Solving Token Gradient Conflict in Mixture-of-Experts for Large Vision-Language Model

## Abstract

Abstract The Mixture-of-Experts (MoE) has gained increasing attention in studying Large Vision-Language Models (LVLMs).
It uses a sparse model to replace the dense model, achieving comparable performance while activating fewer parameters during inference, thus significantly reducing the inference cost.
Existing MoE methods in LVLMs encourage different experts to handle different tokens, and they usually employ a router to predict the routing of each token.
However, the predictions are based solely on sample features and do not truly reveal the optimization directions of tokens.
This may lead to severe optimization interference between different tokens assigned to an expert.
To address this problem, this paper proposes a novel method based on token-level gradient analysis, i.e. , S olving T oken G radient C onflict (STGC).
Specifically, we first use token-level gradients to identify conflicting tokens in experts.
After that, we add a specialized loss tailored to eliminate conflicts among tokens within each expert.
Our method can serve as a plug-in for diverse Large Vision-Language Models, and extensive experimental results demonstrate its effectiveness.
The code will be publicly available at https://github.com/longrongyang/STGC .

## 1 Introduction

Large Vision-Language Models (LVLMs) have recently demonstrated significant advancements by integrating visual processing modules into Large Language Models (LLMs), bringing LLMs closer to Artificial General Intelligence.
Many recent LVLM works (Zhang et al., ; Bai et al., ; Zhang et al., ; Zhao et al., ; Chen et al., ) show that large model size and large dataset size are especially important to enhance intelligence, i.e., the scaling law.
Even when the model size is sufficiently large, models exhibit ”Emergent Abilities”, which are not present in small models but are only present in large models.
Thus, a series of studies (Li et al., ; Dai et al., ; Liu et al., ) have expanded the capacity of LVLMs to 13 billion parameters, leading to state-of-the-art performance on various tasks.

Under realistic applications, deploying such large models requires considerable computational resources, making inference extremely expensive.
For reducing the inference cost, a popular solution is using the Mixture-of-Experts (MoE) architecture.
The MoE, a form of sparsely activated model, has been verified by many works (Fedus et al., ; Zoph et al., ; Komatsuzaki et al., ) to achieve comparable performance with dense models when activating fewer parameters.

With multiple experts in the MoE, a fundamental problem is the routing of tokens.
To route tokens to different experts, existing MoE works (Lin et al., ; Dai et al., ) typically use a router, such as a linear layer, to predict the probability of each token belonging to different experts.
The tokens are then dispatched to the expert with the Top-$k$ predicted probability.
Additionally, to prevent load imbalance, existing methods usually incorporate a load-balancing loss, which aims to prevent the overuse or underuse of some experts.

Figure: Figure 1: (a) In this work, our goal is to resolve interference among diverse data by reducing gradient conflicts among tokens within an expert. (b) We present statistics on the gradient consistency (the mean similarity among gradients of all tokens within an expert). By incorporating the proposed STGC, the gradient consistency increases, thereby verifying that STGC achieves our goal.
Refer to caption: x1.png

Beyond balancing the load, another important goal of routing tokens to different experts is to reduce interference among diverse data.
To achieve this goal, some recent works have performed sample-level clustering on instruction embeddings (Gou et al., ), grouping similar samples sent to the same expert.
However, because routing during training is at the token level, existing methods struggle with interference between tokens within a sample.
Meanwhile, samples with similar embeddings may have distinct optimization objectives, so embedding-based routing is still risky for generating optimization interference among tokens within an expert.
Fortunately, the gradient can directly reveal the optimization direction.
Therefore, in this work, we aim to explore data interference in the MoE through the lens of token-level gradients.
As shown in fig. 1 (a), our basic idea is to reduce gradient conflicts of tokens within an expert, addressing severe data interference during the learning of LVLMs under complex and real-world scenarios.

To this end, we propose a novel regularization loss based on token-level gradients in this paper, for Solving Token Gradient Conflict (STGC).
The proposed STGC consists of two steps:
Specifically:
$(i)$ Conflicting Token Identification.
After processing a batch of data, we perform a backward pass to obtain the token-level gradients for each expert, without updating any model parameters.
Within an expert, we define the average gradient of all tokens as the average gradient, representing the holistic optimization direction of the expert.
Then, all tokens with gradients having negative cosine similarity to the average gradient are identified as conflicting tokens.
$(ii)$ Conflict Elimination Loss.
For the conflicting tokens, we record their routing scores predicted by the router.
We then reduce the routing probabilities of these tokens to other experts.
As shown in fig. 1 (b), the proposed STGC makes the gradient directions of tokens within an expert become more consistent, i.e., reduces the gradient conflicts of tokens within an expert.

In conclusion, our contribution can be summarized as:

- •
Beyond relying on sample-level cues, we propose using token-level gradients to identify optimization interference among tokens within an expert.
- •
We propose a novel conflict elimination loss to resolve gradient conflicts among tokens within an expert, making the gradient directions of tokens within an expert more consistent.
This actually prompts the further specialization of experts.
- •
Designed as a plug-in, our method can be seamlessly integrated into existing Large Vision-Language Models (LVLMs).
Extensive experiments confirm its effectiveness.

## 2 Related Works

### 2.1 Large Vision-language Model

Large Language Models (LLMs) have demonstrated strong instruction following and generalization capabilities.
To maintain these capabilities while incorporating visual information, Large Vision-Language Models (LVLMs) such as GPT-4 and LLaVA utilize frozen visual encoders and trainable visual projectors to integrate visual data into LLMs.
They typically encode visual information into visual tokens and use these tokens to condition the adaptation of language tokens within LLMs (OpenAI, ; Touvron et al., ; Wei et al., ; Touvron et al., ; Zheng et al., ; Team, ; Sun et al., ; Du et al., ; Bai et al., ; Yang et al., ; Penedo et al., ; Taori et al., ).
Recent works have focused on improving performance through two types of methods.
The first type optimizes training strategies, e.g.,  (Bai et al., ; Chen et al., ).
Most works belong to the second type, focusing on enhancing visual components, including expanding visual instruction-tuning datasets (Liu et al., ; Zhang et al., ), improving image encoders (Chen et al., ; Bai et al., ), and aligning the input and projection layers (Lin et al., ; Cha et al., ; Alayrac et al., ; Dai et al., ; Ye et al., ; Zhao et al., ).
These efforts, particularly the expansion of visual instruction-tuning datasets and the increase in model scales, have significantly enhanced the visual understanding abilities of LVLMs.

### 2.2 Mixture-of-Experts (MoE)

The Mixture-of-Experts (MoE) is a hybrid model, consisting of multiple sub-models known as experts, and has shown potential in scaling up models (Shazeer et al., ).
The key concept of MoE lies in the use of a router to determine the token set that each expert handles, aiming for reducing interference among tokens from different types of samples.
Early MoE works have utilized the hard routing mode, where each expert is typically assigned a specific role.
For example, a series of works (Bao et al., ; Long et al., ; Satar et al., ; Wang et al., ; Shen et al., ) consider language and vision gaps in multi-modal data (Liang et al., ), decoupling experts by modal category and assigning a specific role to each expert.
The key feature of hard routers is that they eliminate the need to learn routing assignments.
The hard routing has also been widely applied in task-specific MoEs (Li et al., ; Zhu et al., ; Ma et al., ; Kudugunta et al., ).

Then, soft routers enable a dynamic allocation of tokens among different experts, allowing each expert to focus on its expertise and achieving model sparsity.
Recent LLM (Shazeer et al., ; Lepikhin et al., ; Fedus et al., ; Zoph et al., ; Komatsuzaki et al., ) and LVLM works have mainly focused on soft routers.
For instance, Gshard (Lepikhin et al., ) incorporates MoE into transformers and achieves excellent performance.
Lifelong-MoE (Chen et al., ) uses MoE to mitigate the challenge of catastrophic forgetting in lifelong learning.
MoE-LLaVA (Lin et al., ) and LLaVA-MoLE (Chen et al., ) utilize MoE and its variants to empower LVLMs.
The approach of expert segmentation increases the number of experts to achieve greater specialization.
DeepSeekMoE (Dai et al., ) and QwenMoE (Bai et al., ) segment experts by splitting the FFN intermediate hidden dimension.
Cluster-based methods, such as MoCLE (Gou et al., ), cluster samples and then route those in the same cluster to the same expert.
DEMIX (Gururangan et al., ) clusters samples according to their task type, ensuring that samples in the same cluster are routed to the same expert.
Existing methods primarily operate at the sample level and rely solely on either features or labels, which makes it challenging to address interference between different tokens within a sample. This work aims to utilize token-level gradients to identify and resolve token optimization interference within an expert in the MoE.

## 3 Methodology

Figure: Figure 2: Our pipeline. (a) Conflicting Token Identification. When the gradient of a token has a sufficiently low cosine similarity to the average gradient of its assigned expert, this token is marked as a conflicting token. (b) Conflict Elimination Loss. We propose a loss term aimed at discouraging the routing of conflicting tokens to their assigned experts, achieved by reducing the routing scores for these tokens for their assigned experts.
Refer to caption: x2.png

### 3.1 Overview

Large Vision-Language Model: A Large Vision-Language Model (LVLM) aims to effectively integrate the capabilities of the pre-trained LLM and a visual model.
Specifically, given a RGB image $\mathbf{v}\in\mathbb{R}^{H\times W\times 3}$, where $H$ and $W$ are its height and width, the vision encoder processes the input image to obtain a visual token sequence $\mathcal{Z}=[z_{1},z_{2},\cdots,z_{P}]\in\mathbb{R}^{P\times C}$, where $P$ is the sequence length of visual tokens, calculated as $P=\frac{H\times W}{14^{2}}$.
A visual projection layer is then used to map $\mathcal{Z}\in\mathbb{R}^{P\times C}$ to $\mathcal{V}\in\mathbb{R}^{P\times D}$, where $D$ represents the hidden layer size of Large Language Model (LLM).
Similarly, the text undergoes word embedding by layer $g$ and is projected to obtain the sequence tokens $\mathcal{T}=[t_{1},t_{2},\cdots,t_{N}]\in\mathbb{R}^{N\times D}$, where $N$ represents the sequence length of text tokens.
Subsequently, the visual and text tokens are concatenated together and fed into a large language model.
This model consists of stacked multi-head self-attention (MSA) and feed-forward neural networks (FFN), with layer normalization (LN) and residual connections typically used within each block:

$$ $\mathbf{x}_{0}=[v_{1},v_{2},\cdots,v_{P},\cdots,t_{1},t_{2},\cdots,t_{N}],$ (1) $$

$$ $\mathbf{x}_{\ell}^{\prime}=\mathrm{MSA}(\mathrm{LN}(\mathbf{x}_{\ell-1}))+ \mathbf{x}_{\ell-1},\ell\in\{1,\ldots,L\},$ (2) $$

$$ $\mathbf{x}_{\ell}=\mathrm{FFN}(\mathrm{LN}(\mathbf{x^{\prime}}_{\ell}))+ \mathbf{x^{\prime}}_{\ell},\ell\in\{1,\ldots,L\},$ (3) $$

$$ $\mathcal{Y}=\mathrm{LN}(\mathbf{x}_{L}),$ (4) $$

where $L$ is the layer number of LLM.
The LVLM model generates an output text sequence $\mathcal{Y}=[y_{1},y_{2},\cdots,y_{K}]\in\mathbb{R}^{K\times D}$ by progressively generating each element, where $K=P+D$ represents the total length of the output text sequence.
Then, the outputs are optimized through a generative loss in an auto-regressive manner.
The loss is formulated as:

$$ $\mathcal{L}_{\text{main}}=-\sum_{i=1}^{D}\text{log}\ p_{\theta}\left(\mathcal{ Y}^{[P+i]}\mid\mathcal{V},\mathcal{T}^{[:i-1]}\right),$ (5) $$

where $\theta$ is a trainable parameter.
The auto-regressive loss for the token $t_{n}$ is abbreviated as $\mathcal{L}_{n}(\theta)$.

MoE: The Mixture-of-Expert (MoE) layer is used to replace the FFN layer, e.g.,  (Dai et al., ).
A MoE layer consists of multiple FFNs, each representing an expert, i.e., $\mathcal{E}=[e_{1},e_{2},\cdots,e_{E}]$, where $E$ is the number of experts.
The router is typically a linear layer that predicts the probability of each token being assigned to each expert, and we formulate this process as:

$$ $p_{\text{moe}}(\mathbf{x})_{i}=\frac{e^{z_{\text{moe}}(\mathbf{x})_{i}}}{\sum_ {j=1}^{E}e^{z_{\text{moe}}(\mathbf{x})_{j}}},$ (6) $$

where $z_{\text{moe}}(\mathbf{x})=\mathbf{W}\cdot\mathbf{x}$ and $p_{\text{moe}}(\mathbf{x})_{i}$ is the routing score of $\mathbf{x}$ for the $i$-th expert.
The matrix $\mathbf{W}\in\mathbb{R}^{D\times E}$ represents the lightweight training parameters for routing.
We calculate a weighted sum of the outputs from the Top-$k$ experts with the highest softmax probabilities, where the weighting of each expert is related to the routing score:

$$ $\displaystyle w_{\text{moe}}(\mathbf{x})_{i}$ $\displaystyle=\frac{e^{z_{\text{moe}}(\mathbf{x})_{i}}}{\sum_{j=1}^{k}e^{z_{ \text{moe}}(\mathbf{x})_{j}}},$ (7) $\displaystyle\mathrm{MoE}(\mathbf{x})$ $\displaystyle=\sum_{i=1}^{k}w_{\text{moe}}(\mathbf{x})_{i}\cdot e_{i}(\mathbf{ x}),$ $$

where $w_{\text{moe}}(\mathbf{x})_{i}$ represents the weight of the $i$-th expert for $\mathbf{x}$, and $e_{i}(\mathbf{x})$ is the output of the $i$-th expert.
We express $\mathcal{L}_{n}(\theta)$ as $\mathcal{L}_{n}(\theta_{e_{i}},\theta^{\prime})$, where $\theta_{e_{i}}$ denotes the $i$-th expert, and $\theta^{\prime}$ represents all other parameters except for $\theta_{e_{i}}$.

Our Method: In this work, our goal is to propose a novel learning strategy for the MoE architecture for reducing interference among diverse data.
Specifically, as illustrated in Figure 2, we model the interference among tokens within an expert using token-level gradients, and then design a novel loss function that requires tokens with conflicting gradients to be handled by different experts.
The details of these modules will be introduced in the subsequent sections.

### 3.2 Conflicting Token Identification

For the MoE, the key to reducing interference among diverse data is to prevent optimization interference between tokens within an expert.
A possible approach is to cluster samples based on their features, using the clustering results to decide which expert the samples should be assigned to.
Alternatively, decisions can be made based on the specific task associated with each sample.
However, these approaches have two main limitations: $(i)$ They operate at the sample level, whereas the routing is at the token level; routing all tokens within a sample to the same expert does not effectively address interference between tokens within the sample.
$(ii)$ The optimization direction is jointly influenced by features and labels, but these methods rely on only one of these two factors.
To address these issues, we propose using token-level gradients, which can accurately depict optimization directions at the token level, to identify optimization interference between tokens within an expert.

First, we introduce the negative impact brought by the gradient conflicts.
Without loss of generality, we discuss two distinct text tokens, $t_{n}$ and $t_{n^{\prime}}$, as shown in Figure 2 (a).
Assume that both $t_{n}$ and $t_{n^{\prime}}$ are processed by the expert $e_{i}$.
Let $\mathbf{g}_{n}=\nabla_{\theta_{e_{i}}}\mathcal{L}_{n}(\theta_{e_{i}},\theta^{
\prime})$ denote the gradient of the token $t_{n}$ with respect to the expert $\theta_{e_{i}}$.
A small change in $\theta_{e_{i}}$ in the direction of $-\mathbf{g}_{n}$ is given by $\theta_{\mathrm{e_{i}}}\leftarrow\theta_{\mathrm{e_{i}}}-\delta\mathbf{g}_{n}$, with a step size $\delta$.
The effect of this change on the performance of another token $t_{n^{\prime}}$ is measured by:

$$ $\Delta\mathcal{L}_{n^{\prime}}=\mathcal{L}_{n^{\prime}}(\theta_{e_{i}}-\delta \mathbf{g}_{n},\theta^{\prime})-\mathcal{L}_{n^{\prime}}(\theta_{e_{i}},\theta ^{\prime})=-\delta\mathbf{g}_{n}\cdot\mathbf{g}_{n^{\prime}}+o(\delta),$ (8) $$

where the second equality is obtained by first-order Taylor approximation.
Likewise, the effect of an update of $\theta_{e_{i}}$ in the direction of the negative gradient of token $n^{\prime}$ (i.e., $-\mathbf{g}_{n^{\prime}}$) on the performance of token $n$ is $\Delta\mathcal{L}_{i}=-\delta\mathbf{g}_{n}\cdot\mathbf{g}_{n^{\prime}}+o(\delta)$.
Thus, the model update for token $n$ is considered to negatively affect token $n^{\prime}$ when $\mathbf{g}_{n}\cdot\mathbf{g}_{n^{\prime}}<0$, since it increases the loss of token $n^{\prime}$, and vice versa.
We define $\mathbf{g}_{n}$ and $\mathbf{g}_{n^{\prime}}$ as conflicting gradients when their cosine similarity $\cos{\phi_{nn^{\prime}}}<\tau$, where $\tau$ is a threshold and $\phi_{nn^{\prime}}$ is the angle between $\mathbf{g}_{n}$ and $\mathbf{g}_{n^{\prime}}$.
Gradient conflicts can cause the optimizer to struggle to converge to a desirable solution, especially when there is a large difference in gradient magnitudes.

We then define the conflicting token.
The parameters of the expert $e_{i}$ are updated using the average gradient.
Let the tokens processed by the expert $e_{i}$ be denoted as $\{t_{1},\cdots,t_{N_{e_{i}}}\}$, the average gradient on the expert $e_{i}$ is represented as:

$$ $\mathbf{g}_{mean}=\frac{\sum_{n=1}^{N_{e_{i}}}\mathbf{g}_{n}}{N_{e_{i}}}.$ (9) $$

The average gradient indicates the direction of parameter updates for the expert at each iteration.
When the gradient of a token and the average gradient are conflicting gradients, it suggests that the token is detrimental to the learning of the expert $e_{i}$, so his token should be considered for assignment to another expert.
A formal definition of a conflicting token is provided as follows:

###### Definition 1 (Conflicting Token)

The token $t_{n}$ is said to a conflicting token if $\mathbf{g}_{n}$ and $\mathbf{g}_{mean}$ are conflicting gradients, where $\mathbf{g}_{mean}$ is the average gradient of all tokens in the expert of $t_{n}$.

Lastly, we detail our method for identifying conflicting tokens, as illustrated in Figure 2(a).
Initially, we unfreeze only the expert layer in the MoE and compute the main loss.
We then perform back-propagation to calculate the token-level gradients for each token within the expert layer.
Subsequently, we calculate the average gradient, as well as the cosine similarity between the gradient of each token and the average gradient.
Lastly, when the cosine similarity is less than $\tau$, we mark the token as a conflicting token.
Identifying these tokens allows us to use the precise optimization direction represented by the gradients to reduce interference among diverse data in the next section.

###### Definition 1 (Conflicting Token)

### 3.3 Conflict Elimination Loss

The learning of a conflicting token tends to increase the loss of most other tokens within its corresponding expert.
Thus, once a conflicting token is identified, it should be reassigned to a different expert for processing.
To achieve this goal, we propose a simple yet effective regularization loss by constraining the routing scores predicted by the router, as shown in Figure 2(b).

Specifically, for each expert within every layer, we first identify the conflicting tokens using token-level gradients.
Then, the router predicts the probability of each token being assigned to different experts.
For a conflicting token $t_{n}$, we record the routing logits $z_{\text{moe}}(t_{n})$, the routing scores $p_{\text{moe}}(t_{n})$, and the expert ID $id_{\text{moe}}$ it is currently assigned to.
Using the expert ID $id_{\text{moe}}$, we calculate the loss:

$$ $\displaystyle z^{\prime}_{\text{moe}}(t_{n})$ $\displaystyle=-z_{\text{moe}}(t_{n}),$ (10) $\displaystyle p^{\prime}_{\text{moe}}(t_{n})_{i}$ $\displaystyle=\frac{e^{z^{\prime}_{\text{moe}}(t_{n})_{i}}}{\sum_{j=1}^{E}e^{z ^{\prime}_{\text{moe}}(t_{n})_{j}}},$ $\displaystyle\mathcal{L}_{\text{token}}$ $\displaystyle=\frac{1}{N\cdot E}\sum_{n=1}^{N}\sum_{i=1}^{E}\text{log}(p^{ \prime}_{\text{moe}}(t_{n})_{i})\cdot q_{\text{moe}}(t_{n})_{i},$ $$

Where $N$ is the count of all conflicting tokens, $E$ is the number of experts, and $p^{\prime}_{\text{moe}}(t_{n})$ represents the inverted routing score for the token $t_{n}$.
The ${q}_{\text{moe}}(t_{n})$ define one-hot vectors, with ${q}_{\text{moe}}(t_{n})_{id_{\text{moe}}}=1$.
This loss is designed to encourage the reassignment of conflicting tokens to different experts.
When $k$ in Top-$k$ exceeds 1, a token may be assigned to multiple experts.
Our method focuses on considering all tokens within each expert, regardless of whether a token has also been assigned to other experts.

### 3.4 Total Loss

To encourage experts to handle tokens in a balanced manner, the differentiable load balancing loss, as introduced in (Fedus et al., ), is typically defined for each MoE layer as follows:

$$ $\mathcal{L}_{\text{aux}}=E\cdot\sum_{i=1}^{E}\mathcal{F}_{i}\cdot\mathcal{P}_{ i},$ (11) $$

where $\mathcal{F}$ represents the fraction of tokens processed by each expert $e_{i}$, and $\mathcal{P}$ represents the average routing probabilities assigned to expert $e_{i}$.

In conclusion, the total loss is given by:

$$ $\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{main}}+\alpha\cdot\mathcal{L}_{ \text{aux}}+\beta\cdot\mathcal{L}_{\text{token}},$ (12) $$

where $\alpha$ and $\beta$ are hyper-parameters.

## 4 Experiments

**Table 1: Comparison among different LVLMs on image understanding benchmarks. “Act.”, “V”, “Q”, “P”, “M”, and “S” represent activated parameters, Vicuna (Chiang et al., ), Qwen (Bai et al., ), Phi-2 (Microsoft, ), MobileLLaMA (Chu et al., ), and StableLM (), respectively. Evaluation Benchmarks include VQA${}^{\text{v2}}$ (Goyal et al., ); GQA (Hudson & Manning, ); VisWiz (Gurari et al., ); SQA${}^{\text{I}}$: ScienceQA-IMG (Lu et al., ); VQA${}^{\text{T}}$: TextVQA (Singh et al., ); POPE (Li et al., ); MME (Fu et al., ); MMB: MMBench (Liu et al., ); MM-Vet (Yu et al., ); AI2D (Kembhavi et al., ); ChartQA (Masry et al., ); DocVQA (Mathew et al., ). ^∗ indicates that there is some overlap in the training data. ^† indicates that a stronger visual encoder (siglip-so400m-patch14-384) is used. All “Sparse Model” methods employ the configure 4-Top-2.**
| Method | LLM | Act. | VQA${}^{\text{v2}}$ | GQA | VisWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | POPE | MME | MMB | MM-Vet | AI2D | ChartQA | DocVQA | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Dense Model |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| LLaVA-1.5 | V-13B | 13B | 80.0^∗ | 63.3^∗ | 53.6 | 71.6 | 61.3 | 85.9 | 1531.3 | 67.7 | 35.4 | 49.6 | 18.1 | 24.0 | 55.5 |
| Qwen-VL | Q-7B | 6.7B | 78.8^∗ | 59.3^∗ | 35.2 | 67.1 | 63.8 | - | - | 38.2 | - | - | - | - | - |
| LLaVA-1.5 | V-7B | 6.7B | 78.5^∗ | 62.0^∗ | 50.0 | 66.8 | 58.2 | 85.9 | 1510.7 | 63.4 | 30.5 | - | - | - | - |
| TinyGPT-V | P-2.7B | 2.7B | - | 33.6^∗ | 33.4 | - | - | - | - | - | - | - | - | - | - |
| MobileVLM | M-2.7B | 2.7B | - | 59.0^∗ | - | 61.0 | 47.5 | 84.9 | 1288.9 | 59.6 | - | - | - | - | - |
| LLaVA-Phi | P-2.7B | 2.7B | 71.4^∗ | - | 35.9 | 68.4 | 48.6 | 85.0 | 1335.1 | 59.8 | 28.9 | - | - | - | - |
| Sparse Model |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MoE-LLaVA | S-1.6B | 2.0B | 76.7^∗ | 60.3^∗ | 36.2 | 62.6 | 50.1 | 85.7 | 1318.2 | 60.2 | 26.9 | 48.8 | 15.3 | 18.4 | 49.2 |
| MoE-LLaVA | P-2.7B | 3.6B | 77.6^∗ | 61.4^∗ | 43.9 | 68.5 | 51.4 | 86.3 | 1423.0 | 65.2 | 34.3 | 58.8 | 19.9 | 21.5 | 53.5 |
| DYNMOE-LLaVA | P-2.7B | 3.4B | 77.9^∗ | 61.6^∗ | 45.1 | 68.0 | 51.8 | 86.0 | 1429.6 | 66.6 | 33.6 | - | - | - | - |
| MoE-LLaVA^† | P-2.7B | 3.6B | 79.9^∗ | 62.6^∗ | 43.7 | 70.3 | 57.0 | 85.7 | 1431.3 | 68.0 | 35.9 | 59.5 | 15.4 | 25.6 | 54.9 |
| Our Method^† | P-2.7B | 3.6B | 80.0^∗ | 63.0^∗ | 48.6 | 70.9 | 58.8 | 86.5 | 1481.7 | 71.0 | 40.7 | 64.5 | 44.7 | 42.1 | 61.0 |

### 4.1 Experimental Setup

Benchmark: In this work, we follow existing works (Liu et al., ; Lin et al., ) to evaluate the proposed STGC.
STGC is only used in the instruction tuning stage.
Some academic-task-oriented and instruction-following benchmarks are collected for evaluating the LVLMs.
For academic-task-oriented benchmarks, VQA-v2 (Goyal et al., ) and GQA (Hudson & Manning, ) assess the visual perception capabilities of models through open-ended short answers.
The VizWiz dataset (Gurari et al., ), containing 8,000 images, evaluates the zero-shot generalization of models on visual questions asked by visually impaired people.
ScienceQA (Lu et al., ), a multiple-choice benchmark, evaluates the zero-shot generalization of models on scientific question answering.
TextVQA (Singh et al., ) focuses on text-rich visual question answering tasks.
ChartQA (Masry et al., ) focuses on visual and logical reasoning tasks over charts through open-ended short answers.
DocVQA (Mathew et al., ) focuses on reading comprehension tasks over document images through open-ended short answers.

For instruction-following benchmarks, POPE (Li et al., ) evaluates the degree of hallucination in model responses on three sampled subsets of COCO (Lin et al., ): Random, Common, and Adversarial.
MME (Fu et al., ) assesses the visual perception of models with yes/no questions.
MMBench (Liu et al., ) evaluates the robustness of model answers with all-round shuffling on multiple choice answers.
MM-Vet (Yu et al., ) evaluates the model capabilities in engaging in visual conversations on a diverse range of tasks, and evaluates the correctness and helpfulness of the responses using the GPT-4 evaluation framework.
AI2D (Kembhavi et al., ), a multiple-choice benchmark, evaluates the model capabilities for science diagram comprehension.

Baseline: Our main baseline is set to MoE-LLaVA (Lin et al., ) in this paper.
MoE-LLaVA incorporates a Mixture-of-Experts (MoE) into LVLMs, and has proposed a three-stage training scheme.
It trains only the MoE in the third stage, i.e., the instruction tuning stage.
MoE-LLaVA has 4 experts and selects the Top-2 experts to handle tokens, and we refer to this configuration as 4-Top-2.
Building on MoE-LLaVA, we add a novel regularization loss $\mathcal{L}_{\text{token}}$ in this work during the instruction tuning stage to enhance the MoE.
For the language model backbone, we follow MoE-LLaVA (Lin et al., ) to use StableLM-1.6B and Phi2-2.7B.
The visual encoder is typically set as clip-vit-large-patch14-336.
$\alpha$ is set to 0.01, following MoE-LLaVA (Lin et al., ).
We also compare with DYNMOE-LLaVA (Guo et al., ), which improves the MoE-LLaVA by dynamically setting the selected expert count during inference.

### 4.2 Image Understanding Evaluation

Image Question Answering: We evaluate the performance of our method on five image question-answering benchmarks, as shown in Table 1, and report the number of activated parameters as a measure of efficiency.
Our method demonstrates superior image understanding capabilities, achieving the 80.0%, 63.0%, 48.6%, 70.9%, and 58.8% performance on VQA${}^{\text{v2}}$, GQA, VisWiz, SQA${}^{\text{I}}$, and VQA${}^{\text{T}}$, respectively.
Compared to LLaVA-1.5 with 7B activated parameters, our method brings 0.6%, 1.0%, 4.1%, and 0.8% performance increase on VQA${}^{\text{v2}}$, GQA, SQA${}^{\text{I}}$, and VQA${}^{\text{T}}$, respectively, when only activating 3.6B parameters.

Benchmark Toolkit:
To comprehensively evaluate the multi-modal understanding capabilities of our method, we assess its performance across four benchmark toolkits.
These toolkits typically involve open-ended answers and serve as tools to verify the model ability to engage in natural language questioning.
As shown in Table 1, our method demonstrates superior image understanding capabilities, achieving the 86.5%, 1481.7, 71%, and 40.7% performance on VQA${}^{\text{v2}}$, GQA, VisWiz, SQA${}^{\text{I}}$, and VQA${}^{\text{T}}$, respectively.
Compared to LLaVA-1.5 with 7B activated parameters, our method brings 0.6%, 7.6%, and 10.2% performance increase on POPE, MMB, and MM-Vet, respectively, when only activating 3.6B parameters.

### 4.3 Ablation Study

STGC as a plug-in: We design the proposed STGC as a plug-in for existing MoE methods.
To fully verify the robustness and effectiveness of the proposed STGC, we select different baselines, including different MoE configures (4-Top-2 & 4-Top-1), LLMs (S-1.6B & P-2.7b), and visual encoders (clip & siglip).
Then, we add STGC onto these baselines.
As shown in table 2, under diverse baselines, adding STGC always brings a stable and convincing performance increase.

**Table 2: STGC as a plug-in. We set different baselines and add the proposed STGC. 4-Top-1 means that 4 experts are set and the Top-1 expert is selected to handle tokens. ^∗ indicates that there is some overlap in the training data. ^† indicates that a stronger visual encoder (siglip) is used.**
| Method | LLM | Act. | VQA${}^{\text{v2}}$ | GQA | VisWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | POPE | MME | MMB | MM-Vet | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MoE-LLaVA-4-Top-1 | S-1.6B | - | 74.5^∗ | 58.6^∗ | 25.7 | 55.8 | 45.0 | 85.2 | 1245.3 | 56.2 | 27.2 | 53.5 |
| +STGC | S-1.6B | - | 74.9^∗ | 59.4^∗ | 27.4 | 57.5 | 46.5 | 85.8 | 1276.8 | 56.8 | 28.5 | 54.6 |
| MoE-LLaVA-4-Top-2 | S-1.6B | 2.0B | 76.7^∗ | 60.3^∗ | 36.2 | 62.6 | 50.1 | 85.7 | 1318.2 | 60.2 | 26.9 | 57.3 |
| +STGC | S-1.6B | 2.0B | 76.9^∗ | 60.9^∗ | 37.7 | 62.6 | 50.7 | 85.9 | 1355.1 | 60.7 | 28.2 | 58.0 |
| MoE-LLaVA-4-Top-2 | P-2.7B | 3.6B | 77.6^∗ | 61.4^∗ | 43.9 | 68.5 | 51.4 | 86.3 | 1423.0 | 65.2 | 34.3 | 61.1 |
| +STGC | P-2.7B | 3.6B | 78.0^∗ | 62.1^∗ | 47.2 | 68.1 | 52.3 | 86.9 | 1429.2 | 66.7 | 33.3 | 61.8 |
| MoE-LLaVA-4-Top-2^† | P-2.7B | 3.6B | 79.9^∗ | 62.6^∗ | 43.7 | 70.3 | 57.0 | 85.7 | 1431.3 | 68.0 | 35.9 | 62.9 |
| +STGC | P-2.7B | 3.6B | 80.3^∗ | 63.2^∗ | 45.1 | 70.3 | 57.4 | 86.1 | 1447.6 | 69.7 | 35.7 | 63.5 |

**Table 3: Study about Conflicting Token Identification. Settings for results in Table 2 are highlighted in blue. The best results are indicated by boldface. StableLM-1.6B is used.**
| Strategy | GQA | VisWiz | VQA${}^{\text{T}}$ | MMB | MM-Vet |
| --- | --- | --- | --- | --- | --- |
| embedding-based | 56.1 | 35.4 | 48.7 | 60.6 | 25.2 |
| gradient-based | 60.9 | 37.7 | 50.7 | 60.7 | 28.2 |

Study about Conflicting Token Identification:
Our method uses token-level gradients as the cue to identify conflicting tokens.
To verify the superiority of token-level gradients as the cue over sample-level embeddings, we conduct experiments:
$(i)$ Sample-level expert labels, based on clustering for instruction embeddings, which is similar to (Chen et al., ).
$(ii)$ Token-level expert labels, based on the token-level gradients in each expert.
Besides, when the gradient $\mathbf{g}_{n}$ of the token $t_{n}$ and and the average gradient $\mathbf{g}_{mean}$ satisfy the condition $\cos{\phi_{nmean}}<\tau$, we flag the token as a conflicting token.
We discuss different thresholds for identifying conflicting tokens:
$\tau\in\{0.1,0.0,-0.1\}$.

As shown in Table 3, we find:
$(i)$ The performance using token-level gradients to identify conflicting tokens is significantly higher than using sample-level embedding clusters.
For example, our method achieves a 4.8% higher performance improvement over the embedding-based scheme on GQA.
(ii) When $\tau=0$, the performance on most datasets is the best.
The results are consistent with the common belief, i.e., gradients are considered conflicting when their cosine similarity is less than 0.

**Table 4: Study about Conflict Elimination Loss. Settings for results in Table 2 are highlighted in blue. The best results are indicated by boldface. StableLM-1.6B is used.**
| Layer | GQA | VisWiz | VQA${}^{\text{T}}$ | MMB | MM-Vet |
| --- | --- | --- | --- | --- | --- |
| 0-24 | 60.9 | 37.7 | 50.7 | 60.7 | 28.2 |
| 0-12 | 60.7 | 35.2 | 50.9 | 61.9 | 25.5 |
| 12-24 | 60.5 | 34.9 | 50.7 | 61.6 | 26.7 |

Study about Conflict Elimination Loss:
We propose the Conflict Elimination Loss to reduce interference among diverse data types.
We explore the impact of applying the loss at different layers, with a total of 24 layers:
$(i)$ All layers (0-23).
$(ii)$ The second half of the layers (12-23).
$(iii)$ The first half of the layers (0-11).
We also consider different loss weightings $\beta\in\{0.5,1.0,2.0\}$.

As shown in Table 4, we find:
$(i)$ Applying the proposed loss to all layers yields the best performance on most datasets, verifying its importance across all layers.
$(ii)$ The proposed loss is not sensitive to different loss weightings $\beta$, with the highest performance on most datasets when $\beta$ is set as 1.0.

Figure: Figure 3: Statistical Verification. We conduct a deep analysis of the role of STGC. “Baseline” means MoE-LLaVA. “Baseline + STGC” means our method. (a) We present the change of routing scores of conflicting tokens on their current experts, to verify that STGC is moving conflicting tokens away from their current experts. (b) We compute a novel metric, gradient consistency (the mean similarity among gradients of all tokens within an expert), for verifying that the decrease of the proposed loss leads to the more consistent token gradients within an expert. (c) We further analyze the gradient consistency on different layers.
Refer to caption: x3.png

Scalability of STGC: One necessary hypothesis of this work is that there are severe optimization interference among diverse training data.
Therefore, STGC should bring a more significant performance increase when using a more complex dataset to train.
To verify this, we expand the size of the public dataset we are using for further experiments.
Specifically, we use the training data provided in Open-LLaVA-NeXT(^*^**[https://github.com/xiaoachen98/Open-LLaVA-NeXT](https://github.com/xiaoachen98/Open-LLaVA-NeXT)), i.e., open-llava-next_instruct_mix1M.
As shown in table 5, under more data, STGC brings a more significant average performance increase (1.1% vs. 0.6%).

**Table 5: Scalability of STGC. We consider to employ the larger scale of data. Under more data, the STGC can brings the more significant performance increase.**
| Method | LLM | Data | VQA${}^{\text{v2}}$ | GQA | VisWiz | SQA${}^{\text{I}}$ | VQA${}^{\text{T}}$ | POPE | MME | MMB | MM-Vet | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| MoE-LLaVA-4-Top-2^† | P-2.7B | 665K | 79.9^∗ | 62.6^∗ | 43.7 | 70.3 | 57.0 | 85.7 | 1431.3 | 68.0 | 35.9 | 62.9 |
| +STGC | P-2.7B | 665K | 80.3^∗ | 63.2^∗ | 45.1 | 70.3 | 57.4 | 86.1 | 1447.6 | 69.7 | 35.7 | 63.5 |
| MoE-LLaVA-4-Top-2^† | P-2.7B | 1021K | 79.7^∗ | 63.0^∗ | 42.7 | 71.1 | 56.9 | 84.3 | 1439.9 | 70.4 | 42.2 | 63.8 |
| +STGC | P-2.7B | 1021K | 80.0^∗ | 63.0^∗ | 48.6 | 70.9 | 58.8 | 86.5 | 1481.7 | 71.0 | 40.7 | 64.9 |

Statistical Verification:
The main claim of this work is that STGC can effectively reduce the gradient conflicts of tokens within an expert, i.e., making the gradient directions of tokens within an expert more consistent, thus decreasing data interference and improving model performance.
To verify this, we design a statistical verification experiment.
In this experiment, the training consists of two steps:
The first step is to capture the gradient similarities of tokens within an expert.
In detail, we gather all tokens within an expert $e_{i}$ and compute their cosine similarity to form a similarity matrix.
Then, we compute the mean of the similarity matrix as $\text{mean}(\cos{\phi_{i}})$.
We define the average of $\text{mean}(\cos{\phi_{i}})$ on all experts as gradient consistency, serving as a novel metric to evaluate whether the gradient directions are consistent within an expert.
The second step is to update models.
In this experiment, we only use the proposed loss to update parameters, for undisturbedly observing the impact of the proposed loss to the gradient consistency.

We first provide the change in routing scores of conflicting tokens on their corresponding experts in fig. 3 (a).
Then, we provide the curve graph of the proposed loss and the gradient consistency obtained from the TensorBoard dashboard in fig. 3 (b).
We also provide the gradient consistency on different layers; The total layer number is 32, and we analyze the layers {0, 8, 16, 24}.
We find:
$(i)$ Our method significantly reduces the mean routing score from 0.3866 to 0.3349, verifying that our method encourages conflicting tokens to be assigned to other experts instead of the current ones.
$(ii)$ By reducing the mean routing score of conflicting tokens, the gradient consistency significantly improves, indicating that the token gradient directions within an expert become more consistent.
This verifies the role of STGC in reducing gradient conflicts.
$(iii)$ For different layers, STGC always increases the gradient consistency.
The deeper layers seem to have higher gradient consistency after adding STGC while the shallower layers have lower gradient consistency.

Figure: Figure 4: Expert Loading. We select three validation datasets, i.e., VizWiz (Gurari et al., ), SQA (Lu et al., ), and TextVQA (Singh et al., ), to analyze the expert loading. “Baseline” means MoE-LLaVA. “Baseline + STGC” means our method.
Refer to caption: x4.png

Expert Loading: We present some statistical results about the expert loading during inference on several datasets.
As shown in fig. 4, STGC benefits the expert load balance.
A possible reason is that the expert load imbalance means that many tokens are routed to an expert, significantly increasing the possibility that tokens have gradient conflicts.
By adding STGC, many tokens are moved to the expert with fewer tokens.
This may also be a new perspective on why the expert load balance is important to the MoE system.

## 5 Conclusion

Our study reveals that there is still severe optimization interference of tokens within an expert for the MoE, leading to sub-optimal learning for the experts.
To reduce optimization interference among tokens, we propose employing token-level gradients to identify conflicting tokens, and then adding a novel conflict elimination loss based on the routing scores.
Our method acts as a plug-in, which can be easily integrated into existing MoE-based LVLMs.
Extensive experiments demonstrate the superior performance of our approach across diverse datasets.
Especially when the data diversity is larger, our method brings a more significant performance increase.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Will it introduce additional training costs for the computation of the gradient and similarity?
- It would be better to provide some analysis on the distribution of expert loadings for better understanding.
soundness: 3
strengths: - The paper is well-written and easy to follow. The motivation for solving token-level gradient conflict during training MOE is sound to me.
- The proposed CEL can serve as a direct plug-in to improve the training of MOEs.
- A consistent improvement can be observed in Table 2 and 3, and the proposed STGC can be applicable to a wide range of model and data settings.
- The evaluation of gradient consistency in Figure 3 also provides further evidence to account for the improvement.
summary: This paper introduces an approach to improve the Mixture-of-Expert(MOE) by eliminating token-level gradient conflict during training. The gradient of the token is compared with the average gradient to decide the conflict tokens. A Conflict Elimination Loss (CEL) is proposed as a regularization term to encourage the reassignment of conflicting tokens. The results on various model setting and tasks validate the effectiveness of proposed method.
weaknesses: - It would be better to illustrate the distribution of cosine similarity between gradients of all tokens and the averaged gradient for better understanding.
- Since the average gradient is dynamically changing during training, is it possible that some tokens do not conflict with a specific expert but become conflicting after several steps of training? 
- Why only the Large Vision-Language Model (LVLM) setting is considered? 

- Some minor points:
  - Figure 3b: better label the x-axis with "Training Step"
  - In TL;DR, a novel loss.
  - Figure A in supplementary: part of the figure is not visible

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 2
questions: See weakness section.
soundness: 2
strengths: (1) The proposed approach is novel and is effective in improving the model performance.
(2) Resolving conflicting-token demonstrates a strong correlation with improvement of the performance, which makes it a seemingly good metric to study MoE.
(3) Thorough empirical results are provided to illustrate the effectiveness of the method. Sensitivity analysis are provided to show the robustness of the method.
summary: This paper identifies conflicting token as one of the major problem in MoE-based LVLM training. Furthermore, the author propose a novel Conflicting-Elimination-Loss to resolve the conflict. By applying the proposed loss, it provides performance improvements over baseline.
weaknesses: (1) Visualization on the routing mechanism of the trained model should be studied to provide insights into the trained model. By conducting such visualization, we might be able to know better why the proposed method can yield performance improvements. In addition, this could also validate the proposed method would still effectively utilize every one of the expert, instead of collapsing into only using one expert.
(2) It seems that the optimization process of the proposed CEL is not very stable (as in Figure 3(a)), the reviewer wonders whether this would be a problem when scaling up to larger scale.
(3) Computational overhead should be provided to ensure the proposed method does not incur too much overhead. This added study can show whether the proposed method is promising in terms of scaling up to large scale.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weaknesses section for questions as well.
soundness: 3
strengths: **Relevance**

Routing has been a difficult task, which is pertinent to the Mixture of Experts framework. However, the understanding and analysis of the router predictions has been lacking in the literature. I believe this paper takes a significant step towards identifying and addressing a potential issue that leads to hindrance in learning in the MOE framework. The authors define the notion of a conflicting token, and then addresses it well using their proposed method, comprehensively justifying the performance in a variety of scenarios.

**Presentation**

I enjoyed reading the paper, it was easy to follow (except some parts listed in weaknesses), introduces the problem well, and presents detailed results and ablations to justify their proposed method. I specifically appreciate the statistical verification in Figure 3 along with the results which indicates that token conflict is actually a problem and resolving it leads to better performance.

**Practicality**

The section on memory considerations and the engineering tricks for reducing memory overheads is very helpful and leads to a much practical implementation of the proposed method.
summary: The authors make a very interesting and useful observation regarding the inconsistency/randomness in gradient direction for tokens routed to a particular expert in the Mixture of Experts framework. They then propose adding a loss term that aims to align the expert-wise token gradients to the expert-wise mean gradient over all tokens. The results presented by the authors suggest that the gradient alignment correlates with better performance, and also provide ablations showing that their method actually gets the token gradients to align.
weaknesses: I will combine and list the weaknesses and questions here. 

**Practical Implementation**: While sec A in the supplementary compares the practical implementation with the heavy one in terms of Pearson correlation coefficient between the similarity metrics, I believe additional considerations 
- a quantitative measure of the memory overhead reduction (if not possible then a qualitative measure)
- performance difference on an actual dataset
could make the claim of practicality even stronger. Specific experiments or analysis in the main paper that quantifies the memory savings and any potential trade-offs in performance when using their practical implementation versus the full gradient computation would be helpful.

**Token Conflicts after STGC**: Fig 3 indicates the increase in gradient alignment over the course of the training process, but it would be interesting to understand the tokens that remain conflicting even after STGC. What proportion of the total tokens stay conflicting? Can something be reasoned about this behaviour? Along with the mean gradient consistency, it would be good to report the std deviations as well. It might also be worth tracking the proportion of total tokens that stay conflicting over the course of training. The authors' insights and justifications on this consideration would be useful.

**Conflict Elimination Loss**: Eqn 8 proposes the loss term, and the results do indicate its efficacy, but a more detailed motivation for the particular form of the loss could be useful. Specifically, why is $z_{moe}'(t_n)$ set to $-z_{moe}(t_n)$. Are there other forms that were considered, or insights into why this particular expression works would help.

**MOE Motivation**: The authors work with the assumption that MOE frameworks reduce interference between tokens from diverse data. However, there have been previous works that mention that it is not always the case. Particularly, [1] and [2] suggest that MoE does not necessarily lead to diverse data going to different experts, or observe weak correlation of router decisions with diversity. Additionally, [3] presents a MoE variant that works against this assumption, and uses nested experts. It would be helpful to suggest how STGC could be utilized in such scenarios. It might be useful to include a discussion section in the paper that explicitly addresses these alternative perspectives on MoE behavior 
- how STGC relates to or differs from these other findings, and 
- how STGC could be adapted or extended to work with different MoE variants like nested experts.

[1] Mixtral of Experts, Jiang et. al,, https://arxiv.org/abs/2401.04088

[2] Scaling Vision with Sparse Mixture of Experts, Riquelme et. al, https://arxiv.org/abs/2106.05974

[3] Mixture of Nested Experts: Adaptive Processing of Visual Tokens, Jain et. al, https://arxiv.org/abs/2407.19985

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: Can you provide an analysis of the training costs?
soundness: 3
strengths: 1. Routing is the key problem when training MoE and this paper targets this important problem to reduce interference between tokens within an expert in MoE models by using token-level gradient analysis.
2. The method demonstrated significant performance improvements on various vision-language benchmarks, indicating its effectiveness in enhancing model capabilities.
3. STGC is designed as a plug-in, which means it can be easily integrated into existing MoE-based LVLMs without the need for fundamental architecture changes.
summary: The paper introduces Solving Token Gradient Conflict (STGC), a method to address interference between tokens within experts in Mixture-of-Experts (MoE) models for Large Vision-Language Models (LVLMs). STGC uses token-level gradient analysis to identify conflicting tokens and a regularization loss to optimize token routing, reducing interference and improving model performance. Experiments show STGC's effectiveness across diverse datasets, especially with larger data diversity. The method serves as a plug-in for existing MoE-based LVLMs. The study highlights the importance of managing token interference in MoE architectures and provides a novel approach to enhance their performance in vision-language tasks.
weaknesses: 1. The paper primarily focuses on the empirical validation of the STGC method. While the experimental results are promising, there could be a more in-depth theoretical analysis to understand the underlying principles and limitations of token gradient conflicts and how STGC addresses them.
2. The paper mentions the reduction in inference cost but does not discuss the potential increase in training cost due to the additional computations required for token-level gradient analysis.
3. The paper demonstrates the effectiveness of STGC on vision-language tasks. However, it is not clear how well these findings generalize to other domains or tasks outside of vision-language models. Further experiments in diverse domains could strengthen the paper's claims.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: All relevant questions are included under the “Weaknesses” section. If the authors can address these concerns adequately, I will consider increasing my rating.
soundness: 3
strengths: * **Novel Perspective on Token-Level Interference:** The paper introduces an insightful perspective on token interference by analyzing conflicting optimization directions among tokens assigned to the same expert, which is an important and under-explored issue.
* **Direct and Effective Methodology:** The method efficiently identifies conflicting tokens through comparisons between individual token gradients and the global average gradient within experts. The regularization loss improves routing by discouraging conflicting tokens from remaining with the same expert.
* **Comprehensive Experiments:** The experimental results are extensive and validate the proposed method’s effectiveness. The evaluations across diverse datasets highlight the robustness and general applicability of the approach.
summary: This paper proposes a novel approach to address token interference in Mixture-of-Experts (MoE) architectures for Large Vision-Language Models (LVLMs). While MoE improves computational efficiency by activating only a subset of experts during inference, token interference within the same expert remains a challenge. The authors introduce Solving Token Gradient Conflict (STGC), which leverages token-level gradient selection to detect conflicting tokens and implements a conflict elimination loss to optimize token routing, thereby mitigating interference. The proposed method serves as a plug-and-play solution for various LVLM architectures. Empirical results substantiate the effectiveness of the approach, demonstrating significant improvements across multiple datasets.
weaknesses: * **Clarification on Gradient Conflict Hypothesis:** The authors assert that the learning of conflicting tokens increases the loss of most other tokens within the same expert. However, it is counterintuitive since conflicting tokens should be relatively few, making the directions of parameter optimization align with the global average gradient of all tokens in the expert. Clarifying this assumption would improve the paper's clarity. 
* **Handling Conflicting Tokens Across Experts:** The paper suggests redistributing conflicting tokens to different experts. However, it is unclear how the method ensures that these conflicting tokens do not cause conflicts with tokens in other experts. 
* **Questioning the Gating Network’s Learning Capabilities:** It remains unclear why the gating network fails to learn an optimal distribution for the conflicting tokens. Given that each expert should ideally handle tokens with similar characteristics, why would the gating network assign such divergent tokens to the same expert?

Meta-review: This paper proposes a method to reduce the interference between tokens in Mixture-of-Experts models. This is done by adding a loss term that aligns the per-expert token gradients to the mean gradient over all tokens within the expert. The authors show that this proecedure does indeed improve the alignment of the token gradients, and that this leads to better performance during training.

Reviewers were initially positive about the paper, and the comprehensive rebuttal further addressed the remaining concerns. Therefore, the decision is to accept the paper.

Decision: Accept (Poster)