Divergence Between Different Routing Strategies: An Exploratory Analysis
As depicted in Figure 1 , unchosen experts may contribute little or even negatively to the final performance.
Based on this finding, we are inspired to study the difference of output probabilities using different routing strategies.
Specifically, we conduct an analysis on Mixtral 8x7B [ 16 ] , with two different routing strategies, i.e. , top-2 routing and rank- k routing, which are detailed as follows.

Top-2 Routing .
Top-2 routing (Figure 2 (a)) [ 30 ] is the default routing strategy of Mixtral 8x7B, which activates the two experts with the highest values in \mathbf{w} .
In this setting, the renormalized gate value for the i -th expert, \hat{\mathbf{w}}_{i} , is defined as follows:

\hat{\mathbf{w}}_{i}=\begin{cases}\frac{\mathbf{w}_{i}}{\sum_{j\intop(\mathbf{w},2)}\mathbf{w}_{j}},&\ i\intop(\mathbf{w},2)\ 0,&\ i otintop(\mathbf{w},2)\end{cases} (3)

where \text{top}(\mathbf{w},2) returns the indices of the largest 2 elements in \mathbf{w} .

Rank- k Routing .
The rank- k routing strategy (Figure 2 (b)) only activates one expert, whose initial gate value is ranked at k in \mathbf{w} .
The renormalized gate value \hat{\mathbf{w}}_{i} is defined as follows:

\hat{\mathbf{w}}_{i}=\begin{cases}1,&\ i=rank(\mathbf{w},k)\ 0,&\ i eqrank(\mathbf{w},k)\end{cases} (4)

where \text{rank}(\mathbf{w},k) returns the index of the k -th largest element in \mathbf{w} .
For Mixtral 8x7B, k is enumerated from 1 to 8.
We employ rank- k routing as a representative strategy to utilize unchosen experts of top-2 routing (Additional statistics on the utilization ratio of unchosen experts are provided in Appendix 6 ).

Given an input sequence x_{<t} , we denote the next-token distributions using top-2 routing and rank- k routing as p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-k}}(x_{t}|x_{<t}) , respectively.
Then, we compute the Kullback-Leibler Divergence (KLD) between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-k}}(x_{t}|x_{<t}) on GSM8K dataset.
A qualitative illustration is presented in Figure 3 and a more detailed quantitative study is included in Appendix A .
As shown in Figure 3 , the KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-1}}(x_{t}|x_{<t}) is relatively minor, suggesting a high similarity in their next-token prediction capabilities. However, for k values ranging from 2 to 8, we identify three key findings:

Finding 1: p_{\text{rank-k}}(x_{t}|x_{<t}) with different k values exhibits discernible KLD with p_{\text{top-2}}(x_{t}|x_{<t}) . As k increases from 2 to 8, the KLD increases accordingly. This finding indicates the overall next-token prediction capability gap between top-2 and rank- k routing.

Finding 2: Apparent KLD is observed when generating reasoning sequences, such as mathematical expressions ( e.g. , "1+2+2=5") and the initiation of reasoning steps ( e.g. , "On Thursday"). This suggests notable differences between top-2 and rank- k routing in generating tokens for reasoning.

Finding 3: For the generation of function words ( e.g. , "is") and punctuations ( e.g. , ","), the KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-k}}(x_{t}|x_{<t}) is relatively smaller than that for generating critical reasoning sequences. This indicates that such predictions pose fewer challenges for rank- k routing.

To sum up, it is observed that, in scenarios demanding reasoning capability for next-token prediction, MoE models with top-2 and rank- k routing strategies demonstrate distinct generation behaviors.
This phenomenon suggests that different experts do not always act synergistically, and could in fact exhibit conflicting behaviors.
To harness such information introduced by more experts, a feasible approach is to apply contrastive methods [ 22 ; 20 ] to transform the observed negative impacts into positive ones.

Therefore, we propose to leverage the contrastive information existing between different routing strategies of the MoE model ( e.g. , top-2 routing and rank- k routing) during inference decoding.