RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Method
In this section, we first provide a preliminary introduction of MoE models.
Then, we present an analysis based on next-token distribution KLD to reveal the divergence between different routing strategies in MoE models.
This analysis motivates the introduction of SCMoE, a self-contrast method to leverage the contrastive information existing between different routing strategies in MoE models.
Preliminary
In Transformer-based MoE models, the conventional Feed-Forward Network (FFN) is substitued with the MoE layer [ 19 ] .
Typically, each MoE layer consists of a router R and a set of experts \{E_{i}\}_{i=1}^{N} .
For a given input sequence x_{<t}=(x_{1},x_{2},...,x_{t-1}) ,
the router allocates each token in {x}_{<t} to a specific subset of experts, which are subsequently activated to process the tokens.
Specifically, given each token’s hidden state \mathbf{h} , the router first calculates an initial gate value vector \mathbf{w} across the N experts as follows:

\mathbf{w}=Softmax(\mathbf{W}_{r}\mathbf{h}) (1)

where \mathbf{W}_{r} denotes the weight matrix of the router.
Each element \mathbf{w}_{i} in \mathbf{w} represents the probability of activating the i -th expert.

After that, the router applies a routing strategy ( e.g. , top-2 or rank- k routing in Section 2.2 ) to determine the subset of experts to be activated.
Then the \mathbf{w}_{i} of the unchoosen expert is set to 0
and \mathbf{w} is renormalized to \hat{\mathbf{w}} accordingly.
Subsequently, the output \mathbf{o} of the MoE layer is computed as the weighted sum of outputs from the activated experts:

\mathbf{o}=\sum_{i\in\{j|\hat{\mathbf{w}}_{j} eq 0\}}{\hat{\mathbf{w}}_{i}\cdot E_{i}(\mathbf{h}}) (2)

Once the input sequence x_{<t} has undergone a complete forward pass through the MoE model, the next-token distribution p(x_{t}|{x}_{<t}) is computed based on the output of the final layer.
A decoding algorithm is then applied to predict x_{t} from the vocabulary \mathcal{V} based on p(x_{t}|{x}_{<t}) .
SCMoE: Self-Contrast Mixture-of-Experts
We introduce S elf- C ontrast M ixture- o f- E xperts (SCMoE), an MoE-native self-contrast decoding method.
The fundamental idea behind SCMoE is to determine next-token distribution of an MoE model by leveraging the contrastive information between its strong and weak activation, thereby amplifying the desirable behaviors of the strong activation.
In this context, "strong activation" and "weak activation" of an MoE model refer to the activations obtained by adopting routing strategies with inherent differences ( e.g. , top-2 routing and rank- k routing). An MoE model offers flexible combinations of routing strategies that can be applied for strong and weak activation.
We consider the case of top-2 routing for strong activation and rank- k routing for weak activation.

Specifically, in SCMoE, given the output logits of strong and weak activation,
we use the following equation to obtain the adjusted logits for next-token prediction:

z_{sc}(x_{t}=i|x_{<t})=\begin{cases}(1+\beta)\cdot z_{top-2}(x_{t}=i|x_{<t})-\beta\cdot z_{rank-k}(x_{t}=i|x_{<t})&i\in\mathcal{V}_{valid}\ -\infty&i ot\in\mathcal{V}_{valid}\end{cases} (5)

where \beta\in(0,\infty) is a hyperparameter modulating the intensity of the contrastive penalty. z_{\text{top-2}}(x_{t}|x_{<t}) and z_{\text{rank-k}}(x_{t}|x_{<t}) represent the output logits prior to the softmax operation. \mathcal{V}_{valid} is a subset of the vocabulary \mathcal{V} to restrict the search space:

\mathcal{V}_{valid}=\{i\ |\ z_{top-2}(x_{t}=i|x_{<t})\geq\log\alpha+\max_{j\in\mathcal{V}}z_{top-2}(x_{t}=j|x_{<t})\} (6)

where {\alpha}\in(0,1] is a hyperparameter to control the size of \mathcal{V}_{valid} by masking out tokens that are assigned lower logits.
Empirically, {\alpha} is set to 0.1.

Figure 2 (c) presents an example of how SCMoE works.
In this figure, the output logit of " _= " is consistently high across both top-2 and rank- k routing strategies.
Notably, the logit of the ground-truth token " _+ " shows an apparent increase with the top-2 routing compared to rank- k routing.
SCMoE capitalizes on this contrast to boost the logit of " _+ ", thereby generating more accurate output.
Datasets and Models
To measure the effectiveness of SCMoE, we consider several challenging tasks for LLMs, including mathematical reasoning, commonsense reasoning, and code generation. For mathematical reasoning and commonsense reasoning, we select GSM8K [ 8 ] and StrategyQA [ 13 ] respectively, reporting accuracy. For code generation, we use HumanEval [ 6 ] and MBPP [ 2 ] , reporting pass@1 accuracy. We choose Mixtral 8x7B [ 17 ] as our backbone model.
Setup
As discussed in Section 2.3 , in SCMoE, we use Mixtral 8x7B’s default top-2 routing as the strong activation.
For the weak activation, we only consider the rank- k routing with k=2 .
For the penalty strength \beta , we search from \left[0.1,0.3,0.5,0.7,0.9\right] .

We employ the representative routing-based methods ( i.e. , dynamic and ensemble routing) as the baselines of experts utilization for MoE models. Noting that SCMoE can be seen as a decoding method, we also select commonly used search-based methods ( i.e. , contrastive search, contrastive decoding and Dola) for LLMs as additional baselines. The details of each method are listed below:
Greedy.
Greedy chooses the highest probability token at each step.
Dynamic Routing.
Inspired by [ 14 ] , during inference, the number of activated experts is not fixed. Instead, a threshold is set, and experts are selected in order from highest to lowest scores until the threshold is exceeded. The range of the threshold is \left[0.2,0.3,0.4,0.5,0.6\right] .
Ensemble Routing.
Ensemble routing activates k experts for inference with greedy search, where k ranges from 1 to 8. Note that when k=2 , it is the same as greedy.
Contrastive Search.
Su et al. [ 33 ] use a look-ahead mechanism and penalizes tokens compromising the isotropy of the model’s latent space. We search the penalty degree from \left[0.3,0.4,0.5,0.6\right] .
Contrastive Decoding.
Li et al. [ 20 ] search for tokens that maximize the probability difference between the base LLM and an amateur model. We use Mixtral 8x7B as base LLM and Mistral-7B [ 16 ] as the amateur. We search the strength of the amateur penalty \beta from \left[0.1,0.3,0.5,0.7,0.9\right] .
DoLa.
Chuan et al. [ 7 ] obtain the next-token distribution by contrasting the logits differences between the last layer and a premature layer. The premature layer is dynamically selected from a pre-specified set of layers. Following DoLa [ 7 ] , we test two sets of layers: even-numbered layers from \left[0,16\right) and from \left[16,32\right) respectively.
Unchosen experts can contribute too.
We present the results for each method in Table 1 . For dynamic routing, compared with the greedy approach, dynamically selecting the number of experts to use can enhance Mixtral 8x7B’s performance except for GSM8K (GSM8K -0.68, StrategyQA +1.58, MBPP +1.60, HumanEval + 4.87). This observation indicates that adopting the same top-2 routing strategy during inference as in the training stage may not be optimal for MoE models. Furthermore, for ensemble routing, incorporating additional experts into inference can also improve performance for each task except for MBPP (GSM8K + 2.05, StrategyQA +1.54, MBPP + 0, HumanEval + 3.66). This findings implies that unchosen experts can be further utilized.
SCMoE unleashes MoE models’ power.
SCMoE enhances mathematical reasoning by a +5.10 increase on GSM8K, commonsense reasoning by a +3.46 improvement on StrategyQA.
Moreover, in code generation, SCMoE gets improvements of +2.60 and +7.92 on the MBPP and HumanEval, respectively.
In contrast, traditional search-based methods do not demonstrate substantial improvements on MoE models.
In particular, DoLa’s performance not only fails to surpass, but actually falls below the greedy baseline, particularly due to its inability to terminate generation sequences appropriately (for specific examples, refer to Table 11 in the appendix). Meanwhile, contrastive decoding with Mistral 7B as the amateur model does not result in consistent improvements, and even a decrease in pass@1 accuracy on MBPP (-1.00). Contrastive decoding necessitates a suitable amateur model for effectiveness [ 20 ; 25 ] , but selecting a separate amateur model with same vocabulary is not always feasible.
In comparison, SCMoE capitalizes on the MoE models’ inherent strong and weak activation to conduct self-contrast.
Different weak activation can be viewed as different amateur models, offering higher flexibility and thus help to find the ideal one for contrast.
Impact of Weak Activation
In our main experiments, we use weak activation with rank-2 routing across all benchmarks.
In fact, SCMoE offers the flexibility to employ various routing strategies to determine weak activation.
Thus, in this section, we further explore the effects of selecting different weak activation.
Specifically, we first set rank- k routing with k ranging from 1 to 8 as different weak activation and then investigate corresponding performance changes.
Besides rank- k routing, we also consider random-1 routing strategy to serve as an alternative weak activation for SCMoE. In the random-1 routing strategy, at each MoE layer, the router randomly selects one expert to process current input token.

The experimental results for each candidate weak activation are presented in Figure 4 .
Firstly, compared to the greedy baseline (represented by the red line),
there is a noticeable enhancement in GSM8K, StrategyQA and HumanEval regardless of the chosen weak activation in SCMoE.
Moreover,
when using random-1 routing (represented by the green line), there is still an improvement compared to greedy, which demonstrates the advantage of SCMoE in utilizing its weak activation for self-contrast.
Overall, using rank-2 routing as weak activation can provide consistently good performances, and further exploring rank- k or other routing strategies may bring additional improvements.
Impact of Strong Activation
As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks. For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA. This leads us to consider whether enhancing the strong activation in SCMoE can further unlock MoE models’ potential.
To this end, we adjust the strong activation of Mixtral 8x7B to top-3 for GSM8K, and to top-4 for StrategyQA, MBPP, and HumanEval, while keeping the weak activation with rank-2 routing as before.
The experimental results, as shown in Table 2 , reveal that enhancing the strong activation of SCMoE can further boost MoE models’ performance. Compared to the previous best performance achieved when only utilizing top-2 routing for strong activation, this adjustment improves Mixtral 8x7B’s performance by 1.98 on GSM8K, 0.13 on StrategyQA, and 1.80 on MBPP.
Combining SCMoE with Self-Consistency
Using self-consistency [ 35 ] for multiple sampling and taking a majority vote to determine the final answer is a common method to improve LLMs’ performance. Therefore, we explore whether SCMoE can combined with self-consistency.
For vanilla self-consistency, we use temperature sampling with temperature \tau=0.7 to reach the best baseline performance.
For self-consistency with SCMoE, we simply employ \beta=0.5 , rank-3 routing as weak activation, according to the best hyperparameters setting from Table 8 . It is worth noting that since SCMoE already has a mask \alpha=0.1 to limit the sampling range of the vocabulary, we do not perform any additional temperature processing on the final logits.
As shown in Figure 5 ,
SCMoE (67.94) yields comparable results with major@5 (66.87). Furthermore, SCMoE can enhance the major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K.
Latency
We further evaluate the impact of SCMoE on decoding latency and compare it with other methods on Mixtral 8x7B. Specifically, we first input 32 tokens to each method and then force them to generate a sequence of 512 tokens to calculate the latency.
The results in Table 3 show that SCMoE increases the decoding time by a factor of 1.30x compared to greedy.
When compared with other methods, SCMoE does not introduce a significant amount of latency, especially when compared to contrastive search (x1.62) and contrastive decoding (x1.43).
Moreover, SCMoE even surpasses the results of using self-consistency with major@5 on GSM8K, which has a 5x latency compared to greedy. Therefore, the latency of SCMoE can be considered negligible, making it both effective and efficient approach.
Employ DeepSeekMoE
We further explore the adaptability of SCMoE to other MoE models. We conduct experiments on DeepSeekMoE-16B [ 5 ] . DeepSeekMoE-16B employs fine-grained expert segmentation and shared expert isolation routing strategies, which is different from Mixtral 8x7B [ 17 ] . We detail the hyperparameters settings of experiments in Appendix C .
It is worth noting that contrastive decoding needs a suitable model to serve as an amateur. However, DeepSeekMoE-16B does not have a smaller model with the same vocabulary, so DeepSeekMoE-16B does not have the contrast decoding baseline.
As depicted in Table 4 , SCMoE effectively unleashes the potential of DeepSeekMoE-16B. Specifically, compared to greedy baseline, SCMoE demonstrates improvements across all tasks: it enhances mathematical reasoning by 1.82 on GSM8K, commonsense reasoning by 2.58 on StrategyQA, code generation by 2.00 on MBPP, and 1.22 on HumanEval. In contrast, other methods, regardless of routing-based or search-based, struggle to outperform the greedy baseline. These results demonstrate that SCMoE can be successfully applied to other MoE models.
Mixture-of-Experts
The Mixture-of-Experts (MoE) model was initially introduced by A. Jacob et al. [ 15 ] . Previous studies have demonstrated that sparsely gated MoE models can significantly improve model capacity and efficiency, enabling superior performance compared to dense ones [ 30 ; 10 ; 39 ; 40 ] .
In MoE models, a static number of experts are activated regardless of the varying complexity presented by input tokens. Typically, top-1 or top-2 experts are activated in these models [ 19 ; 12 ] .
In the era of LLMs, numerous extensive open-source models based on MoE architecture have emerged. Specifically, both Mixtral 8x7B [ 17 ] and Grok-1 [ 37 ] introduce an 8-expert MoE that uses a top-2 routing algorithm during inference. DeepSeekMoE [ 9 ] and QwenMoE [ 34 ] , on the other hand, both employ a fine-grained expert segmentation, applying 2 shared experts with N routed experts. As a result, they use k +2 experts for inference, with 2 fixed shared experts and top- k routed experts.

While several works have attempted to examine pruning or dynamic routing algorithms for MoE models [ 23 ; 11 ; 14 ] from the perspective of reducing computational costs while maintaining performance. Our approach differs in that we investigate the utilization of unchosen experts in a self-contrast manner to boost MoE models’ capability without increasing too much computation.
Contrast in Language Modeling
The idea of employing contrast to enhance language modeling has been explored through various approaches. Specifically, the contrast enables language models to discern between desirable and undesirable behaviors, a distinction that the conventional maximum log-likelihood modeling often fails to adequately capture [ 1 ] .
One line of research focuses on training-time optimization. Reinforcement learning from human feedback (RLHF) [ 32 ; 4 ; 27 ] trains reward models by contrasting the rewards associated with desirable outputs to those of undesirable ones, and then optimize the LLM to maximize rewards through reinforcement learning. RRHF [ 38 ] , DPO [ 28 ] , and PRO [ 31 ] eliminate the necessity of constructing reward models and instead directly optimize LLMs by contrasting preferred responses versus dispreferred ones.
Another research avenue focuses on inference-time optimization. DExperts [ 22 ] fine-tunes two models with desirable and undesirable attributes separately, guiding the base model by leveraging the contrast between those models. Contrastive Decoding [ 20 ; 25 ] contrasts base model with an amateur model to mitigate undesirable tendencies of the amateur. Emulated fine-tuning [ 24 ] and proxy-tuning [ 21 ] achieve training-free alignment in a similar way, applying the contrast between aligned and unaligned models as a reward signal to guide the decoding process of a larger unaligned LLM. Contrastive Search [ 33 ] uses a look-ahead contrastive mechanism and penalizes tokens compromising
the isotropy of the model’s latent space. DoLa [ 7 ] obtains the next-token distribution by contrasting the logits differences between the last layer and a premature layer to improve factuality.

Our research focuses on inference-time optimization.
Distinct from the above methods that mainly utilize contrasts between different models, our work leverages the contrastive information among strong and weak activation of MoE models to unleash their potential through self-contrast.
Conclusion
In this work, we develop S elf- C ontrast M ixture- o f- E xperts (SCMoE), a conceptually simple and computationally lightweight strategy to unleash MoE models’ power via self-contrast.
We find that different routing strategies within an MoE model output results with considerable divergent information.
Utilizing this information in a self-contrast manner can further enhance MoE models’ reasoning capabilities in next-token prediction.
Experimental results show that SCMoE improves the MoE models’ performance on multiple benchmarks with only minor latency increase at inference time.
Due to resource constraints, our main limitation is that we cannot further explore the performance of SCMoE on larger MoE models such as Mixtral 8x22B or DeepSeek-V2.
Overall, SCMoE is a critical step to leverage the inherent self-contrast features of MoE models, and offers new insights to the utilization of unchosen experts.
A.1 KLD Supplement for Section 2.2
In Section 2.2 ,
Figure 3 qualitatively illustrates that reasoning ability gap among different expert routing ( i.e. , top-2 and rank- k routing).
To support this,
we also conduct a quantitative study.

Using the questions and ground-truth answers from GSM8K train set as input, we obtain the the next token in a teacher-forcing approach with Mixtral 8x7B.
Then, we calculate the average KLD between the p(x_{t}|x_{<t}) produced by Mixtral 8x7b with top- 2 routing strategy and those generated with different rank- k routing strategies.
Specifically, the average KLD is calculated across three sets of tokens:

(1) "All" : This set includes all tokens in ground-truth answers;

(2) "Expression" : This set comprises tokens from mathematical expressions in ground-truth answers.
The generation of these tokens poses reasoning challenge for MoE models.
We use regular expressions to extract the mathematical expressions within ground-truth answers.

(3) "Stopword" : This set contains tokens from stopwords, which serves as a representative proxy for function words. We utilize the NLTK stopwords list 2 2 2 https://www.nltk.org/ .

The results are presented in Table 5 .
The results further support the three findings in Section 2.2 for k values ranging from 2 to 8:

Finding 1: p_{\text{rank-k}}(x_{t}|x_{<t}) with different k values exhibits notable average KLD with p_{\text{top-2}}(x_{t}|x_{<t}) .
As k increases from 2 to 8,
the average KLD also increases accordingly.
This finding suggests the overall next-token prediction discrepancy between top-2 and rank- k routing.

Finding 2: For each rank- k strategy, apparent average KLD is observed when generating mathematical expressions ( i.e. , "Expression" token set).
This indicates the notable differences between top-2 and rank- k routing in generating tokens for reasoning.

Finding 3: For each rank- k strategy, the average KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-k}}(x_{t}|x_{<t}) is relatively smaller when generating stopword tokens ( i.e. , "Stopword" token set) compared to generating mathematical expression tokens.
This suggests that such predictions pose fewer challenges for rank- k routing.
A.2 Further Analysis on Kullback-Leibler Divergence
We also calculate the KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{random-1}}(x_{t}|x_{<t}) , p_{\text{Mistral-7B}}(x_{t}|x_{<t}) in Table 5 and present further analysis on Kullback-Leibler Divergence:

It is observed that the KLD between p_{\text{top-2}}(x_{t}|x_{<t}) and p_{\text{rank-2}}(x_{t}|x_{<t}) is relatively small for the "Stopword" token set.
This indicates that Mixtral 8x7B with rank- 2 routing exhibit basic stopword generation capability similar to Mixtral 8x7B with top-2 routing.
However, for the "Expression" token set, the KLD increases notably compared to that of the "All" token set (i.e., it increases by 31.13%).
These observations suggest that when shifting routing strategies from top-2 routing to rank-2 routing, the reasoning capability of Mixtral 8x7B decreases more than basic generation capability.

As suggested by prior works [ 20 , 25 ] , this apparent reasoning ability gap can be leveraged to better amplify the reasoning strength of Mixtral 8x7B with top-2 routing.
Thus, in our main experiments, we report results with fixed rank- 2 for the weak activation. The same observation also applies to the weak activations of rank-3, rank-4, and random-1, albeit with varying degrees of significance.
Empirically, results in Section 3.3 and 4.1 also illustrate that contrast with rank- 2 routing yields generally better improvements.

For Mistral 7B, the average KLD between its next-token distribution and that of Mixtral 7x8B with top-2 routing across three token sets is quite small, indicating that their overall distributions is very similar.
This similarity makes Mistral 7B not an ideal weak model to contrast.
Greedy.
Greedy does not have hyperparameters to set.
Dynamic Routing.
The range of the dynamic threshold is \left[0.2,0.3,0.4,0.5,0.6\right] .
Ensemble Routing.
The number of activated experts for inference ranges from 1 to 8.
Contrastive Search.
The penalty degree is \left[0.3,0.4,0.5,0.6\right] .
DoLa.
For DoLa, due to DeepSeekMoE-16B having 28 layers, we test two sets of layers: even-numbered layers from [0, 14) and from [14, 28) respectively.
SCMoE
DeepSeekMoE-16B defaults to taking top-6 routing. Therefore, when implementing SCMoE, we choose top-6 routing as strong activation and top- k routing k\in\left[1,2,3\right] as weak activation. For the penalty strength \beta , we also search from \left[0.1,0.3,0.5,0.7,0.9\right] .
Figure 1 : Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.; Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
Figure 2 : (a & b) Given an input \mathbf{h} , (a) and (b) demonstrate the workflows of top-2 routing and rank- k routing ( e.g. , k =2). We use two MoE layers as a simple schematic, omitting other layers in MoE models. Note that, in the second MoE layer, rank- k routing activates the unchosen expert in top-2 routing; (c) An illustrative example of how SCMoE works, which contrasts z_{\text{top-2}}(x_{t}|x_{<t}) with z_{\text{rank-k}}(x_{t}|x_{<t}) . The complete question and answer for this example are shown in Figure 3 .; (a & b) Given an input \mathbf{h} , (a) and (b) demonstrate the workflows of top-2 routing and rank- k routing ( e.g. , k =2). We use two MoE layers as a simple schematic, omitting other layers in MoE models. Note that, in the second MoE layer, rank- k routing activates the unchosen expert in top-2 routing; (c) An illustrative example of how SCMoE works, which contrasts z_{\text{top-2}}(x_{t}|x_{<t}) with z_{\text{rank-k}}(x_{t}|x_{<t}) . The complete question and answer for this example are shown in Figure 3 .