Introduction
Scaling up model parameters, dataset size and training time has been considered the most direct and effective approach to improving foundation models’ performance [ 18 ; 26 ; 29 ] .
However, scaling dense models substantially increases computational costs, which poses a significant practical challenge.
Mixture-of-Experts (MoE) [ 30 ; 39 ; 17 ; 9 ; 37 ; 34 ] has emerged as a compelling solution for optimizing the balance between model capacity and computation overhead in the era of large foundation models.

MoE models achieve the goal by sparsely activating only a portion of the parameters for each specific input. Specifically, in MoE models, parameters are grouped into a bunch of experts, MoE models only activate some of them for processing a given input.
This selective activation is achieved through a routing mechanism that dispatches each input token to a fixed number of experts (e.g, top- k routing [ 12 ; 10 ; 37 ; 17 ] ).
Therefore, compared to their dense counterparts, MoE models enjoy more efficient training with significantly reduced computational costs [ 10 ; 39 ; 17 ; 9 ; 37 ; 34 ] .
At the inference stage, they typically adhere to the same routing strategy as the training stage, activating only a small fraction of experts. Basically, for each input token, most of the well-trained experts do not contribute to the output prediction.
As a result, the potential of utilizing more experts during the inference stage to enhance performance remains underexplored.

In this paper, we investigate the impact of unchosen experts 1 1 1 Unchosen experts refer to the experts not selected by default routing ( e.g. , top-2 routing in Mixtral 8x7B). on the performance of MoE models and explore their suitable usage. A direct hypothesis is that incorporating more experts improves MoE models and helps solve more difficult problems [ 3 ; 36 ; 14 ] .
However, in our exploratory experiment on Mixtral 8x7B [ 17 ] , we find simply raising the number of activated experts (blue lines in Figure 1 ) does not lead to stable improvements and may even hurt performance on different tasks.
This indicates that unchosen experts may contribute little or even negatively to the final performance, which is contrary to the common perception of unchosen experts as candidates of positive power.

Inspired by the finding, we further dive deep into the difference between the output probability distributions of MoE models applying different routing strategies. As shown in Figure 3 , we calculate the Kullback-Leibler Divergence (KLD) between the token distributions obtained from the default top-2 routing and rank- k routing, and find apparent discrepancy.
The discrepancy is particularly evident in the parts that require rigorous reasoning.
This suggests that different experts do not always act synergistically; instead, they may exhibit conflicting behaviors.

Therefore, we introduce S elf- C ontrast M ixture- o f- E xperts (SCMoE), which can convert the negative effects brought by unchosen experts into positive ones through contrasting the output logits obtained using different routing strategies.
Specifically, the probability of next token is based on the logits difference between strong and weak activation of the MoE models. For "strong activation" and "weak activation", we use the top-2 routing strategy (Figure 2 (a)) and the rank- k routing strategy (Figure 2 (b)) respectively. Thus, SCMoE enables unchosen experts to contribute to the prediction. An overview of how SCMoE works is presented in Figure 2 (c).

Experimental results on various benchmarks across different domains demonstrate that SCMoE significantly enhances Mixtral 8x7B’s reasoning capability (Section 3 ). Specifically, compared to greedy decoding, the accuracy increases from 61.79 to 66.94 (+5.15) on GSM8K, 72.83 to 76.29 (+3.46) on StrategyQA, and the pass@1 accuracy increases from 46.20 to 48.80 (+2.60) on MBPP and 33.54 to 41.46 (+7.92) on HumanEval. Further analysis shows that SCMoE can even surpass the result of using self-consistency with major@5 (66.87) on GSM8K. What’s more, combining SCMoE with self-consistency can further boost the model’s performance, improving major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K.
Regarding inference efficiency, it turns out that SCMoE incurs only a minor (x1.30) delay compared to greedy decoding, which is competitive among several strong decoding baselines.
To sum up, empirical results and comprehensive analyses demonstrate that SCMoE is a both effective and efficient approach to unleashing MoE models’ power.