Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast

Authors: chufan  shi
cheng  yang
xinyu  zhu
jiahao  wang
taiqiang  wu
siheng  li
deng  cai
yujiu  yang
yu  meng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-07 15:39:17

Abstract: Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the input sequence activates a different subset of experts determined by a routing mechanism. However, the unchosen experts in MoE models do not contribute to the output, potentially leading to underutilization of the model's capacity.
In this work, we first conduct exploratory studies to demonstrate that increasing the number of activated experts does not necessarily improve and can even degrade the output quality. Then, we show that output distributions from an MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically. 
Motivated by these findings, we propose **S**elf-**C**ontrast **M**ixture-**o**f-**E**xperts (SCMoE), a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. 
In SCMoE, the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE model.
Our method is conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding. 
Experiments on several benchmarks (GSM8K, StrategyQA, MBPP and HumanEval) demonstrate that SCMoE can consistently enhance Mixtral 8x7B’s reasoning capability across various domains. For example, it improves the accuracy on GSM8K from 61.79 to 66.94. 
Moreover, combining SCMoE with self-consistency yields additional gains, increasing major@20 accuracy from 75.59 to 78.31.

Main paper: 
 introduction: 
 Scaling up model parameters, dataset size and training time has been considered the most direct and effective approach to improving foundation models' performance [1] [2] [3] . However, scaling dense models substantially increases computational costs, which poses a significant practical challenge. Mixture-of-Experts (MoE) [4] [5] [6] [7] [8] [9] has emerged as a compelling solution for optimizing the balance between model capacity and computation overhead in the era of large foundation models.MoE models achieve the goal by sparsely activating only a portion of the parameters for each specific input. Specifically, in MoE models, parameters are grouped into a bunch of experts, MoE models only activate some of them for processing a given input. This selective activation is achieved through a routing mechanism that dispatches each input token to a fixed number of experts (e.g, top-k routing [6, 8, 10, 11] ). Therefore, compared to their dense counterparts, MoE models enjoy more efficient training with significantly reduced computational costs [5] [6] [7] [8] [9] 11] . At the inference stage, they typically adhere to the same routing strategy as the training stage, activating only a small fraction of experts. Basically, for each input token, most of the well-trained experts do not contribute to the output prediction. As a result, the potential of utilizing more experts during the inference stage to enhance performance remains underexplored.Ensemble Routing SCMoEIn this paper, we investigate the impact of unchosen experts3 on the performance of MoE models and explore their suitable usage. A direct hypothesis is that incorporating more experts improves MoE models and helps solve more difficult problems [12] [13] [14] . However, in our exploratory experiment on Mixtral 8x7B [6] , we find simply raising the number of activated experts (blue lines in Figure 1 ) does not lead to stable improvements and may even hurt performance on different tasks. This indicates that unchosen experts may contribute little or even negatively to the final performance, which is contrary to the common perception of unchosen experts as candidates of positive power.Inspired by the finding, we further dive deep into the difference between the output probability distributions of MoE models applying different routing strategies. As shown in Figure 3 , we calculate the Kullback-Leibler Divergence (KLD) between the token distributions obtained from the default top-2 routing and rank-k routing, and find apparent discrepancy. The discrepancy is particularly evident in the parts that require rigorous reasoning. This suggests that different experts do not always act synergistically; instead, they may exhibit conflicting behaviors.Therefore, we introduce Self-Contrast Mixture-of-Experts (SCMoE), which can convert the negative effects brought by unchosen experts into positive ones through contrasting the output logits obtained using different routing strategies. Specifically, the probability of next token is based on the logits difference between strong and weak activation of the MoE models. For "strong activation" and "weak activation", we use the top-2 routing strategy (Figure 2 (a)) and the rank-k routing strategy (Figure 2 (b)) respectively. Thus, SCMoE enables unchosen experts to contribute to the prediction. An overview of how SCMoE works is presented in Figure 2 (c) .Experimental results on various benchmarks across different domains demonstrate that SCMoE significantly enhances Mixtral 8x7B's reasoning capability (Section 3). Specifically, compared to greedy decoding, the accuracy increases from 61.79 to 66.94 (+5.15) on GSM8K, 72.83 to 76.29 (+3.46) on StrategyQA, and the pass@1 accuracy increases from 46.20 to 48.80 (+2.60) on MBPP and 33.54 to 41.46 (+7.92) on HumanEval. Further analysis shows that SCMoE can even surpass the result of using self-consistency with major@5 (66.87) on GSM8K. What's more, combining SCMoE with self-consistency can further boost the model's performance, improving major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K. Regarding inference efficiency, it turns out that SCMoE incurs only a minor (x1.30) delay compared to greedy decoding, which is competitive among several strong decoding baselines. To sum up, empirical results and comprehensive analyses demonstrate that SCMoE is a both effective and efficient approach to unleashing MoE models' power. 
method: 
 In this section, we first provide a preliminary introduction of MoE models. Then, we present an analysis based on next-token distribution KLD to reveal the divergence between different routing strategies in MoE models. This analysis motivates the introduction of SCMoE, a self-contrast method to leverage the contrastive information existing between different routing strategies in MoE models.EQUATIONIn Transformer-based MoE models, the conventional Feed-Forward Network (FFN) is substitued with the MoE layer [15] . Typically, each MoE layer consists of a router R and a set of experts {E i } N i=1 . For a given input sequence x <t = (x 1 , x 2 , ..., x t-1 ), the router allocates each token in x <t to a specific subset of experts, which are subsequently activated to process the tokens. Specifically, given each token's hidden state h, the router first calculates an initial gate value vector w across the N experts as follows:EQUATIONwhere W r denotes the weight matrix of the router. Each element w i in w represents the probability of activating the i-th expert.After that, the router applies a routing strategy (e.g., top-2 or rank-k routing in Section 2.2) to determine the subset of experts to be activated. Then the w i of the unchoosen expert is set to 0 and w is renormalized to ŵ accordingly. Subsequently, the output o of the MoE layer is computed as the weighted sum of outputs from the activated experts:EQUATIONOnce the input sequence x <t has undergone a complete forward pass through the MoE model, the next-token distribution p(x t |x <t ) is computed based on the output of the final layer. A decoding algorithm is then applied to predict x t from the vocabulary V based on p(x t |x <t ).As depicted in Figure 1 , unchosen experts may contribute little or even negatively to the final performance. Based on this finding, we are inspired to study the difference of output probabilities using different routing strategies. Specifically, we conduct an analysis on Mixtral 8x7B [16] , with two different routing strategies, i.e., top-2 routing and rank-k routing, which are detailed as follows.Top-2 Routing. Top-2 routing (Figure 2 (a)) [4] is the default routing strategy of Mixtral 8x7B, which activates the two experts with the highest values in w. In this setting, the renormalized gate value for the i-th expert, ŵi , is defined as follows: Question: Hallie had dance practice for 1 hour on Tuesdays and 2 hours on Thursdays. On Saturdays, she had dance practice that lasted twice as long as Tuesday's night class. How many hours a week did she have dance practice? Answer: On Tuesdays, she had 1 hour. On Thursdays, she had 2 hours. On Saturdays, she had 2 x 1 = 2 hours. 1 + 2 + 2 = 5. The answer is 5. where top(w, 2) returns the indices of the largest 2 elements in w.EQUATIONRank-k Routing. The rank-k routing strategy (Figure 2 (b)) only activates one expert, whose initial gate value is ranked at k in w. The renormalized gate value ŵi is defined as follows:EQUATIONwhere rank(w, k) returns the index of the k-th largest element in w. For Mixtral 8x7B, k is enumerated from 1 to 8. We employ rank-k routing as a representative strategy to utilize unchosen experts of top-2 routing (Additional statistics on the utilization ratio of unchosen experts are provided in Appendix B).Given an input sequence x <t , we denote the next-token distributions using top-2 routing and rank-k routing as p top-2 (x t |x <t ) and p rank-k (x t |x <t ), respectively. Then, we compute the Kullback-Leibler Divergence (KLD) between p top-2 (x t |x <t ) and p rank-k (x t |x <t ) on GSM8K dataset. A qualitative illustration is presented in Figure 3 and a more detailed quantitative study is included in Appendix A. As shown in Figure 3 , the KLD between p top-2 (x t |x <t ) and p rank-1 (x t |x <t ) is relatively minor, suggesting a high similarity in their next-token prediction capabilities. However, for k values ranging from 2 to 8, we identify three key findings:Finding 1: p rank-k (x t |x <t ) with different k values exhibits discernible KLD with p top-2 (x t |x <t ).As k increases from 2 to 8, the KLD increases accordingly. This finding indicates the overall next-token prediction capability gap between top-2 and rank-k routing.Finding 2: Apparent KLD is observed when generating reasoning sequences, such as mathematical expressions (e.g., "1+2+2=5") and the initiation of reasoning steps (e.g., "On Thursday"). This suggests notable differences between top-2 and rank-k routing in generating tokens for reasoning.Finding 3: For the generation of function words (e.g., "is") and punctuations (e.g., ","), the KLD between p top-2 (x t |x <t ) and p rank-k (x t |x <t ) is relatively smaller than that for generating critical reasoning sequences. This indicates that such predictions pose fewer challenges for rank-k routing.To sum up, it is observed that, in scenarios demanding reasoning capability for next-token prediction, MoE models with top-2 and rank-k routing strategies demonstrate distinct generation behaviors. This phenomenon suggests that different experts do not always act synergistically, and could in fact exhibit conflicting behaviors. To harness such information introduced by more experts, a feasible approach is to apply contrastive methods [17, 18] to transform the observed negative impacts into positive ones.Therefore, we propose to leverage the contrastive information existing between different routing strategies of the MoE model (e.g., top-2 routing and rank-k routing) during inference decoding.We introduce Self-Contrast Mixture-of-Experts (SCMoE), an MoE-native self-contrast decoding method. The fundamental idea behind SCMoE is to determine next-token distribution of an MoE model by leveraging the contrastive information between its strong and weak activation, thereby amplifying the desirable behaviors of the strong activation. In this context, "strong activation" and "weak activation" of an MoE model refer to the activations obtained by adopting routing strategies with inherent differences (e.g., top-2 routing and rank-k routing). An MoE model offers flexible combinations of routing strategies that can be applied for strong and weak activation. We consider the case of top-2 routing for strong activation and rank-k routing for weak activation.Specifically, in SCMoE, given the output logits of strong and weak activation, we use the following equation to obtain the adjusted logits for next-token prediction:EQUATIONwhere β ∈ (0, ∞) is a hyperparameter modulating the intensity of the contrastive penalty. z top-2 (x t |x <t ) and z rank-k (x t |x <t ) represent the output logits prior to the softmax operation. V valid is a subset of the vocabulary V to restrict the search space:EQUATIONwhere α ∈ (0, 1] is a hyperparameter to control the size of V valid by masking out tokens that are assigned lower logits. Empirically, α is set to 0.1.Figure 2 (c) presents an example of how SCMoE works. In this figure, the output logit of "_=" is consistently high across both top-2 and rank-k routing strategies. Notably, the logit of the ground-truth token "_+" shows an apparent increase with the top-2 routing compared to rank-k routing. SCMoE capitalizes on this contrast to boost the logit of "_+", thereby generating more accurate output.3 Experiments 
datasets and models: 
 To measure the effectiveness of SCMoE, we consider several challenging tasks for LLMs, including mathematical reasoning, commonsense reasoning, and code generation. For mathematical reasoning and commonsense reasoning, we select GSM8K [19] and StrategyQA [20] respectively, reporting accuracy. For code generation, we use HumanEval [21] and MBPP [22] , reporting pass@1 accuracy. We choose Mixtral 8x7B [6] as our backbone model.As discussed in Section 2.3, in SCMoE, we use Mixtral 8x7B's default top-2 routing as the strong activation. For the weak activation, we only consider the rank-k routing with k = 2. For the penalty strength β, we search from [0.1, 0.3, 0.5, 0.7, 0.9].We employ the representative routing-based methods (i.e., dynamic and ensemble routing) as the baselines of experts utilization for MoE models. Noting that SCMoE can be seen as a decoding method, we also select commonly used search-based methods (i.e., contrastive search, contrastive decoding and Dola) for LLMs as additional baselines. The details of each method are listed below: Greedy. Greedy chooses the highest probability token at each step. Dynamic Routing. Inspired by [14] , during inference, the number of activated experts is not fixed. Instead, a threshold is set, and experts are selected in order from highest to lowest scores until the threshold is exceeded. The range of the threshold is [0.2, 0.3, 0.4, 0.5, 0.6]. Ensemble Routing. Ensemble routing activates k experts for inference with greedy search, where k ranges from 1 to 8. Note that when k = 2, it is the same as greedy. [18] search for tokens that maximize the probability difference between the base LLM and an amateur model. We use Mixtral 8x7B as base LLM and Mistral-7B [16] as the amateur. We search the strength of the amateur penalty β from [0.1, 0.3, 0.5, 0.7, 0.9]. DoLa. Chuan et al. [24] obtain the next-token distribution by contrasting the logits differences between the last layer and a premature layer. The premature layer is dynamically selected from a pre-specified set of layers. Following DoLa [24] , we test two sets of layers: even-numbered layers from [0, 16) and from [16, 32) respectively.Unchosen experts can contribute too. We present the results for each method in Table 1 . For dynamic routing, compared with the greedy approach, dynamically selecting the number of experts to use can enhance Mixtral 8x7B's performance except for GSM8K (GSM8K -0.68, StrategyQA +1.58, MBPP +1.60, HumanEval + 4.87). This observation indicates that adopting the same top-2 routing strategy during inference as in the training stage may not be optimal for MoE models. Furthermore, for ensemble routing, incorporating additional experts into inference can also improve performance for each task except for MBPP (GSM8K + 2.05, StrategyQA +1.54, MBPP + 0, HumanEval + 3.66). This findings implies that unchosen experts can be further utilized. SCMoE unleashes MoE models' power. SCMoE enhances mathematical reasoning by a +5.10 increase on GSM8K, commonsense reasoning by a +3.46 improvement on StrategyQA. Moreover, in code generation, SCMoE gets improvements of +2.60 and +7.92 on the MBPP and HumanEval, respectively. In contrast, traditional search-based methods do not demonstrate substantial improvements on MoE models. In particular, DoLa's performance not only fails to surpass, but actually falls below the greedy baseline, particularly due to its inability to terminate generation sequences appropriately (for specific examples, refer to Table 11 in the appendix). Meanwhile, contrastive decoding with Mistral 7B as the amateur model does not result in consistent improvements, and even a decrease in pass@1 accuracy on MBPP (-1.00). Contrastive decoding necessitates a suitable amateur model for effectiveness [18, 25] , but selecting a separate amateur model with same vocabulary is not always feasible. In comparison, SCMoE capitalizes on the MoE models' inherent strong and weak activation to conduct self-contrast. Different weak activation can be viewed as different amateur models, offering higher flexibility and thus help to find the ideal one for contrast.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes]Answer: [Yes] Justification: We use open-source models and publicly available datasets, and we also include our code in the supplementary materials.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
impact of weak activation: 
 In our main experiments, we use weak activation with rank-2 routing across all benchmarks. In fact, SCMoE offers the flexibility to employ various routing strategies to determine weak activation. Thus, 8 .Table 2 : Experimental results of different strong activations. We set the weak activation with rank-2 routing. For each benchmark, we select the top-k routing yielding the best performance in Figure 1 as the ideal strong activation. The specific hyperparameter settings can be found in Table 9 . in this section, we further explore the effects of selecting different weak activation. Specifically, we first set rank-k routing with k ranging from 1 to 8 as different weak activation and then investigate corresponding performance changes. Besides rank-k routing, we also consider random-1 routing strategy to serve as an alternative weak activation for SCMoE. In the random-1 routing strategy, at each MoE layer, the router randomly selects one expert to process current input token.MethodThe experimental results for each candidate weak activation are presented in Figure 4 . Firstly, compared to the greedy baseline (represented by the red line), there is a noticeable enhancement in GSM8K, StrategyQA and HumanEval regardless of the chosen weak activation in SCMoE. Moreover, when using random-1 routing (represented by the green line), there is still an improvement compared to greedy, which demonstrates the advantage of SCMoE in utilizing its weak activation for selfcontrast. Overall, using rank-2 routing as weak activation can provide consistently good performances, and further exploring rank-k or other routing strategies may bring additional improvements. As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks. For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA. This leads us to consider whether enhancing the strong activation in SCMoE can further unlock MoE models' potential. To this end, we adjust the strong activation of Mixtral 8x7B to top-3 for GSM8K, and to top-4 for StrategyQA, MBPP, and HumanEval, while keeping the weak activation with rank-2 routing as before. The experimental results, as shown in Table 2 , reveal that enhancing the strong activation of SCMoE can further boost MoE models' performance. Compared to the previous best performance achieved when only utilizing top-2 routing for strong activation, this adjustment improves Mixtral 8x7B's performance by 1.98 on GSM8K, 0.13 on StrategyQA, and 1.80 on MBPP. Table 3 : Averaged decoding latency for each method. CS is short for contrastive search and CD is short for contrastive decoding. We set k = 3 for ensemble routing, while for dynamic routing we set threshold = 0.5. The speeds are tested on 4 A100 40G with batch size = 1. Using self-consistency [26] for multiple sampling and taking a majority vote to determine the final answer is a common method to improve LLMs' performance. Therefore, we explore whether SCMoE can combined with self-consistency. For vanilla self-consistency, we use temperature sampling with temperature τ = 0.7 to reach the best baseline performance [27] . For self-consistency with SCMoE, we simply employ β = 0.5, rank-3 routing as weak activation, according to the best hyperparameters setting from Table 8 . It is worth noting that since SCMoE already has a mask α = 0.1 to limit the sampling range of the vocabulary, we do not perform any additional temperature processing on the final logits. As shown in Figure 5 , SCMoE (67.94) yields comparable results with major@5 (66.87). Furthermore, SCMoE can enhance the major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K.We further evaluate the impact of SCMoE on decoding latency and compare it with other methods on Mixtral 8x7B. Specifically, we first input 32 tokens to each method and then force them to generate a sequence of 512 tokens to calculate the latency. The results in Table 3 show that SCMoE increases the decoding time by a factor of 1.30x compared to greedy. When compared with other methods, SCMoE does not introduce a significant amount of latency, especially when compared to contrastive search (x1.62) and contrastive decoding (x1.43). Moreover, SCMoE even surpasses the results of using self-consistency with major@5 on GSM8K, which has a 5x latency compared to greedy. Therefore, the latency of SCMoE can be considered negligible, making it both effective and efficient approach.We further explore the adaptability of SCMoE to other MoE models. We conduct experiments on DeepSeekMoE-16B [28] . DeepSeekMoE-16B employs fine-grained expert segmentation and shared expert isolation routing strategies, which is different from Mixtral 8x7B [6] . We detail the hyperparameters settings of experiments in Appendix C. It is worth noting that contrastive decoding needs a suitable model to serve as an amateur. However, DeepSeekMoE-16B does not have a smaller model with the same vocabulary, so DeepSeekMoE-16B does not have the contrast decoding baseline. As depicted in  
related work: 
 Mixture-of-Experts The Mixture-of-Experts (MoE) model was initially introduced by A. Jacob et al. [29] . Previous studies have demonstrated that sparsely gated MoE models can significantly improve model capacity and efficiency, enabling superior performance compared to dense ones [4, 5, 11, 30] .In MoE models, a static number of experts are activated regardless of the varying complexity presented by input tokens. Typically, top-1 or top-2 experts are activated in these models [15, 10] . In the era of LLMs, numerous extensive open-source models based on MoE architecture have emerged. Specifically, both Mixtral 8x7B [6] and Grok-1 [8] introduce an 8-expert MoE that uses a top-2 routing algorithm during inference. DeepSeekMoE [7] and QwenMoE [9] , on the other hand, both employ a fine-grained expert segmentation, applying 2 shared experts with N routed experts. As a result, they use k+2 experts for inference, with 2 fixed shared experts and top-k routed experts.While several works have attempted to examine pruning or dynamic routing algorithms for MoE models [31, 32, 14] from the perspective of reducing computational costs while maintaining performance.Our approach differs in that we investigate the utilization of unchosen experts in a self-contrast manner to boost MoE models' capability without increasing too much computation.The idea of employing contrast to enhance language modeling has been explored through various approaches. Specifically, the contrast enables language models to discern between desirable and undesirable behaviors, a distinction that the conventional maximum log-likelihood modeling often fails to adequately capture [33] . One line of research focuses on training-time optimization. Reinforcement learning from human feedback (RLHF) [34] [35] [36] trains reward models by contrasting the rewards associated with desirable outputs to those of undesirable ones, and then optimize the LLM to maximize rewards through reinforcement learning. RRHF [37] , DPO [38] , and PRO [39] eliminate the necessity of constructing reward models and instead directly optimize LLMs by contrasting preferred responses versus dispreferred ones. Another research avenue focuses on inference-time optimization. DExperts [17] fine-tunes two models with desirable and undesirable attributes separately, guiding the base model by leveraging the contrast between those models. Contrastive Decoding [18, 25] contrasts base model with an amateur model to mitigate undesirable tendencies of the amateur. Emulated fine-tuning [40] and proxy-tuning [41] achieve training-free alignment in a similar way, applying the contrast between aligned and unaligned models as a reward signal to guide the decoding process of a larger unaligned LLM. Contrastive Search [23] uses a look-ahead contrastive mechanism and penalizes tokens compromising the isotropy of the model's latent space. DoLa [24] obtains the next-token distribution by contrasting the logits differences between the last layer and a premature layer to improve factuality.Our research focuses on inference-time optimization. Distinct from the above methods that mainly utilize contrasts between different models, our work leverages the contrastive information among strong and weak activation of MoE models to unleash their potential through self-contrast. 
conclusion: 
 In this work, we develop Self-Contrast Mixture-of-Experts (SCMoE), a conceptually simple and computationally lightweight strategy to unleash MoE models' power via self-contrast. We find that different routing strategies within an MoE model output results with considerable divergent information. Utilizing this information in a self-contrast manner can further enhance MoE models' reasoning capabilities in next-token prediction. Experimental results show that SCMoE improves the MoE models' performance on multiple benchmarks with only minor latency increase at inference time. Due to resource constraints, our main limitation is that we cannot further explore the performance of SCMoE on larger MoE models such as Mixtral 8x22B or DeepSeek-V2. Overall, SCMoE is a critical step to leverage the inherent self-contrast features of MoE models, and offers new insights to the utilization of unchosen experts. In Section 2.2, Figure 3 qualitatively illustrates that reasoning ability gap among different expert routing (i.e., top-2 and rank-k routing). To support this, we also conduct a quantitative study.Using the questions and ground-truth answers from GSM8K train set as input, we obtain the the next token in a teacher-forcing approach with Mixtral 8x7B. Then, we calculate the average KLD between the p(x t |x <t ) produced by Mixtral 8x7b with top-2 routing strategy and those generated with different rank-k routing strategies. Specifically, the average KLD is calculated across three sets of tokens:(1) "All": This set includes all tokens in ground-truth answers;(2) "Expression": This set comprises tokens from mathematical expressions in ground-truth answers. The generation of these tokens poses reasoning challenge for MoE models. We use regular expressions to extract the mathematical expressions within ground-truth answers.(3) "Stopword": This set contains tokens from stopwords, which serves as a representative proxy for function words. We utilize the NLTK stopwords list 4 .The results are presented in Table 5 . The results further support the three findings in Section 2.2 for k values ranging from 2 to 8:Finding 1: p rank-k (x t |x <t ) with different k values exhibits notable average KLD with p top-2 (x t |x <t ).As k increases from 2 to 8, the average KLD also increases accordingly. This finding suggests the overall next-token prediction discrepancy between top-2 and rank-k routing.Finding 2: For each rank-k strategy, apparent average KLD is observed when generating mathematical expressions (i.e., "Expression" token set). This indicates the notable differences between top-2 and rank-k routing in generating tokens for reasoning.Finding 3: For each rank-k strategy, the average KLD between p top-2 (x t |x <t ) and p rank-k (x t |x <t ) is relatively smaller when generating stopword tokens (i.e., "Stopword" token set) compared to generating mathematical expression tokens. This suggests that such predictions pose fewer challenges for rank-k routing.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We provide the detailed setting in the Experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The experiments results support the main claims of our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We have explained the GPU resources we used to measure latency.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: Our code meets NeurIPS code of ethics' requirements.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We make appropriate references to the open-source models and publicly available datasets we used in the text.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We also calculate the KLD between p top-2 (x t |x <t ) and p random-1 (x t |x <t ),p Mistral-7B (x t |x <t ) in Table 5 and present further analysis on Kullback-Leibler Divergence: It is observed that the KLD between p top-2 (x t |x <t ) and p rank-2 (x t |x <t ) is relatively small for the "Stopword" token set. This indicates that Mixtral 8x7B with rank-2 routing exhibit basic stopword generation capability similar to Mixtral 8x7B with top-2 routing. However, for the "Expression" token set, the KLD increases notably compared to that of the "All" token set (i.e., it increases by 31.13%). These observations suggest that when shifting routing strategies from top-2 routing to rank-2 routing, the reasoning capability of Mixtral 8x7B decreases more than basic generation capability.As suggested by prior works [18, 25] , this apparent reasoning ability gap can be leveraged to better amplify the reasoning strength of Mixtral 8x7B with top-2 routing. Thus, in our main experiments, we report results with fixed rank-2 for the weak activation. The same observation also applies to the weak activations of rank-3, rank-4, and random-1, albeit with varying degrees of significance. Empirically, results in Section 3.3 and 4.1 also illustrate that contrast with rank-2 routing yields generally better improvements.For Mistral 7B, the average KLD between its next-token distribution and that of Mixtral 7x8B with top-2 routing across three token sets is quite small, indicating that their overall distributions is very similar. This similarity makes Mistral 7B not an ideal weak model to contrast.B Quantitative Study of SCMoE's Unchosen Experts' Utilization.As mentioned in Section 1, unchosen experts refer to the experts not selected using default (e.g. top-2) routing. To further evaluate the utilization ratio of unchosen experts in SCMoE, we calculate the proportion of experts that are activated by rank-k routing during weak activation but not activated by top-2 routing in strong activation. Specifically, we take quantitative study of SCMoE's unchosen experts' utilization on GSM8K with Mixtral 8x7B as detailed in Table 6 . In SCMoE, the activation proportion of unchosen experts for rank-2 routing on GSM8K is 46.21% and for rank-3 routing on GSM8K is 72.62%, indicating that unchosen experts can contribute to MoE models.Here, we detail the hyperparameter setting of each baselines for DeepSeekMoE-16B [7] . It is important to note that contrastive decoding needs a suitable model to serve as an amateur. There is one fixed value for the hyperparameter α = 0.1 in Equation 6that generalizes across various domains. To provide some clarity, when α is set closer to 1, the contrastive process activates fewer vocabulary for strong activation, resulting in minimal changes after the self-contrast. Conversely, setting α closer to 0 allows more vocabulary tokens to be considered in the self-contrast process, leading to significant changes and potentially introducing more noisy information. A suitable α should strike a balance between including ideal tokens, which can lead to accurate results in the contrastive vocabulary, and avoiding the introduction of excessive noise from an overly large vocabulary. Previous work [18] on masking vocabulary based on α suggests that α = 0.1 is quite robust and generalizes well across various domains. This guides our choice in this setting.Moreover, we report the performance of each decoding method in Tables 1, 9 , 4, Figure 4 under method-specific hyperparameter settings in 7, 8, 9, 10. The performance of DoLa on Mixtral 8x7B is quite poor. Through a case study, we find that DoLa cannot terminate its generation appropriately and may exhibit degeneration, as shown in Table 11 .Table 11 : Failed examples of DoLa.Tom plants 10 trees a year. Every year he also chops down 2 trees a year. He starts with 50 trees. After 10 years 30% of the trees die. How many trees does he have left?Output: 10 -2 is 8, and 50 -(10 x 2) is 30, and 30 -10 is 20, and 20 -10 is 10, and 10 -10 is 0, and 30% of 0 is 0, and 0 + 0 is 0, and 10 -0 is 10, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -10 is 0, and 10 -You are an expert Python programmer, and here is your task: Write a python function to count number of substrings with the sum of digits equal to their length. Your code should pass these tests: assert count_Substrings("112112",6) == 6 assert count_Substrings("111",3) == 6 assert count_Substrings("1101112",7) == 12 The strength of SCMoE lies in its ability to handle tasks requiring intricate reasoning processes by leveraging both strong and weak activations, which benefits in scenarios demanding reasoning capability for next-token prediction. In contrast, benchmarks like MMLU [42] do not have explicit (verbalized) reasoning paths, which SCMoE is dedicated to helping. Therefore, SCMoE, similar to other generation decoding strategies like contrastive search [23] and contrastive decoding [18] , may not exhibit distinct advantages.NeurIPS Paper Checklist Justification: In the conclusion section, we have discussed the limitations of our work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: There is no societal impact of our work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).Unchosen experts refer to the experts not selected by default routing (e.g., top-2 routing in Mixtral 8x7B).https://www.nltk.org/ 


Decision: This paper proposes a simple inference strategy that utilizes unchosen experts in a MoE framework. The paper starts with two interesting observations: (1) increasing the number of activated experts does not necessarily improve and can even degrade the output quality; (2) output distributions from an MoE model using different routing strategies substantially differ. These observations led them to propose the Self-Contrast Mixture-of-Experts (SCMoE), which modifies the decoding strategy by contrasting outputs from strong and weak activation using the same MoE model. This method improves the performance of various tasks (GSM8K, StrategyQA, MBPP and HumanEval) of Mixtral 8x7B with a slight latency overhead. Combining SCMoE with self-consistency yields further gains.

All reviewers agree that the paper presents very insightful observations, proposes a simple, off-the-shelf decoding strategy that's massively 
effective. The findings and the method can benefit anyone studying and deploying MoE models immediately.

Reviews: review: paper_summary: paper_summary: The paper explores to leverage the contrastive information existing between different routing strategies of the MoE model to facilitate a better token decoding during inference in a training-free fashion. The paper is built upon two interesting observations: 
(1) increasing the number of activated experts does not necessarily improve and can even degrade the output quality (which increases and then drops might be due to noise/irrelevancy of other experts).  
(2) output distributions from an MoE model using different routing strategies substantially differ (which is obvious due to pre-training setting with a routing strategy).

More specifically, for ScMoE the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE mode. The authors conducted experiments using Mixtral on GSM8K, StrategyQA, MBPP and HumanEval and illustrate some noticeable performance benefits on GSM8K.
strength_weakness: strength_weakness: While the paper is well-written and comprehensive, I have following comments/questions/concerns related to the paper:

1. One question- the author submitted numbers for Mixtral default routing strategy seems to be far from the numbers reported by the Mistral authors (Table 2 from https://arxiv.org/pdf/2401.04088). What could be the possible reason for that (curiosity and not a negative point)? 
2. The author must perform the memory bottleneck for storing additional activations for their purpose of using two multiple routing strategy in SCMoE. I believe the memory bottleneck (as well as latency) of decoding will be substantially high while dealing with long-context tasks. 
3. I was wondering if the contrastive behavior can be enforced explicitly by using a small finetuning data and a contrastive loss? What are the authors thought about this?
4. What is the consistency of the findings KLD for reasoning tasks across some other math reasoning tasks apart from GSM8K? 
5. Another major weakness of the work is the tuning of the hyperparameter `\beta` for every dataset? Can authors find some discover a single `\beta` which  should work well enough for say a task category (math reasoning/commonsense).

I still find the work interesting and willing to increase the score with fair discussion during rebuttal.
questions: questions: See above.
limitations: limitations: See above.
paper_summary: paper_summary: This paper introduces SCMoE, a decoding time algorithm which can be applied off the shelf to boost MoE models' performance by contrasting the chosen experts in strong and weak activations. Experiment results show that this algorithm has an empirical advantage over baselines methods in coding, commonsense knowledge, and math benchmarks.
strength_weakness: strength_weakness: 1.	Commonsense reasoning in StrategyQA, which is a multi-hop reasoning dataset about world knowledge, does not seem to be so strongly related to coding and math which requires formal reasoning compared to formal logic (see [1], which mainly focuses on logic, algorithm, and math as reasoning tasks). Do you anticipate performance boost in logic reasoning tasks? 

2.	Relatedly to the last question, do you think the proposed algorithm will also help general world knowledge reasoning without implicit complex reasoning chains such as MMLU? 

	
3.	Is it possible to find the ideal strong activation in real-life workflows only given the user query?

[1] Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., & Xie, X. (2023). Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.
questions: questions: See above
limitations: limitations: The authors briefly mentioned limitations of their paper in the conclusion section but they could further discuss other limitations including limitation of datasets evaluated etc.
paper_summary: paper_summary: This paper proposes a novel approach called Self-Contrast Mixture-of-Experts (SCMoE) to improve the utilization and performance of Mixture-of-Experts (MoE) models. The key contributions are:

1. Exploratory studies showing that increasing the number of activated experts in MoE models does not always improve output quality and that different routing strategies lead to substantially different output distributions.

2. The SCMoE method, which leverages unchosen experts in a self-contrast manner during inference. It determines next-token probabilities by contrasting outputs from strong and weak activation using the same MoE model.

3. Experimental results demonstrating that SCMoE consistently enhances the reasoning capabilities of Mixtral 8x7B across various domains.

4. Combining SCMoE with self-consistency yields further gains.

5. The proposed SCMoE method is conceptually simple, computationally lightweight, and incurs minimal latency compared to greedy decoding.
strength_weakness: strength_weakness: 1. The idea of SCMoE is somewhat similar to the cited paper "Contrastive decoding: Open-ended text generation as optimization (Li et. al. [20])," although I understand that SCMoE is better adapted to MoE LLMs and has lots of novelties in other aspects. The authors could provide more discussion in the related work section about the similarities and differences between the two approaches.

2. The paper lacks clarity on how certain hyperparameters are chosen. For example, in Section 3.2, the authors state that "for the weak activation, we only consider the rank-k routing with k=2" but do not provide an explanation for why k cannot be 3, 4, or 5. It would be helpful to know if choosing k=2 is motivated by faster inference times, as the top 2 models will be used regardless.
questions: questions: 1. See Weakness 2.

2. I am curious why Mixtral has decreased performance when averaging over more expert models. Could the authors give some explanations? It is because of something specific to pretraining methodology of Mixtral/DeepSeekMoE models?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. It can be seen as a decoding method utilizing divergent information from different routing strategies. This method introduces slightly more latency overhead and improves the performance of various tasks through experimental evaluation.
strength_weakness: strength_weakness: * This inference method employs two models, one with top-2 routing and the other with rank-k routing. It produces double memory overhead which is significant, especially for larger models.
* The choosing of α in Equation 6. seems non-trivial, it would be great to provide some insight on this.
questions: questions: * The extra latency cost is minor, is it because the two-model inference is parallelly implemented? 
* The “+2=5 (...)” in Figure 2.(c) is a little bit confusing to me, could you explain further about it?
limitations: limitations: Yes.

Comments: title: Response to Authors.
A Gentle Reminder
comment: Thank you for the thorough response, I encourage you to polish the submitted version. I will raise my score.
Dear reviewer,

Can you let the author and us know if you've read the rebuttal, and if you have any further comments?

Thanks,

AC
Dear Reviewer KVjj,

Thank you once again for your constructive comments on our submission. Your feedback is really valuable to the improvement of our paper. As the discussion phase nears its conclusion, we eagerly await any further comments or questions you might have.

We hope that our responses have adequately addressed your concerns. If you find our revisions satisfactory, we would greatly appreciate it if you could consider raising the score of your assessment. If there are still issues to be addressed, please let us know, and we would be more than happy to engage in further discussion.
Thank you for addressing my concerns! I’ll keep my positive score for the paper.
Thank the authors for their answers to my concerns. They have adequately answered every question I raised to my satisfaction, and therefore, I will keep my current score of an acceptance.