Unchosen Experts Can Contribute Too: Unleashing MoE Models’ Power by Self-Contrast
Authors: Chufan Shi
Cheng Yang
Xinyu Zhu
Jiahao Wang
Taiqiang Wu
Siheng Li
Deng Cai
Yujiu Yang
Yu Meng
Venue: NeurIPS 2024 poster (2024)
Keywords: Mixture-of-Experts
Self-Contrast
Text Generation
Primary area: natural_language_processing
TL;DR: Enhancing Mixture-of-Experts models by utilizing unchosen experts in a self-contrast manner.

Abstract: Mixture-of-Experts (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the input sequence activates a different subset of experts determined by a routing mechanism. However, the unchosen experts in MoE models do not contribute to the output, potentially leading to underutilization of the model's capacity.
In this work, we first conduct exploratory studies to demonstrate that increasing the number of activated experts does not necessarily improve and can even degrade the output quality. Then, we show that output distributions from an MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically. 
Motivated by these findings, we propose **S**elf-**C**ontrast **M**ixture-**o**f-**E**xperts (SCMoE), a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. 
In SCMoE, the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE model.
Our method is conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding. 
Experiments on several benchmarks (GSM8K, StrategyQA, MBPP and HumanEval) demonstrate that SCMoE can consistently enhance Mixtral 8x7B’s reasoning capability across various domains. For example, it improves the accuracy on GSM8K from 61.79 to 66.94. 
Moreover, combining SCMoE with self-consistency yields additional gains, increasing major@20 accuracy from 75.59 to 78.31.

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
Chufan Shil*
Cheng Yang"" Xinyu Zhu Yujiu Yang'
Jiahao Wang
Taiqiang Wu
Siheng Li' Deng Cai
Yu Meng?
TTinnhua University University of Virginia
Thee University Hong Kong *Tencent AI Lab
scf22@mails .tsis..ssiiii ssinghua.edu..
yang yujiuusuuu..... tewaaassiinguuuu edu. on yumeng5evirginia.cc edu
Abstract
Eexture-of-Experet (MoE) has emerged as a prominent architecture for scaling model size while maintaining computational efficiency. In MoE, each token in the imput sequence activates a different subset experts determined by a routing mechanism. However, the unchosen experts in MoE models do not contribute to the output, potentially leading to underutilization the model's capacity. this work, first conduct exploratory studies to demonstrate that increasing the number of activated experts does not necessarily improve and can even degrade the output quality. Then, show that output distributions from MoE model using different routing strategies substantially differ, indicating that different experts do not always act synergistically. Motivated by these findings, we propose Self-Contre Mixture- of of-Experts (SCMoE), a training-fre ffree strategy that utilizes unchosen experts in a SCMoE, the next-token probabilities
self-contrast manner during inference. are determined by contrasting the outputs from strong and weak activation using the same MoE model. Our method conceptually simple and computationally lightweight, as it incurs minimal latency compared to greedy decoding. Exper- iments on several benchmarks (GSM8K, StrategyQA, MBPP and HumanEval) demonstrate that SCMoE can consistently enhance Mixtral 8x7B's 's reasoning capa- bility across various domains. For example, improves the accuracy on GSM8K from 61. .979 to 66.94. Moreover, combining SCMoE with self-consistency yields additional gains, increasing major @20 20 accuracy from 75.59 to 78.31.
Introduction
Scaling up model parameters, dataset size and training time has been considered the most direct and effective approach to improving foundation models' performance [T-3]. However, scaling dense models substantially increases computational costs, which poses a significant practical challenge. Mfxture-of-Expee -xperts (MoE) 4 has emerged compelling solution for optimizing the balance between model capacity and computation overhead the era of large foundation models.
MoE models achieve the goal by sparsely activating only portion of the parameters for each specific input. Specifically, in MoE models, parameters are grouped into bunch of experts, MoE models only activate some of them for processing a given input. This selective activation is achieved through a routing mechanism that dispatches each input token to a fixed number of experts (e.g, top-k [ 10 II). Therefore, compared to their dense counterparts, MoE models enjoy more efficient training with significantly reduced computational costs 9 At the inference stage, they typically adhere to the same routing strategy the training stage, activating only small fraction
routing [6
Equal Contribution, Source code is available at attp:e::/gttubb.oo////baaaaa/a/e Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ensemble Routing
SCMoE
78
50
43.0
5.15
3.46
74
47
38.5
7.92
Accuracy
Pass
07
70
44
34,0
66-
29.5
(a) GSM8K
(b) StrategyQA
(c) MBPP
(d) HumanEval
Figure I: Performance comparison between increasing the value of top-k (i.e., ensemble routing) and
SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
of experts. Basically, for each input token, most of the well-trained experts do not contribute to the output prediction. As result, the potential of utilizing more experts during the inference stage to enhance performance remains underexplored.
In this paper, we investigate the impact of unchosen expert the performance of MoE models and explore their suitable usage. A direct hypothesis is that incorporating more experts improves MoE models and helps solve more difficult problems 12) However, in our exploratory experiment on Mixtral 8x [6], we find simply raising the number of activated experts (blue lines in Figure [T) does not lead to stable improvements and may even hurt performance different tasks. This indicates that unchosen experts may contribute little or even negatively to the final performance, which is contrary to the common perception of unchosen experts as candidates of positive power.
Inspired by the finding, we further dive deep into the difference between the output probability distributions of MoE models applying different routing strategies. As shown in Figure 3 we calculate the Kullback-Leibler Divergence (KID) between the token distributions obtained from the default 2 routing and rank-k routing, and find apparent discrepancy. The discrepancy is particularly evident in the parts that require rigorous reasoning. This suggests that different experts do not always act synergistically; instead, they may exhibit conflicting behaviors.
Therefore, introduce Self-Contran Mixture-of- ofiee-e-t-----tttt-e (SCMoE), which can convert the negative effects brought by unchosen experts into positive ones through contrasting the output logits obtained using different routing strategies. Specifically, the probability of next token is based on the logits difference between strong and weak activation of the MoE models. For "strong activation" and "weak activation". we use the top-2 routing strategy (Figure (a)) and the rank-k routing strategy (Figure ( (b)) respectively. Thus, SCMoE enables unchosen experts to contribute to the prediction. An overview of how SCMoE works is presented Figure (c).
Experimental results on various benchmarks across different domains demonstrate that SCMoE significantly enhances Mixtral 8x7B's reasoning capability (Section 3). Specifically, compared to greedy decoding, the accuracy increases from 61.79 to 66.94 (+5.15) on GSM8K, 72.83 to 76.29 (+3.46) on StrategyQA, and the pass@) accuracy increases from 46.20 to 48.80 (+2.60) on MBPP and 33.54 to 41 .66 (+7.92) on HumanEval. Further analysis shows that SCMoE can even surpass the result of using self-consistency with major @5 (66.87) on GSM8K. What's more, combining SCMoE with self-consistency can further boost the model's performance, improving major@ 20 accuracy from 75.59 to 7 (+2.72) on GSM8K. Regarding inference efficiency, it turns out that SCMoE incurs only a minor 1.) delay compared to greedy decoding, which is competitive among several strong decoding baselines. To sum up. empirical results and comprehensive analyses demonstrate that SCMoE is a both effective and efficient approach to unleashing MoE models' power.
2
Method
In this section, we first provide preliminary introduction of MoE models. Then, we present an analysis based on next-token distribution KLD to reveal the divergence between different routing
3 Unchosen experts refer to the experts not selected by default routing (e.g. top-2 routing in Mixtral 8x7B)

---

Oukkk
hours had
3xc Xeyy
hours had
hours had
Fiup-2
F Xxr
MoE Models
Router
O Expert
top-2 routing rank-k routing
1,2,3,4: Gate Value Rank
G..) On Saturdays, she had 2 hours, 1
(a) T Routing
(b) Rank-k Routing
(c) SCMoE
Figure 2: (a & b) Given an input h, (a) and (b) demonstrate the workflows of top-2 routing and rank-k routing (e.g., k=2). We use two MoE layers as a simple schematic, omitting other layers in MoE models. Note that, in the second MoE layer, rank-k routing activates the unchosen expert in top-2 routing; (c) An illustrative example of how SCMoE works, which contrasts 2x (T<)) with ( The complete question and answer for this example are shown in Figure
strategies in MoE models. This analysis motivates the introduction of SCMoE, a self-contrast method to leverage the contrastive information existing between different routing strategies in MoE models.
2.1 Preliminary
Transformer-basee MoE models, the conventional Feed- Feed-Forward Network (FFN) substitued with the MoE layer [15]. Typically, each MoE layer consists of router R and set of experts ( 1 For a given input sequence Z'z = (I1, 42: xt-1), the router allocates each token in ø<z to a specific subset of experts, which subsequently activated to process the tokens. Specifically, given
i=1-
each token's s hidden state h, the router first calculates an initial gate value vector w across the N experts as follows:
w Softmax(W (W.h)
(I)
where W, denotes the weight matrix of the router. Each element in represents the probability of activating the i-th expert.
After that, the router applies a routing strategy (e.g., top-2 or rank-k routing in Section 2.2 to determine the subset of experts be activated. Then the w, of the unchoosen expert is set to 0 and renormalized to w accordingly. Subsequently, the output o the MoE layer is computed as the weighted sum of outputs from the activated experts:
Σ [0)
W. E.(h)
Once the input sequence Zez has undergone a complete forward pass through the MoE model, the next-token distribution p(x)) is computed based the output of the final layer. A decoding algorithm is then applied to predict z; from the vocabulary V based on p(xa(=<e))
2.2 Divergence Between Different Routing Strategies: An Exploratory Analysis
As depicted in Figure [ unchosen experts may contribute little or negatively to the final performance. Based on this finding, we are inspired to study the difference of output probabilities using different routing strategies. Specifically, we conduct an analysis Mixtral 8x7B [16), with two different routing strategies, i.e., top-2 routing and rank-k routing, which are detailed as follows.
Top-2 Routing. Top-2 routing (Figure 2 (a)) [4] is the default routing strategy of Mixtral 8x7B, which activates the two experts with the highest values in w. In this setting, the renormalized gate value for the i-th expert, w. is defined as follows:
W:
i E top(w. (w,
W:--
Ejempor wy
(3)
き top( (w,
3

---

Question: Hallic had drnce practice for 1 hour on Tuesddyy and 2 hous on Thursdyss. On Saturdays, the had dunce practice thar lasted twice long Tuesday's night class. How hours week did she have dance practio??
Answer: On Tuesdays, she had
Thurdays, had hours. On Saturdays, she had x hours. 1+ ++ 2
The answer S.
5.01
1.47
0.30
7.48
3.0
2:37
1.82
3.00
12:14
6.22
4.57
5.74
3.88
3.63
1.6:
0.32
0.77
0.56
4.24
13:23
21.0:
32.30
18.95
223
4.01
13.23
337
14.03
24.97
7:23
11.05
30.52
:4.34
7:24
0.72
134
0.72
9.39
10.23
13.03
62.33
23.24
213
8.93
223
6.83
13.33
12.78
6.2
19:0
22.69
42.44
23.33
123
232
2.00
4.71
13:13
23.3:
5783
30.13
10.2
13.85
14.18
10.11
18.39
4:.11
278
21,81
8.59
10.05
14.01
25:12
40.22
16.45
11 10
12:85
25.16
13.45
4.77
22:85
16.19
16.5
16:17
71.44
742
13.07
6.30
12:55
17:41
20.31
69.39
16.00
22. 93
25.15
23 57
12.74
37.01
27 45
25.31
25.37
76.56
17.75
16.16
18.52
24:31
6.18
18:52
had
On
s1c
Tokes
0.00
0.05
0.24
3.07
0.00
0.03
0.02
5.00
6.11
0.28
0.53
3.36
3.03
0.02
0.09
0.00
0.00
0:33
5.26
17.42
8.47
3.55
27.30
27.46
0.33
5.50
0.79
4.25
8:15
3.72
2.00
24.51
12:23
22:72
11.93
2.56
3.25
15.87
7.81
.3.89
57447
23.26
0.77
1.29
3.45
5.46
3.50
5.50
1.03
15.82
15.05
4.16
4.55
25.02
10.3 31
5:22
13:25
961
17.92
15:22
9:39
1.1
1.54
16.85
7.57
3.53
12.02
14.05
14.18
25:35
7.55
16:85
19.00
12:55 55
11
76.01
17.27
17:60
2.06
11:40
9.56
16.22
26:15
75.50
27.20
15.63
25.21
24.85
8.33
20,44
22,56
23.00
26.40
40:39
33.33
13.76
13:23
13.53
6.75
2.03
30,45
41.03
23.79
28.30
15.79
18:03
20,35
2
hous
On
.5
days
.be
had Tikes
.X
2
bours
5:11
6:31
0.31
3.01
3.00
0.03
0.53
0.50
5.00
0.53
0.30
0.50
3.11
3.05
0.01
0.03
0.00
0.00
0.39
7.59
10:36
3.5
0.11
18:54
4.30
5.40
0.42
52
1.13
2.00
0.60
0.55
2.02
2.44
1:38
28.47
1.30
1.3
13.25
3722
1.40
8.07
4.23
17:82
34.30
3.7
4.33
212
15:26
21114
3.39
3.20
1.85
3.51
25.00
4.19
9.81
25 15
15.30
15:13
12:45
3.55
49
18.82
11.91
8.14
0.65
7.20
25.0
10.12
K110
1355
15
24.17
8.00
12.52
32.5 04
13:25
39.54
31
4.73
47
20 64
13.39
10.85
927
9.84
22.24
7.47
14.84
22.14
13.09
2
29:35 1
13.05
10:43
54
17.59
16.11
14.50
4.05
31
The
5
Tokea
Figure 3: Heatmap of Kullback-Leible Divergence between the output distribution of top-2 routing
strategy ( 2 (2g xx<))) and different rank-k routing strategies ( Prank k. (x)((=<<)) ). The i in rank-k routing ranges from to 8. The values in the heatmap are scaled by 10 This example is taken from the GSM8K dataset. An additional quantitative study of the KLD is provided in Appendix A
where top(w,2) 2) returns the indices of the largest 2 elements in w.
Rank-k Routing. The rank-k routing strategy (Figure (b) only activates one expert, whose initial gate value is ranked at k in The renormalized gate value w,, is defined as follows:
{
1, = rank(w, k) 0, i 전 rank(w,k)) k)
w. =
where rank(w. (w, k)) returns the index of the k-th largest element in w. For Mixtral 8x7B, k is enumer- ated from to 8. We employ rank-k routing representative strategy to utilize unchosen experts of top-2 routing (Additional statistics on the utilization ratio of unchosen experts are provided in Appendix|B] (B)
Given an input sequence Tec, we denote the next-token distributions using top-2 routing and rank-k routing as 2hop-2 ( and Prank (xx,) (2xa) (ree)). respectively. Then, we compute the Kullback-Leible Divergence (KI.)) between Puop 2 (xee)) and Prank& (xxe((ac)) on GSM8K dataset. qualitative illustration is presented in Figure aadd a more detailed quantitative study is included in Appendix [ As shown in Figure 3 the KLD between Prop-2 Prop-2(z// (x and Prank-1 (xz(=<e)) is relatively minor, suggesting high simmariit in their next-token prediction capabilities. However, for l values ranging from 2 to 8, we identify three key findings:
Finding 1: Prank-k (x2 (x)))) with different & values exhibits discernible KLD with Prop-2 (xe)))))) As k increases from 2 to 8, the KLD increases accordingly. This finding indicates the overall next-token prediction capability gap between top-2 and rank-k routing.
Apparent KLD is observed when generating reasoning sequences, such as mathematical expressions (e.g.. "1+2+2=5") and the initiation of reasoning steps (e.g., "On Thursday"). This suggests notable differences between top-2 and rank-k routing in generating tokens for reasoning.
Finding 2:
Finding 3: For the generation of function words (e.g.. "is") and punctuations (c.g.. "."), the KLD between (x and P (xx( is relatively smaller than that for generating critical reasoning sequences. This indicates that such predictions pose fewer challenges for rank-k routing.
To sum up, it is observed that, in scenarios demanding reasoning capability for next-token prediction, MoE models with top-2 and rank-k routing strategies demonstrate distinct generation behaviors. This phenomenon suggests that different experts do not always act synergistically, and could in fact exhibit conflicting behaviors. To harness such information introduced by more experts, a feasible approach is to apply contrastive methods [17]8] to transform the observed negative impacts into positive ones.

---

Therefore, we propose to leverage the contrastive information existing between different routing strategies of the MoE model (e.g. top-2 routing and rank-k routing) during inference decoding.
2.3 SCMoE: Self-Contrast Mixture-of-Expere
We introduce Self- CoffrConttatt Mixture of Experts (SCMoE), an MoE-native self-contras decoding method. The fundamental idea behind SCMoE is determine next-token distribution of an MoE model by leveraging the contrastive information between its strong and weak activation, thereby amplifying the desirable behaviors of the strong activation. this context, "strong activation" " and "weak activation" of an MoE model refer to the activations obtained by adopting routing strategies with inherent differences (e.g., top-2 routing and rank-k routing). combinations of routing strategies that can be applied for strong and weak activation. We consider the case of top-2 routing for strong activation and rank-k routing for weak activation.
MoE. model offers flexible
Specifically, in SCMoE, given the output logits of strong and weak activation, we use the following equation obtain the adjusted logits for next-token prediction:
{ (1 + B) 2 = i| - B. 2ramk = i((x<)) i E Vvatid -oo
(00 = i1(x0)))
(5)
i i4 Vvalid
where 3 (0, is hyperparameter modulating the intensity of the contrastive penalty. ( (x.ce) and ( (x))c) represent the output logits prior to the softmax operation. Vealid is subset of the vocabulary V to restrict the search space:
Vealid
2 (zt i|ıي<)) log max 2 =)(,<ر))
(6)
jEV
where E (0, 1] is hyperparameter to control the size of Veatid by masking out tokens that are assigned lower logits. Empirically, a is set to 0.1.
Figure (c) presents an example of how SCMoE works, In this figure, the output logit of"="" consistently high across both top-2 and rank-k routing strategies. Notably, the logit of the ground-truth token +" shows an apparent increase with the top-2 routing compared to rank-k routing, SCMoE capitalizes on this contrast boost the logit of " "_",, thereby generating more accurate output.
=" is
Experiments
3.1 Datasets and Models
To measure the effectiveness of SCMoE, we consider several challenging tasks for LLMs, including mathematical reasoning, commonsense reasoning, and code generation. For mathematical reasoning
and commonsense reasoning, we select GSM8K [19] and StrategyQA [20] respectively, reporting
accuracy. For code generation, use HumanEval (1 and MBPP 22 reporting pass@1 accuracy.
We choose Mixtral 8x7B 6 as our backbone model.
3.2 Setup
Mixtral 8x7B's default top-2 routing as the strong activation. For the weak activation, we only consider the rank-k routing with 2. For the penalty strength 3, we search from [0.1, 0.3, 0.5, 0.7, 0.9).
As discussed in Section 2.3 in SCMoE,
We employ the representative routing- based methods (i.e., dynamic and ensemble routing) as the baselines of experts utilization for MoE models. Noting that SCMoE can be seen decoding method, we also select commonly used search-based methods (i.e., contrastive search, contrastive decoding and Dola) for LLMs additional baselines. The details of each method are listed below:
Greedy. Greedy chooses the highest probability token at each step.
Dynamic Routing. Inspired by 14), during inference, the number of activated experts is not fixed. Instead, threshold is set, and experts are selected in order from highest to lowest scores until the threshold is exceeded. The range of the threshold is [0.2, 0.3, 0.4, 0.5, 0.6].
Ensemble Routing. Ensemble routing activates & experts for inference with greedy search, where A ranges from 1 to 8. Note that when A = 2, it is the same as greedy.
5

---

Table 1: Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with Mixtral 8x7B. We report the best results for each method here. The performance of each method with different hyperparameters can found in the Appendix Table 7
Method
GSM8K
StrategyQA
MBPP
HvmanEval
Greedy
61.79
72.83
46.20
33.54
Routing based
Dynamic Routing
61.11
74.41
47.80 46.20
38.41
Ensemble Routing
63.84
74.37
37.20
Search- based
Contrastive Search
60.96
74.85
46,20
36.59
DoLa
49.96
71.04
33,00
12.80
Contrastive Decoding
62.24
74.45
45.20
35.98
SCMoR 66.94
76.29
48.80
41.46
Contrastive Search. Su et al. [23 use look-ahead mechanism and penalizes tokens compromising the isotropy of the model's latent space. We search the penalty degree from [0.3, 0.4, 0.5, 0.6].
Contrastive Decoding. Li et al. [8] search for tokens that maximize the probability difference between the base LLM and an amateur model. We use Mixtral 8x7B as base LLM and Mistral-7B [I6 as the amateur. We search the strength of the amateur penalty 3 from [0.1, 0.3, 0.5, 0.7, 0.9).
24 obtain the next-token distribution by contrasting the logits differences between the last layer and a premature layer. The premature layer is dynamically selected from a pre-specified set of layers. Following DoLa [24), we test two sets of layers: even-numbered layers from [0, 16) and from [16, 32) respectively.
DoLa. Chuan et al.
3.3 Results
Unchasen experts can contribute too. We present the results for each method in Table ! For dynamic routing, compared with the greedy approach, dynamically selecting the number of experts to use can enhance Mixtral 8x7B's performance except for GSM8K (GSM8K -0.68, Strategy +1.58, MBPP +1.60, HumanEval + 4.87). This observation indicates that adopting the same top-2 routing strategy during inference as in the training stage may not be optimal for MoE models. Furthermore, for ensemble routing, incorporating additional experts into inference can also improve performance for each task except for MBPP (GSM8K+ + 2.05, Strategy QA +1.54, MBPP + O, HumanEval + 3.66). This findings implics that unchosen experts can be further utilized.
SCMoE unleashes MoE models' power. SCMoE enhances mathematical reasoning by +5.10 increase on GSM8K, commonsense reasoning by a +3.46 improvement on StrategyQA. Moreover, in code generation, SCMoE gets improvements of +2.60 and +7.92 on the MBPP and HumanEval, respectively. In contrast, traditional search-based methods do not demonstrate substantial improve- ments on MoE models. In particular, DoLa's performance not only fails to surpass, but actually falls below the greedy baseline, particularly due to its inability to terminate generation sequences appropriately (for specific examples, refer Table [T in the appendix). Meanwhile, contrastive decoding with Mistral 7B as the amateur model does not result in consistent improvements, and even decrease pass@1 accuracy on MBPP (-1.00). Contrastive decoding necessitates a suitable amateur model for effectiveness [8 25 but selecting separate amateur model with same vocabulary is not always feasible. comparison, SCMoF capitalizes on the MoF models' inherent strong and weak activation to conduct self-contrast. Different weak activation can be viewed as different amateur models, offering higher flexibility and thus help to find the ideal one for contrast.
Analysis
4.1 Impact of Weak Activation
In our main experiments, we use weak activation with rank-2 routing across all benchmarks. In fact, SCMoE offers the flexibility employ various routing strategies to determine weak activation. Thus,

---

SCMoE w/rank-k
Greedy
SCMoE wirandom-1
79
77
49
39
Accuracy
Pars
65
75
36
73
35
(a) GSM8K
(b) StrategyQA
(c) MBPP
(d) HumanEval
Figure 4: Experimental results of different weak activations. We set the strong activation with top-2
routing in SCMoE. The detailed results with their hyperparameters are report in Appendix Table g
Table Experimental results of different strong activations. We the weak activation with rank-2 routing, For each benchmark, we select the top-k routing yielding the best performance in Figure as the ideal strong activation. The specific hyperparameter settings can be found Table
Method
GSM8K
Strategy QA
MBPP
HumanEval
SCMoF
66.94
76.29
48.80
41,46
SCMoE w/ ideal strong activations
68.92
76.42
50.60
41.46
in this section, we further explore the effects of selecting different weak activation. Specifically, first set rank-k routing with k ranging from to 8 as different weak activation and then investigate corresponding performance changes. Besides rank-k routing, we also consider random-1 routing strategy to serve as an alternative weak activation for SCMoE. In the random- I routing strategy, at each MoE layer, the router randomly selects one expert process current input token.
The experimental results for each candidate weak activation are presented in Figure Firstly, compared to the greedy baseline (represented by the red line), there is noticeable enhancement in GSMSK, StrategyQA and HumanEval regardless of the chosen weak activation in SCMoE. Moreover, when using random-1 routing (represented by the green line), there is still an improvement compared to greedy, which demonstrates the advantage of SCMoE in utilizing its weak activation for self- contrast. Overall, using rank-2 routing weak activation can provide consistently good performances, and further exploring rank-k or other routing strategies may bring additional improvements.
4.2 Impact of Strong Activation
Self Corsisiecy of SCMoE
SCMaE
As revealed by Figure I using default top-2 routing is not optimal for all tasks. For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the high- est accuracy on HumanEval and StrategyQA. This leads consider whether enhancing the strong activation in SCMoE can further unlock MoE models' potential, To this
Self Cof--Corssttncy Ilmmpeeattre Sampling
Greedy
Accuracy
end, we adjust the strong activation of Mixtral 8x7B top- 3 for GSM8K, and to top-4 for StrategyQA, MBPP, and HvmanEval, while keeping the weak activation with rank- 2 routing as before. The experimental results, as shown in Table 2] reveal that enhancing the strong activation of SCMoE can further boost MoE models' performance, Compared the previous best performance achieved when only utilizing top-2 routing for strong activation, this ad- justment improves Mixtral 8x7B's performance by 1.98 on GSM8K, 0.13 on StrategyQA, and 1.80 on MBPP.
60
55
Majorāä
Figure 5: Experimental results on com- bining SCMoE with self-consistency on GSM8K using Mixtral 8x7B.

---

Table 3: Averaged decoding latency for each method. CS is short for contrastive search and CD is short for contrastive decoding. We set h = 3 for ensemble routing, while for dynamic routing we set threshold = 0.5. The speeds are tested on 4 A100 40G with batch size
Method
Greedy
Ensemble
Dynamic
CS
DoLa
CD
SCMoE
Latency (s / 512 tokens)
50.32
59.82
54.85
81.73
53.30
72.04
65.47
Lateney Ratio
x1.00
x1.19
x1.09
x1.62
x1.06
x1.43
x1.30
Table 4: Experimental results on GSM8K, Strategy QA, MBPP and HumanEval with DeepSeekMoFE 16B. We report the best results for each method here. The performance of each method with different hyperparameters can found in the Appendix Table TO
Method
GSM8K
Strategy QA
MBPP
HumanEval
Greedy 18.95
60.41
35,20
26.83
Routing-based
Dynamic Routing Ensemble Routing
19.71
60.63
34.80 35,20
25.00
19.71
60.41
26.83
Search-based
Contrastive Search
19.94
61.77
33.40
25.00
DoLa
18.27
61.72
36.00
22.56
SCMoE
20.77
62.99
37.20
28.05
4.3 Combining SCMoE with Self-Consistency
Using cof-cconsitency 26 for multiple sampling and taking majority vote to determine the final answer is common method to improve LLMs' performance. Therefore, we explore whether SCMoE can combined with self-consistency. For vanilla self-consistency, we use temperature sampling with temperature 0.7 to reach the best baseline performance (27] For self-consistency with SCMoE, we simply employ
0.5, rank-3 routing as weak activation, according to the best hyperparameters setting from Table 8 It is worth noting that since SCMoE already has mask
0.1 to limit the sampling range the vocabulary, we do not perform any additional temperature processing on the final logits. As shown in Figure SCMcE (67.94) yields comparable results with major@ 5 (66.87). Furthermore, SCMoE can enhance the major @20 20 accuracy from 75.59 to 78.31 (+2.72) on GSMSK.
4.4 Latency
We further evaluate the impact of SCMoE on decoding latency and compare it with other methods on Mixtral 8x7B. Specifically, we first input 32 tokens to each method and then force them generate sequence of 512 tokens to calculate the latency. The results in Table show that SCMoF increases the decoding time by factor of 1.30x compared to greedy. When compared with other methods, SCMoE does not introduce a significant amount of latency, especially when compared to contrastive search (x1.62) and contrastive decoding (x1.43). Moreover, SCMoE even surpasses the results of using self-consistency with major@5 5 on GSM8K, which has 5x latency compared to greedy. Therefore, the latency of SCMoE can be considered negligible, making both effective and efficient approach.
4.5 Employ DeepSeekMoF
We further explore the adaptability of SCMoE to other MoE models. We conduct experiments on DeepScekMoE-166 (28]. DeepSeekMoE-161) employs fine-grained expert segmentation and shared expert isolation routing strategies, which is different from Mixtral 8x7B (6). We detail the hyperparameters settings of experiments in Appendix C is worth noting that contrastive decoding needs a suitable model to serve as an amateur. However, DeepSeckMoE-11e smaller model with the same vocabulary, so DeepSeekMoF-166 does not have the contrast decoding baseline. As depicted in Table SCMoE effectively unleashes the potential of DeepSeekMoF. 16B. Specifically, compared to greedy baseline, SCMoE demonstrates improvements across all tasks: it enhances mathematical reasoning by 1.82 GSM8K, commonsense reasoning by 2.58 on
does not have
8

---

StrategyQA QA code generation by 2.00 on MBPP, and 1.22 on HumanEval. In contrast, other methods, regardless of routing- based or search-based, struggle to outperform the greedy baseline. These results demonstrate that SCMoE can be successfully applied to other MoE models.
basting-asseddee
5 Related Work
The Mfxture-of-Expee (MoE) model was initially introduced by A. Jacob al. [29). Previous studies have demonstrated that sparsely gated MoE models can significantly improve model capacity and efficiency, enabling superior performance compared to dense ones [4] 5 I1 30).
Mixture-of-Experts
In MoE models, a static number of experts are activated regardless of the varying complexity presented by input tokens. Typically, top-1 or top-2 experts are activated in these models 15 10 In the era of LLMs. numerous extensive open-source models based on MoE architecture have emerged. Specifically, both Mixtral 8x7B [6] and Grok-I g introduce an 8-expert MoE that uses a top-2 routing algorithm during inference. DeepScekMoE 7 and QwenMoE [9], on the other hand, both employ fine-grat gene-graieed expert segmentation, applying shared experts with N routed experts, As result, they use k+2 experts for inference, with 2 fixed shared experts and top-k routed experts.
While several works have attempted to examine pruning or dynamic routing algorithms for MoE mod- els [31 32 14 from the perspective of reducing computational costs while maintaining performance. Our approach differs in that we investigate the utilization of unchosen experts in self-contrast manner to boost MoE models' capability without increasing too much computation.
Contrast in Language Modeling The idea of employing contrast to enhance language modeling has been explored through various approaches. Specifically, the contrast enables language models to discern between desirable and undesirable behaviors, a distinction that the conventional maximum log -igliiilihood modeling often fails to adequately capture [33]. One line of research focuses on training- -time optimization. Reinforcement learning from human feedback (RLHF) (34 trains reward models by contrasting the rewards associated with desirable outputs to those of undesirable ones, and then optimize the I.L.M to maximize rewards through reinforcement learning. RRHF [ DPO [38, and PRO 339] eliminate the necessity of constructing reward models and instead directly optimize LLMs by contrasting preferred responses versus dispreferred ones. Another research avenue focuses on inference-time optimization. DExperts [17] fine-tunes two models with desirable and undesirable attributes separately, guiding the base model by leveraging the contrast between those models. Contrastive Decoding 118 25
contrasts base model with an amateur model to mitigate undesirable tendencies of the amateur. Emulated fine-tuning [40] and proxy-tumine [41] achieve training- free alignment in a similar way, applying the contrast between aligned and unaligned models reward signal to guide the decoding process of larger unaligned LLM. Contrastive Search 23 uses a look-ahead contrastive mechanism and penalizes tokens compromising the isotropy of the model's latent space. DoLa [24] obtains the next-token distribution by contrasting the logits differences between the last layer and a premature layer to improve factuality.
Our research focuses on inference-tine optimization. Distinct from the above methods that mainly utilize contrasts between different models, our work leverages the contrastive information among strong and weak activation of MoE models to unleash their potential through self-contrast.
Conclusion
In this work, we develop Self-Contrast Mixture-of-Expere (SCMoE), a conceptually simple and computationally lightweight strategy to unleash MoE models' power via self-contrast. We find that different routing strategies within an MoE model output results with considerable divergent information. Utilizing this information in self-contrast manner can further enhance MoE models' reasoning capabilities in next-token prediction. Experimental results show that SCMoE improves the MoE models' performance on multiple benchmarks with only minor latency increase at inference time. Due to resource constraints, our main limitation is that we cannot further explore the performance of SCMoE on larger MoE models such as Mixtral 8x22B or DeepSeek-V2 V2. Overall, SCMoE is critical step to leverage the inherent self-contrast features of MoE models, and offers new insights to the utilization of unchosen experts.

---

Acknowledgements
The authors would like to thank Zicheng Lin, Xinzhe Ni, Yifan Wang, and Qingyan Guo for their valuable feedback and discussions. This work was partly supported by the Shenzhen Science and Technology Program (JCYJ202208 00002208111010111 18101014030) and the "Graph Neural Network Project" of Ping AnTecchology (Shenzhen) Co., Ltd.
References
[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint arXiv:2001.0836/ 2020.
[2] OpenAl. Gpt-4 technical report. 2023.
www.sin..pera....o http://paaaa..ccppparrrs///////e 4.pdf//
[3] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillierap, Jean- baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, arXiv preprint arXiv:2403.05530, 2024.
[4] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
https: / www.w..ww.eeeeeeee............ee net (etrummid=Blcceee
[5] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe Le, James Laudon, et al. Mixture-of-expers with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
[6] Albert Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arXiv preprint arXiv:2401. 040088 2024.
[7] Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of cexture-offexpert of-experts language models. arXiv preprint arXiv:2401.06600 2024.
[8] xAI. Grok-1 model card.
2024.
[9] Qwen Team. Qwenl.5-moe: 5-moe: Matching 7b model performance with 1/3 activated parameters. 2024.
[10] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:3, 2022. URL (tRp:
org htt.a/rroorr/pppps//////1111
html
[11] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, al. Glam: Efficient scaling of language models with mixture-of saturr--o-epperrtt. In International Conference on Machine Learning. pages 5547-5569. PMLR, 2022.
[12] Ran Avnimelech and Nathan Intrator. Boosted mixture of experts: An ensemble learning scheme. Neural computation, 11(2):883-99) 1999.
[13] Dongrui Wu, Chin-Teng Lin, Jian Huang, and Zhigang Zeng. On the functional equivalence of tsk fuzzy systems neural networks, mixture of experts, cart, and stacking ensemble regression. IEEE Transactions on Fuzzy Systems, 28(10):2570-233 25800000, 2019.
[14] Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint 2733v:2403.07652, 2024.
10

---

[15] Dmitry Lepikhin, HyoukJoong Lee. Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with con- ditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. net. forum? d= XHTmYb
[16] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:23/0.06825, 2023.
[17] Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah Smith, and Yejin Choi. Dexperts: Decoding-ting time controlled text generation with experts and anti-experts. arXiv preprint arXiv:2105.03022 2021.
[18] Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXhv:2210.150990 2022.
[19] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. ArXiv preprint, abs/2 10. 14168, 2021.
[20] Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. Did aristotle use laptop? question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9, 2021.
[21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. ArXiv preprint, abs/2107.0333. 2021.
[22] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. ArXiv preprint, abs/2 107.0733,, 2021.
[23] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[24] Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James Glass, and Pengcheng He. Dola: Decoding by contrasting layers improves factuality large language models. arXiv preprint arXiv:2309.0330 2023.
[25] Sean O' Brien and Mike Lewis. Contrastive decoding improves reasoning in large language models. arXiv preprint ar 17 2023.
[26] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V Le. Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023.
[27] Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujuu Yang, and Wai Lam. A thorough examination of decoding methods in the era of 1lms. arXiv preprint arXiv:2402.06925. 2024.
[28] Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint orXiv:2401.0295.1 2024.
[29] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput. 3(1):79-87, 1991. doi: 00.116/NEECO.1911.33 1.799. URL https: /doi. org/ / /0/ 11..0.....eeee
1991 3.1.79

---

[30] Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08900, 2022.
[31] Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruming and skipping for mixture-of- experts large language models. arXiv preprint
14800, 2024.
[32] Dongyang Fan, Bettina Messmer, and Martin Jaggi. Towards empirical understanding of moe design choices. arXiv preprint arXiv:2402. 1.000..3300 13089, 2024.
[33] Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. Director: Generator- classifiers for supervised language modeling. arXiv preprint urXiv:2206.07694 2022.
[34] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances Neural Information Processing Systems, 53:3008-3021 2020.
[35] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint 27X1v2220.05862.
2022.
[36] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[37] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05300, 2023.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[39] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18990-18998, 2024.
[40] Eric Mitchell, Rafael Rafailov, Archit Sharma, Chelsea Finn, and Christopher D Manning, An emulator for fine-tuning large language models using small language models. arXiv preprint vrXiv:2310.12962 2023.
[41] Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, and Noah A Smith. Tuning language models by proxy. arXiv preprint
2024.
[42] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint w0Xi::2009.33300 2020.
12

---

Table 5: Average KLD between (x and different distribution across three token sets using the GSM8K dataset. Specifically, we compare Prop (x (ee)) with p(zg) generated by Mixtral 8x7B with rank-k routing, Mixtral 8x7B with random-1 routing and Mistral-7B, respectively. "I" and "I" the percentage increase and decrease relative to the "All" token set. The values in the table are scaled by 10a.
Mixtral 8x7B
Token Set
rank-k
Mistral-7B
random-1
2
All
0.17
5.05
10.21
12.81
15.80
17.78
19.47
25.36
10.36
0.25
0.13
6.62
12.16
14.60
17.52
19.19
20.50
25.70
12,21
0.23
Expression
123.24%
131.13%
119.08%
113.97%
110.89%
17.92%
15.32%
11.32%
117.89%
47.70%
0.20
3.40
6.84
8.38
11.06
13.09
15.40
21.03
7.22
0.28
Stopword
124.04%
132.07%
133.0455
134.60%
430.00%
128.37%
.20.87%
L17.09%
430.25%
112.53%
Appendix
A Quantitative Study of Kullback-Leibler Divergence
A.1 KLD Supplement for Section 2.2
In Section 2.2 Figure 3 qualitaativel illustrates that reasoning ability gap among different expert routing (i.e., 2 and rank-k routing). To support this, we also conduct a quantitative study.
Using the questions and ground-truth answers from GSM8K train next token in a teacher-forcing approach with Mixtral 8x7B. Then, we calculate the average KI.D between the p(xx(( produced by Mixtral 8x7b with top-2 routing strategy and those generated with different rank-k routing strategies. Specifically, the average KLD is calculated across three sets of tokens:
input, obtain the the
(1) "AIl:: This set includes tokens in ground-truth answers;
(2) "Expression": This set comprises tokens from mathematical expressions in ground-truth answers. The generation of these tokens poses reasoning challenge for MoE models. We use regular expressions to extract the mathematical expressions within ground-truth answers.
(3) "Stopword": This contains tokens from stopwords, which serves representative proxy for function words. We utilize the NLTK stopwords lis()
The results are presented in Table 5 The results further support the three findings in Section 2.2 for values ranging from 2 to 8:
Prank-k ( (x:et) with different & values exhibits notable average KLD with Prop-2 ae (z))). As l increases from 2 to 8, the average KLD also increases accordingly. This finding suggests the overall next-token prediction discrepancy between top-2 and rank-k routing.
Finding 1:
Finding 2: For each rank-k strategy, apparent average KL.D is observed when generating mathemati- cal expressions (i.e., "Expression" token set). This indicates the notable differences between top-2 and rank-k routing in generating tokens for reasoning.
Finding For each rank-k strategy, the average KLD between (zop-2(zr|Ixc (x)et) and Prank-k (2y (+) is relatively smaller when generating stopword tokens (i.e., "Stopword" token set) compared to generating mathematical expression tokens. This suggests that such predictions pose fewer challenges for rank-k routing.
A.2 Further Analysis on Kullback-Leibler Divergence
We also calculate the KLD between 2 and Pramdom- (zre (Mtt)aaa (x in Table and present further analysis on Kullback-Leibler Divergence:
https: Sattps://wie mltk. org/
13

---

Table 6: The proportion of experts that are activated by rank-k routing during weak activation but not activated by top-2 routing in strong activation on GSM8K with Mixtral 8x7B. Unchosen experts refer to the experts not selected using default top-2 routing.
rank-k unchosen expert ratio (%) 2.81
2
3
4
6
7
8
46.21
72.62
80.54
84.61
87.79
90.44
90.96
It is observed that the KLD between 2 (<<)) and ( is relatively small for the "Stopword" token set. This indicates that Mixtral 8x7B with rank-2 routing exhibit basic stopword generation capability similar Mixtral 8x7B with top-2 routing. However, for the "Expression" token set, the KLD increases notably compared to that of the "All" token set (i.e., it increases by 31.13% These observations suggest that when shifting routing strategies from top-2 routing rank-2 routing, the reasoning capability of Mixtral 8x7B decreases more than basic generation capability.
As suggested by prior works [8 25 this apparent reasoning ability gap be leveraged to better amplify the reasoning strength of Mixtral 8x7B with top-2 routing. Thus, in our main experiments, report results with fixed rank-2 for the weak activation. The same observation also applies to the weak activations rank-3, rank-4, and random- 1, albeit with varying degrees of significance. Empirically, results in Section 3.3 and 4.1 also illustrate that contrast with rank-2 routing yields generally better improvements.
For Mistral 7B, the average KLD between its next-token distribution and that of Mixtral 7x8B with top-2 routing across three token sets is quite small, indicating that their overall distributions is very similar. This similarity makes Mistral 7B not an ideal weak model to contrast.
B Quantitative Study of SCMoE's Unchosen Experts' Utilization.
As mentioned in Section unchosen experts refer to the experts not selected using default (e.g. top-2) routing. To further evaluate the utilization ratio of unchosen experts in SCMoE, we calculate the proportion of experts that are activated by rank-k routing during weak activation but not activated by top-2 routing in strong activation. Specifically, we take quantitative study of SCMoE's unchosen experts' utilization on GSM8K with Mixtral 8x7B as detailed in Table 6 In SCMoE, the activation proportion of unchosen experts for rank-2 routing on GSM8K is 46.2 21% % and for rank-3 routing on GSM8K is 72.62%, indicating that unchosen experts can contribute to MoE models.
C Hyperparamters Setting for DeepSeekMoE-168
Here, we detail the hyperparameter setting of each baselines for DeepSeekMoE-161 7 It is important to note that contrastive decoding needs a suitable model to serve as an amateur. However, DeepSeekMoE-1on does not have smaller model with the same vocabulary, so DeepSeekMoE does not have a contrast decoding baseline. For other baselines, we list the details below:
Greedy. Greedy does not have hyperparameters to set.
Dynamic Routing. The range of the dynamic threshold is [0.2, 0.3, 0.4, 0.5, 0.6). Ensemble Routing,
The number of activated experts for inference ranges from to 8.
Contrastive Search. The penalty degree is [0.3, 0.4, 0.5, 0.6].
DoLa. For DoLa, due to DeepSeekMoF--IM 16B having 28 layers, test two sets of layers: even- numbered layers from [0, 14) and from [14, 28) respectively.
SCMoE DeepSeekMoE- 16B defaults to taking top-6 routing. Therefore, when implementing SCMOE, we choose top-6 routing as strong activation and top-k routing k E [1, as weak activation. For the penalty strength B, we also search from [0.1, 0.3, 0.5, 0.,,0,09
D Detailed Results of Different Hyperparamters Setting for Each Method
There is one fixed value for the hyperparameter a = 0.1 in Equation T that generalizes across various domains. To provide some elarity, when a set closer I, the contrastive process activates fewer vocabulary for strong activation, resulting in minimal changes after the self-contrast. Conversely,
14

---

setting a closer to O allows more vocabulary tokens to be considered in the self-contrast process, leading sigmificant changes and potentially introducing more noisy information. A suitable should strike a balance between including ideal tokens, which can lead to accurate results in the contrastive vocabulary, and avoiding the introduction of excessive noise from an overly large vocabulary. Previous work [18] masking vocabulary based on suggests that = 0.1 is quite robust and generalizes well across various domains. This guides our choice this setting.
Moreover, we report the performance of each decoding method in Tables method-specific hyperparameter settings
Figure under
9 10
Table 7: Details for Table |
Experimental results GSM8K, StrategyQA, MBPP and HumanEval with Mixtral 8x7B. The performance of each method with different hyperparameters.
Method
Hyper
GSM8K
Strategy
MBPP
HumanEval
Greedy
61.79
72.83
46.20
33.54
Routing-hased
0.2
44.66
65.35
41.20
26.22
0.3
49.20
68.64
39.80
32.93
Dynamic Routing
0.4
54.13
72.27
44.20
34.76
59.82
74.41
46.20
38.41
0.6
61.11
74.19
47.80
34.15
45.19
64.87
38.60
26.83
2
61.79
72.83
46.20
33.54
63.84
73.45
44.20
34.15
62.93
74.37
46.20
37.20
Ensemble Routing
62.02
73.53
44.80
34.15
6
59.14
73.23
44.00
29.88
7
57.32
72.31
43.80
29.27
8
57.70
72.18
42.40
31.71
Search-bused
0.3
60.42
74.06
46.20
36.59
0.4
60.58
74.02
46.20
36.59
Contrastive Search
0.5
60.96
74.80
41.00
34.76
0.6
59.74
74.85
39.20
21.95
[0, 16)
49.96
71.04
33.00
12.80
DoLa
[16, 32)
36.54
65.22
21.60
6.10
61.03
74.15
45.20
34.76
0.3
62.24
74.45
45.20
35.98
Contrastive Decoding
0.5
61.03
73.58
44.40
34.76
59.97
74.06
43.20
34.15
0.9
60.05
73.97
41.40
31.10
62.62
73.93
48.80
39,02
0.3
65.96
75.28
47.40
39.63
SCMoE
0.5
66.94
76.29
45.00
41.46
0.7
64.37
76.16
42.60
39.63
0.9
64.29
75.59
41.60
38.41
15

---

Table Details for Figure 4
Experimental results of different weak activations with Mixtral 8x7B.
We set the strong activation with top-2 routing in SCMoE.
rank-k
Task
random
2
4
5
6
7
0.1
60.88
62.62
61.87
63.08
63.38
62.09
63.76
63.38
63.38
0.3
62.24
65.96
65.20
65.20
64.82
65.50
65.50
64.29
64.74
GSM8K
0.5
63.31
66.94
67.02
66.79
64.29
65.35
66.03
62.02
64.97
63.91
64.37
66.03
64.14
64.37
64.44
66.26
63.15
65.13
0.9
63.53
64.29
64.97
64.82
64.44
64.37
64.75
61.94
63.84
73.80
73.93
73.36
73.88
74.02
74.19
72.92
73.80
74.58
0.3
74.32
75.28
73.01
75.15
73.53
74.19
73.18
72.79
74.62
StrategyQA AA
0.5
74.93
76.29
73.40
74.23
74.54
74.10
74.63
72.88
75.55
0.7
75,81
76.16
72.35
73.14
74.32
74.98
73.40
72.66
75.24
75.55
75.59
73.23
74.06
75,28
73.14
72.75
73.14
75.11
0.1
44.40
48.80
47.60
46.80
45.40
44.00
43,40
43.80
46.40
0.3
45.40
47.40
46.40
46.40
45.20
42.40
43.20
41.80
45.00
MBPP
0.5
44.00
45.00
45.40
43.80
41.80
38.60
38.80
40.80
44.20
0.7
43,40
42.60
40.60
43.60
40.60
38.00
36.60
39.00
41.80
0.9
43.00
41.60
39.60
39.60
39.40
38.80
35.20
39.60
37.00
0.1
37.20
39.02
39.63
38.41
40.85
35.98
36.59
35.98
38.41
0.3
37.20
39.63
39.02
37.80
39.02
35.98
33.54
38.41
37.80
HumanEval
37.80
41.46
37.80
35.98
34.76
32.93
34.15
33.54
37.20
0.7
34.76
39.63
33.54
31.71
28.05
31.10
32.32
34.15
33.54
0.9
32.93
38.41
29.27
32.32
26.22
29.27
31.10
32.93
28.66
Table 9: Details for Table 2 StrategyQA, MBPP and HumanEval with Mixtral 8x7B. We set the weak activation with rank-2 routing.
Experimental results of different strong activations on GSM8K,
Task
top-k
0.1
0.3
0.5
0.7
0.9
GSMBK
3
63.76
68.92
67.70
66.11
66.41
StrategyQA
74.72
75.50
76.42
76.33
76.38
MBPP
48.00
50.60
49,00
45.40
43.40
Human
4
40.24
39.02
39.63
39.02
41.46
16

---

Table 10: Details for Table 4 Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with DeepSeekMoE-168 The performance of each method with different hyperparameters. In SCMoE, "A/B" refers to top-k and 8, respectively.
Method
Hyper
GSM8K
StrategyQA
MBPP
HumanEval
Greedy
18.95
60.41
35.20
26.83
Routing-hased
0.2
11.60
56.47
29.40
19.51
0.3
16.83
59.36
32.60
22.56
Dynamic Routing
0,4
18.12
60.24
33.80
23.17
0.5
19.26
60.63
36.00
24.39
0.6
19.71
59.97
34.80
25.00
1
4.32
$1.57
20.00
15.24
2
10.92
55.69
30.00
20.12
3
15.47
58.49
31.40
23.17
4
16.98
59.76
33.00
22.56
Ensemble Routing
5
17.82
58.88
35.20
25.00
6
18.95
60.41
35.20
26.83
7
19.71
59.06
34.40
26.21
8
19.41
58.84
34.00
26.83
Search-
0.3
18.95
60.67
33.40
25.00
0.4
19.79
61.77
33.20
24.39
Contrastive Search
0.5
19.94
61.59
33.20
23.17
0.6
18.42
61.42
33.20
21.95
[0, 14)
18.27
61.72
36.00
22.56
DoLa
[14, 28)
10.46
56.17
24.60
15.24
(1, 0.1)
19.86
61.90
35.40
26.83
(1, 0.3)
19.56
62.64
36.60
26.83
(1, 0.5)
20.55
62.99
37.20
23.78
0.,
19.48
62.16
35.60
22.56
(.,)
19.11
61.11
34.80
20.73
(2, 0.1)
18.73
61.29
33.80
26.83
(2, 0.3)
19.41
60.54
34.40
27.44
SCMoE
0.5)
19.71
59.84
36.40
25.61
(2, 0.7)
20.62
60.76
35.20
25.61
(2, 0.9)
18.88
60.32
33,80
24.39
(3, 0.1)
19.56
60.85
34.80
27.44
(3, 0.3)
19.11
60.63
34.60
27.44
0.5)
18.88
60.98
35.20
28.05
(3, 0.7)
20.77
60.19
36.00
27.44
(3, 0.9)
20.24
61.20
36.20
26.22
17

---

E Failed Examples of DoLa on Mixtral 7x8B
The performance of DoLa on Mixtral 8x7B is quite poor. Through case study, we find that DoLa cannot terminate its generation appropriately and may exhibit degeneration, as shown Table II
Table 11: Failed examples of DoLa.
GSM8K
Input:
Tom plants 10 trees a year. Every year he also chops down 2 trees a year. He starts with 50 trees. After 10 years 30% of the trees die. How many trees does he have left?
Output:
10 is 8, and 50 - (10 2) is 30, and 30 10 is 20, and 20 10 10, and 10 10 0, and 30% 0 O, and
+ is 0, and 10 is 10, and 10 10 0, and 10 10 0, and 10 10 is 0, and 10- 10 is 0, and 10 - 10 is 0, and
10- 10 is 0, and 10 - 10 is 0, and 10 10 is 0, and 10- 10 is 0, and 10 10 0, and 10 10 is 0, and 10 10 is O. 0. and 10 10 0, and 10- 10 0, and 10 10
and 10 10 is and 10 10 is 0, and 10-
and 10 10 is 0, and 10 is 0, and 10 is 0, and 10 is 0, and 10 10
0, and 10
is 0, and 10
10
10 0, and 10 10 is 0, and 10 is 0, and 10 10 is 0, and 10 is 0, and 10 10 is 0, and 10 10 is 0, and
10 10 is 0, and 10 10 is 0, and 10
MBPP
Input:
You expert Python programmer, and here is your task: Write python function count number of substrings with sum of digits equal to their length. Your code should pass these tests:
assert count_ Suuutttsssrriigs(" 112112"666 == assert count Substrings""1111 =
assert count Substringss"1101111
Output:
import re def count, Substrings(s.D)
00t10000000001111110100001111110
11911111100001111000000001001110
HUMANEVAL
Input:
from typing import List
def concatenatc(string:: List(str)) -> str:
Concatenate list of strings into single string
> concatenate(()
concatenate(("a" "b","e"" "c"])
"abe"
Output:
#
2.
5.
6.
7.
8.
9.
10.
11
12.
13.
14.
15.
16.
17.
18.
19.
20.
21
22.
23.
24
25.
26
27
28.
29.
30.
31.
32.
34
36.
37.
38
39
40.
41
42
43.
44
45.
47.
48
49
51
52
53
54
56
58
59.
60.
61.
62.
63.
64
65.
66
67.
68.
69.
70.
71.
72.
73.
74.
75.
76.
77.
78
79.
80
81
82
83
84
85.
86
87,
88,
89.
90.
91.
92.
93
94
95
96
97.
98
99.
100.
101.
102.
103.
104.
105.
106.
107.
108.
109
110.
111.
112.
113.
114.
115.
116.
117
118.
119.
120
121.
122.
123.
18

---

Scope of SCMoE's Effectiveness
The strength SCMoE lies in its ability handle tasks requiring intricate reasoning processes by leveraging both strong and weak activations, which benefits in scenarios demanding reasoning capability for next-token prediction. (verbalized) reasoning paths, which SCMoE is dedicated to helping. Therefore, SCMoE, similar to other generation decoding strategies like contrastive search 23 and contrastive decoding [18), may not exhibit distinct advantages.
contrast, benchmarks like MMLU 42 do not have explicit
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We ensure that all claims made in the paper are included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the conclusion section, we have discussed the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide corresponding explanations for the formulas used in our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We use open-source models and publicly available datasets, so our results are reliable and reproducible.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: We use open-source models and publicly available datasets, and we also include our code the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https::/niips..
public/ publicc//uudes httpc///guidees/oeuuuuuml CodeSuummeeeieeiiicooeo
as sswonooooccycy for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nipss.cc/publicc c/ des CodeSubmn bub ss onPol icy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the detailed setting in the Experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The experiments results support the main claims of our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have explained the GPU resources we used to measure latency.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our code meets NeurIPS code of ethics' requirements.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of our work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We make appropriate references to the open-source models and publicly available datasets we used in the text.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswis wwwoosocittcceeeooo com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "dc01zp1j7R", "reviewer": "Reviewer_KVjj", "summary": "The paper explores to leverage the contrastive information existing between different routing strategies of the MoE model to facilitate a better token decoding during inference in a training-free fashion. The paper is built upon two interesting observations: \n(1) increasing the number of activated experts does not necessarily improve and can even degrade the output quality (which increases and then drops might be due to noise/irrelevancy of other experts).  \n(2) output distributions from an MoE model using different routing strategies substantially differ (which is obvious due to pre-training setting with a routing strategy).\n\nMore specifically, for ScMoE the next-token probabilities are determined by contrasting the outputs from strong and weak activation using the same MoE mode. The authors conducted experiments using Mixtral on GSM8K, StrategyQA, MBPP and HumanEval and illustrate some noticeable performance benefits on GSM8K.", "questions": "See above.", "limitations": "See above.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper brings several interesting strength to community:\n\n1. The U-shape performance and top-k routing policy is indeed interesting and changing k wrt. dataset for best performance can encourage MoE designs with adaptive inference based on task difficulty. \n2. The authors proposal to estimate next-token probabilities by contrasting the outputs from strong and weak activation using the same MoE model is interesting and I am surprised to see the impressive performance gain (>5 points) on GSM8K.\n3. The authors presents some findings (ln. 111-120 and Appendix) as well as a comprehensive ablation experiments which make the paper well-grounded and comprehensive.\n4. The author additionally include some latency related issues which is the major bottleneck of their proposed method.", "weaknesses": "While the paper is well-written and comprehensive, I have following comments/questions/concerns related to the paper:\n\n1. One question- the author submitted numbers for Mixtral default routing strategy seems to be far from the numbers reported by the Mistral authors (Table 2 from https://arxiv.org/pdf/2401.04088). What could be the possible reason for that (curiosity and not a negative point)? \n2. The author must perform the memory bottleneck for storing additional activations for their purpose of using two multiple routing strategy in SCMoE. I believe the memory bottleneck (as well as latency) of decoding will be substantially high while dealing with long-context tasks. \n3. I was wondering if the contrastive behavior can be enforced explicitly by using a small finetuning data and a contrastive loss? What are the authors thought about this?\n4. What is the consistency of the findings KLD for reasoning tasks across some other math reasoning tasks apart from GSM8K? \n5. Another major weakness of the work is the tuning of the hyperparameter `\\beta` for every dataset? Can authors find some discover a single `\\beta` which  should work well enough for say a task category (math reasoning/commonsense).\n\nI still find the work interesting and willing to increase the score with fair discussion during rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and willingness to take further discussions with us. We appreciate your praise of the insights (S1, S2) and the detailed experimental setup (S3, S4) in our work. We provide point-to-point response to address your concerns as follows:\n\n> W1: One question- the author submitted numbers for Mixtral default routing strategy seems to be far from the numbers reported by the Mistral authors (Table 2 from [1]). What could be the possible reason for that (curiosity and not a negative point)?\n\nR1:\nWe would like to clarify that the performance reported in Table 2 of the Mixtral [1] are using the self-consistency with major@8 (as detailed in Section 3 of their experimental setup). For direct results without using self-consistency, you can refer to Table 3 in Mixtral [1]. We hope this addresses your concern.\n\n[1] Mixtral of Experts \n\n\n> W2: The author must perform the memory bottleneck for storing additional activations for their purpose of using two multiple routing strategy in SCMoE. I believe the memory bottleneck (as well as latency) of decoding will be substantially high while dealing with long-context tasks.  \n\nR2: \nIn practice, the memory overhead and decoding latency introduced by SCMoE are within acceptable bounds. The primary cause of the additional memory usage is the simultaneous employment of both strong activation (e.g. , top-2 routing) and weak activation (e.g. , rank-2 routing) in an MoE model. Specifically, this additional memory is required for storing additional KV cache, which scales linearly with the sequence length. For a sequence length of 2048, the additional memory amounts to approximately 256.0MB using BF16. Given the model size of nearly 86GB, this represents a marginal increase of about 0.3%. As for decoding latency, it consistently remains at x1.30, and this ratio does not increase with longer lengths. This minor increment underscores the feasibility of our approach in practical deployments. \n\n> W3: I was wondering if the contrastive behavior can be enforced explicitly by using a small finetuning data and a contrastive loss? What are the authors thought about this?\n\nR3: \nThis is a very interesting point. Enforcing contrastive behavior explicitly by using a small finetuning data and a contrastive loss is indeed a promising direction of future work. This is a straightforward approach but would greatly increase training cost. We consider that maybe applying it at the late stage of training using a small amount of fine-tuning data is definitely worth trying.\n\nNevertheless, the key insight we want to share is that we can more effectively utilize the MoE's unchosen experts in a simple yet effective training-free approach since they are already trained and loaded into memory. In terms of further more effective utilization of these experts combined with additional training methods, due to limited space, we leave for future work.. \n\n> W4: What is the consistency of the findings KLD for reasoning tasks across some other math reasoning tasks apart from GSM8K? \n\nR4:  Based on your suggestion, we conduct experiments on a more challenging math reasoning dataset MATH [3]. Due to computational resource limitations, we randomly sample 500 examples from the MATH test set for our experiments. We perform a quantitative study of KLD on the MATH dataset using the same settings as in Appendix A. The results are shown in the table below:\n\n|  Token Set |  rank-k |         |         |         |         |         |        |        |\n|:----------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:------:|:------:|\n|            |    1    |    2    |    3    |    4    |    5    |    6    |    7   |    8   |\n|     All    |   0.31  |   7.81  |  10.43  |  16.13  |  18.84  |  21.96  |  23.84 |  30.92 |\n| Expression |   0.26  |   9.27  |  11.68  |  17.17  |  19.83  |  22.84  |  25.20 |  31.73 |\n|            | -16.69% | +18.77% | +11.99% |  +6.47% |  +5.25% |  +3.99% | +5.74% | +2.60% |\n|  Stopword  |   0.38  |   5.82  |   8.42  |  13.11  |  15.98  |  19.54  |  21.49 |  29.96 |\n|            | +22.50% | -25.42% | -19.28% | -18.70% | -15.19% | -11.03% | -9.85% | -3.10% |\n\nThe results are consistent with our observations on the GSM8K dataset (Lines 446-455), reaffirming that MoE models employing top-2 and rank-k routing strategies exhibit distinct generation behaviors. This finding further supports the analytical foundation of SCMoE in utilizing contrastive information effectively.\nWe also compare the results of SCMoE and other methods on the MATH dataset. The experimental results are shown in the table below. The results indicate that SCMoE continues to be effective even on more challenging math reasoning tasks.\n\n|   Dataset  | Greedy | Dynamic | Ensemble |  CS  | DoLa |  CD  | SCMoE |\n|:----------:|:------:|:-------:|:--------:|:----:|:----:|:----:|:-----:|\n| Math (500) |  20.2  |   21.0  |   21.2   | 21.4 | 16.4 | 20.6 |  22.4 |\n\n[3] Measuring Mathematical Problem Solving With the MATH Dataset \n\n> W5: Another major weakness of the work is the tuning of the hyperparameter \\beta for every dataset? Can authors find some discover a single \\beta which should work well enough for say a task category (math reasoning/commonsense).\n\nR5: \nThank you for your question. In fact, the performance of different $\\beta$ can be seen in Table 7 of Appendix D, where $\\beta$ = 0.5 already achieves the best results on GSM8K, StrategyQA, and HumanEval (except for MBPP). Therefore, $\\beta$ = 0.5 may be sufficient for most scenarios and further tuning could achieve optimal performance. Meanwhile, as shown in Table 7, other baseline methods also face the same issue. However, for SCMoE, we believe automatically adjusting the $\\beta$ based per each instance or token can be a direction worth exploring, which we leave for future work. \n\nWe appreciate your positive feedback and willingness to take further discussions and increase the rating. Hope that our response can address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohg3hs5poO", "reviewer": "Reviewer_bydP", "summary": "This paper introduces SCMoE, a decoding time algorithm which can be applied off the shelf to boost MoE models' performance by contrasting the chosen experts in strong and weak activations. Experiment results show that this algorithm has an empirical advantage over baselines methods in coding, commonsense knowledge, and math benchmarks.", "questions": "See above", "limitations": "The authors briefly mentioned limitations of their paper in the conclusion section but they could further discuss other limitations including limitation of datasets evaluated etc.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality: \nThis paper proposes an original method in MoE decoding. The method is straightforward to understand and easy to deploy. \n\nQuality: \nThis paper is of high quality. The paper is well written with good demonstrations and clear math. The proposed method shows better performance across all presented benchmarks compared to baseline methods. \n\nClarity: \nAs discussed in the last point, this paper is very clear in idea demonstration and methodology illustration. Experiment results are also well organized. \n\nSignificance: \nThis paper is significant in contributing to the after-training development of MoE models.", "weaknesses": "1.\tCommonsense reasoning in StrategyQA, which is a multi-hop reasoning dataset about world knowledge, does not seem to be so strongly related to coding and math which requires formal reasoning compared to formal logic (see [1], which mainly focuses on logic, algorithm, and math as reasoning tasks). Do you anticipate performance boost in logic reasoning tasks? \n\n2.\tRelatedly to the last question, do you think the proposed algorithm will also help general world knowledge reasoning without implicit complex reasoning chains such as MMLU? \n\n\t\n3.\tIs it possible to find the ideal strong activation in real-life workflows only given the user query?\n\n[1] Zhu, K., Chen, J., Wang, J., Gong, N. Z., Yang, D., & Xie, X. (2023). Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback! We appreciate your praise of the Originality, Quality, Clarity and Significance of our work, which is a great encouragement for us. We would like to address your concerns below:\n\n> W1: Commonsense reasoning in StrategyQA, which is a multi-hop reasoning dataset about world knowledge, does not seem to be so strongly related to coding and math which requires formal reasoning compared to formal logic (see [1], which mainly focuses on logic, algorithm, and math as reasoning tasks). Do you anticipate performance boost in logic reasoning tasks? \n\nR1: \nWe conduct experiments on three logical reasoning tasks as mentioned in [1]: abductive logic, boolean logic, and deductive logic. Since Mixtral achieves nearly 100% accuracy in boolean logic, we only present the results for abductive logic and deductive logic below.\n\n|  Dataset  | Greedy | Dynamic | Ensemble |  CS  | DoLa |  CD  | SCMoE |\n|:---------:|:------:|:-------:|:--------:|:----:|:----:|:----:|:-----:|\n| Abductive |  68.2  |   77.0  |   70.2   | 69.2 | 74.0 | 81.2 |  87.6 |\n| Deductive |  78.4  |   80.8  |   78.4   | 77.4 | 84.6 | 84.6 |  86.4 |\n\nFrom these results, it is evident that SCMoE provides significant performance improvements in abductive and deductive logic. This finding verifies that SCMoE can also bring performance boost in logical reasoning tasks.\n\n[1] Dyval: Dynamic evaluation of large language models for reasoning tasks. \n\n> W2: Relatedly to the last question, do you think the proposed algorithm will also help general world knowledge reasoning without implicit complex reasoning chains such as MMLU? \n\nR2: \nThe strength of SCMoE lies in its ability to handle tasks requiring intricate reasoning processes by leveraging both strong and weak activations, which benefits in scenarios demanding reasoning capability for next-token prediction (Lines 111-124). In contrast, benchmarks like MMLU do not have explicit (verbalized) reasoning paths, which SCMoE is dedicated to helping. Therefore, SCMoE, similar to other generation decoding strategies like Contrastive Search (CS) and Contrastive Decoding (CD), may not exhibit distinct advantages. We will further discuss and clarify the applicability of SCMoE in the next version.\n\n> W3: Is it possible to find the ideal strong activation in real-life workflows only given the user query?\n\nR3: \nThe topic of finding the ideal strong activation in real-life workflows based solely on the user query is indeed a topic worthy of further exploration in future research.\nPrevious work [2] offers some insights regarding this problem. It suggests that \"Harder tasks need more experts\" and proposes a routing algorithm that can dynamically adjust the number of experts activated based on the difficulty of the problem.\nWhile for our method SCMoE, it can achieve better performance than the ideal strong activation as shown in Figure 1, reducing the reliance on searching for ideal strong activation. The performance of SCMoE can be further boosted when we can obtain the ideal strong activation as shown in Table 2, this indicates that our method can be further enhanced with other ideal activation searching method, and we leave that for future research.\n\n[2] Harder tasks need more experts: Dynamic routing in MoE models.\n\nThanks again for your valuable feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpO3iIKm04", "reviewer": "Reviewer_cUsK", "summary": "This paper proposes a novel approach called Self-Contrast Mixture-of-Experts (SCMoE) to improve the utilization and performance of Mixture-of-Experts (MoE) models. The key contributions are:\n\n1. Exploratory studies showing that increasing the number of activated experts in MoE models does not always improve output quality and that different routing strategies lead to substantially different output distributions.\n\n2. The SCMoE method, which leverages unchosen experts in a self-contrast manner during inference. It determines next-token probabilities by contrasting outputs from strong and weak activation using the same MoE model.\n\n3. Experimental results demonstrating that SCMoE consistently enhances the reasoning capabilities of Mixtral 8x7B across various domains.\n\n4. Combining SCMoE with self-consistency yields further gains.\n\n5. The proposed SCMoE method is conceptually simple, computationally lightweight, and incurs minimal latency compared to greedy decoding.", "questions": "1. See Weakness 2.\n\n2. I am curious why Mixtral has decreased performance when averaging over more expert models. Could the authors give some explanations? It is because of something specific to pretraining methodology of Mixtral/DeepSeekMoE models?", "limitations": "N/A", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-motivated through exploratory studies, showing that simply increasing the number of activated experts in MoE LLMs can harm performance and that expert models tend to inhibit rather than strengthen each other.\n\n2. The proposed SCMoE method is simple and intuitive, using the difference between the logits of stronger and weaker models. This approach incurs minimal performance overhead and is straightforward to implement.\n\n3. Despite its simplicity, SCMoE consistently yields performance gains on evaluation benchmarks, showcasing its effectiveness.\n\n4. The selected set of evaluation tasks, while not extensive, includes representative tasks from the most important domains of LLM applications, making the experimental results meaningful and easy to interpret. The study also examines multiple mainstream MoE LLMs, such as Mixtral and DeepSeekMoE.\n\n5. The paper is well-written and well-presented.", "weaknesses": "1. The idea of SCMoE is somewhat similar to the cited paper \"Contrastive decoding: Open-ended text generation as optimization (Li et. al. [20]),\" although I understand that SCMoE is better adapted to MoE LLMs and has lots of novelties in other aspects. The authors could provide more discussion in the related work section about the similarities and differences between the two approaches.\n\n2. The paper lacks clarity on how certain hyperparameters are chosen. For example, in Section 3.2, the authors state that \"for the weak activation, we only consider the rank-k routing with k=2\" but do not provide an explanation for why k cannot be 3, 4, or 5. It would be helpful to know if choosing k=2 is motivated by faster inference times, as the top 2 models will be used regardless.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of the novelty, simplicity and effectiveness of our work. This is a great honor for us. We aim to address your concerns below.\n\n> W1: The idea of SCMoE is somewhat similar to the cited paper \"Contrastive decoding: Open-ended text generation as optimization (Li et. al. [20]),\" although I understand that SCMoE is better adapted to MoE LLMs and has lots of novelties in other aspects. The authors could provide more discussion in the related work section about the similarities and differences between the two approaches.\n\nR1: \nThank you for your insightful feedback. The motivation behind SCMoE is to investigate the utilization of unchosen experts. Our final approach leverages the unchosen experts by contrasting predictions of strong activation with weak activation. We acknowledge that there are some existing works sharing the similar spirit in using a contrastive objective function. We have already discussed them in the Related Work section (\"Contrast Language Modeling\", Lines 279-299). To further address your concern, we give more detailed explanations regarding the similarities and differences between our work and [1] as follows:\n\n*Similarity:*\n- Both SCMoE and [1] belong to the category of inference-time optimization with contrast. \n- Both approaches contrast two different distributions to obtain a better distribution to enhance model generation performance.\n\n*Difference:*\n- To obtain two different distributions, [1] requires two different models—an expert model and an amateur model—making the selection of a suitable expert model and amateur model combination a challenge, as it involves ensuring vocabulary consistency, and various expert-amateur model parameter scale combinations.\n- In contrast, SCMoE takes advantage of the architecture characteristic of MoE models by employing different routing strategies to directly obtain two different distributions. Therefore, SCMoE essentially utilizes only one single model, supports dynamic distribution combinations, and effectively employs the unchosen experts in MoE models.\n\n[1] Contrastive decoding: Open-ended text generation as optimization. \n\n> W2 & Q1: The paper lacks clarity on how certain hyperparameters are chosen. For example, in Section 3.2, the authors state that \"for the weak activation, we only consider the rank-k routing with k=2\" but do not provide an explanation for why k cannot be 3, 4, or 5. It would be helpful to know if choosing k=2 is motivated by faster inference times, as the top 2 models will be used regardless.\n\nR2: \nThank you for your insightful feedback. To clarify, the choice of $k$ in the rank-$k$ routing strategy is guided by the criterion of maintaining consistency in generating general tokens, such as stopwords, while providing a notable contrast for tokens requiring reasoning capabilities.\n\nIn Appendix A (Lines 429-473), we qualitatively illustrate the average KLD between  $p_{\\text{top-2}}(x_{t}|x_{<t})$ and different distributions $p_{\\text{rank-k}}(x_{t}|x_{<t})$. It is observed that the KLD between $p_{\\text{top-2}}(x_{t}|x_{<t})$ and $p_{\\text{rank-2}}(x_{t}|x_{<t})$ is relatively small for the \"Stopword\" token set. This indicates that Mixtral with rank-2 routing exhibits basic stopword generation capability similar to Mixtral with top-2 routing. However, for the \"Expression\" token set, the KLD increases notably compared to that of the \"All\" token set (i.e., it increases by 31.13%). These observations suggest that when shifting routing strategies from top-2 routing to rank-2 routing, the reasoning capability of Mixtral decreases more than basic generation capability.\n\nAs suggested by prior works [1], this apparent reasoning ability gap can be leveraged to better amplify the reasoning strength of Mixtral with top-2 routing. The same observation also applies to the weak activations of rank-3, rank-4, and random-1, albeit with varying degrees of significance.\n\nEmpirically, results in Sections 3.3 and 4.1 illustrate that rank-2 routing generally yields better improvements. Hence, rank-2 routing serves as a practical and versatile configuration applicable across various domains. To consistently validate the effectiveness of SCMoE, we present our findings using a fixed rank-2 for weak activation.\nWhile rank-2 is effective, it may not be optimal for all tasks. Future work could explore how to adaptively set the size of weak activation rank-k for different user queries. \n\n[1] Contrastive decoding: Open-ended text generation as optimization. \n\n> Q2: I am curious why Mixtral has decreased performance when averaging over more expert models. Could the authors give some explanations? It is because of something specific to pretraining methodology of Mixtral/DeepSeekMoE models?\n\nR3:\nYour question is very insightful. We believe this is still an open question, and we explain our understanding as follows. Similar to your hypothesis, we also believe it is due to something specific to the pretraining methodology. In addition to the standard Language Modeling loss, additional losses such as Expert-Level Balance Loss and Device-Level Balance Loss are used to address the MoE models' loading balance issue, ensuring stable training. This load balancing loss in pretraining may enable different experts in MoE models to excel at different domains / tasks but not all. When \"non-specialized\" experts are activated, their contributions may become noise in the final ensemble, diluting the strengths of truly specialized experts. Based on this observation, we consider utilizing these \"non-specialized\" (wrt. unchosen ) experts in a self-contrast manner in order to benefit in scenarios demanding reasoning capability for next-token prediction (Lines 111-126).\n\nThank you for your insightful suggestion! We sincerely appreciate your praise for the novelty of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgsQ3uJZ2n", "reviewer": "Reviewer_Vqkj", "summary": "This paper proposes a training-free strategy that utilizes unchosen experts in a self-contrast manner during inference. It can be seen as a decoding method utilizing divergent information from different routing strategies. This method introduces slightly more latency overhead and improves the performance of various tasks through experimental evaluation.", "questions": "* The extra latency cost is minor, is it because the two-model inference is parallelly implemented? \n* The “+2=5 (...)” in Figure 2.(c) is a little bit confusing to me, could you explain further about it?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper is well put together with clear insights and well-articulated motivation. \n* The problem studied in the paper is well-motivated, it is known that expert selection in MoE is not trivial.\n* The idea is novel, and the illustration of the methodology is easy to follow.", "weaknesses": "* This inference method employs two models, one with top-2 routing and the other with rank-k routing. It produces double memory overhead which is significant, especially for larger models.\n* The choosing of α in Equation 6. seems non-trivial, it would be great to provide some insight on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback. We appreciate that you highlight the motivation, idea and the clarity of our work. We provide detailed responses to address your specific concerns as outlined below.\n\n> W1: This inference method employs two models, one with top-2 routing and the other with rank-k routing. It produces double memory overhead which is significant, especially for larger models.\n\nR1:\nThank you for your question. In fact, SCMoE incurs only minimum memory overhead because it does not use two models. Instead, it performs both weak activation forward passes and strong activation forward passes simultaneously using a single model (As we explained in Figure 2 and Section 2.3). \nSpecifically, this additional memory is due to SCMoE doubling the size of the KV cache caused by additional rank-k routing. This additional memory scales linearly with the sequence length. For instance, for a sequence length of 2048 using BF16, the additional memory amounts to approximately 256.0MB. Given the model size of nearly 86GB, this represents a marginal increase of about 0.3%. This minor increment underscores the feasibility of our approach in practical deployments. \n\n> W2: The choosing of $\\alpha$ in Equation 6. seems non-trivial, it would be great to provide some insight on this.\n\nR2: \nThank you for your insightful feedback. There is one fixed value for the hyperparameter $\\alpha$=0.1 in Equation 6 that generalizes across various domains. To provide some clarity, when $\\alpha$ is set closer to 1, the contrastive process activates fewer vocabulary for strong activation, resulting in minimal changes after the self-contrast. Conversely, setting  $\\alpha$ closer to 0 allows more vocabulary tokens to be considered in the self-contrast process, leading to significant changes and potentially introducing more noisy information.\n\nA suitable $\\alpha$ should strike a balance between including ideal tokens, which can lead to accurate results in the contrastive vocabulary, and avoiding the introduction of excessive noise from an overly large vocabulary. Previous work [1] on masking vocabulary based on $\\alpha$ suggests that \"$\\alpha$ = 0.1 is quite robust and generalizes well across various domains.\"  This guides our choice in this setting.\n\nWe will include these details in the next version. We appreciate your thoughtful feedback and hope this response addresses your concern.\n\n[1] Contrastive decoding: Open-ended text generation as optimization. \n\n\t\t\t\t\n> Q1: The extra latency cost is minor, is it because the two-model inference is parallelly implemented? \n\nR3: \nThank you for your valuable question. In fact, we do not use two separate models for inference, we only use a single MoE model's strong activation and weak activation for self-contrast. \n\nSpecifically, upon receiving a test input, we duplicate it. As a result, we have two identical test inputs. During the forward pass of the MoE model, we apply top-2 routing strategy to one test input and rank-$k$ routing strategy to the other input. This approach allows us to achieve both strong and weak activation within a single forward pass, thereby reducing the extra latency cost.\n\n> Q2: The “+2=5 (...)” in Figure 2.(c) is a little bit confusing to me, could you explain further about it?\n\nR4: Thank you for your valuable question. Figure 2(c) presents an example of how SCMoE operates. The complete question and corresponding answer for this example are depicted in Figure 3. In Figure 2(c), the model is tasked with predicting the next token (represented by the \"?\" symbol with the green background indicating the unknown next token). Utilizing the SCMoE method, the model successfully predicts the next token to be \"+\" (represented by \"+\" with the green background). \nThe notation \"2 = 5 (...)\" following the \"+\" represents part of the answer that has not yet been generated (the complete answer can be found in Figure 3). In fact, it is unnecessary to include \"2 = 5 (...)\" in Figure 2(c). We appreciate your suggestion and will clarify this in the next version.\n\nThank you for your insightful reviews! We sincerely appreciate your kind words regarding the novelty of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple inference strategy that utilizes unchosen experts in a MoE framework. The paper starts with two interesting observations: (1) increasing the number of activated experts does not necessarily improve and can even degrade the output quality; (2) output distributions from an MoE model using different routing strategies substantially differ. These observations led them to propose the Self-Contrast Mixture-of-Experts (SCMoE), which modifies the decoding strategy by contrasting outputs from strong and weak activation using the same MoE model. This method improves the performance of various tasks (GSM8K, StrategyQA, MBPP and HumanEval) of Mixtral 8x7B with a slight latency overhead. Combining SCMoE with self-consistency yields further gains.

All reviewers agree that the paper presents very insightful observations, proposes a simple, off-the-shelf decoding strategy that's massively 
effective. The findings and the method can benefit anyone studying and deploying MoE models immediately.
Decision: Accept (poster)