TC-MoE: Augmenting Mixture of Experts with Ternary Expert Choice
Authors: Shen Yan
Xingyan Bin
Sijun Zhang
Yisen Wang
Zhouchen Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Mixture of Experts
Primary area: foundation or frontier models, including LLMs

Abstract: The Mixture of Experts (MoE) architecture has emerged as a promising solution to reduce computational overhead by selectively activating subsets of model parameters.
The effectiveness of MoE models depends primarily on their routing mechanisms, with the widely adopted Top-K routing scheme used for activating experts.
However, the Top-K scheme has notable limitations,
including unnecessary activations and underutilization of experts.
In this work, 
rather than modifying the routing mechanism as done in previous studies,
we propose the Ternary Choice MoE (TC-MoE),
a novel approach that expands the expert space by applying the ternary set {-1, 0, 1} to each expert.
This expansion allows more efficient and effective expert activations without incurring significant computational costs.
Additionally, 
given the unique characteristics of the expanded expert space,
we introduce a new load balance loss and reward loss to ensure workload balance and achieve a flexible trade-off between effectiveness and efficiency.
Extensive experiments demonstrate that TC-MoE achieves an average improvement of over 1.1% compared with traditional approaches,
while reducing the average number of activated experts by up to 9%.
These results confirm that TC-MoE effectively addresses the inefficiencies of conventional routing schemes,
offering a more efficient and scalable solution for MoE-based large language models.
Code and models are available at https://github.com/stiger1000/TC-MoE.

Published as a conference paper at ICLR 2025
TC-MOE: AUGMENTING MIXTURE OF EXPERTS WITH TERNARY EXPERT CHOICE
Shen Yan', Xingyan Bin", Sijun Zhang', Yisen Wang Zhouchen 445
Center for Data Science, Peking University
SonerddFunndttion-MMe-
ByteDance
State Key Lab of General AI, School of Intelligence Science and Technology. Peking University
*Institute for Artificial Intelligence, Peking University
Pazhou Laboratory (Huangpu), Guangzhou, Guangdong, China
ABSTRACT
The Mixture of Experts (MoE) architecture has emerged as a promising solution to reduce computational overhead by selectively activating subsets of model pa- rameters. The effectiveness of MoE models depends primarily on their routing mechanisms, with the widely adopted Top-K routing scheme used for activating experts. However, the Top-K scheme has notable limitations, including unneces- sary activations and underutilization of experts. In this work, rather than modi- fying the routing mechanism as done in previous studies, we propose the Ternary Choice MoE (TC-MoE), a novel approach that expands the expert space by apply- ing the ternary set (, } to each expert. This expansion allows more efficient and effective expert activations without incurring significant computational costs. Additionally, given the unique characteristics of the expanded expert space, we introduce a new load balance loss and reward loss ensure workload balance and achieve flexible trade-off between effectiveness and efficiency. Extensive experiments demonstrate that TC-MoE achieves an average improvement of over 1.1% compared with traditional approaches, while reducing the average number of activated experts by up to 9%. These results confirm that TC-MoE effectively addresses the inefficiencies of conventional routing schemes, offering more ef- ficient and scalable solution for MoE-based large language models. Code and models are available at attttps: / http://gggtttuu./tg/// thub. com/ st ger r
INTRODUCTION
In recent years, large language models (LLMs) (Brown et al. 2020 Touvron et al. 2023 Achiam al. 2023) have demonstrated impressive performance across a wide range of domains. However, modern LLMs still face inefficiencies because they typically utilize all their parameters for every input token during both training and inference. This leads to substantially increased computational resource requirements the models scale. To address these challenges, researchers have introduced the Mixture of Experts (MoE) architecture (Shazeer et al.) (017).. The MoE architecture facili- tates parameter scaling while maintaining reasonable computational costs. Unlike traditional dense models, MoE models incorporate a routing mechanism that selectively activates specific subsets of parameters for each input token. Recent advancements in MoE models (Jiang al. 2024 Dai al.)
2024 Wu et al. 2024) have paved the way for scaling language models to unprecedented sizes while achieving remarkable performance improvements.
Within the MoE architecture, the routing mechanism plays critical role since it significantly influ- ences both the efficiency and effectiveness of model training. Traditional MoE frameworks, includ- ing GShard (Lepikhin et al.] (021). Switch Transformers (Fedus et al. (2022), and ST-MoE (Zoph) al. 2022), all employ the Top-K routing scheme. This method calculates the routing probabil- ity for each combination of experts and tokens. The K experts with the highest probabilities are activated for each token, with the final output being the weighted sum of their outputs.
"Corresponding authors.

---

Published as a conference paper at ICLR 2025
0.06
Positive contribution
contrixx Negathie contrixttin
200
0.06
150
Contribution
0.34
Dersity
0.02
-60%
-40%
Negative contribution
-0.02
0
0.1
0.2
0.3
0.4
0.5
0.8
0.7
0.8
0.9
-0.016 -0.01
-0.005
0.005
Gate Values
Contribution
(a) Distribution contributions from activated ex- perts. The experts are categorized based on their gate values. This shows that some activations contribute negatively to the performance, indicating unnecessary activations.
(b) Distribution of contributions from experts with low gate values after flipping the sign their out- puts. The results demonstrate that some experts can positively impact performance when their out- put signs flipped.
Figure 1: Analysis of the limitations in the conventional Top-K K routing scheme in model with 2 activated experts out 8. We compute the contribution of each activated expert by measuring the difference in model performance when the activation is masked. More details are in Appendix A
However, recent works (Zhou et al. 2022; Huang et al., 2024) combined with our experiments demonstrate the suboptimal nature of this routing scheme. We identify two key limitations:
Unnecessary Activations:
The Top-K scheme activates fixed number of experts for each token, neglecting the possibility of adaptively choosing the number of activated experts. As shown in Figure Ta some activated experts do not positively contribute to model perfor- mance, revealing redundant computations.
Underutilization of Experts: The constraint of non-negative expert weights prevents full utilization of expert potential. Figure [bb demonstrates that approximately 40% of low- weight experts show positive contributions when their output signs are reversed.
In this work, introduce the Ternary Choice Mixture of Experts (TC-MoE) to address these limitations. Unlike previous studies (Zhou et al.) 2022 Yang et al.) 2024 Huang et al.) 2024) that focus on improving the routing scheme, we explore innovative approach through expert space expansion. Inspired by ternary quantization techniques where weights are constrained to (-1, 0, 1}, we propose create an expanded expert space through ternary multiplication of each original ex- pert. As demonstrated in Figure 2b] assigning ternary choices to each expert generates expanded expert space with unchanged parameter counts of the experts. This expanded expert space empow- ers the router to learn sophisticated strategies during the training process, thereby alleviating the aforementioned limitations without modifying the routing scheme.
However, the expanded expert space exhibits unique characteristics differing from the original expert space. As illustrated Figure 2b] parameter-sharing expert pairs coexist with zero-parameter ex- perts that incur computational costs. These structural differences render traditional load balance loss ineffective. To address this, we propose a redesigned load balance loss for equitable workload dis- tribution. Furthermore, introduce novel reward loss enabling flexible efficiency-efeectiveess trade-offs through our analysis.
We conduct comprehensive evaluations of our method multiple benchmarks. The results demon- strate that our TC-MoE outperforms existing approaches. Compared with the baseline, TC-MoE delivers an average performance gain of 1.1% alongside 9% reduction in the average number of activated experts. Notably, under varying computational budgets, our method maintains consistent performance advantages over dynamic routing alternatives. These findings conclusively demonstrate the effectiveness of TC-MoE in overcoming the limitations of conventional routing schemes.
2

---

Published as a conference paper at ICLR 2025
Expert
Expert E
Expert E;
EST
Ro parntis
(-1,0,13
Export
Expert
Expert
Router
Router
Token A
Token A
(a) Conventional MoE.
(b) Our proposed TC-MoE.
Figure 2: Comparison of the conventional MoE architecture and the proposed TC-MoE. The original expert space contains 3 experts, with the router activating 2 experts per token. By multiplying each expert with the ternary set {-1, 1}, TC-MoE obtains an expanded expert space comprising experts. Notably, 2 of these experts are parameter-free free (we maintain 2 instead of 3 as 2 is sufficient for the router to activate any number from to of these experts). The remaining experts are grouped into 3 parameter-sharing sets.
Our contributions can be summarized as follows:
1. We propose the TC-MoE, novel method to overcome classical routing limitations in MoE architectures by expanding the expert space through multiplying the ternary set, achieving effective activation enhancement without routing scheme modification.
2. Given the unique load balance requirements in our expanded expert space, we redesign the load balance loss. Additionally, we introduce novel reward loss to achieve a flexible trade-off between effectiveness and efficiency.
3. Our experimental results demonstrate that TC-MoE achieves superior performance with fewer activated parameters, confirming significant improvements in both effectiveness and efficiency over the baseline.
RELATED WORK
Mixture of Experts Models. MoE models (Jacobs et al., 1991 Jordan & Jacobs 1994) have been extensively studied in artificial intelligence. The concept of using trainable gating network to determine a sparse combination of experts is pioneered by the Sparsely-
MoE (Shazeer et al, 2017). Since then, numerous studies (Lepikhin al., 2021 Fedus al. 2022 Zoph et al. 2022 Jiang et al. 2024 Dai et al. 2024; Wei et al. 2024) have built upon this framework, demonstrating compelling empirical results by scaling MoE models unprecedented sizes.
Routing Schemes. The MoE architecture relies on a routing module to determine the activation of experts, making the routing scheme critical factor for MoE model performance. Early works (Shazeer et al. 2017; Fedus et al.) 2022) employ the Top-K routing scheme, which calculates the routing probabilities for each expert and activates the Top-K experts with the highest probabilities. Recent studies have focused on improving routing schemes. Zhou et al. (2022) introduce the expert choice routing mechanism, which assigns equal capacity to every expert and allows tokens to com- pete for expert selection. Yang et al. (2024) propose threshold based router that uses manually set threshold to control the number of activated experts for each token. Huang et al. (002)) also propose a threshold- based router but take it a step further by incorporating dynamic loss to prevent activating too many experts.
Heterogeneous Experts Design. Unlike classical MoE frameworks that utilize feed-forward net- works with the same configuration for all experts, recent works have explored the design of hetero- geneous experts. Ainslie al. (2023b) propose heavy branch alongside light branch, using
3

---

Published as a conference paper at ICLR 2025
router to select important tokens for processing through the heavy branch. Raposo al. (2024) fur- ther refine this concept in the decoder-only setting by defining the light branch as a skip connection. Additionally, Zeng al. (2024) introduce set of null experts alongside ordinary experts, while Wang et al. 2024) explore more diverse strategies for integrating heterogeneous experts.
In this paper, we propose the TC-MoE, novel method that complements existing research rout- ing mechanisms in MoE models. Our framework provides a comprehensive design for expert spaces by leveraging heterogeneous experts, thereby improving the overall performance and scalability of MoE architectures.
3 PROPOSED TC-MoE
In this section, we begin with an overview of the widely used Top-K routing mechanism in MoE models. We then introduce Ternary Choice MoE, method that expands the expert space with minimal computational overhead. Following this, we propose a new load balance loss to ensure effective load balance across the expanded expert space. Finally, we present reward loss technique that achieves flexible trade-off between efficiency and performance in our approach.
3.1 REVIEW OF TOP-K ROUTING MECHANISM
In typical MoE architecture for transformer language models, Feed-Forward Network (FFN) layers are replaced with MoF. layers. Each MoE layer consists of N independent FFNs, referred to as experts, {E1, Ez.... EN), along with a trainable router. Given a hidden representation h E Idd of the input token, the router computes the probability distribution over the experts follows:
phh) Softmax( (Wa, h + bg);
where W, E Rxd is a trainable weight matrix, and by R is the bias term. Then the Top-K router selects the top K experts with the highest probabilities for each input token. The gate values for the selected experts are set to the normalized probabilities, while those for the other experts are set to 0:
p. (h))
> (y),) JEE
3E9
(h)
O,
E
where E denotes the set of the top experts with the highest probabilities. The final output 0 of the MoE layer is computed the weighted sum of the outputs from the activated experts:
Σ (h) (h). iEg
(3)
3.2 TERNARY CHOICE
Although the Top-K routing scheme is widely used in MoE models, we identify two key limitations of this approach. As illustrated in Figure[] the scheme exhibits unnecessary activations where some activated experts negatively affect model performance. also fails to fully utilize existing experts, neglecting the potential benefits of contrasting expert outputs. While most previous studies have focused on routing scheme modifications, we propose the TC-MoE, which expands the expert space to provide richer activation options for the router. Specifically, as illustrated in Figure 2b] expand the original expert space by applying the ternary set 1-1,00 0, 13. This allows us to project each expert E; into three distinct experts (F. F.P, E. defined as follows:
(b) = Ex),,, (h) = O, E, (h) = (h), Vh E Rd.
(4)
In our design, E and E, share parameters with E, While E contains no parameters and remains identical across all experts. We further simplify by retaining only Eq. EK. this is sufficient for the Top-K K router to activate any number from to K these experts. As a result, TC-MoE has total of 2N + K experts.
As E! and E share parameters with E,, the only additional parameters and computational costs in our method come from the router component. With the number of experts increased to 2N +
4

---

Published as a conference paper at ICLR 2025
K, TC-MoE introduces (N K)d + N K additional parameters and incurs O((N + K)d)
additional computational overhead. Importantly, these additions are negligible compared to the overall computational costs of the MoE block.
For simplicity, we define the sets of each type of expert as follows:
E = [ li E [N]), E = (E E [K]}, == E [N]
(5)
Our method provides an alternative perspective for addressing the aforementioned limitations of the Top-K routing scheme. By incorporating E", the router can avoid unnecessary activations by activat- ing experts from E which do not contribute to the output and require no computation. Moreover, introducing E-1 enables the router explore the potential benefits of flipping the signs of expert outputs.
Furthermore, we find that making a small improvement to the Top-K routing scheme by always activating experts from E is beneficial, which is described in detail in Appendix C
3.3 LOAD BALANCE LOSS
In common MoE models (Fedus et al. 2022 Zoph et al. 2022), an auxiliary loss is typically in- troduced to encourage a balanced workload among experts. However, in our approach, experts are classified into two types: E' U EE!, which incurs computational costs, and E which does not in- cur any computational costs. Therefore, reasonable workload balance considerations in our scenario follows: (1) experts from E. do not need to be balanced with other experts since they do not contribute to computational costs, and (2) the sum of the workloads expert EE and expert Ex; should be balanced, as E and E_1 are distributed on the same device in scenarios involving expert parallelism (Lepikhin al. 2021). Based on these considerations, we propose a new formulation for the load balance loss:
T
J1
(Token j selects expert E or b)²),
(6)
KT
7
f
N
1 T
P: = T j=l
Σ
Pry (h₂) + +P (h))(h))
(8)
N
Loux Σ
(fs F) Pa.
where T' is the sequence length, fi represents the sum of the activation frequencies of experts E and E and p3 denotes the sum of the average probabilities assigned to experts E7 and E,
3.4 FLEXIBLE TRADE-OFF BETWEEN EFFICIENCY AND EFFECTIVENESS
Since E represents special class of experts that incurs computational costs, it is crucial to understand how the router learns to allocate gate values to these experts. Based on our analysis, propose novel auxiliary loss, termed the reward loss, achieve flexible trade-off between efficiency and effectiveness by tuning the activated ratio of experts from ES,
During the backward pass, the gradient of the gate value for each expert is computed follows:
dc
Ea(h)):
ac 0g. (h)
EEg
80
(10)
0.
i i
For each activated expert E, the term on reducing the loss function. Since the sum of the gate values is constrained to competitive
(g) indicates the impact of increasing the gate value
ac
5

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of performance across evaluation benchmarks. "Avg. K" denotes the average number of activated experts that the reduction ratio of FLOPs compared to the Top-K baseline. The bold number indicates the highest value for each benchmark.
incurs computational costs during inference. "#FLOPs 1." denotes
Bur road!
Top-K
35.15
42:00
13:43
Toy.P P
100
0.36
15.36
99.54
35.74
90.3
12.33
41.00
68.05
43.55
53.30
48:53
ReiRijau
TCMME
1.8
0.55
30
25.58
90
43.51
12.90
$8.65
61888
54.85
49.54
praierd Nare coudd
Top-k
8.00
16
56.6
25.15
90 16
43.72
26.6
68:31
61:11
$1.80
41.90
Try molk
Top. &
16
36.00
46.05
17.90
11.33
41:08
51.41
48.75
FacWick
Jase ewudd!
TC-M6
dynamic arises among the activated experts. Experts that significantly contribute to reducing the loss function are assigned higher gate values, verified in Figure
Following Equation
10 for activated expert F.P, we have
OC
JC 80 5 (b)
OC 80
0 =
(II)
dg (h)
This indicates that expert E. has no impact on reducing the loss function. Therefore, when com- peting with other activated experts, expert E! tends receive higher gate values than experts with negative impacts but lower gate values than those with positive impacts. This effectively helps avoid unnecessary activations.
Based on the above analysis, we propose extending our method to achieve a flexible trade-off be- tween efficiency and effectiveness. Specifically, manually assign negative value to ac
go instead of 0, thereby giving expert E positive contribution in reducing the loss function. Conse- quently, the router will learn to promote the activation of these experts, while selectively deactivating other types of experts with minimal positive contributions. To achieve this, we introduce a new aux- iliary loss, termed the reward loss, defined as follows:
K
T
Lrwd = T
Σ
DEN
g (hj),
(12)
1=
1=1
where 7' is the sequence length, and gæp (hj) represents the gate values expert E on token hy.
Linearly combining the language modeling loss (Cim), the load balance loss, and the reward loss, we yield the total loss, formulated follows:
C = Lim + ai Loaz + anhrrdd
(13)
where da is a hyper-parameter known as the load balance factor, and 02 is a hyper-parameter known the reward factor.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Pre-trained Datasets. We train models using the RedPajama dataset (Computer 2023) and the FineWeb dataset (Penedo et al.) 2024). The RedPajama dataset includes diverse sources such Common Crawl (CC), C4, Wikipedia, Github, books, arxiv, and Stackexchange. The Web dataset is an open-source, high-que high-quality training dataset consisting of cleaned and deduplicated En- glish web data from CC. In experiments, all models trained on 100B tokens.
Architecture. We employ a decoder-only transformer model, primarily based on the LLaMA ar- chitecture (Tourron al. 2023). Each transformer layer includes both attention layer and

---

Published as a conference paper at ICLR 2025
Table 2: Configurations of our MoE models.
#Agtiivatd Experts/ #Total Experts
#Activated
Model
Layer
#Hidden Size
#Hendd
#KV Heads
Mnterrediitte Size
#'Taaal Params
Tiny
2/8
298M497M
Base
32
1024
16
2
2816
2/8
681M//2.3
Fine sene-grained
4/16
631M/2.1B
2.52
TC-MoE
TC-MoE
Top-P
Top-P
Random drop
Random drop)
HellaSSag Aoc. (%)
43
42
12
1.4
1.6
1.3
12
1.4
1.5
1.85
Average
Averago
(a)
(b)
Figure 3: Comparison (a) the language modeling loss and (b) the accuracy on HellaSwag under different budgets for the average number of activated experts. The results demonstrate TC-MoF outperforms other competitors under all settings.
MoE layer. RMSNorm (Zhang & Sennrich 2019) is applied to the inputs of both attention layers and MoE layers. Within the attention layer, adopt the Group-Query Attention (GQA) (Ainslie et al. 2023a). Additionally, RMSNorm is used to normalize each key vector (Dehghani et al. 2023)
Each FFN expert employs the SwiGLU activation function (Shazeer 2020). In our experiments, We employ three types of models: tiny, base, and fine- ree-graieed base, as summarized in Table We use the same tokenizer as GPT-NeoX-208 (Black et al., 2022), which has vocabulary size of 50257.
Competitors. We pre-train three baseline methods alongside our proposed TC-MoE:
1. Top-K: A standard Top-K routing scheme that activates the top K experts for each token. We select K architectures (Zoph et al. 2022 Jiang et al. 2024 Wei et al. 2024 Wu et al. 2024).
K 4 as these are the most common configurations in modern MoE
2. Random drop: A variant of the Top-K routing scheme that, with probability p, does not activate the expert with the second highest probability.
3. Top-P: The Top-P routing scheme (Huang et al., 2024), which activates the smallest set of experts whose cumulative probabilities surpass a threshold 1 for each token.
4. TC-MoE (ours): It expands the expert space and adopts the standard Top-K routing scheme to activate the top K experts within this expanded expert space for each token.
The Top-K baseline adopts a fixed number of activated experts, whereas Random drop, Top-P, and TC-MoE allow for flexible trade-off between effectiveness and efficiency by tuning specific hyper- parameters. Details of these hyper--arrmeters are provided in Appendix B
Evaluation. We evaluate these models on seven benchmarks: ARC (Clark et al.
2018 BoolQ
HellaSwar Zellers et al. 2019). OpenBookQA (Mihaylov lov et al. 2018), PIQA (Bisk et al. 2020). SIQA (Sap) et al. 2019), and WinoGrande (Sakaguchi et al. (021)) ). These tasks assess the model performance on logical reasoning, language understanding, commonsense reasoning, and knowledge utilization. Additionally, measure the average number of activated experts that incur computational costs to demonstrate the efficiency of each model. Note that in our TC-MoE, only the activations of E and F counted since E. incurs computational costs. For simplicity, refer the average
Clark et al. 2019), MMLU (Hendryeks et al. 2021), LAMBADA (Paperno et al. 2016).

---

Published as a conference paper at ICLR 2025
Table 3: Ablation study on the contribution of different types of experts. "Multiplication Set" de- notes the set used to multiply the original expert space, "Average K" denotes the average number of activated experts. Specifically, {1} represents the Top-K baseline.
Multiplication Set
Average Performance
Average K
#FLOPs
{1}
2.00
48.71
{ 1-1,1)
2.00
0.0%
49.00 (+0.29)
{0.1}
1.81
6.9%
49.23 (+0.52)
1, 0,
1.82
6.4%
49.54 (+0.83)
Postive contrituton
Posttive contribution
Posilioo contiibution
105-
Combuuttonn
Contuutto
002-
Negetti
Nogaling conttiution
61
63
04 05 Or
02
64 oe
Gate Valuee
Gate Valter
(b)
Figure 4: Distribution of contri- butions from activated experts in TC-MoE on pre-trained data.
Figure 5: Distribution of contributions from activated experts in (a) baseline and (b) TC-MoE on ARC-Easy. The results show significant alleviation of unnecessary activations by TC-MoE.
number of activated experts the average number of activated experts that incur computational in the following sections.
4.2 MAIN RESULTS
Table summarizes the performance of various models across different evaluation benchmarks. The results highlight the superior performance of the proposed TC-MoF.
Specifically, when the base model is pre-trained the RedPajama dataset, TC-MoE outperforms competitors on ARC-Easy, HellaSwag, OpenBookQA, SIQA, and WinoGrande, while achieving comparable results on BoolQ, MMLU, LAMBADA, and PIQA. Notably, TC-MoE achieves an erage accuracy of 49.54% surpassing the Top-K baseline by 0.83%, Random drop by 0.88% and Top-P by 0.71%. %. For the fine-grained base model pre-trained on the RedPajama dataset, TC-MoE also outperforms the Top-K baseline, improving the average accuracy by 0.96%.
When pre-training on the FineWeb dataset, TC-MoE demonstrates even greater accuracy improve- ments. For the tiny model, TC-MoE surpasses the Top-K baseline by 1.18%. Similarly, for the base model, TC-MoE outperforms the Top-K baseline by 1.17%.
Beyond improved accuracy, TC-MoE consistently demonstrates greater efficiency. Specifically, it reduces the average number of activated experts by 9.0% and the required FLOPs by 6.5% com- pared to the Top-K K baseline when using the base model pre-trained the RedPajama dataset. On the FineWeh dataset, TC-MoE reduces the average number of activated experts by 7.0% and the required FLOPs by 5.1%. These results demonstrate that our method achieves significant gains in both effectiveness and efficiency over the Top-K baseline.
Additionally, we conduct a thorough comparison of these methods under different budgets for the average number activated experts. The results are shown in Figure 3 The figure demonstrates that TC-MoE consistently outperforms the other two competitors. Notably, in terms of the language modeling loss, TC-MoE reduces the loss by approximately 0.017 compared to competitors. In terms of the accuracy HellaSwag, TC-MoE improves by up to 0.7% compared other methods.
8

---

Published as a conference paper at ICLR 2025
100
Aclivated Ratio (%)
Activated Ratio (%)
60-
60
31
Layer
Layer
(a)
(b)
Figure 6: The activated ratios of different types experts across layers on (a) the pre-trained data and (b) the test data (ARC-Easy). The results show similar activated pattern different data.
Activated Redoo 1%)
10
Aclivated
80
6-
Export
Expert
(a)
(b)
Figure 7: The activated ratios of different experts in layer 16 on (a) the pre-trained data and (b) the test data (ARC-Easy). The results show the effectiveness of our load balance loss during both training and inference.
4.3 ABLATION STUDY
We conduct an ablation study by evaluating the performance of our method using only a subset of (,1,0,1) 1} to multiply the original expert space. demonstrated in Table expanding the expert space with either 1) {0. }} improves the model performance. Specifically, expanding the expert space with { [11, results in a 0.29% average performance increase, while maintaining the average number of activated experts at 2.0. When expanding the expert space with {0, 1), the average performance increases by 0.52%, and the average number of activated experts is reduced by 0.19. When expanding the expert space with the complete set {-1, 0, 1), the model achieves the best performance, with both improved results and a reduced number of activated experts. In summary, both the F-1 and E. types contribute to the improvement in model performance, while the F type also significantly enhances model efficiency.
4.4 ANALYSIS
Unnecessary Activations. We investigate the effect of our method on reducing unnecessary activa- tions. Figure 4 shows the distribution contributions from activated experts in the TC-MoE model on the pre-trained data. Compared to the distribution of contributions in the baseline model, shown in Figure Ta our TC-MoE significantly reduces the occurrence unnecessary activations. Addi- tionally, when analyzing unnecessary activations ARC-Easy, observe significantly large number of activations that contribute negatively. As shown in Figure [ TC-MoE demonstrates an greater reduction of unnecessary activations in this scenario.

---

Published as a conference paper at ICLR 2025
40
Aclivated Ratio (%)
Adtivated Ratio (%)
30
10-
0-
5000
10000
15000
20000
25000
5000
10000
15000
20000
25000
Step
Step
(a)
(h)
Figure 8: The changing curves of activated ratios of (a) type F-1 and (b) type E of different layers.
Activated Ratio of Different Types of Experts. To analyze the activated ratio of different types of experts, we visualize the activated ratios across layers in the TC-MoE model. The results are shown in 6 We observe that the E type has the highest activated ratio, indicating its major contribution to the output. Additionally, the model spontaneously learns to allocate an activated ratio of totally around 20% to the E type and the E- type, highlighting their necessity for more powerful routing. The activation ratios on ARC-Easy are similar to those on the pre-trained data, demonstrating the generalization of our method.
The distribution of activated ratios varies significantly across different layers. Figure shows the changing curves of activated ratios across layers throughout training. We find that the E and E types exhibit contrasting activation patterns across layers. Specifically, the model activates a higher proportion of E-1 experts in deeper layers, while E experts are predominantly activated in shallow layers.
Load Balance. To explore the effectiveness of our load balance loss, we visualize the activated ratios of different experts in TC-MoE during training. Figure 7 shows the activated ratios in layer 16. To observe the actual load balancing distribution, we stack the activated ratios of each E, and E! pair, as these experts are distributed on the same device when involving expert parallelism. Additionally, we plot the sum of the activated ratios of experts from E the position of expert 9, since these experts do not contribute to computational costs. The results demonstrate that TC-MoE achieves near-perfect workload balance on pre-trained data. The sum of the workloads of experts El and experts E, are balanced, each around 11.5%. Additionally, the activated ratios of expert El and expert E,- are not fixed but instead learned dynamically experts from 0 do not participate in the load balancing, allowing the model to activate E without any constraints. On ARC-Easy, observe slight deviation with maximum workload at 15.0% versus minimum at 8.5%.
the model. Furthermore,
5 CONCLUSIONS
In this paper, we present the Ternary Choice MoE (TC-MoE), novel approach that addresses the limitations of traditional Top-K routing in MoE architectures. By expanding the expert space through multiplying each expert with the ternary set 0, 1}, we introduce greater flexibility and diversity into expert activations without significant computational overhead. Our approach enhances expert utilization, effectively mitigating both unnecessary activations and expert underutilization in conventional MoE models. Extensive experiments demonstrate that TC-MoE achieves consistent improvements in effectiveness and efficiency, surpassing existing methods across multiple bench- marks. These results highlight the potential TC-MoE as a scalable, computationally efficient method for MoE models. We believe this work provides new perspectives for further development in the design and optimization of MoE models, paving the way for more advanced and resource- efficient large-scale models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
Zhouchen Lin and Yisen Wang were supported by National Key R&D Program of China 2022ZD01603000 Zhouchen Lin was also supported by the NSF China (No. 62276004). Yisen Wang was also supported by National Natural Science Foundation of China (92370129, 62376010), and Beijing Nova Program (20230484344, 20240484642).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Joshua Ainslie, James Lee-Thorp. Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. GQA: Training generalized multi-query transformer models from multi-head check- points. arXiv preprint arXiv:2305.1322 2023a.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyan- skiy, David Uthus, Mandy Guo, James Lee-Thorp. Yi Tay, et al. COLT5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752 2023b.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelli-
gence, 2020.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, al. GPT-NeoX-200B, An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologiess, Volume (Long and Short Papers), 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.000 080305555, 2018.
Together Computer. RedPajama: open dataset for training large language models, 2023. URL http:///hh..bbb.oooooooooeeccor/
ama Datal
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, al. DeepSeekMME: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.006 1606666 2024.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning, 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 1-09: 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2025
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder task needs more experts: Dynamic routing in MoE models. In Proceedings of the 62nd Anmual Meeting of the Association for Com- putational Linguisties (Volume I: Long Papers), 2024.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.040800 2024.
Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 6(2):181-214, 1994.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016.
Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The Wen datasets: Decanting the web for the finest text data at scale, 2024.
/ttp:///xxiivv.. abs /ttp://awww...g//////0.......... 17557
David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mixture-of-Depths. Dynamically allocating compute in transformer-bace language models. arXiv preprint arXiv:2404.02256 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An ad- versarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. SociallQa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gaareely-gated mixture-of-expers layer. In International Conference on Learning Representations, 2017.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Wang, Xingwu Sun, Ruobing Xie, Shuaipeng Li, Jiaqi Zhu, Zhen Yang, Pinxue Zhao, JN Han, Zhanhui Kang, Di Wang, et al. Hmoe: Heterogeneous mixture of experts for language modeling. arXiv preprint arXiv:2408. 10681, 2024.
Tianwen Wei, Bo Zhu, Liang Zhao, Cheng Cheng, Biye Li, Weiwei Lü, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Liang Zeng, et al. Skywork-MoE: A deep dive into training techniques for mixture-offexperts language models. arXiv preprint arXiv:2406. 06563, 2024.
12

---

Published as a conference paper at ICLR 2025
Shaohua Wu, Jiangang Luo, Xi Chen, Lingjun Li, Xudong Zhao, Tong Yu, Chao Wang, Yue Wang, Fei Wang, Weixu Qiao, et al. Yuan 2.0-M32: Mixture of experts with attention router. arXiv preprint 47X1v:2405.17976, 2024.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In International Conference on Learning Representations, 2024.
Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. XMoE: Sparse models with fine-grained and adaptive expert selection. In Findings of the Association for Computational Linguistics ACL 2024, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Zihao Zeng, Yibo Miao, Hongcheng Gao, Hao Zhang, and Zhijie Deng. AdaMoE: Token-
adaptive routing with null experts for mixture-of experts language models. arXiv preprint arXiv:2406.1322 13233, 2024.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Advances in Neural Information Processing Systems, 2019.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, al. Mixtureoofeexprtss with expert choice routing. In Advances in Neural Information Processing Systems, 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoF: Designing stable and transferable sparse expert models. arXiv preprint arXiv2220..08906, 2022.
13

---

Published as a conference paper at ICLR 2025
A DETAILS OF CALCULATING THE CONTRIBUTION OF EACH ACTIVATION
To evaluate the effectiveness of the routing scheme, we conduct experiments to analyze the impact of routing decisions model outputs. It is important to note that we not discussing the contribution of each expert, but rather the effect of each activation decision made by the router.
Specifically. we first randomly select 15 sequences from the training set of RedPajama (Computer 2023) and the test set of ARC-Easy (Clark et al.] 2018), respectively. The average sequence length of samples in the training set is 1608, while it is 30 in the test set, We use the language modeling loss on these sequences to measure the quality of model outputs. For a specific activation A, a straightforward method obtain its contribution to forward the model on the input sequence twice: once with this activation and once masking this activation. We then calculate the difference in the value of the loss function. This can be formulated as follows:
(14) where x denotes the input sequence, C denotes the loss function, M denotes the function of the model, and My4a1 denotes the function of the model when masking the activation A. For sequences sampled from the pre-trained data, we compute the loss across all positions, whereas for sequences sampled from the test data, we calculate the loss only over the tokens constructing the answer. When masking activation A results in a higher loss, we calculate a positive contribution for activation A. This indicates that the activation decision has beneficial impact on model performance.
Contribution == (AM, - ((M())),
In practice, we observe empirically that the impact of masking a single activation is too small to analyze due to numerical errors. Therefore, we alternatively categorize the activations into groups based on their gate values and calculate the contribution of each group of activations. Specifically, we divide the gate values from to 1 into 10 intervals: to 0. 0.1 to 0.2, ... 0.9 to 1.0. Nevertheless, the unequal size of different groups still makes it unfair to compare the contributions across groups. To address this, we randomly select the same number of activations mask within each group for a fair comparison. Specifically, we mask 20 activations in each layer of each group for sequences sampled from the training set, and activations in each layer of each group for sequences sampled from the test set. We then calculate the loss difference as shown in Equation [4
For the experiments involving the flipping of expert output signs, we use a similar method. For specific activation A, we forward the model on the input sequence twice: once with this activa- tion and once with the flipped sign activation, then calculate the difference in the value of the loss function. We define this as:
(15) where (A).U((A denotes the function of the model when the sign of activation A is flipped. We randomly flip the sign of 20 activations with gate values lower than 0.2 in each layer, obtaining the distribution shown in Figure lb
Contribution (onthation---CMMMMau---aa--)->-- (()))))(())))))
B HYPER-PARAMETIE
We use the AdamW optimizer with exponential decay rates 31 = 0.9 and 8n 0.95, and apply a weight decay of 0.1 throughout training. The learning rate is warmed up linearly from to 3e-4 during the initial 10% of training. then decays to 3e-5 following cosine decay schedule for the remaining steps. We set the sequence length to 2048 and the global batch size to 2048. Batch size warmup is synchronized with the learning rate warmup schedule.
To achieve a flexible trade-off between effectiveness and efficiency for Random drop, Top-P, and TC-MoE, set hyperparameters as follows:
Random drop: We set the drop probability p to 15%, 45%, and 70% to achieve average activation numbers of 1.85, 1.55, and 1.30, respectively.
Top-P: We set the threshold P to 0.4 as in the original paper (Huang et al. 2024), and the dynamic loss weight le-5, 2e-5, and 5e-5 to achieve average activation numbers of 1.99, 1.56, and 1.14, respectively.
TC-MoE: We set the load balance factor a to 0.01, and the reward factor 02 to 0, 1e-5, and 2e-5 to achieve average activation numbers of 1.82, 1.61, and 1. 1.30, respectively.
14

---

Published as a conference paper at ICLR 2025
15
E
1e:5
2e-5
18-
Activated Ratio (%)
Average K
166
14-
5000
10000
15000
20000
25000
GC
C4
Wikipedia
Gthub
andv Stackaxchange
Step
Source
Figure 9: The changing curves of the average
Figure 10: The activated ratios across different sources of the training data.
number of activated experts when varying the re- ward factor.
For initialization, we adopt an initializer range 0.006. The weight matrix Wg of the router also initialized with standard deviation of 0.006. The bias term by R2N+K of the router is set such that experts of type E² have bias of those of type E-1 have bias of -1, and those of type E have a bias of 10. This particular initialization strategy is designed to guide the router to focus primarily on type E during the early stages of the training process.
C IMPROVING THE TOP-K ROUTING SCHEME
Intuitively, experts from E share similarities with attention sinks (Xiao et al., 2024). The intuition behind attention sinks lies in the fact that the attention scores calculated by the Softmax operation sum up to 1 for all tokens. As a result, the model naturally learns to assign useless attention scores sink tokens. Similarly, in MoE models, the router uses the Softmax operation calculate gate values. In this context, we believe that experts from E serve as sink experts to collect unneeded gate values.
However, within the conventional Top-K routing mechanism, illustrated in Equation TO only the activated experts participate in the competition for gate values. This implies that experts from E can collect useless gate values only when they among the Top-K experts. To address this, refine our design by consistently activating experts type E", enabling them to participate fully in the competition for gate values and more effectively serve sinks. Specifically, activation set & by taking its union with E The updated calculation of gate values is formulated as follows:
revise the
Phhh// Σ (b),, 1EEUE"
(h)
JECUBO
(16)
i E
where & denotes the set of the top K experts with the highest probabilities.
D EFFECT OF THE REWARD LOSS
We also investigate the effect of our designed reward loss. As illustrated in Figure we vary the reward factor from 0 to 2e-5, The average number of activated experts shows different changing curves. By increasing the reward factor, we encourage the model to select experts from E", which incur no computational costs. Consequently, the model tends to have a lower average number of activated experts. Specifically, the average number of activated experts converges to 1.82 when the reward factor 0 and to 1.30 when the reward factor is 2e-5. These results demonstrate that tuning the reward factor enables flexible trade-off between effectiveness and efficiency.
15

---

Published as a conference paper at ICLR 2025
E ACTIVATED RATIO ON DIFFERENT SOURCES
We also investigate the activated ratio on different sources of the training data. The results are shown in Figure O0 We observe that the activated ratio of EA exhibits significant variance across different sources. Notably, The activated ratio of E" is around 11% on data from Github and arxiv, while only 4% on data from Wikipedia. Similarly, the activated ratio of E also varies across different sources. The activated ratio of E is only 4% on data from Wikipedia and Stackexchange, while it 6% on data from CC. The variance across different sources indicates the specialization of experts from E and E-1
16


Reviews:
[{"review_id": "xjBReAxuyO", "reviewer": "Reviewer_h4gE", "summary": "This paper presents an interesting study about Mixture of Experts (MoE). Instead of selecting the top-K experts in the architecture of MoE to be active in prediction, the paper extends the weights of experts to be projected to {−1, 0, 1}, which allows negative contributions from certain experts. To incorporate this idea, it is necessary to enable the router to learn more complex routing strategies during the training process. New loss functions are thus designed. Evaluation was conducted on a decoder-only transformer model, primarily based on the LLaMA architecture. The model consists of 32 transformer layers, each including both an attention layer and an MoE layer. And each MoE layer consists of 8 FFN experts. The evaluation tasks are mainly text-based, such as language understanding and commonsense reasoning. The results demonstrate that the proposed model outperforms mostly the other baselines.", "questions": "Please see the W1 and W2.\n\n---- After the rebuttal\nI have read the response and additional results from authors. W1 and W2 are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: The paper is easy to follow. The motivation is clear. The overall design is reasonable. \nS2: The evaluation is comprehensive, from the overall performance comparison with baselines on downstream tasks, ablation study to analysis of active experts over layers, over steps and so on.", "weaknesses": "W1: In the proposed architecture, will the requirement on memory doubled, or even more? Although no extra computational cost may be introduced.\nW2: The results show that most of the time, only 2 experts (or on average 1.8 experts) are selected for being active. Is this small number of active experts limiting the performance improvement regarding the introduction of negative contribution? By checking Table 1 and 2, we can see that the proposed model is not always outperforming baselines with a notable improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFpou6riFn", "reviewer": "Reviewer_7syP", "summary": "The authors propose an alternative routing scheme to the widely-used Top-K approach in Mixture of Experts (MoE) models. Their method expands the expert space by introducing a multiplicative set {−1,0,1}, allowing each expert E to activate as E\\*1, E\\*(−1), or E\\*0. Here, E\\*0 corresponds to a no-operation choice, effectively reducing the number of active experts to save computational resources. This addition permits the model to activate fewer than K experts when beneficial, lowering computation costs.\nTo optimize the routing, they adapt the load balancing loss to treat each expert Ei equivalently, regardless of whether it is activated with +1 or −1. Additionally, they introduce a loss term that rewards the model for selecting E\\*0, controlled by a hyperparameter that balances efficiency and accuracy. This approach allows the model to expand the effective expert space without adding significant parameter costs. Their results show improvement over three other competitors (Top-K, Random drop, Top-P) in both accuracy and computational savings.", "questions": "- With a variety of datasets used, why were some of the most standard benchmarks, such as MMLU or SuperGLUE, left out?\n- Could the authors provide more justification for choosing (K = 2) in their Top-K approach? It appears they reference Zoph et al. (2022), but additional context would clarify this choice.\n- Some benchmarks categorize tasks by difficulty. Would it be possible to show if the model's accuracy improvements correlate with task difficulty?\n- Why did the authors limit quantization levels to ({-1, 0, 1})? Exploring additional quantization levels might provide further insights.\n- Given the new term introduced in the loss function, is the loss comparison with other baselines still fair and directly comparable?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors introduce a novel routing scheme that minimizes overhead while achieving comparable or improved accuracy and reduced computational cost.\n- The paper is well-written, with a clear, logical flow and excellent presentation.\n- The evaluation is thorough, tested across multiple datasets, strengthening the results.\n- Additional analyses, such as the impact of unnecessary activations, activated ratio of different expert types, Load Balance, and the ablation study, are relevant and well-executed, adding depth to the study.\n- The authors made a commendable effort in providing detailed information for reproducibility, including specific model configurations, dataset details, and experimental setups.", "weaknesses": "- Some figures are not fully self-explanatory; they lack clear takeaways, requiring readers to refer back to the text for important details to understand them.\n- The improvements in accuracy appear marginal and may not be statistically significant (although it does seem to be better in terms of computation needed).\n- Despite reporting the accuracy correctly on some test benchmarks, some of the results presented in the figures (Figs. 3, 4, 5, 6 and 7) are based on training loss or analyses conducted solely on the training dataset. For a more robust evaluation, results should ideally be shown on a validation or test set to confirm generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gXGHqpAGDC", "reviewer": "Reviewer_3e6U", "summary": "This paper identifies the limitation of the commonly-used TopK routing scheme of MoEs:  unnecessary activations and underutilization of existing experts. The authors propose a solution by expanding expert space via a ternary set. Experimental results showcase the effectiveness of the method.", "questions": "- I am having some difficulty understanding how the reward loss functions as you intend. Based on equation (12), minimizing the reward loss seems to result in assigning gating values close to zero for the $E_0$ experts?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to follow and clearly structured.\n\n- The authors propose a novel method for improving MoE, which does not act on the router but takes a different approach to expand expert space. \n\n- The experiments are extensive and the ablation study effectively shows the necessity of having both zero experts and negative experts.", "weaknesses": "- Only the LLaMA architecture is tested, and only Top2 routing is performed. It is questionable if the results can generalize to other base architectures and TopK (k=1, 4, etc.) routing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6z0GjFsCj", "reviewer": "Reviewer_6GcM", "summary": "This paper proposes a scheme for improving MoE models by using the idea of ternary quantization over the experts. The core idea of this paper is based on their observation that the Top-k scheme underutilizes experts and does not have a way of dynamically selecting the number of experts for a particular token. Their scheme proposes to expand the number of experts available by converting the existing experts three types of experts : null, same, negative. In order to deal with the new challenges this scheme brings about, they design a new load balancing and reward loss. The load balancing loss simply extends the standard loss by accounting for the fact that the expert and the negation of that expert are really the same expert and thus on the same device and so their total workload needs to be balanced. Through experiments, they observe this scheme outperforms standard MoE models by about 1% in flop matched settings on downstream tasks.", "questions": "Why did the authors not release their code? How can I verify their results in this paper?\nIn Fig 2B I can see that there is no null expert for Expert 3. This is different from what the text says that the total expert count = 3 x base_num experts. Is this just an illustration error?\n\nIs this paper suggesting that a dense forward pass is worse than a sparse forward pass? Personally, I have performed several experiments with varying the number of K over standard dense MoE models during training and always found performance to monotonically increase with the number of activated experts. It would be beneficial to be able to add some experiments where the authors can show activation how the performance on a benchmark task(s) is impacted by increasing top-k by 1 over some useful range. \n\nHow are the results in Fig 1a computed? Are these results on benchmark scores evaluated for a particular setting of K? A more detailed caption would be helpful for future readers.\n\nHow is the computation for the total number of experts done? It says on Line 210 it is $2N + K$ but if each expert is expanded to three experts, isn't that simply $3N$? Please clarify this.\n\n\nDo the authors find any connection with null experts to attention sinks [1]? It would be interesting to have a discussion of that as I believe something similar to the idea of attention sinks is happening where instead of randomly routing to a useless expert (in the case of fixed top-k) the model can dynamically choose the number of experts needed for each token. \n\n[1] - Xiao, G., et al. \"Efficient Streaming Language Models with Attention Sinks, 2023.\" URL http://arxiv. org/abs/2309 17453.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my opinion, the main strength of this paper is the simplicity of the proposed MoE augmentation scheme. Their observation regarding\nthe negative contribution of some experts is interesting though it is contrary to the idea that dense training / inference is better [1].\nTheir baseline comparisons are correct although I believe comparing simply with Top-k = 2 would be enough and the compute could have been allocated better elsewhere i.e training larger models. While there is limited explanation for why the ternary quantization works, they have good empirical observations. If the schemes load balancing is better than the top-k load balancing loss, then this approach is indeed worth scaling up in my opinion.\n\n[1] - Pan, Bowen, et al. \"Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models.\" arXiv preprint arXiv:2404.05567 (2024).", "weaknesses": "The performance improvement is quite marginal and since the evaluation is done a relatively small models, we do not know if this technique will scale to larger models. I also did not find any theoretical / intuitive justification for why null experts or negative experts should help. In the questions subsection, I have included one potential reason but it would be interesting to know the intuition the authors had for developing this scheme.\nThe evaluation is done on limited benchmarks and the impact on realistically sized models is unknown.  A model with 2.3B total parameters has approximately only 700M activated parameters with a Top-k=2 routing scheme. Did the results hold on any smaller models trained or models trained on a more higher quality pretraining dataset like FineWeb? Can the authors evaluate on MMLU as well?. I would have also liked to see the reduction in FLOPS compared to the standard baseline and how their scheme is impacted by fine-grained experts. My intuition is that it would be less effective since their is more capacity within the base MoE to learn different types of experts and thus the need for negated or null experts might be less.\n\nTypos and grammar errors\nLine 105 - of classical routing scheme", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)