Adaptive Gating in Mixture-of-Experts based Language Models
Authors: Jiamin Li
Qiang Su
Yitao Yang
Yimin Jiang
Cong Wang
Hong Xu
Venue: EMNLP 2023 Main (2023)
Keywords: Mixture of Experts
Adaptive Computation
Training Efficiency

Abstract: Large language models have demonstrated exceptional language understanding capabilities in many NLP tasks. Sparsely activated mixture-of-experts (MoE) has emerged as a promising solution for scaling models while maintaining a constant number of computational operations. Existing MoE models adopt a fixed gating network where each token is computed by the same number of experts. This contradicts our intuition that the tokens in each sequence vary in terms of their linguistic complexity and, consequently, require different computational costs. Little is discussed in prior research on the trade-off between computation per token and model performance. 

This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution. Adaptive gating preserves sparsity while improving training efficiency. We further draw upon curriculum learning to better align the order of training samples and maximize the training time savings. Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality. Moreover, we conduct a comprehensive analysis of the gating decisions and present our insights on which tokens are inherently difficult to process, depending on the specific language task.

Adaptive Gating in Mixture-of-Experts based Language Models
Jiamin Li', Qiang Su', Yitao Yang', Yimin Jiang, Cong Wang', Hong Xu²
City University of Hong Kong, "The Chinese University of Hong Kong jiamin. 1 liemy. ci sitty..edu edu. edu,hk, ytyangêcse cuhk. edu. hk, jymthu@gmail. com congwangêci tyu. edu. hk, hongxu@cuhk. edu.hk
Blaag.su@ii cityu. edu.h.
Abstract
Sparsely-activated mixture-of-experts (MoE) is promising paradigm to address the scalability issue while maintaining a constant number of com- putation FLOPs (Lepikhin et al., 2020; Fedus et al., 2021). MoE utilizes an ensemble of ex- perts to collectively tackle the learning task. Each input activates a subset of experts, resulting in dynamically-changinn and sparse computation graph. This method effectively distributes the com- putation among experts, increases model capacity and improves training efficiency (Du et al., 2022: Rajbhandari et al., 2022). Very recently, there has been quite some prior work on improving the per- formance of Transformers using MoE (Rajbhandari et al., 2022; Zoph et al., 2022; Chen et al., 2023a; Gale et al., 2022).
Large language models, such as OpenAl's Chat- GPT, have demonstrated exceptional language understanding capabilities in various NLP tasks. Sparsely activated mixture-of-expeer has emerged as promising solution for scaling models while maintaining constant number of computational operations. Existing MoE model adopts fixed gating network where each token is computed by the same number of experts. However, this approach contradicts our intu- ition that the tokens in each sequence vary in terms of their linguistic complexity and, conse- quently, require different computational costs. Little is discussed in prior research on the trade- off between computation per token and model performance. This paper introduces adaptive gating MoE, a flexible training strategy that allows tokens to be processed by variable number of experts based on expert probabil- ity distribution. The proposed framework pre- serves sparsity while improving training effi- ciency. Additionally, curriculum learning is leveraged to further reduce training time. Ex- tensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference qual- ity. Moreover, we conduct a comprehensive analysis of the routing decisions and present our insights when adaptive gating is used.
(MoE)
Despite MoE's benefit in scalability, it suffers from suboptimal training efficiency. In particular, focus on the gating mechanism that selects the experts for each token in this work. Existing MoE models adopt a fixed top-2 gating training while employing top-1 gating during inference for shorter response times. Top-2 gating entails twice the com- putational cost per token and doubles the data trans- fer size of all-to-all operations compared to top- 1. Yet, it remains unclear whether top-2 gating actu- ally leads to performance gains that could justify the additional overheads. Therefore, a compre- hensive analysis of the trade-off between training efficiency and model performance increasingly crucial. More practically, how to construct an MoE language model that effectively balances training efficiency and performance, is of great interest and imminent value.
1 Introduction
The field of natural language processing (NLP) has undergone a remarkable revolution driven by the rapid advancements in language models (Cha; Touvron et al., 2023; Bar; pal). They exhibit so- called "emergent" capabilities for a wide variety of applications (Wei et al., 2022). However, as demands for these applications continue to grow, scalability of these models poses an increasingly challenging hurdle due to constraints in compu- tational resources, memory capacity, interconnect bandwidth, etc. (Pope et al., 2023).
Towards this end, we present our first attempt to empirically characterize and improve the efficiency of the gating mechanism in MoE. We observe that across various models and tasks, a large number of tokens display simple linguistic characteristics or a single dominant feature, which allows them to be effectively processed using just the top-1 expert.

---

This observation suggests that the current top-2 gat- ing strategy incurs unnecessary computation costs for significant number of tokens.
We conduct extensive experiments on various NLP tasks and datasets and present a thorough analysis the gating decision of the tokens to prove the effectiveness and efficiency of adaptive gating.
Motivated by this insight, we further introduce adaptive gating in MoE that enables tokens to be processed by a flexible number of experts depend- ing on the gating decision. Our approach, in con- trast to conventional MoE models, preserves the sparsity of MoE. models while enhancing flexibil- ity in token handling. We incorporate a threshold within the gating network to conduct adaptive token routing based on the distribution of expert proba- bilities. With adaptive gating, the majority of to- kens use simple top-1 gating; top-2 gating is selec- tively applied only when necessary and beneficial, thus significantly reducing the computation cost, However, the training efficiency cannot achieve the same improvement the computation cost due to the fact that tokens with top-2 gating always incur a longer training step, thus becoming the bottleneck. Therefore, to enhance training efficiency even fur- ther, we leverage the idea of curriculum learning by strategically adjusting the order of training data samples.
Background
2.1 Mixture-of-Experts
Mixture-of off-Exeets (MoE) has been adopted in var- ious deep neural network models (Shen et al., 2023; Chen et al., 2023b) and has shown great promise in enhancing the performance of language mod- els. For example. GShard (Lepikhin et al., 2020) and Switch Transformer (Fedus al., 2021) effec- tively scale Transformer-based language models with MoE layers.
In particular, these models typically employ an MoE layer to substitute the feed-forward network (FFN) layer. The MoE layer comprises multiple FFNs, each acting as an expert, along with gating network. Each expert i is fully-connected two- layer network utilizing ReLU activation and with its own set of parameters. For given token æ; the output of an expert can be defined as:
We conduct extensive experiments six NLP tasks with different encoder and decoder models. The results show that our approach can effectively reduce the end-to-end training time by at most 22.5%, while achieving comparable inference qual- ity with top-2 gating MoE models. Moreover, we show that the tokens routed to two experts are cou- pled with the nature of each NLP task. For sen- timent analysis, those are the tokens expressing neutral opinions; translation task pays attention sentences with complex structure; Question and Answer connects key words in question and context and assign both with top-2 gating; summarization puts more effort in understanding the pronouns and finding tokens expressing central idea; top-2 rout- ing decision changes along with the token to gen- erated in text completion task and conversational tokens in dialogue response task use top-2 experts frequently. Empirically, we find that small thresh- old value (i.e. 0.1, 0.2) in adaptive gating can lead similar performance as top-2 gating.
FFN((*) - ReLU Wi) Wi.
(I)
where Wi and Wi are the trainable weights of the two linear layers in expert i.
The gating network takes in the embedding tor of each token a and multiplies them with its trainable matrix WG. The gate value for specific token can be determined through:
R = softmar(x Wa).
(2)
This softmax activation R. indicates the weight of each expert in processing the token. The gating network then dispatches this token top-k experts with ie highest activations. The final output of the MoE layer is:
Σ FFN.()) EEE
(3)
that is, the weighted sum of outputs from selected expert(s) E c (FFN,, FFN,,.,FFNN The sparse nature of MoE improves the model scaling in size without increasing the training cost.
Our contributions are as follows:
We propose adaptive gating in the MoE train- ing scheme, which enables tokens to be pro- cessed by a flexible number of experts.
Related work. Several prior works have explored the efficient use of gating or expert selection MoE. Aoki et al., 2022; Zhou et al., 2022: Haz- imeh et al., 2021; Ma et al., 2018 propose different
We leverage curriculum learning to alleviate the training bottleneck caused by varying exe- cution times of tokens.

---

approaches to encourage expert specialization. Dai al., 2022 adopt a pre-defined expert assignment for each input categories. Roller et al., 2021; Zuo et al., 2021 propose to remove gating networks. Zhou et al., 2022 present a novel selection mecha- nism where experts selects token instead of token selecting experts. Hazimeh et al., 2021 introduce multiple routing policies enhance specialization in multi-task scenario. Roller et al., 2021 deter- ministic hashing, while Zuo et al., 2021 use stochas- tie routing. However, could lead to inconsistent inference results. Therefore, they employ a regular- ized loss to penalize the discrepancy of expert se- lection. All existing work adopts a fixed and equal computation capacity for each token and expert, while we look into the trade-off between compu- tation costs and model performance with adaptive gating.
Norm Prob.
Norn Prob
0.15
0.20
0.15
0.10
0.10
0.05
0.05
0.00
0.00
10
Expert
Expert
(b) Taken
0.25
0.25
Norm Prob.
-..20
Nom. Prob.
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
Expert
Expert
(c) Token
(d) Token
Figure I: Normalized expert probability computed by top-2 gating network from four sampled tokens. Here we use the Sentiment analysis task list in Table 2.
Gate
Norm. Computazion
Norm. Moli Layer Running Time
Top-1
0,5
0.67
Adaptive Top-1)
Adaptive (50% Top-1)
0.75x
0.92x
Adaptive Top-1)
Table 1: We compare the computation savings and running time reduction of the MoE layer of varying degrees of top- gating against top-2 gating. The MoE layer running time is measured
3 Design
We now discuss the design of adaptive gating in MoE for training.
testbed Section 4.3. Tokens are randomly selected from the data batch. Here we also use the Sentiment analysis task list in Table 2. We show the results averaged from 40 runs.
3.1 Adaptive Gating in MoE
Load balancing loss. Adaptive gating uses a flexi- ble number of experts to process each token. This flexibility, however, adds extra difficulty to the load balancing problem in training which aims to evenly distribute tokens among all experts. As it is still im- portant to prevent the gating network from overly concentrating
Observation. We first present our empirical find- ings from experiments with classical MoE mod- els. Specifically, we extract the softmax activations and analyze the probability distribution of expert selection for each token in the gating network. Fig- ures depict the normalized activation values of four sampled tokens across 16 experts. We see that for tokens and 4, their activations of the top-1 and top-2 expert are very close as shown in Figures la and 1d, while for tokens 2 and a significant bias towards the top-1 expert exists as in Figures and lc. We find that these ignificantly-biasee distribu- tion accounts for at least 55% of all the tokens in our evaluation.
very small number of experts, in adaptive gating, we impose the soft load balancing constraints on the top-1 gating decisions, while al- lowing top-2 gating decisions to be trained without any soft constraints. That is, the loss of each MoE layer becomes:
I 4= E
(4)
Adaptive gating. Previous work has demonstrated that MoE experts specialize in different linguistic aspects. Building upon our empirical findings, one can see that many tokens can be effectively handled by a single expert during the training stage. To control the number of experts handling each token, we introduce a threshold parameter, denoted as T. If the activation value difference between the top-1 expert, denoted i, and the top-2 expert, denoted j. is within the threshold T, we consider the token as requiring both expert i and expert j for processing. Otherwise, we route the token only to the t expert.
where ft is the fraction of tokens dispatched expert e among those processed by top-1 gating; Pa the average gating probability to expert e over all tokens the current batch, and E, is the number of experts at layer just as in classical MoE (Fedus et al., 2021).
3.2 Batching
Challenge. While adaptive gating provides effec- tive computational savings, Transformer MoE's model architecture poses a significant challenge to training efficiency. Specifically, there is a mismatch

---

Task
Dursee
Model
Architecture
Sentiment analysis
SST-2 (Socher et sl, 2013)
BERT-Base (Deviin et al., 2018)
12-laeer encoder
Translation
WMT19 (De >En) (Foundation)
FSMT (Ng et. 2020)
6 layer encoder, 6 layer decnder
Question and Answer
SQUAD (Rajparker et 2016)
BERT-Base (Deviin et al. 2018)
12-layer encoder
Summarization
CNN/Daily (Iermann 2015; See 2017)
BART (Lewis et 2019)
12. 12-layer encoder, layer decoder
Text gencration
wikitent (Merity 2016)
GPT-2 (Radford al, 2019)
24-tayer decoder
Dialogue response
SODA (Kim et al.,, 2022)
DialoGPT medium (Zhang et al., 2020)
24 layer decoder
Table 2: Overall performance of adaptive MoF and compared basclines in different NL.P tasks. All the models converge the same loss value.
in the data processing granularity between the MoE experts and the Attention layer. The MoE experts operate individual tokens, while the Attention layer requires input in the form of complete sen- tence. As result, although the processing time for large portion of tokens is reduced by half in the MoF. layer, we still need wait until the re- maining tokens (in the same data batch) complete their top-2 processing. Consequently, training step time cannot enjoy the same reduction as in compu- tation. Table shows the computation reduction well empirical MoE layer running time, both normalized conventional top-2 gating. We use PyTorch Profiler to obtain the computation time of MoE layer. For simplicity, here we force a fixed percentage of tokens to be routed to only top-1 ex- pert and mcasure the running time. The reduction in running time is clearly much smaller than the computation savings.
d, we define a complexity vector C::
Ca = [rö 7 A.1
(5)
where is the number of MoE layers the model, and r; represents the ratio of tokens processed by top-2 experts to the sequence length (i.e., the total number of tokens in data sample d) in layer i.
To determine the order of the training data, we identify the data sample with the fewest tokens pro- cessed by top-2 experts, and calculate the cosine similarity using complexity vectors of the remain- ing data samples. Training data is then reordered based on this similarity value, starting from the most similar ones. This approach allows the model to gradually learn from simpler sequences and pro- gressively handle more complex sequences.
Evaluation
Curriculum learning. In adaptive gating, we propose to incorporate the concept of curriculum learning to address the aforementioned training ef- ficiency challenge. Curriculum learning (Bengio et al., 2009), as the name implies, is paradigm where training examples are presented model in increasing order of complexity. aims to en- hance the learning efficiency and generalization performance of models. By carefully designing the curriculum, the model is exposed to easier ex- amples at the initial stages, allowing it to build a solid foundation before tackling more challenging concepts. This gradual learning process has shown promising results in NLP (Wang et al., 2021).
We evaluate adaptive gating in MoE on six NLP tasks using various encoder and decoder models. We then analyze the gating decision to better un- derstand the effectiveness of adaptive gating.
4.1 Tasks and Models
Table 2 summarizes the details.
4.2 Baselines
We use the Transformer models from HuggingFace and convert the FFN layers to MoE layers (Komat- suzaki et al., 2022). We compare adaptive gating's s training efficiency with the following three base- lines and then evaluate the inference performance with top-1 gating MoE.
Adjust training data order. Our intuition is that the number of experts required by each token can be an indicator of the token complexity. We can therefore reorder the training data in way that prioritizes simpler sequences during model train- ing. Additionally, we can group together training data with similar complexity levels to minimize the bottleneck effect caused by difficult tokens in need of t experts.
Dense models. Transformer with no MoE layers. Top-2 gating MoE. MoE models with top-2 gat- ing (Lepikhin et al., 2020; Hazimeh et al., 2021) for training.
Top-1 gating MoE (Switch Transformer). Switch Transformer (Fedus et al., 2021; Kim al., 2021; Xue et al., 2022) uses top-1 gating to mitigate train- ing instabilities.
To quantify the complexity of a training sample

---

Task
Scheme
Norm. Training Time
Computation FT.OPs
Inference Performance
Dense
0.88x
2.18G
0.912
Sentiment analysis
Top-2 Gating
3.28G
0.918
Top-1 Gating
0.99x
0,902
(Accuracy)
Adaptive Gating
0.77x
2.30G
0.919
Dense
0.87x
10.6G
40.9
En->De translation
Top-2 Gating
1x
15.9G
41.1
Tup-1 Gatigg
1,04x
10.6G
39.5
(RI.UU Score)
Adaptive Gating
0.79x
11.5G
41.1
Dense
0.84x
2.18G
75.7
Question and Answer
Top-2 Gating
1x
3.27G
77.6
Top-1 Gating
1.07x
2.18G
75.5
(Fl Score)
Adaptive Gating
0.86x
2.36G
77.4
Dense
0.89x
79G
42.3
Summarization
Tup-2 Gating
43.4
Tup-1 Gating
1.06x
79G
40.8
(ROUGE-I)
Adaptive Gating
0.86x
87G
43.3
Dense
0,84x
3.4T
16.3
Text completion
Top-2 Gating
1x
4.9T
17.8
Top-1 Gating
1,14x
3.4T
16.5
(Perplexity)
Adaptive Gating
3.73T
Dense
0.82x
3.4T
12.5
Dialogue response
Top-2 Gating
4.9T
13.4
Top-1 Gating
0,93x
3.4T
(Perplexity)
Adaptive Gating
0.82x
3.76T
13.3
Table 3: Overall performance of adaptive gating and compared baselines in different NLP tasks. We normalize the training time with reference to the performance of top-2 gating MoF. All the schemes in the same task converge to the same loss.
4.3 Training Configurations
For DialoGPT-medium, the batch size and learning rate are 64 and 0.0001.
We use 8 A100 GPUs, each with 40 GB memory. Data and expert parallel is used for distributed train- ing. We distribute the experts evenly among all the GPUs. In terms of hyperparameters and model architecture, we adopt the default configurations es- tablished in the existing models (Wolf et al., 2020; Kwon and Chung, 2023).
MoE configurations. The parameter size of the FFN in each model is the same in Baseline and MoE models and we set the number of FFNs (i.e. experts) to 16 for all evaluated tasks. The coef- ficient of the load balancing loss is 0.01. No ca- pacity constraints are enabled so no tokens would be dropped. The expert parameters are randomly initialized. We normalize the expert probability in adaptive gating and set the threshold T to 0.1.
Model architecture. BERT-Base has 12 attention heads per layer. The hidden size is 768 and the intermediate dimension is 3072. The Transformer model has 16 attention heads. The hidden size is 1024 and the intermediate dimension in encoder and decoder layers are 8192 and 4096, respectively. BART-Large has 16 attention heads. The hidden size is 1024 and the intermediate dimension is 4096. GPT-2 and DialoGPT-medium have 16 attention heads. The hidden size is 1024 and the intermediate dimension is 4096.
4.4 Overall Performance
We present the overall training and inference perfor- mance in Table 3. Overall, adaptive gating achieves comparable performance to the baselines while sig- nificantly reducing the training time even compared to t gating. This because though top- gat- ing maximizes the computation saving, makes training more difficult to converge to the same loss value, eventually leading to slightly longer training time compared to top-2 gating in 4 out of 6 tasks run. An in-depth analysis of how adaptive gat- ing works in connection each task is presented in Section 4.5.
Hyperparameters. BERT-Bas has a batch size of 24 and the learning rate is 0.00003. The maxi- number of tokens for the translation model is 4096 with learning rate of 0.0005. The maximum number of tokens allowed for BART-Large set to 4096. The learning rate is 0.00001. The batch size of GPT-2 with learning rate of 0.00015.
Sentiment analysis. Adaptive gating in MoE out- performs both Dense models and top-2 gating MoE

---

0.25
0.25
0.25
Top-2 (%6
0.20
0.20
0.20
0.15
0.15
Top- 2
0.15
0.10
Tog
0.100
0.10
0.05
0.05
0.05
0.00
0.00
2 4 6
7 9 1011
0 2 4
5 7 9 1011
0 2 5 7 8 1011
Layer
Layer
Layer
(a) Sentiment analysis
(b) En >DDe Translation
(c) Question and Answer
0.35
0.35
X 0.30
0.30
0.30
c] 0.25
F 0.25
c) 0.25
0.20
0.20
0.20
1: 0.15
0.15
_ 0.15
0.10
0.10
0.10
0 2 6 8 21416)
0 2 4 8 21121416)
0 2 4 8 10T 21 82022
Layer
Layer
Layer
(d) Summarization
(e) Text generation
(1 Dialogue response
Figure 2: Percentage of tokens computed by top-2 experts over all the tokens in each layer when using adaptive gating in MoE.
in all metrics. While the average computation FLOPs per token is higher with adaptive gating compared to top-1 gating MoE, which represents the minimum possible FLOPs in the MoE struc- ture, adaptive gating requires less training time and achieves superior accuracy during the inference stage. This is consistent across all the tasks. No- tably, only 11.3% the tokens in our evaluation receive two experts, which is the lowest among all tasks. Compared to top-2 gating, adaptive gating fo- cuses on assigning more experts to tokens that rep- resent neutral opinions, allowing for more com- prehensive eecision-making process. Conversely, tokens expressing little or obvious sentiment given less attention without degrading accuracy. Translation. Adaptive gating delivers the same per- formance with top-2 gating while reducing training time and FLOPs per token by 25.6% and 38.2%, respectively. Notably, we observe that the gating network in adaptive gating exhibits a particular fo- the complexity of sentence structures. Even tokens that appear linguistically simple can involve two experts when they appear in sentences with intricate structures and grammar. Overall, 25.6% of all trained tokens are routed to two experts. Question and Answer. The training time with adaptive gating is 85.7% that of top-2 gating. Al- though its inference performance is slightly lower, it still outperforms top-1 gating. Through our ex- periments (refer to Section 4.6), we discover that adaptive gating achieves the best results when the threshold is set to 0.2 for Question and Answer. The gating decision is influenced by both the con- text and the specific question being asked. For this task 16.4% tokens receive t2p-2 processing. Summarization. Summarization is the most chal- lenging task in our evaluation, as it involves pro- cessing long and information-rice articles. Adap-
tive gating takes 1 11.8% less training time than top-2 gating, However, its inference performance slightly lags behind. Particularly, in adaptive gating tokens selected for top-2 experts exhibit significant vari- ations across different layers. We provide more detailed analysis of this observation in Section 4.5. Text completion. We use a GPT-like decoder- only architecture for this task. Adaptive gating achieves similar performance as top-2 gating and Dense models while outperforming top-1 gating, When compared to top-2 gating, only 21.8% to- kens rely two experts, resulting in reduction of 23.8% in average computation FLOPSs per token. The selection of tokens utilizing two experts varies considerably due to the diverse nature of the input. Dialogue response. Dialogue response requires more nuanced processing compared to simple text generation, as it involves generating responses in a targeted role based on narrative input and dialogue history. The sparsity introduced by MoE is advan- tageous for this task. All three MoE approaches outperform the Dense model. Among all the tasks evaluated, dialogue response exhibits the highest percentage, 23.4% of tokens routed to two experts, indicating the higher utilization of the top-2 gating mechanism among all the tasks. Upon evaluating the tokens, we observe that this task can be viewed as combination of all the other evaluated tasks.
4.5 Analysis and Insights
While it is intuitive to understand that some minor tokens (e.g., "a", "the". "is"" only need top-1 expert to process, this does not fully explain how and why adaptive gating works different NLP tasks. Thus we analyze how the tokens are processed in training with adaptive gating, and make quite a few interesting observations that can help better answer this question. In a broader sense, we believe

---

our insights are also instrumental towards building better language models.
numbers and the objects those numbers refer to. Summarization. In summarization, the percent- age of tokens using two experts decreases in both encoder and decoder layers (Figure 2d). Based on our analysis of sampled tokens, we identify two patterns for tokens that are likely to be routed to top-2 experts. First, tokens with multiple meanings that rely on both themselves and the surrounding context for their ultimate interpretation. They are often routed to two experts in the shallow layers. Second, pronoun tokens, as understanding their ref- erents is crucial for accurate summarization, use two experts in the deeper layers. This pattern is particularly prevalent in this task. Additionally, cer- tain key tokens (e.g. "in conclusion", "however", "in all") that indicate the beginning the central idea or the main opinion of the context are often sent two experts together with the following tokens.
Note that when BPE tokenizer is used, we ag- gregate the result by mapping the tokens to the natural language word and perform analysis on the aggregated statistics.
Sentiment analysis. Sentiment analysis exhibits the lowest percentage of top-2 gating among all tasks, and the percentage is stable across layers (Figure 2a). The top-2 gating mechanism focuses two main types of input here. First, frequently selects tokens that express a more neutral opin- ion since they are more difficult to classify (Ta- ble 4). Second, tokens associated with sarcastic statements, double negatives, or conflicting opin- ions are also commonly routed to two experts. Adaptive gating effectively identifies these tokens early on in the model they are relatively easy to extract, which explains the stable percentage across layers. special case is when the input does not explicitly convey any sentiment. Adaptive gating tends to initially route all tokens to either the top-1 or t experts and gradually narrows down to more informative tokens. A typical instance of this is "as a dentist's waiting room."
Text completion. Text completion differs from the previous tasks as it is a decoder-only and auto- regressive task. The gating results in text comple- tion are influenced by the current prediction being generated. The focus of tokens changes dynam- ically based on the current prediction. It is chal- lenging to identify specific types of tokens that con- sistently receive two experts. When predicting a pronoun, for example, the focus shifts to the names of individuals. Similar patterns can be observed for numbers and dates. Additionally, we find that the percentage of tokens routed to two experts is linked to the length of the current sequence. Longer sequences have a higher percentage of top-2 gating.
Translation. We focus on English-to-Genn German translation only. We examine the top-2 gating re- sults based on our understanding of the source sen- tences. The distribution of the top-2 gating per- centages varies between the encoder and decoder layers, exhibiting a gradual decrease in the encoder layers and an increase in the decoder layers (Fig- ure 2b). From sampled tokens and the adjusted data order in adaptive gating, we observe that tokens re- quiring two experts are usually within the same sentence. This observation leads to infer that the complexity of sentence structure influences the gating results. In Table 4, we present one sentence containing multiple clauses that are frequently pro- cessed by the top-2 experts.
Dialogue response. Dialogue response, compared to text completion, requires more understanding of the narrative input and the dialogue history. We find that lots of effort are put into processing dialogue history. First, one key distinction is that tokens with conversational meaning occur much more frequently. These words lack informative content but serve to express human-lik sentiments, such as gratitude and politeness. We infer that routing these tokens for two experts indicates that there is a difference between the conversational usage and written text and it is also critical to learn where and when these words should be used. Second, given the nature of the dialogue, many conversations are based on underlying assumptions and conditions. Related tokens are usually processed with two to- kens to improve the understanding of the context. For instance, the dialogue example provided in Ta- ble 4 is built on top of a scenario assuming that
The percentage of top-2 to- kens in question and answer tasks fluctuates across layers (Figure 2c). First, adaptive gating pays ex- tra attention to the question itself. Words listed in Table 4 are some common examples. These tokens often either specify the scope of the ques- tion pose constraints to the answers. Second, in the context side, tokens routed two experts are closely related the question in the input as well. For example, asking a question about numbers and computations would result in t gating on the
Question and Answer:

---

Task
Top-2 gating tokens
Sentiment analysis
realistic, thoroughly, handsome but unfulfilling, simply, is not the worst movie of the year, generic
Translation
believe that anyone who has had the opportunity to visit Algeria during recent months or years make better assessment of what this terrible outbreak of terrorism means to the Algerian people and, indeed, believe that it would be our credit we dealt with this issue in an urgent dehate.
Question and Answer
Which entity, who else, after what, Up until, who was blamed, in terms of, after, Who's death caused this protest?
Summarization
Japanese actress Rinko Kikuchi walks Anjali Rao through the streets of Tokyo. She stunned global cinema audiences with her controversial and Oscar-nominated perfor- mance as a lonely deaf girl in the film "Babel". Rinko Kikuchi is one of Japan's 's hottest young actresses and models, recently working with Karl Lagerfeld as the new face of Channel. Despite her success, she remains an unconventional figure in Japan, at odds with the traditional demure image the Japanese woman and forging a career on her Own terms...
Text completion
Harris announced he would stepping down as rabbi in 201 and the synagogue hired Boris Dolin as his successor, Born and raised in Oregon, Dolin had worked at Temple Beth Israel as a teacher and youth group adviser from 1999 to 2001.
Dialogue response
exactly, definitely, hmm, um, well, guess, [Narrative] Johnathan plans tell his parents that he is gay. He feels anxious because he doesn't 't know they will react. He is worried that they will be disappointed or even angry with him.
Table 4: Examples tokens using top-2 experts different tasks. Underlined tokens use top-2 gating in a sequence.
"Johnathan tells his parents that he is gay" and asks the model to answer questions with this condition.
Nomm.
Training Time
Inference Performance
Sentiment andlysis
1.02x
0.77X
0.92x
1.01x
0.912
0.918
0.917
0
Trandation
0.88X
0183k
01.3k
01588
402
41.1
401.8
41.1
Question
0.925
77.6
4.6 Ablation Study
Summarization
0.98x
1.02x
1.06x
1.01x
408
42.3
43.1
(3.1
Text genenatonn
0.95x
0.93%
0.99%
1.01X
16.6
17.2
17.4
17.4
Threshold T in adaptive gating. We now con- duct an ablation study on the threshold T intro- duced in adaptive gating. Increasing the threshold value results in less sparse model, where more tokens are assigned to the top-2 gating mechanism, subsequently increasing the computational FLOPs. Table 5 shows the inference performance of dif- ferent tasks when the threshold is increased from 0.05 to 0.5. When using a small threshold of 0.05, both the training time and inference performance closely resemble those of top-1 gating MoE. On the other hand, setting the threshold to 0.4 does not always lead to the same performance as top-2 gating. Together with Table 3, we discover that threshold values of 0.1 and 0.2 often strike favor- able balance between training time and inference performance.
Dialogue respones
128
Table S: Overall performance when the threshold T changes.
Training time is normalized with reference to top-2 gating MoE. We highlight the best one with the least training time.
Task
Truining Time Inllation
Inference Performance
Sentiment analysis Translation Question and Answer
22%.
40.00
14%
-0.14
9%
-0,21
Summarization
14%
-0.14
completion
12%
Dialegue response
11%
-0.19
Table 6: Overall performance comparison of adaptive gating when data batch is not adjusted.
iteration cannot achieve the same level of reduc- tion as the computation FLOPs. Consequently, the end-to-end training time is significantly inflated, with an average increase of 13.7%. Additionally, the idea of the curriculum also contributes the improvement in inference performance. The max- imum drop is 0.21 in Question and Answer task when the data is fed and trained in a random man- ner.
Curriculum learning. Essentially, we disable the data order adjustment before each epoch and use the random data loader to feed the training set. We present the performance degradation compared to the full-set adaptive gating Table 6. Since is highly possible that there is at least one token that are routed to top-2 experts, the step time of each

---

5 Limitation
ChatGPT: Optimizing Language Models for Dialogue.
https::/openaal wttpeaai..cooom//////e com/blog/chatgpp/
Choice of k. Adaptive gating MoE eurrently is limited to top-k gating, where can be either I or 2. This is built on the common practice in extensive prior work that top-2 gating shows a promissing resut in MoE. Further evaluation is necessary to validate the performance of wider range of k val- ues. Our experiments were conducted on diverse set of NLP tasks and datasets, but it is essential to note that the effectiveness and efficiency of adap- tive MoE may vary depending on the specific task characteristics. Different tasks may exhibit distinct patterns and complexities, which can impact the performance and generalizability of the proposed approach. Further investigation and evaluation on wider range of tasks would provide more com- prehensive understanding of the limitations and applicability of adaptive MoE.
Google Bard. hittps:/bbrrs google.com/
Raquel Aoki, Frederick Tung, and Gabriel L Oliveira. 2022. Heterogeneous multi-task learning with pert diversity. IEEE/ACM Transactions on Computa- tional Biology and Bioinformaties, 196))3093-3100
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009, Curriculum learning. In Proc. ICML.
Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. 2023a. Sparse Mol as the New Dropout: Scaling Dense and Self-Slimmable Transformers. In Proc. ICLR.
Zitian Chen, Yikang Shen, Mingyu Ding, Zhenfang Chen, Hengshuang Zhao, Erik Learned-Mille and Chuang Gan. 2023b. Mod-squad: Designing mix- tures of experts modular multi-task learners. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 1 11828- 11837.
6
Conclusion
This paper demonstrates the effectiveness and flexi- bility of adaptive gating in MoE models for wide range of natural language processing tasks. By dy- namically adjusting the number of experts based on token characteristics, we achieve improved training efficiency without compromising inference perfor- mance. Additionally, the integration of curricu- lum learning allows tackle the challenge of varying execution times, thereby reducing training costs. Our rescarch sheds light on the trade-off be- tween training efficiency and model performance in sparse and dynamic MoE networks, offering valu- able insights for the development of more scalable and adaptable language models.
Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. 2022. One model, multiple modalities: A sparsely activated ap- proach for text, sound. image, video and code. arXiv preprint arXiv:2205.06126.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR.
Nan Du Yanping Huang. Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph. Liam Fedus. Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pel- lat, Kevin Robinson, Kathleen
Acknowledgement
Heiir-Hellsttem, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. GLaM: Efficient scaling of language models with mixture- of-experts. In Proceedings of Muchine Learning Re- search.
We thank the anonymous EMNLP'23 reviewers and Area Chairs for their constructive and valu- able comments. This work was supported in part by funding from the Research Grants Coun- cil of Hong Kong NN_Cityy11392 C2004-21GE, R 10122-21, R6021-20F, GRF 11209520, and CRF C7004-22G).
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03960
Ethics Statement
Wikimedia Foundation. Acl 2019 fourth conference on machine translation (wmt19), shared task: Machine translation of news.
There is ethic problem in this work.
References
Trevor Gale, Deepak Narayanan, Cliff Young, and Matei Zaharia. 2022. Megablocks: Efficient sparse training with mixture-of-experts. arXiv preprint arXiv:2211.15841.
AI ACROSS GOOGLE; PaLM 2. https://ai.
goggle/discover//pamm//

---

Hussein Hazimeh, Zhe Zhao. Aakanksha Chowdh- ery, Maheswaran Sathiamorrthy. Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. 2021. Dselect- k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. Effi- ciently scaling transformer inference. Proc .MLSys.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Karl Moritz Hermann, Tomás Kociský, Edward Grefen- stette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015, Teaching machines to read and comprehend. In NIPS.
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Min- jia Zhang. Reza Yazdani Aminabadi, Ammar Ah- mad Awan, Jeff Rasley, and Yuxiong He. 2022. DeepSpeed- Dee Advancing Mfxture-of-Expert In- ference and Training to Power Next- eext-Geneatton Al Scale. arXiv preprint arXiv:2201.055:
Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing La, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2022. Soda: Million-scale dialogue dis- tillation with social commonsense contextualization. ArXiv, abs/221 12.222121100 10465.
Pranav Rajpurkar. Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. 2021. Scalable and effi- cient moe training for multitask multilingual models. arXiv preprint arXiv:2109. 109.45...
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. 2021. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34.
Abigail See, Peter J. Liu, and Christopher D. Manning, 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Aran Komatsuzaki, Joan Puigcerver, James Thorp. Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. 2022. Sparse upcycling: Training mixture-of- experts from dense checkpoints. arXiv preprint arXiv:2212.00055.
Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Dar- rell, Kurt Keutzer, and Yuxiong He. 2023. Scaling vision-language models with sparse mixture of perts, arXiv preprint 23xx:2303.07226.
Yoohwan Kwon and Who- Chung. 2023. Mole: Mixture of language experts for multi-lingual auto- matic speech recognition. In ICASSP.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional com- putation and automatic sharding. arXiv preprint arXiv:2006. 16668.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising gguence-to-sequene pre-training for natural language generation, translation, and comprehension. arXiv preprint urXiv:1910 13461.
Xin Wang, Yudong Chen, and Wenwu 7hu. 2021. A survey on curriculum learning. IEEE Transactions Pattern Analysis and Machine Intelligence.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relation- ships in multi-task learning with multi-gate mixture- of-experts. Procedingss of the 24th ACM SIGKDD international conference on knowledge discovery & data mining.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel. Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint 22v.:2006.0766
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- els.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame,
Pier-
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Out, Michael Auli, and Sergey Edunov. 2020. Facebook fair's wmt19 news translation task submission. In Proc. of WMT.

---

Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proc. EMNLP.
Fuzhao Xue, Xiaoxin He, Xiaozhe Ren, Yuxuan Lou, and Yang You. 2022. One student knows all ex- perts know: From sparse to dense. arXiv preprint arXiv:2201. 10890.
Yizhe Zhang. Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. Dialogpt: Large-scale generative pre-training for conversational response generation. In ACL, system demonstration.
Yanqi Zhou, Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, al. 2022. offture-ofeexperts with expert choice routing. Advances in Neural Informa- tion Processing Systems.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yan- ping Huang, Jeff Dean, Noam Shazeer, and William Fedus. 2022. Designing Effective Sparse Expert Models. arXiv preprint arXiv:2202.08800
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jian- feng Gao. 2021. Taming sparsely activated trans- former with stochastic experts. arXiv preprint 1..022210.


Reviews:
[{"review_id": "mb6Q475JV7", "reviewer": "Reviewer_SPuz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\nTo the best of our knowledge, there is no existing work on adaptively controlling the number of experts that each token is being processed. We use various experiments to evaluate the trade-off between the computation per token and model performance and argue that quite a significant portion of computations are actually unnecessary. Besides, we have tried to cover a variety of task types and model types to validate this idea. Could you elaborate on the additional experiments necessary to support our solution?", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and easy to follow. The illustration of motivation and method is clear. \nThe design of adaptive gating in MoE is intuitive and technically novel.", "reasons_to_reject": "This paper uses an existing method and does not mention or lack description of its own.\nInsufficient experiments, as many figures as possible should be used to illustrate the author's point of view.\nMore baseline models are encouraged to be used to evaluate Training Time, FLOPs and Inference Performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces adaptive gating in MoE, a flexible training strategy that allows tokens to be processed by a variable number of experts based on expert probability distribution.  Extensive experiments on diverse NLP tasks show that adaptive gating reduces at most 22.5% training time while maintaining inference quality.\n\n", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"Figures 1 depict the normalized activation values of four sampled tokens across 16 experts.\" See line 209, page3.\nFigure 2 is not clear.\n"}, "extra_text": {}}, {"review_id": "YD3RBw1kfI", "reviewer": "Reviewer_ggKr", "summary": "", "questions": "- How are findings in Section 3.1 / Figure 1 drawn? How are these experiments performed in detail?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments.\n## Response to reasons to reject\n1. To the best of our knowledge, no existing work comprehensively evaluates the performance of adaptive gating in the scope of MoE and the trade-off between the computations per token and the model performance. We provided a variety of experiments with different tasks and models and an in-depth analysis of our findings.  \n2. The performance in Table 3 shows that the FLOPs could be reduced by an average of 1.3x compared with the most widely adopted Top-2 gating while maintaining a comparable inference performance. \n## Response to questions for the authors\nWe will add the experiment details to our paper. The experiments in Figure 1 are measured with the first task in Table 2, that is, using BERT-Base model to perform sentiment analysis tasks using SST-2 dataset. We present four sampled tokens and collect the gating decision of these tokens in a sample iteration during the training stage.\nThe model in Table 1 is the same task as Figure 1 and the statistics are profiled with PyTorch Profiler to obtain the time of all the computation kernels in the MoE layer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is clear and easy to follow. The experimental setup seems solid, with multiple NLP tasks evaluated.\n- The proposed approach looks promising, as it usually outperforms baselines in terms of training time, inference FLOPs and performance. The analysis and ablation study nicely explains model design choices and provides insights on how the method works in each NLP task.", "reasons_to_reject": "- I am skeptical about the novelty of this paper. Despite its potential impact and value, it looks like a combination of existing techniques.\n- The performance improvement (training, inference and test performance) over the baseline is quite small.", "questions_to_authors": "- How are findings in Section 3.1 / Figure 1 drawn? How are these experiments performed in detail?\n", "paper_topic_and_main_contributions": "The paper proposed an adaptive gating mechanism in a Mixture of Experts (MoE) training scheme, which enables tokens to be processed by a flexible number of experts. The proposed training strategy allows tokens to be processed by a variable number of experts based on expert probability distribution. Overall the results show that the method often achieves improved training efficiency without compromising inference performance, with the integration of curriculum learning to be beneficial for reducing training costs. The authors present an interesting analysis where they explain how the gating mechanism adjusts to each NLP task. For instance, for sentiment analysis the tokens expressing neutral opinion are routed to two experts, while this happens in QA for both questions and context text snippets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The authors could use 1-2 sentences in the intro to briefly explain what top-1 and top-2 gating is (L67).\n- Section 2 should be changed structure-wise. It has only a single subsection (2.1), while Related Work is just a \\paragraph{}.\n- Section 4 is also lacking structure. 4.1 is only one sentence.\n- The words 'significantly' should be used more carefully (e.g. L 583), only if a significance test is performed."}}, {"review_id": "S4vOnRTkdK", "reviewer": "Reviewer_P1xP", "summary": "", "questions": "1) line 365: it was not clear how you initialize the experts. As you say use runs use same parameters, I am assuming the effective FF dimension of each expert is original FF dimension/ 16. If yes, this raises the question how do you split the parameters of original FF layer among these 16 experts. Do you initialize them based by splitting the original init across this in order or do you init the params of new expert FFs randomly. I'd be surprised that you match the performance of original dense model even after throwing away most of the original weights.\n\n2) Table 3: it is not convincing that you train top-1 for longer as the numbers are so close that maybe, if you would have kept a long enough wallclock time for all runs, top-1 would have been comparable to the rest irrespective of the loss.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comments. \n## Response to the reasons to reject\n1. We agree that a more sophisticated curriculum design could further improve training efficiency. Currently, we consider the tokens that are handled by two experts as more challenging ones. Therefore, we use a vector that includes the ratio of tokens in each layer receiving top-2 gating to represent the complexity of the data sample.\n2. We agree that top-1 gating in training is not a common practice. We could remove the training top-1 gating baseline results. \n## Response to the questions for authors\n1. We init the parameters of new expert randomly.\n2. There are at least 55% of all the tokens in the dataset that receive biased top-1 gating decision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. MoE/ switch layers is a useful technique and optimizing it is important - the proposed solution is simple and intuitive.\n2. I liked the analysis of complexity of various tokens.\n3. The speedups over top-2 are decent, albeit with the use of curriculum.", "reasons_to_reject": "1. Require sophisticated curriculum making the training pipeline complex and not plug-n-play.\n2. The decision of training top-1 baseline for longer is not convincing given that all results are in same ballpark.", "questions_to_authors": "1) line 365: it was not clear how you initialize the experts. As you say use runs use same parameters, I am assuming the effective FF dimension of each expert is original FF dimension/ 16. If yes, this raises the question how do you split the parameters of original FF layer among these 16 experts. Do you initialize them based by splitting the original init across this in order or do you init the params of new expert FFs randomly. I'd be surprised that you match the performance of original dense model even after throwing away most of the original weights.\n\n2) Table 3: it is not convincing that you train top-1 for longer as the numbers are so close that maybe, if you would have kept a long enough wallclock time for all runs, top-1 would have been comparable to the rest irrespective of the loss.", "paper_topic_and_main_contributions": "Authors study an important direction of use of mixture of experts (MoE) / switch layers at the FF layers and try to optimize the use of top-2 experts (MoE) vs only top-1 expert (switch). As expected, using top-2 experts leads to better performance than top-1 at the expense of twice as many FLOPs and longer runtime. In this work authors, propose to use top-2 experts only when the difference between the highest expert probability and the second highest expert probability is smaller than a pre-decided threshold saving them compute and retaining the performance of top-2 experts. Unfortunately, to gain full computational speedups they also require a sophisticated curriculum learning schedule.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line 124-125 \ntable1 : top-1 missing x\nline 216: what do you mean 55% - is this w.r.t. to a threshhold\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers overall agree that the paper is sound and moderately exciting. While the paper has some flaws, these are mostly because it is difficult to experiment with MoEs if one does not have large computational resources. The initial results are very promising, and the paper is well-written and technically novel.
Decision: Accept-Main