Dense-to-Sparse Gate for Mixture-of-Experts
Authors: Xiaonan Nie
Shijie Cao
Xupeng Miao
Lingxiao Ma
Jilong Xue
Youshan Miao
Zichao Yang
Zhi Yang
Bin CUI
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep Learning
Transformer
Mixture of Experts.

Abstract: Mixture-of-experts (MoE) is becoming popular due to its success in improving the model quality, especially in Transformers. By routing tokens with a sparse gate to a few experts that each only contains part of the full model, MoE keeps the model size unchanged and significantly reduces per-token computation, which effectively scales neural networks. However, we found that the current approach of jointly training experts and the sparse gate introduces a negative impact on model accuracy, diminishing the efficiency of expensive large-scale model training. In this work, we proposed $\texttt{Dense-To-Sparse}$ gate (DTS-Gate) for MoE training. Specifically, instead of using a permanent sparse gate, DTS-Gate begins as a dense gate that routes tokens to all experts, then gradually and adaptively becomes sparser while routes to fewer experts. MoE with DTS-Gate naturally decouples the training of experts and the sparse gate by training all experts at first and then learning the sparse gate.  Our code is available at https://anonymous.4open.science/r/MoE-3D0D/README.md/README.moe.md.

Under review as a conference paper at ICLR 2022
DENSE-TO-SPARSE GATE FOR MIXTURE-OF-EXPERT
Anonymous authors
Paper under double-blind review
ABSTRACT
Mixture-of-experts (MoE) is becoming popular due its success in improving the model quality, especially in Transformers. By routing tokens with a sparse gate to a few experts that each only contains part of the full model, MoE keeps the model size unchanged and significantly reduces per-token computation, which ef- fectively scales neural networks. However, we found that the current approach of jointly training experts and the sparse gate introduces a negative impact on model accuracy, diminishing the efficiency of expensive large-scale model training. In this work, we proposed ing. Specifically, instead of using a permanent sparse gate, DTS-Gate begins as a dense gate that routes tokens to all experts, then gradually and adaptively becomes sparser while routes to fewer experts. MoE with DTS--ate naturally decouples the training of experts and the sparse gate by training all experts at first and then learning the sparse gate. Experiments show that compared with the state-of-the-ar Switch-Gate in GPT-MoE(L.5B) model with Open OpenWebTex, dataset(40GBB, DTS- Gate can obtain 2.0x speed-up to reach the same validation perplexity, as well as higher FL.Os-efficiency of 1.42x speed-up. Our code is available
gate (DTS- Gate) for MoE train-
INTRODUCTION
Transformer models have achieved a huge success in improving model quality via scaling data and model sizes (Brown et al. 22000 Dosovitskiy et al. 2021; Ying et al. 202T). Specifically, Kaplan al. (2020) explored the scaling law of transformer models which shows that the model quality scales as power-law with data sizes, model sizes and the computation. However, with the rapid increasing of the model sizes, is hard to further scale the model to extremely large size due to the limited computation power of available hardware devices. To address these challenges, sparsely- gated Mixture-of-FFi off--eerts- (MoE), a popular form of conditional computation, is proposed to further increase the model size while without increasing the computational cost (e.g., FLOPS) proportion- ally (Bengio et al. 2013 Shazeer et al., 2017: Fedus et al. 202T]] Lepikhin et al. 2021 Lewis et al.) 2021b, Roller et al. 2021). Specifically, tokens routed by a sparse gate corresponding perts that only contain part of the full model for processing, leading to lower computational costs compared a dense model with the same model size.
The routing strategy defined by the gate is key to MoE models, as it not only determines the model sparse structure (i.e., which experts tokens are routed to), but also affects the specialization of ex- perts (i.c., how each expert is trained) (Roller et al. 2021). Statc-of-the-an approaches (Fedus et al.) 2021 Lepikhin et al.) (021)) usually adopt a pre-defined sparse gate (c.g., Top-1 or Top-2), and then initialize and train both the gate functions and experts jointly. However, such join-training over pre-defined gates could significantly limit both the training efficiency and model quality. Particu- larly, at the beginning of training a MoE model, both the gate and the experts are randomly initial- ized. this case, the gate does not have evidence to decide which expert to process a token, and input token. Joint-updating
the experts also do not have experiences to process a randomly-assigne their models requires a vast amount of trial and errors to converge. Furthermore, the pre-defined gates significantly limit the opportunity of exploring all experts to only or 2 at time, and have to rely on some random routing noises to get feedback from other experts and finally reinforce their future routing choices. Our observation shows that such random routing in the initial stage and long- distance reinforce-based model updating in existing approaches could affect both the training time and final model quality.

---

Under review as a conference paper at ICLR 2022
In this work, to overcome the limitations in existing approaches, we advocate new training mecha- nism to decide the sparse gate gradually for MoE models, which is named as Dense- -opeeettt-rrrrrr se gate (DTS-gate -rtSe. Instead of pre-defining the sparse gate in the previous works, DTS-gate -itt starts with a dense gate that routes tokens to all the experts so that each expert gets trained sufficiently at the beginning. Then, after short warming-up training for all experts, the gate adaptively learns the weights of routing to each expert and gradually routes tokens to fewer experts, making the training structure sparser and continuously reducing the computation cost, while keeping the model quality improving as usual. In particular, to implement the DTS--ate, our idea is to carefully control the temperature of softmax- based routing function, so that to adjust the weights distribution among experts and control the sparsity of the MoE layer during training. In short, DTS-gate advances in two aspects. First, compared to the joint-training of gate and experts from scratch, DT'S-gate pro- vides a mechanism of fraining gate after experts, which could reduce a lot of random error-trails at the beginning. Second, compared to the reinforce-basee gate allows to get training feedbacks from all experts and adjust the routing weights directly to the right direction, which not only speedups the convergence of the gate, but also benefits for the experts specialization.
model updating, starting with a dense
Experiments show that compared with the state-of the art Switch-Gate in GPT-MoE(1.5B) model with Open WebText datasen(40GBB, DTSS obtain 2.0x speed-up to reach the same validation perplexity (Figure ((a)), as well as higher FL.OP--effiieency of a 1.42x speed-up (Figure ((b).. Experiments also verify the ability of DST-Gate for scaling models with more experts or more MoE layers.
2 RELATED WORK
Exploiting the sparsity in deep neural networks can reduce both storage and computation requirements for model training and inference. One of the most widely used sparsification methods is weight pruning (LeCun et al. 1990 (Han et al., 2015). Previous stud- ies propose to prune away redundant or less useful weights based on various pruning criteria (e.g.. the importance of individual weights (Han et al., 2015) or groups of weights (Wen et al. 2016 Luo et al. 2017 He et al. 2017) and then fine-ture tune remained weights to regain the lost accuracy. After pruning and fine-tuning, parts of weights are permanently removed, inducing a static sparsity pattern in DNNs. The sparsity pattern/structure is trade-off between model effectiveness and hardware ef- ficiency (Mao et al. 2017). Early works attempt to increase the sparsity ratio or model accuracy by employing unstructured sparsification methods, while recent works focus more structured sparsity for practical speedup on hardware. Interestingly, Frankle & Carbin (2018) points out train- ing a sparse network from scratch is superior or comparable pruning-basee methods. While Liu et al.) (2018) draw a different conclusion under different settings (e.g., training typer-paramertt pruning structures). Therefore, still needs further investigation answer which one is critical to sparse neural networks, the sparsity structure or remained weights. Our DTS is analogous to pruning-basee methods that training all experts first and then learning the sparse gate routing.
Static Sparse Neural Networks
Conditional Computation with MoE Different from static sparse neural networks that perma- nently remove some weights, conditional computation (Bengio et al. 2013) activates only some relevant parts of the model sity structure that remains all weights but brings sparsity into the computation. The MoE model (Shazeer et al.) 2017),
per-example basis, which can be regarded
dynamic spar-
specific form of conditional computation, contains many experts and trainable gating network which selects a sparse combination of experts to process each input sample. Conditional computation is capable of reducing inference cost (without reducing model capacity) or increasing model capacity (without increasing inference cost) from a model accelcration or scaling perspective. On the other hand, the activated parts (experts in MoH) can be regarded as structured sparse blocks, which does not introduce additional computational overhead. However, conditional computation models are often difficult to train, since they require learning discrete routing decisions from individual examples to experts and the gating network tends to converge to a state that only selects the same few experts (Eigen et al. 2013). Shazeer et al. (017)). Lepikhin et al. (2021) and Fe- dus et al. (2021) add auxiliary load- bata--bllannnnggg losses to mitigate this self reifforeffrr phenomenon and improve training efficiency. In such MoE models, the gating network and experts, as two criti- cal components, jointly trained which may interfere with each other. In DTS-gate, we consider
2

---

Under review as a conference paper at ICLR 2022
K=n
K=2
K=1
E
E
Expir 0
Expert
Expert
Exprr
Expert
Expart
>>>
threshold
threshold
threshold
Gate
Gate
Gate
Token
Token
Token
Figure 1: Overview of training MoE with DTS--Gtte At the beginning, the experts not trained and tokens would be distributed to all experts to enable sufficient updates for experts. As the training continues, the output weights of gate begin diverge because experts tend to have their own domain. The gate becomes sparser. Finally, the weights are close to one-hot and each token is distributed to the most suitable expert.
decoupling the training of experts and the gating network by involving all experts starting with high temperature Gumbel-Softmax and then training the gating network be sparser and select the best expert through decaying this temperature. [Lewis et al. (2021b) formulates token-expert al- location as linear assignment problem and guarantees balanced compute loads. [Rlleer et al. (202T) replaces the gating network with hash-based routing strategy, e.g. random hash, clustered hash dispersed hash.
3 METHODS
3.1 PRELIMINARY
The main components sparse MoE layer are network G sparse gate and an expert network
E.
Sparse Gate Network The sparse gate network G takes a token V. (x, E RD) as input and produces the probability of it with respective N experts. Equation formulates Equatioon[]orrullaee the gate function, where WW,, E R DxNN is the embedding of N experts and the top-K outputs are normalized via so fftmaa distribution. The sparse gate network widely adopted by (hazeerr et al. (2017): : Fedus et al.. 2021 Lepikhin et al. (2021) and achieves promising results with constant computation cost.
((x.) = (Top Wg))
Expert Network The expert network E is a set of N experts fei, eN where each is FFN layer and contains its own parameters in our models. For each expert e; (e; R - R D) takes the token v, as input to produce its own output (z,). The final output of the expert network y, is the linearly weighted combination of each expert's output on the token by the gate's output, formulated as Equation 2
g(z.))e((x.)
(2)
3.2 DENSE-TO-SPARSE GATE
Sparse gating has demonstrated its superior model efficiency in both training and inference. but prior work tends to convergence to a sub-optimal model on FLOPs-efficiency or -ffple-efficiency because of the jointly training of the randomly initialized gate network and expert network. In
3

---

Under review as a conference paper at ICLR 2022
Gate Weights Distribution
Tokens Percentage of Each Expert
9 10
35%
0.8
8
6 7
7
MoE Lager
6
5
5
MoE
0.4
15%
4
0.2
2
5%
9101112131415
101112131415
Expert
Expert
(a) Weights Distribution of Single Token
Loads Distribution within Batch (40960 tokens)
The 16 experts' weights and loads distribution of GPT-MoE-Dense (all experts are tivated for each token) among 12 MoE-layers (totally 24 layer). Figure 2(a) shows that the gate's weights each expert is uneven. And in Figure 2(b) route tokens its Top-1 expert based on the DenseMoE gate, and the loads between experts is naturally imbalanced.
Figure 2:
this paper, we propose new mechanism for the gate function, named Dense-to-Spare gate (DTS-Gate), which starts dense gate that routes tokens to all experts and then gradually becomes sparser. DTS--ate benefits from the sufficient training of experts in the early stage and then make the experts selection becomes sparser on the basis of specialized experts. This dense-to-spp to-asss process only occupies small fraction compared with the total training time, which usually takes days to weeks.
Sparsity in Mixture of Experts Sparsity naturally exists in MoE models. Here we train a 24- layer transformer decoder model, with every other FFN layer replaced by 16-expert MoE layer. Each MoE layer is densely connected, where tokens are distributed to all experts but with weighted differently. As shown Figure [ the expert weight matrix is extremely sparse and the loads are naturally imbalanced among experts.
Dynamic Sparsity Gate In order control the sparsity during training, we adopt the softmax temperature to adjust the weights distribution among experts. Formulated as Equation 3 W is the experts embedding, Ç is the extra noise and sampled from Gumbel(0, 1) distribution (Jang et al., 2017), and is the softmax temperature which controls the distribution. When the increases, the distribution becomes more uniform, which evolves more experts into the computation of each token. As the approaching 0, the distribution becomes one-hot, which is more confident for the gate network.
clte-W ΣN Wl,
g(x,)
Sparsity Scheduler By scheduling the temperature of Equation we can control the sparsity of the MoE layer different training stages. There is trade-off between small temperature, where experts distributions are nearly one-hot and lead to large variance gradients learning but less FLOPs in computation. Large temperature is just the opposite. DTS-Gat starts at large and anneal to a small 7 with linearly decaying.
Adaptive Capacity With decreasing, the weights among experts approach one-hot distribution, where most of them are close to zero. To optimize such sparsity into actucal training time, we threshold drop the experts whose weights fall below the threshold as Figure I and extra communication or computation will be wasted. Furthermore, we don't enforce a balanced assign- ment algorithm for D'SS-Gate because observe naturally imbalanced distribution of experts from
4

---

Under review as a conference paper at ICLR 2022
Scele Modill with
Experts
24
24
GPT-0n
GPT-CTS
GPT-On
GPTDTS
22
G2T-Switch
22
GPESwitch
20
20
Velcettoo
18
Voldiatoon
18
28-
15
14-
14
12-
Specdop
12
25x Speeduu
10
6.
1.0
Itorations
FLOPs
1x14
(a) Validation PPL over steps
(b) Validation PPL FLOPs
Figure 3: End-to-end performance comparison between three models. Figure 3(a) and Figure ((b represent the curve of PPL over iterations and FLOPs, where DTS- Tate can obtain 2.0x speed-up to reach the same validation perplexity, well higher FLOPs-efficiency of 1.42x speed-up.
Table 1: Model Architectures
Model
Params
Layers
d
dEFN
NMoE
NEsperts
GPT-Small
117M
12
768
3072
GPT-Medium
345M
24
1024
4096
GPT-MoE
1.5B
24
1024
4096
12
16
Dense-MoE models, shown as Figure 2(b) To meet the demand of above two designs, we enable each expert with adaptive capacity make them well specialized.
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Dataset Models are trained on two datasets separately: English Wikipedia (17.5GB) (Devlin al.) 2019) and WebText (40GB) (Radford al.). Wikipedia is single domain of text, which contains about 2,500 words, and WebText is large and diverse dataset in various domains, which is collected using 45 million links from Common Crawl. We apply GPT2-BPE (vocabulary size of 50257) both datasets for tokenization and binarize the dataset for loading efficiently.
Evaluation Our experiments are conducted on language modeling task, and perplexity (PPL) is reported as the evaluation metric of model performance. We focus exclusively comparing FLOPs- efficiency, defined as the best model performance (PPL) can be achieved given the fixed number of floating-point operations. Mcanwhile, we don't add the FLOPs of balance loss throughout experi- ments in this paper, as we focus gate distribution with or without balance loss. We don't 't report actual training time because the actual training time mixes the computation complexity with system implementation and leads to unfair comparisons as new algorithms without well optimized system implementation. the appendix, we will show the routing pattern of DTS relative stable, with heavy-loaded experts almost keep fixed. And we can utilize system techniques, e.g., partitioning hot experts among multiple GPUs and merging multiple cold experts onto less GPUs, for load balance between GPUs.
Model Architectures We test on three models, including GPT-Small, GPT-Medium and GPT- MoE, and the detailed architectures are shown Table where GPT-Small and GPT-Medium are two standard architectures defined by RRadford et al. We scale the ford-forward layer (FFN) of GPT-
5

---

Under review as a conference paper at ICLR 2022
Medium(345MM from 1 to 16 the MoE Layer, and replace every other FFN as MoE to match the total parameters of GPT-Large(L.SB) We also extend SPPT-Smal(117MM) with various MoE-layer settings to evaluate our DTS-Gate method.
Baselines As for MoE- models, we compare 4 methods totally, including Switch(Top. gate proposed by |edus et al. (202T)), DTS DTS-Dynaaiccdecay the temperature from 2 to 0.3 linearly in the first 15000 steps, and switch to Topl the 20000 step), and Dense-MoE(select all the experts for each token). The rank of theoretical FLOPs per token is PPT-MoE-Densse DTS > Switch SeP-Ssnallexcludding the cost gate network and all to all communication MoE layer).
Training Hyperparameters All models use the same hyperparameters of 4,000 warmup steps, polynomial learning rate scheduler, Adam optimizer, clip norm of 5.0, weight decay of 0.1 and dropout of 0.1. We use GeLU activation functions (Hendrycks & Gimpel 2016) and set sequence length as 1024. We use CrossEntropy as the criterion and 0.01 as the coeffiient of balance loss in Switch. To save the training cost, we use batch size of 256 and sequence length of 768 on the GPT- Small model, other hyper-parameters keep constant. We set the threshold of gates as 0.001 across the training, which is a pprameee determine how large the expert's weight is important and a trade-off between computation and accuracy from our point of view.
4.2 RESULTS AND ANALYSIS
4.2.1 END TO END COMPARISON
To improve the computation efficiency, only part parameters are used for each token in sparse mod- els with the cost of model performance. DTSS-ate aims to shift the model training from dense to sparse, and keep the inference cost same as sparse models, and results are shown in Figure [ Experiments show that compared with the state-of-the-art Switch-Gate in GPT-MoE(1.5B) model with OpenWeb WebText dataset(40GBB) DTS-Gate can obtain 2.0x speed-up to reach the same valida- tion perplexity (Figure ((a). well higher FL.PP-eeficiency of 1.42x speed-up (Figure ((b).. Experiments also verify the ability of DST-Gate for scaling models with more experts or more MoE layers.
Comparison with Dense Models Dense models, where all parameters are participated in com- putation for each token, tends achieve high model performance due to its large parameter pacity but with high computation cost. As shown by Figure [3a] MoE-Dense performs better than other models at the same iteration. By utilizing the dense-to-sparse training mechanism, MoE- DTS shrinks the performance gap between MoE-Dense and MoE-Switch. As for FLOPS-efficiency.. MoE-DTS outperforms both dense models because of its sparse computation.
Comparison with Sparse Models MoE-Switch pre-defines its static Top-1 gating network and jointly training the gate and experts networks. Different from MoE-Switch, MoE-DTS utilizes tem- perature to adjust the distribution of the token-to-experts (one-hot or uniform) and threshold to remove computation of experts with low weights. MoE-IDTS performs better than MoE-Switch in sample-efficiency because of more experts involved in training and updates at beginning, shown as Figure 3(a) As for FLOPS-efficiency, DTS-Gate first involve more experts into warm-up train- ing, which is poor FLOPs-efficency. But with the training going on, DTS-Gate can obtain greater compared with the state-of-the-an Switch-Gas Gate GPT- MoE(1.5B) model with Open WebTex dataset(40GB).
than 5% improvements in
It worth noting that several syper-parametenn of MoE-DTS, such as max/min temperature and decay iterations, needs to be carefully determined. In this paper, we just propose the training mech- anism of So-Saree and prove effectiveness through simple policy.
4.2.2 SCALABILITY
In this section, conduct experiments about the scalability and FLOPs-efficiency of -ate.. Although it doesn't adopt a balanced routing algorithm, experiment results show that DTS-Gate can be utilized in scaling models, because it wouldn't 't converge only few experts.

---

Under review as a conference paper at ICLR 2022
Scale with more layers
Scale with
26
OiMoE
1-expert
L.Me--Layer
4-nepers
24
4.MeE-Layer
expeets
5-MoF-Laayr
16-experts
22
Valldzion
18
16
14
2
2
FLOPs
1c12
FLOPs
1s12
(a) Scale experts
(b) Scale MoE layers
Figure 4: Scalability for MoE-DTS models. It shows that More experts more MoB-layers (larger models with constant FLOPss,, will lead to better FL.Ps-efficiency.
Different Temperature Scheduler
Different Temperature Scheduler
24
24
to 0.1
to 0.1
to 01
to
22
to 0.5
22
to 05
Vaidation Ph
21
21
Veldation
20
19
19
18
18
16
16
8000
10000
14000
16000
18000
20000
1.0
1.5
3.0
3.5
Itraations
1613
(a) Iteration
(b) FLOPs
Figure 5: Different Temperature Scheduler. We vary the temperature interval values and experiments show that larger beginning temperature will cost more computation at early stage, while provide little extra gain (1 0.1 V.S. 2 to 0.1). Small temperature end will lead the model degradation, show by to 0.1 and 2 to 0.1 at 15000 iteration in Figure 5(a)
Increase the Expert Number Based on the 12-layer GPT-Small m17el(117M), we replace the 7th FFN layer by one MoE layer and vary experts number within {1,4,8, 16). As shown by Figure 4(a) MoE-DTS keeps consistent improvements when increasing expert numbers.
Increase the MoE layer number Similarly, we also vary the number MoE layer within (0, 1, and each MoE layer contains 8-experts. Figure 4(b) shows that by increasing MoE layers, MoE-DTS can achieve better model performence with same FLOPs.
4.2.3 EFFECT OF SPARSITY SCHEDULER
Our work proposes the mechanism Dense-To-Sparss training for MoE models, and employ Gumbel-Softtaut to sparsify the MoE layers by decreasing the temperature In this subsection, sev- cral experiments are conducted to show the effect different temperature setecdulers(-schhoeee The used models are GPT-MoE, 24-layer decoder with 12 MoE-layer (16 experts) and dataset is Open (ebG)x((40B)).
We decay from max_value to minuv in the first 15000 iterations and switch to Topl at the 20000 step. Experiments with different max value and min value are evaluated and the results are shown in Figure

---

Under review as a conference paper at ICLR 2022
Tokens Percentage of Each Expert
openweebeext, balanced
6 7 8 9 10
imbalanced eicipediaa balanced
0.5
wicspedin mbalancedd
Validaation PPL
40
MoE Lager ID
0.4
S
0.2
0.0
101112131415
PLOPS
1c12
Expert
(a) with and without Balanced Loss
(b) Expert Loads(40960 tokens)
Figure 6: Effect of Balanced Loads
Max/Min Temperature Under small temperatures, the weight distribution of experts is close to one-hot, which leads to the variance of gradients is large. In the contrast, large temperatures result in nearly uniform dis- tribution gate weights, which evolves more experts into training but the variance of gradients is start from large value and anneal a small r gradually.
distribution and high computation efficiency, but
small. We schedule the temperature Figure shows the sample-efficiency and FLOPs-efficiens of different t schedulers.
By comparing {1 to 0.1] and {2 to 0.1), we find larger begin temperatures will lead to high sample- efficiency but low FLOPs-efficiency, because more experts are participated in computation. By comparing {2 to 0.1} and {2 to 0.3). we find smaller end temperatures cause poor model perfor- mance because of the large variance of gradients. After analysis about these experiments, we choose (2 to 0.3) as our training configs.
Decay Iterations After max/min temperatures determined, need to care about the decay iterations. {2 to 0.1}, {2 to 0.3) and (2 to 0.5) spend about 11800, 13200 and 15000 iterations respectively for decaying temperature from 2 to 0.5. Higher decay rate performs good at sample- efficiency at the expense of FLOOs-efficiency. Choosing suitable decay rate is a balance between sample-efficiency and FL.OPs-efficiency. Actually, considering about the training cost, usually set 5% 15% as the decay iterations.
Dynamic V.S. Static Both large temperatures and small temperatures have their advantages and disadvantages, it would be better to design T scheduler according the models and hardware for the balance between model- perfo-performance and FLOPs-efficiency.
4.2.4 EFFECT OF BALANCED LOADS
Current works utilize either extra balance loss (Lepikhin al. 2021 Fedus et al. 2021 balanced assignment algorithm (Lewis et al., 2021a) to avoid the training of sparse MoFt-layer converge at the status of only few experts frequently used. Balanced loads naturally easy for efficient system implementation. However, is still unknown about which is better for model performance. We conduct several experiments explore this mystery. Meanwhile, DTS-Gate proposed by this paper is orthogonal to the balance loss, because DTS-Gat cares about the weights' polarization at the token level while balance loss cares about the loads of each expert model level/layer level.
With and Without Balance Loss Balance loss can improve the performance of switch trans- former Fedus et al. (2021). Therefore, we conduct experiment with and without balance loss on DTS, to study whether can also improve the model performance of DTS. To analyze the influ- ence of balance loss on TTSt, we replace the 7th layer of GPT-Small(1 17M) models with one MoE layer( 16experts), and training this model with and without balance loss on Open WebText and Wikipedia separately. As shown by Figure [(a) this model performs almost the same on Wikipedia with and without balance loss, while it performs better without balance loss OpenWebTTer the
8

---

Under review as a conference paper at ICLR 2022
begining and then gradually coincide with the one with balance loss. The difference may introduced by that Web is a large and diverse dataset in various domains but Wikipedia only comes from one domain. Therefore, imbalanced distribution is suitable for training OpenWebText. But Why WebText with and without balance loss are similar at the end training. We check the experts' loads in next paragraph.
Expert Loads Distribution We directly load the checkpoints from the end to end experiments, where MoE-DTS is the 1.5B MoE. models described as above. Figure ( that the experts's loads of the 3rd 11th MoE layer is almost balanced, which is learned without balance loss.
5 CONCLUSION AND FUTURE WORK
MoE models suffer from the model quality challenge due to the difficulty of training the pre-defined sparse gate and the experts jointly. To address this challenge in MoE training, we presented DTS- Gate that (1) designs the dense-to-spare training mechanism and (2) provides the scheduling policy for sparsity adjusting- Our evaluations show that DTTS-Gate can not only achieve better model quality in Transformers with given computation budget but also achieve better FL.OPs-effiieecy when comparing with previous works in MoE training. DTS--ate is stepping stone in exploring MoE training. On the other hand, DTS-Gate opens challenges for system execution due to the computation in the early stage and the adaptive capacity of experts. In the future, we would like to design and implement system-level optimizations to achieve efficient training in both model quality and system execution.
REFERENCES
Yoshua Bengio, Nicholas Léonard. and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL
hict:://rxxi.orgg www./rr//////./eeee.eeee abs 3432
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tier Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL htttpsss:/pppcceeingggg.ooo www.unngg....uupr.rr wew.rrrps.... waass.cc/ne pc/ppapper/22///aatt/ wtt.:/06b0oo4441111oobaaaooo.a.. Abtrr:aeataaaa11110ooaoaaaaaaaaa .tml
Abstract
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pp. 4171- 4186. Association for Computational I Linguistics, 2019. doi: 10.18653/v1/n19- 1423. URI. https://doi. http:///o...g//...111 http://...11633///11111..
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. wtt.atpps///geeeeeeeee...o wwwp://wwwwwwweeeeeeeeeeeeeee forum? id=YicbFdNTTy
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. arXiv preprint 13/24443 2013.

---

Under review as a conference paper at ICLR 2022
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion param- eter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL https:
//axxiv.org/aat abs/ 2101.101111000001111002211 03961
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2018
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 2015.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In Pruceedings of the IEEE international conference on computer vision, pp. 1389-1397, 2017.
Dan Hendrycks and Kevin Gimpel. arXiv: 0841, 2016.
Gaussian error linear units (gelus). arXiv preprint
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterizatioo with gumbel-softmax, In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL htt.thts:s//ppeeeeeeeeeee net her/Eorum? 2ttteooruummnrrreee38yeeeeee
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, abs/2001.08366 2020. URL URI.tttp::///rae
wrt. 08361
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang. Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL ltttss://ppeneviier .nt forum? www.ooooowooieeeeeeeoeeeeee 7XHTTXX
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large. sparse models. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6265-6274 PMLR, 2021a. URL httt:://proceeeinnggt m.rr. wttppp/wws/sseeeeeeeee htt.://iinn..n..pp...ess///////e
html
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. arXiv preprint arXiv:2 2021b.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations, 2018.
Jaan-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision, pp. 5058-5066. 2017.
Huizi Mao, Song Han, Jeff Pool, Wenshuo Li, Xingyu Liu, Yu Wang, and William J Dally. Ex- ploring the granularity of sparsity in convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 13-20, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. arXiv preprint arXiv:2 106.04426, 2021.
10

---

Under review as a conference paper at ICLR 2022
Experts' Loads
5000
2000
500
10
id1 Sentence 2 Sentence id Sorterrce id5 Sentonce Sentence
Figure 7: The Loads each expert in 7th MoE-Layer, with 16 experts in total, accumulated of 16 sentence.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
httts:://ppenreiiee net www.mmod=ollmkkke.cooe
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. Advances neural information processing systems, 29:2074-2082, 2016.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie Liu. Do transformers really perform bad for graph representation? arXiv preprini arXiv:2106.05234, 2021.
A APPENDIX
A.I EXPERT LOADS DTS

---

Under review as a conference paper at ICLR 2022
Experts' Loads Per Sentence
200
Sentance id 54
Sentence id 12
Sentence i8
Sentence it.6
Serrence
Surancciidd
Senbience Id
Sentannce 1
Sentenaa 152
SSnntence 5
Sendar id
Sentancee
Sentence Senttnnca kt
Senbince Sentence
Sentenoe Sentence
Sentirce Sentence Sentence Sentenon
Figure 8: The Loads of each expert in 7th MoE-Layer, with 16 experts total, in the sentence level.
Average Experts Per token
14
12
500
1500
2500
3500
4500
5500
6500
7500
8500
9500
10500
11500
12500
13500
14500
15500
16500
17500
18500
19500
20500
21500
22500
23500
24500
25500
26500
27500
28500
29500
Iteration
Figure Average expert per token during training, with 16 experts each MoE layer.
12


Reviews:
[{"review_id": "qyczq-vJS7I", "reviewer": "Reviewer_wm4x", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Notes:\n* The idea is an interesting one. If the quality of sparse models can be significantly improved through using more compute only early on in training, this could become a widely adopted technique.\n* Reasonably well-written paper.\n* The authors use a reasonable baseline of a 24-layer MoE Transformer, 16-experts for every other layer.\n* Visualization of the model dynamics, specifically gate weights distribution, is a nice addition beyond plots and tables of numbers.\n\nImprovements:\n* I suggest a different graphic for the “experts” in Figure 1. Our field already anthropomorphizes neural networks enough; I’d prefer to not see a little human with an etched brain as a stand-in for two matrix multiplications and a non-linearity.\n* A style recommendation. Make each of your captions self-contained with description and result so that the reader can quickly scan your work figure-to-figure. Elaborate over the short phrases you’re using (e.g. “Figure 5: Different Temperature Scheduler”).\n* On page 8, don’t use the abbreviations w/ and w/o.\n* The significance of the gate values in Figure 7 will likely be lost to many readers. As stated above, please elaborate in your captions to describe what is being shown and why it’s important to substantiate your contribution.\n* p8. “Though these tokens are spatial proximity” -> Fix.\n\nQuestions:\n* Figure 3. Why are there periodic dips in the training perplexity curves? If this is from repeating the data in the same order, I’d recommend in the future that you shuffle data each epoch.\n* Figure 4 demonstrates that MoE-DTS improves with more experts and expert-layers. How does this scaling compare to MoE-Switch?\n* “experiments results show that the FLOPs-efficiency can be optimized into time-efficiency” -> Where was this shown? All the plots and data I see are for FLOPs, not wallclock time on a specific hardware.\n* “It is amazing that DTS-Gate can learn to load balance without extra loss.” -> Was this verified to be a property only of DTS-Gate? Please confirm that naive approaches without load balancing do not learn this, or else, perhaps remove this. Also, as a nit, I’d avoid subjective claims like in using the word “amazing”.", "summary_of_the_paper": "This work proposes the Dense-to-Sparse Gate or DTS-Gate. It’s a simple idea of starting training with soft (continuous) routing to all experts and then gradually reducing to hard (discrete) routing. The authors claim that this improves the quality of traditional approaches. The work proposes the novel idea and details how to schedule the sparsity adjustment. The authors suggest this improves traditional approaches by 5% FLOPs-efficiency.", "summary_of_the_review": "I’m positively disposed towards this work, however, my main concern is on the lack of a clear demonstration of the magnitude and the significance of the gains.\n\nPlease clearly show me the asymptotic quality is worse using vanilla approaches (e.g. Lepikhin et al., 2020, Fedus et al., 2021). As far as I can tell, the only substantiating plot for this is 3(b), but in my opinion, this doesn’t clearly showcase the improvement of the method. At the very least, these curves should be extended until MoE-DTS is conclusively better than MoE-Switch on a FLOPs basis. Further, though the idea is interesting, showing stronger empirical results than a 5% FLOPs-efficiency gain is likely needed to convince practitioners and researchers. If this is the case, I will improve from a 6->8.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qp01JFEhtzt", "reviewer": "Reviewer_gUQG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes a simple and easy to implement idea. However, the experiments do not clearly show that Dense-To-Sparse (DTS) is better in terms of accuracy-compute trade-offs than fully sparse baselines (such as Switch-Moe).\n\nIn particular, although Figure 3b should show this (FLOP vs. Accuracy), but the lowest perplexity is achieved by Switch-MoE with $8 \\cdot 10^{33}$ FLOP and DTS-MoE only matches Switch-MoE at around $6.5 \\cdot 10^{13}$ FLOP. Most importantly, FLOP is not very well correlated with actual training time (nor energy consumption), since if some sort of \"masking\" is used to implement DTS, rather than true sparsity, some compute is wasted to compute the experts outputs that are then masked.\n\nTo this reviewer, it is not clear if masking or some thresholding was used to evaluate DTS. Figure 1 depicts a \"threshold\", which could mean that DTS avoid computing the output of some experts (the ones whose weight falls below the threshold), and that the FLOPs depicted in Figure 3b and thorugh the rest of the paper are more correlated with actual runtime. However, this \"threshold\" is not mentioned anywhere else in the paper, so it's very likely that when the temperature is very low (i.e. only 1 of the expert's weights is effectively non-zero) all the experts are being actually used. In any case, notice that even if the reported FLOPs correlate well with runtime, the observation regarding Switch-MoE is still true.\n\nMoreover, many of the experiments do not report results on a test set, not even a validation set. Figures 3 (arguably the main plot) and 5 report *training* perplexity, rather than the perplexity on a separate validation or test set. However, Figure 4 and 6 do report results on the validation set. It's unclear to this reviewer what's the criterion used to report results on one set or the other. In any case, validation perplexity, or even better test perplexity, should have been used through all the paper (training measuraments are relevant sometimes, but not in the context of this paper).\n\nThere are some results from the \"effect of balanced loads\" analysis (Section 4.2.4) which are also not clear to this reviewer, as well. In particular, Figure 6a shows that the use of balanced loads adds an overhead of approximately $5 \\cdot 10^{11}$ FLOPs in the openwebtext dataset. Why? Why only in this task but not in the Wikipedia task? In addition, notice that one of the main practical reasons to use a balanced load (e.g. in Switch-MoE) is to efficiently use all available compute, which is not considered in the analysis presented.\n\nMinor typos and comments:\n\n- Abstract: \"**an** neegative impact\" -> \"**a** negative impact\".\n- Page 4: \"Here we train a 24-layer transformer decoder model, with **and** every other FFN\" -> \"Here we train a 24-layer transformer decoder model, with every other FFN\".\n- Section 4.2.3: There seems to be notes left from the draft of the paper. \"Need We analyze it in token-level's weights?\".\n- Section 4.2.4: \"We directly load**s**\" -> \"We directly load\".", "summary_of_the_paper": "In the context of neural networks with Mixture of Experts (MoE) layers, the paper proposes to decrease the number of activated experts per each input as training progresses, by means of decreasing the temperature in the softmax that is typically used to weight the contribution of the different experts in the MoE layers. The proposed approach is compared with fully Dense MoEs and the recent Switch-MoE baselines in a language modeling task.", "summary_of_the_review": "The main claim of the paper (the current approach of jointly training experts and the sparse gates introduce a negative impact on model accuracy) is not well supported by the experiments. In addition to that, many of the experiments report training measurements rather than test or validation measurements. Comparing training perplexity is not enough to show the superiority of the proposed approach. Some of the details of the implementation are not clear to this reviewer and affect the FLOP vs. Accuracy comparison. Actual runtime comparison (on a particual hardware architecture) would be much better.\n\nGiven all these issues, I recommend to not accept the paper. \n\n\n**Update after rebuttal**\nThe authors have addressed many of my concerns during the rebuttal period satisfactorily. My concern about the claimed \"speed-up\" is still present, but the authors have stated why they decided to use FLOPs rather than actual runtime on a given hardware and implementation. Based on all this, I'm slightly increasing my score for this submission.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TtFgvKuo-er", "reviewer": "Reviewer_XPcr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles an interesting aspect of conditional computation: in the context of MoEs, how do we start training from scratch when individual experts can easily break down and collapse?\n\nThe paper provides an intuitive answer: we can start by training all experts together (i.e. as a dense model), and relax this over time, hopefully, once all individual experts are enough developed. In some setups (which, interestingly, coincide with some of the original motivation ones for MoEs) this may not be possible, for example if the number of experts is just huge.\n\nSome recent work has shown the effect of increasing \"k\" a lot (see [1], Figure 10). It seems that the effect of increasing \"k\" plateaus somewhat quickly. While in those experiments \"k\" doesn't change over time, I think this may suggest there's little to gain in doing so (especially as it's quite expensive).\n\nOverall, I think the experiments in the paper do not show the proposed algorithm offers any advantage over standard top-k routing (Switch, in the paper).\n\nFigure 3 seems to be the only comparison between Switch ([2] i.e. no warm-up) and DTS-Gate (expert warm-up). First, the reported metric is training perplexity. I'm not sure why this is the case: test (or validation) perplexity should be reported. In the second plot (3b), which is the most relevant in my opinion (FLOPs or runtime, rather than \"steps\" as in 3a, as these hide very different costs), it seems that Switch dominates DTS. For any FLOPs budget (vertical line), Switch offers better performance than DTS. Similarly, for any attainable performance level, Switch gets there at a lower FLOPs cost. Thus, I'm not sure how the paper justifies the use of DTS. I may have misunderstood something though. It's not clear to me why the DTS run (in red) was halted, and didn't continue training. There's a mismatch between both plots for the yellow curve (MoE-Dense), in the left one (3a), perplexity is below 14, while in the right one (3b), it's above 15. Why? Was the x-axis (FLOPs) trimmed?\n\nIn the previous to last paragraph in page 6, there's this sentence: \"DTS-Gate can obtain greater than 5% improvements in FLOPs-efficiency compared with state-of-the-art Switch-Gate [...]\". Where can this be seen?\n\nAlso, and as the authors acknowledge in the conclusions, the fact that DTS doesn't impose balance (if I understand correctly) even when using a load balancing loss (as opposed to having a maximum expert capacity that enforces it), can lead to an inefficient use of current hardware (some experts waiting for others).\n\nFor Figure 6, as expert id's are independent across layers, it would be nicer to sort experts per row according to their load, so that clearer patterns emerge.\n\nAlternative idea: One could try something even simpler. Initially train a dense model (with one MLP only, rather than E different MLPs + router) for a number of steps. Then, replicate the dense MLP to all the experts, and add a router from scratch, and apply top-k as usual. In other words, pre-train a dense backbone for a bit, and then initialize the expert model from this. Have the authors tried this? I can see this working well.\n\n\n**Typos:**\n\nSecond paragraph of page 6: \"set tokens of each sample as 1024\" -> \"set tokens' hidden size to be 1024\"?\n\n\"comparsion\" --> \"comparison\"\n\n\n**References:**\n\n[1] Scaling Vision with Sparse Mixture of Experts\n\n[2] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity", "summary_of_the_paper": "The paper proposes an algorithm, DTS-Gate, to warm-up experts in an MoE learning setup. Rather than applying standard top-k selection per input, the algorithm initially applies dense routing (i.e. k = total number of experts), to avoid quick collapse, and gradually relaxes it to become sparser (and cheaper). Experiments and ablations in language tasks are provided to support the advantages of the algorithm.", "summary_of_the_review": "While the algorithm is reasonable, the experimental results (only one plot, Figure 3b) do not suggest it provides any additional benefit compare to the standard baseline.\n\n------------\n\nNote: After the rebuttal, I increased the score from 3 to 5.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NhlDKsMoY5", "reviewer": "Reviewer_prfk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nThe proposed scheme to train dense gating then making it sparser makes sense and could potentially stabilize and improve the training of gating networks. \n\nThe authors conducted extensive experiments to evaluate both the effectiveness and efficiency of the proposed algorithm.\n\nWeaknesses.\nStabilizing the training of the sparse gating network has been the main challenge of the MoE based model architectures, there exists some research with similar ideas trying to make the training of MoE more robust. \n1. This paper uses gumbel noise to stabilize the learning of softmax gating. (https://arxiv.org/abs/1910.04915)\n2. This paper uses a binary gating with similar noise and a l0 regularizer to control the sparsity. (https://ojs.aaai.org/index.php/AAAI/article/view/3788)\n3. This paper uses a binary encoding and diffienable operators to smooth the learning of sparse gating, with initializing to be dense and later converging to sparse. (https://arxiv.org/abs/2106.03760)\n\nGiven these existing literatures, I feel like the novelty of this paper is quite limited. \n\nIn the meantime, since the sparsity of the gating is controlled by the annealing of temperature of the softmax, only Top-1 gating is supported, this also limits the capacity flexibility of the MoE based morel architecture. \n", "summary_of_the_paper": "This paper studied the problem of training the gating network for the mixture-of-experts based model architectures. To make the gating network training more stable and robust, the authors proposed a dense-to-sparse gating training algorithm that uses Gumbel noise and temperature tuning. The authors conducted experiments on large NLP datasets.", "summary_of_the_review": "Overall, I think the training scheme from dense to sparse for sparse MoE gating makes sense, and this is supported by experiments in the paper. However, I also think the technical novelty of this paper is limited given existing work with similar ideas on improving the gating network training.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CbBRAX6Jc2y", "reviewer": "Reviewer_hQQS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strong Points:**\n- using gradual sparsification for the MoE gates of a transformer model seems to offer a final performance (perplexity in this particular case) close to the MoE dense gates, while having lower computational requirements. At least in theory, as I believe that a binary mask is used to emulate sparsity. Is my understanding correct?\n\n**Weak Points and suggestions:**\n- The paper claims are not perfectly aligned with the empirical evaluation. If the proposed gradual sparsification with DST is intended to be for any MoE model than other neural network models besides transformers shall be studied. If the proposed method is particularly designed for transformer models then perhaps vision transformers shall be also studied. Based on the chosen direction, new algorithmic baselines and datasets have to be added for comparison. Otherwise, the broadness of the claims needs to be adjusted accordingly.\n- The related work discussion is missing very important work. Particularly, gradual sparsification for dense-to-sparse training has been introduced in [1], up to my best knowledge. From that moment, a large body of works has been released on this topic. In parallel, sparse-to-sparse training has been studied for vision transformer in [2]. Please add a consistent paragraph to discuss the above directions and to highlight clearly what is novel in this paper in comparison with those works. Otherwise, the proposed method, broadly speaking, would be just a simple application of [1] (or follow ups) on MoE gates for transformers. A recent survey can give more details on these topics [3].\n- Can you please add an algorithm to better illustrate the proposed method?\n- Is the lower bound from the “Adaptive Capacity” paragraph fixed across training? How can one choose it?\n- Why in Figure 3b, the dense model minimum PPL is higher than in Figure 3a? Can you train all models from Figure 3a for the same number of iterations? Now, somehow, it seems that the complete overview picture is missing\n- I suggest to carefully proof read the whole paper. The English usage and general appearance can be improved. Also, there are a number of typos, e.g., “Comparsion with…”. “…As shown in Figure 3.2….” \n\n**References:**\n- [1] Christos Louizos, Max Welling, Diederik P Kingma. Learning Sparse Neural Networks through 𝐿_0 Regularization. ICLR 2018, https://openreview.net/forum?id=H1Y8hhg0b \n- [2] Tianlong Chen , Yu Cheng , Zhe Gan , Lu Yuan , Lei Zhang , Zhangyang Wang. Chasing Sparsity in Vision Transformers: An End-to-End Exploration. NeurIPS 2021, https://arxiv.org/abs/2106.04533 \n- [3] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks, JMLR 2021, https://www.jmlr.org/papers/v22/21-0366.html\n", "summary_of_the_paper": "This paper introduces the idea of using dense-to-sparse (DTS) training gates in mixture-of-experts (MoE) based on a gradual sparsification process. The novel produced method is evaluated on a Transformer model. Overall, it looks to me that the paper has rather limited novelty. While the paper is well-written, in my opinion, the empirical evaluation support is not perfectly aligned with the paper claims.", "summary_of_the_review": "While there are some clear advantages highlighted by this paper in obtaining more efficient transformer models, overall, I believe that the paper is not ready yet for publication.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple approach to improve the robustness of training a sparsely gated mixture-of-experts model, which at a high level simply consists in training initially as a dense gated model, to better warm start a final phase of sparse training. Results are presented to highlight the potential benefits of this approach.

The authors have provided a detailed response and updated results, in response to the reviews. Each reviewer has also responded at least once to the author response. Despite that engagement, all reviewers are leaning towards rejection (though there is one reviewer with a rating of 6, they regardless state that "I'm confident this will make a great resubmission at a future venue", indicating they actually support rejection).

The reviewers point out that the proposed method is not really novel, pointing to an existing recent paper. Even without that prior work, I would also argue that the proposed approach is conceptually straightforward and has benefits that were fairly predictable and not particularly surprising. Given the generally lukewarm reception from the reviewers, I think there is a legitimate concern to be had here about this work's potential for impact.

Though the review process has definitely improved the paper's manuscript since its submission, I unfortunately could not find a reason to dissent from the reviewers' consensus that this submission is not ready to be published. Therefore recommend it be rejected at this time.
Decision: Reject