Taming Sparsely Activated Transformer with Stochastic Experts
Authors: Simiao Zuo
Xiaodong Liu
Jian Jiao
Young Jin Kim
Hany Hassan
Ruofei Zhang
Jianfeng Gao
Tuo Zhao
Venue: ICLR 2022 Poster (2022)

Abstract: Sparsely activated models (SAMs), such as Mixture-of-Experts (MoE), can easily scale to have outrageously large amounts of parameters without significant increase in computational cost. However, SAMs are reported to be parameter inefficient such that larger models do not always lead to better performance. While most on-going research focuses on improving SAMs models by exploring methods of routing inputs to experts, our analysis reveals that such research might not lead to the solution we expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to experts. In this paper, we propose a new expert-based model, THOR ($\underline{\textbf{T}}$ransformer wit$\underline{\textbf{H}}$ St$\underline{\textbf{O}}$chastic Expe$\underline{\textbf{R}}$ts). Unlike classic expert-based models, such as the Switch Transformer, experts in THOR are randomly activated for each input during training and inference. THOR models are trained using a consistency regularized loss, where experts learn not only from training data but also from other experts as teachers, such that all the experts make consistent predictions.  We validate the effectiveness of THOR on machine translation tasks. Results show that THOR models are more parameter efficient in that they significantly outperform the Transformer and MoE models across various settings. For example, in multilingual translation, THOR outperforms the Switch Transformer by 2 BLEU scores, and obtains the same BLEU score as that of a state-of-the-art MoE model that is 18 times larger. Our code is publicly available at: https://github.com/microsoft/Stochastic-Mixture-of-Experts.

Published as a conference paper at ICLR 2022
TAMING SPARSELY ACTIVATED TRANSFORMER WITH STOCHASTIC EXPERTS
Simiao Zuo Xiaodong Liu", Jian Jiao", Young Jin Kim", Hany Hassan', Ruofei Zhang", Tuo Zhaot and Jianfeng Gao
Georgia Institute of Technology Microsoft
{simiaozue tourzhao @gatech. edu, [xiaodl, jian. jiao, youki, hanyh, htt.a/angg..gggeeo/mmooroooooooo com
ABSTRACT
Sparsely activated models (SAMs), such as Mixture-of-Expee (MoE), can eas- ily scale to have outrageously large amounts parameters without significant increase in computational cost. However, SAMs are reported to be parameter in- efficient such that larger models do not always lead to better performance. While most on-going research focuses improving SAMs models by exploring meth- ods of routing inputs to experts, our analysis reveals that such research might not lead to the solution expect, i.e., the commonly-used routing methods based on gating mechanisms do not work better than randomly routing inputs to ex- perts. In this paper, we propose a new expert-based model, THOR (Transformer witH StOchastic ExpeR(s). Unlike classic expert-based models, such as the Switch Transformer (Fedus al., 2021), experts in THOR are randomly acti- vated for each input during training and inference. THOR models are trained using a consistency regularized loss, where experts learn not only from train- ing data but also from other experts as teachers, such that all the experts make consistent predictions. We validate the effectiveness of THOR on machine trans- lation tasks. Results show that THOR models more parameter efficient in that they significantly outperform the Transformer and MoE models across var- ious settings. For example, in multilingual translation, THOR outperforms the Switch Transformer by 2 BLEU scores, and obtains the same BLEU score as that of tath-offthe-art MoE model (Kim et al., 2021) that is 18 times larger. Our code is publicly available at: https:/ /ele:///gtttie /gihhuu......./ com/mmmmm//ooooooo crosoft/
INTRODUCTION
Large neural network models have shown to be effective in many natural language processing tasks such as machine translation (Lewis et al., 2020; Conneau & Lample, 2019), natural language un- derstanding (Devlin et al., 2019; Liu et al., 2019; He et al., 2020), and natural language generation (Radford et al., 2019; Brown et al., 2020). These models are usually densely activated. That is, model uses all its parameters to process all inputs. One drawback of these models is the prohibitive training cost. Moreover, the extreme size drastically reduces inference speed, further limiting the models' practicality.
To address these issues, sparsely activated models (SAMs, Shazeer et al. 2017) have been proposed. A SAM adaptively selects a subset of its parameters for different inputs during model training and inference. This makes it possible train SAMs that activated models without significant increase in computational cost. For example, the sparsely acti- vated GShard (Lepikhin al., 2020) consists of over 600 billion parameters and the Switch Trans- former (Fedus et al., 2021) 1.5 trillion parameters, while GPT-3 (Brown et al., 2020), which is arguably the largest densely activated model. consists only 175 billion parameters.
order of magnitude larger than densely
The building block of SAMs is the expert layer, which contains an attention mechanism and multi- ple feed-forward neural networks (FFNs) in parallel. Each FFN is referred as an expert. During
Work was done during an internship Microsoft.

---

Published as a conference paper at ICLR 2022
training, an input is routed to a fixed number of experts, such that the number of floating point op- erations (FLOPs) of one forward pass remains constant, regardless of the total number of experts. Thus, training SAMs is much more cost-efficient than training densely activated models. For exam- ple, training of Switch-large (Fedus et al., 2021) and that of T5-aage (Raffel et al., 2019) require the same forward FLOPs, despite that the former is 35 times larger (26.3 vs. 0.74 billion parameters).
However, SAMs have been reported to be parameter inefficient. For example, although the Switch- large model 35 times larger than T5-large, its performance on the GLUE benchmark (Wang al., 2019a) is only slightly better (88.5 vs. 87.8). There are also cases where the performance of SAMs even worse than smaller densely activated models. For example, the performance of Switch- large is worse than Ta-large on the ARC Reasoning Challenge (66.0 vs. 68.8) (Clark et al., 2018). In another example, although GShard (eepikhin et al., 2020) shows substantial gains over densely activated models, diminishing return with larger number of parameters has been observed.
Most on- on-going research has focused on improving SAMs by developing effective routing methods. Since only a subset of model parameters (i.e., experts) are updated for each input during training, we need to decide which experts to be activated given an input. Existing works (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2021; Yang et al., 2021) use gating network for input routing. However, the gating mechanism suffers from the notorious load imbalance issue: the gate's weight could collapse such that nearly all the inputs are routed to the same expert. Therefore, many methods are proposed to mitigate this issue, such as noisy gating (Shazeer et al., 2017), expert capacity (Lepikhin et al., 2020), load balancing loss (Lepikhin et al., 2020; Fedus et al., 2021), and k Top-1 gating (Yang et al.. 2021). However, these routing methods have not been proved effective to make SAMs more parameter efficient. To understand why SAMs are not parameter efficient, we analyze the performance of several classic MoE models. Our analysis reveals that a SAM does not always outperform a densely activated model of similar size, confirming the results reported in Yang et al. (2021). Moreover, we also observe that the widely-used routing method based on the gating mechanism does not work better than randomly routing inputs to experts,
Inspired by our findings, we propose new SAM, THOR (Transformer witH StOchastic ExpeRts). Unlike classic SAMs, such as the Switch Transformer, experts in THOR are randomly activated (with need of any gating mechanism) for each input during training and inference. THOR models are trained by minimizing both the cross-entropy loss and a consistency regularization term, such that experts can learn not only from training data but also from other experts as teachers so that all the experts make consistent predictions.
To validate the effectiveness of THOR, we have conducted extensive experiments machine translation using three settings: low-resource, rich-resource, and multilingual. Results show that THOR models outperform state-of-the-art MoE models by an average of 2 BLEU score twelve low-resource translation tasks. In the rich-resource setting. THOR achieves new state-of-thh-ar results on the two widely-used translation benchmarks, WMT' 16 En-De and WMT' En-Fr. On multilingual translation tasks, the THOR model with 300 million parameters achieves 2 BLEU score improvement over a state-of- of-the-art MoE model of the same size. Moreover, our model achieves state-of-the-ar results on these tasks- the same BLEU score that is achieved by the Z-code MoE model (Kim et al., 2021) with 5.5 billion parameters (18 times larger).
BACKGROUND
Transformer. The Transformer (Vaswani et al., 2017) model has demonstrated its superior perfor- mance in many peuuence-to-squeneee natural language processing tasks, such as neural machine translation. The model contains encoder and decoder. The encoder consists of multiple encoder layers, each having an identical structure, An encoder layer employs a self-attention mechanism and feed-forward neural network (FFN). The decoder is similarly constructed, except for an additional cross-attention mechanism in each decoder layer.
Sparsely Activated Models. The building block of SAMs is the expert layer, which is similar to the Transformer layer. Each of these expert layers contain an attention mechanism and multiple FFNs in parallel, where each FFN is referred to denotes the total number of experts. A gating mechanism decides to which expert(s) an input should be routed. At each expert layer, given an input vector I E RR, where d is the embedding dimension, the gate value of routing æ to expert E, is Pr(x: [Softmax (W (1
expert. Let ( ( N denote the experts, and N
2

---

Published as a conference paper at ICLR 2022
where W E R is the trainable weight matrix of the gating mechanism. Given the gate values {x)( f= we select the top-K experts to form an activated set of experts T C ... where
[T| = K. Then the output Zoot of the expert layer is
(l(())((T).
Fout
RE
Notice that in Eq. 2, input x only activates K instead of N experts, where K < e.g. K N = 2048 in GShard (Lepikhin et al., 2020). This implies that the number of FLOPs required for one forward pass does not increase with the number of experts N. Therefore, SAMs can scale to an enormous size without any significant increase in training time and inference time.
and
The gate weight matrix W (Eq. I) is trained together with the rest of the model parameters. Because there is no constraint on the learned weights, it is possible that W,, collapses such that one row dominates, i.e., all the inputs are routed to one expert. This problem referred to as load imbalance. Existing works adopt various ad- hoc heuristics mitigate this issue, e.g.. adding Gaussian noise to Eq. (noisy gating, Shazeer et al. 2017), limiting the maximum number of inputs that can be routed an expert (expert capacity, Lepikhin al. 2020), imposing a load balancing loss (Lepikhin et al., 2020; Fedus et al., 2021), and using linear assignment (Lewis et al.. 2021). There are other works that remove the gating mechanism such that load imbalance no longer an issue, e.g., by incorporating hash functions (Roller et al., 2021). Besides the load imbalance issue, there are also heated discussions on how to construct T Eq. For example, Shazeer et (2017); Lepikhin et al. (2020); Yang et al. (2021) conjecture that routing inputs to K > experts is necessary, while Fedus et al. (2021) argue that using K = 1 is sufficient and more computationally efficient.
ANALYSIS OF SPARSELY ACTIVATED MODELS
We investigate behavior of the gating mechanism of several classic MoE models. We conduct exper- iments on multilingual translation task, (De, Vi} - En. More details are presented in Appendix A.
We consider two MoE models proposed in Shen et al. (2019), referred to as MoE(dec) and MoE(tok), respectively, and three variants of the Switch Transformer proposed in Fedus et al. (2021). The number of experts set to two for all the MoE models. We compare them with the Transformer (Vaswani et al., 2017) model of the same model size.
Figure shows the validation losses and BLEU scores of three models: Transformer, MoE(dec), and MoE(tok). We see that the two MoB models perform very similarly, and neither outperforms the Transformer by a significant margin.
To interpret the results of Figure 1, we examine the load of each expert and the confidence scores of routing inputs to different experts. An expert's 's load is defined as the proportion of inputs that are assigned to it. For an input that is routed to an expert, its routing confidence score (output of the gating mechanism) determines the level of preference, e.g., if the routing confidence score is 0.5, then the gate has no preference for either expert. For each expert, compute the average routing confidence score over all the inputs assigned to it.
Figure 2 shows that after the early stage of training (i.e., the first 200 iterations). the gate weight collapses and nearly all the inputs are routed to expert 2. Also, the average routing confidence score of expert 2 is close 1.0, which means that the gate strongly prefers expert 2 to expert In this case, only one of the experts is sufficiently trained. Figure depicts a different scenario, where the inputs are randomly dispatched the experts. Notice that after approximately 4000 iterations, the two experts are equally loaded, and the probabilities of assigning any input to expert I and expert 2 are almost identical, indicating that the gating mechanism has no preference for either expert.
We have identified two behaviors of the gating mechanism: load imbalance and random routing. The former is also reported in recent papers (Shazeer et al., 2017: Lepikhin et al., 2020; Fedus al., 2021). We further investigate the Switch Transformer (Fedus et al., 2021), which is a state-of-the- art MoE variant that incorporates various methods to resolve the load imbalance issue. In addition, because behavior of the gating mechanism in the Switch Transformer mimics random routing (see Appendix A), we examine the effect of discarding the gate and randomly assigning inputs to experts. Figure 4 demonstrates the validation losses and BLEU scores of the Transformer and three variants of the Switch Transformer, where inputs are routed according to tokens (referred to as Switch(t)), sentences (Switch(s)), or are routed randomly (Switch(r)). Similar to the results in Figure 1,
3

---

Published as a conference paper at ICLR 2022
4.1
1.0
10-
4.0
0.9
3.9
32
0.8
Expert
Exper:
0.6-
Expert
Experi
31
0.6
0.2
BLEU
Transformer MoB(doe) Molstok)
30
100
2000
Dienations
3001
400
5001
0
100
200
Tienniions
300
400
500
Figure I: Validation results of MoE(dec) and MoE(tok).
Figure 2: Gating mechanism of MoE(dec). Left: average rout- ing confidence; Right: load of experts.
Exgert I
Expert Export
Eagert
2.88
Prdalbing
0.5-
4.0
0.7-
0.5-
32
0.6-
0.3
31
Loo
0.5
0.2
6
1000
2000
5000 Derraioos
3000
4000
6000
1000
2000 3000
Facrations
4000
5000 6000
Transformer Swisches)
Switchr
Swishhry
30
Figure 3: Gating mechanism of MoF(tok). Left: average routing confidence; Right: load of experts.
Figure 4: Performance of three vari- ants of the Switch Transformer.
see that the four models perform similarly. This shows that even after we alleviate load imbalance, model performance is not improved (i.e., the Switch Transformers do not outperform the vanilla Transformer), and the performance of the Switch Transformer does not vary much among different routing methods, including random routing.
We remark that in this paper, we focus on natural language processing tasks, particular neural ma- chine translation. There other works in different research fields (c.g., computer vision) that draw different conclusions than ours (Riquelme et al., 2021). We attribute this to the intrinsic differences between image classification and language generation, e.g., each input in the former belongs to clearly-defined category, while no such knowledge exists in the latter.
In summary, the experiments reveal
A sparsely activated model does not always outperform densely activated model of the same model size.
The widely-used routing method based the gating mechanism does not work better than randomly routing inputs to experts.
4 THOR: TRANSFORMER WITH STOCHASTIC EXPERTS
The ineffectiveness of the gating mechanism, as shown in our experiments, motivates us to propose new expert-based model, THOR (Transformer witH StOchastic ExpeRts). In THOR, pair of experts are randomly selected and activated in each layer during a training iteration, and then all the inputs in batch are processed using the same pair of experts. Our method drastically simplifies model design, and has two additional advantages. First, it eliminates the load imbalance issue be- cause randomly selecting pair experts each iteration allows each expert to have fair chance to be sufficiently trained. The ad-hoc heuristics, such as the load balancing loss, as discussed in Section 2, are no longer needed. Second, unlike the gating mechanism, THOR does not introduce any additional model parameters.
One problem of THOR is that without gating mechanism, experts need to be randomly selected during inference, and we may obtain inconsistent inference results due different random seeds. For example, on a Czech-to-Englias translation dataset, our experiments show that randomness can result in 0.5 BLEU score difference.
To address this issue, we introduce a consistency regularizer in the training objective of THOR. Concretely, let N denotes the number of experts, I. the number of layers, and F an activated expert

---

Published as a conference paper at ICLR 2022
Regularization
loss1
P1
Joss2
Softmax
Softmax
Add Normalize
Add Normalize
FEN
FEN
FFN
Normalize
Nonmalize
Self-Antetion
Self-Aneation
Embedding
Embedding
Batch
Figure 5: Illustration of a training iteration with stochastic experts. For conciseness, we show model with only one Transformer layer.
(which FFN) layer I, where 1 S i S N and S L. We use p f(x; (F)) to indicate the prediction probability of imput æ using the model f where experts {E are activated. Figure 5 illustrates one training iteration. Notice that instead of activating one expert for each layer in an iteration, we select to activate a pair of experts in THOR. As result, we obtain two prediction probabilities produced by the two selections, respectively: Pi f(x; (E) and P2 f(; { M=))) Then, the training objective of THOR with respect to training samples (ar, v) the dataset is
min Σ (#:
E(x,y) v) = CE(pi:)) + CE((2:))
aCR(p1:2))
1 where CR(p1:p)) 2 KK(palpp)))
Here, CE is the cross-entropy loss, the consistency regularizer CR is defined as the average of the two Kullback-Leibler (KL) divergence terms, and a is a hyper-parameter that controls the strength of the regularizer. In mini-batch SGD training, randomly sample a pair of experts to activate at each layer for each batch. During inference, we can also randomly select an expert to activate at each layer for each input, similar that in training. We can also use different expert-selection methods, such as expert-ensemble, as to be discussed in Section 5 (Table 5).
The THOR training objective of Eq. 3 forces all the experts minimize training emoos while mak- ing the same predictions as much as possible. Thus, in each training step, each expert optimizes its parameters by learning from both the training data (via minimizing the cross-entropy loss) and its paired expert as teacher (via minimizing the KL divergence). Although these experts are learned make consistent predictions, they converge to different (local) optima given the randomness in- troduced in training, e.g., initialization, mini-br mini-batch SGD, random routing, etc. Thus, every expert learns from a diverse teachers during the course of training, which helps improve model's performance. In addition, by penalizing experts that yield inconsistent predictions from the others, the consistency regularizer also helps reducing the variance of model prediction.
THOR is conceptually similar to dropout (Srivastava et al., 2014) since both methods route an input to some randomly selected sub-net components (i.c., experts in THOR and neurons in dropout). However, THOR differs from dropout in several important aspects, making it better choice for efficient training and serving of large-scale neural models. First, THOR can be applied to both training and inference, while dropout is only used for training. Second, THOR is shown to be more robust large-scale model training than dropout. For example, our models are less likely to overfit with the increase in the number of experts (see Figure 9). Third, THOR leads to sparse model
5

---

Published as a conference paper at ICLR 2022
that is more structured than that of dropout, such that large-scale THOR model can be much more easily trained using GPU clusters, e.g., by putting different experts on different GPUs in parallel.
EXPERIMENTS
We evaluate THOR on neural machine translation. We adopt three settings: low-resource trans- lation, rich-resource translation, and multilingual translation. For low-resource and rich-resource translation, we train all the models using Fairseq' (Ott et al., 2019). For multilingual translation, we use DeepSpeed MoE* (Kim al., 2021)10 to implement the MoE models. All the experiments are con- ducted on NVIDIA V100 GPUs. Additional experiments, including model scale-up and comparison of inference speed, deferred to Appendix D.
5.1 BASELINE
We use two baselines in the experiments.
Transformer (Vaswani et al., 2017) achieves superior performance in many sequence-to- sequence learning tasks, such as neural machine translation.
Switch Transformer (Fedus al., 2021) state-of-the-art MoE model, which employs gating mechanism to route inputs and uses load balancing loss to reduce load imbalance.
To verify the effectiveness of the imposed consistency regularizer in Eq. 3, we also compare THOR with Transformer models trained using two popular regularization methods. We remark that these two methods share similar computational costs with THOR, i.e., they also require two forward passes each training iteration.
SMART (Jiang et al., 2020) utilizes smoothness inducing adversarial regularizer to pe- nalize the worst case difference between predictions of a clean input and perturbed input.
R3F (Aghajanyan et al., 2020) uses regularizer reduce representational collapse. The method has shown to effective in various natural language processing tasks.
the methods are trained for the same number of FLOPs in experiments fair comparison.
5.2 LOW-RESOURE TRANSLATION
We use six language pairs: English to Vietnamese, English to German, and English to French from IWSLT; English to Romanian, English to Latvian, and English to Czech from Europarl?. Dataset statistics summarized in Table (Appendix B).
Table I: Experimental results on low resource datasets.
The best result on each dataset is in bold.
En- Vi
Vi-En
En-De
De-En
En-Fr
Fr-En
Transformer (Vaswani al., 2017)
31.3
29.4
28.1
34.8
39.2
38.1
SMART (Jiang et al., 2020)
32.5
30.5
29.3
35.8
40.0
38.8
R3F (Aghajanyan et al., 2020)
32.2
30.7
29.2
35.7
39.7
38.9
Switch (Fedus et al., 2021)
31.7
29.5
28.4
34.6
39.1
38.2
THOR
34.0
33.0
31.1
37.8
40.7
40.0
En-Ro
Ro-En
En-Ly
Lv-En
En-Cs
Cs-En
Transformer (Vaswani et al., 2017)
23.5
25.0
13.6
15.8
16.1
20.4
SMART (Jiang et al., 2020)
24.6
25.7
14.2
16.3
16.7
21.4
R3F (Aghajanyan et al., 2020)
23.8
25.8
14.4
16.3
16.8
21.6
Switch (Fedus et al., 2021)
23.8
24.4
13.8
16.1
16.1
20.6
THOR
25.2
27.1
15.2
17.4
17.6
22.4

---

Published as a conference paper at ICLR 2022
To evaluate THOR with different model sizes, 2017) architecture on Europarl datasets, and a smaller model on IWSLT datasets. Compared with Transformer- base, the smaller model decreases the hidden dimension from 2048 to 1024, and de- creases the number of heads from 8 to with the dimension of each head doubled. We use two experts for the expert-based models. We remark that though THOR increases the number of parameters, its inference speed (in terms of FLOPs) is the same as Transformer-bai because only one expert is activated for each input. Interested readers refer Appendix C for more details.
the Transformer-base (Vaswani et al.,
The experimental results in Table I show that performance of the Switch Transformer is on par with the vanilla Transformer, e.g., its average BLEU score on the 12 datasets 26.3, the same as the Transformer. The results confirm that SAMs do not outperform densely activated models with similar model sizes. In contrast, THOR achieves more than 1.0 BLEU score improvement over the Switch Transformer in all the 12 tasks. THOR also significantly outperforms the models trained using the two competing regularization methods, SMART and R3F.
5.3 RICH-RESOURE RESOURCE TRANSLATION
We two widely adopted rich rich-resource transla- tion benchmarks: English to German translation from WMT 16 and English to French translation from 1MT'14 The former dataset consists of 4.5 million training sentence pairs, and the latter 36 million pairs. We follow the pre-processing steps in Out et al. (2018).
Table 2: BLEU and sacreBLEU scores on WMT 14 En-Fr and WMT 16 En-De. Re- sults of Jiang al. (2020), Aghajanyan et al. (2020), and Fedus et al. (2021) are from our implementation.
BLEU
En-De
En-Fr
To evaluate THOR, We the Transformer-big architecture (Vaswani et al., 2017) and we set the number of experts for both THOR and the Switch Transformer to 4. Interested readers refer to Ap- pendix C for more details.
Vaswani et al. (2017)
28.4
41.8
Ott al. (2018)
29.3
43.2
Wang et al. (2019b)
29.6
Wu et al. (2019a)
29.7
43.2
So et al. (2019)
29.8
41.3
Jiang et al. (2020)
29.8
43.4
Table reports the BLEU scores and the sacre- BLEU scores (Post, 2018) different mod- els. We see that THOR achieves new state-of- the-art results in the setting where neither data augmentation nor pre-trained language model is used. Specifically, THOR lifts the previ- ous state-of-the-arr (Liu et al., 2020b;c) by 0.3 BLEU score on the En-De translation task and 0.1 BLEU score on the En-Fr translation task. THOR also significantly outperforms the models trained using the other two regularization meth- ods, SMART (Jiang et al., 2020) and R3F (Agha- janyan et al., 2020). Similar to what observed in low-resource translation, the Switch Trans- former (Fedus et al., 2021) does not outperform the vanilla Transformer (Ott al., 2018).
Wu et al. (2019b)
29.9
43.3
Aghajanyan et al. (2020)
29.4
43.3
Liu et al. (2020c)
30.1
43,8
Fedus et al. (2021)
29.3
43.0
THOR
30.4
43.8
sacreBLEU
En-De
En-Fr
Ott al. (2018)
28.6
41.4
Jiang et al. (2020)
29.1
41.5
So al. (2019)
29.2
Aghajanyan et al. (2020)
29.0
41.5
Liu et al. (2020c)
29.5
41.8
Fedus et al. (2021)
28.6
41.1
THOR
29.6
41.9
5.4 MULTILINGUAL TRANSLATION
We have collected 10 language pairs from WMT datasets, and built a 64k-entry dictionary for all the languages. The detailed statisties are summarized Table (Appendix B). Please refer Kim et al. (2021) for more details. We do not use multi-task learning or additional monolingual data in the experiments.
We use the following model architecture: the embedding dimension is set to 768 and the hidden dimension for the FFN set to 3072; we use encoder layers and decoder layers, where each layer has 12 attention heads, and the dimension of each head is 64. We set the number of experts to 4 for both THOR and the Switch Transformer.
Table 3 reports the average BLEU score of translating English to other languages, translating other languages to English, and the overall score the 20 tasks. We see that compared with the Switch

---

Published as conference paper at ICLR 2022
Transformer of the same size (i.e., 300 million parameters), model achieves 2-point improve- ment in the overall BLEU score. In addition, our model is far more parameter efficient than the Switch Transformer. The THOR model with 300 million parameters achieves the same BLEU score (24.4) that is achieved by the Switch Transformer with 5.5 billion parameters, which is more than 18 times larger.
Table 3: Multilingual translation results, Here "E" means the number of experts.
En ~Others
Others =En
Average
Switch (32E, 5.5B)
24.4
Switch (4E, 300M)
20.3
24.6
22.4
THOR (4E, 300M)
21.4
27.4
24.4
Figure 6 shows BLEU scores in all the 20 translation tasks. Notice that THOR outperforms the baseline on 17 out of the tasks. The improvement is in general more significant on the tasks with smaller datasets. For example, our model achieves BLEU score improvement of 4.7 and 6.7 on Gu-En (85k) and Hi-En (264k), respectively. On the tasks with larger datasets, improvement ob- tained by our model is less substantial, but still significant, e.g., +0.9 BLEU score on Cs-En (10M) and +1.1 Fi-En (4.8M). For the only three tasks where our model underperforms the baseline, the gaps are small, c.g., -04, -0,, and -0.4 BLEU scores En-Cs, En-De, and En-Fr, respectively.
30
Swich
THOR
25
10
Bn-Cs
En: De
En-Et
En-Fi
En-Pr
En Ga
En Hi
En-Ly
En Ro
Fa-Tr
35
30
BL.EU
20
Figure Details of multilingual translation results.
5.5 ABLATION EXPERIMENTS
Training Objective. We examine the relative contributions of the three loss terms used in the THOR training objective of Eq. 3: CF₁, CF2 and CR. The result in Table 4 shows that the con- sistency regularizer CR is crucial to the model performance, and that dropping one of the two CE terms leads to only very small BLEU score loss since the two cross-entropy terms play the same role training.
Inference Methods. We compare three inference methods: (1) D Dispatch (s) uses sentence- level random routing, where all tokens in one sentence are routed to the same expert; (2) Dispatch (t) uses token-level random routing, where tokens within a sentence are routed to different experts; (3) Ensemble, where each sentence is routed to all the N experts, and the N hidden representations in each layer are averaged. Note that the number of FLOPs is larger for Ensemble e because need to run forward pass for each input through N experts. Table 5 shows that Di sisattch ch (s) and Dispatch (t) perform similarly, and Ensemble yields the best BLEU score with cost of longer inference time.
8

---

Published as a conference paper at ICLR 2022
Table 4: Effect of the three loss terms in training object of Eq. 3,
Table 5: Performance and costs of three inference methods, tested on Cs- En translation.
tested on Cs-En translation.
Loss terms
BLEU
BLEU
time
CE₁ CE2 CR
22.4
CE + CR
22.2
Di Dispatch (s)
22.4
x
CE₁ CE2
20.8
Di Dispatch (t))
22.4
x
CÉ₁
20.6
Ensemble
22.6
X
20.50
22.5
38.0
22.0
37.5
22.00
21.75
21.5
37.0
Trarsfomerr
21.50
21.0
36.5
THOR
2125
21,00
20.5
36.0
20.75
0.0
2.0
5.0
100
20.0
Swisch
THOR
35.5
Figure 7: Effect of the consis- tency regularization strength on Cs-En translation.
Figure 8: Violin plot of per- formance consistency Cs- En translation.
Dimensenilaperts
Figure 9: BLEU vs. model size on De-En translation.
Regularization strength. To investigate the effect of the regularization strength a, periments on the Cs-En translation dataset in the low-resource setting. Figure 7 shows that model performance not very sensitive to a long the value is large enough, say a 2.0.
Consistency of Model Prediction. We study the variance of model prediction due to the use of randomly activated experts during inference. We compare THOR and the Switch Transformer, where we remove the trained gate during inference. For each model, we compute the variance of model prediction based on 20 runs. As shown Figure THOR makes more consistent predic- tions than Switch Transformer due to the use of the consistency regularizer for model training. The variance THOR is below 0.002, whereas the variance of Switch Transformer 0.008, four times larger. We remark that by removing the trained router from the Switch Transformer, model perfor- mance only marginally decreases (from 20.6 to 20.4). This further indicates that a trained router may not be better than random router.
Overfitting. We compare the THOR model and the Transformer model regarding how likely they overfit the training data when the model size increases. We experiments the De-En data in the low-resource setting, where the dropout rate of the FFNs the Transformer is selected such that the number of parameters trained in iteration is the same the THOR model. As shown in Figure 9, THOR does not show any sign of overfitting BLEU score we increase the number of experts from 2 to 8. In contrast, the Transformer model's performance deteriorates we increase the hidden dimension of its FFN from 2k to 8k. We remark that we also observe the overfitting phenomenon on larger datasets, e-g., the Transformer overfits on the Cs-En dataset when set the hidden dimension of its FFN to 16k.
we observe consistent improvement in
6 CONCLUSION
We present a new expert-based sparsely activated model, THOR. Unlike existing SAMs, such as the Switch Transformer, experts THOR are randomly activated for each input during training and inference. THOR models are trained using consistency regularized loss, where every expert learns not only from training data but also from other experts as teachers so that all the experts make consistent predictions. As result, not only can large-scale THOR models be trained and served as efficiently as classic MoE models, THOR models also demonstrate a better generalization capability in that they more parameter-efficien less likely to overfit, make more consistent predictions, and achieve better results consistently across different settings. We validate the effectiveness of THOR via comprehensive empirical study machine translation. In all the three settings (i.e., low-resource, rich-resource, and multilingual translation), THOR models significantly outperform the vanilla Transformer, and Switch Transformer, state-of-the-ant MoE model.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
We thank Rukmini Lyer, Kevin Duh, Hao Cheng, Chunyuan Li, Johannes Gehrke, colleagues from Microsoft Bing Ads team and Microsoft Research for their valuable discussions and comments.
REFERENCES
Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. ArXiv preprint, abs/2008.031556 2020. URL https::/ http/aaavvvv/..////////..//..avl org/
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: 4457c0d6bfcb4967 (1tpa//c000..1111000010141400400
http:/gs...eiiisss.cc./e//////// wwtpc/pppe///////eeeeeeeeeeeeeee Abstract html.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv preprint, abs/1803.055
2018. URL https: /ttp://aaav.v//ae//ea/eeeeeaaela org/ abs//bs//// / 8038 05457.
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Bus Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 1057-7067, 2019. httptttss///pprceeeteee www.innnn..nn..nneeeeeeeeeeeeee. cc/paper/ 2019 9 c0 el A0A.0000002222210000113733333333 htt.s//rcc.ccccccccoaaaaaaaaaaaa cibstract.htm.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199/ 1423. URL https: tttts://aclanthollee
org/N19 1423.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. ArXiv preprint, abs/2101.03961 2021. URL https:
//arxiv. abs /2101. 03961.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding- bert with disentangled attention. ArXiv preprint, abs/2006.03654, 2020. URL https: /arxiv. org/ 2006. 03654.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled reg- ularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, pp. 2177-2190, Online, 2020. Association for Computational Linguis- URL https: / aaaclaann awl.nnhhloggyyoo.oocomoooooollle org/2020.
tics. doi: 1/20655//122202alllmmm. ael-main.197.
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andres Felipe Cruz Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari. Yuxiong He, and Hany Hassan Awadalla. Scalable and ef- ficient moe training for multitask multilingual models. ArXiv preprint, abs/2109.10465 2021. URL https::////...........tp / / /arxivv www.v.ww.///.oooot.....w....wwow abs 221 2109....00. 11.0091114000 10.
10

---

Published as a conference paper at ICLR 2022
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015. URL http: // //arxiv..rg/aabbib
6980.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. ArXiv preprint, abs/2006. 2020. URL https:
/ //arxiv.org// aww.aaaaaa/// 200. 16668.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequencc pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 7871-7880, Online, 2020. Association for Computational Linguistics. doi: 10. https:
aai.aa.....mmm................ URL
http:///aaaaattloooo/oe /
org/ acl-main. 703.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. ArXiv preprint, abs/2103.16716, 2021. URL
http://rrgabba///ooo /ttp//bb////111..111 attpss://axxiee
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- niew.net, 2020a. URL https: open revi ew. net.e/eww.eeeeeeeeeeeeeeeeeeeeee seet/foorme perrummii=rgg22aEE...
Liyun Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the dif- ficulty training transformers. Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 5747-5763, Online, 2020b. Association for Computational Linguistics. doi: 00.18653/v1/2020 111166//1122220.mmmmmmaaaii emnlp-main.l6663 URL https:// acl allanhhollgg logy- wowg////00. 2200 emn poomlp-mnn m--mal 433.
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural ma- chine translation. ArXiv preprint, abs/2008.07772. 2020c. URL https://aaxiv..rrg /bs/ 2008. 07772.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907. 1692, 2019. URL https: httppps:///axv.../// org/ ab9999990 11692.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 1-9, Brus- sels. Belgium, 2018. Association for Computational Linguistics. doi: 18.15/ 11133/1//11661/11 URL httph:t///// httpps:/aacaantholggg... org/ //18--1-- 6301.
Myle Out, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minneapolis, Minnesota, 2019. Association for Computational Lin- guistics. doi: 10. 10.18653/11/N19-4000
URL https::tp............ a antho .og...ooo..oooooo.oo.......... org /N1 9
org/N1
Matt Post. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Papers, pp. 186,191, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10. 10.118553/11/V 18 6319.
URL https: http:////caaaahhooooo/oo
org/W1 8- 6319. 99
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to- text transformer. ArXiv preprint, hbbs/1910.10683 2019. URL https:/ /arxivv. org/ abs/ 1110 1.000.00660 10683.

---

Published as a conference paper at ICLR 2022
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. arXiv preprint S.S1v:2106.0597720 2021. URL penttss:/arxivv
http:///www..oo//////////////// wttpg//////......///////// 05974.
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large URL https://arxiv. org/ abs/
sparse models. ArXiv preprint, abs/2106. 04426, 2021.
2106. 04426.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisticy (Volume I: Long Pupers), pp. 1715-1725, Berlin, Germany, 2016. Association for URL https: /aclaannnonoooy.
Computational Linguistics. doi: 10.18653/v//P16 1162.
hog/P16-1162
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture- of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL /ppprevi/ppeeevviwww net SauuuidddBccMDLLL.
Tianxiao Shen, Myle Out, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5719-5728. PMLR, 2019. URL http: http::/pproceedinne mwr. press/v97/ shen 9c. html.
David R. So, Quoe V. Le, and Chen Liang. The evolved transformer. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5877-58876 PMLR, 2019. URL http: htttt:://proceed.ngg. mlr. httes//v7/ssos sol 9a. html.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-198, 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shiens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 2818-2826. IEEE Computer Society, 2016. doi: 10.1109/CVR.2206.3000 URL https: //doi. org. 10. 1109/CVPR.... 2016.00130000 308.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. http/paper/22111 2017/hahh/ 5998 6008, 2017. URL https: httttp:://prooeedinngs neuripe - Abstract html. 3f55e2235477dee
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019a. URL https: /wwp//eveeeeeeeeee/e//e net forum?id= T54.m2
Qiang Wang. Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F, Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1822, Florence, Italy, 2019b. Association for Computational Linguistics. doi: 10.18653/v 1/P19-11766.11111111111111111111 URL https: //acl/lclcc woatthooog...oo wtw.y.rgg/1991110000
12

---

Published as a conference paper at ICLR 2022
Felix Wu, Angela Fan, Alexei Baevski, Yann N. Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In 7th International Conference on Learning Rep- resentations, ICLR 2019, New Orleans, LA. USA, May 6-9, 2019. OpenReview.net, 2019a. URL www.rreiiieweee.. net / forum? d=skV 9tt..
Lijun Wu, Yiren Wang, Yingee Xia, Fei Tian, Fei Gao, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. Depth growing for neural machine translation. Proceedings of the 57th Annual Meer- ing of the Association for Computational Linguistics, pp. 3558-5563 Florence, Italy, 2019b. Association for Computational Linguistics. doi: 10.18653/v1/P19-1155 1558. URL https:// ac eclanthology www./oggg///.... P19558.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia. Ang Wang. Jie Zhang, Jiamang Wang, Yong Li, et al. Exploring sparse expert models and beyond. ArXiv preprint, abs/2105. 15082, 2021. URL http:p/s//wwwv.//////e///
110..1101..1.2... 15082.
13

---

Published as a conference paper at ICLR 2022
A ANALYSIS OF SPARSELY ACTIVATED MODELS
A.1 TRAINING DETAILS
We consider two Mixture-of-Experts (MoE) models proposed in Shen et al. (2019), which are de- noted "MoE(dee)" and "MoE(tok)". In the first variant, each expert is a separate Transformer de- coder. In the second variant, each expert is different token, i.e., if route the input to expert one, then we replace the (bos) (begin-of -feetenn--eetence token in the imput sentence with a (expert) token. Note that embeddings of these expert tokens trained together with the rest of the model parame- ters. These models are equipped with an expectation-mxxiiizaaa framework facilitates computing the probability of assigning an input to specific expert according to the gating mechanism. Please refer to Shen et al. (2019) for details about these models.
optimization framework. Such a
We use multilingual translation setting, where adopt datasets: De-En from IWSLT" 14 and Vi-En from IWSLT'15 15. For each dataset, we use byte pair encoding (BPE, Sennrich et al. 2016) with 10, 000 merge operations for pre- pre-processing. Then we concatenate the two pre-processed datasets. We learn a separate dictionary for En and (De+Vi), which resulted in approximately 9k and 12k vocabularies, respectively.
For training, we use Adam (Kingma & Ba, 2015) as the optimizer and we set the learning rate to 0.001. We set the batch size to equivalent 64k tokens, e.g., we use 8k tokens per GPU with GPUs. Other training details follow the Fairseq" implementation. For inference, we use beam size of and length penalty 1.0.
A.2 ADDITIONAL RESULTS
We also plot the average routing confidence score and the load of experts for Switch(s) and Switch(t), similar to Figure 2 and Figure 3. We first investigate the Switch Transformer without the load balancing loss.
Expert
Expert
0.80
Expert 2
0.60
Expert 2
Probability
0.55
0.70
Load
0.50
0.45
0.60
0.40
0.50
0.35
20000
40000
40000
80000 60000
Iterations
Iterations
Figure 10: Switch(s) w/o load balancing. Left: average routing confidence; Right: load of experts.
0.65
Expert
Expert Expert
0.80
Expert 2
0.60
Probabliity
0.55
0.70
Load
0.50
0.65
0.45
0.60
0.55
0.40
0.50
0.35
10000
20000
30000 40000 50000
10000
20000
30000
40000 50000
Iterations
Iterations
Figure 11: Switch(t) w/o load balancing. Left: average routing confidence; Right: load of experts.
htttps://githubb.cmm http://wwweaaaaaaaae//////aa/tta
14

---

Published as a conference paper at ICLR 2022
Figure 10 shows the results for Switch(s) without the load balancing loss, where we route inputs to experts on the sentence-level. We see that after about 10k training iterations, the average routing confidence score of expert and expert 2 becomes similar, and both of these scores around 0.60. Moreover, the load of the experts are not balanced, i.e., there is a 10% difference in the loads (55% 45%). We conclude that behavior of the gating mechanism of Switch(s) is similar Figure 3, i.c., the gate is essentially randomly routing inputs to experts without any preference.
Figure 11 shows the results for Switch(() without the load balancing loss, where we route inputs to experts on the token-level, i.e., different tokens within the same sentence may be routed to different experts. Similar to the Switch(s) case, the average routing confidence score of both of the two experts converges to around 0.55. This indicates that the gate do not prefer any expert given an input. Moreover, the load of the experts are not balanced, the same as in Figure 10. Based on these observations, we conclude that behavior of the gating mechanism of Switch(t) is also random routing.
0.504
Expert Expert 2
0.80
0.503
0.502
Probubility
0.70
Loud
0.501
Expert Expert
0.500
0.499
0.60
0.498
0.497
0.50
0.496
10k
15k
20k
25k
30k
35k
10k
15k
20k
25%
30k
35k
40k
Iterations
lterations
Figure 12: Switch(s) w/ load balancing. Left: average routing confidence; Right: load of experts.
0.85
0.504
Expert Expert
0.503
Expert Expert
0.75
0.502
Probabiity
0.70
0.501
Lood
0.500
0.499
0.60
0.498
0.55
0.497
0.50
0.496
5000
10000
15000 20000 25000 30000
100000 1,5000 25000 30000
Iterations
Iterations
Figure 13: Switch(t) w/ load balancing. Left: average routing confidence; Right: load of experts.
Figure 12 and Figure 13 show behavior of the gating mechanism of Switch(s) and Switch(t) equipped with the load balancing loss, respectively. We see that the load balancing loss indeed balances the load for both Switch(s) and Switch(t), e.g., there is less than 0.4% imbalance for Switch(s) and less than 0.2% imbalance for Switch(t). In comparison, the imbalance is around 10% for the two Switch Transformer variants without the load balancing loss. Also, similar the case without the load bal- ancing loss, the average routing confidence score converges to around 0.60 for Switch(s) and around 0.55 for Switch(t). Based on the observations, we conclude that behavior of the gating mechanism is still random routing when Switch(s) and Switch(t) are equipped with the load balancing loss.
B DATASETS
Statistics of low-resource datasets are shown in Table 6. The English-Vietnamese Vietnamese, English-German and English-Frende Frenchh datasets are from' IWSLT' 14, '15, and '16, respectively. The training data of
15

---

Published as a conference paper at ICLR 2022
English- Rominian, English-Latvian and English- are from Europarl and the validation and testing data are from WMT'
Statistics and data sources used in the multilingual translation task are shown in Table 7.
Table 6: Statisties of low resource translation datasets.
En-Vi
En-De
En-Fr
En-Ro
En-Lv
En-Cs
Train
117,055
160,239
218,256
390,746
591,631
619,029
Validation
5,098
7,283
8,453
1,900
1,949
2,902
Test
1,268
6,750
1,133
1,999
2,001
3,005
Table Statisties of multilingual translation datasets. The other language the translation tasks English (En) for all the datasets.
Language
Czech (Cs)
German (De)
Estonian (Et)
Finnish (Fi)
French (Fr)
Data source # Samples
WMT 19
WMT' 19 4,613,192
WMT 18 695,227
WMT 19 4,838,576
WMT' 15
10,273,696
9.999.995
Language
Gujarati (Gu)
Hindi (Hi)
Latvian (Lv)
Romanian (Ro)
Turkish (Tr)
Data source
WMT' 19
WMT 14 264,199
WMT' 17 1,444,235
WMT' 16 540,562
WMT
# Samples
85,688
182,269
C TRAINING DETAILS
C.I LOw RESOURCE TRANSLATION
We build a joined dictionary for the source and target languages for each dataset. To facilitate this, byte pair encoding (BPE) with 10, 000 and 40, 000 split operations for the IWSLT and the WMT datasets, respectively, Other pre-processing steps follow the Fairsey implementation.
5.0. We set the batch size to be equivalent to 32k tokens, i.e., if we have four GPUs, then we set the number of tokens on each GPU be 4k and accumulate gradients for two steps. We use Adam the optimizer with 3 0.9, 82 = 0.98, and we set the learning rate to be 0.0015. We train the model for 40k steps, and we test the model that yield the highest validation BLEU. For validation and testing, we use beam size and length penalty 1.0. Other training and inference details follow the Fairseq implementation.
For training, the regularization strength is chosen be
C.2 RICH RESOURCE TRANSLATION
Strength of the consistency regularizer set as a 2.0. We use Adam (Kingma Ba, 2015) as the optimizer with 33 0.9, 8 0.98, and the learning rate is chosen 0.001. For inference, we use a beam size and length penalty 0.6 for En-De; we use a beam size and a length penalty 1.0 for En-Fr. Other post-processing steps follow Ott et al. (2018). We report both the BLEU score and the sacreBLEU score (Post, 2018), where the latter is safer akknstigootiic version of BLEU.
C.3 MULTILINGUAL TRANSLATION
For training, set the batch size to be equivalent to 1.6 million tokens, e.g., 4096 tokens per GPU with 24 GPUs, and we accumulate gradients for 16 steps. We use RAdam (Liu et al., 2020a) as the optimizer with parameters 31 0.9 and B3 0.98. The learning rate is set to be 0.05. Also, set the dropout ratio to be 0.1, and we use label smoothed cross entropy (Szegedy et al., 2016) with smoothing factor 0.1. The regularization strength set to be 4.0. For inference, size 5 and a length penalty 1.0.
beam
16

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTS
We further test behavior of THOR and the Switch Transformer when we increase the number of experts. To avoid overfitting, translation dataset. In this experiment, the Transformer model has 48M parameters, models with 2, 16, and 64 experts have 55M, 143M, and 456M parameters, respectively.
a small model (Transformer-WWSLTT) the WMT 16 En-De
Transformer
26.2
Trunsfomer
Switch-2
BLEU (validaatinn)
26.0
Switch
ppl (training)
10-
Switch-16
THOR
Switch-64
25.8
9
25.6
8-
25.4
25.2
24
48
72
96
120
144
168
2
16 Number of Experts
64
Wall-time
Figure 14: Effects of the number of experts on WMT 16 En-De translation. Left: training perplexity (lower the better) with respect to wall-time (measured in GPU hours); Right: validation BLEU (higher the better) after training for 180 GPU hours with respect the number of experts, where the size of Transformer does not change.
Figure 14 demonstrates the results. In Figure 14 (left), notice that the Switch Transformer trains faster than the vanilla Transformer, and this scaling property is more significant when we increase the number of experts.
From Figure 14 (right), we that with experts, the Switch Transformer behaves slightly worse the vanilla Transformer in terms of validation BLEU. However, when we increase the number of experts, performance of the Switch Transformer continues improve and outperforms the vanilla Transformer with the same number of FLOPs. This indicates that in order for sparsely activated model to outperform densely activated one, we need scale the former contain much more parameters than the latter. Our observations are consistent with existing literature (Lepikhin et al., 2020; Fedus et al., 2021). For example, in Fedus al. 2021, the sparsely activated Switch-base outperforms the densely activated T5-base using the same number of FLOPs. However, the former more than 30 times larger (7.5 billion vs. 0.22 billion parameters).
Our method is more parameter efficient than the conventional methods. From Figure 14 (right), we see that THOR significantly outperforms the vanilla Transformer and the Switch Transformer even with only 2 experts. Moreover, when we increase the number of experts, performance of THOR also improves.
We also compare inference speed of Transformer, Switch Transformer, and THOR in Table 8. Note that for THOR, we use the Di chspttch (s) method in Table Note that the inference speed of Switch Transformer and THOR is slower than the vanilla Transformer because of the computation and communication overhead induced by input routing. Such an overhead is more noticeable when the number of experts large. We remark that in Fedus al. 2021, the speed of Switch-base is about half of T5-base (780 vs. 1600 samples per second).
Table 8: Inference speed (tokens/second).
Transformer
Switch
THOR
# experts
2
16
64
2
16
64
Speed
15.2k
15.0k
10.4k
7.4k
15.1k
10.6k
7.5k
17


Reviews:
[{"review_id": "_MtI0w3G_oA", "reviewer": "Reviewer_PQMm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1 The paper is well written with good structure\n2 The method is concise and novel to some degree.\n\nWeakness\n1 The number of experts is limited to 2. Indeed it would maintain the computation load. But the power of experts may not be exhibited sufficiently. It would be great to investigate the performance by changing the number of experts.\n2 I am wondering if this mechanism can be used in the vision transformers? ", "summary_of_the_paper": "In this paper, the authors propose to equip transformer with stochastic experts ( i.e., a number of FFN layers in parallel)  to boost model capacity without increasing much computation. Moreover, they propose a consistency regularization between a pair of experts in the training, which can alleviate performance deterioration of random expert selection in the inference.\n", "summary_of_the_review": "Basically, it is an OK paper for me. The novelty and experiments are good to show effectiveness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_4yXU_uIaHB", "reviewer": "Reviewer_zyWV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies sparse models in the context of deep learning: neural networks that only activate a subset of all parameters depending on the input. This is a field that is gaining lots of attention recently, as models have grown large, expensive and inference time has increased accordingly.\n\nSo far, most works [1, 2, 3] learn a router (usually a linear map + softmax + maybe noise) to assign tokens to experts. This approach has some downsides (lack of differentiability, expert collapse that requires additional auxiliary losses, etc). The paper proposes a new approach that simply samples the experts at random --and enforces some coherence among them. \n\nSection 3 presents some experiments related to standard gating mechanisms, Section 4 presents the algorithm, and Section 5 shows the experimental results.\n\nI have concerns in the three sections.\n\n**Section 3.**\n\nThe number of experts is 2 --and I assume the model selects 1--, which seems maybe too few to me, in order to see generalizable effects.\n\nIf I understand Figures 1-3 correctly, two MoE models were trained (MoE(dec) and MoE(tok)). The first one collapses (only Expert 2 ends up being used) in Figure 2, and the second one seems to reach an equilibrium, in Figure 3. I think the interpretation provided by the paper for Figure 2 is generally wrong and misleading. The fact that both experts are used equally often (i.e. load ~ 0.5 in both cases), and the fact that the *average* routing confidence is 0.5 does *not* imply random routing is happening, as suggested in the paper. If uniformly random routing was happening, we'd see this. However, there are good routing schemes that would lead to this averaged outcome too. To give a simple example, suppose we have a dataset containing sentences about sports and about animals, half and half. The router may send all sports tokens to Expert 1 with weight 1.0, and all animal tokens to Expert 2 with weight 1.0. The average gating weight is 0.5 for each expert, and they get half the tokens. The routing, however, is very far from random.\n\nI'd not agree with the general statement (and, more importantly, I don't think this paper contains enough evidence to back it up): \"The widely-used routing method based on the gating mechanism does not work better than randomly routing inputs to experts\".\n\nThere are published experiments (see [3], Figure 27, layer 21) that replace a trained router with a random router and, as expected, performance goes down. Of course, this does not imply that if routing is random while training, final performance will be worse (but we'll end up with an ensemble of similar experts). Moreover, there's evidence that, under learned routing, experts specialize (even in a human understandable way), see [3] Figure 7. These facts should probably be acknowledged in the paper. Also, note that even fixed routing [4] is fundamentally different from random routing, it still allows for expert specialization if done well.\n\n**Section 4.**\n\nThe proposed algorithm seems to weaken the sparse models advantage, that is, having lots of capacity which the model can specialize per input, while keeping cost kind of constant. By randomly selecting experts in every batch (+ explicitly adding a consistency loss), we are forcing all experts to learn the same thing --and how to deal with all possible inputs/tokens--, which directly counteracts the sparse models core idea. In other words, we are training a computationally-cheap ensemble. This can still be a very good model, nonetheless; but I'd be surprised if it beats a properly tuned MoE transformer (more on this in the next section). As mentioned before, there's evidence that experts specialize [3].\n\nAlso, I was wondering if there's concrete evidence to support this claim in page 5: \"Although these experts are learned to make consistent predictions, they converge to different (local) optima given the randomness introduced in training [...]\". I'd assume there's such type of statements in the ensemble literature.\n\n**Section 5.**\n\nI have some fundamental concerns with the experimental section. If my understanding of the experiments is correct, the comparison between THOR and the Switch Transformer [2] is not fair. Basically, the number of epochs is fixed (or say, datapoints seen by the models during training, batch size * number of steps). However, THOR applies two times the backbone network to each datapoint, leading to a massive mismatch in terms of total FLOPs or runtime (with equivalent hardware).\n\nIn order to have a meaningful comparison, all the tables in the paper should include two additional columns: total training FLOPs and total runtime. This will unhide compute differences across models.\n\nSwitch uses k=1 (i.e. one expert selected per token). Note that even for k=2 (as used in [1, 3]) previous sparse models would be cheaper (and significantly so) than THOR, as all the common --non-experts-- layers are only applied once, regardless of k. One baseline that must be in the paper tables is a Switch model that runs for a FLOPs- or runtime-matched amount of steps, that is, it probably runs for around twice as many steps. I'm open to change my score/reviews if those experiments are added, and are still favorable to THOR.\n\n\nAnother couple of comments regarding experiments. First, for non-ensemble methods to shine (i.e. Switch), probably more than 2/4 experts would help. Using so few experts probably helps mitigate the lack of specialization effect, and benefits the ensemble-like THOR approach. \n\nSecond, I don't understand the following sentence at the end of section 5.3: \"Similar to what is observed in low-resource translation, the Switch Transformer does not outperform the vanilla Transformer\". Looking at Table 2, it seems BLEU for Switch is (quite?) better than for Vaswani et al (i.e. \"vanilla\" Transformer)?\n\n\n\n[1] - GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding\n\n[2] - Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity\n\n[3] - Scaling Vision with Sparse Mixture of Experts\n\n[4] - Hash Layers For Large Sparse Models", "summary_of_the_paper": "The paper proposes a new routing mechanism for sparse models in the context of language tasks. Rather than learning a parametric router that learns how to assign tokens to experts, the proposed algorithm (THOR), randomly selects two experts per mini batch, and applies those experts independently to every input. A consistency loss is used to force experts to provide similar predictions. A number of experiments are provided suggesting the algorithm outperforms previous works.", "summary_of_the_review": "Experiments devote significantly more compute to THOR than to Switch. Accordingly, it is hard to interpret the results.\n\n------\n\nI've raised my score given the authors rebuttal and updated results (3 --> 5).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "M4DYajomwYz", "reviewer": "Reviewer_PXVZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is very well written and clear, proposing the THoR method, including ablations on aspects of its design, and performing three classes of evaluations, comparing to other methods, reporting that THoR outperforms all other methods and establishes at least two state-of-the-art results on WMT. Fundamentally, I am more than willing to believe the key results which show that THoR outperforms Switch Transformer. \n\nHowever, MoEs are complex models, for which it is very easy to subtly apply an incorrect comparison, or to evaluate improperly. There are several results in the paper which claim that Switch Transformer underperforms or just equals the performance of a dense, standard transformer. I find this difficult to reconcile with the results of the Switch Transformer paper [1], which not only claims that Switch outperforms dense models, but has several results (Figure 1A, and Appendix D) specifically saying increasing the number of experts from 1 to larger numbers monotonically improves performance. A general result which has been mirrored in other recent MoE-like papers (Figure 3 in [2], Figure 6 in [3], Figure 4 in [4]). This inconsistency makes me skeptical of the hierarchy of methods suggested by this paper.\n\nFundamentally, the main request of this rebuttal is that the authors address this inconsistency between their MoE/Switch results and those reported by others. I have three concrete asks:\n\n* Are methods FLOP-matched when being compared, or step-matched? My read of sections 4 and 5 are that methods are step-matched, but this seems substantially unfair since THoR (and the other regularization methods) require twice the compute per step. I believe a proper comparison across all the datasets and evaluators should be at a fixed number of FLOPs (i.e., with Switch/Dense running for double the number of steps). In line with this request, the inclusion of validation curves showing the performance of THoR, Switch and Dense Transformers as training progresses would be very valuable.\n\n* Are the authors able to reproduce Figure 1A from [1]? An exact reproduction isn't necessary, differences in model details or dataset are fine, but it would be extremely valuable to know whether or not the author's implementation of Switch mirrors the result that increasing the number of experts should increase the performance of the network. If yes, what is your explanation for Switch not outperforming a Dense transformer on some evaluations? If no, do you have an explanation for the difference? Does THoR exhibit improvement with more experts?\n\n*  This paper uses just 2 or 4 experts for the majority of its experiments. While Switch does already claims performance improvements with so few experts, it is also much smaller than the normal regime which is tested, with number of experts in the 16-128 regime being standard [1, 2, 3, 4, 5]. What do the initial experiments, as well as the later Switch vs THoR evaluation comparisons, look like with a larger number of experts? Does the comparison hold up?\n\nIf the authors address these three points and either explain or reconcile the difference in their reported Switch performance with those of other papers, then I would have no problem recommending an accept (8). In the current form, I feel this paper is borderline due to these unanswered questions.\n\n\n\n\nAdditionally, I have several smaller nits and comments:\n\n* **[General]** There are several recent methods which claim to improve over switch ([2] and [5] come to mind). I don't necessarily think you need to compare against them, but as the claimed performance improvement is high, it seems important to acknowledge.\n\n* **[General]** The authors prefer the term SAM, but this is a very general term (there are lots of different sparsely-activated models out there) and this term doesn't seem standard in the literature. Maybe something more specific would be more apt?\n\n* **[Section 2, \"Transformer\"]** \"the model\" should be replaced with \"our model\", not all transformers are encoder/decoder models.\n\n* **[Section 2, \"Sparsely Activated Models\"]** A nit: Hash Layers don't help solve the load balance issue, they completely obviate the need for a routing layer (also, from Section 4, worth noting that similar to THoR, Hash layers also don't need to introduce new parameters).\n\n* **[Appendix A.2]** I think the conclusions you're reaching are too certain. While it is true that random routing is a possibility based on these results, it is not a foregone conclusion (what manifests as a slight preference in the router might be key). A conclusive experiment you could run would be to replace the routing decisions *only at evaluation time* by random decisions, and see how much the evaluation performance of the networks are impacted.\n\n* **[Section 4]** The idea that THoR proposes, that experts should be similar and consistency-regularized towards each other, has a very reasonable interpretation. However, it is directly in opposition to the standard interpretation (that experts should be diverse). I think it would be good to discuss this difference more.\n\n* **[Section 4, \"different experts on different GPUs in parallel\"]** Can you elaborate on this? I understand expert parallelism, but a key aspect of expert parallelism applied to most MoE models is the expectation that for any training minibatch, we expect to be using all experts more or less equally (which means we can share the compute equally across a bunch of different devices, each with their own experts). In THoR, for any given step we're fixing the number of experts we use to two, so I don't see how this can be effectively parallelized across more than 2 devices (without replication).\n\n* **[Section 5.4, Table 3]** Having numbers from a dense transformer would be useful here.\n\n* **[Section 5.5, Figure 8]** Switch should have input jitter disabled at evaluation time (Figure 15 in [1]). This should make the network at eval deterministic. What is the explanation for the variance seen in this figure?\n\n\n[1] *Switch Transformer* https://arxiv.org/abs/2101.03961\n\n[2] *Hash layers* https://arxiv.org/abs/2106.04426\n\n[3] *GShard* https://arxiv.org/abs/2006.16668\n\n[4] https://arxiv.org/abs/2109.10465\n\n[5] *BASE Layers* https://arxiv.org/abs/2103.16716", "summary_of_the_paper": "The paper proposes THoR, an approach towards training MoE-like models (called SAMs in the paper) that have multiple internal experts which are chosen in a discrete fashion. Unlike other common methods, which generally incorporate a form of router which maps input to an  specific router choice, THoR makes random, global, per-minibatch decisions on experts, and includes a cross-expert regularization term to help align experts to each other. This paper reports that THoR outperforms other methods, including the recent Switch Transformer MoE-style model, on several classes of multilingual translation tasks.", "summary_of_the_review": "The paper is well written and the proposed ideal seems novel. But my inclination to recommend accept (8) is tainted by some doubts about the primary comparison given with Switch Transformer.  I've outlined three key questions/additions which could assuage these doubts.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EzeSYBdAkbX", "reviewer": "Reviewer_hGSm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is easy to follow and the proposed methods are simple and novel. The experiments and ablation study are extensively conducted. The results look promising and the method should be easy to be adopted. \n\nI have a few questions and comments.\n\n1. In Table 2, Transformer-base (Vaswani et al.,2017) architecture is used for Europarl datasets, while a smaller model is used on IWSLT datasets. I am wondering why not apply both model architecture to both datasets. \n\n2. In Table 2, it seems to me that the size of Switch transformer is larger than transformer-base but they have same inference flops? However, I also see this statement `The results confirm that SAMs do not outperform densely activated models with similar model sizes.` in Sec. 5.2. So I am wondering if the model size of switch transformer is same as the size of transformer-based model in Table 2. \n\n3. In page 5, the paper states `During inference, we can also select a pair of experts to activate at each layer for each input, similar to that in training.`. I thought only one expert is activated during inference to gain the inference speed. Also, if two experts are selected, how the final predictions are selected? Is the average embedding used? \n\n4. The paper mostly compares the inference FLOPS. I am wondering if the authors can provide the real inference time for transformer-base model and the THOR model (with same FLOPS) in both batch inference mode and real time inference mode. ", "summary_of_the_paper": "This paper proposed to use randomly selected experts for Mixture-of-Experts models instead of gating function based selection methods. To avoid large performance variance of the random selection during inference, the paper proposed to add a consistency regularization which drives the similarity between different experts. The experimental results show its superior performance over existing MoE models such as switch transformers. ", "summary_of_the_review": "The paper presented a simple and novel MoE method. The experiments and ablation study are extensively conducted. The results look quite promising and the method should be easy to be adopted. I vote for acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors introduce a simple mixture-of-experts model, by greatly simplifying the routing mechanism: experts are randomly activated both at train and inference time. A consistency loss function is added for training the proposed models, enforcing all experts to make consistent predictions. The proposed method, called THOR, is evaluated on machine translation tasks, including multi-lingual MT, and outperforms the recently proposed Switch Transformer MoE.

The reviews note that the paper is well written and easy to follow, and that the proposed method is simple. While the results look promising, the reviewers also raised concerns regarding comparisons to previous work, some of which were addressed in the rebuttal. Finally, a reviewer raised the concern that this method is related to ensembles, which work well for machine translation, but are not discussed or compared to. For these reasons, I believe that the paper is borderline, leaning toward acceptance.
Decision: Accept (Poster)