Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers
Authors: Tianlong Chen
Zhenyu Zhang
AJAY KUMAR JAISWAL
Shiwei Liu
Zhangyang Wang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Sparse Mixture-of-Experts
Random Routing
Transformer Training
Dropout

Abstract: Despite their remarkable achievement, gigantic transformers encounter significant drawbacks, including exorbitant computational and memory footprints during training, as well as severe collapse evidenced by a high degree of parameter redundancy. Sparsely-activated Mixture-of-Experts (SMoEs) have shown promise to mitigate the issue of training efficiency, yet they are prone to (1) $\textit{redundant experts}$ due to representational collapse; and (2) $\textit{poor expert scalability for inference and downstream fine-tuning}$, primarily due to overfitting of the learned routing policy to the number of activated experts during training. As recent research efforts are predominantly focused on improving routing policies to encourage expert specializations, this work focuses on $\textit{exploring the overlooked scalability bottleneck of SMoEs}$ and leveraging it to effectively $\textbf{scale dense transformers}$. To this end, we propose a new plug-and-play training framework, $\textbf{SMoE-Dropout}$, to enable scaling transformers to better accuracy in their full capacity without collapse. Specifically, SMoE-Dropout consists of a $\textit{randomly initialized and fixed}$ router network to activate experts and gradually increases the activated expert number as training progresses over time. Transformers trained by SMoE-Dropout naturally exhibit a $\textbf{``self-slimmable”}$ property subject to resource availability, offering smooth and consistent performance boosts with an increase in activated experts during inference or fine-tuning. Our extensive experiments across diverse transformer architectures on a variety of tasks demonstrate the superior performance and substantial computation savings of SMoE-Dropout, compared to dense training baselines with equivalent parameter counts. In particular, our trained BERT outperforms its densely trained counterpart with consistent improvements of {$1.03\%$, $0.78\%$, $1.09\%$} on challenging reasoning tasks {$\texttt{ASDiv-A}$, $\texttt{MAWPS}$, $\texttt{SVAMP}$}, respectively. Codes and models are available in https://github.com/VITA-Group/Random-MoE-as-Dropout.

SPARSE MOE AS THE NEW DROPOUT: SCALING DENSE AND SELF-SLIMMABLE TRANSFORMERS
Tianlong Chen'r, Zhenyu Zhang' Ajay Jaiswal', Shiwei Liu', Zhangyang Wang' VVITA Group, University of Texas at Austin
{t Itianleng.o chen, zhangu.zhan, a jayjjaissseee jaiswal, shiwei ,iu, aelaswargeeutexxs.cee
ABSTRACT
Despite their remarkable achievement, gigantic transformers encounter signifi- cant drawbacks, including exorbitant computational and memory footprints during training, as well as severe collapse evidenced by high degree of parameter re- dundancy. Sparscly-activated Mixture-of- Expertss (SMoEs) have shown promise to mitigate the issue of training efficiency, yet they are prone to (1) redundant experts due to representational collapse; and (2) poor expert scalability for in- ference and downstream fine-tuning, primarily due to overfitting of the learned routing policy to the number of activated experts during training. As recent re- search efforts are predominantly focused on improving routing policies to encour- age expert specializations. this work focuses on exploring the overlooked seala- bility bottleneck of SMoEs and leveraging it to effectively scale dense transform- ers. To this end, we propose a new plug-and-play training framework, SMoE- Dropout, to enable scaling transformers to better accuracy in their full capac- ity without collapse. Specifically, SMoE-Dropout consists of a randomly ini- tialized and fixed router network activate experts and gradually increases the activated expert number as training progresses over time. Transformers trained by SMoE- -ropout naturally exhibit a "seef-slimmable" property subject to re- source availability, offering smooth and consistent performance boosts with an increase activated experts during inference or fine-tuning. Our extensive exper- iments across diverse transformer architectures a variety of tasks demonstrate the superior performance and substantial computation savings of SMoE- OMoo-Dropout, compared to dense training basclines with equivalent parameter counts. In par- ticular, our trained BERT outperforms its densely trained counterpart with con- sistent improvements of (1.03%, 0.78%, 1.09%} challenging reasoning tasks (ASDiv-A, MAWPS, SVAMP) ), respectively. Codes and models are available in httpps///ggchbbb..com////araoooo
https://githhii com /wtp///IIAAAa www../////.../..o////o/o/ Randemn Bareldo--------rrpppaa Droprut
INTRODUCTION
Scaling neural networks, historically with the blessing of modern hardware, have dramatically im- proved the state-of-the-an boards, conforming to the empirical scaling laws (Kaplan et al., 2020), where the final model quality has been found to have pawer-law relationship with the amount of data, model size, and compute time. Transformers (Vaswani et al., 2017), 7),, swiftly after their introduction, have become de facto choice for many natural language processing (NLP) (Yang et al., 2019c: Liu al., 2019b; Talmor et al., 201 8; Jaiswal et al., 2021; Yang et al., 2019b; Wang et al., 2018 8; Ding et al., 2019; Chowdhery al., 2022; Wei et al., 2022) and computer vision (Dosovitskiy et al., 2020; Han al., 2020; Tou- vron et al., 2021; Mao et al., 2022: Zheng et al., 2021; Parmar et al., 2018) applications and now their parameter counts are typically measured in billions rather than millions. Unfortunately, this exploita- tion of parameters actuates roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase especially for dense advanced transformer-basee (e.g., BERT (Devlin et al., 2018) and GPT (Brown et al., 2020)) and require thousands of GPU days for training. Additionally, these gigantic transformers suffer from the representation collapse issue during vanilla training, which is affirmed by high degree of parameter redundancy (Guo et al., 2019: Ganesh et al., 2020; McCarley et al., 2019) and observed ineffective usage of the transformer expressiveness (Michel et al.. 2019; Chen et al., 2022a).
a wide array of real-world machine learning applications and leader-
models
"Equal Contribution.

---

Sparse Mixture-of -fxturr-ff-Expeets (SMoEs) enable effi- cient scaling of model capacity at a fixed compu- tational cost by performing input-dependent condi-
1.28
Densely Training w. Dropout
1.26
Training w. Learnable SMoE
Training w. SMoE. -roopout
Bite Per. Cha..Chhrntter (BPC)
1.24
tional computing. Such property facilitates train-
ing transformers with significantly high parame- ter counts at moderately increased cost, compared to their dense counterparts, resulting in improved training efficiency. For instance, with similar train- ing FLOPS. Switch- Large (Fedus al., 2021) (a kind of SMoE) is 35 larger than TS-Large dense model (Raffel et al., 2020). Despite their advantages in mitigating computational and en- ergy footprints, SMoEs have many critical limita- tions. Firstly, the current learning-based routing mechanisms in SMoEs tend to push hidden repre- sentations clustering around expert centroids (Chi et al., 2022), implying a trend toward represen- tation collapse, which in turn leads to redundant experts, inferior expert specialization, thereby sub- standard performance (Mittal al., 2022; Chen et al., 2022b). Secondly. SMoEs suffer from poor scalability during inference and downstream fine-tuning prominently due to overfitting of the learned routing policy to the number of activated experts during training. Naive solutions to mitigate such sparsity immutability often lead to perfor- mance degradation. As recent research efforts for SMoES are predominantly focused on improving routing policies to encourage expert specializations, we explore the overlooked scalability bottleneck of SMoEs and ask: Does there exist a principled and pluggable approach to modify SMoE training that can enhance scalability at inference and downstream fine-tuning of large-scale transformers, by dynamically adapting the number of activated experts subject to resource availability?
1.22
1.20
1.18
1.16
1.0
1.2
1.4
1.6
1.8
2.0
Pammeter Count (X10")
Figure 1: Bits-Per-Character (J) on enwik8's
test-set with a 1-layer Transformer-Xi SMoF- Dropout demonstrates "self-xlimmable" property where inference performance is smoothly boosted along with the inerease of activated parameters. Learnable SMoEs tend to overfit certain levels of network capacity. Note that only gray curve is pro- duced by (5) different dense models.
To this end, this paper proposes novel alug-and-play training framework, named -MoE-Dropont to enable scaling transformers to better accuracy in the full capacity setting without collapse. More specifically, OMoE-Dropout employs a fixed router network that is randomly initialized to activate experts and progressively increases their number as training progresses over time. Our simple, yet highly effective strategy has a multi-fold win-win for trained transformers, specifically: 1 obtain- ing "self-slimmable" property during inference and downstream fine-tuning subject to resource availability, which delivers a once-for-all in-situ trade-off between efficiency and performance; @ mitigating representational collapse and effectively utilizing the full model capacity, where activat- ing more experts produces superior performance (Figure 1 (blue)); @ eliminating the overhead of learning routing policies for SMoE. Note that SMoE-Dropout can be swiftly adapted for training any deep learning network (e.g. CNNs), given some splitting techniques (Zhang et al., 2021). but this work primarily focuses on transformers considering their exploding computational footprints. Our innovative contributions can be summarized as:
We propose a new plug-and-play training framework, SMoE-Dropout, to enable scaling transformers in the full capacity setting without collapse. SMoE-Dropout facilitates the randomly and sparsely activated structure of network modules, playing an implicit regu- larization role similar to dropout. Our new framework leads to enhanced generalization and reduced training costs (e.g., up to 37% running time savings) compared to the vanilla training of large dense transformers at equivalent parameter counts.
Transformers trained by S.MoE-Dropout naturally exhibit a "self-slimmable" property that displays smooth and consistent performance boosts when increasing activated experts dur- ing inference or fine- fine-tuning (Figure 1 (blue)). This property enjoys an "in-situ" trade-off between efficiency and performance at deployment, subject to resource availability.
* Our extensive experiments across representative architectures on a variety of tasks vali- date the effectiveness of our proposed SMoF-Dropout. Specifically, during pre-training. our approach has {1.37, 4.10). (2.53, 12.44] and (154.12, 188.00) (x10-2) lower BPC than {vanilla dense training (with the same parameter counts), learned SMoEE for Transformer-XL. BERT, and RoBERTa, respectively; after transferring, SMoE- Dropout obtains f0.07%, 1.03%, 0.78%. 1.09%) performance improvements for BERT and {- 3.88%, 0.07%, 5.04%} for RoBERTa, on (CSCA, ASDiv-A, MAWPS, SVAMP} reasoning tasks compared to its dense training counterpart.
2

---

Point-wise Addition
Summation
N
Multi-Layer Perceptron
Lalti- Pereeptron
Expert
Expert
Expert
Expert
Splitting
Along with Training
Point wise Multiplicazion
Normalization
Expoot
Rorport
wase Addition
Export
Expert
Random Router
k
k
Export
Export
Multi-Head Attention
...
Token Embedding
Normalization
Expert
Expert
Transformer Layer
Modulization
SMoE-Dropout
Gredually Increased
Figure 2: Overview of proposed SMoE-Dropout. Left describes the standard transformer layer, consisting multi-head attention and multi-layer perceptron (MLP) components. Middle Left shows the process mod- ulization. splits the original MLP evenly and constructs a series of experts which are smaller MLPs with a reduced hidden dimension. Middle Right presents the overall procedure of SMoE-Dropont selects the top-k experts given token embedding and then reweights the features from activated experts. In the end, summation is conducted aggregate all features. Right displays the gradually increased numher of chosen experts, along with the training procedure.
The random router
2 RELATED WORKS
Mixture of Experts (MoE). MoE special kind neural network, where its parameters are partitioned into series of sub-modules (commonly referred to experts), and conditional compu- tation is then performed an input-dependent fashion (Jacobs et al., 1991; Jordan & Jacobs, 1994; Chen et al., 1999; Yuksel al., 2012). The traditional dense MoEs computationally intensive, as they adopt all experts for each input (Eigen et al., 2013). Fortunately, recent investigations (Shazeer al., 2017; Lepikhin al., 2020; Fedus et al., 2021) have proved the effectiveness of MoEs with sparsely activated experts (i.e.. SMoE) at both training and inference stages, which greatly trim down the cost and scale language models to enormous sizes like trillions of parameters (Fedus al., 2021). This efficient fashion of SMoEs gains increasing popularity in various NLP (Shazeer et al., 2017: Lepikhin et al., 2020; Zhou et al., 2022: Zhang al., 2021: Zuo al., 2022: Jiang al., 2021) and vision (Riquelme et al., 2021; Eigen et al., 2013; Ahmed et al., 2016; Gross et al., 2017; Wang al., 2020; Yang et al., 2019a; Abbas Andreopoulos, 2020; Pavlitskaya al., 2020) tasks.
However, its sparse-gated manner incurs several downsides, including: (1) Unstable training. Zoph et al. (2022) pointed out that while techniques like gradient clipping can stabilize SMoE training, they often result in lower quality. The router 2-loss (Zoph al., 2022) is preferred solution for achieving both improved performance and stability. (2) Poor specialization. One of the intriguing goals of SMoE to divide-and-comquee the learning task by solving each piece of the task with adaptively selected experts (Aoki et al., 2021; Hazimeh et al., 2021; Ma et al., 2018; Mittal et al., 2022). To encourage specialization and decrease redundancy among experts (Chen al., 2022b), Dai et al. (2022) pre-defined the expert assignment for different input categories, while Hazimeh et al. (2021) advocated multiple, diverse router policies. (3) Representation collapse and load im- balance among experts. As the primary issue of learning-ber based SMoEs, various approaches have been proposed to mitigate their negative effects. Shazeer et al. (2017) injected Gaussian noises into gating networks promote the routing balance. Later, Lepikhin et al. (2020); Fedus al. (2021) applied an auxiliary loss of load balancing regularizers; Lewis et al. (2021) performed the routing by dealing with a linear assignment problem; Clark et al. (2022) utilized reinforcement learners; Zhou et al. (2022) routed top-k inputs per expert instead of selecting top experts per input sample. Beyond learned routing policies, Roller al. (2021) and Zuo al. (2022) designed deterministic hashing and stochastic assignments, respectively, which eliminate the necessity for router networks.
Zuo al. (2022), one closely related prior work, also endorsed the advantage of stochastic expert as- signment. They randomly activate experts for each input during training and inference, which leads
3

---

to inconsistent inference results. To address the prediction randomness, Zuo et al. (2022) employed consistent regularized loss to penalize the discrepancy among different experts. However, such reg- ularization is prone to redundancy in SMoEs and sacrifices the network capacity. In proposal, the fixed router with random weights generates deterministic inferences. Meanwhile, the presented "self-slimmable" attribute suggests the full models' expressiveness is adequately exploited.
Dropout and Other Training Techniques for Transformers in NLP. Dropout (Srivastava al., 2014) was developed to prevent overfitting in over- wver-parameterzzed networks during training. by ran- domly omitting neurons and their corresponding connections. Follow-up studies develop plenty of dropout variants (Zhang & He, 2020; Wan et al., 2013; Ba & Frey, 2013; Kingma et al., 2015; Gal al., 2017; Wu & Gu, 2015; Tompson et al., 2015; DeVrrie & Taylor, 2017; Park & Kwak, 2016; Semeniuta et al., 2016). In parallel, McAllester (2013); Mou et al. (2018); Mianjy & Arora (2020); Zhang & Xu (2022); Neklyudov al. (2017); Gal & Ghahramani (2016) have devoted themselves in deriving the theoretical foundation for dropout and explaining its implicit regularization impacts.
Other notorious bottlenecks of transformer training primarily stem from overfitting and instability caused by poor optimization (Zhang et al., 2020; Liu et al., 2019a; 2020a), insufficient or heteroge- neous downstream data (Varis & Bojar, 2021; Zhang & Vaidya, 2021), etc.. Accordingly, numerous remedies are developed to address the issues. For example, data augmentations (Sun et al., 2020), improved initialization (Liu et al., 2020b;a; Xu et al., 2020: Zhu et al., 2021), upgraded normal- ization (Wang et al., 2022; Yang et al., 2022), enhanced optimizers (Cohen et al., 2022), weight decay (Loshchilov & Hutter, 2017), and early stopping.
3
METHODOLOGY
3.1 PRELIMINARY
Sparse Mixture of Experts (SMoEs). SMoE models leverage conditional computation to activate different subsets of network for different inputs. A building block of SMoEs is the expert layer including a multi-head attention block and multiple experts in parallel. In this work, we consider SMoE for Transformers, where SMoE layers are incorporated into contiguous Transformer blocks. SMoE expert can be normally constructed by either splitting the vanilla MLP of transformers into smaller pieces (Zhang et al., 2021) or replicating the MLP (Fedus et al., 2021). Most existing SMoE works mainly concentrate on the MLP component in transformers since MLPs constitute roughly 2/3 of total model parameters counts storing substantial amounts of learned knowledge as memory networks (Geva et al., 2020; Dai et al., 2021).
Let (E denote the experts, where is the index of expert and N is the total number of experts. A gating network router R is inserted to choose the top-k experts with the largest scores R(x)); and æ represents the input embedding. Usually, h < N. which implies a sparsely activated setting. Specifically, the resultant output of the expert layer can be depicted as follows:
y Σ R(x), Rx)) TopK(softmax(((a)) k); TopK(e, k)
v u is the top
0 otherwise
where G is the critical part of a router R. For learnable routing, G is neural network that can be one few layers MLP (Shazeer et al., 2017: Fedus et al., 2021). (x)) stands for features from the expert E It will be further summed with a scaling coefficient R(x), to form the final output y. The Topk function maintains the largest k values and sets the reset elements to zero. In practice, a load or important balancing loss (Shazeer et al., 2017) is employed to avoid the representation collapse issue, ie., always picking the same experts for different inputs and ignoring others.
Dropout and its variants. Dropout is a conventional training technique employed to alleviate the risk of overfitting. The vanilla dropout typically applied to fully connected layers with dropping probability p. During each training iteration, neurons will be disabled with the probability p. In other words, the omission of neurons follows a Bernoull 1 L (p) distribution. As for the inference phase, there is no dropout and all neurons are activated. To counterbalance the surplus information during training, the output logits are reweighted by 1 - p. In this paper, we selected two representatives among diverse proposed dropout variants, concrete dropout (Gal et al., 2017) and dropblock (Ghiasi et al., 2018) as our comparison baselines.
Concrete Dropout It replaces the discrete Bernoulli0y distribution of dropout with a tinuous relaxation, i.e., Concrete distribution, and allows an automatic tuning of the dropping probability p. For example, considering the one-dimensional case, as shown in Gal et al. (2017),
4

---

Concrete random variable x is described as z sigmodd? x (log(p) - log((1 - p) + log(u)
log(1 - (og(1- where u Uni 1) is a uniform random variable and t denotes a temperature hyperparameter. Note that parameter is optimized in a data-driven way.
DropBlock. Instead of performing Bernoulli dropping per feature map, Ghiasi al. (2018) applies it in areas within feature maps. They claim that DropBlock improves the generalization and limits overfitting by hiding certain areas of features or input samples.
3.2 A NEW TRAINING PIPELINE: SMOE-DROPOUT
Modulization. The first step in our 3MoE-Dropout, turns a large densely connected MLP into multiple smaller MLPs with the same size, as demonstrated in Figure 2. Without loss of generality, in Figure 2, we use a single-layer MLP f with a dimension d for illustrations. After the modulization, it is divided into set of MLPs E2....... {
where they have the same hidden dimension $- Random Routing Policy. Few prior works have investigated some form of random routing poli- cies, such as Roller et al. (2021) utilizes a hash table to enforce a pre-defined deterministic random mapping from inputs to experts and Zuo et al. (2022) adopts a fully random assignment in each training iteration. Although they have shown some benefits from random policies, both methods suffer from inconsistent inference predictions, and can not outperform the densely trained models with equivalent parameter counts. In contrast, our proposed framework, SMoE-Dropout considers a randomly initialized and fixed router network to guide token assignment. Different from previous works, our proposal's 's assignment is (1) implicitly optimized during training. since feature embed- dings remain updated for the same input sample; (2) deterministic during inference thanks to the fixed weights in R. Extensive results in Section 4 verify the superiority of our proposal, compared to existing random policies and the dense baseline with the same model parameters.
Additionally, another crucial design in EMoE-Dropour routing is the progressively enlarged num- ber of activated experts (k). Riquelme al. (2021); Jiang et al. (2021) reveal that altering k in the inference phase incurs significant performance degradation if the SMoE is learned with a fixed k. For example, (Riquelme et al., 2021)'s 's SMoE trained with k has 20% ~ 30% accuracy drops on ImageNet, when activating & 2 7 experts during the evaluation. This drawback substantially restricts the practical use of SMMEs because diverse real-world scenarios require different resource budgets, necessitating flexible and effective network capacity during inference. To tackle this lim- itation, we adopt a training strategy that gradually enriches the active network capacity by linearly increasing the number of selected experts k during training. This approach coincides with the prin- ciple of curriculum learning and provides the attractive "self-slimmable" ability, which consistently boosts performance for transformers as the number of activated experts increases during inference and downstream fine-tuning, as shown in Figure!
SMoE-Dropout. Our effective proposal comprises three simple and highly effective steps, as de- scribed in Figure 2. First, divides the MLP into a series of MLPs with a reduced size for moduliza- tion (Middle Left of Figure 2). Then, random policy parameterized by fixed weights is introduced to route token embeddings to k experts with the largest response (Middle Right of Figure 2). Finally, progressively actives more experts, preventing the overfitting to the amounts of used network ca- pacity during training. (Right of Figure 2).
4 EXPERIMENT
4.1
IMPLEMENTATION DETAILS
Network Architectures and Comparison Baselines. In our experiments, we have adopted three representative wransformer-base networks, including BERT (Devlin et al., 2018), Transformer- XL (Dai et al., 2019), and RoBERTa (Liu et al., 2019b). Specifically, we use double-size BERT base / RoBERTapase that have 12 transformer layers. 768-dimensional encoder layers, 6144-/3072- dimensional feed-forward networks (MLPs), and 12 attention heads. For both Transformer-Xi we choose a reduced size due to limited resources, which has 4 layers, 256-dimensional encoder layers, 8192-dimensional feed-forward networks, and 8 attention heads with a head size of 64.
For sufficient comparisons with our proposal, Training w. SMoE-Dropout, we consider five base- lines: (i) Densely Training w. Dropout, where the vanilla dropout is applied to feed-forward net- works (MLPs); (62) Densely Training w. Concrete Dropout (Gal et al., 2017); (táz) Densely Training w. DropBlock (Ghiasi et al., 2018). Note that both Concrete Dropout and DropBlock are inserted in ferd-forward networks, replacing the vanilla dropout; (iv) Training w. Learnalle SMoE (Fedus et al., 2021); (v) Training w. THOR (Zuo et al., 2022), where THOR is another random SMoE
5

---

that randomly activates a pair of experts for each imput sample and adopts auxiliary consistency regularization based on Kullback-Leibe Leibler (KL) divergence. To compute the regularization term, two forward processes needed in each training iteration.
Pre-Training. > Datasets. Transformer- XL is pre-trained on enwik8 (Mahoney, 2011) dataset, while we use BooksCorpus (Zhu et al., 2015) for BERT and RoBERTa, >> Training Configura- tions. For Tra we follow the official training setups, using Adam optimizer and the learning rate starts from 2.5 x 10-4 and decreases according to a cosine annealing scheduler. We use a batch size of 22 and optimize the network for 4 10 iterations. As for BERT pre-training, we adopt an AdamW optimizer with an initial learning rate of 5 x 10-5 that linearly decays to O. The batch size and total training steps are 64 and 1 x 10', respectively. RoBERTa's pre-training config- urations strictly follow the default from BuugingFacee but with reduced training steps of 1 x 10" Moreover, we conduct a grid search and set the coefficient of THOR's regularization term 2. Similarly, the temperature in Concrete dropout is = 0.1. 1> Evaluation Meirics. Since both perfor- mance and efficiency are essential, we assess the pre-tra pre-training performance via Bits-Per- Per-Character (BPC) on the hold-out validation set, where a smaller BPC value indicates a better pre-training; and we report training time per iteration the number of floating point operations (FLOPs) of single- sample inference, for evaluating the efficiency. (1 RTX A6000, batch size 22) and {8 V100, batch size 64) are adopted for time measurements of Transformer-XL. and BERT/RoBERTa, respectively.
Datasets. Five benchmarks across three downstream tasks amined in this paper, including text classification (SST-2 2 (Socher et al., 2013)), arithmetic reason- ing (ASDiv-A A (Miao al., 2020), MAWPS Keoziel-Kedziosski et al., 2016), SVAMP Patel et al. (2021)), and commonsense reasoning (CSOA (Talmor et al., 2018)). D Training Configurations. We perform dense fine-tunne finettunin for all approaches. Given downstream parameter budget, SMoE-based methods will select the most voted experts based on their routing policies. Detailed training setups are listed as follows. We fine-tune the pre-trained Transformer-XX XI. with a smaller learning rate of 1 x 10-4 and batch size of 64 on SST-2 benchmark. And for BERT and RoBERTa, we fine-tune the models on the aforementioned four reasoning datasets. The learning rate is fixed at 2 x 10 5 and the batch size is 64. In each downstream task, the fine-tuning continues for 3 epochs, while other configurations are kept the same as the ones in pre-training. D Evaluation Metrics. At the evaluation phase, accuracy (%) and the problem solving rate (%) (Wei et al., 2022) are reported on the test set of SST-2 2 and other reasoning tasks, respectively.
Downstream Fine-Tuning
4.2 SUPERIOR PERFORMANCE OF SMOE-DROPOUT
We adopt classical transformer-based models, i.e., (Transformer-XL. BERT, RoBERTa), and train them in dense SMoE-based manner on fenwik8, BockCorpus, BookCorpus). Evalua- tion results are summarized in Table 1, where all models are compared under the same number of parameter counts. The following observations can be drawn: O Our SMoF-Dropout demonstrates superior performance compared to all other training algorithms. Specifically, -MoE-Dropon with all experts selected obtains 1.37 18.49, 0.56 12.44, and 152.82 188.00 (x10- 2) lower BPC for Transformer-XL. BERT, and RoBERTa, respectively. This validates the effectiveness of our proposals. @ Appropriate random routing policies show consistent performance benefits across all three network backbones. Moreover, our randomly weighted router surpasses the completely random allocation in THOR, which is within expectation since our assignment is implicitly "opti- mized" using evolved feature embeddings.
@ In terms of training efficiency, SMoF-Dropout has up to 21,, 37%, and 25% training time savings compared to the dense training of three backbones. If only half of the experts (k = ) are activated, approach enjoys extra 23% ~ 34% inference FLOPs reduction with a comparable BPC, Although the learnable SMoE reaches the best efficiency, it results in inferior performance.
Besides, we report another group of experiments varying the expert numbers (parameter counts) during evaluation. As shown in Figure 3, for SMoE-based approaches. we directly change the number of activated experts at the inference stage, which is an in-situ fashion from the single trained transformer. While for dense training baselines, each dot in their curve requires a separately trained model since it does not allow modificatiion of network capacity without further fine-tuning. Our findings are as follows: O The performance of SMoo-Dropout stably improved along with more parameters used, and it outperforms the others after 1.0, 10, and 8 (x10") parameter counts for three backbones. Such "slimmable" property enables scaling transformers to the full capacity without
"https://huggiin http:innnaaa.eeeeaeaaeeeeeeeeeee http://raaafroooo /tt.e//oooo////aaaaaaeee/eeeeeee rattetereaa..deorertt.t....

---

Table 1: Testing performance of (Transformer-Xi BERT, ROBERTa] network backbones on (enwit 8 BookCorpus, BookCorpus) datasets, respectively. All models are compared under the same number of parameter counts. Training time (s) and inference FLOPs (x10')) are reported. For THOR (Zuo et al., 2022), all (k N) experts activated.
SMoE, and SMoE-Dropout, evaluations are performed with half (k
Transformer-X..
BERT
RoBFRTa
Methods
BPC (
Time
Infer. FL.PPs
BPC ())
Time
Infer. Fl.OPs
BPC (L)
Time
Tnfer. F1.OPs
W. Dropout
1.1623
5.1298
7.7579
7.65:1
0.20688
8,0903
0.1898
101.75
Dense w, Concrete Dropout
1.3335
6.3519
7.7579
7.6419
0.3031
735.72
8,0820
0.2410
101.75
Dense DropBlack
1.2468
5.3902
7.7579
7.6349
0.2119
135.72
8.0773
0.1934
101.75
THOR (k- = N)
1.3110
4,8830
7.7620
7,6434
0.1439
135.73
8,0778
0.1607
101.76
SMoE (k N)
1.1896
4.7962
7.7620
7.7537
0,1387
133.73
8,1291
0.1538
101.76
SMoE. Dropout (k MMooDDDoo
1,1776
5.0220
5.6145
7.6372
0.1905
80.330
6,7893
0.1799
78,558
1.1486
5.0220
7.7620
7.6293
0.1905
135.773
6.5491
101.76
TranSoemer-Xi XL
BERT
RoDERTs
1.325
Despout
100 Per Clands (3P5)
1.200
1,275
7.54
Laurable SMali
THOR
Udde Ihe Chemater (UPC)
8.5
Derne w. Drspout
SMell-Troppse
THGR
2.70
1:225
Leanzile SMat SMLE Dropoct
pa 74
7.68
7,5
Dropout
1.200
THOR
1.175
Lorrable SMal.
2.64
6.-
Drogor
0.8
1.0
1.2
1.5
1.5
2.0
16
10
Pheamdtoo (K10")
Parmnio (X107)
Figure 3: Testing performance over parameter counts of (Transformer-X XL, BERT, RoBERTa] networks on (enwik8, BookCorpus, BookCorpus) datasets, respectively. A smaller BPC suggests better model.
collapse, bringing a once-for-all trade-off respected inference resource availability. @ In contrast, learnable SMoE's and THOR's BPC are quickly saturated and deteriorated when adopting more experts, which implies the existence of expert redundancy (or representation collapse). The potential reasons for their substandard results are (i) the overfitting to fixed # experts utilized during training for learnable SMoE, (fi), and the consistency regularization between experts' predictions for THOR.
4.3 TRANSFER STUDY OF SMOE-DROPOUT: SELF-SLIMMABLE
We further investigate SMoE-Dropout and its intriguing "self-slimmable" property in transfer learning scenario. Pre-trained models from Section 4.2 are densely fine-tuned on various down- stream tasks, including text classification {SST-2} and challenging arithmetic & commonsense reasoning {CSQA, ASDiv-A A. MAWP S, SVAMP] }. The performance is collected in Table 2. We find: equipped with SMoE-Dropout Transformer-Xi achieves 0.47% ~ 2.43% accuracy improve- ments on SST-2, BERT/ / RoBERTa obtain (0.07% ~ 9.72%, 0.42% ~ 3. 3.78%, 0.26% ~ 1.30%, 1.09% ~ 4.90%) and {-. 2. 2.10% 5.88%, 0.07% ~ 0.27%, 5.04% 5.93%) performance boosts on {CSQA, ASDi v A, MAWPS, SVAMP) respectively, suggesting an enhanced transferability.
Similarly, we alter the model capacity during downstream fine-tuning. Starting from one pre- training, the SMoE-based method first calculates the selected times of each expert based on one feedforward pass with downstream data, then chooses the top activated experts to meet certain pa- rameter budgets, and performs the subsequent dense fine-tuning. As displayed in Figure 4, our SMoF-Dropout has a continually increased accuracy or problem-solving rate when involving more parameters, and clearly surpasses the rest of approaches at parameter counts beyond 0.8, 8, and 10.5 (x10") for Transformer-XI. BERT, and RoBERTa respectively. shows a flexible capacity adjustment, i.e., "self-slimmaale", according to the downstream resource constraint.
4.4 EXTRA INVESTIGATION AND ABLATION STUDY
Q1: When does SMoE-Dropout outperform other baselines? A1: Sufficient Model Capacity.
To answer Q1 and understand SMoF--ropout's superiority in diverse scenarios, we investigate our proposal with different model capacities by varying model depth (e.g., layers) & width (e.g., experts).
Duue to limited computation resources. four, official) pre-trained BERT/RoBERTa models are produced with {10 10 training iterations. {128, 256} batch size, [MLM. MLM and NSP) tasks, on (BookCorpus (800M words), BookCorpus (800M words) and Eng1 sish k iedei a (2, 500M words)) dataset, respec- tively. The huge gap of pre-training outlays justifies the difference between and official performance.
7

---

Table 2: Transfer performance { (Acerrcyy (% 1), Problem Solving Rate (% 1)} of (Transformer-XI, BERT, RoBERTa) networks on {SST-2, CSQA, ASDiV-A, MAWPS, SVAMP} datasets. All models are compared under the same number of parameter counts. The same densely fine-tuning is adopted for all approaches, while THOR. SMoE, and SMoE- are tuned with half (k= ) or all (k N) experts activated.
Transformer-XL
BERT
RoBERTa
Methods
SST-2
CSQA
ASDiv-A
MAKES
SVAME
ASDiv-A
MAMES
SVAMP
Dense w. Dropout
81.94
30.44
55.27
80.47
34.24
49.58
78.06
28.90
THOR (f: N)
81.13
20.79
52.52
79.95
30.43
53.36
77.86
28.44
SMoE (k- N)
79.98
29.27
55.88
80.73
33.15
52.10
77.86
27.98
SMoE- Aropoor SMoE-Dropout
81.60
30.32
54.97
80.99
33.65
52.94
76.30
31.19
82.41
30.51
56.30
81.25
35.33
55.46
78.13
33.94
Transformer XL SST
BERT CSQA
BCRT ASDIv.A A
BURT MAWPS
Lowarde site
TIDER
TRE
THIX
(K227)
Pacarates (KCUP))
BEKT on SVAMP
RollERDa ASDiVA
RollEKTs on MAWPP
RotiERlia or SVAMP
T1002
50
5
16
35
16 Fastmsoon (410")
180
12.5
11.
120
125
(+10
PPaaraccm (920")
Figure 4: Transfer performance over # parameter counts of (Transformer-XL. BERT, RoBERTa] networks on downstream (SST-2, CSCA, ASDiv-A, MAWPS. SVAMP] datasets, respectively. Only the fine-tuning of Dense w. Dropout needs multiple pre-trained models with different amounts of network capacity.
Model Depth Different Number of Layers. We conduct experiments on enwik kB dataset with Transformer-X XL that has 2, 4, 8, 12 layers and each layer is turned into the SMoE layer through modularization. The comparison results of Densely Training w. Dropout and Training w. Learnable SMoE are reported in Figure 5 (a). We find that densely trained transformer performs the best when the network capacity is small like 2 layers, while with sufficiently large model capacity (2 4 layers), SMoE-Dropout demonstrates a consistent advantage compared the others. Mcantime, along with the increase of layers, the performance gap of SMoEs between the learned policy and our random policy keeps enlarging, signifying SMoE-Dropout's better scalability.
Model Width Different Number of Experts. Similarly, we study the influence of model capac- ity by examining Transforme-XL with different widths of 2, 4, 8, 16 experts. Results are summa- rized in Figure (b). Consistent observations can be drawn that: (i) Densely Training w. Dropout outperforms SMoE-based training under small network widths such as < 8 experts; (ii) SMoE- Dropout presents enhanced performance when applied to large models with 16 experts; (iti) Learn- able routing policies effective with small number of experts like 8 experts, while it gets worse results than our random routing with a sufficient number of experts, e.g., 16 experts.
Q2: What is better SroEooropout design? A2: Random Weight Router; Later-layer SMoE.
To answer Q2, we focus on the main constituents of SMoE-Dropout: Modularization, Random Routing Policies, and Gradually Increased k. Comprehensive ablations are depicted below.
Ablation on Diverse Random Routing Policies. An appropriate design of random routing poli- cies determines the achievable performance of SMoE-Dropout. We compare our random initialized and fixed router to SMoE with fully random assignments (Zuo al., 2022) and random hash SMoE with pre-defined deterministic random assignment (Roller et al., 2021). Transformer- XL results on 8 are collected in Fig. 5 (c), where our proposed random routing obtains substantially lower BPC of 2.96 ~ 170.11 (x10-2) than the other two under different amounts of model parameters.
Ablation on w/w.o. Gradually Increased k. Figure 5 (d) investigates SMoE- -roppout variants with and without gradually increased k. We see that disabling the progressive manner of enlarg-
8

---

1:2
1.0
144
brek Tucliy Pague
14
Gan Empars Penudd bomerr
Tanig Bepus
LG
115
1
L14
0.4
1.0
Ray
Mil
Prin ML.
Extra studies about SMoF-Drooout, Testing BPC of Transforee--T.. is collected enwi k8. (a) and (b) investigate diverse training mechanisms under different model depths and widths, respectively. (c) is the ablation of random routing policies. (d) examines the effects of gradually increased k. (e) studies the appropriate locations to insert SMoE expert layers.
Figure 5:
ing the number of activated experts causes unsatisfied performance. Also, result, the "self- slimmable" property completely disappears, e.g., adopting all model parameters leads to worse BPC.
Ablation Different Positions for Modularization. It remains mysterious where is the best position to insert SMoE layers. To address this question, we perform modularization to differ- ent transformer layers and record their performance in Figure (e). Specifically, given 4-layer Transformer-Xe XL, we compare four options: (i) Early, the first two layers are SMoE layers; (it) Mid- dle, the 2nd and 3rd layers are SMoE layers; (iti) Later, the last two layers are SMoE layers; (iv) Every-2, there is one SMoE layer every two transformer layers, i.e., the 2nd and 4th layers. From the results, introducing SMoEs to later layers is in general more beneficial than modulizing earlier transformer layers. One possible reason is that shallow layers might capture common features that need to be shared across input samples. More dissections are left for future works.
Q3: Extra benefits from SMoE-Dropout? A3: Improved Distillation and Less Overfitting.
Distilling into Single Expert on Downstream Tasks. Besides all the benefits pre-training in- ference and downstream transfer, we explore additional advantages of SMoE-Dropout under the distillation scheme that usually preferred in resource- restteee-limited applications. As shown in Table 3, we distill all pre-trained Trassformer-XLs into the same smaller variant with a single expert on the SST-2 2 downstream task. Our algorithm produces the most distillable models among all four methods by a clear accuracy margin of 0.76% ~ 1.89%.
Table 3: Distillation results of
Overfitting. We investigate the potential for overfitting to the training data distribution model parameters increase in SMoE- Dropout, SMoE, and densely trained transformers. As shown in Figure (a) and (b), experiments conducted on enwikō 8 with Transformer-XI. and three approaches are compared under the same parameter counts. We observe both SMoF-Dropout and Densely Training w. Dropout do not exhibit any indication of overfitting. That is, the performance is consistently improved as we increase the layers from 2 to 12 or experts from 2 to 16. In contrast, Training w. Learnable SMoE incurs BPC deterioration owing to overfitting when we expend the transformer 12 layers, similar to the findings in Zoph et al. (2022). We attribute the reduced overfitting to the implicit regularization effect of SMoF-Dropout and Dropout.
Transformer XL on SST-2. Method
Accuracy (t)
Dense w. Dropout
81.25
THOR
80.76
SMoE
80.12
SMoE-Dropout
82.01
CONCLUSION
In this paper, we present a novel and-aay SMoE-Dropout strategy for training over- parameterized transformers in full-capacity settings without collapse. We design fixed and ran- domly initialized router to assign experts and gradually increase their number along with the train- ing, As result, our proposal provides an appealing "self-slimmable" property to large transformers during inference and downstream fine-tuning, depending on available resources. It implies allevi- ated representation collapse and delivers an in-situ trade-off between efficiency and performance. Extensive experiments across various combinations of network backbone and dataset, consistently demonstrate the significantly improved performance and training time savings from our algorithm. Future work includes the extension of other network architectures and tasks like vision recognition.
ACKNOWLEDGEMENT
The research of ZWW is in part supported by the US Army Research Office Young Investigator Award IN911NF2010240)

---

REFERENCES
Alhabib Abbas and Yiannis Andreopoulos. Biased mixtures of experts: Enabling computer vision inference under data transfer limitations. IEEE Transactions on Image Processing, 29:7656-7667, 2020.
Karim Ahmed, Mohammad Haris Baig, and Lorenzo Torresani. Network of experts for large-scale image categorization. In European Conference on Computer Vision, pp. 516-532. Springer, 2016.
Raquel Aoki, Frederick Tung, and Gabriel L Oliveira. Heterogeneous multi-task learning with expert diversity. arXiv preprint arXiv:2106. 10595, 2021.
Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26, 2013.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multi- class classification. Neural networks, 12(9):1229-1252, 1999.
Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10020-12030, 2022a.
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task- sseciipec expert pruning for sparse mixture-of-experts arXiv preprint arXiv:2206.0022 00277, 2022b.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj. Xia Song, and Furu Wei. On the representation collapse of sparse mixture of experts. arXiv preprint arXiv:2204.09119, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm:
Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Aidan Clark, Diego de las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. arXiv preprint arXiv:2202.01 /69, 2022.
Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, al. Adaptive gradient methods at the edge of stability. arXiv preprint SxXiv:2207.14488 2022.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. Knowledge neurons in pretrained transformers. arXiv preprint arXiv:2 20l::2/04.000 10400869, 2021.
Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.0612 0220000.01...... 2022.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprint arXiv: 190100280, 2019.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
Terrance Vries and Graham W Taylor. Improved regularization convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
10

---

Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprchension at scale. arXiv preprint arXiv:1905.05460 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever, Learning factored representations in a deep mixture of experts. arXiv preprint 13/24:13/ 2013.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961 2021.
Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems, 29, 2016.
Yarin Gal, Jiri Hron, and Alex Kendall. Concrete dropout. Advances neural information process- ing systems, 30, 2017.
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Deming Chen, Marianne Winslett, Hassan Sajjad, and Preslav Nakov. Compressing large-scale transformer-based models: A case study bert. arXiv preprint arXiv:2002.11985, 2020.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.14913, 2020.
Golnaz Ghiasi, Tsung-Y. Yi Lin, and Quoe V Le. Dropblock: A regularization method for convolu- tional networks. Advances in newral information processing systems, 31, 2018.
Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 6865-6873, 2017.
Fu-Ming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. arXiv preprint 1xXiv:199.124886 2019.
Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, Zhaohui Yang, Yiman Zhang, and Dacheng Tao. A survey on visual transformer. ArXiv, abs/2012. 2020.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamorrthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34, 2021.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Ajay Jaiswal, Liyan Tang, Meheli Ghosh, Justin Rousseau, Yifan Peng, and Ying Ding. Radbert-cl: Factually-awarc contrastive learning for radiology report classification. Proceedings of machine learning research, 198:199-208, 2021.
Hao Jiang, Ke Zhan, Jianwei Qu, Yongkang Wu, Zhaoye Fei, Xinyu Zhang, Lei Chen, Zhicheng Dou, Xipeng Qiu, Zikai Guo, et Towards more effective and economic aparsely-activale model. arXiv preprint arXiv:21 2021.
Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm.
Neural computation, 6(2):181-214 1994.
Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.0836,, 2020.
11

---

Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteri- zation trick. Advances in neural information processing systems, 28, 2015.
Rik Kedziorsk,,,,,,, Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. Mawps: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 1152-11577 2016.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 2265-6274. PMLR, 2021.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv: 032665, 2019a.
Liyuan Liu, Xiaodong Liu, Jianfeng Gao, Weizhu Chen, and Jiawei Han. Understanding the diffi- culty of training transformers. arXiv preprint arXiv:2004.08249, 2020a.
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural ma- chine translation. arXiv preprint arXiv:2008. 07772, 2020b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining 2019b.
approach. arXiv preprint
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 2017.
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relation- ships in multi-task learning with multi-gate mixture-of-expert.. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1930-1939, 2018.
Matt Mahoney. Large text compression benchmark, 2011.
Zhiyuan Mao, Ajay Jaiswal, Zhangyang Wang, and Stanley H. Chan. Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model. ArXiv, abs/2207.100400, 2022.
David McAllester. A pac-bayesian tutorial with a dropout bound. arXiv preprint arXiv: 2013.
J. S. McCarley, Rishav Chakravarti, and Avirup Sil. Structured pruning of a bert-based question answering model. arXiv preprint arXiv:1910.06360 2019.
Poorya Mianjy and Raman Arora. On convergence and generalization of dropout training. Advances 2020.
in Neural Information Processing Systems, 33:21
Shen-yun Miao, Chao-Chun Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, 2020.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. Is a modular architecture enough? arXiv preprint arXiv:2206.0277 2022.
12

---

Wenlong Mou, Yuchen Zhou, Jun Gao, and Liwei Wang. Dropout training, data-dependent reg- ularization, and generalization bounds. In International conference on machine learning, pp- 3645-3653. PMLR, 2018.
Kirill Neklyudov, Dmitry Molchanov, Arsenii Ashukha, and Dmitry P Vetrov. Structured bayesian pruning via log-normal multiplicative noise. Advances in Neural Information Processing Systems, 30, 2017.
Sungheon Park and Nojun Kwak. Analysis on the dropout effect in convolutional neural networks. In Asian conference on computer vision, pp. 189-204. Springer, 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam M. Shazeer, Alexander Ku, and Dustin Tran. Image transformer. ICML,, 2018.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nlp models really able to solve simple math word problems? arXiv preprint arXiv:2103.07199) 2021.
Svetlana Pavlitskaya, Christian Hubschneider, Michael Weber, Ruby Moritz, Fabian Huger, Peter Schlicht, and Marius Zollner. Using mixture of expert models to gain insights into semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition Workshops, pp. 342-343, 2020.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arX00:2210.03350, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: html.
//jmlr. wttp://ppppppp////////0..
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto. Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34, 2021.
Stephen Roller, Sainbayar Sukhhaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In A. Beygelzimer, Y. Dauphin, P. Liang. and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, 2021. URL https: http:////ppeeeee/eeeeeeeeeeeeeee n forum?id= Berum?id=1MMDDDee111.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. Recurrent dropout without memory loss. arXiv preprint arviv: 1603.05 18, 2016.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arviv: / 701.06538. 2017
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
Lichao Sun, Congying Xia, Wenpeng Yin, Tingting Liang, Philip S Yu, and Lifang He Mixup- transformer: dynamic data augmentation for nlp tasks. arXiv preprint arXiv:2010.02394, 2020.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequ:: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv: /8// 1.0000937 2018.
Yixuan Tang. Hwee Tou Ng, and Anthony KH Tung. Do multi-hop question answering systems know how to answer the single-hop sub-questions? arXiv preprint arXiv:2002.09919, 2020.
13

---

Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 648-656, 2015.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv'e e J'egou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Dušan Variš and Ondiej Bojar. Sequence length is a domain: Length-based overfitting in transformer models. arXiv preprini arXiv:2109.07276, 2021.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In International conference on machine learning, pp. 1058-1066.
PMLR, 2013.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint ar / 2018.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers to 1,000 layers. arXiv preprint arXiv:2203.000 000300055, 2022.
Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. Deep mixture of experts via shallow embedding, Uncertainty in artificial intelligence, pp. 552-562. PMLR, 2020.
Jason Wei, Xuezhi Wang, Dale Schourmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint 11X1v:2201.11993, 2022.
Haibing Wu and Xiaodong Gu. Towards dropout training for convolutional neural networks. Neural Networks, 71:1-10, 2015.
Peng Xu, Dhruy Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung. Simon JD Prince, and Yanshuai Cao. Optimizing deeper transformers on small datasets. arXiv preprint arXiv:20/2. 2020.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019a.
Qiming Yang, Kai Zhang, Chaoxiang Lan, Zhi Yang, Zheyang Li, Wenming Tan, Jun Xiao, and Shiliang Pu. Unified normalization for accelerating and stabilizing transformers. arXiv preprint arXiv:2208.013111 2022.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv: 1902,01718, 2019b.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint 1rXiv:1809.080
2018
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019c.
Seniha Esen Yuksel, Joseph N. Wilson, and Paul D. Gader. Twenty years of mixture of experts. IEEE Transactions on Neural Nenworks and Learning Systems, 23 2012. doi: 10.1109/TNNI S.019TNNNNS.20122200229999
14

---

Jingzhao Zhang, Sai Praneeth Karimireddy. Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why (adam) beats {sgd} for attention models, 2020. URL https:
Minjia Zhang and Yuxiong He. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural Information Processing Systems, 33:14011- 14023, 2020.
Wancong Zhang and leshan Vaidya. Mixup training leads reduced overfitting and improved calibration for the transformer architecture. arXiv preprint arXiv:2102.11402, 2021.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefica- tion: Conditional computation of transformer models for efficient inference. arXiv preprint 10X00:21 2021.
Zhongwang Zhang and Zhi-Qin John Xu. Implicit regularization of dropout. arXiv preprint arXiv:2207.05952, 2022.
Minghang Zheng, Peng Gao, Renrui Zhang, Xiaogang Wang, Hongsheng Li, and Hao Dong. End- to-end object detection with adaptive clustering transformer. ArXiv, abs/201 1.09315, 2021.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoc Le, and James Laudon. ofxture-of-experts with expert choice routing. arXiv preprint arXiv:2202.09388 093666,,.. 2022.
Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning initialize neural networks for stable and efficient training, Advances in Neural Infor- mation Processing Systems, 24:16410-16422, 2021.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference computer vision, pp. 19 19-27,22 2015.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. arXiv preprint arXiv:2202.08906. 2022.
Simiao Zuo Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference Learning Representations, 2022. URL https: hwwp//eeeeeeeeeeeeeeeeeeeeeeeeep net forum? 1d=B72Hxs80q4.
15

---

A1 MORE IMPLEMENTATION DETAILS
Algorithm 1: Concrete Dropout in a PyTorch-like style
class ConcreteDropout (nn conuuee
def _init_
drepout_rogulariirriite init_min=0.1, init_max=0.1): super (Conerttttroo,, self) init_
self. self.weight__regulriier weight__eeuuarrzzn self.dropoot_reellattaa
aropout_regulariiz.
init_min ap. (init_miin ap. log init_min)
init_man dog (init_max) tog (1. init_max)
Laf..p_logis ParPaeteteer (torch, empty (1) uniform, (init min, init_max))
def forwa, input, next_layer) p torch. sig + Apply Concrete Dropout
output = colff__oncrete_dree drwpput (input, p]
4 Feed forward through the next layor
output next next_layer (output)
Calculate the weight regularizer sum_of_square
for param in part_laaer.pprmmterre.
sum_of_square +- torch (torch. (param, 2))
weights_rrgularizer self. reeigattreeularizer sum of square (1
p)
$ Calculate the dropout regularizer dropout_regulariies +- (1.
terch. log
Gropout_rregularizer
torch.log (1.
ingut_diimensionaliiy
dropout_reepularzzer
imput_dimensionallls
regularization = veights_regularizer + tropout_regularizee
return output, regularizaticn
def _coocretee_dropoottt (self, input, p):
eps - 1e-7; temp 0.1
Calculate the dropout probability matrix
unif noise torch. lil.:rraad_iieeiiineie (input)
drop prob = (torch.log + eps)
+ lorch.loguunif_1 eps)
torch.lo((1
+ eps)
torch. uni__niiess + eps))
drop_prob = torch (igrhhhiigmiidddrp_pprrooo
temp)
random_tensee = drop_prab
retain_prob
Apply Concrete Dropout
output torch.mul mul (input, random_tenson
output /= retain_prox
return output
Algorithm 2: DropBlock a PyTorch-like style
def drop block (input, drop drop_prob, block block_size) size]: Calculato the mask with zero value for block-wise elements mask = torch. like (x) .1t (drep_ float ()
mask r.max max__poolld
block_size, 1, bizek // 2)
mask mask
output input mask f Apply drepout
return output
A16

---

Concrete- Dropput Details of Concrete Dropout and DropBlock. In experiments, or DropBlock to replace the original dropout layer in MLP blocks. And for Concrete Dropout, we adopt the official implementation oa/oob/terr/// mwwaaa////////m concrete prropout-pyton iyyny method in Ghiasi al. (2018). And the PyTorch-style pseudo codes for both methods presented in Algorithm and 2.
wata/aaaaaaaa/nneeeeeeeeeeeeeeee /ttttp:://itthbbl pynb. For DropBlock, we follow the
bl ob/
A2
MORE EXPERIMENT RESULTS
A2.1 STABILITY ANALYSIS
Table A4: Test accuracy of Transformer-XL on SST-2. Both the average and standard deviation of accuracy are reported across independent runs.
To evaluate the stability of the improvement ob- tained by our SMoE-Dropout, we carry out further experiments of Transformer-XLL on S. The results are reported in Table A4, from which we can observe that our SMoE- -Doppot achieves a statisti- 1.17%
Method
Accuracy (t)
Dense W. Dropout
81.39 0.31
THOR (k = N)
81.15 + 0.55
cally significant improvement of 0.93% accuracy gains compared with other SMoE-variants and the dense network, where there is no overlap between the error bars (one standard deviation).
SMoE (k: = N)
81.20 0.50
SMoE-Dropout (A: )
82.03 = 0.26
SMoE-Dropout (k: = N)
82.32 1 0.14
A2.2 COMPARISON WITH LEARNABLE SMOES W. GRADUALLY INCREASED k
Table A5 demonstrates that both random routing policy and progressively increasing the num- ber of activated experts are beneficial for alleviating representation collapse and providing "self- slimmable" property, yet not as good as combining both. To be specific, when applying the strategy of progressively enlarging the number of activated experts, the learnable SMoEs suffer less repre- sentation collapse and achieve better performance, i.e., 0.31% higher accuracy. Meanwhile, We find that learnable SMoE with curriculum learning has the "self-slimmmale property only when activat- to 8. However, the performance starts to degrade if using more experts like A 16. As for SMFF-Dropout with random routing, it enjoys better "self-slimmable" property from k = 1 to k = 16 (full model capacity), with up to 0.87% higher accuracy on SST-2 across all scenarios, compared its learnable variants.
ing experts from
Table A5: Testing accuracy (%) over # activated experts of Transformer-X XL on SST-2.
# Activated Experts
4
8
16
SMoE w.o. Gradually Increased k
80.06
80. 79
80.58
80.99
81.20
SMoE w. Gradually Increased k
79.40
81.02
81.13
81.71
81.51
SMoE-Dropout
79.02
81.25
82.00
82.03
82.32
A2.3 TRANSFER STUDY ON MULTI-STEP REASONING TASKS
We conduct a further transfer study of the pre-trained BERT networks answering dataset, Hotpot CA Yang et al. (2018). And We use exact match (EM) accuracy to assess networks' performance. Following the same metric in Press et al. (2022), we calculate the compositionality gap, i.e., the gap of EM accuracy between multi-hop question answering and its all single-hop sub-questions Tang al. (2020), of each network. As shown in Table A6, our SMoE- Dropout is beneficial for reducing the compositionality gap, which achieves the best performance with up 0.30% higher EM score and 0.30% narrower compositionality gap, compared with the learnable SMoE and its dense counterpart.
multi-hop question-
Table A6: The EM score and compositionality gap of the pre-trained BERT networks on Hot
Metrics
Dense w. Dropout
SMoE
SMoE-Dropout
EM Accuracy (%)
15.10 14.90
14.90 15.00
15.20
Compositionality Gap (%)
14.70
A17


Reviews:
[{"review_id": "v1J1mmpfGXq", "reviewer": "Reviewer_qsko", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Sparse Mixture-of-Experts (SMoE) have been widely used to increase the representation capacity of transformers with similar computational cost as dense transformers. SMoE consists of many experts, only few of which are activate for any given input example. This increased representational power comes with two issues: \n(a) Representation Collapse -- as the latent space representations get centered around experts leading to lots of redundancy and sub-optimal performance\n(b) Poor Scalability -- During inference and downstream fine-tuning, increasing the number of active experts does not increase the performance due to overfitting to the active experts during training\n\nThis work addresses these issues by introducing SMoE-dropout. It consists of randomly initialized and fixed router network that activates the experts. It gradually increases the number of experts during the training. This yields SMoE that have \"self-slimmable\" property, i.e., offering consistent boosted performance for transformers with an increase in activated experts during inference and downstream fine-tuning. Empirical evaluations show that SMoE transformers trained with SMoE-dropout yields consistent improvements over their dense counterparts on various reasoning tasks.", "summary_of_the_review": "Overall the self-slimmable property is indeed important for a sparse mixture of experts model and the proposed architecture helps in achieving the same. The paper is its current state needs to add more details to the method and explain some of the design choices. It is also lacking in novelty as mentioned earlier. \n\n\n----\n\n\nI've read other reviewer comments and author's response. I'll increase my score accordingly.", "strength_and_weaknesses": "\nStrengths:\n- \"Self-Slimmable\" -- the consistent boost in transformer performance with an increase in activated experts during inference and downstream fine-tuning\n- Extensive empirical evaluation that demonstrates the improvements on pre-training and reasoning tasks\n\nWeaknesses:\n- Does not explain the proposal in details (see below)\n- Limited novelty ( progressive activation of experts by https://arxiv.org/abs/2112.14397v1, other works have looked at progressive activation of kernels in CNNs such as Once-for-All networks  )\n\n\nQuestions for Authors:\n\n(1) SMoE-Dropout needs to be explained in more detail than what is presented in 3.2. What does it mean to say that the router network is \n - randomly initialized\n - and fixed \n - and implicitly optimized during training\n\n Either it is fixed or you train the network. May be elaborate any points that I have missed.\n\n(2) How do you integrate the Concrete-Dropout and DropBlock in the proposed SMoE-Dropout? A simple pseudo-code that shows these steps end-to-end would help a lot to clarify things. \n\n(3) Why is only the linear schedule used to gradually increase during the training? How about other ways to progressively activate the experts? How does dropout and expert activation schedule interplay?\n\n(4) How is Figure~1 generated? \n   - For SMoE-Dropout --> for the same SMoE, you increase the number of activate experts --> yielding in higher parameter count.\n   - How are green and black curves generated? \n   - Since black represents the dense network --> it should be a collection of different dense models\n   - Why is the proposed scheme worse in the low parameter count regime?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "l3-AEIOFl3K", "reviewer": "Reviewer_DuzK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "While many MoE research focused on improving routing policies to encourage expert diversity and specialization, this paper studies another aspect of expert scalability, i.e., gradually increasing the portion of experts being activated per time. The authors show that by making MoEs smoothly scalable to the number of experts, it can turn into a new structured dropout approach that effectively trains larger transformer models in their full capacity. ", "summary_of_the_review": "See the strength and weaknesses.", "strength_and_weaknesses": "First of all, it is very interesting to see the connection drawn between the modern MoEs and the “old-fashioned” dropout. The former typically refers to both sparse training and sparse inference (at the same or similar sparsity), while the later conducts sparse training yet dense inference in full model capacity. Despites their formation similarity, traditionally they pursue different goals: this paper makes a revisit of them and neatly bridges their gap.\n\nTheir proposed solution, SMoE-Dropout, simply re-parameterizes a transformer with MoE layers for their MLPs, and then use a fixed random router to activate experts with a “curriculum learning” strategy, i.e., the number of activated experts gradually improved. Such training naturally yields a once-for-all in-situ trade-off between efficiency and performance when considering the deployment – a bonus that most current MoEs do not naturally enjoy. The method is also potentially generalizable to other non-transformer models.\n\nThe authors report very extensive empirical studies across various combinations of network architectures and datasets, as well as some transfer learning settings. They consistently demonstrate the significantly improved performance as well as training time savings from the SMoE-Dropout over a number of baselines. The experiment analyses are particularly well done and provide many useful insights on how SMoE-Dropout recipe works.\n\nThe following weakness points are identified, and clarifications are requested:\n-\tIs random routing really the major contributing factor in your recipe? What will happen if we apply the same curriculum learning of k to training learnable MoEs? Will they also become self-slimmable and suffer less collapse?\n-\tAnother main critique I hold against this work is that in many experiments, the empirical gains of SMoE-Dropout seem just marginal over other baselines, although the performance increase trend seems consistent as more experts are activated.\n-\t In Figure 1, it also appears that SMoE-Dropout performs worse than learnable MoE when the model is small.\n-\tI am also curious that, given the “self-slimming” property displayed on the same task, whether and how SMoE-Dropout could help address the compositional generalization challenge, that most MoEs nowadays can only solve with the assistance of clever prompting? For example, it would be very interesting to see the authors testing their trained models on multi-step commonsense or algorithm reasoning cases, perhaps with or without prompting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ED33CGo2sv", "reviewer": "Reviewer_EKmy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel plug-and-play strategy for training large transformer models, which leverages sparse MoEs in a dropout-like manner to scale transformers to better performance in their full capacity without collapse. The method is simple, and the experiments are thorough.", "summary_of_the_review": "Overall, I would tend to accept this paper. If the author can address my concerns. I would be more convinced.", "strength_and_weaknesses": "Pros\n-\tSMoE-Dropout demonstrates an attractive “self-slimming” property during inference and downstream fine-tuning, which delivers a once-for-all in-situ trade-off between efficiency and performance\n-\tLike classical dropout, SMoE-Dropout is able to mitigate the representation collapse that standard MoEs usually suffer from, i.e., activating more experts do not improve or even hurt performance.\n-\tApplying SMoE-Dropout is extremely cheap as only random router is adopted\n-\tTransfer study is strong, which provides another evidence that more pre-training information is effectively retained by the full transformer capacity. It is further validated in ablation studies 4.4\n\nCons:\n-\tThe performance gain of SMoE-Dropout is sometimes marginal, such as on BeRT (section 4.1). Moreover, as seen from Figure 1 and Figure 3, it seems SMoE-Dropout does sacrifice performance at low parameter counts compared to Learnable MoE, why?\n-\tWhen breaking a single-stream models into MLP MoEs (the modularization step in section 3.2), how to decide the number of MLP experts needed, i.e., N? This seems to be an important hyperparameter but not discussed. Also, why not applying SMoE-Dropout to MLP layers but not self-attention? \n-\tFor training with Learnable SMoE, it is further unclear how the authors selected its k?\n-\tWhy training time is the same between SMoE-dropout k=N/2 and k=N, in Table 1?\n-\tTable 1 again: are all those numbers averaged across three datasets? The authors didn’t explain\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "38exqnP52iq", "reviewer": "Reviewer_3ZBm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "SMoE-Dropout is motivated by the representation collapse and expert scalability issues observed in learned MoEs, by dynamically adapting number of activated experts as needed. It consists of a randomly initialized and fixed router network to activate experts and gradually increase their number as training progresses over time.", "summary_of_the_review": "The paper proposed a new method for dynamically adapting number of activated experts as needed and the overall quality is good. If the  questions as described above are well addressed, I tend to increase my score.", "strength_and_weaknesses": "Main Strength Points\n(1)\tSMoE-Dropout facilitates randomly and sparsely activated structure of network modules, playing an implicit regularization role similar to dropout – but more scalable to large transformer models. It has very low overhead and is very easy to use.\n(2)\tSMoE-Dropout naturally provides a “self-slimmable” property offering consistent performance boost for transformers with an increase in activated experts during inference and downstream fine-tuning. The transfer gains are especially obvious.\n(3)\tExperiments are very extensive and thorough, including a variety of tasks, architectures, datasets, and strong baselines beyond vanilla dropout. Both standard training and transfer learning are reported, accompanied with in-depth ablation studies. The experiment and analysis are very well organized and read convincing.\n\nWeakness and/or clarification:\n(1)\tDoes SMoE-training cost higher memory overhead, compared to dense training?\n(2)\tIt is very interesting to see that the SMoE-trained model can be distilled into a single-expert backbone. But details are lacked on how the distillation was done: is it common logit matching or something else? Also, how other MoE-variants will performance under distillation?\n(3)\tFor learnable MoE overfitting and collapse, it is an already well-known phenomenon that can also be mitigated by various diversity-regularized routers. It would be nice to compare SMoE-dropout with some of them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%