AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers
Authors: Ganesh Jawahar
Subhabrata Mukherjee
Xiaodong Liu
Young Jin Kim
Muhammad Abdul-Mageed
Laks V. S. Lakshmanan
Ahmed Hassan Awadallah
Sebastien Bubeck
Jianfeng Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Mixture-of-expert models
Neural architecture search
Efficiency

Abstract: Neural architecture search (NAS) has demonstrated promising results on identifying efficient Transformer architectures which outperform manually designed ones for natural language tasks like neural machine translation (NMT). Existing NAS methods operate on a space of dense architectures, where all of the sub-architecture weights are activated for every input. Motivated by the recent advances in sparsely activated models like the Mixture-of-Experts (MoE) model, we introduce sparse architectures with conditional computation into the NAS search space. Given this expressive search space which subsumes prior densely activated architectures, we develop a new framework AutoMoE to search for efficient sparsely activated sub-Transformers. AutoMoE sparse models obtain (i) 3x FLOPs reduction over manually designed dense Transformers and (ii) 23% FLOPs reduction over state-of-the-art NAS-generated dense sub-Transformers with parity in BLEU score on benchmark datasets for NMT. AutoMoE consists of three training phases: (a) Heterogeneous search space design with dense and sparsely activated Transformer modules (e.g., how many experts? where to place them? what should be their sizes?}; (b) SuperNet training that jointly trains several subnetworks sampled from the large search space by weight-sharing; (c) Evolutionary search for the architecture with the optimal trade-off between task performance and computational constraint like FLOPs and latency.

Under review as a conference paper at ICLR 2023
AUTOMOE: NEURAL ARCHITECTURE SEARCH FOR EFFICIENT SPARSELY ACTIVATED TRANSFORMERS
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural architecture search (NAS) has demonstrated promising results on iden- tifying efficient Transformer architectures which outperform manually designed ones for natural language tasks like neural machine translation (NMT), Existing NAS methods operate on space of dense architectures, where all of the sub- architecture weights are activated for every input. Motivated by the recent ad- vances in sparsely activated models like the Mixture-of Experts (MoE) model, we introduce sparse architectures with conditional computation into the NAS search space, Given this expressive search space which subsumes prior densely acti- vated architectures, cient sparsely activated uub-ransformnes. MutooM-E-geeattee sparse models obtain (i) 4x FLOPs reduction
develop new framework Aut to search for effi-
manually designed dense Transformers and (ii) 23% FLOPSs and 10% latency reduction over state-of-the-art NAS NAS-generated dense Tub-Tranfformers with parity BLEU score on benchmark datasets for NMT. OMoE consists of three training phases: (a) Heterogeneous search space design with dense and sparsely activated Transformer modules (c.g., how many ex- perts? where to place them? what should be their sizes?); (b) SuperNet training that jointly trains several subnetworks sampled from the large search space by weight-sharing: (c) Evolutionary search for the architecture with optimal trade-off between task performance and computational metries like FLOPs and latency.
INTRODUCTION
Transformers have demonstrated state-of- the-art performance in several tasks, but the larger their size. the more difficult it is to use them in resource constrained settings (Strubell et al.) 2019). Recent works in neural architecture search (NAS) (Wang et al. 2020 Xu al.. 2022a 2021 So
et al., 2021] Xu et al., [022b] Javaheripi et al. 2022) have focused on identifying computationally
efficient sub- Transformers that are easier to deploy on edge devices. However, existing works on NAS only focus on the subspace of dense Transformer architectures, where all the network weights are activated for every input.
In contrast to the above dense models, sparsely activated ones like the Mixture-ote -o-ture-of-Bre -aiee-of--xxppp Experts (Fedus) al. (2022b) perform conditional computation which only subset of the weights of the network are activated per input. Selective compute allows us to design neural networks with large number of model parameters, without significant increase in the computational cost. With increased capacity, these sparse models have demonstrated state-of- of-the-art performance in natural language tasks such as neural machine translation (NMT) et al., 2021 Kudugunta et al. 2021 Zuo et al.) 2022).
The goal of this work is explore the spuce of sparsely activated MoE architectures for NAS to identify computationally efficient sparse sub- Tabbrrssffrrrrss.
Incorporating MoF architectures in the search space requires one to make several design choices. (a) Expert placement: Identifying the Transformer layers for introducing expert sub-networks. (b) Number of experts: How many experts to introduce in different layers? (c) Expert FFN size: What should be the feedforward network (FFN) size for each expert? Given the large search space of
Terminologies: (1) Dense architectures refer to fully activated networks for every input. (2) Sparse ar- chitectures refer to sparsely activated ones with conditional computation per input. (3) Optimal architectures refer to Pareto-optimal ones with the best trade-off between task performance and computational metries.

---

Under review as a conference paper at ICLR 2023
potential architectures and the exorbitant computational cost of training and evaluating them isting approaches manually design MoE architectures with a For instance, they the same number of experts of the same capacity in different layers and make ad-hoc decisions like introducing experts in every other layer (Fedus et al.
homogeneous space.
2022b Kim et al. 2021
Zuo al. 2022 Du et al. 2022 Artetxe et al. (022T) every four layers Zoph al. 2022).
These design choices are not necessarily optimal. The decoder should be lighter than the encoder for auto-regressive NMT tasks due to the cumulative latency of generating tokens one a time (Liu et al. 2020 Kasai et al. 202T). This impacts the design choice for the number of decoder layers and the number of experts to use in each. For instance, the loss of capacity with decoder layer reduc- tion can be compensated by adding experts on the remaining ones. On the encoder side, a vanilla placement of the maximum allowable experts in each layer results in increased latency from expert communication and activation, although theoretical FLOPs can remain unaffected. These suggest that the optimal MoE's could heterogeneous when resources like latency or FL.OPs are con- strained. In recent review on sparsely activated models, Fedus et al. (2022a) note that the optimal hyperparameters depend on application and resource specifications where a systematic simulation of the compute, memory and communication cost can aid practitioners quickly determine optimal settings without costly trial-and-erros launches. Aut OMoE provides such a framework to identify optimal byper-parameter configurations for sparse models under computational constraints.
The above observations are depicted in Table which shows demonstrative examples of manually designed architectures those found by outE framework from the search space. We compare these architectures against various computational metrics (e.g., latency, FLOPs, active MoE parameters), architectural configurations and task performance.
Machine Translation
#Experts in each layer
Accuracy
Computational Footprint
Design Approach
Encoder
BLEU
Latency
Active Params
(G)
Manually designed (every layer)
4-444
444444
27.87
586ms
56M
Manually designed (every other layer)
14.144.14
14.114.144
28.48
506ms
56M
3,4
AutoMoE (4 Experts)
24-1-1-3-1
1-1-111-
28.22
239ms
49M
3.1
AutoMoE Experts)
1-1-44-4-1
4-1-1-1
194ms
22M
Table Manually designed vs. AMt searched architecture for 6-layer encoder-decoder Trans-
former. We report various computational footprint metrics (measured on 1 V100 GPU) and BLEU
score of sparse expert models on WMT' 14 En-De machine translation task. We show the number of experts per layer separated by hyphen (-) for encoder and decoder.
Novelty: To the best of our knowledge, AutoMoE introduces the first end-to-end framework to automatically design efficient MoE models under resource constraints. AutoMoE is also the first MoE framework to support adaptive computation due to heterogeneous experts, where input tokens are routed to experts of different sizes.
With this desiderata, develop AutoMoE with the following components and contributions:
We introduce a heterogeneous search space for Transformers consisting of variable number, FFN size and placement of experts in both encoders and decoders; variable number of layers, attention heads and intermediate FFN dimension of standard Transformer modules.
We extend Supernet training this new search space which combines all possible sparse chitectures into a single graph and jointly trains them via weight-sharing, yielding a reduced amortized training cost.
We use an evolutionary algorithm to search for optimal sparse architecture from Supernet with the best possible performance on downstream task (e.g., BLEU score for NMT tasks) satisfying a user-specified computational constraint.
Experiments on several NMT benchmarks demonstrate that Aut scaehcrrc sparse models obtain (i) 4x FLOPS reduction over manually designed dense Transformers and (ii) 23% FLOPs and 10% latency reduction over sate-of-the-ar -AS-generated dense uubTransformers with comparable BLEU scores.
Changes made as part of the revision are highlighted in red.
2

---

Under review as a conference paper at ICLR 2023
2 BACKGROUND
Sparse expert models: Mixture-of-expert models have rich literature in machine learning dat- ing back to the early 90s (Yuksel et al. 20122. Sparsely activated expert models, where only a small subset of experts are active at any given time, have received sigmificant attention with works such as Shazeer et al.] (2017). Switch Transformers (Fedus et al., (022b), GShard (Lepikhin et al.)
2020), BASE (Lewis et al. 202T), Hash (Roller et al., 2021). GLaM (Du et al., 2022), Stochas- tic Experts (Zuo et al.) 2022), Gating Dropout (Liu et al. 2022) and ST-MoE (Zoph et al.) 2022).
Some crucial differences these works include choice of expert routing function, expert placement technique, stability/performancee enhancement techniques and nature of the task (pre-training vs. fine-tuning). Some challenges in building sparse expert models include: (i) lack of diversity in ex- pert design (expert layer selection, number of experts, expert size, etc.), (ii) training instability and (iii) expert load balancing issue, to name a few. A comprehensive review of recent sparse expert models can be found at Fedus et al. (2022a).
Expert design:
Prior work on designing sparsely activated expert models has largely relied on ad- hoc manual choices in terms of expert layer selection, number of experts and their sizes. Existing approaches mostly use manual design, where they add experts on (i) alternate layers (Fedus et al,,
20226, Kim et al. 2021 Zuo et al., 2022 Du al. 2022 Artetxe al. 2021), (ii) every four layers Zoph et al., 2022), or (ili) final few layers Rajbhandari et al. (2022) The resulting sparse models have homogeneous expert layers, i.e., same number of experts of the same size in all expert
layers. These choices are generally agnostic to the computational constraints (e.g., latency, memory) of the hardware in which the sparse expert model has to be deployed.
Switch Transformers: The variant of sparse expert model used in this work is Switch Transform- ers (Fedus et al., (0026), mainly chosen due to popularity and high performance. Switch Trans- formers replace every Feed-Forward Network (FFN) layer with an expert layer consisting of col- lection of experts (independent FFN networks). Each expert layer is preceded by parameterized routing network that is trained route each token to top-k experts in the expert layer. In this work, we adapt Switch Transformers to an encoder-decoder model which is trained from scratch on the machine translation task using top-1 routing.
Neural Architecture Search (NAS): Given a search space of architectures and efficiency con- straints (e.g., model size, latency), NAS typically aims to identify the optimal architecture that maximizes the task performance, while satisfying the efficiency constraints. The main challenges in building NAS framework include: (i) constructing a search space that covers diverse architectures for the task, (ii) building a fast, accurate performance predictor and latency estimator for candidate architecture evaluation, and (iii) designing search algorithm to find Pareto-optimal architectures. NAS has been recently used for natural language understanding tasks to build efficient BERT (De- vlin et al.
2019) and GPT (Brown et al. 2020) based pre-trained language models (Xu et al. 2021
Yin et al. 2021: Xu et al., 2022a b. Gao et al. 2022] Dong et al.] 2021 [oo et al.] [021 Javaheripi
(ct al.] 2022)) as well as for machine translation tasks (So et al.
| 2019 Wang et al., 2020). Hardware
aware transformers (HAT) (Wang et al.) 2020) is ofate-of-the-art NAS framework for machine translation that uses hardware latency as feedback for optimization.
However, all of the above NAS works consider search space with densely activated Transformer models, and primarily search over typical Transformer architectural syper-parametery like number of layers, attention heads and hidden size. In contrast, we propose the first NAS framework that considers sparsely activated Transformer models like the Mixture-of-Experte which subsume all prior densely activated Transformer models special case (i.e. expert per layer). This allows us to expand the search space and make it more diverse by considering heterogeneous architectures and in the process address some longstanding design choices for MoE's like how many experts? which layers to place the experts? what should be the expert size? and so on.
3 NEURAL ARCHITECTURE SEARCH FOR MIXTURE-OF-EXPERO
In this section, present the components of the Aut OMoE framework (as illustrated in Figure I for designing efficient sparse networks under computational constraints.
3.1 HETEROGENEOUS SEARCH SPACE
3

---

Under review as a conference paper at ICLR 2023
(1) Design space construction
(2) Supermet Training
Mic expe
Encoder Layer
Decocer
Sampll_Archtsctuuet Archtocturej)
expeit s.r
Encoder
Earcep (13044))
FFN
FFN
FIN
Docodar Layer1
Sample Archiecturr]]]
(3) Search
Rouber (taken)
Eneader
Lueency
GPU
Rouner (tsken)
Grods AoddAArantion-HHaad
Candalae Architactree
Evolutionary Search
Self Aell-Atemiinn -ieddss
Se oniteads
Aedraattur
Welcklion
Embeddinn Size
Embadding Sue
(4) Efficient Subnet Training
Standalone
[ffierr
training
Otx (es
GPU
Figure 1: AutoMoE Framework. (1) Heterogeneous search space with variable dimensions for dense Transformer blocks and sparsely activated expert modules. (2) Supernet training by sampling subnetworks from search space and training them by sharing common weights with Supernet. (3) Evolutionary search to find efficient architectures by (a) sampling subnetworks from the search space; (b) using latency measured in the target device; and (c) performance estimation from Supernet as feedback for iterative optimization via crossover and mutation. (4) Efficient subnetwork(s) from evolutionary search trained on downstream task.
Attributes
Dimensions
Existing MoE approaches greatly restrict their design space by considering uniform distribution of capac- ity/size and number of expert subnetworks placed in different Transformer layers. For instance, the stan- dard MoE design (Fedus et al., (2022b) for an L-layer Transformer with M experts placed in alternate lay- ers has only two possible configurations viz., {1-M- 1.... (M-1-M-...) J. Our design space allows vari- able number of experts in each layer resulting in M possible configurations. (b) Furthermore, our design space also allows variable expert capacity, e.g., by modulating the width of the feedforward (FFN) sub- networks for different experts. Considering N possi-
Encodor-Embedding EinbeddinggSS
(512, 640)
[512. 640)
#Encuder Layers
[6)
dDuuder- Layers
2,, 4,
Encoder-QKV-Dim
(512)
Decoder-KK V-Dim
[512)
#Encoder-Self- Art- Art-Heads (PI.)
8)
#Deeoder-Self-ae Sel------ (PL)
[4, 8}
Cressder-Cress-Atc An-Heeds (PL)
(4.8)
dDecoder Arbitrary (PI.)
Encoder- I-teeeermmde Size
{1024, 2048, 3072}
(PL, PE)
Decoder.FFN.imermmeet Ineerrrmediite-Siz (PL, PE)
(1024, 2048, 3072)
#Fncoder- (PI.)
(12 M)
#Deooder-Experts (PL)
[1. M)
ble FFN dimensions for each expert results in N possible configurations for designing the expert space. (c) Finally, given the autoregressive nature of tasks like neural machine translation, the inference cost is domi- nated by the decoder (Kasai et al. 2021). For instance, for token-based MoE, decoders take 200x the time per step compared to encoders at peak throughput (Kudugunta et al. 202T). Therefore, we further con- sider variable number of decoder layers along with the above choices for expert placement and expert capacity. To the best of our knowledge, our work is the first to study such a flexible and exhaustive design space for modeling sparse architectures.
Table 2: Search space of Aut oMoE. 'PL' and 'PE' refer per layer and per expert search dimensions. Decoder arbitrary attn. searches last k encoder layers to attend. FFN size varies across layers and experts. M denotes maximum experts per layer.
In addition to the heterogeneous experts, we allow flexible design for the non-expert Transformer modules like the number of attention heads, hidden size and intermediate feedforward dimensions. This heterogeneous design of the non-expert, i.c., dense Transformer modules, has been explored in prior works such as HAT (Wang et al. 2020) for generation tasks like NMT, and AutoDistil (Xu
al. 2022a) for understanding tasks like those in the GLUE benchmark Wang et al. 2018).
A typical challenge in designing an expressive search space for NAS the increased computational cost to search over all viable configurations. Some recent works like AutoDistil (Xu al. 2022a) and few-shot NAS (Zhao et al.) (202I) demonstrate that a curated search space can alleviate gradient conflicts in weight-sharing of candidate subnetworks and improve Supernet training and stability.
4

---

Under review as a conference paper at ICLR 2023
Expert innemediree C. 1004)
23400
Samphii Erieedd 5
(og. Scat
Suia Transformer Weight for Output
Sample Erteading
P.S. 5331
SubTranstermer Weight for Ousput FFN-2
Enteedihigg
Sampled Embedding (e.g.
Internee Sine
Infermesh lea, 3072)
Sampled Experts
Outi
TubTransformer Weight
Input
Router
Mox Euperta (e.
2081
Saropted Feebeddding
SubTransformer
(n 3129
Weight for
MON DovetPi Irie www.eeeeee th 30221
Expat medte
Sampled Emieediing
Weight Input FFN-2
SubTrransfrmer
Iea. 5021
Mac Sepurt PPM
IPE PEETAN Sun Striee.
20021
Max. Embedding Size (0.s, 640]
(a) Router
(b) Experts (e.g., 2 FFN experts)
Figure 2: Weight sharing in the Supernet for sparsely activated expert modules. Illustrations of weight sharing for the standard dense Transformer modules can be found Wang et al. (202))
Such conflicts result from sharing weights between subnetworks of very different sizes that have different convergence rates. To this end, we make the following design choices for our expert and non-expert Transformer modules. (i) For dense non-expert modules, we leverage existing search space from prior work (Wang al., 2020) which already demonstrated state-of the-art performance for tasks like machine translation and forms the strongest baseline for our work in the dense Trans- former space. (ii) We further incorporate various features for the sparse expert modules in the search space corresponding to the number of experts and their capacity in the form of their intermediate FFN size. We make the above choice for both encoders and decoders where each expert FFN can have a variable size and capacity. In both settings, each layer can have flexible choices for the stan- dard Transformer hyper-parameters. (iii) Given the auto-regressive nature of machine translation tasks, the inference latency in the decoding phase can be twice as much as that in the encoding phase (Kasai al., 202I) (demonstrated many decoder layers to use and subsequent expert distribution based on latency constraint. We keep the number of encoder layers fixed for all our experiments and study this choice via ablation in Table ( shows our search space.
Figure (e)). Consequently, we also search over how
Table 5
3.2
SUPERNET TRAINING
A typical challenge in the NAS framework is to develop performance estimator that can efficiently compute the accuracy of a candidate architecture. The naive approach training candidate architec- tures from scratch to convergence and then evaluating on the validation set is prohibitively expensive given the large search space for all possible candidate architectures (see Section 3.3). To address this challenge, Aut
performance estimator is based weight-sharing via Supernet (Cai al., 2020 Wang et al. 2020). which is commonly used technique in recent NAS works. The Supernet for AMoE is the largest sparsely activated expert network in the search space. It consists of the maximum number of experts (M) placed in every layer of the Transformer in both encoder and decoder. Each expert FFN has the maximum intermediate hidden size the search space. Similar principles apply to the non-expert dense modules initialized with corresponding full dimension.
The Supernet trained with the following steps: (i) sample candidate architecture randomly from the search space (Guo et al.) 2020); (ii) train the sampled architecture by extracting the common portion of weights from different layers in the Supernet (i.e., by weight sharing) for one training step on the task; (iii) repeat steps (i) and (ii) until the training budget is exhausted. Once the Super- net training is complete, we can obtain a quick accuracy estimate for a candidate architecture (i.c. subnetwork) by extracting its shared weights from the Supernet and evaluating the validation set.
The key challenge here is to build weight sharing techniques for expert-specific components, which include: (i) router: neural network that is trained route each token (of 'embedding size") in an incoming example to exactly one expert (out of M experts) for top-1 routing; (ii) FFN expert: a standard Transformer FFN block that has unique weights and is learned independently. AutoMoE's
5

---

Under review as a conference paper at ICLR 2023
expert layers follow the Switch Transformer (Fedus et al. 2022b) specification. For subnetwork extraction from the Supernet, AutoMob extracts front rows and front columns of the Supernet's router weight matrix, corresponding to the subnet design. For example, consider the Supemet's router to be designed for experts and 640 embedding size with the shape of the router weight matrix as 4 640. Consider sampled subnet during Supernet training, consisting of 3 < experts and 512 < 640 embedding size with the subnet's router matrix as 3 x 512. To populate this matrix, we extract the first rows and first 512 columns from the Supernet's weight matrix (as illustrated in Figure 2 (a)). Such a weight sharing technique allows us to design architectures with varying number experts in each Transformer layer.
AutoMoa also extracts front rows and front columns from the weight matrices of each FFN ex- pert from the Supernet, corresponding to the subnet design. For the previous example, assume the intermediate FFN size of each expert in the Supernet to be 3072 (shape of weight matrix for first FFN layer is 3072 x 640 and second FFN layer 640 x 3072). Assume the sampled subnet to be designed for 2 experts with intermediate FFN size of one expert to be 2048 while the other to be 1024. For the first expert, the weight matrices of the subnet of shape 2048 x 512 (Input) and 512 x 2048 (Output) are extracted from the first 2048 rows, 512 columns (Input) and first 512 rows, 2048 columns (Output) of the corresponding Supernet weights. For the second expert, the weight matrices of shape 1024 x 512 (Input) and 512 x 1024 (Output) are extracted from the first 1024 rows, 512 columns (Input) and first 512 rows, 1024 columns (Output) of the corresponding Super- net weights. This example is illustrated in Figure (b). The subnet extraction technique does not extract weights from the third and fourth experts of the Supernet as the subnet is designed to have only two experts (not shown in the figure). Such weight sharing technique allows us design ar- chitectures with varying intermediate FFN size for each expert. Additional techniques for improving expert capacity such as stacking FFN layers, and techniques for improving Supernet performance with sandwich sampling (Yu et al. 2019), in-place knowledge distillation (Yu et al., 2019), gradient conflict reduction (Gong et al. 2022) are left for future work.
3.3 SEARCHING FOR EFFICIETT SUB- TEA-TFARNNFREEE WITH COMPUTATIONAL CONSTRAINT
out search is based on an evolutionary algorithm that takes the hardware computational con- straint (c.g., latency < 200ms) as input and aims to identify the abbaarchiee which achieves maximum accuracy for the task while satisfying the constraint. The algorithm works by randomly sampling from the design space an initial set of sparsely activated candidate architectures; evolving the top architectures iteratively by mutation followed by crossover; until the search iterations are exhausted. Candidate architectures are easily ranked by the Supernet performance estimator based on the validation score for the task. Latency estimate for each architecture is obtained by measuring the latency directly on the target device. The standard approach measures gold latency for forward propagation of a batch of examples for large number (e.g.. 300) of passes and then computes the truncated mean (after removing bottom and top 10% outlier latencies). This latency estimation can be costly given the large space of candidate architectures. To overcome this challenge, AotE uses partially gold latency, which is obtained by forward propagation of a batch of examples for a small number (e.g., 100) of passes and then computing truncated mean. After the search is com- pleted, the architecture with the highest performance is selected as the optimal one.
3.4 TRAINING EFFICIENT SUB- TRANSFORMER
Once the optimal sparsely activated architecture is identified, we train the model weights for the final architecture from scratch to convergence for the same number of training steps following the evaluation setting of HAT (Wang et al. 2020) for a fair comparison.
EXPERIMENTS
Datasets and evaluation metries.
We evaluate oMOE standard machine translation benchmarks: WMT 14 En-De, WMT' 14 En-Fr and WMT' 19 En-De with dataset statistics in Table 10 We use pre-processed datasets and evaluation setup from Wang al. (2020). We report BLEU score (Papineni et al.) 2002) performance metric with beam of size 5 and a length penalty of 0.6 (for WMT).
Baselines and AutoMoB variations. We compare Aut oMoE against both manually designed and NAS-searched architectures. For the manual baseline, we consider: (a) densely activated Transform- ers Vaswani et al. (2017a) with no experts; (b) sparsely activated MoE with homogeneous experts

---

Under review as a conference paper at ICLR 2023
Dataset
Network
#Active Params (M)
Sparsity
FLOPs (G)
BLEU
GPU
(%)
hours
W4 En-De
Transformer
Transformer
Sparse (every layor)
71
39
4.3
28.7
Evolved Transformer
NAS over Dense
47
0
2.9
28.2
2,192,000
HAT
NAS Dense
56
0
3.5
28.2
264
Random Search
NAS over Sparse
27
2.2
27.3
126
AutoMoF (6 Experts)
NAS Sparse
45
2.9
28.2
224
14 En-Fr
Transformer
Dense
176
0
10.6
41.2
240
Transfomner
Spurse (every layer)
71
4.3
41.5
Evolved Transformer
NAS Dense
175
0
10.8
41.3
2,192,000
HAT
NAS over Dense
57
0
3.6
41.5
248
Random Search
Sparse
27
2.2
AutoMoEl (6 Experts)
NAS over Spurse
46
72
2.9
41.6
236
AutoMoFF Experts)
Sparse
19 En-De
Transformer
184
Trunsformer
Sparse fevery layer)
71
39
4.3
46.3
Dense
63
Random Search
NAS over Sparse
27
50
2.2
43.7
126
AutoMoB (2 Expers)
NAS over Sparse
45
2.8
45.5
AutoMoE (16 Experts)
NAS Sparse
69
3.2
45.9
248
Table 3: FLOPs improvement of AutoMoE baselines with Pareto-optimal architectures high- lighted in blue. Changes in revision are highlighted in red. We report both active model parameters; and sparsity measured non-active parameters a percentage of total parameters. We report training time for one Nvidia V100 GPU and training cost following HAT (Wang et al. 2020) corre- sponding 40K training steps.
(i.e. same number and FFN size) placed in every other layer (Fedus et al. 2022b Kim al. 2021
Zuo et al.
2022 Du et al. 2022, Artetxe et al. 2021). For the NAS baselines, we consider (c) satte-of-the-art NAS framework for identify- ing efficient densely activated sub- Transformers for neural machine translation (same task setting as ours); and (d) Evolved Transformer (So et al. 2019) which one of the earlier works on finding efficient dense sub-Transe Transformers with evolution based architecture search.
HAT (Wang al. 2020). which Supernet-bseed
Training configurations. All the baselines and Aut oMoE variants including the Supernet and final model are trained with the same setting for fair comparison, following HAT. All the models are trained for 40K steps, with a warmup of 10K steps from 10-7 to 10 10-3 and use cosine annealing to 10-7 7 for the rest of the steps. All models are trained using fairseq toolkit (Ott al. 2019) with an effective batch size of 524K tokens on 16 V100 GPUs.
Evolutionary search setup. For performance estimation, monitor the validation loss of the sub- net on the NMT task. We compute latency by measuring the time taken to perform translation from source sentence target sentence with same desired input/output length (30 for WMT) and orig- inal beam settings (see Section 4 on the target device (NVIDIA V100 GPU). We measure latency 300 times for gold (to report final metrics) and 100 times for partially gold (during evolutionary search) respectively; discard the top and bottom 10% (outlier latency) and compute mean of the rest.
settings for evolutionary search include: 15 as iterations, 125 as population size, 25 as parents' size, 50 as mutation population size with mutation probability of 0.3 and 50 as crossover population size. Unless otherwise stated, the latency constraint for all the experiments set to 200 ms.
RESULTS
5.1 AUTOMOE BASELINES PERFORMANCE COMPARISON
Table 3 presents the comparison of Aut oMoE with baseline models several computational met- rics and task performance. We report the number of parameters without embedding weights, and FLOPs without the last decoding layer for all the models, consistent with WWag al. (2020) eval- uation settings. Aut wMM.-MMeeeratttttooeeeeeeeeeeaae sparsely activated sub- Transformers obtain 4x reduction in FLOPs manually designed (densely-activated) Transformer-big. Compared to NAS baselines

---

Under review as a conference paper at ICLR 2023
4.00
0.80
0.76
0
3.23
0..88
0.64
2.75
2.00
0.55
0.52
#Decoder
(a) #Deonder layers vs. Fl.OPs.
(b) Encoder experts as ratio of total experts.
0.270
Deccoder Labency (ma)
4 9.255
0.72
0.16
3.225
0164
0.230
0262
500
5 0.380
0.52
230
300
152
400
Encoder Layer
Decoder Layer
Overal Lateency (ms)
(c) Encoder layer vs. #Expert (%) (d) Decoder layer vs. #Expert (%).
(e) Overall vs. Decoder latency.
Figure Architecture analysis for Aut gengeerre sparsely activated models. We sample several architectures from the Pareto frontier for Aut.oMEE-variants and baselines, and report the aggregate statistics in terms of the impact on different computational metrics.
like Evolved Transformer So et al. (2019) and HAT Wang et al. (2020) that generate densely ac- tivated nub-Transformers, Aut oMoE improves on both FLOPs and BLEU score on an aggregate across all tasks. Notably, Supernet-basee
oMo and HAT have massively reduced amortized training cost (GPU hours) compared Evolved Transformer with progressive evolutionary search.
Compared to all the other densely-activated models, we observe that AutoMoE generates networks with high sparsity resulting in mas- sively reduced active parameters and active model size. For the NAS models, we train the top-2 Sab-rransformers in the Pareto fron- tier and report the one with the best BLEU FLOPs trade-off on the validation set. The op- timal maximum experts vary for different tasks, with 6 experts for WMT" En-De and 16 ex- perts for WMT' 14 En-Fr and 1M En-De given that the latter two datasets are 10x larger than the former.
Dataset
FL.OPs
Latency (ms)
BLEU
WM
HAT
3.5
205
28.2
AutoMoE
2.9
190
28.2
IM En-Fr
HAT
3.6
AutoMoB
2.9
196
41.6
WM En-De
HAT
4.1
212
45.8
AutoMoE
2.8
180
Table 4: AutoMoE reduces FLOPs by 23% and latency by 10% over optimal HAT (Wang et al. 2020) architectures under same latency constraint (200 ms) in aggregate with similar task perfor- mance demonstrating NAS over sparse networks is more efficient than NAS over dense networks.
5.2 ANALYSIS
Decoder layers vs. FLOPs. Figure (a) shows the average FLOPs for all Aut OMcE variants with different decoder layers as obtained during our search (varying from 3 to 6) and baseline mod- els. Notice that the FLOPs increase with increase in decoder layers, given the auto-regressive nature of NMT tasks which require generating tokens sequentially. In contrast to manually designed Trans- formers with 6 decoder layers (both dense and sparsely activated MoE variants), AutoMoE. and HAT-searched architectures reduce the number of decoder layers with resulting decrease in both FL.OPs and latency. This is also evident Figure 3 (e) which shows that decoder latency dominates the total inference latency for all the models more than 90%.
Expert distribution in encoder vs. decoder. Figure (b) plots the number of encoder experts as ratio of total experts for AutoMoE-generauee sub- Transforrrooss We observe that AutoMoE assigns a significant number experts to the encoder as compared to the decoder. As a result, encoders have much higher capacity (i.e.. encoder parameters proportion of overall parameters) than decoders. This correlates with the earlier observation that models with higher encoder layers compared to decoder layers enjoy better latency- performacc trade-off (Kasai al. 2021). Our findings from
8

---

Under review as a conference paper at ICLR 2023
oMt designed architectures indicate that the number of layers and experts two knobs that jointly help in modulating encoder capacity and decoder latency to design efficient models.
Expert distribution in different layers. Figures 3 (c) and (d) show the percentage of experts allo- cated to different layers for encoders and decoders averaged over several sampled architectures. Notice that the middle encoder layers (3rd 5thh are allocated the maximum number of experts, while the first layer receives the least. The trend reverses for decoder, with the first layer receiving most ex- perts with gradual reduction in expert allocation. This also consistent with keeping decoders light by dropping layers to reduce latency, while compensating for the reduced capacity with increased experts in the first few layers.
Pareto-optimal AutoMoE designed architectures. Table 7 in Appendix shows the sparsely ac- tivated expert architectures designed by two variants of Aut oMoE ("std-expert': expert FFN size same in each layer and variable across; 'fract-expert:: fully heterogeneous expert size) for different datasets with the best trade-off in BLEU vs. latency. On aggregate 69% of the experts are allocated to the encoder compared to the decoder. Meanwhile, 70% of the expert layers in 'fract-expert' ar- chitectures have 2 or more experts, out which more than 75% of the expert layers have varying capacities (i.e., experts with different FFN intermediate size).
Search space variations. Table 5 presents the im- pact of different search space choices the efficiency and performance trade-off. The first variation is to make "#Encoder Layers' an elastic search dimension. Note that both HAT and AutoMoE consider the num- ber of encoder layers be fixed (refer to Table 2 We observe that varying encoder layers degrades the model efficiency in terms of FLOPs (top major row), re-inforeing our prior observations on the importance of encoder capacity and depth.
Search Space Variation
BLEU FLOPs
(G)
Varying number of encoder layers HAT
28.2
3.5
Layees E. (1 6)
28.1
3.4
AutoMoE (Std-expert, 2 Experts) w/
28.2
2.9
fixed encoder layers
AutoMol: (Std-expert, Experts) w/ aFncoder- Layers E {1 6)
28.3
3.7
AutoMol:
28.2
2.9
AutoMoE (Sid-expert, Experts) w/ manually designed homogeneous experts 1-2-1-2.1-2
28.3
3.5
In the second variation (Table 5 second major row),
1-1-1-2-2-2
28.3
3.8
2-2-2-1-1-1
28,3
3.1
we fix the expert architecture (with experts manu- ally placed uniformly) in the search space and only search for the standard Transformer hyper-parameters. Observe that AutoMo E-designed experts have better FLOPs than such manually designed MoE architec- tures.
AutoMoE: (Fract-expert, experts) AutoMoE (Fract-oxpert) Identity Expert - PFN Intermediate
28.4
3.5
28.1
2.7
size E {0, 3072)
Table 5: Variations in space on WMT' 14 En-De dataset.
search
The last variation introduces identity or dummy experts (i.e., expert with 0 intermediate FFN size, equivalent to identity operation). This explores the idea that we can skip the computation for some of the tokens based on context rather than always forcing them through an FFN. We observe that identity experts marginally hurt the performance but significantly reduce FLOPs (last major row).
6 CONCLUSION
AutoMoa is the first framework to explore the space of sparsely activated Mfxture-of-Experts (MoE) models for neural architecture search (NAS). AMtE identifies efficient sparsely-activate sub- Jub-Transformess with reduction in FLOPs and latency over both manually designed and NAS- searched architectures, with parity BLEU score on benchmark machine translation (MT) tasks. AutoMoE explores fully heterogeneous search space with variable number of experts, their size and placement choices in different layers for encoders and decoders, alongside other standard Trans- former architectural hyper-panee parerrparameers.. Our experiments show that AutoMoE reduces FLOPs by 23% and latency by 10% over optimal HAT architectures under the same latency constraint in the aggregate, across benchmark MT tasks, while maintaining similar task performance.
Given our focus on finding efficient MoE models under computational constraints, Aut oMoE search space and evaluation has been restricted in scale to big- sig-sized Transformer models for benchmark MT tasks. A natural extension of this work is to explore the limits of MoE models like SwiichhTransfome ers (Fedus et al. 2022b) and GShard (Lepikhin et al. 2020) that are significantly larger containing billions to trillions of parameters; as well as designing sparse and transferable efficient expert mod- els (Zoph al. 2022) for diverse types of tasks like reasoning, summarization and understanding.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
The source code to reproduce Aut OMOE results can be found as part of the supplementary material. There will be a 'readme.md' file that has scripts to run the complete pipeline:
1. Download WMT datasets from web.
2. Train Supernet for performance estimation (Section
3.2
3. Run evolutionary search to identify best subnet (Section
3.3
4. Compute latency of best subnet (Section
5. Compute FLOPS of best subnet (Section
6. Train best subnet from scratch (Section 3.4
7. Compute BLEU score of best subnet.
Training configurations of supernet, evolutionary search hyperparameters and training configura- tions of subnet has been discussed in Section 4
Aut is built over fainseq (Ot al. 2019) and HAT (Wang al. 2020) implementation (pub- mim com/mit han- Lab/ htbpaarwware aware trans formers) For fair comparison, we retain most of the param-eas,,,ters training and evaluation recipes, pre- processed datasets from the original HAT implementation.
licly available at hhtpps:/ //t::://gttttt com/
REFERENCES
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan lyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa Kozareva, and Ves Stoyanov. Efficient large scale language modeling with mixtures of experts. CoRR, abs/2112.1066 2021. URL https: /ttp:///xxvvvv.. org/
wtwp/////////ooooo/ooooo//saaaea abs 2.1.2211.10688
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Vose Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL httpp::/pproceeeiiii neur 1457c0d6bfcb4 cb4966ffcc64666 741 Bofbbac112f64aaPPaae0 4457:0055.c00002244ccabaaaaaa2.2
neur ps. cccppaper//22///11ee pdf
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URLp:ttss:///raa http:///wwwxwwp.pppp/////// wwtppo//
09791. 1df
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. (0.18653/v1/N19-115 1423. URL https: haaelaancholooy..oo//111355
Chenhe Dong, Guangrun Wang, Hang Xu, Jiefeng Peng, Xiaozhe Ren, and Xiaodan Liang. Effi- cientBERT: Progressively searching multilayer perceptron via warm-up knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18653//1/20222 100.6655 www.iings-eennl..122222 URL htttt:/://eeaanttoooo..ooooz
findi nggiiigs-emmt 123
10

---

Under review as a conference paper at ICLR 2023
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. GLaM: Efficient scaling of language models with mixture-of-experts. In Ka- malika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5547-5569 PMLR, 17-23 Jul 2022. URL https: ottps://proceediing. mlr press www du22c. html
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning, 2022a. URL htlL:htts://// wwwp:/iv.rrggabbbabeoo 01667
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter
models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39 2022b. URL WRLhhtt:://
jm wwwpoorppp.pp.../o pttppperrsv////// v23 21 0998.998.... .html
Jiahui Gao, Hang Xu, Han Shi. Xiaozhe Ren, Philip L. H. Yu, Xiaodan Liang. Xin Jiang, and Zhengue Li. Autobert-zero zero: Evolving BERT backbone from scratch. In Thirty-Sixti AAAI Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 10663-10671. AAAI Press, 2022. URL https://ojs.com www../ossaaaiiooo.oo.. wereaai.orgiindeee indei.ddxx.. phhp/AAA atticle/view/2332
Conference on Artificial Intelligence, AAAI 2022, Thirty-
Chengyue Gong. Dilin Wang. Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, qiang liu, and Vikas Chandra. NASVit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations, 2022.
URL https:://pperreiieeee / forum? id=Qawl 6njk
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI, vol- ume 12361 of Lecture Notes in Computer Science, pp- 544-560. Springer. 2020. doi: 10.1007/ URL https:/ /tw..//oiiiaiaa wtt.oioorg/1...00000 10t.:/1..000.0000 100-111000077700000-55---1111111
32
Mojan Javaheripi, Shital Shah, Subhabrata Mukherjee, Tomasz L. Religa, Caio C. T. Mendes, Gus- tavo H. de Rosa, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. Litetransform- ersearch: Training-free on-device search for efficient autoregressive language models, 2022. URL
https: /tapsv///axvv..... org/ abs / 2203. 02094
Jungo Kasai, Nikolaos Pappas, Hao Peng, James Cross, and Noah Smith. Deep encoder, shal- low decoder: Reevaluating non-autoregressive machine translation. In International Confer- ence on Learning Representations, 2021. URL
net/ forum?id
KpfasTaLUpg
Young Jin Kim, Ammar Ahmad Awan, Alexandre Muzio, Andrés Felipe Cruz-Salinas, Liyang Lu, Amr Hendy, Samyam Rajbhandari, Yuxiong He, and Hany Hassan Awadalla. Scalable and ef- ficient moe training for multitask multilingual models. CoRR, 14/2109.10465, 2021. URL
httpps://artt hppps:///rxivv http://wwwvv.../.ee////o/o
arg//bs// abs 2109. 10465
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh- Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-of-experts for efficient inference. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/w1/200 1/225/////222..immmnnnn. findiing--emmll..0444 URL
Findings www.p.n..m.mm...mmm..mmm.ee.mw.. 304
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional
11

---

Under review as a conference paper at ICLR 2023
computation and automatic sharding. CoRR, abs/2006.16668, 2020. URL https:
h0s//bs/20055 16668
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang (eds.), Pro- ceedings of the 38th International Conference Machine Learning. volume 139 of Proceed- ings of Machine Learning Research, pp. 6265-6274. PMLR, 18-24 Jul 2021. URL https:
7/proceed ngs ml www.s/ressss...... lewi s21a. html
Rui Liu, Young Jin Kim, Alexandre Muzio, and Hany Hassan. Gating dropout:
Communication- efficient regularization for sparsely activated transformers. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 13782-13792. PMLR, 17-23 Jul 2022. URL https: / pttp::://proceediggs. mmr.
0osas/vv62/1i22220
Xiaodong Liu, Kevin Duh, Liyuan Liu, and Jianfeng Gao. Very deep transformers for neural ma- chine translation. arXiv preprint Stt::200000077222 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/11--0000 URL https::t///// http:///alaaathhocoo
org/N19-4009
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu Bleu: method for automatic
evaluation of machine translation. Proceedings of the 40th Annual Meeting of the Associa- fion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.3115/1073033.1073333.. URL https:
//aclattholloay..rr//0000---
Samyam Rajbhandari, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Mee:Speed-MoE: Advancing mixture-of- experts inference and training to power next-generation AI scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 18332-18346. PMLR, 17-23 Jul 2022. URL https: //proceedings.o mlr wwwessvvl22.
wtw.:///ww.......pp.eesssss//e./ htt.1/////aaabaaaaa.a..a ra jbhandar 22a ahmml
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Ad- vances in Neural Information Processing Systems, 2021. URL httlittts///ppeeeiieee net/ Forum? 1d= 1
MgDDWb 1ongDwbllll. ULM
Noam Shazeer, *Azalia Mirhoseini, KKzysztoff Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https: //openzeviews eww. beet/forrue d=B1 ckMDqlg)
David So, Quoe Le, and Chen Liang. The evolved transformer. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5877-5886. PMLR, 09-15 Jun 2019. htttttss:/ppooeeeeeeeeee .mlr www.sppsepee.pp..pppppp..eee
html
David So. Wojciech Manke, Hanxiao Liu, Zihang Dai, Noam Shazeer, and Quoc V Le. Searching for efficient transformers for language modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Infommtion Processing Systems, volume 34, pp. 6010-6022. Curran Asso- ciates, Inc., 2021. URL httpt:tt/////m httpttt://pcccce..aottte hewriing.sneeuuss... wttperrpsss.cpppppr////11111 7
pdf
12

---

Under review as a conference paper at ICLR 2023
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3645-36535 Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10.18653/v1// 1/P19/1111991355 URL https: //ac//acccc wrtccholog..rroo 10PP-19-130 9-1355
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017a. URL httttt:// httttp:://prooeed.ng..eeeee pdf
(155ee23357dees33 11te:77eeee11100011ccccca4aaaaa.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017b. URL htttps://pproeeeeettte www.innggs.eeuiine wwwrine.ipp.... htt.:ipp...ppppr.eeeeeee pctppapper/211///1eee 1155e223547dee31155 aaaaaa5seaeeeeea0000ccccccaaaaaa Papeeeaaaaaaaaaaaa...pe.. pdff
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/W1s 11tp:///////---5/1 5446. URL httlhhtps://// http:p::/acaaantcoooooooooo or1/188----11111 5446
Hanrui Wang. Zhanghao Wu, Zhijian Liu, Han Cai. Ligeng Zhu, Chuang Gan, and Song Han. HAT: Hardware-awarr transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7675-7688, Online, July 2020. Association for Computational Linguistics. doi: 10.18665\/1/2222aalmmaiill
URL.
686
Dongkuan Xu, Subhabrata (Subho) Mukherjee, Xiaodong Liu, Debadeepta Dey, Wenhui Wang, Xiang Zhang. Ahmed H. Awadallah, and Jianfeng Gao. Autodistil: Few-shot task-agnostic neural architecture search for distilling large language models. ArXiv, February 2022a. URL https:
//arxiv.orq/ant abs 2201222121.0 08539
Jin Xu, Xu Tan, Renqian Luo, Kaitao Song, Jian Li, Tao Qin, and Tie- Yan Liu. Nas-bert: Task- agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, pp. 1933-1943, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 9781450383325.
doi: 10.1145/3445588.4477888
URL https: http:///cc..oggg//// 1145/
3447548. 3467262
Jin Xu, Xu Tan, Kaitao Song, Renqian Luo, Yichong Leng, Tao Qin, Tie- Yan Liu, and Jian Li. Analyzing and mitigating interference in neural architecture search. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16646-24662. PMLR, 17-23 Jul 2022b. URL https:
7 //proceedings.. mir. mlr. www...p..ee....... wwwss/eeevve v xu22h. html
Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. AUEoT:nyBERTT Automatic hyper-parameter optimization for efficient pre-trained language models. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5146-5157, Online, August 2021, Association for Computational Linguistics. doi: 18.18/53/v1/ 2021 lon-long..400 URL wttpt:tt:///tm https:::/cclaaatoollog/o/o/ wwtp:/ooooooo.oo/////eeee acl.1o1......oon.....
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. In International Conference on Learning Representations, 2019. URL /tRlhhttss://peeeeee h/tppp//e/e//.../////////.
gMCsAqY?
13

---

Under review as a conference paper at ICLR 2023
Seniha Esen Yuksel, Joseph N. Wilson, and Paul D. Gader. Twenty years of mixture of experts. IEEE Transactions on Neural Networks and Learning Systems, 23(8):1177-1193, 2012. doi: 10.119/TNNLLS2022200222
Yiyang Zhao, Linnan Wang, Yuandong Tian, Rodrigo Fonseca, and Tian Guo. Few-shot neural architecture search. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12707-12718. PMLR, 18-24 Jul 2021. URL https::/pxooceeinnggsss .mr. press/ v139/zhao211 html
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, 2022. URL
https://arxxi ott/abs//aaaeoooooooooeoeeeaeeat /tt.a/////2220000000
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In International Conference on Learning Representations, 2022. URLhttps: http///epppe/eeeee//e/ee//e/eeee net forum? id=B72HXs80qq
14

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Hyperparameter
HAT
AutoMoE (std-experts)
AutoMoB
(fract-
Trans-Base BBg
experts)
Broddr-Emmbedding-Siz
(512, 640)
{512, 640)
(512, 6403
/ 1024 512/ 1024
Decoder- Fonnler-Fmhedddingg- Size
(512, 640)
(512, 640)
(512, 640)
LEncodor-Layers
{6}
[6}
[6}
6
#Deoodr-Layars
3,
L1 6,
(1.2, 3,
Encoder Bondee--QKV.DDee QKV-Dia
{512)
[512)
[512]
$12 / 1024 1024
QKV-Dim
(512)
(512)
Simcoder-Self-At-HHeeee (PL)
(4,
(4,8)
(4, 8.
8 16
#Deeoder-Seff-Al--E (PL)
(4,8)
(4,8)
(4.8)
iDeeoder-Cross-tt--H (PL.
(4.8)
(4.8)
(4.8)
8/
#Decode-Arbittary (PL)
[-1. 1.
[-1, 1, 2
[-l. 1,
Eacoder-HNN Intemmeeddei (PL)
1024, 3072
1024, 2048, 3072
[1024, 2048, 3072]
2048 4096
Decoder FFN Intermediate Size (PI.)
{ 1024, 2048, 3072
10024 2048, 3072
[1024, 2048, 3072]
2048 / 4096
#Encoder-Exppee (PL.
11.2. M
21.2. M] (1.2. M) [1024, 2048, 3072) [1024, 2048, 3072)
#Deunder Experts (PI.)
(1.2 M)
Enc-Exper-FFFN WWW.-FNNN--SSSSSS Inter-Sizee (PI., PF)
Doc-ExpettPFN-Inooee (PL, PE)
Table 6: Search space of AutoMoE vs. HAT vs. manually designed Transformer-Base / Big (Vaswani et al., 2017b). 'PL'' refers to per layer while 'PE' refers to per expert. M denotes maximum number of experts layer.
Qualitative analysis of expert architectures. Table 7 presents the expert architectures designed by both variants of AutoMoE across datasets. On an average, more than 69% of the experts get assigned the encoder compared to the decoder. As result, overall capacity of the encoder significantly increased by introducing more experts, without having a drastic impact on the overall model latency and FLOPs. On average, more than 70% of the expert layers in 'fract-expert' architectures have 2 or more experts, out of which more than 75% of the expert layers have varying capacities (i.e.. expert with different FFN intermediate size).
be
Model Dataset
Encoder
Decoder
#Experts per layer
Expert FFN Inter Size
#Experts per layer
Expert Size
PPN
Inter
Std-expert WM En
3-2-4-1-5:2
2048 3072 3072-30777777 301-333--3377---- 3072-3072
2-1-1-1
3048-3072-
3072
W4 En-Fr
1-1-3-1-3-3
007-2048-3072--472--7777
2-1-1-1
3072-3072-3072-
3072
WM En-De
1-2-5-1-2-4
3072-3072-3072
4-4-1-1
3072-3072-3072-
3072
Fract-expert WM En-De
3-2-3-4-1-3
3-1-1-1
(3072-1024-2048)-
(3072-3072- 10244-33022221
3072-3072-3072
3072 2048) (3072
3072]
IM En-Pr
6-2-3-4-4-5
2244-1104-2044)
1124-2048-1024-110 1024-1024-
2-1-4-2
[3072-3072)-3072-
(072--2048-2044 22--202337-33-3772
13072-3072-3072
2048]-13072 (3072-307) 2048 3072]
2048]-[3072 (3072-2048)
[3072- 1024 1024-2048)
1M En-De
2.3-1-266-1
[3072.007] (3072)
2-4-1-1
(3072-3072) 1024-2048-37723 3072-3072
00A-3072-11224-3388--37777
Table 7: AutoMoE-generated Pareto-optimal architectures for different datasets. FFN intermediate sizes for fractional experts (i.e. varying expert sizes within each layer) are enclosed within square brackets.
Full Architecture Design. Figure 4 architectures generated AutoMoE.
5 and present the full architecture design of pareto-efficient
Partially Gold Latency vs. Latency Predictor. Soolws the comparison of different latency estimators: proposed partially gold latency and latency predictor. On all datasets, proposed par-
15

---

Under review as a conference paper at ICLR 2023
5072 FFN-lnter
Expert
Encoder Layer
Router (token)
8 Selt Attention-Heade
Output
2048 FiN-inter
Expert
Expert
Decoder Layer
Encoder
(token)
Router (token)
8 Self Self-Attenttonn-een
Cross Attention-
FEN--1nter Esperrs
2
Encoder Layer4
3F72 FFN weer
1 Expert
Decoder Layer
Router
ASclf-Attention-Heze
Expert
Heads
Sell-Attennioo--leddd
Router (token)
3072 Inter
Expert
Self- Sollf-Attention--e Heads
Decoder Layer
Router (token)
Encoder Layen 2
Cross Attention Heads
Router (token)
Self Self-Attention-Headda
Seai--Attentio-Heass
Expert
Decoder Layer
Encoddr Lyer
Router (token)
Router (token)
Attention
Self-Atttentoon-Heaadk
Sclff-Attention-Hoadd
FmbbeddingSSie
Embedding.soo
Figure 4: AutoMoS-generated architecture for WMT" 14 En-De.
16

---

Under review as a conference paper at ICLR 2023
Encoder Layer
Router (token)
Sell-Atennioon-eeaee
Output
3072 FFN-Inder Experts
Encoder Layer
Decoder
Router (taken)
Router
8 Sualf-Atteintion-Hie Heads
Cross Attention-
3072 FEN Inter & Experts
Encoder Layer4
3072 FFN vitrr
1 Expert
Decodar Layer
SSelf-Aftention Heads
(token)
FFN-amter Expers
Attention- Heads
Encoder Layer
Self-Attention Heads
Router (token)
FTA-inter
Expert
Sell--Attertion-iieeed
Decoder Layer
Router (token)
Encoder Layen2
A Crows Attention Heads
Router (token)
8 Self-AttenntionHHaaad
Decoder Layer
Expert
Envoder Layer
Router (token)
Router (token)
Attention
SelfAAttentioo--teadk
Self-Attention-Hess Heads
FmbbeddingSiie
Embedding.soo
Figure 5: AutoMoB-gennrated architecture for WMT" 14 En-Fr.
17

---

Under review as a conference paper at ICLR 2023
Envoddr Layer B
Router (token)
Self Atwenttonnneneeee
Output
3072 FFN-inerr
Niinter, Experts
Encoder Layer
Decoder Leyer
Router (token)
Router (token)
8 Self Seaf-AAttenioon--eess Heads
FF7 Inter
9eelf-AttentionHoadd
1 Expert
Encoder Layer4
3672 FEN-iner
Expert
Decoder Layer
(token)
@.Seff-Ateen Sellf-ttentioon-ieet Heads
Router (token)
Ne-i-teer Experts
Attention Heads
2
Self-Attention Heads
Encoder
Router (token)
FTA-inter
Expert
Self-Aittention--leedd
Decoder Layer
Router (token)
Encoder Layer
A Cross Attention Heads
8 Sell-AAttention-HHaad
Router (token)
Self-Artention-Haade
1 Expert
Decoder Layer
Expert
Enooder Layer1 1
Router
Router (token)
Attention
Self-Atttention-ieeadk
Fribedding-Sive
Embedding.soo
Figure 6: AutoMoS-generated architecture for WMT" 19 En-De.
18

---

Under review as a conference paper at ICLR 2023
Dataset
Latency Estimator
#Active Params (M)
Sparsity
FLOPs (G)
BLEU
Lateney (ms)
(%)
WM En-De
AutoMoFF Experts)
Panially Latency
29
AutoMoE (2 Exports)
Latency Prodictor
46
29
2,9
28.2
189
WM En-Fr
AutoMoE (2 Experts
Partallly Gold Latency
29
2.9
41.2
176
AutoMoB Experts)
Latency Predictor
41.1
19
AutoMoB (2 Experts)
Partially Gold Lateney
40
2.8
45.5
180
AutoMoE: (2 Experts)
Latency Predictor
40
3.2
46.0
229
Table 8: Latency improvement with the proposed partially gold latency vs. latency predictor. We report both active model parameters; and sparsity measured as non-active parameters percentage of total parameters.
Search Constraint
BLEU FLOPs
Latency (ms)
(G)
Latency & 200ms
aNok (2 Experts)
41.23
2.9
176
aMoz Experts)
FLOPS S GFLOPs
Experts)
41.09
3.0
216
oMt (4 Experts)
41.10
3.0
229
Table 9: Impact of different search constraints
WMT 14 En-Fr dataset.
tially gold lateney yields better latency, for same better FL.OPs, BLEU, active model parameters and sparsity.
Latency vs. FLOPs as constraint for search. Table presents the impact of latency and FLOPs as computational constraints the performancc-efficcinnyy trade-off. Constraining FLOPs results in models that fully exhaust the FLOPs budget for GFLOPs and GFLOPs; while leading to higher latency. On the other hand, constraining the latency tends to underutilize the budget and leads to relatively superior FLOPs and latency, thereby providing a stricter control.
Dataset Year
Source Target #Train
#Valid
#Test
Lang
Lang
WMT
2014
English
German 4.5M
3000
3000
(en)
(de)
WMT
2019
English German 43M
2900
2900
WMT
2014
English French 35M
26000
26000
(en)
Table 10: Machine translation benchmark data.
19


Reviews:
[{"review_id": "o4XH_D7CCh", "reviewer": "Reviewer_CMSP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces AutoMoE, an AutoML framework for searching efficient sparsely activated models. Experiments are conducted on three machine translation benchmark datasets, including WMT'14 En-De, WMT'14 En-Fr, and WMT'19 En-De. Compared with previous models (e.g., HAT), AutoMoE achieves a better trade-off between FLOPs and BLEU scores.  ", "summary_of_the_review": "I have some concerns regarding the novelty of this work. And it is unclear to me whether AutoMoE is better than previous models when comparing their efficiency on hardware (e.g., GPU).  ", "strength_and_weaknesses": "Strength:\n1. This paper is well-written. \n2. The practical improvements over previous methods are clear, and the search cost is affordable. \n\nWeakness:\n1. 1. I feel this work is a straightforward application of existing NAS methods on MoE with some engineering efforts. The novelty of the proposed framework is limited. According to the experiments, the improvements are not surprising, given that the search space and the overall framework are much more complicated than previous methods. \n2. In addition to FLOPs, the GPU latency of the models should be included in Table 4. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6AOAgzVgIbB", "reviewer": "Reviewer_vutH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on neural machine translation and develops a new NAS approach. Specifically, the authors propose a transformer-based search space by incorporating the MoE module into it. Nevertheless, the novelty seems very limited since the proposed method can be regarded as a direct application of HAT on MoE architectures. Moreover, the performance improvement over HAT is marginal.", "summary_of_the_review": "The novelty is limited since the proposed method can be regarded as a direct application of HAT on MoE architectures. Moreover, the performance improvement over HAT is marginal.\nMaybe I missed something important. If there are some other essential differences, I am happy to change the rating.", "strength_and_weaknesses": "Strengths:\n1. This paper develops a new NAS approach by incorporating MoE module into the search space\n2. This paper provides sufficient details of the proposed search space as well as interesting analysis.\n\n\nWeaknesses:\n\n1. Compared with HAT, the novelty of this paper seems very limited. If I understand correctly, the essential difference from HAT is including the MoE module into the search space. Besides this, all the others (including supernet training, evolutionary search algorithm) remain exactly the same as HAT.\n\n2. How do the authors evaluate latency in Table 1? If the latency is measured based on a single sample, the practical latency for a mini-batch may be very slow since different sub-networks are activated for different inputs. Thus, it would be better to provide the comparisons of latency on both single sample and a mini-batch.\n\n3. The performance improvement over HAT seems very marginal in Table 4.\n\n4. In NAS papers, random search is always a strong baseline. Thus, it would be better to include the comparisons with random search in Table 4.\n\n5. In Table 5, this paper only shows the results of the proposed method. It is interesting to see the comparisons between the proposed method and HAT, which is also able to produce architectures under specific constraints. \n\n6. Does AutoMoE also need to train a latency predictor like HAT? If not, how to efficiently obtain the latency for candidate architectures during the search process?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Tso3M4JZm", "reviewer": "Reviewer_Yxc3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies efficient sparsely activated transformers. The authors expand the search space of transformers with a few mixture-of-experts (MoE) design choices: i.e., the number and size of experts after each transformer block. Then, the authors apply weight-sharing super network training and evolutionary search to explore the best sparsely activated MoE architecture under the given resource constraints. The proposed AutoMoE has achieved moderate #FLOPs reduction compared with the dense transformer.", "summary_of_the_review": "My current recommendation is primarily based on the limited novelty of this paper. However, I would love to see the authors' response before making the final recommendation.", "strength_and_weaknesses": "---\nStrengths:\n- The paper is very well-written and easy to follow.\n- The proposed framework is technically sound and performs well on machine translation benchmarks.\n- The paper provides sufficient implementation details (e.g., design space configuration, and search and training hyperparameters). The submission also comes with a code implementation, which will facilitate other researchers' reproduction.\n\n---\nWeaknesses:\n- The technical novelty of this paper is a bit limited. The core contribution lies in the search space design since the other components (i.e., super network training and evolutionary search) are standard (used in most neural architecture search papers). However, the authors adopt most of their search space designs from HAT except those related to MoE. The support for elastic MoE also seems straightforward (basically the same as FFN). Though I like the idea of incorporating MoE into the transformer's design space, I am not confident whether these technical contributions are sufficient for publication.\n- The experimental evaluation needs improvement:\n  - Despite mentioning the baseline with homogeneous experts in the text, the authors have not provided the corresponding results in Table 4.\n  - The authors have only presented #FLOPs in Table 4. However, #FLOPs reduction does not necessarily translate into measured speedup.\n  - The authors have compared AutoMoE with manually-designed baselines in Table 1. However, I suspect the shallower encoder contributes the most to the latency and #FLOPs reduction.\n\n---", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject