Merging Experts into One: Improving Computational Efficiency of Mixture of Experts
Authors: Shwai He
Run-Ze Fan
Liang Ding
Li Shen
Tianyi Zhou
Dacheng Tao
Venue: EMNLP 2023 Main (2023)
Keywords: Mixture of Experts
Computational Efficiency
TL;DR: In this work, we propose merging experts into one (MEO) as a drop-in replacement of the mixture of experts (MoE) to significantly improve the computational efficiency, e.g., reducing FLOPs from 72.0G to 28.6G.

Abstract: Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of growing computational cost. Although a sparse Mixture of Experts (MoE) can reduce the cost by activating a small subset of parameters (e.g., one expert) for each input, its computation escalates significantly if increasing the number of activated experts, limiting its practical utility. Can we retain the advantages of adding more experts without substantially increasing the computational costs? In this paper, we first demonstrate the superiority of selecting multiple experts and then propose a computation-efficient approach called \textbf{\texttt{Merging Experts into One}} (MEO), which reduces the computation cost to that of a single expert. Extensive experiments show that MEO significantly improves computational efficiency, e.g., FLOPS drops from 72.0G of vanilla MoE to 28.6G (MEO). Moreover, we propose a token-level attention block that further enhances the efficiency and performance of token-level MEO, e.g., 83.3\% (MEO) vs. 82.6\% (vanilla MoE) average score on the GLUE benchmark. Our code will be released upon acceptance. Code will be released at: \url{https://github.com/Shwai-He/MEO}.

Merging Experts into One:
Improving Computational Efficiency of Mixture of Experts
Shwai He! Run-Ze Fan Liang Ding Li Shen
Tianyi Zhou' Dacheng Tao?
'University of Maryland, College Park The University of Sydney
Universiity of Chinese Academy of Sciences "JD Explore Academy
shwaiheeumd.com edu, liangding. liamegmail.com
tianyleumd.cm
Abstract
650
660-
Scaling the size of language models usually leads to remarkable advancements in NLP tasks. But it often comes with a price of grow- ing computational cost. Although sparse Mix- ture of Experts (MoE) can reduce the cost by ac- tivating a small subset of parameters (e.g., one expert) for each input, its computation escalates significantly if increasing the number of acti- vated experts, limiting its practical utility. Can we retain the advantages of adding more ex- perts without substantially increasing the com- putational costs? In this paper, we first demon- strate the superiority of selecting multiple ex- perts and then propose a computation-efficien approach called Merging Experts into One (MEO), which reduces the computation cost to that of a single expert. Extensive experi- ments show that MEO significantly improves computational efficiency, e.g., FLOPS drops from 72.0G of vanilla MoE to 28. 28.9G (MEO). Moreover, we propose token-level attention block that further enhances the efficiency and performance of token-level MEO, e.g., 83.3% (MEO) vs. 82.6% (vanilla MoB) average score on the GLUE benchmark. Our code will be re- leased upon acceptance. Code will be released at: attps://gil /hgitbb...tbt..../m thub.
640
47.-
670
12-12"
10"
2A 10
12"10"
Performance (S)
Small =MEO
Small WAMoE
Bare-WIMEO
(ath 1.2m Loft Floeling Point Operations (FLOP))
2.4 iot
Figure I: Performance vs. FLOPs of MoE and MEO the token level when different numbers of experts (i.e., 2, 4, 8, 16, 32) are selected. We take three different sizes of BERT as the expert model.
Although increasing the experts selected for each input can improve the representation diver- sity (Yang ct al., 2019) and downstream task per- formance (Shazeer et al., 2017; Yang et al., 2019), it usually comes with a price of significantly growing computational cost. Our empirical study (Figure and Table 1) verifies the Pros and Cons (superior performance vs. high computational cost) of se- lecting multiple experts at MoE inference. Hence, to retain the advantage of MoE on computational efficiency, existing work mainly selects only one expert per input in applications or experiments (Fe- dus et al., 2021), which inevitably compromises the performance.
1 Introduction
Scaling language models has achieved promising progress in the field of NLP (Brown et al., 2020; OpenAI, 2023). To further increase the model size under a computational budget, sparsely activated networks (Du et al., 2022; Artetxe et al., 2022) only employ few parameters for each input. A widely studied approach is the Mfxture-of-Exppee Expprts (MoE, Shazeer et al., 2017), which trains multiple expert networks but only selects subset of them for a specific input (Jacobs et al., 1991; Jordan and Jacobs, 1994). Compared dense networks of the same model size, MoE effectively reduces computational costs.
Our work aims to improve the computational efficiency of MoE inference with multiple experts selected, for greatly rejuvenating the compromised performance. The computation involved in MoE. primarily consists of the inference each selected expert and the summation of their outputs, with the former dominating the cost, Hence, the cost linearly grows with the number of selected experts. To overcome the computational bottleneck, we in- stead propose Merging Experts into One (MEO),
"Corresponding author

---

Input
Output
Wrg. 0g
W1. 01
Input.x x
Wa, By
Output
Gating Network
Wa. bg
W.B
Gating
Wa. 0g
Y/3
Network
W3. 0g
Bgg
(a) MoE
(b) MEO
Figure 2: The diagrams of (a) MoE and (b) our proposed MEO, with case of experts are selected. MoE linearly combines the outputs from experts, while MEO first merges experts into one and then computes the input.
which alters the calculation order of the two op- erations, i.e., first merging the parameters of the selected experts into one expert followed by infer- ence on the merged expert. Since the parameter merging only requires summation, an MEO layer (approximately) only consumes the computation of single-expert inference, no matter how many ex- perts are selected. This leads to nearly constant inference cost when scaling up the model capacity (i.c., the number of selected experts) to improve the performance.
ding (or task ids):
GATE(a:).
Token-level
G(a.)
GATE( > nz), Sequence-level, GATE(task_idas
Task-level
where "GATE" denotes the gating function.
Table 1: Effects of the number of selected experts on performance. The results are bold.
#FLOPs.
SST-2
STSB
MNLI
QNLI
Avg.
MEO can be applied as drop-in replacement for MoE, which has been deployed at various lev- els, e.g., selecting experts for each token (Shazeer al., 2017), each sequence (Ye et al., 2022), each task (Kudugunta al., 2021), etc. On the se- quence/task level, our empirical studies demon- strate that replacing MoE with MEO significantly improves computational efficiency, e.g., reducing FLOPs from 72.0G to 28.9G, without hurting the performance. In addition, we propose token-level attention mechanism that further enhances the effi- ciency and performance, e.g., from 82.6% (MoE) 83.3% (MEO) on BERT-Base (Figure 1).
7.5G
86.1
77.8
2
9.6G
87.9
86.8
78.2
86.2
84.8
13.9G
88.2
87.1
78.3
86.4
85.0
8
22.5G
88.3
87.7
79.1
86.8
85,5
16
39.7G
88.4
87.5
78.8
86.6
85.3
32
74.1G
88.2
87.6
78.6
85.2
Motivation. While many predominant MoE mod- els tend to select the top-1 expert (Fedus et al., 2021), selecting multiple experts has the potential of boosting the representation power (Chen et al., 2020; Yang et al., 2019). Empirically, we conduct preliminary experiments the BERT-Small (Bhar- gava et al., 2021) to verify it.
2 Methodology
In Table 1, it evident that selecting multiple experts contributes to better performance. Even though selecting excessive experts is suboptimal as introduces the interference between experts that hinders the performance (Mustafa et al., 2022; Zhu al.. 2022), our preliminary experiments necessi- tates the selection of multiple experts.
Review of Mixture of Experts. Given a token in the input sequence x Raad MoE selects experts from (m n) experts (Ei,..., En) based on gating network. We denote as the gat- ing scores and T the indices of selected experts. MoE linearly combines the outputs of selected ex- perts:
However, selecting more experts leads to a sub- stantial increase in FLOPs (e.g., 74. v.s. 7.5G when increasing from to 32). This phc- nomenon urges reflect whether there exists an efficient approach to achieve both high perfor- mance and computational efficiency. Our goal is to ensure consistent computational cost, regardless of the number selected experts.
EnST Gu(os) F& (T:).
MoE. performs at various levels, e.g., token, se- quence, and task, where MoE selects experts based on a single token, input sequence, or task embed-

---

Merging Experts into One. The computation cost of MoE primarily involves the computation of individual experts (i.e., ExeT O(E)) and the mixture of outputs from experts (i.e., Ogg) and k(Eke 9k- F.)). Notably, the computation of individual experts plays dominant role, with even the cost of single expert being significantly out- weighing that of the mixture:
However, when directly applying MEO the token level, the situation is different. Since the gating scores of each token within a sequence are unique, the straightforward usage of MEO would require the aggregation of multiple sets of weights and biases, resulting in increased deployment cost. Therefore, we refine and enhance the framework of token-level MEO specifically.
O > O + O 5 9h- Ek);
(3)
Token-Level MEO. Our proposed token-level MEO aims to incorporate token-level information with minimal extra computational cost. Specifi- cally, the expert selection is performed at the se- quence level, thereby preserving context informa- tion and eliminating the necessity of aggregating multiple weights and biases for individual tokens. To capture the identification of each token, we leverage the token attention mechanism inspired by Houlsby et al. (2019); Li et al. (2021).
KET
where O-)) measures the computational cost. On the other hand, as the number of selected experts m increases, the term ΣKET O(Fa) ex- periences a substantial increase, whereas the in- crease in O(E GR Eg) is marginal. There- fore, it is essential to address the growing trend of ΣKet O(Ex) to enhance computational efficiency.
As illustrated in Figure 2, we propose the method called Merging Experts into One (MEO), where the key idea is leverage the gating scores to aggregate the parameters of the selected experts (which is akin to the simple weighted model fusion mechanism (Li et al., 2023)):
Specifically, given the input sequence E Rxxd we employ specialized bottleneck block, inspired by adapter-like structures (Houlsby et al., 2019; Pfeiffer et al., 2021). The bottleneck layer incor- porates down-projection weights W Wdown C dx an activation function f and up-projection weights Wup R xd with reduce factor = 64 that en- sures low extra computational cost. By operating on each token individually, the bottleneck applies token-level attention to the input sequence x::
W.- I9((a)) WR. by RET
[n((x)) by (4)
RET
where Wx, by represent the weight and bias of the k-th expert, while W., b, are the aggregated weight and bias for x4 The output of MEO is given by:
- x+ f( W Wewnn) W op
(6)
With the inclusion of token identification in the updated input, MEO performs aggregation of W and b through sequence-level expert selection. Sub- sequently, these aggregated W and b are used compute the output in conjunction with the input.
y: = a + 6,),
(5)
where a represents the activation function. The computation cost of MEO primarily con- sists of (W,,, + b,)), OlEkeT Gx Wi). O Gk bR), and O Among them. Olo(Wi,, + b)) is the dominant factor. It is worth noting that WWoWW, + b,)) is equivalent to the computation cost of a fully connected net- work and independent of the number of selected experts. Therefore, MEO compresses computation costs significantly.
Empirical Evaluation
Experimental Setup. Experiments were con- ducted on Four widely-used benchmarks, span- ning understanding and generation tasks: (1) GLUE (Wang et al., 2019), containing understand- ing tasks like natural language inference, sentiment analysis, and sentence similarity evaluation; (2) XSum (Narayan et al., 2018) 8), summarization dataset where the models are required to generate short summary for given article; (3) WikiText- (Merity et al., 2016), collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia where the models are utilized to generate the next tokens; (4)
MEO at Difffrrnt Levels. In the case of se- quence and task level MEO, all tokens within a sequence share the same gating scores, as well as the aggregated parameters W and ó !. This property allows for easy adoption of MEO these levels.
'e omit subscripts of W and at the sequence and task level given each token shares the same aggregated parameters.

---

Table 2: Empirical results for MEO and MoE in task-level (lask) and sequence-level (seg). We also report the performance of vanilla feedforward layers ("Vanilla") as reference. The shown results are the averaged score for 5 runs. The best results are bold. * indicates the method with the fewest the fewer FLOPs ("Vanilla" is not included).
Method
#LOOPs.
CoLA
SST-2
MRPC
STS-B
QQP
MNLI
QNLI
RTE
Avg
Vanilla
28.5G
54.6
91.1
84.6
85.8
90.2
80.6
90.4
66.4
80.5
MoErask
72.0G
58.5
91.3
85.8
89.2
90.5
82.7
90.5
69.3
82.2
+28.9G
59.1
91.2
85.5
89.3
90.4
83.0
90.9
68.9
82.3
MoEscg
72.0G
59.8
91.5
86.5
89.5
90.6
83.4
90.7
70.4
82.8
MEOsag
*28.9G
60.1
91.9
86.3
89.4
90.7
83.7
70.3
83.0
SQuAD v1.1 (Rajpurkar et al., 2016), pair-wise dataset for questions and Wikipedia paragraphs where models select the answer span to the ques- tion from the paragraph.
We follow Zhong et al. (2022a.b); He et al. (2023a) to conduct experiments on the widely-used GLUE benchmark, containing understanding tasks like natural language inference, sentiment analy- sis, sentence similarity evaluation, etc. We use Adam (Kingma and Ba, 2015) the optimizer with 31, B2 0.9, 0.98. For regularization, we the weight decay as 0.1 and grid-search the learning rate from (le-5, Se-5, le-4, 5e-4), where we warm up the learning rate in the first 10% steps (of the total training steps). For different data scales, we grircs the training epoch and batch size from (5, 10, 15, 20), and (8, 16, 32, 64), respectively. The maximum length is 128 for GLUE, 1024 for WikiTeex, and 384 for SQuAD. For XSum, we the max length of source articles be 512 and the max length of the target summary be 128. We follow previous works (Phang et al., 2018; Lee et al., 2020; Dodge et al., 2020; Wang et al., 2022; He et al.. 2023b) to fine-tune the pretrained lan- guage models, e.g. BERT (Devlin et al., 2019), on the downstream training set and report results using the last checkpoint.
Inference
16
Selected Experts
Comparison of inference time between MoE and MEO under series of different numbers of selected experts (i.e., 2, 4, 16).
Figure 3:
Analysis of Reduced Computation. Compared to a fully connected layer, MEO only intro- duces computation in gating network (z)()) and merging experts (i.e., O(EtTT 9r Wv) and O{Σ Gx bx)). The additional computation is minimal compared to that of individual experts.
In practice, we use eight NVIDIA V100 Tensor Core GPUs to measure the inference time MEO and MoE on BERT-Base when selecting different numbers experts (i.e., n 1, 4, 8, 16). Infer- ence time is calculated by the total running time on the MNLI validation dataset with batch size 16. According to Figure 3, as the number of selected experts increases, the inference time of MEO is relatively consistent, while MoE exhibits signif- icantly increased inference time. This highlights the advantage of MEO in computational efficiency, which becomes even more pronounced as the num- ber of selected experts grows.
Main Results. Following Shazeer et al. (2017); Gao al. (2022), conduct experiments on BERT-Base (Devlin al., 2019) and replace feed- forward layers ("Vanilla") with MoE or MEO, with the setting m = 4 and n = 16. In Table 2, we carefully compare our proposed MEO with MoE at task and sequence levels, in terms of computational efficiency and performance. Compared to MoF, MEO significantly reduces the computation cost while achieving comparable performance. Specifi- cally, compared to vanilla feed-forward layers, the Floating Point Operations (FL.Oss) of MEO only increase marginally (i.c., about 1%), while MoE multiplies the FLOPs about 2.53 times.
Table 3: Comparison between MEO and MoE with different activation function usage (i.e., activation function within (in) and outside (ouf) experts).
Method
FLOPs
SST-2
QQP
MNLI
QNLI
Avg.
Vanilla
7.5G
86.9
89.1
77.2
85.2
84.6
MoE..,
22.6G
87.9
89.4
77.8
85.7
85.2
MoHout
22.5G
87.6
89.2
78.0
85.6
85.1
MEO
*7.7G
88.1
89.7
78.2
86.2
85.6

---

MoE 2NN on BERT-Base and 77.8% V.S. 77.3% on BERT- Small.
88.0
MEO
We also implement the token-level MEO on BERT-Large, utilizing 8 experts and selecting 2 experts, resulting in a model with about 1.75 bil- lion parameters, As demonstrated in Table 4, MEO consistently enhances performance across various tasks, e.g.. 0,4% improvement in SST-2 when com- pared MoE. Notably, the additional computa- tional cost is minimal, with only 4.0 GFLOPs increase over the Vanilla model. Therefore, token- level MEO proves to be an efficient and effective alternative to token-level MoE.
86.0
Performance
78.0
Figure 4: Performance comparison between token level MoEE and MEO, where we take BERTT-Smal as the backbone with the setting m and 32.
Analysis of Activation Function. In many cases where an expert EG represents linear layer without nonlinear activation function, the output of MEO (y: = (W, + 6.)) is equivalent to that of MoE (y/ = < 9k (Waæ; + by))). However, the expert E; involves an activation function, the output of MoE is y; = EkeT(Gn (Wa,;; bx)), which leads to differences in outputs and poten- tially in performance. As depicted in Figure 3. we compare MEO with MoE with different usage of activation, where we consider two scenarios: activa- tion function within or outside experts. The results demonstrate that the performance gap between the two scenarios is minimal and indicates the effee- tiveness of MEO in handling expert networks that incorporate activation functions.
Transfer to different architectures and tasks. Utilizing MEO in BERT architectures enhances computational efficiency and performance, and we further validate the effectiveness of MEO on a wide range of architectures for different tasks. In Table 5, we use BART-Large (Lewis et al., 2020) for XSum (Narayan et al., 2018), GPT-2-Small (Radford et al., 2019) for WikiText (Merity et al., 2016), and T5- Base (Raffel et al., 2020) for SQuAD (Rajpurkar et al., 2016). MEO and MoE are deployed at the token level. Considering the limited computation resource, we
2 and 8 for BART and GPT-2, while m and 16 are set for T5. Clearly, MEO outperforms the standard MoE in three tasks, showing its universality in both natural language understanding and generation.
Table 4: Performance of Token-Level MEO, where we take BERT-Large as the backbone with the setting m = and n = 8.
Table 5: Effectiveness on different architectures and tasks. XSum, WikiText, and SQuAD evaluated with ROUGE-2 (R2.). Perplexity (PPL), and Exact Match (EM), respectively.
Method
FLOPs
SST-2
MRPC
STSB
QNLI
Avg.
Vanilla
87.2G
86.8
89.1
91.8
90.2
MoF.
139.0G
93.7
87.2
89.7
92.2
90.7
Method
XSum
WikiText
SQuAD
MEO
*91.2G
94.1
87.5
89.8
92.4
91.0
FLOPs
R2.
FLOPs
PPL
FLOPs
EM
Vanilla
369.4G
21.9
295.4G
21.9
90.2G
81.6
Effectiveness of Token-Level MEO. For MEO the token-level MEO, we have incorporated teven-level attention blocks. To assess the deploy- ment cost of newly added blocks, we first calculate the extra parameters and FLOOSs with BERT-Small as the backbone. The extra cost of added blocks is minimal (i.e.. 0.6M parameters and 0.15 GFLOPs). Furthermore, in Figure 4, we present performance comparison between token level MEO and MoE in four natural language understanding tasks, where MEO outperforms MoE consistently across these tasks, e.g., 78.9% v.s. 78. 1% on MNLI For the av- crage score on the GLUE benchmark, MEO boosts the performance significantll, i.e. 83.3% v.s. 82.6%
MoE
576.6G
22.2
412.2G
21.1
221.3G
82.0
MEO
*383.6G
22.4
*303.2G
20.9
*93.5G
82.1
Conclusion
In this work, we systematically investigate the com- putational cost of the Mixture of Experts. Based on our findings, we propose a drop-in replacement called Merging Experts into One (MEO) to enhance computational efficiency. Additionally, we propose Token-Level attention mechanism that further boosts performance. Our study empirically indi- cates the potential to make MEO a golden standard efficient architecture within the NLP community.

---

5 Limitations
2020. Language models are few-shot learners. NewrlPS.
Despite the progress we have made, there are still limitations in our work. While our architecture for the mixture of experts demonstrates improved ef- ficiency, there is a need for further exploration in terms of its deployment. Specifically, determining the optimal number of experts in specific layers and selecting different levels of MoEs require ad- ditional investigation. We believe that with the implementation of efficient deployment strategies, our method has the potential to become even more competitive.
Yinpeng Chen, Xiyang Dai, Mengchen I Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020. Dynamic convolution: Attention over convolution kernels. In CVPR.
Jacob Devlin, Ming-Wei Beng-Wei Chang, Kenton Lee, and Kristina Toutanova, 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah Smith. 2020. Fine-tuning pretrained language models: Weight ini- tializations, data orders, and early stopping. arXiv preprint.
Acknowledgements
We are grateful to the anonymous EMNLP review- ers and the area chair for their insightful comments and suggestions.
Nan Du. Yanping Huang, Andrew Dai, Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus. Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier- Hellstern, Toju Duke, Lucas Dixon, Kun Zhang. Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. Glam: Efficient scaling of language mod- els with mfxture-of-experts In ICML.
Ethics Statement
We take ethical considerations seriously and strictly adhere to the EMNLP Ethics Policy. This paper fo- cuses the higher efficiency of dynamic networks, e.g., the mixture of experts. Both the datasets and models used in this paper are publicly available and have been widely adopted by researchers. We en- sure that the findings and conclusions of this paper are reported accurately and objectively.
William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Mach. Learn. Res.
7e-Feng Gao, Peiyu Liu, Wayne Xin Zhao, Zhong- Yi Lu, and Ji-Rong Wen. 2022. Parameter-efficcee mixture- offexee-of-experts architecture for pre-trained lan- guage models. In COLING.
References
Mikel Arietxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Out, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O' O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa Kozareva, and Veselin Stoyanov. 2022. Efficient large scale language modeling with mixtures of ex- perts. In EMNLP.
Shwai He, Liang Ding, Daize Dong, Boan Liu, Fuqiang Yu, and Dacheng Tao. 2023a. PAD-net: An efficient framework for dynamic networks. In ACL.
Shwai He, Run-Ze Fan, Liang Ding, Li Shen, Tianyi Zhou, and Dacheng Tao. 2023b. Mera: Merging pretrained adapters for few-shot learning. arXiv preprint.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilbe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nip. In ICML.
Prajjwal Bhargava, Aleksandr Drozd, and Anna Rogers. 2021. Generalization in NLI: Ways (not) to go be- yond simple heuristics. In the Second Workshop on Insights from Negative Results in NLP.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretehen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Firie Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.
Robert A. Jacobs. Michael Jordan, Steven Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Compur.
Michael Jordan and Robert A. Jacobs. 1994. Hier- archical mixtures of experts and the EM algorithm. Neural Comput.
Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.

---

Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Lu- ong, and Orhan Firat. 2021. Beyond distillation: Task-level mixture-of-experts for efficient inference. In EMNLP.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le. Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural net- works: The sparsely-gated mixture-of-experls layer. In ICLR.
Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. 2020. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In ICLR.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising requence-to-sequeee pre-training for natural language generation, translation, and com- prehension. In ACL.
Yequan Wang, Hengran Zhang, Aixin Sun, and Xuying Meng. 2022. Cort: A new baseline for comparative opinion classification by dual prompts. In EMNLP.
Brandon Yang. Gabriel Bender, Quoe Le, and Jiquan Ngiam. 2019. Condconv: Conditionally parameter- ized convolutions for efficient inference. In NeurlPS.
Chao Li, Aojun Zhou, and Anbang Yao. 2021. Omni- dimensional dynamic convolution. In ICLR.
Qinyuan Ye, Juan Zha, and Xiang Ren. 2022. Eliciting and understanding cross-task skills with task-level mixture-of- experts. In EMNLP.
Weishi Li. Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. 2023. Deep model fusion: A survey. arXiv preprint.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022a. E2s2: Encoding-enhanced sequence-to-seqquece pretraining for language un- derstanding and generation. arXiv preprint.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod-
cls.
Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhua Liu, Bo Du, and Dacheng Tao. 2022b. Improv- ing sharpness-awe sarpeess-aware aware minimization with fisher mask for better generalization on language models. In EMNLP.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. 2022. Multi- modal contrastive learning with limoe: the language- image mixture of experts. In NewrIPS.
Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang. Hongsheng Li, Xiaogang Wang, and Jifeng Dai. 2022. Uni-perceiver-man moe: Learning sparse generalist mod- els with conditional moes. In NeurIPS.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for ex- treme summarization. In EMNLP.
OpenAL. 2023. GPT-4 technical report. arXiv preprint.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. EACL.
Jason Phang. Thibault Févry, and Samuel Bowman. 2018. Sentence encoders on stilts: Supplementary training intermediate labeled-data tasks. arXiv preprint.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100, 000+ questions for machine comprehension of text. In EMNLP.


Reviews:
[{"review_id": "XWWhxhWHJX", "reviewer": "Reviewer_6rLC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments! We carefully address your concerns as follows:\n\n## Q1. The difference between MEO and MoE in communication cost. \nThere is no significant difference between their required communication costs, though MEO is faster in terms of the inference speed. Both MEO and MoE require distributing inputs to multiple experts and combining the parameters or outputs of selected experts [1, 2]. \n\n## Q2. The experimental setup. \nA detailed experimental setup can be found in Appendix A and Line 220-225. Due to the limited computation resources, we use a NVIDIA V100 to conduct each experiment. Our deployment of experts is identical to previous works and uses their used pytorch packages to allocate experts in parallel. \n\n## Q3. Performance of selecting more experts. \nThe results in Table 1 demonstrate the advantage of selecting multiple experts (when compared with selecting only one expert) but selecting excessive experts degrades the performance to be suboptimal. This is consistent with previous work [3] results reported in their Appendix. The suboptimal performance, as discussed in Line 113-116, could result from the interference between experts [4, 5]. Moreover, selecting excessive experts reduces the number of possible combinations of selected experts (e.g., 1 when selecting 32 from 32 v.s. 35960 when selecting 4 from 32).\n\n## References: \n[1] Sparse MoE with Random Routing as the New Dropout: Training Bigger and Self-Scalable Models. ICLR 2023. \n\n[2] Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models. COLING  2022. \n\n[3] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. \n\n[4] Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts. NeuIPS 2022. \n\n[5] Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs. NeuIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an interesting angle to reduce the computation cost of the experts in sparse MoE models.\n\n2. The authors provide an analysis of the different levels of merging experts into one, as well as the reduced computation. ", "reasons_to_reject": "1. One significant concern raised in this paper is the lack of discussion on the communication overhead of MoE. It's not clear where the performance gain comes from, basically merging experts into one also reduces the communication (all_to_all).\n\n2. The experiment setup is not clear. For example, how many GPUs are used? what is the deployment of the experts?\n\n3. It seems the performance gain is very limited when selecting more experts in table 1, what is the reason?", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors introduce a new solution to reduce the computation cost of Mixture of Experts models. Existing MoE systems typically involve high computation overhead on individual experts, to reduce these, the authors propose to merge the experts into one, by reordering the computation. Specifically, the parameters of multiple experts are merged into one. The evaluation results show that the proposed approach significantly improves the computational efficiency of MoE.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vanVnZAwqc", "reviewer": "Reviewer_tZMq", "summary": "", "questions": "- Does this approach work for routing token / seq to FFN (e.g. stacks of linear layers with nonlinearities after each one, without having to have a gating mechanism per linear layer?).\n- Why are the inference times for BERT 20+ seconds even for 1 expert? This seems significantly longer than what I'd expect -- what is the batch size?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments, which help us further improve our paper. We carefully address your concerns as follows:\n\n## Q1. The usefulness of our proposed method. \nMost works on MoE recently focus on linear-layer experts [1-3] without nonlinearity and they achieved promising results in practice. So we mainly follow their setting. In Line 236-251, we also discussed how to directly apply our method to nonlinear experts with non-linear activations. The flexible mixture of different networks architecture is out of the main scope of the settings studied in this paper but deserves further exploration in the future. We will add more detailed discussion in the next version.  \n\n## References:\n\n[1] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 2022.\n\n[2] Multi-modal contrastive learning with LIMoE: the language-image mixture of experts. NeuIPS 2022.\n\n[3] VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts. NeuIPS 2022. \n\n## Q2. Experiments on different tasks and model architectures. \nAs suggested by the review comments, we report the experimental results on three different sizes of BERT, including BERT-tiny, BERT-small, BERT-base. To further address your concern, we followed your suggestions and conducted extra experiments on three benchmark tasks (SQuAD, Wikitext and XSum) up on different LMs (including BART-large, GPT-2, T5-base), with the experimental results reported below. \n\n*BART-large on XSum (max_source_length=512, select 2 experts from 8):*\n| | FLOPs | Rouge-2 |\n| :------ | :------: | :------: |\n|Vanilla | 369.3G | 21.9 |\n|MoE | 576.5G\t| 22.2 |\n|**MEO**| **369.7G** | **22.4** |\n\n\n*GPT-2 on Wikitext (max_length=1024, select 2 experts from 8):*\n| | FLOPs | PPL |\n| :------ | :------: | :------: |\n| Vanilla | 279.2G | 21.5 |\n| MoE | 395.9G | 21.1 |\n| **MEO** | **279.3G** | **20.9** |\n\n\n*T5-base on SQuAD (max_length=384, select 4 experts from 16):*\n| | FLOPs | Exact_Match |\n| :------ | :------: | :------: |\n| Vanilla | 90.3G | 81.6 |\n| MoE | 221.0G | 82.0 |\n| **MEO** | **90.4G** | **82.1** |\n\nIn the new experimental results, our proposed MEO achieved consistent improvement across different tasks. Moreover, MEO’s advantage on computational efficiency still holds when applied to different architectures (i.e., BERT, BART, T5, GPT-2), indicating the universality of our method. \n\nWe further evaluate the practical inference speed of Vanilla, MoE and MEO, using T5-base as the base model and “eval_samples_per_second” as the metric. The inference of MEO is significantly faster than that of MoE (176.97 v.s. 106.52), while the speed of Vanilla is 184.7. \n\n## Q3. Applying MEO to FFN layers. \nFollowing existing works on MoE, we mainly focus on replacing linear layers in FFN with MoE/MEO layers. In Line 236-251, we have compared MoE layers with and without nonlinearities, and empirically verified that their difference can only cause minor difference in performance, though their outputs differ. Hence, though we discuss less about nonlinear activations in MoE/MEO layers, our method can be applied to routing token / seq to FFN. \n\n## Q4. The inference speed of MoE and MEO. \nThe inference time (when selecting one expert for each input) includes the computation of gating function and allocating specific experts to each input, which costs more than inference on a single model. The total number of the demos is ~10k and the batch size is 16, so the whole process requires more than 600 iterations, which takes a relatively long time. However, further increasing the batch size will make CUDA out of memory, due to the limited resources and memory on the Nvidia V100.  \n\n## Q5. Line 115 - 74G v 7.4G, should be swapped.\nThank you for pointing out the typo! We will fix it in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Very simple idea that improves efficiency and accuracy of MoE (thought severely limited compared to MoE)\n- Overall improvements on GLUE", "reasons_to_reject": "- The proposed method seems to only work for experts that are essentially an affine transform / linear layer. This is just one interpretation of MoE, and feasibly we can imagine an expert to be any type of network. The method for aggregating experts (e.g. here a weighted sum) only work mathematically if there are no nonlinearities, limiting the usefulness of this approach\n- Approach has only been applied to a single model, BERT-Small, it would be good to contain additional datasets outside of GLUE and a more varied treatment of different model architectures and sizes.", "questions_to_authors": "- Does this approach work for routing token / seq to FFN (e.g. stacks of linear layers with nonlinearities after each one, without having to have a gating mechanism per linear layer?).\n- Why are the inference times for BERT 20+ seconds even for 1 expert? This seems significantly longer than what I'd expect -- what is the batch size?", "paper_topic_and_main_contributions": "The authors propose a new approach for MoE called Merging Experts into One (MEO) which first selects N experts and combines them into a single expert. This method significantly improves computational efficiency (over 2x wrt FLOPS), and outperforms vanilla MoE on GLUE.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 115 - 74 v 7.4G, should be swapped."}, "extra_text": {}}, {"review_id": "OyQIKfZgUX", "reviewer": "Reviewer_92Jb", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your positive comments and support! We carefully address your concerns as follows:\n\n## Q1. While GLUE benchmark is used, the experiments are still limited to mostly NLU tasks. Testing MEO on a wider range of tasks could be beneficial.\n\nThank you for the suggestions! We conducted new experiments on other benchmark tasks beyond the GLUE benchmark, i.e. T5-base on SQuAD, GPT-2 on Wikitext and Bart-large on XSum, the experimental results for token-level MoE and MEO are compared in the tables below. \n\n*BART-large on XSum (max_source_length=512, select 2 experts from 8):*\n| | FLOPs | Rouge-2 |\n| :------ | :------: | :------: |\n| Vanilla | 369.3G | 21.9 |\n| MoE | 576.5G | 22.2 |\n| **MEO** | **369.7G** | **22.4** |\n\n*GPT-2 on Wikitext (max_length=1024, select 2 experts from 8):*\n| | FLOPs | PPL |\n| :------ | :------: | :------: |\n|Vanilla | 279.2G | 21.5 |\n| MoE | 395.9G | 21.1 |\n| **MEO** | **279.3G** | **20.9** |\n\n*T5-base on SQuAD (max_length=384, select 4 experts from 16):*\n| | FLOPs | Exact_Match |\n| :------ | :------: | :------: |\n| Vanilla | 90.3G | 81.6 |\n| MoE | 221.0G | 82.0 |\n| **MEO** | **90.4G** | **82.1** |\n\nOur proposed MEO achieved consistent improvement across different tasks and language models. Moreover, its advantage on computational efficiency still holds when applied to  different mainstream architectures (i.e., Encoder-Only (BERT), Encoder-Decoder (BART and T5), Decoder-Only(GPT-2)), which indicates the universality of our method. \n\n## Q2. Testing MEO on larger size models could be beneficial.\n\nWhen applied to 16-expert MEO or MoE, the total number of parameters in BERT-base is about 1 billion. We also conducted experiments on BERT-large with 8 token-level experts (1.75 billion parameters). The new experimental results are reported below: \n\nBERT-large on the GLUE benchmark (max_length=128, select 2 experts from 8):\n| | FLOPs | SST-2 | MRPC | STSB | QNLI\t | Avg. |\n| :------ | :------: | :------: | :------: | :------: | :------: | :------: |\n| Vanilla | 87.2G | 93.2 | 86.8 | 89.1 | 91.8 | 90.2 |\n| MoE | 138.9G | 93.7 | 87.2 | 89.7 | 92.2 | 90.7 |\n| **MEO** | **87.5G** | **94.1** | **87.5** | **89.8** | **92.4** | **91.0** |\n\nWhen increasing the model size, we also witness the advantages of MEO on both performance and computational efficiency. For different tasks or architectures, please refer to our response to Q1.  \n\n## Q3. Missing References  \nAlthough both our work and AdaMix[1] merge the blocks of language models, our work is fundamentally different from it. First, we focus on enhancing the computational efficiency of MoE, while Aadmix aims at improving the performance of parameter-efficient tuning. Second, the merging operations are different: AdaMix averages the parameters of adapters while our method follows the aggregation process of MoE. That being said, AdaMix’s high-level idea of merging adapters is related. We will cite and discuss AdaMix in the next version. \n\n## References: \n\n[1] AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning, EMNLP 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is clearly written and easy to follow. The method and results are conveyed effectively.\n2.\tThe paper proposes an idea of merging expert parameters before computation to reduce costs. This is a simple but impactful innovation.\n3.\tThe proposed MEO is shown to be effective as a drop-in replacement for MoE at multiple levels (token, sequence, task) with minimal changes. And, the authors further boost efficiency and performance of token-level MEO compared to vanilla token-level MoE inspired token attention.", "reasons_to_reject": "1.\tWhile GLUE benchmark is used, the experiments are still limited to mostly NLU tasks. Testing MEO on a wider range of tasks could be beneficial.\n2.\tOnly BERT-base model is evaluated. Testing MEO on larger size models could be beneficial.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The main topic is improving the computational efficiency of mixture of experts (MoE) models for natural language processing tasks. The key contributions are:\n\n1.\tProposing a new method called Merging Experts into One (MEO) that reduces the computational cost of MoE models to be equivalent to using just a single expert. This is achieved by first merging the parameters of selected experts before computing the input, rather than computing each expert's output separately.\n2.\tDemonstrating that MEO can be applied at various levels (token, sequence, task) as a drop-in replacement for MoE, while significantly reducing FLOPS. For example, it reduces FLOPs from 72G for MoE to 28.6G for MEO at the sequence level.\n3.\tIntroducing a token-level attention mechanism for token-level MEO that further enhances efficiency and performance. MEO with token attention outperforms vanilla MoE, e.g. 83.3% vs 82.6% on GLUE benchmark with BERT-Base.\n4.\tProviding extensive experiments that validate the efficiency and effectiveness of MEO compared to MoE. For instance, MEO achieves comparable performance to MoE on GLUE while using far fewer FLOPs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning -- https://arxiv.org/abs/2205.12410"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to merge experts of mixture of experts models into one expert during inference, which improves inference efficiency and task performance

**Pros**:
- the proposed expert merging technique is simple drop-in replacement to MOE models to effectively improve inference efficiency and task performance.
- The authors provide additional positive results on larger-scale models and on more generative tasks to demonstrate the effectiveness of the approach.

**Cons**: I find that the work doesn't exhibit notable drawbacks, and the concerns and questions raised in the reviews have been adequately resolved.
Decision: Accept-Main