HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts
Authors: Truong Giang Do
Le Huy Khiem
Quang Pham
TrungTin Nguyen
Thanh-Nam Doan
Binh T. Nguyen
Chenghao Liu
Savitha Ramasamy
Xiaoli Li
Steven HOI
Venue: EMNLP 2023 Main (2023)
Keywords: Sparse mixture of experts
hypernetwork
efficient training of LLMs
large language models

Abstract: By routing input tokens to only a few split experts, Sparse Mixture-of-Experts has enabled efficient training of large language models. Recent findings suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts eventually learn similar representations. However, this strategy has two key limitations: (i) the policy derived from random routers might be sub-optimal, and (ii) it requires extensive resources during training and evaluation, leading to limited efficiency gains. This work introduces \texttt{HyperRouter}, which dynamically generates the router's parameters through a fixed hypernetwork and trainable embeddings to achieve a balance between training the routers and freezing them to learn an improved routing policy. Extensive experiments across a wide range of tasks demonstrate the superior performance and efficiency gains of \texttt{HyperRouter} compared to existing routing methods. Our implementation is publicly available at {\url{{https://github.com/giangdip2410/HyperRouter}}}.

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts via HyperNetwork
Giang Doï, Khiem Let, Quang Pham M TrungTin Nguyen", Thanh-Nam Doan Binh T. Nguyen*, Chenghao Liu Savitha Ramasamy', Xiaoli Li', Steven Hoi Institute for Infocomm Research (I'R), A STAR, Singapore, University of Tennessee at Chattanooga, & VinUniversity, A Independent Researcher, 1 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, 38000 Grenoble, France, AISIA Lab, University of Science, Vietnam National University Ho Chi Minh City, Salesforce Research Asia, Singapore Management University x Correspondence to: phquang@i24 2r a a-star.
Abstract
WikIText-103
By routing input tokens to only a few split ex- perts, Sparse Mixture-of- -ettert-ff-Experts has enabled efficient training of large language models. Re-
SMoE
SMoE-Drepaut
HyperRouter
1000
500
Perplexily
cent findings suggest that fixing the routers can achieve competitive performance by alleviating the collapsing problem, where all experts even- tually learn similar representations. However, this strategy has two key limitations: (i) the pol- icy derived from random routers might be sub- optimal, and (ii) it requires extensive resources during training and evaluation, leading to lim- ited efficiency gains. This work introduces HyperRouter, which dynamically generates the router's parameters through fixed hyper- network and trainable embeddings to achieve balance between training the routers and freez- ing them to learn an improved routing policy. Extensive experiments across a wide range of tasks demonstrate the superior performance and efficiency gains of HyperRouter compared to existing routing methods. Our implementation is publicly available at https: //gi t giangdip2411 1ttpddip241///// //Hyper
100
50
Number of Experts
Figure I: Perplexity (log-scaled) on the dataset with varying numbers of experts used for infer- ence. All methods have the same FLOPs.
of the most effective strategies thus far is the Sparse -oxturee-f--Exertts (SMoE) (Shazeer et al., 2017; Fedus et al., 2022b), which utilizes routers to direct each input token to a small subset of network pa- rameters (experts). SMoE improves both efficiency and performance compared to dense training ap- proaches (Lewis et al., 2021; Artetxe et al., 2022; Zhang et al., 2022: Du al., 2022).
Despite the encouraging results, SMoE has been found to encounter the issue of representation col- lapse, where all experts converge to similar rep- resentations (Chi et al., 2022; Chen et al., 2022). This problem arises due to the learning process of the router encouraging experts cluster around centroid (Chi al., 2022). Therefore, significant efforts have been dedicated to addressing the repre- sentation collapse issue while preserving the sim- plicity and efficiency of SMoE training. To this end, one of the most effective strategies is freezing the router, demonstrated by SMoo-Dropon (Chen et al., 2023), where a randomly initialized router re- mains fixed throughout the training process. Addi- tionally, Chen al. (2023) found that progressively increasing the number of selected experts can be beneficial. However, we argue that such a naive strategy exhibits two key limitations. Firstly, the random routing policy may be sub-optimal, which
1 Introduction
Recent years have witnessed tremendous successes of the Transformer model (Vaswani et al., 2017) and its variants across wide range of tasks, rang- ing from natural language and speech processing (Bao al., 2022b; Gulati et al., 2020), computer vision (Dosovitskiy et al.. 2021; Ruiz et al., 2021; Bao et al., 2022a), reinforcement learning (Chow et al., 2023), to life sciences (Rives et al., 2021). Since then, scaling up to larger models has be- come the prevailing approach for advancing the state-of-the-art in pre-training and finetuning tasks. However, training such large models comes with high computational cost (Lin al., 2022); there- fore, there is a growing need develop efficient strategies that facilitate the training of large lan- guage models (LLMs) (Fedus et al., 2022a). One

---

hinders the overall training process. Secondly, we will show in Section 3.3 that a fixed router will restrict the model's representation capabilities, ne- cessitating a progressive increase in the number of chosen experts to achieve satisfactory performance. Therefore, AMoE-Dropond inherently suffers from limited representation and does not offer efficiency gains during inference.
experts converge to similar representations (Chi et al., 2022; Chen al., 2022). Such issues result in poor specialization among experts, parameter redundancy, and limited performance gains. Ex- isting works address the issue (i) by employing various ad hoc-heuristics, c.g., adding Gaussian noise (Shazeer et al., 2017), limiting the maximum number of inputs that can be routed to an expert (Lepikhin al., 2021), imposing load balancing loss (Fedus al., 2022b), using linear assignment (Lewis et al., 2021), or eliminating the necessity for router networks ((Zuo et al., 2022) employed a consistent regularized loss for stochastic expert assignment, (Roller et al., 2021) incorporated deter- ministic hashing assignments). To resolve the issue (ii), (Chi et al., 2022) proposed an X-MOE layer to improve the routing algorithm via dimension duction, normalization, and gating temperature. Furthermore, (Dai et al., 2022) proposed Stable- MoE with two training stages to reduce the router's fluctuations. The work related to ours the most is SMoE-Dropout (Chen et al., 2023), which is considered randomly initialized and fixed router to route tokens. Thus, the routing policy is sta- ble during training and deterministic during infer- ence. Our work improves upon SMoE-Dropout by extending the fixed router to hypernetwork further improve the routing policy during training while alleviating the collapsing issucs.
This work introduces a novel approach called HyperRouter to address the trade-off between fixed and trainable routers in SMoE training. In- stead of employing a random but fixed router, HyperRouter utilizes a random but fixed hyper- network (Ha et al., 2017) to generate the router's parameters based trainable router embedding. By doing so, HyperRouter Router can improve the rout- ing policy throughout training while mitigating the representation collapse issues. To demonstrate its effectiveness, we conduct extensive evaluations on various NLP tasks, comparing HyperRouter to several state-of-the-ee SMoE routing strategies. Moreover, HyperRouter achieves the same perfor- mance threshold with fewer experts (compute) dur- ing inference, thereby significantly enhancing the efficiency of deploying LLMs in real-world applica- tions. Fig. shows that hyperRouted consistently outperforms other competitors when the same num- ber of experts is used during inference.
2 Related Work
3 Methodology
Existing Routing Mechanism for SMoE. One of the most important components for training SMoE is the expert routing strategy, which specifies ex- perts to process input tokens. There are two com- mon classes of token-expert assignment algorithms for SMoE training: (1) letting tokens select the top-k experts and (2) letting experts select the top- k tokens. For the first approach, (Fedus et al., 2022b, Switch Transformer), (Lepikhin et al., 2021, GShard), (Zuo et al., 2022, THOR), (Lewis et al., 2021, BASE), (Clark et al., 2022, S-BASE), and SMoE-Dropout (Chen al., 2023) are represen- tative methods. Meanwhile, Zhou al. (2022) introduced expert choice that enables selecting dif- ferent experts for each token and demonstrates the potential of the second approach.
This section describes SMoE training and details of the proposed HyperRouter method.
3.1 SMoE Training
We first describe SMoE training of LLMs, which consists of router R(-) with parameter W, and N expert networks { with parameters W We follow the most common implementation (Fe- dus al., 2022b) use a linear network as the router and split the feedforward networks LLMs into N separated experts.
Switch Transformer. Given an input token æ with its representation h E Rd, the SMoE output is calculated by routing h only to k-most suitable experts determined by the router, i.e..
N
On the Representation Collapse of SMoE. A major research focus is how improve the token- expert assignment avoid the representation col- lapse issue where: (i) all the inputs are routed to the same expert (Zuo et al., 2022) or (ii) all
y = Σ R(h), Ey(h) and
(I)
R(h) TopK(o x h),
where the TopK(,, k)) function keeps the largest &

---

out training, starting k 2 and ending N. Fig. 2 visually illustrates our proposed HyperRouter method.
3.3 A Deeper Analysis HyperRouter
FFN
FFN
FFN
FFN
...
We now investigate the representation capabilities of HyperRouter compared the naive SMoE and SMoE-Dropout. Due space constraints, only present the main results here and provide the de- tailed calculations in Appendix B. First, the naive SMoE jointly trains the router and expert parame- ters, lcading to entanglement and eventually the representation collapse issue (Zuo et al., 2022). SMoE-Dropouut proposed to fix the router parame- ter W,, which will show to result in restricted representation. To do so, need to calculate the Jacobian with respect to h, which characterizes how the feature h is involved during training.
Topk
HyperRouter
Router
Router Emd.
Token Emd.
Figure 2: An illustration of our HyperRouter that dynamically generates the router's parameters from fixed hypernetwork. Yellow modules including the router embedding and experts are trainable while the gray module the hypernetwork, frozen. Best viewed colors.
Jacobian of SMoE and SMoE-Dropout. Let { SSMOE(h) (h) = oW,,))) S (h) = a(H(e) h)), 15s be the indicator function, and W be the i- column of Wr, the Jacobians for SMoE and SMoE-
values of a given vector while setting the remaining values to zero; (-) is the standard Softmax func- tion. In practice, only a few experts are activated for the current token (k < N) for efficient training. SMoE-Dropout. SMoE-Dropout (Chen et al., 2023) is a statc-of-the-arr strategy that addresses the representation collapse issues (Sec. 2) by sim- ply fixing randomly initialized router improve the token routing consistency. Furthermore, SMoF- Dropout gradually increases the number of experts chosen throughout training, starting with k = 2 and ending at = N. During the evaluation, SMoF. Dropout proposes to use half or all experts to bal- ance its efficiency and performance.
Dropout calculated
Vn -
J V.C
= J- +
G Wri where (3)
k
G = Σ SSMOE (h) (1se SMOE (S))E(h))
(E,(h)
Here, the Jacobian VnC is combination of two terms related to J1 and J2. The J1 component represents how h contributes to the output y and is the same for all methods. The second compo- nent related to J2 characterizes learning better ex- perts' representation of h and is the main differ- ence. Since SMoE-Dropout fixes W,, its J2 Jg VyC term is expressed of columns in W,. which lies in lower dimen- sional subspace because W,, E Rxxd with k < d number of experts is usually smaller than the fea- ture dimension. Thus, SMoF-Dropout alleviates the entanglement between the router and experts at the cost of restricting the experts' representation capabilities.
3.2 HyperRouter
We describe our proposed HyperRouter strategy that balances between a random but fixed router and router optimized during train- ing. HyperRouter employs fixed hypernet- work (Ha et al., 2017) H(-) to dynamically generate the router's parameters conditioned on trainable router embedding Specifically, HyperRouter obtains the output y as:
linear combination
y Σ R(h), Eh) and
(2)
Rhh) = Top k) and W, H()), where e is low dimensional trainable vector as- sociated with the current layer. All parameters are jointly optimized to minimize the task loss. Lastly, like EMoo-Dropout. HyperRouter also gradually increases the number of activated experts through-
Jacobian of HyperRouter. By similar calculation, the Jacobian of HyperRouter is shown in equation 4. Given that e is trainable, HyperRouter's Jacobian not ex- pressed as a simple linear combination in a much lower-dimensional subspace. In contrast,

---

HyperRouter maps the feature subspace whose dimension can be controlled by e, which we can easily set to greater than Furthermore, by fixing the hypernetwork, the Jacobian does not change freely as in SMoE, which helps alleviate the entanglement. Overall, HyperRouter can alleviate the representation collapse while not sacrificing the representation capabilities.
Table 1: Bit-per-character and Perplexity on the enwik8 and WikiText-101 test sets, respectively. Lower is bet- ter. denotes the number of experts chosen during inference. The best results are in bold.
anwikö
k
Dense
Dense-Dmpout
SMoE
SMoE. Dropout
HyperRouter
7.20
3.02
1,48
4
1.68
1.29
1.19
Jacobian of HyperRouter:
VnC V yf J v
16
1.25
1.16
1,27
1.16
1.16
MikiText-119 193
- V + cHi((e), where
(4)
896. 44
560.93
65.17
$4.69
40.00
30.45
h J1
16
32.13 31.55
32.27
27.66
27.57
$ sllyper Vh) (E)(h))
baselines use the same amount of trainable pa- rameters, while our HyperRouter introduces ne- glectable 1024 trainable parameters for the router embeddings.
k
Hyper (h) [158 sHyyer (h)
1=1
(E) (e,(h)TVv,,) v)
We consider two training tasks. First, pre- training on the enwik8 (Mahoney, 2011) and WikiText-103 (Merity et al., 2017) datasets, where we follow the same pre-training procedure as (Chen et al., 2023) to train the small and medium models for 400K iterations. We only train the large models for 100K iterations due resource con- straints. Second, finetuning on the SST-2 (Socher et al., 2013), SST-5 (Socher et al., 2013), IMDB (Maas et al., 2011), and BANKING77 (Casanueva et al., 2020) datasets using the model pre- pre-trained on enwik8. Similar to (Chen et al., 2023), we also perform dense finetuning (always choosing all ex- perts) for several epochs. In all experiments, we report the evaluation metrics the test set when using different numbers of experts for routing meth- ods. More implementation details and additional results are provided in the Appendix.
4 Experiments
We conduct experiments to investigate the follow- ing hypotheses. First, HyperRouter improves ef- ficiency compared to existing routing strategies, requiring less computation to achieve similar per- formance thresholds during evaluation. Second, with the same amount of experts used during in- ference, HyperRouter can achieve better perfor- mances compared to other competitors.
4.1 Experiment Setting
Most of our experiments follow Chen et al. (2023) and consider a small TransformerXL. (Dai et al., 2019) with four layers in our experiments due to the limited resources. We also investigate the scal- ability by considering larger variants of the Trans- formerXL with eight twelve layers. We compare our HyperRouter with several state-of-the-ar rout- ing strategies: SMoE (Fedus et al., 2022b)- train- ing with trainable routers; (ii) THOR (Zhou et al., 2022) replacing routers with a stochastic experts selection strategy and a consistency regularizer; (iii) SMoE-Dropout (Chen et al., 2023) using random router and gradually increasing the number of selected experts during training. We also include traditional baselines: (i) Dense- the standard training of transformers where no routing mecha- nisms are implemented; and (ii) Dense+ Dropout similar to Dense but with Dropout (Srivastava et al., 2014) inserted in the fully connected layers. All
4.2 Pre-training Result
Tab. reports the bit-per-character and perplex- ity on the enwik8 and WikiText-103 datasets, re- spectively. First, we observe that SMoE-based training offers improvements over Dense train- ing. Furthermore, adding Dropout can help with Dense training, which achieves comparable per- formances with SMoE-Dropout, though its im- provements diminish the larger dataset of Tex-Text- The benefit of SMoE training is the efficiency during inference by using fewer ex- perts. Our HyperRouter substantially outperforms both SMoE and SMoE-Dropout on both datasets this regime. Notably, HyperRouter significantly

---

Table 2: Sit-per-character on the enwik8 test set using the TransformerXL. medium and large models. Lower better. A denotes the number of experts chosen during inference, and SD denotes the SMoE- -ropout method. The best results are in bold.
also report the results of using only half the ex- perts during evaluation. Overall, we observe consis- tent accuracy gains from HyperRouter compared to other baselines on all datasets. Notably, on SST-2 and IMDB datasets, HyperRouter with only eight experts could substantially outperform other strategies when using all experts.
Medium TransfomerXX
Lurge TransfomerXX
SMOE
HyyerRooter
HyperRouter
2,170
3.741
1.457
1,370
1.766
1.215
2
1,395
1.642
1.201
1,216
1.29%
L1881
4.4
Router Analysis
1.260
8
1.230
1,167
1.337
1,182
1.197
L,711
1.236
1,170
Table 4: Average entropy of the distribution of the routers on the enwik8 dataset. Lower better.
Table Transfer performance (Accuracy) on the SST-2, SST-5, IMDB, and BANKING77 datasets. Higher is better. k denotes the number of experts chosen during inference. The best results in bold.
Mathod
Router
Roater 2
Router 3
Ronter
SMaE
2.5164
2.2126
2.2424
2.2583
SMot-Dropoot
2 2,5849
2,1624
2,2579
2.2690
HyperRouter
1,4993
0.8894
1.1269
1.3477
SST-2
SST-5
IMDS
BANKING77
We now investigate the distributional output from the routers (softmax of router's ' output) trained with different strategies. Such outputs determine how tokens are assigned to experts. We hypothesize that high-entropy distributions are not preferred since they are closer to being uniform, indicating the router's low confidence in choosing experts for the current token. In the extreme, the router outputs a uniform distribution and assigns tokens to all experts, eventually causing the collapse is- sue. Therefore, a router with a lower entropy is preferred since it confidently assigns tokens to a few experts, which can improve experts' special- ization. To this end, we report the entropy of each router in the small TransformerXL. trained the enwik8 dataset in Tab. 4. The entropy is calcu- lated on all samples in the test set using the model obtained after training. The result clearly shows that HyperRouter achieved much lower entropy than SMoE and SMoE-Dropout. Due to space con- straints, we will provide additional visualizations and discussions in Appendix D.
Dense DemsetDupoul
82,18
39.72
88.30
84.54
$1.94
39.95
88.37
84.34
SMot-Dropout (k=-)
78 88.43
HyperRouter (k=8)
82.87
41.26
87.66
SMeE (k=16)
79.98
39.45
88. 16
86.52
SMoF-Dropont (lm16))
87 81
87.89
HyperRouter (6=16)
83.33
41.80
88.46
88.41
outperforms SMoE-Dropout when using only one expert, reducing BPC from 3.02 to 1.48 enwik8, and perplexity from 560.93 to 65.17 on WikiText-103. Overall, with eight experts or less, HyperRouter performs competitively with SMoE- Dropout while only using half of the experts during evaluation.
Tab. 2 reports the BPC on the enwik8 dataset using the medium and large TransformerXL. Notably, we only train the large model 100K iterations due re- source constraints. With larger backbone networks, we observe the gap between our HyperRouter and the baselines becomes more significant, indicating our Router enjoys good scalability with the model complexity. Lastly, we observe that SMoE- Dropout does not achieve satisfactory results with the large backbone when only trained with 100K and outperformed by the naive SMoE. This result shows that MMMoE-Dropout requires significant re- sources to achieve good performance. On the other hand, our HyperRouter consistently outperforms both baselines, regardless of the backbone size and number of experts activated.
5 Conclusion
In this work, we investigated the potentials and limitations of SMoE for training LLMs and pro- posed HyperRouter to balance between two ex- tremes of SMoE: training and freezing the router. Consequently, HyperRouter can learn better rout- ing policies while alleviating the representation collapse of conventional SMoE training. Experi- ments on both pre-training and fine-tuning tasks demonstrated promising Hyper capabili- ties in facilitating efficient, effective training and inference compared to ofate-o-the-ar strategies.
4.3 Finetuning Result
Tab. reports the results of the finetuning exper- iment on the SST-2, SST-5, IMDB, and BANKING77 datasets. Although we perform dense finetuning,

---

Limitations
lñigo Casanueva, Tadas Temčinas. Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, Online. Association for Computational Linguisties.
Our work focuses on the efficiency and efficacy of training LLMs using SMoE. Despite the encourag- ing results, our experiments are conducted only on medium--calee datasets with a small TransformerXL due to computation limitations. Thus, further em- pirical evaluations are required to validate the scal- ability of HyperRouter and other SMoE strategies on recent LLMs and larger datasets.
Oscar Chang. Lampros Flokas, and Hod Lipson. 2020. Principled Weight Initialization for Hypernetworks. International Conference on Learning Representations.
Tianlong Chen, Zhenyu Zhang, AJAY KUMAR JAISWAL, Shiwei Liu, and Zhangyang Wang. 2023, Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers. In The Eleventh Interna- tional Conference on Learning Representations.
Ethics Statement
Despite encouraging results, training large-scale LLMs is inevitably costly and requires extensive computational resources, which need to be properly managed. Moreover, our work used data collected on the web, which has been known to suffer from gender and racial biases and requires additional efforts to mitigate its negative impacts. Lastly, our study promising step towards facilitating the development of new LLMs, which still requires careful regularization to avoid potential misuses in harmful applications.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. 2022. Towards Understanding the Mixture- of of-Experts Layer in Deep Learning. In Advances in Neural Information Processing Systems.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Ba- jaj. Xia Song. Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022. On the Representation Collapse of Sparse Mixture of Experts. In Advances in Neural Information Processing Systems.
Yinlam Chow, Azamat Tulepbergenov, Ofir Nachum, Dhawal Gupta, Moonkyung Ryu, Mohammad Ghavamzadeh, and Craig Boutilier. 2023. A Mixture-of- Expert Approach to RL-based Dialogue Management. In The Eleventh International Conference on Learning Representations.
A Acknowledgement
This research/project is supported by the National Research Foundation, Singapore, under its Al Sin- gapore Programme (AISG Award No: AISG2-RP- 2021-027).
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bog- dan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, George Bm Van Den Driessche, Eliza Ruther- ford, Tom Hennigan, Matthew J Johnson, Albin Cas- sirer, Chris Jones, Elena Buchatskaya, David Bud- den, Laurent Sifre, Simon Osindero. Oriol Vinyals. Mare' Aurelio Ranzato, Jack Rae, Erich Elsen, Koray Kavukcuoglu, and Karen Simonyan. 2022. Unified Scal- ing Laws for Routed Language Models. In Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learning Research, pages 4057-4086 PMLR
References
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mi- haylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O' O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab. Zoriitsa Kozareva, and Veselin Stoyanov. 2022. Efficient Large Scale Language Modeling with Mixtures of Experts. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Pro- cessing, pages 1 1699-1191 11199-1776 11732, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. StableMoE: Stable Routing Strategy for Mixture of Experts. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics (Volume 1: Long Papers), pages 7085-7095, Dublin, Ireland. Association for Computational Linguistics.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. 2022a. BEÎT: BERT Pre-Training of Image Transform- ers. International Conference on Learning Represen- tations.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer- XL: Attentive Language Models beyoud a LengthLenngt Context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 2978-2983 Florence, Italy. Associa- tion for Computational Linguistics.
Hangbo Bao, Wenhui Wang. Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022b. VLMo: Unified LangoneLanguage Pre- Tra-nirinn with Mixture-of- Modality- Modality-Experts, In Advances in Neural Information Processing Systems.

---

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthis Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An Image is Worth 16x 16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations.
Matt Mahoney, 2011. Large text compression bench- mark.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer Sentinel Mixture Models. In International Conference on Learning Representa- tions.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong. Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P Bosma, Zongwei Zhou, Tao Wang, Emma Wang. Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen Meier-Hellstem, Toju Duke, Lu- cas Dixon, Kun Zhang, Quec Le. Yonghui Wu, Zhifeng Chen, and Claire Cui. 2022. GLaM: Efficient Scaling of Language Models with Mixture-of Ex-perttts. In Pro- ceedings of the 39th International Conference Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pages 5547-5569. PMLR.
Hao Peng. Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015. A comparative study on regular- ization strategies for embedding based neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2106- 2111.
Quang Pham, Chenghao Liu, Doyen Sahoo, and Steven CH Hoi. 2022. Learning fast and slow for online time series forecasting. arXiv preprint 2rXiv:2202.11672.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. 2021, Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118.
William Fedus, Jeff Dean, and Barret Zoph. 2022a. A review of sparse expert models in deep learning, arXiv preprint arXiv:2209.0166
William Fedus, Barret Zoph, and Noam Shazeer. 2022b. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. Journal of Machine Learning Research, 13)1
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason E. Weston. 2021. Hash Layers For Large Sparse Models. In Advances in Neural Information Processing Systems.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Par- mar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang. Zheng- dong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augnae
Transformer for Speech Recognition. Proc. Interspeech 2020. pages 5036-5040.
Carlos Riquelme Ruiz, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. 2021. Scaling Vision with Sparse Mixture of Experts. In Advances in Neural Information Processing Systems.
David Ha, Andrew M. Dai, and Quoe Le, 2017. Hy- perNetworks. In International Conference on Learning Representations.
Noam Shazeer, * *Azalia Mirhoseini, * *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely- Gated Mixture-off -ffture-of-Expertts Layer. In Inter- national Conference on Learning Representations.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, De- hao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2021. GShard: Scal- ing Giant Models with Conditional Computation and Automatic Sharding. In International Conference on Learning Representations.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive Deep Models for Se- mantic Compositionality Over Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pages 1631-1642 Seattle, Washington, USA. Association for Computa- tional Linguistics.
Mike Lewis. Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE Layers: Simplifying Training of Large. Sparse Models. In Pro- ceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6265-6274. PMLR
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. 2022. A survey of transformers. Al Open, 3:111- 132.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Re- search, 192961919
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 2011. Learning Word Vectors for Sentiment Analysis. ceedings of the 49th Annual Meeting of the Associa- tion for Computational Linguistics: Human Language Technologies. pages 450, Portland, Oregon, USA. Association for Computational Linguistics.
Pro-
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc.

---

Haiyan Yin, Ping Li, 2021. Mitigating forgetting in online continual learning with neuron calibration. Advances in Neural Information Processing Systems, 34:10260-10272.
that HyperRouter obtains the output y as follows:
y Σ R(h); Ey) ==
R(h) (To(e(((H(e)) k),
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. MoEfication: Trans- former herward Layers are Mixtures of Experts. In Findings of the Association for Computational Lin- guisties: ACI. 2022, pages 877-890, Dublin, Ireland. Association for Computational Linguistics.
Here, ((-) is the standard Softmax function given by:
a (HH(e) h)j exp((H((e) x h);))
Σn exx(ex)((((e) h),))
= S Vj I....,..
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang. Vincent Zhao, Andrew M Dai, zhifeng Chen, Quoc V Le, and James Laudon. 2022. Mixture-of- Experts with Expert Choice Routing. In Advances in Neural Information Processing Systems, volume 35, pages 7103-7114. Curran Associates, Inc.
Without loss of generality and for simplicity. by rearranging the index of the top k-experts, HyperRouter can be rewritten as follows:
S Hyper (h) EE where sulyper (h) exp((H(() h)y)
y
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. 2022. Taming Sparsely Activated Transformer with Stochastic Experts. In International Conference Learning Representations.
Σ (xp((H(e) x h)))² Vj l,..,,,
The Jacobian matrix J of the output y with respect to h can be decomposed into two terms follows:
The supplementary document of our work is orga- nized as follows. Appendix A provides additional illustrations to compare our HyperRouter with other strategies. Appendix B provides a detailed derivation of the Jacobians. Appendix C provides the detailed experimental settings, additional re- sults, and analysis. Lastly, we conclude this work with a discussion on promising future research venues in Appendix E.
k Hyyper (h) EE)(t)
hy =Vn
Hyper Vn (E)(
Vh [sttper (h) E7(h)
j=1
A Additional Figures
S
Hyyeer Vn) (E)
Fig. 3 illustrates the conceptual difference be- tween the traditional SMoE (Fedus al., 2022b), SMoE-Dropout (Chen al., 2023), and our HyperRouter. In summary, SMoE uses a train- able router to send tokens to experts. On the other hand, SMoF-Dropout fixed a randomly ini- tialized router and gradually increased the num- ber of experts chosen throughout training. Lastly, HyperRouter improves upon SMoE- -Dopout by replacing the router with fixed hypernetwork and trainable router embedding.
k
Hyper((h)
Hyper (h) (1 sHiyper(n)
x (e( (since e is independent of h).
= J1 J2
(5)
Note that the last equation in (5) is obtained by using the chain rule, the logarithmic derivative, and the inner product as follows:
Øs; 0z: 5j
log(s;) = 8j -I, (a4-s)) s:). where
Ozi
exp(2,)
Yj = 11, 1....,k.
B Derivation of the Jacobian
exp(x;
It is worth mentioning that the first term Jn means to produce better token representation given the current activation router s sffyee(()) while the sec- ond term J2 represents learning a better gating function for the appropriate activation score router X HHyper (h). After the back propagation, the gradi-
This Section details the calculations of the Jaco- bians presented in Section. 3.3.
The Jacobian of SMoE and SMoF. -MoF--Dropouti is sim- ilar to the Jacobian of HyperRouter. Therefore, only the later detailed calculation is shown. Recall

---

FFN
FPN
PPN
FEN
FFN
FFN
FFN
FFN
FFN
FFN
PPN
FFN
Tops
Topk
Topk
Router
HyperRouter
Token Emd.
Taken Ema.
Router Emd.
Token Emnd.
(a) SMoE
(b) SMoE- MMoooouu
(c) HyperRouter
Figure 3: An illustrative comparison among SMoE, SMoE-Dropodde and our PlyperRouter. The input token is a representation vector (could be output from the previous layer). Yellow modules trainable. frozen, indicated by lock symbol
Gray modules
Best viewed in colors.
ent of the loss function L is obtained from the two paths mentioned above and is written follows
Model architecture. The small TransformerXL. variant (Chen et al., 2023) consists of Trans- former decoder layers with an input dimension of 256. Each layer consists a self-attention layer with attention heads, followed by a feed- forward network with an inner dimension of 512. The dropout ratio is kept at 0.1. We split the feed- forward network into 16 experts with the same dimension. For the HyperRouter, we initialize the embeddings with size 256 and employ 2-layer perceptron with an inner dimension of 256 the hypernetwork. The ReLU function is used as the activation function for the hypernetwork. Totally our HyperRouter introduces an additional 1024 trainable parameters in the TransformerXL. model. It is worth noting that the parameter overhead is fixed we scale up larger transformer models. For the medium and large variants, scale the model to eight and twelves layers, respectively.
VnC vV Vny VaY
vyC
- = J2) vyC
(using (5))
=J V V y.C.
(6)
Finally, by expanding the second term as follows,
we obtain the desired result:
1 (h) (1,+ s(lyper
(n) Vyc) Hl)
x
Σ (e), where
1=1
G Sj Sttyper((h (h) (15r s.ttppe (h)
(v
Additional Experiments
C.2 Pre-training Experiments
This section provides the implementation details of our experiments Sec.
Tab. Al provides the implementation details for pre-training our TransformerXI. small and medium enwik8 and WikiText- -03.. The Trans- formerXL large network trained in the same maner, but only for 100K iterations.
C.1 General Setting
Our experiments conducted based the pub-
licly available SMoF-Dropout (Chen et al., 2023)
implementation' Moreover, the pre-training exper- iments were conducted on single A40 A 100
Table A1: Implementation details for pre- mre-training exper- imentson enwik8 and WikiText- 103 datasets.
GPU, while the finetuning experiments were con- ducted on a single GeForce RTX 3090 GPU. We also emphasize that parallel training on multiple GPUs might yield different results.
Datasct
Inpur length
Ratch size
Oprimizer
1rr
* Iterations
enwikö
512
22
Adam
2.5e 4
400000
wikiText-183
2.5c-4
'ttpps: https:////iee htt.b//ttttthooog//ee... com/VITA-Graup Tandom-MoE-an as- Droport

---

C.3 Finetuning Experiments
Table A4: Number parameters in different compo- nents of SMoE-Dropout and HyperRouter during train- ing. Blue parameters are trainable, red parameters are
To conduct finetuming experiments, we use the same model architecture as in pre-training. Tab. A2 provides the implementation details used for finetuning experiments on four different datasets.
frozen, and underline parameter are dynamically gener- ated in each iteration.
Mothod
Router Emb..
Router
HyporNetwork
Transtomerr
Table A2: Implementation details for finetuning experi- ments on four different datasets.
Srapoot HyperRouter
16.448
19. 506.350
16.448
112.57
19, 005.350
Dainset
Input length
Batch size
Optimizer
Epochs
SST-2
S12
16
Adam
1o-4
3
backbone and the router embeddings; (ii) frozen parameters which are the routers SMoE--roppout and hypernetwrrss in HyperRouter (iii) dy- namic parameters that are dynamically gener- ated in each iteration, such as the routers HyperRouter Overall, the additional trainable parameters in HyperRouter is neglectable. More- over, the number of frozen parameters (hypernet- works) is quite small compared to the transformer backbone (5.7% (5).. Investigating into sharing the hy- pernetwork across layers or generating the routers coordinately (Pham et al., 2022) can further reduce this cost while improving the performance, which we will leave for the future work.
99T-5
512
16
Adam
1e.4
3
IMDB
512
Adam
1o-4
3.
BANKING77
512
16
Adam
le.4
15
C.4 Inference Efficiency
We discuss an implementation detail that makes our HyperRouter have the same efficiency during in- ference as SMoE and SMoE- -oop--ur Particularly, before performing inference on any input samples, we use the hypernetwork to generate the router for each layer and then store them. During infer- ence, HyperRouter does not need to re-generate routers. Therefore, HyperRouter enjoys the same inference complexity SMoE and SMoE-Dropout because their routers have the same dimensionality. Tab. A3 reports the number of FLOPs of different methods during inference on the enwik8 dataset according different number of experts chosen (k). Note that the FLOPs do not scale linearly with A since SMoE training is only applied to the fully connected layers, while other layers (multi-head self-attention, layer normalization, etc.) remain the same across baselines. In the extreme, using only one expert only incurs 45% of the FLOPs compared to using all 16 experts.
D Routing Visualization
Table A5: Average entropy of the distribution of the routers' output on the enwik8 dataset. Lower is better.
Mathod
Router
Roater 2
Router 3
Romter
2.5164
2.2126
2.2424
2,2583
0.17
+ 0.31
2: 0.33
+ 0.25
SMoE. Drepout
2,5849
2,1624
2.2579
2.2690
0.11
0.37
:: 0.28
+: 0.27
HyperRouter
1,4993
0.8894
1.1269
1.3477
0.46
1. 0.50
This Section provides the full details of the routers' entropy and visualizes its distributional output. This is supplementary to Section 4.4. Tab. A5 reports the mean and standard deviations of the entropy at each router. This table is the full ver- sion of Tab. A5. We can see that all methods have rather low standard deviation, indicating that the differences are significant.
Table A3: Inference FLOPS (10" on the enwik8 dataset, denotes the number of experts used during inference.
Dense DemoetDopoull THOR SMoE SMoE: SMoE-Dropoot HyperRoocter
3.47
3.47
3.47
4.00
4.00
4,54
4.54
4.54
16
7.76
7.76
7.66
7.56
7.66
7.66
We also provide an illustrative example of the routers' outputs using randomly picked sample on the test set in Fig. 4. Here we can clearly see that the policies from SMoE and SMoE-Dropout are much closer uniform while HyperRouter poli- cies are much sharper and have lower entropy. We emphasize that this example is not cherry- since we already calculated the averaged entropy
C.5 Parameter Comparison
Tab. A4 provides the number of parameters in different components of SMoF-Dropout and HyperRouter There are three categories: (i) trainable parameters which are the transformer

---

all samples in Tab. A5. Overall, this result shows insights into how HyperRouter can perform better than other state-of-the-art SMoE strategies.
E Future Work
Our HyperRouter opens several promising venues for future research. Particularly, believe that investigating two HyperRouter components: (i) fixed hypernetwork, and (ii) trainable embedding can yield further performance and efficiency gains. Potentials directions include incorporating regular- ization such as lz-penalty or dropout (Peng et al.. 2015) or using better hypernetwork initialization techniques (Chang et al., 2020). Furthermore, the current implementation uses hypernetwork for each transformer layer, and it generates all param- eters of the router. Sharing hypernetworks among layers or generating the router coordinate-wist wise can offer knowledge sharing (Yin et al., 2021; Pham et al., 2022), which can further improve the result.

---

Rouser
Telken
Token 2
Tokkn 3
Tuken
Token
Taken
Token ?
Token 8
Honter
Token
Token 2
Token
Token
Tokon 5
Taken
Tokon 7
Token 8
Router :5
Token
Token 2
Token
Token
Token 5
Token
Token
Token
Rouxer 4
Toloen
Token
Token
Token
Token
Tokeo
(a) SMoB
Router
Tokeen
Token 2
Token 3
Token
Token
Token
Token ?
Tokeo 8
Touser
Tedeer
Takwer 2
Torken
Tokkr
Tokknn
Tokar
Tokknn 7
Rourer
alln
Token
Token
3
Token
Token
Taken
Takeen ?
Token
Houter
Mat 1
Tokeen
Token 2
Token
Token
Token
Token
Token
Tokeo
(b) SMoE-Dropout
Router
Tokeen
Token 2
Token
Token
Token >
Taken
Toben ?
Token 15
Router
Token
Token 2
Tokknn 3
Token
Token 5
Token
Token 7
Token 8
Rourer
nxt w.t.l..nane.l
not La
Toleen
Token 2
Tokken 3
Token
Token 5
Token
Token 7
Token
Rouser
Token
Token
Token
Token
Token
Token
Token
Tokeo
(c) HyperRouter
Figure 4: Visualization of the distribution of the routers' output.


Reviews:
[{"review_id": "xbaLgNzseo", "reviewer": "Reviewer_EK3o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We refer the Reviewer to our general comments for the intuition of our method and the large scale experiment results.\n\n> The paper needs a comprehensive analysis of sparse MoE, including the communication overhead (all to all).\n\nThank you for the interesting suggestion. However, we would appreciate if the Reviewer could elaborate more on this concern regarding all-to-all communication overhead.\n\nFor this work, we focus on designing a more effective routing policy by addressing the weaknesses of naive SMoE and SMoE-Dropout, which are two extremes of SMoE training. Our HyperRouter incurs neglectable additional trainable parameters (layer embeddings) and has the same training/inference complexities as SMoE-Dropout (as we showed in Appendix B.4 and B.5).\n\n> The evaluation needs experiments on distributed deployment and a larger model\n\nWe appreciate the Reviewer's feedback. The main focus of our work is to propose a simple and effective router for SMoE-based training of LLM. We have shown encouraging results of HyperRouter compared to the standard SMoE training and a state-of-the-art strategy, SMoE-Dropout. Unfortunately, we are not able to conduct large scale distributed training due to our limited computation resources. Even so, we tried our best to conduct larger scale experiments with TransformerXL-Medium and TransformerXL-Large in this rebuttal. The results consistently show that our HyperRouter can offer improvements to the baselines across different network sizes and number of experts used during evaluation. Thus, we firmly believe that any advanced in distributed training of SMoE can be easily adopted to HyperRouter with similar performance gains.\nHowever, a comprehensive evaluation of large-scale distributed training is beyond the scope of this study and we will leave this for the future work.\n\n> For the arguments that the existing approach has two key limitations, the authors should present key experiment results for demonstration.\n\nOur work argue two limitations of SMOE-Dropout: (i) the routing policy from the randomly initialized gating might be sub-optimal; and (ii) SMoE-Dropout requires more experts to achieve good performances.\n\nRegarding (i), it is clear that a better routing policy will result in a better final performance, giving the same network architecture. This result is supported by the experiments throughout our study, e.g. Table 1, 2 and Figure 1 where HyperRouter consistently outperforms SMoE-Dropout given the same network architecture. Moreover, in Appendix C, we also visualize the routing policies and report their entropy from different methods. We can clearly see that HyperRouter's routing policy has much lower entropy, indicating the router can confidently assign tokens to experts. For a more detailed discussion, please refer to Appendix C.\n\nThe arguement (ii) is supported by Figure 1, Table 1, and 2 where our HyperRouter can achieve similar performances with SMoE-Dropout when using only half of the experts during evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an interesting approach that facilitates dynamic router parameter generation.\n\n2. The evaluation demonstrates the potential and performance improvement of the proposed approach.", "reasons_to_reject": "1. The paper needs a comprehensive analysis of sparse MoE, including the communication overhead (all to all). Currently, it's not clear where the performance gain comes from, basically, different number of experts incurs different communication overhead.\n\n2. The evaluation needs experiments on distributed deployment and a larger model.\n\n3. For the arguments that the existing approach has two key limitations, the authors should present key experiment results for demonstration.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors introduce a new approach that features dynamic router parameter generation in sparse MoE. The authors argue that the existing approach has limitations on the sub-optimal random routing policy and limited efficiency gains for inference. To mitigate these limitations, this paper presents HyperRouter, which generates the router's parameters based on a trainable router embedding. The evaluation shows that HyperRouter efficiently improves the inference performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PSjLbwupWX", "reviewer": "Reviewer_1481", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We refer the Reviewer to our General Response \"Large Scale Experiments\" for the results and detailed discussions on the large scale models. In summary, we observe that the gap between our HyperRouter and SMoE-Dropout becomes larger as we increase the model complexity, which is promising for scaling our method to larger scales LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- HyperRouter introduces a novel approach that balances fixed and trainable routers in SMoE training, enhancing the routing policy and mitigating representation collapse issues, resulting in more efficient and effective large language model training.\n\n- The authors conduct experiments on both pre-training and fine-tuning for various datasets and note that HyperRouter:\na) Outperforms both SMoE and SMoE-Dropout (in the pre-training regime)\nb) Significantly outperforms SMoE-Dropout when using only one expert (in the pre-training regime)\nc) Performs competitively with SMoE275 Dropout while only using half of the experts (in the pre-training regime)\nd) In the fine-tuning regime- it outperforms all the other strategies on SST-2 and IMDB datasets with only 8 experts\n\n", "reasons_to_reject": "- The experiments in the paper have a narrow scope since the authors use only one kind of transformer architecture. The results are are not very conclusive for future research to build on.\n\n- I'm not sure if utilizing just 4 layers (out of 18) from TransformerXL would accurately predict the model's behavior. The authors might have opted for a smaller model for directly translating the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents HyperRouter, a dynamic routing approach that generates the router's parameters through a fixed hypernetwork and trainable embeddings. This method strikes a balance between training the routers and freezing them, resulting in improved routing policies and alleviating the representation collapse issue encountered in Sparse Mixture-of-Experts (SMoE). The authors claim that extensive experiments across various NLP tasks demonstrate the superior performance of HyperRouter compared to existing routing methods for SMoE while using fewer experts during inference.\n\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MRZU3GNVAk", "reviewer": "Reviewer_wUFz", "summary": "", "questions": "A. What are the empirical results on large and medium scale transformers (LLMs)? If there is indeed resource limitation, could you provide some hypothesis and minimal experiment results to validate your hypothesis?\n\nB. In section 3.2,  it's not clear why the decoupling of HyperRouter's embeddings from the token representation will improve the routing policy. Is it from some property of hypernetwork or something else? There is not enough context on hypernetwork and why the proposed approach will lead to improvements. It might not be obvious to readers. \n\n[Update] The authors addressed the two questions with proper empirical evidence and explanation in the rebuttal.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We refer the Reviewer to our general responses for the detailed analysis of the performance gain from HyperRouter. We also conducted a larger scale experiment with TransformerXL-Medium and TransformerXL-Large. The result indicates that the gap between HyperRouter is consistent, and even becomes larger, as we increase the model complexity. This result is also consistent with our analysis (please see \"Explaining the Performance Gains from HyperRouter (comments from Reviewer EK3o and wUFz)\"), which does show that the expert's representation capabilities in SMoE-Dropout is restricted to a low dimensional subspace.\n\n> Confusion regarding Line 108-110\n\nWe apolgize for the confusion in L108-110. There are 2 categories: (i) each token chooses k experts; and (ii) each expert chooses k tokens. SMoE, SMoE-Dropout, and our HyperRouter fall into the first category while THOR (Zhou et al., 2022) falls into the second category.\n\nIn the revision, we will revise this section to discuss the literature more clearly.\n\n> The section B.4 (inference FLOPS) and C (routing visualization) are very useful\n\nThank you for the suggestion. In the revision, with an additional page, we will make sure in include and discuss this result in the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The results based on a small scale Transformer-XL on both pretraining and finetuning tasks are very convincing. It shows for pretraining tasks and during inference, HyperRouter achieves the same performance as competitors with half the number of experts, warrant its advantage for LLM deployment. It also shows for finetuning tasks, there is consistent accuracy gains over competitors on all datasets. \n\nThe additional experiment results about the inference FLOPs, parameter comparison, and entropy analysis and routing visualization in the appendix add more strength to the analysis. They provide extra angles to help understand why the proposed approach is advantageous.", "reasons_to_reject": "All experiment results are based on a small-scale transformer-XL. Based on the configuration in the paper, the model has about 1.8M parameters. The SMoE-Droput paper reported results on BERT base (110M parameters), RoBERTa base (125M), and Transformer-XL (18M).  So the reader of this paper may not have a clear picture of what the performance gains and inference advantages are on large and medium scale transformers (LLMs). Will the conclusions generalize to larger-scale LLMs or the gain will diminish as it scales? We need more empirical evidence. \n\n[Update] The authors addressed the two questions with proper empirical evidence and explanation in the rebuttal.", "questions_to_authors": "A. What are the empirical results on large and medium scale transformers (LLMs)? If there is indeed resource limitation, could you provide some hypothesis and minimal experiment results to validate your hypothesis?\n\nB. In section 3.2,  it's not clear why the decoupling of HyperRouter's embeddings from the token representation will improve the routing policy. Is it from some property of hypernetwork or something else? There is not enough context on hypernetwork and why the proposed approach will lead to improvements. It might not be obvious to readers. \n\n[Update] The authors addressed the two questions with proper empirical evidence and explanation in the rebuttal.", "paper_topic_and_main_contributions": "This paper proposes HyperRouter to address the tradeoff between fixed and trainable routers in SMoE training. \n\nSMoE trains the router and suffers from representation collapse. SMoE-Droput effectively mitigates the collapse, but its random routing policy might be sub-optimal and its inference requires half or all of the experts, leading to limited efficiency gains during inference. This work proposes using a random but fixed hyper-network to generate the router's parameters based on a trainable router embedding. \n\nThis work shows empirically that HyperRouter achieves the same performance with fewer experts (compute) during inference, significantly enhancing inference efficiency. It also consistently outperforms competitors when the same number of experts is used during inference.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 108-110. \"enable selecting different experts for each token\" falls into the first category, ie., let token select the top-k experts, not let expert select the top-k tokens. Do you summarize it incorrectly or is there another paper for the 2nd category?\n2. The section B.4 (inference FLOPS) and C (routing visualization) are very useful for understanding the results. I do wish they are included, or summarized in the main paper."}}]

Author rebuttal:

Meta-review: The paper presents a dynamic routing approach for Sparse Mixture-of-Experts (SMoE) training and inference. This method aims to mitigate the limitations of existing SMoE approaches by generating the router's parameters through a combination of a fixed hypernetwork and trainable embeddings. 

Generally, the reviewers appreciated the novelty of the proposed approach and that experimental results were given for both pre-training and fine-tuning settings. The results indicate that the approach appears to outperform existing routing strategies in the SMoE regime. Some concerns were raised by reviewers around the size of models evaluated, to which the authors provided additional results. These results indicate that the proposed approach also works on larger models (at least in the architectural family). The paper would be even stronger if additional Transformer architectures were explored. 

There was also concerns related to communication overhead/amount of computation of the approach raised by one reviewer. It is not clear to me why this approach would suffer additional communication overhead compared to relevant existing SMoE approaches. The authors state clearly in the rebuttal that it is a fair comparison in terms of number of experts used across the different approaches.

Finally, the authors also did a good job in the rebuttal of providing additional analysis to argue why the representational power of HyperRouter could be considered better than prior work using dropout.
Decision: Accept-Main