Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
Authors: Yongxin Guo
Zhenglin Cheng
Xiaoying Tang
Zhaopeng Tu
Tao Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Mixture of Experts
Transformer Model
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose DynMoE, a method designed to automatically determine the  hyper-parameter values for MoE models without compromising performance.

Abstract: The Sparse Mixture of Experts (SMoE) has been widely employed to enhance the efficiency of training and inference for Transformer-based foundational models, yielding promising results. However, the performance of SMoE heavily depends on the choice of hyper-parameters, such as the number of experts and the number of experts to be activated (referred to as top-$k$), resulting in significant computational overhead due to the extensive model training by searching over various hyper-parameter configurations. As a remedy, we introduce the Dynamic Mixture of Experts (DynMoE) technique. DynMoE incorporates (1) a novel gating method that enables each token to automatically determine the number of experts to activate. (2) An adaptive process automatically adjusts the number of experts during training. Extensive numerical results across Vision, Language, and Vision-Language tasks demonstrate the effectiveness of our approach to achieve competitive performance compared to GMoE for vision and language tasks, and MoE-LLaVA for vision-language tasks, while maintaining efficiency by activating fewer parameters. Our code is available at \url{https://github.com/LINs-lab/DynMoE}.

Published as a conference paper at ICLR 2025
DYNAMIC MIXTURE OF EXPERTS: AN AUTO-TUNING APPROACH FOR EFFICIENT TRANSFORMER MODELS
Yongxin Guo Zhenglin Cheng4.5.6.. Xiaoying Tang 12,3. Zhaopeng Tu Tao Lin
1 Schooo of Science and Engineering. The Chinese University of Hong Kong, Shenzhen 518172, China Shenzhen Institute of Artificial Intelligence and Robotics for Society (AIRS), Shenzhen, China
@Gunngoogg Provincial Key Laboratory of Future Networks of Intelligence, Shenzhen, China *Zhejiang University Schooo of Engineering, Westlake University SIII
Research Center for Industries of the Future, Westlake University Tencent Lab
ABSTRACT
The Sparse Mixture of Experts (SMoE) has been widely employed enhance the efficiency of training and inference for Transformer-base foundational models, yielding promising results. However, the performance of S.MEE heavily depends the choice ryper-parameters, such as the number of experts and the number experts be activated (referred top-k), resulting in significant computational overhead due to the extensive model training by searching over various hyper- parameter configurations. As a remedy, we introduce the Dynamic Mixture of Experts (DYNMOE) technique. DYNMoE incorporates (1) novel gating method that enables each token to automatically determine the number of experts to activate. (2) An adaptive process automatically adjusts the number of experts during training. Extensive numerical results across Vision, Language, and Vision- Languuge tasks demonstrate the effectiveness of our approach to achieve competitive performance compared to GMoE for vision and language tasks, and MoE-LLa VA for vision- languaee tasks, while maintaining efficiency by activating fewer parameters. Our code is available at https: htt///gttttt gigttuu....... com/ com/LINs LINS- lab/ /DyMoD....
INTRODUCTION
The scalable nature of Transformer models (Kaplan al., 2020) has gained remarkable successes across a spectrum of applications, ranging from language (Achiam et al., 2023; Touvron et al., 2023a;b) and vision Kirillov et al. (2023); Peebles & Xie (2023) to coos-mmddality domains (Liu et al., 2024; Li et al., 2022b; 2023b). To further enhance performance while maintaining high efficiency, Sparse Mixture of Experts (SMoE) has emerged as promising technique that significantly reduces computation costs during both training and inference stages (Fedus et al., 2022; Lepikhin et al., 2020; Zhang et al., 2022), and has been shown to achieve comparable or superior performance compared to traditional dense models (Li et al., 2022a; Jiang et al., 2024; Dai et al., 2024).
Despite its success, SMoE has an unavoidable drawback: the performance of SMoE heavily relies on the choice of hyper-parameters, such as the number of activated experts per token, referred as top-k, and the number of experts (Clark et al., 2022; Fan et al., 2024; Yang et al., 2021). denoted as K. As illustrated in Figure (()), the performance discrepancy of MoE models under various configurations can be approximately 1%-3%. Notably, identifying the optimal number of ablation studies is challenging. As the size of the models continues to grow, this limitation could result in a significant waste of computational resources, and in turn, could hinder the efficiency of training MoE-based models in practice.
without a sufficient
To tackle the above problems, the objective of this paper is to explore a novel training technique for MoE models, with the aim of addressing the following core question:
Is it possible to develop a MoE training strategy that can automatically determine the number of experts and the number of activated experts per token during the training process?
"Equal contributions.
Tao Lin and Xianying Tang are corresponding authors.

---

Published as a conference paper at ICLR 2025
COLA
65.5
MoB w/ vacions (K, 6)
DynMoB (Oun)
65.0
Vogu Tha dew Averray
DynMioks
Acourecy (16)
61.5
BLIP--ug
64.0
MoE (Avg.): 64.30
Ind ore
HLIP VIT COPT
63.5
Aotivated
(dllicn))
(a) Performance Fluctuation Illustration
(b) Performance-Ffficeency Illustration
Figure 1: Illustration of performance and efficiency of DyNMOE. In Figure ((a), we carried out experiments on GLUE benchmark (Wang et al., 2018). employing BERT-large (Devlin et al., 2019) as backbone. In Figure((b), we follow the MoE-LLaVA (Lin al., 2024) settings, the r-axis represents the number of activated parameters. while the y-axis shows the performance on the Visual Question Answering (VQA) task.
Hence, we introduce the Dynamic Mixture of Experts (DYNMOE) method, which addresses the aforementioned question through the introduction of two innovative components: (1) top-any gating method that enables each token to autonomously determine the number of experts to activate, thereby allowing different tokens to activate varying numbers of experts; (2) an adaptive training process that dynamically adjusts the number of experts, increasing it when the current quantity is inadequate and removing redundant experts as necessary. Additionally, we introduce new auxiliary loss function specifically designed to encourage sparsity when employing the top-any gating approach. This loss encourages different experts to be diverse, rather than mandating that all experts be activated with the same frequency. We summarize the contributions of this paper as follows:
Introducing DYNMOF, a novel method frees the burden of pivotal hyper- paper--arameter selection for MoF training, which is capable of autonomously determining the number of experts and the number of experts to be activated per token.
Conducting extensive empirical experiments across Vision, Language, and Vision-Languag tasks. The results illustrate that DYNMoE achieves comparable or superior performance and efficiency compared to the well-tuned MoE settings (Figure 1(b)).
2 RELATED WORKS
The Sparse Mixture of Experts (SMoB) approach (Eigen et al., 2013; Shazeer et al., 2017; Lepikhin et al., 2020) has been proven to effectively enhance the training and inference efficiency of founda- tional models. Contemporary studies primarily modify the MLP layer of transformer models into multiple expert models and employ a gating network to determine which expert to select. They only choose subset of experts for each token during both training and inference (Lepikhin al., 2020; Fedus et al., 2022). Recently, the SMoF. structure has shown success in various research areas. For in- stance, GMoE (Li et al., 2023a) has demonstrated that SMoE can enhance generalization performance in vision tasks. Large Language Models (LLMs) have also employed MoE to simultancously reduce training and inference costs while improving model performance (Fedus al., 2022: Jiang al., 2024; Dai et al., 2024; Ren et al., 2023; Lin et al., 2024). However, most of these models employ standard SMoE structures and apply the SMoE to various tasks. Our paper focuses on improving the MoE training process, which can be easily integrated with these methods.
Recently, some attempts have been made improve the architecture MoE. models. For example, researchers have investigated the benefits of sample-wise (Ramachandran & Le, 2018; Gross et al., 2017) and token-wise (Shazeer et al., 2017; Riquelme et al., 2021; Fedus et al., 2022) routing. Some studies introduce load balancing loss to ensure that the experts are activated an equal number of times (Lepikhin et al., 2020; Fedus et al., 2022). Expert choice routing (Zhou et al., 2022) addresses load balance by allowing experts to choose tokens; however, this approach also suffers from dropped tokens. SoftMoE (Puigcerver et al., 2023) uses a slot mechanism to simultancously resolve the issues of load balance and dropped tokens. Nevertheless, these approaches also require pre-defined
2

---

Published as a conference paper at ICLR 2025
hyperparameters, such the number of experts the number of experts be activated. Some studies enable tokens to activate a varying number of experts (Huang et al., 2024; Yang et al., 2024; Huang et al., 2024; Yang et al., 2024). However, these approaches either rely on modifying the routing mechanism from top-k to top-p (which introduces the additional hyperparameter p), or use dense training during the initial stages, neither of which provide an optimal implementation. In this paper, we tackle this problem by presenting DYNMOE, an algorithm that automatically determines the number of activated experts for each token and dynamically adds or removes experts during the training process. Furthermore, we introduce a new auxiliary loss function that ensures sparsity when utilizing the DyNMOE algorithm.
METHOD
In this section, we introduce the Dynamic Mixture of Experts (DYNMOE), an algorithm capable of auto- matically determining the number of experts and the number of experts to be activated for both training and inference stages. This is achieved through the incorporation of two crucial components:
(1) The top-any gating method (Figure 2), which models the gating mechanism as a multi-label classification problem, allowing tokens to decide the number of experts to be activated on their own. This enables different tokens to activate varying numbers of experts, including the option to activate no experts.
Wa
G+
Wesd
Figure 2: Illustration of the top-any gating method. The input tokens pass through the gating weights Wa g.e corresponding to each expert e, obtaining the gating scores. The scores surpass gates G. will activate the subsequent expert. Finally, the expert outputs are combined to produce the oulput tokens.
(2) A carefully designed adaptive process that adds new experts when tokens choose not activate any existing experts, and removes any surplus ex- perts that have not been activated by any tokens.
The overall process is summarized in Algorithm 1.
3.1 TOP-ANY GATING
In this section, we present the superior gating method to eliminate the need for tuning the top-k value. We further improve the test-time inference procedure and introduce an additional auxiliary loss to prevent token dropping and boost efficiency.
Traditional top-k gating and the limitations. The traditional top-k gating method takes the token embedding x as input and employs an additional gating network to predict the gating scores. These gating scores are then used to determine which experts will be activated for the input tokens. Typically, given token x E IR as input, the gating process is defined as the follows (Rajbhandari et al., 2022; Hwang et al., 2023):
(xx E R softmax WT W.Xx x)
where W, R the parameter of the gating network, and K is the number of experts. Then the output of the MoE layer is defined by
Σ
Exx))
EeeTop-(O(O) g(x)
eETop-k(g(x))
where Ex(x) E dd is the output of e-th expert given input x, and g(x). is the e-th entry of g(x).
Despite the considerable success of the top-k gating method in enhancing training and inference efficiency, two limitations persist:
The value of k must be fine-tuned to optimize model performance. As demonstrated in Figure (()),
the performance of MoE models vary significantly with different t values. This observation has also been noted in recent studies (Clark et al., 2022: Fan et al., 2024: Yang et al., 2021) Consequently, substantial computational resources are needed to identify the optimal value of k.
3

---

Published as a conference paper at ICLR 2025
2. The top-k gating approach assumes that each token must activate the same number of experts, which may not always hold in practice. For instance, when considering different tasks, there could exist tokens shared by all tasks and those specific certain tasks, i.c. different tokens could activate different numbers of experts.
Addressing the limitations of top-k gating by tuning-free top-any gating. To address the aforementioned limitations, propose the top-any gating method, which does not require a pre- defined value of and allows different tokens to activate varying numbers of experts during both training and inference stages.
The design of the top-any gating method draws inspiration from the multi-label classification problem. We consider each expert an individual class and calculate the classification (gating) score for each class (expert) independently. Subsequently, all classes (experts) with scores exceeding the threshold are deemed positive (activated). In detail, given the expert representation matrix W,, E R where the k-th row of Wg acts as the representation of expert k, and input token RRd the key steps of top-any gating can be formulated by the following equation:
(x. W
s(x) =
(3)
|
g(x) = sign (ar (s(x)) - (G))
(4)
where W, R and G RRK To illustrate, we first compute the cosine similarities between the token and the expert representation matrix Wg and obtain the similarity score s(x) R Then the sigmoid function a is applied to the similarity score s(x) to obtain the scores between 0 and 1. Finally, experts with similarity scores greater than the trainable per-expert threshold G are considered to activate experts for the token x. It is important to note that the sign function does not support back prockkkpppgggionn and thus we customize the back- proppaaattttinnn process of this part by directly copying the gradient of ((x) to a (s(x)) - o (G) to effectively bypass the sign function.
Given the gating score g(x) E 12 the number of activated experts is then defined by
k: == sum (g(x))
where & represents the number of experts be activated for token The model output of the MoE layer with the top-any gating method can be derived as follows
Σ Fx(xx).
y
9(x).>0
Remark 3.1 (Discussion on not consider the magnitude of scores when averaging the expert outputs.). In our top-any gating approach, the scores of different experts are calculated independently. As a result, the scores of different experts may have different scales and ranges. For instance, there may be cases where the scores of Expert are within the range of (0.1, 0.2), but the scores of Expert 2 are within the range of (0.8, 0.9). To avoid this mismatch, we have decided not to consider the magnitude of scores in Equation (6). Ablation studies can be found in Table 23.
Improving the top-any gating during test-time to prevent token dropping. To facilitate the design of the adaptive expert number process, we did not impose a minimum value on k. Consequently, some tokens may not activate any experts. To address this issue, during model performance evaluation, we modify the top-any gating to enable top-1 gating for tokens that do not choose to activate any experts. In detail, for the input token with sum(g(x)) = O, the modified gating score ğ(x) is obtained by
{
0
A: # arg max (s(()), k == arg max (ss(x))
x(x); = (s(())
Guarding efficiency for top-any gating by auxiliary loss. is improve the training and inference efficiency. However, in the absence of cap the maximum number of activated experts, tokens might activate all experts, which is counterproductive to our primary goal.
The primary goal of using MoE models
4

---

Published as a conference paper at ICLR 2025
Using auxiliary loss regularization over experts may alleviate our issue. However, existing auxiliary loss methods (Lepikhin et al., 2020; Fedus et al., 2022; Wu et al., 2024) are primarily designed to ensure load balancing across experts and thus cannot align with our objectives. While activating all experts can indeed achieve load balancing, it contradicts our aim of improving efficiency by limiting the number of activated experts. Therefore, we need solution that not only ensures load balancing but also restricts the number of activated experts
As remedy, we propose new auxiliary loss, namely sparse and simple gating loss, as shown in (8). The diversity loss and simplicity loss in (8) work together to improve the efficiency of the model by addressing different aspects of the expert representations. On one hand, the diversity loss encourages independence among the W, representations of various experts. It serves two purposes: First, it prevents high degree of similarity between experts, thereby enhancing the model's representational capacity; Second, it guides tokens to avoid simultaneous activation of all experts, thereby promoting sparse gating for improved efficiency. On the other hand, the simplicity loss normalizes W, to avoid excessively large values within the matrix, which helps maintain numerical stability and prevents overfitting due to extreme parameter values. The detailed loss function is defined as follows:
K
L = W WT,, W, diversity loss
IK 11
+
W
w..ell2
(8)
K
simplicity loss
where In is the identity matrix with dimension K, and Wg,e E Rd is the e-th element of W. indicating the representation of the e-th expert.
3.2 ADAPTIVE TRAINING PROCESS
In this section, we elaborate on the adaptive training process, which is designed to automatically determine the number of experts. As illustrated in Figure 3, the adaptive process consists of three parts, namely (1) Routing Recording: recording the routing results during training; (2) Adding Experts: adding new experts when tokens choose not to activate any existing experts; and (3) Removing Experts:
removing experts that have not been chosen by any tokens. To promising efficiency and avoiding burden communication, we only check if experts required to be added or removed every 100-300 iterations.
Routing Recording. To facilitate the removal and addition of experts, it is essential to track the routing status. Specifically, we record two key pieces of information for each MoE layer: (1) For each expert e, we record the time at which expert is activated, denoted as Rg E R (as shown in Line 9 of Algorithm 1). (2) For input data that does not activate any expert, we compute the sum of their embeddings x as R.s E TRd (as outlined in Line 10 of Algorithm 1). Note that this approach simplifies the expert addition process: by using the token embeddings to initialize the expert representation W 9, we can achieve high similarity score between these tokens and the new experts, ensuring that the new expert will be activated by these tokens when added.
As demonstrated in Algorithm 1, we utilize flag, and flags to determine when to start and stop routing recording. Users can control these two flags as needed.
Adding Experts when there exist tokens that choose not to activate any experts. We add new experts when the recorded Rs * O, as some tokens do not activate any experts and Rs is the sum of these tokens. Therefore, given K activated experts and new expert K 1, we initialize Rs and GK+1 0. Moreover, due to the device constrain, the maximum number of R
W ++ experts should be constrained. We set the maximum number of experts 16 for vision and language tasks, and 4 for vision-language tasks in practice. Discussions on additional strategies for initializing new experts can be found in Appendix 8.4.
Removing Experts when there exist experts not activated by any token. We remove experts when there is an expert e such that R = 0 (as shown in Line 13 in Algorithm 1), which indicates that there is no token choose to activate the expert
We also conducted experiments incorporating other auxiliary losses with DYNMOF, shown in Table 15.
5

---

Published as a conference paper at ICLR 2025
Sipart
Eqpat2
Eipar
Coped
Opers
lepott
Copedd
rood!
Expard
venuzahne
Expet3
upen
Fapert
Figure 3: Elaboration on the adaptive training process. We visualize the adaptive training process of DYNMOE, including record routing, experts adding, and experts removing. The green strip connecting the token and the expert indicates records of token routing to expert. The red arrow the bottom part of the figure shows where and when expert addition and removal happens.
4 EXPERIMENTS
In this section, we carry out experiments to address the following questions:
Q1: Can DYNMOE achieve competitive performance among different MoE settings? See 4.2.
Q2: Can DYNMOË handle tasks with varying modalities and scales? See 4.3.
Q3: Will the model trained by DYNMOE maintain sparsity to ensure efficiency? See 4.4.
Q4: Can DYNMOE offer insights that could guide the design of MoE models? See 4.5.
Additional numerical results, including (1) detailed results on vision and language tasks, (2) ablation studies on auxiliary losses, (3) comparison top-p gating basclines, (4) pretraining and fine-tuning results on more vision tasks, (5) training efficiency evaluation, (6) overhead of introducing top-any gating, and (7) ablation studies the weights for averaging expert outputs, can be found in the Appendix 8.
4.1 EXPERIMENT SETUP
To answer the above four questions, we conduct experiments on Vision, Language, and Vision- Language tasks. The details are shown in the following.
Vision Task. For the vision tasks, we follow the same settings as in GMoE (Li et al., 2023a). We employ the pre-trained ViT-S/16 Dosovitskiy et al. (2020) model and evaluate it the DomainBed (Gulrajani & Lopez-Paz, 2020) benchmark. Our experiments encompass four Do- main Generalization datasets: PACS (Li al., 2017), VLCS (Albuquerque et al., 2019), Office- Home (Venkateswara et al., 2017), and DomainNet (Peng et al., 2019). All results are reported using the train-validation selection criterion.
Language Task. The language tasks adhere to the same settings as those in MoEfication (Zhang al., 2022) and EMoE (Qiu et al., 2023). The MoE models are built upon the BERT-large (Devlin et al., 2019) architecture using the MoEfication method and are fine-tuned on GLUE (Wang al., 2018) tasks, which include COLA (Warstadt al., 2019), QNLI (Wang et al., 2018), RTE (Bentivogli et al., 2009), MNLI (Xu et al., 2020), and MRPC (Dolan & Brockett, 2005). For each MoE setting, we tune the learning rates in (2e-5, 3c-5, 5c-5) and report the best results. Langua-Lagguage Task. The vision-language tasks follows the setting in MoE-LLa
(Lin et al., 2024), where we use StableLM-2 2-66B (Bellagente et al., 2024), Qwen- (Bai et al., 2023) and Phi-2-2.7B (Hughes) backbone language models, and use clip-vit-large-pain patch 14-336 (Radford et al., 2021) as the vision encoder. The models are evaluated on image understanding benchmarks including VQA-v2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), VisWiz (Gurari et al., 2018). ScienceQA-
(Lu et al., 2022). TextVQA (Singh et al., 2019), POPE (Li et al.. 2023c), MME (Yin et al., 2023), MMBench (Liu et al., 2023). LLa -enchhhh in-the-We -n-the-Wild) (Liu al.. 2024), and MM-Vet Vet (Yu et al., 2023). Furthermore, we keep routing records in our model during testing time. For each benchmark, we collect the number of experts' activations per MoE layer and total processed tokens during testing. The ppprm--rrrmeter settings are the same to MoE-LLaVA for fair comparision.

---

Published as a conference paper at ICLR 2025
COLA
MRPC
QNLI
Top-1 Acourecy Tines
03.5
MoeE x/ varions (K.A)
Top- Accuracy Tines
MuE wf catome (8
Top- Accurcy Tines
MoR w/ varson (8,
DoeMMiF (Kem,, me k- S
DynMof (K am D
DyuMoR (K=9, ang 7
$5.0
Doan 65.17:
$0.3
DouMo
DynMol: 02.50
$4.5"
560
MoE (A Gogg
88.3 MaB (Avg.) $0.04
MoE (Avg]: 92.49
180
MaE (K kod))
Mol: (K--8, kmm))
(i,, kmws)
MaE: 1K5=16, kmm km
MaE 16, kou
MaE (K,10 kms))
MoE 16,16, kowg)
DyuMoE (Ours)
MoE (Kisdd, kmw))
Most (K mu kmua)
MoE (K-m kmod)
MoE (K (K kmw))
MaE 116. bm=1)
Mal 16, kea))
MoE (Kestti km)8)
km=2)
DeubleE (Ours)
MoE (Kom8, kow))
MaE (K mit kew2)
Madi (Kim bens))
MoE (Kusd, kmw))
MoE 16, km
MaE 10 k. km=2)
MaE 16, kww))
MaE (kmit bos))
DyubfoE (Oura)
MoE MrE (K (K-ub,
MaE (Ki11
MNLI
RTE
Top-2 Accuracy Times
Top Anmesry Thmos
MoE w/ varions (K.A)
Top-1 Aceusey Thmos
22
MoE w/ vamtons (K.E)
Top22 Accurac Tenes
MoB w/ surous (K,
DemMoE (K-9, org 8-0))
DrnMaE (K-9, 6.9)
DyuMoE (Duss)
03
DynMaE: 5,00
24
MoR j 86.61 DynMoE: 80.1
23
MaE (Ang.): 74.77
DeiMeE: 73:11
MoE (Kes), keol)
MaE (1ii88 kood)
MoE (Koo8, kmd)) MoE (K=a), bost)
MaE (h-110, kwo))
MoB (K-10, kwo) MaE (limin, boud)
MoE (Keal, kwo))
DyaMcE (Our)
MoE: (Kis8,, keol)
MoE (K bood) Made (1vi8, kmsd)
MoE' (Ke koo)) MoE (N-itt boall ()
MoE (Kim10, keol)
MoE (K-10, boad)
MaE (Eesta, kes))
DynMEE (Oue)
Mod: (Neis koal)
MoE (8 kor))
Mal. (1 bood)
Madi (l6m8, keot))
MoE (f--10, be MaE (K-m10, kw2)) MoE Wim16, koad) MaE (K-in, bass)
DynMaE (Ouri)
Figure 4: Performance of DYNMOE on language tasks. We conduct experiments on the GLUE benchmark. The z-axis represents MoE settings with varying K and top-k values. The y-axis denotes the model'ss performance. Dashed lines indicate the average performance across different settings, as well as the performance of DYNMOE. For all the MoE settings, we tune the learning rates in (2e-5, 3c-5, 5e-5) and report the best results. We also report the times when each MoE setting attains the top-2 best results across all configurations.
4.2
A1: DYNMOE ACHIEVES COMPETITIVE PERFORMANCE AMONG VARIOUS MoE SETTINGS
In this section, we carry out experiments the GLUE benchmark (Wang et al., 201 8), varying the number of experts (K) and the value of top-k: The results of these experiments can be observed in Figure 4. More detailed results of each MoE setting can be found in Tables 6-10 of Appendix.
The performance of DYNMOE surpasses the average performance among various MoE settings. As seen in Figure 4, we can observe that The DYNMOE outperforms the average performance for various K and top-k values in most tasks. DYNMOE also achieves the highest number of top-1/2 best performances among all MoE settings, demonstrating its competitive performance.
2. The performance fluctuates considerably with different K and top-k values, such as up to 3.0% on the RTE task and 1.3% on the COLA task. DYNMoE overcomes this issue by not requiring pre-defined K and top-k values.
The performance gain specific K and top-k choice is not consistent among tasks. For instance, the K = 16, k- setting performs well on QNLI but poorly on MRPC. In contrast, the DYNMOE always achieve competitive performance among tasks.
4.3 A2: DYNMOE CAN HANDLE VISION, LANGUAGE, AND VISION-LANGUAGE TASKS
In addition to Language tasks, we also conduct experiments on Vision and Vision-Language tasks to verify the performance of DYNMOE on different modalities and task scales. The results can be found in Tables 1, and 2.
The effectiveness of DYNMOE remains consistent in both Vision and Vision-Language tasks. Compared to the standard MoE, we can observe the following: A. DYNMoE outperforms standard MoE with well-tuned learning rate, number of experts, and top-k (Qiu et al., 2023) in Vision tasks. The performance difference between DYNMOE and another well-tuned MoE setting in (Li et al., 2023a), falls within the range of random fluctuation. B. When using M-- and
7

---

Published as a conference paper at ICLR 2025
Table 1: Performance of DYNMOË vision tasks: Our study investigates the performance of DYNMOE on vision tasks using the DomainBed benchmark, with ViT-small serving as the backbone model. The effectiveness of GMoE is elucidated based on meticulously tuned results as presented in the previous works Li et al. (2023a) and Qiu et al. (2023). In our implementation of DyNMoE, we configure the maximum number of experts to 8, with initial setting of 6 experts. The number of experts is dynamically adjusted in each iteration for DYN MoE, We also report the performance of DYNMOE using Gshard loss (Lepikhin et al., 2020) as the auxiliary loss.
Algorithms
PACS
VLCS
OfficeHome
DomainNet
Average
GMolF (in Li et al. (2023a))
88.1
80.2
74.2
48.7
72.8
GMoE (carefully tuned (Qiu et al., 2023))
87.7
79.6
73.1
GMoE (with DyNMoE. Gshard Loss)
88.4
79.4
73.6
47.4
72.2
GMoE (with DYNMoE, Diverse and Simple Gating Loss)
87.6
80.3
73.5
48.2
72.4
Table 2: Performance of DYNMOE on vision-language tasks: Our study investigates the performance of DYNMoE-LLaV. on image understanding benchmarks. Evaluation Benchmarks include VQA-v2: GQA; VisWiz; SQA' (ScienceQA-IMGG) VQA (TexiVQA); POPE; MME; MMB (MMBench); LLaVA W (ILaaVA- Bench (in-the-Wilde Wild)); MM-Vet. For fair comparison, we the maximum number of experts to 4 for DYNMOE-LLaVA (the same as the number of experts in MoB-LLaVA) and set the initial number of experts to 2. NA indicates the number of activated parameters.
Algoritims
N₂
VQA
GQA
VisWiz
SQA!
VQA"
POPE
MME
MMB
LLavA
MM-Vet
Dense
LLaVA-1.5 (Vicuna- 13B)
13B
80.0
63.3
53.6
71.6
61.3
85.9
1531.3
67.7
70.7
35.4
11.VA-1.5 (Vicuma 7B)
7B
78.5
62.0
50.0
66.8
58.2
85.9
1510.7
64.3
63.4
30.5
LLaVA-Phi (Phi-2-2.7B)
2.7B
71.4
35.9
68.4
48.6
85.0
1335.1
59.8
28.9
Sparse (StableLM-I.6B)
MoE-LLaVA
(K = 4,k -
2.06B
76.7
60.3
36.2
62.6
50.1
85.7
1318.2
60.2
86.8
26.9
DYNMoF-LLaVA (avg k 1.25)
1.75B
77.4
61.4
40.6
63.4
48.9
85.7
1300.9
63.2
86.4
28.1
Sparze (Qwen-1L88 L.88)
LLVAA (K = 4,K =
2.24B
76.2
61.5
32.6
63.1
48.0
87.0
1291.6
59.7
88.7
25.3
DYNMOE-LLavA (avg = 1.86)
2.19B
76.4
60.9
32.4
63.2
47.5
85.8
1302.4
61.3
89.2
24.2
Sparse (Phi-2-2.78) MoE-LLaVA (K = 4,k
77.6
61.4
43.9
68.5
51.4
86.3
1423.0
65.2
94.1
34.3
DYNMOF-LLaVA (avg & 1.68)
3,35B
77.9
61.6
45.1
68.0
51.8
86.0
1429.6
66.6
95.6
33.6
Phi-2-2.7B as the backbone, the performance of DyNMOE-LLaVA surpasses that of MoE-LLaVA. C. With Qwen- 1.8B as the backbone, the performance of DYNMOF-LLaVA remains comparable MoE-LLaVA, In this setting, the average top-k of DYNMOE-LLaVA (avg k 1.86) is also close to the MoE-LLaVA setting (k: = 2). D. In the BERT experiments (Figure 4), DyNMoE generally activate more experts for each token compared larger scale MoE-LLaVA experiments (Table 2). This observation aligns with the BERT experiments results obtained when using a fixed k value, i.c., k=4 generally performs better among the set { 1.2.4.8).
4.4 A3: DYNMOE MAINTAINS EFFICIENCY BY ACTIVATING LESS PARAMETERS
In this section, we aim to demonstrate that although we did not enforce sparsity on the DYNMOE models, the trained DYNMOE models are still sparse, promising improved inference efficiency.
DYNMOE-LLaVa activates fewer parameters compared MoE-LLaVA. In Table 2, we display the number of activated parameters in the "NA" column. When using StabeLM-1.66 as the backbone, DYNMOE-LLaVA activates approximately 15.0% fewer parameters than MoE-LLaVA VA. For Qwen-1.8P 1.8B, DYNMOE-LLate VA activates about 2.2% fewer parameters than MoB-LLaVA, For Phi-2-2.7B, MYNNoEE-LLVV activates about 7.5% fewer parameters than MoE-LLaVA. these three cases, the reduction in activated parameters does not compromise the model's performance.
Ablation studies the value of top- during test. In Table 3, we examine the performance of DYNMoE-LEP Latto--LLaV, VA when using different top-k values during the testing phase. The results indicate that (1) The original DyNMoF-LLas LLVAA outperforms other settings in most cases while activating the
8

---

Published as a conference paper at ICLR 2025
Vightd 2
POPS
Valiee
MiBBask
Lost Light
Loyar
Layer
5055 Vei
Dossild any sipe;
Brsalai iou topel
Zys36e ang tipil 1.b
Eracict my tepel 121 Haal
Dys3ke ana tiget 1.2
Figure 5: Average top-k activated experts of DYNMOE on vision-language benchmarks. We record average top-k activated experts for each MoE layer when using StableLM-11668 the language model backbone.
Table 3: Ablation studies on the value of top-k during test. We train the models using DYNMOE and set different values of top-k during the test. Training and evaluation settings are identical to that of Table 2.
Algorithms
Na
VQA
GQA
VisWiz
SQA!
VQA?
MME
MMB
LLavA "
MM-Vet
SublelMM-J.6 1.68
DYNMOE LLaVA
61.4
63.4
48.9
1300.9
63.2
DYNMOF-LL (k 2)
2.06B
76.9
61.0
39.1
62.1
49.2
85.7
1320.4
62.4
73.6
28.2
DYNMOE LLaVA
60.7
1306.9
DYNMoF.LLaVa. (k-
2.89B
76.8
60.5
34.8
61.9
49.0
85.8
1321.9
61.9
75.8
27.8
Qwen-1.88
DYNMoE-LLaVA
2.19B
76.2
61,5
32.6
63.1
48.0
87.0
1291.6 6
59.7
88.7
25.3
DYNMOE-LLavA (k
2.24B
76.2
60.8
33.8
62.2
47.7
87.5
1281.3
60.4
91.3
23.0
DYNMOE-LLaVA (k 3)
2.65B
76.2
60.5
32.2
62.9
48.1
88.4
1263.7
60.7
87.8
23.4
DYNMOE-LLavA (k 1)
3.05B
75.7
60.0
31.6
62.8
48.3
88.1
1263.4
61.0
86.7
23.7
Phi-2-2.7B
DYNMOE-LLavA
3.35B
77.9
61.6
45.1
68.0
$1.8
86.0
1429.6
66.6
95.6
33.6
DYNMOE LLaVA (k-
3.62B
77.8
61.5
41.6
67.6
51.8
85.5
1433.5
66.8
95.1
32.7
DYNMOE-LLaVA ik =
4,46B
77.7
61.8
42.0
68.0
$2.3
86.3
1438.1
66.8
94.3
30.8
DYNMOE-LLaVA (k
5.30B
77.5
61.4
41.7
68.0
52.4
87.0
1431.5
66.5
95.8
32.8
Roettokittaaa by Low IXQAAC
Equaii An Actinaic (FQgud)
Elastt Acraaaan Lass (ND)-
+
Lowe
(a) Activation frequency (Qwen) (b) Activation frequency (StableLM) (c) Activation frequency (Phi-2)
Figure 6: Statistics of expert activation frequency different layers. We report the frequency of expert activations in various layers for the VQA task. Larger circles indicate experts that are activated more frequently. Table Efficiency evaluation DYNMOE comparing to MoE-LLaVA.
conduct experiments on single A100 GPU (80 GB) paired with CPUs using identical environment and identical inference configurations. We report the performance of MoE-LLaVA using DeepSpeed's top-2 gating implementation. The symbols 4 and t indicate that lower and higher values, respectively, denote better performance. The results in this table averaged 5 trials, with the first sample excluded to avoid measuring unnecessary memory allocations. Other metrics are reported Table 24.
Model
Memory (GB)
Thrughout Pirst Token Latency J. Wall clock Time 1. (assond sample) (uns sungle x layer) (ne/ sample x laye)) (ue. / sungle x layer)
Routing Time
Gating Time
Expert Passing Time
[loken scaom))
(ne))
32
MoR-1 LAVA (SmMle M-I 6Mx4)
63
0.04
1.21
1.30 1.17
DYNMOE (SubiELM--LLS x)5
5.96
30
124
37
0.52
0,81
fewest number of parameters. (2) Compared to the StableLM-1.6 backbone, DYNMoE-LLaVA trained with the Qwen-1.87 backbone sometimes favors activating two experts. This observation aligns with the fact that DYNMOE-LLaVA also chooses to activate about 2 experts (see Table 2).
Inference efficiency of DyNMoE. To further evaluate the inference efficiency of DynMoE, have compared its FLOPs, MACs, speed, and memory usage to those MoE-LLaVA. The results in Table 4 show that: (1) DYNMOE chieves higher throughput, lower latency, and reduced wall-clock
9

---

Published as a conference paper at ICLR 2025
0OLA
MIRFCC
Standed MoE
Standard MoB
DynMoE
2001
5030
T11.0
Figure Convergence curve w.r.t training FLOPs. We present the convergence curve with respect to training FLOPss for DYNMoE and the best best-performance MoE setting on the GLUE benchmark.
time compared to MoE-LLaVA, indicating improved efficiency. (2) The top-any gating introduces additional cost in the router, but the gating and expert passing steps are more efficient than in MoE- LLaVA. (3) In the current implementation, all experts. loaded unloaded, occupy GPU memory. resulting in the same memory usage MoE-LLaVA, Offloading unloaded experts from GPU memory could improve efficiency.
Training efficiency DyNMOE. We present training FLOPs for both Language (Figure and Vision-Language (Table 24) experiments. The results show that DYNMoE achieves comparable or lower FLOPs than standard MoE, ensuring both efficiency and performance without extensive parameter tuning.
4.5
A4: DYNMoE PROVIDE INSIGHTS ON MoE ARCHITECTURE DESIGN
MoE structure is required for bottom layer rather than top layer. In Figures 5 and 6, present the average top-k of DYNMoE-LLaVA and the frequency of expert activation across various layers. Our observations indicate that: (1) In the top layer (the layer closest the LM prediction head), tokens tend to select the same expert, while the bottom layer. tokens activate all experts uniformly. This suggests that there is need convert the top layer to MoE layer, whereas the bottom layer should be transformed into MoE layer. (2) Different LLM backbones may exhibit distinct expert activation frequency patterns. For the StableLM backbone, most MoE layers activate only one expert, whereas for the Phi-2 backbone, experts are more likely to be activated uniformly.
Shared experts exist in each MoE layer. Figures 18- 22 display the threshold values for each MoE layer. We notice that typically, one expert per layer has significantly lower threshold, making it more easier to be activated. This observation is consistent with Deepseek-MoE's (Dai et al., 2024) design of incorporating shared experts for all tokens each MoE layer.
CONCLUSION AND FUTURE WORKS
In this paper, we introduce DYNMOR, a method that automatically determines both the number of experts and the number of experts activate. Our results show that DYNMoR delivers comparable or even superior performance across various MoE model configurations, while maintaining efficiency. This demonstrates DYNMOE's potential to save researchers time and computational resources in hyperparameter tuning. Additionally, our visualizations reveal interesting insights, such as the reduced number of experts needed for the top layers, which could inspire future advancements in MoE model design. For future work, as discussed in Han et al. (2021), MoE can be considered dynamic model because different tokens may activate different experts, thereby enabling adaptive computation and enhancing the model's 's ability to adapt input data. While DYNMOE addresses dynamic challenges through adaptive top-k selection and an adaptive number of experts, exploring integration with other dynamic techniques, such as layer skipping (Zhao et al., 2024), would also be valuable. Morcover, the current adaptive process and top-any gating method are not sufficiently efficient. Developing more optimized implementations, such designing CUDA kernels, would be valuable in the future.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work is supported in part by the National Science and Technology Major Project (No. 20222D0115101), Research Center for Industries of the Future (RCIF) at Westlake University, Westlake Education Foundation, and Westlake University Center for High-performance Computing. This work is also supported in part by the funding from Shenzhen Institute of Artificial Intelligence and Robotics for Society, in part by the Shenzhen Key Lab of Crowd Intelligence Empowered Low- Carbon Energy Network (Grant No. ZDSYS202200061 10000000000000 1002). in part by Shenzhen Stability Science Program 2023, and in part by the Guangdong Provincial Key Laboratory of Future Networks of Intelligence (Grant No. 2022B1212010000
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874 2023.
Isabela Albuquerque, João Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.0000 1.00044, 2019.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. 2023.
Marco Bellagente, Jonathan Tow, Dakota Mahan, Duy Phung, Maksym Zhuravinskyi, Reshinth Adithyan, James Baicoianu, Ben Brooks, Nathan Cooper, Ashish Datta, Meng Lee, Emad Mostaque, Michael Pieler, Nikhil Pinnaparju, Paulo Rocha, Harry Saini, Hannah Teufel, Niccolo Zanichelli, and Carlos Riquelme. Stable Im 2 1.6b technical report. 2024.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXivv2401.000 16066,, 2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (/WP2005), 2005.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
11

---

Published as a conference paper at ICLR 2025
David Eigen, Mare' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXiv: 2013.
Dongyang Fan, Bettina Messmer, and Martin Jaggi. Towards an empirical understanding of moe design choices. arXiv preprint arXiv:2402. 130-9999... 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, (3(120):1-39 2022.
Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1904-6913, 2017.
Sam Gross, Mare' Aurelio Ranzato, and Arthur Szlam. Hard mixtures of experts for large scale weakly supervised vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6863-6873, 2017.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2018.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11); 7436-7456, 2021.
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint 203.0:2403.07652, 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Alyssa cls.
Hughes.
Phi-2: The surprising power of small language mod-
URL https://ww..ndt http::/wwwwwww.oooo////t/o cwt./coooocooooooooooooooo..c.o. en wtw.:///t...cceeeeeeeeeeeeeeaaae powrrissnnn-ower--dee -aadss-mmmm--aaagaa-------a..-
phi
Changho Hwang. Wei Cui, Yifan Xiong, Ziyue Yang, Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin Jose, Prabhat Ram, et al. Tutel: Adaptive mixture-of-experts at seale. Proceedings of Machine Learning and Systems, 5, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint (XXX::0001.0866/ 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Yann Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 707)33, 2015.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2020.
12

---

Published as a conference paper at ICLR 2025
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse mixture-of-cre of-experts are domain generalizable learners. In The Eleventh International Conference Learning Representations, 2022a.
Bo Li, Yifei Shen, Jingkang Yang, Yezhen Wang, Jiawei Ren, Tong Che, Jun Zhang, and Ziwei Liu. Sparse miixture-of-expert domain generalizable learners. In The Eleventh International Conference on Learning Representations, 2023a. URL https: : /openreviii ew net. forum?
(d=Rec29nBB9)
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference computer vision, pp. 5542-5550, 2017.
Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -mngge pre- training for unified vision-language understanding and generation. In International conference on machine learming, pp.
PMLR, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023b.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint 2.N5::230...3355 2023c.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large lasion-language models. arXiv preprint arXiv:2401.15944 15947, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/ 2023.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song- Chug-Chnn&u Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-255 2521,, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 1195-4205. 2023.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406- 1415, 2019.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. arXiv preprint S.0iv:2308.009: 2023.
Zihan Qiu, Zeyu Huang, and Jie Fu. Emergent mixture-of-experts: Can dense pre-trained transformers benefit from emergent modular structures? arXiv preprint arXiv:2310.10008, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Samyam Rajbhandari, Conglong Li, Zhewei Yao. Minjia Zhang, Reza Yazdani Aminabadi, Am- mar Ahmad Awan, Jeff Rasley, and Yuxiong He. Deepspeed-med Advancing mixture-of-expee of-experts inference and training to power next-generation ai scale. In International conference on machine learning, pp. 18332-18342 18346. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Prajit Ramachandran and Quoc V Le. Diversity and depth in per-example routing models. In International Conference on Learning Representations, 2018.
Xiaozhe Ren, Pingyi Zhou, Xinfan Meng, Xinjing Huang, Yadao Wang, Weichao Wang, Pengfei Li, Xiaoda Zhang, Alexander Podolskiy, Grigory Arshinov, et al. annuu((/Sggma Towards trillion parameter language model with sparse heterogeneous computing. arXiv preprint 21X1:22333.10453 2023.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8558 2021.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exper layer. arXiv preprint arXiv:1701.00000 065388 2017.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018 1088-527,22 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Xun Wu, Shaohan Huang, Wenhui Wang, and Furu Wei. Multi-head mixture-of-expert. arXiv preprint arXiv:2404. 15045, 2024.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng. Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762- 4772, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. 00.18653/11/2220.ool www.:/200.coiiinmmmi...... main. 419. URL https: /aaaaaaaannntooooo org/ 2020. cww.ng.olnngggooooo main.........mmmmm..mm 419.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, al. M6-t: Exploring sparse expert models and beyond. arXiv preprini arXiv:2105.15082, 2021.
Yuanhang Yang, Shiyi Qi, Wenchao Gu, Chaozheng Wang, Cuiyun Gao, and Zenglin Xu. Xmoe: Sparse models with fine-grained and adaptive expert selection. In Findings of the Association for Computational Linguistics ACL 2024, pp. 11664-11674, 2024.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
14

---

Published as a conference paper at ICLR 2025
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 200.:2000.00490, 2023.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 877-890, 2022.
Wangbo Zhao, Jiasheng Tang, Yizeng Han, Yibing Song, Kai Wang. Gao Huang, Fan Wang, and Yang You. Dynamic tuning towards parameter and inference efficiency for vit adaptation. arXiv preprint arXiv:2403. 1/808, 2024.
Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Morley Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to be efficient: Build structured sparsity in large language models. Advances in Neural Information Processing Systems, 37: 37:101999-101991 2025.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-expert experts with expert choice routing. Advances in Neural Information Processing Systems, 35:103--114 2022.
15

---

Published as a conference paper at ICLR 2025
CONTENTS OF APPENDIX
6 Experiment Settings
16
7
Detailed Algorithm Framework
17
8
Additional Experiments
17
8.1
Detailed Results on Language and Vision Tasks
17
8.2
Combine DYNMOE with Load Balance and Efficiency Losses
17
8.3
Comparision to Top-p Gating Baseline
19
8.4
Numberical Results on More Vision Tasks
19
8.5
Efficiency Evaluation
20
8.6
Ablation Studies on Aggregation Weights
20
9
Additional Visualization Results
20
9.1
Activation Frequency
20
9.2
Average Top-k
................................
................................
9.3
Layer-wise Expert Similarity Matrix
................................
9.4
Visualization of G
EXPERIMENT SETTINGS
We conduct experiments on Vision, Language, and Vision-Language tasks. The detailed experiment settings are shown in the following.
Vision Task. For the vision tasks, we follow the same settings as GMoE (Li et al., 2023a). We employ the pre-trained ViT-S/16 Dosovitskiy et al. (2020) model and evaluate it on the DomainBed (Gulrajani Lopez-Paz, 2020) benchmark. Our experiments encompass four Do- main Generalization datasets: PACS (Li et al., 2017), VLCS (Albuquerque et al., 2019), Office- Home (Venkateswara al., 2017), and DomainNet (Peng et al., 2019). All results are reported using the train- validation selection criterion. We conduct all experiments GPU, and the reported results are averaged over three random seeds. For DYNMoE, we set the maximum number of experts to 8 and the initial number of experts 6. The adaptive process is executed for each iteration.
single RTX 3090
Language Task. The language tasks adhere to the same settings as those in MoEfication (Zhang et al., 2022) and EMoE (Qiu et al., 2023). The MoE models are built upon the BERT-large (Devlin al., 2019) architecture using the MoEfication method and are fine-tuned on GLUE (Wang et al., 2018) tasks, which include COLA (Warstadt et al., 2019), QNLI (Wang et al., 2018). RTE (Bentivogli al., 2009), MNLI (Xu al., 2020), and MRPC (Dolan & Brockett, 2005). We conduct all experiments on single RTX 3090 GPU, and the reported results are averaged over three random seeds. For DYNMOF, we set the maximum number of experts to 8 and the initial number of experts 6. For each epoch, we begin recording routing 1/3 of the epoch and complete recording routing and execute the adaptive process at 2/3 of the epoch.
Vision-Language Task. The vision-language tasks follows the setting in MoE-LLaVA (Lin et al., 2024), where we use StableLM-2211.66 (Bellagente et al., 2024), Qwen- 1.8B (Bai et al., 2023) and Phi-2 (Hughes) as backbone language models, and use www.--itta-paacclleeee 14-336 (Radford et al., 2021) as the vision encoder. We conduct model training on 8 A 100 (80G) GPUs, completing within 2 days, detailed hyper-parameters setting are shown in Table 5. The models evaluated on image understanding benchmarks including VQA-v2 (Goyal et al., 2017). GQA (Hudson & Manning. 2019), VisWiz (Gurari al., 2018), ScIG (Lu al., 2022), TextVQA (Singh et al., 2019), POPE (Li et al., 2023c), MME (Yin et al., 2023), MMBench (Liu et al., 2023). -LaVA-Bende (in-the- Wild) (Liu et al., 2024), and MM- Vet (Yu et al., 2023). Furthermore, we keep routing records in our model during testing time. For each benchmark, we collect the number of experts' activations per MoE layer and total processed tokens during testing.
16

---

Published as a conference paper at ICLR 2025
Table 5: Detailed training hyper-parameters and configuration.
Models
Config
StableLM
Qwen
Phi-2
Maximum experts
4
Deepspeed
Zero2
Zero2 LLaVA-Finetuning
Zero2_offlodd
Data
Image resolution
336 x 336 CLIP-Large/336
Image encoder
Feature select layer
-2
Image projector Epoch
Linear layers with GeLU
1
Learning rate
2e-5
Learning rate schedule
Cosine 0.0
Weight decay
Batch size per GPU GPU
8
8
4
x A100 (80G)
8 x A A100 (80G)
x A100 (80G)
Precision
Bf16
Algorithm Pseudo cade DyNMoF on each iteration and MoE layer.
Require:
Input data x, initial gating network parameters W,, G, and т, experts E .. ER, start record routing flag flage, fimish record routing flag flagy-
Ensure: MoE layer output y, auxiliary loss value.
I: if flag. then
routing flag flagrout 1.
3: Initialize routing records Reor Ok-
4: Initialize non-activate sample records R.m = Os-
5: Get the gating outputs 9(x) and k by Eg (4) and (5).
6: Get MoE layer output y by Eg (6).
Calculate auxiliary loss by Eq (8).
8: if flagrout then
Re Rg sum (g (x(),, dim = 0).
10:
Rs Rss + E -x
11: if flag, then
flagrout
13:
if Exists that R$ 0 then
14:
Remove experts
15:
if Rs,e # 0 then
Add new expert + with expert representation Wg,441
Rs/ [Rs|].
DETAILED ALGORITHM FRAMEWORK
8 ADDITIONAL EXPERIMENTS
8.1 DETAILED RESULTS LANGUAGE AND VISION TASKS
In this section, we present the detailed results of our experiments on the GLUE benchmark (Wang et al., 2018) in Table 11 and on the DomainNet dataset in Table 12. These results demonstrate that incorporating the specially designed diversity and simplicity loss significantly enhances the model's performance.
Moreover, we present the detailed results using different learning rates on the GLUE benchmark in Tables 6- 10.
8.2 COMBINE DYNMOE WITH LOAD BALANCE AND EFFICIENCY LOSSES
Tables 13, 14, 15, and 16, we report the following metrics:
Performance (Table 13): The performance of different settings.
17

---

Published as a conference paper at ICLR 2025
Table Detailed performance of DYNMOË and various MoE settings COLA dataset
COLA
A - =
8,4
16, =
16,
K 16,
1
16,
DyaMoE
I 2e-5
64.10
64.51
64.94
43.00
63.63
64.71 22.04
64.12 63.42 20.24
64.37 63.13
65.17
I 3c-5
63.86
62.10
64.73
64,03
61.76 37.26
62.80
30-3
41.83
39.68
62.63
0.00 (fail)
38.30
25.79
40.68
Table 7: Detailed performance of DYNMoE and various MoE settings on MRPC dataset
MPRC
K 8,A
K 8.4
2 5.
8,4
16,5 16,
K 16,
K 16,
DynMol:
= 20-5
89.74 90.11
89.63
89.74 89.50
89.36 88.67
88.07
89.02 90.
89.74 89.38
89.56
89.57
30-5
90.19
89.81 88.30
90.35
90.64
30-3
88.70
84.62
88.72
84,48
89.08
37.40
79.95
90.09
Table 8: Detailed performance of DYNMOE and various MoE settings on QNLI dataset
QNLI
K
16, 16,
DynMoE
20-5
84.04
Ro-3
02.45
92.39
92.01
78,39
78.22
92.53
92.59
I Se-3
50.54
64.46
78.13
64.43
50.54
50.54
64.27
64.43
75.50
Table Detailed performance DYNMOE and various MoE settings MNLI dataset
MNLI
K 8,A
K
8,
8,6
16, 16, K
16,
K 16,
DynMoE
Ir= 20-5
86.56
86.70
86.57
86.61
86.63
86.51
86.37
86.46
56.3
51.44
35.45
35.45
34.54
Table 10: Detailed performance DyNMoE and various MoF. settings RTE dataset
RTE
K 8
K 8,K
K
k
K
8,
R
16,8 K
K
16,
2 R
16,
K
16,
8 DynMoF
Ir 2e-5
73.04 72.44
70.52
74.13
74.37
74.01
66.19
75.33
72.56
72.80
3e-5
74.85
75.09
73.53
73.16
72.32
75.21
73.53
73.41
5c-5
58.48
34,39
62.45
65.10
63.78
63.06
58.84
63.66
65.22
Table 11: Performance of DyNMoE on language tasks: Our study investigates the performance of DYNMoE on language tasks using the GLUE (Wang al., 2018) benchmark, with BERT-large serving as backbone model. The baselines including traditional MoE methods with different number of experts K and top-k. In our implementation of DYNMOEE we configure the maximum number of experts to 16. with an initial setting of 8 experts. The number of experts is dynamically adjusted in each epoch for DyNMoE. The represents experiment failure, final results could be obtained using Gshard loss.
Algorithms
COLA
MRPC
QNLI
MNLI
RTE
Average
MoE [K
8,
64.10.1..4
90.14:10.60
92.48-0.21
86.56=0.06
73.041:2.13
81.26
MoE (K
8,k 2)
64.51=10.1
90.19-0.17
92.39=0.08
86.70=0.23
74.85+1.96
81.73
MoE (K
64.9410.62
89.74:10.99
92.52=0012
86.57=0.28
75.09+1.84
81.77
MoE 8
64.03310.54
89.3610.09
92.46=0.09
86.61=0.26
74.37:10.78
81.37
MoE 16. 1)
63.6310.20
89.8110.30
92.39=0.21
86.63=0.17
74.0110.29
81.29
MoF (K
16, 2)
64.71+1.21
90.18+1.33
92.53-0.07
86.73-0.43
72.32+3354
81.29
MoE (K- 16, 4)
64.1241.42
89.74+0.40
92.65-0.09
86.59=0.16
75.33+00995
81.69
MoE (K- 16, 8)
64.37::1.14
90.35:+0.68
92.49-0.111
86.51=0.20
73.53+2.21
81.45
DYNMOE, Gshard Loss
64.88:10.86 65.17+0.26
89,85-10229
92.42=0.07
73.41:0.68 73.41+11.96
DYNMoE
90.64++0.26
92.59=0.08
86.37=0.13
81.64
Table 12: Detailed results on DomainNet dataset: We report the detailed test results each domain of the DomainNet dataset.
Algorithms
clip
info
paint
quick
real
sketch
Averuge
GMoE (with DYNMOE, Gshard Loss)
66.8
23.8
54.1
15.9
68.7
54.9
47.4
GMoE (with DYNMoF, Diverse and Simple Gating Loss)
68.0
24.4
55.4
16.6
69.5
55.1
48.2
Load Balance (Table 14): The frequency with which each expert is activated, calculated as (expert activation time total token count).
Efficiency (Table 15 and 16): The top-k values per layer and the top-k activation frequency, calculated (number of tokens that activate k experts total tokens). We can find that
18

---

Published as a conference paper at ICLR 2025
Table 13: Performance of DYNMOE with load balance and efficiency losses. We conduct experiments using the MoE-LLaVA setup, incorporating (1) ballbaaanning loss and (2) an efficiency loss enforce sparsity, as proposed by Zheng et al. (2025).
Performance (StableLM)
VQAv2 GQA
VizWiz SQA
TexVVQA
POPE
MME
MMBench
DYNMOË
77.4
61,4
40.6
63,4
48.9
85.7
1300.9
63.2
DYNMOE + load balance
77.1
61.6
37.0
61.4
50.3
85.3
1313.5
61.7
DYNMOË + load balance + efficiency
77.1
61.8
39.4
62.9
49.7
85.4
1321.2
61.9
Table 14: Activation frequency per expert of DYNMoË with load balance and efficiency losses. We conduct experiments using the MoE-LLaVA setup, incorporating (1) load-balancing loss and (2) efficiency loss enforce sparsity. as proposed by Zheng al. (2025). We report the activation frequency of each expert at layer on the VQAv2 dataset.
Activation Frequency per Expert (VQAv2, layer 0) Expert Expert 2 Expert Expert 4
MoE (top-2) DYNMOE
0.36
1.29
0.16
0.19
0.29
0.97
0.48
0.35
DYNMOE + load balance
0).1
0.50
0.90
0.68
DYNMOE load balance + efficiency
0.45
0.52
0.42
0.63
Table 15: Sparsity of DyNMOE with load balance and efficiency losses. We conduct experiments using the MoE-LLaVa, VA setup, incorporating (1) load haddbalanciin loss and (2) an efficiency loss to enforce sparsity, as proposed by Zheng al. (2025). We report the average top-k value of each expert at each layer on the VQAv2 dataset.
Top-k per Layer (VQAv2)
Layer 0 Layer
Layer
Layer 6 Layer 8 Layer 10 Layer 12
Layer 14
Layer 16
Layer 18
Layer 20 Layer 22
DYNMoE
2.09
1.07
1.57
1.06
2.04
1.03
L.3
1.00
1.03
1.02
1.02
1.00
DYNMOE load balance
2,88
1.25
1.59
1.27
1.26
1.13
L.77
1.20
1.12
1.33
1.30
1.00
DYNMOF lead balance a efficiency
2.02
1.25
1.81
1.37
1.6
1.20
1.47
2.30
1.07
1.37
1822
1.00
Table Top-k frequency of DYNMOE with load balance and efficiency losses. conduct experiments using the LLLVA setup, incorporating (1) a load balancinann loss and (2) an efficiency loss enforee sparsity, as proposed by Zheng et al. (2025), We report the frequency of activating top-k experts for cach configuration.
Top-k Frequency (VQAv2)
Top-1
Top-2
Top-3
Top-4
DYNMoE
0.79
0.16
0.04
0.01
DYNMoE load balance
0.65
0.26
0.06
0.03
DYNMoE load balance + efficiency
0.58
0.32
0.09
0.01
Although it does not explicitly enforce load balancing, the original DynMoE achieves load balancing comparable to that of the standard top-2 MoE (Table 14).
Adding the load balance loss slightly decreases the performance of DynMoE (Table 13) while increasing the number of activated experts (Table 15). However, it improves load balancing (Table 14).
Adding an additional efficiency loss top of the load balance loss improves performance (Table 13) and helps overcome some extreme cases, such as the reduction of the top-k values in the bottom layer from 2.88 2.02 (Table 15), and reduce the number of tokens that activate all experts (Table 16). Moreover, the efficiency loss further enhances load balancing (Table 14).
8.3 COMPARISION TO TOP-P GATING BASELINE
In Table 17, we compare DYNMoE with the method proposed by Huang et al. (2024), which replaces the traditional top-k gating with top-p gating. We set p 0.4 suggested in the original paper. Our results show that DynMoE achieves better performance without requiring the additional parameter.
8.4 NUMBERICAL RESULTS ON MORE VISION TASKS
In Tables 18, 19, and 20, we show the performance of DYNMoE on more vision tasks, and also investigate the impact of strategies on initialize the new experts, including
Average: Averaging the parameters of existing experts initialize the new expert.
19

---

Published as a conference paper at ICLR 2025
Table 17: Comparision to top-p gating method. We conduct experiments using the MoE- LLaVA setup, and compare DYNMOE to top-p gating method (Huang et al., 2024).
StableLM
VQAv2
GQA
VizWiz
SQA
TextVQA
POPE
MME
MMBench
DYNMOE
77.4
61.4
40.6
63.4
48.9
85.7
1300.9
63.2
top p (p=0.4)
77.1
61.7
36.0
62.8
48.6
85.2
1332.9
62.3
Table 18: Fintuneing results on Tiny-ImageNet dataset. We fine-tune the pretrained ViT-S (Dosovitskiy al.,
2020) model on the TinyImageNet (Le & Yang, 2015) dataset and report the accuracy every two epochs.
TinyImaget ImageNet (Finetune, VIT-S, 2 MoE layers)
E1
E3
ES
E7
E9
E1
E13
B15
E17
E19
E:00
MoE (K 8, k=
78.32
82.79
84.03
84.83
83
85
85
85.82
86
27
86
86.61
86.65
MoE (K 8, 8, 2
78.53
82.95
84.05
84.74
84
99
85.00
85
86
86.63
86.58
86.72
MoE (K k=
79.25
83.38
83.73
84.72
85.00
85.50
86.64 86.56
MoE (K k
79.20
83.30 84.02
80
84 86
85
62
86.08
86 12
86
86.73 86.58
DynMoF, (Original, avg topk=6.5)
79.10
83.09
84
84.84
85.
18
85.
56
85.91
86.09
86.37
86.40 86.70
DynMoE (Average, avg topk=6.0)
79.19
83.48
84,21
84.84
85
85
76
86
86.41
86.49
86.70
86.75
DynMoF. (W-Avergee, avg topk=6.0)
78.96
83.18
85
84
85 34
93
86.
10
86
30
86.60
86.70 86.80
DynMoE (Most activated, avg topk=6.5)
79.09
83.57
84.21
84.62
85.40
85.87
86.45
86.40
86.66
86.70
W-Average: Using weighted averaging of the parameters of existing experts, where the weights correspond to the number of experts to be activated.
Most activated: Initializing the new expert using the parameters of the most frequently activated expert.
Results show that (1) DynMoE converges faster than standard MoE settings; (2) W-Average achieves the best performance in most cases; and (3) incorporating load balance loss and efficiency loss accelerates training while improving performance.
8.5 EFFICIENCY EVALUATION
In Table 21, compare the training efficiency of DYNMOE with standard top-1 and top-2 gating MoE models. The results show that DYNMOË trains faster than top-2 gating but slower than top-1 gating.
In Table 22, we show the overhead introduced by the top-any gating method. The results indicate that the primary source of overhead comes from the routing process, which likely explains why DynMoF (with forced top-1) is noticeably slower than MoE (also with forced top-1).
8.6 ABLATION STUDIES ON AGGREGATION WEIGHTS
In Remark 3, we discussed why we did not consider the magnitude of scores when averaging the expert outputs. In Table 23, we show that using different expert scores significantly reduce the model performance.
9 ADDITIONAL VISUALIZATION RESULTS
9.1 ACTIVATION FREQUENCY
We present the activation frequency of experts across various MoE layers and evaluation tasks using different backbones: StableLM-1166 (Figures and 10), Qwen-1.88 (Figures 11 and 12), and Phi-2- 2.7B (Figures 13 and 14). The results suggest that compared the StableLM-1.c backbone, experts are more uniformly activated for models utilizing Qwen-1.80 1.8B and Phi-2-2.7B as backbone LLMs.
Table 19; Pretrain results in CIFAR-10 dataset. We pretrain the VIT-S (Dosovitskiy et al., 2020) model on the CIFAR-I0 dataset and report the accuracy every 100 epochs.
CIFARIO S. 2 McE Ace 100 Epoch
200
400
500
MoE MoFF [K
IK = k
72.86
77.51
30.10
$1.08
75 79
50
80
X1.11
DynMol (Onginal, lopk=]]
76.81
79.2
81.77
83.50
DysMoE (Average, avg topk-7)
74.70
92
82.51
33.57
DyaMot: (W-Avenge, avg tock-6655
26.16
78.77
81.30
82.01
DyaMoE (Maa avg (opk-6.5)
20

---

Published as a conference paper at ICLR 2025
Table 20: Pretrain results ImageNet dataset. We train VIT-S from scratch for 200 epochs ImageNet dataset (Deng et al., 2009) with batch size 512 across A100 GPUs. 3, 6, 9] are replaced with MoE layers, and the maximum number of experts per layer is 4. The learning rate set to le-4, and we use the Adam optimizer with parameters [0.9, 0.99] and cosine learning schedule, while the weight decay is set to 5c-5. During evaluation, indicates DynMoE with simple and diverse gating loss, and DynMoF-B indicates DynMoR with simple and efficiency loss.
ViT-S architecture. layers [0,
set the batch size to 128 and use A100 GPU. DynMoF-A
diverse gating loss load balance
Train Time Train Whll-Clock Time
Evolsaiis Time: Inferener Tame en Roslint Inferetaat Tine out Cating Inferenes Time Expert Pasxing AscdF layet)
(dag
(s/huich)
(ES
laper) (ms
layer)
(9)
DeepSpeed MoE (ug-]
0.39
2.3
0.53
0.05
14.54
34.33 47
64.
DertSpeed (np-1)
010
(k=1.63)
B.9
DynMisll-t (km1.25
0.41
0382
0.46
31.04
148 11
68.5
Table 21: Training efficiency comparision.
Time (s/batch)
ViT-S, ImageNet (Train, A100, batch size=32) (Train, A100, batch size=512) (Test, A100, batch size=256)
MoB-LLaVA
ViT-S, ImageNet
Top-1 MoE
1.31
0.39
0.18
Top-2 MoE
1.60
0.63
0.32
DynMoE
1.48
0.51
0.27
Table Overhead of top-any gating. We report the time taken key steps top-any routing, including routing, gating, and expert passing.
Time (per sample and MoE layer) Routing (ms) Gating (ms) Expert Passing (ms)
MoE-LLaVA VA (enforced top-1) DynMoE (enforced top-1)
0.04 0.45
0.60 0.75
1.08 1.10
Table 23: Ablation studies on aggregation weights. We conduct experiments using the MoB-LLaVA setup, the "weighted scores" indicates averaging the expert outputs based on the gating outputs.
StableLM
VQAv2 GQA
VizWiz SQA
TextVQA POPE MME MMBench
ours
77.4
61.4
40.6
63.4
48.9
85.7
1300.9
63.2 52.1
weighted scores
73.9
57.4
32.1
61.3
46.9
84.2
1176.8
21

---

Published as a conference paper at ICLR 2025
MaVA-LS-B
D 168x4 Qon-VI-C-- 14.7
ME LoVA
VQN Test-dos Aounaey
DLIP 12g
HLIP OPT
ROSMOS- 1-1 or
#MRFIIC--R
"BLIP-2
Activated Parometess (Billooss
Figure 8: Comparing the performance efficiency models. The æ-axis represents number of activated parameters, while the y-axis shows the performance on the Visual Question Answering (VQA) task.
Reparts Artivation by Loper (GmghQA)
Expertt Artinstinn 'ye Tayer (1 (LlaV)-Ponch)
Expert
70
Layre
Lagrr
(a) GraphQA
(b) LLaVA-Bench
Exgaril Activo Jonts by Layer (MB3AG6)
Eacartia Activsationn Lapar (MMBBadh)
Eagert ItS
Zaguet
Lave
(c) MM-WE Vet
(d) MMBench
Experts Aucivation (MNN
Esperis Activations by Layer (UUP
Fagen
Eagert
10
34
16
21
2
10
34
16
18
21
Tagnr
Tager
(e) MME
(f) POPE
Figure 9: Activation frequency of experts on various MoE layers and evaluation tasks using StableLMM as backbone.
9.2 AVERAGE TOP-k
In Figures 15 and 16, we illustrate the average top-k of DyNMoE models using Qwen and Phi-2 as backbone LLMs.
9.3 LAYER WISE EXPERT SIMILARITY MATRIX
In Figures 17, 19. and 21, we illustrate the similarities between various expert representations, specifically, different rows of W across multiple MoE layers. These comparisons utilize Mt.
22

---

Published as a conference paper at ICLR 2025
Eqperts Actisaiaon by Lagur (A
Zaparte be Art inaiiio by Lopor (Tea
Tagent
Eapent
Layre
Leger
(a) ScienceQA
(h) TextVQA
Eeperte Actrntionn by Leper (VieWad
Espert
Lanis
VisWiz
Figure 10: Activation frequency of experts on various MoF. layers and evaluation tasks using StableI.M .. as backbone.
Exports Activation Loper (GmghQA)
Experts Artinttone Tayer (LlaVA-Bencc)
Expert
Exuert
0
Layre
Leyt
(a) GraphQA
(b) LLaVA-Bench
Experts Activecions by Leetr (3M Vet)
Expecto Artivatios by Love (MMDendh)
Expert
Espert
Laye
Layer
(c) MM-
(d) MMBench
Eapuet by Layur (M8)
Esperts by Layer (UPE
Eogert
Eigeet
10
125
21
10
21
Layee
Tager
(e) MME
(f) POPE
Figure 11: Activation frequency of experts on various MoE layers and evaluation tasks using Qwen as backbone.
23

---

Published as a conference paper at ICLR 2025
Eqperis Artisaiane by Lager (SammaAA
Zaparte be Art Intisiiion by Lopor (Tea
Eapent
0
Layre
Lager
(a) ScienceQA
(h) TextVQA
Eeperte Actrntions by Loper (VieWad
Esport
VisWiz
Figure 12: Activation frequency of experts on various MoE layers and evaluation tasks using Qwen as backbone.
Exports Activation Loger (GmghQA)
Experts Artinttone 'y' Tayer (LlaVA-Bench)
Expurt
Expert
70
Loyre
Layer
(a) GraphQA
(b) LLaVA-Bench
Experts Activntions by Leetr (3M Vet
Expects Artivatios by Love (MMDench)
Expert
Espert
Laye
Laye
(c) MM-Ver
(d) MMBench
Eapust to by Layur (MM)
Experts by Layer SUPE
Eogert
Exgert
16
18
211
26
28
Tagne
Tager
(e) MME
(f) POPE
Figure 13: Activation frequency of experts on various MoE layers and evaluation tasks using Phi-2 as backbone.
24

---

Published as a conference paper at ICLR 2025
Experto Artinationa Lager (8 Q4)
Tapurtx instion by Layer (Teat(QA)
Pippert
Eigper
20
26
(a) ScienceQA
(b) TextVQA
Esparle Artivations Layer (Ni4Wia)
Eagert
20
30 16
18
20
22
24
26
28
Tagne
(c) VisWiz
Figure 14: Activation frequency experts on various MoE layers and evaluation tasks using Phi-2 as backbone.
Vigatz
POPE
Valiee Dua leg
MiDBaak Dy Ring di
Dyalk eny ke
By-MaR me lop 185
Ta eng bay MaRALLAA 2
MaSittatA 2
MAGALLAA tepol
346-4LaNA tepul
toper
352
Lec:
Laose
Laver ID
Lo
Ltt
Dystiall iou 280
2yy354 lipll LST 12-5 1
Drallit ligol: 132 Malll 11alh log d
Dysakill light LST
Mall 1aal upel
Veal Wl teg
4 10
02427 11
2 1211111 14
Figure 15: Average top-k activated experts of DYNMOE vision-language benchmarks, using Qwen as language backbone.
POFE
Decklit
DeibGi tipl: 10N LIAAN LOUR
2eddi 106 Sppa
Eretid up4 Lob
Teetfif kepdl 106 Sical
mon The MMll bell 102
Viaff
Figure 16: Average top-k activated experts of DyNMoE on vision-language benchmarks, using Phi-2 as language backbone.
25

---

Published as a conference paper at ICLR 2025
Ligar
Layer ?
Lager
Layee 6
Popert
Popert
Degine
LeVa
Export
Expert
3M0
Exsent
3014
Expect
6.8
Popert
-MMG
1800
Repert
1.00
MML
Dogert
MXQ
100g
Expert
M/C
1,00
Export
Expert
6M
400
100
Exsert
4:14
100
Expert
4:001
100
040
1/0
Exper
Eapeet 2
Expret
Eappet
Expost
Dxpett
Eaport
Expest
Expen
Papers
Papor
Expent 3
Exper
Expert
Lugaer
Layer 10
Loger
Export
Experi
Bigent
Expert
Export
Expert
Popert
Poyert
Expert
Export-3
Expert
Esport
Esport
Brpeet
Eaport
Erpect
Enpert
Export
Brgent
Export
Export
Emrpet
Export
Eppert
Lager
Liger
Lager
Logne
Export
180
Expert
1:00
400
3805
Exsort
1.00
4:00
MM13
Expert
-164
Expert
MB0
Report
Deport
AMGS
Expert
084
Expett
Eecert
Expest
Eaport
Espert
Eipert3
Expert
Expert
Engert 2
Espert
Erpert
Enpert
Export
Expert
Export
Export
Expert
Expert
Figure 17: wise expert similarity matrix LM), We record the experts' cosine similarity per layer during test time. It out the cosine similarity between experts is close
Layer
Layer
Layer
Layer
Layre
Layer
Layer
Layer
Layer Layer
Layer
Layer
Expert
01000
24015
0.009
+LEG3
-0.00
Expert
Expert
0:303
10.000
0.011
3:011
-1.01
1/014
0.055
Expert
0.90
0.091
3018
01000
2:011
0.066
.0077
0.000
6.31
0.00
0.333
Figure 18: Layer-wise expert activation threshold (StableL.M), Darker-colored experts are likely be activated compared to lighter-colored experts.
1.6B, Qwen-1.80 1.8B, and Phi-2-2.7B the backbone LLMs. The findings demonstrate that these expert representations are nearly orthogonal, suggesting that different experts capture diverse features, which could potentially enhance the model's capacity.
9.4 VISUALIZATION OF G
In Figures 18, 20, and 22, present the values of the learned threshold G, employing Mta 1.6B, Qwen-1.8B. and Phi-2-2.7B as the backbone LLMs. The results reveal that for each MoE layer, there is one expert that is more readily activated, This observation is consistent with the design of Deepseek-MoF (Dai et al., 2024).
26

---

Published as a conference paper at ICLR 2025
Lagar
Lagarr
Layar
Eppert
Reyert
Doport
Leta
2016
Expert
-012
Expert
Expert
Excort
Expert
Export
Esperi
Exyert
Expest
Export
Export
Expert
Export
Export
Export
Depest
Export
Expest
Expert
Expor
Expert
Expor
Exeert
Expest-
Export
Lagner
Layer 10
Lattr
12
Ligher 11
Expert
1.:33
Expert
1:400
ABY2
Eigent
6:13
6:33
Expest
1.00
1B34
6:43
Poport
MBC
Popert
6.42
1,00
Doger
AML
1.50
X091
Expert
L00
4.31
-U05
Export
1.00
Expert
Eigent
AMRI
Expest
180
Popert
400
M0
Peyert
AM0
Deport
AMUS
0.6
X00
Expert
-M0
Exper
Espett 2
Expor.3
Eapert
Esport
Erpert? 2
Fapott
Ergert
Erper
Eapert
Brgeot
Expott
Eapott
Enpent 2
Expert13
Export
Laget 16
Layer 18
Layer
Lagtr 22
Expert
A335
Expert
Degort
ABIL
1.50
100
031
Expert
100
Expezi
Boxe
Export
1000
Export
440
1M
Experi 4
0:00
100
Exjent
<Bc
130
Expest
Eaporr
Eigert
Siport
Esport
Export
Expert
Eapet
Expert
Gipert
Esport
Expert
Expor
Expor
Exper 2
Eaport
Eigent
Figure 19: Layer-wise expert similarity matrix (Qwen). We record the experts' cosine similarity per layer during test time. It turns out the cosine similarity between experts is close to 0.
Layer 0
Layer 2
Layer
Layer
Layer
Layer 10 Layer 12 Layer
14 Layer
16 Layer
18 Laver 20 Layer 22
Expert
Expert
0:00
0.000
A.007
-B/02
0.006
2/001
0.000
.0/0/
.0.004
0/005
00/01
0.313
Expert
0.90
0.011
0.306
3
0.006
2.003
0.000
6/005
0.610
.0/033
0.006
0.312
Expert
Figure 20: Layer-wise expert activation threshold (Qwen). Darker-colored experts are more likely to be activated compared to lighter-colored experts.
27

---

Published as a conference paper at ICLR 2025
Lager
Layer
Lager
Leyer
Export
Expert
Regert
LAVB
and
Exgesk
Expor
Eapesi
Expert
Expert
Expert
Regert
Expest
Expot
Expor
Dapnet
Expertt
Eapner
Enppree
Expen
Eapeet
Eapret
Expest
Expest
Eapeet
Expert
Eaper
Bapert
Expeet
Eayeet
Expeet
Layner
Layer 10
Later 12
Loynr 14
Export
Expezi
Export
Export
-007
Expert
MA12
Expert
Exgent
1.00
1010
Expert
1B04
6:10
Export
100
Expssi
Export
Expert
6:00
Expor
6:80
Expert
6.10
180
Expert
1.01
Export
8:40
180
Eaport
Elpertt 2
Risentt
Expert
Enpprt
Eupert
Enpert
Erpest
Eapert
Eigpert
Eaport
Export
Elspett 2
Eapert 3
Eipertt
Laynr
Layer J8
Laytr
Export
Expert
Export
Popert
Pepert
Repert
Fopert
Expert
Popert
Popert
Pegprrt
Expert
Erpent-2
Expert
Ripert-
Export
Brpert
Export
Brrect
Enpert
Export2
Export
Expect
Eapert
Expert
Eppert
Lagetr 24
Layer
26
Layer 28
Ligntr 30
Pspert
6:423
Popert
1:m0
MBQ
AUBL
Pegert
0.00
6:3G
Papert
1a00
6:00
E/Rt
Expert
1:00
Expert 2
6:4
1,00
4:00
MBC
Depertt
1,00
Export
1:0K
4:80
-000
Paport
6.40
1,00
Expert
Pepurtt
Poport
-200
Epert
Expert
Bepert
Ju0
AM0
Export
Depert
Expest
Expert
Expert
Exper
Fapert
Eepar
Expatt
Raper
Eqpor
Paper
Eapert
Enpert
Depar
Expar
Figure 21: Layer-wise expert similarity matrix (Phi-2), We record the experts cosine similarity per layer during test time. turns out cosine similarity between experts close to
Lager
Lagze 2
Lawer
Layer
Leper 8
Lawer
Layer 12 Lager
Lawe 15 Layer 18 Layer 20 Lavve 22 Lave 24 Layer 26 Loger
Lawe 30
Expert
2013
Expest 3
6443
canm
6/10
44121
0.233
6:4c
4/1
0423
Figure 22: Layer-wise expert activation threshold (Phi-2). Darker-colored experts are more likely be activated compared to lighter-colored experts.
28

---

Published as a conference paper at ICLR 2025
Distribttion of Number of Activatedd Experta Per Laver (StableLM)
1.0
cimes
Relalver Prsqanee
0.5
0.6
0.4
0.0
8
12
16
22
Layns
Destribation of Number of Autivated Experts Per Layer (Qwen)
1.0
timees
Reladive Prequeeee
0..5
0.6
0.4
0.0
222
Layers
Distribusion of Number of Activated Experts Per Layer (Phi-2)
1.0
timnes
Relalive Prequmey
0..
timnes
times
0.6
times
0.4
0.0
0
Layers
Figure 23: Distribution of number of activated experts in each layer. We report the results of StableLM. Qwen, and Phi-2 models, respectively.
Nev
CIFAR-IC
DyakluE (Orggza)
DenMeE (Crzza))
DyabioE (Anmmar)
22
DvaMLE (Araagd)
Dyabbok (Woboemrr))
BsaMLE (WSAverage)
DyaMaR (Nien Avthoate))
RenMMe (Midd Aitboned)
(DC
XGeB (K-5, k-2))
MrE (SCed, 1o-2
Nolk (K-5, k-d)
13
100
150
370
200
Kgooh
Egoch
Figure 24: Convergence curve on CIFAR10 TinylmageNet datyasets.
29

---

Published as a conference paper at ICLR 2025
Table 24: Efficiency evaluation of DYNMOE comparing to MoE-LLaVA. We conduct experiments on single A100 GPU (80 GB) paired with 16 CPUs using identical environment and identical training/inferent contigurations. We report the performance of MoE-LLaVA using DeepSpeed's top-2 gating implementation. The symbols and indicate that lower and higher values, respectively, denote better performance.
Model
Training FLOPs | Inference FLOPs 4 Inference MACs | Memory Usage (GFLOPs/token)
(TFLOPs/step)
(GMACs/token)
(GB)
MoE-LLaVA (StableLM)
18.23
27.62
13.34
5.98
DynMoE-LLaVa (StableLM)
17.97
25.25
12.13
5.98
MoF-LLaVA (Qwen)
34.27
23.36
11.30
6.37
DynMoE-LLaVA (Qwen, Ours)
34.61
22.17
10.73
6.37
MoE-LLaVA (Phi-2)
63.43
22.73
10,46
DynMoE-LLaWe (Phi-2)
63.36
44.92
21.72
10.46
30


Reviews:
[{"review_id": "GgZt27ArA2", "reviewer": "Reviewer_y8zU", "summary": "This work presents DynMoE a mixture-of-experts approach which uses 2 mechanisms:\n\n  * Top-any routing which computes a score for each token-expert pair and a learnable threshold per expert, therefore allowing a rather dynamic use of tokens/expert computations.\n  *  keeping track of expert utilisation to grow/shrink throughout the training, and initialising the new experts routing based on the non-routed tokens.\n\nIt shows results in vision, language, and vision-language tasks and shows it comparable to training with a fixed (but sweeped) number of K experts and top-k tokens routed.", "questions": "My main questions are highlighted on the main weaknesses above.\n\n\nAfter rebuttal from authors with added experiments on (1) Included a load-balance loss in our model; (2) Introduced an efficiency loss to enforce sparsity, as suggested in [2]. I have raised my score from below acceptance to accept.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Together, the two proposed mechanisms achieve an expert architecture which automatically decides the number of expert activations and the total number of experts throughout the training. Those are two problems that if addressed correctly can make MoE simpler to use by not having to fix them ahead of time.", "weaknesses": "In my understanding, early design of MoEs [Shazeer et al., 2017] had many engineering challenges in consideration such as efficient use of compute units and network, which led to many of the fixed top-k routing decisions (e.g. why should the compute unit that processes expert A process only 10 tokens.. if it has to wait for another compute unit that has to process 100 tokens?) which is also similar to the concerns of equal utilisation of experts with the purpose to efficiently utilise hardware. With that in mind I find it hard to understand from the text in this paper: (a) why those concerns don’t apply here, (b) how has hardware changed overtime such they are no longer important, (c) if the approach here would not scale to utilise hardware as efficiently as a fixed top-k routing as one scales the method.\n\nUnclear on what mechanisms are at play to optimise the gating mechanisms. In particular it appears to this reviewer that the only gradient to the gating and gating thresholds is via the (1/k) factor of eq (6).. If so one can tend to understand that no per-expert specific gradient is observed.. By that I mean, for a token, all of the params used to compute its gating values will observe similar gradient directions regardless of which expert they refer to.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hu5Z3lFq2F", "reviewer": "Reviewer_4q9o", "summary": "This work proposess DynMoE, an enhanced version of the Mixture of Experts layer. The method enables top-any gating, allowing for the execution of different number of experts for each token. In the proposed method experts that are unused for a predefined period of training time are discarded, and new experts are added if there are tokens that do not activate any experts.\n\nThe proposed method is relatively simple and appears to achieve its intended effect of reducing the hyperparameter tuning burden associated with MoE layers. However, the method is compared to only a single variant of MoE, and improvements are visible on only a subset of the results. There are several other issues, with not adequately detailed description of the method being one of them. As of right now, I recommend rejection of this work, but I am willing to increase the score if the authors properly address my concerns.", "questions": "- How are experts initialized at the start given a pre-trained model? Are parameters of experts that are added during training initialized randomly, or are the weights of the most recently discarded expert re-used?\n- Are the auxiliarry losses somehow weighted? If yes, why is it not reported in the paper?\n- Is the MoE implementation used in this work dropless [6], i.e. does it drop tokens if the predefined capacity of an expert is exceeded? If it is not dropless, how often are tokens dropped and how does DynMoE compare to standard MoE in terms of load balance?\n- In the appendix the authors write: \"For each epoch, we begin recording routing at 1/3 of the epoch and complete recording routing and execute the adaptive process at 2/3 of the epoch.\" Can the authors clarify why is this needed?\n- Currently the method relies on a linear router. However, some works assume a deeper router [7,8]. Can the method be extended to non-linear routers?\n- Remark 3.1 is not fully convincing to me. Can the authors explain more in detail why different ranges of expert scores are supposed to be undesirable? \n\n#### References:\n[6] Gale, Trevor, et al. \"Megablocks: Efficient sparse training with mixture-of-experts.\" Proceedings of Machine Learning and Systems 5 (2023): 288-304.\n\n[7] Zhang, Zhengyan, et al. \"MoEfication: Transformer Feed-forward Layers are Mixtures of Experts.\" Findings of the Association for Computational Linguistics: ACL 2022. 2022.\n\n[8] Chi, Zewen, et al. \"On the representation collapse of sparse mixture of experts.\" Advances in Neural Information Processing Systems 35 (2022): 34600-34613.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Elimination of hyperparameters introduced by MoE, which was the motivation of the paper, is practical and useful, and thus the work could be of interest to the community. The method is also relatively simple, which may be regarded as an advantage.\n- The main evaluation is relatively thorough in terms of number of datasets and models. The authors evaluate their method on three different setups, with one being a multimodal task.", "weaknesses": "- **Limited novelty and lack of proper attribution:** While the authors demonstrate some improvement over standard MoE layers, the idea of dynamically selecting the number of executed experts on a per-token basis is not new [1,2,3,4]. Moreover, none of these are mentioned in the related work section. The authors should properly discuss these works and highlight the differences of the proposed method.\n- **Lack of comparison to existing MoE methods from the literature:** Given the above, does the proposed method provide any advantages over the existing top-any literature? An empirical comparison with these methods would strengthen the contribution. As of right now, the authors compare their method with only a single baseline (one of the top-k MoE variants), which is an extremely low number for an empirical-only work. Would the benefits of the proposed method hold even when stacked against alternative top-k MoE variants? Some of these variants also allow for an effectively different number of experts executed per token in inference [5]. \n- **Unconvincing real latency/throughput evaluation:** A major flaw of the work is that authors only briefly mention the actual speedup that the method supposedly provides. The authors write (line 488): \"To ensure a fair evaluation, MoE-LLaVA employs the expert dispatching implementation from DynMoE by fixing the top-k values.\". This statement is unclear, and for me it suggests the opposite of a fair evaluation, as the alternative implementation may significantly slow down the original MoE-LLaVA. The readers should be aware of any overhead stemming from the introduction of top-any routing. I would at least expect a non-negligible overhead stemming from the non-constant number of token-expert executions. Finally, to provide a complete picture, the authors should also measure and report the throughput of dense models for comparison.\n- **No from-scratch training experiments:** All experiments are conducted on pre-trained models, which are then converted to sparse MoE models during the fine-tuning. Mixture-of-Experts layer was originally proposed to scale up the number of parameters without slowing down from-scratch training. Although the pre-trained setup is as important, demonstrating that the proposed method is also appropriate for the from-scratch setup would strengthen the paper.\n- **Standard deviations not being reported:** The authors do not report standard deviations of the results in any of the experiments. This is particularly surprising as the authors write in the appendix that the scores are averaged over three random seeds. Without standard deviations it is hard to estimate the significance of the results, especially since some of the scores in Tables 1, 2, 3, and Figure 4 are really close. The authors also do not present any evidence that training of the proposed DynMoE is stable in any separate experiment.\n- **Method description and experimental setup not being adequately explained:** After multiple passes through the text some crucial aspects are still unclear to me - see the questions section below.\n- **Readability:** I would recommend to do a pass through the paper to improve grammar and readability (e.g. \"and uses an additional gating network g to predict the scores that the input token embedding assigned to each expert.\" or \"To promising efficiency and avoiding burden communication, we only check if experts required to be added add or removed.\").\n\n#### References:\n\n[1] Nie, Xiaonan, et al. \"Evomoe: An evolutional mixture-of-experts training framework via dense-to-sparse gate.\" arXiv preprint arXiv:2112.14397 (2021).\n\n[2] Zheng, Haizhong, et al. \"Learn to be efficient: Build structured sparsity in large language models.\" arXiv preprint arXiv:2402.06126 (2024).\n\n[3] Huang, Quzhe, et al. \"Harder Tasks Need More Experts: Dynamic Routing in MoE Models.\" arXiv preprint arXiv:2403.07652 (2024).\n\n[4] Yang, Yuanhang, et al. \"XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection.\" Findings of the Association for Computational Linguistics ACL 2024. 2024.\n\n[5] Riquelme, Carlos, et al. \"Scaling vision with sparse mixture of experts.\" Advances in Neural Information Processing Systems 34 (2021): 8583-8595.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iuj87308TD", "reviewer": "Reviewer_kDxr", "summary": "This paper introduces the DYNMOE, an algorithm that automatically determines the number of experts activated for each token, addressing the dependency on hyperparameters (such as the number of experts and the number of activated experts) in SMoE models. DYNMOE uses an innovative top-any gating method and an adaptive training process to dynamically adjust the number of experts, while incorporating auxiliary loss functions to ensure sparsity and diversity among experts, improving training and inference efficiency. Experiments demonstrate that DYNMOE achieves comparable or superior performance and efficiency to well-tuned MoE configurations across vision, language, and vision-language tasks.", "questions": "see weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The paper clearly analyzes existing issues in MoE, particularly how the fixed top-k approach may not be optimal.\n\n2 The authors consider the extreme case of activating all experts in the top-any scenario and design a regularization loss to address this.\n\n3 The writing logic is clear.", "weaknesses": "1 In the top-any routing, is G a hyperparameter that needs to be predefined? Would this lead to cumbersome tuning?\n\n2 In the top-any routing, the selected k experts considered equally important by default. Would introducing varying importance levels for different experts yield better results? Although the authors mention related content in Remark 3.1, ablation experiments seem necessary.\n\n3 Why top-any MoE training strategy dones't lead to increased training time?\n\n4 The author could further elaborate on how the diversity loss in the auxiliary loss prevents activating all experts simultaneously.\n\n5 How exactly does DYNMOE remove experts? For example, if the interval for adding/removing experts is set to 300 steps and expert x is activated in the first 299 steps but not in the 300th step, will it be removed? Alternatively, if expert x is activated in the first step but not for the remaining 299 steps, will it be removed?\n\n6 For new experts added in DYNMOE, Would averaging the weights of other experts to initialize the new expert’s weights lead to better results? Or would random initialization yield better performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X18n0YsA1a", "reviewer": "Reviewer_DhtB", "summary": "This manuscript proposes an MoE framework in which the activated expert numbers could be learned, and each expert can be removed or added during the training phase. This framework removes the hand-design requirements for expert numbers, and outperform the hand-designed MoEs in vision, language and multi-modal tasks.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The studied problem is interesting and well-motivated, as the design of MoE have been adjusted by hand in previous studies;\n- The proposed method is novel and reasonable;\n- Empirical evaluation is relatively comprehensive to support the claims, and the observations provide useful insights for researchers in the field;\n- Good readability, which helps readers clearly understand the contribution and advantages.", "weaknesses": "Overall, this is a novel and solid work. I have some kind suggestions for further improving it:\n\n- Experiments\n  - For vision tasks, why not conduct experiments on the well-known ImageNet dataset? To my understanding, MoE is designed for larger-scaled data and model sizes. Experiments with larger backbones on ImageNet would make the method more convincing.\n  - Comparison with existing MoE models would strengthen the proposed approach, such as Swin-MoE [1], VoE [2] (on vision) and Switch Transformer [3], Mistral MoE [4], DeepSeek-MoE [5] (on language).\n  - Maybe it is useful to discuss the advatanges of this well designed MoE over those conventional backbones at the same computation budgets.\n\n- Literature review. To my understanding, the proposed method lies in the field of dynamic neural networks [6]. It is encouraged to discuss more about the relationship to other dynamic paradigms, such as layer skipping [7]. When it allows a token to select 0 expert, the proposed framework would naturally allows layer skipping.\n\n[1] Tutel: Adaptive Mixture-of-Experts at Scale.\n\n[2] Scaling Vision with Sparse Mixture of Experts. NeurIPS, 2021.\n\n[3] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR, 2023.\n\n[4]  Mixtral of Experts.\n\n[5]  Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.\n\n[6] Dynamic Neural Networks: A Survey. TPAMI, 2021.\n\n[7] Dynamic tuning towards parameter and inference efficiency for vit adaptation. NeurIPS, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)