Ada-K Routing: Boosting the Efficiency of MoE-based LLMs
Authors: Tongtian Yue
Longteng Guo
Jie Cheng
Xuange Gao
Hua Huang
Jing Liu
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Mixture-of-Experts
Reinforcement Learning
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: In the era of Large Language Models (LLMs), Mixture-of-Experts (MoE) architectures offer a promising approach to managing computational costs while scaling up model parameters. Conventional MoE-based LLMs typically employ static Top-K routing, which activates a fixed and equal number of experts for each token regardless of their significance within the context. In this paper, we propose a novel Ada-K routing strategy that dynamically adjusts the number of activated experts for each token, thereby improving the balance between computational efficiency and model performance. Specifically, our strategy incorporates learnable and lightweight allocator modules that decide customized expert resource allocation tailored to the contextual needs for each token. These allocators are designed to be fully pluggable, making it broadly applicable across all mainstream MoE-based LLMs. We leverage the Proximal Policy Optimization (PPO) algorithm to facilitate an end-to-end learning process for this non-differentiable decision-making framework. Extensive evaluations on four popular baseline models demonstrate that our Ada-K routing method significantly outperforms conventional Top-K routing. Compared to Top-K, our method achieves over 25% reduction in FLOPs and more than 20% inference speedup while still improving performance across various benchmarks. Moreover, the training of Ada-K is highly efficient. Even for Mixtral-8x22B, a MoE-based LLM with more than 140B parameters, the training time is limited to 8 hours. Detailed analysis shows that harder tasks, middle layers, and content words tend to activate more experts, providing valuable insights for future adaptive MoE system designs. Both the training code and model checkpoints will be publicly available.

Published as a conference paper at ICLR 2025
ADA-K ROUTING: BOOSTING THE EFFICIENCY OF MOE- MOE-BASED LLMS
Tongtian Yue Longteng G Jie Cheng! Xuange Gao Hua Huang', Jing Liu Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence, University of Chinese Academy of Sciences 3 School of Artificial Intelligence, Beijing Normal University
ABSTRACT
the era of Large Language Models (LLMs), Mixtureoof-Experts (MoE) architec- tures offer a promising approach to managing computational costs while scaling up model parameters. Conventional MoE-based LLMs typically employ static Top-K routing, which activates fixed and equal number of experts for each token regardless of their significance within the context. In this paper, we propose novel Ada-K routing strategy that dynamically adjusts the number of activated experts for each token, thereby improving the balance between computational efficiency and model performance. Specifically, our strategy incorporates learnable and lightweight allocator modules that decide customized expert resource allocation tailored to the contextual needs for each token. These allocators are designed to be fully pluggable, making it broadly applicable across all mainstream MoE-based L.MSs. We leverage the Proximal Policy Optimization (PPO) algorithm to facil- itate an end-to-end learning process for this non-differentiable decision-making framework. Extensive evaluations on four popular baseline models demonstrate that our Ada-K routing method significantly outperforms conventional Top-K routing. Compared to Top-K, our method achieves over 25% reduction in FLOPs and more than 20% inference speedup while still improving performance across various benchmarks. Moreover, the training of Ada-K K is highly efficient. Even for Mixtral-8x22B, MoB--bsed LLM with more than 140B parameters, the training time is limited to 8 hours. Detailed analysis shows that harder tasks, middle layers, and content words tend to activate more experts, providing valuable insights for future adaptive MoE system designs. The code and checkpoints will be released at https: www.ggtthhuu.oe wwwtommivvttttteeeeemmem Adda-K
INTRODUCTION
Over the past few years, the rapid development Large Language Models (LLMs) (Brown et al.. 2020b:
Raffel et al., 2020 Touvron et al. 2023a Chiang et al. 2023) has marked a significant leap towards Artificial General Intelligence (AGI). Generally, increasing the number of parameters in an LLM enhances its performance across diverse tasks, demonstrating emergent capabilities (Kaplan ct al. 2020 [Brown et al. 2020a)). However, this improvement comes with substantial computational costs for both training and inference, posing barriers to the broad application and efficiency.
In response to these challenges, the Mixture-of- offturr-of-EExerrts (MoE) (Jacobs et al. 1991 Eigen et al. 2013) architecture has gained popularity as a scalable solution that balances parameter increase with computational cost. MoF implementations in Transformers (Vaswani et al. 2017) have shown that significant model scaling can be achieved without a proportional rise in computational burden, thus maintaining efficient performance. These successes highlight the promising potential of MoB-based Team: Dai et al. 2024).
LLMs (Jiang et al. 2024
The core of the MoE architecture comprises set of expert networks governed by routing strategy. This routing strategy, executed via a learnable router (Fedus et al.) [2022a] Du et al. (022a),, selectively assigns each token to a limited number of experts. This sparse expert selection pivotal for MoE efficiency. The most common routing strategy is Top-K routing (Shazeer et al., 20176). It selects the best-suited experts for each input based on preliminary probability calculations, activating the top experts. Although recent studies ZZoph et al. 2022
Lewis et al. 2021a) have introduced adjustments
"Corresponding author.

---

Published as a conference paper at ICLR 2025
to ensure more balanced activation across experts, major limitation remains: the fixed activation numbers does not account for the varying importances of different tokens.
The importances of tokens can vary significantly, potentially influenced by factors such as the different demands of tasks (Rogers et al. [202]). the inherent characteristics of words (Schick & Schütze (2020), and the contexts in which they are used (Guu al. 2020). This oversight of token differences can lead to resource inefficiencies. Simpler tokens with minimal semantic significance may receive more processing power than needed, leading to inefficiencies. Conversely, more complex tokens those representing critical information or requiring advanced logical reasoning mmight not get adequate attention. This misallocation of expert resources not only suboptimizes performance but also hinders further improvements in computation efficiency.
To address the limitations of traditional Top-K routing in MoE models, we propose a novel, learnable Ada-K routing strategy that adapts expert allocation based on each token's inherent importance and difficulty. Specifically, this strategy introduces pluggable, lightweight allocator, easily integrable into existing MoE-based LLMs. The allocator dynamically determines the optimal number of experts for each token by sampling from the probability distribution it outputs over the possible expert counts. However, this sample operation is inherently non-differentiable. Hence, we employ the Proximal Policy Optimization (PPO) algorithm (Steennnn et al. 2020; Nakano et al.) (02T) to optimize the allocators end-to-end, towards ideal balance between model performance and efficiency.
We validate the effectiveness of our proposed Ada- K routing strategy by integrating it into four popular MoE-based LLMs and assessing their performance across multiple benchmarks. Compared to baseline models utilizing Top-K routing, our Rda-K routing consistently reduces the number of activated experts by 30% to 40%, while simultaneously enhancing overall performance. This reduction in expert activation directly translates to computational gains, achieving more than 25% reduction in FL.OPs and 20% speedup in inference time. Notably, this favorable balance is attained with minimal additional training, requiring the tuning of only approximately 2M parameters, with training times under 8 hours for all baseline models.
Furthermore, we conduct detailed analysis of the Ada-K routing strategy the task, layer, and token levels. Our findings indicate that harder tasks, middle layers, and content words tend to activate more experts, demonstrating the strategy's 's ability to allocate computational resources efficiently based on the importance of input tokens. These conclusions also provide valuable insights for future adaptive MoE system designs. Our contributions are summarized as follows:
An advanced dynamic routing strategy. We propose dynamic Ada-K routing strategy that adjusts the activated experts number on a per-token basis to enhance the conventional Top-K routing. Compared Top-K K routing, Ada-K K routing manages save over 25% FLOPs and achieve more than 20% acceleration in inference, while enhancing performance.
An efficient RL-based training framework. We introduce a learnable and lightweight allo- cator module that determines the optimal number of activated experts for each token through end-to-end training. Reinforcement learning techniques have been carefully designed and introduced to facilitate the training of this non-differentiable decision-making framework.
Comprehensive quantitative and qualitative experiments. We extensively evaluate our method across four popular MoE-based L.Mss, ranging from 14.3B to 140B parameters. Ada-K -K routing demonstrates consistent advantages over these baselines. The effectiveness of our proposed method is further validated by extensive qualitative analyses.
2 RELATED WORK
2.1 MIXTURE-OF-EXPERT
The sparse Mfxture-of-Eane Experts (MoE) layer, which includes a predetermined number of experts and routing network, is initially introduced to enhance the capacity of deep neural networks for NLP tasks in LSTM models (Shazeer et al. 2017a). This architecture is later extended to Transformers (Lepikhin et al. 2020) and adapted for computer vision (Riquelme et al. 202 Daxberger et al.) 2023), gaining popularity due to its robust scaling properties (Du et al. [022b] 1022k Clark et al. (022). In the MoE framework, extensive research has focused on refining routing algorithms (Hazimeh et al.,
2021 Lewis et al. 20216; Roller et al., 200T Zhou et al. (002). Approaches range from random routing (Zuo al. 2021) and activating all experts via weighted averages (Eigen al. 2013). to
2

---

Published as a conference paper at ICLR 2025
Takent
Tokon2
el
Router
Router
k Z
k"
Allocator
Taken1
Token2
(a) Top-K Routing MoE
(b) Ada-K Routing MoE
Figure I: Comparison Top K and Adaa K routing strategies for MoE: (a) In Top-E K routing, each token consistently selects fixed number of & experts based on predefined model configuration. (b) In Ada-K routing, each token dynamically activates a customized number of experts via our newly introduced learnable allocator module.
selectively engaging single or multiple experts (Fedus et al. 2022b Du et al. 2022b). However, they universally deploy fixed number of experts, irrespective of the differing complexities of input tokens (Lepikhin et al., 2020; Fedus et al., 20226). In our study, we introduce a parameter-efficien and data-efficient training framework that introduces lightweight learnable modules, i.e. allocators. These allocators seamlessly work with routers to dynamically assign expert resources across individual tokens. Our framework markedly reduces computational costs while achieving improved performance compared to the baseline models.
2.2 REINFORCEMENT LEARNING IN LLMS
Recent advancements in Large Language Models (LLMs) have been significantly influenced by Reinforcement Learning (RL.). For instance, the Reinforcement Learning from Human Feedback (RLHF) approach has demonstrated efficacy in aligning L.LMs with human-centric values and preferences (Bai et al.) 2022 Ouyang et al. 2022 Cheng et al. 2024). This method involves training a reward model (RM) that encapsulates human preferences and subsequently refining LLMs based
the reward signals generated by the RM. Additionally, some studies (Lee et al. 2023 Xu et al. 2023
Yue al. 2024) have investigated the LLM-centric multimodal representation learning. utilizing RL to improve the alignment between the semantic spaces of LLMs and visual concepts. Typically, they
often leverage the Proximal Policy Optimization (PPO) algorithm (Stiennon et al. 2020 Nakano et al. (021) to perform parameter optimization. In this work, we present novel PPO-based training framework tailored for MoE, a widely-used architecture in modern LLMs. This plug-and-plas framework seamlessly integrates with MoE models and its variants. We execute end-to-end agents training via RL, with these agents tasked with resource scheduling for expert activation, thereby significantly enhancing the inference efficiency.
METHOD
3.1 PRELIMINARY
We first provide concise overview of the sparse Top K routing MoE model. Structurally, MoE layer substitutes the feed-forward network (FFN) sub-block of the original Transformer layer with an expert network comprising N experts E {e1,e2,
For each token z; in the input sequence X, the activation probabilities for each expert are determined via router layer W:
P(x;) Softmax (W.z)) .x))
with W CCN being lightweight, trainable projection matrix. Top-K routing MoE employs routing strategy where the & experts with the highest weights in P(x,) are selected. The weights of the chosen experts then normalized, while that of the remaining experts set zero, indicating
3

---

Published as a conference paper at ICLR 2025
their inactivity:
P(xs)
g(m;) = E 2jETopk(P) Pj
(PTopK(P))
(2)
where (P) is an indicator function that is if E (P) and otherwise. The final output, derived from g(z;), a weighted average of the & chosen experts:
MoB(a;) = Σ gn(z;) en (xi)
(3)
n=l
3.2 ADA-K ROUTING
In this section, introduce the proposed Ada-K routing framework outlined in Figure It enhances the conventional Top-K routing by incorporating a novel component called allocator. The allocator works synergistically with the original router dynamically perform expert allocation for each token T₁. Structurally, the allocator is a lightweight, trainable linear layer, similar to the router, and is responsible for determining the optimal number of experts k* for each token. The allocator takes x; as the input and then produces a probability distribution over the possible number of experts:
Patlor (m) Softmax (Walloc x3;)
Based on this distribution, k* is obtained through non-differentiable sampling operation.
k*
(alloc
(5)
Subsequently, k- and F; passed to the router, which then activates the top k- experts to produce the weighted representations as described SecB.1] However, direct optimization for these allocators via backpropagation is precluded due to the non-differentiable nature of the sampling operation during the forward pass. To address this challenge, we propose a RL-based optimization framework, which will be detailed in the following section.
3.3 LEARNING STRATEGY
In our proposed framework, the entire training is specifically tailored the newly introduced alloca- tors, with the original LLM maintained in a frozen state to preserve its inherent capabilities.
The training objective is enhance both performance and efficiency, which can be decomposed into optimizing linguistic capabilities and minimizing the average number of activated experts. Regarding linguistic capabilities, we design PPO loss, which circumvents the need for differentiable action sampling. Regarding the activated expert counts, we incorporate a regularization loss to minimize the expectation value of the allocator's output distribution.
PPO Loss. In the setting of RL, the allocator of I-th layer is considered policy π parameterized by 9, We introduce a warm-start strategy to initialize 0,, which will be discussed later. The representation of at [-th layer is regarded the state st. The number of activated experts čt, determined through sampling, serves as the action taken under policy no Le., é (-1sr). The objective is maximize the expected return E 7' R(c,, st)] over the policy v, where 7 serves as the discounted factor, 7 E (0, 1], and L represents the number of layers in the model. We define the reward as follows:
agent with
R(a, 81) log P(
11-1 11=L
where only the allocator at the last layer receives the log likelihood as the reward. In other words, the expected return could be simplified [ log P(r, .., (1;)),. Through the reward defined in this way, maximizing the expected return is equivalent to minimizing caption loss in NLP. We employ the PPO algorithm (Stiennon et al., 2020 Nakano al. [021] optimize the policy within the trust region for stable training. The RL loss function is formulated as follows:
L (R) -E, [min (r, clip (r(0,),
+ ++) ()x)) Ar)
where e is a hyperparameter and clip function is introduced to constrain values within a specified range. The importance sampling ratio r(O,) is formulated as follows:
r (6)) ) ne, (c1l=)) Topte (&a (81)
(8)
4

---

Published as a conference paper at ICLR 2025
where 8i is the policy parameters before update. In standard PPO algorithm, it needs another value function to compute the advantage, which requires additional network and computations. Alterna- tively, we apply the form of advantage function in reinforce with baseline algorithm (Sutton & Barto 2018) formulated in (q. where the baseline employed to reduce variance theoretically:
AtCr, 81) Σ - (R(Cm= 8m) R( sm) = > - R(ct, st)
The superscript denotes the baseline, which define as default Top-K routing. There is no need of gradients for action 4, reward R(a, st), advantage A(4,, 8t). and old sampling probability Ta (Gr (s)).
only latest sampling probability ne, (61(st needs to calculate gradient training loss.
Activation Regularization Loss. The regularization loss reduces the activated expert counts by optimizing the expectation of the distribution produced by every allocator:
L n (n.
Lrea (6)
= Σ
Σ
(10)
i=1 n=1
The final loss a combination the PPO loss and regularization loss, where a hyper-parameter to control the reduction degree of the activated expert counts.
(()) L RL (0) + (
(11)
Warm Start. Given the large decision space encompassing all expert counts, allocators require an effective parameter initialization mitigate instability and inefficiency caused by arbitrary or incorrect choices. In this paper, we propose a warm-start approach "P-Warm to pre-train the allocators. This pre-training process utilizes the nucleus sampling (Holtzman et al. 2019) (i.e., Top-P) to generate pseudo-labels Specifically, we first choose the minimal subset of experts whose cumulative probability, as determined by the original router, surpasses the threshold p. As illustrated in the Eq. 12), for given token Tı, the number of experts within the subset is denoted by ny)
(p(p) argmán kE(1...N) j<=b
Pi
(12)
where P., represents the probability distribution arranged in descending order. For each baseline, we compute the average expert counts across different p values using moderate amount of token set T. The p value whose average counts is closest to the default activation value is then selected.
pd = argmain
Σ ne(p) kl]
(13)
Finally, for every token zy in the training dataset, we utilize mzy(x") ) as pseudo labels to facilitate the warm start training of the allocators.
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
Model Settings. We verify the effective- ness and universality of our proposed train- ing framework on four prevailing MoE based LLMs, i.e. Mixtral-8x22B (Jiang et al., 2024). Mixtral-8x7B (Jiang
Table I: Architecture details of four baseline models.
Config
Mixtral 8x22B
Mixtral 8x7B
DeepSeek MoE-16B
Qwen MoE-A2.7B
(004). DeepSeck-MoE-116 16B (Dai et al. 2024) and Owenl.5-MofE-A2.77 (Team). The archi- tectural details of the four baseline models are presented in [able Mixtral-8x7B and
Tup-K
2
2
Shared Experts
Routed Experts
8
8
64
60
Moł Layers
Activated Purams
39.0B
12.9B
2,8B
2.7B
Total Params
140.6B
46.7B
16.4B
14.3R
Mixtral-8x22B utilize a standard Top- K routing strategy for all experts. Additionally, DeepSeek-MoEi 16B and 1.ven1.5-Mo-A2..777 classify experts into shared and routed categories. Each token inherently activates all shared experts and selects the Top-K experts from the routed categories. Ada-K is applicable to any routing-bart based expert module. Besides, we keep the shared experts when present.
5

---

Published as a conference paper at ICLR 2025
Benchmark and Evaluation Details.
Following previous works TTuvron al. 2023b Le Scao 2023] et al. 2023 Black et al. 2022), we employ the Im-evaluation-hamee (Gao et al. 2021 evaluate our model. This tool serves as the backend for the HuggingFace Open LLM Leaderboard (Beeching et al., 2023). Our model is assessed on 6 key benchmarks aligned with Open
et al.
LLM Leaderboard. We firstly examine the model's accuracy across various benchmarks. Then, evaluate the computational cost with four metrics: average activated expert counts per token (Act), activation reduction rate (Rate), total floating point operations (FLOPs), and inference time speedup (Speedup). "Rate" is calculated as (1 - -1-Act/ee Act/k) x 100%. where k represents the default activation value. "Speedup" is the cumulative total across six benchmarks and "FLOPs" is measured by one single sample with a length of 256. Refer to Appendix for additional details.
Training Details. We adopt AdamW Loshchilov & Hutter 2017) as the opti- mizer. All baseline models are trained for one epoch using a consistent set of 10k sam- ples. The batch size and learning rate is set to 64 and le-3, respectively. We leverage 2 PPO epochs for reinforcement learning. For all four baseline models, we uniformly set A as 3c-3. This ratio prioritizes ensuring suf- ficient reduction rate, while guarantee a performance advantage over the original model. The training overhead for is relatively efficient. As shown in Table [] the trainable parameters for all baseline models are the scale of M,, which is negligible compared to the total parameters. Additionally, we present the training hours for the four baseline models. We employ NVIDIA A800 GPUs train Mixtral 8x22B, whereas each of the other three utilizes NVIDIA A800 GPUs. The training time for all baseline models is limited to 8 hours. Further details can be found in the Appendix.
Table 2: Training overhead of four baseline models.
Total Params
Trainable Params
Training Hours
Baseline Model
Mixtral-8x224
140.6B
2.75M
7.92h
Mixtral-8x7B
46.7B
1.05M
4.96h
DeepSeel-MoE--16B
16.4B
3,54M
1.79h
Qwenl.5-MofE-22.78
14.3B
2.95M
1.58h
4.2 PERFORMANCE EVALUATION
Table 3: Performance for four baseline models across six benchmarks. Results using default Top-K routing and our method are highlighted in brown and blue respectively. Each arrow and its associated numeric annotation indicate the performance disparity with the default Top- X baseline.
Method
Accuracy
Computation
ARC-C
Hella
MMLU
GSM
Truth
Wino
Average
Act
Rate t FLOPs
Speedup
Mixtral-&x22B
Top-K (k
66.22
85.43
73.15
70.96
61.54
82.77
73.35
15.30
1.00
50.0%
11.3TT
1.39x
Tog-K (k
72.70
89.08
77.77
82.03
79.15
0.0%
20.04T
1.00x
Ada.
73.57
89.76
78.22
82.98
69.97
85.02
79.92
10.77
1.31
34.4%
14.08T
131x
Mixtral-&:7B
Top-K (k
60.41
83.13
34.67
17.68
3.68T
1.35x
Top- 10-KKL (K
66. 72
86.48
70.39
58.38
41.25
82.24
67.58
2.00
0.0%
6,56T
1.00x
Ada.
68.49
87.23
70.40
58.61
42.85
81.45
68.19
1.40
30.0%
4.42T
1
Top-K (A 2)
$1.88
77.99
59.20
14.21
38.87
70.11
52.04 12:09
2.00
50.0 %
0.88T
1.25x
Tog: (k 3)
52.78
78.69
60.65
14.78
40.96
72.20
53.34 13.00
3.00
25.0
L.O0T
1.14x
Top-K (At
$4.69
79.45
61.30
15.62
42.50
73.00
54.43
4.00
0.0%
1.23T
1.00%
Ada-K
54.41
79.65
60.99
21.21
41.96
72.55
55.13
10.20
2.58
35.5%
0.92T
1.22x
DeepSeck-MoE-166
Tog K (k 3)
51.37
78.33
40.11
12.59
30.16
72.22
47.46 12:45
3.00
50.0%
0.96T
1.27x
Top-K (k 4)
52.73
79.44
43.04
14.86
30.34
72.88
48.87 11.04
4.00
33.3%
1.08T
1.18x
Top-K (A 5)
52.39
79.71
44.00
15.30
30.92
73.72
49.34
jousy
5.00
16.7%
1.20T
1.08
Top-K (A 6)
52.22
79.84
44.72
16.45
31.07
75.14
49.91
6.00
0.0%
1.36T
1.00x
53.44
80.24
44.98
16.86
31.83
75.92
50.55
10.64
40.0%
L.0OT
1.24x
For the four baselines, the relevant accuracy and computation metrics (detailed in Sec 4.1 are reported in Table Additionally, we present the metrics of the baseline models under lower Top-K activation level (e.g., = for Mixtral-8x7B) as supplementary comparison. The implementation of Ada K routing leads to significant and consistent improvement in accuracy across all baselines compared to the default settings. This enhancement is particularly noteworthy, as models employing Ada-K routing not only achieve these accuracy gains but also reduce FLOPs by over 25% and accelerate inference by more than 20%, demonstrating a compelling balance between performance and efficiency. We believe these results arises from more effective expert resource allocation.

---

Published as a conference paper at ICLR 2025
4.3 ABLATION STUDY
In this section, we perform a comprehensive ablation analysis of the proposed training framework. It should be noted that the conclusions are consistent across four baseline models. However, due to space limitations, we uniformly present the numerical results based on Qwen Swenl..-Moff 5-M..--A..BB
Trade-off between performance and activation re- duction rate. We first investigate the trade-off be- tween the activation reduction rate and model per- formance. As detailed in Sec 4.1 the results in Table 3 are based on a fixed value of À (i.e., 3e-3)
55.5
55.0
K=4
Accuracy
in Equation I1 Actually, Ada-K could facilitates more flexible balance between accuracy and compu- tational efficiency through it. By varying X, we gen- erate 15 data points to construct the trade-off curve illustrated in Figure 2 The trade-off point in the figure correspond to Table 3 Analysis reveals that, prior to the trade-off point, accuracy declines grad- ually with increasing activation reduction. However, beyond this point, the rate of decline accelerates sharply. Importantly, our method consistently sur- passes the default Top-K K model until the activation reduction rate reaches approximately 44%. Compared traditional Top- K routing, Ada-K provides both adjustable flexibility and a more effective balance between performance and efficiency.
54.0
$3.0
52.5
Accuracy Activarion Reduction
Tradk off Point
20
30
50
Activation Reduction Rate
Figure 2: Trade-off curve between perfor- mance and activation reduction rate.
Effect of dynamic routing. To the best of our knowledge, Ada-K is the first learnable dynamic expert allocation strategy. In this section, we aim explore and validate the advantages of this pure dynamic paradigm. Actually, as shown in Table 3 at comparable activation reduction rates, base- line models with lower Top- Pop-K activation levels ex- hibit a significant performance gap when compared to Ada-K routing. To adapt these baseline mod- els, which have reduced numbers active experts, to their new activation patterns, we fine-tune their routers using the same 10K dataset as Ada-K. This process effectively imparts pseudo-dynamic qual- ity to the static Top-K K routing. Additionally, some prior works (e.g.. MoE-D (Huang et al. (0224) and D2DMoE (Piórczyński et al.) 2023) achieve quasi- dynamic routing through fixed thresholds. They determine the activation of each expert by compar- ing the routers' output against these thresholds. We
Table 4: Ablation study about the dynamic routing. The results of default Top-K rout- ing and our method are highlighted in brown and blue respectively.
Route
Tuned
Acc
Rate
52.54 11.89
50.0%
Top-K (k = 2)
/
52.85
11.58
50.0%
Top-K K (K = 3)
53.34
L1.00
25.0%
53.44
10.99
25.0%
x
54.43
0.0%
Top-K (A 4)
53.89 1o.54
0.0%
MoED (p 0.3) MoED (p 0.4)
53.45
10.98
32.4%
53,60 1o.sa
28.6%
D2D (T 0.1)
53.73 1o.70
27.8%
D2D (T 0.2)
53.64 1o.79
31.5%
Ada-K
55.13 no.70
35.5%
also include these methods in our comparison by reproducing them using their official code. The results Tagges that improving the adaptability of static routing leads to modest performance gains relative to the original baselines. Additionally, hhreshold-basee routing demonstrates slight advantage over static routing. However, these methods still fall significantly short compared to Ada- K, highlighting the critical need for dynamic and efficient expert allocation strategy.
Effect of activation regularization. In this sec- tion. we explore different schemes for redueing ex- pert activations. The related results are reported in Table 5 Since the expectation of each allocator's s output distribution is differentiable, it allows for direct optimization via backpropagation (row "As Loss"). So we choose it the default method. Al- ternatively, combining the language modeling likeli- hood and the activate expert count of each token to form a reward presents another viable strategy (row "As Reward"). It introduces reward-level trade-off
Table 5: Ablation study about the activation regularization methods. The results of default Top routing and our method are highlighted in brown and blue respectively.
Method
Acc
Act
Rate
Original
54.43
4.00
0.0%
As Reward
54.64 10.21
2.56
36.0%
As Loss
55.13 10.70
2.58
35.5%
7

---

Published as a conference paper at ICLR 2025
where all training objectives are uniformly optimized through the PPO loss. For fair comparison, employ the same training data and ensure consistent activation reduction rates. The results indicate that direct optimization of expectations yields marginally better performance than treating them as rewards. However, the difference is not significant. It highlights the effectiveness and robustness of Ada-K routing training.
Effeet of training data. In this section, exam- ine the impacts of varying training data, i.e. pretrain and supervised fine-tuning (SFT) data. For fair- ness, each data type comprises 10k samples sourced from prominent open-source corpus. We report the The results indicate that
Table 6: Ablation study about the training data type. The results of default Top-K routing and our method are highlighted in brown and blue respectively.
training results in Table 6 the training not sensitive to the data domain. achieves comparable performance across two data settings, demonstrating both the effectiveness and robustness of Ada-K routing training. We detail the collection and organization processes for two types of data in the Appendix.
Data Type
Acc
Act
Rate
Original
54.43
4.00
0.0%
Pretrain SFT
55.78 11.35
2.68
33.0%
55.13 10.70
2.58
35.5%
Effect of warm up strategy. In this section, we conduct an ablation study on different warm start strategies. The results are reported in Table [ If no warm-up strategy is employed, the allocators are initialized randomly (the second row). The K-Warm strategy involves pretraining the allocators to con- sistently output k. For P- Warm, detailed in Sec 3.3 pseudo-labels are generated using a specific p value. The findings indicate that both the K-Warm and P-Warm strategies obviously surpass the perfor- mance of the default baseline, whereas the random strategy shows only comparable performance. It is largely due the pre-training mitigates the sam- pling arbitrariness caused by random initialization. facilitating the learning of more optimal strategies. Additionally, the more flexible warmup strategy P- Warm yields marginally improved performances compared to the static K- K-Warm.
Table Ablation study about the warm up strategy. The results of default Top-K routing and our method are highlighted in brown and blue
Strategy Original
Acc
Act
Rate
54.43
4.00
0.0%
x
54.18 10.25
2.88
28.0%
K- Warm P-Warm
54.97
10.54
2.60
35.0%
55.13
70.70
2.58
35.5%
4.4 VISUALIZATION AND ANALYSIS
In this section, we provide a detailed analysis and visualization of the policies learned by the agents (i.e., allocators) after training with Ada-K routing. Due to space limitations, we uniformly present the numerical results based on Owenl.5-MoE-A22.75
Expert resource allocation is more adaptive.
We initially analyze the probability distribution r(k)
of activated expert counts per token: P(k) Σ r(n]' where r(k) represents the number of
tokens activating k experts. The results for each benchmark are illustrated in Figure 3 using logarithmic scale for enhanced clarity. The analysis reveals: (1) The decision space available to trained allocators ranges from I to the total number of experts, ensuring adaptive resource allocation. (2) The training effectively reduces the activation levels of the majority of tokens to approximately or 3. Concurrently, around 10% of essential tokens are identified and subjected to enhanced processing by engaging more than the default number of experts. (3) The expert activation distribution differs across benchmarks, demonstrating that the trained allocators are capable of adapting to diverse domains and devise customized solutions accordingly.
Middle layers tend activate more experts. As illustrated in Figure 4 variations in layer depth consistently impact average expert activations across multiple benchmarks. Specifically, both the shallow and deep layers utilize fewer experts, whereas the intermediate layers employ a higher number of experts. We hypothesize that this can be attributed to the varying complexities at different stages of processing. Shallow layers mainly engage in basic feature extraction, e.g. recognizing simple syntactic patterns and semantic elements, which might generally necessitates minimal specialized knowledge and thus reduces the requirement for experts. Conversely, middle layers might play a pivotal role in more intricate tasks, including the integration of basic features into sophisticated
8

---

Published as a conference paper at ICLR 2025
ARC-C. X pm 0.57%
TruthgA
6.11%
Fiebas, 1-1.1.%%
Winoo, 2 9.3.7%
K-4
COMOK
10-1
Activatee Expent
10
10-1
10
13
37 41 Activated Exports
25
29
3
37
10
21
27
Layer mndex
Figure 3: Probability distribution curves of ex- pert activations per token across six benchmarks.
Figure 4: Curves of expert activations per layer across six benchmarks.
representations, ambiguity resolution, and the comprehension of contextual nuances. These tasks could likely benefit from diverse array of specialized inputs, possibly necessitating larger pool of experts to enhance the processing robustness. In contrast, deep layers might primarily concentrate on refining these integrated features and finalizing the output, mainly involving the application and optimizing of processed information, which can typically be achieved with fewer experts.
Content words tend to activate
experts. To evaluate the impact of token attributes, statistically analyze both part-of-speech (POS) and expert activations for total of million tokens across six benchmarks. To reduce variability due to randomness, that occur more than 1,000 times within these corpora. POS tagging is performed using the NLTK toolkit (Bird 2006). We calculate average activations for prevalent POS categories, namely nouns, verbs, conjunctions, that enhanced expert resource tends concentrate on verbs and nouns, which are central to syntactic construction and convey clear semantic meanings. In contrast, the expert activations elements with weaker semantic content, e.g. special symbols and conjunctions, are relatively less. This conclusion largely substantiates the intuitive basis for research motivation. The strategies learned by the agents (i.e., allocators) tend to allocate more expert resources to tokens rich in semantic information, allowing for thorough modeling. Conversely, tokens with weaker semantics require only minimal expert resources for effective representation. This more rational resource allocation strategy benefits both semantic understanding and computational efficiency.
analysis only includes tokens
sepeetive//aveerbs... and punctuation. The findings presented in Figure 3 suggest
Load balance is maintained. In this section, we investigate whether the previously established expert load balance of the default Top-K model is maintained after fine-tuning with our Ada-K routing. Intuitively, since the router that determines expert allocation remains frozen throughout the training process, framework is anticipated to minimally disturb the original load balance. We assess the activation probability of each expert across six benchmarks, as illustrated in Figure The results demonstrate that the load each expert remains nearly unchanged before and after traming, preserving a nearly uniform distribution.
0.0200
0.0175
Achowted (xpeette
0.0150
0.0125
0.0100
Con Speech
Punt
0,0000
19
25
25
32
43
65
Expoort Ind
Figure 5: Average expert activation number per token across different parts of speech.
Figure 6: Distribution curves of expert workloads before and after Ada-K fine-tuning. The gray horizontal line represents the uniform distribution of expert activations.

---

Published as a conference paper at ICLR 2025
Table Average activated experts and accuracy delta across tasks of varying difficulty levels. AAce represents the absolute performance im- provement compared to default Top-K K routing.
Table 9: The relationship between the ratio of layers equipped with the allocator to all layers and the corresponding metries. The results with default setting are highlighted in blue.
Benchmarks
Act
AAcc
Training Params
Training Time
Ratio
Acc
FLOPs
ARC-E
2.23
-0.40
ARC-C
2.84
+0.12
0.125
0.37M
1.54h
54.98
1.19T
Collection
2.58
+0.70
0.25
0.74M
1,55h
55.06
1.14T
BBH
3.43
+5.54
0.5
1.48M
1.56h
55.15
1.04T
1.00
2.95M
1.58h
55.13
0.92T
Hard tasks tend to activate more experts. As shown in Table 8] difficulty on activated expert counts across both intra-benchmark and inter-benchmark dimensions. For both dimensions, we report the average number of activated experts and the performance gains (relative to the default Top-K K setting) of our method. For intra-benchmark dimension, we leverage ARC, a multiple-choice question question-answering answering dataset consisting of science exam questions for grades to 9, we analyze two levels of task difficulty, namely Easy (E) and Challenge (C). For inter-benchhee benchmarrk dimension, we compare the commonly used six benchmarks (detailed Sec[4.Tp with BBH (BIG- Bench Hard) (Suzgun et al. 2022). BBH comprises 23 demanding tasks that require sophisticated cognitive skills such as multi-hop reasoning, causal inference, and logical deduction, markedly exceeding the difficulty of the collection of six benchmarks. The results indicate that model with Ada- X routing activates more experts on harder tasks (i.e., 2.23 vs. 2.84 for mntha-benchmark and 2.58 3.43 for inter-benchmark)
we examine the effect of task
Furthermore, we discover that the model with Ada-K excels at handling challenging tasks. Specif- ically, for the intra-benchmark dimension, our method demonstrates a moderate performance loss compared to the baseline with Top K routing in ARC-E. However, when the task difficulty is in- creased to ARC-C, the model with Ada-K exhibits a performance advantage. Similar conclusions even more evident in the inter- benchmaenc dimension. This indicates that, when faced with highly complex tasks, the flexibility of Ada-K K in concentrating expert resources on key tokens enables more effective modeling of the tasks, leading to improved adaptability.
Equipped more layers with allocator yields better performance. In this section, we investigate the impact of varying allocator deployment ratios. By default, we integrate an allocator at each layer of the baseline models. The relevant results are presented in Table 9 All selected layers are sampled at equal intervals across the model. Our findings indicate that, due to the lightweight nature of the allocators, even when all layers are equipped with them, the number trainable parameters remains minimal, resulting in only a negligible increase in time overhead. Moreover, while this effect does not significantly impact accuracy in benchmark evaluations, equipping more layers with an allocator clearly reduces computational overhead. Consequently, deploying allocators on per-layer basis emerges as the optimal choice, as it enables a more refined expert allocation strategy at each layer, facilitating the scheduling of expert computations throughout the model.
5 CONCLUSION
In this paper, we present a novel Ada -K routing strategy for MoE-based LLMs, which dynamically adjusts the number of activated experts based on token importance. Ada-K routing enhances the balance between computational efficiency and model performance through a lightweight allocator module optimized via PPO algorithm. Extensive evaluations demonstrate that Ada-K routing significantly reduces expert activation by 30%-40% while improving benchmark performance across various MoE-based LLMs. This reduction in expert activation directly translates to computational gains, achieving more than a 25 % reduction in FLOPs and 20% speedup in inference time, showcasing practical efficiency gains for large-scale applications. Moreover, this method is highly efficient, requiring minimal additional training, and can be easily integrated into existing MoE-basee LLMs with low training overhead. Our analysis also provides valuable insights into adaptive resource allocation in large MoE. models.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGMENT
This research is supported by Artificial Intelligence National Science and Technology Major Project (2033DD0121000), National Natural Science Foundation of China (No. 62437001, 62436001), Strategic Priority Research Program of the Chinese Academy of Sciences (XDA0460305), Key Research and Development Program of Jiangsu Province under Grant BE202301633, and CCF- Tencent Rhino-Binn -ird Open Research Fund.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 2022.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https
11m leaderboard. 2023.
Steven Bird. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pp. 69-72, 2006.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:187771901, 2020b.
2020b.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fei- Yue Wang. Rime: Robust preference-basee based reinforcement learning with noisy preferences. arXiv preprint 172252240..12557, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuma. Imsys. org (accessed 14 April 2023), 2023.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann. Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference on machine learning, pp. 4057-4086. PMLR, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0544 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reñichrr Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.14668, 2021.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.066 0006066666,6 2024.
Erik Daxberger, Floris Weers, Bowen Zhang, Tom Gunter, Ruoming Pang, Marein Eichner, Michael Emmersberger, Yinfei Yang, Alexander Toshev, and Xianzhi Du. Mobile v-moes: Scaling down vision transformers via sparse mixture-of-experts. arXiv preprint 2rXiv:2309.04433 2023.
11

---

Published as a conference paper at ICLR 2025
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten P. Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster. Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellsterr Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 2547-5569. PMLR, 2022a. URL URL
Toju Duke, Lucas Dixon, Kun Zhang, Quoe V. Le, Yonghui Wu,
mlr wttppsssvll//uu222ee222 html
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pp. 5547-5569. PMLR, 2022b.
David Eigen, Mare' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint arXiv: 2013.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, J. Mach. Learn. Res., 23: 13:120:1-120:3: 2022a. URL
http::/ fmwr../mmr... www.ar.orgpppppppppee vvt
html
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, Journal of Machine Learning Research, 23(120):1-39, 2022b.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold- ing, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https: /oi .
10. wwtgg/../0////eeee..ee
5371628
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 34:29335-29344 29347, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint 0tti:22009.033000 2020.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text 2019.
degeneration. arXiv preprint
Quzhe Huang, Zhenwei An, Nan Zhuang, Mingxu Tao, Chen Zhang, Yang Jin, Kun Xu, Liwei Chen, Songfang Huang, and Yansong Feng. Harder tasks need more experts: Dynamic routing in moe models. arXiv preprint arXiv:2403.07652. 2024.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-88 87, 1991.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint wrXiv:240J.04088 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088: (08611, 2020.
12

---

Published as a conference paper at ICLR 2025
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 6265-6274. PMLR, 202 URL http::t::pppccceeeiiiiie /tt.://eeeennnnp.rrr.rrs//////// ewwsslll htmll
m.r wttppp///3///
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference Machine Learning, pp. 2265-6274 PMLR, 2021b.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint (rXXi:2300506666, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.0799 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2021.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Mikolaj Piórczyński, Filip Szatkowski, Klaudia Balazy, and Bartosz Wójcik. Exploiting transformer activation sparsity with dynamic inference. arXiv preprint 20.iv:2310.04300 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(5)5588--551, 2020.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595 2021.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866, 2021.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555-17566. 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106.
2021.
Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676 2020.
13

---

Published as a conference paper at ICLR 2025
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated cexture-of-expene layer. arXiv preprint wtX.v:1771.0655.0 2017a.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017b. URL https:77 openrevi ew. www.:ee/tffoeeo fooooordduml
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.0926:1 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-followwe model. Stanford Center for Research Foundation Models. https://erfm. stanford. edu/2023/03/11//lpeer htmll 36))7, 2023.
Qwen Team. Qwenl.5-moc: moc: Matching 7b model performance with 1/3 activated parameters. https github io/blog/qwen mooe/ Accessed: 2024-03-28.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint ar ww.XXx:2304.059777 2023.
Tongtian Yue, Jie Cheng, Longteng Guo, Xingyuan Dai, Zijia Zhao, Xingjian He, Gang Xiong, Yisheng Lv, and Jing Liu. Sc-tune: Unleashing self-consistent referential comprehension in large vision language models. arXiv preprint arXiv:2403. 13263, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 190500990 2019.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 15:77003-7144 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models, arXiv preprint arXiv:2202.08906, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo 7hao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. arXiv preprint 10.0222 2021.
14

---

Published as a conference paper at ICLR 2025
A EVALUATION BENCHMARKS
2024). we assess models across six standard benchmarks utilizing the Eleuther AT Language Model Evaluation Harness, a comprehensive framework designed to evaluate generative language models on a diverse array of tasks, These tasks include A12 Reasoning Challenge (ARC-C) (Clark et al. 2018), HellaSwag (Hella) (Zellers et al. 2019). MMLU (eendrycks et al. 2020), TruthfulQA (Truth) (Lin et al. (021)) Winogrande ( (Wino) (Sakaguchi et al.) al.j )021) and GSM8K (GSM) (Cobbe et al. 2021). In these evaluations, higher scores indicate better performance. We selected these benchmarks because they assess a range of reasoning abilities and general knowledge across multiple disciplines in both zero-shot and few-shot scenarios.
Following previous works (Team: Jiang et al. 2024 Dai et al.
Table 10: Details of benchmarks. We follow the setting of HuggingFace Open LLM Leaderboard.
Benchmark
Ashots
# Samples
Details
ARC-C Clark et
2018
25
2,59k
set grado-school science questions.
HellaS Zellers
2019
10
70%
A test of commonsense inference,
which is casy for humsas but challenging SOTA models.
test to mensure text model's is multirakk accuracy.
MMLU
Hendrycks
2020
14.99 9K
The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
Diverse grade school math word problems to measure a model's
GSM8K Cabbe al
2021
8.5k
ability solve multi-step muthematical reasoning problems.
TrutQQA Lin cl.
2021
0.8k
test to measure model's propensity to reprodacee falschoods commonly found online.
WinoG (Sakaguchi
2021
44k
An adversarial and difficult Winograd benchmark at scale, for commonsense reasoning.
B IMPLEMENTATION DETAILS
In this section, we detail the training protocols of the proposed framework. The specific hyper- parameter configurations for training are reported in Table II These protocols are applicable to all the baseline models, utilizing& 8 A800-80G GPUs. Throughout all training phases, we consistently conduct single epoch to prevent overfitting. The batch size per GPU is set at 8. Gradient checkpointing is activated for both training phases to enhance memory efficiency.
Table 11: Additional training details.
Fine-tuning
Configuration
Warm-Start
PPO
Optimizer
AdamW
AdamW
Base LR
1e-3
le-3
Precision
bi16
bf16
Weight Decay
0.1
0.1
Batch Size
64
64
LR Decay Schedule
cosine
constant
Gradient Checkpoint
True
True
Training Epochs
1
Max Length
2048
2048
Threshold p
0.3
Regularization Coef
3e-3
PPO Epoch
2
C CHOICE OF X
In this section, present the trade-off curves obtained by scanning À for the remaining three baseline models. The patterns observed are similar: as the compression rate increases, the accuracy initially decreases gradually, but the rate of decline accelerates thereafter. We uniformly select À =3 -3 x 10-3 as the optimal balance point across all models.
D TRAINING CURVES
In this section, we present the advantage and loss curves of four baseline models during training, which are calculated according (q.(9) and Eq. [1), respectively. Generally, a consistent trend
15

---

Published as a conference paper at ICLR 2025
6655
5A5
550
K-1h
16.0
Tecincy
Accuracy
03.5
67.5
15:6
665
Acvittor Indkoffnert
Actival on Reduction Bare
Activation Reduction Part
Activation Reduction Rate
(a) DeepSeck-MoE-166
(b) Mixtral-8x7B
(c) Mixtral-8x x2B
Figure Trade-off curve between perfor- mance and activation reduction rate for other three baseline models.
across all four MoE models is observed: a gradual decrease in loss and an increase in advantage over the course of training. This suggests that the models effectively explored and adopted optimal strategies, leading to higher rewards.
4.5
43-
435
10.0
4310
15
3
3.0
4.00
30-
232
25
0.30
2.-
15
08
Qwenll.-MoeEAA2.77
DeepSeek-MoE-166
2.4-
935
15
0225
23
12
30
1020
020
25
0335
0350
148
1.4
4.10
15
1.2
0075
ba3dh
b00cc
(c) Mixtral-8x78
(d) Mixtral-88 22B
Figure 8: Loss and advantage curves for four baseline models
E TRAINING DATA
Table 12: Datasets used for training. We collect data from various sources to empower the model with broad spectrum of linguistic capabilities. We ensure that all datasets are publicly available in the community.
Usage
Source
#Sample
Alpaca GPT4, UltraChat 200k, LIMA, OpenPlatypus
Supervised Fine-tuning
10.2k
CodeAlpaca 20k, Wiki QA, MathInstruct
Pretrain
Wiki Demo, RedPajama V2, Wikipedia, StarCoder
10. lk
16

---

Published as a conference paper at ICLR 2025
In this section, introduce the data utilized in training. For default setting, we employ dataset comprising 10k randomly sampled from mainstream public supervised fine-tuning datasets. Additionally, the sources of the pretrain corpora used in the ablation experiments are also reported in Table [2] All these data are widely employed in the training of prevalent LLMss (Touvron et al.
2023a Taori al. 2023 Chiang et al. 2023)
17


Reviews:
[{"review_id": "C0URVCbhRC", "reviewer": "Reviewer_vr7C", "summary": "The paper studies the dynamic routing strategy in MoE architectures. Conventional MoE architectures use a static Top-K routing, activating a fixed number of experts regardless of the token's complexity or importance. They propose the Ada-K routing strategy. Ada-K routing dynamically adjusts the number of activated experts based on the significance of each token, balancing efficiency and performance. The allocator module is based on RL. Ada-K achieves over 25% reduction in FLOPs and 20% faster inference speeds, with performance improvements across various benchmarks.", "questions": "1. Is the usage of DRL necessary? How is the performance of the proposed method compared to naive Top-k selection distillation?\n\n2. What is the state space of the DRL agent?\n\n3. As each layer $l$ is paired with a DRL agent, are all DRL agents trained together or in a layer-by-layer manner? Will the selection for a layer $l$ influence the selection of layer $k$, where $k\\neq l$? If so, how is the influence taken into account in the desigm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "MoE is a scalable solution that balances parameter increase with computational cost. Targeting the limitations of prior efforts with a fixed number of experts, this paper works to make the expert number dynamic, which can bring additional efficiency and potential performance gains. The experiments include studies across multiple scales of models to test the effectiveness of the proposed method. An advantage of Ada-K is that it is pluggable, making it applicable across different MoE-based LLMs.", "weaknesses": "1. The Ada-k routing design works for the post-training of MoE-based LLMs. For post-training, as the gate is already trained, a question is whether it is necessary to use RL to learn the gate selection again, which complicates the overall design. For instance, a simple solution is to distill Top-k selection into binary selection with a separate gate, similar to the MoD design. The paper doesn't include comparison studies with this naive Top-k selection distillation, making it hard to say whether RL is necessary. I also didn't see a concrete motivation for using DRL.\n\n2. Some parts of the paper writing need to be improved. For instance, for each layer $l$ equipped with the allocator, the state design and action design for the DRL agent are unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PvGpnzMKj3", "reviewer": "Reviewer_3rXb", "summary": "This paper introduces the Ada-K Routing method, which incorporates an additional RL agent into pre-trained MoE models to dynamically control the top-k in MoE routing. This approach enables dynamic adjustment of top-k at a lower cost, thereby enhancing the inference efficiency of MoE models. The effectiveness of the proposed method has been validated on multiple open-source models, demonstrating improved accuracy compared to existing threshold-based methods when achieving similar acceleration effects. Additionally, the paper presents detailed ablation studies and analytical experiments that provide valuable insights into the research on MoE models.", "questions": "- The reported speedup in the paper is based on the actual inference time during evaluation tasks, which likely applies to scenarios with batch sizes of 1 or smaller. However, MoE models have large parameter volumes, and their efficiency typically becomes evident with larger batch sizes during actual deployment. The reviewer is concerned that the RL-based method of controlling top-k for each token might lead to imbalance in batch inference, potentially affecting the acceleration effect. Therefore, the reviewer would like to know the relationship between acceleration effect and batch size.\n- Would combining Ada-K with the standard SFT setting yield better results? Or, under the control of Ada-K, would the performance of SFT be affected?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Innovatively proposes to introduce an RL agent for controlling top-k in pre-trained MoE models, optimizing the allocation of computational resources and improving inference efficiency. The method has a low training cost and shows robustness to training data, outperforming existing methods across various downstream tasks.\n- Thorough ablation studies demonstrate the relationship between acceleration effects and accuracy, providing practical guidance. It also validates the effectiveness of activation regularization and warm-up strategies.\n- Performs meticulous analysis revealing that intermediate layers of trained models require the activation of more experts, reaffirming that more challenging tasks necessitate the activation of more experts.", "weaknesses": "- Some comparisons with baseline methods are not entirely reasonable. Existing work suggests that freezing the router yields better results when tuning MoE models. Ada-K freezes the router and introduces additional agent parameters for training, while other comparison methods only train the router. The reviewer recommends supplementing the results by (1) conducting full fine-tuning of the model with a frozen router and comparing the effects of introducing only threshold methods versus Ada-K; or (2) maintaining the current Ada-K setup and adding LoRA parameters to the threshold-based method while freezing the router for tuning.\n- The paper \"AdaMOE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models\" (ArXiv: 2024-06-19), which has a very similar title, also achieves dynamic adjustment of top-k during post-training and improves accuracy while saving FLOPs during the tuning phase. Given the short time interval, the lack of comparison with this baseline can be understood, but the reviewer suggests discussing it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sC8GOubCq8", "reviewer": "Reviewer_gGQ7", "summary": "This work proposes to introduce an adaptive computation budget for MoE LLMs. Specifically, the proposed method fine-tunes a pretrained MoE LLM to activate the adaptive number of experts with PPO training and a trainable allocator layer. The proposed model achieves comparable performance with baselines but uses less computation.", "questions": "1) What is the setting of LLM inference speedup? Batch inference or batch size == 1? How does the inference speed trend if we ablate the batch size? And are you using expert parallelism during training and inference?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) Well-motivated. It is well-known that MoE LLMs are very effective and promising, but the efficiency of MoE LLM deployment is limited due to the huge amount of trainable parameters. It is good to improve the efficiency of LLMs.\n2) Clear writing and comprehensive ablation studies.", "weaknesses": "1) An important baseline is missing -> Mixture of Depth (https://arxiv.org/abs/2404.02258). Due to the layer skip in this paper, the computation cost for each token is adaptive as well,\n2) Due to the imbalanced computation cost in different layers, the pipeline parallelism is more difficult and challenging to use, during both training and inference.\n3) There are many other ways to introduce adaptive computation budget, e.g. ACT algorithm in universal transformer (https://arxiv.org/abs/1807.03819), PonderNet (https://arxiv.org/abs/2107.05407). Need to discuss and compare. Why do you select PPO to train the allocator? More justification is required. It seems that the model is unnecessarily complex. Why not just ACT or so? Any algorithm or training difficulty? Introducing the PPO in such an early stage will make the LLM training pipeline much more complicated, which may make this approach not that useful, even if it is effective to some extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xtD34alO3u", "reviewer": "Reviewer_wxUn", "summary": "This paper introduces a new Ada-K routing strategy for MoE-based large language models (LLMs), which dynamically adjusts the number of activated experts based on token importance. Due to the non-differentiable nature of the decision, the paper leverages the Proximal Policy Optimization (PPO) algorithm to facilitate the end-to-end learning process. The proposed method improves model performance while reducing computational costs. Extensive evaluations on various benchmarks, along with comprehensive ablation studies, demonstrate the effectiveness of Ada-K routing compared to traditional Top-K routing strategies.", "questions": "In addition to the issues mentioned in the Weaknesses section, there are a few other concerns:\n\n1. What does $i$ represent in Equation (8)? Perhaps it should be $n$?\n2. The authors should provide more details on how the policy $\\pi$ was designed in this work and the rationale behind this design choice? Additionally, how was the number of training parameters calculated?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper investigated a timely problem.\n2. The routing strategy proposed in this paper is a new approach that dynamically adjusts the number of activated experts.\n3. The paper presents a comprehensive set of experiments and analyses.", "weaknesses": "1. The structure of the Method section can be improved. Section 3.1 describes the traditional routing method of MoE, which would be more appropriately placed in the Related Work section. The Method section needs more elaboration. At present, it is rather brief, and expanding on key details would greatly improve the clarity and comprehensibility of the research.\n2. This method has certain limitations, particularly in its application to models that use top-1 routing, such as Switch Transformer, making optimization more challenging.\n3. Regarding the experiments in Section 4.2, a direct comparison between the warm-up Ada-k routing and the baseline top-k routing with different k values may be somewhat unfair. The models are likely to have different performance levels even before training due to the differences in routing methods. Providing a loss curve during training could better demonstrate the effectiveness of the proposed method.\n4. Figure 8 shows that the use of the Ada-k strategy leads to performance degradation on simpler tasks while achieving better results on more complex tasks. This seems counterintuitive based on prior experience. Perhaps the authors could provide a more plausible explanation for this phenomenon.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S5caDQcn1c", "reviewer": "Reviewer_6jjx", "summary": "This paper presents Ada-K routing, an additional adapter on mixture-of-expert model that can decide the number of experts activated for each token. The dynamic number of experts helps reducing the inference cost of the model, while still keeping a descent performance. The authors used proximal policy optimization algorithm to make the ada-K adapter trainable, and evaluated it on four MoE models at different scales. The authors additionally run a sweep of experiments to study the effect of Ada-K at different scale of MoE, the trade-off between accuracy and efficiency, and the ablation study of different hyper-parameters. The authors further introduced visualization of the expert allocation pattern, which helps understand the MoE architecture's dynamism.", "questions": "- I noticed that the authors claim that all conclusions in the ablation study is generally applied to all the four models. Providing the trade-off point of all other three models (or are they also 3e-3?) could answer my doubt on the second point of the weakness section.\n\nThe paper is well written in general. The question below will not influence my scoring.\n- In section 3.3, paragraph PPO Loss, \"Eq.(4) could be simplified as ...\". I'd suppose it is \"The expected return could be simplified as ...\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The idea of an adapter to control the number of experts at the granularity of token and layer can be universally migrated to most of the MoE models, and is simple to implement.\n- The paper presents detailed optimizations to make the reinforcement learning of adapter more stable.\n- The experiment is abundant and at a large scale, including:\n  - models at different scale of both parameters and activated parameters, as well as different baseline number of activated experts. \n  - the pareto frontier between performance and the cost reduction\n  - comparing Ada-K with existing heuristic-based dynamic expert allocation algorithms\n  - ablation study on dataset, regularization, and warmup strategy\n- The visualization of the result is novel and inspiring.", "weaknesses": "- Although Ada-K significantly reduces the total FLOPs at inference, an efficient implementation could be nontrivial, which limits the use of Ada-K technique in practice. Providing experiment data on the average latency/throughput could help make the inference cost improvement more significant.\n- Figure 2 suggests that the hyperparameter $\\lambda$ is crucial to the balance between performance maintenance and the inference cost reduction. However, the corresponding paragraph (\"Trade-off between performance and activation reduction rate\") lacks enough information on how to find the $\\lambda$. What is the range of $\\lambda$ in the figure? Is the best $\\lambda$ generally applied to all the four models, or should a user of Ada-K also finetune on a sweep of different $\\lambda$'s? Providing a plan for the user to find a good $\\lambda$ could help improve Ada-K's usability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)