Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion
Authors: Filip Szatkowski
Bartosz Wójcik
Mikołaj Piórczyński
Simone Scardapane
Venue: NeurIPS 2024 poster (2024)
Keywords: inference efficiency
activation sparsity
dynamic-k gating
mixture-of-experts
conditional computation
dynamic neural networks
Primary area: other
TL;DR: We speed up inference by novel mixture-of-experts conversion method

Abstract: Transformer models can face practical limitations due to their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be leveraged to reduce the inference cost by converting parts of the network into equivalent Mixture-of-Experts (MoE) layers. Despite the crucial role played by activation sparsity, its impact on this process remains unexplored. We demonstrate that the efficiency of the conversion can be significantly enhanced by a proper regularization of the activation sparsity of the base model. Moreover, motivated by the high variance of the number of activated neurons for different inputs, we introduce a more effective dynamic-$k$ expert selection rule that adjusts the number of executed experts on a per-token basis. To achieve further savings, we extend this approach to multi-head attention projections. Finally, we develop an efficient implementation that translates these computational savings into actual wall-clock speedup. The proposed method, Dense to Dynamic-$k$ Mixture-of-Experts (D2DMoE), outperforms existing approaches on common NLP and vision tasks, reducing inference cost by up to 60\% without significantly impacting performance.

Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Expertt Conversion
Filip Szatkowski"
Bartosz Wójeik* IDEAS NCBR
IDEAS NCBR
Warsaw University of Technology
Jagiellonian University t
Mikolaj Piórczyúski*
Simone Scardapane Sapienza University of Rome
Warsaw University of Technology
Abstract
Transformer models can face practical limitations due their high computational requirements. At the same time, such models exhibit significant activation sparsity, which can be leveraged reduce the inference cost by converting parts of the network into equivalent Mixture-of-Expere played by activation sparsity, its impact on this process remains unexplored. We demonstrate that the efficiency of the conversion can be significantly enhanced by a proper regularization of the activation sparsity of the base model. Moreover, motivated by the high variance of the number of activated neurons for different inputs, we introduce a more effective dynamic-k expert selection rule that adjusts per-token basis. To achieve further savings,
(MoE) layers. Despite the crucial role
the number of executed experts we extend this approach to multi-head attention projections. Finally, we develop an efficient implementation that translates these computational savings into actual wall- clock speedup. The proposed method, Dense to Dynamic-k Mixture-of-Experts (D2DMoE), outperforms existing approaches common NLP and vision tasks, reducing inference cost by up to 60% without significantly impacting performance.
Introduction
Transformers have become a predominant model architecture in various domains of deep learning such as machine translation [47], language modeling 66] 31), and computer vision 7 21]. The widespread effectiveness of Transformer models in various applications is closely related to their ability scale efficiently with the number of model parameters [20), prompting researchers train progressively larger and larger models 45 19]. However, the considerable computational demands of these models often restrict their deployment practical settings with limited resources.
At the same time, Transformer models exhibit considerable activation sparsity in their intermediate representations 24). which suggests that most of their computations redundant. Conditional computation methods can reduce these unnecessary costs by using only a subset of the model parameters for any given input [14]. In particular, Mixture-of-Experts (MoE) layers [38], consisting of multiple experts that are sparsely executed for any input token, are an effective way to decouple
the number of parameters of the model from its computational cost 3 As shown by [52], many
pre-trained dense Transformer models can be made more efficient by converting their FFN blocks into MoE layers, process they call MoEfication.
"quual contribution
1 Coorespnnding author: b. wojocck @ wjj.toral.u...du..... FFcully of Mathematics and Computer Science, Doctoral School of Exact and Natural Sciences
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

a) Sparsification
b) Dense to MoE conversion
c) Dynamic-k expert selection
MSR
E1
LAKA
Gate
B
Gate
Figure 1: Key components of D2DMoE: (a) We enhance the activation sparsity in the base model. (b) We convert FFN layers in the model to MoE layers with routers that predict the contribution of each expert. (c) We introduce dynamic-k: routing that selects the experts for execution based on their predicted contribution.
Contributions of this paper: We consider the following research question: what is the optimal way to convert a generic Transformer model into an equivalent sparse variant? We identify series of weaknesses of the MoEfication process limiting the resulting accuracy-sparsic tradeoff, and propose corresponding mitigations as follows. We call the resulting algorithm Dense Dynamic-k Mixture-of of Settrr--f-Exppeeee Experts (D2DMoE) and outline it in Figure
First, we analyze the relationship between the activation sparsity the starting model and the efficiency of the final MoE model. We show that computational savings directly related to sparsity levels, and we correspondingly enforce higher activation sparsity levels before conversion through lightweight fine-tuning process, which leads to substantially improved cost-to- t-r--ormmmaeeceeeeee-ee trade-off.
2. We identify the router training scheme in the original MoEfication algorithm as limitation of the conversion process. We propose to frame the router training regression problem instead, hence our routers directly predict the norm of the output of each expert.
3. We show that Transformer models exhibit significant variance of the number of activated neurons, and standard top-k expert selection in the MoE layers is inefficient. We propose an alternative dynamic-k expert selection scheme that adjusts the number of activated experts per-token basis. This approach enables the model to efficiently allocate compute between easy and hard inputs, increasing the overall efficiency.
4. We generalize the conversion method to any standalone linear layer including gated MLP variants commonly found in modern LLMs 45 42 and projections in Multi-Head Atten- tion (MHA) layers (which often account for over 30% of total computations Transformer models (39).
For MHA, we propose a replacement procedure in which every dense layer is substituted by a two-layer MLP module trained to imitate the output of the original layer.
We evaluate D2DMoE across benchmarks in text classification, image classification, and language modeling, demonstrating significant improvements in cost pes--performacc trade-offs in all cases. D2DMoE is particularly well-suited for contemporary hardware, as evidenced by our efficient GPU implementation, which we contribute alongside our proposed method.
2 Motivation
MoE. models have gained a lot of traction over the last years as an effective architecture to decouple the parameter count from the computational cost of the models (56). In MoE layer. hard sparsity is usually enforced explicitly by applying a top-k operation on the outputs of a trainable gating layer. However, many recent works [53] 2 30] have shown that most Transformers, when trained scale, build intrinsically sparse and modular representations. Zhang et al. [52) proposed to leverage this naturally emerging modularity with MoEfication method that converts dense transformer models into MoE models by grouping FFN weights into experts and subsequently learning small routers that determine which experts to activate. Models converted with MoEfication are able to preserve the performance of the original dense models while using only a fraction of their computational cost. However, we believe that the MoEfication procedure is not optimal, and therefore aim obtain dense-to-sparse conversion schemes that obtain a better cost-performance trade-off.

---

60.0
a [sparsty] 0.05 [89%)
Non sctvotios
50.0
Scanaieed
Base model
20.0
(88%)
0.30 [90%)
10.0
Gaae
[94%)
06
CV/pameeant
Dysantte gataa (onn)
83
Gate
3
100
GFLOPs
120
140
0.0
laysr Index
10 11
Impact sparsity MoF: conversion
NonNonzzen activations distribution
Top-k dynamic- & gating
Figure 2: (a) Cost-accuracy tradeoff' for MoFfied [27] GPT-2 model obtained starting from models with different levels of activation sparsity. Sparsification correlates with the model performance. (b) Distribution non-zero activations in the FFN layers in GPT-2-base on OppeneeTexx, with and without the sparsity enforcement phase. Both models exhibit significant variance, and the mean-to- variance ratio increases in the sparsified model. (c) We propose to exploit the variation activations through dynamic-k routing procedure that adapts the number of experts allocated to sample.
Intuitively, MoE converted from sparser base model would be able to perform the original function using a smaller number of experts. To validate this hypothesis, we perform MoEfication on different variants of GPT2- GPT2-basq with varying activation sparsity levels and show the results in Figure 2a As expected, MoEfication performs better with sparser models. We further investigate the per-token mean and the variance of non-zero neurons in the base and sparsified model, and show the results in Figure 2b Observe that different layers use different number neurons on average. Moreover, the variance of the number of activated neurons is quite high and becomes even more significant in the sparsified model. This means that static top-k gating as used in MoEfication is not optimal for dense-to-MoE converted models, and a more flexible expert assignment rule that would be able to handle the high per-token and per-layer variance could be beneficial to the efficiency of such models, as illustrated at Figure 2c Such dynamic-k gating requires routers that reliably predict the contribution of each expert. We observe that routers obtained through MoEfication do not accurately reflect this contribution. Morcover, their router training procedure depends on the strict sparsity of the model guaranteed by the ReLU activation function. Therefore, we design novel router training scheme that directly predicts the contribution of each expert and generalizes to the broader family of activation functions. We combine the proposed components (sparsity enforcement, expert contribution routing, and dynamic-k gating) into a single method that we call Dense Dynamic-k Mixture-of of-Experts (D2DMoE), which we describe in detail in the next Section.
Method
D2DMoE reduces the computational cost of the model by splitting every MLP module into MoE layer. In this section, we describe all of its components in detail. A high-level overview of the entire procedure presented in Figure T The conversion process can be optionally preceded by MHA projection layer replacement (Sec. 3.5), which allows us to apply the same transformation pipeline on all replacement modules.
3.1 Enforcing activation sparsity
We expect that enforcing higher levels of activation sparsity may allow for the execution of an even smaller number of experts, resulting in overall computational savings. To this end, we induce activation sparsity by fine-tuning the model with an additional loss term that induces activation sparsity [ We apply the square Hoyer regularization 22 I7
on the activations of the model:
2.(x)
1 L
Σ
|
(1)
Σ
a
'We provide the experimental details for this analysis in Section 4.3 and Appendix

---

where a' is the activation vector from the middle layer of the 1-th MLP for imput and L is the total number of MLPs in the model. Overall, the model is trained with the following cost function:
[(x) = Len(D. v) + (z.
(2)
where LcE is cross-entropy loss, and a is the hyperparameter that controls the strength of sparsity enforcement. We find that the pre-trained models recover the original performance with only fraction of the original training budget (eg. IB tokens for GPT2-base or Gemma-2B, which is less than 1% of the tokens used for pretraining).
3.2 Expert clustering
We split the two-layer MI.P modules into experts using the parameter clustering method proposed by Zhang et al. (52). Assuming the MLP layers are composed of weights W1. W2 and corresponding biases bj. b2. we treat the weights of each neuron from W1 as features and feed them into the balanced k-means algorithm [26] that groups neurons with similar weights together. Then, we use the resulting cluster indices to split the first linear layer W1, the first bias vector b1, and the second linear layer W₂ into n experts of the same size. The second bias by is not affected by this procedure.
MoEfication process was designed for standard two-layered MLPs [52]. Recent LLMs 44 have shifted towards gated FFNs, where the activation is realized through a Gated Linear Unit (GLU) 37 which contains an additional weight matrix for the gate projections. To adapt the expert clustering procedure described above to gated FFN layers, we cluster the weights of the gating matrix W g instead of W1, and use the obtained indices to divide the weights of the two other layers. We provide more intuition and details on our method for gated FFNs in Appendix G
3.3 Expert contribution routing
In a standard MoE-based model, the gating networks are trained in an end-to-end manner. Contrary to this, we train each gating network independently. We propose to frame the problem of training the router as a regression task and directly predict the l -orm of the output of each expert with the router. Formally, given an input token 2, train D2DMoE router R to minimize the following loss:
L. (x) - - (R(=); [=)]1))))
where Eg is the i-th expert. We use a small two-layer neural network as the router R and apply an absolute value activation function to ensure non-negative output. This prrcesiionbaasee formulation is still compatible with the commonly used top-k expert selection, but enables more precise attribution of the contribution of each expert,
show later in the experimental section.
Note that Zhang et al. [52] also trains each routing network independently, but their method constructs artificial labels for each input, and then subsequently trains the router as a classifier. We discuss the differences in detail in Appendix A
3.4 Dynamic-k gating
Commonly used MoE layers always execute top-k experts for each token, where l is a predefined hyperparameter. This means that, regardless of the difficulty of the input, the model spends the same amount of compute on each batch (54) or token (38). While this may be appropriate if the model is trained with the same restriction, it suboptimal for model that was converted from dense model, as we show in Section 2
Since router directly predicts the & -n-omm of the output of each expert, propose a dynamic-k expert selection method that skips experts for whom the router predicts relatively small output norms. Given router output vector R(z), we select hyperparameter E [0, 1] and define the expert selection rule G for the i-th element as:
G(z); = {
if R(2); 2 T max R(2) if R(2;; <T T max R(2)
Note that as 7 increases, the number of executed experts and the overall computational cost decrease. We emphasize that after model deployment can be adjusted without retraining.

---

1ettaate DEOWWEE
MotFcaton DODMKE
Baod
DEDANE
ZTW
ZFW
Baseline
Banalls
25.
50
75 120 125 GFLOPs
155
10
150
110
120
140
150
0.75 1.88 125
1.5 22 TFLOPs
2.20
ViT-B InageNet-1k
BERT CARER
GPT GPT-2-base OpenWebTest
(d) Gemma 2B C4
Figure 4: ProPs-performance tradeoff comparison our method and MoEfication 52 on CV and NLP benchmarks. We also include early-exit (ZTW, (4) and token dropping baselines (A ViT, (5I)) for classification. Our method outperforms these baselines across multiple computational budgets.
3.5 Conversion standalone dense layers
significant amount of computing in deep neural networks is often spent on dense layers that are not followed by any activation function. Dense-to-sparee sparse-Moo conversion methods cannot reduce costs such layers due to a lack activation sparsity. This determines an upper bound on the possible computational savings. To overcome it, we substitute dense layers with small MLPs with approximately the same computational cost and number of parameters. Each MLP trained to imitate the output of the original dense layer given the same input by minimizing the mean squared error between the two (akin to a distillation loss).
In our case, for Transformer architectures, we substitute projection matrices along with their biases in every MHA layer, as depicted in Figure 3 This means that the final model has four MoE layers the MHA layer and one MoE layer in the FFN layer (either plain gated) for each Transformer block. Note that we not modify the computation of the scaled dot-product attention itself and this scheme can be applied to any standalone dense layer.
Seated Doe Produur Atteaciia
Staled Produck Astentire
X
Figure 3: tion conversion scheme.
Attention projec-
4 Experiments
To analyze the impact of our method, we evaluate its performance on language modeling, text classification, and image classification. We obtain performance versus computational cost character- istics for each method by evaluating the methods with different inference hyperparameters (either T described in Section 3.4 for D2DMoE or number of experts k for MoEfication; we mark them on the plots with dot markers). We report the computational cost of each method FLOPs, as it is deviee-independent metric that has been shown to correlate well with latency [27]. In addition, measure the wall-clock execution time of an efficient implementation of our method.
For MoEfication, we follow the procedure described by Zhang et al. 52 by converting the activation functions the pre-trained model to ReLU and then fine-tuning the model. In the case D2DMoE, we also replace activation functions with ReLU, except for Section[5.4] where we demonstrate that our method performs well also with GELU. To provide fair comparison, the total training data budget always the same between different methods. See Appendix [1 for a detailed description of our setup. The source code for our experiments is available at: hhtpp: : //githubb com/ www.coommtaaeeeeeee battxooocck//2/MMe
4.1 Image classification
Vision Transfomer 7 is one of the most popular architectures in computer vision. Since our method applies any Transformer model, we evaluate on the popular ImageNet- 1k (35) dataset. We use pre-trained VIT-B checkpoint as the base model and compare D2DMoE with MoEfication in terms of the computational cost versus accuracy trade-off. broader comparison, we also evaluate state-of-the-art early-exit method Zero-time Waste (ZTW) [49], well our re-implementation of A-VIT, an efficient token dropping method proposed by Yin al. (5I) Our results, presented in Figure 4a demonstrate the significant gains from applying method over MoEfication.
5

---

Table 1: Relative downstream performance D2DMoE and MoEfication on BoolQ dataset. Our method only starts to degrade at around 70% compute budget, while MoEfication gradually decreases.
Compute budget
100%
90%
80%
70%
60%
50%
25%
10%
MoEfication 100%
92.24%
92. 19%
92.15% 15%
38.79%
75.40%
86.70%
77.53% 90.89 %
D2DMoE 100%
99.68% %
99.37%
98.69 %
97.60%
94.34%
92.75 %
4.2
Text classification
evaluate our method with BERT-base [6] on the CARER dataset (36 that contains text samples categorized into 6 different emotion categories. We compare the accuracy versus FLOPs trade-off for D2DMoE, MoEfication, and ZTW. We show the results in Figure 4b] The performance of MoEfication gradually deteriorates and completely collapses when the number of executed experts approaches zero. In comparison, D2DMoE maintains the original performance for a wide range of computational budgets, and the performance drop starts at significantly lower budget. While carly-exiting performs well for the lowest budgets, it obtains worse results than D2DMoE at medium budgets and suffers from gradual performance decline.
4.3 Language modeling
We evaluate our method on language modeling and compare with MoEfication using GPT-2- base [31] and Gemma-2B [2).. We initialize GPT-2 models from publicly available OpenAI checkpoint pre-trained on a closed-source WebText dataset and use OpenWebTexx [12] in all of our experiments. For Gemma-2B, we also start from the publicly available pretrained model and evaluate its language capabilities on the C4 dataset [32 after finetuning. For both models, we use around IB tokens for the finetuning phase (less than 1% of the cost of original pretraining) and 8- 16M tokens for router training. We report the results in this section without the MHA projection replacement, as this task is highly sensitive to changes in attention layers, leading to noticeable loss degradation. For more training details, see Appendix J.3
We present test losses for D2DMoE and MoEfication at different compute budgets for GPT-2-base and Gemma-28 Figures 4c and 4d respectively. Our method outperforms the baseline at every computational budget. The loss of D2DMoE plateaus for higher budget levels, while the baseline displays consistently worse results whenever we lower the computational budget. Notably, for the larger Gemma-22 2B model our method performs well for most compute budgets, while the performance of MoEfication collapses. The failure of MoEfication can be explained by the emergence of massive activations in large models 40), which makes it unable to learn reliable routing, as we analyze in
more detail in Appendix
We also provide downstream evaluation of our Gemma models on the BoolQ dataset. We take the base model, which achieves 68.40% zero-shot evaluation accuracy, and convert it to MoE with D2DMoE and MoEfication. In Table I we report the relative accuracy of the models at different compute budgets. Our method largely retains the performance across multiple compute budgets, while the performance of MoEfication decreases significantly. This shows that the FOLSPP results for D2DMoE and MoEfication directly translate downstream performance on language tasks.
4.4 Execution latency
0.25
Exper unage bacMet with 93% original SIAATOY
For any model acceleration method to be practically useful, it must reduce end-to-end inference execution time modern GPU hardware. To achieve this, we implement the forward pass of our MoE-layer in the Triton intermediate language 43), and em- ploy several optimizations for our implementation, including an efficient memory access pattern, kernel fusion, and configuration auto-tuning. As suggested by Tan et al. [4T] our implementation also avoids unnecessary copies when grouping tokens.
(ett cook te 2e8
020
015
010
0.05
ML7
200M52 mated (trinn)
0.0
0.0
02
0.4
3.6
0.8
1.0
FLO%
169
We verify the performance of our implementation for single D2DMoE layer (24 experts with expert dimensionality 1 128) layer
Figure 5: Single D2DMoE layer execution wall-clock time.

---

in isolation by comparing it with the corresponding MLP module (inner dimensionality 3072) on an NVIDIA A100 GPU. We fill tensor of size [256 x 197 x 768] (batch size, sequence length. and hidden dimension, respectively) filled with Gaussian noise and use it as input to both modules. The gating network of D2DMoE is included in measurements, but the decisions are overridden with samples from Bernoulli distribution, and we control how many experts are executed on average by changing the Bernoulli probability. The results, presented in Figure [ show that our implementation scales linearly with the number of executed experts, and has negligible overhead. Our method can be almost three times as fast as standard MLP while preserving 99% the original accuracy. In Appendix C we provide additional wall-clock measurement results along with more detailed description of our implementation.
4.5 Compatibility with model compression techniques
To accelerate inference D2DMoE leverages input-dependent ac- tivation sparsity, a property inherent to almost every Transformer model. However, interaction between D2DMoE and other popular network acceleration techniques, such as pruning [16] or quanti- zation [3 28), J is unclear. We evaluate D2DMoE in combination with such techniques to demonstrate their complementarity.
0.90
Accuracy
0.88
CuR 2:38 +OZDMLE
0.86
a-2.7
+0ZDNCE
CuRL 2-238
+OZENLE
0.82
m-2.3
First, we evaluate D2DMoE applied on top of networks pruned with CoFi, structured pruning technique introduced by Xia et al. [50 CoFi removes redundant neurons, attention heads, and sub- layers to achieve the desired sparsity ratio, and then subsequently fine-tunes the reduced network. We first prune the base model with CoFi to the desired sparsity level, apply D2DMoE to it, and then evaluate both models QNLI [48] In Figure 6 show that D2DMoE successfully accelerates inference even on networks pruned to high sparsity levels.
0.80
YO2ENCE
GFLOPs
Figure 6: D2DMoE applied to models pruned with CoFi.
0.90
Accuracy
0.80
we also investigate the applicability of D2DMoF to quantized models using dynamic post-training quantization from Py P on BERT trained on the CARER dataset. Our method is robust to 8- and 16-bit quantization and exhibits only slight variations in performance after quantization. As FLOPs do not take bit width into account, we show quantized models in the same FLOPs range as the original model. In Appendix C we also present wall-clock time measurements for quantized D2DMoE.
In Figure 7
0.70
DIDANE
DEDANE (1628)
0.65
DEDANE (161)
7
8 GFLOPs
10
Figure 7: D2DMoE applied to quantized models.
5 Analysis
In this section, we present in detail additional experiments that provide insights into the performance of our method. Additionally, in Appendix(E) we analyze the performance of MoEfication with Gemma. in Appendix [F we provide the results of router architecture analysis, in Appendix H we conduct experiments corresponding to the ones in Section 5.3 with GELU function, and in Appendix ! we show additional visualizations for expert activation patterns.
5.1 Expert selection patterns
The dynamic-k rule introduces variability in the allocation of the computational budget along the model depth. To explore its scale, we investigate the distribution of the number of executed experts, with and without the activation sparsification phase. number of activated experts for each FFN layer of the BERT-base model trained on the CARER dataset (additional results are available in the appendix in Appendix M. As expected, the model with enforced activation sparsity requires fewer experts for given threshold. Both base and sparsified models exhibit significant variance in the number of activated neurons across different layers, which justifies the dynamic-k selection and indicates that computational adaptability mechanisms are crucial for efficient inference in Transformer-basee models.
Figure Sa we show the histograms of the

---

12
Layer
Layer
20
40
60
80
100
Ratio of selected experts (1)
(a) Compute along the model depth
(b) Compatational load maps for ImageNet. IK sample images
Figure 8: D2DMoE allows for dynamical allocation of computation for each layer and each input independently. a) Per-layer distribution of the number of executed experts on CARER dataset in D2DMoE with 0.01 for standard model (top) and a sparsified model (bottom). Sparsification leads to a significantly lower number of selected experts. b) Computational load maps of selected ImageNet-Ik k samples for our converted VYT-B model with r 0.0025. D2DMoE allocates its computational budget to semantically important regions of the input.
D2DMoE also allows the model to allocate different computational resources various layers. We expect the model to allocate more compute to tokens containing information relevant to the current task. Since each token position the input image, we can easily plot a heatmap to indicate the regions of the image where the model spends computational budget. In Figure 8b) we present such an analysis for our converted ViT-B model. As expected, the dynamic-k routing enables the model to minimize the computational effort spent on regions that contain insignificant information.
ViT model corresponds
separate and non-overlapping part of
5.2 Ablation study
Since our method consists of several steps, the positive impact of each one of them may not be evident. To show the significance of every component,
perform ablation study by incrementally adding each component to the baseline method. We take a BERT-base model and evaluate the ablated variants the same setting as described in Section 4.2 The results of this experiment are presented in Figure Da As expected, each ablated version of the method improves upon the previous one. The sparsity enforcement phase leads to enhanced performance compared to plain MoEfication. Alternative router training objective and dynamic-k expert assignment further improve the results, but as the method only operates on the FFN layer- the computational cost cannot go below the cost of the remaining part of the model. Extending D2DMoE to MHA projection layers allows our method to reduce the computational cost further, and the resulting full method retains the accuracy of the original model at around twice fewer FLOPSs than MoEfication.
5.3 Base model activation sparsity
To justify our proposed activation sparsity phase, we investigate the impact of the activation sparsity of the base dense model on the performance MoE obtained with our method, We conduct a study similar to the one presented in Figure 2a we train multiple base models with different activation sparsity enforcement loss weights a and convert them to of models with our method.
The results, shown in Figure 9b highlight the positive correlation between the activation sparsity and the performance of the converted MoE, as higher sparsity in the base model always translates to better performance for D2DMoE. This is consistent with results previously observed for MoEfication. However, our method achieves better results for every base model in all cases, proving that regression routing and dynamic-k selection better utilize the induced sparsity.
5.4 Sparsification and reliance on the activation function
Activation sparsity works focus their analysis on networks with ReLU activation, as other functions (such GELU SiLU) do not guarantee exact sparsity. When analyzing non-ReLU models, such
8

---

7.0
6.
Appuree)
LONK
PALli
0.0
4.0
3.6
0.5
12
102
1.0
140
100
112
170
130
192
152
102
122
120
100
180
GFLOPs
GFLOPs
Phases ablation hase
(b) Sparsification (GPT2.base)
(c) ReL.U vx GELU (GPT2-haxe) (d) Expert gramularity (GPT2-base)
Figure 9: Analysis experiments with D2DMoE. (a) Impact of different phases of our method. Each phase improves upon the baseline. (b) Sparsification improves the cost-accuracy trade-off of the final D2DMoE model. (c) Sparsification allows us to apply our method to GELU-based model without significant drops in performance. (d) Smaller experts display favorable performance and allow for larger computational savings.
works require fine-tuning with the activation function changed to ReLU (relufication) 27 which limits their practical applicability. We hypothesize that relufication is not necessary and the models with many near-zero activations effectively function similarly to standard ReLU-based models. To evaluate this hypothesis, we extend the sparsity enforcement scheme to the commonly used GELU activation by penalizing the model for pre-activation values larger than a certain threshold. We first transform the pre arevviiivttion values as z' max(0, d), where is the actvvctivation value and d is a displacement hyperparameter. Then, we apply the loss from Equation (I) on z'. This transformation penalizes only pre-activation values larger than d, and as a result, the model learns to produce values that effectively become negligible post-activation. We empirically find that = - 10 works well with GELU as the output below this value is zero.
To validate our hypothesis, we follow the methodology from Section 4.3 and we train ReLU- and GELU-based GPT-2 with and without sparsity enforcement loss. We show the results in Figure 9c D2DMoE with a sparsified GELU-based model performs similarly to a sparsified ReLU-based model, while the performance of the non-sparsified GELU-based variant collapses. Within ReL.Ubbsed models, the sparsification still enhances the performance of D2DMoE, but the improvements are less drastic, and the behavior of our method does not significantly change in the case of GELU, This shows sufficient activation sparsity enforcement relieves the model from the dependence on ReLU.
5.5 Impact of expert granularity
crucial hyperparameter in D2DMoE is the selection of expert size. Smaller experts may allow more granular selection of executed neurons, likely resulting in a lower computational cost. However, decreasing the expert size increases the number of experts, which translates to larger router, potentially negating any computational gains. To study the impact of this hyperparameter our method, we evaluate D2DMoE on GPT-2 with different expert sizes, and show the results in Figure 9d
We observe that our method generally performs better with smaller experts. Those results differ from the ones presented in [52] where the expert size significantly higher. The positive correlation between granularity and performance can be explained by the increased levels of activation sparsity in our model, which requires significantly fewer activated neurons (experts). As expected, the perfor- mance decreases for the extreme choice of expert size equal to I due to significantly higher routing costs. We include additional results for expert granularity with GELU activation in Appendix H
Related Work
Mflxture-of-xpperte MoE layers were introduced as an efficient way further increase the
capacity of deep neural networks applied for NLP tasks, initially in LSTM models 38 and later
in Transformers 23). Since then, they have also been applied to computer vision 33
5).
MoE
layers have gained significant popularity primarily due to their excellent scaling properties [8 B. Nonetheless, training such models challenging, primarily because gating decisions must be discrete to ensure sparse expert selection. Various methods of training were proposed, some of which include reinforcement learning
[] weighting the expert output by the probability to allow computation of the gradient of the router 38 using the Sinkhom algorithm [3. Some of those approaches also suffer

---

from the possibility of load imbalance and therefore require auxiliary losses or alternative expert selection methods (9 144. Interestingly, in many cases fixed routing functions perform similarly to trainable routers 34 which suggests that current solutions are largely suboptimal. MoE models
can also be derived from pre-trained dense models by splitting the model weights into experts and independently training the routers for each layer 52 57.. which avoids most of the problems present in end-to-end training.
Activation sparsity in Transformers. ei (4 show that Rel.U-based Transformer models produce sparse activations in their intermediate representations, an effect that is prevalent across architectures, layers, and modalities. They propose a simple rule for keeping only top-k activations in each MLP layer, which results in model with comparable performance. Similarly, Mirzadeh et al. [27 demonstrate that ReLU activation function in LLMs encourages ensuing activation sparsity that can be leveraged to skip redundant computations. Tuli and Jha 46] take step further and design dedicated Transformer architecture accelerator that also exploits activation sparsity, while Liu et al. [25] proposes to predict activation sparsity structure in LLMs and reduce the model latency by skipping redundant computations. Jaszezur et al. [18) demonstrate that is possible to train Transformer models from scratch with a fixed level of activation sparsity and obtain similar performance. Finally, a related line of works focuses on sparsity in the attention distributions instead of intermediate representations (4. None of the above- mentioned methods explore induced activation sparsity as a way to increase computational gains, nor do they address variance of the number of sparse activations per-token basis.
7 Conclusion
We introduce Dense to Dynamic-k Mixture-of-Experts (D2DMoE), a novel approach that induces activation sparsity to improve the efficiency of Transformer-basee models by converting their layers to Mixture-of -f--xxerts (MoE). We demonstrate the interplay between the activation sparsity of dense models and the efficiency of converted MoEs. Moreover, we introduce regression- based router training and dynamic-k routing, which enable our method to efficiently utilize the induced sparsity. Finally, we show how dense-to-sparse-Mon projections and gated MLPs. Our approach is compatible with the existing Transformer architectures and significantly improves upon existing MoE conversion schemes. Our findings contribute the ongoing efforts to make Transformer models more efficient and accessible for a wider range of applications, especially in resource-constrained environments.
conversion approaches can be extended to MHA
Limitations and Broader Impact
While D2DMoE displays promising results in reducing the computational cost of inference in Trans- former models, few limitations should be acknowledged. Our proposed sparsity enforcement and router training phases require additional training time. This overhead, while small, must be considered when evaluating the benefits of our approach. Moreover, existing approaches on common NLP and CV tasks, but the scope of our experiments restricted due limited access computational resources. Further research needed explore its applicability extremely large models.
demonstrate improved performance
Our work focuses primarily on fundamental machine learning research and we do not see any specific risks or ethical issues associated with our method. Nevertheless, we recognize the potential for misuse of machine learning technology and advocate for responsible AI practices to mitigate such risks.
Acknowledgments
Filip Szatkowski is supported by National Centre of Science (NCP, Poland) Grant No. 2022/45/B/ST6/0288 Bartosz Wójcik is supported by National Centre of Science (NCP, Poland) Grant No. 2023/49//SST6/025513 Simone Scardapane was partly funded by Sapienza grant 1G123188BEEFGABI (CENTS). This paper has been supported by the Horizon Europe Programme [HORTZON-CIA-202-HUUMAA
under the project "ELJAS: European Lighthouse of Al for Sustainability", GA no. 101120237. For the purpose of Open Access, the authors have applied
10

---

CC-BY public copyright license to any Author Accepted Manuscript (AAM) version arising from this submission.
We gratefully acknowledge Poland's high-performance Infrastructure PL.rrid (HPC Centers: ACK Cyfronet AGH, PCSS, CI TASK, WCSS) for providing computer facilities and support within computational grants no. PLG/2023/016399, PLG//023/0163211, and PLG/20240017385.
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.3432, 2013.
[2] Xuanyao Chen, Zhijian Liu, Haotian Tang, Li Yi, Hang Zhao, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 2061-2070, June 2023.
[3] Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. International Conference on Machine Learning, pages 4057-4086. PMLR, 2022.
[4] Gonçalo M Correia, Vlad Niculae, and André FT Martins. Adaptively sparse transformers. 2019.
arXiv preprint arXiv:
[5] Erik Daxberger, Floris Weers, Bowen Zhang, Tom Gunter. Ruoming Pang, Marcin Eichner, Michael Emmersberger, Yinfei Yang, Alexander Toshev, and Xianzhi Du. Mobile v-moes: Scal- ing down vision transformers via sparse mixture-of-experts. arXiv preprint (iXiv:2309.04354 2023.
[6] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18X10:1810.04805. 2018.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[8] Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference Machine Learning, pages 5547-5569. PMLR, 2022.
[9] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232-5270, 2022.
[10] Wikimedia Foundation. Wikimedia downloads. URL wttp:tttt:///mnntie http:///ddmmmsssiiiisieeeeee wikimedia .oreeidaaa..rg....oooooooooo
[11] Georgios Georgiadis. Accelerating convolutional neural networks via activation map compres- sion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7095-7095, 2019.
[12] Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. OpenieeTeettCorpus 2019.
http:://Styiion077.itttti io/
[13] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural net- works with pruning, trained quantization and huffmaan coding, arXiv preprint 4rX.v:1510.00149, 2015.
[14] Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(11):7436-746, 2021.

---

[15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint S503:.55030025535 2015.
[16] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 10010882--1005, 2021.
[17] Patrik O Hoyer. Non-negative matrix factorization with sparseness constraints. Journal of machine learning research, 5(9), 2004.
[18] Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojciech Gajew- ski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. Advances in Neural Information Processing Systems, 34:9895-99007, 2021.
[19] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 1rX0v22310006825, 2023.
[20] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown. Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2011.08661 2020.
[21] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54 (10s):1-41 2022.
[22] Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference Machine Learning, pages 5533-5543. PMLR, 2020.
[23] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference Learning Represen- tations, 2020.
[24] Zonglin Li, Chong You, Srinadh Bhojanapalli, Daliang Li, Ankit Singh Rawat, Sashank J Reddi, Ke Ye, Felix Chern, Felix Yu, Ruiqi Guo, et al. The lazy neuron phenomenon: On emergence of activation sparsity in transformers. In The Eleventh International Conference on Learning Representations, 2022.
[25] Zichang Liu, Jue Wang, Tri Dao. Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivas- tava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient Ilms at inference time. In International Conference on Machine Learning, pages 22137-22170 PMLR, 2023.
[26] Mikko I Malinen and Pasi Fränti. Balanced k-means for clustering, In Structural, Syntactic, and Statistical Pattern Recognition: Joint IAPR International Workshop, St SSPR 2014, Joensuu, Finland, August 20-22, 2014. Proceedings, pages 32 32-41. Springer, 2014.
[27] Iman Mirzadeh, Keivan Alizadeh, Sachin Mehta, Carlo C Del Mundo, Oncel Tuzel, Golnoosh Samei, Mohammad Rastegari, and Mehrdad Farajtabar. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04564 2023.
[28] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. white paper on neural network quantization. arXiv preprint vrXiv:2106.08290 2021.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

[30] Zihan Qiu, Zeyu Huang, and Jie Fu. Unlocking emergent modularity in large language models. In 2024 Anmual Conference of the North American Chapter of the ACL, 2024.
[31] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
[33] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583- 8595, 2021.
[34]
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34: 15555-1666,,, 2021.
[35] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICV), 115(3):211-2555) 2015. doi: 10.700 12000//5126-----0116666
[36] Elvis Saravia, Hsien-Chh Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/
11/D18-1404 URL
httph:t////// hwtp:///wwwwwwwwww..//..... aclweb. org/ anthology/ (arplllyyyyy111110 1404
[37] Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202. 2020.
[38] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exxpe layer. In International Conference on Learning Representations, 2016.
[39] Zhuoran Song, Yihong Xu, Zhezhi He, Li Jiang, Naifeng Jing, and Xiaoyao Liang, Cp- vit: Cascade vision transformer pruning via progressive sparsity prediction. arXiv preprint urXiv:2203.04570 2022.
[40] Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint 2rXiv:2402.17762 2024.
[41] Shawn Tan, Yikang Shen, Rameswar Panda, and Aaron Courville, Scattered mixture-of-experts implementation. arXiv preprint arXiv:2403.0824 2024.
[42] Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829. 2024.
[43] Philippe Tillet, Hsiang-Tsuun Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pages 10-19, 2019.
[44] Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European Conference Computer Vision, pages 516-533. Springer, 2022.
[45] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.09288 2023.
[46] Shikhar Tuli and Niraj K Jha. Acceltran: A awars accelerator for dynamic inference with transformers. IEFE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023.
13

---

[47] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[48] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 07461, 2018.
[49] Bartosz Wójcik, Marcin Przewiežlikowski, Filip Szatkowski, Maciej Wolczyk, Klaudia Balazy, Bartlomiej Krzepkowski, Igor Podolak. Jacek Tabor, Marek Smieja, and Tomasz Trzciński. Zero time waste in pre-trained early exit neural networks. Neural Networks, 168:580-601, 2023.
[50] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint arXiv:2204.00408. 2022.
[51] Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809-10818. 2022.
[52] Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022, pages 877-890, 2022.
[53] Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xiaozhi Wang. Xu Han, Zhiyuan Liu, Ruobing Xie, Maosong Sun, and Jie Zhou. Emergent modularity in pre-trained transformers. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 4066-4083, Association for Computational Linguistics, July 2023.
[54] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoe V Le, James Laudon, et al. Mixture-of-expprts with expert choice routing, Advances in Neural Information Processing Systems, 35:7103-7114 2022.
[55] Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. The IEEE International Conference on Computer Vision (ICCV), December 2015.
(56) Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXi::2202.088800 2022.
[57] Simiao Zuo, Qingru Zhang, Chen Liang, Pengcheng He, Tuo Zhao, and Weizhu Chen. MoE- BERT: from BERT to mixture-of-experts via importance- -uiided adaptation. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguis- ties: Human Language Technologies, pages 1610-1623, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 1/20/2/ www.h-main.naaaamaaaa 116. URL
https:: ttttp::/aclantholooo org 2022. naacl-main. 116
14

---

A Difference between router training in D2DMoE and MoEfication
Our router training procedure is similar the proposed in MoEfication [52] but the source code of the method provided by the author Sf contains a different routing objective than the one reported in the paper. While the paper describes their router training objective prediction of the sum of ReLU activation values in each expert, the source code uses prediction labels created from the sum of the activations in the intermediate layer divided by the maximum value in the entire batch and minimize the binary cross-entropy loss. Assuming that dg.j is the activation vector in the hidden layer of expert for sample k, the label generation for their router be expressed
maXI, Ei At,m.i
In comparison to their approach, the router training procedure in D2DMoE differs multiple aspects:
Our router considers the output of each expert instead of looking at the activations in the intermediate layers.
Instead of using artificially created labels based on the sums of activation values, we predict the ne-morm of the output. This has the additional benefit that our router can work with alternative activation functions.
Our router trained with the mean-squan mean-squared error instead of the binary cross-entropy loss. The output of our router is constrained to positive values, while the MoEfication router is constrained to outputs [0, 1].
We find that the above differences are responsible for the improved performance of our router (sce Figure 9a).
B Comparison of FLOPs between standard FFN layer and dynamic-k MoE
To compare the efficiency of our method with standard MLP layer in Transformer, we estimate FLOPs in both modules. We assume the layer is composed of two linear transformations, with imput and output size dm and hidden dimension edm- where the expansion factor, which is usually equal to 4 in standard Transformer models. We skip the negligible cost of the biases and activation functions for simplicity.
One can estimate the cost of the MLP layer in FLOPs, CMLP, as:
CEn = dm edm + edm dm = dm 2e.
(6)
For dynamic-k: expert selection with total experts and i experts selected for a given input, the cost of the forward pass composed of the cost of a forward pass through & experts and the cost of the 2-layer router with input dimension dm hidden dimension d,, and output dimension The cost of the single expert pass can be expressed as:
Ce = (dm edm + edm dm)) d 2. 2e
and the routing cost can be estimated as:
CR dm dn dn n.
(8)
Therefore, the full cost of dynamic-k Coynk can be estimated as:
2ek
Coynk = k CE + CR = dm
+ dy, (dm + nn,
(9)
n
and the cost of our method compared the cost of standard MLP can be expressed as:
Caynk OMLP
d,n 2. Zek + dn (dm n)
(10)
d 2 2.2e
k
d + +
(11)
d 2c
'MoEfication source code for router training is publicly available at: http::://ttttttisiee MoEfication/// ooffaaoooobbbosssaaaaaaaaoooooar
L188- L260
15

---

dm
dm
6dh,
dm = 3dh. e
dm 2dh.
cost ratio
1.00
0.10
1
e dm
e dm
e e.dm
Figure 10: FLOPs ratio between dynamic-k expert layer and standard two-layer MLP for different values of the total number of experts n and number of selected experts k. We assume the hidden dimension of router d₁, is based on model dimension dm and set standard expansion factor = 4. For different sizes of router, dynamic-k uses fewer FLOPs than standard MLP long the total number of experts is sufficiently large and the number of selected experts is not equal to the total number of experts. For the clarity of presentation, we plot discrete values of & and n as continuous.
As long as the number selected experts does not approach the total number of experts n and the hidden dimension of the router does not approach the size of hidden dimension dms the ratio is significantly below one.
Assuming the worst case for second term (n = edm), we can estimate the cost ratio as:
d, 1te
(12)
dm 2e
which shows that dynamic-k expert selection only exceeds the FLOPs cost of the standard network when the dynamic-k rule selects almost all experts the number of experts becomes very high. For an even more detailed analysis, we refer to where the cost ratio between our method and standard MLP is shown, assuming different router sizes and 4 as standard for most Transformer models. In practice, we use d = 128, so in all our experiments dm 6d/.-
C Efficient implementation of D2DMoE
In Listing we present the pseudocode for our efficient imple- mentation the forward pass of the D2DMoE module. We skip the pseudocode of the kernel of the second layer as it is similar, but provide the full source code our code repository. Note that our implementation has multiple points where it could be im- proved for further performance gains: I) metadata that is required for the kernels could also be computed with dedicated kernel to reduce overhead; 2) atomic operations are currently used in the second layer to merge the results from different experts, alternative implementation that does not use atomic operations could be faster; 3) it could be rewritten in CUDA make of dynamic parallelism. We leave those improvements for future work.
1.0
1.2
1.4
1.6
18
2.0
2.2
Wall clock dime (mi)
Figure 11: Wall-clock time mea- surements of the ViT-B model and its corresponding D2DMoE model.
In the main paper, we have presented wall-clock time measure- ments of single D2DMoE layer. Below, we also ensure that our implementation works and performs well when used for the ViT-B model in which each FFN is replaced with a D2DMoE module. In Figure II measure the averaged processing time and the accuracy of our model. We perform the experiments on an NVIDIA A100 GPU using batch size of 256. Each point on the x-axis corresponds to single threshold and shows the wall-clock time of processing single input averaged over the entire ImageNet- lk test set. Dynamic inference with D2DMoE offers up to 30% reduction in processing time without affecting the accuracy.
To show that D2DMoE also reduces the execution latency of quantized models, we modify our we perform similar experiment to the kernels to handle f1oat16 and int8 data types. In Table 2
16

---

one from Figure 5 We sample gating decisions from the Berouulli distribution with probability and measure the execution time of our experts for the three data type variants.
Table 2:
Wall-clock time measurements (f8) of execution of our D2DMoE layer when using different data types and GPUs.
GPU
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
float32
5
13
18
23
28
33
38
42
47
52
RTX 4090
float16
4
7
9
11
14
16
18
24
27
int8
4
5
7
8
9
10
11
12
13
14
float32
6
12
15
22
28
35
38
A100
float16
6
7
8
10
11
13
14
16
17
19
21
int8
7
9
10
12
15
16
19
The results show that both the higher activation sparsity (lower p) of our method and lower-precision data types are complementary terms of wall-clock time reduction. While see smaller improvement from using int8 over f float16 on A100, we attribute this to differences between GPU architectures and software support for low-precision arithmetic.
D Compatibility with knowledge distillation
0.8
0.7
In Section 4.5
have demonstrated that method is compat-
Accuracy
0.6
ible with two popular model compression methods: quantization and pruning. A natural question is whether method can be effectively applied to models compressed via knowledge distil- lation. Since distilled models also exhibit activation sparsity that our method relies on, D2DMoE should be applicable to such mod- els. In Figure 12 we demonstrate the results of D2DMoE when applied on VIT-S model, which has been trained via knowledge distillation [15] with the torchvision ViT-B being used as the teacher model. We see that D2DMoE is also able to reduce the cost of this smaller model.
0.5
0.4
0.3
Distil ed VIT-S D2DMoE
0.2
Distilled VIT:S
2.5
3.0
4.5 5.0 GFLOPs
3.5
4.0
Figure 12: Performance of D2DMoE applied on a VIT-S distilled from the larger ViT-B model.
E Routing analysis for large models
As presented in Figure Ad, comparison other considered benchmarks MoEfication visibly underperforms on language modeling with Gemma-2B. We attribute this the emergence of massive activations in LLMs that reach a specific scale [40].
McEfication
9
MREficstoo-RRR
X
Beseline
Loss
7
Massive activations are outliers along certain feature dimensions whose magnitudes are thousands of times larger than the magni- tudes of other activations. The training objective of MoEfication described in Equation (5) uses maximum activation over the en- tire batch to normalize the target label for each expert. Upon encountering large outlier values, those labels become effectively meaningless, as the values for most of the experts become very close to zero. In this case, the router effectively learns output zero labels for most of the experts aside from the ones correspond- ing to the outlier values.
6
1.2 14
TFLOPs
Figure 13: Comparision of perfor- mance on Gemma-2B for MoEfi- cation with vanilla routing and with our regression routing.
In comparison to MoEfication, our router training scheme does not make use of such normalization, and should therefore be robust to the emergence of massive activations. To validate this, we apply MoEfication on Gemma-2B, but with our regression routing instead the original router training strategy. We compare the resulting model with vanilla MoEfication in Figure 13 and notice that replacing the routing scheme is enough for the model to learn effective expert assignment, as even though the expert choice is static and the base model is not sparsified, the cost-loss trade-off has
17

---

7.0
0.9
6.5
Apsuracy
Router depinel
Loss
deptic
deginng
5.5
6.0
0.7
Roder depttng hoteen dinnde
5.0
Rodder deptnn2 heteen cennii
Rodeer deginn2 coeen con-12n
4.5
Bxpul size
0.8
Router depiri2 hriden (81-255
4.0
0.5
Basedin
Router deptrin hriden, dliv0122
3
3.5
10
12
90
100
110
120
130
140
150
160
100
120
140
160
180
GFLOPs
GFLOPs
GFLOPs
architecture ablation BERT
(b) Router archittecture ablation for
(c) Expert gramularity with GELU.
Figure 14: Additional ablations with router architecture and expert granularity.
significantly improved. This simple experiment shows that our regression routing objective is more robust than MoEfication when scaling larger models.
F Router architecture
In comparison to standard linear routers used in MoE models trained from scratch, routers in MoEfication are 2-layer MLPs. To obtain the best performance with D2DMoE, we compare the linear design with MLPs with different hidden sizes for BERT-base and GPT-2-base on Figures [4a] and [4b respectively. Linear routers do not perform well with our method, and overall 2-layer MLP with a hidden dimension of 128 results in the best performance for both models. Note how for BERT-base, the accuracy curve for model with the hidden dimension of 128 is slightly worse than for smaller routers, but for harder task with GPT-2 more complex router is required. Following this analysis, we use 2-layer MLP with a hidden dimension of 128 for most of our experiments in the paper, with the only exception being the larger Gemma-2B model where we scale the hidden dimension accordingly to 512 to match the increase in model dimensionality.
G D2DMoE extension to GLU-based layers
Standard FFN
W;
To provide better intuition behind the extension of our method to
GLU-based gated MLPs mentioned in Section 3.2 we visualize the
differences between standard FFN and Gated FFN and the applica-
tion our method in Figure T5] Standard Transformer MLP realizes the following function:
W;
7(x) W, W.WWWAWWW (W2x),
(13)
Gated FFN
Wi
where W W2 are the weights for the upscale and downscale projections and A stands for the activation function. In comparison, gated MLP can be written down as:
((x) = (AA(W gz) W22),
(14)
A
W
where W, is the weight for the added gate projection.
Duste
The intuition behind MoEfication, which method also follows for standard FFNs, is that the sparsity of the intermediate, post-activation representations determines the sparsity of the output representation. sion Gated MLP. Therefore, the expert split is performed based on the weights of the upscale projection, as zeroed neurons in the upscale activations will also result in zeroed outputs of the downscale projection. When extending D2DMoE to Gated MLPs, our intuition is that the gating projections determine the sparsity of all the later representations, gating values. Therefore, we propose to build the experts through clustering performed on the gating weights W and use the indices obtained through expert split on gating weights construct experts from W, and W2. Following similar reasoning, for GLU-based models, we also perform activation sparsity enforcement on the gating projections instead of upscale projections as described originally in Section 3.1
Figure 15: D2DMoE exten-
both upscale and downscale
multiplied with the
We omit biases for simplicity.
18

---

H Additional results with expert size and GELU
In addition to experiments in Section 5.3 we present the results of similar ablation carried on the sparsified GPT-2 model with GELU activation. The results, presented in Figure 14c follow the same pattern as before, which supports our claim that the sparsification enables the GELU based models to function similarly to ReLU-basee ones.
I Expert activation patterns for attention projection layers
Following the analysis for MoE-converted FFN layers in Section 5.1 we present full results for FFN in Figure I6 and investigate the activation patterns in MHA projections modified with our method in Figures 17 to 20 The projection modules display lower levels of sparsity than FFNs, which is to be expected as our projection layers have lower intermediate dimensionality. Expert selection distribution patterns in Q and K show significant similarity, and the patterns in V and also similar to lesser degree. The variance of the number of selected experts in MHA projections is higher than in FFN layers, but it still exists and the distribution in some of the layers seems to be bimodal, which provides further justification for the dynamic-k selection rule.
output projections
Training and hardware details
In this Section, we describe the technical details used in the D2DMoE conversion procedure. For full reproducibility, we share the source code that we used for conducting the experiments. All experiments were performed using the PyTorch library [29] on the NVIDIA A100 and V100 GPUs on internal clusters. We utilize the fvcore library to count model FLOP
J.1 Image classification
All methods start with the same pre-trained ViT-B from the torchvision library and are trained on ImageNet- 1k using the augmentation proposed by Touvron et al. 44 We use mixup (0.8), cutmix, label smoothing (0.1), gradient clipping (1.0) and the Adam optimizer with a cosine learning rate schedule without warm-up. For D2DMoE, we replace the MHA projections and train the replacements for 3 epochs with the initial learning rate 0.001 and batch size 128, and then finetune the model for 90 epochs with sparsity enforcement weight a = 0.2, initial learning rate 2 10-5 and batch size 512. We then convert the modules into MoE layers. and train the gating networks for 7 epochs with the initial learning rate set to 0.001 and batch size 128. We train ZTW for 100 epochs in total, allocating 5 epochs for ensemble training, while keeping the rest of the original hyperparameters unchanged. For MoEfication, we first convert the pre-trained model to ReLU-based one and finetune for 90 epochs with initial learning rate of 0.0001 and batch size 256. We then split the weights and train the routers for 10 epochs with the initial learning rate 0.001 and batch size 256.
J.2 Text classification
All experiments start from the same pre-trained BERT-base checkpoint. For methods requir- ing ReLU activation function, we replaced GELU with ReLU and continue model pretrain- ing concatenated wikipedia [10] and books [55) corpora for 5000 steps on 8 GPUs using /wta//eammllsssseeeee pyroorchllaguuggg-oooee ing/ run__uu__e mlm. py, per device batch size 96 and learning rate 5 10 For MHA projections replacement we use the same corpus and train replaced MLP modules on a single GPU with batch size 128 and learning rate 0.001 for 3000 steps. We finetuned base dense models on CARER dataset for 5 epochs with 2 10-5 learning rate. For sparsity enforcement in D2DMoE use linearly increasing from zero 0.0001 over training. For both MoEfication and D2DMoE we train routers with batch size 64 and initial learning rate 0.001 for 5 epochs. In all experiments, we use Adam optimizer with linear learning rate decay, For MoEfication we use expert size 32, for D2DMoE we use more granular expert size equals 6. For ZTW we trained ICs for 5 epochs with batch size 32 and learning rate 0.01.
main setup from /tstps://githun com
19

---

J.3
Language modeling
We base our code and hyperparameterrs for GPT2-base on the nanoGPT repository provided at htsts 7/github com http://aaaaaaaaaaaaaaooao
We initialize the model from https: htop::://huggnggg http///hgggnggac.ce.e co/ openai- coommunity/gaan gpt2 In all pretraining experiments, we initialize models from a publicly available OpenAl checkpoint pre-trained on closed-souree WebText dataset and finetune for the fixed number of 1000 steps with the effective batch size equal to the value in the repository through gradient accumulation. The alpha values for sparsity enforcement can be found at Figure 9h We train the routers for D2DMoE and MoEfication for 2000 steps using one GPU and tuning the learning rates for given expert size from the range between 0.002 0.005. For router training, we use Adam optimizer and cosine warmup scheduler.
co/ For Gemma-2B, we start from the checkpoint at httttt://hugginggteeee ct/ google/ genma- 2b We also finetune the model for Ik steps with an effective batch size 1024, sequence length of 1024 and Adam optimizer with a learning rate of le-4. As Gemma's 's hidden dimension is much larger than the other considered models, we change the hidden dimensionality of the routers 512 for both our method and MoFfication, but keep the other hyperparameters the same as in the rest of the experiments. For MoEfication, Gemma, we use 512 experts to obtain an expert size comparable the one in their paper. For our method, we use 2048 experts. In D2DMoE, we set sparsity enforcement weight to 0.00003. We train the routers for 500 steps with Adam and effective batch size 16 and use learning rate of 0.001.
We report the results for language modeling without the MHA projection replacement step, as we find that it is especially sensitive to changes in the attention layers, which always result in visible loss degradation.
K
Contributions
Filip integrated the codebase and ran the experiments for GPT-2 and Gemma, performed the activation sparsity analysis, and all the analyses on language modeling models. He contributed to the design of dynamic-k gating and played primary role in designing the experiments and writing the article.
Bartosz set the research direction of the project and proposed the alternative routing scheme, dynamic- k expert selection, the additional activation sparsity enforcement phase for ReLU and GELU, and the replacement of MHA projection layers. He wrote the shared codebase for the experiments, carried out the ViT-B experiments, implemented the custom Triton kernels for the efficient implementation of the method, and also played primary role in the writing and editing the article.
Mikolaj made this paper possible by performing all of the experiments at the initial stages of the project and implementing MoFffication and numerous variants of our method. He carried out the BERT experiments, performed weight sparsity compatibility analysis, the ablation study, and contributed to the crafting the paper.
Simone significantly improved the paper's 's readability and provided invaluable advice for revising it.
20

---

def forvard_ www.etrrtoooiiieitooeeee (slmi routing.teneor);;
# compute the necessary netadata
split the batch into groups: executed that expert
# (for each expert independently) sort_inndcee
outing.teneeo arggert "ygoor(ddin-) descending-True)
# get the number of sauples executed by each expert
expert_
satting.tensor.aundddi # actual forward pass
intermediato_ace Hoejzestayyelmmlmmmnattiii (gply apply(...) final_out aaaaaaadllggeeetttniaaaaaatt...d
11
return final_out
13 class Koefireslayyrrinllanntttttt (torch. autograd Function); )tntor........n.iiii.))) @staticnethad
15
def forvard (imput. veight, biss, sort_indices, expert_bincour
16
grid karnel
which divide computational work
18
in multiple dimensions: batch dimension (sauple_din)
output dimension (expert_din) and nunber of experts dimension (nun_oxperts) grid (cdav(sample_da)) BLOCK_SIZE_DD cdiv(expart_dts BLOCK_SIZE_ED), nun_experta) toe_firet_kerae (...)
20
23
25
Striton. jit
keret
s_.ptr
27
weight_ptr,
28
bias-ptr,
29
output_ptr,
30
sort tort_indices_ptr,
Dxpert_,incouute.ptrr.
32
34
# based on tl progaag I (axia=0)) compute the tile indices for the batch and output dimensions
35
36
# (grouped, column najer pid_bd, pid_ed
ordering)
37
kernel expert_index _inee sl.progran_iddarii assigned load
experte
nany-to-one relationehip
39
number of (expertt.tnccouma
expert
41
expert_samples_come = tl.
expert_index)
42
calculate the nunber instancee that need to be used process all tokene bd_pids_for_exper
43
t1. (it_eeeerrr__sppplee # instances that have no computation to perform exit early
(OCK_SIZE_BD)
44
pid_bd bd_pide_for__xpert
46
calculate offsets that will be used for addressing data in memory offs_bd .... offs_ed
47
48
19
offs_hd
pick the data load based in_deta_indices tl. oodi indices_ptter expert_inded calculate nenory a during loading this sill group samples for the current learner only x_ptra x_ptr in_data_indiceall Nona]
sort indices
51
offe_bd ...)
the input
veights
53
55
v_ptrs veight_ptr index the result will be sccusulated in this variable accumulator
56
BLOCK_SIZB_ED) Etsp==tl.floatti.
58
iterate over the inmermost dimension in range (0, cdiddd(nhiee BLOCK_SIZE_HD)).
60
load the nemory fran the current input and zeight tiles t1.1oad(x_ptra, mask..... other=0.0)
62
(vl_te, mack..... other-0.0)
63
perforn natrix multiplication for these tiles and accunulate (since data grouped, this accumulator += t1. tl.dot v)
performed in efficient manner)
65
the pointers atride_x_ho x-ptrs += BLOCK_SIZE_RE v_pirs BLOCK_3IZE,
67
68
veight,
load and add biases to the accunulated result offs_b_ed
20
b_ptra bias_ptr expert_indes accumulator +- t1.load (d. nask...., other-0.0) apply if ACTIVATION == 'relu':
72
73
74
75
accumulator relu(accunulator)
76
77
calculate the nemory addresses for the output offa_aut_bd output_ptr expert_ index out_ptra offa_ont_bd[: None] ... offs_b..... bettsbbber [Nona,
79
81
out_nask
82
store the result to the main GPU nenory
(out_ptrs, accumulator, mackk
Listing 1: Simplified pseudocode of our efficient D2DMoE, implementation for GPUs
21

---

t= 0.1
t= 0.01
t= 0.001
Model ayer
9
Base model
8
Model ayer
Sparse model
40
40
100
40
100
Ratio of selected experts (%)
Ratio of selected experts (%)
Ratio of selected experts (%)
Figure 16: Per-layer distribution of the number of executed experts in D2DMoE trained on the CARER with different thresholds for a standard, non-sparsified model (top row) and sparsified model (bottom row). The high variability of that number explains the computational gains from using dynamic-k.
T- 0.01
0.001
12
Model ayer
Base model
10
3
12
Model ayer
Sparse model
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Ratio of selected experts (%)
Ratio seleecte experts (%)
Ratio
selected experts (%)
Figure 17: Distribution of the number of executed experts in each layer for query projections.
t=
T= 0.01
T= 0.001
Model ayer
Base model
12
Model ayer
8
Sparso model
20
10
60
100
20
10
60
100
20
40
60
100
Ratio of selected experts (%)
Ratio of selected experts (%)
Ratio of selected experts (%)
Figure 18: Distribution of the number of executed experts in each layer for key projections.
22

---

0.1
0.01
0.001
Model ayer
Base model
Model ayer
Sparse model
20
60
100
20
60
100
20
60
100
Ratio of selected experts (%)
Ratio ofiselecter experts (%)
Ratio
selected experts (%)
Figure 19: Distribution of the number of executed experts in each layer for value projections.
0.1
T= 0.01
0.001
Model ayer
Base model
Sparso model
Model ayer
20
60
100
20
40
60
100
20
60
100
Ratio of selected experts (%)
Ratio scletee experts (%)
Ratio
selected experts (%)
Figure 20: Distribution of the number of executed experts in each layer for output projections.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We put a lot of effort to accurately present our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include separate "Limitations" section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We do not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have a dedicated appendix section to thoroughly describe our experimental settings (we also provide the source code).
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification: We provide the source code along with README. Code is easy to run once the environment is up, and generates the plots automatically.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https /nipss.cc/pulli//guu guwde//ddee/eee Cot//ooodeeesiseeeeeeeeee hwtpbbmmsssioooooooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We list the necessary facts in the main paper, and provide all the details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification: We do not have the computational resources to run our main experiments with multiple seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
26

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: We did not gather this information during our work on this paper we were not aware this requirement. Gathering this data now close to impossible. We provide information the type of GPUs used.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tppss:///rrrpps.t/
Answer: [Yes]
Justification:
We went through the guidelines and do not see any ethical issues.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the impact of our work, which we believe is positive.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release new data or new models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We always cite or credit the original authors. We always check the license of the assets we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release the source code. We apply the MIT license code and provide a README description.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer[[NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "ZVOf1MtIWX", "reviewer": "Reviewer_VwY8", "summary": "The paper introduces a method called D2DMoE aimed at enhancing the efficiency of transformer models. D2DMoE implements a dynamic-k routing mechanism that allows the model to select a variable number of experts based on the input. The method leverages the inherent activation sparsity in transformer models to reduce the number of active parameters during inference, leading to significant computational savings—up to 60%—without compromising performance. The approach demonstrates that by converting dense layers into Mixture-of-Experts (MoE) layers, transformer models can achieve better accuracy-sparsity trade-offs, making them more efficient for various NLP and vision tasks.", "questions": "See weaknesses.", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a thorough empirical evaluation across multiple tasks (image classification, text classification, language modeling) and model architectures (ViT, BERT, GPT-2, Gemma). The experiments compare against relevant baselines and demonstrate consistent improvements.", "weaknesses": "In my view, this paper appears to be primarily a repackaged sparse-activated pruning technique stemming from the MoE concept. Several concerns arise:\n\n1. Limited comparison with alternative sparsification methods: The paper predominantly contrasts with MoEfication, neglecting a thorough analysis against other compression or sparsification strategies beyond early-exit techniques. Consequently, the comparative experiments presented lack depth and fail to be fully convincing.\n\n2. Lack of practical acceleration results: The paper only presents theoretical reductions in FLOPs and parameter counts. Without actual inference acceleration results on real hardware (e.g., V100, H100, GTX-4090Ti), it's impossible to assess the practical benefits of the method. The additional overhead from the gating mechanism could potentially negate some of the theoretical gains.\n\n3. Questionable novelty of Dynamic-k gating: The proposed expert selection based on ℓ2-norm of output is indeed more reminiscent of pruning techniques than traditional MoE approaches. This calls into question the novelty of the method when viewed in the context of pruning literature, and makes the comparisons with MoE methods potentially unfair or irrelevant.\n\n4. Limited novelty and inadequate discussion of related work: Many of the proposed operations bear similarities to existing sparse activation pruning methods. The paper fails to adequately discuss these connections, instead focusing on less relevant work. This omission of crucial related work in pruning literature significantly undermines the claimed novelty of the approach.\n\n5. Lack of discussion of some SOTA sparse methods: such as Pruner-Zero: Evolving Symbolic Pruning Metric From Scratch for Large Language Models. ICML2024.\n\n6. Theoretical foundation: The paper lacks a strong theoretical justification for why the proposed methods work better, which becomes even more critical given the concerns about its relationship to existing pruning techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent reviewing our work. We are grateful that the reviewer recognizes that our paper presents a thorough empirical evaluation and compares the proposed method against relevant baselines. Below, we present the responses to the issues listed by the reviewer.\n\n> ​​In my view, this paper appears to be primarily a repackaged sparse-activated pruning technique stemming from the MoE concept.\n\n> Limited comparison with alternative sparsification methods: (...)\n\n> Lack of discussion of some SOTA sparse methods: (...)\n\nWe would like to highlight that our method is not a pruning technique, but a dynamic inference method. Dynamic inference methods (also known as conditional computation or adaptive inference) [8] are a distinct area of research fundamentally different from pruning approaches. We list the exact differences below:\n1) Pruning focuses on model **weights**, while dynamic inference results in sparsity in model **activations**.\n2) During inference, the pruned model **statically** uses the same subset of weights for all inputs. In contrast, dynamic inference methods **dynamically** select an appropriate subset of weights on a per-input basis instead of removing them.\n3) Pruning is usually concerned with **model compression**, while dynamic inference mostly focuses on **inference acceleration**. The two objectives are not always correlated; in particular, unstructured pruning fails to provide any speedups on GPUs in practice due to the lack of proper hardware support for sparse matrix multiplications [1, 2].\n\nIt has already been shown that combining weight sparsity and activation sparsity can lead to higher speedups than with weight sparsity alone [2]. Since conditional computation methods such as MoE can be seen as a form of structured activation sparsity [1], we consider our method and weight pruning as complementary.\n\nCrucially, we highlight that we do explore the relationship of our method to pruning in our paper. In Section 5.6 we perform an experiment that tests our method for compatibility with pruning, where we demonstrate that our method indeed improves the performance of a network pruned by the CoFi structured pruning method [9].\n\n> Questionable novelty of Dynamic-k gating: (...)\n\n> Limited novelty and inadequate discussion of related work: (...)\n\nWe would like to gently disagree with the reviewer in regards to the lack of novelty of our method and point out that other reviewers did not raise similar concerns and even praised the novelty of our work (reviewer uQnr). To the best of our knowledge, we were the first to propose MoE routing based on the norms of the outputs of the experts. Our expert contribution routing is conceptually consistent, significantly improves performance (Figure 6a), and is not dependent on ReLU (Figure 6c). The use of the ℓ2-norm (or any norm) is widespread in the machine learning literature (e.g. adversarial examples [3], knowledge distillation [4], normalization [5]), and we do not see any connection to pruning in particular. \n\nPlease also note that expert contribution routing and dynamic-k gating are separate contributions (Figure 6a). We show that static top-k gating is inappropriate for MoEs converted from dense models (Figure 2b), a setup first proposed by Zhang et al. [6]. Similarly, we are the first to show the existence of this problem, and we propose dynamic-k gating as a remedy - that in our view also has no connection to the pruning literature.\n\nFollowing the discussion with the reviewer, in addition to Section 5.6, we add the discussion on pruning and its relation to our method to the paper to better highlight the differences for the readers.\n\n> Theoretical foundation: (...)\n\nWhile we acknowledge the importance of strong theoretical foundations in machine learning, we focus on empirical evaluation as done in works similar to ours [2,6,7,8]. \n \n> Lack of practical acceleration results: (...)\n\nWe address this concern in the joint response to all reviewers and in the rebuttal PDF in Figures 1 and 2, where we show our wall-clock time measurement results.\n\nWe would like to thank the reviewer again for his time spent on reviewing our work. We hope that our answers resolve the reviewer’s concerns, and we are open to further discussion in case of any further questions. We also kindly ask the reviewer to reassess our work and adjust the score accordingly, considering our answers and the additional rebuttal experiments.\n\n##### References\n[1] Hoefler, Torsten, et al. \"Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks.\" Journal of Machine Learning Research 22.241 (2021): 1-124.\n\n[2] Hunter, Kevin, Lawrence Spracklen, and Subutai Ahmad. \"Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks.\" Neuromorphic Computing and Engineering 2.3 (2022): 034004.\n\n[3] Costa, Joana C., et al. \"How deep learning sees the world: A survey on adversarial attacks & defenses.\" IEEE Access (2024).\n\n[4] Gou, Jianping, et al. \"Knowledge distillation: A survey.\" International Journal of Computer Vision 129.6 (2021): 1789-1819.\n\n[5] Hoffer, Elad, et al. \"Norm matters: efficient and accurate normalization schemes in deep networks.\" Advances in Neural Information Processing Systems 31 (2018).\n\n[6] Zhang, Zhengyan, et al. \"Moefication: Transformer feed-forward layers are mixtures of experts.\" arXiv preprint arXiv:2110.01786 (2021).\n\n[7] Shazeer, Noam, et al. \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.\" arXiv preprint arXiv:1701.06538 (2017).\n\n[8] Han, Yizeng, et al. \"Dynamic neural networks: A survey.\" IEEE Transactions on Pattern Analysis and Machine Intelligence 44.11 (2021): 7436-7456.\n\n[9] Xia, Mengzhou, Zexuan Zhong, and Danqi Chen. \"Structured Pruning Learns Compact and Accurate Models.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2jcfwmcDJ", "reviewer": "Reviewer_uQnr", "summary": "This paper improves over MoEfication with four innovations: (i) enforcing higher activation sparsity; (ii) directly predict the norm of the output of each expert; (iii) dynamic-k expert selection scheme; and (iv) generalization to any standalone linear layer. The resulting method achieves significant improvements in terms of cost-vs-performance trade-offs in text classification, image classification, and language modeling.", "questions": "NA", "limitations": "Please see weakness.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The proposed method is a nice addition of MoEfication, in terms of both performance and generalization (to any standalone linear layer). It makes this kind of method more usable in practice. \n\n- I like the idea of directly predicting the norm so as to achieve dynamic-k expert selection. This dynamic scheme is novel and bridges the fields of dynamic inference and MoE.\n\n- The experiments illustrate that the proposed method is applicable in various settings, including text classification, image classification, and language modeling.", "weaknesses": "- The baseline seems weak. ZTW only receives a few citations and is not published at a top machine learning conference/journal.\n\n- Only FLOPs are reported and it is a poor indicator of practical latency/throughput, especially in the LLM era.\n\n- It is not clear if the proposed method could be combined with other methods to show its broader applicability. In particular, quantization is a very general method and is almost adopted as a default in many practical applications.\n\n- The writing could be improved. For example, more technical details of MoEfication should be included.\n\n- In all figures, the lines are continuous. However, these lines should have been a result of interpolation. The original point values are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort spent on our paper. We are pleased that the reviewer recognizes the novelty, applicability, and performance of our method. Below, we address the issues raised by the reviewer.\n\n> The baseline seems weak. ZTW only receives a few citations and is not published at a top machine learning conference/journal.\n\nZTW [1] was published at NeurIPS 2021, and its extended version [2] was published in Neural Networks in 2023. We apologize for the confusion. We cited the extension believing it is more relevant due to being more recent, but in our new revision we cite both versions of that paper.\n\n> Only FLOPs are reported and it is a poor indicator of practical latency/throughput, especially in the LLM era.\n\nWe address this concern in the joint response to all reviewers and in the rebuttal PDF in Figures 1 and 2, where we show our wall-clock time measurement results.\n\n> It is not clear if the proposed method could be combined with other methods to show its broader applicability. In particular, quantization is a very general method and is almost adopted as a default in many practical applications.\n\nMultiple studies such as [4,5,6,7,8] show that dynamic computation methods combine well with quantization and pruning. In Section 5.6, we demonstrate that our method integrates effectively with the existing structured pruning method CoFi [3]. For reviewers' convenience, we perform a similar experiment with 8-bit and 16-bit dynamic quantization and provide the results in the rebuttal PDF. Our method integrates seamlessly with quantization.\n\n> The writing could be improved. For example, more technical details of MoEfication should be included.\n\nFollowing the reviewer's suggestion, we have included a more detailed description of MoEfication in the paper.\n\n> In all figures, the lines are continuous. However, these lines should have been a result of interpolation. The original point values are missing.\n\nWe changed the plots to add points as suggested by the reviewer.\n\nWe would like to thank the reviewer again for his time spent on reviewing our work. We hope that our answers resolve the reviewer’s concerns, and we are open to further discussion in case of any further questions. We also kindly ask the reviewer to reassess our work and adjust the score accordingly after taking into consideration our answers and the additional experiments from the response PDF.\n\n##### References\n\n[1] \"Zero time waste: Recycling predictions in early exit neural networks.\", Wołczyk et al., NeurIPS2021.\n\n[2] \"Zero time waste in pre-trained early exit neural networks.\", Wójcik et al, Neural Networks 168 (2023).\n\n[3] \"Structured Pruning Learns Compact and Accurate Models.\", Xia et al., ACL2022.\n\n[4] \"Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness.\", Young, arXiv 2023.\n\n[5] \"The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Models.\", Kurtic et al., EMNLP 2022.\n\n[6] “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, Han et al., ICLR2016.\n\n[7] “McQueen: Mixed Precision Quantization of Early Exit Networks”, Saxena et al., BMVC2023.\n\n[8] “Two sparsities are better than one: unlocking the performance benefits of sparse–sparse networks.”, Hunter et al., Neuromorphic Computing and Engineering 2022 Volume 2 Number 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IAZXuXn01n", "reviewer": "Reviewer_R1eJ", "summary": "This paper proposes a method to convert a dense pre-trained transformer network into its sparse counterpart. The approach begins by fine-tuning a pre-trained network to sparsify its activation values. Subsequently, the method clusters the neurons in the MLP layers to form distinct experts and introduces a router network. Finally, the router network is fine-tuned to predict the norm of the output of each expert. Additionally, this work explores the possibility of modifying attention layers and clustering MLP layers using the GLU activation function. The results demonstrate that the proposed method is robust to high sparsity and outperforms baseline model", "questions": "After converting the attention weight matrices, is any router network introduced to determine which head is activated? In the main paper, I only read that the attention matrices are replaced with two-layer networks with comparable parameters and cost. I didn't find any rationale provided for this conversion.", "limitations": "1. Currently, this method is only evaluated on small networks, so it is unknown whether the proposed method can be effectively applied to larger networks.\n2. Although this method leverages sparsity in activations to accelerate inference, it does not reduce the number of parameters, and may even increase them, unlike other methods such as pruning that simultaneously reduce the parameter count.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The computational cost of large models is a significant research problem. This work proposes a promising method that can reduce FLOPs by up to 60% without compromising performance.\n2. This study conducts extensive experiments across different modalities, models, and datasets, making the results convincing.", "weaknesses": "1. This method introduces several hyperparameters during training and inference, which could potentially make it difficult to reproduce and deploy.\n2. I find that Figure 2 does not justify dynamic-k very well, as most layers have a similar percentage of non-zero activations. However, this concern is alleviated after reviewing Figures 12 through 16.\n3. Although this work sparsifies at a coarse granularity, which is beneficial for reducing latency, reporting wall-time would be more convincing than solely reporting FLOPs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort spent on our paper. We are pleased to see that the reviewer recognizes the significance of our work and the robustness of our method across different experimental settings. Below, we answer to the issues raised by the reviewer:\n\n### Weaknesses\n\n> This method introduces several hyperparameters during training and inference, which could potentially make it difficult to reproduce and deploy.\n\nWhile our method introduces several hyperparameters, we show that its performance is quite robust to hyperparameter selection (please see Figures 6b, 6d, and 11 for analysis of the impact of different hyperparameters). We found out that the staged nature of the conversion process significantly simplifies the tuning for both D2DMoE and MoEfication, as it allows us to tune and verify the performance of a stage before proceeding to the next one.\n\n> I find that Figure 2 does not justify dynamic-k very well, as most layers have a similar percentage of non-zero activations. However, this concern is alleviated after reviewing Figures 12 through 16.\n\nWe want to clarify that the *average* (aggregated over all tokens from the entire test set) percentage of non-zero activations does not need to be different between layers for the use of dynamic-k gating to be justified. For a single FFN layer, we are interested in the variance of the number of non-zero activations. We show that this variance is significant (error bars in Figure 2b, top; we also plot the coefficient of variation in Figure 2b, bottom) for every FFN layer, hence executing the same number of experts for each input would be suboptimal.\n\nNote that some layers do exhibit different numbers of average non-zero activations (Figure 2b, top). While one could tune $k$ for each layer separately, it would be a cumbersome process, and to our best knowledge, there are no works in that direction. Our dynamic-k gating allows for an uneven spread of computation throughout the depth of the model without the need for tuning any hyperparameters (Figure 5a).\n\n> Although this work sparsifies at a coarse granularity, which is beneficial for reducing latency, reporting wall-time would be more convincing than solely reporting FLOPs.\n\nWe address this concern in the joint response to all reviewers and in the rebuttal PDF in Figures 1 and 2, where we show our wall-clock time measurement results.\n\n### Questions\n\n> After converting the attention weight matrices, is any router network introduced to determine which head is activated? In the main paper, I only read that the attention matrices are replaced with two-layer networks with comparable parameters and cost. I didn't find any rationale provided for this conversion.\n\nPlease note that we do not introduce routing across the attention heads nor change how attention is calculated. We only replace the projection matrices (W_q, W_k, W_v, W_o). We introduce this modification to extend the computational savings from our method to attention layers without changing the self-attention mechanism itself.\n\n### Limitations\n\n> Currently, this method is only evaluated on small networks, so it is unknown whether the proposed method can be effectively applied to larger networks.\n\nWhile our evaluation is done mostly on small-scale models, please note that we also demonstrate that our method works well with the 2B parameter Gemma model. Our computational budget does not allow for larger models, and as such we leave the exploration of this question for future work. However, since bigger models are more overparameterized and exhibit higher activation sparsity, we do not see any reasons why our method would not scale further. Finally, we see more improvement upon the baseline when D2DMoE is applied to the larger model (Gemma 2B) than when applied to smaller models; this suggests that our method scales better than the baseline (please see the discussion on why in Appendix C).\n\n> Although this method leverages sparsity in activations to accelerate inference, it does not reduce the number of parameters, and may even increase them, unlike other methods such as pruning that simultaneously reduce the parameter count.\n\nWhile this is true, the parameter overhead from our method is very low. We consider model compression methods such as pruning to be complementary to D2DMoE (see Section 5.6).\n\nWe would like to thank the reviewer again for his time spent on reviewing our work. We hope that our answers resolve the reviewer’s concerns, and we are open to further discussion in case of any further questions. We also kindly ask the reviewer to reassess our work and adjust the score accordingly after taking into consideration our answers and the additional experiments from the response PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "erFLUYuB8I", "reviewer": "Reviewer_v3M2", "summary": "The paper presents Dense to Dynamic-k Mixture-of-Experts (D2DMoE), a method to convert a dense model into a MoE model, that exploits the fact that activations in Transformer models are typically very sparse. The sparsity can be further improved by the addition of square Hoyer regularization during a light fine-tuning phase. Since the sparsity pattern changes over examples, the use of a dynamic selection rule is preferred. The strategy used to convert the dense model into a MLP is almost identical to that of Zhang et al. from 2022, but the loss function used to train the router is different. The router in this paper is trained to predict the norm of the output of the corresponding expert (and the experts with the an expected l2 norms over a certain dynamic threshold are selected). Finally, the present work also MoEfies not only the FFN layers as usual, but also all the linear layers in the MHA (for that, each linear layer is converted into a MLP of the same cost, minimizing the l2 error, similar to distillation). \nThe proposed approach is evaluated on different architectures (ViT, BERT, GPT2, Gemma) and benchmarks (including both vision and language). In all cases the proposed approach offeres a better FLOP-quality trade-off than that of Zhang et al. (2022), and achieves a quality similar to the original dense baseline but at significantly fewer FLOPs.", "questions": "See the comments mentioned in the weaknesses.", "limitations": "No particular ethical considerations affecting this work, in my opinion.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well structured and written. All the steps involved in the proposed method are clearly explained in the main paper, or briefly explained first and then thoroughly documented in the appendix. \n- Figure 5a shows that the step done to improve sparsification is clearly useful. Although this doesn't show how the quality of the resulting model is affected, figure 4 shows that the quality of the original dense model is matched, or very nearly matched, with significantly fewer FLOPs (and better FLOP-quality trade-off than the MoEfication proposed by Zhang et al. in 2022).\n- Figure 5b shows that more FLOPs (i.e. experts) are allocated to image regions with higher semantical meaning, as one intuitively would expect. This might not be a strength if one only cares about FLOP vs quality, but it certainly makes the approach more interpretable.", "weaknesses": "- The use of MoEs from scratch is criticized because the training difficulties that these have. However, the training recipes for these have improved significantly in the past two or three years, and are now a component of many state-of-the-art models (e.g. [Mixtral](https://mistral.ai/news/mixtral-of-experts/), [DeepSeekMoE](https://arxiv.org/abs/2401.06066), [Gemini 1.5](https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf)). Thus, I believe that these should definitely be considered as a baseline, otherwise the potential impact of this work is more limited.\n- The proposed method involves many steps. They are well documented (see strengths), but makes the whole approach a bit cumbersome, in my opinion. Alternativelly, I wonder how the dynamic routing, the sparsity-indicuing regularization and other of the components used would perform when training a model from scratch.\n- As usual, one must be carefull when doing only FLOP vs quality comparisons, since the proposed method, especially the dynamic selection of experts, may be hard to efficiently implement on modern hardware (GPUs, TPUs). I couldn't find this discussed anywhere in the paper, so I would appreciate more clarity on this (perhaps even additional plots showing Runtime vs quality, even if it's only on the appendix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for assessing our work and recognizing that our paper is well-written. Below, we address the reviewer's concerns.\n\n> The use of MoEs from scratch is criticized because the training difficulties that these have. However, the training recipes for these have improved significantly in the past two or three years, and are now a component of many state-of-the-art models (e.g. Mixtral, DeepSeekMoE, Gemini 1.5). Thus, I believe that these should definitely be considered as a baseline, otherwise the potential impact of this work is more limited.\n\nMoE models trained from-scratch may at first seem like an appropriate baseline due to similar architecture. However, note that the primary goal of training MoE from-scratch is to scale the parameter count of the model without affecting its *training* cost. In contrast, our aim is to reduce the *inference* cost of existing pre-trained models by skipping redundant computations while preserving the model size. All our baselines start from the same pre-trained model checkpoint, and we assume only a small budget for model adaptation, so we cannot conduct a fair comparison between our method and models suggested by the reviewer.\n\n> The proposed method involves many steps. They are well, documented (see strengths), but makes the whole approach a bit cumbersome, in my opinion. Alternativelly, I wounder how the dynamic routing, the sparsity-indicuing regularization and other of the components used would perform when training a model from scratch.\n\nWhile our method is multi-step, we show the robustness to the change of hyperparameters in Figures 6b, 6d, and 11. We agree that applying our method to training MoE from-scratch is an interesting future work direction. Expert contribution routing could be useful in augmenting the training of gating networks, which often perform similarly to fixed mappings to experts [1]. Moreover, as activation sparsity levels change as training progresses [2], one might investigate the application of dynamic-k gating for faster training. According to our knowledge, nothing similar to our contributions appeared in the context of MoE literature before.\n\n> As usual, one must be carefull when doing only FLOP vs quality comparisons, since the proposed method, especially the dynamic selection of experts, may be hard to efficiently implement on modern hardware (GPUs, TPUs). I couldn't find this discussed anywhere in the paper, so I would appreciate more clarity on this (perhaps even additional plots showing Runtime vs quality, even if it's only on the appendix).\n\nWe address this concern in the joint response to all reviewers and in the rebuttal PDF in Figures 1 and 2, where we show our wall-clock time measurement results.\n\nWe would like to thank the reviewer again for his time spent on reviewing our work. We hope that our answers resolve the reviewer’s concerns, and we are open to further discussion in case of any further questions. We also kindly ask the reviewer to reassess our work and adjust the score accordingly after taking into consideration our answers and the additional experiments from the response PDF.\n\n##### References\n\n[1] Roller, Stephen, Sainbayar Sukhbaatar, and Jason Weston. \"Hash layers for large sparse models.\" Advances in Neural Information Processing Systems 34 (2021): 17555-17566.\n\n[2] Wild, Cody, and Jesper Anderson. \"Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers.\" arXiv preprint arXiv:2407.07848 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9xdxuaIMfg", "reviewer": "Reviewer_85BP", "summary": "This paper proposes a more effective dynamic-k expert selection rule that adjusts the number of executed experts on a per-token basis to reduce the high computational cost of the transformer models. They claim their D2DMoE model outperforms existing approaches.", "questions": "1. Training Overhead: Can you provide more details on the additional training time required for sparsity enforcement and router training? How does this overhead compare to the overall training time of the original dense models?\n2. Real-world Deployment: What steps or considerations are necessary for deploying D2DMoE models in real-world applications, particularly in terms of latency, hardware compatibility, and integration with existing pipelines?\n3. Adversarial Robustness: Have you evaluated the robustness of D2DMoE models against adversarial attacks? If not, do you have any plans to explore this aspect in future work?\n4. Comparison with Pruning Techniques: How does D2DMoE compare with other model pruning or compression techniques in terms of performance, computational savings, and implementation complexity?\n5. Scalability: Can the proposed method be effectively scaled to extremely large models like GPT-3? Are there any specific challenges or considerations when applying D2DMoE to such models?", "limitations": "The paper does not sufficiently address potential challenges in real-world deployment, such as latency variations, hardware compatibility, and integration with existing systems.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation of this paper is good. They try to save up the inference cost of the transformer models.\n2. They perform experiments on several tasks.", "weaknesses": "1. The most important contribution is the dynamic-k routing algorithms. However, the main work is a small modification to MoEfication. The difference of the proposed method with MoE-based methods is limited.\n2. The experiments are not conducted well. This paper lacks a comprehensive comparison with other methods. There exist many MoE-based VIT methods, such as [1, 2]. Furthermore, why does the authors does not provide any performance number in the table? This paper claims they outperforms existing approaches on common NLP and vision tasks, however, only two methods are compared.\n3. The major advantage of D2DMoE compared with MoEfication is saving inference cost. However, the saved cost is quite limited observed from fig. 4.\n\n[1] Mod-squad: Designing mixtures of experts as modular multi-task learners\n[2] M3 vit: Mixture-of-experts vision transformer for efficient multi-task learning with model-accelerator co-design", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time spent on our work. We are glad that the reviewer recognizes the proper motivation of our method and the thoroughness of our experiments. Below, we address the concerns raised by the reviewer.\n\n### Weaknesses\n\n> The most important contribution is the dynamic-k routing algorithms. However, the main work is a small modification to MoEfication. The difference of the proposed method with MoE-based methods is limited.\n\nThe contributions of our work are not only dynamic-k gating, but also the sparsification phase, expert contribution routing, and extension to MHA projections. All these contributions improve upon the baseline method (Figure 6a). Figures 2a and 6b show the impact of the sparsification phase on the model performance. Regression routing significantly improves the performance on Gemma2B, even when applied in isolation to MoEfication (see Appendix C), while the baseline method performs very badly in this setting. This indicates our method is more robust when scaling to larger models (note how our routing significantly differs from the MoEfication objective, as explained in Appendix A).\n\n> The major advantage of D2DMoE compared with MoEfication is saving inference cost. However, the saved cost is quite limited observed from fig. 4.\n\nWe respectfully disagree with the reviewer’s claim that the cost savings with our method are insignificant. Note how our method matches the performance of the dense model using around 60% (ViT), 30% (BERT), 80% (GPT-2) or 70% (Gemma2B) of its computation, significantly reducing the cost when compared to the MoEfication baseline.\n\n> The experiments are not conducted well. This paper lacks a comprehensive comparison with other methods. There exist many MoE-based VIT methods, such as [1, 2].\n\n> This paper claims they outperforms existing approaches on common NLP and vision tasks, however, only two methods are compared. \n\nThe references provided by the reviewer are MoE-based methods for multi-task learning, and therefore are inapplicable in our setting. Our setting assumes a pre-trained dense model checkpoint, and MoEfication is the only MoE-based method fulfilling this condition that we are aware of.\n\nTo strengthen the evaluation of our method, we implement dynamic inference method A-ViT [1] and include it as an additional baseline. A-ViT is spatially-adaptive in terms of its computational effort spent on an input image, similarly to our method (Figure 5a). Moreover, it also starts with a pre-trained dense model checkpoint, and thus is applicable in our setting. We provide the results of the ViT experiment updated with A-ViT baseline in Figure 2 in the rebuttal PDF. Our method outperforms the A-ViT baseline.\n\n> Furthermore, why does the authors does not provide any performance number in the table?\n\nWe had considered plots as a more informative way to show the performance of our method. However, as per the reviewer’s suggestion, we have added them to the appendix of our current revision. \n\n### Questions \n\n> Training Overhead:  (...)\n\nWe provide those numbers in multiple places in the paper (e.g. L101-102; L193-194; Appendix H). Overall, this overhead is less than 1% of the overall training cost.\n\n> Adversarial Robustness:  (...)\n\nThe adversarial robustness of dynamic inference methods is an issue that warrants an in-depth investigation in the form of a separate paper, as in Hong et al. [2]. We consider it too complex to fit within the scope of our work, but it’s an interesting and understudied future work direction.\n\n> Comparison with Pruning Techniques:  (...)\n\nD2DMoE is not a pruning or compression method, and therefore we do not consider such methods as suitable for direct comparison. We rather see model compression techniques as a complementary approach that can be applied alongside our method (see Section 5.6).\n\n> Scalability: (...)\n\nWe do not see any specific issues that would prevent the application of our method to larger models, aside from the limited computational resources that prevent us from running such experiments.\n\n> The paper does not sufficiently address potential challenges in real-world deployment, such as latency variations, hardware compatibility, and integration with existing systems.\n\n> Real-world Deployment: (...)\n\nWe did not list deployment limitations as they are not specific to our method but are general across dynamic inference methods. However, we followed the reviewer's suggestion and listed them in our paper to make sure that the reader understands them:\n1. In contrast to static models, latency depends on the sample. In our case, it is upper-bounded by the latency of executing the forward pass with every token being routed to every expert.\n2. The software has to support dynamic computational graphs.\n3. Hardware compatibility is dependent on the method. We provide the wall-clock time measurement results for our implementation in the rebuttal PDF.\n\nWe would like to thank the reviewer again for his time spent on reviewing our work. We hope that our answers resolve the reviewer’s concerns, and we are open to further discussion in case of any further questions. We also kindly ask the reviewer to reassess our work and adjust the score accordingly after taking into consideration our answers and the additional experiments from the response PDF.\n\n##### References\n\n[1] Yin, Hongxu, et al. \"A-vit: Adaptive tokens for efficient vision transformer.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Hong, Sanghyun, et al. \"A panda? no, it's a sloth: Slowdown attacks on adaptive multi-exit neural network inference.\" arXiv preprint arXiv:2010.02432 (2020).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the time and effort spent on our work and their valuable comments that helped us improve our work. We appreciate that our work has been praised by the reviewers for its thorough empirical evaluation (reviewers VwY8, R1eJ), novelty (reviewer uQnr), the generality of our method (reviewer uQnr), good empirical results (reviewers VwY8, uQnr, and R1eJ), and the paper being well-written (v3M2).

## Rebuttal summary
We highlight the main points that changed in our new revision thanks to the valuable feedback that we received from the reviewers:
- Reviewers VwY8, uQnr, R1eJ, and v3M2 requested wall-clock time measurements of the performance of our method. To strengthen our work we provide an efficient implementation of D2DMoE and show its performance in the rebuttal PDF. In Figure 1, we show that FLOPs for D2DMoE correspond to actual latency and that our method can significantly speed up the inference (63% reduction of wall-clock time with a negligible performance drop). Moreover, in Figure 2 we show end-to-end accuracy vs latency plots that show D2DMoE can reduce the end-to-end computation by around 25% time without any performance drop. We consider our efficient implementation of D2DMoE as a strong technical contribution of our work.
- Reviewer 85BP requested additional baselines. As a result, we add A-ViT [1] into our comparison. A-ViT is a dynamic inference method that saves compute by dropping tokens. The results are presented in Figure 3 of the rebuttal PDF. D2DMoE outperforms this baseline as well.
- Reviewer uQnr asked about the integration of D2DMoE with quantization. Therefore, in Figure 4 of the rebuttal PDF, we provide an additional experiment with quantization. We show that our method integrates seamlessly with 16-bit and 8-bit quantization, further proving the robustness of our method. 

Below, we elaborate on wall-clock measurements and our rationale for using FLOPs in more detail.

## Wall-clock time measurement results

We implement the forward pass of our MoE module using GPU kernels written in Triton [2] and employ several optimizations for our implementation, including an efficient memory access pattern, kernel fusion, and configuration auto-tuning. As suggested by Tan et al. [4], our implementation also avoids unnecessary copies when grouping tokens.

We verify the performance of our implementation for a single FFN MoE layer in isolation by feeding it with synthetic random data. We measure the execution time of our layer for a range of different numbers of experts executed on average and compare it to the corresponding MLP module. The results, presented in Figure 1 of the rebuttal PDF, show that our implementation has almost no overhead, and that FLOPs for D2DMoE perfectly correlate with the wall-clock time of execution.

Furthermore, we want to ensure that these gains translate to real-world usage speedups. We therefore use our new implementation in a ViT-B-converted D2DMoE model and measure the average sample processing time on the entire ImageNet-1k test set. As suggested by reviewer v3M2, in Figure 2 of the rebuttal PDF we plot the accuracy vs wall-clock time trade-off of our method. While a small overhead when compared to the ViT-B baseline is visible, our method still allows for significant savings. This shows the speed-up potential of D2DMoE in practice.

We have added the above wall-clock measurement experiments to our paper, along with a detailed description of our Triton implementation and its source code.

## FLOPs rationale
Finally, we provide our reasoning for using FLOPs as the main metric in our work instead of using wall-clock time measurements:
1. Wall-clock measurements are heavily affected by the choice of device hardware. Even if the device model (i.e. GPU model) is the same, the environment (cooling efficiency, temperature, package versions, I/O load by other users) may be different and can significantly affect the result. As such, wall-clock time does not allow for reliable comparisons to results presented in other works.
2. Implementing efficient GPU kernels is often non-trivial, and entire papers have been published with their sole contribution being an efficient implementation of MoE [2, 3]. Latencies obtained with poorly implemented and slow kernels could suggest to the reader that the method is GPU-unfriendly, even if it is a problem of only that specific implementation. We emphasize the rapid progress in research on efficient MoE implementations, exemplified by the recent work of Tan et al. [4], which demonstrates significant advancements compared to Gale et al. [3].
3. Finally, as discussed in "The hardware lottery" paper by Hooker, S. [5], compatibility with the current hardware should not overshadow the ideas proposed in research papers. FLOPs, as a metric for computational complexity, are independent of hardware choice and can be used to compare algorithms rather than their specific implementations.

We thank the reviewers for their valuable insights. We hope that our response and additional experiments fully address the concerns of the reviewers, and we are open to further discussion should reviewers have any additional inquiries. 

##### References
[1] Yin, Hongxu, et al. "A-vit: Adaptive tokens for efficient vision transformer." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.

[2] Tillet, Philippe, Hsiang-Tsung Kung, and David Cox. "Triton: an intermediate language and compiler for tiled neural network computations." Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. 2019.

[3] Gale, Trevor, et al. "Megablocks: Efficient sparse training with mixture-
of-experts." Proceedings of Machine Learning and Systems 5 (2023).

[4] Tan, Shawn, et al. "Scattered Mixture-of-Experts Implementation." arXiv preprint arXiv:2403.08245 (2024).

[5] Hooker, Sara. "The hardware lottery." Communications of the ACM 64.12 (2021): 58-65.

Meta-review: The submission proposes a dynamic-k routing mechanism to select the number of experts in a MoE setting.  The submission initially received split reviews, which improved during the rebuttal process.  All but one reviewer now recommend acceptance.  The remaining reviewer has concerns about novelty, noting other papers that have used a dynamic-k routing mechanism in some way.  The submission also provides an interesting practical implementation and benchmarking results that appear to be of value in themselves.  The authors are strongly recommended to include citations to and discussion about the references pointed out by Reviewer 85BP - this will strengthen the correctness and relevance of the paper.
Decision: Accept (poster)