ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Authors: Shangqian Gao
Ting Hua
Reza Shirkavand
Chi-Heng Lin
Zheng Tang
Zhengao Li
Longge Yuan
Fangyi Li
Zeyu Zhang
Alireza Ganjdanesh
Qian Lou
Jie Xu
Yen-Chang Hsu
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their high computational demands. Traditional pruning methods reduce these costs by permanently removing parameters, which inevitably leads to performance degradation. To mitigate this issue, we propose ToMoE, a method that transforms dense LLMs into Mixture-of-Experts (MoE) models by uncovering experts inherently present within dense models, without requiring any weight updates. ToMoE leverages dynamic structural pruning to unify expert construction and router training in a single stage, achieving consistently strong performance. Remarkably, even without fine-tuning \revise{the model weights}, ToMoE consistently outperforms state-of-the-art pruning and MoE techniques across Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5 models. The code for this paper is available at https://github.com/gaosh/ToMoE.

Published in Transactions on Machine Learning Research (01/2026)
ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
Shangqian Gao
sg24bi
Deportment of Computer Science, Florida State University
Hua Ting
thwa edu
Department of Computer Science and Engineering, University of Notre Dame
Reza Shirkavand
rezashku@cs.umg und. edu
Deportment of Computer Science, University of Moryland, College Park
Chi-Heng Lin Samsung Research America
chiheng..nn@saamnnnnen www../.iis@amsssoommmm
Zheng Tang Samsung Research America
Begengttnn@@samuung.oom com
Zhengao Li Department of Computer Science, Florida State Universily
x123 edu
Longge Yuan Deportment of Computer Science, Florida State University
1y23a edu
Fangyi Li
fongyili upenn. edu
School of Engineering and Applied Science, University of Pennsylvania
Zeyu Zhang Amozon AGT
zeyzhan seyzhan@amazon.com
Alireza Ganjdanesh
aliganj@umd. edu
Department of Computer Science, University of Moryland, College Park
Lou Qian
qion.lou@ucf edu
Deportment of Computer Science, University of Central Florida
Xu Jie
rujie@uul..dd edu
Department of Health Outcomes and Biomedical Informatics, University of Florida
Yen-Chang Hsu
baanchnn.hsuussmmmann com
Samsung Research America
Reviewed on OpenReview: https: // o/perrevee.. formm id= RFHg446666
Abstract
Large Language Models (LLMs) demonstrate remarkable capabilities but face deployment challenges due to their high computational demands. Traditional pruning methods reduce these costs by permanently removing parameters, which inevitably leads to performance degradation. To mitigate this issue, we propose ToMoE, method that transforms dense LLMs into Mixture-of-Exper (MoE) models by uncovering experts inherently present
"Corresponding author.

---

Published in Transactions on Machine Learning Research (01/2026)
within dense models, without requiring any weight updates. ToMoE leverages dynamic structural pruning to unily expert construction and router training in single stage, achieving consistently strong performance. Remarkably, even without fine-tuning the model weights, ToMoE consistently outperforms state-of-the-art pruning and MoE techniques across Phi- 2, LLaMA-2, LLaMA-3, and Qwen-2.5 models. The code for this paper is available at https://g thtn:b////tubb.....mm thub. com/ www/ gaosh/ToMoE
1
Introduction
Although LLMs demonstrate remarkable capacity to perform diverse tasks Brown 2020 Kenton & Toutanova
2019 Raffel et al.) 2020; OpenAl
2022 Anthropic
2023
Dong et al.) 2022 Radford et al. 2019: Kaplan
et al. 2020), their huge model size often limits their usability on devices with limited resources. As a result,
considerable efforts (Ma et al. 2023 Ashkboos et al. 2024 Frantar et al., 2022) are focused on minimizing
the computational and memory costs of these models. Structural pruning (Ma et al.. 2023) has emerged as a promising solution this challenge because, unlike unstructured pruning, it achieves compression without the need for specialized implementations. However, the problem with structural pruning methods is that they will substantially reduce the model capacity, resulting in an obvious performance gap compared to the dense model. The fine-tuning cost for even partially recovering this gap is tremendous.
To achieve a better trade-off between the number of parameters and performance, sparse Mixture of Experts (MoE) models (Shazeer et al. 2017 Lepikhin et al. 2021b are designed to activate only a subset of the model's parameters, corresponding to the selected exports. Recently proposed MoE models, such as DeepseckMoE (Dai et al. 20 demonstrated that they can match the performance of dense models with a similar total parameter count while using small number of active parameters. Following this lead, transforming dense models into MoE models could offer promising approach to bridging the performance gap left by structural pruning methods. Unlike prior efforts to construct MoE models from dense models (Zhang et al.) 2022 Lee et al.
2024b Zhu et al. 2024), our findings reveal that MoF inherently exists within dense models and can be uncovered without updating model weights (continue pretraining). Specifically, we show that these experts can be identified through dynamic structural pruning. These results represent a novel contribution that has not been demonstrated in previous studies.
The core idea of MoE models is conditional computation, where experts are dynamically selected based input tokens. This concept aligns closely with dynamic pruning methods (Gao el al. 2019), which make pruning decisions given imput features. Leveraging this connection, we propose to construct MoE models from dense models by using dynamic structural pruning. Specifically, for Multi-Hoad sclf-Attention (MHA) layers, we apply top-K routing and static pruning for compression, while for MLP layers, we transform them into MoF. layers using top-1 expert routing. The routing mechanism learned for dynamic structural pruning can be directly applied to serve as the routing module for MoE layers. With differentiable discrete operations, the MoE conversion process can be formulated as differentiable dynamic pruming problem. With this formulation, can efficiently convert dense model to an MoF model at cost similar to or lower than regular structural pruming methods. The comparison between our method, static pruning, and the original LLM is shown in Fig. [
Built upon the above findings and techniques, we proposed ToMoE to effectively convert dense LLMS to MoE. models with dynamic pruning. The contributions of this work can be summarized as follows:
We introduce novel approach to convert dense models into MoE models through dynamic pruning. Specifically, implement top-K routing and static pruning for MHA layers along the head dimension and top-1 routing for MLP layers across the learned experts. This formulation ensures sparse and efficient computation while retaining model capacity.
Dense-to-MoE Conversion Through Dynamic Pruning:
The proposed method involves jointly optimizing routing modules and expert configurations by solving a regularized optimization problem. Our approach leverages differentiable operations to enable efficient. and flexible MoF constructions.
Joint Optimization for Routing and Exports:
2

---

Published in Transactions on Machine Learning Research (01/2020)
LiMe
LLMe
LiMe
LLMa
LUM
Vise
eggnt
Wile
.011
orgin
(a) Original LLMs
(b) LLMs with structural pruning.
(c) LLMs with dynamic pruning for MoE.
Figure 1: (a): The original LLM uses all parameters to process all the input text. (b): The static pruning for LLMs permanently removes model parameters, and the rest of the parameters are used to process all the input text. Our approach (c): L.Mss with dynamic pruning use different sub-networks (illustrated by different colors) to process different tokens. We incorporate MoE to achieve a fixed expected budget in inference.
Consistent Performance Improvements: Our method consistently outperforms ttateoof-hheaat structural pruning and MoE construction techniques various tasks while training only the router, without fine-tuning the model weights. This performance improvement is demonstrated across widcly used public models such as Phi-2, LLaMA-2, LLaMA-3, and Qwen-2.5.
Detailed Analysis: We extensively analyze the resulting model from ToMoE across multiple perspectives, including parameter allocation, router behavior, and the ablation of different design components. We hope these analyses provide valuable insights and guidance for future research in this area.
2
Related Works
Pruning:
Structural pruning Li et al. 2017: Kurtic et al. 2022 Ma et al. 2023 is attractive technique
for real-world deployments since it removes redundant parameters to reduce model size without requiring
specialized implementations. Structural pruning methods fall into two main categories: static pruning (A (Awar
cet al. 2017 Molchanov et al. 2019 Fang et al. 2023 and dynamic pruning [Gao et al. 2019 Chen et al. 2020
Anagnostidis et al. 2023 Dong et al. 2024a). Static pruning removes parameters based on input-agnostic
importance metrics. For example, Prunerr (Ma et al., 2023) eliminates non-essential coupled structures
using gradient-based criteria. The problem with structural pruning is that it often creates noticeable performance gap relative to dense modcls (Ma et al. 2023 Ashkboos al. 2024). In contrast, dynamic pruming removes weights based on input-dependent metries. Early allempls for dynamic pruning (Gao el al., 2019 Chen et al.) 2020) focus on Convolutional Neural Networks, where channels are selectively activated for input samples. Recent works, such as D-LLM (Wang et al. 2024), incorporate the concept of conditional computation into LLMs by selectively skipping layers based on input tokens. The problem with dynamic pruning methods that they do not have fixed budget given different inputs, which creates problems when serving LLMs in mini-batch setting or in the prefilling stage. Our method, on the other hand, converts the dense LLM to sparse MoE model with fixed per-token budget.
Another line of research applies contextual sparsity for LLMs (Liu et al. 2023 Zheng et al. 2024 Lee et al. 2024ah, where neurons are selectively activated given certain conditions. Although there are some promising results, they are generally more difficult to achieve better inference efficiency due to their irregular sparsity patterns. In contrast, MoF. models have more comprehensive support from the system side, making them a popular choice for scaling up the model. Thus, our method mainly focuses on converting dense models MoE models.
MoE: Sparse of (MoE) models improve upon pure structural pruning by maintaining even enhancing model capacity without proportional increase in computational cost. For instance, Sparscly-Gated MoE (Shazeer el al.) 2017) employs a trainable gating network to select a small subset of experts for each input, enabling the model to scale to thousands of experts efficiently (Lepikhin et al. 2021b). More recent methods like DeepSockMoE (Dai et al. 2024) further address expert specialization, matching denso-model performance with similar number of activated parameters. Previous methods constructing MoE from the
dense model (Zhang et al. 2022 Lee et al. 2024b Zhu et al. 2024) separate the expert construction and
3

---

Published in Transactions on Machine Learning Research (01/2026)
KIKA with Tap-k Bootnng
MLP with Mut
Recter
Eapert
expert By
Modi-lted Aftetion
Projp
Selection matricns for sach roben
Selection mutrices For eash expert
Expert
Ws
Eapert
Wa
W
Original ML Weighi
Figure 2: ToMoE uses top-1 routing for MLP layers, and static and dynamic pruning along the head dimension for MHA layers.
router training into two distinct stages, often leading to sub-optimal performance. In contrast, our method integrates expert construction directly into the pruning process, treating as a unified step with router learning and thereby largely improving the performance without fine-tuning.
3 ToMoE
Most recent LLMs, like GPT (Radford et al. 2018), LLaMA Touvron et al. 2023a ) etc., adapt decoder-only
architectures and thus our method focuses on decoder-only architectures. A typical decoder block consists of Multi-Head Attention (MHA) and Multi-Layer Perceptron (MLP) layers. For clarity, we denote the sequence length by T, the hidden dimension by d, the MLP intermediate dimension by dmidd and the number of attention heads by H.
To reduce the computational costs of the decoder-only architecture, we propose to convert the original model into Mož models. For MHA layers, utilize top-K routing and static pruning along the head dimension H. Top-K routing and static pruning for MHA layers ensure that, during prefilling or model serving, all tokens maintain the same head dimension, enabling parallel processing. For MLP layers, our approach transforms them into MoE layers along the MLP middle dimension dinid and employs top-1 routing. A key distinction between our method and previous dynamic pruning approaches is that the converted model maintains consistent computational costs for all inputs. This property could be crucial for efficient processing.
3.1 Expert Embeddings
Inspired by the recent success of using hypernctworks (Ha et al. 2016 Ganjdanesh et al. 2024 Gao et al. 2024 to generate pruning decisions, adopt hypernetwork to generate expert embeddings:
Ean = HN(z).
(1)
where is the input to the hypernetwork drawn from random distribution, and Ean [E,, contains embeddings for all layers and E, N-x,, where N is the number of experts and d, is the expert embedding dimension. Each embedding E,,; will then be used to generate the configurations of experts. The purpose of having the hypernetwork to generate Ean is to introduce inter-layer dependencies across different layers and operations. This design has been shown to accelerate the learning process in practice (Gao et al. 2024 More details are given the Appendix A
E,. Ez]
3.2
Expert Construction
In this section, we will talk about how to construct experts from MLP layers. In decoder layer, the formulation of MLP is: fuur(X) oXXW) @ (XWu)Wn, where matrices Wo Bdddmdd WG E Rddd
4

---

Published in Transactions on Machine Learning Research (01/2026)
and WD E Rdmiadd denote up, gated, and down projection matrices. In addition, denotes nonlinear activation functions and @ denotes the Hadamard product (element-wise product).
Assume the target is to use N experts, under the setting of structural pruning, each expert can be represented by:
(x, =X) O (X.)s WD,
(2)
where = 1,... N, and S; = Diag(s:) (s; E RRdmis S; E Rddid Xdmis) ) is a binary diagonal matrix selecting a subset of weight vectors for the ith expert. X, is the tth token, which is assumed to be routed to the ith expert. Once each expert is formulated, its configuration is learned as follows:
s = STT-GSig(Proo) (GE)), G = (TT-GSSmaxRRouer(X))
(3)
where E RTxN is the output of the router module, E (1 omitted for clarity) is the expert embeddings, ProjD Rd + Rdmi is projection module project the latent embedding the MLP middle dimension, Router()) : TRd + R.V is the router module that maps the inputs to N-dimensional routing score vector for expert selection, and ST-GSig and ST-GSmax are
Gumbel-Sigmoid and Gumbel-Softmax functions respectively (Jang et al. 2016). Under this setting, s, will contain retained positions (represented by 1) for the ith expert, and G contains one-hot routing decisions for tokens in X.
3.3 MHA top-K Routing
An MHA layer can be represented as [M) = -Ee as (XXXX.))eeXWW... k.))) XW Wo,,, where
W q. W Kis V. E Rdx# #, Wo, IR # xd are the query, key, value, and output matrices for each attention head, and X RTxd is the input hidden states. and as denote positional embedding and the softmax function.
For MHA layers, we perform two kinds of pruming: dynamic top-K pruning and static pruning, both along the head dimension. Justifications regarding the design choice are provided in the Appendix Like ML.P layers, we also insert selection matrices:
fMHA (X,) Σ [o, (d(X. (X, (X.Wk,)) X,
x. W.W S, Wo...
(4)
where So. Sy E * are selection matrices. So is the shared selection matrix for static pruning of query and key matricos, while S, the token-specific selection matrix for the value and output matricos of the 1-th token. We apply the same selection matrix across all heads, ensuring that all heads have the same head dimensions at inference time. To generate the selection matrix, we calculate its diagonal vector 8 as:
1 N 1TE)),
s, ST-GGig(Proo) MMA
MMA (X.) (Proj"
(5)
where 1 RV is all-one vector, 1 1 reprosents the average expert embedding of size de, ProjMMa Rd R is a projection module to map the latent embedding to the head dimension and Proje MHHA Rd + Rd is also a projection module to project input tokens to the space of expert emboddings, and ST-GSig is defined in Sec. 3.2 When 0, we initialize X = 0, and set So ST-GSig(Projjjp) (+ (11 E)), since is input. independent.
During training, the number of ones in vary freely. After training is complete, compute K round() T ΣT (t,))) for subset of tokens and use it for top-K routing during inference. Note that the k in top- K for sp and s; (t 1) can be different, allowing for larger flexibility. Also, note that must follow specific structural constraints to be compatible with the position embedding e(.), and more details can be found in Appendix A.2
st,;))
3.4
Regularizations for MoE Constructions
In Sec. 3.2 and Soc. 3.3 we briefly introduced the design space for constructing MoE models using dynamic structural pruning. In this subsection, we will introduce regularizations customized to the characteristics of MoE, models.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Union of Experts Regularization. An ideal sparse MoE model converted from dense model should maximize parameter utilization, which means that the total mumber of parameters in the MoF model should closely approximate the dense model. To add this regularization to our learning process, we push the union of experts to closer to the original model. More specifically, we use MHA layers as an example:
T
T II 11 11-8)) s:),
U ==1
8:
(6)
where V is the union operator, and is the union of all kept positions for each token. For MLP layers, it can be calculated similarly. We then push
E" [
(Ju) represents the size of u) to 1:
u 1),
Rv L
SToR
where (r) can be any regression loss functions, and we will choose freg later.
Parameter Rogularization. For a sparse MoE model, we also need control the number of active parameters given the provided budget. To achieve this goal, we can directly penalize the maximum width across different experts. We choose the maximum width over experts instead of the mean, median, other alternatives because the maximum provides precise control over the upper bound of the number of active parameters.
Denote the width of a layer as di, where * E MHA). For MLP layers, it can be calculated by d, MLP = max((11ma)), where Iama E Rddid an all-one vector of size dmidd slmid produces the width of all experts, and d, MLP represents the maximum width across all experts. The width of MHA layers can be caleulated similarly. Based on di, we can calculate the number of active parameters in the model T(dMaE), where dM.E= = [di,... . dd To push the number of active parameters to predefined rate p, the following objective is applied:
Rp Sreg (T(dMR)), total).
where Trotal is the total number of parameters, and E (0, 1] represents the ratios of the active paramcters. For frog in Eq. g and Eq. 7 the following function frog is used:
freg(z, y) log(max(z;, v)/ min(x, ()))))
Load Balancing Regularization. When determining the configurations of experts, we also apply the load balancing regularization to encourage balanced load across experts (Lepikhin et al. 2021a Fedus et al. 2022 The load balancing loss from the Switch Transformer (Fedus et al. 2022) is adopted:
RL N
R.. 1=1
where F; = ΣT, 1(Gr,, 1). The indicator function 1(-) returns 1 if the condition is true and 0 otherwise. F; represents the fraction tokens assigned to the i-th expert. P; = = Σ GSmax(Gr.)) where G Router(X) is the router output before ST-GSmax. P. is the fraction of the router probability allocated for the i-th expert. Rr. will encourages uniform routing across different experts as shown in (Fedus et al.] (022). The combination of Eq. g and Eq. g creates an interesting phenomenon where they encourage uniform allocation of width among experts.
3.5 Learning to Construct MoEs
Based on the aforementioned techniques, MoEs can be constructed from the dense LLMs by training router parameters, projection parameters, and hypernetwork parameters, while keeping all original model parameters frozen. This approach enables the rapid construction of an effective MoE. model with resource budget.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Perplexity comparisons of structured pruning methods and ToMoE for LLaMA-2 7B and 13B models on WikiText-2.
LLaMA-2 7B (ppl: 5.12
LLaMA-2 -2 13B (ppl: 4.57
Method
70%
60%
50%
70%
60%
50%
MLM-Pr Ma
2023
13.56
17.90
31.05
12.19
19.56
32.20
LLM Surgoon
Von der Ouderan ml.
2024
7.83
10:39
15.38
8:21
7.25
9.43
ShortGPT
Men et al
2024
33.21
71.04
268. 11
30,48
48.83
187 187.23
SLEB
(oog at al
2024
11.23
29.10
103.38
8.24
11.76
27.67
SloeGPT (Ashkhons
2024
10.47
15.19
24.82
8.08
12.56
20.57
ModeCITT Lin
2124
7.51
8.41
11.88
6.10
6.95
8.95
DISP-LLM Gao
2021
6.85
8.11
5.77
ToMoE (ours)
7.17
comparable that of structural pruning. The overall framework of our method shown Fig. 2 and the corresponding training objective function can be formulated as:
min Ban, f(x)) + aRp + BRU + RRL.
(10)
where = HOHN Router Piroj-MHA OProf-MPP],
OHN trainable parameters for the hypernetwork in Eq. Router and Prroj-MIP are trainable pa- rameters for the router and the project module in Eq. Proj-MHA is the trainable parameters of the projection modules given in Eq. 3 Rp, Ru. and RL are regularization terms defined in Sec. 3.4 And a,
Table 2: Comparisons with semi-structured pruning on LLaMA-2.
Method
Structure
50% (7B)
50%
(13B)
SparseGPT (2:1)
Semi-structured
10.17
8.32
Wanda (2:4)
Seni-structured
11.02
8.27
8, and are hyperparameters to control the strength of these regularization terms. Here,f represents the
Pruner Zero (2:4)
Semi-structured
10.52
7.41
ToMoE (ours)
Structured
8.36
6.78
original dense model, and f' the model equipped with our designed modules for MoE construction, Under this setting, we use (();-) to calculate the KI. divergence between the logits of and f", which used as the guidance to preserve the capacity of the dense model (Hinton et al. 2015). We also found that using the KL divergence alone can lead to the best performance, and this observation complies with the experimental setup in (Muralidharan et al.) 2024h. Also, note that we perform in-place knowledge distillation since the original model weights are frozen. Thus, the knowledge distillation process does not introduce overheads in terms of GPU memory.
After learning how to construct the MoE, we convert the MLP layer pruning the MLP layer, save } E for MHA layers the bias of the Projø and drop Projøue and convert Eq. 5 into Top-K routing function as well use So for pruning We and W Our construction also enables converting the MoF. back into pseudo-MoF model. The MoE model and the pseudo-MoF model are equivalent, and more details can be found in Appendix B.3
experts with shared weights. After
4 Experiments
4.1
Settings
Models. Our ToMoE method evaluated using several LLMs with decoder blocks. Specifically, we choose the
following models: LLaMA-2 Touvron et al. 2023b): LLaMA-2 7B and LLaMA-2 13B; LLaMA-3 8B (Dubey
al. 2024); ; Phi-2 (Javaheripi et al., 2023 2.5 (Yang et al.) 2024): Qwen-2.5 7B and Qwen-2.5 14B.
Results for LLaMA-2 13B and Qwen-2.5 14B are presented in the A Appendix D
Implementations. ToMoE is implemented by Pytorch (Paskke et al. 2019 and Hugging Face transformer library (Wolf et al. 2020). The model weights are frozen when training the modules with learnable parameters θ Obj. 10 We use the AdamW (Loshchilov & Huttor (019)) optimizer to optimize O, which trained for 10,000 iterations for all models. For all experiments, we set 16, 2.0, and y 1.0, where a, 3, and 7 are defined in Obj- T0 Without specific descriptions, the number of experts for ToMoF is across all
7

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Zero-shot task performance of compressed LLaMA-2 7B, LLaMA-3 8B, Qwen- 7B.
ARC-c
PIQA
WinoG.
HrllaS.
Model
Active Parameters
Method
Averauge
Aco-norm
Acc-norm
acn-norm
ano-norm
100%
Dense
74.58
46.25
79.11
69:06
75.99
60.00
ShortGPT
al.
M024
60.46
43.67
SlieeGPT
Ashichoos
34.80
LLaMA-2 7B
LLM Surgoon
Viu der Diberau
0024
60%
ModeGPT
Lb
20724
49.45
30.03
64.96
61.96
53.01
51.88
ModeGGPT Alpaca
2024
59.76
31.73
70.35
64.
58.63
57.58
ToMoE (Ours)
63.64
38.74
72.85
62.51
65.84
60.72
100%
Dense
17.69
50.58
80.63
72.69
79.16
72.75
ShoetGPT-Alpe
Men al
2024
38.13
31.40
60.94
54.22
31.02
43.24
LLaMA-3
78%
BiceGGT-AAlpnn
Asikkoos
2012
11:44
29.27
57.56
58.48
41.06
46.17
ModeGPT-AIpp
Jin al. 21724
67:05
41.13
75.52
69.61
66.49
63.99
70%
ToMoE (Ours)
71.55
44.71
76.28
68.98
71.84
66.67
60%
ToMoE (Ours)
65.87
11.64
73.61
63.30
66.42
69.11
100%
Denze
79:42
50.17
79.54
71.35
78.36
71.77
DISP-LLM Alpaca (aoo el al. 2024
66.35
34.22
70.29
53.50
55.00
53.69
ToMoE (Ours)
61.83
36.77
71.82
57.70
39.55
57.53
5006
ToMoE N=16 (Ours)
64.81
41.72
73.45
58.45
61.06
59.90
Qwea--2.
ToMoE N-24 (Ours)
64.09
39.85
72.96
68.56
63.21
09.85
DISP-LLM Alpaca (aoo
27:99
53.43
44.11
ToMoE N (Ours)
55.49
40%
(Ours)
54.36
TuMoE (Ours)
57.85
(a) Loss C
(b) Loss Rp
(c) Loss Ru
(d) Loss Ry
Figure 3: The training dynamics give different ratios of active parameters the Qwen-2.5 7B model.
settings. Depending on the size of the base model, to NVIDIA A100 GPUs are used to train More implementation details can be found Appendix
Datasets. Two training scttings provided for all modules with learnable parameters Q: (1) using WikiText Merity et al. (2016). and (2) using mixed dataset comprising WikiText, Alpaca Taori et al. 2023
and Code-Alpaca Chaudhary 2023) (mixing ratio: 1:1:1). Based on our observation, ToMoE benefits from diverse mixture of datasets to effectively construct experts. Following previous methods (Ashiboos et al.
2024 Gao et al.] 2024). our method and other methods are evaluated on five well-known zero-shot tasks:
PIQA Bisk et. al. 20200; WinoGrande (Sakaguchi al. 202T); HellaSwag (Zellers et al.] 2019); ARC-e and
ARC-c Clark et al. 2018). We further evaluate our method on the following tasks and configurations
ensure consistency with comparison baselines: 32-shot BoolQ (Clark et al. 2019), SeiQ (Johannes Welbl 2017h, 5-shot WinoGrande, 25-shot ARC-c, 10-shot HellaSwag, TruthfulQA Lin et al. (2022), and 5-shot
MMLU (Hendrycks et al. 2021 We use llm-eval-harness (Gao et al. 2021 to evaluate the compressed
models.
Baselines. ToMoE is compared to baselines from structural pruning methods
MMa et al. 2023
Ashkboos Gao al.
ct al. 2024 Men et al.) 2024; [Song et al.) 2024 van der Ouderaa et al.) 2024 Lin al. 2024), semi-structural pruning methods (Frantar & Alistarh, 2023 Sun et al. 2024 Dong all
2024
2024b) and
MoE construction methods (Zhu et al. 2024 Lee
al. 2024b: 4b: Qu et al. 2024 Pei et al.
2025
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Compassion against MoE construction methods.
ARC-
ARCC-
PIQA
WinoG,
HellaS.
Model
Active Parameters
Method
Average
acc-poria
acc-norm
acc-noem
acc
acc-noem
100%
Dense
74.58
46.25
70:11
69.06
75.90
69.00
LLaMA-MoE E16A1
Zhu
2024
27. 02
29.53
49.13
49.49
26.19
36.27
LLaMA-2
7B
+ live-tuning
35.52
26.88
19.13
55.71
31.07
10.35
50%
LLaMA-MokE MoE ESA2
Zha
2024
25.76
27.22
50.92
47.51
26.02
35.49
lin-stning
37.71
27.56
58.11
51.78
36.37
42.31
ToMoE (Ours)
56.65
33.87
71.00
58.56
60.26
56.07
100%
Dense
79.11
72.17
Phi-2
78%
G-Mooliescion
Lee al
2024b
66.46
38.91
70:29
65.75
57.90
59.86
70%
G-MoPfiecron
LLe al
2024b
57.95
35.24
64.90
60:30
49.07
53.47
ToMoR (Ours)
70.79
43.86
77.00
66.38
62.66
64.16
Table 5: Comparison against MoE construction methods on LLaMA-2 7B.
Parametens
Method
Tokoss
BoolQ (32)
SeiQ
PIQA
WinoG. (5)
ARC-C (25)
HellsS. (10)
Average
A00
are
ano-norm
100%
LIaMA-2 7B
21
82.04
90.80
78.78
73.96
53.15
78.35
76.21
LLaMA- MoE E8A2 Zlb et al 2020
1.2B
37.83
20:00
49.73
50.12
25.79
26.18
34.27
LLnMA-MaE-v2 E8A2 (Ou es nl. 2024
1.2B
51.25
67.00
56.64
52.88
25.68
35.10
48.59
30%
CMoE ESSIA1 (Pei 43..0. 2025
0
16.09
65.30
52.77
48.70
23.50
30.12
44.13
tuning
77.50
ToMoK (Ours)
34.39
Table 6: Comparison against LLaMA-MoE- on LLaMA-3 SB.
Method
* Tokens
Buuliq
SeQ
PIQA
ARC-C (25)
ThushfulAA
Hellss,
MMLU
a00
acc aorm
a0c
s0c 200m
axc
acc mocm
acc
100%
LLMMA-3
93:20
78.51
78.79
67.22
LLMA MoB v2 ESA2 Og al
2024
70
74.62
90.60
60.26
42.83
45.62
58:55
$7.41
50.61
00%
LLaMA ESAISI
2026
70
88.80
67.90
46.85
52.67
ToMoR (Ous)
0.028
63.07
87.60
72.08
37.29
40.08
63.91
36.91
57.40
4.2 Language Modeling
Tab. presents the perplexity results of structured pruning methods applied to LLaMA-2 models of sizes 7B and 13B on the WikiText-2 dataset, comparing various methods with 70%, 60%, and 50% of active parameters setting corrrspodirggg to pruning ratios of 30%, 40%, and 50% for pruning, respectively. Across all pruming ratios, ToMoE consistently achieves the lowest perplexity compared to other methods, even performmiing many approaches with significantly larger numbers of active parameters. For instance, ToMoE with 50% active parameters achieves perplexity of 8.36, which is superior to LLM-Pruner, ShortGPT, SLEB, and SliceGPT at 30% pruning ratio. Furthermore, ToMoE with 50% active parameters surpasses ModeGPT and LLM Surgeon at 40% pruning ratio. While the gap between ToMoF, and DISP-LLM is smaller, it is still obvious at a 50% pruning ratio: ToMoE achieves a perplexity that is 1.48 points lower than DISP-LLM. ToMoE also exhibits superior performance with the LLaMA-2 13B model, maintaining similar advantage over other methods as observed with the LLaMA-2 7B model. This demonstrates the effectiveness of ToMoE with much fewer active parameters. Tab. 2 presents a comparison of our method against semi-struclural pruning techniques. Our approach consistently achieves the lowest perplexity while retaining 50% of the active parameters. Moreover, the performance gap between our method and the semi-structural pruning methods is also obvious. On the LLaMA-2 7B model, SparseGPT achieves the second- best performance, with our method improving upon it by 1.81 in terms of perplexity. For the L.L.MA-2 13B model, Pruner-Zero shows the second-best performance, while ToMoE. further reduces the perplexity by 0.63. The comparison against semi-structural pruning methods further demonstrates the advantage of our method on the language modeling task.
in maintaining strong language modcling performance,
9

---

Published in Transactions on Machine Learning Research (01/2026)
Table ToMoE Visualization for the last layer of the LLaMA-2 7B model with 50% active parameters
Expert
<a> Grand Theft Auto VI is un upcoming video game in development by Rockstar Games, It is due to be the eighth main Crand Theft Auto game, following Grand Theft Auto V (2013), and the sixteenth entry overall. Set within the fictional open world state of Leonidabased om Floridaand its Miamiiinspired Vice City, the story expected to follow the criminal duo of Lucia and her male partner. '/F ofFlooinng years of anticipation, Rockstar confirmed February 2022 that game was in development, That September, footage from unfinished versions was leaked online in what journalists described as one of the biggest lraks the history the video game industry;
Color:
Expert
Expert 2
Expert
speculation
Expert
Expert
game was formally revededd December released in late 2025 the Play Station and Xbox Series X/S. \nGrand
Expert
2023 and scheduled the fictional open world state of Leouidabasod
Expert
Floridawhich includes Viee City, fictionalised version of Miami, Vice City previously featured Grand Theft Auto (1997) and the main setting of Grand Theft Auto Viee City (2002) and Grand Theft Auto: Vice City Stories (2006]] The world parodies 2020s American culture, with satirical depictions of social media and influencer culture, end references to Internet memes such Florida Man, The story follows a criminal duø; Lucia, the series' first female protagonist since] and her male partner; the first trailer depiets Lucia prison inmate, and later evading custody with her partner
Theft Auto VI
Expert
2
0.7
0.6
Expere
0.4
0.3
10
15
25
30
Layers
Figure 4: Experts token allocation of ToMoE for the LLaMA-3 model collected on the WikiText datasot.
4.3 Zero-Shot and Few-Shot Performance
In Tab. 3 we present the zero-shot performance of various methods LLaMA-2 7B, LLaMA-3 8B, and Qwen- 2.5 7B. Our method consistently achieves the best average performance across all models. For LLaMA-2 7B, compared to weaker methods like GhortGPT and SliceGPT, our approach demonstrates significant advantages (ToMoE 50%: 60.72 vs. SliceGPT 40.84 and ShortGPTT 70%: 47.07). The advantage against stronger baselines is also obvious. Although ModeGPT performs closer to ToMoE, the gap remains significant. With 60% active parameters, ToMoE is 3.14 times better than ModeGPT. For L-aMA-3 8B, the performance advantage of ToMoE is even larger, where it reduces 5% more active parameters than ModeGPT while still achieving a 2.71 performance gain. Furthermore, when removing 15% more active parameters compared to ShortGPT and SliceGPT, ToMoE exceeds their average performance by 18.93 and 16 points, respectively. For Qwen-2.5 7B, ToMoF. significantly outperforms DISP-LLM, consistent with previous findings other models. We further investigate the effect of the number of experts N when 40% to 50% of the parameters are active. The results indicate that increasing the number experts to 16 is beneficial. However, further increasing to provides only marginal or no improvement, likely because a too-large number of experts burdens the learning process. Thus, we recommend choosing the number of experts N to be smaller than 16.
In Tab. 4 method demonstrated superb advantages compared to existing MoE construction methods. In " "+iinettuinng setting of LLaMA-MoE, the resulting model is trained for the same number of iterations as ToMoE for updating model weights. In Tab. 5 and Tab. 0 we further compare our method with CMoE, LLaMA-MoE, and LLaMA-MMB-v2, following the experimental settings in their papers. Our approach consistently surpasses all baselines while requiring significantly fewer tokens. Notably, in Tab. 6 when compared against. the fully trained LLAMA-MoE-v2, our method achieves comparable performance even without additional fine-tuning of the model weights. In summary, our method shows that learning routers and experts together is more promising solution compared to existing works.
10

---

Published in Transactions on Machine Learning Research (01/2026)
Preserved Rales
0.4
QK
vo
Layers
Figure 5: Model width after ToMoE for the Qwen-2.5 7B model when the number of active parameters equals 50%.
Tokket
Qwen-22570
ToMot
Withor
Design
Without Selt- Sell-Knowledge Diatitation
Expec an
Traeirg
Charge
Training on Aplaca Datumet
130
Training Whkdfostt
(a) Width and union of experts
(b) Training costs
(c) Ablation study on Qwen-2.5 7B
Figure 6: (a) Model width and union of experts. (b) Costs of different learning-based methods. (c) Ablation study Q.en-2.5 7B.
4.4 Analysis of ToMoE
Training Dynamics. In Fig 3 all p, the knowledge distillation loss L (F), the parameter regularization loss Rp, and the union of experts regularization loss Ry decrease over the course of training. Notably, the parameter regularization loss quickly drops to 0 in the early stages of training, while using a smaller p requires more iterations. The peak of the union of experts regularization loss increases when using smaller values of p, indicating that the initial solution lends to only cover small portion of the dense model. Regarding the load balancing loss, it oscillates around 0.15, demonstrating that ToMoE maintains a relatively balanced load distribution during the training process.
we visualize the training dynamics under different values of p. Across
Ablation Study. Fig. bc we present the average zero-shot task performance under different set lings. For p= 0.4 and p = 0.5, replacing the knowledge distillation loss with the language modeling loss significantly impacts performance. At p 0.4, removing Ry also results in substantial performance drop, whereas the impact is much smaller at p = 0.5. We hypothesize that this difference arises because reducing p makes the learning process more challenging. Without the guidance provided by Ru, the model struggles to effectively utilize the parameters of the original model. Additionally, the choice of dataset affects performance, particularly when switching to the WixiText dataset. This demonstrates that a mixing dataset is beneficial to the overall performance.
Other Analysis. (1). Fig. 5 presents the width of our ToMoE model for Qwen-2.5 7B, which shows the wiser-wise configuration is highly non-uniform. It demonstrates that our method can flexibly set the width of different layers and operations. (2). Fig, 6a. shows that the union of experts is close to the full model capacity, even though the width of experts across different layers is highly non-uniform, demonstrating the effectiveness of our loss design. US dollars. ToMoE costs similarly compared to DISP-LLA -LMM with LLaMA-2 7B and 13B models, and both of them are much cheaper than LLM Surgoon. (4). Fig. shows the token allocation across experts on the Wikitext dataset. We observe that the early and late layers exhibit relatively balanced expert utilization, while the middle layers have certain experts activated more frequently. (5). Finally, we visualize the expert.
(3). Fig, 6b plots the costs of different sarning-based methods in terms of
11

---

Published in Transactions on Machine Learning Research (01/2026)
selection for LLaMA-2 7B in Tab. 7 We observe that each expert aligns syntax rather than semantic meanings, resembling the observations in
JJang et. al. 2024
5 Conclusion
In this paper, we propose novel algorithm, ToMoE, for converting dense models into MoE models through dynamic structural pruning. The resulting MoE models significantly outperform state-of-the-art structural pruning methods while using similar or lower training costs compared to other learning brarning-based pruning methods. Our findings reveal the presence of meaningful experts within the MLP layers of dense models, even without fino-tuning the model weights. ToMoE serves as powerful tool for uncovering these experts within the original dense LLM.
References
Joshua Ainslie, James Leo-Thorp Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-head checkpoints. Tn Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 4895 4901, 2023.
Sotiris Anagnostidis, Dario Pavllo, Luca Biggio, Lorenzo Noci, Aurélien Lucchi, and Thomas Hofmann. Dynamic context pruning for efficient and interpretable autoregressive transformers. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Newral Information Processing Systems 2023, NewrlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::/pppprss.nnitiii nips. html
paper http:/aaper/22/////ae
Anthropic. Claude: A safer conversational ai assistant.
https://www.atttrrrro..ccoo/////
claude 2023.
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM J. Emerg. Technol. Comput. Syst., (2:):322:132:11 2017. doi:
URL
https://doi.. org/ 10/ 1145/3005348
Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprinl X3xx:1308..432, 2013.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. In Proceedings of the AAAI conference on arlificial intelligence, volume 34, pp- 7432 7439, 2020.
Tom Brown. Language models are few-shot learners. arXiv preprint arXiv:2005 2020.
Sahil Chaudhary. Code alpaca: An instructiinn-folloinng llama model for code generation. https://githu/ com/sahi128011t/ccaaaaaaaa.. 2023.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the LEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11030-11039. 2020.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint 15l:1803.05.4 2018.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. arXiv preprint arXiv:2401.0000 000666,,,, 2024.
Harry Dong, Xinyu Yang, Zhenyu Zhang, Zhangyang Wang, Yuejie Chi, and Beidi Chen. Get more with LESS: synthesizing recurrecce with KV cache compression for efficient LLM inference. In forty-forst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview net, 2024a. URL ittts::/openrreviet http:/pppreeriieeeeeeforuuurdddd wett/foouumiid-uuhaaWkYY.
Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero- Evolving symbolic pruning metric from scratch for large language models. In Forty-forst International Conference on Machine Learning, 2024b.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey in-context learning. Xiv preprint N.i:2230100023, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishok Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprinl 20X.::2407.21783, 2024.
Gonglan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinchao Wang. Depgraph: Towards any structural pruning- Tn Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16091 16101, 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(10)): 39, 2022.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323 10337. PMLR, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprinl or Xiv:22200. 1 1323,, 2022.
Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang, Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. arXiv preprint arXiv:2406. 12042, 2024.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021. URL https:///ooii org/ 10. 5281/zenodo. 5371628
Shangqian Gao, Chi-Heng Lin, Ting Hua, Zheng Tang, Yilin Shen, Hongxia Jin, and Yen- Ven-Chang Hsu. Disp-Ilm: inmeeession-nddpendeen structural pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng zhong Xu. Dynamic channel pruning: Feature boosting and suppression. In International Conference on Learning Representations, 2019. URL
https://openrrvview wwwt:/oorrmmddd//xthaaaa... http:://pppeeeeeeeeesss//oooreep
David Ha, Andrew Dai, and Quoc V Lo. Hypernetworks. arXiv preprint (XXi::609.09106 2016.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network, 2015. URL httpps:/aarxiv.org/// w0tp:///..rg//aa//55..... 1
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel- softmax. orXiv preprint arXi::661..0114/ 2016.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprini arXiv:2401.0400 2024.
Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08800 008661, 2020. URL httts:///aaraatee www.oogg//........oooooooooo.o.r httg/abs/200111 2tt.p//00010111111111111
Jacob Devlin Ming-Wes Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacI-HLLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Flias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The optimal BERT surgeon: Scalable and accurate second-order pruning for large language models. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 4163-4181. Association for Computational Linguistics, 2022. doi: /Ttp////2222222222222
URL https://doii http:////bii..rr.1..1188.6////// 1ttg///1....163//////222 emnlp-main. 279
Donghyum Lee, Jacyong Lee, Genghan Zhang, Mo Tiwari, and Azalia Mirhoseini. CATS: Context-aware thresholding for sparsity in large language models. In First Conference Language Modeling, 2024a. URL
httpps://openreview http:://pppporrrreeeeeeeeertrtrr net//orumm1dd=v322772
Jaeseong Lee, Seung-won Hwang, Wonpyo Park, and Mingi Ji. Breaking relu barrier: Generalized moefication for dense pretrained models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10097 10097-10107, 2024b.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. (GS)hard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021a. URL https:
//openreview net/forumiid-qrwe YXHTmbb
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In International Conference on Learning Representations, 2021b.
Hao Li, Asim Kadav, Igor Durdanovie, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In International Conference on Learning Representations, 2017. URL https: httpp::/ //perrvview ket/forum?idd=rJq0T1
Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, and Yen-Chang Hsu. Modegpt: Modular decomposition for large language model compression. arXiv preprint XrXi:22408. 09662, 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimie human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 0224-3252. 2022.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Ré, and Beidi Chen. Deja vu: Contextual sparsity for efficient LLMs at inference time. In Proceedings of the 40th International Conference on Machine Learning, pp. 22137-22176. PMIRR 2023.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Represenlations, 2019. URL http:///eerrreeeeeeeeeerrrrrrrra
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Lim-pruner: On the structural pruning of large language models. Advances in newral information processing systems, 00:2170--21722 2023.
Xin Men, Mingyu Xu, Qingru Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853, 2024.
Stephen Mcrity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture modols, 2016.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on compuler vision and pallern recognilion, pp. 11264-11277 2019.
Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Bhuminand Joshi, Marcin Chochowski, Mostola Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. In The Thihty-eighth Anmual Conference on Neural Information Processing Systems, 2024.
OpenAL. Chatgpt: Optimizing language models for dialogue. https:///ponai.comm
2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pigh-peeformance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-800000 8035, 2019.
Zehua Pei, Lancheng Zou, Hui-Ling Zhen, Xianzhi Yu,, Wulong Liu, Sinno Jialin Pan, Mingxuan Yuan, and Bei Yu. Cmoe: Converting mixture-of-experts from dense to accelerate 1lm inference. arXiv preprint arXiv:2502.04416 2025.
Xiaoye Qu, Daize Dong, Xuyang Hu, Tong Zhu, Weigao Sun, and Yu Cheng. Llama-moe v2: Exploring sparsity of llama from perspective of mixture-of-experts with post-training, arXiv preprint 0rXiv:2411.1500 2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Πlya Sutskever. Improving langnage un-
derstanding by generative pre-training. 2018. URL https://cdn, http:////ddooooni..... openai. ccm/ressaaccc-ccorrrr.... language-unsen wuwpperriiddd///uaggeeeee ottpe///weeeeeggeeeeoommmnrrnnia understanding paper.ppeee Technical report, OpenAl.
Alec Radford, Jeffrey Wu, Dario Amodei, Jack Clark, Miles Brundage, and Πlya Sutskever. Language models are unsupervised multitask learners. https::/ccd..ppeeee
models are unsupervised htwpaaeedddmtttttteeeeeea Sealliitask_eeanerrss learners. pdff 2019. OpenAl Technical Report.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 2021.
Noam Shazeer. Fast transformer decoding: One write-head is all you need. arXiv preprint Xix:::9911.00 2019.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gass gated mixture-of-experts layer. In International Conference on Learning Representations, 2017.
Jiwon Song, Kyungsook Oh, Tacsu Kim, Hyungjun Kim, Yulhwa Kim, et al. Sleb: Streamlining llms through redundancy verification and climination of transformer blocks. In Forty-first International Conference on Machine Learning, 2024.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocompuling, 568:127063, 2024.
Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL
httpss://pperrevier http://nnnntttoorrrraaoooooooo.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovingg Ilama model. https://giithb..cm
tatau-lab/sstanord alpaca 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arxiv22002. 13971,, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Sounya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint WxXi:2337.00928, 2023b.
Tycho FA van der Ouderaa, Markus Nagel, Mart Van Baalen, and Tijmen Blankevoort. The Ilm surgeon. In The Twelfth International Conference on Learning Representations, 2024.
Huanyu Wang, Lei Xie, Hanbin Zhao, Chao Zhang, Hui Qian, John CS Lui, et al. D-llm: A token adaptive
computing resource allocation strategy for large language models. In The Thirty-eighth Annuul Conference Neural Information Processing Systems, 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, 25-45, Online, October 2020. Association for Computational Linguistics. URL omnlp-demos.6 demos. 6
https://www.cc awwweb...bbb...... org/ htgg/anthology/200
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fci Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 12.i:2412.15115 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer feed-forward layers are mixtures of experts. In Findings of the Association for Computational Linguisties: ACL 2022, pp. 877 890, 2022.
Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Zhuoqing Mao, Beidi Chen, Fan Lai, and Atul Prakash. Learn to efficient: Build structured sparsity in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Syslems, 2024. URL httpps://openrevien net hett/Eorumiid-iSSCChhBO
Tong Zhu, Xiaoye Qu, Daize Dong, Jiacheng Ruan, Jingqi Tong, Conghui He, and Yu Cheng. Llama-moc: Building mixture-ofeexpets from llaaa with continual pre-training. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15913 11923,,,,,,,,111 2024.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A Details of trainable modules
A.1 Module Configurations
We present the details of trainable modules in Tab. 8 In short, we project the imput tokens to low- dimensional space and add them to the output of the HyperNetwork. The inputs to the HyperNetwork are fixed random vectors of size N x 32 sampled from a Normal Distribution. Except for the HyperNetwork, other individual trainable modules are created for each MHA and MLP layer. If we have L blocks, then we will have L Proje L Proj with output size of H: L Projn with output size of d 2H L, ProjD Projer and L
Router layers. Notations of d, H, dmidd and N are already defined in Sec. 3
Module Types
Removed?
Structures
HyperNetwork
/
Input : > Bi-GRU
Proje
X
Linear(d, d, 128)
Proja
X
LayerNorm(de = 128) Gel.U-I.innardde 128, #
Projh P
/
LayerNorm((e 128) GeLU GeLU->Limar(de 128, dmaid)
Router
X
Linear(d, N))
Table 8: Detailed configuration of trainable modules.
After we complete the training of ToMoE, we do not have to preserve all modules. The embeddings from the HyperNetwork will be saved, so the HyperNetwork can be removed without impacting the model. Proj PLo brings most additional parameters, fortunately, it can also be removed. After the training of ToMoE, E and Projo can be used to directly generate experts:
Se = =T ST--GSi((Proo (E)),
(11)
where ST-GSig again is the Straight-Througe Gumbel-Sigmoid function. Se {0, 1] Nxdmii binary vectors to select experts from the dense model. Once Se is generated, it can be reused, and thus we no longer need Projbue. Let St Diag(s$), 1,... .N. Similarly, use & Remi xd to represent the actual column row selection matrix by removing zero columns rows, where d'mid dmid and it is the width of each expert. The ith expert can be represented as:
Xdmid is the resulting
har X) = oXWoaS' @ (XW
(12)
After ToMoE, given the result of the routing function SST-GSmax(Router(XX (X)), the MLP calculation with MoE can be written as:
Y, G.,
(13)
where X, is the feature map of ith token, and i represents the index where G.,, = 1. Note that Eq. 13 is still differentiable with respect to the parameters of the Router.
Another question is how many parameters we need after introducing Top-K routing for MHA layers and Top-1 1 routing for MLP layers. Analytically, the additional parameters can be calculated by L x 128 + L x 128 (1) L x x N. Let's use LLaMA-2 7B as an example, L 32, 4096, N 8, H 32, x 32 4096 x 128 + 32 128 (128) 32 x 4096 0.0184B. This is equivalent to 0.27% of the total parameters of the LLaMA-2 7B model, and thus, the additional parameter is not significant compared to the original number of parameters.
the additional parameters
A.2 Head Dimension Pruning vs. RoPE
Rotary Position Embedding (RoPE) Su et al. 2024 is popular positional encoding method, and it is regularly used in LLMs like LLaMA Touvron et al. into 2 sub-spaces, and they are applied on query and key. This means that if we want to perform head dimension pruning for query and key matrices, we need to follow the sub-spaces resulting from RoPE and
2023b). RoPE divides the म dimensional space
17

---

Published in Transactions on Machine Learning Research (01/2026)
Table 9: Zero-shot task performance of compressed LLaMA-2 13B and Qwen-2.5 14B.
ARC-
ARC--
PIQA
WinnG.
HellaS.
Model
Active Parameters
Method
Averago
BOC morm
Roo-norin
act-norn
arc
acc-s1orm
100%
Dense
77.48
40.23
80.47
72.22
79.39
71.76
SliceGPT
2024
60:27
36.18
09.42
64.09
49.74
55.94
Surgeon
Ouderta
2014
30.42
63.07
70%
DISP-LLM Alpaca
Gao al
2021
68.98
11.28
77.31
67.32
68.98
63.39
MoDeGPP T-Appcca
Lin et a..
2022
70.21
11:47
77.15
71.27
71.81
66.39
ToMoE (Ours)
74.58
45.14
77.97
75.44
68.19
68.26
SlieePPT Ashiboos ot al.
2024
48.99
32,51
63.17
56.75
39.85
48.25
LLaMA-2 13B
LLM Surgnon Van der Ouderaa al
2024
63.50
37.12
73.16
65.75
65.64
60.94
DISP-LLM Gao
2022
62.67
35.63
73.39
62.67
65.86
60.01
DISP-LLM Alpacs
Gao
2024
66.79
42.75
75.30
64.25
67.52
63.32
MoDeGPT-Alpern
Lin
2024
63.72
38.82
71.87
66.30
62:10
00.56
ToMoE (Ours)
67.63
66.38
65.00
Surgnon van Ourlens
2024
37.12
56.19
56.32
DISP-LLM Gao et aL. 2014
58.27
36.87
68.67
59.27
37.18
54.30
DISP-LLM Alpuco (aa et al.
2024
58.72
37.54
72.20
59.59
62.39
57.49
ToMoE (Ours)
63.51
37.97
73.29
62.59
68.30
61.18
100%
Deuse
79.12
59.13
83.09
71198
82.10
75.71
DISP-LLM Gao et al.
2024
65.87
37.99
72.20
58.51
60.63
59.04
Qwen-2.5 14B
ToMoE (Ours)
66.37
39.16
60.69
65.57
61.28
DISP-LLM
32.62
70.08
52.96
40%
ToMoE (Ours)
59.85
34.73
71.49
56.60
56.06
make these two sub-spaces share the pruning mask So, and the final pruning mask for query and key is si [sop: So[1: f011:111: and clearly the size of $0[1:4]] is 2H In short, simply select the first. half of elements from So and repeat twice to make the final pruning decision. We also found that applying dynamic pruning for query and key matrices along the head dimension is difficult and unreasonable since different tokens may have different positions after pruning. It becomes a problem when calculating the inner product between the query and key matrices given different tokens.
By applying head dimension pruning, our method also does not need to be modified when facing different. attention mechanisms like GQA (Grouped-Quess Attention) (Ainslie et al. 2023) and MQA (Multi-Queel
Attention) Shazeer 2019
A.3 Details of Gumbel-Softmant and Gumbel- Sigmoodd
The Gumbel-Softmax function (Jang al. 2016 allows for differentiable sampling from categorical distribution. Given logits x, the Gumbel-Softmax sample is computed
x
y softmax
where each element. of g is drawn from Gumbel(0. 1), and is the temperature parameter that controls the gradient Estima-
smoothness of the distribution. Combining Stumbel-Softase with the tor (Bengio et al. 2013), we have the following equation:
I; + Di
ST ST-GSmax(x) one-hot
arg max IDD
(14)
where D f1,2,... N), N again is the number of experts our setting, and one-hot will assign corresponding to the position of the maximum value in xtg and assign 0 to other positions.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Table 10: Zero-shot task performance of the compressed Phi-2.
ARC-
ARC-c
PIQA
WinaG.
HellaS.
Active Parameters
Method
Avg
acc-boem
acc-norm
acc-norm
acc
acc-norm
100%
Dense
78.21
51.01
79.11
75.61
73.86
72.17
SliceGPT
Asháboos al.
2024
58.00
35.32
71.87
67.80
57.76
58.15
80%
|Fieetuning
56.61
38.91
71.27
67.17
54.86
57.76
DISP-LLM
Goo
67.09
63.43
SliceGPT
52.40
54.46
75%
[Fme-tuning
65.
52.48
55.17
DISP-LLM
Geo et
2021
65.93
13.31
71.27
65.
59.95
61.72
SliceGTT
Asháboos et al.
2014
53.08
30.29
63.94
63.
47.56
31.99
[Fine-tming
46.38
32.68
66.16
63.54
49.72
51.70
70%
DISP-LLM
Gao et al.
2021
63.59
38.48
73.31
65. 19
54.43
59.00
ToMoE (Ours)
70.79
13.86
77.09
66.38
62.68
64.16
The Gumbel-Sigmmid function is special case of the Gumbbel-Sfttma function, designed for binary distribu- tions. Given logits x, the Gumbel-Sigmoid sample is computed as:
y sigmoid
where is sampled from (0mbel(O, 1) and again is the temperature parameter. Combining with the Straight-Throug gradient Estimator, we have the following equation:
x + g +
ST-GSig(x) cound(sigmoit
(15)
where b is constant bias in our implementation and it ensures that all experts start from the whole model, round(.) will round the input values to the nearest integer, and in our case, it rounds inputs to or 1. For all
experiments, we set b = 3.0 in Eq. 15 and we set T = 0.4 for Eq. 14 and Eq. 15
Listing 1: Pseudo-code for self self-knowledge distillation.
with torch no no_grad ():
Disable trainable modules for ToMoE helper... set set_modu modummodleee.emm status (model, False)
Get logits from the teacher (original model) teacher_ output = podel(inputa Boacher_logite teacher teacher_output, output.logita logits
Enable trainable modules for ToMoE helper.. set_ set_module, status (model, True)
10
# Get logits from the student model from ToMoE learning process
model_outpur model (inputs) logits model_ logits
14
B More Details of the Loss Design
B.1 Implementation of the Keff-Knowledge Distillation
During the ToMoE learning process, we freeze the parameters of the original model. This approach offers the additional benefit of enabling self-knowledge distillation without the need to load extra model. Tn
19

---

Published in Transactions on Machine Learning Research (01/2026)
Table 11: ToMoE Visualization of LLaMA-2 7B with 50% active parameters
Expert Color
Expert Expert Expert 3 Expert 4 Expert Expert 6 Expert Expert 8
MLP
<a> Homarus gammarus, known the European lobster common lobster, is spectes of clawed lobster from the castern Atlantic Ocean, Moditerranean Sea and parts the Black Sea, It is closely related American lobster, H. americanus. It may grow to a length of 60 cm (24 in) and a mass of 6 kilograms
[13 lb), and bears conspicnous pair of claws. In life, the lobsters are blue, only becoming "lobster red on cooking. Mating occurs the summer, producing eggs which are carriod by females for up year before hatching into planktonic larvae. Homarus gammarus is a highly esteemed food, and is widely caught using lobster pots, mostly around the British Isles.
MLP 16
<a> Ioomaaus gammarus, known the European lobster common lobster. is species of clawed lobster from the castern Atlantic Ocean, Mediterrancan Sea and parts the Black Sea. It closely related the American lobster, II. americanus. It may grow to a length of 60 em (24 in) and mass of 6 kilograms 1b), and bears conspicuous pair of claws. In life, the lobsters are blue, only becoming 'lobster red on cooking. Mating occurs the summer, producing eggs which are carried by the females for up year before hatching into planktonic larvac. Homarus gammarus is a highly esteemed food, and is widely caught using lobster pots, mostly around the British Isles.
MLP 32

 Homarus gaminruss, known the European lobster common lobster, is species of clawed lobster from the eastern Atlantic Ocean, Mediterranean Sea and parts of the Black Sea, It closely related the American lobster, H. americanus. It may grow to a length of 60 cm (24 in) and masss of 6 kilograms (13 lb), and bears conspicuous pair of claws, In life, the lobsters are blue, only becoming 'lobster red cooking. Mating occurs the summer, producing eggs which carried by the females for up year before hatching into planktonic larvae. Homarus gammmrru is a highly esteemed food, and is widely caught using lobster pots, mostly around the British Isles.
Table 12: Zero-shot task performance of compressed LLaMA-2 7B with more settings.
Active Parameters
Mothod
ARC-c
ARC-c
PIQA
WinoG.
HellaS.
Average
8cc-norm
aco-norm
ac0-norm
acc
accnorm
100%
Dease
74.58
46.25
79.11
69.06
75.59
69.00
ShortG1TT
al.
2024
64.31
64.33
56.13
53.25
SliceCPT
Ashikbou
68.55
58.01
53.73
LLM Surgeon
kan der Ouderas et al. 2021
63.09
13.
61.09
60.72
59.00 00.03
70%
DISP-LLM
Cio ot ml.
2024
59.81
33.19
71.82
62.27
63.43
58.10
DISP-LIM
Alpaon (Can et
2024
80.10
37.03
73.72
63.93
62.87
59.53
ModeGIPT
Lin et al
2124
63.26
38. 713
70.40
67.
633.26
60.78
ModeCPT-AR Alpoca
Lin
2024
65.49
30.16
13.34
66.22
65.90
62.02
ShortGPT Men al.
2024
41.16
29.94
60.12
60.46
43.67
47.07
ST
Ashithoos nl.
2024
36.49
24.57
54.90
53.43
34.50
40.84
60%
LLM Surgeon (van der Ouderaa al.
2024
52.31
5:L.38
50.86
Mo
Lan al.
40.45
30.03
64.96
61.96
53.01
51. .88
ModeCPT. Apecn
2024
50.76
34.73
70.35
64.44
ToMoE (Ours)
63. 63.64
38.74
72.85
62.51
65.84
60.72
LLM Surgeon
vau der Ouderaa et. al.
2024
11.91
26.28
61..6
51.55
40.29
45.68
DISP-I.IM GGan et al.
2024
43.06
25.85
63.93
54.54
63.43
46.72
50%
DISP-LIM Alpaca (Gao et nl. 2024
61.14
30.20
08. 08.34
56.20
49.33
51 .05
ToMoE (Ours)
56.65
33.87
71..0
58.56
60.26
56.07
Lst. 1 present the pseudo-code for the self-knowledee distillation process. In summary, we first disable the trainable modules associated with ToMoE and compute the output logits from the original model. Next, we re-enable the trainable modules for ToMoE and perform a regular forward pass. The logits from the original model are then used to guide the learning of ToMoE.
B.2 Efficient Implementation of R.
Recall from Eq. 6 that the union regularization for MLP and MHA layers is defined as:
T
T
R₂ = Us: =s II st).
i=1
20

---

Published in Transactions on Machine Learning Research (01/2026)
For MLP layers, this equation incurs high computational costs since E RTxddme whereas for MHA layers, the cost is significantly lower because d H < dmid- To simplity Eq. 6 note that all 8; 1i=1...., 7') are derived from N experts. Using embeddings from the hypernetwork, we calculate the configuration of N experts as:
ST-GGSigPProDD (E)),
and substitute Se into Eq.
RMP
II s
(16)
This reduces computation by factor of N- 2048 256 times.
T For example, in LLaMA-2, the computational cost is reduced by
Table 13: Ablation study on design choices of ToMoE and the impact of temperature 7 on performance.
Settings
ARC-e
ARC-c
PIQA
WinoG.
HellaS.
Avg
Local Emb
55.22
32.94
66.32
53.12
57.85
53.09
Head Pruning
45.12
25.85
64.82
49.49
39.65
44.99
w/o VO Routing
55.51
32.34
70.40
57.14
59.12
54.90
7 = 0.3
57.28
33.53
70.35
57.54
60.22
55.78
7 = 0.4
56.65
33.87
71.00
58.56
60.26
56.07
0.5
55.89
33.36
71.00
36.75
59.31
55.16
ToMoE
56.65
33.87
71.00
58.56
60.26
56.07
B.3 Equivalence of MoE and pseudo-MoE
One major challenge when training MoE models is maintaining appropriate expert capacity, defined the number of tokens each expert processes (Fedus al 2022). This is typically addressed using a load balancing loss. Without this loss, some experts may become overloaded while others remain underutilized, leading to bottlenecks where a few experts dominate the computation.
Although ToMoE also requires load balancing loss, the potential overhead introduced by load balancing is mitigated by the pscudo-MoE approach after ToMoE. After applying ToMoE, the resulting model can be trained using pseudo-MoE, which resembles the training of dense model. This is straightforward to implement as follows:
fMLP = o (XWc)S (XWrS)SWD
(17)
where S, in represents the routed expert from S. Eq. 11 as determinod by the router. The pscudo-MoE is useful when the active number of parameters is relatively large. In such cases, pseudo-MoE training be more time-efficient than conventional MoE training.
C More Implementation Details
During training the modules of ToMoE, we use AdamW optimizer to optimize it with constant learning rate 10-3 and weight decay 0.05. For different models, we always set the mini-batchsize to 1 on each GPU. For LLaMA-2 7B, and Qwen-2.5 7B models, we use 2 NVIDIA A100 GPUs, For LLaMA-3 8B, we use 3 NVIDIA A100 GPUs. For LLaMA-2 13B and Qwen-2.5 14B models, we use NVIDIA A100 GPUs. For all the rest models, we use NVIDIA A100 GPU. We set p = (0.6, 0.5. 0.4, 0.3) when the ratios of active parameters equals to (40%, 50%, 60%, 70%).
For the Alpaca dataset D we use the 'text' column within the dataset, which combines the columns of 'instruction' and 'output' For the Code Alpaca datasct we combine the 'instruction', 'input', and 'output' columns as one training sample.
httpe///aaaglggfeeeo////atttttts labbaaaaaa hhttes//gthhub.cmm//s///2200 hooos///
21

---

Published in Transactions on Machine Learning Research (01/2026)
Table 14: ToMoE Visualization of LLaMA-2 7B with 50% active parameters (contimued).
Expert Color
Expert Expert Expert 3 Expert 4 Expert 5 Expert 6 Expert Expert 8
MLP
<a> Find the equstion of the line passing through the points (3, and (7. 9) using the slope-interoee intercept
form linear equation. find the equation of the line passing through the points (3, 5) and (7, 9) using the slope-intercept form (y mx + b), we first need to find the slope (m) and the y-intercept (b).
1. Find the slope (m]: (y22 yT) [x2 x1)
(9 5)
[7
3)
47 4
2. Use one of
the points to find the y-intercept (b). We'll use (3. 5): y
b X 2 The equation of the line
Write the equation in slope-intercept form: mxx b passing through the points (3, and (7, 9)
1x
2
2.
MLP 16
<a> Find the equstion of the line passing through the points (3. and (7. 9) using the slope-intercept form linear equation. using the slope-intercept form (y Mx + b), we first need to find the slope (m) and the y-intercept (b).
find the equation of the line passing through the points (3, 5) and (7, 9)
1. Find the slope (m): (y2 y1) (x2 x1)
(9 5)
[7
3)
4/ 4
2. Use one of
the points to find the y-intercept (b). We'll use (3, 5): y mx
+
Write the equation in
form: mx + by = 1x 2
x + 2 The equation of the line
passing through the points (3, and (7, 9)
2.
MLP 32

 Find the equation of the line passing through the points (3. and (7, 9) using the intereitt
form of linear equation. To find the equation of the line passing through the points (3. 5) and (7. 9)
using the intereitt form (x mx + b), we first need to find the slope (mn) and the y-intercept (b).
1, Find the slope (m]e (y2 y1) (x2 x1)
(9 5) [7
3) m
//
1 Use one of 2 The equation of the line
4
the points to find the y-intercept (b). We'll use (3. 5): y-y- mx
1
3
=
Write the equation in intereett form: mxx bb 1x
2
x
+
passing through the points (3, and (7, 9) is
vo MLP
Preserved Rolos
15
25
30
Layers
Figure 7: Model width after ToMoE for the LLaMA-2 7B model when the number of active parameters equals 50%.
D More Experimental Results
Model
Aclive Parameters
Batch Size 512
Batch Size 1024
Batch Size 1536
LLaMA-2 7B
100%
1781.76
tokens/s
1805.80 80 tokeus/s
1838. 45 tokeus/s
LLaMA-MoE ESA2 2hm al. 2024
2685.35
tokuus/s
2796.07 tokens/s
2887 .98 tokeus/s
50%
ToMaE (ours)
2360.38
takens/s
tokens/s
2919.45 tokens/s
Table 15: Inference throughput (tokens per second) under different mini-batchsizes..
In Tab. and Tab. 10 we present the zcro-shot performance of various methods on LLaMA-2 13B, Qwen-2.5 14B, and Phi-2 models. From Tab. g it is evident that ToMoE consistently outperforms other methods. Compared to 7B or 8B models, the performance gap between method and other approaches is smaller, which also holds for the differences between bascline methods. This is likely due to the larger model sizes. Table 12 presents the zero-shot performance of the LLaMA-2 7B model across more baselines and active parameters. As shown in the table, ToMoE consistently achieves significantly better performance than all competing methods.
On the LLaMA-2 13B model, ToMoE surpasses structural pruning methods even with smaller compression rates. For instance, ToMoF with 50% active parameters performs better than MoDeGPT and LLM Surgeon
22

---

Published in Transactions on Machine Learning Research (01/2026)
0.6
Normalzied Expert Siza
0.55
0.45
0.35
0.3
10
11
12
13
17
18
19
22
23
24
25
26
27
Layers
Figure 8: Box plot of widths across different experts for the LLaMA-2 7B model when the number of active parameters equals 50%.
Live: 1
Läver 2'
Layer
Leyer a'
Layer 5
Layer 6"
Liver 1
Layer
Liven e
Laver
Layer ri
Layer 13
Layer 13
Liyyz
Loyer 16
Layer
Liver
tayor
Talpor
Layer
Layer
Laye: 25
Layer
Layer
Layer 24
Layer
Laper
Layer 31
Layer
Figure 9: The similarity of different experts from different layers of ToMoE of the LLaMA-2 7B model.
with 40% compression rate. The performance gap becomes even more obvious when comparing methods with the same number of active parameters. Similarly, from Tab. 10] ToMoE demonstrates superior performance compared to SliceGPT and DISP-LLM. Specifically, ToMoE with 70% active parameters achieves better results than all three compression levels of SliceGPT and DISP-LLM.
In Tab. 15 we report the inference throughput (measured in tokens per second) of different models under varying batch sizes. Compared to the dense LLaMA-2 7B baseline, both converted MoE models achieve higher throughput due reduced active parameters. Our resulting model has a similar throughput LLaMA-MoE when the batch size is large enough.
ToHat
In Fig. 10 we present the accuracy-paramette trade-off on the more challenging MMLU dataset with the LLaMA-3 8B model. The results show that our method can still provide meaningful results when activating only 50% of the model parameters.
Actis Parerittrs
Figure 10: MMLU accuracy vs. active parameters.
In Fig. 7 we illustrate the width of ToMoE for the LLaMA-2 7B model. highly non-uniform pattern emerges in the allocation of active parameters, indicating
23

---

Published in Transactions on Machine Learning Research (01/2026)
that ToMoE can effectively determine the ideal distribution of active parameters, even when the allocation is highly non-uniform.
In Tab. 13 we present an ablation study to analyze several design choices in ToMoE, focusing on architectural components and the effect of the temperature parameter т. Although ToMoE applics contextual sparsity to the value and output (VO) projections within the attention layer, their contribution to overall performance is relatively minor due the small head dimension (128, in the case of LLaMA-2 7B). To validate this, we disable dynamic attention sparsity and instead apply only static structural pruning to the attention layer (denoted as "w/o VO routing"). This leads to only modest performance drop of about 1% using 50% active parameters.
We also examine the role of global expert embeddings with GRU in conveying cross-layer architect.ral information. Specifically, we compare the default global expert embedding with a local-only variant ("local emb"), where expert emboddings are used only MLP layers and removed from attention layers. Results show slight decrease in performance, suggesting that global expert embeddings contribute to better coordination across layers.
Additionally, we evaluate the scnsitivity of ToMoE to the temperature T in the routing mechanism. The results with 7 E (0.3, 0.4, 0.5} show that performance remains relatively stable, indicating robustness to the choice of r within a reasonable range.
Finally, we explore head pruning in the early stages of ToMoE development. However, this approach yielded significantly lower performance. This may be due to the distortion of attention feature maps when heads are removed, which makes it more difficult to train effective MLP experts.
These results highlight the effectiveness of ToMoE in preserving the capacity of LLMs compared to structural pruning methods. Additionally, they demonstrate that ToMoB performs robustly across various scales and types of LLMs.
E Visualization of Experts
In this section, we analyze the properties of the experts produced by our method. Tab. II and Tab. 14 present visualizations of the routed tokens among experts across different layers and input texts.
In Tab. 11 observe distinct patterns in the allocation of tokens to specifie experts, which aligns with our obscrvations in Tab. An interesting trend emerges when comparing layers: the first layer exhibits more diverse token distribution, while subsequent layers prefer to assign continuous tokens to the same expert. Tab. 14 focuses on inputs related to a math problem. Unlike the visualization in Tab. II the MoE routing for the math problem reveals clcarer semantic patterns. For instance, Expert 2 in MLP 16 is predominantly activated by numbers and mathematical notations, and a similar behavior is observed for Expert 8 MLP 32. This suggests that the experts in ToMoF, may encode more distinct semantic meanings compared to MoE, models trained from scratch. Further investigation is required to fully understand the precise semantic roles of ToMoE experts.
In Fig. g we present box plot showing the expert sizes across different layers. The figure reveals that the maximum and minimum expert sizes are closely aligned across layers. This outcome is direct result of applying constraints from Eq. g and Eq. g as well as only penalizing the largest expert in Eq. g During training, minimizing the task loss (self-knowledge distillation loss) encourages experts to grow size. Consequently, smaller experts do not remain small due to the task loss and they are not penalized by the parameter regularization loss. This iterative process leads to all experts event converging to similar sizes. After completing the ToMoE training process, adjust the width of all experts to match the maximum size among them. This ensures uniform computational cost across all experts.
In Fig. we present a visualization of the similarity between different experts across all layers of the LLaMA-2 7B model. Within the same layer, experts generally exhibit comparable similarity values, indicating that while the experts share the same size, their weights remain distinct. Notably, certain layers, such as layer and layer 30, show lower similarity values. This observation aligns with expectations, as the expert sizes in these layers smaller.
24

---

Published in Transactions on Machine Learning Research (01/2026)
F Design Choice for the MHA
Ideally, to achieve maximum flexibility, one might consider applying dynamic pruning to all projection matrices in the MHA layer, including the query (Wq), key (Wk), value (Wv), and output (Wo) matrices. However, there is a fundamental limitation when attempting to apply dynamic pruning along the head dimension for the query and key matrices.
Suppose we generate pruning masks St {0, 1]ddd each time step t based on the input X2 Rixd, and consider two distinet time steps, a and b. For the i-th attention head, the attention score between queries and keys is influenced by the pruning masks. Specifically, the effective attention score between the a-th query and the e-th key is given by:
while the attention score between the a-th query and the b-th key is:
W(. c(.,WK..)
The effective width-thar is, the dimensionality over which attention is computed-between between the a-th query and the a-th key is [|S.S] Ilo = S, = K, assuming the mask has exactly K active elements. However, for cross-position pairs like (a, b). the effective width becomes S| llo SS @ Sello S min(E Sa. Sc) K. The equality holds only when S. Ss, which generally does not hold for arbitrary a + b.
This observation implies that dynamically pruned query and key matrices fail to fully utilize the allocated capacity K unless the pruming masks are identical across all positions. Moreover, the variability of the effective width across different query-key pairs introduces instability and inconsistent capacity utilization, making this approach less favorable compared to static pruning for the query and key matricos.
In contrast, dynamic pruning does not encounter this issue when applied the value and output matrices, as these are not involved in pairwise comparisons like the query-key dot products. Therefore, we adopt dynamic pruming only for the value and output projections, while kceping the query and key projections pruned stalically to maintain stable and full-capacity attention compulation.
25


Reviews:
[{"review_id": "arlCBnBlnR", "reviewer": "Reviewer_65R1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The work proposes ToMoE, which introduces **a novel structured pruning approach** by converting dense LLMs into MoE models via dynamic structural pruning, achieving consistently strong performance without weight updates.\n2. ToMoE offers a **low-cost alternative to build MoE from dense LLMs**, avoiding expensive fine-tuning or pretraining, compared with Llama-MoE[1][2].\n3. ToMoE features a **unified, efficient training scheme** with joint expert and router optimization, plus tailored strategies for MLP (top-1) and MHA (top-K + static) layers.\n4. Extensive experiments prove its outperforms state-of-the-art pruning methods across multiple models (LLaMA-2/3, Qwen-2.5) in both perplexity and zero-shot tasks.\n\n\n- [1] LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training.\n- [2] LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. **ToMoE outperforms existing structural pruning methods while maintaining or reducing computational cost**: This is strongly supported by empirical evaluations across multiple model families (LLaMA-2, LLaMA-3, Qwen-2.5) and tasks. Tables 1 and 3 present perplexity and zero-shot performance metrics showing that ToMoE consistently achieves lower perplexity and higher accuracy than state-of-the-art baselines like LLM-Pruner, SliceGPT, LLM Surgeon, and DISP-LLM, even with fewer active parameters. \n\n2. **ToMoE effectively converts dense models into MoE models without fine-tuning the original weights**: The ablation study in Figure 5(c) shows that removing this self-knowledge distillation component significantly degrades performance, confirming its importance in preserving model capacity without updating the core weights. The union of experts regularization (R_U) is shown in Figure 5(a) to ensure that collectively, the experts utilize nearly all original parameters, addressing concerns about capacity loss from pruning. Table 5(b) proves its superior efficiency in existing structure purning methods.\n\n3. But the claims of the advantages of ToMoE as MoE construction method still lack sufficient proof, which will be explained in 'Request Changes'. \n\nIn sum, the claims in this paper are mostly supported with visualizations or experiments, and are very convincing.", "audience_yn": "Yes", "audience_explanation": "This work proposes a new method in the field of structure pruning, and there is still a lot to explore. It opens up a promising direction for future research in model efficiency and sparsity, and we expect it to inspire a wave of follow-up studies exploring dynamic, parameter-efficient architectures without full fine-tuning.", "requested_changes": "1. ToMoE currently only uses top-1 routing for MLP layers in ToMoE, but exploring top-k routing is both possible and might be helpful.\n2. Table 4 is supposed to show that ToMoE is the better MoE construction method, but the comparison isn't fair. Llama-MoE is an example of this. Llama-MoE is randomly initialised and needs extensive continue-pretraining, but in this work, it is evaluated before training. The '+finetune' result is only trained with equal iteration with ToMoE, which is far less than the required training cost of Llama-MoE. It is wrong to say that ToMoE is better than LLaMA-MoE when the number of training steps is the same, because LLaMA-MoE was not designed to work with few training steps. The author should try to show that ToMoE can achieve performance close to LLaMA-MoE, even with a simpler training process and a lower training budget. It is very important to explicitly evaluate this comparison. As an alternative, you can compare your results directly with the officially released LLaMA-MoE checkpoint.\n3. Llama-MoE v2[2] is another advanced MoE construction method. I would like to see a comparison.\n\nOverall, I believe that ToMoE is an promising work in terms of structure pruning, but its advantage as a MoE construction method still lacks sufficient proof.\n\n- [1] LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training.\n- [2] LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training", "broader_impact_concerns": "1. This work presents a technical method for converting dense large language models into Mixture-of-Experts architectures for dynamic structural pruning. It does not introduce new capabilities (e.g., generation of harmful content, surveillance, or decision-making in sensitive domains) beyond those already present in the original models. \n2. This work does not involve human subjects, personal data, or deployment in high-stakes applications; we believe it raises no significant ethical or societal concerns that warrant a detailed Broader Impact Statement. \n3. The primary impact is positive: enabling more efficient and accessible deployment of existing LLMs with reduced computational cost.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bUa5amIOwx", "reviewer": "Reviewer_N19L", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary\nThis paper proposes ToMoE, a novel method to convert dense models to MoE while mitigating performance loss and taking care of high computational cost. The core summary is as follows. ToMoE pioneers converting dense LLMs into sparse MoE models without updating the original model weights. It leverages dynamic structural pruning to \"uncover\" inherent expert structures within dense models. ToMoE solves a critical limitation of traditional dynamic pruning by ensuring fixed per-token active parameters. \n### Strenths\nThe following are key strengths of this paper.\n1. Low performance cost. ToMoE avoids the high fine-tuning cost of structural pruning (which requires expensive retraining to recover performance losses). ToMoE freezes original model weights and only trains lightweight components (routers, projection modules, hypernetwork), adding minimal parameters.\n2. Practical deployment advantages, for exapmle, fixed latency. Unlike dynamic pruning (variable costs per input), ToMoE’s fixed per-token budget simplifies deployment in latency-sensitive applications.\n3. Relatively high reproducibility. Provides detailed method introduction and specific implementation introduction.\n\n### Weaknesses\nHowever, this paper has some key shortcomings that would be greatly improved if the authors had the opportunity to further clarify or revise them. The following are key weaknessesof this paper.\n1. The formatting of the paper requires significant adjustments. This may not be relevant to comprehension, but it will have a significant impact on rigor. Detailed information is provided in the following sections.\n2. Expert count sensitivity. Table 3 show that increasing N beyond a threshold (e.g., N=16 for Qwen-2.5 7B) provides minimal or no performance gain, even introducing training burdens. This means users must tune N for each model, adding hyperparameter optimization costs.\n3. The baselines and evaluations could be more comprehensive. Table 4 only covers two MoE construction baseline methods. A detailed investigation of more recent dense-to-MoE baselines, such as CMoE and DIVE, in subsequent versions would significantly enhance the paper's persuasiveness.\nIn addition, some in-depth experiments on the ToMoE mechanism can also add depth to the paper. For example, the authors can try to visualize and study whether ToMoE has a preference for specific parameters or heads during the pruning process, and try to derive more insights.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper's core claims include four main areas: 1. ToMoE can convert dense LLMs into MoEs through dynamic structural pruning without updating the original model weights; 2. Joint optimization of the routing module and expert configuration enables efficient MoE construction; 3. This method outperforms state-of-the-art pruning and MoE methods on multiple models (Phi-2, LLaMA-2/3, Qwen-2.5) and multiple tasks without fine-tuning; 4. Three types of regularization, namely expert union, parameters, and load balancing, are crucial for performance.\n\nOverall, Tables 3 and 4 demonstrate the effectiveness of the paper's approach and show that ToMoE can maintain good performance even with a lower activation ratio. However, as shown in the weaknesses in the previous section, a richer baseline and more thorough research would significantly improve the paper's quality.", "audience_yn": "Yes", "audience_explanation": "The paper’s core insights revolve around uncovering latent experts in dense LLMs via dynamic structural pruning to build efficient MoE models, without sacrificing performance or incurring heavy fine-tuning costs.\n1. Dense LLMs inherently contain latent experts, no weight updates needed. Unlike prior MoE construction methods, ToMoE reveals that dense LLMs already have functional experts embedded in their layers. These experts can be extracted purely via dynamic structural pruning, avoiding the computational overhead of re-training model weights.\n2. Dynamic pruning unifies expert construction and router training. Traditional MoE methods split expert building and router learning into two separate stages, leading to suboptimal coordination. ToMoE leverages dynamic pruning to merge these steps.\n3. Fixed per-token computational budget solves dynamic pruning’s key flaw. Prior dynamic pruning methods lack a fixed budget for different inputs, making them incompatible with mini-batch inference or pre-filling (common in LLM deployment). ToMoE’s MoE conversion enforces a fixed per-token budget, balancing efficiency and deployment practicality.\n4. No fine-tuning, yet SOTA performance across models. ToMoE outperforms sota structural pruning (e.g., LLM-Pruner, SliceGPT) and MoE construction methods (e.g., LLaMA-MoE, G-MoEfication) on perplexity and zero-shot tasks—without updating the original dense model’s weights.", "requested_changes": "Overall, this paper is of acceptable quality. It is suggested that the supplementary work mentioned in the previous sections can improve the quality of the paper. **However**, the following formatting issues must be brought to the attention of the authors. These are irrelevant to the understanding of the paper, but can significantly affect its rigor:\n1. Table 5 has a formatting issue. I believe the author intended the left side to have a regular color display, but the current state makes the whole effect very imprecise.\n2. Conventionally, mathematical formulas must end with punctuation. Some formulas in this paper have punctuation, but others do not. For example, Eq 5-10 is correct, but Eq 1-4 is not.\n3. The figures formats in the paper are not uniform. Some figures were imported as pdfs, while others were imported as pngs. For example, the three images in Figure 5 are in different formats.", "broader_impact_concerns": "Overall, this paper's basic reproducibility and methodological description are sufficient. The paper provides detailed implementation details, including hyperparameters and training methods. Beyond that, some minor issues may be considered:\n1. A broader comparison of dense-to-MoE baseline methods would enhance the authority and dissemination of this paper's methods, as mentioned in the previous section.\n2. A wider range of model sizes (larger dense models) would enhance the richness of the evaluation.\n3. The paper does not discuss efficiency issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "c7BjDBikjc", "reviewer": "Reviewer_o8BZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary\n\nThis paper provides a new aspect to dynamically prune dense models to MoE models, which could utilize the scaling and large model capacity properties of MoE to maximize the model performance. ToMoE performs pruning on both attentions and FFNs, and converts the model to a sparse one effectively. Experiments are conducted under different activation settings and model series, and the performance indicate its effectiveness.\n\nStrengths\n\n- ToMoE combines the structural pruning concept to converting dense models into dynamic sparse MoEs, which is a novel contribution.\n- The training process is highly efficient. Comparing to continual pre-training and structural pruning, ToMoE could reach relative high results with minimal training costs.\n- The experimental results are promising, and it achieves superior performance compared to structural pruning, semi-structural pruning, and continual pre-training.\n- The ablation studies are insightful, providing sufficient results to understanding the training dynamics.\n\nWeaknesses\n\n- The main results do not include harder tasks. According to previous dense-to-MoE baselines, MMLU seems to be challenging and have a significant performance drop after model conversion. It would be better to evaluate models in MMLU, BBH, and GSM8K benchmarks.\n- Table 4 is a case study (expert 1 is less utilized), which is less quantitively useful. Please change it to a expert load heatmap with layers as x-axis and experts as y-axis.\n- The definition and initialization methods of the router are missing, and it would be better to explain them in the main content.\n- It is not clear if this method could bring efficiency improvements. Although the number of activated parameters could be reduced by pruning, such a dynamic structural pruning may not be more efficient due to those selection matrices (masks). Could you provide an efficiency analysis (especially on the inference stage)?\n- `without fine-tuning` in abstract and introduction may be over claimed, and please consider revise this statement since ToMoE still has to train routers and selection matrices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The most important claim is that structural pruning would remove necessary model parameters and lead to performance decline. Based on the overall experimental results, I think ToMoE has shown the potential to surpass those baselines.\n\nHowever, the `without fine-tuning` claim may be over stated and should be revised since ToMoE still needs router tuning.", "audience_yn": "Yes", "audience_explanation": "Yes. MoE, model pruning, and model conversion communities may find this paper useful for their research or take it as a baseline.", "requested_changes": "- WikiText, WikiText-2, Alpaca, CodeAlpaca are not properly cited or referred.\n- The conclusions in Table 11 and the others are insightful, please consider summarize some takeaways at the end of the introduction section to strengthen the overall contributions.\n- Formats\n    - LLama → LLaMA\n    - For vectors, please omit $\\times 1$ in shapes, e.g. \\textbf{1}_{d_{mid}} \\in \\mathbb{R}^{d_{mid} \\times 1} → \\textbf{1}_{d_{mid}} \\in \\mathbb{R}^{d_{mid}}\n    - than → then before Eq. 7\n    - missing comma at the end of equations since there are `where` following after\n    - some symbols are conflicting, e.g. the `f` is both a regression function and a model", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)