HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space
Authors: Ke Li
Zheng Yang
Zhongbin Zhou
Xuefeng
Zhonglin Jiang
Wenxiao Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Pruning; MoE
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memory requirements, limiting practical deployment. While existing pruning methods primarily focus on expert-level pruning, this coarse granularity often leads to substantial accuracy degradation. In this work, we introduce HEAPr, a novel pruning algorithm that decomposes experts into smaller, indivisible atomic experts, enabling more precise and flexible atomic expert pruning. To measure the importance of each atomic expert, we leverage second-order information based on principles similar to the Optimal Brain Surgeon theory. To address the computational and storage challenges posed by second-order information, HEAPr exploits the inherent properties of atomic experts to transform the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity from $\mathcal{O}(d^4)$, where $d$ is the model’s dimensionality, to $\mathcal{O}(d^2)$. HEAPr requires only two forward passes and one backward pass on a small calibration set to compute the importance of atomic experts. Extensive experiments on MoE models, including DeepSeek MoE and Qwen MoE family, demonstrate that HEAPr outperforms existing expert-level pruning methods across a wide range of pruning ratios and benchmarks. Specifically, HEAPr achieves nearly lossless compression at pruning ratios of $20\% \sim 25\%$ in most models, while also reducing FLOPs by nearly $20\%$. The code can be found at [https://github.com/LLIKKE/HEAPr](https://github.com/LLIKKE/HEAPr).

Published as a conference paper at ICLR 2026
HEAPR: HESSIAN-BASED EFFICIENT ATOMIC EXPERT PRUNING IN OUTPUT SPACE
Ke Li', Zheng Yang", Zhongbin Zhou², Feng Xue Zhonglin Jiang", Wenxiao Wang
School of Software Technology, Zhejiang University
2 FABU Inc.
3 Hangzhou Kuaidi Science and Technology Co., Ltd.
4 Geely Automobile Research Institute (Ningbo) Co., Ltd
{1 1like2248, wenxiaowang jed.uu.eduuu... on
ABSTRACT
Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memory requirements, limiting practical deployment. While existing pruning methods primarily focus on expert-level pruning, this coarse granularity often leads to substantial accuracy degradation. In this work, we introduce HEAPL, a novel pruning algorithm that de- composes experts into smaller, indivisible atomic experts, enabling more precise and flexible atomic expert pruning- To measure the importance of each atomic expert, we leverage second-order information based on principles similar to the Optimal Brain Surgeon theory. To address the computational and storage chal- lenges posed by second-order information, HEAPr exploits the inherent properties atomic experts to transform the second-order information from expert parame- ters into that of atomic expert parameters, and further simplifies it to the second- order information of atomic expert outputs. This approach reduces the space com- plexity from (d'*), where d is the model's dimensionality, to (dd)). HEAPP re- quires only two forward passes and one backward pass on a small calibration set to compute the importance of atomic experts. Extensive experiments on MoE mod- els, including DeepSeck MoE and Qwen MoE family, demonstrate that HEAPT outperforms existing expert-level pruning methods across a wide range of pruning ratios and benchmarks. Specifically, HEAPr achieves nearly lossless compression at pruning ratios of 20% 25% most models, while also reducing FLOPs by nearly 20%. The code can be found at https//gihhubbommL///KK//II
INTRODUCTION
Mixture-of- Experts (MoE) models have recently emerged as promising alternative dense large language models (L..Ms), replacing dense feed-forward layers with sparsely activated experts and dynamic routing. This design allows MoF models to match or surpass the performance of dense LLMs while activating only fraction of parameters during inference (Fedus et al. [022 Zhu al., 2024 Liu et al. 2024a). making them particularly attractive for large-scale, concurrent deployment. However, while sparse activation reduces computational cost, it exacerbates memory requirements. For example, DeepSeek- V3 (Liu et al., 2024a) activates only 37B parameters per inference, yet all 671B parameters must still be stored in GPU memory, resulting in prohibitively high deployment costs. Notably. MoE layers typically account for over 97% of total model parameters, and they represent the dominant storage bottleneck. Therefore, compressing MoE layers becomes critical to overcoming inference inefficiency and making deployment feasible in resource-constrained devices.
Model pruning has been widely explored as an effective compression strategy to reduce storage and improve efficiency. Yet fundamental trade-off persists: fine-grained pruning typically preserves ac- curacy but yields limited speedups on hardware, whereas coarse coarse-grained grained pruning directly enables ac- celeration but often incurs obvious accuracy loss. Within MoE models, parameter sparsification (Xie) jet al.) (2024) faces similar limitations, as hardware inefficiencies constrain its practical benefits. "Corresponding author

---

Published as a conference paper at ICLR 2026
Consequently, recent research has shifted to- ward expert-level pruning, offering more di- rect gains in both acceleration and memory re- duction. Existing expert-level approaches at this level can be broadly divided into expert dropping and expert merging. Expert dropping methods (Lu al. 2024; Huang et al. 2025) completely remove experts deemed unimpor- tant, but relying solely on calibration to dis- card entire experts risks losing valuable com- plementary expertise, consequently often lead- ing to notable performance degradation. Ex-
Atomic Expert Prening
Esport
WGate
Atamic Expert
Expert
Internal View
W
W down
Atomic Expert level
pert merging methods (Li et al. 2024; Chen eet al.] 2025 Huang et al.) (2255 instead aim to consolidate functionally similar experts to more effectively preserve overall model capac- ity. However, their clustering- based similarity measures are notoriously unstable, and naive merging strategies (e.g., averaging or frequency- based weighting) often introduce destructive pa- rameter conflicts, resulting in suboptimal and inefficient outcomes. To alleviate these critical con- flicts, recent decomposition based approaches (Li et al. [225cc] Gu et al., (225)) represent individual experts as a mixture of shared and specialized components. While this advanced framework helps to preserve model capacity, it still requires computationally expensive decomposition and merging operations, and unfortunately still incurs a non-negligible accuracy loss.
Illustration of atomic expert-level prun- ing, which removes the t-th column from the W gate and up matrices, and the corresponding t-th row from the Woown matrix.
Figure 1:
To identify pruning units that are more flexible than expert-level pruning, we introduce the concept of an atomic expert, in which each expert is decomposed into smaller, indivisible units. Concretely, each atomic expert is defined by jointly grouping the relevant columns of W Waate, and the corresponding row of Wydown(as shown in Figure D}. The output of a full expert can be represented as the sum of outputs from multiple atomic experts. Pruning this granularity directly removes atomic experts, thereby isolating pruning effects and avoiding interference with remaining compo- nents. By eliminating atomic experts that contribute little to final predictions, inference efficiency can be improved and deployment overhead reduced in a more straightforward and essential way.
The key challenge now lies in how quantify the importance of each atomic expert overall per- formance. To tackle this problem, we propose HEAPr, a principled framework for efficient and high- nigh-performance atomic expert pruning, Our approach is inspired by the classical Optimal Brain Surgeon (OBS) theory (Hassibi et al.) 1993 LeCun et al. 1989), which approximates the effect of weight pruning via Taylor expansion of the loss function and leverages second-order information to identify parameters with minimal contribution. However, applying OBS to modern deep archi- tectures is computationally prohibitive due to the cost of Hessian estimation, and this is why layer- wise Hessian estimation has become widely adopted (Dong et al.) [017]] Frantar & Alistarh 2022 Frantar et al. 2023). Despite this, the space complexity of Hessian estimation at the expert level remains O (33dmodd ) ? I which is still unacceptable. Therefore, we propose two optimiza- tions to improve Hessian matrix computation. First, by decomposing experts into atomic experts, demonstrate that the second-order derivatives of parameters between different atomic experts zero. This observation allows us to significantly reduce the space complexity of the Hessian matrix, lowering O ((3dmode1) dimerr). ). Second, we further optimize the Hessian matrix by shifting the pruning constraints analysis from the parameter space of atomic experts to their output space. This shift enables us to leverage the Fisher information matrix, which theoretically equivalent to the expected Hessian but significantly more efficient to compute (Bishop & Nasrabadi] 2006 Singh & Alistarh] 2020), and by combining this with a Taylor expansion of the atomic expert function, we can estimate each atomic expert's contribution to the final loss. This further reduces the Hessian complexity to (ddand)) for each expert, ensuring high efficiency in both computation and storage. HEAPT is not only tractable but also highly efficient: all atomic expert importance can be computed with just two forward passes and one backward pass on a small calibration set. We evaluated HEAPT on seven zero-shot tasks, achieving nearly lossless pruning with 20% pruning on DeepSeekMoB- W.MMoE-A2.7B--hee 7BCha, and 40% pruning on Qwen2-57B-A
16B-Base, 25% pruning on
dimer the intermediate dimension after the W.p transformation, and dmudel the hidden size of the model.
2

---

Published as a conference paper at ICLR 2026
Additionally, on the latest (wen3-30B-ANe model, the average accuracy only drops by 0.03 at 25% pruning ratio. Overall, our contributions are summarized as follows:
We introduce second-order approximation scheme for atomic expert pruning in MoE models, which transforms the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity of second-order infor- mation from O ((3dmods) )= to (dmode))
Building this efficient scheme, we propose HEAPT, a highly efficient and scalable prun- ing algorithm that accurately estimates the importance of all atomic experts with just two forward passes and one backward pass small calibration set.
We conduct extensive experiments on DeepSeckMoE- 1eae--16--Basse BoB-Base, Chat, Qwen2-57B-A -BB-1BB and Qwen3-30B-A3B across diverse benchmarks. HEAPr out- performs current SOTA methods and achieves nearly lossless compression at pruning ratios of 20%-25% in most models, while also reducing FLOPss by nearly 20%.
RELATED WORKS AND PRELIMINARY
Mixture of Experts Compression. Model compression for MoE architectures has recently at- tracted growing attention due to the remarkable performance of MoE models. MoE-Pruner (Xic) al. 2024) performs weights sparsification based on activation magnitude, weight magnitude, and router importance, yet its acceleration is gerdware-dependern and relies on distillation to recover accuracy. Expert-level pruning has been more extensively explored due its friinnyy acceleration. NAEE (Lu et al. 2024) selects subset of experts to minimize calibration error, but this can lead to overfitting and the loss of specialized knowledge. Similarly, MoF-12 (Yang et al., 2024) combines expert pruning with low-rank decomposition, yet requires additional fine-tuning for recovery. To alleviate such issues, expert merging methods aim to retain similar experts rather than discarding them. MC-SMoE (Li et al. 2024) merges experts by clustering based routing poli- cies, and HC-MoE (Chen et al. (025) does so by grouping experts with similar outputs. However, limited expert similarity makes merging prone to parameter conflicts. EEP (Liu et al.] (024b) uses gradient-free evolutionary search to combine expert dropping and expert merging, cutting SMoE experts and active experts while maintaining or improving downstream performance. To further ex- ploit redundancy. D (Gu et al. 2025) constructs shared expert via weighted combinations and compresses residuals through low-rank decomposition, while Sub-MoE (Li et al. (2025a) applies SVD to extract shared subspace across experts, both of which require computationally expensive decomposition and merging operations. We decompose the expert into atomic experts and propose HEAPr, a method that measures importance by utilizing a second-order approximation to assess the importance of atomic experts. This approach enables more flexible pruning units and provides an efficient highly algorithm, preserving model performance while eliminating the need for retraining.
Optimal Brain Surgeon in Pruning. The OBS framework (Hassibi et al. 1993; LeCun et al,, 1989) approaches pruning
optimization problem, aiming minimize the increase in the loss function when a parameter is removed. Consider a model that has already been trained and con- verged, with parameters 0 and corresponding loss E(O). We can analyze the effect of perturbing the parameters by analyzing the second-order Taylor expansion of the loss function around 0. Specif- ically, the change in the loss At when perturbing the parameters by 50 is given by the following:
Al E(O + 50) ((0) Ve(
T 50
500T H00 + 501))
2
where H is the Hessian matrix of second derivatives of the loss with respect to the model parameters. Since the model has already converged to local minimum of the loss function, the first-order term can be removed (Vε(0) = 0), and the higher-order terms can be ignored for small perturbations.
For pruning, the constraint is 0g + 80,- 0 for the target, leading to the optimization problem as:
min 88,
} 3,80 H50,
s.t. 80g + 0g 0,
(2)
where 9 denotes the index of the pruned parameter. Solving this optimization problem yields the minimal increase in loss from pruning parameter 8y. which is Al
0²
H 1
3

---

Published as a conference paper at ICLR 2026
Directly computing the full Hessian in deep neural networks is practically infeasible. Existing OBS methods to adopt significant approximations. For instance, K-FAC approximation (Martens & Grosse 2013) provides an efficient approximation of second-order information and Hessians are computed layer-wise to guide pruning (Dong et al.] 2017] Frantar & Alistarh 2022] Prantar et al.) 2023). Previous work (Singh & Alistarh| 2020) shows that the Fisher information matrix serves reliable Hessian estimate and allows for more efficient computation. Some apply OBS to structured pruning (Yu et al. 2022), but these efforts limited they consider only the trace of the Hessian.
3 METHOD
3.1 ATOMIC EXPERT IN MIXTURE-OF-EXPERTS.
The MoE architecture has been widely adopted in LLMs as a replacement for the dense feed-forward network layer, which effectively increases the model capacity while reducing the number of activated parameters. Formally, given an input token representation x E [dodd, the output of the MoE layer with Neap experts is defined as:
g(x) - (g) (x), (x),....9)(000. .x(x)))),,,,() Top -x( W IR""
y
where W/Rate IR Napx duuds produces router scores and Top-x(-) denotes the router function that selects the top-K experts. Each expert (-) is a gated feed-forward block:
Ex(x) W.doom SILU(W; gate x) @ (W;"x)) x)].
where WiT Wgate E &Rdux Wodown E Rd Xdie C denotes the Hadamard product, and SiLU(-) is the SiLU activation. Within each expert, computations can be decomposed into atomic wP experts. Let wig and wgaae denote the j-th rows WiI and Wgate, respectively, and let doww denote the j-th column of W.down Then the j-th atomic expert of the i-th expert is
1 (x) down
wis
SiLU (WL xx) (w
(5)
where w. i.j1 Wiy E R Ixdast and wowwn Rdaaaxx1 Consequently, each expert is linear combi- nation of its atomic experts:
gate
dusse
Ex
(x).
In this framework, each expert Ex(-) can be viewed linear combination of its atomic experts. This decomposition allows pruning at the atomic expert level without compromising the other atomic expert structure, leading to both computational acceleration and deployment efficiency directly.
3.2 ATOMIC EXPERT IMPORTANCE ANALYSIS IN THE OUTPUT SPACE
Importance of Atomic Experts. As discussed in Section 2 the OBS theory provides an excellent framework for analyzing the impact of parameter pruning on model performance. However, its major limitation is the large Hessian matrix, even when only computed layer-wise. In the case of MoE, directly applying OBS at the expert level is still infeasible, as it requires constructing an exceedingly large Hessian with space complexity of ((3dmodel dz per expert, leading to prohibitive computation and storage costs. Fortunately, by decomposing the expert into smaller atomic experts, property is revealed: the parameters of different atomic experts are decoupled, i.e.,
8 Ex a 805) 200) 20G)
Pell (x)
0, Vi
where ( E 3d represents the parameters of the i-th atomic expert. This means that the cross-Hessians between different atomic experts are zero, which provides a valuable and simplifying property that allows us to focus exclusively the Hessian of each individual atomic expert with
4

---

Published as a conference paper at ICLR 2026
respect its own specific parameters. Based on this observation, the second-order Taylor expansion of the change in the loss function with respect to each expert's parameters can be expressed as:
dune
Al 80" H80
( 50
here, Reddansttrdae denotes the parameters of given expert, and H is the corresponding Hessian matrix with space complexity O (3dmmode dimer)²), And each () represents the Hessian for the i- th atomic expert. This decomposition leads to a significant reduction in the complexity of summing over the Hessians E H which is reduced O ((3dmodel) dmnee):
However, the resulting Hessian matrix computation remains unacceptable due to its high computa- tional and storage cost. To further alleviate the bottleneck, introduce second optimization that reformulates the pruning constraint. The original parameter-space constraint (equation 2 implies that the atomic expert's output ep (x; Op + d0p), where Op E Rd denotes the parameters of the atomic expert to be pruned, would be zero for every possible input x. Although theoretically sound, enforeing such a universal constraint is computationally infeasible. This motivates a more targeted reformulation: for specific token x, what is the minimum loss increase A required to force the expert's output to zero? To make this question concrete, we impose the per-token constraint ep (x; Op + 80p) = 0, treating as given. Since the atomic expert functions are not optimized with respect to the parameters Op through gradient descent, applying a Taylor expansion of the atomic expert functions around Op results in the first-order term dominating, yielding:
ep [x; Op + 80p) = (x Op) JpdOp= =0 =0
where Jp& E Rd desos x3d denotes the Jacobian of ep (x; Op). This leads to the following problem:
deuur
min 6
s.t. Jp + ep = 0.
(10)
2
To solve the problem in equation 10 consider the LLMs trained with a negative log-likelihood loss & (e.g., cross-entropy loss). In this setting, the Fisher Information Matrix F is equivalent to the expected Hessian (Bishop & Nasrabadi 2006), providing a computationally efficient alternative:
F E E =
(V
(11)
where é the sample-wise loss. Previous work (Singh & Alistarh, 2020) has shown that for well- converged neural networks, few hundred representative samples are already sufficiently reliable to estimate TE [H. Expanding the gradient of with respect to the parameters gives Ve, = Jpgp, where gp dmodel is the gradient of the loss with respect to the pruned atomic expert output ep. Substituting this expression into the objective equation
10 yields the expected loss increase when pruning the atomic expert ep, with 80(1) = 0 for all atomic experts not pruned:
S p 80p Rs EX-D [ ep E[grg,] ep].
(12)
This leads us to define the Importance of the atomic expert ep as
= Al x Ex~D
[ e Elgr,
(13)
where smaller indicates that the corresponding atomic expert has less impact on the overall model loss and should be pruned with higher priority. The detailed derivation is provided in Appendix
At this point, we have shifted the analysis from the parameter space of atomic experts to their output space, further reducing both computational and storage requirements. Next, we introduce a remarkable property of atomic expert outputs: the outputs of atomic experts within the same expert share identical gradients, i.e..
at i (x)
ae OE(x)'
Vi (1, (1...., dmmer) ) ( E F.
(14)
This property allows us to further significantly reduce storage requirements. Instead of maintaining separate gradient covariance matrices for each atomic expert, only need to store single matrix per expert. As a result, the space complexity for computing the importance of the atomic expert within the same expert is drastically reduced to O (dhodd)) enabling efficient storage management.
5

---

Published as a conference paper at ICLR 2026
Global Ranking of Atomic Experts. The metric of each atomic expert's importance has been introduced by equation [3] Next, an important question how to rank the importance of the atomic experts. Consider that our importance metric evaluates experts based their overall contribution to the model's change in the loss function (as shown in equation 2). it provides a natural basis for effectively compare experts across layers consistently, ensuring
global ranking. This allows that pruning decisions are made based on the entire model's behavior rather than isolated layer-wise.
3.3 HEAPR AL.GORITHM
Building on the above analysis, we propose HEAPI, a pruning strategy for MoE feedforward layers that ranks the importance of atomic experts (as defined in equation 13) and removes those with negligible contribution to the overall loss. To effectively compute the importance of atomic experts, we leverage small but representative calibration set 'D and estimate importance in two stages.
Shared Gradient Covariance Estimation. For given expert E,, the gradients of the loss with respect to all its constituent atomic experts' output are identical. Therefore, rather than performing redundant computations, we execute single backward pass to obtain the gradient for the expert's output, gr, a This shared gradient is used to compute gradient covariance matrix G; for all atomic experts belonging to E,, accumulated over the subset tokens T, C D routed to E;:
G. =
1 T
Σ XET;
gE, (x)gE, (x)
(15)
Importance Computation. Subsequently, during forward pass, we compute the importance for each individual atomic expert eμ. Although the gradient covariance matrix G; is shared among all atomic experts within same expert E,, the output of each atomic expert, e₃ (x), remains unique. This difference in output allows us to distinguish their individual contributions. The importance of an atomic expert eş (where e₁ E E2) is calculated by averaging over the tokens processes:
8k
T
Σ XET.
(x) (xe)(x).
(16)
This approach relies solely standard forward and backward computations, making it both excep- tionally time- and memory-efficient. The space complexity of each gradient covariance matrix is only (dlavse)) significantly alleviating the storage bottleneck. After computing the importance & across all atomic experts in the model, we perform a global ranking and prune the lowest r% of ex- perts across all MoE layers. The complete and optimized procedure is summarized Algorithm T
Algorithm HEAPF: Hessian- Efficient Atomic Expert Pruning in Output Space
Require: MoE model fo. calibration set 'D pruning ratio r
Ensure: Pruned model fo'
1: for each expert E; do
b> Stage 1: Gradient Covariance Estimation
2: Collect routed tokens TV
3: Compute shared gradient gE, aE;
at
4:
Compute G.
1 |T. ΣxET, gE, (x)gE; (x)T
Space complexity O(d)
5: end for
6: for each atomic expert e₁ in E, do
>> Stage 2: Importance Computation
7: Compute 8 = ... EXET, 2 (x) (x)x
8: end for
9: Global rank {5x] and prune lowest r% across all experts
10: return Pruned model for
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Models and Setup. We evaluate our approach on a broad spectrum of model architectures and scales assess its generality and effectiveness, including Dee (Dai et al.

---

Published as a conference paper at ICLR 2026
Table 1: Performance of HEAPr with DeepSeckMoF:166-BBBal L.ven..5-MoF.. M...Mo.-A2..B--C
Qwen2-57B-A144 and Qwen3-30B-A3B on seven zero-shot tasks, reported in terms of accuracy. The results marked with are obtained from the official implementation.
Ratio
Method
Wiki!
PTBL
Openh.
ARC_e
WinoG.
HellaS.
ARC.e
PIQA
MathQ
Avg-f
DeepSeekMoE-IHe
0%
Original
6.38
9.47
0.32
0.76
0.71
0.58
0.45
0.79
0.32
0.56
NAEE
9.44
15.02
0.32
0.71
0.66
0.55
0.40
0.77
0.29
0.53
MoE-12
7.69
11.59
0.26
0.71
0.68
0.49
0.38
0.73
0.29
0,50
20%
MoE-SVD
6,92
10.48
0.75
0.70
0.53
0.76
0.31
D²-MoE
6.84
11.10
0.74
0.69
0.55
0.76
0.31
6.54
9.88
0.32
0.76
0.71
0.57
0.45
0.79
0.32
0.56
NAEE
8.55
14.47
0.67
0.67
0.41
0.69
0.26
0,46
MoE-I2
9.73
15.75
0.64
0.66
0.31
0.26
40%
D2-MoE
7.93
14.07
0.26
0.69
0.65
0.45
0.36
0.72
0.28
0.49
HEAPP
6.80
10.86
0.30
0.74
0.69
0.52
0.41
0.76
0.30
0.53
D.wenI.5MMoF..e
Chat
Original
8.12
12.97
0.70
0.59
0.79
0.35
0.54
MC-SMoE
12.76
17.45
0.25
0.65
0.65
0.53
0.37
HC-SMoE
11.62
16.39
0.66
0.76
0.29*
0.50
25%
Sub-MoE
9.48
14.84
0.30
0.69
0.66
0,56
0.37
HEAPT
8.31
14.12
0.32
0.69
0.67
0.56
0.38
0.76
0.35
0.53
MC-SMoE
5c2
le3
0.18
0.33
0.52
0.29
0.19
HC-SMoB
25.50
38.18
0.23
0.61
0.65
0.47
0.35
0.58
0.23*
0.45
50%
Sub-MoB
17,51
29.00
0.25
0,58
0.58
0.46
0.25
HEAPT
9.24
17.58
0.27
0.64
0.64
0.46
0.33
0.71
0.33
0.48
Awen3-30B-A331
U
Original
8.64
15.40
0.34
0.79
0.71
0.60
0.54
0.79
0.59
0.62
IIC-SMoE
18.86
31.11
0.22
0.64
0.61
0.40
0.35
0.59
0.41
0.46
25%
Sub-MoE
13.59
23.48
0.25
0.70
0.66
0.47
0.44
HEAPE
9.10
16.80
0.33
0.76
0.70
0.55
0.49
0.78
0.50
0.59
IIC-SMoE
72.33
162.99
0.13
0.44
0.50
0.29
0.23
0.44
0.32*
0.34
50%
Sub-MoB
21.05
43.19
0.23
0.68
0.63
0.41
0.40
HEApF
11.22
26.29
0.25
0.67
0.63
0.38
0.41
0.67
0.36
0.48
Qwen2-57B: A1-411 14B
0%
Original
5.12
9.18
0.33
0.75
0.74
0.63
0.46
0.81
0.39
0.59
NAEE
6.81
11.34
0.31
0.73
0.73
0.55
0.46
0.76
0.36
0.55
MoE-12
24.90
77.05
0.26
0.70
0.46
0.71
0.41
0.75
0.30
0.51
40%
D.MMoE
8.19
11.23
0.33
0.75
0.75
0.61
0.45
0.79
0.36
0.58
HEAPT
5,92
9.34
0.33
0.75
0.74
0.63
0.46
0.81
0.39
0.59
2024), Qwenl 5-.-0--A2.7B-CHE TTeam 2024a), Qwen2-57B-A143
A4B [Team] 2024b), and Qwen3- 30B-A3B. All experiments are calibrated on WikiTexx-2 using 128 sequences of 2048 tokens (see Appendix for details). Notably, our method introduces no additional tunable hyperparameters.
Baselines. For our comparisons, we evaluate seven recently proposed perf-mmeeeeeance compres- sion methods, including expert dropping (NAEE (Lu et al. 2024), MoE-1 (Yang et al. 2024 expert merging (MC-SMoE [i al. 2024). HC-SMoE (Chen al. 2025)), and expert decomposi- tion (Sub-MoE (Li et al. 2025a). D MoE (uu et al. 2025), MoE-SVi (Li et al. 2025b)). Baseline data were collected from prior publications, prioritizing original sources, and details are provided in Appendix B Missing data for open-source implementations
obtained from official code.
Evaluation. We report results seven zero-shot benchmarks using the T1-Evaluation-Harreee
(version 0.4.7) (aoo et al. 2024), including HellaSwag (Zellers et al. 2019). MathQA (Amini et al.
2019), Op (OBQA) Mihaylov et al. 2018), PIQA (Bisk et al. (2020). WinoGrande (Sak- aguchi et al., 222T), ARC-Easy and ARC-Challenge (Borakko et al. 2018). These tasks collectively enable repeated and consistent evaluation of our method across varied domains.

---

Published as a conference paper at ICLR 2026
4.2 MAIN RESULTS
Compression Performance. As shown in Table 1 HEAPr achieves exceptional performance across various MoE models and pruning ratios. Notably, our method delivers near-lossless com- pression. At pruning ratios of 20% 25%. HEAPT matches the performance of the original models on DeepSeckMoE-162 Base and QwenI.5-MofE Mao..-----AA2.--Chh.e More impressively, on Qwen2-57B- A14B, HEAPT maintains performance almost identical the original model even at high 40% compression ratio. In contrast, our method outperforms recent approaches such as Sub-MoE, D2- MoE, and NAEE under the same compression ratio. Furthermore, on the latest Qwen3-30B-A3B model, HEAPT incurs only a minimal performance loss at a 25% pruning ratio, with the average accu- racy dropping slightly from 0.62 to just 0.59. These results strongly highlight the unique advantage of HEAPT in pruning at the atomic expert level, enabling substantial model efficiency improvements while maintaining and preserving core model performance effectively.
Compare to CAMERA-P. In this section, we compare HEAPr with a concurrent related work CAMERA-P (Xu et al. 2025), which evaluates the importance of an atomic expert using the con- cept of decoding-ting time energy. Specifically, the importance of the j-th atomic expert in the i-th expert is given by Ei.j = (1ob, + a [| 1|,, where Pi, SiLU E. x) (w.">)) CAMERA-P uses heuristic approach to measure atomic expert importance based on the output magnitudes on a calibration set. However, this method has two main drawbacks: it is local, neglect- ing atomic experts' impact on overall model performance and cannot be globally applied for pruning due to varying activation magnitudes across layers. In contrast, our method HEAPP, built upon the OBS framework, leverages the Hessian matrix to assess the impact of atomic experts on the overall model performance. And HEAPr yields a globally consistent importance metric for atomic experts, thereby enabling principled global pruning, as analyzed in Section 3.2 Table 2 performance of HEAPP and CAMERA-P released its open-source implementation, we evaluate HEAPT using the acc_norm as reported in the original paper. At 20% pruning ratio, HEAPP outperforms CAMERA-P by an average of 1.2 in accuracy. Even when applying the same layer-wise pruning strategy as CAMERA-P, HEAPT still achieves average accuracy improvement of 0.5. Notably, 40% pruning ratio, the performance gap between the two methods narrows. We attribute this to the reduced redundancy at higher pruning ratio, where the non-essential atomic experts identified by both methods become nearly identical.
gate
we compare the DeepSeekMoE- 16B--6ass Since CAMERA-P has not
Performance Boundary of Pruning. Fig-
0.7
0.7
ure 2 reports the performance of HEAPT on DeepSeckMoE-16B--
-D-
Average Arturacy
PLOPs Reduction Ratin
using a random 128- sample subset of WikiText-2 with 2048 to-
0.6
0.6
Average Accuracy
0.5
kens under different compression ratios, where the ratio denotes the fraction of parameters re- moved relative to the full model size. For compression ratios below 0.4, the pruned mod- els retain 93% of the baseline accuracy while achieving a 20% reduction in FLOPs. In this regime, the accuracy curve is nearly flat, in- dicating substantial redundancy among atomic experts and suggesting that HEAPT can effec- tively identify and remove them. As compres- sion increases further, accuracy degrades grace- fully, highlighting clear trade-off between ef- ficiency and performance. Even an extreme compression ratio of 0.9, the model retains about 38% of the baseline accuracy while achieving a 55% reduction in FLOPs. These results demonstrate the robustness of HEAPT under moderate prun- ing and its effectiveness in enabling aggressive acceleration while retaining reasonable performance.
1.10x
0.4
0.4
0,20x
0.3
0.3
0,21
0.2
0.13x
0.15x
0.2
0.1
0.0
0.2
0.8
0.0
Compression (s)
Figure 2:
Performance of DeepSeekMoE- BeotsMMME---BB-BE- 16B-Base under varying compression ratios, with corresponding FLOPs saving on Weki data.
4.3 ABLATIONS
Global ys. Layer-wise Pruning. As shown in Table layer-wise pruning (CAMERA-P, HEAPT- L.) ranks the importance of atomic experts within each MoE. layer and prunes the bottom r%, whereas
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of layer-wise pruning (CAMERA-P and HEAPI-L) global pruning (HEAPI-G) on DeepSeckMoE-1GP Base and 1-o.l-5---E-2227555 tasks, with acc norm reported for DeepSeekMoE- 16B-Bse and accuracy for others.
across seven zero-shot
Ratio
Method
Openb.
ARC.e
WinoG.
HellaS,
ARC_
PIQA
MathQA
Average
DeepSeckooE166-44
CAMERA-P
44.00
71.80
70.17
75.02
45.56
78.62
31.46
59.52
HEAPT-L
44.01
72.64
70.01
75.55
47.27
79.76
30.92
60.03
HEAPI-G
44.80
73.73
71.43
76.57
47.01
79,82
31.42
60.68
CAMERA-P
43.20
70.71
68.51
69.04
42.24
75.41
29.01
56.87
40%
HEAPE-L
42.80
70.45
68.35
68.10
43.69
76.
29.41
56.99
HEAPT-G
41.40
72.05
69.06
70.79
45.05
76.39
29.85
$7.80
HEAPT-L
30.60
66.58
66.77
55.09
38.05
76.61
33.97
$2.52
25%
HEAPT-G
31.80
68.60
67.22
55.67
37.56
76.39
34.87
53.59
HEAPT-L 27.00
63.26
64.01
47.00
33.70
69.80
32.29
48.15
50%
HEAPI-G
27.01
63.89
64.32
46.35
34.22
70.86
33.37
48.57
global pruning (HEAPT-G) ranks the importance of all atomic experts across the entire model. Com- pared with CAMERA-P, our layer-wise pruning HEAPr-L. achieves superior performance, indicat- ing that the atomic expert importance metric, as derived from equation [3] provides more effective pruning criterion within individual layers. Furthermore, HEAPr-G, by leveraging global pruning and importance scores across all layers, achieves even stronger and more consistent results, validating the global consistency of the atomic expert importance thoroughly analyzed in Section 3.2
Impact of Pruning Granularity, To better demonstrate the importance of atomic expert decom- position, conduct an ablation study comparing pruning the atomic expert level and expert level. Based on equation 8 the importance score for an expert computed via equation [3] can be expressed the sum of the importance scores of its constituent atomic experts. As reported in
33 expert-level pruning behaves similarly to Expert Dropping (Lu et al. 2024): the activated experts unchanged after pruning does not lead to noticeable computational speedup. In contrast, pruning at the atomic expert level reduces the dimensionality within each expert, thereby enabling real acceleration. Empirically, atomic-level pruning consistently outperforms expert-level pruning across multiple benchmarks, highlighting its effectiveness and necessity.
Empirical Correlation of Loss and Atomic Expert Importance 81.- In Section 2 follow- ing the principles of OBS theory, we define the atomic expert importance score 8y for ep based on the expected change in model loss. The goal of this metric is to identify atomic experts whose removal induces the smallest increase in the overall loss. However, because both the OBS formulation and the output-space approxi- mation neglect higher-order terms, an exact nu- merical match between loss change Al is not expected. Importantly, pruning ultimately requires reliable ranking of atomic expert importance rather than an ac- curate prediction of At. To evaluate the rank- ing quality of ss. we infer the atomic experts on the calibration set and then group them into 10% bins according to their original indices. As shown in Figure 3 the observed loss increase Al for each bin closely follows the cumulative trend of the corresponding normalized importance scores sk- This result indicates that, despite the approximations involved, the proposed 8ı metric
1.0
Norma lired Wildd Iscore
2.188
2.100
0.8
2.122
0.
2.126
9.178
HC 1256
>b Nervolioe Soor
2.130
096-10%
10% 30%
20%-3166
300--49%
40% 1th
30% 409
Rook Rok-7pp 70%%
70% o09.
50K-309
90% 109%
and the empirical
Expert
Figure 3: Consistency between atomic expert normal- ized importance score 3 and the change in loss. The figure plots actual loss increase Af observed upon pruning atomic experts within 10% quantile bins (or- dered by original expert index) against the cumulative importance
sk.

---

Published as a conference paper at ICLR 2026
provides a globally consistent and reliable ranking of atomic experts. It effectivell identifies atomic experts whose removal causes minimal performance degradation, thereby offering solid basis for the HEAPP algorithm and supporting the accuracy of our pruning decisions.
Table 3: Comparison of pruning granularities at the expert level and the atomic expert level, where expert importance computed by summing the importances of its atomic experts, evaluated across seven zero-shot tasks. FL.PPs n. denotes the FL.OPs reduction ratio. Results are reported on
DeepSeekMoF- 16B-Base.
Ratio
Level
FLOPs n.
Wiki,
Openb.
ARC.e
WinoG.
HellaS.
ARC.c
PIQA
MatQQA
Expert
0%
6.90
31.40
75.76
71.35
57.99
44.31
78.40
30.75
20%
Atomic Expert
8%
6.54
31.54
75.88
71.43
57.39
44.62
79.05
31.52
Expert
0%
8.00
30.60
73.19
63.93
51.15
42.49
77.09
28.24
40%
Atomic Expert
21%
6.80
30,00
73.78
69.06
52.29
40.61
76.50
30.12
Impact of Calibration Data. Figure the average aceuracy with error bars over ran- dom subsets of the calibration data, indicat- ing that the performance of our HEAPT algo- rithm is largely unaffected by the choice of calibration data, whether they are WikiText-2 C4 dataset. This highlights the remarkable robustness and generalizability of our method, it consistently performs well across differ- ent calibration corpora and domains. Further- more, the table also explores the significant im- pact of calibration set size pruning perfor- mance. the number of calibration samples increases, the model's performance improves consistently, indicating that larger calibration sets offer richer statistical coverage, which pro- vides more reliable and informative signals for effective compression. These results strongly suggest that our method is not only robust to variations in calibration data but also benefits from the inclusion of additional diverse samples.
Average Accuracy
0.55
0.53
Wilkos
32
128
256
Calibration samples
Figure Performance of DeepSeckMoF- 16B-Base under 20% compression ratio, using calibration data randomly sampled from WikiText-2 and C4.
CONCLUSION
In this work, introduce HEAPr, novel method that refines expert-level pruning MoE models by enabling more flexible and fine-grained pruning strategy at the atomic expert level. Inspired by the principles of the Optimal Brain Surgeon theory, we evaluate the importance of atomic ex- perts using second-order information. By transforming the analysis from the expert parameter space to that atomic expert parameters, and further shifting to the atomic expert output space, we significantly reduce the computational and storage bottlenecks associated with the second-order in- formation matrix. HEAPT requires only two forward passes and one backward pass to efficiently compute the importance of atomic experts. Extensive experiments various modern MoE models demonstrate that HEAPP outperforms state-of-the-art pruning methods, achieving near-lossless prun- ing with pruning rates of 20% 25%. More importantly, perspective on MoE expert pruning, which we hope will contribute to deeper, more comprehen- sive understanding of MoE models. Future work will explore large-scale experiments across wider range of models and investigate the potential of parameter compensation methods after the pruning.
method provides much finer-grained
ACKNOWLEDGMENTS
This work was supported in part by The National Nature Science Foundation of China (Grant NO.: 62303406), in part by Ningbo Key R&D Program (NOs.: 20252055, 2025Z035), in part by Yongjiang Talent Introduction Programme (Grant NO.: 2023A- 4023A-194-G), in part by Zhiyuan Lab- oratory (NO.: Z.L.2Y222000 (YI.2004022b).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorsk, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms. arXiv preprint arXi:: 13319, 2019.
Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, vol- ume 4. Springer, 2006.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj. Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the dataset. arXiv preprint arXiv: 2018.
I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen- Ven-Chang Hsu, and Chun-Y Yi Lee. Retraining- free merging of sparse moe via hierarchical clustering. Forty-second International Conference on Machine Learning, 2025. URL httpt:t////// http:///ppeeevviieeee net forum? di hs OxzxxxX
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401. 1.66666, 2024.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475-4488, 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL httpp:://ppp..eeeeeeeeee
net. htt.://ommmmm.oocrr.f...
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL URL/httts://eeeoodeee org/ records/ 12608602
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based iims compression. arXiv preprint arXhv:2502.17298, 2025.
Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference neural networks, pp. 293 299-299. IEEE, 1993.
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI. Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreviess net. forttfoorrr httt://orrumi..cheeeereooo
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
11

---

Published as a conference paper at ICLR 2026
Lujun Li, Zhu Qiyuan, Jiacheng Wang. Wei Li, Hao Gu, Sirui Han, and Yike Guo. Sub-mo Efficient mixture-of-eepert Ilms compression via subspace expert merging. arXiv preprint arXiv:2506.23266. 2025a.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tian- long Chen. Merge, then compress: Demystify efficient SMoe with hints from its routing policy. The Twelfth International Conference on Learning Representations, 2024. URL
https: / http://pppeevvveeeeee net. -ett/
Wei Li, Lujun Li. Hao Gu, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Moe-SVD: Structured mixture-of-experts LLMs compression via singular value decompo- sition. In Forty-second International Conference on Machine Learning. 2025b. URL https:
//openzeview.oo
Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Struc- tured mixture-of-experts LLMs compression via singular value decomposition, 2025c. URL
https://epenrevier: net wow.:/ooddmooouuu-
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. 2024a.
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-experts language models: Enhancing performance and reducing inference costs. arXiv preprint arXiv:2407.00945 2024b.
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts are equal: Efficient expert pruning and skipping for mixture-of of-experts large language models. Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pp. 6119-61722 2024.
James Martens and Roger Grosse. Optimizing neural networks with cronecker-factorcc approximate curvature. In International conference on machine learning, pp. 2408-2417. PMLR, 2015.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.000 027999 2018.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 2021.
Sidak Pal Singh and Dan Alistarh. Woodfisher: Efficient second-order approximation for neural network compression. Advances Neural Information Processing Systems, 33:18098-18109, 2020.
Qwen Team. Qwenl.5-moe Matching 7b model performance with 1/3 activated parameters", February 2024a. URL htttps:::////en gi thub. www.hh.......oooooeo http://bllog/wwemmmoooe
Qwen Team. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024b.
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruming mixture-od cocuure-of-exppee of-experts large language model using the hints from its router. arXiv preprint arXiv:2410. 2024.
Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, and Wanx- iang Che. Camera: Multi-matrin joint compression for moe models via micro-expen redundancy analysis. arXiv preprint arXiv:2508.022 023220 2025.
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. In Findings of the Association for Computa- fional Linguistics: EMNLP 2024, pp. 10456-10466 2024.
12

---

Published as a conference paper at ICLR 2026
Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Schoon Kim, Michael W Mahoney, and Kurt Keutzer. Hessian-aware pruning and optimal neural implant. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3880-3891, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 1XXi::9055007860 2019.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang. Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. Deepseek-coder-ve Breaking the barrier of closed-soure models in code intelligence. arXiv preprint arXiv:2406. 11ri222466.99990 1931, 2024.
APPENDIX OVERVIEW
Section
A
Derivation of the importance of atomic expert.
Section
Detail Analysis of Main Results.
Section
Analysis of Runtime Speedup and Memory Usage.
Section
Compression Rate Analysis under Global Pruning.
Section
Use of LLM.
Section
Reproducibility Statement.
Section
G
Ethics statement.
A DERIVATION OF THE IMPORTANCE OF ATOMIC EXPERT
We provide a detailed derivation of the importance measure introduced in equation 13
Let θ denote the model parameters of dimension P. Let (x) E Rd be the output of the k-th atomic expert for given input x. We define the Jacobian of the atomic expert output with respect
to the parameters (x) (x) Rd Let g(y) Ve, log P(y/x, θ) Rd be the gradient
of the log-likelihood with respect to the expert output.
Under the negative log-like liellkikehoodd loss, the Hessian matrix H can be approximated by the Fisher Information Matrix F:
H F Ex-Pan (By--Pmalvis)) Ve log P(y|x, V log P(y/ x, θ) [].
(17)
Using the chain rule, the gradient of the log-likelihood with respect to the parameters can be decom- posed as:
Ve log P(y|x, Jx()) g(y)-
(18)
Substituting this into the expression for F:
F Ex [Ja g())g(y) Jx(x)]]
(19)
Since (x(x) depends only and is constant with respect to the inner expectation over y, can extract it:
F Ex (x)
T)) (x)]
(20)
We define (x(x) A Eyix (g())g())
J]. Then, the parameter-space FIM becomes:
F Fx [ (x)))
(21)
We aim to find parameter perturbation 80 that minimizes the change in the overall loss while canceling the output of the k-th atomic expert (i.e. Ja(x)50 + (x) = 0). The optimization problem is formulated
min de
1 T F80
s.t.
Jx(x)80+en) ex())
(22)
Substituting the derived form of F:
min
1 80
E2 [x(x) Gx()) 80.
(23)
80
2
13

---

Published as a conference paper at ICLR 2026
Since d6 is global parameter update independent of the specific sample x, move it inside the expectation:
min be
Ex (Jx((x)))) GA (x(x)
(24)
Using the constraint (x(x)0 = -x we substitute this into the objective function:
S = Ex (xx))xx) (xxx)xxxx)
(25)
B DETAIL ANALYSIS OF MAIN RESULTS.
Calibration Set Sampling Strategy. To construct the calibration set, we first load the entire dataset (either WikiTexx-2 or C4) and concatenate all sentences into single corpus using "\n \ as the separator. We then tokenize the full corpus and split the resulting token stream into consecutive samples, each consisting of 2048 tokens. With fixed random seed (random. seed (0) ) for repro- ducibility, we randomly select 128 such samples to form the calibration set. The 128 samples drawn from Wexi were used obtain all results reported in Table T and the impact of the calibration set is discussed in Section
Details of Baseline Experiments. For DeepScckMMoE-GGE Basc, the results for NAEE, MoE-1², and M are taken from the paper (Gu et al. 2025 while MoE-SVD results are sourced from all results are from the paper (Li al., 2025a); any missing results with available official open-source code were reproduced by us. For Qwen3-300 Wwen3-30B-A31 30B-A3B, the results for HC-SMoE and Sub-MoE are from the paper (Li al. 2025a). For Owen2-57B-A144 the results for NAEE, MoE-I2 and D*-MoE are taken from the paper (Gu et al.
its paper (Li et al. 2025b). For -B..l.--MM0-AA27B-CC
2025). Table 4 shows the calibration dataset size for various methods.
Table 4: Calibration sizes for different methods (2048 sqlen).
Method
NAEE
D2-MoR
Sub-MoF
HEAPr
Calibration Set Size
128
512
128
128
C ANALYSIS OF RUNTIME SPEEDUP AND MEMORY USAGE
Table 5 summarizes the computational cost and performance of HEAPr compared with competitive baseline methods (NAEE and D2²MoFF) on two representative MoF models: DeepSeek MoF. 16B- Base and Qwen2-57B-A A1.BB.. The table reports the number of calibration samples used for pruning, the theoretical FLOPs (TFLOPs) required for pruning, GPU time cost, peak memory usage.
Table 5: Comparison of computational cost between HEAPr and baseline pruning methods.
Method Samples
TFLOPs
GPU Time Cost
Memory
DeepSeekMoE 16B-Base
NAEE
128
11
2 min
27GB
D--MoE
512
227
30 min
53GB
HEAPP
128
44
6 min
44GB
Qwen2-57B-A -14BB
NAEE
32
60GB
D
512
1205
90 min
127GB
HEAPP
128
123
20 min
91GB
14

---

Published as a conference paper at ICLR 2026
D COMPRESSION RATE ANALYSIS UNDER GLOBAL PRUNING
In this section, we analyze the compression rates across different layers when applying a 25% and 50% global pruning strategy based on the global ranking of atomic experts' importance. As shown in Figure 5 and 6 the compression rate is initially high in the early layers, suggesting that the experts in these layers are less important and can be pruned with minimal impact on the model's performance, As we move deeper into the network, the compression rate decreases, indicating that experts in these layers are more important to the model's performance. Interestingly, after certain point, the compression rate starts to increase again in the deepest layers, suggesting that some experts in these layers become redundant, allowing for further pruning without significant loss of model performance. This non-monotonic behavior highlights the varying importance of experts across layers in MoE-based models.
MoE A2.7B.C...
DeepseckMMo-.
-wen3-30B-4 -00--BBBBB3B
0.33
Compresseo on Ratto
0.30
0.30
0.25
0.25
0.20
0.20
100
15
20
20
1.5
20
25
10
20
30
40
Layer Index
Layer Index
Layer Index
Figure 5: Compression ratios across different layers under 25% global pruning for Qwenl.5-Mol Mo-BB--- A2.7B-Chat, CeepSeckMoE-16b-Biin and Qwen3-30B-A3B.
Dwen1.5MM0EA2.788 5...---77222.
DeepseekMoo-166
Owen3-308-A38
0.60
0.60
0.55
0.6
Compressen Rato
0.55
0.50
0.50
0.4
0.65
0.40
0.2
0.35
10
15
20
0.35
2.0
1.5
20
25
10
20
30
40
Layer Index
Layer Index
Layer Index
Figure 6: Compression ratios across different layers under 50% global pruning for Qwenl M-MoE--- A2.7B-Chat, DeepSeekMoE-166- 16b-BBas, and Qwen3-30B-A3B.
E USE OF LLMS
In this work, Large Language Models (LLMs) were primarily utilized for tasks such as text refine- ment, offering writing suggestions, and improving the overall structure and clarity of the manuscript. important to note that LLMs did not contribute to the ideation development of the method- ology section. The authors guarantee that all LLMMggnnrrted content was thoroughly reviewed and edited to ensure its accuracy and coherence.
F REPRODUCIBILITY STATEMENT
To ensure reprodueibility, we have made the code and checkpoints obtained in our computational environment available at http:://gitabb.cm//LLLKXX//// While have taken every effort to
15

---

Published as a conference paper at ICLR 2026
consistency, results may exhibit slight variations due to the random selection of calibration sets, as well as potential version differences in libraries such as transformers and LM-Evaluation- Hamess. These fluctuations are expected and considered acceptable.
G ETHICS STATEMENT
This work adheres to ethical guidelines in conducting research and reporting results. We have used publicly available datasets and models, ensuring that our methods comply with their respective terms of use. The research itself aims enhance existing technologies and does introduce any ethical concerns. No personal or sensitive data was used in this study, and the methods employed do not raise any known ethical issues.
16


Reviews:
[{"review_id": "RHM1Sz76LN", "reviewer": "Reviewer_Xbwv", "summary": "This paper introduces HEAPr, a pruning algorithm for MoE LLMs that decomposes each expert into smaller ''atomic'' sub-experts and scores them using Hessian information. By showing that cross-derivatives between atomic units are zero, the authors reduce the Hessian complexity drastically and then shift the analysis to the output space using a Fisher-Taylor approximation. In practice, HEAPr computes atomic importances with just two forward passes and one backward pass on a small calibration set, without introducing extra hyperparameters. Experiments show that HEAPr can prune 20–25% of parameters (refuce ~20% in FLOPs) with nearly no loss in accuracy.", "questions": "Kindly refer to the section above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-organized and easy to follow.\n- The method is lightweight. Importance scores are computed with only two forward and one backward pass over a small calibration set. By storing just one gradient-covariance matrix per expert, HEAPr can achieve a $O(d^2)$ storage complexity per expert.\n- HEAPr consistently outperforms existing expert-level pruning methods on multiple MoE architectures.", "weaknesses": "Thank you for submitting your work to ICLR’26. While the paper introduces a promising method for fine-grained MoE pruning, my concerns are as follows:\n\n- (Approximation assumptions) The method assumes that atomic experts within the same expert share identical output gradients, which may not always hold in practice. Could the authors provide empirical evidence, such as measuring how well the predicted loss changes align with actual loss increases, to support the validity of this?\n\n- (Evaluation scope) The experiments focus only on several zero-shot reasoning tasks using the LM-Eval-Harness. \n\n- (Mixed results) Table 2 somehow mixes results from prior papers it is unclear if baselines were run under identical conditions. Could the authors clarify which baselines were rerun and whether all methods used consistent evaluation settings?\n\n- (Sample detail) The method relies on a calibration set (128 WikiText-2 sequences), but the sample selection process isn't described. Given that `random choices can cause 'slight variations'`, could the authors clarify how calibration samples are chosen and whether results are sensitive to this choice?\n\n- (Ablation)  Have the authors evaluated how pruning full experts compares to pruning at the atomic level? An ablation would help isolate the benefit of the decomposition strategy.\n\n- (minor) The bolding in Table 1 is sometimes inconsistent (e.g. under the Qwen2-57B-A14B model at a 40% ratio).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9LxfrxG7gs", "reviewer": "Reviewer_7buB", "summary": "The paper presents a principled framework for fine-grained MoE pruning with strong theoretical motivation from OBS theory and an efficient practical implementation. The atomic expert decomposition $E_i(x) = \\sum_j e_i^{(j)}(x)$ where $e_i^{(j)}(x) = W_{\\text{down}, i, j} [ \\text{SiLU}(W_{\\text{gate}, i, j} x) \\cdot (W_{\\text{up}, i, j} x) ]$ provides flexible pruning granularity (Eq. 5-6; Sec. 3.1; p.4). A key theoretical insight is that the cross-Hessians between atomic experts are zero ($\\frac{\\partial^2 E}{\\partial \\Theta^{(i)} \\partial \\Theta^{(j)}} = 0$ for $i \\neq j$), which enables a dramatic reduction in complexity (Eq. 7; Sec. 3.2; p.4). Experimental results show that 20% pruning on DeepSeekMoE-16B-Base maintains a 60.68 average accuracy vs. 60.03 for the layer-wise method, and 25% pruning on Qwen3-30B-A3B drops only 0.03 in average accuracy (Table 2; Sec. 4.3; p.8). However, approximations from the Taylor expansion (first-order dominance assumption) and the equivalence of Fisher information to the expected Hessian require stronger empirical validation (Eq. 9, 11; Sec. 3.2; p.5).", "questions": "- **Validate approximations empirically**\n  - Compute the predicted loss increase ∆ℓpred using Eq. 12-13 for pruned models and compare it with the actual measured loss ∆ℓactual across varying pruning ratios (10-50%). Report the correlation coefficient, mean absolute error, and provide scatter plots to quantify the approximation quality.\n  - Analyze the Fisher-Hessian alignment: compute both $F = \\mathbb{E}[(\\nabla\\_\\Theta \\ell)(\\nabla\\_\\Theta \\ell)^T]$ and the empirical Hessian H (possibly via finite differences for a subset) and measure $\\frac{\\|F-H\\|\\_F}{\\|H\\|\\_F}$ Vary the calibration size $N \\in \\{32, 64, 128, 256, 512\\}$ and show convergence.\n  - Provide Taylor expansion validation: for selected atomic experts, compute the actual function change $e\\_P(\\Theta\\_P + \\delta\\Theta\\_P) - e\\_P(\\Theta\\_P)$ vs. the linear approximation $J\\_P \\delta\\Theta\\_P$ and report the approximation error distribution. Show empirically when first-order dominance holds.\n- **Expand baseline comparisons and ablations**\n  - Include comprehensive comparisons: adapt MoE-SVD/MoE-I2 to atomic granularity by applying low-rank decomposition/hybrid approaches to atomic weight groups. Compare accuracy, compression ratio, and FLOPs under matched conditions.\n  - Execute component ablations: (1) Atomic decomposition with random pruning (no OBS), (2) Expert-level OBS without atomic decomposition, (3) Atomic OBS in the parameter space (without the output space transformation), and (4) Per-expert normalized importance vs. global. Report the isolated impact of each contribution.\n  - Provide a comprehensive efficiency analysis: measure and report the memory footprint (GPU memory usage during pruning and inference), actual inference latency (ms/token), throughput (tokens/sec), and hardware utilization across models and pruning ratios in the main text.\n- **Complete mathematical derivations and clarify notation**\n  - Show the Lagrangian derivation in the main text: write $\\mathcal{L}(\\delta\\Theta_P, \\lambda) = \\frac{1}{2} \\sum (\\delta\\Theta^{(i)})^T H^{(i)} \\delta\\Theta^{(i)} + \\lambda^T (J_P \\delta\\Theta_P + e_P)$, derive the optimality conditions ∇L=0, substitute into the objective, and obtain Eq. 13. Move detailed algebra to the appendix but keep the outline in Sec. 3.2.\n  - Unify notation: explicitly define a pruning index map P:(expert_idx, atomic_idx)→global_atomic_idx and use it consistently. Clarify whether the $\\Theta^{(i)}$ indexing refers to atomic experts within one expert or globally across all experts.\n  - Revise Algorithm 1: make the $T\\_i$ vs. $T\\_k$ distinction explicit ($T\\_i$ are tokens routed to expert $i$; $T\\_k = T\\_i$ for atomic experts $k \\in \\text{expert } i$). Add complexity annotations for each line. Specify parallelization opportunities.\n- **Enhance reproducibility documentation**\n  - Detail the calibration procedure: specify the random seed for sampling, the exact sequence handling (padding/truncation strategy), batch vs. online covariance computation (streaming updates or full batch), and the token-level activation/gradient collection methodology.\n  - Analyze and report the global ranking complexity: provide the time complexity $O(K \\cdot \\log(K))$ where $K = \\text{total atomic experts}$, measure the actual sorting time vs. total pruning time, and discuss efficient implementations (e.g., approximate top-k selection for very large $K$).\n  - Address numerical stability: add regularization $\\epsilon$ to the gradient covariance $\\bar{G}\\_i \\leftarrow \\bar{G}\\_i + \\epsilon I$, specify the $\\epsilon$ value and selection rationale, and discuss routing imbalance handling when $|T\\_i| < \\text{threshold}$ (e.g., uniform fallback or expert merging). Report failure cases, if any.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Principled theoretical framework with efficient complexity reduction**\n  - The OBS-based formulation $\\Delta \\ell = \\frac{1}{2} \\delta\\theta^T H \\delta\\theta$ provides a rigorous pruning criterion that minimizes the increase in loss (Eq. 1-2; Sec. 2; p.3), grounding the approach in established theory rather than heuristics.\n  - The atomic expert decomposition reveals parameter decoupling $\\frac{\\partial^2 E}{\\partial \\Theta^{(i)} \\partial \\Theta^{(j)}} = 0$, reducing the Hessian from $\\mathcal{O}((3d_{\\text{model}} \\cdot d_{\\text{inter}})^2)$ to $\\mathcal{O}((3d_{\\text{model}})^2 \\cdot d_{\\text{inter}})$ per expert (Eq. 7-8; Sec. 3.2; p.4), which makes the computation tractable.\n  - A further shift to the output space leverages the Fisher information $F = \\mathbb{E}[(\\nabla_\\Theta \\ell)(\\nabla_\\Theta \\ell)^T] = \\mathbb{E}[H]$, reducing complexity to $\\mathcal{O}(d_{\\text{model}}^2)$ with an importance score $s = \\frac{1}{2} e_P^T \\mathbb{E}[g_P g_P^T] e_P$ (Eq. 11-13; Sec. 3.2; p.4-5), achieving a dramatic improvement in efficiency that is critical for scaling.\n- **Elegant algorithmic design with minimal computational overhead**\n  - The shared gradient property $\\frac{\\partial \\ell}{\\partial e^{(i)}} = \\frac{\\partial \\ell}{\\partial E}$ enables the computation of a single gradient covariance matrix Ḡi per expert rather than per atomic expert (Eq. 14-15; Sec. 3.3; Algorithm 1; p.5-6), eliminating redundant computation.\n  - A two-stage process: (1) a backward pass for $\\bar{G}\\_i = \\frac{1}{|T\\_i|} \\sum\\_{x \\in T\\_i} g\\_{E\\_i}(x) g\\_{E\\_i}(x)^T$, and (2) a forward pass for $\\bar{s}\\_k = \\frac{1}{|T\\_i|} \\sum\\_{x \\in T\\_i} \\frac{1}{2} e\\_k(x)^T \\bar{G}\\_i e\\_k(x)$ efficiently computes all importance scores (Eq. 15-16; Algorithm 1; p.5-6), demonstrating practical elegance.\n  - Global ranking across layers based on a loss contribution metric enables consistent pruning without layer-specific thresholds (Sec. 3.2; p.5), improving coherence compared to layer-wise methods.\n- **Comprehensive experimental validation with strong empirical results**\n  - Near-lossless compression: 20% pruning on DeepSeekMoE-16B-Base achieves 60.68 avg accuracy vs. original (Table 2); 25% pruning on Qwen1.5-MoE-A2.7B-Chat reaches 53.59; 40% pruning on Qwen2-57B-A14B shows minimal degradation (Tables 1-2; Sec. 4.2-4.3; p.7-8), demonstrating effectiveness across model scales.\n  - Outperforms CAMERA-P by 1.2 accuracy points at 20% pruning with a principled global importance metric vs. a heuristic local energy (Table 2; Sec. 4.3; p.8), validating the theoretical approach.\n  - Ablations confirm the superiority of global pruning over layer-wise (HEAPr-G 60.68 vs. HEAPr-L 60.03 at 20%), robustness to calibration data sources (C4 vs. WikiText-2), and performance scaling with calibration set size (Fig. 3; Tables 2-3; Sec. 4.3; p.8-9), supporting the design choices.", "weaknesses": "- **Approximation validity and empirical validation gaps**\n  - The Taylor expansion $e_P(\\Theta_P + \\delta\\Theta_P) \\approx e_P(\\Theta_P) + J_P \\delta\\Theta_P$ assumes first-order dominance but neglects higher-order terms $\\mathcal{O}(\\|\\delta\\Theta_P\\|^2)$(Eq. 9; Sec. 3.2; p.5). For large pruning perturbations (20-40% ratios), this approximation's accuracy is not empirically verified—no comparison of predicted vs. actual loss increases is provided.\n  - The Fisher information equivalence F=E[H] holds for well-converged networks but relies on a citation (Singh & Alistarh, 2020) without model-specific validation (Eq. 11; Sec. 3.2; p.5). Convergence analysis with varying calibration sizes is incomplete.\n  - The Taylor expansion for the atomic expert function claims that \"functions are not optimized with respect to parameters through gradient descent,\" so the first-order term dominates (Sec. 3.2; p.4), but this reasoning lacks rigorous justification or empirical loss landscape analysis.\n- **Limited scope of baseline comparisons and ablations**\n  - Comparisons focus on expert-level methods (NAEE, D2-MoE, Sub-MoE) and one concurrent atomic-level work (CAMERA-P); comparisons with recent decomposition methods (MoE-SVD, MoE-I2 at atomic granularity) or hybrid pruning+quantization approaches are missing (Sec. 4.2; Tables 1-2; p.7-8).\n  - No ablation isolates the contributions of: (1) atomic decomposition alone, (2) the OBS framework without the output space transformation, or (3) global vs. per-expert importance normalization—making it difficult to attribute gains to specific innovations.\n  - Efficiency analysis (Fig. 2; Sec. 4.3; p.8) reports FLOPs savings but omits memory footprint reduction, actual inference latency measurements, or GPU utilization metrics, which are critical for deployment assessment.\n- **Mathematical derivation completeness and notation clarity**\n  - The derivation from the constrained optimization (Eq. 10) to the importance metric (Eq. 13) is sketched, with intermediate steps \"detailed in Appendix A\" (Sec. 3.2; p.4-5). The main text would benefit from showing the Lagrangian formulation and optimality conditions.\n  - Notation switches between the atomic expert e(j)i(x)∈R^dmodel (Eq. 5) and the pruned atomic expert eP without clearly defining the relationship or indexing scheme (Eq. 9-13; Sec. 3.2; p.4-5). An explicit mapping from (i,j) to P would improve clarity.\n  - The pseudocode in Algorithm 1 uses Tk (tokens for atomic expert k) vs. Ti (tokens for expert i) inconsistently (Line 2 vs. Line 7; p.6). The relationship between expert-level and atomic-level token routing should be clarified.\n- **Reproducibility and implementation details**\n  - Calibration uses 128-256 samples, but the exact sampling strategy, sequence length handling, and token-level vs. sample-level statistics aggregation are not specified (Tables 2-3 mention a \"128-sample subset\"; Sec. 4.1 & 4.3; p.7,9). This is critical for replication.\n  - The computational complexity of the global ranking procedure is not analyzed—sorting dinter×nlayers×nexperts atomic experts could be expensive for large models (Sec. 3.2; Algorithm 1; p.5-6). Practical implementation considerations are absent.\n  - There is no discussion of numerical stability: the gradient covariance G̅i may become ill-conditioned with a small |Ti| (routing imbalance); how degenerate cases are handled is not specified (Eq. 15; Sec. 3.3; p.5-6).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ibh0S0zMuM", "reviewer": "Reviewer_Y8vd", "summary": "This paper introduces HEAPr, a new method for expert pruning in mixture-of-experts (MoE) models. HEAPr first decomposes each expert into smaller atomic experts and then selects those whose removal results in the smallest increase in loss. Experiments across multiple model families, including DeepSeek and Qwen, demonstrate that HEAPr achieves a better compression–quality trade-off than baseline approaches in most cases.", "questions": "* How sensitive is HEAPr to randomness in selecting samples from a given calibration dataset? Please quantify this by adding standard deviation bars in Fig. 3, computed over multiple calibration subsets drawn with different random seeds.\n\n* About the notations and theoretical claims:\n    * Line 176: The notation E (the number of experts) is not defined. \n    * Line 215: Should the complexity term \"O((3d_{model}d_{inner})^2)\" actually be \"O((3d_{model})^2)d_{inner})\"? My reasoning is that each H^{(i)} has shape (3d_{model})^2, and there are d_{inner} such matrices. Please correct me if my understanding is incorrect.\n    * Line 220: \"the constraint in equation 2 can be equivalently enforced in the atomic expert output domain\" is not accurate. If Equation 2 is satisfied (i.e., the parameters of the atomic expert are zero), then the output of the atomic expert is indeed zero for any input. However, the reverse does not necessarily hold — even if the atomic expert’s output is zero for all inputs, its parameters are not guaranteed to be zero. Therefore, enforcing the constraint in the output domain is not strictly equivalent to enforcing it in the parameter domain. \n    * (Contining the above point) Furthermore, the algorithm does not actually ensure that \"the output of the atomic expert with respect to any input is always zero,\" since it evaluates this condition only on a limited set of samples. To be clear, I am not suggesting that the authors modify the algorithm — such simplifications and approximations are understandable and acceptable. However, the claims should be stated more precisely, and these approximations should be explicitly acknowledged and clarified in the paper.\n    *  Earlier in the paper, the input to the expert functions is the tokens (e.g., Eq. 3 and Eq. 7). However, in Line 222, the input to the expert function (e_p) becomes the parameters, and the input tokens are omitted. This inconsistency in notation may cause confusion for readers and should be clarified for consistency. \n    * Line 223: \"... denotes the parameters of the pruned atomic expert.\" Should it be the parameter of the atomic expert before pruning?\n\n* Table 1 does not consistently highlight the best results in bold. In some cases, the proposed method is shown in bold even when a baseline achieves a better result. This inconsistency could mislead readers and should be corrected to ensure fair and transparent comparison. For example,\n    * When baselines achieve the same (best) score as HEAPr, only HEAPr’s numbers are highlighted in bold: 0.32 for \"NAEE, 20% ratio, Openb. dataset\", 0.76 for \"HC-SMoE, 25% ratio, PIQA dataset\", 0.63 for \"Sub-MoE, 50% ratio, WinoG. dataset\", 0.75 for (D^2-MoE, 40% ratio, ARC_e dataset)\n    * Baselines have better results than HEAPr, yet HEAPr’s numbers are still highlighted in bold while the baselines’ numbers are not: 0.75 for \"D^2-MoE, 40% ratio, WinoG. dataset\", 0.71 for \"MoE-I^2, 40% ratio, HellaS. dataset\".\n\n* Line 367: it is mentioned that \"Table 1 are primarily drawn from the respective original papers, ensuring consistency.\" Given the use of \"primarily,\" could the authors clarify whether all baseline numbers are taken from the original papers? If not, please specify which results are directly from the original papers and which ones were reproduced or obtained otherwise.\n\n* Why are different compression ratios used for different datasets? Please clarify the rationale behind this choice.\n\n* Line 382: \"In contrast, competing new methods such as Sub-MoE, D 2-MoE, and NAEE show significant accuracy degradation under the same conditions\" appears to be an overclaim. For example, D^2-MoE performs only 0.01 lower than HEAPr, which does not constitute a *significant* degradation..\n\n* The proposed pruning strategy could also be applied directly to the original experts (rather than the atomic experts). How does this variant perform? It would be helpful to include an ablation study comparing the two settings. If pruning the original experts yields worse results, that would effectively highlight the importance of the atomic expert decomposition described in Section 3.1.\n\n* It would be helpful to discuss EEP (https://arxiv.org/abs/2407.00945), an expert pruning and merging method that appeared online over a year ago.\n\n* Other minor questions/suggestions:\n    * Line 44: lacks a conjunction in the sentence \"MoE layers typically account for over 97% of total model parameters, they represent the dominant storage bottleneck.\"\n    * Line 138: \"algorithmic\" should be \"algorithm\"\n    * Line 320: there might be a typo w.r.t. \"experiences\"\n    * Line 391: what is \\Phi?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is generally well written.\n\n* The proposed method is highly efficient.\n\n* The idea of decomposing each expert into smaller atomic experts is interesting and may have broader applications.", "weaknesses": "* The notations in the paper require further clarification to improve readability. See the detailed comments below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VFtLQYBRhk", "reviewer": "Reviewer_UZ1E", "summary": "This paper introduces HEAPr, a novel pruning method for Mixture-of-Experts (MoE) models that decomposes experts into smaller atomic experts and leverages second-order information based on Optimal Brain Surgeon theory to assess their importance. By transforming the analysis from expert parameter space to atomic expert output space, the method reduces computational complexity and enables more precise fine-grained pruning.", "questions": "1. How does HEAPr perform on other major MoE model families beyond DeepSeekMoE and Qwen?\n\n2. How does HEAPr perform on other tasks?\n\n3. What are the quantitative comparisons of computational costs between HEAPr and baseline methods?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is well-grounded in Optimal Brain Surgeon theory and provides clear mathematical derivations for adapting second-order information to MoE architectures. The theoretical framework is comprehensive and enhances the method's credibility.\n-  The paper effectively addresses the fundamental challenge of applying Optimal Brain Surgeon theory to modern large-scale models by cleverly shifting the pruning analysis from high-dimensional parameter space to a more tractable output space, thereby overcoming the computational difficulties in MoE", "weaknesses": "- The experimental evaluation of the proposed HEAPr method is conducted solely on models from the DeepSeekMoE and Qwen families. This narrow scope raises concerns about the method's generalizability to other model families (such as Mixtral or GPT-OSS), and it remains unclear whether the method would maintain its effectiveness across diverse MoE implementations and architectures.\n- The evaluation is limited to commonsense reasoning and QA benchmarks, notably omitting crucial domains like code generation, which prevents a comprehensive assessment of the pruned model's full capabilities.\n-  The paper emphasizes the efficiency of HEAPr, yet the experiments provide no quantitative data to support this claim. The evaluation is missing a crucial comparison of the computational cost of the pruning process itself when compared to baseline methods.\n- The paper presents the \"atomic expert\" as a key contribution, but acknowledges and compares against concurrent work CAMERA[1], which introduces an identically defined structure called \"Micro-Expert.\" This suggests that the atomic expert pruning granularity may not be original to this work, but rather represents an emerging common approach in the field. Moreover, this pruning approach is conceptually similar to structured pruning within experts, as both involve removing corresponding columns/rows across multiple matrices.\n\n[1] CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)