HEAPr: Hessian-based Efficient Atomic Expert Pruning in Output Space

## Abstract

Abstract Mixture-of-Experts (MoE) architectures in large language models (LLMs) deliver exceptional performance and reduced inference costs compared to dense LLMs. However, their large parameter counts result in prohibitive memory requirements, limiting practical deployment. While existing pruning methods primarily focus on expert-level pruning, this coarse granularity often leads to substantial accuracy degradation. In this work, we introduce Heap r, a novel pruning algorithm that decomposes experts into smaller, indivisible atomic experts, enabling more precise and flexible atomic expert pruning. To measure the importance of each atomic expert, we leverage second-order information based on principles similar to Optimal Brain Surgeon (OBS) theory. To address the computational and storage challenges posed by second-order information, Heap r exploits the inherent properties of atomic experts to transform the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity from O ​ ( d 4 ) O(d^{4}) , where d d is the model’s dimensionality, to O ​ ( d 2 ) O(d^{2}) . Heap r requires only two forward passes and one backward pass on a small calibration set to compute the importance of atomic experts. Extensive experiments on MoE models, including DeepSeek MoE and Qwen MoE family, demonstrate that Heap r outperforms existing expert-level pruning methods across a wide range of compression ratios and benchmarks. Specifically, Heap r achieves nearly lossless compression at compression ratios of 20 % ∼ 25 % 20\%\sim 25\% in most models, while also reducing FLOPs nearly by 20 % 20\% . The code can be found at https://github.com/LLIKKE/HEAPr .

## 1 Introduction

Mixture-of-experts (MoE) models have recently emerged as a promising alternative to dense large language models (LLMs), replacing dense feed-forward layers with sparsely activated experts and dynamic routing. This design allows MoE models to match or surpass the performance of dense LLMs while activating only a fraction of parameters during inference (Fedus et al., 2022; Zhu et al., 2024; Liu et al., 2024), making them particularly attractive for large-scale, concurrent deployment. However, while sparse activation reduces computational cost, it exacerbates memory requirements. For example, DeepSeek-V3 (Liu et al., 2024) activates only 37B parameters per inference, yet all 671B parameters must still be stored in GPU memory, resulting in prohibitively high deployment costs. Notably, MoE layers typically account for over $97\%$ of total model parameters, they represent the dominant storage bottleneck. Therefore, compressing MoE layers becomes critical to overcoming inference inefficiency and making deployment feasible in resource-constrained environments.

Model pruning has been widely explored as an effective compression strategy to reduce storage and improve efficiency. Yet a fundamental trade-off persists: fine-grained pruning typically preserves accuracy but yields limited speedups on hardware, whereas coarse-grained pruning directly enables acceleration but often incurs obvious accuracy loss. Within MoE models, parameter sparsification (Xie et al., 2024) faces similar limitations, as hardware inefficiencies constrain its practical

Figure: Figure 1: Illustration of atomic expert-level pruning, which removing the $t$-th column from the $W_{gate}$ and $W_{up}$ matrices, and the corresponding $t$-th row from the $W_{down}$ matrix.
Refer to caption: x1.png

benefits. Consequently, recent research has shifted toward expert-level pruning, offering more direct gains in both acceleration and memory reduction. Existing expert-level approaches at this level can be broadly divided into expert dropping and expert merging. Expert dropping methods (Lu et al., 2024; Huang et al., 2025) completely remove experts deemed unimportant, but relying solely on calibration to discard entire experts risks losing valuable complementary expertise, consequently often leading to notable performance degradation. Expert merging methods (Li et al., 2024; Chen et al., 2025; Huang et al., 2025) instead aim to consolidate functionally similar experts to more effectively preserve overall model capacity. However, their clustering-based similarity measures are notoriously unstable, and naive merging strategies (e.g., averaging or frequency-based weighting) often introduce destructive parameter conflicts, resulting in suboptimal and inefficient outcomes.
To alleviate these critical conflicts, recent decomposition-based approaches (Li et al., 2025b; Gu et al., 2025) represent individual experts as a mixture of shared and specialized components. While this advanced framework helps to preserve model capacity, it still requires computationally expensive decomposition and merging operations, and unfortunately still incurs a non-negligible accuracy loss.

To identify pruning units that are more flexible than expert-level pruning, we introduce the concept of an atomic expert, in which each expert is decomposed into smaller, indivisible units. Concretely, each atomic expert is defined by jointly grouping the relevant columns of $\bm{W}^{\mathrm{up}}$, $\bm{W}^{\mathrm{gate}}$, and the corresponding row of $\bm{W}^{\mathrm{down}}$(as shown in Figure [1](https://arxiv.org/html/2509.22299v1#S1.F1)). The output of a full expert can be represented as the sum of outputs from multiple atomic experts. Pruning at this granularity directly removes atomic experts, thereby isolating pruning effects and avoiding interference with remaining components. By eliminating atomic experts that contribute little to final predictions, inference efficiency can be improved and deployment overhead reduced in a more straightforward and essential way.

The key challenge now lies in how to quantify the importance of each atomic expert to overall performance. To tackle this problem, we propose Heapr, a principled framework for efficient and high-performance atomic expert pruning. Our approach is inspired by the classical Optimal Brain Surgeon (OBS) theory (Hassibi et al., 1993; LeCun et al., 1989), which approximates the effect of weight pruning via a Taylor expansion of the loss function and leverages second-order information to identify parameters with minimal contribution.
However, applying OBS to modern deep architectures is computationally prohibitive due to the cost of Hessian estimation, and this is why layer-wise Hessian estimation has become widely adopted (Dong et al., 2017; Frantar & Alistarh, 2022; Frantar et al., 2023). Despite this, the space complexity of Hessian estimation at the expert level remains $\mathcal{O}\left((3d_{\text{model}}\cdot d_{\text{inter}})^{2}\right)$(^1^11$d_{\text{inter}}$ is the intermediate dimension after the $W_{\text{up}}$ transformation, and $d_{\text{model}}$ is the hidden size of the model.), which is still unacceptable. Therefore, we propose two optimizations to improve Hessian matrix computation.
First, by decomposing experts into atomic experts, we demonstrate that the second-order derivatives of parameters between different atomic experts are zero. This observation allows us to significantly reduce the space complexity of the Hessian matrix, lowering it to $\mathcal{O}\left((3d_{\text{model}})^{2}\cdot d_{\text{inter}}\right)$. Second, we further optimize the Hessian matrix by shifting the pruning constraints analysis from the parameter space of atomic experts to their output space. This shift enables us to leverage the Fisher information matrix, which is theoretically equivalent to the expected Hessian but significantly more efficient to compute (Bishop & Nasrabadi, 2006; Singh & Alistarh, 2020), and by combining this with a Taylor expansion of the atomic expert function, we can accurately estimate each atomic expert’s contribution to the final loss. This further reduces the Hessian complexity to $\mathcal{O}(d_{\text{model}}^{2})$ for each expert, ensuring high efficiency in both computation and storage.
Heapr is not only tractable but also highly efficient: all atomic expert importance can be computed with just two forward passes and one backward pass on a small calibration set.
We evaluated Heapr on seven zero-shot tasks, achieving nearly lossless compression with $20\%$ pruning on DeepSeekMoE-16B-Base, $25\%$ pruning on Qwen1.5-MoE-A2.7B-Chat, and $40\%$ pruning on Qwen2-57B-A14B. Additionally, on the latest Qwen3-30B-A3B model, the average accuracy only drops by $0.03$ at a $25\%$ compression ratio. Overall, our contributions are summarized as follows:

- •
We introduce a second-order approximation scheme for atomic expert pruning in MoE models, which transforms the second-order information from expert parameters into that of atomic expert parameters, and further simplifies it to the second-order information of atomic expert outputs. This approach reduces the space complexity of second-order information from $\mathcal{O}\left((3d_{\text{model}}\cdot d_{\text{inter}})^{2}\right)$ to $\mathcal{O}(d_{\text{model}}^{2})$.
- •
Building on this efficient scheme, we propose Heapr, a highly efficient and scalable pruning algorithm that accurately estimates the importance of all atomic experts with just two forward passes and one backward pass on a small calibration set.
- •
We conduct extensive experiments on DeepSeekMoE-16B-Base, Qwen1.5-MoE-A2.7B-Chat, Qwen2-57B-A14B, and Qwen3-30B-A3B across diverse benchmarks. Heapr outperforms current SOTA methods and achieves nearly lossless compression at compression ratios of $20\%–25\%$ in most models, while also reducing FLOPs by nearly $20\%$.

## 2 related works and Preliminary

#### Mixture of Experts Compression.

Model compression for MoE architectures has recently attracted growing attention due to the remarkable performance of MoE models. MoE-Pruner (Xie et al., 2024) performs weights sparsification based on activation magnitude, weight magnitude, and router importance, yet its acceleration is hardware-dependent and relies on distillation to recover accuracy. Expert-level pruning has been more extensively explored due to its hardware-friendly acceleration. NAEE (Lu et al., 2024) selects a subset of experts to minimize calibration error, but this can lead to overfitting and the loss of specialized knowledge. Similarly, MoE-I^2 (Yang et al., 2024) combines expert pruning with low-rank decomposition, yet requires additional fine-tuning for recovery. To alleviate such issues, expert merging methods aim to retain similar experts rather than discarding them. MC-SMoE (Li et al., 2024) merges experts by clustering based on routing policies, and HC-MoE (Chen et al., 2025) does so by grouping experts with similar outputs. However, limited expert similarity makes merging prone to parameter conflicts. To further exploit redundancy, $D^{2}$-MoE (Gu et al., 2025) constructs a shared expert via weighted combinations and compresses residuals through low-rank decomposition, while Sub-MoE (Li et al., 2025a) applies SVD to extract a shared subspace across experts, both of which require computationally expensive decomposition and merging operations. We decompose the expert into atomic experts and propose Heapr, a method that measures atomic expert importance by utilizing a second-order approximation to assess the importance of atomic experts. This approach enables more flexible pruning units and provides a highly efficient algorithmic, preserving model performance while eliminating the need for retraining.

#### Optimal Brain Surgeon in pruning.

The OBS framework (Hassibi et al., 1993; LeCun et al., 1989) approaches pruning as an optimization problem, aiming to minimize the increase in the loss function when a parameter is removed. Consider a model that has already been trained and converged, with parameters $\theta$ and a corresponding loss $\ell(\theta)$. We can analyze the effect of perturbing the parameters by analyzing the second-order Taylor expansion of the loss function around $\theta$. Specifically, the change in the loss $\Delta\ell$ when perturbing the parameters by $\delta\mathbf{\theta}$ is given by the following:

$$ $\Delta\ell=\ell(\theta+\delta\theta)-\ell(\theta)=\nabla\ell(\theta)^{\top}\delta\theta+\frac{1}{2}\delta\theta^{\top}\bm{H}\delta\theta+O(\|\delta\theta\|^{3}),$ (1) $$

where $\bm{H}$ is the Hessian matrix of second derivatives of the loss with respect to the model parameters. Since the model has already converged to a local minimum of the loss function, the first-order term can be removed ($\nabla\ell(\theta)=\mathbf{0}$), and the higher-order terms can be ignored for small perturbations.

For pruning, the constraint is $\theta_{q}+\delta\theta_{q}=0$ for the target, leading to the optimization problem as:

$$ $\min_{\delta\theta_{q}}\;\;\tfrac{1}{2}\delta\theta^{\top}\bm{H}\delta\theta,\quad\text{s.t. }\delta\theta_{q}+\theta_{q}=0,$ (2) $$

where $q$ denotes the index of the pruned parameter. Solving this optimization problem yields the minimal increase in loss from pruning parameter $\theta_{q}$, which is $\Delta\ell=\tfrac{1}{2}\frac{\theta_{q}^{2}}{[\bm{H}^{-1}]_{qq}}.$

Directly computing the full Hessian in deep neural networks is practically infeasible. Existing OBS methods to adopt significant approximations. For instance, K-FAC approximation (Martens & Grosse, 2015) provides an efficient approximation of second-order information and Hessians are computed layer-wise to guide pruning (Dong et al., 2017; Frantar & Alistarh, 2022; Frantar et al., 2023). Previous work (Singh & Alistarh, 2020) shows that the Fisher information matrix serves as a reliable Hessian estimate and allows for more efficient computation. Some apply OBS to structured pruning (Yu et al., 2022), but these efforts are limited as they consider only the trace of the Hessian.

## 3 Method

### 3.1 Atomic Expert in Mixture-of-Experts.

The MoE architecture has been widely adopted in LLMs as a replacement for the dense feed-forward network layer, which effectively increases the model capacity while reducing the number of activated parameters. Formally, given an input token representation $\mathbf{x}\in\mathbb{R}^{d_{\text{model}}}$, the MoE output is defined as:

$$ $\mathbf{y}=\sum_{i=1}^{\kappa}g_{i}(\mathbf{x})\,E_{i}(\mathbf{x}),\qquad\mathbf{g}(\mathbf{x})=\big(g_{1}(\mathbf{x}),\dots,g_{\kappa}(\mathbf{x})\big)=\operatorname{Top}\text{-}\kappa\!\big(\bm{W}^{\mathrm{gate}}\mathbf{x}\big)\in\mathbb{R}^{\kappa},$ (3) $$

where $\bm{W}^{\mathrm{gate}}\in\mathbb{R}^{E\times d_{\text{model}}}$ produces router scores and
$\operatorname{Top}\text{-}\kappa(\cdot)$ denotes the router function that selects the top-$\kappa$ experts. Each expert $E_{i}(\cdot)$ is a gated feed-forward block:

$$ $E_{i}(\mathbf{x})=\bm{W}_{i}^{\mathrm{down}}\big[\,\operatorname{SiLU}\!\big(\bm{W}_{i}^{\mathrm{gate}}\mathbf{x}\big)\odot\big(\bm{W}_{i}^{\mathrm{up}}\mathbf{x}\big)\big],$ (4) $$

where $\bm{W}_{i}^{\mathrm{up}},\bm{W}_{i}^{\mathrm{gate}}\in\mathbb{R}^{d_{\text{inter}}\times d_{\text{model}}}$,
$\bm{W}_{i}^{\mathrm{down}}\in\mathbb{R}^{d_{\text{model}}\times d_{\text{inter}}}$,
$\odot$ denotes the Hadamard product, and $\operatorname{SiLU}(\cdot)$ is the SiLU activation.
Within each expert, computations can be decomposed into atomic experts.
Let $\mathbf{w}_{i,j}^{\mathrm{up}}$ and $\mathbf{w}_{i,j}^{\mathrm{gate}}$ denote the $j$-th rows of
$\bm{W}_{i}^{\mathrm{up}}$ and $\bm{W}_{i}^{\mathrm{gate}}$, respectively, and let
$\mathbf{w}_{i,j}^{\mathrm{down}}$ denote the $j$-th column of $\bm{W}_{i}^{\mathrm{down}}$.
Then the $j$-th atomic expert of the $i$-th expert is

$$ $\mathbf{e}_{i}^{(j)}(\mathbf{x})=\mathbf{w}_{i,j}^{\mathrm{down}}\,\Big[\operatorname{SiLU}\!\big(\mathbf{w}_{i,j}^{\mathrm{gate}}\mathbf{x}\big)\cdot\big(\mathbf{w}_{i,j}^{\mathrm{up}}\mathbf{x}\big)\Big]\;\in\;\mathbb{R}^{d_{\text{model}}},$ (5) $$

where $\mathbf{w}_{i,j}^{\mathrm{up}},\mathbf{w}_{i,j}^{\mathrm{gate}}\in\mathbb{R}^{1\times d_{\text{model}}}$ and
$\mathbf{w}_{i,j}^{\mathrm{down}}\in\mathbb{R}^{d_{\text{model}}\times 1}$.
Consequently, each expert is a linear combination of its atomic experts:

$$ $E_{i}(\mathbf{x})=\sum_{j=1}^{d_{\text{inter}}}\mathbf{e}_{i}^{(j)}(\mathbf{x}).$ (6) $$

In this framework, each expert $E_{i}(\cdot)$ can be viewed as a linear combination of its atomic experts. This decomposition allows pruning at the atomic expert level without compromising the other atomic expert structure, leading to both computational acceleration and deployment efficiency directly.

### 3.2 Atomic expert Importance Analysis in the Output Space

#### Importance of Atomic experts.

As discussed in Section [2](https://arxiv.org/html/2509.22299v1#S2.SS0.SSS0.Px2), the OBS theory provides an excellent framework for analyzing the impact of parameter pruning on model performance. However, its major limitation is the large Hessian matrix, even when only computed layer-wise. In the case of MoE, directly applying OBS at the expert level is still infeasible, as it requires constructing an exceedingly large Hessian with space complexity of $\mathcal{O}\left((3d_{\text{model}}\cdot d_{\text{inter}})^{2}\right)$ per expert, leading to prohibitive computation and storage costs. Fortunately, by decomposing the expert into smaller atomic experts, a property is revealed: the parameters of different atomic experts are decoupled, i.e.,

$$ $\frac{\partial^{2}E(\mathbf{x})}{\partial\Theta^{(i)}\,\partial\Theta^{(j)}}=\frac{\partial^{2}\mathbf{e}^{(i)}(\mathbf{x})}{\partial\Theta^{(i)}\,\partial\Theta^{(j)}}=0,\quad\forall i\neq j$ (7) $$

where $\Theta^{(i)}\in\mathbb{R}^{3d_{\text{model}}}$ represents the parameters of the $i$-th atomic expert. This means that the cross-Hessians between different atomic experts are zero, which provides a valuable and simplifying property that allows us to focus exclusively on the Hessian of each individual atomic expert with respect to its own specific parameters. Based on this observation, the second-order Taylor expansion of the change in the loss function with respect to each expert’s parameters can be expressed as:

$$ $\Delta\ell=\frac{1}{2}\delta\Theta^{T}\bm{H}\delta\Theta=\frac{1}{2}\sum_{i=1}^{d_{\text{inter}}}(\delta\Theta^{(i)})^{T}\bm{H}^{(i)}\delta\Theta^{(i)}$ (8) $$

here, $\Theta\in\mathbb{R}^{3d_{\text{model}}\cdot d_{\text{inter}}}$ denotes the parameters of a given expert, while $\bm{H}$ is the corresponding Hessian matrix, which has a space complexity of $\mathcal{O}\left((3d_{\text{model}}\cdot d_{\text{inter}})^{2}\right)$. The matrix $\bm{H}^{(i)}$ represents the Hessian for the $i$-th atomic expert. This decomposition leads to a significant reduction in the complexity of summing over the Hessians $\sum_{i=1}^{d_{\text{inter}}}\bm{H}^{(i)}$, which is reduced to $\mathcal{O}\left((3d_{\text{model}})^{2}\cdot d_{\text{inter}}\right)$. However, the resulting Hessian matrix computation remains unacceptable due to its high computational and storage cost. To further alleviate the bottleneck, we introduce a second optimization that shifts pruning constraints from the parameter space of atomic experts to their output space, where the constraint in equation [2](https://arxiv.org/html/2509.22299v1#S2.E2) can be equivalently enforced in the atomic expert output domain. Specifically, pruning the atomic expert $\mathbf{e}_{\mathcal{P}}$ is equivalent to imposing: $\mathbf{e}_{\mathcal{P}}(\Theta_{\mathcal{P}}+\delta\Theta_{\mathcal{P}})=\mathbf{0},$ where $\Theta_{\mathcal{P}}\in\mathbb{R}^{3d_{\text{model}}}$ denotes the parameters of the pruned atomic expert. Since the atomic expert functions are not optimized with respect to the parameters $\Theta_{\mathcal{P}}$ through gradient descent, applying a Taylor expansion of the atomic expert functions around $\Theta_{\mathcal{P}}$ results in the first-order term dominating, yielding:

$$ $\mathbf{e}_{\mathcal{P}}(\Theta_{\mathcal{P}}+\delta\Theta_{\mathcal{P}})\approx\mathbf{e}_{\mathcal{P}}(\Theta_{\mathcal{P}})+\bm{J}_{\mathcal{P}}\delta\Theta_{\mathcal{P}}=\mathbf{0},$ (9) $$

where $\bm{J}_{\mathcal{P}}\in\mathbb{R}^{d_{\text{model}}\times 3d_{\text{model}}}$ denotes the Jacobian of $\mathbf{e}_{\mathcal{P}}(\Theta_{\mathcal{P}})$. This leads to the following problem:

$$ $\min_{\Theta_{\mathcal{P}}}\frac{1}{2}\sum_{i=1}^{d_{\text{inter}}}(\delta\Theta^{(i)})^{T}\bm{H}^{(i)}\delta\Theta^{(i)}\quad\text{s.t.}\quad\bm{J}_{\mathcal{P}}\,\delta\Theta_{\mathcal{P}}+\mathbf{e}_{\mathcal{P}}=\mathbf{0}.$ (10) $$

To solve the problem in equation [10](https://arxiv.org/html/2509.22299v1#S3.E10) , we consider the LLMs trained with a negative log-likelihood loss $\ell$ (e.g., cross-entropy loss). In this setting, the Fisher Information Matrix $\bm{F}$ is equivalent to the expected Hessian (Bishop & Nasrabadi, 2006), providing a computationally efficient alternative:

$$ $\mathbb{E}\left[{\bm{H}}\right]=\bm{F}=\mathbb{E}\!\left[(\nabla_{\Theta}\ell)(\nabla_{\Theta}\ell)^{T}\right],$ (11) $$

where $\ell$ is the sample-wise loss. Previous work (Singh & Alistarh, 2020) has shown that for well-converged neural networks, a few hundred representative samples are already sufficiently reliable to estimate $\mathbb{E}\left[{\bm{H}}\right]$. Expanding the gradient of $\ell$ with respect to the parameters gives $\nabla_{\Theta_{\mathcal{P}}}\ell=\bm{J}_{\mathcal{P}}^{\top}\mathbf{g}_{\mathcal{P}},$ where $\mathbf{g}_{\mathcal{P}}\in\mathbb{R}^{d{\text{model}}}$ is the gradient of the loss with respect to the pruned atomic expert output $\mathbf{e}_{\mathcal{P}}$. Substituting this expression into the objective equation [10](https://arxiv.org/html/2509.22299v1#S3.E10) yields the expected loss increase when pruning the atomic expert $\mathbf{e}_{\mathcal{P}}$, with $\delta\Theta^{(i)}=\mathbf{0}$ for all atomic experts not pruned:

$$ $\Delta\ell=\tfrac{1}{2}\,\delta\Theta_{\mathcal{P}}^{T}\mathbb{E}\left[{\bm{H}}_{\mathcal{P}}\right]\,\delta\Theta_{\mathcal{P}}=\tfrac{1}{2}\,\mathbf{e}_{\mathcal{P}}^{\top}\,\mathbb{E}[\mathbf{g}_{\mathcal{P}}\mathbf{g}_{\mathcal{P}}^{\top}]\,\mathbf{e}_{\mathcal{P}}.$ (12) $$

This leads us to define the Importance of the atomic expert $\mathbf{e}_{\mathcal{P}}$ as

$$ $s=\tfrac{1}{2}\,\mathbf{e}_{\mathcal{P}}^{\top}\,\mathbb{E}[\mathbf{g}_{\mathcal{P}}\mathbf{g}_{\mathcal{P}}^{\top}]\,\mathbf{e}_{\mathcal{P}},$ (13) $$

where a smaller $s$ indicates that the corresponding atomic expert has less impact on the overall model loss and should be pruned with higher priority. The detailed derivation is provided in Appendix [A](https://arxiv.org/html/2509.22299v1#A1).

At this point, we have shifted the analysis from the parameter space of atomic experts to their output space, further reducing both computational and storage requirements. Next, we introduce a remarkable property of atomic expert outputs: the outputs of atomic experts within the same expert share identical gradients, i.e.,

$$ $\frac{\partial\ell}{\partial\mathbf{e}^{(i)}(\mathbf{x})}=\frac{\partial\ell}{\partial E(\mathbf{x})},\quad\forall i\in\{1,\dots,d_{\text{inter}}\},\ \mathbf{e}^{(i)}\in E.$ (14) $$

This property allows us to further significantly reduce storage requirements. Instead of maintaining separate gradient covariance matrices for each atomic expert, we only need to store a single matrix per expert. As a result, the space complexity for computing the importance of the atomic expert within the same expert is drastically reduced to $\mathcal{O}(d_{\text{model}}^{2})$, enabling efficient storage management.

#### Global Ranking of Atomic Experts.

The metric of each atomic expert’s importance has been introduced by equation [13](https://arxiv.org/html/2509.22299v1#S3.E13). Next, an important question is how to rank the importance of the atomic experts. Consider that our importance metric evaluates experts based on their overall contribution to the model’s change in the loss function (as shown in equation [2](https://arxiv.org/html/2509.22299v1#S2.E2)), it provides a natural basis for global ranking. This allows us to effectively compare experts across layers consistently, ensuring that pruning decisions are made based on the entire model’s behavior rather than isolated layer-wise.

### 3.3 Heap r Algorithm

Building on the above analysis, we propose Heapr, a pruning strategy for MoE feedforward layers that ranks the importance of atomic experts (as defined in equation [13](https://arxiv.org/html/2509.22299v1#S3.E13)) and removes those with negligible contribution to the overall loss. To effectively compute the importance of atomic experts, we leverage a small but representative calibration set $\mathcal{D}$ and estimate importance in two stages.

1. Shared Gradient Covariance Estimation. For a given expert $E_{i}$, the gradients of the loss with respect to all its constituent atomic experts’ output are identical. Therefore, rather than performing redundant computations, we execute a single backward pass to obtain the gradient for the expert’s output, $\mathbf{g}_{E_{i}}=\partial\ell/\partial E_{i}$. This shared gradient is used to compute a gradient covariance matrix $\bar{\bm{G}}_{i}$, for all atomic experts belonging to $E_{i}$, accumulated over the subset of tokens $\mathcal{T}_{i}\subseteq\mathcal{D}$ routed to $E_{i}$:

$$ $\bar{\bm{G}}_{i}=\tfrac{1}{|\mathcal{T}_{i}|}\sum_{\mathbf{x}\in\mathcal{T}_{i}}\mathbf{g}_{E_{i}}(\mathbf{x})\mathbf{g}_{E_{i}}(\mathbf{x})^{\top}.$ (15) $$

2. Importance Computation. Subsequently, during a forward pass, we compute the importance for each individual atomic expert $\mathbf{e}_{k}$. Although the gradient covariance matrix $\bar{\bm{G}}_{i}$ is shared among all atomic experts within same expert $E_{i}$, the output of each atomic expert, $\mathbf{e}_{k}(\mathbf{x})$, remains unique. This difference in output allows us to distinguish their individual contributions. The importance of an atomic expert $\mathbf{e}_{k}$ (where $\mathbf{e}_{k}\in E_{i}$) is calculated by averaging over the tokens it processes:

$$ $\bar{s}_{k}=\tfrac{1}{|\mathcal{T}_{i}|}\sum_{\mathbf{x}\in\mathcal{T}_{i}}\tfrac{1}{2}\,\mathbf{e}_{k}(\mathbf{x})^{\top}\bar{\bm{G}}_{i}\mathbf{e}_{k}(\mathbf{x}).$ (16) $$

This approach relies solely on standard forward and backward computations, making it both exceptionally time- and memory-efficient. The space complexity of each gradient covariance matrix is only $\mathcal{O}(d_{\text{model}}^{2})$, significantly alleviating the storage bottleneck. After computing the importance ${\bar{s}_{k}}$ across all micro-experts in the model, we perform a global ranking and prune the lowest $r\%$ of experts across all MoE layers. The complete and optimized procedure is summarized in Algorithm [1](https://arxiv.org/html/2509.22299v1#alg1).

Figure: Algorithm 1 Heapr: Hessian-based Efficient Atomic Expert Pruning

## 4 experiment

**Table 1: Performance of Heapr with DeepSeekMoE-16B-Base, Qwen1.5-MoE-A2.7B-chat, Qwen2-57B-A14B and Qwen3-30B-A3B on seven zero-shot tasks, reported in terms of accuracy.**
| Ratio | Method | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA | MathQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeekMoE-16B-Base |  |  |  |  |  |  |  |  |  |
| 0% | Original | 0.32 | 0.76 | 0.71 | 0.58 | 0.45 | 0.79 | 0.32 | 0.56 |
| 20% | NAEE | 0.32 | 0.71 | 0.66 | 0.55 | 0.40 | 0.77 | 0.29 | 0.53 |
| MoE-I^2 | 0.26 | 0.71 | 0.68 | 0.49 | 0.38 | 0.73 | 0.29 | 0.50 |  |
| $D^{2}$-MoE | 0.30 | 0.74 | 0.69 | 0.55 | 0.41 | 0.76 | 0.31 | 0.54 |  |
| Heapr | 0.32 | 0.76 | 0.71 | 0.57 | 0.45 | 0.79 | 0.32 | 0.56 |  |
| 40% | NAEE | 0.23 | 0.67 | 0.67 | 0.41 | 0.32 | 0.69 | 0.26 | 0.46 |
| MoE-I^2 | 0.23 | 0.64 | 0.66 | 0.41 | 0.31 | 0.68 | 0.26 | 0.45 |  |
| $D^{2}$-MoE | 0.26 | 0.69 | 0.65 | 0.45 | 0.36 | 0.72 | 0.28 | 0.49 |  |
| Heapr | 0.30 | 0.74 | 0.69 | 0.52 | 0.41 | 0.76 | 0.30 | 0.53 |  |
| Qwen1.5-MoE-A2.7B-Chat |  |  |  |  |  |  |  |  |  |
| 0% | Original | 0.31 | 0.70 | 0.66 | 0.59 | 0.40 | 0.79 | 0.35 | 0.54 |
| 25% | MC-SMoE | 0.25 | 0.65 | 0.65 | 0.53 | 0.37 | - | - | - |
| HC-SMoE | 0.27 | 0.66 | 0.63 | 0.55 | 0.35 | 0.76 | 0.29 | 0.50 |  |
| Sub-MoE | 0.30 | 0.69 | 0.66 | 0.56 | 0.37 | - | - | - |  |
| Heapr | 0.32 | 0.69 | 0.67 | 0.56 | 0.38 | 0.76 | 0.35 | 0.53 |  |
| 50% | MC-SMoE | 0.18 | 0.33 | 0.52 | 0.29 | 0.19 | - | - | - |
| HC-SMoE | 0.23 | 0.61 | 0.65 | 0.47 | 0.35 | 0.58 | 0.23 | 0.45 |  |
| Sub-MoE | 0.25 | 0.58 | 0.58 | 0.46 | 0.25 | - | - | - |  |
| Heapr | 0.27 | 0.64 | 0.64 | 0.46 | 0.33 | 0.71 | 0.33 | 0.48 |  |
| Qwen3-30B-A3B |  |  |  |  |  |  |  |  |  |
| 0% | Original | 0.34 | 0.79 | 0.71 | 0.60 | 0.54 | 0.79 | 0.59 | 0.62 |
| 25% | HC-SMoE | 0.22 | 0.64 | 0.61 | 0.40 | 0.35 | 0.59 | 0.41 | 0.46 |
| Sub-MoE | 0.25 | 0.70 | 0.66 | 0.47 | 0.44 | - | - | - |  |
| Heapr | 0.33 | 0.77 | 0.70 | 0.55 | 0.49 | 0.78 | 0.50 | 0.59 |  |
| 50% | HC-SMoE | 0.13 | 0.44 | 0.50 | 0.29 | 0.23 | 0.44 | 0.32 | 0.34 |
| Sub-MoE | 0.23 | 0.68 | 0.63 | 0.41 | 0.40 | - | - | - |  |
| Heapr | 0.25 | 0.67 | 0.63 | 0.38 | 0.41 | 0.67 | 0.36 | 0.48 |  |
| Qwen2-57B-A14B |  |  |  |  |  |  |  |  |  |
| 0% | Original | 0.33 | 0.75 | 0.74 | 0.63 | 0.46 | 0.81 | 0.39 | 0.59 |
| 40% | NAEE | 0.31 | 0.73 | 0.73 | 0.55 | 0.46 | 0.76 | 0.36 | 0.55 |
| MoE-I^2 | 0.26 | 0.70 | 0.46 | 0.71 | 0.41 | 0.75 | 0.30 | 0.51 |  |
| $D^{2}$-MoE | 0.33 | 0.75 | 0.75 | 0.61 | 0.45 | 0.79 | 0.36 | 0.58 |  |
| Heapr | 0.33 | 0.75 | 0.74 | 0.64 | 0.46 | 0.81 | 0.39 | 0.59 |  |

### 4.1 Experimental Setup

#### Models and Setup.

We evaluate our approach on a broad spectrum of model architectures and scales to assess its generality and effectiveness, including DeepseekMoE-16B-Base (Dai et al., 2024), Qwen1.5-MoE-A2.7B-Chat (Team, 2024a), Qwen2-57B-A14B (Team, 2024b), and Qwen3-30B-A3B. All experiences are calibrated using Wikitext-2, with our method employing 128 sequences of 2048 tokens. Notably, our method introduces no additional tunable hyperparameters.

#### Baselines.

For our comparisons, we evaluate six recently proposed high-performance compression methods, including expert dropping (NAEE (Lu et al., 2024), MoE-I^2 (Yang et al., 2024)), expert merging (MC-SMoE (Li et al., 2024), HC-SMoE (Chen et al., 2025)), and expert decomposition (Sub-MoE (Li et al., 2025a), $D^{2}$-MoE (Gu et al., 2025)). For fairness and transparency, the reported results in Table [1](https://arxiv.org/html/2509.22299v1#S4.T1) are primarily drawn from the respective original papers, ensuring consistency.

#### Evaluation.

We report results on seven zero-shot benchmarks using the LM-Evaluation-Harness (version 0.4.7) (Gao et al., 2024), including HellaSwag (Zellers et al., 2019), Mathqa (Amini et al., 2019), OpenBookQA (OBQA) (Mihaylov et al., 2018), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2021), ARC-Easy and ARC-Challenge (Boratko et al., 2018). These tasks collectively enable repeated and consistent evaluation of our method across varied domains and reasoning tasks.

### 4.2 Main Results

#### Compression Performance.

As shown in Table [1](https://arxiv.org/html/2509.22299v1#S4.T1), Heapr achieves exceptional performance across various MoE models and compression ratios. Notably, our method delivers near-lossless compression. At pruning ratios of $20\%\sim 25\%$, Heapr matches the performance of the original models on DeepSeekMoE-16B-Base and Qwen1.5-MoE-A2.7B-Chat. More impressively, on Qwen2-57B-A14B, Heapr maintains performance almost identical to the original model even at a high $40\%$ compression ratio. In contrast, competing new methods such as Sub-MoE, $D^{2}$-MoE, and NAEE show significant accuracy degradation under the same conditions. Furthermore, on the latest Qwen3-30B-A3B model, Heapr incurs only a minimal performance loss at a $25\%$ pruning ratio, with the average accuracy dropping slightly from $0.62$ to just $0.59$. These results strongly highlight the unique advantage of Heapr in pruning at the atomic expert level, enabling substantial model efficiency improvements while maintaining and preserving core model performance effectively.

#### Compare to CAMERA-P.

In this section, we compare Heapr with a concurrent related work CAMERA-P (Xu et al., 2025), which evaluates the importance of an atomic expert using the concept of decoding-time energy. Specifically, the importance of the $j$-th atomic expert in the $i$-th expert is given by $\varepsilon_{i,j}=(||\Phi_{i,j}||_{2}+\alpha||\Phi_{i,j}||_{2})\cdot||\mathbf{w}_{i,j}^{\mathrm{down}}||_{2},$ where $\Phi_{i,j}=\operatorname{SiLU}\left(\mathbf{w}_{i,j}^{\mathrm{gate}}\mathbf{x}\right)\cdot\left(\mathbf{w}_{i,j}^{\mathrm{up}}\mathbf{x}\right)$. CAMERA-P uses a heuristic approach to measure atomic expert importance based on the output magnitudes on a calibration set. However, this method has two main drawbacks: it is local, neglecting atomic experts’ impact on overall model performance and cannot be globally applied for pruning due to varying activation magnitudes across layers. In contrast, our method Heapr, built upon the OBS framework, leverages the Hessian matrix to assess the impact of atomic experts on the overall model performance. And Heapr yields a globally consistent importance metric for atomic experts, thereby enabling principled global pruning, as analyzed in Section [3.2](https://arxiv.org/html/2509.22299v1#S3.SS2.SSS0.Px2). In Table [2](https://arxiv.org/html/2509.22299v1#S4.T2), we compare the performance of Heapr and CAMERA-P on DeepSeekMoE-16B-Base. Since CAMERA-P has not released its open-source implementation, we evaluate Heapr using the acc_norm as reported in the original paper. At a $20\%$ pruning ratio, Heapr outperforms CAMERA-P by an average of $1.2$ in accuracy. Even when applying the same layer-wise pruning strategy as CAMERA-P, Heapr still achieves an average accuracy improvement of $0.5$. Notably, at a $40\%$ pruning ratio, the performance gap between the two methods narrows. We attribute this to the reduced redundancy at higher pruning ratio, where the non-essential atomic experts identified by both methods become nearly identical.

#### Performance Boundary of Pruning.

Figure: Figure 2: Performance of DeepSeekMoE-16B-Base under varying compression ratios, with corresponding FLOPs saving on WikiText2 data.
Refer to caption: x2.png

Figure [2](https://arxiv.org/html/2509.22299v1#S4.F2) reports the performance of Heapr on DeepSeekMoE-16B-Base using a random 128-sample subset of WikiText-2 with $2048$ tokens under different compression ratios, where the ratio denotes the fraction of parameters removed relative to the full model size. For compression ratios below $0.4$, the pruned models retain over $93\%$ of baseline accuracy while already reducing FLOPs by $21\%$. In this regime, the accuracy curve remains nearly flat, revealing substantial redundancy among micro-experts and confirming that Heapr can effectively identify and remove them. As compression increases further, accuracy degrades gracefully, highlighting a clear trade-off between efficiency and performance. Even at an extreme compression ratio of $0.9$, the model preserves about $55\%$ of its baseline accuracy while achieving a $38\%$ reduction in FLOPs. These results demonstrate both the robustness of Heapr under moderate pruning and its effectiveness in enabling aggressive acceleration while retaining performance.

### 4.3 Ablations

**Table 2: Comparison of layer-wise pruning (CAMERA-P and Heapr) versus global pruning (Heapr) on DeepSeekMoE-16B-Base and Qwen1.5-MoE-A2.7B-Chat across seven zero-shot tasks, with acc_norm reported for DeepSeekMoE-16B-Base and accuracy for others.**
| Ratio | Method | Openb. | ARC_e | WinoG. | HellaS. | ARC_c | PIQA | MathQA | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DeepSeekMoE-16B-Base |  |  |  |  |  |  |  |  |  |
| 20% | CAMERA-P | 44.00 | 71.80 | 70.17 | 75.02 | 45.56 | 78.62 | 31.46 | 59.52 |
| Heapr-L | 44.01 | 72.64 | 70.01 | 75.55 | 47.27 | 79.76 | 30.92 | 60.03 |  |
| Heapr-G | 44.80 | 73.73 | 71.43 | 76.57 | 47.01 | 79.82 | 31.42 | 60.68 |  |
| 40% | CAMERA-P | 43.20 | 70.71 | 68.51 | 69.04 | 42.24 | 75.41 | 29.01 | 56.87 |
| Heapr-L | 42.80 | 70.45 | 68.35 | 68.10 | 43.69 | 76.12 | 29.41 | 56.99 |  |
| Heapr-G | 41.40 | 72.05 | 69.06 | 70.79 | 45.05 | 76.39 | 29.85 | 57.80 |  |
| Qwen1.5-MoE-A2.7B-Chat |  |  |  |  |  |  |  |  |  |
| 25% | Heapr-L | 30.60 | 66.58 | 66.77 | 55.09 | 38.05 | 76.61 | 33.97 | 52,52 |
| Heapr-G | 31.80 | 68.60 | 67.22 | 55.67 | 37.56 | 76.39 | 34.87 | 53.59 |  |
| 50% | Heapr-L | 27.00 | 63.26 | 64.01 | 47.00 | 33.70 | 69.80 | 32.29 | 48.15 |
| Heapr-G | 27.01 | 63.89 | 64.32 | 46.35 | 34.22 | 70.86 | 33.37 | 48.57 |  |

#### Global vs. Layer-wise Pruning.

As shown in Table [2](https://arxiv.org/html/2509.22299v1#S4.T2), layer-wise pruning (Camera-P, Heapr-L) ranks the importance of atomic experts within each MoE layer and prunes the bottom $r\%$, whereas global pruning (Heapr-G) ranks the importance of all atomic experts across the entire model. Compared with Camera-P, our layer-wise pruning Heapr-L achieves superior performance, indicating that the atomic expert importance metric, as derived from equation [13](https://arxiv.org/html/2509.22299v1#S3.E13), provides a more effective pruning criterion within individual layers. Furthermore, Heapr-G, by leveraging global pruning and importance scores across all layers, achieves even stronger and more consistent results, validating the global consistency of the atomic expert importance thoroughly analyzed in Section [3.2](https://arxiv.org/html/2509.22299v1#S3.SS2.SSS0.Px2).

#### Impact of Calibration Data.

Figure: Figure 3: Performance of DeepSeekMoE-16B-Base under a 20% compression ratio, using calibration data randomly sampled from WikiText-2 and C4.
Refer to caption: x3.png

Table [3](https://arxiv.org/html/2509.22299v1#S4.F3) demonstrates that the performance of our Heapr algorithm is largely unaffected by the choice of calibration data, whether they are WikiText-2 or C4 dataset. This highlights the remarkable robustness and generalizability of our method, as it consistently performs well across different calibration corpora and domains. Furthermore, the table also explores the significant impact of calibration set size on pruning performance. As the number of calibration samples increases, the model’s performance improves consistently, indicating that larger calibration sets offer richer statistical coverage, which provides more reliable and informative signals for effective compression. These results strongly suggest that our method is not only robust to variations in calibration data but also benefits from the inclusion of additional diverse samples, further enhancing its overall effectiveness and stability.

## 5 conclusion

In this work, we introduce Heapr, a novel method that refines expert-level pruning in MoE models by enabling a more flexible and fine-grained pruning strategy at the atomic expert level. Inspired by the principles of the Optimal Brain Surgeon (OBS) theory, we evaluate the importance of atomic experts using second-order information. By transforming the analysis from the expert parameter space to that of atomic expert parameters, and further shifting it to the atomic expert output space, we significantly reduce the computational and storage bottlenecks associated with the second-order information matrix. Heapr requires only two forward passes and one backward pass to efficiently compute the importance of atomic experts. Extensive experiments on various modern MoE models demonstrate that Heapr outperforms state-of-the-art pruning methods, achieving near-lossless pruning with pruning rates of $20\%\sim 25\%$. More importantly, our method provides a much finer-grained perspective on MoE expert pruning, which we hope will contribute to a deeper, more comprehensive understanding of MoE models. Future work will explore large-scale experiments across a wider range of model and investigate the potential of parameter compensation methods after the pruning.

## References

- Amini et al. (2019)
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi.
Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
*arXiv preprint arXiv:1905.13319*, 2019.
- Bishop & Nasrabadi (2006)
Christopher M Bishop and Nasser M Nasrabadi.
*Pattern recognition and machine learning*, volume 4.
Springer, 2006.
- Bisk et al. (2020)
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al.
Piqa: Reasoning about physical commonsense in natural language.
In *Proceedings of the AAAI conference on artificial intelligence*, volume 34, pp. 7432–7439, 2020.
- Boratko et al. (2018)
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang, Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al.
A systematic classification of knowledge, reasoning, and context within the arc dataset.
*arXiv preprint arXiv:1806.00358*, 2018.
- Chen et al. (2025)
I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun-Yi Lee.
Retraining-free merging of sparse moe via hierarchical clustering.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=hslOzRxzXL](https://openreview.net/forum?id=hslOzRxzXL).
- Dai et al. (2024)
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al.
Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models.
*arXiv preprint arXiv:2401.06066*, 2024.
- Dong et al. (2017)
Xin Dong, Shangyu Chen, and Sinno Pan.
Learning to prune deep neural networks via layer-wise optimal brain surgeon.
*Advances in neural information processing systems*, 30, 2017.
- Fedus et al. (2022)
William Fedus, Barret Zoph, and Noam Shazeer.
Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
*Journal of Machine Learning Research*, 23(120):1–39, 2022.
- Frantar & Alistarh (2022)
Elias Frantar and Dan Alistarh.
Optimal brain compression: A framework for accurate post-training quantization and pruning.
*Advances in Neural Information Processing Systems*, 35:4475–4488, 2022.
- Frantar et al. (2023)
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh.
OPTQ: Accurate quantization for generative pre-trained transformers.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=tcbBPnfwxS](https://openreview.net/forum?id=tcbBPnfwxS).
- Gao et al. (2024)
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.
A framework for few-shot language model evaluation, 07 2024.
URL [https://zenodo.org/records/12608602](https://zenodo.org/records/12608602).
- Gu et al. (2025)
Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo.
Delta decompression for moe-based LLMs compression.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=ziezViPoN1](https://openreview.net/forum?id=ziezViPoN1).
- Hassibi et al. (1993)
Babak Hassibi, David G Stork, and Gregory J Wolff.
Optimal brain surgeon and general network pruning.
In *IEEE international conference on neural networks*, pp. 293–299. IEEE, 1993.
- Huang et al. (2025)
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI.
Mixture compressor for mixture-of-experts LLMs gains more.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=hheFYjOsWO](https://openreview.net/forum?id=hheFYjOsWO).
- LeCun et al. (1989)
Yann LeCun, John Denker, and Sara Solla.
Optimal brain damage.
*Advances in neural information processing systems*, 2, 1989.
- Li et al. (2025a)
Lujun Li, Zhu Qiyuan, Jiacheng Wang, Wei Li, Hao Gu, Sirui Han, and Yike Guo.
Sub-moe: Efficient mixture-of-expert llms compression via subspace expert merging.
*arXiv preprint arXiv:2506.23266*, 2025a.
- Li et al. (2024)
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen.
Merge, then compress: Demystify efficient SMoe with hints from its routing policy.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=eFWG9Cy3WK](https://openreview.net/forum?id=eFWG9Cy3WK).
- Li et al. (2025b)
Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo.
Structured mixture-of-experts LLMs compression via singular value decomposition, 2025b.
URL [https://openreview.net/forum?id=ho7ZUS1z8A](https://openreview.net/forum?id=ho7ZUS1z8A).
- Liu et al. (2024)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024.
- Lu et al. (2024)
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li.
Not all experts are equal: Efficient expert pruning and skipping for mixture-of-experts large language models.
In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 6159–6172, 2024.
- Martens & Grosse (2015)
James Martens and Roger Grosse.
Optimizing neural networks with kronecker-factored approximate curvature.
In *International conference on machine learning*, pp. 2408–2417. PMLR, 2015.
- Mihaylov et al. (2018)
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.
Can a suit of armor conduct electricity? a new dataset for open book question answering.
*arXiv preprint arXiv:1809.02789*, 2018.
- Sakaguchi et al. (2021)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
Winogrande: An adversarial winograd schema challenge at scale.
*Communications of the ACM*, 2021.
- Singh & Alistarh (2020)
Sidak Pal Singh and Dan Alistarh.
Woodfisher: Efficient second-order approximation for neural network compression.
*Advances in Neural Information Processing Systems*, 33:18098–18109, 2020.
- Team (2024a)
Qwen Team.
Qwen1.5-moe: Matching 7b model performance with 1/3 activated parameters”, February 2024a.
URL [https://qwenlm.github.io/blog/qwen-moe/](https://qwenlm.github.io/blog/qwen-moe/).
- Team (2024b)
Qwen Team.
Qwen2 technical report.
*arXiv preprint arXiv:2407.10671*, 2024b.
- Xie et al. (2024)
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu.
Moe-pruner: Pruning mixture-of-experts large language model using the hints from its router.
*arXiv preprint arXiv:2410.12013*, 2024.
- Xu et al. (2025)
Yuzhuang Xu, Xu Han, Yuanchi Zhang, Yixuan Wang, Yijun Liu, Shiyu Ji, Qingfu Zhu, and Wanxiang Che.
Camera: Multi-matrix joint compression for moe models via micro-expert redundancy analysis.
*arXiv preprint arXiv:2508.02322*, 2025.
- Yang et al. (2024)
Cheng Yang, Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan.
Moe-i2: Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition.
In *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 10456–10466, 2024.
- Yu et al. (2022)
Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer.
Hessian-aware pruning and optimal neural implant.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, pp. 3880–3891, 2022.
- Zellers et al. (2019)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.
Hellaswag: Can a machine really finish your sentence?
*arXiv preprint arXiv:1905.07830*, 2019.
- Zhu et al. (2024)
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al.
Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence.
*arXiv preprint arXiv:2406.11931*, 2024.

## Appendix Overview

## Appendix A Derivation of the importance of atomic expert

We provide a detailed derivation of the importance measure introduced in equation [13](https://arxiv.org/html/2509.22299v1#S3.E13).

Consider the negative log-likelihood loss $\ell$, whose per-sample gradient with respect to the parameters $\Theta$ can be written as

$$ $\nabla_{\Theta}\ell=\bm{J}_{k}^{\top}\mathbf{g}_{\mathbf{e}_{k}},$ (17) $$

where $\bm{J}_{k}\in\mathbb{R}^{d\times P}$ is the Jacobian of the atomic expert output $\mathbf{e}_{k}$ with respect to its parameters $\Theta$, and $\mathbf{g}_{\mathbf{e}_{k}}\in\mathbb{R}^{d}$ is the gradient of the loss with respect to $\mathbf{e}_{k}$.
By definition, the Fisher Information Matrix is

$$ $\bar{\bm{H}}=\bm{F}=\mathbb{E}\big[(\nabla_{\Theta}\ell)(\nabla_{\Theta}\ell)^{\top}\big].$ (18) $$

After model convergence, the Jacobian $\bm{J}_{k}$ can be treated as independent (Martens & Grosse, 2015), substituting the expression of $\nabla_{\Theta}\ell$ gives

$$ $\bm{F}=\bm{J}_{k}^{\top}\,\mathbb{E}\!\left[\mathbf{g}_{\mathbf{e}_{k}}\mathbf{g}_{\mathbf{e}_{k}}^{\top}\right]\bm{J}_{k}.$ (19) $$

Returning to the quadratic optimization problem in the OBS framework:

$$ $\min_{\delta\Theta}\ \tfrac{1}{2}\,\delta\Theta^{\top}\bm{F}\,\delta\Theta\quad\text{s.t.}\quad\bm{J}_{k}\delta\Theta+\mathbf{e}_{k}=0,$ (20) $$

we define the auxiliary variable $\mathbf{u}=\bm{J}_{k}\delta\Theta$. The constraint becomes $\mathbf{u}+\mathbf{e}_{k}=0$, i.e., $\mathbf{u}=-\mathbf{e}_{k}$, and the objective reduces to

$$ $\min_{\mathbf{u}}\ \tfrac{1}{2}\,\mathbf{u}^{\top}\mathbb{E}\!\left[\mathbf{g}_{\mathbf{e}_{k}}\mathbf{g}_{\mathbf{e}_{k}}^{\top}\right]\mathbf{u}\quad\text{s.t.}\quad\mathbf{u}+\mathbf{e}_{k}=0.$ (21) $$

Plugging in the constraint yields the optimal cost:

$$ $\Delta\ell=\tfrac{1}{2}\,\mathbf{e}_{k}^{\top}\,\mathbb{E}\!\left[\mathbf{g}_{\mathbf{e}_{k}}\mathbf{g}_{\mathbf{e}_{k}}^{\top}\right]\mathbf{e}_{k}.$ (22) $$

We therefore define the importance of the $k$-th atomic expert as

$$ $s\;=\;\tfrac{1}{2}\,\mathbf{e}_{k}^{\top}\,\mathbb{E}\left[\mathbf{g}_{\mathbf{e}_{k}}\mathbf{g}_{\mathbf{e}_{k}}^{\top}\right]\mathbf{e}_{k}$ (23) $$

which is a scalar since $\mathbf{e}_{k}\in\mathbb{R}^{d}$ and $\mathbb{E}[\mathbf{g}_{\mathbf{e}_{k}}\mathbf{g}_{\mathbf{e}_{k}}^{\top}]\in\mathbb{R}^{d\times d}$.
This formalizes equation [13](https://arxiv.org/html/2509.22299v1#S3.E13) in the main text: the smaller the value of $s$, the less impact the $k$-th atomic expert has on the overall model loss, making it a better candidate for pruning.

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: Kindly refer to the section above.
soundness: 3
strengths: - The paper is generally well-organized and easy to follow.
- The method is lightweight. Importance scores are computed with only two forward and one backward pass over a small calibration set. By storing just one gradient-covariance matrix per expert, HEAPr can achieve a $O(d^2)$ storage complexity per expert.
- HEAPr consistently outperforms existing expert-level pruning methods on multiple MoE architectures.
summary: This paper introduces HEAPr, a pruning algorithm for MoE LLMs that decomposes each expert into smaller ''atomic'' sub-experts and scores them using Hessian information. By showing that cross-derivatives between atomic units are zero, the authors reduce the Hessian complexity drastically and then shift the analysis to the output space using a Fisher-Taylor approximation. In practice, HEAPr computes atomic importances with just two forward passes and one backward pass on a small calibration set, without introducing extra hyperparameters. Experiments show that HEAPr can prune 20–25% of parameters (refuce ~20% in FLOPs) with nearly no loss in accuracy.
weaknesses: Thank you for submitting your work to ICLR’26. While the paper introduces a promising method for fine-grained MoE pruning, my concerns are as follows:

- (Approximation assumptions) The method assumes that atomic experts within the same expert share identical output gradients, which may not always hold in practice. Could the authors provide empirical evidence, such as measuring how well the predicted loss changes align with actual loss increases, to support the validity of this?

- (Evaluation scope) The experiments focus only on several zero-shot reasoning tasks using the LM-Eval-Harness. 

- (Mixed results) Table 2 somehow mixes results from prior papers it is unclear if baselines were run under identical conditions. Could the authors clarify which baselines were rerun and whether all methods used consistent evaluation settings?

- (Sample detail) The method relies on a calibration set (128 WikiText-2 sequences), but the sample selection process isn't described. Given that `random choices can cause 'slight variations'`, could the authors clarify how calibration samples are chosen and whether results are sensitive to this choice?

- (Ablation)  Have the authors evaluated how pruning full experts compares to pruning at the atomic level? An ablation would help isolate the benefit of the decomposition strategy.

- (minor) The bolding in Table 1 is sometimes inconsistent (e.g. under the Qwen2-57B-A14B model at a 40% ratio).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - **Validate approximations empirically**
  - Compute the predicted loss increase ∆ℓpred using Eq. 12-13 for pruned models and compare it with the actual measured loss ∆ℓactual across varying pruning ratios (10-50%). Report the correlation coefficient, mean absolute error, and provide scatter plots to quantify the approximation quality.
  - Analyze the Fisher-Hessian alignment: compute both $F = \mathbb{E}[(\nabla\_\Theta \ell)(\nabla\_\Theta \ell)^T]$ and the empirical Hessian H (possibly via finite differences for a subset) and measure $\frac{\|F-H\|\_F}{\|H\|\_F}$ Vary the calibration size $N \in \{32, 64, 128, 256, 512\}$ and show convergence.
  - Provide Taylor expansion validation: for selected atomic experts, compute the actual function change $e\_P(\Theta\_P + \delta\Theta\_P) - e\_P(\Theta\_P)$ vs. the linear approximation $J\_P \delta\Theta\_P$ and report the approximation error distribution. Show empirically when first-order dominance holds.
- **Expand baseline comparisons and ablations**
  - Include comprehensive comparisons: adapt MoE-SVD/MoE-I2 to atomic granularity by applying low-rank decomposition/hybrid approaches to atomic weight groups. Compare accuracy, compression ratio, and FLOPs under matched conditions.
  - Execute component ablations: (1) Atomic decomposition with random pruning (no OBS), (2) Expert-level OBS without atomic decomposition, (3) Atomic OBS in the parameter space (without the output space transformation), and (4) Per-expert normalized importance vs. global. Report the isolated impact of each contribution.
  - Provide a comprehensive efficiency analysis: measure and report the memory footprint (GPU memory usage during pruning and inference), actual inference latency (ms/token), throughput (tokens/sec), and hardware utilization across models and pruning ratios in the main text.
- **Complete mathematical derivations and clarify notation**
  - Show the Lagrangian derivation in the main text: write $\mathcal{L}(\delta\Theta_P, \lambda) = \frac{1}{2} \sum (\delta\Theta^{(i)})^T H^{(i)} \delta\Theta^{(i)} + \lambda^T (J_P \delta\Theta_P + e_P)$, derive the optimality conditions ∇L=0, substitute into the objective, and obtain Eq. 13. Move detailed algebra to the appendix but keep the outline in Sec. 3.2.
  - Unify notation: explicitly define a pruning index map P:(expert_idx, atomic_idx)→global_atomic_idx and use it consistently. Clarify whether the $\Theta^{(i)}$ indexing refers to atomic experts within one expert or globally across all experts.
  - Revise Algorithm 1: make the $T\_i$ vs. $T\_k$ distinction explicit ($T\_i$ are tokens routed to expert $i$; $T\_k = T\_i$ for atomic experts $k \in \text{expert } i$). Add complexity annotations for each line. Specify parallelization opportunities.
- **Enhance reproducibility documentation**
  - Detail the calibration procedure: specify the random seed for sampling, the exact sequence handling (padding/truncation strategy), batch vs. online covariance computation (streaming updates or full batch), and the token-level activation/gradient collection methodology.
  - Analyze and report the global ranking complexity: provide the time complexity $O(K \cdot \log(K))$ where $K = \text{total atomic experts}$, measure the actual sorting time vs. total pruning time, and discuss efficient implementations (e.g., approximate top-k selection for very large $K$).
  - Address numerical stability: add regularization $\epsilon$ to the gradient covariance $\bar{G}\_i \leftarrow \bar{G}\_i + \epsilon I$, specify the $\epsilon$ value and selection rationale, and discuss routing imbalance handling when $|T\_i| < \text{threshold}$ (e.g., uniform fallback or expert merging). Report failure cases, if any.
soundness: 3
strengths: - **Principled theoretical framework with efficient complexity reduction**
  - The OBS-based formulation $\Delta \ell = \frac{1}{2} \delta\theta^T H \delta\theta$ provides a rigorous pruning criterion that minimizes the increase in loss (Eq. 1-2; Sec. 2; p.3), grounding the approach in established theory rather than heuristics.
  - The atomic expert decomposition reveals parameter decoupling $\frac{\partial^2 E}{\partial \Theta^{(i)} \partial \Theta^{(j)}} = 0$, reducing the Hessian from $\mathcal{O}((3d_{\text{model}} \cdot d_{\text{inter}})^2)$ to $\mathcal{O}((3d_{\text{model}})^2 \cdot d_{\text{inter}})$ per expert (Eq. 7-8; Sec. 3.2; p.4), which makes the computation tractable.
  - A further shift to the output space leverages the Fisher information $F = \mathbb{E}[(\nabla_\Theta \ell)(\nabla_\Theta \ell)^T] = \mathbb{E}[H]$, reducing complexity to $\mathcal{O}(d_{\text{model}}^2)$ with an importance score $s = \frac{1}{2} e_P^T \mathbb{E}[g_P g_P^T] e_P$ (Eq. 11-13; Sec. 3.2; p.4-5), achieving a dramatic improvement in efficiency that is critical for scaling.
- **Elegant algorithmic design with minimal computational overhead**
  - The shared gradient property $\frac{\partial \ell}{\partial e^{(i)}} = \frac{\partial \ell}{\partial E}$ enables the computation of a single gradient covariance matrix Ḡi per expert rather than per atomic expert (Eq. 14-15; Sec. 3.3; Algorithm 1; p.5-6), eliminating redundant computation.
  - A two-stage process: (1) a backward pass for $\bar{G}\_i = \frac{1}{|T\_i|} \sum\_{x \in T\_i} g\_{E\_i}(x) g\_{E\_i}(x)^T$, and (2) a forward pass for $\bar{s}\_k = \frac{1}{|T\_i|} \sum\_{x \in T\_i} \frac{1}{2} e\_k(x)^T \bar{G}\_i e\_k(x)$ efficiently computes all importance scores (Eq. 15-16; Algorithm 1; p.5-6), demonstrating practical elegance.
  - Global ranking across layers based on a loss contribution metric enables consistent pruning without layer-specific thresholds (Sec. 3.2; p.5), improving coherence compared to layer-wise methods.
- **Comprehensive experimental validation with strong empirical results**
  - Near-lossless compression: 20% pruning on DeepSeekMoE-16B-Base achieves 60.68 avg accuracy vs. original (Table 2); 25% pruning on Qwen1.5-MoE-A2.7B-Chat reaches 53.59; 40% pruning on Qwen2-57B-A14B shows minimal degradation (Tables 1-2; Sec. 4.2-4.3; p.7-8), demonstrating effectiveness across model scales.
  - Outperforms CAMERA-P by 1.2 accuracy points at 20% pruning with a principled global importance metric vs. a heuristic local energy (Table 2; Sec. 4.3; p.8), validating the theoretical approach.
  - Ablations confirm the superiority of global pruning over layer-wise (HEAPr-G 60.68 vs. HEAPr-L 60.03 at 20%), robustness to calibration data sources (C4 vs. WikiText-2), and performance scaling with calibration set size (Fig. 3; Tables 2-3; Sec. 4.3; p.8-9), supporting the design choices.
summary: The paper presents a principled framework for fine-grained MoE pruning with strong theoretical motivation from OBS theory and an efficient practical implementation. The atomic expert decomposition $E_i(x) = \sum_j e_i^{(j)}(x)$ where $e_i^{(j)}(x) = W_{\text{down}, i, j} [ \text{SiLU}(W_{\text{gate}, i, j} x) \cdot (W_{\text{up}, i, j} x) ]$ provides flexible pruning granularity (Eq. 5-6; Sec. 3.1; p.4). A key theoretical insight is that the cross-Hessians between atomic experts are zero ($\frac{\partial^2 E}{\partial \Theta^{(i)} \partial \Theta^{(j)}} = 0$ for $i \neq j$), which enables a dramatic reduction in complexity (Eq. 7; Sec. 3.2; p.4). Experimental results show that 20% pruning on DeepSeekMoE-16B-Base maintains a 60.68 average accuracy vs. 60.03 for the layer-wise method, and 25% pruning on Qwen3-30B-A3B drops only 0.03 in average accuracy (Table 2; Sec. 4.3; p.8). However, approximations from the Taylor expansion (first-order dominance assumption) and the equivalence of Fisher information to the expected Hessian require stronger empirical validation (Eq. 9, 11; Sec. 3.2; p.5).
weaknesses: - **Approximation validity and empirical validation gaps**
  - The Taylor expansion $e_P(\Theta_P + \delta\Theta_P) \approx e_P(\Theta_P) + J_P \delta\Theta_P$ assumes first-order dominance but neglects higher-order terms $\mathcal{O}(\|\delta\Theta_P\|^2)$(Eq. 9; Sec. 3.2; p.5). For large pruning perturbations (20-40% ratios), this approximation's accuracy is not empirically verified—no comparison of predicted vs. actual loss increases is provided.
  - The Fisher information equivalence F=E[H] holds for well-converged networks but relies on a citation (Singh & Alistarh, 2020) without model-specific validation (Eq. 11; Sec. 3.2; p.5). Convergence analysis with varying calibration sizes is incomplete.
  - The Taylor expansion for the atomic expert function claims that "functions are not optimized with respect to parameters through gradient descent," so the first-order term dominates (Sec. 3.2; p.4), but this reasoning lacks rigorous justification or empirical loss landscape analysis.
- **Limited scope of baseline comparisons and ablations**
  - Comparisons focus on expert-level methods (NAEE, D2-MoE, Sub-MoE) and one concurrent atomic-level work (CAMERA-P); comparisons with recent decomposition methods (MoE-SVD, MoE-I2 at atomic granularity) or hybrid pruning+quantization approaches are missing (Sec. 4.2; Tables 1-2; p.7-8).
  - No ablation isolates the contributions of: (1) atomic decomposition alone, (2) the OBS framework without the output space transformation, or (3) global vs. per-expert importance normalization—making it difficult to attribute gains to specific innovations.
  - Efficiency analysis (Fig. 2; Sec. 4.3; p.8) reports FLOPs savings but omits memory footprint reduction, actual inference latency measurements, or GPU utilization metrics, which are critical for deployment assessment.
- **Mathematical derivation completeness and notation clarity**
  - The derivation from the constrained optimization (Eq. 10) to the importance metric (Eq. 13) is sketched, with intermediate steps "detailed in Appendix A" (Sec. 3.2; p.4-5). The main text would benefit from showing the Lagrangian formulation and optimality conditions.
  - Notation switches between the atomic expert e(j)i(x)∈R^dmodel (Eq. 5) and the pruned atomic expert eP without clearly defining the relationship or indexing scheme (Eq. 9-13; Sec. 3.2; p.4-5). An explicit mapping from (i,j) to P would improve clarity.
  - The pseudocode in Algorithm 1 uses Tk (tokens for atomic expert k) vs. Ti (tokens for expert i) inconsistently (Line 2 vs. Line 7; p.6). The relationship between expert-level and atomic-level token routing should be clarified.
- **Reproducibility and implementation details**
  - Calibration uses 128-256 samples, but the exact sampling strategy, sequence length handling, and token-level vs. sample-level statistics aggregation are not specified (Tables 2-3 mention a "128-sample subset"; Sec. 4.1 & 4.3; p.7,9). This is critical for replication.
  - The computational complexity of the global ranking procedure is not analyzed—sorting dinter×nlayers×nexperts atomic experts could be expensive for large models (Sec. 3.2; Algorithm 1; p.5-6). Practical implementation considerations are absent.
  - There is no discussion of numerical stability: the gradient covariance G̅i may become ill-conditioned with a small |Ti| (routing imbalance); how degenerate cases are handled is not specified (Eq. 15; Sec. 3.3; p.5-6).

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * How sensitive is HEAPr to randomness in selecting samples from a given calibration dataset? Please quantify this by adding standard deviation bars in Fig. 3, computed over multiple calibration subsets drawn with different random seeds.

* About the notations and theoretical claims:
    * Line 176: The notation E (the number of experts) is not defined. 
    * Line 215: Should the complexity term "O((3d_{model}d_{inner})^2)" actually be "O((3d_{model})^2)d_{inner})"? My reasoning is that each H^{(i)} has shape (3d_{model})^2, and there are d_{inner} such matrices. Please correct me if my understanding is incorrect.
    * Line 220: "the constraint in equation 2 can be equivalently enforced in the atomic expert output domain" is not accurate. If Equation 2 is satisfied (i.e., the parameters of the atomic expert are zero), then the output of the atomic expert is indeed zero for any input. However, the reverse does not necessarily hold — even if the atomic expert’s output is zero for all inputs, its parameters are not guaranteed to be zero. Therefore, enforcing the constraint in the output domain is not strictly equivalent to enforcing it in the parameter domain. 
    * (Contining the above point) Furthermore, the algorithm does not actually ensure that "the output of the atomic expert with respect to any input is always zero," since it evaluates this condition only on a limited set of samples. To be clear, I am not suggesting that the authors modify the algorithm — such simplifications and approximations are understandable and acceptable. However, the claims should be stated more precisely, and these approximations should be explicitly acknowledged and clarified in the paper.
    *  Earlier in the paper, the input to the expert functions is the tokens (e.g., Eq. 3 and Eq. 7). However, in Line 222, the input to the expert function (e_p) becomes the parameters, and the input tokens are omitted. This inconsistency in notation may cause confusion for readers and should be clarified for consistency. 
    * Line 223: "... denotes the parameters of the pruned atomic expert." Should it be the parameter of the atomic expert before pruning?

* Table 1 does not consistently highlight the best results in bold. In some cases, the proposed method is shown in bold even when a baseline achieves a better result. This inconsistency could mislead readers and should be corrected to ensure fair and transparent comparison. For example,
    * When baselines achieve the same (best) score as HEAPr, only HEAPr’s numbers are highlighted in bold: 0.32 for "NAEE, 20% ratio, Openb. dataset", 0.76 for "HC-SMoE, 25% ratio, PIQA dataset", 0.63 for "Sub-MoE, 50% ratio, WinoG. dataset", 0.75 for (D^2-MoE, 40% ratio, ARC_e dataset)
    * Baselines have better results than HEAPr, yet HEAPr’s numbers are still highlighted in bold while the baselines’ numbers are not: 0.75 for "D^2-MoE, 40% ratio, WinoG. dataset", 0.71 for "MoE-I^2, 40% ratio, HellaS. dataset".

* Line 367: it is mentioned that "Table 1 are primarily drawn from the respective original papers, ensuring consistency." Given the use of "primarily," could the authors clarify whether all baseline numbers are taken from the original papers? If not, please specify which results are directly from the original papers and which ones were reproduced or obtained otherwise.

* Why are different compression ratios used for different datasets? Please clarify the rationale behind this choice.

* Line 382: "In contrast, competing new methods such as Sub-MoE, D 2-MoE, and NAEE show significant accuracy degradation under the same conditions" appears to be an overclaim. For example, D^2-MoE performs only 0.01 lower than HEAPr, which does not constitute a *significant* degradation..

* The proposed pruning strategy could also be applied directly to the original experts (rather than the atomic experts). How does this variant perform? It would be helpful to include an ablation study comparing the two settings. If pruning the original experts yields worse results, that would effectively highlight the importance of the atomic expert decomposition described in Section 3.1.

* It would be helpful to discuss EEP (https://arxiv.org/abs/2407.00945), an expert pruning and merging method that appeared online over a year ago.

* Other minor questions/suggestions:
    * Line 44: lacks a conjunction in the sentence "MoE layers typically account for over 97% of total model parameters, they represent the dominant storage bottleneck."
    * Line 138: "algorithmic" should be "algorithm"
    * Line 320: there might be a typo w.r.t. "experiences"
    * Line 391: what is \Phi?
soundness: 3
strengths: * The paper is generally well written.

* The proposed method is highly efficient.

* The idea of decomposing each expert into smaller atomic experts is interesting and may have broader applications.
summary: This paper introduces HEAPr, a new method for expert pruning in mixture-of-experts (MoE) models. HEAPr first decomposes each expert into smaller atomic experts and then selects those whose removal results in the smallest increase in loss. Experiments across multiple model families, including DeepSeek and Qwen, demonstrate that HEAPr achieves a better compression–quality trade-off than baseline approaches in most cases.
weaknesses: * The notations in the paper require further clarification to improve readability. See the detailed comments below.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: 1. How does HEAPr perform on other major MoE model families beyond DeepSeekMoE and Qwen?

2. How does HEAPr perform on other tasks?

3. What are the quantitative comparisons of computational costs between HEAPr and baseline methods?
soundness: 3
strengths: - The method is well-grounded in Optimal Brain Surgeon theory and provides clear mathematical derivations for adapting second-order information to MoE architectures. The theoretical framework is comprehensive and enhances the method's credibility.
-  The paper effectively addresses the fundamental challenge of applying Optimal Brain Surgeon theory to modern large-scale models by cleverly shifting the pruning analysis from high-dimensional parameter space to a more tractable output space, thereby overcoming the computational difficulties in MoE
summary: This paper introduces HEAPr, a novel pruning method for Mixture-of-Experts (MoE) models that decomposes experts into smaller atomic experts and leverages second-order information based on Optimal Brain Surgeon theory to assess their importance. By transforming the analysis from expert parameter space to atomic expert output space, the method reduces computational complexity and enables more precise fine-grained pruning.
weaknesses: - The experimental evaluation of the proposed HEAPr method is conducted solely on models from the DeepSeekMoE and Qwen families. This narrow scope raises concerns about the method's generalizability to other model families (such as Mixtral or GPT-OSS), and it remains unclear whether the method would maintain its effectiveness across diverse MoE implementations and architectures.
- The evaluation is limited to commonsense reasoning and QA benchmarks, notably omitting crucial domains like code generation, which prevents a comprehensive assessment of the pruned model's full capabilities.
-  The paper emphasizes the efficiency of HEAPr, yet the experiments provide no quantitative data to support this claim. The evaluation is missing a crucial comparison of the computational cost of the pruning process itself when compared to baseline methods.
- The paper presents the "atomic expert" as a key contribution, but acknowledges and compares against concurrent work CAMERA[1], which introduces an identically defined structure called "Micro-Expert." This suggests that the atomic expert pruning granularity may not be original to this work, but rather represents an emerging common approach in the field. Moreover, this pruning approach is conceptually similar to structured pruning within experts, as both involve removing corresponding columns/rows across multiple matrices.

[1] CAMERA: Multi-Matrix Joint Compression for MoE Models via Micro-Expert Redundancy Analysis


Decision: Accept (Poster)