REAP the Experts: Why Pruning Prevails for One-Shot MoE compression
Authors: Mike Lasby
Ivan Lazarevich
Nish Sinnadurai
Sean Lie
Yani Ioannou
Vithursan Thangarasa
Venue: ICLR 2026 Poster (2026)
Keywords: mixture-of-experts
moe
compresson
expert pruning
expert merging
merging
pruning
LLM
evaluation
Primary area: foundation or frontier models, including LLMs
TL;DR: We argue that pruning experts is superior to merging them for one-shot compression of MoE LLMs and introduces a new method, REAP, that achieves nearly lossless performance on generative tasks by minimizing the upper bound of the reconstruction error.

Abstract: Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert *merging* on discriminative benchmarks, we find that expert *pruning* is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.

Published as a conference paper at ICLR 2026
REAP THE EXPERTS: WHY PRUNING PREVAILS FOR MOE COMPRESSION
Mike Lasby! 1,2,1 Ivan Lazarevich', , Nish Sinnadurai', Sean Lie', Yani Ioannou?, Vithursan Thangarasa'
Cerebras Systems Inc., SShuulich School of Engineering, University of Calgary
, com/ /eep///rrrbbrsssseeccccrrrrpp
https://hff col
ABSTRACT
Sparsely-activated Mfiture-of-Expere (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, mo- tivating research into expert compression. Contrary recent findings favouring expert merging on discriminative benchmarks, we find that expert pruning is a superior strategy for generative tasks. We demonstrate that existing merging techniques introduce an irreducible error due to the loss of fine-grained routing control over experts. Leveraging this insight, we propose Router- weighted Expert Activation Pruning (REAP), novel pruning criterion that considers both router gate-values and expert activation norms to minimize the reconstruction error bound. Across a diverse set SMoE models ranging from 20B to IT parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression code generation tasks with Swen3-Coder-4805 and Kimi-K2 K2, even after pruning 50% of experts.
INTRODUCTION
Interest in the Sparsely-activated Hiittur--of-Expett (SMoE) architecture for Large Language Models (LLMs) surged following the release of DeepSeek-V? V3 (DeepSeek-Al et al., 2024) and other high-quality open-weight SMoE LLMs (Jiang et al., 2024; Meta Al Team, 2025; Yang et al., 2025a; Zeng et al., 2025; Baidu, 2025; Kimi Team et al., 2025). Compared to dense models, SMoEs offer lower latency and more efficient pre- pre-training (Fedus et al., 2022). However, SMoEs require more parameters than dense models to achieve similar accuracy, resulting in significant memory overhead. Further, expert usage imbalance during inference causes poor accelerator utilization, leading to increased latency or compromises such as dropped tokens (Balmau et al., 2025). Expert usage imbalance also represents an opportunity, motivating prior work which investigates whether experts can be compressed without negatively impairing accuracy (Li et al., 2023; Lu et al., 2024). By eliminating or compressing redundant experts, memory overhead is reduced, A more uniform distribution of expert usage would also improve hardware utilization. Expert compression is particu- larly valuable for use cases which feature small batch sizes such as local deployments and academic research.
Initial expert compression efforts focused on expert pruning, the removal of experts in their entirety. However, expert pruning is a strong intervention on the model's weights. Techniques such as quantization, low-rank compression, and expert merging also offer memory savings but maintain lossy representation of the less important experts. Crucially, expert merging has recently been demonstrated to outperform expert pruning when evaluated with perplexity and on Multiple Choice (MC) question answering benchmarks (Li et al., 2023; Liu et al., 2024b). However, an evaluation comparing these methods on generative benchmarks has yet to be conducted. In this work, demonstrate that when paired with suitable saliency criterion expert pruning outperforms expert merging, particularly on generative benchmark tasks such as code generation, creative writing, and mathematical reasoning. Specifically, our main contributions
follows:
We demonstrate that existing expert merging techniques introduce irreducible error due to the loss of the router's 's independent, input-dependent modulation of the expert outputs. In high-granularity SMoEs, the loss of fine-grained routing control results in functional subspace collapse;
Empirically, we find that expert merging distorts the functional manifold topology due the introduction of novel functionality. Conversely, as a coordinate subspace operation, pruning preserves the topology;
Correspondence to mklasby @ ucalgary.ca & vithu @cerebras.net + Work completed while intemship Cerebras

---

Published as a conference paper at ICLR 2026
We introduce weught-weighted Expert Activation Pruning (REAP), novel expert pruning salieney criterion. By considering both router gate-values and expert activation norms, REAP explicitly minimizes the upper bound of the reconstruction error derived in our analysis by pruning experts which contribute minimally to the layer output;
Across diverse SMoE architectures ranging from 20B to 1T parameters and a suite of generative evaluations, we demonstrate the significant and consistent advantage of REAP over existing expert pruning and merging approaches, particularly at 50% compression. Notably, our method achieves near-lossless compression (Dace < 2%) on code generation tasks after pruning 50% of experts from Owen3-Coder-cee and Kimi-K2:
We open-source our code and select compressed model checkpoints to facilitate further research on compressed SMoEs and their applications.
2 RELATED WORK
Sparsely activated SMoE architecture. A Mixture-of-Expere (MoF) layer is comprised of multiple, specialized ferd-fowwrd subnetworks known experts and router which produces gate-values (i.e., gates) to dynamically modulate the output of the experts based on the input. The architecture was revived in the deep learning era by the introduction of the SMoE by Shazeer et al. (2017). SMoEs layers only select subset of experts to use for each input, enabling massive scaling of model parameters without commensu- rate increase computational cost (Lepikhin et al., 2021: Fedus et al., 2022). In transformer- based LLMss SMoE layers are integrated by replacing the traditional feed-forward layers. Further innovations such as auxiliary-loss citfarr--lss-fress free load balancing (DeepSeek-A) Al et al., 2024), shared experts, and fine-grained experts (Dai al., 2024) have propelled SMoE architectures to become the de facto standard for LLMs in recent months.
Expert pruning. Although SMoE layers effectively decouple total model parameters from inference costs, the memory overhead of storing large SMoEs restricts their deployment in resource-constraieed environments, motivating research in expert pruning to reduce total number of parameters. Early efforts demonstrated that progressively pruning experts based on router weights during fine-tuning until a single expert remained could preserve model quality in task-specific settings (Chen et al., 2022). Koishekenov et al. (2023) found expert pruning to be effective without further fine-tuning despite aggressively pruning up to 80% of experts. Muzio et al. (2024) found that global pruning using gate-values as saliency criterion was more effective than uniform, layer-wise frequency-basse pruning, Other sophisticated pruning criteria have been proposed: Lu et al. (2024) introduced an exhaustive search strategy which prunes experts that minimize the reconstruction loss between the original and pruned layer outputs; Liu et al. (2024a) used a gradient-free evolutionary algorithm to prune experts. Both of these works demonstrated significant improvements over naive frequency-bed -base pruning. A comprehensive evaluation of 16 diverse pruning criteria was conducted by Jaiswal et al. (2025). Expert Activation Norm (EAN) was empirically found to be the highest performing criterion and the benefits of iterative pruning were presented.
Expert merging. While the above-noted works prove that expert compression is feasible via pruning, an alternative compression technique is to merge experts. Generally, merging requires both a clustering algorithm and merging technique. Li et al. (2023) introduced Merge Sparse Mixture of Experts (M-SMoH) which first initializes expert cluster centres by identifying the dominant experts with the highest usage frequency globally across all layers. The remaining non-dominant experts are clustered based on the cosine similarity of router logits. Finally, expert weights are aligned via permutation with the weight matching algorithm (Ainsworth et al., 2023) and merged using requenccyweiighee
parameter averaging. Li et al. (2023) found that their technique outperformed Chen et al.'s 's (2022) pruning method on MC benchmarks. Chen et al. (2025) proposed Hierarchical Clustering for Sparsely activated Mixture of Experts (HC-SMoE). HC-SMoE clusters experts based on the euclidean similarity of their representative vectors the average activation of each expert measured on every token in a calibration dataset- - using hierarchical agglomerative clustering. Similar to M-SMoF, HC-SMoF uses frequency- reeuuency-eiighte weighted parameter averaging to merge clusters into a single merged expert. Without any fine-tuning, Chen et al. (2025) found that their technique outperformed expert pruning based on router logits (He et al., 2025a), frequency, and Lu et al. 's (2024) method when benchmarked on a suite of MC question answering tasks.
Other compression techniques. In addition to pruning and merging, experts may be compressed through quantization (Huang et al., 2025; Li et al., 2025; Duanmu et al., 2025), low-rank decomposition (Yang et al., 2024a; Gu et al., 2025; He et al., 2025b), weight sparsity (He et al., 2025a). or a combination of any of the above techniques (Liu et al., 2025a). These other approaches are orthogonal expert pruning
2

---

Published as a conference paper at ICLR 2026
and merging; however, note that expert merging necessitates re- re-quanizzation for block quantization formats that share common scaling coefficients across a group of weights whereas pruning does not.
Model merging. Model merging aims to combine parameters from multiple trained neural networks and has been rapidly adopted as a cost-effective way to improve model quality across diverse domains. The initial motivation for merging was based on the finding that mode connectivity exists between the loss landscapes of two or more trained neural networks, enabling interpolation of their parameters without incurring an increase in loss (Garipov et al., 2018; Ainsworth et al., 2023; Ito et al., 2024), Simple parameter averaging remains an effective technique; however, more sophisticated strategies based on task vectors have also been proposed to minimize interference in the merged model parameters (Ilharco et al., 2023; Yadav et al., 2023; Yu et al., 2024). Much of the existing literature focuses on the setting in which multiple fine-tunes of single checkpoint are merged. Non-local merging in which the models do not share common checkpoint is more closely related to expert merging. Sharma et al. (2024) found that re-scaling of model activations was necessary to achieve high-quality non-local merging.
LLM evaluation. Evaluating LLMs is challenging; prior work demonstrated that simple metrics such as perplexity can be misleading when used to evaluate compressed LLMs (Jaiswal et al., 2024). MC benchmarks typically measure the log-likelihood of answer tokens determine model's is response to question (Gao et al., 2023; Chandak et al., 2025). As such, each response choice is evaluated in single for- ward pass, without any tokens being generated by the model. Perplexity and MC accuracy can therefore be viewed as discriminative metrics. In contrast, generative benchmarks require the model to output response, more closely corresponding with real-world use-cases of L.MMs. Tasks such as code generation, mathe- matical reasoning with structured outputs, and creative writing are examples of generative benchmarks.
3
MOTIVATION
Setup. To motivate our proposed expert pruning method, we derive the expected errors of both expert merging and pruning. Consider SMoE layer with K experts fı,...//,, each function fx Rd Rd. Let Txx) denote the set of indices corresponding to the top-k router scores. The router produces sparse gating vector g(x) 20 where 94 (1:) > 0 if & (1r) and (x) 0 otherwise. We assume the active gates are normalized such that ExET (z) = 1, an operation commonly included in SMoE architectures.
The output of the layer is
h(z) = Σ (x())f((r)
(E)(2)
Two operations at fixed compression. To analyse the fundamental difference between compression operations, we focus on the elementary case of reducing two experts, (ferf)), to one by comparing the
mean squared reconstruction error, & [[]((<))>- [[((())((()))))) (((=)] l where h(x) is output of the layer after compression. Pruning removes expert j and re-normalizes the router outputs over the remaining K-1 1 experts. Merging replaces (fe.fz) with new expert f. Existing one-shot expert merging methods such as HC-SMoE and M-SMoE sum the gates of the original experts gi (x))) + +g (x). The pruned, M(x)), and merged, h(x)) layer outputs are
M(x): Σ [x(x)
1- (z( Sa(w),
(2)
h(z): (o))) (z))
Σ (g((t)f(rr. Aefis
(3)
3.1 MERGING INDUCES AN
TARGET A SINGLE EXPERT CANNOT REALIZE
Define the router's input-dependent mixing ratio r(x) = [1)(( E [0, locally on the set where g >0. O. Substituting (x) and (z(x) in terms of r(z), the original contribution of the pair (i.j) can be written as
(x)
= =o. (()))(()++
(4)
The ideall ingut deperdent largel expert
3

---

Published as a conference paper at ICLR 2026
After merging, the router must apply the summed gate, (z(2++ ()) to constant convex combination of the constituent experts which is independent of z. The core issue is that the merged model forced to upproximate the dynamic. input-dependent target expert with static one. The following quantifies this unavoidable approximation error.
with constant E [0,1] and define A; fi()) (z)). This definition of assumes that the experts are linear functions of x which is generally not the case; however, this simplified model approximates the behaviour of frequency- parameter averaging used by expert merging techniques in practice. Emerge is minimized when is chosen to be the expected mixing ratio, a² == E[r(x]]] Omitting the argument (a)
Irreducible error of merging, Let
for brevity, this minimal error is
((a+t))((((++()))))))))))))))))) (g)+ (a f1-(1-a)) )))) = EE, (g0+ (r-a*) |
(5)
rouler scale policy variability expert gAP
In particular, if the router's policy is not constant (Var[r(#) > 0) and the experts are not functionally identical اااl 00), then every constant-o merge incurs positive emror. Let Giy: E. (1)(1))) Under a simplifying assumption that the router scale, policy variability, and Gij are weakly correlated across inputs, the error tern may be decomposed to:
En (ro))))))
(x)
(6)
Consequences. This is standard least-squares problem minimized when == Elr). and the minimal value is Var(r]. Based on the assumptions noted above, we conclude that merging with summed gates fundamentally flawed whenever: (i) the router has learned an input-dependent policy for mixing two experts (Var(r) > O); and (ii) the experts are themselves distinct (J) [|اار [|]ll~0) 0). Any fixed a cannot overcome the irreducible error bound established in Equation (6).
3.2 PRUNING PRESERVES INDEPENDENT CONTROL
Pruning removes one function but importantly does not tie the remaining gates. The router still modulates each surviving expert independently. In contrast, merging removes a degree of freedom in the policy by replacing individual experts with their mergers. For direct comparison under no fine-tuning, the error of pruning expert j where j ET (z). After pruning, the router promotes previously inactive expert i with the new gate-value of g((x)) 00, producing the error
consider
-,(x)-$((1))
Eprune = (m)jET(T)) [gg (x) (1) 1 (z) (x)
(7)
substination
renomalizaaion emorr
Substitution error is the dominant term in the above expression the renormalization error coefficient sim- ply scales the magnitude of the surviving expert outputs without changing their direction. In contrast, the sub- stitution error includes the output of the promoted expert which may introduce significant error. With top-k routing of S gj and the maximum substitution error occurs when of وزg with magnitude upper bounded by
(8)
Synthesis. While neither method clearly superior for all distributions, our simplified analysis above isolates specific sources of error. Merging with summed gates couples the experts, incurring error whenever either expert is active, unless the experts are functionally identical (A,, 0). The router loses the ability to independently modulate the merged experts in an input-dependent manner. Equation (6) establishes that summed gate merging incurs an irreducible error directly proportional to the router's policy variability (Varlr(=))).
In contrast, pruning only incurs errors when the pruned expert is in the top-k set, j E T(x). Unlike Equation (5), Equation (8) does not penalize policy variability; the router still controls surviving experts independently. The substitution error from pruning (Eq. 7) is proportional to its gate-value (g,) and is insensitive to policy variability. Highly-granan granular SMoEs with many experts per layer use highly variable routing policies (high Var (x)) to combine many small contributions (small gj (x)). ). In this setting, we expect merging with summed gates be fundamentally disadvantaged.
4

---

Published as a conference paper at ICLR 2026
Remarks. (i) The constant-mixtur model f is mathematically related to the frequency weighted parameter averaging merge used in practice. (ii) Even if f was dependent æ, the router after merging cannot independently modulate the two latent functions, so the original policy invalidated. (iii) With top-k routers, the specifie irreducible error from policy variability (Varfr(=)) is generated exclusively on the support where both experts are selected. Outside that support, this component vanishes, leaving only static error term that depends on the functional expert gap. (iv) See Appendix for an extension of the above analysis to hierarchical clustering.
3.3 EMPIRICAL EVIDENCE FOR LOSS OF INDEPENDENT CONTROL.
Setup. We analyse the functional expert output manifolds across four diverse state-of-the-art SMoE architectures by recording mean expert activations from samples of 2048 tokens from the C4 dataset (Raffel et al., 2020).
Functional subspace collapse. By projecting expert activations onto their first two principal components, we visualize how pruning and merging affect the learned representations. Figures 1, A5a and A5b demonstrate striking progression of functional subspace collapse from early late layers in architectures such as Qwen3 and ERNIE-4.5. In early layers, the original experts form relatively compact manifolds with moderate spread. After pruning, the surviving experts maintain their positions on the original manifold, preserving its geometric structure with reduced density. In contrast, merging produces visible contraction toward the manifold's centre. The contrast becomes dramatic in late layers, where experts are more specialized.
The progression from early to late layers validates our theoretical prediction that the irreducible error is proportional to Varlr(:)). Early layers, which typically learn more generic features, exhibit lower policy variability and thus less dramatic collapse. Late layers, where experts have specialized for distinct computational roles, demonstrate high policy variability, resulting in the severe functional collapse observed when these specialized experts are merged into static averages.
Functional manifold distortion. While collapse is less apparent in low-granularity models, the introduction of novel functions due merging distorts the topology of the original expert manifold to a greater degree than pruning. To quantitatively measure this phenomenon, we measure the 1-Wasserstein distance (Earth Mover's distance) between the original and compressed expert output manifolds, see Appendix B.2 for details. As depicted in Figure 2, the merged outputs consistently exhibit higher transport cost from the original manifold.
Manifold geometry preservation. Across all models and layers, we observe that pruning preserves the topology of the functional manifold while merging fundamentally alters it. The preservation of manifold geometry under pruming reflects the mathematical structure of the operation: the pruned expert class is a co- ordinate subspace of the original, with the router maintaining independent control over each surviving expert.
In contrast, the subspace collapse observed in merged highly-granular SMoFs visualizes the loss of independent control. When gates gi and gj are tied by their sum (ge + gj), the router can no longer independently modulate the two underlying functions, forcing the model to approximate the dynamic mixture r(x)) (1-r(()) with static merged expert J.
With low-granularity SMoEs, such -- and Mixtral, functional subspace collapse due expert merging is less apparent, see Figures A5e to A5f. With few experts per layer and active experts per token, these architectures have less variable routing and higher gate-values, which better preserves the variance of the original manifold. However, the introduction of novel functions by merging introduces greater manifold distortion than the substitution етrоr associated with pruning. These observations reveal that the core issue is not the reduction in the number of experts per se, but rather the qualitative change in the router's 'ss control structure and the introduction of novel functionality, See Appendix for additional discussion.
4 ROUTER- WEIGHTED EXPERT ACTIVATION PRUNING (REAP)
The motivation in Section 3 demonstrates that the functional output space a SMoE layer is defined by the coordinated behaviour of the router and experts. As established in Equation (8), the magnitude of the substitution error incurred by promoting expert i in lieu of pruned expert j is upper bounded by (x)) + [+)])) Naive frequency- -hased pruning considers neither the coordination between
5

---

Published as conference paper at ICLR 2026
Original Experts
Surviving
Merged
Original Experts
Surviving
Merged
(a) Qwen3-30B Layer 0
(b) Qwen3-30B Layer 47
Figure I: (a) Functional subspace (PCA) for early SMoE layers in Qwen3-30B. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (b) Functional subspace (PCA) for late MoE layers. The contraction under merging is dramatically more pronounced, with up to 100x reduction in spread for models with many experts. See Figure A5 for results from other models.
0.5
Stuned
Mhagpal
Wasserstein distance
0.4-
Wasserstein distance
0.4
Wasserstein distance
Wasserstein distance
0.4
0.4
0.2
0.2
0.2
0.1
0.1
Runed
Fune Poygz
Pruned Magou
Strgel
0.0
0.0
0.0
0.0
20
40
Layer
Layer
Layer
Layer
(a) Qwen3-30B
E.52E-4.-2IB
(c) Mixtral-8x7B
(d) Llama-4-Scout
Figure 2: 1- I-Wasserstein distance between the compressed and original expert output manifolds measured normalized angular distance. Expert merging introduces novel functions which distort the manifold.
router and expert (z)(z)) nor the functional properties of pruned expert (+))), effectively assuming that all active experts contribute equally to the output. By ignoring these terms, frequency-basee methods fail minimize the error bound derived above.
Since the identity of the promoted expert (and thus [=)(() varies across tokens, directly minimizing the pruned expert's impact g, (f)ll,(()) is an effective heuristic to minimize the total error. This strategy targets the known components of the error bound (gy [ while simultaneously shrinking the scaling coefficient (o²) of the unknown component ([]. Intuitively, this identifies experts which contribute minimally to the layer output, yielding the minimal difference between the original and pruned layer outputs in expectation. To select which experts to prune, we propose a novel saliency criterion, REAP. Specifically, the saliency score, Sj. is defined as the average of the expert's weighted magnitude over tokens for which is active:
S3= X
Σ ME
where X, is the of tokens where expert j is active (i.e.. =x laje (()))) Crucially, calculating this average conditionally over X, rather than globally decouples the expert's 's functional impact from its frequency of activation. A global average may be dominated by usage frequency and risks pruning specialist experts which are rarely activated but contribute significantly to the layer output when selected. By pruning experts with the lowest Sy. REAP targets those that provide weak functional contribution even when specifically requested by the router, thereby minimizing the substitution error bound for every active token.
EXPERIMENTS
Setup. We implement REAP and other expert compression baselines in PyTorch (Ansel al., 2024). We collect router logits and expert activation data to calibrate the compression algorithms using a variety of general pre-training and domain-specifie Supervised Fine-Tuning (SFT) datasets. For calibration, 1,024

---

Published as a conference paper at ICLR 2026
Table I: Comparison of SMoE models included in our study.
Model
Routed Experts
Experts
Top-K
Spunsity
Parameters (Ie9)
Parama, (Ic9)
First layer
dense
Granularity
Expert
87.88%
High
Qwen3-30B-A3B
93.75%
30.5
No
High
Mixtral-XX7 Bx7BB-Intruct-////
8
2 75.00%
46.7
13
No
Low
GLM Ais
128
8
93.02%
106.9
12
Yes
Hlghh
16
1
88. 24%
107.8
17
No
Low
Oweat-Codx--8UB-AAA -0---8008--ASS---msu--------
160
8 95.00%
480.2
35
No
High
Kimi-K2-Instratt 14 (RedHarAT, 2025)
184
8
97.66%
1026.4
32
Yes
High
samples are randomly selected and packed to 2,048 sequence length for models with S 110B parameters. For models with 110B parameters, select 12,228 samples with a maximum sequence length of 16,384 tokens without truncation or packing.
We compress models by pruning or merging 25% 50% of experts in each layer, except for M-SMoF which determines the number of clusters per layer based on glohal expert usage frequency. When evaluating models with < 50B parameters on coding and MC, we calibrate and compress the models using three different seeds and report the mean. Larger models, creative writing, and mathematical reasoning evaluations are reported using a single seed, except where explicitly noted otherwise. All models are evaluated in the one-shot setting, with no additional fine-tuning after compression.
Models and data. We evaluate the expert compression algorithms on a diverse set of six SMoE architectures covering model sizes from 21B to IT with varying degrees of sparsity and expert granularity, see Table for details. For MC question answering and code generation benchmarks, we et al., 2020; Allen Institute for AI, 2024) and evol-codealpaca (Chaudhary, 2023; Luo et al., 2024; Tam, 2023) datasets assess both general and domain-specific calibration. Models with 2 1 110B parameters are additionally calibrated with data from lam--functoon-callin (Liu et al., 2024c; Salesforce, 2025) and
C4 (Raffel
trajectoresss (Yang al., 2025c;b) datasets. For creative writing and math benchmarks we employ WritingPrompts curated (Pritsker, 2024) and mula---sf-perronna-maal Allen Institute for AI, 2025), respectively. The default chat template is applied to all SFT datasets and </think> tags are explicitly closed to disable reasoning in hybrid reasoning models.
(Lambert et al., 2025;
Evaluation. Compressed SMoE models are evaluated on a suite of benchmarks including MC question answering, code generation, mathematical reasoning, creative writing, and tool calling. See Appendix for details. We implement HC-SMoE. and M-SMoE as expert merging baselines. Average linkage criterion used for HC-SMoE. M-SMoE does not include low-rank compression from the complete MC-SMoE method. Pruning baselines consist of frequency- -based pruning and EAN and experts with the lowest saliency scores according to each method's 's criterion are pruned. See Appendix for formal definitions.
5.1 RESULTS
Table 2 and Figure code generation, creative writing, math reasoning, and MC results are presented for Qwen3-30B and GL.--.5-Air after calibration with domain-specific datasets. Table 3 contains results for large-scale SMoE pruned models on code generation, tool calling, and MC benchmarks. See Table A6 and Table A7 for detailed MC and code generation results, respectively. Figure A6 depicts coding generation and MC accuracy versus model parameters. See Appendix F for additional results.
Zero-shot MC question answering. Both merging and pruning are capable of producing accurate com- pressed SMoE models for MC question answering. HC-SMoE and REAP have mean decrease in accuracy of approximately 4% and 13% for compression ratios of 25% and 50%, respectively, excluding large-scale SMoEs. REAP achieves first or second rank among all methods, models and compression ratios, suggesting strong consistency regardless of specific model architecture. When calibrated on C4, we find slightly improved accuracies for all compression methods with similar rankings as noted above, see Table A8.
Generative benchmarks. Compared to MC, generative benchmarks are more representative of world cases of LLMs. In this setting, pruning emerges as the clearly superior compression method on the gener- ative task benchmarks. Excluding large-scale SMoEs, REAP achieves a mean decrease in accuracy of 1.9% and 6.9% 25% and 50% compression ratios, respectively, on coding. In comparison, both HC-SMoB and
7

---

Published as a conference paper at ICLR 2026
Table 2: MC and generative benchmark results for Qwen3-30B and GLM-4.5-Air.
Coding LiveCede
Creative Writing Wikllench
Math MATH-50I
MC
Model
Compression
Techuiqqu
Method
Eval+
Code Avg
GSMBK
Math
Avg
Reoeline
0,302
0U811
0,003
01872
0.721
Marging
M-SMaF:
SMeE
0.781
0.752
0.201
0.258
0.537
0,975
0.497
0.901
0.564
0.873
0.83
0.819
0.886
0.674
0.558
Froquency
0,302
0,807
0.10
0.868
0.85
0,600
Quess 308-A388
Pruning
EAN
0,797
0.311
0.811
0.004
0877
0U893
0.00
REAP
0.797
0.304
0.551
0.804
0.896
0.881
0.888
0.665
Merging
M.SMal
MC-SMcl
0.543
0.590
0.185
0.206
0.397
0.364
0.00
0.725
0.824
0,760
0.838
0.696
0.724
0.831
0:451
0.542
50%
Requessy
0.668
0.236
0.52
0.677
8171
0180
0.66
0.485
Praming
EAN
0.753
0,306
0.530
0.702
0.874
0.855
0.864
0.495
REAP
0.750
0,302
0,541
0.718
0877
01838
0.857
0.505
Baseline
0.786
0.374
0,500
0.839
0.846
0
0.683
0,747
Merging
M-SMoE
SMoc
0.726
0,363
0.350
0.528
0.781
0.78M
0242
0.488
01908
0.880
0.864
0.704
0,596
2.%
Rroquency
0.759
0.341
0,795
0182
01908
0U870
0.648
4.55AA Air
Paming
FAN
0.374
0.834
0.800
0.908
0.874
0.677
0,412
0.831
0.839
0.920
0.674
Marging
MiSMal:
HC-SMcR
0.688
0.099
0.220
0.419
0,391
0.903
0.465
0.667
0
0.732
0.700
0:448
0.564
50%
Frequancy
0.511
0.104
0.508
0.604
DA1S
0.612
0.613
Pruming
EAN
0.721
0.253
0.482
0.202
0.78]
0.838
0.809
0.511
REAP
0.679
0.352
0.515
0.754
01815
(1900)
0.857
0.554
Table 3: Large-scale pruned SMoEs on agentic, non non-agentic coding, tool-use tasks, and MC benchmarks.
Non-Agentic Coding
Agentic Coding SWW--Rneh--erithh
Tool-Use (BFCLxc)
MC
Modd
Compressioo
Method
Evai+
LiveCode
Code Avg
Non-Live
Live
Multi-Turn
Overall
MC Avg
Raseline
0.841
0431
0.636
0.540
01866
0,825
0,380
0,600
0,750
Qwen3-Cudkes
25%
FAN
Frequency
0.737
0.419
0.296
0.623
0.516
0.378
0.534
0.844
0.511
0,813
0,763
0.384
0.355
0.676
0.654
0.202
0.606
Instruct-FP'
REAP
0446
0.624
0.540
08878
0.392
0.688
0.748
Frequency
0.007
0.012
0.010
0.000
0.200
0.392
0.000
0.197
0.506
SOSS
FAN
0.777
0.382
B.500
0,536
01822
0.774
0.383
0.650
0.991
REAP
0,622
0.415
0,619
0.522
0.899
0
0.371
0.674
0.692
Bascline
0,828
0.434
0.631
0.554
0.800
0,802
0.355
0.696
0,780
Instruct-
Kimi-K2-
W4A16
25%
REAP
FAN
Frequency
0.840
0.002
0.379
0:440
0.284
0.640
0.579
0,560
0.562
0,19
0.644
0.812
0.501
0.045
0.263
0.451
0.682
0.635
0.604
0.20
0.773
Frequeney
0.112
0.000
0.006
0.090
0.255
0.397
0.008
0.218
0.49
500
EAN
0.722
0:253
0.497
0.576
0.778
0.767
0.173
0,573
0.587
REAP
0.819
0.49
0.624
0.576
0.785
0.743
0.164
0.564
0.643
M-SMoE produce mean decreases in accuracy >5% at 25% compression and >20% at 50% compression. Notably, REAP maintains significantly higher accuracy at 50% compression than other pruning methods. M-SMoE achieves significantly better code generation accuracy on low-granularity SMoE architectures.
On creative writing, REAP and EAN are near-lossless 25% compression with REAP offering improved quality at 50% compression. Merging methods are less consistent across various model architectures and compression ratios. For example, M-SMoE is the best method for Qwen3-30B at 50% compression, but the worst on GI G----.i- REAP attains the best mathematical reasoning results with a remarkable mean decrease in accuracy of just 0.1% at 25% compression. In comparison, HC-SMoE and M-SMoE offer high accuracy at 25% compression but are significantly less accurate than pruning at 50% compression.
Expert pruning at scale. To assess whether pruning remains viable at scale, we prune Qwen3-Coder 480B and Kimi-K2- On MC questions, REAP outperforms other pruning methods. On non-agentic coding tasks, REAP achieves near-lossless accuracy with a 0.16% and 1 1.2% mean decrease in accuracy compared to baseline at 25% and 50%, respectively, outperformin EAN and frequency- particularly at 50% compression. On the challenging SWE-Bench task, both REAP and EAN maintain high accuracy at 25% and 50% compression, with some scores slightly exceeding the baseline. On tool use, EAN and REAP are comparable, with REAP slightly outperforming at 50% compression with a mean decrease in accuracy of 5.9% versus 6.2% for EAN. Frequency-basee pruning suffers from a sharp degradation in quality at 50% compression, highlighting the importance of pruning saliency criteria which consider expert activations. Scaling the pruming methods is relatively trivial. Unlike HC-SMoE, calibration for pruning does not require recording activations from every expert for every token, facilitating efficient calibration. Further, pruming can be easily applied to quantized models without any additional steps required to reconcile block scales re-quantize following compression. See Appendix E for further discussion.
pruning,
8

---

Published as conference paper at ICLR 2026
90
Campression
Mean Accuracy (%)
80
70
Pruning metheds
60-
REP (ous)
FAN
50
Prequency
40
HC-SMEE
Morgling methods
30
N-SMoE
20
10
Coding
Math
Creative Writing
MC
Coding
Math
Creative
MC
Writing
GLM 4.5 - Air
Qwen3
30B - A3B
Figure 3: GLM-4.5-Air and Qwen3-30B accuracy vs. task type. REAP offers significant improvements compared to other methods at 50% compression. Note the significant performance drop for merging methods on generative tasks (Coding, Math, Creative Writing) compared to their relative strength on MC.
1.0
0.7
0.6
N-gram diversity
0.8
Cross perplexity
0.5
Type
0.4
0.8
alginer
0.6-
0.3
0.6
Rane
0.2
HNSH
0.4
M-Slet
10a
0.1
REAP
M.Mett pormuted
0.2
0.0
REAP
MiSMoE SMoE
10
20
0
16
32
N-gram size
Generator model
Token Position
Layer
(a) N-Gram diversity
(b) Cross perplexity
(c) Completion logit JSD
(d) Expert distance
Figure (a) & (b) N-Gram diversity and cross-perplexity of compressed Qwen3-30B-A3B models 50% compression, respectively. (c) Jensen-Shannon Divergence (JSD) of compressed and baseline model logits completion token position for Qwen3-30B-A3B at 50% compression. Initially, all compressed models share close alignment with the baseline model. However, as the completion token position increases the merged models diverge from the baseline more rapidly than the REAP pruned model. (d) The mean relative 1.2-distance and singular-vector alignment between Qwen3-30B expert weights at 50% compression. Expert merging is more challenging than model merging due large distances between experts in weight space and low singular-vector alignment.
Quantifying merged SMoE generation quality. While merged expert SMoEs offer reasonable quality for discriminative tasks such as MC question and answering, they fail remain competitive on generative tasks. To help explain the performance gap of merged models between discriminative and generative tasks, we perform an analysis the compressed model outputs and compare with REAP pruned models. We prompt 50% compressed Qwen3-30B models with 100 questions randomly sampled from the evol-codealpaca dataset and record their outputs. In Figure 4a, we measure the N-gram diversity and find that the merged models have significantly lower diversity across all N-gram sizes measured. In contrast, the REAP pruned model remains similar to the base model, albeit slightly less diverse. In Figure 4b, we measure the perplexity of the text generated by the compressed models with the original baseline model. The text generated by the merged models has both a higher mean and higher variance than the pruned model generations, suggesting that the REAP pruned model outputs are more closely aligned to the original model. The alignment between the baseline and REAP pruned SMoEs is further supported by Figure 4c, which plots the JSD of the compressed and baseline logits vs. output token position. The merged model logits diverge from the baseline more rapidly than the pruned model.
The challenges expert merging. Model merging has been widely adopted to facilitate LLM fine- tuning. Why does expert merging miss the mark? In addition to the loss of the router's input-dependent modulation of experts explored in Section 3, we argue that the non-local nature of expert merging and high cardinality of expert clusters pose significant unresolved challenges. In Figure 4d, plot the mean relative L2-distance between experts clustered by HC-SMoE or M-SMoE and compare with the distance between ex- pert weights from the pretrained to Instruct Fine-Tuned (IFT) checkpoints. We find that the distance between

---

Published as a conference paper at ICLR 2026
clustered experts within the same layer greatly exceeds that of experts in the IFT checkpoint after fine-tuning Ito et al. (2024) found that weight matching pernutations improved alignment of parameters' singular vec- tors. Following their approach, decompose expert weights with Singular Value Decomposition (SVD) and plot the singular-vector alignment in Figure 4d. Even after applying weight matching permutations, the M-SMoE expert clusters remain far apart both in weight space and singular-vector alignment. The relatively poorly aligned experts highlight the considerable challenge of coherently merging their parameters.
When merging works well, it's more closely related to pruning than one might expect. In Figure A7a, we depict the frequency of singleton clusters - clusters containing a single expert - for both HC-SMoE and M-SMoE. A singleton cluster is directly analogous to an expert that remains after pruning. We find that HC-SMoE in particular has a high prevalence of singleton clusters, leaving important experts unadulterated and compressing the rest into a few megu-clusters containing tens of experts. This particularly true of the high granularity models which contain more experts per layer. We hypothesize that the cardinality of these mega-clusters poses a challenge for existing merging algorithms and test this intuition in Figure A7b. Unfortunately, even modest restrictions of the maximum cluster size to 32 - half the number of experts to compress - results in large decreases in model quality on coding tasks.
The importance of domain-speciftr calibration. In Figure A8, we plot the code generation accuracy of the various compression methods and models when calibrated on either C4 or vvol-codealpaca. The difference is stark, C4 calibration results in a collapse in accuracy, with several compressed model instances failing produce coherent outputs, resulting in 0% accuracy. In Figure A9, we compare the accuracy of compressed Qwen3-30B models calibrated with either domain-specifie data or the combined calibration data across all generative tasks. While oppain--pecific calibration results in higher accuracies, REAP best preserves accuracy compared to other compression methods in the combined data calibration setting.
6 DISCUSSION
Similar to prior work, we find that expert merging performs reasonably well on MC benchmarks. This may be because MC tasks only require discriminative function that can be approximated by an average expert. In contrast, merging fails to maintain model quality on generative tasks, particularly 50% compression and high-granularity architectures. Generative tasks require auto-regressive generation, a capability that impaired when the router's fine- graiied control removed or novel expert functions are introduced. Compared to expert pruning, merging is less consistent, exhibiting higher variance across models and compression ratios. The outputs of expert merged models are more repetitive and less closely aligned with the base model compared with pruned model's s outputs. Taken together, these observations are direct evidence of functional manifold distortion of the SMoE layers discussed in Section 3.3.
Overall, expert pruned models offer consistently higher accuracy than merged models on generative tasks. REAP is a robust pruning criterion that generalizes across a wide array of SMoE architectures, compression ratios, and generative tasks. By taking into consideration both the router gate-values and expert activation norms, REAP minimizes the reconstruction error bound by pruning experts which contribute the least to each layers output. REAP is scalable, achieving near-lossless compression on coding tasks with Owwe3-CCode--pee and Kimi-K2. The successes of REAP highlight the crucial importance of preserving coordination between the router and experts. Compression methods which impair the router's ability to independently modulate expert outputs or distort the original functional manifold are less likely to succeed.
Finally, this work highlights the importance of comprehensive downstream evaluations and the significant challenges involved with evaluating L.M Discriminative metrics such as perplexity and log-likelihood based MC benchmarks are not necessarily good proxies for generative model quality.
7 CONCLUSION
Our analysis of current SMoE expert merging techniques introduces irreducible error due to the loss of the router's independent control over experts. In contrast, expert pruning produces coordinate subspace of the original layer which maintains the topology of the functional manifold. We introduce REAP, a novel expert pruning method which prunes experts that contribute the least to the layer's output, thereby minimizing the reconstruction error bound. Empirically, we demonstrate that REAP retains remarkably high accuracy wide arry of generative tasks across diverse set of model architectures. We hope that this work inspires further compression techniques for SMoEs and facilitates the deployment of accurate, domain-specific models in resource constrained settings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
We would like to acknowledge the helpful feedback of Valavan Manohararajah, Mohammed Adnan, and Rohan Jain. This work was supported in part by RBC Borcalis through the RBC Borealis AI Global Fellowship Award. ML and YI gratefully acknowledge the support of Alberta Innovates (ALLRP 577350-22, ALLRP 600038-24) the Natural Sciences and Engineering Research Council of Canada (NSERC)
Defence Research and Development Canada and NSERC/Agence Nationale de la Recherche (ANR) (ALLRP 602719-24). This project was supported thanks funding from IVADO and the Canada First Research Excellence Fund. This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca) YI is supported by a Schulich Research Chair.
ETHICS STATEMENT
This research focused on the algorithmic compression of SMoE models and does not involve the use of human subjects, personally identifiable information, or sensitive data. The datasets used for calibration and evaluation (e.g.. C4, evol-codealpaca) are publicly available. Our aim is to enable the use of large-scale SMoE models in resource constrained settings. However, we acknowledge that compression techniques such REAP could potentially facilitate deployment of models for malicious purposes. Further, our compression methods are applied to pre-trained models and any biases related to fairness, discrimination, or representation inherent in the original models may be present in their compressed versions. We make no attempt in this work to mitigate these potential biases. The primary contribution of this paper is technical, and we do not foresee any new, direct ethical concerns arising from our proposed methodology beyond those already associated with the deployment of large language models.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our research. We have open-sourced our code and released select compressed model checkpoints facilitate further research on compressed SMoEs. REAP is formally described in Section 4. The baseline methods we compare against, including frequency-basee pruning, EAN, M-SMoE, and HC-SMoE, are formally defined in Appendix D. Section provides a detailed description of our experimental setup. including the specific models used, the calibration and evaluation datasets, and the implementation details for all compression experiments. Further evaluation details are provided in Appendix C.
REFERENCES
Red Hat AI and vLLM Project. LLM Compressor, 8 2024. URL https::// com/ vl vllm-project/ wtepfjeeetmmmmmmmeeeeeeeeeeeeeee compreess.o.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In Proceedings of the Eleventh International Conference on Learning Representations, 2023. htttt:://tppeeeeeeeeeee wet.ieewweee///eeeeem. forum? 1d=CQsmMym1P51
Allen Institute for AI. allenai/e4 Datasets at Hugging Face, August 2024. URL https:
//huggingfacee co/ /ttaaattt/aallaaaaiiiiai
Allen Institute for AL allenaiiulls3--ttpperronsss--s htt.ff/cc...coctctttttt/aaaeeeae httpaaaitttult-eaeee www.--teesssssamaaaaaa.eee
Datasets at Hugging Face, 2025. URL
https:
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary Vito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sher- lock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Trans- formation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume (ASPLOS 24). ACM, April 2024. doi: 10. 11453320065.3644000 36403666 URL ht htsttpp:://osss /occs. wrroccspytorre pytorch. org/ seagg/asets.... wttpttsssttttttthttt pof.

---

Published as a conference paper at ICLR 2026
Artificial Analysis.
Artificial analysis intelligence benchmarking methodology. https: lartificialanaall http://////tttooooooooo/eeeeeeee ber 2025. Version 3.0.
Septem-
Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning on realistic long-core cong-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3639-3664, 2025.
Baidu. Emie 4.5 technical report. 2025.
URL https://yiyen www.iyannnann.u..ommmma bwidu. wtt/a/lcoooolooooo /bog// publication /ERRIIE BERNIEeTeehiicaaaaaaa Report. .ddf.
Oana Balmau, Anne-Marie Kermarre, Rafael Pires, André Loureiro Espírito Santo, Martijn de Vos, and Milos Vujasinovic. Accelerating moe model inference with expert sharding. In Proceedings of the 5th Workshop on Machine Learning and Systems, pp. 192-199, 2025.
Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. -9-bench: Evaluating conversational agents dual-control environment. arXiv preprint arXiv:2506.09982, 2025.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. Answer Matching Outperforms Multiple Choice for Language Model Evaluation, July 2025. URL
http::/aaae org/abs/ 02856. arXiw:2507.02886 [cs].
Sahil Chaudhary, Code alpaca: An instruction-followinng llama model for code generation. http:/1//ooooaapaaee/ 2023.
pettss:/gitthne cow.
I-Chun Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hao Chao, Yen-Chang Hsu, and Chun- Yi Lee. Retraining-free merging of sparse moe via hierarchical clustering. In Proceed- ings of the Forty-second International Conference on Machine Learning, 2025. URL
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-Specific Expert Pruning for Sparse fixture-of-Expette June 2022. URL 00277. arXiv:2206.00277 [cs].
http://arxivs org/ abs 2000//0277.2
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ 1/N19 1300. URL https: acwannhhhlooooooo logy Mtgg/N11-1300000
Peter Clark, Isaac Cowbey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https: /atiiiv//rrivv.///aa/ org/ alos 11001180010 05457. arXiv:1803.0544 [cs].
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, Christopher Hesse, and John Schulman. Train- ing verifiers solve math word problems, 2021. URL http:: /ttpss:////vvv.. org/ ala/////// /21/2.10.1111 14168. arXiv:21 12.1111.4110 14168 [cs].
Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. DeepSeckMoE: Towards Ultimate Expert Specialization in Mfxture-of-Expeee Lan- guage Models, January 2024, URL http:/// wawpiviivv. wwwpi/...rraabbb.b.eaao.o..ooose 2240 201. 06066. arXiv:2401.0600 [es] version: 1.
12

---

Published as a conference paper at ICLR 2026
DeepSeek-AL, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang
Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. 7hang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. Cai, Jian I Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song. Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan. Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun. Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang. X. Q Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang. Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan. Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su. Xuheng Lin, Y. K. Li, Y. Q Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi. Yiliang Xiong. Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yonggiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu. Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. DeepSeek- V3 Technical Report, December 2024. URL htth: /awpiv org/ hatp///b///// 2222221222211 ..2...9444 19437. arXiv:2412. 19437
[cs]
version: 1.
Haojie Duanmu, Xiuhong Li, Zhihang Yuan, Size Zheng, Jiangfei Duan, Xingcheng Zhang, and Dahua Lin. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design. In Pruceedings of the Forty-second International Conference on Machine Learning, June 2025. URL https:/ htte:://ppperrrieeeee ew. fortt/form??ta wet.eeeooo2i.ionnnoooooeeeeeeee.
William Fedus, Barret Zoph, and Noam Shazeer. Switch Transformers: Scaling to Trillion Parameter Mod- els with Simple and Efficient Sparsity, June 2022. URL tattp://arxiin org/ MeXiv:2101.0396 [cs].
httg/ba/////oo//////////// abs /2001.21111.. 03961.
Leo Gao. Multiple Choice Normalization in LM Evaluation, October 2021. https://blog. eleuther.ail wwt.mmmiptlleechee.... www.cceeeoomaaaateiiieie
URL
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Lau- rence Golding, Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model eval- uation, 12 2023. URL. https: //gi thub. cww/ccmm///c..c. EE MenchhrrAAlmmeeee l uut wwwoaarnesssseeee
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems, 31, 2018.
Hao Gu, Wei Li, Lujun Li, Zhu Qiyuan, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decom- pression for moc-based LLMs compression. In Proceedings of the Forty-second International Conference on Machine I Learning, 2025. URL https: : / /twp//ppeevvveeeeeeeeeeeeeeeevve ew. net/ / forum? i d=ziezVi PoN1.
Shwai He, Daize Dong, Liang Ding, and Ang Li. Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques, March 2025a. URL WRRphttt:////ti..cot abs 220. 02500. uxXiv:2406.02500 [cs] version: 3.
Yifei He, Yang Liu, Chen Liang, and Hany Hassan Awadalla. Efficiently Editing Mfxture-of-Expert Models with Compressed Experts, March 2025b. URL http:/ ptst:://arxxe http///xxivvrrroe hasp//bb//200 22003. 00634. arXiv:2503.0M0 006300000344443 [cs].
13

---

Published as a conference paper at ICLR 2026
Dan Hendryeks, Collin Burs, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In Proceed- ings of the Ninth International Conference Learning Representations, 2021a. URL /ttps://openreeee ew, ew. www.nee//oormmiiimmm (o 3GmQ.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b. URL https:/ /ttps/////peeeeeieeeeee net forum? iid- 7Bytt
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He. Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QL Mixture compressor for mixture-of-experts LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ /oppereveerreeeeeeee net/ dorrummrmh.hhee.eosso......
HuggingFace, Open rl: A fully open reproduction of deepseek-rl, January 2025. URL
https://github. http:///gtttuuuboo/ooeees www.///ggongaaaeeeeeeeooooeeeeee baanngiingfacceeppnn-rti oope-rr..
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.. In The Eleventh International Conference on Learning Representations, 2023. URL https::///// httpp::://ppeeeee wwpp//eeeeeeeeeeeeee/eeeee net/ forum? id 6t OKwEB- jrjj..
Akira Ito, Masanori Yamada, and Atsutoshi Kumagai. Analysis of Linear Mode Connectivity via Permutation-Bast Bassdd Weight Matching: With Insights into Other Permutation Search Methods. In Proceedings of the Forty-Second International Conference on Machine Learning, October 2024. URL https:/ www.ereeeveeeeeeeeeeee eew net. forum? d= 1RKGGZZZK9 9D.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Proceedings of the Thirteenth International Conference on Learning Representations, 2025. URL httpt: http:://ppppevveeeeeeee net feet/fformm3 dd=.hfaayccccf1111....
Ajay Jaiswal, Jianyu Wang. Yixiao Li, Pingzhi Li, Tranlong Chen, Zhangyang Wang, Chong Wang, Ruoming Pang, and Xianzhi Du. Finding Fantastic Experts in MoEs: A Unified Study for Expert Drop- ping Strategies and Observations, April 2025. URL http: / httpiv.vvvvvooo// org/ abs /2504005588 05586. 3XXi:2554.05556 [cs].
Ajay Kumar Jaiswal, Zhe Gan, Xianzhi Du, Bowen Zhang, Zhangyang Wang. and Yin- fei Yang. Compressing Ilms: The truth is rarely pure and never simple. In Proceed- ings of the Twelfih International Conference on Learning Representations, 2024. URL
https:/ http:://pperrvveee wttt//orumm.dd99.15557777
Lewis Tunstall Ben Lipkin Roman Soletskyi Shengyi Costa Huang Kashif Rasul Longhui Yu Albert Jiang Ziju Shen Zihan Qin Bin Dong Li Zhou Yann Fleureau Guillaume Lample Jia LI, Edward Beeching and Stanislas Polu. Numinamath tir. [https: : http:s:///uhggaaaa h/thggggggfacceeee/eo/ooooo/a/ee
/o AI MM/ MO /nt//NNumiiaaaaa TIR] (https: /ttttt/b////bb////////ge thub. cot/projecttcccccoooooooooooocce numina na/ nammmooaaamommma/////ao//oooooo hataaaammooooreeeeeeee progress pwtpress--rizeeeee wwteeepiiiiiiiieeeeeeeeeeeeeeeee report/ www.umiianattttttt.ooom pdff,, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Amne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of Experts, January 2024. URL http://tp/eeee
httta:://axxvv..e http//axivvv...aaaaa/// abs/ /ta.bs//4400 401 04088. arXiv:2401 .04088 [cs].
Carlos E Jimenez, John Yang. Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github is- sues? In The Twelfth International Conference on Learning Representations, 2024. URL
https:/ http//ppreeeieeeeeeeee
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, Jialei Cui, Hao Ding, Mengnan Dong, Angang Du,
14

---

Published as a conference paper at ICLR 2026
Chenzhuang Du, Dikang Du, Yulum Du, Yu Fan, Yichen Feng, Kelin Fu, Bofei Gao, Hongcheng Gao, Peizhong Gao, Tong Gao, Xinran Gu, Longyu Guan, Haiqing Guo, Jianhang Guo, Hao Hu, Xiaoru Hao, Tianhong He, Weiran He, Wenyang He, Chao Hong, Yangyang Hu, Zhenxing Hu, Weixiao Huang. Zhiqi Huang. Zihao Huang, Tao Jiang. Zhejun Jiang. Xinyi Jin, Yongsheng Kang. Guokun Lai, Cheng Li, Fang Li, Haoyang Li, Ming Li, Wentao Ii, Yanhao Li, Yiwei Li, Zhaowei Li, Zheming Li, Hongzhan Lin, Xiaohan Lin, Zongyu Lin, Chengyin Liu, Chenyu Liu, Hongzhang Liu, Jingyuan Liu, Junqi Liu, Liang Liu, Shaowei Liu, T. Y. Liu, Tianwei Liu, Weizhou Liu, Yangyang Liu, Yibo Liu, Yiping Liu, Yue Liu, Zhengying Liu, Enzhe Lu, Lijun Lu, Shengling Ma, Xinyu Ma, Yingwei Ma, Shaoguang Mao, Jie Mei, Xin Men, Yibo Miao, Siyuan Pan, Yebo Peng, Ruoyu Qin, Bowen Qu, Zeyu Shang, Lidong Shi, Shengyuan Shi, Feifan Song. Jianlin Su, Zhengyuan Su, Xinjie Sun, Flood Sung, Heyi Tang, Jiawen Tao, Qifeng Teng. Chensi Wang. Dinglu Wang, Feng Wang. Haiming Wang, Jianzhou Wang. Jiaxing Wang, Jinhong Wang, Shengjie Wang, Shayi Wang, Yao Wang. Yejie Wang. Yigin Wang, Yuxin Wang. Yuzhi Wang, Zhaoji Wang. Zhengtao Wang. Zhexu Wang. Chu Wei, Qianqian Wei, Wenhao Wu, Xingzhe Wu, Yuxin Wu, Chenjun Xiao, Xiaotong Xic, Weimin Xiong, Boyu Xu, Jing Xu, Jinjing Xu, L. H. Xu, Lin Xu, Suting Xu, Weixin Xu, Xinran Xu, Yangchuan Xu, Ziyao Xu, Junjie Yan, Yuzi Yan, Xiaofei Yang. Ying Yang. Zhen Yang. Zhilin Yang. Zonghan Yang, Haotian Yao, Xingcheng Yao, Wenjie Ye, Zhuorui Ye, Bohong Yin, Longhui Yu, Enming Yuan, Hongbang Yuan, Mengjie Yuan, Haobing Zhan, Dehao Zhang. Hao Zhang, Wanlu Zhang, Xiaobin Zhang, Yangkun Zhang. Yizhi Zhang. Yongting Zhang, Yu Zhang, Yutao Zhang, Yutong Zhang, Zheng Zhang, Haotian Zhao, Yikai Zhao, Huabin Zheng, Shaojie Zheng. Jianren Zhou, Xinyu Zhou, Zaida Zhou, Zhen Zhu, Weiyu Zhuang, and Xinxing Zu. Kimi K2: Open Agentic Intelligence, July 2025, URL http: / / arxiv. org/ abs/ 2507 20534. wrXiv:250.225544 [cs].
Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory-efficiet NLLB--000: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Model. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3567-3585, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.11655/11/2003.accl URL https: / aaaaaaaaaatttoooo .argggy..rrg org/ 22033.222.222 acl-long.... 198/.
Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmeettd generation, 2024. URL hwtph:tt//stt. /tt//axxvvv.rr/. /ww.orr/////////o///e/eeee/eeeee abs abaaaass//00 /2409249000 12941.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling, 2025. URL https:////// openrevi ew net /formm? ? iid= ieeiiluubffHt luGfHHHHH.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In Proceedings of the Ninth International Conference on Learning Representations, 2021. URL https: / openreview net./ / forum? i id=qrwe TooveexxHTe TY.TmYb.
Pingzhi Li, Zhenyu Zhang, Prateek Yadav, Yi-Lin Sung. Yu Cheng. Mohit Bansal, and Tianlong Chen. Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policy. In Proceedings of the Twelfth International Conference on Learning Representations, October 2023. URL https:/ http///ppeeeeeeeeeeeeeeeeeeeeeee net wttt:/ooommmmmeeeeeeCC 3WK.
Pingzhi Li, Xiaolong Jin, Zhen Tan, Yu Cheng, and Tianlong Chen. OuantMoE-Benede Examining Post-Training Quantization for Mixture-of-Experts, February 2025.
URL.
http://arxiv. http://wxxii.oog/ab///////o//s// 2206. 201.2220611100 55.55. arXiv:2406.08 0555 [cs).
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
Bill Yuchen Lin, Yuntian Deng. Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Benchmarking L.LMs with Challenging Tasks
15

---

Published as a conference paper at ICLR 2026
from Real Users in the Wild. In Proceedings of the Thirteenth International Conference Learning Representations, October 2024a. URL Stth:tts:// httttt::://ppeeeeiiieee wppp/eeeee.eeeeeeee/pp/e. .nt/
http://wewiii..i.......aaaaaaaaa htte:/ooommmmmimmm/eeoeee 5xp..
Ji Lin, Jiaming Tang, Haotian Tang. Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024b.
Enshu Liu, Junyi Zhu, Zinan Lin, Xuefci Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient Expert Pruning for Sparse Mixture-of-FEperts Language Models: Enhancing Performance and Reducing Inference Costs, July 2024a. URL
http:///artt http://axvvv..../aa/eee org/abss2407 00945.. MtXiv:2407.00944 [cs].
James Liu, Pragaash Ponnusamy, Tianle Cai, Han Guo, Yoon Kim, and Ben Athiwaratkun. Training-Free Activation Sparsity in Large Language Models, October 2024b. URL
http://arxx: orgg/bs//4055 14690. arXiv:2408. 14690 [cs].
Jiacheng Liu, Peng Tang, Wenfeng Wang, Yuhang Ren, Xiaofeng Hou, Pheng-Am Heng, Minyi Guo, and Chao Li. A Survey on Inference Optimization Techniques for Mixture of Experts Models, January 2025a. URL http://///aae .ww.ivoor//aaaa abs 2412. 14219. #EXiv2241.142211 [cs] version: 2.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558 21572, 2023.
Zechun Liu, Changsheng Zhao, Hanxian Huang. Sijia Chen, Jing Zhang, Jiawei Zhao, Scott Roy, Lisa Jin, Yunyang Xiong, Yangyang Shi, Lin Xiao, Yuandong Tian, Bilge Soran, Raghuraman Krishnamoorthi, Tijmen Blankevoort, and Vikas Chandra. Paretoq: Improving scaling laws in extremely low-bit L.MM quantization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025b. URL https: hpwnr/eeeeeeeeeeeeeeeeeeee net. /oormm? PMSNNxXTX.
Zuxin Liu, Thai Quoe Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh R N, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang. Shelby Heinecke, and Caiming Xiong. APIGen: Automated Plpeline for generating verifiable and diverse function-calling datasets. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Truck, 2024c. URL https:/ http:://ppppeeveeeeee http://ee//oooooe wtt.:o/rru...idg0o///22/22
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 6159-6172, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1/255/w long.334. URL https: htttlps://cccie /aclanthhooogy...... 2024 1...024.cc. aalllllllnnno 334/-
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma. Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In Pruceedings of the Twelfth International Conference on Learning Representations, 2024. URL https: / http:///pppeeeeeeeeeeeeeeeeeeeea .et. forum? (iwoorummmnnnnuussiss555
Meta Al Team. The Llama herd: The beginning of a new era of natively multimodal AI innovation, 2025. URL https: / //ai meta. cwm/blooooooooooooooooo olog//////ooooo 1lama- 4-uul cimodal-iinni well-nnneelll liine/e//.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
ModelScope Team. EvalScope: Evaluation framework for large models, 2024.
URL
https://github.. ctt../.coooooooooo. wwweeessccceeeeeoooo eeaoopppeealscopp...mm
Alexandre Muzio, Alex Sun, and Churan He. SEER-MoE: Sparse Expert Efficiency through Regu- larization for Mixture-of Experss,. April 2024. URL http: wwwpixxvvvvoooo org/ abs//404..44 05089. arXiv:2404.0508) [cs].
16

---

Published as a conference paper at ICLR 2026
Neil Chowdhury, James Aung, Chan Jun Shem, Oliver Jaffe, Dane Sherbum, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang. Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified, August 2024. URL https:/ / i com/ ndex/ cew.roooouiiiisseeeee beeeee-bbnch-eeeee fied/.
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model con- nected with massive APIs. In Proceedings of the eiihty Annual Conference on Neural Information net.
Processing Systems, 2024. URL https: http:::://///peevveeee
BRNG- 6YemY.
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference Machine Learning, 2025. URL
atttps://opens
net. forrm//ormmmm/ 2mDDaDD
Jade Pritsker. euclaissWVittngPPoppsscrraaceee Datasss Datasets at Hugging Face, December 2024. URL https://busginggaces http:///ca...cacttttttttt//a/aaa htwpr//aaseeeetttinggeeooooeeeee wowptttsssrroossssssseossmssssss _urrtted.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
RedHatAI.
Hugging Face, September 2025. URL 14.16.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial
winograd schema challenge at scale. Communications of the ACM, 64(9)99-1066 2021. URL
htttps///dl..ac..oo//cl////// https: //l. acm. wtt.c//wwwo//../////ooooooo /tt1////////11111//44444553
Salesforce. Swwefffore/tannntttttoooonlnnnnn Datases
Datasets at Hugging Face, May 2025. URL https: ///puggiigaace.co//ttttttt/////s http:////...aaaaaeeaaaaaaaaaaaaa function-callinge 60k.
Ekansh Sharma, Daniel M. Roy, and Gintare Karolina Dziugaite. The Non-Local Model
Merging Problem: Permutation Symmetries and Variance Collapse, October 2024. URL
http://arxiv.org/// wwwp//rg///b/aaeeee 11.....6........ arXiv:2410.12766 [cs].
Noam Shazeer. GLU Variants Improve Transformer, February 2020.
URL
http:
//arxiv.org abs/ 2002. 05202. arXiv:0002.05202 [es].
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously Large Neural Networks: The Sparsely-Gate Gated Mixture-of- -fxtuuee-ofEExperss Layer, January 2017. URL hatp::/axxxi /bsg//////// 111111111111 06538. arXiv:1701.06553 [es, stat].
Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony. Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaci, Irem Ergün. Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrnann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024.
Zhi Rui Tam. thebaekkcal//eeeooccccaaappeee/e Datasets at Hugging Face, July 2023. URL http///eeeoooooaaacaaaaaaa
https: httpss//hhggnnngacceeee http://aaaaaaaaataaaaaeeteaeaeae
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. In seirth Conference on Neural Information Processing Systems, 2023. URL https: /ttpr/vieee/eeeeee///m/em/e net. forum? wit ax3WyC
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang,
17

---

Published as a conference paper at ICLR 2026
Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 Technical Report, May 2025a. http:tp:/ arxiv. org/ abs / 2505. 09388. DXXv:2505.09388 [es].
Cheng Yang. Yang Sui, Jinqi Xiao, Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. MoE-i": Compressing mixture of experts models through inter-expert pruning and intra-expert low-rank decomposition. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 1046610466, Miami, Florida, USA, November 2024a. Association for Computational I Linguisties. doi: 10. 1/2234 findings-emmlp.612 URL https: httaaacaannnoooooooooooooooooooe .rg 2024. f Findings-ennt -aiggg-emmm 1p. 612/.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Proceedings of the Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL htw.s://///// openrevi ew .net/ /ftt//foumm// ? du= 8J3.
John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. WWWE-benc/SWE-eee -aap://ss---miitttt-rrrrrtrtttie Datasets at Hugging Face, May 2025b. URL https: /hugg 1 nacace. co wttp//aaaasaaaaeeeeeeeeaa SwB- bench// /Sw////WEa-/ smi th-tta jectories.
John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig. Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling Data for Software Engineering Agents, May 2025c. URL http: /aapiv/aaxivvvv/aaa org/ abs 2abs/2550 25004 21798, arXiv:2504.21798 [cs].
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4808 1-4000, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10.18653/1//119-4777 URL https: / aalaacaaaa //////aaaanntttoo orw.eooooooooooooooooooooo
Aohan Zeng, Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang. Da Yin, Hao Zeng, Jiajie Zhang, Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang. Zilin Zhu, Bohan Zhang. Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang. Ding Ai, Erle Zhu, Fei Wang. Gengzheng Pan, Guo Wang, Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng. Hao Tai, Haoke Zhang. Haoran Wang, Haoyu Yang. He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang. Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang. Lekang Yang. Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang. Man Zhang, Minghao Li. Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang, Shaoyou I Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang. Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang. Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li. Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng, Zhuorui Liu, Zichen Zhang, Zihan Wang, Zijun Yao, Zikang Wang. Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang. Juanzi Li, Yuxiao Dong, and Jie Tang. GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation http//bb//55000 06471. arXiv:2508.066 066401 [cs].
Models, August 2025. URL http::/://// /tt::://aaxiii
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024. URL https: httpss//ppppeeieeeeee .ee. forum? id Su 7RRMMM
18

---

Published as a conference paper at ICLR 2026
A EXTENSION TO HIERARCHICAL CLUSTERING
While Equation (5) analyses pairwise merging, practical implementations often employ hierarchical clustering to form groups of experts. Consider a cluster C²= (.,,) of & experts merged into a single representative fc. The original contribution of this cluster can be decomposed as:
Eg,
Σ 9i,(x) (x)
(w,(t)fi(x))
(10)
JEC
JEC
JEC
Dynamicc input- dependeni mixture
where wy(x) 9y (i)
the within-cluster mixing ratios that sum 1.
9t)
After hierarchical merging, the router must apply the summed gate Ejecii, to a single, static cluster representative fc. typically computed as a weighted average the cluster members based on calibration data. This induces an irreducible error.
Hierarchical clustering error For a cluster merged into fc Ej=Ejecayff with fixed weights aj 2 E = =1, the minimal 2 error is:
min fos
Σ IEC
gis fú
Σ JEC
9hg
fc
=E
Σ
gis
Var
(11)
JEC
jEC
The error grows with both cluster's total gate-value and the variance of the dynamic mixture that cluster must approximate with static representative.
Implications for cluster formation. The hierarchical error bound reveals a fundamental tension:
Large clusters
(/C) large) aggregate more gate-value Ejechh, amplifying any approximation
еттог
Diverse clusters (high | for i.j C') increase the variance term, as the static representative must approximate wider range functions
Imbalanced clustering (many singletons, few mega-clusters) combines the worst aspects:
mega-clusters suffer severe collapse while singletons provide minimal compression
Distance metrics like Euclidean distance that consider magnitude can exacerbate these issues by creating clusters based on norm similarity rather than functional role, potentially grouping experts with different specializations but similar scales. The resulting mega-clusters force the router apply single control signal to what were previously dozens of independently modulated experts, explaining the catastrophic functional collapse observed empirically in late layers where Var[wy(=:] is highest.
B ADDITIONAL EMPIRICAL EVIDENCE FOR LOSS OF INDEPENDENT CONTROL
B. FUNCTIONAL SUBSPACE PCA ANALYSIS
Qualitative evidence of functional subspace collapse. In Figure 1a, Qwen3's layer 0 exemplifies the contraction of the functional output space by merging in early layers. The original 128 experts span from -0.4 to 1.0 along PC1, pruning maintains this full range with 64 experts, while merging contracts the distribution to approximately 0.22,0.3) 3, 5-fold reduction. This contraction is dramatic in late layers, where experts are more specialized, as can be seen in Figure 1b. Figures A5a and A5b exhibit similar contractions of the expert output manifold under merging, whereas pruning often preserves outlier experts and the span of the original expert output manifold.
In Table A4, we tabulate the total cumulative variance explained by PCI+PC2 for the PCA projections in Figures and A5. For low- www.granllrrity SMoES such as Llama-4-Scoue and Mixtral, PCA1 and PC2 capture most of the variance in the activations. Even in gigh-rraullarit SMoEs such as Qwen3 and ERNIE. large portion the total variance is captured by PC1 and PC2. The merged variance explained is consistently higher than the baseline, suggesting that the merged outputs have lost some of their high-dimensional
19

---

Published as a conference paper at ICLR 2026
Original Experts
Surviving
Merged
Original Experts
Surviving
Merged
2
20
-50
-50
PC
PC1
PCE
PC
PC2
PC3
(a) BRNIEE4.5-21B Layer
(b) ERNTE-4.5-21 Layer 27
Original Experts
Surviving
Merged
Original Experts
Surviving
Merged
20
0
00
-20
-04
PCL
PCL
AC3
PC1
(c) Mixtral-8x Layer 0
(d) Tixtral-8x7B Layer 31
Original Experts
Surviving
Merged
Original Experts
Surviving
Merged
0.6
200
0.00
g
-0.0
-200
(c) Llama-4 S-out Layer 0
(f) Llama 4-Souut Layer 47
Figure A5: (a,c,e) Functional subssace (PCA) for early SMoE layers. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (b,d,f) Functional subspace (PCA) for late MoE layers.
complexity. In contrast, the pruned variance explained consistently lower than the baseline, suggesting that pruning preserves outlier experts and the high-dimensional complexity of the baseline model.
The role expert granularity. Both Qwen3-30B-A3B and ERNIE-4.5-21B are highly-grunlarr SMoEs containing 128 and 64 routed experts per layer, respectively, and 8 and 6 routed experts per token, respectively. Functional subspace collapse due to expert merging is more pronounced in these models than in low-granalrrit models such Mixtral-&x7B and Llama-4-Scuut. With fewer experts and lower amount of experts per token, low-granularigg SMoEs appear to better preserve the variance of their expert output manifolds under expert merging. For example, in Figure A5c, the merged manifold spans along PC1 from approximately [-0.1,0.2] whereas the pruned manifold spans from approximately [0.0,0.2) along PC1. Similarly, as depicted Figure A5e, the pruned and merged manifolds span PC1 along 0.00 and 0-005,00.05), respectively.
However, the merged manifold is distorted by the introduction of novel expert functions. For example, in Figure A5e, expert merging introduces novel functions which occupy approximately 0.05,0.005 and 0-0025.000055 which are significantly different than any of the original experts. This best exemplified by Figure 2, which plots the Wasserstein distance between the original and compressed expert output manifolds in terms of normalized angular distance. Compared to the pruned models, the higher distances between the merged and original expert output manifolds suggest lower degree of similarity. The distorted manifold the merged expert outputs represents a loss of fidelity with the original manifold which cannot be restored in the one-shot compression setting.
B.2 WASSERSTEIN DISTANCE
To quantify the distortion of the original expert output manifold, we measure the 1-Wasserstein distance (Farth Mover's distance) between the original and compressed expert output manifolds, see Figure 2.
The distance is calculated between the discrete empirical distributions of the original expert outputs, F= (fi,...J (K) ), and compressed expert outputs, f= [(xz). projected onto the unit hypersphere
20

---

Published as a conference paper at ICLR 2026
Table A4: Cumulative variance explained by PCI and PC2 across compression methods. Compared to
pruning, merging results in consistently higher explained variance suggesting that the merged models have lost some of their high-dimensional complexity.
Model
Layer
Baseline
Merged
Pruned
Qwen3-30B-A3B
0
0.2343
0.2700
0.1845
Qwen3-30B-A3B
47
0.7195
0.7437
0.6860
ERNIE-4,5-2IB
0
0.3836
0.2851
0.2733
ERNIE-4.5-2IB
26
0.2563
0,4599
0.0785
Llama-4-Scour
0
0,9032
0.9343
0.8480
Llama-4-Scout
47
0.9473
0,9546
0.8754
Mixtral-8x71
0
0.6486
0,8479
0,4016
Mixtral-8x7B
31
0.8580
0,8140
0.7027
K
K/2
1 79
fz-Jz |
W1(F
inf
arccos
(12)
(EN(MP))
π
where f and v are uniform probability measures over the indices of F and JEr respectively, I(u,v) is the set of all transport plans (joint distributions) with marginals and and the cost function is defined as the nor- malized angular distance. This metric quantifies the minimum "work" required to transport the probability mass from the compressed functional manifold to cover the original manifold, thereby penalizing both the contraction of variance (subspace collapse) and the introduction of functionally distinct artifacts (distortion).
C EVALUATION DETAILS
Multiple choice (MC) evaluation. Following Chen et al. (2025), our MC benchmarks include: A12 Reasoning Challenge (ARC-c & ARC-e) (Clark et al., 2018), BoolQ (Clark et al., 2019), Hel- laSwag (Zellers al., 2019), MMLU (Hendrycks al., 2021a), OpenBookQA (OBQA) (Mihaylov et al., 2018), Recognizing Textual Entailment Challenge (RTE) (Bentivogli et al., 2009), and WinoGrande (WinoG.) (Sakaguchi et al., 2021). We evaluate the models in the zero-shot setting using the standard log-likeline lellikiiihodd approach with Im-eval-harness (Gao et al., 2023). We report byte-length normalized accuracies for ARC-c, ARC-e, HellaSwag, and OBQA'.
Coding evaluation. For code generation, all models are evaluated on EvalPlus (Liu et al., 2023) and 182 LiveCodeBench (Jain al., 2025) questions collected between January and April 2025. We extend the original source code for these benchmarks to evaluate our models. We additionally evaluate Kimi- K2-Instruct-Ve -4rcct-4A1666 16 and Qwen3-Coder-48001 on the agentic coding benchmark SWE-Bench (Jimenez al., 2024) and tool-calling benchmark BFCLv3 (Patil et al., 2025). For BFCLv3, use the original Gorilla framework for evaluating our models (Patil et al., 2024).
For SWE-Bench evaluation, run our compressed models with the mini-SWE-ageen scaffolding (Yang et al., 2024b) and report the score on the SWE-Bench Verified test set (Neil Chowdhury et al., 2024). We use 4,096 and 16,384 as the maximum number of output tokens for evaluating Tive3-CCoder-800 and Kimi-K2- Somi-K2-Inntrrin Instrnctutttee W4A16 on SWE-Bench, respectively. The input context length for both models limited to 65,536. We do not limit the number of turns in mini-SWE-agent flow, but restart the rollout in cases where the model could not generate a valid patch (that is, in the case when the output of the final turn does not contain diff --git substring). We set the maximum number of restarts to 20. which we found to be sufficient to generate patches for all samples with pruned models, unless the model produces degenerate responses like repeating strings. We use the cloud-based evaluation provided with the sb-cli tool to get the final scores for all evaluated models.
For be-hench Barres et al. (2025), we use greedy decoding and 4,096 as the maximum number of output tokens for each LLM call. For user simulation, we use the gpt-4 1el.11-22----- model;
Reportedd the acc_norm field in the FleutherAT evaluation hamess outputs. See Gan (2021) for details.
21

---

Published as a conference paper at ICLR 2026
maximum number of steps is 100 and number of trials is set to three for each domain. Following Artificial Analysis (2025), we additionally implement an LLM- repetition checking step. Every 30 steps of the simulation, model (in case, gpt-4. 114..-1mmnii2222550 20000-20200---0
is given the past 30 episodes of the conversation trajectory with a repetition checking prompt to determine whether the agent is stuck in the loop or making meaningful progress. This allows early task termination if the agent is stuck. We use the same decoding parameters for the repetition model as for the user and assistant models.
Math and creative writing evaluation. Mathematical reasoning assessed on GSM8K (Cobbe al., 2021) and MATH-500 (Hendrycks al., 2021b; Lightman al., 2023) benchmarks using the evalscope (ModelScope Team, 2024) framework. To assess creative writing, we use 146 creative writing prompts sampled from WildBench (Lin et al., 2024a) with gpt pot-4o-2000 2024-05-22---- -13 used as the judge to evaluate the model responses. We report normalized scores using the WildBench rubric.
Generation configuration. For models with < 110B parameters, we use greedy sampling (i.e, temperature 0.0) evaluate code generation and math reasoning. For creative writing use the default temperature, top-P, and top-K settings for each respective model. The maximum number of output tokens is extended to 16,384 for all generative tasks to account for the verbosity of some models. For hybrid reasoning models such as Qwen3-30B- -0---BBBBB3B,, we disable reasoning on all tasks by setting enable_tiin potplllethinkinggFaaaee se the chat template.
For larger models with 110B parameters, we use greedy sampling for EvalPlus, SWF--ench, and BFCLv3. On LiveCodeBench, Qwen3-Coder-480N and Kimi-K2 are evaluated with default sampling parameters and greedy sampling, respectively. We report the mean and standard deviation for Qwen3-Coder-4800 on LiveCodeBench over five random seeds. We use a repetition penalty 1.05 for all large model evaluations. For EvalPlus we use 768 the maximum number of output tokens and 16,384 for LiveCodeBench. For BFCLv3 we set the maximum number output tokens to 4,096.
Model details. The SomiiK2-Instruct--ar weight-quantized version of Kimi-K2- In-trut released by RedHatAl (2025).
model used throughout this study is an INT4
D BASELINE METHODS
The following formally describes the baselines compression methods consider.
Notation. Let Xeat be calibration dataset. Consider a SMoE model with layers, Ln, K experts per layer f1,... fK. each a funetion fx Rd + R4, and a router producing non-negative gates g(r)= (x)....... A The output of layer L,, is
hn Σ (x))).(.).
The expert usage frequency, V4, for expert f. is the number of tokens in Neat for which fa is activated
14 = ==
where (x Xeat |i TopK(g(z)))
Given saliency scores, S S pruning removes experts with the minimum saliency score. For merging, we first cluster experts based on their pairwise distances, D e xx and then merge the parameters of experts contained within each cluster.
Firquuencybaase pruning, The frequency-based pruning saliency criterion prunes experts with the lowest usage frequency across the calibration dataset. The saliency of f/ is simply S, 5.
EAN pruning. EAN pruning introduced by Jaiswal al. (2025) accumulates the activation norm of each expert across tokens for which the expert is activated. The saliency of fa is
S. = Σ [l)(())
(13)
22

---

Published as a conference paper at ICLR 2026
M-SMoE merging. Proposed by Li et al. (2023), M-SMoE first uses weighttmmtchhnn (Ainsworth al., 2023) to find a permutation matrix Pj which aligns expert f, to expert fi. In the models we study, each expert is two-layer feed-forward SwiGLU block (Shazeer, 2020) with up, gate, and down projections: gates Wil do The permutation matrix is applied to the intermediate dimension of experts such that the expert outputs are invariant to the transformation
ʃj = (Wap) w
W up
= W
W gate
W
gate j.
W down Pj W
down-
The permuted expert defined fj (W W Wa gate down
To initialize the expert clusters, M-SMoE identifies the set of m dominunt experts IF Fdoms as the experts across all layers with the highest usage frequency 'The pairwise expert distance based the cosine distance of the router gate-values measured on the calibration dataset
Daj
1
Σ
1-
(:(x) (g) (1,,,)) (x)]
(14)
Keall
Non-dominant expert j is clustered by selecting the dominant expert with the smallest pairwise distance
= argmin
iET dom
The merged expert fa is created by calculating the requency-weightee average of the permuted parameters, W', of all experts in the cluster C.
W. Eiec, WW
(15)
Execom
HC-SMoE merging. Chen (2025) clusters experts based on their representative vectors, A,, defined as the average activation across every token in the calibration dataset
A A.=
Σ
f((x).
Krral
The expert pairwise distance is defined as the cosine distance between representative vectors
A-A,
D
(16)
|A
Clusters are formed using hierarchical agglomerative clustering with average linkage criterion. We start by initializing each expert as singleton cluster. At every iteration, the closest pair of clusters, C2C are joined and the pairwise distances updated as the average of the constituents
Eieco Di.k [Ca
=argminDis:
Da,k
ij
The clusters are merged with Equation (15).
E QUANTIZATION AND EXPERT PRUNING
Pruming easily combined with quantization, as demonstrated by our large-scale evaluations. We calibrate, prune, and evaluate (pwen3-Coder-442 and Kimi-K2 using FP8 and W4A16 quantization, respectively. In contrast, combining expert merging with quantization more complex; the fine-grained shared quantization scales across parameter groups must be reconciled during merging, whereas pruning requires no such adjustment
Generally, weight quantization offers superior accuracy at given compression ratio down to 4-bits. For instance, quantizing Qwen3-30B-AM to 4-bits using AWQ (Lin et al., 2024b) outperforms a 50% expert- REAP model at 16-bits, despite having half the checkpoint size. However, combining these methods enables compression rates that neither can achieve in isolation. For example, our Kimi-K2 results combine 4-tit weights with 50% expert pruning, representing a total size reduction of 87.5 87.5%. Achieving
23

---

Published as a conference paper at ICLR 2026
similar footprint through quantization alone would require 2-bit weights, which are not natively supported on most current accelerators and typically suffer from significant accuracy degradation (Liu et al., 2025b).
To further illustrate this, we calibrate and quantize Qwen3-30B 301--3BBBB3BBB using the cool-codealpuca-v/ dataset to 4- and 2-bit weights with the LLM Compressor library (AI & vLLM Project, 2024). We additionally compress the 4-bit checkpoint by using REAP calibrated on the same dataset to prune half the experts. In Table A5, we report EvalPlus accuracy versus checkpoint size relative to the uncompressed 16-bit model. This preliminary analysis suggests that REAP combined with 4-bit weights outperforms more aggressive quantization. A more thorough exploration of various low-bit schemes and pruning combinations remains an important direction for future work.
Quantization
Num. Experts
Approx. Relative Size
Eval+
W16A16
128
100.0%
81.4
W16A16
64
50.0%
78.0
W4A16-G128
128
25.0%
80.5
W4A16-G128
64
12.5%
77.6
W2A16-G128
64
12.5%
28.6
Table A5: Qwen3-30B-A3B EvalPlus accuracy vs. relative checkpoint size with AWQ quantization, REAP expert pruning, and quantization combined with expert pruning. Overall, quantization outperforms expert pruning up to 4-bit weights. Beyond 4-bit weights, combining quantization with expert pruning yields significantly better accuracy than more aggressive quantization.
F ADDITIONAL RESULTS
Table A6 shows the full suite of MC question answering benchmarks and the average result across all models and methods. Table A7 tabulates code generation accuracy of compressed SMoE models calibrated on evol-codealpaca. Eval+ is the average of MBPP+ and HE+. The Code Avg column is the average of Eval+ and LiveCodeBBench (Liveeode)). Table A8 summarizes the accuracy of the various compression methods studied when calibrated with the C4 dataset on coding and MC benchmarks. Notably, while the MC performance is generally slightly higher than models calibrated on evol-codealpaca, the resulting code generation quality is abysmal, with most models failing to generate coherent output.
Figure A6 plots non-agentic coding and MC accuracy versus compressed model size. Figure A7a depicts the proportion of singleton clusters for HC-SMoE and M-SMoE. Figure A7b plots accuracy vs. maximum cluster sizes when the maximum cardinality of clusters is restricted. Figures A8 and A9 show the importance of using domain-spocifie calibration data, particularly at high compression ratios.
Table A9 presents the complete r"-bench results across three domains (Retail, Airline, and Telecom) for the baseline model and REAP compression at 25% and 50% levels. The results show pass"k "kss metrics for k=1, 2, and 3, demonstrating the impact of pruming evaluating conversational agents, specifically designed to test their ability to collaborate with a user in real-world scenarios.
Table A10 presents additional -2-bench results with REAP compression for of non-reasoning and reasoning models (Qwen3-Coder-308B -irr--ii,, MiniMax-M2), Furthermore, Table A1l presents BFCLv3 scores for the same of
models (with an inclusion of REMP-compresseed GLM-4.6FFPS variants in thinking mode). The calibration data mix used for non-reasoning models in these experiments is the same as for the other large-scale models in this study (12,228 samples including tool calling and agentic trajectory data). For the models that support reasoning, we include an extra 12,228 samples from the Open-R Mixture-of- of-Thoughts dataset (HuggingFace, 2025), drawing 4,096 samples from the Code, Math and Science subsets with a maximum sequence length of 16,384 tokens.
Table A12 shows results for the Kimi-Linear-48f. -ineer-48B---- Instruct model as an example of REAP compression applied to hybrid linear attention architecture. The calibration data mix is made up of 24,414 samples with a maximum sequence length of 16,384 tokens. The samples are drawn from evol-codealpaca (Chaudhary, 2023; Luo et al., 2024; Tam, 2023), xlam-function-calling (Liu et al., 2024c; Salesforce, 2025). -VE-smmith-trajectories (Yang et al., 2025c;b), WildChat (Zhao et al., 2024). NuminaMath (Jia LI & Polu, 2024) and Aya (Singh et al., 2024). The calibration data mix composition reflects coding, tool calling, multilingual and general conversational use cases. We evaluate
Linear-48B-A3 A3B-IInsttttt on coding and math benchmarks at 30% REAP compression, well long-context muestion-nnswerigg
24

---

Published as a conference paper at ICLR 2026
Table A6: Detailed benchmark results for multiple-choice QA tasks.
Medii
Cargmokn
Techeique
Nethed
ABICE
ABIC
Beelq)
NHILU
OBRAA
IEEE
MLC AvK
0361
0:31
MerpPn
M-3304
0.596
3306
0717
5549
5166
1000 TURO
HAIK 1-99
25%
0:466
3100
0100
3957
DATT
DACR
0750
DATA 103
06-88 11011
KTAP
0:366 9- 5761
5.996 01006
0958
D100
6/784 D10
DV06
D/00
0100
0/705
0006 000
[U67 (199)
HC:SMIE
0.411 6: 9.009
0.41: 0.002
0.22
0.00
0523- 0000
B4R5 0000
0350 0005
0.72 0011
0.97 0009
0.59 0.001
EAM
0417 3306
2.33
0165
03:38
5160
0372
0209
DUKC
0336
DURO
00014
(U22
0,50
RTAP
0.400 3766
0108
0/06
09.00
0100
0/0A
0.00
0000
0.511
0000
0.00
000
DALL
0000
0.901
0.790
0:97
0.71
0279
TU516
U21C
10721
Mirping
HD:SM
0.478
3306
0.723
5.66-
D529 TUNC
TUNA
1530 2 JUNC
IUSE
25%
0:488
3.011
3700
0105
33117
D100
UMM 11015
RCAAP
01111 5161
5.333 01000
5355
0161
5.706
D101
10966
07708
100
0/06
100
HC:SMIE
03368 8.00
0.53 0.009
0.440
0/00
0473 0000
0516-
0000
0.501
0007
0.724
0000
0620 0005
0,543 0.001
Prasin
EAM
0.36
3306
0436
0109
5.99
5000
DA31
DA43
0001
0229
0U00
00220
0390
0000
0448 0U01
RTAP
9.94 3106
0495
0106
0/06
0/661
D101
0/066
D/00
0.066
0000
0/96
000
0.60
0.42
0.87
0363
1022
1L756
D106
0553 (UMG
1010
1004
HC MM
0.990 5766
5.477
0/661
59688
25%
0.36
2.04
39.66
51.00
3315
2106
0325
BMCG
LOM
REAAP
0030 0, 5101
0933 01000
0100
D830
D100
D/66 11
10771 100
17706 100
17715 (199)
HC-5M
0.39 6 5.00
5.99- 8 0/000
0551-
0/00)
0.791 8 0/001
0.433 -1 0/00
0143
0/00
0.700
0000
0.113 4 0000
01667 0.000
Prining
BAN
0911
5261
0.131
5106
0999
0161
6/911
D100
D100
D/63
17717
TATT (L9HT
RGAAP
03% 3006
0.89
0/601
0811
0/00
0/77
(/00)
D/00
01/58
1000
0667
1095
(1571 11001
0667
0.441
0,79
0323
0000
03422
0216
0301
6051
0319
0200
0671
HI-SMa
6.94
0311
0451
25%
Frogurk
Prankng
6.62
0335
0382
6313
0775
0446
0221
Intrect
REAP
6:544
0330
0372
0294
0776
0453
0260
1005
ILT0R
Mapis
HU-SMal
M-SK5E
6:11
6.35
611
6.91
0876
0981
6371
0676
0003
0086
0026
1060
Proving
BAN
6.00
6390
0381
6371
0396
0063
1666
BAKS
KEAP
6561
6302
6969
6345
0962
0432
0202
0.994
6689
0325
0382
0229
0289
0473
0047
HC-SMcE
6397
6382
6969
0815
0722
0428
0.66
0.755
0,714
25%
GLMA55-Ai
BAM
6358
0365
6.76
lues
1208
1060
KEAP
6.555
6.799
098
0799
0700
0430
0600
0.755
Mapic
HC-SMal
M-SMB::
6423
0:51
6153
0,01
0961
6781
0.113
0.50
0.54
0315
00266
0.30
0000
0.511
0005
0L964
6535
Proning
BAN
6358
0530
0981
0351
0
1300
10066
1008
6413
6586
0628
0312
0.311
0605
0661
0333
0966
0881
0000
0.66
0751
0717
(175))
Quack-Codar
25%
Praking
EAN
8555
6366
638)
6769
0755
0.04
0347
0691
0.202
RIGAP
6671
6331
0960
080
0006
0406
0051
0022
Proquecc
6354
6470
0391
0302
0481
0222
0070
08800
(LSN))
50%
Preking
EAN
6.02
6566
0851
0925
0605
0316
0764
0996
Roselline
6313
0334
0913
076
0002
005
0760
6001
(1780)
Kirs- letruc-
250g
Prasigg
EAN
6513
6663
6338
671
0387
0835
0781
0347
0646
0665
REAM
6.01
6354
0907
0260
0089
0479
1086
(089)
6385
6154
0005
0416
631
DAT7
0.500
Pradin
6425
0463
0388
0/26
benchmarks LongBench v2 (Bai et al., 2025) and FRAMES (Krishna et al., 2024). For long-context benchmarks, a truncation in the middle is done for input prompts exceeding 131,072 tokens in length. The accuracy scores for FRAMES are estimated with LLM-as-a-judge (gpt-4. 1-011--22------- as judge). Notably, even though the maximum sequence length during calibration is 16,384 tokens, the REA DEEpeecomeesseed model retains performance at context lengths far exceeding that value.
model
25

---

Published as a conference paper at ICLR 2026
Table A7: Detailed benchmark results for non-agentic code generation tasks, Eval+ is the average of MBPP+ and HE+, The Code Avg column the average of Eval+ and LiveCodeBench (LiveCode).
Medel
Compresdon
Technigu
Mithol
HF:
HE+
MHRPP
MHPP-
Fval+
Live
Cada
Basclin
0,922
[L965
0.910
0.765
0315
0.231
0.523
Meriing
M-SM-E
HCSSMdf
0.857 = 0.007
0.774 = 0001
0.815 :: 0.000
0.730 # 0.009
0.768 + 0105
0,827
= 0.003
(1647 #: 0.017
0.696
:: 0.005
0.68
0.750
0.007
0.004
0,207 :: 0.000
0.194
0.022
0.441
0.470
0.011
0,00
ERNIE-AS 4.5 21B A3B PT
25%
Prusiny
Frquaacy
0:00 0.00
0:00
0.846
0.84
0.000
0.011
0.857 0.000
0:540 0.006
0.709
01777
0.010
0.004
0.77
0.87
0.095
0.033
REAP
0.09 0.12
0.865
0.004
0.66 =: 0.006
(1735
0.002
0.00
0.512
Merging
M-SM-
HC
0:022
0.004
0.007 0.404
0.009
0,688
0.250
0.036
0.088
0.11
0.207
0,040
0.00
+0005
50%
0,699 0031
01640
0U022
0,06
0.14
0.584
0,005
0612
0.01
0.966
0,348
Prining
EAN
0.655 6 0.619
(1642
+ 0.000
0.713
+ 0.15
(1.0)
# 0.015
0.617
e 0.012
0.112
0.064
0.364 + 0/044
REAP
0.767 = 0.038
0.752 : D.0S5
0,749
=: 0.005
01636
: 0.013
0,685
0.024
IL187 0.005
0,441 : 0,014
Baseline
0927
0.834
0381
0.743
0314
0.3022
0.558
Maging
M-SM-F
HC-SMolf
0,88 + 0.12
0.566-00011 00011
0.13 + 0.077
01815
+ 0.016
0,50 + 0006
0344 + 0000
0.7% + 0.007
0.68 + 0.005
0.811
0.752
+ 0/006
0.00
0.203
01256
+ 0.000
0.017
0.530
0.505
0.009
0.006
25%6
Froquecc
0921 0.00
0.34 0.007
0388 0000
0.75 0.003
0380
0.06
0.901
0.011
0,00
Qwen?: 30B A3B
Prosing
0500 +000
01864
0.004
0389
0.09
0.79
0.005
0090
0.06
0.511
00000
0354
REAP
0011
0.80
0.004
0.547 + 0.004
0.005 (1725
0.004 0397
0.994 0.003
Maring
HC
0.68
50%
0.67 0006
0.756
0.022
0,692
0.066
0.79 a 0.015
0,08
0.19
0.206
0.005
0,452
0,02
Prining
FAN
0.866 0022
0887 + 0.00
0.798
+ 0.006
01999
0.008
0.753
0011
0.906
0.003
0.5.0
0,00
RFAP
0.17 0.13
01888 + 0.015
0318 + 0.06
01703
0.004
0.780
0.006
11902
0.000
0:541 BARR
Bascline
0.244
0.475
0.556
0.463
0.409
0.123
0.296
Morging
BC-SMdd
M-MMel
0318 a 0,007
0439 = 0.028
0.23 a 0.015
0,385
2 0.000
0.466
0,550
=: 0002
a: 0100
0.300
0.44
a 0.015
: 0.007
0.325
0.144
:: 0.007
a: 0015
000033
0.110
:: 0.10
0.010
0.262
0.179
0.001
0.011
Mictrakia.BB-
25%
Phusing
Irequency
BAN
0400
0413:0002
0.358
0.365
a: 0.004
a: 000S
0341
0477
a 0009
1: 0.00
01653
0.90
0.012
0.013
0015 0318
0,48
0.09
0.099
0.111
0.014
0.005
0:109
0.252
0.000
0.000
REAP
0.45 0.066
0.385
0.025
0.548 0.000
0.470
0.013
0429
0.005
Merging
0085
0,095
3: 0.002
0.0:9
0.11
0.102
1: 0001
IK/SMolt
0 0015
0.146
0.000
0.35
0.233
02114:::00 0:015
0.013:2
Prequency
0.167 0015
0.145
00007
0,342
0.016
0.207
0.002
0,023
0.004
0,000
Pruning
BAN
0:230 0.006
189
0,005
0,315
0.000
0.525
0.015
0357
0.06
0.001
0.161
REAP
0.288 0.10
0,200
0.016
0,381
0.00
0.311
0.005
0,78
0.011
0.065
0.000
Baseline
0829
0.765
0.758
0.640
0.744
0.541
0,922
Merging
M-SM-F
HC-SMuF
0.787
0833
0.788
02882
0.766
0.755
0635
0.587
0.663
0,699
n 1.804
0.148
0.405
0.511
I. Berra---Seratt
25%
Pequencc
0335
0.78
0788
0.030
0.09
0.11
0.508
178-
Prising
0825
0262
0.944
0.48
0.066
0.906
Restruct
REAP
0829
0.787
0.788
0622
0.004
0.242
Merging
HC SMall
SMRE
0767
0.644
0.322
0.30
0.62
0.500
0.144
0.599
0,46
0.07
50%
Froquency
0823
0.755
0.751
0.995
0.676
Frusing
EAN
0.86
0.744
0.24
0.601
0,44
REAP
0.765
0,622
0.624
0.696
0.345
Ruweline
0848
0.29
0,800
0.743
0.766
0.334
0,580
Merging
M-SMel
HC-SMolf
0386
0872
0.055
0.793
0380
0,355
0.099
0.669
0.797
0:266
0.963
0.500
0,550
0,528
255
Frequsacc
0.48
0811
0.54
0.706
0.759
0.54)
0.550
GB M-- M-i
Prusing
BAN
03/2
0817
0306
0.200
0288
0.544
0.911
REAP
0.84
0.39
0.39
0.888
0,059
0412
0.388
Merging
MISM&E
0318
0,500
0.899
0.519
0.766
0.437
0.77
0
0.688
0.573
0.534
0.450
0.511
0.104
Prusing
EAN
0841
0.780
0.807
0.661
0.721
REAP
0823
0.780
0712
0577
0.679
0.382
0,115
Basellie
0951
0,80
0.93
0.791
0:631
a 0.00
06636
Owent Coder- 4508 A38B
25%
Prusing
Frequency
BAN
0354
0.39
0.05
0875
0,10
0911
0.669
0.755
0737
03877
11296 0.017
01419
0.015
0.516
0622
Rssruci 194
REAP
0.57
0.99
0917
0.722
0381
0.166
1: 0.13
0A14
Prequency
0020
0002
0000
0.003
0000
0.012 0.000
0010
5016
Prusing
BAN
0915
[84]
0354
0714
0777
0.012
0.110)
REAP
0839
0.72
0910
0/722
0882
0.015 0.415
0,619
Rasellise
0863
0.921
0013
0.735
0,828
Kini- W4A16
25%
Prining
Fraqueacy
0.530
0.09
0.88
0.463
0.905
0.857
0.908
REAP
0.57
0.921
0.188
0.759
0.940
0.988
0.079
0.175
(1146
0,112
01000
04556
50%
Prining
EAN
0.66
0811
0.750
01622
0,722
01253
04427
REAP
0.15
0.884
0,899
0.754
0,819
01429
0.624
26

---

Published as a conference paper at ICLR 2026
Table A8: C4 calibrated results for coding and MC tasks.
Coding Lieeadee
MBC MMLUU
Mode
Comgredoon
Terhniqu
Methed
Dvale
Cade
ABCC
ARRC-
Buxik)
Hedbsorag
RTV.
WhnGG.
MC
0:815
0.251
0.523
0.564
0.782
0.873
0.:15
0.737
0.482
0.822
0.724
0.721
Morging
HC-Skbil
ER-IE-A.5-2IB-
A38--F
25R
Prining
RAN
Froquescy
0:36
0.254
(1000)
0.000
0.111
0.127
0.335
0.338
0.555
0.235
0.790
0.784
0.81
0.41
0.811
03300
0,319
0.588
0.58
0.89
0.642
0.582
0763
0.58
0.91
0.00
0.83
0700
6.566
0,677
Meriia
HUSSMbll
01000
0.006
0.515
0.500
50%
Froqecticy
0100
0.000
0.00
0.00
0.655
0.717
0.350
0.006
0.535
0.758
0.69
0.563
Proniig
EAN
0187
0.86
10005
0.41
0.666
0741
0687
0.474
0.508
0.766
0.01
0,60
0.713
0.600
Marging
M-Shaat
HUSSill
D1033
0:788
01001
0140
0.511
0.00
07068
0.11
0381
0331
0361
0A32
0.733
0.66
0.615
0.376
03818
0.61
0.011
0.718
0.611
Owanh-3B-ABB
Prining
EAN
0100
0.59
03012
0388
0.07
REAP
0:763
0.253
0.586
0.35
0.71
0.64
0.40
0.736
0.422
0.85
0.93
0.702
Marging
M SMoaE
0J00
0.001
0.00
0.62
0336
0338
0.03
0715
0479
0.27
0.20
0.533
0.542
0.536
0,422
01000
01001
01:00
01960
0.88
0.50
DA32
0.905
0.564
0.58
0669
0.585
Prening
EAN
6100)
0.000
0.000
0440
0.36
0.76
0.500
0.007
0.424
0.622
0.994
0.667
REAP
0.00
0.004
01312
0.404
0/66
0.24
0644
0.52
0.940
0.613
0.67
0.59
0.206
0.50
0.740
Morging
HC-SMaBB
0.385
0.121
0.53
0.770
0.66
0.11
0.21
0.766
0336
0.31
0.44
0716
0162
0.726
0.714
Micaul 8u7B.
25R
Proning
Froqueacy
EAN
0:570
0.000
0.23
0.20
0612
0316
0.25
0.63
0.366
0.93
0.80
0675
0.751
0.29
0.70
Merging
M M.53M
HUSShbal
0.159
6.00
0.000
01073
0,000
0.540
0.500
0.993
0.764
0.64
0.807
0.395
01395
0.240
0.561
01548
0.302
0,527
0.75
0.526
0.70
0.466
0160
50R
Froquency
0.645
0.899
0.047
Table A9: b2-bench results with REAP compression across different benchmark domains on Qwen3-480B- A35B-Coder-FPP
Dataset
Compression
Method
pass"1
pass 2
pass' 3
Baseline
0.643
0.544
0.500
Retail
25% 50%
REAP REAP
0.661
0.535
0.465
0.632
0.515
0.456
Baseline
0.460
0.340
0.280
Airline
25% 50%
REAP
0,487
0.367
0.320
REAP
0.447
0.333
0.280
Baseline
0.500
0.398
0.325
Telecom
25%
REAP REAP
0.529
0.456
0.421
50%
0.471
0.339
0.263
Table A10: Additional -2-bench results (pass"1 scores) across non-reasoning and reasoning models and compression levels with REAP. Interleaved thinking is not applied for MiniMax-M2 (thinking traces from previous turns are discarded multi-tum trajectories).
Model
Dataset
Baseline
20%
25%
30%
40 %
Airline
0.393
0.407
Qwen3-Coder-30Bh A33
Retuil
0.626
0.620
Telecom
0.336
0.322
Airline (Thikiing off)
0.633
0.640
Retail (Thinking off)
0.728
0.751
GLM-4,5-irr
Telecom (Thinking off)
0.284
0.307
Teleoom (Thinking
0.272
0.269
MiniMax-M2
Telecom
0.591
0.576
0.591
0.553
27

---

Published as a conference paper at ICLR 2026
Table A11: Additional BFCLv3 results across across non-reasoning and reasoning models and compression levels with REAP. Interleaved thinking is not applied for MiniMax-Mi (thinking traces from previous turns are discarded multi-tum trajectories).
Model
Benchmark
Bascline
20%
25%
30%
40%
Swen3-Coder-30F
BFCLv3
0.632
0.622
GLM-4.5-ir
BFCLv3 (Thinking)
0.768
0.763
GLM-4.6-FP8
BFCLv3 (Thinking)
0.784
0.773
0.768
0.742
MiniMax-M2
BFCLv3
0,626
0.615
0.599
0.579
Table A12:
results at 30% REAP compression across coding, math, and
long-context evaluations.
Coding
Math
Long-Context
Compression
HumanEvall
MBPP
LiveoddBench
MATH-500
GSMSK
LongBench
FRAMES
Kimi Linear 48B Insincct
Haseline
08233
0.669
0.276 0.302
0.818
0,873
0,368
0.557
0,523
Non-Agentic Code Acc. (%)
60
75
Bsseite Pruning methods (oun)
50
MC Accuracy (%)
Froque ncy
40
65
Merging methods
HCC 5Mo
N--Met
30-
60
Modele
ERNIE 4.5 4.5211-338
55
Owin 300033300--00
Hirt BxiB
10
50
HB- IdE-nvenee
0
102
10a
101
10a
Owen dade AsSD-incnuci-frs
Log Scaled Total Parameters (in billions)
Log
Parameters billions)
Km--az
Figure A6: Coding and MC accuracy across all models vs. parameters. The benefits of REAP over other compression methods are evident at 50% compression. For large-scale SMoEs, REAP is near-lossless whereas the shortcomings of frequency- based pruning become apparent.
100
MC SMIL
Codlig
N-SMIE
Singleton clusters (96)
Accuracy 1%)
0.34
60
0.1
AMB
118
None
32
Max cluster size
16
(a) Singleton cluster proportion
(b) Restricted cluster sizes
Figure A7: (a) Average proportion singleton clusters vs. model for HC-SMoE and M-SMoE. We find that the clustering algorithms used by our baseline merging methods tend to generate high proportion of singleton clusters containing just single expert. In order to achieve the desired compression ratio, the large number of singletons conversely results in some clusters which contain many experts, in some cases N/2-+ 1 experts for a layer with N experts are grouped into single cluster. (b) Accuracy vs. maximum cluster size using M-SMoE to compress 50% of experts in Qwen3-30B. While MC accuracy remains stable up to a maximum cluster size 4, generative coding capabilities are severely diminished by restricting the clustering algorithm.
28

---

Published as a conference paper at ICLR 2026
Compress lon
50
0%
50%
Coding Accuracy (%)
25%
Dataset
Codokipac
30
10
R2AP
Fres
HEOPLE V-PRE
REA
14A
HIC MORE
REA
148
HCSPPE
HiSHLE
ERNIE-4.5-21B-A3
Qwen3-30B-A38
Figure A8: Coding accuracy vs. calibration dataset. Using domain-specitf calibration datasets substantially improves compressed model quality within the target domain. Fine-grained models such as Qwen3-30B and ERNIE suffers greater degradation, with several compression methods failing to produce any coherent output when calibrated on C4.
Mean Accuracy (%)
80
jourl lspectic
jouns (penenl))
ospec
(penenal)
40
Reauency (spesitc
Requeency (peneni)
HC SMaE (secific)
20
HC SMIE (goncra)
Ic M-St ispeci
SMot (genaral)
Coding
Math
Multiple Choice
Creative Writing
Figure A9: Mean accuracy vs. task type for models calibrated with domain specific data versus general data. The "general" calibration data consists of the combination of evol-codealpaca-vi. Writing- Prompts curated, and tull-3-sff-perronna-maal and includes three times the total number of samples as the domain- speaii- calibration datasets. Compared other compression methods, REAP best preserves accuracy across tasks when calibrated on the general dataset.
29


Reviews:
[{"review_id": "PNssUl91Fk", "reviewer": "Reviewer_KnfQ", "summary": "This paper investigates compression techniques for Sparse Mixture-of-Experts (SMoE) models, challenging recent claims that expert merging outperforms pruning. The authors provide theoretical analysis showing that merging introduces irreducible error through \"functional subspace collapse\" - essentially, when you merge experts, the router loses its ability to independently control them based on input. They propose REAP (Router-weighted Expert Activation Pruning), which selects experts to prune by considering both router gate values and activation norms. The method is evaluated across models ranging from 20B to 1T parameters, with testing on generative tasks like code generation, creative writing, and math reasoning. Results show REAP consistently outperforms merging methods on generative benchmarks, particularly at 50% compression ratios.", "questions": "1. Would merging still underperform pruning if merged experts were allowed limited post-merge fine-tuning? How much of the “functional subspace collapse” could be recovered through optimization?\n\n2. How sensitive is REAP to the assumed linearity between gate-values and activation magnitudes? Would a nonlinear scaling or normalization improve robustness across layers?\n\n3. Can REAP and merging be hybridized—e.g., pruning weak experts first and merging only within strongly correlated clusters—to achieve better compression-performance trade-offs?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors offer a clean, intuitive analysis showing that merging induces an irreducible error proportional to router policy variability. \n\n2. REAP’s design combines router gate strength with expert activation norms. It is conceptually simple yet empirically robust. It scales efficiently to trillion-parameter models and operates without fine-tuning.\n\n3. The experiments span multiple model families, from 20B to 1T parameters, and cover diverse benchmarks (code, creative writing, math, tool use). The generative evaluation is a standout—showing that REAP maintains performance where merging sharply degrades.\n\n4. The PCA visualizations in Figures 1 and A4 are interesting. They effectively demonstrate the functional collapse phenomenon and show how pruning maintains the geometric structure of the expert manifold while merging contracts everything toward the center.", "weaknesses": "1. This paper focuses on one-shot compression, which is fair for the “no retraining” setting, but most real-world deployments fine-tune after merging. It’s unclear whether the theoretical and empirical gaps between merging and pruning persist once merged experts are lightly retrained.\n\n2. The independence assumption between router policy and expert function in the irreducible error derivation (Eq. 5) is strong and may not hold in practice, especially in late layers where experts co-adapt. This limits how far the theorem generalizes.\n\n3. Section 5.1 highlights that calibration data domain critically affects compression success. This introduces a sensitivity that could complicate real deployment, but the paper doesn’t analyze how REAP behaves under mismatched calibration domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2YsFv3FSBU", "reviewer": "Reviewer_5xpD", "summary": "This paper proposes REAP (Router-weighted Expert Activation Pruning) for pruning experts in Mixture-of-Experts model. They show thorough empirical validation that pruning outperforms merging methods on generative tasks. REAP prunes MoE experts based on router gates and activation norms. The central claim seems to be that merging causes \"functional subspace collapse\" by losing the router's input-dependent control, while pruning preserves independent expert modulation.", "questions": "1) Section 5 (experimental setup):  *We compress models by pruning or merging 25% or 50% of experts in each layer* - are these experts selected at random (for non M-SMoEs)?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1) Well written introduction (and comprehensive related work) of concepts like model merging, expert pruning etc.\n2) The intuition is sound in section 3.1: merging forces a static convex combination when you need a dynamic one.\n3) I appreciate the effort to formalize the merging and pruning problem and provide mathematical intuition for why pruning outperforms merging.\n3) Strong/thorough empirical validation.", "weaknesses": "In general, I understand the authors' core ideas behind REAP, and empirical results are convincing. My major concern is that the mathematical presentation has significant issues (in my view). Perhaps there needs to major rewording where authors state them as \"mathematical intuition\" rather than formal proofs, explicitly acknowledging the simplifying assumptions while emphasizing that the approach is validated by strong empirical evidence.\n\nMajor Weaknesses\n1) I am not convinced by Section 3.3 experiments and the related claim. Authors show PCA (PC1, PC2) plots to claim it as the empirical evidence for loss of independent control of \"degrees of freedom\" when one does model merging vs pruning. \n- PCA by definition is reducing from $n$ dimensions to $k << n$ dimension ($k=2$) and what if independent control is preserved in dimensions 3-100 but lost in the first 2 PCs? What about variance? (should report variance explained by PC1+PC2)\n- Since this is towards understanding functional subspace (potentially high dimensional) a more convincing setup/experiments would require quantitative metrics like: rank of expert output matrix, any information-theoretic metric etc. \n- Current setup would be stronger as \"preliminary empirical observations\" rather than conclusive evidence.\n- The real evidence should come from performance metrics and more direct measures of expert independence.\n\n2) Missing Rigor in \"Irreducible Error\" Claim in general: I have similar issues in this part as well as point 3 below. I understand the intuitive sketch but I think stating it as formal proof is a bit of stretch. For this to be a rigorous claim one needs to proof that 1) that this is a lower bound over all possible merged representations or 2) this is the optimal form or that no other parameterization could do better.\n\n3) Theorem 1: Despite stating the independence assumption I believe there are mathematical errors in this and subsequent sections. For example: authors write the $||\\Delta_{i,j}||^2$ outside the expectation, but $\\Delta_{i,j}$ depends on $x$, so that expression does not look correct (at-least without more explanation). Similarly the three terms in Equation 5 are not rigorous and one cannot generally factor this into three separate expectations without making strong independence assumptions (which are also not stated completely) . Equation 6 also seems to have similar issues.\n\n4) *However, since our goal is to prune unimportant experts, we can reasonably assume their gate-values are small w* --> At $50\\\\%$ compression, one is pruning half the experts - Some of these will NOT have small gate values - authors use the approximation to derive the metric, then apply it globally - so I believe the explanation in this section is not entirely correctly stated with missed statements.\n\n5) There is a disconnect between Section 3 and 4. Whatever the authors use in section 3 does not reflect in Section 4 (ex: no mention/use of expert differences, policy variability? among other issues.)\n\n6) Not sure if I missed this but it's hard to see in Figure 2 the claim of near-lossless compression at any task. Ex: wrt to EAN (another method). I saw that the difference between the two is not much? Since this is an impactful claim - it might be better or more cleanly demonstrated? \n\nMinor Weaknesses\n1) Section 3.1: the equations or the explanation in general in this section can be cleared a bit (especially *\"define the router’s input-dependent mixing ratio ...\"*).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W8sOvEAp0D", "reviewer": "Reviewer_Yea5", "summary": "This paper challenges the recent trend favoring expert merging for compressing Sparse MoE models, arguing instead that expert pruning is superior for generative tasks. The authors provide theoretical analysis showing that merging causes \"functional subspace collapse\" due to loss of independent router control over experts, while pruning preserves this control. They propose REAP (Router-weighted Expert Activation Pruning), a novel pruning criterion combining router gate values and expert activation norms. Experiments across models from 20B to 1T parameters demonstrate REAP's advantages on generative benchmarks, achieving near-lossless compression at 50% on code generation tasks.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The theoretical insight in this work, that expert merging introduces irreducible error proportional to router policy variability, is insightful, novel, and well-presented.\n\n* Evaluation is extensive across many models and tasks\n\n* REAP is simple to implement and computationally efficient\n\n* The paper is overall well-structured with good motivation and intuitive explanations", "weaknesses": "* Theorem 1's proof assumes \"independence between the router policy and expert functions.\" This is barely possible in trained MoE models where routers and experts are jointly optimized. \n\n* The proposed method does not always outperform expert merging baselines, such as Qwen3-30B-A3B on WildBench at 50% ratio. \n\n* I would love to see more comparisons with other MoE-specific compression methods, such as quantization [1] [2].\n\n\n[1] Li, P., Jin, X., Tan, Z., Cheng, Y. and Chen, T., 2024. QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts. arXiv preprint arXiv:2406.08155.\n\n[2] Duanmu, H., Li, X., Yuan, Z., Zheng, S., Duan, J., Zhang, X. and Lin, D., 2025. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design. arXiv preprint arXiv:2505.05799.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)