MoBE: Mixture-of-Basis-Experts for Compressing MoE-based LLMs
Authors: Xiaodong Chen
Mingming Ha
Zhenzhong Lan
Jing Zhang
Jianguo Li
Venue: ICLR 2026 Poster (2026)
Keywords: large language models
model compression
structured pruning
Primary area: foundation or frontier models, including LLMs

Abstract: The Mixture-of-Experts (MoE) architecture has become a predominant paradigm for scaling large language models (LLMs). Despite offering strong performance and computational efficiency, large MoE-based LLMs like DeepSeek-V3-0324 and Kimi-K2-Instruct present serious challenges due to substantial memory requirements in deployment. While recent works have explored MoE compression to address this issue, existing methods often suffer from considerable accuracy drops (e.g., 7-14% relatively) even at modest compression rates. This paper introduces a novel Mixture-of-Basis-Experts (MoBE) method that achieves model compression while incurring minimal accuracy drops. Specifically, each up/gate matrix in an expert is decomposed via a rank decomposition as W = AB, where matrix A is unique to each expert. The relatively larger matrix B is further reparameterized as a linear combination of basis matrices {Bi} shared across all experts within a given MoE layer. The factorization is learned by minimizing the reconstruction error relative to the original weight matrices. Experiments demonstrate that MoBE achieves notably lower accuracy drops compared to prior works. For instance, MoBE can reduce the parameter counts of Qwen3-235BA22B-2507, DeepSeek-V3-0324 (671B) and Kimi-K2-Instruct (1T) by 24%-30% with only 1%-2% accuracy drop (about 2% drops when measured relatively).

Published as a conference paper at ICLR 2026
MOBE: MIXTURE-OF-BASIS-EEPBRI FOR COM- PRESSING MOE-BASED LLMs
Xiaodong Chen Mingming Ha', Zhenzhong Lan Zhang Jianguo Li Inclusion AI, China 2School of Information, Renmin University of China, Beijing, China WWestlake University, Hangzhou, China
[chenxiaodong, (hang- edu. 1iig wew..eeattggoou..coo
ABSTRACT
The Mixture-of-Experts (MoE) architecture has become a predominant paradigm for scaling large language models (LLMs). Despite offering strong performance and computational efficiency, large MoE-based LLMs like DeepSeek -V3-S2e--V3-0324 V3-0324 and Kimi-K2-Instruct present serious challenges due to substantial memory re- quirements deployment. While recent works have explored MoE compression to address this issue, existing methods often suffer from considerable accuracy drops (e.g., 7-14% relatively) even at modest compression rates. This paper in- troduces novel Mixture-of-Bee Basis-Experts (MoBE) method that achieves model compression while incurring minimal accuracy drops. Specifically, each up/gate matrix in an expert is decomposed via rank decomposition W AB, where matrix A is unique to each expert. The relatively larger matrix B is further re- parameterized linear combination of basis matrices (B') shared across all experts within a given MoE layer. The factorization learned by minimizing the reconstruction error relative to the original weight matrices. Experiments demonstrate that MoBE achieves notably lower accuracy drops compared to prior works. For instance, MoBE can reduce the parameter counts of Qwen3-235B. A22B-2507. DeepSeek-V3 V3-0324 (B) and Kimi K2- Instruttt (1T) by 24%-30% with only 2%-2% accuracy drop (about 2% drops when measured relatively).
INTRODUCTION
Transformer- based large language models (LLMs) Vaswani et. 2017) have revolutionized natural language processing, achieving tfute-of-the-ar performance in domains such creative writing, code generation, and mathematical reasoning. This progress has been largely guided by scaling laws (Kaplan et al. 2020 Hoffmann al. 2022). which posit that model performance improves with increases in parameter count and training data size. However, scaling dense architectures beyond certain Creeshodd-typcceee
hundreds of billions of parameters (> 100B) has proven challenging and prohibitive. Therefore, the Mixture-of-Expera
(MoE) (Jacobs et al. 1991 Jordan & Jacobs 2024) architecture has become popular since the sparse activation makes MoEs
1994 Cai
much easter and more efficient to scale to more than several hundreds of billions of parameters (Liu
al. 2024 Yang et al. 2025 Team et al., 2025a) since last year.
Despite the computational advantages of sparse activation, the large total parameter counts of MoE- based L.MSs present significant bottleneck for practical deployment. For instance, leading open- source LLMss such as SeopSeekk-3-03322 (6718 parameters) (Liu et al.) (024) exhibit performance comparable to top closed-source models. However, their scale imposes prohibitive demands on GPU memory; even high-end infrastructure, such as machine with 8x H100 GPUs, may be insufficient for efficient inference.
This work done when Xiaodong Chen research intern at Ant Group. Corresponding authors.

---

Published as a conference paper at ICLR 2026
MoRE
MsLAE
D-MMOE
$9.2
96.1
Relatve Aucuracy (%)
NL 6
32.2 21.0
DrepSeek V5622
Figure I: Relative performance comparison of different MoE compression methods. Relative ac- curacy is the ratio of the compressed model's performance to that of the original model. The ac- curacy are averaged over 15 benchmarks as shown in Table 3 Applying D²-MoE to large models like Bwen3-235B-A22B-
is computationally pro- hibitive on an 8x H100 GPU machine; therefore, it is excluded from these comparisons. MoBE is evaluated at compression rates similar or higher than baseline methods (MoLAE, D²-MoE).
DeepSeek-V3 V3-0324 and Kimi-K2-Instre
To address this challenge, much research have been proposed for MoE-based LLM compression, which could be generally categorized into two major categories. First, pruning techniques reduce
total parameter counts by either removing entire experts (Xie et al. [0244]] Lu et al.
2024 Yang et al.)
2024) or merging similar ones (hao Liu et al., 2024 Li et al. 2023b Chen et al. 2024). However, this approach often leads permanent loss of specialized knowledge and significant performance degradation (Gu et al.] 2025). Second, decomposition techniques employ matrix factorization to
compress each expert's weight matrices (Gu al., [025 [iu et al.) 2025) Li et al., (0255).. Typical works include D2-MoF (Gu et al. (2025), which extracts shared weights and applies singular value decomposition (SVD) to the residual delta weights, and MoI.AE (Liu et al. 2025), which uses SVD to represent each expert weight as a product of its unique transformation matrix and shared latent matrix. Although these SVD-based methods generally outperform expert pruning, they can still in- substantial information loss. This is evidenced by the high Mean Squared Error (MSE) between the original and reconstructed matrices, as shown in our reconstruction error analysis (Figure
In this paper, introduce the Mixture- oftturre-o-BBais--- MasssttEpperts (MoBE), novel method for efficient, parameter compression for MoE-based LLMs. MoBE factorizes weight matrix W in an expert with rank decomposition W AB, where A is unique for each expert and B is re- re-parameterized as a linear combination of set of basis matrices ( (B') that are shared across all experts within each MoE layer. This formulation achieves parameter reduction for two reasons. First, the number of basis matrices m is much smaller than the number of experts n, i.c. m < n, and basis {B'} is shared across all experts within each layer so that we could save considerable parameters for B. Second, the unique transformation matrix A is smaller than W, so that the whole MoBE factorization achieves parameter savings. The MoBE factorization is optimized by minimizing the reconstruction error between the factorized representation and the original pretrained weight matrices, typically using the gradient descent method.
We conduct comprehensive experiments diverse set of MoE-based L.MMs, including Ling-Lite- Chat Team et al.) 2025b), SeepSeek-V2-LLte--e
(Shao et al. 2024). DeepSeek- -330324 (Liu al. 2024), Qwen Diven3300--- A0B--ABB2500) Owen3-235BB-A22BB 2507 Yang al. 2025) and Kimi-K2- Instruct (Team et al. 2025a). A direct comparison of reconstruction error on 30---33--33-----
demonstrates that MoBE achieves consistently lower MSE than both MoLAE and -MMMEE, often with reductions of over 50%, across all layers (Figure 2 Similar results for more models are pre- sented in Appendix C To assess downstream task performance, evaluate the compressed models
2

---

Published as a conference paper at ICLR 2026
10
2.0 10-4
1.5
MSB Less
1.5
1.0
MSE
1.0
McE
D MrE
0.5
44
12
16
20
28
32
36
Layer
Layer
(a) Gate matrices
(b) Up matrices
Figure 2: Comparison of per-layer MSE loss for compressing the gate a and up Qwen3-30B-A3B--25 using MoBE, D2-MoE and MoLAE
(b) matrices of
wide range of benchmarks. As shown in Figure 1 MoBE exhibits superior performance advance compared to MoLAE and D2-MoE at similar or even higher compression rates.
In summary, contributions be summarized as follows:
(MoBE), parameter-efficient architecture for MoE model compression. Our analysis shows that this design yields significantly lower reconstruction error compared to existing decomposition techniques.
We introduce the
We demonstrate through extensive experiments on leading MoE models, including Qwen3- 235B-A22B- A22B-2507 DeepSeek- V3-0324 and Kimi-K2-Insse -i---K2-IInseee that MoBE can reduce total parameter counts by 24%% while retaining up to 98% of the original performance, outperforming state-of-the the-art MoE counterparts by large margin.
RELATED WORKS
Research on MoE compression can be categorized into expert gruning-based (Xie al. 2024 Gu al.
et al. 2024 Yang et al. 2024 and decomposition--assed (Li et al. 2025b; Liu et al. 2025
2025
Below we elaborate on related works under these two categories.
2.1
EXPERT PRUNING-BASED MOE COMPRESSION METHODS
Expert pruning-base based methods aim to reduce the total parameter counts of MoE-based LLMs by either directly removing entire experts or merging them. For instance, NAEE (Lu et al., 2024) removes unimportant experts by evaluating expert combinations
calibration dataset minimize model loss, while STUN (Lee et al.) 2024) groups experts based on co-activation frequency and routing weight similarity, retaining only one expert per group. Other approaches focus on merging similar experts. DEK (Zhang et al. 2024), for example, identifies and groups similar experts in the feature space and then merges them the weight space reduce redundancy. MC-SMoE (Li al. 2023b) organizes experts into distinct groups according to routing strategies and merges each group into single expert. Because these methods remove entire expert modules, they risk a permanent loss of specialized knowledge, often leading to notable accuracy degradation certain tasks.
2.2 EXPERT MATRIX DECOMPOSITION--E BBSEDD MoE COMPRESSION METHODS
In contrast to expert pruning, expert matrix decomposition based methods compress MoE-based LLMs by factorizing each expert's weight matrices into relatively smaller representations. D MoE (Gu et al. 2025) and MoLAE (Liu et al. 2025) are two state-of-the-art examples of this cat- egory. D approximates each expert matrix with shared matrix and residual delta matrix, in which the shared weight is obtained via a Fisher- weighted average of the original weights, and the residual delta weights (the difference between original and shared weights) are decomposed into low-rank matrices using SVD. MoLAE first groups a set of up/gate matrices in each MoE layer, and then approximates each matrix in group by expert-specific transformation matrix and the
3

---

Published as a conference paper at ICLR 2026
product of group-shared latent matrix. The approximation is achieved using SVD the stacked up/gate matrices within the group.
Although these methods are effective in reducing parameter counts, their reliance low-rank sumptions can be a limitation. The resulting matrix factorization does not always capture the full information of the original weights, which introduce substantial reconstruction errors and lead to notable performance drops in downstream tasks. In Appendix B we analyze the effective rank of expert weight matrices in several leading open-source MoE models. Our results show that this rank consistently exceeds the compression threshold of SVD-meaning that to achieve parameter reduction, the number of retained singular values must fall below this threshold. Eliminating this excess rank reduces the matrix's 's expressive power, likely explaining the performance degradation observed in these SVD-based compression methods.
3 METHODOLOGY
In this section, first briefly review the standard Mixture-of-Expers (MoE) architecture (Sec- tion(..1). Then, we elaborate our proposed Hixxtue-of-BBasi--xxeeee (MoBE) architecture and detail the algorithm for converting a pretrained MoE model to MoBE architecture (Section 3.2 Finally, we describe the activation functions in MoBE (Section B.3) and a specific Z-score normalization technique applied to the expert weight matrices during the conversion process (Section 3.4)
3.1 STANDARD MIXTURE-OF-EXPERE ARCHITECTURE
A standard MoE layer replaces the dense Feed-Forward Network (FFN) in the Transformer with sparsely activated structure comprising router and multiple experts. For each input token, the router dynamically selects small subset of these experts for processing, which yields significant computation cost reduction. In a typical MoE layer with n experts, the i-th expert (F*) often employs SwiGLU formulation (Shazeer 2020 process input token embedding E Rd as
E' (I) = Wi Wdown (Wi up C SiLU(W" (lat g)t
(1)
up/gate RRxd and W down dx denote the up, gate, and down projection matrices of E', is the intermediate dimension of MoE experts, and d is the hidden dimension of the model. It observed in most open-source MoE models that p dd. The router calculates a gating score for each expert and selects the top-K experts for the token:
where W
G(z) (Softaaam
(2)
where W, E Raxd denotes the weight matrix of the router G. The final output of the MoE layer a weighted sum of the outputs from the selected experts:
K y= Σ (z)) 1=1
33)
where (z) denotes the gating value (i.e., the router score) of the i-th expert E This operation is applied independently to every token in the input sequence.
3.2 MIXTURE-OF-BANA EXPPRTTS ARCHITECTURE
While large MoE models are much more efficient in inference than dense models of a similar size, they are also constrained by higher memory and storage requirements during deployment. To allevi- this, we introduce the Mixture-of-Baan -as-ooff------eaaeeee Rasis-EExperts (MoBE) architecture, illustrated in Figure 3 The MoBE formulation begins by factorizing the up/gate matrix W e Rexd of the i-th expert from the perspective of rank decomposition (Golub Van Loan 2013) as
W = AB
4

---

Published as a conference paper at ICLR 2026
Output Hiddeen States
O0
Outpui Hidden States
W dena
Expert
Expert
Expert
Al Ap
at U,,Bao
up
B
per
Bir EAC
Input Hiddln States
Input Hidden States
Figure 3:
The Mixture-of-Basi--Exxae Bx--s--exexxtss (MoBE) architecture. For clarity of explanation, we omit the activation function following the gate matrix.
where A' E R B E Brxd, and is the rank of W; with S min d) p. MoBE further considers re-parameterizing B' with a set of shared basis matrices as
B
B
with a 2 0,
Σ a 1,
where (Bá E Rexdjm J1=1 is set of basis matrices shared in one MoE layer, and {ald)m J=t are learnable, expert-specific weighted coefficients. Combining these components and introducing non-linear activation function (e.g., SiLU (Ramachandran et al. 2018)) to enhance representa- tional power, we define the final MoBE factorization as:
W = A'A B')
where W the reconstructed version of W
This factorization allows the shared basis matrices {B'} to capture common information across all experts in one layer, while the expert-specific transformation matrices A' encode specialized infor- mation. We demonstrate in the Appendix DiiiiiiDhaaa that this factorization is more powerful than the simple SVD approach. We apply this factorization to both the gate and up projection matrices. However, we do not decompose the down projection matrices, as prior research indicates they store critical knowl- Meng et al. 2022) and are less amenable to effective compression (Liu et al.
edge (Geaa et al. 2020 2025
We convert a pretrained MoEE-hased LIM into our proposed MoBE formulation by learning the factorized components. This is achieved by minimizing the reconstruction error between the original expert weight matrix Wi and the reconstruction matrix V/
min Σ
W
w
Σ W A
i B')
(5)
A
This optimization problem can solved using various algorithms, such as gradient- based optimiz- ers like Adam (Kingma & Bal 2014) the Alternating Optimization (AO) method (Wu & Lange 2008). In our practice, we find that the Adam optimizer performs sufficiently well across layers and various models, while AO suffers from unstable behavior during its alternating optimization steps. Algorithm details the full procedure for converting a standard MoE model to the MoBE formulation.
5

---

Published as a conference paper at ICLR 2026
Algorithm Converting standard MoE into MoBE
1: Require: L-layers model MMOR with experts per layer; target basis count m & n; activation
function ʃ.
2: Ensure: Parameter-efficient MoBE model MMOBE-
3: Initialize non-MoF parts in MMOBE with parameters directly from MMOE:
4: for each MoE. layer - in [MMEE do
5: for type & {gate, up} do
6: Let [W 1= be the expert matrices of the 1-th layer
7: Solve 5 with Adam optimizer
8: Obtain the factorized components {A.). [B)){()))
9: end for
10: Copy the 1-th layer down projection matrices (W down Di=1 from MMOBE
Assemble the [-th MoBE layer with (r, aie3 and (Wown).
12: end for
return MM.BE
We further analyze the parameter complexity of MoBE compared standard MoE as illustrated in Table I Note that this analysis considers only the total and activation parameter count for a single MoE layer, excluding other components such as the embedding and attention layers. The total parameter counts for one MoBE layer is ndp 2npr + 2mrdd, where the first term is for the down matrices Wdown. the second term is for the transformation matrices A in the up and gate projection, and the third term is for the basis matrices (B*). The parameter count ratio (r) from MoE to MoBE can be computed as
ndp + 2npr + 2mrd
1
2r 2mr
3ndp
3
3d
3np
Since r S p < Jd, the second term 2r
32
<
3.
1 For the last term, m < n, for an MoE with
3np 12' Therefore, 7 < + + 1/2 < When using MoBE replace MoE, the compression ratio by MoBE 1-7. From the analysis, we can draw the conclusion that the MoBE architecture could substantially compress the standard MoE models.
n 128 experts, even if we set m = 16,
we could have the last term 2mr
Notably, while MoBE reduces the total parameters quite lot, its activation parameter count requires closer examination. The matrices and the down matrices W Wdown contribute 2krd + kdp < 3kdp (since < p) to the activation parameter count, while the transformation matrices A introduce an additional 2kpr. This may lead to an increase in the number of activation parameters. To compensate for this increase, inspired by previous work (Chaudhari et al. 2025). we propose variant MoBEI, which reduces the number of activated experts during inference from l to a smaller value k. In many modern MoE models, the number of activated experts l is typically set to 8. In MoBEt, reduce this to (i.e., k' 6).
3.3 ACTIVATION FUNCTION IN MOBE
In Eq(4), we employ an activation function enhance representational power. However, not all activation functions are equally suitable. For instance, we posit that the commonly used ReLU (Glo-) not et al. 201T) activation function is suboptimal for this task. ReLU can induce excessive sparsity in the matrix B J(E wid B²), which may cause notable information loss. As the transfor- mation matrix A' pexr is smaller than B grxd it may struggle compensate for this loss with such a limited representation capacity. Therefore, a bipolar activation function (i.e., one that outputs both positive and negative values like tanh) is highly desirable.
Consequently, activation functions such as Tanh (LeCun et al. 1989). SiLU (Ramachandran et al., 2018), and GeLU (Hendrycks Gimpell (0166) are more suited for this task, while Sigmoid (Rumel- hart et al. 1986) and ReLU are expected to yield inferior results. Our ablation study in Section 4.4 provide evidence supporting this hypothesis.
'The method (Chaudhari al. 2025) reduces only activation parameters, not total parameters. Therefore, consider it complementary approach and did not include in experimental comparisons.

---

Published as a conference paper at ICLR 2026
Table Comparison of total and activation parameter count for one standard MoE and MoBE layer. MoBEt is a MoBE variant with further activation expert number reduction.
Standard MoE
MoBE
MoBE
#Total Parameters
3ndp
ndp + 2npr + 2mrd
ndp + 2npr 2mrd
#Activation Parameters
3kdp
kdp + 2kpr + 2krd
k dp 2k pr + 2k'rd
Table 2: Means and stds of the gate matrices and up matrices in various MoB-based LLMs.
Ling Lite Char
DeepSeek
V2 Lite Chat
DeepSeek V5 0324
Qwen3 30B
A3B 2507
Quent 235B A22B 2507
Kimi K2 Instruct
4.2e.6
2,5c
Sid
Up Matrices
Mesn Sid
2.3e77 2.8-2
-1.6-7
3.0p2
-5.3c.9 1,2e-2
5.3-77
233-2
18k44 1.6e-2
4.2o-8
2.5e22
3.4 Z-SCORE NORMALIZATION MOBE
To address the impact of a wide range of weight values and obtain stable results in seeking the basis, we consider normalizing all expert weight matrices in each MoE layer. We introduce Z- score normalization by subtracting the mean and dividing by the standard deviation (std) across all experts' weights:
pw = mean(W* W ..... W"),
(6)
dw std(W2 W ..... W
W uw
W
ow
This normalization introduces additional inference overhead. After factorization, ow term can be folded into the transformation matrix A', and the pw term will require extra bias operation during inference compared to the original form E)
=ww Wi = oow- W + pw (aw A')( B') + pw-
However, empirically study different off-the-shelf MoE models and find that jw is typically negligibly small as shown in Table We can therefore omit the term μa in Eq(9). That means, we only require absorbing dw into A' without introducing extra parameters and computing overhead during inference.
4 EXPERIMENTS
In this section, we evaluate the proposed MoBE approach on popular open-source MoE models and compare state-of-the-art MoE compression methods (Section 4.3). We then conduct set of ablation studies on activation functions (Section 4.4) and normalization schemes (Section 4.5)
4.1 SETUP
Models. We evaluate our method, MoBE, on a suite of popular open-source MoE-based LLMs: Ling-Lite-Chie (Team et al. 2025b), DeepSeek-V2-LLite-Chee (Shao et al. 2024), DeepSeek-V V3- 0324 (Liu al. 2024). Qwen3-30n 30--333--3---55-5 Qwen3-235BB-A22H,
Yang et al. 2025) and
Mimi-K2-Instrud (Team et al. a025a
Baseline. We compare our approach against two state-of-the-art MoE compression baselines, D MoE (Gu et al., 2025) and MoLAE (Liu et al.) 2025). Both MoBE and MoLAE are data-free compression methods, whereas M requires a calibration dataset, for which we use tulu-v3-sft- mixture (Lambert al. 2024). Due to the high computational cost of its backward pass, applying M2E to very large models like Qwen3-235B. A22B-222--00 DeepSeck-V3-nt V3-0324 and Kimi-K2- Instruct is infeasible single 8xH100 GPU machine. Therefore, comparisons involving D2-MoFE

---

Published as a conference paper at ICLR 2026
excluded from these three larger models. In addition, we compared two additional baseline methods, MoNE Zhang et al. (2025) and Sub-MoE Li et al. (2025a), on Dwen3-30B-A3B-2555 and Qwen3-235B 211---222-----2---- 2207,2507, with the results presented in the Appendix
Hlpeer-aaaamtters Hyper-parameteer are configured per case (models or methods). We provide a more detailed explanation in the Appendix [ basis matrices m and the rank
regarding the impact of the values of the number of
For Ling-Lite- and DeepSeek-V2-Lite-Ge MoBE uses m = basis matrices and MoLAE uses 8 latent matrices. To compensate extra computing cost introduced by extra activation parameters in MoBE (Section 3.2). we reduce vetivated experts from = 6 to k' = 4 in MoBEt.
. For Qwen3-30B- Bive3-30B-A3B-11 A0B-533-5 and Bree3-2335-A22BB 2-007, both MoBE and MoLAE use 32 basis/latent matrices. MoBE reduces activated experts from k = 8 to = 6 in MoBE
For DeepSeek- -e3-S2,-- both MoBE and MoL.A use basis/latent matrices, with MoBE reducing / from 8 to 6 in MoBEt.
For Limi-K2-Instruee
both MoBE and MoLAE use 128 basis/latent matrices, and MoBE similarly reduces & from to 6. Due to optimization challenges with 384 experts per layer, we split them into two groups, each trained with 64 basis matrices.
For D²-MoE, the rank of delta weights is set to 700 for Ling- Lite-Chat and DeepSeck-Vi V2- Lite-Chat, and 420 for Dwee3-30B-A33--25
For simplicity, we set the rank r p in all our studies. It gets more compression ratio when setting r < p while may increasing the accuracy drops.
Implementation Details. All experiments are conduct on H100 or H20 GPUs using the Adam optimizer (Loshchilov & Hutter 2017) with a 0.07 learning rate. We set the batch size equal to the number of experts n and train for a maximum of 50,000 epochs, employing early stopping with a patience of 2,000 epochs based on the training loss.
4.2 EVALUATION BENCHMARK
We perform comprehensive evaluation across wide spectrum of benchmark. The evaluation
suite covers four primary domains: (1) General Knowledge: BBH (Srivastava et al. 2022).
MMLU (Hendrycks et al. 2020), CEval (Huang et al. 2023). and CMMLU (Li et al. 2023a);
(2) General Reasoning: ARC-Challenge (Clark et al. 2018 IFEval (Zhou et al.] [202]), and
GPQA (Rein et al.] 2023); (3) Mathematics: Math (Hendrycks et (021), GSM8K (Cobbe al.,
2021). AIME24, and AIME25; and (4) Coding: MBPP (Austin et al. 2021), HumanEval CCen al. 2021). LCB (LiveCodeBench-v5) (Jain et al. 2024), and MultiPL-E (Cassano et al. 2022).
For AIME24 and AIME25, we run 16 inference trials per question and report average accuracy; for IFEval, the final score the mean of strict accuracies at both the prompt and instruction levels.
4.3
MAIN RESULTS
All the compared results of the origin model (MoE) and different compression methods (MoBE, MoBE!, D2-MoE, and MoLAE) are shown in Tables 3 It shows that our proposed MoBE method generally outperforms all the compared compression methods across various benchmarks. For in- stance. for the Long-Lite-Chal and DeepSeek- Biiicek-V---iie V2-Lite- Chat models, MoBE improves performance by 2-3% accuracy over the baseline. The performance gains are even more notable for Qwen3-30P5 A3B-2507, 222---2223---22--222
DeepSeek-V3-0133 V3-0324 and Kimi-K2-Instrret, reaching 4-8% accuracy advantages over compared compression methods.
We note that converting MoE models into MoBE architecture results in an average performance degradation of 1.4% accuracy compared to the original MoE models. For comparison, MoBEI that only reduces the number of activated experts from & to k', leads to smaller degradation of 0.5% accuracy. It suggests that it is more challenging to compress the total parameters than activation parameters for an MoE model. As the sparsity ratio (#activated recent MoE models becomes larger and larger so that the total parameter counts reach trillion-level (21T), ät is more useful and practical to compression the total parameters.
wttperrssttttatlaaaaeeteeeee of
8

---

Published as a conference paper at ICLR 2026
Table Performance comparison different compression methods on various MoE-based LLMs, where "I" indicates that this model activates fewer experts than the original model to compensate for the increase activation parameters. The column Ratio refers to the proportion of compressed parameters to the total parameters in the LLMs.
ABC [RPul
(204
Rast
VMMEII
(Thel
[MMI 11
W6n
EEVeS:
AMMEN
ATMES
Herastie
123
52.4
15.3
11.3
21.3
35.0
66.
15.1
303
12.8
2113
14.6
30.1
125
85.1
66-3
$3.9
714
8239
03.3
1910
76.
385
663
651
13.6
11.
52.0
02.4
31.5
17-36
133
823
81.8
3011
31.5
23.2
302
25.3
367
Che
16CAE
115
63.3
43.3
360
46.0
285
M1607
15%
15.5
30.5
93.9
538
553
25.0
(8.0
2.5
$1.5
586
74.0
3.0
36.1
P7
553
460
655
9611
53.0
71.3
540
44.0
25.0
56.4
76.
42.1
3.76
66.9
60.5
91.8
61.
M-d17
375
65.
151
$1.0
33.3
66.
555
K14
92.
16.1
510
45.6
85.3
90.2
61.3
38,01
26.7
DeyySer Vi 1131
Mr.AB
97.3
833
585
52.9
823
54.5
15.5
16.5
24.5
57.2
56.5
61.0
14.6
75.1
08
415
39-
715
41.5
Qweed 2008-222222220 2218-2011
Mr
345
96.3
83.9
15.5
93.4
986
933
962
4055
96.3
648
54.
1912
953
510
71.
43.7
913
MuBIC
340
55.7
915
93.9
16.9
00.1
03.2
NGB
09
05.5
93.
17.4
36.3
311
60-
90.
96.3
06.5
61.
53.
16.1
06.1
00.3
12.1
08.1
My00
319
913
960
68.8
81.1
M:01:
399
91.3
11.7
746
3601
913
96.5
99.
951
16.6
63.
41.
904
961
650
41.
$1.5
1e-5
MSE Loss
MSE Loss
Zone
siu
sigmold
2
Z Z-Sooce Nomalizatonn
gela
selu
wro Z Score Normlizztion
0
10
20
30
40
20
30
40
Layer
Layer
Figure 4: Comparison of per-layer MSE loss for compressing the gate matrices of Qwen3-30B- A3B when using different activation functions.
Figure 5: Comparison of per-layer MSE loss for compressing the gate matrices of Qwen3-30B- A3B with/without Z-score normalization.
4.4 ABLATION STUDY ON ACTIVATION FUNCTIONS
In Eq we apply non- linear activation function to enhance representational capacity. We conduct experiments on the Owen3-30B-A33 model's gate matrices select the optimal activation function. As shown in Figure 4 Sigmoid demonstrates inferior performance to the case without activation in terms of the reconstruction MSE, while ReLU has an order-of-man -fagiitu- higher MSE loss. This result is consistent with our analysis in Section 3.5 GELU, SiL.U, and Tanh achieve similar results and outperform the case without activation, while we finally choose SiLU and Tanh as our activation function they offer favorable trade-off between performance and computational efficiency.
4.5 ABLATION STUDY ON Z-SCORE NORMALIZATION
To evaluate the impact of the Z-score normalization introduced in Section 3.4 we conduct an abla- tion study using the -wwen330B-AABB gate matrices. All experiments use identical hyperparameter and optimization settings, varying only the application normalization. Figure 5 shows a notable reduction in MSE loss when Z-score normalization is applied. We hypothesize that the normaliza- tion can rescale the weight values from wide and wild ranges to a normal distribution with a mean of 0 and std of that the optimization becomes more stable and effective.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
In this paper, propose the Mixture-of-Bie Baarre-ff-Bas--EEpeeee (MoBE), paramette--efficiet architecture that addresses memory challenges in deploying large-scale MoE-based LLMs. MoBE effectively combines shared basis matrices with expert-specific transformation matrices via rank decomposition to overcome limitations of prior work. Extensive experiments demonstrate that MoBE outperforms existing counterpart methods like MoLAE and D²-MoE with large margin in preserving higher performance and better model compression rate. MoBE can compress leading models such as Qwen3-235B-A22 1-5SB-A22B-250- DeepScek-V3-De V3-0324 and Kimi-K2 -2-n-Ke-Instruet by up to 24%-30% while taining up to 98% of their original performance across diverse benchmarks. Such practical and effective method may help enable large MoE models for more scalable and efficient applications.
STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
In preparing this manuscript, Large Language Models were used exclusively for refining language, grammar, and clarity. The core ideas and content remain entirely the author(s)' own, who bear full responsibility for all information presented herein.
LIMITATIONS
While our method performs well in compressing MoE models, still causes a slight drop in accuracy compared to the original model. To fix this gap, potential direction is employ full network knowledge distillation (KD) between the original and our compressed models. This requires mod- ifying existing training frameworks to support KD training for large LLMs. Another limitation is that MoBE requires multiple times calling of current optimized kernel fused- Tused-MoE to mimic the fac- torization, which is relatively inefficient. Hence, it requires implementing a specifie mega-kernel for the whole factorization to unleash the power of the MoBE architecture. Future work will address these two limitations.
ACKNOWLEDGMENTS
This work is supported by the National Key Research & Development Plan (2023YFF0725000) and the National Natural Science Foundation of China (92570121, 62322214, U23A20299, U24B20144) 44). We also acknowledge the support of the Ant Group Research Intern Program.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Do- han, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, 100//2208007732. 2021. URL lhttps://ast
semanticscholat wwwp:oorps..:
Weilin Cai, Juyong Jiang, Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint arXiv: 2407:02407.0624 2024.
Federico Cassano, John Gouwar, Daniel Nguyen, Sy Duy Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. Multipl-e: A scalable and extensible approach to bench- marking neural code generation. 2022. URL https: htt.://app...mmmanccccccc..... org/
CozpusID:25485411
Marmik Chaudhari, Idhant Gulati, Nishkal Hundia, Pranav Karra, and Shivam Raval. Moe lens expert is all you need. In Sparsity in LLMs (SLLM): Deep Dive into Mixture Experts, Quantization, Hardware, and Inference, 2025. URL https:/ http:////pereevieeeeom net forum?
Td==S4WXncws
10

---

Published as a conference paper at ICLR 2026
I-Chum Chen, Hsu-Shen Liu, Wei-Fang Sun, Chen-Hat Chao, Yen-Chare Hsu, and Chun- Yi Lee. Retraining-free merging of sparse moe via hierarchical clustering. 2024. URL /
semanticschol wwtp/ooooss:
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 0001:22007.033744 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457, 2018. URL https: sem....sseettttchool...... .rrg (CorpusDD:3922BL))
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXi::2100.1466. 2021.
Mor Geva, Schuster, Jonathan Berant, and Omer Levy. Transformer foed-forwadd layers are key- value memories. ArXiv, abs/2012.14913, 2020. URL
http:/CoopuSI::229933333
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In International Conference Artificial Intelligence and Statistics, 2011. URL https: //api.
semant cscho wws.ccohla...oo/ooorussssl..seee 2239473
Gene H Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
Hao Gu, Wei Li, Lujun Li, Qi Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yi-Ting Guo. Delta decompression for moe-based iims compression. ArXiv, foss/2502.129999 2025. URL https:
//api. wtt.eeeeaaiccccoolllaooooo/roooe
En hao Liu, Junyi Zhu, Zinan Lin, Xuefei Ning, Matthew B. Blaschko, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Efficient expert pruning for sparse mixture-of-expprrs language models: Enhancing performance and reducing inference costs. ArXiv, abs/2407.00945,, 2024. URL httpt: www.appp...aaaiim.mmm..... www.nticschooooo.ee..e....ccocso
Dan Hendrycks and Kevin Gimpel. Gaussian emror linear units (gelus). arXiv: Learning, 2016. URL ptttps://api.semantiiee http:///g....aaaaaaaa..oa...oooa 125617077
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. Measuring massive multitask language understanding. ArXiv, Abs/2009.03300
2020.
URL https::/api... www.nticscoolar..rrogo.....
221516475 6475
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Xi- aodong Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. ArXiv, abs/2103.03887, 2021. URL https: seats:://ap..emmmttt semant icscholar: oooggoorrrrrroo/oooooooooo
CozpusID:23213465)
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego De Las Casas, Lisa Anne Hendricks, Johannes Welbi, and Aidan Clark. Train- ing compute-optimal large language models. 2022.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. arXiv preprint arXiv:2305.088322, 2023.
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. ArXiv, abs/2403.07974, 2024. :
//api. semant wwwchhaar........ .r. wwwecchollrrrrr/ooooeeeeeeeeeeee
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
Jared Kaplan, Sam Mccandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. TŞ" ulu 3: Pushing frontiers open language model post-training. arXiv preprint arXiv:2411.15124 2024.
Yann LeCun, Bernhard E. Boser, John S. Denker, Donnie Henderson, Richard E. Howard, Wayne E. Hubbard, and Lawrence D. Jackel. Backpropagation applied to handwritten zip code recogni- tion. Neural Computation, 1:541-551, 1989. URL batpss:/appie stt.apppssmttticcccccaaccccaeaee
arg/CooruuTD:44111785
Jaeseong Lee, Seung won Hwang, Aurick Qiao, Daniel Campos, Zhewei Yao, and Yuxiong He. pruming for scalable moe pruning. ArXiv, abs/2409 01211,, Stun: URLthttss://apt http://ppp...aa.....mm..mmm semant weeamantccccoole...o cschole wwwchoal....o/oooooo...... oogp/orpssr:: 2D:2555001 2024.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timo- thy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.092 092112 2023a.
Lujun Li. Zhu Qiyuan, Jiacheng Wang, Wei Li, Hao Gu, Sirui Han, and Yike Guo. Sub-moe: Efficient mixture-of-expert 1lms compression via subspace expert merging. arXiv preprint arXiv:2506.23266, 2025a.
Pingzhi Li, Zhenyu (Allen) Zhang, Prateek Yadav, Yi-Lin Sung, Yu Cheng, Mohit Bansal, and Tianlong Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy. ArXiv, abs/2310.01134, 2023b. URL https: /tpp. sew.apiisssmaatccchhlhla..a o
CorpusID:263605B8
Wei Li, Lujun Li, You-Liang Huang, Mark G. Lee, Shengjie Sun, Wei Xue, and Yike Guo. Struc- tured mixture-of-experts LLMs compression via singular value decomposition, 2025b. URL
https://openrew//
net wewt:/ooomidddho72555555
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024.
Zehua Liu, Han Wu, Ruifeng She, Xiaojin Fu, Xiongwei Han, Tao Zhong, and Mingxuan Yuan. Molae: Mixture latent experts for parameter-efficient language models. 2025. URL https:
//pii iorii.semanicscchnn hotpgi/eeattttteeooooa..rrrrrrpp
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017. URL aattps://apil http:/pppp...aa.t..cccccoaaa....
Corpus ID: :D:::D::322 53592270
Xudong Lu, Qi Liu, Yuhui Xu, Aojun Zhou, Siyuan Huang, Bo Zhang, Junchi Yan, and Hongsheng Li. Not all experts equal: Efficient expert pruning and skipping for mixture-of-expert. of-experts large language models. In Annual Meeting of the Association for Computational Linguistics, 2024.
URI.httss: /tt:://ap. /appi. semanticschon hww.chhh.....oogoccopppsao////11
12

---

Published as a conference paper at ICLR 2026
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations gpt. In Neural Information Processing Systems, 2022. URL htttp:://aaa/
semanticscho atgg/oorrusI::2255335533
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. Searching for activation functions. ArXiv,
abs/1710.05941, 2018. URL https: //api Senaatticccoollttooo//oooooooo
10919244
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R. Bowman. Gpqa: A graduate-level google-proof q&a
benchmark. ArXiv, abs/2311.12022 2023. URL BRLLhttps://aet .tt..//ca...maattcccccccooo.....
CorpusID:26529985
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representattiissnnn by back- propagating errors. Nature, 323:533-536, 1986. URL httpss:/api..eetee stp.a///cco.seeettticcccooooooo
Ctc/CorpusID:20500110
Zhihong Shao, Damai Dai, Daya Guo, Bo Liu (Benjamin Liu), Zihan Wang. and Hua- jian Xin. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. ArXiv, abs/2405. 04444, 2024. URL httpps://api.semmmicccsstt/
CorpusID:2696138B)
Noam M. Shazeer. Glu variants improve transformer. ArXiv, abs/2202..00000022.. 2020. URL https: wtt.:ssemaiicccccccc wwwpschoolar.oo/oooouuuuuu. : :11
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint vrXiv:2206.046.1 2022.
Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, Zhuofu Chen, et al. Kimi k2: Open agentic intelligence, 2025a. URL http://///aaxxv.oo//abassse////s
Ling Team, Binwei Zeng, Chao Huang, Chao Zhang, Changxin Tian, Cong Chen, Dingnan Jin, Feng Yu. Feng Zhu, Feng Yuan, al. Every flop counts: Scaling a 300b mixture-of-experts ling I1m without premium gpus. arXiv preprint arXiv:2503.051399 2025b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Tong Tong Wu and Kenneth Lange. Coordinate descent algorithms for lasso penalized regression. 2008
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pruner: Pruning mixture-of-expee of-eeertss large language model using the hints from its router. ArXiv, abs/2410.12013, 2024. URL htt..httss//// /taps:://ap5 /api semant cccholl oorgg /rg/
CorppsSD:2733375611
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.093388, 2025.
Cheng Yang, Yang Sui, Jinqi Xiao. Lingyi Huang, Yu Gong, Yuanlin Duan, Wenqi Jia, Miao Yin, Yu Cheng, and Bo Yuan. Moe-iP: Compressing mixture of experts models through ixter-expert pruning and intra-expert low-rank decomposition. ArXiv, abs/2411.0 2024. URL https:
//api. semant wwa.mmalccccharr.cooo httpaaatccccccc..oooooo/o/rroooo 233
Geng Zhang, Yuxuan Han, Yuxuan Lou, Wangbo Zhao, Yiqi Zhang, and Yang You. Mone: Replacing redundant experts with lightweight novices for structured pruning of moe. ArXiv, 0bss2507.003390 2025. URL https::// //api. semanticscholat
2801 48222
13

---

Published as a conference paper at ICLR 2026
Zeliang Zhang, Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruming in sparse of ArXiv, 0atp22407.099900 2024.
BtRLhtts::// /app.pp.......... semant cttp/oorpus1::2211122...
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instmuction-following evaluation for large language models. ArXiv, abs/231 107911, 2023. /ttpttps:://////tettt //atps://ap... wwt.::p..a.aaatatnc..eaaeae.cmee wtt.://cco....a..oooooooooaaaaaa
14

---

Published as a conference paper at ICLR 2026
A
ABSOLUTE PERFORMANCE COMPARISON OF MOE COMPRESSION METHODS
224
ma
Acauro
SA.E
LingChhCCall
Figure 6: Absolute performance comparison of different MoE compression methods.
We present the absolute performance comparison of MoE compression methods Figure
B ANALYSIS OF THE EFFECTIVE RANK OF EXPERT WEIGHT MATRICES
We evaluate the effective rank of expert weight matrices Qwen3-235B-A222B-5507 DeepSeek- V3-0324, and Kimi-K2-Iin Ma--KK2IInnttt The effective rank is defined as:
j==
Fe min
k N
> 0.95
E
where a; is the i-th largest singular value (sorted in descending order) and is the matrix rank. The expert weight matrices in Qwvnn-2231B-A22B--200
have dimensions 4096x1536, while those 7168x2048. Figures I0 illustrate the per-layer average effective rank Te and its range for each model. Taking the expert weight matrices of Kimi- K2-Instruct as an example, rank decomposition could realize parameter compression only if the intermediate rank satisfies
-33--344 and Kimi-K2-Instruu
7168 2048
re
1593.
7168 + 2048
However, according to Figure 10 the average effective rank Te is larger than 1593 most lay- This discrepancy implies that the pure tet--ddecoooostton-bbeeeeee method can't produce model compression without performance loss. An interesting finding can be drawn from the analysis: Owen3-2355BAA22B-2555 shows much broader effective rank range than the other two, which may indicate that its experts are far from being well-balanced during the training phase.
C ADDITIONAL MSE COMPARISONS
We present a comparison of reconstruction errors on Ling-Lite- Chht, DeepSeek V2-Lite, Qwen3- 235B-A22B-2507, DeepSeek-V3-0322 and Kimi-K2-Inntrruct in the Figure 15
D AN EXPRESSIVE POWER ANALYSIS MOBE
In this section, we demonstrate that the MoBE possesses greater expressive capacity than conven- tional low-rank factorization derived from Singular Value Decomposition (SVD). Assume there are experts in one layer, each with up/gate matrix of dimension x d. To construet the based baseline, we first partition these n experts into m groups, with each group containing 2 matrices (assuming is multiple of m). For the j-th group, the up/gate matrices stacked row-wise
15

---

Published as a conference paper at ICLR 2026
form consolidated matrix Wa this stacked matrix:
stack E R B2
We perform Singular Value Decomposition (SVD)
U E V =SVD(W))
stack
(10)
where ( E x E E top r singular values, obtaining the components U E R mh xr Σ E R. and V E Rxxd We then incorporate D into U and V by setting U = U (2))// and V = (Ε>)1/3 For the i-th up/gate matrix, assuming it belongs to the j-th group, its low-rank approximation is given by:
and V E IRaxd The decomposition is truncated to retain the
W A
(11)
where A' corresponds Ü**(K- 1)p kp with A -i- indicating that the i-th up/gate matrix is the k-th matrix in the j-th group, and is Vi. Comparing this factorization with 4q observe that the SVD-based decomposition is special case of MoBE. Specifically, it is equivalent to MoBE where the a are restricted to one-hot vectors (assigning each expert to a single group) and the activation function is an identity mapping. Therefore, MoBE exhibits significantly greater expressive power than this SVD-based approach.
E ANALYSIS OF BASIS MATRIX COUNT AND RANK
The number of parameters for the basis matrices is mrd, where m is the number of basis matri- ces, r is the rank of the basis matrices, and is the hidden dimension of the model. We conduct experiments on construction errors presented in Figure 7
with different values of m and r; comparison of the re-
As shown in the Figure 7 for a fixed parameter budget, the reconstruction error for the configuration m 32, r 768 is significantly lower than that for m 64, r 384. This indicates that the rank r of the basis matrices is a more influential factor than the number of basis matrices m. Therefore, our main experiments, we set r to be equal to the hidden dimension of MoE to maximize representational capacity.
Additionally, we conduct experiments on Qwen3-30B-AAB-22555 with r fixed at 768 while varying investigate its impact downstream task performance. The results presented in Table 4 These results show that when m = 16, the performance the compressed model degrades sub- stantially compared when better than at m 32, this improvement comes at the cost of considerably lower compression rate. Therefore, in our main experiments, select configurations corresponding 25%-30% compression rate to achieve a favorable balance between compression efficiency and model perfor- mance.
32. Conversely, while performance m 64 is marginally
Table 4: Performance comparison different configuration settings on Qwen3-308 3----30-------- The column Ratio refers the proportion of compressed parameters to the total parameters in the LLMs.
LLM
Raio
ABCCC
12gA
B811
SINLU
(lbdl
CMNLL
Mut
GSMMic
A1M8131
AIMU2
MBPP
Muttpiit:
LED
Qnea1-328-A11 2397
14%
$5.0
$6.9
$2.1
83.5
$3.6
88.1
92.5
95.2
55.0
452
$7.4
91.8
35.6
13.8
5%
$1.5
K5.5
13.8
866
87.0
96.8
$7.1
18.
850
457
344
770
F ADDITIONAL BASELINE COMPARISONS
We compared two additional baseline methods on Qwen3-30B-A3B. -33-25557 and Owwn3-235B--222 2507: MoNE and Sub-MoE. MoNE prunes less important experts based on their routing weights and output variance, replacing them with their average outputs. Sub-MoE groups experts by output cosine similarity and merges each group into one. This merge uses SVD on the expert matrices, weighting their components by activation frequency to create a new, single expert. For both of these methods, we use tula-v3-sft-mixture as the calibration dataset. As shown in the Table 5 MoBE consistently outperforms these baseline methods.
16

---

Published as a conference paper at ICLR 2026
1e
1.5
me12 pe208
MSS Loss
m-:4 pr334
1.0
MSE
0.5
0.5
12
20
Layer
Layer
(a) Gate matrices
(b) Up matrices
Figure 7: Comparison of pre-layer MSE for compressing the gate (a) and up ( matrices of Qwen3-
30B-A3B- 2507 using different configuration settings.
Table 5: Performance comparison of different compression methods on various MoE-based LLMs, where "f" indicates that this model activates fewer experts than the original model to compensate for the increase activation parameters. The column Ratio refers to the proportion of compressed parameters to the total parameters in the LLMs.
ARC C BTTcal
dRgA
BS:1
MX
(Tiil
ESSSL
AIVE34
ATNE35
MTPP
Flruufral
'Saagg E.
1,71
17-30e
05.1
01.5
43-2
01.3
B13
21.2
68.5
06.1
$2.0
20.1
18.3
$4.3
44.0
16.0
96.1
823
533
Qwea1-00 10033-0--1110 AID 2007
Saks3ie
10.3
85.1
68.3
22.8
823
1012
83.3
12.11
$5.0
31.3
63.2
$2-
61-2
13:
16.1
15.1
31.0
968
11.
853
92.3
76.
MiLAS
65.6
15.5
66.2
855
13.5
13.1
11.3
10.0
13.5
75.2
Oweed -228-2207
18.7
12.3
03.
55.3
Modt:
06.3
39.0
99.8
98.4
966
850
96.3
96.1
19.3
13.9
43.7
91.1
Mean Bonk
Aange (Mtn Maat
Mean
Rank fonge 24 Mod
Mean
1Vin
12.00
1200
1202
Expert Rank
Expert Expert Miarixx Hank
800
800
800
600
600
6.0
Expert
200
200
200
86
Layer
Layer
Layer
(a) Gate matrices
(b) Up matrices
(c) Down matrices
Figure 8: Average effective rank and effective rank range of the gate, up, and
down
matrices at each layer in Qwen3-235B- --2.----22--257777
Expert Matris Rank
ano
000
ana
Expert
400
400
400
30
20
30
40
50
60
35
20
30
40
50
60
30
20
30
40
35
60
Layer
Layer
Layer
(a) Gate matrices
(b) Up matrices
Down matrices
Figure 9: Average effective rank and effective rank range of the matrices each layer DeepSeek.
(a) gate,
up. and
down
PeepSeek-33-032. V3-0324.
17

---

Published as a conference paper at ICLR 2026
fange Maav
Ranc Parge Mi
Sear
tange
1600
1400
Dipertt Metrit Panki
2200
2200
800
800
15
20
30
46
10
60
16
20
16
4b
15
46
Layer
Layer
Layer
(a) Gate matrices
(b) Up matrices
(c) Down matrices
Figure 10: Average effective rank and effective rank range of the Qa gate,
up. and
down
matrices at each layer in Kimi-K2 mnni-K2-Instrutt.
le-4
le-4
MARE
MARE:
MSE Loss
McLAI:
MSE Loss
MeLAE
16
20
24
0
4
12
16
20
24
Layer
(a) Gate matrices
(b) Up matrices
Figure 11: Comparison of pre-layer MSE for compressing the gate and up matrices of Ling- Lite-Chat using MoBE, D2-MoE and MoLAE.
MSE Lo8K
D1
Layer
Layer
(a) Gate matrices
(b) Up matrices
Figure 12: Comparison of pre-layer MSE for compressing the gate and
b matrices of
DeepSeek- DeepSeek-V2-LLite-- V2-LLit--hht using MoBE, D2-MoE and MoL.AE.
Ic 5
1.0
Molii
MoSE MrlAIl
MrlAA
MSE Loss
MSE Loss
0 10 15
30
45 50 60 65 70 85 Layer
10 2025 3035 40 50.55 60 70 80 Layer
(a) Gate matrices
(b) Up matrices
Figure 13: Comparison of pre-layer MSE for compressing the gate and up (b) matrices of Qwen3- 235B-A22B 2507 using MoBE and MoLAE.
18

---

Published as a conference paper at ICLR 2026
S
10.5
7.5
7.5
MoLAA
MolAf!
MSE Loss
3.0
5.0
2.5
Layer
Layer
(a) Gate matrices
(b) Up matrices
Figure 14: Comparison of pre-layer MSE for compressing the gate and up (b) matrices of
DeepSeek- V3-0324 using MoBE and MoLAE.
---Seck-V3-0324
MeLAll
MSE Loss
MSE
Layer
Layer
(a) Gate matrices
(b) Up matrices
Figure 15: Comparison of pre-layer MSE for compressing the gate (a) and
b) matrices of Kimi-
K2- Instruct using MoBE and MoLAE.
19


Reviews:
[{"review_id": "BedLeG8N08", "reviewer": "Reviewer_Bric", "summary": "The paper proposes MoBE, Mixture-of-Basis-Experts, to compress MoE-based LLMs. Each expert’s up/gate matrix is factorized by the rank decomposition W=AB; B is shared as a linear combination of a set of shared basis matrices, while A is specific for each expert. By minimizing the reconstruction error plus z-score normalization and suitable activations (e.g., SiLU/Tanh), MoBE reduces total parameters by 24%-30% while keeping 98% performance on many benchmarks, outperforming MoLAE and D^2-MoE at similar ratios.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem is important for deploying trillion-level MoE. The idea is simple but quite effective. The paper proposes to decompose the up/gate matrix into shared basis matrices B across experts to capture the common information across experts and keep matrix A per expert to encode specific information, and to add non-linearity inside the matrix factorization to enhance representational power.\n- The paper is well-written. The equations and algorithm steps are easy to follow.\n- The paper conducts extensive experiment to show that this method is applicable to very large MoE including Qwen3- 235B-A22B-2507, DeepSeek-V3-0324 and Kimi-K2-Instruct where many baselines are infeasible. It is very pratical for memory-bound inference.", "weaknesses": "- The paper comes with limited theory of formal approximation guarantees. Most are from empirical studies.\n- The choice of hyper-parameters lacks guidance, including the choice of basis count m and the rank r. The compression rate and the accuracy frontiers are not fully mapped.\n- No study of light-weight finetuning or knowledge distillation to close the last 1%-2% gap.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UtieJMrlXr", "reviewer": "Reviewer_1CGV", "summary": "The paper proposes an elegant and theoretically sound framework for MoE compression that leverages shared basis learning. The core concept of representing experts as combinations of a shared basis is well-motivated by the observation of functional redundancy (Fig. 1; Sec. 3.2; p.4). The experimental results appear strong, with claims of maintaining high relative accuracy (e.g., 96.8% for Qwen3-235B at 24% compression) as shown in Figure 1. However, the paper suffers from significant presentation issues, most notably the absence of critical referenced tables (e.g., Table 3), which prevents a full verification of the quantitative claims. The analysis of effective rank is referenced to a non-existent figure (Fig. 9), further hindering a complete assessment.", "questions": "- **Include all referenced tables and figures**\n  - The most critical suggestion is to **include the missing Table 3, Table 8, and Figure 9** in the manuscript. Without these, the paper is incomplete and its claims cannot be verified.\n  - Ensure that all references in the text point to the correct tables, figures, and sections. The manuscript should be carefully proofread to correct any such errors.\n- **Provide comprehensive details on the training process**\n  - Add a dedicated subsection or appendix detailing the hyperparameters used for both stages of the training process, including learning rates, batch sizes, schedulers, and convergence criteria.\n  - Include a sensitivity analysis on the number of basis experts. This could be a plot showing the trade-off between the number of basis experts, the compression ratio, and the model's performance on a validation set.\n- **Clarify the experimental setup and results**\n  - Once Table 3 is included, ensure it is well-described and that all columns and rows are clearly labeled. The main text should walk the reader through the key results in the table.\n  - Provide a more detailed analysis of the results, going beyond relative accuracy to discuss performance on specific tasks or benchmarks.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Novel and theoretically sound compression framework**\n  - The MoBE formulation, where each expert is a weighted sum of basis experts, provides a principled way to capture and exploit inter-expert redundancy ($\\text{Expert}\\_i = \\sum\\_j \\alpha\\_{ij} \\cdot \\text{Basis}\\_j$, Eq. 1; Sec. 3.2; p.4). This is a clear and impactful contribution.\n  - The framework naturally separates shared knowledge (the basis experts) from specialized knowledge (the combination coefficients), offering a more structured approach to compression than unstructured pruning.\n  - The two-stage training process, which first initializes the basis and then jointly optimizes all components, is a logical and practical approach to tackling the complex optimization problem (Sec. 3.3; p.5).\n- **Strong conceptual motivation and clear illustrations**\n  - The paper provides a clear motivation for the MoBE approach, contrasting it with existing methods and highlighting the limitations of pruning and independent decomposition (Sec. 2; p.2-3).\n  - The architectural diagram (Fig. 3; Sec. 3.2; p.4) is clear and effectively communicates the core components of the MoBE block, including the basis experts and the learned combination coefficients.\n  - The conceptual illustration of relative performance (Fig. 1; p.1) provides a high-level summary of the method's claimed effectiveness, showing favorable comparisons against other methods like MoLAE and MoE-SVD.", "weaknesses": "- **Missing key experimental results and references**\n  - The paper repeatedly references **Table 3** for key quantitative results that are central to its claims of outperforming baselines. However, **Table 3 does not exist** in the manuscript or its appendices. This is a critical omission that makes it impossible to verify the core experimental findings.\n  - The review references **Table 8** and **Figure 9** in the appendices for further analysis, but these elements are also **not found** in the provided document. This suggests either a flawed manuscript or a flawed review process.\n  - Without these key tables and figures, the paper's claims of superior performance are unsubstantiated and rely solely on high-level plots (e.g., Fig. 1).\n- **Ambiguous or incorrect references**\n  - The analysis of effective rank is attributed to \"Appendix C; Fig. 9; p.14\". As noted, Figure 9 is missing. The paper does contain an analysis of MSE loss in **Figure 2** (p.3), which might be what was intended, but this discrepancy creates confusion.\n- **Limited details on the training and optimization process**\n  - The paper provides a high-level overview of the two-stage training process but lacks specific details on the hyperparameters, learning rates, and convergence criteria for each stage (Sec. 3.3; p.5). This limits reproducibility.\n  - The mechanism for determining the number of basis experts (a critical hyperparameter) is not well-described or justified. A sensitivity analysis on this parameter would be crucial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6qaRg9bQ1G", "reviewer": "Reviewer_cwwa", "summary": "The paper presents a framework for Mixture of Basis Experts for compressing LLMs with a relatively low lose of accuracy.  \n\nPaper’s core claim is clearly stated (24–30% parameter reduction; ~98% relative performance retained on very large MoE LLMs).\n * Method novelty vs MoLAE/D²-MoE and pruning/merge lines is explicit. \n * Theoretical parameter-count analysis (γ formula) is correct and assumptions are reasonable\n\nwith a mixture of shared basis matrices (convex weights) and a non-linear activation—is a clear step beyond SVD-style decompositions used by MoLAE , which rely on linear low-rank sharing.\n\nIt's not entirely novel - It builds on established rank factorisation/dictionary-learning instincts; the novelty is the specific mixture-with-activation sharing and where it’s applied in MoE.  Retaining down matrices is motivated by prior findings on where knowledge resides, rather than a new theoretical claim. \n\nHowever the results appear impressive and the work is fully justified. I think in places the writing is a little terse and perhaps the author could look at providing more of a rationale of why this should work at the start of the paper.", "questions": "Explain the expert-grouping trick for Kimi (384 experts → two 64-basis groups) and study its effect on accuracy and reconstruction error.\n\nCould you share end-to-end inference metrics (latency, throughput, peak VRAM) for MoBE vs MoLAE/D2-MoE across sequence lengths and batch sizes, and clarify any kernel-fusion needs or runtime overheads (e.g., from Z-score normalisation)?\n\nWhat practitioner guidance can you provide for selecting the number of bases \nand activated experts for MoBE), ideally with sensitivity curves showing the accuracy–compression Pareto frontier per model family?\n\nDid you test light compression of the down projections (e.g., small-rank/partial sharing) or a short KD step, and what impact did this have on the residual accuracy gap?\n\nHow does MoBE conversion affect routing dynamics—gate logits, expert-utilisation entropy/load balance—and do you observe drift or mode collapse on tasks sensitive to routing?\n\nHow was baseline parity ensured (matched compression ratios, calibration/data, training budgets), and do you have controlled results on smaller models where D2-MoE is feasible?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "A meaningful architectural re-parameterisation of MoE experts that is novel relative to linear SVD-sharing approaches and practically validated at unprecedented model scales. \n\nResults seem impressive and should be reproducible (I'm assuming there will be a link to code if the paper is accepted).", "weaknesses": "Report end-to-end efficiency, not just parameter counts\n\nStrengthen parity and scalability of baselines - D2-MoE is omitted on trillion-scale models for feasibility; include either (a) scaled-down controlled runs at matched ratios, or (b) additional scalable baselines, so large-model wins aren’t confounded by method availability. \n\nBroaden ablations/analyses - in particular I'd be interested in an analysis involving downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)