KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Models
Authors: Zukang Xu
Zhixiong Zhao
Xing Hu
Zhixuan Chen
Dawei Yang
Venue: ICLR 2026 Poster (2026)
Keywords: vector quantization
llm
Moe
Primary area: foundation or frontier models, including LLMs

Abstract: Mixture of Experts (MoE) models have achieved great success by significantly improving performance while maintaining computational efficiency through sparse expert activation. However, their enormous parameter sizes and memory demands pose significant challenges for deployment in resource-constrained environments.
Vector Quantization (VQ) offers a promising approach for ultra-low-bit compression in Large Language Models (LLMs) by constructing and leveraging a codebook—where weight vectors are mapped to the most similar discrete codewords within the codebook. 
However, its direct application to MoEs suffers from significant performance degradation caused by two critical obstacles:  (1) redundant representation among experts leads to VQ repeatedly quantizing similar representations for each expert, resulting in inefficient utilization of the limited codebook capacity; and (2) cumulative outputs bias, amplified by expert aggregation, leads to distributional shifts in the quantized outputs, resulting in degraded model accuracy.
To this end, we propose KBVQ-MoE, a novel VQ framework to enhance extremely low-bit quantization for MoE-based LLMs. 
KBVQ-MoE introduces two lightweight and offline techniques that introduce negligible runtime computational and memory overhead:
(1) Input-driven redundancy elimination, where a Karhunen–Loève Transform (KLT) guided singular value decomposition (SVD) extracts and shares dominant weight components across experts. 
(2) Bias-corrected output stabilization, where vector quantization is applied to expert-specific (i.e., non-redundant) representations and the quantized outputs are corrected with channel-wise affine compensation.
Experiments on various MoE LLMs demonstrate that KBVQ-MoE preserves accuracy substantially better than existing quantization methods. 
For instance, 3-bit quantization of Qwen1.5-MoE-A2.7B achieves an average accuracy of 67.99, nearly identical to the FP16 baseline of 68.07, underscoring the potential of KBVQ-MoE for efficient deployment on edge devices and other resource-constrained platforms.

Published as a conference paper at ICLR 2026
KBVQ-MOE: KLT-GUIDED SVD WITH BIAS- CORRECTED VECTOR QUANTIZATION FOR MOE LARGE LANGUAGE MODELS
Zukang Xu
Zhixiong 2.21
Xing Hu²
Zhixuan Chen
Dawei Yang
Houmo AI
2 Nanyang Technological University
ABSTRACT
Mixture of Experts (MoE) models have achieved great success by significantly improving performance while maintaining computational efficiency through sparse expert activation. However, their enormous parameter sizes and memory demands pose significant challenges for deployment in resource-constrained environments. Vector Quantization (VQ) offers a promising approach for ultra-low-bin com- pression in Large Language Models (LLMs) by constructing and leveraging a codebook- weight vectors are mapped to the most similar discrete code- words within the codebook. However, direct application MoEs suffers from significant performance degradation caused by two critical obstacles: (1) redundant representation among experts leads to VQ repeatedly quantizing similar represen- tations for each expert, resulting in inefficient utilization of the limited codebook capacity; and (2) cumulative outputs bias is amplified by experts aggregation in MoE layers, leading distributional shifts in the quantized outputs. To this end, we propose KBVQ-MoF, novel VQ framework to enhance extremely low-bit quanti- zation for MoF-based L.LMs. KBVQ-MoE integrates two novel techniques: (1) Input-driven redundancy elimination, where a Karhunen-Loève Transform (KLT) guided singular value decomposition (SVD) extracts and shares dominant weight components across experts. (2) Bias-corrected output stabilization, where vector quantization is applied to expert-specific (i.e., non-reddndant) representations and the quantized outputs are corrected with channel-wise affine compensation. Experi- ments on various MoE LLMs demonstrate that our KBVQ-MoE preserves accuracy substantially better than existing quantization methods. For instance, 3-bit quan- tization of Qwen1.5-Mool A2.7B achieves an average accuracy of 67.99. nearly identical to the FP16 baseline of 68.07, underscoring the potential of KBVQ-MoE for efficient deployment on edge devices and other resource-constraing platforms.
INTRODUCTION
Mixture-of of -ffxxxssts (MoE) models have recently achieved state-of-the-art performance in natural
language processing (NLP) (OpenAl 2023) Kimi Team| 2025] DeepSeck-AI al.) 2024] Jiang
et al. 2024 Yang et al. 2025). By activating only small subset of experts through gating mechanism. MoE enables linear scaling of capacity with the number of experts while keeping
inference cost manageable. However, the growth in expert count substantially increases parameter storage and memory bandwidth requirements. For instance, Qwwen3-Next-O0-AA33 A3BrBBaaaa Cloud
AAlibaaa
2025) requires more than 160GB of GPU memory under FP16 inference. These extreme resource requirements make deployment on edge devices largely infeasible. Post- bost-Trainnng Quantization (PTQ) has emerged as promising solution for compressing L.LMs without the need for retraining (Hu) et al. 2024) Frantar et al. 2022). A typical subcategory of PTQ is Scalar Quantization (SQ), which represents each weight independently by mapping it to discrete value from a lower bit-width set.
"Equal contribution
This work was conducted during his internship at Houmo Al Corresponniig author

---

Published as a conference paper at ICLR 2026
100
KBVQ-MoE VPTQ
1:0
6:37
9.00
6:29
0.20
0.25
0.23
6.31
0.22
1:00
6:99
0:11
0.09
0.31
0.12
0.69
6.17
0.16
GPTQ gB4
09
VQ
1.00
0.34
0.20
0.20
0.32
3.02
0.02
0.50
0.00
0.32
08
Aurg Accuracy
MoeQuant
616
626
0.33
0.36
0.11
A37
0:35
6.15
0.66
0.33
0.29
0:54
0.36
1.00
0.22
0.22
0.50
0.20
0.35
0:30
0.08
0.31
2.00
3.06
0.03
0.28
0.15
0.60
60
9:4
0:20
0.28
0,22
100
0.29
0.27
9.32
0.8 0:35 007
100
035
0.35
05
0.23
0.22
0.30
1.2
0.30
0.00
0.03 0.5
1.00
0.38
0.5
0.22
(0322
0.26
0.33
0.28
0.2
0.20
1.02
0.36
0.12
0:31
000
0.30
0.16
6.15
1.00
Ute Owend
Owen1.5 Moo A2.78
Expert similarity
(b) Expert similanity after KBVQ-McE
Figure 1: Average accuracy across multiple MoE architectures, showing that KBVQ-MoE achieves superior performance under 2-bit quantization.
Figure 2: Similarity of expert outputs before and after redun- dancy elimination by KBVQ-MoE.
SQ performs well from medium to high bit-width (2 4 bits) (Xu al. 2025 Sun et al. 2024
Zhao et al. 2025), ) but its representational capability drops sharply at extremely low bit-width (< 3 bits), leading to significant accuracy degradation. In contrast, Vector Quantization (VQ), another subcategory of PTQ, shows strong potential for ultra-low-bit dense LLM quantization( Yue et al.
2025: Liu al., 2024, Tseng al., 2024b). This advantage realized mainly through leveraging predefined codebook-wiin where weight vectors are mapped to the most similar discrete codewords within the codebook- significantly reducing the data volume while maintaining an acceptable level of information retention.
However, directly applying VQ to MoE archi- tectures suffers from significant performance degradation caused by two key obstacles. O Re- dundant representation among experts. MoE experts often capture similar feature patterns(Du let al. 2025 Sankar & Dimitri 2025 Omi al. 2025
Original Quantized Data: Hean and Variance Comparinon
0. 0.0005
The varfance of vector ouantized expe t outputs
sariance of the expert outpals
The vartance of the eportt ourputa with KBVG Mot
0000
0015
Gu et al. 2025 Li et al. 2025a), resulting in substantial parameter redundancy. As shown in Fig. ((a), experts within the same layer pro- duce highly similar outputs for identical acti- vations, reflecting their overlapping functional roles. This redundancy wastes quantization ca- pacity and prevents limited codebook resources from being concentrated on axpert-specific (i.e.. non-redundant) representations. @ Cumulative and amplified outputs bias. Quantization er- rors accumulate across layers, resulting in bi- ased layer outputs. In MoE. architectures, this bias becomes more pronounced because expert aggregation further amplifies it, leading to more severe distributional shifts than in dense LLMs. As shown in Fig both the mean and variance of outputs shift after quantization. When biased outputs from multiple experts are aggregated through gating, the bias is amplified and propagates across layers, leading to distributional drift and degraded model performance.
0000
0005
The muan of the er grignal expert autous The mean the quartized expert outputs mean the exzert ourputs KiVG-Mot
0.010
2,005
2,000
Index.
Figure 3: Distributional Shifts in Jwve33-30B-A3B Layer 20 Outputs: (Top) Per-channel Mean Com- parisons (FP, Direct VQ, KBVQ-MoE); (Bottom) Per-channel Variance Comparisons (FP, Direct VQ, KBVQ-MoE).
To this end, we propose KBVQ-MoP the first VQ framework tailored to MoE architectures. KBVQ- MoE is built on two efficient innovations: Input-driven redundancy (limination(DDRE)) First, we employ the Karhunen-Loève Transform (KLT) to align expert weights with input activation statis- tics, thereby mapping them into a common latent space, referred to here as a unified representation (see Eq. 3). Next, we apply SVD to this unified representation in order to extract the dominant shared representation, which are retained at full precision, making the remaining expert-specific representations easier to quantize effectively. As illustrated in Fig. bb) after redundancy elimination
2

---

Published as a conference paper at ICLR 2026
the outputs of experts exhibit much lower similarity compared Fig. validating the effective- ness of redundancy climination. @ Bias-corrected output stabilization(BCOS) We apply vector quantization only to expert-specific(ic. non-redundant) representations and stabilize the outputs of quantized experts through lightweight linear scaling and bias correction. As a result, with proposed IDRE and BCOS, KBVQ-MoF provide effective solution for ultra-low- low-bit quantization in MoE LLMs, achieving both efficient codebook utilization and stable output distributions.
We conducted experimental evaluations of the proposed method on a variety of MoE LLMs, including Qwenl.5-Moll 1.5-MoE-A2...7 (Qwen Team, 2024), Qwen3-30B- 30B-A3BBBBBB Yang et al. 2025). Mixtral-8x7B<Iian
al. 2024) and Deepseek V2-Liee DeepSeek consistently outperforms existing scalar and vector quantization methods under the same compression ratio, with particularly strong gains in ultra-low precision settings. For example, at 2-bit quantization on the Qwen3-A3BB-3fH our method improves perplexity by 6 and raises average accuracy by nearly 10%, demonstrating its potential for deploying MoE LLMs on ressurce-constrained devices such as edge platforms.
Al al. 2024). As shown Fig. I KBVQ-MoE
The main contributions of this paper are as follows:
We identify two key challenges that arise when vector quantization is applied to MoE-based LLMs: the waste codebook resources caused by common redundant representation among experts, and outputs bias in quantization errors exacerbated by expert aggregation.
We propose the KBVQ-MoE framework, which integrates Input-driven redundancy elimina- tion and Bias-corrected outputstabilization.
Both theoretical analysis and experimental results demonstrate the effectiveness of the KBVQ-MoE method. It exhibits significant advantages over existing methods mod- els such as the Qwen series and Mixtral, and even achieves near-floating point accuracy performance under 2-bit quantization.
2 RELATED WORK
2.1 POST-TRAINING QUANTIZATION (PTQ)
Scalar Quantization (SQ) assigns an independent scaling factor and zero point to each tensor (e.g., layer weight matrix), mapping continuous values to discrete integers. Methods such as GPTQ (Frantar et al. 2022) and GPTAQ (Li al.) 20255) leverage Hessian information to optimize the error propagation path, while I-LLM (Hu et al. 2024) achieves quantization of large models through smoothing and full integer inference approximation. Approaches like Quarot (Ashkboos et al.) 2024) and OSTQuant (Hu et al. 2025b) incorporate orthogonal transforms to improve quantization efficiency. However, SQ suffers from representational bottlenecks ultra-low bit-width scenarios (Σ bits), making it difficult to balance the compression ratio and accuracy of MoE LLMs.
Vector Quantization (VQ) clusters weight vectors into shared codebooks and leverages structural redundancy to achieve higher compression ratios (Gersho 1979). GPTVQ (Van Baalen al. 2024) combines sxpecrationmmaxiiizztioo with error feedback to optimize codebooks; VPTQ (Liu 2024) and AQLM (Egiazarian et al. 2024) employ residual quantization
reduce error
accumulation; PCDVQ (uue et al.) (2025) achieves efficient quantization by decoupling the magnitude and direction of vectors; QulP# (Tenng al. 2024a) and QTIP (Tseng et al. 2024b) utilize geometric transformations to improve error distribution. These studies have demonstrated the superiority of VQ at extremely low bit-widths for dense LLMs. Nevertheless, when existing VQ techniques are directly applied to MoE LLMs, they fail to account for the unique structural information of MoE, resulting in suboptimal compression performance.
2.2 MoE LLMs COMPRESSION METHODS
Research model compression for MoE architectures is still in its infancy. Most methods directly adopt general model compression techniques, lacking special consideration for the structural char- acteristies of MoE gating and multi-expert architectures. EAC-MoE (Chen al. 2025) reduces parameters by pruning redundant experts, but pruning causes irreversible loss of functionality. D2- MoE (Gu et al. 2025) and SubMoe (Li al. 2025a decompose expert weights into low-rank factors,
3

---

Published as a conference paper at ICLR 2026
yet their compression ratios are limited by rank constraints, failing meet the demands of extreme resource-constrrr constrained scenarios. MoEQuant (Hu et al. (222a) uses routing statistics to balance the contributions of each expert during calibration, but its performance remains unsatisfactory under quantization of v 4 bits.
In summary, existing MoE LLMs compression methods lack collaborative optimization mechanism that integrates the statistical characteristics of input activations they neither fully exploit input- related common patterns shared across experts nor specifically correct distribution shifts caused by quantization errors of experts. This makes it difficult for them to balance model accuracy at high compression ratios. The KBVQ-MoE framework proposed in this paper is designed fill this gap. significantly improving the quantization performance of MoE LLMs through input-driven redundancy elimination and bias-corrected output stabilization.
3 PRELIMINARIES
Eecture-of-Experts Mixture-of-EEperts (MoE) architectures modify standard Transformer layers by replacing conventional feed-forward (MLP) modules with specialized MoE modules, enabling efficient scaling of model capacity while maintaining computational effcciency(acobs et al.] 1991 Jordan & Jacobs 1994). MoE layer typically comprises two types of components: a set of m shared experts ([Fi..... ..... pm D. a set of routing experts ({Ef (Ef,.... E D), and gating network (or router network) that determines which experts process each input token.
Each exptrt-whether whether shared (effectively a compact MLP), with shared experts designed to handle general patterns across inputs and routing experts specialized for specific input subsets. When processing input hidden state x E Rd, the MoE layer operates through a structured sequence:
couting-function
lightweight feed-forward sub-network
1. Gating Score Calculation: The gating network computes weights (or affinities) (x) for all experts, quantifying how well each expert aligns with the input æ. These weights reflect the probability of routing the corresponding expert.
Expert Selection: For routing experts, only the top k experts with the highest affinities are selected, denoted by the subset K topk (z)( j E .i....,,]]]). In contrast, all shared experts typically utilized to preserve general representational capacity.
3. Weighted Output Computation: The final output y of the MoE layer is weighted sum of outputs from the selected experts, where the weights are the corresponding gating scores. Mathematically, this is formalized as:
(()g(()))))) +
Σ Fy (x) JEK
shared experts
top & routing experts
While this formulation captures the core mechanism, specific designs vary across architectures example, in how gating scores are computed, how experts are structured (e.g., depth, activation functions), or how the balance between shared and routing experts tuned.
4 METHOD
This section introduces the proposed KLT-guided SVD with Caas-Coreeeted Vector Quantization for MoE. Large Language Models (KBVQ-MoF). The framework is built on two innovations that align with the challenges identified in Section T (1) Input-driven Redundancy Elimination (IDRE), where the Karhunen-Loève Transform (KLT) maps expert weights into a unified representation aligned with input statistics, and singular value decomposition (SVD) is then applied to extract dominant shared components retained at full precision, leaving expert-specific (i.e., non-redundant) representations that more amenable to quantization; (2) Bias-Corrected Output Stabiliza- tion (BCOS), where vector quantization is applied only to the expert-specific representations, and lightweight linear scaling with bias correction is used to stabilize outputs and mitigate distributional
4

---

Published as a conference paper at ICLR 2026
shifts. The overall workflow of KBVQ-MoE is mathematically formulated in Eq algorithmic flow is provided in Appendix !
while detailed
W
W
Wehare
+
W
quant
BCOS
Wehare +
W
quant
(8,
KLT+SVD
Bias Correction
Sharod Part
Specific Part
no quant
quant
scaled&bias.
4.1 INPUT-DRIVEN REDUNDANCY IOMINATIONNIDREE
In MoE architectures, experts often exhibit substantial redundancy, as their weights encode overlap- ping mappings for identical activations. This redundancy wastes quantization capacity and prevents limited codebook from focusing on expert-specific representations. To address this issue, we introduce Input-driven Redundancy Elimination (IDRE), which leverages the statistical char- acteristics of model inputs to construct unified representation of expert weights. By isolating shared components and retaining them full precision, IDRE reduces redundancy and improves the efficiency of codebook utilization for subsequent quantization. IDRE is applied uniformly to the MIP weights of all experts in each MoE layer- layer-including both shared and routing experts. The detailed transformation of the MoE structure before and after applying IDRE is illustrated Fig. 4 This procedure consists of the following steps:
input
input
Uherc:
E0
En
Gate
V/1
En'
mul
mul
mul
mul
mul
mul
(add
edd
output
output
(a) Original Moe Block
(b) Moe Block after IDRE
Figure 4: The comparison of structural changes in the MoE (Mixture-of (Mixture-of-Experte Expeettstt structure before and after IDRE.
Step 1: KLT Decomposition of Input Activations: Constructing the Input Coherence Space. To reduce redundancy, we begin by applying the Karhunen-Loève Transform (KLT) the input activations in order to construct coherence basis that captures their dominant directions. Let the input activation matrix of the expert layer be X E Rbxic, where denotes the number of samples and ic is the input dimension. The procedure is as follows:
1. The covariance matrix computed as Cx 1 XT Ricxic
B-1 11,
2. An eigenvalue decomposition is performed on Cx:Cx = (UKLTA (UKLrAkarr)
where UKLT [u...... dice] is an orthonormal basis, and ARIT = diag(A1 2 ... 2 die) is diagonal matrix of eigenvalues. Here, U represents the j-th coherence direction of the input, and X, denotes the energy magnitude this direction.
3. Based on this, we construct the input coherence basis: Ux U The column vectors of the resulting Ux are sorted by input energy magnitude, forming orthogonal coordinate system based on input statistics. This facilitates prioritizing the extraction common weight structures related high-energy input directions in subsequent steps.
detailed theoretical derivation linking this decomposition to the minimization of quantization emor provided in Appendix A.2

---

Published as a conference paper at ICLR 2026
Step 2: Mapping Weights to the Input Coherence Space. To ensure that weight analysis is guided by the dominant directions of the input, we project the original weight matrix W E R onto the input coherence basis obtained in Step I: W WUx E IRooxic, where oc denotes the output channel dimension and ic denotes the input channel dimension. In this transformed representation, each column of W corresponds the mapping of specific coherence direction of the input the output space. The j-th column, in particular, represents how the weights project along the j-th most energetic input direction uy. This alignment establishes direct connection between weight structures and input characteristics, which facilitates the subsequent extraction of shared and expert-specific components.
Step 3: Extracting Input- @ppt-CChereet Common Structures: Precise Redundancy Elimination. The common structures of MoE experts manifest as cross-expert shared weight patterns in the input coherence space (c.g., similar mappings for the high-energy direction U1). To capture these patters,, concateniate the mapped weights of all experts(W<() denotes the i-th expert)-along their output channel dimension-iin into (n oc) x ic-dimensional unified representation:
W()
WOUx
17
Wx
W
(n
(3)
[W()]
We then apply SVD to W in order to extract the dominant shared representation:W = (UEVT)"
1. U E Riek denotes the shared left singular vectors (shared mapping directions across experts, where k is the number of retained dominant singular values, focusing on input- coherent common patterns); Σ = diag( ......... ok) is the singular value matrix (the larger jj is, the more prominent the corresponding dominant shared representation).
2. The common structures in the input coherence space are mapped back to the original weight space via the inverse transformation the input coherence basis (Ux )):
Ushare UT Uxx R*xic
V E xkk represents the right singular vectors, which are partitioned by expert into:
(1)", (2) ,...., ,,..,, V...,"""""]"
(5)
where v(i Roex is the private right singular vector for expert
3. The truncated low-rank components Ustare and V/ obtained from SVD are retained in full precision to preserve the fidelity of shared and expert-specific representations.
Thus, IDRE explicitly decouples shared structures (kept at full precision) from expert-specific representations (subject to quantization), enabling more efficient use of codebook capacity. rigorous spectral characterization of this decomposition, including its optimality and error bounds, is provided in Appendix A.3
4.2 BIAS-CORRECTED OUTPUT STABILIZATTIO(BCOOS)
While IDRE reduces redundancy and improves codebook utilization, quantization of the remaining expert-specific weights W Wanntt still introduces cumulative bias that distorts output distributions. To mitigate this problem, BCOS stabilizes the outputs through vector quantization followed by lightweight channel- -iiee affine compensation.
Step 1: Vector Quantization of Expert-Specific Weights. We perform VQ quantization the remaining expert-specific weights Wquant in the original space: partition Wquant into subvectors E Rdd, map these subvectors to indicesq = arg min, || cyll² via codebook C = {cI,... ..., Cx }. and the quantized result is 2g = Cq- The quantized matrix is denoted as: W Wqunmvo E ocxie The compressed weight is the sum of the shared component and the quantized component:
Wvg
Whhre
W Wquam.VQ
(6)

---

Published as a conference paper at ICLR 2026
The quantization error, arising solely from the quantization of the specific component, is defined as e = WvQ W.
Step Channel-Wiss Bias Correction, As the number of layers increases, the quantization error leads to distributional shifts in the outputs. we apply channel-wise affine compensation to align the statistics of quantized outputs with those of the original model. The corrected output of the linear layer is formulated as
Yoor = Wvqz + s s (Wvqx) + b- (s + (Wvoa) + b,
where 8 E Rec is a channel-wise scaling factor; b E Roce is channel-wise bias term; @ denotes element-wiss multiplication. Here, Wvo is the original quantized output. We choose s and b to ensure that the corrected output Ycor matches the mean and variance of the full full-precision output y = Wx. Following unbiased estimation in the sense of Minimum Mean Square Error (MMSE) (see Appendix A.4), the optimal parameters approximated as
89
yj - 1, by = Pyy - (1 +1++0 43)
(8)
where a and ju denote the standard deviation and mean, respectively, and both operations are chinnel-wise. In other words, each channel is scaled by the ratio of standard deviations between the original and quantized outputs, and shifted by the corresponding mean difference. After this correction, both the mean and variance of each channel effectively eliminating the distributional shift caused by quantization. This correction introduces only 2oc additional parameters per layer and requires only lightweight per-channel scaling and shifting operations during inference, resulting in negligible computational and storage overhead.
aligned with the full-precision baseline,
5 EXPERIMENT
Models and Datasets. To comprehensively verify the performance effectiveness and scenario appli- cability of the KBVQ-MoE framework, we conducted experiments on multiple sets of representative Mixture-of- (MoE) models and diverse evaluation tasks. For the selection of evaluation models, we prioritized currently mainstream pre-trained MoF. L.MSs to ensure the reference value of the experimental results. Specifically, these models include the Qwen series (e.g., Qwenl.5-MoF. M- A2.7B (Qwen Team 20044) 1wwen3-30B-A33 (Ynng et al.. 2025)). Deepseek V2-Lite (DeepSeck-A) AI al. 2024), and Mixtral (Jiag et al.) 2024). In terms of the design of evaluation tasks and datasets, we focused on natural language reasoning and understanding benchmarks, and comprehensively examined the framework's 's performance through multi dinliiidimenional metrics: On the one hand, we tested the perplexity (ppl) of the language model (LLM) output with a sequence length of 4096 on the Wexi'exx22 dataset (Merity et al., 2016), to measure the coherence of language generation and semantic modeling capability of the compressed model: - On the other hand, we evaluated the model accuracy on zero-shot datasets, including Are- Are-Challenge (Clark et al. 2018). Arc-Easy (Boratko et al. 22188) HellaSwag, (Zellers et al. 2019). LAMBADA LAMBADA-openall (Paperno al. 2016)). LAMBADA-standar (Paperno et al. 2016). PIQA (Bisk et al. 2020), and WinoGrande (Sakaguchi
al. 2021 This was done to comprehensively verify the generalization capability of the compressed
model across different task scenarios.
Baselines. To demonstrate the effectiveness and superiority of our method, we compare KBVQ- MoE with range of well-validated excellent methods, such Round To Nearest (RTN) and GPTQ (Frantar et al. 2022), the MoEQuant (Hu et al. 2025a) (proposed specifically for MoE quantization), as well VQ. We respectively compare the accuracy differences of these methods under ultra-low bit-widths (23 bits), with their compression ratios evaluated based on actual storage usage.
Implementation Details. All experiments were conducted on NVIDIA RTX A6000 GPU. The calibration dataset used in our experiments was sampled from the Red PajaPaaaaaaaaaa dataset (Together Computer 2023) ; we fixed the random seed to 42 and randomly sampled 256 data samples with a sequence length of 4096 from the Red Pajama dataset, which served as the calibration set for the IDRE and BCOS methods in this paper. In the IDRE process, we set the truncated rankk KLT-SVD to 1/ 128 of the full rank, and under this configuration, the average number of parameters increased
7

---

Published as a conference paper at ICLR 2026
Table Comparison of average accuracy between the KBVQ-MoE method and other quantization methods on various different moe models
QwenT A-M00--22.78
Qwen3 30B-A38
Mixtral-8x7B
Deepsock V22-oo
Bit
Method
(J)
Avg Acc
(1)
Avg Ace
(J)
Avg
W2
Avg Ace (1)
16
FPI6
7.22
68.07
8.70
70.24
3.88
78.57
5,92
70.68
RTN
638309
25.64
765922
25.89
274952
25.27
174653
25.12
GPTQ
12.69
49.07
438.42
33.06
5.69
67.56
8.49
58.04
MoeQuant
583542
37465
28.94
13.43
40.16
25893
25.59
VQ
26.95
47.84
115.30
30.61
5.99
59.22
10.96
49.85
KBVQ-MoE
9.61
62.78
11.87
63.37
5,39
75.69
7,94
63.10
RTN
I6689
25.68
G8637
25,84
45136
25.80
97.75
33.32
GPTQ
7.58
66.36
11.32
63.51
4.17
77.43
6.98
68.89
MoeQuant
66.79
37.32
72.21
7.52
66.34
VQ
11.47
55.94
18.72
52.06
5.52
73.98
7.94
62.26
KBVQ-MoE
7.74
67.99
9.26
69.09
4.07
78.35
6.95
68.73
by SVD redundancy extraction is only 0.12. this paper, the length the vector quantization vector length is set 4. This selection will reduce the occupation of the codebook low level. Meanwhile, the k-means algorithm adopted in the vector quantization process uses kmeans++ for initialization and sets the iteration number 100, which helps balance the stability of codebook generation and the efficiency of quantization computation. All evaluations were performed using the open-source AM-EEaluation-Hanae toolkit (Gao al. 202I
Main results. Table 1 presents the results of direct comparisons between method and other approaches. Overall, our method demonstrates significant advantages across different models and quantization bit-widths, achieving average accuracy (Avg Acc) close full precision (FP16) while maintaining low eгror. Particularly in low-bit quantization scenarios (S. 3 bits), our method exhibits notable robustness and generalization capability. For Qwen models, take Qwen3-30B-AN 30B-A3B Yang et al., 2025) under 2-bit quantization: its PPL decreases by nearly 6 points, and Avg Acc increases by more than 10 points. For Qwen 1.5-MoE-AAA A..7.2..7B A..7B(Qween Team 2024) and Mixtral-8x7B(iaal et al.) 2024) under 3-bit quantization, their Avg Acc reaches 67.99 and 78.35, respectively, which are nearly identical to the FP16 precision. Detailed accuracy results for each dataset and model are provided in Appendix
A.7 A.7
Table 2: Ablation studies on the pre-process and post-process of KBVQ-McE respectively
Model
Bit
IDRE
BCOS
W2
ARC-E
ARC-C
HE
PIQA
WI
FP16
8.70
79.25
56.40
59.60
80.30
70.48
18.72
57.83
40.87
63.23
71.82
57.70
Qwen3-30B-A3B
V
X
11.67
71.35
50.55
73.51
77.75
66.92
14.32
65.49
47.33
68.37
75.52
60.42
9.26
77.27
53.24
75.53
78.89
70.01
Ablation Experiments. As shown on Table 2 In the 3-bit quantization Swven-33B-A33. our complete scheme (IDRE BCOS) achieves the best performance across the board: the perplexity (ppl) on Wikitext2 decreases to 9.26, representing a 50.5% reduction compared to the unprocessed baseline (18.72) this is 20.6% decrease compared to the IDRE-only scheme and 35.3% decrease compared to the BCOS-only scheme. Additionally, comprehensive improvements are observed across five benchmark tasks. From the ablation of individual modules, is evident that IDRE is the primary source of performance gain, while BCOS further corrects residuals and unleashes a synergistic effect on this foundation. Their combination enables a significant leap in overall accuracy, demonstrating that the proposed two innovations can approach full-precision performance under extremely low bit-widthh This validates strong robustness and practical value for real-world deployment.
Table 3: Ablation on using KLT to guide SVD for expert redundancy
Model
Bit
Method
W2
ARC-E
ARC-C
PIQA
WI
SVD+VQ
14.03
68.21
43.59
67.32
76,60
66.80
Qwenl 5-MoE---M
2
KLTSVD++Q
11.87
70.33
47.61
67.49
76.66
66.46
8

---

Published as a conference paper at ICLR 2026
In IDRE, the Karhunen- Transform (KLT) to extract the input coherence basis for redundancy removal across experts, which enables more direct redundancy extraction. Table 3] we compare the direct performance between redundancy extraction without input coherence and that with KI.T-based input coherence. Clearly, the IDRE scheme yields greater accuracy improvement: model, the perplexity (ppl) Wikitext2 decreases more
specifically, the @weel1.5MMoF-AN than 2 points.
For the IDRE, we conducted ablation studies determine the optimal ratio of the truncated rank of SVD the full rank n. Table 4 presents the ablation results of Qwen3-30BB 3--3BBBB3BBBB under 2-bit quantization: without IDRE, the model performs poorly, with perplexity (ppl) of 15.3 on Wikitext2. However, after applying SVD truncation to pre-extract expert redundancy, the ppl drops sharply to 11.87. Further increasing
Table 4: Ablation study the rank of SVD trun- cation in the Pre-Process procedure
Model
Bit
k/n
W2
2.01
15.30
2.08
1/128
11.87
Qwen3-30B-A3B
2.11
1/64
11.30
2.20
1/32
11.01
the truncated rank, performance improvements become marginal, while the average bit-width in- creases instead. Therefore, recommend setting the truncated rank to 1/128 of the full rank.
To further validate the effectiveness of our ap- proach on MoE architectures, we integrate our IDRE and BCOS modules with different vector quantization baselines, treating them as a general plug-in component for MoE quantization. This design demonstrates that our method is both ver- satile and efficient, it can be seamlessly com- bined with existing quantization techniques to enhance their performance. As shown in Table 5 on Owen1.5-MoBF 1.5 -anGE-AA.755 A2w7B(Qwen Team 2024) ap- plying method GPT'VQ(VVa Baalen al. 2024) under 3-bit quantization yields nearly 30% ppl improvement, while under 2-bit quanti- zation with VPTQ(Liu et al. (024), our method still achieves more than a 15% performance gain.
Table 5: Experimental Results of Different VQ Methods Integrated with KBVQ-MoB for Qwenl 1.5MMe-A22.7B Model Under 2-bit Quan- tization
Base Method
IDRE
BCOS
W2
X
12.88
X
11.03
GPTVQ
X
/
11.97
9.43
X
X
10.17
V
X
9.21
VPTQ
X
9.77
8.78
We conduct a speed evalua- tion in the decoder stage. In our tests with 1k input to- kens, as shown in T the 5wenl.5--Mo-AA2. under 2-bit quantization (in- tegrated with KBVQ
Table 6: Decoder speed test
Decoder Speed (Tokens/s)
Model
BF16
Quantized
Speed Up
QwenI.5-Moo 1.5-MoeA22.7B
22.31
35.24
1.58
model
Qwen3-30B-A3B
10.85
17.37
1.60
-MoE and corresponding VQ methods) achieves an inference speedup of over 1.5x, further validating the practical deployment value of the proposed framework in balaneing accuracy retention and efficiency.
CONCLUSION
This paper proposes KBVQ-MoE, the first vector quantization framework tailored to Mixture-of- Experts (MoE) architectures. We address two critical challenges of applying vector quantization MoE LLMs: (1) redundant representation among experts, which wastes limited codebook capacity, and (2) cumulative and amplified outputs bias, which manifests during expert aggregation and leads to severe distributional shifts. KBVQ-MoE integrates two key techniques: Input-driven Redundancy Elimination (IDRE), which isolates shared structures and retains them full precision, and Bias-Corrected Output Stabilization (BCOS), which applies vector quantization only to expert-specific representations while correcting output distribution shifts through channel-wise affine compensation. Both theoretical analysis and empirical evaluations on diverse MoE LLMs (e.g.. Qwen and Mixtral) demonstrate that KB VQ-MoE achieves superior performance under ultra-low-bii quantization, preserving accuracy with negligible runtime overhead. In future work, will extend the framework to adaptive codebook design and expert pruning, further improving the deployment efficiency of large-scale MoE models in resource-constrained environments.

---

Published as a conference paper at ICLR 2026
REFERENCES
Tongyi Qianwen Team Alibaba Cloud. Qwen3-next: Next-generation foundation model architecture.
ctttps://modessco// en/ en/models// ct//mooeels/mee//eeeeen Owwn/wwen3 Next 80B A3B- Instruct 2025.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorsk Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 conference the North American chapter of the association for computational linguistics: Human language technologies, volume (long and short papers), pp. 2357 2367, 2019.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated llms. Advances Neural Information Processing Systems, 111100011-100440 2024.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7432-7439, 2020.
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang. Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, et al. A systematic classification of knowledge, reasoning, and context within the arc dataset. arXiv preprint /806 00358, 2018
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss William H. Guss, Andrew N. Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Sandeep Balaji, Sharan Jain, William Saunders, Christopher Hesse, and John Schulman. Evaluating large language models trained on code, 2021.
Yuanteng Chen, Yuantian Shao, Peisong Wang, and Jian Cheng. Eac-moe: Expert-selection aware compressor for mixture-of-experrt large language models. arXiv preprint arXiv:2508.0162. 2025.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rX3vv:800005555 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021.
DeepSeek-Al AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Deng, Chong Ruan, Damai Dai, Daya Guo. Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fuli Luo. Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qihao Zhu Qinyu Chen, Qiushi Du, R. J. Chen. L. Jin, Ruiqi Ge. Ruizhe Pan, Rumxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang. Tian Pei, Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang. Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Zheng. Yichao Zhang, Yiliang Xiong, Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong, Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang,
10

---

Published as a conference paper at ICLR 2026
Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhewen Hao. Zhihong Shao, Zhiniu Wen. Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. CoRR, Abs/2405.04434, 2024. Ittphhtts://aaxiiv... wwwp://wwwrr/a/aaooo/oo/oaoaaaa
abs
04434
Zhenbang Du, Jiayu An, Yunlu Tu, Jiahao Hong, and Dongrui Wu. Mixture-of-experts for open set domain adaptation: A dual-space detection approach. IEEE Transactions Artificial Intelligence, 2025.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Bahenko, and Dan Alistarh. Extreme compression large language models via additive quantization. arXiv preprint arXiv:2401.061 08111. 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint 2rXiv:2210.17332 2022.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, and Niklas Muennighoff. A framework for few-shot language model evaluation. Technical report, GitHub, September 2021.
Allen Gersho. Asymptotically optimal block quantization. IEEE Transactions on information theory, 25(4):373-380, 1979.
Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, and Yike Guo. Delta decompression for moe-based lims compression. arXiv preprint arXiv:2502. 17298, 2025.
Xing Hu, Yuan Cheng, Dawei Yang. Zhihang Yuan, Jiangyong Yu, Chen Xu, and Sifan Zhou. I-llm: Efficient integer-only inference for fully-quantized low-bit large language models. arXiv preprint arXiv:2405. 17..29999.789 2024.
Xing Hu, Zhixuan Chen, Dawei Yang. Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. Moequant: Enhancing quantization for mixture-of-expee large language models via balante sampling and affinity guidance. arXiv preprint xxivv25555.03804 2025a.
Xing Hu, Yuan Cheng, Dawei Yang. Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. arXiv preprint ar 13987 2025b.
Robert A. Jacobs, Michael Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-8)) 87, 1991. doi:
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts. CoRR, abs/2401. 104088, 2024. https://::/ http:///axxvvv.///eeee
04088
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the algorithm. Neural Computation, 6(2):181-2) 1994, doi: 10.1 00.1162/neco.199444 1994.6.2.181.
Kimi Team. Kimi-vl technical report. CoRR, abs/2504.07491, 2025. URL https: //apxii////xvvv
07491
Lujun Li, Zhu Qiyuan, Jiacheng Wang, Wei Li. Hao Gu, Sirui Han, and Yike Guo. Sub-moe: Efficient mixture-of-expert llms compression via subspace expert merging. arXiv preprint arXiv:2506.23226, 2025a.
Yuhang Li, Ruokai Yin, Donghyun Lee, Shiting Xiao, and Priyadarshini Panda. Gptaq: Efficient finetuning-fie quantization for asymmetric calibration. arXiv preprint arXiv:2504.02699 2025b.

---

Published as a conference paper at ICLR 2026
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang. Ting Cao. Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. arXiv preprint arXiv:2409.17066 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint (efi::1609.7788800 2016.
Nabil Omi, Siddhartha Sen, and Ali Farhadi. Load balancing mixture of experts with similarity preserving routers. arXiv preprint arXiv:2506. 2025.
OpenAl. Gpt-4 technical report. CoRR, abs/2303.08874, 2023. doi: 10445500aaXxv.2200.007774
URL 1. http:///......5///.......
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaclla Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016.
http//axxiv...g/abba///o abs/
1506606031
Qwen Team. Qwenl.5-moe: Matching 7b model performance with 1/3 activated parameters. https: //gwenlm. //qwenlm.github.. git.ub. htt.chh.......................ee wt/llo/lloooo /wwp///wweemmee/e/ee////mm 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Eustache Sankar and Vasily Dimitri. Mixture of experts models in deep learning and their techniques applications and challenges. Authorea Preprints, 2025.
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, et al. Flatquant: Flatness matters for Ilm quantization. arXiv preprint arXiv:2410.09426. 2024.
Together Computer. Redpajama-data- It: An open dataset for training large language models. https ReePP jama-Data- 2023.
//buggingface co/
Available on Hugging Face Datasets.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quipi:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396 2024a.
Albert Tseng, Qingyao Sun, David Hou, and Christopher M De Sa. Qtip: Quantization with trellises and incoherence processing. Advances in Neural Information Processing Systems, 37:59597-59622 2024b.
Mart Van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing of dimensionality for I1m quantization. arXiv preprint 1rX1v:2402.15319, 2024.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Aruiraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi- task language understanding benchmark. Advances in Neural Information Processing Systems, 37: 95266-95290, 2024.
Zukang Xu, Yuxuan Yue, Xing Hu, Zhihang Yuan, Zixu Jiang, Zhixuan Chen, Jiangyong Yu, Chen Xu, Sifan Zhou, and Dawei Yang. Mambaquant: Quantizing the mamba family with variance aligned rotation methods. arXiv preprini arXiv:2501.1 3484, 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng. Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin,
12

---

Published as a conference paper at ICLR 2026
Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang. Zeyu Cui. Zhenru Zhang. Zhipeng Zhou, Zihan Qiu, et al. Qwen3 technical report. CoRR, abx/2505.093888 2025. URL https:
//arxiv. org/ abs/ 250..09358888
Yuxuan Yue, Zukang Xu, Zhihang Yuan, Dawei Yang, Jianlong Wu, and Liqiang Nie. Pedvq: Enhancing vector quantization for large language models via polar coordinate decoupling, arXiv preprint arXiv:2506.054333 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint w7Xiv:7905007884 2019.
Zhixiong Zhao, Fangxin Liu, Junjie Wang, Chenyang Guan, Zongwu Wang, Li Jiang, and Haib- ing Guan. Specquant: Spectral decomposition and adaptive truncation for ultra-low-bit llms quantization. arXiv preprint arXiv:2511.11663, 2025.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1
USE OF LLMS
Large language model (LLM) tools were employed solely for text polishing, including refinement of sentence structure, lexical optimization, and enhancement of language fluency.
All core components of this research-such as the conception of research ideas and framework, algorithm design and implementation, experimental protocol design, data collection and processing. execution and analysis of experiments, and validation of conclusions- were independently completed by the research team.
LLM tools were not involved in the conception of research content, generation of technical solutions, execution of experiments, or derivation of conclusions. The authors affirm full responsibility for the scientific validity, authenticity, and originality of the work in accordance with academic integrity standards.
A.2 THEORETICAL PROOF OF THE INPUT COHERENCE BASIS
Let the linear mapping of the i-th expert be y = ( x, with its quantized version denoted as W and the error matrix defined as E(0 & ( W(O, For the input covariance Cx = E[xx"' that
reflects the true distribution during inference, natural definition of task distortion is the output mean squared error (MSE):
A
Wr x W(
2 =
E Cx))).
This metric directly measures the impact of quantization on the output side, rather than the unweighted difference of the weights themselves. Perform the Karhunen Loèvee Transform (KLT) on Cx such that Cx UKI.TAKITUKKK KLT- where AKIT diag(A) 2 ... Xie). Define the input coherence basis as Ux e UKIT Then the following identity holds:
1/2
Tr(ECxE*) TT(UUxUX)) 1E
(10)
thus
L= Σ -W
(11)
This indicates that: in the input coherence space, the output MSE is equivalent to the Frobenius norm of the weight error. Therefore, all "extraction/quantizatioo" operations aimed at reducing output distortion should performed in this space. Denote Vi((i) A W0
A.3 SPECTRAL CHARACTERIZATION OF THE OPTIMAL SHARED SUBSPACE (KLT-SVD).
To formalize the redundancy removal performed by IDRE, we stack all expert weights along the output dimension:
(1)
W
E R (noc)
S 2 W.W.
(12)
(n(n)
Here (i) denotes the i-th expert weight expressed in the input KLT coordinates introduced in Appendix. A.2 The matrix S is the (input-aligned) Gram matrix of all experts. practice, IDRE computes a truncated SVD the stacked matrix W E xin W UEV T_ Note that the Gram matrix.S = W W admits the eigen-decomposition S = VV*VT i.e., the eigenvalues of S are the squared singular values of W, and the eigenvectors of S coincide with the right singular vectors of W. Therefore, selecting the top-k eigenvectors of S is exactly equivalent to performing rank-k truncated SVD on W and keeping the top-k k right singular vectors. The parameter & used in our analysis and implementation is the dimension of this shared right-singular subspace.
14

---

Published as a conference paper at ICLR 2026
Optimal shared subspace. energy of all experts:
We seek k-dimensional shared subspace that maximizes the stacked
max U UTU=I
TU SU)
max
WUU
(13)
O UT
By Ky Fan's theorem, the optimizer Un is given by the top-k eigenvectors of S associated with its largest eigenvalues {o}} Let Ps. - U&U be the projector onto this shared subspace. In the KLT space, each expert admits the orthogonal decomposition
PR
(I-P). specific (difference)
(14)
shared (recuundancy)
and mapping back to the original coordinates yields W W PaT and W = W - share
share"
where T is the KLT transform defined in Sec. A.2. Intuitively, the shared component captures directions that are simultancously important for many experts, while the specific component collects expert-unique variations.
Why KLT before SVD. Let z be the input to the MoE layer with covariance Σx B[ax UxAxUx U and define the input KLT transform by = UxAX² In this basis, the stacked Gram matrix takes the form
8 W W =T
Σ WOO W T
Σ WOT UxAX
(15)
Thus the spectrum {o}} of S jointly reflects both the input energy (through Ax) and the expost weight energy (through E, W()T w() along each input principal direction. Selecting the top- k eigenvectors of therefore retains those directions that are simultaneously dominant the input distribution and heavily utilized across experts, which is precisely the notion of "shared" structure we aim to preserve in full precision.
Truncation error and redundancy removal. The residual (expert-specific) part after projecting onto PA satisfies
Σ (I
( Σoj j>k
(16)
where { are the eigenvalues of S sorted in decreasing order. Hence the truncation error is exactly controlled by the tail eigenvalues discarded by Pç.. Correspondingly, the redundancy removal ratio of IDRE at rank can be quantified as
Σ Σ
ビj=
Pk
A
(17)
which measures the fraction of stacked expert energy captured by the shared subspace.
This characterization provides explicit trade-off: increasing k monotonically raises P and reduces the truncation error in equation 16] but also enlarges the full-precision shared component. Empirically, we observe that the spectrum
for MoE LLMs such as -0wenI.5-MoE-e Mo5--277. and Qwen3-30B-AA3 of S is strongly low-rank and decays rapidly (approximately heavy-tailed). As shown in the Fig[5] demonstrates the obvious low-rank characteristics of the expert groups and their posterior singularity.. Under a simple power-law approximation of x j-a with a > 1, the residual fraction 1 pu behaves like Ck--o; once h exceeds moderate threshold, the marginal gain from further increasing l quickly diminishes. Across layers and models, choosing &o Rs ie/128 typically yields Pk A 0.6-0.8, while larger ranks (e.g.. l i./64) provide only marginal perplexity improvements at noticeably higher storage cost. [ shows a comparison of k selection for two moe models of Qwen, and the low rank of 1/128 already achieve very good results. This explains why k / /18 serves robust operating point that balances reconstruction error (controlled by the tail eigenvalues) and the overhead of the shared full-precision subspace across different models and tasks.
15

---

Published as a conference paper at ICLR 2026
energy up proj experts
up sxperts
1.0
1.0
Normalteed stegutae value magnituute
Norma simpular magnitude
0.5
CLOZE
CUEORE
1V006 (1/23)
16 Singular value Olgi scale, scale corresponds k=2"x)
2004
255 Singular value (legs scale corresponds to
(a)
(b)
Figure 5: The low-rank characteristics after expert fusion activation of the first block, (a) Qwenl.5-
MoE-A2.7B, (b) Qwen3-30B-A33
Table 7: Low-rank value ablation
k
1/256
1/128
1/64
1/48
1/32
Compress Ratio
2.05
2.08
2.11
2.15
2.2
Qwen -5-MoE-A2.78
20.71
9.61
9.60
9.58
9.58
Qwen3-30B-A3B
15.30
11.87
11.30
11.30
11.01
A.4 SYSTEM BIAS CORRECTION
In the paper, we reformulate bias correction as residual affine transform on the quantized output:
Ycom = Wvo + a @ (Wvoz) + b (1 + s) @ (Wvoz) + b
(18)
where s, E Roe denote Channel-wise residual scaling and bias, respectively. We now show that this correction statistically optimal under the mean squared error (MSE) criterion.
1. Problem Setup Let the full cuel-iico output be = Wr x and the quantized output be 9 = Wvgæ. Our goal is to determine and b such that
Yeorr
+ 5) 2 b
approximates as closely as possible. The optimization problem can be written as
(s" 6*) = arg min [llyy ((1 + 5) +
(19)
2. Closed-form Solution For the j-th output channel, this reduces to:
(5), b7) arg min
E [(1s ((1 + s,)ū; + b,)))]
(20)
This equivalent to univariate linear regression with slope a, = + 8, and intercept by. Its closed-form solution is
aj = Cov(yj. 11)
6 = Pv; - -a313,
(21)
Var(ū,)
and therefore
87 o 1.
(22)
16

---

Published as a conference paper at ICLR 2026
3. Approximation under High Correlation Because y/j and J differ only by quantization noise, they are highly correlated. Thus, we approximate
Cov 81) yu og.
(23)
Substituting into the regression solution gives
Uj
1,
bj = Pv - (1 83)
(24)
4. Conclusion This yields exactly the mean-variance matching rule used in the main text, showing that the residual affine correction is not a heuristic adjustment but an unbiased MMSE-optimal estimator.
A.5 COMPRESSION CONDITION
In the current experiment, all methods are matched by weight bit-width (2 bits or bits), while the additional overhead of KBVQ-Mot (shared low-rank components and per-channel biases) only slightly increases the average number of bits per weight (by approximately 0.08 bits). In the table8] we report the actual storage usage and effective bits per parameter of all baselines, and provide comparisons under equal compression ratios.
Table 8: compression achieved by various techniques
Meihod
Compress ralio
Qwen A5MM0E-A2.7B
Qwen3-30BB-A3
Mixiral-8x7B
DeepseekV2-Lit
16
FPI6
0%
27.9GB
59.GGB
89.6GiB
29.3GB
2
RTN
87.5%
4.3GB
8.4GB
12.6GB
4,3GB
2.25
GPTQ
85.9%
4.7GB
9.3GB
13.5GB
4,7GB
MocQuant
87.5%
4.3GB
8.4GB
12.6GB
4.3GB
2.08
VQ
87%
4.3GB
8.7GB
13GB
4.4GB
2.08
KBVQ-MoB
87%
4.3GB
8.7GB
13GB
4,4GB
We provide a comprehensive quantification from three perspectives : offline computation overhead, inference-time overhead, and memory/parameter cost. These analyses complement the "Decoder speed test" Table in manuscript and present a complete view of the efficiency characteristics of KBVQ-MoE.
(I) Offline Computation Overhead (Does Not Affect Inference Latency)
KBVQ-MeE introduces additional one-time KLT-guided SVD decomposition during the cali- bration phase. This operation applied after stacking all expert weights: The cost of KLT-SVD is comparable to single forward pass the calibration set. occurs entirely offline during calibra- tion. introduces no additional inference-time latency or compute cost. Therefore, the additional computation introduced by KBVQ-MoE does not affect actual inference performance.
(2) Inference-Time Overhead (Nearly Negligible)
During inference, KBVQ-MoE introduces only: Two extra parameters per output channel (scale and bias). A lightweight per-channel affine correction after each expert's linear layer: Ycor (1 + s) (Wvoæ) + b. This involves only element-wise additions and multiplications, whose complexity is far lower than the original MatVec operation inside each expert MLP. Empirically, this affine correction accounts for less than 0.1 % of the expert forward FLOPs consistent with the 1.5-1.6x inference speedup reported in Table 6. Thus, KBVQ-MoE introduces minimal additional compute overhead during inference.
(3) Detailed Memory and Parameter Overhead Analysis
Let: Expert weight dimension: m Number of experts: n, IDRE rank ratio: k, VQ subvector length: Quantization bit-width: b.
The total bit consumption of KBVQ-MoE consists of four parts: Original FP16 parameters:1 6rmmd, IDRE shared low-rank components:16(m + In) min(m. 1)k, VQ codebook + indices:mllm
2b0 e n. BCOS scale/bias (per channel):16- In. Total compression
ratio: 16(m++I)) min(m,))) 1Dk m2bx vn4322n
17

---

Published as a conference paper at ICLR 2026
Example: Owenl.55-MoB-2.. 2.7B (gate proj weight). WWigght dimension: 5632 2048, Number of experts: 64, Original storage:5632 x 2048 x 64 x 16; bit = 1.38; GB, KBVQ-MoE configuration: 2-bit VQ. vector length 4. &
Total compressed storage: ((5632 2048 x 64) 2048/128
16)
(5632 2048 2 64 2 4 16 64) (2048 16 64), bit 0.18, GB.
This corresponds to an 87% compression ratio, equivalent to 2.08 effective bits per weight.
Through the analyses above, provide complete efficiency characterization of KBVQ across: Offline computation: acceptable one-time cost, no inference impact. Inference computation: negligible additional overhead (<0.11 % FL.OPs), Memory cost: clear and formula-based quantification (87% compression demonstrated).
A.6
4BIT ABLATION EXPERIMENT
Our primary focus in this work is to address the challenges of ultra-low-bit compression (e-g., 2-3 bits), where existing methods often suffer from severe accuracy degradation. In higher-bil bit regimes, standard scalar quantization (SQ) can already achieve strong performance, and the relative advantage of vector quantization (VQ) gradually diminishes. Moreover, quantization, whose codebook training via k-means clustering incurs a computational cost that grows exponentially with the target bit-widthh making VQ less attractive in practice for high-bit bit settings. For completeness, we additionally report the performance of KB at 4-bit Tabl which shows that method remains competitive in this regime, though ultra-low-bi compression is the main target of our design.
method is based vector
Table 9: 4bit ablation experiment
Model
Method
W2
ARC-C
ARC-E
PIQA
FP16
7.22
69.53
44.03
80.47
69.30
RIN
10.83
64.95
41.12
76.84
67.28
GPTQ
7.43
68.03
43.78
78.90
68.14
Owenl.5-MoE-A2.77
MocQuant
7.55
68.97
44.04
79.76
67.86
VQ
8.92
64.68
40.88
77.20
64.91
KBVQ-MoE
7.59
69.05
44.39
80.33
68.56
FP16
3.84
85.39
66.38
85.20
76.72
RTN
5.41
75.84
55.48
77.84
70.73
GPTQ
4,03
84.89
63.78
84.21
74.97
Mixtral-8x77
MoeQuant
4.12
83.88
63.70
84.20
75.02
VQ
4.27
84.18
64.21
84.93
74.81
KBVQ-MoE
4.16
84.97
65.04
84.84
74.17
18

---

Published as a conference paper at ICLR 2026
A.7
ALL EXPERIMENTS
Table 10: Comparison experiment of the QwenI.5-MoE-A2. model
Mathod
W2
ARC-E
ARC-C
HE
LAMBADA-O
LAMBADA-S
PIQA
16
FP16
7.22
69.53
44.03
77.26
71.28
64.62
80.47
69 30
68.07
RTN
638509
24.87
27.50
26.39
31.47
48.30
25.64
GPTQ
12.69
47.14
60.77
43.72
69.97
49.07
2
MoeQQiit
585542
34.54
35.09
33.55
12.03
8.05
59.08
58.21
34.64
VQ
26.95
58.16
34,64
63.35
28.90
22.92
71.82
55 (1)
47.84
KBVQ-MoE
9.61
65.66
39.59
64.52
67.16
77.20
65.11
62.78
116689
27.52
15.08
GPTQ
7.58
68.60
43.34
15.35
68.68
62.80
79.22
66.5
66.36
MoeQQuii
7.87
68.56
44.21
74,29
70.54
64.97
79.51
65.45
66.79
601.6
KBVQ-MoE
7.74
68.90
44.54
75.08
72.81
66.64
80.14
67.80
67.99
Table 11: Comparison experiment of the Qwen3-30B-A3B model
Mathod
ARC-E
LAMBADA-0
LAMBADA-S
FP16
6.70
79.25
36.40
79.60
61.70
62.91
80.30
70.45
70.23
KTN
768922
24.49
26.71
25.93
0
51.31
52.80
25,89
GPTQ
438.42
34.49
27.85
33.89
14.62
18.11
52.23
50.23
33,06
MoeQuant
26.59
25.38
VQ
115.30
33.08
24.49
35.74
6.79
7.35
56,86
49.95
30.61
KBVQ MoE
11.87
70.33
47.61
67.49
00.68
$4.34
76.66
66.45
63.37
RTN
68657
21.48
34.91
27.38
$1.80
25,84
GPTO
MoeQuant
24.85
30.49
37.77
31.49
18:38
35.11
55.82
52.23
37.32
VQ
18.72
57.83
40.87
63.23
40.44
32.51
71.82
$7.70
$2.06
KBVQ-MoE
9.26
77.27
$3.24
75.53
66.37
62.35
78.89
Table 12: Comparison experiment of the Mixtral-8x7B model
151
Method
W2
ARC-EE
ARC
HE
LAMBADA-0
LAMBALIA
PIJA
WI
AVG
FPI6
3,88
65.39
06.38
85.93
77.28
75.06
85.20
96.72
75..77
RTN
274952
28.41
26.09
25.27
GPTQ
5.69
72.35
48.89
76.93
68. 39
61.44
77.15
63.72
67.56
MoeQuini
13.43
49.85
38.93
40.12
22.98
18.94
60.38
19.91
40.16
VQ2
70.08
KBVQ&MoF
RTN
45136
26.39
25.50
25.38
0
0
$1.38
45.54
25.80
GPTQ
4.17
84.01
64.42
85.12
76.77
71.76
83.79
76 16
77.43
MoeQuant
VQ
5.52
81.65
82.12
72.97
67.44
81.88
73.48
73.98
KBVQ-Mol
4.07
84.89
63.05
84.99
80.50
75.53
83.90
75 63.61
78.33
Table 13: Comparison experiment of the V22-ite
V2-Lite model
LAMBADA-O
LAMBADA-S
PIDA
16
FPT6
5.92
76.22
48.9%
17.91
72.35
67.90
80.20
11.19
70.68
RTN
174653
24.12
26.45
26.30
0
49.89
19.09
25.12
GPTQ
MoeQuant
25893
26.94
25.40
25.45
50.14
51.29
25.59
VQQ
10.96
53.99
38.71
50.92
48.19
40.11
57.73
59.33
49.85
KBVQ-MoE
7,94
68-47
40.60
67.93
61.91
55.94
75.91
67.91
63.10
97.75
GPTQ
6.98
74.04
46.35
76.44
70.4
65.65
79.05
70.27
68.89
3
MoeQuant
7.52
70.25
42.91
71.19
70.04
65.32
76.84
67.83
66.34
VO
7.94
67.91
43.22
68.81
67.90
60.30
67.82
59.83
62.26
KBVQ-MoE
19

---

Published as a conference paper at ICLR 2026
A.8 KBVQ-MON ALGORITHM
Algorithm KBVQ-MoE: KITT-uided SVD with Bias-Corrected VQ for MoE Models
Input: Expert weight matrices {W()) Ji=1 input activations X, codebook size K, sub-vector length
d
Output: Quantized weights (WO) 12ml with bias correction parameters (s, b)
Pre-Process: KLT-guided SVD (Redundancy Removal)
1: Compute input covariance: Cx B- X E IRiexic
2: Figen-decompose: Cx U ALL-AKITUKL
3: Input-coherent basis: Ux = AKTT
4: for i = 1 to n do
5: Project expert weights (right projection): W = W
6: end for
7: Stack all experts (row-wise): W- [W().... ..... WWin) E xixo
8: Product SVD: W (UEVT)T
>V EVERRoog R U E Rexic
9: Select top-k: UR = UI.1ok), 1/2 Vi.1ok) Σg E1i,,
Partition Vi by experts: V (E&V(), En.;; V each each V(()
E Booc
11: for = 1 to n do
Define shared output loader: Ustare + Ux'Ux
13: Shared part in projected basis: Wil)) = (Ushare (Z
share
X ic
14: Map back to original input space: W(1)
share = (1(s)
share
Special (residual) part: W((t) quant (i()) W (f)
share
16:
end for
Quantization: Vector Quantization of Special Part
17:
for i 1 to do
Split W(() quant into sub-vectors {2} Initialize codebook via K-means++
18:
19:
20:
Train VQ codebook C = {cI,..., CKK by k-means
21:
for each sub-vector do
22:
Assign index: q = arg min, ||2 c/
Replace: 2q = Cq
24: end for
Form quantized special part: W-(
quant
end for
Post-Process: Bias Correction
27: for i = 1 to do
Define quantized weights: W) = w
share + w-(i)
28:
quant. VQ
29:
Estimate per-channel statisties from calibration data:
My, By for original outputs y W x
Jlg, Fg. for quantized outputs ý = ())
30:
Compute correction parameters:
8j og 1, by = Mv, (1 sy)
31:
Corrected output: Yeor = (1 + 8) 2 (WV(0)) +
32:
end for
33: return Usbare, Vk. C, (s,
A.9 ABLATION OF IDRE AND BCOS
To further clarify the contribution of each component within the IDRE and BCOS modules, we conduct fine graindd component-wise ablations 2-bit Oweenl.5MMoEA22.5 This directly
20

---

Published as a conference paper at ICLR 2026
addresses the reviewer's concern regarding the lack of quantitative analysis for the internal steps of each module.
(a) Effect of the KLT step in IDRE. IDRE consists of (i) input-driven KLT transform and (ii) subsequent SVD on the stacked expert weights in the KLT space. To isolate the effeet of the KLT step, we compare the full IDRE against variant that performs SVD directly the original stacked weights (without KLT alignment). The results at 2-bit are summarized in T4b
Table 14: Ablation of KLT in IDRE on Qwenl M-..--5--- (2-bit).
Model
IDRE
W2
HE
WI
w/o KLT
11.77
35.89
63.01
QwcnL.5-MBE-A2.7
KLT
9.61
39.59
65.11
(b) Effect of mean and variance correction in BCOS. BCOS performs channel-wise bias correction using both the mean and variance of the channel outputs. To disentangle their respective roles, evaluate four variants: (i) no correction, (ii) mean-only, (iii) variance-only, and (iv) full mean+variance correction. The 2-bit results on Owwnll5-MMoE-A2.777
given in Tab [5
Table 15: Ablation of mean and variance correction in BCOS Owenl.5-MoE-A2.. (2-bit).
Model
Mean
Variance
W2
HE
WI
X
x
11.03
35.18
63.49
x
11.01
35.35
63.40
Owe.l.5-MoE-A2.7B
V
10.38
36.70
64.23
9.61
39.59
65.11
A.10 COMPARISON OF MOE COMPRESSION METHODS
The core designs of D2-MoE, SubMoE, and EAC-MoE all revolve around "expert layer structure optimization", which belongs to different technical branch of MoE compression compared to KBVQ-MoE's "weight quantization". The specific differences can be summarized as follows:
SubMoE: Merges similar experts through subspace clustering and extracts cross-expert shared low-rank features via SVD to reduce the number of experts. Essentially, it is structural compression method combining "expert-level pruning low-rank decomposition" The compression ratio is mainly determined the expert merging ratio (e.g., 50% means merging half of the experts).
D2-MoE:
Decomposes expert weights into a "shared base matrix + expert-specific delta matrix" and performs low-rank approximation the delta matrix reduce the number of parameters. Essentially. it is a structural compression method combining "weight de- composition low-rank approximation". The compression ratio is determined by the rank truncation ratio of the delta matrix (e.g., retaining 60% of the rank).
EAC-MoE: Starts from "expert selection behavior" and jointly models two operations: quantization and pruning. It proposes QESC (Quantization with Expert-Selection Calibra- tion), which explicitly calibrates the router during low-bit quantization to mitigate expert selection bias caused by quantization noise. also proposes PESF (Pruning based on Expert- Selection Frequency). which prunes "cold experts" using their actual selection frequency to further improve inference speed and reduce memory usage while minimizing accuracy loss. Thus, EAC-MoE can be regarded as "routing-aware quantization expert pruning hybrid compression scheme".
KBVQ-MoE (ours): Does not change the number or structure of experts. First, extracts cross-expert shared low-rank subspaces in the input KLT space via IDRE to explicitly remove redundancy, then performs vector quantization on expert-specific residuals. Essentially, is "MoE structure-aware weight quantization method". The compression ratio is determined
21

---

Published as a conference paper at ICLR 2026
by the quantization bit-width (2-3 bits) and codebook reuse efficiency, achieving 80%-90% memory compression under the same structural constraints.
Overall, MoE are more inclined to directly reduce the number of parameters (removing experts reducing rank), EAC-MoE follows hybrid route of structural pruning + quantization, while KBV focuses on achieving high compression ratios through structure-award low-bit quantization while preserving the original expert structure. Different routes may exhibit distinet trade-offs terms of "expert specificity preservation", "inference speed", and "implementation complexity", which need be characterized through empirical comparisons.
To ensure fairness, we uniformly Mixtral-8x7B as the baseline model (retaining the original expert structure: 8 experts, 32 layers) and adopt identical experimental settings:
Fair Comparison Experiments: Settings and Results.
Evaluation tasks: WikiText2 (language modeling. metric: perplexity (PPL), lower is bet- ter), ARC-Challenge (ARC_C), ARC-Easy (ARC_E), WinoGrande (WinG) (all zero-shot inference, metric: accuracy, higher better):
Compression ratio definition: Memory saving uniformly calculated as "the ratio of compressed parameter storage to the original FP16 model" to avoid misunderstandings caused by inconsistent definitions of "compression ratio" across different methods;
Experimental environment: NVIDIA RTX A100 GPU, PyTorch 2.1, evaluation tool: LM- Evaluation- Harnesss (v0.4.0).
The comparison results shown in the Tab 16 (Note: ARC_C, ARC_ E, and WinG values accuracy, retained to two decimal places for precision):
Table 16: Ablation KBVQ-Mot and other MoE. compression methods
Method
Memory saving
WikiText2 (PPL)
ARC_ C
ARC E
WinG
Sub-MoE
50%
6.97
0.45
0.75
0.72
D2-MoE
60%
6.46
0.38
0.72
0.71
EAC-MoE
84%
4.58
0.55
0.81
0.75
KBVQ-MoE (ours)
87%
4.07
0.63
0.85
0.76
In the experiments on Mixtral-8x75 significant differences exist among the three compression approaches: KB KBVQ-MoE (quantization compression) achieves an 87% memory saving. which is significantly higher than that of Sub-MoE (50%) and D2-MoE (60%), and slightly higher than that of EAC-MoF (84%). Moreover, it performs optimally across all tasks: its Perplexity (PPL.) on and 0.51 lower than that of EAC-MoE;
WikiText2 2.39-2.90 lower than that Sub-MoE/D2-Mo its accuracy ARC_ C is 18-0.25 higher than that of Sub-MoE/D2 of EAC-MoE. These results confirm that KBV can well retain the expressive ability of MoE experts under high compression ratio (>80%).
and 0.08 higher than that
EAC-MoE (hybrid compression), which combines routing calibration and expert pruning based selection frequeney, outperforms Sub-MoE and D2-MoE (structural compression) in perfor- mance. However, KBVQ-MoB still shows better performance under a similar compression ratio and does require modifying the expert structure. Additionally, KBVQ-MoF preserves the original expert topology, achieving an accuracy of 0.63 on complex reasoning tasks (e.g.. ARC_ far exceeding the 0.38-0.45 of Sub-MoE/D2-Mott and the 0.55 of EAC-MoE. It also features greater flexibility in deployment and migration: it can be directly adapted to various MoE LLMs (e.g. Qwen, DeepSeek) with only one round of offline calibration and codebook training, demonstrating prominent comprehensive advantages.
A.11 CHALLENGING BENCHMARKS
We carried ablation studies between method and diverse baselines more arduous tasks shown on Tab[I7] Main tasks include: MMLU(Wang et al. 2024), MathQA(Aminn et al. 2019).
GSMbK et al. 202T), Human et al. 2021)
22

---

Published as a conference paper at ICLR 2026
Table 17: more challenging benchmarks
Model
Bits
Method
MMLU
MathQA
GSMKK
HymanFval
16
FPI6
59.60
37.55
62.55
32.32
GPTQ
26.94
19.33
15,42
13.88
Owen1.5-MoE--A2.7
MoeQuant
34.75
22.42
23.21
18.12
2
VQ
48.79
28.34
46.51
28.03
KBVQ-MoE
52.16
30.75
$6.39
30.11
16
I-P6
79.5
58.53
85,44
71.5
GPTQ
42.11
34,9
49.34
50.12
Qwen3-30B-A3B
MoeQuant
45.1
39.86
57.38
55.32
VQ
60.94
49.87
64.83
61.2
KBVQ-MoF
68.19
50.11
78.92
66.89
16
FPI6
44.60
31.49
20.16
26.83
GPTQ
28.48
18.82
11.26
14.87
DeepSeckMoE-1oBB
MoeQuant
34.82
24.93
12.88
16.95
VQ
30.78
24.49
13.55
17.89
KBVQ-MoE
41.39
28.97
17.32
22.62
16
FPI6 70.50
42.41
65.88
32.93
GPTQ
40.80
22.96
5.89
9.97
Mixtral-xx8B
MocQuant
49.39
27.11
20.67
14.74
2
VQ
55.84
30.83
23.21
KBVQ-MoE
61.11
34.8
57.86
29.9
A.12 LIMITATIONS
Despite the strong empirical performance of KBVQ-MoE ultra-low-bit quantization of decoder- only MoE large language models, the method exhibits several limitations that highlight potential directions for future work.
(1) Dependence on empirically selected SVD truncation rank. The truncated rank (k) in IDRE currently chosen based on an empirical balance between reconstruction fidelity and storage overhead (e.g., & across evaluated models, KBVQ-MoE does not yet include an adaptive mechanism to automatically determine the optimal rank for different MoE architectures, task regimes, input statistics. This limits its ability to fully optimize the trade-off between redundancy removal and quantization efficiency across diverse settings.
of the full rank in our experiments). Although this choice works robustly
(2) Limited validation beyond decoder-only architectures. Our experiments focus primarily on decoder-only MoE models such as Qwen-MoE and Mixtral-8x7B. While this allows for clean examination of the effects IDRE and BCOS, the method has not yet been systematically evaluated mncoerrddcode MoE models multimodal MoE architectures. These settings may require modified input-statistics estimation or revised forms of redundancy modeling. Extending KBVQ-MoE bidirectional cross-modal MoE structures therefore important for future exploration.
(3) Lack of evaluation extreme bit regimes. KBVQ-MoF achieves near-FP16 accuracy at 2-3 bits, but has not been tested in more extreme quantization regimes such as 1-bit binary quantization or hybrid bit-widths (c.g., 1.5-bit). These regimes pose substantially greater information bottlenecks, and may require enhanced bias-correction mechanisms, more expressive codebook structures, or hybrid compression strategies to remain effective.
Together, these limitations delineate the boundaries of the current investigation and point toward promising extensions of KBVQQ KBVQ-MoE in future research.
23


Reviews:
[{"review_id": "t810g73iMx", "reviewer": "Reviewer_7GqJ", "summary": "This paper presents KBVQ-MoE, a vector quantization framework tailored for ultra-low-bit compression of Mixture-of-Experts (MoE) large language models (LLMs). The study is motivated by the substantial performance drop observed when conventional quantization methods are directly applied to MoE architectures. In particular, KBVQ-MoE mitigates expert redundancy and output bias through input-driven redundancy elimination and bias-corrected output stabilization mechanisms.", "questions": "Given that the calibration set contains only 256 samples from RedPajama, could the authors provide ablation results on calibration size to verify its representativeness for computing reliable KLT statistics and bias correction factors? Additionally, how do the authors ensure that no potential data leakage occurs during calibration?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is clear and convincing, highlighting MoE-specific issues of expert redundancy and output bias.\n\n2. The proposed KBVQ-MoE is validated on several representative MoE architectures, demonstrating consistent improvements.", "weaknesses": "1. While IDRE and BCOS are ablated individually, there is no fine-grained study of codebook size sensitivity.\n\n2. Lack more advanced or concurrent MoE-aware compression baselines (e.g., D2-MoE, SubMoE mentioned in related work).\n\n3. The evaluation of computational efficiency is insufficient. The paper only reports a simple “Decoder speed test” in Table 6, without providing detailed analysis of computational or memory overhead.\n\n4. The core motivation of the paper lies in the claim that redundancy elimination and bias correction help stabilize expert output distributions; however, the supporting evidence (e.g., Fig. 2–3) is insufficient. It would be more convincing if the authors compared other methods reported in Table 1 to quantitatively validate the claimed effect.\n\n5. The paper’s presentation lacks rigor in notation and consistency. For instance, the dimension oc in Step 2 is undefined, and the superscript in Equation (3) for the routing expert is unexplained. There are also typos (e.g., uantization → quantization) and inconsistent use of MoE/moe.\n\n6. The paper does not explicitly discuss the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zcX0jBopBv", "reviewer": "Reviewer_XsNC", "summary": "The paper proposes a vector quantization approach targeted for mixture of expert layer. The idea is to extract redundant representation across experts and keep it in higher precision while expert specific components are vector quantized. Within vector quantization of expert specific components, scaling and bias is applied to improve quantization. The paper is well written and easy to understand. The techniques proposed are intuitive and I appreciate the authors providing actual hardware speedup numbers. The paper is missing some recent non linear quantization baselines and comparison at iso-compression.", "questions": "1. How does this approach compare with ResQ (https://openreview.net/pdf?id=4qIP1sXcR1)? Although ResQ does activation quantization as well and integer quantization of weights, it also uses eigen value decomposition to isolate high precision components. How does this approach compare with using IDRE proposed in this work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Paper is easy to understand and well written.\n2. Technique proposed is intuitive.", "weaknesses": "1. The paper is missing comparison with recent non linear quantization baselines : VPTQ (https://arxiv.org/abs/2409.17066), AQLM (https://arxiv.org/pdf/2401.06118), QUIP (https://arxiv.org/pdf/2307.13304), QUIP# (https://arxiv.org/pdf/2402.04396), SqueezeLLM (https://arxiv.org/pdf/2306.07629), GPTVQ (https://arxiv.org/pdf/2402.15319),  etc.\n2. Among the baselines presented, the compression achieved by various techniques is missing. \n3. Iso-compression results are missing.\n4. Evaluation on complex tasks is missing : math understanding, coding, reasoning, long context abilities, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KcKVjHebNl", "reviewer": "Reviewer_TstW", "summary": "This paper proposes KBVQ-MoE, a vector quantization framework designed specifically for compressing Mixture-of-Experts (MoE) language models. The method addresses two key challenges: (1) redundant representations across experts that waste codebook capacity, and (2) cumulative output bias from quantization errors that get amplified through expert aggregation. The approach combines Input-Driven Redundancy Elimination (IDRE), which uses KLT-guided SVD to extract and preserve shared components at full precision, with Bias-Corrected Output Stabilization (BCOS), which applies vector quantization only to expert-specific weights and corrects distributional shifts via channel-wise affine transformations. Experiments on models like Qwen and Mixtral show strong performance at 2-3 bits, with 3-bit Qwen1.5-MoE-A2.7B achieving near-FP16 accuracy.", "questions": "1. typo: Qwen1.5-Moe-A2.7B->Qwen1.5-MoE-A2.7B", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper makes a novel contribution by adapting vector quantization specifically for MoE architectures. The identification of expert redundancy and amplified quantization bias as key bottlenecks is insightful, and the KLT-guided SVD approach creatively aligns weight decomposition with input activation statistics.\n2. The technical approach is sound with theoretical justifications provided in the appendices. The experimental evaluation is comprehensive, covering multiple MoE models with thorough ablations. Results show consistent and meaningful improvements at ultra-low bit-widths (2-3 bits), where baseline methods struggle significantly. The modular design demonstrates compatibility with existing VQ methods, increasing practical impact.", "weaknesses": "1. The paper mentions \"negligible\" computational overhead but provides limited quantitative analysis. How long does the KLT-SVD calibration take compared to standard VQ? What is the actual inference-time cost of the channel-wise bias correction operations? These practical considerations matter for deployment. Could you provide some results on this, like time cost of quantization method.\n2. The baseline methods, especially MoEQuant, show surprisingly poor performance at 2-bit in Table 1 (e.g., W2 of 583542 for Qwen1.5). More discussion on this failure would help.\n3. The choice of truncated rank k=n/128 appears empirically driven from Table 4 but lacks theoretical justification. While the ablation shows diminishing returns beyond this point, why this specific ratio is optimal across different models and tasks remains unclear. The paper would benefit from analysis connecting rank selection to properties of the expert weight matrices or input distributions.\n4. The evaluation relies on simple zero-shot reasoning tasks (ARC, HellaSwag, PIQA, etc.) that may not fully capture model capabilities. Including more challenging benchmarks like MMLU, MATH, and code generation tasks (MBPP, EvalPlus) would better demonstrate the method's effectiveness across diverse domains.\n5. Table 1 only shows 2-bit and 3-bit results. Adding 4-bit and 8-bit comparisons would provide a more comprehensive results, especially since 4-bit quantization is common in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lZfNHMBBNq", "reviewer": "Reviewer_v8EJ", "summary": "This paper introduces a novel quantization method, KBVQ-MoE, designed for MoE-based large language models. The method first applies the KLT and SVD to extract dominant weight structures shared across experts, thereby reducing redundancy and improving codebook utilization. Building on this foundation, it employs vector quantization for expert-specific representations and further proposes a channel-wise affine compensation module to refine the mean and standard deviation of the quantized outputs. Extensive experiments demonstrate superior performance across multiple datasets and LLM architectures.", "questions": "1. Is the IDRE technique applied exclusively to router experts or to all experts? The notation in Eqs. (1) and (3) suggests it concerns router experts, while the workflow in Eq. (2) seems to encompass all experts. Clarification on whether the symbol $n$ refers to different concepts across equations would be helpful.\n2. What is the rationale for maintaining the shared structure at full precision while quantizing only the expert-specific weights? Why not quantize the shared weights as well?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Figures 2 and 3 provide highly illustrative examples that effectively support the paper’s claims on redundancy extraction and channel-wise bias correction.\n2. The overall motivation is solid and clearly articulated. The proposed method is both conceptually sound and straightforward, making it easy to understand and reproduce. Moreover, it successfully addresses two critical challenges outlined in the introduction.\n3. Table 5 convincingly demonstrates that the proposed IDRE and BCOS modules can be seamlessly integrated into other MoE quantization approaches, highlighting the method’s flexibility and potential for broad applicability.", "weaknesses": "1. Some mathematical symbols should be revised (e.g., matrices should be boldfaced). Additionally, minor grammatical and formatting issues should be corrected, and certain technical details could be described more clearly.\n2. The contribution of each component within the IDRE and BCOS modules remains unclear due to the lack of quantitative results and in-depth analysis in the ablation study.\n3. The comparison with other MoE-based compression methods, such as EAC-MoE, D2-MoE, and SubMoE, which are illustrated in the related work, should be included for a more comprehensive comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqosa63hLR", "reviewer": "Reviewer_Dyjg", "summary": "The paper proposes KBVQ-MoE, a post-training vector quantization framework for MoE LLMs combining a KLT-guided SVD to remove cross-expert redundancy (IDRE) with a lightweight channel-wise affine correction (BCOS) to stabilize outputs after VQ. The key idea is to map expert weights into an input-coherent basis via KLT, extract dominant shared components with SVD and keep them in full precision, then quantize only the expert-specific (i.e., non-redundant) representations and correct distributional shifts by mean–variance matching. The authors claim the affine correction is “not a heuristic adjustment but an unbiased MMSE-optimal estimator.”", "questions": "* How is `Qwen3-30B-A3B ` evaluated? Specifically, did you enable \"reasoning\" of this model? I would love to see the quantization results on a reasoning LLM.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* This paper is clearly motivated by two obstacles in MoE VQ, including redundant representation among experts and cumulative outputs bias.\n\n* Empirical result looks promising, particularly at low bits.\n\n* The method acts as an easy-to-use plugin that improves multiple VQ baselines and yields speedups.", "weaknesses": "* Evaluation tasks are limited. The evaluated tasks are normally for pre-trained models, but some of the models are post-trained models (e.g. Qwen3). I would love to see some post-training benchmark (e.g. MMLU, AIME24, HumanEval) results on the Qwen3 model.\n\n* Some typos: (line 756) “A.5 ALL EXPRIMENTS” → “All Experiments.”; (line 81) \"uantization\" -> \"quantization\"; (line 105) \"non-redundan\" -> \"non-redundant\"\n\n* Some of the related works on MoE quantization can be further discussed or compared with, e.g. [1] [2] [3].\n\n(I'm more than happy to increase my score if my questions are adequately addressed)\n\n[1] Kim, Y.J., Fahim, R. and Awadalla, H.H., 2023. Mixture of quantized experts (moqe): Complementary effect of low-bit quantization and robustness. arXiv preprint arXiv:2310.02410.\n\n[2] Li, P., Jin, X., Tan, Z., Cheng, Y. and Chen, T., 2024. QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts. arXiv preprint arXiv:2406.08155.\n\n[3] Duanmu, H., Li, X., Yuan, Z., Zheng, S., Duan, J., Zhang, X. and Lin, D., 2025. MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design. arXiv preprint arXiv:2505.05799.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)