CodeQuant: Unified Clustering and Quantization for Enhanced Outlier Smoothing in Low-Precision Mixture-of-Experts
Authors: Xiangyang Yin
Xingyu Liu
Tianhua Xia
BO BAO
Vithursan Thangarasa
Valavan Manohararajah
Eric Sather
Sai Qian Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture-of-experts
quantization
Primary area: foundation or frontier models, including LLMs

Abstract: Outliers have emerged as a fundamental bottleneck in preserving accuracy for low-precision large models, particularly within Mixture-of-Experts (MoE) architectures that are increasingly central to large-scale language modeling. Under post-training quantization (PTQ), these outliers induce substantial quantization errors, leading to severe accuracy degradation. While recent rotation-based smoothing techniques alleviate the problem by redistributing outlier magnitudes, residual errors remain and continue to impede reliable low-precision deployment.

In this work, we tackle this challenge by introducing CodeQuant, a unified quantization-and-clustering scheme that contains smoothing activation outliers via learnable rotation and absorbing weight outliers into fine-tuned cluster centroids for MoE. This design reduces the influence of extreme values by fitting them within cluster centroids, thereby lowering quantization error while maintaining expressive capacity. Coupled with a dedicated kernel design for GPU and CPU, CodeQuant achieves up to $4.15\times$ speedup while delivering significantly higher accuracy than state-of-the-art quantization approaches across diverse MoE models. Our results highlight CodeQuant as a promising direction for efficient and accurate deployment of MoE-based large language models under low-precision constraints. Our code is available at https://github.com/SAI-Lab-NYU/CodeQuant.

Published as a conference paper at ICLR 2026
CODEQUANT: UNIFIED CLUSTERING AND QUANTIZA- TION FOR ENHANCED OUTLIER SMOOTHING IN Low- PRECISION MIXTURE- OF-NPERRTSS
Xiangyang Yin-* Xingyu Liu Tianhua Xia
Bao Vithursan Thangarasa"
Valavan Manohararajah Eric Sather Sai Qian Zhang
Courant Institute of Mathematical Sciences, New York University "Tandon School of Engineering, New York University
"Cerebras Systems Inc.
(shawn. yin, x15444, tx856, sai. zaanggaannnnne Bnny.. edu (bo. bao, vithu, valavan, eric. sather @cerebras net
ABSTRACT
Outliers have emerged fundamental bottleneck in preserving accuracy for low- precision large models, particularly within cexture-of-Experrs (MoE) architectures that are increasingly central to large-scale language modeling. Under post-traii post-training quantization (PTQ), these outliers induce substantial quantization errors, leading to severe accuracy degradation. While recent rotation-basee smoothing techniques alleviate the problem by redistributing outlier magnitudes, residual errors remain and continue to impede reliable low-precision deployment.
In this work, we tackle this challenge by introducing CodeQuant, unified aaaatizztion-and--luuterrrgg scheme that contains smoothing activation outliers via learnable rotation and absorbing weight outliers into fine-tuned cluster centroids for MoE. This design reduces the influence of extreme values by fitting them within cluster centroids, thereby lowering quantization eтror while maintaining expressive capacity. Coupled with dedicated kernel design for GPU and CPU, CodeQuant achieves up to 4.15x speedup while delivering significantly higher accuracy than state-of-the-an quantization approaches across diverse MoE models. Our results highlight CodeQuant
promising direction for efficient and accurate deployment MoE-ban based large language models under low precisocn constraints. Our code is available at hhttt:::: / hit. com/ SAI Lab- NYO
INTRODUCTION
Mixture-offEExpert (MoE) has emerged as one of the most effective paradigms for scaling large language models (LLMs). By activating only subset of experts for each input token. MoE in- troduces conditional computation, allowing different experts to specialize in distinct linguistic or multimodal patterns. This specialization enables MoE-based models to achieve superior performance across diverse tasks. Consequently. MoE architectures have been adopted in many state-of-the--e LLMs (bdiin al.) [2024] Yang al. 2025 DeepSeek-Al et al.) 2024). Despite these advantages, MoE models still carry substantial computational and system-level costs. Although only a fraction of experts active per token, the total parameter size is extremely large, leading high memory requirements and increased communication overhead during distributed training and inference. These factors increase processing latency and pose serious challenges for real-worke deployment.
To address these costs, low- fow-precision quantization has become widely adopted strategy. By representing weights and activations with fewer bits, quantization substantially reduces memory footprint and improves computational throughput. Recent hardware innovations further accelerate this trend: NVIDIA's Hopper (NVIDIA Corporation| (022b) and Ada GPUs (NVIDIA Corporation 2022ab natively support FP8 arithmetic, while the Blackwell series extends support to FP4. These developments provide strong foundation for efficient MoB deployment with low precision. However,
"Equal contributions.

---

Published as a conference paper at ICLR 2026
CodeQuant Framework
Stage 1: Activation-oriante Outller
Stage 3: Adaptive Weight
Stage 4: Kernel Design and LUT-based System Implementation
MoE FFN
Clustering and Centroid Finetuning
Smoothing
aczamin [X Q(X2))
Classiting
X
x
Feedrorward layar (FFN)
Uiainen
For blockwise setting
Self-ateetion layer (SA)
Stage Permutation-invariant outlier grouping
A
Wu
Wo
SA
Figure 1: Overview of the CodeQuant framework. The left panel illustrates the target architectures, including MoE FFN and Self-Attention blocks. The right panel depicts the four-stage calibration and deployment pipeline: Stage applies Icarnable rotations to smooth activation outliers; Stage 2 permute weight for optimized distribution; Stage 3 introduces clustering fine-tune mechanism to align with objective; and Stage 4 deploys the quantized model using specialized LUT kernel.
quantizing MoE architectures remains challenging due to the prevalence outliers (Dettmers et al. 2022: Sun et al. 2024). Large-magnitude activations expand the dynamic range, leading to severe quantization errors and significant accuracy degradation under post-training quantization (PTQ), particularly low-bit settings such 4-bit quantization. While recent nutlier-smoothee methods (Xiao et al. 2024 Ashkboos et al. (222)) alleviate the issue, residual errors persist and continue to hinder reliable now-precision deployment.
In parallel, bodebook-basedd approaches such as clustering have emerged as compelling alternative to uniform quantization. By mapping weights or activations to compact set of representative centroids, clustering mitigates quantization error and effectively handles outliers, extreme values can be absorbed into centroids rather than expanding the overall dynamic range. Beyond its algorithmic robustness, clustering is also hardware-efffcieett: lookup table (LUT) implementations enable rapid centroid mapping and streamlined memory access, making well suited for large-scale deployment. Notably, several commercial accelerators have already adopted such designs, including Apple's Neural Engine (Inc. 2024a) and Arm Ethos-U U (Inc.) 2020). The sparsity indexing mechanism in the Cerebras Wafer-Scale Hngiine (Inc.) 2024b) further enables gerooperformance LUT implementation. Collectively, these developments underscore clustering as a practical, hardward aeddwarr-alggeed solution for LUT-driven quantization.
Motivated by the challenge of activation outliers and the efficiency potential of LUTs, we present CodeQuant, unified codebook-basee clustering and quantization framework for low precision MoE models. CodeQuant exploits clustering robustness and LUT-based quantization to improve performance without any runtime overhead. Our contribution can be summarized as follows:
We first introduce crrivatiio-orientti Outlier Smoothing (AOS), which suppresses activation outliers through rotation matrix adjustment, effectively relocating them into the weight space.
We then propose Adaptive Weight Clustering with Centroid Finetuning (ACCF) and Permu- tation Invariant Outlier Grouping (POG). which substantially reduce weight quantization eтror even in the presence of significant outliers.
We develop a LUT kernel to demonstrate improvements in hardware efficiency, Across Phi- Mini-MoE-Inntrune Dwen3-30B-Al DeepSeek-V2-Lite V2-Lite and Mixtral 8x7B, CodeQuant consistently accelerates inference, lowers memory footprint, and preserves accuracy.
BACKGROUND AND RELATED WORK
2.1
OUTLIER IN LLMS
Activation outliers have been widely recognized as major obstacle to effective quantization of LLMs since they expand the dynamic range and induce severe activation quantization errors. Prior
work (Dttteers et al. 2022 Sun et al. 2024 An al. 2025) highlights predominant forms:
2

---

Published as a conference paper at ICLR 2026
channel-wise outliers and massive activations. Moreover, residual connections exacerbate the problem by propagating outliers across layers and amplifying the adverse effects (Guo et al. (024).
Mixture-of -ffpprrs (MoE) LLMs are likewise affected by the outlier problem. Prior studies MoE (Sun et al. 2024 Lo ct al. 2023) report that massive activations frequently arise in the hidden states between decoder layers and further propagated through residual connections, compounding their impact across subsequent layers. More recently, the notion of super experts has been introduced (Su al. 2025), revealing an additional source of large-magnitude outliers specific to MoE architectures.
2.2
OUTLIER AWARE QUANTIZATION
Prior efforts on LLM quantization have pursued two directions for addressing the outlier problem. The first explicitly isolates outliers and applies mixed-precision quantization (Dettmers et al. [022|022 Kim
et al.
2024. van Baalen et al. 2025 Huang et al. 2025 Dong & Zhang 2025: Liu & Zhang 2024).
ensuring that extreme values are preserved at higher precision. The second seeks to mitigate outliers through invariant matrix transformations. Within this line, one strategy redistributes outliers between activations and weights (Xiao et al., 2024 Lin et al. 2024b Xiang & Zhang, 2024). SmoothQuant (Xiao al. 2024) is representative work, which jointly smooths activations and weights to mitigate their impact. The other strategy is to smooth activation via orthogonal transformation. QulP (Chee et al. 2024) and QuIP# (Tseng al. 2024) initiate this line of work by leveraging rotation transformations to mitigate outliers. Building on this idea, QuaRot (Ashkboos et al. (0224) applies rotation activations for outlier-free inference. DuQuant (Lin al., 2024a) combines permutations for dual handling of outliers. SpinQuant (Liu et al.) (025) introduces learnable orthogonal rotation matrices that are optimized during post-training quantization, and subsequent work such as OSTQuant (Hu et al. 2025b) further incorporates a KL-bbased objective to fine-tune these rotations together with smoothing parameters. QSVD Wang et al. (2025) combines low-rank decomposition with rotation-bast based quantization, achieving superior accuracy and improved hardware efficiency.
In the context of weight quantization, most existing works nonetheless adopt uniform quantization schemes such as GPTQ (Frantar et al. 2022) and AWQ (Lin et al. 2024b), even though weight distributions in practice are far from uniform. To address this mismatch, early studies (Dettmers et al.)
2023 Yoshida 2023] (Blumenberg et al. (025) introduce quantile-based non-uniform quantization, codebooks. Meanwhile, SqueezeI.LMM (Kim et al., 2024) demonstrates that dynamic non-uniform quantization better adapts to the empirical weight distribution in LLMs. Building on carlier clustering- based compression techniques (Han al., 2016; Xu al.) 2018), SqueezeLLM integrates K-means clustering into LLM quantization, yielding more robust results. Moreover, efficient algorithms for low-pre low-precision MoE remain largely underexplored. MoEQuant (Hu et al.) (2025a) demonstrates that directly applying conventional quantization methods to MoE models yields suboptimal results, underscoring the importance accounting for expent affinities.
leveraging the normal distributions assumption of weights to construct information-opiin
2.3
LUT AND HARDWARE IMPLEMENTATION
General Matrix Multiply (GEMM) with clustered multiplicands requires LUT support for efficient deployment. Without hardware-friendee LUTs, centroids must be stored as floating-point values and reloaded during computation, incurring significant overhead. Studies on both CPUs and GPUs address this by exploring LUT-based execution to bridge non-ume uniform quantization and practical deployment. On CPUs, DeepGEMM (Gajji et al. 2023) uses LUT--driven kernels for llwr precision CNNs, LUTTN (Lin et al. 2024c) optimizes memory use via hyperparameter tuning, and T-MAC (Wei et al., 2025) reformulate mixed- preccssnr GEMM as table lookup for LLM inference. On GPUs, LUT-GEMM (Park et al. 2024) and FLUTE (Guo et al., (025) design optimized kernels to minimize unpacking overhead, while LUT Tensor Core (Mo et al. 2025) integrates LUT primitives into censor-core pipelines through software- hardware co-design.
3 METHODOL.OGY
The overview of CodeQuant is shown in Figure I which comprises three stages. In the first stage, we apply Antivation-Orientte Outlier Smoothing (AOS) exclusively to the input activations, effectively mitigating activation outliers (Section 3.1. In the second stage, we optionally employ
3

---

Published as a conference paper at ICLR 2026
Permutation Invariant Outlier Grouping (POG), which reorders the columns of the weight matrix to better support the subsequent clustering process (Section(..3/ Stage three introduces Adaptive Weight Clustering and Centroid Finetuning (ACCF), which identifies optimal groupings and refines centroids to minimize output difference (Section (.2). Finally, the resulting MoE is deployed using a LUTT-based system, achieving superior computational efficiency (Section 3.4.
3.1 ACTIVATION-ORIENTI OUTLIER SMOOTHING
As illustrated Figure 2. the rotational method introduces an additional matrix R E g4 xdin applied to the activation the Self- Aeff-Attention (SA) and Feed-Forward Net- work (FFN). The SA blocks in MoE models share the same structure as those in standard LLMs, and therefore rotation transformation is invariant discussed in (Ashkboos et al., 20024). The incorporation rotational matrix R within the FFN layers is illustrated in Figure 2 Al- though an FFN in MoE consists of router and multiple experts, the router is simply a linear layer and each expert is structurally identical to standard FFN. As a result, the MoE module is invariant to rotation transformations. To avoid introducing any online computation, we adopt an SA-style design in which the router and all experts share the same rotation matrix. Taking a single expert an example, this rotational invariance can be expressed as follows:
E N in both
Router R' Wmuner
Expert Expert
R
YR
Expert N
wns
R
Wga
SiLU
W
R
Wup
Expert
Figure 2: FFN layers within MoE is applied with rotational matrices for outlier smoothing.
(6(X,RR W gate)) @ X,RR Wup) W down (o(XW gate) C W down
where o6-) denotes a nonlinear activation function (e.g., SiLU) and X, E RRaxd. denotes the subset of tokens assigned that expert.
Although random rotation improves quantization, replacing weight quantization with clustering reveals activation quantization as the dominant bottleneck. To address this, we fine-tune the rotation matrix It via the Cayley transform to smooth activations (Nishimori & Akaho 2005; Li et al. 2020). Specifically, for any matrix M Rd xdsa, where dnn denotes the model's s hidden dimension, we first extract its skew-symmetric component and then derive an orthogonal matrix via the Cayley transform:
S 2 1 M²)
(I-
By parameterizing the matrix M. this construction guarantees that the matrix R remains orthogonal while keeping the process fully differentiable. AOS employs learnable rotation matrices to minimize the quantization error of rotated activations, defined as Xn R.. By minimizing the quantization error of rotated activations, the rotation explicitly reduces the influence of outliers on the activation side, leaving the weights accommodate more of the variation. Formally, the optimization objective is defined as:
arg min |X Q(xR)1?
where Q(-) denotes the quantization function (i.e. integer quantization). Using WikiText2 (Merity et al., 2016) as the calibration dataset, we observe consistent reduction in quantization error during fine-tuning. On the held-out test set, fine-tuned rotations yield lower quantization error than random rotations, demonstrating that the learned rotations generalize beyond calibration. We provide ablation study on fine-tuned rotation in Section 4.4
3.2
ADAPTIVE WEIGHT CLUSTERING AND CENTROID FINETUNING
Building on the smoothed input activations enabled by AOS, we introduce the ACCF method, which refines grouping and centroid search to further reduce clustering error in the outputs of matrix products. Specifically, let WR R W denote rotated weight matrix. We adopt row-wise parameterization for clustering. Specifically, let C Rdo denote the centroid matrix where the i-th row C,,; serves the codebook for the i-th row of weights, and let A {0, d x be the binary assignment
4

---

Published as a conference paper at ICLR 2026
tensor satisfying Ek= As..k 1. The reconstructed weight matrix We is defined element-wist as:
Wess = ECu Auskk
(4)
To minimize the changes in the output, we set the target as:
arg min |XaWa XnWall2
(5)
C.A
where XR E Nxdes denotes the input activations at this layer when the upstream weights have already been clustered. Equation within the SA layers of the MoE through the hybrid operation of input quantization and weight clustering.
sseecifies Sspeciiss the objective function for enabling matrix computations
However, unlike in SA, applying the same operation to the routing mechanism of the MoE FFN may cause mismatches in token-expert assignments compared with the original MoE, thereby degrading performance. To address this, we design a MoE-specifie objective utilizing MoE weighted sum, Meanwhile, prior works have shown the importance of token-expert affinity (Dai et al. 2022 Li et al.,
2025 Hu et al. 2025a Liang et al. 2025). Thus, we add KL divergence loss router logits during fine-tuning to preserve the original exper- assignment. In general, we modify the objective
function in Equation D as follows:
[[RWWR XRWall..
if E (WRQ. WR.K, WR.).
==
E Σ W. + ADRi WR E (WRigate WRep]:
(6)
||
where F denotes the number of experts, Y E Nxdi is the weighted sum produced by the MoE module on the calibration set using the non-clustered weights, and II and II represent the router outputs corresponding to X and XR, respectively. (: returns the KL divergence between the two inputs and X specifies the relative importance of the objective functions.
The optimization problems in Equation 6 can be addressed in alternating, iterative manner. We first fix the assignment matrix A and optimize the centroid matrix C. To this end, we employ a local fine-tuning procedure following Equation 6oto update via gradient descent. To determine the assignment matrix A given the centroids C' while minimizing the output difference, straightforward approach to the nearest-neighhor rounding method in the standard K-means algorithm. However, this does not perfectly align with the objective functions in Equation 6 To mitigate this, we design an analytical solution derived using gradient update assignment matrix. For case of illustration, we adopt the loss function defined in Equation 5 though similar technique can also be applied to the loss function in Equation 6 clustered weights.
We begin by deriving the gradient expression for the
OC 2 2X XX
W. aW.
Set Dj = X and D. Xn. For computational efficiency, we approximate these matrices by retaining only their diagonal entries, i.e., D, = Diag( D1). D2 = Diag( Π2). For each element W Raa. the corresponding error introduced by assigning the k-th centroid Ci is:
(Wnnij Ci,K) |[Di.jj, Ci D2.j; W Wn,vil?
n.ij;
(8)
where Di.jj and D2.j denote the j-th diagonal elements of Dj and D2, respectively. Hence, the optimal assignment for We,3 is obtained by searching over the row-specific centroids {C,,)
k* arg min (Wr..,..CC C.,. kE(L....)
3.3 PERMUTATION-INVAIIANT OUTLIER GROUPING
The ACCF algorithm described in Section 3.2
directly applied to the rotated weight matrices WR- In practice, achieving strong MoE accuracy under ACCF critically depends on initializing Wn to be cluster-friendly, such that a low-error clustered solution can be readily obtained. Since AOS minimizes the quantization error of rotated activations only, the remaining variability is left to the weights, making a fuiendy initialization crucial for ACCF to achieve high performance.
5

---

Published as a conference paper at ICLR 2026
However, in practice, observe that WR is
Clustering group Bubgroup
(b) After
sometimes not amenable to clustering, as shown in Figure 3 (a). Consider weight vector Wi n
Before
Custiingg youp Casterrgnaa
W. 10
2
W
partitioned into clustering groups of size 9 = 4, highlighted by the orange boxes. Each cluster- ing group is allocated a centroid budget of k = 2. Owing to the high variance within group 1, the optimal clustering solution still incurs a cluster- ing error of 17. To reduce the error, we propose the POG method, as illustrated in Figure 3 (c).
1 (.Cuuwiigg
Casterng
W.5.5 5.5 23 a a
We
17 1771
20 23
Clustoring error (High)
POG
Clastering crror (Low]
(c)
Step
Step 2:
10
20 26
2
20
20
SD mb4
0.0
63
00
1.4
4.2
Specifically, the weight vector is first divided into smaller sub-groups (shown in green boxes),
Figure 3: The overview of the POG framework.
each of size in this example. In Step I, the variance is computed across the elements within each uub-group. In Step 2, the sub-groups as to redistribute high- and low-variance sub-groups more evenly across the larger groups of size g= = 4. This reordering helps reduce the variance within each clustering group and thereby lowers the overall clustering error for the resultant R. as shown in Figure 3 (b). The key intuition is that, in the original WR. R. group contains weights that would require more than two centroids achieve low error, while group 2 is much easier to cluster. By permuting elements at the sub-group level, we obtain a more duster-frrendly W It important to note that this idea differs from prior work designed to facilitate quantization (Lin et al. (0244),, since the reordered matrix W is not necessarily amenable to quantization. The resultant Wi is then used as the initialization for the subsequent ACCF operations, and leading to improved performance. The detailed POG algorithm is shown in the Appendix [
permuted indivisible units, ordered by their variance, so
In practice, directly using the permuted matrix W Prior work (Lin et al., 2024a) addresses this by formulating permutation as a matrix multiplication. Specifically, permuting WR R can be achieved by multiplying it with a permutation matrix P. which encodes the permutation pattern shown in Figure 3 As an orthogonal matrix, P can be folded into the SA and FFN components of MoE using the same method rotation matrix R. In CodeQuant, the permutation matrices P and PT are introduced after P.P and P Wout in the self-attention block, and after W.,, up P as well as before P W down in the feed-forward block, ensuring output invariance and improving ACCF performance.
ft alters the output will lead to incorrect results.
3.4 CODEQUANT KERNEL AND SYSTEM IMPLEMENTATION
To evaluate the potential woal-world performance of CodeQuant, we design and simulate an efficient LUT-bbaed GEMM kernel. While a full hardware implementation is beyond the scope of this work, our simulation, based on the validated Accel-Sim framework (Mo et al., 2025] Guo et al., 2023] Avalos Baddouh et al., 2021), models realistic architectural modifications. First, the input and weight matrices are tiled by the weight group size. Each group of weights shares the same set of centroids and is multiplied with multiple activation channels, shown in Figure 4 (a). To reduce redundant multiplications, for each weight group we precompute a LUT using the 16 centroid values and the 16 possible 4-bit integer activation values, as shown in step 1 of Figure 4 (b). The LUT consists of 16 subtables, each computed from one centroid value over 16 activation values when the activations are quantized to 4-bit. CodeQuant uses two-level Mux select the output as shown in step 2 in Figure (b). By pairing activation and weight for shared-memory access, shared-memory conflicts are reduced compared with separate activation and weight accesses (Guo et al. (025). The LUT resides in SM shared memory, as shown in Figure 4 (c) and occupies only a small fraction of the blb).
shared memory available on modern GPUs (NVIDIA Corporation
Although CodeQuant GEMM kernel is promising due to its advantages in eliminating dequantization and multiplication through simple table lookup, existing GPU implementation still faces challenges. This is mainly due to limited instruction support for efficient lookup table precomputation (Mo et al. 2025) and shared memory bank conflicts from extensive random indexing operations (Guo et al. 2025). To make better use of the precomputed lookup tables, the number of activation channels in the input matrix in Figure 4 (a) should increase. However, modern GPU uses the CUDA tensor core for high performance matrix multiplication and the tensor core instruction only supports fixed size of matrix tiles multiplication (8 x 4 x 16 INT8 matrix multiplication in Nvidia RTX A100 GPU IVVIDIA Corporation] a)). To achieve better LUTT-based GEMM performance and keep fair

---

Published as a conference paper at ICLR 2026
Lookup Table Precomputation
2. Lookup
Lookup
nuv
Sharec Memory
Curda Core
Cudl Core
Cude
Weight Group
Weight Matrix
B Centroid
D003 x
DO01 G1X
MUX
LUT
Cude Core
Cuda Core
Cuda
Core
M-nd Activotion
Core
Activation Channel
Teraor Care
Curda
Cudds
Cuda
MUX
Lookup Table 16
bi
6 Controid 18
SM
SM
SM
Input Matrix
Output Matrix
DOS C16x
NUX
bi [ccivotion
CIM
DRAM
(a)
(b)
(c)
Figure 4: (a) One tile of the matrix multiplication. (b) The steps of CodeQuant kemnel, including one-time lookup table precomputation and table lookup. (c) The precomputed lookup table is stored in the shared memory in the Streaming Multiprocessors (SM) in GPU.
comparison with tensor cores, we simulate the GPU performance with optimized matrix sub-tile shape under the same floating point operation numbers per cycle using Accel-Sim (Khairy et al. 2020). To mitigate the bank conflicts, the LUT can be duplicated into more memory banks (Lo et al. 2025) to reduce the chance of multiple threads accessing the same memory bank. To keep the same total shared memory size, we can increase the number banks (32 banks in A100 GPU) and reduce the size of each memory bank, which requires the shared memory structure improvement. We use Accel-Sim simulate the LUT-based GEMM performance with optimized GPU shared memory structure.
EXPERIMENTS
We evaluate CodeQuant across MoE models of varying sizes and architectures, including Phi- Mini-MoE-Instrree (Abdin al. 2024), Qwen3-30B-A3h (Yang al., 2025). DeepSeek- V2-
Lite (DeepSeek-AI et al. (2024), and Mixtral 8x7B (Jiang et al. 2024). The evaluations cover language generation, commonsense QA tasks, and mathematical reasoning tasks. For language modeling, we report perplexity on Wiki'eext2 (Merity et al. 2016) and C4 (Raffel et 2023 For zero-shot QA, we measure accuracy on ARC (Clark et al. 2018), HellaSwag (Zellers al. 2019), MMLU (Hendrycks et al. 202I]. PIQA (Bisk et al. (020), and WinoGrande (Sakaguchi et al. 2021
For few-shot mathematical reasoning, we evaluate CodeQuant using GSM8K (8-shot) (Cobbe ct al. 2021 and MATH500 (4-shot) (Hendrycks et al., 2021).
In the AOS stage, we apply the Cayley transform to optimize the cntivaaion-qaanizaattn rotation matrix R, using 1,024 WikiText2 samples over 128 iterations. In the ACCF stage, we optimize centroids over 64 iterations with 512 WikiText2 calibration samples, setting the KI divergence coefficient 1.0. We study the impact of À in Section 4.4 In terms of preprocessing time, the AOS SeapSeekk-2--Line
stage requires approximately I 15/20/30/50 minutes for Owen3-30B-AR and Mixtral 8x7B on H100 GPUs, respectively. The subsequent ACCF stage requires 30/40/1 10/240 minutes for the same models. Despite the preprocessing cost, our framework fully offlinn meaning no on-the-fly computation. During inference, the weight matrices remain fixed, and inference proceeds in the same way as standard MoE. Section ?? shows that this leads to a net inference speedup.
We compare CodeQuant with several PTQ methods, including RTN (Round-to-Nearest). SmoothQuant (Xiao al. 2024), QuaRot (Ashkboos et al. al.]2024), 2024), Squeezel.LMM (Kim et al. 2024). DuQuant (Lin al.) (0244) and SpinQuant (Liu et al. (0255] baseline methods. For methods that rely on online Hadamard transforms, we adopt the same setting to ensure methodological consistency,
We use the same activation bitwidth across methods, including SqueezeLLM, where input activations are quantized with RTN. For weights, we match the total number discrete representation values. For instance, when QuaRot uses 4-bit quantization, configure CodeQuant with 16 centroids for weight clustering to yield an equivalent representation capacity, using the same centroid-selection strategy Squeezel.L.M. All algorithms evaluated under quantization/clusterie configurations. In
7

---

Published as a conference paper at ICLR 2026
Table 1: Performance in perplexity (PPL) on Wiki2 and C4 dataset, and accuracy on Are-Challenge (A-c), Arc-easy (A-e), HellaSwag (HS), MMLU (ML), PIQA (PQ) and WinoGrande (WG). For each setting, we report the BF16 baseline in the first row. More results are shown in the Appendix.
Models
Methods
Wiki2 (1)
(4)
A-c (1)
A-e (f)
HS (t)
ML (?)
PQ (t)
WG <t)
Avg (t)
BF16
6.83
13.05
0.581
0.813
0.759
0
0.797
0.753
0.731
KIN
9811.22
7431.27
0.287
0.268
0,261
0.232
0.501
0.516
0,344
Phi-mini- MoE-IInstuct
SqueezeLLM
K383.63
5619.01
0.279
0.281
0.263
0.236
0.515
0.500
0.346
SmoothQuunt
24071.25
16320.79
0.263
0.280
0,270
0.240
0.528
0.503
0,347
QuaRot
7.93
14.44
0.545
0.784
0.725
0,633
0.775
0.702
0.694
CodeQuant
13.94
0.790
0.728
0.644
0.784
0.716
0.700
BF16
6,69
9.32
0,491
0.759
0.780
0,551
0.804
0.709
0.682
KTN
812.90
600.45
0.226
0.295
0.283
0.237
0.483
0.339
A4W4 Embeddingg wixe
DeepSeek-
SQ
806.71
614.70
0.257
0.301
0.277
0,238
0.541
0.508
0.354
V2-Lite
SmoothQuant
11,57
16.10
0,381
0.645
0.658
0.305
0.747
0.581
0.553
QuaRot
7.75
10.75
0.457
0.720
0.745
01,550
0.787
0.682
0,640
CodeQuant
9.85
0.749
0.767
0.684
0.664
BF16
9.04
14.05
0,566
0.793
0.776
0,778
0.805
0.694
0.735
KTN
232.49
0.385
0.367
0.445
0.371
Owen3-
S
100.47
121.55
0.222
0.352
0.367
0.243
0.576
0.504
0.377
30B-A3B
SmoothQuant
33.39
0.584
0,490
0.547
0.522
QuaRot
16.04
24.27
0.386
0.596
0.609
0.585
0.735
0.575
0
CodeQuant
10.31
15.75
0.757
0.688
0.735
0.685
0.694
BF16
4.01
7.41
0.579
0.851
0.720
0.677
0.856
0.799
0.747
KTN
10502.14
14045.38
0.284
0.350
Mixtral- 8x7B
Sq
13952.66
19725.12
0.297
0.282
0,279
0.251
0.527
0.519
0,359
SmoothQuant
77.32
96.01
0.222
0.349
0.303
0.236
0. 0.565
0.497
0.362
QuaRoe
16.79
24.29
0.348
0.570
0.512
0.286
0.708
0.560
0.497
CodeQuant
4.65
8.06
0.565
0.819
0.715
0.644
0.827
0,780
0.725
RIN
20.86
30.75
0.345
0.540
0.475
0.318
0.657
0.529
0.477
SquezzeLLM
20.21
0.607
0.590
0.455
0.687
0.572
0.552
Phi-mini-
SmoothQuant
15.34
24.18
0.356
0.559
0,532
0.464
0.656
0.577
0.524
A4W4 Block wixe
Instrictt
QuaRoc
7.63
13.82
0.534
0.790
0.728
0.633
0.783
0.719
0.698
CodeQuant
7,28
13.54
0.562
0.800
0.733
0.646
0.792
0.729
0.710
KIN
161.88
159.65
0.236
0.368
0.344
0.236
0.515
0,380
SqueezeLLM
115.66
112.59
0.238
0.379
0.364
0.234
0.590
0.500
0,384
DeepSeek- V2-Lite
SmoothQuant
9.11
12.72
0.387
0.682
0.687
0,347
0.761
0.613
0.574
QuaRox
7.62
10.59
0.462
0.719
0.745
0.483
0.781
0.668
0.643
CodeQuant
7.03
9.79
0.480
0.741
0.764
0.525
0.794
0.698
0.667
the first, referred to as Block-wise, quantization or clustering is applied within groups of g 1024 weight values along the embedding dimension. In the second, termed Embedding- quantization is applied across the entire embedding dimension, spanning the full embedding vector.
We evaluate CodeQuant GEMM kernel using Accel-Sim (Khairy al. 2020). a state-of offte--f-the-ar GPU simulator, configured to model an A100 80GB GPU with CodeQuant-optimizzed tensor cores. Detailed simulation settings are provided in the Appendix
A..3 As baselines on real A100 hardware, we measure the latencies of HuggingFace (Wolf et al.) 2020) BF16 models, QuaRot (Ashkboos) ctt al., 2024) A4W4 quantized models, and SqueezeLLM Kim et al. 2024) A4W4 quantized models. Sq serves baseline for weight clustering and activation quantization without GPU architectural modification, helping isolate the latency performance gains from CodeQuant hardware kernel design. Experiments use prefill length of 512, decoding length of 1 128, and batch size of 16. Additionally, we measure the real hardware performance of CodeQuant by benchmarking the A8W4 T-MAC kernel (Wei al., 2025), mixed-precision LUT-based CPU GEMM kernel, against Llama.cpp (Gerganov & ggml-org contributors 2023) BF16 and A8W4 models on CPU.
4.1 MAIN RESULTS
Table T summarizes the evaluation results of CodeQuant under different configurations. For clarity, we adopt the 'AxWx' notation. For instance. in QuaRot, RTN, and SmoothQuant, 'A4W4' denotes 4-bit quantization of activations and 4-bit quantization of weights. In contrast, under CodeQuant, '4W4'' corresponds to applying 4-tit linear quantization activations and clustering weights into 2 = 16 centroids. In the Embedding wise setting, POG has no effect on the final performance. Therefore, POG is not applied here. Detailed explanation will be provided in Appendix A.2.2
We first present the Embedding-wind wise evaluation results. For A4W4, CodeQuant delivers substan- tial improvements over existing methods. On Qwen3-30B-A3B, it reduces perplexity by 5.73 on
8

---

Published as a conference paper at ICLR 2026
Table 2: Rotation- method performance comparison. CodeQuant indicates that online
Hadamard transforms are enabled during the quantization process.
Models
Methods
Wiki2 (1)
C4
A-c (t)
A-c (1)
HS (1)
ML (1)
PQ (1)
WG (1)
Avg (f)
Embedding soneedding-wsse wiss
DeepSeek- V2-Lite
DuQuant
8.43
11.94
0.455
0.708
0.623
0.400
0.775
0.693
0.624
SpinQuant
9.24
0.427
0.706
0.774
0.638
0.610
A4
,odeQuant,.ad
8.16
11.38
0.445
0.723
0.727
0.454
0.782
0.644
0.629
DuQuant
13.52
20.10
0.472
0.739
0.637
Qwen3-
30B ABB
Sp
14.61
22.07
0.415
0.600
0.628
0.584
0.692
0.622
0.590
CodeQuant,...
12.69
19.89
0.739
WikiText2 and 8.52 on C4, while increasing average accuracy by 11.3% compared to QuaRot, with even larger gains over SmoothQuant on both metrics. On DeepSeek-V22 V2-LLtte, CodeQuant again improves performance, lowering perplexity by 0.67 on WikiText2 and 0.9 on C4, alongside 2.4% accuracy increase over QuaRot. On Mixtral 8x7B, CodeQuant shows the same trend, reducing perplexity by 12. on WikiText2 and 16.23 on C4 compared to QuaRot, and increasing average accuracy by 22.8%. These results highlight CodeQuant's consistent advantages across architectures and demonstrate that its effectiveness remains stable across both model structure and model scales. The A8W4 Embedding- results are detailed listed in Appendix AA.4
With Block-wise setting and POG enabled, we evaluate Phi-mini-MoF. Moninnn--Mo--Istrruuc and DeepSeek- V2- Lite. Under A4W4, both models show clear improvements over the Embedding-win baseline. However, when moving to A8W4, Phi-miin-MMo--Instruc benefits only marginally, and DeepSeek- V2-Lite even drops by 0.3% relative the baseline. We attribute this to DeepSeek's already strong accuracy without POG, with less than a 1 % gap compared to BF16. These results suggest that permutation is effective under extreme compression, as detailed in Appendix
In addition, we evaluate CodeQuant against two strong DuQuant, both of which suppress outliers through trainable or structured transformations and utiize online Hadamard transformation. For faimess, we adopt online Hadamard transforms and denote this variant as CodeQuant,.ad matching the SpinQuant setup. shown in Table CodeQuant,.od consistently outperforms both baselines. On Qwen3-30B-A3B it reaches average accuracy of 0.653 compared to 0.637 for DuQuant and 0.590 for SpinQuant achieves 0.629, again exceeding DuQuant 0.624 and SpinQuant,od, at 0.610, demonstrating robust advantages across language modeling and downstream tasks.
PTQ baselines, SpinQuant and
DeepSeek-V2-Lite, V2-Lite,
4.2 MATHEMATICALLY REASONING PERFORMANCE
further assess whether CodeQuant preserves reasoning-heavy capabilities, which are typi- cally more sensitive to quantization. We eval- uate DeepSeek- V2-Lite and Qwen3-30B-AN under the A4W4 Embedding wise configura- tion on GSM8K (8-shot) and MATH500 (4-shot) (DeepSeek-Al) AI et al. 2024), where each k-shot prompt includes & worked examples before the test question. As shown in Table CodeQuant substantially outperforms QuaRot and remains close to the BF16 baseline. On DeepSeek-V22 Lite, the degradation minimal, only 3.4% on GSM8K and 1.3% MATH500. For Owwe3--30B-A3B the advantage becomes nounced: CodeQuant improves over QuaRot by 35.9% on GSM8K, and 11.3% on MATH500, highlighting its strength reasoning-heavy tasks.
Table 3: A4W4 Embedding-wind results on GSM8K (8-shot) and MATH500 (4-shot).
Models
Methods
GSMSK (f)
MATH500 (t)
BF16
0.364
0.121
DeepSeck.VV.I. V2-Lite
QuaRot
0,231
CodeQuant
0.330
0.108
BF16
0.924
0,322
Qwen3-30B-A30
QuaRot
0.508
0.128
CodeQuant
0.867
0.241
pro-
4.3 LATENCY EVALUATION
presents the normalized speedups of all baselines, with BF16 latency normalized to 1. Compared with the BF16 models, CodeQuant achieves an average 2.63x speedup, which underscores the effectiveness of low-bit activation and weight quantization together with the LUT--based GEMM
Figure 5

---

Published as a conference paper at ICLR 2026
Table AOS Impact
Table 5: KL Loss Impact
Table Bitwidth Budgets Impact
Method
DeepSeek VieLite
Method
Tisk
mini
Deepseck- V2-1.ie
Model
Wikii
DeepSeek V2-Lite
C4
Acc 1
7.29
7.10
Squeezel. L4w
24.36
32.58
0.496
Random
C4
10.16
W/O KL
C4
13.95
9.57
SqueezeLLLM M
8.40
11.69
0.619
Acc
0.652
0.694
0.658
Squeezel.LMM Arr
7.17
10.01
0.652
Wika2 v
7,06
WIkI2
7,06
7,03
CodeeQuantzau ssw
10.68
14.39
0.368
AOS
C4
9.85
W/ KL
C4
13,80
9.79
CodeQuant
7,59
10.58
0.639
0.667
CodeQuant, cowe
7,06
design. The speedup of CodeQuant over QuaRot highlights the advantage of replacing repetitive multiply-accumulate operations with direct LUT indexing, thereby reducing redundant multiplications. The improvement over Sq reflects the benefit of deploying a GPU implementation that uses optimized LUT operations. Considering the strong accuracy results of CodeQuant shown in Table T CodeQuant achieves the optimal performance among the baselines. Further- more, we validate these performance trends real hard- ware by benchmarking CPU kernel, where CodeQuant achieves up 4.15x speedup over a BF16 baseline. Additional evaluations can be found in Appendix A.5
BF16
Quarot
SqueszeLLM
CodeQuant
Speadup
2.8
2.8
3.0
2.0
DenpSeck
Figure 5: Normalized speedup on one A100 GPU.
4.4 ABLATION STUDIES
Impact of Activation Smoothing We evaluate whether fine-tuning the rotation matrix improves accuracy on DeepSeek-Ve -eeeLiee-V2-Liee under the A4W4 Embedding-wis. configuration, keeping all other settings fixed. Specifically, we compare a random rotation with its fine-tuned version produced by AOS, As shown Table rotational matrix finetuning yields consistent improvements, boosting accuracy by 1.4% and reducing perplexity by 0.23 on WikiText2 and by 0.31 on C4.
Impact of KL Penalty We evaluate the effectiveness of the KL divergence term defined in Equa- tion 6 The ablation is conducted on Phi-mini- SontiiMoE-liee M-E--MMttucc and DeepSeek- V2-LLte under the A4W4 Block-wise configuration, comparing two settings: (i) centroids fine-tuned without the KL divergence term () 0.0), and (ii) centroids optimized with the full ACCF loss (A 1.0). As shown in Table ACCF with the KL penalty outperforms the version without it. Additional analysis in Appendix (A.7 further shows that the KL penalty also stabilizes the router behavior, indicating that KL regularization helps preserve the original expert-routing pattern after quantization.
Impact of Low-Bit Compression We examine CodeQuant performance under different centroid budgets on DeepSeek- Baneek-V--LLie V2-Lite with Embedding quantization. We apply the same rotation matrix to quantize activations for both CodeQuant and SqueezeLLM, and evaluate over three settings: A4W2, A4W3, and A4W4. As shown in Table 6 CodeQuant consistently outperforms SqueezeL across all budgets. Under the most aggressive case (A4W2). CodeQuant's 's average accuracy decreases by 9.9% relative to the A4W4 case, whereas SqueczeLLM drops by 15.6%. Moreover, CodeQuant's advantage widens as the budget shrinks from 1 1.5% at A4W4 to 7.2% at A4W2, indicating robustness under extreme compression.
CONCLUSION
We present CodeQuant, unified CodeQuant reduces quantization error while preserving accuracy, achieves up to 4.15x latency reduction. Experiments confirm that CodeQuant delivers superior accuracy-efficiency trade-offs compared to other baseline algorithms, enabling more reliable low-precision deployment of MoE.
framework for low-precision MoE.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics. CodeQuant is post-training quantization frame- work evaluated on pretrained models and public datasets, without the use of private or user-specific data. Our research does not involve human subjects, private sensitive data, personally identifiable information. The method modifies only internal representations through weight quantization and routing, introducing no new risks in fairesss, privacy, security beyond those inherent to the base models. We are not aware of any direct ethical concerns specific to this work.
REPRODUCIBILITY STATEMENT
Code and models: All experiments in this paper are conducted on publicly available datasets with specified preprocessing steps. Detailed configurations, including hyperparameter, training procedures, and hardware specifications, are reported in the experiment section. Baselines are re-implemented following their original papers, with reference to the authors' released code when available. While the source code for CodeQuant is not released at submission time, we will make it publicly available upon acceptance to facilitate reproducibility.
Datasets: All datasets used in this work are publicly available.
Randomness: All experiments are run with fixed random seeds in the scripts, to ensure consistent results.
Compute resources: Our experiments conducted on NVIDIA RTX H100, RTX A100, Accel-Sim GPU simulator, and Intel CPU as described Section 4.
USE OF LARGE LANGUAGE MODELS
Large language models (LLMs), such as ChatGPT, were used only for polishing language and improving readability. All technical ideas, analyses, experiments, and conclusions were conceived, implemented, and validated by the authors. The final manuseript was carefully reviewed to ensure accuracy and correctness.
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httpps://arrd..orrr/aa///00.
Yongqi An, Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Systematic outliers in large language models, 2025. URL. htttts:://aattt org/ 2502.
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms, 2024. URL/ttps:: //tax//aaiii/ii//aa org/ abs 2404. 00456
Cesar Avalos Baddouh, Mahmoud Khairy, Roland N. Green, Mathias Payer, and Timothy G. Rogers. Principal kernel analysis: A tractable methodology to simulate scaled gpu workloads. In MICRO-54: 54th Annual IEEF/ACM International Symposium on Microarchitccture MICRO '21, pp. 724-737, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 978 1450385572. doi: /tg.////ii..o.....//// 1ee...1113346655 00144454466752.3881111 3480100
N0.114433466552.4400100
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical commonsense natural language. Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Patrick Blumenberg, Thomas Graave, and Tim Fingscheidt. Improving block-wise Ilm quantization by 4-bit block- blick-wise optimal float (bof4): Analysis and variations, 2025. URL https: : //axiivv org//bb/ 2505. 06653
11

---

Published as a conference paper at ICLR 2026
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees, 2024. URL httpttss///axxw.orogg 13304
2307
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URR.htts:///ar/t wwwpzivv..rrg..ooooo /aazg//b//// /18///803.00 803.05457
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL (tthttps: / arxiv.org/
abs/2211 141111666
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baohao Chang, and Furu Wei. Stablemoe: Stable routing strategy for mixture of experts, 2022. 08396
DeepSeek-AI, Aixin Liu, Bei Feng, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model, 2024. ULL.ccpss://axxi..orr////// 04444.04.4444 04434
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Lim.:emmoooe..mm.mmma...... Llm.int00: 8-bit matrix multi- plication for transformers at scale, 2022. URL UtL.hcpss.//aaxiv.orr//aba// 07339
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer Qlora: Efficient finetuning of quantized Ilms, 2023. URL
Zhenyuan Dong and Sai Qian Zhang. Ditas: Quantizing diffusion transformers via enhanced activation smoothing. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 4606-4615. IEEE, 2025.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Darshan C. Ganji, Saad Ashfaq, Ehsan Saboori, Sudhakar Sah, Saptarshi Mitra, MosammadHossein AskariHemmat, Alexander Hoffman, Ahmed Hassanien, and Mathieu Léonardon. Deepgemm:
Accelerated ultra low -owppiccsi inference on cpu architectures using lookup tables, 2023. URL
lattps://arxiy> httpp///aaxiiv.rgg/aoooooo. wrg//bs//////oooooooooooooo 223/g//bs//33444. 09049
Georgi Gerganov and ggml-org contributors. llama.cpp. httpp::/github.com ggml org/
11ama. cpp, 2023. Accessed: 2025-09-23.
Cong Guo, Jiaming Tang, Weiming Hu, Jingwen Leng, Chen Zhang, Fan Yang, Yunxin Liu, Minyi Guo, and Yuhao Zhu. Olive: Accelerating large language models via victim pair quantization. In Proceedings of the 50th Annual International Symposium Computer Architecture, ISCA '23, New York, NY, USA, 2023. Association for Computing Machinery. wttphttps://oi.org/////
outlier-
ISBN 9798400700988. doi: 01114//357937135590009 3579371 3589038
Han Guo, William Brandon, Radostin Cholakov, Jonathan Ragan- Kelley, Eric Xing, and Yoon Kim. Fast matrix multiplications for lookup table-quantized Ilms, 2025. URL http::ttps://t //arxiv
org/abs/ 2407 10960
Tianyu Guo, Druv Pai, Yu Bai, Jiantao Jiao, Michael Jordan, and Song Mei. Active-dormant attention heads; Mechanistically demystifying extreme-token phenomena in 11ms. CoRR,
0os/2410.138333 2024, doi: 10.48550/arxiw.2410.. 13835. URL httpttttt///aatt.a wwwpxis..rg//ss///// 2410. 13835
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016. URL httlaattt/aaxi..oot/ abs 01/1/1100 0. 00149
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. | https: //arxivv
org/ abs/ 2009.0.. 03300
12

---

Published as a conference paper at ICLR 2026
Xing Hu, Zhixuan Chen, Dawei Yang, Zukang Xu, Chen Xu, Zhihang Yuan, Sifan Zhou, and Jiangyong Yu. Moequant: Enhancing quantization for mixture-of -feeeeerts- large language models via expert-balaneed sampling and affinity guidance. arXiv preprint arXiv:2505.33804 2025a.
Xing Hu, Yuan Cheng, Dawei Yang, Zukang Xu, Zhihang Yuan, Jiangyong Yu, Chen Xu, Zhe Jiang, and Sifan Zhou. Ostquant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting, 2025b.
orq
13987
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and Xiaojuan Qi. Slim-Ilm: Salience-driven mixed-precision quantization for large language models, 2025. URL httpss//aaaxxx..crr/////////
14917
Apple Inc. Apple palettization, 2024a. URL http:://appl...iiuu.../////eeee/
docs gwwdes/source/ http:://ww..............otaat/a.
Arm Inc. Arm ethos-u npus, 2020. URL /ttp:/tttee///cmmmaaatiiooerree.
static/ 0aat:cc//6ob22ab02222222222
Cerebras Systems Inc. Cerebras cs-3, 2024b.
URL kttps://w...eeerras...ll
Cerebras-cs3
Intel Corporation. Intel xeon w7-3445 processor (52.5m cache, 2.60 ghz) specifica-
tions. https://wwwz inteii com/ wwtpoteeeoo/oooetttttteooccte//t http:///s//s//s////////////s////
intel w
52 5m cache 2- wht6:00-hhhssp/e wtwpgh///pppcccaaiiooooo
html 2025. Accessed: 2025-09-24.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang. Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024. URL /tttps://axxv...o www.:xiw..w.g.....oobeaa htw.sa/////01.... 04088
Mahmoud Khairy, Zhesheng Shen, Tor M. Aamodt, and Timothy G. Rogers. Accel-sim: an extensible simulation framework for validated gpu modeling. In Proceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture, ISCA '20, pp. 473 486. IEEE Press, 2020. ISBN 9781 9781728146614,, doi: 10.1 100110999SCAA4466722000000000 URL https: /
I109/I9CA45697. 0ttp:/45666.222..00055
Schoon Kim, Coleman Hooper, Amir Gholami. Zhen Dong, Xiuyu Li. Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spare quantization, 2024. URL. https?
attrg/abs//ee abt.//bs/22333 2006 07629
Jing Li, Zhijie Sun. Dachao Lin, Xuan He, Binfan Zheng, Yi Lin, Rongqian Zhao and Xin Chen. Expert-token resonance moe: Bidirectional routing with efficiency affinity-driven active selection, 2025. URL (RL/atts:///rxxtt
org/ aa////bb/2226 2406. 00023
Jun Li, Li Fuxin, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. arXiv preprint arXiv:2002.01113 2020.
Jingcong Liang, Siyuan Wang, Miren Tian, Yitong Li, Duyu Tang, and Zhongyu Wei. Not all models suit expert offloading: On local routing consistency of mixture-of-expert models, 2025. URL
org/
htrg/abs/// abs 220..2555..3. 16056
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger
quantized Ilms, 2024a.
URL https://::///aa /ttttps://axiiv.. org/ 255/
Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of machine learning and systems, 87-100, 2024b.
13

---

Published as a conference paper at ICLR 2026
Shi-Zhe Lin, Yun-Chih Chen, Yuan-Hao Chang, Tei-Wei Wei Kuo, and Hsiang- Pann Li. Lutin: Efficient neural network inference with table lookup. Proceedings of the 29th ACM/IEEE International Symposium Low Power Electronics and Design, pp. 1-6, 2024c.
Wenxuan Liu and Sai Qian Zhang. Hq-dit: Efficient diffusion transformer with fp4 hybrid quantization. arXiv preprinl arXiv:2405. 2024.
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Lim quantization with learned rotations, 2025, URL | http:: /ttss///rxxiii wwwpa/wwwwwg//bb///4555... 16406
Ka Man Lo, Zeyu Huang, Zihan Qiu, Zili Wang, and Jie Fu. A closer look into mixture-of-experts in large language models, 2025. URL https://arxiv http://w....aaa///220000 18200 18219
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Zhiwen Mo, Lei Wang, Jianyu Wei, Zhichen Zeng, Shijie Cao, Lingxiao Ma, Naifeng Jing. Ting Cao, Jilong Xue, Fan Yang, and Mao Yang. Lut tensor core: software- hardware co-design for lut-based low-bit IIm inference. In Proceedings of the 52nd Annual International Symposium on Computer Architecture, SIGARCH '2, pp. 514 528. ACM, June 2025. doi: 10. 10.11457569503333310577 URL
org 11..111..1..1111
Yasunori Nishimori and Shotaro Akaho. Learning algorithms utilizing quasi-geodesic flows on the stiefel manifold. Neurocomputingg 67:106-135, 2005,
NVIDIA Corporation. Nvidia a100 tensor core gpu. cetaaeeeeeeerrr/ae//a/aa/ a. Accessed: 2025-09-18.
NVIDIA Corporation. Nvidia h100 tensor core gpu.
http:://ww.nvidiaa.cc.l/// data chtoaeeeenerr110 Accessed: 2025-09-18.
NVIDIA Corporation. Nvidia ada gpu architecture whitepapee.
NVIDIA Corpora- tion, 2022a. URL / nvidia. com/ aem http://add////llnniioo///eeerrrr
NVIDIA Corporation. Nvidia h100 tensor core gpu architecture whitepaper. Techni- cal report, NVIDIA Corporation, 2022b. URL
www.ssourree..iii....... nvvdia com/
en www.shoopp----cctteeeeeeeeeeeeee
Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung
Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models, 2024.
URL httlt::p/// /tap::://arxivvi org/ abs/2200 09557
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https: /ttp/v.vvvv/..////////ea///a.aav wrw. hatgg//bb//aae
10683.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106.
2021.
Zunhai Su, Qingyuan Li, Hao Zhang, YuLei Qian, Yuchen Xie, and Kehong Yuan. Unveiling super experts in mixture-of-expprtt large language models, 2025.
abs/
2507.233
Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models, 2024. URL /ttttps://axxi...om www.xiii.rg...... abs /b/p///2.//2227722
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip: Even better Ilm quantization with hadamard incoherence and lattice codebooks, 2024. URL
http:s//arrxv..rrr///////22220.
0439
14

---

Published as a conference paper at ICLR 2026
Mart van Baalen, Andrey Kuzmin, Ivan Koryakovskiy, Markus Nagel, Peter Couperus, Cedric Bastoul, Eric Mahurin, Tijmen Blankevoort, and Paul Whatmough. Gptvq: The blessing dimensionality for Ilm quantization, 2025. https://///xxi
2402 15319
Yutong Wang, Haiyu Wang, and Sai Qian Zhang. Qsvd: Efficient low-rank approximation for unified query-key-value weight compression in low-precision vision- languale models. arXiv preprint arXiv:2510.16292. 2025.
Jianyu Wei, Shijie Cao, Ting Cao, Lingxiao Ma, Lei Wang, Yanyong Zhang, and Mao Yang. T-mac: Cpu renaissance via table lookup for low-bit Ilm deployment on edge. In Proceedings of the Twentieth European Conference on Computer Systems. EuroSys '25, pp. 278 292. ACM, March 2025, doi: 10.1145/368903133666000 URL hhttp://dak doi org 10 1ee..114555388 3696099
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison. Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-art natural language processing, 2020. URL htt.Lntp:://axi..orr///aaaa 011.0/00///191...... 03771,
Jingyang Xiang and Sai Qian Zhang. Dfrot: Achieving outlier-free and massive activation friieation-free for rotated Ilms with refined rotation. arXiv preprint arXiv:2412.00644 2024.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post training quantization for large language models, 2024. URL https
7/arxiv. org/ abtp//1..22111... 10438
Yuhui Xu, Yongzhuang Wang, Aojun Zhou, Weiyao Lin, and Hongkai Xiong. Deep neural network compression with single and multiple level quantization, 2018. URL
abs/1803. 03289
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388 2025.
Davis Yoshida. Nf4 isn't information theoretically optimal (and that's good), 2023, URL https 7 /a.xiv.orr/abs//3333.... 06965
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. httt:://///axxx http:/iwwwwggaa/////aoooaa 1100 07830
A APPENDIX
A.1 ROTATION MATRIX DEEPSEEK- V2-LITE
In Section 3.1
we integrate the rotation matrix into the weight parameters. Due the architectural differences between the DeepSeek- V2-Lite model and Qwen3-30B-A3B, in SeepSeek--2LLinn SA block, the rotation matrices are applied to W₂ and W In the MoE FFN block, DeepSeek -2-kLee includes a shared expert; therefore, the rotation matrices are also applied to the shared expert's s Wup and W gate-
A.2 POG ALGORITHM AND ANALYSIS
A.2.1 ALGORITHM
In Section3.. 3.3 propose permutation method. In this section, will introduce how construct permutation matrix that makes the weights more amenable to clustering in detail.
First, need to obtain permutation sequence using Algorithm Given weight matrix WR Rd Xdout, we compute a permutation sequence π E R defined as a bijective sequence in which each element मं specifies the original column relocated to the i-th position in the permuted
15

---

Published as a conference paper at ICLR 2026
Algorithm 1: POG Algorithm
Input:
Wn E IR xdui is the weight matrix after rotation; 9 E is the quantization group size; E N is the small subgroup size, which is the unit to swap, and it satisfies 9s g. column permutation order x of {1...., Jem)
Output:
Procedure
Ng &- dan/g.
N₂ +- dam/gs. gn.
Tt 6 g/9s:
dn >
Compute mean absolute value each column: S E Rd where sj
Rir
dig
Isdz <- argsort(S. (S, desc); Partition Lıs into N. groups of size ga, such that cach group G. L.a=(( 1) + 1: ig.] I
,,..,, Noi
for to N. do
WG. = Wnl: Gi];
+ Mean( (WennSSt dim = 1), dim = 0):
{ei, ..., UN J:
10
Iv + argsort(V, desc), Iv + argsont( asc);
11
ι + []:
12
for = 11-10 to N do
13
append Tv [i]
append Tv [(i 1)(n 1) + d(n 1)] m;
15
return
arrangement. Concretely, we first sort the columns by their mean absolute value and partition them along the column dimension into small subgroups. Then, the subgroup with the largest average variance is paired with subgroups of the smallest variance to form the first group, and this process is repeated until all subgroups are assigned.
Second, after obtaining the permutation order ग,, we construct the corresponding permutation matrix P. defined as follows:
[1, Ps 0, otherwise.
if =i==(j), where P E (0, 1)axn
(10)
Lastly, we fuse the permutation matrix into the weight parameters eliminate additional online computation. For the Phi-mini-MoE-Innttuu and Owwnn-33B-AA3 models, the permutation is applied in both the self-attention and MoE-FFN blocks. the self-attention block, we multiply the permutation matrix with WRy and apply its transpose to WR In the MoE-FFN block, the permutation matrix multiplied with WRap- while its transpose is applied to WR for each expert.
For -eppSeek-V2-Litee the permutation is applied to all experts, including the shared expert, in the MoE-FFN block. Specifically, WRap multiplied by the permutation matrix, and WR.down is multiplied by its transpose for every expert. In the self-attention block, due to the unique structure of the DeepSeek family, additional steps are required to preserve output invariance. First, the layer normalization is absorbed into the weight matrix. Then, we decompose
W WRAV [W Woooppeeeeddd. WRIKpe
into WR Romppreeee kv and WR The permutation matrix is multiplied with We compesseed kv. while the transpose of the permutation matrix is applied to W WRKVD preserve output invariance.
A.2.2 ANALYSIS
POG is designed to reduce weight clustering error under the Block-wise setting, where weights are partitioned into clustering groups (of fixed size along the embedding dimension clustered independently. In this regime, a column permutation changes which weight values are placed into the same group, and therefore solutions. By constructing a permutation matrix that reorders columns with different variability, POG aims form groups that are better conditioned for clustering.
change the within-gres group distribution and the resulting K-means
16

---

Published as a conference paper at ICLR 2026
Table 7: Performance in perplexity (PPL) on Wiki2 and C4 dataset, and accuracy on Are-Challenage
(A-c), Arc-easy (A-e), HellaSwag (HS), MMLU (ML), PIQA (PQ) and WinoGrande (WG). Code-
Quant is set as A8W4 Embedding-wir wise.. We report the BF16 baseline in the first row, and mark the
methods as BF16.
Models
Methods
Wiki2 (L)
C4())
A-c (f)
A-e (1)
HS (t)
ML
PQ (T)
WG (1)
Avg (f)
BF16
6.83
13.06
0.581
0.813
0.759
0.681
0.797
0.753
0.731
RTN
12.13
20.46
0,460
0.730
0.618
0.497
0.741
0.632
0.613
Phi-mini- MoE-IInstrcct
7.41
13.65
0.565
0.795
0.736
0.658
0.791
0.746
0.715
SmoethQuant
9.50
16.23
0.481
0.741
0.653
0.569
0.756
0.638
0.634
QuaRot
7.69
14.15
0,549
0.787
0.735
0.652
0.786
0.737
0.708
CodeQuant
13.73
0.79
0.741
0.796
0.732
0.719
A8W4 Emhedding- wise
BF16
9.04
14.05
0,566
0.793
0.776
0,778
0.805
0.694
0.735
RIN
14.09
21.65
0.284
0.446
0.692
0.643
0.656
0.626
0.558
Qwen3-
LM
9.37
14.56
0,529
0.768
0.743
0,764
0.770
0.671
0.707
30B-A3B
SmoothQuant
11.77
17.82
0.463
0.703
0.721
0.695
0.773
0.667
0.670
QuaRot
11.18
16.58
0.471
0.671
0.696
0,708
0.766
0,654
0.661
CodeQuant
9.81
15.11
0.535
0.779
0.754
0.757
0.797
0.679
0.717
BF16
6.69
9.32
0.491
0.759
0.780
0,551
0.804
0.709
0.682
RTN
7.72
10.89
0.469
0.719
0.732
0.457
0.790
0.671
0.640
DoepSook-
SquazeLLMM
9.60
0.485
0.755
0.760
0,525
0.803
0.701
0.658
V2-Lite
SmoothQuant
7.61
10.70
0.457
0.729
0.754
0.480
0.794
0.674
0.648
QuaRot
7.29
10.08
0,466
0.737
0.757
0.493
0.792
0.705
0.658
CodeQuant
6.84
9.50
0.487
0.764
0.773
0.533
0.798
0.709
0.678
RTN
14.93
0.530
0.777
0.683
0.770
0.671
0.668
Sq
7.18
13.46
0,576
0,801
0.744
0.670
0.797
0,759
0.724
Phi-mini-
SmoothQuant
14.67
0.516
0.768
0.697
0.769
0.688
0.673
A8W4 Block-wisc
MoE-IIntrruc
QuaRot
7.48
13.65
0,550
0,794
0.737
0.645
0.786
0.737
0.708
CodeQuant
7.11
13.33
0.575
0.817
0.744
0.661
0.792
0.751
0.723
RTN
7.47
10.40
0,455
0.743
0.764
0.488
0.788
0.687
0.654
LQ
6.86
9.53
0.469
0.754
0.773
0.535
0.796
0.706
0.672
DeepSeek-
SmoothQuant
7.42
10.39
0,459
0.724
0.748
0.476
0.792
0.665
0.644
V2-Lite
QuaRot
7.22
10.03
0.466
0.746
0.760
0.508
0.795
0.688
0.661
CodeQuant
6.83
9.49
0.472
0.756
0.775
0.535
0.804
0.708
0.675
In contrast, under the Embedding-wise setting where clustering is performed over the entire embed- ding dimension single set (i.e., without block partitioning). POG has effect on the clustering result. This is because permutation only reorders the elements of the set being clustered, and K-means gnver-invariant. In Block-wiss wise setting, however, each block is formed by selecting a subset of elements along with the embedding dimension after permutation. As a result, the composition of each subset depends on the ordering, and thus permutation can change which elements belong the same block. Consequently, POG will only benefit Block-wise setting.
A.3 HARDWARE EVALUATION SETTINGS
We use Accel-Sim (Khairy et al. 2020 state-of-the-art open-source GPU simulator, and modify its configuration and trace files to model both the original RTX A A100 80GB GPU and an A with LodeeQuant-opimizze tensor cores, as shown Section 3.4 The simulator calibrated against real A100 measurements, achieving less than 1% latency error, consistent with prior GPU module design studies (Mo et al. 2025) Guo et al. 2023 Avalos Baddouh et al. 202T). We configure tensor cores with matrix multiplication size of 16 X X 8 and 64 shared memory banks to improve lookup table reuse and reduce bank conflicts.
A.4 CODEQUANT 48W4 EMBEDDING- WISE ACCURACY PERFORMANCE
With 8-bit activation quantization, the error of activation quantization is minimized. In this testing setup, the effectiveness of quantivation/clustrrig method will stand out. results of offline version CodeQuant under Embedding-wii wise setting. Our method is generally better than all baselines. Meanwhile, we notice that SqueezeLLM also performs well in this case showing the promising of clustering method. Table 3 summarizes the comparison between CodeQuant SpinQuant and DuQuant. As we discussed in the main content, with more layers being quantized or clustered in this online scenario, the accuracy is lower than Table 7 but make the same quantization/clustee layers setup to make sure fair comparison. Compared with these two strong rotation methods, our algorithm excels them proving the effectiveness of CodeQuant.
the main
17

---

Published as a conference paper at ICLR 2026
Table 8: Rotation-basee method performance comparison. CodeQuant indicates that online
Hadamard transforms are enabled during the quantization process.
Models
Methods
Wiki2 (1)
C4
A-c (t)
A-e (1)
HS (1)
ML (1)
PQ (1)
WG (f)
Avg (f)
Embedding sateedding-wise grsssss
DeepSeek- V2-Lite
DuQuant
8.43
11.94
0.455
0.723
0.728
0.406
0.815
0.665
0.632
SpinQuant
8.62
0.431
0.696
0.726
0.432
0.781
0.650
0.619
A3W4
CodeQuant,.ad
7,80
10.85
0.457
0.726
0.742
0.472
0.787
0.676
0.643
DuQuant
12.70
0.455
0.604
0.777
0.622
0.649
Qwen3-
30B A3B
Sp
13:55
19.53
0.474
0.716
0.669
0.677
0.762
0.616
0.652
CodeQuant,.ad
16.80
0.481
0.719
0.766
A.5 LUT KERNEL PERFORMANCE ON CPU
Table 9: Latency and Memory Evaluation on CPU
Phi-mini- MoE- MoE-Instruct
DeepSeck- DeepSeck-V2-Lite
Qwen3-30B-in --3--3
Bit Width
Method
Mem. (GB) 4 Lat. (5)
Mem. (GB) L Lat. (6)
Mcm. (GB) J.
Lat. (s)
BF16
Llama.cpp (CPU)
14.3
40.1
29.3
50.0
56.9
66.1
Llama.cpp (CPU)
4.1
15.0
8.8
17.1
16.2
20.1
A8W4
CodeQuant (CPU)
4,1
13.3
8.9
14.2
16.5
15.9
T-MAC (Wei et al.
2025) implements mixed-precision GEMM via lookup table-based kernel
within the Llama.cpp framework (Gerganov & ggml-org contributors 2023), enabling efficient CPU execution. We evaluate CodeQuant by benchmarking the A8W4 T-MAC kernel against BF16 and A8W4 models in Llama.cpp The experiments are conducted on an Intel(R) Xeon(R) w7-3445 CPU (Intel Corporation] 2025) using 20 threads. CPU, CodeQuant achieves up to 4.15x speedup over BF16 baselines and consistently outperforms the quantization baselines. The gains are larger on CPU than on GPU primarily because CPU inference exposes less parallelism and is more memory- bound (Wei et al. (025), making the improvements over the baseline more pronounced. In addition, efficient LUT instructions on CPUs further amplify CodeQuant's 's advantage over quantized baselines.
Table 10: Impact of POG
Table 11: KL Penalty Impact on Router
Thh-mini-Mnfe Mnn- Instruut
DeepSeek V2-Lite
Model
Method
Change Rute (%)
Method
Wiki2
C
Ace
Wiki2
C4
Acc
DeepSeek- V2-Lite
QuaRor
41.47
W/O POG
7.31
13.66
0.710
7.08
9.88
0.663
CodeQuant wio KL
CodeQuant w/ KL.
22,82
24.33
W/ POG
7.28
13.54
0.714
7.03
9.79
0.668
30B-A38
Qwent.
CodeQuant wlo KL
Qualox
60.21
72.13
CodeQuant
A.6 IMPACT OF POG
We evaluate the impact of POG operation on Phi-mini-MoE-Instruuc and PrepSeek-V2-Linn under the A4W4 Block-wise configuration with fixed group size of g 1024. As shown in Table IO removing permutation consistently degrades performance. On -hi-mini-MoE- -MoE-Intrrutt, with POG applied, perplexity increases by 0.03 on WikiText2 and 0.12 on C4, while accuracy drops by 0.4%. A similar pattern is observed on DeepSeek-V22 V2-Lite, confirming the generality of this effect.
A.7 IMPACT OF KI. PENALTY ON ROUTER LOGITS
We measure the effect of KI. divergence on router stability for DeepSeek LeanSeee--V-LLiiee V2-LLte and Qwen3-30B- A3B under the A4W4 Embedding- Wise setting. The change rate is defined as the layer-wise average change in Top-K expert indices (with A3B) computed by comparing the router outputs before and after quantization. Results are averaged over 50 samples from the WikiText-2 test set.
for DeepSeck- -2-LLte and
for Qwen3-30B:
As shown in Table II adding the KL penalty consistently reduces routing perturbation. On DeepSeek- V2- LitLi the change rate drops from 24.33% to 22.82%. A similar trend is observed on Qwen3-30B. A3B, where KI. regularization yields reduction from 60.21 % 59.58%, despite its larger 128-expert
18

---

Published as a conference paper at ICLR 2026
MoE blocks. These results indicate that KL regularization helps preserve the expert-routing pattern during quantization and mitigates performance degradation.
19


Reviews:
[{"review_id": "HfsBDpb3gc", "reviewer": "Reviewer_JwHU", "summary": "The paper proposes CodeQuant, a unified clustering + quantization framework tailored for Mixture-of-Experts (MoE) models. Empirical results on several MoE models show improved perplexity. The paper claims up to 4.15× speedup with negligible accuracy loss.", "questions": "- On which layers or experts does CodeQuant fail? Are there cases where AOS rotation worsens quantization error or diverges?\n- How is the trade-off between the number of centroids and quantization precision chosen?\n- What is the formal connection between ACCF loss and quantization error？How do you ensure centroid fine-tuning does not simply overfit the calibration set?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Addressing outliers in MoE quantization is highly relevant since activation and weight spikes are key bottlenecks in scaling low-bit inference.\n- Combining activation smoothing (AOS), column reordering (POG), adaptive clustering (ACCF), and custom LUT kernels forms a complete end-to-end quantization pipeline.", "weaknesses": "- The evaluation omits strong baselines such as Duqant and GPTQ.\n- Applying learned rotations within MoE layers risks disturbing the routing behavior. Although the paper mentions avoiding direct interference with router inputs, no quantitative analysis or bounds on routing consistency are provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PaWVucAjdd", "reviewer": "Reviewer_yk1W", "summary": "This paper proposes a codebook-based clustering method for quantizing Mixture-of-Experts (MoE) models. To mitigate the outlier issue, it combines 1) activation smoothing, which transfers outliers to weights, and 2) a training-based clustering method to reduce weight quantization error. The paper focuses on MoE LLMs, and the training loss is carefully designed to align the token routing behavior of the quantized model with the original model.  The authors validated inference performance benefit via a simulation framework and proposed insights for hardware and software co-design.", "questions": "- What is the quantitative impact of the KL loss on final accuracy? Is there evidence that preserving the original token-expert assignment is necessary after quantization alters the weights?\n- Computational Overhead:** What percentage of the total inference time is spent on the clustering operations? How does the performance benefit differ between the prefill and decoding stages?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "-  The work focus the timely and challenging problem of quantizing Mixture-of-Experts (MoE) models. The training loss is well-tailored for these architectures, giving it practical significant relevance for industry deployment.\n- It provides a analysis on the computational patterns for clustering-based GEMM operations, offering a perspective for future hardware and software co-design.", "weaknesses": "- The technical innovation appears limited, as the method primarily combines previously established techniques of outlier transformation and clustering.\n- The benchmarking and accuracy evaluation are conducted on relatively small models. Since MoE models are typically very large, validating the method on larger-scale models (e.g., Qwen-235B, DeepSeek-R1) would strengthen the claims. Furthermore, evaluation is limited to next-token prediction tasks; testing on more complex reasoning benchmarks (e.g., GSM8K, MATH500) would be more compelling for modern LLM capabilities.\n- While inference performance is compared, a more detailed breakdown is lacking. It would be beneficial to provide detailed testing parameters and analyze performance separately for the prefill and decoding stages, as they have different computational characteristics (memory bandwidth vs. compute density).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X6jDHuj0r1", "reviewer": "Reviewer_8Fns", "summary": "The paper introduces CodeQuant, a quantization method for MoE models. It combines Activation-oriented Outlier Smoothing (AOS), Adaptive Weight Clustering with Centroid Finetuning (ACCF), and Permutation-Invariant Outlier Grouping (POG), and provides kernels for LUT-driven execution. The experiments evaluate both quantization accuracy and latency.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. AOS directly optimizes activation quantization error on calibration data. The use of Cayley transforms to enforce orthogonality is simple and numerically stable compared with naive orthogonalization schemes.\n\n2. ACCF alternates optimization over centroids and assignments, which is well motivated. The additional KL term for router logits is also reasonable and conceptually sound.\n\n3. POG targets a very practical issue: improving block-wise clustering quality under tight centroid budgets.\n\n4. The LUT-based kernel is evaluated via simulation to estimate speedups and memory savings, and the reported numbers appear reasonable.\n\n5. Overall, the motivation is clear, and both the method and the experimental design are appropriate to support the paper’s claims.", "weaknesses": "1. Some of the bolded entries in the tables appear incorrect. For example: (i) W4A4 Embedding, Phi-mini-MoE-Instruct, A-c dataset; (ii) W8A4 Embedding, Phi-mini-MoE-Instruct, WG dataset.\n\n2. The GPU evaluation of the proposed method is based solely on simulation, whereas QuaRot and SqueezeLLM are benchmarked on real GPUs. In addition, there may be non-trivial overhead, especially from POG, which could introduce extra computation or memory traffic. The authors should discuss these potential costs in more detail.\n\n3. The experiments are limited to perplexity and commonsense QA benchmarks. Additional benchmarks such as GSM8K or MATH would provide a more comprehensive assessment of the method’s impact on reasoning-heavy tasks.\n\n4. The paper does not report the pre-processing (quantization runtime). Including this would help clarify the practical cost of applying the method in real deployments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VXShcBrdsE", "reviewer": "Reviewer_BbsM", "summary": "This paper investigates the efficiency of MoE-based LLMs and proposes a vector quantization method named CodeQuant, which incorporates learnable rotation and clustering operations to achieve competitive W4A4 performance. The authors further design optimized LUT kernels to demonstrate improvements in execution latency.", "questions": "1. What is the additional training cost introduced by the learnable rotation matrix?\n2. Could you include comparisons with more competitive rotation-based quantization baselines mentioned in the Experiments section, such as DuQuant and SpinQuant?\n3. Could you report actual latency and memory measurements on real Nvidia GPUs (e.g., RTX 4090 or A100) to support the Accel-Sim results?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed hardware-oriented implementation effectively enhances the efficiency of MoE-based LLMs.\n2. The specially designed component for MoE, such as the KL loss term, is validated to be effective through ablation studies.", "weaknesses": "1. Since learnable rotation has been extensively explored in prior LLM post-training quantization (PTQ) works such as SpinQuant and OSTQuant, the paper should more clearly articulate its key differences and advantages over these baselines.\n2. Some of the compared baselines appear outdated and do not represent the current state-of-the-art.\n3. The proposed kernel is evaluated only within the Accel-Sim framework, lacking validation on real hardware", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)