Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
Authors: Mohammed Nowaz Rabbani Chowdhury
Kaoutar El Maghraoui
Hsinyu Tsai
Naigang Wang
Geoffrey W. Burr
Liu Liu
Meng Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Mixture-of-Experts
Quantization
Theoretical Generalization Guarantees
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a theoretically provable method for efficient quantization of large Mixture-of-Experts models.

Abstract: Sparse Mixture-of-Experts (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this reduces computation, the large number of parameters still incurs substantial memory overhead during inference. Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose a theoretically grounded expert-wise mixed-precision strategy that assigns bit-width to each expert primarily based on their *change in router’s* $l_2$ *norm* during training. Experts with smaller changes are shown to capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, to avoid allocating experts to lower precision that inject high quantization noise, experts with large *maximum intra-neuron variance* are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment.

Published as a conference paper at ICLR 2026
EFFICIENT QUANTIZATION OF MIXTURE-OF-EXPERTS WITH THEORETICAL GENERALIZATION GUARANTEES
Mohammed Nowaz Rabbani Chowdhury's Kaoutar EI Maghraoui', Hsinyu Tsai", Naigang Wang', Geoffrey W. Burr?, Liu Liu', Meng Wang Rensselaer Polytechnic Institute, IBB Research
ABSTRACT
Sparse Mixture-of- (MoE) allows scaling of language and vision models efficiently by activating only a small subset of experts per input. While this re- duces computation, the large number of parameters still incurs substantial memory overhead during inference. Post- Post-training quantization has been explored to address this issue. Because uniform quantization suffers from significant accuracy loss at low bit-widths, mixed-precision methods have been recently explored; however, they often require substantial computation for bit-width allocation and overlook the varying sensitivity of model performance to the quantization of different experts. We propose theoretically grounded expert- wise mixed seeeedprrcision strategy that assigns bit-width to each expert primarily based on their change in router's 's l2 norm during training. Experts with smaller changes shown capture less frequent but critical features, and model performance is more sensitive to the quantization of these experts, thus requiring higher precision. Furthermore, avoid allocating experts to lower precision that inject high quantization noise, experts with large maximum intra-neuren nura-nnuron variance are also allocated higher precision. Experiments on large-scale MoE models, including Switch Transformer and Mixtral, show that our method achieves higher accuracy than existing approaches, while also reducing inference cost and incurring only negligible overhead for bit-width assignment [
INTRODUCTION
The sparse Mixture of Experts (MoE) architecture allows the construction of larger pre-trained language and vision models without increasing training costs (Shazeer et al. 2017: Lepikhin et al. 2021: Riquelme et al., 2021 Fedus et al. 2022, Allingham et al. 2022). In this architecture, the transformer block's ' forward network (FFN) is replaced by muluple FFN modules, each referred to as an expert. Each expert is paired with a trainable router, which selectively activates small subset of experts for each input token. Compared to dense models (Fedus et al.) [002]] (Coowhhry (ct al. 2023), MoE enables faster convergence and reduces the amount of training data required. Additionally, MoE maintains similar inference FLOPs to dense models despite having a larger parameter count (Riquelme et al. 2021 Zhou et al. 2022).
Despite these advantages, MoE incurs substantial memory costs during inference due to their large size, limiting their deployment. Since experts learn diverse features during pre- pre-training. not all are equally relevant for specific downstream task, some recent pruning strategies attempt to mitigate memory usage by eliminating task-irrelevant experts (Chen et al.) 2022a Koishekenov et al. 2023 Chowdhury al. 2024). However, the effectiveness of pruning diminishes in complex tasks, where
larger set of experts remains essential.
Post-training weight quantization (PTWQ) focuses on quantizing weights after training and has emerged as another promising technique for reducing the memory footprint of large language models (LLMs) (Shao et al., [2024] Hubara al. [021; Lin et al. 2024 Frantar et al. 2023 Badri & Shaji
2023). Several works have applied quantization to large MoE models by uniformly reducing all expert
weights to fixed bit-width (Kim al. 2022 2023b Frantar & Alistarh 2024). However, this uniform
"Corresponding author. Email: wangm @mi.cdd
Code available at: http:e////..ttt...o.orrrreeeeaaa

---

Published as a conference paper at ICLR 2026
approach overlooks the varying importance of different experts, resulting in substantial performance degradation under extremely low-bit settings (e.g.. sub-3-bit quantization). Although various mixed- precision quantization methods have recently been developed for other model architectures and could potentially be applied to MoEs, e.g., block-wise mixed precision of MoEs Li et al. (2024), these do not leverage the varying relevance of experts in MoE. models. An expert-wise mixed-precision approach, in which bit-width varies across experts based on their sensitivity, offers greater potential to preserve accuracy under low-bit constraints. Yet, this direction remains largely unexplored. To our knowledge, only two recent works (Li et al. 2024 Huang et al., 2025a) have explored this approach, using metrics such as expert usage frequency and mean routing weights estimate experts' sensitivity. However, these heuristics are suboptimal and lack theoretical justification (Chowdhury et al. 2024). This raises a fundamental question:
What metric provably categorizes experts in the mixed seeeedprecision quantization of an MoE layer?
This paper addressed the question both theoretically and empirically. Our theoretical analysis reveals that allocating higher bit-width to a group of experts with a smaller change in the router's 's l2 norm during training, corresponding to experts that learn less frequently used but important features, while allocating the rest of the experts in lower bit-width can significantly reduce model size without hurting performance. Moreover, allocating some of the experts with high maximum intra-neuron variance to higher bit allows further compression. Extensive empirical evaluations support these findings, demonstrating that large state-of-the ofute-of-the-art (SOTA) MoE models (e.g., Switch Transformer, Mixtral) can be quantized to ultra-low-bie bit regimes (e.g., below 3-bit) without sacrificing accuracy. Our major contributions are summarized as follows:
1. A theoretically grounded metric, the change in a router's 12 norm during training, for expert-wise mixed-precision quantization. We theoretically analyze the training dynamics and generalization behavior of a simplified two-layer MoE model fine-tuned on classification tasks. This model is a SOTA theoretical model in understanding training and generalization of MoEs and general neural networks. We prove that experts capturing less prevalent features exhibit smaller changes in their router's l2 norm during training. We further prove that these experts exhibit lower activation levels. Hence, the model's ' generalization performance is more sensitive to the quantization of these experts, requiring them to have higher precision. Unlike the prior work (Chowdhury et al.) 2024) that uses the router's l2 norm to distinguish between relevant and irrelevant experts for pruning, our analysis offers a finer-grained view that identifies varying levels of expert importance, enabling a principled approach to diverse expert-wise bit-width allocation for mixeddppeeisson quantization.
2. Empirical Validation of Expert-wise Mixed Precision on large MoE models, including Switch Transformer and Mixtral. Our results show that assigning precision based on changes in the router's
l2 norm during fine-tuning outperforms alternative heuristics, such as expert activation frequency and activation weights. Moreover, for large pretrained models like Mixtral, where fine-tuning is computationally expensive, we demonstrate that using the router's l2 norm from the pretrained model alone, without any fine-tuning, achieves test accuracy comparable to the existing expert- muxedd-reccison strategies (Table while reducing inference computation (Figure Importantly, our approach incurs negligible computational overhead to determine expert bit-widths, while the alternative methods require significant GPU computation.
2 RELATED WORKS
Quantization of large models. Parallel to PTWQ, some other methods focus on minimizing quantization eиror through quantization aware (re- )training (QAT), but these methods are every expensive and not suitable for large models (Wang et al.) 2022 Liao et al.) 2024 Gu et al. 2024). Mixed-precision strategies have also been explored, where bit-widths vary across different model components (c.g., MLP blocks, attention heads) (Dong al. 2020 Li et al.) 2024 Huang et al.) 2025b Dettmers et al. 2024). However, intra-layer variation of bit-widths remains underexplored.
MoE compression. To compress MoE models, some approaches focus expert pruning, either targeting specific downstream tasks during fine-tuning (Chen et al., 2022a Koishekenov et al. 2023 Chowdhury et al., 2024), removing irrelevant experts from the pre- prained model (Zhang et al. 2024 Xie et al. (024). However, the effectiveness of pruning diminishes for complex tasks. PTWQ has also been applied to compress large MoE models, with most methods focusing on uniform
2

---

Published as a conference paper at ICLR 2026
2024 which often results in degraded performance under ultra low-bit bit settings. Two very recent works have
quantization of experts (Kim al. 2022 2023a bj Yi et al. 2025 Frantar & Alistarh
explored expert-wise
sexxed-recision quantization for MoE models (Li et al., 2024; Huang et al. 2025a), but they either rely on suboptimal metrics or require extensive memory and computational resources determine the expert-specific bit-width distribution.
Optimization and generalization analysis of neural networks. Several works have established optimization and generalization guarantees for neural networks (NNs) using neural tangent kernel (NTK)-based approaches (Jacot et al. 2018: Lee et al. 2019 Du et al. 2019; Allen-Zhu et al. 2019)
Li al. 2022). However, such analyses can not capture realistic training dynamics they require the weights remain close to initialization throughout training. More recent studies have focused on
the feature learning dynamics of NNs to derive generalization guarantees (Karp al. 2021] Brutzkus & Globerson 2021 Li et al. 2023 Zhang et al.) 2023] Chowdhury et al. 2023), offering better alignment with practical neural network behavior. These analyses are typically restricted shallow networks, and our work falls within this framework.
EXPERT-WISE MIXED-PRECISION QUANTIZATION OF MoE
3.1 THE BASICS OF MIXTURE-OF-EXPEE ARCHITECTURE
In MoE models, the standard feed-forward net- works (FFNs) in transformer MLP blocks are replaced with multiple parallel FFN experts. A gating network of routers assigns input tokens to specific experts.
Taken
Taken
Assending order normichange
Export
Expert
Export
Expert
Higher (by)
(b)
Consider example MoE block that includes k experts, each of which is two-layer FFN. Let = [1) =1 .x(2) :.... (n))
Router 1
Router 2
Router
Routter 4
Rnd denote the
Iil..
Gating Network
input sequence, consisting of n tokens, each of dimension d. For each token z05) Rd with j E [n]. the MoE block produces a d'- dimensional output token, forming the output se-
Taken 1
Token
Figure 1: A schematic of Mfxture-of-Expperr with our proposed approach. Experts with smaller router norm changes in higher bit. Experts with large max intra-neuron variance are reordered.
quence Tout = (1) (2)
Fout (n) E R2nd",
The output Fout E IRd corresponding to the in- put (i) is given by,
( (w/() 02
of if G 0
( Σ fa((6) where, fa(())) (E(M]
(I)
if Gf
Here fz((6)) E Rdd denotes the contribution of the s-th expert. w:(0) gm and w-(a) gd'' represent the weights of the first and second layer, respectively. The activation function a(-) is applied element-wise to the hidden layer output. 0gr denotes the zero vector in Rd",
The Gating Network. For each token x (j E [n]) and each expert (s E [k), the gating network computes gating value G. E [0. The network includes & trainable router vectors w, Rd
(s E [k)), one per expert.
In loken-choice routing(Fedus al. 2022). given input token v Rd. the routing network computes set of routing scores { (w.,, x b ) for all & experts. The top-l experts with the highest scores (where < k) are selected, and their corresponding gating values are computed via a softmax over the top-l scores, while the remaining experts receive a gating value of zero. In contrast, in expert-choice routing (Zhou et al. 2022). each expert computes routing scores x(), J over all n tokens and selects the top-1 tokens with the highest scores. The gating values for the selected tokens computed via softmax over the top-/ scores, and the rest assigned zero.
3

---

For pre-tr models where the initial router vectors whs (0) are unavailable, we use Published as a conference paper at ICLR 2026
3.2 THE POST-TRAINING WEIGHT QUANTIZATION (PTWQ)
PTWQ methods compress neural network weights by representing them as low-bit fixed-point integers. During inference, these quantized weights are dequantized back to floating-point values. Given weight matrix W Rinxout, and a target bit-width b, the de-quantized weights W are computed as,
W A. ([W/A + oux
2 | xnxouu)
where A == ( -1) is the quantization bin size, and == (w)_i/inn////// (W)/A]
2b-1 is the zero-point of the quantized weights. 1 is an all-ones matrix with dimension (in x out),
and [-] is the dlement-wise rounding to the nearest integer.
Most PTWQ methods select A and by minimizing either the loss in activation calibration data (e.g., GPTQ (Frantar et al. 2023), AWQ (Lin et al. 22024)). An alternative line of work aims to minimize the reconstruction error directly, without calibration data (e.g., HQQ (Badri & Shaji 2023
3.3 THE PROPOSED MIXED-PRECISION QUANTIZATION METHOD
Our quantization approach proceeds in two main steps: (1) ordering the experts based on the change in the router's nomn and the maximum intra-neuron variance, defined in Defs. [.d assigning bit-widthh (two-level or three-level quantization) according to the ordering.
and 3.2 and (2)
STEP 1: Expert Ordering. We first introduce two metrics used ordering the experts.
Definition 3.1. For expert E [k), let w. and (T) be its router vectors in the initial and trained models, respectively. We define change in router's l2 norm as follows,
A = ||
Definition 3.2. Let W! be the first-layer weight matrix of expert s the trained model, containing neurons with weights in Rd. The maximum intra-neuron variance evaluates the maximum variance of weight entries in each neuron, i.e.,
MaxVar = max rE/m]
Σ
( [r. Σ w( .))
where (((,.T(rr [r. denotes the i-th element of the r-th row of the matrix W/(.T)
We first rank experts by the change of router's l2 norm A where those with smaller (T) are placed higher, which later correspond higher precision. This ordering theoretically justified in Section 4 and the intuition is that the model performance is more sensitive to the quantization of those experts with smaller (TT)
Then, to assign the experts in higher precision that inject high quantization noise to the model, we adjust the ordering by promoting experts with larger maximum intra-neuron variance higher ranks. Specifically, if lower-ranked expert has its MaxVar! at least ( Bf times greater than Mar (T) of expert s', where s' ranks higher than in the ordering by router norm change, we move 8 to above s' the adjusted ordering. This process repeated until no further changes are needed. The intuition is that larger intra-neuron variance arises either from wider weight ranges or from more skewed weight concentrations, both of which induce higher quantization noise compared to experts with smaller intra-ncuron variance under the same bit-width assignment.
Special Case:
the router's la norm itself as surrogate for (T) since initial weights are typically small-variance random initializations. Max is computed directly from the pre-trained model as well.
STEP 2: Bit Assignment. Based on the obtained ordering, we can assign two-level or three-level quantization as follows.
We use (= = 3 in experiments, since the variance of any bounded distribution is most three times that of uniform distribution with the range. This adjustment affects only 4-5% of experts in experiments.
4

---

Published as a conference paper at ICLR 2026
Two-level assignment. Given bn bt and target average bit-width barg, quantize the top fraction of experts to bn and the rest to by.
Bus bi
bg-bi
Three-level assignment. With wid-wsdths by > bm > by and target Dang. we assign higher-ranked experts to bn. mid-ranked experts bm. and lower-ranked experts bt. In general, multiple assignment strategies are possible according to the ranking order while still achieving the same bavg- We select the best strategy based on the intuition to balance between maximizing the number of experts assigned to the highest precision and minimizing those assigned to the lowest precision, depending on the value of bavg relative to the three levels.
Specifically, when lavg is in (bn, - (hh - br)/3, 3, by). i.e., close to by. we maximize the number of experts assigned to by- When davg is in [bs. 2(b, - b1)/3, b, (b, /r], we again maximize the number of experts in bn. but subject to the constraint that the number in by does not exceed those in bm. When bavg in (bi, bn. 2(bb. b3), we minimize the number of experts in bj.
4 GENERALIZATION GUARANTEES FOR THE ROUTER-NORM-BAN EXPERT-WISE MIXED-PRECISION QUANTIZATION OF MOE
4.1 SUMMARY OF THEORETICAL INSIGHTS
Before formally presenting theoretical setup and results, we first summarize the key theoretical insights. We consider a setting where an MoE model is fine-tuned for a binary classification problem. Each input sequence contains single task-relevant token that determines the label, while the remaining tokens are task- irreeantt.. For each class, there are two distinct task-relevant tokens: one more prevalent, appearing in a (1 - a:) fraction of the data, while the other appears in an a fraction (a < t). Although based on the simplified setup, our theoretical insights are validated empirically on practical MoE models in different language tasks. Our major theoretical takeaways include:
1. Experts specialized in learning less-prevalent tokens undergo smaller changes in their router's 12 norm than experts that learn more-prevalent tokens. We show that different experts specialize in different task-relevant tokens. The routers associated with experts that exclusively learn the less-prews cess-prevalent token exhibit a smaller l2 norm change after fine- sine-ttning, compared to routers for experts that learn more- preveeeee tokens. This observation suggests that the router norm change can serve as a useful indicator for distinguishing between these two types of experts.
2. Experts that learn lrev-prevalent tokens produce weaker activations, and the model's generalization performance is more sensitive to the quantization of these experts. We prove that experts are primarily activated by the task-relevant tokens they learn. Experts that learn less- prevalent tokens generate significantly weaker activations than experts that learn more-prevalent tokens. Therefore, the model performance more sensitive to the quantization of the former ones.
3. Quantizing experts with smaller router la norm changes to higher precision, while rest of the experts in lower precision achieves the same generalization as full-precision quantization. Because the model's generalization is more sensitive to the quantization of the experts learning eesealeevlnntt tokens, and as they can be identified via router's lg norm change, quantizing them b, allows safe reduction of other experts" precision by log, (--a) bits without hurting generalization.
4.2
DATA MODEL AND ASSUMPTIONS
The MoE model and binary classification task. We consider neural network that contains single MoE block, fine-tuned on a binary supervised classification task, where each input sequence x is labeled with y E {+1, -1}. The MoE block generates deddimmnniionll output tokens, i.e., d' = 1 for (1))) in Q1. and the model output is computed by aggregating all the output tokens, i.e., for an input sequence æ, the model's output is
f = Σ
( Dout
Σ
Σ
(2(
(5)
5E[m]
jE(n
(E(A]
5

---

Published as a conference paper at ICLR 2026
Let (.) denote the model after T steps of finetuning. 2 is correctly classified if yf (1)
(x) 0.
For each expert s E [k]. the second layer weights are fixed during training and defined as W :=
al*). 1 11xm, where a(*) E [+1,-1]. We refer to each expert as positively connected to the final
output if (( = 1, and negatively connected (()) = -1. Let S4,5. S_ C [A] denote the set of positively and negatively connected experts, respectively. The activation function a(-) is rectified linear unit (ReLU). The routing mechanism follows expert-choice routing, where each expert selects I tokens, satisfying 1 I L for some constant L.
Although our theoretical analysis is based on a two-layer MoE model, it already captures the key components, including routers, experts, and nonlinear activation, and the learning problem is already highly non-convex. In fact, the two-layer network model is SOTA for theoretical analysis of training
dynamics and generalization in MoEs (Chen et al., 2022b Chowdhury et 2023), and in general deep neural networks (Li et al. 2023 Zhang et al.
2023 Allen-Zhu & Li 2023 Bu et al. 2024).
iwwoprecisioon-eeel quantization. To simplify the theoretical analysis, we consider two precision levels and only the first layer weights of each expert 8, i.e., W(e:r) are quantized. The top s--raction of experts in S+ and the top s--frationn in S_, each with the smallest values of A! are quantized to
the higher bit-width by. while the remaining experts in both sets are quantized to the lower bit-width bt. The quantization is applied in a column-wise fashion: for each expert and its corresponding bit-width, the bin size A is computed independently for each column of w| w(e.T) Without loss of generality, we assume the zero-point = 0.
The data model. Let P C Rdd denote a set of orthonormal vectors with [P] S =d=2((1)) Two vectors 01 and 05 in P, and their negatives -01 and -02 are called task relevant, denoted by set P. = #0 while all vectors in Po } are task-irrelevant.
Each sequence and label pair (x, 3) follows distribution D, where contains exactly one token from Pr, which determines y. sequences containing =o1 are labeled as class (i.e., = +1), and those containing =o2 are labeled as class (i.e., y = -1). The remaining tokens in z are drawn independently from the dusk-irrelevant set Po J. each with probability (1/d). With probability a, where the constant a (0, 1/4), ) a sequence contains the less prevalent telkvntl tokens 01 or O2, and with probability 1 - a: it contains the more prevalent tokens -01 or -2
Our data model is similar to Bu et al. (2024) except that our iireeeetele vectors are drawn from an orthonormal set instead of Gaussian distribution. The assumption of orthonormal task-irrelevant
vectors have been widely deployed in theoretical analysis (Brutzkus & Globerson 2021 Shi et al.
2022 Zhu & Li 2022 Chen al.
2022b Zhang et al. 2023 Li et al. 2023
We next introduce some useful notations in presenting our theoretical results. After training iterations, we define the activation of expert in response task-relevant token as follows:
Definition 4.1. For expert 8 E [k:), the activation of expert s by a task relevael vector defined as o(t == I (W<(ss))
v), V U
(6)
where w(s:0)) is the first-layer weights of expert s at iteration t, and I is an all-ones vector in R..
Intuitively, a lower activation of an expert for a task-relevant vector leads to a smaller gap between the output of this expert and the output of another expert not selecting the token, leading to weaker predictions against quantization noise.
The same Chowdhury et al. (2024), we define expert's proficiency measure to quantify the router's s ability to select task-relevant tokens from a sequence. Specifically,
Definition 4.2. The proficiency of expert after I. training iterations in selecting lask-relevant vector u is measured by the probability that it assigns a gating value of at least 1/1 to token v, i.e.,
pl (..1)) = pP {) 2 1/1 (1) u for some j E [mi]],
V VEP,
Alignment of the pretrained model. In a pretrained model (t = 0), we say the router for expert s is aligned to task-relevant vector (v E if pl. 2(11, i.e., it selects with nontrivial gating
Fixing the output layer for analytical convenience is standard in the literature and has been adopted in prior works (La & Liang 2018 Brutzkus et al. 2018 Arora al. 2019 Zhang et al. 2023 Chowdhury al. 2023

---

Published as a conference paper at ICLR 2026
value for constant fraction of samples containing t. We assume that in the pretrained model, routers of the experts in S4 are aligned to either 0₁ or -01, and routers of the experts in S_ are aligned to either 02 -02. This assumption reflect the common intuition that pretrained MoE model learns to specialize experts for different subtasks or feature types,
Let S. (u P,) denote the set of experts whose routers aligned with in the pretrained model. Let max( So, Smyl denote the maximum fraction of routers that are aligned to less-prevalent [ [
task-relevant vectors in 5+ and S_.
4.3 MAIN THEORETICAL RESULTS
Lemma 4.3. Suppose the pretrained model is fine-tuned for T ((((//ool// iterations, the returned ((T) has the following properties,
(i) the routers' alignment to cask-relevant vectors are enhanced during training, specifically,
=1 1, (a.T) -. E Su, Vu { =
(8)
(ii) the l2 norm change of the routers aligned with the more prevalent -1 and 00 higher than those aligned with the less prevalent Or and o2, specifically,
A
(TT).
Vs So,, E S_o,, = 1,2
(9)
and (iii) the expert activation by -o and 00 are higher than that by on and o2, specifically,
o ( Ω (1=00) ml log?
(e T
1 - 2a
(x.T) = (2 log!), 0%
-o 2
(10)
do. (x.T)
2a
Lemma summarizes key properties of the fine-tuned model that can leveraged for expert-wise mixed precision implementation. First, (8) shows that if the router of expert is aligned with mixed-preeisee task-relevant vector the pretrained model, that is, p. ((..0) 1211, then after fine-tuning, the
alignment becomes stronger: Pe (e.T) 1. Moreover, the expert s suppresses - by assigning zero or negligible gating value to the negative token -v, i..., (= 0. This implies that each expert becomes specialized in single task-relevant vector after fine-tuning.
Second, (9) shows that the change in the router's l2-norm larger for experts aligned with more prevalent features, and smaller for those aligned with less prevalent ones. This property allows us to distinguish two types of experts based on their router norm changes. Third, (TO) demonstrates that experts aligned with less prevalent vectors produce weaker activations than those aligned with more prevalent ones, resulting from the less frequent occurrence of these tokens in the data. The ratio of their activations is at least (1 22x)(2a.. Thus, the model's generalization is expected to be more sensitive to the quantization of the experts aligning with less prevalent vector, and hence these experts need higher precision. Lemma 4.3 is verified by synthetic data in Figs. and in Appendix A
We next formally establish Theorem 4.4 that characterizes the generalization guarantee of the quantized model r8²) after applying the two-level mixed seeccs-preiision quantization method in Section
3.3 the tunet model ((T), Let Var(s.T) denote the variance of the r-th column of W!(s.2) Theorem 4.4. Suppose the number of fine-tuning iterations satisfies T O (log)//),, and max,e(m) Var(e.1) (11) for every expert s. If i 2 7, and the two quantization levels satisfy
by 2 logg (1 2(d (dd (og((ookkmd)) (ogg!
(11)
and
by 2 logz I- R(dv hot///og(kkk/////mm log log!)),
a
(12)
then with high probability the quantized model has guaranteed generalization, i.e.,
P(((,,,)) DD:
( (x) Du:
=1
(13)
7

---

Published as a conference paper at ICLR 2026
Remark 4.5. Theorem 4.4 states that if the maximum intra-neuron variance of all the experts close to each other, i.e., V's [R+] MaxVar!") (11) sorting the experts in ascending order of their router norm change A!7", and quantizing the top re-fraction (k 2 7) of experts in this sorted list to by bits and the remaining experts to by bits, where bn and by satisfy conditions (11) and (12), allows the quantized model to preserve the generalization of the full-precision model. Each low-precision expert can use log, (1=2) fewer bits than its high-precision counterpart. This is verified by synthetic data in
Fig. in Appendix A Note that all the experts aligned with less prevalent vectors are among the top k-fraction (by Lemma 4.3 (ii) and exhibit smaller activation values (by Lemma 4.3 (iii). Therefore, they require higher precision. In contrast, the experts quantized to lower precision are those aligned with more prevalent vectors and have larger activations, and hence can be quantized aggressively.
5
EXPERIMENTAL RESULTS
5.1 EXPERIMENTS ON FINETUNED SWITCH-TRAND REEHHHTTTAASRREEEEE MODEL
Here, we present quantization results on Switch Transformer (Fedus et al. 2022 finetuned on CNN/Daily Mail (CNNDM) text summarization task (See et al., 2017). All non MoE weights are quantized to 8 bits. We apply HQQ (Badri & Shaji 2023) for quantizing the mode See section B in appendix for more implementation details.
Two-level expert-wise
As shown in Figure uniform 3-bit expert quantization nearly preserves generaliza- tion, while uniform 2-bit severely degrades the generalization. We therefore use mixed- -eeccdprr with two bit levels: 3 and 2.
16.2
X unifeem
-o- act freq
19.3
-G- act weight
Rouge-2 Score
$8.0
Our method outperforms existing expert-wise mixed- precision methods: We benchmark against two prior expert- wise sexxed-rrecison strategies: (i) activation frequency (aver- age tokens routed per expert) and (ii) activation weights (aver- age gating weights on a calibration set) (Li et al. 2024 Huang iet al. (2025a), where
FP-32 Seore: 19,87
17.
router sorn change (ours)
moter nonn change
(ours)
17.3
22
28
Avg. bits/expert
experts are as- signed higher precision. In contrast, our corac-e ordering itself preserves generalization down to 2.5 average bits/expert, outperforming both baselines. Furthermore, addi- tional reordering by Max Var affects only 3.7% of experts, extending preservation to 2.125 bits.
Figure 2: Expert-wise mixed-
precision of Switch Transformer on CNNDM. Bit choices: 2, 3
5.2 EXPERIMENTS ON PRETRAINED MIXTRAL MODELS
We quantize the pretrained Mixtral 8x7B (46.7B parameters) and Mixtral 8x22B (140.6B parameters) models (Jiang et al. 2024) using GPTQ (Frantar et al. 2023) to evaluate on eight zero-shot benchmark LLM tasks. All non non-MoE parameters are assigned to 3 bits. See section B in the appendix for details.
Baselines. We compare against the state-of-the-art expert-wise method, Pre-loading Mixed-precision Quantization (PMQ) (Huang et al. 2025a), which assigns bit-widths by minimizing Frobenius-norm output errors (per expert and bit level) weighted by activation frequency and gating scores. As PMQ outperforms the activation frequency and activation weights based methods, the comparison with these method are in Appendix (see Figure 7 [ in Appendix). We also evaluate non-expert-wise approaches, including layer-wise (Hessian (Dong et al. 2020), BSP (Li et al. 2024g) and group-wise (Slim-LLM (Huang et al. 2025b)) methods. Our method has advantages as follows.
Three-level expert-wise mixed precision. We consider three-level bit- bit-assignment of (1,2,3) bits. As shown in Table 1 the uniform 3-bit expert quantization almost maintains generalization, but uniform 2-bit quantization significantly degrades performance.
High accuracy with robust scaling. Our method surpasses PMQ above 2.0 average bits in terms of accuracy for Mixtral 8x7B Extending to Mixtral-8x22B (140.6B parameters), our method
'We use eight V100 GPUs for fine-tuning and one NVIDIA A5000 GPU (48GB) for quantized inference. $Compressing helow 2.0 bits/expert is too aggressive, since the compressed model size ($13.1 GB) falls helow the equivalent dense model (13.6 GB) that is trained with far more data and has better generalization.
8

---

Published as a conference paper at ICLR 2026
again outperforms PMQ, demonstrating robustness to model scale. It also outperforms exp-ex- wise methods (e.g., Hessian (layer-wist) wise) (Dong et al. 2020). BSP (layer-wise) (Li et al. 2024). and Slim-LLM (group-wise) (Huang al. 2025b)) by large margins.
Low inference cost. Figure shows inference time on Wikitex12 (Merity et al.) (016). For the same average bits/expert,
(Mand
PMQ
method is
Unitom
faster than PMQ because PMQ assigns higher precision to frequently
lnffeetrc come
activated experts, whereas our method allocates higher precision less frequent experts, reducing computation.
500
Negligible assignment overhead. Unlike PMQ. which requires evaluating all experts across bit levels on calibration set (e.g., 110 GB GPU memory and 2227s for Mixtral-8x7B: 350 GB and 6000s for Mixtral-88 Mixtral-8x22B), our method only sorts experts by router norm with minor reordering. This requires no GPU and negligible computation, enabling scalable compression of large MoE models.
2.3 Aug. bnatoppr
Figure 3: Inference time dif- ferent expert-wisg wise methods.
Table I: Task accuracy (%) of different methods the benchmark LLM tasks.
Avg. bitwexp.
Memory (GB)
Model
Method
PIQA
ARC=e
ARC-
BoolQ
HellaS,
Wino.
MathQA
MMLU
Avg.
Full prel-precision
(FP)
96.8
83.68
83,50
59.64
85.05
83,99
76.4
41.61
67.85
72.72
Uniform
3
19.3
82.32
80.05
57.42
86.09
81.51
75.14 14
39.43
64.84
70.85
2
13.1
76.44
67.68
43.60
72.51
72.93
65.27
28.58
42.79
58.73
2.75
17.7
81.83
80.47
56.31
85.57
81.05
74.98
38.29
61.55
70.01
2.625
16.9
81.45
78.62
$4.86
85.60
80.57
74.66
36.75
59.78
69.04
2,5
16.1
80.79
78.41
54.44
85.14
79.36
74.35
36.28
58.23
68.38
Rouler
2:375
15.3
80,20
75.38
51.19
84.92
78.69
73,80
33.47
56.07
66.72
MMx
73.95
(Ours)
2.125
13.8
78, 94
74.45
51.02
80.12
76.56
70.17
31.29
51.56
64.26
74.17
Mixtral
L.75
11.7
75.03
69.53
42.92
73.64
70.03
68.35
27.04
44,82
58.95
8x7B
2.75
17.7
82.05
78.87
56.48
84,80
75.30
38.39
61.79
69.85
2.625
16.9
81.56
78.41
52.99
83.67
80.04
74.66
38.26
59.76
68.67
2.5
16.1
80.63
76.94
53.33
83.15
80.02
74.9%
37.15
$4,05
67.53
2.375
15.3
80.47
73.32
50.00
81.93
78.54
74.66
35.18
53.34
65.93
PMQ
2.25
14.5
80.14
72.14
49.32
83.15
77.62
74.19
33,70
51,00
65.16
2.125
13.8
79.00
75.55
72.26
72.30
34.07
2.0
13.1
76.93
71.93
46.59
78.65
74.88
73.24
31.83
48.60
62.83
2.5
17.0
80.21
76.38
51.20
81.11
78.05
72.97
35.27
56.21
67.18
Hessian
15.3
79.21
79.15
71.25
2.0
13.6
75.32
67.26
45.01
70.29
71.90
69.11
31.07
40.85
58.85
BSP
17.0
68.16
62.19
Slim-LLM
2.0
13.6
61.70
49.07
28.24
66.18
44.10
57.54
23.62
25.43
44.49
Full Full-precision
(FP)
77.40
Uniform
3
57.5
81.45
76.68
53.07
78.53
74.23
68.19
36.21
55.46
65.48
38.6
55.96
22.78
57.92
50.12
21.64
2.5
46.7
80.14
71.25
47.27
73.49
65.11
64.40
30.62
48.
60.10
Mixtral 8x22B
(Ours)
2.0
38.6
78.56
64.18
41.30
68.26
61.91
61.25
27.14
40.09
55.34
L.75
35.2
75.14
59.59
2.5
46.7
79.49
70.62
46.84
74,28
68.64
65.35
29.58
50.40
60.69
PMQ
2.25
43.0
2.0
38.6
76.55
66.16
39.85
69.48
60.11
$9.83
27.00
39.43
54.80
1.75
35.2
72.20
56.90
32.59
39.33
49.43
57.38
24.82
30.30
47.87
5.3 ABLATION STUDY
We determine the importance the two stages of the experts' ranking: the router norm based ranking and the maximum intra-neuron variance (i.e.. Max based reordering by providing an ablation study among (i) only Max based ranking, (ii) only router norm based ranking, and (ii) router norm based ranking + Max Var based reordering described in section 3.3 We conduct the study on Mixtral 8x7B for the eight zero-shot benchmark LLM tasks. We report the average accuracy across the tasks for both the teoelevel assignment (bit choices: 2, 3), and threc-level assignment (bit choices: 1, 2, 3) in Table 2 As we can see, for the two-level assignment, only router norm based ranking performs better than only MaxVa based ranking for most of the average bit points. However, for the

---

Published as a conference paper at ICLR 2026
threc-level assignment, there is an abrupt drop in performance in the only router-norm- based ranking as some of the unusually large MaxVar experts are placed in bit (see our MaxVar visualization of Mixtral 8x7B in Appendix|E] E)), which injects an unbearable amount of quantization noise into the model. Reordering these experts (only 11 out of 256 for Ç 3) to higher rank completely removes this issue and significantly outperforms the only MaxVan based method and other competitive baselines provided in Table [ of the paper. We provide an empirical justification for our selection of Ç in Appendix
Table Average accuracy (%) of different expert ranking methods
Tevelevel assignment (hit choices: 2,3)
Three level assignment (bit choices: 1,
Method
Avg.
bits/expert
Avg. bitslexpert
2.75
2.625
2.5
2.375
2.25
2.125
2.75
2.5
2.25
2.0
1.75
66.01
63.88
60.65
69.37
67.90
63.97
60.44
Router norm
69.92
68.35
67.01
64.97
63,84
61.54
54.23
49.78
48.12
44.96
42,92
Router borm Max (Our meibod)
70.01
68.38
65.79
62.56
5.4 JUSTIFICATION FOR USING FINAL ROUTER NORM AS SURROGATE FOR CHANGE IN NORM OF PRETRAINED MoE MODELS
As stated in section 3.3 for the experiments on zero-shot evaluation of pre-trained models, we propose to use the final router norm (w: (T)) to approximate the change in the router's norm (A!T)), when the randomly initialized model is publicly available for computing the initial router norm (w."), The rationale behind the approximation comes from the fact that the initial routers are generally initialized randomly with small variance (e.g.. parameters of DeepSeekMoE are initialized randomly with variance 0.000036 (Dai et al. 2024)), which leads to a very small difference between the two methods. We provide theoretical justification of our claim in Appendix G Here, we provide an empirical justification for the claim by reinitializing the routers of the pre-trained switch transformer randomly from N (0, a²) with a 0.0005. We finetune the re-initialized model on the CNN/Daily Mail dataset and compare the rank correlation between the two expert ranking methods measured via Spearman's p and Kendall's 7 (p, T ss 1 implies high correlation). The results are provided in Table The high rank correlation implies that the rank orders using both methods very similar.
Table 3: Correlation between final router norm and change in norm across different layers
Enc-1
Enc-3
Enc-5
Eac-7
Enc-9
Ene-11
Dee-1
Dec-3
Dec-5
Dec-7
Dec-9
Dec-11
Spearman's 0,9997
0.9994
0.9995
0.9992
0.3989
0.9990
0.9990
0.9997
0.9995
0.9997
0.9998
0.9999
Kendall's r 0,9950
0,9900
0.9920
0.9871
0.9851
0.9861
0.9871
0.9960
0,9920
0.9950
0.9960
0.9980
We provide the quantization results for both methods in Table 4
As expected, due to the high correlation of the rank order between the final norm and the change in norm based method, the scores of both methods are very similar.
Table 4: Quantization results for final router norm and change in router norm based method
Initial router
Original pretrained
Random router
Method
Full-precision
Full-precision
Change norm
Final norm
Avg. bits/expert
32 (FP)
32 (FP)
2.75
2.5
2.25
2.75
2.5
2.25
Rouge-2 score
19.87
19.46
18.79
18.60
18.37
18.81
18.59
18.38
6 CONCLUSION
This paper proposes expert-wise mixed-precisio quantization method for MoE models that allocates higher bit-widths to experts with smaller router norms and lower bit-widths otherwise. It can use pretrained router norms alterative to avoid costly fine-tuning while maintaining accuracy. The approach is theoretically supported and empirically effective on large MoE models. It reduces memory and inference costs, promoting energy efficiency and a smaller carbon footprint. Future work will combine the method to waser-wise and block-wise quantization.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the 2024 IBM PhD fellowship. in part by the National Science Foundation (NSF) under Grant 2430223, in part by Army Research Office (ARO) under Grant 1000000000 and in part by the RPI-IBM Future of Computing Research Collaboration (http://aire.rpieedd)) part of the IBM Al Horizons Network (htt:///bmmbiz/AlHHriiooz
REPRODUCIBILITY STATEMENT
We provide the complete setup for our theoretical analysis in section 4.2 We include additional details related to our analysis Appendix H We provide the proof Lemma in Appendix and the proof of Theorem 4.4 in Appendix K We provide the details of our experimental setup, including model architecture, parameter size, values of the hyperparameters related to the implemented quantization methods, and the evaluation datasets in Appendix B We include the code of our experiments for reproducibility.
REFERENCES
Zeyuan Allen-ZNt Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. In 2021 IEEE 62nd Anmual Symposium on Foundations of Computer Science (FOCS), pp. 977-988. IEEE, 2022.
Zeyuan Allen-Zhu and Yuanzhi L.. Towards understanding ensemble, knowledge distillation and self- distillation in deep learning. The Eleventh International Conference on Learning Representations, 2023. URL httpt: http:://oppeeeeeeeee /tpps//e///peeeeeiieee net htt.://ww..e..orrummmmooooooooo.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. In International conference on machine learning, pp. 242-252. PMLR, 2019.
James Urquhart Allingham, Florian Wenzel, Zelda Mariet, Basil Mustafa, Joan Puigcerver, Neil Houlsby, Ghassen Jerfel, Vincent Fortuin, Balaji Lakshminarayanan, Jasper Snoek, Dustin Tran, Carlos Riquelme Ruiz, and Rodolphe Jenatton. Sparse moes meet efficient ensembles. Transactions Machine Learning Research, 2022. ISSN 2835-8856. URL/https: /ttp//ppppreee..eeooee net/ forummid=i 000e0-ii003ddd2200 Expert Certification, Expert Certification.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Kedzes-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. MathQA: Towards interpretable math word problem solving with ceraatoon-bsee formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and
Short Papers). pp. 2357-2361 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1111///1//111142///
URLhttps: ac www.hologhoyy....oomooooooo org/N19-1244
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International conference on machine learning, pp. 322-332. PMLR, 2019.
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Alon Brutzkus and Amir Globerson. An optimization and generalization analysis for max-pooling networks. In Uncertainty in Artificial Intelligenee pp. 1650-1660. PMLR, 2021.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev-Shwartz. Sgd learns parameterized networks that provably generalize on linearly separable data. In International Conference Learning Representations, 2018
11

---

Published as a conference paper at ICLR 2026
Dake Bu, Wei Huang, Taiji Suzuki, Ji Cheng, Qingfu Zhang, Zhiqiang Xu, and Hau-San Wong. Provably neural active learning succeeds via prioritizing perplexing samples. In Proceedings of the 41st International Conference Machine Learning, pp.
2024.
Tianyu Chen, Shaohan Huang, Yuan Xie, Binxing Jiao, Daxin Jiang, Haoyi Zhou, Jianxin Li, and Furu Wei. Task-specific expert pruning for sparse mixture-of-expert. arXiv preprint arXiv:2206.00227, 2022a.
Zixiang Chen, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. Towards understanding the mixture-of-experts layer in deep learning. In Advances in Neural Information Processing Systems, pp. 23049-23062, 2022b.
Mohammed Nowaz Rabbani Chowdhury, Shuai Zhang, Meng Wang, Sijia Liu, and Pin- Yu Chen. Patch-level routing in mixture-of-experss is provably sample-efficient for convolutional neural networks. In International Conference on Machine Learning, pp. 6074-6114. PMLR, 2023.
Mohammed Nowaz Rabbani Chowdhury, Meng Wang, Kaoutar El Maghraoui, Naigang Wang, Pin- Yu Chen, and Christopher Carothers. provably effective method for pruning experts in fine-tuned sparse mixture-of-experts. In International Conference on Machine Learning, pp. 8815-8847. PMLR, 2024.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. (eXiv:8803.05557/0 2018.
Damai Dai, Chengqi Deng, Chenggang Zhao, Rx Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1280-1297, 2024.
Tim Dettmers, Ruslan A. Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. SpQR: sparse-quantized representation for near-lossless LLM weight compression. In The Twelfth International Confer- ence on Learning Representations, 2024. URL hettps://openreee
net/ forum?? d=
@lu25ahSuy
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trace-weighted quantization of neural networks. Advances in neural information processing systems, 33:1 1551-18529, 2020.
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International conference on machine learning, pp. 1675-1685 PMLR, 2019.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Elias Frantar and Dan Alistarh. Qmoe: Sub-1-bir compression of trillion param- eter models. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.). Proceed- ings of Machine Learning and Systems, volume 6, pp. 439-451, 2024. https://proceecingss http://...........o...........pa (11108888666111111333366--00---0
URL
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan-Adrian Alistarh. Optq: Accurate post-
training quantization for generative pre-trained transformers. In 11th International Conference on Learning Representations, 2023.
Yuling Gu, Oyvind Tafjord, Bailey Kuchl, Dany Haddad, Jesse Dodge, and Hannanch Hajishirzi. Olmes: A standard for language model evaluations. arXiv preprint (tt.:2400008444 2024.
12

---

Published as a conference paper at ICLR 2026
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference Learning Representations (ICLR), 2021.
Wei Huang, Yue Liao, Jianhui Liu, Ruifei He, Haoru Tan, Shiming Zhang, Hongsheng Li, Si Liu, and XIAOJUAN QI. Mixture compressor for mixture-of- -eepeertss LLMs gains more. In The Thirteenth International Conference on Learning Representations, 2025a. URL htttpss//oppereeieeeee
[Os
Wei Huang, Haotong Qin, Yangdong Liu, Yawei Li, Qinshuo Liu, Xianglong Liu, Luca Benini, Michele Magno, Shiming Zhang, and XIAOJUAN QL Slim-LLM: Salience-driven mixed-precision quantization for large language models. In Forty-second International Conference on Machine Learning, 2025b. htthttts:://// /pt.ttps//oeeneevii..... www/ffrruute (o-orummm1-PPPb
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry, Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pp. 4466 4475. PMLR, 2021.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088. 2024.
Stefani Karp, Ezra Winston, Yuanzhi Li, and Aarti Singh. Local signal adaptivity: Provable feature learning in neural networks beyond kernels. In Advances in Neural Information Processing Systems, pp. 114883244897 2021.
Sakaguchi Keisuke, Le Bras Ronan, Bhagavatula Chandra, and Choi Yejin. Winogrande: An adversarial winograd schema challenge at scale. 2019.
Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Who says elephants can't run: Bringing large scale moe models into cloud scale production. In Proceedings of The Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). pp. 36-43, 2022.
Young Jin Kim, Raffy Fahim, and Hany Hassan Awadalla. Mixture of quantized experts (moqe): Complementary effect of low-bit quantization and robustness. arXiv preprint 2r.0v:2310.024100
2023a.
Young Jin Kim, Rawn Henry, Raffy Fahim, and Hany Hassan Awadalla. Finequant: Unlocking efficiency with fine-grained weight-only quantization for llms. arXiv preprint 2302v22308.09723, 2023b.
Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. Memory-efficient nllb-200: Language-specific expert pruning of a massively multilingual machine translation model. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023.
Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. (GS) haadd: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Representa- fions, 2021. URL htttpss:/
foret//ooummiii wtt.:orrumumddaaaeeeeeeetm.
Hongkang Li, Meng Wang, Sijia Liu, Pin- Yu Chen, and Jinjun Xiong. Generalization guarantec of training graph convolutional networks with graph topology sampling. In International Conference on Machine Learning, pp. 1301--3051. PMLR, 2022.
Hongkang Li, Meng Wang, Sijia Liu, and Pin- Yu Chen. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. In The Eleventh International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2026
Pingzhi Li, Xiaolong Jin, Yu Cheng, and Tianlong Chen. Examining post-training quantization for mixture-of-experts. A benchmark. arXiv preprint 2xXi::2406.08155, 2024.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. Advances in neural information processing systems, 31, 2018.
Baohao Liao, Christian Herold, Shahram Khadivi, and Christof Monz. Apiq: Finetuning of 2-bit quantized large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20996- 21020, 2024.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583- 8595, 2021.
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, 2017.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In ICLR, 2024.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of-experrs layer. In International Conference Learning Representations, 2017.
Zhenmei Shi, Junyi Wei, and Yingyu Liang. A theoretical analysis on feature learning in neural networks: Emergence from inputs and advantage over fixed features. In International Conference on Learning Representations, 2022.
Naigang Wang, Chi-Chun Charlie Liu, Swagath Venkataramani, Sanchari Sen, Chia- Yu Chen, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Leland Chang. Deep compression of pre-trained transformer models. Advances in Neural Information Processing Systems, 35:14140-14154, 2022.
Yanyue Xie, Zhi Zhang, Ding Zhou, Cong Xie, Ziang Song, Xin Liu, Yanzhi Wang, Xue Lin, and An Xu. Moe-pu poe-ppuner: Pruning mixture-ou -fcure--o-eree experts large language model using the hints from its router, arXiv preprint arXiv:2410. 12013, 2024.
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Empowering sparse large language models on mobile devices. IEEE Transactions on Mobile Computing, 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
Shuai Zhang, Meng Weng, Pin- Yu Chen, Sijia Liu, Songtao Lu, and Miao Liu. Joint edge-model sparse learning provably efficient for graph neural networks. In International Conference on Learning Representations, 2023.
Zeliang Zhang. Xiaodong Liu, Hao Cheng, Chenliang Xu, and Jianfeng Gao. Diversifying the expert knowledge for task-agnostic pruming in sparse mixture-of offerress--eppers... arXiv preprint arXiv:2407.09590. 2024.
Yangi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y Zhao, Andrew M. Dai, Zhifeng Chen, Quoe V Le, and James Laudon. Mixture-of-expert with expert choice routing. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
14

---

Published as a conference paper at ICLR 2026
A VERIFICATION OF THEORETICAL RESULTS ON SYNTHETIC DATA
On bi loq (L-20
Couppoons Stag
u 0-25.5
0=0 0.1875
0.125
0.0025
Ronter ascending onder zomm)
Ratio activations experts learning more and less prevalent tokens, respectively (class-1)
Figure 5:
Figure 4: Router vector pro- jection along the class-1 task- relevant feature 01.
Figure 6: Change of bit differ- ence between higher bit experts and lower bit experts with
We validate our theoretical claims using synthetic data generated as described in Section 4.2 Tokens are drawn from an orthonormal matrix obtained via QR decomposition of d x Gaussian matrix with d = 200. We set k = 20, = 800, = 100, and = 5. Model weights are initialized from zero-mean Gaussian distribution with variance 0.0001 and trained with a learning rate of 0.2.
Figure shows the projection of the router vectors onto the direction of O₁, class- task-relevant vector, for the experts in S.. The experts are sorted in ascending order based on the change in router norm. Routers exhibiting larger norm changes tend to have significant component along the more dominant -0, direction, consistent with Lemma (i). Figure presents the minimum ratio of activation of -Oi-aligned experts by -0 that of Oraaligned experts by Ø1, minimized over all such expert pairs. This ratio is compared against the theoretical lower bound of (1 /2x, as established in Lemma 4.3 ii).
Experts with large compo- nents along the -0, and 00 directions are quantized to a lower bit-width by. unless high maximum row variance is observed, in which case they are quantized to higher bit-width b, The value of bn is determined empirically as the minimum bit-width required to achieve zero test error when all experts are uniformly quantized to by. We then choose by as the minimum bit-width that still maintains zero test error in the mixed-precision setting. Figure shows that the gap b,, by increases as a decreases, aligning with the theoretical bound 1ogg((1 /2x) discussed in Remark 4.5
We quantize the weights as described in Section 4.2 using equation (2)
B DETAILS ON THE QUANTIZED MODELS AND EVALUATION TASKS
B. SWITCH TRANSFORMER
We fine-tune pre-trained Switch Transformer (Fedus et al. 2022) which contains 64 experts per MoE block CNN/Daily Mail (CNNDM) text summarization task (See et al. 2017). The model follows an encoder-decoder architecture with 12 transformer blocks each in the encoder and decoder; every even numbered block an MoE block, resulting in 12 MoE blocks total. The model has about 2 billion parameters, with 90% residing in MoE. blocks. All non-MoE weights are quantized to 8 bits. We apply HQQ (Badri Shajil (023) for quantizing the model. Weights each row of the weight matrices are quantized together.
B.2 MIXTRAL.
We quantize the pretrained Mixtral 8x7B and Mixtral 8x22B models (Jang et al. 2024), which adopt a decoder-only architecture. The Mixtral 8x7B contains 32 transformer blocks, and the Mixtral 8x22B contains 56 transformer blocks. All blocks are MoE blocks of 8 experts. Mixtral 8x7B contains 46.7B parameters, with 97% residing in the MoE blocks. Mixtral 8x22B contains I 140.6B parameters, with 99% residing in the MoE blocks. We quantize the non-MoE parameters to 4 bits and apply GPTQ (Frantar et al.) 2023) for model quantization with group size 128, and 1% damping. We use 128 samples of length 2048 from Wikitext2 the GPTQ calibration data. Model performance is evaluated on eight zero-shot benchmark LLM tasks: PIQA (Bisk et al.) 2020),
and ARC-Easy (Clark et al. 2018). BoolQ (Clark et al. 2019), HellaSwag (Zellers et al. 2019)
15

---

Published as a conference paper at ICLR 2026
WinoGrande (Keisuke et al., 2019), MathQA (Amini al. 2019), and MMLU (Hendrycks et al.
2021 using the EleutherAT LM Harness (Gu et al. 2024).
JUSTIFICATION FOR THE SELECTION OF C
As stated section 3.3 we select ( = 3, since the variance of any bounded distribution is at most three times that of a uniform distribution with the same range. Our selection of Ç alters the initial router norm based order by very small amount (only 11 out of 256 experts are reordered in Mixtral 8x7B, and only 28 out of 768 experts reordered in Switch Transformer for selection of () Indeed, our selection of ( only reorders the experts that have unusually large MaxVar values in an MoE layer (see our MaxVar visualization of Mixtral 8x7B in Appendix E We conduct sweep of for Mixtral 8x7B on the eight benchmark downstream tasks and report the average accuracy in Table
As we can see, the performance picks around Ç 3.0, which justifies our selection.
Table 5: Avg. accuracy (%) for different values of (
Avg. bits/expert
1.0
2.0
2.3
3.0
4.0
5.0
2.0
60.44
61.56
62.28
62.56
61.32
61.74
D MORE RESULTS ON MIXTRAL 8x7B
act weight
70
froq
max (ours)
Avg. accumicy
-Q router norm (ours)
2.1
22
13
244
2.5
2.6
3.7
2.8
Avg. bits/expert
Figure 7: Expert-wise mixed-precision results for Mixtral 8x7B eight benchmark LLM tasks; expert bit-choices: 2, 3. Only 4.3% of the experts are reordered to higher ranks maximum intra- neuron based reordering.
- act weight
Avg. nccuracy
froq
47
37
1775
Avg. bits/expert
Figure 8: Expert-wise mixed-precision results for Mixtral 8x7B eight benchmark LLM tasks; expert bit-choices: 2, Only 4.3% of the experts are reordered to higher ranks in maximum intra- neuron based reordering.
16

---

Published as a conference paper at ICLR 2026
E VISUALIZATION OF MaxVar FOR MIXTRAL 8X7B
4.00
LMG
Sugus
Suges Voderr
Lapen
Lapen
Accoddig
Tayse
Layer
E2
Expus E
Liper:14
Exper
Figure 9: Visualization of the maximum intra-neuron variance of the experts in Mixtral 8x7B. Only handful of experts (11 out of 256, colored in red) have unusually large Max Var values.
17

---

Published as a conference paper at ICLR 2026
F VERIFICATION OF THEORETICAL INSIGHTS IN PRACTICAL MOE MODELS
For accurate verification of our theoretical insights, first need identify the task-relevan tokens of different experts, which hard to determine in practical MoE models. This is because, for some experts, many tokens can be task-relevant, but each of them may appear less frequently in data. On the contrary, for some experts, only few tokens can be relevant, but each of them may appear very frequently in data. We consider the tokens with high gating values the task-relevant tokens of each expert.
Larger router norm experts exhibit higher activation. We verify our claim that larger router norm experts exhibit higher activation in practical MoE models. To verify that claim, we plot the average activations of tokens with top gating values (sampled from the WikiText2 dataset) for each expert in the first five layers of Mixtral 8x7B. The results are provided in Figure I0
Avg. activation
Avg. activation
rodar nom
Rare Rodrr
X:00
muner 1om
--8--
20
6
12
20
No. of tokens
No. of tokens
(a) Layer 0
(b) Layer
11.84
Nader
-0- Rexter0
rourr 12m
outar tom
185
Avg. activation
103-
Roder
Avg. activation
Rexter2
11.
No. of tokens
No. of tokens
(c) Layer 2
(d) Layer 3
Hinoo
Avg. activation
Notee
No. of tokens
(e) Layer 4
Figure 10: Average activation over tokens with top gating values of different experts of Mixtral 8x7B. The top one/two router norm experts exhibit larger activations.
18

---

Published as a conference paper at ICLR 2026
As see, the largest router norm expert (in some cases, the largest two) always exhibits significantly large average activation compared to other experts, except for the second layer. In this case, the lowest one has unusually high activation. However, from maximum intra-neuron variance, i.e., Max Var visualization given Appendix [R] we can see that this expert has an unusually large MaxVar value than other experts. Therefore, this expert will be placed on higher bit regardless of its position in the router norm order according to our method.
Finally, visualize the top gating value tokens (sampled from the WikiText2 dataset) through their corresponding model input embeddings for the first MoE block (closest to the model input embeddings) of Mixtral 8x7B. The visualization of the first few tokens with top gating values (highlighted in yellow) of the smallest, and the first two largest router norm experts are provided in Figure 11 along with their adjacent tokens.
The trackways called batrachchhii. are usually found in strata deposited around opholis considered a prolscartiform reptie <0x04==<(x00>=<0000
<0x0Ao-<0r0A> Chad Benekrs Chizad) - guitar| 1994
(Dx04-<Dx00--000 w00t000000000-00000-----0-000000 Oxaziidine deriveives were first reported the mid
Mauritania, Moracee
Portugal, Marino,
=: Amalgamation = -00db--Oob---bcck--dd---------
On
Bettalion near Yongsan, Stragglers from this position continued stream next and Geer recrulted gultarist Chiadd, bassist Mowk, crummer
was to occupy French North Africa: Morocae, Algaris, and Turisia
Battallon amalgamated with several other special
Company, but Jenson collected str stragglers from it and seized high ground astride this main
<O wotokko---CC---000---------
Ovazindino-
An
twenty
day
a. 740 hectares 577 q-@
2005, American Society of Magazine Editors unvelldd its of the top
Comunitaria Lorenzo Tenochillánn Texistepec. stands
(a) Expert 7 (smallest router norm)
co0tt-cii0oo-tiett The battles of the American C /Wr and at lscs ware very influeniial on the designs
undred Days, his dofest the Battle Waterioo the pursult of his amy and hinsaff
Soundireck cOaDAoxobOOo <hebAe-dircko--eeee Selected
services used to se provided by the Metropolitan Wotorworks and Sowerage System, which served main guns -00000--0000--0000
World Worlll the guns used Type 91 armour
coming, 12 August. As passed through Waterfoo Place, on the edge of the Sogoide Britain World Warlll and decining health.
Operation
German insasion of USSR, Operation marbermnee began, and USSR became Dryigha byxDA--OAAD= Kora Saludong (The Kingdom of catching enemy commerce raiders Wortp to
completed
= Fire control and sansora = cbblk--dada--chhooo-ddo Downtown coNoAo indpencientl was cer fied platinum the Argentine Chamber Phonograms and Videogram
referred
refgieus community was disparsed curing the Eng tan Civil War butwean Parlamentarians and Royalist but reformed, ending
(b) Expert (second largest router norm)
hill accupied by D Company was in reality the western tip of a large mountain mass that original estimates were 6 @-@ 10 hPa
progression the Doctor was "a different phase of his life now and phoons 66 greater than normal. Philippine
character development, although felt was blunted by the knowlodge that series
Lolita Nevertheless, Hentzi believed that the film 's themes of materialism and conform 17. Charles El Wagner of the Exhibitor Trade Review
legs into the ringpost, as he did to Spanky and Gowen, and Inter
got my belt * years later, the former French colony made its début the 1964 Summer Olympics in Tokyo
injury Mir,
unification
video received a premier on MTV 's Headbangers Ball on June
was Jewish sage who lived in the time of the Mishna prominent supp
graded it to Tropical Storm Parma October 2 Park area commission affecting the population. Annually,
established arbitrate claims emanating from British
surrounding them that are most likely mucous coatings. They
(c) Expert (largest router norm)
Figure 11: Token visualization of the smallest and the largest two router norm experts
19

---

Published as a conference paper at ICLR 2026
As can see, the lowest router norm expert (Expert-7) activates subwords of unusual names/nouns (e.g., batrachichni, prolacertifomm, Chizad, Oxaziridine, Morocco, Amalgamation, Stragglers, hectares, Tenochtitlán). Each of them is rare in data, but be critical in the context. This verifies our intuition that the lower router norm experts learn critical but infrequent tokens. On the other hand, the largest router norm expert (Expert-6) activates many common full words of the English language, such as pronouns (e.g., that), prepositions (e.g., to, in, on), etc. The second largest router norm expert (Expert-1) activates on sentences implying war or military operations, which are common in many documents. This verifies our claim that the larger router norm experts learn more frequent tokens.
G THEORETICAL JUSTIFICATION FOR USING FINAL ROUTER NORM AS A SURROGATE FOR CHANGE IN NORM OF PRETRAINED MOE MODELS
As stated in section 3.3, for the experiments on zero-shot evaluation of pre-trained models, we propose the final router norm ([Iwe")) (]) approximate the change in the router's norm (A = 11 [D. as the randomly initialized model is not publicly available for computing the initial router norm (Iws (Iw The rationale behind the approximation comes from the fact that the initial routers are generally initialized randomly with small variance (e.g.. parameters of DeepSeekMoE are initialized randomly with variance 0.000036 (Dai et al.) 2024)). that case, the initial router norm differences among the routers are too small to alter the change of router norm based order when approximated by final router norm.
Specifically, for any two routers (router 1 and router 2), if router I's change in norm is larger than router 2's change in norm, i.e., A(T) > A!" then
A (T)
A!
(2) > 0
(1lo) (T) - || 10) I]) (1lw2 (T) || 1| (0)
(1lw) (T) |um (T) [1) - [l|w] (0) || 111 I|) 0
Now, due the small-variance initialization, |||= (0) 1|m0 (0) ||| is very small quantity. Therefore,
it is highly likely that ||w] 11 || > 0, long A!7 (2) (T) is not too close zero.
Based on the above intuition, we provide a formal theorem to justify the claim:
Theorem G.1. Let the routers of the initial model be randomly initialized from ((0,a²) with = (1/dd) Then, with probability at least 1 1
de, for any two routers $1,82 E [k] such that
AST Asz = 2(1/Vdd) we have [[wer (T) > || [
Proof. As, (T) A (T) (21///d), we have, ||
1 2 (1 (0) - 1| +2(1/va).
w
Now, with probability 1 - d?' we have ||:!
wu:2 (0)
= = O(1/vd) for our selection
of a which completes the proof.
The theorem confirms that, for small-variance initialization (i.e., ((1d)), the final norm based order preserves the change in norm based order for any two routers unless they are very close to each other (i.e., A! A (11/((()))))))
20

---

Published as a conference paper at ICLR 2026
H
PRELIMINARIES
For any fine-tuning iteration the equation 5 can be represented as,
x (x) Σ (x) where, (x) a
m
Σ
G(out)
> ReLU ((u()0))
(14)
(EJ
Here, 1 (x:) C [n] is the indices of the tokens of the input sequence x that are routed the expert s [k] at time and wf
(e.0) is the r-th column of W((.") Note that ( (x)) L.
As analyze the expert-choice routing, for any j E _ (Ir), the gating value Gfaet) evaluated as,
Gfa0)) exp((wu?) ,,()))
(15)
EiEl exp((w! (t) x(1))
We analyzed the case where the model is fine-tuned to minimize the Hinge loss
(0, v) max(1 y (z).
(16)
while the gradients evaluated on similar to the setting of Zhang et al. 2023
( (z)
(17)
For any input (x, y). the gradient for the column E [m] of W((,.() is evaluated as,
(x.
ys())
Σ
(f()))2000) 20)22
(18)
(E) (x)
and the gradient for the router is evaluated as,
01 v)
(*)
Σ
Gla
(2l1) - x())
(19)
Ow
(E (0a)
(E)
where, (a.1) = En - ReLU (0.1) ,.))))))))
We consider that the model fine-tuned via Stochastic Gradient Descent algorithm (SGD) with batch size B, where the expert weights are updated with learning rate and the router weights are updated with learning rate 7/-- The batch gradient for the column E [m] of (s.t) is evaluated as,
Øl
1
(1()) (x,z)
(20)
Ow (a.1)
B
(w.(..0)
and the batch gradient for the router w, is evaluated as,
Øl au (x, 8) aw.
(21)
ow:
B
ZEB
Notations:
1. Ô()) and (2-)) hides the factor log(poly(d) with sufficiently large polynomial poly(.)
2. With high probability (abbreviated as w.h.p. refers to the probability 1 poly(d)
21

---

Published as a conference paper at ICLR 2026
Definitions:
For any 9 PMo01002) we define the activation of the expert s E [A]] by g as, (a.t) 0q
E m == ReLUU(wr (e.0) .g)).
For any E Pr, we define complementary expert proficiency measure for the expert at time t as,
p = P (a. ~ D : :jj E !! such that x f/3i [m] such that x! = v)
Note that (s.1) 2 pD
(s.4)
Without the loss generality, we assume that for any 8 E Se, p/4,0) = O(1dd) We define,
G(+:0); Gating value of the token x(0) for some [n]] and e E P, at expert and iteration t
G(*:1); Gating value of the token x(0ر q for some j E [n] and q Pioo, 02) at expert and iteration
Jl (x) x00 / is the number of copies of the task-irrelevant vector ((x)0) == | E
q Plfo,, 02) in the set of top tokens for the input sequence expert s and iteration
{ w 00)
and C2 max (1055000) DE]K)
We define C1 max
sE[R]
Without the loss of generality, we analyze the case that 1 2 e20s
Therefore, Vs E
[k]. w
logl.
|S for } and 7o for E {#o2).
[S [s_
We define, %o S+
Therefore, max max(7m) (we 1v)
Without the loss generality, we assume that Vv E EP, No = ((1).
We define, C, == min((w)) (00 We assume that Cp >
We assume that for any v E P, and any 8 E S,, (( w))00 2(1),
and ||S5.| S_|| = O(vk).
Components of the routers' gradients.
For any input (ar, y) DD the router's gradient component of the expert [k] along any task- relevant vector E Pr and along any task-irrelevant vector E Po 02) at iteration are evaluated as follows:
if Ej E E (2) S.t. ا00 =
(0(xz v) !t .9
if E (x)
yal ((()))(())) c(aut)
Σ
of (o.00 o((.0)
(0.)
s.t. (1) =
(x)
(22)
22

---

Published as a conference paper at ICLR 2026
0
if if E (s)
s.t. x(1 =
and ((d+() = -
@l(a,y) v)
v)
ya G( (x) (x) (x) (sse)(ss/ ((a(t))))))) (f (ofta0) ofb.0))
if if E _
Ow!!
s.t. xهح
yals) (x) (0) G/a00) ( (o(2x)) - of(ull)
if E J! (x)
s.t. x(: (23)
Components of the experts' column gradients.
For any input (z, ~ D, the gradient component of the column r E [m] of w((*t) along any task-relevant vector v E P. and along any task-irrelevant vector q E Pl po 02) iteration f are evaluated as follows:
if (uu .4) 0
0l(2, v) Ow: (...5.17.9
0
if (w/ .4) 2 0 but, Aj E J! (I) s.t. x!1)= (24)
(v) (x) ( ((..)) ((,)))
if (wr
(a.t) , 2 and, 3j 1! (z) s.t. z(1) =9
0
if (wt
.b) < 0 but Aj ( (z) s.l. z(1)
ya G(11)) (x)
if (e.))
e) < 0 and
Ej E (1) (x) s.t. z(5) =v
(l(x,)) Owl(s
(25)
0
if (w (a.1)
0) 2 0 but
Aj E ح (x) s.t. x(5) =
ya ya G(a00 (x) if (w (e 0) 2 0 and Ej E " (z) s.t. x05) =
I PROOF OF LEMMA 4.3
Proof sketch. Lemma 4.3 provides the results for training dynamic analysis of the analyzed model. Primarily, our training dynamic analysis provides insights about the learning characteristies of the experts learning different task-relevant tokens. Moreover, the analysis provides necessary bounds of the router norm changes and expert activations required for the mixed-precision quantization analysis, along with the generalization guarantee of the trained model. We categorize the training into two phases:
(i) The expert alignment phase
iii) The router-expert co co-learning phase
(i) The expert alignment phase. Given the relative alignments of the routers to different task relevant tokens, the expert alignment phase confirms that, regardless of the initial alignment of the columns of the expert-weights (i.e., the columns of w-(*)) they sufficiently align with the task-relevant tokens to which their respective routers are initially aligned to. Therefore, the batch gradients during the SGD updates for the router weights maintain large components along the initial alignment direction after this phase. We quantify the number of iterations required to complete this phase of training, along with the bounds of expert activations by different task-relest relevant tokens after this phase (see Lemma I.5 and Lemma
23

---

Published as a conference paper at ICLR 2026
(ii) The router-expert co-learning phase. After the expert alignment phase, due to the large batch- gradient components of the routers along the initial task-relevant token directions, they become further aligned to these directions in the subsequent updates of SGD. This allows the expert weights to be more aligned with the task-relevant token directions of their respective routers, further increasing the routers' batch-gradien components along these directions. Therefore, the routers and the experts co-learn the task-relevant tokens at least by quadratic rate. Hence, the model generalizes after this phase of training. However, due the larger frequency of more-prevalent tokens, the experts learning them receive larger updates in their router and expert weights, allowing larger norm change and expert activations after training, compared to other experts. As shown in Lemma [43] we quantify the sufficient number of iterations required to complete the training, along with the router norm changes and expert activation bounds for different experts.
Lemma 1.1 (Full version of Lemma
4.3 Suppose the expert learning rate ne the router learning
Cp
rate 7,, 0
the batch size B = {ddf), and the pre-trained model is trained for Cll
PC
log l
T=
(26)
amll
Cp
iterations. Then, the returned f(T) has the following properties:
(i) For all E So and P, {to,, +02), have
pb: p(o?)) 0, and
1 U) = v for some j E [n], plur) > 2
(ii For all 8 E So and s' E S Soop i= 1,2, we have
A > AT)
(iii) For all 8 E So and s E S Sop 1,2, we have
log
(xT) = Ω ml
0
Cp
(3,T) = Ω
(1 a) mlC2y
log l Cp
-01
(E'T) T)
2x
2 2a
(s
0o:
(iv) For all g E 02,,02), 8 So, v V P, ,,-
{ t02], and v E \ (+u), we have
d( = 0mmC2)
d (3.T) O(mC2).
Proof. (i) Let consider E S.. From Lemma J.5 we can show that, for T 0 1Cg
am
4o <t S T', and for any q P Pofo, 02), ww)
9) < log
Therefore, using Lemma J.4 and Lemma J.5 by selecting = 92(d²) we have,
Ou! (TT) 01) < - amC2
0l
ai
am
Therefore,
01) 2 ②
Jun (
mC
On the other hand, from Lemma J.5
Ow J 4) (s.1' +1)
0
d
Therefore, poi poi
(a,T'+1) (e.T')
and p p
[s,T')
24

---

Published as a conference paper at ICLR 2026
Ou! (T)) 01) 2 -0(mC2), for our selection of yr, we have (w +1)
01
Again, as ( Therefore.
01 - -q) - 22 log I.
(4 01) 5 -S2( mmmm
Øl
a
0l
2 S2( a aLmmle
02(
and hence
-01)
Ow Ow!
Ow. Oug [T++1):
12
O amC2
On the other hand,
Ow. +I" 9) S 50( mmee
O2
+ 0 mC²,
) and
+1 9) 2 -0(
al
(
mmle
00
mC²
d
d
Ow:
d2
d
Therefore, for any t s.t. VT s S t 1, if for all q E P\for 02) it holds that (wl)) 01 -9) -)) S
2 log!. I, by induction we can show that, po; (s.7)
(s,t')
and p p:o - p ((ait))
2 po
In that case, we have, Ol
01) -2( a mole
2 and hence
01
a ammel
+ l armC2
1)
52(
01) 2 52(
Ow (3)
12
1
Ow, (1)
12
On the other hand,
Ol
① S 0 mme 1) 0 mC2
Ø1
mile - 0 mC2
) and
① 2 -0
Ow. O
d
Ow (
Therefore, (ws (t) 01 9) 2 (w:
(T)
01 q) 22
a
12 17p (t T*)3 + -2(
ammC2
7r (t )))).
Now, we can show that, (wa
(1")
we :9 01) OCp)). Also.
(0)
(ww) (1) - logl. l. V2
.0 -0)
1
P22 log
Therefore, we need T 0
one V
Cp
steps to ensure that, for all task iask-irrelevant pattern g.
(w!T) .01 q log.. I. In that case, for any t 2 T", ((s.1) and Vz(i) o1, Gf.a)) N
Now, if there exists q' Pi{01, s.t. (w: (T-1)
01 - 9/) 2oggl, then for any q P Pot
for which (wo! (1) 01 - 9) S logl, we have,
-9- - 9) S (1 + log 1 + 0 12
(w: (T-1) ,0 - 9¹) = (w)
(2' 11- 01 4) + (w0 (T-1)
7
log l) as,
(w. 11) .q 1') S
1
log l + 0 - log I). This creates contradiction.
[lgg!)
/2
Therefore, VT <t' < T, we have for all iakeiirrlevvant pattern g, (w 01 q) S 2 2ogo I.
3
3
Now, (T ,)))) > log Therefore, (w ; <
.o1) 2
2 <r>,<<-- <-olog l.
Therefore, for any E P\o0i, 02). (w! 01 - 9) < log 1 - (w! .g)-
2
12 log I 0
On the other hand, (w. (T) (0) .
Therefore, (w> (T)
- q) < 0, which implies
add
(la.T) O.
Similarly, for any E P, and any E S,,, we can show that p( (c.T 1, p_ 0. and
(r for some j E [n), G. ( 1
2
25

---

Published as a conference paper at ICLR 2026
(ii) Let s Son and s' S-o1. From the proof of statement (i), know that, we have for any (0)
{w=2)) . -g> 0 12
o. E P(i(01, such that, for any t s.t. t T,
(uo')) (t)
o -q)
d log 1).
12
Similarly, [ . -q) -)- (w o' 9) = 0 a²d d
log().
Now, for any 1, for any task-irrelevant pattern 4- (ws (w)++1), .0 - q) I " 01 q) + O(amm2n,t) + Oaa2mma,,
Therefore, at least up to t O iteration, for E 02)
(ws) 01 -) S 3logl, which implies VL > (1)) for all q E Pi(or, (w! 01 q) 3logl, 1, and hence, Vz- = 01
Gf (1 (1- - 12*
1
Therefore, for any 6 <> Ti, for any task-irrelevant pattern q.
(wa (++1) .1) S (w: for all iirelireelvaant pattern q (ws
(Ti) ..)) + O a
p mmCon T1)) + 0 a²
1 ( me - T)) which implies,
(T) 01) S (w. (Ti) .11 + Olog()).
Now, as there exists task-irrelevant pattern g such that, (w!T), 00 9) Lllog,, L, we have, (w.") (T) .01) - (w?") (0) 01) < logg
Now, for any i, we have, 01
s 0 - t) + O(mne) + O(mC2).
Ow: (2) 02)
d
(ws (T)
! o2) s ((VC)) Similarly, (w!?) w o2) sO(Vo).
Therefore,
On the other hand, as shown in the proof of (i), for any q E (P]oo,,00) 02 (w! .g) -ww 10) .g)
(2 log I O a²d
Therefore, ( < log
Now, if ATT > AAT) does not hold, then
(T)
(T)
||
< 4.5 log
Therefore, (w g' 01) < 4.5 log which implies, for any a E Pifor, 02), (w -0
(T)
-01-0) -9) < 5logg
12 (w -0 -g) < 5 5logl, then Va!t) = -01, and 17 <
as, (w (7) g) (w> . = 0 a²d
(0)
log!).
However, if for all E
T. (1 (1 Da 2 3/a
aT
Now, using the same procedure as in the proof of (i), after T
< 1-
steps, we have for all
q E Pifor 02). (w (T
3
_01) > log I, which implies, 4 ST
2
(w! (t+l) -01) > logl + 2(
3
(1 a1a) a)
mt - T")) + 52(
(1 a)²
(t-
2
14
1
Therefore, (T -01) > log + 2 2
3
a)² (1)
22 log l) which implies AT) > A(T)
(iii) Let us assume s E So, and s' E S-o. Then, Vr [m] such that (wo(..0) .01) > 0. from the proof
of (i) have for any L, ((.t)) po; (e).0) which implies, Vi T, ( (w.
Øl
(.u)) 01) < - () + ò 1
I/B
which implies. Vr [m] such that (wl .o)) 2 0. vt T. (w) (2,t+1) ,01) 2 (w) (w/ ,01) + 20 (
Oole for the choice of B o(.)).
26

---

Published as a conference paper at ICLR 2026
Therefore, Vr E [mm] such that (1ot"00000)
,0) 0.
(wi
(c.T) (w) .01)
(wt tal .1)
2
03/ (TTT)
21122
log Cp )
which implies dol
(s.2'
A(mlC2 AmmlC2ylog//Ca
aT
Again, using the same procedure as the proof of (i), after T" 1
we have, voli
1-a
-01. cft
and Vr E [m] such that (wt (4'00) -01) 2 0, we have (w) (o(t)T)))
-01)
/ logi
Cp
Therefore, have, Ar E [m] such that (w -01) 2 0,
(wf''1) (F r) ,01) Ω
(1 (1 p222
a.)
log l
which implies (6612)) 52 (1101111 1
m (1=00mt0
log Cp
Cp
Similarly, for E Soz and s' Sooz can show that (s.T) ([mml22//ogg/C/) and o -02
(x'TT)
1 mll mFPC2 C2/
log1
α
Cp
Now, suppose, T K K= 12
log'
where K is the constant satisfies equation (26).
an7
Cp
Then, for any r [m] of s So, such that (wf (a.0) .1)) 2 0, we have (wf (T.T)
01) s C2
log! K & Cp
Again, for any E [m] of s E S., such that (wf (s,0)
.0)) < 0, we have (w, .01 <0.
Similarly, for any [m]] of s' SSoo s.t.
(w (0)00) .) -o1 2 O, have
logh K PC
log!
(w) ( T) -o) 2 2(1C2) Cp 2 Cp
Therefore, (f /oh
()
(s.T) (1.T)) 2 -2a))/2a
Similarly, we can show that for any E Sog and s' E Smog o / (o00)) 2 (1 - 22)/2a.
(n. (x.7)
(iv) Now, Vs [k], Vy E P)f01.02) 02) and Vr [m] such that (w) .9) 2 0, Vt, ( Øl
dw (2,0) 9) 2
0 1 ō
VB
Therefore, (w/ (s,T')
(4) < (w.
(3.0) .g) .g)+0 +0 = C2 C2+0
1
log
C2) = O(C2) which implies
ad
Cp
0 (a.T) = =(mC2).
Again, Vs E S+, Vt and, Vr E [m]] such that (w-
(3,0) (.0) 02) > 0 we have,
awv 02) > 0, and
Øl
for all r E [m] s.t. (w (s.0) -02) 2 0, al
(w (x,T') .02) < C2 and
-02) 2 0 which implies Ow (sa))
(wf (eTT) -02) < C2. Therefore, (a.2) 0(mC2) and x (a.2) 0(m22).
Similarly, we can show that Vs E S_, do. S_ (a.7)
= 0mm2)) and o ( = 0(mC2).
(x.1)
27

---

Published as a conference paper at ICLR 2026
J LEMMAS USED TO PROVE LEMMA 4.3
Lemma J.1. Let, S D such that p= P (xx, v) ~ D: (x. v) E S). Then, w.h.p. any randomly
sampled batch Be of size B at the iteration I, ns - By = Ô (vB)
Proof. Let define random variable X associated with any sample (xr, y) D such that, if (x. S
X
0 if (x. $
Therefore, X Ber(p).
Now, for any randomly sampled batch B, ((1111()) (222,)2)..., (F yB)} of size B, we can denote the B i.idd random variables following the same distribution as X by X1, X2, XB corresponding to the B samples of the batch, respectively.
Therefore, |Bns| Ee= X₁.
Now, E [ s|]
h Σ E [X,] Bp.
B [Bns] Bp = =Ō(VB) 21 poly(d)
Therefore, using the Hoeffding's inequality, P completes the proof.
which
Lemma J.2. For any expert So with u, v 02) such that + v. any q P (01, and any E [m), w.h.p. over randomly sampled batch of size B we can ensure that,
01
mC²
mC²
(i)
4)
0
+
Owss
(0)
d
VB
al
1
(ii)
(..0).0) (5 9)
50
+
VB)
al
< (amC2) +0 (
( (11-a) ml2 ( +0 mC2
(iii)
Ows (0) v)
VB
(iv)
Ou (3
v) < -2
7
a
+
I/B
if (Ww (a.0)
,0) 2 0,
0l
1) 0
(1-a)
a)
+
1
if (1o(..0)
.0) O.
Ow
d
VB
(.. dl
1) 2
a
2
δ
BB]
if (w/
.)) 2 0, and
2 0 if (x.0)
Ow...00)*
1)
u) <0
dl
mC2 vB
(v)
Ow (0) 0²)
$ 0 (mC2) + +
Ou (5,0 v²) 2 0 if (wr (a.0) ,,²) 2 0,
al
(vi)
dl
v/ > -0 ((1 - a)) Ô
if (w) (uff00) (2.0)
,2/) <00
/B
28

---

Published as a conference paper at ICLR 2026
(w(3,0) v²) - 0 (a) (a)) +0
1
if f(w/
,20,
2 0. and
VB/
Owl (.0) v) - if (w!=00) .00) <0
(a.0)
Proof. For any Pr and any g E Pl (((0,,00);
(a.0) (s.0) Ov 0 = Ere ReI.U((W, (5.0) .)) - Ereim ReL.U((W (x.0)
.0) = (mmC2).
Similarly, for any d' E Pion 02) such that y # d' (s.0) of (x.0) = 0(mC2).
We denote Bo as the randomly sampled batch before the first update of SGD.
01
1
(x, v)
ow. 0
B
Ow: 10)
B
Let us define the Sq (0)
((x, v)
D: Aj E J s.t. x11) q) and,
Pq (e.0) P (z.
D (I, E S
Here, P4
(=.0)
7
01
0((, v)
1
al y)
Therefore,
9
Σ
Ow: (0) 9)
Ow: (0)
B
Ow
(0)
B
(0)
BEBanDi,
Now, from equation 22 for any (x,8) Di.5"" !
Øl v)
Ow, (0)
al
0l v)
Therefore,
9
Ow:
B
Ows (0)
(0)
zEBons?
Now, for any (I, v). (x.0) ((..0)
Therefore, as (e.0) og
(a.0)
(mmC)) and for any d'Pifo 023 such that + q. 10.10/000 (1.0)
Bo n S (()
(6,00) 0(mC2), from equation 22
al
9)
O(mC2)
Ow
(0)
B
(0)
Bo
1
Now, from Lemma J.1
w.h.p.
Pq
(s.0)
+
+
1
which implies,
B
al awi
C2
mC
m o
VB
(ii) Using equation (24 and the fact that for any (a, v) D. (*,1) c(( and by following the same procedure as in the proof of the statement (i) we can complete the proof.
(iii) Let us define the set, 5.5"" == (xx v) ~ D 3j E J( s.t. (1) = v}.
29

---

Published as a conference paper at ICLR 2026
Now,
IP (z, v) D (z,y)= -
P (x.y) v) D: (z. E _
y= +1 and Ej [n]] s.t. z01)
x (k. ~ D: y +1 and Aj E [n] s.t. x011 v)
On the other hand, p':,0) p==,0)==000000 O(1/d).
=
Now, using equation complete the proof.
23 by following the same procedure as in the proof of statement (i), we can
(iv) Let us define the set, S.. 00 : (2, ~ D: D: E ! s.t. x00 = and ((=.0)) 2 7
1
Now,
PP (x, 9) ~ :: (2, E s. 10)
= P (x,1) ~ D: (x, E s. [0)
= +1 and Ej E [m] s.t. x00) =
x P
(x, ~ D y = +1 and Ej E [n] s.t. x00) v)
-!0.00 a
?(a)
[As. (x.0) 2(1)
2
On the other hand, plo,0) (1/d).
Now, using equation 25 and by following the same procedure in the proof of the statement (i) can complete the proof.
(v) Using equation complete the proof.
23 and by following the procedure in the statements (iii) and (i)
(vi) Using equation 255 and following the same procedure as in the proof of statement (ii) and (iv)
we can complete the proof.
Lemma J.3. For any expert E S, with v, d' E (-01,--0 -02) such that v + v', any o2)
and any r E [m]. w.h.p. over a randomly sampled batch of size we can ensure that,
Øl
mC² d
mC²
(i)
(o) !)
50
+
Ow
BB
(ii)
Ow G.op 9)
00
+
/B
+ +0 (Emcs)
(mC2
(iii)
Ow (0) v)
+ +
/B
al
(1 (1-a) a)
is (u (a.0)
(iv)
Ow (5,0) v) < -Ω
+
.u) 2 0,
1
I/B
Ow. 6) 0
dl
+ ò
1
if (wo (x.))..)
.0) < O,
VB
30

---

Published as a conference paper at ICLR 2026
0l
(1 a)
-0 -0(1)
0) 2
if (w! .0) 2 0. and
2
VB/
d
0) 2 if (w/ .0) <0
Owl (3,))
0l
mC²
(v)
Ows (0) 1/)
s (m(2) +0
VB
(vi)
0l
1 2 0 if (wt (2.0)
,00) 2
Ow! (+,0)
01
1) > (a) - -0
if (wt (c,0) ,20) 0.
(3,))
/B/
(w. =01-(()))) (()-))) + +
01
1
if (w//+0) ,v) N 0,
and
/B/
Our (2.0) v)) - 0 if (w, (c.0) ,²) <0
Proof. Using the same procedure as in Lemma J.2 we can complete the proof.
Lemma JA. For any expert s E [k]. any v E Pr, and at any iteration t, every q E P\ for, 02) that
satisfies the condition (w?² (1) .) < (w.s (t) v) also satisfies the condition (w/ (8) u) (w?))
.9) S 2 log I,
then for any j E ( where x = e and G. 2 1/1, we have, G( (1 (1- 2 47
G((,))
Proof. If for all E Poon, o2) with (w. ( .4) < (w, : o) we have, (w! u) - (w:, 9) < 2 log L
then V(z,y) D s.t. Àj E (x) with z01) v and G(oot) (x) 2 G(a0)) (I), Vi E J
(x) (x)
and
(1-1)
i i we have, (x) (x)
of (x)) (x))
minf
12
(1-1). 1)
1
12
(1 + 12
)2
(1 +
)2
1
(1
12
Now,
(1 (1 - 1) 113
1
Now, let there exists a constant C > 0 such that 1)
2 T + 1 0) - ((1+20) +
1)2
CI2 + 2CC 0.
Now, CP2 + 2C1 - C > 0 as 1 > 2. Therefore, [3(1 + 20) S 1a(1 - C*) satisfies (01-C)-M(1 (11+ +C +CC1- C20.
1-1
Now, 1-1 2
Now, ((1+20) < 14(1 - C) + C <
1 > 2. Hence, picking C = [+2 7
T+2
4
(2(1 1)
satisfies that
2 41
which implies (z) ((a.0) (x)) z 41
(12 +1 - 1)2
Lemma J.5. For any expert 8 E So such that v E 02). and Aq E (2) by selecting
1) Cp and B = 5 (d?), we can ensure that after T = 0 112
7 =
C2
iterations,
ame
(i) of (s.TT) = f2 (mC2), o o (aTT)) = 0(mC2), og (xTT) = 0(mCz)
(ii) (err)) 2 pe (5.0) and, pl. < p(a.0)
31

---

Published as a conference paper at ICLR 2026
Proof. Suppose al
01 From the statement (i) of the Lemma 12
w.h.p. over randomly sampled
mC
2C
batch.
Owg 10)- 9) s 0
+
0
d
VB
Therefore,
Kwh (1) 9) - (w/ 10) .g)
S 0
mC² Tx) + Ō
mC²
Tr))
VB
On the other hand, from the statement (iii) of the Lemma
w.h.p over a randomly sampled batch, mC²
J.2
al
(1 (1-9) ( 0
.1) - {(ammc)+
+0 +0
mC2
Ow
VB
( mC2n) + +0 mC2
Therefore,
( - -wx (0) 01) 2 O(amC2n)) + 0
VB Rr))
d
Now, by selecting //- O Cp
(0) .q) < (w03 01) we get, (222) for (w)
amC² and B = 5
(mra .1) (wo! .9) (2(Cp) which ensures that poi
(e.1)
2 poi
(s.0)
Similarly, we can show that, (uss (1) 01 - g) S 2 logl and plo) p.000)
Now, for any E [m]] such that (w!+.0) .0) 2 O, from the statement (iv) of the Lemma J.2 ol
w.h.p. 0
Ouf(..0) 01) s -
a)
+ O )), and for any E [m] such that (w(=0) ,0) < 0,
1
Ouff 01) S 50( (1-a) =)) + O(1/VB), which implies, for () .1) 2 0,
d
(wf (1.1) .01 (w/a0) .1) + s
.1) < O. (m77 Ō Te and for (w ,2) < 0, (wf= (
WB
Hence, dot
(s.1) (s.0)
2 doi
+ l
amanee
Ō
mmle I/B" B
Similarly, using statement (ii), (iii), and (iv) of Lemma [.2] we can show that,
(a.1) (e.0)
Oaamm))) + Ō m
<00.
VB ne).
(x.0)
1
+ Ô ( m
(x,1)
(s.0)
_ S o
+
mille
03 2
203
d
VB"
(a.1)
(s.0)
<0 m Ne)) + Ō
mile
09 og
d
VB
Therefore, by selecting B = {2) get,
miJe 4) S <0(0)
00(-
+ 0
mC2
Ow. (1) 9) 2 -0
al
MiTe
O
mC²
Ow (1)
d
d
d2
d
a²mon mm
Øl
(1) () 01) S 50(am(s) ②
Sw D (I)+ 01) 2 -0(mm(2) - mm)c)
Ows
1
01
Øl
01) -0(amC2) + 52( a²mme
Jul -1) S (0mm22) + 0l
dm (1)'
(Condition 1) Suppose, there exists a T" such that v0 < 1 5 T", (a.)) 2 po: ppe00) pl000)
(a,0)
32

---

Published as a conference paper at ICLR 2026
Now, if condition 1 holds then, doi 2 (3,0) + 2 am which implies we need T = 0 102
(x.))
1
ame
steps to ensure that, oo (x.7) ((mC2)) Also, as ( S o (s.0) + 0 0' mmneT we have,
(..T) O(mC2). Similarly, o4 (s.T) O(mC2).
Again, if condition 1 holds, then using Lemma 0.2 and equation (22) and equation (25 that, v0 S t S T", we have,
we can show
01
②) s (
me
0
mC2 d
Øl
4)
2 -0
Mille
1)
0
-0-00-
mC²
Ows (3)
Ows
d²
d
01
a mme
01
01) < O(amm2) 12(
01) -O(amO2) - Oaa2mall))
Ow. (1)
Ow (2)
01
Øl
a2mme
Ow. (1) 01) & ((amCa) + Olaa mmm)),)))
01) 2 -((amC2) + 22(
Ows (t)
1
Therefore, by selecting 7, = O( Cp our selection of T"
Tl
am T ) = (
we can ensure that, condition holds for cmmCC2
Similarly, we can prove the case of v O2.
Lemma J.6. For any experi E Sy such that E {- 01, 002), and Vy E PP(01, 02). by selecting
Mr = 0 C and B = 5² (d we can ensure that after T = 0 (1 a)
1C² iterations,
Cl2Cz
(i) of (s.T") = 12 (mm2)) o (x,T") = O(mC2), 0q (o.T") =(mm2)
(ii) (xT") pe (s,0) and, plotT)) s p
Proof. The proof is similar to the proof of Lemma J.5
K PROOF OF THEOREM 4.4
Proof sketch. The results of Theorem 4.4 are provided by the post-training quantization analysis. Given the experts' activation bounds of the trained model, we estimate how much the activations produced by the quantized weights are allowed to deviate from their original values yet correctly classify the sequences. As the activations of the experts that learned more prevalent tokens are larger compared to the experts that learned less prevalent tokens, the former are allowed to deviate more than the latter. We use the maximum allowable deviations of expert activations for the two groups of experts (i.c., the experts that learned less prevalent tokens, and the experts that learned more prevalent tokens) to estimate corresponding quantization bin sizes via equation 2 Finally, we evaluate the sufficient bit-widths of the two groups of experts from their corresponding maximum allowable bin sizes.
Theorem K.1 (Full version of Theorem 4.4) Suppose the number of fine-tuning iterations satisfies = O quantization levels satisfy
P²2 log!
and max,e[m] Varle.)) O(1) for every expert s. If k 2 7 and the two
ame Cp
by > loogg(1 M(dy C, log(kmd3) log
(27)
and
b, > 1ogg(1
(lgg///..o. log log!)),
(28)
1 1-a
33

---

Published as a conference paper at ICLR 2026
then w.h.p. the quantized model has guaranteed generalization, i.e. (T) (x) > 0] =1
PVV(r, ~ : D
(29)
Proof. For any [m]] of [k), we denote the quantized representation of
Wr (x.T) = Wri wiz ... Wra (+TT) by, wr
(a.T)
(x.T)
(s.7:Q)
(s.T:Q) (s.T:Q)
(s.T:Q) (aT11)
wri
Wrz
y.,. Wra
(s,T) Wri
Awri
(a.T.Q)
wes
(eT))
Awiz
(aTTQ))
Wes
(e.)
+ Awrs
(+T:12) is the quantization-noise generated from the quantization of the Here, for any E [d], Awt, weight wt; (a.T)
Now, the randomness of the pre-trained model, for any E [m] of any [k], for any
(()
(
(x.T:Q) i E [d), Awe,
Unif
2 2 where (s) is the quantization bin size of the column
r E [m]] the expert s E
[k:] Here we assume that, for i i2 E [d] s.t. d # 12. Awk,
(a,TiQ) and
Awny (s.T:Q) are independent to each other. Similarly, we assume that for any r1, E [m]] s.t. ri + r2.
A Awrry Autz,
(a.T.Q) ( Awring and, Awra, (a.TiQ) are independent each other, We further (n,T:9)
assume that, for any 82,82 E [k] s.t. 81 # 82, Awny (ai.):Q) Awrai (12,11Q) Awny
Awra,
Au Aurr (:2:T.Q) (22.7:Q)
(Q))T1Q) independent to each other. (1Q) and, Awrzz
Awrng
Aura,
(s..T) Now, from statement (i) of Lemma L1 for any Son Poi
1 and Az 01 for some
log 1 C
j E [n]. Gf cla.T) Furthermore, from statement (iii) of Lemma I.1 dō:
(s1:T) 2mmll22/
Therefore, for any (x, y) DD such that Aj [n] with (1 o1,
log C₂
EmEs,, (x) M(
On the other hand, from statement (i) of Lemma
[1 for any 83 E S orr (sa.7) = 0.
Therefore, for any (x, y) ~ D such that 3j E [m] with 201) = 01.
log Cp
E.Es, fs (()(()) = kml
Again, from statement (iv) of Lemma I.1 for any q E Pifoi, 02). Vs E [k], [k), (s.T) Vs E (5.7 (s.T) Therefore, for any (x,y) y) ~ such that 3j [m] with wn1) = o1
OmmC2), and
O(mO2).
log
ELEs, 13) Exes_ 1Jt km
OkkmmC2), which implies for any
Cp
(x. v) D such that 3j E [n] with x(1) 01, of
(x) >
Therefore, to ensure that for any (a, v) ~ D such that 3j E [n] with xl0) or, vl6 (x)
logl for all E [m] of all sı E So, that satisfy Cp
need (w)....T) w.. (:1) 01) s O((C2)/
(w (=100) .0)) 2 0.
34

---

Published as a conference paper at ICLR 2026
Now, for an E [m]] of $1 Son.
log l Op
(- C2(12 log 1/)p)
P ( ( - (707) o1) 2 2
S eexp
dA (s1)²
logl
Therefore, for all r E [m]] of all 81 E So,, we need (()1) 1 =0(1c2/
to ensure
Cod
that, for all E [m]] of all $1 E Son we have,
log! l Cp
(w) (s1.2) w (.))) 01) S 501c2
1 d²
log
Similarly, for all E [m]] of all E So,, we need (02)
S O11C21
(adlog(7o,kmll to ensure
that, for all r E [m] of all 82 E Sog, we have
log Cp
(w) (w) ( wr (2))) 02) s 50
2 d²
(1
logd
for all r E [m]] of all 83 E S we need (0))
I 0
PCa
Codlog(7-o) to
ensure that, for all E [m] of all 53 E Soo we have
(w ( wo (1) -01) -0)) so( (1 a)
pCxy
log Cp
1
>
1
and
d²
a
log
for all r E [m]] of all 84 E Smags we need (,م00
<
0
PC2
Cpdlog(7-og kmd?y
to
ensure that, for all E [m] of all 84 E S Sagz we have
(w) (T wo (())) -02) -02) S 00
(1 a)
PC2
log
1
1
a
Cp
Now, for all r E [m] of all & E S 1 = max
(11) ( (ns)
we have
V(I, v) ~ D such that 3j E [m] with ,(بi) to,,
log C
P
ESES fQ. (x) (x) O OlVkmlC2
> 1 de
Here, 16 (x) is the quantized output for the expert 8.
Similarly, for all E [m] of all SI, A
max { A.w)). we have
v(z,y) ~ D such that 3j E [m] with x0)) t02
log 1
E.es, 1 (x)= O
Cp
d²
Therefore, for all $1 So, 52 E Soy.33 S
81 E S-oz, for all r E [m]] we need
A(n)) =0(102) Cydlog(% kmdd
logl
A A = 011C2)/ O
logl and (odlog((o,,kmd))
(+)) = 0 (1 a)
pCp C,dlog Yoo kmd?
log l
a
(14) 0
(1-a) PC2) (odllo(7-0
log/
a
35

---

Published as a conference paper at ICLR 2026
Now, for all [k], maxre(m) Var(w, (T (1))
On the other hand, for any [k] and
gla.T)²
any E [m], using the Von-Szokefalli-Nagy inequality, Var(w, (e 2 2d Therefore, for all s E [k), maX,e[m] 3!..2) @(vd).
Let us denote the bit-width of the expert 81 E , 82 E Soy 83 E S-o. and 84 E S_oz by Dei Dsg: Oag, and beg, respectively.
Therefore, need
be. log2 1+ max ((e1.I))
d
Cp Colog(To, kmmd))
loga
1 1+
minre(m) (01)
1C22
log
d
Cp log(Tooz kond))
Similarly, we need bos 2 log2
1+
IC2
log l
ad
Cp Colog(y-o kmd?)
bag
loga
1+0 (1 (1-a)PC2
logl
1+52
ad
Cplog(7-o kmd?) log
and bag 2 log2
(1-
Now, from statement (ii) of Lemma [.1 by selecting k 2 7, we can ensure that V81 E So and sss E Soz b bag b/.
As Torr Yozr 7-o1 7-02 are ((1), we need
bn 2 1o + O(dy/ Cp (og log ()), and
bi 2 10 1-a
2(dv Cplog(kmd))/PCC3 log ()), to ensure that,
(T) (x) F(V(,,1) D: Dyyle'
=1
36


Reviews:
[{"review_id": "3LbLdYgWeq", "reviewer": "Reviewer_8VPJ", "summary": "This paper proposes a novel expert-wise mixed-precision quantization strategy for Mixture-of-Experts (MoE) models, aiming to reduce their substantial memory footprint without significant performance degradation. The core contribution is a theoretically-grounded, two-stage heuristic for assigning bit-widths to experts. First, experts are ranked based on the change in their router's L2 norm during training $\\Delta^T= || w^T||_2 - || w^0||_2$, the paper's theory suggests that experts with a smaller norm change are more sensitive to quantization and thus require higher precision. Second, this ranking is adjusted by promoting experts with high maximum intra-neuron variance to higher ranks to mitigate quantization noise. The authors provide a theoretical analysis for a simplified two-layer MoE model to justify their primary metric. They validate their approach empirically on large-scale models, including Switch Transformer and Mixtral (8x7B and 8x22B), demonstrating superior performance compared to several baselines, including uniform quantization and prior expert-wise methods. A key advantage highlighted is the negligible computational overhead of this bit-assignment method compared to more expensive SOTA approaches like PMQ.", "questions": "Roughly same as weakness:\n\n- The use of the final router L2 norm as a surrogate for the change in norm is the most critical unsupported step in the paper. Can you provide any empirical evidence, for instance on a smaller model that can be fine-tuned, that these two metrics produce a similar expert ranking?\n- The proposed theory posits that experts with smaller router norm changes learn \"less frequent but critical features.\" Can the authors show what types of tokens or inputs are processed by the high-precision vs. low-precision experts in a real-world task, does this align with your \"critical but infrequent\" hypothesis?\n\n- Please provide an ablation study that evaluates the performance of your method under three conditions: (i) using only the router norm ordering, (ii) using only the MaxVar ordering, and (iii) the proposed combination. This will help understand where the gains are coming from.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method in this paper for bit-width assignment is computationally trivial, requiring a simple sort based on router norms. This stands in stark contrast to computationally expensive SOTA methods like PMQ, which require extensive calibration. This is a major practical advantage.\n\nThe method demonstrates strong performance on challenging benchmarks with large-scale Mixtral models, often outperforming existing expert-wise and non-expert-wise quantization methods.", "weaknesses": "1. For the key experiments on pre-trained models, the method abandons its primary metric (change in norm) for a surrogate (final norm). This switch is poorly justified and severs the link to the paper's own theoretical analysis. \n\n2. The `MaxVar` reordering step is not theoretically motivated and feels like an engineered solution to patch deficiencies in the primary metric. The lack of an ablation study makes it impossible to disentangle its effect, obscuring the true source of the performance gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2V7D7wtMJn", "reviewer": "Reviewer_LANt", "summary": "This paper presents an expert-wise mixed-precision quantization method for MoE models. It allocates higher bit-widths to experts that are more sensitive to quantization, identified through the router’s L2 norm change and neuron variance. The paper provides theoretical support showing that experts learning rare but important features require higher precision. Experiments show that the method achieves well performance than existing quantization methods with lower average bit-width and minimal overhead.", "questions": "see the questions", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper proposes a novel and efficient expert-wise mixed-precision quantization strategy based on router norm change.\n2.\tThe paper provides theoretical analysis explaining why experts learning rare features are more sensitive to quantization errors.\n3.\tAchieves well performance and inference efficiency under the same average bit budget.", "weaknesses": "1.\tWhen only pretrained router norms are used without fine-tuning, if the pretraining corpus and downstream data distributions differ, can the norm-based ranking still reliably identify “rare but important” experts?\n2.\tCan the claim that experts learning rare tokens exhibit weaker activations and smaller router norm changes be directly verified through visualization or statistical analysis on real LLM corpora?\n3.\tWould lightly retraining the low-bit experts further reduce the average bit-width or improve model robustness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7stLWKAiZj", "reviewer": "Reviewer_CgYX", "summary": "This paper proposes a novel quantization method for Mixture-of-Experts (MoE) models that dynamically selects the precision for each expert based on changes in the $l_2$ norm of its weights. Theoretical analysis is provided for a toy MoE architecture, and numerical experiments demonstrate that the proposed approach outperforms heuristic methods in low-bit quantization scenarios.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Theoretical results are provided, and the method design is well aligned with the theoretical justification.  \n2. Numerical results demonstrate that the proposed method achieves performance gains over baseline approaches, delivering comparable or even superior performance to heuristic methods and techniques that rely on calibration sets.", "weaknesses": "1. The theoretical analysis is based on a highly simplified MoE architecture focused on binary classification tasks. The gap between the theory and empirical results is not sufficiently addressed. In particular:  \n   (1) Can the insights from Theorem 4.4 be used to *adaptively* determine quantization precision—rather than relying on predefined bit levels?  \n   (2) Do activations in practical MoE models align with the conclusions of Theorem 4.3? If not, what are the key discrepancies, and how do they impact the method’s effectiveness?\n\n2. Several important experimental details are missing:  \n   (1) In Figure 2, the “activation weights” baseline—which specific method  does it correspond to?  \n   (2) How are the hyperparameters (e.g., $\\zeta$) selected? Is their choice task- or model-dependent, and how sensitive is the final performance to variations in $\\zeta$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)