MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Authors: Haojie Duanmu
Xiuhong Li
Zhihang Yuan
Size Zheng
Jiangfei Duan
Xingcheng Zhang
Dahua Lin
Venue: ICML 2025 poster (2025)
Keywords: Quantization
Mixture-of-Expert
LLMs
Efficiency
Primary area: deep_learning->large_language_models
TL;DR: We propose MxMoE, automatically discover mixed-precision quantization schemes and generates efficient kernels for parallel GPU execution, improving both quantization accuracy and computational efficiency.

Abstract: Mixture-of-Experts (MoE) models face deployment challenges due to their large parameter counts and computational demands. We explore quantization for MoE models and highlight two key insights: 1) linear blocks exhibit varying quantization sensitivity, and 2) divergent expert activation frequencies create heterogeneous computational characteristics. Based on these observations, we introduce MxMoE, a mixed-precision optimization framework for MoE models that considers both algorithmic and system perspectives. MxMoE navigates the design space defined by parameter sensitivity, expert activation dynamics, and hardware resources to derive efficient mixed-precision configurations. Additionally, MxMoE automatically generates optimized mixed-precision GroupGEMM kernels, enabling parallel execution of GEMMs with different precisions. Evaluations show that MxMoE outperforms existing methods, achieving 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit and delivering up to 3.4x speedup over full precision, as well as up to 29.4% speedup over uniform quantization at equivalent accuracy with 5-bit weight-activation quantization. Our code is available at https://github.com/cat538/MxMoE.

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Haojie Duanmu" 12 Xiuhong Li Zhihang Yuan Size Zheng* Jiangfei Duan
Xingcheng Zhang?
Dahua Lin 256
Abstract
Yang et al., 2024; Muennighoff et al., 2024; Liu et al., 2024b). These models replace dense MLP blocks in dense Large Language Models (LLMs) with specialized MoE com- ponents, each containing a routing mechanism and multiple expert networks. Through dynamic token- ceen-to-epper alloca- tion during inference, MoE architectures achieve superior model capacity while maintaining computational efficiency equivalent to their dense counterparts. However, these per- formance advantages introduce critical deployment chal- lenges that existing hardware systems struggle to address. First, the memory footprint of MoB model is usually several times that of Dense model. For instance, DeepSeek- V3's 67 IB parameters exceed the memory capacity of eight H100 GPUs in standard configurations (Liu et al., 2024b). Second, while the architecture activates only subset of experts per token during inference, scenarios such as prefill phase or large- serving can trigger widespread expert activa- tion, resulting in considerable computational overhead (Kim et al., 2022a).
Mixture-of ofxteres (MoE) models face deploy- ment challenges due to their large parameter counts and computational demands. We explore quantization for MoE models and highlight two key insights: 1) linear blocks exhibit varying quan- tization sensitivity, and 2) divergent expert acti- vation frequencies create heterogeneous compu- tational characteristics. Based on these observa- tions, we introduce MxMoE, mixed preccsson optimization framework for MoE models that considers both algorithmic and system perspec- tives. MxMoE navigates the design space de- fined by parameter sensitivity, expert activation dynamics, and hardware resources to derive effi- cient mixed-precision configurations. Addition- ally, MxMoE automatically generates optimized sexed-precision Group-GEMM kernels, enabling parallel execution of GEMMs with different pre- cisions. Evaluations show that MxMoB out- performs existing methods, achieving 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit and delivering up 3.4x speedup over full precision, as well up 29.4% speedup over uniform quan- tization at equivalent accuracy with 5-bit weight- activation quantization. Our code is available at https: //gp:/ggitttbb/ www/ca/caacoooooo cat 538/ M.
Mixed-precision quantization, which allocates different bit- widths to different parts of the model based certain cri- teria, has been shown to enhance MoE model performance (Huang et al., 2024; Tang et al., 2024). However, the added complexity of mixed seexedprreision schemes often leads to in- creased system overhead, making it more challenging to achieve tangible improvements in wall-clock time (Dettmers et al., 2022).
In this paper, focus achieving mixed-precision scheme that balances both model accuracy and hardware efficiency. Our goal is to enhance MoE model quantization while simultaneously achieving meaningful acceleration. To this end, we first conduct an experimental analysis of the MoE block, leading to two key insights: There is sig- nificant variation in quantization sensitivity across different linear blocks within an MoE block. Unlike previous work (Li et al., 2024; Huang et al., 2024), our findings suggest that allocating bitwidth at the linear block level, rather than the expert level, may yield better results. The activa- tion frequencies of different experts exhibit large variance within an MoE block. From a hardware perspective, this variance implies that different computational characteristics
1. Introduction
Mfxture-of-Fit -xpertss (MoH) architectures have established themselves cornerstone of modern large language mod- els, driving oftte--f-theaaae art performance across diverse AI tasks (Jiang et al., 2024; The Mosaic Research Team, 2024;
'Shanghai Jiao Tong University 'Shanghai AI Laboratory PPeking University 'ByteDance Seed 'The Chinese University of Hong Kong "CPII under InnoHK. Correspondence to: Haojie Duanmu <duanmuhaojie @ sjtu.edu.cn>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MxMoE; Mixed-precision Quantization for MoE with Accuracy and Performance Co- Co-Design
(e.g., memory-bound and compute-bound) coexist within an MoE. block, indicating potential benefits from applying diverse quantization schemes tailored to these distinct com- putational characteristics.
W down (a gate (X.)) & (X.))
(1)
where We gate up and W are linear transformations, represents the activation function and @ represents element- wise multiplication. The final output F is the weighted sum of the outputs from all experts:
W
Building on these insights, we propose MxMoE, a frame- work designed to derive an optimal mixed-precision quan- tization strategy through accuracy and performance co- design. MxMoE first navigates the interplay among pa- rameter sensitivity, expert activation patterns, and hardware resources, optimizing within this multidimensional design space with other constrains to identify the most effective mixed-pree pieee-preeision quantization schemes. Furthermore, Mx- MoE automatically generates a GPU kernel tailored to the identified strategies, efficiently orchestrating linear blocks with different precision in a parallel manner exploit hard- ware.
F
Σ
W down
(o Lage (X.)) W (X.)) O We
(2)
A straightforward method for executing such operation on hardware is sequential execution, where the summation in Eq. 2 is expanded, and each expert processed individually before aggregating the results. However, since the com- putation independence, this operation can be parallelized. Notably, the shape of the input X. and the corresponding weight matrix may vary across different experts. The ap- proach that concurrently processes multiple independent General Matrix Multiply (GEMM) with different shapes is referred to as Group-GEMM, Unlike Batched-GEMMM, where uub-rroblems have exactly the same shape, Group- GEMM deals with sub-problems that have varying dimen- sions, requiring a more careful design. CUTLASS (Thakkar et al., 2023) provides a high- high-performance implementation capable of efficiently handling the parallel execution of in- dependent GEMM operations.
2. Background
2.1. Quantization
Quantization aims to map a continuous value x to a discrete set of values. This can be represented as: = Q(x)), where 2(-) represents the quantization function, which can be either non-uniform or uniform. We focus on uniform quanti- zation where the set of discrete values is evenly spaced. The uniform min-max quantization operation can be expressed as:
round
Fmin
A Tmin
3. Motivation
3.1. Heterogeneous Quantization Sensitivity
where Tiin is the minimum value of the quantization range, A is the quantization step size, and the rounding function applies rounding to target domain, which is the source of the quantization error.
Recent studies have demonstrated that neural network com- ponents exhibit heterogeneous sensitivity to bitwidth, with quantization affecting different parameters to varying ex- tents (Wang et al., 2019; Dong et al., 2019). This heteroge- ncous sensitivity can be leveraged through mixed -pexcccireiiioon where different bitwidths are allocated to parameters based on their sensitivity. Such schemes typically outperform uniform-precision quantization in terms of accuracy.
LLM quantization mainly focuses on weight-only or weight- activation quantization. Weight-only methods (Frantar et al., 2022; Lin et al., 2024a; Kim et al., 2023) only involves quantization of model weight. Weight-activation meth- ods (Dettmers al., 2022: Xiao al., 2022) quantize both model weight and intermediate activation. In this work, we focus on MoE block quantization, covering both methods. Another lines of work, KV cache quantization (Liu et al., 2024c; Hooper et al., 2024; Yue et al., 2024; Duanmu et al., 2024) is orthogonal to our work.
In the context of MoE models, several works have inves- tigated the behavioral differences among experts. These studies show that, due to the influence of training dynamics, not all experts are equal. Some experts specialize in spe- cific tokens, contributing less to the overall generation (Liu et al., 2024b; Xue al., 2024). Building on this idea, we extend the concept of heterogeneity to the quantization of MoE models. Specifically, we systematically investigate quantization sensitivity across different architectural dimen- sions of MoE models by analyzing the sensitivity of various experts and their corresponding linear blocks.
2.2. MoE Mechanism and Group-GEMM
In MoE models, the routing mechanism dynamically selects a subset of E experts for each input token's 's hidden state, assigning corresponding weights {we)E_ For the e-th expert, the computation is given by:
As illustrated in Fig. 1a, our analysis reveals two key struc-
2

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co- Do-Design
w8a8. g-1_gy91 __ym
w8a16 9-g_g__syymmmmm
Quant Loss
[aa4_g-1_5ym
[aall_gg22__a00 _aym
150
w4a4_g128_ste
waal6_g1228_ps
FP16
100
600
WBA8
Activation Pyy
W4A16 Fri Frendly
50
W4A4
WSAB Friendly
450
W2A16
200
Gate_pro
TOPS
15.35
Quant Loss
Down_proj
150
300
100
150
50
20
40
60
150
40 Expert
Expart
(FLOPs/Byte)
(a)
(b)
Figure 1. (a) Quantization loss across experts DeepSeek 11th layer under various quantization schemes (top), and across linear components (Gate-proj/Doonnppo under the w424.g-1.sym configuration (bottom). The quantization notation wxay-gz.3 denotes x-bit weights, y-bit activations, group size z (-1 indicates per-channel/oken) with symmetric (sym) or asymmetric (asym) quantization. The quantization loss metric is formally defined in Section 1.2.1. (b) Roofline performance analysis for RTX 4090 GPU (left) and expert activation frequency distribution in DeepSeekV2-Litee s 11th layer (right).
tural patterns. First, experts exhibit divergent sensitivity profiles: for example, Expert 40 suffers significantly greater performance degradation under quantization compared to Expert 37. Second, sensitivity varies considerably across the components within single expert: the Down proj block in Expert 40 requires higher precision than the Gate_proj block within the same expert.
roofline analysis on the Nvidia RTX-4090 (Fig. 1b) identi- fies distinct performance regimes: for GEMM operations with shape [m, n, k] where n, > m, the arithmetic inten- sity simplifies to A m. For example, our analysis shows that W4A16 16 outperforms W8A8 when A < 83 and W2A16 outperformes W4A4 when A < 42.
In addition, we observe that MoE architectures exhibit significant computational heterogeneity. For instance, our evaluation of DeepSeek V2-Lite on the HumanEval-X dataset (Zheng al., 2023) reveals that expert activation frequencies within individual MoE blocks vary by over 10x (Fig. 1b). Considering W8A8 and W4A16, the computa- tional heterogeneity implies that within a single MoE block, operations that are suited for W8A8 and W4A16 coexist simultaneously, as predicted by the roofline model. This characteristic, distinet from dense LLMs, suggests that by strategically combining quantization schemes across experts, we can potentially achieve better performance than uniform precision quantization.
These observations motivate our linear-block granularity strategy: assigning different bitwidth to linear-blocks in MoE blocks to preserve model accuracy. Unlike recent studies that adopt expert-level mixed-precision schemes (Li et al., 2024; Huang et al., 2024), our approach focuses allocating bitwidths the linear block level. In Section 5.4, we demonstrate the superiority of this Lmnar-block--oee allocation strategy. Another line of recent research explores fine-grained mixed-precision approaches at the channel or element level (Kim al., 2023; Zhao et al., 2024). However, these approaches ineur significant computational overhead due irregular memory access patterns and the need for bitwidth lookup operations (Dettmers et al., 2022).
From a hardware-friende perspective, it is possible to select the most efficient quantization schemes based on compu- tational characteristics. For example, W2A16 generally outperforms W4A16, and W4A4 outperforms W8A8. How- ever, as discussed in Section 3.1, the allocation of bitwidth plays critical role in model accuracy. Simply optimiz- ing for performance may degrade model accuracy, while focusing exclusively on accuracy can result in suboptimal performance.
3.2. Hardware Friendly Quantization
The computational efficiency of different quantization schemes varies depending on the specific characteristics of the computation (Lin et al., 2024b). The effectiveness of these schemes is fundamentally determined by the arith- metic intensity (AI), defined as the ratio of FLOPs to mem- ory access in bytes (Williams et al., 2009). Weight-only quantization mitigates memory bandwidth limitations by reducing data transfer, whereas weight-activee activation quantiza- tion leverages low low-precision arithmetic units to accelerate compute-intensive operations (Frantar et al., 2024). Our
3

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
HQQ (W))()))))))
85.44
tural flexibility to handle precisioon-heterogeeeouuu expert computations efficiently.
80
Martin 74.90
MxMoE(W4)
70
MxNOE((VAAB)
We propose MxMoE to address above challenges. Mx- MoE tightly couple 1) hardware-aware bitwidth allocation scheme that respects parameter sensitivity and activation patterns with 2) a specialized computation engine that elim- inates kernel launch overhead and enables parallel mixed- precision expert execution.
ModioE(mix precision
3.08%
TFLOPS
50
2.70x
40
39.10
1.41x
30
FPIG(CUTLASS)
20
0.41x
0.11x
11.45
10
3.14
4. Method
0
4.1. Overview
Figure 2. Comparison of the computation throughput of low- precision MoE block. W4 denotes 4-bit per-channel symmetric weight-only quantization, while W8A8 refers to 8-bit per-channel symmetric weight-activation activation quantization. The problem consists of 60 experts, each with shape of [N, K] = [2816, 2048] (from M.E..,, with each token activating 4 experts. The total number of input tokens is set to 512.
Procision
Offilnne Statsticss
Otber Prices
Quantized MoB Block
Gruu GFMM
MaF Block
Menory
imput
CIAI CINA
CIA
Rulgri
Bange
Gate
down
Quani Loss
Ptotile Uata
ExportAct Puliem
Ex
gate
up
A Allocator
Sele-Shhhhlrr
Avalichie Stritettes
3.3. Aggorithm-System Co-Design and Challenges
wfol6 28 myym
F,
galle
down
wBad g-1 syt
The analysis presented raises a fundamental question: can we design a quantization scheme specifically tailored to MoE models that effectively balancing model accuracy and computational efficiency? Our findings suggest that het- erogeneous quantization sensitivity at the linear-block level within MoE models significantly affects accuracy, while hardware resources determine the maximum achievable computational efficiency. Moreover, the variation in expert activation frequencies introduces divergent computational demands, which is crucial for identifying the optimal quan- tization strategy. Therefore, an effective mrixed-precisio quantization scheme must take into account three key fac- tors: 1) parameter sensitivity, 2) expert activation frequen- cies, and 3) hardware characteristics.
0-1 syz w2al6 g) asgm
Hartware Information
Figure 3. Overview of MxMoE.
The workflow of MxMoE is illustrated in Fig. 3. First, MxMoE's allocator takes statistical data specific to the MoE model input, navigating the interplay between param- eter sensitivity, expert activation patters,, and hardware resources. It then optimizes within this multidimensional design space to identify the mixee-prrecision quantization scheme. Next, MxMoE generates a
Sroup-GEMiR kernel tailored to the identified scheme, ef- ficiently orchestrating linear blocks with varying precision. During runtime, the automatically generated tile scheduler maps mixed-precision computation tasks hardware in a manner, fully parallelizing the MoE block.
For given mixed-precision scheme, system-level support is necessary to translate theoretical performance improve- ments into actual wall-clock time reductions. While nu- merous works have optimized low-precision operators for dense L.MSs (Zhao et al., 2024; Lin et al., 2024b), these ap- proaches are often ill-suited for the MoE models. To demon- strate this, we leverage two widely used low-precision ker- nels: HQQ and VLLM-Marlin MoE to build a MoE blocks with 4-bit weight, as shown in Fig. 2. HQQ, which does not fuse dequantization, significantly underperforms the full- precision baseline. Marlin (Frantar al., 2024) is highly optimized W4A16 kernel achieves SOTA performance for W4A16 GEMM. The VLLM community (Kwon et al., 2023) adopts Marlin to build the VL.LM- Marlin kernel. It se- quentially invokes the Marlin kernel multiple times for each expert, which results in suboptimal GPU utilization. These shortcomings intensify when introducing mixed- precissoe configurations, as existing kernel designs lack the architec-
We begin by formalizing the impact of quantization and ex- pert activation frequency on model accuracy and execution performance, providing a comprchensive understanding of the design space for mixed- preccisinn scheme. Subsequently, we present our solution and discuss the system-level support required for mixed-precision MoE blocks.
4.2. Hardware-Aware Bitwidth Allocation
For a Given M-layer MoE model with parameter W and input X, the objective of bitwidth allocation in MxMoE is minimize both the perturbation introduced by quantization and the total execution time of all MoE blocks in the model:
M
min(L(W.X)) L(W X.))" C
T
(3)

---

MxMoE; Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
where is a hyper-parameter balancing the trade-off be- tween model accuracy loss and execution time. In this study, we adopt the setting presented in (Choukroun et al., 2019) which assumes a positive correlation between the change in the intermediate output of the quantized model and the final output. Therefore, minimizing the intermediate output loss leads to minimize the loss item in Eq.3. Furthermore, since the model is executed sequentially, minimizing the ex- ecution time of individual MoE. blocks contributes directly to reducing the overall execution time item in Eq.3. Thus, the objective simplifies to minimizing the output loss I. and execution time I of single mixed -peccd-preciiinn MoE block:
is decomposed into multiple sub-problems, known tiles, mapped to SMs for parallel execution. MxMoE generates candidate tile configurations for each quantization scheme and profiles their runtime costs e₁ ahead-ot -f- off--fff---mm time. For a lin- ear block with scheme s, tile decomposition is represented as (C), where n, denotes the tile count.
The computational task of a MoE block can be represented as list of such pairs, and our goal is estimate the ex- ecution time for this set of tasks mapped to the GPU. In our system design, all tiles are parallelized across the SMs, and the total execution time of the MoE block depends on the longest execution time across all SMs: T = max T; where P is the number of SMs. However, the use of the maximum operator complicates the optimization procedure. To address this, we approximate based on the observed fact: the number of tiles decomposed from a MoE block substan- tially exceeds the number of SMs. Therefore, the execution time of the entire MoE block can be approximated as the serial execution time of all tiles, divided by the number of SMs P:
min T
(4)
To further detail the formulation, we decompose the terms L and T' systematically.
4.2.1. .. QUANTIZATION LOSS FORMULATION
Let S denote the set of hardware-supported quantization schemes (c.g., W6A6 is still unsupported by most exist- ing hardware, while FP8 is supported on Nvidia RTX-4090 but not A A100). For an MoE block comprising E experts, each containing N linear blocks (typically = ern architectures, corresponding to gate-proj, up-proj, and down.proj). The composite loss aggregates individual quan- tization effects as:
E
N
S
IT
3 for mod-
T=
Craket Bidik,t Tiikk
where T represents candidate tile configurations, CLa.k.t denotes execution time of linear-block (i. under scheme k with tile configuration i, and Vr.j.K.t (0, 1} is indicating variable for selecting tile configurations. Our experiments show that this approximation is effective because the total number of tiles is typically much larger than the number of SMs.
N S A.j.k Fij,k
L
(5)
where Fij.k E {0, 1} denotes the binary selection variable for applying the k-th quantization scheme the j-th lin- ear block in expert i. The perturbation coefficient A.jj, quantifies the output distortion when using scheme k. com- puted via Euclidean distance between full-precision (O) and partially quantized (Ô) MoE block outputs:
The optimization under certain memory budget M formu- lated as an ILP problem:
MINIMIZE
L" -((((-))
N Σ
E
[S|
A Ô 0
(6)
WHERE
L
Aig.e Tig.k
2
18 1
E
N
IT
For practical estimation, we employ small calibration set (e.g., 128 samples from WikiText2) to compute A.j,k val- ues. Each linear block in expert is sequentially quantized with scheme E S. with the corresponding output pertur- bation statistically estimated across calibration samples.
WHERE
T
Cij.a W.J.A. Bij.
P
k=1t=l
[S]
SUBJECT TO Jii.k E 6,1,
Σ
Zij.k =1, Vi.j
k=
4.2.2. RUNTIME COST MODELING
T
SUBJECT TO Vij.k, E,,,,
Σ
Hiih = 1, Vi,j.k
To model the execution time T of mixed-precision MoE blocks, we first analyze individual linear blocks. Given input token distributions and expert activation frequencies, we derive GEMM shapes for each liner-block based on per-expert token allocations. On modern GPUs, GEMM
1=1
E N
SUBJECT TO
ΣΣΣ 1=
W.,.. Fig. S M
5

---

MxMoE; Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
We first gather offline statistics A and expert activation patterns. The execution time e for each linear block, un- der every quantization scheme and tile configuration, is then estimated based on pre-profiled single-tile runtime costs. This IIP is subsequently solved to determine de- sired sixxed-rrecssion schemes and tile configurations while ensure the quantized weight strictly adhere the memory bud- get. Following quantization scheme allocation, we apply randomized Hadamard transformations to model weights us- ing the incoherence processing used in QuaRot (Ashkboos et al., 2024), then perform GPTQ-based quantization (Fran- tar et al., 2022). Activations are dynamically quantized at runtime according to the corresponding allocated scheme.
quantization with bit manipulation techniques for optimized integer-to-float conversion (Kim et al., 2022b), while the W4A4-g variant employs multistage software pipelining that enforces strict adherence to 128 quantization group con- straints. Following bitwidth allocation, Mx.ooF generates a tile scheduler with a precision posing heterogeneous GEMM operations into unified kernel execution streams. We discuss in detail the advantages of micro kernel specilization over other possible approaches in App. A.2.
routing logic, com-
Resource Configuration. We next address the configu- ration of computational resources for horizontally fused mixed-precision Group-GEMM kernels. Building on the hardware-aware bitwidth allocation in Eq. 7, MxMoE de- rives tile configurations optimized for each quantization method under two critical constraints. First, warp count consistency is enforced across all micro-kernel tile con- figurations as shown in Fig. 4. Second, shared memory allocation follows the maximum requirement among fused operations. These two constraints ensures compliance with the CUDA programming model's requirement for uniform resources across CTAs (as shown in Fig. 3).
4.3.
GEMM Orchestration
Compatible Tile Configs: same warp count
wbitt:: abits: 2156e ggiza: -1,
sym: false:
BM 16, BM: 128, BK: 647)
8
WM::
NK: 23
wbits: 3, abits: 81 gaize: -1, sym: trues
BM: 128, BN:
128, BK: 647
8
WM: 2, WN:
WK:
13
X Incompatible Tile Configs: different warp count
wbite: 4,_ abate:10 1166 ggize: -1, sym: Louez
EM: 64, BM: 128, BK: 327
To mitigate shared memory waste from divergent tile sizes, we employ k-dimension tiling (slice-K). As illustrated in Fig. 4, the tile size for W4A16 is substantially smaller than that of W8A8. This disparity results in shared memory under-utilization for W4A16 micro-kemell Our solution introduces additional parallelism along the k-dimension for W4A16 configurations through strategic tile partition- ing. This dual- dual-purpose optimization simultaneously reduces warp under-utilization while increasing shared memory uti- lization.
WM: 2,
NK:
wbite: 9, abits:l6, geize: -1,
sym: trues
BM: 125, BM: 22.28, BK:
WM: WN: 4, MK: 8
Figure 4. MxMoE ensures that tile configurations for different pre- cisions have the same number of warps.
As discussed in Section 3.3, sequentially processing com- putations for each expert is inefficient. CUTLASS does not support heterogencous precisions. Fusing heterogeneous- precision GEMMs introduces two fundamental challenges: 1) Optimal tile sizes and warp layouts vary across precisions, making a unified kernel for all possible precisions inherently suboptimal, and 2) The large number of possible precision combinations makes developing a custom kernel for each combination prohibitively costly. MxMoR addresses these challenges through automated kernel generation frame- work that consists of three key components: micro-kemed specialization, resource configuration, and tile scheduling.
Tile Schedule. Finally, MxMoE optimizes the scheduling of tiles with heterogeneous precision requirements. The execution time of tiles varies significantly across different precision and tile shape configuration, making the schedul- ing order critical determinant of overall completion time. This is classic makespan minimization problem. While dy- namic programming can achieve optimal solution, MxMoE implements an efficient greedy heuristic that prioritizes com- putationally intensive tiles. Given that the number of tiles in MoF blocks typically exceeds the available SM count by a substantial margin, this approach achieves near-optimal per- formance (Graham, 1966) while significantly reducing the scheduling overhead compared to dynamic programming solutions.
Micro-Kernel Specialization. We introduce configurable CTA-level micro-kernels implemented as CUDA device (CTA)
functions, designed with index independence to enable subsequent horizontal fusion. Each micro-kernel's 's resources are specified via C++ tem- plate parameters, while memory access patterns are opti- mized for specific quantization schemes through meticulous hand-tuning of compute-to to memory access pipelines.
5. Experiments
5.1. Experimental Setup
Model Configurations. We MxMoE on three open- source MoE architectures: DeepSeek (Liu al., 2024a),
For instance, the W2A16 16 micro-kernel integrates fused de-
6

---

MxMoE: Mixed-precisiee Quantization for MoE with Accuracy and Performance Co-Design
Table We evaluate the following datasets: Are-Challenge (AC), Arc-Easy (AF), HellaSwag (HS), L.MBAAAA--penal (I.O), LAMBADA-standar (LS), PIQA (PQ), and WinoGrande (WG). GPTQ" denotes GPTQ with random Hadamard transformation
preprocessing. # Bits indicates the quantization bitwidth for GPTQ and Quarot (uniform bitwidth), and average bitwidth for MxMoE.
Model
Method
#Bits (W-ACT)
AC1
AET
HSt
LO
LSt
PQT
WG1
Avg.t
PPL.
Baseline
16-16
48.98
76.22
77.91
72.33
67.90
80.20
71.19
70.68
5.92
GPTQ
3.25-16
47.35
75,04
76,44
70.41
65.65
79.05
71,27
69.32
6.18
GPTQ
2.25-16
37.63
63.47
65,45
52.53
48.55
74.59
64.09
58.04
8.49
DeepSeckV2-4 VitLite
QuaRot
4-4
41.81
67.51
74.12
50.01
45.86
75.52
63.38
59.74
8.44
MxMoE
3.25-16
47.87
74.58
76.85
71.10
65.85
79.27
70.09
69.37
6.08
MxMoE
2,25-16
40.36
68.86
68.63
59.56
54.01
75.08
67.80
62.04
7.01
MxMoB
5-5
46.76
74.37
77.38
68,41
64.99
79.38
69.22
68.64
6.16
Bascline
16-16
44.03
69.53
77.26
71.28
64.62
80.47
69.30
68.07
6.79
GPTQ
3.25-16
43.34
68.60
75.35
68.68
62.80
79.22
66.54
66.36
7.15
GPTQ
2.25-16
30.89
47.14
60.77
43.72
34.81
69.97
56.20
49.07
11.19
Qwenl.5-Mol
QuaRot
4-4
27.13
40.74
57.10
35.61
25.33
66.43
51.93
43.47
18.44
MxMoF
3.25-16
43.77
66.04
75.92
69.71
62.82
79.11
68.03
66.49
7.02
MxMoB
2.25-16
31.66
53.28
62.80
56.43
51.00
71.33
61.25
55,39
8.79
MxMoE
5-5
42.92
66.04
76.27
70.06
63.40
80.58
67.80
66.72
7.01
Baseline
16-16
55.20
77.19
84.09
74.35
62.62
82.32
72.14
72.56
5.84
GPTQ
3,25-16
53.67
75,88
82.90
73.36
63.24
81.01
70.96
71.57
6.11
GPTQ
2.25-16
38.82
57.66
71.27
58.99
49.72
73.29
60.30
58.58
7.98
Qwen2-MoE
QuaRot
4.4
33.19
42.72
54.34
23.02
9.53
63.87
50.12
39.54
110.66
MxMoE
3.25-16
53.84
76.30
82.81
72.39
60.95
81.34
69.69
71.05
6.18
MxMoE
2.25-16
45,05
68.86
77.13
66.00
56.61
75,41
62.90
64.57
7.57
MxMoB
5-5
54.86
75.55
82.69
72.87
62.68
79.49
70.96
71,30
6.25
Bascline
16-16
66.38
85.39
85.95
77.28
73.06
85.20
76.72
78.57
3.88
GPTQ
3,25-16
64.42
84.01
85.12
76.77
71.76
83.79
76.16
77.43
4.17
GPTQ
2.25-16
48.89
72.35
76.95
68.39
61.44
77.15
67.72
67.56
5.69
Mixtral-8x79
QuaRot
4-4
50.60
68.69
75.65
40.95
38.83
76.88
61.01
58.94
9.06
MxMoE
3.25-16
64.25
84.22
85.04
76.98
71.86
84.17
75.93
77.49
4.15
MxMoE 2.25-16
48.98
72.77
77.44
68.68
62.18
76.28
68.90
67.89
5.63
MxMoB
5-5
64.08
83.71
85.10
76.21
71.78
83.79
73.80
76.92
4.20
Mixtral (Jiang et al., 2024), and Qwen (Yang et al., 2024) as detailed in Table 2. DeepSeek-V2-1 V2-Lite employs hy- brid architecture: the first layer employs dense MLP instead of MoE blocks in other layers which are quantized with GPTQ 4-bit per-channel asymmetric quantization in our experiments. We focus the quantization of MoE blocks, retaining full precision for the attention modules. Experi- ments conducted on Nvidia RTX-4090
tion accuracy, we apply a random hadamard transformation. We disabled online rotations because they failed on some models like DeepSeck-V2 V2-Lite due to the shape constrain. For weight quantization, MxMoE employs GPTQ, using the same calibration with that of bitwidth allocation.
5.2. Accuracy Results
For weight-only quantization we test 3-bit and 2-bit quanti- zation, comparing with GPTQ, configured with group size 128, asymmetric min-mat quantization, where the scale and zero-point are stored in 16-bit format, resulting in an aver- age bitwidth of 3.25 and 2.25, respectively. To ensure fair- ness, we apply the same random Hadamard transformation for GPTQ. MxMoE use r = as extremely low-bitwidth implies wecource-constrainne environment, where model ac- curacy more important. The results in Tab. 1 show that GPTQ suffers significant performance degradation 2.25- bit, while MxMoE consistently outperforms GPTQ across all models. At 3.25-brl bit, MxMoE outperforms GPTQ in most models except Qwen2-MoF. This exception may stem from inaccuracies in the sensitivity statistics, amplified by inter-layer dependencies (Yue et al., 2024). Using cross- layer loss as sensitivity metric instead layer loss in Eq.
Table Architectural Specifications of Evaluated MoE Models
Model Variant
Params (GB)
Experts
TopK
Mixtral-8 x 8x7B--Isttuet-oO1.
Instruct-v0.1
92.9
Qwen1.5-MoE
26.7
60+4
106.9
DeepSeck-V2 V2-Lite
29.3
6442
Calibration. MxMoE requires offline calibration to deter- mine the sensitivity of tinear-block and expert activation frequencies for bitwidth allocation. For all experiments, we use 128 sequences, each of length 4096, drawn from the Wikitext2 training set (Merity et al., 2016). This calibra- tion process typically takes from several minutes to a few hours depending on the model size. To enhance quantiza-

---

MxMoE: sixed-precision Quantization for MoE with Accuracy and Performance Co- So-Design
FP16
W4A16
W8A8
W444
MyMMoE(44.25AL) Qwen2-MoE
MxMoE(WSAS)
DeepSeek-V2-Lite
Qwen1.5-MoE
Mixtral 8x7B
100
1000
125
5.35
4.00
100
1000
TFLOPS
75
55.00
75
3.55
50
5.88
-0
25
25
-3.00
0-
592
0-
3.00
Takens: 512
Tokens: 512
#Tokens: 512
Tokenss: 512
600
8.50
-3..50 800
100.00 8.00
5.00
0.00
8.00
TFLOPS
100
400
15.09
600
600
7.00
12.99
400
400
6.00
200
200
200
5.00
-7.50
Tolenss: 8192
Takenss: 8192
Tokens:
#Tokens: 8192
Figure 5. Computational throughput of MoE blocks across models and precision settings. W4A16 denotes 4-bit weight-only per-channel asymmetric quantization; W8A8 and W4A4 denote 8/4-bit weight-activation per-channel symmetric quantization. Number followed MxMoE represents the average bitwidth for weight and activation. The x symbols indicate corresponding perplexity values on WikiTex12.
may mitigate this issue, and leave this to future work. Improvement of MxMoE on Mixtral is relative margin. This due to Mixtral's fewer experts (Tab. 2), resulting a more limited mixed- precdssre design space.
Table 3. Comparison of different allocation granularities. Test with 5-bits weight- activatiit quantization. Evaluation metrics are the same described in settings.
PPL
Avg-Acct
Model
For weight-activation quantization, we compare MxMoE with Quarot (Ashkboos al., 2024). MxMoF use 0.75. We focus 4-bit 8-bit is almost lossless. Quarot experiences significant accuracy drop 4-bit across all models, while MxMoE achieves substantial improvements with just additional bit (i.e. average 5-bit). These gains are attributed the higher sensitivity certain activations bitwidth, as shown prior studies (Dettmers al., 2022; Sun et al., 2024). By allocating higher bitwidth for the sensitive activation, MxMoE delivers notable performance improvements. The sources of accuracy gains under this configuration are rigorously dissected in App. A.I.
Linear
Expert
Linear
Expert
DeepSeek- V2-Lite
6.11
6.32
69.01
67.88
Qwenl 1.5MoE
6.95
6.98
67.35
67.11
For the memory-bound scenario (512 tokens), W8A8 con- sistently underperforms both W4A16 and W4.25A15.5, as the limited tokens per expert make MoE block computations memory- bound. MxMoE (W4.25A15.5) not only achieves better accuracy than W4A16 but also delivers up to 25% higher throughput on Qwen 1.5-MoE. This improvement stems from our hardware-aware bitwidth allocation strategy, which assigns lower-precision activations frequently acti- vated experts that form compute-bound operations (as dis- cussed in Section 3.2). In the compute-boune scenario (8192 tokens), we compare against W4A4 and W8A8. While W4A4 offers significant speedup at the cost of substan- tially increased perplexity, and W8A8 maintains accuracy with minimal acceleration, MxMoE (W5A5) achieves up 29 4444% performance improvement over W8A8 while main- taining comparable perplexity to full-precision models.
5.3. Performance Analysis
Due to the lack of established low-precision Group-GEMM baselines, we evaluate both precision kernels generated by MxMoF. MxMoE use r 0.75 for all the mixed-precision scheme. 16-bit Group- GEMM kernel from CUTLASS. we assess the computa- tional throughput of MoE blocks across different models, only expert computation is counted as other operations gating, topk, sort etc.) is negligible. We randomly sam- ple sequences from WikiText-2 with lengths of 512 and 8192 tokens, respectively representing memory-bound and compute-bout bound workloads. As shown in Fig. 5, mixed- precision scheme achieve substantial performance improve- ments: 1.6 2.7x throughput increase for memory-bound workloads and 3 3.44 for compute-bound workloads com- pared to full full-preicision.
and mixed-
5.4. Ablation Studies
Effect of bitwidth allocation granularity. MxMoE employ linear-block level allocation instead of expert-level alloca- tion in previous studies. We also perform bitwidth allocation at expert level as shown table 3. The results demonstrate that finear-block allocation consistently outperforms expert- level allocation.
8

---

MxMoE: Mixed- seeed-precision Quantization for MoE with Accuracy and Performance Co- Ro-Design
8.5
References
650
68
8.0
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B.,
625
66
Avg-Acc.
Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in ro- tated llms. arXiv preprint ar 24xv:2204.004566, 2024.
TOPS
TOPS
7.5
600
PPL
64
Avg-Acc.
7.0
575
Choukroun, Y., Kravchik, E.,, Yang. F., and Kisilev, P. Low- bit quantization of neural networks for efficient inference. In 2019 IEEE/CVF International Conference on Com- puter Vision Workshop (ICCVW), pp. 3009-3018. IEEE, 2019.
6.5
162
550
6.0
0.0
0.25
0.5
0.75
1.0
Payper-Parametr-
Figure 6. Impact of the hyperparameter the trade-off between model accuracy and performance. Model: DeepSeck-V22 V2-Lite.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318-3033 2022.
Dong, 7.,, Yao, Z.,, Gholami, A., Mahoney, M. W., and Keutzer, K. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 293-302, 2019.
Impact of the hyperparameter. MxMoE introduces the hyperparameter to balance efficiency and accuracy. We employ 0.75 in all experiments except extremely low- bitwidth weight-only quantization in Tab. 1, where r = 1. Intuitively, = prioritizes maximizing accuracy, while r = focuses solely on efficiency. Now we quantitatively investigate the impact of the tradeoff parameter. As shown in Fig. 6, performance improves as decreases, at the cost of reduced accuracy. Notably, when optimizing for both objectives, such as at r 0.75, we observe significant per- formance gains with minimal accuracy drop. This highlights the effectiveness of awwa bitwidth allocation.
Duanmu, H.,, Yuan, Z., Li, X., Duan, J., Zhang, X., and Lin, D. Skvq: Sliding- wwnndww key and value cache quantization for large language models. arXiv preprint D.Xiv:2405006219 2024.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for genera- tive pre-trained transformers. ArXiv, abs/2210.11323,, 2022. URL https: //aps:///ppp semanticschol/ cacholar. org/ CCrpporrriiiiiaaa :D:323370000
6. Conclusion
We propose MxMoE, an accuracy--erffommacc co-design framework for MoE mixed-precisioe quantization. MxMoE allocates bitwidth through joint optimization of computa- tional efficiency and model accuracy and generates opti- mized mixed-precision Group-GEMM kernels, achieving significant acceleration while preserving model accuracy.
Frantar, E., Castro, R. L., Chen, J., Hoefler, T. and Alis- tarh, D. Marlin: Mixed-precisioe auto-regressive paral- lel inference on large language models. arXiv preprint arXiv:2408.1177 2024.
Graham, Bounds on multiprocessing timing anomalies. Bell System Tech. J., 45:1563-1581, 1966.
Acknowledgements
Hooper, C., Kim, S., Mohammadzadeh, H., Mahoney, M. W., Shao, Y. S., Keutzer, K., and Gholami, A. Kvquant: Towards 10 million context length Ilm in- ference with ky cache quantization. arXiv preprint 2024.
The authors would like to thank the diligent anonymous re- viewers for their constructive feedback. Project supported by Shanghai Municipal Science and Technology Major Project. This study was supported in part by the InnoHK initiative of the Innovation and Technology Commission of the Hong Kong Special Administrative Region Government.
Huang, W., Liao, Y., Liu, J., He, R., Tan, H., Zhang, S., Li, H., Liu, S., and Qi, X. Mc-moe: Mixture compressor for mixture-of-expprtt llms gains more. arXiv preprint arXiv:4410.06270 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must specifically highlighted here.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., Casas, D. d. I., Hanna, E. B., Bressand, F., et al. Mixtral of experts. arXiv preprint arXiv:2401.0400 2024.

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.0766 2023.
N., al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.02060, 2024.
Sun, M., Chen, X., Kolter, J. Z., and Liu, Z. Massive activations in large language models. arXiv preprint arXiv:2402. 17762, 2024.
Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can't tt run: Bringing large scale moe mod- els into cloud scale production. ArXiv, abs/2211.10017, 2022a. URL https: http:pp/ppp..smmattttisssseeeees www.nticschol..........o org/Co Corpus ID: 253708032.
Tang, P., Liu, J., Hou, X., Pu, Y., Wang, J., Heng, P.A.,, Li, C., and Guo, M. Hobbit: A mixed precision expert offloading system for fast moe inference. arXiv preprint arXiv:4411.0133, 2024.
Kim, Y. J., Henry, R., Fahim, R., and Awadalla, H. H. Who says elephants can't run: Bringing large scale moe models into cloud scale production. arXiv preprint arXiv:2211. 2022b.
Thakkar, V., Ramani, P., Cecka, C., Shivam, A., Lu, H., Yan, E., Kosaian, J., Hoemmen, M., Wu, H., Kerr, A., Nicely, M., Merrill, D., Blasig, D., Qiao, F., Majcher, P., Springer, P., Hohnerbach, M., Wang, J., and Gupta, M. CUTLASS, January 2023. URL https://github.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H.. Gonzalez, J. E.,, Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
com Ctt.//VIIIA//AAA cut.Isss....ss...
The Mosaic Research Team.
Introducing DBRX:
A New State-of-the-Art Open LLM,, 2024. URL. https: hteps::/www.dart wwtp::/wwwabbiicss.ooo cotpttarrriicc..cooooooooo////// Larrocuccigg-dbbxeee---ttttteeee
Li, P., Jin, X., Cheng, Y., and Chen, T. Examining post- training quantization for mixture-of-experts: A bench- mark. arXiv preprint arXiv:2406. 08155, 2024.
Wang, K., Liu, Z., Lin, Y., Lin, J., and Han, S. Haq: Hardware-aware automated quantization with mixed pre- cision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Lin, J., Tang, J., Tang, H., Yang. S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-awan weight quantization for on-device 1lm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Williams, S., Waterman, A., and Patterson, D. Roofline: an insightful visual performance model for multicore architectures. Communications of the ACM, 52(4):65-76. 2009.
Lin, Y., Tang, H., Yang, S., Zhang, Z.,, Xiao, G., Gan, C.,, and Han, S. Qserve: W4a8kv4 quantization and sys- tem co-design for efficient Ilm serving, arXiv preprint 24Xi:24005.0432. 2024b.
Xiao, G., Lin, J., Seznec, M., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quanti- zation for large language models. ArXiv, abs/2211.104333
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024a.
2022. URL https: www.nticschollat org/ CorpusID:2537095
Xue, F., Zheng, Z., Fu, Y., Ni, J., Zheng, Z., Zhou, W., and You, Openmoe: An early effort on open mixture-of-experts language models. arXiv preprint arXiv:2402.01739, 2024.
Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek- v3 technical report. arXiv preprint arXiv22412.19437, 2024b.
Yang, A., Yang. B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. technical report. arXiv preprint arXi::4412.15115. 2024.
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z.,, Braverman, V., Chen, B., and Hu, X. Kivi: A tuning-free free asym- metric 2bit quantization for kv cache. arXiv preprint arXiv:2402.02250. 2024c.
Yue, Y., Yuan, Z., Duanmu, H., Zhou, S., Wu, J., and Nie, Wkvquant: Quantizing weight and key/value cache for large language models gains more. arXiv preprint 2.ti:2242..2000 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Zhao, Y., Lin, C.-Y., Zhu, K., Ye, Z., Chen, L.,, Zheng, S., Ceze, L., Krishnamurthy. A., Chen, T., and Kasikci, B. Atom: Low-bit quantization for efficient and accurate Imm
Muennighoff, N., Soldaini, L., Groeneveld, D., Lo, K., Mor- rison, J., Min, S., Shi, W., Walsh, P., Tafjord, O., Lambert,
10

---

MxMoE: Mixed-prees precision Quantization for MoE with Accuracy and Performance Co- Co-Design
serving. Proceedings of Machine Learning and Systems, 6:196-200 209, 2024.
Zheng, Q.. Xia, X., Zou, X., Dong, Y., Wang, S., Xue, Y., Wang, Z.,, Shen, L.., Wang. A., Li, Y., Su, T. Yang, Z., and Tang, Codegeex: A pre-trained model for code gener- ation with multilingual benchmarking humaneval-x. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 5673-5684, 2023.

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co- Co-Design
A. Appendix
A.1. Case Study: Detailed Analysis MxMoE WSAS Scheme
We conduct in-depth analysis of MxMoE's W5A5 mech- anism and its accuracy advantages. As shown in Tab. 4, reducing activation bitwidth from 5 to 4 bits causes sig- nificant model quality degradation, demonstrating critical quantization sensitivity around 4-bit activation precision. This can be attribute to the massive outlier observed in the input activation of Dowj_proj (Sun et al., 2024), where heavy-tailed activation distributions require higher preci- sion preservation. MxMoE dynamically identifies these quantization senissivve components (whose bitwidth re- duction causes substantial model quality degradation) and allocates elevated bitwidth accordingly. The heteroge- neous bitwidth allocation strategy for Qwen 1.5.Mo.EEE... is visualized in Tab. 7.
Table 4. WikiTex12 perplexity under different weight-activation bitwidth (RTNNtokcm/channel quantization). Column: activation bitwidth, row: weight bitwidth.
#Bits
7
8
4
68079.039
41.433
11.298
9,406
8.068
5
12305.585
38.707
9.715
8.169
7.335
6
14251.822
26.297
9,216
8.196
7.204
7
18151.474
34.775
9.747
8.182
7.325
8
19091.917
38.990
9.525
8.260
7.278
To further validate
benefits, we com- pare against QuaRot (Ashkboos et al., 2024) in Tab.
Table 5, WikiText2 perplexity of Qwen .5 5-MoE under different quan- tization bitwidth settings. Both MxMoE and QuaRot employ RTN weight quantization.
While uniform bitwidth scaling shows similar perplex- ity improvement trends in QuaRot, practical deployment remains constrained by the capacity of model hardware, on which 5-bits operation is not supported. In contrast, MxMoE achieves better accuracy while maintaining hard- ware compatibility through fexxed-rrecssion allocation that leverages existing low-bitwidth arithmetic units.
Setting
w4a4
w5a5
w6a6
w7a7
w8a8
QuaRot (Uni)
36.385
7.998
6.990
6.852
6.814
MxMoE (Mix)
7.160
A.2. The Necessity of Automated Gernel-Generation
MxMoE mitigates the combinatorial explosion of mixed mixee-precision configurations by automating kernel generation 4.3. To illustrate the effectiveness of this approach, we compare our strategy with two alternative solutions:
Developing a universal kernel to handle all precision combinations: This approach would compromise kernel per- formance. We provide a breakdown demonstrating the limitations of this method relative to micro-kernel specialization in MxMoE. Specifically, the kernel for W4A4-per-chhanel could theoretically share the same software pipeline with W4A4-group128, but enforcing universality significantly degrades performance. shown in Tab. 6, we test different kernels under the shape (8192, 8192, 8192], and the specialized kernel always outperform unified kernels. The reason is that unifying the two pipelines requires introducing runtime condition checks, which hinder loop unrolling in the MAC-loop. Moreover, to support group-size= sizu-ssize=128,, the per-channel kernel's tile-size selection is constrained, making configurations such as tile_k = 256 infeasible.
Table Performance comparison of different W4A4 quantization kernels on GPU TOPS
Kernel Type
W4A4 per-channel TOPS W4A4 group128 TOPS
W4A4-per-channed (Specialized)
1070.5303
N/A 667.3349
W4A4 groupl28 (Specialized)
N/A
Unified Kernel
929.1997
412.0268
Developing separate kernels for each configuration: While handerafted kernels could match performance, they require substantial engineering effort. If a given hardware platform supports five quantization candidates (e.g., w2a6, w4a16, w8a8, w4a4, w4a4 with group-size 128), implementing individual kernels for all possible configurations would require 5! = 120 kernels. In contrast, our micro-kernel specialization approach requires implementing only configurable micro- kernels, which are automatically combined by the kernel generator to form optimized fused operators.
12

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co-Design
Table 7: W5AS mixed-precision scheme allocated by MxMoE. Qwenl.5-MoE, layer 5.
Gate
Up
Down
Expert
w-act
gsize
a gsizz
w-act
gsize
gsize
w-act
gsizz
gsize
4-4
128
128
4-4
128
128
4-4
128
128
1
4-4
128
128
4-4
128
128
8-8
-1
-1
4-4
128
128
4.4
128
128
8-8
-1
4-4
128
128
4-4
128
128
8-8
4
4-4
-1
-1
4-4
-1
4-4
128
128
5
4-4
128
128
4-4
128
128
4-4
128
128
6
4-4
128
128
4-4
128
128
8-8
-1
-1
7
4-4
-1
-1
4-4
-1
-1
4-4
128
128
8
4-4
128
128
4-4
128
128
8-8
-1
4-4
128
128
4.4
128
128
8-8
10
4-4
128
128
4-4
128
128
8-8
11
4-4
128
128
4-4
128
128
8-8
12
4-4
128
128
4-4
128
128
8-8
13
4-4
128
128
4-4
128
128
4-4
128
128
14
4-4
-1
-1
4-4
-1
-1
4-4
128
128
15
4-4
128
128
4-4
128
128
8-8
16
4-4
128
128
4-4
128
128
8-8
17
4-4
128
128
4-4
128
128
8-8
18
4-4
128
128
4-4
128
128
8-8
19
4-4
128
128
4-4
128
128
8-8
20
4-4
128
128
4.4
128
128
4-4
128
128
21
4-4
128
128
4-4
128
128
8-8
-1
-1
22
8-8
-1
-1
8-8
-1
-1
8-8
-1
-1
23
4-4
128
128
4-4
128
128
4-4
128
128
24
4-4
128
128
4.4
128
128
8-8
-1
25
4-4
128
128
4-4
128
128
4-4
128
26
4-4
128
128
4-4
128
128
8-8
-1
27
4-4
128
128
4.4
128
128
4-4
128
128
28
4-4
128
128
4-4
128
128
8-8
-1
29
4-4
128
128
4-4
128
128
4-4
128
128
30
4-4
128
128
4-4
128
128
4-4
128
128
31
4-4
128
128
4.4
128
128
8-8
-1
32
4-4
128
128
4-4
128
128
4-4
128
128
33
4-4
128
128
4-4
128
128
4-4
128
128
34
4-4
128
128
4-4
128
128
8-8
35
4-4
128
128
4-4
128
128
8-8
36
4-4
128
128
4-4
128
128
8-8
37
4-4
128
128
4-4
128
128
8-8
38
4-4
128
128
4-4
128
128
8-8
39
4-4
128
128
4-4
128
128
4-4
128
128
40
4-4
128
128
4-4
128
128
4-4
128
128
41
4-4
128
128
4-4
128
128
8-8
42
4-4
128
128
4-4
128
128
8-8
-1
-1
43
4-4
128
128
4-4
128
128
8-8
-1
-1
44
4-4
-1
-1
4-4
-1
-1
4-4
128
128
45
4-4
128
128
4-4
128
128
8-8
-1
46
4-4
128
128
4-4
128
128
8-8
-1
47
4-4
128
128
4-4
128
128
4-4
128
128
48
4-4
128
128
4-4
128
128
8-8
-1
13

---

MxMoE: Mixed-precision Quantization for MoE with Accuracy and Performance Co- Dosiggg
Table 7: MxMoE W5A5 scheme (continued)
Gate
Up
Down
Expert
w-act
gsize
a gsize
w-act
gsize
gsizze
w-act
g-gizz
gsize
49
4-4
128
128
4-4
128
128
8-8
-1
-1
50
4-4
128
128
4-4
128
128
4-4
128
128
4-4
128
128
4.4
128
128
4-4
128
128
52
4-4
128
128
4-4
128
128
8-8
-1
53
4-4
128
128
4-4
128
128
4-4
128
128
54
4-4
-1
4-4
-1
4-4
128
128
55
4-4
-1
-1
4-4
-1
4-4
128
128
56
4-4
-1
-1
4-4
-1
4-4
128
128
57
4-4
128
128
4-4
128
128
4-4
128
128
58
4-4
-1
4.4
-1
4-4
128
128
59
4-4
128
128
4-4
128
128
8-8
-1
60
4-4
4-4
-1
8-8
14


Reviews:
[{"review_id": "vmMvwkSQDg", "reviewer": "Reviewer_ELGx", "summary": "The paper introduces MxMoE, a mixed-precision quantization framework tailored for Mixture-of-Experts (MoE) models, aiming to address deployment challenges posed by their large memory footprint and computational demands. Key insights include:\n1. Heterogeneous quantization sensitivity: Linear blocks within MoE experts exhibit varying sensitivity to bitwidth reduction.\n2. Divergent expert activation frequencies: Computational characteristics (e.g., memory-bound vs. compute-bound operations) differ across experts.\nMxMoE optimizes bitwidth allocation at the linear-block granularity (rather than expert-level) and generates specialized GPU kernels for parallel execution of mixed-precision Group-GEMM operations. It formulates the problem as an Integer Linear Programming (ILP) task, balancing quantization-induced accuracy loss and runtime efficiency. Evaluations on models like DeepSeek-V2-Lite and Mixtral-8×7B show MxMoE achieves 2.4× lower perplexity than GPTQ at 2.25-bit and 3.4× speedup over full-precision execution, outperforming uniform quantization baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s constructive feedback. Below, we address the two key concerns raised:\n\n1. Exploration of Trade-off Parameter $r$ for Different Models and Use Cases\n\n    Our ablation study (Figure 6) explores the impact of the hyper-parameter $r$ on model accuracy and hardware efficiency. Our analysis demonstrates that increasing $r$ prioritizes accuracy (e.g., lower perplexity) at the expense of efficiency (e.g., lower throughput).\n\n    **MxMoE can automatically adjust $r$ according to use case**: there are typically some constraints such as memory budget, precision budget, and latency budget in model quantization. MxMoE satisfy these constraints by automatically adjusting $r$. Additionally, users may manually tune $r$ to balance hardware efficiency and model accuracy, selecting a suitable value based on their specific requirements. Basically, if a user prioritizes higher accuracy, $r$ should be set closer to 1, whereas if better speed is desired, $r$ should be closer to 0.\n\n    While Figure 6 presents results for a single model, we emphasize that the observed trend is consistent across all evaluated architectures. **For simplicity and reproducibility, we adopt $r=0.75$ as the default value in all experiments, except for low-bit weight-only quantization**, where we set $r=1.0$ (Lines 363, 373, 378). This exception aligns with edge deployment scenarios, where memory constraints dominate, making it crucial to maximize compression while preserving accuracy.\n\n2. Validity Analysis for Method Components\n\n    **Micro-Kernel Specialization**\n\n    MxMoE mitigates the combinatorial explosion of mixed-precision configurations by automating kernel generation (Line 289). To illustrate the effectiveness of our approach, we compare our strategy with two alternative solutions:\n\n    - **Developing a universal kernel to handle all precision combinations**: This approach would compromise kernel performance. We provide a breakdown demonstrating the limitations of this method relative to our micro-kernel specialization. Specifically, the kernel for W4A4-per-channel could theoretically share the same software pipeline with W4A4-group128, but enforcing universality significantly degrades performance (test shape $[8192, 8192, 8192]$):\n\n        | Kernel Type | W4A4_per-channel TOPS | W4A4_group128 TOPS |\n        |-| - |-|\n        | W4A4_per-channel(Specialized) | 1070.5303| N/A  |\n        | W4A4_group128(Specialized)| N/A  | 667.3349|\n        | Unified Kernel| 929.1997 | 412.0268  |\n        ||||\n\n        Unifying the two pipelines requires introducing runtime condition checks, which hinder loop unrolling in the MAC-loop. Moreover, to support group-size=128, the per-channel kernel’s tile-size selection is constrained, making configurations such as tile_k=256 infeasible.\n\n    - **Developing separate kernels for each configuration**:\n\n        While handcrafted kernels could match performance, they require substantial engineering effort. If a given hardware platform supports five quantization candidates (e.g., w2a6, w4a16, w8a8, w4a4, w4a4 with group-size 128), implementing individual kernels for all possible configurations would require $5!=120$ kernels! In contrast, our micro-kernel specialization approach requires implementing only 5 configurable micro-kernels, which are automatically combined by the kernel generator to form optimized fused operators.\n\n    **Resource Configuration**\n\n    To resolve resource mismatches caused by heterogeneous micro-kernels (e.g., varying warp counts per thread block, as shown in Figure 4), we employ the following strategies:\n\n    - Max-resource allocation: Ensuring all parallel units reach the resource ceiling for correctness.\n\n    - Slice-K optimization: This optimization improves shared memory utilization and achieves speedup compared to a baseline without carefully config, we provide a case study: The W4A16-per-channel quantization with/without resource configuration optimization(test shape $[16, 8192, 8192]$).\n\n        | Method | Best Config| W4A16 TOPS |\n        | -- | - | -- |\n        | $\\mathrm{MxMoE}$   | Tile: [16, 64, 128]; Warp: [1,2,2]  | 74.8983    |\n        | $\\mathrm{- Resource Configuration}$ | Tile: [16, 128, 128]; Warp: [1,4,1] | 52.4288    |\n        ||||\n\n        The result shows that with resource configuration optimization, MxMoE automatically discovered a better tile configuration (assign 2 warps along k-dimension, which is not a common configuration in previous kernel such as Marlin[1] e.t.c.), which brings 42% speedup.\n\nWe will enhance the revised manuscript with additional ablation studies and quantitative comparisons to reinforce these claims. We sincerely appreciate the reviewer’s valuable suggestions, which we believe further strengthen the technical rigor and reproducibility of our work.\n\n[1] Frantar, Elias, et al. \"Marlin: Mixed-precision auto-regressive parallel inference on large language models.\" Proceedings of the 30th ACM SIGPLAN 2025.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The trade-off parameter $r$ requires tuning for different models and use cases, which is not thoroughly explored.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "There is a lack of validity analysis experiment for each part of the method, especially Micro-Kernel Specialization and Resource Configuration.", "relation_to_broader_scientific_literature": "Not sure.", "essential_references_not_discussed": "Not sure.", "other_strengths_and_weaknesses": "1. Novel granularity: Linear-block-level quantization allocation improves accuracy-efficiency trade-offs compared to expert-level or uniform approaches.\n2. Hardware-algorithm co-design: Integrates parameter sensitivity, activation patterns, and hardware constraints (e.g., roofline model) into a unified framework.\n3. Performance gains: Demonstrates significant improvements in perplexity and throughput across multiple models and workloads (memory/compute-bound).", "other_comments_or_suggestions": "No.", "questions_for_authors": "Please see \"Experimental Designs Or Analyses\" and \"Methods And Evaluation Criteria\"", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QVMjOOiK3w", "reviewer": "Reviewer_CPad", "summary": "This paper presents MxMoE, a mixed-precision quantization framework for MoE models. The development of MxMoE is driven by three key factors:\n1. Parameter Sensitivity,  linear blocks exhibit significant variability in their sensitivity to quantization.\n2. Expert Activation Frequencies, activation frequencies of experts vary considerably across MoE blocks.\n3. Hardware Characteristics, the weight-activation quantization scheme can be optimized to align with its actual arithmetic intensity for enhanced performance.\n\n## update after rebuttal\n\nI asked some questions regarding the rebuttal, and the authors' response makes sense. I have maintained my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work. Your insightful questions and suggestions have been instrumental in enhancing the quality of our manuscript. Below, we address your concerns in detail:\n1. Rationale for the W5A5 Configuration in Table 1\n  - **Why does the \"strange\" W5A5 setting arise?**\n\n    This configuration is intentional. During bitwidth allocation, we set an average 5-bit activation and weight budget for MxMoE. On our experimental platform (RTX 4090), Tensor Cores support 4-bit and 8-bit operations, allowing both W4A4 and W8A8 configurations to benefit from low-precision Tensor Core acceleration. The additional 1-bit budget provides MxMoE with greater flexibility in bitwidth allocation, meaning that certain activations are assigned 8-bit precision while others remain at 4-bit.\n\n    Although the average bitwidth increases by only 1 bit, MxMoE automatically identifies activations that are more sensitive to quantization through sensitivity analysis and assigns them a higher bitwidth to mitigate accuracy degradation. This exemplifies the advantage of mixed-precision quantization over uniform quantization.\n\n  - **Does the accuracy gain stem solely from the additional 1-bit?**\n\n    Yes. The improved accuracy of W5A5 over W4A4 is primarily due to extreme outliers in the input activations of the down_proj layer—a phenomenon well-documented in prior research (e.g., [1, 2]). Unlike standard outlier features, these activations are particularly difficult to quantize effectively at 4 bits, leading to precision loss. MxMoE mitigates this issue by allocating higher bitwidths to such activations, thereby preserving model accuracy.\n    \n    Our findings align with prior work [2], which demonstrates that suppressing extreme outliers substantially improves post-quantization model performance. We will clarify this rationale in the revised manuscript and provide a case study on bitwidth allocation.\n\n2. Throughput Evaluation with Varied Length Distributions\n  \n    We agree that evaluating computational throughput across diverse input lengths is essential for real-world applicability. In response to your suggestion, we have expanded our throughput experiments. Due to time constraints, we conducted experiments on DeepSeekV2-Lite using the Humaneval-X dataset.\n  \n    In this study, we employed the exact same MxMoE(W5A5) configuration as in Table 1 and Figure 5. To assess performance across varied sequence lengths, we set the batch size to 1—ensuring that input lengths is varied, unlike in the compute-bound scenarios in Figure 5. We then processed the entire dataset, computing the average computational throughput (total FLOPs divided by execution time) and comparing it to FP16 performance. The length distribution of the dataset can be found at https://huggingface.co/datasets/THUDM/humaneval-x.\n\n    | Dataset     | FP16  TOPS | MxMoE(W5A5) TOPS | Speedup |\n    | ----------- | ----- | ----------- | ------- |\n    | humaneval-x | 38.19 | 107.36      | 2.8     |\n    ||||\n\n    As shown in the results, although the speedup is slightly lower than in the fully compute-bound scenario, MxMoE(W5A5) maintains a strong acceleration ratio even on datasets with varied-length short sequences. This highlights the robustness of the mixed-precision approach.\n    \n    We will expand this experiment in the revised manuscript and add more models and dataset tests. Thank you very much for your suggestions!\n\n3. Typos and Figure Refinements\n\n    We sincerely appreciate your careful review and apologize for the typographical errors. These will be corrected in the revised manuscript. Additionally, we will refine the caption of Figure 3 to explicitly define terms such as g128 (quantization group size of 128) and g-1 (per-channel/token quantization) to enhance clarity. Thanks again for your advice!\n\nOnce again, we deeply appreciate your valuable feedback. Your suggestions have been instrumental in improving the quality of our manuscript. If you have any further suggestions or concerns, we would be delighted to discuss them!\n\n [1] Sun, Mingjie, et al. Massive Activations in Large Language Models. First Conference on Language Modeling.\\\n [2] Lin, Haokun, et al. Duquant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs. NeurIPS 2024.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "The benchmarks and evaluation criteria are widely used to assess the performance and computational efficiency of quantized models. For instance in baseline method, QuaRot [1].\nTo thoroughly evaluate computational throughput, we recommend that the authors incorporate additonal benchmarks with varied length distributions.\n\n[1] https://proceedings.neurips.cc/paper_files/paper/2024/hash/b5b939436789f76f08b9d0da5e81af7c-Abstract-Conference.html", "theoretical_claims": "This paper does not contain proofs for theoretical claims.", "experimental_designs_or_analyses": "To the best of my knowledge in this field, most of the experimental designs or analyses are sound. I still have concerns regarding why MxMoE uses the 5-5 setting in Table 1 for weight-activation quantization. Could the observed improvements simply be attributed to the 1-bit increase in precision? I would appreciate it if the author could clarify the rationale behind that.", "relation_to_broader_scientific_literature": "I have not come across any papers that share the key contributions of this one.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "There are some typos for the user's further improvements, such as the incorrect use of capitalization in line 194 and Figure 3.\nCaptions can be further refined; for instance, directly presenting a configuration like g128 and g-1 is not reviewer-friendly for those unfamiliar with the field.", "questions_for_authors": "[1] Why MxMoE uses the 5-5 setting in Table 1 for weight-activation quantization. Could the observed improvements simply be attributed to the 1-bit increase in precision? (See Section \"Experimental Designs Or Analyses\")  \n\n[2] Can the authors incorporate additional benchmarks with varied length distributions to conduct throughput-related tests? （See Section Methods And Evaluation Criteria）", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K9NXyCPQMG", "reviewer": "Reviewer_eUec", "summary": "This paper presents a framework for exploring the design space of mixed-precision quantization in mixture-of-experts (MoE) models. It considers the variation in quantization sensitivity of linear blocks within models, allocating different bitwidths based on sensitivity. Additionally, it takes into account the frequency of block activation to apply different quantization strategies. The proposed mixed-precision approach is claimed to achieve a speedup over uniform quantization while maintaining equivalent accuracy. The method is evaluated using DeepSeek, Mixtral, and Qwen MoE architectures, with comparisons across various quantization methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s engagement.", "claims_and_evidence": "The primary claim is that the proposed mixed-precision quantization framework improves inference speed compared to uniform quantization at the same accuracy level. The claim is supported by experimental results demonstrating speedups on multiple MoE models. However, some key methodological aspects, such as how sensitivity and activation frequency incorporated into the quantization strategy, could be elaborated further. Additionally, the paper should provide more explanation of the quantization methods used in comparison, such as GPTQ and Quaro.", "methods_and_evaluation_criteria": "The evaluation is based on well-known MoE models (DeepSeek, Mixtral, and Qwen) and considers several quantization techniques. The datasets used for evaluation appear appropriate, and the comparisons are reasonably extensive. However, more clarity is needed in describing how different quantization configurations are selected. Additionally, while the paper repeatedly mentions hardware-friendly quantization, it does not explicitly define what hardware efficiency entails—whether it refers to latency, throughput, or other tradeoffs.", "theoretical_claims": "The paper does not present any theoretical claims, and therefore, there are no proofs to verify. Its contributions are primarily empirical.", "experimental_designs_or_analyses": "The experimental setup appears reasonable, but certain details are missing. Specifically, the paper describes the allocator, kernel generator, and task scheduler in Section 4.1 and Figure 1, but their implementation details remain unclear. Some information is presented in Section 4.3, but it does not sufficiently explain how these components function together. A more detailed breakdown of the experimental setup would be beneficial for reproducibility and understanding.", "relation_to_broader_scientific_literature": "The paper references several relevant works but lacks sufficient background discussion on some key concepts. Specifically, more context on GPTQ and Quaro, two quantization methods used in the experiments, would be helpful. Similarly, the features, differences, and challenges of the selected MoE models (DeepSeek, Mixtral, and Qwen) should be elaborated upon to provide a clearer motivation for their inclusion.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n* The paper presents a useful and practical approach to improving MoE model efficiency through mixed-precision quantization.\n* A good comparison is made between different MoE models and quantization schemes.\n* The experimental results demonstrate performance improvements and validate the approach.\n\nWeaknesses:\n* The paper lacks clarity in several areas, including definitions of key terms (e.g., hardware-friendly quantization) and details on the allocator, kernel generator, and task scheduler.\n* The related background on quantization methods (GPTQ and Quaro) and MoE models is insufficient.\n* Some sections are repetitive, restating parameter sensitivity and activation frequency without elaborating on their exact implications.\n* The experimental design could be explained in greater detail for better reproducibility.", "other_comments_or_suggestions": "* Clarify the definition of hardware-friendly quantization—does it prioritize latency, throughput, or another metric?\n* Reduce redundancy in discussions of parameter sensitivity, activation frequency, and hardware characteristics. Instead, elaborate on how they are measured and affect performance.\n* Provide clearer explanations of the quantization methods used in comparison.\n* Expand the descriptions of DeepSeek, Mixtral, and Qwen MoE architectures, highlighting their relevance to this study.", "questions_for_authors": "* Can you clarify what hardware-friendly quantization means in your context?\n* Could you provide more details on the allocator, kernel generator, and task scheduler implementation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QLoZ4XLIIb", "reviewer": "Reviewer_sMX8", "summary": "Mixture-of-expert LLMs are getting important to reduce the traning cost without sacrificing the final model quality. With more MoE LLMs released, it's important to optimize the serving performance for them. Quantization is an important technique to optimize the LLMs serving by use low-precision data types to store the parameters. The authors propose to use different quantization schemes (i.e., the low-precision data type for parameter and activation, and the group-size for channel-wise quantization) at the **linear-block** level instead of expert level. Besides, the authors also implement more efficient kernels to execute the quantized layers. Experiments show that the proposed method MxMoE achieves 2.4 lower Wikitext-2 perplexity than GPTQ at 2.25-bit and delivering up to 3.4x speedup over full precision.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback. We would like to provide further clarification on several aspects of our work that you have mentioned:\n1. Novelty of the Approach and Similarities with Prior Work\n - **Linear-Block-Level Quantization**: Our approach is driven by empirical observations rather than a straightforward extension of prior work. Figure 1(a) highlights significant sensitivity differences between linear blocks, and Table 3 confirms the effectiveness of our approach.\n - **Objective: Balancing Model Accuracy and Hardware Efficiency**: Unlike prior work that solely optimizes model accuracy, MxMoE jointly considers both accuracy and system efficiency. Hardware-friendly quantization strategies improve efficiency but often degrade accuracy (Figure 6). Our objective function is designed to incorporate both quantization perturbation and the runtime cost of a mixed-precision scheme. And accurately modeling runtime cost is nontrivial, requiring deep analysis of hardware characteristics and parallel algorithm design, as detailed in Section 4.2.\n - **Scope: Weight-Only vs. Weight + Activation Quantization**: MC-MoE focuses on extremely low-bit weight-only quantization, whereas MxMoE optimally allocates bit-widths for both activation and weight parameters. This approach enables MxMoE to leverage low-precision arithmetic units on GPUs more effectively (Figure 5).\n - **Performance: Theoretical vs. Real Speedup**: Previous methods relied on existing kernels like HQQ, achieving only theoretical speedups without real-world reductions in wall-clock time (Figure 2). In contrast, MxMoE is designed from the ground up for hardware efficiency, leading to actual runtime improvements.\n - **Automation: End-to-End Mixed-Precision pipeline**: MxMoE is the first framework to fully automate the generation of mixed-precision schemes and fused operators tailored to the generated scheme which is not achieved in any prior works.\n2. MxMoE Is Engineering Optimization\n- MxMoE is not simply providing one kernel but rather offering an automated method for the mixed-precision allocation and generating customized fused operators. The design and optimization of mixed-precision GEMM have been an ongoing area of research[3][4], which continue to push the boundaries of quantization research. MxMoE addresses the more challenging problem of mixed-precision GroupGEMM, which is a much more complex problem that goes beyond simple parallelization of different precision tasks in thread-blocks or streams.\n- In fact, we initially implemented MxMoE with multi-stream, but unfortunately we found it could not fully exploit hardware. We show the comparison(RTX 4090, Qwen1.5, W4A16 and W8A8 mixed MoE-Block):\n\n  | #Tokens 1024|TFLOPS|\n  |-|-|\n  |FP16|70.117|\n  |W4A16|119.32|\n  |W8A8|140.95|\n  |MxMoE|163.20|\n  |multi-stream|142.19|\n\n  The results demonstrated that our approach achieves superior efficiency.\n- Assigning task to CTAs is fundamentally the minimum makespan problem, an NP-hard challenge. \"parallelize different experts with different quantization schemes within a single kernel by using separate thread blocks\" won't work as it suffers from workload imbalance. To address this, we design a tile scheduler that distributes workload evenly among all CTAs(Line 295).\n3. Experimental Results\n- At 3.5 bit, quantization perturbation is relatively limited. Both GPTQ and MxMoE perform well at this setting. As the bit-width decreases, the accuracy advantage of MxMoE becomes more evident. This extreme low-bit compression is highly relevant for resource-constrained model deployment, as explored in several works such as [1][2].\n- MxMoE performs bit-width allocation for both activation and weight. As demonstrated in Table 1 and Figure 5, MxMoE shows a significant accuracy advantage over existing methods in weight-activation quantization, pushing the accuracy-efficiency Pareto frontier to new heights.\n\nIn conclusion, we believe **MxMoE introduces novel contributions in both algorithmic design and system-level optimization**. We sincerely appreciate the reviewer’s feedback and the opportunity to clarify our contributions. We hope our responses have effectively addressed the concerns raised. Given these clarifications, we hope the reviewer may reconsider the overall evaluation of our work. We would be happy to further discuss any remaining questions or concerns.\\\n[1] Ma, Shuming, et al. \"The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits.\" CoRR (2024).\\\n[2] Yuan, Zhihang, et al. \"PB-LLM: Partially Binarized Large Language Models.\" The Twelfth International Conference on Learning Representations.\\\n[3] Wang, Lei, et al. \"Ladder: Enabling Efficient {Low-Precision} Deep Learning Computing through Hardware-aware Tensor Transformation.\" 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24).\\\n[4] Lin, Yujun, et al. \"Qserve: W4a8kv4 quantization and system co-design for efficient llm serving.\" arXiv preprint arXiv:2405.04532 (2024).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Efficient execution of quantized MoE LLMs.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**  \n- Provides an efficient implementation for quantized MoE layers.  \n\n**Weaknesses**  \n- The novelty of quantizing at the block level is incremental.  \n- Although the authors implemented new kernels to support the proposed quantized MoE layer, these contributions seem more like engineering improvements rather than groundbreaking innovations.  \n\n**Details**  \nThe novelty of quantizing at the block level is incremental. As the authors mention in the background section, prior works like MC-MOE have already observed that different experts have varying levels of significance or importance and could be quantized into different low-precision data types. These works also employ linear programming to determine the optimal quantization strategy.  \n\nThe new kernel implementation appears to be more of an engineering improvement. Since the quantization granularity is at the linear level, it is natural to parallelize different experts with different quantization schemes within a single kernel by using separate thread blocks to execute these layers or by utilizing different CUDA streams to launch distinct kernels. The innovations in this aspect seem more like engineering contributions to the community rather than fundamental algorithmic advancements.  \n\nThe accuracy does not show significant improvement in the W3.5-A16 setting compared to the uniform baseline GPTQ. From Table 1, MxMoE (W3.5-A16) achieves similar average accuracy to GPTQ (W3.5-A16), and MxMoE only demonstrates better accuracy at even lower precision (e.g., 2.25-bit). If MxMoE or similar methods that use different quantized types for different experts only prove effective at very low precision (<3-bit), their impact and contribution would be considerably diminished.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a new MoE model quantization method with mixed precision considering the sensitivity of the linear layers considering various factors. Also, the proposed MxMoE automatically generates optimized Group-GEMM kernels for mixed precision.

In the review process, there were several clarification questions from the reviewers including trade-off parameter r, micro-kernel specialization, usage of W5A5 and novelty of the proposed method. In the rebuttal and discussion phase, the authors provided relevant answers and information to those questions. 

After the discussion, all the reviewers excluding the one detected as LLM-generated had a consensus of weak accept. (3 weak accepts) In the reviewer discussion, a reviewer shared that the proposed method made sense, but was not exciting.

Combining all those information, I recommend a weak accept if there's an available spot for this paper as it potentially has some contributions even though it may not be significant.
Decision: Accept (poster)