MicroMix: Efficient Mixed-Precision Quantization with Microscaling Formats for Large Language Models
Authors: Wenyuan Liu
Haoqian Meng
Yilun Luo
Peng Zhang
Xindian Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Mixed-precision Quantization
Microscaling Formats
Post-training Quantization
Large Language Models
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: A mixed-precision quantization method using MXFP8, MXFP6 and MXFP4.

Abstract: Quantization significantly accelerates inference in large language models (LLMs) by replacing original high-precision matrices with low-precision counterparts. Recent advances in weight-activation quantization have primarily focused on mapping both weights and activations to the INT4 format. Although the new FP4 Tensor Cores in NVIDIA’s Blackwell architecture offer up to 4$\times$ speedup over FP16, existing INT4-based kernels fail to fully exploit this capability due to mismatched data formats. To bridge this gap, we propose MicroMix, a co-designed mixed-precision quantization algorithm and GEMM kernel based on Microscaling (MX) data formats. Tailored for the Blackwell architecture, the MicroMix kernel supports arbitrary combinations of MXFP4, MXFP6, and MXFP8 channels, and produces BFloat16 outputs. To achieve a favorable trade-off between accuracy and efficiency for each linear layer, we introduce quantization thresholds that identify activation elements where lower-precision formats (MXFP4 or MXFP6) incur excessive quantization error. Our algorithm selectively allocates higher-precision channels to preserve accuracy while maintaining compute efficiency. On the Llama and Qwen model families, MicroMix achieves near-FP16 performance across diverse downstream tasks with an average precision of 5 bits. In particular, Qwen2.5-32B-Base, Coder and Math exhibit lossless accuracy on zero-shot, code generation, and mathematical reasoning benchmarks. In addition, on RTX 5070Ti laptop and RTX 5090 GPUs, our kernel achieves 2.29-3.38$\times$ acceleration compared to TensorRT-FP16. Our code is available at https://github.com/lwy2020/MicroMix.

Published as a conference paper at ICLR 2026
MICROMIX: EFFICIENT MIXED-PRECISION QUANTI- ZATION WITH MICROSCALING FORMATS FOR LARGE LANGUAGE MODELS
Wenyuan Liu', Haoqian Meng', Yilun Luo', Peng Zhang'', Xindian Ma¹ College of Intelligence and Computing, Tianjin University, Tianjin, China [Iwy2020, typedef, ly12023, pzhang, xindianma @ @tju.edu.cn
ABSTRACT
Quantization significantly accelerates inference in large language models (LLMs) by replacing original high- precision matrices with low- low-precision counterparts. Re- cent advances in weight-activation quantization have primarily focused map- ping both weights and activations to the INT4 format. Although the new FP4 Tensor Cores in NVIDIA's Blackwell architecture offer up to 4x speedup over FP16, existing INT4-based kernels fail to fully exploit this capability due to mis- matched data formats. To bridge this gap, we propose MicroMix, co-designed mixed- sxxeedprecision quantization algorithm and GEMM kemell based on Microscal- ing (MX) data formats. Tailored for the Blackwell architecture, the MicroMix kernel supports arbitrary combinations of MXFP4, MXFP6, and MXFP8 chan- nels, and produces B6 outputs. To achieve a favorable trade-off between accuracy and efficiency for each linear layer, we introduce quantization thresh- olds that identify activation elements where lower-precision formats (MXFP4 MXFP6) incur excessive quantization error. Our algorithm selectively allo- cates nigher-precision channels to preserve accuracy while maintaining compute efficiency. On the Llama and Qwen model families, MicroMix achieves near- FP16 performance across diverse downstream tasks with an average precision of 5 bits. In particular, Qwen2.5-32BBBB.. curacy on zero-shot, code generation, and mathematical reasoning benchmarks. In addition, on RTX 5070Ti laptop and RTX 5090 GPUs, our kernel achieves 2.29-3.38x accelcration compared to TensorRT-FP16. Our code is available at
Coder and Math exhibit lossless ac-
INTRODUCTION
In recent years, large language models (LLMs) have demonstrated remarkable performance across wide range of tasks /Vaswani et al. 2023 Brown et al. 2020). However, these capabilities come with substantial computational and energy costs. To mitigate this, quantization techniques replace high- precision matrix multiplications with more efficient low-bit alternatives (Yao et al. 2022 Xiao et al.) 2024], significantly improving L.IM inference speed. Quantization techniques are broadly classified into weight-only and aettht-activation approaches. Weight-only methods (Lin et al. 2024b) Frantar jet al., 2023) Yang al. 2025) have substantially mitigated the precision loss associated with 4-bit weights and 16-bit activations (W4A16). In parallel, weight-activation methods (Dettmers et al.) 2022 Xiao et al. 2024) suppress activation outliers effectively, enabling accurate 8-bit quantization of both weights and activations (W8A8). More recently, mixed-precisio and rotation-basee quan- tization algorithms (Ashkboos et al.] 2024; Zhao et al.) 2024) have pushed the frontier further to W4A4, achieving strong performance downstream tasks.
Despite these advances, two key bottlenecks continue to restrict the kernel-level efficiency of INT4- based quantization: (1) The widely adopted group-wise integer quantization scheme requires de- quantizing each integer group to floating-point values followed by partial summations. This proce- dure is executed on slower CUDA Cores, as INT8 Tensor Cores only support INT32 accumulation.
"oorreponning Author: Peng Zhang

---

Published as a conference paper at ICLR 2026
(2) NVIDIA's latest Blackwell architecture introduces FP4 Tensor Cores that offer up to 4x higher throughput than FP16 and 2x higher than FP8 or INT8. However, existing INT-based quantization kernels are incompatible with these new tensor cores and thus fail to leverage their full potential. As a result, significant room remains for optimizing quantization kernel throughput on the Blackwell architecture.
Blockoclling
Quantiie
MXTI
MIXFP
MEFPPS
T(6)
Activation
(4)
Static Reordee
Weight!
Blackscaling
Dynamic Receder
Quanrine
(b) Quantization Threshold
Accemulate
FP
Fh
FP8
Output
(a) Overview of MicroMix
(c) MSE Loss on 32 Sumples from WikiText2
Figure 1: (a) MicroMix reorders channels and allocates different bit-widths accordingly. (b) The quantization thresholds T(4) and (66) partition elements into three groups based on their quantiza- tion error magnitude. (c) MicroMix consistently achieves lower quantization error across all layers.
In this paper, we propose MicroMix, a mixed-precision quantization framework based on Microscal- ing (MX) data formats, featuring co-designed algorithm and kernel. The key components of Mi- croMix are as follows:
(1) Flexible bit-width ratios (4, 6, and 8 bits). To balance efficiency and accuracy, MicroMix assigns customized ratios of three precision levels to each linear layer. The quantization kernel supports multiple Microscaling formats (MXFP8, MXFP6, MXFP4) and arbitrary mixing ratios. By leveraging CUTLASS GEMM, we instantiate optimized matrix multiplication kernels tailored to specific data types and problem sizes. In addition, dequantization operations are deeply fused into MMA instructions, introducing negligible overhead on Blackwell Tensor Cores.
(2) Low-error precision assignment strategy. We propose a bit allocation algorithm that adapts to input distributions from the perspective of quantization error. The key idea is to ensure that the quantization error of lower-bit formats remains below the upper bound of higher-precision formats. To this end, we define explicit quantization thresholds for MXFP4 and MXFP6: elements exceeding the threshold at a given bit-width are reassigned to higher-preciste formulation introduces explicit outlier thresholds for MXFP4 and MXFP6, addressing limitation of prior work. As a result, MicroMix significantly reduces the quantization error induced by MXFP4, as shown in Figure ee).
formats (see Figure [b)). This
(3) Efficient morderr-nndquanticn operation. Since adjacent channels may be assigned differ- ent bit-widths, channels of the same precision need to be reordered into the same block. Without reordering, applying mixed-precision quantization directly results in irregular memory access and considerable overhead. To address this, MicroMix integrates the reordering step into the quanti- zation kernel (Figure a)), enabling high throughuut quantization across heterogeneous precision levels with negligible additional latency.
We evaluate MicroMix on multiple downstream tasks, including zero-shot and few-shot learning, language modeling, code generation and mathematical reasoning. Across various L.lama and Qwen models, MicroMix generally maintains at least 98% FP16 accuracy on zero-shot, code and math benchmarks, achieving comparable to or better than state-of-the-art baselines. In particular, Mi- croMix achieves near-FP16 performance erage bits about 5.2. For efficiency analysis, we evaluate the MicroMix kernel on RTX 5070Ti laptop, RTX 5090 and RTX PRO 6000 GPUs. Compared with TensorRT-FP16. MicroMix achieves a kernel-level speedup of 2.45-2.93x x on the RTX 5070Ti laptop and 2.29-3.38x on the RTX 5090. When integrated into the Transformer architecture, MicroMix achieves 1.98-2.02x higher end-to-
Qwen2.5-32B models (Base and Coder) with av-
2

---

Published as a conference paper at ICLR 2026
end compared to FP16. For end-to-end efficiency, MicroMix delivers at least 1.82 times higher decoding throughput than INT4 baselines on RTX PRO 6000.
2
PRELIMINARY AND MOTIVATIONS
2.1
PRELIMINARY
Given an activation tensor X and a weight tensor W, quantization approximates the original matrix multiplication with low-precision computation:
= Q(X)Q(W) 8x8W, Q(X) roundd X
where sx and sw are the scaling factors of X and W respectively. VX, E X, the quantization error for Xj is defined as:
E(X,) = - XX, round X
where 7 = |round( (5/a)= - -x//s| is the rounding error. Appendix C.I we further analyze relationship between quantization error and model accuracy. Empirically, observe that model accuracy remains close to the FP16 baseline as long as the quantization error constrained within a exceeds this threshold, accuracy degrades
specific threshold. However,
the quantization rapidly. For recent LLMs, INT8 quantization typically remains within the high-accuracy region, whereas INT4 often lies near the onset of significant accuracy degradation.
Microscaling data formats (MX) are advanced numerical formats designed for deep learning. The basic unit of MX is block of size k;, consisting of & scalar elements {X,} and single shared scaling factor in E8MO (Darvish Rouhani et al. (023a)). Recently, DeepSeek V3.1 (DeepSeek-AT] 2024) was trained using the UE8MO FP8-scaled data format for both model weights and activations, ensuring compatibility with microscaling formats. Given a FP16 tensor X E IR LxX quantization MXFF8MMXFP/MXXPP first partitions X into blocks of 32 elements (X; 1==1 applies per- block symmetric quantization for VX, E X, as follows:
L-1 then 32
Q(X,) round X )
(3)
where round(.) denotes rounding to the nearest MXFP value and the exponent bias format- specific (see Appendix B for more details).
2.2 MOTIVATIONS
The primary motivations of this paper stem from addressing the limitations in current quantization methods and their corresponding kernels.
Motivation 1: Adaptive Mixed-precision Al- location for Diverse Activation Distributions. Existing mixed precision quantization methods such as Atom (Zhao et al. 2024), employ fixed number of high-precision channels across all layers. This uniform allocation fails to ac- count for the heterogeneous activation distri- butions observed in different layers (see Fig- ure 2 Specifically, layers with larger activa- tion values across channels require more high- precision channels to reduce the quantization enror. Consequently, directly applying current fixed-allccation mixed-precisie MX formats leads to noticeable degradation in accuracy (see Table II Appendix[D... DD33.. To overcome this, we propose novel strategy that
lryenut ser air pij
Syye' Lettee
lpen 21 40.9
Mose prggigicniin
320
layss ur disage
MissMMi
2007
Layer Adaptive
Figure 2: Channel-wise mean values of three ac- tivation tensors from Llama3. 1-8B, with outlier channels reordered to the end. Compared to prior methods, MicroMix assigns a larger portion of channels to higher-precision formats and applies layer-wise adaptive precision ratios across all lin- ear layers.
algorithms to

---

Published as a conference paper at ICLR 2026
flexibly allocates the number of 4, 6, and 8-bit channels per layer. This adaptive approach ensures that all linear layers consistently maintain low errors, thereby improving model accuracy.
Leveraging FP4 Tensor Cores for Enhanced Kernel Efficiency. Current INT- based kernels, exemplified by Atom and QuaRot (Ashkboos et al. 2024), require dequantization on CUDA Cores because INT8 Tensor Cores only produce INT32 partial sums. The dequantization process on CUDA Cores limits the performance of these INT kernels (Lin et al.] 20256). In stark contrast, FP4 matrix multiplication allows for direct dequantization on FP4 Tensor Cores, leading to a significant improvement in computational efficiency. This fundamental advantage highlights the critical need for developing -exerrenntttion mixed- precision methods with kernels specifically designed for FP formats.
Motivation 2:
Motivation 3: Quantization Error Management through Adaptive Thresholding for Outliers. Quantization error, inherent to the format conversion between original activations X and their quan- tized representations Q(X), cannot be entirely eliminated. Therefore, emror remains within acceptable bound. While prior works have introduced various techniques, such as smoothing, rotation, and clipping, to mitigate the impact of outliers in activations. There is still notable gap in research concerning the precise threshold above which outliers should be constrained for MXFP4 and MXFP6. In this paper. we define specific quantization thresholds for MX. Elements exceeding these defined thresholds will be preferentially stored in higher bit-width, thereby effectively minimizing quantization error and maintaining high model fidelity.
critical to ensure this
3
METHOD
To address the accuracy degradation observed in INT4 quantized models on downstream tasks, prior work has explored various solutions. However, post-training quantization for Microscaling (MX) formats remains underexplored. Leveraging the inherent flexibility of multiple MX data formats, propose MicroMix, novel co-designed mixed-precision quantization algorithm and kernel.
3.1 ALGORITIMM
In MicroMix, the activation tensor channels are partitioned into three groups, G., Go. and Gs, which are quantized MXFP4, MXFP6, and MXFP8, respectively. The corresponding weight channels are quantized to the same bit-width as their activation counterparts.
Reducing Quantization Error through Permutation. Due to the limited bit-width, the quanti- zation error of MXFP4 or MXFP6 cannot, in general, be lower than that of INT8. Given token X E R our key idea is to constrain the quantization error of MXFP4 and MXFP6 such that remains within the upper bound of the error introduced by INT8:
S E(X)NNSS VX E G
(4)
According to Equation 2, the reduction of quantization error MX primarily depends on lowering the maximum value within each block of X,. A straightforward approach is to group large val- ues into the same blocks while keeping smaller values together. To achieve this, we introduce permutation that rearranges the elements of X in ascending order:
0: X - -x-o(XX (X)
(5)
Defining Quantization Threshold for Partitioning. After permutation, the next step is to deter- mine the groups G4, Go, Gg. To accurately distinguish outliers from regular elements, we define the quantization threshold as follows:
Definition 1. Given high-precision bit-width (e.g., 8-bit for recent LLMs) and target bit-width n, the quantization threshold T(n) is defined as:
T(n) = 2b_ 2n-1
E(X)INTS
(6)
Jmar
To maintain low quantization error at MXFP4 MXFP6, the maximun allowable magnitude within group must satisfy:
max(|G <(n))
4

---

Published as a conference paper at ICLR 2026
Here, denotes the number of bits, is the exponent bias and qmaz represents the maximum repre- sentable value in the target format. A detailed derivation of the quantization threshold is provided in Based on the thresholds, the groups G4, Go, and Gg are defined as:
Appendix C2
G = [X]X T G6 = (x < X S T(6)) G. = (XIT(6) < X
(8)
We calculate the proportions P4, Ps. and Py corresponding the channel groups Go. G5. and Gg for each linear layer in 1lama3.1-8B. The results are shown in Figure observations:
We summarize three key
Layer-wise Adaptivity: The proportions vary dynamically across layers, reflecting the diverse in- put distributions in each activation. This demonstrates that the mixed-precision allocation is layer- specific rather than fixed globally.
FP4 Dominance: The proportion P4 consistently exceeds 50%, indicating that FP4 computations dominate the mixed-precision tional efficiency of the model.
workflow. This dominance contributes significantly to the computa-
Datsset Stability: The variations of P4. Pa- and ps across different datasets and sampling strategies are minimal, suggesting that the mixed-precision assignment remains relatively stable.
V.V.Po Poj
G Fruj
Gate Fruj
Boen Paoj
Muni
2s Mua
Max
Boage
Bagge
Regg
Figure 3: Distribution statistics of P4 (E2M1), Ps (E3M2). and ps across Llama3.1-887 We evaluate 32 samples selected from WikiTeex2 (Merity et al. 2016) and the Pile dataset (Gao et al.) 2020), covering batch sizes of 8, 16, 32, and 64, and sequence lengths of 512, 1024, 2048, and 4096. For each sample, p4, P6, and ps are computed over all linear layers. The figure reports the mean values and min-max ranges of p4. Ps, and pa across all samples.
Offiin Channel Assignment Strategy. Online evaluation of channel partitioning would introduce substantial runtime overhead. Instead, leveraging the observed stability of p4, Ps, and ps. we pre- compute {ri P p o*} for the kth linear layer offline using calibration data. To allocate higher precision to more critical channels, we sort the activation channels according to their absolute mean values. Specifically, for the kth linear layer input tensor X E R the channel-wise absolute mean vector M E IR' is computed as:
M
-
Σ 1=1
T,
The permutation a² is obtained by sorting the elements of M in ascending order. Let ph, pê. and p denote the proportions obtained for X*; the channel partitioning then defined as:
G1 = ot o (X) ().rpt Go = x*((( (X)
G3 = x (X)..tt +r
(10)
3.2 KERNEL DESIGN
Low-bit quantization offers significant performance improvements but presents considerable chal- lenges in kernel design, especially for Recent advancements in GPU architectures, particularly the increased throughput of Tensor Cores for low-bit floating-point operations, combined with underlying support for block-scaled formats, have diminished the competitive advantage of traditional INT-type and GEMM quantization ker- nels. This simultaneously creates opportunities for low-bit floating-polt quantization.
and fine-grained schemes like MicroMix.
Mixed -eecidirreision Quantization. Driven by the goal of deep algorithm-harre hardware integration, we have designed mixed-precision, block-scaling quantization kemnel for MicroMix. Complementing
5

---

Published as a conference paper at ICLR 2026
this, adopt MXFP- type GEMM kemells from CUTLASS, resulting in kernel suite that delivers both excellent performance and high accuracy.
Fine-grained Block-scaled Data Formats. Quantization error is also influenced by the number of elements sharing a single scale factor. To fully harness the representational power low-bit data types, fine-grained group quantization has become widely adopted and proven efficient in related works such as Atom and QuaRot. This used to be a tough trade-off between accuracy gains and dequantization overhead. However, the NVIDIA Blackwell architecture changes the game. Black- well's mma instructions directly support new 4, 6, and 8-bit floating-poine data types with integrated scale factors (known as MXFP formats), making block-scaled quantization truly practical solution.
GEMM Kernel. As shown in Figure 4 (a), the output matrix is divided into blocks within each GEMM kernel, with iterations for each block performed along the K dimension. After loading fragments of input matrices and their scale factors into Shared Memory or Tensor Memory, MMA instructions fused with dequantization operations are continuously executed on Tensor Cores. These operations accumulate FP32 partial sums into the 16 result matrix. The process is highly decoupled, as matrices specific data types invoke their corresponding GEMM kernels. This design allows for easy adjustment of data type categories and their ratios.
Seriiions
Scctooainng VIAA
dkgantie
MEXEP4
O(W)
Q
MXFFS
B
Badooakar NDAA PENTESSRRR
Sendions
MCXEPN
X
Roullt
Hecationg
Shokeddig MIMA
Drane
Jonds
Segazin
FPI
(b)
Figure 4: (a): The fused GEMM kernel of MicroMix. (b):
The fused reorder- and -uantiiattize operation. The quantization of weights is one-time cost and could be performed offline.
Quantization Kernel. Mixed-precision quantization often faces ir- regular memory access, leading to significant performance degrada- tion. To tackle this, MicroMix adopts strategy similar Atom and RPTQ (Yuan et al. 2023) by reordering channels to enable regular memory access. Our algorithm divides channels of activation into three distinct parts, to which we then apply block-wise scaling quan- tization 32-element blocks. To ensure correct matrix multiplica- tion, weights are correspondingly permuted match the reordered activations before undergoing similar three-part block-wise scaling quantization. Crucially, the reordering and quantization of activations must occur dynamically, while these processes for weights can be handled offline as a pre-processing step. To mitigate the overhead of dynamic reordering, we employ a kernel fusion technique (see Fig- (b)), which combines the quantization and reordering operations into a single kernel. As shown in Figure our fused kernel introduces little overhead compared to mixed-precisee quantization only.
Becyder Moaderee dan Quantiatiin fazier Danntlation
Length
Figure 5: Comparison of
the latency between single and fused operations with batch size of 32.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Quantization. MicroMix performs block-wise symmetric quantization with block size of 32 for both weights and activations, using the E8MO scaling format. The data formats are MXFP8 (E4M3), MXFP6 (E3M2). MXFP4 (E2M1) respectively. In Appendix D.I Table 9 provides summary of the quantized model information, including average using bits, offline calibration time and the size of quantized models.

---

Published as a conference paper at ICLR 2026
Baselines. We compare MicroMix against four INT-based weight-activation quantization methods: Atom (Zhao et al. 2024). QUIK (Ashkboos et al. (023)). QuaRot (Ashkboos et al. 2024). FlatQuant
(Sun al. 2025) and one MX- -X-based method, AMXFP4 (Lee et al. 2025). All basclines repro-
duced on both Llama (Grattafiori et al.] 2024) and Qwen (Qwen et al. 2025). Since MicroMix
employs non-fixed bit-widths across linear layers, additionally report the average bit-width per
token element for all methods in Table Implementation details are provided in Appendix D.1
Benchmarks.
For zero-shot evaluation, use ARC.C (Clark al. 2018),
Lambada (Paperno 2019). and PIQA (Lourie
et al.
2016). Winogrande (Sakaguchi et al. 2019), BoolQ (Clark et al.
al. 202 p. For five-shot aceuracy, we adopt MMLU (Hendrycks et al., 2021a). WikiText2 (Merity
et al. 2016) is used to evaluate perplexity (PPL). Additionally, we assess the Code and Math capabil-
ities of the Qwen2.5 model series. Code benchmarks are Human-Eval (Chen et al. 202T) and MBPP
Austin et al. 202 while Math benchmarks cover GSM8K (Cobbe et al. 202 D. MMLU-STEM
Henddyycks al. 2021a). CMATH (Wei et al. 2023) and MATH (Hendryeks al. 2021b)
4.2
MAIN RESULTS
Table reports zero-shot and five-shot accuracy, along with WikiText-2 perplexity, for MicroMix and six baselines on 3lama3.1-8B and Qwen2.5-32B. Across the five zero-shot benchmarks, Mi- croMix is the only quantization method that consistently preserves least 98% of FP16 average accuracy on both models (Llama: 71.56 vs. 73.03; Qwen: 75.20 vs. 75.55). On the five-shot MMLU benchmark, MicroMix retains least 96% of FP16 accuracy (Llama: 62.65
65.24; Qwen: 81.79 vs. 83.32), outperforming all competitors by 2 2.33 points on Llama and 20,27 points on Qwen. For WikiText2, MicroMix incurs only marginal perplexity increase of 0.48 Llama3.1- 8B (6.72 vs. 6.24) and 0.46 on Qwen2.5-223 5-3BB (5.56 vs. 5.02). The results of KV cache quantization demonstrated in Table 10 of Appendix D.3 D.3
Table 1: Zero-shot and few-shot accuracy and perplexity of Llama3. 1ammaa..180 1-BB and Qwen2.5-32B evalu- ated with 1m Am-eval (Gao et al., 2024). "Avg. Bits" denotes the average bit-width per token element. INT6 is implemented using symmetric per-token quantization.
Avg. Bits
O-shat (f)
5 (f)
PPL-(L)
Model
Method
ARC_C
BoolQ
Lambada
PIQA
Winogrande
Avg.
MMLU
WikiText2
FP16
16.00
53.58
81. 81.99
75.47
80.09
74.03
73.03
65.24
6.24
QuaRot
4.12
46.42
76.24
68.48
77.91
70.96
68.00
55.23
6.98
QUIK
5.95
44.62
77.09
73.28
77.09
69.19
68.05
56.65
7.29
Llama3.1-8B
Atom
4.25
50.17
76.15
69.45
78.02
70.01
68.76
58.05
6.79
FlatQuant
4.19
51.54
78.87
73.32
79.16
71.98
70.97
61.33
6.95
AMXFP4
5.00
74.80
75.19
66.34
INT6
6.00
48.38
77.37
69.86
78.29
70.01
68.78
58.67
7.53
MicroMix
5.51
50.26
81.13
74.13
80.14
72.14
71.56
62.65
6.72
FP16
16.00
55.89
87.46
76.21
82.26
75.93
75.55
83.32
5.02
QuaRot
4.12
84.77
80.96
73.36
5.86
QUIK
6.21
52,90
85.87
74.21
80.36
71.82
73.03
78. 89
5.92
Qwen2.5
Atom
4.21
54.78
86.54
75.92
81.45
73.48
74.43
79.54
5.89
FlatQuant
4.71
56.23
86.30
75.41
81.50
74.19
74.72
81.52
5.74
AMXFP4
5.00
51.54
87.09
75.24
80.85
73.51
73.64
79.96
5.85
INT6
6.00
72.15
MicroMix
5.22
56.66
87.13
77.37
80.65
74.19
75.20
81.79
5.56
Notably, higher average bit-width does not necessarily translate into higher accuracy. For instance, QUIK and INT6 employ more bits than MicroMix, yet provide limited performance gains.
Table 2: Mixtral-8x777 8x7B-v0.1 Instruct performance comparison between FP16 and MicroMix.
Arc.C A
BoolQ
Lambada
PIQA
Winogrande
Avg.
Execution Time
FP16
65.70
88.50
77.37
84.49
76.87
78.58
5min 18s
MicroMix
64.25
88.07
78.52
84.00
76.16
78.20
2min 03s
As shown Table 2
MicroMix attains accuracy comparable to FP16 on Mixtral-8x7B-v Instruct, with average score drop of only 0.38 points (78.58 78.20) and per-task differences
7

---

Published as a conference paper at ICLR 2026
typically within +1.5 points. Notably, this minor accuracy trade-off comes with substantial runtime reduction, cutting execution time from 5m18s to 2m03s.
Math benchmarks. Table El shows that MicroMix incurs average accuracy drop of less than 4% compared to FP16, while retaining at least 98.4% of FPI6 accuracy on GSM8K, MATH, and CMATH, with average bit-width of 5.16.
Table 3: Accuracy (1) of Owen2.5-Math-- Matt-rt-Inntruct Instruct math benchmarks: STEM, CMATH, and MATH. FP8 is implemented by vLLM (Kwon et al. 2023
GSM8K, MMLU-
Model
Method
GSM8K
MATH
MMLU-STEM
CMATH
Average
FP16
95.8
83.7
77.8
91.5
87.2
7B
FP8
95.5
83.4
68.7
91.7
84.8
MicroMix
95.1
82.4
66.5
91.5
83.8
Code benchmarks. As reported in Table MicroMix achieves accuracy comparable to or better than INT8 on the 14B (Avg. Bits: 5.54) and 32B (Avg. Bits: 5.18) models. Relative to FP16, the accuracy degradation remains within 1.5%.
Table 4: Accuracy (f) of Qwen2.5 5- 5-:55---Crrr-ii (14 on Code benchmarks: Human-Eva and MBPP. INT8 is implemented by Bitsandbytes (Dettmers et al. 2022)
Model
Method
Human-Fval
Human Human-Fxal+ Eval+
MRPP
MBPP+
FP16
87.8
84.1
81.0
69.2
14B
INT8
86.6
82.9
86.0
73.0
MicroMix
87.4
82.9
85,4
70.1
FP16
88.4
84.1
84.5
70.9
32B
INT8
89.0
86.2
73.5
MicroMix
89.0
86.8
74.1
4.3 ABLATION STUDIES
In this section, we analyze the potential impact of different data formats and calibration datasets.
MXFP6 and MXFP8 Variants. We examine the impact of different MXFP6 (E2M3 and E3M2) and MXFP8 (ESM2 and E4M3) variants on zero-shot accuracy and perplexity. As shown in Table all four configurations yield comparable results, suggesting that the specific exponent-mantissa trade- off has only minor effect on MicroMix. This robustness arises because the definition of quantiza- tion thresholds explicitly accounts for the influence of both exponents and mantissas on quantization егror, thereby mitigating sensitivity to data format choices.
Table Zero-shot accuracy (1) on Winogrande, Lambada, PIQA, and perplexity (L) on WikiText2, using different exponent and mantissa bits for MXFP6 and MXFPS on Llama3.1-8 MXFP4 is E2MI consistently.
MXFP8
MXFP6
Winogrande
Lambada
PIQA
WikiTex12
E3M2
72.53
73.36
80.25
6.84
ESM2
E2M3
72.69
72.83
80.14
6.81
E3M2
72.14
74.13
80.14
6.72
B4M3
E2M3
71.51
74.07
80.09
6.73
Impact of Calibration Datasets. To assess the robustness of offline partitioning, we test different calibration datasets, including WikiText2, Pile and C4 (Raffel et al. 2019). As shown in Table 12 of Appendix D.3] zero-shot and perplexity results remain stable across datasets with performance fluctuations within approximately 1%.
Time Breakdown by Component. We use the average values of p4. Pa. and ps from Llama3.1 1- 8B to compute the runtime breakdown of reorder-and-quantize and GEMM relative to the total
8

---

Published as a conference paper at ICLR 2026
MicroMix kernel time, shown in Table 6 Fused reorder-and aat-annddiqzzzz operation takes less than 20% MicroMix kernel runtime across different lengths.
Table 6: The proportion of runtime for each part on RTX 5090.
Part
Length 128
Length 256
Length 512
Length 1024
Length 2048
Length 4096
Reorder-and- Suant--Qttieee
7.9%
9.7% 90.3%
11.8% 88.2%
14.3% 85.7%
17.0%.
16.9%
GEMM
92.1%
83.0%
83.1%
4.4 EFFICIENCY EVALUATION
In this section, assess the efficiency of MicroMix from three perspectives: (1) single-kernel execution speed: (2) speedup of our custom kernel relative to CUTLASS; and (3) end-to-end per- formance in the prefill and decode stages. We evaluate MicroMix on three Blackwell architecture GPUs: RTX 5070Ti laptop, RTX 5090, and RTX PRO 6000, to examine its applicability across consumer and server GPUs.
Kernel Efficiency. We measure the latency of the MicroMix kernel across varying sequence lengths and hidden sizes. As baselines, tensor), W4A16 (per token), and FP16. Because MicroMix employs nonfixed combination of 4-, 6- and 8-bit channels, all kernel and transformer-block experiments report the minimum maximum ranges alongside mean-value curves. As shown in Figure 66] MicroMix consistently outperforms TRT FP8 on both GPU platforms. On the RTX 5070Ti laptop. it achieves 2.45 2.93 speedup over TRT FP16 and up to 1.45x over TRT FP8. On the RTX 5090, MicroMix delivers 2.29 to 3.38x speedup over TRT FP16 and up to 1.74x over TRT FP8.
strong TensorRT implementations: TensorKT FP8 (per
Uama2-78 Lastag)
Laptop)
7ue42.5-114 5090)
Latency (02)
Latency (nc)
0.6
1.0
1.0
0.5
0.0
512
2040
4286
0.0
1024
0.0
2040
4095
0.0
1024
2048
0152
Length
Length
Length
Length
Microiix Range
MorI Mean
TRT
WAT
FRT
Figure 6: Computation latency of single kernel with different lengths. "MicroMix-Range"" denotes the latency span from the fastest to the slowest time.
Performance of our customized kernel. Table 7 reports the speedup of our customized GEMM kernel relative to CUTLASS. For small problem sizes with N = K 4096, our kernels consistently outperform CUTLASS, achieving speedups of approximately 2.6 to 4.0x for W4A4, 3.1 to 5.0x for W6A6, and 1.2 3.1 for W8A8 M increases. The gains are most pronounced moderate M values, for example M = 32, indicating improved utilization and kernel efficiency for low-precision formats, particularly W6A6.
Table 7: Customized GEMM Kernel Speedup over CUTLASS on Small Problem Size (N=K=4096),
W4A4
W6A6
W8A8
CUTLASS TELOPS
Customized TFLOPS
Speedup
CUTLASS TFLOPS
Customized TFLOPS
Speedup
CUTLASS TFLOPS
Customized TFLOPS
Speedup
L.44
2.72
2.62x
0.65
2.04
3.14%
1.02
1.63
1600
2.63x
3.59
3.27
1.55
4.16
10.91
2.62x
2.62
8.19
3.13%
4.16
6.55
1.57%
8.09
26.17
3.23%
5.23
18.72
3.58%
8.46
16.37
1.93%
$2.37
37.42
32
32.58
130.99
4.02x
20.95
104.75
5.00%
33.58
104,80
3.12%
64
66.82
209.50
3.14%
41.92
134.61
3.21x
67.28
130.96
1.95%
260.90
1.99
161.35
1.92
1.19x
Comparison of 4-bit baselines. To demonstrate the end-to-end efficiency of MicroMix. we com- pare against two 4-bit baselines, Atom and QuaRot. As shown in Figure MicroMix reduces prefill

---

Published as a conference paper at ICLR 2026
latency by about 85 percent compared to Atom and QuaRot RTX PRO 6000. In the decode stage, MicroMix increases throughput by 1.82 to 3.02 times compared to Atom.
Prefill Latency (ms))
Decade Stage (tokenys)
1259
1859
850
330
620
430
320
150
NO. 16
10--66
20-11
20.
17
Figure 7: Prefill latency (left) and decoding throughput (right) of three methods on RTX PRO 6000.
Comparison of FP16 and INT8 baselines. On RTX 5090, we further compare the prefill perfor- mance of MicroMix against two baselines: FP16 from HuggingFace and INT8 from Bitsandbytes. Figure 8 reports the prefill latency and peak memory usage of Llama2-7B and Llama3. 1-BB on the RTX 5090 with batch sizes {8. 12) and sequence length 2048. Compared with FP16, MicroMix duces memory usage by 2.29-2.84 x and latency by approximately 2.0x at batch size 8. Compared with INT8, MicroMix further reduces memory usage by 1.60-2.01 x and latency by 1.80-1.44x at batch size 12.
Llama2-76 78 Prefill latency
Llama2-78 Peakmem
Ulam3311 Prefill latency
Llama3.1.80 Poakmem
1800
1500
Resk (G8)
Latenie (me)
1000
2000
Peeak
O0'
D0M
ODM
bsc-11
12
152--8
Buch 12
tnd-12
Insc- I
Inc- Mecher-2048
2018
FPTM
INTA
NirreMix
Figure 8: Prefill latency and peak memory usage of MicroMix compared with FP16 and INT8.
CONCLUSION
In this paper, we present MicroMix, co-designed mixed-preedee precisieccnnnn quantization algorithm and ker- nel that supports MXFP4, MXFP6, and MXFP8 formats. Our algorithm introduces the quantization threshold to identify elements that incur excessive quantization eтror at the target bit width. We also propose an offline calibration strategy to determine the optimal channel assignments for each precision level on calibration dataset. To enable efficient inference, we design matrix multiplica- tion kernel that integrates three GEMM precisions and a fused reorder-and-quantiz operation. Mi- croMix kernel achieves significant speedups over TensorRT baselines on both RTX 5070Ti laptop and RTX 5090 GPUs across various configurations. On the RTX PRO 6000, MicroMix consistently outperforms FP16, INT8 and INT4 baselines.
ACKNOWLEDGMENN
The authors would like to thank Yafei Zhao for his valuable contributions to the technical imple- mentation.
REFERENCES
Tushar Krishna Akshat Ramachandran, Souvik Kundu. Microscopiq: Accelerating foundational models through outlier-aware microscaling quantization, 2025. URL https::/aaxxi
abs/2411 05282
Saleh Ashkboos, Ili Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Quik: Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.0923. 2023.
10

---

Published as a conference paper at ICLR 2026
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. In The eighth- Annual Conference fgs.
Neural Information Processing Systems, 2024. URI http://ppeeevviieeee / /ttp://wwweeeeeeeeeeeeeeeeeoe/e forum? i ddd
qsw38v1x
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https::////.m wttp:///axxvvvvv. www.ooggg/gao....ooooo.......aes 2108.1100 07732
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.
Yuxiang Chen, Haocheng Xi, Jun Zhu, and Jianfei Chen. Oscillation-reduccd mxfp4 training for Vision transformers, 2025,
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprint arXiv:/810.0430 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 1rX1v:1803.05457v. 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Bita Darvish Rouhani, Nitin Garegrat, Tom Savell, Ankit More, Kyung-Nam Han, Mathew Zhao, Ritchie amd Hall, Jasmine Klar, Eric Chung, Yuan Yu, Michael Schulte, Ralph Wittig, Ian Bratt, Nigel Stephens, Jelena Milanovie, John Brothers, Pradeep Dubey, Marius Cornea, Alexander Hei- necke, Andres Rodriguez, Martin Langhammer, Summer Deng. Maxim Naumov, Paulius Micike- vicius, Michael Siu, and Colin Verrilli. OCP Microscaling (MX) Specification. Open Compute Project, 2023a.
Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Ven- mugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gau- rav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahi, Andres Ro- driguez, Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Mi- cikevicius, Maxim Naumov, Colin Verrilli, Ralph Wittig, Doug Burger, and Eric Chung, Mi- croscaling data formats for deep learning, 2023b. URL htttttttt///xxiiii wwtpaawwwwwwwww/////a/a//a/////e 10537
DeepSeek-AI. Deepseek-v3 technical report, 2024. URL httppp::/aaaiivoorg/aassss 19437
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8):: 8-bit matrix multi- plication for transformers scale, 2022. URL htttpp::/aaxiiv.rrg///att 07339

---

Published as a conference paper at ICLR 2026
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gpty: Accurate post-training quantization for generative pre-trained transformers, 2023. URL btttps://axii....
0as/2220011322
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020. URL httpp::/aaxxii/ abs 00027
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h. Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf. Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024.
//zeodo wwweaanddooorreeeeeeeee recp//ecoaaaa /d2612088112222
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and et al. Akhil Mathur. The llama 3 herd of models, 2024. URL
https://arxs wrg/abs/////ooooo.or.o..///..rtr 2407 2207221783 21783
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021b.
Coleman Hooper, Charbel Sakr, Ben Keller, Rangharajan Venkatesan, Kurt Keutzer, Sophia Shao, and Brucek Khailany, Fgmp: Fine-grained mixed-precision weight and activation quantization for hardware-accelerated Ilm inference, 2025. URL htttthtss////r//// org. abtp//aa/2/5000 141
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization, 2024. URL https:
//axxiv.orr/abs//20000 07629
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Outlier-aware weight quantization for efficient fine-tuning and inference of large language models, 2024. URL
https::/aaxiis org abs////////////eeeooooa/s////ae 2306. 02272
Janghwan Lee, Jiwoong Park, Jinseok Kim, Yongjik Kim, Jungju Oh, Jinwook Oh, and Jungwook Choi. Amxfp4: Taming activation outliers with asymmetric microscaling floating-poim for 4-bit Ilm inference, 2025. URL /ttpt::/tt///ttt /attp:://arxi, arxiv. org/ abs/ 2411 09909
Haokun Lin, Haobo Xu, Yichen Wu, Jingzhi Cui, Yingtao Zhang, Linzhan Mou, Linqi Song, Zhenan Sun, and Ying Wei. Duquant: Distributing outliers via dual transformation makes stronger quan- tized Ilms. arXiv preprint arXiv:2406.0177 011221,,, 2024a.
Haokun Lin, Haobo Xu, Yichen Wu, Ziyu Guo, Renrui Zhang, Zhichao Lu, Ying Wei, Qingfu Zhang, and Zhenan Sun. Quantization meets dlims: A systematic study of post-training quanti- zation for diffusion llms. arXiv preprint arXiv:2508. 148899968...8866, 2025a.
Haokun Lin, Xinle Jia, Shaozhen Liu, Shujun Xia, Weitao Huang, Haobo Xu, Jumyang Li, Yicheng Xiao, Xingrun Xing, Ziyu Guo, et al. Efficient diffusion language models: A comprehensive survey. Authorea Preprints, 2026.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for Ilm compression and acceleration. In Sys, 2024b.
12

---

Published as a conference paper at ICLR 2026
Yujun Lin, Haotian Tang, Shang Yang. Zhekai Zhang, Guangxuan Xiao, Chuang Gan, and Song Han. Qserve: W448kv4 quantization and system co-design for efficient Ilm serving, 2025b. URL
https://arxiv: org/ abs 2405. 04532
Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krish- namoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: Llm quantiza- tion with learned rotations, 2024. httphhtts:/aaaae
org/ abs 2405. 16406
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark, 2021. URL https:
//axxiv.rrg/ab//210000 13009
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. CoRR, abs/1609.07843, 2016. URL. http:ttp:://xxx..or///////////
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. The LAMBADA dataset:
Word prediction requiring a broad discourse context. In Proceedings of the 54th An- nual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp- 1525-1534, Berlin, Germany, August 2016. Association for Computational Linguisties. URL
http: http://wwwww.o wwt. wwwwwa.wwwb..... org/ waatthooogy/PPi. P16-1144
Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
URL /ww.ivaivvvv...///av orgo abs/ / 22122412222 22..2.12 15115
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, htss/9910.1068. 2019. URL /ttp:i:////xv....//a www.a/iww....ooaaoooooaaaooeeeee httpg///b./.. htabb//1910) 1910.10683
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale, 2019. URL http:t/pe//arii..orr///aas//////
10641
Utkarsh Saxena, Sayeh Sharify, Kaushik Roy, and Xin Wang. Resq: Mixed-preccssion uuaniie quantization of large language models with low-rank residuals, 2025. URL /tt/:ttt/////xxxorr////b// 2412. 14363
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang. Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models, 2024. URL https:://axxii wwtp///////.......///r// 13137
Sayeh Sharify, Utkarsh Saxena, Zifei Xu, Wanzin Yazar, Ilya Soloveychik. and Xin Wang. Post training quamtization of large language models with microscaling formats, 2024a. URL https:
7/arxiv. wttpii/.orr/bb///4000000 07135
Sayeh Sharify, Utkarsh Saxena, Zifei Xu, Wanzin Yazar, Ilya Soloveychik, and Xin Wang. Post training quantization large language models with microscaling formats, 2024b. URL https:
7/axxiv. wwtp:/orgga///bbo 4005 07135
Yuxuan Sun, Ruikang Liu, Haoli Bai, Han Bao, Kang Zhao, Yuening Li, Jiaxin Hu, Xianzhi Yu, Lu Hou, Chun Yuan, Xin Jiang, Wulong Liu, and Jun Yao. Flatquant: Flatness matters for I1m quantization, 2025. URL https: patps:///arxixi arwiv..iivvv///ii///...aa...aavv org/ abs aw.aabs24440 24100.9422
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL pRtpps::://axxt
org/ prg/abs/ 1706. 03762 62
Tianwen Wei, Jian Luan, Wei Liu, Shuang Dong, and Bin Wang. Cmath: Can your language model pass chinese elementary school math test?, 2023.
13

---

Published as a conference paper at ICLR 2026
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient tostiing quantization for large language models, 2024. URL https:
/laxxiv.org/aaoo..e. abs / 2211. 10438
Lianwei Yang, Haokun Lin, Tianchen Zhao, Yichen Wu, Hongyu Zhu, Ruiqi Xie, Zhenan Sun, Yu Wang, and Qingyi Gu. Lıq-dit: Log-rotation post-training quantization of diffusion trans- formers for image and video generation. arXiv preprint arXiv:2508 2508::2500,,388,,, 2025.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers, 2022.
URL httph: /ttp://aaxvvv... org. abs 2206. 01861
Zihao Ye. Lequn Chen, Ruihang Lai, Yilong Zhao, Size Zheng, Junru Shao, Bohan Hou, Hongyi Jin, Yifei Zuo, Liangsheng Yin, Tianqi Chen, and Luis Ceze. Accelerating self-attentions for Ilm serving with flashinfer, February 2024. URL httpsss//llasinfee...ro introduce- f aashnnfer htmml
Zhihang Yuan, Lin Niu, Jiawei Liu, Wenyu Liu, Xinggang Wang, Yuzhang Shang, Guangyu Sun, Qiang Wu, Jiaxiang Wu, and Bingzhe Wu. Rptq: Reorder-based post-training quantization for large language models, 2023.
wbw..///////s/304440./ 0304000002000 01089
Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng. Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantiza- tion for efficient and accurate Ilm serving. In P. Gibbons, G. Pekhimenko, and C. De Sa (eds.), Proceedings of Machine Learning and Systems, volume 6. pp. 196- 209, 2024. ittp:tttss://prooeeeet ngs. mlsys wrtpllsssrrgppper_rrrrrerea fttpreerrrflleeeepppr//eepp Ftt1://ee5570000..00000000011113
A RELATED WORKS
Post-training Quantization can be broadly divided into two categories: weight-only methods and weight-activation methods Lin et al. (2025a). Weight-only approaches (Frantar et al. 2023 Kim et al., 2024] Lin et al. 2024b) compress model weights into low-bit formats while dequantizing them back to high precision (e.g.. FP16) during GEMM operations. Although this reduces memory bandwidth requirements, the computation itself still relies on high-precision operations, leaving significant bottleneck in inference efficiency (Lin al. 20256). Consequently, there remains sub- stantial room for accelerating LLM inference. Weight-
methods (Yao et al. 2022 Lee cct al. 2024 Lin et al. 2026), in contrast, quantize both weights and activations into low-bit formats, enabling GEMM to be executed entirely in low precision. This approach alleviates both band- width and computational bottlenecks but often suffers from severe accuracy degradation due to the presence of outlier activations. To address this challenge, mathematically equivalent transformation methods (Xiao et al.] 2024 Shao et al. 2024) adopt a channel-level smoothing strategy. By shifting activation outliers into the weights, these methods effectively reduce quantization error. Rotation- based -achh--activttion methods (Ashkboos et al.) 2024 Liu et al.) 2024: Lin et al., 2024a) have recently emerged, achieving notable success preserving model accuracy even at 4-bit precision.
Mixed-precision quantization retains outliers in higher bit-widths while quantizing the remaining
elements lower bit-widths (Dettmers al. 2022; Saxena et al., 2025 Ashkboos al., 2023 Hooper et al. 2025). The central challenge is designing efficient fused GEMM kernel. Atom (Zhao al.) 2024) achieves state-of- state-of-the-art performance by preserving 128 outlier channels INT8 and quantizing the rest to INT4. Although Atom demonstrated 7.73 x speedup over FP16 on the RTX 4090, its current kernel is limited to Llama2-7B and can only handle up to 128 high-precision chan- nels. Unlike previous approaches that use fixed number of high-precision channels for all linear layers, our method enables flcxible, fine-grained prexisson configurations, and is specifically designed to leverage the advantages of Microscaling data formats.
Applications of Microscaling data formats. Recent works (Darvish Rouhani et al. 2023b Sharify let al., 2024a b) begin to study the applications of MX in both training and inference. AMXFP4 (Lee et al.] 2025) handles outliers and asymmetries in activation by introducing asymmetric shared scales. Furthermore, Chen et al. (2025) significantly improved the FP4 training accuracy of Vision
14

---

Published as a conference paper at ICLR 2026
Transformers by identifying and solving the weight oscillation problem in forward propagation. MicroScopiQ (Akshat Ramachandran 2025) optimizes the quantization by combining pruning with outlier-awand miniaturization. Although these works have made significant progress in the inference and training of low-width MX formats, there still lack of systematic work on using microscaling data formats for general mixed-precision quantization.
B MICROSCALING DATA FORMATS (MX)
According to Darvish Rouhani et al. (2023a) we give some supplementary information of MX in this section. An MX-compliant format is consisted three components: scaling block size k; scalar elements (zx)/== and a shared scale in E8MO format (see Figure 9) 'he special scale format enables the Microscaling data format to achieve dequantization operations solely through shift operations, thereby enhancing the running speed. Here, {x,}e= is already quantized, so the original value is {sr; The specific parameters of MX data formats are shown in Table More details on MX please refer to OCP Microscaling Specification DDaviihh Rouhani et al. 2023a).
bits
shared scale
S
d bits
x1
X²
X3
X
scalar elements
Figure 9: A schematic diagram of the basic unit of Microscaling block. The block encodes the original & values sz; into & elements in MX and shared scale s.
Table Format names and parameters of concrete MX-compliant formats 2023a).
(Darvish Rouhani al.,
Format Name
Element Bits
Element Data
Exponent Bias (b)
Max Normal
Scaling Block Size
Scale Data Type
Scale Bits
Type
(k)
(w)
FP8 (ESM2)
+ 57344
MXFP8
32
E8MO
8
FP8 (E4M3)
7
448
IP6 (E3M2)
3
+ 28
MXFP6
32
E8M0
8
FP6 (E2M3)
1
+ 7.5
MXFP4
FP4 (E2M1)
+: 6
32
E8M0
8
MXINTS
INT8
N/A
163/64
32
E8M0
8
LLaMA3.1-88
Owen2.5-78
Ministral-88
Phi4-148
0.72
0.6
0.05
3.55
0.00
3.50
0.55
3.55
0.50
0.50
048
3.03
0.02
0.28
Qmin
Fmax
Stage
Stable Stage
Figure 10: After quantizing weights to INT4 using per-channel symmetric quantization, the zero- shot average accuracy of the models on Winograde, PIQA, BoolQ, ARC.C, and Lambada changes with qman-TTe quantization process of activations corresponding to different ymaz is implemented through fake-quant simulation. A lower value of amax corresponds higher upper bound on the quantization error.
15

---

Published as a conference paper at ICLR 2026
C QUANTIZATION ERROR ANALYSIS
C.I OBSERVATIONS
In this section, we discuss the quantization error in detail. We observe the variation relationship between the accuracy of the quantization model and the quantization error, which supplements the deficiency in the deseription of the continuity relationship between accuracy and error in previous works.
Given a FP16 tensor RRII VX, E X, the quantization error EXx)) between Q(X;) and X; is:
E(X,) = X - Q(x.) = X; round
7
(11)
where 7 |round(X,/s) X;/s is the rounding error. INT quantization is similar to FP quanti- zation:
Q(X,) round( X )
max
(12)
qmax
where round(.)) is rounding to the nearest INT value and ymaz 2n is the maximum value of INT range. For INT format, there is 7 E [0, 0.5], so we can get the quantization error upper bound
E(X,) of INT format:
E(X;) 7 S 0.5
max(1
max(
(13)
= 0.5 2- -1
F((.))
2
in particular, for INT8:
B(X:)INTS max(1
(14)
254
We reformulate Equation as following:
E(X,) max (X)
(15)
Imax
Then we control Imaz to observe the relationship between the quantized model accuracy and the quantization error upper bound, as shown in Figure TO, We have three observations:
(1) The curve in Figure I0 clearly illustrates how model accuracy varies with quantization error. In general, the accuracy the model decreases with the increase of the upper bound of the quantization enror.
(2) There is "Stable Stage" for each model maintaining high accuracy of variation Jmaz: INT8 (2ma==127) is located in this stage. For all four models, INT8 is high-precision format.
(3) When Jmar is below threshold, the accuracy of quantized model degrades significantly, which we name as "Decline Stage", and INT4 (fmns=7) is located at the end of this stage.
In conclusion, enhancing the accuracy of quantized model requires reducing its quantization error to bring it within the stable stage. The relationship between the quantization error upper bound and the model accuracy inspires us to divide values into three parts from the view of quantization error upper bound.
C.2 DERIVATIONS
In this section, we show the detailed derivation processes of quantization threshold, which is based on the motivation of controlling the quantization error of MXFP4/MXFP6 below E(X)INTS. The quantization error of MXFP4/MXFP6 is:
E ((XX))MXFPPMMMMPPG) 7 21lo (max((( ( -b
(16)
Since the gap between adjacent FP values is not constant, we use
Jmar
(17)
2
16

---

Published as a conference paper at ICLR 2026
to approximately express the rounding error in Equation 16 where Imar is the maximum value of
MXFP4/MXFP6. Substituting Equation into Equation 16
gives:
Omax
2 (max( ( xD))-
2n-1
Omax
2 (max [X
x
(18)
2n-1
Umar
max
2
26
Let (XXPPLMXX) & E(X.)INTS- Then we have
(16))
E(X)
ymar
2 S E(X,)INTS
(19)
2
If the inequality on the right-hand side holds, it follows that
2
max
(
(axx(G/(4,6))) s 26
(20)
Gman
254
According Table 8 when 4 6, the corresponding values of Jmax and b can be substituted directly into Equation 20 At last, we get the definition of quantization threshold:
T(n) 2 2n max X
(21)
Gman 254
D SUPPLEMENTARY MATERIALS OF EXPERIMENTS
D.I EXPERIMENTAL SETTINGS
In this section, we demonstrate some reproduction details, especially claiming how "Avg.Bin Aitg.Bits" in Table iis calculated.
QuaRo QuaRot uses symmetric INT4 quantization of group size 128. a-clip_ratio is 0.9, and w clip is used. For QuaRot, its online Hadamard transformation depends Hadamaaadaarddoo dadamaadd.Trannfforr... kemel without introducing extra matrices. So its average bits
is:
4 128
16 4.12
(22)
Atom The activation-sort metric is chosen "hessian" according to Atom's default settings. a c.cppp ratio 0.9, w clipppaatii ratio is 0.85 and keeper size is 128. The "Avg, of Atom calcu- lated as following:
(hidden_size 128) 44 128 8 hidden.
128
(23)
hidden_size
QUIK The value of fp features num is set to 256. following the settings used in QUIK. The part TNT4 is quantized using asymmetric per-token quantization. clip and int8 down proj is used. Since QUIK adopts pure INT8 for all Down Projs and mixed-precision for other linear layers,
its "Avg. Bits" is:
(hiddeenssize 256) 4 256 16 2 16) 6 (intermediaale_size 8 16) hidden_sin size + intermediate.sia
(24)
where 6 counts for Q K. O. Up and Gate Projs.
http://ghhhb..omm/pp/lqquuuutttt http://gttttlooomnbooollllllllla hhtps///giinubccmmcfeelaaatttto http:://gghhh..coo//////SSAAA/A
17

---

Published as a conference paper at ICLR 2026
AMXFP We use the fpi.emmlassym element format as specified in AMXFP4, scale_bits is and bizek_size is 32. scale_mode (default setting in run.sh) needs two FPI6 scales for each block, its "Avg. Bits" is:
32
2-
16 =
(25)
32
FlatQuan f We adopt the per-token and per-channel INT4 symmetric quantization for FlatQuant, with parameters such as Iwe, lac, cali trans, and add add.diog. Since FlatQuant introduces 7 addi- tional square transformation matrices per layer during the forward pass, and the elements these matrices differ across decoder layers, its "Avg. Bits" is computed as follows:
(hidden_size 6 intermediate_sizze 4 seqlen + numel( 16
(26)
(hidden size intermediate size) seqlen
where numel(P) P)) denotes the of the elements of the transformation matrices per layer:
64 64 4+ 32 + 112 112 + 128 128 = 46336, Llama 3.1-8B
numel P) = 64 64 2 80 80 2 40 40 144 144 192 192 80192, Qwen 2.5-32B
(27) Since the additional introduced matrix be reused for all tokens, when the seqlen longer, the average number of bits caused by transformation matrices is lower. When seglen > 190, the average number of additional bits introduced by P less than 0.01. But at the same time, the single-token situation in the decode stage has to be taken into consideration. In conclusion, we uniformly set seqlen 100.
D.2 INFORMATION ON POST-QUANTIZATION MODELS
Table 9 shows some information of quantized models. In general, MicroMix utilizes 5-5.6 bits for Llama and Qwen series models. The offline calibration and quantization time are relatively fast, which only takes 2min23s to get the quantized model of --...-5-a---BB----t
Table 9: Average bit-width per element and memory consumption quantized weights across all evaluated models. "Quantization Time" denotes the total offline time cost, including reordering and quantization the original model weights.
Models
Avg. Bits
Memory
Quantization Time
1lama3.1-88
5.51
5.09 GB
179s
Qwen2.5-32B
5.22
24.54
406s
Qwen2.5-Code -nn
5,54
9.10 GB
260s
Oween.5--ooerr332 Innsstrut
5.18
24.53 GB
406s
Owenn.55MMthh777 BB-Intruutt
5.16
4.79 GB
143s
In Figure 3 we tallied p4. Ps and ps of each layer of Llama3. We supplement the average number of bits for each layer in Figure 11
Proj
Down Proj
5.2
Serroge
533
Wecrratt Bit
5.20
4.7
4.55
515
15
20
20
20
25
20
75
Layer Indees
Layyrr S1ider
1AYPY Irddes
Figure 11: The average number of bits per layer of Llama3.1-80
18

---

Published as a conference paper at ICLR 2026
Figure 12 illustrates the precision mapping in MicroMix. The reorder-and-quanee operation is fused into LayerNorm. and the resulting quantized activations are reused by subsequent linear layers.
In addition, the KV cache is quantized with FlashInfer (Ye et al. 2024) to further reduce memory usage.
MicroMix Kemee
MXFP4
BFP16
LayerNomn
Fused Op
Reorder and Quantize
LayerNorm
Reorder and Quantize
Cache
G
Attention
Activation
Reorder and Quantize
Reorder and Quantize
Down
Figure 12: Precision mapping MicroMix for Transformer block in LLM.
D.3 SUPPLEMENTARY RESULTS
The results of KV cache quantization are reported in Table where all methods adopt INT4 asym- metric quantization with group size of 64. MicroMix retains 97.6% of the FP16 zero-shot accuracy under this setting. For five-shot accuracy and perplexity, MicroMix also achieves state-of- the-art performance.
Table 10: Zero-shot, few-shot accuracy and perplexity of Llama3. 1-8B, using Im-eval Gao et al. 2024.. All methods use asymmetric INT4 quantization of group size 64.
Avg. Bits
O-shot (t)
5-shot (f)
PPL (1)
Model
Method
ARC.C
BoolQ
Lambada
PIQA
Winogrande
Avg.
MMLU
WikiText2
53.58
6.24
QuaRot
44.88
66.74
66.61
65.90
8.03
Llama3.1-88
QUIK
5.95
49.66
77.77
71.08
78.51
67.88
68.98
57.63
7.32
Atom
4.25
47.95
79.94
72.81
35
70.72
69.95
57.10
7.43
FlatQuant
4.19
50.17
79.33
72.15
79.27
71.59
70.50
59.34
7.12
AMXFP4
5.00
46.84
73.24
69.59
77.15
67.56
66.87
53.11
7,33
MicroMix
5.51
52.22
80.64
74.00
79.38
70.96
71.44
60.90
6.97
Table II shows the results of Atom and QUIK directly applied to MXFP4 and MXFP8, with significant performance drop compared MicroMix. Since the kernels of Atom and QUIK do support the MXFP format, we use the MicroMix kernel to keep the number of MXFP8 channels at 128 and 256 respectively.
Table 11: Zero-shot accuracy (f) and WikiTexi2 perplexity (L) results of mixed-precision methods on MXFP formats using Llama3.1-8B.
Methods
ARC-C
BoolQ
Lambada
PIQA
WikiText2
FP16
51.28
82.05
75.80
80.03
6.24
Atom
43.60
76.36
66.52
75.57
8.02
QUIK
47.27
76.15
68.52
76.72
7.86
MicroMix
50.17
81.13
74.13
80.14
6.72
19

---

Published as a conference paper at ICLR 2026
Table 12: shot accuracy (f) ARC. C, BoolQ. Lambada, PIQA, and perplexity (L) Wiki- Text2 of MicroMix on different datasets using Qwen2.5-14B.
Calib Data
ARC.C
BoolQ
Lambada
PIQA
WikiText2
WikiTex12
57.59
86.18
74.29
81.12
5.87
Pile
57.68
85.41
74.13
80.14
5.92
C4
58.36
86.57
73.65
81.56
5.92
Table 13: Using Qwen2.5-32B, we report the accuracy and performance of MicroMix on various
average bits.
Avg Bits
Arc.C
BoolQ
Lambada
PIQA
Winogrande
Avg.
Execution Time
4.48
56,91
85.87
76.54
81.28
73.88
74.89
5min 37s
4.84
55.46
86.51
77.18
81.50
74.19
74.96
5min 42s
5,09
56.14
85,66
77.14
81.39
74.98
75.06
5min 44s
5.22 (ours)
56.66
87.13
77.37
80.65
74.19
75.20
5min 46s
5.30
55.38
86.09
77.24
81.66
75.06
75.08
5min 48s
5.74
55.63
86.02
76.65
81.77
74.19
74.85
5min 49s
6.01
55.78
86.27
76.67
81.72
74.66
75.02
5min 50s
We further test the stability and flexibility of MicroMix to adapt different average bit-widths (4,
and 6), shown in Table 13
20


Reviews:
[{"review_id": "h85hTUI2iW", "reviewer": "Reviewer_x7qv", "summary": "This paper proposes a mixed-precision post-training quantization framework using MX formats with 4, 6, and 8-bit channels. The method adaptively assigns precision to channels based on quantization thresholds and fuses channel reordering into the quantization kernel. The approach is designed to leverage NVIDIA Blackwell FP4 tensor cores. Experiments on Llama and Qwen models show better accuracy with approximately five-bit effective precision and latency improvements.", "questions": "Rather than only showing results around five bit width, could you please analyze results at different target bit widths such as approximately four, five, or six bits to demonstrate the flexibility of the method and its ability to adapt to different accuracy and latency budgets?\n\nCould you please analyze more model architecture like different attention modules and MoE etc to show its generality?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors provide an end-to-end implementation and report real latency improvements on Blackwell GPUs, which increases the paper's practical value. \n\nThe use of flexible bit-width allocation per layer and integration with FP4 tensor core support matches new hardware capabilities. It is timely. \n\nThe empirical results show high accuracy retention across zero-shot, math, and code benchmarks. The fused reorder-and-quantize kernel design is efficient and demonstrates careful engineering.", "weaknesses": "Despite the solid engineering contribution, the technical novelty appears limited. The paper extends prior mixed-precision ideas, such as Atom, by allowing flexible channel ratios. The threshold-based assignment and layer-wise granularity resemble existing mixed precision heuristics and tuning methods, so the algorithmic insight is incremental.  Fusing permutation into quantization kernel is primarily an engineering effort. \n\nThe end-to-end speedup analysis uses FP16 and FP8 baselines, but not the low-bit kernels like 4bit. A comparison against an equivalent or similar bit-width kernel would clarify relative benefits. Separating prefilling and decoding latency would also give better insights. The evaluation could extend to more settings of different batch sizes, context lengths and decoding lengths. Right now only a few settings are covered. The evaluated models are only Llama and Qwen language models.   \n\nFigure 6 compares kernel speedups against FP8, FP16, and W4A16, but it remains unclear how much of the speed improvement arises from the authors’ design versus underlying CUTLASS kernels. A breakdown of kernel time contributions would clarify the engineering contribution.\n\nFor the accuracy comparison, the proposed method MicroMix is only set to ~5 bit. However, there are baselines like QuaRot use ~4bit.  It would be fair to also show ~4bit result for MicroMix, as well as for the latency comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NblOGUqiui", "reviewer": "Reviewer_MRQT", "summary": "This paper presents MicroMix, a mixed MXFP8/FP6/FP4 quantization method that mainly targets NVIDIA's Blackwell generation GPU and benefits from the new FP4 tensor cores. A quantization algorithm is proposed to allocate different bitwidths to different channels, and a fused quantization kernel is implemented to support the dynamic reordering and mixed precision quantization. The performance is evaluated on consumer-grade GPUs (RTX5070Ti laptop & RTX5090), and it demonstrates a 2.29-3.38x speedup compared to the FP16 baseline.", "questions": "- What is the hardware utilization of the mixed precision kernel? I would recommend using the ratio of FP4/FP6/FP8 and the peak performance numbers to calculate the theoretical latency and compare it with the measured latency. (Be aware that on GeForce GPUs, the bitwidth of accumulation (FP32 vs FP16) will affect the peak performance.) I'm concerned about this because MicroMix uses 3 cutlass kernels to perform a single GEMM, and this might lead to some overhead. \n\n- In end-to-end throughput results, why does INT8 quantization provide little performance improvement? Also, what are the performances of the prefilling stage (TTFT) and decoding stage (token/s)?\n\n- I would recommend adding some accuracy+performance results in the same table. How do quantization methods in Table 1 perform?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "It is good to see a work that benefits from the new technologies on a cutting-edge GPU and demonstrates end-to-end gains and memory savings.", "weaknesses": "- The mixed precision schema partitions the weight+activation into MXFP8+MXFP8, MXFP6+MXFP6, and MXFP4+MXFP4. Among them, MXFP6+MXFP6 does not make sense to me. I believe FP6 has the same peak performance on Blackwell GPUs (correct me if I'm wrong), and quantizing **activations** into FP6 does not reduce the model size, nor increase the inference speed. W6A8 might be a better choice here, and cutlass has an example for W6A8 kernel (https://github.com/NVIDIA/cutlass/blob/main/examples/79_blackwell_geforce_gemm/79c_blackwell_geforce_mixed_mxfp8_mxfp6_bf16_gemm.cu)\n\n- The real efficiency of the implemented kernel is not very clear to me. Although there is a comparison against TRT-FP16, W4A16, and FP8 in section 4.4, the actual utilization (achieved vs peak performance) is not provided.\n\n- Though this paper claims they conduct experiments on server-grade GPUs (RTX5090), the RTX5090 still has the same architecture as consumer GPUs (sm_120) instead of real compute cards like B200 with sm_100 architecture. It is totally understandable that the authors may not have access to these latest-generation GPUs, but I would not recommend claiming the work supports server-grade GPUs. Also, some explanations in the paper (for example, tcgen05.mma instructions) only apply to these compute cards and do not exist on RTX5090.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vcc1jOIvr9", "reviewer": "Reviewer_6987", "summary": "This paper proposes MicroMix, a mixed-precision quantization framework designed for Microscaling (MX) floating-point formats (MXFP4, MXFP6, MXFP8), targeting NVIDIA’s Blackwell FP4 Tensor Cores.\nIt introduces:\n\nA quantization threshold–based precision assignment, determining which channels use 4-, 6-, or 8-bit MX formats.\n\nAn offline calibration to precompute per-layer precision ratios.\n\nA fused reorder-and-quantize kernel optimized for MX formats.\n\nMicroMix reportedly achieves near-FP16 accuracy at ~5 bits average precision and up to 3.3× kernel-level speedup on RTX 5090, outperforming FP16 and INT8 baselines.", "questions": "The paper is competently executed and well-written, with solid experiments and system results.\nHowever, the conceptual novelty is limited—it mostly integrates existing techniques (mixed precision + threshold + MX formats + kernel fusion) rather than proposing a fundamentally new approach. Could you clearly articulate what aspect of MicroMix is genuinely novel compared to prior mixed-precision or microscaling quantization methods?\n\nHow sensitive is the threshold-based precision assignment to the choice of calibration data and ratio parameters?\n\nCould the proposed mixed-precision calibration generalize to other quantization formats (e.g., MXINT4/8,NVFP4)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper targets a relevant and timely topic, leveraging upcoming FP4/6/8 Tensor Cores for efficient LLM inference.\n\nThe system–algorithm co-design is well-executed, integrating quantization thresholds and fused CUDA kernels.", "weaknesses": "The main algorithmic ideas—mixed-precision assignment, threshold-based outlier handling, and fused reorder kernels—are incremental extensions of prior works. The combination feels straightforward rather than conceptually new.\n\nThe method mostly refines known strategies (precision partitioning, block scaling) for a specific hardware (Blackwell FP4), rather than introducing a new quantization principle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LaVWGNr17g", "reviewer": "Reviewer_BbmQ", "summary": "This paper proposes MicroMix, a novel mixed-precision quantization approach that leverages the MXFP4, MXFP6, and MXFP8 formats supported by recent NVIDIA architectures. The method performs channel-wise mixed-precision quantization for both weights and activations, assigning different bit-widths based on value distribution thresholds. In addition to the quantization algorithm, the paper presents a system-level co-design, including fused kernels for quantization, permutation, and GEMM operations to minimize latency and memory overhead. Extensive experiments on large language models (Llama3.1-8B and so on) demonstrate that MicroMix achieves state-of-the-art performance without retraining. The work highlights both algorithmic insights and system-level engineering optimizations for efficient mixed-precision inference on modern GPUs.", "questions": "I doubt the permutation and mixed-precision channels multiplication may cause extra latency. \n\nIn Figure 7, what exactly does “end-to-end throughput” measure?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The method adopts a channel-wise mixed-precision strategy, which is more flexible and effective than conventional layer-wise approaches. \n\nThe insight for permutation and assign different bit-widths through threshold is conceptually clean yet empirically powerful.\n\nThe work tightly couples algorithmic design with low-level kernel fusion\n\nExtensive experiments have been conducted to prove the effectiveness of MicroMix for both accuracy and system efficiency.", "weaknesses": "The work is more engineering-oriented than fundamentally novel from a research standpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)