Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization
Authors: Vage Egiazarian
Roberto L. Castro
Denis Kuznedelev
Andrei Panferov
Eldar Kurtic
Shubhra Pandit
Alexandre Noll Marques
Mark Kurtz
Saleh Ashkboos
Torsten Hoefler
Dan Alistarh
Venue: ICLR 2026 Poster (2026)
Keywords: efficiency
quantization
large language models
Primary area: infrastructure, software libraries, hardware, systems, etc.
TL;DR: We provide state-of-the-art methods for the new FP4 quantization formats, specifically NVFP4 and MXFP4.

Abstract: The recent hardware-accelerated microscaling 4-bit floating-point formats such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to revolutionize large language model (LLM) inference. Yet, their practical benefits remain unproven. We present the first comprehensive study of  MXFP4 and NVFP4 for post-training quantization, revealing  gaps between their promise and real-world performance. Our analysis shows that state-of-the-art methods struggle with FP4, due to two key issues: 
(1) NVFP4's small group size \emph{provably} neutralizes traditional outlier mitigation techniques; 
(2) MXFP4's power-of-two scale quantization severely degrades accuracy due to high induced error. 
To bridge this gap, we introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm that tailors the quantization process to FP4's unique properties, by using block-wise Hadamard transforms and format-specific optimizations. We support our proposal with a set of high-performance GPU kernels that enable the MR-GPTQ format with negligible overhead, by rotation fusion into the weights, and fast online computation of the activations. This leads to speedups vs. FP16 of up to 3.6x layer-wise, and 2.2x end-to-end on NVIDIA B200, and of 6x layer-wise and 4x end-to-end on RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches or outperforms state-of-the-art accuracy, significantly boosting MXFP4, to the point where it nears that of NVFP4. We conclude that, while FP4 is not an automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock a new frontier of accuracy-performance trade-offs.

Published as a conference paper at ICLR 2026
BRIDGING THE GAP BETWEEN PROMISE AND PERFORMANCE FOR MICROSCALING FP4 QUANTIZATION
Vage Egiazarian" & Andrei Panferov* Institute of Science and Technology Austria
Denis Kuznedelev Yandex Research
Shubhra Pandit, Alexandre Marques Mark Kurtz Red Hat Al
Saleh Ashkboos & Torsten Hoefler
ETH Zürichh
Roberto L. Castro", Eldar Kurtié & Dan Alistarh Institute of Science and Technology Austria & Red Hat AI
ABSTRACT
The recent hardwarr-accellerated microscaling 4-bit floating-point formats such as MXFP4 and NVFP4, supported on NVIDIA and AMD GPUs, promise to revolu- tionize large language model (LLM) inference. Yet, their practical benefits remain unproven. We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization, revealing gaps between their promise and real-world performance. Our analysis shows that state-of-the-ar methods struggle with FP4, due to two key issues: (1) NVFP4's small group size provably neutralizes tradi- tional outlier mitigation techniques; (2) MXFP4's power-of-two scale quantization severely degrades accuracy due high induced error. To bridge this gap, we introduce Hiicr-RRotated-PPee
(MR-GPTQ), variant of the classic GPTQ quan- tization algorithm that tailors the quantization process to FP4's unique properties, by using block-wise wise Hadamard transforms and formaat-peccffc optimizations. We support our proposal with a set of gegh-performanc GPU kernels that enable the MR-GPTQ format with negligible overhead, by rotation fusion into the weights, and fast online computation of the activations. This leads to speedups up to 3.6x layer-wise, and 2.2x end-to-end on NVIDIA B200, and of 6x layer- and 4x end-to-end on RTX5090. Our extensive empirical evaluation demonstrates that MR-GPTQ matches or outperforms state-of state-of-the -hee--o--th-ttte accuracy, significantly boosting MXFP4, to the point where it nears that of NVFP4. We conclude that, while FP4 not an automatic upgrade over INT4, format-specialized methods like MR-GPTQ can unlock a new frontier of accuracy performanee trade-offs.
FP16 of
INTRODUCTION
Post-training quantization (PTQ) [41; 21; 36] is one of the most well-researched areas in model compression, in which the objective to take an existing pre-trained model and reduce its size or computation while preserving most of its accuracy. With the advent of large language models (LLMs), PTQ has become highly-active research area, e.g., [21; 59; 3; 15; 50; 19; 51] with significant industry adoption and practical impact [29].
"Equal contributions.
1 Corresponnence dan al starhe wwwrreitttc..... ac.

---

Published as a conference paper at ICLR 2026
In this paper, we focus on quantization using the drently-introuucee microscaling floating-poit precision formats, specifically MXFP4 [47] and NVFP4 [43]. In a nutshell, these formats work by grouping elements into blocks of 32 or 16 elements, respectively, quantized together with shared scale; to reduce the storage overhead, the scales themselves are also compressed, to distinct 8-bit format: a standard sharing between Exponent and Mantissa bits (F4M3) for NVFP4, and 1SM0-essentian
rounding scales to powers-of-two- for MXFP4. As such, the NVFP4 format trades off additional space (4.5 bits per element on average, relative to 4.25 bits for MXFP4), in favor of additional precision. The promise of these formats is two-fold: first, they are claimed to be more accurate than the prior-generation integer precision formats such as INT4 [40]. Second, they are supported in hardware: NVIDIA Blackwell GPUs support matrix multiplications across both NVFP and MXFP formats, whereas AMD GPUs will support MXFP4 [1]. Despite these developments, little known about the accuracy of these formats on real models or their practical performance.
Contributions. In this paper, we provide a first thorough study of the accuracy and performance limitations of the NVFP4 and MXFP4 formats through the prism of current state-of-the-ard quanti- zation methods, coupled with computational support. We focus primarily on quantization 4-bits per parameter, and investigate the interaction between these new formats, real parameter distributions, and state-of-the-ar quantization algorithms. Our main findings are:
We begin with an analysis of quantization error induced by the NVFP4 and MXFP4 formats over both Laplace-like like heavy-tailed distributions, which arise in real- world weights and activations [2; 17], and over Normal parameter distributions, arising when processing weights and activations via rotations in popular methods such as QulP/QulP## [6; 50] or QuaRot [4]. Interestingly, we can prove analytically and show empirically that rotations improve MXFP4 accuracy, but hurt NVFP4 accuracy when coupled with standard Round-to-Nearte (RTN) quantization.
Based on this analysis, we propose a new variant of the GPTQ weight quantization algorithm [21], called Miicr-RotateddGeer
(MR-GPTQ), explicitly designed to maximize accuracy across both MXFP4 and NVFP4. The algorithm employs Hadamard rotations at the group level to "normalize" weights and activations, but in novel block-wise fused form, which, we show, can be supported without any runtime overheads on Blackwell GPUs. In addition, MR-GPTQ introduces new efficient variant of the activation re-ordering heuristic for GPTQ, along with format-specific scale search optimizations.
We perform the first extensive study of NVFP4 and MXFP4 practical accuracy, across standard Llama-3 [18] and Qwen-3 [55] models of different sizes, evaluated on standard zero-shot tasks [23]. We investigate broad set of compression methods, including RTN, GPTQ [21], SmoothQuant [54], QuaRot [4], and SpinQuant [38], as well as our new MR-GPTQ approach. Results show that: (1) both NVFP4 and MXFP4 are lossy, with MXFP4 inducing major accuracy drops (~ 10% relative), and (2) that existing techniques are not well-suited for these new formats, as they do not always outperform RTN. On the positive side, we show that GPTQ and the MR-GPTQ variant yield consistently good recovery for NVFP4. Moreover, MR-GPTT works particularly well in conjunction with MXFP4, recovering accuracy within 1-2% of NVFP4. For large models, we show that both formats can recover up to 98-99% of the baseline FP16 accuracy.
Our main technical contribution is new of GPU kernels specific the Blackwell architecture called QuTLASS, showing that the "micro-rotation" component of MR-GPTQ can be supported without loss of performance relative to standard multiplications. Specifically, this comes in the form of a lightweight fused kernel for online rotation of the activations. Remarkably, our kernel for MXFP4 can obtain higher throughput than ideal NVFP4 matrix multiplication. Our kernels obtain near-ideal layer-wise speedups for both B200 and RTX5090 GPUs, of 3.6x and 6x, respectively, leading to end-to-end inference speedups of 2x and 4x respectively.
BACKGROUND ON MICROSCALING FLOATING- FORMATS
General Definition. The microscaling MXFP4 and NVFP4 formats employ hierarchical quantization, where elements within block share common scale factor, enabling efficient hardware imple- mentation. Given a tensor divided into -ieddnmnnioonll groups, we define a Microscaling Block Floating-Poin PPint (MFP) representation as follows. The Group Size (G) is the number of elements in each group before quantization. The Element Representation (E) is the format used to represent the individual elements within each block. The Scale Representation (S): The format used to represent the scale values for each group.
2

---

Published as a conference paper at ICLR 2026
MXFP4
NVFP4
Quantized
Per Block Scaling
Quantized
Block Scaling
Per Scaling
32 volue Boook with FR4 values
EBMO Scaling Foctor
valuc Blook with FP4 values
E4M33 Scaling Focoor
FF22 (EBME3) Scaling Footor
EdMO Soatig Facker
E4M03 Staling Facur
Sign hit
X
Expoower bi
Marrises 3ix
Figure 1: Schematic illustration of the MXFP4 (left) and NVFP4 (right) microscaling formats.
For floating-point (FP) formats, we use the notation ExMy to say that bits allocated to the exponent, and y bits are allocated to the mantissa. For instance, in the standard FP4 E2M1 representation, each FP4 element consists of 1 sign bit, exponent bits, and 1 mantissa bit, providing 7 distinct positive values {0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0} plus zero and the negatives.
The MXFP4 (Microscaling FP4) Format. This format [47] follows the specification (G = 32, E FP4, S = E8M0). Its distinguishing features are the group size of 32 and its quantization of group scales to powers-of-twar given the use of E8MO, which dedicates all bits the exponent and none to the mantissa. This design choice simplifies hardware multiplication; yet, as our experiments reveal, often introduces quantization artifacts that can significantly impact model accuracy.
The NVFP4 (NVIDIA FP4) Format was introduced by NVIDIA for the Blackwell architecture [43], and employs a more flexible approach with (G = 16, E FP4, E4M3). While sharing the FP4 element format with MXFP4, NVFP4 differs in two key aspects. First, it uses a 16-element group size, and, second, uses full FP8 representation for scales E4M3, preserving more precise scaling information relative to E8M0. NVFP4 trades off more accurate representation for weight and activation distributions, the cost of increased pets-eer (4.5 NVFP4 vs 4.25 for MXFP4).
Related Work. Early work on LLM quantization focused primarily on integer formats, with INT8 being the first to be investigated [13; 56], conjunction with round-to- over groups of consecutive weights and activations. FP formats introduce new possibilities but also challenges: while FP8 quantization known to be near-lossless [29]. the distribution of representable values in NVFP4/MXFP4 changes quantization dynamics. The GPTQ method [21] reached near- lossless 1NT4 compression via second-order weight adjustments. Its effectiveness for FP4 formats remains unexplored. Methods like AWQ [35], SqueezeLLM [28]. and SpQR [15] relied on outlier- aware quantization strategies that assume uniform grids and large group sizes. The FP4 formats' small group sizes (16 or 32) and non-uniform grid inherently perform outlier mitigation, as we discuss in our analysis. Recent extreme compression techniques like QulP [6]. QulP# [50] and QTIP [51] use rotation matrices to normalize the weight distributions. As we will see, this is not necessarily helpful for FP4 microscaling formats.
(RTN) assignment
LLM activations are known to be extremely challenging to quantize, due to outlier features, defined roughly elements up to 100x larger than average [13]. SmoothQuant [54] addresses this for INT8 by resealing to redistribute outliers between weights and activations. Recent rotation-basee bbased methods like QuaRot [4] and SpinQuant [38] mitigate outliers through Hadamard transforms. In this paper, we discover novel trade-offs for these approaches.
Prior work investigating accuracy trade-off's under quantization, e.g., Yao et al. [56]; Liu et al. [37]; Huang et al. [27]; Gong al. [25]; Li al. [34]; Gong al. [24]; Lee al. [32]; Kurtic et al. [29] focuses almost exclusively on INT quantization. Despite industry claims about FP4's accuracy superiority (40; 43), rigorous evaluation remains absent so far, likely due the recent introduction of this format. Our work addresses this gap.
3 A QUANTIZATION ERROR ANALYSIS OF NVFP4 AND MXFP4
Prior work quantization [41; 13; 16) identified the average and top-element (outlier) mean-square error (MSE) as key quantities that can predict quantized model accuracy. In this section, we perform model-based analysis of the NVFP4 and MXFP4 formats from the prism of these metrics.
3

---

Published as a conference paper at ICLR 2026
Luplace Nutvve acivationn Kunosiis8.75
Gousian Rotatal activations Kartosis-..02
Laplace Native weight
Gausiun fit
Rotaee weights
Kurteis-1.47
Kurtoii-0.05
Density
Distribution fits for aggregate weights and activations of Llama-3. 3.1-88-Insttruc,, with and without rotations. The Normal distribution is clearly good fit for rotated weights and activations, while the Laplace distribution provides a good fit for the native distributions. Although native weights appear Normal, they have much heavier tails, as evidenced by the Kurtosis value.
Figure 2:
Modeling Distributions. Early work modeling L.M parameters assumed Normal (Gaussian) distribution [14), consistent with common initialization schemes. Yet, more recent studies have identified that distributions with high kurtosis, such as the Laplace Student-t distributions, better model the sharp peaks and outlier-prone tails weights and activations [2; 17].
Here, we follow the latter line of work and model weights and activations as following Laplace distribution. At the same time, interestingly, it can be proven that, after the Hadamard rotation, these tensors tend to follow normal distribution [6; 50]. We empirically validate these findings via fits over common models, illustrated in Figure 2. Formally, our modeling is as follows:
Definition 1 (Modeling). We assume that the "native" weights and activations follow the Laplace distribution W ~ Laplace(O, b) with density Jw Sw(w) 25 - and variance Var(W) = 212. We fix unit variance throughout, b 1/v2. The magnitude Z |W] is Exp(A) with rate x 1/b V2, that is fz(2) de A= and F2(2)
e A for 2 2 0.
We assume that weights and activations rotated via the Hadamard transform follow a Normal distribution V (O0 1). The magnitude Z = [V] is half-normal with fz(22 Fz(2) = erf(=//2) 2 O, where where erf(z) is the standard Gauss error function /F J et dt.
V 2 12 and
2
Quantization. We model Microscaling Block Floating-Poine (MFP) quantization as follows. Consider ii.d. blocks containing G > 2 elements drawn from some distribution: X = (X)...., ..) with Var(X,) = 1 and Z, = X,|- We assume a grid @ C [0, 1] that is finite, symmetric around 0 and includes both 0 and 1; we write Q Q n [0, 1] and gmin min( @ \ {0}). We use round-to- nearest (RTN) quantization, assuming probability for rounding ties. Next, we formally define the scaling process. For simplicity, we will not not quantize the scale s itself, and assume that values are normalized to [-] We remove these assumptions in our numerical validation (Section 3.2).
Definition 2 (Scales). For a block of elements X, we define the unquantized scale 8 = maxigisa [Xx), the normalized entries U; X:/s E [-1,1), the quantized normalized entries U, = R and the de de-normalized quantized values X, := Ü.. x
Definition 3 (Quantization Metrics). For group size G. we define: (i) The per-element MSE:
MSE(G) [[((X, X1) (by symmetry, the choice of index can be arbitrary). (ii) The top- element MSE per block: Let I, = arg max giça [Xa], ignoring ties. Define MSE =
(X,, - - We always use the same MFP map, i.e. same scale s, for both metrics.
Remark (Quantization Dead-zone). The first positive quantization level in the grid Q, which denote by quin induces the dead-zome zone half-width 9min./ on [0, 1). If UU} < then Ú,
3.1 ANALYTICAL MSE BOUNDS
Next, we derive bounds on quantization error across top and average elements. First, notice that, in a simplified setting, applying the Hadamard rotation spreads the MSE evenly among elements.
Lemma (Top-Element MSE). Assume a vector x E IRG with coondinates i.i.d. N (( 1), to which we apply a Hadamard rotation, perform MFP quantization in the d-doain produce D. and reconstruct
4

---

Published as a conference paper at ICLR 2026
Laplace
Waights
Activations
DAMI EAMS
MSE
Weighs Activation
Density
Llama 3B
MSD
opec
Llama
S- B43
15
125
16
32
64
128
16
10"
10'
Group size G
Group size
Group size
X,,
Figure 3: The effect of Hadamard Transform (HT) on MXFP4 (ESM0) and NVFP4 (E4M3) quantization on Laplace distribution samples and Llama-3.1-88 1-8n--33--BB-Innttuuc weights and activations for various group sizes.
Figure 4: Ranges of FP8 scale format and observed weight and activation mag- nitudes.
i == H Define the quantization error vectors Ey y and E. = --
JG HHE Ey. The expected squared error on the original top coordinate I, arg max, (xxg] is the per-element MSE:
MSEtop (G) IE((=)),,1 1 Ellev122 MSE(G).
Remark 2 (Outlier preservation). By contrast, it is immediate that MSEsap (G) 0 in the absence of the Hadamard rotation, since we are doing absmax scaling, which preserves the top element.
Asymptotic MSE Analysis. Thus, MSE is the key quantity for any fixed grid with dead zone d > O, for both Laplace and Normal models, limg MSE(G)) = Var(X;) 1. Intuitively, this is because,
analyze. First, notice that,
G grows, the block maximum M diverges, [U] [Xi]/M + 0 in probability; the mass that survives the dead zone vanishes. Consequently, the dominant part of the MSE E[(X; X1 becomes [[ < 811 + B[X =.
To get more granular variant, assume the large G domain and examine the "preserved mass":
R(G) = MSE(G) B[x ( 2 23},
which captures the mass that escapes underflow. A precise calculation yields the following:
Lemma 2 (Rates). Let 8 qmin (0, (1) be the dead-zone halfwidth in the normalized domain.
For Laplace, we have: (G. = @(log G , and for Normal: Rx =0(Vog
Discussion. Since < 8 < 8 we have that, for small G, the Laplace MSE should be below the MSE for the Normal distribution. Yet, for sufficiently large G. the Normal rate dominates the Laplace rate, meaning that (G) < MSEE As such, we predict crossover phenomenon, where the MSE gap in favor of the (native) Laplace distribution will be inverted for larger group size G in favor of the transformed Normal distribution. In short, transforms should hurt the original weights small group sizes, and become effective as we increase it.
3.2 NUMERICAL VALIDATION
practice, the weight and activation distributions are not of unit variance. Shared scales give us control over the variance during the quantization process, but the aggregation of the proposed quadratic errors will be dominated by groups with higher variance. To address this, when analyzing real weights and activations, we use the relative version of the errors proposed above.
Relative Errors.
Definition 4 (Relative Metrics), Let I, arg maxigisa [Xa] be the top group element. We define
(X 8, G
the relative per-element MSE as MSE (G)
X?), and the top-element
MSE per block: MSE (G) = E[((,, \
MSE is key metric in compression theory [49]; in the context of LLM compression, Malinovskii al. [39] present a linear dependence between MSErel and end-to-end accuracy decline. Ad- ditionally, recent lattice-based PTQ methods explicitly optimize for MSFrel when designing their
5

---

Published as a conference paper at ICLR 2026
lattice [50; 51; 39]. For MSE Lemma 3 shows how it accurately reflects the outliers' relative error as long as outliers are large, rare, randomly positioned, and MS is consistent for outliers and non-outliers (as shown by the shared scale quantization analysis below).
Figure validates the analysis from Section 3.1 on samples from Laplace distribution, as well as on model, For MSF (top row) and NVFP4 (G= 16), the Hadamard Transform has negative effect for small G and positive effect for larger G. exactly as predicted. To interpret the other effects, we have to better understand the effect of the shared scales quantization.
real weight and activation matrices from the 1aam--31--8B-nsmmee
Shared Scales Quantization. Under fixed bit-width, microscaling floating point formats with a shared scale (stored, e.g., in E8M0 or E4M3) trade range for accuracy. We begin our analysis by examining the range required to fully cover weights and activations.
Figure 4 shows the logarithmic dynamic ranges of several FP8 formats and compares them with the empirical distributions of shared scales for weights and activations across multiple models. One can see that the dynamic range of S = E4M3 covers the full range of these distributions. Trivially, = ESMO, having more range, can easily cover it too. When shared scales range is less than the dynamic range of S, they can always be represented by normal floating-poii values with their relative error (a) bounded by 2- for mantissa precision M and (b) translation-invarian
power-of-two shifts. For ab absmax quantization without rotations, this leads MSE being insensitive the shared scale magnitude in expectation over high dynamic range intervals, and, as the results, to G. We formalize this in Lemma 4.
This allows us to explain the effects of shared scale quantization on MSE by relating it to the precision of the shared scales data type S and the base data type E. We observe the following:
(1) For MXFP4, top values inherit their precision from the base data type, and not the shared scale data type. This is because S = E8M0 is coarser than E = E2M1, leading to shared scales inheriting effectively constant relative error from E2M1 regardless of G. as visible in Figure (2) By contrast, for NVFP4, shared scales inherit effectively constant relative error, regardless of G. This is because S E4M3 finer than Hadamard Transform is applied, the maximum element error is spread across the whole group. This follows Lemma 1. From Figure one can see that this leads to better precision than pure E2M1, but worse than pure F4M3. Moreover, one can see that for heavy-tailed distribution, such as Laplace or the observed model tensors, Xi, grows faster than MSF(G) with G, leading to the error being reduced as we increase the group size G. Yet, this effect alone is not enough for it to improve over the E4M3 precision for reasonable group size G.
E2MI, as visible in Figure 3. (3) Once the
Discussion. Our analysis so far showed that the MXFP4 format induces higher MSE for RTN quantization relative to NVFP4, and is worse at outlier preservation. At the same time, the format has lower memory and computational costs relative to NVFP4, and is likely to benefit from normalization via the Hadamard transform. By contrast, the NVFP4 format has lower MSE due the smaller group size, and top value preservation as it is "promoted" to E4M3. In addition, the NVFP4 MSE may not benefit from normalizing transforms. In the following, we incorporate our analysis into the classic GPTQ algorithm, obtaining a variant that is designed for FP4 formats, called MR-GPTQ.
4 MR-GPTQ: AN FP4-FOCUSED VARIANT OF THE GPTQ ALGORITHM
Standard GPTQ. Given a layer's weights W and calibration inputs X, GPTQ [21] aims to find
quantized weights W that minimize the output reconstruction error: X - - Assuming a fixed quantization grid, GPTQ builds upon the Optimal Brain Quantization (OBQ) framework [22] to iteratively quantize and update remaining weights to compensate for the error leveraging second-order information, while avoiding OBQ's high computational complexity. Specifically, while OBQ employs a dynamic, greedy weight selection strategy for selecting the next weight to quantize, GPTQ observes that this greedy approach offers low benefits over quantizing weights in an arbitrary, fixed order, for heavily-parameteer parameterized layers. Thus, GPTQ quantizes weights across all rows in the same fixed order. This enables it to share the Hessian information, used to compute error updates, among rows. GPTQ typically implements this fixed order by processing the dimensions sequentially, column-by-column (front-to-back). The inverse Hessian must be updated only once per column (deal times) rather than once per weight (dow deat times), which reduces the overall computational complexity from

---

Published as a conference paper at ICLR 2026
Od dea) for OBQ, to (max {drow Leats deat)), providing orders-of- of-mannitude speedup, for weight matrix size dlow X deet.
4.1 ADAPTING GPTQ TO FP4 FORMATS
Our analysis showed that, with RTN quantization, NVFP4 provides lower MSE relative to MXFP4, due to better outlier preservation and smaller group size. GPTQ induces an orthogonal direction in the design space, relative RTN, as it allows for MSE error to be "corrected" by shifting it to other weight blocks. This suggests three general solution strategies: (1) GPTQ applied to the standard NVFP4 grid, with absmax scaling, leveraging the natural properties of NVFP4. This simply extends RTN to GPTQ; (2) MR-GPTO-MXFP4, GPTQ applied to the MXFP4 grid, on rotated weights and activations, this reduces MSE for RTN; (3) -RTQ-TT--::: GPTQ on an MSE-oplimized NVFP4 grid, with rotated weights and activations.
While the first two approaches follow naturally from analysis, the third approach wagers that the higher per-group local MSE caused by applying Hadamard rotations to NVFP4 can be compensated by optimizing the scales, together with the GPTQ updates. As such, options 2 and 3 would offer unified rotated/normalized format, that would apply to both NVFP4 and MXFP4. Next, we describe three key technical additions to the GPTQ algorithm that help bridge the gap between variants.
Ingredient I: MSE-Optimized Grids, Our first step in MR-GPTQ is to identify good initial grid. Recall that NVFP has both tensor (global) and per-group scales, which we denote by ST and sG. respectively. The quantized variant of the element X; will be represented as X; 8T 8G Q(X./(8T (G))) where Q is the quantization operation. To minimize error, we solve the following optimization problem for each tensor, across its groups: miney.ea, .....,, E.11 - X, where (sc,): )s are the quantization scales for the h groups. We solve this by using alternating optimization over the block scales and the per-tensor scale, respectively. For NVFP4 without rotations, we have found this to yield consistent improvements. For MXFP4 with rotations, we have found that single static value works stably across all layers, and we therefore use this approach in our implementation.
Ingredient 2: Static Activation Reordering. The original GPTQ algorithm heuristically re-orders the weight columns following the "dynamic act-order", i.c., a descending order of the corresponding Hessian diagonal entries. This matrix shuffle is applied before the quantization grid and scales computed. While this consistently improves accuracy, it also requires re-shufflin the matrix columns dynamically at runtime, which results in a 10-20% end-to-end inference slow-down.
Instead, we observe that we can apply the activation re-ordering statically, i.e. after the scales and the quantization grid have been computed in the first step, based on the original (arbitrary) column order. In practice, we first fix the grid and scales for each group, shuffle the columns before GPTQ is applied, and then finally shuffle the columns back, maintaining the microscaling group structure of the original matrix. Importantly, this benefits from the improved behaviour during the quantization process itself, without any runtime penalties. This can be applied to GPTQ over any grid, and provides similar improvements to standard "dynamic" act-order, without the runtime overheads.
Ingredient 3: Fused Online Rotations. Our MR- AR-GPTQ variants rotate the weights and activations via a block- wwse Hadamard transform Hg. with l x i diagonal blocks, where is a power-of-two. Mathematically, for linear layer with weights W and activations X, both quantized, the operation occurs as Q(W H₂) QX where H₃ is the block-wise rotation, and Q is the quantization function. In the next section, we describe how this format can be supported efficiently at runtime.
MXFP scale fitting Additionally, for MXFP in Appendix H we propose a simple scale-fitting modification, mapping the excessively large H8M0 range into data range that significantly improves performance of MXFP quantized models.
4.2 GPU KERNEL SUPPORT FOR MR- GRTT VIA QUTLASS
To support the methods described above, we introduce set of kernels optimized for NVIDIA Blackwell GPUs. These kernels constitute QuTLASS v1.0, a high-performance library for low- precision deep learning quantization, building on NVIDIA CUTLASS [42]. QuTLASS provides full support for quantization- and matmul-related operations in both NVFP4 and MXFP4 micro-scaling formats. In addition, we release architecture-optimize implementations for different NVIDIA
7

---

Published as a conference paper at ICLR 2026
Blackwell compute capabilities, namely SM100 [44] and SM120 [46]. The kemels in QuTLASS can be grouped into two categories, which will handle the computation of Q(W (x))(X
1. Quantization-related kernels. While the product WH+ is pre-fused in the weights, XH₃ occurs online. To avoid diminishing the benefits of FP4 hardware acceleration, QuTLASS provides lightweight fused kernels for online rotation. These kernels support "unimodal" k x & block diagonal matrices with k E {16, 32, 64, 128). For l < 256, dense transformations remain memory-bound meaning that any rotation (not just Hadamards) can be applied at essentially the same cost, as the full matrix can be loaded at runtime (e.g., see Tables 13 and 14). To further reduce overhead, quantization and scale calculation
fused into the transformation kernel as a custom epilogue function. QuTLASS currently supports MSE and Abs-Max quantization methods, while its template- based design allows new methods to be easily integrated.
2. Matmul-related narrow precision kernels. Between FP4 quantization and matrix multiplication, mandatee-mandated rearrangement of scaling factors is required [45] for tcgen05 mma. QuT- LASS implements this step using a Triton kernel. For the matmul itself, QuTLASS supports multiple backends, including CUTLASS [42] and FlashInfer [57], enabling flexible plug-and aagg-nddppay backend selection depending on workload and hardware.
EXPERIMENTAL RESULTS
1. Experiments with Emulated Quantization. We first evaluate the highly-popular Llama 3. 3.1-8B- Instruct model [18], examining the impact of quantizing both weights and activations for all linear layers in this model to the INT4 and FP4 formats, using different algorithms. To ensure compatibility, experiments are performed using simulated quantization in PyTorch. We use subset of tasks from the Open LLM Leaderboard V1 [5] for evaluation: GSM8K for grade school math [10], MMLU for world knowledge and reasoning [26; 9], Winogrande and HellaSwag for language understanding [48; 58). (Other tasks in this harness yield similar scores across top methods.) The INT4 experiments use group size 32 with FP16 scales, matching the average bit- bit-w of NVFP4.
Algorithms. We consider both weights--nd-activane
quantization (W4A4, our main focus) and
weight-only quantization (W4A16, as a "control")). For W4A4, we implement the following: (1)
-oundto--eearet (RTN) quantization to the corresponding format, with absmax scales. In addition. we add Hadamard rotations matching the quantization group size (32), denoted as RTN HT. (2) SmoothQuan (54] diagonal rescaling, with a tuned a smoothening factor. We identified a = 0.6
to be the best in our experiments. (3) QuaRot [4], which adds Hadamard rotations strategically at each linear layer. These should reduce quantization error, and most of them can be folded into the model. We use RTN for quantization post-rotation. (4) SpinQuant [38], which adds trainable rotations to the model, similarly to QuaRot. A subset of 1024 calibration sequences from Fine Web is used for training the matrices. (5) GPTQ [21] weight quantization and RTN on the activations, with absmax scales. A subset of 1024 calibration sequences from FineWeb, absmax scales, standard Hessian dampening factors (A = (0- and standard quantization order are used. (6) MR-GPTQ weight quantization, i.c., GPTQ with block rotations, MSE scale optimization, and static activation re-ordering over the rotated weights, as described in Section 4.1, with RTN on the activations. As a control, we also implement weight-only quantization, via RTN, GPTQ, AWQ [36], as well as
Hadamard rotations followed by RTN, denoted as RTN + HT. These results closely follow our findings for W&A quantization, and are thus deferred to the Appendix. In Appendix K, we perform an exhaustive sweep over DCT, DST, Hadamard, and GSR transforms and block sizes showing that the Hadamard transform matching the quantization group size provides the best results on average.
Discussion. The accuracy results for W4A4 experiments on Llama-3. 1..ma--.11-B-.... 1-B--BBIImm -nsttuut are presented in Table 1. The variance for the NVFP4 experiments (i.e., for entries in the 7th column over 5 distinct seeds) is of approximately 0.3 average points, whereas the variance for the INT4 experiments is of approximately point. We mark all top aggregate entries (within 2 standard deviations) as bold in the corresponding columns. We observe the following:
Across all formats, the accuracy drop is noticeable. The lowest average drop is for the NVFP4 format with SmoothQuant, GPTQ, or MR-GPTQ (these results are within variance of each other). The weight quantization results (Appendix Table 2), show that the induced error is roughly evenly split between weight and activation quantization. These results suggest that micro-scaling is not direct solution for accuracy recovery. (2) NVFP4 provides the best accuracy,
(1) No Lossless Format:
8

---

Published as a conference paper at ICLR 2026
Format
Method
MMLU-CoT
GSM8K
HellaSwag
WinoGrande
Avg.
Recovery
Baseline
FP16
72.76
85.06
80.01
77.90
78.93
100
RTN
72.50
84,80
80.20
77.40
78.73
99.74
INT8
GPTQ
72.40
84.40
80.00
77.30
78.53
99.48
RTN
72.40
84.70
79.80
77.70
78.65
99.64
FP8
GPTQ
71.80
84.50
79.90
78.10
78.58
99.55
RTN
65.96
74,68
77.62
74.19
73.1
92.63
INT4
RTN+HT
68.30
79.61
77.60
73,48
74.75
94.71
GPTQ
66.36
76.65
77.38
72.48
73.21
92.75
RTN
68.26
78.39
78.15
74.11
74.73
94.67
RTN + HT
67.41
78.01
77.31
73.48
74.05
93.82
QuaRot
66,50
77.40
77.25
75.14
74.10
93.80
NVFP4
SpinQuant
66.50
76.10
76.96
75.32
73.70
93.40
SmoothQuant
68,90
79.50
79.50
74.70
75.70
95.90
GPTQ
68.85
82.60
78.26
74.51
75.72
95.92
MR-GPTQ
69.12
80.80
78.17
75.24
75.84
96.08
RTN
62.21
67.85
73.99
73.24
69.32
87.83
RTN + HT
62.38
72.48
75.29
71.67
70.45
89.26
SmoothQuant
63.93
68.54
75.10
73.56
70.30
89.06
MXFP4
QuaRot
49.86
56,94
73.50
71.43
62.90
79.70
SpinQuant
61.80
68.16
74.87
72.93
69.40
88.00
GPTQ
63,49
68.46
76.01
74.51
70.62
89.47
MR GPTQ
67.19
75.70
76.91
74.80
73.65
93.31
RTN
68.56
78.17
78.64
75.14
75.13
95.18
RTN + HT
68.59
81.73
78.38
74.27
75.74
95.96
NVINT4
GPTQ MR-GPTQ
68.69
81.58
77.59
73.40
75.32
95.42
69.71
82,26
79.14
75.53
76.66
97.12
RTN
55.06
56.79
72.06
68.27
63.05
79.87
RTN + HT
58.44
61.64
73.94
71.19
66.30
84.00
MXINT4
GPTQ
61.22
67.70
75.04
71.67
68.91
87.30
MR-GPTI GPTQ
65.48
74.83
76.63
73.09
72.51
91.86
Table I: Unified accuracy comparison of 1-mm--3.-8BBIInttruec W4A4 under different quantization formats and methods. each format, top methods within variance are marked in bold.
with INT4 second, and MXFP4 third: On average, NVFP4 and INT4 quantization provide similar quality, with INT4 quantization having higher variance. The MXFP4 format is distant third in terms of accuracy, regardless of the method used, but benefits significantly from AR-GPTQ. (3) Quantization Method Efficiency: First, note the good performance of standard RTN for INT4 (with rotations) and NVFP4 (without). Second, the Hadamard transform appears effective for INT4 and MXFP4 (which use group size 32), but is less effective for NVFP4 (which uses group size 16), confirming our analysis. In particular, for round-to- no-aresst quantization, adding the Hadamard transform to NVFP4 hurts accuracy. Finally, the GPTQ and SmoothQuant methods appear to be consistently but moderately effective across all three formats.
2. Real Quantization. We integrate our kernels in vLLM [30], and perform accuracy evaluations directly in this setup over additional models, such as Llama-3.3- 3.B-33B-------- 77B--Instrue [18], and the Qwen3 [55] family of models. The results presented in Table 10 and Figure 7. For this experiment, provide results for Quantization- Aware Training (QAT) performed using the balanced Generalized Divergence loss [20; 33] between the quantized and the unquantized (frozen) model token distributions on a subset 92,995 samples (10%) from the Tülu 3 [31] instructions dataset. The results show that accuracies measured over real kernels for the 3lama-3.1-88 -B---nsttuct model track closely with the results from simulation, with slightly lower recoveries (within 0.2-0.3% Smaller models (< 8B) and famaly models tend to have lower recovery rates, whereas Qwen3 models can achieve more than 99% average recovery in NVFP4. For NVFP4, standard GPTQ provides the highest recoveries on average, although RTN and MR-GPTQ are also competitive, with QAT only providing very limited benefits. For MXFP4, MR-GPTQ provides the best recovery among PTQ methods, while QAT consistently reduces the gap to full precision. Additionally, in Appendix we analyze differences between GPTQ variants the less noisy PlatinumBench benchmark [53].
also

---

Published as a conference paper at ICLR 2026
3. Integer Microscaling Formats. Although integer-based microscaling formats not specified in the OCP standard [47], one could easy extend the idea to them. For validation purposes, we propose the two following microscaling INT4-based formats: NVINT4 and MXINT4 that are identical to NVFP4 and MXFP4 respectively except for the base type being INT4 instead of FP4. The error and outlier preservation analysis in Figure 10 predicts that NVINT4 should profit from HT and yield performance superior all other formats. "seudo-quantization evaluations Tables 7 and Figure verify that for O-shot (RTN) and 1-shot (MR-GPTQ) quantization finding similar to that of Chen al. [7].
Shape: 57344x8 192 (OUTXIN)
6000
vLLM (QuTLASS) Llama3.3 70B MXFP4
Ideal MXFP4
Specdup over BF16
IxB200, Prompt:Deecdde 32/128
Actual
5000
MXFP4+H; 128
Ideal NVFP4
TFLOPs's B200
4000
Actual NVFP4
52
1336
2566
512
1024
2048
4505
8193
torch BF16
size
Figure QuTLASS performance end-to-end using our vLLM integration.
2000
100
Avg Recovery, %
Hadamard
1000
80
52
64
128
256
512
10242048 8192
70
NVINT MANN
SVNN
Batch size
RTN
(MR-GGPTQ
Figure Average (over Llama 3 1,3,8B) recoveries for microscaling formats and the effect of HT.
Figure 5: QuTLASS performance for single linear LLM layer.
Kernel and Inference Performance. Finally, Figures and 6, we examine the performance of our kernels. Figure shows throughput for a single layer extracted from Llama-333-700% model using FlashInfer backend. The curve labeled with "ideal" represents the upper bound for a real 4-bit weight and 4-bit activation matrix multiplication, i.e., the measured matmul throughput not including the overhead of quantization-related operations. In contrast, the curves labeled "actual" show real measurements including the costs of Hadamards, quantization, and scale computation. The comparison highlights the small gap between idealized efficiency and practical implementations with our kernels, with speedups of up to Rs 3.6x (out of 4x) on B200 and R 6x (out of 8x) on RTX5090. Interestingly, MXFP4 outperforms NVFP4 on B200, with up 15% higher throughput, despite their closely related numerical formats. Possible contributing factors include MXFP4's ''s use of potentially more efficient power-of-twe scales as well as larger group sizes, which could reduce overhead.
Figure 6 shows end-to-end speedup of vLLM running 1lama-3.3-70B with MXFP4 quantization compared the baseline BF16 implementation on single B200 GPU. The results demonstrate consistent performance gains across batch sizes, with speedups reaching up to 2.2x over the BF16 baseline, and nearly 4x on an RTX 5090 GPU (see Appendix for more details).
CONCLUSION
We present first comprehensive study of the recently introduced MXFP4 and NVFP4 formats for LLM quantization, revealing gaps between the promise of these formats and their performance using state-of-the of-the-art methods. To bridge these gaps, introduce novel GPTQ variant adapted to these formats. We support this approach with QuTLASS, a suite of high-pperformacce GPU kernels that implement MR-GPTQ' micro-rotaan rotations with negligible overhead. We hope that our results will provide a basis and motivation for future work on improving accuracy for these novel formats.
(MR-GF ARR-GPTQ,
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
Code. The source code required to reproduce the quantization and accuracy-evaluation experiments available in the FP-Quant repository. The implementation of QuTLASS can be accessed here.
Models. MR-GPTQ quantized models are hosted in dedicated Hugging Face collection.
ACKNOWLEDGMENTS
We thank members of the Red Hat Al team, particularly Rob Shaw, Michael Goin, and Kyle Sayers, for their support and useful suggestions. We would like to thank Lambda Cloud for their generous computational grant. We thank the NVIDIA and Google corporation for their grants, which supported part of this research. Roberto I.. Castro and Andrei Panferov Cluster of Excellence program (GreenAI), whereas Vage Egiazarian and Eldar Kurtic were supported by the ERC Proof-of-Concept Grant FastAI. This work was supported under project IDD 40 part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure.
supported in part by the BILAI
REFERENCES
[1] Advanced Micro Devices, Inc. AMD CDNATM 4 Architecture Whitepaper.
White
paper, Advanced Micro Devices, Inc., June 2025. URL https: www. amd. com/ content/ tatient/daammmm/enn aw///n///////e//aaaaaaa///aaaaee /eaii/////ooo wtwp///////mmeen///eeeee/ /twp//oetstiineeeeeeeeeeeecceeee inst thw.-ttttcc-ooooootttttteeeeeeee
cana- lecenn---arccieee wwwaaraccctttrreeeeee whitreappettptee pdf. Accessed: 2025-09-24.
[2] Mohammad Sadegh Akhondzadeh, Aleksandar Bojchevski. Evangelos Eleftheriou, and Martino Dazzi. Kurtail : Kurtosis-based IIm quantization, 2025. URL hwtptttss//aiii..ottt / .503.01 01483.
[3] Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong. Xineheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09255 2023.
[4] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated Ilms. arXiv preprint 2rXhv:2404.00455 2024.
[5] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard (2023- 2024). https://uugiiggffee co/ htte/ppace//pppee 11m leaderboard
open open_1lmm_eeeeeeepppppppppp_emee Leme_lllmeesddrbaarddd 2023.
[6] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantiza- tion of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
[7] Mengzhao Chen, Meng Wu, Hui Jin, Zhihang Yuan, Jing Liu, Chaoyi Zhang, Yunshui Li, Jie Huang, Jin Ma, Zeyue Xue, Zhiheng Liu, Xingyan Bin, and Ping Luo. Int v.s. fp: A comprehensive study of fine-grat fine-grained low-bit quantization formats, 2025. URL https:// 25602.
[8] Euntae Choi, Sumin Song, Woosang Lim, and Sungjoo Yoo. Grouped sequency-arranged rotation: Optimizing rotation transformation for quantization for free, 2025. URL https: //aaxiiv.. wwtpxii..rrg/..oo httgg/bb////// /205 03810.
[9] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick. and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.052.
2018.
[10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
11

---

Published as a conference paper at ICLR 2026
[11] Tri Dao. Fast hadamard transform in cuda, 2023. URL https://github.com Fao-AILab/Ear fast hadamard- transform
[12] Tim Dettmers. 8-bit approximations for parallelism in deep learning. International Conference on Learning Representations (ICLR), 2016.
[13] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 2022.
[14] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314, 2023.
[15] Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for ncar-lossless Ilm weight compression. arXiv preprint arXiv:2306.03078 2023.
[16] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
[17] Johannes Dotzel, Zirui Liu, T.S. Jayram, and G. Edward Suh. Learning from students: Applying t-distributions to explore accurate and efficient formats for llms. In Proceedings of the 4/st International Conference on Machine Learning, Proceedings of Machine Learning Research. PMLR, 2024.
[18] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[19] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06// 2024.
[20] Erik Englesson and Hossein Azizpour. Generalized jensensshamoon divergence loss for learning with noisy labels, 2021. URL http://:///aaaw..ooo/////a
04522.
[21] Elias Frantar, Salch Ashkboos, Torsten Hoefer, and Dan Alistarh. GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17322 2022.
[22] Elias Frantar, Sidak Pal Singh, and Dan Alistarh. Optimal Brain Compression: A framework for accurate post-training quantization and pruning. arXiv preprint arXiv:2208.11580) 2022. In NeurIPS 2022.
[23] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few- shot language model evaluation, September 2021. URL https: //doi.org/00..2211 zenodo. 5371628. 628.
[24] Ruihao Gong, Yang Yong, Shiqiao Gu, Yushi Huang, Chentao Lv, Yunchen Zhang, Xianglong Liu, and Dacheng Tao. Llme: Benchmarking large language model quantization with versatile compression toolkit, 2024. URL https: /ttpos///axxiiiie arxiv.orgaaas............o /240 26000.000...............
[25] Zhuocheng Gong, Jiahao Liu, Jingang Wang, Xunliang Cai, Dongyan Zhao, and Rui Yan. What makes quantization for large language model hard? an empirical study from the lens of perturbation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18082-18089 2024.
[26] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. 2020.
12

---

Published as a conference paper at ICLR 2026
[27] Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized llama3 models? an empirical study, 2024.
[28] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and aaae--nn--ppseee quantization. arXiv preprint arXiv:2306.07629, 2023.
[29] Eldar Kurtic, Alexandre Noll Marques, Shubhra Pandit, Mark Kurtz, and Dan Alistarh. "give me BF16 or give me death"? accuracy-performance trade-off's in Ilm quantization. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 16872-268866 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-25111 doi: 10. 1/251//11/222555.../ www.5ssccclloo..... 1304. URL https: / //aclanthology..o...om org.oogg////// 2025 long 1304/.
[30] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp. 61 611-626, 2023.
[31]
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang. Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post- post-training. arXiv preprint 2rXiv:2411.15124, 2024. URL https::///// /ttps:///rxxvvv org/ abs/ 2411 15124.
[32] Jemin Lee, Sihyeong Park, Jinse Kwon, Jihun Oh, and Yongin Kwon. A comprehensive evaluation of quantized instruction-tuned large language models: An experimental analysis up to 405b. arXiv preprint arXiv:2409.1105. 2024.
[33] Jung Hyun Lee, Seungjae Shin, Vinnam Kim, Jaeseong You, and An Chen. Unifying block-wise plq and distillation-based qat for progressive quantization toward 2-bit instruction-tuned Ilms, 2025. URL https::////. :ttp::::://rxxtte /aaxiviivvv//////////a//////aavv www.i.www........aa........oooot aaaaaaba//2500 09104.
[34] Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, and Yu Wang. Evaluating quantized large language models. arXiv preprint arXiv:2402. 18158, 2024.
[35] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awars weight quantization for I1m compression and accelcration. In Proceedings of the Learning on Systems (MLSys) Conference, 2024.
[36] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awan weight quantization for on-device IIm compression and acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024.
[37] Peiyu Liu, Zikang Liu, Ze-Feng Gao, Dawei Gao, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. Do emergent abilities exist in quantized large language models: An empirical study. arXiv preprint arXivv23307.00077 2023.
[38] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant-IIm quantization with learned rotations. arXiv preprint 2.K1::2005166400 2024.
[39] Vladimir Malinovskii, Andrei Panferov, Ivan Iin, Han Guo, Peter Richtárik, and Dan Alistarh. HIGGS: Pushing the limits of large language model quantization via the linearity theorem. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 10857-10886, Albuquerque,
13

---

Published as a conference paper at ICLR 2026
New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89176-189 doi: 10. 18.13v 1/2025. naacl-long.543. URL naacl-long 543/.
[40] Gunjan Mehta, Justin Xin, Riyad Islam, Yiheng Zhang, Asfiy Baig, Akhil Goel, and Sandro Cavallari. NVIDIA TensorRT Unlocks FP4 Image Generation for NVIDIA Blackwell GeForce RTX 50 Series GPUs. NVIDIA Technical Blog, May 2025.
[41] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up down?? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
[42] NVIDIA. Cutlass: Cuda templates for linear algebra subroutines, 2017. URL https:// github. wwtp//////////////////////. ctt//uulaasssss
[43] NVIDIA. Nvidia blackwell architecture technical brief. hattps://resouuces.nviiit
com/en-us waaans-baacwellll www.ll--rchhtttuureeee 2024.
[44] NVIDIA. NVIDIA DGX B200.
rttps://resoorrce..viiie
un-us-dgx- systems/////... /dgx //ddxxbb00--00 11:200-daa 2are00dddaaasttettteee tic 2025.
[45] NVIDIA. cuBLAS. https://dbcs.nvidiz. hatpaaa.ccc//uuaacaa/a//s/// wwwphlaaaa/iiii....aaa html# d-block- scaling- factors- 2025.
[46] NVIDIA. NVIDIA RTX
Blackwell GPU
Architecture.
https://
images.nvidie com/ aem www.//ammdam//oooaaaaaeaeaaeeeee htwaammmaaaannolttiiooo/eeeeoeea http:///geooooe///ceelll/l hbt/aawwell////eeeeeooe
pdf, 2025.
[47] Open Compute Project Foundation (MX Alliance). OCP Microscaling Formats
(MX) Specification Version 1.0. Open Compute Project Foundation Technical Speci- fication, September 2023. URL https:/ www. opencompute. org/ documents/
ocp ocp-microseeee microscaling- formats --mm---------------------- mxxvv--0 0 spec final -ddf.
[48] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
[49] Claude E. Shannon. Coding Theorems for Discrete Source With Fidelity CriterionInstiuute of Radio Engineers, International Convention Record, vol. 7, 1959., pp. 325-350. 1993. doi: 10. 1001109999884477542222.h.22
[50] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#:: Even better I1m quantization with hadamard incoherence and lattice codebooks, 2024. URL https: / arxiv. org/ abs/ 2402. 04396.
[51] Albert Tseng, Qingyao Sun, David Hou, and Christopher De QTIP: Quantization with Trellises and Incoherence Processing. In Advances in Neural Information Processing Systems (NeurIPS) Spotlight, December 2024. URL https::tp//. rxiv.. www.v.rrg/aassee. /20/ 2406.446 11235.
[52] Albert Tseng, Tao Yu, and Youngsuk Park. Training Ilms with mxfp4, 2025, URL https: 1/arxiv. org abs 2502 20586.
[53] Joshua Vendrow, Edward Vendrow, Sara Beery, and Alcksander Madry. Do large language model benchmarks test reliability?, 2025. URL wttp://wwwwoogo//////oo////goaap 2502. ...00..033.. 03461.
/ arxi wrw.orgg//ao.oo abs
[54] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
[55] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.1 15115, 2024.
[56] Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems (NeurIPS 2022), 2022.
14

---

Published as a conference paper at ICLR 2026
[57] Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, and Luis Ceze. Flashinfer: Efficient and customizable attention engine for IIm inference serving. arXiv preprini arXiv:2501.00000 2025.
URL. https: :ttttts:://rxeee /arxiv..vv//./////////a/////aaa/ .rgg hbs///b///0 2501 01005.
[58] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.0700 2019.
[59] Yilong Zhao, Chien- Yu Lin, Kan Zhu, Zihao Ye, Lequn Chen, Size Zheng, Luis Ceze, Arvind Krishnamurthy, Tianqi Chen, and Baris Kasikci. Atom: Low-bit quantization for efficient and accurate IIm serving. arXiv preprint 2xXi::3300.99100 2023.
15

---

Published as a conference paper at ICLR 2026
TECHNICAL APPENDICES AND SUPPLEMENTARY MATERIAL
A WEIGHT-ONLY QUANTIZATION RESULTS
The results for weight-only quantization provided in Table 2. One observe that similary the weight and activation quantization case INT4 and NVFP4 perform similarly, while MXFP suffers much significant accuracy drop. Even for weight-only case there is 2% accuracy drop average relative to the original model.
Format
Quantization
MMLU
GSM8k
HellaSwag
WinoGrande
Avg.
Recovery% %
FP16
72.80
85.10
80.00
78.90
79.70
RTN
69.38
81.80
79.41
77.90
77.12
97.71
INT4
RTN+Had
70.27
82.56
79.18
76.64
77.16
97.76
GPTQ
70.25
80.52
79.01
76.64
76.60
97.05
RIN
70.64
82.26
79.24
77.35
77.37
98.02
RTN+Had
69.26
80.82
78.52
77.03
76.41
96.80
NVFP4
GPTQ
70.52
82.49
79.35
76.95
77.33
97.96
AWQ
70.57
82.71
79.30
77.03
77.40
98.06
RTN 68.23
80,36
77.26
75.93
75.44
95.58
RTN+Had
66.24
77.56
77.34
74.11
73.81
93.51
MXFP
GPTQ
68.79
81.43
78.40
76.88
76.37
96.76
AWQ
68.16
78.70
78.56
75.30
75.18
95.25
Table 2: Performance 1aama--.1-8BIInsttii under different weight-only quantization settings.
B REAL QUANTIZATION RESULTS
In this section we provide complete set of evaluation results for Llama-3 Llama-3.2-3B-Insteee Llama-3.1-8B-Inttrue. -Jama-3.3-70B-Instruue and Qwen-3 (Qwen-3-8B, Qwen-3-1444 Qwen-3-32B) model families. We turn off thinking mode for Qwen as it turned out that long reasoning saains-of-thougee turned out be detrimental for performance GSM8k and MMLU-CoT. The scores were produced using QuTLASS vLLM integration.
Table 10 present more details on accuracy recovery for micro-scaling formats, including NVINT4 and MXINT4 and QAT.
16

---

Published as a conference paper at ICLR 2026
Format
Quantization
MMLU
GSM8K
HellaSwag
WinoGrande
Avg.
Recovery
FP16
46.20
46.32
59.78
61.56
53.47
RTN
45.90
44.20
59.80
61.30
52.80
99.55
INT8
GPTQ
45.40
44.90
59.60
60.10
52.50
98.99
RTN
46.10
44.70
59.50
59.50
52.50
98.99
FP8
GPTQ
45.80
45.00
59.10
60.60
52.63
99.22
RTN
36.08
31.39
54.77
57.22
83.91
RTN+Had116
32.80
25.02
56.24
59.04
43.28
80.94
RTN+Had128
38.28
29.95
54.27
58.41
45.23
84.59
GPTQ
37.79
29.80
55.48
60.22
45.82
85.71
NVEP
GPTQ+Had16
38.99
32.98
56.66
58.17
46.70
87.35
GPTQ+Had125 28
35.47
31.16
57.02
59.19
45.71
85.50
QAT
27.85
38.51
57.52
60.30
46.05
86.12
QAT+H
32.72
37.60
57.53
58.41
46.57
87.09
RTN 30,46
11.83
48.28
54.22
36.20
67.70
RTN+Had32
30.89
19.41
51.64
57.22
39.79
74.42
RTN+Had128
34.48
25.55
53.98
58.01
43.01
80.44
GPTQ
26.84
13.50
49.29
56.75
36.60
68.45
MXFP
GPTQ+Had32
29.44
27.60
54.89
58.72
42.66
79.80
GPTQ+Had128
35.68
28.13
54.60
58.72
44.28
82.83
QAT
15.60
20.32
53.34
56.51
36.44
68.16
QAT+Had32
28.12
36.85
57.04
58.80
45.20
84.55
RTN
37.33
26.08
52.62
58.56
43.65
81.64
RTN+
33.41
32.52
57.12
59.19
45,56
85.21
NVINT4
GPTQ
37.15
27.60
55.94
59.35
45.01
84.19
MR-GPTQ
36.69
33.36
57.95
58.96
46.74
87.42
RIN
21.85
4.55
45.07
$5.49
31.74
59.37
RTN+Had32
13.17
9.48
48.91
53.99
31.39
58.71
MXINT4
GPTQ
23.42
13.27
50.02
55.88
35.65
66.67
MR-GPTQ
21.81
23.12
54.96
55.41
38.83
72.62
Table 3: Performance Llama-3.2 -...a...--B-----B I.n-tttInntttuct different weight & activation quantization settings.
17

---

Published as a conference paper at ICLR 2026
Format
Quantization
MMLU
GSM8k
HellaSwag
WinoGrande
Avg.
Recovery%
FP16
64.43
78.01
73.42
70.09
71.49
RTN
64.00
77.70
73.30
69.60
71.15
99.47
INT8
GPTQ
64.00
77.90
73.40
69.90
71.30
99.67
RTN
63.40
77.70
73.00
69.70
70.95
99.19
FP8
GPTQ
64.10
77.80
73.00
69.90
71.20
99.54
RTN
60.62
70.43
70.99
68.03
67.52
94,45
RTN+Hadd16
59.91
64.82
69.77
65.59
65.02
90.96
RTN+Had128
54.34
67.48
69.69
66.93
64.61
90.38
GPTQ
61.76
70.36
71.07
69.93
95.51
NVFP
GPTQ+Had16
60.26
68.76
71.05
67.80
66.97
93,68
GPTQ+Had128
60.19
70.89
70.97
68.19
67.56
94.51
MicroQAT+HadI6
60.66
69.98
70.55
67.01
67.05
93.79
QAT
62.06
75.06
71.27
67.96
69.09
96.64
QAT+Had16
62.03
72.93
70.95
66.46
68.09
95.25
RTN
56.81
60.80
67.30
64.56
62.37
87,24
RTN+Had32
55.58
57.77
68.56
64.33
61.56
86.11
RTN+Had128
55.95
60.80
67.57
64.88
62.30
87.15
GPTQ
57.68
62.32
63.87
64.88
62.19
86.99
MXFP
GPTQ+Had32
59.79
68.92
69.50
66.85
66.27
92.69
GPTQ+Had128
59.56
67.78
70.08
68.03
66.36
92.83
MicroQAT+1aad32
59.49
65.66
69.05
67.32
65.38
91.46
QAT
56.17
64.90
69.51
67.17
64.44
90.14
QAT+Had32
59.83
72.48
70,27
66.54
67.28
94.11
RTN
60.22
71.65
70,92
66.77
67.39
94,27
RTN HT
56.75
71.95
69.24
66.61
66.14
92.52
NVINT4
GPTQ MR-GPTQ
60.5
71.42
27.01
51.07
52.50
73.44
60.8
72.25
71.74
70.48
68.82
96.27
RTN
46.03
47.54
64.21
61.56
54.84
76.71
RTN + HT
50.55
51.33
64.61
59.83
56.58
79.15
MXINT4
GPTQ
53.48
60.73
51.44
58.80
56.11
78.49
MR-GPTQ
57.73
67.48
68.92
66.69
65.21
91.21
Table 4: Performance of 1itma--.2-3B-Inssee for different weight & activation quantization settings.
18

---

Published as a conference paper at ICLR 2026
Format
Quantization
MMLU-CoT
GSM8K
HellaSwag
WinoGrande
Avg.
Recovery%
FP16
72.80
85.10
80.00
77.90
78.90
RTN
72.50
84.80
80.20
77.40
78.73
99.74
INT8
GPTQ
72.40
84.40
80.00
77.30
78.53
99.48
RTN
72.40
84.70
79.80
77.70
78.65
99.64
FP8
GPTQ
71.80
84.50
79.90
78.10
78.58
99.55
RTN
68.70
78.70
78.40
73.40
74.80
94.80
RTN+Had
67.00
77.40
77.30
74.40
74.00
93,80
RTN+Had128
66.60
77.00
77.50
75.50
74.10
93.90
GPTQ
68.60
79.60
78.70
75.50
75.60
95.70
NVEP
GPTQ+Had
69.40
79.60
78.40
75.10
75.60
95.80
GPTQ+Had128
68.90
79.50
78.30
73.60
75.10
95.10
QAT
68.20
79.80
78.90
74.40
75.30
95.40
QAT+Had
68.90
81.60
79.00
75.10
76.10
96.50
RTN
62.20
69.50
73.80
72.60
69.50
88.10
RTN+Had
62.60
71.80
75.20
72.30
70.50
89.30
RTN+Had
64.50
72.70
76.00
73.30
71.60
90.70
GPTQ
63.74
70.20
75.52
7364
70.78
89.66
MXFP
GPTQ+Had
67.20
77.50
77.00
73.10
73.70
93.30
GPTQ+Had128
66.80
78.30
76.90
74.90
74.20
94.00
QAT
65.00
76.00
77.60
72.90
72.90
92.30
QAT+Had
67.60
80.30
78.30
74.90
75.30
95.40
RTN
68.56
78.17
78.64
75.14
75.13
95.18
RTN HT
68.59
81.73
78.38
74.27
75.74
95.96
NVINT4
GPTQ
68.69
81.58
77.59
73.40
75.32
95.42
MR-GPTQ
69.71
82.26
79.14
75.53
76.66
97.12
RTN
55.06
56.79
72.06
68.27
63.05
79.87
RTN HT
58.44
61.64
73.94
71.19
66.30
84.00
MXINT4
GPTQ
61.22
67.70
75.04
71.67
68.91
87.30
MR-GPTQ
65.48
74.83
76.63
73.09
72.51
91.86
Table 5: Performance Llama-3. -nstruuttttttct different weight & activation quantization settings.
Format Quantization
MMLU
GSM8k
HellaSwag
WinoGrande
Avg.
Recovery %
FP16
86.55
95.07
86.22
84.93
88.19
RTN
85,50
93.48
85.63
83.27
86.97
98.61
RTN+HadI6
85.02
93.63
84.97
83.82
86.86
98.49
KTN+Had128
85.24
91.81
84.91
83.35
86.33
97.89
NVEP
GPTQ
85.54
94.09
85.49
84.37
87.37
99.07
GPTQ+Had16
85.58
93.40
85.45
82.40
86.71
98.32
GPTQ+Had128
85.59
94.16
85.56
84.77
87.52
99.24
RTN
83.42
92.65
83,93
81.45
85.36
96.79
RTN+Had32
83.86
93.56
84.13
83.58
86.28
97.83
RTN+Had128
84.37
94.47
84.22
82.40
86.37
97.93
MXFP
GPTQ
83.77
94.47
84.41
82,64
86.32
97,88
GPTQ+Had32
84.82
94.54
84.66
83.11
86.78
98.40
GPTQ+Hadd12
84.90
93.90
84.80
83.80
86.86
98.48
Table 6: Performance of Llama-3.3-70B. -nsttuutttttt for different weight & activation quantization
settings.
19

---

Published as a conference paper at ICLR 2026
Format Quantization
MMLU
GSM8k
WellaSwag
WinoGrande
Avg.
Recovery%
FPI6
72.98
90.90
73.52
70.56
77.49
RTN
70.78
90.30
74,63
70.72
76.61
98.86
KTN+Had16
70.19
86.35
73.02
68.11
74.42
96.04
KIN+Had128
69.09
86.66
73.47
67.96
74.30
95.88
GPTQ
70.90
88.17
75.01
70.09
76.04
98.13
NVFP
GPTQ+Had16
71.06
88.32
74.58
68.03
75.50
97.43
GPTQ+Had128
70.45
87.41
74.25
68.90
75.25
97.11
QAT
70.94
89.08
74.67
68.51
75.80
97.82
QAT+Had16
71.34
89.23
75.24
70.40
76.55
98.79
RTN
67.69
84.23
71.24
67.40
72.64
93.74
KTN+Had32
67.57
83.78
71.32
67.32
72.50
93.56
67.27
81.58
71.41
66.38
71.66
92.48
GPTQ
68.01
84.23
71.65
67.80
72.92
94.11
MXFP
GPTQ+Had32
69.13
84.84
73.17
68.03
73.79
95.23
GPTQ+Had128
69.53
86.43
73.55
65.75
73.82
95.26
QAT
69.45
87.34
74.03
69.85
75.17
97.00
QAT+Had32
70.35
89.61
74.61
70.56
76.28
98.44
Table 7: Performance of Qwen-8B for different weight & activation quantization settings.
Format Quantization
MMLU
GSM8k
HellaSwag
WinoGrande
Avg.
Recovery
FP16
77.18
91.96
79.84
74.27
80.81
RTN
75.73
91.28
78.36
73.16
79.63
98.54
RTN+Had16
74.98
92.04
77.76
72.38
79.29
98.12
RTN+Had128
74.46
91.13
77.60
71.98
78.79
97.50
NVFP
GPTQ
74.88
91.28
78.40
74.51
79.77
98.71
GPTQ+Had16
75.49
91.43
78.38
74.51
79.95
98.94
GPTQ+Had128
75.10
90.52
78.30
72.77
79.17
97.97
RTN
72.92
90.22
76.68
71.51
77.83
96.31
RTN+Had32
73.19
89.54
75.95
71.67
77.59
96.01
RTN+Had128
73.17
85.60
76.80
72.14
76.93
95.19
MXFP
GPTQ
72.57
89.54
76.50
72.45
77.77
96.23
GPTQ+Had32
74.36
89.92
77.64
72.53
78.61
97.28
GPTQ+Had128
74.11
89.92
77.77
71.11
78.23
96.80
Table 8: Performance of Qwen-14B for different weight & activation quantization settings.
Format Quantization
MMLU
GSM8k
HellaSwag
WinoGrande
Avg.
Recovery
FP16
80.81
92.04
83.97
76.56
83.35
RTN
79.85
94.24
83.27
75.22
83.15
99.76
1TN+
78.90
89.23
82.60
76.48
81.80
98.15
RIN+Had128
78.49
89.69
82.47
75.37
81.51
97.79
NVFP
GPTQ
79.54
92.87
83.24
75.93
82.90
99.46
GPTQ+Had16
78.60
90,90
82,93
75.14
81.89
98.26
GPTQ+Had128
79.11
90.52
83.15
76.09
82.22
98.65
RTN
77.07
72.33
81.52
75.22
76.54
91.83
RTN+Had32
78.22
93.03
81.76
75.93
82.24
98.67
RTN+Had128
78.36
88.10
81.66
75.30
80.86
97.01
MXFP
GPTQ
77.01
88.55
81.79
74.90
80.56
96.66
GPTQ+Had32
78.46
82,41
82.72
75.06
79.66
95.58
GPTQ+Had128
78.90
90.90
82.29
75.22
81.83
98.18
Table 9: Performance of Qwen-32B for different weight & activation quantization settings.
20

---

Published as a conference paper at ICLR 2026
Table 10: Per-model recoveries with real (NVFP4 and MXFP4) and hypothetical (NVINT4 and MXINT4) quantization.
Format
Method
HT
Llama3
Qwen3
IB
3B
8B
70B
8B
14B
32B
RTN
83.9
94.4
94.8
98.6
98.9
98.5
99.8
RTN
80.9
91.0
93.8
98.5
96.0
98.1
98.1
GPTQ
85.7
95.5
95.7
99.1
98.1
98.7
99.5
NVFP4
MR-GPTQ
16
87.3
93.7
95.8
98.3
97.4
98.9
98.3
QAT
86.1
96.6
95.4
97.8
QAT
16
87.1
95.3
96.5
98.8
RTN
67.7
87.2
88.1
96.8
93.7
96.3
91.8
RTN
32
74.4
86.1
89.3
97.8
93.6
96.0
98.7
GPTQ
68.4
87.0
89.7
97.9
94.1
96.2
96.7
MXFP4
MR-GPTQ
32
79.8
92.7
93.3
98.4
95.2
97.3
95.6
QAT
68.2
90.1
92.3
97.0
QAT
32
84.5
94.1
95.4
98.4
RTN
81.6
94.3
95.2
96.3
98.1
RTN
16
85.2
92.5
96.0
98.1
99.2
NVINT4
GPTQ
84.2
92.5
95.4
MR- MR-GPTQ
16
87.4
96.3
97.1
RTN
59.4
76.7
79.9
83.9
92.4
RTN
32
58.7
79.1
84.0
89.7
94.9
MXINT4
GPTQ
66.7
78.5
87.3
GRTG
32
72.6
91.2
91.9
21

---

Published as a conference paper at ICLR 2026
C SCALE QUANTIZATION ANALYSIS
As discussed in the main text, microscaling formats adopt scale quantization to reduce memory storage overhead and accelerate dequantization operations. However, scale quantization may introduce additional error due to rounding of scales onto coarser grid. Below provide analysis and explore alternative choices for scale quantization.
MXFP format adopts E8MO grid with exponentially spaced levels. It allows to represent values with very small and large magnitude, yet the distance between adjacent levels can be pretty large resulting in large approximation errors. E4M3 grid used in NVFP, on the other hand, has much narrower dynamic range [-448, 448] with levels spread more uniformly. We note, that the sign bit is in fact never utilized, given that the scale is non-negative value by definition.
Below, we explore several choices for 8-bit scale quantization with fixed group size of 16. Specifi- cally, measure weight and activation MSE for range of FeMm formats with e well as for E8M0 and INT8. For E8M0 scale quantization, we multiply the scale by 4/3 following [52], which yields an unbiased estimate of the original scale and reduces quantization error. Results for weight and activation quantization are shown in Figure and Figure 9, respectively.
7, as
self attm proj
self attn proj
self attn.y proj
self attn.o proj
[W Q/W0[] [W18 [WW -Q000 B [[]11]
FP16 Scales
I Scales
2:18
FP16 Scales
FP15 Scales
INTS Scales
FP16 Scales
INTS Scales
80.5
ERMO Scales
ERMD Scales
EXMD Scales
ERMO Scales
E314
0014
2012
0012
8009
ORIE
ETM Distt EMM (MM
GIME
ECAM ITAs E8M IS5e EXM
FIMM IIM (463 ESM2 (MM1)
IIIM 117M B334 BAMT 1960 1980
mlp. palp-gate proj
mip.up peoj
mlp dolp.down proj
FP16 Scales
FP16 Scalex
Scales
INTS Scales
INT8 Scales
0119
INTH Scales
8005
DRMO Scales
DSMO Scales
ESMO Scales
2014
8062
0112
B15t Bi54 EING EIM Br50
Eide EINS Babte E552 sekci
Eib4 560
HP8
EP8
Figure 8: MSE for the weights of 15th block in the Llama-3.1 ------nntttt model.
self q proj
self sttn.k proj
self proj
self attn.o proj
QU0ill: [x]
FP16 Scales
FP16 Scales
EP16 Scalos
FP15 Scales
INT8 Scales
13900
INTS Scales
INTA Scales
INTS Scales
Scales
Scales
Ea175
80175
0014
E3199
E3199
E013
0010
01146 BIM ITA64 DIME PMN bxb
RIME ESMS 19MI EAN BGM2 B5M
ITM: EIM BTMC Dib B5N2 nsM
11T86 BOME TOMP BAM DIMe DIM
mip.gate proj
mip.up proj
mip dowd proj
+ [|X
Scales
FP16 Scales
FP16 Scales
INTS Scales
IN Stales
INTS Scales
8000
ESMO Seales
0038
ESMP Scales
ESMI Seales
8000-
0036-
20121-
10150
80125-
00.00
50130
EX03
E384 BAME
EIMS
E3X2 sski
EIM4 B4GC
FP8 Format
FP8 Format
HPS Format
Figure 9: MSE for the activations of 1 15th block in the Llama-3. 3.1-8B I-strruct model.
22

---

Published as a conference paper at ICLR 2026
Laplace
Weights
Activations
0.020
MSE
0.010
MSE
top
0,002
E8M0
S= F4M3
HT
16
64
128
8
16
32
64
128
16
32
64
128
Group size G
Group size G
Group size
Figure 10: The effect of Hadamard Transform (HT) on MXINT4 (E8M0) and NVINT4 (E4M3) quantization on Laplace distribution samples and Llama-3, various group sizes.
weights and activations for
One can observe that the E4M3 and ESMO scales are not optimal for weight scale quantization. E4M3 and E8MO increase MSE by 10%, 40% on average, respectively. At the same time, FP8 options with larger mantissa (EIM6-F3M4)
well INT8 perform close to FP4 without scale quantization. The pattern for activation pattern is similar except for the case of down_proo in feedforward layer, which is known have more heavy-tailed distribution with pronounced outliers. We note that the observed behavior generalizes to other models considered in our study.
D INT4-BASED MICROSCALING FORMATS
To bridge this analysis gap and inform future hardware designs, we now present new analysis of hypothetical microscaling INT4 formats.
Since INT4 is not a part of the "OCP Microscaling Formats (MX) Specification" [47], we define it ourselves as follows:
1. We define INT4 base data type uniform symmetric grid of 15 (to match FP4) elements:
[-7, -6, -5, -, -3, -2, -1, 0, +1, +2, +3, + +4, +5, +6, +7].
We define NVINT4 as a microscaling format with E4M3 shared scales for groups of 16 elements (same as NVFP4) over the INT4 base element data type.
3. We define MXINT4 microscaling format with ESMO shared scales for groups of 32 elements (same as MXFP4) over the INT4 base element data type.
Applying the error analysis from Section these formats, as demonstrated in Figure 10, reveals that NVINT4 performs close to NVFP4 without the normalizing transforms. Yet. the proposed group Hadamard Transform has positive effect it (as opposed negative for NVFP4), making micro- rotated NVINT4 the most accurate of all the analyzed formats. MXINT4, however, performs poorly, and the normalizing transforms yield limited improvement. These findings extrapolate seamlessly to evalutations in Tables 1,3,4 and 5, confirming the usefullness of the Hadamard Transform and the superiority of the micro-rotated NVINT4 format.
OUTLIERS ANALYSIS
Proof of Lemma 1. Let U vā H be the normalized Hadamard matrix. U is orthogonal (UT U IG). The error vectors related by e. T
-T UT v) UT ey. Since U
23

---

Published as a conference paper at ICLR 2026
orthogonal, it preserves the Euclidean norm: Squared Error (MSE) is defined as:
|| ll2
|
(U'eyl1
-1Eyll2-
The per-element Mean
1 T
1 G
MSE(G)
Bllle
Bl
12
This establishes the second equality.
To prove the first, we rely on the standard assumption in quantization analysis that the quantization error By is statistically independent of the signal y. Since and y are related by the invertible transformation
U y. By is also independent of Consequently, the reconstruction error ex - Ey is also going be independent of xr
The index I, = arg max; | a function of x. Therefore, the error vector ez (and its components) is independent of the random index I.. Further, since the coordinates of i.i.d., we can apply symmetry to obtain that the probability that any coordinate has the largest magnitude uniform: P(L. = i) 1/G.
calculate the Top-Element MSE using the Law of Total Expectation:
MSEoop (G) B[(ex)]. )7.1
G
E((ex)). -- i]plL, =
[((()) 1. = 1
Because (Ex)? is independent of the event {T. i}, the conditional expectation simplifies to E((E.); [sx))) Substituting yields:
G
MSLRrop (G)
E
G
t=1
E
(Ew)²
(by linearity of expectation)
MSE(G).
This completes the proof.
Lemma 3 (Outliers MAPE). Let distribution be mix of two distributions: Noase and Nonttiers with portions 1 p and p such that:
1. mint((outtieral) > max((Xnase))
2. MSE (X- ~ X1, ~ Nouttiers) = MSErel (X ~ X1Xx, ~ Xrase),
3. P-G & MSE (X).
Then the expected outlier relative quadratic error equals MSF.el (X) up to O(pG):
8x,
A Xx
X
E Ex
MSE (X X))
Xx,xdauttioa
24

---

Published as a conference paper at ICLR 2026
Proof. We expand the expectation conditioned on X1, Routliers
(x) Xf
i= XX
Ex^x
Ei Ax
[X N1,-8,17 8,,) + Lift, AXeNNouttiers X7.
(x, (x)
x3
Ex N1,, Nontierra
1 Eitl. XXexRouttiers
(X1, 81,)² X7.
= Ex [Xr, Nouttierr
+ +O(G)).
By Assumption 2 this conditional expectation equals MSE (X), up O(pG) from Assumption 3. Hence the claim follows.
Discussion, Assumption satisfied for outliers chosen by absolute value thresholds. Assumption 2 holds for floating-point quantization due to constant relative accuracy (no overflow/underflow) verified in Section 3.2. Assumption holds in practice for LLMs since outliers are typically about 0.1% of elements [12].
Lemma 4 (Consistency of MSE for smooth distributions). Let X be distribution of values to quantize with a power-of-two translation-imvariant quantization function
Q : Vx R4, Bk. Z: Qx 2*) 2 (x)).
Assume:
1. supp X' C [2 2 for integers < b.
2. Vo E supp X, Vy [x/ 22, x. : - fx(u)) Sa a
3. (a-Q(x))²
MSE max:
Then
(x Q(x)) 2 (x Q(x))2
dr 0(12--200 MSE a
Exax
x2
x²
Proof. We decompose the expectation over dyadic intervals:
(x - -(x))))
(x Q(x)) fx(x) dz.
E
E
x²
x2
ima
Within each interval, write fx(x) = fx(2') + (Jx(x) Sx(2')) The first term yields
{ (x (x=)(())²
6-1
2 dr Σ (x(2)
The second term is bounded using Assumption 2 and 3, giving
MSErel MAxE O(a) dx = (2' - 24) MSE O(a).
Finally, the normalization error in the discrete approximation of f fx contributes an additional O(a) factor. Combining terms gives the stated result.
Discussion. Assumptions and hold for absmax Xr, quantization since floating-point values bounded with bounded relative error. Assumption is supported empirically (Figure 4), where seale distributions observed to be smooth.
25

---

Published as a conference paper at ICLR 2026
MXF14 Weight Shape: 5120 (OUTxIN)
Owen3-6B End fo-Endd Profill specdiup MXFFAA vs. BF16 KTX5090
Torch BF16
Sequeece Largth
Ideal W4A4
TFLONS/ RTX3080
(no quant)
2.3
Actual W4A---Hall Had52
X03
Actial
W4A--HHad128
230
0.1
Balih stot
20
Figure 11: Illustration of QuTLASS performance for weights and activations on MXFP4 while increasing batch size, for single linear LLM layer, showing the low-overhead of the quantization- related ops, and end-to-end using the Transformers library.
15000
Throughput, tok/s
12500
Speedup over BF16
3.5
Latency, s
10000
1.5
7500
1.0
BFL6
BF16
BF16
5000
F78
MIR-NVTM
MR-NVVP4
0.5
MR-NVEPM
MR. MXFP4
2500
MR MXFTN
ME MXEIN
NVEP4
NVPP4
2.0
0.0
16"
10'
10
500
1000
Batch Size
Batch Size
Batch Size
Figure 12: End-to-end speedups for 1lamm-3.33-7OB 3.ssrot- The gray vertical line roughly separates small-batch inference (which covers single-user text generation) from large-batch inference (which includes prefill).
F QUTLASS RESULTS ON GEFORCE GPUs
Figure 11 illustrates additional QuTLASS performance results NVIDIA RTX5090 GPU. The figure on the left shows throughput for single layer extracted from MXPF4 quantized Qwen3-32B model, while the figure on the right shows the end-to-end speedups on Transformers rumning Qwen3- 8B with MXFP4 quantization compared to the BF16 baseline implementation on single RTX5090 GPU.
G QUTLASS SPEEDUPS IN VARIOUS INFERENCE REGIMES
Figure 12 demonstrates end-to-end inference speedups of FP8 and various FP4 formats relative to BF16 for small and large batch sizes for 3.ama-3.3-70BIIne 3.3-700- rumning single NVIDIA B200 GPU in vLLM.
For large batch workloads, it is easy to see how our Micro- Rotat-d (MR) kernels outperform both BF16 and FP8, with MR-MXFP4 providing the highest throughput around 15,000 tok/sec: 2.2x increase over BF16 and 1.3x increase over FP8.
These kernels, however, were optimized for small batch workloads, where they show little to no improvement over BF16 and FP8. In that regime, which is characterized by memory-bound inference, it is preferable to weight-only quantization, benefits to inference speed. We present accuracy measurements of weight-only quantized models in Appendix A. In Figure 12, present latency and throughput measurements for weight-only (WO)
activation quantization brings
26

---

Published as a conference paper at ICLR 2026
micro-rotated FP4 quantization scheme, which is already supported in vLLM. One that it shows approximately 20% lower latency than FP8 for small batch size inference.
H MXFP SCALE FITTING
The principal cause of the poor performance of MXFP quantization is the large-scale quantization error. On the one hand, the ESMO format allows representing extremely small values, such as 2-127 127 and extremely large values, such as 2 On the other, shown in Figure 4, the actual range of weights and activations is much more narrow, making the ESMO grid too coarse. To address this mismatch, we propose a simple modification: fit the quantization grid to the data range.
The original MXFP quantization grid, with 4/3 re-scaling, quantizes scales follows:
SESMO = (4/3) gchunp(ronud((og, -),-)),,...))) (12,,128)))
(I)
One can estimate the smallest Smin and the largest Smax values in given tensors, such that the smallest value is mapped to - 1288, and the largest to 127:
$gsM0 (log2 Smax logg Amin /clamp(round 255 long amax long Amin
"min
0,255))+lgg emin
(2)
The modification reduces the scale-quantization error, by rescaling the exponent. In effect, we replace the exponent by value 2 (0, followed by rescaling:
200+
(3)
We provide the comparision between the original MXFP and modified version, called MXFP4 in Table 11. In almost every setting, MXFP4 yields substantial performance boost relative to vanilla MXFP4 and performs close NVFP4. One should note that MXFP4 requires 4.25 bit per parameter compared to 4.5 for NVFP.
Method
Format
Llama3 (8B)
Qwen3 (8B)
MXFP4
87.8
93.7
RTN
MXFP4
94.3 (+6.5)
96.3 (+2.6)
NVFP4
94.7
98.9
MXFP4
89.2
93.6
RTN + HT
MXFP4
93.9 (+4.7)
96.3 (+2.7)
NVFP4
93.8
96.0
MXFP4
89.5
94.1
GPTQ
MXFP4
95.2 (+5.7)
92.3 (-1.8)
NVFP4
95.7
98.1
MXFP4
93.6
95.2
MR-GPTQ
MXFP4t
94.9 (+1.3)
98.5 (+3.3)
NVFP4
95.8
97.4
Table 11: Per-model recoveries with vanilla MXFP4 format and the proposed modification denoted by MXFP41
Efficient arithmetic with MXFP scales requires that the weight and activation tensors in a layer share the same exponent. Consequently, the product of the activation scale S and the weight scale sw be expressed as:
-AA.AA-A +Aw = 2 + +ataaaatat
In the above expression, 0.A and a have to the same for one to express scale multiplication in terms of power addition.
I COMPARISON BETWEEN VARIANTS ON THE PLATINUM BENCHMARK
Next, we perform a detailed analysis between different versions of the GPTQ algorithm, on the two formats. As visible in Table the standard evaluation harness struggles to distinguish variants, likely
27

---

Published as a conference paper at ICLR 2026
because of high noise in some of the evaluations. To address this, examine the differences between GPTQ variants on the less noisy PlatinumBBnnh benchmark [53], which includes carefully curated tasks and questions. These experiments are performed with "real" kernels, via our vLLM integration. Figure 13 include full results across tasks, for the following variants:
Transform matrices: Hadamard with different block sizes, denoted by e.g. Had128.
Scale optimization: Our approach (MSE) or the default (MinMax).
Quantization ordering: Static Activation Ordering (ActOrder) or arbitrary/initial (Default).
Mesiledbe Seleed
60C
604
(a) NVFP4 accuracy across GPTQ variants.
(b) MXFP4 accuracy across GPTQ variants.
(c) NVFP4 averages and standard deviations.
(d) MXFP4 averages and standard deviations.
Comparison of NVFP4 and MXFP4 quantization methods Platinum benchmark tasks. Top row shows recovery results across different GPTQ/MR-GPTQ component combinations. Bottom row shows average recovery scores and standard deviations for each method.
Figure 13:
Discussion. We observe the following:
The results suggest that Hadamard rotations provide a statistically-sinniiie. advantage to our MR-GPTQ variant, with group-aligned Hadmard rotations (Had16), MSE and ActOrder, in the NVFP4 case as well. All other variants appear to be within variance of eachother on this benchmark, for NVFP4.
We observe large gap (> 4 points on average) between the top NVFP4 recovery (96.6%) and the top MXFP4 recovery (92.3%).
Finally, the MXFP4 results show very large recovery gap between the variants with rotations and the variants without. Moreover, for MXFP4, larger Hadamard rotations (128 vs 32) appear clearly help, whereas, for NVFP4, matching the rotation size the group size appears ideal.
J STANDARD DEVIATION
We estimate the variance of evaluation scores by performing multiple quantization runs on Llama- Instruutt varying the seeds for GPTQ calibration set sampling, as well as the strategies for scale selection and quantization ordering. These results were generated using our vLLM integration with QuTSASS kernels. Figure 14 displays the scores as bar plots, while Table lists the average recovery scores and their standard deviations.
Additionally, we report the average recovery scores and their standard deviations for the Platinum benchmark suite [53] in Figure 17 and Table 12, We also report per-task recovery (%) in Figure 15.
28

---

Published as a conference paper at ICLR 2026
NVFP Recovery by Method
97.0
96.5
Recovery %
95.0
Had16 Hod16-Minhaa Haodc-eeihhac AccOrder
Had16+ NSE AAcorddr
Hadl 28- Mi-MMMMMaa Aeddder
Had128 USE AccOOder
Idert= AccOrder Accorde
Ident Hccdrdr
Idant Idant- Mini--MiMaxxccc DefOrder
Ident MSE DefOrder
MXFP Recovery % by Method
Recovey
90
89
Had MinMax MMn22- Actorder
Had32+NGE Actorder
Hadf MinMaMiiaaaccc Artarddr
Had128 MSE ActOrder
Ident MinMac Mdintt+MinMax ActOrder
Ident HSE Actider
Ident MinMarMMMoaaa Duffrder
Ident++ MAAASS-DDDeeot DetOrddr
Figure 14: Accuracy results for NVFP4 and MXFP4 across different combinations of MR-GPTQ
components, averaged over five random seeds using vLLM kernels on the benchmark suite.
29

---

Published as a conference paper at ICLR 2026
Figure 15: Aceuracy results for MXFP4 across different MR- MR-GPTQ component combinations on the
Platinum benchmark tasks.
Platitum Bench Recovery (mean %)
Hod122-MMiiMixxxeeeedddde
76.4
Hudl 28-MSE ArtOrdrr
99.0
98.0
97.3
93.9
93.5
75.3
88.1
7%.6
91.3
86.2
99.7
100.2
95.3
91.8
92.1
Hhdf2--MiiMaxx<Arrrhee
99.9
98.8
96.4
92.9
92.2
74.3
91.4
77.8
89.7
90.3
98.6
95.55
97.4
89.1
91.7
HadS2-MEE ActOrder
99.7
98.5
96.1
95.2
93.1
76.5
91.5
76.1
88.3
78.5
95.0
95.33
94.0
90.6
90.6
Mem--MinMMxxtArrrheea
97.9
98.0
94.6
92.4
89.7
67.0
82.3
73.2
86.0
67.4
90.4
93.5
85.2
85.4
85.9
97..0
95.
84.4
63.9
83.2
92.
92.2
88.3
Ideni-MinMax DetOrder
97.2
87.4
66.4
79.1
69.7
88.5
88.3
84.5
83
84,9
SinglsO3
Single
MaltAAith
SVAMP
OSMSK
OoH-LagiaaDDdorrti Dedictinndddiin..... [Ch)
MMLtUMMbb
BBIl- Ofjecff wamting
BBil-Nadgatt
TahFact
HopaaOA
SQUAD
DROP
Winugrad WSC
Aecrage
Figure 16: Accuracy results for NVFP4 across different MR-GPTQ component combinations on the Platinum benchmark tasks.
Platinum Bench Recovery (mean %)
Had16-MSE ActtOrree
100.1
90.5
102.3
Ident MinMax DefOrder
99.9
99.8
97.0
97.3
83.7
92.8
91.1
92.5
99.1
97.4
97.6
90.0
94.2
Ideut- DDDt-MMSEDeefRdddrr
99.7
99.4
96.8
95.2
97.6
85.9
96.3
84.7
95.9
78.6
101.1
96.6
95.6
92 8
94.0
Hadd 28 MSE AcctOrder
100.4
.9.4
96.9
98.3
98.3
86.4
89.0
86.1
96.1
85.2
95.9
96.0
94.6
92.6
94.0
Hadi G-MMiMMax+Asttee
99.9
98.6
97.3
96.8
98.2
85.6
94.11
85.0
97.77
75.2
96.9
101.8
96.9
88.8
93.8
Mene-MMsEEAArredder
100.1
99..3
97.6
95.7
97.0
89.1
95.0
82.2
92..3
83.3
96
96.3
92.5
92.8
93.5
Men--MinMaaxAArrrhaad
100.2
99.8
97.6
96.8
83.8
94.2
83.4
91.8
R5.5
95.7
97.2
92.3
91.6
93.3
SingleOp
S
MaltAAith
SNAMP
aSMaK
BBH Law.-LLogoDDDleeeeeeeee Ded.eaaiiion--- 10hg
M-Mals
ba Chfeco camttig
BBrL-Nadger Nargaee
TakePadt
SQaAD
DROP
Wanogradh WeC
Avesage
30

---

Published as a conference paper at ICLR 2026
Recovery
97
93
94
91
Holle-ASD: POdi--AD--A AurOhdrr
Hol MSSL Aannda
Heal M Anthrder
MatrsM ActDndr
Had MalleeMaaaarttrtttt
Idmi MannAdaalar AnDuler
MXFP Recovery % by Method
94.
92.
Roconayy
88
34.
Hald He Metdar Antirder
Heul Malll-MaMooAnatne Antiide
Hall2Mag- ArtCrde
Idiaa MinMar Antirdar
Mele MansMMB--AAttRe Anttide
IdmrMta- Mam-MGB-Dufiie DufDddrr
Home MintMMaMaa--iii Dont Duftireer
Figure 17: Average recovery scores and standard deviations for NVEP and MXFP methods on the
Platinum benchmarks.
31

---

Published as a conference paper at ICLR 2026
Format
Method
Standard Bench %
STD
Platinum Bench %
STD
Had
Hadl66--iiMax-ActtOdrr
95,88
0.332
93.77
1.680
Hodl
96.33
0.163
96.57
0.746
95.52
0.416
93,42
1.618
Had/285+MSSE4ActtOrdddd
96.11
0.347
93.95
1.143
NVFP
ldent-MinMMax++AcOOOddr
95.84
0.487
93.27
1.263
Ident-MSEE+ActOdderr
96.18
0.589
93.51
1.304
ldcot-MinnMaxx+Deefrdder
96.06
0.655
94.20
1.358
Ident-MSE-DDeOrdderr
96.38
0.441
94.01
1.053
Had32--MimMax-AcrOOdrr
92.79
0.554
91.73
1.183
Had32-+MSF=Acctrdder
93.78
0.445
90.51
2.294
93.42
0.416
92.32
1.128
Hadl:25-MMSE5AAcrOrddr
93.63
0.817
91.86
0.743
MXFP
Idenn+MiniMax++4<(Ordderr
89.78
0.570
85,93
1.459
Ident-MSSE+ActOrdeer
90.54
0.330
85,26
1.112
ldenttMMinMaxx-DerOrdderr
89.16
0.372
84.85
1.95
Ident-MSEE-DelOrderr
90.02
0.387
85.21
0.798
Table 12: Average Recovery scores and standard deviations for NVFP and MXFP methods across the Standard benchmark and Platinum benchmark.
32

---

Published as a conference paper at ICLR 2026
K
THE EFFECT OF DIFFERENT LINEAR TRANSFORMS
In this section we ablate various choices of transforms adopted for outlier mitigation of outliers. Specifically, we consider the following options:
Identity transform.
Discrete Cosine Transform (DCT).
Hadamard rotation [50; 4; 11].
Grouped Sequeney-arranged Rotation (GSR) [8].
We sweep over different options of transform sizes ([16, 32, 64, 128, 256}) both for NVFP and MXFP formats. The average score on tasks from LM Evaluation Harness (piqa, winogrande, hellaswag, arc-easy, arc-challenge) is reported.
From these results, one can observe that rotations yield small improvement relative to identity transform for MXFP format and minor degradation for NVFP with RTN quantization. Different transform sizes perform more less the same.
33

---

Published as a conference paper at ICLR 2026
Transformation
Trunsformation Size
Weight Quant
PIQA
winogrande
hellaswag
arc-casy
are-challenge
Avg
FP16
0.8074
0.7301
0.792
0.7769
0,5307
0.7274
RTN
0.802
0.7261
0.7731
0.7466
0,4923
0.708
GPTQ
0.7933
0.7214
0.7698
0.7664
0.5111
0.7124
RTN
0.6859
0.7583
0.742
0.4889
0.693
16
GPTQ
0.7824
0.7111
0.766
0.7559
0.4991
0.7029
RTN
0.7786
0.7119
0.7572
0.7353
0.4693
0.6905
GPTQ
0.7813
0.7135
0.7718
0.7117
0.4829
0.6922
RTN
0.7862
0.7024
0.7695
0.7306
0.4599
0.6897
DCT
64
GPTQ
0.7878
0.7198
0.7673
0.7765
0.5068
0.7116
RTN
0.7737
0.7206
0,7676
0.7466
0.4701
0.6957
128
GPTQ
0.7873
0.708
0,7715
0.7399
0.494
0.7001
RTN
0.7916
0.7135
0.7698
0.7563
0.4983
0.7059
GPTQ
0.7911
0.7017
0.7692
0.7694
0.506
0.7074
RTN
0.7824
0.7143
0.7575
0.7256
0.4804
0.692
16
GPTQ
0.7878
0.7198
0.7628
0.7395
0.4855
0.6991
RTN
0.7856
0.7198
0.7399
0.7395
0.4667
0.6903
GPTQ
0.7889
0.7096
0,7633
0.7731
0.5026
0.7075
RTN
0.7911
0.7253
0.7536
0.7635
0.4804
0.7028
DST
GPTQ
0.7911
0.7088
0.7638
0.7614
0.5
0.705
RTN
0.7856
0.7024
0.7625
0.7677
0.4881
0.7013
128
GPTQ
0.7824
0.7064
0.7637
0.7778
0.5009
0.7062
RTN
0.7867
0.6993
0.7579
0.737
0.4804
0.6923
GPTQ
0.7856
0.7048
0.7674
0.7462
0,4812
0.6971
RTN
0.7927
0.7096
0.7674
0.7471
0.465
0.6963
16
GPTQ
0.7873
0.7096
0.7697
0.5034
0.7056
RTN
0.784
0.719
0.7639
0.7534
0.4881
0.7017
GPTQ
0.7965
0.7348
0.7668
0.7538
0.506
0.71 16
RTN
0.7818
0.7032
0.763
0.7395
0.4863
0.6948
Hadamard
GPTQ
0.7856
0.7151
0.7657
0.7614
0.5017
0.7059
RTN
0.7884
0.7206
0.766
0.7551
0.506
0.7072
128
GPTQ
0.7938
0.7111
0.7729
0.7681
0.5273
0.7146
RTN
0.7878
0.6969
0.7643
0.7681
0.4983
0.7031
GPTQ
0.79
0.7253
0.7738
0.7673
0.4949
0.7102
RTN
0.7056
0.7694
0.7513
0.4744
0.6988
16
GPTQ
0.7998
0.6985
0.7683
0.7635
0.4863
0.7033
RTN
0.7873
0.6985
0.762
0.7702
0.494
0.7024
GPTQ
0.79
0.7214
0.77
0.7593
0.5
0.7081
RTN
0.7911
0.7151
0.7627
0.7588
0.4821
0.702
GSR
GPTQ
0.796
0.7222
0.7717
0.7622
0.4949
0.7094
RTN
0.7878
0.7174
0.7656
0.7546
0.4898
0.703
128
GPTQ
0.7894
0.7143
0.7721
0.7668
0.506
0.7097
RTN 0.7797
0.6906
0.7626
0.7454
0.4735
0.6904
GPTQ
0,8014
0.7293
0.7756
0.763
0,4991
0.7137
Table 13: Performance of Llama-3-8B with different transformations with NVFP4 format.
34

---

Published as a conference paper at ICLR 2026
Transformation
Trunsformation Size
Weight Quant
PIQA
winogrande
hellaswag
arc-casy
are-challenge
Avg
FP16
0.8074
0.7301
0.792
0.7769
0,5307
0.7274
RTN
0.7704
0.6875
0.7481
0.7121
0,471
0.6778
GPTQ
0.7699
0.693
0.753
0.7327
0.4718
0.6841
RTN
0.7628
0.7072
0.7447
0.7205
0.4582
0.6787
16
GPTQ
0.7753
0.7009
0.7534
0.7365
0.4846
0.6902
RTN
0.7699
0.6914
0.7405
0.6987
0.4437
0.6688
GPTQ
0.7508
0.6969
0.7465
0.7281
0.4531
0.6751
RTN
0.7693
0.7127
0.7454
0.7079
0.4556
0.6782
DCT
64
GPTQ
0.7889
0.7111
0.7524
0.7529
0.465
0.6941
RTN
0.7541
0.6851
0.7398
0.6616
0.4036
0.6488
128
GPTQ
0.7731
0.7088
0.7455
0,7462
0 0.4804
0.6908
RTN
0.7791
0.6953
0.7392
0.6987
0.4411
0.6707
GPTQ
0.7894
0.6946
0.7541
0.7504
0.4744
0.6926
RTN
0.7731
0.6906
0.7493
0.7391
0,4522
0.6809
16
GPTQ
0.7835
0.6898
0.7593
0.7399
0.4787
0.6902
RTN
0.7639
0.6906
0.7441
0.7332
0.4582
0.678
GPTQ
0.7802
0.6985
0.7563
0.7483
0.4753
0.6917
RTN
0.7704
0.689
0.7402
0.7054
0.4599
0.673
DST
GPTQ
0.7769
0.6875
0.7599
0.7189
0.4693
0.6825
RTN
0.7612
0.6772
0.7491
0.7003
0.4497
0.6675
128
GPTQ
0.7693
0.6914
0.7567
0.7462
0.1923
0.6912
RTN
0.7731
0.6906
0.7493
0.7391
0.4522
0.6809
GPTQ
0.778
0.7064
0.7544
0.7412
0.4923
0.6945
RTN
0.7737
0.6906
0.7499
0.4616
0.675
16
GPTQ
0.7867
0.7206
0.7623
0.7218
0.4701
0.6923
RTN
0.7715
0.6946
0,7518
0.7466
0.5034
0.6936
GPTQ
0.7807
0.7032
0.763
0.7471
0.4778
0.6944
RTN
0.7862
0.7088
0.7511
0.7315
0.4667
0.6889
Hadamard
GPTQ
0.796
0.6993
0.7625
0.7635
0.4923
0.7027
RTN
0.7807
0.7064
0.7529
0.7306
0.4548
0.6851
128
GPTQ
0.7807
0.6946
0.7646
0.7538
0.4915
0.697
RTN
0.778
0.7024
0.7491
0.7104
0,4625
0.6805
GPTQ
0.7818
0.7032
0.7624
0.7576
0.4795
0.6969
RTN
0.7813
0.6977
0.7522
0.6982
0.4684
0.6796
16
GPTQ
0.7845
0.7048
0.7682
0.7546
0.4735
0.6971
RTN
0.7748
0.693
0.7514
0,742
0.4991
0.6921
GPTQ
0.7856
0.7111
0.7631
0.7517
0.5026
0.7028
RTN
0.7889
0.7072
0.7464
0.7226
0.4514
0.6833
GSR
GPTQ
0. 0.7949
0.7009
0.7613
0.7412
0.4829
0.6962
RTN
0.7753
0.7001
0.7538
0.7226
0.4659
0.6835
128
GPTQ
0.7813
0.7056
0.7595
0.4846
0.6941
RTN
0.7753
0.6819
0.7494
0.7197
0.4642
0.6781
GPTQ
0.778
0.7151
0.7542
0.7445
0,4923
0.6968
Table 14: Performance of Llama-3-8B with different transformations with MXFP4 format.
35


Reviews:
[{"review_id": "O0D3aqkTsL", "reviewer": "Reviewer_2iLw", "summary": "This paper analyzes the quantization error in the NVFP4 and MXFP4 formats and shows that rotation-based quantization could help MXFP4 but hurts the accuracy of NVFP4 when using round-to-nearest quantization. A new quantization method, MR-GPTQ, is proposed to use block-wise Hadamard micro-rotations, static activation reordering, and format-specific optimizations to improve the accuracy of both NVFP4 and MXFP4. The evaluation results show that MR-GPTQ could achieve better accuracy recovery than prior art. The performance results show the overhead introduced by MR-GPTQ is very small, and 2.2x and 4x end-to-end speedup is demonstrated on server-grade (B200) and consumer-grade GPUs (RTX5090).", "questions": "- How does the proposed method compare to FP8 and INT8 in quality and end-to-end performance?\n- There is a gap between single-layer speedup and end-to-end speedup. Could you provide some analysis of this? Also, in Figure 5 (right), why is the speedup of FP4 low in a small batch size? I think attention is not a bottleneck here since the sequence length is not very long.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A good theoretical analysis and empirical validation of quantization error on the latest quantization schemas (NVFP4 and MXFP4).\n- A good kernel implementation that minimizes the overhead of the proposed quantization method and demonstrates strong end-to-end speedup on cutting-edge GPUs.", "weaknesses": "- It would be helpful if the proposed method could be compared against existing quantization methods, FP8 and INT8, which is already used in many production situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tUqehn5Pmf", "reviewer": "Reviewer_34aN", "summary": "This paper proposes MR-GPTQ (Micro-Rotated GPTQ), a post-training quantization scheme tailored for FP4 formats, aiming to overcome the limited accuracy recovery observed when applying existing PTQ methods to FP4-quantized LLMs.\n\nAlthough recent support for MXFP4 and NVFP4 on Blackwell-class GPUs enables efficient 4-bit inference, the authors show that conventional INT4-oriented techniques do not transfer well to FP4 due to format-specific constraints—most notably, NVFP4’s small group size and MXFP4’s power-of-two scale quantization (E8M0). Moreover, contrary to observations in prior low-bit quantization work, the authors find that Hadamard-based rotations can degrade MSE in NVFP4, exposing a mismatch between FP4 properties and existing rotation-based normalization strategies.\n\nTo address these issues, the authors present MR-GPTQ, which adapts GPTQ’s second-order update mechanism to FP4 via block-wise Hadamard rotation, format-aware scale optimization, static activation reordering, and fused GPU kernels (QuTLASS) optimized for Blackwell.\n\nExperimental results on Llama-3 and Qwen models show that NVFP4 combined with MR-GPTQ recovers approximately 98–99% of FP16 accuracy, while MXFP4—despite its inherently larger quantization error—benefits substantially and approaches NVFP4-level performance. In addition, end-to-end inference achieves 2.2×–4× speedups on B200 and RTX5090 GPUs, demonstrating that FP4 can deliver strong accuracy–performance trade-offs when paired with format-specialized PTQ.", "questions": "If the authors could clarify or further substantiate the major weaknesses mentioned above, I would be happy to reconsider and potentially raise my evaluation. I would particularly appreciate further clarification on these major concerns.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Provides a systematic and quantitative analysis of the MXFP4/NVFP4 formats, clearly identifying the root causes of accuracy degradation in FP4 PTQ.\n* Demonstrates, both empirically and analytically, that rotation-based PTQ techniques are not consistently effective under FP4, motivating the need for format-aware methods.\n* Proposes MR-GPTQ, an FP4-specialized variant of GPTQ, which delivers meaningful accuracy improvements by explicitly accounting for FP4 constraints.\n* Introduces Blackwell-optimized QuTLASS kernels that minimize the runtime overhead of micro-rotations, yielding practical performance gains (2.2×–4× end-to-end).\n* Presents a well-structured pipeline—from analysis to algorithm design, kernel implementation, and evaluation—resulting in a cohesive and well-justified contribution.", "weaknesses": "### Major Weaknesses\n* Insufficient motivation for choosing FP4 over INT4 (potential logical gap):\n    * The paper adopts FP4 (MXFP4/NVFP4) as the primary target format, yet it does not clearly articulate why FP4 should be preferred over INT4 in PTQ scenarios.\n    * A more principled comparison—including distributional properties, quantization behavior, and rotation sensitivity—would help substantiate the motivation for focusing on FP4.\n    * If strong INT4 + rotation baselines (e.g., QuaRot/QuIP) can achieve performance comparable to or better than FP4 + MR-GPTQ, the necessity and relative advantage of the proposed approach may become less evident.\n* Limited justification and characterization of MR-GPTQ design choices:\n    * The paper introduces several components—format-aware scaling, static activation reordering, and block-wise rotation—but does not clearly explain why each is particularly necessary or effective for FP4.\n    * A more principled analysis that isolates the contribution of each component (e.g., through ablations or sensitivity) would clarify whether these choices are inherently beneficial, or simply empirical heuristics.\n    * It is also unclear under which conditions these components fail, making it difficult to assess the robustness and general applicability of the proposed design.\n\n### Minor Weaknesses\n* The source of FP4 rotation degradation is not clearly isolated; the paper does not distinguish format-intrinsic effects from configuration factors (e.g., block size, scaling, calibration), making the necessity of the proposed approach less firmly supported.\n* The observed superiority of NVFP4 over MXFP4 is largely expected from format specifications (E4M3 vs. E8M0), so the empirical confirmation offers limited new insight.\n* The method primarily applies heuristic extensions to GPTQ without demonstrating a substantial methodological advance, which may make the contribution appear incremental rather than novel.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QA0xkYAZjV", "reviewer": "Reviewer_KK3i", "summary": "This paper systematically explores the accuracy–efficiency trade-offs of 4-bit floating-point formats (MXFP4, NVFP4) and introduces QuTLASS, a GPU kernel template that integrates the Hadamard transform at effectively zero additional overhead. The study also examines how INT-family quantization methods interact with FP4 to assess practical benefits for real-world inference.", "questions": "- With FP8 (E4M3/E5M2) included as a baseline under identical tuning, how large and consistent is the gap to FP4 across tasks?\n- What is the minimal reproducible setup for MR-GPTQ (GPU count, wall-time, memory)? How does cost scale for small vs. large models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well structured and easy to follow.\n- Broad empirical exploration of combining INT quantization methods with FP4 (MXFP4/NVFP4).\n- Introduction of QuTLASS, which integrates the Hadamard transform with minimal overhead.", "weaknesses": "- The text states FP8 is “near-lossless,” yet Table 1 lacks an FP8 baseline (e.g., E4M3/E5M2). Including it under identical conditions would anchor FP4’s position more convincingly.\n- Although MR-GPTQ runs offline and does not affect inference latency directly, reproducibility details are missing (GPU type/count, hours). Please add a minimal, reproducible configuration.\n- (Minor) L238: typo “teh” → “the.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTKII6v5Fi", "reviewer": "Reviewer_WXdo", "summary": "This paper provides the first comprehensive study of FP4 microscaling floating-point formats—MXFP4 and NVFP4—used in recent NVIDIA and AMD GPUs for LLM quantization. The authors analyze why these formats underperform despite their hardware support: (1) NVFP4’s small group size limits outlier mitigation, and (2) MXFP4’s power-of-two scale quantization causes high quantization error. To address this, they introduce Micro-Rotated GPTQ (MR-GPTQ), a GPTQ variant tailored to FP4, using block-wise Hadamard rotations, MSE-optimized scales, and static activation reordering. They implement this through QuTLASS, a GPU kernel suite supporting fused online rotations with negligible overhead. Experiments on Llama and Qwen models show that MR-GPTQ significantly improves MXFP4 accuracy (closing most of the gap to NVFP4) and achieves up to 3.6×–6× layerwise and 2.2×–4× end-to-end speedups on Blackwell GPUs. The paper concludes that FP4 is not inherently superior to INT4, but format-specific optimization can unlock its full potential.", "questions": "The current speed evaluation focuses mainly on end-to-end throughput. Is there any latency measurements for the prefill and decode stages, respectively?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles an important and emerging problem—quantization under new FP4 microscaling formats (MXFP4 and NVFP4)—that directly relates to current-generation GPU hardware (NVIDIA Blackwell, AMD CDNA4). This makes the work highly relevant for both research and deployment.\n2. The paper implements QuTLASS, optimized GPU kernels that fuse quantization, rotation, and matrix multiplication for FP4 inference. Integrated with the Blackwell tensor core pipeline, these kernels add almost no overhead, effectively bridging algorithm design and hardware deployment.\n3. The authors present a clear analytical treatment of quantization error for FP4, explaining the contrasting effects of rotations under Laplace vs. Normal distributions, and showing why existing PTQ methods (e.g., GPTQ, SmoothQuant) underperform on FP4 formats.", "weaknesses": "1. Despite the proposed improvements, FP4 quantization—especially with MXFP4—still suffers from noticeable accuracy degradation. Even with MR-GPTQ, the recovery remains 3–5 % below FP16 for many models, suggesting that FP4 formats are not yet competitive with INT4 or FP8 in accuracy.\n2. The methodological novelty is moderate, as MR-GPTQ mainly adapts the existing GPTQ framework to FP4 formats using known ideas such as block-wise rotations and scale optimization. However, the work shows strong engineering and system-level originality—it is the first to deeply study NVFP4 and MXFP4, propose an FP4-specific GPTQ variant, and implement efficient GPU kernels with near-zero overhead. Overall, the contribution is not conceptually new but is practically impactful and timely.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)