Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Authors: Roberto L. Castro
Andrei Panferov
Soroush Tabesh
Oliver Sieberling
Jiale Chen
Mahdi Nikdan
Saleh Ashkboos
Dan Alistarh
Venue: NeurIPS 2025 poster ()
Keywords: quantized training
large language models
Primary area: infrastructure
TL;DR: We provide a method for accurate end-to-end FP4 training of Large Language Models.

Abstract: Training large language models (LLMs) models directly in low-precision  offers a way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we  investigate hardware-supported FP4 training and introduce a new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy-vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-based training is a competitive alternative to FP16 half-precision and to FP8 training. Our code is available at https://github.com/IST-DASLab/Quartet .

Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Roberto L. Castro" ISTA & Red Hat AI
Andrei Panferov" ISTA
Soroush Tabesh
Oliver Sieberling ETH Zürich
ISTA
Jiale Chen ISTA
Mahdi Nikdan
Saleh Ashkboos
Dan Alistarh ISTA Red Hat AI
ISTA
ETH Zürich
Abstract
Training large language models (LLMs) models directly in low-precision offers way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on muxeedpprecisie FP4 training and introduce new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) low precision. Through extensive evaluations on Llama-type models, we reveal new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy- vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-hased training is a competitive alternative to FP16 half-pren and FP8 training. Our code available htttps://gitee com IST DwSSAITT-DDLLAAAaaa.aa
fallbacks. In this paper, we investigate sardpare-supported
Introduction
Over the past decade, the capabilities of large language models (LLMs) have surged, unlocking state-of-the-aar performance in Al reasoning, coding, and multimodal understanding. These advances have come at the cost of an unprecedented rise in compute costs, the floating- ppint operations (FLOPs) required to train a frontier model have been doubling every few months [16).
One key lever for reducing compute costs is lower-precision computation: executing the matrix- multiplication (MatMul) kernels that dominate training workloads at lower bit-widths yields near-linear gains in throughput and energy efficiency. On the inference side, it is known that 4-bit quantization-oo even lower-can preserve accuracy, via sophisticated calibration and rotation schemes 2 9. For training, recent work has pushed the precision frontier from FP16 33 to 8-bit pipelines, responsible in part for efficiency breakthroughs such as DeepSeek-V. V3 [31] In this context, NVIDIA's Blackwell architecture introduces efficient hardware support for even lower-precision microscaling formats [36] such as MXFP and NVFP, which natively support 4-bit floating-poii point operations at higher MiaFFLPP-per-war efficiency: for instance, moving from 8- to 4-bit multiplies on the B200 GPU can almost double arithmetic throughput, while cutting energy roughly in half [35).
Yet, today's s algorithmic support for accurate end-to-end training in such low precision is missing. State-of-the- quantized training methods such as Switchback [54) Jetfire [57] HALO 3 and
Equal contribution. Correspondence to: dan. alistarheist.ac.o
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

INT4-fransformers [56] either (i) lose precision and stability when training current models in 4-bit formats, or (ii) fall back to higher precision for selected matrix multiplications. Bridging this gap calls for both deeper understanding of quantization error during back paok-propngation and new algorithmic safeguards tailored to hardware-native FP4 formats.
Contributions. In this paper, we address this challenge via a first systematic study of hardware-sup hardware-supported FP4 training, focusing on the high-efficiency of the MXFP4 format [36] 35) Based on this analysis, we introduce an algorithm for MXFP4 native training-in which all matrix multiplications occur in MXFP4 called Quartet, which provides the best accuracy-effiiinny trade- off among existing methods, and is near-lossless for LLM pre-training in the large-data regime. Our main technical contribution speedups of almost 2x relative to FP8 for linear layer computations on an NVIDIA Blackwell RTX 5090 GPU. One key achievement is that Quartet enables MXFP4 precision to be "optimal" on the accuracy-efficiency trade-off: at fixed computational budget, the accuracy impact of lower-precision training in Quartet is fully compensated by the higher efficiency of our implementation. In more detail, our contributions are as follows:
highly-efficient GPU implementation of Quartet, which achieves
1. We propose and implement a new approach for comparing quantized training methods, via their induced scaling law, which dictates the loss achievable under specific computation and data budget. We propose and fit such a law for all existing methods, isolating two key parameters: the parameter efficiency e of each method, and its data efficiency effp. A method superior to another if it improves upon both these metrics.
2. We find that the parameter efficiency is directly linked to the forward compression error of each training method, whereas data efficiency is linked to the bias in the method's gradient estimator, which we measure via novel misalignment metric. Given computational and data budget, and real-world speedups due to lower precision, these metrics allow us to predict the "optimal" low-precision setup to train given model to a target accuracy, maximizing accuracy-vs-runting
3. We apply this framework to MXFP4 precision, seeking to determine if there are practical settings under which native training in this precision can be optimal on Blackwell GPUs We isolate an algorithm, called Quartet, which achieves this by maximizing both parameter and data efficiency, building on previous SOTA methods for QAT [37] and quantized backward-pad optimization [47]. Our key technical contribution is complex, highly eefficeeffiiient GPU implementation of Quartet specialized to the new Blackwell architecture.
4. We validate our approach experimentally by pre-training Llama-family [46 models on the C4 dataset (39. Our experiments show that 1) Quartet provides superior accuracy relative to prior methods 55 58 3 across different computing budgets and model sizes, and that 2) its fast
implementation allows to outperform highly-optimized FP8 kernels. This establishes that MXFP4 can indeed provide "optimal" training in practice.
Our work bridges the gap between emerging low-precision hardware capabilities and the algorithmic support needed for accurate, end-to-end quantized model training. Specifically, we show for the first time that the new MXFP4 format can be competitive with FP8 in terms of accuracy-vs-speed which we hope can enable significant reductions in the rising computational costs of AI.
Related Work
Training in 8-bit formats. Early work on low-precision neural network training focused on 8-bit or higher precisions, mainly on CNNs. Banner et al. 4 demonstrated accurate 8-bit training via careful scaling and pighsio-recision accumulation. Yang et al. [59] proposed a framework that quantized weights, activations, gradients, errors, and even optimizer states to INT, achieving for the first time completely integer-only training with comparable accuracy. SwitchBack (55) and JetFire (58) build on this progress, targeting 8-bit training for Transformers [50]. Specifically, SwitchBack uses hybrid INT8/BF16 linear layer for languae models, performing forward and grautggrddeet MatMuls in INT8 while computing weight gradients in 16-bit; this yielded 13-25% end-to-end speedups CLIP models with accuracy within 0.1% of full precision.
JetFire 58 achieved fully INT8 training for Transformers by using novel per-block quantization scheme to handle activation and gradient outliers. By partitioning matrices into small blocks and scaling each block independently, JetFire preserved accuracy comparable to FP16 training while

---

obtaining ~ 40% end-to-end speedup and 1.49x reduction in memory usage. The JetFire approach conceptually similar to the FP8 DeepSeek training technique BI), which used larger block sizes. Recently, HALO 3 improved upon JetFire in terms of the accuracy-speedde specifically focusing on low-precision fine-tuning. In our work, we will treat FP8 as the idealized baseline that has the quality of BF16 and the speed of raw FP8 GEMM operations. That is, when comparing agains FP8, we compare against simultaneously the most accurate and the fastest FP8-based methods could ever be.
trade-off in INT8,
End-to-end lower-precision training. As our results and prior work suggest, going below 8-bit precision in training using the above approaches is extremely challenging, due to the narrower dynamic range and higher error. This frontier was first explored by Sun et al. [42), who achieved 4-bit training on ResNets by using custom numeric format, which unfortunately is far from being supported in hardware. Chmiel et [IT] introduced a logarithmic unbiased quantization (L.UQ) scheme to this end, combining two prior ideas: (1) log-scale FP4-type format to cover wider dynamic range, and (2) applying stochastic unbiased rounding on the backward. For reference, LUQ incurs a 1.1% top-1 accuracy drop on ResNet50/lmageNer and has not been validated on FP formats. Xi et al. [56) proposed a method to train Transformers using INT4 effective precision for all linear layers, using specialized quantizers: block-wise Hadamard transform and LSQ [20] for outlier mitigation on the forward pass, and leverage score sampling on the backward pass to exploit structured sparsity, together with custom INT4-effective format. Their approach trains FERT-family models within 1-2% accuracy gap relative to FP16, with a 2.2x speedup on individual matrix multiplies (relative to 4x theoretical speedup), leading up to 35% faster training end-to-end.
hardware-supportee
We compare relative to these techniques in Section significantly in terms of accuracy and stability.
5 and show that Quartet outperforms them
Mixed-precision training in low- ow-pprcission formats. Given the importance of inference cost reductions, there has been significant work on quantization-aware training (QAT) [14 7 20 5 291. i.e. methods that only quantize the forward pass. Two key difficulties in this setting are I) minimizing the error induced by quantization on the forward pass, and 2) obtaining a stable gradient estimator over the resulting discrete space. With regards to eгror reduction, existing methods either try to find good "learnable" fit w.r.t. the underlying continuous distribution [1 20 or perform noise injection during QAT in order to make the network more robust to quantization (5) Closer to our work, Wang et al. (53] explored FP4 QAT, introducing a "smoother" gradient estimator, together with outlier clamping and compensation to handle activation outliers. While their approach shows good accuracy, is fairly complex and not validated in terms of efficient support. Prior work by (37 provided a simpler alternative approach, based on more precise MSE fitting, an optional Hadamard rotation, and a clipping-aware "trust" gradient estimator. By contrast with these forward-only approaches, recent work by Tseng et al. [47] investigated backward-only quantization with the MXFP4 format, signaling the importance of stochastic rounding and outlier mitigation in low-precision backpropagation. Since the first publication of this work, number of alternative methods for FP4 pre-training have been
[2))
proposed, either as concurrent alternatives 13
or extensions to Quartet 34 10
3 Background
Quantization grids. Quantization maps high- high-precision internal model states, such as weights, activations, or gradients, to a (lwwer-precisee discrete set set-i.e., the quantization grid. This grid can be uniform, e.g., for integer quantization, or non- uniform, e.g., floating point (FP) quantization, where the value spacing is roughly exponential for fixed exponent. Since the original values may differ in scale compared to the grid, a higher-precision scale & is typically stored alongside the quantized values. For vector ær the quantization process can be written q(x:)= = round ( grid), and the original values can be approximately reconstructed as 2 = q(xx). Common choices for the scale are setting it to the maximum absolute value (absmax) in x (to avoid clipping) or optimizing it to minimize the mean squared quantization error, e.g. [3].
Quantization granularity. Apart from grid choice, quantization methods also differ in the gramularity of the scales. A single scale value can be shared across an entire tensor, e.g. [3) across each row or column [37], or over more fine- fin-ngraiedd custom-defined blocks, such as 2D blocks (57. 3I or 1D blocks 36 47). Notably, the latest Blackwell GPU architecture [35) introduces hardware support for
3

---

MXFP4/6/8 and NVFP4 formats. MXFP 36 formats share an FP8 oower-of-two scale over each 1D block of 32 elements, while NVFP4 35 uses FP8 (E4M3) scales and ID blocks of 16 elements.
Rounding. Quantization typically involves rounding, e.g., via deterministic rounding to the nearest grid point, results in the lowest mean squared error (MSH). In contrast, stochastic rounding introduces randomness, rounding up or down with probabilities based on the input's distance to nearby grid points. While may introduce higher MSE, stochastic rounding helps control bias, which can be crucial for maintaining the convergence of iterative optimization algorithms
Outlier mitigation. One key issue when quantizing neural networks is the existence of large outlier values in the network weights, activations, and gradients [18]. One standard way of mitigating such outliers 43
47 is via the Hadamard transform: given vector defined h(z) = HaI, where Ha gdxd is the normalized Hadamard matrix with elements from {=1}. Hadamard matrices have a recursive structure Ha = 1/2 H2 & Hd/2. which enables efficient computation when is power of two [22]. Optimized FWHT implementations for GPUs are available [77 44 When d is not a power of two, the input vector is typically either zero-padded
9
3
2
R h(x) is
to the next power of two or transformed using a Grouped Hadamard Transform, where x is split into equal-sized blocks (each with power-of- length), and the Hadamard transform is applied independently to each block.
Blackwell Architecture Support. NVIDIA's 5th-gen. Tensor Cores in Blackwell [35] provide native
4-bit floating-pon execution. The cores support different block-scaled formats such as MXFP4 [36] and NVFP4 35 which roughly double the peak throughput over FPS/FP6, with single B200 GPU
peaking at 18 PFLOPS of dense FP4 compute [35]. Interestingly, our investigation shows that, as of now, MXFP4 the only microscaling format with support for all required layouts for both forward and backward multiplications in low precision on Blackwell [45]. Therefore, we adopt MXFP4 for our implementation. This format stores each value using 1 sign bit mantissa bit + 2-bits for exponent. Every group of 32 elements shares common 8-bit scaling factor, represented with exponent bits, and no bits for mantissa. Blackwell's s 5th-gen. Tensor Cores handle the required on-the-fly rescaling in hardware, without the need for software-based rescaling at CUDA level. Additional details are provided in Section 4.4 4.4
LLM pre-training. We pre-train Transformers S1 of the Llama-2 46 architecture in the range of 30, 50, 100, 200 million non-embedding parameters across wide range of potarmm--rrretter ratios raging from 25x (around compute-optmal (27)) to 800x (extreme data saturation). We additionally selectively scale the model size up to around billion parameters to verify training stability. We train all models on the train split of the C4 [19 dataset and report C4 validation loss as the main metric. We use the AdamW optimizer (32 with weight decay of 0. 1, gradient clipping of 1.0, 10% LR warmup and cosine schedule. We identify the optimal LR for one of the small unquantized baseline models, scale it inverse- propoooioooo to the number of non-embedding parameters and reuse for every quantization scheme we evaluate. We present all hyper-parameters in Appendix A.2
4 Quartet: Four Ingredients for "Optimal" Quantized Training
4.1 Ingredient 1: Comparing Quantized Training Approaches via their Induced Scaling Laws
The ability of LLMs to scale predictably with both model size and data across orders of magnitude is cornerstone of the current Al scaling landscape (8). Mathematically, this says that the expected loss is a function of model and data parameters, often described in the form of parametric function. This function can be fitted on a set of training runs, and then used to determine the optimal computational training regime [27] to extrapolate model performance (21).
In this paper, we investigate scaling laws relating evaluation loss to the precision in which the forward and backward passes performed, denoted by Pforward and P Pbcckward, respectively. For this, we propose a scaling law of the following functional form:
-(x
A (N- - eff (Poorward))
B
L(N, D, Pforwords P Poockward)
+ F, (1)
D -ff (P Phackwardd) 3
4

---

Scaling Law Fit
BOPS Speedups
RTX 5090 Speedups
1008
100B
34
C4 Vol Loss
1081
JOB
Nmx
Optimal Precision
Nmur
3.0
FP8:FP8
Model
18
FP4:PP
18
Llama 3
FP8:FP4
Qwen
FP4:FP4
Gemma
10
10a
10a
10a
10a
DIN
D Daax
D Dmxx
(a)
(b)
(c)
Figure 1: Analysis of Quartet: (a) Scaling-law I fit for various FORWARD.BAKKWARK precisions. (b) Regions where each FORWARD:BACK WARD precision optimal under the BOPS speedup model. (c) Same (b) but with RTX 5090 speedups. Interestingly, popular models such larger Llama3 or Qwen2.5 models fall into the FP4:FP4 optimality region, implying that training similar models in FP4 might have been optimal.
where A, B, at, 33, are constants describing the general loss scaling w.r.t. model parameter count N and training corpus size D.
The key addition is given by the fitted parameters eff (Pforwar))) representing the parameter efficiency of the precision Pforward used in the forward pass, and ( (Pbackward) representing the "data efficiency" the backward pass occurring in potentially different precision Pheckward- (Both these factors naturally in the interval (0, 1], where the value is reached for full-precision.) Specifically, our parametrization postulates that the impact of the forward-pass precision is felt primarily w.r.t. the trainable parameters, i.e., lowering precision "effective" parameter count to N x ( (forward) < N. This follows the general trend of modeling the effect of forward pass quantization For the data term, we postulate that lowering backward term D, so we effectively need additional data to reach the same the same loss, precisely by factor of 1/eff (Pff ([PPockwart). This is a novel way to model backward pass quantization that we propose. consistent with optimization theory results []. as well as observed performance gaps (see Figure D (a)). We present experimental data to justify these assumptions and compare against alternative scaling laws 30 Appendix A.3
forwward lowers the model's
multiplicative factor on parameter count 22 30 26 37].
precision primarily impacts the data
Experimentally, we observe that different quantized training methods, e.g., STE 6 vs. QuEST (37, induce different scaling laws, and in particular different efficiency parameters. While, usually, scaling laws are used to extrapolate model performance across different parameter and data sizes, we propose to use scaling laws to compare different training methods. Specifically, we say that quantized training method A is superior to method B if it offers both higher parameter efficiency eff and higher data efficiency effp
4.2 Ingredient 2:
Induces Inference-Trrining Trade-Offs
The above scaling law suggests that, given a set of scaling parameters and target loss we wish the model to achieve, can directly solve for the "optimal" forward and backward precisions which allow us to match the loss. However, as pointed out by Sardana et al. [11, it often the case in practice that we wish to put larger weight on inference cost, rather than training cost, which can lead to different results when determining the "optimal" training precisions. Because inference latency depends solely on the forward pass (~~ 33% of training compute) while the backward pass consumes the remaining ~ 66%, these trade-offs may need to be analyzed separately.
Specifically, we can state set of simple guiding principles:
Forward pass. Low-precision induces a trade-off between reduced parameter efficiency, and increased inference speed: for instance, we could train a larger model in terms of parameters N, but quantize its forward pass to lower precision, and obtain better trade-off. As such, Pforward should be picked to optimize this trade-off.
5

---

Backward pass. Similarly, training speedup due to a quantized backward pass can offset the reduced data efficiency p: we could train more quaanii-quantized model on more data under the same computing budget. Thus, backkaard should be picked to optimize this trade-off.
We contrast this with previous work, which often requires lower precision to suffer no accuracy loss (c.g., Chmiel et al. [2]. This unnecessarily reduces these trade-offs to simple selection of the fastest lossless precision. We argue that scaling-law analysis enables more fine-grained approach needed to decide upon the "optimal" set of forward and backward precisions.
Example speedup model. To illustrate this, we assume a hardware- sardware-agnostic bit- bit-wise ops (BOPS) model, which states that speedup is inversely proportional to datatype bit-width. The speedups are stated in T relative to FP8 baseline:
Then, given forward-pas pass compute budget Nina and training budget Nmax Dmaxs the effective loss will be given by:
Loss (Nmax spfw, Dmax sptr / spfw, Powd: Powa):
which we evaluate with the scaling law from Equation [.. leading to the fit from aa). One can see how spfw and sptr propagate as multiplicative factors on effy and effD and directly counter the suboptimal parameter and data efficiencies.
Figures (ig) (c) illustrate the optimal- ity regions: specifically, it tells us for which model sizes (Y axis) and corresponding relative training com- pute (X axis) FP4 is optimal relative to FP8 (red vs. orange region). The green area is the region in which train- ing using our MXFP4 implementation would be optimal by this metrie. In Figure [ we demonstrate that valida- tion loss, on which we build the com- parison, is consistent with downstream performance, meaning that the optimality propagates there as well.
Table I: Speedups relative to an FP8 baseline for forward (spfw), backward (spbw); sptr is the harmonic mean of spfw and spbw with weights 1 1/3 (forward) and 2/3 (backward).
Operation
FP4:FP8
FP8:FP4
FP4:FP4
Forward / Inference (spfw)
2.0
1.0
2.0
Backward (spbw)
1.0
2.0
2.0
Training (sptr)
1.2
1.5
2.0
In summary, Ingredient says that low-precision impact should be analysed under the compute budget; scaling-law fits then reveal when given precision is the optimal choice for either pass.
4.3 Ingredient 3: Minimal Forward-Pass Error and Error-Bias Trade-off
The above ingredients should allow us to determine the "best" quantized training method among existing approaches, focusing on the hardware-supporrte MXFP4 36 format.
Forward pass quantization. As detailed in Section 2 existing QAT (forward-only) approaches can be split into "noise injection" [5] and "error-minimizztionn strategies, e.g. (37]. Focusing the forward pass, by the above discussion (Ingredients and 2), we seek the approach which maximizes the parameter efficiency factor e For this, implement four standard schemes for QAT: 1) stochastic rounding (SR) with standard AbsMax per- grrugroup normalization [47]; 2) vanilla round-to- nearest (RTN) quantization with AbsMax per-group normalization; 3) learnable scale clipping (LSQ) with RTN quantization 20 56]; 4) Hadamard normalization followed by RMSE-based clipping (QuEST) 337. For fairness, we apply the Hadamard transform to weights and activations for each one of these schemes before quantization. We compare these approaches following Section 4. we train models using each technique, apply scaling law fitting, and register their resulting eff factors. For additional information, we also show representations' mean-squared error (MSE) for fitting random Gaussian data. The results are provided in the first rows/columns of Table 2
The results in Table Tsho show that QuEST has the best parameter efficiency eff among all existing methods. Moreover, e appears correlate heavily with MSE, suggested by Panferov et al. 337 38). Additionally, the results align with the analysis of Chmiel et al. [12 that determined deterministic RTN always be preferable to stochastic rounding for the forward pass.
Backward pass: a novel error-bias trade-off. The above findings do not transfer to backward pass quantization, as optimization theory shows that unbiased gradient estimation is critical for

---

Table Illustration of error- error-bias trade-off between different quantized forward and backward pass approaches. For the forward (given by the effx metric) the best performing method is QuEST, correlating with superior MSE over Gaussian input data. By contrast, for the backward pass (the data efficiency p). the best performing method is RTN, that balances quadratic error with magnitude alignment. This justifies our choice of method, which combines block-wise QuEST on the forward, with RTN the backward pass.
Rounding
cff'N
MSE
eff
Misalignment ( (1/S))
Stochastic Rounding AbsMax
0.42
2.77
10
0.88
0 9.3 x 10
Round-to-nearest AbsMax
0.59
1.37
10
0.93
QuEST (Hadamard + RMSE)
0.64
1.32 x 10-2
0.83
1.3 x 10
convergence, e.g. [1 This leads to a trade-off between the error minimization can obtain on the forward pass, and the bias induced over the backward pass for a given method. We study this trade-off via novel analysis of gradient alignment between different quantization methods.
who studied RTN quantization with randomized rotations, approximated by the randomized Hadamard transform, which we denote by H. They show that, while RHT makes quantization unbiased in direction, it adds bias in magnitude. To address this, they proposed an approach that makes RTN projections of post-RHT RHT vectors unbiased, denoted by Q. via the following input (X) and randomness (() specific group-wer wise rescaling factor S:
To study gradient bias, we follow the analysis of 48 49
XX RTX
Ec(Q(X, X if = S. RTN where S =
Unfortunately, their re-scaling is incompatible with coarse group-wise scaling of the MXFP4 format, so we cannot use it in practice. However, we can still use their approach to gauge the degree of misalignment for different quantizers by simply studying their corresponding expected value of 1- E (SS), which we call the projection magnitude misalignment. This factor is presented in Table
2 along with the MSE across different schemes. Focusing on stochastic rounding (SR) vs round-to-nearest (RTN) with AbsMax, one can that SR trades higher error for perfect alignment.
To connect those quantities with training dynamics, we amalyze the cumulative effect of misalignment and егтоr on backward quantization for a 30M-parameters Llama model. In Figure 2 (a) and (c), we plot the alignment metries Cosine Similarity and Projection Magnitude Misslignment- inter-layer activation gradients as a function of back-propagation "depth"". We can again observe the trade-off between similarity and magnitude misalignment. Finally, (c) connects those quantities to final model quality (loss gap vs. full-precisio model) for increasing data-vs-parameters.
We observe that RTN backward quantization consistently outperforms the alternatives, although the gap with stochastic rounding (SR) decreases as the training duration grows ?
Summary. Our analysis outlines new trade-off between parameter efficiency on the forward (equated with quantization MSE), and data-efficiency on the backward (which characterize with with trade-off between error and misalignment). In the following, we will adopt "best of both worlds" approach, aiming to perform forward pass that minimizes MSE (based on QuEST (37]) together with an RTN backward pass that is balances quadratic error with and magnitude alignment. The novel challenge, which we address next, will be an extremely efficient GPU-aware implementation of such approach.
4.4 Ingredient 4: Fast GPU Support for Accurate Quantized Training
Quartet Overview. We integrate our prior discussion into Algorithm | (also shown in Figure 3) which aims to perform accurate training while executing all three matrix multiplications of linear
In the first draft of the paper, an implementation bug caused large performance degradation for RTN backward quantization for longer training runs. This degradation led to our arriving at a different conclusion (SR superiority), which we reject.
7

---

0.6
1,00
0.5
0.0175
Cosine Similarity
Vall Loss Gap
0.0150
[[1/S]
0.0125
0.0100
0.2
0.90
AR.Ab
0.1
0.055
0.55
RTN AlaMax
0.0
0,0025
N-100M
X-50M
X-2004
0,0000
10
10
Backprop Distance, Blocks
Backprop Distance, Blocks (b)
D/N (c)
(a)
Figure 2: The effect of backward quantization LLM training gradient quality and impact on performance: (a, left) and (b, middle) shows cosine similarity and projection magnitude misalignment with unquantized reference, while (c, right) shows performance gaps with mon-quantized baseline for a set model sizes and data to 1o-parameter ratios (D/N).
Layer (1)
BF
Haderard Transkens
Q-EST Daaetiee
MIXER
MXFIN
Doquatia
RTY Quantic
Transpix
Hebonard Transhera
Hadamand Trundenn
Tranpoc
Helereard
Transfern
Fron Lagar
Laswe
Apply QuEST Yank
Hadietard Trasskore
BF16 Gradicnt
?
8932 Weights
Hattrnard Trasoferms
RIN Quaetto MXPP
MXXM GEMM
Hadamaard
MIXEP GEMM
Optaaber
Apphy QuEST Mak
Imn, Wadommrrd Trarofarm
RIX Quantio MXPPA
Figure 3: Quartet computation flow for a fully-MXFP4 linear layer training.
layer in low precision. The forward pass applies fixed Hadamard transform H,, (of block size equal to the quantization group size) and QuEST projection to low precision and multiplies the resulting tensors with an MXFP4 kernel. The backward pass decorrelates the multiplied tensors with an identical block- wise random Hadamard transform Hg. applies round-to to-ncarest (RTN) to MXFP4, performs the two gradient GEMMs in MXFP4, rescales to compensate for RTN range matching (47). applies QuEST masks (M,.Mw. and inverts the Hadamard transform Hg.
Costs and format specialization. The key added cost of the above pipeline is that of the Hadamard rotations and their inversion: specifically, two Hadamard/Inverse transforms are added over standard training. Our key observation is that, since the MXFP4 already groups 32 consecutive weights (in ID), sharing scales, we can and should apply the Hadamard rotations and their inversion at the same group size. With fast Hadamard implementation, the theoretical cost Olg log g) negligibbee for S 256 compared with the GEMMs.
GPU kernel support. While the above blueprint appears simple. implementing it efficiently Blackwell GPUs-in order to leverage fast MXFP4 support-is extremely challenging. For illustra- tion, direct implementation of the above pattern would be slower than FP16 unquantized training, let alone optimized FP8. Our fast implementation builds on CUTLASS 3.9 45 which provides templates for the new Blackwell architecture. Computation happens in stages: Stage fuses the Hadamard transform, quantization, scale calculation, and QuEST clipping mask generation (only on forward) into a single kernel: Stage 2 performs GEMM using dedicated kernel.
Stage I: Fused quantization-relatee operations. First, we observe that, thanks to small group size, the Hadamard transform be implemented as a direct GEMM between the corresponding

---

MXFP4 vs FP8 Speedup
MXFP4 va BF16 Speodup
Tmining Dynamies
5.0
Bloekwise Specdup
4.0
Forward
Backward
Blockwine Speedup
4.0
4.5
MXFP4 Quamet
FP8 Baseline
C4 Val Loss
3,0
Training
3.0
4.0
2.0
2.0
1.5
1.0
1,0--
3.0
1B
128
201
401
60
80
Model Size
Model Size
Consureed
(A)
(b)
(c)
Figure 4: (a, left), (b, middle): Quartet kernels block-wise speedup across model sizes relative to FP8 and BF16. (c, right): Training dynamics for the 7B model trained with Quartet relative FP8.
input matrix and a fixed x Hadamard matrix (see Sec. B producing output in FP32, which is stored in GPU Shared Memory (SMEM). This allows us to implement the Hadamard operation efficiently by leveraging CUTLASS's multilevel tiling templates to optimize data movement. All subsequent operations are integrated via custom CUTLASS epilogue, which utilizes the intermediate results previously stored in higher levels the memory hierarchy and operates locally in the Register File (RF). At this stage, Blackwell's (E2MI) using the PIX instructions for this purpose. To construct the final MXFP4 format, we compute scaling factors of shape x 32. These scales are represented in 8-bit using the E8MO format. Finally, the clipping mask is computed, and the three resulting tensors (values, scales, and mask) written to Global Memory (GMEM). Throughout, data storage is optimized use the widest memory instructions possible.
hardware support is used to downcast FP32 values to FP4
Stage 2: Dedicated GEMM kernel. Blackwell introduces the tcgen05. mma instructions, which
natively support matrix multiplication with scale factors the form D = C+ (A SFA) (B These scale factors are applied along the inner (K) dimension of the GEMM. For MXFP types, every 32 elements along the -idimensioon of matrices and B share a corresponding scale factor. This implics that M X K matrix A is associated with scale matrix SFA of size M x [K/32]. Our dedicated kernel is based on CUTLASS block-scaled GEMM for narrow precision. As part this implementation, also included the necessary functions reorganize the scale factors generated in the Stage 1, aligning them with the layout required by this architecture 35
To our knowledge, our implementation is the first to efficiently support quantization- operations for microscaling formats on the Blackwell architecture. We release as part of "QuTLASS", an open-source library that can be accessed here
Experiments
We now provide additional experimental support for the validity of Quartet, shown in Figure 3 focusing on accuracy comparisons with existing INT4/FP4 training methods, and examining kernel speedups.
Experimental setup and scaling law fit. As described in Section we pre-train Llama-style models on C4 and report validation loss after fixed token budget. All baselines reuse the optimizer, schedule, and hyper-parameten accuracy across methods by fitting the full scaling law in Eqn. T Jaross methods, as follows: we fit parameters A, a, B, 3, E and on a grid of baseline precision runs (FP8 forward, FP8 backward) fit the parameter and data efficiencies eff and effp separately for every forward and backward quantization scheme we evaluate. The law is fitted identically to prior work in this area [27 30 8. For a more detailed description we refer to Appendix A.3
as described in Appendix A.2 Following Section 4. we compare
shown Figure [ (). Then
Accuracy comparisons. We compare accuracy (validation loss) as well as the efficiency factors against four recent, and backward passes: 1) LUQ [12] applies to both INT4 and FP4, using unbiased quantization that pairs 4-bit weights/activations with stochastic underflow, and logarithmic stochastic rounding: 2) HALO 3 which uses Hadamard rotations to mitigate outliers, evaluated FP4 at their most accurate HALO-2 setting; 3) Jetfire (58) performs quantization in blocks of 32 32, originally
training pipelines that operate in 4-bit precision for both forward

---

introduced for INT8, and adapted to FP4 for our setup; 4) LSS [56] for INT4 training, that combines a Hadamard-basse forward pass with "leverage-see scorre"escore" sampled INT4 gradients.
55
Table 3: Validation loss (lower is better) on C4 for Llama models with 30M parameters and efficiency coefficients fit- ted on them. Columns show the tokens cosasstt-puue to-parameters ratio (D/N). All methods share identical setups; only the quan- tization scheme varies. NaNs for LSS-INT4 appeared at arbitrary stages of training without any irregularities.
50
Aocuracy (%)
Quurtet
45
FPS
HellaSWAG
40
WinoGrande
35
Are-Easy
Method
25
50x
100
200
400x
cffix
effp
30
LUQ-INT4
3.73
3.68
3.66
3.43
3.40
0.49
0.15
2.4
2,5
2.6
2.7
2.9
LUQ-FP4
4.81
4.91
4.88
4.84
4.80
0.01
0.07
C4 Val Loss
Jetfire-FFD FP4
7.03
6.94
6.76
6.62
6.58
Unstable
Figure 5: Correspondence between validation loss on C4 and various few- shot benchmarks for Ilama models with 30-200M parameters.
HALO-FP4
6.65
7.04
6,55
6.50
5.38
Unstable
LSS LSS-INT4
NaN
3.40
NaN
NaN
NaN
Unstable
Quartet
3,49
3.38
3.29
3.24
3.20
0.65
0,95
Accuracy discussion. As can be seen in Table across all attains the lowest loss, often by very large margins. At a tokens per parameter ratio of 100x, Quartet improves upon
ratios, Quartet
by 10% relative loss, and the gap widens as we increase data size. We note that Jetfire and HALO incur large degradation and are unstable when ported to FP4. Interestingly, LSS competitive only for shorter runs, and diverges for longer training budgets, beyond 50x, matching observations from prior work [23]. Overall, LUQ-INT4 is the strongest prior work; however, Quartet reaches significantly higher parameter and data efficiency, suggesting that it requires, roughly, 15% fewer parameters and 5x less data to reach the same loss. Figure (c) additionally demonstrates the stability of Quartet for training models two orders of magnitude larger (7B parameters).
Additionally, we trained 100M, 200M, 430M, 800M and 1.6B parameters Llama models with Quartet and FP8, with D/N 100. We evaluated them set of few-shot benchmarks, including HellaSwag [60]. WinoGrande [40] and ARC-easy [13]. Figure demonstrate that those evaluations consistent with C4 validation loss for larger models.
Speedup results. Next, we evaluate the efficiency of our implementation on NVIDIA RTX 5090 GPU by measuring its performance across single layers of standard shapes, and aggregating across an entire transformer block. Speedup results are shown Figure 4 using batch size 64 and sequence length of 512. The FP8 baseline is provided by CUTLASS MXFP8 kernels, while the BF16 baseline uses Py PoTcch, both using Blackwell-optimized kernels. Inference speedups are more pronounced due to the lower cost of the forward pass compared to the backward pass, and the latter's higher computational complexity. The speedup scales with the arithmetic intensity (i.e., model size), reaching up to 2x over FP8 and 4x over BF16 the forward pass, where it stabilizes. In the backward pass, our implementation achieves up to 1.5x over FP8 and 2.6x over BF16, resulting in an overall training speedup of up to around 1.6x, and 2.9x, respectively.
6 Discussion and Limitations
We provided a set of guidelines to modeling, comparing and designing fully-quantized training schemes for large language models. Moreover, we followed those guidelines to arrive Quartet: new SOTA full MXFP4 training algorithm. One current limiting factor is that Quartet was designed with a specific (standard) data-type and compute architecture in mind. In future work, we plan to look into generalizing our approach to alternative formats, as well as larger-scale distributed model execution.
10

---

Acknowledgments
This research was funded in part by the Austrian Science Fund (FWF) 10.55776/COE12. i.e., the Bilateral AI Cluster of Excellence, and through generous gifts by NVIDIA and Google. This work was supported under project ID 40 as part of the Swiss AI Initiative, through a grant from the ETH Domain and computational resources provided by the Swiss National Supercomputing Centre (CSCS) under the Alps infrastructure. We would like to thank our contacts at Datacrunch/Vern /eedda (Paul Chang and Antonio Dominguez) for hardware support that was essential to this project.
References
[1] Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. Qsgd: Communicationn efficient sgd via gradient quantization and encoding, Advances in neural information processing systems, 30, 2017.
[2] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci. Bo Li. Pashmina Cameron. Martin Jaggi. Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated lims. arXiv preprint (iXx::44040004566 2024, URL ftttpstts:///rxii/ attt::://arxvv.or//bbb/////// /404. .00456
[3] Saleh Ashkboos, Mahdi Nikdan, Soroush Tabesh, Roberto L. Castro, Torsten Hoefler, and Dan Alistarh. Halo: Badamardd-sssiste lower- lower-precision optimization for Ilms, 2025, URLhttps://arxx /bs/ 2501. 02625
Ron Banner, Itay Hubara, Flad Hoffer, and Daniel Soudry. Sealable methods for 8-hit training of neural networks. Advances in Neural Information Processing Systems (Neur[PS). 2018.
[5] Chaim Baskin, Natan Liss, Eli Schwartz, Evgenii Zheltonozhskii, Raja Giryes, Alex M Bronstein, and Avi Mendelson. Uniq: Uniform noise injection for non con-uniform quantization of neural networks. ACM Transactions on Computer Systems (TOCS), 37(1-4;: -15, 2021.
[6] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint urXiv:1308.3433 2013.
[7] Yash Bhalgat, Jinwon Lec, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+; Improving low-bit quantization through Icarnable offscts and better initialization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
[8] Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws, 2025. URL https://aaxxi http:///aaaawwwwaaa//////ea wwwps////2/ba//........ 08606
[9] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. arXiv preprint 13311:2300.133304, 2023. URL (tt.hhtts:///rra /tt.iv.vvxvooo///////// abs/2307 13304
[10] Yuxiang Chen, Xiaoming Xu, Pengle Zhang, Michael Beyer, Martin Rapp, Jun Zhu, and Jianfei Chen. Tetrajet-v2: Accurate nvfp4 training for large language models with oscillation suppression and outlier control, 2025.
org/ /abb///510..mm 27527
[11] Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben- Yaacov, and Daniel Soudry. Accurate Neural Train- ing with 4-bit Matrix Multiplications at Standard Formats. In International Conference on Learning Representations (ICLR), 2023.
[12] Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Accurate neural training with 4it matrix multiplications standard formats, 2024. URL URL (tttjjttt::///rrrt org/ /b////////// aaaaa/aa//111 /212/2111.11111 10769
[13] Brian Chmiel, Maxim Fishman, Ron Banner, and Daniel Soudry. Fp4 all the way: Fully quantized training of Ilms, 2025. https://arxiv org/ h2tg//bb//500..... 19115
[14] Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pieree I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint Xiv: 805.06085, 2018.
[15] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https: //te:://arxae //trxivv.v////////////ee////atll org/ abs/aassss/// 05457
11

---

[16] Ben Cottier, Robi Rahman, Loredana Fattorini. Nestor Maslej, and David Owen. The rising costs of training frontier ai models. arXiv preprint arXiv:2405.21015, 2024. URL htttps://arxvv.orrggg///
#405.21015,
[17]
Dao-AILab. Fast hadamard transform in cuda, with a pytorch interface. Dao AlLab/fast-haaeeeeee hadamard-trana
https:///githnn com/
2024, Accessed: 2025-05-13.
[18]
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int((: 8-bit matrix multiplica- tion for transformers scale, arXiv preprint urXiv:2208.07339) 2022. UtLttttps://axiii http::///.rwvvro...../ooo
2208. 07339
[19] Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus, 2021. URL battps::/aaattt www..rgg////.....o..eo....... abwp/aas////// 08758
[20] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint 4rXi::1902.087555 2019.
[21] Meta et al. The Ilama 3 herd of models, 2024. URL httLhttpp///xxiii .rg/ ht////bb/2200 2aobs/2400..0 .27833
[22] Fino and Algazi. Unified matrix treatment of the fast walsh-hadaman transform. IEEE Transactions on Computers, 100(11): 1):1 142, 1976.
[23] Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token llms. arXiv preprint arXiv:2409. 12517, 2024.
[24] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210. 17323, 2022. URL https://arxiv.orgg attpa//bb/////../// 2210. 17323
[25] Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-comneeted foundation models. In International Conference on Learning Representations, 2024.
[26] Elias Frantar, Utku Evci, Wonpyo Park, Neil Houlsby, and Dan Alistarh. Compression scaling laws:unifying sparsity and quantization, 2025. URL httphh:tss//arrr . Barggabb//5500 16440
[27] Jordan Hoffmann, Sehastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL httt:htt////// atx/axiiivv..c http://wxivv....aaaaaaa www.aos/./aaoooo.oo/...o... http//abb/222..... 2203. 15556
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL attps://arxiv.
org/ abs/ 2001. 08361
[29] Ayush Kaushal, Tejas Vaidhya, Arnab Kumar Mondal, Tejas Pandey, Aaryan Bhagat, and Irina Rish. Spectra: Surprising effectiveness of pretraining ternary language models at scale. arXiv preprint arXiv:2407.123277 2024.
[30] Tanishq Kumar, Zachary Ankner, Benjamin F. Spector, Blake Bordelon, Niklas Muennighoff. Mansheej Paul, Cengiz Pehlevan, Christopher Ré, and Aditi Raghunathan. Scaling laws for precision, 2024. URL
https:// arwiv. wrg//ba///1111.//oo 2433000031300 04330
[31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu,, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo. Dejian Yang. Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin. Huazuo Gao, Hui Li. Hui Qu. L. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, Runxin Xu, Ruoyu Zhang. Ruyi Chen, S. S. Li. Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao. Wengin Yu. Wentao Zhang, X. Q Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin
12

---

Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun. Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang. X. Wei. Y X. Zhu. Yang Zhang, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang, Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying Hc, Ying Tang. Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yuc Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yumxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, 2. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang. Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma. Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu. Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li. Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v v3 technical report. arXiv preprint wwx.:22122.999335 2024. URL https://arxiv org/ htsp/bb///211// 19437
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https://arxxa org/ hts/abb/1711...0 05101
[33] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. 2018. URL
wttp://wwwrr.bba//1.....o 033740
[34] NVIDIA. Felix Abecassis, Anjulie Agrusa, Dong Ahn, Jonah Alben, Stefania Alborghetti, Michael Ander- sch, Sivakumar Arayandi, Alexis Bjorlin, Aaron Blakeman, Evan Briones, Ian Buck, Bryan Catanzaro, Jinhang Choi, Mike Chrzanowski. Eric Chung, Victor Cui. Steve Dai. Bita Darvish Rouhani, Carlo del Mundo, Deena Donia, Bure Eryilmaz, Henry Estela, Abhinav Goel, Oleg Goncharov, Yugi Guvvala, Robert Hesse, Russell Hewett, Herbert Hum, Ujval Kapasi, Brueek Khailany, Mikail Khona, Nick Knight, Alex Kondratenko, Ronny Krashinsky, Ben Lanir, Simon Layton, Michael Lightstone, Daniel Lo, Paulius Micikevicius, Asit Mishra, Tim Moon, Deepak Nar Narayanan, Chao Ni, Abhijit Paithankar, Satish Pasumarthi, Ankit Patel, Mostofa Patwary, Ashwin Poojary, Gargi Prasad, Sweta Priyadarshi, Yigong Qin, Xiaowei Ren, Oleg Rybakov, Charbel Sakr, Sanjeev Satheesh, Stas Sergienko, Pasha Shamis, Kirthi Shankar, Nishant Sharma, Mohammad Shoeybi, Michael Siu. Misha Smelyanskiy. Darko Stosic, Dusan Stosic, Bor- Yoing Su, Frank Sun, Nima Tajbakhsh, Shelby Thomas, Przemek Tredak, Evgeny Tsykunov, Gandhi Vaithilingam. Aditya Vavre. Rangharajan Venkatesan, Roger Waleffe, Qiyu Wan. Hexin Wang. Mengdi Wang. Lizzie Wei, Hao Wu, Evan Wu, Keith Wyss. Ning Xu, Jinze Xue, Charlene Yang. Yujia Zhai Ruoxi Zhang, Jingyang Zhu, and Zhongho Zhu. Pretraining large language models with nvfp4, 2025. URL https://arxiv.. org/
wtt.oorgg///aaaaeeeeeeeeeee abs//bbb../////////// 25149
[35] NVIDIA Corporation. Nvidia blackwell architecture technical brief. httips://resourcee.nnt nvidia com/en ee//en--ss us- blackvell architectur 2024, Accessed: 13.
[36] Open Compute Project. Ocp microscaling formats (mx) specification version 1.0. https: / www opencompute. wttp///mmtttggguurrrraaiiiiinaee Accessed: 2025-05-13.
ppd 2023.
[37] Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L. Castro, Mahdi Nikdan, and Dan Alistarh. Quest: Stable training of IIms with 1 1-bit weights and activations, 2025. URL httph::/tts::s org/ abs/2502 05003
[38] Andrei Panferov, Alexandra Volkova, Lonut-Vlad Modoranu, Vage Egiazarian, Mher Safaryan, and Dan Alistarh. Unified scaling laws for compressed representations, 2025. httpp:://///// org////xxxiv..//aaea.. /bs/
2506. 01863
[39] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. In Proceedings of the 37th International Conference on Machine Learning, pages 13962 13982. PMLR, 2020. C4 dataset introduced as part of this work.
[40] Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: wwtp:/s////a/.//////////// 10641 Accounting
[41] Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beoond for inference language model scaling laws, 2025. http/attte////rrv.oo//aa//////// 004488
[42] Xiao Sun, Naigang Wang. Chia- Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkatara- mani, Kaoutar El Maghraoui, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Ultra-Lww Precision 4 bit Training of Deep Neural Networks. In Advances in Neural Information P Processing Systems (NeurIPS). 2020.
13

---

[43] Ananda Theertha Suresh, X Yu Felix, Sanjiv Kumar, and H Brendan McMahan. Distributed mean estimation with limited communication. In International conference on machine learning, pages 3329-3337. PMLR, 2017.
[44] PyTorch Team. Hadacore: Accelerating large language models with fast hadamard transforms. https
2024. //pytrcch wwwpboogggg//eaeeeeooeeeoooo hadacor//acccceeeaa 2024. Accessed: 5025-05-13.
[45] Vijay Thakkar, Pradeep Ramani, Cris Cecka, Aniket Shivam, Honghao Lu, Ethan Yan, Jack Kosaian, Mark Hoemmen, Haicheng Wu, Andrew Kerr, Matt Nicely, Duane Merrill, Dustyn Blasig, Fenggi Qiao, Piotr Majcher, Paul Springer, Markus Hohnerbach, Jin Wang, and Manish Gupta. CUTLASS, January 2025.
URL htt.:/ttttt///iii
com/ Caem/NVIIIA////AA cutlass
[46] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut I Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molyheg, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor. Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023. Wttts:tts://aatt
org/ abaaaba///33 /307 .99288
[47] Albert Tseng, Tao Yu, and Youngsuk Park. Training Ilms with mxfp4, 2025. Attptttss://arzi Aba/2502.25 2tt222262.2226
[48] Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzen- macher. Drive: One-bit distributed mean estimation, 2021. URL https://aaxxi org/ abs/2105 08339
[49] Shay Vargaftik, Ran Ben Basat, Amit Portnoy, Gal Mendelson, Yaniv Ben-Itzhak, and Michael Mitzen- macher. Eden: Coomunicationneeffcconn and robust distributed mean estimation for federated learning, 2022. URL htttp:////ax org/ www.//ggg//bbbeeeeeeo.eeeee abs//b/...././///////aa//a/ 08842
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[SI] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez. Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL (ttps:/tt: org/ abc/ 1706 03762
[52] Hongyu Wang. Shuming Ma, Li Dong. Shaohan Huang. Huaijie Wang. Lingxiao Ma, Fan Yang. Ruiping Wang, Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.1 1453, 2023.
[53] Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang, Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing Large Language Model Training Using FP4 Quantization. arXiv preprint urXiv.2501.77116, 2024.
[54] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low low-precision training for scale vision-language models. Advonces in Neural Information Processing Systems, 36:70 110988, 2023.
[55] Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari S. Morcos, Ali Farhadi, and Ludwig Schmidt, Stable and Low-Precision Training for Large-Scale Vision-I anguage Models. arXiv preprint ar0iv:2304. 13013, 2023.
[56] Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training Transformers with 4-bit Integers. In Advances in Neural Information Processing Systems (NeurlPS). 2023.
[57] Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu. Jetfire: Efficientt and accurate transformer pretraining with int8 data flow and block quantization. arXiv preprint arXiv:2403.1222, 2024.
14

---

[58] Haocheng Xi, Yuxiang Chen, Kang Zhao, Kaijun Zheng, Jianfei Chen, and Jun Zhu. Jetfire: Efficient and Accurate Transformer Pretraining with INTS Data Flow and Per- BercBlock Quantization. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
[59] Yukuan Yang. Shuang Wu, Lei Deng. Tianyi Yan, Yuan Xie, and Guoqi Li. Training High-Perffrmance and Large-Scale Deep Neural Networks with Full 8-bit Integers. arXiv preprint wrXiv:1909002388 2020.
[60] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL https://araat org/ www./995.... 07830
15

---

A Technical Appendices and Supplementary Material
A.1 Algorithm
Algorithm Quartet MXFP4 Forward-Backward Algorithm
Require: Hadamard Transform (Hg. H,
function BACKWARD(outtuu gradient dy, ctx, seed E
block size
2:
Unpack (X,, Wg. Ma. Mw) from cix
1: function FORWARD(input X. weights W)
G₁ + (dydy E); W + H .
2: Xn. f (X)::: Wi, + (W)
G. + (IN( W. < RTN(WW))
(Xy, (...M)))QQETTTC
4:
(Wq, Mw)+ QU)ST(WA)
5:
dr, -GEMMMI(G W
5:
f GEMM (X,, Wa)
6:
dr +- 9. 1HH ① Ma)
6: return ctx = {X,, g. W Mc, Mo}
7:
G < Hyydd" (E); X t HH .)
7: end function
8:
G. + RTN(1GH) X t RIN()X)
9:
dW, + GEMMI .x)
HdW, @ Mw) 11: return de, dW
10:
dW 16 H (dW,
12: end function
A.2 Training Hyper-parameters
Table lists model-specific hyper-parameters.
Table
lists hyper-parameters shared across all
experiments.
Hyperparameter
30M
50M
100M
200M
Number Layers (Niayer)
8
10
Embedding Dimension (Nemba)
640
768
1024
1280
4096
Attention Hcads (Nnead)
5
6
8
10
32
Learning Rate (LR)
0.0012
0.0012
0.0006
0.0003
9.375-10
Table 4: Model-specific hyperparameters used in experiments.
Hyperparameter
Value
Sequence Length
512
Batch Size
512
Optimizer
AdamW Cosine decay with 10% warm-up
Learning Rate Schedule
Gradient Clipping
1.0
Weight Decay (t))
0.1
Number GPUs
8
Data Type (optimizer/accumulltto)))
FP32
Table 5: Common hyperparameters used across all model sizes and quantization setups.
A.3
Scaling Law fitting
We fit the scaling law in two stages: Stage 1. Identical to prior work 8, we fit the unquantized scaling law of the form
A
B 7)² +E D
L(N, D)
No
on baseline BF16 runs for E [30M, 50M, 100M. 200M] and D/N E [25.50, 50, 100, 200, 400, 800) (see Figure I (a)) using Huber loss with 8 10-1 on logarithm of L. Table shows the resulting fit.
16

---

Hetroriial
Modd, Kapla
F' crrur: 1.65e-06
Fii sturr: mour:66400--- 6.40e-06
Fil etor 8.65e-06
C2 Vat LaNs
Satropetatie
Eciropetatin
Ratrapttatcs
DIN
D/N
DVN
Figure 6: Comparison of various scaling law fits and their errors.
Stage 2. Using the fixed fitted parameters from stage 1, we fit the additional e and effp parameters using the same loss function.
For the isolated methods compared in Section we fit e and eff independently for forward-only and backward-only quantization respectively.
For the end-to-end 4-bit comparison in Section 5 present in Table
fitted the parameters jointly for the setups
Alternative forms. We additionally for the scaling law forms with fixed y 1 27 The fits are presented in Figure alongside the mainly used of Busbridge et al. 8
and 8 1 28
Parameter
A
B
3
E
Value
1.52 10
0.589
5.25
10
0.544
1.35
0.274
Table 6: Fitted scaling law coefficients.
A.4
Performance breakdown
Hand
NXPPhamme:
Quartization kenet: 32652 ShreodNock tile
Quantization kerne: 126x52 theadblock tile
Tritonn MXPP rearangement kemel
130
Porcansae
(1096.11005)
(11008,4156)
(4196,0000)
11009)
(31000.000)
(909)
(1905
(1100K (056)
Figure 7: Breakdown of runtime composition across three linear layer shapes of Llama-7B model, for input of batch size 64, and sequence length 512.
presents breakdown of runtime composition across three linear layer shapes in a Llama-7B model, laking the MXFP4 forward pass as an example. Each subplot shows the percentage total runtime spent in three key kernel stages: matrix multiplication, quantization-rehated operations, and rearrangement of scaling factors for the mma instruction 351
Figure 7
The figure compares three kernel configurations. The left subplot shows our fused kernel for quantization-relan operations using basie 32 x 32 threadblock tile size. The center subplot increases this tile size to 128 x 32, resulting in more efficient quantization stage. The right subplot includes a custom Triton kernel, which further improves performance by optimizing the MXFP rearrangement stage. All results normalized to 100%.
17

---

As the figure illustrates, tuning the quantization kernel significantly reduces the proportion of time spent in the quantization stage-patreee lagge-partccularll for large matrix shapes. Increasing the threadblock tile size leads to more active warps per block, enhancing arithmetic intensity and enabling better latency hiding. In CUTLASS--bar (threadblock, warp, and instruction-level tiling), which is designed to optimize data movement through shared memory and registers [45]. The Triton backend exhibits similar trends, with rearrangement overheads further reduced and matrix multiplication dominating the total runtime.
implementations, this change influences the multilevel tiling strategy
A.5 End-to-end Prefill Speedups
End to End Prefili Spoclup MXF14 FP8 RTX5090
1.50
Speedup over FP8
1.25
1.00
0.75
0.50
0.25
0.00
128
236
size (Sequence Length 256)
Figure 8: End-to-end prefill speedups for Quartet MXFP4 vs. FP8, across different batch sizes, using the 7B parameter model on a single RTX 5090.
Figure g illustrates the inference prefill speedup of MXFP4 over FP8 as function batch size, evaluated at fixed sequence length of 256 7B parameter model. The results demonstrate consistent improvement in performance using MXFP4 across all batch sizes, with speedup increasing progressively and peaking at 1.41 relative FP8 at batch size of 128, where it plateaus.
A.6 Post-Training Quantization Results
We compare the results of applying post-training quantization (PTQ) against QUARTET using the MXFP4 format the largest 7B model. For the PTQ baseline, evaluate against QUAROT (2). where the weights are quantized using GPTQ [24). To ensure a fair comparison, we introduce two key modifications the original QUAROT approach:
1. Attention Module: We remove the use of online Hadamard transformations and instead apply a fixed Hadamard transformation of size 128 to the output dimension of the v v_proj layer and the input dimension of the out proj layer. This optimization accelerates the overall process by eliminating per-head online Hadamard computations, without affecting accuracy, since we use a group size of 32 in the MXFP4 format.
2. MLP Down-Projection
For down_projection layers with non-power-of-twe dimensions in the MIP, apply grouped Hadamard transformations using the largest power-of-two size that evenly divides the intermediate dimension of the MLP.
QuaRot (PTQ)
Model Size
BF16
Quartet
7B
16.40
18.19
17.77
Table Perplexity results on C4 dataset using MXFP4 quantization. We use 128 samples from the training set (of the same dataset) the calibration set in GPTQ.
Table presents the comparison between the PTQ scheme (QuaRot) and QUARTET. QUARTET achieves a 0.42-point lower perplexity (PPL) compared to QuaRot when applied to the same model.
18

---

Notably, QUARTET is also more efficient than standard QAT methods, as it quantizes both forward and backward passes.
A.7 Compute Resources
The pre-training experiments were conducted on datacenter-grade machines with 8xH100 NVIDIA GPUs for total compute of around 6,000 GPU-hours. Although most experiments do not require such an elaborate setup, we found the 7B pre-training experiment specifically to be very DRAM- demanding and to require such specific hardware.
The speedup results were obtained on a consumer- grade NVIDIA RTX5090 GPU with total runtime of under hour.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All the claims made are supported by thorough analysis.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Section added.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Not applicable.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Setup described in full.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Full codebase with instruction included.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Setup and parameters fully described.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars reported where applicable.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Compute resources described in supplementary material.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppw...ppppp..o////////aaa
20

---

Answer: [Yes]
Justification: Verified.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper's findings are strictly technical.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper's findings are strictly technical.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Licences respected.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Documentation provided.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Not used.
21


Reviews:
[{"review_id": "J4ZKPNQgjn", "reviewer": "Reviewer_dMMD", "summary": "The paper proposes Quartet and claims it is the first to enable a full FP4 training pipeline for large language models ensuring numerical stability and delivering performance faster than FP8 on NVIDIA’s new Blackwell GPUs. The authors present a precision-aware scaling-law framework that introduces two key concepts: The \"parameter efficiency\" concepts that is linked to the forward-pass error (eff_N) and the \"data efficiency\" concepts that is linked to the backward-pass gradient bias (eff_D).  Based on these concepts, Quartet was designed with these motivations and techniques:  \n\n- Forward-pass error reduction: The authors apply a block Hadamard rotation plus QuEST clipping before quantising, spreading outliers so weights/activations fit cleanly in 4 bits.\n- Backward pass unbiased gradients: They use the same Hadamard decorrelation for backward tensors, but apply stochastic rounding to eliminate gradient bias and stabilize the learning. \n- FP4 assessment: They derive a lightweight precision-aware scaling law that predicts final accuracy from model size, data volume and bit-width, revealing the regimes where FP4 could outperform FP8.\n- Efficient kernel-level implkementation. They implement fused CUDA kernels (rotation to quantisation to scaled MXFP4 GEMM) that leverage Blackwell’s new FP4 tensor-core instruction, delivering ≈2× FP8 throughput. Using  custom CUTLASS 3.9 kernels, they fuse rotation, quantisation and MXFP4 GEMMs via the tcgen05.mma instruction.\n\nThe empirical evaluation demonstrates that Quartet was able to train Llama-style models up to 7 B parameters on C4 with no loss relative to FP8 while delivering about 2.4 × forward and 1.6 × backward kernel speed-ups.", "questions": "- Regarding the PMA proxy limits, can you provide intuition or data showing that PMA continues to correlate with convergence for larger models (> 7 B) or other tasks?\n- How would Equation (1) break if quantization error interacts non-linearly with model width or depth?\n- Your analysis of the error-bias trade-off (Figure 2c) suggests that stochastic rounding (SR) on the backward pass, as used in Quartet, is optimal for data-saturated regimes (high D/N), while round-to-nearest (RTN) may be better for compute-optimal regimes. Does this imply Quartet is primarily tuned for the \"long training\" setting? Have you considered an adaptive strategy, perhaps starting training with an RTN-based backward pass and switching to SR later, to potentially achieve optimality across all data budgets?\n- The implementation of Quartet is specialized for the MXFP4 format. The Blackwell architecture also introduces the NVFP4 format. Could you comment on the challenges and potential performance implications of adapting Quartet to NVFP4?\n- Table 3 shows that the LSS-INT4 baseline becomes unstable and diverges on longer training runs. Can you leverage your gradient analysis framework to provide any insights that might explain why leverage-score sampling on the gradients would lead to such instability?\n- Great speedup results in (Figure 3). To better understand the overheads, can you provide a more granular breakdown of the execution time for a typical linear layer? Specifically, what is the latency contribution of the fused quantization kernel (Stage 1, including the Hadamard transform) versus the GEMM kernel itself (Stage 2)?\n- The Quartet algorithm is a complex pipeline involving multiple components (Hadamard transforms, QuEST, stochastic rounding). This complexity might limit broad adoption or modification by researchers who are not experts in low-level GPU programming. Have you considered providing a higher-level Triton implementation (or at least Triton reference kernels) to lower the entry barrier?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- A technically well-written and structured paper. It was easy to read and follow.\n- Clear demonstration of stable 7 B FP4 training curves.\n- Hardware measurements demonstrated per-kernel speed-ups and back up the kernel-design claims.\n- Scaling-law framework is fitted to data and not just asserted in the paper. \n- The related work section gives a very good coverage of the INT8 → INT4 training paradigm (SwitchBack, JetFire, HALO, INT4-Transformers/LSS, LUQ). It clearly separates “noise-injection” vs. “error-minimization” QAT methods and shows how the forward path builds on them. It also connects algorithmic needs to hardware by highlighting Blackwell’s MXFP4/NVFP4 support. Citations are current (2023–2025) and well grouped, which makes the survey easy to follow.\n\nWeaknesses: \n- Only C4 perplexity is reported; no downstream or instruction-tuning tasks.\n- The accuracy table limited to 30 M parameters; larger models lack final-epoch metrics and variance analysis.\n- Speed-up numbers are kernel-level, there are no end-to-end wall-clock or energy-per-token measurements.\n- No statistical error bars or standar deviation. The single-run results may not be statistically significant.\n- Baselines sometimes diverge. It is unclear if with better tuning the authors might close the gap.\n- The optimizer still uses FP32 so the “end-to-end FP4” efficiency claims are not fully validated.\n- A big portion of the work performance advantage is derived from the highly specialized kernel’s implementation based on the NVIDIA Blackwell architecture and its MXFP4 format. While this makes the work practical/ impactful for this type of hardware, it also presents a limitation. The exact implementation and performance gains may not be directly transferable to other hardware platforms without similar native support for these specific operations. \n- The proposed scaling law is an empirical model fitted on a specific architecture (Llama) and dataset (C4). The paper did not look at downstream tasks or other models. The idea that backward-pass precision primarily affects the data efficiency term is compelling but would benefit from further theoretical grounding or broader empirical validation across more diverse tasks and model families.\n- In related work section, there are some notable omissions that would be good to add:\n   - Native FP8 training (Micikevicius ’22, Fishman ’24). This critical because FP8 is the paper’s “lossless” baseline.\n   - Optimizer-state compression (8-bit Adam, ZeRO-3 8-bit states) is relevant to any “end-to-end FP4” claim.\n   - Foundational stochastic-rounding papers (Gupta ’15; DoReFa-Net ’16)  would complete the historical context for SR.\n   - Sparsity/quantisation scaling laws (Frantar ’24, ’25)  are closely related to this paper’s precision scaling law.\n   - Extreme low-bit LLMs (BitNet, 1-bit)  work is useful to situate Quartet against the lower-than-4-bit frontier.\n   - SmoothQuant / AWQ equalisation  are other mainstream outlier-mitigation routes beyond Hadamard rotations.\n   - FP8 kernel / Hopper Tensor Core literature. Adding a short comparison would clarify how the paper’s Blackwell FP4 kernels improve over existing FP8 paths.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and address their questions in their original order.\n\n> Only C4 perplexity is reported…\n\nWe note that we determine optimality based on pair-wise evaluation loss comparisons. That means that as long as downstream (e.g. zero-shot) performance is consistent with loss, the comparisons remain valid. To demonstrate this consistency, we present 0-shot evaluations of the trained model in the table below:\n\n| N     | D    | Method  | C4 Val Loss | HellaSWAG | WinoGrande | Arc‑Easy |\n|-------|------|---------|-------------|-----------|------------|----------|\n| 1600M | 160B | Quartet | 2.410       | 41.66%    | 54.85%     | 54.71%   |\n| 800M  | 80B  | Quartet | 2.526       | 37.75%    | 50.27%     | 49.53%   |\n| 430M  | 43B  | Quartet | 2.642       | 34.58%    | 50.75%     | 49.03%   |\n| 200M  | 20B  | Quartet | 2.814       | 31.26%    | 50.36%     | 43.56%   |\n| 800M  | 80B  | FP8     | 2.461       | 39.51%    | 53.91%     | 53.28%   |\n| 430M  | 43B  | FP8     | 2.571       | 36.21%    | 51.78%     | 48.95%   |\n| 200M  | 20B  | FP8     | 2.736       | 32.77%    | 50.59%     | 46.21%   |\n| 100M  | 10B  | FP8     | 2.886       | 30.21%    | 51.30%     | 43.27%   |\n\n\nOne can see that the zero-shot results show that the models provide non-random performance, and are consistent with the evaluation loss across the training methods.\n\n\n> Regarding the PMA proxy limits…\n\nThe effect can be observed from two sides:\n 1. The statistical effect of magnitude alignment.\n 2. The fact that RTN is better for shorter training and SR - for longer.\n\nThe first effect is expected to persist regardless of model size as it is a statistical property of quantized matrix multiplication during the backpropagation. The latter can be observed in the training runs of Tseng et al. [1] (see their Figure 5): one can observe that RTN converges faster in the beginning, but is overtaken by SR during the later stages of the training. As such, we claim that PMA effects will persist at that scale.\n\n> How would Equation (1) break if quantization error interacts non-linearly with model width or depth?\n\nEquation (1) models the loss for a particular training setup, determined, among everything else, by model hyper-parameter (depth, width, LR) scaling with parameter-count. In our experiments, we scaled model depth and width proportionally, leading to depth \\~ width \\~ N^(⅓). One can notice, that if quantization had separate multiplicative effects on depth and width, it could still be represented as a multiplicative factor on N. To separate these effects, one would need to construct a scaling law that would decouple depth from width and accurately model the effect of quantization as a function of these two parameters. That could possibly lead to an alternative depth-to-width strategy in the presence of quantization. That would constitute an interesting future research direction.\n\n> The implementation of Quartet is specialized for the MXFP4 format. \n\nWe chose the MXFP4 format as it is the standard OCP Microscaling Format, with broader possible adoption, since it may also be supported by AMD devices. \nHowever, our kernel design is compatible with NVFP as well, and should perform similarly in terms of performance, and slightly better in terms of accuracy (since the scale quantization is more flexible in NVFP). \nWe are working on an NVFP version of Quartet. \n\nAlthough NVFP4 could provide slightly higher accuracy due to smaller group sizes and finer scales, we expect it to perform similarly to MXFP4 due to similar  distributional properties of the underlying FP4 format that were shown to be connected to efficiencies in Table 2. Additionally, limited range of the FP8 shared exponents of NVFP4 could create problems for the backward pass with small gradient norm, requiring additional scaling techniques to prevent underflows.\n\n> Table 3 shows that the LSS-INT4 baseline becomes unstable…\n\nIn our experiments, this baseline doesn’t simply diverge, but rather produces NaN iterations out of nowhere and without prior loss spikes. Its performance being close to Quartet while stable and the use of an unbiased backward pass indicate good alignment. We theorize that the lack of Hadamard preprocessing on the gradient outliers in the backward pass is what’s leading to overflows for LSS-INT4. Determining the exact cause would require extensive debugging of their implementation, which is unfortunately beyond the scope of our work. \n\n> Great speedup results in (Figure 3)...\n\nFrom our benchmarks, quantization-related operations constitute 20-40% of latency depending on the batch size and layer shape. We provide the breakdown for certain matrix shapes in “A.3 Performance breakdown” in Appendix.pdf in supplementary materials. Additionally, see response to reviewer **PgVh** for slightly adjusted performance measurements.\n\n> … providing a higher-level Triton implementation …\n\nWe do provide a full reference Triton pseudo-quantization implementation in `./notebooks/benchmark_mxfp4.ipynb` in supplementary materials. In fact, we used it for some of the training runs because they were performed on Hopper-generation machines without hardware FP4 support.\n\n[1] https://arxiv.org/abs/2502.20586"}, {"review_id": "Dy9MpbhGWI", "reviewer": "Reviewer_PgVh", "summary": "The authors propose quartet, a recipe to train models. Furthermore, they propose parameter efficiency and data efficiency metrics, in tight relation to the forward pass precision and backward pass precision. They utilize stochastic rounding and MXFP4 format on the latest Blackwell GPUs and attempt to come up with a pure MXFP4 based training recipe.", "questions": "1. Do you have data points for Table 3 with FP4-FP8, FP8-FP4, FP8-FP8 precisions in the forward and backward passes respectively?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n* The paper utilizes the strengths of existing methods, such as utilizing Quest in the forward pass and stochastic rounding in the backward pass.\n* The parameter efficiency and data efficiency definitions make sense and the paper is sound overall.\n\nWeaknesses:\n* The comparison is focused on methods that solely utilize 4-bit precision, Table 3 could have included an MXFP8 baseline so that we can see how much quality loss is incurred by choosing to use 4-bit precision matmuls everywhere.", "was_revised": true, "final_justification": "I have read the other reviews and the rebuttal by the authors. My score stays the same due to speedup concerns over fp8 baseline.", "rebuttal": "We would like to clarify the use of FP8 as a baseline in this paper: \n\n - From the accuracy perspective, we reported the BF16 results as an upper bound on FP8 accuracy whenever we needed to compare against FP8. This is a more-than-fair comparison, since FP8 methods do not outperform BF16 precision training, while some methods were shown to match it. \nWe thus believe that this is a fair comparison against the best accuracy results FP8 training could possibly offer.\n\n\n - From the speedup perspective, our intention was to compare against an ideal FP8 baseline method that only performed FP8 cast + GEMM, which would ignore any overheads due to error mitigation and would represent the fastest these FP8 kernels can be. \n\nHowever, after the submission deadline, we have realized that our evaluation induced a small but unnecessary performance overhead in the FP8 benchmarking relative to the ideal version described above. \nAfter repeating all the runs for the FP8 baseline, the impact of removing this overhead is the following: \n\n- On the smaller (800M) model, the ideal FP8 version reaches 20% higher inference throughput than in our submission, and 25% higher training throughput. \nFor instance, our inference speedup relative to FP8 for this model size is reduced 1.9x to 1.6x. \n- On the larger (52B) model, the ideal FP8 version reaches **the same** inference throughput as in our submission, and <= 10% higher training throughput. \nOur end-to-end speedup stays roughly the same at this model size. \n  \nWe apologize for this error, but stress that the performance difference is small enough that it does not significantly affect our FP4 optimality claims: in particular, the optimal training precision for the Llama and Qwen models would still be FP4, whereas the Gemma3 model remains borderline.   \n\n\nTo sum up, our FP8 baseline aimed to **upper bound both the accuracy and the fastest possible performance of FP8 training**, representing the performance of an ideal FP8 training method. \n\n\nWe thank the reviewer for their perspective, apologize for the error, and promise to add a more thorough explanation of our comparison setup and its implications. \n\n> Do you have data points for Table 3…\n\nAll the data points and the code to produce all the figures, including Figure 1 (a) that shows the data you requested, are present in `./notebooks/plots.ipynb` in the supplementary materials."}, {"review_id": "kUOscFXAqh", "reviewer": "Reviewer_dhqV", "summary": "This work presents Quartet, an end-to-end FP4 training recipe for all linear layers. Achieves state-of-the-art accuracy for billion-scale LLMs using FP4 only, measured by the data and parameter efficiency in the compute optimal scaling law proposed by this work. The scaling law that models how accuracy scales with forward/backward pass precision, capturing both parameter efficiency and data efficiency. Specifically, it:\n- in the forward pass: Uses QuEST quantization (based on Hadamard transform and RMSE clipping) for minimal quantization error.\n- in the backward pass: Uses stochastic rounding to reduce gradient bias and improve convergence in long training runs.\n\nWith efficient GPU Implementation, it leads to 1.8× training speedup overall compared to fp8, with higher accuracy than prior FP4/INT4 methods (e.g., Jetfire, HALO, LUQ) on model size up to 7B.", "questions": "- what is the portion of time spent on attention forward/backward operations in your benchmark, and if they're done in bf16, or fp8, or fp4?", "limitations": "- limited to the llama architecture. not verified on MoE, or other non-transformer architectures.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "strength:\n- novel framework for analyzing the performance of low-precision algorithm via scaling law. \n- good convergence result with fp4\n- it shares the insight that the backward pass may be impacted by bias in gradient estimation and it's important to correct it\n- if the code will be open sourced as stated, it will be quite helpful for the community\n\nweakness:\n- lack of clarity: does quartet perform the all attention matmuls in mixfp4 as well? \n- insufficient hyper-parameter and model architecture sensitivity analysis: it looks like the experiment only involve the llama architecture, which is quite limited. It's covered in the literature that model architecture affects the quantization performance, for instance swiglu activation, or the position of the norm layers [1][2]. It's unclear if the proposed method works across different model architecture variant other than llama. \n- it would be helpful to introduce the definition of mxfp4 format, for audience who is not familiar with the number representation\n- So far the exploration for fp8 attention training is still ongoing. It's unclear if there's any caveat when applying mxfp4 to the attention layer, especially for long sequences training. The context length used for training is not specified in the paper either.\n\n[1] https://arxiv.org/abs/2405.19279 \n[2] https://arxiv.org/abs/2409.12517", "was_revised": true, "final_justification": "My concerns are partially addressed. However, the context window 512 is much shorter than typical foundation model pre-training which requires at least 4k or 8k. I think overall this work has good merit but the speed up factor may not be as high as claimed in the paper in real world scenarios.", "rebuttal": "We are grateful to the reviewer for their work.\nTo address their concerns, we highlight that we performed quantization of linear operation in transformers, retaining attention operations in high (BF16) precision. More specifically, we isolated the quantization logic inside the linear layers, retaining all the remaining operations in the architecture in their original form.\n\nThis was motivated since:\n - Linear layers’ are the most computationally expensive part of LLM training and inference for short-to-medium context training and inference.\n - Work on low-precision attention and efficient long-context modeling is orthogonal to linear layer speedups.\nThe training sequence length of 512 tokens that we utilized and the subsequent omission of the attention operation from individual latency benchmarks reflects this choice.\n\nWe acknowledge this limitation of our work, and we will include it in the discussion, but we emphasize that we consider the effect of attention quantization and its impact on long-context training an orthogonal research direction. (In fact, since submission time, orthogonal work on FP4 attention was posted independently [5].)  \n\n*Regarding architectural verification*, we note that our setup closely follows experimental design of prior work on precision scaling laws, notably Kumar et al. [1], Panferov et al. [2], Tseng et al. [3] and Busbridge et al. [4].\nAll these references focus on dense (non-MoE) Llama-like transformers. \n\nWe emphasize that the proposed principles for scaling-law-based and iso-compute comparisons can be directly applied to other model architectures, such as MoE, but that would require designing, verifying and fitting of novel scaling law forms. As such, we leave this for future work.\n\n\n[1] https://arxiv.org/abs/2411.04330 [2] https://arxiv.org/abs/2502.05003 [3] https://arxiv.org/abs/2502.20586 [4] https://arxiv.org/abs/2502.08606 \n[5] https://arxiv.org/abs/2505.11594"}, {"review_id": "uuvKzVcOZS", "reviewer": "Reviewer_7M4U", "summary": "This paper introduces an algorithm called Quartet, which allows for FP4 training on the NVIDIA Blackwell architecture. Background and motivation is provided, including related works. In order to develop Quartet, the authors claim 4 contributions. A scaling method comparing quality parameters for quantized training methods, a way to relate these quality metrics to concrete training algorithms via certain [novel] methods, an optimized implementation of Quartet on modern hardware, and experimental evaluation against previous baselines.", "questions": "### Major\n\nCan you clarify how you arrived at the scaling law in section 4.1?\n\nCan you clarify section 4.2, especially from lines 233 to the effective loss formula?\n\nWhy was LSQ omitted in Table 2?\n\n### Minor\n\nThe NVIDIA 5090 GPU is a slightly different compute capability to B200 and GB200 cards. Are there any differences here in terms of the FP4 capabilities?\n\nAs the paper is focused on using a specific accelerator architecture (NVIDIA Blackwell) why not focus on NVFP4?\n\nIn line 387, how are the baselines implemented? For example, are they both using stochastic rounding? Is this actually using a framework like Megatron or Deepspeed?\n\nWere any loss spikes present during training?\n\nFigure 1: Isn’t the scaling law fit dependent on the training method used for those precisions? Why isn’t this specified?\n\n[1] NVIDIA, CUDA GPU Compute Capability, https://developer.nvidia.com/cuda-gpus", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I would first like to thank the authors for submitting their work to NeurIPS 2025.\n\n### Pros\n\nThis paper targets a relevant and important area today, which is low precision/quantized training. The paper starts with a well written problem statement and primer to low precision training and inference. The claimed contributions are clearly stated.\n\nOverall, the related work section also gives a concise but useful description of prior works.\n\nBackground sections on quantization grids, quantization granularity, rounding, architecture support, and pre-training are clear and informative.\n\nSection 4 is again well written in a smart manner with the four different “ingredients” of Quartet. The structure of the scaling law in 4.1 at least makes intuitive sense as well.\n\nSection 4.3 experiments are well constructed and targeted and overall easy to understand. This is primarily due to the introduction in this section of the misalignment metric, as well as Table 2 which makes it easy to see the trade off between $\\text{eff}_N$ and $\\text{eff}^*_D$ for the different methods.\n\nSection 4.4 is clearly well thought out, and represents a significant contribution given the non-triviality of implementing these kinds of low precision operators in an efficient way on hardware. Also, Algorithm 1 is compact but descriptive, and I feel that it would help if someone were to independently attempt to reproduce the implementation.\n\nAlthough the evaluations are quite concise, it is satisfactory in terms of validating the method presented, going over speedup, loss curve, $\\text{eff}_N$/$\\text{eff}_D$ improvement over previous methods.\n\n### Cons\n\nLines 93-94: Treating FP8 as the lossless baseline is fine, but the reason should be clearly stated in that sentence. There is also no discussion of “master weights” techniques, which appears to have been used for MXFP4 previously [1] that some readers may be familiar with.\n\nIn section 3, outlier mitigation, it isn’t clear why the grouped hadamard transform helps. Although there is some background on the transform itself, it is less clear how this translates to helping with outliers.\n\nIn section 4.1, it should be made clear that the general form of scaling-law (1) is built on prior work [8] in the main paper as opposed to the supplementary material and that that form is being assumed for that reason.\n\nSection 4.2 was quite confusing. My understanding is, In lines 233-234, $N_{max}$ is supposed to be the forward pass parameter budget, which influences the compute budget, but isn’t the compute budget itself. And the training budget similarly is based on $D_{max}$. Speedups are clear, but it would be better to show how $\\text{spfw}$, $\\text{spbw}$, $\\text{sptr}$, are functions of $P_{fwd}$ and $P_{bwd}$ given this simplified BOPS setting. Some derivations here would be good.\n\nSection 4.3 should include some information on the experimental setting itself. Also, it’s not clear why LSQ is omitted from Table 2.\n\nAlthough it’s mentioned as a future extension, it would be good to include at least some discussion regarding larger model sizes. 7B is on the smaller size today, and it’s not clear how Quartet (especially the scaling law) would generalize to larger model sizes where the same scaling may not hold.\n\nThe evaluation doesn’t include any other model architectures (mainly MoE), but this can be excused due to (presumably) the difficulty of writing those kernels.\n\n---\n\nOverall, I find that the work is polished, and its significance, originality, and quality to be high. I have some issue with the clarity in the (fairly critical) sections 4.1 and 4.2 which the majority of the paper relies on.\n\n[1] Rouhani et. al, Microscaling Data Formats for Deep Learning, https://arxiv.org/pdf/2310.10537\n\n#### Other Comments\n\nI would have liked to see some experiments on the memory footprint of the new, lower precision implementation. But it’s fine not to include since that’s not the focus of the paper.\n\nNits:\n\nLine 148: To be pedantic: *normalized* Hadamard matrices have that recursive property. Good to repeat there.", "was_revised": true, "final_justification": "This paper is a solid addition to the area of low precision training, which is a timely and important topic. The paper is clear and well written, while being novel, significant and overall well justified by experiments and theory. I believe this will be of interest to many machine learning researchers and practitioners.", "rebuttal": "We thank the reviewer for providing an extensive evaluation of our work and provide the requested clarifications:\n\n> Can you clarify how you arrived at the scaling law in section 4.1?\n\nConceptually, we proposed this form to unify the laws proposed for forward-only compression by Kumar et al. [4], and the theoretical analysis of training with noisy gradients of e.g. QSGD [2], which suggest a separation of the impact of low precision on the forward and backward passes, in the parameter and data terms, respectively. \nPractically, the ablation for the general form is presented in Appendix A.2 in the supplementary material. In short, the A,B,E,$\\alpha$,$\\beta$,$\\gamma$ form had by far the lowest fit error and was originally tested (by Busbridge et al.[1]) for models up to 7B parameters. This verified that it’s accurate and robust. The inclusion of eff$_N$ models fundamental reduction in capability, modeled similarly in numerous works cited around line 193 of our submission. On the contrary, eff$_D$ implies that the quantized gradient estimation allows for (albeit slower) convergence to the same loss, which aligns with the theoretical analysis of noisy gradient training [2], and the closing of the performance gap on Figure 1 (a).\n\n> Can you clarify section 4.2, especially from lines 233 to the effective loss formula?\n\nAssuming baseline FP8 training and inference, parameter count $N_{max}$ corresponds 1 to 1 to a certain inference cost (latency). Additionally, under fixed model size, training compute can be expressed in training corpus size $D_{max}$ (compute ~ $N_{max} \\cdot  D_{max}$) or data saturation ratio $D_{max}/N_{max}$. Combined, these two quantities constitute a bias in the model cost functional space. Assuming certain inference and training speedups (spfw, sptr) for a certain training scheme and model sizes, various schemes can be mapped into that space for iso-compute comparisons. The effective loss formula presents cost-normalized parameter count and corpus size. Figure 1 (b, c) presents the regions in that space where certain precisions yield lower loss.\n\n> Why was LSQ omitted in Table 2?\n\nLSQ was shown by Panferov et al.[3] to consistently yield higher loss and ~20% lower eff$_N$ than QuEST. We chose the more modern and better performing method as the candidate. Additionally, LSQ can’t be directly applied to backward pass quantization as it has learnable parameters of its own. We will clarify this. \n\n> The NVIDIA 5090 GPU is a slightly different compute capability to B200 and GB200 cards. Are there any differences here in terms of the FP4 capabilities?\n\nYes. According to the Blackwell data sheet [4], we have that: \n - CC=120 (RTX 5090) compared to CC=100 (B200) has a higher relative speedup of FP4 compared to FP8 (4x in theory vs 2x in theory).\n - CC=120 (RTX 5090) compared to CC=100 (B200) doesn’t offer a stochastic rounding instruction.\n\nThe first fact means that FP4 would have a potentially lower performance “ceiling” on the B200 relative to the 5090. These aspects were not well-documented at the time of submission. We’ll make sure to include them in the discussion. We are currently working on a B200 version of our kernels.\n\n> As the paper is focused on using a specific accelerator architecture (NVIDIA Blackwell) why not focus on NVFP4?\n\n\nWe chose MXFP4 specifically to not focus exclusively on Blackwell: MXFP4, being the standard OCP Microscaling Format, will see wider adoption among AI accelerator hardware. For instance, it will be supported on AMD hardware as well, whereas NVFP4 will (most likely) not be supported. \n\n\nAt the same time, our kernel design does apply to NVFP as well. Specifically, we have already extended our forward kernels to NVFP4 (see response to reviewer **dMMD**) and plan to do so for the backward pass as well. \n\n> In line 387, how are the baselines implemented? For example, are they both using stochastic rounding? Is this actually using a framework like Megatron or Deepspeed?\n\nThe baselines are implemented as either custom GEMM CUDA kernels for FP8 or basic `torch.nn.functional.linear` for BF16.\nFor hardware support reasons discussed above, RTN instructions were used instead of stochastic rounding.\nThe speedups are reported for quantized forward and backward passes over linear layers that would constitute models of certain size, effectively omitting non-linear operations and optimization steps from the comparison. They are expected to have negligible impact for larger models.\n\n> Were any loss spikes present during training?\n\nWe observed no spikes in either loss or gradient norm. Figure 3 (c) shows a sample of the training dynamics.\n\n> Figure 1: Isn’t the scaling law fit dependent on the training method used for those precisions? Why isn’t this specified?\n\nWe acknowledge this question and will clarify. The parameter and data efficiencies eff$_N$ and eff$_D$ indeed depend not only on the precision but generally the quantization scheme (method+precision). That’s why it can be used to both compare different methods and different precisions for the same method. We’ll adjust the wording to make it clearer.\n\n[1] https://arxiv.org/abs/2502.08606 [2] https://arxiv.org/abs/1610.02132 [3] https://arxiv.org/abs/2502.05003 [4] https://images.nvidia.com/aem-dam/Solutions/geforce/blackwell/nvidia-rtx-blackwell-gpu-architecture.pdf"}]

Author rebuttal:

Meta-review: This paper introduces an algorithm called Quartet, which allows for FP4 training on the NVIDIA Blackwell architecture. All reviewers agree that this work makes solid contributions, and the AC recommends acceptance.
Decision: Accept (poster)