FP4 All the Way: Fully Quantized Training of Large Language Models
Authors: Brian Chmiel
Maxim Fishman
Ron Banner
Daniel Soudry
Venue: NeurIPS 2025 spotlight ()
Keywords: Quantization
Low precision
Training
FP4
Accelerations
Compression
Primary area: deep_learning
TL;DR: We demonstrate for the first time, fully quantized training of  a 7B LLM using FP4 format.

Abstract: We demonstrate, for the first time, fully quantized training (FQT) of large language models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients on datasets up to 200 billion tokens. We extensively investigate key design choices for FP4, including block sizes, scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2M1) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and round-to-nearest for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately $\sqrt{3}$ times the quantization noise, quantized training becomes less effective. Leveraging these insights, we successfully train a 7-billion-parameter model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to a standard BF16 baseline, confirming that FP4 training is a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in https://github.com/Anonymous1252022/fp4-all-the-way.

FP4 All the Way: Fully Quantized Training of LLMs
Brian Chmiel A+t Maxim Fishman A+t
Ron Banner At Daniel Soudry e
ANiiiia, Israel
DDeparmment of Electrical and Computer Engineering: Technion, Haifa, Israel
brianchmiel,
mazim. faxim.fishmm fishman.d d
daniel.soudry soudry) ronbanner RB,
Abstract
We demonstrate, for the first time, fully quantized training (FQT) of large lan- guage models (LLMs) using predominantly 4-bit floating-point (FP4) precision for weights, activations, and gradients datasets up IT tokens. We extensively investigate key design choices for FP4, including block sizes. scaling formats, and rounding methods. Our analysis shows that the NVFP4 format, where each block of 16 FP4 values (E2MI) shares a scale represented in E4M3, provides optimal results. We use stochastic rounding for backward and update passes and cound-to-nearee for the forward pass to enhance stability. Additionally, we identify a theoretical and empirical threshold for effective quantized training: when the gradient norm falls below approximately v3 times the quantization noise, quan- tized training becomes less effective. Leveraging these insights, we successfully train 7-billion-panl parameeer model on 256 Intel Gaudi2 accelerators. The resulting FP4-trained model achieves downstream task performance comparable to stan- dard BF16 baseline, confirming that FP4 training a practical and highly efficient approach for large-scale LLM training. A reference implementation is supplied in htttps://githhe com/ Ant///oonommuuumooomommom 12 ft/p///ff4aallaa all--alltthhe -the-wy
Introduction
The rapid advancement of Large Language Models (LLMs) has led unprecedented breakthroughs in natural language understanding and generation. State-of-the-art models now scale to hundreds of billions of parameters, enabling remarkable capabilities across diverse applications. However, this progress comes at a significant cost, with training and inference demanding immense computational power and memory bandwidth. As model sizes grow, hardware constraints become major bottleneck. necessitating innovations in numerical precision and memory-efficient architectures.
Until recently, the dominant numerical format for pretraining Large Language Models (LLMs) was BF16, which provided a balance between precision and efficiency. However, as model sizes and dataset scales have grown, researchers have explored lower lower-precision alternatives to improve compu- tational efficiency and reduce memory requirements. few pioneering studies have demonstrated that full training in FP8 is not only feasible but also effective. [K3 showcased the potential of FP8 training on small-scale datasets (100B). g extended it to trillions of tokens dataset with 7B parameters model, and ( demonstrated FP8's viability 671B parameter Mfxture-of--Expert (MoE) model on a vast dataset, achieving state-of-the-ard results. FP8 quickly emerging as the new standard for large-scale LLM training. As research continues to push precision boundaries further, the next logical step is exploring FP4, which promises even greater efficiency while maintaining training stability and model accuracy.
even larger scale, successfully training massive
'Equal contribution
Work done while at Intel.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recently, NVIDIA introduced its new GPU architecture, Blackwell M. marking a major milestone as the first GPU to support FP4 matrix multiplication hardware. This advancement enables nearly 2x acceleration in matrix multiplication throughput compared to FP8, significantly boosting performance and energy efficiency for large-scale deep learning workloads. Blackwell supports two distinct FP4 MXXPP and NVFP4 each with different design choices in block size and scale encoding. In this paper, we aim to systematically investigate the full range of block sizes and scaling formats for FP4 training, analyzing their impact accuracy and stability and highlighting the advantages of the NVFP4 format.
Beyond the choice of numerical format and block size, another critical aspect of low- wwwpreccsioo quanti- zation is the rounding mode used during quantization. The two most commonly used rounding modes are round- cound-to-neddd to-neerest (RIN), which deterministically maps each value to its closest representable value, and stochastic rounding (SR), which prohabilistically rounds based on the distance to neighboring values to reduce quantization bias. While prior empirical work [19] has highlighted the importance of SR, particularly in the backward pass for stabilizing gradient updates, benefits have largely been observed heuristically. In this work, we provide analysis of noisy gradients training and identify the point where they stop being effective and show the importance of SR in low-precision training.
Recent studies have explored the use of FP4 formats for training large language models, yet none achieved full quantization across all key components-weiigee weights, activations, and gradients. In contrast, our approach introduces. for the first time, fully quantized FP4 training framework that addresses all these components simultaneously, and on significantly larger datasets. Table 2 highlighhs these important distinction. This end-to-end FP4 training setup enables us evaluate the practical viability of low-precision methods at scale. Moreover, we evaluate our FP4 models on various downstream tasks showing on-par results with the BF16 baseline-demonstraaee that efficiency gains do not come at the cost of quality.
We make several key contributions:
1. FP4 Format Optimization. We conduct comprehensive experiments on block size and scal- ing formats for FP4 training, revealing that block sizes below 16 elements offer diminishing returns on accuracy. Our comparison of different exponent-mantisn configurations (E1 M6 through E8M0) found the best performance was achieved by E4M3 the format used in NVFP4, validating NVIDIA's hardware design choices.
2. Split Rounding Strategy. We establish that combining stochastic rounding in the backward pass with round-to-nearest in the forward pass significantly improves training stability and final model accuracy in FP4 training compared to using either method alone throughout the training process.
3. Precision Transition Analysis. We provide theoretical framework that identifies the critical point at which FP4 precision becomes less effective for continued training progress. Specifically, when the full- full-precision gradient standard deviation falls below v/ times the quantization noise standard deviation. We suggest, at the end of the training, to apply higher precision Quantization Aware Finetuning (QAF) to increase the signal- signal-to-nois to-sose ratio higher above this threshold and quickly converge to the baseline.
4. End-to-Enn Large-Scale FP4 Training. We successfully train 7B- 1B-parameter LLM entirely in FP4 precision on a trillion tokens, using 256 Intel Gaudi2 accelcrators. While a slight gap in final training loss is observed compared to the BF16 bascline, it is fully closed through the brief QAF phase. This leads to downstream task performance on par with BF16, confirming the practical viability of FP4 for
large-scale applications.
2
Related work
Quantization is a key area of research in the effort to compress neural networks for more efficient deployment and reduced resource consumption. The two predominant approaches in this space are Post- Dost-Trainin Quantization (PTQ) [22]1 9 and Quantization Aware Training (QAT) 7 3 12]. PTQ focuses on converting pre-trained models to low-bit representations without additional training, making it attractive for rapid deployment, especially in inference scenarios. In contrast, QAT incorporates quantization effects during model training or fine-tuning, enabling the network to adapt and maintain accuracy under oo-precciioon constraints. Both methods have seen significant

---

advancements, with recent work demonstrating competitive performance at 4-bit precision 9 and below (20].
Fully Quantized Training (FQT) is a more challenging task than PTQ or QAT, as it requires training from scratch with low-precision weights, activations, and gradients to accelerate all matrix multiplica- tions. Until recently, applying FQT beyond 16-bit precision was considered difficult due to instability
and convergence issues. However, recent works have demonstrated its feasibility. [13] presented the first FQT of a large language model in FP8 on a dataset of up to 100 billion tokens. [8] extended this to
2 trillion tokens, revealing stability issues in later training stages and proposing modified activation function to address them. [6] further advanced the field by training large Mixture of offeure--f-- (MoE) model with FP8 FQT, mitigating instability through finer-grained quantization.
Finer granularity guantization is emerging as a key direction for enabling Fully Quantized Training (FQT) beyond FP8, particularly in the context of FP4 precision. By reducing the quantization block size, these methods aim to better capture local variations in data distributions, improving stability and accuracy. Notable examples include MXFP4 [15) and NVFP4 [
The two works most closely related to ours are 21 19 models using vector-wise FP4 format combined with two key techniques: Differentiable Gradient Estimator (DGE) to replace the standard Straight- Straight-Through Through Estimator (STE), and Outlier Clamp Compensation (OCC) to handle activation outliers wih additional sparse residual matrix. Their models are trained on up to 100 billion tokens, but they quantize only weights and activations, keeping gradients in higher precision-thus accelerating only of the three matrix multiplications involved in training. [9], in contrast, focuses on gradient quantization using the MXFP4 format and applies stochastic rounding alongside the Hadamard transform stabilize training. They train models up to 40 billion tokens. However, like (21]. they only accelcrate part of the three matrix multiplications. In contrast, our work is the first to demonstrate full FP4 Fully Quantized Training (FQT) of large-scale LLMs, enabling the acceleration of all matrix multiplications during training.
21 proposes training large language
FP4 training
Going beyond FP8 to FP4 training presents significant challenges due to the limited dynamic range of FP4, making it difficult to capture the full variability of activations and gradients without excessive quantization error. However, the recently introduced microscaling floating-poin family (MXFP) [5] offers promising alternative by dynamically adjusting the scale at finer granularity, mitigating precision loss.
MXFP4, includes sign bit, 2 exponent bits, and mantissa bit (E2M1) is floating-point format that enhances low-precision training by dividing data into blocks of size 32, with each block sharing a common scale. The scale for each block is stored using the E8MO format, an 8-bit exponent-only representation that provides wide dynamic range without a mantissa and sign. Another potential format for FP4 training is NVFP4, which uses the same E2MI data representation as MXFP4 but differs in block size and scaling format. NVFP4 divides data into smaller blocks of size 16, compared to MXFP4's s 32, allowing for finer- mher-grained scaling adjustments. Additionally, NVFP4 employs an E4M3 format for storing scales, providing a balance between dynamic range and precision. Both MXFP4 and NVFP4 are supported in NVIDIA's Blackwell architecture [1. In Table [ we compare these 2 formats.
Table I: Comparison of MXFP4 and NVFP4 Formats
Datatype
MXFP4
NVFP4
Data Representation
E2M1
E2M1
Block Size
32
16
Scale Format
E8MO
F4M3
Per-Tensor Scale
No
Yes
3

---

3.1 Exploring block size and scale format
Seeing the differences between the two FP4 formats supported in Backwelll-MFFP and NVFP4-1 in terms of block size and scale format, we decided to investigate their impact further. Specifically, we aim to compare the full range of possible seale formats, while maintaining the FP8 data format. Additionally, we explore different block sizes to understand their effect on numerical stability, training efficiency, and model accuracy. This analysis will provide deeper insights into the trade-offs between dynamic range, precision, and computational efficiency in low-precision training.
In Fig. we train a 350M Llama-style model with FP4 format (E2MI) with block size 16 and different scaling formats. Note that, similar to NVFP4, most configurations (except for ESMO, which corresponds to MXFP4) do not utilize the sign bit the scale. This may represent a potential inefficiency that future work could aim to exploit. We notice that the best results are achieved with E3M4 and E4M3, where the latter is used in the NVFP4 format. In Fig. we compare different block sizes both with scales formats E8MO and F4M3, which are the scales used in MXFP4 and NVFP4, respectively. Note that while selecting the appropriate scale has a significant impact on final accuracy-for for example, M6 leads to complete divergence trereenceethheee block size has more modest effect, with smaller block sizes generally yielding better results. This leads us to proceed with the NVFP4 format, which uses a block size of 16 and a scale format of E4M3.
4.0
Baseline
E1M6
3.8
E2M5
E3M4
3.6
E4M3
E5M2
Loss
E6M1
3.4
E8M0
3.2
3.0
2.8
0.0
2.5
5.0 7.5 10.0 12.5 15.0 17.5 20.0
Proceed tokens (Billions)
Figure 1: Formats E4M3 (used in NVFP4) and E3M4 achieved the best results. Comparison of different scaling formats (E1M6, E2M5, E3M4, E4M3, E5M2, E6M1, E8M0) when training 350M Llama model using FP4 format (E2MI) with block size 16. The formats E3M4 and E4M3 achieve the best results (recall E4M3 is used in NVFP4), whereas M1 results in complete divergence.
3.2 Exploring the rounding modes
After completing our exploration of the various block sizes and scales within the FP4 format, we now turn our attention another critical aspect: the choice of rounding modes. In the next section, we will delve into the different rounding strategies available for FP4 and analyze their impact on numerical stability and model performance.
Fully quantized training encompasses the quantization of three key general matell-mtttimmmiee mattiiiioulisilsaaionns (GEMMs): forward, backward, and update. Each GEMM involves two quantized operands opeuands-resulting six distinct quantization points across the training pipeline:
[Forward] 21 Q(Wi)Q(a--);
a= f(()))
(1)
[Backward]
|t-1 Q(W?)Q(0)) Q(()); b= (() 0g
(2)
ac Q(8))(LL)
[Update]
(3)
aw;

---

4.0
4.0
Baseline
Baseline
3.8
Block
3.8
Block 64
Block
Block 32
3.6
Block
3.6
Block 16
Block 8
Block
Loss
Loss
3.4
3.2
3.2
3.0
3.0
2.8
2.8
0.0
2.5
5.0
7.5
10.0 12.5 15.0 17.5
20.0
0.0
2.5
5.0
7.5
10.0 12.5 15.0 17.5 20.0
Processed tokens (Billions)
Processed tokens (Billions)
(a)
(b)
Figure 2: Block size 16 is the best option. We examine the impact different block sizes (8, 16, 32, 64, 128) training accuracy using scaling formats: (a) E8M0 (used in MXFP4) and (b) E4M3 (used in NVFP4). Smaller block sizes yield modest improvements in accuracy, with diminishing returns below 16 elements per block. Thus, block size of 16 provides optimal compromise between performance and computational overhead.
where is the loss function, Q a quantization operation, @ is a component-wise product and, in each layer 1, fi is the activation function, Wi is weight matrix, žι are the pre-activations, and A ac Ba."
3.20
3.15
3.10
Loss
3.05
Full RtN
Weights Forward SR Activations Forward
3.00
Weights Backward SR
Activations Update SR
2.95
Grad Update SR
Grad Backward SR
2.90
6
13 14 15 Proceed tokens (Billions)
8
9
10
11
12
Figure 3: Comparison of different rounding schemes when training a 350M Llama model using NVFP4 format. In each graph, we apply SR in one of the six elements in one of the GEMMs while the round-to-nearest (RtN), Notice that applying SR to neural gradients during both 'Update' and 'Backward' GEMMs and activations during the 'Update' GEMM leads to lower training loss, while applying SR other components has the opposite effect, increasing the loss.
Notably, we have the flexibility to select the rounding mode independently for each of these six elements. Fig 3] we present results of training a 350M Llama model using NVFP4 where stochastic rounding (SR) is applied to each of these elements separately while the rest of the elements use round-to-nearest (RtN), allowing us to evaluate individual contribution. Applying SR to neural gradients during update and backward GEMMs and activations during the update GEMM helps reduce training loss, whereas using SR in other components leads to an increase in loss. Fig. 7 in the Appendix, present additional experiments that support the same conclusion. As result,
5

---

adopt the following selective rounding scheme:
[Forward] (RiN(W))RRi((az))
= f((z))
(4)
[Backward]
91-1 = QRIN QRs
51 = (()) o
(5)
ac
[Update]
= QSR (sR(())QsR((()
(6)
dw;
4 Analysis of Quantized SGD with Stochastic Rounding
This section analyzes when training with low- low-precision gradients stops being effective when using stochastic rounding (SR). While SR removes bias and enables stable descent during much of training, its benefits diminish once gradients become too small relative to quantization noise. We derive threshold on the gradient-to-noise ratio that signals when further progress stalls, motivating precision switch for backward and update passes. This switch improves convergence without altering the forward pass or the deployed model.
Key takeaways from the analysis:
SR enables unbiased updates, allowing stable descent even under aggressive FP4 quantiza- tion. In contrast, deterministic rounding introduces persistent bias, leading to irreducible error floor and preventing convergence (see Appendix [B.2).
There exists critical threshold: With SR, show in Section 4.1 (and in more detail in Appendix B.I) that the average per-coordinate gradient magnitude falls approximately below v times the quantization noise standard deviation, training no longer yields effective loss reduction. This threshold is derived under simplifying assumptions- eeg.. gradient descent with optimal step size, Taylor approximation of the loss, and a concentrated Hessian spectrum.
in Section 4.2 for both synthetic and real- model settings, we show empirically performance degrades sharply below this threshold. A precision switch guided by the theory restores convergence.
Empirical evidence supports the theory:
4.1 Theoretical Derivation
We begin with a second-order Taylor expansion of the loss function around the current parameter vector 92, which reveals a descent term proportional to - -VL" A and a curvature term involving the Hessian H. We then replace the full pell-preciioon gradient V L with its quantized version Jy VL where E is sero-mean noise introduced by stochastic rounding.
Using the update rule A -199 and taking expectations under SR (E(e) B[ee?) og T), obtain the expected loss change:
|[ 2 1
(VL tr(H))
Balancing the descent and noise terms yields the optimal step size:
|
ALTHVL of tr(
Substituting 17* back into the expected loss gives:
4 E|AL] 2 2 + tr()))
Differentiating with respect to og and using some simplying assumptions reveals that sensitivity to quantization noise peaks when:
|V
Ceritical
3dd
Once the per-coordinate gradient magnitude drops below v g. the descent becomes negligible and prechsr-recision gradients are needed to continue improving the loss.

---

4.2 Empirical Validation
To validate this theoretical threshold, we present two types of experiments.
First, we simulate training on a simple quadratic loss with adaptive noise schedule. We scale the quantization noise og = k certicaa for k 1, 0.5. As shown in Fig. convergence completely stalls at high noise levels (c.g., k = 2), slows near the critical threshold ( 1), and closely tracks full- full-precision training when noise is reduced below the threshold (k: 0.5).
10a
100
Loss
10
Baseline
10
2 Ceritical
Oeriiial
10
0.5 Oentical
10
20
30
40
Iterations
Figure 4: Validation of theoretical derivation simple quadratic loss. Training loss with noise levels 0g = k Cerit for k 2, 1, 0.5 in toy quadratic model. High noise blocks descent; low noise allows continued progress.
Second, we test in Fig. the threshold on real 60M-paramett Llama model. During training, we monitor the ratio / /d), and switch to
gradients at the 1000th iteration. When this ratio crosses v3, the loss gap to the full-precision baseline closes immediately, validating the predictive power of the threshold.
9-
difference
0.05
3-
1-
0.00
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Iterations
Iterations
Figure 5: Validation theoretical prediction in a Llama 60M model. (Left):: The difference between the loss curve the baseline and a model with increasing precision mid-training (1000th iteration, vertical dashed orange line). After increasing the precision, the loss difference is completely reduced. (Right): Gradient-to-noise, ratio with the v threshold (black dashed line).
Experiments
Setup. We used the Llama2 model [18 as our baseline. This model is decoder-only Transformer 22 with reenoomalization RMSNorm 23), Smooth-SwiGLU activation function (8), and rotary positional embeddings 17. We trained the models on open-source Red Pajama dataset (5) for tokens, maintaining hyperparameters consistent with [88. including train-test split and initialization. Specifically, we used Adam optimizer with 31 = 0.9, 82 0.95. We used cosine learning rate schedule, with 2000 steps of warmup, peak learning rate of x 10 and decay to 0.1 of the peak

---

learning rate. We used global batch-size of 4M tokens. All training was conducted on 256 Intel Gaudi2 devices, during 30 days.
FP4 training. In Fig. 6a we present our main experiment and present the training loss of Llama2 7B with the proposed FP4 scheme, which includes the use of the NVFP4 format (block size 16, scale format E4M3) and applying SR in the neural gradients (update + backward GEMMs) and activations (update GEMM), while applying RuN for the weights (forward + backward GEMMs) and activations (forward GEMM). In Table 2 we compare the quantization settings of our work with two previous FP4 training works 19 21], showing we are the first work that allows the acceleration of all matrix multiplication during training. In the Appendix Table we show the similarity of the FP4 training losses at different seeds, showing the noise robustness of the proposed method.
4.0
2.00
BF16
1.95
3.5
FP4
1.90
3.0
Loss
Loss
1.85
2.5
1.80
BF16
2.0
FP4
1.75
QAF
1.5 0
1.70
200
400
600
800
1000
0
200
400
600
800
1000
Processed Tokens (Billions)
Processed Tokens (Billions)
(a)
(b)
Figure 6: Full FP4 training 7B model achieved same loss as BF16 baseline. (a): Training loss of Llama2 7B using the proposed FP4 scheme which include NVFP4 format (block size 16, scale format E4M3) with SR in the neural gradients (update backward GEMMs) and activations (update GEMM). Notice a small gap in training loss is observed. (Table (b): Quanization aware finetuning (QAF) training loss of Llama2 7B using NVFP4 format in the forward GEMM and BF16 in the backward and update GEMMs. Notice this short QAF is able to completely close the gap with BF16 baseline.
Table 2: Comparison of the different FP4 training works. DGE refers Differentiable Gradient Estimator, OCC to Outlier Clamp Compensation, SR to Stochastic Rounding and RHT to Random Hadamard Transform. While previous works quantize only part of the GEMMs to FP4 with additional overhead of residual sparse matrix (OCC (21) or Hadamard transform (RHT 19),, our work is the first work that show quantization FP4 of all GEMMs operands, without adding significant overhead.
Weight
Activation
Neural gradients
Tokens
21
FP4+DGE+OCC
FP4+DGE+OCC
BF16
100B
19
BF16
BFI6
MXFP4+RHT+SR
21B
Ours
NVFP4 (RIN)
NVFP4 (RIN SR)
NVFP4 (SR)
IT
Quantization Aware Fine-tuning (QAF). FP4 training results (Fig. 6a) small gap in training loss compared to the BF16 baseline. As shown in Fig. increasing the precision raises the gradient- to-noise ratio higher above the critical threshold. To close the remaining gap, we introduce brief quantization-aww aware finetuning (QAF) phase, where pretraining continues on the same dataset with the forward pass kept in FP4, while the backward pass executed in BF16. Keeping the forward path in FP4 ensures that the model remains fully compatible with low- oow-precisoon inference, without requiring any additional processes such as rate and apply short warmup (40 iterations) followed by cosine decay schedule with an initial peak learning rate. In Fig. (bb we present that this short QAF can completely close the gap with BF16 baseline, achieving average bits of 4.3 bits for GEMMs across training QAF, In the Appendix
wuw.iiiinngguaniiatiinnnnn During this stage, we reset the learning

---

Table 5 we show an ablation study of the length ratio required to get similar final loss as BF16. We find it decreases for larger datasets.
Zero-shot Performance. Table compares the zero-shot performance (accuracy and perplexity) on downstream tasks between the BF16 baseline and our FP4 model both after the FP4 training phase (IT tokens) and after QAF phase (+40B tokens). Notice that while a small gap is observed in part of the tasks after the FP4 training phase, it is completely closed after the QAF.
Table 3: Zero shot accuracy and perplexity comparison between the BF16 baseline and the proposed FP4, both after the FP4 training phase (IT) and after QAF phase (+40B). HS refers HellaSwag, WG refers to Winogrande, BQ refers to BoolQ, AC refers to Arc-C, PQ refers to PiQA, LA refers to Lambada, GQ refers GPQA. IE refers to If Eval. MB refers MBPP, TQ refers TrivialQA. XS refers to XSum. Notice that after the QAF, the proposed FP4 model achieved on-par results with the BF16 baseline.
Precision
Data
Accuracy
Perplexity Wiki
LA
H8
WG
AC
BQ
PQ
GQ
TF
MB
TQ
X8
Avg
T.A
BF16
1T
61.52
68.71
66.54
38.14
69.33
76,33
24.54
33.81
8.2
34.99
11.97
45.63
5.54
6.1
BF16
1.04T
61.46
68.57
64.48
38.91
70.09
75.41
24.73
30.94
8.6
34.19
12.33
45.13
5.54
6
FP4
17
58.39
67.31
64.01
38.65
69.33
74.86
24.73
32.13
6.4
34.3
12.09
44.46
5.83
6.77
QAF
+40B
67.71
68.53
65.98
39.25
68.9
76.01
27.29
32.25
9.4
38.31
12.11
45.75
5.56
5,97
6
Discussion
This work presents the first demonstration of fully quantized FP4 -overinng weights, activa- tions, and gradients-att large scale. Our experiments on Llama2 7B model show that, while a small gap in training loss initially appears compared to BF16, this gap can be fully closed with a short QAF phase, where the forward pass remains in FP4 and only the backward pass switches to 6F16. Importantly, downstream task performance remains on par with BF16, confirming FP4's practical viability.
key contribution our investigation of FP4 format design. We find that NVFP4 (E4M3 with block size 16) offers the best trade-off between dynamic range and precision. Other blocks size or alternative exponent/mantissa configurations lead to instability or diminishing returns, aligning with NVIDIA Blackwell's hardware decisions.
We also introduce a split rounding strategy. using stochastic rounding only in the backward pass. which substantially improves training stability. Furthermore, our theoretical analysis identifies critical transition point: when the full-precision gradient standard deviation falls approximately below v3 times the quantization noise, training stagnates. This insight guides the design of the final fine-tuning phase to boost the ggnal-to-noise ratio and match BF16 convergence.
Limitations. A key limitation of this work is the lack of dedicated FP4 support in current Gaudi hardware, which prevents us from directly measuring the potential speedup and energy efficiency benefits of native FP4 execution. As result, all experiments Gaudi2, which incur additional overhead from precision casting and lead to longer runtimes. Based on previous FP8 works [13 g in comparison to FP8, which corresponds to ~ 85% time-to-train acceleration in comparison to the BF16 baseline. Our work centers on the LLaMA architecture, one of the most widely adopted frameworks in modern LLMs. Preliminary experiments indicate that the approach can be directly applied to Mixture-of-Experre (MoE) architectures. Further analysis of MoE models and extensions to vision tasks are reserved for future work.
conducted using FP4 simulations in
expect in rough estimation ~ 35 40% time-to-train acceleration
Acknowledgements
The research of DS was Funded by the European Union (ERC. A-B A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in AI.

---

References
[1] Nvidia blackwell architecture.
URI. hstps://resources..iiiii
en- un-us-an en-us-blacken www.-baaakllll-rchieeeeeeee
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal. Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ily Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, Curran Associates, Inc., 2020. URL https: http::::s///pooceeennneeee neurips. cc/ htsp/pppe__ff__ fil.ffiles///eeee httpeer_iiieesppper//////////// pppppr/p02//fiillee -
[3] Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv, and Yi. Liu. Optimize weight rounding via signed gradient descent for the quantization of Ilms. ArXiv, abs/2309.05516 2023. URL https: semanticscholar. .arearrrgg///gee org I 261697282
[4] Brian Chmiel, Ron Banner, Elad Hoffer, Hilla Ben Yaacov, and Daniel Soudry. Accurate neural training with 4-bit matrix multiplications standard formats. In International Conference on Learning Representations, 2021. URL https://api. semanticscholar seenartt.cccholaaar.rrr///ooooop
246634451
[5] Together Computer. Redpajama: an open dataset for training large language models, 2023.
URL httpp://githuu. com/toggtthhrrcomppteerre jeeeea amaajata
[6] DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bing-Li Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dong-Li Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang, Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Jun-Mei Song. Kai Dong. Kai Hu, Kaige Gao. Kang Guan, Kexin Huang. Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang, Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shao-Ping Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang. Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng. Wanjia Zhao, Wei An. Wen Liu, Wenfeng Liang, Wenjun Gao, Wen-Xuan Yu, Wentao Zhang, X. Q. Li, Xiangyu Jin, Xianzu Wang, Xiaoling Bi, Xiaodong Liu, Xiaohan Wang. Xi-Cheng Shen, Xiaokang Chen, Xiaokang Zhang. Xiaosha Chen, Xiaotao Nie. Xiaowen Sun, Xiaoxiang Wang. Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao Yaofeng Sun, Yao Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yi Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yi-Bing Ma. Yiyuan Liu, Yongqiang Guo, Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxiang Ma, Yuting Yan, Yu-Wei Luo, Yu mei You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Zehui Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhen guo Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu. Zijia Zhu. Zijun Liu, Zi-An Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng Pan. Deepseek-ve v3 technical report. ArXiv, 11122412.99947.. 2024. URL httths:t/app...... www.sseeannnncchaaaa.eeeem wwtg/oorppsDD:2511.......
[7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetun- ing of quantized llms. ArXiv, abs/2305.14311 2023. URL, httpss/tt bttttp:://ap..smmaaee semanticschollar
org/CorpusID::20 1D:88841128
10

---

[8] Maxim Fishman, Brian Chmiel, Ron Banner, and Daniel Soudry. Scaling fp8 training to trillion-token 11ms. ArXiv, abs/2409.12517 12517, 2024. URL URLpttts:: https://pppp http:/ppp..smmaiiccccchcoo..o... erg/Corpus : 2777530088
[9] Elias Frantar, Saleh Ashkboos, Torsten Hoefter, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. ArXiv, abs/2210.17323, 2022. URL
https://app..emmatiicss www.smeesacccccoo.....o org/ Trrg/CoppsID:: (orppusID:2532372000 253237200
[10] Amirata Ghorbani, James Wexler, and Been Kim. Automating interpretability: Discovering and testing visual concepts learned by neural networks. ArXiv, abs//992.03111111 2019. URL
https::/aap..sematticccct
stt.appii.mmmmaiiccchholleeeeeee wrtpaaarrroooooootoosae00 59842921
[11] Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. GetMo- bile Mob. Comput. Commun., 28:12-17, 2023. URL UtL.Attpss/ppp....mmiicccoo....p
www.:eeeeaaaaatiooe........... org/
CorpusID 258999941
[12] Andrei Panferov, Jiale Chen, Soroush Tabesh, Roberto L. Castro, Mahdi Nikdan, and Dan Alis- tarh. Quest: Stable training of Iims with 1-bit weights and activations. ArXiv, abs/2502.05003, 2025. URL U https: httt::/:/api.smmatticcchhhh//// http:/cchcchaaaoor/rrrrreee ht:://oppIII::22233000
[13] Houwen Peng, Kan Wu, Yixuan Wei, Guoshuai Zhao, Yuxiang Yang. Ze Liu, Yifan Xiong, Ziyue Yang. Bolin Ni, Jingcheng Hu, Ruihang Li, Miaosen Zhang, Chen Li, Jía Ning, Ruizhe Wang, Zheng Zhang. Shuguang Liu, Joe Chau, Han Hu, and Peng Cheng. Fp8-Im: Training fp8 large language models. ArXiv, abs/2310. 18313, 2023, CtL.https://// httpp:::/pp..seanntcccchho//// semanticscholar
org/Corpus ID: 264555252
[14] Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. In I. Guyon, U. Von Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpp:tts/// http::/dcaaaaeeeuueeeooe.ooepeee 0f5004eeec8444bbcccc0000008883.3 .pdf
[15] Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Sum- mer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Dusan Stosic, Venmugil Elango, Maximilian Golub, Alexander Heinecke, Phil James-Roxby, Dharmesh Jani, Gaurav Kolhe, Martin Langhammer, Ada Li, Levi Melnick, Maral Mesmakhosroshahh, An- dres Rodriguez. Michael Schulte, Rasoul Shafipour, Lei Shao, Michael Siu, Pradeep Dubey, Paulius Micikevicius, Maxim Naumov, Colin Verilli, Ralph Wittig, Doug Burger, and Eric Chung. Microscaling data formats for deep learning. ArXiv, abs/2310. 10537, 2023. URL https://api www..sseaaaanccclaaaaam... wrg.laar..rrgooooo Berg//CrpusID:::..
[16] Levent Sagun, Utku Evci, V. Ugur Güney, Yann Dauphin, and Léon Bottou. Empirical analysis of the hessian of parrparraetrrized neural networks. ArXiv, abs/1706.0445.. 2017. URL arg/CorpusIiy : 35432793
https://app:
semanticschol
[17] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomput. 568(C), mar 2024. ISSN 0925-2312. doi: 10. new.m..n.ummm.2..... neucom. 2023. 127063
htthhtss:: 1tt.:/o.rg//0111111111.
[18] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel. Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu. Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, V. Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez,
11

---

Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned
chat models. ArXiv, abs/2307.09288,, 2023. URL https: //ap1 sww.nticscholaa........ccccccc org/
CorpusID: 259950999
[19] Albert Tseng. Tao Yu, and Youngsuk Park. Training Ilms with mxfp4. ArXiv, abs/2502.20586, 2025. URL httpp: www.nticscccoaa...ccoommmmm org/CorpusIDD--.C..ooo.. Dttp://aacccooaaoooooooooorooaaa
:26722117
[20] Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. ArXiv, abs/2310.11453. 2023, URL https://api. semasmiccccocoo......
htt//oorpuID::24177700
[21] Ruizhe Wang, Yeyun Gong, Xiao Liu, Guoshuai Zhao, Ziyue Yang. Baining Guo, Zhengjun Zha, and Peng Cheng. Optimizing large language model training using fp4 quantization. ArXiv, abs/2501.17116, 2025. URL https://apll semanticscholar. wwwpocchooaar.ooooooppeeee Crg/Corp 275932873
[22] Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accu- rate and efficient post-training quantization for large language models. ArXiv, abs/2211.10433.
2022. URL hittps://apy http:/ppp..seaanncccccolalaaaaaa httppp/seeaaaaaeeaa.aaa...rr/aaa www./oorrgoo.......oo :537082277
[23] Biao Zhang and Rico Sennrich. Root mean square layer normalization. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett, edi- tors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019, /ttp//ttp::p/prrceeeiin.g.g..oe new.ggs..eeuippcc.mmmmm cc/ wwtpcccppppppaaeee hpapprrfillee cnmeer_ffllsspeeeee /ttpp//////pppr///////1ee/// fe91199222222a3333eeeeffffffr2-2
12

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claim of the paper is presenting the first full fp4 training scheme on 7B model. This is presented in the experiment section.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 6 we present the limitations of the paper.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciflcciiii asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions
online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification: The theoretical results presented in Appendix B includes all assumptions taken and is a complete proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and roos-reeferencd.
13

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly refereneed.
Experimental result reproducibility
Question: Does the paper fully disclosc all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, authors should describe steps taken make their results reproducible or verifiable.
Depending on contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access model. general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), relcasing of model checkpoint, other means that appropriate to the rescarch performed.
While NeurlPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily reproduce that algorithm.
algorithm, the paper should make clear how
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source source models, it may be that access to the model is limited in some way (e.g.. registered users). but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does paper provide open access to the data code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper use open source dataset and publish the full code to reproduce all experiments.
Guidelines:
answer means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (https://nips... htttpp::/nniss.cc/publlee cc/public/ htt////eeebiiiiiiooooooo
for more details.
While we encourage release of code and data, we understand that might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nnis///..///// cc/public/ oui.dess/CooeeSttmiessssoooooo.. for more details.
14

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section inceludes all details about the experiments in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We explain in the experiments section about the use of standard opensource train-lest split, initialization.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some paramcter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error hars that would yield results that are out of range (c.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time exccution) needed reproduce the experiments?
Answer: [Yes]
Justification: The paper explain in Section all details about the computer resources of the experiments. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
15

---

The paper should provide the amount of compute required for each of the individual experimental well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Does the research conducted the paper conform. every respect, with NeurlPS Code of Ethics http:////eeripp..pppppptt//bbbba
Question:
Answer:
[Yes]
Justification:
The paper conform the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
authors should make sure preserve anonymity (e-g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix A we discuss the broader impacts of the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will he foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example. it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example hy requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
16

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g.. code, data, models), used in the paper, properly eredited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
All data used in the paper is open source. The paper include full citation of all related works.
Justification:
Guidelines:
answer means that paper does not use existing assets.
The authors should cite the original paper that produced the code package or datasct.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperssi www.ehthooe.... com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should he provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are alongside assets?
introduced in the paper well documented and is the documentation provided
Answer: [Yes]
Justification; The paper include anonymous github with the code to reproduce the experiments. The code include full documentation required.
Guidelines:
The answer means paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Fthics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
17

---

Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage LLMs it an important. original. or non-standard component the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the rescarch, declaration is not required.
Answer: [NA]
Justification: LLM was used only for editing formatting purposes.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy /ttp://////////rr/////////////// should or should be described.
for what
18

---

Appendix
A
Broader impacts
Accelerating the runtime of large language models (LLMs) plays pivotal role in shaping modern digital experiences, especially systems like ChatGPT and Gemini become increasingly embedded in everyday applications. Improving their speed efficiency addresses major bottleneck large-scale Al deployment. In addition to boosting performance and reducing memory demands, faster models broaden accessibility, enabling more users to adapt and innovate with LLMs for their specific necds. However, this increased accessibility also raises concerns about potential misuse, underscoring the need for responsible development and oversight.
B Analysis of quantized SGD
B.1 Analysis of quantized SGD with stochastic rounding
We study how quantization noise affects the expected loss decrease during gradient descent.
Let L()) be twice-differentiable scalar loss function on Rt
Step 1: Start with the Taylor expansion of the loss. We consider a small step A from 8.. The sccond-order Taylor expansion of L at 0, is:
1 A0 H(0,) A0 4..
L(0: - A0) [(0,) VI())) Ae + 2
where ((:)) - L(0.).
Step 2: Apply quantized gradient update. We noisy estimate of the gradient due quantization:
9y LL.,))++ +
where e is quantization noise. The update rule becomes:
0.+1
19
AO 11 94.
Step 3: Substitute into the Taylor expansion.
Plugging AO -1 9, gives
1 n 94 (0:)) 94-
L(0(41) L(0.) n VL(O.) 94 + 2
Step Expectation over quantization noise.
Under stochastic rounding, the noise e satisfics
E[s] = 0, Taking expectations the Taylor expansion gives
E[ee"
T.
- VL.) B[g.] + 1 E[9 (O.)) gu]. E[L(O++1)] Re -
Since
= [V)(0)) + VL.).)
the linear term simplifies to
VL(00)' \V
Next,
Elg.9.11 EE -
= VLVET + +
T (0.) 0.) tr (H( = VI(0.) H(0)) (L(). (H(()))))))..
Putting everything together,
B[L(0++1)] = I(O.) - 1 / n (V.( H(0)) ((())))(((.)) (()))))(())+)))))))))))))))))))) + tr
19

---

Step 5: Convergence dynamics with SR
From Step 4. the expected change in loss is:
[[I(0,+1) I(O+)] x -7 (00)) =n²² (VL(O)) (10) + + +2 o.
This can he written as:
[[ - L(0:)] (n ((8.))
descent component
quantization
The useful descent component is negative if:
2 V n (V.)) > ( F100111) =(0)) =1 ( 00 VL.
more conservative condition is:
7 Amax (H.)
Step 6: Optimal step size r/" To find the optimal step size, we define
(nn) E|L(0(+1) L(0)) = 2nd (VL(0.)" +
Setting its derivative to 0: dU dn
- -V + - -
Solving for n
VI HLO (H())((())))
Step 7: Training with optimal step size r/" Substitute 17"= back into
Unn) = 17 A 1 B,
where
(L. VL(0.) + (.
Then
A B
A2
1 1
1
U(7')
B
2
B
i.c.
U(n')= 2( ( (O.) VL(O.) + 2f (r
X VL tr(H)) and Z = || Then
Z
U(n') = 2(X + o
Step 8: Maximum sensitivity to noise. We start with the sensitivity
(U(n")
ZYag og
f(a.) = 0dg
(X oY
To find its maximum, compute the derivative w.r.t. Og:
df zY (X (X+Yo3)² 0. 2 + +Yo?) -o,))-2Yo.)
do,
(X Yoz)²
We factor this expression and simplify it:
df ZY (X Yo, [X + +Yo,) 4Y
x - -3Yo
dog
(X + Yo?)
(X x Yogz)
20

---

Set this to zero to locate the minimum:
X --
0 3Y
Thus the critical noise level is
X
Fertiical
3Y
That is:
VI(0.) H(O)) VL(O.)
critical
(O.)
Finally, assuming
VI(0.) H(O,) VI(0.) tr(H(()))
[V.(0.)
get
VL(00) = 3do critical
[]L((l)]
/ Oeritionl.
Therefore,
LL(00)
Ceritical
v3d
In other words, once the average per-coordinate gradient falls to v times the quantization-nois std, FP4 gradients lose efficacy and it is time switch to higher precision. As shown in Figure setting the noise std to k Gerisical with & 2.0, 1.0, 0.5 yields markedly different convergence behaviors around that threshold. In Appendix [322 we show similar analysis without SR, which shows the "useful descent" vanishes zero as we train, while the biased-noise term remains even after long training..
B.2 Impact of nonzero mean noise (deterministic rounding)
this section, we exemplify the problem with biased quantization schemes (such as RtN), in simple scalar optimization problem with a quadratic loss
(()) x(0 00 (()) A(0 0²),
a step size update with quantization noise whose mean μr [8[]] * O:
o = 0: ((.)) ++) = E[O+41] = [[O:] (A(B(O))) 0²) + pa).
Define the error
e a E[0:] 0"
Then
C++1 = [0(++]] 0¹ [E[0.]] (>ee + [z)] - 0¹ 77e= 1//2
Therefore,
e111 = (1 - TTA) ce - Me.
Unrolling this recursion gives, for a = 1 n1,
R-1
= a'eo
17
0=0
Since Σe=1 a² = I-o" T-a and 1 = n7, we get
en = a" co
B tee
(1 a²) -aa==== a"co
Me
X (1-a)).
'In the high-dimensional regime (d, N - co and d/N - X, where N is number of training samples), previous works [14] showed, using random-matrix theory and empirical results, that the bulk of the Hessian values be approximately represented by the Marchenko-Pastur distribution, especially for small loss values. (which is the common regime LLMs), Marchenko Pastur distribution implies that bulk of Hessian eigenvalues concentrates near their mean tr(H) /d. Empirical investigations [16] [0) confirm that the gradient predominantly occupies this bulk subspace rather than aligning with the few extreme modes. Hence the curvature experienced in the gradient direction approximates the average eigenvalue.
Thus, when
21

---

The loss at step v is
Ln = L( L(B(O,)) = x
As n. a² (for 0, which is required for successful optimization), yiclding the stationary error and residual loss
Me X
Loe
2X
Thus, instead of converging with zero loss, biased SGD settles at
E|0xo] = 0"
and leaves a residual loss
[(B(Oso))
HE
2X
C
Additional Experimental Results
3.20
3.15
3.10
Loss
Full
Weights Forward Activitions Forward RtN Weights
3.00
2.95
Aclivations Updete RIN
Update RIN Grad Bockward RIN
2.90
9
11
12
13
14
15
Proceed tokens (Billions)
Figure 7: Comparison of different rounding schemes when training 350M Llama model using NVFP4 format. In each graph. we apply RtN in one of the six elements in one of the GEMMs while the rest use SR. Notice that applying RtN to neural gradients during both the 'Update" and 'Backward' GEMMs, and to the activations during the 'Update' GEMM leads higher training loss, while applying RtN to the other components has the opposite effect, reducing the loss.
Table 4: Training loss of Llama 125M over 30B tokens with different seeds, yield a similar loss, resulting in standard deviation of 0.001.
Training loss
Seed
3.03
1337
3,027
1234
3.025
2345
3,027
3456
3.029
4567
Table 5: Ablation study determine how many tokens are required to reach the same loss as the BF16 baseline. Notice the QAF ratio decrease for larger dataset. In all QAF experiments we use the peak learning rate equal the last learning rate in the FP4 training.
Starting point
QAF length
Ratio
2008
20B
10%
500B
28B
5.6%
1T
40B
4%
22


Reviews:
[{"review_id": "15Dv9clNcz", "reviewer": "Reviewer_hLJD", "summary": "This paper presents the first successful demonstration of FP4 training for weights, activations, and gradients. Leveraging the NVFP4 format and a carefully selected rounding strategy (stochastic rounding for backward and round-to-nearest for forward), the authors train a 7B LLaMA2 on 200B tokens using 256 Intel Gaudi2 devices. The trained FP4 model achieves performance on par with BF16 after a brief quantization-aware finetuning (QAF) phase.", "questions": "Could the authors clarify the scope and applicability of the proposed FP4 training method? Specifically, are there particular model types, sizes, or tasks for which this approach is especially suitable or advantageous? Or do the authors believe it can generalize effectively to any LLM training scenario?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper presents the first demonstration of FP4 across all three GEMMs (forward, backward, update) at scale (7B model, 200B tokens).\n- The use of FP4 for both weights and activations has the potential to largely accelerate training.\n- The theoretical analysis of the gradient-to-noise threshold—supported by empirical evidence—offers valuable insight into when higher-precision training should be resumed.\n- It is great to see the open-sourced code.\n\nWeaknesses\n\nWhile large-scale experiments are understandably expensive, it would strengthen the paper to include broader evaluations. \n\n- It would be great to evaluate your methods across different model sizes or architectures.\n- A key limitation is that the proposed FP4 training is only simulated on Gaudi2 hardware, which lacks native FP4 support. As such, the actual speed and memory benefits remain speculative. It would be great if you could provide a solid prediction for the cost reduction. \n- The paper lacks qualitative or in-depth comparison with other recent FP4 training approaches. The baseline is only BF16. \n- Details of the training configuration—such as optimizer type, learning rate schedule, batch size, and initialization—are only briefly mentioned and could be more thoroughly documented for full reproducibility.", "was_revised": true, "final_justification": "This is a solid paper that I recommend for acceptance.", "rebuttal": "$\\textbf{Q1}:$ \"It would be great to evaluate your methods across different model sizes or architectures\"\n\n$\\textbf{A1}:$ In the camera-ready version, we will include additional experiments involving smaller models with 125M and 350M parameters, achieved final on-par training loss with the BF16 baseline (125M: 2.922(BF16) Vs 2.918 (FP4). 350M: 2.67 (BF16) Vs 2.673 (FP4)). Furthermore, we have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline  with FP4 model on downstream tasks (similar to Table 3), including new tasks - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS): \n\n| Precision | Tokens | LA    | HS    | WG    | AC    | BQ    | PQ    | GQ    | IE    | MB   | TQ   | XS   |  *   | Avg   |  *   | Wiki(PPL) | LA(PPL) |\n|-----------|--------|-------|-------|-------|-------|-------|-------|-------|-------|------|------|------|-----|--------|-----|------------|----------|\n| BF16      | 1T     | 61.52 | 68.71 | 66.54 | 38.14 | 69.33 | 76.33 | 24.54 | 33.81 | 8.2  | 34.99| 11.97| *   | 45.63 |  *   | 5.54       | 6.10     |\n| BF16      | 1.04T  | 61.46 | 68.57 | 64.48 | 38.91 | 70.09 | 75.41 | 24.73 | 30.94 | 8.6  | 34.91| 12.33|  *   | 45.13 |   *  | 5.54       | 6.00     |\n| FP4       | 1T     | 58.39 | 67.31 | 64.01 | 38.65 | 69.33 | 74.86 | 24.73 | 32.13 | 6.4  | 34.30| 12.09|   *  | 44.46 |  *   | 5.83       | 6.77     |\n| +QAF      | +40B   | 61.71 | 68.53 | 65.98 | 39.25 | 68.90 | 76.01 | 27.29 | 32.25 | 9.4  | 38.31|  12.11   |  *   | 45.75 |  *   | 5.56       | 5.97     |\n\nThe Scaling to larger model sizes is not feasible with our current resources. Our experiments are based on the LLaMA 2 architecture, which is widely adopted in most LLMs. \n\n$\\textbf{Q2}:$ \"A key limitation is that the proposed FP4 training is only simulated on Gaudi2 hardware, which lacks native FP4 support. As such, the actual speed and memory benefits remain speculative. It would be great if you could provide a solid prediction for the cost reduction.\"\n\n$\\textbf{A2}:$ We agree with the reviewer that this is one of the limitations of our paper (as we wrote in our limitation section). It is worth noting that even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators. As stated in the limitations section, based on prior time-to-train improvements from BF16 to FP8 (35-40 \\% time-to-train acceleration) and the additional 2× matrix multiplication speedup from FP8 to FP4, we provide a rough estimate of an 85\\% reduction in training time.\n\n$\\textbf{Q3}:$ \"The paper lacks qualitative or in-depth comparison with other recent FP4 training approaches. The baseline is only BF16.\"\n\n$\\textbf{A3}:$ Our paper is the first and only work to demonstrate full FP4 training, where all three general matrix multiplications (GEMMs) described in Equations 1, 2, and 3 are quantized. Therefore, there is no prior work that allows for a fair comparison. The most related studies, summarized in Table 2, apply only partial quantization (offering less potential acceleration) and are conducted on smaller datasets.\n\n$\\textbf{Q4}:$ \"Details of the training configuration—such as optimizer type, learning rate schedule, batch size, and initialization—are only briefly mentioned and could be more thoroughly documented for full reproducibility.\"\n\n$\\textbf{A4}:$ We maintain all hyperparameters consistent with Llama2 paper, which includes the use AdamW optimizer with $\\beta_1=0.9$, $\\beta_2=0.95$. We use cosine learning rate schedule, with 2000 steps of warmup, peak learning rate of $3\\times10^{-4}$ and decay to 0.1 of the peak lr. We use global batch-size of 4M tokens. All these details are available in the supplied code and will be added in the camera-ready version.\n\n\n$\\textbf{Q5}:$\n\"Could the authors clarify the scope and applicability of the proposed FP4 training method? Specifically, are there particular model types, sizes, or tasks for which this approach is especially suitable or advantageous? Or do the authors believe it can generalize effectively to any LLM training scenario?\"\n\n$\\textbf{A5}:$ The primary applicability of the proposed FP4 training is to accelerate the pretraining phase, which is the most computationally expensive step and becomes increasingly demanding with larger datasets and models. Our work focuses on the LLaMA architecture, which is one of the most widely used architectures in modern LLMs, and we believe our approach can be effectively extended to most LLM models. While vision transformers were not the focus of this work, their similar architectural structure suggests that our method could also provide acceleration in that domain."}, {"review_id": "zjo2xFJx3x", "reviewer": "Reviewer_34ZT", "summary": "This paper explores FP4 training of LLMs using NVFP4 format. The authors use stochastic rounding for the backward path, which is instrumental in getting unbiased gradients. Furthermore, the authors provide theoretical and empirical analysis of the quantized SGD convergence. The proposed method also explores switching to higher precision training when quantization noise dominates gradient updates. The paper is well structured, well-written, and provides insightful information.", "questions": "As mentioned in the previous section:\n\n1. More evaluation results such as GPQA, IFEveal ( Open LLM Leaderboard 2 ), EvalPlus will add more insight on how well the quantized model is trained.\n\n2. Did authors use any sort of Maximal Update Parametrization (muP)? What was the effect of quantization on that? [1]\n\n3. Is it possible to derive some mathematical guarantee that quantized SGD converges given that higher precision SGD converges? Example: refer to Section 4 of [2]\n\nReferences:\n\n[1] Yang, Greg, et al. \"Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer.\" arXiv preprint arXiv:2203.03466 (2022).\n\n[2] Ghaffari, A, et al. \"Is integer arithmetic enough for deep learning training?.\" Advances in Neural Information Processing Systems 35 (2022): 27402-27413.", "limitations": "Authors adequately addressed the limitation of this work.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper is proposing a solid methodology and mathematical derivation. I personally enjoyed seeing mathematical derivation accompanied with empirical evidence,\n\n2. Fig 1 and Fig 2  provide insightful intuition regarding the number formats. Also Fig 4 and Fig 5 are informative and provide empirical evidence for mathematical derivation of critical quantization noise level.\n\n3. Fig 6 shows quantized training loss is similar to the baseline training, showing the effectiveness of this algorithm for Llama 7B model.\n\n4. Experimental evidence with perplexity and zero shot tasks shows strong evidence that training procedure was effective.\n\n**Weaknesses**\n\nThis paper does not have too many weaknesses; here are my suggestions to further improve the quality of the presentation:\n\n1. As mentioned by the authors, the lack of validation in FP4 hardware is a key limitation of this paper.\n\n2.  It would also be insightful to see the results of FP4 training for more models and more evaluation tasks such as EvalPlus and GPQA.\n\n3. Section 4.1 can include quantized SGD convergence analysis to make sure and understand the conditions of quantized SGD convergence.", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1}:$ \"As mentioned by the authors, the lack of validation in FP4 hardware is a key limitation of this paper.\"\n\n$\\textbf{A1}:$ We agree with the reviewer that this is one of the limitations of our paper. It is worth noting that, even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators. \n\n$\\textbf{Q2}:$ \"More evaluation results such as GPQA, IFEveal ( Open LLM Leaderboard 2 ), EvalPlus will add more insight on how well the quantized model is trained.\"\n\n$\\textbf{A2}:$ We have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline  with FP4 model on downstream tasks (similar to Table 3), including new tasks as required by the reviewer - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS): \n\n| Precision | Tokens | LA    | HS    | WG    | AC    | BQ    | PQ    | GQ    | IE    | MB   | TQ   | XS   |  *   | Avg   |  *   | Wiki(PPL) | LA(PPL) |\n|-----------|--------|-------|-------|-------|-------|-------|-------|-------|-------|------|------|------|-----|--------|-----|------------|----------|\n| BF16      | 1T     | 61.52 | 68.71 | 66.54 | 38.14 | 69.33 | 76.33 | 24.54 | 33.81 | 8.2  | 34.99| 11.97| *   | 45.63 |  *   | 5.54       | 6.10     |\n| BF16      | 1.04T  | 61.46 | 68.57 | 64.48 | 38.91 | 70.09 | 75.41 | 24.73 | 30.94 | 8.6  | 34.91| 12.33|  *   | 45.13 |   *  | 5.54       | 6.00     |\n| FP4       | 1T     | 58.39 | 67.31 | 64.01 | 38.65 | 69.33 | 74.86 | 24.73 | 32.13 | 6.4  | 34.30| 12.09|   *  | 44.46 |  *   | 5.83       | 6.77     |\n| +QAF      | +40B   | 61.71 | 68.53 | 65.98 | 39.25 | 68.90 | 76.01 | 27.29 | 32.25 | 9.4  | 38.31|  12.11   |  *   | 45.75 |  *   | 5.56       | 5.97     |\n\n\n$\\textbf{Q3}:$ \"Section 4.1 can include quantized SGD convergence analysis to make sure and understand the conditions of quantized SGD convergence.\nIs it possible to derive some mathematical guarantee that quantized SGD converges given that higher precision SGD converges? Example: refer to Section 4 of [2]\"\n\n\n$\\textbf{A3}:$ Thank you for this interesting reference. The global convergence guarantee in [2] and our local analysis in Section 4.1 work together. The theory in [2] predicts that training will eventually stall in a noise-dependent neighborhood, while our diagnostic tool helps identify in real-time when that stalling has begun. \n\n\n$\\textbf{Q4}:$ \"Did authors use any sort of Maximal Update Parametrization (muP)? What was the effect of quantization on that? [1]\"\n\n$\\textbf{A4}:$ We thank the reviewer for this valuable suggestion. In our work, the primary goal was to demonstrate that full FP4 training can achieve convergence with results on par with the BF16 baseline, using the same hyperparameters. This means we did not explore techniques such as muP, but we consider this an interesting direction for future work. Notice, as mentioned in A2, the main experiment (1T tokens) requires approximately 30 days on 256 devices."}, {"review_id": "3CxSgchFLS", "reviewer": "Reviewer_TyRD", "summary": "This paper demonstrates the first fully quantized training (FQT) of large language models using only FP4 precision for all matrix operations, including weights, activations, and gradients on NVFP4 format, with mixed rounding modes, round-to-nearest in the forward pass and stochastic rounding in the backward pass, to ensure stable training. They also propose a straightforward gradient noise threshold heuristic, based on a simple quadratic loss, to determine when to switch from FP4 to higher precision during training. This is implemented as a brief mixed-precision \"Quantization-Aware Fine-tuning\" phase using BF16. The study features systematic ablation experiments on quantization format, block size, and rounding strategies, achieving competitive zero-shot performance on multiple downstream tasks. Although currently emulated on Gaudi2 hardware, the method is designed as a blueprint for efficient FP4 training on future hardware such as Blackwell.", "questions": "I would like to kindly ask the authors to address the issues stated in the “Weaknesses” section. I would gladly increase my scores if the issues are properly addressed.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. First end-to-end FP4 fully-quantized training at non-toy scale, bridging the gap between FP8 and the state-of-the-art hardware.\n2. Clear exploration and ablations on scale formats, block sizes, and per-operand rounding choices.\n3. A full 7B parameter pre-training run on an open-source corpus with +200B tokens, demonstrating realism beyond toy tasks\n4. The noise threshold provides an actionable heuristic for when to up-switch precision, matching empirical behavior.\n\n\n\n\nWeaknesses:\n1. Lacking an isolated round-to-nearest impact per matrix, similar to Fig. 3. Without it, we cannot decide where SR would beat RtN.\n2. While the theoretical derivation in 4.1 is intuitive and somehow hints why we might have to switch to a higher precision, it lacks proper argument on why this critical measure is the direct cause of instability, i.e. it can be concluded that the worst expected error happens at this critical variance, yet doesn’t imply that the absolute value of $E[\\Delta L]$ is critically high when variance reaches this critical value. The derivation does not bound the absolute loss change $E[\\Delta L]$, nor prove divergence or “instability” at that point.\n3. The 220 B-token run yields a token-to-parameter ratio of ~30×, which scaling-law literature[3,4] deems under-trained and thus less likely to expose late-stage instabilities; training a smaller model on the same token budget would be a stronger stress test.\n4. No measured total BOPS, throughput, or average bit-width across FQT + QAF; a quick estimate suggests roughly ~5.2 bits overall.\n5. Experiments rely on Gaudi2 simulation rather than widely deployed Blackwell hardware, so real-world performance gains remain speculative.\n6. Stochastic-rounding implementation is unspecified; the probability mass function is unclear.\n7. Related-work survey omits two sub-FP8 methods, QuEST[1] and AdaBin[2], which should appear in Table 2.\n8. Training hyperparameters are incomplete; learning rate schedules, batch sizes, and optimizer settings, among others, are missing.\n9. All results are based on a single seed. Table 3 lacks error bars or confidence intervals.\n10. No empirical study on how long QAF must run, nor on LR and optimiser sensitivity in FP4.\n11. No exploration of combining the method with PTQ for additional compression. i.e., applying post-training quantisation methods such as SmoothQuant or GPTQ on top of the FP4-trained model, and inspect the combined effect on accuracy.\n\n\n\n\nMinor text issue:\n1. Possible typo on Fig.1 and Fig.2 x-axis label “proceed” -> “processed”\n2. Line 238: “close completely” -> “completely close”\n3. Typo: “trainng” in Fig. 6 caption.\n\n\n\n\nReferences:\n\n[1] QuEST: Stable Training of LLMs with 1-Bit Weights and Activations, Panferov et al.\n\n[2] AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets, Tu et al.\n\n[3] Scaling Laws for Neural Language Models, Kaplan et al.\n\n[4] Training Compute-Optimal Large Language Models, Hoffmann et al.", "was_revised": true, "final_justification": "The author's rebuttal has addressed all of my concerns, and in addition, they have provided stronger evaluations which better justify their method, and I think this paper is a high-impact paper in the field.", "rebuttal": "$\\textbf{General comment:}$ In response to the reviewer's questions, we would like to clarify a potential misunderstanding regarding the different training phases:\n(1) Long Pretraining: All three general matrix multiplications (GEMMs) described in Equations 1, 2, and 3 are quantized to FP4.\n(2) Short QAF: The forward GEMM (Eq.1) is quantized to FP4, while the backward (Eq.2 ) and update (Eq.3) steps are performed in BF16.\n\nThis means the final model contain FP4 weights and can run inference directly in the FP4 datatype without requiring any additional processes (such as PTQ). We plan to improve and clarify this explanation in the camera-ready version.\n\n$\\textbf{Q1}:$ \"Lacking an isolated round-to-nearest impact per matrix, similar to Fig. 3...\"\n\n$\\textbf{A1}:$ In response to the reviewer’s request, we conducted an additional experiment similar to Fig. 3, where we independently applied RtN to one of the six elements while the remaining elements used SR. The resulting training losses are as follows: Full SR: 3.022; Weights Fwd RtN: 3.002 ; Activations Fwd RtN: 3 ; Weights Bwd RtN: 3.021 ; Activations Bwd RtN: 3.032 ; Gradient Update RtN: 3.041 ; Gradient Backward RtN: 3.039.\n\nThis experiment highlights the effectiveness of the proposed regime, as described in Equations 4, 5, and 6, where SR is applied only to neural gradients during the update and backward GEMMs, and to activations during the update GEMM. We will include this experiment in the camera-ready version\n\n$\\textbf{Q2}:$ \"While the theoretical derivation in 4.1 is intuitive ...\"\n\n$\\textbf{A2}:$ We appreciate the reviewer’s comment and agree with their observation. Our analysis identifies a threshold for training ineffectiveness (i.e., when progress stalls) rather than proving instability. This occurs when the gradient signal becomes too weak to overcome the quantization noise, causing the optimization process to plateau.\nBecause our goal is to define this practical point of failure, the derivation in Section 4.1 is intended as a local metric to guide training decisions, not as a global proof of convergence. Consequently, we do not claim to bound the absolute change in loss or present a full convergence guarantee. As shown in Figure 5, this interpretation is supported empirically: when the metric falls below the proposed threshold, continuing training in low precision is no longer effective, and an intervention such as increasing the numerical precision is required to resume progress.\n\n\n$\\textbf{Q3}:$ \"The 220 B-token...\"\n\n$\\textbf{A3}:$ We have extended the primary training of the 7B model (Figure 6) to 1 trillion tokens, followed by 40B tokens of QAF, which requires approximately 30 days on 256 devices. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16). Then we compare the BF16 baseline  with FP4 model on downstream tasks (similar to Table 3), including new tasks - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS): \n\n| Precision | Tokens | LA    | HS    | WG    | AC    | BQ    | PQ    | GQ    | IE    | MB   | TQ   | XS   |  *   | Avg   |  *   | Wiki(PPL) | LA(PPL) |\n|-----------|--------|-------|-------|-------|-------|-------|-------|-------|-------|------|------|------|-----|--------|-----|------------|----------|\n| BF16      | 1T     | 61.52 | 68.71 | 66.54 | 38.14 | 69.33 | 76.33 | 24.54 | 33.81 | 8.2  | 34.99| 11.97| *   | 45.63 |  *   | 5.54       | 6.10     |\n| BF16      | 1.04T  | 61.46 | 68.57 | 64.48 | 38.91 | 70.09 | 75.41 | 24.73 | 30.94 | 8.6  | 34.91| 12.33|  *   | 45.13 |   *  | 5.54       | 6.00     |\n| FP4       | 1T     | 58.39 | 67.31 | 64.01 | 38.65 | 69.33 | 74.86 | 24.73 | 32.13 | 6.4  | 34.30| 12.09|   *  | 44.46 |  *   | 5.83       | 6.77     |\n| +QAF      | +40B   | 61.71 | 68.53 | 65.98 | 39.25 | 68.90 | 76.01 | 27.29 | 32.25 | 9.4  | 38.31|  12.11   |  *   | 45.75 |  *   | 5.56       | 5.97     |\n\nAs the reviewer can see, the proposed method also generalizes for larger datasets. We will upload these results in the camera-ready version. \n\n\n$\\textbf{Q4}:$ \"No measured total BOPS, throughput, or average bit-width across FQT + QAF...\"\n\n$\\textbf{A4}:$ As stated in A3, we extend the pretraining phase to 1T tokens, followed by a 40B-token QAF phase. As noted in the general comments, during pretraining all three GEMMs operate in FP4, while during QAF only one of the three GEMMs uses FP4 (with the remaining two in BF16). The resulting average bit calculation is:$ [(1000\\times3\\times4) + (40\\times1\\times4) + (40\\times2\\times16)] / [1040\\times3] = 4.3$ bits. We will include this result in the camera-ready version. Note the 5.2 bits result mentioned by the reviewer assumes fully 16bit QAF, which is wrong (see general comment above).\n\n$\\textbf{Q5}:$ \"Experiments rely on Gaudi2 simulation...\"\n\n$\\textbf{A5}:$ We agree with the reviewer that this is one of the limitations of our paper, as wrote in the limitation section. It is worth noting that even for Nvidia’s Blackwell, FP4 training is supported at the hardware level, but their software stack currently supports only inference. It is common for algorithmic research to precede full software support, and we expect that our work will encourage broader adoption of FP4 training across hardware accelerators. As stated in the limitations section, based on prior time-to-train improvements from BF16 to FP8 (35-40 \\% time-to-train acceleration) and the additional 2× matrix multiplication speedup from FP8 to FP4, we provide a rough estimate of an 85\\% reduction in training time.\n\n$\\textbf{Q6}:$ \"Stochastic-rounding implementation is unspecified...\"\n\n$\\textbf{A6}:$ The paper includes in the abstract anonymous full code reference to reproduce the experiments, specifically notice \"experimental/fp4.py\" line 6 which includes the stochastic rounding implementation. In general, the implementation includes taking the full precision value, adding random uniform noise with support equal to the bin size, then rounding to nearest quantization value:\n$Q(x) = \\mathrm{round} (x+ U)$, where $x$ is full precision value, $U\\sim [-0.5\\delta,0.5\\delta]$ is random uniform noise and $\\delta$ is the local spacing between two representable FP4 values around x. \n\n$\\textbf{Q7}:$ \"Related-work survey omits two sub-FP8 methods...\"\n\n$\\textbf{A7}:$ Thank you for these references, we will add them in the camera ready version. Notice that, neither QuEST nor AdaBin represents fully quantized training. In these works, quantization is applied only to the forward general matrix multiplication (Equation 1), while the backward (Equation 2) and update (Equation 3) steps are kept in high precision. This significantly reduces the potential acceleration compared to our approach. As explicitly stated in QuEST [1], page 4: \"although the backward computation is performed w.r.t. the quantized weights and activations, the multiplications and gradient operands are performed in standard 16-bit precision.\". \n\n$\\textbf{Q8}:$ \"Training hyperparameters are incomplete...\"\n\n$\\textbf{A8}:$ We maintain all hyperparameters consistent with Llama2 paper, which includes the use AdamW optimizer with $\\beta_1=0.9$, $\\beta_2=0.95$. We use cosine learning rate schedule, with 2000 steps of warmup, peak learning rate of $3\\times10^{-4}$ and decay to 0.1 of the peak lr. We use global batch-size of 4M tokens. All these details are available in the supplied code and will be added in the camera-ready version.\n\n$\\textbf{Q9}:$ \"All results are based on a single seed...\"\n\n$\\textbf{A9}:$ Table 3 includes running 0-shot downstream tasks on the final FP4 model and the BF16 baseline, which is a deterministic process (No stochastic process during inference). A possible way to get confidence bars is run all training again with different seed, but it is not feasible (full 1T tokens training take 30 days). In order to get an approximate confidence bar, we run the inference process presented in table 3 with 3 close checkpoints and calculate the std of these runs: \n| Precision | LA    | HS    | WG    | AC    | BQ    | PQ    | Wiki | LA |\n|-----------|-------|-------|-------|-------|-------|-------|---------|------|\n| BF16      | 1.27 | 0.13 | 0.89 | 0.82 | 0.66 | 0.45 | 0.016  | 0.2  |\n| FP4       | 0.33 | 0.15 | 1.36 | 0.39 | 0.76 | 0.25 | 0.02  | 0.04  |\n\n$\\textbf{Q10}:$ \"No empirical study on how long QAF must run...\"\n\n$\\textbf{A10}:$ Great question! We conducted an ablation study to determine how long QAF must run, i.e., how many tokens are required to reach the same loss as the BF16 baseline. Starting from three different checkpoints, we obtained the following results:\n* From 200B tokens – QAF requires 20B tokens (10%).\n* From 500B tokens – QAF requires 28B tokens (5.6%).\n* From 1T tokens – QAF requires 40B tokens (4%).\n\nAs the reviewer can observe, for larger and real-world datasets, the QAF ratio decreases. Additionally, we experimented with different peak learning rates for QAF and concluded that it should be similar to the training learning rate—using a larger LR causes divergence, while a smaller LR prevents QAF from converging to the baseline. We plan to include this ablation study in the camera-ready version.\n\n$\\textbf{Q11}:$ \"No exploration of combining the method with PTQ for additional compression...\"\n\n$\\textbf{A11}:$ SmoothQuant and GPTQ are post-training quantization (PTQ) methods designed to take models trained in high precision and quantize them to lower precision (typically INT4) after training, while attempting to minimize quantization error for inference. As noted in the \"general comment\" above, our proposed pretraining + QAF approach produces a model directly in 4-bit precision (FP4), with no additional steps required to enable 4-bit inference. Therefore, we believe PTQ methods are not necessary in our proposed framework. We decided to focus on FP4 and not in INT4 since it is supported natively in modern hardware, such as Blackwell.\n\n$\\textbf{Q12}:$ \"Minor text issue: ..\"\n\n\\textbf{A12}: Thank you, we will fix them in the camera-ready version"}, {"review_id": "GcpWSbITbe", "reviewer": "Reviewer_vv7s", "summary": "The authors in this work demonstrate end-to-end 4-bit FP training at LLM scale (7B, 200B tokens), extending FQT capabilities beyond existing FP8 approaches. The authors claim their work to be the first line of works demonstrate complete FP4 end to end training. The authors evaluate key design choices (block size, scaling format, rounding modes) and identify NVFP4 (E2M1 data with shared E4M3 scale in blocks of 16) with mixed rounding (stochastic for backward/update, round-to-nearest for forward) as optimal. Lastly, they successfully train a 7B-parameter model on Intel Gaudi2 accelerators to performance comparable to BF16.", "questions": "1. Please refer to weakness points (1), (2), (3). Including ablations to address these concerns would strengthen their claims.\n2. How does the FP4-trained model compare to FP16 baseline on downstream tasks (e.g., QA, summarization, code generation)? Does full quantization affect generalization across domains or modalities? The authors are encouraged to include a few difficult baselines to demonstrate the at-par performance wrt BF16 baseline.\n3. How sensitive is the training stability to the √3 gradient-to-noise threshold? Also, can the authors elaborate on how and when precision switching is triggered in practice?\n4. How does the method behave under varying batch sizes? Smaller batches often lead to noisier gradients, does this make FP4 more prone to training collapse? Also, how sensitive is the FP4 training pipeline to initialization schemes?\n5. Given that the authors use of stochastic rounding for backward pass, how reproducible are the training runs across seeds or hardware environments? Are there significant variance in convergence / final performance for different random seeds?", "limitations": "Yes, the authors have discussed limitations of their method.\n\nAs a suggestion, the authors are encouraged to further consider robustness related limitations, especially the potential vulnerability of their approach to noisy gradients or its applicability in vision and multimodal settings, where different modalities may exhibit different gradient distributions or magnitudes that could affect the stability of FP4 training.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "This is a timely work which focuses on lower precision training of LLMs. If generalizable across models, it could have moderate to high impact on an industrial scale.\n\nStrengths:\n1. This authors claim their work to be the first to enable end-to-end 4-bit FP training at LLM scale (7B, 200B tokens), extending FQT capabilities beyond existing FP8 approaches.\n2. The authors systematically explore block sizes (8 to 128), scaling formats (MXFP4 vs NVFP4), and rounding schemes, concluding NVFP4 with block size 16 delivers optimal stability. Also, they identify that rounding strategy using stochastic rounding for gradients and round-to-nearest for forward passes, significantly improves stability, especially for backward and update paths. These insights can be very beneficial for the community to further push ultra low precision training of large scale models.\n3. The paper introduces a theoretically derived and empirically validated threshold for gradient norms droping below √3 × quantization noise, suggesting dynamic precision adaptation.\n4. Demonstrated training of a 7B LLM on Gaudi2 chips with only FP4 showing practical viability. The trained model also achieved downstream results on par with BF16 baselines.\n\nWeakness:\n1. Although the 7B training shown performs at par with BF16 baseline,  it's unclear if FP4 FQT scales to larger models (30B+) or more diverse architectures beyond the 200B token setting. Additionally, as recent work [1] has also proposed an end-to-end FP4 training framework, the authors are encouraged to clarify the distinctions and provide insights into how their approach compares or complements this effort.\n2. Due to the use of shared scaling across blocks, stochastic rounding, and dynamic precision switching, this could likely incurs non-trivial computational cost. Providing a quantitative analysis of these costs would offer clearer insight into the trade-offs between efficiency and precision stability.\n3. The authors are also encouraged to include comparisons to all-nearest or all-stochastic rounding configurations. This would be beneficial in understanding the relative comparison of different schemes for stable training.\n\n[1] Wang, Ruizhe, et al. \"Optimizing Large Language Model Training Using FP4 Quantization.\" arXiv preprint arXiv:2501.17116 (2025).", "was_revised": true, "final_justification": "The authors answered most of my concerns with respect to understanding the sensitivity of the algorithm to different batch sizes, seeds and architectures. The proposed method seems promising however understanding and discussing its usecase and limitations is important for generalization.", "rebuttal": "$\\textbf{Q1a}:$ Although the 7B training shown performs at par with BF16 baseline, it's unclear if FP4 FQT scales to larger models (30B+) or more diverse architectures beyond the 200B token setting. \n\n$\\textbf{A1a}:$ We continued the training presented in Fig. 6 to 1T tokens, followed by 40B tokens of QAF. We achieved training loss of 1.752 (FP4) Vs 1.76 (BF16), showing the ability of the proposed method to work in a larger scale dataset. Then we compare the BF16 baseline with FP4 model on downstream tasks (similar to Table 3), including new tasks as required by the reviewer - GPQA (GQ), IfEval (IF), MBPP (MB), TrivialQA (TQ), XSum (XS): \n\n| Precision | Tokens | LA    | HS    | WG    | AC    | BQ    | PQ    | GQ    | IE    | MB   | TQ   | XS   |  *   | Avg   |  *   | Wiki(PPL) | LA(PPL) |\n|-----------|--------|-------|-------|-------|-------|-------|-------|-------|-------|------|------|------|-----|--------|-----|------------|----------|\n| BF16      | 1T     | 61.52 | 68.71 | 66.54 | 38.14 | 69.33 | 76.33 | 24.54 | 33.81 | 8.2  | 34.99| 11.97| *   | 45.63 |  *   | 5.54       | 6.10     |\n| BF16      | 1.04T  | 61.46 | 68.57 | 64.48 | 38.91 | 70.09 | 75.41 | 24.73 | 30.94 | 8.6  | 34.91| 12.33|  *   | 45.13 |   *  | 5.54       | 6.00     |\n| FP4       | 1T     | 58.39 | 67.31 | 64.01 | 38.65 | 69.33 | 74.86 | 24.73 | 32.13 | 6.4  | 34.30| 12.09|   *  | 44.46 |  *   | 5.83       | 6.77     |\n| +QAF      | +40B   | 61.71 | 68.53 | 65.98 | 39.25 | 68.90 | 76.01 | 27.29 | 32.25 | 9.4  | 38.31|  12.11   |  *   | 45.75 |  *   | 5.56       | 5.97     |\n\nWe agree with the reviewer that it will be interesting to see the proposed method on larger models (30B+), however it requires more compute resources than we currently have: just our experiments on a 7B model for 1T tokens took around 30 days on 256 devices. \n\n\n\n$\\textbf{Q1b}:$ Additionally, as recent work [1] has also proposed an end-to-end FP4 training framework, the authors are encouraged to clarify the distinctions and provide insights into how their approach compares or complements this effort.\n\n$\\textbf{A1b}:$ Please note we already cited this work as [18], and its distinctions from us are explained in lines 103-109 in the \"related work\" section and Table 2.  Specifically, our work is the first work that shows full fp4 training, i.e, quantization to fp4 of the 3 general-matrix-multiplications (GEMMs) (forward, backward, and update - Equations 1,2,3). In contrast, [18] quantized only the forward GEMM, i.e can accelerate only $\\sim 33$% of the GEMMs, which dramatically reduces the potential acceleration. Moreover, they trained only up to 100B tokens, which is an under-trained regime; we extend it to a 10x larger dataset. Please see Table 2 for a full comparison with [18].\n\n$\\textbf{Q2}:$ Due to the use of shared scaling across blocks, stochastic rounding, and dynamic precision switching, this could likely incurs non-trivial computational cost. Providing a quantitative analysis of these costs would offer clearer insight into the trade-offs between efficiency and precision stability.\n\n$\\textbf{A2}:$ Shared scaling across blocks is supported natively in hardware in Blackwell architecture without any acceleration cost, i.e it allows a 2x acceleration in comparison to FP8 and 4x in comparison with BF16. We expect additional AI accelerators also to support it natively without additional cost. \n\nStochastic rounding is also supported natively in the latest Nvidia's CUBLASS (v8.8) or in Gaudi2, without any significant overhead. \n\nNotice we do not use dynamic precision switching. We only increase once the precision of the backward (Eq.2) and update (Eq.3) GEMMs when starting the QAF process. In order to calculate the total cost, we can calculate the average bits operations during all process (pretraining + QAF), as we detail next.\nRecall we extended the pretraining phase to 1T tokens, followed by a 40B-token QAF phase. During pretraining all three GEMMs operate in FP4, while during QAF only one of the three GEMMs uses FP4 (with the remaining two in BF16). The resulting average-bits calculation is:$ [(1000\\times3\\times4) + (40\\times1\\times4) + (40\\times2\\times16)] / [1040\\times3] = 4.3$ bits. We will include this result in the camera-ready version.\n\n\n\n$\\textbf{Q3}:$ The authors are also encouraged to include comparisons to all-nearest or all-stochastic rounding configurations. This would be beneficial in understanding the relative comparison of different schemes for stable training.\n\n$\\textbf{A3}:$ In Fig. 3 we show the comparison to all-nearest (marked as \"full RtN\"). Moreover, we conducted an additional experiment similar to Fig. 3, where we independently applied RtN to one of the six elements while the remaining elements used SR. The resulting losses are as follows: Full SR: 3.022; Weights Forward RtN: 3.002 ; Activations Forward RtN: 3 ; Weights Backward RtN: 3.021 ; Activations Backward RtN: 3.032 ; Gradient Update RtN: 3.041 ; Gradient Backward RtN: 3.039.\n\nThis experiment highlights the effectiveness of the proposed regime, as described in Equations 4, 5, and 6, where SR is applied only to neural gradients during the update and backward GEMMs, and to activations during the update GEMM. We will include this experiment in the camera-ready version.\n\n$\\textbf{Q4}:$ How does the FP4-trained model compare to FP16 baseline on downstream tasks (e.g., QA, summarization, code generation)? Does full quantization affect generalization across domains or modalities? The authors are encouraged to include a few difficult baselines to demonstrate the at-par performance wrt BF16 baseline.\n\n$\\textbf{A4}:$ Please see the response A1a, where we show the evaluation on additional tasks for summarization (xsum), QA (trivial qa) and code generation (mbpp) as required by the reviewer.\n\n\n\n$\\textbf{Q5}:$ How sensitive is the training stability to the $\\sqrt{3}$ gradient-to-noise threshold? Also, can the authors elaborate on how and when precision switching is triggered in practice?\n\n$\\textbf{A5}:$ The $\\sqrt{3}$ gradient-to-noise threshold serves as a useful indicator for assessing the effectiveness of quantized training and identifying when it is no longer beneficial to continue training in low precision. Figure 5 illustrates a scenario where this threshold is crossed, suggesting that full low-precision training should be stopped. In contrast, for the experiment shown in Figure 6, the gradient-to-noise ratio remains above the threshold. This was a significant positive result that justified our decision to allocate the resources to continue training in 4-bit all the way to 1T tokens. It also demonstrates that with the optimal format (NVFP4) and our split rounding strategy, large-scale FP4 training can be successfully completed without even encountering the stagnation point that our theory predicts.\n\n\n$\\textbf{Q6}:$ How does the method behave under varying batch sizes? Smaller batches often lead to noisier gradients, does this make FP4 more prone to training collapse? Also, how sensitive is the FP4 training pipeline to initialization schemes?\n\n$\\textbf{A6}:$ Following the reviewer’s request, we run 3 runs with different micro-batch sizes (1K,10K,20K) in a 125M model trained over 30B tokens using the proposed scheme on Gaudi2 devices, we get similar training losses: 3.027, 3.025, 3.03\nWe believe that the use of per-block quantization (in contrast to per-tensor quantization) reduces the effect of batch size, mentioned by the reviewer.\nRegarding different initializations, please refer to response A7 below, where we tested multiple seeds (including different initializations) and did not observe any convergence issues.\n\n$\\textbf{Q7}:$ Given that the authors use of stochastic rounding for backward pass, how reproducible are the training runs across seeds or hardware environments? Are there significant variance in convergence / final performance for different random seeds?\n\n$\\textbf{A7}:$ Following the reviewer’s request, we conducted 5 runs with different seeds on a 125M model trained over 30B tokens using the proposed scheme on Gaudi2 devices. The resulting losses are: 3.03, 3.027, 3.025, 3.027,3.029 yield a loss standard deviation of 0.001, which is not significant. Moreover, we ran the same experiment on an A100 GPU and obtained an equivalent loss of 3.024. As the reviewer can see, we did not observe any convergence issues across different seeds or hardware platforms. We plan to include these ablation studies in the camera-ready version.\n\n$\\textbf{Q8}:$ As a suggestion, the authors are encouraged to further consider robustness related limitations, especially the potential vulnerability of their approach to noisy gradients or its applicability in vision and multimodal settings, where different modalities may exhibit different gradient distributions or magnitudes that could affect the stability of FP4 training.\n\n$\\textbf{A8}:$ Thank you for this suggestion. We will add it to the limitations section, explicitly emphasizing that our experiments focus on language tasks, while vision tasks are left for future work."}]

Author rebuttal:

Meta-review: This paper presents a practical approach to training large language models using FP4 format, along with comprehensive evaluations of various components (e.g., rounding functions and scaling formats). The research effectively bridges theory and practice by providing theoretical justifications supported by empirical evidence. All reviewers are clear to accept this paper, and the authors adequately addressed reviewer concerns during the rebuttal period. This timely work delivers remarkable results on an important research direction. I believe it will provide valuable guidance to the community for low-precision training of LLMs.
Decision: Accept (spotlight)