Optimizing Large Language Model Training Using FP4 Quantization
Authors: Ruizhe Wang
Yeyun Gong
Xiao Liu
Guoshuai Zhao
Ziyue Yang
Baining Guo
Zheng-Jun Zha
Peng CHENG
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Machine Learning
Quantization
FP4
Quantized Training
Primary area: deep_learning->large_language_models
TL;DR: This work introduces the first FP4 training framework for LLMs, achieving accuracy comparable to BF16 and FP8 with minimal accuracy degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens.

Abstract: The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training.

Optimizing Large Language Model Training Using FP4 Quantization
Ruizhe Wang 12f Yeyun Gong
Xiao Liu
Guoshuai Zhao
Ziyue Yang 32
Baining Guo
Zhengjun Zha'
Peng Cheng
32
Abstract
6
The growing computational demands of training large language models (L.MMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 preci- sion has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantiza- tion errors and limited representational capacity. This work introduces the first FP4 training frame- work for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an out- lier clamping and compensation strategy to pre- vent activation collapse. To ensure stability, the framework integrates mixed-precision training scheme and vector-wise quantization. Experimen- tal results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B to- kens. With the emergence of gext-eennrationn hard- ware supporting FP4, framework sets foun- dation for efficient ultra-low precision training.
FP4(Direct Cast)
FP4((uur))
5
BF16 Baseline
Loss
4
3
Tokens/Billion
2
Figure Directly casting to FP4 results in significantly higher training loss, whereas our proposed FP4 method achieves accuracy comparable to the BF16 baseline. These results are based on experiments with 400M LLaMA2 model.
lenges, demanding extensive time, energy, and financial resources. For example, Llama 3 (Dubey et al., 2024) 405B is trained on up to 16K H100 GPUs for 54 days. Similarly, GPT-4 (Achiam al., 2023), with an estimated 1T param- eters, required an extraordinary amount of computational power. These examples highlight the urgent need for more efficient training methods to keep up with the increasing demands of LLM development.
Model quantization has proven to be an effective technique for reducing training costs, as low-bit arithmetic kernels can save memory and accelerate computations when used appropriately. Most LLM training systems traditionally rely FP32 (full precision) FP16/BF16 (half precision) data formats, but quantization enables these formats to be reduced to lower precision, such as 8-bit or even 4-bit.
1. Introduction
In the past two years, the rapid development of large lan- guage models (LLMs) has significantly reshaped both re- search priorities and industrial practices. Theoretical anal- yses and empirical evidence consistently demonstrate that scaling up model size leads to substantial performance im- provements (Kaplan et al., 2020; Bi et al., 2024). However, training such large-scale models poses considerable chal-
Recent advancements in computational hardware, such as NVIDIA's H100 GPUs (Nvidia, 2023) and the upcoming B200 GPUs (Nvidia. 2024), have introduced support for low-bit arithmetic kernels, enabling more efficient compu- tation. The Hopper series GPUs feature tigh-performncee FP8 tensor cores, delivering 2x speed-up compared to FP16 tensor cores. Mcanwhile, the Blackwell series GPUs extend this capability by supporting FP6 and FP4 formats, with FP4 offering the potential to double computational throughput over FP8. Studies like FP8-LM (Peng al.,
Work done during internship in MSRA 'University of Science and Technology of China "Microsoft SIGMA Team "Microsoft Research Asia. Correspondence to: mitinng@@irrrrr.mmm..... Peng Cheng
Yeyun Gong <pengo@ www.mmicccoooooommm...
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Optimizing Large Language Model Training Using FP4 Quantization
2023) and NVIDIA's Transformer Engine (Nvidia, 2022) have demonstrated the feasibility of FP8 tensor cores for model training. But the application of FP4 tensor cores in model training remains an open research question.
For example, FP16 uses E5M10 and BF16 uses E8M7. FP8 typically has two variants: E4M3 and E5M2. In our work, we adopt the E2M1 format for 4-bit floating-point represen- tation, as defined in prior studies (Rouhani et al., 2023b;a), with 2 bits for the exponent and bit for the mantissa.
However, leveraging 4-bit data formats for neural network training presents significant challenges due to the extremely limited bit width. Directly quantizing LLMs to such a low- bit format often results in substantial accuracy degradation, as shown in Figure 1. This is primarily because low-bit formats are constrained by a limited dynamic range, which increases the risk of overflow and underflow. Even existing methods for 8-bit quantization experience some degree of accuracy loss, underscoring the difficulties of employing a 4-bit format, which provides only 16 distinct representable values.
Unlike integer (INT) quantization, floating-point (FP) quan- tization features uneven quantization intervals and larger dynamic range. To quantize a high-precision tensor like FP16 FP4, we employ the commonly used absmax method (Dettmers et al., 2022; Peng et al., 2023):
MAX
Tip4 - Q(1pp1 ?), 7 max (Tiiee]
Here, MAX134 fps represents the maximum absolute value in the FP4 format, and y serves as the scaling factor. For the E2MI configuration, MAXip4 is calculated to be 6.0. The quantization function Q( is implemented using a look-up table for quantization in a custom CUDA kernel since the FP4 format supports only 2 16 distinct values, Detailed format regulations and quantization implementation can be found in Appendix A.
In this study, pioneeringly propose framework for training language models using the FP4 format, providing validation of the feasibility of this ultra-low precision rep- resentation. To tackle the significant quantization errors associated with weights and activations during model train- ing, we present series of optimization techniques: (1) For weights, we present differentiable quantization estimator to improve gradient updates in FP4 computations. By ana- lyzing the impact of quantization on neural network forward and backward passes, we derive function with correction terms for accurate gradient estimation; (2) For activations, we develop an outlier clamping and compensation strategy to address the issue of outlier values commonly observed during LLM training. By analyzing activation distributions in LLMs, we introduce a clamping method and a sparse aux- iliary matrix to preserve quantization accuracy and maintain model performance.
3. Methodology
In a typical linear layer of a Transformer architecture, the computation can be expressed as Y = A W, where A is the activation tensor and W is the weight tensor. To fully leverage the capabilities of FP4 tensor cores, both A and W need to be quantized to FP4, as shown in Figure 2. How- ever, directly quantizing these tensors into FP4 introduces significant quantization errors. To address this challenge, we propose the differentiable gradient estimator method for weight tensors (Section 3.1) and the outlier clampping and compensation method for activation tensors (Section 3.2) to mitigate these issues.
We conduct comprehensive experiments to demonstrate that our FP4 training framework achieves accuracy compara- ble to models trained in BF16 or FP8 formats with the same hyperparameters. Leveraging the FP8 tensor cores of NVIDIA H100 GPUs to emulate FP4 computations, we train LLMs with up to 13B parameters and 100B training tokens, with minor training loss gap. For zero-shot evalu- ation on downstream tasks, model trained with FP4 show competitive results against BF16 models. We anticipate better speed performance gains with the availability of next- generation hardware like NVIDIA's B-series GPUs. Our training framework can be found at aka.ms/MS.AMP..
3.1. Differentiable Gradient Estimator
Quantization functions are inherently non-differentiable, preventing the reverse flow of the gradient during back- propagation. The widely used Straight-Thhouge Estimator (STE) (Bengio et al., 2013) bypasses this issue by assum- ing that the gradient of the quantized tensor is equivalent to that of the original tensor. However, this simplification introduces inaccuracies low-bit settings, as noted in prior studies (Yin et al., 2019; Gong et al., 2019).
2. Preliminaries
According to the IEEE 754 standard (Kahan, 1996), a binary floating-point number consists of three components: 1-bit sign (S), exponent bits (E), and mantissa bits (M). This is commonly represented as Ex ExMy, where x and y denote the number of bits for the exponent and mantissa, respectively.
To overcome these limitations, we propose Differentiable Gradient Estimator (DGE) that reduces estimation errors. DGE maintains direct quantization for forward computation to preserve hardware efficiency while introducing a gradient correction term derived from a differentiable approximation of the quantization function.
2

---

Optimizing Large Language Model Training Using FP4 Quantization
FP4 look-up table
1110
1100
1011
1010
1001
eave
0901
6010
0011
0100
0101
0110
0111
scaling factor
Input
Q
FP4 Tensor Core
-0.53
-1.39
-1.06
0.86
Oulput
Q
-1.13
1.25
-0.24
0.17
4
6
-1
0.5
Weight
Q
-0.80
0.70
0.29
0.57
quantize fumction
4
1.5
2
BF16 tensor
FP4 tensor
Figure 2. The structure of the proposed FP4 training scheme during the forward pass of linear layer. A high-precision tensor, such BF16. quantized into the FP4 format using look-up table quantization. During the GeMM computation. both weight and activation tensors quantized into FP4 leverage the FP4 tensor Two scaling factors then applied to the final result computational correctness.
Suppose we quantize the model weight with non- differentiable quantization function f: = f(W). Con- sidering the backward gradient computation for linear function with quantized weight, the forward pass be expressed as:
Since f non-differentiable quantization function, its derivative f' is almost everywhere zero, leading van- ishing gradients and causing the weight gradient compu- tation to fail, shown in Equation (6). The Straight- Through Estimator (STE) addresses this issue by assuming f'(W) = 1, thereby bypassing gradient vanishing. In other words, it directly assumes that OL/OW = OL/WW,
=
Y = AW, = Af(w)
(2)
During backpropagation, the loss gradient with respect to the weight 0T./ /0W and the activation 0T./0.t A are computed using the gradient propagated from the subsequent layer aL/OY. For the weight gradient, the chain rule gives:
To achieve more accurate gradient computation, we propose alternative approach: approximating the quantization function with well-chosen differentiable function, com- puting its derivative, and incorporating into Equation (6). Specifically, we use the following function to simulate the quantization behavior:
aL OL aW, aw aw, aw
(AT OL OW, OY aw
Where OW, /oW represents the derivative of the quantiza- tion function f. since f is element-wise function, its derivative f' is also alement-wise Thus we have:
8 f(x) - - sign(
2x
2x 1)
1)-1
1
(7)
Figure 3(a) illustrates this function under for the range [0, 0.5). which represents the first positive quantization in- terval in the E2M1 quantization scheme. This figure also shows that under the assumption of STE, forward quantiza- tion function is equivalent to f(z) because f'(x) 1. In Equation (7), 5 represents the quantization interval, and k a parameter that controls the degree of approximation. As increases, the function curve becomes sharper and more closely resembles the behavior of the original hard quantization function. For details on how Equation (7) is specified and derived, please refer to Appendix C.
aw, (i.1) aW 1
((((i,)) 10.
if (i.j) (k; otherwise.
(4)
Therefore aW, is diagonal matrix. When applied to the chain rule Equation (3), this diagonal structure allows simplification of the gradient computation, reducing it to an clement-wise multiplication between the two items:
ØL aw
ØL aw, i. (i)
j.
(5)
The derivative of Equation (7) can be expressed as:
or to be simplified:
1 2r -
aL = aL dw
f( -
(8)
aw, @ fWW).
(6)
Figure 3(b) and Figure 3(c) show the complete quantization
Where @ denotes the element-wise (Hadamard) product.
curve f(x) and its derivative f'(x:) under k 5 within
3

---

Optimizing Large Language Model Training Using FP4 Quantization
0.5
Hard
6
Hard
3.0
Hard
Quantized x
0.4
DGE
Quantized x
DGE
2.5
DGE
STE
2
STE
2.0
STE
0.3
0
1.5
0.2
1.0
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
-6
mppu x
imput X
input X
(a) single interval quantization
(b) full quantization
(c) full quantization derivative
Figure 3. Visualization of the Differentiable Gradient Estimator (DGE). (a) Comparison of three quantization methods: hard quantization. differentiable quantization, and STE quantization, demonstrated on a single quantization Step. (b) The full quantization curve for E2M1 quantization within its dynamic range' 6.0, 6.0]. (c) The derivative for the three methods, highlighting that hard quantization has a gradient of (x) 0, while STE assumes constant gradient of (x) 1.
the full E2M1 quantization framework. This framework consists of 14 distinct quantization intervals. In practice, the magnitude of f'(x) is capped at 3.0 to prevent infinite gradient spikes at 6/2 point, impacting only very small subset of elements. For the mathematical soundness of this operation, as well as the supplementary integration process and proof for the DGE method in actual training process, please refer Appendix C.
16'
16f
RF16
FB4
16f
161
16
16f
16
10
In practical model training, the Differentiable Gradient Estimator (DGE) is seamlessly integrated into the process. During the forward pass, we retain the hard quantization function for computational efficiency. For the backward pass, a correction term derived from Equation (8) is applied the weight gradient calculation following Equation (6).
-75
-50
-25
25
50
75
Activaation Value
clamp
FP6
3.2. Outlier Clamping and Compensation
During LLM training, activation tensors are significantly more challenging to quantize than weight tensors. This dif- ficulty arises from the complex distribution of activation tensor values, often dominated by outliers-speciff values that are substantially larger than the rest. Outliers pose significant challenge to tensor quantization by dispropor- tionately expanding the dynamic range of the target tensor, causing most values to underflow to zero after quantization.
-75
-50
-25 0
25
50
75
Adtivsation
Figure 4. Visualization of the outlier clamping method, based the first transformer layer's output of the LLaMA 1.3B model after 30,000 training itcrations. Up: Quantization performed without outlier clamping, lcading to severe loss of information. Down: Quantization after applying outlier clamping, effectively preserv- ing tensor structure.
To address this issue, we propose the Outlier Clamping and Compensation method (OCC) restrict the range of acti- vation tensors and mitigate the underflow problem. Specif- ically, we identify outliers-values -vaues with the largest abso- lute magnitudes- them to predefined threshold. Given a pre-defined quantile a, the clamping function can be expressed as:
without outlier clamping, based on a real activation tensor extracted from the first transformer layer's output of the LLaMA 1.3B model after 30,000 training iterations, where 0.999. This approach significantly reduces the mean squared error (MSE) between the original and quantized tensors, enhancing quantization quality and maintaining training stability.
quantile identification and clamp
clamp(Y, max a, min a)
(9)
We also observed that while clamping effectively reduces quantization error, it inherently introduces some error by
Figure 4 illustrates the impact of quantization with and

---

Optimizing Large Language Model Training Using FP4 Quantization
load, with this proportion increasing for larger models. Con- sistent with prior works (Xi et al., 2023; Yang al., 2020; Dettmers et al., 2022), we focus on 4-bit quantization for GeMM operations, a core feature of FP4 tensor cores. In GeMM computation Y = AW, where A (sequence length x input channels) is the activation tensor and W (input chan- nels x output channels) is the weight tensor, quantization is applied along distinct dimensions to align with matrix multiplication logic: A is quantized token-wise (sequence length dimension), while W is quantized channel-wise (out- put channels dimension). The aforementioned accuracy- preserving techniques are integrated to minimize quanti- zation error. Since FP4 Tensor Cores are unavailable, we validate FP4 performance using Nvidia H-series GPUs' FP8 Tensor Cores, which encompass FP4's dynamic range and enable accurate simulation.
Table 1. Quantitative analysis of mathematical accuracy between original and quantized activation tensors. Results represent the average values obtained across all activation tensors on the 30,000 training iterations of the LLaMA 1.3B model.
CLAMP
COMP
QUANTILE
SIM f
MSF 4
SNR
92.19%
0.1055
8.31
99.9
98.83%
0.0366
14.25
V
99.9
99.61% %
0.0245
15.31
99
100%
0.0099
18.38
V
97
100%
0.0068
20.88
disregarding the outlier values. To further preserve accu- racy, we propose compensating for this error using sparse outlier matrix. In our experiments, the quantile clamping threshold a is set relatively high (around 0.99 ~ 0.999), making the residual matrix AY = = - Y. highly sparse, with only about 0.2% ~ 2% non-zero elements. During computation, the clamped matrix Y. is processed using FP4 GeMM, while AY is handled with high-precision sparse matrix multiplication.
In mixed-precision training (Micikevicius et al., 2017), non- GeMM operations, which account for a minor computa- tional fraction, are performed at higher precision to preserve accuracy. Following the framework in (Peng et al., 2023), we perform gradient communication in FP8 format to reduce bandwidth usage and adopt their mixxed-precssion Adam op- timizer to conserve GPU memory. Gradients and first-order moments are stored in FP8, while second-order moments are stored in FP16. Remaining operations, comprising a smaller computational portion, are executed in FP16 or BF16 for stability and precision.
Table 1 provides a quantitative analysis of cosine similarity (SIM), mean squared error (MSE), and signal-to-noise ratio (SNR) between the original activation tensors and quantized tensors. These results represent average values obtained across all activation tensors on the 30,000 training iterations of the LLaMA 1.3B model, demonstrating the impact of outlier clamping and compensation on preserving tensor fi- delity during real model training. The data shows that outlier clamping significantly improves both cosine similarity and SNR. Moreover, incorporating outlier compensation further reduces quantization loss. Notably, lowering the quantile threshold increases the compensation scale, further reduc- ing quantization loss. However, this introduces trade-off between computational efficiency and numerical accuracy that must be carefully considered.
We adopt the widely recognized LLaMA model (Tou- vron et al., 2023) as the primary model architecture. The training is conducted from scratch using the DCLM dataset (Li et al., 2024), comprehensive dataset well-suited for language model pretraining. Hyperparameters remain con- sistent across precision settings for fair comparison. The learning rate follows a warm-up and cosine decay schedule, with the warm-up phase spanning 5% of total steps and the learning rate gradually decreasing to 10% of its peak over the remaining 90%. The peak learning rate is 3 x 10-1, with weight decay of 0.1. For the Adam optimizer, we use 31 = 0.9, 32 = 0.95, and e = 1 x 10-8, For spe- cial hyperparameters used in FP4 method, we use k = 5 for differentiable gradient estimator and select a = 0.99 as the activation clamp and compensation quantile. Input sequences are fixed at 2048 tokens, and the batch size is 2048, comprising approximately 4M tokens.
4. Experiment
In this section, we evaluate the proposed FP4 training frame- work across language models of various sizes. Section 4.1 details the implementation of our FP4 training framework, including the model architecture and hyperparameters. Sec- tion 4.2 presents the main results, showcasing training curves and zero-shot performance on downstream tasks. Finally, Section 4.3 provides ablation studies to further vali- date the effectiveness.
4.2. Main Results
We validate the effectiveness of our proposed FP4 training framework by comparing it against the widely adopted 6F16 training scheme. Figure 5 presents the training loss curves for LLaMA models (1.3B, 7B, and 13B) trained with BF16 and FP4 precision. All models are trained on 100B tokens using the same dataset and identical
4.1. Experiment Setup
During LLM training, General Matrix Multiplication (GeMM) accounts for over 95% of the computational work-
5

---

Optimizing Large Language Model Training Using FP4 Quantization
Table 2. Zero-shot evaluation for downstream tasks between BF16 models and FP4 models under different model sizes.
Model Size
Precision
Average
PIQA
Hellaswag
ObQA
Arc-C
Arc-E
BoolQ
LogiQA
SciQ
Lambada
BF16
53.23
71.11
50.80
36.60
36.69
68.60
57.83
30.26
83.30
1.3B
(P4(O)r))
53.13
70.89
50.82
36.20
36.86
67.47
58.23
29.49
83.90
44.30
BF16
53.87
71.22
52.03
37.40
38.99
67.47
60.55
27.65
85.00
44.56
7B
FP4(Ours)
54.42
71.87
52.97
38.40
39.85
67.97
62.20
27.96
84.70
43.88
BF16
54.44
72.80
53.56
38.60
38.82
67.97
57.40
29.65
86.30
44.87
13B
FP4(Ours)
54.95
73.78
54.12
39.60
39.68
67.89
55.90
30.88
85.80
46.89
8
8
BF16 (Pum)))))
BF16
7
7
7
FP4(Ours)
[Pu(()us))
6
6
Loss
Loss
4
4
4-
3
3
3-
2
2
20
40
60
80
100
20
40
60
80
100
0
20
40
60
80
100
Billion takens
Billion tokens
Billion tokens
(a) LLaMA-1.3B
(b) LLaMA-7B
(c) LLaMA-13B
Figure 5. Training curves for BF16 models and FP4 models under different model sizes. (a) Training curves for 1.3 LLaMA model. (b)
Training curves for 7B LLaMA model. (c) Training curves for 13B LLaMA model.
hyperparameters. The curves for BF16 and FP4 largely overlap across different model sizes, with the FP4 curve exhibiting slightly higher training loss compared to the BF16 curve. Specifically, after training 100B tokens, the training losses for the 1.3B model, 2.17 (FP4) vs. 2.07 (BF16) 6) for the 7B model, and 1.97 (FP4) vs. 1.88 (BF16) for the 13B model.
Table 3. Perplexity evaluation for downstream tasks between BF16 models and FP4 models under different model sizes.
follows: 2.55 (FP4) vs. 2.49 (BF16)
Size
Precision
Average
L.b.LAAT
1.bd.tté
Pile 10k
Wikitest
BF16
37.38
14.98
25.10
82.77
26.65
L33
FP4(Ours)
36.86
15.33
23.07
82.52
26.51
BF16
35.06
14.34
23.33
77.72
24.86
In addition to training loss, we evaluate the models on di- verse set of downstream tasks datasets in zero-shot manner, including Are (Clark et al., 2018), BoolQ (Clark et al., 2019), HellaSwag (Zellers et al., 2019), LogiQA (Liu al., 2021), PiQA (Bisk et al., 2020), SciQ (Welbl et al., 2017), Open- bookQA (ObQA) (Mihaylov et al., 2018), and Lambada (Paperno et al., 2016). These results are obtained through the widely used in-evaluation-hure 2024). As presented in Table 2, models pre-trained with FP4 demonstrate competitive performance in intrinsic in- context learning capabilities. Under the same model size, the average accuracy of FP4-trained models is comparable to, or even slightly exceeds, that of BF1 6-trained models. Additionally, the results follow the general trend: larger models achieve higher accuracy under the same number of training tokens.
FP4(Ours)
35.62
14.29
24.42
78.42
25.36
BF16
33.69
12.42
22.45
75.06
24.81
13B
TP4(Ours)
33.99
13.67
21.62
75.84
24.83
Table 3 further presents the perplexity (PPL) evaluation results for several downstream datasets including Lambada OpenAl (Lbd.OAI), Lambada standard (Lbd.std) (Paperno et al., 2016), the Pile 10k (Gao et al., 2020) and Wikitext (Merity et al., 2017). The results demonstrate that FP4 models achieve comparable or even slightly lower PPL than BF16 models. As expected, larger models achieve lower perplexity under the same training token budget.
library' (Gao et al.,
These results highlight that despite the reduced precision, FP4 training achieves nearly equivalent performance to BF16 both in terms training loss and downstream task ac- curacy, making a promising approach for efficient training of large language models.
'ttpps: //g/ thub. com http//eeetherAAA//// evaluation ion harness
6

---

Optimizing Large Language Model Training Using FP4 Quantization
8
5.0
W4A4
W4A8
7
WAA+-DGE+OCC
4.5
WAAB-DGE(K-))
MSAMP FP8
3.40
W4AR-DGE(=S)
2.40
TE FP8
W4AR-DGE(K-10
BF16
4.0
DF16
Loss
Loss
5
2.56
3.5
4
3.0
10
10
Billion tokens
Billion tokens (b) Ablation on weight
(a) Ablation on precision
10
10
9
W9A++0CC((-0.9999
NaN
Wifine
Ascoarse
Arconese
2.50
WBA++OCC(=00.99)
Wiccarsc Affine
WM14HOCC((-0.97)
W-fine
A:fine
Loss
BF16
2.79
1.40
Loss
BF
2.15
6
6
5
4
4
3
3
10
10
Billion tokens
Billion tokens
(c) Ablation on activation
(d) Ablation on granularity
Figure 6. Ablation studies. (a) Training curves under different precision frameworks. (b) The effect of proposed Differentiable Gradient Estimator (DGE). (c) The effect proposed Outlier Clamping and Compensation method (OCC), Note that directly casting activation into 4-bit leads divergence, and the loss value into NaN (Not Number). (d) Training curves under different quantization granularities of FP4.
4.3. Ablation Study
Weights. For weight-only 4-bit quantization (W4A8), we evaluate our Differentiable Gradient Estimator (DGE) method alone against direct quantization. As shown in Figure 6(b), the DGE method significantly improve conver- gence. Notably, direct quantizing weight into 4-bit doesn't t introduce a substantial training loss gap, suggesting that weights easier quantize than activations. For the hy- perparameter k in this method, larger & can better model the quantization function, but it can also lead to more unstable correction term for the gradient. It can also be seen in the figure that a moderate = 5 gives better final performance.
We divide ablation study into smaller parts to better highlight the findings of FP4 training. All experiments are conducted on the LLaMA 1.3B model, trained with 10B tokens from subset of the DCLM dataset. To acceler- ate convergence for this smaller model. the batch size is reduced from 2048 256, while other hyperparameters remain consistent with the main experiments.
Precision. Figure 6(a) presents training curves across various precisions, including BF16 (baseline), MS-AMP FP8 (Peng et al., 2023), Transformer-Engine FP8 (Nvidia, 2022), directly-casted FP4, and our FP4 method. We use W4A4 to denote direct quantization, meaning that quantizing both weight and activation to fp4. Meanwhile, W444+DGE+OCE denotes our fp4 quantization method that incorporates the Differentiable Gradient Estimator (DGE) and Outlier Clamp and Compensation (OCC) meth- ods introduced in Section 3. The loss curves show that two FP8 methods and our FP4 approach maintain pretraining ac- curacy, while directly-casted FP4 has a significant training loss gap.
Activation. For activation-only 4-bit quantization (W8A4), evaluate our Outlier Clamp and Compensation (OCC) method alone against direct quantization. Figure 6(c) re- veals that directly quantizing activations in FP4 results in curve divergence, where the loss values turn into NaN (Not a Number) after certain training steps. Outlier clamping and compensation effectively reduces this loss gap, ensur- ing a good convergence. This experiment re-emphasizes the importance of appropriate treatment of outliers in the absmax quantization framework. For the hyperparameter

---

Optimizing Large Language Model Training Using FP4 Quantization
in this method, smaller a implies stronger compensation, but an increased computational cost. Figure 6(C) shows the model loss under three settings = 0.999, 0.99, 0.97, corresponding to the non-zero elements of the sparse com- pensation matrix of 0.2%.2% and 6%, respectively. Al- though experiments show that a smaller a leads to better model accuracy, which is consistent with the conclusion of Table we believe that a 0.99 is a better choice for comprehensive computational performance considerations.
LLM inference (Dettmers & Zettlemoyer, 2023; Wu et al., 2023). However, these methods focused on LLM infer- ence, requiring additional computation like calibration set fine-tuning (Wang et al., 2024), rotary matrix and low-rank compensation (Lin et al., 2024b; Ashkboos et al., 2024; Li et al., 2025), quantization parameters searching (Liu et al., 2023a), or even retraining the whole network (Ma et al., 2024). In the field of FQT, an early study (Sun et al., 2020) applied a 4-bit radix-4 FP4 format convolutional neural networks (CNNs). MXFP (Rouhani et al., 2023b) intro- duced novel quantization data for GPT-style models, but lacked feasibility validation on full FP4 settings. (Xi et al., 2023) proposed an INT4 training framework, but their focus was on fine-tuning tasks with limited applicability to LLM pretraining. In contrast, our work is the first to propose an FP4 training framework tailored for LLMs, validated from scratch, and designed to align with next-generation hardware like Nvidia's B-series GPUs.
Granularity. We also observe that the granularity of FP4 quantization plays a critical role. While FP8 training schemes (Peng et al., 2023; Nvidia, 2022) achieve sufficient accuracy with coarse- grained tensor-wise quantization, Fig- ure 6(d) shows that tensor-wise scaling in FP4 introduces significant errors. To address this, we adopt vector-wise scaling, with token-wise quantization for activations and channel-wise quantization for weights, aligning with GeMM computation rules as discussed in Section 4.1. Notably, applying coarse-graneed quantization to activations alone result in more severe accuracy degradation than applying it to weights alone, revealing that activations are harder to quantize than weights, consistent with the activation outlier issue described in Section 3.2.
Differentiable Quantization. Unlike previous methods focusing on differentiable quantization (Gong et al., 2019; Uhlich et al., 2019; Chen et al., 2019; Li et al., 2022; Huang et al., 2022), which rely on learnable quantization param- eters updated through backpropagation, our differentiable gradient estimator method uses fixed quantization func- tion. We directly change the gradient estimator from STE to DGE during the backward pass, avoiding the need for continuous updates to the quantization function, which is not friendly to specialized hardware designs. Our approach is more efficient and more suitable for hardware acceleration in large-scale training.
5. Related Work
Quantized Training and Inference. When discussing the quantization of large language models (LLMs) for train- ing, we typically refer to Fully Quantized Training (FQT). Related research efforts have generally used Mixed Preci- sion Training (Micikevicius et al., 2017; Mellempudi et al., 2019) frameworks to accelerate model training while main- taining model accuracy. While previous research has mainly concentrated on CNNs or DNNs(Sun et al., 2019; Wang et al., 2018; Banner et al., 2018; Yang et al., 2020), recent studies have demonstrated the feasibility of low-bit mixed precision training for LLMs (Peng et al., 2023; Nvidia, 2022; Fishman et al., 2025; Xi al., 2024). In contrast to the FQT scheme, research on low-bit computation for infer- ence has focused on Post-Training Quantization (PTQ) and Quantization Aware Training (QAT). While PTQ directly quantizes pre-trained models for inference (Dettmers et al., 2022; Frantar et al., 2023; Lin et al., 2024a; Xiao et al., 2023; Yao al., 2022: Liu al., 2024), QAT involves fine- tuning or pre-training the model for better low-bit inference performance (Liu et al., 2023b; Cheng et al., 2023; Wang et al., 2023; Dettmers et al., 2023). Our method differs from QAT, as we aim to accelerate the training process while maintaining performance, rather than solely focusing on improving inference efficiency without consideration for the training speed.
Handling Outliers. Our method for handling activation out- liers in LLMs differs significantly from existing approaches, which mainly target model inference (Liu et al., 2023a; Li et al., 2025; Ashkboos et al., 2024; Liu et al., 2024). Activa- tion outliers in LLMs are typically channel-specific (Xiao et al., 2023; Wei et al., 2022). Channel quantization would reduce quantization loss but conflicts with the compu- tation structure of matrix multiplication in linear layers (Xi et al., 2024; Lee et al., 2024). Previous strategies to solve this problem like smoothing outliers (Xiao et al., 2023) or us- ing rotary matrices (Ashkboos et al., 2024; Liu et al., 2024) rely on offline pre-processing, making them incompatible with pretraining tasks. In contrast, our method addresses outliers dynamically during real-time training without re- quiring separate calibration datasets, which is critical for maintaining efficiency in pretraining large models.
6. Limitation
One primary limitation of this work lies in the absence of dedicated FP4 Tensor Cores in existing hardware. Conse- quently, we are unable to directly measure the potential speedup and energy efficiency gains achievable with native
4-bit Quantization. Recent works in PTQ and QAT have successfully applied 4-bit, 2-bit or even I-bit quantization to
8

---

Optimizing Large Language Model Training Using FP4 Quantization
FP4 support. All current experiments rely on FP4 simula- tions, which introduce additional computational overhead due to extra precision casting and significantly prolong run- time. Additionally, due to constraints on computational resources, we have not yet extended our experiments to extremely large-scale models or to datasets comprising tril- lions of tokens. Investigating such scalability remain as critical directions for future research.
Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable Methods for 8-bit Training of Neural Networks. Advances in Neural Information Processing Systems, 31, 2018.
Bengio, Y., Léonard, N., and Courville, A. Estimat- ing or Propagating Gradients Through Stochastic Neu- rons for Conditional Computation. arXiv preprint arXiv: 1308.333,, 2013.
Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q. Fu, 2., et al. Deepseek LLM: Scaling Open-Source Language Models with Longter- mism. arXiv preprint arXiv:2401.02955 2024.
7. Conclusion
We propose the first FP4 pretraining framework for mod- em Large Language Models (LLMs), overcoming the chal- lenges of limited dynamic range and quantization precision in 4-bit formats. By proposing a differentiable gradient estimator and an outlier compensation mechanism, we ef- fectively reduce the accuracy gap between FP4 and higher- precision baselines like FP8 or FP16, achieving compara- ble performance across diverse model scales. Our findings demonstrate the feasibility of FP4-based training, provid- ing insights into improving quantization methods for ultra- low-precision computing, and may also serve as a call for next-generation hardware designs to enable efficient 4-bit computation kernels.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reason- ing about Physical Commonsense in Natural Language. In Proceedings of the AAAI conference on artificial intel- ligence, volume 34, pp. 7432-7439, 2020.
Chen, S., Wang, W., and Pan, S. J. MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization. Advances in Neural Information Process- ing Systems, 32, 2019.
Cheng, W., Zhang, W., Shen, H., Cai, Y., He, X., Lv, K., and Liu, Y. Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs. arXiv preprint MWXi:223090055/6 2023.
Impact Statement
This work demonstrates the feasibility of using ultra-low precision formats like FP4 for training large language mod- els, offering a pathway toward energy conservation and reduced carbon emissions in Al development. By signifi- cantly lowering computational demand, FP4-based methods can democratize access to advanced Al systems while pro- moting environmental sustainability.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M. and Toutanova, K. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pp. 2924 2936, 2019.
Additionally, this research calls for sext--eneration Al accel- erators optimized for 4-bit computations, potentially shap- ing future hardware innovations. However, broader socictal implications must be considered, including the risks of mis- use and the amplification of biases inherent in large-scale AI models. Addressing these challenges is essential to ensure responsible and equitable adoption of this technology.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Chal- 2018.
lenge. arXiv preprint
Dettmers, T. and Zettlemoyer, L. The case for 4-bit preci- sion: k-bit Inference Scaling Laws. In International Con- ference Machine Learning, pp. 7750-7774. PMLR, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.088 087744 2023.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. GPT3.int80) 8-bit Matrix Multiplication for Transform- ers at Scale. Advances in Neural Information Processing Systems,
2022.
Ashkboos, S., Mohtashami, A., Croci, M., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. QuaRot: Mutlier-Free 4-Bit Inference in Rotated LLMs. Advances in Neural Information Processing Systems, 37: 100213-100240, 2024.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. Advances in Neural Information Processing Systems, 36: 10088-10118 2023.
9

---

Optimizing Large Language Model Training Using FP4 Quantization
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024.
Li, J., Fang, A., Smyrnis, G., Ivgi, M., Jordan, M., Gadre, S. Y., Bansal, H., Guha, E., Keh, S. S., Arora, K., et al. DataComp-LM: In search of the next generation of train- ing sets for language models. Advances in Neural Infor- mation Processing Systems, 17:14200-14287 2024.
Fishman, M., Chmiel, B., Banner, R., and Soudry, D. Scal- ing FP8 training to trillion-token LLMs. In The Thirteenth International Conference on Learning Representations, http:/iwwwweee///teeeeeeee
Li, M., Lin, Y., Zhang, Z., Cai, T., Guo. J., Li, X., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. SVDQuant: Ab- sorbing Outliers by Low-Rank Component for 4-Bit Dif- fusion Models. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL https: / openreview net/ dorrmmidd-wRR3K..
2025. URL https:
id=E1EH00
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. GPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview wwwpreeieeeeeeet/oruumeeeee....
Li, Y., Xu, S., Zhang, B., Cao, X., Gao, P., and Guo, G. Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer. Advances in Neural Information Processing Systems, 35:34451-34460 2022.
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, arXiv preprint arXiv:2101.00027, 2020.
Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang. W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration. Proceedings of Machine Learning and Systems, 6:87-100, 2024a.
Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi,
A., Foster, C., Golding, L., Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang, J., Reynolds, L.., Schoelkopf, H., Skowron, A., Sutawika, L., Tang, E., Thite, A., Wang. B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: zenodo. org/ records// 12608602.
Lin, Y., Tang, H., Yang. S., Zhang, Z., Xiao, G., Gan, C., and Han, S. QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving. arXiv preprint wrxiv:2405.04532, 2024b.
Liu, J., Cui, L.., Liu, H., Huang, D., Wang, Y., and Zhang, Y. LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning. In Proceedings of the Twenty-Ninth International Conference on Inter- national Joint Conferences Artificial Intelligence, pp. 3622-3628, 2021.
Gong. R., Liu, X., Jiang. S., Li, T., Hu, P., Lin, J., Yu, F., and Yan, J. Differentiable Soft Quantization: Bridging Full- Precision and Low-Bir Bit Neural Networks. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4852-4861, 2019.
Liu, S., Liu, Z., Huang. X., Dong, P., and Cheng, K.-T. LLM-FP4: 4-Bit Floating-Poin Point Quantized Transform- ers. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023a. URL https: / //openreview.cr new. http://ommmmmmdwiimmmeda
Huang, X., Shen, 7., L.i, S., Liu, 7.,, Xianghong, H., Wicak- sana, J., Xing, E., and Cheng, K.-T. SDQ: Stochastic Differentiable Quantization with Mixed Precision. In In- ternational Conference on Machine Learning, pp. 9295- 9309. PMLR, 2022.
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mehdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LLM- QAT: Data-Free Quantization Aware Training for Large Language Models. arXiv preprint arXiv:2305. 17888, 2023b.
Kahan, W. IEEE standard 754 for binary floating-point arithmetic. Lecture Notes on the Status of IEEE, 754 (94720-1776):11, 1996.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling Laws for Neural I Language Models. arXiv preprint arXiv:2001.0836) 2020.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. SpinQuant: LLM quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
Lee, C., Jin, J., Kim, T., Kim, H., and Park, E. OWQ: Nutlier-Aware Weight Quantization for Efficient Fine- Tuning and Inference of Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pp. 113644, 2024.
Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv preprint arXiv:2402.1 17764, 2024.
10

---

Wang. N., Choi, J., Brand, D., Chen, C.-Y., and Gopalakr- Optimizing Large Language Model Training Using FP4 Quantization
Mellempudi, N., Srinivasan, S., Das, D., and Kaul, B. Mixed Precision Training With 8-bit Floating Point. arXiv preprint arXiv:1905. Iet.i. 12334, 2019.
Sun, X., Choi, J., Chen, C.-Y,, Wang. N., Venkataramani, S., Srinivasan, V. V., Cui, X., Zhang, W., and Gopalakr- ishnan, K. Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks. Advances in Neural Information Processing Systems, 32, 2019.
Merity, S., Xiong, C.,, Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. In International Conference on Learning Representations, 2017.
Sun, X., Wang, N., Chen, C.-Y., Ni, J., Agrawal, A., Cui, X., Venkataramani, S., El Maghraoui, K., Srinivasan, V. V., and Gopalakrishnan, K. Ultra-Low Precision 4-bit Training of Deep Neural Networks. Advances Neural Information Processing Systems, 33:1796-1807,, 2020.
Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., Ginsburg, B., Houston, M., Kuchaiev, O., Venkatesh, G., et al. Mixed Precision Training. arXiv preprint 171.:07100007400. 2017.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open Foundation and Fine- Tuned Chat Models. arXiv preprint arXiv:2307.09288. 2023.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can Suit of Armor Conduct Electricity? New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
Uhlich, S., Mauch, L., Yoshiyama, K., Cardinaux, F., Gar- cia, A., Tiedemann, S., Kemp, T., and Nakamura, A. Differentiable Quantization of Deep Neural Networks. arXiv preprint arXiv: /452, 2(8), 2019.
Nvidia.
Using FP8 with Transformer Engine, 2022. URL https://docs /ttp:::/d/ocss.niiie.... nvidia. com/ deepleeammicccttttttttrereerggge
Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang. F., Wang, R., Wu, Y., and Wei, F. BitNet: Scaling 1-bit Transformers for Large Language Models. arXiv preprint arXiv:2310.1 1453, 2023.
Nvidia. NVIDIA H100 Tensor Core GPU Architecture,
2023. URL https: wttp:////eesorreseeeeeee nvidia. com/
en-us-tensor-com
Wang. J., Liu, H., Feng, D., Ding, J., and Ding, B. FP4- Quantization: Lossless 4bit Quantization for Large Lan- guage Models. In 2024 IEEE International Conference on Joint Cloud Computing (JCC), pp. 61-67. IEEE, 2024.
Nvidia. NVIDIA Blackwell Architecture Technical Brief, 2024. URL https::/
en-us- wtwe---awacaeeeeeeeeeeeeeeeeeeee
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word predic- tion requiring& a broad discourse context. In Proceed- ings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1525-1534, 2016.
ishnan, K. Training Deep Neural Networks with 8-bit Floating Point Numbers. Advances in Neural Information Processing Systems, 31, 2018.
Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q.. Yu, F., and Liu, X. Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models. Ad- vances in Neural Information Processing Systems, 35: 17402-17414, 2022.
Peng, H., Wu, K., Wei, Y., Zhao, G., Yang, Y., Liu, Z.,, Xiong, Y., Yang, Z., Ni, B., Hu, J., et al. FP8-LM: Training FP8 Large Language Models. arXiv preprint arXiv:2310.18313. 2023.
Welbl, J., Liu, N. F., and Gardner, M. Crowdsourcing Multiple Choice Science Questions. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 94- 106, 2017.
Rouhani, B. D., Garegrat, N., Savell, T., More, A., Han, K.-N., Zhao, R., Hall, M., Klar, J., Chung, E., Yu, Y., et al. OCP Microscaling Formats (MX) Specification, 2023a. URL https: www.ooppute.cceeeee.mmo...oo org/ Abcuments/oop-mmierrssaall formass mx-v1- fina-ppe--faappppd
Wu, x., Li, C., Aminabadi, R. Y., Yao, Z., and He, Y. Un- derstanding Int4 Quantization for Language Models: La- tency Speedup, Composability, and Failure Cases. In In- ternational Conference on Machine Learning, pp. 37524- 37539. PMLR, 2023.
Rouhani, B. D., Zhao, R., More, A., Hall, M., Khodamoradi, A., Deng, S., Choudhary, D., Cornea, M., Dellinger, E., Denolf, K., et al. Microscaling Data Formats for Deep Learning. arXiv preprint arXiv:2310. 10537, 2023b.
Xi, H., Li, C., Chen, J., and Zhu, J. Training Transform- ers with 4-bit Integers. Advances in Neural Information Processing Systems, 36:491 (9--9968, 2023.

---

Optimizing Large Language Model Training Using FP4 Quantization
Xi, H., Chen, Y., Zhao, K., Teh, K. Chen, J., and Zhu, Jetfire: Efficient and accurate transformer pretraining with int8 data flow and per-block quantization. In Inter- national Conference on Machine Learning, pp. 54049- 54063. PMLR, 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and Efficient Post- Training Quantization for Large Language Models. In In- lernational Conference on Machine Learning, pp. 38087- 38099, PMLR, 2023.
Yang. Y., Deng, L., Wu, S., Yan, T., Xie, Y., and Li, G. Training PereoPeeffrmance and Large-Scale Deep Neural Networks with Full 8-bit Integers. Neural Networks, 125: 70-82,20 2020.
Yao, Z., Yazdani Aminabadi, R., Zhang, M., Wu, X., Li, C., and He, Y. ZuroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. Advances in Neural Information Processing Systems, 35: 77168-27182 2022.
Yin, P., Lyu, J., Zhang, S., Osher, S., Qi, Y., and Xin, J. Understanding
Estimator in Training Activation Quantized Neural Nets. In International Con- ference on Learning Representations, 2019. 2019.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800, 2019.
12

---

Optimizing Large Language Model Training Using FP4 Quantization
A. Implementation of FP4 Quantizaiton
Floating-poin numbers in a computer are represented using a binary format defined by the IEEE 754 standard (Kahan, 1996). Each number is divided into three components: the sign bit (S), the exponent (E), and the mantissa (or significand, M). This commonly represented ExMy, where x and y denote the number of bits for the exponent and mantissa, respectively, The sign bit determines whether the number is positive (S 0) negative (S 1). The exponent, stored in biased format, encodes the power of two that scales the number, enabling the representation of wide range of values. The mantissa contains the significant digits of the number, capturing its precision. A normalized floating-pon -pint number is decoded as:
Value (-1)$ x (1.M) 2
Where 1.M represents the normalized mantissa with an implicit leading 1, and the bias (e.g., 127 for single precision or 1023 for double precision) adjusts the exponent to account for its encoding. Subnormal numbers, where the exponent all zeros, are handled separately with no implicit leading 1. This representation allows for efficient computation but introduces rounding eπrоrs due to the limited number of bits in the mantissa.
The IEEE 754 standard does not define rules for floating-por point formats with precision below 16 bits, such FP8 and FP4, For 4-bit floating-poin representation, we adopt the E2M1 format as defined in prior studies (Rouhani al., 2023b;a). According to the IEEE definition, an exponent field (E) filled with ones does not correspond to a valid numeric value; instead, it represents infinity (Inf) when the mantissa (M) is all zeros or an invalid number (NaN, Not a Number) when the mantissa contains nonzero bits. However, this rule often disregarded FP8 and FP4 formats due their limited bit width, as the priority is maximize the representation of meaningful numerical values. For example, EP8-E4M3 format doesn't t define Inf, FP6 and FP4 formats don't define both Inf and NaN.
Based on the distribution of exponent and mantissa bits, all representable numbers in the FP4 format are listed in Table 4.
Table FP4 Quantization Table under different FP4 formats.
SEQUENCE
FORMAT
1111
1110
1101
1100
1011
1010
1101
1000/0000
0001
0010
0011
0100
0101
0110
0111
-2.5
1:0
#2M1
-4
-3
-2
-1.5
-1
-0.5
4:0
0.5
1.5
2
3
4
=3H0
-16
-8
-4
-2
-1
-0.5
-0.25
4:0
0.25
0.5
2
4
8
16
-3
-2
-1
0
2
3
.3.
225
-1.
J05
0.5
1.5
25
3.5
E1M2
-6
-4
33
-2
-1
0
2
3
6
05
13
H2M1
-16
-1
0
8
E3M0
Figure 7. Visualization of all representable numbers in different FP4 formats.
The "E0M3" format is not included here because it is equivalent to the INT4 format, it doesn't have any exponent bits. From Table and Figure 7, we observe that increasing the number of exponent bits expands the dynamic range, while increasing the number of mantissa bits improves the precision of quantization intervals. We select the E2M1 format in our main experiments as it offers a balanced trade-off' between dynamic range and quantization precision.
Since the FP4 format supports only 16 distinct values, we implement a look-up table for FP4 quantization in a custom CUDA kernel. Quantization functions typically involve element-he deeenntby-eeeeeee by-element operations on large amounts of data, which can
13

---

Optimizing Large Language Model Training Using FP4 Quantization
be parallelized to take advantage of the highly parallel computing power of GPUs. The following code paragraph shows the implementation of the quantization kernel.
global void quantize kernel (const float* x, float* output, x size) {
2
int idx blockidx.X * blockDim.x x threadidx.x;
3 if (idx < x_size) {
4
float value x[ixx];
5
float closest;
7
closest
(value < -5.0f)
? -6.0f
8
(value < -3.5f)
? -4.0f
9
(value -2.5f)
-3.0f
(value -1.75f)
-2.0f
(value < -1.25f)
-1.5f
(value < -5.75f)
-1.0f
13
(value < -0.25f)
-0.5f
14
(value 0.25f)
0.0f
(value 0.75f)
0.5f
(value < 1.25f)
1.0f
17
(value < 1.75f)
1.5f
18
(value < 2.5f)
2.0f
(value 3.5f)
3.0f
20
(value < 5.0f)
4.0f
6.0f;
21
22
output [idx] = closest;
23
25
26
void quantize (at: :ensor input, at::Te output, int size)
27
const float* input_data = aaput.ddta_ptrrflostt ();
28
float: output_data output wttpttddaaaaaptttaa...... ();
29
30
const int threadsPerBlock 256;
31
const int blocks = (size + threadsPerBlock - 1) / threadsPerBlock;
32
eudaStream_t stream at: gerre
quantize_ kereekkeeeelkccokk.... thrcadsPerBlock, 0, stream>>>> (input_data, output_data, size)
35
B. Theoretical Analysis on Speed Performance and Overhead
To assess the theoretical performance benefit of FP4 acceleration in our mixed-precision training framework, we analyze the theoretical speedup at the level of individual Transformer layer components. Specifically, we compute the floating-poits operations (FLOPs) required for each subcomponent in both FP32 and FP4 precision. Let the hidden size be h, batch size b, and sequence length The FLOP breakdown per Transformer layer is summarized in Table 5.
Given that the backward pass typically incurs approximately twice the FLOPs of the forward pass, the total computational cost for both forward and wark passes is three times that of the forward pass alone. Thus, the theoretical speedup factor from FP4 acceleration (excluding any overhead) can be expressed as:
3 (24bsh² 5bs + 36bsh)
24h 5s 36
3 x (6bsh? + 5h + 36bsh)
6h + 5s 36
For a representative case of a 7B model with hidden size 4096 and sequence length s = 2048, the resulting thcoretical
24 x 4096 + 5 x 2048 + 36
speedup is approximately 6 x 4096 + 5 x 2048 36 = 3.12.
In addition to the theoretical speedup from FP4 compute, we also account for two primary sources of computational overhead introduced by our proposed methodologies: Differentiable Gradient Estimator (DGE) and Outlier Clamp and Compensation (OCC).
14

---

Optimizing Large Language Model Training Using FP4 Quantization
Table 5. FI.OP breakdown of standard Transformer layer under FP4 acceleration.
Component
Subcomponent
FLOPs (FP32)
FLOPs (FP4)
Speedup Factor
Input LayerNorm
4bsh
4bsh
1x
Query, Key, Value Projections
6bsh?
1.5bsh?
4x
Atlention Scores Computation
4bs2%
4bs
1x
Multi-Head Attention
Softmax Computation
b
6s%A
1x
Oulput Projection
2bsh?
0.5bs/2
Post-Attention LayerNorm
4bsh
4bsh
1x
Up Projection
Sbah
2bsh
4x
Feed-Forward Network (FFN)
GeLU Activation
28hsh
28bsh
1x
Down Projection
8bsh?
2bsh
Total
24bsh² + -bss + 366sh
66sh2 2 55s 2
365s/
DGE Overhead. DGE introduces an additional nonlinear function in the backward pass of GEMM operations used for weight updates. As shown in Equation (8), this adds approximately 8 FLOPs per input element. Accumulating over all relevant GEMM operations-inc attention query/key/value projections, attention output projection, and MI.P up and down projections-the total FLOP overhead per iteration is:
8 x (3bsh + bsh + 4bsh + 4bsh) = 96bsh
This overhead occurs only once per forward-backward iteration.
OCC Overhead. OCC introduces extra FP8 sparse matrix multiplications during outlier compensation. Given an activation outlier matrix sparsity of 2(1 a:), these sparse GEMMs are applied to each of the four primary GEMM computations in Transformer block, resulting in additional 2(1 a) x (12bsh²) FLOPs per iteration. In our setup, we choose 0.99 to maintain high sparsity in the AY matrix. While the computational FLOP overhead remains small due to high sparsity, hardware inefficiencies in sparse GEMMs make a high value of a essential for runtime performance.
Adjusted Speedup Estimate. Taking both DGE and OCC overheads into account, the revised theoretical speedup becomes:
3 x (24bsh² h 36bsh)
24h 58 36
x (6bsh² + 5hs2h + 36bsh + 2(1 - (1)bshh²) + 96bsh 6h - 24(1 - az)h + 5s + 68
24 x 4096 + 5 x 2048 + 36
Substituting h = 4096, 8 = 2048, and a= = 0.99, we obtain
2.95
6 x 4096 + 24 x 0.01 x 4096 + 5 2048 + 68
Overhead Impact. The DGE overhead accounts for 32/(6/ + 5s + 36) = 0.1% and the OCC overhead contributes: 24(1 a)h/(6h- + 5s + 36) = 5.6% of the total computation. These result in a modest reduction of the ideal FP4 speedup from 3.12 to 2.95, which we consider an acceptable trade-off between computational efficiency and model accuracy.
C. Supplementary Proof for Differentiable Quantization Estimator
C.1. The Derivation of Differentiable Quantization Function
In Section 3.1, we introduce differentiable function to simulate the quantization funciton in Equation (7). Here we describe in detail how this formula is derived step by step.
To construct a smooth and differentiable approximation to the quantization function, we begin with the power function fz) = xº, where 0 < a < 1. This function is particularly suitable due to its saturating behavior: it approaches 1 as r + 1, and rapidly decays toward as - These properties align well with the behavior of the right half of a typical
15

---

Optimizing Large Language Model Training Using FP4 Quantization
quantization function. To extend the function to the entire real line while maintaining central symmetry around the origin, first symmetrize by applying the sign function. This gives:
f(x) sign(x) |
(10)
This symmetrized function now resembles "soft step" centered at the origin, increasing smoothly from -1 +1. We first scale the x-axis adjust the quantization interval from [-1, to 8/22, 6/2], which need to scale by 6/2, resulting in:
2x f(x) = sign(
2x
)- 5 )
(11)
To shift the center of this function from the origin the midpoint of the quantization interval [0, 6], we perform a horizontal (x-axis) translation. Specifically, we translate the input by 8/2, which centers the function at 6/2. This gives:
f(z) sign
2(x - 2)
2(x 2)
sign
2x 8
1)
2x
a
(12)
5
This modified function now transitions smoothly from negative to positive around the midpoint of the quantization interval, rather than around zero. Similarly, to ensure the output(y) range of the function maps to the desired quantization interval [0, 6]. we apply vertical (y-axis) translation and scaling. Specifically, we shift the function upward by 1 and then scale it by 5/2, resulting in:
f(x) = 8. (1+ sign 2
(2 2xx 8
15-1 -2-1
2x (
1))
) 5
(13)
This transformation maps the function output to the interval [0, 0]. with steep but smooth transition occurring around 8/2.
Finally, to make the steepness of the approximation more intuitively controllable, we reparameterize the exponent as the reciprocal of a positive constant & > i.e., a = 1/k. Larger values of & lead to steeper transitions, making the function better approximate a hard quantization step, while smaller values of result in smoother transitions.
C.2. Proof of DGE with Vector-wise Scaling Factors
Here we present the complementary proof procedure for the Diffcrentiable Gradient Estimator (DGE) method under actual quantization with rector-wise scaling factors. In the GeMM operation Y AW, where A is the activation tensor with dimensions (s c. sequence length input channels) and W the weight tensor with dimensions (c; c,, input channels x output channels), quantization is applied along distinct dimensions adhere to the mathematical logic of matrix multiplication. For the weight tensor with dimensions (c: X co), channel-wist quantization is performed follows:
Wasted W of Q(W (Wccaled
(14)
Wescaled q
(15)
W Wiscaled
(16)
sf
Here, af is the scaling factor, and @ represents the element-wise (Hadamard) product. In tensor- quantization, sf is scalar. For channel-wise quantization, sf is vector with dimensions (1 x C.). In this case, the ( operation involves broadcasting sf to each column of the matrix W (c, x co), followed by element-wise multiplication.
For Equation (16), is crucial to note that multiplying by 1/sf ensures mathematical correctness. Practically, however, this step performed after the GeMM kernel execution. In other words, the quantized weight tensor provided to the GeMM
16

---

Optimizing Large Language Model Training Using FP4 Quantization
kernel is the scaled quantized weight tensor W/scaled from Equation (15). Nevertheless, for mathematical analysis, the quantized weight tensor Wg must be re-scaled.
In the backward computation, the loss gradient with respect to W is derived from the forward operation Y AW q According to the matrix multiplication rules for differentiation, the gradient @L/aW, is computed using the activation gradient / from the subsequent layer.
Fwd: Y AW, Bwd: dL
A² aL
(17)
aw,
0Y
By applying the chain rule and referring to Equations (14) to (16), the relationship between 0L/oW, and the actual weight gradient 0L/OW is established. According to Equation (16), the gradient @L/0W can be expressed in terms of @L/aW, using the scaling factor sf:
aL. OW scaled
aL ow, sf
(18)
Subsequently, the differentiable gradient estimator correction term ('(x) is applied compute aL/0W scaled.
aL
al WWscaled
O Q'(Wsaated
(19)
aw scateed
Where (xxx is the differentiable gradient estimator correction item introduced in Equation (8). Finally, the relationship between 0T./0Wwated and 0L/0W is derived by incorporating sf:
ÖL
Off
(20)
OW
aw scaledd
By combining all these steps, the formula for calculating the true weight gradient aL/aW is obtained:
ØL
åL
@'((scalee)
② @sf
(21)
0W
OW, ÖL
sf
G ('(Wscated
(22)
aw,
Importantly, the scaling and un-scaling steps cancel each other due to the element-wiss nature of the operations, resulting in simplified expression. This final formula matches the previously demonstrated Equation (6) in the main body of the paper, with the only difference being that the variables within the DGE correction term must account for scaled weights. No changes are required for the Q and Q' functions.
C.3. Mathematical Soundness of DGE Clipping
As introduced in Section 1. the differentiable quantization scheme includes DGE correction term whose derivative can become unbounded near the midpoint of the quantization interval. Specifically, the derivative f'(x) is clipped to maximum value of 3.0 in practice to prevent instability during training. In this appendix. we provide a detailed mathematical justification for this clipping operation and explain how smoothing technique can be used ensure continuity and numerical stability.
The derivative of the DGE approximation is given by:
--1
1 f'(x)
2r
(23)
k
8
17

---

Optimizing Large Language Model Training Using FP4 Quantization
also stated in Equation (8). This expression arises from differentiating the power-based approximation of the quantization function centered at 8,2, where 8 is the quantization range, and kt > 1 is hyperparameter that controls the steepness of the transition.
The critical issue lies in the exponent - 1, which is negative for all &o > 1. Negative exponents in the form |a|* with a < are known to diverge as x - 0, due to the reciprocal relationship. For instance, when = 3, the exponent becomes -3. leadinn to:
2x
1
(z)
(24)
(1/ - 1|2/3)
3
which becomes singular 6/2. In this limit, the term - O, causing the denominator to vanish and (x) oc. This unbounded growth can lead to gradient explosions during optimization, especially when many input values are near 5/2.
To eliminate the singularity at = 8/2, we redefine the absolute value function in a differentiable and bounded manner. Instead of using the sharp absolute value |z]]. we adopt a smooth approximation defined as:
| R
x2
(25)
where is small positive constant. Substituting this into the derivative expression yields a smoothed version:
8 I'amooth (I) = 1 k. (v(2 2x. -1 2 2
(26)
This formulation ensures that the denominator never reaches zero, as /0++2> < 0, and therefore famoon (2) remains bounded for all x E RR In effect, the singularity is replaced with a smooth transition that asymptotically approximates the original function as e + 0, while maintaining differentiability and numerical stability during training.
We observe from the smoothed derivative expression presented in Equation (26) that the DGE correction term becomes bounded due to the regularization introduced by the epsilon term. Specifically:
1
2r
2
lim I'mooo x--8/2
lim 2-16/2
V
-1
2
(27)
1 I (Vo+&))**
t-1 1 c
(28)
In practice, the DGE correction item is clipped with predefined value, which is proved in Equation (28) to be mathematically equivalent since k and are all constents. This approach is aligned with the simplicity principle in algorithm design: it avoids the need to modify the functional form while still achieving equivalent mathematical behavior around the singular point. As demonstrated above, both the smoothed and clipped versions share the key property of bounding the gradient magnitude in the vicinity of æ = 6/2, thereby ensuring stability during backpropagation.
D. Analyzing Quantization Difficulty Through Tensor Distribution
Section 3 highlights the necessity of quantizing both weight and activation tensors to fully leverage the FP4 tensor core. It also points out that activation tensors are significantly more challenging to quantize compared to weight tensors. To further support this observation, we provide the actual distributions of weight and activation tensors during model training.
Figures to 10 illustrate the distribution weight tensors, while Figures 11 to 13 show the distribution of activation tensors. These results are derived from training the LLaMA 1.3B model over 30,000 iterations. The y-axis set to logarithmic
18

---

Optimizing Large Language Model Training Using FP4 Quantization
lyyns2 adE akrrissdees
laye 2 selt stantiendaac
4adi sacsies dims
kayca Rodi. svenioeaienn
10
Mikay 5000
10'
Mouer 6 0-0330 Sac
16
More: 6-000 8.0170
Mon: 2.0000
0.1122
10'
Moie: 6-3030
Mare reco
10
16
15
16
10
16
90
10
1111
0.5
-6.25
-0.25
-0.2
Bryens 12 www.stiie.cooo Seese
tapors SAR, MANCERAC Reise
Jnyers UA wAT www.ereeeeeeeeeee. ene
Sapers 15.AAT www.eresseeeeoooooooo
Ioyers200 20 SilT, MANIEIIO desse
10'
61226
19'
9163 Sal. 0.8240
Sad. 6-120
0.803 Skl 0.8244
10'
Shat: 6-0217
(11 0.0000
1660
16
10
10'
12"
16'
10
10'
13
10
10
Valore
Figure& 8. Visualization of the weight tensors in the dense projection layers of the self- module.
byyoou mippepp peppppptttion
2und up p.poeecctoon
Eagore enly up projection
bycs capp-ppp.ee. spppeepcciooon
Jyyoe R.mty wp. projection
minlp up projection
10
Mear: 8393
16
Maa:: 10000
Mar: 4.0911
Wiass d.00)
10
167
10
10
10
32
167
10
10
10
10
30
30l
-025
0.00
0.25
-42
0.5
0L0
0.1
-0.2
-0.1
00
0.1
-0.2
0.00
625
Eapore zmlo up. spprrojactie
Lapen 14 calp ey prwjjccttion
bryam Montppppprrroeee projectioon
Syyers (Korly prpjaeete
Supers:20 mijp 1p perppeactionnn
Tayaer oll up, projectiion
Miurr 44.001
Muse: 4.5000
220000
Miea: 5:0000
Moxx 2.000
10
16'
12"
10'
16'
10'
167
10'
101
16
10
10
16
-0.2
0.5
0.2
0.2
-022
0.0
-0.1
0.0
0.1
-4.25
0.20
0.25
-0.25
0.25
Veluoe
Figure 9. Visualization of the weight tensons in the up-projection linear layers of the MLP module.
layan, Buzty town, layers 2.2t down, AyypnAiippdoww projattos
boyenui
layorc iats
layex dow.
22000
skl 8.8255
Stc. 65115
Stll 8.00%
Sal 0:8157
Sut 0L0633
Sil 28165
10
107
10
15'
35
10
167
10
142
15
12'
Inquory
Mosos.i-annssdww... projcctionn
projecttioon (Sud: 9.0283
HRusd deever prngocttio
layes Javea pogesttttiiio
1ayos22 praeiin
8215
5AJ-0 0.0044
Sal. 0:8279
Sub 0,0250
28272
16
10
10
10
16'
10'
tn²
10
16
10
10
12'
0.5
8.5
0.6
24
6.00
-0.5
6.5
Figure 10. Visualization of the weight tensors in the down-projection linear layers of the MLP module.
19

---

Optimizing Large Language Model Training Using FP4 Quantization
1syos.n wit Litereee uore peerrie
Iowrec 2o Jevamis oon, pmoriis
layaad oilf CArT, pereece
Seylec SIT, www.Er...O....... Jonerio
byyer wilf www.eteeeeeooccooo can, pervice
16'
Mese 002025 Sha 0021
Moes: 440013 542 0.3376
-48664 S1A: 3.1377
Ma 3,000 542-0
8 Sid 0,213
Mare BUK21 037446
16'
10
10
16
101
0.0
layera 22dd atuntonnccee afrectin
koani anciticc.oor station
Jayn 16rdt antiction cae atcmtion
Sayem adf afcctie.cco RECTECO
Mousc 009026
News 0.8556 0.P915
Moe re0e
Mare: 00003
Mose D00IA
10
11
10
16
10
10
16?
-2.5
6/0
2.5
Valuos
Figure 1.. Visualization the activation tensors from the core attention output.
10
10"
Sat 4004
Wii #764
Su 64776
Sie BARM
Aot a 9464
16"
10
td
10
1d
16"
16'
10
18
layensees Soot 1.2006
1apers22.per:
Sal- 9980
167
9vt. 1.9990
54d 3.1716
(9ut 1 12245
16'
Sati: 1.2366
16
10
10
10
16
10?
16
Valux
Figure 12. Visualization of the activation tensors from the post-attention layer normalization output.
byyenu milp down, projectio
Layes.3 www..oow. prjecttee
bypcs4msppooe jrojectss
bycsh my:- doord projectoo
Dayors: 1. Eiat downe projectie
16
Mars 460109
32"
Mare 4:2005
Mass -.0001
Maps. - 2012
10'
Mop 10.03
Maa -3.005
Suc EV9AG
12at 6,1411
5/836
2683
03360
16
33a
102
10
10'
10
10
10'
10
10'
16
10'
12:mmppdiaer dir.pppddoo....... penjecttions
loyore 14-trp
Tayore 16.mmyy dewne
iiit dennm projectioon
layore 35 anlp dorem, projection
1oyor:33 exto
30"
10"
5.0005 Sid. 0,3066
20004
Suc 03102
146 8-4355
SkJ 8.120
Sib 07013
16
102
to²
10?
tol
16
10'
10
.II I.
.11.]
23
Veloes
Figure 13. Visualization of the activation tensors from the MLP down-projection layer output.
20

---

Optimizing Large Language Model Training Using FP4 Quantization
scale enhance visualization. From these figures, it is evident that weight tensors generally exhibit a smaller dynamic range, while activation tensors have significantly larger dynamic range, making them more challenging quantize.
Regarding distribution characteristics, weight tensors typically follow a normal distribution, with certain tensors exhibiting small outliers. In contrast, activation tensors vary widely in their distributions. For example, core attention outputs often follow a regular distribution with minimal outliers. However, many activation tensors, such as layer-norm outputs and transformer layer outputs, display irregular distributions with numerous outliers, making them particularly difficult to quantize.
Notably, the outliers in activation tensors during LLM training tend to appear in specific channels. This observation is further validated through heatmap analysis in Figure 14. The result is obtained through the activation function (GeLU) output from the first transformer layer.
These analyses underscore the critical importance effectively addressing activation tensors during quantization, especially their outliers. Future research could gain valuable insights by exploring the complex distributions and outlier behavior of activation tensor values.
0
513
4
1026
1539
Sequence Length Batch Size
2052
2565
3078
3591
4104
2
4617
5130
5643
6156
1
6669
7182
7695
0
241
482
723
964
1205
1446
1687
1928
2169
2410
2651
2892
3133
3374
3615
3856
Hidden Dimension
Figure 14. Heatmap visualization of the activation function (GeLU) output from the first transformer layer the 30,000 training iteration
LLaMA 1.3B model. The vertical light lines the heatmap correspond specific channel dimensions the activation tensor. highlighting the channel-wise distribution of outliers.
21


Reviews:
[{"review_id": "9xaFWcl24V", "reviewer": "Reviewer_iCFi", "summary": "The paper presents an innovative framework for training large language models (LLMs) using FP4 quantization. The key contributions include:\n1.Differentiable Quantization Estimator: This method improves gradient updates during FP4 computations by analyzing the impact of quantization on both forward and backward passes of neural networks, deriving a function with correction terms for accurate gradient estimation.\n2.Outlier Clamping and Compensation Strategy: This addresses the issue of outlier values in activation tensors during LLM training, which can cause significant quantization errors. The strategy involves clamping these outliers to a predefined threshold and compensating for the introduced error using a sparse outlier matrix.\nDetailed experiments are presented in the paper to validate the effectiveness of the framework:\n1.Focused on 4-bit quantization for GeMM operations\n2.Quantization is applied along distinct dimensions for activation and weight tensors, aligning with matrix multiplication logic.\n3.The framework is tested on LLMs with up to 13B parameters trained on 100B tokens.\n4.Results show that the FP4 framework achieves accuracy comparable to BF16 and FP8 with minimal degradation.\n\n## update after rebuttal\nLow-precision training has emerged as a clear trend in reducing computational costs for machine learning. After carefully considering the rebuttals from the authors and the feedback from other reviewers, I believe this work makes a significant contribution to both the academic community and the industry. Its technical insights and practical implications are particularly valuable in advancing the field of efficient training methodologies. Therefore, I will maintain my original rating of accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and insightful evaluation of our work, as well as for recognizing the theoretical and practical contributions of our FP4 training framework. Below, we address the questions and suggestions:\n\n**Q1 (Theroretical Claims): Clarification on OCC’s $\\alpha$ value inconsistency (Sec. 3.2 vs. Sec. 4.3)**\n\nA1: Thank you very much for pointing out this issue. This is indeed a typo in Section 4.3—the correct statement should be that **a lower $\\alpha$ leads to better model accuracy**. Theoretically, a lower $\\alpha$ results in reduced quantization loss, as stated in Section 3.2 and shown in Table 1. Experimentally, Figure 6(c) in Section 4.3 confirms that lower $\\alpha$ corresponds to a lower training loss curve. We will correct this in the final version. Again, we appreciate your keen observation!\n\n**Q2 (Experimental Designs Or Analyses): Details on speed evaluation / Theoretical analysis on the computation overhead or other cost**\n\nA2: Thank you for the suggestion. We recognize the importance of analyzing FP4’s theoretical speedup and computational overhead. The theoretical speedup of FP4 **(excluding DGE and OCC overhead)** is:\n\n$$\\frac{3*(24bsh^2+5bs^2h+36bsh)}{3*(6bsh^2+5bs^2h+36bsh)}=\\frac{24h+5s+36}{6h+5s+36}=3.12 \\quad (h=4096, s=2048)$$\n\nFor detailed analysis on the decomposition of computational components, we kindly refer the reviewer to the response to reviewer gQsM (A1) due to space constraints.\n\n**Regarding computational overhead:**\n\n- DGE overhead: DGE introduces an additional nonlinear function during GEMM backward for weight updates, adding ~8 FLOPs per input element (Eq. 8). Accumulating all GEMM operation, this will cause a total overhead of $8*(3bsh+bsh+4bsh+4bsh)=96bsh$ (Four additions in brackets are weight shapes of attn qkv proj, attn out proj, MLP up proj and MLP down proj). Note that this overhead occurs only once per forward-backward iteration.\n\n- OCC overhead: OCC incurs extra sparse matrix multiplication. FP8 sparse GEMM is used  with an activation sparsity of $2(1-\\alpha)$. These FP8 sparse GEMM are added into every GEMM computation, adding an extra $2*(1-\\alpha)(12bsh^2)$ FLOPs, where $12bsh^2$ comes from the cumulative sum of all GeMM calculations in the decomposition table that can be accelerated by FP4 (attn qkv proj, attn out proj, MLP up proj and MLP down proj), but in FP8 format with twice as many FLOPs as in FP4 $\\bigg(2*(1.5bsh^2+0.5bsh^2+2bsh^2+2bsh^2)\\bigg)$. Since we set $\\alpha=0.99$, meaning these GEMMs are highly sparse, the overhead remains small, though hardware inefficiencies in sparse matrix multiplication necessitate choosing a larger $\\alpha$ to ensure the high sparsity of the $\\Delta Y$ matrix.\n\nAccounting for these overheads, the adjusted theoretical speedup is:\n\n$$\\frac{3*(24bsh^2+5bs^2h+36bsh)}{3*(6bsh^2+5bs^2h+36bsh+2*(1-\\alpha)(12bsh^2))+96bsh}=\\frac{24h+5s+36}{6h+24*(1-\\alpha)h+5s+68}=2.95 \\quad (h=4096, s=2048, \\alpha=0.99)$$\n\nThe overhead from DGE and OCC acounts for $32/(6h+5s+36)=0.1\\\\%$ and $24(1-\\alpha)h/(6h+5s+36)=5.6\\\\%$ of the total computation, respectively, reducing the theoretical speedup ratio for FP4 compared to BF16 reduced from 3.12 to 2.95. We believe this is an acceptable trade-off between accuracy and efficiency since they can largely reduce quantization errors during training.\n\n**Q3 (Experimental Designs Or Analyses): Loss curve spikes in Fig. 5 (larger models):**\n\nA3: Thank you for the insightful observation. Larger models (e.g., 13B) exhibit more pronounced loss spikes compared to smaller models (e.g., 7B, 1.3B). While similar spikes occur in BF16, they are more frequent and severe under FP4 due to its limited representation range and the larger number of elements per FP4 quantization vector, increasing quantization error. Addressing this may require more aggressive accuracy compensation strategies, like decreasing the parameter $\\alpha$. Additionally, the issue may stem from the optimizer, as we currently use a FP8-based mix-precision optimizer. Larger models may demand higher precision for the optimizer.\n\nThank you again for your valuable comments. We appreciate your constructive feedback, which has helped us clarify key aspects of our methodology and presentation.", "claims_and_evidence": "The claims are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are well-aligned with the problem of optimizing LLM training using FP4 quantization. They address the key challenges and provide a comprehensive assessment of the framework's effectiveness.", "theoretical_claims": "I checked the derivation of DGE and OCC. Both of them are theoretically correct.\nHowever, there is a small issue of OCC. In Sec. 3.2, the author provides a study of the influence of CLAMP/COMP/QUANTILE (Table 1). It can be seen that lower alpha value yields less loss, because that more values are moved to high precision residuals. However, in Sec. 4.3 (Activation part), they claim higher alpha leads to better model accuracy. This is inconsistent with that in Sec. 3.2. It should be clarified.", "experimental_designs_or_analyses": "The experimental designs and analyses in the paper appear sound and well-considered, including Comparison to Baselines/ Model Sizes and Training Data/ Training Loss Curves/ Downstream Task Evaluation/ Ablation Studies/ Quantization Granularity Analysis.\nHowever, the authors do not give more details on speed evaluation. Though there is no hardware that support FP4 natively. A theoretical analysis on the computation overhead or other cost should be given.\nAnother issue is the spikes of the loss curve, Fig. 5. It seems more and larger spikes are observed for larger models. It's better to give the reasons, as it may indicate some potential risk scaling to larger models.", "relation_to_broader_scientific_literature": "The key contributions of the paper are closely related to the broader scientific literature in several specific ways:\n1. Advancement of Quantized Training Methods\n2. Differentiable Quantization Estimator\n3. Outlier Clamping and Compensation Strategy\n4. Mixed-Precision Training and Fine Grained Quantization\n5. Hardware Considerations", "essential_references_not_discussed": "All important literatures are cited in this paper.", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "See above", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HCVofjrJxR", "reviewer": "Reviewer_Le3f", "summary": "The paper introduces a framework for training LLMs using 4-bit floating-point (FP4) quantization to address the computational burdens of LLM training. It introduces a Differentiable Gradient Estimator (DGE) for weight updates and an Outlier Clamping and Compensation (OCC) strategy to manage activation outliers, mitigating quantization errors. The framework integrates mixed-precision training and vector-wise quantization. Experiments conducted on LLaMA2 models (up to 13B parameters,100B tokens) using FP8 tensor cores to emulate FP4 suggest performance comparable to BF16 and FP8 baselines in training loss and zero-shot tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive critique. We appreciate your recognition of the technical effort and alignment with hardware trends, as well as your insightful suggestions. Below, we address your concerns and questions:\n\n**W1: Hardware Dependency**\n\nA1: We acknowledge the lack of specific hardware as a limitation, as noted in Sec. 6. We provide a theoretical analysis of FP4 acceleration and demonstrate a **theoretical 3× speedup** (depending on model parameters). The detailed analysis is included in our response to reviewer gQsM (A1) due to space constraints. Additional analysis on DGE and OCC overhead, addressed in reviewer iCFi’s reply (A2), indicates the overhead remains acceptable while preserving accuracy. However, this is not speculative. As reviewer 1Hhi noted, while the absence of hardware testing affects empirical validation, it does not detract from our goal—to explore FP4 quantization for large-scale training in alignment with future hardware trends.\n\n**W2: Insufficient Scale (13B parameters, 100B tokens)**\n\nA2: We agree that scaling to trillion-parameter models is crucial. However, within academic compute constraints, our focus was to establish FP4’s feasibility. A **13B model trained on 100B tokens represents a reasonable scale in research**, even if it is smaller than industrial models (e.g., GPT-4, LLaMA3). Scaling stability is essential but is best explored in future industry-driven work.\n\n**W3: Clarification on DGE**\n\nA3: We apologize for not including the full mathematical derivation of Eq. 7 due to paper space constraints. We choose power function $f(x)=x^a$ (a<1) as the base differentiable estimation function because it saturates at 1 as x tends to 1, and quickly drops to 0 as x tends to 0, very much in line with the right half of the quantization function. To fully simulate the quantization function, we take absolute values of x and apply sign function for central symmetry: $f(x)=sign(x)\\cdot|x|^a$. We then translate the function on the x-axis and the y-axis to move to the quantization range of [0, $\\delta$]: $f(x)=\\delta(1+sign(x-\\frac{\\delta}{2})\\cdot|x-\\frac{\\delta}{2}|^a)$. Finally, we adjust the value of the power exponent to control the steepness and write a(a<1) as 1/k(k>1). We will clarify this in the final version to avoid possible confusion.\n\n\n**W4: Memory Usage and Training Speed**\n\nA4: We thank the reviewer for the suggestion. Below, we report real-time training memory usage and throughput:\n|Model Size|Precision|Memory Usage|Tokens per second|\n|-|-|-|-|\n|1.3B|BF16|51.03GB|470.3k (1.0x)|\n|1.3B|FP4|46.52GB **(-9%)**|395.1k (0.84x)|\n|7B|BF16|72.04GB|255.7k (1.0x)|\n|7B|FP4|52.75GB **(-27%)**|276.1k (1.08x)|\n|13B|BF16|70.28GB|118.4k (1.0x)|\n|13B|FP4|53.80GB **(-24%)**|126.9k (1.07x)|\n\nNote that our method is based on FP8-LM framework, where its speed up for three models are 1.03x, 1.24x and 1.27x (paper reported). In our implementation, **we need to do extra precision casting** for BF16 to FP4 (for simulation) and FP4 to FP8 (for FP8 GEMM computation), **leading to large overhead since native FP4 hardware is inaccessible**. In other words, evaluating speed performance during a real training process without dedicated hardware support can only serve as a point of reference.\n\n**W5: OCC’s Novelty and Computational Burden**\n\nA5: We thank the reviewer for pointing out that we did not state the OCC methodology very well in the paper. OCC’s novelty lies in its effective clamping strategies for characterizing numerical ranges and sparse compensation which avoids dense high-precision residuals. This method is suitable for online use and better matches the pre-training task. For the computation burden of $\\Delta Y$, please refer to reviwer iCFi's reply (A2) due to the charactor limit. We'll refine these statements in the final version.\n\n**Q6: Perplexity (PPL) Metrics**\n\nA6: We acknowledge PPL’s importance as evaluation metric. Due to the character limitation, we kindly refer to reviwer gQsM's reply (A2) for detailed PPL results.\n\n**Q7: Convergence with Increasing Tokens**\n\nA7: Our preliminary analyses of existing scaling trends suggest that extended token training would likely achieve stable convergence. While scaling analysis is crucial for understanding model convergence, resource limitations currently prevent comprehensive token scalability studies. We highlight this as a critical research direction and commit to open-sourcing our framework to support collaborative exploration of token scaling dynamics.\n\n**Q8: Open-Sourcing the Code**\n\nA8: We sincerely appreciate the reviewer’s interest in implementation details. We are fully committed to open-sourcing the code and will release it once it has been thoroughly organized to ensure clarity and usability.\n\nThank you again for your rigorous feedback. We hope these clarifications address your concerns.", "claims_and_evidence": "The core claim—that FP4 can train LLMs with minimal accuracy loss compared to BF16—is supported by training loss curves (e.g., 1.97 vs. 1.88 for 13B models) and zero-shot task accuracies (e.g., 54.95% vs. 54.44% for 13B). DGE and OCC are presented as solutions to FP4’s quantization challenges, with ablation studies showing their necessity (e.g., direct FP4 casting diverges). However, the evidence is undermined by the absence of native FP4 hardware testing, limiting efficiency claims to speculation.", "methods_and_evaluation_criteria": "The methodology employs mixed-precision training, quantizing GeMM operations to FP4 while using higher precision elsewhere. DGE approximates quantization differentiably, and OCC clamps outliers with sparse compensation. Evaluation includes training loss, quantization fidelity metrics (cosine similarity, MSE, SNR), and common zero-shot accuracy. Besides, the author need report the ppl metrics.", "theoretical_claims": "This paper presents that FP4’s 16-value representation suffices for LLM training via DGE and managing outliers with OCC. However, The introduction of Equation 7 appears somewhat abrupt, and its specific definition, as well as the rationale for its particular form, can be quite confusing. These claims lack comparisons to other low-bit formats (e.g., INT4). The theoretical novelty is modest, leaning heavily on prior quantization concepts.", "experimental_designs_or_analyses": "Experiments test LLaMA 2 models (1.3B, 7B, 13B) on 100B tokens from the DCLM dataset, with ablation studies isolating DGE and OCC effects. While the design is systematic, its scale is insufficient—13B parameters and 100B tokens pale against modern LLMs (e.g., Llama 3’s15T tokens).", "relation_to_broader_scientific_literature": "This work builds on FP8-LM , aiming to extend quantization to FP4 for training. This paper insufficiently engages with related FP4 works, like LLM-FP4.", "essential_references_not_discussed": "The paper discusses a relatively comprehensive review of related work.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Interesting topic: This paper explored FP4 for LLM training, aligning with the push for future hardware trends (e.g., Blackwell GPUs).\n2. Technical Effort: DGE are thoughtful attempts to address FP4’s quantization issues, supported by derivations (e.g., Equations 6-8) and ablations (Figure 6).\n3. The comparison is extensive, including FP8-LM and TE, demonstrating the effectiveness of this paper.\n\nWeaknesses:\n1. Hardware Dependency: The lack of native FP4 hardware testing renders efficiency claims speculative, a drawback for a paper emphasizing training cost reduction.\n2. Insufficient Scale: Experiments at 13B parameters and 100B tokens seems to outdate compared to current LLM scales (e.g., GPT-4’s 1T parameters/ more tokens).\n3. Confused DGE: The direct presentation of Equation 7 is overly abrupt and somewhat confusing, lacking appropriate reasoning and motivation.\n4. I understand that current hardware do not yet support FP4 tensor core; however, the authors should at least present the memory usage or training acceleration of existing methods. The absence of such data makes it difficult to accept their motivation for reducing training costs.\n5. The novelty of OCC is straightforward and seems to lack novelty. Besides, author should present the computation burden of delta_Y.", "other_comments_or_suggestions": "See above strengths and weaknesses.", "questions_for_authors": "2. Does increasing the number of training tokens further still result in consistent convergence effects?\n1.The implementation details will be useful. Could the authors consider open-sourcing the code for testing?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dydm1lMDEP", "reviewer": "Reviewer_1Hhi", "summary": "The paper tackles the challenging problem of FP4 training for LLMs. They propose two innovations: a differentiable quantization estimator (DGE) for precise weight gradient updates and an Outlier Clamping and Compensation (OCC) strategy for activations. The authors also provide extensive experiments on model scales (up to 13B parameters) show that the proposed FP4 method achieves comparable accuracy to traditional FP8 and BF16 baselines.\n\n## Update after rebuttal\nI maintain my original score. I am generally satisfied with the authors’ response.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful evaluation of our work and for recognizing the significance of our contributions. We appreciate your positive feedback on the effectiveness of DGE and OCC, as well as the thoroughness of our experiments. Additionally, we are grateful for your understanding of the hardware constraints that necessitated our simulation-based approach, which, while unavoidable, slightly limits empirical hardware-level assessments.\n\n**Q1: OCC’s Computational Overhead During Actual Training Scenarios**\n\nA1: The computational overhead of OCC primarily arises from additional sparse matrix multiplications. Specifically, the input activation tensor Y is decomposed as: $Y=Y_c+\\Delta Y$, where $\\Delta Y$ contains outlier values processed in higher-precision FP8 GeMM. Since $Y_c=\\text{clamp}(Y, max=\\alpha,min=1-\\alpha)$ and $\\alpha$ is very close to 1, the $\\Delta Y$ matrix remains highly sparse, with a sparsity ratio of $2*(1-\\alpha)$.\n\nA detailed theoretical analysis of the computational cost of $\\Delta Y$ GeMM is provided in our response to reviewer iCFi (A2). We kindly refer you to those details if you're interested. The results show that under the training configuration of **hidden_size=4096, sequence length=2048 and $\\alpha=0.99$** (as used inour 7B model training), the theoretical speedup of FP4 training decrease slightly from 3.12 to 2.95 due to OCC. Considering the large accuracy benefits of the OCC method, we think this performance drop is acceptable. Given OCC’s significant accuracy benefits, we consider this tradeoff acceptable. However, careful tuning of $\\alpha$ is crucial to maintain high sparsity, as current hardware struggles with efficient sparse matrix multiplications at lower sparsity levels.\n\nHowever, the situation in real training scenarios is a little different. In our current training setup, FP8 GeMM is used to simulate FP4 behavior, and $\\Delta Y$ is naturally handled in FP8 format. Consequently, both $Y_c$ and $\\Delta Y$ are computed within FP8 GeMM, allowing their results to be directly summed without requiring an additional GeMM operation, thereby eliminating OCC-related overhead in this simulation setup.\n\nAt present, the primary computational overhead stems from:\n\n1. DGE computations (negligible).\n2. Precision casting between BF16, FP4, and FP8 (significant).\n\nHowever, if native FP4 hardware becomes available and is used for $Y_c$ computations, then OCC’s computational cost must be carefully reassessed.\n\nOnce again, we appreciate your constructive feedback and your recognition of this work’s broader relevance in advancing resource-efficient LLM training.", "claims_and_evidence": "The paper effectively supports its claims, showing through extensive experimentation that the FP4 quantization framework performs closely to higher precision baselines in training LLMs. The proposed differentiable gradient estimator (DGE) and outlier clamping and compensation (OCC) methods are convincingly validated by clear ablation studies (e.g., Figures 3, 4, and Table 1).", "methods_and_evaluation_criteria": "The proposed evaluation criteria, such as training loss and zero-shot task performance, are appropriate for quantization research. The authors selected established benchmarks like PiQA, HellaSwag, ObQA, and Lambada, which are standard for assessing language models.", "theoretical_claims": "There are no theoretical claims/proofs.", "experimental_designs_or_analyses": "Experimental designs appear sound, but direct access to FP4 tensor cores is simulated due to hardware constraints. This limitation slightly impacts the empirical assessment of true hardware-level gains but cannot be attributed to the authors.", "relation_to_broader_scientific_literature": "The problem is very relevant, especially with the recent interest in training LLMs in a resource-efficient manner.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n\n- The proposed Differentiable Gradient Estimator (DGE) method is particularly innovative, providing a differentiable approximation that improves gradient accuracy significantly over the Straight-Through Estimator (STE).\n- The Outlier Clamping and Compensation strategy is well-motivated and effective, especially given the difficulty of handling outliers in FP4 quantization.\n- Clear presentation with illustrative figures and well-structured experimental analyses.\n\nAuthors have also addressed other limitations of their work in the paper such as using FP8 tensor cores to emulate FP4 computations due to hardware limitations", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Could you elaborate on the computational overhead introduced by the OCC method during actual training scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4s14vLLdCB", "reviewer": "Reviewer_gQsM", "summary": "This paper presents a FP4 training framework for large language models, which could potentially reduce the costs of LLM training. This work addresses the accuracy challenges caused by FP4 quantization with two key innovations: 1) Differentiable Gradient Estimator (DGE) for accurate weight updates and 2) Outlier Clamping and Compensation to tackle activation outliers. As a result, this framework maintains accuracy comparable to BF16 and FP8 while effectively scaling to models with up to 13B parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thorough and constructive feedback on our work. We appreciate your recognition of the framework’s potential and your insightful questions, which have helped clarify key aspects of our methodology and claims. Below, we address each point in detail.\n\n**W1: Speedup Estimation and Kernel-Level Latency Breakdown**\n\nA1: We completely agree with the importance of evaluating theoretical speedup. Since we employ a mixed-precision training framework, we analyze the impact of FP4 acceleration on each computational component of a standard Transformer layer. Given a hidden size *h*, batch size *b*, and sequence length *s*, the FLOP breakdown is as follows:\n\n|Component|Subcomponent|FLOPs(FP32)|FLOPs(FP4)|Speedup Factor|\n|-|-|-|-|-|\n|Input Layernorm| |$4bsh$|$4bsh$|1x|\n|Multi-Head Attention|Query,Key,Value Projections|$6bsh^2$|$1.5bsh^2$|4x|\n| |Attention Scores Computation|$4bs^2h$|$4bs^2h$|1x|\n| |Softmax Computation|$bs^2h$|$bs^2h$|1x|\n| |Output Projection|$2bsh^2$|$0.5bsh^2$|4x|\n|Post-Attention Layernorm| |$4bsh$|$4bsh$|1x|\n|Feed-Forward Network(FFN)|Up Projection|$8bsh^2$|$2bsh^2$|4x|\n| |GeLU Activation|$28bsh$|$28bsh$|1x|\n| |Down Projection|$8bsh^2$|$2bsh^2$|4x|\n|Sum| |$24bsh^2+5bs^2h+36bsh$|$6bsh^2+5bs^2h+36bsh$| |\n\nSince the backward pass requires approximately twice the FLOPs of the forward pass, the theoretical FP4 speedup (excluding DGE and OCC overhead) for a 7B model (h=4096, s=2048) is:\n\n$$\\frac{3*(24bsh^2+5bs^2h+36bsh)}{3*(6bsh^2+5bs^2h+36bsh)}=\\frac{24h+5s+36}{6h+5s+36}=3.12$$\n\nThe reviewer also point out the importance of kernel-level latency breakdown. Currently, we use FP8 GEMM kernels for all matrix multiplications in the above table, leading to a theoretical speedup of:\n\n$$\\frac{3*(24bsh^2+5bs^2h+36bsh)}{3*(12bsh^2+5bs^2h+36bsh)}=\\frac{24h+5s+36}{12h+5s+36}=1.83$$\n\nHowever, in practice, the framework of [FP8-LM](https://arxiv.org/pdf/2310.18313)  reports only a 1.28× actual speedup for h=4096 (7B model). Our implementation incurs additional precision casting overhead due to extra precision casting from BF16 to FP4 (for simulation) and FP4 to FP8 (for FP8 GEMM computation). These conversions are unnecessary with specialized FP4 hardware, where quantization could be fused into the GEMM kernel. So although we write the FP4 quantization kernel, it will still cause heavy overhead in the training process, reduceing speedup from 1.28× to approximately 1.08× in real training.\n\n**W2: Perplexity (PPL) Evaluation**\n\nA2: We agree that PPL is a sensitive metric for language model training. Below, we present the evaluation results:\n|Model|Precision|Lambada_openai|Lambada_standard|Pile10k|Wikitext|Average|  \n|-|-|-|-|-|-|-|  \n|1.3B|FP4|14.98|25.10|82.77|26.65|**37.38**|  \n|1.3B|BF16|15.33|23.07|82.52|26.51|**36.86**|  \n|7B|FP4|14.34|23.33|77.72|24.86|**35.06**|  \n|7B|BF16|14.29|24.42|78.42|25.36|**35.62**|  \n|13B|FP4|12.42|22.45|75.06|24.81|**33.69**|  \n|13B|BF16|13.67|21.62|75.84|24.83|**33.99**|  \n\nThe results demonstrate that FP4 models achieve comparable or even slightly lower PPL than BF16 models. As expected, larger models achieve lower perplexity under the same training token budget.\n\n**Q3: For Equation (8), should there be a $\\delta$ in the numerator of the constant term?**\n\nA3: Yes, it was a typo error and we're really sorry! We sincerely appreciate your attention to detail and will correct this in the final version.\n\n**Q4: Memory Consumption vs. Impact Statement Claim**\n\nA4: We appreciate the reviewer’s feedback on this potentially misleading claim. While our experiments confirm reduced GPU memory usage compared to BF16, this reduction primarily results from:\n\n1. FP8 optimizer states in Mixed-precision optimizers.\n2. FP8 activation storage (from FP8-LM framework).\n\nThe current FP4 online quantization strategy does not reduce GPU memory. Further reductions would require FP4 optimizer states and activation storage, which we leave for future work. To prevent confusion, we will remove the memory claim from the Impact Statement in the final version.\n\n**Q5: Quantization Granularity**\n\nA5: Yes, each token (activation) or each channel (weight) has a single scaling factor. Currently the provided quantization kernel in Appendix A only covers the quantization of the scaled tensor, and per-channel/per-token max value computation is not yet fused into the kernel.\n\nWe sincerely appreciate the reviewer’s insightful feedback, which has strengthened both our analysis and the clarity of our claims. We will incorporate all necessary revisions into the final manuscript.", "claims_and_evidence": "Please refer to **“Methods And Evaluation Criteria”**.", "methods_and_evaluation_criteria": "The accuracy evaluation and ablations appear to be plausible. It covers models with sizes from 1.3B to 13B and provides both training loss curves and downstream evaluations. However, to make the claim stronger, it would be better to have:\n1) Perplexity (PPL) evaluation of the models trained with the proposed method and BF16 baseline, as PPL is more sensitive and commonly used than downstream evaluations.\n2) Efficiency evaluation for this paper is lacking. Although FP4 tensor cores are not available, some analysis/estimations regarding the speedups would be helpful.", "theoretical_claims": "I have checked the formulations for DGE in this paper. The overall derivation looks good to me. One question on Equation (8): Should there be a $\\delta$ in the numerator of the constant term?", "experimental_designs_or_analyses": "Please refer to **“Methods And Evaluation Criteria”**.", "relation_to_broader_scientific_literature": "The paper follows the framework of [FP8-LM](https://arxiv.org/pdf/2310.18313), targeting the accuracy issues regarding FP4 LLM training.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "[Strength]\n1. The paper targets an important topic: efficient training of LLMs. Despite room for further improvements, the evaluations and ablations appear to be plausible and comprehensive.\n2. The DGE design in this paper seems interesting and effective for FP4 training.\n\n[Weakness]\n1. It would be better if the authors could estimate the speedups in the paper. While the FP4 tensor cores may not be available, the estimation of theoretical speedups can be obtained with kernel-level latency breakdown. For example, what are the latency proportions of FP8 GEMM kernels (for FP4 simulation) and quantization kernels in the training process? \n2. Perplexity (PPL) evaluation of the models trained with the proposed method and BF16 baseline are not provided. As PPL is more sensitive and commonly used than downstream evaluations, it would be helpful to include PPL evaluation.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "1. According to Figure 2, the input activations and weights are quantized from BF16 to FP4 on-the-fly. As a result, the proposed method in this paper should not be able to reduce memory consumption during the training process. Is this contradictory with the claim in the Impact Statement that “By significantly lowering computational and memory demands, FP4-based methods can …” ?\n2. The authors mention that activation is quantized token-wise and weight is quantized channel-wise. I would like to confirm that does it mean that each token in activation (each channel in the weight) has only one scaling factor? If so, is the per-channel/per-token max value calculation process included in the quantization kernel?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques to enable the training of large language models (LLMs) using FP4 weights and activations. The methods are applied to train LLMs with up to 13B parameters and on datasets of up to 100B tokens, demonstrating training dynamics comparable to BF16 baselines.

The paper is very clearly written and addresses an important topic. While the theoretical contributions and technical innovations are not particularly novel--the use of approximate gradients for step functions, quantile-based clamping, and high-precision sparse outlier matrices have been explored in prior quantization work--the strength of the paper lies in its experimental results. The framework demonstrates FP4 training with minimal degradation at a reasonably large scale, which is practically valuable and impactful. In addition, during rebuttal, the authors provide further clarification and additional analysis regarding computational overhead. 

All things considered, the paper is recommended acceptance.
Decision: Accept (poster)