Scaling Laws for Floating–Point Quantization Training
Authors: Xingwu Sun
Shuaipeng Li
Ruobing Xie
Weidong Han
Kan Wu
Zhen Yang
Yixing Li
An Wang
SHUAI LI
Jinbao Xue
Yu Cheng
Yangyu Tao
Zhanhui Kang
Cheng-zhong Xu
Di Wang
Jie Jiang
Venue: ICML 2025 poster (2025)
Keywords: Floating-point quantization
Scaling law
Training performance
Primary area: deep_learning->large_language_models
TL;DR: This paper examines floating-point quantization in LLM training, proposes a unified scaling law, and offers key recommendations for improving model performance and cost-efficiency.

Abstract: Low-precision training is considered an effective strategy for reducing both training and downstream inference costs. Previous scaling laws for precision mainly focus on integer quantization, which pay less attention to the constituents in floating-point (FP) quantization, and thus cannot well fit the LLM losses in this scenario. In contrast, while FP quantization training is more commonly implemented in production, it's research has been relatively superficial. In this paper, we thoroughly explore the effects of FP quantization targets, exponent bits, mantissa bits, and the calculation granularity of the scaling factor in FP quantization training performance of LLM models.
In addition to an accurate FP quantization unified scaling law, we also provide valuable suggestions for the community: (1) Exponent bits contribute slightly more to the model performance than mantissa bits. We provide the optimal exponent-mantissa bit ratio for different bit numbers, which is available for future reference by hardware manufacturers; (2) We discover the formation of the critical data size in low-precision LLM training. Too much training data exceeding the critical data size will inversely bring in degradation of LLM performance; (3) The optimal FP quantization precision is directly proportional to the computational power, but within a wide computational power range. We estimate that the best cost-performance precision should lie between 4-8 bits.

Scaling Laws for Floating-Poimi Quantization Training
Xingwu Sun 12 Shuaipeng Li L Ruobing Xie Weidong Han' Kan Wu Zhen Yang' Yixing Li An Wang 14 Shuai Li Jinbao Xue
Yu Cheng' Yangyu Tao
Zhanhui Kang"
Chengzhong Xu 12
Di Wang
Jie Jiang'
Abstract
1. Introduction
Scaling laws of large language models (LLMs) could help developers effectively select superior parameter settings before experiments and accurately predict the model perfor- mance under different configurations. They are regarded as excellent guidance in LLM training. The widely acknowl- edged scaling law efforts such as Kaplan et al. (2020). Hoff- mann et al. (2022), and Li et al. (2024) mainly concentrated the central factors, i.e., model size and trained token size, which significantly impact the performance of LLMs. With the rapid growth of both model and data sizes, there has been increasing attention to the efficiency and cost of LLM training. Training and serving with lower precision hecomes popular solution. Currently, lots of representative LLMs were trained BF16 and even lower precision (Dubey et al., 2024: Sun al., 2024; Liu al., 2024; Yang al., 2024; Ma et al., 2024; Wang et al., 2023; Peng al., 2023), aiming to balance effectiveness and efficiency. Compared to inte- ger quantization, floating-point (FP) quantization can better maintain LLMs' accuracy at extremely lower bit rates and thus is often equipped in low- www-peccsion LLMs. Therefore, exploring the scaling laws of LLM performance under dif- ferent low precision settings with FP quantization becomes essential to shed light future low-precision LLM training.
Low-precision training is considered an effective strategy for reducing both training and down- stream inference costs. Previous scaling laws for precision mainly focus on integer quantiza- tion, which pay less attention to the constituents in doating-point (FP) quantization, and thus can- not well fit the LLM losses in this scenario. In contrast, while FP quantization training is more commonly implemented in production, it's re- search has been relatively superficial. In this paper, we thoroughly explore the effects of FP quantization targets, exponent bits, mantissa bits, and the calculation granularity of the scaling fac- tor in FP quantization training performance of LLM models. In addition to an accurate FP quan- tization unified scaling law, we also provide valu- able suggestions for the community: (1) Exponent bits contribute slightly more the model perfor- mance than mantissa bits. We provide the optimal exponent- exponent-mantissa bit ratio for different bit num- bers, which is available for future reference by hardware manufacturers; (2) We discover the for- mation of the critical data size in low- precision LLM training. Too much training data exceeding the critical data size will inversely bring in degra- dation of LLM performance; (3) The optimal FP quantization precision is directly proportional to the computational power, but within a wide com- putational power range. We estimate that the best cost-performance precision should lie between 4-8 bits.
Recently, there was pioneer work that conducted in-depth analyses and explorations on the LLM's scaling laws for precision in both training and inference (Kumar et al., 2024), quantitatively measuring the degradation rules of post-train quantization and quantized training. This seal- ing law provides an appropriate conclusion explaining the potential damage of excessively increasing training data to low-precision L.MMs' performance. However, Kumar et al. (2024) directly adopted the bit width the precision in its low-precision scaling laws, which might lose finer-grained modeling of the relationship between various parameter set- tings related to the FP quantization and the final loss of LLMs. In practice, the key factors of FP quantization such the exponent, mantissa, and the block size of scaling factors may have different impacts on the final loss. A more comprehensive, precise, and practical scaling law for FP quantized training related to the data size (D), model size
'qqual contribution 'Tencent Hunyuan Uniieersity of Macau 'The Chinese University of Hong Kong "Institute of Science Tokyo. Correspondence to: Shuaipeng Li <shuaipengliee tencent.com>. Chengzhong Xu <czxu waa.@uuu.eduu... Di Wang <di- wang mang@toncent.com²
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Scaling Laws for Floating-Pone Point Quantization Training
(N), exponent (E), mantissa (M), and block size of scaling factors (B) is urgently desired.
M1
3-4-
350
Our work concentrates on establishing, verifying, and ana- lyzing the scaling law for FP quantized training in LLMs. At the beginning, we first predict the model performance via the precision-related scaling law from previous work under different data/model sizes and precision settings. Surpris- ingly, we discover that the predictive performance was not perfectly satisfactory under different FP quantized training settings. Subsequently, we carefully design a comprehensive set of explorations with experiments of different precision settings (training 366 models), exploring the basic scaling law formation, as well as the potential impact of the quan- tization targets, exponent and mantissa, and block sizes on the loss. Finally, we aggregate these factors get our final scaling law for FP quantized training with valuable insights to guide the LLM training under low precision.
32-
325
M6
3.--
3.00
28-
2775
26
275
3.0
(a) Kumar et al. (2024).
(b) Ours.
Figure I. Comparing Eq. (6) from Kumar et al. (2024) and Eq. (13) from our work, Capybara scaling law fits data better in low - precision scenarios. Specifically, Kumar ct al. (2024)'s fitting results show considerable bias in the EIMI case. In the subfigures, data point magnitudes are roughly proportional to E.
size of LLM pre-training cannot be added indefinitely with- out harming the performance under low precision, while large model sizes, higher precision settings (measured by exponent and mantissa), and smaller block sizes could in- crease the extreme point of effective trained tokens for LLM training. (c) Intuitively, the negative impact of fow-precision training in LLMs would be proportionally amplified with the "knowledge intensity"". (d) The exponent and mantissa have their optimal settings under different bit widths. Expo- nent bits contribute slightly more to the model performance than mantissa bits. (c) The optimal FP quantization pre- cision exhibits a direct proportionality with computational power. Nonetheless, across a broad spectrum of computa- tional power, our estimated optimal pos--performance preci- sion should reside within the 4-8 bits range.
Our FP quantization training scaling law, namely Capybara (Appendix B), is formulated as follows:
T(N, D, M,
n
+ +
No
D3
D
log, B
+ Na (E + (M + 0.5)
The first two factors D and N indicate the data size and model size respectively, which show the main impacts on training loss given by the key factors of data and model size similar to the Chinchilla scaling law (Hoffmann et al., 2022); represents the bias; The last factor could be regarded as the additional negative impact deriving from low precision D No implies a certain form of "knowledge intensity" in LLM, and log, B, (E + 0.5) and (M O. jointly reflect the "low precision information loss" of FP quantized training. We have conducted extensive fitting ex- periments with various possible scaling law formulations to ensure the accuracy and simplicity of our scaling laws. Note that the exponential and data sizes are exactly the same as those in the first two factors. The product of the above "knowledge intensity" and "low precision information loss" forms the last factor.
training, where
2. Preliminary
Classical Scaling Laws. Scaling laws have become a fun- damental framework for understanding the relationship be- tween essential factors such as model size (N), data size (D), and the resulting loss (L) in deep learning. Two classical scaling laws have been widely recognized in the industry: Chinchilla scaling law (Hoffmann et al., 2022) and OpenAl scaling law (Kaplan et al., 2020). The Chinchilla scaling law is expressed as:
a and 33 of model
Figure I illustrates the fitting results of our Capybara scaling law compared with other ones, demonstrating our advan- tages on predicting LLM performances under different float quantized training settings. Throughout our experiments and analyses related to our Capybara scaling law, we also discover the following observations and insights that could facilitate future low precciccsoon LLM training: (a) It has been discovered that the impact of quantized weights on the per- formance is relatively minor during both forward and back- ward computations. Meanwhile, activations demonstrate a higher degree of quantization tolerance specifically when computing gradients pertaining to themselves. (b) The data
L(N,D) = n
+
(2)
No
D
The OpenAl scaling law is given by:
(
d D
L(N, D)
+
(3)
N
where n, d, B, and are positive fitted constants. The balance between N and D) emerges critical for compute- optimal training.
Scaling Laws for Precision. Subsequent research extends this framework by incorporating the role of precision in
2

---

Scaling Laws for Floating-Poiin Quantization Training
quantized training and inference, so to provide insights into how precision affects model performance. In Kumar et al. (2024), precision-aware scaling laws were introduced to capture the trade-offs between model size N, data size D, and precision P. For integer quantized training, they pro- posed the tradeoff between weight N and weight precision P as:
as illustrated in Figure la. The unsatisfactory fit, especially for training results using low-bit FP formats, suggests that the previous relationship proposed in Kumar et al. (2024) does not adequately capture the nuanced dynamic impacts of FP quantization on L.MM performance.
In this work, we address these shortcomings by re-deriving the scaling law for FP quamtized training. Our re-derivation incorporates a more nuanced understanding of how the finer factors of exponent, mantissa, and block size affect low- precision training. By refining the theoretical framework and aligning it more closely with observed behaviors, we aim to establish a more accurate and predictive scaling law so as to bridge the gap between theoretical insights and real-world applications.
NN, P) N(1 - P/T),
(4)
where Nar indicates the "effective parameter count" of mod- els, and y is constant representing the sensitivity of model weights to precision. Incorporating New into the Chinchilla scaling law yields:
D,,, = N(1 - PPr)ja + DB t
π
d
(5)
3. Setup and Scaling Laws
This framework highlights that reducing weight precision P can be compensated by increasing the parameter count N maintain performance, which is a critical insight for low-precision model optimization.
3.1. Method and Implementation
Quantization Method. We quantized tensor into low- precision FP format, following the IEEE 754 standard (Ka- han, 1996), which includes both normal and subnormal representations. The format consists of sign bit, F ex- ponent bits and M mantissa bits. To expand the dynamic range, the special bits are adopted for normal values instead of representing Infinity and Not Number (NaN). Since the modern hardware does not support arbitrary FP format, we simulate them using QPyTorch (Zhang et al., 2019) with nearest rounding. Due to the narrow dynamic range and low representation precision of the low-precision format, we employ scaling techniques (Sun et al., 2019; Micikevicius et al., 2022). The original tensor is multiplied by a higher- precision scaler before being cast into the www-prccision format. The scaling factor is computed as follows:
Current Scaling Laws cannot Fit Well in FP Quanti- zation. Note that most previous work focused on integer quantized training. FP quantization is more prevalent in real-world applications due to its hardware compatibility and finer granularity. For instance, formats such as FP16 and BF16 are standard in many large-scale training pipelines, and emerging formats like FP8 and FP4 are gaining traction. Despite this, scaling laws specifically tailored to FP quanti- zation are still largely unexplored. The primary distinction between FP and integer quantization lies in the allocation and usage of bits. FP numbers allocate bits to represent both the exponent and the mantissa, with each set of bits serving distinet purposes: the exponent mainly captures dynamic range, while the mantissa mainly encodes precision within that range. In contrast, integer formats uniformly distribute all bits to refine the quantization lattice, providing consistent resolution across the representable range. This fundamental difference highlights the need for dedicated scaling laws for the unique characteristics of FP formats.
S FP max Timas/man /max (()i)(+))))).
where FP represents the maximum normal value of the low-precision FP format. A scaling factor can be shared ev- ery B elements along the channel dimension. It a unified representation for tensor-wing scaling (B bdja), channel- wise scaling (B d ) and block-wise (1 < B<d,,) scaling for a tensor with the shape of b dn-
Kumar et al. (2024) hypothesized that the exponent and mantissa bits should be scaled jointly (i.e., increase together as total bit count does). Then, in FP formats, precision is determined by the exponent E and mantissa M, with the total precision: P = E + M + 1. By substituting P in their precision-aware scaling law, we have:
Implementation. The quantization is applied to the linear layers in transformer (Vaswani et al., 2017), excluding the dot-product attention and the classifier. In a linear layer's 's computation, there is one matrix multiplication during the forward phase and two during the backward phase as:
d
L(N,DEE,M)=
+
D + c, (6)
N(1 - e 1+
Y
-XW
However, upon conducting experiments and applying this scaling law to fit empirical results, particularly for low- precision training regimes, we observed significant devia- tions between the law's predictions and actual performance,
dx = =dY, Woad bad
(8)
dW = =dY2)) X "Xnodd
where X E Roxde We E Redee ddn and Y E Ruxdda rep-
3

---

Scaling Laws for Floating-Poiin Quantization Training
resent the input tensor, the weight matrix and the output tensor, respectively. b denotes the batch size, while dm and dow refer to the number of input and output channels. The two imputs per matrix multiplication are converted into a low-precision format with scaling factors. The inputs are de-quantized into BF16 tensors (Abadi et al., 2016), and BF16 multiplication is performed. The accumulators are stored in FP32 format, and the result of the accumulators converted into a BF16 tensor the output.
is expand the understanding of scaling laws include the impact of numerical precision on model performance, especially focusing the trade off's among precision, computational efficiency, and model accuracy.
In the following, we will present our methodology for inves- tigating float - precision scaling laws, experimental results, and implications for model design and training different numerical precision levels. Experiments will be used to discuss the marginal effects of exponent (E), mantissa (M), and scaling factor size (B) on LLM performance.
Modeling Object. As shown in Eq. (37) of (Kuzmin Noise Ratio (SQNR) for GEMM operations could unify the effects of exponent (E) and mantissa (M) precision while accounting for input distribution modifications like random orthogonal transformations. However, SQNR calculation inherently de- pends tensor distributions, which may limit its practical applicability. Therefore, we directly select the raw exponent and mantissa bits rather than the SNR as essential factors in our Capybara scaling law for more precise prediction ability.
et al., 2022), using the
3.4. Quantization Targets
In pursuit of balancing practicality with academic rigor, we have chosen to focus on the quantization of inputs to the General Matrix Multiplication (GEMM) computations within the Transformer architecture. Transformer consists of three main GEMM operations: forward computation, input gradient computation, and parameter gradient computation. The inputs to these matrix multiplications in both forward and backward passes include six distinct elements: X, W, dY 1. W Wodd. d 22 and Xowd, which can be quantized to (P1) through (P₂) respectively (see Figure 2a).
3.2. Setup
We trained and evaluated a range of LLaMA (Dubey et al.,
Experimental Findings. Figure 2a illustrates the classical quantization targets P1 to P6 to be explored. Through our experiments, we observed that the FP3 quantization of these inputs has varying impacts on L.M results. As illustrated in Figure 2b, our key observations related to quantization tar- gets are as follows: (1) The quantization of P1, P3, and P5 has significant effects on the model's 's performance, leading to a substantial increase in loss. Notably, the quantization of P5 results in a pronounced degradation of performance, with losses increasing by up to 2%. This suggests that compress- ing and losing information in the input embedding during the backward pass could lead to considerable performance penalties. (2) Quantizing only one target of P4 or P6 yields the optimal performance. (3) Quantizing both P2 and P6 together results in similar overall performance to quantizing P2 alone. Quantizing P2, P4, and P6 together also leads to overall results comparable to quantizing P2 alone.
2024) architecture models on a subset of the Dolma V1.7 dataset (Soldaini et al., 2024), using the same sampling pro- portion as for the OLMo 7B-v1.7 model (Groeneveld et al., 2024). Our experiments systematically explored language model pretraining across N E {41, 85, 154, 679) million parameters and D E {10, 20, 50, 100} billion tokens. Fur- thermore, we conducted two additional pretraining sessions with 1.2 billion-paramete models to validate our Capybara scaling law equation. For every (N, D) combination, we ran over 36 experiments, systematically varying exponents and mantissas, adjusting the quantization target during train- ing, and exploring different block sizes for quantization. In total, we carried out 366 runs. Detailed hyperparameters and ablation studies are provided in Table I and Table 3.
3.3. Basic Scaling Law Form
Our research, building on foundational scaling laws for train- ing data and model size that are crucial for understanding machine learning models' efficiency and effecctiveness, first evaluates the classical Chinchilla (Hoffmann et al., 2022) and OpenAI (Kaplan et al., 2020) scaling laws and then presents our precision- aware design. To find which scaling law better fits empirical data, we ran experiments with BF16 precision for model sizes from 41 million to 679 million parameters and plotted scaling law curves to show the fit between predicted and actual losses. Since the Chinchilla scaling law had a better fit in our BF16 precision experi- ments (as shown in Figure 8), we used it as the basis for exploring our proposed float precision scaling law, Our aim
Optimized Quantization Target. To balance efficiency and effectiveness, we' ve selected P2, P4, and P6 for quan- tization. Future research will build on this setup, chosen for its minimal impact on performance compared to other options. This approach preserves model integrity while gaining computational benefits.
3.5. Exponent and Mantissa
Exponents and mantissas are key elements in FP represen- tations. Proper bit-width assignments for them can signif- icantly mitigate information loss during FP quantization. Here, we aim to explore the hidden associations between
4

---

Scaling Laws for Floating-Poli Point Quantization Training
Forward
2.0
Val Loss Gap with 2-4-6
1.5
Backward
dt:
0.5
ow of.. XT
2:4
2-4-5 WWW. W.W... MoM
4-6 466 Wi
JW
W
3-
Word
0Y2
Mond
Quantization Target
(a) Quantization Targets.
(b) Loss Gaps.
Figure 2. For the subsequent exploration of scaling laws, we choose P2, P4, and P6 as our quantization targets.
exponents/mantissas and LLM performance.
jointly scaling law of N. D, as follows:
Exponent. Firstly, we investigate the scaling law when Exponent serves as an independent variable. Experiments of different Exponents with various other parameter settings have been conducted, followed by our attempt of parameter fitting. It is assumed that the Exponent-related scaling law conforms more to power-law relationship form as:
D 1
L(N,D.E) No 7E +
+ LLBBFIG.
(10)
LBF16 represents the BF16 loss L(N, D) given in the chin-
chilla scaling law of Eq. (2), i.e LaFIS No D + 6. Elegantly, we find that & s B, n 2 a, and s I in the fitting. Therefore, the Exponent scaling law is as:
a d
L(F) 7
1E+005) +
(9)
D L(N,D.E) Na 7E +05)
n
+
D3 De te. (1I)
No
We discussed other forms of relationships (e.g., Kumar et al. (2024)) and conducted relevant comparative experiments, ultimately finding that the power-law relationship is more consistent with the experimental results. The 0.5 in Eq. (9) functions as a good bias fit extreme values. According the IEEE 754 standard (Kahan, 1996), when either E (exponent) or M (mantissa) is set to 0, default information is still retained. This can also account for the existence of half-bit bias here.
Finally, we re-fit the data using Eq 11, obtaining the results shown in Figure 9a.
Mantissa. Similar Exponent, we assume the Mantissa- related scaling law conforms power-law form L(M)
x (M40.5) + c'. Surprisingly, jointly considering the effects of D and N in representing 7/ and i', we also find that & =s 3, ny su a, and z 1. Ultimately, we adopt the form of Eq. (12) to fit the joint scaling law of Mantissa with N and D.
Next, we attempt to fuse the relationship with those of data size D and model size N. Parameter fitting is carried out under various E, D. and N configurations, and the results are shown in Figure 10, where 7 is negatively correlated with D and is negatively correlated with both N and D. With N
D
1
n d Na D +e. (12)
L(N, D. No 7( + +.5)
regard to y, we re-parameterize it as function of D for i, simply multiplying the original form of the chinchilla scaling law by a coefficient precisely satisfies the pattern we discovered. Subsequently, we fit [[N, D, F), which is the
7, As
The fitting result is shown in Figure 9b.
Joint Exponent Mantissa. Integrating the scaling law results of Exponent and Mantissa when each serves as an independent variable, we can naturally organize their joint
5

---

Scaling Laws for Shaaiig-PPoin Quantization Training
3,5
36
A92
Channel-wise Strategy. To investigate the scaling law un- der the channel-wise strategy, first utilize Eq. (15) to inversely derive the equivalent block size for different N and D cases that achieve the same validation loss as when employing the channel-wise strategy. is found that this equivalent block size of the channel-wise strategy is ap- proximately constant: logz Behannet 13.1567, which is natural since the batch size of gradient (mostly) much larger than the hidden size (din in Section 3.1). After incor- porating this equivalent block size into Eq. (15), the fitted scenario of the channel-wise strategy is in Figure 13a.
944
3,4
Zin 615M
34
9028
3296
Fredier
Frudet
350
22-
3.0
3.0
-77
Figure 3. Our scaling law of Eq. (15) precisely forecasts validation loss for diverse block sizes. Data poiat sizes are directly propor- tional to D and B in the respective left and right sub- sub-figures.
Tensor-wise Strategy. Using the tensor-wise strategy, we apply Eq. (15) to predict the equivalent block size. As shown in Figure 12, this size follows a power-law relation- ship: log, Biensor <. Eq. (15) produces the fitted results in Figure 13b.
scaling law with N, D, E, M into the form of Eq. (13), The final fitting effect presented in Figure lb.
N= Substituting this prediction into
D
1
L(N, D) = NO (E 0.5) + +LBI6-
(13)
4. A Unified Scaling Law for Floating-Point Quantization Training
3.6. Block Size of Scaling Factor
In this section, provide the unified scaling law for FP quantization training, with its fitting and predictive perfor- mance as well as the insightful findings deriving from our Capybara scaling law.
In this subsection, we discuss the correlations of the block sizes and L.LM losses, and extend the block-related sealing law to channel-wise and tensor-wise strategies.
Block-wise Strategy. In the quantization process, de- fine the statistical range of the scaling factor as the block size (B). Since the scaling factor employs high-precision caching, when B- = 1, it is equivalent to retaining a high- precision copy of the tensor to be quantized. At this point, the model's expressiveness should be approximately the same that of the high-precision model: L(N, D, B 1) LBFI6 (NN, D). After comparing the power, linear, and logarithmic law forms, we ultimately select the following logarithmic form as the scaling law when block size serves as an independent variable:
4.1. The Unified Scaling Law Formation
The unified scaling law for FP quantization training should be able to jointly consider all factors of the data size (D), model size (N), exponent (E), mantissa (M), and block size of scaling factors (B) for precise low-precision LLM per- formance prediction. Based the conclusions drawn in Eq. (2), Eq. (13) and Eq. (15), we could intuitively design the unified scaling law as follows:
n d No
D,
+
DE +C+ (EC B). (16)
L(B)= k. log2 B + tb
(14)
Here, N + DJ + represents the classical BF16 loss of the Chinchilla scaling law, and p(E, M, B) indicates the additional negative impacts brought by low-precision float quantized training. Deriving from Eq. (13) and (15), we can formulate follows:
In Figure 11, we demonstrate the changes in fitted x and d under different N and D conditions. It can be observed that, similar the exploration of Exponent and Mantissa in Section 3.5, is positively correlated with N. while vb is negatively correlated with N and D, respectively. Further- more, after re- parameterizing them, we similarly found that the fitted exponents of N and D are approximately and 8, and the correction coefficient of & based on the chinchilla scaling law is approximately equal to I. Therefore, we ulti- mately build the scaling law for block size in conjunction with N and D as follows:
D
log, B
p(E,
(17)
No 7F + 0.5) 0.5)
where the exponent E, mantissa M. and block size B jointly represent the possible information loss of FP quantized train- D ing, and No reflects, in sense, the knowledge intensity of an LLM of size N trained on D data. Note that we could smoothly fuse the factors of E, M, and B with unified set of hyper-parameters of a, 3, and 7,
loga B
d D3
(NN Na
e.
(15)
K
No
We adopted all above 358 experiments in Section 3 contain- ing various N, D, E, M, B settings to obtain the specific
The fitting effect on experimental data is shown in Figure
6

---

Scaling Laws for Floating-Polle Point Quantization Training
4,0
41M
Optimal FP-layout
3.8
4M
10
679M
Volidetion 128
Volidetion 78
3.4
Validetion
(E: 0.5)%(( 0.5/
Predict
3,2
DMG
10*
722 CMU
101
155 CM6
2.8
10
1pr
2.6
10
1att 4433
(7N)
tyt mmy
10
2,50
2.75
3,00
3.25 Actual
3.50
3.75
4.00
0.0
2.5
5.0
7.5
10,0
12.5
15.0
Montisse
Figure 4. The fitting results of Capybara scaling law for FP quantization training. Data point size is proportional to D. The star points (1.2B, 7B and 70B models) our validation.
Figure The optimal float layouts of different bit widths.
Prodee Coarts
nypee-parraettes in Eq. (16) and Eq. (17).
To ensure the simplicity and universality of our Capybara scaling law, we selected Occam's Razor fuse or expurgate unnecessary typeee-arameetes. Ultimately, the final scaling law for FP guantization training is articulated as follows:
Figure 6. Variation of loss with data size under different FP quanti- zation settings.
n d No
L(N, F,
= +
+ D
D
log22 B
More details in Appendix C. The corresponding loss scaling equation incorporates this optimal layout as:
N y E (..5)((M 0.5)**
(18)
n d
D log,
LN.D.P.B) = No D3 +et Na Yop "6P6""
(20)
where the corresponding hyper-parameters are given in Ta-
ble 2. The fitting performances of Eq. (26) are given in Figure 4, which show superior capability compared pre- vious scaling laws in low-precision training.
where 7/p 16+ Figure 5 visualizes the predictive per-
75 PM
formance for different P, demonstrating the effectiveness of the derived layout in preserving performance under varying precisions. The optimal float layouts of FP4, FP8, and FP16 are E2MI, E4M3, and E8M7 (BF16), respectively.
Furthermore, we evaluate our Capybara scaling law to pre- dict the losses 1. 1.2B, 7B and 70B L.LMs with different low-precision settings and trained tokens (which are viewed as our validation models that are not used in calculating arper-parameters of our Capybara scaling law). The consis- tently accurate fitting results demonstrate that our Capybara scaling law performs well in larger model sizes.
4.3. Implication-2: Critical Data Size for Optimal Performance
From Eq. (26) and Figure 6, we can observe that there are two factors that contain and they have opposite impacts LLM loss. Intuitively, it implies that there is an optimal data size under certain FP quantization setting. The de- termination of critical data size (Darit) stands as critical juneture within the quantized training regimen. Upon ex- ceeding the threshold of Derit with the training dataset, any additional data introduction negatively impacts the model efficacy, manifesting in a rise in validation loss instead of a decline. comprehensive derivation for the estimation of
4.2. Implication-1: Optimal Float Layout Analysis
The optimal float layout for given precision P E M 1 is derived to minimize the impact of precision-related information loss on model performance. Based on Eq. (26), the optimal mantissa is expressed as:
vP Mop
0.5.
(19)
5

---

Scaling Laws for Floating-Poole Point Quantization Training
Derit is delineated in Appendix D:
Optimal Precision Pope(C)
dy + (M (M 1AAM+000 0.5) (21)
Derit
8
log2 B
Notably, a positive correlation emerges between model size (N) or training precision (P) and the occurrence of this pivotal point, indicating its delayed emergence under such conditions. Utilizing our parameter estimation framework, 1 billion-parameter model trained utilizing BF16 exhibits Derit value of 1730T, which is much larger than current data size, elucidating the previous lack of observation of this phenomenon. Conversely, when the same model is trained with FP8-EAM3, the Devit value swiftly diminishes to 27T, and with FP4-E2MI, further plummets to 0.4T. This phenomenon implies the potential harmness of larger data size on low-precision LLM training.
7
6
5-
4
1021
1024
1027
10:0
1033
Computational budget (C)
The optimal cost- cest-performacce ratio precision as func- tion of the total compute budget, illustrating the relationship be- tween precision (P") and computational budget (C) when the block size (B) is set to 128 and & 6/16.
Figure 7.
4.4. Implication-3: Compute-Optimality with Fixed Configurations
We control the total computation cost C kNPD and analysis the optimal configuration under Eq. (26).
Minimization over N, D, P with Fixed Compute. Through a joint analysis of the impacts of N, D, and P on the final validation loss, the relationship between cost- effective precision and expected compute budget as:
Fixed Data Size D. For fixed D, the optimal precision Per minimizes the loss while accounting for computational constraints. From Appendix E.1, Port expressed as:
p ((+tt) a42
2+3
= logz B)
(25)
k
(D) = D log, B) 540
(22)
where X カ-リー2 In Appendix E.3, present 0+4+8" detailed derivation processes. Based the parameters fitted from our experimental data presented in Table 2, with the block size (B) set to 128 and k 6/ 16, as illustrated in Figure when the total compute budget is in the range of (1021, 1091) FP operations, the optimal cost-performance ratio precision is found to lie between and bits. This finding implies that training larger models with lower preci- sion and utilizing less data yields more cost-effective ap- proach. Developers could rely our implications from our Capybara scaling law to decide their optimal float-pointing quantization settings.
where YD 8+2-a which consolidates the relationships "ary
between model precision and compute efficiency. Eq. (22)
suggests that we can adopt such a quantization strategy: in the early stage training, employ aggressive quantization strategies such FP8- EP8-E4M3, or even FP4-E2MI which may quickly converge
be available in the future hardware, the model to better level. Subsequently, as the data vol- ume as well as the "knowledge intensity" further increase, gradually enhance the training precision to BF16, or even FP32, in order to maintain the optimal cost-effectiveness of training.
Fixed Model Size N. For fixed N, the optimal preci- sion depends balancing the compute resources and main- taining the required training data size. The corresponding scaling law and derivations are in Appendix E.2. We have:
5. Related Work
Scaling Law of LLMs. Due to the extremely large cost of resource and time for LLM training, discovering appropriate scaling laws to accurately predict LLM capabilities under different parameters is essential for product-level training. Kaplan al. (2020) gave the classic form of the scaling law and concludes that the performance penalty depends predictably on the ratio No074/ /0.71/D. Hoffmann et al. (2022) modeled the final loss as a parametric function of the count of model parameters and the number of trained tokens, i.e. E+A/N* +B/D Bahri et al. (2024) and Lin et al. (2024) theoretical analysis of how loss scales with the size of the
C K
Popa PNV)
YN
N (a+2)) log, B
(23)
finding analogous to that presented Kumar et al. (2024)
observed herein, specifically, under the limitation of com- putational resources, equilibrium exists between preci- sion and model size from Eq. (23) is as:
+ 22++2 Constant.
(24)
8

---

Scaling Laws for Floating-Pooie Quantization Training
training dataset and the number of parameters in a power- law manner. Previous work (Dettmers & Zettlemoyer, 2023) explored the scaling laws of LLMs with different bit preci- sions. Recently, Kumar et al. (2024) focused on the impact of model parameters and data volume with low precision, highlighting the possible negative impacts of more trained tokens in low-precision LLM training and serving. Other recent work (Ouyang et al., 2024) also investigated the corre- lations of integer quantization and LLM performance. Nev- ertheless, it is still not that particularly clear to conclude the scaling law for FP quantization training with respect to the specific selection of the exponent, mantissa, and block size of scaling factors.
Acknowledgement
Ruobing Xie is supported by the Young Elite Scientists
Sponsorship Program by CAST (2023QNRC00).
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Quantization of L.Mss. The quantization technique of large language models (Lang et al., 2024; Shen et al., 2024) has received widespread attention. Xiao et al. (2023) re- duced the accuracy loss during quantization by smooth- ing the distribution of activations and weights. Dettmers et al. (2024) combined Quantization-Awarr Training and LoRA methods to implement an efficient fine-tuning method. Egiazarian et al. (2024) explored techniques for compress- ing large language models at very low bit rates, and Behdin et al. (2023) proposed a framework that allows each layer to be quantized independently. Although previous work (Zhang et al., 2023; Yoshida, 2023) studied the impact of exponent, mantissa and blocksize on the quantification of LLMs, the comprehensive impact of these indicators has not been systematically studied and summarized.
Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al. Tensorflow: Large-scale machine learning heterogeneous distributed systems. arXiv preprint arXiv: /603.66467, 2016.
Bahri, Y., Dyer, E.,, Kaplan, J., Lee, J., and Sharma, U. Ex- plaining neural scaling laws. Proceedings of the National Academy of Sciences, 121 121((7):c23118781) 2024.
Behdin, K., Acharya, A., Aman Gupta, S. K., and
Mazumder, R. Quantease: Optimization- based quan- tization for language models-an efficient and intuitive algorithm. stat, 1050:5, 2023.
Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Proceedings of ICML, 2024.
6. Conclusion, Limitation, and Future Work
In this work, we propose our Capybara scaling law, which functions satisfactorily as a precise guidance of future low- precision LLM training. The key factors of the data size (D), model size (N), exponent (E), mantissa (M), and block size of scaling factors (B) are carefully considered in our Capybara scaling law throughout several hundred of experi- ments with various precision and model settings. Besides the scaling law, we also discover some insightful implica- tions that could instruct and enhance future FP quantization training in I.MMs. We hope our findings could shed light on better prediciiin LLM training to facilitate the LLM community.
Dettmers, T. and Zettlemoyer L. The case for 4-bit preci- sion: k-bit inference scaling laws. In International Con- ference on Machine Learning, pp. 7750-7774. PMI.R, 2023.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized 1lms. Advances in Neural Information Processing Systems, 36, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2024.
In the future, will verify our scaling laws for FP quan- tization training under larger model sizes and data sizes. Currently, our explorations are conducted based on the clas- sical Transformer architecture. Whether our scaling laws could also be smoothly applied to LLMs of other archi- tectures (e.g., Mamba series (Dao & Gu, 2024)) is worth confirming. Moreover, our experiments focus on the clas- sical FP quantization strategies, while other new-proposed low-bit LLM quantization methods and hardware should also be covered in the future.
Egiazarian, V., Panferov, A., Kuznedelex, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. In Proceedingy of ICML, 2024.
Groeneveld, D., Beltagy, I., Walsh, P., Bhagia, A., Kinney, R., Tafjord, O., Jha, A. H., Ivison, H., Magnusson, I., Wang, Y., Arora, S., Atkinson, D., Authur, R., Chandu, K., Cohan, A., Dumas, J., Elazar, Y., Gu, Y., Hessel, J., Khot, T., Merrill, W., Morrison, J., Muennighoff, N.,

---

Scaling Laws for Ploating-Poin Quantization Training
Naik, A., Nam, C., Peters, M. E., Pyatkin, V., Ravichan-
Micikevicius, P., Stosic, D., Burgess, N., Corea,, M., Dubey, P., Grisenthwaite, R., Ha, S., Heinecke, A., Judd, P., Kamalu, J., et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.04333 2022.
der, A., Schwenk, D., Shah, S., Smith, W., Subramani, N., Wortsman, M., Dasigi, P., Lambert, N., Richardson, K. Dodge, J., Lo, K., Soldaini, L., Smith, N. A., and Ha- jishirzi, H. Olmo: Accelerating the science of language models. Preprint, 2024.
Ouyang, X., Ge, T., Hartvigsen, T., Zhang, Z., Mi, H., and Yu, D. Low-bit quantization favors undertrained Ilms: Scaling laws for quantized Ilms with 100t training tokens. arXiv preprint 2tii::2111.1769999 2024.
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L.., Hendricks, L.. A., Welbl, J., Clark, A., al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 2022.
Peng, H., Wu, K., Wei, Y., Zhao, G. Yang, Y., Liu, Z., Xiong, Y., Yang, Z., Ni, B., Hu, J., et al. Fp8-Im: Training fp8 large language models. arXiv preprint arXiv:2310.1833 2023.
Kahan, W. Ieee standard 754 for binary floating-pont point arith- metic. Lecture Notes on the Status of IEEE, 754(94720- 1776):11, 1, 1996.
Shen, A., Lai, Z., and Li, D. Exploring quantization tech- niques for large-scale language models: Methods, chal- lenges and future directions. In Proceedings of the 2024 9th International Conference on Cyber Security and In- formation Engineering, 2024.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.08361, 2020.
Soldaini, L., Kinney, R., Bhagia, A.. Schwenk, D., Atkinson, D., Authur, R., Bogin, B., Chandu, K., Dumas, J., Elazar, Y., Hofmann, V., Jha, A. H., Kumar, S., Lucy, L., Lyu, X., Lambert, N., Magnusson, I., Morrison, J., Muennighoff, N., Naik, A., Nam, C., Peters, M. E., Ravichander, A., Richardson, K., Shen, Z., Strubell, E., Subramani, N., Tafjord, O., Walsh, P., Zettlemoyer, L., Smith, N. A., Hajishirzi, H., Beltagy, I., Groeneveld, D., Dodge, J., and Lo. K. Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. In Pruceedings of ACL, 2024.
Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muen- nighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghu- nathan, A. Scaling laws for precision. arXiv preprint arXiv:2411. 1,004330, 2024.
Kuzmin, A., van Baalen, M., Ren, Y., Nagel, M., Peters, J., and Blankevoort, T. Fp8 quantization: The power of the exponent. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 14651-14662. Curran Associates, Inc., 2022.
Lang. J., Guo, Z., and Huang, S. A comprehensive study on quantization techniques for large language models. In 2024 4th International Conference on Artificial In- telligence, Robotics, and Communication (ICAIRC). pp. 224-231. IEEE, 2024.
Sun, X., Choi, J., Chen, C.-Y,, Wang, N., Venkataramani, S., Srinivasan, V. V., Cui, X., Zhang, W., and Gopalakr- ishnan, K. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. Advances in neural information processing systems, 32, 2019.
Li, S., Zhao, P., Zhang, H., Sun, S., Wu, H., Jiao, D., Wang. W., Liu, C.,, Fang, Z., Xue, J., Tao, Y., CUI, B., and Wang, D. Surge phenomenon in optimal learning rate and batch size scaling. In The Tiirty-eigtth Annual Conference on Neural Information Processing Systems, 2024.
Sun, X., Chen, Y., Huang, Y., Xie, R., Zhu, J., Zhang, K., Li, S., Yang, Z., Han, J., Shu, X., et al. Hunyuan-large, An open-source moe model with 52 billion activated pa- rameters by tencent. arXiv preprint arXiv:241 02:224111022555 2024.
Lin, L., WU, J., Kakade, S. M., Bartlett, P. L., and Lee, J. D. Scaling laws in linear regression: Compute, parameters, and data, arXiv preprint arXiv:2406. 244.6.666000. 2024.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of NeurlPS, 2017.
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., al. Deepseek-v2: A strong, economical, and effficintt mixture-of-experts language model. arXiv preprint arXiv:2405.044
Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang. F., Wang, R., Wu, Y.. and Wei, F. Bitnet: Scaling 1- bit transformers for large language models. arXiv preprint W:Xiv:2310.11455 2023.
2024. Ma. S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., and Wei, F. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402. 17764, 2024.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International
10

---

Scaling Laws for Floating-Poli PPint Quantization Training
Conference Machine Learning, pp. 38087-38099.
PMIR, 2023.
Yang. A., Yang, B., Zhang, B., Hui. B., Zheng, B., Yu,
B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.55115, 2024.
Yoshida, D. N14 isn't ist't information theoretically optimal (and that's good). arXiv preprint arXiv:2306.06966 2023.
Zhang, C., Cheng. J., Shumailov, I., Constantinides, G. A.,
and Zhao, Y. Revisiting block-based quantisation: What is important for sub-8-bit Ilm inference? In Proceedings of EMNLP, 2023.
Zhang. T., Lin, Z., Yang. G., and De Sa, C. Qpytorch: A low- precision arithmetic simulation framework. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-NeurlPS Edition (EMC2-NIPS), pp- 10-13, IEEE, 2019.

---

Scaling Laws for Floating-Pone PPintt Quantization Training
A. Hyperparameter Details
The detailed hyper-parameters of our LLMs are given as follows:
Table 1. Model hyper- for cach size.
41M
85M
154M
679M
1.2B
7B
70B
Layers Hidden Size
12
12
12
24
24
35
88
512
768
1024
1536
2048
4096
8192
FEN Hidden Size
1536
2048
2816
4096
5632
10752
22016
Attention Heads
8
16
24
32
64
Attention Head size
64
64
64
64
64
64
64
Optimizer
Adam
Adam (2),
(0.9, 0.95)
Adam
x 10
Weight Decay
0.1
Clip Grad Norm
1.0
Max LR
3.0 10
Min LR
0 Cosine
LR Decay
Decay Rate
10%
Seqence Length
2048
Batch Size Tokens)
2M
Warmup Steps
500
B. Fitting Details of Our Scaling Laws
We name our scaling law for Floating- point quantization training as the Capybara scaling law. Under constrained resources and space, increasing the number capybaras can significantly reduce their survival rate and quantity once certain density threshold surpassed. We observe similar phenomenon our scaling law: with fixed model size, expanding the data size does not consistently yield improvements when the "knowledge density" becomes too high under the pressure of low-precision training. More fitting details of our Capybara scaling laws introduced as follows.
B.I. Numberical Fits
Our Capybara scaling law is formalized as:
d
D
log, B Na
DN,, No D
(26)
<E (M.)%(MMM) (MM0M+0000 +.5)**
where the detailed fitted constants and values are shown as follows:
Table Fitted constants and their values Eq. (26).
Constant
Value
n
69.2343
a
0.2368
d
68973.0621
0.5162 B
e
1,9061
11334.5197 ?
8
3.1926
2,9543 D
12

---

Scaling Laws for Floating-Poue -Pint Quantization Training
B.2. Fitting Results of Classical Scaling Laws
We give the fitting results of two classical scaling laws, i.e., the Chinchilla scaling law and the OpenAl scaling law, in Figure 8. We select the Chinchilla scaling law for reference considering its better fitting performance.
41M
41M
3,4
85M
3,4
85M
154M
154M
679M
679M
3.2
3.2
Predict
Predict
3.0
3.0
2.8
2.8
2.6
2.6
2.6
2.8
3.0 Actual
3.2
3.4
2.6
2.8
3.0
3.2
3.4
Actual
(a) Chinchilla.
(b) OpenAL.
Figure 8. Fitting performance of classical scaling laws, with data point size proportional to D. Left: Curve based Chinchilla scaling law shows excellent empirical training loss alignment with predicted losses. Right: Curve based on OpenAl scaling law also demonstrates good match, though less precise than Chinchilla.
B.3. Fitting Results of the Scaling Laws for Exponent and Mantissa
Figure 9 shows the fitting performance of our scaling law related to the exponent mantissa, and Figure 10 shows the correlations between essential parameters in Eq. (9).
41M
41M
3.75
85M
3.75
154M
154M
3.50
679M
3.50
679M
Predict
Predict
3.25
3.25
3.00
3.00
2.75
2.75
2.75
3.00
3.25 Actual
3.50
3.75
2.75
3.00
3.25 Actual
3.50
3.75
(a) Exponent.
(b) Mantissa.
Figure 9, The fitting outcomes of our scaling law related to the exponent (left)/mantissa (right). Data point size is proportional to D.
13

---

Scaling Laws for Floating-Point Point Quantization Training
1.25
1.25
41M
41M
41M
05
1,20
1,20
154M
154M
15-M
679M
1.15
1,15
679M
0.0
logy
log
1.10
1,10
-0.5
1.05
1.05
-1.0
1.00
1.00-
-1.5
0,95
0.95
-8
-7
-6
-4
-3
19 log M
20
23.0
23.5
24,0
24.5
25.0
log m
log
Figure The correlations between 7: Eq. (9) and N.D. y,u can viewed as functions N.D. Data point size proportional to
B.4. Fitting Results of the Scaling Laws for Scaling Factor
These figures show different correlations and fitting results related the scaling factor B in our scaling law.
-3.0
41M
41M
41M
1.2
85M
85M
154M
154M
154M
-3.5
log
log
1.1
loge
1.1
-4.0
-7
-6
-5
-4
-3
18
19
20
18
20
log
log
log
Figure The correlations between ka3 in Eq. (14) and N,D, K.g could viewed as functions of N,D. The data points are scaled proportionally to the value of D.
4,00
41M
85M
3.75
154M
log log (logz B)
679M
3.50
3.25
3,00
2.75
-8
-7
-6
-5
-4
-3
log
Figure correlations between log, and 73
The size the data point proportional D.
14

---

Scaling Laws for Floating-Point Point Quantization Training
4.0
41M
41M
3.8
85M
3.9
85M
154M
154M
3.6
679M
3.8
679M
Predict
Predict
3.7
3.4
3.6
3.2
3.5
3,0
3,4
3,0
3.2
3.4
3.6
3.8
3,4
3,6
3.8
Actual
Actual
(a) Channel-wise.
(b) Tensor-wise.
Figure 13. Fitting results of the scaling law under different sharing strategics of the scaling factor. The size of the data point is proportional
D.
C. Optimal Float Layout
Given specified total precision (P), the process determining the optimal allocation of exponent bits (E) and mantissa bits (M) involves substituting the equation:
E = P M 1.
(27)
into the proposed scaling law delineated in Eq. (17):
D
logz B
P(M)
Na
7(P - 1 - M + 0. + 0.5)"
(28)
D
log, B
No
-
M) M + 0.5)
Subsequently, the loss function L, with respect to the mantissa bits M, is expressed as:
d D
log, B
L(M)
+ +et
(29)
Na D
Na 7P 0.5 - M) + 0.5)*
To optimize this function, we compute the partial derivative of with respect to M:
ØL
D
log, v(P 0.5 MM + 0.5) 5(P 0.5 M) (M + 0.55"
9M
Na
(P - 0.5 (M
(30)
D
loga B
V
No
7 -P- 0.5 (M
M 0.5
P 0.5 M
By setting this partial derivative equal to zero, we obtain the optimal value for M which given by Eq. (31):
15

---

Scaling Laws for Floating-Poole Point Quantization Training
OL
0M
8
0
(31)
M + 0.5
P 0.5
M
vP
M
0.5,
+ U
and the corresponding value for E is:
Bopt P- 1 Mop opt
8P
(32)
0.5.
8+
Next, we introduce the optimal values Mop and Popt into the proposed scaling law, formulated in Eq. (17):
D
log,
B
Popt P)
No
6P
uP
8+0
(33)
D
loga B
(5 + 1)6+v
Na
80
pô+.
For the sake of simplification, we introduce the parameter n/p defined as follows:
75 nn
7pp (5 1)
(34)
As result,
D logg B No 7o P84v
Popt (P)
(35)
By substituting Eq. (35) into the unified scaling equation, namely Eq. (16), we arrive at Eq. (20), which further simplified to:
n
D
log, B
(P)
+
+et
Na
D8
Na
7op P6+v
D
log, B
+ d
1+
D te
Na
n
7 p8+u
(36)
n
+
+ c.
D8
Na
1+ D loga B Pa
Furthermore, let Yn. denote constant value:
7n Yp..
(37)
In accordance with the Chinchilla scaling law (Hoffmann et al., 2022), we define Ne as the count of effective parameters, which aligns with the model size specified in Eq. (2):
16

---

Scaling Laws for Floating-Poll Point Quantization Training
Neff = N
(38)
1
logg B "/a
When the condition D logg B > in po+y is satisfied, the effective number of parameters, Ness, can be simplified as follows:
Neff
7n D log, B
N
(39)
Hence, we discern a power-law relationship between the number of effective parameters, Neff, and the precision, P. important to emphasize that Ne.s is influenced not solely by the quantization technique employed but also by the volume of data. When both the model size and the quantization method are held constant, increase in data size leads to decrease in the number of effective parameters.
D. Critical Data Size
For FP quantization training, overtraining may occur, that is, when the amount of training data exceeds the critical value, the loss will increase instead. Given exponent bits (E), mantissa bits (M), block size (B), and number of model parameters (N), we aim find the critical data size before over-training. Based Eq. (16), we derive the expression for D when the partial derivative with respect D is zero.
Preliminarily, we compute the partial derivative of the loss function L with respect to the data size D:
aL
0 d
0
of OF, M, B)
aD
aD
No
OD
D
0D
OD
d
D3-1
log,
(40)
= B D8+T
+ 3
No
7(E + 0.5) [ + 0.5)
By setting this partial derivative zero and then solving for D we obtain:
aL
0D
D
log, B
D3+1
Na
7(E + 0.5) (M 0.5)
(41)
D (.5) 0.5)*
log, B
Consequently, the critical value of D is given by Eq. (21).
E. Compute-optimality
In order to investigate the optimal precision P under constrained computational budget, we define the computational expenditure associated with FP quantization training as follows:
C C=
(42)
Here, l signifies a proportionality constant, denotes the computational cost per model parameter, and accounts for the additional expense incurred during the multiplication of scaling factors.
17

---

Scaling Laws for Ploating-Poin Quantization Training
E.1. Fixed data size D
For the critical precision P in relation the data size Eq. (42) can be incorporated into the proposed scaling law, as expressed in Eq. (20):
d
D
log, B
L(D.)) P)
+c+
D
C
7/ p6-v
M(P+bDD
M(P+b)D
(43)
kD n c
(P+ b)a0 + D log,
kD
(P
d
D5 te.
7
C
p
We then compute the partial derivative of the loss function L(D,P) P) with respect to P:
OL(D,)) P)
kD) a C
Do loga B
kD
a a (PPP +
1)0 - (6 + DP + b)app-u-1 p2(6+v)
(P+b)a-1
+
0P
Pp
C
(44)
kD
D log2 B
kD
(P
6)
8 v
(P + b)a0))a--0)) +
na
C
C
p6+v
P+b
P
Upon setting the partial derivative of the loss function with respect to precision Pequal to zero, and solving for P, we obtain:
OL(D, P)
OP =0.
(45)
P+b
+d+v V/
a
nypa
p6+v
P
P 9+1
D logz B
Assuming
+ v nay
TD
(46)
and considering that b = 0, the critical precision P is determined as:
pstv = =DD log, B.
(47)
As illustrated in Figure 14, Eq. (47) suggests that the amount of training data increases, the most economical precision also correspondingly rises under the constraint of limited computational power.
E.2. Fixed model size N
With respect to the critical precision P in relation to the model size N, we can streamline Eq. (20) to:
C
)a
d
K[P+0]N
log, B Yp pô+0
L(N
+et
Na
No
K[PDDJN
(48)
KN B
log B
C
8
d
(P + b)
No
K:N
(P+ b-0
No
Subsequently, we evaluate the partial derivative of the loss function I(N, P) with respect to
18

---

Scaling Laws for Floating-Point Point Quantization Training
9
Optimal Precision Pope(
8
7
6
5
4
10
10"
10
10
10
10
Data size (D)
Figure 14, Under the constraint of computing the budget with block size (B) set to 128, and based on the results of experimental data fitting, the optimal precision (P) values for different data sizes (D) can be deduced. As depicted, across substantially broad range of data sizes from O.IT to 100T, the optimal precision value consistently falls within the range of 4 8 bits.
OL(N,), P) OP
KN
8
1ogz B
C
3(P + 1) + (6 - (P + b) pô+v-1
d3
(P (P+0)) b)8-1
C
No
KN
(P+ ()+8
(49)
kN
log2 B
C
B 6+
d3
(P + 6)8
C
Na
kN
(P po+v
P+b
Upon setting this partial derivative to zero, and solving for P. we arrive at:
AL(N, P)
=0
OP
log2 B
C 8
8
8 ++
KN B
d3
(P + 6)8-1111
(50)
Na
A:N
(P + b)3 p8+v
P+b b
C
1
3
8 + V
da
(NN)
28
+
(P + 6)28 p8+
P+b b
P
log, B
By introducing
8+0+1
YN
(51)
d8%p
and under the assumption that the critical is deduced be:
1
3 p
8 8+D
d3
KN
28
p23- pôto
P
log, B
C
(3 + 6 v) log,
28
po+u+23
(52)
Na
KN
C IN
2
p5+0+28
N (a+22)) logg B.
K
E.3. Minimization over N, D, with Fixed Compute Based on the results from Section F. E.2 and specifically Eq. (42) with b=0, we proceed address the system of equations:
19

---

Scaling Laws for Floating-Polle PPonnt Quantization Training
OL(D.P) 8D = 0.
aL(D.P) O. 0P
(53)
C= kPND.
This subsequently leads to the expression of P in relation the computational budget C::
plo+u) ( (ND log2 B) whd
d3 6 +
/ (')
(54)
na
+ v B
F. Detailed Settings of Experiments
We show the detailed configurations of our experiments follows:
Table All configurations for the ablation experiments.
N
E
M
Fitting support
0
40894464
10485760000
0
7
channel
1
40894464
10485760000
1
1
32
2
40894464
10485760000
I
I
64
3
40894464
10485760000
I
1
128
4
40894464
10485760000
I
256
5
40894464
10485760000
I
512
6
40894464
10485760000
I
channel
7
40894464
10485760000
I
tensor
8
40894464
10485760000
I
2
channel
9
40894464
10485760000
I
3
channel
10
40894464
10485760000
I
4
channel
11
40894464
10485760000
1
5
channel
12
40894464
10485760000
1
channel
13
40894464
10485760000
2
channel
14
40894464
10485760000
2
3
channel
15
40894464
10485760000
3
1
channel
16
40894464
10485760000
3
2
channel
17
40894464
10485760000
4
channel
18
40894464
10485760000
4
3
channel
19
40894464
10485760000
4
5
channel
20
40894464
10485760000
5
channel
21
40894464
10485760000
5
2
channel
22
40894464
10485760000
6
channel
23
40894464
20971520000
0
7
channel
24
40894464
20971520000
I
32
25
40894464
20971520000
I
64
26
40894464
20971520000
I
128
27
40894464
20971520000
256
28
40894464
20971520000
I
512
29
40894464
20971520000
1
I
channel
30
40894464
20971520000
I
tensor
31
40894464
20971520000
1
2
channel
32
40894464
20971520000
1
3
channel
20

---

Scaling Laws for Floating-Poiie Quantization Training
33
40894464
20971520000
1
4
channel
34
40894464
20971520000
1
5
channel
35
40894464
20971520000
1
6
channel
36
40894464
20971520000
2
1
channel
37
40894464
20971520000
2
3
channel
38
40894464
20971520000
3
channel
39
40894464
20971520000
channel
40
40894464
20971520000
4
channel
41
40894464
20971520000
4
3
channel
42
40894464
20971520000
4
5
channel
43
40894464
20971520000
5
1
channel
44
40894464
20971520000
5
2
channel
45
40894464
20971520000
6
channel
46
40894464
52428800000
O
channel
47
40894464
52428800000
I
32
48
40894464
52428800000
I
64
49
40894464
52428800000
I
128
50
40894464
52428800000
256
51
40894464
52428800000
512
52
40894464
52428800000
I
channel
53
40894464
52428800000
I
tensor
54
40894464
52428800000
I
2
channel
55
40894464
52428800000
I
3
channel
56
40894464
52428800000
1
4
channel
57
40894464
52428800000
I
5
channel
58
40894464
52428800000
I
6
channel
59
40894464
52428800000
2
channel
60
40894464
52428800000
2
3
channel
61
40894464
52428800000
3
channel
62
40894464
52428800000
3
2
channel
63
40894464
52428800000
4
1
channel
64
40894464
52428800000
4
3
channel
65
40894464
52428800000
4
5
channel
66
40894464
52428800000
5
channel
67
40894464
52428800000
5
2
channel
68
40894464
52428800000
6
channel
69
40894464
1048576000G
0
7
channel
70
40894464
104857600000
1
1
32
71
40894464
104857600000
64
72
40894464
104857600000
128
73
40894464
256
74
40894464
104857600 7600000
I
512
75
40894464
104857600 7600000
channel
76
40894464
(048576000)
I
tensor
77
40894464
104857600000
I
2
channel
78
40894464
1048576000m
1
3
channel
79
40894464
1
4
channel
80
40894464
(048576000))
1
5
channel
81
40894464
1
6
channel
82
40894464
(0485760000)
2
channel
83
40894464
104857600000
2
3
channel
84
40894464
104857600000
3
1
channel
85
40894464
104857600000
3
2
channel
86
40894464
104857600000
4
1
channel
21

---

Scaling Laws for Floating-Poiie Quantization Training
87
40894464
104857600000
4
3
channel
88
40894464
104857600000
4
5
channel
89
40894464
104857600000
5
1
channel
90
40894464
104857600000
5
2
channel
91
40894464
104857600000
6
channel
92
84934656
10485760000
0
7
channel
93
84934656
10485760000
I
32
94
84934656
10485760000
I
64
95
84934656
10485760000
I
128
96
84934656
10485760000
256
97
84934656
10485760000
channel
98
84934656
10485760000
I
tensor
99
84934656
10485760000
I
2
channel
100
84934656
10485760000
3
channel
101
84934656
10485760000
I
4
channel
102
84934656
10485760000
I
5
channel
103
84934656
10485760000
1
6
channel
104
84934656
10485760000
2
1
channel
105
84934656
10485760000
2
3
channel
106
84934656
10485760000
3
channel
107
84934656
10485760000
3
2
channel
108
84934656
10485760000
4
channel
109
84934656
10485760000
4
3
channel
110
84934656
10485760000
4
5
channel
111
84934656
10485760000
5
1
channel
112
84934656
10485760000
5
2
channel
113
84934656
10485760000
6
channel
114
84934656
20971520000
O
7
channel
115
84934656
20971520000
I
32
116
84934656
20971520000
I
64
117
84934656
20971520000
I
128
118
84934656
20971520000
I
256
119
84934656
20971520000
I
channel
120
84934656
20971520000
I
tensor
121
84934656
20971520000
I
2
channel
122
84934656
20971520000
I
3
channel
123
84934656
20971520000
I
4
channel
124
84934656
20971520000
1
5
channel
125
84934656
20971520000
1
6
channel
126
84934656
20971520000
2
1
channel
127
84934656
20971520000
2
3
channel
128
84934656
20971520000
3
channel
129
84934656
20971520000
3
channel
130
84934656
20971520000
4
channel
131
84934656
20971520000
4
3
channel
132
84934656
20971520000
4
5
channel
133
84934656
20971520000
5
1
channel
134
84934656
220700
5
2
channel
135
84934656
120000
6
channel
136
84934656
52428800000
0
7
channel
137
84934656
52428800000
I
1
32
138
84934656
52428800000
I
I
64
139
84934656
52428800000
I
I
128
140
84934656
52428800000
1
1
256
22

---

Scaling Laws for Floating-Poing Quantization Training
141
84934656
52428800000
1
1
channel
142
84934656
52428800000
1
1
tensor
143
84934656
52428800000
1
2
channel
144
84934656
52428800000
I
3
channel
145
84934656
52428800000
I
4
channel
146
84934656
52428800000
I
5
channel
147
84934656
52428800000
I
channel
148
84934656
52428800000
2
channel
149
84934656
52428800000
2
3
channel
150
84934656
52428800000
3
1
channel
151
84934656
52428800000
3
2
channel
152
84934656
52428800000
4
channel
153
84934656
52428800000
4
3
channel
154
84934656
52428800000
4
channel
155
84934656
52428800000
5
channel
156
84934656
52428800000
5
2
channel
157
84934656
52428800000
6
1
channel
158
84934656
10485760000
0
7
channel
159
84934656
1048576 (04857600000
I
32
160
84934656
104857600 7600000
I
64
161
84934656
104857600000
7600000
I
128
162
84934656
I
256
163
84934656
10485760 760000
I
channel
164
84934656
(0485760000)
1
tensor
165
84934656
104857600000
I
2
channel
166
84934656
1048576000 760000
I
3
channel
167
84934656
1048576000g)
I
4
channel
168
84934656
(0485760000)
I
5
channel
169
84934656
(0485760000)
1
channel
170
84934656
10485760000
2
channel
171
84934656
104857600000
2
3
channel
172
84934656
104857600000
3
1
channel
173
84934656
10485 760000
3
2
channel
174
84934656
104857600000
4
channel
175
84934656
104857600000
4
3
channel
176
84934656
10485760000
4
5
channel
177
84934656
10485760000)
5
channel
178
84934656
104857600000
5
2
channel
179
84934656
104857600000
6
channel
180
154140672
10485760000
O
7
channel
181
154140672
10485760000
I
32
182
154140672
10485760000
I
64
183
154140672
10485760000
128
184
154140672
10485760000
256
185
154140672
10485760000
I
channel
186
154140672
10485760000
I
tensor
187
154140672
10485760000
I
2
channel
188
154140672
10485760000
I
3
channel
189
154140672
10485760000
1
4
channel
190
154140672
10485760000
I
5
channel
191
154140672
10485760000
1
6
channel
192
154140672
10485760000
2
1
channel
193
154140672
10485760000
2
3
channel
194
154140672
10485760000
3
1
channel
23

---

Scaling Laws for Floating-Poine Quantization Training
195
154140672
10485760000
3
2
channel
196
154140672
10485760000
4
1
channel
197
154140672
10485760000
4
3
channel
198
154140672
10485760000
4
5
channel
199
154140672
10485760000
5
channel
200
154140672
10485760000
5
2
channel
201
154140672
10485760000
6
channel
202
154140672
2097 10971520000
0
7
channel
203
154140672
20971520000
I
32
204
154140672
20971520000
64
205
154140672
20971520000
128
206
154140672
20971520000
256
207
154140672
20971520000
channel
208
154140672
20971520000
tensor
209
154140672
20971520000
I
2
channel
210
154140672
20971520000
I
3
channel
211
154140672
20971520000
I
4
channel
212
154140672
20971520000
I
5
channel
213
154140672
20971520000
1
6
channel
214
154140672
20971520000
2
channel
215
154140672
20971520000
2
3
channel
216
154140672
20971520000
3
channel
217
154140672
20971520000
3
2
channel
218
154140672
20971520000
4
channel
219
154140672
20971520000
4
3
channel
220
154140672
20971520000
4
5
channel
221
154140672
20971520000
5
channel
222
154140672
2097152000
5
2
channel
223
154140672
2097 1520000
6
channel
224
154140672
52428800000
O
7
channel
225
154140672
52428800000
1
32
226
154140672
52428800000
I
64
227
154140672
52428800000
I
128
228
154140672
52428800000
256
229
154140672
52428800000
I
channel
230
154140672
52428800000
tensor
231
154140672
52428800000
I
2
channel
232
154140672
52428800000
I
3
channel
233
154140672
52428800000
I
4
channel
234
154140672
52428800000
I
5
channel
235
154140672
52428800000
1
6
channel
236
154140672
52428800000
2
channel
237
154140672
52428800000
2
3
channel
238
154140672
52428800000
3
channel
239
154140672
52428800000
3
2
channel
240
154140672
52428800000
4
1
channel
241
154140672
52428800000
4
3
channel
242
154140672
52428800000
4
5
channel
243
154140672
52428800000
5
channel
244
154140672
52428800000
5
2
channel
245
154140672
52428800000
6
channel
246
154140672
104857600000
0
7
channel
247
154140672
104857600000
1
I
32
248
154140672
104857600000
I
I
64
24

---

Scaling Laws for Floating-Poins Quantization Training
249
154140672
104857600000
1
1
128
250
154140672
104857600000
1
1
256
251
154140672
104857600000
1
1
channel
252
154140672
104857600000
1
1
tensor
253
154140672
104857600000)
I
2
channel
254
154140672
104857600000
I
3
channel
255
154140672
104857600000
I
channel
256
154140672
I
5
channel
257
154140672
104857600000
1
6
channel
258
154140672
104857600000
2
1
channel
259
154140672
104857600000
2
3
channel
260
154140672
104857600000
3
channel
261
154140672
104857600000
3
2
channel
262
154140672
104857600000
4
channel
263
154140672
104857600000
4
3
channel
264
154140672
104857600000
4
5
channel
265
154140672
104857600000
5
1
channel
266
154140672
104857600000
5
2
channel
267
154140672
104857600000
6
channel
268
679477248
10485760000
O
7
channel
269
679477248
10485760000
I
32
270
679477248
10485760000
64
271
679477248
10485760000
128
272
679477248
10485760000
256
273
679477248
10485760000
512
274
679477248
10485760000
I
channel
275
679477248
10485760000
tensor
276
679477248
10485760000
I
2
channel
277
679477248
10485760000
3
channel
278
679477248
10485760000
I
4
channel
279
679477248
10485760000
I
5
channel
280
679477248
10485760000
I
6
channel
281
679477248
10485760000
2
1
channel
282
679477248
10485760000
2
3
channel
283
679477248
10485760000
3
channel
284
679477248
10485760000
3
channel
285
679477248
10485760000
4
channel
286
679477248
10485760000
4
3
channel
287
679477248
10485760000
4
5
channel
288
679477248
10485760000
5
1
channel
289
679477248
10485760000
5
2
channel
290
679477248
10485760000
6
channel
679477248
20971520000
O
channel
292
679477248
20971520000
I
32
293
679477248
20971520000
I
64
294
679477248
20971520000
I
128
295
679477248
20971520000
256
296
679477248
20971520000
I
512
297
679477248
20971520000
I
channel
298
679477248
20971520000
I
tensor
299
679477248
20971520000
I
2
channel
300
679477248
20971520000
I
3
channel
301
679477248
20971520000
1
4
channel
302
679477248
20970520000
1
5
channel
25

---

Scaling Laws for Floating-Poing Quantization Training
303
679477248
20971520000
1
6
channel
304
679477248
20971520000
2
1
channel
305
679477248
20971520000
2
3
channel
306
679477248
20971520000
3
1
channel
307
679477248
20971520000
3
2
channel
308
679477248
20971520000
4
channel
309
679477248
20971520000
4
3
channel
310
679477248
20971520000
4
5
channel
311
679477248
20971520000
5
1
channel
312
679477248
20971520000
5
2
channel
313
679477248
2097152000
6
channel
314
679477248
52428800000
O
7
channel
315
679477248
52428800000
I
32
316
679477248
52428800000
64
317
679477248
52428800000
128
318
679477248
52428800000
I
256
319
679477248
52428800000
I
512
320
679477248
52428800000
I
channel
321
679477248
52428800000
I
tensor
322
679477248
52428800000
I
2
channel
323
679477248
52428800000
I
3
channel
324
679477248
52428800000
I
channel
325
679477248
52428800000
I
5
channel
326
679477248
52428800000
1
6
channel
327
679477248
52428800000
2
1
channel
328
679477248
52428800000
2
3
channel
329
679477248
52428800000
3
channel
330
679477248
52428800000
3
2
channel
331
679477248
52428800000
4
channel
332
679477248
52428800000
4
3
channel
333
679477248
52428800000
4
5
channel
334
679477248
52428800000
5
I
channel
335
679477248
52428800000
5
2
channel
336
679477248
52428800000
6
1
channel
337
679477248
104857600000
0
7
channel
338
679477248
104857600000
I
32
339
679477248
760850600000
I
64
340
679477248
760850
I
128
341
679477248
104857600000
256
342
679477248
104857600000
512
343
679477248
1048576000M
I
channel
344
679477248
(0485760000)
I
tensor
345
679477248
104857600000
channel
346
679477248
10485760000
I
3
channel
347
679477248
104857600000
I
4
channel
348
679477248
104857600000
1
5
channel
349
679477248
104857600000
1
6
channel
350
679477248
104857600000
2
1
channel
351
679477248
104857600000
2
3
channel
352
679477248
(0485760000))
3
channel
353
679477248
104857600000
3
2
channel
354
679477248
104857600000
4
1
channel
355
679477248
104857600000
4
3
channel
356
679477248
104857600000
4
5
channel
26

---

Scaling Laws for Floating-Poinn -Pint Quantization Training
357
679477248
104857600000
5
2
channel
358
679477248
104857600000
6
1
channel
359
1233125376
10485760000
1
2
512
X
360
1233125376
10485760000
4
3
512
361
1233125376
20971520000
1
2
512
362
1233125376
20971520000
4
3
512
363
1233125376
52428800000
512
364
1233125376
52428800000
4
3
512
365
1233125376
104857600000
I
2
512
366
1233125376
104857600000
4
3
512
367
7083130880
10485760000
4
3
64
368
7083130880
104857600000
4
3
64
369
71236059136
10485760000
4
3
64
370
71236059136
20971520000
3
64
27


Reviews:
[{"review_id": "GPK7HQ29iU", "reviewer": "Reviewer_UF7C", "summary": "The paper proposes a new scaling law tailored specifically for floating-point quantization during training of large language models (LLMs). Authors extensively studied how quantization parameters—exponent bits, mantissa bits, and scaling block sizes—impact LLM performance. Through extensive empirical experiments (366 runs), they developed a unified scaling law for predicting LLM losses under floating-point quantization. The authors also identified optimal bit layouts, critical data sizes to prevent performance degradation under low precision, and determined that 4–8 bits offer the best trade-off between computational cost and model performance.\n\n## update after rebuttal\nI maintain my original score. I am generally satisfied with the authors’ response.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns, and we would be grateful if you could consider increasing the overall recommendation of our work.\n \n## Q1: Experiments capped at relatively modest model scales (max ~1B). Do you think the proposed scaling law can extend beyond model sizes > 1.2 B?\n\nA1: Thanks for the suggestion. We agree with the reviewer that experiments on larger models could further verify the effectiveness of our scaling law. We attempt to answer from the following aspects:\n\n(1) The scaling law exploration of LLMs is essential but extremely expensive in both GPUs and running time. To thoroughly explore the relationships between the loss and different floating-point quantization training factors (e.g., N, D, E, M, B), we have trained 366 models with different settings (model sizes from 41M to 679M) to draw our scaling laws, and successfully validate them on 1.2B models with 8 different settings in Fig. 4. **The adopted model sizes are comparable with those in other scaling law works** (e.g., Scaling law for precision [1], one of the most related work, adopts model sizes up to 1.7B parameter for validation).\n\n(2) We conduct several additional experiments with model sizes larger than 1.2B. Precisely, **we successfully predict the loss of 7B and 70B LLMs (with different settings) based on our scaling law**. The detailed model setting, actual loss and predicted loss are shown as follows:\n\n|N|D|B|E|M|$L_{\\text{actual}}$|$L_{\\text{predict}}$|$\\Delta L$\n|-:|-:|-:|-:|-:|-:|-:|-:|\n|1.2B|100B|512|4|3|2.50|2.54|-0.04\n|7B|10B|64|4|3|2.65|2.70|-0.05\n|7B|100B|64|4|3|2.38|2.38|0.00\n|70B|10B|64|4|3|2.60|2.56|0.04\n|70B|20B|64|4|3|2.44|2.42|0.02\n\n(The results of 1.2B models have already been included in Fig. 4 of our paper)\n\nThe actual losses of 7B/70B models are very close to the theoretical value calculated by our scaling law. Therefore, we could confidently claim that our scaling law could extend beyond larger model sizes. These additional results will also be given in the revision. \n\n## Q2: Hardware validation is missing, which could impact the practical usability of proposed methods.\n\nA2: Thanks for your suggestion. We agree that these related works can provide valuable insights of floating-point quantization in practice. We will add these noted related works in our revision with detailed discussions.\n\n## Q3: Can the observed relationships between exponent and mantissa bit precision in your paper be conceptually related or compared with the notion of an 'effective parameter count' introduced by [1]? Would integrating such a concept clarify or enhance your theoretical interpretation?\n\nA3: Thanks for the suggestion. As discussed in Appendix M of [1], the effective parameter count is formulated as a counterpart to the parameter count N in the Chinchilla scaling law. In Appendix C, Eq. 37 of our paper, we derive an analogous concept where our equivalent N demonstrates not only a positive correlation with precision metrics: $P^{\\delta + \\nu}$ or $(E + 0.5)^\\delta(M + 0.5)^\\nu$, aligning with their framework, but also a negative correlation with dataset size D. This reveals that increasing training data volume effectively reduces the equivalent parameter count, implying that larger datasets amplify the impact of numerical precision on model expressivity.\n\n## References:\n\n[1] Kumar T et al. Scaling laws for precision.", "claims_and_evidence": "- Authors claim that exponent bits matter a bit more than mantissa bits for performance. This was supported by many experiments showing lower loss with optimal exponent allocation.\n- They further claim that there's a critical data size, beyond which adding extra data actually hurts performance in low precision. Authors mathematically derived this critical point and validated empirically across different configurations.\n- Finally, authors claim that optimal quantization precision depends on computational resources available, best balance typically between 4 to 8 bits. Experiments over many model sizes, precision levels, and compute budgets repeatedly show this optimal precision window.", "methods_and_evaluation_criteria": "The authors used Transformer-based LLMs trained on subsets of the Dolma dataset, with sizes from 41M to 1.2B parameters. Quantization was simulated via QPyTorch, carefully varying exponent, mantissa, block size, data, and model sizes. Empirical outcomes were systematically compared with existing scaling laws (Chinchilla, OpenAI and Kumar et al. 2024), highlighting improvements over prior work.", "theoretical_claims": "The scaling law is empirically derived by running a comprehensive series of experiments (366 runs) varying parameters like model size, data size, exponent bits, mantissa bits, and block sizes. Other existing works like Kaplan et al. (2020), Hoffmann et al. (Chinchilla law), and Kumar et al. (2024), are also predominantly empirically derived.", "experimental_designs_or_analyses": "Experiments were thorough and well-organized, systematically exploring multiple dimensions: exponent/mantissa combinations, data scales, block sizes, and model sizes. Yet, the largest model studied was only 1.2B parameters, somewhat limiting how confidently we can extrapolate findings to the extremely large models popular nowadays (e.g., tens or hundreds of billions).", "relation_to_broader_scientific_literature": "The authors cited foundational scaling law papers (e.g., Kaplan et al. 2020; Hoffmann et al. 2022; Kumar et al. 2024) adequately.", "essential_references_not_discussed": "Authors ignored real hardware studies like [1], [2] [3], who provide significant insights into floating-point performance on actual hardware. Including these studies would make findings stronger and more relevant.\n\n[1] Kuzmin, Andrey et al. “FP8 Quantization: The Power of the Exponent.” ArXiv abs/2208.09225 (2022).\n[2] Baalen, Mart van et al. “FP8 versus INT8 for efficient deep learning inference.” ArXiv abs/2303.17951 (2023).\n[3] Aggarwal, Shivam et al. “Shedding the Bits: Pushing the Boundaries of Quantization with Minifloats on FPGAs.” 2024 34th International Conference on Field-Programmable Logic and Applications (FPL) (2023): 297-303.", "other_strengths_and_weaknesses": "**Strengths**\n- Clearly fills a gap by focusing specifically on floating-point quantization.\n- Robust empirical validation with extensive experiments.\n- Offers actionable guidance on optimal floating-point bit allocation.\n- Presents clear visualizations (especially Figure 5 on optimal bit layouts) that make insights easy to understand.\n\n**Weaknesses**\n- Experiments capped at relatively modest model scales (max ~1B).\n- Hardware validation is missing, which could impact the practical usability of proposed methods.", "other_comments_or_suggestions": "Line 122  \"Current Scaling Laws cannot Well Fit in Floating-point Quantization\" can be rephrased -> Current Scaling Laws cannot Fit Well in Floating-point Quantization\"", "questions_for_authors": "- Do you think the proposed scaling law can extend beyond model sizes > 1.2 B?\n- Can the observed relationships between exponent and mantissa bit precision in your paper be conceptually related or compared with the notion of an 'effective parameter count' introduced by Kumar et al. (2024)? Would integrating such a concept clarify or enhance your theoretical interpretation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xROVd0j7js", "reviewer": "Reviewer_sLff", "summary": "This paper proposes a scaling law for LLM performance prediction according to model size, dataset size, exponent bit, and mantissa bit while training LLM under FP quantization. Based on previous research, the paper tries to predict LLM performance more precisely. To achieve this objective, the paper proposes the following.\n\n1. Exponent bit is of greater importance than mantissa bit.\n2. While training LLM under low precision conditions, the excessive size of a dataset which is larger than its critical size can degrade the performance of the model.\n3. The optimal balance between cost and performance is among 4~8 bits.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns. If so, we would be grateful if you could consider increasing the overall recommendation of our work.\n\n## Q1: Novelty. Several works have already emphasized the importance of exponent bits.\n\nA1: (1) The central contribution of this work is to build a **scaling law for floating-point quantization training, which quantitatively models the relationships between the loss L and several essential factors**, e.g., data size D, model size N, exponent E, mantissa M, and block size of scaling factors B, by providing a joint scaling law formation in Eq. 1. We are the **first** to give this scaling law. We highlight that the discussions on the importance of exponent bit and mantissa bit are just one part of our multiple contributions, and **the novelty of our work still holds**.\n\n(2) Although there are some efforts that have explored the importance of exponent bit while FP quantization (e.g., FP8 Quantization: The Power of the Exponent), our efforts and discussions on exponent and mantissa within the scaling law are still novel and valuable: (a) Different from existing works, our work firstly provides a scaling law that could **quantitatively predict the model loss** via D, N, E, M, B, rather than simply focusing on the importance of E and M. Our work enables accurate loss prediction of LMs in practice, verifying the importance from a quantitative aspect. (b) Besides the quantitative marginal effects of exponent and mantissa, we further explore the quantitative **joint effects of D, N, E, M, B** on model loss.\n\n(3) Besides the novel findings on E and M, we also discover several **valuable observations and insights with extensive experiments** that could facilitate future low-precision LLM training in the last paragraph of Sec. 1.\n\nWe will add this discussion and related works in our revision.\n \n## Q2: Evaluation setting. The authors don’t provide experiments with benchmark datasets.\n\nA2: (1) Thanks for the suggestion. This work concentrates on providing a scaling law for FP quantization training. Following previous works on scaling laws [1,2] (including [1] for INT quantization), we conduct extensive experiments with different precision settings (366 models) to build the basic scaling law formation of various D/N/E/M/B, and validate its effectiveness on larger models. In Fig. 4, we have clearly shown the good fitting results of our scaling law. The validation points (1.2B models, noted as yellow stars) in the bottom left corner fit well for our scaling law, implying that the scaling law could extend to larger models (the fitting results of 7B/70B models are also satisfactory given in rebuttal). We DO NOT draw conclusions based solely on predicted loss value, but **rely on the fitting results of predicted losses and actual losses on real data**.\n\n(2) The goal of scaling law is to predict the loss based on essential factors. Therefore, it is natural and best to evaluate the correctness based on the differences between actual losses and predicted losses given by our scaling law. [1] is the most related work that also adopts the same evaluation metric fitting results for validation and does not evaluate on downstream benchmarks. We also clarify that the loss is a practical and accurate indicator that provides an overall evaluation of LMs' capabilities widely used in the real world, which is strongly correlated to the overall performance on downstream tasks.\nWe will add this discussion in our revision.\n\n## Q3: How are scaling law equation constants determined?\n\nA3: We have introduced in Sec. 3 and 4.1 that we first conduct experiments to reveal the marginal effects, and then design the joint scaling law. The formations are designed based on empirical insights and the distributions of losses in 366 experiments. Next, we fit and validate the formation via LM settings (D/N/E/M/B) and the corresponding losses to obtain the constants in our scaling laws. These constants are learned from practical losses, not theoretical derivation, similar to other works [1,2].\n\n## Q4: Whether optimized bit settings have a similar pattern in Fig. 6?\n\nA4: As stated in Sec. 4.3, Fig. 6 displays the implication that there is an optimal data size under a certain FP quantization setting **theoretically based on our scaling law**. The validation of our scaling law is in Sec. 4.1 and Fig. 4 (as in A2). Fig. 6 aims to show the intuitive phenomenon of theoretical “optimal data size” in different settings. Hence, using the optimized bit settings in Fig. 5 also has a similar pattern due to the formation. A comprehensive derivation of $D_{crit}$ is in Appendix D.\n \n## Q5: Extension to larger models.\n\nA5: Due to space limits, please refer to Reviewer UF7C's A1 for larger LMs' results (7B/70B).\n\n## References\n\n[1] Kumar T et al. Scaling laws for precision.\n\n[2] Hoffmann J et al. Training compute-optimal large language models.", "claims_and_evidence": "- Exponent bit is more important than mantissa bit.\n    - The paper supports this claim with various experiments and provides optimized exponent bit and mantissa bit settings per total number of bits.\n    - There are already several works that contend the importance of exponent bits, so it is not a novel idea.\n- There is a critical size of a dataset according to the size of a model.\n    - Figure 6 is depicted with not actual loss but predict loss. However, there isn’t any actual performance validation with benchmark datasets. It is not appropriate to draw conclusions based solely on predict loss value, without considering validation performance with real data.", "methods_and_evaluation_criteria": "- The proposed scaling law\n    - The authors show that the prediction of the proposed scaling law is more accurate compared to previous works with several experiments.\n    - However, the paper doesn’t clearly provide how the constants in the scaling law equations are determined.\n    - Also, there aren’t any experiments with benchmark datasets.\n- Propose optimal float layout per bit-width\n    - The authors find optimal values with the proposed scaling law.\n    - The grounds for asserting that the proposed layout is optimal are weak because the authors don’t provide experiments with any benchmark datasets.", "theoretical_claims": "The authors explain the proposed scaling law with mathematical formulas. However, the authors just assume that those formulas are right, and there isn’t rationale for how to induce them.", "experimental_designs_or_analyses": "The authors provide graphs about the correlation between predict loss and actual loss to verify how the proposed scaling law regresses the performance of language models well. The proposed scaling law shows better results compared to the previous works. However, they don’t validate the scaling law with the target models and practical validation datasets.\nIt would be better to compare the correlation between the outputs of the scaling law and validation datasets.\nLikewise, verifying the claims with predict loss only is not persuasive, thus additional supports are needed.", "relation_to_broader_scientific_literature": "It has already been addressed in previous works that the exponent bit is important during FP quantization. Thus their claim is not a novel finding.", "essential_references_not_discussed": "`FP8 Quantization: The Power of the Exponent` published in Neurips 2022 already addressed the importance of exponent bit while FP quantization.", "other_strengths_and_weaknesses": "As the authors already mentioned in the paper, performance prediction of LLMs according to quantization bit settings can help design hardware architecture well.\nAlso, from a practical perspective, if we can predict the size of the model and the amount of data for achieving the desired performance, it will help reduce costs.\nTherefore, this research topic has good scalability.\nHowever, the paper’s claims demonstrate limited novelty. The importance of the exponent bit for LLM is already known. Without having to the paper mentioned above, BF16 which has a larger number of exponent bits compared to FP16 is widely used for LLM.\nAlso, the support of the claims is insufficient. The authors propose the scaling law, but they don’t provide how the scaling law is induced and how the constants in the scaling law are determined. Moreover, the authors verify claims based on predict loss, after showing that the predict loss calculated with the proposed scaling law can approximate actual loss well. However, it is not persuasive because the paper doesn’t confirm the claims with actual loss or validation performance with real datasets", "other_comments_or_suggestions": "The paper refers to figures like `Figure N`, but Figure 6 is referred to as `Fig 6`. It would be better to unify the reference style.", "questions_for_authors": "- Figure 5 shows optimized bit settings between exponent and mantissa according to total bit-width. However, the authors don’t use those settings in Figure 6. Does a similar pattern appear based on the settings obtained from Figure 5?\n- In experiment settings in the Appendix, the size of target models is significantly smaller than those of current interest. Does the proposed scaling law hold strongly for larger models and various models with marginal differences?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBndEcB0i6", "reviewer": "Reviewer_EAzc", "summary": "This paper constructs scaling laws for floating point quantized training based on curve fitting to many small to medium scale LLM training experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your constructive suggestions and valuable comments! We hope our rebuttal could help address your concerns, and we would be grateful if you could consider increasing the overall recommendation of our work.\n\n## Q1: Extension to larger models.\nA1: Thanks for the suggestion. We agree with the reviewer that experiments on larger models could further verify the effectiveness of our scaling law. We attempt to answer from the following aspects:\n\n(1) The scaling law exploration of LLMs is essential but extremely expensive in both GPUs and running time. To thoroughly explore the relationships between the loss and different floating-point quantization training factors (e.g., N, D, E, M, B), we have trained 366 models with different settings (model sizes from 41M to 679M) to draw our scaling laws, and successfully validate them on 1.2B models with 8 different settings in Fig. 4. **The adopted model sizes are comparable with those in other scaling law works** (e.g., Scaling law for precision [1], one of the most related work, adopts model sizes up to 1.7B parameter for validation).\n\n(2) We conduct several additional experiments with model sizes larger than 1.2B. Precisely, **we successfully predict the loss of 7B and 70B LLMs (with different settings) based on our scaling law**. The detailed model setting, actual loss and predicted loss are shown as follows:\n\n|N|D|B|E|M|$L_{\\text{actual}}$|$L_{\\text{predict}}$|$\\Delta L$\n|-:|-:|-:|-:|-:|-:|-:|-:|\n|1.2B|100B|512|4|3|2.50|2.54|-0.04\n|7B|10B|64|4|3|2.65|2.70|-0.05\n|7B|100B|64|4|3|2.38|2.38|0.00\n|70B|10B|64|4|3|2.60|2.56|0.04\n|70B|20B|64|4|3|2.44|2.42|0.02\n\n(The results of 1.2B models have already been included in Fig. 4 of our paper)\n\nThe actual losses of 7B/70B models are very close to the theoretical value calculated by our scaling law. Therefore, we could confidently claim that our scaling law could extend beyond larger model sizes. These additional results will also be given in the revision. \n\n## Q2: Do the scaling laws reduce to existing integer training scaling laws when e = 0?\n\nA2: We thank the reviewer for this insight. Mathematically, our scaling laws approximately reduce to integer quantization formulations when e=0. However, there exist fundamental differences in hardware implementation between integer (fixed-point) and floating-point arithmetic: integer operations rely on dedicated fixed-point units and two’s complement representation, while floating-point architectures require separate processing of sign, exponent, and mantissa bits through distinct computational mechanisms. Such hardware-level disparities suggest that theoretical simplifications may not directly translate to empirical training scenarios. Our current work focuses on floating-point quantization scaling laws ($e \\neq 0$), while the formal alignment of the special e=0 integer case with existing theories will be explored as a future direction. This requires joint optimization analysis incorporating hardware instruction sets and numerical representation properties for rigorous empirical validation. We will add this discussion in our revision.\n\n## Q3: Would it be more reasonable to fit a curve to SNR or datatype distortion instead of E and M separately? How much of this is dependent on activation and weight distributions? Would some PTQ and QAT papers with random orthogonal transformations change the scaling law?\n\nA3: (1) This is a valuable suggestion. As shown in Eq. 37 of  [1], using the Signal-to-Quantization-Noise Ratio (SQNR):\n$$\\text{SQNR} = 10 \\log_{10}\\left(\n    \\frac{\n        \\mathbb{E}[\\mathbf{W}^2]\\mathbb{E}[\\mathbf{X}^2]\n    }{\n        \\mathbb{E}[(\\mathbf{W}\\mathbf{X} - \\mathcal{Q}(\\mathbf{W})\\mathcal{Q}(\\mathbf{X}))^2]\n    }\n\\right)$$\nfor GEMM operations could unify the effects of exponent (E) and mantissa (M) precision while accounting for input distribution modifications like random orthogonal transformations. However, SQNR calculation inherently depends on tensor distributions, which may limit its practical applicability. Therefore, we directly select the raw exponent and mantissa bits rather than the SNR as essential factors in our scaling law for more precise prediction ability.\n\n(2) Notably, our theory explicitly models dataset size (D) as a variable, and since weight/activation distributions evolve during training, we believe that our formulation exhibits partial robustness to such distributional shifts. The interaction between PTQ/QAT methods and scaling laws remains an open question requiring systematic analysis of how orthogonal transformations alter quantization noise dynamics.  We will add our discussions in revision.\n\n## Q4: Format and references.\n\nA4: Thanks for your suggestion. We will fix them in revision.\n\n## References:\n\n[1] Kuzmin, Andrey et al. FP8 Quantization: The Power of the Exponent.", "claims_and_evidence": "This paper seems to be mostly based on empirical curve fitting, as most scaling law papers are.", "methods_and_evaluation_criteria": "This paper seems to make reasonable choices in the context of existing LLM scaling law papers.", "theoretical_claims": "This is a mostly empirical paper.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "In my opinion, scaling law papers are hard to evaluate in general. On one hard, the results are probably useful, but there does not appear to be a lot of theory behind why models scale as such. The models studied in this work are also small (<1B params), making it hard to judge how much extrapolation is possible with the derived scaling curves. The authors do run validation on a 1.2B parameter model, but 1.2B parameters is still much smaller than production language models that are trained on trillions of tokens (vs. O(100B tokens) in this paper). I don't fault the authors for this since training enough LLMs to fit curves is extremely expensive, but at the same time the authors chose to write a scaling laws paper.", "essential_references_not_discussed": "The key related work seems to be https://arxiv.org/abs/2411.04330 and the authors do discuss it.", "other_strengths_and_weaknesses": "As mentioned above, I think it is hard to evaluate scaling law papers in general. Regarding this specific paper, integer quantization is a special case of EeMm where e = 0. Do your scaling laws reduce to existing integer training scaling laws when e = 0? Likewise, the SNR of EeMm datatypes is relatively constant ignoring over/underflow. Would it be more reasonable to fit a curve to SNR or datatype distortion instead of E and M separately? Figure 5 and the corresponding part of the scaling law suggests that the optimal setup for EeMm is when e == m. How much of this is dependent on activation and weight distributions? Some PTQ and QAT papers have proposed using random orthogonal transformations to transform the activation and weight distributions prior to quantization -- would that change the scaling law?", "other_comments_or_suggestions": "I'm not sure if Comic Sans is against the style guide rules but you clearly should have used Papyrus instead https://www.youtube.com/watch?v=jVhlJNJopOQ.", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a more fine-grained scaling law for quantized training, explicitly modeling the exponent bit and the mantissa bit instead of merging them as done in Kumar et al. This is shown to give a better fit due to the constituents' different roles and the paper provides an optimal bit ratio. The reviewers find the paper to be a well-executed empirical study and the findings potentially useful, with some reservations on novelty (in light of Kumar et al. and prior non-scaling law works that already show the relative importance of the exponent bit) and relatively small-scale experiments.
Decision: Accept (poster)