Scaling Laws for Precision
Authors: Tanishq Kumar
Zachary Ankner
Benjamin Frederick Spector
Blake Bordelon
Niklas Muennighoff
Mansheej Paul
Cengiz Pehlevan
Christopher Re
Aditi Raghunathan
Venue: ICLR 2025 Oral (2025)
Keywords: quantization
scaling laws
precision
language models
Primary area: foundation or frontier models, including LLMs
TL;DR: We model the effects of precision on language model loss scaling, both during and after training. We find that overtrained models degrade more when quantized at inference time, and that training larger models in lower precision can be optimal.

Abstract: Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's "effective parameter count," allowing us to predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventually making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss of a model with different parts in different precisions, and suggest that training larger models in lower precision can be compute optimal.  We unify the scaling laws for post and pretraining quantization to arrive at a single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions on model sizes up to 1.7B parameters trained on up to 26B tokens.

Published as a conference paper at ICLR 2025
SCALING LAWS FOR PRECISION
Tanishq Kumar*
Zachary Ankner' 34
Benjamin F. Spector Blake Bordelon'
Niklas Muennighoff
Mansheej Paul
Cengiz Pehlevan'
Christopher R
Aditi Raghunathan
Harvard University MIT *Databricks
Stanford University *Carnegie Mellon University
ABSTRACT
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware" scaling laws for both training and inference. We propose that training in lower precision reduces the model's effective parameter count, allow- ing us predict the additional loss incurred from training in low precision and post-train quantization. For inference, we find that the degradation introduced by post-training quantization increases as models are trained on more data, eventu- ally making additional pretraining data actively harmful. For training, our scaling laws allow us to predict the loss a model with different parts different preci- sions, and suggest that training larger models in lower precision may be compute optimal. We unify the scaling laws for post and pretraining quantization to arrive at single functional form that predicts degradation from training and inference in varied precisions. We fit on over 465 pretraining runs and validate our predictions model sizes up 1.7B parameters trained up 26B tokens.
INTRODUCTION
Scale has emerged as a central driver of progress in deep learning (Brown et al. 2020). Key work on scaling (Kaplan et al., 2020, Hoffmann et al. 2022) studied tradeoffs between model/dataset size to balance performance and compute. However, the precision in which models are trained and served is an important third factor that contributes both cost and performance. Deep learning is trending towards lower precision: current frontier models like the Llama-3 series are trained in BF16 (Dubey et al., 2024), and there is widespread effort to move the pretraining paradigm to FP8 Micikevicius al. 2022). The next generation of hardware will support FP4, and advances in weight-only quantization have led to training in binary and ternary at scale (Ma et al. 2024 How far will these paradigms go? Specifically, ask:
Wang
al. 2023
What are the tradeoff's between precision, parameters, and data? How do they compare for pretraining and inference?
Studying scaling in precision is challenging because work on scaling laws generally aims to drop fine-grained implementation details in pursuit of universal functional forms while work on quantiza- tion generally does the opposite, focuses on the details: how quantization is done, with what type, to what part of the model. In seeking balance, we consider variety of plausible functional forms, and choose one that abstracts implementation details of quantization away from loss scaling, allowing us to predict loss scaling in many situations of practical interest. This functional form that posits bit precision and parameter count interchangeably contribute to model's "effective parameter count," Non. and implementation details like which parts of model are quantized to what precision, interact with loss scaling only through their effect this quantity.
Overall, we study the scaling of the effects of precision on loss as we vary data and parameters, both during and after training. We first study how the degradation induced by post-train quantiza- tion scales with parameters and data. We find that the degradation increases with data, so that for
Equul contribution. Correspondence kuma @co1 lege. aag..harvrrd.ed.

---

Published as a conference paper at ICLR 2025
Scaling: Post-Train Quantization
Scaling: Quantized Training
Training models precision can be compute optimal
Val Loss (Post-Oun (ast-Quant)
Final Val Loss
More pretralninng compute
INT3
inference
2.997
3,009
3.057
INTS
IN16
PTQ
100
1906
FF4 (1.760)
FPG (1-1783
FPR (BROM)
BF16 (445M)
FP32 (720M)
Teken/Parametor Ratio
Training Precision (Model Size)
Figure 1: Schematic of key findings. (Left) Training a fixed model size to various data budgets in BF16 and quantizing weights at the end. We find that degradation due to post-train quantization increases with tokens seen during pretraining, that eventually additional pretraining data can be harmful. (Right) Our scaling suggests training larger models in lower precision can be compute- optimal according to the cost model in Section 4.3 Weights, activations, attention quantized, all models trained on the same data budget, details in Appendix
fixed model, training on additional data after certain point can be actively harmful if the model will be quantized after training. We then shift our focus to quantized training, examining both the quantization-awer -a--a--war--ttnnngg (weights only) and low-precision training (weights, activations, tention all quantized) settings. Our scaling laws for pretraining suggest that the compute-optimal pretraining precision is in general independent of compute budget. Surprisingly, however, this inde- pendence ceases to be true if model size is constrained, in which case the compute-optimal precision grows slowly in compute.
In all, pretrain suite of 465 language models in 3 16 bit precisions, well post-train quantize each to multiple precisions. For language model with N parameters, trained on D tokens with training precision Purin. and post-train weight precision Ppusa. we ultimately find unified scaling law that takes the following form:
L(N, Panins Prost)
AN
+BD + (Netr, D, Prrnin P Ppost)
Training time Ellects
Post-Training Ellects
Usual Chinchilla
where A, B, are positive fitted constants, and ôpTQ refers the loss degradation induced by post-training quantization before inference. Altogether, our results for post-train quantization illus- trate how more pretraining FLOPs do not always lead to better models at inference-time, and our results for low-precision pretraining suggest that both the standard practice training models in 16-bit, and the race to extremely low (sub 4-bit) pretraining precision, may be suboptimal.
2 BACKGROUND, RELATED WORK, AND SETUP
Notation. Throughout, D denotes dataset size in tokens and N denotes model size in parameters. Pw, Pa, PLy refer to the bit precision, in integer-type, of the weights, activations, and key-value cache ("attention") durring training, and pust refers to the precision we post-train quantize (PTQ) weights to at the end for model inference. When P Punin is used without reference to part of the model, all three model parts are tied to the same precision. The inference-ting by post-train quantization will be denoted (N, D, Punins Ppoa), and it is defined as the change in loss from performing post-training quantization compared to the end of pretraining. We use "high precision" to mean 16-bit above.
loss degradation induced
We study KV, rather than QKV, because understanding scaling in the KV cache alone is important for many inference settings. For pretraining claims in Section 4.3 we quantize the entire attention computation, including queries, finding additionally quantizing the query vectors makes negligible difference to scaling.
2

---

Published as a conference paper at ICLR 2025
2.1 QUANTIZATION FUNDAMENTALS, How, WHAT, WHEN
The Problem: Compute vs Memory-Bound Workloads. Most deep learning workloads are bot- tlenecked by either compute, in the form of matrix multiplications, or memory bandwidth, in the form of data movement between different parts of the GPU. Different types of workloads have dif- ferent bottlenecks: most time is spent doing large matrix multiplications during pretraining, so is compute-bound in contrast, small-batch inference is bandwidth-boog bound by model weights; long- sequence decoding is bandwidth bound by KV cache, etc. This motivates studying scaling in the training precision of the (weights, activations, KV cache) both in isolation and in combination.
Quantization: How. Quantization of an operation typically refers to rounding of values in matrices involved in some computation the forward or backward pass, depending on what is quantized, and when. Quantization is usually done to integer or floating-pont type.
Quantization: What. Only weights. "Quantization-avar" training" Quantizing only weights dur- ing training does not offer any compute savings because matrix multiplications are still done in high precision. However, this is commonly done allow weights to adapt to low precision so they can be served at very low precision at inference-time, thereby alleviating memory bottlenecks (Ma et al.) 2024 Wang et al. 2003). We will refer this as "quantization-awarre aware-training" and defer additional discussion to Appendix D
Weights, activations, attention. "Low-precision training" Quantizing and activations and attention in addition to weights allows for compute gains because matrix multiplications can be done in low precision (if the hardware supports it) since everything is in the same precision. We will refer to this setting as "low-precision training" to distinguish it from quantization-aware training.
Quantization: When. Quantization can be done during or after training. In practice, when seek- ing to reduce inference-time memory costs, one first attempts post-train quantization. If that de- grades the model too much, arantization-aware-rree
used. Post-train quantization is typically
only applied to model weights (Frantar et al. 2022 Dettmers et al. 2022 Lin et al. 2023 Xiao
jet al., 2023). To reduce pretraining costs, low-pre Low-precision-e precision-training is needed. We will study scal- ing laws for post-training quantization in Section B for quantized training in Section 4 (examining
both quantization-awer aware training and low precision training) and unify the two in Section 5 The numerical values of all our fitted constants can be found in Appendix K
2.2 SCALING LAWS AND PARAMETRIC FITS
Scaling Laws. Hoffmann et al. 2022 model loss scaling using the functional form L(N, D) = AN- + BD + E where A, B, a, B, E are positive fitted constants, finding that data and param- eters should be scaled in roughly equal proportion as more compute becomes available. We will refer to the scaling (Hoffmann al. 2022) as "Chinchilla-optimaa"" or just "Chinchilla" and note this often used colloquially D/N x 20 being pretraining compute-optimal On the theoretical front, work on scaling laws (Bahri et al., 2024; Bordelon et al. 2024| Lin et al.) 2024b) finds that noise to various parts of model or data affects loss in predictable way. While previous works have explored the scaling behavior of post-training quantization in terms of total model bits (Dettmers & Zettle- moyer 2023) and knowledge capacity (Allen-Zhu
Li (2024), we focus instead on data scaling. We note that in general the exact fitted values of all coefficients and exponents can vary drastically based on small implementation differences: [Besiroguu et al. (2024) find different constants when attempting to replicate (Hoffmann et al. 2022). Sardana & Frankle (2023) fit coefficients A, B of different orders of magnitude. For this reason, we emphasize our contribution not the numerical values we fit, but the trends and functional forms we identify.
Overtraining. In practice, accounting for inference costs means training smaller models for sub- stantially longer than Chinchilla-optimaa (Sardana & Frankle, 2023 Gadre et al.) [024]. For in- stance, Llama-3-8B is trained to D D/N R3 2000 (Dubey et al. 2024 and the Gemma-2 series up to D/N > 1000 (Team et al.
2024). We refer to such models as "overtrained" in this paper, with
the token/parameter ratio D/N being a key quantity throughout. Work on inference- cetrennce-timeeeee compute
2024 and on synthetic and multimodal data (Yang et al. 2024| Fan suggests future models may be even more overtrained. Therefore, modern work scale must consider ratios much larger than Chinchilla-optimal, and in this work
(Snell et al. 2024 Brown al.
cet al., 2024; Bauer et al.) 2024)
3

---

Published as a conference paper at ICLR 2025
N= 30M
N 60M
N 110M
N 220M
Val Loss (Post- Qutt-Qua)))
INT6
4.25
INTS
4.00
INT4
INT3
3.75
No PTQ
3.50
3.25
Degradation, GETO
10-1
10-7
10-2
100
1000
100 10
100
10
Token/Parameter Ratio
Figure 2: Loss degradation from PTQ increases with data. Top row is loss after PTQ, bottom row loss degradation compared to end of training, before PTQ. The top row is thus the gray line in each plot plus the corresponding value in the bottom row. We can see that degradation grows with data, bottom row is fitted with Equation 2 For D/N sufficiently large (left), loss can increase in data. Even at lower D/N, where post-quant loss continues to decrease with data, the value of data reduced compare to the baseline. R² = 0.97 over fitted points (bottom row).
we perform experiments up to D/N ~ 10 and analyze the predictions found by our scaling law for up to D/N z 10a. See Appendix for additional related work.
2.3
SETUP
We train and evaluate suite of OLMo-style models the Dolma V1.7 dataset (Groeneveld al.
2024 Soldaini et al. 2024), using a standard Transformer++ implementation; see Appendix A for hyperparameters and ablations. Our experiments consist of sweep of language model pretraining runs over N e [30, 60, 110, 220] million parameters (non-embedding) and D E [1.5,3, 6, 13, 26 billion tokens. Our model sizes are relatively small because we train up to very high D/N 10 to study data scaling and set off over 20 runs at every (N, D): we sweep 8 values of precision for each of the (weights, activations, attention).
SCALING LAWS FOR POST-TRAIN QUANTIZATION
The easiest and most common quantization technique post-train quantizing a model -et-the-shlff (Chee et al.) 2024 Huang et al. [0244] Dettmers et al. 2022 Lin et al.] 2023 Xiao et al.) 2023). In this section, we consider models trained BF16 and use GPTQ (Frantar al. 2002) to post-train quantize them, replicating our findings with two other methods in Appendix [F] We quantify the resulting loss degradation ôpro- finding that post-train quantization scales poorly in data.
OVERTRAINED MODELS DEGRADE MORE WHEN POST-TRAIN QUANTIZED
We consider different model sizes (columns) trained various data budgets (x-axis of each plot) and plot in Figure both the loss after post-train quantization (top row) and the degradation incurred relative to end of training (bottom row). We find that the degradation ôpro increases in training data size across all model sizes, but that for a fixed dataset size larger models incur a smaller degradation. We additionally observe that ôPTQ increases exponentially as we decrease the precision we quantize to. Based on these observations we model ôpro as taking the form:
D
(N, D,
D.Pspss Ppost) = Cr
e Pppal "ippa
(2)
NYx
4

---

Published as a conference paper at ICLR 2025
where Cr, ND YN Poost are positive fitted constants. As find the fitted values of YD and " to be similar (see Appendix K for numerical values), we can think of this as an approximate power law in the token/parameter ratio D D/N. The intuition for this poor data scaling might be that models train on more data, they compress more information into their weights, so that perturbations to weights in the form of quantization more harmful loss, all else equal. We diseuss formal theoretical interpretations in Appendix H
This finding implies that for models that will be post-train quantized, there exists an amount of pretraining data beyond which additional data is actively harmful to performance at inference-time (see top-left, Figure 2 This can be defined as the point where additional data increases post-trai degradation more than it decreases loss during pretraining. We solve analytically for this critical data size in Appendix E as well analyze cost model for workloads where inference-cost is the primary concern. We thus summarize our first scaling finding as follows.
Finding Overtrained language models more sensitive to post-training quantization. For models trained in BF16 or above, we can model this loss degradation as
D NYn
ôPro N, D, Pposa) = Cr
e Ppesl "poas
where Cr, Yo, ,N, Tipost are positive fitted constants. This implies that when D/N is suffi- ciently large, Pp post sufficiently small, loss after quantization increase models pretrained for longer, as in Figure 2 We will revisit and modify Equation in Section to account for the effects of training in low-precision on òprQ-
SCALING LAWS FOR QUANTIZED TRAINING
In this section we study pretraining with weights, activations, and KV cache in various precisions. Importantly, only training precision, not test-time precision, is varied in this section; we discuss the interaction between train and test-time precision in Section 5 We sweep the training precisions of the weights, activations, and KV cache Pw. Pa. PL. E [3. 12] individually, as well as training BF16 baselines. We also pretrain models with arbitrary combinations of Pw,, Pa. P&v validate our scaling laws. To perform quantization during training, we quantize the forward pass in integer type unless otherwise noted, see Appendix for implementation details.
4.1 QUANTIZATION- WWA.ON---AAAR-RRRINNNNNE QUANTIZING WEIGHTS DURING TRAINING IIAS A CONSISTENT AND PREDICTABLE EFFECT
We first examine the trade-off between weight precision Pw and parameters N while holding P. Pvr fixed high precision. We fix D 13B tokens and perform grid sweep over combinations of N and Pa.. We plot the resulting IsoLoss contours where we linearly interpolate the final loss values in Figure 3 We observe that the bit precision the weights can be traded off for the number of parameters, i.e., a model with smaller N but larger Pw can achieve the same loss model with larger N but smaller Pv. Additionally, we find that the gains from increasing the bit precision of the weights are large at lower precisions but saturate at higher precisions (typically around 6-7 bits per weight).
In line with the empirical trends in Figure 3 we find the best fit for the tradeoff between weight precision and parameters is N the sensitivity of model weights (alternative fits explored in Appendix K). We therefore modify Chinchilla scaling to account for New by making the substitution N + Nea(N, Pw), giving the modified form:
where "w is fitted constant measuring
L(N, D) ()N,))=([((1 - ( BD -D- ++E
where we recall that A, B, E, a, 8 are fitted positive constants in the usual Chinchilla scaling form, and "w is a fitted constant we introduce. We plot the predictions of our fit compared to observed values in Figure for range of (N, D).
5

---

Published as a conference paper at ICLR 2025
Nus/V vs Precision
Empirical Isoloss Contours
Predicted Loss Contours
P4 (000
100
Precision (bits)
N (millions)
N (millions)
Figure 3: (Left) N from final scaling law. Our fit of Ner(NN, Pw) in this section is the first step towards this (blue). Empirical (center) and predicted (right) IsoLoss contours illustrating the tradeoff. Y-axis is weight precision during quantized training. All runs plotted trained on D = 13B tokens. Predictions from a fitted version of Equation 3 darker lines correspond to lower loss.
3.38 tokens
13.18 tokens
26.28 tokens
Final V Loss
Model Size
24:
18
16
60M
30M
110M
Pw (training precision, bits)
Pwi (training precision, bits)
Pw (training precision, bits)
Figure 4: Predicting final validation losses L(N, D, Pw)) for various N, D, Pw to test proposed functional form. Points are experimental values, lines are predictions of a single parametric fit of the form in Equation 3
We train only two model sizes 26B due to compute constraints.
4.2 PE--P-RRIN-TRRIIIIIII THE EFFECTS OF QUANTIZING WEIGHTS, ACTIVATIONS, AND ATTENTION ARE COMPOSITIONAL AND MULTIPLICATIVE
Quantization-awan training does not change the cost of pretraining. This is because modern GPUs require inputs to matrix multiplication to have the same precision, i.e. P. Pa PLY (Micikevi- cius al. 2022). To understand the interplay between precision and pretraining compute we must now analyze the scaling behavior of Pa and Pey well. Note that in our training experiments, only quantize on the forward pass to ensure a fair comparison between quantization aaraattton-aaar--taiinnnnn (weights only) and the additional quantization to activations/KV cache, see Appendix D
Precision activations and KV cache affect loss a similar way. We first verify in Appendix
Figure 20 that varying Pa and Pay in isolation give rise scaling behavior that is best fit by
functional form analogous to the form for Pw (Equation R
Figure 5 left).
We refer the scaling coefficients computed by varying the precision of just one part of the model a time as marginally fitted constants, and those found by fitting on runs that include multiple model components in low precision the same time jointly fitted constants.
Constants fitted marginally and jointly make similarly good predictions. We now turn our attention understanding the interactions between weights, activations, and attention. If the effects of quantizing weights, activations, and attention are independent, then a factorized, multiplicative interaction of the following form natural proposal.
Nen(P) = N e PP//mm c- -ee
(4)
We test whether this independence approximately holds by comparing the predictive power model with marginally fitted constants and model with jointly fitted constants. We show the predictive power of both models in F[gure c). finding that both methods for fitting constants have approximately the same predictive power. These results suggest that the independence assumption reasonable. We both present further evidence that this "factorized" functional form a strong fit to the data as well as discuss alternative factorization schemes in Appendix M

---

Published as a conference paper at ICLR 2025
P., Marginal Sweep 0.0028, 0.9655
Joint fit, fiP.. Pa. P 0.0086, 0.9006
Combined Marginais, 0.0089,
4.3
Prodcoed
3.8
Actumi
Actual
Actua
Figure 5: (Left) Predicted loss based on fitted values with Equation 4 (center) Fitting parameters jointly on sweeps with combinations of precisions vs (right) fitting them on "marginal" sweeps where only one model part is in low precision at time. Outliers are those at extremely low precision whose training runs are sometimes unstable.
Finding 2. The effects of quantizing the weights, activations, and KV cache during training are well modeled as independent and multiplicative so that
L(N, Pw.. Pa. Pa.)
O eff + BDD + E
where
New(P Pa, Pww =
-- + -Bo//w)
N h --
for which we fit constants Two Ta. Tkv that reflect the different sensitivities of weights, acti- vations, and KV cache. If the three precisions are set to the same value P. as in pretraining, this simplifies to Ner(P) s N(1 - e P/9333 where y is the average of the three parameters. We visualize this functional form with our fitted values in Figure 3
4.3 IMPLICATIONS FOR PRETRAINING
When training in a precision
P. meaning Ps = P&v = P, compute cost scales linearly in P Hoffmann
(2022) performed all experiments 16-bit precision and use cost model of 6ND) FLOPs. We generalize this to C
Abdelkhalik al. 2022
6 NDP to account for the linear relation between compute and precision, which reduces to the Chinchilla cost function for p = 16. We now examine three practically relevant variants of the following optimization problem.
To
6 NDP
min L(N, D, A[N(1 - - P/1/y11- BD- + subject to N.D.P
(5)
16
Since derivations algebraically involved, will work up proportionality and verify proposed solutions numerically. See Appendix D for mathematical details. We note that the implications of functional form are no matter the scale which future experiments are done, but the numerical values we predict depend on our fitted constants which are fitted on smaller-scale, integer- type experiments.
4.3.1 IF YOU MUST TRAIN IN LOW PRECISION, INCREASE PARAMETERS BEFORE DATA
Minimizing L(N, D) with P fixed, subject C x NDP. We get with some algebra that at precision P and compute budget C, the optimal allocations N*, D of parameters and data relative Chinchilla-optimal Nca, Det. will be given by
(P.C.)) (C) x
[1-6 1/5
30
P and DP(P.C) Dcn(C)) x 11-c-
-P/
atd
Pats
(6)
which suggests as precision of training decreases at fixed compute, we should increase param- eters and decrease data. The interpretation of this is that at very low precisions, our effective parameter count vanishes so that increasing parameter count compute-optimal since data egre- giously outstrips effective parameters.
II practice, the gains less than linear due to systems overhead.

---

Published as a conference paper at ICLR 2025
Predicted: Quartiized Training (NTT)
Empirical: Quantized Training (FP)
P (D) for Various N
Prodicted Va Loos
Freah VAl Lowe
PModeel Practsion)
Iredict ble Loos
0.768
Trinning Precsion (Model
0.1291
-0214
12:M
150m
Training Precision (Model
13.51gg
18/001
14810
D DDDataseet Size, Trillion Takens)
124
Figure 6: Scaling law predictions (left, fitted on integer type) empirical values (right, floating- point type). Precision weights, activations, attention fixed to Prain- Predictions closely match the empirical trend, but are shifted up by a small amount since floating-point is more expressive type and will incur lower loss at the same precision. (Right) When N held fixed, compute-optimal pre- cision increases approximately logarithmically with data. Markers correspond to predicted compute- optimal precision for Llama-3 (8b, 70b, 405b), denoted by (circle, triangle, star) at each IsoFLOP (lines), illustrating how compute-optimal precision increases data when model size is held fixed.
4.3.2 COMPUTE-OPTIMAL PRETRAINING PRECISION IS IN GENERAL INDEPENDENT OF COMPUTE
Jointly minimizing L(N, D, P) with C' x NDP. This is the setting of pretraining without con- straints on N, D. except for a fixed compute budget. Solving this joint minimization problem gives an implicit equation for P (C). Denoting u (P) [1 P/3] 3a, we find (see Appendix Ep that this equation takes the form
3a
Sat1
e -15 P
which reveals that in general the optimal pretraining precision is independent of compute budget. This suggests that compute-optimal precision should be held fixed to J= while N, D are scaled We find this Pa to be around 7-8 bits when fitting our scaling law on runs with quantization done to integer type. This has two consequences: first, this means the de- facto practice of training models in 16-bit may be suboptimal. Second, the race to low- low-precision training may have to stop before going below 4-bits, since this would force model sizes become disproportionnaely (more than 4x) larger to maintain loss scaling (see Figure 3 left).
according to Equation
We test predictions in Figure 6 at larger scale. We train compute- matphed models at various parameter count and precision ranging from FP4 to FP32 and 220M to 1.6B parameters. We train in floating-point type since that is standard in pretraining (Groeneveld al. 2024 Deitke al. 2024).
though our scaling laws are fitted on integer type. We plot our predicted trend in Figure 6 (left) and the empirical values in the middle. We find that scaling fits on integer type are strong until 4-bit precision, at which points the difference between the two types becomes more apparent. The matching of qualitative trends throughout, with the optimum being close to the predicted optimum of p* near 7-8 bits suggests that similar scaling laws may exist across types. We initiate similar analysis for floating-pote point type in Appendix ??.
4.3.3 BUT COMPUTE-OPTIMAN PRETRAINING PRECISION CAN INCREASE IN COMPUTE IF MODEL SIZE N IS CONSTRAINED
Minimizing L(D,P) with N fixed, subject to C x NDP. A common use case in practice to train suite of models of various sizes on similar data. The Llama-3 and Gemma-2 series (Dbbey et al. 2024, Team et al. 2024) are examples. In this setting, N is fixed in advance and only D, P are jointly optimized. Surprisingly, our scaling laws predict that models of differing sizes should not necessarily be trained in the same precision, and that compute-optimal precision scales as (C) log Since N is held constant and we show Appendix F that log as log in proportion, we can write P* (C' x IDg(D The intuition for this is that, for fixed N, precision acts as new lever to bring highly overtrained models closer to pretraining optimality by reducing D D/Neft
An important subtlety here is that since models overtrained for inference, want keep the cost of forward pass- which is proportional fixied, not just N. While N the same for both model of No
8

---

Published as a conference paper at ICLR 2025
byrolNun. D, Pvais Poutt
MSE: 5.06e-02, R?:
Empirical 6yr
Predicted 6pro
Panc post trainirinn preciiion (bits)
parr post post-training precision (bits)
109
10
Pred
10
10
62
10-
1220
Actual Gmr
Pa. training precision (bits)
Pe. training precision (bits)
Figure 7: Combined plots for predicting degradation. (Left) demonstrates the quality of fit all our runs, including all combinations of pre and post-training precisions. (Center, right) illustrate visually that our unified degradation form can predict degradation when training and serving in any precision. Plots (center, right) vary Pa only, but fits in (left) include runs where Pa Pky are also jointly varied.
Finding When N, D, optimized jointly, compute-optimal pretraining precision is independent of compute. 16-bit has many unnecessary bits. and 4-bit requires increasing the model size disproportionately to maintain loss scaling. Our fits imply that 7-8 bits are compute-optimal. In contrast, when N is fixed in advance, such as when training model family on similar data, (C) x log C. This suggests that for models that will be signifi- cantly overtrained, higher precision during training may be compute-optimal
A UNIFIED SCALING LAW FOR PRECISION
In this section, combine the scaling laws presented into unified functional form that predicts both training/post-trainng effects, including interactions between the two. We now treat ópro as function ÓpTo (N, D, Purain Prest) rather than just OPTo(N, D, Peea) as did earlier in Section We find two competing effects at play when predicting бpro- but overall, models trained in Tower precision are more robust post-train quantization in the sense of incurring lower degradation.
3
Two competing effects at play during post-train quantization. Intuitively, training any of Pw. Pa, PL. in low precision forces the model to learn weights that are robust to "quantization noise," so they degrade less under PTQ. However, the reduced precision will degrade more because ôpTQ increases with N-7x we found in Section 3 We call this second effect the "overtraining" effect. In practice, the first "robustification" effect wins out, so that models trained in lower precision overall degrade less when post-train quantized. We confirm using New rather than to predict degradation given various training precisions leads to a substan- tially stronger fit in Figure 21 top left, top center), to verify the competing overtraining effect.
Nar implies that models trained in low
Modifying ôrro to account for training precision. We assume training precision is strictly greater than inference precision, and define degradation identically zero if they equal. We begin by studying how degradation scales with just weight-precision during training, Pw-
Consider Figure center)) We fix (N, D) and each cell of the heatmap represents the empirical
degradation dprol?w, Pres)) We observe that degradation very quickly increases to its exponentially large value from Section 3 if there is any gap between training and inference-time precision. This
parameters in 16-bit and one with 2No parameters in 8-hit, the latter has higher Nen with our so will reach lower pretraining loss the data with the training/inference costs.

---

Published as a conference paper at ICLR 2025
motivates modifying our initial functional form fitted in Section
3
D
(N, D, Pw: Prss) Cre Paes/ Tpon
CulPe Ppee)
(8)
N
Robuxtification effect
Overtraining effect
where Cw is the only new fitted value. Then, we can extend this to include the precision effects of activations/aateenioon in the natural way:
D'D II
Orro (N, , D, Pwsy Par Pav: PpoSt) = Cre Pron
e C2(Pa-
N'W off
XE
(w,aa,kvv
We measure the fit to the data of such a functional form in Figure 7 and find strong fit with R 0.90 over 1000 data points (each of 465 pretraining runs post-train quantized multiple precisions).
An interpretable, unified functional form. Now we simplify and interpret the resulting functional form. Consider training with only weights in low precision and take Cw = for illustrative purposes so we can simplify Equation 9 Denote oe == e Pa/% as "training noise" reflecting the decrease in effective parameter count due training weights in lower precision. Then, Equation simplifies to
D NOW
BeTo(N, D, Ptrain Ppos) = OT (opio oit
(10)
Robusification effect
Overtruining effect
which we note the intuitive modification one might make to the form of the initial post-training quantization degradation we fitted in Section in Finding with small competing effects factor from New pushing in the opposite direction. It cleanly reflects the intuition that models are robustified PTQ noise the extent they were trained with similar noise.
Finding 4 (Unified Scaling Laws). Modeling cwwpprecision effects during pretraining as independent and multiplicative noise that accumulates, and including post-training quan- tization degradation, the predicted loss for language model with parameters, trained on 1) tokens, with training precision Pw, Pa. Pkv to end-time weight-precision post. can be predicted as
D,, PEv, , Pooa) AN + BD- + + ôPTQ
(11)
where 6pro( (N, D. Pw, Pa, Pkvv Ppma) in general as Equation 9 and Nan N, Pw, Pa, P.v) in Finding 4.2
CONCLUSION AND LIMITATIONS
We find that the common inference-time technique of post-train quantization can incur large degra- dation very high data budgets, demonstrating striking example of how more pretraining com- pute does not always imply stronger models at inference-time. Seeking better data scaling, we study quantization-awarc and low precision training. We find that parameters and bit precision are well modeled as interchangeably controlling an "effective parameter count" of the model allows us to predict umite-precisien loss effects accurately during both training and inference.
There are limitations to our analysis. First, we use fixed architecture throughout to examine the effects of precision, parameters, and tokens in a controlled manner. In contrast, low precision train- ing often involves architectural tweaks (Ma et al. [2224] Zhu et al. (024)) that can close much of the gap from a vanilla full precision model. Second, while compute costs do scale linearly with preci- sion, the gains from halving precision usually less than 2x due to systems overhead. Third, we only consider loss scaling without downstream model evaluations. We emphasize that the trends we find aim to be suggestive rather than prescriptive, and hope future work can more comprehensively examine these effects at larger model scale. In all, we find that the effects of precision on loss are predictable and consistent, with important and surprising implications.
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
We study the efficient training of language models, and as such do not see any new ethical concerns arising result of our work.
ACKNOWLEDGEMENTS
Tanishq Kumar thanks Tim Dettmers, Chris De Sa, Neil Band and Luke Bailey for helpful comments and discussion, as well as Ludwig Schmidt for spotting an early typo. Blake Bordelon is supported by Google PhD Fellowship. Cengiz Pehlevan is supported by NSF grant DMS-2134157. NSF CAREER Award 2IS-2239780, and a Sloan Research Fellowship. This work has been made possible in part by gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence. Aditi Raghunathan acknowledges support from A12050 program by Schmidt Sciences (Grant G2264481). Google Research Scholar, Apple, NSF, Cisco.
We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF un- der Nos. CCF2247015 (Hardware-Aware), CCF1763315 (Beyond Sparsity), CCF1563078 (Vol- ume to Velocity), and 1937301 (RTML); US DEVCOM ARL under Nos. W911NF-23-2- 0184 (Long-context) and W91 INF-21-2-0251 (Interactive Human-AI Teaming): ONR under Nos. N00014231 1633 (Deep Signal Processing); Stanford HAI under No. 247183; NXP, Xilinx, LETI- CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASE, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Re- search program, the Stanford Data Science Initiative (SDSI). Benjamin F. Spector supported by a Hertz Fellowship. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, or endorsements, either expressed or implied, of NIH, ONR, or the U.S. Government.
REFERENCES
Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural In- formation Processing Systems, 34:26989-270MM 2021.
Emmanuel Abbe, Euric Boix Adsera, and Theodor Misiakiewicz. The merged-stairease property: necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. In Conference on Learning Theory, pp. 4782-4887. PMLR, 2022.
Hamdy Abdelkhalik, Ychia Arafa, Nandakishore Santhi, and Abdel-Hameed A Badawy. Demysti- fying the nvidia ampere architecture through microbenchmarking and instruction-level analysis. In 2022 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-8. IEEE, 2022.
Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. Scaling laws for gen- erative -oxed-modal language models. In International Conference on Machine Learning, pp- 265-279 PMLR, 2023.
Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Zhen Stephen Gou, Phil Blun- som, Ahmet Üstün, and Sara Hooker. Intriguing properties of quantization at scale. Advances in Neural Information Processing Systems, -4422444,,,, 2023.
Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300-22312 2022.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models. arXiv preprint 162. 2024.

---

Published as a conference paper at ICLR 2025
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't t reach for the stars! arXiv preprint arXiv:2301.03968 039988 2023.
Zeyuan Allen-Zho and Yuanzhi Li. Physics of language models: Part 3.3, knowledge capacity scaling laws, arXiv preprint arXiv:2404.05405, 2024.
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The on- set of variance-limited behavior for networks in the lazy and rich regimes. arXiv preprint arXiv:2212.12147, 2022.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. Proceedings of the National Academy of Sciences, (21( (177)))22311111122 121, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2399.16609 2023.
Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hid- den progress in deep learning: Sgd learns parities near the computational limit. Advances in Neural Information Processing Systems, 35:21750-21764, 2022.
André Bauer, Simon Trapp, Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive exploration of synthetic data generation: A survey, arXiv preprint wrXiv:2401.02524 2024.
Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404. 10102, 2024.
Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. arXiv preprint vrXiv:2309.6602 2023.
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. dynamical model of neural scaling laws. arXiv preprint 2rXiv:2402.01092, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXhv:247.221887 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https://arxivg: org/ wwt./a/.////ooooooo/oooo/o.aaee/ 2005. 14165
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher M De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive Language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 2818-2829, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
12

---

Published as a conference paper at ICLR 2025
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International conference machine learning, pp. 4057- 4086. PMLR, 2022.
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs the edge of stability, In International Conference on Learning Representations, 2021.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv preprint 1rXiv:1412.7000 1412.7024, 2014.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-arr multimodal models. arXiv preprint arXiv:2409. 2024.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning, pp. 7750-7774, PMLR, 2023.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via Block-wise quantization. arXiv preprint arXiv:2110.02880 2021.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35: 30318-3033 30332, 2022.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint arXiv:2306. 03078, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.087588 2021.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 2Xiv:2407.2178. 2024.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7382-7392, 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Samir Yitzhak Gadre, Georgios Smymiss, Vaishaal Shankar, Suchin Gururangan, Mitchell Worts- man, Rulin Shao, Jean Mereat, Alex Fang, Jeffrey Li, Sedrick Kch, et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXivv2403.08554. 2024.
Justin Gilmer, Behrooz Ghorbani, Ankush Garg, Sneha Kudugunta, Behnam Neyshabur, David Car- doze, George Dahl, Zachary Nado, and Orhan Firat. loss curvature perspective on training 2021.
instability in deep learning. arXiv preprint arXiv:2110.044.
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. arXiv preprint arXiv:2402..0 00838, 2024.
Alexander Hägele, Elie Bakouch, Atli Kosson, Loubna Ben Allal, Leandro Von Werra, and Martin Jaggi. Scaling laws and compute-optimal training beyond fixed training durations. arXiv preprint arXiv:2405. 18392, 2024.
13

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXtv:2203.15550 2022.
Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, and Xiaojuan Qi. Billm: Pushing the limit of post- post-training quantization for llms. arXiv preprint arXiv:2402.04220 04291, 2024.
Berivan Isik, Natalia Ponomareva, Hussein Hazimeh, Dimitris Paparas, Sergei Vassilvitskii, and Sanmi Koyejo. Scaling laws for downstream task performance of large language models. arXiv preprint 20l::24002.01177. 2024.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient -nneer--rithmmeti-onnly inference. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2704-2713, 2018.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b. arXiv preprini arXiv:2310.0683) 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
Jakub Krajewski, Jan Ludziejewski, Kamil Adamczewski, Maciej Pióro, Michal Krutul, Szymon Antoniak, Kamil Ciebiera, Krystian Król, Tomasz OdrzygóždZ, Piotr Sankowski, et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023.
Jeffrey Li, Alex Fang, Georgios Smymiss, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.1179. 2024.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06111/ 2023.
Lin, Jiaming Tang, Haotian Tang. Shang Yang, Xingyu Dang, and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arxiv. MLSys 2024, 2023.
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang. Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024a.
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data. arXiv preprint arXiv:2406. 21V:2406.084666, 2024b.
Qian Liu. Xiaosen Zheng. Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang. and Min Lin. Regmix: Data mixture regression for language model pre-training. arXiv preprint wwxiv:2407.014900 2024.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emni Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. /7888, 2023.
Anton Lozhkov, Raymond Li. Loubna Ben Allal, Federico Cassano, Joel Lairieoir,er, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. Starcoder and the stack v2: The next generation. arXiv preprint arXiv:2402. 19173, 2024.
14

---

Published as a conference paper at ICLR 2025
Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al. Fingpt: Large generative models for small language. arXiv preprint arXtv:2311.056400 2023.
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang. Shaohan Huang, Li Dong, Ruiping Wang. Jilong Xue, and Furu Wei. The era of 1-bit Ilms: All large language models are in 1.58 bits. arXiv preprint arXiv:2402. 17764, 2024.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 2017.
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisen- thwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, et al. Fp8 formats for deep learning. arXiv preprint arXiv:2209.08 05433, 2022.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Xin Yong, Hailey Schoelkopf. et al. Crosslingual gen- cralization through multitask finetuning. arXiv preprint arXiv:2211.01786, 2022.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Alcksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin A Raffel. Scaling data-constrained language models. Advances in Neural Information Processing Systems, 36, 2024a.
Niklas Muennighoff, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Jacob Morrison, Sewon Min, Wei- jia Shi, Pete Walsh, Oyvind Tafjord, Nathan Lambert, al. Olmoe: Open mixture-of-experts language models. arXiv preprint arXiv:2409.00000 240,00060, 2024b.
Quynh Nguyen, Marco Mondelli, and Guido F Montufar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In International Conference on Machine Learning. pp. 8119-8181 8129. PMLR, 2021.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Teddy Ferdinan, Haowen Hou, Przemystaw Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint 2rXiv:2404.05892 2024.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann,, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 22 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 1-7 2020.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405. 2024.
Nikhil Sardana and Jonathan Frankle. Beyond chinchilla-optimal:: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.004453 2023.
Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al. What language model to train if you have one million gpu hours? arXiv preprint 15X14:2210.15424, 2022.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202, 2020.
Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang. Christopher Ré, Ion Stoica, and Ce Zhang. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Learning, pp- 31094-31116 PMLR, 2023.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv: 100900803 2019.
15

---

Published as a conference paper at ICLR 2025
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint WxXv::2408.03314, 2024.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, et al. Dolma: An open corpus of three trillion tokens for language model pretraining research. arXiv preprint arXiv:2402.00159, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeh, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonss al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: enhanced transformer with rotary position embedding. corr abs/2104.09866 098644 (2021). arXiv preprint arXiv:2104.09800 2021.
Xiao Sun, Naigang Wang, Chia-Yo Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar EI Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakrish- nan. Ultra-low precision 4-bit training of deep neural networks. Advances in Neural Information Processing Systems, 33: 182:1796-887,, 2020.
Chaofan Tao, Qian Liu, Longxu Dou, Niklas Muennighoff, Zhongwei Wan, Ping Luo, Min Lin, and Ngai Wong. Scaling laws with vocabulary: Larger models deserve larger vocabularies. arXiv preprint 2XXv:2407.13623 2024.
Yi Tay, Mostafa Dehghami, Samira Abnar, Hyung Won Chung. William Fedus, Jinfeng Rao, Sharan Narang, Vinh Q Tran, Dani Yogatama, and Donald Metzler. Sealing laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207. 10551, 2022a.
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So, Siamak Shakeri, Xavier Garcia, Huaixiu Steven Zheng, Jinfeng Rao, Aakanksha Chowdhery, et al. Transcending scaling laws with 0.1 % extra compute. arXiv preprint arXiv:2210.11399 2022b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11800 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv: m 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 13X11
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.08 092888 2023b.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An in- struction finetuned open-access multilingual language model. arXiv preprint arXiv:240..07827. 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you (nips, 2017. arXiv preprint 130600770 10:50140525X 16001837, 2017.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang. Yi Wu, and Furu Wei. Bitnet: Scaling 1-bit transformers for large language models. arXiv preprint arXiv:2310.1 /453, 2023.
16

---

Published as a conference paper at ICLR 2025
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language 2022.
models. arXiv preprint
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. Advances in Neural Information Processing Systems, 36:10271-10296 2023a.
Mitchell Wortsman, Peter Liu, Lechao Xiao, Katie Everett, Alex Alemi, Ben Adlam, John Co- Reyes, Izzeddin Gur, Abhishek Kumar, Roman Novak, et al. Small-scale proxies for large-scale transformer training instabilities. arXiv preprint arXiv:2309. 14322, 2023b.
Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quan- tization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602, 2020.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Greg Yang. Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ry- der, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs V: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466, 2022.
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candès, and Tatsunori Hashimoto. Synthetic
continued pretraining, arXiv preprint arXiv:2409.0743/ 2024.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Rui-Jie Zhu, Yu Zhang, Ethan Sifferman, Tyler Sheaves, Yiqiao Wang, Dustin Richmond, Peng Zhou, and Jason K Eshraghian. Scalable matmul-free language modeling. arXiv preprint arXiv:2406.000 022-22286.0228. 2024.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. arXiv preprint arXiv:2308.07663, 2023.
17

---

Published as a conference paper at ICLR 2025
Weights
KV Cache
Activations
4.27
4.49
4.23
4.25
Final Loss
4.25
Final Loss
4.35
Final Loss
4.22
4.24
4.21
4.30
4.23
4.20
4.22
4.25
4.19
4.21
4.18
Procision (bits)
Precision (bits)
Precision (bits)
Figure 8: L(Pw), L(Pa), ((Pv)) ) for ablated hyperparameters, N 30M, D = 1.5B. We can see the trends persist, where the first few bits reduce final val loss significantly, with diminishing/saturating returns quickly setting in at higher precision. We do not fit constants on these ablated runs.
A HYPERPARAMETER DETAILS AND ABLATIONS
We launch over 20 runs for each (N, D) combination to study scaling in precision, trained and validated the common crawl split of the Dolma dataset (Soldaini et al.) 2024). We use a standard causal Transformer++ implementation: SwiGLU activations (Shazeer) 2020). RoPE embeddings (Su et al. 2021 RMSLayerNorm, Adam 8 values of (0.9, 0.95). We adopt a cosine learning rate schedule with 10% warmup period and peak learning rate of 6e-4 for the smallest model and learning rates scaled with width and depth according to depth-jPP for the larger models (Yang et al.) [022] Bordelon et al. 2023). We use sequence length of 1024 and batch size of 256 throughout, with Adam e 1e-15, following (Wortsman et al., 2023b). We use weight decay of 0.1, as (Ahmadian et al. 2023) find some results in the quantization literature may be artifacts of insufficient weight decay. We follow (Ma et al. (024)) in including LayerNorm before projections because they find is important for low precision training to be stable. These are the hyperparameters and settings used for the main scaling law experiments.
To check robustness, ablation we use a sequence length of 512 with batch size 128, weight decay of 1e-3, Adam c of 1e-10, a peak learning rate of le-4 and a warmup period of duration 3%. We train models with these alternative hyperparameters at various weight, activation, and KV cache precisions. We train and val on C4 (Raffel et al.) 2020 Dodge et al. 2021) instead. Though these ablations are at rather smaller scale due to compute constraints, the loss curves follow the same trends rapid decrease in final loss with an initial increase in precision from 4 bits, then diminishing returns as we approach higher precision as in the main text, suggesting the trends are robust to hyperparameter choices.
then ablate these hyperparameter choices, with results in Figure In our
B ADDITIONAL RELATED WORK
Efficient training and inference Low precision has been key to improving the efficiency of train- ing and using LLMs (Micikevicius et al. 2017
Shoeybi et al. 2019: Wortsman al. 2023a Zhu
et al. 2023). Prior works generally study either precision during training (Courbariaux et al. 2014]
Dettmers et al. 2024: 2021 Sun al. 2020
Liu et al. 2023) or the effects of changing the pre-
cision after training (post-traee training quantization) Frantar et al. 2022; Lin et al.) 2024a] Dettmers) et al. 2022. Xiao et al. 2023 [heng et al. 2023 Dettmers et al. 2023p. In this work we study both,
the precision during training and after, and unify them from scaling perspective. Other important works include recent popular work on
(Ma et al., (024) where weights are quantized to extreme precisions (ternary) the forward pass during training. This work is con- sistent with ours in that they can quantize weights so aggressively because weights are less sensitive than activations KV cache. Further, while a controlled comparison across precision, they use a nonstandard architecture, learning rate, and weight decay schedule specifically to make training with ternary weights stable.
fixed architecture throughout to maintain
18

---

Published as a conference paper at ICLR 2025
Large language models and scaling By scaling up the transformer architecture Vaswani al.
2017) a variety of large language models have been proposed (Brown et al. 2020, Rac et al. 2021
Touvron al. 2223a b. Dubey et al., 2024
Le Scao al., [2023]
Muennighoff et al. 2022 2024b
Groeneveld et al. 2024 Jiang et al. 2023;
Zhang et al. 2022] Allal et al. 2023; Li et al. 2023
Lozhkov et al. [2024] wu konnn al. 2023]
Bai al. 2023] Chowdhery al. 2023] Team al.
2023 [stünn et al. 2024, Deitke et al., 2024). To improve our understanding of these models,
various works have investigated their scaling properties (Ruan al. 2024 Allen-Zhu & Li 2024
Hägele et al., 2024). Many aspects are relevant to scaling including the architecture (Tay et al. Krajewski et al. 2024] Tao al., 2024 Clark al. [022] Tay et al. 2022b; Scao et al.
2022a
2022
Peng et al.] (024)) the modalities considered (Aghajanyan et al. 2023; Alabdulmohss et al.
2022
Cherti et al. (023). the performance metrics (Wei et al., 2022 Srivastava et al. 2022 Isik
et al.
2024), the data composition (Li et al.) [024] Liu et al. 2024 Allalak et al., 2024) and data
repetitions (Muennighoif et al. 2024a). Our work analyzes one such aspect, which is key to better
scaling: the numeric precision during and after training.
C ALTERNATIVE FUNCTIONAL FORMS
There are several plausible functional forms to try a priori. The key junctions are whether form is 1) additive or multiplicative and 2) interacts with parameters/data is independent, 3) power law or exponential. We try a variety of combinations main text one of the best fits, notably with the fewest fitted parameters. We emphasize that several fitted forms are likely to be reasonable fits to the data, and an important desiderata for choosing a functional fit is interpretability. Several scaling law papers find multiple fits plausible in terms of predictive power (Muennighoff et al. 2024a Kaplan et al. 2020), and ultimately make a decision based on interpretabilityy.
these three and find the formulation in the
We make these fit choices on sweeps of the form I(N, D, Pw) and discuss alternatives to the de- composition/fictooiiaaioo to account for activations and KV cache in Appendix Section M which assumes an effective parameter count formulation. In this section, a power law refers to a term of the form C. P-ow oow where Cw,, Ow are fitted. general, we find modeling precision effects with power law fits on their own causes the fitted constants A, B to blow up, whereas this does not happen with exponential fits, suggesting the power law does not change sharply enough to match the change in loss induced by precision. We note that while fitting parameters using double notion of effective parameters and effective data leads to slightly better fit, requires more fitted parameters so we stick with the Ner formulation for simplicity and interpretability. When choosing between fits validate on held-out data and the R² values below reflect the fit on the held out data. This is in contrast our plots in the main text, where have chosen functional form and we fit and plot the same data, as is standard in scaling laws (Muennighoff et al. 2024a).
Functional Form
Val R
Number of Fitted Parameters
Neif
0.82
3
Addinive/independent power law
0.71
2
Dent
0.74
Ne and Den (tied)
0.79
3
Net and Den (not tied)
0.84
4
Multiplicative power law, N,P
0.75
Table 1: Comparison of Functional Forms with R2 and Number of Fitted Parameters
D QUANTIZATION IMPLEMENTATION DETAILS AND TYPES
Two canonical types for neural network quantization are floating-pols (FP) and integer (INT) quan- tization. Despite their differences in representation, we hypothesize the scaling behavior between floating-pott -oitingg-oontttoe and integer quantization can be described by similar functional forms, where [[b) provides preliminary evidence for this.
19

---

Published as a conference paper at ICLR 2025
D.1 INTEGER QUANTIZATION AND IMPLEMENTATION DETAILS
In integer quantization, continuous values are mapped to discrete integer values. Typically, this is done by scaling the original values according fixed scale factor. Mathematically, for real number z, the quantized integer value Tint is computed as:
iint
where s is the scaling factor, and 1-1 denotes rounding the nearest integer specified by the number of bits. The value can then be dequantized back to an approximate real value by multiplying by s:
docquatt
Zint
This process introduces quantization error, defined as the difference between the original value & and the dequantized value zdoquant- The goal of quantization is minimize this error while still reducing the precision. One can think of this as rounding to the nearest point on uniform lattice. More complicated quantization schemes involve selecting the lattice points in a data or model-dependent manner. Integer quantization, as implemented, uses fixed-point scaling based on the maximum absolute value of the tensor, and then scales the values within the range [Q,, Opl. where Qn (b- and Qp = 2 1) - 1, with being the number of bits.
Integer quantization first rescales the inputs into the range specified by the number of bits by
max [x=
for tensor-based scaling, or
Op
max [r]] dim = k))
for channel- scaling. After scaling, the result is rounded to the nearest integer and then clamped to the range [Qn, Qp). After matrix multiplication, the result is rescaled back into the original range. We quantize only the forward pass this work, to ensure fair comparison between quantization- wware-training (weights only) and low- low-precision training (weights, activations, KV cache). This is because the backward pass is not usually quantized during quantization htttii/attoonaaa---ree sware-training (Ma et al. 2024), so comparing sensitivities of weights (forward only) activations/KV V cache (forward and backward) would not be a principled comparison. In production pretraining in low precision, the matrix multiplications on the backward pass are also quantized, leading further compute savings. We leave detailed analysis of how our observations change when accounting for the backward pass to future work. We use integer quantization throughout fit our scaling laws for simplicity.
D.2 FLOATING-POINT QUANTIZATION
Floating-poii poiit quantization is slightly more sophisticated, aiming to make a non-uniform lattice roughly matching the distribution of the weights, which are assumed to be Gaussian. A floating- point number in general represented as:
Ttp (-1)2-m m2
where s the sign bit, is the mantissa, and is the exponent. In foating-point quantization, both the mantissa and exponent are quantized to reduce the bit width. For exponent exponent-mantissa mantissa allocations of bits and details of exponent bias, we follow the guidelines from (Micikevicius et al. 2022) and quantize weights per channel and activations per-tensor.
Making a full scaling law for floating poitt quantization is more involved than our integer treatment, because the effects of scaling mantissa vs exponent bits are not the same. In contrast, in integer quantization, each additional bit simply causes us to round into a finer-grained lattice after rescaling, thereby reducing quantization error by predictable amount. In floating-pons -pinnt quantization, altering the exponent affects the dynamic range, while altering the mantissa changes the precision within that suitable for model training,
range. This flexibility at once makes floating poating-pointt quantization but harder to analyze. We leave a commensurately detailed analysis mantissa vs exponent and more generally floating point- scaling future work.
20

---

Published as a conference paper at ICLR 2025
D.3 HARDWARE DETAILS
Weight-only quantization can accelerate inference because software can be written to accommodate moving data between GPU parts (HBM-SRAM) in smaller units (types), that given bandwidth can move more data per second. This reduces memory (IO) bottlenecks that often dominate during inference, even with high-batch workloads. However, we emphasize that the type and therefore speed at which the GPU can do matrix multiplications in natively is determined by the hardware provider, so that even when Pw Pa = PaRy (including queries), compute savings are only achieved when these correspond with both bit-width and type that the GPU supports. We aim to study scaling in a fairly hardware-agnostic manner so that our work may be useful in the future, and make no claims about hardware details or optimality. We train all our models with fake (simulated) quantization on NVidia H100 GPUs to remain hardware agnostic, not taking advantage of any true low- lowpprecision computation. The only assumption is that when hardware does implement support for integer quantization, it done in a way that involves some combination of rescaling and rounding,
is standard the time of writing (Dettmers & Zettlemoyer 2023 Dettmers al. 2022
Wu al.
2020 Jacob et al. 2018).
E DERIVATIONS
E. CRITICAL DATASET SIZE FOR PTQ
We seek Denit that satisfies @L(D.m) presented in the main text and equating their opposing effects, we get the equation
05vgg( AD Taking both derivatives for the functional forms
aD
BD BD-9-1 9-1 cnt
NDCNNN e- -Ppaal "io D20-
(12)
cril
which implies
BBN e Poea/ Nppa
D
Denit
(13)
YDCT
the predicted point after which pretraining on more data can increase loss of a model that is post- train quantized. Note that this quantity explodes in P, that truly unreasonable amount of data required for longer pretraining to be harmful at commonly used precisions (eg. 8-bit). However, we find that on overtrained models D/N > 103, these overtraining-degradatte degradation effects become nontrivial around 5-bits, and dominant below that.
E.2 COMPUTE-OPTIMALITY CALCULATIONS
We a constraint C & NDP throughout. Working up proportionality is essentially rescaling the compute constraint, so it doesn't affect the scaling trends we identify, which is our focus.
E.2.1
FIXED PRECISION COMPUTE OPTIMAL SCALING
Under fixed precision, the loss takes the form
L= i=u(P)AN-* + BD-3
(14)
where (uP) = [1- e e-P/1] -3a 3a is a fixed constant. The compute optimal scaling when minimizing the loss over N. D gives
L == 8
(15)
by replacing D C P Optimizing over N, that this is equivalent to the original chinchilla optimization problem but with A Au(P) and B BP Performing this optimization, we find
"u(P) Aax
+
(u(P)Aa) -at
(*,CCC)
Cata
DD((,C)=
Cate
(16)
BP3
BP3
21

---

Published as a conference paper at ICLR 2025
We relate the above expressions to the original Chinchilla-optimal N, D at full precision (ca(C), Dcn
() (Cr(C) & [1-E- 1/8 a-8 p prald and
()((O) (cn(C) x [ 1-e- --/8
a-d
pafe
(17)
E.2.2 FIXED MODEL SIZE N
Now, investigate the case where model size N is fixed but precision and data are jointly optimized at fixed compute C = NDP. This optimization problem takes the form
= u(P)AN + BD-
(18)
Under fixed compute, we have D = NP so replacing the second term, we have
C
L= ===(P))N== + BC- N pd
(19)
where N is constant. We therefore have single variable P to minimize the above formula over
OL OP WP)ANN + -C 8 p8-1=0
(20)
First, we note that (P) has the following form
1 - p/y
3a -r/1 x u(P) 2odl
u' u'(P) -3a[1 p/11-aa- 3a-1 We thus desire a solution to the implicit equation
(21)
7
3a e P/v x (P)) 3a-1
p
(22)
We now aim to find approximate asymptotic relationship between P and C' as C + xo. Taking a logarithm both sides, we find (neglecting additive constants that are independent of C, P)
- (3a + 1) 1) - c Pi7)
Cn
(23)
The correct dominant balance at large C' is to take P ~ By In C, as can be verified numerically. With the constraint that = NPD we have that D* N3y
E.2.3 MINIMIZATION OVER N. D. P WITH FIXED COMPUTE
Recall our three-way loss function is given as below. We separate Ner into terms involving (N, P) explicitly here as it makes the math easier follow.
L(N, D. P) -N-
- "u(P) + BD- w(P) [1 -1-e-- - -PP/1-30
(24)
Under the constraint C' x NDP, we can replace D in terms of C, N. P giving the loss expression
= AN - P) + BN pd -
(25)
0L aAN 0- P) + 8BN8 1 88
(26)
ON
0L 3--////NNN
P/T
(27)
OP
Multiplying the first equation by N and dividing the second equation by it reveals that the optimal P satisfies compute-indee innppeddindd implicit equation
3
(28)
This exercise reveals that the compute optimal strategy when allowed to jointly optimize N. D. to choose a fixed precision that satisfies the above equation and then to scale up N, with the prescription in Appendix I.I.1.
22

---

Published as a conference paper at ICLR 2025
pa C= 2NP
D vs C 2NP
N vs 2NP
15"
Post-train Precision (P*)
11
Model Parameters (NP)
10
1277
304
Lama3.a0
2077
Lama3-100
Lanaa 405b
103
124
18'
16
60
16
36
40
16-
16"
16"
102
16.
Inference FLOPs, C= 2NP
Inference FLOPs, C= 2NP
Inference FLOPs, C 22NP
Figure 9: Numerically minimizing model of inference-time costs with respect to N, D, after accounting for post-train-qqaaniizztonn degradation and its relation to overtraining.
E.3 INFERENCE-TIME COST MODEL
For many, inference is the primary cost of training and serving models. Here, we present prelimi- nary analysis of an inference-time cost model. The key tension that inference cost scales as NP, so that inference costs at fixed pretraining loss can be reduce by either reducing model size (and overtraining more) or quantizing post-training
We will assume here that P= poss refers to the precision weights will be quantized to. In practice, inference costs may depend on the precision of the KV cache and activations to some extent as well, but we assume this for tractability of the following mathematical model, and to get sense of how overtraining and post-train quantization concerns play out at inference-tinas We can phrase this minimization problem in the following way.
D Nuv
min L(N, AN BD- CT e -Pr subject C' NP N.D.P
(29)
The system of first-order conditions that results from this constrained optimization problem is not in general tractable analytically, so we solve the above constrained optimization problem for (C(C), (C)) (C) numerically via simple grid search. We find that ,*, D* grow as a power law in C while pa x log C. The clumping in points is an artifact of the numerics of the grid search; the fitted lines represent the loglinear (left) and loglog (middle, right) trends overall.
It might be surprising that D is not taken to infinity since it does not appear in the cost function. The reason for this is because if it was, post-train degradation (the third term) would become large. might also be surprising that D changes with compute at all. The reason for this is because, once again, of the third term: we allow more inference-tine compute we more N, and at larger N we can now tolerate a larger data budget for a given post-train quantization degradation, so being compute-optimal means taking advantage of this and training that larger parameter count on more data.
The intuition for why p log C might be as follows. Consider a situation in which p* is in- dependent of compute: the third term will come to be a bottleneck in loss as compute gets larger because N, D are both being scaled as power laws in compute, and eventually the effect of e PPr will become non noglngglie in comparison to the first two terms in the loss function. To continue decreasing loss at this point, we must make this term smaller at a rate commensurate with the other terms, which go as a power law in compute. Since precision is inside the exponential, this can be done by taking P ~ log C. An important thing to note is that since we are ignoring pretraining costs here, the absolute values of predicted D* are much larger than would be realistically possible in any reasonably training regime, where pretraining costs do matter, even if less than inference costs. But the empirical trends in N P showcase how overtraining with post-train quantization in mind can outperform vanilla overtraining without accounting for its effects post-train quantization.
23

---

Published as a conference paper at ICLR 2025
N= 30M
N 60M
N 110M
N 220M
Val (Post-Quae (Post-Quant)
INT3
4.2
INT4
INTS
INT6
No PTQ
3.6
Degradation, 6ptQ
10
10-1
10
Token/Parameter Ratio
Figure 10: Replicating Section results with AWQ.
N= 30M
N 60M
N 110M
N 220M
Val (Post: Quant)
5.5
INT3
INT4
5.0
INTS
INT6
PTO
4.0
3.5
10c
Degradation, 6pTO
10
Token/Parameterr Ratio
Figure 11: Replicating Section results with RTN.
F REPLICATING PTQ SCALING WITH OTHER QUANTIZATION METHODS
Here replicate the finding that post-train degradation due post-train quantization increases with token/parameter ratio as D /N We fit the same functional form as in the main text, but get slightly different values fitted constants, as expect. We replicate on AWQ (Lin et al., 2023) and round-to-nearest quantization. The former is modern and sophisticated technique, and the latter simple and naïve approach quantization. The fact they, as well as GPTQ the main text, share the same failure modes suggests that poor post-training quantization data scaling should be default expectation any newly proposed PTQ technique.
G PTQ: LEARNING RATE SCHEDULE ABLATION
Here, we ablate our learning rate and schedule to use warmup with linear decay, as opposed to a cosine schedule, check it not artifact of choice of learning rate schedule. We do so
24

---

Published as a conference paper at ICLR 2025
Val Loss Post--Quntt)
N 30M
4.4
4.2
INTS
INT4
4.0
PTQ
100
Token/Parameter Ratio
Figure 12: Linear LR Schedule Ablation
on our 30M model due to compute constraints, finding the degradation with token/parameter ratio persists, as expected.
H WHY DO LANGUAGE MODELS GET MORE SENSITIVE WITH OVERTRAINING?
This section is speculative.
Sharpness. A canonical line of work in optimization demonstrates that model sharpness increases during learning until it hovers at a maximal value (the "edge of stability") (Cohen al. 202T Gilmer et al. 202I), so that movement along the top Hessian eigenvector degrades loss by more throughout training. Though sharpness is formally worst-case sensitivity, results hold for average case, such as loss degradation induced by isotropic noise. It may be possible that sharpness during language model pretraining does not reach its maximal value for long time, which is why sensitivity to noise monotonically seems to increase as D/N + DO on realistic data budgets. Closely related is the largest eigenvalue of the neural tangent kernel (NTK) which captures the magnitude of the variance of the predictor under parameter noise, This quantity is known to empirically increase during training in a variety of settings, and is closely related to generalization guarantees (gguyen et al. 2021 Atanasov et al. (022).
conjecture similar
Hierarchical learning strategies become more sensitive throughout training. Our expectation that overtrained language models may degrade more when quantized at inference-time is motivated in part by the following results. The hierarchical nature of learning is by now well understood in some toy settings: in (Abbe et al.) [021]. it is shown that "staircase" polynomials of increasing de- gree are learned faster than high- monomials since neural networks combine existing features to learn ones. In (Abbe al. 2022) this result
strengthened to show that such hierarchical structure both necessary and suffcient to learn sparse functions with SGD in two layer neural net- works. In this setting, damage to features encoding lower-order polynomials affects all higher-order ones, so that such networks are increasingly sensitive to fixed feature noise throughout learning. Another result of similar flavor is that of (Barak al., 2022), who explicitly require high-precision gradients for sparse parity to be learned, since sparse parity is learned by the amplification of small initial signal. If language models learn hierarchically, is possible that the features that learned late into overtraining as D/N - co are reliant on base features, so that noise harms the base features and therefore significantly damages higher-order features.
I GRANULARITY ABLATIONS
Here, we ablate our choice of quantization granularity (per-tensor vs per-channel) compared to the main text, where we do weights per-channel and activations per-tensor. Per-tensor quantization involves keeping one scalar to rescale all values in tensor into the quantization codebook range, and per-channel means keeping a scalar per channel dimension; therefore, the latter is strictly more expressive and thus incurs lower quantization loss, than the former, at the cost of slightly more memory usage. Here, we ask: is the increased sensitivity of activations a result of them being inherently sensitive, due the per-tensor design choice.
25

---

Published as a conference paper at ICLR 2025
Varying Quantization Granularity
Weights iper tomsor)
5.00
Weights
Activations (per tensor)
4.75
Activations (per channel)
Loss
4.50
4.25
4.00
3.75
6
8
12
Precision (bits)
Figure 13: Quantization granularity ablation: all combination of (training weight precision, training activation precision) x (per-tensor, per-channel). Dashed lines are per-channel and solid are per- tensor.
These results show that activations are generally more sensitive than weights, since their loss penalty lower precision goes up faster even when granularity kept fixed across the two. In fact, quan- tizing activations per-channel is almost hard quantizing weights per-tensor. This is consistent with a broad line of work in quantization finding that activations comprise the central difficulty in quantization (Dttmeess & Zeetlemoyer 2023
Ma et al. 2024).
J MAIN FIGURE DETAILS
The model on the left is N = 30M parameters, chosen because we could train it to the highest token/parameter ratio given our compute budget. On the right train suite of models with NP kept constants on 16B tokens (so that C' 16 N DP is matched throughout under our cost model). We plot val loss Dolma, as throughout the main text, and floating-pe point (rather than integer) to make the pretraining claims as realistic as possible.
K NUMERICAL FITS
Following (Muennighoff et al. 2024a), we tie a = 3 so they do not become very different, though this is not required. Distinct ax 3 only add expressivity to the model and we have verified the plots look similar without tying. We also only use the full scaling law when specified in the text, since the law is developed piccewise through the text. For instance, Figures 3 and 4 solely fit Chinchilla with a substitution N > Nen(Pw) because at that point Pa, Pvv have not been introduced. Figures 5, 6, and use full scaling law, for instance make predictions. We emphasize constants are unlikely to be useful because as (Hoffmann et al.) 2022; Sardana & Frankle (2023) show, fitted constant depend heavily on the architecture and dataset used, which differs from setup to setup. Rather, the trends we identify are the key findings. With that said, our fitted constants are as follows.
numerical
Note that we include biases in our exponent fits, for instance when modelling Net as a saturating exponential, we find that the different parts model cause numerical instability different values of low precisions, so even if they are the same functional form, they may be translated (left/right shifted versions) of eah other. For instance fit of the form <=/7a in the main text is really computed with offset tn but including biases everywhere clutters notation and obscures mathematical insight.
26

---

Published as a conference paper at ICLR 2025
Constant A
Value
4.299e3
a
0.4965
B
1.806e4
2.7648
Tw
2.6745
nw
0,3037
78
2.2102
1,4072
Tkv
0.9578
Ylkv
2,4185
CT
0.0598
YDD
0.5068
Y/N 0.3439
Y 0.5907
1.1277
Table 2: Fitted constants and their values
params, Weights
params, tokens
KV params,
3.50
32
32
110M params, 26.28 tokens
1100 params, 26.28 tolens
110M params, 26.28 takens
3.50
53
2.0
340
Prodsion (6l5))
Produton (blss
Prodsdon (bits)
Figure 14: Sweeping L(P) for the three model parts at various N, D.
L ARE WEIGHTS, ACTIVATIONS, AND K. CACHE EQUALLY SENSITIVE?
We find that training runs with P. < P&y S 3 are not numerically stable, and often diverge, while Pv is still well behaved. In particular, we find activations are more sensitive, though this could be because quantize activations per-tensor and weights-per channel, rather than activations being inherently more sensitive. Consequently, we do not fit or validate on runs with activations or attention bits equal to 3. We leave more detailed analysis of fine- areiindde sensitivity across layers and types of parameters to future work. The Figure below illustrates the empirical sensitivity by plotting L(P) for the three quantities for various (N, D).
27

---

Published as a conference paper at ICLR 2025
Weights
Activations
Cache
NoofPine
0.4
10
10 Pg (training precision, bits)
12
10
12
Pw (training precision, bits)
Ps, (training procision, bits)
Figure 15: Plotting what Nar looks like empirically. Each black point is pretraining run, mathe- matical details what is plotted here in Appendix Blue lines are parametric fits of a saturating exponential.
M EMPIRICAL NEFF
Consider a model trained with some arbitrary (N, D, Pw). Assuming a Chinchilla function form with N + Nen(Pw), we can write the difference between its loss and the loss of full precision model as
A[N - N
as the terms involving B, D, E cancel. Note that Ner(Pw = 20) = N by construction. In practice use BF16 model as the "infinite- uuninite--prcisionn" model, finding no real difference we use an FP32 model or even functional fit estimating P. - xo based on our integer quantization loss results. Our goal is to plot what f(P) looks like where Nen N f(P). Therefore, we can rearrange the above equation as follows
F(P) Nar 1 L(N.D.P)) D, 20) + N
1/a
(30)
N
Then plotting this quantity using our fitted numerical values (See Appendix K) gives the empirical tradeoff between precision and parameters. We can see that the tradeoff is quickly saturating in P to value near 1. While the functional form is the same for the three model parts, the fitted constants are different. For instance, runs with P. < 3 or Pkv < 3 often diverged, and this was not the case with weight precision. Further, higher bit value, but very quickly becomes sensitive around 4-5 bit precision.
that the KV cache is not sensitive quantization
Then far as the joint functional form for Nor Pw., Pa. Pkv) is concerned, acknowledge that alternative factorizations that do not decompose the model into weights, activations, and KV cache, may have an equally good fit. For instance, decomposing the weights term into a product of layer- wise effects has reasonable fit though introduces more parameters, and more coarse- coarse-grained version may not decompose the model into parts all, but only consider tied precisions. We choose this factorized form because QAT considers weights only, and activations and attentions are the two other things that must then kept low precision to see compute gains. Since practitioners often care about KV cache its own, chose to decompose "activations and attention" as "activations and KV cache." " We emphasize that our main point not that this factorization is objectively correct, but in observing that such factorization that assumes approximate independence is possible in the first place.
N FLOATING--PINT EXPERIMENTS
The key difference between floating point and integer type is that the former allocates some bits to the exponent representation and some to the mantissa, and these bits play different roles, unlike in integer type where every bit plays the same role in making the quantization lattice uniformly more fine-grained. We hypothesize that if exponent and mantissa bits are scaled jointly (ie. increase
28

---

Published as a conference paper at ICLR 2025
Predicted vs Actual MSE: 0.0027, RP: 0.9678
N=220M, D=13.1B
N-30M, D=6.78
4.5
3.40
3.36
Prodicted
3.34
LOGS
3.30
3.95
326
3.90
3.2
4.0
4.2
4.4
3:2
11
10
Actual
Weight bits (Ph. fllating point)
Woight (P flatting point)
Figure 16: Fitting an effective parameter form to floating-point precision for weight training. (Left) involves checking quality of fit 140 training runs in floating point precision for weights during training.
Loss vs Exponent/Mantissa Weight Bits
3.9
Loss
3.8
MxE3
M3Ex
Pw: (exponent/mantissa))
Figure 17: Exponent-mantisaa bit allocation sweep. We can see the two types of bits have different scaling behavior, but both fit the saturating form where the first few bits reduce loss lot, with diminishing returns after that.
together as total bit count does), the overall trend will still be predictable with a functional form like ours. To test this, fit parametric form like Equation 3 with the constants A, B, E, a listed in the table. The overall fit results in values of Twr 2.811 and an exponent bias of b = 0. 240. showing the functional form is still good fit the data, even for floating point, under reasonably standard bit allocation schemes between mantissa and exponent. On the middle and right, we fit the same parametric form for particular values of (N, D) and visualize the quality of the resulting predictions.
We use bit allocations of E2MO, E3M0, E4M1, E3M2, E4M2, E5M2, and E5M6 for 3, 4, 6, 7, 12 bits, respectively, with one sign bit throughout. Since exponent and mantissa bits play in general different roles (ie. the effect of bit on loss and dynamics depends a on whether it comes from the mantissa exponent in floating point), expect our functional form does well here because mantissa and exponent allocations both increase jointly as precision rises, so overall the trends are predictable in similar way. We check directly the role of the two by sweeping ExM3 and E3Mx directly, confirming this intuition. This suggests one route for making fine-grained fits for general arbitrary ExMy combinations is decompose the effects of mantissa and weights, for instance form like Nor Pw, Pw, e, .). Since this is not needed for standard bit allocation choices as we can in Figure [6 do delve into this complexity.
O ADDITIONAL PLOTS
29

---

Published as a conference paper at ICLR 2025
5.00
Val Loss
4.50
Training-timg Effects, Prrain
Post-Training Effects, P Ppst
4.00
3.75
10
10
12
Takens (billions)
Takens (billions)
Figure 18: Illustration what fmite-preccsion effects during training and inference look like on learning curves.
30

---

Published as a conference paper at ICLR 2025
Empirical Inference-time Degradation N=30M, D=1.6
Predicted inference-time Degradation N=30M, D=1.68
PRe cne frr womeeme Ceg metetattion
port pestarain quantaation
post train quantraztion
sout
training prediion (bits)
training precision (bits)
Empirical Inference.tiee time Degradation N-60M,
Predicted Inforence. timi Degradation N-60M, D-6.6B
Deet Reeticed Ditente Uiee DOO coltio
postarain quarttattion
pozt-rriin quanttration
10
12
Ps. training (bits)
training procision (bits)
Empirical Inference-time Degradation
Predicted Inferince-tinee Degradation N=110M,
N=110M,
Trediter mm semo mr Oegradation
Sen quantitation
post.rrai train quanizzation
6.2
62
P... training predsion (bits)
Pe. training precision (bits)
Empirical Inference-time Degradation
Predicted Inference-time Degradation
N=220M, D=6 68
N=220M, D=6.60 6.68
crdd Fete Doe medttoo
quantrration
pozt train quantrration
6
Pp. training precision (bits)
P.. trtaiinnng procisian (bits)
Figure 19: Predicted vs actual ôpro for several N, D.
31

---

Published as a conference paper at ICLR 2025
P. Sweep MSE: 0.0055, R 0.9410
Pey Sweep MSE: 0.0003, R 0.9965
Predicted
3.2
4.2
4.4
3.2
3.4
3.6
3.8
4,0
4.2
Actual
Actual
Figure 20: Marginal sweeps for precision of activations and KV cache, along with predictions from an Ner functional form analogous Equation fitted from scratch.
Without Nur
With
22
12
Actual Lins
Actual kmng
Taiiin Prods.n
Empircia Lans
Predicted 4ro
Pace pottonitig
Figure 21: Combined plots for predicting degradation. (a) and (b) illustrate different fitting ap- proaches to model degradation, demonstrating a stronger when + Ne is used. (c), (d) (e) illustrate our unified degradation form can predict degradation when training and serving in any precision. Plots (c-e) made for varied Pw, but fits in (a) and (b) include runs where Pa, Pay are also jointly varied.
32


Reviews:
[{"review_id": "Et5TAvrgTm", "reviewer": "Reviewer_Xi3X", "summary": "This manuscript provides a thorough investigation into the impact of bit precision on inference performance and introduces a scaling law that correlates performance with precision. The paper is commendable for its extensive experimental validation. The study addresses a significant problem in the field of deep learning optimizations and offers practical insights for efficient model deployment. The manuscript is well-structured and the arguments are clearly presented", "questions": "potential typos: \n\n1. row303: P_a and =P_{kv} as well", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n\n1. The paper tackles an important issue with the introduction of a bit precision scaling law. While this topic has been explored before, the theoretical scaling law presented in this work offers valuable guidance for the efficient deployment of models in real-world applications. The implications of this work could be transformative for the field.\n\n2. The authors have provided a wealth of experimental results that not only validate the existing scaling laws across different model sizes but also demonstrate the generalizability of previously unseen scenarios. This thorough experimental section strengthens the paper's contributions and is persuasive.\n\n3. The manuscript is particularly strong in its methodological rigor, with a clear articulation of the scaling laws and their implications for precision in deep learning models.", "weaknesses": "no clear weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4vGzSUaOGc", "reviewer": "Reviewer_jtGw", "summary": "The paper studies the scaling law for precision, including exploring the #parameters, #tokens, pretraining precision, and inference precision.\nThe paper first introduces the background via (1) giving a decent introduction to quantization, (2) presenting the existing scaling laws on #parameters and #tokens, and (3) experimental setup.\nThen the paper introduces the scaling laws for post-train quantization, and quantized training, sharing interesting findings.\nFinally, a unified scaling law is introduced.", "questions": "I respect the amount of experiments to support that the proposed scaling law works well. However, the counterintuitive findings are more attractive to me. The paper summarizes the findings in Fig 1. Could the author further explain the underline reasons/mechanism of such counterintuitive phenomenons?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper studies a meaningful topic, the scaling laws of precision, which is a new topic following the scaling law of data and parameters.\n\n(2) The paper gives a good presentation. I especially appreciate the introduction to quantization. I'm not familiar with how quantization works in detail, so it helps a lot.\n\n(3) The paper shows interesting findings in Sec. 3.1 Fig. 2: more pretraining tokens result in lower performance for post-train quantization with a high quantization rate.\n\n(4) The paper shows interesting findings in Sec. 4.1 Fig. 3: KV cache is more sensitive to the change of precision when precision is low, but when precision is high, KV cache is more robust to the change of precision compared with weights and activations.\n\n(5) The paper shows interesting findings in Sec. 4.3 Fig. 6: there would be cases where training in low precision leads to better evaluation loss.\n\n(6) The paper generally shows that the proposed scaling law works well in the experimental setting of the paper.", "weaknesses": "(1) The paper uses the dataset Dolma for experiments. Though it's hard, it would be interesting to see how pretraining data affects this law.\n\n(2) The paper uses the OLMo-style models for experiments. It would be great to give a general introduction to OLMo-style. Are they transformer-based model? While the abstract states the scaling law for language models, there would be other types of language models other than OLMo-style models, such as SSM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQT31pQi6W", "reviewer": "Reviewer_TyGC", "summary": "This paper explores how precision -- specifically, low-precision training and inference -- affects the performance and compute cost of large language models. The authors propose new \"precision-aware\" scaling laws to predict the degradation in model performance when trained or quantized at different precision levels. Their work is motivated by the increasing trend toward low-precision training, driven by the need to reduce computational costs while maintaining model quality. While previous research has focused on scaling laws that balance model size and dataset size (for example Hoffmann et al. Chinchilla scaling laws), these do not account for the role of precision. The authors argue that precision is a critical factor that influences both compute efficiency and model performance, especially as hardware evolves to support lower precisions. They aim to fill this gap by developing scaling laws that incorporate precision as a third variable alongside model size and dataset size.", "questions": "I already specified some of them above, but the questions are particularly as in the following:\n\n- While the paper primarily focuses on integer-type quantization, you mention that floating-point quantization  is commonly used in practice, especially in pretraining. Can you elaborate on how your scaling laws might differ when applied to floating-point quantization? \n\n- You mention in the paper that activations and KV cache are more sensitive to low precision than weights, particularly when precision drops below 4 bits. Could you provide more detailed insights into why activations and KV cache are more sensitive? Is this primarily due to the per-tensor vs per-channel quantization method, or are there other factors at play?\n\n- Your experiments are conducted using specific hardware such as Nvidia H100 GPUs. How do you expect the scaling laws to generalize across different hardware architectures, especially those that may handle precision differently, for example future GPUs with native support for FP4 or binary/ternary quantization?\n\n- Given that your largest model size is 1.7B parameters, do you anticipate any limitations or deviations from your scaling laws when applied to much larger models with hundreds of billions or trillions of parameters?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a new dimension to the well-established scaling laws by incorporating precision as a critical factor. This is an important contribution because most prior work focused on model size and dataset size without considering precision, which is increasingly relevant due to hardware advancements supporting lower-precision computations. By doing so, the authors offer a more comprehensive framework for understanding and optimizing model performance under different training and inference conditions.\n\n- The authors fit on over 465 pretraining runs across different precisions (3-bit to 16-bit) and sizes (up to 1.7 billion parameters), providing a robust dataset to validate their proposed scaling laws. The empirical results are consistent with the theoretical predictions, achieving high R^2 values (e.g., R^2 = 0.97 for post-training quantization degradation). \n\n- The paper offers actionable insights into how low-precision training can be compute-optimal, particularly in scenarios where hardware constraints or cost considerations are paramount. For example, it shows that training larger models at lower precision can sometimes be more efficient than using higher precision, which is a valuable insight for practitioners looking to optimize both performance and computational costs.", "weaknesses": "- While the paper focuses extensively on integer-type precisions (e.g., 3-bit, 8-bit), it does not explore floating-point types like FP8 or BF16 in as much depth. Given that floating-point formats are widely used in modern hardwares, this omission limits the generalizability of the findings to real-world applications where floating-point precision is common. This could limit the applicability of the scaling laws in environments where floating-point precision dominates, potentially requiring further research to adapt these findings.\n\n- The experiments are conducted on specific hardware setups that support low-precision computations, such as GPUs optimized for integer-type operations. The fitted constants and trends may not generalize well across different hardware architectures or future technologies that handle precision differently. This may reduce the long-term relevance of the paper’s findings as hardware evolves.\n\n- Maybe I'm missing this, but the paper suggests that compute-optimal precision is around 8 bits but does not deeply explore scenarios where precision drops below 4 bits (e.g., binary or ternary quantization). Given that future hardware may support even lower precisions, this limits the scope of the findings.\n\n- While pretraining cost optimization is thoroughly explored, inference-time costs -- especially in real-time or latency-sensitive applications -- are not given as much attention. In many practical deployments, inference-time efficiency is more critical than pretraining cost savings. This imbalance might limit the practical applicability of some of the findings in scenarios where inference-time efficiency is more important than pretraining considerations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1zAK6vPWsA", "reviewer": "Reviewer_heoa", "summary": "This paper propose the scaling laws for precision through replacing the N in the original Chinchilla with the effective parameter count $N_{eff}$ and adding the post-training effects.", "questions": "As shown in above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThe proposed scaling law unify the post train quantization and quantized training into a single functional form.\n2.\tThe finding in the section 4.3 is inspired and the conclusions are consistent with usual experience and give a theoretical explanation.\n3.\tThe experiment is adequate and reasonable and the paper is well written.", "weaknesses": "1.\tThe paper use the $N(1-e^{P_{w}/\\gamma_{w}})$ to fit the left in the figure 3. But I think the power law is the most commonly used in all kinds of scaling law form. I suggest the author could compare the exponential with power law like $N(1- A*P_{w}^{\\alpha})$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)