Unified Scaling Laws for Compressed Representations
Authors: Andrei Panferov
Alexandra Volkova
Ionut-Vlad Modoranu
Vage Egiazarian
Mher Safaryan
Dan Alistarh
Venue: NeurIPS 2025 poster ()
Keywords: scaling laws
large language models
model compression
quantization
sparsity
Primary area: deep_learning
TL;DR: We investigate new scaling laws which predict the scaling of LLMs when training them over quantized or sparse representations.

Abstract: Scaling laws have shaped recent advances in machine learning by enabling predictable scaling of model performance based on model size, computation, and data volume. Concurrently, the rise in computational cost for AI has motivated model compression techniques, notably quantization and sparsification, which have emerged to mitigate the steep computational demands associated with large-scale training and inference. This paper investigates the interplay between scaling laws and compression strategies, exploring whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations, such as sparse, scalar-quantized, sparse-quantized or even vector-quantized formats. Our key contributions include proposing and validating a general scaling law formulation applicable both individually but also composably across compression types. We demonstrate both theoretically and empirically that a simple metric based on Gaussian mean squared error fitting can robustly predict parameter efficiency across compressed models. Additionally, we extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats. Finally, we identify conditions under which these unified scaling laws fail.

Unified Scaling Laws for Compressed Representations
Andrei Panferov*
Alexandra Volkova'
Ionut- Modoranu
Vage Egiazarian ISTA
ISTA
ISTA
ISTA
Mher Safaryan
Dan Alistarh ISTA & Red Hat Al
ISTA
Abstract
Scaling laws have shaped recent advances in machine learning by enabling pre- dictable scaling of model performance based on model size, computation, and data volume. Concurrently, the rise in computational cost for AI has motivated model compression techniques, notably quantization and sparsification. which have emerged to mitigate the steep computational demands associated with large-scale training and inference. This paper investigates the interplay between scaling laws and compression formats, exploring whether a unified scaling framework can accu- rately predict model performance when training occurs various compressed representations, such as sparse, scalar-quantiee sparse-quantized or even vector- quantized formats. Our key contributions include validating general scaling law formulation and showing that is applicable both individually but also composably across compression types. Based on this, our main finding is demonstrating both theoretically and empirically that there exists simple "capacity" metric based on the representation' 's ability to fit random Gaussian data data-which which can robustly predict parameter efficiency practical side, we extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats. Our source code is available at: LST-DASLab/ unif ied- Sc-laws
multiple compressed representations. On the
1 Introduction
A key recent advance in machine learning has been the idea of predictable scaling of learning performance with respect to model, computation and data sizes. This approach is encompassed by the scaling laws Kaplan et al. [17], which allow researchers to predict the values of these three parameters required reach certain model performance. This powerful idea has been ex expanded
upon in several directions, e.g. [1 22 and is key ingredient behind the massive expansion of
computational power for AI 112
parallel research direction, motivated by this massive increase in computational cost, has been model compression, which proposes series of techniques to reduce the computational and memory footprint of model inference and training, via techniques such as sparsification [14] and quantizution [[]] In this paper, we focus on the interplay between scaling laws and the degree of compression of the representation over which learning occurs. While there is significant emerging work in this direction. e.g. 19 33) 31), current scaling laws are specialized to single representations (c.g., quantization or sparsity) and/or formats (e.g., integer quantization), and cannot yet address the question of predicting model scaling behavior when training over general compressed representations.
"Equal contribution. Corresponneece to: dan. alistara @t
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

Contributions. This paper is structured two main questions, and their practical ramifications:
Q1: Is there unified compression scaling law? First, wish to find single general law that not only applies to sparse [9] or quantized [19] representations in isolation, but that also provides a good fit for hybrid formats, such as sparse-nd--uanniinn weights, or compound compression, i.e. sparse weights and activations. Through extensive experimentation, we identify this law to be of the form
B- D-.D-BB Loss (N, D) A (N. BB F
where N is the number of model parameters. D is the dataset size, E is the irreducible error, A. B, and B are constants, and p is a parametric function of the representation R. Crucially, we find that, even for very complex representations- e. 3-bit quantization with group size 32 and 1% outliers in full-precision-thee parametric function P can still predict the scaling of model performance w.r.t. the parameter count N. We call p(R) the representation capacity of R. Consequently. there is always a "dense equivalent" parameter count N' N p(R) which would yield the same loss during training. The capacity p(R) lies naturally in the interval (0, 1]. and the key goal of compression is to maximize the trade-off between model accuracy and the size and computational cost of the representation.
Q2: Is capacity an "intrinsic" property of the representation? While related forms of the above law have been proposed in prior work [10] 19], we are the first show that capacity is an intrinsic property of the representation, independent of the model and task for which the scaling law is obtained, but relatable to standard information-theoree applicability of the law across hybrid (e.g. sparse-quantized weights) composite (e.g. quantized representations.
measures. Moreover, we establish the
Beiightsand-activall
More precisely, our main finding is that capacity is tighlly-orrelated with the representation's 'ss ability to fit random Gaussian data, measured in terms of minimal mean-squan muann-squared error (MSE). Concretely, p(R) is a simple parametric function of the MSE of the representation R when fitting random Gaussian data, i.e. p(R) = ğ( (R)) where instances of the same representation R. e.g. and 4-bit integer quantization, share the same parametric form p. This finding, which we validate across quantized, sparse, quantized-sparse, and even vector-quantized representations, provides simple metric to "rank" different formats implementing the same representation. In addition, this also allows us to determine the "optimal" capacity certain bit-width, which is given by theoretical bounds on Gaussian fitting for a given support, which can be casily estimated via Monte Carlo algorithms. In addition, we also provide non-trivial theoretical justification for this relationship in Theorem] [ for Adam-optimized compressed models: we relate the convergence of Adam over compressed representations with the product between the number of parameters N and the average root -quared error of compression across optimization, which connects to our notion of capacity.
Our second finding is that, except for pathological cases, capacity factorizes across composite representations: concretely, the capacity of 4-bit and 2:4 sparse model is the product between the capacity of the 4-bit dense model, and that 2:4-sparse but unquantized model. Factorization allows us to evaluate the capacity of complex representations based on simple ones, and also holds when compressing different model representations, e.g. both weights and activations.
Practical Implications. The analytical metrics suggested by representation capacity also have non-trivial practical applications. First, the fact that we are able to relate the predictive parameter P to intrinsic properties of the underlying representation gives us the ability to analytically predict accurately
the representational power of different compressed numerical formats. This way, compare and predict the efficacy of various formats such as Pooting-PPintt Integer (INT with and without grouping), or sparse- quantized formats (2:4 + INT) at different compression budgets. Second, this framework inspires an improved approach for sparse training, which we show provides significant improvements (above 20% in some sparsity regimes) in capacity at the same number of parameters.
Overall, our results provide a new lens to view the scaling properties of compressed models, with respect to intrinsic properties of the representation over which training is performed. Thus, we believe that capacity-aware scaling has the potential to become practical design principle for the next generation of efficient foundation models.
2 Preliminaries
Scaling Laws. We start from the "Chinchilla" scaling law formulation [15] that proposed to model loss scaling function of the number of parameters in the model N and the number of data points

---

Table Representation scaling laws (rows) versus the quantities of interest (columns). For all laws, N represents the number of parameters, D is the data, and E is the irreducible error. For the sparsity scaling law of Frantar al. (8) S is the sparsity and the lowercase parameters
learnable constants. For the precision scaling law of Kumar el al. [19] Pa: is the weight precision, and ?pP is a learnable weight sensitivity parameter. For the law of Frantar al. [10], eff:: is the "effective parameter multiplier," that is explicitly fitted for every instance of compression C. By contrast, our formulation postulates that the parameter efficiency is simple parametric function of the representation's 's capacity to fit random Gaussian data (CMSE(R).
Sparsity fit Quantization fit (Error)
Parametrization
Formulation for Loss(N, D)
(Error)
Sparsity S
as(1 S) +
aan
+ E D
bp
N/A
Frantar et al. 9
NbN
Quantization to P.
A[N(1 - -Po/ }] -a BDD +
N/A
4.5 10-3
Kumar al. 19
Compression C
B
+E
4.2-10 10
1.9 100-
Frantar er al. 10
effe D
Representation
B
+
E
4.7 17-1
2.1 10-3
(OURS)
(N 7 (M))(
DB
D the model was trained on, in the form the parametric function:
Loss D) AN- + BD + E,
(2)
where A, B, E, a, and 33 are the scaling law parameters that can be fit empirically, It is important to note that such scaling laws assume an ideal, well-tuned training setup, and that the parameter may vary slightly depending on architecture, optimizer, and parameerr......
Compressed Representations. For sparsity, we assume that a specific fraction, within each parameter group of certain size G, is set zero. Sparsity is unstructured if the group is the whole tensor, whereas it is semi-structured (N:M) if N parameters out of every M are set to zero. For quantizaaion, unless otherwise stated, we assume that parameters are mapped onto scalar, symmetric grid corresponding to the number of bits available for quantization, as is standard [ل]. (We will also consider vector quantization in Section 4.1) For sparse-quantized representations, we follow [13 by first applying sparsification, and then quantization, to map continuous parameters onto this format.
Prior Scaling Laws. The relationship between the learning representation and the the scaling law formulation considered by Frantar et al. [9] for sparsity, and by Kumar al. [19] for quantization. The scaling laws they propose are described in Table together with their parametrization, for the special case of weight-only compression. While both these laws can predict loss with respect to training over their target representations, their formulation is not designed to generalize to other representations, or to hybrid ones (e.g. sparse quusse-quantized).
The unified law we consider extends preliminary work by Frantar al. [10) who, assuming that training happens over weights compressed in representation C. proposed a simple parametric law similar to Equation[] []] but which is fitted independently for each instance of compressed training, yielding a value of the corresponding parameter efficiency factor, called eff Frantar al. [10] focuses on quantization; they fit sparsity in limited experiments, and do not consider hybrid formats.
Our Approach. By contrast, the representation R: in their parlance, we show that, across all instances of a given compressed representation Rt e.g. uniform integer (INT) quantization, the parameter efficiency has the same parametric form p (R), and, in fact, this parametric form is simply a function of the MSE for the representation R w.r.t. random Gaussian data, i.e. p(It) pl GM))( Importantly, GMSE Rt) an intrinsic property of R, and only depends on its own parametrization: for INT, this would be the number of bits we employ per parameter.
focus is relating parameter efficiency intrinsic properties of
The fact that this parametric form is shared across instances of the same representation (Section 4.1 is powerful since it allows us to compare and transfer parameters between instances of the same representation R. Clearly, if GMSE ~ 0, then p(P) 1, and recover the original "dense"
3

---

scaling law [5]. Interestingly, Table I shows that our unified law can provide better fit than the representationnspeeciii formulations of Frantar et al. [9] and Kumar al. [19], and (almost) matches the formulation of [00]. which is fitted for each compression instantiation C..
Setting for Experimental Validation. For our scaling law investigations, we pretrained decoder-only
Transformers following the Llama architecture [44 for 30M, 50M, 100M and 200M non-embedding parameters. The models were trained on the C4 dataset (28] using the Llama-2 tokenizer [34). To ensure we operate in a data-rich regime, we use 50, 100, and 200 training tokens per model parameter for each training configuration. and train on fixed-length context windows of 512 tokens. We used
AdamW [8] 23 with 0.1 ratio of warm-up epochs with cosine scheduler. Our experimental setup is very similar to that of (9] 19 10I
More details are provided in Appendix [
We follow standard quantization-awan unstructured weight sparsity. For quantization we employ the gradient estimator of [27], per-layer uniform quantizer with static scaling factors and gradient masking. Quantization levels range from 1-bit to 8-bit precision. We consider configurations with quantized weights only, activations only, and both simultaneously. For sparsity, we apply unstructured magnitude pruning via top-k thresholding on per-layer basis. The sparsity mask is recomputed dynamically each optimization step. For Vector Quantization (VQ), we follow QuEST scalar quantization and apply it to 2- and 4-dimensional HIGGS grids (24). To restrain outliers we use the trust estimation method 27 that zeros out gradients for any point lying outside a hypersphere certain radius.
training (QAT) methods, combined with various levels of
3 Theoretical Analysis
One key focus of our work is whether, given a compressed representation R over which learning is performed, we can identify a predictive metric that correlates with the representation's 's efficiency p(R). To identify this metric, we first model the standard weight-compresee LLM optimization process, which combines the Adam algorithm have:
[8] with the straight-through estimator (STE) [ We
STE Gradient: Jt v(()),,
where O. = C(O.),
Optimizer states:
m
3mm--1 + (1 B1)
Ut B2u--1 + (1 32)9i
T max(Ut) 11
m
Model update: 0,+1 = 0, Vūr E
where V represents stochastic mini-batch gradient operator, C: R.V - R.V the compression scheme, 31, A3 E (0, 1) are momentum parameters, e >> 0 is a small constant for numerical stability and 17 >> 0 is the learning rate or the step-size. All vector operations are element- including the max operation. Our analysis relies on the following assumptions, which are standard in adaptive
optimization 336 29
25 30)
Assumption (Lower bound and smoothness). The loss fanction f: R.V - lower bounded by some 1 E RR and is I--mmooth, namely, [[V - V f(0')ll2 I T. 0'112, for any 0, 0' e N
Assumption 2 (Unbiased and bounded stochastic gradient). For all iterates t 1, the stochastic gradient 9t 0, unbiased, E[ge] ( and uniformly bounded, ||gellzo S Goe, by some constant Go 2 O.
Assumption 3 (Bounded variance). For all iterates t 2 1, the variance of the stochastic gradient gr 0, is uniformly bounded by some constant o² 2 0, namely Elllg: (0))) S 0².
In this context, our main claim is the following:
Theorem 1 (Non-convex convergence analysis). Let Assumptions 7 <(1 <(1-81)
2 and 3
hold. Then, choos-
ing step-size 1 min(no. /T with no 2LCVN and C = 2 G2 + +/NN, a randomly chosen
compressed iterate θ from {01 (0...... ērY satisfies
CLG.o Ve
1
CVN /T
f(()) fr Lo
4 T
| <
EE
0, Bell2
.
+0
T
Discussion. Similar convergence analysis for Adam under compressed iterates 0, was performed in the setup of convex online learning with bounded domain condition (16), and in nonconvex

---

1.0
QUEST
0.8
LSQ
((R)
0.6
p(R)
p(R)
Decoupled fit Kurna et al. fit Our Fit
0.2
x
Naise Injection
x
Noise Injection W+A
QUEST
QUEST W+A
0.0
10
10-3
10-2
10-1
100
10
10-3 10
10
109
Bitwidth (a)
GMSE (b)
GMSE
(C)
Figure 1: Comparison of fits for scaling law forms from Table | (a, left) shows quantizations scaling laws, (b, middle) and (c, right) demonstrate the match between noise injection and QuEST quantization for weight-only and weights+activatioon quantization.
optimization with restricted hyperparameter choices and slower rate in terms of constants and extra log-terms (3]. We now interpret this bound, whose complete proof can be found in the Supplementary. Specifically, the bound shows the ergodic convergence of the gradients taken at compressed iterates, which is the strongest property shown even in the uncompressed case [36]. In turn, this term is bounded by 3 terms on the RHS. The second and third bounding terms standard in the analysis of uncompressed Adam [6]., showing that our analysis fairly tight. We focus our attention on the first term, whose key part is highlighted in blue: this term consists of absolute constants, multiplying the
critical term N-E E ET= 11. Bell2] Specifically, this term consists of the number of parameters
N times the average & compression error over the model parameters throughout the execution. Thus, this analysis suggests that the parameter efficiency of such models may depend multiplicatively on both the number parameters, and the compression root-mean- square-error root-mea--square-errrt (RMSE) throughout the execution. Importantly, this bound is independent of the compression type.
4 Findings
4.1 Finding 1: Gaussian RMSE Predicts Representation Capacity
Table presents Trrssensssentss number of scaling laws that model the same functions via different parametrizations. One can notice, that both the Sparsity form of Frantar et al. [9] and the Quantization form Kumar et al. [19] can be reduced to the Decoupled form of Frantar et al. [10] in the third row, by imposing - e PP// for quantization). Naturally, the Decoupled form can achieve lower fit error, but does not provide any information about the interpretation of the capacity term, which we call p pR)), across different representations R. The Sparsity form and the Quantization form, on the other hand, feature intertwining and interpretable parameters. For simplicity, we first focus on the Quantization form for now.
additional constraints (e.g. effp
The Functional Form. Kumar al. [19 choose the functional form p(Par) = 1 e-e to model quantization efficiency. By contrast, we propose a different form model p(R):
pl GMSE(R)) R)) L tanh F log (GMSE(RR))
(3)
which depends only on the representation R's Gaussian-MSE fit, denoted by GMSE(R). and on the scalars L. F, and C, detailed below. The GMSE Rt) is easily computable for any representation, and allows us bypass the dependency on eppresentationnspeciii sparsity. Specifically, we fit the scalar parameters for each compression type, e.g. scalar quantization, and then re-use these parameters while varying GMSE(R) w.r.t. compression parameters, e.g. bit-width. The scalar parameters L, F, and C' allow us to accurately model observed effects such as:
parametrization, such as bit-width or
Imperfect convergence in high- precisioe:::: While modern QAT algorithms such as QuEST reach efficiency 1 for low quantization error, older algorithms such as LSQ (Figure (a)). have an efficiency limit strictly below 1, since for instance its gradient estimator introduces consistent bias. The factor J., defaulting to 1 for saturating representations, allows us to model this imperfection.
Various low-precision curvature: As seen in Figures (b) and (c), different representations be- have differently around GMSE 1, with some have noticeably higher curvature (" ("braakdown").
5

---

From Figure (a), one can see how that region disproportionally affects the law of Kumar et al. 19 leading to a very poor fit at higher bitwidths. The factor C, closer for representations "more linear" around GMSE = 1, allows us to more accurately model p(R).
Quality of Fit. Table I shows that our approach leads similar or better quality-of-fii relative to prior laws, covering both scalar quantization and sparsity, while Figure [ shows p (t) alignment between scaling law forms, compared Kumar al. [1]. for the QuEST and LSQ quantizers. Again, our approach provides significantly better fit. In Figure 2 a), we show that our method can also provide a good fit for models trained with vector- rector-quantized (VQ) weights, using the projection method of [2]] for lattice dimensions 2 and 4. This shows both the versatility of our approach, and the necessity of the L term, since ligee-dimennional VQ appears to have clear sub-unit saturation due to higher bias. We provide further examples in Section
4.2 Finding 2: Noise Injection as a Scaling Law Predictor
Next, we turn our scaling law on its head, and ask: what if we plug the optimal achievable GMSE for certain bit-width into the scaling law? In that case, the scaling law should allow us to compute a lower bound on the achievable parameter efficiency given a certain type of representation. In turn, we can find out how close existing quantization- sparsity-aware training techniques, numerical formats, are to the information-theoreticc lower bound for that specific representation.
Figure I (b) illustrates the "optimality gap" for the QuEST algorithm for scalar weight-only quan- tization across bit-widths, suggesting that this approach is fairly close optimal. In Figure (c), we compare the fit between actual runs of this QAT algorithm across bit-widths, and the predicted values via noise injection [1] (plugging in the equivalent GMSE) into the scaling law, showing near-perfect fit.
(c).
4.3 Finding 3: Representation Capacity Is Multiplicative Across Compression Types
In prior work, [19] have claimed that, for their formulation of the law, the representation capacity factorizes independently for quantization of weights and activations. Our experimental findings extend this result, showing that representation capacity, p(I),, also factorizes naturally across a wide range of compression approaches, whether for the same tensor (sparse-and-quantzee weights) for different state tensors (sparse weights and sparse activations). We follow the experimental setup from Section 2 training models with sparse weights and activations, or sparse-quantized weights, both. We fit a scaling law in the 100 toks/param regime. We show that representation capacity factorizes for the following scenarios:
1. Sparse weights and activations: For sparsity, independently applied to weight and activations,
p(Rs.....) = p(Rs...) p(Rs...
We summarize the fitted values of p(R) levels in matrix M (Figure b)), where each entry corresponds to the fitted efficiency for a model trained with a specific sparsity configuration. Remarkably, the matrix can be accurately approximated by a rank-1 outer product of the first column Mo.: (weight-only) and the top row M;, 0 (activations-only elements, i.e. M s Mo, & Mo,-- The resulting parameter efficiencies closely match the product of efficiencies obtained for runs with welght-only and activations-onf, configurations.
2. Sparse and quantized weights: Given weight sparsity level s combined with q-bit quanti- zation, we claim that the representation capacity can be represented as the product: p Rg.s) p(R,)= p(..). We report the results for different sparsity levels and bit width in Figure 9 Similarly, the matrix p(R) factorizes into the outer product of marginal vectors for quantization- only and sparsity-only representation. Apart from extreme quantization to 2-bit precision, the approximation maintains the error of order of 10-2
3. Sparse and quantized weights, and quantized activations: Finally, observe that factor- ization extends to quantization of activations as well. In supplementary experiments, we apply quantization to activation tensors alongside with weight sparsity and quantization. Our results indicate that the representation capacity with weight sparsity su and quantization bitwidth qu. and activation sparsity yo follows p( (m) p( (R pl ((....) p(Bq..).
This result allows for low-cost comparison across compression comparison. Morcover, it facilitates compression hyperparameter tuning and thus predictable model training in a compressed regime.

---

True plP)
Approximated piP)
Absolute error
0.50
1.00
Weight Sparsty
Weight Sparsity
Weight Sparstt
0.2
0.84
0.84 0.42
0.27 0.17
0.84
6.63 0.42 6.27
0.56
0.2
6.00
0.01 0.01 6.01
0.6
0.15
0.47
0.36
0.25 0.16 0.12
66
0.47
0.36 0.24 0.15 0.09
0.00
0.02 0.01 0.03
0.-
dm=z
dimed
0.8
0.27
0.21 0.16 0.12
0.08
0.8
0.27
0.21 0.14 0.09 0.05
0.8
0.01
0.02 0.03 0.03
0.8
0.0
10
15
MSE
10
12"
Activation Sparsity
Activation Sparsity
Activation Sparsity
(a)
(b)
Figure 2: (a) Scaling law for 2- and 4-dimensional vector quantization. (b) Representation capacity across weight and activation sparsity levels: baseline, factorized prediction, and relative errors. Note the low errors for the factorized predictions, with slight increases at the larger sparsity levels.
GMSE, Val Loss by Floating Point Configuration
D4N1
Compression
GMSE
10
FaV3
E3N4
12N5
INTR
10-1
bits
bits
bits
0.90
3.24
F-N1
EOM1
0.05
FP4 vartants
FP6 variants
0.80
3.22
E392
5.255
NTS
ESM2
FIH5
Noise injection
4 bits
bits
bits
4
10
Figure 3: Comparison of floating point and integer data-types terms of GMSE, and C4 Validation Loss when trained using the corresponding formats via QuEST, and the resulting capacity p(R). Observe the high correlation between ranking in terms of GMSE (top), and Val. Loss (bottom).
Applications
5.1 Application 1: Comparing Compressed Numerical Formats
Practical Formats. The scaling law for compressed representations enables systematic comparison of numerical formats used in quantization, such as INT8, INT4, FP4, or custom low-precision representations, based just on their GMSE, which can be determined via fast Monte Carlo algorithms. Thus, provides clear guidance on which low-precision format delivers the best performance for given resource constraints. Figure 3 iillsstrate this for number of floating-pone and integer
0.9
Bits:
1.0
Dutliers:
0.8
+
5.0%
0.8
0.7
0.6
Groups
0.6
16
64
0.2
10-2
10
10-4
10-3
10-2
10
MSE
MSE
Figure 4: Representation capacity R() versus MSE for (a) group-wise quantization, with markers indicate group counts (color encodes quantization bitwidth), and (b) outlier-aware quantization.

---

(a)
(b)
(c)
(d)
Figure Backward heuristics: (a) forward mask Mew determined by the T3 threshold, (b) backward mask determined by threshold Tp. (c) banded backward mask determined by both T3 and Tp. (d)
capacity plot compared the Magnitude Pruning bascline with MBw -Mww
data-types. Specifically, we observe direct correlation between the ranking of GMSE values (top) and the actual C4 validation loss obtained in actual experiments (bottom). This suggests that our GMSE metric is an accurate predictor of compressed pre-training performance. For instance, it suggests that switching to FP4 (E2MI) will not bring gains relative INT4 training, and that both formats are close to the theoretical lower bound at bits.
The Impact of Parameter Grouping and Outlier Preservation. A related question regarding formats is whether more complex approaches, such as group wise quantization, or outlier preservation in higher precision, can disrupt the scaling law. We examine this in Figure 4 which it shows that preserving no outliers (0 %) lies on the Pareto-optimal boundary: higher outlier ratios ratios achieve worse trade-off between the MSE and the representation capacity p R).. This suggests that, for pre-training is more effective allocate bits encoding the values distribution rather than outlier preservation or careful grouping. This further demonstrates that the proposed RMSE dependency is a general property and remains valid even under diverse structured compression techniques.
Compositionality, An immediate practical application of the multiplicative behavior of the law (Section 4.3) is the ability to estimate the model's performance in advance for arbitrary compression configuration. Given the individual efficiencies of different compression methods, such quanti- zation or sparsity, applied to weights or activations, one can predict the combined effect without spending additional compute for training.
5.2 Application 2: Increasing the Capacity of Sparse Training
The Sparse Training Problem. For our second application, investigate implications of the RMSE law to maximize the capacity of a sparse representation during training. Specifically, standard sparse training methods such as gradual magnitude pruning (GMP) [37|9] compute forward sparsity mask, which we denote by Mew during the forward pass based on the absolute-magnitdde Top-K operation applied to the model parameters 0 with respect to a target sparsity. Then, a gradient (oTo (0)) taken w.r.t. the sparsified weights. Standard basclines, such the ones we use for sparse training, re-use the forward sparsity mask for the backward, preventing the pruned weights from receiving any gradient. We are interested in heuristics to improve the parameter efficiency of this standard approach, increasing capacity at the same sparsity level.
RMSE-Banded Gradient Masking, For this, we follow the RMSE law and align the parameters R with the standard normal distribution by dividing / by its root mean square RM
N E 0z which results in [10/RMS(0)) deviation parameter p E 0.5), which determines the threshold for the backward mask T RMS(() (0.5 + p), where p is the inverse cumulative distribution function of the standard Normal distribution. The multiplication by RM 5(x) "converts" the threshold for the standard Normal distribution to the threshold for the vector 0. As a result, M = 10] > Tp-
= N. We allow the user to provide a median
Effectively, our approach, which we call RMSE-Banded Backward Masking (RBBM), sets a backward mask Mnw that may be different than the TopK mask for the forward, whose sparsity controllable via the parameter p. To address the fact that it may not allow gradient flow for small parameter values, we allow gradients to flow for the smallest and largest parameters, and create band between Tp and the TopK threshold Th, where we do not allow gradient flow. Let m = min(Tp, (TnpTTT Tk) and M max(T, p. TR) and define Maw = (10) < m) V (10) > M).. Since do not control the
8

---

relationship between Tр and Tk. we need to ensure that the band is defined correctly. Concretely, the values 0g < m and values 0, > M will receive gradients, while the values 8, E [m, M] will not.
To illustrate, in Figure Sa we show the structure of forward mask MEW. were the red region corresponds to values 10,1 < T3 that will not receive any gradient, while the green region corresponds to the values |0:| 2 T. which will receive gradient. The top-k threshold T; fixed. In Figure 5b we have a similar behavior for the backward mask MBW determined by the threshold Tp. which 1s now user-controlled via the median deviation parameter p. In Figure 5e we show an example for T < Ts. where we obtain a banded-mask: the values |0,| E [Tp. Tk] will not receive any gradient (red region), while the other values will receive gradient (green region). The band width can be controlled via the parameter E (0, 0.5). When is close to zero, the TT value will decrease, having the effect of increasing the width of the red band, where the corresponding weights do not get gradient. When p = O. the value of T will be equal to the median and this will be equivalent to the baseline (e.g. MEw MFww illustrated in Figure Sa
Results. We apply RBBM for sparse training in our pretraining scenario, for the 30M parameter Llama model, using our training setup from Sec. [ and for unstructured sparsities between 10% and 90%. We compute the capacity of the sparse representation. The results for our RMSE-based heuristic and the standard sparse training baseline (Magnituee Pruning) are provided in Figure (d The results show that our RMSE-based approach enables consistently higher capacity than the baseline.
6 Related Work
We focus on studies that extended classical scaling laws [7 15 to model compression. Frantar et al [9] presented the first scaling law for weight-sparse Transformers, across vision and language and unstructured and stritstructured sparsity. Earlier work by Clark er al. (5] studied mixture-ofeexpens sparsity, deriving scaling laws in terms of total parameters and compute per token, reinforcing the idea that only effective parameters govern scaling.
A recent breakthrough by Kumar et al. [9] introduced scaling laws that incorporate numerical quantization during training and inference, showing that, as for sparsity, a model trained in low precision behaves like smaller quantization (PTQ), showing that PTQ quality worsens as training data increases. For training- training-time quantization, their laws suggest that using lower precision allows training larger models on the same compute budget. Relative to their pioneering work, we bring the following improvements. First, investigate different and arguably simpler scaling law, showing that it yields a considerably better fit for quantization itself (see Table D. Second, our key focus is different, we provide a first interpretation of the notion of representation "capacity"", together with theoretical justification, and ample experimental validation. Finally, we validate the factorization property posited by Kumar et al. [191 as well as extensions to hybrid formats. Follow-up work by Sun et al. (33) examines scaling laws for floating-poin (FP) formats, finding that the law of Kumar et al. [19] does not provide a good fit in this case, and investigates an extension of the law via additional parametrization.
model. They also apply their approach to post-training
Preliminary work by Frantar et al. [10 proposed the single- parameerr scaling law on which build, and showed that it can be applied to instances of weight quantization and sparsity, by directly fitting the efficiency parameter. By contrast, identify a general law, in the sense that the same parametric form can transfer between compression types, to hybrid sparse-quantized formats, as well as to instances where both weights and activations are compressed. More interestingly, we equate the representation capacity factor in the law with a natural notion of representation capacity, show that the law factorizes across representations. In concurrent work, ParetoQ (22) aimed to unify the fragmented landscape of LLM quantization by systematically evaluating the interplay between training strategy, quantization function design, and bit selection. Our results complement their findings: for instance, we obtain that, for the architectures we consider, 2-bit weight-only quantization is Pareto optimal.
7 Discussion and Limitations
Our study introduces representation capacity-roughly defined as a simple monotone transform of the Gaussian MSE- unified metric when training compressed models across various representations. Capacity enables format comparisons without retraining or exhaustive grid searches, so that future hardware designers can expose any format whose capacity dominates the Pareto fronticr, confident
9

---

that software will exploit it optimally. Moreover, our law factorizes, further simplifying the search for the "optimal" training format.
few caveats remain. First, in line with prior work in this area, our experiments are limited to decoder-only Llama-style architectures trained on C4 in the data-rich regime (100 toks/param); we plan to extend this at larger scale. Second, the law may need specific fits for ultra-low precision (e.g. 2-bit or ternary formats) and for vector-quantization codebooks below 8 entries, suggesting second-order effects may need to be taken into account. Third, while our theoretical evidence uses standard assumptions, it could be extended to more complex representation types.
Acknowledgments
We would like to thank Lambda Cloud for their generous computational grant. We thank the NVIDIA and Google corporation for their grants, which supported part of this research. Alexandra Volkova and Andrei Panferov were supported in part by the BILAI Cluster of Excellence program.
References
[1] Baskin, C., Liss, N., Schwartz, E., Zheltonozhskii, E., Giryes, R., Bronstein, A. M., and Mendelson, A. (2021). Uniq: Uniform noise injection for non-uniform quantization of neural networks. ACM Transactions on Computer Systems (TOCS), 37(1-4), 1-15.
[2] Bengio, Y., Léonard, N., and Courville, A. (2013). Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /.0.:333
[3] Chen, C., Shen, L.,, Huang, H., and Liu, (2021). Quantized Adam with Error Feedback. arXiv preprint arXiv:2004. /4/80.
[4] Chen, X., Liu, S., Sun, R., and Hong, M. (2019). On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization. International Conference on Learning Representations.
[5] Clark, A., de Las Casas, D., Guy, A., Mensch, A., Paganini, M., Hoffmann, J., Damoc, B., Hechtman, B., Cai, T., Borgeaud, S., el al. (2022). Unified scaling laws for routed language models. International Conference on Machine Learning, pages 4057-4086.
[6] Défossez, A., Bottou, L.,, Bach, F., and Usunier, N. (2022). simple convergence proof of adam and adagrad. Transactions on Machine Learning Research.
[7] Diao, S., Yang. Y., Fu, Y., Dong, X., Su, D., Kliegl, M., Chen, Z., Belcak, P., Suhara, Y., Yin, H., el al. (2025). Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training. arXiv preprint arXiv:2504.15 13161.
[8] Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. (2022). Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17325
[9] Frantar, E., Ruiz, C. R., Houlsby, N., Alistarh, D., and Evci, U. (2024). Scaling laws for sparsely-connected foundation models. In International Conference on Learning Representations.
[10] Frantar, E., Evci, U., Park, W., Houlsby, N., and Alistarh, D. (2025), Compression scaling lawstunifying sparsity and quantization.
[11] Gholami, A., Kim, S., Dong, Z., Yao, 2., Mahoney, M. W., and Keutzer, K. (2022). A survey of quantization methods for efficient neural network inference. In Low-power computer vision, pages 291-326. Chapman and Hall/CRC.
[12] Gibney, E. et al. (2022). How to shrink ais ballooning carbon footprint. Nature, 607(7920), 648-648.
[13] Harma, S. B., Chakraborty, A., Kostenok, E., Mishin, D., Ha. D., Falsafi, B., Jaggi, M., Liu, M., Oh, Y. Subramanian, S., and Yazdanbakhsh, A. (2025). Effective interplay between sparsity and quantization: From theory to practice. In International Conference on Learning Representations. arXiv:2405.20935.
10

---

[14] Hoefler, T., Alistarh, D., Ben-Nun, T., Dryden, N., and Peste. A. (2021). Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22(241), 1-124.
[15] Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hemnigan, T., Noland, E.. Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L. (2024). Training compute-optimal large language models. In Advances in Neural Information Processing Systems.
[16) Hou, L.,, Zhang, R., and Kwok, J. T. (2019). Analysis of quantized models. In International Conference on Learning Representations.
[17] Kaplan, J., McCandlish, S., Henighan, T.,, Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001 1.0361.
[18] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. International Conference on Learning Represeniations (ICLR). 1412.61411.690.
[19] Kumar, T., Ankner, Z., Spector, B. F., Bordelon, B., Muennighoff, N., Paul, M., Pehlevan, C., Ré, C., and Raghunathan, A. (2024). Scaling laws for precision. arXiv preprint arXiv:2411.04330.
[20] Li, X. and Li, P. (2022). Analysis of Error Feedback in Federated Non-Convex Optimization with Biased Compression. arXiv preprint 1rXiv:2211.1422
[21] Liu, Z.,, Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning. J., Cao, Y., Zhang, Z., Dong, L., al. (2022). Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 12009-12015
[22] Liu, Z., Zhao, C., Huang, H., Chen, S., Zhang. J., Zhao, J., Roy, S., Jin, L., Xiong, Y., Shi, Y. Xiao, L., Tian, Y., Soran, B., Krishnamoorthi R., Blankevoort, T., and Chandra, V. (2025). Paretoq: Scaling laws in extremely low-bit IIm quantization. arXiv preprint arXiv:2502.02631.
[23] Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In International Conference on Learning Representations.
[24] Malinovskii, V., Panferov, A., Ilin, I., Guo, H., Richtárik, P., and Alistarh, D. (2025). HIGGS: Pushing the limits of large language model quantization via the linearity theorem. In L. Chiruzzo, A. Ritter, and L. Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics; Human Language Technologies, pages 10857-10886, Albuquerque, New Mexico. Association for Computational Linguistics.
[25] Modoranu, I.-V., Safaryan, M., Malinovsky, G., Kurtic, E., Robert, T., Richtarik, P., and Alistarh, D. (2024). Microadam: Accurate adaptive optimization with low space overhead and provable convergence.
[26] OLMo, T., Walsh, P., Soldaini, L., Groeneveld, D., Lo, K., Arora, S., Bhagia, A., Gu, Y., Huang, S., Jordan, M., et al. (2024). olmo 2 furious. arXiv preprint arXiv:2501.00656.
[27] Panferov, A.. Chen, J., Tabesh, S., Castro, R. L.. Nikdan, M., and Alistarh. D. (2025), Quest: Stable training of Ilms with 1-bit weights and activations. arXiv preprint arXiv:2502.05003.
[28] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. (2020). Exploring the limits of transfer learning with a unified text-to-text transformer. In Pro- ceedings of the 37th International Conference on Machine Learning (ICML), pages 13934-13944. PMLR. T5 and C4 Dataset.
[29] Reddi, S. J., Kale, S., and Kumar, S. (2019). On the convergence of Adam and beyond. arXiv preprint arXiv: 1904.99249997
[30] Robert, T., Safaryan, M., Modoranu, I.-V., and Alistarh, D. (2024). LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statistics. arXiv preprint arXiv:2410.16103.
11

---

[31] Sardana, N., Portes, J., Doubov, S., and Frankle, J. (2024). Beyond pannhilla-optimaal Ac- counting for inference in language model scaling laws. International Conference on Machine Learning.
[32] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. (2024). Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568, 127063.
[33] Sun, X., Li, Xie, Han, W., Wu, K., Yang, Z., Li, Wang, A., Li, S., Xue, J., Cheng, Y.. Tao, Y., Kang, Z., Xu, C., Wang, D., and Jiang, J. (2025). Scaling laws for floating-point quantization training. arXiv preprint 2501:22500.02233.
[34] Touvron, H., Martin, L., Stone, K., Albert, Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, I.., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fermandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S.. Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nic, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 0928880
[35] Zhang. and Sennrich, (2019). Root mean square layer normalization. Advances in Neural Information Processing Systems, 32.
[36] Zhou, D., Chen, J., Cao, Y., Yang. Z., and Gu, Q. (2018). On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization. arXiv preprint arXiv:1808.0567 05671.
[37] Zhu, M. and Gupta, S. (2017). To prune, or prune: exploring the efficacy of pruning for model compression. arXiv preprint 17110:1710.0187..
12

---

Appendix Roadmap
This appendix provides supporting material organized as follows:
Experimental Setup (Appendix A Model architectures, hyperparameters, and training
configurations.
Factorization of Representation Capacity (Appendix B): Detailed analysis showing how
representation capacity matrices can be factorized for various compression techniques including quantization, sparsity, and their combinations.
Ablation Studies on Law Formulation (Appendix []: Investigation of different noise distributions (Gaussian, Logistic, Student's Laplace) and functional forms (tanh, logistic) for the scaling law formulation.
Implementation details and algo- rithms for vector quantization approaches, including forward and backward pass descriptions for HIGGS- based training.
Scaling Laws for Vector Quantization (Appendix D):
Theoretical Support (Appendix Ey: Convergence analysis for Adam optimizer with Straight- Through Estimation (STE), including complete proofs and supporting lemmas.
Improved Sparse Training via RBBM (Appendix FD: Comparison of backward mask heuristics against RigL and Gradual Magnitude Pruning, with detailed descriptions of different masking strategies.
A Experimental setup
Hyperparameters. Table summarizes the architectural and optimization hyperparameters used in this study.
Model size
# Layers
# Heads
# Embeddings
Learning rate
30
6
640
10-3
50 M
7
6
768
1.2 10-3
100 M
8
8
1024
6 10
200 M
10
10
1280
3-104
Table 2: Key architectural and training hyperparameters for Llama family models.
Generalization Across Model Families and Datasets. To validate that our findings are independent of the model family and dataset, we conducted comparable experiments on the OLMo2 26 family models trained on the ClimbMix dataset (7]. OLMo2 models use no biases, employ rotary positional
embeddings (32], RMSNorm (35), and reordered pre-normalization [2] 26). We also used ReLU activation function for linear layers.
We matched the model size range, data-model size ratios, and optimizer hyperparameters to those used in our original setup. Scaling laws of the same functional form as in Table where p(GMSE) takes the form of Eq. were refitted from scratch. We report the corresponding estimates for parameters (a, [8, E, L. F. C) in Table 3 Confidence intervals of one standard deviation were obtained by the bootstrapping procedure. The results match closely with those from the Llama / C4 setup, with overlapping confidence intervals.
We use 8x80GB H100 machines for efficient training, and training one model takes on average 1 hour. To produce the full set of results we ran in total approximately 250 such training runs for various compression configurations.
B Factorization of Representation Capacity
Figures 6 show factorization of the representation capacity matrix for various in-training compres- sion techniques:
1. Quantized weights and activations (Fig. 6
13

---

Llama family C4
OLMo2 family ClimbMix
0.13 0.05
0.18 0.03
B
0.33 + 0.06
0.26 = 0.02
1.3 + 0.5
1.4 = 0.3
L
1.0 + 0.7
0.84 = 0.10
F
0.41 + 0.02
0.37 = 0.04
1.39 0.08
1.24 = 0.13
Table 3: Scaling Law hyperparameters for Llama and OMLo2 model families.
2. Sparsity QuEST quantizer (Fig.
3. Joint sparse & quantized weights + activations (Fig. 8). for all combinations (8a, las %6) for sparsity Sa E [0.25, 0.5, 0.75] and bit widths Ja. 9b E [2, 4, 6].
4. Sparsity + uniform quantizer with maximum absolute value as scale (Fig. 9
From the factorized represeetation-ccaacity matrices we observe the following:
1. The element- error of the fitted coefficients (from our scaling law) is of order (0-310-2
2. A rank-1 row-column outer product accurately approximates the matrix, confirming the multiplicative property of representation capacity p in various scenarios.
3. Approximation error remains of the order 10-2, except for the cases of extreme 2-bit quantization, where p S 0.1. We explain this gap due to the poorer performance of the optimizer in these extreme compression regimes, which not taken into account currently by our model (as uses the same coefficients for both 16 and bits).
True plP)
Approximated p(P)
Absolute
Weight bits
2-0.15
0.30 0.40 0.45 0.45
Weight bits
2
0.09
0.22 0.35 0.41
0.45
Weight bits
2 0.06 0.09 0.06 0.04
0.0.19
0,70
0.36
0.57
0.67
0.74
0.03
0,03
4-0.20
0.46
0.72
0.85
0.90
4.
0.19
0.43
0.69
0.82
0.90
4- 0.01 0.03 0.03 0.03
0.21
0.48
0.76 0.91
0.21
0.48 0.76 0.91
1.00
2
16
16
16
Activation bits
Activation bits
Activation bits
Figure 6: Representation capacity coefficients for independent quantization of weights and activations. Element-wise p fitting error is not greater than 5 10-3
C Ablation studies on Law Formulation
C.1 Evaluating RMSE across Different Distributions
We investigate how the choice of noise distribution used in our law formulation from Sec. 4.1 affects the predicted representation capacity. In Figure 10a we plot the mapping p(MSE) for different bit widths using Logistic, Student's t, and Laplace noise distributions. Each distribution rescaled to have zero mean and unit variance.
We observe that, matter which noise distribution we choose, the mapping p p(MSE) always remains strictly monotonically decreasing. In principle, one could use heavytailed distributions (for example, Student-t Laplace) to give more weight to extreme outlier errors. However, this leads to smaller range of MSE values. By contrast, assuming Gaussian noise which we propose propose-producee produces the widest spread of MSE, which in turn allows for better fit for the scaling law. In short, although
14

---

True p(P)
Approximated p(P)
Absolute error
2 0.57 54.00 c 0. 38
2-
0.48
0.39
2
0.7 Saan 0.66a 0...aee...775000 S7azzed 35
0.75
0.63
0.52
0.37
0.02
0.05
0.02
Bits
Bits
2.07 0. 60 45ga
n 9 90 6.7.
0.99
0.84
0.69
0.49
0.06
0.06
0.02
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
0.0
0.25
0.5
0.75
Sparsity
Sparsity
Sparsity
Figure 7: Representation capacity coefficients with fit errors in case of sparsity combined with the QuEST quantization.
0.08
Abs. Error
0.06
X
0.04
0.02
0.00
0.3
0.4
0.5
0.6
0.7
Sparsity
Figure 8: Representation capacity fit errors for sparse+quantized weights and quantized activations. Error bars denote 11 standard deviation from the mean.
True p(P)
Approximated p(P)
Absolute error
0.05
0:35
0.85
1:00
1.00
0:05
0.58
0.55
1.00
1:50
0.1
0.09
0.52
0.81
0.95
0.55
0.1
0.04
0.55
0.81
0.94
0.35
0.1
0.05
0.03
0.00
0.00
0.2
0.2
0.52
0.2
Coarsic)
3.4
0.49
0.84
0.00
0.00
0.4
0.15
0.44
0.7
2.75
0.78
0.4
0.64
0.6
0.57
0.78
0.78
0.4
0.10
0.01
0.00
0.00
0.5
0.5
0.42
0.5
0.7
9.09
0:35
0.49
0.56
0.56
0:83
0.33
0.48
0.56
0.56
0.06
0.02
0.01
0.00
0.8
5.06
0.47
0.8
0.62
0.27
0.47
0.8
0.64
0.01
0.01
0.9
0.35
4
16
Bits
Bits
Bits
Figure 9: Representation capacity coefficients matrix for sparsity applied with uniform quantization. Element-wise y fitting error is not greater than 2 10-3
monotonicity is preserved under various distributions, the Gaussian MSE delivers the best overall representation capacity prediction, so we adopt it as our default formulation.
Throughout this work, unless specified otherwise, MSE is computed over standard Gaussian input.
C.2 Functional form of the Law
The behavior of p( GMSE) observed in our experiments can be captured by fitting multiple smooth, monotonically decreasing functions, with no more than additional parameters. In principle, a wide range of such functions can be used to model this relationship. depending on the desired fit properties.
For lower overall fitting error, found it beneficial constrain the function satisfy boundary conditions f(0) = and f11 = 0. This way the correct behavior in the high-error region MSE S 1 is enforced, which is critical for stable predictions in the extreme compression cases. The corre- sponding fits are summarized Table 4, the fitting error is calculated for the combined scaling law L(MSE) L))(M))))
15

---

1.0
30 M
100 M
0.7
0.6
Gaussian
3.1
Logistic
0.4
Student's
3.0
papacce
2.9
0.3
10
10
10
10a
10
MSE
MSE
(a) Effect of input noise distribution on the mapping p(MSE)
(b) Different functions used to fit ((MEE)
Functional form
Fitting error (RMSE)
Decoupled
Independently fitted p
4.2 10
Tanh Logistic
tanh F log1/4 MSEJC
4.7 10-4
(1 + B MSEA)
4.9 10-4
1 MSEA
Logistic (1, 0)
1.1 10
B MSFA
Table 4: Functional form choices and associated fitting error.
Throughout this work, we adopt the functional form of hyperbolic tangent as it provides the smallest fitting error.
D Scaling Laws for Vector Quantization
In this section, we provide detailed information about the Vector Quantization approach used to produce the results Figure[(ar Algorithms and describe the forward and backward passes over linear layer actively quantized with HIGGS for row-major weights. As was described carlier, method is combines ideas from Panferov al. (27 for the gradient estimator, and Malinovskii et al. (24) for the lattice representation. We use the trust estimation method that zeros out gradients for any point lying outside hypersphere of radius Rt ||a:||2 R2. Our experiments on 30M and 50M models using the same set hyperparameters as Sec.
conducted
Algorithm vQ Training Forward
1: Input: Input activations x, row-major weight w
2: Wh. HT(w)
3: WA projgrid W/.
4: y = =xW]
5: Return: y, x, Whs Mgrid Aorid(we) (Wh; Wa)
E Theoretical Support
Here provide the full proof of Theorem I giiving convergence analysis of the Adam optimizer when used with STE. For completeness, the description of the algorithm is presented in the Algorithm
3
16

---

Algorithm 2 VQ Training Backward
1: Input: By
x, Who Mgrid(Wer (Wh; Wa)
2: ØL
aL Wh
0x
by
3:
ar.
x
AL by
(hw)
4: Ow
aL
IHT
Morid(Win Wh: Wh) O
al
Own
5: Return: aL BL
8x
Fw
Algorithm 3 Adam with Straight Through Estimation (STE) and AMSGrad normalization
Input: parameters B1, B2 E (0, 1), > step-size 17 0, 01 Fd mg
Up vi 0,
2: for (1,2,.., 7 do
3:
02 -C(O.)
Compress the model via quantization and/or sparsification
4:
gt Vof(0.)
Compute STE for compressed model o Update first-order gradient momentum Update second-order gradient momentum o Apply AMSGrad normalization Update the uncompressed model parameters
5:
me = 31m--1 + (1 (1- 31),et
6:
Ut B2Ur 1 + B2)gi
7:
Ur max Ut, =11
8:
0.+1 0i 11 Voete
9: end for
Proof. Let G be the gradient bound with respect l norm, that is, llgelle I G. Using the relationship between le and foo norms, we conclude < ddG Let I, = Diag Va + E) be the preconditioning (diagonal) matrix and rewrite the main update rule
8i+1 = 0g - mm
Letting 0g = 01, define virtual iterates 2, as follows:
It 1 B1 1 31
01-1-
In particular, I1 01. Then, the update rule for the virtual iterates becomes
1
81
T++1
(0.+1
00,
(0,
01-1)
1- 31
1
B
"
11.31
Teme +
Tt
1
B1
1 81
11.31
+ 1131 Tmm4-
T T,m+ +
71-1m--11
1 B1
1-1
31
7
T,(m, - -3mm-1) +
n99
(Tr-1 Te) m,
1 Bi
1
B
n131
-nyT ttt +
AT A
1- 31
Next we apply smoothness (Assumption) 1 of the loss function f over the iterates .yy
L 1 - 12l12.
f(2141) S + (f(x)),
(V)))) 2x) + 2
17

---

Taking expectation and splitting the inner product into two part, we obtain [[f( - -()
B1
AT Alme-1)
7B
1- 31
1 I + E
B1
19
Figt
ATom-1
1-31
+ +nBB (VS(OE))
B1
B (V f((.), r
AT ATem-1)
1- B1
11
7 + 2
91
E
Figh
Alam-
+ TIlE ((V)(0)) V1(z)),
(5)
1- B1
III
In the following, we bound all the four terms mentioned above.
Bounding term I. Let || |r', be the operator norm (with respect to l norm) of the matrix Ar,- Since Al', is diagonal, the spectral norm coincides with the largest diagonal value in magnitude. Using unbiasedness of the stochastic gradients, we have (V)) nE
I
T/
[V f(O.). Ar :9:1
7/EE
V V[(0),,,---())))
(V
nB (V -f(.)). A
7/EE
< - -T/Amin 11
0:11] B[]ATT)
" Vf
my + 0.11 || +
(6)
Co
where we used Assumption 2 and Lemma 3 to bound
Amin (Tz-1) 2 (1 Imaa + +)) 2 (G + -112 def
Co
Bounding term II. Splitting the inner product again and bounded each term, we get
( (V)(O))
31 AT Pm
)) (V -((())))V(()))
B1
I1 = n)
+ nB
AT Almm-1)
1- $1
1- B
n1
L,8 E [[[[[--m---ll |
E V
AT mm- 11l + (1- B1)
1 1 31
1131
G
1-33
(1-
where we used the fact that the largest eigenvalue Amax (Ty) = ||T|||= 0f min + - 1/2 The second inequality is due to the smoothness of f, and the last inequality is due to Lemma Assumption and the property of norms.
Bounding term III. This term can bounded as follows:
L
III < 1/ LE
+ (1 (1- 81)2
AT A 11m4-1113
LL
L82
Ellgg ( 12 (1 B1)2
||
LL
L2
- + (1- 3-
El1 [T11111111
2L
Lo
L22
+
(1 3
EE[ Ar, |12],
(8)
18

---

where we used Assumption that 9i is unbiased with bounded variance o². Bounding term IV. Finally, for the fourth term, we have
IV - E (VV ((0)) - -VV00
1 E
I n ( (Vf(()) V((a))
1-31
(0)
+ 2 -V/(m\)))
v/(=a))
3 LG
B[| AT
2c
2p
(1-Bi)e
(1) ne
T² E
[[ [1²)
2(1 31'pp
(T-31)v-
3/2
1e 1L
| 112
10 Al.
(9)
2E
2(1 - 31
(1
(1- -i
where (a) is due to Young's inequality and (b) is based on Assumption into 5
1 Now integrating 6
I S
"
Bell
+ G |IT Ar,
Co
n
LG2
11 S 1- Bi
G2
+
E1| |I
(1 32
L
LO
39
LG
III <
(1 B1) BI
IV < 7P
392
E [|me-ll?]
(1 (1-BBi)e" Arl..,
LB
2E
2(1 - 31
and taking the telescoping summation over I. 1, ..... T, obtain
f [x(+)) - E[f(2z)]
Tip
13,397.2
11
+ nP
Σ
E [
+
Co
2e
2(1 2(1-31 31)2pp pe
G22
B.G2
T.
BI
E11
0ell -I
1 31
(1 -1- 1--1)2e
(1-31)²
T
"
+ L
+ nP +
1323312
Σ
f 12
Lo²
T T
Co
2c 2(1- - 211-B1)²pp
2(1 - 3:)2pe
+
+ n/G
+
LG
Σ
T
B| /1I + TT2 T
Σ B|Jlr, 111
4L T
Σ
Bl1@0 - -ell -[l,-ll,
1- B1
(1- (1-B1)2 Ve
(1-81) B1)2
T
t=1
t=1
def c(1-81) and using Lemma ALCo
where we used Lemma
Choosing p 20c
and 17 < no
we get
T
+ Lo2 T
E[((2TH)) - [(a)))
1)
2Co
(1 - 31)
4r
+ nLG T T
+ (1- (1-Bi)ve
+
Σ
Ell@e - 0.1l [[].
(1 B.
t=1
19

---

Re-arranging terms, we get
1 T EIVV1 & 2Co
(f(0))-S*
TT)
(1 - 3-)
(
G2
T/3
2C
LG
T
10: Oz 2|
+4C,
+
+
Σ
TT(1-1iVV
T(1 - 81)
T
VE 0t- min
where in the last inequality we used 21 01 and the lower bound f S f() for 0 RRd. Finally, and considering the two cases, we continue
choosing 1j min )/)
T
f(01) -
Lo²
L2C03902
T
<
2Co
max
1,
1oVT
VT
oVT
+
(1
B1 222
G²
B1LG²
200LG T
T
110g 0 1|
+4C
E
T(1 BI)VE
T -1)
Ve
+
00- min
1=1
[(01) S"
Lo² B2
2Co
VT
EVT
(1 -33 PTT
(f(())
G
B1
+4C
T(1 - BilVE +
2nTT
- 19/4(1-11))
LG,L
1
T
E
100
0cl12
VE
T
Using the bounds G < VNG.o. Co s bound
N
C and surpessing higher order terms, we simplify the
2
Σ BI
CLG
E
1
Σ
10,
0el12
.N
CVN
(r())
-
Lo²
+0
N//2
T
VT
T
which completes the proof of the theorem.
Lemma 1. For any 2 the following bounds hold:
T
T
|mlll < G,
+ Σ E [=1
13
To
Σ
E [v)(1
(10)
Proof. Let us start with the proof of the first bound on my.
|| m, + (1 - 31 31) 1|
s 3111mm/1 + +(1 (1))1ll0l (4))1 11?
1+1
1+1
Bilmml |I + (1 B1) Σ g |g 12
(1) Σ
8
1g 1
Using the bounded gradient assumption, get
||me| (1
31)
Σ g SC.
SC.
20

---

To derive the bound with expectation, we apply Cauchy-Schwartz Schwartz inequality and the bounded variance assumption:
T
T
B
17]
$
(1-
E
11
T
E
[[lgel]] = Σ
E
|19: - -)(() + (
T
2 +
E
[ TO? + Σ R
| (0.)
Lemma 2. For AT, P Fe we have
Σ |ATII
1 Ve
Σ IAT 1A
1
Proof. From the definitions of r, = Diag (ī, + c) and oN = max (h-1) imply that AT, = T--1 - re is positive semidefinite. Hence, ||r Amax (Ar,) 2 Using the convexity of Xmax over symmetric matrices, we get
T
Σ |rr
= max
1=1
1
1
max
$ Ve
UUL
Vūr, +e
Vo,s +e
Vūr, te
For the second sum of squared norms, notice that for scalars 2 bb 2 0. it holds that
(a- b)² < (a - b) + b) = a² - b².
Therefore, the above derivation can be repeated without the square roots as follows:
T
|r
max
Ar
f,
2
1
max
Vūr-l.+e +
Vūn, +E
1
1
1
max
= max
5 7
+6
De,i
+
0o.i +
Ør. + c
which completes the proof.
Lemma 3. For all iterates t 2 1 the following bound holds
1z 1l S G².
Proof. From the update rules we get the bound for using the initialization up = 0:
||ue|loo S |[vllla < B211v--111 + (1 1
s B211u--111 + +1 G2
d-1
@ + +(1 - G2 Σ G
Hence, using the update rule of é, and initialization op 0, we conclude
|e I max([]o:/loo: [/0))\/=)) G².
21

---

Next, we simplify the optimization setup by considering SGD optimizer over (still generally non- convex) quadratics. In this special case, we provide improved and generally optimal asymptotic convergence rate. Moreover, we do not use the bounded gradient condition (i.c.. l S Goo of Assumption in this analysis.
More formally, consider iterates 0.+1 = -/Vo where 0, C(O.) is the compressed model. Suppose that the loss function is quadratic with Hessian matrix H E R- Ngxo x and our compression scheme C: RV R- unbiased, namely 1 [c] 0.. Since the loss is quadratic, we have
Vf(0.)
ff(0, (0, 9,) ff() H 02 0z).
Denote by IR, = B [A) the conditional expectation conditioned on iterate O.. and apply unbiasedness of the compression to get
E. f( = =, 0.)11
(11)
Therefore,
Be[s(0+t))-7) - [*]
< (f(O.) -f* -B.((V v((.)) LnF E. V ((
Laj² R. (0. || + Lai o²
s (S(O.) - /)) - (V.(0))))) ()
2
= (S(O.) - / - -. Lnj² E2 V f(0,) 12 + Lai
Li²
E, H (0. 0c)11 +
o²
2
2
= (S(O.) / 711 /// 1n/2) [1 [+ Ln2 (Boll 02 0111:r1 +
s (f(0.) - ff) " " Ln²
[1. 0 11:z 1 02))
where we used E, (()) = v (0²) due to the unbiasedness of compression and enforced the bound 17 S in the last step. Hence,
T
EI 2(((r)) S*)
T Oeller + 0²
+ nT.
E
Σ
10.
T/T
Choosing the step size min( ) and applying L-smoothness, we get 0 convergence rate for the uncompressed iterates 0,:
T
T 100
& - VT (2(()))))+++ Lo² + L
1,
L
EIV 1=1
0.113
max
For the convergence bound with respect to the compressed iterates 0G apply exact difference in average gradient norms with the following identity:
11 to quantify the
+ + Σ B
E
F
H
0.)11
T
Thus, a randomly chosen compressed iterate Ō from (ū,.... θT} satisfies
|| < L2. E
10. 0. 12
+ +
T
VT
F Improved Sparse Training via RBBM
F.1 Comparison against RigL.
In this subsection we compare our backward mask heuristic in Figurd5d with the RigL method of (Evci et al., 2020). We run two instances of RigL: 1) the default one that updates the mask once at
22

---

100 steps (i.e. A = 100) and updates the mask for the last time at 75% of training and 2) version of RigL that closer to our RBBM setup, which changes the mask each step (i.e. A = 1) during the entire training. In Figure Iaa we observe that both versions of RigL induce lower capacity than our naive baseline for fixed sparsity.
Fig. (1 1000
Doce
6.7
Dasad ICMP)
Bivel (SMP)
Fay. 1o4Ro-12
RLON (OMP)
54
54
Capaciy
0.3
82
02
0.3
9.5
044
D19
0.6
285
0.11
01.7
13.0
Sparsity
Sparsity
(a) Pre-training 10M with different sparsities using our MP baseline, RBBM heuristic and Rigl. variations.
(b) tre-training Llama-30M with different sparsities using our constant MP (CMP) baseline, GMP and RBBM heuristic with GMP schedule.
Figure 11: Comparison of sparse training methods for Llama-30M.
F.2 Comparison against Gradual Magnitude Pruning (GMP)
In this section we show our results for applying the GMP sparsity schedule [37] for our setup in [ib Our first baseline is the constant Magnitude Pruning (CMP), where the backward mask is identical to the forward mask (determined by Top-K) and the sparsity is kept fixed during training. The second baseline is the original GMP where sparsity increases gradually and we compare against the gradual sparsity schedule applied to our b-rms heuristic.
We observe our RBBM heuristic with GMP schedule has lower capacity than both CMP and GMP baselines when sparsity is < 40%. However, for sparsities 2 40% there is no significant difference in capacity between CMP and GMP schedules.
F33 Backward Mask Heuristics
In this section we provide more details about our backward heuristics.
Our purpose is to perform sparse training for both forward and backward passes. All models trained with the same learning rates as in the Quest project.
Notations. Let 0 be the model parameters, My be the be the mask for the forward pass, and Baw be the mask for the backward pass.
Forward Pass. The mask for the forward pass is computed using Top-K operator, where is chosen based on the target sparsity. Supposing Top-K returns the indices of largest entries by magnitude, the 7th entry the forward mask for a tensor x is computed using the indicator function as follows:
Mew (1) Tli TopK (I+))
(12)
Backward Pass. The mask for the backward pass is computed using a few heuristics described below.
1. fw: the backward mask is simply set to the forward mask: Mew = Mrw. This heuristic allows gradients to flow only through the largest parameters by magnitude selected by Top-K, while the low-magnitude parameters will have zero gradient
2. rms: we align the tensor & with the standard normal distribution by dividing x by its root mean square RMS(() = V Ex 2 which results in /|z/ (x)) = n.. For this heuristic, the
23

---

user sets a median deviation parameter p E (0, 0.5), which is used to determine the threshold for the backward mask (RMMs(E,)))) p) RMS(z) (0.5 + p), where ppf is the inverse cumulative distribution function of the standard normal distribution (see the acipy.stats. scipy.stats.normpp norm.ppf function).
The multiplication with (M) has the purpose of converting the threshold for the standard normal distribution to the threshold for the vector I. As a result, MEW = [x| > T TRass( p).
3. banded-rms (b-rms): the rms heuristic has the property that the absolute values of æ that are larger than the threshold TRMS(5 (x. (z, p) will have value 1 in the mask, while the smaller ones will have value This banded hcuristic determines the backward mask using the threshold T T'RMS s (æ, p) computed for the rms heuristic in conjunction with the Top-K threshold (which we denote by Tx). We want to allow gradients to flow for the small parameters and create band between T TRMS (2.P) and T3 where we do not allow gradients. Concretely, the backward mask is set as follows: MBw (|r] < min (T p), Ti)) (max naM (zz p), Tx) < (z:)) Since the median deviation p is a Slyper-paameee and we are using the min and mar functions to make sure the band is valid, e.g. the parameters
we do not have any control over the relationship between 7 RRMS (z, p) and Tk
do not receive gradient if they lie in the interval [min (T'," T TRMS (z,p), p)), max(T's TRMSsSSS p))]
area-banded-rus (a-b-rms): in the b-rms heuristic we do not have any control over the re- lationship between the Top-K threshold Tl and TRMss (2, p)- Let discuss the two possible cases:
(a) Tk < <TRMSII.)) : MBw = ((x] < Tb) V (TRMs(IF,) p) < [z]), which means that all values from x with a lower magnitude than T3 and larger magnitude than TRass(R, (ar, p) will get gradient, while the values in the range [Tg, TRass(x. p)] will not receive gradient, even though they were selected among the Top-K during the forward pass.
(b) TRMS (2. p) < Tx : Maw = (|x) < (RMS((. p)) V (TA < [=], which the desired case we developed the b-rms heuristic for: the largest entries from according to the Top-K rule will receive gradient, as well as the entries smaller than TRAiS (x. The entries lying the interval [TnMs(x.P)) p), Ts. will not receive gradient.
We want to make sure that case (a) above does not happen in practice and force the heuristic to behave as in the case (b). For this, we need change the way we compute the threshold TRaas(2,)) p).
The area-b-rms heuristic uses the area papammerrretter a E [0, 1] (instead of median deviation p) and expresses the width of the band starting from the Top-K K parameter Tç towards zero to compute the threshold T₃ to make sure the condition T. Tı always holds. As a result, MBW = (|x] < Ta) v (TR < |æ:). For example, = 0 yields TІ = Тa and this heuristic turns into Iw, while a = 1 yields T, and equivalent to Mew La (all entries set to meaning all parameters get gradients). When E (0, 1), the parameters smaller than T, or larger than Th get gradients, while the parameters lying in the interval [T., Tx] do not get gradients.
How to compute the threshold 7,? Compared to the threshold computation for the previous heuristic, the definition for T. is slightly more complicated and was computed graphically. Let be the cdf function and f be the ppf function (inverse cdf) for the standard normal distribution.
T. RMS(=) 8-1 (-..0... 0.5 (0.5+(1 (1-))
())(((RMT))))
Tk
-0.5) 0.5
(13)
Explanations for the formula above. Suppose the Top-K threshold Tk has a corresponding
edf of 0.8 and we set a = 0.5 (which means 50%). We need to set the threshold T. such that 0.5) a. where 0.5 is the edf of the mean (which is identical the median for a standard normal distribution). This ratio expresses the length of the band fT,,) in the edf space starting from fTk) towards the median. As consequence, the threshold T. = (0.65) because the quantile 0.65 is the center of the interval [0.5, (Tk)) = [0.5, 0.8).
(J(T)) ((T)))/((TT)
The cxplanations of each term follow:
24

---

Ta (MS(a)
To RMM(2) 8
0.5 + a)
-0.5
A: f = edf computes the corresponding quantile of the Top-K threshold T₃ normalized by RM)(x)
B: subtract 0.5 from term A to compute the length of the interval [0.5, ((T
C: multiply by 1 because we take into consideration the band length that starts at
f(T towards
compute the edf of T, by offsetting again by 0.5 (the quantile the median)
E: use ppf = f to obtain the value that corresponds to edf (Ta) for the standard normal distribution
F: multiply by RMS(x) to obtain T. in the space
Technical note. One could determine the threshold T, naively by employing the formula Thaive (1 - (1-a)7k. Despite simpler, this naive approach leads to narrower band because the edf space is non-linear.
Conclusion. The mask computed using the a-b-rms heuristic is more straightforward to under- stand because the parameter describes the area of the red band (where parameters do not receive gradients) as a percentage of the area between 0 and the Top-K threshold Tμ. This heuristic can be used as replacement for b-rms and the parameter should be tuned, similarly to parameter p for b-rms, with the distinction that [0, (for a-h-rms) and p E (0, 0.5) (for b-rms).
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See final section.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: See Section
S
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Code for reproducing the results is provided.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Code is provided.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This is standard and specified. Code is provided.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide quality of fit for the scaling laws.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide this in the supplementary material.
9. Code of ethics
Question: Does the research conducted in the paper conform. in every respect, with the NeurlPS Code of Ethics http://pp//.aoooop..ccc/aa//aaaa
Answer: [Yes]
26

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper is technical in nature, so there isn't t anything significant to discuss on this point.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
16.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
27


Reviews:
[{"review_id": "D1iDRjWKqC", "reviewer": "Reviewer_9G9f", "summary": "The paper investigates the scaling law that predicts model performance when training occurs over compressed representations. It introduces a \"representation capacity\" metric that estimates parameter efficiency, based on fitting random Gaussian data. Experiments are provided to show that this capacity metric can predict model performance across different compression formats.", "questions": "N/A", "limitations": "yes", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- The paper provides a unified scaling law that incorporates various compression formats, including sparse, quantized, and hybrid representations.\n- The proposed representation capacity metric is new.\n\nWeaknesses:\n- The paper is not easy to understand for readers not in the field of scaling laws and model compression.\n- To my best understanding, there is no experiment showing how good the proposed scaling law fits the real data. I think a figure showing the predicted loss vs. the real loss would be helpful (similar to Figure 1 in [8] and Figure 7 in [18]).", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback!\n\n\n**1. Accessibility for readers outside scaling laws/model compression.**\n\nThank you for pointing this out. We agree that the current presentation is quite technical, and, in the revision we will include a brief “reader’s guide” to our notation and assumptions. We expect these additions to make the paper more self‑contained.\n\n\n**2. Empirical fit of the proposed scaling law.**\n\nWe fit the scaling laws identically to existing literature[1,2,3]: we minimize the Huber loss over the logarithm of the validation set loss with $\\delta=1e-4$. The fits show MAE of around $6\\cdot10^{-3}$ for validation loss prediction. We will include a visualisation of estimated vs actual loss in the revised version of the paper.\n\n\n**3. Title mismatch between PDF and OpenReview.**\n\nWe apologize and will fix this discrepancy: the OpenReview title will be updated to match the PDF’s title.\n\n\n[1] https://arxiv.org/abs/2203.15556 [2] https://arxiv.org/abs/2411.04330 [3] https://arxiv.org/abs/2502.05003"}, {"review_id": "kQOwyeTs35", "reviewer": "Reviewer_ygts", "summary": "The paper studies the scaling behavior of compressed models, unifying different ways and levels of sparsity and quantization. The proposed law can be used to predict representation capacity that enables comparing sparse, quantized, or both sparse and quantized models.", "questions": "Some questions are listed above.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The idea of factorizing capacity scores is very interesting.\n- The proposed laws allows for predicting representation capacity in general, without restricting the space in only sparsity or quantization.\n\nWeaknesses:\n- The title in the PDF and the registered title do not match.\n- While the capacity factorization is interesting, the high-level idea was introduced by Kumar et al.\n- The paper has extensive results on Llama trained on C4. But it's unclear whether the findings and comparisons are generalized to other models and datasets. \n- Looking at Table 1, Frantar et al [9] reaches smaller fit error for both sparsity and quantization compared to the proposed law in this paper. So, there is no definite gain over this baseline. Could there be a way to make better predictions using these two different laws?", "was_revised": true, "final_justification": "I thank the authors for clarifying the connections to Kumar et al and Frantar et al. The additional experimental results also strengthen the claims of the paper. I increase my score to Borderline Accept.", "rebuttal": "We thank the reviewer for their feedback! \n\n\n**1. Title mismatch.**\n\nWe apologize and will fix this: the OpenReview title will be updated to match the PDF internal title.\n\n\n**2. Relation to Kumar et al.**\n\nIndeed, Kumar et al. introduced a notion of “representation capacity” in the context of quantization. Our contribution is different in three key ways: \n  1. Our functional form of representation capacity is more general, as it covers multiple forms of compression, e.g. sparsity and quantization. Further, it yields a much better fit in terms of prediction error, that is shown in Figure 1a, for example. \n  2. Even focusing on quantization, Kumar et al. only investigate scalar quantization with straight-through estimators, whereas we conducted experiments on formats including sparse + quantized, vector quantization, and multiple gradient estimators (STE, LSQ, QuEST).\n  3. We are the first to provide theoretical evidence for the parameter dependency being a function of the MSE.  \n\n**3. Generalization beyond LLaMa on C4.**\n\nWe agree it is important to demonstrate that our findings are not specific to the LLaMa/C4 setting. To address this, we now present a series of experiments in an alternative training setup:\n - Architecture: two‑layer ReLU² MLPs (vs three-layer scheme for LLaMa), RMSNorm applied after each sublayer (vs pre-normalization for LLaMa), and weight‑tied embeddings (vs untied for LLaMa).\n - Dataset: high-quality NVIDIA ClimbMix dataset (vs C4 used previously).\n\nWe conducted training with identical model sizes, data-saturation ratios, and optimizer hyperparameters. We refitted all the scaling law parameters from scratch. \nWe found that the new estimates for parameters (α,β,E,F,L,C) match the LLaMa/C4 fit: their point estimates coincide within the confidence intervals, obtained by the bootstrapping procedure described in our paper.\n\n|      | LLaMa/C4 | Transformer/ClimbMix10M (new) |\n|:---------|:--------:|:---------:|\n| $\\alpha$  | $0.13 \\pm 0.05$  | $0.18 \\pm 0.03$  |\n| $\\beta$  | $0.33 \\pm 0.06$ | $0.26 \\pm 0.02$  |\n| E  | $1.3 \\pm 0.5$ |  $1.4 \\pm 0.3$  |\n| L | $1.0 \\pm 0.7$ |  $0.84 \\pm 0.1$  |\n| F | $0.41 \\pm 0.02$  | $0.37 \\pm 0.04$  |\n| C  | $1.39 \\pm 0.08$  | $1.24 \\pm 0.13$  |\n\nThis indicates that the same underlying law governs both model families and datasets. We will add those results in the revised version of the paper. We thank the reviewer for the suggestion.\n\n\n**4. The “effC” form of Frantar et al. [9]**\n\nWe emphasize that the meaning of effC is subtly different between our work and that of Frantar et al. [9]: They treat all representation capacities as **independent variables** in the fitting process, and will naturally obtain lower error since it directly fits capacities for each bitwidth in turn. \n\nBy contrast, assuming a functional form over them (such as our GMSE form or the inverse exponential of Kumar et al. [18]) effectively constrains the optimization problem, trading fit error for explainability and robustness of the law. The independent-variable “effC” solution provides the lower bound on fit error for the otherwise constrained solutions. \n\nFrom Table 1, one can see that the parametrization we propose achieves the error closest to the lower bound constituted by the “effC” form while also connecting capacity to GMSE. To further clarify this, we’ll expand the discussion around *Line 178*  with a discussion on how parametrizing capacity affects the scaling law fitting optimization problem."}, {"review_id": "db5VSGllMp", "reviewer": "Reviewer_4EjT", "summary": "This paper introduces a unified scaling law that models the performance of compressed neural network representations, including sparsity, quantization, and their hybrid combinations.\nTo generalize across compression methods, the authors propose a novel metric called representation capacity $\\rho$, which is independent of specific compression schemes and avoids reliance on traditional metrics like bit-rate or sparsity level.\nThis capacity $\\rho$ is quantitatively defined by a compressed model's ability to fit random Gaussian data, measured using Gaussian Mean Squared Error (GMSE).\nUsing this representation capacity, the authors derive a generalized scaling law of the form:\n${\\rm Loss}(N,D) \\sim A(N \\rho(R))^{−\\alpha} +BD^{-\\beta} + E $, where $N$ is the number of model parameters, $D$, the data size, and $A, B, E, \\alpha$ and $\\beta$ are fitted constants\nFurthermore, they show that the representation capacity of hybrid compression methods factorizes across components. For example, combining sparsity and quantization leads to:$\\rho_{\\rm total} = \\rho_{\\rm sparsification}*\\rho_{\\rm quantization}$.\nThe proposed scaling law and factorization principle are empirically validated across a wide range of compression schemes, including scalar quantization, vector quantization, sparse representations, and sparse-quantized models.", "questions": "1. The theoretical analysis in section 3 explains (very elegantly, I must say) the reasoning behind the RMSE contribution for the weight sparsification/quantization.\nHow does activation sparsification/quantization affects the loss. Why does it enter the formulation in Eq. 1 as a factor of $N$, the model weights.\nwill it hold also for convolutional networks, where $N$ can be small (i.e., convolution weights) while the activations might be much larger (i.e., image size).\n\n2. Measuring of actual $\\rho(R)$ values is not fully clear from the paper. I think I understand it as follows:\nFor each compression format $R$ you vary only $N$ (the model size) and then fit the training loss $L$ to the loss approximation, Eq. 1.\nBy fixing the data size $D$ you can avoid fitting $B$ and $\\beta$ but rather only a total constant $E'=B*D^{-\\beta}+E$.\nHaving a multitude of such experiments, per $R$, you fit the global parameters $A, \\alpha$ and $E'$ as well as $\\rho(R)$ per experiment.\nI think what confused me is that you have 2 fit procedures -- one for Eq. 1 and then another one for Eq. 3.\nPlease verify in your responding to the review. In addition, whether I am right or wrong, the paper should clarify this methodology more elaborately.\n\n3. Please put a refernece on how you obtain/calculate the theoretical lower bound (e.g., at 257). It might be also that I missed it...\n\n4. In the checklist you mention \"Code for reproducing the results is provided\". However, a) you do not refer to it in the paper, b) I couldn't find such code in any link or supported material\n\n5. How do you derive Eq. 3? This was unclear to me. \nLater on I found some explanation in appendix D.2. Please refer to it from within the text.\n\n6. small typos that I found. I can't avoid noting on it :) \n\nline 217 shiowing -> showing \n\nline 277 \"the the\" -> the \n\nline 605 \"the the\" -> the", "limitations": "Should address limitation about the generality of the model to other architectures. see also comment above about activation quantization and CNN models", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Novel unifying framework: Introduces a capacity-based scaling law that generalizes multiple prior works and applies to hybrid compressed formats, addressing a significant open problem in scaling compressed models.\n- Strong theoretical grounding: Provides convergence analysis under Adam optimization using compressed iterates, connecting capacity with optimization behavior.\n- Empirical validation: Demonstrates strong empirical fits across a wide variety of representations and model configurations (quantized, sparse, and their combinations).\n- Practical relevance: The ability to rank compressed representations without retraining or exhaustive grid search is immediately useful for both research and deployment.\n- Factorization finding: The observation that capacity factorizes across representation components is elegant and potentially impactful for compression-aware hyperparameter tuning.\n\nWeaknesses\n- The experimental scope is rather narrow and conducted on small Llama-style decoder-only models (up to 200M params) trained on the C4 dataset. Thus, it is not empirically shown how this law generalizes to other architectures (e.g., CNNs)\n- The factorization of the representation capacity $\\rho$ breaks at low bit rate, probably due to substantial second-order effects, as acknowledged by the authors.\nHowever, there is no deep analysis where and why these breakdowns occur.", "was_revised": true, "final_justification": "I thank the authors for their detailed reply and addressing of my concerns\n\nI keep my score at 5\n\nComparing to scores provided by my peer reviewers strengthen my decision.", "rebuttal": "Thank you for your feedback!\n\n\n### Answers to your questions:\n\n\n**1. The theoretical analysis in section 3.**\n\n\nThe theory in Section 3 indeed justifies RMSE reasoning for weight compression and does not consider activation/gradient compression. \n\n\nIf we quantize/sparsify activations, then instead of mini-batch stochastic gradient $g_t$, we would get noisy gradients $\\hat{g}_t$ with another level of noise. We can repeat the analysis of Theorem 1 and derive similar rate where RMSE term would be expressed in terms of gradient bias, namely $|| \\hat{g}_t - g_t ||_2$ instead of $L|| \\hat{\\theta}_t - \\theta_t ||_2$. \nHowever, it is important to note that $|| \\hat{g}_t - g_t ||_2$ is not the compression RMSE of gradients since $\\hat{g}_t$ is not the quantized/sparsified $Q(g_t)$ as it was the case for weight compression (activation compression affects gradient computation in a more complicated manner). \n\n\n**2. The fitting procedure.**\n\n\nThanks for bringing this up, we need to clarify that:\n - We always fit the parametric forms (e.g., Equation 3) on the validation loss prediction in the form of Equation 1 using the Huber loss.\n - For the main setups of each compression format (143 runs for uniform quantization and noise injection, 250 runs for basic sparsity) we vary model sizes N, token count D and precision/sparsity to measure the final validation loss L as a function of N,D and compression degree to independently fit laws of forms from Table 1. Table 1 and Figure 1 present results based on these fits.\n - For more focused additional experiments (Sections 4.3, 5.1, 5.2), we reuse the previously fitted general A,B,E,$\\alpha$,$\\beta$ parameters from the Decoupled scaling law fit and perform a pin-point fit of just the representation-related parameters. We do this to increase the fit robustness.\n\n\n**3. On the theoretical lower bound.**\n\n\nSee the lower bound (13.24) in Theorem 13.3.2 of Cover, T.M. and Thomas, J.A. (2006), Elements of Information Theory, John Wiley & Sons, Inc., Hoboken. For a given compression rate or bit-rate the MSE is lower bounded by $\\sigma^2 \\cdot 4^{-\\text{bit-rate}}$. \n\n\n**4. The codebase.**\n\n\nWe apologize for the omission. Unfortunately we are not allowed to provide code links during the rebuttal, but all our code will be made open source and we will add a link to a public github repository in the revised version of the paper.\n\n\n**5. Cross-references and typos.**\n\n\nThanks for catching this, we will make sure to refer to the appendix section in the revised version of the paper.\n\n\n### Comments regarding the stated weaknesses:\n**1. Generalization beyond LLaMa on C4.**\n\n\nWe agree it is important to demonstrate that our findings are not specific to the LLaMa/C4 setting. To address this, we now present a series of experiments in an alternative training setup:\n - Architecture: two‑layer ReLU² MLPs (vs three-layer scheme for LLaMa), RMSNorm applied after each sublayer (vs pre-normalization for LLaMa), and weight‑tied embeddings (vs untied for LLaMa).\n - Dataset: high-quality NVIDIA ClimbMix dataset (vs C4 used previously).\n\n\nWe conducted training with identical model sizes, data-saturation ratios, and optimizer hyperparameters. We refitted all the scaling law parameters from scratch. \nWe found that the new estimates for parameters (α,β,E,F,L,C) match the LLaMa/C4 fit: their point estimates coincide within the confidence intervals, obtained by the bootstrapping procedure described in our paper.\n\n\n|      | LLaMa/C4 | Transformer/ClimbMix10M (new) |\n|:---------|:--------:|:---------:|\n| $\\alpha$  | $0.13 \\pm 0.05$  | $0.18 \\pm 0.03$  |\n| $\\beta$  | $0.33 \\pm 0.06$ | $0.26 \\pm 0.02$  |\n| E  | $1.3 \\pm 0.5$ |  $1.4 \\pm 0.3$  |\n| L | $1.0 \\pm 0.7$ |  $0.84 \\pm 0.1$  |\n| F | $0.41 \\pm 0.02$  | $0.37 \\pm 0.04$  |\n| C  | $1.39 \\pm 0.08$  | $1.24 \\pm 0.13$  |\n\n\nThis indicates that the same underlying law governs both model families and datasets. We will add those results in the revised version of the paper. We thank the reviewer for the suggestion.\n\n\n**2. On extreme sparsity.**\n\n\nWe agree that separability breaks in the < 4‑bit regime. Namely, most optimizers' performance degrades, see for example Learned Step Size Quantization by K. Esser et al., where they address low-bit optimizer issues by modifying the training procedure (learned step sizes, gradient scaling, hyper-parameter tweaks). \nAlso, kernel support for low bitwidth is limited  and prior work also shows similar degradation.\nThese effects jointly account for the decoupling in this range, so we intentionally scoped our main claims to ≥4‑bit where the factorization is stable. \nWe believe it should be possible to improve fit by focusing specifically on this low-bitwidth regime: currently, our laws span an extremely wide model size range."}, {"review_id": "Z3T2gyo6g4", "reviewer": "Reviewer_W7rD", "summary": "This paper proposes a unified framework for compression scaling laws. It uses Gaussian Mean Square Error (GMSE) to define representation capacity, which helps predicting the loss for different compression techniques including quantization, sparsification, vector quantization, and their hybrids. Their representation capacity factorizes across different compression types. Applications include comparing different numeral formats and designing new methods (RBBM) for sparse training.", "questions": "1. Line 213: Could you explain why the noise injection gives a proxy for the ”information-theoretic lower bound”?\n2. Is it possible to do experiments on different D/N ratios, even just on small N? If your scaling law holds, should the losses decrease at the same speed w.r.t. D, for different compression settings?\n\nTypos:\n\nline 217: shiowing —> showing\n\nline 230: M:,0 —> M_{:,0}", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a simple and performant compression scaling law, generalizing all previous related works such that the prediciton can be done over a combination of compression techniques. \n2. The experiments for factorization of representation capacity into up to 3 terms are a highlight point.\n\nWeaknesses:\n1. All models were trained up to a maximum of 200M parameters. This is relatively small compared with other papers on scaling law such as [1] (1.7B). The reliability of extrapolating the law to billion parameter regime is to be explored.\n2. The experiments are based on a single architectural familty (Llama-style decoder-only) and a fixed D/N ratio (100toks/params). \n3. The writing could be clearer. For example the representation R is not defined. Do you mean it the same as \\hat{\\theta_t}?\n\n[1] Kumar, Tanishq, et al. \"Scaling laws for precision.\"", "was_revised": true, "final_justification": "The author addressed all my questions in the rebuttal. I think the additional experiments are convincing.", "rebuttal": "Thank you for your feedback!\n\n**1. On generalization to billion parameters.**\n\nOur original experimental design is very similar to those of prior work on compression scaling laws, e.g. Kumar et al. \nNevertheless, to address your question, we trained an additional 1.6B‑parameter LLaMA‑family model with 4‑bit weight and activation quantization on 160B C4 tokens. The table below shows the observed C4 validation loss in the D/N=100 regime, as well as the prediction from the Decoupled scaling law fitted on models up to 200M parameters.\n\n| N    | Predicted Loss | Real Loss |\n|------|----------------|-----------|\n| 430M | 2.627          | 2.614     |\n| 800M | 2.523          | 2.495     |\n| 1.6B | 2.421          | 2.378     |\n\nNotably, the prediction errors are ~2x smaller than the gaps in performance between adjacent models even for models as large as the whole order of magnitude above those used for scaling law fitting.\nAdditionally, we’d like to note that Kumar et al. did not properly scale to 1.6B parameters, as their 1.6B model was heavily undertrained at D/N=10, way below the compute optimality ratio and their main explored data saturation regions.\n\n**2. Generalization beyond LLaMa on C4.**\n\nWe agree it is important to demonstrate that our findings are not specific to the LLaMa/C4 setting. To address this, we now present a series of experiments in an alternative training setup:\n - Architecture: two‑layer ReLU² MLPs (vs three-layer scheme for LLaMa), RMSNorm applied after each sublayer (vs pre-normalization for LLaMa), and weight‑tied embeddings (vs untied for LLaMa).\n - Dataset: high-quality NVIDIA ClimbMix dataset (vs C4 used previously).\n\nWe conducted training with identical model sizes, data-saturation ratios, and optimizer hyperparameters. We refitted all the scaling law parameters from scratch. \nWe found that the new estimates for parameters (α,β,E,F,L,C) match the LLaMa/C4 fit: their point estimates coincide within the confidence intervals, obtained by the bootstrapping procedure described in our paper.\n\n|      | LLaMa/C4 | Transformer/ClimbMix10M (new) |\n|:---------|:--------:|:---------:|\n| $\\alpha$  | $0.13 \\pm 0.05$  | $0.18 \\pm 0.03$  |\n| $\\beta$  | $0.33 \\pm 0.06$ | $0.26 \\pm 0.02$  |\n| E  | $1.3 \\pm 0.5$ |  $1.4 \\pm 0.3$  |\n| L | $1.0 \\pm 0.7$ |  $0.84 \\pm 0.1$  |\n| F | $0.41 \\pm 0.02$  | $0.37 \\pm 0.04$  |\n| C  | $1.39 \\pm 0.08$  | $1.24 \\pm 0.13$  |\n\nThis indicates that the same underlying law governs both model families and datasets. We will add those results in the revised version of the paper. We thank the reviewer for the suggestion.\n\n\n**3. Definition of the representation.**\n\nThank you for bringing this up, please see the definition below:\n\nDef. Representation \\textbf{R}: a functional mapping of a full-precision tensor $W$ (for instance, BF16) to a compressed format $R(W)$ (e.g. quantization, sparsification, etc.). The mapping may be point-wise (e.g., scalar quantization) or group-wise (e.g., vector quantization). \n\nWe will add a complete definition in the revised version of the paper. \n\n\n**4. On the theoretic lower bound.**\n\nSee the lower bound (13.24) in Theorem 13.3.2 of Cover, T.M. and Thomas, J.A. (2006), Elements of Information Theory, John Wiley & Sons, Inc., Hoboken. For a given compression rate or bit-rate the MSE is lower bounded by $\\sigma^2 \\cdot 4^{-\\text{bit-rate}}$.\n\n\n**5. D/N ratios.**\n\nWe apologize if it wasn't clearly stated in the text, we conducted the experiments for all models on a set of D/N ratios = [50, 100, 200]."}]

Author rebuttal:

Meta-review: This paper investigates the scaling law that predicts model performance when training occurs over compressed representations. 
A key contribution is the introduction of the "representation capacity" that quantifies the capacity of such representations to support learning, with the goal of predicting model performance across different compression schemes. 
The authors provide empirical evidence to support the utility of this metric across several settings.

The reviewers are in agreement that the proposed idea of factorizing capacity scores is novel and thought-provoking. 
The paper is generally well-structured and clearly written. 
At the same time, reviewers noted that the empirical validation could be significantly strengthened, particularly through evaluation on a broader range of architectures and compression formats.

Overall, this work introduces an interesting perspective on measuring and understanding performance in compressed regimes. 
I encourage the authors to extend their empirical study in the revised version to fully realize the potential of their framework.

I thus recommend acceptance.
Decision: Accept (poster)