The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws
Authors: Tian Jin
Ahmed Imtiaz Humayun
Utku Evci
Suvinay Subramanian
Amir Yazdanbakhsh
Dan Alistarh
Gintare Karolina Dziugaite
Venue: ICLR 2025 Poster (2025)
Keywords: pruning
sparsity
large language model
pretraining
Primary area: foundation or frontier models, including LLMs
TL;DR: We make dense scaling laws fit sparsely pretrained models.

Abstract: Pruning eliminates unnecessary parameters in neural networks; it offers a promising solution to the growing computational demands of large language models (LLMs). 
While many focus on post-training pruning, sparse pre-training--which combines pruning and pre-training into a single phase--provides a simpler alternative. 
In this work, we present the first systematic exploration of optimal sparse pre-training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations.
We find that initiating pruning at 25\% of total training compute and concluding at 75\% achieves near-optimal final evaluation loss.
These findings provide valuable insights for efficient and effective sparse pre-training of LLMs.
Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs, unifying scaling laws across pre-training paradigms.
Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference.

Published as a conference paper at ICLR 2025
THE JOURNEY MATTERS:
AVERAGE PARAMETER COUNT OVER PRE-TRAINING UNIFIES SPARSE AND DENSE SCALING LAWS
Tian Jin' Ahmed Imtiaz Humayun Utku Evei* Suvinay Subramanian
Amir Yazdanbakhsh' Dan Alistarh Gintare Karolina Dziugaite
MIT CSAIL RRice University 'Google Research 'Google DeepMind 'Google FIST Austria
ABSTRACT
Pruning eliminates unnecessary parameters in neural networks; it offers a promis- ing solution the growing computational demands of large language models (LLMs). While many focus on post-training pruning, sparse pre-training-whie combines pruning and pre- pre-training into a single phase-provides a simpler alterna- tive. In this work, we present the first systematic exploration of optimal sparse pre- training configurations for LLMs through an examination of 80 unique pruning schedules across different sparsity levels and training durations. We find that ini- tiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss. These findings provide valuable insights for efficient and effective sparse pre-training of LLMs. Furthermore, we propose a new scaling law that modifies the Chinchilla scaling law to the average pa- rameter count over pre- tre-training. Through empirical and theoretical validation, we demonstrate that this modified scaling law accurately models evaluation loss for both sparsely and densely pre-trained LLMs,, unifying scaling laws across pre- training paradigms. Our findings indicate that while sparse pre-training achieves the same final model quality as dense pre-training for equivalent compute budgets, it provides substantial benefits through reduced model size, enabling significant potential computational savings during inference.
INTRODUCTION
As language models grow in size and train on more data, they consistently demonstrate improved performance (Brown et al. 2020 Kaplan et al., 2020 Hoffmann et al., 2024 Nakkiran al. 2020). However, their enormous size poses an increasingly pressing challenge to their efficient deployment and equitable access. Sparse pre-training integrates neural network pruning (Han al. 2015 LeCun
et al., 1989; Hassibi et al. 1993 He et al. 2017) into pre-training and offers promising solution to these challenges by activating only a subset of parameters during both training and inference, reducing computational costs; it gained prominence when Lottery Ticket Hypothesis (Frankle le & Carbin 2019) presented compelling evidence for its feasibility. Subsequent work introduced a series of sparse pre-training algorithms (Evci et al. 2020 Peste et al. 2021 Kuznedelev et al. (2024).
While a growing body of research investigates pruning large language models (LLMs) post-training (Sun et al.) 2024; Frantar & Alistarh 2023 Xia et al., 2023), our work focuses on sparse pre- training. The combined challenges of LLM pre-training costs and pruning algorithm design present substantial obstacle to this direction of research. For example, Lottery Ticket Hypothesis (Frankle & Carbin 2019) identifies trainable sparse sub-networks using iterative pruming and retraining, a process that becomes prohibitively expensive the scale of contemporary L.Mss. This enormous expense limits investigation to smaller-scale models, leaving the optimal strategies for sparse pre- training LLMs largely unknown. Scaling laws which predict how language modeling loss varies with model and data size can help us extend insights from small-scale experiments to large models.
This leads critical question: how does sparsity change the scaling laws, which inform practical LLM training design (Sardana et al. 2024 Grattafiori et al. 2024)
"Correspondence tianjinécsal 1.m.m.m.... .iit.edu, gkôz@gocgle..com

---

Published as a conference paper at ICLR 2025
Previous work on sparse pre pre-training by Frantar et al. (2024) introduced sparsity- aware scaling law that departed from the established scaling laws for dense models. The modified laws included extra terms dependent on the final sparsity, aiming to capture the sparsity-specific scaling effects.
Our approach. Instead, we revisit the original dense scaling laws and explore how to adjust the parameter count term for sparse pre-training, where it gradually decreases the course of pre- training. Our experiments show that dense scaling laws effectively model sparse pre-training when using the average parameter count - the mean of per-step parameter count over pre-training. Our findings suggest that the core principles of dense scaling laws remain applicable in the sparse pre- training regime, with the key adjustment being a more nuanced consideration of model size.
Optimal sparse pre-training configuration. To validate this approach, we evaluate over 80 combi- nations of sparse pre-training schedules, sparsity levels, and training durations. Our work provides the first systematic analysis of sparse pre-training configurations across design dimensions critical to sparse pre-training training. Our systematic evaluation uncovers the optimal configurations and offers new insights into the dynamics of sparse pre-training.
Our results show that, for a fixed compute budget, initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss across various training durations and sparsity levels. We find that sparse pre-training performs best using the same optimal learning rates and batch sizes as the initial dense model under equivalent compute budgets. We also provide additional analysis into failure modes that occur with non-optimal sparse pre-training contigurations, highlighting the importance of properly tuned configuration. Collectively, analysis presents practical prescriptions that ease the transition from dense to sparse pre-training configurations.
Scaling analysis. We extend the Chinchilla scaling law to model sparsely pre-trained LLMs by using the average parameter count over pre-training. We only count active parameters, which receive gradient updates; they become inactive once pruned. This adaptation allows us predict evaluation loss across a range of model sizes, sparsity levels, and training durations. Using empirically verified assumptions about sparse pre-training dynamics, we provide theoretical justification for why average parameter count accurately models the evaluation loss of sparsely pre-trained LLMs.
Sparsity=8D %
1eB
Sparsity=60 %
Active Param Count
1.5
Sparse
Active Param Count
Sparse Dense
2:90
Sparse
Dense
Dense
1.0
2.85
0.5
0.5
0.0
0.0
Final Eval Loss
2.80
50
100
150
25
50
100
Steps
K Steps
1eB
Sparsity=40 %
1eB
Sp Sparsity=20 %
2.75
Active Param Count
1.5
Activo Param Count
1.5
2.70
0.5
Sparse
Sparse Dense
0.0
0
20
40
60
80
20
40
60
2.65
6.'
19'
Steps
Steps
Average Active Parameters
Figure 1: We show the predictive power of average active parameters by creating two families of models. The first sparse, starting from a dense model with 138 million prunable parameters in the linear layers and targeting final sparsity levels of 20%. 40%, 60%, and 80%. The second is dense, created by adjusting the hidden dimension to match the average number of active parameters throughout pparse-rre-trainie
for each sparse models. In the left plot, we represent sparse models with dashed lines and dense models with solid lines. Each sparse-dense pair, with matching average active parameters, is shown in the same subfigure. Each pairs of model shares the same total training compute. In the right plot, despite differences in pre-training techniques, sparse and dense models with matching average active parameters (indicated by matching colors) achieve similar final loss.
2

---

Published as a conference paper at ICLR 2025
Average parameter count. To demonstrate how average parameter count predicts sparse pre- training loss, we pre-train and compare four pairs of sparse and dense models. In each pair, we match their average parameter count" and training token count. As shown in Figure [ while each sparse-dense pair differs in parameter count during pre-training (left), they achieve similar final loss (right). This comparison demonstrates that average parameter count predicts final evaluation loss equally well for both sparse and dense models. In Appendix C we replicate this result on a larger scale using models exceeding IB parameters, and in Appendix D
we validate that sparse and dense
pairs also match in downstream task evaluations.
Contributions. We make the following contributions in our work:
1. We unify sparse and dense scaling laws by modifying the parameter count term in the Chinchilla model and extending it to model sparse pre-training. This modified scaling law accurately models evaluation loss across model sizes, sparsity levels, and training durations.
2. We present theoretical analysis, based empirically validated assumptions about sparse pre- training dynamics, that justifies using average parameter count to model sparse pre-training loss.
3. We search over 80 sparse pre-training configurations and present a simple prescription that achieves optimal or near-optimal loss across different sparsity levels and training duration.
4. We present an analysis over the failure modes when sparse pre-training configurations deviate from the said prescription, highlighting their practical importance.
Implications. Our work enables practitioners to extend the familiar dense scaling principles to sparse pre-training. By providing optimal sparse pre-training configurations and unified scaling law, our work facilitates the transition from dense to sparse pre-training, promoting the development of more computationally efficient large language models (LLMs).
2 RELATED WORK
Neural scaling laws. Neural scaling laws provide framework for understanding how neural net-
works' performance scales with parameters, data, and compute (Banko & Brill 2001 Goodman 2001] Ghorbani et al., 2022, Kaplan et al. 2020, Hoffmann et al.] 2024 Bansal et al. 2022 Gor- don al. 1021)) Kaplan et al. (2020) showed that for modern transformer- based language models, model loss decreases predictably with increasing model size, dataset size, and compute, following power-law relationship. [Hofmman et al. (2024) later refined these insights by optimizing hyper- parameter configurations, such as learning rate schedules, and proposed a new scaling law that emphasizes scaling training data more aggressively than Kaplan al. (2020)'s 's original recommen- dations. Most relevant to our work are Rosenfelld et al. (2021) and Frantar et al. (2024). Rosenfeld al. (202T) focused on small-scale CNNs for image classification, while Frantar et al. (2024) fo- cused on transformer-basee
vision and language models; both modeled their respective performance as a function of model size and pruning configurations. Our work is different in that we unify the functional forms of scaling laws for both dense and sparse pre-training. This unification is partly enabled by our novel exploration of optimal hyperparameter configurations for sparse pre-training.
Additionally, our study is the largest-scale investigation of sparsely pretrained L.MMs to date, with our largest model using over 5 times the compute of the largest model examined in prior work Frantar et al. 2024). The largest model we investigate requires 4.5 x 1020 FLOPs training compute.
Pruning. Sparse pre-training involves pruning parameters an LLM during the pre-training pro-
cess. While there are many pruning algorithms available (LeCun et al., 1989. Hassibi et al. 1993]
Han et al. 2015 He et al.) 2017] Frankle & Carbin 2019; Renda et al., 2020 Peste et al. 2021
Evci al. 2020). , we focus on simple class of algorithms known as iterative magnitude pruning
(Zhu & Gupta 2011 7 Frankle & Carbin 2019 Renda et al. 2020). Since sparse pre-training results in sparse model the end of training, our method can also be viewed
pruning algorithm. In contrast to prevailing approaches that train large dense model and then prune it while preserving
accuracy (Frantar& Alistarh| 2023 Sun al. 2024). our analysis shows that, within the sparsity
We adjust the hidden dimension of dense models to match the sparse models, but since it must be divisible by the number of attention heads and chips, we create two dense models that hest approximate the target parameter count and linearly interpolate hetween them.
3

---

Published as a conference paper at ICLR 2025
levels examined, it is possible directly train sparse model that achieves the same final loss using the same compute budget as training the large dense model. This simplifies the model devel- opment process by eliminating the need for pruning a post-training step.
Dynamic parameter schedule. While practitioners typically pre-train LLMs with fixed number of active parameters throughout the training process (Groeneveld et al. 2024 Shoeybi et al. 2020),
a growing line of work explores varying this number during pre-training to improve computational
efficiency (Yao et al.) 2024, Panigrahi al. 2024 Yano et al., 2024). This line of work often focuses on gradually increasing the number of parameters during training. Yao et al. (2024) proposed progressive growth during pre-training using stage, multi-axis growth schedule. Panigrahi et al. (2024) introduced layer dropout, progressively reducing the number of dropped layers during training. Yano et al.) (2024) developed STEP, which begins pre-training with small model and gradually increases its size in stages. Our work may be viewed proposing another dynamic parameter schedule. However, differs by focusing on compute-optimal strategies that gradually reduce model size, optimizing for inference efficiency. Since the final model is smaller, our approach leads to more efficient inference compared to methods that progressively increase model size.
3 PRELIMINARIES
Algorithm. We adopt the iterative magnitude pruning (IMP) algorithm for pre-training LLMs ZZuu & Gupta 2017 Renda al. 2020) Frankle & Carbin 2019 Samar 2022 Frantar et al.) 2024). We
score each parameter's importance based on its magnitude. At each pruning iteration, we rank all model parameters globally and prune those with the lowest magnitudes. No structural constraints imposed the sparsity pattern. Our sparse pre-training algorithm consists of three phases:
1. Dense training phase: We train the dense starting model with all its parameters for Npre steps;
2. Iterative pruning phase: We iteratively remove parameters. Each pruning iteration starts by removing fixed fraction of the remaining parameters, and then training for P gradient steps. This continues for Nprune pruning iterations, until the model reaches the desired sparsity S.
3. Sparse recovery phase: Fixing the sparsity pattern from the previous phase, further train the sparse model for Npost steps to recover any accuracy that is lost due to pruning.
Based on Frantar al. (2024) Zhu & Gupta (2017): Bambhaniya et (2024) we fix P 100.
Given starting dense parameter count, target sparsity S, and a compute budget, we systematically search for the optimal allocation of compute across these three phases, Npres Nprune and Npost
within a grid of possible allocations, to achieve the best evaluation loss in Section 6
Effective compute. Following (Kaplan et al., 2020 Hoffmann et al. 2024), we approximate the total compute as 6 times the number of parameters multiplied by the number of training tokens. We scale this linearly with sparsity to report effective compute for sparse models (Frantar et al., 2024), noting that our implementation masks pruned parameters rather than eliminating their computation.
Chinchilla scaling law. Neural scaling laws model changes in final validation loss under the growth of parameters, data, compute, etc. One widely used scaling law is the Chinchilla scaling law by Hoffmann et al. (2024). This law models the relationship between the loss L, the number of param- eters N, and the number of training tokens D using the following equation:
A L(N, D) = No
+ E,
(I)
D
where A. B, a, B3, and are free parameters: and a describe how loss decreases with increasing model size N, while B and 3 describe how loss decreases with increasing training tokens D. The constant E represents an irreducible loss. Scaling laws allow practitioners to optimize training configurations to fit within their compute budgets without running costly experiments (Sardana et al. 2024). The laws also hold theoretical value, as they capture the dynamics of how loss changes with data and parameter scaling. We present overview of existing sparse scaling law in Appendix
4

---

Published as a conference paper at ICLR 2025
4 METHODS
Models. We pre-train both sparse and dense models with starting parameter count ranging from 58M to 468M. We use the LLaMA 2 (Touvron et al., 2023) base model architecture. For each unique model size, we train two versions: one using over 10x the number tokens corresponding to Chinchilla optimal, and the other using 20x Chinchilla optimal. We train substantially past Chinchilla-optimal dataset size, following prevailing practice (Touvron et al. 2023 Grattafiori et al. which benefits inference efficiency.
2024
Table provides model details. Each model name contains the parameter count and suf- fix indicating training duration: "10x" means training tokens exceed 10x Chinchilla-opimmal and "20x" means they exceed 20x (Hoffmann al. 2024). We prune only linear layer pa- rameters, leaving embedding and normalization layers dense. For each dense model, we list its total training tokens and the ratio of tokens to prunable parameters (in parentheses). We train four sparse variants for each dense model, matching their training compute while varying sparsity from 20% to 80%.
Name
# Prunable
# Tokens
58M-10x
42M
14.7B (205x)
58M-20x
42M
29,4B (409x)
162M 10x
138M
33.5B (207x)
162M-20x
138M
67.1B (414x)
468M-106
435M
94,4B (217x)
468M-20x
435M
188.8B (434x)
Table 1: Training details for models by size and token count. Numbers in parentheses show the token-to-prunable parameter ratio.
Dataset. We use the 'en' partition of the C4 dataset (Raffel al., 2020). Using the LLaMA 2 tokenizer, this dataset can be tokenized to 197.71 billion tokens.
Software and hardware. Our work TPUv4 and TPUv5 hardware for training LLMs. We modify MaxText (Davidow et al. 2024) to support sparse pre-training of LLMs.
SCALING ANALYSIS
While Equation ( effectively models dense pre-training loss, we show that by changing the pa- rameter count term to average parameter count, the same functional form extends to model sparse pre-training. Here, we derive this modified scaling law analytically and validate empirically.
5.1 UNIFIED SCALING LAW THAT MODELS DENSE AND SPARSE PRE-TRAINING
Let T denote the total number of pruning iterations, where each iteration consists of parameter removal step, followed by training at that fixed sparsity. Thus, we may represent a sparse pre- training sequence (N1, Di), (N2, D2) ...,, (Nr, DT), where N₂ is the number of remaining parameters, and D3 is the number of tokens at iteration & E {1...., "i....,Ty T). Let N denote the average parameter count during training, i.c., N = + Ng. When N₂ = N for all k, k', we recover dense training, where the number of parameters does not change throughout training, and N = Ni.
We model the relationship between the final evaluation loss L, the average number model param- eters N, and the total number of training tokens using the following equation:
A
L(N,)) No
+
(2)
DE
where is the total number of training tokens, and A, B, E, a, and 3 are free positive parameters. Importantly, our proposed scaling law retains the same functional form as the Chinchilla scaling law in Equation (T). but replaces the number of dense parameters N with the average parameter count N. When sparsity set to 0, scaling law reduces the original Chinchilla scaling law.
5.2 THEORETICAL ANALYSIS: AvG PARAM COUNT AS EFFECTIVE MODEL SIZE
Using certain assumptions either standard prior work or validated through empirical observa- tions, provide analytical derivation demonstrating that the average parameter count in sparse pre- pre-training plays the equivalent role of the fixed parameter count in dense training. This motivates unifying both pre-training techniques under the same scaling law shown in Equation (2)
5

---

Published as a conference paper at ICLR 2025
Assumptions. Our analysis rests two key assumptions: empirically justified and one exten- sively used in prior work:
1. Log loss decays linearly with log compute for fixed parameter count pre-training;
2. Total compute for processing fixed number of tokens is proportional to the number of active parameters in the model.
Kaplan et al. (2020) presented empirical evidence for Assumption (1) when fixed parameter count pre-training falls within the "compute optimal" regime, meaning that model is appropriately sized to fully utilize the available compute. In our experiments, we extensively tune our sparse pre- training configurations to bring us close to this regime. In this setting, it is known that the loss evolves function of the training compute C as
(A/C)),
where L(C) represents the loss at compute C, and a > 0 is constant that governs the of loss decay compute increases. We apply this relationship to model loss decay within each pruning iteration, where parameter count stays fixed.
1e-17
Fit for Compute 1e19
Coefficient C, [_(0=)><<aa) (-alpha-1
10.0
Fit Compute le19
alpha=0.0411
alpha=0.203
Final Eval Loss
3-
5
1046
10
10
10
10
102
10a
Compute (FLOPs)
Pruning Iteration (k)
Figure 2: Left: Loss effective compute for 410M models for various sparsity levels and training durations. Right: Estimated a coefficient in the scaling law for 410M model.
Assumption (2) has been heavily used in previous scaling law work (Kaplan al. 2020 Sardanal et al.] 2024] Frantar et al.] 2024), which model compute as proportional to the number of parameters, batch size, and number of iterations. Since we use fixed number of tokens per step, the compute per step is directly proportional to the number of active parameters at that step.
Loss modeling. To derive the average parameter scaling law, start with Taylor series expan- sion of the loss, as modeled by Equation (3) (Assumption 1), at pruning iteration k, where Co.k denotes the cumulative training compute up to iteration This yields an approximation for the change in loss due to an incremental increase in compute AC₁:
AL -a -aA A ACC.
(4)
Applying the above approximation across all pruning iterations, and relying on Assumption 2, we express the total changes in loss over T training steps as proportional to
A Liottll
Σ AL O
Coe-l x No.
(5)
where we have used the fact that computation is known to be linear in the number of parameters Ka- plan et (020) (Assumption 2): AC₁ x Nç.
For realistic values of C and a, we find that the terms C remain very stable. In Figure (right), plot C-a-1 function of pruning iterations for the 410M model experiments, with

---

Published as a conference paper at ICLR 2025
empirically fitted values of 0.041 and 0.203. After about 100 pruning iterations, we observe that this coefficient becomes essentially constant. Further, note that this sum of loss decreases does not take into account the temporary increase in loss due to pruning at specific step. This is justified as we have observed empirically that, during training, the loss spikes at the pruning step do not significantly affect the final pre-training loss. One may observe this from Figure 2 (left), where present the loss curves for sparse pre- pre-training 410M models. Instead, the loss only depends on the number of non-zero parameters and on the amount of computation during the iteration.
Effective model size. Summing across all T iterations, we find that the total change in loss AL proportional to the average parameter count during sparse pre-training. Taking the log of this relationship shows that the log change in loss varies linearly with log average parameter count. This demonstrates that sparse pre-training follows the same log-linear relationship between loss and model size as dense scaling Equation (3) suggests, motivating to extend established dense scaling principles to sparse settings by using average parameter count as effective model size.
Further validation. Our analysis predicts that sparse pre-training, despite its varying parameter count, should maintain the same log-linear relationship between loss and training compute as dense pre-training. Figure (left) shows the relationship between the training loss of sparsely pretrained 410M models and effective training compute consumption. The data confirms that the log-linear relationship largely holds. We empirically observe transition point around 10 floating point operations, corresponding to the first 2.6% of training steps. A linear fit on the loss data before and after this point estimates the scaling parameter to be approximately 0.041 and 0.203, respectively.
5.3 EMPIRICAL ANALYSIS
In this subsection, fit our proposed scaling law with empirical data.
Fitting method. We optimize key aspects of sparse tre-iingn configuration, including the learning rate, batch size, and compute al- locations across the three stages of sparse pre- training (see Section 6 for details). We fit our proposed scaling law (Equation (2) using the final evaluation loss obtained from sparse pre- training experiments with these optimal config- urations. Our experiments cover 5 sparsity lev-
Actual v.s. Predicted Eval Loss
3.2
Predicted
3.0
2.8
els (0%, 20%, 40%, 60%, 80%), 3 model sizes (58M, 162M, 468M), and 2 training durations (10x Chinchilla optimal and 20x Chinchilla op- timal), producing 30 data points.
2.6
2.2
Following the methodology in Hoffmann et al. (024). used the L-BFGS algorithm (Liu & Nocedal 1989) and a Huber loss with d == 1x 1x improve robustness against outliers. We set the maximum number of L-BFGS iter- ations 1000, which we empirically find suit- able for ensuring convergence. We initialized the scaling law's free parameters with the same ran- dom values as in (offfman et al. (2024). To account for possible local minima, we sampled 100 initializations from the random grid and selected the parameters with the best Huber loss, following the precedent in Hoffmann et al. (2024); Frantar et al. (2024).
2.25
2.50
2.75
3.00
3.25
3.50
Actual
Figure 3: Predicted eval loss from fitted scal- ing law versus the actual achieved final loss.
Results. We present the predicted model evaluation loss and the actual final evaluation loss in Figure 3 Across different model sizes and training durations, our fitted scaling law models the final model loss with sufficient accuracy. Specifically, the average absolute difference between the predicted and actual loss is 0.016. The distribution prediction error varies across sparsity levels: the maximum mean absolute difference occurs at 60% sparsity with 0.03, while the minimum occurs at 0% sparsity with 0.007. We attribute this disparity to the sealing analysis not fully accounting for the regularization effects of sparsity (Jin et al. 2022

---

Published as a conference paper at ICLR 2025
3.20
20%
20%
2.3
230
2:22
2.1
3,00
Sparsity
40%
2.0
230
2.43
2.1
2.79
334
231
Normallzon los
Sparsity
40%
274
2.24
2.2
2.95
60%
187
2:44
2.07
2.34
2:00
60%
12
1.3
2.71
2.38
2.85
2.80
80%
80%
75%
100%)
(25% 25%
50%0
(75%, 25%)
(0%, 25%)
(0%. 50%
(0%. 75%
(0%, 100%
125%. 25%)
125% 50%)
150% 25%
150% 50%)
175% 25%0
(0%
(0%,
(0%
(0%
28%
(50%
(50%
Sparsity Schedule
Sparsty Schedule
Figure 4: Optimal sparsity schedule sweep for 162M-10x (left) and 162M-20X (right) models.
Each tuple on the x-axis, (ta, t,), represents the percentage of training time spent for dense traning (ta), and percentage time spent gradually pruning (tz).
Conclusion. By replacing the parameter count with average parameter count in Chinchilla scaling law (Equation (T) we demonstrate that this modified scaling law, presented Equation 2 can effectively predict the final evaluation loss of sparse pre-training.
SEARCHING FOR THE OPTIMAL SPARSE PRE-TRAINING CONFIGURATION
In Section 3 we introduced the three phases of sparse pre-training-dee dense training, iterative pruning, and sparse recovery. We now examine how to optimally allocate compute across these pre-training phases. In this section, present experimental search for optimal sparse pre-training configu- rations; we used these searched optimal configurations to validate our scaling law Section
6.1 SPARSE PRE-TRAINING CONFIGURATION SWEEP.
Sparsity schedule sweep. For each combination of initial dense parameter count, target sparsity, and training compute budget, we optimize compute allocation across the three training phases (Section to maximize final model evaluation loss.
large body research shows that pruning too early can trap models in suboptimal minima (Gale al. [2199] Frankle al. 22199) Paul et al. 2023: Bambhaniya et al.) 2024 Lu al. (0033). These
findings suggest that one needs to allocate substantial compute to the dense training phase before iterative pruning phase may begin. Similarly, research also shows that removing
many weights
in one iteration leads to poor final model quality (Renda et al.) 2020).. Therefore, reaching target sparsity requires sufficiently large number of pruning iterations, each removing only moderate fraction of weights. These competing demands make sparsity schedule design challenging.
Focusing the 162M-10x and 162M-20x models, we systematically search for the optimal spar- sity schedule by evaluating all valid combinations of compute allocated to the dense training phase (0%, 25%. 50%, 75% of total training FLOPs) and the weight removal phase (25%, 50%, 75%, 100%). For a schedule to be valid, the combined compute for the dense training and weight removal phases must not exceed 100%, with the remaining compute allocated to the sparse recovery phase. For this sweep, we adopt the same hyperparameter configurations (learning rate, batch size, etc.) used for equivalent model configurations in the Pythia suite of models (Biderman et al. 2023).
Sparsity Schedule Results. We present the results of our sparsity schedule sweep in Figure We encode each schedule on the x-axis with a 2-tuple. The first value represents the proportion of compute allocated to the dense training phase, and the second value represents the compute allocated to the weight removal phase. Our results consistently show that allocating 25% of total compute to the dense training phase and 50% the weight removal phase yields either the optimal training loss or a result within 0.01 of the minimum.
Learning rate and batch size sweep. We subsequently investigate the optimal learning rate and batch size to use for sparse pre-training. We sweep through grid of [0.0004, 0.0016, 0.0064] for learning rate and (0. 125M, 0.5M, 2M] for batch size.
8

---

Published as a conference paper at ICLR 2025
31.28
Toks, 0%
35,08 Toks, 20%
40.8B Toks, 40%
51.2B Toks, 60%
78.38 Toks, 80%
Normalized loss
Batch Size
128K
2.7
2:72
2:92
2.72
2.74
2.92
2.76
2.79
2:97
2.87
512K
2.73
2.67
2.78
2.78
2.72
2M
2.83
2.7
2.82
2.83
2.7
2.84
2.84
2.71
2.83
2.87
2.75
2.88
2.99
2.85
2.93
3
(xle-4) Leaming
(xle-4) Learning
(xle44 Learning Rate
(xle-4)
Learnnng (xle-4) Learning
62.5B
Toks, 0%
70.18 Toks, 20%
81.4B Toks, 40%
102.38 Toks, 60%
156.68 Toks, 80%
-10
Baxch Size
128K
2.67
2.72
2.73
2.9
2.74
2.78
2.92
2.82
2.85
2.98
Normalized
512K
2.68
2.64
2.73
2.69
2.65
2.74
2.7
2.67
2.78
2.73
2.7
2.82
2.83
2.79
2.83
0.5
2M
2.74
2.65
2.69
2.74
2.66
2.7
2.76
2.67
2.75
2.8
2.7
2.74
2.92
2.81
2.85
0.0
1
12
(x1e-4) Leaminn Rate
(x]e-4) Learning Rate
(:le4) Learning Rate
(x1e-4) Leariing Rate (x]e-4) Learning Rato
Figure 5: Batch size and learning rate sweep for 162M models.
Learning rate and batch size results. We present our learning rate and batch size sweeps in Fig- ure Our findings consistently show that using the optimal hyper-parameter for dense pre-training either achieves the optimal evaluation loss for sparse pre-training or comes very close (within 0.01 difference).
6.2
CLOSER LOOK PRUNING SCHEDULE
20%
20%
3.1
20%
50%
3.0
40%
3.1
40%
80%
3.0
40%
80%
3.0
2.0
2.9
2.9
Tinal
2.9
2.8
2.0
2.7
2.7
2.7
2.7
2.6
2.6
10
20
30
40
10
30
50
30
50
70
30
50
70
Dense Training 1% Compute!
Dense Training (% Computee
Iterative Pruring Compute!
Reeative Prining (% Computel
(a) Pruning Start (10x)
(b) Pruning Start (20x)
(c) Pruning Duration (10x) (d) Pruning Duration (20x)
Figure 6: closer look failure modes for non-optimal sparsity schedules.
In this section, we visualize the impact of allocating different fractions of the total compute to the dense training and iterative pruning phases on final evaluation loss.
Dense training compute. To determine the optimal fraction of compute to allocate to the dense pre-training phase, we vary the dense compute between 0%, 25%, and 50%. Throughout these experiments, we keep the iterative pruning phase compute fixed at 50%, as this was previously found to optimal, and adjust the sparse recovery phase compute to maintain a constant total compute for sparse pre-training. The results are shown in Figures 6a and
Our findings indicate clear optimal allocation of dense training compute 25% of total compute, where 7 out of cases reach their lowest loss. This trend is consistent across different sparsity levels (ranging from 20% to 80%) and both training regimes (10x and 20x Chinchilla-optimal).
Additionally, the results suggest that allocating too much compute to dense training (50%) leads to worse final loss, particularly for high-sparsity models. This underscores the importance of allocating sufficient compute to the sparse recovery phase.
Iterative pruning compute. We explore the effect of varying the compute allocation to the iterative pruning phase. First, fix the dense training compute 25%, this was previously found to be optimal. Then, we vary the pruning duration while adjusting the sparse recovery phase to keep the total compute constant. We visualize these results in Figures 6c and 6d
We find that the optimal allocation iterative pruning compute varies across training regimes. For the 10x Chinchilla model, the lowest evaluation loss occurs with 25% allocation to iterative prun-

---

Published as a conference paper at ICLR 2025
ing (2.83), while for the 20x Chinchilla model, the optimal allocation is 50% (2.77). Despite these differences, allocating 50% total compute to iterative pruning consistently results in reasonable evaluation loss across both regimes. Interestingly, extending the iterative pruning allocation beyond 50% tends to degrade performance, particularly in high-sparsity models (80%). These findings sug- gest that high-sparsity models benefit most from moderate allocations iterative pruning, ensuring sufficient compute for sparse recovery after weight removal.
7 CONCLUDING REMARK
Our work examines sparse pre-training for large language models (LLMs) and presents unified scaling law that effectively models both sparse and dense scaling.
Value of sparse pre-training. Sparsely pre-trained LLMs match the final evaluation loss of dense models when their average parameter counts are the same (Figure DT. Unlike dense pre- traininnnng which maintains a constant parameter count, sparse pre-training starts with more parameters and gradually reduces them, effectively redistributing parameter count over training steps. For the same average parameter count, this redistribution does not change total training compute. Within the model sizes, sparsity levels, and schedules explored, redistributing parameters maintains the same training compute and evaluation loss trade-off as dense pre-training while achieving smaller final model size. This makes sparse pre-training valuable: it improves the three- way trade-off among training compute, evaluation loss, and final model size.
Scaling prescription. Chinchilla (Hoffmann et al. 2024) defines compute-optimal LLM training configurations by minimizing training compute alone, yet practical LLMs like LLaMA TTourron al. 2023; Grattafiori al.] (024) often train longer reduce inference costs. [Sardana et al. (2024) formalizes this practice by minimizing lifetime compute-the sum of training and inference compute. Applying the same analysis with our scaling law (see Appendix BB, we find that training with less data can achieve the same target loss while reducing lifetime compute, compared with the prescription by [Sardana et al. 2024). This surprising advantage stems from sparse pre-training's key feature: it decouples the average parameter count during training, which governs model quality, from the final parameter count after training, which determines inference compute. This decoupling enables better balance between training efficiency and inference costs, making sparsity a crucial factor in designing compute-optimal training configurations for LLMs.
Compression rate. Our findings suggest that up to a certain compression rate, sparse pre-training compresses the LLM during training without loss in quality. For a given sparse pre-training con- figuration, we define its compression rate as the ratio between two model sizes: the smallest dense model that matches the sparse model's evaluation loss (while keeping other factors like compute, data constant) and the final sparse model size. From analysis, we know that a dense model needs as many parameters as the sparse model's average parameter count during pre-training to match its quality. Therefore, for sparse pre-training configuration, its compression rate is also the ratio be- tween average and final parameter counts. Within the LLMs we explore, we reach the maximum compression rate 80% final sparsity, where our sparsity schedule results in an average parameter count of about 40% of the initial dense parameter count, yielding a 2x lossless compression rate.
Limitation. We note that, due to the lack of adequate software and hardware support for executing matrix multiplications with unstructured sparsity, we are unable to demonstrate computational sav- ings from sparse pre-training. For the same reason, the scale of our study is limited, as the actual compute required may be up to 1/(1 sparsity) times the effective compute we estimate for sparse pre-training. For experiments with 80% sparsity, this factor reaches 5x.
Conclusion. In this work, we unify sparse and dense pre-training through single scaling law that uses average parameter count, showing its effectiveness in modeling evaluation loss across model sizes, sparsity levels, and training durations. Our analysis reveals several practical benefits: sparse pre-training reduces inference costs without sacrificing model quality or increasing training compute; compared to existing dense pre-training preseriptions, incorporating sparsity maintains model quality while reducing both training data and lifetime compute costs. Together, our work establishes sparse pre-training promising alternative to dense pre-training.
10

---

Published as a conference paper at ICLR 2025
Reproducibility statement. We have detailed our dense LLM training configurations in Section 4 along with our hardware and software setup. Additionally. in Section 6 discussion of the pruning configurations explored and identified.
we provide an extensive
Acknowledgment. We are deeply grateful to Elias Frantar, Naveen Kumar, Sanjiv Kumar, Daniel M. Roy, and Clemens Schaefer for their valuable feedback and thoughtful review of this paper. We also acknowledge the critical support provided by the Google CoreML Performance Team, and Google Research during this project. We further recognize the extended team Google DeepMind. who enabled and supported this research direction.
This work was in part supported by the Sloan Foundation, the MIT-IBM Watson Al Lab, Apple, and SRC JUMP 2.0 (CoCoSys).
REFERENCES
Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-Chun Kao, Shivani Agrawal, Utku Evci, and Tushar Krishna. Progressive gradient flow for robust n.m sparsity training transformers. arXiv preprint 240v:22402.04444, 2024.
Michele Banko and Eric Brill. Scaling very very large corpora for natural language disambigua- tion. In Anmual Meeting of the Association for Computational Linguistics, 2001.
Yamini Bansal, Behrooz Ghorbani, Ankush Garg, Biao Zhang, Colin Cherry, Behnam Neyshabur, and Orhan Firat. Data scaling laws in NMT: The effect of noise and architecture. In International Conference on Machine Learning, 2022.
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprins arXiv:2304.01377 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Matthew Davidow, Mohit Khatwani, and Michelle Yoo et al. Maxtext: framework for training large language models.
com.AI htt.a/a--epperrouurerrraaaa //wxtexxee//axxtxttteeeeeee 2024.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference Machine Learning. 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. International Conference on Machine Learning, 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.000 0070744 2023.
Elias Frantar, Carlos Riquelme Ruiz, Neil Houlsby, Dan Alistarh, and Utku Evci. Scaling laws for sparsely-connected foundation models. International Conference on Learning Representations, 2024.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 1904.09544 2019.
11

---

Published as a conference paper at ICLR 2025
Behrooz Ghorbani, Orhan Firat, Markus Freitag, Ankur Bapna, Maxim Krikun, Xavier Garcia, Ciprian Chelba, and Colin Cherry. Scaling laws for neural machine translation. In International Conference on Learning Representations, 2022.
Joshua Goodman. A bit of progress in language modeling. arXiv preprint arXiv:es CL/0108005, 2001.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan, Data and parameter scaling laws for neural machine translation. In Conference on Empirical Methods in Natural Language Processing, 2021.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Ko- renev, Arthur Hinsvark. Arun Rao, Aston Zhang, Aurelien Rodriguez. Austen Gregerson, Ava Spataru. Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak. Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind That- tai, Graeme Nail, Gregoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Kore- vaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jong- soo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Aríni, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins. Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh. Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathicu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Ku- mar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bushlykov, Nikolay Bogoy- chev, Niladri Chatterji, Ning Zhang, Olivier Duchemne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ra- mon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Ro- hit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mi- haylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Gold- schlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song. Yuchen Zhang, Yue Li. Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexci Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, An- drew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, An- nie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leon- hardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic. Brian Gamido, Britt Mon- talvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia
12

---

Published as a conference paper at ICLR 2025
Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Elconora Presani, Emily Hahn, Emily Wood, Erie-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smoth- ers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harri- son Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang. Jen- nifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill. Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Jun- jie Wang. Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhaut, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vattskov, Mikayel Samvelyan. Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Ro- driguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang. Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ra- maswamy, Shaun Lindsay. Shaun Lindsay, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar. Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satter- field, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Ku- mar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang. Xiao- jian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhao- duo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Au- thur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik. Crys- tal Nam, Matthew Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, William Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan I Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah Smith, and Hannanch Hajishirzi. OLMo: Accelerating the science of language models. In Annual Meeting of the Association for Computational Linguistics, 2024.
Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems, 2015.
B. Hassibi, D.G. Stork, and G.J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
13

---

Published as a conference paper at ICLR 2025
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In IEEE International Conference on Computer Vision, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hen- nigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae, and Laurent Sifre. Training compute-optimal large language models. In Advances Neural Information Processing Systems, 2024.
Tian Jin, Michael Carbin, Daniel M. Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Prun- ing's effect generalization through the lens of training and regularization. In Advances in Neural Information Processing Systems, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Denis Kuznedelev, Eldar Kurtic, Eugenia Iofinova, Elias Frantar, Alexandra Peste, and Dan Alis- tarh. Accurate neural network pruning requires rethinking sparse optimization. Transactions on Machine Learning Research, 2024.
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Informa- tion Processing Systems, 1989.
Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1):503-528, 1989.
Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yazdanbakhsh. Step: learning n:m structured sparsity masks from scratch with precondition. In International Conference on Machine Learning, 2023.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang. Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference on Learn- ing Representations, 2020.
Abhishek Panigrahi, Nikunj Saunshi, Kaifeng Lyu, Sobhan Miryoosefi, Sashank Reddi, Satyen Kale, and Sanjiv Kumar. Efficient stagewise pretraining via progressive subnetworks. arXiv preprint arXiv:2402.059113 2024.
Mansheej Paul, Feng Chen, Brett W. Larsen, Jonathan Frankle, Surya Ganguli, and Gintare Karolina Dziugaite. Unmasking the lottery ticket hypothesis: What's encoded in a winning ticket's mask? In International Conference on Learning Representations, 2023.
Alexandra Peste, Fugenia Iofinova, Adrian Vladu, and Dan Alistarh. AC/DC: Alternating com- pressed/dccompressed training of deep neural networks. In Advances in Neural Information Pro- cessing Systems, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(1), 2020.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020.
Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In International Conference on Machine Learning, 2021.
Anshul Samar. Creating sparse gpt-3 models with iterative pruning. https:// ai/ bloq/ wertiing-sppssss opt - 3- models sonddllswiiii siwwwthh---- www.iiiereeeeunnniii. 2022. Ac- cessed: 2025-01-12.
14

---

Published as a conference paper at ICLR 2025
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Ac- counting for inference language model scaling laws. International Conference on Machine Learning. 2024.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint arXiv:1909.0880 000000 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. In International Conference Learning Representations, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshor, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar 23x.:2307700288 2023.
Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared Ilama: Accelerating language model pre-training structured pruning. arXiv preprint urXiv:2310.06694 2023.
Kazuki Yano, Takumi Ito, and Jun Suzuki. Step: Staged parameter-efficient pre-training for large language models. In Annual Meeting of the Association for Computational Linguistics (Student Research Workshop), 2024.
Yiqun Yao, Zheng Zhang, Jing Li, and Yequan Wang. Masked structural growth for 2x faster lan- guage model pre-training. International Conference on Learning Representations, 2024.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878, 2017.
A EXISTING SPARSE SCALING LAW
Frantar et al. 2024 propose model for the evaluation loss of a sparsely pre-trained LLM based on the final sparsity S. the number of non-zero parameters N the end of pre-training, and the amount of training data D, as follows:
bx
bp
L(S,N, = (as(1-S)) syds + cs)
+
+
Compared to the Chinchilla scaling law, this model introduces the following modifications:
1. A sparsity term as(1 S)²s which introduces two new sparsity-speciff parameters, 45 and by. allowing the model account for the effect of sparsity on loss.
2. Instead of using the total number of parameters N as in dense training, the model uses the number of non-zero parameters remaining at the end of sparse pre-training.
B SPARSE SCALING SUGGESTS TRAINING WITH LESS DATA
Following the methodology of Sardana et al. (024). we compute the recommended dataset size and model size to achieve specific model loss according to Chinchilla Hoffmann al. (2024).
15

---

Published as a conference paper at ICLR 2025
Beyond Chinchilla Sardana et al. (2024), and our proposed scaling laws. We find our proposed sparse pretraining technique recommends using less data.
For target training loss of 1.89, and total expected inference traffic of 100T tokens, have these recommendations:
Chinchilla: Optimizing for training compute, Chinchilla scaling law recommends training 70B model for 4.26T tokens, the model consumes 15.8 x 10 FLOPs over its lifetime.
Beyond Chinchilla: Optimizing for training and inference compute, Chinchilla scaling law recommends training a 23.5B model for 24.4T tokens. With 100T inference tokens, the model consumes 8.14 x 1021 FLOPs over its lifetime.
Optimizing for training and inference compute and fixing sparsity level at 80%, our proposed scaling law recommends training a model with average parameter count of 28.0B, resulting in 14.0B model at the end of pretraining for 16.6T tokens. With the same 100T inference tokens, the model consumes 5.58 x 10 FLOPs over its lifetime, resulting in 31.4% additional lifetime compute saving. This number grows even larger with increased inference traffic, since the benefit of sparse pretraining lies predominantly in inference compute saving.
Ours:
Compared with Sardana et al. (204)), our prescription achieves the same target loss while using less data and less compute, suggesting that sparse pretraining more data efficient.
The intuition behind our counter-intuitive prescription is that sparse pretraining decouples average parameter count during training (which correlates with quality) from inference parameter count (which correlates with inference compute). This decoupling allows us to train models with larger average parameter counts while still producing a smaller model for inference compared with tradi- tional dense scaling.
C VALIDATION USING LARGER MODELS
In Figure [ we demonstrated that sparse and dense pre-training runs, starting with 138 million prunable parameter models and matching average parameter counts, achieve similar perplexity. This section extends the same analysis to models exceeding billion parameters to evaluate the generality and robustness of these observations.
Method. We pre-trained a sparse model starting with 1.14 billion parameters, pruning it to 50% sparsity over 144 billion tokens. The corresponding dense model was configured to match the sparse model's average parameter count during training. Both models used the same compute budget of 6.8 x 2020 FLOPs, approximately 5x the Chinchilla-optimal budget for 1 1B-parameter model. The sparse model uses global iterative magnitude pruning, while the dense model followed standard training setup.
Results. The sparse model achieved final perplexity of 2.44, closely matching the dense model's 2.46. At the end of training, the dense model has 1.34 times as many parameters as the sparse model, highlighting that sparse pre-training achieves significant parameter reduction while main- taining comparable performance.
Conclusion. These results confirm that sparsely pre-trained language models with matching average parameter counts achieve comparable perplexity to their dense counterparts, even at a larger scale. This consistency demonstrates the generality and robustness of our observations.
D DOWNSTREAM TASK PERFORMANCE
While perplexity is important metric, downstream task performance is critical for understanding the practical utility of sparse pretraining. This section evaluates the pair of sparse and dense models produced in Appendix[C on various downstream benchmarks to assess their task performance.
Method. To evaluate downstream task performance, we selected four commonly used benchmarks that test reasoning and commonsense knowledge:
16

---

Published as a conference paper at ICLR 2025
PIQA (Physical Interaction Question Answering): This dataset evaluates model's abil- ity to reason about everyday physical commonsense, such as choosing the most plausible way to perform physical task.
ARC (A12 Reasoning Challenge): ARC consists of grade-school science questions de- signed to test reasoning and application of scientific knowledge, divided into Easy and Challenge subsets.
Lambada: This dataset tests model's ability for long-range coherence by requiring it to predict the last word passage, where understanding the full context critical.
Winogrande: This dataset evaluates commonsense reasoning by asking the model to resolve pronoun references in ambiguous sentences.
We evaluated both the sparse model (1.14B parameters, 50% sparsity) and the dense model (match- ing average parameter count) on the aforementioned datasets.
Results. As shown in Table the sparse and dense models achieved nearly identical performance across all benchmarks. The sparse model's average geometric mean aceuracy was 42.42%, com- pared to 42.45% for the dense model, with no significant difference observed.
Table 2: Downstream task evaluation results for sparse and dense models.
Model
PIQA
ARC Easy
ARC Challenge
Lambada
Winogrande Geomean
50% Sparse
69.7
42.05
24.74
36.52
51.85
42.42
Matching Dense
69.26
41.58
26.37
34.62
52.41
42.45
Conclusion: These results demonstrate that sparsely pre-trained language models are equally ca- pable their dense counterparts in downstream tasks when matched on average parameter count, highlighting the practical utility of sparse pretraining.
17


Reviews:
[{"review_id": "ZhqD5PP1Wt", "reviewer": "Reviewer_poHq", "summary": "This paper conducts an overwhelming analysis of the effect of sparse pretraining on large language models (LLMs). Specifically, it proposes a new scaling law that is modified from the Chinchilla scaling law using a novel concept of the average number of active parameters during training, i.e., the averaged number of parameters that receive gradients in LLM pretraining among all training steps. The experimental results show that the proposed new scaling law sufficiently fits the evaluation loss prediction of LLM pretraining, while leveraging sparse pretraining could match the sparse, pruned model's evaluation loss to a larger, dense LLM.", "questions": "- What is the exact pruning method being used in the iterative pruning phase? A detailed description of the pruning criteria could be helpful for the audience to understand (e.g., structured/unstructured pruning and how is the pruning decision being made)\n- $\\bar{N}$ and $D$ are independent variables shown in equation (2), however, there seems no analysis in this paper showing that the effects of $\\bar{N}$ and $D$ bring to the model's evaluation loss are independent.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper extends the Chinchilla scaling law and adapts it to the scenario of sparse pretraining, where the modified scaling law uses the concept of averaged activate parameter number to sufficiently model the evaluation loss of LLMs.\n- When design a pretraining pruning schedule using the scaling law proposed in this paper, one can pretrain an LLM with fewer parameters yet matching the evaluation performance of a dense model.\n- The theoretical and empirical analysis in this paper demonstrates strong proof of the scaling law proposed in this paper.", "weaknesses": "- The scale of models being used in this paper is limited, as the authors have addressed in the limitation section. Larger model experiments are definitely useful and could bring wider impact to this paper.\n- The only evaluation metric being used in this paper is the pretrained model's perplexity (or pretraining evaluation loss) without downstream task evaluations. As models could forget knowledge in pretraining because of pruning, a finer-grained analysis with task-specific evaluations could be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2IJXAVYJz", "reviewer": "Reviewer_tYDa", "summary": "This paper investigates sparse pre-training as an alternative to post-training pruning for large language models (LLMs) and proposes a modified scaling law based on the average active parameter count. Through extensive experimentation with over 80 configurations, the authors claim that sparse pre-training can yield models with similar performance to dense pre-training, offering a more efficient approach by beginning with dense training and progressively sparsifying. However, due to limitations in hardware and software, the paper lacks direct evidence of computational savings, which is a primary motivation for sparse pre-training.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Modifying the Chinchilla scaling law to consider average active parameters is a novel approach, bridging dense and sparse pre-training frameworks effectively\n+ The study evaluates a wide array of configurations and sparsity schedules, providing a thorough analysis of optimal sparse pre-training practices", "weaknesses": "- Despite the theoretical focus on efficiency, the paper lacks a demonstration of actual computational savings due to the current limitations in sparse matrix support in hardware/software, which could weaken the case for real-world applicability\n- The study uses evaluation loss as the sole metric, without investigating the effects on real-world downstream tasks. This limits the ability to gauge the model’s practical effectiveness or generalization capabilities\n- The framework’s reliance on finely tuned, phase-specific compute allocations (dense, pruning, recovery) introduces implementation complexity, which could be challenging to replicate or scale, particularly in resource-constrained environments", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g758uDeLge", "reviewer": "Reviewer_Kf8Y", "summary": "The paper studies the scaling law for pretraining sparse language models. It argues that sparse pretraining reduces both pretraining and inference costs since the final model can be smaller and requires less computation during inference. The authors modify the Chinchilla scaling law for dense models to use the average number of active parameters (receiving gradients) in training. The modified scaling law unifies both sparse and dense models.  They also conduct extensive studies to search for optimal sparse pretraining configurations for LLMs. The key findings include pruning starting at 25% and ending at 75% obtains near optimal final eval loss, optimal learning rates, and batch sizes for sparse training match dense models under the same compute budget. The paper further provides both theoretical justifications and empirical validation for the scaling law and presents a simple recipe for sparsely pretrained models.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper studies an interesting research problem, the scaling law for pretraining sparse models. It develops a modified scaling law that unifies both dense and sparse models. \n- The analysis and empirical validation are solid and convincing. The findings are supported by both theoretical justifications and experiments over 80 sparse pretraining configurations. The optimal settings for pretraining sparse models provide valuable practical lessons.\n- The paper itself is well-written, clearly structured, and easy to follow.", "weaknesses": "- One major concern is that the scaling law for dense models seems to have less impact, especially given that generally more data and compute resources are better to obtain better and more capable LLMs, LLaMA 2 and LLaMA 3 papers provide such evidence. How would unifying the scaling law for both dense and sparse models be useful under such a context?\n- The paper mainly focuses on LLaMA style model architecture for sparse pretraining, any discussion on other sparse architectures like the mixture of experts (MoE)? Otherwise, it may be a bit overclamied. \n- The other concern is that previous scaling laws at least study model parameters with 1B scale, but this paper only discusses models under 500M parameters, making the claim of scaling law for LLM weaker. \n- While the paper acknowledges the limitation of evaluation loss, it would be very useful to report the performance numbers of downstream tasks, especially given the size of models is small. No compute limitation prohibits such evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)