Scaling Laws for Sparsely-Connected Foundation Models
Authors: Elias Frantar
Carlos Riquelme Ruiz
Neil Houlsby
Dan Alistarh
Utku Evci
Venue: ICLR 2024 spotlight (2024)
Keywords: sparsity
scaling
optimal sparsity
efficiency
foundational models
transformers
structured sparsity
pruning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: This paper proposes scaling laws for parameter sparse foundational models.

Abstract: We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i.e., "foundation models"), in both vision and language domains. In this setting, we identify the first scaling law describing the relationship between weight sparsity, number of non-zero parameters, and amount of training data, which we validate empirically across model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to characterize the "optimal sparsity", the sparsity level which yields the best performance for a given effective model size and training budget. For a fixed number of non-zero parameters, we identify that the optimal sparsity increases with the amount of data used for training. We also extend our study to different sparsity structures (such as the hardware-friendly n:m pattern) and strategies (such as starting from a pretrained dense model). Our findings shed light on the power and limitations of weight sparsity across various parameter and computational settings, offering both theoretical understanding and practical implications for leveraging sparsity towards computational efficiency improvements. We provide pruning and scaling law fitting code at: github.com/google-research/jaxpruner/tree/main/jaxpruner/projects/bigsparse.

Published as a conference paper at ICLR 2024
SCALING LAWS FOR SPARSELY-CONNECHE FOUNDATION MODELS
Elias Frantar Carlos Riquelme', Neil Houlsby', Dan Alistarh", Utku Evei* Google DeepMind, ISTAustria; correspondence: elias www.taarannntt......r..rrr
ABSTRACT
We explore the impact of parameter sparsity on the scaling behavior of Trans- formers trained on massive datasets (i.e., "foundation models"), in both vision and language domains. In this setting, we identify the first scaling law describ- ing the relationship between weight sparsity, number of non-zero parameters, and amount of training data, which we validate empirically across model and data scales; on ViT/JFT-4B and T5/C4. These results allow us to characterize the "optimal sparsity". the sparsity level which yields the best performance for a given effective model size and training budget. For a fixed number of non-zero parameters, we identify that the optimal sparsity increases with the amount of data used for training. We also extend our study to different sparsity structures (such the hurdware-friendee n.m pattern) and strategies (such starting from a pretrained dense model). Our findings shed light on the power and limitations of weight sparsity across various parameter and computational settings, offering both theoretical understanding and practical implications for leveraging sparsity towards computational efficiency improvements. We provide pruning and scal- com ing law fitting code at: how.uuuooccoooooglleeeeeerrrrrr tree/ ma in httpa///aaappeeeeeeero/eeeee/e//
INTRODUCTION
Foundation models (Bommasani al. 2021), loosely defined as large (often Transformer-based (Vaswani et al. 2017)) networks that are trained on massive quantities of highly general data, have driven significant progress in deep learning, for both natural language (Brown vision tasks (Dosovitskiy et al.] 202T). One key property of such models is the predictability of their performance when scaling various model attributes, such as the number of parameters and the amount of data or computation used (Kaplan et al., 2020p. This is encapsulated by scaling laws, which make it possible to accurately predict the final performance of model given its size and training budget (i.e., amount of training data).
al., 2020) and
A parallel trend, motivated by computational costs, has been the focus towards increased efficiency for large models. This usually achieved by employing compressed parameterizations via quantiza- tion (Gholami et al. 2021) or sparsification (Hoefler et al. (021), during inference and/or training, which can lead to reduced run-time via both software and hardware support (Elsen al. 2020 Yao et al. (022).. Despite major community interest in efficiency, the impact of these compressed representations, in particular of parameter/wweiht sparsity, the scaling behavior of foundation models is not well understood; especially, when applying powerful but expensive training- based compression methods (Jacob et al. 2018 Zhu & Gupta (017)).
In this paper, we address this gap by studying the relationship between sparsity and scaling laws for foundation models. We focus on weight sparsity, that is, on networks whose individual connections are pruned, and on Transformer (Vaswani et al.) (2017) models for both vision (Dosovitskiy et al., 2021) and language (Raffel et al. 2020bb domains. We use the massive JFT-4B (Google) 2023a) and C4 (Raffel et al.) 2020a) datasets, which are several orders of magnitude larger than what has been employed so far by the vast majority of work on sparsity. In this massive dataset regime, dense models continue
improve with prolonged training and be competitive in a fair comparison, using equal amounts of training compute and data. This is in contrast to popular pruning benchmarks, e.g., ImageNet (Deng et al.] 2009). executing for many training epochs, where dense models tend to saturate (Kuznedelev et al. (0035)). allowing sparse models to achieve major gains relative to dense models with comparable number of parameters.
unclear if sparse models can

---

Published as a conference paper at ICLR 2024
Quality of fit for L(S, N. scaling law
10a2
Optimal sparsity contours for VIT/IT
ntting data
extrapalation
3x 10
#pretraining LLPPS
1000
Measured
Dense FLOPs FLOPs
10th
87. 50%
2 10
10th
50. 50%
00%
103r
Chinchi lt
2 102
3 3x10
10
10"
16ah
10"
Predicted L
parameters
Figure (Left) Fit and extrapolation quality of the L(S, N, D) scaling law on T5/C4. (Right) Optimal sparsity Sept contours fitted on VIT/IFT, for sparse and dense costs (details in Section 2.2).
In order to quantify the benefits of sparsity, or the lack thereof, this large-dataset regime we develop joint scaling laws that relate the sparsity of a network, its effective size and the amount of data used for training. We show that, for sparsity S, number of non-zero parameters N and amount of training data/steps D. the validation loss L approximately satisfies the following law, for both vision and language tasks:
LSS D) (as(1-))) + tcs)
)
aD D
+
+
(I)
Intuitively, the first two summands capture the power law scaling in terms of capacity, i.c. sparsity and non-zero parameters, and data, respectively, while c is lower bound on the achievable task loss. In more detail, the first multiplicative term captures the impact of sparsity, here expressed as remaining density (1 S). which itself follows a saturating power-law with coefficient a,ş, exponent by and limit constant cş. The exponents by and by scale the (non-zero) parameter count N, and the data D term, respectively, as is common in classical scaling laws (Kaplan et al. 2020).
We validate this formula empirically using large vision and language datasets, several model sizes, (Left) for an illustration of the
amounts of training data and sparsity levels. Please see Figure | scaling law fit and extrapolation quality. In turn, this law allows us obtain several new insights for sparsely connected foundation models:
First, the sparse scaling law suggests that sparsity affects each model size in similar way, i.e., as a multiplicative constant to the size scaling. At the same time, sparsification does not appear to interact significantly with the data scaling; the original dense term in D is preserved.
Second, we can use our scaling law in Equation (T) to analytically derive the optimal sparsity Sept for a given inference size and training budget, allowing to predict the regime where sparsity could actually provide benefits over simple dense model rescaling and extended training.
Our analysis of optimal sparsity Sap, demonstrated in Figure ( shows that its iso-contours run parallel to the dense compute optimal Chinchilla line (Hoffmann et al. 2022) of the respective model and task. Importantly, the optimal sparsity increases with longer training. Further, while optimal dense models define a line on the parameter-FLLPp surface, optimal sparse models form a half-plane (with different sparsities unlocking multiple optimal sizes for fixed training cost).
In addition, we find that the main conclusions of our law hold also for hardware- handware-friendly n:m spar- sity patterns (Mishra al. 202I), that relative capacity gains through sparsity are consistent across domains, and that pruning well-trained dense models is more efficient than training from scratch (while sparsifying), if dense checkpoints already exist, but is significantly slower otherwise.
In sum, our results provide the first scaling law for characterizing the impact of sparsity on the perfor- mance of Transformers trained on massive datasets. From the conceptual perspective, this provides a simple tool to understand the puwer-but also the limitations-of sparsity for a given task/model com- bination. From the practical side, this be used to determine whether sparsity can be reasonable option for inference or training speedups, in settings where specific software/hardware support for such compressed representations is available.
2

---

Published as a conference paper at ICLR 2024
2 SCALING LAWS FOR PARAMETER-SPARE SPARSE TRANSFORMERS
Fair evaluation in the presence of strong scaling. In the context of modern Transformers trained
on massive datasets, popular evaluation approaches for pruning (Gale et al.) 2019 Singh & Alis-
tarh 2020; Sanh et al., 2020 Schwarz et al. 2021 [enbaki et al. (023) that have been reasonable
for standard benchmarks s like ResNet50/lmageNe (Singh & Alistarh 2020; Schwwarz et al., 2021)
or BERT/GLUE (Sanh et al. 2020 Kurtic et al.] 2022), require careful reconsideration to ensure
meaningful comparisons. Specifically, it is critical to adopt the resource-euuivalent setting of Liu
ett al. (2118) and Jin et al. 20022) which we illustrate below.
For example, assume that model pretrained for 100k steps is pruned to 50% sparsity over an- other 100k steps (a standard setup for ResNet50/lmageNe)) The resulting network should not be compared to the original one, as it has had 2x more training overall. Further, even comparison against a 2x smaller model (same non-zero parameter count) trained for 200k steps (same amount of training) is not necessarily fair, as training this smaller dense model requires less overall compute than producing the larger sparse one (as we perform sparsification only gradually). In both cases, due to the strong scaling properties of Transformers trained on massive quantities of data (Kaplan jet al., 20200 Hoffmann et al. 2022), the respective baseline would have most likely improved sig- nificantly with more data/compute as well. Thus, the proper comparison point for sparse network a dense model with the same number of parameters trained for equivalent compule. This resource normalization, required by strong scaling and no overfitting, renders this setting very challenging.
Experimental setup overview. We execute extensive training sweeps across sparsity, size and data, which we then subsequently use to develop scaling laws. Now follows a very brief summary of our main setup; detailed discussion of all our choices, including the experiment grid and hyper- parameters, can be found in Appendix AA] In terms of models and datasets, we focus on Vision Trans- formers (Dosovitskiy et al., 202T) trained for multi-label image classification on the JFT-4B dataset (Dehghani et al. 2023). consisting of 4 billion images, as well as encoder-decoder T5 models (Raf-) fel et al.] 2020b) (improved 1.1 version (Google 2023b)) trained for maskedd-anguage-mmone on C4 (Raffel et al. 2020b), consisting of 150+ billion tokens. We follow the model's respective original training recipes (Zhai al., 2022, Raffel et al., (020b) and carry out sparsification during training via gradual magnitude pruning (Zhu & Gupta 2017). using a cubic schedule starting at 25% of training and ending at 75%. Our setup is optimized for robustness and consistency across scales rather than maximizing pruning performance on one particular setting (see also Appendix (B).
2.1 DERIVING THE CORE LAW
Dense sealing, It well established (Kaplan et al., 2020 Hoffmann et al. 2022) that the pre- training validation loss of dense Transformers can be approximately modeled, in terms of parameter count N and amount of training data D, by functions of the following form:
L(N, D)
aN N
by
ODD D
bD
+c.
The first two summands capture the power law scaling in terms of size and data, respectively. Mean- while, e represents the inherent stochasticity of the modelling problem as lower bound on the loss. The scaling exponents by and by are usually quite stable for a particular task, whereas the constant coefficients v and ap vary with minor process changes like a different architecture or optimizer.
Scaling laws usually assume an ideal training setup with no data repetition and focus on modelling the non-botttenecceer regime (e.g.. with sufficient steps//datahattchsizeeeec. rather than on edge cases (Kaplan et al., 2020 Hoffmann et al. 2022); we follow suit. Further, we deliberately consider the pretraining loss and mfinite data setting to assess the effectiveness of sparsity in its most challenging (one essentially needs to fit the data well possible) yet also most useful application (all further post-processing would directly benefit from a compressed base model; see also Appendix
Preliminary observations. The key question we hope address is how parameter sparsity S enters this core scaling relationship: understanding this will enable studying other interesting aspects like optimal sparsity or limit performance. A priori, it is not obvious how S should enter into Equation (2) to form L(S, N, D), where N denotes the number of non-zero parameters. Are larger models easier to sparsify, does longer training help highly sparse models more, or is sparsity mostly
3

---

Published as a conference paper at ICLR 2024
250K steps
500K steps
1M steps
val loss. low bou-bound
10a
6
10'
10a
10'
10
enon-zero parameters
enon-zero parameters
@non-zero parameters
00.00%
50.00%
75.00%
87.50%
Figure 2: Visualization of T5/C4 sweep results for all sizes and sparsities, grouped by training steps.
independent of other parameters? Therefore, to identify the functional form of the scaling law, we T5 training sweep over sparsity, size and steps. Figure shows bound e = subtracted to account for power law saturation against the inherent uncertainty limit) versus model size for all sparsity levels, grouped by the number of training steps. Please observe that the scaling of this plot, as well as most other visualizations in this paper, is log-log.
shows validation loss (with lower
We make three major observations from these graphs:
1. The loss vs. #non-zero curves for all sparsity levels seem to form almost parallel lines, differing primarily in the intercept.
2. The higher the sparsity the lower the loss, but gains are quickly diminishing.
3. The overall shape of all curves very similar for each training duration, the y-values just tend to shift a bit downwards with more training steps.
Sparse scaling law. We now use the previous insights to construct our L N, D) formula. Ob- servation 1 suggests that the model size power law scaling for all sparsity levels differs primarily by constant factor (intercept in a log-log plot; by (the slope) stays fairly consistent. Based on observation 2, we model this sparsity factor as a (quickly) saturating power law. Finally, observation 3 indicates that sparsity and data scaling are mostly independent, hence simply keep the original D-term. In summary, these observations lead us to the following joint scaling law:
L(S,N,D) D) (as(1 - 5y + +cs)
(
(T)
a
+c.
D
To properly model that 0.75 is twice as sparse as 0.5, we define the sparsity power-law part via the corresponding compression rate 1/(1 5). Further, an subsumed by as and cs, leaving free parameters. On a high level, our scaling law combines capacity limit term, comprised of size and sparsity (which can encode extra information via zero pattern), with the standard data limit term. Lastly, S = 0 recovers the established L(N, D) form.
TS/C4 results. Next, we fit the coefficients of L, our entire T5 sweep data. This accomplished, following (Hoffmann et al. 2022), by minimizing the Huber-loss of log L with 5 0.001 (for robustness against outliers) using BFGS, for multiple random starting points. We plot actual vs. predictions in Figure [ (Right) to judge the quality of our final fit (see Appendix C for coefficient values). All in all, the predictions match the observed data quite closely (despite having # 7 datapoints per free parameter), demonstrating the compatibility of the law in (3) with the observations.
Furthermore, we evaluate extrapolation performance by pruning 2.3 billion parameter model to 75% sparsity. This constitutes an R 6.75x larger target number of non-zero parameters than the maximum in our fitting data, which is a similar level of extrapolation as was done in the Chinchilla study (Hoffmann et al. 2022). To avoid any architecture bottlenecks and achieve better training utilization. we use the TS-XL architecture (rather than rescaled TS-base) and train with batchsize 256 for 250k steps (rather than 500k with batchsize 128). Despite these changes to our setup, the prediction of our fitted scaling law is quite close the actual validation loss; see Figure (Right).
ViT/JFT-4B results. Lastly, we execute a ViT training sweep and also fit a scaling law of the same 3 form as for the T5 data. Here we use 8 0.01 and do not take the log of I. as we find the
4

---

Published as a conference paper at ICLR 2024
NLP NLP-optimized settings from before to exclude outliers too aggressively for VIT data (which gives a poor fit for smaller models). We note that this sweep contains >> 2x more datapoints, leading to more robust coefficient estimates. We qualitatively compare predicted and actual loss-vs-data curves in Figure organized by sparsity level. We strongly emphasize that the predictions in all subplots here are produced by a single joint law with the same parameters (nol one fit per image). As can be seen, for the most part, our law appears to match the collected datapoints very well. Only at the lowest amount of training, some points are a bit off the prediction curve; we suspect that this may be related to the fact that these runs only involve comparatively few training steps, which may be slight bottleneck for the optimization process, Finally, we train 75% ViT-L for 3.6 billion images to validate extrapolation, ending up only 0.09 better than the corresponding prediction, in line with being a slightly better base architecture than the family used for our sweep (see Appendix A..
00.00% sparsity
50.00% sparsity
75.00% sparsity
87.50% sparsity
10'
9
8'"
5x 20*
Pretraining
Pretraining data
Protralning
Pretraining
0.66M
1.33M
2 25M
5.31M
10.
21. 23M
42 47M
84 93M
169 169.87M
Figure 3: Visual comparison of the ViT scaling sweep data and the corresponding fitted scaling law.
2.2 OPTIMAL SPARSITY
One particularly interesting feature of the joint scaling law just derived is that allows easily com- paring models with different sparsities but the same number of non-zero parameters and training cost. Thus, we determine in which situations sparse models are better than dense ones, accord- ing to all criteria discussed in Section 2 Specifically, we can define the following quantity:
Optimal sparsity. The sparsity value (N, which yields the lowest validation loss for fixed number of non-zero parameters N and training cost c
There are two ways of defining training costs in this context: (a) densely, as the cost of training a dense base model of size N/(1 - S) for the same amount of training steps, or (b) sparsely, as the actual FL.PPs spent to produce the sparse model, assuming that sparsity can be perfectly exploited during training as soon as it appears. For our particular sparsification schedule, (b) can be calculated by multiplying the training costs of dense model, approximated as 6ND (Kaplan et al. 2020 (or half for encoder-decoder architecture models), by (see Appendix D for this and other derivations):
Graa (8) = (0.25 + 0.50 (1 0.75 s))/(1 S) + 0.25.
(4)
As we have assumed that the amount of training equals the amount of new data, we can determine the performance of sparsity S model trained for compute C 6NN D Cmm((S) by querying L with Ds (C/ C/ cmm (S), i.c., scaling down the D corresponding to C' by the increase in training costs of the sparse model. Inserting Ds and then differentiating with respect to gives the contour line for which sparsity S is optimal, i.c., achieves the lowest loss among all possible sparsity choices, when training for the same compute:
bp (S)
an bp uD) D5 =0,b,(1 - -S)5--1.
Crnut (S) (S)
(5)
An interesting property about this contour is that it implies Dg O(Nbw/bD) meaning that if data- is stronger than size-scaling, then the same sparsity is optimal for a smaller data-to-size ratio process bottlenecked more by capacity than by data will
on larger models. This is sensible benefit more from increasing the former, e.g.. by adding sparsity. Finally, we want to point out that Sope can often also be determined explicitly by solving (4) for S, e.g., here for dense training costs with (mm(()) = 1/(1- - S):
Sopt C)= max [ exp (log by
agbs + (oylooN) + bs)] ( ( by (o/(bo ths)
( .). (6)
'We note that it is common in the literature Hoffmann al. 2022 to define scaling laws in terms of parameters N and data D. but switch to expressing scaling terms computational cost C whenever relevant.
5

---

Published as a conference paper at ICLR 2024
Empirical results. We now compute optimal sparsity curves for our experimental T5 and ViT data, for which we fit scaling laws in the previous subsection. Fig- ure (Right) and | show the optimal sparsity contours, both for dense and sparse costs. An interesting feature of Equation (5) is that all sparsity contours are, by con- struction, parallel to the Chinchilla compute optimal line
Optimal sparsity contours for T5/C4
10
ppretraining FLOPs
1000
10
Dense FLDPs
Sparse FLOPs
10
87, 07.50%
(Hoffmann et al. 2022). which denotes ideal utilization of training FLOPs for fully dense models; this can be clearly observed in the plots as well. However, we note that the Chinchilla line does not necessarily correspond to the S case since non-zero sparsity may be optimal in this regime (this is the case for sparse-FLOPs).
75. 0..00%
50, 00%
10
00. 00.00%
Chinchilla
10t
10a
10
107
neon-zero parameters
Figure 4: Optimal T5 sparsity contours.
The key taky from these results is that one trains significantly longer than Chinchilla (dense compute optimal). more and more sparse models start to become optimal in terms of loss for the same number of non-zero parameters. This is because the gains of further training dense models start to slow down significantly at some point, allowing sparse models to overtake them. We further illustrate this effect on a subset of our actual VIT data in Figure 5
0.66M #nonzeros
1.33M #nonzeros
2.65M #nonzeros
Validaation loss
10
9.0
8.0
10
10
10
10:0
10:
10th
1029
#sparse pretraining FLOPs
#sparse pretraining FLOPs
ssarse pretraining FLOPs
00.00%
50.00%
75.00%
87.50%
Figure 5: Loss vs. sparse pretraining FLOPs for ViT models of varying sparsity.
The practical question now is how much longer training is necessary? In terms of sparse FLOPs. 50% sparsity is already optimal for < 2x (ViT) and < 3x (T5) longer training than Chinchilla; for dense FLOPs is R 5x and R 70x, respectively. While the latter number seems quite high at first glance, we note that language models of the sizes we consider here are already typically trained for 100x longer than Chinchilla (Brown et al., 2020). Additionally, larger models are being trained with more and more data as well, c.g., Llama2-75 7B with z 14x Chinchilla (Touvron et al. 2023b). In general, the optimal sparsity at given point (N, C) lower for dense than sparse FLOPs since the former assumes that sparsity provides no benefits during training.
2.2.1 LIMIT PERFORMANCE
In the previous section, we have focused only on when sparse models become optimal but not how much better they can be compared to dense models. this section, we study the following question: How much larger, and thus computationally more expensive, does a dense model need to be in order to match the loss of a smaller sparse model with very long training? Since we have found the scaling term in D to not interact with sparsity in Section 2. it suffices to compute the increase in N required to lower the loss by the same factor the increase in S via:
gain(S) = -
(as(( - S))e + cs
-1/bx
+ cs
The gains for our particular scaling coefficients are shown in Table D (eft). They are to be inter- preted in the following way: for example, 75% sparse ViT with N non-zeros will perform similar to a dense one with x 2.17.V parameters, when both are trained with the same amount of data. Crucially, this holds for any amount of data and thus also in the infinite limit when training is purely capacity bound. Hence, this expresses an equivalence between dense capacity and sparse capac- ity. We find that sparsity gains are very similar across vision and text domains, with the sweet-spot being around 75% sparsity at 2.15x gain. This is in alignment with the view of sparsity

---

Published as a conference paper at ICLR 2024
general increase in modeling power; we note that the gain is defined relatively to -emaii- improvements through increases in model size.
Family
0.500
0.750
0.875
Pattern
0.50
0.75
VIT/JFT
1.60x
2.17x
2.63
n:4
1.56x
1.62x
T5/C4
x.59x
2.16x
2.63
n:8
1.67
1.81x
Table 1: (Left) Equivalent dense size multiplier to match performance of a sparse model. (Right) Dense size multipliers for n:m sparsity on T5/C4.
250k steps
3 EXTENSIONS
2x10'
cense
2:4
val Hose towebrund
1:44
N:M sparsity. Complementing the unstructured sparsity ploration, we now also consider structured n:m sparsity, which can be accelerated on hardware (Pool & Yu [202]] Hubara jet al. 2021). Similar to how minor changes in the process (optimizer, model shape) generally only affect the multiplica- tive constants dense scaling laws (Kaplan et al.) (220)), we also expect minor changes in the sparsification process to only affect the sparsity term in (3). This can be exploited to laws based on significantly less runs: if the dense base scaling is known, one only has to fit as, by and cs (just 3 rather than parameters) to find the corresponding L(S, N, D). We utilize this in the context of n:m sparsity by fitting new laws for 2:4 and 1:4 as well 4:8 and 2:8 patterns, respectively, based only on subset of our full grid in Appendix [ Concretely, we execute all runs involving either the least amount of steps, or the smallest model.
2:8
4:8
107
10
#ESSTHOEE patuttedaes
Figure Loss vs. size plot for a subset of T5/C4 n.m sparsity data.
Figure O visualizes subset of the collected data, displaying a very similar form 2 which indicates that the general scaling law shape also holds for n:m sparsity. We also fit scaling laws (with Huber = 0.01 as 0.75 patterns will otherwise be treated as an outlier) and calculate sparsity gains in Section 2.2.1 see Table (Right). In general, 2:4 and 4:8 perform both very similar to 50% (see Table I and also Figure 6). although the n:m estimates are slightly more noisy due to less data used fitting the curves. Meanwhile, 1:4 brings almost no advantage and 2:8 only a slight improvement, which is contrary to our unstructured results. We suspect that the 75% patterns may simply be too stringent to significantly increase capacity beyond their 50% variants.
Decoder-only models. While we have focused on encoder-decoder models for language to cover a wide range of architectures, we now also validate our sparsity scaling laws in the context of standard decoder-only Transformers trained for auto-regressive language modeling. We follow the same setup as in our TS/C4 experiments, changing only model architecture and loss function. We again execute the experiment grid defined in Table 3 for 250K and 500K training steps, as well as a subset of the more expensive 1M step runs. Figure] demonstrates that all key sparse scaling properties still hold.
Quality of for US, N. sew wewiing Inw.
250K stops
SO0K stops
11ang
lowdound
DC, 00.00%
00.00%
87.50%
104
Knoon zere parometers
'0o
1000
non zore paramoters
16
3X1P
wndiindc 4x
Figure 7: Visualization of decoder-only experiment and corresponding scaling fit quality.
Pruning pretrained models. Lastly, we consider a practical scenario where set of existing very well-trained dense models should be made more efficient via pruning, using a small fraction of the compute spent for the initial pretraining. Our main interest here is to compare the efficiency of sparsifying from scratch and sparsifying from a pretrained checkpoint. For that purpose, we train ViT S/16, M/16 and B/16 models for full epochs on JFT (i.e., 16 billion images) and then start the same gradual sparsification procedure used before from these checkpoints, for 5.6% of the pretraining budget (as the model is already pretrained, we start to sparsify immediately rather than after 25% of training). Finally, we our scaling laws from Section 2.1 to determine the
7

---

Published as a conference paper at ICLR 2024
amount of training necessary to produce equivalent models of the same quality when starting from scratch. T shows how much more/less data is required to achieve equivalent performance for sparsifying from scratch, when excluding/iclludngg the pretraining cost, respectively.
0.500
0.750
0.875
Model
exc.
inc.
exc.
inc.
exc.
inc.
S/16
4
0.25
4.27x
0.23
2.45x
0.13
M/16
4.76x
0.25x
4.18x
0.22
2.57x
0.14x
B/16
4.35x
0.23x
4.00
0.21x
2.72x
0.14x
Table 2: Relative amount of data required for sparsifying from scratch to match the validation loss of pruning from pretrained model, when pretraining cost is excluded (exc.) and included (ine.).
If the model already exists and there thus no pretraining cost, then starting from such a checkpoint is > 4x more efficient then sparsifying from scratch for 0.5/0.75, and > 2x for 0.875 sparsity, respectively. The reason why the efficiency gains are decreasing with higher sparsity is most likely the increased divergence from the initial starting point. At the same time, when the pretraining cost is counted well, pruning throughout the whole training process appears to be 2 4x more efficient, relative to the x 5% pruning of pretraining budget. Overall, these results clearly demonstrate that, while the sparsification process benefits significantly from better trained initial model, it only does so to a limited extent.
4 RELATED WORK
Sparsity & pruning. Sparsity has long history (LeCun al.
1989 Hassibi et al. 1993) and
a large number of works have been published on this topic (Hoefler et al.
[002]) State-of-the-ar
methods range from simple gradual removal of the smallest weights (Zhu & Guptal 2017). to partial
or full sparse training (Mocanu et al. 2018 Jayakumar et al.) 2021 Peste et al. 2022 approximate
Hessian-based metrics (Singh & Alistarh 2020; Frantar et al. 2021 Kuznedelev et al.) 2023a) and
"soft" sparse optimization (Kusupati et al. 2020 |anh et al.) 2020). Sparsity can lead to substantial
practical speedups with specialized inference algorithms (Kurtz et al. 2020 Elsen et al. 2020 Yet,
most of those works focus on relatively simple tasks like ResNet500/mmgeeet or BERT/GL.UE
In contrast, much less is known when it comes to sparsifying Transformers trained on massive datasets: The Appendix of Gopher (Rae et al. (02I) conducts pruning experiments for generative language modelling task and finds that, when trained for the same amount of steps, sparse models can outperform dense ones, but leaves open whether this is also possible when accounting for the significantly increased compute spent for producing those sparse models, relative to equivalently sized dense ones. Similarly, [erebras (202)) prunes GPT-like model, also using significantly more data than its dense baseline. Recently, SparseGPT (Frantar & Alistarh| (203)) showed that is possible to impose weight-sparsity extremely large language models, even without retraining; yet, it remains unclear if this can also be done on smaller networks that are trained on more data.
Scaling laws. The key behind the success of Transformers is their exceptional scaling: increasing model size and/or data brings consistent performance improvements, even at huge scale. Further, this scaling behavior is predictable, following simple paw curves (Kaplan et al. (020). This
can be utilized to construct a family of compute optimal models (Hoffmann et al. 2022). More recently, scaling laws are being extended to more specialized applications, e.g.: optimizing model shapes ( (Alabdulmohsin et al. 2023), routing mechanisms (Clark et al., (022), repeating training data multiple times (Muennighoff al. 2023) and several downstream tasks (Caballero al. 2023) However, not much is known about the scaling of weight sparsity for such models.
Rosenfeld et al. (202 studies the relationship between width, depth and weight density for pruning pretrained ResNets trained primarily on the CIFAR dataset (Krizhevsky et al. 2009), which is nowa- consider modern Transformers trained on datasets many orders of magnitude larger and focus particularly on the data/compute dimension that is crucial in this context, but not relevant in the setting of Rosenfeld et al. (2021).
days considered very small. Contrarily,
Transformer efficiency. Making (large) Transformers more efficient is currently a highly active area of research. Probably the currently most popular and practical approach is quantization, that is reducing the numerical precision (Frantar et al. 2022) Dettmers & Zettlemoyer 20233 Xiao et al.) 2022). Further, there are also many works Mfxtureoo--Expert (MoE) models, which bound
8

---

Published as a conference paper at ICLR 2024
the overall computation cost per sample (Du et al. Riquelme et al. 202T). MoES are form of dynamic activation sparsity, which is very different from the static weight sparsity that we study; the former trades off increased memory for faster inference, whereas the latter reduces both inference and memory costs. In general, quantization, MoEs and weight sparsity are complementary and may be stacked for compound gains (Kurtic et al. 2022.
2022 Fedus al., 2022;
Artetxe et al. 2022
5 DISCUSSION
Limitations. While our study is based on extensive experiments across models and domains, also has limitations, which we discuss here and plan to address in future work. First, our sparsifica- tion recipe was optimized for robustness and scalability across a wide range of setups, rather than to fully maximize sparsity versus accuracy in particular setting. We believe that specific coefficient values can be improved with extensive getting-speciffee tuning and better sparsification techniques; however, the general nature of our scaling law will remain consistent.
We focus on settings where pruning is applied to pretraining tasks with massive amounts of data and compute. This is ideal in terms of usability, as down-stream (finetuning) applications directly benefit from the pruned model, but it also makes compression quite challenging. We think higher sparsity rates can be achieved if pruning is applied directly for specialized applications that only require a subset of the model's capabilities. Similarly, our study focuses on the infinite data setting, which essentially climinates overfitting, as only a single pass over the data is performed. Sparsity could be particularly effective when data is limited and thus multiple epochs are performed.
Our proposed sparse scaling law suggests that higher sparsity is always better (but with potentially quite quickly saturating improvements), which may not be true in extremes. For very high sparsity (e.g., 64x compression) we sometimes see slightly worse performance, presumably due to imper- fections in the pruning and optimization process. This phenomenon could potentially be modelled by a quadratic, but the present study treats as bottleneck that is not necessarily captured.
Finally, the main goal of this study is understanding core scaling relationships. Thus, we focused on the most fundamental cost metric, non-zero parameter count. However, in practice, sparsity acceleration can be complex: current software/hardware may not provide ideal speedups and models generally also contain operations (e.g., layer-norms, attention) which do not benefit from weight sparsity. Extending our scaling results to more target metrics is an interesting topic for future work.
Compatibility with other works. We will now briefly discuss how our scaling insights line up with existing sparsification results on similar models/datasets. First, the results in the Appendix of Rae et al. (2021), for a decoder-only text-generation model, are consistent with our scaling laws; the improvement through sparsity appears to be similar for each model size and their maximum size advantage of 2.5x observed at 0.9 sparsity is quite close to our limit gains in Section 2.2.1
In contrast, Cerebras 2022) report a significantly better gain of = 5x, but in quite different setting where the baseline is training (not inference) compute optimal and sparsification uses > 5x more data than the dense comparison point. This not inconsistent to our results: if we query our fitted T5 scaling law (see Section 2. with this setup, we predict 1 1.54 loss (dense 1B params, 20B tokens) vs. 1.48 loss (80% sparse & 200M non-zeros, 100B tokens), in favor of the sparse model.
Finally, SparseGPT (Frantar& & Alistarh 2023) notes that post-training pruning becomes easier as the model size increases. However, they do not perform any retraining, and observe this effect primarily relative to the respective unpruned base model, not in terms of improvements over the Pareto size- vs-loss frontier that we focus on. Hence, we believe that this is likely more related to the pretrained models' initial robustness to pertubations rather than the architecture's 's inherent sparsifiability.
Practical implications. Our scaling insights lead to a number of practical consequences: Sparsity seems to affect each model size in approximately the same way, while remaining mostly independent of the amount of training data used. This provides evidence that good pruning performance in less expensive settings should generalize to performance at scale, which will hopefully accelerate research on new sparsification recipes and algorithms. Additionally, we have shown that optimal sparsity levels continuously increase with longer training. Sparsity thus provides a means further improve model performance for a fixed final parameter cost. In particular, when training beyond Chinchilla optimality, where simple dense training starts to run into diminishing returns, sparsity can provide a clear alternative. Thus, our findings can be interpreted as providing practical motivation for further developing sparsity support.
9

---

Published as a conference paper at ICLR 2024
6
ETHICS STATEMENT
The primary goal of this research is to examine the interaction between parameter sparsity and the scaling behavior of foundation models, and such it does pose immediate ethical concerns. The techniques explored focus on compression and computational efficiency, and thus will lead to no harm to individuals or potential bias towards users. The recommendations provided for leveraging sparsity aim to advance technology and should serve to improve the computational burden caused by large models. As such, they should lead to positive societal outcomes.
REFERENCES
Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting VIT in shape: Scaling laws for compute-optimal model design. arXiv preprint arXiv:2305. 1035,, 2023.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, et al. Efficient large scale language mod- eling with mixtures of experts. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and Rahul Mazumder. Fast as CHITA: Neural network pruning with combinatorial optimization. In International Conference on Machine Learning (ICML), 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg. Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint 10X.::2008.07258. 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. In International Conference on Learning Representations (ICLR), 2023.
Cerebras.
Models attps://www. http://wwwwwwerr....e/.. net/bett/gooo cetboor..e....te//ooog///////// creating sparse gpt- 3 models with ceerattvvepruninng
Creating
Sparse
GPT-3
with Iterative Prun-
ing.
2022.
URL
Aidan Clark, Diego De Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. International Conference on Machine Learning (ICML), 2022.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski. Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. In International Conference on Machine Learning (ICML), 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Puttern Recognition (CVPR), 2009.
Tim Dettmers and Luke Zettlemoyer. The case for 4-bit precision: k-bit inference scaling laws. In International Conference on Machine Learning (ICML), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations (ICLR), 2021.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: Efficient scaling of language models with mixture-of-expers, In International Conference Machine Learning (ICML), 2022.
10

---

Published as a conference paper at ICLR 2024
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Murali Emani, Sam Foreman, Varuni Sastry, Zhen Xie, Siddhisanket Raskar, William Arnold, Ra- jeev Thakur, Venkatram Vishwanath, and Michael E Papka. A comprehensive performance study of large language models on novel ai accelerators. arXiv preprint arXiv:2310.04607 2023.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning (ICMI.), 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1): 5232-5270, 2022.
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning (ICML), 2023.
Elias Frantar, Eldar Kurtic, and Dan Alistarh. M-FAC: Efficient matrix-free free approximations of second-order information. In Conference Neural Information Processing Systems (NeurIPS), 2021.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. In International Conference on Machine Learning (ICML). 2019.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer.
A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.136600 2021.
Google. Big vision, 2023a.
URL http:s//ggthbb..ccooooogg/aaaaaa
vision
Google. T5x, 2023b. URL http://ss//g.cccb.o.oo/o////////
Babak Hassibi, David G Stork,, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, 1993.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint arXiiv:210..00 102.00554, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Itay Hubara, Brian Chmicl, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Eugenia Iofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse ImageNet models transfer? In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient waapeerrriitmmticcoon inference. In Conference on Computer Vision and Pattern Recog- nition (CVPR), 2018.
Siddhant M Jayakumar, Razvan Pascanu, Jack W Rae, Simon Osindero, and Erich Elsen. Top- KAST: Top-K always sparse training. arXiv preprint arXiv:2106.035
2021.
Tian Jin, Michael Carbin, Dan Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. Pruning's effect on generalization through the lens of training and regularization. In Conference on Neural Information Processing Systems (NeurIPS). 2022.

---

Published as a conference paper at ICLR 2024
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
Eldar Kurtic and Dan Alistarh. GMP*; Well-tuned global magnitude pruning can outperform most bert-pruning methods. arXiv preprint arXiv:2210.063884 2022.
Eldar Kurtic, Daniel Campos, Tuan Nguyen, Elias Frantar, Mark Kurtz, Benjamin Fineran, Michael Goin, and Dan Alistarh. The Optimal BERT Surgeon: Scalable and accurate second-order pruning for large language models. In Empirical Methods in Natural Language Processing (EMNLP), 2022.
Eldar Kurtic, Denis Kuznedelov, Elias Frantar, Michael Goin, and Dan Alistarh. Sparse finetuning for inference acceleration of large language models. arXiv preprint arXiv:2310.0622 2023.
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Bill Nell, Nir Shavit, and Dan Alistarh. Inducing and exploiting activation sparsity for fast inference on deep neural networks. In International Conference on Machine Learning (ICML), 2020.
Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateck Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference Machine Learning (ICML), 2020.
Denis Kuznedelev, Eldar Kurtic, Elias Frantar, and Dan Alistarh. CAP: correlation-aware pruning for highly-accurate sparse vision models, 2023a.
Denis Kuznedelev, Eldar Kurtic, Eugenia Iofinova, Elias Frantar, Alexandra Peste, and Dan Alis- tarh. Accurate neural network pruning requires rethinking sparse optimization. arXiv preprint vrXiv:2308.02060, 2023b.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Conference on Neural Information Processing Systems (NeurlPS), 1989.
Joo Hyung Lee, Wonpyo Park, Nicole Mitchell, Jonathan Pilault, Johan Obando-Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, et al. JaxPruner: concise library for sparsity research. arXiv preprini 2r04v:2304. 14082, 2023.
Tao Lin, Sebastian U Stich, Luis Barba, Daniil Dmitriev, and Martin Jaggi. Dynamic model pruning with feedback. In International Conference on Learning Representations (ICLR), 2020.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. In International Conference on Learning Representations (ICLR), 2018
Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yazdanbakhsh. STEP: Learning n: M structured sparsity masks from scratch with precondition. In International Conference on Machine Learning (ICML), 2023.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelcrating sparse deep neural networks. arXiv preprint /rXiv:2/04.08378, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 911: 2018.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo. Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint 1.. 2023.
Mahdi Nikdan, Tommaso Pegolotti, Eugenia Iofinova, Eldar Kurtic, and Dan Alistarh. SparseProp: efficient sparse backpropagation for faster training of neural networks. In International Confer- ence on Machine Learning (ICMI), 2023.
12

---

Published as a conference paper at ICLR 2024
Alexandra Peste, Eugenia Iofinova, Adrian Vladu, and Dan Alistarh. AC/DC: Altemating com- pressed/deecompressed training of deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Jeff Pool and Chong Yu. Channel permutations for N:M sparsity. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Jack Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2 /2.22//2.11 1446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140): 1-67, 2020a.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(1):5485-5511 2020b.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit. On the predictability of pruning across scales. In International Conference on Machine Learning (ICML), 2021.
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by fine-tuning, arXiv preprint XrXiv22005.07683 2020.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascanu, Peter Latham, and Yee Teh. Powerprop- agation: A sparsity inducing weight reparameterisation. Conference on Neural Information Processing Systems (NeurIPS), 2021.
Noam Shazeer and Mitchell Stem. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning (ICML), 2018.
Sidak Pal Singh and Dan Alistarh. WoodFisher: Efficient second-order approximation for neural network compression. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. SPDF: Sparse pre-training and dense fine-tuning for large language models. arXiv preprint arXiv:2303.1 10464, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /39771
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint (tt::2330700288 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Conference on Neural In- formation Processing Systems (NeurIPS), 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations (ICLR), 2020.
Haojun Xia. Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu. Yong Li. Wei Lin, and Shuaiwen Leon Song. Flash-LLM: enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. arXiv preprint arXiv:2309. 20X.:2209..0285... 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. 1..1::2111.10438, 2022.
arXiv preprint
13

---

Published as a conference paper at ICLR 2024
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Efficient and affordable post-training quantization for large-scale transformers, arXiv preprint 2xxiv:22060088000 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hong- sheng Li. Learning N:M fine-grained structured sparse neural networks from scratch. In Interna- tional Conference on Learning Representations (ICLR), 2021.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression, arXiv preprint arXiv: 17000177,, 2017.
14

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETUP
This section discusses our experimental choices and hyper-parameters, as well as technical details for sparsity-aware AdaFactor and iterative n.m pruning.
Models & datasets. We consider two standard deep learning applications: vision and language. For the former, we focus on Vision Transformers (Dosovitskiy et al.) (02I]) trained for multi-label image classification on the JFT-4B dataset (Dehghani et al. [2023), consisting of 4 billion images; for the latter, we consider encoder-decodee T5 models (Raffel et al. (020b) (improved 1.1 version (Google 2023bl) trained for masked- laaikkkd--anggg---mdoeeeeee 150+ billion tokens. These choices allow us to study the generality of our laws not just across vision and language but also for different kind of pretraining objectives and variations of Transformer architectures.
on C4 (Raffel al. 2020b), consisting of
Training syper-parameters For the most part, we reuse the optimized training preerpparaeetes of the original ViT-scaling (Zhai et al. 2022) and T5 paper (Raffel et al. 2020b), respectively. Our only notable change is that we do not factor the second moment of the respective AdaFactor-based (Shazeer & Stern 2018) optimizers (however, we still apply relative learning rate scaling and RMS clipping for T5); this is done since factorized moments for pruning and sparse optimization are not yet very well studied. Further, we train T5 models with batchsize 128 (similar to most ablation studies in the original paper (Raffel et al., 2020b)) in order to perform sufficiently many optimization steps also for experiments with lower total amounts of training data, which we found important to obtain stable sparse results through model and data scaling.
Model sizes. When it comes to selecting our particular model dimensions, two things must be taken into account: (a) we are particularly interested in the inference-optimal overtraining regime (Touvron al. 2023a) where models get close to their capacity limit, and (b) produce model
with N non-zeros and sparsity S, we actually need to train a model that is 1/(1 - S) times larger than dense model of size N. In combination with the fact that need to repeat the entire training sweep for multiple sparsity levels, this limits the size of models we can study while keeping compute requirements feasible. Specifically, b5-base). Then we generate models of appropriate sizes by scaling only the Transformer's ' hidden dimension and keeping all other shape parameters constant. This way matches between models and sparsities, facilitating direct comparisons (not all default family models are exactly 2 steps apart and a 50% sparse model would thus not always be directly comparable to the next smallest dense variant); we did not observe any notable performance decrease for dense models using this scaling strategy, at the sizes we study.
start from the base variants of both VIT and T5 (B/16 and
get quite precise size-
Sparsity configurations. We focus primarily on the most fundamental sparsity type. unstructured sparsity, but also perform some investigations for the more practical n.m pruning pattern (Zhou et al.) 2021 Pool & Yu, (02I)) where only n out of m consecutive weights are non-zero. We uniformly sparsify all linear layers in the Transformer backbone, which effectively avoids layer collapse (Wang) al. 2020), or other edge cases that may otherwise occur in our sweeps, and generally works decently well for Transformer models. On T5 models, we also sparsify the rather large embeddings to the amount necessary for parameter matching a smaller dense version.
Preliminary experiments indicated quickly diminishing returns for very sparse models, which, as discussed previously, in addition quite expensive to train. Thus, we focus three medium sparsities: 50%, 75%, 87.5%, corresponding to 2x, 4x and 8x compression rate, respectively. Our implementation is based on the recently proposed Jaxpruner library (Lee et al. 2023), which is easy to integrate into the offical ViT (Google 2023a) and T5 (Googge 2023b) codebases.
Pruning strategy. As we intend to execute substantial sweeps across model size, training data and sparsity, it is essential that our pruning method is highly robust and does not require hyper- parameter retuning for each run. A natural candidate is gradual magnitude pruning (GMP) (Zhu & Guptal 2017). which is well studied and known to be very reliable. At the same time, GMP is usually quite competitive with more complex techniques (Singh & Alistarh 2220 Kurtic & Alistarh 2022), , especially at the medium sparsity levels that we focus on. We also tested variation of GMP which incorporates diagonal second-order information (Kurtic et al.] (022)) but found it to perform
almost identically in our setting. Further, we tried AC/DC (Peste al. 2021) STE (Lin et al.
15

---

Published as a conference paper at ICLR 2024
2020) and RigL (Evci al. 2020) (which achieve strong results classic benchmarks) but saw similar or worse performance, while being more sensitive to hyper-parameterss as we scale (see also Appendix B). Thus, we ultimately decided to use GMP.
In terms of specific hyper-parameters, we prune using cubic schedule starting after 25% of training and ending at 75%, updating every 100 steps. Our sparsification interval was chosen so that pruning begins with a reasonably well trained model and ends with sufficient finetuning of the final sparse structure. However, we performed ablations for frequency/starrtend in Appendix [B and did not find the process be too sensitive those hyper-parameters (except for when the pruning interval is really short).
Sweep grids. Table lists the grid parameters that we sweep over. For ViTs, we consider 7 target models sizes in 2x increments each, while we use 4 targets sizes in increments of 4x for T5. Vision Transformers are trained for 4 different lengths, with the longest corresponding to s 1.8 billion trained for 3 different lengths up
images; language models sparsity targets is the same in both cases, corresponding to 2, and 8x compression rate. Overall, the VIT grid was designed be more extensive whereas the T5 setup was chosen to be more efficient.
65 billion tokens. The set of
Model family
ViT
T5
#Non-zer zero params
0.66M, 1.33M, .... 42.4M 55K, 110K, 220K, 440K 0.0, 0.5, 0.75, 0.875
1.3M, 5.3M. 85M
Training steps
250K, 500K, 1M 0.0, 0.5, 0.75, 0,875
Sparsities
Total #runs
48 Table 3: Grid definition for our main scaling sweeps.
112
A.1 TECHNICAL DETAILS
Sparsity-awand RMS. AdaFactor (Shazeer & Stern 2018) as employed by T5 (Raffel et al,, 2020b) defines the learning rate relatively, scaling it by the root-mean-square (RMS) of each weight tensor, respectively. We find that this does not interact well with high sparsity, as a tensor with many zeros tends to have lower RMS, resulting in smaller learning rate, which is especially problematic as high levels of sparsification require more recovery during the pruning process. To work around this, we always caleulate the RMS only over unpruned weights, which effectively alleviates this problem. We also apply the same technique to AdaFactor's 's RMS clipping threshold, but note that this is much less critical than the learning rate scaling.
Iterative n:m pruning. Sparsifying to the pattern is usually done by pruning in one-shot, followed by finetuning (Zhou et al., 202T), or directly training with dynamic pruming mask via through gradient estimation (Lu et al., 2023). We take a different approach in this work: we gradually remove the smallest weights while ensuring that at least n weights remain in each group of size m. This effectively generalizes the highly robust gradual pruning paradigm to the n:m setting. Not only does gradual n:m pruning unify our setups between unstructured and structured sparsity experiments, we also found it to work reliably across scales with the same hyper- hyper-parament parameters, a highly useful property for scaling studies. is shown in Algorithm [] the key is to temporarily set the largest items in each group to oc, thus ensuring that they are always picked by an unstructured topk selection.
simple and efficient implementation of this scheme
Algorithm Prune weights w sparsity s < n/m where each group of m weights contains at most zeros.
Ju + topk-nm( w|, n, m)
w copy of
w/ I. + DC
Tunstr t topk-unstr( |/|,
16

---

Published as a conference paper at ICLR 2024
B PRUNING ABLATIONS
We ablate gradual magnitude pruning hypee-parameters by sparsifying ViT-B/16 for 900M images to S 0.9375 sparsity. A high S was chosen in order to amplify differences between parameter settings; we vary the (relative) start and end point gradual pruming as well as the update frequency of the mask; the pruning schedule is always cubic. As can be seen in Table 4 most configurations perform very similar, except when the pruning period is too short overall. We ultimately pick the 25-75/100 setup to ensure that there is sufficient time for training a decent model before starting pruning, as well as for properly finetuning the final sparse version, which we think could be helpful for some points in our main scaling experiment grid.
start
end
freq
accuracy
0.250
0.750
100
45.28
0.250
0.750
50
45.08
0.250
0.750
200
45.13
0.125
0.875
100
45.33
0,475
0.625
100
44.65
0.125
0.625
100
45.13
0.375
0.875
100
45.04
Table Ablation study of gradual magnitude pruning hyper-parameters.
AC/DC. We also experimented with the AC/DC method (Peste et al. 2021), a sparse training ap- proach that was recently shown to yield very strong results on standard (non-foundation model) benchmarks Kuznedelev et al. (023b). We use sparse and dense cycle length of 20K steps (10K each phase) and apply AC/DC only during the same pruning period as our GMP setup to ensure the same pre- and post-sparsificatiio finetuning. On smaller TS models, AC/DC works well but yields very similar results to GMP. On larger models, however, AC/DC appears to require some hyper-parameter reconfiguration for higher sparsities. Since per-model hyper-parameter tuning is inconvenient for large scaling sweeps, while initial results also did not suggest clear improvements, stuck well established GMP. In general, differences between pruning methods tend to appear mostly at very high sparsities (Singh & Al- istarh] (2020). Nevertheless, think that more extensively investigating advanced sparsification approaches in the context massive pretraining datasets an interesting topic for future work.
note that even for classic benchmarks, major
0.500
0.750
0.875
#nnz
GMP
AC/DC
GMP
AC/DC
GMP
AC/DC
1.3M
17.11
17.11
15.64 5.87
15.96
14.73
14.59
42M
6.11
6.11
6.05
5.81
6.11
Table 5: Comparing validation perplexity of T5 models trained for 250K steps using GMP and AC/DC, respectively; we show perplexity as losses at these levels should be compared in log-scale.
C SCALING COFFFICIENTS
Table 6 lists the fitted coefficient values for the scaling results presented in the main paper; D is assumed to the number of images for ViT and the number tokens for T5. The fitting errors are also shown, where we note again that they correspond to the Huber-loss with 5 = 0.01 for VITNIFT and the Huber-loss of log L with 8 0.001 for T5/C4, following (Hoffmann et al. 2022).
For n:m sparsity, we only refit the sparsity coefficients ag, by and cg, preserving the other values from the corresponding unstructured results. While the fitting procedure may not be guaranteed to be convex, we find the process to converge to virtually the same values from different random starting points.
17

---

Published as a conference paper at ICLR 2024
Model VITJIFT
Sparse
as
by
cs
by
ap
bp
c
Error
unstr.
2.94++2
0.821
4. 4.68e+2
0.392
2.37e+8
0.890
4.517
4.93e-4
T5/C4
unstr. 1.68e+1
0.722
4.50e+1
0.245
6.90e+8
0.203
0.651
7.60e-6
T5/C4
n:m 8.64e+1
2.752
5.36e+2
2.1e-5
Table 6: Fitted coefficients of the scaling laws presented in the main paper.
D OPTIMAL SPARSITY DERIVATIONS
Sparse costs. We now discuss how to derive the sparse cost factor (Sm given by Equation in Section 2.2 for our particular pruning setup. For the first 25% of training, we use dense model that is 1 5) times larger, incurring cost 0.25/(1 (1 5), while for the last 25% we are training a model with sparsity S of the same cost the dense reference model, thus contributing 0.25 term. For the middle 50%, we prune in the cubic $ S. (1 t)² schedule (Zhu & Gupta 2017), where t E [0, 1]. The cost spent over this period is given by I minus (we care about density rather than sparsity) the integral over the full range of t, which is (1 0.75 5), further multiplied by 0.5 to cover the duration.
Contour lines. To determine the contours of Sopt C.,, i.e. the regions where a particular spar- sity S is optimal, we first query L(S, N, D) with Ds = D/Cmal where D = C/6NN is the amount of data corresponding to dense training FLOPs C' relative to which Cmul defined, giving:
(as(1 - S) + cs)
1
bw +
(<mm(())
aD hD
+c.
(8)
N
D
Next, we optimize for by differentiating and setting the result equal to O:
(1-
bD (Cmal (S)
aD
bn-1
(cmm(())
QD
0.
(9)
Ds
Ds
Finally, the form presented in the main paper is constructed by few algehraic simplifications:
bn
Crual (S)
aD bn
Cmaa (S)
an Dg
= a,bb (1 - -yps-1 .N..NN- -N
(10)
Ds
Smm (S)
by aD
bD
(Cmun (S)
as,b((1 Syss-1 .N bNb.
(11)
Cmul (S)
Ds
Closed form solution. The closed form solution for Sopt(M N,, C) assuming dense costs Caul (S) 1/(1- S) can be found by solving the above contour equation for St
1 bb+1
bp
(1-s)
6D
=(1 (1- sy0s-1 agbs- -
(12)
Ds
1
bp+
asbs
-N -
bx
CD Ds
bp
(13)
S
S
bn
(bD + bs) log
1
= llo asbs
NN
AD Ds
-bp1
(14)
1- S
bD
1 1 S exp log
1
rasbs
N- aD
-bp1
/(bo + bs)}
(15)
bD
Dg
ragbs
OD -bo
S = 1 - e
- log
-N
/(b0 + bbs)
(16)
bD
Ds
Further algebraic manipulations yield:
= 1 cxp log
bD
b aD
bp
/(bo + bs)
(17)
Labb
Ds
S = 1 exp
(log agbs
bn
+ bxlog.N + bolog aD
/(bD bs)
(18)
Dg
S= = 1 exp
(10g
bD
+ bylog /bb + bs)
bxyogg)
0D
bn/ bp/(bb+be)
(19)
agbs
Ds
Substituting Ds by its expression in C and guaranteeing that the result is always non-negative (since sparsity does not exist) brings the formula given in Section 2.2
18

---

Published as a conference paper at ICLR 2024
E IMPACT OF SPARSITY ON DOWNSTREAM TASKS
As discussed in the main text, the focus of this study, similar to most other works on scaling laws, lies on modeling the pretraining task validation loss. For dense models it is known, that this loss strongly correlated with performance on downstream (e-g. few-shot) applications (Hoffmann al.) 2022 This effect has also been observed for sparse models, for example the context transfer learning for ImageNet models (Iofinova et al. 2022). We now provide additional evidence, for our particular sparse models by evaluating them on suite common few-shot tasks.
Concretely, we consider all dense and sparse models from our main sweep and plot few-shot accu- racy vs. pretraining loss; the results are shown Figure 8 As can be seen, sparse models exhibit very similar behavior to dense ones, both terms of accuracy achieved for fixed validation loss well in terms of overall noise level that is typical for each task. This confirms that sparse models with better pretraining validation loss, like dense models, indeed also perform, on average, better on few-shot tasks.
imagenet_ 5shot
imagenet_ 25shot
cifar100_10shor _0shho
0.7
00.00%
00.00%
00.00%
50.00%
0.7
50.00%
50.00%
Few-shol accuracy
0.6
75.00%
87.50%
Few shot acouracy
0.6
87. 97.50%
0.6
87.50%
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.2
0.1
Pretraining loss
Pretraining less
Pretraining loss
col. hist 10shot
uc. merredd 10shot
dtd 10shot
0.95
00.00%
00.00%
0.70
00.00%
0.80
50.00%
0.90
50.00%
0.65
50,00%
Fewssol accuracy
0.78
87.50%
75.00%
Few-shot accuracy
0.85
87.50%
25.00%
0.60
75.00%
0.76
0.55
0.80
shot
0.50
0.72
0.45
0.70
0.70
0.40
0.65
8
9
7
8
Pretraining
Pretraining
Protraining loss
Figure 8: Few-shot accuracy vs. pretraining loss for dense and sparse VIT/JFT models.
PRACTICAL SPARSITY ACCELERATION
Although we focus primarily on developing general understanding of sparsity scaling, rather than on accelerating sparse model in practice, there many works that study the latter. We now sum- marize some of them.
Network weights that are exactly zero must not be stored and can be safely skipped during computa- tion. This means that, principle, sparse models require both less memory well less compute to run than their dense counterparts. For CPU inference, this compute reduction can be effectively translated into end-to-end speedups using sparsity-awaa matrix algorithms (Elsen al. 2020 Kurtz et al., 2020). Custom hardware can also achieve strong throughput improvements for sparse models: for example, Thangarasa et al. (202)) report almost 4x speedup on a matrix multiplication with 75% sparse weights. On modern GPUs, effectively utilizing fully unstructured sparsity is challenging due irregular computation patterns. However, newer generations of NVIDIA GPUs include dedicated
hardware support for semi-structured n:m sparsity (Pool & Yu 2021), yielding to 2x speedup over fully dense matrix multiplications.
19

---

Published as a conference paper at ICLR 2024
While effectively utilizing weight sparsity during training is still active of research, there are al- ready some promising results. [Emani et al. (2023) report substantial end-to-end training speedups for training large, unstructured sparse, Transformer models custom hardware. Meanwhile, Nikdan al. 2023) develop sparsity-aware backpropagation kernels for efficient CPU training.
Finally, we note very recent line of work which uses unstructured sparsity (Xia et al., 2023 Kurtic al., 2023) primarily for reducing model size, in order fit massive Transtormer models onto smaller accelerators. Further, for memory bound generative inference, this brings practical as well, by reducing transfer costs between GPU main memory and registers due.
20


Reviews:
[{"review_id": "lx2iumBtIX", "reviewer": "Reviewer_pZ1g", "summary": "This paper proposes a formulation of scaling laws for sparsely-connected foundation models. Beginning with the scaling laws proposed by previous works, which are related to the parameter count N and the amount of training data D, the authors add parameter sparsity S into consideration and construct a formulation based on observations and actual data points. After obtaining the formulation, the authors investigate the optimal sparsity and limit performance for training sparse models. Furthermore, the authors consider two practical applications, including N:M sparsity and pruning, to validate the scaling laws.", "questions": "Please refer to the weakneses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper investigates a topic highly relevant to current trends in large-scale models, namely the scaling laws of sparse large models. The authors present the formulation of scaling laws along with their derivation process and validate them on T5 and ViT models.\n2. The authors also validate several applications related to scaling laws, providing reasonable experimental designs and detailed experimental results.\n3. The writing throughout the article is quite commendable.", "weaknesses": "1. This paper posits that sparsity can reduce training costs, thus presenting research on optimal sparsity. However, in reality, the training costs of current sparse training methods are often the same as those without sparsity. Under these circumstances, is it meaningful to study optimal sparsity?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oJwJedal8m", "reviewer": "Reviewer_jJsH", "summary": "The key message of this paper is a scaling law describing the relationship between sparsity (weight sparsity rate), the number of nonzero parameters, and the amount of training data proposed. The most intuitive effect on our training of sparse models is that for a fixed number of non-zero parameters, it is found that optimal sparsity increases with the amount of data used for training. And the benefit of long training seems more obvious compared with dense models. The experiments are using Vision Transformer and T5 language models and conducted on large-scale C4 and JFT4B datasets.", "questions": "Please see the weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This study is the first paper to use large-scale experiments and theoretical analysis to explore in detail the effects of sparsity on neural network training and efficiency. The authors propose new scaling laws describing the relationship between sparsity, the number of non-zero parameters, and the amount of training data, which is important.\nOn the one hand, the findings of this paper can help us configure the training parameters more scientifically and thus improve the efficiency and performance of the model in large-scale training. It also shows that as the amount of training data increases, the model sparsity required to achieve optimal performance also rises. In this case, appropriately increasing the model sparsity and optimizing the model structure may be a more effective solution.\nMeanwhile, according to this finding, the sparsity of the model can be dynamically adjusted according to the change in data volume during the training process to balance the model performance and computational cost better.", "weaknesses": "Overall, I think this is a good paper; my only concern lies in the scale of the data and modeling. Compared to Chinchilla's law, the author's data and model sizes are significantly smaller. Also, since the authors mentioned the optimal sparsity setting as an empirical design or product of this claim, could you elaborate on what specific improvements in downstream task or operational efficiency the sparsity of our actual model would bring us under the authors' theoretical guarantee?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6jIHfoWUd", "reviewer": "Reviewer_1amU", "summary": "This paper studied the scaling laws for sparse foundation models. Specifically, they built a scaling law for how sparse models would perform as a function of sparsity, model size and pre-training data size. The authors evaluated on ViT and T5 architectures and showed that the constructed scaling laws can be used to easily predict the optimal sparsity without manually re-training the models.", "questions": "1. I am curious as to whether lottery ticket hypothesis would hold as to foundation models, as it is the most impactful pruning paper in recent years. Since this work is closely related, i am wondering if the authors have explored the LTH setting for sparse training. If not, it would be good to add a discussion on LTH [2].   \n2. Can the proposed scaling laws be useful in the extreme sparsity regime, e.g. 95% or even 99%? Or is it applicable only for the sparsity level within a range? \n\n[2] The Lottery ticket hypothesis: Finding Sparse, Trainable Neural Networks. ICLR 2019.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental results are comprehensive. I appreciate that the authors include experiments on hardware friendly structured sparsity.  \n2. Deriving scaling laws for sparse foundation models is important. As foundation models require huge compute to train, understanding their scaling behavior and relationship to sparsity is crucial for predicting model performance.  \n3. The empirical results show that ViT and T5 show strong scaling curves for sparsity and their performances can be well predicted from the constructed scaling law.", "weaknesses": "1. My main concern lies in the contribution of section 2 on fair evaluation, which argued that a sparse network should be compared with a dense model with the same number of parameters and compute. A very similar argument is made in [1] (See “training budget” in section 3), which argued that the compute budget for comparing sparse and dense models should be the same. It would be good to clarify the differences to [1] in the paper.\n2. The evaluation on the encoder-decoder architecture T5 seems limited. The paper do not evaluate the more popular autoregressive decoder-only architecture but the original scaling laws is built on such autoregressive language models.\n\n[1] Rethinking the value of network pruning. ICLR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ryUxb4boMZ", "reviewer": "Reviewer_Tdrh", "summary": "This work uses scaling law to describe the relationship between weight sparsity, number of non-zero parameters and amount of training data.", "questions": "1, What are the 7 free parameters fitted and how are these parameters changes for different dataset and different hyperparameters $N, D, S$?\n\n2, Procedure to identify $S_{opt}$ in 2.2 Optimal Sparsity is confusing. Could you explain this procedure with more details?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1, In-depth investigation of the relationship between validation loss and sparsity, number of non-zero parameters and training data. Useful for understanding sparsification better.\n\n2, Derived optimal sparsity for a given inference size and training budget.", "weaknesses": "1, Experiments are only tested on two examples. Though due to limited resources, more examples may be challenging, two examples may not be very convincing.\n\n2, The scaling law has too many degrees of freedom (7 free parameters) that could be hard to generalize to new dataset without sufficient information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)