Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Authors: Nolan Simran Dey
Shane Bergsma
Joel Hestness
Venue: NeurIPS 2024 poster (2024)
Keywords: sparsity
large language models
scaling laws
Primary area: optimization_for_deep_networks
TL;DR: We introduce the sparse maximal update parameterization (S$\textmu$Par) which ensures optimal HPs remain the same for any width or sparsity level. This dramatically reduces HP tuning costs, allowing S$\textmu$Par to achieve superior losses.

Abstract: Several challenges make it difficult for sparse neural networks to compete with dense models.  First, setting a large fraction of weights to zero impairs forward and gradient signal propagation. Second, sparse studies often need to test multiple sparsity levels, while also introducing new hyperparameters (HPs), leading to prohibitive tuning costs. Indeed, the standard practice is to re-use the learning HPs originally crafted for dense models.  Unfortunately, we show sparse and
dense networks do not share the same optimal HPs. Without stable dynamics and effective training recipes, it is costly to test sparsity at scale, which is key to surpassing dense networks and making the business case for sparsity acceleration in hardware.

A holistic approach is needed to tackle these challenges and we propose S$\textmu$Par as one such approach. For random unstructured static sparsity, S$\textmu$Par ensures activations, gradients, and weight updates all scale independently of sparsity level. Further, by reparameterizing the HPs, S$\textmu$Par enables the same HP values to be optimal as we vary both sparsity level and model width. HPs can be tuned on small dense networks and transferred to large sparse models, greatly reducing tuning costs. On large-scale language modeling, S$\textmu$Par shows increasing improvements over standard parameterization as sparsity increases, leading up to 11.9\% relative loss improvement at 99.2\% sparsity. A minimal implementation of S$\textmu$Par is available at https://github.com/EleutherAI/nanoGPT-mup/tree/supar.

Sparse maximal update parameterization: A holistic approach to sparse training dynamics
Nolan Dey
Shane Bergsma
Joel Hestness
Cerebras Systems jnolan..jellBeerebbree net
Abstract
Several challenges make difficult for sparse neural networks to compete with dense models. First, setting large fraction of weights to zero impairs forward and gradient signal propagation. Second, sparse studies often need to test multiple sparsity levels, while also introducing new hyperparameters (HPs), leading prohibitive tuning costs. Indeed, the standard practice is to re-use the learning HPs originally crafted for dense models. Unfortunately, networks do not share the same optimal HPs. Without stable dynamics and effective training recipes, it is costly to test sparsity at scale, which is key to surpassing dense networks and making the business case for sparsity acceleration in hardware.
show sparse and dense
A holistic approach is needed to tackle these challenges and we propose sparse maximal update parameterization (SyPar) as one such approach. For random unstructured static sparsity, SuPar ensures activations, gradients, and weight up- dates all scale independently of sparsity level. Further, by reparameterizing the HPs, SuPar enables the same HP values to be optimal as vary both spar- sity level and model width. HPs can be tuned on small dense networks and transferred to large sparse models, greatly reducing tuning costs. On large- scale language modeling, SuPar shows increasing improvements over standard parameterization as sparsity increases, leading up to 11.9% relative loss improve- ment at 99.2% sparsity. A minimal implementation of SuPar is available at ssuar.
https: thub. com httm///lehherAAaanoP//
1 Intro
Sparsity has emerged as a key technique to mitigate the increasing computational costs of training and inference in deep neural networks. This work focuses weight sparsity, whereby significant fraction model weights are kept at zero. It has long been known that dense neural networks can be heavily pruned after training [30]. With the goal of reducing costs during training, recent work has explored static weight sparsity from initialization. In this work we focus on random unstructured static sparsity, which has re-emerged
surprisingly effective strategy [33, [33, This type of sparsity can be accelerated by CPUs, Cerebras, Graphcore, and SambaNova. Furthermore, GPUs and TPUs support 2:4 block structured sparsity which is quite similar to 50% unstructured sparsity.
Unfortunately, several challenges have hindered progress weight-sparse neural networks. First, sparsity impairs signal propagation during training [31, 11, 1]. Second, with today's 's techniques, sparse training is costly. Sparse techniques typically introduce extra hyperparameters (HPs), e.g.. number of pruning iterations initialization [60, 7, 56]. and it is common to train models across different sparsity levels. Since tuning should be performed at each level and the search space grows exponentially with the number of HPs, the tuning costs essentially "defeat the purpose" of sparsity, iee., to reduce computation [60]. Finally, today there only nascent ecosystem of hardware acceleration for unstructured sparsity, so most researchers get little sparsity benefit when tuning.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Standardd practice
SuFar (Our work!
Standard practice
SuPar approach
Sparsty
Prohbitve tuning
Cheep tuning
3.9
0.25
Trai Loss
3.5
2.875
Inconclusive results
Robust to widthidensity
0.227
96825
0.343355
Unclear path to scale
Ready for next lovel scale
3.6-
optimum shifts
optimum stable
Derse still dominant
Ready surpess
2-42
2-
2-
2-
2-4
2-
2-4
Leamiing Rate
Leaning Rate
Figure 2: SuPar enables sparse training at scale, helping to surpass dense and motivate sparsity in hardware.
Figure 1: SuPar (Our work) allows stable optimum HPs for any sparsity level, unlike standard practice.
These costs have led to the standard practice of simply re-using HPs that were previously optimized for the baseline dense models (Section 2). One might hope that sparse models thrive with the same learning rates and other HPs as their dense counterparts. Unfortunately, they do not: optimal HPs systematically vary with sparsity level (Figure 1, left). With impaired training dynamics, prohibitive tuning cost, and lacking the established training recipes enjoyed by dense models, is often inefficient to train sparse networks at scale (Figure 2).
To remedy this situation, propose sparse maximal update parameterization (SuPar, pronounced "soo-pahr"), a novel, holistic approach to stabilize sparse training dynamics. SuPar fulfills the Feature Learning Desiderata (Section 3) by parameterizing weight initialization and learning rates with respect to change in width and sparsity level. As a generalization of maximal update parameteri- zation (uP) [64, 63], SyPar enjoys well-controlled activation, gradient, and weight update scales in expectation, avoiding exploding or vanishing signal when changing both sparsity and model width.
By reparameterizing HPs in this way, SyPar enables the same HP values to be optimal as sparsity varies (Figure 1, right). We therefore enjoy pTrranffe: we can tune small proxy models and transfer optimal HPs directly to mod- els scale. In fact, we discovered our μP HPs, tuned for dense models in prior work (and equivalent to SuPar with sparsity=0%) correspond to the optimal learning rate and initial weight variance for all sparse models tuned in this paper! As sparsity increases, our formulation shows the standard parameterization (SP) and μP suffer from vanishing signal, further clarifying prior observations of gradient flow issues in sparse networks. The improve- ments enabled by SuPar set the frreto-frontier best loss across sparsity levels. Figure 3 previews this improvement for large language models trained from compute-optimal configurations [23]. Here, SuPar benefits grow with in- creasing sparsity, to 11.9% better loss than SP and 1.9% better loss than yP at 99.2% random unstructured sparsity. See Section 4.3 for details on this experiment.
SP
3.6
UP
SyPar
Validation Loss
3.4
3.0
2.8
2'
2
Density
Figure 3: For LLMs SuPar forms the Pareto frontier loss across sparsity levels, with no HP tuning required.
2 Related work
Sparse training landscape Sparse training can be divided into static sparsity, where the connec- tivity is fixed (our focus) and dynamic sparsity. where the sparsity mask can evolve [22]. We use unstructured sparsity, though our approach generalizes to structured approaches where particular sparsity pattern increases efficiency on specific hardware [67, 26, 38, 14, 29, 1]. Unstructured connec- tivity may based on both random pruning [40, 18, 57, 33, 58) and various
criteria [32, 60, 61, 56, 7]. Liu et al. [33] found that models scale, the relative performance of randomly pruned networks grow. Furthermore, Frantar al. [15) found the optimal level of sparsity increases with the amount of training data. Together, these findings suggest that neural networks

---

continue to get wider and deeper. and trained on more and more data, very sparse sandomly-pruncd networks may emerge as an attractive option.
Improving sparse training dynamics Many prior works identify various sparse training dynamics
issues. In particular, prior works note sparsity impacts weight initialization [35, 31, 49, 11], activation
variance [29], gradient flow [61, 37, 57, 11, II, and step sizes during weight updates [15]. These prior works each only address subset these issues in targeted ways, often showing benefits to sparse model training loss. We advocate for holistic approach, and discuss the relationship between these prior works and our approach in Section 5 after describing and evaluating SuPar.
Sparse sensitivity HPs Due the costs of training with fixed weight sparsity, re-using dense HPss is standard practice. Such re-use is typically indicated in appendices or supplemental materials, e.g., [40, 32, 35, 31, 16. 60, 61, 56, 13, 7, 18, 57, 33, 58]. Also, dynamic sparsity approaches often compare to fixed sparsity; these baselines are likewise reported to re-use the dense HPs [2, 41, 10, 34, 11, 59 However, some prior work has suggested such training is sensitive to HPs, e-g., learning rates [35, 57], learning rate schedules [16], or training length [28]. although systematic tuning was not performed. For dynamic sparse training (DST), is also conventional to re-use dense HPs, whether in dense-to-sparse [37, 15] sparse-to-spars (evolving mask) training [2. 8, 34, 11, 59]. As with fixed sparsity, work here has also suggested sensitivity to HPs, e.g., to dropout and label smoothing [16]. DST may also benefit from extra training steps [10] smaller batch sizes [34], although in DST this may mainly be due to a greater number of opportunities for connectivity exploration [34].
3 Sparse maximal update parameterization (SuPar)
We now provide background, motivation, and derivation for SuPar, first introducing notation (Section 3.1) and then defining Feature Learning Desiderata (Section 3.2) with brief overview of μP (Section 3.3). Finally we motivate SyPar and provide an overview of the parameterization (Section 3.4).
3.1 Notation
The operations for single sparse training step illustrated in Figure 4. The definition and dimensions are: batch size B, learning rate loss function L, forward pass function F, input dimension dua input activations X E Exd=, imput activation gradient 8x Vx E xdd. output dimension dout, output activations Y E "xxdu, output activation gradient aY Vy C E R Axda weights W E Idd Xdou, initialization variance ow for weights W, weight update AW E xda and AY Bxd is the effect of the weight update on output activations: AY = X O M). Unless otherwise specified, M {0, 10 x an unstructured random static mask with sparsity and density P =
ac
ac
When changing model scale or sparsity, we refer to width multiplier deas and density multiplier m, Pess
de
d. base
don. base
Forward
X
W M
AW M
Optimizer
VYL
Weight Update
VxC
Backward
Figure 4: The three operations associated with training layer with weights that perform the function F: Forward activation calculation, backward gradient propagation, and the weight update.
If we apply sparsity to linear layer (i.c., F fully-connected layer), our aim is to control:
1. Forward pass: Y = FXX W о M) X(W @ M).

---

2. Backward pass: VxC = (VyL)- (W O OM).
3. Effect of weight update AW on Y: AY - X(AW & M)'.
3.2 Feature learning: Defining the goal of μP and SuPar
Prior works [64, 63, 65] introduce the Feature Learning Desiderata (FLD) ensure stable training dynamics as width is varied. Building on prior works, we include gradients V in the desiderata.
Feature Learning Desiderata (FLD); For layer and token i, we desire that |Y ( |V
@(v Y 2 = O (o) Vi, Vl.
Recall that if all the entries of vector RR some constant c, then ||v]]2 = O (()) with respect to width n. Therefore we can satisfy the FLD by ensuring the typical element size of Y, Vx C, and AY is (11) with respect to some variable(s) we would like scale. Variables scale include width [64, 63, 65], depth [66, 4], and sparsity (this work). The FLD prescribes a holistic signal propagation approach of controlling each of the three operations in a training step, subset".
3.3 Maximal update parameterization (μP)
Here we provide a brief overview of maximal update parameterization (μP) [64, 63, 65]. With the standard parameterization (SP), Yang and Hu [64] show the scale of activations throughout training increases model width increases, motivating the development of μP. jP [64, 63) is defined as the unique parameterization that satisfies the FLD by ensuring the rypical element size of Y. VxL, and AY is (1)) with respect to change in width my. The FLD can also be satisfied by controlling the spectral norm of weights [65]. μP enables μ μTrannffe:: the optimum learning rate, initialization weight variance, scalar multipliers, and learning rate schedule all remain consistent width is increased for μP models [63]. μTrransfer can leveraged to take a tune small, train large approach where hyperparameters are extensively tuned for small model then transferred, enabling reduced tuning budgets and superior tuning for large models compared to standard practice.
3.4 Sparse maximal update parameterization (SuPar)
Yang et al. [63] show activation magnitudes explode with increasing model width. In Figure 5 show sparsity has the opposite effect: increasing sparsity causes shrinking activation magnitudes.
SP
SuPar
10a
Activation
increasing sparsity
Mean Attention
107
30-1
Moai Absoluta Adtivviittttttiion Adtivaiion
304
109
302
20
2.22 Deniity
2.2
Dansity
Dansity
Figure 5: Mean absolute value activations for attention and feed forward blocks after training step t (10 seeds). In SP and uP models, decreasing density causes activations to vanish (note axes log-scale). In SuPar models, density has little effect on activation scales and there is no vanishing.
After a weight update AWis applied, new output activations can be written as Y +AY X(W & M) + X(AW & M). Our goal control AY.
For example, initialization methods alone only control |Y|| and V| the first time step.

---

Finding 1:
Increasing sparsity causes vanishing activations and gradients with both SP and pP.
SuPar is defined as the unique parameterization that satisfies the FLD by ensuring the typical element size of Y. VxC, and AY is (1) with respect to change in width ma and change in density mp. SuPar enables stable activation scales across sparsity levels (Figure 5, right). In this section, walk through the changes required to control each of the three operations in a sparse training step, providing overview of the SuPar derivation. We focus on the AdamW [36] optimizer used in experiments. For a more detailed derivation, including both SGD and Adam, see Appendix D.
Forward pass at initialization To ensure the typical element size of Y is (1) with respect to change in width m&, and change in density ma, we can control the mean and variance of Yij. Since initialization E [[] EE E O, and W Y, the mean is controlled. The variance of Y ij can be written as:
Var(Y = M.d base pPasse i (Var(X (X) + E[X|3)
To ensure Var (Y:) scales independent of d and mp we choose ow
dW "W.be m.a,
Backward gradient pass at initialization To ensure the rypical element size of V is (1) with respect to change in width md... and change in density mp, we can control the mean and variance of VxC. Since at initialization E [[] = 0, E[Vx = and the mean is controlled The variance of Uy can be written
Var V =
Mila Ma 2 Var( V
(2)
W dare maaa m.
To ensure (iy)V scales independent of mi and mps we choose aw Mdd = -,d,, allowing the same ow to control both forward and backward scales.
Typically
Effect of Adam weight update AW on Y To ensure the typical element size of AY @(1) with respect to change in width den and change in density mp- By the law of large numbers, the expected size of each element can be written as:
]. as (dmmP) - 00
ΣS 7
Sigj
AAY [g] - mpMid din.bose PPasee E
Xik
(VEi We Ed
X V 1
To ensure AY and | P are scale invariant to Mdd Mpp we choose 7)
m.. mp'
Implementation summary Table 1 summarizes the differences between SP, pP, and SuPar. Since we only sparsify hidden weights, SyPar matches yP for input, output, bias, layer-norm, and attention logits. Also note width multipliers my and density multipliers mp are usually the same for all layers, allowing simplified notation. This correction is equivalent to μP [63] when p = 1 and mp = 1. The correction to hidden weight initialization we derive is similar to the sparsity-aware initialization in prior work [35, 49, 11]. SyPar should also easily extend to 2:4 sparsity patterns because, in expectation, the rows and columns of M' should have equal density. A minimal implementation of SuPar is available at https: / github. wtt.//oeo/eeeeeeeeeeeee EwwtthhrrA//aaooooP wwweereeeeeeeeeeeeeeeeeeeeeeeeee supar.
4 SuPar Training Results
Here, we present empirical results showing the effectiveness of SuPar over SP and μP when training sparse models. When using SP or uP, optimal HPs drift as we change the sparsity level, possibly leading to inconclusive or even reversed findings. SuPar has stable optimal HPs across both model width and sparsity level, and we show it improves over SP and μP across different scaling approaches. Taken together, we see that SuPar sets the Pareto frontier best loss across all sparsities and widths,
AAthhough the gradients V will have some correlation with weights W even at initialization, we assume for simplicity that they fully independent. Future work could investigate this assumption more deeply.
5

---

Table Summary SP, JuP and SuParr implementations.
Parameterization
SP
uP
SuPar
Embedding Init. Var.
base
basc
base
Embedding LR
Thase
Thase
Thase
Embedding Fwd.
X
W emb
Olimput
X
W emb
Orinut
W
Hidden Init. Var.
base
base! ma
base!
(mamp)
Hidden LR (Adam)
Towe
Truse
Mousee
(mamp
Unembedding Fwd.
L W
cmb
Goupput X I.
W
emb
/
Croutgut
XWW
omb/
mid
Attention logits
Q J'K/v Chead
Q
K
dhead
K //deaad
including when we scale to a large dense model with width equal to GPT-3 XI. [5]. Optimal dense uP HPs-when adjusted using SuPar- are also optimal HPs for all sparse models that test here.
All tests in this section use GPT-like transformer language models [48, 9], trained on the SlimPajama dataset [54] with a 2048 token context length. We apply random unstructured static sparsity to all projection weights attention and feedforward blocks while keeping embedding, layer normalization, and bias parameters dense. We refer the reader to Appendix E for full methodology of all experiments.
4.1 Sparse hyperparameter transfer
We first show sparsifying dense model using either or μP leads to significant drift in optimal HPs as the sparsity level changes. Figure 6 shows train loss for SP, pP, and SuPar models when trained with varying sparsity levels and sweeping across different peak learning rates. For the SP configuration, sparsity increases, the optimal learning rate increases in somewhat unpredictable way. uPP experiences similar shift in optimal learning rate, though shifts are even more abrupt. For SyPar, the optimal learning rate is consistently near 2 across all sparsity levels.
AP
5uPar
4.0
3.9
5-000
0.5
3.8
0.75
Train
0.9975
3.7
0.00875
= 0.984375
Optimum um
3.6
Derse Opt
pTrondforre
3.5
2 Learning Rate
Learning
Learring Rate
Figure SuPar ensures stable optimal learning rate for any sparsity s, unlike SP and pP (3 seeds).
also sweep base weight initialization values and find even more chaotic behaviors for SP and μP with different sparsity levels (Figure 7, left and center, respectively)+ μP even shows discontinuities in optimal initialization values at different sparsity levels. We attribute this discontinuity to widely varying expected activation scales between embedding and transformer decoder layers, where embed- ding activation scales will tend to dominate for high sparsity levels. SuPar shows consistent optimal initialization (right plot). Figures 6 and demonstrate second finding.
Finding 2: With SP and μP, dense and sparse networks do not share the same optimal HPs.
HP transfer tests, showing loss for each parameterization across all sparsities. Even when selecting the best learning rate at each sparsity level for SP and μP, SuPar (largely) forms the Pareto frontier with an average gap of 0.8% better than SP and 2. 1% better than pP.
Figure 8 summarizes
'These results are taken from point early in training as models with widely varying initialization tend to become unstable later training.

---

0.5
K- 0.75
4.0
0.875
0.5
0.5375
8.0
Optinum
Dense optimu pTrenslerred
7.0
6.5
at Inttaaliztioon
at Initialicaaionn
at inttalliztionn
Figure 7: Across sparsity 8, SP and μP show unstable optimal initialization. SuPar is stable (3 seeds).
AP
SyPar
4.5
5w0.5, d-1088
433
875, d-2176
4.2
Optimum
4.1
Denis Optimum
Train
ulranstemed
3.8
37
2:30
Learnng
Learning
Learning
Figure 9: SuPar ensures stable optimal learning rate in Iso-Parameter sparse + wide scaling seeds).
4.2
Studying SuPar indicates how some sparse scaling techniques appear work
So far, we see SuPar can transfer optimal HPs across spar- sity levels, but have also designed it transfer HPs across different model widths (hidden sizes), similar to uP. Here, further demonstrate that SuPar transfers op-
3.85
SyPar
lows with
timal HPs across width. More generally, sparse scaling that keeps fixed number of non-zero weights per neuron allows SP and uP to also transfer HPs.
3.65
Figure shows learning rate transfer tests when changing both the model's 's hidden size, dmodel, and sparsity level in common scaling approach called So-Paarameter scaling [18, 10, 59]. Iso-Parameter scaling keeps the model's num- ber of non-zero parameters approximately the same, width and sparsity are varied Here, we see the common result that SP models starting from dense HPs do tend significantly improve as we increase width and sparsity. Note, though, the optimal learning rate for each sparsity level still shifts. When we correct dense HPs using μP SuPar, the dense baseline significantly improves, but only SuPar shows consistent loss improvement and stable HPs.
Figure Summarizing loss results from Figure 6 with the optimal learning rate for each parameterization and sparsity.
Based on the SuPar formulation: When the number of non-zero weights per neuron (WPN) in the network is the same, μP and SuPar become synonymous, because initialization and learning rate adjustment factors will be constant (i.e., dmodee p WPN O(1)). Optimized SP HPs will also tend to work well. We define this new scaling setting, which we call Iso-WPN, to verify this hypothesis. In Figure 1I, we test SP HPs with ISO- scaling and see the optimal learning rate stays consistently between 2-7 and 2 with roughly aligned curves (we omit similar μP and SuPar plots for space, because their corrections are the same). The conclusion that when scaling SP models in an Iso-WPN sparse setting, HPs should maintain similar training dynamics. More generally, as WPN decreases
Not perfectly Iso- Parameter due unsparsified layers (embedding, bias, layer-norm, etc.)

---

(e.g., by increasing sparsity), the optimal learning rate will tend to increase proportionally, and vice versa".
SP pP
2i G-O
0:5 Dance Optimam
3.3
6-768
d-1088
d-1136
2178
Learn ng Rate
Figure 10: Losses at the end of training when Sso-Parameter scaling.
The SP optimized LR is relatively stable with iso- scaling (3 seeds).
Figure 11:
Figures 5, 6, 7, and 9 show SuPar is the only parameterization that ensures stable activation scales and stable optimal HPs across model widths and sparsities, satisfying the FLD.
Finding 3:
SuPar enables stable activation and stable optimal HPs for any width and sparsity.
4.3 SpPar scaling to large language model pretraining
We conclude this section reflecting on the demonstration of SuPar improvements in large-scale language model. We train 6 610M parameter models starting from a Chinchilla [23] compute-optimal training configuration with 20 tokens per parameter from the SlimPajama dataset. This larger model- with hidden size 2048, 10 layers, and attention head size -permitss sweeping over larger range of sparsity levels, so we test up to 99.2% sparsity (density 2 2-7).
Figure shows validation loss for each parameterization as we sweep sparsity levels. Additionally, in Table 2, we evaluate the models from Figure 3 on five downstream tasks: ARC-easy, lambada, RACE, PIQA, and BoolQ, which collectively test for common sense reasoning, world knowledge, and reading comprehension. As sparsity increases, results across pretraining loss and average downstream task accuracy consistently show SP and μP fall farther behind SuPar. Since these models are trained with large number of tokens, we attribute the widening loss gap mostly to increasingly under-tuned learning rates for SP and jP as sparsity increases-tus the weight updates lose gradient information throughout training. Figure 8 shows retuning SP and uP could recover some of the gap to SuPar, but that could be costly: These runs take 3-6 hours on Cerebras CS-3 system (or >> days on an NVIDIA A100 GPU).
Finally, returning to the Iso-Parameter scaling setting, Figure 10 shows losses for 111M parameter models trained on IB tokens and scaled up while using dense optimal HPs. The SP and uP models experience detuning sparsity increases, allowing SuPar achieve superior losses?.
Finding 4: Sparse networks trained with SuPar improve over SP and uP due improved tuning.
4.4 Dynamic sparsity hyperparameter transfer
In Figure 12 we test the transfer of optimal learning rate across sparsity levels for two popular dynamic sparse training methods: Rigging the Lottery (RigL) [10] and Gradual Magnitude Pruming (GMP) [68]". We show that none of SP. μP, or SyPar achieve transfer of optimal learning rate across sparsity levels. For SP and uP we see that higher sparsity levels have higher optimal learning rates.
"Our results generalize the Yang et al. finding that optimal LR decreases width increases [63, Figure 1]. 7 Note this is not an Iso-FLOP comparison because increasing dmoda also increases attention dot product and embedding FLOPs, which aren't be sparsified. This is significant that our 87.5% sparse model from Figure 10 has double the training FLOPS of the dense bascline, with virtually unchanged loss.
$ Rigl.: Uniform sparsity distribution, drop fraction 0.3, and mask updates every 100 steps.
GMP: Uniform sparsity distribution, cubic sparsity schedule, and mask updates every 100 steps.
8

---

This is because sparsity reduces activation and gradient scales such that a larger learning rate is needed to counteract this. SuPar sees the opposite trend where higher sparsity levels have lower optimal learning rates, indicating that SyPar is "overcorrecting".
Dynamic sparse methods make updates to the weight mask such that the distribution of unmasked/non-zero weights changes to something non-Gaussian, which prevents SuPar from being correct in expectation. Compared random pruning, mask obtained from magnitude pruning will better preserve the size of activations and gradients seen in the dense network. Since SyPar assumes weights are drawn from Gaussian distribution, SuPar ends up "overcorrecting" the initialization and learning rate. In future work it would be impactful to develop a parameterization which generalizes SuPar to work for an arbitrary sparse training algorithm.
RigL
Rigl.
RigL SpPar
5.2
5.0
4.6
4.4
4.2
2-41
2-
Learning rate
Learning rate
Learning rate
GMP SP
GMP APf
GMP SuPar
5.2
540.0
5.0
4.8
540 8..
516 9175
S-0 5=0.96875 00.96875
0-0.984375
#frarsterrad
4.0
3.8
Learnin
Learing
Learning rate
Figure 12: For dynamic sparse training methods RigL and GMP, none SP. uP, or SuPar achieve stable optimal learning rate across sparsity (3 seeds). Missing points indicate diverged training runs.
5
Discussion
To improve sparse training, prior works make targeted corrections which arise from observations that sparsity can cause degraded activation, gradient, and/or weight update signal propagation. We review these observations and corrections to advocate for holistic control of sparse training dynamics.
Sparsifying Can Cause Vanishing Activations Evci et al. [11] note that by initializing weights using dense methods (e.g. [17, 21]). the "vast majority" of sparse networks have vanishing activations. Lasby et [29, App. A] analyze activation variance guide for selecting structured sparsity. The FLD suggest activation norms be measured and controlled with respect to sparsity, so activation variance can be considered a proxy to whether sparsity might negatively impact training dynamics. Evci et al. [11] ultimately initialize variances via neuron-specitt sparse connectivity, while Liu et al. (35) and Ramanujan et [49] propose scaling weight variances proportional to layer sparsity. These corrections, however, only target controlling activations but not weight updates.
Gradient Flow Partially Measures the Weight Update pDesideratum Sparsity also impairs gradient foodientflowww--o flow-the the magnitude of the gradient to the weights ddurnng training [11. 1]. Since gradient flow is measured using the norm of the weight gradients, measures a piece of the weight update. Unfortunately, gradient flow does not directly measure the effect of the weight update step, which can also involve adjustments for things like optimizer state (e.g., momentum and velocity), the learning rate, and weight decay. Prior works propose techniques to improve gradient flow during sparse

---

training and pruning by adjusting individual hyperparameters or adding normalization [61, 37, 11, 11. However, these techniques might overlook the effects of the optimizer and learning rates in weight updates. Notably, Tessera et al. [57] do consider some of these effects, but their proposed techniques maintain gradient flow only in the Iso-Parameter scaling setting rather than arbitrary sparsification.
Frantar et al. [15, App. A.I] also endeavor to control weight updates, where they observe diminished step sizes when optimizing sparse networks with Adafactor [52]. They correct this by computing Adafactor's 's root-mean-squree scaling adjustments over unpruned weights and updates. However, such normalization does not prevent activations from scaling with model width [63, 65]. In this sense, sparsity- -awrr fixes to Adafactor can improve dynamics, but will not address instability holistically. In Figure 14 we show the SyPar LR correction alone is not even sufficient achieve stable optimal r7.
Weight Initialization Only Controls Dynamics at Initialization We noted works above that adjust sparse weight initializations [11, 35, 49]. Additionally, Lee et al. [31] explore orthogonal weight initialization [46], both before pruning (to ensure SNIP [32] pruning scores are on a similar scale across layers) and after (to improve trainability of the sparse network). While adjusting weights can improve sparse training dynamics at initialization, such adjustments are insufficient to stabilize signals after multiple steps of training, in the same way that standard weight initializations fail to stabilize training of dense networks. In Figure 13 we show the SuPar init. alone is not even sufficient to achieve stable optimal ow-
6 Limitations
As Section 4.4 shows. SuPar requires further extension for dynamic sparse training due to unpre- dictable changes in weight distributions. The same applies to methods which prune at initialization or after pretraining in non-random fashion. Iterative magnitude pruning (IMP) is an interesting case since involves rewinding weights back to their initial values while maintaining the same mask [12]. If the IMP mask at initialization still allows the non-zero weights to have Gaussian distribution, then SuPar would apply to this case. Therefore, it's possible SuPar could prevent "HP detuning" in later IMP iterations, and potentially improve IMP losses, though we do not explore this. SuPar would also work for random structured pruning of entire neurons at initialization because this case simply reduces to training with narrower dense model.
For weight sparsity more generally, the most pressing limitation is the lack of hardware accelera- tion [38]. While new software [50, 29, 43] continues to better leverage existing hardware, the growth of software and hardware co-design is also encouraging [59, 6]. effective sparsity techniques can be specifically optimized in deep learning hardware. But to effectively plan hardware, we need to train and test sparse prototypes at next-level sizes, at scales where the optimum sparsity level may be higher than in current networks [15]. Performing such scaling law-style studies requires incredible resources even for dense models with well-established training recipes [27, 23). As SuPar reduces training and tuning costs, it can help unlock these studies and guide future hardware design.
Finally, the scaling factors for the weight update need to be derived for each optimizer, which might limit the usability of SuPar in practice. For a discussion of the broader impacts of SuPar, see Appendix A.
7 Conclusion
Nobody said training with sparsity was easy. We showed that with the standard parameterization and μP, increasing sparsity level directly correlates with vanishing activations. Impaired training dynamics prevent sparse models from sharing the same optimal hyperparameters. suggesting prior results based on re-use of dense HPs merit re-examination. In contrast, by holistically controlling the training process, SuPar prevents vanishing activations and enables HP transfer (across both width and sparsity). L.MSs trained with SuPar improve over μP and the standard parameterization. As such, we hope SuPar makes things little easier for sparsity research going forward.
10

---

Acknowledgements
We would like to thank Gavia Gray, who provided helpful feedback on the manuscript, and Gurpreet Gosal, who tuned the yTTranfferred hyperparameters seen throughout the document.
References
[1] Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-
Chun Kao, Shivani Agrawal, Utku Evci, and Tushar Krishna. 2024, Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers. arXiv preprini arXiv:2402.04744 (2024).
[2] Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. 2017. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv:171..05136 (2017).
[3] Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountabiliry, and transparency. $10-623.
[4] Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. 2024. Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit. In The Twelfth International Conference on Learning Representations. https: //tp:////eeee...e.e//mm net/ httf//ormmmmd/////
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems 33 (2020), 1877-1901.
[6] Cerebras Systems. 2024. Train a Model with Weight Sparsity. Cerebras Systems Documen- tation. http:::///ocs..o/ cerebras net/ httpc//cwsttttt// http://oootttttudddeessspparaeee .ttml Version 2.1.1.
[7] Pau de Jorge, Amartya Sanyal, Harkirat S Behl, Philip HS Torr, Gregory Rogez, and Puneet K Dokania. 2020. Progressive skeletonization: Trimming more fat from a network at initialization. arXiv preprint arXiv:2006.000 0908/ (2020).
[8] Tim Dettmers and Luke Zettlemoyer. 2019. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv: arXiv: 1907.044700 (2019).
[9] Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, and Joel Hestness. 2023. BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model. arXiv:2309. 1566 [cs.Al]
[10] Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In International conference on machine learning. PMLR, 2943 29522.
[11] Utku Evci, Yani Ioannou, Cem Keskin, and Yann Dauphin. 2022. Gradient flow in sparse neural networks and how lottery tickets win. In Proceedings of the AAAI conference on artificial intelligence, Vol. 36. 6577-6586.
[12] Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint I 03633 (2018).
[13] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. 2020. Pruning neural networks initialization: Why are we missing the mark? arXiv preprint arXiv:2009.08576 (2020).
[14] Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive language models can be accurately pruned in ohe-shot. In International Conference Machine Learning. 10323-10337.
11

---

[15] Elias Frantar, Carlos Riquelme, Neil Houlsby, Dan Alistarh, and Utku Evci. 2023. Scaling laws for sparsely-connecter foundation models. arXiv preprint 0XXv:2200.005200 (2023).
[16] Trevor Gale, Erich Elsen, and Sara Hooker. 2019. The state of sparsity in deep neural networks. arXiv preprini arXiv: 1902.09574 (2019).
[17] Xavier Glorot and Yoshua Bengio. 2010. Understanding the Difficulty of Training Deep Feedforward Neural Networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics (PMLR).
[18] Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. 2020. Are wider nets better given the same number of parameters? arXiv preprint arXiv:2010. 14495 (2020).
[19] Hila Gonen and Yoav Goldberg. 2019. Lipstick on pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. arXiv preprint arXiv: / 903.00330 (2019).
[20] Yiwen Guo, Chao Zhang, Changshui Zhang, and Yurong Chen. 2018. Sparse dnns with improved adversarial robustness. Advances in neural information processing systems 31 (2018).
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In Proceedings of the IEEE international conference on computer vision. 1026-1034.
[22] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. 2021. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. Journal of Machine Leurning Research 22, 241 (2021), 1-124.
[23] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George den Driessche, Bogdan Damoe, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. 2022. An Empirical Analysis of Compute-optimal Large Language Model Training. In The Conference on Neural Information Processing Systems (NeurIPS).
[24] Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. 2019. What do compressed deep neural networks forget? arXiv preprint
(2019).
[25] Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058 (2020).
[26] Hycong-Ju Kang. 2019. Accelerator-aware pruning for convolutional neural networks. IEEE Transactions on Circuits and Systems for Video Technology 30, 7 (2019), 2093-2103.
[27] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling Laws for Neural Language Models. arXiv:2001.0832 [cs.LG] https: /ttp://aaxvvvvv. wwt.://////.....////333
[28] Denis Kuznedelev, Eldar Kurtic, Eugenia lofinova, Elias Frantar, Alexandra Peste, and Dan Alistarh. 2023. Accurate neural network pruning requires rethinking sparse optimization. arXiv preprint arXiv:2308. 02060 (2023).
[29] Mike Lasby, Anna Golubeva, Utku Evei, Mihai Nica, and Yani Ioannou. 2023. Dynamic Sparse Training with Structured Sparsity. arXiv preprint arXiv:2305.0222 (2023).
[30] Yann LeCun, John Denker, and Sara Solla. 1989. Optimal brain damage. Advances in Neural Information Processing Systems 2 (1989).
[31] Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. 2019. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint ar.iv: 06307 (2019).
[32] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. 2018 SNIP: Single-shot network pruning based on connection sensitivity, arXiv preprint 1rX0v:1810002340 (2018).
12

---

[33] Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, and Mykola Wang, Zhangyang an d Pechenizkiy. 2022. The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXiv:2202.0200 (2022).
[34] Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. 2021. Do we actually need dense vieepparameterizatio?? In-time over-parameterization in sparse training. In International Conference on Machine Learning. PMLR, 6989-7000.
[35] Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2018. Rethinking the value of network pruning. arXiv preprint arXiv: (2018).
[36] Iyy Loshchilov and Frank Hutter. 2017. Decoupled Weight Decay Regularization. In Interna- tional Conference on Learning Representations.
[37] Ekdeep Singh Lubana and Robert Dick. 2020. A gradient flow framework for analyzing network pruning, arXiv preprint arXiv:2009. / 1839 (2020).
[38] Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. 2021. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08377 (2021).
[39] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Bares,, Lucy Vasserman, Ben Hutchin- son, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model cards for model reporting. In Proceedings of the conference on faimess, accountability, and transparency. 220-229.
[40] Decebal Constantin Mocanu, Elena Mocanu, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. 2016. A topological insight into restricted Boltzmann machines. Machine Learning 104 (2016). 243-270.
[41] Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibeseu, and Antonio Liotta. 2018. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications 9, (2018), 2383.
[42] Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.0944 (2020).
com/neuralmagic// [43] Neural Magic. 2024. DeepSparse. GitHub repository. http::///ggbbbbb.c///aaaaaangg// deepsparse
[44] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L.. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training Language Models to Follow Instructions With Human Feedback. arXiv:2203.0211 [cs.CL] /tttss//arxivvorg///
02155
[45] David Patterson, Joseph Gonzalez, Quoc Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. 2021. Carbon emissions and large neural network training. arXiv preprint arXiv:2104. 10350 (2021).
[46] Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. 2017. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. Advances in neural information processing systems 30 (2017).
[47] Ofir Press, Noah Smith, and Mike Lewis. 2022. Train Short, Test Long: Linear Biases Enables Input Length Extrapolation. In International Conference on Learning Representations.
Attention with
[48] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
2019. Language Models are Unsupervised Multitask Learners. https: : /tt////famaaa.w....c../ cloudfront wwt.beeeeeeeeeeee
-ooudronnt.ett//tteeee------ wttpb//ttteernngggeeeeeeeee sanguage-modelll haageeeeooeeee///neeee-eee-e
aanguage--moeell. pdf
[49] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. 2020. What's hidden randomly weighted neural network?. Proceedings of the IEEE/CVF conference computer vision and pattern recognition. 11893-11902.
13

---

[50] Erik Schultheis and Rohit Babbar. 2023. Towards Memory-Efficient Efficient Training for Extremely Large Output Spaces Learning with 670k Labels on Single Commodity GPU. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 689-704.
[51] Noam Shazeer. 2020. GLU Variants Improve Transformer. arXiv:2002.05202 [cs.LG] https:
/ //arxiv. org/ www.//////////o///////////e/ee// /002 05202
[52] Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. PMLR, 4596-4604.
[53] Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. The woman worked as a babysitter: On biases in language generation. arXiv preprint 1xii:1000000300 (2019).
[54] Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. SlimPajama: A 627B token cleaned and deduplicated version of RedPajama.
https: pttps://wwee /www. cerebras net/blog/ -adaaaadddddddddddddeeeettted-ee
-atpaaaaaaaaa--2---tt--eee https:/ https://hugging http://guggnngac.e.oooo
www.eiattddddeeeiio-oooooooo redpa ama.
http://eeeee.ettttttteteeee////e jaa.-a22aamm22277 627B
[55] Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considera- tions for deep learning in NLP. arXiv preprint arXiv:1906.0222: (2019).
[56] Hidenori Tanaka, Daniel Kunin, Daniel I. Yamins, and Surya Ganguli. 2020. Pruning neural net- works without any data by iteratively conserving synaptic flow. Advances in neural information processing systems 33 (2020), 6377-6389.
[57] Kale-ab Tessera, Sara Hooker, and Benjamin Rosman. 2021. Keep the gradients flowing: Using gradient flow to study sparse network optimization. arXiv preprint arXiv:2 (2021).
[58] Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. 2023, SPDF: Sparse pre-training and dense fine-tuning for large language models. In Uncertainty Artificial Intelligence. 2134-2146.
[59] Vithursan Thangarasa, Shreyas Saxena, Abhay Gupta, and Sean Lie. 2023. Sparse-IFT: Sparse Iso-FLOP transformations for maximizing training efficiency. arXiv preprint urXiv:2303.1152. (2023).
[60] Stijn Verdenius, Maarten Stol, and Patrick Forré. 2020. Pruning via iterative ranking of sensitivity statistics. arXiv preprint arXiv:2006. 00896 (2020).
[61] Chaoqi Wang, Guodong Zhang, and Roger Grosse. 2020. Picking winning tickets before training by preserving gradient flow. arXiv preprini 2rXiv:2002.07376 (2020).
[62] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems 36 (2023).
[63] Greg Yang, Edward Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. 2021. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer. In Advances in Neural Information Processing Systems.
[64] Greg Yang and Edward J Hu. 2020. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14553 (2020).
[65] Greg Yang. James B Simon, and Jeremy Bernstein. 2023. A spectral condition for feature learning. arXiv preprint arXiv:2310.17811 (2023).
[66] Greg Yang, Dingli Yu, Chen Zhu, and Soufiane Hayou, 2023. Feature Learning in Infinite Depth Neural Networks. In International Conference on Learning Representations.
[67] Zhuliang Yao, Shijie Cao, Wencong Xiao, Chen Zhang, and Lanshun Nie. 2019. Balanced sparsity for efficient DNN inference on GPU. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 5676-5683.
[68] Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.0187. (2017).
14

---

A Broader impacts
Sparsity is recognized to reduce carbon emissions [45] and offset well-known environmental and financial costs of large model training [3]. For example, unstructured sparsity can be accelerated by the Cerebras Wafer-Scale Engine" 10 and 2:4 block sparsity can be accelerated by NVIDIA Ampere GPUs There is growing recognition that HP tuning is key contributor to these costs. HP tuning is costly, possibly undermining equity in Al research due to financial resources [55]. During model retraining, sensitivity to HPs also leads to downstream costs [55]. SuPar reduce these costs and sensitivities and thus improve equity.
Sparsity also has potential drawbacks. Hooker et al. [24] showed that even when top-line performance metrics are comparable, pruned networks may perform worse on specific subsets of the data (including on underrepresented groups [25]), may amplify sensitivity to adversarial examples, and may be more sensitive to distribution shift. These sensitivities may depend on the degree of sparsity [20]. It remains an open question whether such drawbacks occur only with pruning when training with sparsity from scratch (as in SuPPar) [22], and how such sensitivity may impact susceptibility to misuse [62]. methods to detect [53, 42] and mitigate [19, 44] harm. Moreover, since many large models are later pruned for deployment, we recommend testing and documenting in the model card [39] any adverse affects of sparsification the time of model release.
We require sparsity-speciie
B Downstream task comparison of parameterizations
In Table 2, we evaluate the models from Figure on five downstream tasks: ARC-easy, lambada, RACE, PIQA, and BoolQ, which collectively test for common sense reasoning, world knowledge, and reading comprehension. We also specifically chose tasks that are easy enough for even extremely sparse models to significantly outperform random chance.
Sparsity
0
0.5
0.75
0.875
Rand.
SP
SpPar
SP
μP
SuPar
SP
uP
SuPar
SP
uP
SjPar
ARC-easy
0.25
0.49
0.51
0.51
0.45
0.49
0.48
0.44
0.45
0.45
0..22
0.43
0.44
LAMBADA
0.00
0.32
0.36
0.36
0.27
0.31
0.32
0.22
0.27
0.28
0.20
0.23
0.25
RACE
0.25
0.30
0.30
0.30
0.29
0.31
0.30
0.28
0.30
0.29
0.27
0,28
0.28
PIQA
0.50
0.67
0.67
0.67
0.63
0.65
0.67
0.63
0.64
0.64
0.62
0.63
0.63
BoolQ
0.50
0.53
0.57
0.57
0.58
0.55
0.51
0.57
0.62
0.52
0.61
0.62
0.62
Avg.
0.30
0.46
0.48
0.48
0.44
0.46
0.46
0.43
0.46
0.44
0.42
0.44
0.44
Sparsity
0.9375
0.96875
0.984375
0.992188
Rand.
μР
SuPar
SP
μP
SpPar
SP
uP
SuPar
uP
SuPar
ARC-easy
0,25
0.41
0.40
0.43
0.39
0.42
0.41
0.38
0.38
0.41
0.37
0.38
0.38
LAMBADA
0.00
0.19
0.20
0.21
0.16
0.18
0.19
0.13
0.15
0.17
0.12
0.13
0.14
RACE
0.25
0.25
0.27
0.28
0.24
0.25
0.27
0.24
0.24
0.25
0.25
0.24
0.26
PIQA
0.50
0.61
0.61
0.61
0.60
0.61
0.60
0.59
0.60
0.59
0.58
0.60
0.60
BoolQ
0.50
0.62
0.62
0.61
0.57
0.62
0.62
0.45
0.62
0.61
0.41
0.61
0.61
Avg.
0.30
0.42
0.42
0.43
0.39
0.42
0.42
0.36
0.40
0.41
0.34
0.39
0.40
Table Downstream evaluation accuracy; higher better: SuPar performs best or within 0.01 of
best across all sparsity levels and tasks, except boolg at 50% and 75%
sparsity. Even at 99% sparsity,
SuPar models maintain 40%+ average accuracy, whereas the SP model drops to 34%, close to the
30% accuracy of the random baseline.
C Individual ablations of SuPar initialization and learning rate corrections
In Figures 13 and 14, individually ablate the effect of the SpPar initialization and the SuPar learning rate. We show that using only the SuPar initialization in conjunction with μP (uP + SuPar initialization only) does not allow for transfer of optimal initialization standard deviation optimal learning rate across sparsity levels. We also show that using only the SpPar learning rate in conjunction with uP does not achieve transfer either. Therefore, both the SyPar initialization and learning rate corrections are required to achieve optimal hyperparameter transfer across sparsity levels.
1onttpa://wwr werebraa..et/blog/aaaaeeeee phttps://wwe nvidia,
spw.sstttt-- gpt- mi-dell
cetiiia..cm//eeee com/en-o/////eeeeeeeeeeeeee us- wataa
15

---

36.0
5 0.5
6- 0.75
0.755
8.5
0.5375
8.0
0.984375
Optmmum
Danse Optimum UTransferred
6.5
Inittalization
Initializationn
at
Figure 13: SuPar ensures stable optimal weight initialization standard deviation, unlike SP, μP, μP + SuPar init only, and μP SuPar LR only.
JPP SyPar only
SyPar only
SyPar
4.0
2 0.5
1.8
5- 0.875
0,75
0.9375
4- 6.994273
Optimam
3.4
Derie Optimunt ursnneereed
35
2-1
2
2-1
2
Learning Rate
Learning Rate
Leamning Rate
Figure 14: SuPar ensures stable optimal learning rate (Bottom), unlike SP, μP, μP and μP + SuPar LR only.
SuPar init only,
D SuPar detailed derivation
D.1 Forward pass at initialization
The first stage where we would like to control training dynamics is in the layer's forward function. For random unstructured sparsity mask M, since each column of M has daap non-zero elements in expectation, we can rewrite the forward pass as:
dup
Yu = X(W O M) Σ (g (W of M,j- =
Σ
X W
kj
(4)
k:M,==1
To satisfy the FLD, we desire the typical element size of Y O(1) with respect change in width Md, and change in density mp. To achieve this we can ensure the mean and variance of Y are invariant to md, and mpp
Mean: As expectation is linear and X and W are independent at initialization:
diap Σ
dap Σ
dup
Y 1
E
Y WaJ
E| W
E ky]
(5)
M
Therefore, since at initialization E|W sy 0. E Y s5 0 and the mean is controlled.
Variance: As expectation linear and each weight element IID:
dap Σ
dip
Var(Y.)) Var
XaR W
Σ
Var(X.s Wky)
(6)
M
k:M,==1
16

---

Then, since X and W are independent at initialization:
dup
Var(Yi;)
(Var(X,)) + (Var + B[W&,]3)) (E (W,))) (7)
A:M,==1
Finally, since at initialization E| [[W kn] = 0 and redefining Var( (Kx) = ow:
dup
Var (i)))
(Var(X.x) + E[Xx]") Var Wkj) = dinpo (Var(X) + B[X])
(8)
k:M,y=1
Rewriting in terms of multipliers for the width Mid den, and the change in density mp
Var (Y m.,, din, base MaaemppPPaeeee w Var E[X]²)
Solution: To satisfy the FLD and ensure Var Y sz) scales independently of md, and mp, we choose This ensures typical entry size of Y is invariant changes in width md, and
to set ow "Wbbas m., m,,
density mpp
Note that this correction is equivalent to pP [63] when m, 1. Further, the sparsity factor in the denominator matches the correction for sparsity-awan aware initialization from Evei al. [11].
D.2
Backward gradient pass at initialization
The next stage we would like to control training dynamics is the layer's backward pass. For random unstructured sparsity mask M, since each of M has down non-zero elements in expectation, we can rewrite the backward pass as:
diee
damp
VxLy = [VyL(W ( M) - VyLi (W Mja)
Σ
V Wjx
(10)
k:M,e=1
To satisfy the FLD, we desire the typical element size of Vx O(1) with respect change in width mi., and change in density my- To achieve this, we can ensure the mean and variance of VxL invariant m.d.. and mpp
Mean: Although the gradients V will have some correlation with weights W even at initializa- tion, we assume for simplicity that they fully independent. Future work could investigate this assumption more deeply. As expectation linear:
douP
domp Σ
doup Σ
E
Li Wsk
V E
W ju]
M
k=M
(11)
Therefore, since at initialization E jk] = 0, E(Vxyl = 0, the mean is controlled.
Variance: As expectation is linear and each weight element is IID:
deup
duap
Var(VxXug)= = Var
ViLix
W
Var(VY W₁
(12)
A:M ==
M
From the backward pass mean derivation, know E 0. Then, similar to the forward pass variance derivation, we can simplify using the facts that at initialization, Vy and W are (roughly) independent and E[W] 0. Similarly
also define Var(W/ vyk) ow and rewrite in terms of
width multiplier mid...
dur and changes in density m.
Paue
Var Par(VxLs)) Mild. dout.base Poase w Var(VVL)
(13)
Solution: To satisfy the FLD and ensure Var(VxLi)) scales independently of ma... and mps we This ensures the typical entry size of V C is invariant to changes
ow haw man map
choose to set ow
17

---

in width md. and density mp. Typically, we scale model width such that Meg.. m. This equal scaling allows the same initialization variance correct both forward activation and backward gradient scales, making them independent of width. Further, since we assume random sparsity across layer's weights, the sparsity initialization correction factor, mps the same for both the forward activations and backward gradients.
D.3 Effect of weight update AW on Y
To satisfy the FLD, we desire the typical element size the weight update AY ((1) with respect to change in width ma, and change in density mp- To achieve this examine the expected size of each element. Here, we use to be the learning rate for this layer. For a random unstructured sparsity mask M, since each column of M has daa non-zero elements in expectation:
di Xig AWW, oj Mnj) =
dup Σ
AY = (AW O M), =
X (14)
A=
Mean: As expectation in linear:
dal Σ M:Ma
E AAY = E
A
1
Σ
BX ky)
(15)
k:Ma== =1
Since AW derived from X, there is covariance between these variables and E E A W is non-zero. By the Law of Large Numbers:
E|AY, ,y - 7dmpB Ax, as (dmp) - 00
(16)
Rewriting in terms of width and density multipliers:
o[ + 1,,,, ban pfrrase I W] as (dmpp) - 00
(17)
Equation will be used intermediate result in the following sections.
D.3.1 Effect SGD weight update AW Y
Following the formulation in [63], stochastic gradient descent (SGD) weight updates take the form:
(X) (Vx
B
A w kj
b (VyL)
(18)
dim
dim
kj
so we can rewrite Equation 17 as:
B
AA b + Phase E.
X
XBe V (Vx))s) as (dimp) + 50
(19)
For SGD updates, to satisfy the FLD and ensure E AAY Y úg] and the typical entry size of AY are scale invariant to ma and mps we choose Mosse mpp Note this correction equivalent to μP [63] when p 1, m, 1.
Solution:
D.3.2 Effect of Adam weight update AW on Y
Following the formulation in Yang et al. [63], Adam weight updates take the form:
Σ 7h
V
L)
AW kjj
6k
(20)
E
Σ
X 6k
VY 1
where 7 is the current training step and 7ts w/ are the moving average weights at each training step. Here, just consider the weight update associated with an unpruned weight, since pruned
18

---

weight will have value and update (i.e., pruned weights trivially satisfy that their effect on forward activations cannot depend on width or sparsity). We can rewrite Equation 17 as:
1. as (dinP) -
E VEi We Ee
ve Eo
Ciny
B 7 Mp din. Mp pPrase E
Xik
X 2,y)
(21)
Solution: For Adam updates, to satisfy the FLD and ensure B|AY xy] and the typical entry size of AY Y are scale invariant to md, and mp. we choose 77 to uP [63] when = 1,, mp
Thess mh, Note that this correction equivalent
D.4 Additional notes about derivation
We make few supplementary notes about the above derivation:
Throughout our derivation, we use p to refer to the density level. Note that since this derivation is local to a single layer in the model, the density (or sparsity) level can also be parameterized independently for each layer. a sparsity technique will use layer-wise independent sparsity levels, appropriate corrections should be made for each layer.
Similar to the notation, we use to denote the learning rate, but this learning rate can be layer-specific depending on sparsity level. Appropriate corrections must be made using layer-wise independent sparsities.
The use of the Law of Large Numbers in portions the above derivation indicate that SuPar is expected to provide stable training dynamics the number of non-zero weights per neuron (WPN) tends to infinity. However, in sparse settings, the WPN can tend to be small. If WPN is small, training dynamics may be affected, and this might be direction for future work.
this work, we only consider sparsifying linear projection layers. As a result, SuPar matches yP for other layers like input, output, bias, layer-norm, and attention logits. Depending on the sparsification technique, these other layers might need to be reviewed for their effects on training dynamics.
E Experimental details
SuPar base hy typeparreeeeer tuning To find the optimized set of hyperparameters for SuPar, we actually tune uP HPs dense proxy model. By formulation of SuPar, these HPs transfer optimally to all the sparse models trained for this work. This dense proxy model is a GPT-2 model, but with small changes: ALiBi position embeddings [47] and SwiGLU nonlinearity [51]. We configure it with width: dunudel = dumadel.buse = 256, number layers: Paayers 24, and head size: d 64, resulting in 39M parameters. We trained this proxy model 800M tokens with batch size of 256 sequences and sequence length 2048 tokens. We randomly sampled 350 configurations of base learning rates, base initialization standard deviation, and embedding and output logits scaling factors. From this sweep we obtained the tuned hyperparameters listed in Table 3.
Table 3: Tuned hyperparameters for our dense proxy model.
Hyperparameter
Value
0.08665602
Thoae
1.62E-2
Oiinpul
9.1705
Oouuput
1.0951835
Experimental details for all figures In Table 4, we provide extensive details on hyperparameters, model size, and training schedule for all experiments this paper. models in this paper were trained on the SlimPajama dataset [54], cleaned and deduplicated version of the RedPajama dataset.
19

---

Tokens
306M
1.6M
12.13B
82K
306M
18
306M
306M
Steps
1169
100
11752
1169
1907
1169
1169
LR warm-
up steps
116
1175
116
190
116
116
Decay to zero
LR decay
10x linear
Constant
Constant
10x linear
10x linear
10x linear
10x linear
Table 4: Experimental details for all figures in this paper.
Ccompi
1.095
1.095
1.095
1.095
1.095
N/A
1.095
Okipus
9.1705
9.1705
9.1705
11.22
9.1705
9.1705
N/A
9.1705
Init. Stdev.
SP: 2. 166E-2
μP, SpPar: 0.087
Variable
SP: 0.02
μ... SuPar 0.087
0.101
SP: 0.02
μP, SuPar: 0.087
SP: 0.02
pP. SpPar: 0.087
SP: 2. 166E-2
μP. SuPar: 0.087
0.087
LR
Variable
1.62E-2
SP: 2e-4
μP,, SuPar: 1.62B-2
1.88--02
μP. μP. SpPar: 1.62E-2
Variable
SP. dondd S 1088: 6E-4
2F-4
μP, SpPar: 1.628-2
Variable Variable
SP. dmodd > 1088:
B
128
504
128
256
128
128
dhoad
64
64
64
64
64
L
2
10
2
2
10
2
2
desodel
4096
4096
2048
2048
Variable
Variable
Variable
1024
Fig. 7, 13
Figure
6, 8, 14
Fig. 10
Fig. 12
Fig.
Fig.
Fig.
Fig.
Fig.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Sections 6 and A contain a discussion of limitations and broader impact.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification: In Section 3 we provide short proof sketch to provide intuition behind both SuPar and pP. Then Section we provide a detailed derivation for SuPar (which also reduces to uP when p 1).
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: To ensure reproducibility, in Section E we provide extensive details all experiments contained in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All tests in this paper use GPT-like transformer language models [48, 9], trained on the SlimPajama dataset [54], which are both open. Section E. contains sufficient detail to faithfully reproduce the main experimental results. A minimal implementation of SuPar is available at tttps://gtt thub. com/EleutherA AI/tahoor// mmup/ tt/e/ supar. We also provide simple breakdown of the code changes required in Table 1.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
public/guiden CcodeeSbbissiionee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /nips. /twppscc/pppiii wwtpccpwwwcc.pcecceeeeooope
CodeSubmiss ssionsolicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: To ensure reproducibility, in Section we provide extensive details on all experiments contained in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In most of our experiments, we sample multiple seeds and plot the mean loss as a solid line, as well as the standard error of the mean as a shaded area of the same color. We disclose the number of seeds used for each experiment in the corresponding figure captions.
Guidelines:
The answer NA means that the paper does not include experiments.
23

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section we provide extensive experimental details which can be used to easily derive compute requirements based on reader's specific hardware setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https : ////:///neuee /wwuuriur.p...// cc cc/public//eieeccccceecc httppppllllihisaii/iaepppp http:/tthiiiiiiieeeeeeeeeeeee chh.idelinedddee
Answer: [Yes]
Justification: We do not conduct any research with human participants. Also, consider the broader impacts and limitations our work in Sections A and 6. Finally, we provide extensive experimental details in Section E improve reproducibility.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]

---

Justification:
See Section A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper presents a new neural network parameterizzation SyPar and does not release any models or datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all datasets and architectures in accordance with their licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
25

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce new assets in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not conduct any research involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not conduct any research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "7E7csyAcEA", "reviewer": "Reviewer_F4Ay", "summary": "This paper studies the effect of various hyperparameters on static sparse training while having a holistic approach. It highlights that there is a correlation between their settings and neural network performance (practically loss function values). The experiments are performed on smaller and larger models, including a large-scale language model.", "questions": "I don't have questions which may change my evaluation. Still, I consider this study promising, and I hope that my comments will help the authors to improve the next version of the paper.", "limitations": "n/a", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This is an interesting study with a relatively low level of originality (in my opinion). It practically puts together and discuss a bit more various findings from various papers. Also, the paper structure and clarity could have been improved. Currently, it is not easy to understand it. If the paper would be more mature, may have a relatively fair impact in the community.", "weaknesses": "I believe that one main weakness is that the networks performance is reported just with loss values. For a more comprehensive view, and deeper understanding other performance metrics (task dependent) shall be also presented for all experiments.\n\nWith respect to the problem motivation, I believe that the paper exaggerates a bit the difficulty of training static sparse neural networks. Perhaps, for static sparsity, the effect of hyperparameters on network performance is more pronounced, but the alternative dynamic sparsity is quite robust to hyperparameter choice (e.g., sparsity level) and can obtain quite easy performance on par (or even better) with dense neural networks while having much fewer theoretical computational requirements as reported in a number of related works. For a fair overview, the paper shall discuss also comparatively the performance of static sparsity against dynamic sparsity and pruning, or alternatively, to motivate better its settings. \n\nThe paper readability can be improved with a better structure, more clarity in arguments and qualitative discussions, and more clear statements about the paper novel contributions.\n\n(minor) The source code shall be provided to ensure easy reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very useful suggestions, your positive comments regarding the promising/interesting findings, and your note on the potential for impact in the community. \n\n```\nThis is an interesting study with a relatively low level of originality (in my opinion). It practically puts together and discuss a bit more various findings from various papers. Also, the paper structure and clarity could have been improved. Currently, it is not easy to understand it. If the paper would be more mature, may have a relatively fair impact in the community.\n```\nWe take your point regarding the level of originality, however we note that we are the first to acknowledge and propose a solution to the systematic misuse of dense hyperparameters in sparse training. On Lines 90-91 and Footnote 1 (page 3) from our submission, we demonstrate how widespread the issue is by providing numerous examples of sparse training works which use dense hyperparameters for sparse training. So, regardless of the perceived level of innovation, the work is important and the potential for impact is fairly high, as you noted.\n\n```\nI believe that one main weakness is that the networks performance is reported just with loss values. For a more comprehensive view, and deeper understanding other performance metrics (task dependent) shall be also presented for all experiments.\n```\n\nThank you for suggesting this. Please see Section 3 \"Downstream Task Evaluations\" from our \"Global Author Rebuttal\". We hope this addresses your concerns.\n\n```\nWith respect to the problem motivation, I believe that the paper exaggerates a bit the difficulty of training static sparse neural networks. Perhaps, for static sparsity, the effect of hyperparameters on network performance is more pronounced, but the alternative dynamic sparsity is quite robust to hyperparameter choice (e.g., sparsity level) and can obtain quite easy performance on par (or even better) with dense neural networks while having much fewer theoretical computational requirements as reported in a number of related works. For a fair overview, the paper shall discuss also comparatively the performance of static sparsity against dynamic sparsity and pruning, or alternatively, to motivate better its settings.\n```\nThis is very perceptive –- and a point that we did not fully appreciate previously.\nBut first of all, we should have done a better job of motivating static random sparsity – the topic of the paper – to begin with.  We will note in the paper that static sparsity remains a very important technique and intense focus of research for ourselves and other labs, not least because it enjoys native hardware support (e.g., 2:4 sparsity in NVIDIA GPUs, full unstructured sparsity in Cerebras chips).\n\nThat being said, investigating hyperparameter shift in DST methods is insightful. Please see Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\". In this section we show that the optimal learning rate significantly changes across sparsity levels for both RigL and GMP, contradicting your claim that dynamic sparsity is quite robust to hyperparameter choice and further highlighting the importance of developing systematic solutions like S$\\mu$Par. As we discussed on Lines 274-278 of our submission, S$\\mu$Par is not correct for dynamic sparse methods, leaving an impactful direction for future work.\n\nWe really appreciate your suggestion here as these interesting findings definitely help expand the paper’s contribution!\n\n```\n(minor) The source code shall be provided to ensure easy reproducibility.\n```\nUnfortunately we do not have an open-source repository for S$\\mu$Par. However, we do provide Table 1 in our submission, which outlines the simple implementation changes required to implement $\\mu$P and S$\\mu$Par. Based on this it should be straightforward to extend existing implementations of $\\mu$P to implement S$\\mu$Par (e.g. https://github.com/microsoft/mup).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ot7s37qCHE", "reviewer": "Reviewer_6wqj", "summary": "The paper introduces Sparse Maximal Update Parameterization (SµPar), a novel approach designed to address challenges in training sparse neural networks. Sparse networks, despite their computational advantages, often struggle with signal propagation and optimal hyperparameter (HP) tuning across different sparsity levels. SµPar aims to stabilize training dynamics by ensuring that activations, gradients, and weight updates are scale-invariant with respect to sparsity. This method allows for consistent optimal HPs across varying model widths and sparsity levels, thus reducing the tuning costs associated with sparse models. The empirical results demonstrate that SµPar outperforms standard parameterization (SP) and maximal update parameterization (µP) in maintaining stable optimal HPs and achieving better loss metrics across different sparsity levels and model scales.", "questions": "Please refer to weaknesses.", "limitations": "Please refer to weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The idea of this paper is interesting. The authors select a trending topic with very clear and strong motivation. The preliminary experimental results is very clear and helpful to the readers.\n\n2. The discussion about forward/backward pass and weight updates are very clear. Figures are also clear and good to read.\n\n3. The writing is good and the paper is well structured.", "weaknesses": "1. This paper has many supporting experiments reporting the training loss, validation loss, and transfer loss for SµPar. However, loss values are not always reliable for evaluation. The accuracy data is a more convincing data to demonstrate the performance. I wonder what is the accuracy for the experiments. It would make reader easy to compare against other methods.\n\n2. What is the method that is used to prune the network? There are lots of method such as SNIP and GraSP for static sparse training. However, this part is unclear according to the paper. \n\n3. The author claims SµPar is a holistic solution. However, from what the paper has demonstrate (discussion, experiments), the reviewer cannot be convinced. According to weaknesses 2, we don’t know what is the pruning method used in sparse training, and according to the paper, the author also not performs applying SµPar on different sparse training methods. Therefore, we don’t know if SµPar works for different sparse training algorithms. Furthermore, I assume this paper only discuss static sparse training (such as SNIP), without discussing the performance on dynamic sparse training (DST) which is a more reliable and promising sparse training domain (e.g., better accuracy, flexible). Therefore it is not suitable to claim SµPar is a holistic solution. Even the authors don’t make such claim, without sufficient experiments the reviewer mentioned in this part, this paper is still lacks fundamental evidence to demonstrate effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the helpful comments, and for your kind words regarding the paper's key idea, motivation, and clear and well-structured presentation.\n\n```\nThis paper has many supporting experiments reporting the training loss, validation loss, and transfer loss for SµPar. However, loss values are not always reliable for evaluation. The accuracy data is a more convincing data to demonstrate the performance. I wonder what is the accuracy for the experiments. It would make reader easy to compare against other methods.\n```\n\nThank you for suggesting this. Please see Section 3 \"Downstream Task Evaluations\" from our \"Global Author Rebuttal\". We hope this addresses your concerns.\n\n```\nWhat is the method that is used to prune the network? There are lots of method such as SNIP and GraSP for static sparse training. However, this part is unclear according to the paper.\n```\nThank you for bringing this lack of clarity to our attention! In this work we solely used random static unstructured pruning at initialization, as stated on Lines 107, 512, 528, 549, and 564. To improve clarity we will update our abstract, intro, and conclusion to further emphasize our choice of pruning method. As we noted on Lines 27-28, recent work has shown random sparsity to be a surprisingly effective strategy for sparse training, so we adopted this in our work.\n\n```\nThe author claims SµPar is a holistic solution. However, from what the paper has demonstrate (discussion, experiments), the reviewer cannot be convinced. According to weaknesses 2, we don’t know what is the pruning method used in sparse training, and according to the paper, the author also not performs applying SµPar on different sparse training methods. Therefore, we don’t know if SµPar works for different sparse training algorithms. Furthermore, I assume this paper only discuss static sparse training (such as SNIP), without discussing the performance on dynamic sparse training (DST) which is a more reliable and promising sparse training domain (e.g., better accuracy, flexible). Therefore it is not suitable to claim SµPar is a holistic solution. Even the authors don’t make such claim, without sufficient experiments the reviewer mentioned in this part, this paper is still lacks fundamental evidence to demonstrate effectiveness.\n```\n\nThank you for raising these points. Please see Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\". This section shows that none of SP, $\\mu$P, or S$\\mu$Par achieve transfer of optimal learning rate across sparsity level for RigL and GMP. We hope this addresses your concerns around applying S$\\mu$Par to dynamic sparse training methods.\n\nAlso, we agree that ``holistic'' is a vague term, as it might imply S$\\mu$Par solves challenges with other forms of sparse training, such as dynamic sparse training.  We should have been more clear that S$\\mu$Par holistically controls the scale of the forward, backward, and weight update operations with respect to model width and level of unstructured static random sparsity –- some of these operations have been handled \\emph{in isolation} in prior work, but not all together.\n\nWe will also update our introduction to make it clear that S$\\mu$Par does not extend to pruning or dynamic sparse training because these methods can cause the unmasked/non-zero weight distribution to become non-Gaussian (as noted currently only in L274-L278).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3zrv4nOBN", "reviewer": "Reviewer_hqe9", "summary": "A parameterization method is proposed to ensure that the sparse models with different weight sparsity share the same set of optimal hyperparameters. The sparse models with S$\\mu$PaR achieve lower validation accuracy using the same hyperparameters the dense model uses.", "questions": "Major questions\n- Is the proposed parameterization method generalizable to any structured sparsity patterns (e.g., channel pruning) for the density-invariant hyperparameters? As mentioned in the paper, the unstructured sparsity is extremely hard to speed up. Discussing S$\\mu$PaR on more structured sparsity patterns, such as channel pruning, will make the paper stronger. \n- Why does S$\\mu$PaR stabilize the sparse network training even after multiple training steps? How is S$\\mu$PaR technically different from the previous sparse weight initialization methods? Also, what is the accuracy difference between S$\\mu$PaR and other sparse weight initialization methods?\n\nMinor questions\n- Line 113 about the weight update $\\mathbf{X}^{l+1} + \\Delta \\mathbf{X}^{l+1}$ is confusing because the weight update is denoted by $\\Delta \\mathbf{W}^l$ in Line 105.\n- Is there any connection between the proposed method and the iterative magnitude pruning (IMP)used in Lottery Ticket Hypothesis [2]? Can S$\\mu$PaR improve the final performance of IMP?\n\n[2] Frankle, Jonathan, and Michael Carbin. \"The lottery ticket hypothesis: Finding sparse, trainable neural networks.\" arXiv preprint arXiv:1803.03635 (2018).", "limitations": "The authors have addressed the limitations and broader impacts in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. The experimental results support the claim well.\n- The proposed method links the maximal update and sparse network training, which is an interesting found.", "weaknesses": "- The proposed method is discussed and evaluated only with the random unstructured sparsity pattern, which is hardly used in practice to speed up the DNN training and inference.\n- The theoretical contribution is somewhat weak. The proposed parameterization method is simple extension of maximal update parameterization to the sparse version, which I don't see significant technical improvement. \n- The experimental results should include the accuracy comparison between S$\\mu$PaR and other sparse weight initialization methods (e.g., [1]). Although the proposed method was compared with SP and $\\mu$P, the superiority of S$\\mu$PaR is somewhat predictable since the other two are not designed for the sparse weights. I am curious how much improvement S$\\mu$PaR can make from the previous sparse weight initialization schemes.\n- The scaling factors for the weight update need to be derived for each optimizer, which might limit the usability of S$\\mu$PaR in practice.\n\n[1] Lee, Namhoon, et al. \"A signal propagation perspective for pruning neural networks at initialization.\" arXiv preprint arXiv:1906.06307 (2019).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed and constructive feedback, and your kind words regarding the paper’s writing, experimental results, and core idea.\n\n```\nSuPar is discussed and evaluated only with the random unstructured sparsity pattern, which is hardly used in practice to speed up the DNN training and inference.\n```\nIt is true that the current adoption rate of unstructured sparsity in training and inference is low, as we acknowledge on Lines 35-36 and 73-75 of our submission. While we currently mostly motivate unstructured sparse training on Lines 27-28 and 279-286, we will note in the paper introduction that unstructured sparsity remains a very important technique and intense focus of research for ourselves and other labs, as accelerators such as CPUs, Cerebras, GraphCore, and SambaNova can take advantage of unstructured sparsity during training and inference. Furthermore, GPU and TPU systems support 2:4 block structured sparsity which is quite similar to 50\\% unstructured sparsity.\n\nResearch has shown that the less structure imposed on sparsity masks, the better the performance, making this setting relevant to study in training and inference. For example, SparseGPT [Frantar and Alistarh(2023)], the SOTA LLM pruning technique creates unstructured sparse masks that require hardware acceleration to realize an inference benefit. They report a CPU inference speedup, demonstrating the potential for real-world impact of unstructured sparsity.\n\nAdditionally, in Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\", we test RigL and GMP dynamic sparse training and show that none of SP, $\\mu$P, or S$\\mu$Par achieve transfer of optimal learning rate across sparsity level. We hope this helps address your concerns around applying S$\\mu$Par outside the context of random unstructured sparsity.\n\n```\nThe theoretical contribution is somewhat weak. The proposed parameterization method is simple extension of maximal update parameterization to the sparse version, which I don't see significant technical improvement.\n```\nIt is true that S$\\mu$Par is a simple extension of $\\mu$P. However, we note that we are the first to acknowledge and propose a solution to the systematic misuse of dense hyperparameters in sparse training. On Lines 90-91 and Footnote 1 (page 3) from our submission, we demonstrate how widespread the issue is by providing numerous examples of sparse training works which use dense hyperparameters for sparse training. So, regardless of the perceived significance of technical improvement, the work is important and the potential for impact is fairly high.\n\n```\nWhy does S$\\mu$Par stabilize the sparse network training even after multiple training steps? How is S$\\mu$Par technically different from the previous sparse weight initialization methods?\n```\nWe take your point, and indeed we discuss this on Lines 267-273. $\\mu$P and S$\\mu$Par differ from pure initialization-based methods due to the presence of a learning rate correction. The combination of an initialization and learning rate correction allows S$\\mu$Par to stabilize sparse network training even after multiple training steps, as seen in Figure 5 of our main submission. In Section 1 \"Individual ablations of S$\\mu$Par initialization and learning rate corrections\" from our \"Global Author Rebuttal\", we also show that the S$\\mu$Par initialization adjustment alone is not sufficient to achieve transfer of optimal HPs across sparsity levels.\n\nAdditionally, please see Section 3 \"Downstream Task Evaluations\" from our \"Global Author Rebuttal\" where we provide accuracy comparisons between SP, $\\mu$P, and S$\\mu$Par.\n\nWe also think the comparison across PaI methods is somewhat orthogonal to our work because we are not studying the optimal sparse training recipe (eg. PaI, DST, prune-retrain), but instead how to best control sparse training dynamics.\n\n```\nIs the proposed parameterization method generalizable to any structured sparsity patterns (e.g., channel pruning) for the density-invariant hyperparameters? As mentioned in the paper, the unstructured sparsity is extremely hard to speed up. Discussing S$\\mu$Par on more structured sparsity patterns, such as channel pruning, will make the paper stronger.\n```\n\nGreat question! On Lines 160-161 we remark that S$\\mu$Par should easily extend to random 2:4 structured sparsity as it closely resembles random unstructured sparsity. S$\\mu$Par would also work for random structured pruning of entire neurons at initialization because this case simply reduces to training with a narrower dense model. However, S$\\mu$Par does not transfer to dynamic sparse training because weights become non-Gaussian. Please see Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\" for a discussion on the limitations of S$\\mu$Par in dynamic sparse training. We will update the main body to more thoroughly discuss these possible extensions. Thank you for the feedback.\n\n```\nCan S$\\mu$Par improve the final performance of IMP?\n```\nThis is an interesting question. S$\\mu$Par improves random static unstructured sparse training and IMP rewinds weights back to their initial values while maintaining the same mask. If the IMP mask at initialization still allows the non-zero weights to have a Gaussian distribution, then S$\\mu$Par would apply to this case. Therefore, S$\\mu$Par *could* prevent ``hyperparameter detuning'' in later IMP iterations, and *potentially* improve IMP losses. Given the popularity of IMP/LTH, we think this discussion would make an excellent addition to Section 5 of our paper. Thank you for bringing this up!\n\n```\nDeltaX vs DeltaW notation\n```\n$\\Delta W$ refers to the weight update whereas $\\Delta \\mathbf{X}$ refers to \"the effect of the weight update”. We will update Line 113 accordingly.\n\n[Frantar and Alistarh(2023)] Elias Frantar and Dan Alistarh. 2023. SparseGPT: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Pxian8kd7", "reviewer": "Reviewer_Grts", "summary": "This work (similarly to \\muP method) proposes a specific parameterization for weights, gradients and updates such that the optimal training hyperparameters (i.e. learning rate) are transferrable across different sparsity levels. The approach is validated on the task of language modeling within wide range of sparsities.", "questions": "Does the SμPar transfer to other domains in the same way? For example, one could try pretraining a vision transformer in a setup similar to [3]. \n\nSparsity scaling laws paper [3] showed that given sufficiently long compute, sparse models turn out to be compute-optimal. Given that SμPar finds better hyperparameters, could one expect that higher sparsity levels may be favored compared to the recipe used in the aforementioned work?\n\nIt seems like the value of train loss decreases with the increase of sparsity (i.e., sparse models are better given the same number of parameters), whereas in Figure 10, the performance of SμPar loss seems to be independent of sparsity. I would expect an increase in the quality gap for longer training according to sparsity scaling laws.\n\n---\n[1] Evci, Utku, et al. \"Gradient flow in sparse neural networks and how lottery tickets win.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 36. No. 6. 2022.\n\n[2] Liu, Zhuang, et al. \"Rethinking the value of network pruning.\" arXiv preprint arXiv:1810.05270 (2018).\n\n[3] Frantar, Elias, et al. \"Scaling laws for sparsely-connected foundation models.\" arXiv preprint arXiv:2309.08520 (2023).", "limitations": "See Weaknesses and Questions.\n\nWould be interesting to see whether the introduced parametrization is optimal for training methods with dynamic sparsity (RigL [1], Top-Kast [2], AC/DC [3]). \n\n---\n[1] Evci, Utku, et al. \"Rigging the lottery: Making all tickets winners.\" International conference on machine learning. PMLR, 2020.\n\n[2] Jayakumar, Siddhant, et al. \"Top-kast: Top-k always sparse training.\" Advances in Neural Information Processing Systems 33 (2020): 20744-20754.\n\n[3] Peste, Alexandra, et al. \"Ac/dc: Alternating compressed/decompressed training of deep neural networks.\" Advances in neural information processing systems 34 (2021): 8557-8570.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The proposed approach is theoretically sound and intuitive. \n\n* SμPar appears to produce a stable optimal learning rate with a wide range of sparsities, whereas the values for standard parameterization and dense \\muP have to be tuned according to the level of sparsity. \n\n* SμPar allows one to achieve strong results for larger, outperforming the tuned SP and \\muP configuration, given a small proxy tuned model. With significantly smaller tuning cost one can achieve better results.", "weaknesses": "Whereas all the components in SμPar seem to be important, the importance of each individual parameterization is not studied. It could be the case that SμPar for weights while keeping \\muP parameterization for gradients and learning rate may suffice and produce the same performance. For the sake of completeness, I would suggest ablating individual terms. \nPrior [1], [2] work shows that proper weight initialization (scaled according to sparsity) already improves training dynamics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful review.  It is gratifying to know that you found the method theoretically sound and intuitive, and that the experimental results were a strength of the paper.\n\n```\nWhereas all the components in S$\\mu$Par seem to be important, the importance of each individual parameterization is not studied. It could be the case that S$\\mu$Par for weights while keeping $\\mu$P parameterization for gradients and learning rate may suffice and produce the same performance. For the sake of completeness, I would suggest ablating individual terms. Prior [1], [2] work shows that proper weight initialization (scaled according to sparsity) already improves training dynamics.\n```\nThis is an excellent suggestion! In Section 1 \"Individual ablations of S$\\mu$Par initialization and learning rate corrections\" from our \"Global Author Rebuttal\" we perform the ablation you suggested and show that both the initialization and learning rate correction are required for S$\\mu$Par to achieve transfer of optimal initialization standard deviation or optimal learning rate across sparsity levels. We will include these ablations in the camera version of the paper.\n\n```\nDoes the S$\\mu$Par transfer to other domains in the same way? For example, one could try pretraining a vision transformer in a setup similar to \"Scaling laws for sparsely-connected foundation models\".\n```\n\nThis is a good question because much of the sparse neural network literature has focused on the vision domain. Since ViT shares most of its architecture with language transformers, we believe the extension of $\\mu$P and S$\\mu$Par to ViT should also follow Table 1 from our main submission, but we leave testing it as future work.\n\nHowever, the setup you are referencing also uses gradual magnitude pruning (GMP). In Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\", we show that S$\\mu$Par is not correct for GMP and provide an explanation as to why (continuing our explanation from Lines 274-278).\n\n```\n\"Scaling laws for sparsely-connected foundation models\" showed that given sufficiently long compute, sparse models turn out to be compute-optimal. Given that S$\\mu$Par finds better hyperparameters, could one expect that higher sparsity levels may be favored compared to the recipe used in the aforementioned work?\n```\nThis is a very interesting paper with potentially high impact. On Lines 262-266, we discuss how the setup used in \"Scaling laws for sparsely-connected foundation models\" adjusts the learning rate proportional to sparsity. This correction likely has a similar effect to the S$\\mu$Par learning rate correction however without both an appropriate initialization and learning rate correction, optimal hyperparameter transfer is not guaranteed. Without HP transfer, it is likely that many of the models trained in this paper experience \"de-tuned hyperparameters\" as width and sparsity are varied. As we discuss in Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\", if one could extend S$\\mu$Par for arbitrary sparse algorithms, then that parameterization would likely improve the sparse models in the aforementioned work, especially at high sparsity levels. We leave the development of this method as future work, as stated on Line 278.\n\n```\nIt seems like the value of train loss decreases with the increase of sparsity (i.e., sparse models are better given the same number of parameters), whereas in Figure 10, the performance of S$\\mu$Par loss seems to be independent of sparsity. I would expect an increase in the quality gap for longer training according to sparsity scaling laws.\n```\nIso-FLOP efficiency isn't the focus of our paper. However in Figure 10 we do include the iso-Parameter scaling tests similar to the scaling strategy in \"Scaling laws for sparsely-connected foundation models\" as well as several other papers. However \"iso-Parameter\" doesn't necessarily mean iso-FLOP because increasing $d_\\text{model}$ also increases attention dot product FLOPs, which can't be sparsified. This increase in attention FLOPs is so significant that our 87.5\\% sparse model from Figure 10 has just over double the training FLOPs of the dense baseline, with virtually unchanged loss. Unfortunately, \"Scaling laws for sparsely-connected foundation models\" did not model the effect of attention FLOPs in their scaling law functional forms, instead assuming that sparsity and parameter count alone are sufficient. We suspect that many of the sparse models in that paper use considerably more total training FLOPs than their dense baselines, making for an unfair comparison. It is unclear whether their results hold under more robust FLOP comparisons. For this submission though, we consider the work of beating dense FLOP efficiency with sparse training as future work. We will update our manuscript to reflect the subtleties of FLOP counting in sparse models.\n\n```\nWould be interesting to see if S$\\mu$Par is optimal for DST (eg. RigL, Top-KAST, AC/DC)\n```\nThis is a good suggestion for an experiment! Unfortunately S$\\mu$Par is not optimal for DST methods. We demonstrate this and provide an explanation in Section 2 \"Dynamic sparsity hyperparameter transfer\" from our \"Global Author Rebuttal\". We leave the solution as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for taking the time to read our submission and provide helpful feedback. Please find attached our 1 page PDF containing additional results. We believe these additions help address many reviewer concerns and strengthen our submission. Here we provide a discussion of the results in our 1 page Rebuttal PDF.

# Section 1: Individual ablations of S$\mu$Par initialization and learning rate corrections
Thank you to reviewer Grts for suggesting we individually ablate the effect of the S$\mu$Par initialization and the S$\mu$Par learning rate. In Figure 1 of our Rebuttal PDF (blue and orange boxes), we show that using only the S$\mu$Par initialization in conjunction with $\mu$P ("$\mu$P + S$\mu$Par init only") does not allow for transfer of optimal initialization standard deviation or optimal learning rate across sparsity levels. This result also helps address some of the feedback from reviewer hqe9. We also show that using only the S$\mu$Par learning rate in conjunction with $\mu$P does not achieve transfer either ("$\mu$P + S$\mu$Par LR only"). Therefore, both the S$\mu$Par initialization and learning rate corrections are required to achieve optimal hyperparameter transfer across sparsity levels.

# Section 2: Dynamic sparsity hyperparameter transfer
Every reviewer mentioned dynamic sparsity in some capacity, which motivated us to study it more closely in the context of S$\mu$Par. In Figure 1 of our Rebuttal PDF (green box), we test the transfer of optimal learning rate across sparsity levels for two popular dynamic sparse training methods: Rigging the Lottery (RigL) [Evci et al.(2020)] and Gradual Magnitude Pruning (GMP) [Zhu and Gupta(2017)]. We show that none of SP, $\mu$P, or S$\mu$Par achieve transfer of optimal learning rate across sparsity levels. For SP and $\mu$P we see that higher sparsity levels have higher optimal learning rates. This is because sparsity reduces activation and gradient scales such that a larger learning rate is needed to counteract this. S$\mu$Par sees the opposite trend where higher sparsity levels have lower optimal learning rates, indicating that S$\mu$Par is ``overcorrecting''.

On Lines 274-278 of our submission we mention that dynamic sparse methods can make updates to the weight mask such that the distribution of unmasked/non-zero weights changes to something non-Gaussian, which prevents S$\mu$Par from being mathematically correct. Compared to random pruning, a mask obtained from magnitude pruning will better preserve the size of activations and gradients seen in the dense network. Since S$\mu$Par assumes weights are drawn from a Gaussian distribution, S$\mu$Par ends up ``overcorrecting'' the initialization and learning rate. In future work it would be impactful to develop a parameterization which generalizes S$\mu$Par to work for an arbitrary sparse training algorithm.

# Section 3: Downstream Task Evaluations
Thank you to reviewer F4Ay and 6wqj for pointing out the limitations of limitations of relying on loss alone for evaluating LLMs. We recognize this shortcoming since ultimately LLMs are being trained for use in downstream tasks. Following their suggestion, in Table 1 of the Rebuttal PDF, we evaluated the models from Figure 1 of our submission to provide a head-to-head comparison between SP, $\mu$P, and S$\mu$Par. Results across pretraining loss and average downstream task accuracy consistently show that S$\mu$Par models achieve superior performance compared to SP and $\mu$P. We measured accuracy on five downstream tasks: ARC-easy, lambada, RACE, PIQA, and BoolQ, which collectively test for common sense reasoning, world knowledge, and reading comprehension. We also specifically chose tasks that are easy enough for even extremely sparse models to significantly outperform random chance. 

[Evci et al.(2020)] Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. 2020. Rigging the lottery: Making all tickets winners. In International conference on machine learning. PMLR, 2943–2952.

[Zhu and Gupta(2017)] Michael Zhu and Suyog Gupta. 2017. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.01878 (2017).

Meta-review: This paper proposes a specific parameterization for weights, gradients and updates such that the optimal training hyperparameters (i.e. learning rate) remain the same across different sparsity levels or widths. The approach is validated on the task of language modeling across various levels of sparsities. Some main concerns about the novel contributions, evaluation metrics (loss), and dynamic sparse training, etc. During the rebuttal period, the authors clarify with new results. Most of reviewers are satisfied with the authors rebuttal and propose to accept the paper. Although one reviewer still thinks the paper needs more polishing work with another round of review, this reviewer also does not object to accept if all the others agree to accept the paper. Therefore, we propose to accept the paper, suggesting the authors to include all the clarifications and new results in the final version.
Decision: Accept (poster)