AutoSparse: Towards Automated Sparse Training
Authors: Abhisek Kundu
Naveen Mellempudi
Dharma Teja Vooturi
Bharat Kaul
Pradeep Dubey
Venue: Submitted to ICLR 2023 (2023)
Keywords: sparsity
sparse training
deep learning

Abstract: Sparse training is emerging as a promising avenue for reducing the computational cost of training neural networks. Several recent studies have proposed pruning methods using learnable thresholds to efficiently explore the non-uniform distribution of sparsity inherent within the models. In this paper, we propose Gradient Annealing (GA), a gradient driven approach where gradients to pruned out weights are scaled down in a non-linear manner. GA eliminates the need for additional sparsity-inducing regularization by providing an elegant trade-off between sparsity and accuracy. We integrated GA with the latest learnable threshold based pruning methods to create an automated sparse training algorithm called AutoSparse. Our algorithm achieves state-of-the-art accuracy with 80% sparsity for ResNet50 and 75% sparsity for  MobileNetV1 on Imagenet-1K. AutoSparse also results in 7× reduction in inference FLOPS and > 2× reduction in training FLOPS for ResNet50 on ImageNet at 80% sparsity. Finally, GA generalizes well to fixed-budget (Top-K, 80%) sparse training methods, improving the accuracy of ResNet50 on Imagenet-1K, to outperform TopKAST+PP by 0.3%.

Under review as a conference paper at ICLR 2023
AUTOSPARSE: TOWARDS AUTOMATED SPARSE TRAIN- ING
Anonymous authors Paper under double-blind review
ABSTRACT
Sparse training is emerging a promising avenue for reducing the computational cost of training neural networks. Several recent studies have proposed pruning methods using learnable thresholds efficiently explore the non-uniform distri- bution of sparsity inherent within the models. In this paper, we propose Gradient Annealing (GA), gradient driven approach where gradients of pruned out weights are scaled down in non-linear manner. GA provides an elegant trade-off' between sparsity and accuracy without the need for additional pparsity-inducing regulariza- tion. We integrated GA with the latest learnable threshold based pruning methods to create an automated sparse training algorithm called AutoSparse. Our algorithm accuracy with 80% sparsity for ResNet50 and 75% spar- sity for MobileNetVi on Imagenet- AutoSparse also results 7x reduction inference FLOPS and 2x reduction in training FLOPS for ResNet50 Ima- geNet at 80% sparsity. Finally, GA generalizes well to fixed-budget (Top-K, 80%) sparse training methods, improving the accuracy of ResNet50 to outperform TopKAST+PP by 0.3%. MEST (SotA method for fixed- budget sparsity) achieves comparable accuracy as AutoSparse at 80% sparsity, however, using 20% more training FLOPS and 45% more inference FLOPS.
achieves
Imagenet- IK,
INTRODUCTION
Deep learning models have emerged the preferred solution for many important problems in the domains of computer visiorHe et al. (2016); Dosovitskiy et al. (2021), language modeling Brown et al.
(2000), recommender system/Naumor et al. (2019) and reinforcement learningSilver Models have grown larger and more complex over the years, as they are applied to increasingly difficult problems on ever-grov wver-growing datasets. In addition, DNN models are designed to operate in overparameterized regime Arora et al. (2018); Belkin et al.] (2019); Ardalani et al. (2019) to facilitate easier optimization using gradient descent based methods. As a consequence, computational costs of performing training and inference tasks state-of-ther -he-tttt art models has been growing an exponential ratdAmodes to noise during training reduced precision training Micikeviii et al. (201)): Wang et al. (2018); Sun et al. (2019) have successfully exploited this aspect of DNNs to speed up training and inference tasks significantly. Today, state-of-the-a training hardware consists of significantly more reduced-precision FLOPs compared to traditional FP32 computations.
al. (2017).
Hernandez) The excess model capacity also makes DNNs more resilient
Sparsity is another avenue for improving the compute efficiency by exploiting excess model capacity, thereby reducing the number of FL.OPs needed for each iteration. Several studies have shown that while overparameterized model helps ease the training, only a fraction of that model capacity is needed to perform inference Li et al. (2020); Han et al. (0155); Li et al. [2020)[Narang et al. (2017); Ström 1997): Gale et al. (2019). A wide array of studies have also proposed methods to prune dense networks to produce sparse models for inference
to-se----- Molchanov et al.] (2017), Zhu & Gupta (2017): Frankle & Carbin )0 Renda et al (2020). More recently, there is a growing interest in sparse-to-sparse Frankle & Carbin (2019 Mostafa & Wang (2019): Bellec et al. (2018); 1); Dettmers & Zettlemoyer (2019): Jayakumar et al. (2021) Zhang et al. (2022); Schwarz et al. 2021)> Yuan al. (2021) methods training models with end-to-ens sparsity to reduce the computational costs of training. This paper presents techniques to improve and generalize
Evci et al. (2021) 1; Lee (2021):
sparse training methods for easy integration into different training workflows.

---

Under review as a conference paper at ICLR 2023
Sparse training methods can be broadly divided into two categories a) deterministic pruning methods, that initialize the model with a desired fixed sparsity budget at each layer and enforce it throughout the training cycle, b) learnable threshold pruning methods attempt to discover the sparsity distribution within the model by learning layer-wise threshold parameters. While these latter methods can aim for desired sparsity level by selecting appropriate combination of initialization and hyper-parametent the final model sparsity may not be exactly what was desired- - hence they are non-deterministic. Please refer to Hoefler et al. (202T) for detailed categorization and discussion on various sparsification methods.
Deterministic pruming methods expect a prior knowledge of how much sparsity can be extracted out of any given model. This sparsity budget is often determined by trial- and-error or extrapolated from previously published studies. Once the sparsity budget is determined, choice must be made between a uniform or a non-uniform distribution of sparsity across the
Deterministic Pruning:
layers. Majority of the methods in this category FFrnnlle & Carbin (2019); Bellee et al. (2018);
Evei et al. (2021); Lee (202l); Jayakumar et al. (2021): y: Zhang et al. (2022) Schwarz et al. (022
Zhou et al. (202T): Yuan et al. (2021) opt for unitorm sparse distribution because it requires fewer
hyperparameters
subset of these methods Jayakumar et al. (2021); Zhang et al. (2022): |Schwarz
et al. (202I) maintain first and last layers in dense while the sparsity budget is uniformly distributed
across rest of the layers. Fewer methods in this category Mostafa & Wang (20T9); Dettmers &
Zettlemoyer (2019) use non-uniform distribution across layers using dynamic weight reallocation heuristics. Non-uniform distribution allows more degrees of freedom to explore ways improve accuracy at any given sparsity budget. The best performing methods in this category are the ones that skip pruning the first and last layers.
1.1 LEARNABLE THRESHOLD PRUNING
Learnable threshold pruning methods offer two-fold advantage over deterministic pruning methods, 1) computationally efficient, as the overhead of computation (e.g. choosing Top-K largest) used for deriving threshold values at each layer is eliminated, 2) learns the non-uniform sparsity distribution inherent within the model automatically, producing a more FLOPs-efficient sparse model for inference. For example, 80% sparse ResNet50 produced via fixed- Exdget method MEST requires 50% more FLOPS than learned sparsity method AutoSparse (discussed later) as compute profile for various layers is non- non-uniform (Figure Iap.
Current state-of-the-art methods in this space Liu et al. (2020); Kusupati et al. (2020) rely on L.2 regularization based approaches to guide threshold updates and penalize pruned weights. Dynamic Sparse Training Liu ized to 'zero' and pushes the small thresholds up using an exponential loss function added to L2 regularization term. Soft Threshold Reparameterization Kusupati et al. (2020) initializes threshold parameters with large negative values and controls the induction of sparsity using different weight decay (A) values for achieving different sparsity levels using L2 regularization. Sparsity vs accuracy trade-off is a challenge and is discussed below.
Liu et al. (2020) is sparse training method that starts with threshold values initial-
Exploring Accuracy Vs. Sparsity Trade-Off:
Deterministic pruning methods, not withstanding their limitations, offer a consistent level of sparsity throughout the training which in turn can lead to predictable performance improvements. For learnable threshold methods, the performance can be measured in the form of reduction in training FLOPs measured across the entire duration of the training. In order to meet this goal, learnable threshold methods must have the ability to induce sparsity early in the training and provide algorithmic means to explore the trade-off increase model sparsity while also reducing accuracy loss. Empirical studies on aforementioned method& et al. (2020) Kusupati et al. (2020) have indicated that L2 regularization based approach offers at best, a weak trade-off of better accuracy at the expense of lower average sparsity. We also found these methods to be susceptible to runaway sparsity (e.g. hit 100% model sparsity), if higher levels of sparsity were induced right from the start of the training. To mitigate this problem. DST Liu et al./ (2020) implements hard upper limit checks (e.g. 99%) on sparsity to trigger a reset of the offending threshold and all the associated weights to prevent loss of accuracy. Similarly, STR Kusupati et al.) (2020) methods uses combination of small initial threshold values with an appropriate to delay the induction of sparsity until later in the training cycle (c.g. 30 epochs) control unfettered growth of sparsity.
2

---

Under review as a conference paper at ICLR 2023
Motivated by the above challenges, we propose Gradient Annealing (GA) method, to address the aforementioned issues related to training sparse models. When compared existing methods, GA offers greater flexibility explore the trade-off between model sparsity and accuracy, and provides greater stability by preventing divergence due to runaway sparsity. We also propose a unified training algorithm called AutoSparse, combining best of learnable threshold methods KKsupati et al. (2020) with GA that attempts to pave the path towards full automation of sparse-to-sparse training. Additionally, we also demonstrated that when coupled with deterministic pruning methods (TopKAST), Gradient A Annealing can extract better accuracy at fixed (80%) sparsity budget. The key contributions of this work are as follows:
We present novel Gradient Annealing (GA) method (+ and more accurate gradient approximator than STE (Bengio et al. (2013) and ReLUU (e.g. STR Kusupati et al. (2020).. For training end-to-end sparse models, GA provides greater flexibility to explore sparsity and accuracy trade-off than other methods.
a) which is generalized
We propose AutoSparse, a unified algorithm that combines learnable threshold methods with Gradient Annealing to create an automated framework for end-to-end sparse training. Our algorithm outperformed state-of-the-art methods by achieving better accuracy while maintaining consistently high sparsity throughout the training for ResNet50 and MobileNetV1 on Imagenet- IK dataset.
AutoSparse achieves best FLOPs efficiency for inference and training among methods with compa- rable sparsity and accuracy.
We demonstrate the efficacy of Gradient Annealing general learning technique independent of AutoSparse by applying to TopKAST method to improve the Top-1 accuracy of ResNet50 by 0.3% on ImageNet- 1K dataset using same fixed sparsity budget of 80%.
2 GRADIENT ANNEALING (GA)
A typical pruning step of deep networks involves masking out weights that are below some threshold T. This sparse representation of weights benefits from sparse computation forward pass and in computation of gradients of imputs. We propose the following pruning step, where w is weight and a threshold that can be deterministic (c.g., TopK magnitude) or learnable:
(sparse) v sign(w) ha (lw) - T)
x, 0,
@ha(1) Ox
11, a,
>>> 0 x S 0
Forward pass (x:)
(Proxy) Gradient
(1)
where 0 < < 1. S is O if |m| is below threshold T. Magnitude-banse pruning is greedy, temporal view of parameter importance. However, some of the pruned-out weights (in early training epochs) might turn out to be important in later epochs when more accurate sparse pattern emerges. For this, ho (-) in eqn ( allows the loss gradient to flow to masked weights in order to avoid permanently pruning out some important weights. The proposed gradient approximation is inspired by the Straight Through Estimator (STE) Bengio et al. (2013) which replaces zero gradients of discrete sub-differentiable functions by proxy gradients in back-propagation. Furthermore, decay this a as the training progresses. We call this technique the Gradient Annealing.
We anneal at the beginning of every epoch and keep this value fixed for all the iterations in that epoch. We want to decay a slowly in early epochs and then decay it steadily. For this, we compare several choices for decaying a: fixed scale (no decay), linear decay, cosine decay (same learning rate (2)) sigmoid decay (defined in (3)) and Sigmoid-Cosine decay (defined in (4)). For sigmoid decay in (3) Lo -6 and L, 6. For total epochs T, scale for in epoch is
Cosine-Decay (1, 7)
C;
=
(1 + cosine (T i/T))/2
(2)
Sigmoid-Decay(<4 7)
8; 1 sigmoid Lo + (In La) i/TT
Sigmoid-CCosin--eecay//
= max(si,
Figure I shows the effect of various linear and non-linear annealing of a dynamic sparsity. Fixed scale with no decay (STE) does not give us a good control of dynamic sparsity. Linear decay is better than this but suffers from drop in sparsity towards the end of training. Non-linear decays in eqn
3

---

Under review as a conference paper at ICLR 2023
4 provide much superior trade-off between sparsity and accuracy. While eqn and eqn (4 show very similar behavior, sharp drop of eqn (3) towards the end of training push up the sparsity a bit (incurring little more drop in accuracy). These plots consistent with our analysis of convergence of GA in eqn (5). Annealing schedule of a closely follows learning rate decay schedule.
Analysis of Gradient Annealing
Here we analyze the effect of the transformation (-) on the convergence of the learning process using simplified example follows. Let |w| T, (v), optimal weights be w' and optimal threshold be T", i.e., v" |w1 - T Let us consider the loss function as
min (t)) 0.5 (ha 8²)²
v
and let (v) denote the gradient
Oh. (w) dv
We consider the following cases for loss gradient for v.
( 0 if (a( =
(ha(v).(B v*) = =1.(v --) if v v> 0 and V- >
ac Ov = (ha)())(ha(()) -b0))
(ha(v)(( (v*)) = (1-(v+lo")) if v 0 and 0- S 0
(5)
(ha(u) (-v") =a-(-v") if 2 0 and 8*
@ha(v) ((v")) = -(="(=))) if v U and oz S
Correct proxy gradients for (-) should move towards v² during pruning opposite direction of gradient for gradient descent) and stabilize it at its optima (no more updates). Therefore, Ōh, (v) > should be satisfied for better convergence of v² Our ha satisfies this condition for > 0. Furthermore, for v > 0, gets updated proportional to v., i.e., how far e is from v* As training progresses and gets closer to v", receives gradually smaller gradients to finally converge to v². However, for S O, receives gradient proportional to magnitude of a vv,, irrespective of how close is to v², Also, note that we benefit from sparse compute when
We set initial T high in order to achieve sparsity from early epochs. However, this likely leads to a large number of weights following condition 4 in eqn ()) Fixed, large a (close to 1) makes large correction to and moves it to ex quickly. Consequently, moves from condition 4 to condition 2, losing out the benefit of sparse compute. A lower a 'delays' this transition and enjoys the benefits of sparsity. This is why we choose < rather than identity STE as proxy gradient (unlike Tang et al. (2022). ). However, as training progresses, more and more weights move from condition
condition 2 leading to drop in sparsity. This behavior is undesirable to reach target sparsity at the end of training. order to overcome this, we propose to decay a with training epochs such that we enjoy the benefits of sparse compute while being close to v². That is, GA provides more controlled and stable trade-off between sparsity and accuracy throughout the training.
Note that, GA is applicable when compute loss gradients for superset of active (non-zero) weights that take part in forward sparse computation using gradient descend. For an iteration f, let the forward sparsity be S. If a 0, then need to calculate gradient for only those non-zero weights as other weights would not receive gradients due to ReLU STE. order to benefit from such computational reduction, we can set 0 after several epochs of a annealing.
AUTOSPARSE: SPARSE TRAINING WITH GRADIENT ANNEALING
AutoSparse is the sparse training algorithm that combines the best of learnable threshold pruning techniques Kusupati et al. [2020): Liu et al. (020)) with Gradient Annealing (GA). AutoSparse meets the requirements necessary for efficient training of sparse neural networks, outlined in Section [.]
Learnable Pruning Thresholds Eliminate threshold computation, reduce sparsification over- head compared deterministic pruning JathounJayakumar al. (2021. Learn the non-uniform distribution of sparsity across the layers.
Sparse Model Discovery: Discover elegant trade-off between model accuracy vs. level of sparsity by applying Gradient Annealing method (as shown in Figure [ Produce a sparse model at the end of the training with desired sparsity level guided by the hyper-parameter a.
4

---

Under review as a conference paper at ICLR 2023
140
90
AutoSperse: FLOPS
100
STR: 0.18x FLOPS
70
FLGPS
60
MEST 80%: 0.21x FLOPS
40
40
fied stale
20
linear anneal
Cog-Cos arneal
11
21
Layers
31
41
51
Epochs
(a) Inference FLOPS for 80% sparse ResNet50 pro- duced by ixed-budger and leamned sparsity methods
(b) Sparsity achieved for various gradient annealing
Figure Sparse ResNet50 training on ImageNet
Accelerate Training/Inference Reduce training FLOPs by training with sparse weights from scratch, maintaining high levels of sparsity throughout the training, and using sparsity in both forward and backward pass. Produce FLOPS-efficien sparse model for inference.
Previous proposals using leanable threshold methods such as Liu et al. (2020) and Kusupati et al. (2020) address the first criterion but do not effectively deal with accuracy sparsity trade-off as they are outperformed by our method. These methods also do not accelerate training by reducing FLOPS as effectively our method. Deterministic pruning methods such as Jayakumar al. (202T)
Evci et al. [202T); Yuan et al. (202T) address the third criterion of accelerating training, but incur higher sparsfication overheads for computing threshold values and cannot automatically discover the non-linear distribution of sparsity. This leads to sub-optimal sparse models for inference (Figure La).
Formulation: Let D == {(x Rdd Yr E [R)} be the observed data, W be the learnable network parameters, L be loss function. For an L-layer DNN, W is divided into layer-wise trainable parameter tensors, [We] As various layers can have widely different number of parameters and also unequal sensitivity to parameter alteration. we use one trainable pruning parameter, s, for each layer E, i.e., $ = ..., Sr. the vector of trainable pruning parameter. Let g - R be applied wlement-wise. For layer f, Ti g(sz) is the pruning threshold for W,. We seek to optimize:
min L(Sh.. (W,s), ((S,...(W,ss). D)
(6)
W.s
where, function Sn., g: parameterized by ha E TR IR that is applied element-wise.
We = Sh..g (We, = sign We) ha ((Wel - g(x))
Gradient annealing is applied via (+) as discussed earlier.
Sparse Forward Pass: At iteration f, for layer e, sparse weights are Wil() = Sh.. (W()) ( as defined in eqn 7 Let the non-zero (active) set of weights A
w((h) > 0}. For simplicity, let us drop the subseript notations. W is used in the forward pass as Y X @ W where @ is tensor MAC operation, e.g., convolution. Let A denote the fraction of non-zero elements of W belonging to A, i.e, out forward sparsity is - A. W is also used in computation of input gradient during backward pass. Every iteration, we update W and construct W from updated W and learned threshold.
Sparse Compute of Input Gradient: For an iteration and layer (we drop the subscripts E for simplicity), output of sparse operations in forward pass need not be sparse, i.e., Y @ W is typically dense. Consequently, gradient of output Vy is also dense. We compute gradient of input
Vx W & Vy. Computation of V V is sparse due to sparsity in W.
Sparse Weight Gradient: Gradient of weights Vw is computed as X @ Vy. Note that, for forward sparsity S, 0 implies weight gradient sparsity S as no gradient flows pruned weights. We can have a hyperparameter that specifies at which epoch we set a = O, to enjoy benefits of sparse weight gradients. However, need to keep a 0 for several epochs to achieve high accuracy results, losing the benefits of sparse weight gradient. In order to overcome this, we propose the following for the epochs when o 0.
5

---

Under review as a conference paper at ICLR 2023
We make sparse V if we compute loss gradient using subset B of W.
B = {i: : W, E Top B)}, B2A A
(8)
where B is a superset of A and TopK A) picks indices of largest magnitude elements from W. This constitutes our weight gradient sparsity 1 1-B. B.
We apply gradient annealing on set \ A, i.e. gradients of weights in B \ are decayed using a.
Note that, a = 0 implies B = A. Gradient computation for parameters W and are in Appendix.
4 RELATED WORK
Learnable Threshold Methods
STR Kusupati al. (2020) is state-of-the-an method that learns pruning thresholds along with weights. STR prunes weights that are below the threshold using ReLU as a mask function. Setting = 0 7 reproduces the formulation of STR (however, our method learns completely different distribution than that of STR). This = 0 implies identical forward and backward sparsity which are computationally beneficial. However, this prevents gradients from flowing to pruned out weights resulting in a sub-optimal trade-off between sparsity and accuracy. This forces STR to run fully dense training for many epochs (implemented by initializing threshold parameter 8 to a very large negative value, e.g., -3200 so that ssgmoid(s) Re O, and weight decay s takes many epochs to make sigmoid(s) large enough to induce sparsity. Note that, extracting sparsity from early epochs using STR results in run-away sparsity. Using gradient annealing technique, we are able to extract sparsity from early epochs to maintain high average sparsity throughout the training, yet achieve SOTA accuracy.
DST et al. (2020) is also a sparse training algorithm based on learned weights and thresholds, where binary weight mask is created by suppressing the weights that are below the threshold (using step function). they impose exponential decay of thresholds as regularizer. A hyperparameter controls the amount of regularization, leading to a balance between sparsity and accuracy. In order to reduce the sensitivity of the hyperparameter, they manually reset the sparsity if it exceeds some predefined limit (thus occasionally falling into dense training regime). Further, they approximated the gradient of pruning step function by linearly scaling up the gradient of the elements that are close to the pruning threshold, and scaling down (fixed scale) the gradient of other elements. This helps some of the pruned out elements to receive loss gradient.
SCL, (Tng et al. (2022) learns both network weights as well as mask parameters during training. These masks are binarized during forward pass to represent network sparsity. This mask information, along with a decaying connectivity hyperparameter are used as a sparsity-inducing regularizer in the objective function. During training, the mask is learned as dense parameter tensor. This increases the effective model size during training, which might create overhead moving parameters from memory. LTP Azarian et al. (202T) learns the pruning thresholds using soft pruning and soft Lo regularization where sigmoid is applied on transformed weights and sparsity is controlled by a hyper-parameter Savarese et al. (2021) used sigmoidal soft-threshold function as sparsity-inducin regularization.
Deterministic Pruning Methods
GMP Zhu & Gupta (017)) suggests to gradually increase the number of removed weights until desired sparsity is reached. RigL.Evci et al. (022)) proposes to prune out a fraction of weights, and activates/revives new ones iteratively using infrequent full gradient calculations. Such restriction of gradient flow causes more accuracy drop.
Jayakumma et al.] (202T) always prunes Top-K weights. but updates superset of active weights based on gradients so that the pruned out weights can be revived. Moreover, they penalize the non-active weights by factor inversely proportional to density. This way, the active set of weights becomes stable with epochs as small fraction of pruned out weights become active again. ITO et al. (2021) maintains a fixed amount of sparsity throughout training. They use gradient based weight regrowth used in RigL and SNFS Dettmers & Zettlemoyer (2019). Applying ITOP in RigL grow stage, i.e., exploring new weights based on (full) gradient information in Rigl grow stage, RigL+ITOP achieves higher accuracy. PowerPropagation Schwarz et al. )021) is technique transform weight distribution of trained weights. As an effect, they observe that weights initialized close to 0 are likely to be pruned out, and weights are less likely to change sign. PP, when applied on TopKAST, i.e.,
= u|o[--1 I, s.t. it creates heavy-tailed

---

Under review as a conference paper at ICLR 2023
TopKAST+ PP, improves the accuracy of TopKAST. OptG Zhang et al. (2022) learns both weights and a pruning supermask in a gradient driven manner. They argued in favour of letting gradient flow to pruned weights that it solves the 'independence paradox' problem that prevents from achieving high-accuracy sparse results. However, they achieve a given sparsity budget by increasing sparsity according some sigmoid schedule (sparsity is extracted only after 40 epochs). This suffers from larger training FLOPs count. MEST (Yuan et al. (202I) always maintains fixed sparsity in forward and backward pass by computing gradients of survived weights only. For better exploration, they remove some of the least important weights (ranked proportional to magnitude plus the gradient magnitude) and introduce same number of random 'zero' weights to maintain the sparsity budget. Similar to RigI., they also need to train a lot longer (250 epochs vs 500 epochs in RigL). Gradmax (Evei et al. (2022) proposed to grow network by adding more weights gradually with training epochs in order to reduce overall training FLOPS. SWAT (Raihan & Aamodi (2020) sparsifies both weights and activations by keeping TopK magnitudes in order to further reduce the FLOPS.
EXPERIMENTS
5.1 EXPERIMENTAL. SET UP
Vision Models: ImageNet-IK IK (Deng et al. (2009) )) is widely used large-scale image classification dataset with IK classes, We show sparse training results on ImageNet- IK for two popular CNN architectures: ResNet50 He al. (2016) and and MobileNetV1 [Howard et al. (2017). to demonstrate the generalizability of our method. For AutoSp training, we use SGD as the optimizer, momentum 0.875, learning rate (max) 0.256 using cosine annealing with warm up of 5 epochs. We run all the experiments for 100 epochs using a batch size 256. We use weight decay A = 0.00003051 75781 25 (picked from STR Kusupati al. (2020)). label smoothing 0.1, So 55.
We presented our results
only using Sigmoid- decay of a (defined in eqn
Language Models: We choose Transformer models Vaswani et al. (2177) for language translation on WMT14 Singlish-Geemann data. We have 6 encoder and 6 decoder layers with standard hyperparameter setting: optimizer is ADAM with betas (0.9, 0.997). token size 10240, warm up 4000, learning rate 0.000846 that follows inverse square root decay. We apply AutoSparse by introducing a learnable threshold for each linear layer, and we initialize them as = -7.0. Also, initial a = 0.4 is annealed according to exponential decay as follows. For epoch and , > 0 (we use 3 = 1):
Exponential-Decay (t: B) (B-c)
We keep the first and last layers of transformer dense and apply AutoSparse to train it for 44 epochs.
We repeat the experiments several times with different random seeds and report the average numbers. Notation:
We define the following notations that are used in the tables. 'Base:: Dense baseline accuracy, 'Sopl(S)': Top-1 accuracy for sparse models, 'Drop%': relative drop in accuracy for sparse models from Base, '5%;: percentage of model sparsity. "Train F': fraction of training FLOPs comparing to baseline FLOPs, 'Test F": fraction of inference FLOPs comparing to bsaeline FLOPs, BBack 5%': : explicitly set sparsity in weight gradients (B in eqn R 'BLEU(S):: BLEU score for sparse models. Smaller values of 'Train F' and "Test F' suggests larger reduction in computation.
5.2 EFFICACY GRADIENT ANNEALING
Comparison with Other Learnable Threshold Methods:
We compare AutoSparse results with STR Kusupati al. (2020) and DST Liu et al. (2020). Lack of gradient flow to pruned out elements prevents STR to achieve the optimal accusiyy trade-off. For example, they need dense training for many epochs in order to achieve high accuracy results, losing out the benefits of sparse training. Our gradient annealing overcomes such problems and achieves much superior sparsity-accurace trade-off. In Table [] we achieve better accuracy than STR for ResNet50 at various sparsity levels 80%, 90%, and 95%, while gaining in FLOPs reduction for both training and inference. Also, Table 3 shows our superior accuracy for MobileNetV1. Similarly, method achieves higher accuracy than DST for both 80% and 90% sparsity budget for ResNet50 (Table However, DST uses separate sparsity-indueing regularizer, whereas our gradient annealing itself does the trick. SCI. Tang et al. (022) is another dynamic sparse method that learns both
7

---

Under review as a conference paper at ICLR 2023
Method
Base
TopI(S)
Drop% S%
Train
Test F
comment
RigL
76.8
74.6
2.86
80
0.33
0.22
fixed sparsity budget
*SWAT-U
76.8
75.2
2.08
80
0.24x
0.22
TopK weight, act
GMP
77.01
75.6
1.83
80
0.2x
fixed sparsity budget
TopKAST
76.8
75.7
0.94
0.48
0.22x
fixed sparsity budget
+PP
76.8
76.24
0.73
80
0.48
0.22x
fixed sparsity budget fixed sparsity budget
TOpKAST
76.8
76.47
0.43
80
0.48x
0.22x
MEST
76.9
76.71
0.25
80
0.57x
0.21x
fixed sparsity budget
DST
74.95
74.02
1.24
80.4
0.15x
learnable sparsity
STR
77.01
76.19
1.06
79.55
0.54x
0.18x
learnable sparsity
AutoSparse
77.01
76.77
0.31
79.67
0.46x
0.14x
0u=:75,==0 epochh9
AutoSparse
77.01
76.59
0.55
80.78
0.36x
0.14x
01= 00==8,0=0@@pochh0
MEST +EM
76.9
75.91
1.29
90
0.25x
0.11x
fixed sparsity budget
OptG
77.01
74.28
3.55
DST
74.95
72.78
2.9
90.13
0.087x
learnable sparsity
STR
77.01
74.31
3.51
90.23
0.38x
0.083x
learnable sparsity
AutoSparse
77.01
75.9
1.44
85.1
0.37x
0.096
0o=99,==0@epoch00
AutoSparse
77.01
75.19
2.36
89.94
0.36
0.081x
00==9,==0@epoch55
STR
77.01
70.4
8.58
95.03
0.246x
0.039
dense to sparse
AutoSparse
77.01
70.84
8.01
95.09
0.182x
0.036
Table I: ResNet50 on ImageNet:
Comparing accuracy, sparsity and the FLOPS (dense 1x) for training and inference for selected sparse training methods. TopKAST*: TopKAST with 0% backward sparsity. TopKAST +GA: TopKAST* with Gradient Annealing boosts the accuracy despite having same training/inferene FLOPS. For AutoSparse 79.67% sparsity, ag=0.755 is decayed till epoch and then set to (implying ~ 80% forward and backward sparsity after epoch 90). Similarly, for AutoSparse 95. 09% sparsity, 0y=0.8 8 is decayed till epoch 20 and then set to 0. Standard deviation results for AutoSparse is less than 0.06.
Method
S%
Topl(S)
Drop%
Train F
Test F
Back(S)
comment
AutoSparse
83.74
75.02
2.58
0.328x
0.128x
50
Og=1.0, 80=-8, w grad 50% sparse
TopKAST
80
75
2.34
0.32x
0.22x
50
fwd & in grad 80%, grad 50% sparse
Table 2: ResNet50 on ImageNet: AutoSparse with explicitly set sparsity for weight gradient. For AutoSparse 83.74 74% sparsity, on 1 is decayed till epoch 100 resulting in identical sparsity for forward and input gradients, along with 50% sparsity for weight gradient throughout the training (involves invoking TopK method).
weights and sparsity mask parameters using explicit regularizer with the objective function. They report 0.23% drop in accuracy 74% sparsity for ResNet50 on ImageNet. However, their much lower dense baseline, lower reported sparsity and lack of training FLOPs make it harder for direct comparison (inference FLOPs 0.21 x baseline). LTP produces 89% sparse ResNet50 that suffers from 3.2% drop in accuracy from baseline (worse than ours). Continuous Sparsification Saaarese et al.) (2021) induces sparse training using soft-thresholding as a regularizer. The sparse model produced by them is used to test Lottery Ticket Hypothesis (retrained). Lack of FLOPs number makes it harder to directly compare with our method. GDP (Guo et al. (202I) used FLOPS count as a regularizer to achieve optimal accuracy compute trade-off (0.4% drop in accuracy with 0.49x training FLOPS. Our method achieves slightly higher accuracy with lower FLOPS count.
Comparison with Deterministic Pruning Methods:
RigL Evei et al. 202T) activates/revives new weights iteratively using infrequent full gradient calculation. Such restriction of gradient flow causes accuracy drop: 80% sparse model loses
8

---

Under review as a conference paper at ICLR 2023
Method
Base
TS)
Drop%
S%
Train F
Test F
comment
STR 71.95
68.35
5
75.28
0.37
0.18x
dense to sparse
AutoSparse
71.95
70.1
2.57
75.1
0.51x
0.22x
a + 0
STR 71,95
64.83
9.9
85.8
0.32x
0.1x
dense to sparse
AutoSparse 71.95
64.87
9.84
86.36
0.25x
0.1x
a=0 @ epoch 20
AutoSparse 71.95
64.18
10.8
87.72
0.22x
0.08x
a=0 @ epoch 20
Table 3: MobileNetV1 ImageNet:
Comparing accuracy, sparsity and FLOPS (dense 1 1x) for training and inference for selected sparse training methods. AutoSparse achieves significantly higher accuracy for comparable sparsity. For AutoSparse 75.1% sparsity, a 0.4 is decayed till epoch 100, resulting in identical sparsity for forward and input gradient computation. For AutoSparse (87.7%%866. .3%)) sparsity, (a 0.6,a 0.8) decayed till epoch 20 and then set to 0, which implies identical forward and backward sparsity after epoch 20.
2.86% top-1 accuracy from dense baseline (table D ITOP Liu et al. 2021 maintains a fixed amount of sparsity throughout training. They use gradient based weight regrowth used in Rigl. and SNFS Dettmers & Zettlemoyer (2019). Applying ITOP in RigL. grow stage, i.e., exploring new weights based on (full) gradient information in Rigl grow stage, RigL+ITOP achieves accuracy 75.84 for sparsity budget 80% ~ 1.25% off from baseline. OptG Zhang et al. (2022) learns both weights and a pruning supermask in gradient driven manner where pruned weights also receives loss gradient. AutoSparse is better than their accuracy at 90% sparsity budget, despite achieving sigmificantly higher average sparsity. PowerPropagation Schwarz al. (2021) transforms weight as w u|u|--1 s.t. it creates heavy-tailed distribution of trained weights that are more amenable to pruning. PP, when applied on TSPKAST, i.e., TSPKAST + PP, improves on TopKAST and achieves 0.73% drop in accuracy for 80% sparsity budget and 2% drop in accuracy for 90% sparsity budget. Gradient Annealing, when applied with TopKAST method (without additional regularization) improves these accuracies while maintaining similar training and inference FLOPs. SWAT-U sparsifies models in the forward pass and activation the backpass to reduce the training FLOPs. For models with ReLU activation, we typically have 50% default sparsity. This would lead to up to 2x further reduction of training FLOPS for AutoSparse. However, in our analysis we do not count such inherent sparsity of activations. Finally, MEST (SOTA for fixed-budget sparse training) achieves comparable accuracy for 80% sparse Resnet50, however, using 20% more training FLOPS and 45% more inference FLOPS (as their sparse model is not
5.3 AUTOSPARSE FOR LANGUAGE MODEL.S
We applied our AutoSparse on Transformer model using exponential decay for a 0.4. This annealing schedule resulted in a better trade-off between accuracy and sparsity. Also, we set a = after 10 epochs to take advantage of more sparse compute in backward pass.
AutoSparse achieves BLEU score of 27.57 with overall sparsity 60.99% which incurs an accuracy drop of 0.93% from dense baseline BLEU score 27.83.
CONCLUSION
We propose Gradient Annealing, novel technique that decays the gradients of pruned out weights in a non-linear manner. This potentially eliminates the need for explicit regularizer to induce sparsity in model training. This provides with tool to develop an automated sparse training method to explore the inherent model sparsity without the knowledge of an appropriate sparsity budget. AutoSparse helps us to achieve state-of-the-art sparsity/accuracy trade-off for two different type of CNN models: ResNet50 and MobiletNetV1 on ImageNet dataset. Further, we demonstrate the applicability of our method on Transformer models for language translation task. Interestingly, among methods producing comparable sparsity, AutoSparse learns non- mon-linear sparsity distribution that is the most FL.PS-efficient.
Declaration: The authors read and adhered to the ICLR Code of Ethics and ICLR Code of Conduct.

---

Under review as a conference paper at ICLR 2023
REFERENCES
D. Amodei and D. Hernandez. ai ann-udddcomme/////
AI and Compute.
N. Ardalani, J. Hestness, and G. Diamos. Empirically Characterizing Overnarameterriatiio Impact net/forummid=aa0
Convergence. In https:://///// http:////ppppeeeeee.eeee///oooaa rett//oorrmm//dsss1111saaaaa 2019.
Arora, Cohen, and E. Hazan. the optimization of deep networks: Implicit acceleration by overparameteriiaaioo. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244 253. PMLR, 10-15 Jul 2018. URL Aroral8a.html
mlr wwt..mm....p....
K. Azarian, Y. Bhalgat, J. Lee, and T. Blankevoort. Learned Threshold Pruning. In https: 00075. pdf, 2021.
w/axxiv.org/pdff//.
M. Belkin, D. Hsu, S. Ma, and S. Mandal. Reconciling modern machine-learning practice and the classical birs-variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849-15854, 2019, doi: 10. http://paaa..10001111 URL /ttp://wwwwwww www. pnws. www.pppna..og//....ooo abs 01000.113330000 pnas 19030701111
G. Bellec, D. Kappel, W. Maass, and R. Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018. URL https://
Openreview Beeerrevee.net//ooreee mett/formmss netf//orruu??d_BBBBMI wNO1C-
Y. Bengio, N. Léonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. In arXiv preprint arXiv:1308$ 3432, 2013.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J D Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, Agarwal, Herbert-Voss, G Krueger, T Henighan, R Child, A Ramesh, D Ziegler, J Wu, C Winter, C Hesse, M Chen, E Sigler, M Litwin, S Gray. B Chess, J Clark, C Bereer, McCandlish, Radford, Sutskever, and D Amodei. Language Models are Few-Shot org/ abs 2005. 141 65 2020.
Learners. In htttt::://axxiii wwtp:/xww..or///o.oaaaaaa/ba/a
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009.
T. Dettmers and L. Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance. In btlttttss//aaxiv.. wttp:/www.wg/////////..... 04840, pdf. 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL /ttppttss////peeeeeeee
net for.m/ormmmmmm Yicbb
U. Evei, T. Gale, J. Menick, P.S. Castro, and F. Elsen. Rigging the Lottery: Making All Tickets Winners. In inthttps: //xxiv,
org/pa//1911 httpp/////aaxavvoorr////////11 11134. pdf 2021.
U. Evci, B. van Merriënboer, T. Unterthiner, M. Vladymyrov, and F. Pedregosa. GRADMAX: Growing Neural Networks Using Gradient Information. In International Conference Learning Representations (ICLR), 2022.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. URL htttps://operreiieee
net/Eo wtr.///rmmmiiiee wil--id-rr-bb-cc--
Gale, E. Elsen, and Hooker. The State of Sparsity in Deep Neural Networks. In https; //arxiv. wwtgp//////p/./////////////// 09574. pdf, 2019.
Y. Guo, H. Yuan, J. Tan, Z. Wang, S. Yang, and J. Liu. GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization. In /n/ wttp:xiwwwor////////0000 02000. pdf 2021.
10

---

Under review as a conference paper at ICLR 2023
S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
K. He, X. Zhang, S. ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
T. Hoefler, D Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, pp. 1-124, 2021.
A.G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In arXiv preprint arXiv:/704.4 7704.0486, 2017.
S.M. Jayakumar, R. Pascanu, J.W. Rae, S. Osindero, and E. Elsen. Top-KAST: Top-K Always Sparse Training. In 2021.
A. Kusupati, V. Ramanujan, R. Somani, M. Wortsman, P. Jain, S. Kakade, and A. Farhadi. Soft Threshold Weight Reparameterization for Learnable Sparsity. In htttttss//ariiii org/abs/ 2002.03233 2020.
Y. Lee. Differentiable Sparsification for Deep Neural Networks. In htttt:::/ htfp::://operevieww 2021.
net/Forumidd= 1ePP1iid_x4xA1221-11
Z. Li, E. Wallace, S. Shen, K. Lin, K. Keutzer, D. Klein, and J. Gonzalez. Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers. In https://arxii: wttpp/wwww..g/pp/////111717/ pdf 2020.
Liu, Xu, Shi, R.C.C Cheung, and H.K.H So. Dynamic Sparse Training: Find Efficient Sparse Network from Scratch with Trainable Masked Layers. In International Conference on Learning Representations (ICLR), 2020.
S. Liu, L Yin, D. C. Mocanu, and M. Pechenizkiy. Do We Actually Need Dense Over- Parameterization? In-Time Paramtttettiiioooonnnn in Sparse Training. In https://arxiv org/pdf/2102 02887.88880 pdf 2021.
Micikevicius, S. Narang, Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. Mixed precision training. In arxivprrprintar..
1710. 03740 2017.
D. Molchanov, A. Ashukha, and D. P. Vetrov. Variational Dropout Sparsifies Deep Neural Networks. 7a html] 2017.
In lattp://proceediingg mlr. ptwps//700mmmmcaanaooooe http:///coaaaaloooooooos/sssss/s
H. Mostafa and X. Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the 36th International Conference on Machine Learning, pp. 4646-4655. PMLR, 2019.
Narang, G. F. Diamos, S. Sengupta, and E. Elsen. Exploring sparsity recurrent neural networks. CoRR, abs/1704.05119, 2017. URL httt::://arxi..orrg/a////100000 00119
M. Naumov, D. Mudigere, H.-J. M. Shi, J. Huang, N.n Sundaraman, J. Park, X. Wang, U. Gupta, C-J Wu, A G. Azzolini, D. Dzhulgakov, A. Mallevich, I. Cherniavskii, Y. Lu, R. Krishnamoorthi, A. Yu, V. Kondratenko. S. Pereira, X. Chen, W. Chen, V. Rao, B. Jia, L. Xiong, and M. Smelyanskiy. Deep learning recommendation model for personalization and recommendation systems. CoRR, abs/1906 1906.0009,, 2019. URL ULLLaappp///aaatt.orrr////// 00091
Md. A. Raihan and T.M. Aamodt. Sparse Weight Activation Training. In Conference Neural Information Processing Systems (NeurIPS 2020), 2020.
A. Renda, J. Frankle, and M. Carbin. Comparing rewinding and fine-tuning in neural network pruning. In International Conference on Learning Representations, 2020. URLbtttss://// //openeevee////e//
11

---

Under review as a conference paper at ICLR 2023
P. Savarese. H. Silva, and M. Maire. Winning the Lottery with Continuous Sparsification. In 2021.
J. Schwarz, S. M. Jayakumar, R. Pascanu, P.E. Latham, and Y. W. Teh. Powerpropagaation: A sparsity inducing weight reparameterisation. In Ittpp::/arxiv..tt wttp:///p//////..22///. pdf 2021.
D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap. Hui, L. Sifre, G. van den Driessche, Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-, October 2017. URL/ittpp:/dxx.doo org. /0 1038 2tt///arrrr2222
N. Ström. Sparse Connection And Pruning In Large Dynamic Artificial Neural Networks. In http://ww.iikk... www.nikkkkstrroo ron.s////ww..ikkottrro..
cat http:a/accaaaooooooeeeeeeeeeeeee
pdf 1997.
X. Sun, J. Choi, C-Y Chen, N. Wang, S. Venkataramani, Srinivasan, X. Cui, W. Zhang, and K. Gopalakrishnan. Hybrid 8-bit Floating Point (HFP8) Training and Inference for Deep Neural Networks. In The Conference and Workshop on Neural Information Processing Systems, 2019.
Tang, L. Luo, Xie, Y. Zhu, Zhao, I. Bi, and C. Lu, Automatic Sparse Connectivity Learning for Neural Networks. In httttt:s//aaxiiii wwtp://wwwwpppp//.....oo httppp////221..0000.2222 2022.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In Neural Information Processing Systems, 2017.
N. Wang. J. Choi, D. Brand, C-Y Chen, and K. Gopalakrishnan. Training Deep Neural Networks with 8-bit Floating Point Numbers. In htttss://arke:
org. wttp://wwwwrppd///./ee////// 08011 påf 2018.
G. Yuan, X. Ma, W. Niu, Z. Li, Z. Kong, N. Liu, Y. Gong, Z. Zhan, C. He, Q. Jin, S. Wang. M. Qin, B. Ren, Y. Wang. S. Liu, and X. Lin. MEST: Accurate and Fast Memory-Economte Sparse Training Framework on the Edge. In Neural Information Processing Systems (NeurIPS), 2021.
Y. Zhang, M. Lin, M. Chen, Z. Xu, F. Chao, Y. Shen, K. Li, Y. Wu, and R. Ji. Optimizing Gradient- driven Criteria in Network Sparsity: Gradient is All You Need. In https::/arxxv..org/ pdf/2201. 12826. p 2022.
A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning n.m fine fine-grained structured sparse neural networks from scratch. In International Conference on Learning Repre- sentations, 2021. URL https: http://pppeeeiiieeeeeee htt.//eunnrttteeee
net forum? 1d= d vgp__
M.H. Zhu and S. Gupta. To prune, not to prune: exploring the efficacy of pruning for model compression. In Intttps:// //tr////vvvv//////////////////// wwtp///gppppp/////oo/ 01878. pdf, 2017.
12


Reviews:
[{"review_id": "tLSsnSAJhF", "reviewer": "Reviewer_EMVQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first introduces a technique called Gradient Annealing, which utilizes a similar concept of the straight-through estimator to enable weight updates on the pruned weights. Specifically, the gradient on the pruned weights will be shrunk by a factor, which is decaying over time. They argue that using a non-linear annealing scheme can help stabilize the sparse budgets. The authors further propose a sparse training algorithm called AutoSparse, that involves a layer-wise learnable threshold for pruning, and also applies the Gradient Annealing technique on a subset of parameters and can enhance efficiency as claimed by the authors. Experiments show performance gain on image classification and NLP tasks. \n", "summary_of_the_review": "Overall I appreciate the authors' work, and it could be the community's interest and inspire future research. However, there are still several things that I believe are necessary to address. Therefore I give a marginal score, but I am also open to changing my score based on the authors' responses.", "strength_and_weaknesses": "Strengths:\n\n- Provided intuitive examples and explanations (Page 3-4)\n- Substantial amount of experiments on large-scale datasets (ImageNet and WMT). \n\nWeakness: \n\n- Some details are hard to follow.  \n- Evaluation is not sufficient to justify the effectiveness of the proposed techniques", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c1Ef_XDl1bl", "reviewer": "Reviewer_djDX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces Gradient Annealing (GA) as a key component for learnable sparsity training to further improve training and inference efficiency alongside accuracy. The authors propose AutoSparse, a training sparse training algorithm that combines GA with the generalized formulation of STR (Kusupati et al., ICML 2020). The paper argues that GA is a crucial component and improves accuracies further compared to vanilla STR formulation, at the same time also improving accuracy for deterministic pruning methods like TopKAST.\n\nThe paper also provides the choices for the GA hyperparameter ($\\alpha$) alongside an analysis of why having a GA-style non-linear decay of gradients of potentially inactive weights helps in having higher accuracy than an STR + STE variant or STR with no GA. \n\nThe paper also supports the method with extensive experimentation on ImageNet with MobileNet and ResNet50 compared against various baselines across efficiency and accuracy metrics both for STR style methods and deterministic methods. Auto Sparse was also extended to an LM task. \n\nNote that the brevity of the review should not be taken as a negative aspect, as the paper's point is clear and direct given the similarity to STR in the formulation. ", "summary_of_the_review": "While the idea of GA alleviates some of the issues in learnable sparsity, it brings in a bit of design complication -- which is fine. However, the bigger concerns are with comparisons to some missing baselines as well as the writing. I think the paper has potential and with a strong revision is worth publishing. \n\n----\nAfter rebuttal, I think the paper might be a good  addition to the iclr community. However, it still has issues that need to be addressed as pointed by the other reviewers as well.", "strength_and_weaknesses": "I will go sequentially for both strengths and weaknesses.\n\nStrengths:\n1) The paper is well motivated and understands the one major problem plaguing learnable sparsity methods and exposits well.\n2) The related work is well-placed.\n3) The explanation and exposition of the GA method, design choices, and analysis are done well. \n4) The experiments are thorough and in line with the baseline papers like STR. \n5) The experiments with training and inference metrics are appreciated. \n6) Backward sparsity results showcase the generality of GA\n\nWeakness:\n1) The writing is not clear and would definitely benefit from revision during the course of the discussion. This also includes the aesthetics and the issues with citation style (whatever is used is not the default for ICLR 2023) -- The second aspect has not affected my perception of the paper, but I would recommend a revision to fix them.\n2) While the proposed solution of GA is interesting and states that it alleviates some of the hparam issues of learnable sparsity, its design choices often are indicative of the potential search through human function design. For example, the non-linear decay comes from the observation of not letting sparsity hit 100% soon for STR when the s_init is small in magnitude. While I like that the design helps with better scalability and applicability, it feels like each of the aspects is being handled specifically -- not a major weakness but something that struck me.\n3) AutoSparse is a generalized formulation of STR, however, when comparing, AutoSparse was only compared to STR. I think it makes a lot of sense to compare AustoSparse to STR + STE (by setting $\\alpha=1$) -- This experiment would show if GA is the major factor for the gains or if is it just the dead gradients through ReLU -- this is a major concern. \n4) In Table 2, why don't we have an apples to apples comparison with the same sparsity for TopKAST as it is deterministic -- I would love to see the accuracy of 85% sparse topKAST solution.\n5) While the application to LM is interesting, it serves little purpose in helping us understand without any baseline -- Please add a strong baselines like STR on AutoSparse for this -- I understand STR might not have done this, but it is the duty of the authors to apply an existing technique and its obvious variants as baselines for a strong paper. \n6) Lastly, I also do not agree with the claim that the top-k sorting in deterministic pruning is a compute-intensive step. -- any thoughts?\n\nI am open to discussion when the rebuttal and revisions come with answers to my mentioned weaknesses. I am looking forward to it and potentially changing my score. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "apmjMXvpru", "reviewer": "Reviewer_Xqxh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on sparse training, which aims to reduce the computational overhead of deep neural networks. Specifically, this work proposes a non-linear gradient-based method, namely, Gradient Annealing (GA), to address the trade-off between model sparsity and accuracy. Meanwhile, this paper combines one latest sparse training method with GA, arriving at a unified training algorithm, i.e., AutoSparse. Extensive experimental results demonstrate that the proposed method could achieve the state-of-the-art model sparsity of 80% on ResNet and of 75% on MobileNetV1. Besides, GA outperforms TopKAST+PP by 0.3% in terms of classification accuracy on ImageNet.", "summary_of_the_review": "This paper proposes Gradient Annealing (GA) to address the trade-off between model sparsity and accuracy, and then, combine GA with the recent state-of-the-art method, arriving at a unified training algorithm called AutoSparse. The technical contributions seem to be novel. Meanwhile, this paper is well-written and the proposed method is reproducible. Based on the above consideration, I recommend accepting this paper. But, since I am not an expert in this field, I am open to changing my score, based on the authors' responses and other reviewers' comments.", "strength_and_weaknesses": "### Strengths\n\n- This paper is well-written and easy to follow.\n\n- The technical contributions,  i.e., GA, a gradient-based non-linear method, for addressing the trade-off for model sparsity and accuracy, are novel.\n\n- The experimental evaluation is solid, including baseline comparisons in the main paper and ablation studies in the appendix.\n\n  \n\n### Weaknesses\n\n- Vision Transformers (ViT) is a popular backbone network in recent years. The empirical evaluations shall conduct baseline comparisons by using ViT on ImageNet.\n\n\n\n[1] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, ICLR 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Az1mDQkwYc", "reviewer": "Reviewer_5Hu8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new sparse neural network training method. The key component in the proposed method is called Gradient Annealing (GA). GA can automatically find a sparse subnetwork in the end of training. The authors also propose some tricks to sparsify the computation in training. A series of experiments are used to evaluate the performance of the proposed method. ", "summary_of_the_review": "1. The proposed method seems tricky and not elegant. It is unclear whether these tricks would produce negative impacts in training. \n\n2. This paper is not well motivated. \n\n3. It is difficult to control the final sparsity in the proposed method, which is very important in real applications. \n\n4. Some details, such as how to calculate the computational saving, are missing.  \n", "strength_and_weaknesses": "Strength:\n1.\tThis paper is well written and easy to read. \n2.\tThe experiments show that the proposed method can find a sparse subnetwork during training. \n\nWeaknesses:\n\n1. Some overclaim statements. In page 4, the authors claim that “minimize sparsification overhead” and “optimal trade-off”. How to prove these optima, as the proposed method is heuristic?\n\n2. This paper is not well motivated. In the abstract, the authors say that the proposed method eliminates the need for sparsity inducing regularizer. I don’t think this is an advantage. Moreover, in the proposed method, it seems that it is difficult to control the sparsity of the finally trained neural networks, which is very important in real applications.\n\n3. The authors propose some tricks to sparisfy the computation in training, such as in sparsifying the weight gradient. It is unclear whether these tricks would produce bad impacts in training as no theoretical analysis is provided in this paper. \n\n4. The authors claim that the proposed method can reduce the FLOPS in training. Since the structure of sparse computation in this method is complicated, how to calculate/count the computational savings in FLOPs? The authors are recommended to release the code to  show this counting procedure. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3MB7g0ICwd5", "reviewer": "Reviewer_7rY9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Gradient Annealing (GA) and AutoSparse, two complementary approaches to achieve a towards optimal sparsity-accuracy trade-off during the training of sparse neural networks.", "summary_of_the_review": "Interesting work, but it seems to be a bit immature and not ready yet for publication.", "strength_and_weaknesses": "Strength:\n* Interesting idea with a small level of novelty\n* The proposed method seems to be able to slightly improve the performance on top of the baselines.\n* Experiments performed on large datasets and models\n\nWeaknesses:\n* The categorization of sparse training methods from the first page and the related work is shallow and misleading. For instance, “deterministic pruning” takes the reader to the idea of a deterministic process, while in reality just the sparsity level may be fixed (or some rules of pruning a part of the connections, etc.), but the output itself (e.g., the sparse connectivity) is a result of (in most cases) a random process (e.g, random pruning, random sparse initialization, stochastic gradient descent, etc). This needs serious adjustment (including of a number of statements in the paper) to reflect better the state-of-the-art and to clarify sharply the paper contributions on top of the existing work. One could start from latest survey papers such as [1]\n* The proposed method has a small level of novelty. Some of the main paper claims are perhaps too strong.\n* The improvement achieved by the proposed method over the baselines seems marginal. The statistical significance of the results has to be studied in order to boost the paper quality. If the large datasets/models hinder this operation then smaller datasets and other type of layers (e.g., fully-connected) could contribute in offering a more comprehensive understanding of the proposed method behaviors. \n* Minor: TopKAST doesn't achieve anymore the state of the art performance. For instance, the results reported in [2] seems higher than the results achieved in this paper. A direct comparison or a study to see if the proposed method can improve also MEST is necessary. Please note that this may lead to claims re-adjustments.\n* Minor: Some parts of the used terminology can bring confusion. It has to be made more rigorous. For instance, GA is an acronym typically used for Genetic Algorithms. \n\nNon-exhaustive list of missing references:\n\n[1] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, Alexandra Peste, Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks.\nAuthors, JMLR 2021, https://www.jmlr.org/papers/volume22/21-0366/21-0366.pdf \n\n[2] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, Xue Lin, MEST: Accurate and Fast Memory-Economic Sparse Training Framework on the Edge, NeurIPS 2021, https://arxiv.org/abs/2110.14032 \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1JFpPf30yT", "reviewer": "Reviewer_G8BR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes gradient annealing method in dynamic sparse training to improve the performance. Based on gradient annealing, the author of the paper proposes AutoSparse algorithm that uses a learnable threshold to find better sparse topology and sparsity distribution during DST. Multiple experiments are conducted on ImageNet.", "summary_of_the_review": " I think this paper needs major revision, both on the technical contribution and writing. My suggestion is reject.", "strength_and_weaknesses": "Strengths\n\n1. This paper investigates different types of non-linear annealing methods and did a thorough analysis\n\n2. The author of the paper not only performs experiments on CV models but also on Language models.\n\n\nWeaknesses\n\n1. The overall contribution is incremental to existing approaches. Investigating and adding different annealing methods to sparse training is not a major contribution. Additionally, I don’t see a very strong motivation for using this GA method. It is more like a training “trick” of dynamic sparse training.\n\n2. The accuracy improvements are not significant. Many existing works on sparse training are not compared, such as MEST (NeurIPS 2021) or GradMax (ICLR 2022).\n\n3. Writing is not clear. For example, in the introduction, the author first claims deterministic pruning sets desired fixed sparsity for each layer before training, then at the deterministic pruning subsection, they claim method [26] (DSR) is belong to deterministic pruning. However, DSR changes sparsity distribution during training, which is not what the author claimed in the earlier part of this paper. So I think some of the writing is not rigorous.\n\n4. It is not true that non-uniform sparsity distribution “did not show any improvement” to uniform distribution. RigL (ICML 2020) and GaP (ICLR 2022, Effective Model Sparsification by Scheduled Grow-and-Prune Methods) are all showing non-uniform sparse distribution achieves lot more better accuracy. So it is not rigorous to have that strong claim in this paper without any experimental data to prove such claim.\n\n5. The overall writing and paper organization is not good. The introduction part is not entirely clear to me, and the structure is confusing. Some part of the introduction feels like a related work survey.\n\n6. Very limited experimental results in this paper. The author didn’t show any data for motivation, and no ablation study showing why the proposed method is effective. No analytical or empirical experiments are performed to convince the reader on why the results is outperforming others.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject