Search Spaces for Neural Model Training
Authors: Darko Stosic
Dusan Stosic
Venue: ICLR 2022 Submitted (2022)
Keywords: search space
sparsity
neural models
deep learning

Abstract: While larger neural models are pushing the boundaries of what deep learning can do, often more weights are needed to train models rather than to run inference for tasks. This paper seeks to understand this behavior using search spaces -- adding weights creates extra degrees of freedom that form new paths for optimization (or wider search spaces) rendering neural model training more effective. We then show how we can augment search spaces to train sparse models attaining competitive scores across dozens of deep learning workloads. They are also are tolerant of structures targeting current hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.

Under review as a conference paper at ICLR 2022
ADVANCES IN SPARSE NEURAL MODEL TRAINING
Anonymous authors Paper under double-blind review
ABSTRACT
While larger neural models are pushing the boundaries what deep learning achieve, often more weights are needed to train models rather than to run inference for tasks. This paper explores how can we remove weights during training without impacting the ability to traverse parameter spaces (or the set of all weights a model can take). We first discover weights pruned for inference do not learn meaningful information over time but their short-term behavior is necessary for training. We then provide recommendations for removing such weights in order to train sparse neural models. With these recommendations, we attain competitive scores across dozens of deep learning workloads. We also find sparse models are tolerant of structures targeting existing hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.
INTRODUCTION
In the area of deep learning, increasing the size of neural models has led to dramatic advances (Hes- (ness et al., 2017 Kaplan et al. 22020; Henighan et al.) (2020). motivating training with hundreds of billions of parameters (Brown et al.) Fedus al. 2021). ). However, larger models incur higher memory costs and runtimes, which limits training and inference tasks that can run on existing hard- ware Thompson et al. 2020). Nevertheless, larger models have been shown to train faster (Li al. 2020 and better (Kaplan et al. 2020), which drives their adoption.
An area of research that seeks to compensate the increasing costs of model size is sparsity (Hoefler et al. (021), which moves weights to zero so they can be discarded from storage or computations. Sparsity emerges as a promising option to reduce costs of inference (Luo et al.) 2017 Allen-Zhu ett al. 2019), as neural models capable of performing tasks sizes smaller than they were trained for (Narng et al. 2017: Renda et al. 2020). However, since overparameterization remains critical for training, sparsity observes limited success there (Lee et al., 2019; Frankle & Carbin 2019 Wang et al. 2020; Tanaka et al. 2020 Frankle et al. [201] Bellec et al.) 2018)) Mocanu et al,
2018 Dettmers Zettlemoyer 2019
Evci et al. 2020a Jayakumar et 2020).
The ability to run inference but not training with fewer weights points to a problem with search (or how well can models navigate parameter spaces during training) rather than model capacity (LeCun) cet al., 1990). Literature shows adding more weights to training creates extra degrees of freedom that form new paths for optimization, rendering neural model training more effective (Evci et al. 2020b). While it is widely believed that having more weights facilitates escape from critical points during training, the question remains whether these weights are always needed (e-g.. learn meaningful representations) or could be eventually discarded if they only help with the optimization process.
In this paper we explore how to remove weights from training without restricting its ability ef- fectively traverse parameter spaces towards good solutions. Surprisingly, we discover that weights pruned at inference time are transient and do not learn representations these weights do not need to be stored throughout all of training, in contrast to weights that learn meaningful information for inference and thus need to be stored. The pruned weights however do learn short-term representations that provide the model extra degrees to escape regions of bad sad- dle points (Kawaguchi, 2016) and high error plateaus (Dauphin et al. 2014) that can slow down learning, after which they can be discarded.
the long-term. As such,
With this understanding we propose new recommendations for training sparse models that consol- idate recent advances in sparsity literature: (1) rewire weights to expand the parameter space, (2)

---

Under review as a conference paper at ICLR 2022
update gradients of all weights to encourage alternate paths of optimization, and (3) discard inter- mediate representations to reduce noise from gradient accumulations. Following these recommen- dations, show that sparse neural models achieve competitive results dozens of deep learning workloads, even when satisfying constraints needed to accelerate training and inference using Sparse Tensor Cores (Mishra al. 2021) in NVIDIA GPUs.
The paper is organized as follows. In Section 2 we introduce a metric for learning and investigate the duration which weights retain meaningful information throughout training. We then devise recom- mendations for removing weights that do not learn long-term representations when training sparse models in Section E] Section 4 summarizes our methodology for sparse training. Section shows sparse models constructed in this fashion perform competitively across a plethora of deep learning workloads and can target current hardware accelerators (Mishra et al. 202T) for training and ference. Section relates to prior research. In Section work.
we conclude with directions for future
METRIC FOR LEARNING
The crux of this paper is to understand whether weights that are being added for training are learn- ing meaningful representations or solely being used to facilitate the optimization process. For this purpose, we use correlations to measure the duration that weights in neural model retain (or learn) information that is relevant for inference.
sets of data. When applied weights learned time, correlations measure how weights in future training time steps are similar to past ones. We postulate correlations represent the degree with which weights in neural model learn. Weights that learn meaningful representations for inference are repeatedly reinforced throughout training, and their values form distinct temporal patterns that are correlated to past values. In contrast, weights that do not learn (and can be removed during inference) exhibit random behavior and have no dependence over time.
Correlations determine how similar
We measure correlations between pair of time series x and y using the Pearson coefficient (Rodgers) Nicewander, 1988): : Pz.y =\t F[(x (y) [y)]/a=dy: where jr the mean, the standard deviation, and B.l the expected value. Coefficients of value -1 denote two series have identical trends, indicates the series are random, and -1 represents series with opposite behavior. We treat each individual weight
time series E {w1, ......... we) and compute Pearson coefficients be- tween windows x [Wi, W2..... W and y E [w,, w w.+1..... Wi representing the similarity between the weights and their values some future time Correlations naturally decay with since temporal patterns are less likely to persist over long durations. An important metric we adopt is the time A after which weights no longer correlate their future values, namely when p 0.
1.0
0.8
0.20
// Training TIME
0.8
0.15
0.6
0.4
0.10
0.4
0.2
0.05
0.0
10a
0.0
10-2
10a
0.00
10
10-1
10a
WEIGHT MAGNITUDE
WEIGHT MAGNITUDE
WEIGHT MAGNITUDE
Figure 1: Time (A) it takes for weights to decorrelate (normalized by the total number of training steps) function of weight magnitudes obtained after training. Points correspond to the me- dian A over weight bins sampled from single neural layer. Shaded regions distinguish between weights that are needed (white) or can be removed (yellow) at inference time. From left to right: Transformer-X XL. (Language Modeling), GNMT (Machine Translation), ResNet50 (Image Classifi- cation).
2

---

Under review as a conference paper at ICLR 2022
Figure [ plots the time (A) after which weights become completely uncorrelated with their past values. We observe that the correlation length grows with weight magnitudes obtained after training. While weights needed for inference exhibit correlations that persist across significant fraction of training, weights that are not needed have short-term correlations and behave randomly over short periods. As result, long-term correlations of large weights characteristic of learning, signifying repeated reinforcement along the weight direction. On the other hand, the near-random motion of small weights suggests they don't learn useful representations and can be removed during inference.
Because weights removable during inference are absent of correlations over long periods of training, we conjecture they do not need to be stored all the time. However, still need pruned weights during training due to their short-term correlations. More specifically. their short-term interactions are critical for training to take different paths for optimization, facilitating escape from bad saddle points (Kawaguchi 2016) or high emor plateaus (Dauphin et al. 2014)) that can slow down learning. Combining this understanding. in later sections we discover their learned values can be periodically destroyed throughout training (once correlations fall to zero) with no impact accuracy, which paves the road for sparse training.
RECOMMENDATIONS FOR SPARSE TRAINING
This section outlines recommendations for training sparse neural models. While many of these ideas have been circling in sparsity literature, they remain misunderstood which hinders adoption. We use the understanding from the previous section to provide a deeper intuition in these cases. To summarize, we uncover the following steps train sparse neural models:
(1) Frequently rewire weights that participate in training.
Perform gradient updates for weights that do not participate in training.
(3) Induce exploitation (e.g., stop rewiring after some amount of time, reset non-participating weights to zero, regularize them).
3.1 REWIRING OF NEURAL WEIGHTT
Sparse models are commonly trained by rewiring (or sampling) subset weights from neural model according to some criteria (magnitude, sign, gradients, etc), allowing them to learn which weights are needed for inference, Some works rewire weights every training iteration (Bellec et al. 2018 Wortsman et al. 2019: Raihan & Aamodt 2020 Jayakumar et al.] 2020 Zhou et al. [021), while others rewire every hundreds of training steps (Evci et al. (2020)) or after an entire pass through the data (Mocanu et al., [018 Dettmers & Zettlemoyer 2019). Since there are clear reasons behind these choices, an important question becomes how often must weights be rewired so neural models learn effectively.
We study the effects rewiring rates have on accuracy when training sparse models of various sizes d, where d denotes the fraction of total weights being used in the sparse model. Figure (left) plots the task error (or accuracy difference between dense and sparse models) as function of the number of training steps r taken between rewirings. We find rewiring frequency does not matter when sparsity is low or moderate (d ~ 0.5), but errors increase with models get sparser (d < 0.5). Therefore, weights should be rewired frequently (within a few training steps) avoid losing accuracy.
Correlations can help us explain why training improves when we rewire sparse neural models. While weights that do learn long-term representations can discarded during training, they have short- term correlations that create extra degrees of freedom for escaping critical points (Evci et al. 2020b). But this escape only occurs when weights are rewired. For example, neural models always operate on reduced parameter spaces if weights being used remain unchanged throughout training (r - x),, and expand exactly once after going through the entire data when they are rewired every epoch. By swapping between weights that participate (are used in forward and backward propagations) and do not participate (are fixed to zero during training), training can take different paths for optimization using the short-term correlations, which helps neural models explore the parameter space.
3

---

Under review as a conference paper at ICLR 2022
= 0.2
3
Task ERROn
0.3
d = 0.1
3
1.0
0.5
0.0
109
10a
10a 10
10
109
109
10
10a
10a
REWIRIGG STEPS
GRADIENT SCALING
RESETTING STEPS
Figure 2: Investigations into various aspects of training sparse models using ResNet50. Left: Task error (or accuracy difference between dense and sparse models) as function of rewiring steps r. Middle: Task error function of scaling factor applied to gradients of weights that do participate in training. Right: Task error where non-participating weights are reset to zero every training steps. We consider sparse models of different d denoting the ratio of weights being used. Lines represent polynomial fits of sample points. Appendix A covers data for a broader span of neural models and tasks.
3.2 UPDATES FOR WEIGHTS NOT PARTICIPATING IN TRAINING
Previously, found neural weights not needed for inference form short-term interactions that make training more effective. Since sparse models lack weights that have similar roles, we can approxi- mate this behavior using gradient updates for non-pariicipating weights. Because non-participating weights do not contribute to the loss, their gradients determine whether another optimization path is better suited traverse the region being explored in the loss landscape. Repeating gradient updates registers the importance of these paths over some period of training and can trigger rewiring when weights exceed threshold, which allows training to explore different regions of the parameter space while operating on (or forward and backward propagating) smaller set of weights.
We evaluate the importance of gradients updates on conpatticcipaiigg weights by reducing their contribution with scale factor Figure 2 (middle) shows the task error as function of the scale factor using various model sizes We observe error increases as 8 0 (no gradients contribute when = 0), which can be attributed to premature convergence when training lacks expressive power to explore different paths. Dampening the gradient updates affect sparser models (d < 0.5), which are more sensitive to critical points, more than larger models (d ~ 0.5), which may still have sufficient weights to train effectively. As a result, non-participating weights should be updated regularly to retain accuracy.
Gradient updates for weights that do not participate in training could mean they also learn represen- tations, such that more capacity rather than search improves accuracy. We verify this by resetting non-participating weights to zero after every training steps, thus removing any representations they might have learned. Figure 2 highest when values are reset every iteration (z 1), which is equivalent when training without any rewiring. Conversely, error decreases weights are reset less frequently and saturates after suffi- cient training steps (z ~ 1k)) Interestingly, this also represents the time it takes for added weights to decorrelate, shown in previous sections, supporting our conjecture that correlations are indicative of learning, and weights can be discarded once they cease to exist. The ability to remove information from con-paticiipatng weights reinforces the idea that they make training more effective rather than augment model capacity.
(right) shows the task error function of z. Errors are the
While most literature trains sparse models using fewer weights and gradients (Bellec et al. 2018 Mocanu et al. 2018 Dettmers & Zettlemoyer 2019), more recent success has been found by updates
gradients for weights that do not participate in training (Wortsman et al.] 2019 Liu et al. 2020 Zhou al. 2021 Hubara et al., 2021) as described above. However, far there has been little understanding of why this helps improve sparse training, which we attribute to the use of short-term correlations escape critical points.
4

---

Under review as a conference paper at ICLR 2022
3.3 BALANCING BETWEEN EXPLORATION AND EXPLOITATION
Deep learning like other optimization problems treads a delicate balance between exploration and exploitation. In early stages of training neural models explore search spaces using high learn- ing rates, whereas late stages exploit specific regions using small learning rates. Training sparse neural models can also affect this balance, as fewer weights reduces the degrees of freedom and thus hinders exploration during training. On the other hand, gradient updates on non-participating weights introduces noise as any nonzero value will not reflect what being used for training, which limits exploitation (training bounces around basins of attraction due to gradient noise).
Figure shows task error degrades without proper exploration, by training smaller mod- els with neural layers of reduced widths (No EXPLORE), or exploitation, by training sparse models delineated in previous sections (No EX- PLOIT). Therefore, we seek ways to induce ex- ploitation when augmenting search spaces for more exploration. One course of action is to remove noise introduced by non-participating weights during late stages, so training can take steepest descents towards the minima. To achieve this we stop rewiring weights af- ter sufficient training (FIx), such that non- participating weights can no longer contribute to the loss. Figure 3 shows this decreases error rates tremendously.
25
No EXPLORE
No EXPLOIT
20
FIX
RESET
15
REGULARIZE
10
5
0
0.50.4 0.3 0.2
0.1
0.05
MODEL SIZE d
Figure 3: Task error after training Transformer- XL Base using different exploration versus ex- ploitation strategies d. Appendix neural models and tasks.
Another option is to draw analogies with added weights, which can be safely discarded af- ter sufficient training since they do not learn representations participating weights to zero roughly every 1K training steps (or the time it takes for added weights to decorrelate) in order remove gradient noise that may trigger unnecessary rewiring, allowing train- ing to exploit nearby regions. Figure g shows error decreases substantially when non-participating weights are either reset zero (RESET) or regularized with decay factor (REGULARIZE) sug- gested in (Zhou et al. 202T). Sparsity literature introduces alternative courses for inducing exploita- tion (Mocanu et al. 2018 Dettmers Zettlemoyer, 2019 Evei et 2020a).
function of model size [A] covers data for a broader span of
time. We reset non-
4 METHODOLOGY
Using recommendations from the previous section, summarize methodology for sparse training in Algorithm| [ Namely, we rewire weights w of sparse neural models based on their magnitudes in order preserve long-term correlations that were discovered in Section 2 At each point in time, the top fraction d of weights are chosen to participate in training. Participating weights p are used to compute the loss and activation gradients, while the optimizer performs updates on the weight gradients. We then induce exploitation on non non-participating weights using any of the approaches discussed earlier.
We note some limitations of the methodology described above. Sparse training observes memory savings because all weights need to be stored for their gradient updates. Sparsity is also only applied on weights, which means can only accelerate the forward pass and part of the backward pass that computes activation gradients. For further accelcration, we can consider applying sparsity to activations their gradients to accelerate weight gradient computations (Raihan & Aamodt 2020
The methods delineated above make a tradeoff between exploration and exploitation using variables and B. Stopping rewiring after half of training (v 0.5) often provides the best results. should be large enough to retain short-term correlations (on the order 1k training steps). = 0.0002 chosen in (Zhou al. 202T) roughly matches choice for All methods perform equally
5

---

Under review as a conference paper at ICLR 2022
well with their optimal variables, these variables can either be automated or kept as default. For most experiments we adopt RESET and use Fix in a few select cases.
Algorithm TRAIN(D,
2,
1: Initialize neural weights w at random
2: for each training iteration
3: Sample a mini batch of data '
4:
f w; if Wi 2 T, where T is chosen so that |p| = d
> Get participating weights
5:
- W; if W.
Get non-participating weights
6:
{ t L(p. D)
Forward pass
7:
Øt + BLI..D
Backward pass
dw
Ow
8:
W +
? Dun
de
Optimizer step
9:
ift 2 v then n t-0
>> Fix
10:
it mod = 0 then n +
RESET REGULARIZE
1 An
EMPIRICAL DATA
We are now in the position to train sparse neural models using our recommendations. This section presents empirical evidence that short-term correlations can help sparse models achieve better curacy for inference tasks. We first demonstrate that our strategy performs competitively against state-of-the-art sparsity research. Then, we explore sparse models are tolerant to sparsity structures that target hardware acceleration using Sparse Tensor Cores (Mishra et al. 2021).
5.1 COMPARISONS TO DENSE MODELS
Experiments are conducted across a wide range of deep learning tasks and neural architectures trained on large data sets, detailed in Appendix B We draw comparisons between dense models (DENSB) and sparse models as described in this work (SPARSE). We also compare to smaller dense models (SMALL) consisting of neural layers with reduced widths that run on existing dense hardware. We reduce the widths of neural layers by a factor of 1/d along one of the dimensions to match the number of weights used in the sparse models. Appendix C demonstrates the benefit of sparse training compared to smaller dense models.
Table lists accuracies for dense models and their differences for sparse models (adding the two numbers produces accuracies for sparse models) across various tasks as a function of model size d (or the ratio of weights being used for the sparse models). We find using our recommendations most sparse models can be halved in size (d = 0.5) without sacrificing any task accuracy, whereas training with a quarter of the weights (d = 0.25) often reduces accuracy by less than 1%. Some exceptions include efficient convolutional models and sparser models (d 0.1)), which may be constrained by capacity.
5.2 COMPARISONS TO SPARSITY RESEARCH
We also draw comparisons to sparsity literature as detailed below.
LOTTERY. We construct lottery tickets (LOTTERY) by training models to completion (e.g., k steps) and removing weights based on their trained values (Frankle & Carbin 2019 Frankle et al. 2019). Sparse models are initialized with weights obtained after some amount of training
e) and then trained for k- steps. We choose e E [k/ 10, k/100] across various workloads (Frankle et al. 2019).
SET AND RIGL. Participating weights are rewired over time based magnitude and adding new ones either randomly (SET) (Mocanu et al. 2018 or based on their gradients (RIGL) (Evci et al.) 2020a). Weights initialized rewire decays linearly throughout training. We sweep across workloads for the best initial value for the rewiring fraction and frequency.
when they become participating. The fraction of weights to

---

Under review as a conference paper at ICLR 2022
Table Accuracies for dense and their differences for sparse (positive means better) for different d.
Model
DENSE
d
d 0.25
0.1
Model
DENSE
0.5
d
0.25
d 0.1
ResNet18
70.30
+0.01
0.92
-2.75
SqueezeNet V1
60.77
-0.88
-3.0
ResNer34
73.87
-0.20
-0.65
-2.27
MobileNet V2
71.53
-0.87
-0.87
ResNet50
76.71
-0.05
-0.59
-2.17
Stacked UNet-64 64
69.53
-1.41
-3.79
-8.19
ResNet101
77.50
-0.25
-0.11
-1.2
SSD-ResNet18
19.15
-0.1
-5.55
ResNeX150
77 77.68
+0.02
0.49
2.01
SSD-ResNet50
24.93
0.52
2206
5.31
ResNeXt101
79.27
+0.20
+0.19
Faster R-CNN
37.71
0.25
-161
55.1
WideResNet50
40.13
1.06
R-CNN
38.26
1.42
4,80
WideResNet101
78.63
-0.1
+0.04
-1.00
Mask R R-CNN
35.03
-0.8
-2.65
-7.44
33
77.10
- 0.08
-188
33.25
Mask R-CNN 3x
40.78
0.12
-1.12
3.51
Xception
79.28
+0.04
0.28
-1.26
R-CNN
37.05
0..3
-0.81
2.98
DenseNet121
75.46
-0,66
4.35
RetinaNet
36.48
6,00
DenseNet161
78.77
+0.01
0.86
2.35
RPN
57.61
-0.16
-0.90
-256
DenseNet169
76.97
10.04
0.96
3.23
DETR
39.90
+0.10
0.60
0.80
VGGI1-BN
70.70
0.33
2224
Pix2PixHD
08.83
+2227
3,02
3.52
VGG16-BN
74.00
0.25
0.46
1.75
Few-Shot Vid2Vid
25.78
+0.52
0.63
4.52
VGG19-BN
74.88
+0.09
-0.48
-1.52
FAZE
2.94
0.04
0.002
0.04
DRN-C-26
75.22
0.30
U.74
-2355
Vaswami Base
26.87
0.68
1.92
3.62
DRN-C-42
76.78
-0.22
-1.88
Vaswani Large
28.43
2212
DRN-A-50
78.30
-0.23
-0.4
-2.28
Levenshtein
6.16
-0.11
0.23
0.45
DeiT Tiny
72.70
-281
-8.09
16.449
GNMT
24.81
-0.12
+026
-0.15
Small
80.08
-153
3.75
8.30
Base
22.88
2.04
5.41
DeiT' Base
81.95
0.75
XI. Large
17.90
-0.16
-1.1
-265
ShuffleNet22
68.44
0.43
-144
BERT Base
87.66
-0.44
MNASNet
71.80
-1.09
33.66
BERT Large
90.92
-0.02
TigureAplor the task error across the sparsity methods. We find that our strategy vastly outperforms competing approaches on all tasks and model sizes by a wide margin that increases with sparsity. Interestingly, the methods compared typically omit least one of the recommendations listed in this work. For example, LOTTERY performs worse even when starting from a dense model because does not rewire. Conversely, SET and RIGL underperform because they do not accumulate gradient updates. However, other works (Zhou et al. [222 Hubara et al. (021) that adopt concepts similar those explored this paper should pertorm similarly our approach. As a result, leveraging short-term correlations seems to be critical component for recent success in sparse training.
5.3 APPLICATION ON HARDWARE ACCELERATORS
train sparse neural models while maintaining accuracy. However, such models cannot be accelerated on modern hardware with current matrix-math pipelines (Park al., 2017| Gale et al. 2020) without imposing particular structures (or positions of weights used during training and inference). On the other hand, neural structures targeting hardware accelcration
We have shown earlier that
(e.g., removing blocks (Gray et al. 2017). channels filters (Wen al. 2016
al. 2017
CLASSIFICATION
TRANSLATION
LANGUAGE MODELING
SMALL
SET
TASK ERROR
15
LOTTERY
RIGL
SPARSE
4
10
2
2
0.50.4 0.3
0.2
0.1
0.05
0.50.4 0.3
0.2
0.05
0.50.4
0.3
0.2
0.05
MODEL Stze
MODEL SIZE
MODEL
Figure 4: Accuracy difference between dense and sparse models comparing various methods as function of model size d. Left to right: ResNet50 (Classification), Transformer (Translation), Transformer-2 XL (Language Modeling), Appendix covers broader span data.

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracies for dense and their differences for 2:4 sparse (positive means better).
Model
DENSE
2:4 ID
2:4 2D
Model
DENSE
2:4 ID
2:4 2D
ResNet18
72.17
+0.00
-0.28
VGG19
74.88
+0.04
00.1
ResNet34
75.14
+0.06
-0.27
Xception
79.28
+0.04
-0.11
ResNe150
77.67
+0.05
+0.09
DETR
39.90
-0.30
-0.0
ResNet101
78.94
-0.09
-0.31
Pix2PixHD
68.83
+1.34
0.68
Inception
78.11
-0.11
-0.18
Few-Shot Vid2Vid
26.06
+0.49
0.04
ResNext50
78.36
-0.21
-0.99
FAZE
2.49
+0.08
ResNext101
79.27
+0.28
+0.36
GNMT
24.81
+0.15
+0.09
WideResNe150
78.13
-0.07
-0.08
Transformer Large
28.43
-0.10
-0.39
WideResNet101
78.63
+0.08
00.7
BERT Large
90.92
+0.03
0.55
DRN 26
77.66
-0.05
-0.11
layers (Michel al. 2019)) often degrade accuracy. This apparent tradeoff between aceuracy and performance (or speed) has hindered their adoption.
Therefore, we explore whether our recommendations can make hardware-aware structures more amenable for deep learning. As case study, consider Sparse Tensor Cores introduced in NVIDIA Ampere GPU architecture (Mishra et al. 202T), which have twice the math throughput of regular matrix operations. The hardware expects 2:4 sparsity structure that takes least two values to be zero for each group of four values. Appendix illustrates examples of 2:4 sparsity for inference (1D) and for training (2D).
Table lists accuracy differences between sparse and dense models (positive values mean sparse performs better), where extend learning rate schedules for some workloads compared to what is typically used in literature (see Appendix for details). From the table, we find that structured sparse neural models can generally retain accuracy for tasks. While neural models adopting coarser structures such as block sparsity (Narang et al. 2017 Gray et al. 2017) ) fail to benefit from our recommendations and perform better than smaller models (see Appendix E}. using 2:4 for train- ing (2D) and inference (1D) roughly matches accuracy of the dense models. Therefore, we can conclude 2:4 sparsity is particularly effective at leveraging short-term correlations due to its finer granularity. This suggests possible avenues towards accelerating training (Hubara et al.) (021) as well as obtaining efficient models for inference without having to repeat the training process (Mishra al. 2021 Zhou et al. 2021).
RELATED WORK
Applying sparsity to reduce the size of neural models has been topic of interest for the past three
decades (LeCun et al. 1990 Hassibi & Stork 1993, Reed 1993 Castellano et al. 1997 Han et al.) 2015 Jayakumar et al. 2020). Typically, sparse models constructed by training much larger
models (that are casier to train) and removing some of their weights either after training (Han et al.
2015 Mishra al. 202 or gradually alongside training (Naang al., 2017: Zhu Gupta 2018
Gale et al.) 2019). However, the above are only useful to reduce costs for inference.
For training acceleration, early works (Lee al., 2019 Wang et al. 2020
Tanaka al. 2020
sought to remove weights before training with limited success (Gale et al. 2019 Frankle et al., 2021. Surprisingly, it was shown sparse models can be trained when initialized the same way as the trained dense model (Frankle & Carbin 2019). After this breakthrough, many works (Dettmers) Zettlemoyer 2019 Evei et al. [2200a] Jayakumar et al., (020)) tried to dynamically learn sparse models by rewiring their weights throughout training, though some works existed carlier (Bellec ct al. 2018 Mocanu et al. 22018). While this greatly improved accuracy, it was not enough to match that of dense models. More recent advances (Wortsman al., 2019 Liu al. 2020 Savarese al., 2020 Zhou et al. 2021 Hubara et al. 2021) introduced gradient updates for all weights in a neural model. including ones that do not participate in training, and observed better success.
Our recommendations are similar to recent works (Zhou et al., 2021 Hubara et al. 2021) that were conducted in tandem with research, but differ in some crucial aspects. (Zhou et 2021 adopts
8

---

Under review as a conference paper at ICLR 2022
2:4 ID sparsity patterns which cannot accelerate training. We also show alternatives to regularizing the panatiiicii weights which achieve the same effect. (Hubara et al. (202)) uses 4:8 sparsity patterns which are not useful for Sparse Tensor Cores, but they also adopt 2D patterns as we do.
While sparsity has been extensively explored for neural model training, our paper presents new perspectives on why certain methods should or not work. Our explanations about how additional weights do not learn meaningful information throughout training complement existing observations about the benefits of overparameteriiztion (Evei al. 2020b). We use this understanding con- solidate many ideas that have been circling around in sparsity literature but remain not clearly un- derstood. For example, how frequently and why must weights be rewired.
CONCLUSION
In this paper we provide better understanding of what is needed for sparse training. We first discover weights that can be pruned during inference do not learn meaningful representations over time, but exhibit short-term correlations that are necessary to train effectively. Based on this under- standing, we uncover recommendations to remove such weights when training sparse models and conduct extensive empirical studies to determine the effects of different hyperparameters. We then demonstrate that sparse training can work across dozens of deep learning workloads that have not been investigated before sparsity literature. Lastly, we are the first to employ sparsity patterns that can accelerate training of sparse models on existing hardware.
We believe these results open many questions. On the practical side, it may be interesting to con- sider how could this strategy be adopted to accelerate real-world training and inference workloads today, reducing the environmental impact from training very large models. On the theoretical side, would like to understand better ways to approximate the short-term behavior in cases where accuracy still suffers as well as how to address potential biases induced by sparse models which may affect applications. We hope that our results will spur further research on these unconventional architectures, which challenge the default choice held by massive neural models today.
REFERENCES
Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In ICML, 2019.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In ICLR, 2018
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, 2020.
Giovanna Castellano, Anna Maria Fanelli, and Marcello Pelillo. An iterative pruning algorithm for feedforward neural networks. IEEE Transactions Neural Networks, 1997.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdi- nov. Transformer-XL. Attentive language models beyond fixed-length context. arXiv preprint 1rXiv: 2019.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcchre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high-dimensional non-convex op- timization. In NewrlPS. 2014.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXiv:1907.04840. 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint / 2018.

---

Under review as a conference paper at ICLR 2022
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In ICML. 2020a.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv:1906.1072 2020b.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03900 2021.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In ICIR, 2019.
Jonathan Frankle, Gintare K. Dziugaite, Daniel M. Roy, and Michael Carbin. The lottery ticket hypothesis at scale. arXiv preprint arXiv:1903.0161,, 2019.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks initialization: Why are we missing the mark? In ICLR, 2021.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: 190779957.9.4 2019.
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kemels for deep learning. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
.dff 2017. S. Gray, A. Radford, and D. P. Kingma. GPU kemelss for block-sparse weights, https://odr. openai cotpr///llcckppprr/////aaaeeeeee
Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein transformer. In NeurlPS. 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In NeurlPS. 2015.
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NewrIPS. 1993.
K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In ICCV, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, and Sam Mc- Candlish. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:
2017.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. arXiv preprint 102.:2202200054: 2021.
Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Seffi Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n:m transposable masks. arXiv preprint arXiv:2102.08124, 2021.
Siddhant M. Jayakumar, Razvan Pascanu, Jack W. Rae, Simon Osindero, and Erich Elsen. Top- KAST: Top-k always sparse training. In NeurIPS. 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836/ 2020.
10

---

Under review as a conference paper at ICLR 2022
Kenji Kawaguchi. Deep learning without poor local minima. In NeurIPS. 2016.
Yann LeCun, John S. Denker, and Sara A. Solla. Optimal brain damage. In NewrIPS. 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: Single-shot network pruning based on connection sensitivity. In ICLR, 2019.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR. 2017.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gonzalez. Train big, then compress: Rethinking model size for efficient training and inference of transform- ers. ICML, 2020.
T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, 2017.
Junjie Liu, Zhe Xu, Runbin Shi, Ray C. C. Cheung, and Hayden K.H. So. Dynamic sparse training: Find efficient sparse network from scratch with trainable masked layers. In ICIR, 2020.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C. Berg. SSD: Single shot multibox detector. arXiv preprint arXiv:1512.022. 2015.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Jian-Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. ICCV, 2017.
Paul Michel, Omer Levy, and Graham Neubig, Are sixteen heads really better than one? In NeurIPS, volume 32, 2019.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural networks. arXiv preprint arXiv:2104.08. 08378, 2021.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature Communications, 2018.
Sharan Narang, Eric Undersander, and Gregory Diamos. Block-sparse recurrent neural networks.
arXiv preprint arXiv:1711.02782) 2017.
NVIDIA. Deep learning examples.
http:::://gthuub.coo///
Deeplearningixapplee 2020a.
NVIDIA. Imaginaire.
httpss///tthuu...oo/oo////////ss 2020b.
NVIDIA. Megatron-LM,, httpps//ggihnbb.ccm////////////e
LM, 2020c.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In NAACL-HLT, 2019.
Jongsoo Park, Sheng Li, Wei Wen, Ping Tak Peter Tang, Hai Li, Yiran Chen, and Pradeep Dubey. Faster cnns with direct sparse convolutions and guided pruning. In ICLR, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In NewrIPS. 2019.
11

---

Under review as a conference paper at ICLR 2022
A Radford, K Narasimhan, T Salimans, and Sutskever. Improving language understanding by gen-
erative pre- htwp:///aaaaaauuuuuuooooa/aaaaar understanding, research- cowers/
http://awwwwwawaaaa.oooeeaaaeaee com/ openai-asnetsett http://www....aa..o.o.ooooaaaaaa
paper.paf2 2018.
Md Aamir Raihan and Tor M. Aamodt. Sparse weight activation training. In Neur/PS. 2020.
R. Reed. Pruning algorithms-a survey. IEEE Transactions Neural Networks, 1993.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NeurlPS. 2015.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In ICLR, 2020.
Joseph Lee Rodgers and W. Alan Nicewander. Thirteen ways to look at the correlation coefficient. The American Statistician, 1988.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Pedro Savarese, Hugo Silva, and Michael Maire. Winning the lottery with continuous sparsification. In NeurlPS, 2020.
Sohil Shah, Pallabi Ghosh, Larry S. Davis, and Tom Goldstein. Stacked U-Nets: A no-frills approach to natural image segmentation. arXiv preprint 1rXiv:1804.10343 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
org/ abs/ / 1t////1194444 4842.
Hidenori Tanaka, Daniel Kunin, Daniel I.. K. Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In NeurlPS, 2020.
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. The computational limits of deep learning. arXiv preprint NNNi:20077005558. 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers. distillation through attention. arXiv preprint arXiv:2012.12877 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS. 2017.
Chaoqi Wang, Guodong Zhang, and Roger Grosse Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.
Ting-Cle Wang, Ming-Yu Yu Liu, Jun- Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. In NeurIPS, 2018a.
Ting-Chun Wang, Ming-Yo Yu Liu, Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018b.
Ting-Chin Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few- shot video-to-video synthesis. In NeurlPS, 2019.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. NeurlPS. 2016.
Mitchell Wortsman, Ali Farhadi, and Mohammad Rastegari. Discovering neural wirings. In NeurIPS. 2019.
12

---

Under review as a conference paper at ICLR 2022
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey. Jeff Klingner, Apurva Shah, Melvin John- son, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws. Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. Google's s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.088 08144, 2016.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2.
https::/gi com/ http://h.....b.ooooooocccccceeee dttprrrroodeetteenenno.. 2019.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In CVPR, 2017.
A. Zhou, Y. Ma, J. Zhu, J. Liu, Z. Zhang, K. Yuan, W. Sun, and H. Li. Learning fine-grained structured sparse neural networks from scratch. In /CLR, 2021.
M. Zhu and S. Gupta. To prune, or not to prune: Exploring the efficacy of pruning for model compression. In ICLR. 2018.
13

---

Under review as a conference paper at ICLR 2022
A RECOMMENDATIONS FOR TRAINING
We expand our investigations how remove weights that do not learn meaningful information when training sparse models, covering more neural architectures and deep learning tasks.
plots the task emror as function of rewiring steps for sparse models of different d. We observe that error increases with less frequent rewiring (r + oo) for other vision tasks, since rewiring is related to how often traversal through the parameter space expands.
Figure 5
IMAGE CLASSIFICATION
SEMANTIC SEGMENTATION
IMAGE GENERATION
0.5
d - 0.4
TASK EXnOR
40
d 0.3
0.25
= 0.2
2-
20
d = Osl
10'
10'
10a
10a
90"
10'
102
10
10
10'
10a
109
REWIRING STEPS
REWIRING STEPS
REWIRING STEPS
Figure 5: Same as Figure 2 (left). From left to right: Inception (Image Classification), Mask
RCNN (Semantic Segmentation), Pix2PixHD (Image Generation).
F shows the task error as a function of the scale factor applied to gradient updates for non- participating weights. We observe the error increases with decreasing contributions of the gradients (s + 0), which suggests updates to non-participating weights are also important during training for other tasks.
SEMANTIC SEGMENTATION
MACHINE TRANSLATION
LANGUAGE MODELING
TAsk Eknon
0.5
2
d=
d=
0.4
10
10"
f0-2
10"
10"
00
10
10a
GRADIENT SCALING
GRADIENT SCALING
GRADIENT SCALING
Figure 6: Same as Figure 2 (middle). From left to right: Mask R-CNN (Semantic Segmentation), Transformer (Machine Translation), Transformer-Xe (Language Modeling).
Figure demonstrates the task error as a function of the number of training steps at which non- participating weights are reset to zero. Similar to results in Section 3.2, error rates saturate after sufficient training (z 1k), which reinforces the idea that non contaatiiiiittnn weights augment pa- rameter spaces rather than model capacity.
14

---

Under review as a conference paper at ICLR 2022
SEMANTIC SECMENTATION
LANGUAGE MODELING
MACHINE TRANSLATION
12
TASK ERnOR
d=5.5
9
d= 0.2
3
d=0.1
d= 0.1
d=
6
2
3
109
10-
10
10"
10a
10a
RESETTING STEPS
RESETTING STEPS
RESETTING STEPS
Figure 7: Same as Figure 2 (right). From left to right: Mask RCNN (Semantic Segmentation). TrL (Language Modeling), GNMT (Machine Translation).
compares various exploration and exploitation strategies for training sparse models, as described in Section 3.3 While task accuracy degrades with lack of proper exploration or exploita- tion, inducing exploitation by removing gradient noise from non-participating weights (FIX, RESET, REGULARIZE) substantially decreases the error rates.
Figure 8
TRANSLATION
LANGUAGE MODELING
RESET
No EXPLORE
REGULARIZE
No
TASK ENRoR
FIx
0.50.4 0.3
0.2
0.1
0.05
0.50.4
0.3
0.2
0.1
0.05
MODEL Size
MODEL SIze d
Figure 8: Same as Figure From left to right: Transformer (Translation), Transformer-Xi Large (Language Modeling).
The above results across more deep learning workloads further validate recommendations put forth in Section 3 for training sparse models.
B EXPERIMENTAL SETUP
We design experiments on PyTorch (Paszke al. 2019) using custom autograd functions for con- volutions and linear layers. Our functions emulate sparsity using binary tensor (or mask) that we multiply elementwise with the weights of each layer during forward and backward propagation. We determine masks based on weight magnitudes, as described earlier, after the optimizer step and before next training iteration.
B. IMAGE CLASSIFICATION
We train popular convolutional models like ResNets (He et al.) 2016). VGG (Simonyan & Zis- serman 2015), Stacked U-Nets (Shah et al. 2018), Dilated Residual Networks (Yu et al. 2017). Inception (Szegedy et al. 2015). MobileNet Sandler al. 2018), well as vision transformers like DeiT Touvron et al. 2020 Training involves standard pipelines for image classification on ImageNet-2012 deseribed in literature and found in public code repositories. For most workloads, we adopt learning rates with linear warmups for the first epochs, drop the learning rate by a fac- tor of ten epochs 30-60-80, and stop training after 90 epochs. few select neural models (e.g..
15

---

Under review as a conference paper at ICLR 2022
mobilenets and vision transformers), however, trained for more epochs using linear cosine schedules.
We measure model quality using top-1 classification accuracy. We apply sparsity to convolutions and linear layers with some exceptions: convolutions whose input channels are not divisible by 16 (e.g., first convolution layer, group and depthwise separable convolutions).
B.2 IMAGE SEGMENTATION AND DETECTION
Image segmentation and detection tasks include both regression and classification components. Pop- ular detectors and segmentors are typically trained in phases: first a backbone is trained for image classification, followed by the addition of model components that are trained for detection or segmentation. Backbones are trained on ImageNet-2012, while downstream tasks are trained on COCO. We adapt training scripts and code from Detectron2 (Wu et al. 2019).
We train neural models such regions with convolutional neural networks (R-CNN) variants (Ren 2015; He et al., (2017). vanilla one-shot detectors (Liu et al., (2155), and with focal loss (Lin al. 2017). Convolution and linear layers encountered pretrained backbones are sparse, like for classification tasks. Detection and segmentation heads are also targeted.
et al.
B.3 GENERATIVE MODELING
Generative Adversarial Networks (GANs) contain two subnetworks: a generative model and a dis- criminative model which combine regression and discrimination tasks during training. For image and video tasks, the generator model regresses pixel colors. We explore conditional GANs for super image-to-imasee translation and video-to-vie (odvideo synthesis using Imaginaire (NVIDIA (020bb. We measure quality of generated outputs using the Frechet Inception Distance (FID). We experiment with generative neural models like Pix2PixHD (Wang et al. 2018b), Vid2Vid (Wang et al. 2018a), and FewShot- Vid2Vid (Wang et al. 2019), targeting convolution and linear layers.
B.4 MACHINE TRANSLATION
We explore transformer and recurrent neural models for language translation. All models encoder-decoder style architectures trained for English to German (En-De) translation on WMT. We adapt model and training code from Fairseq (Ott al., 2019 and NVIDIA Deep Learning Ex- amples (NVIDIA 2020a). We measure model quality using BLEU scores.
We experiment with GNMT (Wu al. 2016) and transformer-based architectures (Vaswani al. 2017 Gu et al. 2019). All linear layers are sparse, except for embeddings and vocabulary projec- tions.
B.5 LANGUAGE MODELING
We consider recent advances in word-level language modeling using transformer decoder (left-to- right) or encoder (bi-directional) architectures (Radford et al. 2018 Devlin et al. 2018). We pretrain language models in an unsupervised fashion on WikiText- 103 or Wikipedia corpus, and evaluate on downstream tasks that are 7ero shot or require additional finetuning. We train them using Mega- tron (NVIDIA 2020c) and NVIDIA Deep Learning Examples (NVIDIA 2020a). Model quality is measured in terms of perplexity or F1 score.
We train language models such Transformer-XL. Dai et al. (2019) and BERT Devlin et al. 2018)) We make all linear layers sparse, except for embeddings, vocabulary projections, and classification heads for downstream tasks.
C COMPARISONS TO DENSE MODELS
is also interesting to understand where does accuracy of our sparse models (SPArSE) fall be- tween accuracies of dense (DENSE) and smaller neural models (SMALL). We define metric = (DENSE SEARCH) (DENSE - SMALL), where a value one means accuracy matches that of dense, and zero implies accuracy is better than smaller model.
16

---

Under review as a conference paper at ICLR 2022
function of model size d for various tasks. For d ~ 0.5. sparse models able to approximate dense models across all neural architectures and tasks. While decreases for sparser models (d < 0.5), they still much more efficient than smaller models. Even at d 0.1, most models can retain a large amount of accuracy.
Figure 9 iillustrates
CLASSIFICATION TASKS
VISION TASKS
NLP TASKS
1.0
0.9
0.8
FRACTION
0.8
ResNer50
MRCNN
0.8
DENSENETI61
RETINANKT
GNMI
0.7
INCEPTION33
RPN
TRANSFORMER
0.6
VeG19
PIX2PIXHD
TRANSFORMER-XI
0.6
0.4
0.50.4
0.3
0.2
0.05
0.50.4
0.3
0.05
0.50.4
0.2
0.1
0.05
MODEL Size
MODEL SIze d
MODEL SIZE
Figure 9: Fraction of accuracy that sparse models achieve between dense and smaller models as a function d across various tasks and neural architectures. We measure the fraction as = (DENSE SEARCH)(DENSE: SMALL).
D EFFECTS OF LONGER TRAINING
Since conventional models achieve higher accuracy when trained on larger data sets with longer training schedules (Liu et al. 2019). another interesting direction is to explore the effeets of sparsity models that are trained to the limits of their capacity. We train neural models longer by extending their learning rate schedules after warmup e.g. a schedule 1/t becomes 2/t.
function of training time t, which denotes the ratio of training steps to the original training schedule. The fact accuracy improves with more training suggests neural models often not trained capacity using conventional schedules. We observe sparse models achieve worse accuracy than dense models that are undertrained (1 ~ 1), but can match accuracy when trained to capacity (t 1).
Figure 10 (left) plots accuracy differences
Figure 10 (middle) illustrates task eтrоrs between dense and sparse models that have been trained for the same number of steps. We find error scales inversely with since more training gives neural models better chances to explore parameter spaces. Particularly, errors can reach zero with sufficient training (Evei al. 2020a) for models that are not constrained by capacity (d 0.25). Figure I0 (right) shows the time it takes for sparse models to recover accuracy of dense models is relatively short when 1, but significantly longer as accuracy saturates >>>>> 1).
Investigations on the effects of longer training are expanded to more neural architectures and deep learning tasks. Figure [I illustrates accuracy deltas as a function of training time t. For vision tasks, we find sparse models of moderate sizes (d > 0.25) can match accuracy of dense models after sufficient training (t 2 3). On the other hand, for language modeling, sparse models cannot match accuracy for any time t because dense models already capacity for the task at hand. Obviously, when using popular training schedules (t ~ 1), sparse models can be trained bit longer recover the accuracy lost.
E APPLICATION ON HARDWARE ACCELERATORS
This appendix explores various sparsity structures considered in the paper that are amenable for acceleration using modern matrix-math hardware.
17

---

Under review as a conference paper at ICLR 2022
Iso-TIME
ISO-PERF
3
15
ACCURACY DIFPIRENCE
TASK ERROR
2
10
TIME
REGULAR
1
d-0.5
d=0.25
d=0.1
16
2
4
TRAINING TIME
TRAINING TIME 8
TRAINING TIME
Figure 10: Effects of sparsity with longer training of ResNet18. Left: Accuracy differences
function of training time / (or ratio of steps to the original training schedule). Middle: Task error between dense and sparse models trained for the same duration. Right: Training time it takes for sparse models to match accuracy of dense models.
CLASSIFICATION
CLASSIFICATION
LANGUAGE MODELING
ACCURACY DIFPERINCE
-3
REGULAR
D=0.5
-0
D=0.25
D=0.1
-12
3
8
4
2
3
TRAINING TIME
TRAINING TIME t
TRAINING TIME t
Figure 11: Same as Figure 10 (left). Left to right: ResNet50 (Classification), Inception (Classifi- cation), Transformer-2 XL (Language Modeling).
E BLOCK SPARSITY
We first consider block sparsity (Narang al., 2017: Gray et al.) 2017). which removes blocks of
contiguous elements (or weights) in a neural layer as shown in Figure 12 Block sparsity addresses common issues that present for unstructured formats: indices for active blocks reduce storage overhead by factor of the block size, blocks are stored contiguously in memory which reduces irregular memory accesses, and their computations exploit faster matrix-math hardware, such as Tensor Cores in NVIDIA GPUs.
We construct block sparse structures by (1) partitioning a neural layer into set of blocks, (2) aggregating elements in each block into metric, and (3) removing blocks according to some criteria based on their metries. While we remove blocks based on largest magnitude maX4e(1.99) wy Other
choices such as the p-norm (Σ/
1.2
[ /w/l")// achieve similar results.
Because structures restrict the combination of weights that can be formed in a neural model, an important question then for what block sizes b (if any) can sparse models retain accuracy. Table lists accuracy differences between sparse and dense models for d = 0.5 using various block sizes. We find block sparse models fail to maintain accuracy, performing no better than smaller models. Notably, accuracy deteriorates for all tasks and block sizes, including smaller blocks (b < 4) that are less amenable for hardware acceleration. In other words, block structures are too coarse for retaining short-term correlations. For example, different weights in block may have different roles during training: a block that participates in training may contain weights that are not important, wheres porttiiiiiingiing block may have weights that were crucial keep. Both cases prevent sparse
18

---

Under review as a conference paper at ICLR 2022
BLOCK SPARSE
2:4 SPARSE (ID)
2:4 SPARSE (2D)
Block sparse, 2:4 ID, and 2:4 2D structures for a 4 x neural layer. Blank cells represent weights that do not participate in training (are assumed zero) and colored cells denote weights that participate (have nonzero values). Arrows indicate direction along which structure is imposed.
Figure 12:
Table 3:
Accuracies for dense models and their differences for block sparse models using different block sizes
Model
DENSE
SMALL
b=
b
b 4
8
b 16
b 32
Transformer-X
22.88
-2.88
-0.82
-203
-2335
-222
2.36
22.4
Transformer
28.43
-0.77
-0.21
-0.75
-1.10
-1.47
-1.98
- -1.94
GNMT
24.81
-2.75
-0.15
-0.15
+0.02
-0.08
-0.15
+0.30
ResNet50
76.71
-1.63
-0.46
-0.99
-2.19
-2.79
Mask RCNN
35.03
-0.88
-0.28
-1.17
-2.43
-3.74
-4.41
-5.00
models from retaining weights over time that are relevant for traversing the parameter space, and thus impacting accuracy.
E.2 2:4 SPARSITY
We consider Sparse Tensor Cores (Mishra al. 2021 introduced in NVIDIA Ampere GPU architecture which exploit 2:4 sparsity and have twice the math throughput of regular matrix units. Figure shows that 2:4 sparsity mandates each group of four values must have at least two values that are zero. Typically, 2:4 is applied on weights w in the forward pass, = wr. However, we can also apply 2:4 on weight transposes w" for the backward pass, ØL 02 -LLaa w". We denote these two options as 2:4 1D that accelerates forward pass for inference Mishra et al. 2021 and 2:4 2D that accelerates both forward and backward passes for training.
The 2:4 sparsity structure must always be imposed along the inner dimension of dot products. For linear layers, we apply 2:4 on n weight tensor along or n (for forward or backward pass, respectively). For convolutions, a or l x r X s (for forward or backward pass, respectively). where denotes output channels, r and kernel dimensions.
apply 2:4 on & x r weight tensor along input channels
We can satisfy 2:4 ID constraints by removing weights with lowest magnitudes. Since 2:4 2D constraints have trivial solution, seek minimize the cumulative magnitude of the removed weights. In other words, for each x 4 block in the tensor, we construct all possible combinations of 2:4 2D patterns, compute their 1-norm, and choose the structure that has the largest norm.
F COMPARISONS TO SPARSITY RESEARCH
We expand comparisons literature covering more neural architectures and deep learning tasks. Figure 1 illustrates the task eπrr across various methods, neural architectures, and tasks. We find strategy outperforms competing approaches in most cases with few exceptions: we do not
19

---

Under review as a conference paper at ICLR 2022
CLASSIFICATION
CLASSIFICATION
CLASSIFICATION
12
10
15
SMALL LOTTERY
RIGL
10
SET
TASK ERROR
SPARSE
4
2
0.50.4 0.3
0.05
0.50.4 0.3
0.2
0.1
0.05
0.50.4
SEGMENTATION
GENERATION
TRANSLATION
15.0
40
12.5
TASK ERROR
7.5
5.0
2.5
0.0
0.50.4
0.50.4 0.3
0.2
0.1
0.05
0.50.4
MODEL SIzE
MODEL SIzE d
MODEL SIzE
Clockwise: Inception V3 (Classification). DenseNet161 (Classifica- tion), VGG19 (Classification), GNMT (Translation), Pix2PixHI (Generation), Mask RCNN (Seg- mentation).
Figure 13: Same as Figure
outperform lottery tickets in segmentation tasks because detectors and segmentors are trained with small learning rates which limit exploration. For generation tasks, our models are slightly worse for d ~ 0.5 (though noisy scores make it difficult to draw conclusions), but the asymptotic behavior of emrrr rates as d + clearly indicates that our approach is superior.
20


Reviews:
[{"review_id": "d9HRKr-sTw", "reviewer": "Reviewer_kChj", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This paper is well written and easy to understand\n- They make clear the positioning of the paper in relation to prior work and what they aim to demonstrate and compare against.\n- The central idea about extra removable weights being extra degrees of freedom is well explained and may be somewhat novel\n- They demonstrate significantly better results compared to prior work that does not use frequent \"rewiring\" with gradient backpropagation on non-contributing weights.\n- They demonstrate the effects of the different methods and recommendations with their thorough ablation and exploration of the effects of different hyperparameter values.\n\n\nWeaknesses\n- The novelty of the paper (though admitted by the authors) could be more significant. The method itself does not seem novel and has been explored and used in depth in several other papers.\n- These papers are cited, but not compared against. While it is understandable that the main focus of the paper is not on doing significantly better than these methods since they share a similar concept, the paper may be more novel if it can more clearly differentiate their contribution. It is difficult to judge the empirical results in that respect without those comparisons.\n- While the related works are mentioned and cited, the central idea could be better compared to the variety of recent work analyzing Deep Neural networks from a loss landscape perspective. It is somewhat difficult to ascertain how novel this central idea is without this comparison. It is quite similar in effect and possibly concept with many other works on analyzing escaping local minima and overparameterized networks.\n\nQuestions:\nIt may be helpful to directly state this in the response especially if I am misunderstanding or underappreciating something. What is the main novel contribution of this paper that differentiates it from other works in this space?\nBased on Figure 3, it seems like resetting weights to 0 may have a negative effect on model performance? Doe this trend continue if the length between resets in lengthened even further?\n\nCan you baseline your analysis of the weight movement of active and inactive weights compared to random walks of the model weights? Some of the trends are difficult to differentiate from the numerical properties of the metrics. The trend for active and inactive weights must be monotonically increasing for all the weights due to the definition since at the end of training there is no more movement of the weights. For the correlation, larger weights with a random walk for a fixed step size would naturally require more time to uncorrelate. What is the measure for distance traversed in the second graph in figure 1?\n\nThe paper may be improved by exploring the benefits of this sparse training in terms of inference speedup and training speedup.\n\n\n\nMinor comments:\n\"While LOTTERY (Frankle & Carbin, 2019) determines at initialization which weights participate in sparse models\"\nThat cited paper determined based on a full training run based on weight magnitude and then reset the initialization.\n\nTable 1 and 2 would be easier to understand if all the results were consistently error or accuracy or at least if + or - delta was consistently better or worse\n", "summary_of_the_paper": "This paper seeks to explore and understand the improved performance of Neural Networks trained with additional parameters even though a majority of the weights can be pruned during inference. They analyze the magnitude, correlation, and movement of weights during training and propose a hypothesis that the reason is that the weights which can be pruned during inference provide degrees of freedom in the loss landscape which allows better optimization of the non-prunable weights. Based on this, they explore and recommend a methodology for sparse training as a method to efficiently approximate a higher dimensional weight space that maintains these good optimization properties. They ablate the effects of the different recommendations and explore the hyperparameter space to have a better understanding of the dynamics of the sparse training.", "summary_of_the_review": "This paper is well written, and clearly demonstrates the advantages of sparse training with rewiring, gradient updates for non-participating weights, and an induced exploitation step. The paper also includes very thorough and useful empirical results on sparse training and through ablation experiments, but lack comparisons to similar methods. Unfortunately, I currently recommend weak rejection since I don't believe that there are quite enough significant novel contributions. While the experiments and ablations shed some further light into how to effectively use these methods and provides some better understanding, this space and the concepts have been thoroughly explored in previous works.\n\n\n===============================================================\nPost-rebuttal\nI thank the authors for their clarifications in the comments and changes in the paper. I believe much of it is clearer now. Due to this, I will be raising my recommendation to marginally below the acceptance threshold. Still believe that as other reviewers have mentioned, the paper would be significantly stronger if they compared their recommendations more directly with other recent work in the space since they claim to be providing extensive hyperparameter recommendations and comprehensive recommendations for how to train sparse models.\n\n\nThere are also minor changes which would make the figures much clearer. Figure 3 should include their method with the combination of  Fix, Reset, and Regularization. The trend with rewiring and resetting steps in Figure 2 would also be clearer if the graphs were slightly extended to demonstrate at what frequency resetting and rewiring is required.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YcdSXgAupgo", "reviewer": "Reviewer_yWXp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tIt is interesting to understanding the training of deep models from the perspective of search space.\n2.\tThis paper proposed a series of steps to train the sparse models as larger models, and quantitative experimental results show that these steps can effectively reduce the training error of the sparse model\n\nWeaknesses:\n\n1. The third paragraph of Section 2.1 introduces the changes in the role played by model weights during the training process (shown in Fig. 1). And authors make an assumption that the weights which are eventually discarded (or added weights) play a crucial role in determining what will be used for inference. Nevertheless, it is hard to verify the role of these weights according to existing results. More results and explanations are required.\n\n2. Since this paper focuses on training sparse models, it would be better to compare the proposed method with model compression methods (e.g., [a,b]). \n\n3. The authors propose several techniques to encourage the exploration and exploitation when training deep models. However, it is still unclear how to derive the final sparse model from a full network. More details should be provided.\n\n\nReference:\n\n[a] Discrimination-aware network pruning for deep model compression. PAMI 2021.\n\n[b] ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting. ICCV 2021.\n", "summary_of_the_paper": "The authors proposed a new point of view to explain why deep learning models often require more weights are needed to train models rather than to run inference for tasks. One example is that sparse models usually perform well in inference, but do not perform well in training. The main content of this viewpoint is that more weights during training can expand the search space, create extra degrees of freedom for search, and form new optimization paths when model optimization falls into critical points, thereby facilitating model training. Based on this point of view, this paper proposes a series of steps to augment search spaces of sparse models during training to approximate the behavior of larger models. The experimental results show that this series of steps can effectively reduce the model error.", "summary_of_the_review": "It is an interesting paper and the experiments are strong for me.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Rgydut7BXFI", "reviewer": "Reviewer_t5MH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work has two main contributions. First authors argue that over-parameterization is useful for `search`. Later authors provide extensive experimental results (maybe the largest I've seen in this field) on comparing various sparse training techniques. I thank authors for their extensive experimental results, which is quite useful to the community, as many researcher can't do such extensive experiments. This contribution alone could have a significant impact on the field. Do they plan open-sourcing their code? The idea of grouping weights into 3 subsets (inactive/active/search-space) is also interesting and novel (though this idea is mentioned in various work). Though this work has quite a bit potential to impact and inspire future research, I found few things confusing and have questions about its novelty.\n\n## Concerns\n0. Main method used in the paper reminds me previous work like Discovering Neural Wirings and TopKast, in which forward pass is made on top-magnitude weights and gradients are applied to all weights. It would be nice to make a clear comparison with previous work; i.e. what is the novelty? \n1. Naming in Figure-4 is a bit confusing, I recommend using terms like 'Small-Dense', 'Static' instead of no-explore/exploit (also NO EXPLORE becomes REDUCE in the next figure, which is not consistent) as it is used in previous work. Furthermore 'Fix' and 'Reset' have similar curves. So resetting doesn't help? Then, the text needs to be updated as it claims 'fix' brings better results. Maybe bringing results from Figure:12 that shows the gains of resetting weights might be useful.\n2. rigl is known to take longer training steps to converge. Are your experiments made with default dense training steps? I recommend authors to do extended training comparisons (like 2x or 5x) for the Resnet50 classification experiments. Similarly do you use the ITOP recipe proposed by [4] in your experiments? It would be nice to make this clear in the text, as if using limited training steps, ITOP improves the performance of rigl. And finally the difference between 'rigl/set' and 'search' needs to be clear in the text. fully-sparse DST methods (set/rigl) don't need to store values of the inactive weights, which reduces the peak memory consumption significantly. I recommend authors to make this distinction early in the paper, motivate the setting they are focusing on and clearly indicate before each experiment.\n3. I think part of Appendix C should be brought to the main text (like the algorithm), as it would help the reader to understand the experiments and contributions better. One way to enable that would be to make Section 2 more succinct, as few ideas are repeat in different subsections.\n4. Section 2.4 seems very similar to the experiments/observation in [3] (and some other parts of Section 2). Authors should discuss similarities and differences. I am not sure what new insights are provided in Section 2 compared to [3]. Can you make this clear? \n5. Hypothesis and evidence provided in section 2.3 is not convincing to me. When weights approach to zero, they naturally become uncorrelated. Therefore the correlation tells more about which weights go to zero than anything else I think. Furthermore since low magnitude weights are selected at the end it becomes a bit circular. Finally, the relationship between correlation and exploration is not clear. \n6. In appendix it says \"For smaller models, we reduce the width of neural layers by a factor of 1/d to match the number of weights used in sparse models.\" The factor should be 1/sqrt(d), as width=0.1, would have roughly 1% of the parameters. \n7. weight decay has an important effect on exploration and used successfully in literature [1,2]. Do you use weight decay in your experiments?  Are they included in the gradients when you scale the them for Figure:3?\n8. I am not sure the illustration on Figure-2 is helpful giving an example where extra dimensions can be helpful. Since even with the vertical dimension, the optimization would stay at the bad local minima for the given example. Maybe \"mexican-hat\" example would be a better fit. \n\n## Minor\n- First paragraph of Section 2.1, authors should cite the original work on each point made as much as possible. Happy to suggest early references if needed. \n- \"since they contribute the most\" -> the least\n- \"They are also are tolerant\"\n- \"...use search spaces to describe reasons more weights are needed for training\" - \"to describe why more weights are needed...\"\n- \"Task error\" doesn't seem like the right term. I would recommend using something like 'Delta Accuracy' or 'Compression Error'\n- In captions ResNet50 is mentioned without the dataset. I assume it is Imagenet?\n- \"by training sparse models delineated so far\" Not clear what this refers to. So far when?\n- RIGL (Mocanu et al., 2018) (typo)\n- Figure 17: Clockwise is ambiguous. update titles with model names?\n- Related Work/ 'which improved accuracy but not enough to match that of larger models' This statement is vague and possibly not correct. This depends on the sparsity. Maybe you mean dense-to-sparse training? Then, rigl and topkast do match pruning results. \n- 'these results open many question' \n\n[1] https://arxiv.org/abs/2110.00296\n[2] TopKAST: https://proceedings.neurips.cc/paper/2020/file/ee76626ee11ada502d5dbf1fb5aae4d2-Paper.pdf\n[3] https://arxiv.org/abs/1906.10732\n[4] https://arxiv.org/abs/2102.02887\n\n## After Rebuttal\nI read author's response and looked the revised version quickly. A lot seems to be changed during rebuttal, which is probably a good thing; though makes it difficult to evaluate. Regardless, as I mentioned above, the experimental results (which includes basically everything except RL) is unique and thus important for informing future research, thus I raised my score to 6. \n\nIf rejected, venues that accept longer papers (like JMLR) might be a good fit. It would be great to discuss experiments in different domains in detail and try to understand/quantify why different behaviours are observed in different domains. ", "summary_of_the_paper": "This paper studies the training of sparse neural networks and the effect of extra parameters (dense connectivity) during training. They hypothesize that pruned connection are useful, because they help optimization to take some random steps in those direction which help optimization escaping the local minima. Authors support their claims with experiments and finally share a very long list of experimental results on training sparse networks.  ", "summary_of_the_review": "As mentioned above the experimental results provided at the end of the paper (one of the most extensive set of experiments I have seen in a paper), would be a strong contribution alone. However, authors use half of the paper focusing on explaining why more connections are needed for better results, which I find not very novel and a bit confusing. Authors say that pruned connections are used in training for exploration and search very vaguely; which doesn't provide any new insights on the existing understanding of the training of sparse neural networks. I am willing to increase my score if my concerns are addressed and looking forward to read the next version of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-hxWCZMyNRx", "reviewer": "Reviewer_LwfS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThey provide a very intuitive explanation of the problem with good drawings and figures. This gave me a new intuition about how neural networks learn and optimize\nThere's a lot of comparisons with current regular neural networks.\nThe project details are well documented in the appendix\nThe paper is well written\n\nWeaknesses:\nI can't find any comparisons with other, similar, sparsity methods. I would like to see how this holds up against state-of-the-art in sparsity.\nI would like to see confidence intervals on the different model performances (if you have the resources of course).\nThe code is not available (as far as I can see).", "summary_of_the_paper": "The domain is the optimization of deep and large neural networks. The author argues and presents information that supports (figure 1), a hypothesis that a large portion of neurons is pushed towards dormancy while optimizing a neural network and only a subset of the neurons are actively used. The authors, moreover, argue that these dormant weights are important during optimization as they can propose new solutions during training (figure 2). However, these dormant weights can also be an issue as they introduce gradient noise. The authors propose a new scheme to balance the role of the dormant weights in order to maximize new solutions while minimizing gradient noise. The authors do this by either setting them to zero at a certain iteration or regularizing their sizes. They only optimize a set amount of the weights at once (rewind). The authors show results on multiple benchmark datasets highlighting their performance and how new sparsity-oriented hardware works well with their algorithms. The dormitory vs active weights are picked by a threshold of their magnitude.", "summary_of_the_review": "I think this is interesting research and highlights a general and contemporary challenge when training deep neural networks. The problem is well motivated and the implementation well described. The experiments are extensive. however, as this is not compared against any other model in the field I cannot assess whether this approach is truly useful. I would be able to change my mind if the authors provide such results, or argue why they don't provide it. Also, please make your code available in an anonymous repository, and publicly available at the end of the review session. If you have the compute, please provide confidence intervals on your results. Also, please elaborate on what it means for weights to \"decorrelate\". E.g. why is the threshold important.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were generally split on this paper. On the one hand, reviewers generally appreciated the clear presentation, discussion, and explanations, and the experiments. On the other hand, most reviewers commented on the lack of comparative evaluation to other works, including works that are related conceptually. While the authors have a potentially reasonable argument for omitting such comparisons, in the balance I do not believe that the reviewers were actually convinced by this. Particularly when the novelty of the contribution is not crystal clear, such comparisons are important, so I am inclined to not recommend acceptance at this point (though I acknowledge that the paper is clear borderline and could be accepted).
Decision: Reject