Year: 2021

Abstract:
While larger neural models are pushing the boundaries on what deep learning can achieve, often more weights are needed to train models rather than to run inference for tasks. This paper explores how can we remove weights during training without impacting the ability to traverse parameter spaces (or the set of all weights a model can take). We first discover weights pruned for inference do not learn meaningful information over time but their short-term behavior is necessary for training. We then provide recommendations for removing such weights in order to train sparse neural models. With these recommendations, we attain competitive scores across dozens of deep learning workloads. We also find sparse models are tolerant of structures targeting existing hardware, opening avenues for training and inference acceleration. Our work encourages research to explore beyond massive neural models being used today.

heading: 1 INTRODUCTION
text: In the area of deep learning, increasing the size of neural models has led to dramatic advances (Hestness et al., 2017; Kaplan et al., 2020; Henighan et al., 2020), motivating training with hundreds of billions of parameters (Brown et al., 2020; Fedus et al., 2021). However, larger models incur higher memory costs and runtimes, which limits training and inference tasks that can run on existing hardware (Thompson et al., 2020). Nevertheless, larger models have been shown to train faster (Li et al., 2020) and better (Kaplan et al., 2020), which drives their adoption.
An area of research that seeks to compensate the increasing costs of model size is sparsity (Hoefler et al., 2021), which moves weights to zero so they can be discarded from storage or computations. Sparsity emerges as a promising option to reduce costs of inference (Luo et al., 2017; Allen-Zhu et al., 2019), as neural models are capable of performing tasks on sizes smaller than they were trained for (Narang et al., 2017; Renda et al., 2020). However, since overparameterization remains critical for training, sparsity observes limited success there (Lee et al., 2019; Frankle & Carbin, 2019; Wang et al., 2020; Tanaka et al., 2020; Frankle et al., 2021; Bellec et al., 2018; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2020a; Jayakumar et al., 2020).
The ability to run inference but not training with fewer weights points to a problem with search (or how well can models navigate parameter spaces during training) rather than model capacity (LeCun et al., 1990). Literature shows adding more weights to training creates extra degrees of freedom that form new paths for optimization, rendering neural model training more effective (Evci et al., 2020b). While it is widely believed that having more weights facilitates escape from critical points during training, the question remains whether these weights are always needed (e.g., learn meaningful representations) or could be eventually discarded if they only help with the optimization process.
In this paper we explore how to remove weights from training without restricting its ability to effectively traverse parameter spaces towards good solutions. Surprisingly, we discover that weights pruned at inference time are transient and do not learn representations in the long-term. As such, these weights do not need to be stored throughout all of training, in contrast to weights that learn meaningful information for inference and thus need to be stored. The pruned weights however do learn short-term representations that provide the model extra degrees to escape regions of bad saddle points (Kawaguchi, 2016) and high error plateaus (Dauphin et al., 2014) that can slow down learning, after which they can be discarded.
With this understanding we propose new recommendations for training sparse models that consolidate recent advances in sparsity literature: (1) rewire weights to expand the parameter space, (2)
update gradients of all weights to encourage alternate paths of optimization, and (3) discard intermediate representations to reduce noise from gradient accumulations. Following these recommendations, we show that sparse neural models achieve competitive results on dozens of deep learning workloads, even when satisfying constraints needed to accelerate training and inference using Sparse Tensor Cores (Mishra et al., 2021) in NVIDIA GPUs.
The paper is organized as follows. In Section 2 we introduce a metric for learning and investigate the duration which weights retain meaningful information throughout training. We then devise recommendations for removing weights that do not learn long-term representations when training sparse models in Section 3. Section 4 summarizes our methodology for sparse training. Section 5 shows sparse models constructed in this fashion perform competitively across a plethora of deep learning workloads and can target current hardware accelerators (Mishra et al., 2021) for training and inference. Section 6 relates to prior research. In Section 7 we conclude with directions for future work.

heading: 2 METRIC FOR LEARNING
text: The crux of this paper is to understand whether weights that are being added for training are learning meaningful representations or solely being used to facilitate the optimization process. For this purpose, we use correlations to measure the duration that weights in a neural model retain (or learn) information that is relevant for inference.
Correlations determine how similar are two sets of data. When applied to weights learned over time, correlations measure how weights in future training time steps are similar to past ones. We postulate correlations represent the degree with which weights in a neural model learn. Weights that learn meaningful representations for inference are repeatedly reinforced throughout training, and their values form distinct temporal patterns that are correlated to past values. In contrast, weights that do not learn (and can be removed during inference) exhibit random behavior and have no dependence over time.
We measure correlations between a pair of time series x and y using the Pearson coefficient (Rodgers & Nicewander, 1988): ρx,y ≡ E[(x − µx)(y − µy)]/σxσy , where µ is the mean, σ the standard deviation, and E[·] the expected value. Coefficients of value +1 denote two series have identical trends, 0 indicates the series are random, and −1 represents series with opposite behavior. We treat each individual weight as a time series w ∈ {w1, w2, . . . , wt} and compute Pearson coefficients between windows x ∈ {w1, w2, . . . , wt−τ} and y ∈ {wτ , wτ+1, . . . , wt}, representing the similarity between the weights and their values at some future time τ . Correlations naturally decay with τ since temporal patterns are less likely to persist over long durations. An important metric we adopt is the time ∆ after which weights no longer correlate to their future values, namely when ρ = 0.
Figure 1 plots the time (∆) after which weights become completely uncorrelated with their past values. We observe that the correlation length grows with weight magnitudes obtained after training. While weights needed for inference exhibit correlations that persist across a significant fraction of training, weights that are not needed have short-term correlations and behave randomly over short periods. As a result, long-term correlations of large weights are characteristic of learning, signifying repeated reinforcement along the weight direction. On the other hand, the near-random motion of small weights suggests they don’t learn useful representations and can be removed during inference.
Because weights removable during inference are absent of correlations over long periods of training, we conjecture they do not need to be stored all the time. However, we still need pruned weights during training due to their short-term correlations. More specifically, their short-term interactions are critical for training to take different paths for optimization, facilitating escape from bad saddle points (Kawaguchi, 2016) or high error plateaus (Dauphin et al., 2014) that can slow down learning. Combining this understanding, in later sections we discover their learned values can be periodically destroyed throughout training (once correlations fall to zero) with no impact on accuracy, which paves the road for sparse training.

heading: 3 RECOMMENDATIONS FOR SPARSE TRAINING
text: This section outlines recommendations for training sparse neural models. While many of these ideas have been circling in sparsity literature, they remain misunderstood which hinders adoption. We use the understanding from the previous section to provide a deeper intuition in these cases. To summarize, we uncover the following steps to train sparse neural models:
(1) Frequently rewire weights that participate in training.
(2) Perform gradient updates for weights that do not participate in training.
(3) Induce exploitation (e.g., stop rewiring after some amount of time, reset non-participating weights to zero, or regularize them).

heading: 3.1 REWIRING OF NEURAL WEIGHTS
text: Sparse models are commonly trained by rewiring (or sampling) a subset of weights from a neural model according to some criteria (magnitude, sign, gradients, etc), allowing them to learn which weights are needed for inference. Some works rewire weights every training iteration (Bellec et al., 2018; Wortsman et al., 2019; Raihan & Aamodt, 2020; Jayakumar et al., 2020; Zhou et al., 2021), while others rewire every hundreds of training steps (Evci et al., 2020a) or after an entire pass through the data (Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019). Since there are no clear reasons behind these choices, an important question becomes how often must weights be rewired so neural models can learn effectively.
We study the effects rewiring rates have on accuracy when training sparse models of various sizes d, where d denotes the fraction of total weights being used in the sparse model. Figure 2 (left) plots the task error (or accuracy difference between dense and sparse models) as a function of the number of training steps r taken between rewirings. We find rewiring frequency does not matter when sparsity is low or moderate (d ∼ 0.5), but errors increase with r as models get sparser (d 0.5). Therefore, weights should be rewired frequently (within a few training steps) to avoid losing accuracy.
Correlations can help us explain why training improves when we rewire sparse neural models. While weights that do not learn long-term representations can be discarded during training, they have shortterm correlations that create extra degrees of freedom for escaping critical points (Evci et al., 2020b). But this escape only occurs when weights are rewired. For example, neural models always operate on reduced parameter spaces if weights being used remain unchanged throughout training (r →∞), and expand exactly once after going through the entire data when they are rewired every epoch. By swapping between weights that participate (are used in forward and backward propagations) and do not participate (are fixed to zero during training), training can take different paths for optimization using the short-term correlations, which helps neural models explore the parameter space.

heading: 3.2 UPDATES FOR WEIGHTS NOT PARTICIPATING IN TRAINING
text: Previously, we found neural weights not needed for inference form short-term interactions that make training more effective. Since sparse models lack weights that have similar roles, we can approximate this behavior using gradient updates for non-participating weights. Because non-participating weights do not contribute to the loss, their gradients determine whether another optimization path is better suited to traverse the region being explored in the loss landscape. Repeating gradient updates registers the importance of these paths over some period of training and can trigger rewiring when weights exceed a threshold, which allows training to explore different regions of the parameter space while operating on (or forward and backward propagating) a smaller set of weights.
We evaluate the importance of gradients updates on non-participating weights by reducing their contribution with a scale factor s. Figure 2 (middle) shows the task error as a function of the scale factor s using various model sizes d. We observe error increases as s → 0 (no gradients contribute when s = 0), which can be attributed to premature convergence when training lacks expressive power to explore different paths. Dampening the gradient updates affect sparser models (d 0.5), which are more sensitive to critical points, more than larger models (d ∼ 0.5), which may still have sufficient weights to train effectively. As a result, non-participating weights should be updated regularly to retain accuracy.
Gradient updates for weights that do not participate in training could mean they also learn representations, such that more capacity rather than search improves accuracy. We verify this by resetting non-participating weights to zero after every z training steps, thus removing any representations they might have learned. Figure 2 (right) shows the task error as a function of z. Errors are the highest when values are reset every iteration (z = 1), which is equivalent when training without any rewiring. Conversely, error decreases as weights are reset less frequently and saturates after sufficient training steps (z ∼ 1k). Interestingly, this also represents the time it takes for added weights to decorrelate, as shown in previous sections, supporting our conjecture that correlations are indicative of learning, and weights can be discarded once they cease to exist. The ability to remove information from non-participating weights reinforces the idea that they make training more effective rather than augment model capacity.
While most of literature trains sparse models using fewer weights and gradients (Bellec et al., 2018; Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019), more recent success has been found by updates gradients for weights that do not participate in training (Wortsman et al., 2019; Liu et al., 2020; Zhou et al., 2021; Hubara et al., 2021) as described above. However, so far there has been little understanding of why this helps improve sparse training, which we attribute to the use of short-term correlations to escape critical points.

heading: 3.3 BALANCING BETWEEN EXPLORATION AND EXPLOITATION
text: Deep learning like other optimization problems treads a delicate balance between exploration and exploitation. In early stages of training neural models explore search spaces using high learning rates, whereas late stages exploit specific regions using small learning rates. Training sparse neural models can also affect this balance, as fewer weights reduces the degrees of freedom and thus hinders exploration during training. On the other hand, gradient updates on non-participating weights introduces noise as any nonzero value will not reflect what is being used for training, which limits exploitation (training bounces around basins of attraction due to gradient noise).
Figure 3 shows task error degrades without proper exploration, by training smaller models with neural layers of reduced widths (NO EXPLORE), or exploitation, by training sparse models delineated in previous sections (NO EXPLOIT). Therefore, we seek ways to induce exploitation when augmenting search spaces for more exploration. One course of action is to remove noise introduced by non-participating weights during late stages, so training can take steepest descents towards the minima. To achieve this we stop rewiring weights after sufficient training (FIX), such that nonparticipating weights can no longer contribute to the loss. Figure 3 shows this decreases error rates tremendously.
Another option is to draw analogies with added weights, which can be safely discarded after sufficient training since they do not learn representations over time. We reset non-
participating weights to zero roughly every 1k training steps (or the time it takes for added weights to decorrelate) in order to remove gradient noise that may trigger unnecessary rewiring, allowing training to exploit nearby regions. Figure 3 shows error decreases substantially when non-participating weights are either reset to zero (RESET) or regularized with a decay factor (REGULARIZE) as suggested in (Zhou et al., 2021). Sparsity literature introduces alternative courses for inducing exploitation (Mocanu et al., 2018; Dettmers & Zettlemoyer, 2019; Evci et al., 2020a).

heading: 4 METHODOLOGY
text: Using recommendations from the previous section, we summarize a methodology for sparse training in Algorithm 1. Namely, we rewire weights w of sparse neural models based on their magnitudes in order to preserve long-term correlations that were discovered in Section 2. At each point in time, the top fraction d of weights are chosen to participate in training. Participating weights p are used to compute the loss and activation gradients, while the optimizer performs updates on all the weight gradients. We then induce exploitation on non-participating weights n using any of the approaches discussed earlier.
We note some limitations of the methodology described above. Sparse training observes no memory savings because all weights need to be stored for their gradient updates. Sparsity is also only applied on weights, which means we can only accelerate the forward pass and part of the backward pass that computes activation gradients. For further acceleration, we can consider applying sparsity to activations or their gradients to accelerate weight gradient computations (Raihan & Aamodt, 2020).
The methods delineated above make a tradeoff between exploration and exploitation using variables v, z, and β. Stopping rewiring after half of training (v = 0.5) often provides the best results. z should be large enough to retain short-term correlations (on the order of 1k training steps). β = 0.0002 chosen in (Zhou et al., 2021) roughly matches our choice for z. All methods perform equally
well with their optimal variables, so these variables can either be automated or kept as default. For most experiments we adopt RESET and use FIX in a few select cases.
Algorithm 1 TRAIN(D, γ, d, v, z, β) 1: Initialize neural weights w at random 2: for each training iteration t do 3: Sample a mini batch of data D 4: p← wi if wi ≥ τ , where τ is chosen so that |p| = d . Get participating weights 5: n← wi if wi < τ . Get non-participating weights 6: `← L(p,D) . Forward pass 7: ∂`∂w ← ∂L(p,D) ∂w . Backward pass
8: w ← w + γ ∂`∂w . Optimizer step 9: if t ≥ v then n← 0 . FIX 10: if t mod z = 0 then n← 0 . RESET 11: n← n− βn . REGULARIZE

heading: 5 EMPIRICAL DATA
text: We are now in the position to train sparse neural models using our recommendations. This section presents empirical evidence that short-term correlations can help sparse models achieve better accuracy for inference tasks. We first demonstrate that our strategy performs competitively against state-of-the-art sparsity research. Then, we explore sparse models are tolerant to sparsity structures that target hardware acceleration using Sparse Tensor Cores (Mishra et al., 2021).

heading: 5.1 COMPARISONS TO DENSE MODELS
text: Experiments are conducted across a wide range of deep learning tasks and neural architectures trained on large data sets, as detailed in Appendix B. We draw comparisons between dense models (DENSE) and sparse models as described in this work (SPARSE). We also compare to smaller dense models (SMALL) consisting of neural layers with reduced widths that can run on existing dense hardware. We reduce the widths of neural layers by a factor of 1/d along one of the dimensions to match the number of weights used in the sparse models. Appendix C demonstrates the benefit of sparse training compared to smaller dense models.
Table 1 lists accuracies for dense models and their differences for sparse models (adding the two numbers produces accuracies for sparse models) across various tasks as a function of model size d (or the ratio of weights being used for the sparse models). We find using our recommendations most sparse models can be halved in size (d = 0.5) without sacrificing any task accuracy, whereas training with a quarter of the weights (d = 0.25) often reduces accuracy by less than 1%. Some exceptions include efficient convolutional models and sparser models (d = 0.1), which may be constrained by capacity.

heading: 5.2 COMPARISONS TO SPARSITY RESEARCH
text: We also draw comparisons to sparsity literature as detailed below.
LOTTERY. We construct lottery tickets (LOTTERY) by training models to completion (e.g., k steps) and removing weights based on their trained values (Frankle & Carbin, 2019; Frankle et al., 2019). Sparse models are initialized with weights obtained after some amount of training (t = ) and then trained for k−t steps. We choose ∈ [k/10, k/100] across various workloads (Frankle et al., 2019). SET AND RIGL. Participating weights are rewired over time based on magnitude and adding new ones either randomly (SET) (Mocanu et al., 2018) or based on their gradients (RIGL) (Evci et al., 2020a). Weights are initialized to zero when they become participating. The fraction of weights to rewire decays linearly throughout training. We sweep across workloads for the best initial value for the rewiring fraction and frequency.
Figure 4 plots the task error across the sparsity methods. We find that our strategy vastly outperforms competing approaches on all tasks and model sizes by a wide margin that increases with sparsity. Interestingly, the methods compared typically omit at least one of the recommendations listed in this work. For example, LOTTERY performs worse even when starting from a dense model because it does not rewire. Conversely, SET and RIGL underperform because they do not accumulate gradient updates. However, other works (Zhou et al., 2021; Hubara et al., 2021) that adopt concepts similar to those explored in this paper should perform similarly to our approach. As a result, leveraging short-term correlations seems to be a critical component for recent success in sparse training.

heading: 5.3 APPLICATION ON HARDWARE ACCELERATORS
text: We have shown earlier that we can train sparse neural models while maintaining accuracy. However, such models cannot be accelerated on modern hardware with current matrix-math pipelines (Park et al., 2017; Gale et al., 2020) without imposing particular structures (or positions of weights used during training and inference). On the other hand, neural structures targeting hardware acceleration (e.g., removing blocks (Gray et al., 2017), channels or filters (Wen et al., 2016; Li et al., 2017),
layers (Michel et al., 2019)) often degrade accuracy. This apparent tradeoff between accuracy and performance (or speed) has hindered their adoption.
Therefore, we explore whether our recommendations can make hardware-aware structures more amenable for deep learning. As a case study, we consider Sparse Tensor Cores introduced in NVIDIA Ampere GPU architecture (Mishra et al., 2021), which have twice the math throughput of regular matrix operations. The hardware expects a 2:4 sparsity structure that takes at least two values to be zero for each group of four values. Appendix E illustrates examples of 2:4 sparsity for inference (1D) and for training (2D).
Table 2 lists accuracy differences between sparse and dense models (positive values mean sparse performs better), where we extend learning rate schedules for some workloads compared to what is typically used in literature (see Appendix D for details). From the table, we find that structured sparse neural models can generally retain accuracy for tasks. While neural models adopting coarser structures such as block sparsity (Narang et al., 2017; Gray et al., 2017) fail to benefit from our recommendations and perform no better than smaller models (see Appendix E), using 2:4 for training (2D) and inference (1D) roughly matches accuracy of the dense models. Therefore, we can conclude 2:4 sparsity is particularly effective at leveraging short-term correlations due to its finer granularity. This suggests possible avenues towards accelerating training (Hubara et al., 2021) as well as obtaining efficient models for inference without having to repeat the training process (Mishra et al., 2021; Zhou et al., 2021).

heading: 6 RELATED WORK
text: Applying sparsity to reduce the size of neural models has been a topic of interest for the past three decades (LeCun et al., 1990; Hassibi & Stork, 1993; Reed, 1993; Castellano et al., 1997; Han et al., 2015; Jayakumar et al., 2020). Typically, sparse models are constructed by training much larger models (that are easier to train) and removing some of their weights either after training (Han et al., 2015; Mishra et al., 2021) or gradually alongside training (Narang et al., 2017; Zhu & Gupta, 2018; Gale et al., 2019). However, the above are only useful to reduce costs for inference.
For training acceleration, early works (Lee et al., 2019; Wang et al., 2020; Tanaka et al., 2020) sought to remove weights before training with limited success (Gale et al., 2019; Frankle et al., 2021). Surprisingly, it was shown sparse models can be trained when initialized the same way as the trained dense model (Frankle & Carbin, 2019). After this breakthrough, many works (Dettmers & Zettlemoyer, 2019; Evci et al., 2020a; Jayakumar et al., 2020) tried to dynamically learn sparse models by rewiring their weights throughout training, though some works existed earlier (Bellec et al., 2018; Mocanu et al., 2018). While this greatly improved accuracy, it was not enough to match that of dense models. More recent advances (Wortsman et al., 2019; Liu et al., 2020; Savarese et al., 2020; Zhou et al., 2021; Hubara et al., 2021) introduced gradient updates for all weights in a neural model, including ones that do not participate in training, and observed better success.
Our recommendations are similar to recent works (Zhou et al., 2021; Hubara et al., 2021) that were conducted in tandem with our research, but differ in some crucial aspects. (Zhou et al., 2021) adopts
2:4 1D sparsity patterns which cannot accelerate training. We also show alternatives to regularizing the non-participating weights which achieve the same effect. (Hubara et al., 2021) uses 4:8 sparsity patterns which are not useful for Sparse Tensor Cores, but they also adopt 2D patterns as we do.
While sparsity has been extensively explored for neural model training, our paper presents new perspectives on why certain methods should or not work. Our explanations about how additional weights do not learn meaningful information throughout training complement existing observations about the benefits of overparameterization (Evci et al., 2020b). We use this understanding to consolidate many ideas that have been circling around in sparsity literature but remain not clearly understood. For example, how frequently and why must weights be rewired.

heading: 7 CONCLUSION
text: In this paper we provide a better understanding of what is needed for sparse training. We first discover weights that can be pruned during inference do not learn meaningful representations over time, but exhibit short-term correlations that are necessary to train effectively. Based on this understanding, we uncover recommendations to remove such weights when training sparse models and conduct extensive empirical studies to determine the effects of different hyperparameters. We then demonstrate that sparse training can work across dozens of deep learning workloads that have not been investigated before in sparsity literature. Lastly, we are the first to employ sparsity patterns that can accelerate training of sparse models on existing hardware.
We believe these results open many questions. On the practical side, it may be interesting to consider how could this strategy be adopted to accelerate real-world training and inference workloads today, reducing the environmental impact from training very large models. On the theoretical side, we would like to understand better ways to approximate the short-term behavior in cases where accuracy still suffers as well as how to address potential biases induced by sparse models which may affect applications. We hope that our results will spur further research on these unconventional architectures, which challenge the default choice held by massive neural models today.

heading: A RECOMMENDATIONS FOR TRAINING
text: We expand our investigations on how to remove weights that do not learn meaningful information when training sparse models, covering more neural architectures and deep learning tasks.
Figure 5 plots the task error as a function of rewiring steps r for sparse models of different d. We observe that error increases with less frequent rewiring (r → ∞) for other vision tasks, since rewiring is related to how often traversal through the parameter space expands.
Figure 6 shows the task error as a function of the scale factor s applied to gradient updates for nonparticipating weights. We observe the error increases with decreasing contributions of the gradients (s→ 0), which suggests updates to non-participating weights are also important during training for other tasks.
Figure 7 demonstrates the task error as a function of the number of training steps z at which nonparticipating weights are reset to zero. Similar to results in Section 3.2, error rates saturate after sufficient training (z ∼ 1k), which reinforces the idea that non-participating weights augment parameter spaces rather than model capacity.
Figure 8 compares various exploration and exploitation strategies for training sparse models, as described in Section 3.3. While task accuracy degrades with lack of proper exploration or exploitation, inducing exploitation by removing gradient noise from non-participating weights (FIX, RESET, REGULARIZE) substantially decreases the error rates.
The above results across more deep learning workloads further validate recommendations put forth in Section 3 for training sparse models.

heading: B EXPERIMENTAL SETUP
text: We design experiments on PyTorch (Paszke et al., 2019) using custom autograd functions for convolutions and linear layers. Our functions emulate sparsity using a binary tensor (or mask) that we multiply elementwise with the weights of each layer during forward and backward propagation. We determine masks based on weight magnitudes, as described earlier, after the optimizer step and before the next training iteration.
B.1 IMAGE CLASSIFICATION
We train popular convolutional models like ResNets (He et al., 2016), VGG (Simonyan & Zisserman, 2015), Stacked U-Nets (Shah et al., 2018), Dilated Residual Networks (Yu et al., 2017), Inception (Szegedy et al., 2015), MobileNet (Sandler et al., 2018), as well as vision transformers like DeiT (Touvron et al., 2020). Training involves standard pipelines for image classification on ImageNet-2012 as described in literature and found in public code repositories. For most workloads, we adopt learning rates with linear warmups for the first 5 epochs, drop the learning rate by a factor of ten at epochs 30-60-80, and stop training after 90 epochs. A few select neural models (e.g.,
mobilenets and vision transformers), however, are trained for more epochs using linear or cosine schedules.
We measure model quality using top-1 classification accuracy. We apply sparsity to convolutions and linear layers with some exceptions: convolutions whose input channels are not divisible by 16 (e.g., first convolution layer, group and depthwise separable convolutions).
B.2 IMAGE SEGMENTATION AND DETECTION
Image segmentation and detection tasks include both regression and classification components. Popular detectors and segmentors are typically trained in two phases: first a backbone is trained for image classification, followed by the addition of model components that are trained for detection or segmentation. Backbones are trained on ImageNet-2012, while downstream tasks are trained on COCO. We adapt training scripts and code from Detectron2 (Wu et al., 2019).
We train neural models such as regions with convolutional neural networks (R-CNN) variants (Ren et al., 2015; He et al., 2017), vanilla one-shot detectors (Liu et al., 2015), and with focal loss (Lin et al., 2017). Convolution and linear layers encountered in pretrained backbones are sparse, like for classification tasks. Detection and segmentation heads are also targeted.
B.3 GENERATIVE MODELING
Generative Adversarial Networks (GANs) contain two subnetworks: a generative model and a discriminative model which combine regression and discrimination tasks during training. For image and video tasks, the generator model regresses pixel colors. We explore conditional GANs for super image-to-image translation and video-to-video synthesis using Imaginaire (NVIDIA, 2020b). We measure quality of generated outputs using the Frechet Inception Distance (FID). We experiment with generative neural models like Pix2PixHD (Wang et al., 2018b), Vid2Vid (Wang et al., 2018a), and FewShot-Vid2Vid (Wang et al., 2019), targeting convolution and linear layers.
B.4 MACHINE TRANSLATION
We explore transformer and recurrent neural models for language translation. All models are encoder-decoder style architectures trained for English to German (En-De) translation on WMT. We adapt model and training code from Fairseq (Ott et al., 2019) and NVIDIA Deep Learning Examples (NVIDIA, 2020a). We measure model quality using BLEU scores.
We experiment with GNMT (Wu et al., 2016) and transformer-based architectures (Vaswani et al., 2017; Gu et al., 2019). All linear layers are sparse, except for embeddings and vocabulary projections.
B.5 LANGUAGE MODELING
We consider recent advances in word-level language modeling using transformer decoder (left-toright) or encoder (bi-directional) architectures (Radford et al., 2018; Devlin et al., 2018). We pretrain language models in an unsupervised fashion on WikiText-103 or Wikipedia corpus, and evaluate on downstream tasks that are zero shot or require additional finetuning. We train them using Megatron (NVIDIA, 2020c) and NVIDIA Deep Learning Examples (NVIDIA, 2020a). Model quality is measured in terms of perplexity or F1 score.
We train language models such as Transformer-XL Dai et al. (2019) and BERT Devlin et al. (2018). We make all linear layers sparse, except for embeddings, vocabulary projections, and classification heads for downstream tasks.

heading: C COMPARISONS TO DENSE MODELS
text: It is also interesting to understand where does accuracy of our sparse models (SPARSE) fall between accuracies of dense (DENSE) and smaller neural models (SMALL). We define a metric f = (DENSE − SEARCH)/(DENSE − SMALL), where a value of one means accuracy matches that of dense, and zero implies accuracy is no better than a smaller model.
Figure 9 illustrates f as a function of model size d for various tasks. For d ∼ 0.5, sparse models are able to approximate dense models across all neural architectures and tasks. While f decreases for sparser models (d 0.5), they are still much more efficient than smaller models. Even at d = 0.1, most models can retain a large amount of accuracy.

heading: D EFFECTS OF LONGER TRAINING
text: Since conventional models can achieve higher accuracy when trained on larger data sets or with longer training schedules (Liu et al., 2019), another interesting direction is to explore the effects of sparsity on models that are trained to the limits of their capacity. We train neural models longer by extending their learning rate schedules after warmup , e.g. a schedule 1/t becomes 2/t.
Figure 10 (left) plots accuracy differences as a function of training time t, which denotes the ratio of training steps to the original training schedule. The fact accuracy improves with more training suggests neural models are often not trained to capacity using conventional schedules. We observe sparse models achieve worse accuracy than dense models that are undertrained (t ∼ 1), but can match accuracy when trained to capacity (t 1). Figure 10 (middle) illustrates task errors between dense and sparse models that have been trained for the same number of steps. We find error scales inversely with t, since more training gives neural models better chances to explore parameter spaces. Particularly, errors can reach zero with sufficient training (Evci et al., 2020a) for models that are not constrained by capacity (d ≥ 0.25). Figure 10 (right) shows the time it takes for sparse models to recover accuracy of dense models is relatively short when t ∼ 1, but significantly longer as accuracy saturates (t 1). Investigations on the effects of longer training are expanded to more neural architectures and deep learning tasks. Figure 11 illustrates accuracy deltas as a function of training time t. For vision tasks, we find sparse models of moderate sizes (d ≥ 0.25) can match accuracy of dense models after sufficient training (t ≥ 3). On the other hand, for language modeling, sparse models cannot match accuracy for any time t because dense models are already near capacity for the task at hand. Obviously, when using popular training schedules (t ∼ 1), sparse models can be trained a bit longer to recover the accuracy lost.

heading: E APPLICATION ON HARDWARE ACCELERATORS
text: This appendix explores various sparsity structures considered in the paper that are amenable for acceleration using modern matrix-math hardware.
E.1 BLOCK SPARSITY
We first consider block sparsity (Narang et al., 2017; Gray et al., 2017), which removes blocks of contiguous elements (or weights) in a neural layer as shown in Figure 12. Block sparsity addresses common issues that are present for unstructured formats: indices for active blocks reduce storage overhead by a factor of the block size, blocks are stored contiguously in memory which reduces irregular memory accesses, and their computations can exploit faster matrix-math hardware, such as Tensor Cores in NVIDIA GPUs.
We construct block sparse structures by (1) partitioning a neural layer into a set of blocks, (2) aggregating elements in each block into a metric, and (3) removing blocks according to some criteria based on their metrics. While we remove blocks based on largest magnitude maxi∈(1,b2) wi. Other
choices such as the p-norm ( ∑b2 i |wi|p)1/p achieve similar results.
Because structures restrict the combination of weights that can be formed in a neural model, an important question is then for what block sizes b (if any) can sparse models retain accuracy. Table 3 lists accuracy differences between sparse and dense models for d = 0.5 using various block sizes. We find block sparse models fail to maintain accuracy, performing no better than smaller models. Notably, accuracy deteriorates for all tasks and block sizes, including smaller blocks (b ≤ 4) that are less amenable for hardware acceleration. In other words, block structures are too coarse for retaining short-term correlations. For example, different weights in a block may have different roles during training: a block that participates in training may contain weights that are not important, wheres a non-participating block may have weights that were crucial to keep. Both cases prevent sparse
BLOCK SPARSE 2:4 SPARSE (1D) 2:4 SPARSE (2D)
models from retaining weights over time that are relevant for traversing the parameter space, and thus impacting accuracy.
E.2 2:4 SPARSITY
We next consider Sparse Tensor Cores (Mishra et al., 2021) introduced in NVIDIA Ampere GPU architecture which exploit 2:4 sparsity and have twice the math throughput of regular matrix units. Figure 12 shows that 2:4 sparsity mandates each group of four values must have at least two values that are zero. Typically, 2:4 is applied on weights w in the forward pass, y = wx. However, we can also apply 2:4 on weight transposes wT for the backward pass, ∂L/∂x = ∂L/∂y × wT . We denote these two options as 2:4 1D that accelerates forward pass for inference Mishra et al. (2021), and 2:4 2D that accelerates both forward and backward passes for training.
The 2:4 sparsity structure must always be imposed along the inner dimension of dot products. For linear layers, we apply 2:4 on a n × k weight tensor along k or n (for forward or backward pass, respectively). For convolutions, we apply 2:4 on a k× c× r× s weight tensor along input channels c or k× r× s (for forward or backward pass, respectively), where k denotes output channels, r and s are kernel dimensions.
We can satisfy 2:4 1D constraints by removing weights with lowest magnitudes. Since 2:4 2D constraints have no trivial solution, we seek to minimize the cumulative magnitude of the removed weights. In other words, for each 4 × 4 block in the tensor, we construct all possible combinations of 2:4 2D patterns, compute their 1-norm, and choose the structure that has the largest norm.

heading: F COMPARISONS TO SPARSITY RESEARCH
text: We expand our comparisons to literature covering more neural architectures and deep learning tasks. Figure 13 illustrates the task error across various methods, neural architectures, and tasks. We find our strategy outperforms competing approaches in most cases with a few exceptions: we do not
outperform lottery tickets in segmentation tasks because detectors and segmentors are trained with small learning rates which limit exploration. For generation tasks, our models are slightly worse for d ∼ 0.5 (though noisy scores make it difficult to draw conclusions), but the asymptotic behavior of error rates as d→ 0 clearly indicates that our approach is superior.

Review summaries:
This paper seeks to explore and understand the improved performance of Neural Networks trained with additional parameters even though a majority of the weights can be pruned during inference. They analyze the magnitude, correlation, and movement of weights during training and propose a hypothesis that the reason is that the weights which can be pruned during inference provide degrees of freedom in the loss landscape which allows better optimization of the non-prunable weights. Based on this, they explore and recommend a methodology for sparse training as a method to efficiently approximate a higher dimensional weight space that maintains these good optimization properties. They ablate the effects of the different recommendations and explore the hyperparameter space to have a better understanding of the dynamics of the sparse training.

This paper studies the training of sparse neural networks and the effect of extra parameters (dense connectivity) during training. They hypothesize that pruned connection are useful, because they help optimization to take some random steps in those direction which help optimization escaping the local minima. Authors support their claims with experiments and finally share a very long list of experimental results on training sparse networks.  

The authors proposed a new point of view to explain why deep learning models often require more weights are needed to train models rather than to run inference for tasks. One example is that sparse models usually perform well in inference, but do not perform well in training. The main content of this viewpoint is that more weights during training can expand the search space, create extra degrees of freedom for search, and form new optimization paths when model optimization falls into critical points, thereby facilitating model training. Based on this point of view, this paper proposes a series of steps to augment search spaces of sparse models during training to approximate the behavior of larger models. The experimental results show that this series of steps can effectively reduce the model error.

The domain is the optimization of deep and large neural networks. The author argues and presents information that supports (figure 1), a hypothesis that a large portion of neurons is pushed towards dormancy while optimizing a neural network and only a subset of the neurons are actively used. The authors, moreover, argue that these dormant weights are important during optimization as they can propose new solutions during training (figure 2). However, these dormant weights can also be an issue as they introduce gradient noise. The authors propose a new scheme to balance the role of the dormant weights in order to maximize new solutions while minimizing gradient noise. The authors do this by either setting them to zero at a certain iteration or regularizing their sizes. They only optimize a set amount of the weights at once (rewind). The authors show results on multiple benchmark datasets highlighting their performance and how new sparsity-oriented hardware works well with their algorithms. The dormitory vs active weights are picked by a threshold of their magnitude.