Dynamic Sparse Training with Structured Sparsity
Authors: Mike Lasby
Anna Golubeva
Utku Evci
Mihai Nica
Yani Ioannou
Venue: ICLR 2024 poster (2024)
Keywords: Machine Learning
dynamic sparse training
structured sparsity
N:M sparsity
efficient deep learning
RigL
SRigL
constant fan-in
dynamic neuron ablation
neuron ablation
structured and fine-grained sparsity
online inference
accelerating inference
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: SRigL: a dynamic sparse training method that learns structured and constant fan-in sparsity to match SOTA generalization performance while accelerating inference

Abstract: Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically less computationally expensive, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work, we propose a sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity by imposing a constant fan-in constraint. Using our empirical analysis of existing DST methods at high sparsity, we additionally employ a neuron ablation method which enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST performance on a variety of Neural Network (NN) architectures. Using a 90% sparse linear layer, we demonstrate a real-world acceleration of 3.4×/2.5× on CPU for online inference and 1.7×/13.0× on GPU for inference with a batch size of 256 when compared to equivalent dense/unstructured (CSR) sparse layers, respectively.

Published as a conference paper at ICLR 2024
DYNAMIC SPARSE TRAINING WITH STRUCTURED SPARSITY
Mike Lasby', Anna Golubeva Utku Evci", Milai Nica².6, Yani A. Ioannou' University of Calgary, *Massachusetts Institute of Technology, SIAIFI
'Google DeepMind, University of Guelph, Vector Institute for Al
ABSTRACT
Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training, matching the generalization of dense models while enabling sparse training and inference. Although the resulting models are highly sparse and theoretically less computationally expensive, achieving speedups with unstructured sparsity on real-world hardware is challenging. In this work, we propose sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity by imposing a constant fan-in constraint. Using our empirical analysis of existing DST methods at high sparsity, we additionally employ neuron ablation method which enables SRigL to achieve state-of-the-art sparse-to-sparse structured DST performance on a variety of Neural Network (NN) architectures. Using 90% sparse linear layer, we demonstrate a real-world acceleration of 3.4 x/2.5x x on CPU for online inference and 1.7 x/13.0x GPU for inference with batch size of 256 when compared to equivalent dense/unstructuree (CSR) sparse layers, respectively.
INTRODUCTION
Dynamic Sparse Training (DST) methods such as RigL (Evci et al., 2021) are the sfate-of-the-art in sparse training methods for Deep Neural Networks (DNNs). DST methods typically learn unstructured masks resulting in 85-95% fewer weights than dense models, while maintaining dense-like general- ization and typically outperforming masks found via pruning. Furthermore, sparse-to-sparss DST algo- rithms are capable of employing sparsity both during training and inference, unlike pruning and dense- to-sparse DST methods such as SR-STE (Zhou et al., 2021) which only exploit sparsity at inference time.
While models trained with DST methods are highly sparse and enable large reduction in Floating Point Operations (FLOPs) in theory, realizing these speedups on hardware is challenging when the sparsity pattern unstructured. Even considering recent advances in accelerating unstructured Sparse Neural Networks (SNNs) (Gale et al., 2020; Elsen et al., 2020; Ji & Chen, 2022), structured sparsity realizes much stronger acceleration on real-world hardware. On the other hand, structured sparse pruning often removes salient weights, resulting in worse generalization than comparable unstructured SNNs for the same sparsity level (Fig. la). Our work presents a best-of- both--hh--oo wet-botthhwlllee worlds approach: we exploit the DST framework to learn both a highly-sparse and structured representation while maintaining generalization performance. summary, our work makes the following contributions:
1. We propose novel sparse-to-sparse DST method, Structured RigL (SRigL), based on RigL (Evei et al., 2021). SRigL learns a SNN with constant fan-in fine-grained structured sparsity (Fig. la) while maintaining generalization comparable with RigL up to high sparsity level (99%) for a variety of network architectures. This structure is a particular case of "N:M sparsity" which requires N out of M consecutive weights to be non-zero (Mishra et al., 2021).
2. Our empirical analysis shows Rigl., at sparsity levels > 90%, ablates whole neurons. By allow- match Rigl. generalization even in this high-sparsity regime. 3. We enable neuron ablation in SRigl. across all sparsity regimes. We find this structured sparsity is complementary to the constant fan-in sparsity in improving real-world inference timings while maintaining generalization comparable to unstructured DST methods.
ing neuron ablation in SRigl.,
Imklaaby.yan ooniiiaoo ioannou] @ucalgary.ca, golubeva@mit.edu, evcu coou@googleccom. nicam @ @uoguelph.ca
Our source code is available here.

---

Published as a conference paper at ICLR 2024
most salient
least salient
maskad
Conot Per Layer Theary
Corst-Fand Theory
layer L+1
Conoh Pen- Experiment
Exporiment
unstructured pruning
conse. fan-in pruning
sparsity
0.5
0.15
0.8
0.10
0.05
Layer Width
(a) Constant fan-in pruning v.s. unstructured pruning.
(b) Output-norm variance analysis.
Figure 1: Constant fan-in pruning keeps the most salient weights per neuron. while unstructured pruning keeps the most salient weights per layer. A constant fan-in weight matrix has the same number of non-zo zero elements (here 2) per column allowing condensed representation. While pruning may remove salient weights affecting gen- cralization, with SRigL structure and weights are Icarned concurrently. (b) Output-norm variance: Theorctical predictions and simulation results (see Appendix A) demonstrating that sparse layers with constant fan-in have consistently smaller output-norm variance than layers with the sparsity but w/o the constant fan-in constraint.
4.
We demonstrate that constant fan-in sparsity enables compact representation that is not only parameter- and memory-efficieee but also amenable real-world acceleration. We observe significantly reduced real-world timings for online inference using our CPU-based Py implementation and for batched inference using a GPU-based implementation from Schultheis & Babbar (2023) over dense and unstructured baselines.
RELATED WORK
Dynamic sparse training Unlike with pruning, where weights are typically pruned after the dense network was trained (Han et al., 2015; 2016), or at initialization (Wang et al., 2020), DST methods learn the sparse connectivity during training by periodically adding and removing weights based on various saliency criteria. For instance, Sparse Evolutionary Training (SET) (Mocanu et al., 2018) removes weights with the smallest magnitude and adds weights randomly; similarly, RigL (Evei et al., 2021) prunes weights with the smallest magnitude and regrows weights that have large- magnitude gradients. Liu et al. (2021c) further improved the original RigL results by increasing the extent of the parameter space explored by modifying the sparse connectivity update schedule and drop rate.
Many recent works have examined the effect of different grow and prune saliency criteria on unstructured DST approaches, including SET, Deep Rewiring (DeepR) (Bellec et al., 2018), Sparse Networks from Scratch (SNFS) (Dettmers & Zettlemoyer, 2019), Dynamic Sparse Reparameterization (DSR) (Mostafa & Wang, 2019), Top-K Always Sparse Training (Top-KAST) (Jayakumar et al., 2020), and Memory-Econowd
Sparse Training (MEST) (Yuan et al., 2021 In Section 4, we compare SRigL to several of these methods. While the above-noted DST methods are highly effective at finding SNNs which reduce thcoretical inference cost, they result in unstructured SNNs which are difficult accelerate in practice common hardware architectures.
In a contemporaneous work, Yin et al. (2023) also identified the existence of sparse amenable channels in existing unstructured DST algorithms. Their method, Chase, achieves state-of-the-aee art generalization performance by including a soft memory bound similar to Yuan et al. (2021b) and calculating the saliency of parameters based global instead of layer-wise statistics. Chase requires that the structured sparsity level be set prior to training. In contrast, SRigL dynamically learns to ablate channels based on the number of remaining weights that are considered salient.
Accelerating unstructured sparse neural networks Elsen al. (2020) proposed method for accelerating unstructured SNNs based on one-dimensional tiling of non-zero elements, which demon- strated significant speedups on both Central Processing Unit (CPU) (Elsen et al., 2020) and Graphics Processing Unit (GPU) (Gale et al., 2020). However, like most approaches to accelerating unstructured SNNs, this method relies on imposing structure on an existing sparse weight matrix after training. Our method can be considered way of adding structure to SNNs during training, allowing the model to maximally utilize non-zero weights since structure and weights are learned concurrently.
2

---

Published as a conference paper at ICLR 2024
DeepSparse Engine (Neural Magic, 2021) accelerates inference of unstructured sparse networks on CPU by applying several innovations. In Appendix K, we compare our timings with SRigL to the DeepSparse Engine.
Learning block structured sparsity from scratch Block sparsity is a particular type of structured sparsity in which blocks of non-zero weights are grouped together in arrangements that reduce the memory overhead required to store the indices of the non-zero weights. Blocks can be generated out of contiguous weights in 1D (sometimes called tiles) or 2D or by utilizing a fixed number of non-zero weights per row or column group in the case of block-balanced sparsity (Hoefler et al., 2021). Spurred by the success of DST in learning unstructured sparse models, recent works have attempted to upply DST principles to learn block-structured sparsity. Jiang et al. (2022) introduced a novel -ware DST algorithm known as Dynamic Shuffled Block (DSB). DSB reshuffles non-zero weights into a block sparsity pattern after sparse connectivity updates, thereby improving memory access efficiency. Wall-clock speed-ups of up to 4x were reported with this method; however, generalization performance was reduced compared to Rigl. at comparable sparsities. Dietrich et al. (2022) applied modified variant of RigL to BERT models (Devlin et al., 2019). The resulting method is capable of learning models with block-structured sparsity.
Learning N:M structured sparsity from scratch N:M sparsity is specific form of block-balanced sparsity in which ID blocks with M contiguous elements contain exactly N non-zero elements. N:M sparsity is particularly amenable to acceleration and several attempts have been made to train models with N:M fine-grained structure using DST methods.
Yang et al. (2022) extended the DST method proposed by Liu et al. (2021b) to train multiple sparse sub- networks sampled from single dense super-network. Their proposed method, Alterntting Sparse Training (AST), switches the network topology between sparse sub-networks after each mini-batch during training. Yang et al. (2022) demonstrated state-of-the-arr performance several typical sparse training benchmarks. However, the dense model weights and gradients are required throughout the majority of training, greatly increasing the overall compute and storage requirements. While AST demonstrated a tantalizing possibility of training multiple sparse sub-networks within a single training loop, the gradual
to-sarse training paradigm used by (Liu et al., 2021b) is not directly comparable to Rigl. or other similar end end-to-end sparse DST methods.
Zhou et al. (2021) explored how N:M sparsity can be achieved during training using magnitude- pruning during the forward pass and a Straight- Throughh Estimator (STE) (Bengio et al., 2013) on the backward pass. In their method, the dense network weights are projected into a sparse network during each training iteration. The sparse network is obtained by selecting the top-N out of every M contiguous weights and STE is used to propagate the approximated gradients through the projection function. A regularization term is applied to the gradients of pruned weights to reduce instabilities during training. Their approach- - Sparse-Refinge Straight- Tarhhtt Estimator (SR-STE) - was applied to networks with N:M ratios of 1:4, 2:4, 2:8, 4:8, 1:16.
Although SR-STE utilizes sparse operations in the forward pass and can find sparse models optimized for inference, it does not reduce the training cost significantly. Specifically, SR-STE training requires (1) storing original parameters in their dense format, and (2) calculating dense gradients during each train- ing iteration. This makes SR-STE training as expensive as the original dense training in terms of memory and compute cost'. On the other hand. DST methods such as RigL, and our proposed method SRigL, are capable of end-to-end sparse training and use sparse parameters and gradients throughout training.
Accelerating fine-grained N:M structured sparsity Nvidia (2020); Mishra al. (2021) introduced the Ampere Tensor Core GPU architecture (e.g. A100 GPUs) and proposed the 2:4 fine-grained structured sparsity scheme that enables SNNs to be accelerated on this hardware at inference time. This scheme places a constraint on the allowed sparsity pattern: For every contiguous array of four weights, two are pruned, yielding 50%-spars net. The resulting regular structure of the weight matrix allows one to compress it efficiently and to reduce memory storage and bandwidth by operating on the nonzero weights only. Since the focus is on acceleration at inference time, the authors proposed to use the standard method of magnitude- bassed based pruning post training to achieve the 2:4 sparsity. Importantly, this work considered exclusively the 2:4 ratio; other N:M ratios cannot be accelerated on Ampere GPUs
'To be precise, SR-STE can use some sparse operations and reduce training cost up to two thirds of the original dense training. However this is still far from fully sparse acceleration for training.
3

---

Published as a conference paper at ICLR 2024
Constant fan-in N:M structured sparsity The constant fan-in constraint represents a special case of N:M sparsity where N is the number of non-zero weights per neuron and M is the dense fan-in for each neuron within given layer. While commodity hardware acceleration currently exists only for 2:4 sparsity on Nvidia's Ampere and later architectures (Mishra et al., 2021), a constant fan-in constraint can also take advantage of the efficient memory access and throughput increase that N:M sparsity yields, as recently demonstrated by Schultheis & Babbar (2023). Constant fan-in sparsity has several attributes which differentiate from N:M sparsity:
Constant fan-in sparsity is more flexible than N:M sparsity, enabling arbitrary global sparsity values to be applied to the mode whereas N:M sparsity is limited specific sparsity ratios.
With the constant fan-in constraint, per-layer sparsity distributions such as Erdös-Réón Rényi-Kernel (ERK) can be applied to the model. The ERK distribution has been demonstrated to outperform uniform sparsity distributions by reallocating parameters to layers with fewer parameters (Mocanu et al., 2018; Evci et al., 2021). In contrast, N:M sparsity can only be applied with uniform sparsity distribution.
Hardware support for acceleration of N:M sparsity is currently limited to 2:4 sparsity on Nvidia GPUs, offering a modest acceleration on the order of x2. contrast, the potential promise of highly sparse models (>=90% sparsity) to be x 10 faster than an equivalent dense model. As we demonstrate in Section 4.4 and Appendix I, our condensed sparse representation with constant fan-in sparsity can achieve significant acceleration over wide range of sparsities even without specialized hardware.
Online inference In many applications, DNNs are used in an online manner, i.e. by using only single inputs and not batches of inputs. Online inference is common in real-time and latency-sensinnn applications, or applications without significant numbers of simultaneous requests allowing batching. Online inference, especially for real-time applications, does not typically benefit from accelerators such as GPUs that require host to device transfers, since the cost of the transfer itself often negates any benefit in compute. Accelerating online inference workloads remains an open research problem, with many systems engineering solutions proposed to achieve acceleration (Kumar et al., 2019; Li et al., 2020; Wang al., 2022; Wu et al., 2020). Our condensed representation CPU implementation. which exploits both structured and constant fan-in sparsity. offers complimentary, orthogonal solution to these engineered solutions by directly accelerating model inference for single samples.
3
METHOD
Our goal in this work is introduce structural constraints the sparse mask learned by RigL. in order to make it more amenable to acceleration at inference time while not affecting RigL's generalization performance. We first performed theoretical analysis to explore the effect of various sparsity distributions with different degrees of structural constraints on the training dynamics of SNNs, detailed in Fig. la and Appendix A. Based on this analysis, we did not find any evidence to suggest that the constant fan in constraint would impair SNN training dynamics and performance, motivating the use of constant fan-in sparsity in our method outlined in Section 3.1.
3.1 STRUCTURED RIGL
As motivated by Appendix A, propose to enforce the constant-fan-in constraint within sparse-to-sparse DST method to learn structured sparse connectivity from scratch. Specifically, we use RigL by Evci al. (2021), which can obtain highly sparse networks with generalization performance comparable to their dense baselines.
In brief, the methodology of RigL is to update the SNN connectivity during training by praning weights with the smallest magnitude and regrowing those with the largest corresponding gradient magnitude in each layer. This occurs in periodic, but relatively infrequent mask update steps throughout most of training. In SRigL, weight saliency must be determined at the neuron level (in convolutional layers, the level of each filter), since we enforce that every neuron (output channel) has the same number of unmasked incoming weights, thereby satisfying the constant fan-in constraint. (Fig. la).
However, this approach alone significantly lags behind RigL's 's generalization at very high sparsities (>90%) and with transformer architectures, as shown in Fig. 3a and Table 4. This is because the constant fan constraint has an important side-effect: under a strict constant fan-in constraint, neurons
4

---

Published as a conference paper at ICLR 2024
layer L+1
most salient
ablation
ablation
least salient masked
layer
Figure Neuron ablation. At sparsity levels over 90%, RigL learns to completely mask (ablate) large number of neurons within each layer, effectively reducing layer width. Imposing constant fan-in in constraint requires all neurons to have the same numher of (non-pruned) incoming weights and therefore inhibits ablation, which results in worse generalization performance than Rigl.. Allowing SRigl. ablate neurons restores -level performance.
can never be entirely masked (ablated), as illustrated in Fig. 2. At very high sparsity levels this can lead to many neurons that have only 1-2 weights, limiting the capacity to learn complex features and consequently reducing generalization performance. Indeed, at high sparsities we observed empirically that RigL ablates large numbers of neurons (Figs. 3b, 11 and 12). Effectively, RigL. reduces the width of the model af high sparsities to maintain generalization performance; we believe we are the first to explicitly identify this behaviour within a DST method. To resolve this issue in SRigL, we implement neuron ablation method, allowing SRigL to maintain both a constant fan-in constraint and to reduce layer width at high sparsities. We introduce a new hyperparameter, 7sal. which defines the required minimum percentage of salient weights per neuron. Given neuron with constant fan-in of k; if fewer than Yea! & weights are considered salient by either the drop or grow criteria, then the neuron ablated and its weights redistributed to other neurons within the same layer. Notably this neuron ablation method allows SRigL to exploit neuron ablation structured sparsity at much lower sparsity levels than we identified it occurring at in RigL, while maintaining good generalization, demonstrated in Table 4.
The steps below outline our final SRigL method with neuron ablation. In the following procedure, the first two steps the same as in RigL. while the other steps are specific to SRigL, containing modifications to include the constant fan-in constraint and dynamic neuron ablation. We first set an ablation threshold Ysal- Then, for each layer we do the following:
1. Obtain magnitudes of the active weights and gradient magnitudes the pruned weights; these will serve as prune and growth criteria, respectively.
2. Compute K, the number of weights to be grown and pruned in the current step in this layer. prune.
We always grow the same number of connections
3. Count the number of salient weights per neuron. A weight is considered salient if it is in the top-K of either the maggnitde weights the largest-magnitud gradients.
4. Ablate neurons that have fewer salient weights than Ysal *k, where k is the fan-in. Ablation is done by pruning all incoming weights. These pruned weights are redistributed to the remaining neurons in the following steps.
5. Compute the new constant fan-in constraint, k', based on the number of ablated neurons.
6. Prune the K maalesttmmagnnne weights in the current layer. Note that this pruning criterion considers all weights within a layer rather than pruning only the smallest weights in each neuron.
7. For each active neuron, regrow as many weights as required, proceeding in order of decreasing gradient magnitude, until the target fan-in, K, is achieved.
4 RESULTS
We implement SRigL in PyTorch by extending existing implementation of RigL (McCreary, 2020). We evaluate our method empirically on image classification tasks: on the CIFAR-10 dataset (Krizhevsky, 2009) we train variant of ResNet-18 (He et al., 2016) suitable for CIFAR-10 and Wide ResNet-22 (Zagoruyko & Komodakis, 2017); on the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC-12) dataset (Russakovsky al., 2015) commonly referred to as ImageNet- - we train ResNet-50 (He et al., 2016), MobileNet-V3 (Howard et al., 2019), and Vision Transformer (ViT-B/16) (Dosovitskiy et al., 2021). See Appendix C and Appendix D.4 for Wide ResNet-22 and MobileNet- V3 experimental results, respectively.
Unless noted otherwise, we use the same hyperparameter configuration the original RigL method. A detailed summary of our hyperparameter settings and training details can be found in Appendix D.
5

---

Published as a conference paper at ICLR 2024
78
100
72
Test Accuracy (%)
Active Nerons (%)
90
80
RigL
60
SRigL ablation
70
RigL
SRigL
SRigL w/o ablation
SRigL x2
SRigL
54
SRigL x5
60
SRigl. x2
Dense benchmark
SRigl x5
50
75
80
85
90
95
100
80
85
90
95
100
Sparsity (%)
Sparsity (%)
(a) ResNee-500/mageNet
(b) Neuron Ablation
Figure 3: (a) ResNet-50//mageNet top-1 test accuracy when trained with SRigL. for a range of sparsities is comparable to RigL. Extended training durations of x2 and x5 are also reported for SRigL. Results reported single runs. (b) Neuron ablation: The percentage active neurons (i.., ablated) following RigL/SRigL training 50/mmaeNe RigL ablates large number neurons high sparsities.
We set the ablation threshold, Yealt to 30% for all SRigL results, except for our ViT-B/16 experimens This value was selected based on a hyperparameter sweep performed by training ResNet- and Wide ResNet-22 on the CIFAR-I 10 dataset, see Appendix E.
4.1 RESNET-I 8 TRAINED ON 10
We use a variant of ResNet-18 18 with reduced kernel dimensions and stride in the first two convolutional layers obtain model suitable for CIFAR- 10; our training regimen generally follows Evci (2021), see Appendix D.1 for more information. We repeat training with five different random seeds for both methods and report the mean and 95% confidence interval compared a densely-connectee model in Table 2. These results confirm that imposing constant fan-in constraint during sparse training does not significantly degrade generalization performance of the SNN compared the Rigl. method. In Fig. 11 plot the number of neurons ablated at ablation thresholds of 0%, 30%, and 50% to demonstrate how the Yeal hyperparameter can be used to guide the final model width during training.
benehmark
4.2 RESNET-50 TRAINED ON IMAGENET
Our training regimen for the ImageNet dataset generally follows Evei et al. (2021), see Appendix D.2 for more details. We investigate the effect of extended training with x2 and x5 the original number of training epochs. We train each model with a single seed and report the results Fig. and Table 1.
yields similar generalization performance Rigl. across each sparsity and training duration considered. At high sparsities, SRigL with ablation outperforms SRigL without ablation, highlighting the importance of neuron ablation as sparsity increases. Notably, RigL x5 results 99% sparsity in Evci et al. (2021) used a dense first layer, unlike all other results reported in Table 1. Despite this difference, SRigL x5 at 99% sparsity comparable the RigL x results. We expect that the 99% sparse models would be improved by using dense first layer for all SRigL results. Similar to RigL, we observe that SRigL generalization performance improves with increasing training time.
We inspect the connectivity of ResNet models trained with the Rigl. method and find, as shown in Fig. 3b, that at 95% sparsity 10.9% of neurons are removed completely. Thus, Rigl. results in fewer, but more densely connected neurons, whereas the fan-in constraint enforces that all neurons are retained.
In Table we compare SRigL to variety of DST algorithms. SRigL performs comparably other methods, even those which learn unstructured sparsity. Methods with memory footprint listed as dense require training with the dense network and therefore are not directly comparable other sparse-to-spare DST methods. The most directly comparable method to ours is DSB; we note that SRigl. outperforms DSB at all sparsity ratios reviewed.

---

Published as a conference paper at ICLR 2024
Table 1: Top-1 ImageNet test accuracy of ResNet-50 trained with Rigl. or SRigl. at high sparsities and with various training times (as in Evei et al. (2021)), e.g. 5x training epochs than dense ResNet-50.
Table Test accuracy for ResNet-18 CIFAR-10 trained with Rigl. or SRigl. with/without neuron ablation at varying sparsities repeated with five different random seeds.
RigL.
SRigL.
RigL
SRigL
sparsity
w/o
w/ ablation
sparsity
w/o
w/ ablation
(%)
1x
5xt
1x
1x
2x
5x
(%)
80
74.9
77.1
74.8
75.0
76.5
77.2
80
95.2
95.2
95.
90
72.8
76.6
72.6
72.7
74.7
76.2
90
95.1#0.1
95.0=0.1
95.1
95
69.6
74.6
68.8
69.1
71.5
73.6
95
94.6#0.2
94.5++0.3
94.7=00.2
99
51.4
61.9*
48.7
51.5
55.3
59.0
99
92.9
91.5+0.3
92.8
0
dense ResNet-50:
76.7
0
dense ResNer-18:
95.5
x RigI. are from Evei al. (2021)
uses a dense first layer, unlike results
results
Table 3: Top-1 ImageNet test accuracy of ResNet-50 trained with variety of DST methods, highlighting methods that both sparse-to-sparse (i.e. sparse training) and learn structured sparsity similar to SRigL. only DSB-16 (2:4 and sparsity) directly comparable in this regard. RigL and SRigL. results are from our experiments, other values are obtained from each method's corresponding paper, unless noted otherwise.
training
sparsity
method
method
structured
50%
75%
80%
90%
93.75%
Static
sparse no
7..6
65.8+0.04
SET
sparse
7.39
69.6=0.23
DeepR$
sparse no
71.7
70.2
DSR
sparse no
73.3
71.6
Top-KAST
sparse
74.76
70.42
MEST
sparse no
75.39
72.58
RigL
sparse no
74.98
72.81
D6B-16
sparse yes
76.33
74.04
Chase ft
sparse yes
75.27
74.03
SRigL (Ours)
sparse yes
76.60
75.55
75.01
72.71
70.56
SNFS (ERK)*
dense no
75. 73.2
0.12
73.0
AST+GC*
dense
no
73.1
SR-STE
dense yes
76.2
71.5
dense ResNer-50:
76.7
* Values obtained from Evei et al. (.021). $values obtained from Mostafa & Wang (2019). +Values for the MEST (x0.6 (xM.67+EM) variant, matched to the same number of training FLOPs RigL. &Values tabulated for Top-KAr KAST correspond the backwards sparsity as HOp-KAST uses different sparsities in the forward backward passes. For more information see Table in Jayakumar et al. (2020). Values from Yin et al. (2023) for channel sparsity (S.) set to 40%. *+50% initial sparsity. Values from Yang et al. (2022)
Table 4:
Top-1 test accuracy of VIT-B/16 trained on ImageNet with w/o neuron ablation
Table 5: SRigL sparsity and FLOPs for ResNet-500/mageNer training and inference. See Appendix G for more details.
RigL
SRigL
SRigL FLOPs
sparsity (%)*
w/o
w/ ablation
sparsity (%)
training (x1e18)
inference (x 1c9)
80
77.9
73.5
77.5
80
1.13
3.40
90
76.4
71.3
76.0
90
0.77
1.99
dense ViT-B/16:
78.35
95
0.40
1.01
+Sparsity level set for all modules except multi-headed attention input projections, which remain dense. See Appendix D.3 for more details.
99
0.09
0.21
0
3.15
8.20

---

Published as a conference paper at ICLR 2024
4.3 VISION TRANSFORMER TRAINED ON IMAGENET
We train the vision transformer variant ViT-B/16 on ImageNet generally following the original training recipe per Dosovitskiy et al. (2021) with select modifications, see Appendix D.3 for more information.
Similar to our Convolutional Neural Network (CNN) experiments, RigL ablates significant number of neurons when applied to the ViT-B/16 architecture with sparsities of 80 and 90%. Additionally, we find that RigL learns sparse connectivities with high variance of fan-in between neurons (see Fig. 12). At 90% sparsity, some neurons are allocated up to x10 more active weights than the mean number of active weights in the same layer. We hypothesize that these more densely connected neurons found in our Rigl. experiments are important for generalization performance; therefore, a high "sol threshold should improve performance of SRigI. by ablating neurons until sufficient density of sparse fan-in is reached. Indeed, we find that SRigL's generalization performance is sensitive to "Yeal and that high Yood thresholds 90% to 99% perform best. See Fig. 9a and Appendix E for more details on how Yaal affects the generalization performance of ViT-B/16. For the following results, we used a Ysal of 95%.
We train each model with single random initialization and report the results in Table 4. SRigL without ablation is unable to match the generalization performance of RigL at very high sparsity. However, with neuron ablation enabled, SRigL's performance greatly improves and is closely comparable to Rigl. at 80% and 90% sparsity.
4.4 ACCELERATION OF CONSTANT FAN-IN SPARSITY
Algorithm 1 "Condensed" linear layer with constant fan-in sparsity forward pass
I: Input: the input matrix of shape (batch_aa sizc,
w: the condensed weight matrix of shape nun_features)
constant_fan
3:
indx: indices of non zero dense weights of shape constant_fan_
4:
5: output --ttorch.zeroo (size= ze- (batch_siat si neurons))
6: forb range (batch_size) do
For each sample in mini-batch >. each active neuron in layer & For each non-zero weight
for in range (neurens)
8: for in range ( (constant_ Battttant_faa____ fan iin do
9: source _ix <- idx/nn, kl
10:
feature + x[b, source_idx]
11:
output [b, n] += feature # w[n, k]
12:
return output.
While SRigL shows promising theoretical speedups (i.e. FLOPs) demonstrated in Table 5 and Appendix G, FLOPs are limited in demonstrating the real-world acceleration potential of a proposed sparse representation in general. Yet conversely, creating fully-optimized software or hardware implementation of a novel representation typically requires significant engineering effort outside of the scope of this paper.
Here we show that even straight-forward forward Py implementation of our proposed condensed neural network representation (see Appendix F) can demonstrate this real- world acceleration. The algorithm to accelerate our condensed sparsity representation is shown in Algorithm 1, demonstrating that embarrassingly parallel. Additionally, leveraging CUDA kernels from Schultheis & Babbar (2023), we also demonstrate that constant fan-in sparsity can be accelerated on commodity GPUs.
To accelerate our condensed linear layer we exploit both structured and constant fan-in sparsity by removing ablated neurons and zero-valued weights from active neurons. In Fig. 4, we present real- world timings comparing our condensed linear layer structured and unstructured sparse representations. We extract the trained layer weights and bias from ViT-B/16 models trained with SRigL to obtain an accurate representation of the sparse topology produced during real training run with SRigL.
Our condensed representation is significantly faster than the dense benchmark and other sparse representations across all sparsities investigated. This real-world speed-up immediately applicable to applications where latency is critical. In some instances, we found structured sparsity yields the best acceleration. By including both structured and constant fan-in sparsity, models trained with SRigL can either the fully condensed (structured constant fan-in) or purely structured sparse representations to obtain real-wort acceleration across broad range of applications with the same set of weights.
8

---

Published as a conference paper at ICLR 2024
200
Dense
80
90
95
99
175
Time (uS)
125
100
75
50
SRigL (ours)
Structured
Unstructured
(a) CPU online inference
104
Dense
10a
Time (μs)
10a
10a
SRigL (aurs)
Structured
Unstructured
(b) GPU inference with batch size of 256
Figure 4: Comparing real-world timings for a fully-connected layer extracted from a 6/T-B/16 model trained with SRigL when compressed using the condensed representation learned by SRigL to structured (i.c. the same layer accelerated using only the ablated neurons without exploiting the fine-grained sparsity). and unstructured (i.e. minimum of
Compressed Sparse Row (CSR) representations. The median
shown, while the to
bars show std. dev. Note: the increased timings for the 95 & 99% sparse structured representations SRigL ablating relatively fewer neurons at these sparsities compared to 80 and 90%. (a) CPU wall-clock timings for online inference on Intel Xeon W-2145. For online (single input) inference, our condensed representation 90% is 3.4x faster than dense and 2.5 faster than unstructured sparsity. See Appendix I. (b) GPU wall-clock timings for inference with batch size of 256 on NVIDIA Titan V. At 90% sparsity, our condensed represen- tation 1.7x faster than dense and 13.0x faster than unstructured (CSR) sparse layers. Note y-axis log-scaled.
See Appendix I and Appendix J for details wall-clock benchmarks across range of threads and batch sizes. Furthermore, we expect that a more optimized software implementation and/or explicit hardware support would enable use of SRigL across wider range of applications.
5 CONCLUSION
In this work present SRigL, novel DST method that learns sparsity mask incorporating both structured and constant fan-in sparsity. SRigL is capable sparse-lo-sparse training while maintaining generalization performance par with state-of-the-ar unstructured sparse training methods on wide variety of network architectures. Our observation that RigL ablates neurons at high sparsities inspires neuron ablation method which enables SRigL match the performance of RigL, sparsities and on the ViT-B/16 network architecture. SRigl.'s constant fan-in constraint and neuron ablation results in real-world acceleration for CPU online inference and GPU batched inference. We hope this work will motivate the implementation of additional fine-graines structured sparsity schemes and the engineering efforts required accelerate them further.
high

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We acknowledge the support of Alberta Innovates, the Natural Sciences and Engineering Research Council of Canada (NSERC), and the NSF Al Institute for Artificial Intelligence and Fundamental Interactions (IAIFI). We are grateful for computational resources made available to by Deavr Dataworks, Google, Amazon, and the Digital Research Alliance of Canada. We also acknowledge the very helpful feedback of Erik Schultheis and Trevor Gale.
REFERENCES
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein, Deep Rewiring: Training very sparse deep networks. In International Conference on Learning Representations, Feeerna 2018. URL /eeetts::/opennee openrevieneeeeeeeeeeeeeeeereeeee net/ Soorumiid-2a Bid--u_wN--------- wNo1C-
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv pre-print, 2013.
Ting-Wo Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression via learned global ranking. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), June 2020.
Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. AutoAugment: Learning Augmentation Policies from Data, April 2019. URL httggabb//80// 09501. 1rXiv:1805.09500 [cs, stat].
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoc Le. RandAugment: Practical Automated Data Augmentation with Reduced Search Space. In Advances in Neu- ral Information Processing Systems, volume 33, pp. 18613-18624. Curran Associates, Inc., 2020. URL https: neurips ec cc/ pttpp//ppp////////////// /hash/ Abst html.
Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance. Technical Report arXiv: 04840, arXiv, August 2019.
Jacob Devlin, Ming-Was Chang, Kenton Lec, and Kristina Toutanova. of Deep Bidirectional Transformers for Language Understanding, May 2019. URL http:/ // /arxivv....//////////aaa////avvv wttp//og/////////ooooo 0400000000..
BERT: Pre-training
04805. arXiv:1810.04805 [cs].
Anastasia S. D. Dietrich, Frithjof Gressmann, Douglas Orr, Ivan Chelombiev, Daniel Justus, and Carlo Luschi. Towards Structured Dynamic Sparse Pre-Training of BERT. Submit- ted to the International Conference on Learning Representations, January 2022. URL https:://pennre htteriiwwwwweeeeeeee forum? iid 7awddWisOc.
Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. Advances in
Neural Information Processing Systems, 32, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations, January 2021. URL pettps:/copenreet nete /tto//oodddiiiiibbioeo
Erich Elsen, Marat Dukhan, Trevor Gale, and Karen Simonyan. Fast sparse convnets. In Proceedings the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners. Technical Report arXiv:1911.11133 arXiv, July 2021.
arXiv:1911.11134.
Utku Evci, Yani Ioannou, cem Keskin, and Yann Dauphin. Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win AAAI 2022 Poster, February 2022.
Trevor Gale, Matei Zaharia, Cliff Young, and Erich Elsen. Sparse gpu kernels for deep learning, 2020.
10

---

Published as a conference paper at ICLR 2024
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour, April 2018.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, 2015.
Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recog- nition. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 770-778, Las Vegas, NV, USA, 2016. IEEE. ISBN
doi: 10. 1O9/C/P...00........
Yang He, Ping Liu, Ziwei Wang, Zhilan Hu, and Yi Yang. Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks. arXiv:2/02.00554 [cs], January 2021. URL httthttp////aree org/ /abg///b//// 22102.2 00554. arXiv: 2102.00554.
Andrew Howard, Mark Sandler. Grace Chu, Liang-Chies Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
Eugenia Iofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well do sparse imagenet transfer? CoRR, abs/2111. 2021. URL https: // arxiv. org/ abs 13445.
Sebastian Jaszezur, Aakanksha Chowdhery, Afroz Mohiuddin, Lukasz Kaiser, Wojeiech Gajewski, Henryk Michalewski, and Jonni Kanerva. Sparse is enough in scaling transformers. In A. Beygelz- imer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URLhttps: https //tptreei///peeeeiiieee net pnee/fforumm iid 2rtrrmmmdd--00SCydd00
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-KAST: Top-K Always Sparse Training. In Advances Neural Information Processing Systems, volume 33, pp- 40744-207544 Curran Associates, Inc., 2020.
Bo Ji and Tianyi Chen. FSCNN: A Fast Sparse Convolution Neural Network Inference System,
December 2022.
Peng Jiang, Lihan Hu, and Shihui Song. Exposing and Exploiting Fine-Grained Block Structures for Fast and Accurate Sparse Training. In Proceedings of the Neu- ral Information Processing Systems Conference (NeurIPS), October 2022. URL https: http::://ppenae https///ppeeeviieeeeee .nt/ www./eeeeeeaeeeeeeeeeeeeeeeeeeee /formmiddssFapsuuht...
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, University
of Toronto, 2009.
Adarsh Kumar, Arjun Balasubramanian, Shivaram Venkataraman, and Aditya Akella. Ac- celerating deep learning inference via freezing. In 11th USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 19), Renton, WA. July 2019. USENIX Association. URL https::/ /ttt:///wwwww http://wueii..orr......oe http:///oooeeee///oeccceeeeeeeee wewrrenceehoteloouutt wtt.:oo///e.ppeeeettttttoo/////e
www.uuseeii.... htttee////.w..uneeneoooooooootrp
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks. In Proceedings of the 37th International Conference on Machine Learning, pp. 5533-5543. PMLR, November 2020. URL html. ISSN: 2640-3498.
https::/
mlr, www.ppsssviiig/

---

Published as a conference paper at ICLR 2024
En Li, Lickang Zeng, Zhi Zhou, and Xu Chen. Edge AI: On On-Demand Accelerating Deep Neural Network Inference via Edge Computing. IEEE Transactions on Wireless Communications, 19(1):447-457, January 2020. ISSN 1558-2248. doi: 10.1 10.1109TTC.2219929994 29900.9999944..... URL / /eexxeeeepplle ore. .ee.org/
Conference
Name: IEEE Transactions Wireless Communications.
Mufan Li, Mihai Nica, and Daniel M. Roy. The future gagussian::: Resnets and their infinite- depth- and-width limit at initialization. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021.
Mingbao Lin, Rongrong Ji, Yan Wang, Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1529-1538, 2020.
Etai Littwin, Ben Myara, Sima Sabah, Joshua Susskind, Shuangfei Zhai, and Oren Golan. Collegial ensembles. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18738- 18748. Curran Associates, Inc., 2020.
Kuang Liu. pytorch-cifar, 2017. URL https: /ttp::://gttttt com/ comm//uanggiiu/pyyoocch-oee farr.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference Machine Learning, pp. 7021-7032. PMLR, 2021a.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, 7ahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Sparse Training via Boosting Pruning Plasticity with Neuroregeneration. In Advances in Neural Information Processing Systems, volume 34, pp. 9908-9922. Curran Associates, Inc., 2021b.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do We Actually Need Dense PareeParamttriizaaoonn In-Time Parreaaramettiizttoonn Sparse Training. In Proceedings of the 38th International Conference on Machine Learning, pp. 6989-7000, PMLR, July 2021c. URL https::///// http:///ooceedengeeeeeee mlr. poress/v1390 wttp//1191///// httm.. ISSN: 2640-3498.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In In-
ternational Conference on Learning Representations, December 2018.
URL https:
//openreview..o/////// http:/nee/ooooeeoo hetpe//eettttttttteeeeeeee
Maintainers and Contributors. Torchvision: Pytorch's computer vision library. https:
2016.
Dyllan McCreary. Pytorch implementation of rigging the lottery: Making all tickets winners, Nov 2020. Re-impleeenntationeexteeiiiii of the work done by Google Research: https://gitubb.co//eee research/rigl.
Asit Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius, Accelerating Sparse Deep Neural Networks, April 2021, URL http:// www.xxx...rg/as///oe/e
08378. arXiv:2104.08377 [cs].
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1):2383, December 2018. ISSN 2041-1723.
doi: 10.1033/441447-01--4444
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In Proceedings of the 36th International Conference on Machine Learning, pp. 4646-4655. PMLR, May 2019. ISSN: 2640-3498.
Neural Magic. Deepsparse engine: Sparsity-aware deep learning inference runtime for CPUs, 2021. URL http:: /ttp//ittttt thub. com/ neura aaw.ralmmiiiom c/ deepsparse. /eepppp....ee.e.p....e..ee....
Nvidia. Nvidia A 100 Tensor Core GPU Architecture, Technical report, Nvidia, 2020.
12

---

Published as a conference paper at ICLR 2024
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
Erik Schultheis and Rohit Babbar. Towards Memory-Efficient Training for Extremely Large Output Spaces - Learning with 500k Labels on a Single Commodity GPU, June 2023, URL attp://arxive org/ http/abs/22 /206 03725. arXiv:2306.03725 [cs].
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhut- dinov, Dropout: A Simple Way to Prevent Neural Networks from Overfitting Jour- nal of Machine Leurning Research, 15(56):1929-1958, 2014. ISSN 1533-7928. URL html.
.rgg wwwppss/vi/sssiiaaeaaeeeeepseeee
Xiu Su, Shan You, Tao Huang. Fei Wang, Chen Qian, Changshui Zhang, and Chang Xu. Locally free weight sharing for network width search. arXiv preprint arXiv:2102.05258, 2021.
Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- based pruning for compact neural networks. Advances in Neural Information Processing Systems, 011224644246116 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2826,, June 2016. doi: 10. 109/CVPR.2016.000
ISSN: 1063-6919.
Yehui Tang. Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning, Advances in Neural Information Processing Systems, 33: 13:10936-19947 2020.
Tijmen Tieleman, Geoffrey Hinton, et Lecture 5.--rmopo::: of its recent magnitude. COURSERA: Neural networks for machine learning, (12):26-3,, 2012.
Divide the gradient by a running average
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In ICLR, 2020.
Zehuan Wang, Yingcan Wei, Minseok Lee, Matthias Langer, Fan Yu, Jie Liu, Shijie Liu, Daniel G. Abel, Xu Guo, Jianbing Dong, Ji Shi, and Kunlun Li. Merlin HugeCTR: GPU-accelerated Recommender System Training and Inference. In Proceedings of the 16th ACM Conference on Recommender Systems, RecSys '22, pp. 534-537, New York, NY, USA, September 2022. Association for Computing Machinery. ISBN 978- 4577---5003--27--5 doi: 10.1 10.1145/55232733544405 URL hww.t:ttps/// / /ál www.aoo.o..........e...e....eeee www.ooog//oooooo htt///oo//1....0 11..1145.44 / 3523227 3547405.
amm.
Xiaorui Wu, Hong Xu, and Yi Wang. Irina: Accelerating DNN Inference with Efficient Online Scheduling. In Proceedings of the 4th Asia-Pacific Workshop on Nenworking, APNet '20, pp. 36-43, New York, NY, USA, August 2020. Association for Comput- ing Machinery. ISBN 978-1-4503-8876-4
doi: 241145/3411029.3411 http://.......rroooo.///0/111//.
URL
Li Yang, Jian Meng, Jae-sun Seo, and Deliang Fan. Get More at Once: Alternating Sparse Training with Gradient Correction. In Proceedings of the Neural Information Processing Systems Conference (NeurIPS), October 2022. Utthttps: / oww.reeeiivieeeeeeeee ew nnt./ for d= ezs.p
Lu Yin, Gen Li, Meng Fang, Li Shen, Tianjin Huang, Zhangyang Wang, Vlado Menkovski, Xiaolong Ma, Mykola Pechenizkiy, and Shiwei Liu. Dynamic Sparsity Is Channel-Leve Sparsity Learner. In Neural Information Processing Systems. arXiv, November 2023. doi: 00448500axi..2205.44444...
URL htt:ttp::/ / arxivv org/ abs/ (olbs/23000 2305 94.4.444444 arXiv:2305. 19454 [cs].
Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. Xixiv preprint arXiv: 1903. 1728, 2019.
13

---

Published as a conference paper at ICLR 2024
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang. Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. MEST: Accurate and Fast Memory-Economic Sparse Training Framework the Edge. In Advances in Neural Information Processing Systems, volume 34, pp. 20838-208511 Curran Associates, Inc., 2021a. URL
ngs reurrps... cc wtt.cc/pppeee pptprr/pp///2////// www.eeattbtttatt.ttt..e... html.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et Mest: Accurate and fast memory-economic sparse training framework on the edge. Advances in Neural Information Processing Systems, 34:20838-2088 2021b.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Scong Joon Oh, Youngjoon Yoo, and Junsuk Choe. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6022-6031 Scoul, Korea (South), October 2019. IEEE. ISBN 978-1 778-1-7281444 978-1-72814-803-8 doi: 10. 100////C.V22111.001222. URL
https://ieeexpleee ieee. org/ cocument/90082960
Sergey Zagoruyko and Nikos Komodakis. Wide Residual Networks, 2017. arXiv:1605.07146
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond Empirical Risk Minimization. In International Conference Machine Learning, May 2023. URL noteId=r1Dip1-B0
https: https:///pentei /ppereeieeeeeeeeeeee .nt./ httpr///weeeeeee/ttouemeeeeeeee
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random Erasing Data Augmenta- 1708.. 04896. arXiv:1708. 04896 [cs].
tion, November htt: htt.ittpp//rriiv.orttt
Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n.m fine-graines structured sparse neural networks from scratch. In International Conference on Learning Representations, 2021.
14

---

Published as a conference paper at ICLR 2024
Appendices
A SPARSITY AND OUTPUT-NORM VARIANCE
Consider a SNN with ReLU activations, where each neuron has on average k connections to the previous layer (i.e., fan-in). has been shown by Evci et al. (2022), that by normalizing the weights on initialization by a factor of v2/k, one achieves the following desirable normalization property for each layer with output 2
E
Meaning that on average the variance of the norm of each layer's output is constant. However, the variance of this ratio is non-trivial. In networks with large depth, it can accumulate, leading to exponentially large variance at the final layer (Li et al., 2021). Minimizing this variance on initialization has been shown to have a positive effect on training dynamics in some network models (Littwin al., 2020), as it stabilizes the gradients. We therefore analyze the output norm variance as a guiding quantity for sparsity-type selection.
In the following, we consider three different types of sparsity distributions, which respectively correspond to different degrees of sparsity structure in the SNN, and derive analytic expressions for the behaviour of output norm variance in SNNs with the given sparsity type. The derivations for the following results can be found in Appendix B:
"Benoulli sparsity": A connection between each neuron in layer ++ 1 and each neuron in layer appears independently with probability p= & resulting in each neuron having & connections on average and each layer having ná connections on average. The variance is:
||2++1133 ||
5n 8+18*
Var
(1)
(nn+2)
"Constant Per-Layer sparsity": Exactly kn connections are distributed at random in the layer connecting the n neurons in layer E+ 1 and the n neurons in layer É, resulting in each neuron having & connections on average. The variance is:
| (1 1|=
(n² 3)Cn,,++88. -n² -2n
VarCConst Payer
n
where On, a-1/k Note that when n >>>> Cn, mية is close to 1, and with Cn, à 1 recover the formula for Bernoulli sparsity, meaning that this sparsity type and Bernoulli sparsity are very similar.
n- 1/n
"Constant Fan-In sparsity": Each neuron in layer ++1 connected to exactly & neurons from layer f, chosen uniformly at random. In this case, the variance is:
[12 11
5n 5
3(n-k) kn(n+2))
VarConstFan--n
+(n+2)
In deriving the above results assumed that the direction of the layer output vector ! is uniformly distributed on the unit sphere. We compare our theoretical predictions with simulations in Fig. 1b and verify their accuracy. Bernoulli and constant-per-layer distribution result in unstructured sparsity, and most the current DST approaches, including RigL, operate with penstant-per-laye sparsity. In contrast, the constant-fan-in type imposes strong structural constraint. Therefore are somewhat sparsity always produces slightly smaller output-
surprised to find that, in fact, constant-fan variance than the other types. The difference is larger when h n, i.e., for very sparse networks. This indicates that, at the very least, the constant fan-in constraint should not impair SNN training dynamics and performance, motivating our method of maintaining the constant fan-in sparsity constraint within a DST approach.
15

---

Published as a conference paper at ICLR 2024
B COMPUTING THE OUTPUT NORM VARIANCE
Definition B.1. Let E {0,1)A be binary vector. Let / E {0,1) be an N x N binary matrix. L be any vector. Let W NNN be matrix of id (0,1) random variables.
Define the vector = by:
2 k
(W (goou)
(4)
iec. the entries 2; are given by:
2 24 V
W
(5)
k
Proposition B.2. The variance of each entry 2; is:
2 k
Var(z;)
(6)
and therefore the distribution of each can be written
2
96
(7)
j=1
where 9s are N iid ((,1)) random variables.
Proof. By the properties of variance:
2
Var(z:)
Tog E
(8)
he
3.8'
2
Σ
Tty Tey E
(9)
k
3.''
2
u2
(10)
k
2
D k TuyEy
(11)
since 12, = I,j and E3=C, because they are binary valued. Once the variance is established, notice that is a linear combination of Gaussians with 1 zv. because the row W,y Wyy- Hence the 2; are independent Gaussians, so the form 2, 9.V E ung follows.
Corollary B.3. The norm || can be written as:
n || d Σ ot E i-j=
(12)
Proposition B.4 ("Bernoulli Sparsity")), Suppose that u E 12th is uniform from the unit sphere, the entries Iyy ~ Ber( h). E -Br(() all independent of each other. Then:
[|=1|²
=1
(13)
5n 8+18
Var [1=11²)
(14)
n(n+2)
16

---

Published as a conference paper at ICLR 2024
Case
Num. Terms
E
[9
B[E)Ey]
I
j=ド
n²
3
k
n(n+22)
i
n² (
1
(#)
m(n++2)
n² (n-1)
3
(4)²
2
[12
m(++2)
n (n-
1
(#)*
(2)
m(++22)
Table Overview terms for Bernoulli type sparsity.
Proof. We have
E
[1=113)
2
k Σ
E [ uy
(15)
2
Σ
E Elg
(16)
k
1,==
2 n
1
(17)
h 1J=1
12 m
(18)
Similarly, we compute the 4-th moment as follows:
2 k
E
||
Bl4y u3]
(19)
".J.""
We split this into four cases and evaluate these based on whether or not i= and =j"" in the following table.
Combining the value of each term with the number terms gives the desired result for the variance.
Proposition B.5 ("Constant-per-layy sparsity")). Suppose that E RM is uniform from the unit sphere and & (er()) are independent of each other. Suppose the entries of the matrix Liş are chosen such that:
There exactly kn ones and exactly n²² nik zeros in the matrix I, and their positions in the matrix are chosen uniformly from the (n²) possible configurations. Then:
E
[|=||
=1
(20)
Var [|x13) (n² + 77n 8)Crk 18 +18 -n² 22
(21)
n(n++2
Proof. Note that ([[4))=k/n still holds, since there are kn ones distributed over n² locations. Thus the computation for B [[|=|1²) is identical to the previous proposition. Note also that when there
two entries, we have:
if and j j=j' otherwise
T Tegl
(22)
nk
if
n
(23)
Cr.k
otherwise
where C,,
B-1/k n-1/n" The table with terms for computing [([|=|1*)
becomes: The extra factor of C,.
in the entries leads to the stated result.
17

---

Published as a conference paper at ICLR 2024
Case
Num. Terms
E
[9
B[E,y
I U[L
i=ドj=j
n²
3
k
n(n+22)
i
(
1
(+)
Cn,k
n(n
(n-
3
Cn,
(2)
+(++22
n (n-
1
Cn,,k
(2)
m(+++2)
Table 7: Overview of terms for Constant-per-layer type spansity.
Proposition B.6 ("Constant Fan-In sparsity"), Suppose that u R" is uniform the unit sphere, and Ej ~ (er()) all independent of each other. Suppose the entries of the matrix 1új are chosen so that:
1. There are exactly k ones in each row of the matrix and exactly n - zeros in the matrix I, chosen uniformly from the (2) possible ways this can happen.
2. Different rows of I are independent.
Then:
||=11²)
=1
(24)
5n 8 k:)
Var [|=11²)
(25)
kn(n+2)
Proof. Same arguments before apply, but we have
ifi= and j j= i and j $ j
k k
(26)
otherwise
(27)
and the table for the variance computation becomes:
Case
Num. Terms
E [[g898]
Iil
B[E,E,1
B[usu²]
i=
n²
3
3
+(++22
n² (2
1
3
m(++22
iード
n² (
3
k-1
(
n-1
+(n+2)
n² (n-
(4)
(1)²
m(n+2)
Table Overview of terms Constant-fan-ii type sparsity.
Which leads to the stated result.
(28)
18

---

Published as a conference paper at ICLR 2024
96
RigL
SRigL.
sparsity (%)
w/o
w/ ablation
Test Accuracy (%)
92
50
94.6+1011
94.7+0.1
94.6++011
88
60
94.6+10.1
94.5+0.1
94.6-10.1
94.5=0.11
94.4+0.1
94.4=0.1
84
RigL
80
94.0++.11
94.1+0.2
94.0+0.1
SRigl. w/o ablation
90
93.340.1
93.1
93.3
80
SRigL.
95
92.1#0.1
91.410.1
91.8=0.2
Dense benchmark
76
99
84.9+0.2
76.9
82.7=0.8
50
60
70
80
90
100
Sparsity (%)
0
dense Wide ResNet-22:
95.0
Figure 5 & Table 9: Test accuracy of Wide ResNet-22 trained on CIFAR-10. Mean and 95% confidence intervals reported five runs.
C WIDE RESNET-22 22 TRAINED ON CIFAR-10
In Fig. 5 we present results of training Wide ResNet-22 (Zagoruyko Komodakis, 2017) with RigL or SRigL on the CIFAR-IO dataset. The training details for this experiment identical those reported in Section 1. SRigL. without ablation performs poorly at very high sparsities. With ablation, SRigL achieves generalization performance comparable RigL.
D HYPERPARAMETED AND TRAINING DETAILS
D.1 RESNET-18 TRAINED ON CIFAR-10
As per Liu (2017), we modify the original ResNet-18 network by changing the kernel dimensions of the first convolutional layer to 3x3 instead of 7 x7. Further, reduce the stride in the first two convolutional layers to one to avoid excessive reduction of the feature map's 's spatial dimensions.
We train each network for 250 epochs (97,656 steps) using batch size of 128. An initial learning rate of 0.1 is reduced by a factor of 5 every 77 epochs (about 30,000 steps). We use stochastic gradient descent (SGD) with momentum, with an 1.2 weight decay coefficient of 5e-4 and momentum coefficient of 0.9. We train each model using single Nvidia V100 GPU.
We achieve the desired overall sparsity by distributing the per-layer sparsity according the ERK (Evci et al., 2021; Mocanu et al., 2018 8) distribution, which scales the per-layer sparsity based on the number of neurons and the dimensions the convolutional kernel, if present. We set the number of mini-batch steps between connectivity updates, AT, to 100. Ysal is set at 30% based on the results of a small grid search performed on CIFAR 10 with ResNet- 18 and Wide ResNet-22. See Fig. 8 for details.
For each trial, we select a desired sparsity in the range from 0.5 to 0.99. At each connectivity update, the portion of weights to be pruned or regrown is based on cosine annealing schedule (Dettmers Zettlemoyer, 2019) with an initial value = 0.3. The portion of weights to be updated decays from the initial value to zero once 75% the total training steps have been completed, after which the weight mask remains constant.
D.2 RESNET-50 TRAINED ON IMAGENET
We use mini-batch size of 512 instead of 4096, We linearly scale the learning rate and AT to account for our smaller batch size. Linearly scaling the learning rate in this manner was included in the original Rigl source source code and is further motivated by Goyal et al. (2018) 8). We increase AT to 800 and average the dense gradients over cight mini-bated steps to ensure that SRigL has the same quality of parameter saliency information available RigL at each network connectivity update. We set Ysal to 30% based on our grid search presented in Fig. 8.
19

---

Published as a conference paper at ICLR 2024
Our learning rate uses linear warm-up up reach maximum value of 0.2 at epoch five and is reduced by a factor of 10 at epochs 30, 70, and 90. Using mini-batch of 512, we train the networks for 256,000 steps to match RigL's 's training duration. We use cosine connectivity update schedule with ===00 0.3. We initialize the sparse model weights per Evei et al. (2022). We train the networks using SGD with momentum, 1.2 weight decay, and label smoothing (Szegedy et al., 2016) coefficients of 0.9, le-4 and 0.1, respectively.
We use the same standard data augmentation in our data preprocessing as RigL, including randomly resizing to 256 x 256 or x80 480 pixels, random crops to 224 x224 pixels, random horizontal flips, and per-image normalization to zero mean and unit variance using identical per RGB channel mean and standard deviation values as RigL. We train each model using either four Nvidia V100 or A100 GPUs.
D.3 VISION TRANSFORMER TRAINED ON IMAGENET
For our ViT-B/16 experiments, we used sparsity on the convolutional projection (input projection to patches), the fully connected layers in the feed forward (MLP) blocks and the output projections of the multi-headed attention (MHA) modules. We performed a lightweight ablation study on four ViT-B/16 networks trained on ImageNet to determine the affect of sparsifying the first convolutional projection layer as well as the input projection layers in the multi-headen attention modules. Based on the results of our ablation study. we did not use sparsity on the MHA input projection layers or the scaled-dol products. See Fig. 9b for more details. This setup is similar to the "Sparse FF" models investigated by Jaszezur et al. (2021). The global model sparsity level reported in Table 4 is calculated based on the sparse modules only. If we also consider the parameters in the MHA input projections as part of our parameter budget, the global model sparsities tabulated in Table 4 correspond to 60.35% and 67. 77.0% for the rows labelled 80% and 90% sparsity, respectively.
We add additional data augmentations following the standard Torch Vision (Maintainers & Contributors, 2016) V/T- training procedure for ImageNet. These data augmentations applied include: random cropping, resizing the cropped image to 224 by 224 pixels, randomly horizontal flips, randomly augmenting with RandAugment algorithm (Cubuk al., 2020), and normalizing with the typical RGB channel mean and standard deviation values. We also randomly choose one of random mixup (Zhang al., 2023) or random cutmix (Yun et al., 2019) and add it to the above-noted augmentations. We use 0.2 and 1.0 for the alpha parameter values for mixup and cutmix, respectively. We omit Dropout (Srivastava et al., 2014) from the model entirely to avoid potential layer collapse in the case where all non-zero weights are dropped from a layer and to avoid any other unintended interference with SRigL's 's sparse training procedure.
We sample eight mini-batch steps with 512 samples per minc-batch and accumulate gradients before applying the optimizer, resulting in an effective mini-batch size of 4096. We train the model for 150 epochs using AdamW (Loshchilov & Hutter, 2018) optimizer with weight decay, label smoothing, 81, and 32 coefficients of 0.3, 0, 0.9, and 0.999, respectively. We use cosine annealing with linear warm- up for our learning rate scheduler with initial learning rate of 9.9c-5 that warms-up to maximum value of 0.003 at epoch 16. We clip all parameter gradients to max L.2 norm of 1.0. We apply uniformly distributed sparsity across all layers in the model. AT is set to 100 to update network connectivity every 100 mini-batch steps. We train each model using either four Nvidia V100 or A100 GPUs.
D.4 MOBILENET-V V TRAINED ON IMAGENET
We follow the Torch Vision (Maintainers & Contributors, 2016) training recipe for MobileNet- V3 Large and Small for ImageNet. We set AT to 100 and Yeal to 30% similar to our other CNN experiments. We train the models from scratch for 600 epochs using an R.MSProp (Tieleman et al., 2012) optimizer with momentum, L.2 weight decay, and smoothing constant coefficients of 0.9, 1e-5, and 0.9, respectively. The networks are trained with step learning rate decay schedule with initial learning rate of 0.064, multiplicative factor of 0.973, and we decay the learning rate every two epochs.
The input data is augmented with random cropping to 224 by 224 pixels, random horizontal flips, Autmentgtnoiiion using the ImageNet policy (Cubuk et al., 2019), normalizing to standard RGB mean and standard deviation values, and random erasing with a probability of 0.2 (Zhong et al., 2017). Similar the above, we omit Dropout (Srivastava al., 2014) to avoid potential layer collapse. Unlike the Torch Vision recipe, we do not average the trained parameters across the last three checkpoints that improved the top-1 accuracy. We train with a batch size of 512 and accumulate gradients across
20

---

Published as a conference paper at ICLR 2024
two mini- resulting in effective mini-batch size of 1024. We train each model using four Nvidia A100 GPUs.
80
70
Test Accuracy (%)
64
Test Accuracy (%)
56
48
42
32
28
16
RigL.
14
RigL
SRigL
SRigL
Dense benchmark
Dense benchmark
90
85
100
Sparsity (%)
Sparsity (%)
Figure 6: MobileNet-V3 Large/ ImageNet top-1 accuracy. SRigL compares well against RigL both both models perform poorly compared the denes baseline 99% sparsity.
Figure MobileNet-Va V3 Small / ImageNet top-1 accuracy. SRigL compares well against RigL both both models perform poorly compared the denes baseline 99% sparsity.
E TUNING Vsald MINIMUM PERCENTAGE SALIENT WEIGHTS PER NEURON
Fig. 8 depicts the generalization performance of highly sparse ResNet- 18 and Wide ResNet-22 models trained on the CIFAR-10 1-A dataset. SRigL's L' generalization performance at high sparsities is improved with neuron ablation; however, the specific value selected for Ya does not have a significant effect on performance. Our experiments demonstrate that SRigL performs well with variety of 7 values. In Section we report the results of SRigl. models trained with "sal set to 30%. With dynamic ablation enabled, we the minimum salient weights per neuron in value less than onc. In Fig. 10, many layers in ResNet-50 are set to the minimum threshold of one when we apply "sai of 30% for all model types other than VIT-B/16. This minimum threshold explains the invariance of the model's performance when comparing against multiple values for Yeall
if the user-defined threshold results
95.5
$4.5
Sparsity (%)
Sparsity (%6
Test Test.Accul Accuracy (%0
90.2
90%
Tesk Accuracy
95%
50% Ablation
95% wib AB artin
95%
90%
92.8
Ablotion
$2.6
wrl
91.9
78.0
75.0
10
20
40
50
10
20
30
40
50
Minimum Salient Weights per Neurrn (96
Minimumm Salient Weights Neuron (%)
(a) 1//CFFRR//
(b) Wide ResNet-22/CIFAR-10
Figure 8: (a) ResNeti8/CIFAR Test Accuracy vs. Ysal when trained with SRigL with without ablation for a range of sparsities. The mean and 95% confidence intervals are shown for five different random seeds for the runs with ablation. For the runs without ablation, report the mean of five different random seeds. (b) Wide ResNet- 22 Test Accuracy Ysu. The and 95% confidence intervals shown for five different random seeds.
Fig. 9a demonstrates how ViT-B/16's generalization performance is much more sensitive to Teal- We find that RigL learns a sparse connectivity pattern with a large variance in sparse fan-in between neurons within given layer, with some neurons having an order of magnitude more fan-in connection than the mean fan-in.
21

---

Published as a conference paper at ICLR 2024
80
77
Test Accuracy (%)
74
Dense benchmark
65
Sparse first layer
71
Sparsity (%)
Dense first layer
80%
80% w/o Ablation
90%
55
90%
Ablation
65
100
50
Minimum Salient Weights per Neuron (%)
Dense MHA
Sparse MHA
(a) ViT-B/16/ImageNet Test Accuracy vs. Ysal when trained with SRigL with and without ablation enabled for 80% and 90% sparsity. ViT-B/16's performance is much more sensitive to Ysat and generally performs best with high ablation thresholds. Based this data set Yaol to 95% for our results reported Section
ViT-B/16 ablation study. The best performing variant used a sparse first layer and dense input projections in the MHA modules.
F CONDENSED MATRIX MULTIPLICATION
Using a constant fan-in sparse representation presents an advantage compared the general N:M sparse representation in that we can represent our weight matrices in compact form, since every neuron/convolutionnal filter has the same number of non-zero weights. Here we demonstrate how this can be used to accelerate fully-connected layer.
Consider the standard vecror product:
Way
W11
W12
Wid
Wa2
12
W2y
WV
(29)
Wwa
Wi.d/
Way
When W E Raxd is sparse and has only & non-zero elements per row, the sums representing each element of out will be limited & terms, i.e.:
k
OHI 2/
Σ
Wis Via with ja (),,...,d), ja Fja
(30)
Note that the expression on the right-hand side of Eq. (29) can be represented as an operation between dense matrix W Rank (we call "condensed W") and vectors yx u t elements are drawn from with replacement (we call them "recombinations of o²"). The operation a sum over element-wise products between the i-th column of W and the i-th column vector v::;
R" whose
We W.O..
(31)
1=1
Mathematically, these methods are equivalent for any matrices. Computationally. the condensed method can more efficient, particular for sparse matrices with constant small fan-in By construction, this method requires the sparse matrix W to be stored in dense representation which involves two 2D arrays of shape n x k: One holds the values of the non-zero elements of and the other their respective column indices, which are used to generate input vector re-combinations. An efficient computational implementation of this method subject of ongoing work on this project Based our results, the constant fan-in constraint does not appear to have limiting effect SNNs.
22

---

Published as a conference paper at ICLR 2024
70 30
4015-10%
4011-10%
layent
Invert 2011
1yerl10.comvv
layeri.1.onv1
1ayeri.1.oov2
lorars
Isyer2 0. conv1
lorars 2000
Ixyor2.0.com2
128
layer2.0.thrraa.t
Ward
bayer2.1.covv1
128
1syer2.1.comv.
layer co
1eyer3.0 connv
256
256
layor3 1 sherroout
laye
Isyer3.1.com
256
layer denu
1ypo3311.omv2
byerl com/3
Miyer4.0.com
512
byeri 00172
Isyent conov
leyerd shoroout:
512
leyers 20n12
layer4.1.comv
Minimum Sallent Weights per Nearon
100 200
400
500
Figure 10: ResNet-50 Layer vs. Minimum salient weights per neuron. SRigL sets the minimum salient weight per neuron the sparse fan-in per neuron
Figure 11: ResNet-18/CIARR-10 layer widths at the end of training at 99 % sparsity. Without ablation, constant fan-in constraint enforces that sparse layers retain their original width. When ablation is enabled, the Yaal threshold (minimum percentage salient weights per neuron) used to control the amount of ablation.
the product between Taal and than 1. Therefore, even a relatively large network such as ResNet50 many of the layers only require that single weight be active keep the neuron active. We believe this why SRigL's performance relatively invariant various ablation thresholds when applied to CNNs
800
600
Sparse Fan-In
500
Test Accuracy (%)
400
300
62
100
56
SRigL
3
4 5
6
7
8
generalization
Layer Index
50
10
20
30
40
50
Normalized Training FLOPS (%)
Figure 12: Sparse Fan-In vs. ViT-B/16 layer index at the end of training with RigL at 90 % sparsity. Only the first 10 layers shown for clarity. We find that RigL learns sparse connectivity with large vari- ance fan-in between neurons within the same layer with some neurons receiving to x 10 the number active connections than the mean for the same layer.
Figure 13: Training FLOPs for SRigL on ResNet- 50/ImageNet at a variety of sparsitics compared with dense generalization. FLOPss are normalized by dense training FLOPs.
23

---

Published as a conference paper at ICLR 2024
G FLOPS ANALYSIS
In Fig. 13, we present an analysis of the FLOPs required during training and inference for SRigL. and compare with SR-STE. We calculate FLOPs using the same methodology as Evci et al. (2021) by considering only operations induced by convolutional and linear layers and their activations. FLOPs for add and pooling operations are ignored. For training FLOPs, we also disregard FLOPs required for mask updates, this step is amortized over AT steps and is negligible compared the FLOPs required otherwise for training. The open-source code for counting operations from the NeurIPS 2019 MicroNet Challenge and is available on GitHub².
Similar to other DST methods, SRigL. obtains generalization performance comparable to a dense network benchmark fraction of the FL.OPs required for both training and inference.
H IN TIME OVERPARAMETERRIZATONN RATES
In Figs. 14 to 17 present the In Time Overparameterriztton Rate (TTOP) (Liu et al., 2021c) for various models and datasets. In this same work, Liu et al. (2021c) proposed modified hyperparameters for RigL that may yield higher generalization performance; however, detailed investigation of these hyperparameters for SRigl. is left to future work.
100
100
80
80
ITOP Rate
ITOP
40
40
20
RigL
20
RigL
SRigL ablation
SRigL ablation
SRigl.
SRigl.
50
60
70
80
90
100
50
60
70
80
90
100
Sparsity (%)
Sparsity (%)
Figure 14: ResNet-I8/CIFAR ITOP rate
Figure 15: ResNet-18/CIFAR ITOP rate
100
80
ITOP Rate
ITOP Rate
60
RigL
SRigL w/o ablation
RigL
20
SRigL
SRigL. w/o ablation
SRigL
SRigL
SRigL.
80
82
84
86
88
90
80
85
95
100
Sparsity (%)
Sparsity (%)
Figure 16: ViT-B//6//mageNet ITOP Rate
Figure 17: ResNet-50//maeeNell ITOP Rate
MeeroNeet Challenge Github Repository
24

---

Published as a conference paper at ICLR 2024
I CONDENSED LINEAR CPU BENCHMARK DETAILS
For each sparsity level, we used the trained weights from the last linear layer in the final multi-layer perception block from the ViT-B/16 transformer encoder. This layer has width of 768 neurons and an input of 3072 features. The input and layer parameters all set to 32 bit floating point type. Across all sparsities, batch sizes, and number threads investigated, our condensed representation utilizing both structured and fine-grat fne-grained sparsity yields the fastest online inference speed. However, higher batch sizes and modest sparsities, structured sparsity is often faster than our condensed representation. See Figs. 18 20 for benchmark results from 1-8 threads and batch sizes 1-64. We note that SRigL with either condensed or structured sparse representation yields the fastest benchmark times.
We used torch. cercchcommiic le with the inductor backend. For compiler options, used the max- autotune mode and full graph output. However, full graph output is not compatible with CSR formats we omit this parameter for the unstructured benchmarks. The benehmark script was run with n ceness value of - 15 to ensure as accurate results possible. The apparent slow down 99% structured sparse benchmarks compared other sparsities due the fact that SRigL. ablates fewer neurons at 99% sparsity. At extreme sparsities, each neuron has very few active weights resulting more neurons being considered as salient by SRigL.
J GPU BENCHMARKS
Using GPU CUDA kernels developed by (Schultheis & Babbar, 2023), we accelerate our sparse networks and demonstrate significant acceleration for batched inference and modest acceleration for online inference at high sparsities (>90%), see Fig. 21. All runs conducted on an NVIDIA Titan V. Note y-axis scale is logarithmic.
80%
90%
Structured
2500
Unstructured
Time (us)
1500
1000
SRigL lours)
500
Unstructured
95%
99%
Structured
Structured
2500
Unstructured
Unstructured
Time (Us)
1500
1000
500
Batch Size
Batch Size
Figure 18: CPU benchmarks with thread up batch size
25

---

Published as a conference paper at ICLR 2024
80%
90%
Structured Unstructuee
Structured
Unstructured
1000
800
Time (μs
600
95%
99%
SRigL
Structured Unstructured
Structured
Unstructured
1000
800
Time (us)
600
Batch Size
Batch Size
Figure 19: CPU benchmarks with 4 threads up to batch size 64
80%
90%
SRigL, lours)
Structured
Unstructured
Dense
Time (us)
400
200
SRigL
100
Structured
Unstructured
Dense
95%
99%
(ours)
SRigL lours)
600
Structured
Structured
Unstructured Dense
Unstructured
Dense
Time (us)
400
300
200
100
12
18
32
64
Batch Size
Batch Size
Figure 20: CPU benchmarks with 8 threads up to batch size 64
26

---

Published as a conference paper at ICLR 2024
10+
Dense
10a
Time (μs)
10a
10
SRigL (ours)
Structured
Unstructured
(a) GPU online inference (batch size of 1)
10
Dense
10a
Time (μs)
10a
104
SRigL (ours)
Structured
Unstructured
(b) GPU batched inference with batch size of 128
10
Dense
80
90
95
99
Time (μs)
10a
10
10
SRigL (ours)
Structured
Unstructured
(c) GPU batched inference with batch size of 2048
NVIDIA Titan V. We compare timings for a folly-connected layer extracted from ViT-B/16 model trained with SRigL when compressed using condensed representation learned by SRigL, structured (i.e. SRigL with only neuron ablation) and unstructured (i.c. CSR) representations. Batch sizes are 1, 256, and 2048 for sub-fig sub-figures 21a, 21b, 21c, respectively. The median over minimum of 5 runs shown, while the error bars show the std. dev. Note: a-axis scale is logarithmic
Figure 21: Real-world GPU wall-clock timings for inference
27

---

Published as a conference paper at ICLR 2024
250
Dense
80
90
95
99
200
Time (us)
150
100
50
SRigL (ours)
DeepSparse (Unstructured)
Structured
CSR (Unstructured)
Figure Online inference with DeepSparse compared SRigL on an Intel Xeon W-2145 threads. The median over a minimum of runs shown, while the error bars show the std. dev.
K DEEPSPARSE CPU BENCHMARKS
Here we present online inference benchmarks for CPU using the DeepSparse Engine library (lofinova al., 2021). DeepSparse library includes several engineering innovations accelerate unstructured sparsity on CPU. For instance, a depth-wise asynchronous execution algorithm is used that takes advantage of the relatively large cache size for CPUs compared to hardware accelerators such as GPUs, Other additional innovations used include pre-loading the input data to hide latency via CPU pipelining, compressing sparse activations into CSR format on-the-fly, and keeping convolutional kernels in L2 cache. For more details see Kurtz et al. (2020).
We compare our CPU timings for SRigL DeepSparse in Fig. 22 and find similar latency; however, we note that DeepSparse is subject to higher variability as evidenced by a larger standard deviation. Further, many of the innovations used accelerate unstructured sparse networks with DeepSparse could equally be applied to networks trained with SRigL.
L COMPARISON WITH STRUCTURED PRUNING METHODS
In the following table we compare several structured pruning methods to SRigL. The tabulated structured pruning methods typically prune and fine-tune a pretrained model, resulting extended training duration compared to typical dense training. We report the inference FLOPs, top-1 accuracy, and number of epochs for each method Table 10.
28

---

Published as a conference paper at ICLR 2024
Table 10: Top-1 ImageNet test accuracy of ResNet-50 for various structured pruning methods compared with SRigL and Chase (Yin et al., 2023). All values, except SRigL. are obtained from et al. (2023).
Methods
Inference FLOPs
Top-1 I Accuracy
Epochs
Uniform
2.0G
75.1%
300
Random
2.0G
74.6%
300
GBN (You et al., 2019)
2.4G
76.2%
350
LEGR (Chin et al., 2020)
2.4G
75.7%
FPGM (He al., 2019)
2.4G
75.6%
200
TAS (Dong & Yang, 2019)
2.3G
76.2%
240
Hrank (Lin al., 2020)
2.3G
75.0%
570
SCOP (Tang al., 2020)
2.2G
76.0%
230
CHIP (Sui et al., 2021)
2.2G
76.3%
Group Fisher (Liu al., 2021a)
2.0G
76.4%
AutoSlim (Yu & Huang, 2019)
2.0G
75.6%
CafeNet-R (Su al., 2021)
2.0G
76.5%
300
Chase-1'(Ying et al., 2023)
1.5G
76.6%
250
SRigL
2.0G
74.7%
205
SRigL.
2.0G
76.2%
515
Uniform
1.0G
73.1%
Random
1.0G
72.2%
300
Group Fisher (Liu et al., 2021a)
1.0G
73.9%
CafeNet-R (Su al., 2021)
1.0G
74.9%
300
CafeNet-F (Su et al., 2021)
1.0G
75.3%
300
Chase-2X(Yin et., 2023)
0.9G
75.7%
250
SRigL
1.0G
71.5%
205
SRigL*
1.0G
73.6%
515
DST methods. All other methods tabulated are structured pruning methods.
29


Reviews:
[{"review_id": "KCZvgDQTBg", "reviewer": "Reviewer_43nv", "summary": "This paper proposes SRigL with a type of structured sparsity - constant fan-in sparsity that can be applied to dynamic sparse training, leading to real-world clock-time savings while maintaining the optimal performance.", "questions": "(1) What is the technical difference between SRigL and Chase? \n\n(2) Can SRigL also accelerate the training process for the online inference, with real-world wall-clock saving? Moreover, I noticed that the Constant Fan-in sparsity can be accelerated by GPUs with some custom CUDA implementation, such as Schultheis & Babbar (2023). I am wondering how difficult to make SRigL accelerated by GPUs using the CUDA implementation provided by Schultheis & Babbar (2023). \n\n(3) Why SRigL w/ ablation outperforms SRigL w/o ablation? I suppose that SRigL w/o ablation is essentially the unstructured version of RigL. \n\n(4) Since the real-world clock-time is measured on CPU. I am wondering how different it is to implement GPU kernel to support SRigL in common GPUs?\n\nOverall, I think this paper is a good asset and I am willing to increase my score if the above weaknesses can be resolved.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The research topic of structured DST is timely and important for the ML community. With the popularity of LTH and DST, sparse neural networks have received upsurging attention due to its promising capacity to reduce training/inference costs while maintaining the original performance of dense counterparts. However, the benefits of sparse NNs can largely constraint by the limited support from common hardware - GPUs. Research works to improve the progress of this direction make significant contributions to the community.  \n\n2. This paper provides a comprehensive and precise related work that covers the most state-of-the-art structured/unstructured sparse training approaches. I very much appreciate such rich related works that provide enough credits and credits to previous works. \n\n3. The detailed step of SRigL in Section 3 provides a good overview to understand the methodology. \n\n4. SRigL is able to find small-sparse NNs that enjoy better real-world wall-clock for online inference than structured (SRigL with only neuron ablation) and unstructured NNs. \n\n5. Compared with N:M sparsity, SRigL enjoys uniform layer-wise sparsity, which is more desirable for performance.", "weaknesses": "(1) Partial of the ideas used in SRigL has some certain levels of overlaps with the previous work (Chase: https://arxiv.org/pdf/2305.19454.pdf). For instance, Chase also uncovers that a large proportion of channels (termed sparse amenable channels) tend to be sparse during DST. They also perform channel pruning to produce a mixture of structured and unstructured sparsity at the end of training. It is better to clarify the difference and similarity between Chase and SRigL, even though Chase does not introduce the hardware-friendly constant fan-in sparsity for the unstructured part. \n\n(2) Can SRigL also accelerate the training process for the online inference, with real-world wall-clock saving?\n\n(3)  In many cases in Table 1 and 3, SRigL w/ ablation even outperforms SRigL w/o ablation, which is a bit counter-intuitive. Cause SRigL w/ ablation essentially produces a smaller-sparse model if I understand correctly, which would decrease the model capacity. Can the authors elaborate more about this? Does  SRigL w/o ablation means only pruning these dean channels without weight regrowing?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bqGG34IfQA", "reviewer": "Reviewer_LY1T", "summary": "This paper presents Structured RigL (SRigL), a Dynamic Sparse Training (DST) method that excels in training sparse neural networks. SRigL achieves state-of-the-art performance in DST by incorporating fine-grained N:M sparsity and continuous fan-in constraints for sparse interstructures. Through heuristic analysis and neuron removal, SRigL outperforms existing methods across various neural network architectures, demonstrating a substantial 3.6x/2x speedup on CPU at 90% sparsity compared to equivalent dense or unstructured sparse layers.", "questions": "* In Section 4.4, the description of Algorithm 1 mentions, \"The algorithm to accelerate our condensed sparsity representation is shown in Algorithm 1, demonstrating its embarrassingly parallel nature.\" I'm interested in knowing the throughput on real GPU or CPU based on the sparsity levels in the matmul unit-test.\n* While discussing \"Constant fan-in sparsity,\" it occurs to me that there might be performance variations across tasks and networks depending on the information included by input features. Have there been any experiments applying this approach to tasks other than vision?\n* It is anticipated that the proposed SRigL method is significantly influenced by the $\\gamma_{sal}$ value. I think this sensitivity might be more pronounced in networks with lower redundancy, such as MobileNet. Has there been an observation of trends by sweeping through different $\\gamma_{sal}$ values? If so, what were the findings?\n* How does the application of SRigL affect the practical outcomes (latency, throughput) in the context of \"end-to-end sparse training\" on GPUs?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The authors propose SRigL method that learns a SNN with constant fan-in fine-grained structured sparsity while maintaining generalization comparable to RigL even at high sparsity levels across various network architectures.\n* Experimental results show that the SRigL method can not only improve the efficiency of parameters and memory, but can also enable acceleration during training.\n* The proposed SRigL demonstrates minimal accuracy drop even at high sparsity levels exceeding 90% in both ResNet and ViT architectures.", "weaknesses": "* The proposed SRigL primarily demonstrates its efficacy on networks with relatively high redundancy, such as ResNet or ViT. However, there is a lack of experimentation on networks with lower redundancy, such as MobileNet.\n* The experimental results are limited to vision tasks.\n* While the comparison from the perspective of Dynamic Sparse Training (DST) in Table 3 is crucial, I believe that the results regarding structured pruning performance are equally significant. However, there is a notable absence of experiments, and comparisons with other techniques with 'structured' attributes are challenging.\n* A minor concern is the absence of a comprehensive figure that provides an overview of the entire process of SRigL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0IjgaygBCe", "reviewer": "Reviewer_UhkR", "summary": "The paper proposes a method for dynamic sparse training that leads to structured N:M sparsity, here realized as a constant fan-in degree for neurons. The authors modify RigL to achieve this. The proposed constant fan-in N:M sparsity can theoretically achieve faster inference speeds on new GPU chips that are produced for general consumers but can specifically accelerate this type of structure.\n\nThe authors show that the output norm variance can in expectation be reduced with constant fan-in sparsity. The authors experimentally show that their proposed SRigL leads to the similar performance as RigL while enforcing the imposed structure.", "questions": "For the wall-clock time comparison, it seems that the median for at least 5 runs is shown. Given that the inference time is in the microsecond range, why not show the median of millions of forward passes, to reduce noise?\n\nWhat should be the main takeaway from you showing SRigL x2 and SRigL x5 in Figure 3? It seems to be included to inflate the presentation of results of SRigL, but RigL performs the same under x2 or x5 training times. Either include RigL x2 and RigL x5, or omit both from Figure 3.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper tackles an important issue in deep learning, is well-motivated, and written in a clear fashion. The motivation for the benefits of structured sparsity are clear. The main method is simple to explain, and seems to work approximately as good as RigL.\nI have reviewed the paper before, and appreciate that the authors include a wall-clock time comparison of a fully connected layer on a CPU.", "weaknesses": "I have reviewed this paper before and still remain unconvinced by the author's argumentation on the main benefits of SRigL. While the authors claim that sparse-to-sparse training such as SRigL is beneficial over dense-to-sparse methods in terms of memory usage and computational time, the evidence presented for this is marginal. While the benefit of sparse models at inference time is obvious, the benefit of sparsifying models at training time (that reach the same generalization error) should be either faster training times or lower memory footprint. The authors show no evidence that SRigL trains models faster than dense-to-sparse methods such as SR-STE. On a similar note, a real-world quantitative evaluation of the difference in memory footprint for dense-to-sparse methods (such as SR-STE) vs. SRigL would make this paper much more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zQkcTPiOJe", "reviewer": "Reviewer_qWXP", "summary": "The paper introduces Structured RigL (SRigL), an advancement in Dynamic Sparse Training (DST). SRigL combines structured sparsity with neural network training, resulting in models that are both computationally efficient and performant. The method achieves faster real-world inference times on CPUs and outperforms other sparse training techniques in various neural network architectures.", "questions": "(1) with comparison to [1], what's the advantages of SRigL?\n\n(2) Why AST and SR-STE is not comparable to SRigl?  it would be better to report the results of AST and SR-STE since they are very related works.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) SRigL creatively combines structured N:M sparsity with a constant fan-in constraint which is from existing DST methodologies.\n\n(2) The paper provides extensive empirical evidence to show its superior performance.\n\n(3) SRigL's ability to achieve faster real-world inference times on CPUs is of paramount importance.", "weaknesses": "(1) an important baseline is missing [1].\n\n(2) Lack of novelty: A combination of sparse training and N: M sparsity have been shown in previous study [1]. \n\n[1] Yin, Lu, et al. \"Dynamic Sparsity Is Channel-Level Sparsity Learner.\"  NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qDmxGc30Jj", "reviewer": "Reviewer_HJFn", "summary": "This work proposes a dynamic sparse training method with constant fan-in constraint. The method is validated on several computer vision datasets and architectures.", "questions": "Wall-clock timings for structured sparsity look suspicious. Does it mean that N:M sparsity with higher sparsity may be slower than the one with lower sparsity? Indeed, it is hardly possible to achieve linear speed-up, but I would expect at most saturation for high N:M sparsity. \n\nHow well does the method perform when combined with quantization? One is expected to achieve even higher speed-ups for sparse+8bit quantized model.  \n\nHow much does the method explore new connections compared to RigL? Would be interesting to compare some measure similar to In-Time Over-Parameterization, introduced in [1] for RigL and SRigL. \n\n---\n[1] https://arxiv.org/abs/2102.02887", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The introduced sparsity pattern is quite flexible and allows achieving performance close to unstructured sparsity while being more hardware-friendly. Neuron ablation procedure looks very sound and improves noticeably performance of the method. The appendix involves theoretical analysis motivating the good optimization behavior of the proposed SRigL and the choice of constant fan-in sparsity pattern. \n\nMethod is validated on several models and attains pretty good performance at high sparsity.\n\nThe algorithm is simple and leads to speedups even without the need to write highly-customized code.", "weaknesses": "I think the proposed SRigL with constant fan-in sparsity should be compared to a similar sparse training procedure with N:M sparsity. One could perform it as follows: with some interval, a fraction of weights with the smallest magnitude among the group of M consecutive weights is dropped and the same fraction with high gradient magnitude is regrown. Indeed, the difference from original RigL is that the space of possible updates is much more constrained - i.e. for 2:4 sparsity one can prune a single non-zero weight inside the group of 4 and regrow one among the zeros. Likely, such strategy would perform poorly, but this comparison would motivate the necessity for constant fan-in sparsity. The other alternative is RigL with blocked sparsity that prunes whole groups of consecutive `block_size` weights. This sparsity pattern is known to be more CPU-speedup friendly compared to unstructured sparsity. \n\nMethod lacks comparison with dedicated inference engines that leverage sparsity. DeepSparse engine [1] achieves significant speed-ups for unstructured sparsity, especially on Intel CPUs. \n\n*Minor* In the Appendix E plots w/o ablation seem to be absent on Figure 7. \n\n---\n[1] https://github.com/neuralmagic/deepsparse", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)