Keep the Gradients Flowing: Using Gradient Flow to study Sparse Network Optimization
Authors: Kale-ab Tessera
Sara Hooker
Benjamin Rosman
Venue:  (2021)
Keywords: neural networks
sparsity
gradient flow
sparse network optimization

Abstract: Training sparse networks to converge to the same performance as dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider various choices made during training that might disadvantage sparse networks. We measure the gradient flow across different networks and datasets, and show that the default choices of optimizers, activation functions and regularizers used for dense networks can disadvantage sparse networks. Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results.   

Under review as a conference paper at ICLR 2021
KEEP THE GRADIENTS FLOWING: USING GRADIENT FLOW TO STUDY SPARSE NETWORK OPTIMIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Training sparse networks to converge to the same performance dense neural architectures has proven to be elusive. Recent work suggests that initialization is the key. However, while this direction of research has had some success, focusing on initialization alone appears to be inadequate. In this paper, we take a broader view of training sparse networks and consider various choices made during train- ing that might disadvantage sparse networks. We measure the gradient flow across different networks and datasets, and show that the default choices of optimizers. activation functions and regularizers used for dense networks can disadvantage sparse networks, Based upon these findings, we show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime. Our work suggests that initialization is only one piece of the puzzle and a wider view of tailoring optimization to sparse networks yields promising results.
INTRODUCTION
Over the last decade, a "bigger is better" race in the number of model parameters has gripped the field of machine learning (Amodei et al., 2018 Thompson et al.] 2020), primarily driven by over-
parameterized deep neural networks (DNNs). Additional parameters improve top-line metrics, but drive up the cost of training (Horowitz) 2014 Strubell et al. 2019 Hooker 2020) and increase the
latency and memory footprint at inference time (Warden & Situnayake 2019] (Samala et al.) 2018
Lane & Warden 2018). Moreover, overparameterized networks have been shown to be more prone to memorization (Zhang et al. 2016).
To address some of these limitations, there has been renewed focus compression techniques that preserve top-line performance while improving efficiency. A large amount of research focus has centered on pruning, where weights estimated to be unnecessary
removed from the network
at the end of training (Louizos et al. 2017 Wen et al. 2016 Cun et al. 1990; Hassibi et al. 1993a
Ström 1997: Hassibi et al., 1993b; Zhu & Guptal 2017| See al.) 2016 Narang al. 2017).
Pruning has shown remarkable ability to preserve top-line metrics of performance, even when
removing the majority of weights (Hooker et al. 2019 Gale al. 2019). However, most pruning techniques still require training a large, overparameterized model before pruning a subset of weights.
Due to the drawbacks of starting dense prior to introducing sparsity, there has been recent focus on methods that allow networks which start sparse at initialization, to converge to similar performance as dense networks (Frankle & Carbin 2018: Frankle al. 2019b Liu al.) 2018a). These efforts have focused disproportionately on trying to understand the properties of initial sparse weight dis- tributions that allow for convergence. However, while this work has had some success, focusing on initialization alone has proven to be inadequate (Frankle et al. 2020 Evci et al. 2019).
In this work, we take a broader view of why training sparse networks to converge to the same performance as dense networks has proven to be elusive. We reconsider many of the basic building blocks of the training process and ask whether they disadvantage sparse networks or not. Our work focuses the behaviour of networks with random, fixed sparsity at initialization and we aim to gain further intuition into how these networks learn. Furthermore, we provide tooling tailored to the analysis of these networks.

---

Under review as a conference paper at ICLR 2021
In order to effectively study sparse network optimization in controlled environment, propose an experimental framework, Same Capacity Sparse vs Dense Comparison (SC-SDC). Contrary to most prior work comparing sparse to dense networks, where overparameterizee dense networks are compared to smaller sparse networks, SC-SDC compares sparse networks to their equivalent capacity dense networks (same number of active connections and depth). This ensures that the results are a direct result of sparse connections themselves and not due to having more or fewer weights (as is the case when comparing large, dense networks to smaller, sparse networks).
We go beyond simply comparing top-line metrics by also measuring the impact on gradient flow of each intervention. Historically, exploding and vanishing gradients were a common problem in neural networks (Hochreiter et al.] [200]] Hochreiter, [991] Bengio et al.) 1994] Glorot & Bengio
2010 Goodfellow et al. (016). Recent work has suggested that poor gradient flow an exacerbated issue in sparse networks (Wang et al., 2020; Evci et al. 2020). To accurately measure gradient flow in sparse networks, we propose a normalized measure of gradient flow, which we term Effective Gradient Flow (EGF)- this measure normalizes by the number of active weights and thus is better suited to studying the training dynamics of sparse networks. We use this measure in conjunction with SC-SDC, to see where sparse optimization fails and to consider where this failure could be result of poor gradient flow.
Contributions Our contributions can be enumerated follows:
1. Measuring effective gradient flow We conduct large scale experiments to evaluate the role of regularization, optimization and architecture choices on sparse models. We evaluate multiple datasets and architectures and propose new measure of gradient flow, Effectivv Gradient Flow (EGF), that we show to be stronger predictor of top-line metrics such as accuracy and loss than current gradient flow formulations.
2. Batch normalization plays disproportionate role in stabilizing sparse networks We show that batch normalization is more important for sparse networks than it is for dense networks, which suggests that gradient instability is a key obstacle to starting sparse.
3. Not all optimizers and regulizers are created equal Weight decay and data augmentation can hurt sparse network optimization, particularly when used conjunction with accelerating, adap- tive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam (Kinmma & Ba 2014) and RMSProp (Hinton et al. 2012). We show this highly correlated to high EGF (gradient flow) and how batch normalization helps stabilize EGF.
4. Changing activation functions can benefit sparse networks We benchmark wide set of ac- tivation functions. specifically ReLU (Nair & Hinton 2010) and non-saturating activation func- tions such as PReLU (He et al. 2015). ELU (Clever et al. 20155). SReLU (Jin et al. 2015), (Ramachandran et al. (017)) and Sigmoid (Neal) 1992). Our results show that when using adap- tive optimization methods, Swish is promising activation function, while when using stochastic gradient descent, PReLU preforms better than the other activation functions.
Implications Our work is timely as sparse training dynamics are poorly understood. Most training algorithms and methods have been developed to suit training dense networks. Our work provides insight into the nature sparse optimization and suggests a wider viewpoint beyond initialization necessary to converge sparse networks to comparable performance as dense. Our proposed approach provides a more accurate mcasurement of the training dynamics of sparse networks and can be used inform future work on the design of networks and optimization techniques that are tailored explicitly to sparsity.
METHODOLOGY
2.1
SAME CAPACITY SPARSE VS DENSE COMPARISON
Our goal is to measure what architecture and optimization choices favor sparse networks relative to dense networks. To fairly compare sparse and dense networks, we propose Same Capacity Sparse vs Dense Comparison (SC-SDC), simple framework which allows us to study sparse network optimization and identify what training configurations are not well suited for sparse networks.
SC-SDC can be summarized as follows (See Figure T for overview):
2

---

Under review as a conference paper at ICLR 2021
2. Match initial active weight distribution Before training, ensure the active weights for both sets of networks sampled from the same distribution.
1. Initialize Create sparse and dense networks the same capacity.
3. Train both sets of networks for 1000 epochs.
Evaluate the better architecture Complete the Wileoxon Signed Rank Test.
Increasing network widthicapacity from 10% to 90% of the max network width.
Sparac Denac
Sporse
Denac
Repeat varying
Figure Same Capacity Sparse vs Dense Comparison (SC-SDC)
1. Initialize For a chosen network depth (number of layers) L and a maximum network width NMaaw, , compare sparse and dense networks various widths, while ensuring they have the same paramater count.
Initially, we mask the weights Os of sparse network S:
ass = 0 @ m'm
a' = for 1=1,...,
(1)
where O's Om' denotes element-wise product of the weights Og of layer and the random binary matrix (mask) for layer m', a's is the nonzero weights in layer I of sparse network S and a'D is the nonzero weights in layer of dense network D (all the weights since no masking occurs).
For fair comparison, need ensure the same number of nonzero weights for sparse network S and dense network D. across each layer L.
| lo ||abll,, for 1= 1=,,..,,
We provide more implementation details of how achieve this in Appendix A.I
2. Match active weight distributions Following prior work (Liu et al.) ensure the nonzero weights initialization of the sparse and dense networks sampled from the same distribution at each layer as follows:
2018b Gale et al. 2019
pl a' Pl, for 1=1,..., L
where pl refers the initial weight distribution at layer I, for example Kaiming initialization (He et al. 2015). This ensures that both sets of active weights (sparse and dense) are initially sampled from the same distribution.
3. Train We then train the sparse and dense networks for 1000 epochs (allowing for convergence).
Evaluate the better architecture We gather the results across the widths/capacity levels and conduct a paired, one-tail Wilcoxon signed-rank test (Wilcoxon) 1945) to evaluate the better archi- tecture. Our null hypothesis (Hp) that sparse networks have similar or worse test accuracy than dense networks (lower or the same median), while our alternative hypothesis (H₂) is that sparse networks have better test accuracy performance than dense networks the same capacity (higher median). This can be formulated as:
Ho Sparse <= Dense
H1 Sparse Dense
Our goal of SC-SDC is to compare sparse and dense networks at the same capacity level. By same capacity, we are referring to the same number of active weights, but other notions of same capacity can also be used. We briefly discuss this in Appendix A.
3

---

Under review as a conference paper at ICLR 2021
Table 1: The average correlation between gradient flow measures and generalization performance
Measure
Correlation to Test Loss
Correlation to Test Accuracy
Sparse
Dense
Sparse
Dense
||]11
0.3705
0.3940
0.3551
0.3750
CIFAR-100 CIFAR
[
0.3732
0.3181
0.3167
0.3840
egfi
0.4155
0.4168
0.3992
0.4041
cg.2
0.4373
0.3323
0.3833
0.3774
||9||
5
0.4030
0.4411
0,4286
0,3720
11
0,3998
0,4008
0,3974
0.3913
cg
0.4362
0.4506
0,4418
0.3821
eg.2
0.4048
0.4121
0,4142
0.3990
We compare the average absolute Kendall Rank correlation between different formulations of gradient flow and generalization. The subscript denotes the p-norm (I1 or /2 norm). We see that EGF has higher absolute correlation when compared to standard gradient flow measures. We also see that is consistent across Fashion MNIST (see Appendix B.
2.2 MEASURING GRADIENT FL.OW
Gradient flow (GF) is used to study optimization dynamics and typically approximated by taking the norm of the gradients of the network (Pascanu al. 2013: Nocedal et al. [2002] Chen et al. 2018 Wang et al. 2020 Evei et al. 2020).
We consider a feedforward neural network f : R + R, with function inputs x E RD and network weights O. The gradient norm is usually computed by concatenating all the gradients of network into a single vector, g = 00
ac
where C' is our cost function. Then the vector norm is taken as follows:
gfp ||gl|p.
(5)
where p denotes the pth-norm.
Effective Gradient Flow Traditional measures of gradient flow take the 11 or 12 norm of all the gradients (Chen et al. 2018: Pascanu et 2013 Evci et al. 2020). This is not appropriate for sparse networks, as this would include gradients of masked weights which have no influence the forward pass. Furthermore, computing /1 or 12 across all weights in the networks gives disproportionate influence layers with more weights. We instead propose simple modification of Equation 5 which we term Effective Gradient Flow (EGF), that computes the average, masked gradient (only gradients active weights) norm across all layers.
We calculate EGF as follows:
UC
OC
ac ..... agc @ ml) for 1= 1,....,;
=
001
3 m
002
o m
(6)
egs, Σ 11
Oc
where L is number of layers and 0g
O'' denotes an element-wiss product of the gradients of layer
1. OC
0el and the mask m' applied the weights of layer 1. For fully dense network, m' is matrix
of all ones, since gradients masked.
EGF has the following favourable properties:
Gradient flow is evenly distributed across layers EGF distributes the gradient norm across the layers equally, preventing layers with a lot of weights from dominating the measure and also preventing layers with vanishing gradients from being hidden in the formulation, as is the case with cquation[][whte (when all gradients are appended together).
Only gradients of active weights are used EGF ensures that for sparse networks, only gradients of active weights used. Even though weights are masked, their gradients not necessarily
4

---

Under review as a conference paper at ICLR 2021
Table 2: Different network configurations for sparse and dense comparisons
Configuration
Varlants
Optimizers
SGD SGD with mom (0.9).
Adagrad. RMSprop Adam
Aegutarization/oomalizazt method
No regularization, L2/Weight Decay, Data Augmentation,
Skip Connections and Batchnorm.
Number of hidken layers
1.2.4.
Dense Width
923, 2153, 2768.
Activation functions
ReLU, PReLU, , ELU, SReLU and Sigmoid.
Size
Learning Rate
0.001, 0.1
since the partial derivative of the weight wrt. the loss, is influenced by other weights and activations. Thereby weight can be zero, but its gradient can be nonzero.
Possibility for application in gradient-based pruning methods Tanaka et al. 2020) showed that gradient-based pruning methods like GRASP (Wang al. (020) and SNIP (Lee al.) [00 (01 disproportionately prune large layers and are susceptible to layer-collapse, which is when an al- gorithm prunes all the weights across layers, maintaining EGF (as opposed to standard gradient norm) could possibly be used as pruning criteria. Furthermore, current approaches measuring or approximating the change in gra-
specific layer. Due to the fact that EGF is evenly distributed
dient flow during pruning in sparse networks (Wang et al. 2020 Evci et al., 2020 Singh Lubana & Dick 2020), could benefit from this new formulation.
To evaluate EGF against other standard gradient norm measures, such as the /1 and norm, we empirically compare these measures and their correlation test loss and accuracy. We take the absolute average of the Kendall Rank correlation (Kendall 1938), across the different experiment configurations. We follow similiar approach Diang et al. (2019), but unlike their work which has focused on correlating network complexity measures to the generalization gap, we measure the correlation of gradient flow to performance (accuracy and loss). We measure gradient flow at 10 points evenly spaced throughout training, specifically at the end of epoch 0, 99. 199, 299, 399, 499, 599, 699, 799, 899 and 999.
Our results from Table Ihowss that EGF has higher average absolute correlation to both test loss and accuracy. This is also true of Fashion MNIST (see Appendix B.Jp. Due to the comparative benefits of EGF, we use it for the remainder of the paper to measure the impact of interventions. We include all measures of gradient flow in Appendix for completeness.
2.3 ARCHITECTURE, NORMALIZATION REGULARIZATION AND OPTIMIZER VARIANTS
We briefly describe our key experiment variants below, and also include for completeness all unique variants in Table
Activation functions ReLU networks (Nair & Hinton 2010) are known to be more resilient to vanishing gradients than networks that use Sigmoid or Tanh activations, since they only result in vanishing gradients when the input is less than zero, while on active paths, due to ReLU's lincarity, the gradients flow uninhibited (Glorot et al., 201T). Although most experiments are run on ReLU networks, we also explore different activation functions, namely PReLU (He et al., 2013). ELU (Clevert et al. 2015), Swish (Ramachandran et al. 2017). SReLU (Jin et al. 2015) and Sigmoid Neal 1992).
Batch normalization and Skip Connections Other methods help alleviate the vanishing gradient problem include the addition of skip connections (cvery two layers) (Srivastava et al. [015 He et al. 2016) and batch normalization (Toff & Szegedy 2015). We empirically explore these methods.
Optimization and Regularization techniques We explore the impact of popular regularization methods: weight decay/l2 regularization (0.0001) (Krogh & Hertz 1992 Hanson & Pratt 1989) and data augmentation (random crops and random horizontal flipping (Krizhevsky et al.) 2012)). Fur- thermore, we benchmark the impact of the most widely used optimizers such as minibatch stochastic gradient descent (with momentum (0.9) (Sutskever et al. 2013 Polyak 1964) and without momen-
5

---

Under review as a conference paper at ICLR 2021
tum Robbins & Monro 1951 RMSProp (Hinton et al. 2012).
Adam (Kingma & Ba 2014). Adagrad (Duchi et al. 201 and
3 EMPIRICAL SET-UP
SC-SDC MLP Setting We use the SC-SDC empirical setting (section 2.1 for all experiment vari- ants. We train over 6000 MLPs for 1000 epochs and evaluate performance on CIFAR-10 and CIFAR- 100 (Krizhevsky al. 2009). We compare sparse and dense networks across various widths, depths, learning rates, regularization and optimization methods as shown in Table
We choose a network width Mazw of + 4, where is the input dimension of the network. In the case of CIFAR, = 3072 and so our maximum width Na 3076. We repeat these ex- periments with one, two and four hidden layers, with the number of active weights these networks ranging from 949, 256 to 31, 765, 568 weights. In section 4 we discuss results achieved using four hidden layers on CIFAR-100 100 and we provide the one and two hidden layer results Appendix [
Dense Width Following from SC-SDC, these networks are compared at various network widths, specifically a width of 308, 923, 1538, 2153, 2768 (10%, 30%, 50%, 70% and 90% of our maxi- mum width NarawW (3076) as shown in Table We use the term dense width to refer to the width of a network if that network was dense. For example, when comparing sparse and dense networks at dense width of 308, this means the dense network has width of 308, while the sparse network has a width of N Maw (3076), but has the same number of active connections as the dense counter- part. We provide more detailed discussion of the choices made in the SC-SDC implementation in Appendix A
Extended CNN Setting We also extend our experiments to Wide Resnet-50 (the WRN-28-10 vari- ant) (Zagoruyko & Komodakis 2016) and use the optimization and regularization confi onfigurations from the paper.
4 RESULTS AND DISCUSSION
4.1 COMPARISON OF DENSE AND SPARSE INTERVENTIONS USING SC- CDC
In this section, we use the results of the Wilcoxon signed rank test from SC-SDC to identify where optimization choices are currently well suited for sparse networks and which are not. Furthermore, for each variant, we also measure the gradient flow using (GF(egJ2 (6) as described in the previous section. Our main findings show that:
1. Batch normalization is critical to training sparse networks, more than it is for dense networks. This suggests that gradient instability is a key obstacle for sparse optimization.
2. Weight decay (with and without batch normalization) and data augmentation ( (without batch nor- malization) can hurt both sparse and dense network optimization. This particularly occurs when using accelerated, adaptive optimization methods that use an exponentially decaying average of past squared gradients, such as Adam and R.MPProp (Ruder) 2016). these methods, large EGF (gradient flow) strongly correlates to poor test accuracy.
3. Non-saturating activation functions, such Swish (Ramachandran al. 2017) and PReLU (He et al. 2015), achieve promising results both the sparse and dense regime. However, these results
statistically significant when sparse networks are used and so this could motivate for the use of similar activation functions for training sparse networks.
Batch normalization plays a disproportionate role in stabilizing sparse networks Batch normal- ization ensures that the distribution of the nonlinearity inputs remains stable as the network trains, which was hypothesized to help stabilize gradient propagation (gradients do not explode or vanish) [loff & Szegedy 2015). Following from Table 3a and 3b) we see that batch normalization sta- tistically more important for sparse network performance than it is for dense networks, across most configurations and learning rates.
Note we measure RGF 10 points throughout training and take the average

---

Under review as a conference paper at ICLR 2021
(a) Effect of Different Regularization Methods 0.001 Learning Rate
DA
L2
BN
SC
BALBN_SC DA.L2_BN.SS
Adagrad
0.9997
0.9997
0.9983
0.0062
0.2385
0.0006
0.0035
Adam
0.0003
0.0547
0.1984
0.0789
0.0025
0.0391
0.1184
RMSProp
0.0005
0.0003
0.3001
0.1165
0.1664
0.0957
0.0234
SGD
0,9997
0.9997
0.9997
0.0003
0.2478
0.0013
0.0035
Mom (0.9)
0.9997
0.9997
0.9995
0.0006
0.9989
0.0005
0.0025
(b) Effect of Different Regularization Methods 0.1 Learning Rate
BN
BN SC
DA BN
L2 BN
DA BN SC
DA L2 BN SC
Adagrad
0.0003
0.0016
0.0018
0.9626
0.0234
0.0140
Adam
0.0698
0.0035
0.0023
0.0434
0.1912
0.3766
RMSProp
0.0021
0.5624
0.0267
0.0079
0.8943
0.0016
SGD
0.0005
0.0045
0.0003
0.0481
0.0004
0.0126
Mom (0.9)
0.0004
0.0027
0.0016
0.4435
0.0053
0.2118
(c) Effect of Different Activation Functions 0.1 Learning Rate
Relu
Swish
PRelu
SRelu
Sigmoid
Elu
Adagrad
0.0234
0.0045
0.0498
0.1817
0.4797
0.0032
Adam
0.1912
0.1817
0.0391
0.0620
0.0045
0.0004
R.MPPrpp
0.8943
0.1671
0.0023
0.0115
0.9973
0.1533
SGD
0.0126
0.0267
0.0045
0.0778
0.0304
0.0559
Mom (0.9)
0.2118
0.0134
0.0013
0.0778
0.0013
0.9733
NR No Regularization, BN and L2- weight decay.
Batchnorm, SC Skip Connections, DA
Data Augmention
Table 3: Wilcoxon Signed Rank Test Results for ReLU networks with four hidden layers, trained on CIFAR-100, using different learning rates. We use p-value of 0.05, the bold values indicate where sparse networks perform better than dense networks in statistical significance manner (reject Ho from while non-bold values indicate that it possible dense networks have the same or better test accuracy in that configuration. The performance results for these networks are presented in Figure
2a 12
and 10a
From Table 3a and Figure 10a we see methods such as L2 and data augmentation usually favour dense networks (apart from Adam and RMSProp. when using data augmentation). However, with the addition batch normalization (Table 3bb L2 to L2 BN and DA to DA_ ), these methods favour sparse variants. This is especially apparent in Figure 10a where batch normalization im- proves performance across all sparse optimizers, while resulting in a lower, more stable EGF. This further emphasizes the importance of stabilizing gradient flow, particularly in sparse networks.
Weight Decay and Data Augmentation can hurt sparse network optimization When we take closer look at the effects of weight decay and data augmentation on sparse network accuracy (Figure Figure [3}, we see that weight decay (even with batch normalization) drastically decreases accuracy when used with adaptive optimization methods that use exponentially decaying average of past squared gradients (Adam and RMSProp). Furthermore, it results in distinctively larger EGF values, which hints at Adam and RMSProp being more sensitive to larger gradient norms than other optimizers. This agrees with Loshchilov & Hutter (2017), who proposed a different formulation of weight decay for adaptive methods, since current 7.2 regularization formulation for adaptive methods could lead weights with large gradients being regularized less, although this not experimentally verified. In the context of data augmentation, we see poor test accuracy when is used without batch normalization (Figure T0a). If used with batch normalization (Figure 2 results in lower EGF and best test accuracy. This further emphasized the need to stabilize gradient flow and how EGF can be used to this end.
The potential non-saturating activation functions Swish and PReLU We also explore the effect of different activation functions sparse network optimization. For the activation function

---

Under review as a conference paper at ICLR 2021
(a) Test Accuracy for Dense and Sparse Networks CIFAR-100
Accurity Adrged
Netunoy fdam
loureyy wih narkklt
DURUCCA
DLUR PAL
(b) Gradient Flow for Dense and Sparse Networks on CIFAR-100
harage Credire Tae idayad
Aue ugg Gradeet flee
Aurrage Gnd mont)
NR No Regularization, BN= Batchnorm, SC Skip Connections, DA and L2- weight decay.
Data Augmention
We show the test accuracy (upper image) and gradient flow (lower image) results for Sparse MLPs with four hidden layers and a methods and promising activations. The results for all optimizers can be found in Figure 13
Figure 2:
large learning rate (0.1), across different regularization
variants, the best configuration for each optimizer was chosen. For Adagrad, Adam and RMSProp we use BN, SC and DA, while for SGD, we use BN, SC, L2 and DA.
From Table 3c we see that Swish, PReLU and Sigmoid favour sparse architectures, but from the per- formance results from Figure 12 we see that only Swish and PReLU are viable activation choices. We continue to see consistent trend for adaptive methods (most notably in Adam and RMSProp), that higher EGF values, for example in SReL.U, correspond to poor performance (Figure 11b). while promising methods result in lower EGF value (such Swish). This further emphasizes how EGF can be used to guide advances in network optimization.
4.2 GENERALIZATION OF RESULTS ACROSS ARCHITECTURE TYPES.
In this section, we move on from SC-SDC and extend our result to Wide ResNet-50. We note from Figure 33] that most of our results from SC-SDC also hold on larger, more complicated models. We see that L2 regularization (even with batch normalization) hurts performance for adaptive methods (Adagrad and Adam) and also results in higher EGF values (Figure [4). Furthermore, we also see data augmentation is beneficial when used with batch normalization. Finally, we see that Swish is a promising activation function for adaptive methods and leads to lower EGF (Figure (4). This shows that the SDC results are not constrained to small scale experiments and that can be used to learn about dynamics of larger, more complicated networks.
8

---

Under review as a conference paper at ICLR 2021
Accuracy Adagrad
Mddgacy-Adagad
Accuracy Mam
Accuracy- -S0 with mom10.9
50
18
Denslty
Densty
Density
Network Variant
DA, BN
Raiu
BN, SC. Ralu
DA BR, Swan
DA. BN SC, Prilu
Relu
MAJL2_BN_SC_RAO
DALL2 Swahh
DA SCB
widorcsnce50
Figure 3: Wed5 Test Accuracy on CIFAR-100. The density ranges from 1% to 100%. The gradient flow results can be found in Figure 14
5 RELATED WORK
Pruning at Initialization Methods that prune at initialization aim to start sparse, instead of first pre-training an overparameterized network and then pruning- These methods use certain criteria to estimate at initialization, which weights should remain active. This criteria includes using the con- nection sensitivity (Lee et al. 2018b), gradient flow (via the Hessian vector product) (Wang et al.) 2020) and conversation of synaptic saliency (Tanaka et al. 2020). Another branch of pruning is Dy- namic Sparse Training, which uses information gathered during the training process, to dynamically update the sparsity pattern of these sparse networks (Mostafa & Wang 2019; Bellec et al. 2017
Mocanu et al. 2018:] Dettmers & Zettlemoyer, [2019] Evci et 2019)) While our work is motivated by the same goal of allowing networks to start sparse and converge to the same performance as dense networks, we instead focus on the impact of optimization and regularization choices on sparse networks.
Sparse Network Optimization as Pruning Criteria Optimization in sparse networks has often been neglected in favour of studying network initialization. However, there has been work that has looked at sparse network optimization from different perspectives. mainly as guide for pruning criteria. This includes using gradient information (Mozer & Smolensky 1989, [eCun et al.) 1989
Hassibi & Stork 1992 Karnin 1990p, approximates of gradient flow (Wang et al., 2020 Dettmers
& Zettlemoyer 2019] Evei et al.] (2020) and Neural Tangent Kernel (NTK) (Liu & Zenke 2020) to
guide the introduction of sparsity.
Sparse Network Optimization to study Network Dynamics Apart from use as pruning criteria, optimization information has been used to investigate aspects of sparse networks, such their loss landscape (Evci et al.) 2019). how they are impacted by SGD noise (Frankle et al.] (019a),, the effect of different activation functions (Dubowski) 2020) and their weight initialization (Lee et al. 2019). Our work differs from these approaches as we consider more aspects of the optimization and regularization process in controlled experimental setting (SC-SDC), while using EGF to reason about some of the results.
CONCLUSION AND FUTURE WORK
In this work, we take a wider view of sparse optimization strategies and introduce appropriate tooling to measure the impact of architecture and optimization choices on sparse networks (EGF SC-SDC ). Our results show that weight decay and data augmentation can hurt optimization, when adaptive optimization methods are used and this usually corresponds to much higher EGF Furthermoree, we show how batch normalization is critical to training sparse networks, more so than it is for dense networks helps stabilize gradient flow. We also show the potential of non-saturating activation functions for sparse networks such as Swish and PReLU, Finally, we show that our results extend to more complicated models like Wide ResNet-50.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Dario Amodei, Danny Hermandez, Girish Sastry, Jack Clark, Greg Brockman, and Ilya Sutskever. Ai and compute, 2018. URL /ttpps://ppeaa...t... www..ooooooob.ooeee and- compute/
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprint arXiv: 1711.55. 2017.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-ten dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning. pp. 794 803. PMLR, 2018.
Ajórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:151 1.02289, 2015.
Yann Le Cun, John S. Denker, and Sara A. Solla. Optimal brain damage. In Advances in Neural Information Processing Systems, pp. 598-605. Morgan Kaufmann, 1990.
Janez Demsar. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research, 1(Jan):1-30, 2006.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint urXiv:1907.04844 2019.
Adam Dubowski. Activation function impact on sparse neural networks. B.S. thesis, University of Twente, 2020.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pp. 907-940, 2016.
Utku Evci, Trevor Gale. Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the Lottery: Making All Tickets Winners. arXiv e-prints, art. 1rXiv:1911.11131 November 2019.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint arXiv: 10732, 2019.
Utku Evci, Yani A Ioannou, Cem Keskin, and Yann Dauphin. Gradient flow in sparse neural net- works and how lottery tickets win. arXiv preprint arXiv:2010.03533, 2020.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: /00.0000033 2018.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. arXiv preprint arXiv:1912.05671 2019a.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M Roy, and Michael Carbin. Stabilizing the lottery ticket hypothesis. arXiv preprint weXiv:1903.0111
2019b.
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Pruning neural networks initialization: Why we missing the mark?, 2020.
Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural networks, ((3):183-11) 192, 1989.
Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks. arXiv e-prints, art. arXiv: 1ttp:::99020999990 09574, February 2019.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv preprint arXiv: /900: 09574, 2019.
10

---

Under review as a conference paper at ICLR 2021
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourleenth international conference on artificial intelligence and statisticx, pp- 315-323, 2011.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Stephen José Hanson and Lorien Pratt. Comparing biases for minimal network construction with back-propagation. Advances in neural information processing systems, pp. 177-185, 1989.
B. Hassibi, D. G. Stork, and G. J. Wolff. Optimal brain surgeon and general network pruning. In IEEE International Conference on Neural Networks, pp. 293-299 vol.1, March 1993a. doi: 10. 199.ICCNN.199..9887777777
Babak Hassibi and David G. Stork. Second order derivatives for network pruning: Optimal brain surgeon. In NIPS, pp. 1641771. Morgan Kaufmann, 1992.
Babak Hassibi, David G. Stork, and Stork Crc. Ricoh. Com. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in Neural Information Processing Systems 5, pp. 164-171. Morgan Kaufmann, 1993b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8), 2012.
Sepp Hochreiter. Untersuchungen dynamischen neuronalen netzen. Diploma, Technische Uni- versität München, 91(1), 1991.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et al. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
Sara Hooker. The Hardware Lottery. arXiv e-prints, art. arXiv:2009.06448 September 2020.
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What Do Com- pressed Deep Neural Networks Forget? arXiv e-prints, art. arXiv:191 1.11525448, November 2019.
Kurt Hormik, Maxwell Stinchcombe, Halbert White, et al. Multilayer feedforward networks universal approximators. Neural networks, 265):359-366, 1989.
M. Horowitz. 1.1 computing's 's energy problem (and what we can do about it). In 2014 IEEE International Solid-State Circuits Conference Digest of Technical Papers (ISSCC), pp. 10-14, 2014.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, abs/1502.031677, 2015. URL http::::tt:a..orrrtrt// abs/1502. 03167
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan. and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint 192.v.19,, 2019.
Xiaojie Jin, Chunyan Xu, Jiashi Feng, Yunchao Wei, Junjun Xiong, and Shuicheng Yan. Deep learning with s-shaped rectified linear activation units. arXiv preprint arXiv: / 2015.

---

Under review as a conference paper at ICLR 2021
Ehud D Karnin. A simple procedure for pruning sack-propagation trained neural networks. 1EEE transactions on neural networks, (2)):339-244 1990.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar-100 datasets. https://www. cs. toronto. edu/rrizcifar. html, 6:1, 2009.
URI:
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992.
N. D. Lane and P. Warden. The deep (learning) transformation of mobile and embedded computing. Computer, 51(5):12-16, May 2018. ISSN 1558-0814. doi: 10.1 00.1000/2
Yann LeCun, John S. Denker, and Sara Solla. Optimal Brain Damage. In NIPS, pp. 598 605.
Morgan Kaufmann, 1989.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip S. Torr. SNIP: single-shot network pruning based on connection sensitivity. CoRR, abs/1810.02340, 2018a. URL http::/axxx..orgg
abs f1as/1820 111//1110000 02340
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:/810.02340) 2018b.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv:1900 063007 2019.
Tianlin Liu and Friedemann Zenke. Finding trainable sparse networks through neural tangent trans- fer. arXiv preprint arXiv:2006.08228. 2020.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of 2018a.
network pruning. arXiv preprint
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the Value of Network Pruning. CoRR, 0tos/180000570., 2018b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 17X 2017.
C. Louizos, M. Welling, and D. Kingma. Learning Sparse Neural Networks through L.0 Regu- larization. ArXiv e-prints, December 2017.
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Advances in neural information processing systems, pp. 6231-6239, 2017.
John H McDonald. Handbook of biological statistics, volume 2. sparky house publishing Baltimore, MD, 2009.
Duane Merrill and Michael Garland. Merge-based parallel sparse matrix-vectos multiplication. In SC' Proceedings of the International Conference for High Performance Computing, Network- ing, Storage and Analysis, pp. 689. IEFE, 2016.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connec- tivity inspired by network science. Nature communications, 91):1-12, 2018.
12

---

Under review as a conference paper at ICLR 2021
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. arXiv preprint arXiv:1902.05967, 2019.
Michael C Mozer and Paul Smolensky. ming the fat from network via relevance assessment. zky (ed.), Advances in Neural Information Processing Systems 1, pp. 107- 115. Morgan-Kaufinnn
Skeletonization: A technique for trim- In D. S. Touret-
1989.
URL http://papers. nips cc/paper/
119- Si-skelle wetllelooiii inttonizaiionooo tee-ttechnqquuue que for-trrimiiggthee fat from a- net - pdf
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Sharan Narang, Erich Elsen, Gregory Diamos, and Shubho Sengupta. Exploring Sparsity in Recur- rent Neural Networks. arXiv e-prints, art. arXiv:1704.05119 Apr 2017.
Radford M Neal. Connectionist learning of belief networks. Artificial intelligence, 56(1):11-113, 1992.
Jorge Nocedal, Annick Sartenaer, and Ciyou Zhu. On the behavior of the gradient norm in the steepest descent method. Computational Optimization and Applications, 22(():5-35, 2002.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com-
putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Swish: a self-gated activation function. arXiv preprint arXiv:1710.05944 7, 2017.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv:1609.0444 (444709904777, 2016.
arXiv preprint
Ravi K Samala, Heang-Ping Chan, Lubomir M Hadjiiski, Mark Helvie, Caleb Richter, and Kenny Cha. Evolutionary pruning of transfer learned deep convolutional neural network for breast cancer diagnosis in digital breast tomosynthesis. Physics in Medicine & Biology, 63(9):095005, may 2018. doi: 10. 10100//1361-666//aabbbe
Abigail See, Minh-
Minh-Thang Thang Luong, and Christopher D. Manning. Compression of Neural Machine Jun 2016.
Translation Models via Pruning. arXiv e-prints, art.
Ekdeep Singh Lubana and Robert Dick. A gradient flow framework for analyzing network pruning. arXiv e-prints, pp. arXiv-2009, 2020.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. arXiv preprint arXiv:1505.00387. 2015.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. 2019.
Nikko Ström. Sparse connection and pruning in large dynamic artificial neural networks, 1997.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initial- ization and momentum in deep learning. In International conference on machine learning, pp. 1139 1147, 2013.
Hidenori Tanaka, Daniel Kunin, Daniel LK Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. arXiv preprint M0Xi:200000054677 2020.
Neil C. Thompson, Kristjan Greenewald, Kecheon Lee, and Gabriel F. Manso. The Computational Limits of Deep Learning. arXiv e-prints, art. arXiv:2007.055:5 July 2020.
13

---

Under review as a conference paper at ICLR 2021
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376,220 2020.
P. Warden and D. Situnayake. TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers. O'Reilly Media, Incorporated, 2019, ISBN 9781492052043. URL httpps///bookssggggl.e.oooo com/ books 3=
W. Wen, C. Wu, Y. Wang, Y. Chen, and H. Li. Learning Structured Sparsity in Deep Neural Net- works. ArXiv e-prints, August 2016.
Frank Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80-83, 1945.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 708.07777 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/ 160.067146, 2016.
URL
07146
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint 1.X0v:3 2016.
Yue Zhao, Jiajia Li, Chunhua Liao, and Xipeng Shen. Bridging the gap between deep learning and sparse matrix format selection. In Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming, pp. 94-108, 2018
M. Zhu and S. Gupta. To prune, or not prune: exploring the efficacy of pruning for model compression. ArXiv e-prints, October 2017.
14

---

Under review as a conference paper at ICLR 2021
A SC-SDC
In this section, we provide more information about SC-SDC and its benefits.
A.1 SC-SDC IMPLEMENTATION DETAILS
Wilcoxon Signed Rank Test This is non-parametric test that compares dependent or paired sam- ples, without assuming the differences in between the paired experiments are normally distributed (MDDonald 2009; Demšar 2006).
Random Sparsity Our work focuses the training dynamics of random, sparse networks. This ensures that what learned is not dependent on a specific pruning method, but rather can be used to better understand sparse training in general. Going forward, it would be interesting explore these dynamics on pruned networks.
We achieve random sparsity, by generating random mask for each layer and then multiply the weights by this mask during each forward pass. The sparsity is distributed evenly across the network. For example, 20% sparse MLP has 20% of the weights remaining in each layer.
Dense Width A critical component to how we specify our experiments a term we define as dense width. In order fairly compare sparse and dense networks, we need them to have the same number of active connections at each depth. In the case of sparse networks, this means ensuring they have the same number of active connections as the dense networks, while remaining sparse. Dense width refers to the width of a network if that network was dense. This process of comparing sparse and dense networks different dense widths illustrated figure
Fair comparison of Sparse and Dense networks As can be seen from figure SC-SDC ensures the exact same active parameter count, but the sparse networks will be connected to more neurons. It is possible that the increased number of activations being used can lead to sparse networks having higher representational power, however most work on expressivity of neural networks looks at this from depth perspective and proves certain depths of networks are universal approximators (Eldan & Shamir 2016 Hornik et al. 1989 Funahashi 1989).
To this end, ensure these networks have the same depth. but believe going forward an inter- esting direction would ensuring they have similar amount of active neurons.
SC-SDC comparison details For completeness, we provide more details of how we ensure sparse and dense networks are of the same capacity.
Following from equation2) 2 ensure the same number of weights in sparse and dense networks, can ensure they have the same number of active weights at each layer as follows:
| llo ||abllo> for 1= 1=,,...,4
(8)
This is achieved by masking each of the weight layers of sparse network S:
a 0's ml for ==
(9)
where m' is a random binary matrix (mask) for layer s.t. || Ioo = a/D- where alD is determined by the chosen capacity, these networks will be compared at.
For SC-SDC, we need a maximum network width Nazzw and comparison width Nw. We choose max network width NMazw of n + 4, where is the input dimension of the network. In the case of CIFAR, n 3072 and so our maximum width N Ma = 3076. The choice of n + 4 follows from|Lu et al. (2017), where the authors prove universal approximation theorem for width-bounded ReLU networks, with width bounded to n + 4. Our comparison width, Nw. equivalent to dense widths vary in experiments 308. 923, 1538, 2153, 2768.
The dimensions of each of layers are as follows:
First Layer:
0 E R INNN
03 R IxXNNaw m² E (0, 10
(10)
15

---

Under review as a conference paper at ICLR 2021
2. Intermediate Layers:
0 E RR-ww.Ww
0's E R.MassXX NNMaw m 11 E (0, 1] Merww
(11)
3. Final Layer:
0 E Nwx
0 R NM m' E {0,1} M
(12)
where NMsxw is maximum width of the sparse layer, Nw is the comparison width, is the input dimension, O is output dimension, L is the number of layers in the network, 0'2 is the weights in layer/ 1 of sparse network S and 0', is the weights in layer ( of dense network D.
This process would be the same for convolutional layers, but there would third dimension to handle the different channels. In figure we provide an illustrative example showing how to ensure sparse and dense networks are compared fairly.
Densely
Sparsely
6
1
4
4
%
%
5,
%
%
1
%
6
4,
%
1
Figure 4: Fair comparison of sparse and dense neural networks
A.2 BENEFITS
The benefits of SC-SDC can be summarized as follows:
We can better understand sparse network optimization. SC-SDC allows us to identify which optimization regularization methods are poorly suited to sparse networks in trolled setting, ensuring the results are a direct result the sparse connections themselves.
Learn at what parameter and size budget, sparse networks are better than dense. Comparing sparse and dense networks of the same capacity allows us to see which archi- tecture is better at different configurations. In configurations where sparse architectures perform better, we could exploit advances in sparse matrix computation and storage (Zhao) et al. 2018 Merrill & Garland, 2016) to simply default sparse architectures.
16

---

Under review as a conference paper at ICLR 2021
Densely connected
Sparsely connected
Plo
mL
A bu bg
Dense Width of
QUILE
O
10
[10] Sprine la
| Sperne Ih bg
Wilth
Inpec
Diaanoe
Dene die
Wilt
100
100
vupu Cinesaoon
Ouget
Dironden
Dinestoo
Mon H+C
Figure 5: Comparing sparse and dense neural network fairly at different widths
B GRADIENT FLOW
B.1 EGF RESULTS ON FMNIST
We extend our experiments to Fashion MNIST (Xiao et al, 2017). a dataset that is distinctively different the CIFAR datasets used in section 2.2 We 450 experiments with networks with four hidden layers, using 0.001 as a learning rate and for 500 epochs. We varied configurations as follows:
Optimizers Adagrad, Adam and SGD with momentum.
Regularization methods no regularization, batchnorm, skip connections, 12 (0.0001) and data augmentation.
From table 4
that out of the gradient flow formulations, EGF still correlates better to gener-
alization performance.
For completeness, present the full set of results using the different formulations of gradient flow
on CIFAR-100. Namely, we show |[g|] (5) (Figure 6) .|gl|2 (5) (Figure 7 .egffi (6) (Figure g and
eg.f2 (6) (Figure
17

---

Under review as a conference paper at ICLR 2021
Table 4:
The average correlation between gradient flow measures and generalization performance
for FMIST
Measure Correlation Test Loss
Correlation to Test Accuracy
Sparse
Dense
Sparse
Dense
FMNIST
0.3259
0.2522
0.3536
0.3487
0.3207
0.2702
0.3139
0.3318
c9.1
0.3534
0.2522
0,3748
0,3487
eg.fa 0.3672
0.3017
0.2314
0,3335
Gradient Flse Dorner und Sparsel MU's with hout rooken Lapats DIAA-1100
Figure 6: Gradient Flow in CIFAR-100 using |gg|]
C DETAILED RESULTS FOR SC-SDC
In this section, we presented the detailed results for our experiments.
1. Detailed Results with low learning rate (0.001).
2. Detailed Results with high learning rate (0.1).
3. Results for different activation functions.
18

---

Under review as a conference paper at ICLR 2021
Gradient Flow Dence
OIFAR
Figure 7: Gradient Flow CIFAR-100 using و
Gradient Too Sence MLPs Hidden CIFA-1100
(o0 Bey
Figure 8: Gradient Flow in CIFAR-100 using eg/1
19

---

Under review as a conference paper at ICLR 2021
Gradlent Floe Carus Sparse MUDs itaden Layara
Figure 9: Gradient Flow in CIFAR-100 using egf2
Figure 10: Effect of Regularization Accuracy and Gradient Flow for Dense and Sparse Networks on CIFAR-100, with low learning rate (0.001)
(a) Test Accuracy for Dense and Sparse Networks CIFAR-100
Aecurac Adegred
Act Adam
Accurny BMM5hpp
Act pnacy 500
Accurery SCD with manona
Demse Widi
Donue Medes
Deros Welh
0uts Wall
Type Sparm
(b) Gradient Flow for Dense and Sparse Networks on CIFAR-
Awi log Groekent Adayod
degrago Groekent Plww Adam
Average Dradee Riw. ro40ivs
Aucriat Cradlent Flw. 500
Recruse Orofiee File. 500 meveb.))
150 2M0 202
1500
203
K00 1500 DM00
190 JO 200
130
250
Detaook Yorkat
Ocedy 100
20

---

Under review as a conference paper at ICLR 2021
Effect of Activation Functions Accuracy and Gradient Flow for Dense and Sparse Networks on CIFAR-100, with large learning rate (0.1)
Figure 11:
(a) Test Accuracy for Dense and Sparse Networks on CIFAR-100
Acorary Adagaad
Accurcyy Adain
Acourcyy RMSSoop
Accu 500
ACCu SOD with (1
Cente Vidii 200
DEC
Conde Wid
3130
180g 130
Cerrae
Denes Withh
300
300
Denia With
2000
Network Wriken
Sgrood
swich
Denie
Sparse
(b) Gradient Flow for Dense and Sparse Networks on CIFAR-100
Awii aaea Crockint Ploy Adagaed
Angr Crockent (ley Adamm
Pvurage Cradient Tioe FPMPV12
Avarage Cradiet Fion SCD
Anarpe Cradint Fiw 5C0 with merst.i
Denos 203
Denis
1000 Denion M6-77
Devio #6271
190 20M
200
1000 20M
750
Striint
Dense
Sparse
Figure 12: Test Accuracy for Dense and Sparse Networks on CIFAR- 100. Each configuration is the best configuration for that optimizer. For Adagrad, Adam and RMSProp we use BN, SC and DA, while for SGD, we use BN, SC, L2 and DA.
Table 5: Test Accuracy summary for CIFAR-10 with low learning rate (0.001)
(a) One Hidden Layer
Regolarization Sporse
Augeemmation Sparse
Butchnonn
Dense
Dense
Sparse
Adagrad
54,537 a/ 0.91
58,259 e/. 0.36
56,095 ef 2,395
55.497 et 3.022
$4.505 et 0,909
55.269 s/ 0.545
$4.279 ef 2,054
$5.951 it 0.349
Adam
$1.391 +/ 1.066
52,475
e/ 0.48
37.965 mf 2,874
42 aga ml 6.494
48.671 e/ 0
49.312 o/ 1.033
52.668 +1 1,275
53.321 m/ 0.392
RMSPhop
48.000 44-0.0000 0470776
$1,507
41- 0.817
36.280 44. 4,053
40.62 44- 105
47.7 4/- 1.926
48.145
46. 468
$1. 58.581
+/- 1.617
53,045 4)- 1:283
SGD
53.714 1.564
53,023
a/. 0.46
57.091
44. 1.524
55.059 a4 3,06
53,775
45 1.559
55,091
a6- 6-46
54.364
a/: 2.363
55.996 +1: 0.696
SGI with momitl))
35/7744
+/ 1.98
54,105
36,98 +/ 1.007
57.849 +/ 0.754
540083
+/ 1,72
55,051
+/ 1.06
54 34.
+/ 2.033
35.931 +/ 0.004
(b) Two Hidden Layers
No Repabriztiion Sparse
Data Augmentation Spans
1.2 Spuese
Harchromm
Dense
Dense
Dense
Dense
Spanse
Adagrad
54.1 + 1.92
St. 333 +i- 508
50 739
wd 3.0%
5S 957
-l- 4.113
53813 wi 1
56 123
+ 301
S1. 11.003 + 2-11
S5s +/- 0.119
Adam
$2.003 af 0.91
53,785
al. 1.146
42.095
ef 7.723
49.468
al 7.545
$0.588
al 0.685
$1.681
0.668
$7.541
a/. 1,525
57.886 et: I.17]
RMSPhop
$1.352 af 1.198
53,499
af 1.5
46.872
ef 4.916
$1.999
3.436
50.382
0,602
50.52
0,915
57,432
1.512
57.972 mt 1.15]
SGID
1.918
1.735
2.16
965
54.
$6.459 -/-
SGID with momntl 9)
53:777
1.583
4/-
61.814
62.367
2.17
53,835
1.728
53.26
3.509
$6.081
2.1.8
$7.575 4)-
(c) Four Hidden Layers
Ragikattzionn
Augaoratic
Scip Cranection
Deaia
Diros
Spura
Thensr
Sparss
Deau
Admmn
sk. 17.86
1%
2011
113 199
50 p15 683
<30th 121
28. 30000 al 1.395
39 13h
32:91 0191
34.54 8085 515
KNS0/pp
ex 195
$1490
10.06
1.24
59 103 1:53
M6.7h 16.65
52.70 al 2313
58.209
0341
51.099
1.98
90T wtbh scedbay
$1399-0
(7)
(5:34 64624
63 3.30
51 11-988-1 19:15
$1382-0
1456
44661 14911
59.3%
61366 1.7%
13330 er 3.35
21

---

Under review as a conference paper at ICLR 2021
Figure 13: Effect of Different Interventions on Accuracy and Gradient Flow for Dense and Sparse Networks on CIFAR-100, with large learning rate (0.1) All Optims
(a) Test Accuracy for Dense and Sparse Networks on CIFAR-100
Aroumey Acagoe
Accundy. meppepp
Aerunry 100
WUTTS
Danio With
DA PFN
(b) Gradient Flow for Dense and Sparse Networks on CIFAR-100
Adgrut
Mooer
D.BLLCCSA 1022 2112MALC Soih
PC D0L12 JUSC PoN.
Dasty Dow
00.32 MOCC
Average Cradiont Flow Adagrad
Average Gradiont Flow Adam
Average Gradient Flow SCD with momjo. 9)
1.2
1.0
1.0
1.5
Aver aaje Goddinnt Flow
0.5
0.5
32
0.2
0.2
3.0
0.5
0.5
100%
30%
322%
1024
Denstty
Network Varlant
BN Relu
BN, Relu
DA BN, SC Swish
DA, BN SC, Prelu
COnv type
BN, Relu
12 BN Reli
DA L2. BN SC Retu
DA L2 BN SC. Swish
DA L2, 3N SC. Prolu
wideresnet50
Figure 14: Wed50 Grad Flow CIFAR-100. The density ranges from 1 1% 100%.
22

---

Under review as a conference paper at ICLR 2021
Table 6: Test Loss summary for CIFAR-10 with low learning rate (0.001)
(a) One Hidden Layer
No Regutarmntion
Data Argmennation
1,2
Boxchoom
Deese
Sperse
Demse
Spame
Denso
Spare
Dense
Sparse
Adagrad
102. 0,216
el
1.64
1.923
6.242
RM5Prup
15.672
62,005 25.008
25,578
4.582
L216
5.503
1.936
6.64
1.000
SGID
1897 +
0.161
1.25
0.04
1.305 ei 0.88
1999
00-006
1,595
+1-0115
1.009 -4- 0.000
1,683 40-00017
SGD with momple))
3,006 0.56
2 2.998-8 0.04)
1.438 -4 0.11
1.407 a/- 0.113
2,092
41-0.055
2,208 +/- 0.12
2,145 40-0.001
(b) Two Hidden Layers
Repulatization Spurse
Augitemation
Dense
Spanie
Spane
Sparse
0.678
1 40-00677
2,631
2.151 0:443
2.60
4:30.
Adam
116.25
25,294
143.085
2,006 0.305
2,232
0,474
3,98
0.234
3,379
7,009
1,768
7,309
RMSPhop
148,
30.047
161.511
74983
2.192 0.179
2,832
0.981
4.339
0.144
3985 0.514
7,406
1,568
1,495
SGD
2,907 0.488
2,008
0,366
0.051
1,268
0,145
2889
0.356
2.48 +f200.89
2.01 0000-/0.114
SGD with momel1
-/ 0.818
3,971
1,438
2442 45-552
40-0008
2,95 +i 0.594
2,988 +/-
2. 2.888 0.399
2,376 0.128
(c) Four Hidden Layers
Donec
Sparee
Dowe
Spans
Detie
Spans
Deau
Stund
Doves
Spane
647
1117 0:20
0:35
11847
WE 115)
7:210 ea:
5:11
71181
SGD
3,578 6: 1,255
L tutsS
1518 64-0 872
4.812 2029
3458 an: 1.102
2.79 63-4.305
2:385 0,15
4.311 en 0.512
2.442 1U768
Test accuracy for Sparse and Dense Networks using CIFAR-10
Antenry wertt wngges
Ridee myyttr Alen
Ancarttyy for Todden sperth
Accemay for Nertt 500
Agrnsry botee ayerrt
A00/00
Regass www.e/oooaaaaooooooooooooooooo Drstze Netivol
farmioo
Regy Evtoniio
Saly Conrarttoo
bwech Type
- Daya
S389
Figure 15: Test Accuracy for CIFAR-10 with 0.001 Learning Rate
23

---

Under review as a conference paper at ICLR 2021
Table 7: Test Accuracy summary for CIFAR-100 with low learning rate (0.001)
(a) One Hidden Layer
Regolariotion Sporse
Augmentiation Spans
Rotchturm
Dense
Dense
Sparse
Adagrad
26.76 of 0.386
27.39 of 0.545
26,958 rf 2.257
26.22 of 2,099
27.023 1/ 1,055
27.588 o/ 0.874
21,451 of 1.413
22,972 ol 0.299
Adam
23,216 +/ 1.688
24.185
of 0.657
13,401 of 1.361
15.3 4.14
21.771 +/ 1.059
22.191
of 0.493
24,255 +1 2.204
25.111 e/ 0.505
RMSPhop
22:115 46 1.05
23.764
4/- 0.737
12,805 -4- 1,663
18415 at 3,053
20.55 46 1.012
22,063
46 0.747
24.009 -/- 2.01
24.715 4/- 1.092
SGD
36.911 45 1.177
27.063
a/. 1.958
26.525
44. 1.354
24.081 a4 3.155
27.051
45 1.155
27.086
2.041
21.014 m/ 2.475
25.002 +/- 0.699
SGD with mocertt.)))
25:155
4/- 2.71
26.243
c/. 1.264
29,007
44. 1.125
29:577 at 1.315
25.663
66 2.55
26.632
6/ 1.299
21.649
m/ 2.645
23.565 4/ 0.631
(b) Two Hidden Layers
No Regubaizztion
Data Augmentation
1.2
Batchrom
Densc
Sprew
Dans:
Spanc
Denaz
Spore:
Done:
Spanc
Adagrad
18.21 1497
28.365 + N011
2k 715
+ 2.741
22.433 -t- 4
78.49s - 1.522
29 20100 + Win
24.912 + 5.763
77.518-a w+ 1:316
Adam
20.423 a/ 1.523
23.28 at 2
12.156 el 4.33
17.349 at 6.332
20.977
at 0.507
22.596 1/ 1.217
28.15 al 1.368
28.538 at 1,666
RMSPhop
18.595 1/ 2.791
21,013 e/ 4.295
12:551
el 4,55
10.099 of 5,901
210005
0.566
21,967 m/ 1.24
28.151 m/ 2.013
28.132 m/ 1.84]
SGD
22.125 */ 1.51
24,099
e/ 7.827
27.951
of 1,25
22.25
of 6.112
27.271
1.202
24,009
e/ 7938
23 80
m/ J,0K3
26,022 m/ 1,006
SGDD with mount))) 9)
26.972
2.252
26.289
4/- 3.761
31,985
31.633
2.95
27.328
2.575
26.621
3.022
27.196 4/-
(c) Four Hidden Layers
No Regalamaiis
Das Asgrenaäre
Bikteem
Adars
15,965 3,256
en
33
3,39
22.40 al
3.833
10300 wh-
991
Hobee 2.899
17,555 8.55
24852
4:51
20.00 en
23.54
40: 17711
25.153 a0-0.577
4.99 s,08s
17 0833 el 9995
11 195 2:311
20.000 0t 643
19205 v1-8
16.78 mm
29 938 1980
06 1.36
10-----2330 7.297
21.300 64-4
SGD with 4) 9y
25.209 wi 1-466
17.456 44 5:95
35:55 61-3146
24.45-40 46- 13:44
26744 6/ 1:38
16.907 -i 8:631
27:331 3407
38.83344 46 1.536
36381 61 1.83
25,506 23-3.155
Table 8: Test Loss summary for CIFAR-100 with low learning rate (0.001)
(a) One Hidden Layer
No Regelarizaion
Data Anginemation Spunee
12
Baxhuoon
Dane
Sparse
Densc
Dense
Spars
Dess:
Spare
Adagnull
1/215 4h
1:181 4 911
0.1
31177 4h 116
TA11 0 214
3:04 4.20
wi
3.845
RM
$1.721
544
24.3
743.694
37,436
14.38
15824
(A: 1,500
1.196
SCD
3,655 12 0.00
1,305 11 0.14)
5.184 0.052
3,255 11 0.134
3,517
0.03
3.289 (10013
4,055 14: 15-0.001
3.867 0.052
SKib with mome)).9)
1,038 +/- 1.577
5.677 + 0.N
5.558 44 0.231
3.335 4 0.195
5,008 el 1:201
4,22 -0-006
5008 -1-0-455
4 4.600-) 11.09
(b) Two Hidden Layers
No Reqularizaiius Spane
Data Asgrecetation Sparse
1
Bachnern
Dense
Dense
Dense
Sparse
Dense
Spare
Adagrad
4,687 -/ 0.712
4.178 -d 0.871
3:128 st 0.083
3.154 at 0,152
4,251
0,559
3.854 -/. 0.023
4,37 ex 0.129
3:784 of 0.154
Addmm
441 033 ed-
a1- 3.566
13.361
6.434
9.121
2.255
15,406 +5-
666
RMSP
346 064
212
4.157 0.293
6.024
18,274
0.776
11,817 3.779
16,005
17.701 1.539
+t 0.31
SGD with (0
7.993 1.534
6.507 1.225
4.759
1.465
5.623
1.258
0.873
4,357 0.068
(c) Four Hidden Layers
Denae
Spone
Deve
Dreae
Ypuww
Deew
Ypare
Drese
Spene
6.7.4000 2-133
IADL and 5.155
4.84 1332
Bull en 3.60
10 1al 43%
4,19 GARE
5,00 2.37
222105 4NLS1
Mall 764 128.22
14 444 1AT
5.9 46 8.31
22345 413311
23 957 46
461 548
3123
12:044
42 6532
ml 5.5
k.EW 1775
4.35 6.855
ol 1195
Table 9: Test Accuracy summary for CIFAR-10 with high learning rate (0.1)
(a) Four Hidden Layers
Tre Regakomaainn
I140 Angrenstiion
Halllaare
Aluusd
21 123777-0
FAIRS
11,664
DU56 13.99
135 $4.52
41712
al 17.113
15.0 OUT
8309
100
107 4.
5:58
5099 en
100
1.545
55
457 3.90
0008 1,66
8,25 15499
30.001 26518
99199
91.35 0060
$7.899 1.302
56,99 al. 2.00
5GD (0
9300 e: 2
12
1400
(00 02
10086 45 19.11
10279 33.540
64 23.74
$7.822 AA 1,11
99888 0.86
9.177 41-9777
Table 10: Test Loss summary for CIFAR- with high learning rate (0.1)
(a) Four Hidden Layers
Bicaa
Dana
2pane
Doan
Spuae
Soates
Disag
Spana
Acagra
2320 SAMG
230 10-00
230 wi
450 Wi Addi 101446
1.38
51.00 20 34313
85.55 18837
4150 133
360 011
6-000 8-34
24

---

Under review as a conference paper at ICLR 2021
Test accuracy for Sparse and Dense Networks using CIFAR 100
Accoray do Wort 150
Atretyy 21
Boyala
Auprontattonn
Figure 16: Test Accuracy for CIFAR-100 with 0.001 Learning Rate
Test accuracy for Sparse and Dense Networks using CIFAR-10
Srewny spert degger
Morth
Agpnsry Syeersl
hogular www.oootoeaaaaaaooooooooo lestie Nottod
Butomam
Dute Asgrnotation
Skip Conractonn
Das Typ
Dens
Figure 17: Test Accuracy for CIFAR-10 with 0.1 Learning Rate
Table 11: Test Accuracy summary for CIFAR-100 100 with high learning rate (0.1)
(a) Four Hidden Layers
Data Negrestiien
Adbaral #X05Pv
11.333 T203
14822 10al-00 0.0
23.355
R.E
0.
5" est
10 6.0
10e4-00
1:16
SOD cauas3.55
5.5 03-44-4.30 62796
10-4-00
4:411 3.42
12:255 30.25
31.541 2.244
31.37% 46 1,700
1300 02-0
1,009 e 2477
25

---

Under review as a conference paper at ICLR 2021
Gradient Sparse Networks on
512
Figure 18: Gradient Flow for CIFAR-10 with 0.1 Learning Rate
Table 12: Test Loss summary for CIFAR-100 with low learning rate (0.1)
Four Hidden Layers
Saine
Spece
1M0 AU-LORT
413 11
5.00 40-30
ASTI 334
E.IIT 1856
e 58347)
13.0 2005
3-48
1159 1.2
$322-0 2.9
45.8504
5:00 028
+4-27 7.99
6A11 1850
Test accuracy Sparse
using CIFAR-100
Eoadly
Figure 19: Test Accuracy for CIFAR-100 with 0.1 Learning Rate
Gradient Flow for Sparse Networks and Dense Networks CIFAR-100
Gradect teyert
levarti memeatt
Figure 20: Gradient Flow for 100 with 0.1 Learning Rate
Test Accuracy for Sparse MLPs with Four Hidden Layers an CIFAR-100
sobutty 1889 Astrnc arang
Aouano -540
Figure 21: Test Accuracy for Different Activation Functions
26

---

Under review as a conference paper at ICLR 2021
Gradlent Flow for Sparse MLPS with Four Hidden Layers CIFAR-100
Figure 22: Gradient Flow for Different Activation Functions
27


Reviews:
[{"review_id": "m3dJVdCq0uT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper looks at how choices in optimization affect how well you can train sparse networks. The authors come up with a new measure of effective gradient flow, which is important for good performance. They also compare sparse vs. dense networks across various optimizers, hyperparameters and activation functions, and find that batch norm and certain activation functions are beneficial for sparse networks\n\nPros\n* It is important to understand what’s better for training of sparse networks. There are previous studies of the effect of overparameterization, but much less about sparse networks vs. dense networks at the same size.\n* The authors did a multitude of experiments thoroughly to compare different training configurations\n\nCons\n* While it is interesting to compare sparse network to their same capacity dense networks, this comparison is less practical because small dense networks are rarely used in practice.\n* I’m unsure about the novelty or rationale of the EGF definition: shouldn’t masked weights already have a gradient of 0? What is the rationale for not weighing each layer’s gradient equally, even if some have more weights than others?\n* How are the sparse connections determined? If it is random, then the findings are less applicable because people would use sparse networks found by a good pruning method rather than a random sparse network in practice.\n* While table 1 shows that EGF is better than simple gradient norms, gradient flow measures in general do not seem that important to the rest of the paper: why do we need gradient flow when we can just look at accuracy? Is EGF used for anything besides as an observation? Also, it is unclear in figure 2 if high EGF can explain good performance or if they are just somewhat correlated.\n\nOther questions:\n* How do you specify the number of active connections if $a_D$ of a fully connected layer is determined by number of inputs * number of outputs? Do you adjust the number of inputs or outputs, or both? How does the procedure work with conv layers? \n* Table 1: when do you take EGF measures - throughout training or at the beginning or end? Also, for the experiments in figure 2, how does EGF change over time and is it averaging EGF over training the best way to use this measure?\n* Why use a Wide ResNet rather than a regular ResNet?\n* Doesn’t data augmentation hurt dense networks more for adam?\n* What sparsity level does table 3 use?\n\nAdditional suggestions\n* Perhaps it would also be useful to see the effect of adjusting more common/simple hyperparameters such as learning rate, batch size, and momentum.\n* I would like to see accuracies of the experiments in table 3: even though it is scientifically interesting to see where sparse networks do better relative to dense networks, ultimately the goal is to get the best performing sparse network in terms of absolute accuracy. Figure 2 shows that, but not for all the combinations.\n* Minor: table 4 might be easier to visualize as a chart, similar to figure 2.\n* Minor: “batch normalizationalization” typo on page 5, and figures 7 and 9 do not show up\n\nOverall: This paper examines many experiments to improve the optimization of sparse networks, but I am not convinced that the approaches are the most effective (how important is EGF and comparing sparse to same size dense). Thus, I recommend rejection but am open to changing my stance based on the authors’ response.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jFRI1qTzR97", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis work performs a systematic study to understand how various design choices affect the performance and the training behavior of sparse neural networks. In order to do this, it proposes a simple framework to compare sparse and dense networks (SC-SDC) and a simple common-sense metric to measure the gradient flow in sparse neural networks (Effective Gradient Flow). It is the first time when I see such study, and I believe that it can contribute to the way how we perceive sparse neural networks as it brings a new perspective. \n\nStrong Points:\n\nS1) The paper can guide the researcher or the practitioner to make good design choices with respect to sparse networks in a fast manner.\n\nS2) The well-designed experiments show the counterintuitive fact that even static random generated sparse network can outperform dense networks at the same capacity (number of parameters) if proper settings which improve the gradient flow are made, e.g. PReLU or SReLU as activation function, batch normalization.\n\nS3) The results suggest that sparse networks do not necessarily falls under the same behavior of their dense counterparts and highlight the importance of having specialized algorithms and methods for them.\n\nS4) In general, the paper is clear and well written, with some exceptions which likely show that it has been developed until the very last moment before submission.\n\n\nWeak Points:\n\nW1) The paper address just static sparse networks. As static networks still have a limited capacity, I believe that it would be much more informative if (ideally) the same sparse random generated networks would be studied in the same conditions using also dynamic sparse training.\n\nW2) The claim that comparing sparse networks with dense networks having the same number of parameters (thus fewer neurons) is fair does not sound completely fair to me. I agree that it is a tricky situation overall, but I would consider also the fact that more neurons mean more representational power and, thus, sparse networks have an \"unfair\" advantage from this perspective.\n\nW3) The general appearance of the paper suffers from, probably, the last-minute submission.\n\n\nWhile the paper has clearly added value, overall, I found it a bit immature for publication. During the rebuttal phase, I would recommend to the authors to address the above weak points and the following (minor) comments:\n\nC1) Make it clear from the begin (not just in the appendix) that you are discussing about static sparse networks to avoid confusion. Auxiliary, perhaps you can add some details about the sparse networks initialization in Table 2.\n\nC2) It is very interesting in Table 4, first row, that for this particular case the networks fail to converge in the two “extreme” cases (Dense and 20% density), while the middle density levels work fine. Do you have any idea why this happens? Same reasons for the sparse and the dense networks? Perhaps, this particular case, can help in developing more robust training methods for neural networks in general?\n\nC3) The clarity of the Dense Width paragraph from page 5 can be improved\n\nC4) Perhaps you can add also the dense network performance in Fig. 2?\n\nC5) Try to perform a thorough proof-read of the paper. There are many typos (e.g. page 3, “layer in l”), and other sloppy written aspects (e.g. detail MAW, tables 11 and 12 have the same learning rate value, figures are missing from the appendix, mention somewhere in tables 9-12 the activation function, etc.) \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "WDnXTWsgOC", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Recently, initialization has been found critical to the model accuracy attained by sparse training [1]. In this paper, the authors studies the impact of factors other than initialization on the model accuracy attained by sparse training relative to dense training (under the same model parameter count). At the core of this paper, the authors argue that the effective gradient flow (grad norm from only activate model weight dimensions) is an effective indicator on the model accuracy attained by sparse training. Firstly, the authors show that the effective gradient flow attains higher correlation with model accuracy than the norm of full gradient (including gradients on sparsified weight dimensions) in sparse training. Secondly, the paper empirically demonstrate that in sparse training, 1) weight decay and data augmentation can hurt model accuracy, 2) batch normalization plays significantly role for model accuracy in sparse training and 3) non-saturating activations boost the magnitude of effective gradient flow and consequently improve model accuracy. \n\nI think the direction this paper worked on can have meaningful influence on designing model structure and general training algorithms for sparse training. Specifically, the effective gradient flow can potentially be an objective to improve for innovate methods in sparse training. Additionally, the investigation presented the impact of different model and training factors on model accuracy that has not been systematically studied before; this could bring up discussions for more broader techniques in sparse training.\n\n\nCurrently I give marginal reject because I have the following comments on technical aspect and clarity regarding meeting the acceptance criteria. I am happy to raise the rating if the authors can resolve them during the rebuttal phase.\n\n1. In Table 1. the authors show that effective gradient flow achieves higher correlation with model accuracy than full grad norm. However the difference in the correlation attained by effective gradient flow and full grad norm is not significant. I was wondering which correlation is used here? Is it the Pearson Coefficient for linear correlation or rank correlation? If the rank correlation is not used here, I would suggest the authors to present results on rank correlation. Given that the effective gradient flow and model accuracy might not follow a linear relationship, the rank correlation can help better evaluate whether better effective gradient flow tends to imply better model accuracy in sparse training.\n\n2.  In section 4.1, how are the training hyperparameter tuned for different methods such as using and without using weight decay? Some quick elaboration on the fairness for tuning each setting will be useful to help evaluation the conclusion drawn in section 4.1. Also is the model trained for enough number of optimizer steps to attain almost-saturating accuracy for each specific setting?\n\n3. In section 4.1, it is a bit surprising to me that data augmentation can hurt generalization performance in sparse training. Are there any intuitive reasons for this observation?\n\n\nBesides the above major comments, here are the minor comments for further improving the paper:\n\n1. The definition and use of symbol a_s is confusing. In Equ. (2), a_s is a deterministic scalar value while in Equ. (3) a_s becomes a random variable.\n\n2. It would be helpful to clarify why the authors choose to use 1000 epochs (or is it 1000 optimizer steps?) uniformly in the experiments; this can help readers to better evaluate the experiment protocols.\n\n3. Symbol m in Equ. (6) should be bold to be consistent with vector notations on gradient.\n\n4. Larger font in figures will be useful for smooth reading.\n\n5. I totally understand that in section 4.1 that MLP for CIFAR100 is a micro experiment for study and < 40% accuracy can be acceptable. But giving dense training accuracy numbers/curves under the same model parameter count can be helpful better evaluating the sparse training accuracy relative to dense training.\n\n\nReference \n[1] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks. Jonathan Frankle, Michael Carbin. ICLR 2018.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GumbPK8xTzP", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes effective gradient flow (EGF), which is a layer-wise normalized gradient flow. Compared to (unnormalized) gradient flow, the paper shows that the proposed EGF is (slightly) better correlated with metrics like test loss and test accuracy (see Table 1). Given that this claim is supported with experimental results, the paper would become much stronger if a larger number and a more diverse set of data-sets were used (in addition to CIFAR-10 and CIFAR-100, which are two very similar image-data-sets) as to show that the claim holds more generally. Apart from that, given that the correlations are (only) about 0.4 in Table 1, it seems that only some aspects are explained by EGF. \n\nThis said, if we continue with the insight that EGF is helpful, how would EGF help to design better sparse models and optimization methods? Why is it easier to compute EGF instead of directly the relevant metrics (like test loss etc.) to determine what training methods and model architectures work better?  In fact, Figure 2, top row, exactly does this: test-accuracy is directly used to determine which training methods work best for sparse models. What is the additional insight/benefit of using EGF (like in the bottom row of Figure 2) instead of test-accuracy ?\n\nThe paper also proposes the SC-SDC framework, see also results in Table 3. The key idea is to compare  sparse and dense networks that have the same number of (non-zero) weights. While this is a good start, I am not sure that this is really a fair comparison, though. I would expect the sparse network to have a larger capacity than a dense network with the same number of parameters. While there are many ways to see this, the simplest  might be that  a weight in a sparse model requires two parameters, its value and its index if we wanted to encode the model. Another way to see it might be that a dense model can be pruned to become sparse(with fewer weights), but without losing much prediction accuracy. As a consequence of potentially comparing sparse and dense networks of different capacities, the results in Table 3 might be biased in favor of sparse models. From this perspective, it is remarkable that sparse models do not clearly outperform dense models in Table 3, which might indicate that the training of sparse models with the analyzed approaches still suffers from the problems outlined earlier in the paper.\n\nEquipped with the proposed methods EGF and SC-SDF, the paper then analyzes several (standard) approaches like batch normalization etc as to determine which of these approaches are useful for training sparse models.  Again, given that only two very similar data-sets were used (CIFAR-10 and 100), it is unclear if the found results would generalize to other data sets. Moreover, it is not clear why EGF and SC-SDF are needed to determine which training-methods work well for sparse models. In fact, Figure 2 (top row) directly shows the test-accuracy for the various approaches--i.e., without using the proposed EGF and SC-SDF. EGF is shown in the bottom row in Figure 2, as to illustrate that EGF has a similar behavior as test accuracy in the first row in Figure 2. What is the additional insight obtained from EGF compared to using only test-accuracy as to determine that  batch-normalization works well ? \n\nSome minor points:\n\nHow are “test loss“ and “test accuracy” defined in this paper, and what is the difference? I did not immediately find the answer in the referenced paper (Jiang 2019).\n\nAre really all 6 digits statistically significant in Table 1 ?\n\nFigures 7 and 9 in the appendix are missing the actual image.\n\nThe paper has several typos, like “and.”, or the first sentence in Appendix A\n\n+++ updates after authors' feedback +++\n\nI  thank the reviewers for their detailed response. I still feel that more work (experimental and theoretical, as outlined in my review) is needed. \n\nI also would like to make sure that my point is not misunderstood when I said that the sparse model requires twice as many parameters to be stored (the value and the index),  compared to the dense model. Using compression algorithms, the number of bits to store the model can obviously be reduced (below the factor of two). Anyways, the deeper point that I wanted to make  was the connection between minimum description length  (number of bits to store the model) in information theory and the model complexity /capacity in statistics/machine learning: see BIC in https://en.wikipedia.org/wiki/Minimum_description_length\nHence, the bits to store a model are directly related to the model-complexity/capacity in machine-learning/statistics.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a new measure of effective gradient flow (EGF), and also compared sparse vs. dense networks on CIFAR-10 and CIFAR-100. The notion of EGF would be interesting, but the paper did not present enough evidence to support this notion.
Decision: Reject