A Signal Propagation Perspective for Pruning Neural Networks at Initialization
Authors: Namhoon Lee
Thalaiyasingam Ajanthan
Stephen Gould
Philip H. S. Torr
Venue:  (2020)
Keywords: neural network pruning
signal propagation perspective
sparse neural networks

Abstract: Network pruning is a promising avenue for compressing deep neural networks. A typical approach to pruning starts by training a model and then removing redundant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done at initialization prior to training, based on a saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as a form of gradient, we formally characterize initialization conditions to ensure reliable connection sensitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifications to the existing pruning at initialization method lead to improved results on all tested network models for image classification tasks. Furthermore, we empirically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, can be useful in various scenarios where pruning is applied to non-standard arbitrarily-designed architectures.

Published as a conference paper at ICLR 2020
A SIGNAL PROPAGATION PERSPECTIVE FOR PRUNING NEURAL NETWORKS AT INITIALIZATION
Namhoon Lee', Thalaiyasingam Ajanthan", Stephen Gould', Philip H. S. Torr
University of Oxford
Australian National University
[naahhonn, phst robots... ox wx. ac.uk
2 (thalaaiysingamm a ajant han, stephen, gould (anu. edu au
ABSTRACT
Network pruning is promising avenue for compressing deep neural networks. typical upproach to pruning starts by training model and then removing redun- dant parameters while minimizing the impact on what is learned. Alternatively, a recent approach shows that pruning can be done initialization prior to training, based on saliency criterion called connection sensitivity. However, it remains unclear exactly why pruning an untrained, randomly initialized neural network is effective. In this work, by noting connection sensitivity as form of gradient, we formally characterize initialization conditions to ensure reliable connection sen- sitivity measurements, which in turn yields effective pruning results. Moreover, we analyze the signal propagation properties of the resulting pruned networks and introduce a simple, data-free method to improve their trainability. Our modifica- tions to the existing pruning at initialization method lead to improved results on empir- all tested network models for image classification tasks. Furthermore, ically study the effect of supervision for pruning and demonstrate that our signal propagation perspective, combined with unsupervised pruning, be useful in various scenarios where pruning is applied to non-standard arbitrarily-designee architectures.
INTRODUCTION
Deep learning has made great strides in machine learning and been applied to various fields from computer vision and natural language processing, to health care and playing games (LeCun et al. 2015). Despite the immense success, however, it remains challenging computational and memory requirements of large neural network models. To this end, lightweight models often preferred, and network pruning, technique reduce parameters in network, has been widely employed to compress deep neural networks (Han et al. 2016). Nonetheless, designing pruning algorithms has been often purely based on ad-hoc intuition lacking rigorous underpinning, partly because pruning was typically carried out after training the model as a post-processing step or interwoven with the training procedure, without adequate tools to analyze.
deal with the excessive
Recently, Lee et al. 2019) have shown that pruning can be done on randomly initialized neural net- works in a single-shot prior training (i.e., pruning at initialization). They empirically showed that as long as the initial random weights are drawn from appropriately scaled Gaussians (e.g., Glorot & (2010), their pruning criterion called connection sensitivity can be used to prune deep neural networks, often an extreme level of sparsity while maintaining good accuracy once trained. How- ever, it remains unclear as to why pruning at initialization is effective, how it should be understood theoretically and whether it can be extended further.
Bengio (20T0),,
In this work, we first look into the effect of initialization on pruning, and find that initial weights have critical impact connection sensitivity, and therefore, pruning results. Deeper investigation shows that connection sensitivity is determined by an interplay between gradients and weights. Therefore when the initial weights are not chosen appropriately, the propagation of input signals into layers of

---

Published as a conference paper at ICLR 2020
these random weights can result in saturating error signals (f..,, gradients) under backpropagation, and hence unreliable connection sensitivity, potentially leading to a catastrophic pruning failure.
This result leads us to develop signal propagation perspective for pruning at initialization, and to provide a formal characterization of how a network needs to be initialized for reliable connection sensitivity measurements and in turn effective pruning. Precisely, we show that sufficient condition connection sensitivity is layerwise dynamical isometry, which is defined as all singular values of the layerwise Jacobians being concentrated around 1. Our signal propagation perspective is inspired by the recent literature on dynamical isometry and mean field theory (Saxe) al. 2014 Poole al., 2016; Schoenholz et al., 22077: Pennington et al., 2017). in which the general signal propagation in neural networks is studied. We extend this result to understanding and improving pruning at initialization.
to ensure Coinecionooomectiioon
Moreover, we study signal propagation in the pruned sparse networks and its effect on trainability. We find that pruning neural networks can indeed break dynamical isometry, and hence, hinders signal propagation and degrades the training performance of the resulting sparse network. In order to address this issue, we propose a simple, yet effective data-free method to recover the layerwise orthogonality given the sparse topology, which in turn improves the training performance of the compressed network significantly. Our analysis further reveals that in addition to signal propagation, the choice of pruning method and sparsity level can influence trainability in sparse neural networks.
Perfect layerwise dynamical isometry cannot always be ensured in the modern networks that have components such ReLU nonlinearities (Pennington et al. (017) and/or batch normalization (Yang) et al. 2019). Even in such cases, however, our experiments on various modern architectures (includ- ing convolutional and residual neural networks) indicate that connection sensitivity computed based on layerwise dynamical isometry is robust and consistently outperforms praning based on other ini- tialization schemes. This indicates that the signal propagation perspective not only important to theoretically understand pruning at initialization, but also it improves the results of pruning for a range of networks of practical interest.
Furthermore, this signal propagation perspective for pruning poses another important question: how informative is the error signal computed on randomly initialized networks, or can we prune neural networks even without supervision? To understand this, we compute connection sensitivity scores with different unsupervised surrogate losses and evaluate the pruning results. Interestingly, our re- sults indicate that we can in fact prune networks in an unsupervised manner to extreme sparsity levels without compromising accuracy. and it often compares competitively to pruning with supervision. Moreover, test if pruming at initialization can be extended to obtain architectures that yield better performance than standard pre-designed architectures with the same number of parameters. In fact, this process, which call neural architecture sculpting, compares favorably against hand-designed architectures, taking network pruning one step further towards neural architecture search.
2 PRELIMINARIES
Pruning at initialization. The principle behind conventional approaches for network pruning is to find unnecessary parameters, such that by eliminating them the complexity of the model is reduced while minimizing the impact on what is learned (Reed) 1993). Naturally, a typical pruning algorithm starts after convergence to minimum or training to some degree. This pretraining requirement has been left unattended until Lee et al. (2019) recently showed that pruning can be performed on untrained networks at initiailzation prior to training. They proposed a method called SNIP which relies on a new saliency criterion, namely connection sensitivity, defined as follows:
(9; D) D)
where 93 (w: = aL(c O w: D)
sy
w:: D)
(I)
PF A-= [g
dc,
Here, 8, is the saliency of the parameter j. w E RTh is the network parameters, C {0. 1) is the auxiliary indicator variables representing the connectivity of network parameters, m is the total number of parameters in the network, and D is a given dataset. Also, 9j is the derivative of the loss with respect to Cj. which turns out to be an infinitesimal approximation of the change in the
The term faithful is used to describe signals propagating in network isometrically with minimal ampliti- cation attenuation, and borrowed from Saxe et al. (2014), the first work to introduce dynamical isometry.
2

---

Published as a conference paper at ICLR 2020
loss with respect to removing the parameter j. Designed to be computed initialization, pruning is performed by keeping top-k (where & denotes a desired sparsity level) salient parameters based on the above sensitivity scores.
Dynamical isometry and mean field theory. The success of training deep neural networks is due in large part to the initial weights (Hinton & Salakhutdinov 2006 Glorot & Bengio 2010 Pascanu et al., 2013). In essence, the principle behind these random weight initializations is to have the mean squared singular value of network's s input-output Jacobian close to so that on average, an error vector will preserve its norm under backpropagation; however, this is not sufficient to prevent amplification or attenuation of an error vector on worst case. A stronger condition that having as many singular values as possible near is called dynamical isometry (Saxe et al.) (0144). Under this condition, error signals hackpropagate isometrically through the network, approximately preserving its norm and all angles between error vectors. Alongside dynamical isometry, mean field theory is used to develop theoretical understanding of signal propagation in neural networks with random parameters (Poole et al. 2016). Precisely, the mean field approximation states that preactivations of wide, untrained neural networks can be captured as a Gaussian distribution. Recent works revealed maximum depth through which signals propagate initialization, and verified that networks are trainable when signals can travel all the way through them (Schoenholz et al.) 2017
Yang &
Schoenholz 2017 Xiao al. 2018).
3 SIGNAL PROPAGATION PERSPECTIVE TO PRUNING RANDOM NETWORKS
Problem setup. Consider a fully-connected. feed-forward neural network with weight matrices W E N biases b' E R.,, pre-activations h' E IR,, and post-activations x E RR,, for E {1 (1... K} up to K layers. Now, the feed-forware dynamics of a network can be written as,
x ohh)),
h'= w'X'++bb
where ø : + R is an elementwise nonlinearity, and the input is denoted by x². Given the network configuration, the parameters are initialized by sampling from a probability distribution, typically zero mean Gaussian with scaled variance (LeCun et al. 1998 Glorot & Bengio 2010).
3.1 EFFECT OF INTIIALIZATION ON PRUNING
is observed Lee et al. (2019) that pruning results tend to improve when initial weights are drawn from scaled Gaussian, so-called variance scaling initialization (LeCun et al. 1998] Glorot & Bengio 2010 He et al.] 2015). As we wish to better understand the role of these random initial weights pruning, we will examine the effect of varying initialization on the pruning results.
In essence, variance scaling schemes introduce normalization factors to adjust the variance a of the weight sampling distribution, which can be summarized that depends on architecture specification such as the number of output neurons in the previous layer (e.g., fan-in), and a is a global sealar throughout the network. Notice in case of a network with layers of the same width, the variance can be controlled by a single scalar y as 1/1 vb for all layers I. In particular, we take both linear and tanh multilayer perceptron networks (MLP) of layers K = 7 and width N = 100 on MNIST with a = 1 as the default, similar to Saxe et al. (2014). We initialize these networks with different 7, compute the connection sensitivity, prune it, and then visualize layerwise the resulting sparsity patterns c as well as the corresponding connection sensitivity used for pruning in Figure
- o, where th is layerwise scalar
It is seen in the sparsity patterns that for the tanh network, unlike the linear case, more parameters tend to be pruned in the later layers than the earlier layers. As result, this limits the learning capa- bility of the subnetwork critically when a high sparsity level is requested; e.g., for R = 90%, only few parameters in later layers are retained after pruning. This is explained by the connection sensi- tivity plot. The sensitivity of parameters in the nonlinear network tends to decrease towards the later layers, and therefore, choosing the top-k parameters globally based on the sensitivity scores results in subnetwork in which retained parameters are distributed highly non-uniformly and sparsely to- wards the end of the network. This result implics that the initial weights have a crucial effect on the connection sensitivity, and from there, the pruning results.
3

---

Published as a conference paper at ICLR 2020
R
linear (K=7)
tanh (K=7)
linear (K=7)
tanh (K=7)
10
3.0
3.0
2.5
2.5
30
CS (x
50
70
1.0
1.0
0.5
0.5
90
0.0
0.0
2 3
4 layer
5
6
2
3
4 layer
5
6
2
3
4
5
2
3
4
5
6
layer
layer
Figure 1: (left) layerwise sparsity patterms E (0, 1) 100110010000 obtained as a result of pruning for the sparsity level & = {10, .., 90) %. Here, black(()/white(() pixels refer to pruned/retained parameters; (right) connection sensitivities (CS) measured for the parameters in each layer. All networks are initialized with 7 = 1.0. Unlike the linear case, the spansity pattern for the tanh network is non- uniform over different layers. When pruning for high sparsity level (e.g., R 90%), this becomes critical and leads to poor learning capability as there are only a few parameters left in later layers. This is explained by the connection sensitivity plot which shows that for the nonlinear network parameters later layers have saturating, lower connection sensitivities than those in earlier layers.
3.2 GRADIENT SIGNAL IN CONNECTION SENSITIVITY
We posit that the unreliability of connection sensitivity observed in Figure T due poor signal propagation: initialization that projects the input signal to be strongly amplified attenuated in the forward pass will saturate the error signal under backpropagation (i.e., gradients), and hence will result in poorly calibrated connection sensitivity scores across layers, which will eventually lead to poor pruning results, potentially with complete disconnection of signal paths (e.g., entire layer).
Precisely, give the relationship between the connection sensitivity and the gradients follows. From Equation D connection sensitivity is a normalized magnitude of gradients with respect to the connectivity parameters Here, we the vectorized notation where denotes all learnable parameters and denotes the corresponding connectivity parameters, From chain rule, we can write:
aL(c ④ w; D) Ōc
aL(c @ D) @
aT(w;; D) Ow
c=1
Therefore, 0L/0c is the gradients OL/Ow amplified (or attenuated) by the corresponding weights w, i.e., 0L/Oc, OL/ôw, Wj for all j (1...m), m). Considering aL/Oc; for given j. since Wy does not depend any other layers or signal propagation, the only term that depends on signal propagation in the network is the gradient term al./ww,. Hence, a necessary condition to ensure faithful 0L/0c (and connection sensitivity) is that the gradients åL/ôw need to be faithful. In the following section, we formalize this from a signal propagation perspective, and characterize an initial condition that ensures reliable connection sensitivity measurement.
3.3
LAYERWISE DYNAMICAL ISOMETRY
3.3.1
GRADIENTS IN TERMS OF JACOBIANS
From the feed-forward dynamics of network in Equation 2 the network's input-output Jacobian corresponding to a given input x" can be written, by the chain rule of differentiation, as:
OxK Ox0
JoK
ID'''', 1=1
where D' E NxN is diagonal matrix with entries D', o' o'(h)) with ø' denoting the deriva- tive of nonlinearity o, and d,p = 1li = jl is the Kronecker delta. Here, we will use J to denote the Jacobian from layer & to layer L. Now, we give the relationship between gradients and Jacobians:
Proposition 1. Let 0T./ax* K denote the error signal and x denote the input signal. Then,
4

---

Published as a conference paper at ICLR 2020
1. the gradients satisfy:
B = 6 x
(5)
where J K 0x* /0x' is the Jacobian from layer l to the output and @ is the Kronecker product. 2. additionally, for linear networks, i.e., when Φ is the identity:
Sw J (Jo-
+ aa)
(6)
0x'-1/0x0 the Jacobian from the input to layer I and E R. is constant term that does not depend on x
where J0.-1
Proof. This can be proved by an algebraic manipulation of the chain rule while using the feed- We provide the full derivation in Appendix A
forward dynamics in Equation 2
Notice that the gradient at layer constitutes both the backward propagation of the error signal e up to layer and the forward propagation of the input signal x up to layer 1-1. 1. Moreover, especially in the linear case, the signal propagation in both directions is governed by the corresponding Jacobians. We believe that this interpretation of gradients is useful as sheds light on how signal propagation affects the gradients. To this end, we next analyze the conditions the Jacobians, which would guarantee faithful signal propagation in the network, and consequently, faithful gradients.
3.3.2 ENSURING FAITHFUL GRADIENTS
Here, we first consider the layerwise signal propagation which would be useful to derive properties on the initialization to ensure faithful gradients. To this end, let us consider the layerwise Jacobian:
Ox
0x = D'w'.
Note that it is sufficient to have layerwise dynamical isometry in order to ensure faithful signal propagation in the network.
Definition 1. (Layerwise dynamical isometry) Let J ax
ax--T RMM 1 be the Jacobian matrix of layer 1. The network is said to satisfy layerwise dynamical isometry if the singular values of 1 are concentrated near for all layers, ie., for a given e >> 0, the singular value oy satisfies [1-oz oy S e for all j.
This would guarantee that the signal from layer I to / - (or vice versa) is propagated without am- plification attenuation in any of its dimension. From Proposition [ and Equation 7 by induction, is easy to show that if the layerwise signal propagation is faithful, the error and input signals will faithfully propagate throughout the network, resulting in faithful gradients.
For linear networks, JL-1, = W Therefore, one can initialize the weight matrix to be orthogonal such that (W')' = I, where I is the identity matrix of dimension N. In this case, all singular values of W are exactly (i.e., exact dynamical isometry), and such an initialization guarantees faithful gradients. While linear network is of little practical use, we note that helps to develop theoretical analysis and provides intuition as to why dynamical isometry is a useful measure.
For nonlinear networks, the diagonal matrix D' needs to be accounted for as it depends on the pre-activations h' at layer l. In this case, it is important to have the pre-activations h' fall into the linear region of the nonlinear function é. Precisely, mean-field theory assumes that for large-N limit, the empirical distribution of the pre-activations h' converges to Gaussian with zero mean and variance q', where the variance follows recursion relation (Poole et al. 2016). Therefore, to achieve layerwise dynamical isometry, the idea becomes to find a fixed point q" such that h' ~ NN(0, q*) for all e {1...K) K). Such a fixed point makes D' = D for all layers, and therefore, the pre- activations are placed in the linear region of the nonlinearity Then, given the nonlinearity, one can su I. The procedure for finding the
find a rescaling such that (DW')T (DW') = (W')" /o/ rescaling of for various nonlinearities are discussed in Pennington al. 2017 2018). Also, this easily extends to convolutional neural networks using the initialization method in Xiao el al. (0188)
Dynamical isometry can hold for antisymmetric sigmoidal activation functions (e.g., tanh) shown in Pennington et al. (0077). A recent work by Tarnowski et (2019) have also shown that dynamical isometry achievable irrespective of the activation function in ResNets.
5

---

Published as a conference paper at ICLR 2020
Table 1: Jacobian singular values and resulting sparse networks for the 7-layer tanh MLP network considered in section 3.1 SG, CN, and Sparsity refer to Scaled Gaussian, Condition Number (ie., / where smax and mmin are the maximum and minimum Jacobian singular values), and a ratio of pruned prameters to the total number of parameters, respectively. SG (>=10- 3) is equivalent to the variance scaling initialization as in LeCun et al. (1998); Glorot & Bengio (2010). The failure cases correspond to unreliable connection sensitivity resulted from poorly conditioned initial Jacobians.
Jacobian singular values
Sparsity in pruned network (across layers)
Initialization
Mcan
Sid
CN
3
4
5
Error
SG (y=10 +)
2.46e-00
9.90e-08
4.66e+000
0.97
0.80
0.80
0.80
0.80
0.81
0.48
2.66
SG (y=10 9)
5.74e-01
2.45e-01
8.51e+00
0.97
0.80
0.80
0.80
0.80
0.81
0.48
2.67
SG (y=10-3)
4.49e-01
2.51e-01 01
5.14c+001
0.96
0.80
0.80
0.80
0.81
0.81
0.19
2.67
SG (G-1001))
2.30e+001
2.56e4001
2.92e+404
0.96
0.81
0.82
0.82
0.82
0.80
0.45
2.61
SG (y=109)
1.03e+00
2.61e403
3.31e+11
0.85
0.99
1.00
1.00
1.00
90.2
SG (1=10")
3.67e+01
2.64e+005
inf
0.84
0.95
1.00
1.00
1.00
1.00
1.00
90.2
We note that dynamical isometry in fact weaker condition than layerwise dynamical isometry. However, in practice, the initialization suggested in the existing works (Pennington et al. 2017: Xiao ct al. 2018), i.e., orthogonal initialization for weight matrices in each layer with rescaling based on mean field theory, satisfy layerwise dynamical isometry, even though this term was not mentioned.
Now, recall from Section 3.1 that a network is pruned with global threshold based on connection sensitivity, and from Section 3.2 that the connection sensitivity is the gradients scaled by the weights. This in turn implies that the connection sensitivity scores across layers are required to be of the same scale. To this end, require the gradients be faithful and the weights to be in the same scale for all the layers. Notice, this condition is trivially satisfied when the layerwise dynamical isometry is ensured, each layer is initialized identically (i.e., orthogonal initialization) and the gradients are guaranteed to be faithful.
Finally, we verify the failure of pruning cases presented in Section 3.J based on the signal prop- agation perspective. Specifically, we measure the singular value distribution of the input-output Jacobian (JO.K) for the 7-layer tanh MLP network, and the results are reported in Table [ Note that while connection sensitivity based pruning is robust to moderate changes in the Jacobian singular values, it failed catastrophically when the condition number of the Jacobian is very large (> 1e++11). In fact, these failure cases correspond to the completely disconnected networks, a consequence of pruning with unreliable connection sensitivity resulted from poorly conditioned initial Jacobians. As will show subsequently, these findings extend to modern architectures, and layerwise dynamical isometry yields well-conditioned Jacobians and in turn the best pruning results.
4 SIGNAL PROPAGATION IN SPARSE NEURAL NETWORKS
So far, we have shown empirically and theoretically that layerwise dynamical isometry can improve the process of pruning at initialization. One remaining question to address is the following: how well do signals propagate in the pruned sparse networks? In this section, first examine the effect of sparsity on signal propagation after pruning. We find that indeed pruning can break dynamical isometry, degrading trainability of sparse networks. Then we follow up to present simple, but effective data-free method to recover approximate dynamical isometry on sparse networks.
Step 1. Initialize a network with a variance scaling (VS) or layerwise dynamical isometry (LDII) satisfying orthogonal initialization. Step 2. Prune at initialization for sparsity level based connection sensitivity (CS); we also test random (Rand) and magnitude (Mag) based pruning for comparison. Step 3. (optional) Enforce approximate dynamical isometry, if specified. Step Train the pruned sparse network using SGD. We measure signal propagation (e.g., Jacobian singular values) on the sparse network right before Step 4, and observe training behavior during Step Different methods are named as [A}- (C)) where A, B, C stand for initialization scheme, pruning method, (optional) approximate isometry, respectively. We perform this on 7-ayyer linear and tanh MLP networks as before 5
Setup. The overall process is summarized as follows:
We conduct the same experiments for ReI.U and Leaky-Rel.U activation functions (see Appendix

---

Published as a conference paper at ICLR 2020
linear (Km7)
tanth (K=7)
linear (K=7)
tarhh (Km
3,40
0.35
0.8
0.6
9.20
0.20
Ur- 11 C5.
1.5
Lonce
0.4
singuua
2.13
3.0
2.10
2.00
0.5
202
320
Sparsity
Sparsity
Iteration
(a) Signal propagation
(b) Training behavior
Figure 2: (a) Signal propagation (mean Jacobian singular values) in sparse networks pruned for 90%. Signal
varying sparsity levels K, and (b) training behavior of the sparse network at propagation, pruning scheme, and overparameterization affect trainability of sparse neural networks. We train using SGD with the initial learning rate of 0.1 decayed by 1/10 at every 20k iterations. All results are the average over 10 runs. We provide other singular value statistics (max, min, std), accuracy plot, and extended training results for random and magnitude pruning in Appendix
Effeet of pruning on signal propagation and trainability. Let us first check signal propagation measurements in the pruned networks (see Figure In general, Jacobian singular values decrease continuously as the sparsity level increases (except for {-I-(---AI which we will explain later), indicating that the more parameters removed, the less faithful network is likely to be with regard to propagating signals. Also, notice that the singular values drop more rapidly with random pruning compared to connection sensitivity based pruning methods (i.e., [-)-Rand vs. (-J-CS). This means that pruning using connection sensitivity is more robust to destruction of dynamical isometry and preserve better signal propagation the sparse network than random pruning. We further note that, albeit marginal, layerwise dynamical isometry allows better signal propagation than variance scaling initialization, with relatively higher mean singular values and much lower standard deviations especially in the low sparsity regime (see Appendix C
Now, we look into the relation between signal propagation and trainability of the sparse networks. Sigureebjshows training behavior of the pruned networks (R = -90%) obtained by different methods. We can see a clear correlation between signal propagation capability of network and its training performance; ie., the better a network propagates signals, the faster it converges during training. For instance, compare the trainability of network before and after pruning, That is, compared to LDI-Dense (R 0), LDI- Mag. Rand) decrease the loss much slowly; random pruning starts to decrease the loss around 4k iteration, and finally reaches to close to zero loss around 10k iterations (see Appendix C which is more than order of magnitude slower than network pruned by connection sensitivity. Recall that the pruned networks have much smaller singular values.
Enforcing approximate dynamical isometry. The observation above indicates that the better signal propagation is ensured on sparse networks, the better their training performs. This motivates us to think of the following: what if we can repair the broken isometry, before we start training the pruned network. such that we can achieve trainability comparable to that of the dense network? Precisely, we consider the following:
min |I O (') @ (w) - W
(8)
where C¹, W I' are the sparse mask obtained by pruning, the corresponding weights, the identity matrix at layer I, respectively, and || || is the Frobenius norm. We optimize this for all layers identically using gradient descent. Given the sparsity topology C' and initial weights W', this data- free method attempts to find an optimal W such that the combination of the sparse topology and the weights to be layerwise orthogonal, potentially the full rank capacity. This simple method (i.e.. (-AI, where Al is named for Approximate Isometry) turns out to be highly effective. The results are provided in Figure 2 and we summarize our key findings below:
Signal propagation (LDI-ICS, Rand} vs. LDI-(CS, Rand)-AD. The decreased singular values (by pruning R > 0) bounce up dramatically and become close to the level before pruning. This means that orthogonality enforced by Equation 8 is achieved in the sparse topology of the pruned

---

Published as a conference paper at ICLR 2020
Table Pruning results for various neural networks on different datasets, All networks are pruned at initialization for the sparsity R 90% based on connection sensitivity scores as in Lee et al. (2019). We report orthogonality scores (OS) and generalization errors (Error) on CIFAR- (VGGI6, ResNets) and Tiny-ImageNe (WRN16); all results are the average over 5 runs. The first and second best results are highlighted in each column errors. The orthogonal initialization with enforced approximate isometry method (i.e., LDI-AI) achieves the best results across all tested architectures.
VGG16
ResNet32
ResNet56 Erorr
ResNet110
WRN16
Initialization
OS
Enor
Erorr
OS
Ervor
OS
Eror
VS-L
13.72
8.16
4.50
11.96
4.64
10.43
4.65
9.13
11.99
45.08
13.60
VS-II
15.44
8.36
4.41
12.21
4.44
10.63
4.39
9.08
13.49
46.62
13.33
8.11
4.43
LDI-AI
6.43
7.99
2.62
11.47
2.79
9.85
2.92
8.78
6.62
44.12
Table 3: Pruning results for VGG16 and ResNet32 with different activation functions CIFAR-10. We report generalization errors (avg. over 5 runs), and the first and second best results are highlighted.
Table 4: Unsupervised pruning results for K K-layer MLP networks on MNIST. All networks are pruned for sparsity at orthogonal initialization. We report gen- eralization errors (avg. over 10 runs).
90%
VGG16
ResNet33
Initialization
tanh
I-reu
selu
fanh
Irelu
selu
Loss
Superv.
K=3
K=$
K=7
VS-L
7.78
VS.G G
9.06
7.84
8.82
13.44
12.02
12.32
GT
2.16
2,43
2.61
VS-H
8.43
13.12
11.66
12.21
Pred. (mu)
3.31
3.38
3.60
LDI
8.76
7.53
8.21
13.22
11.58
11.98
Pred. (softmax)
X
3.11
3.37
3.56
LDIAI
8.72
7.47
8.20
13.14
11.51
11.68
Unif.
2.77
2.77
2.94
network (i.e., approximate dynamical isometry), and therefore, signal propagation on the sparse network is likely to behave similarly the dense network. As expected, the training performance increased significantly (e.g., compare LDI-CS with LDI-CS-AL AI for trainability). This works more dramatically for random pruning; i.e., even for randomly pruned sparse networks, training speed increases significantly, implying the benefit of ensuring signal propagation.
Structure (InDI-Rand-Al vs. LDI-CS-AI). Even if the approximate dynamical isometry is en- forced identically, the network pruned using connection sensitivity shows better trainability than the randomly pruned network. This potentially means that the sparse topology obtained by differ- ent pruning methods also matters, in addition to signal propagation characteristics.
Overparameterization (LDI-Dense vs. LDI-(CS, Rand)-AD). Even though the singular values restored to a level close before pruning with approximate isometry, the non-pruned dense network converges faster than pruned networks. We hypothesize that in addition to signal propa- gation, overparameterization helps in optimization taking less time find minimum.
While being simple and data free (thus fast), our signal propagation perspective not only can be used to improve trainability of sparse neural networks, but also to complement common explanation for decreased trainability of compressed networks which often attributed merely to a reduced capacity. Our results also extend to the case of convolutional neural network (see Figure & in Appendix
VALIDATION AND EXTENSIONS
In this section, we aim to demonstrate the efficacy of our signal propagation perspective on wide variety settings. We first evaluate the idea of employing layerwise dynamical isometry on various modern neural networks. In addition, we further study the role of supervision under the pruning at initialization regime, extending it to unsupervised pruning. Our results show that indeed, pruning can be approached from the signal propagation perspective at varying scale, bringing forth the notion of neural architecture sculpting. The experiment settings used to generate the presented results are detailed in Appendix B The code can be found here: http:///gthub..om//manoooooooo
8

---

Published as a conference paper at ICLR 2020
5.1 EVALUATION ON VARIOUS NEURAL NETWORKS AND DATASETS
Here, we verify that our signal propagation perspective for pruning neural networks at initialization indeed valid, by evaluating further on various modern neural networks and datasets. To this end, provide orthogonality scores (OS) and generalization errors of the sparse networks obtained by different methods and show that layerwise dynamical isometry with enforced approximate isometry results in the best performance; here, we define OS as 1 Σ: II(C' ④ " can be used to indicate how close are the weight matrices in each layer of the pruned network to being orthogonal. All results are the average of 5 runs, and we do not optimize anything specific for a particular case (see Appendix for experiment settings). The results are presented in Table 2
(C's & (') - which
The best pruning results are achieved when the approximate dynamical isometry is enforced on the pruned sparse network (i.e. LDI-AI); across all tested architectures. Also, the second best results are achieved with the orthogonal initialization that satisfies layerwise dynamical isometry (i.e., LDI). Looking closely, it is evident that there exists a high correlation between the orthogonality scores and the performance of pruned networks; Le., the network initialized to have the lowest orthogonal- ity scores achieves the best generalization errors after training. Note that the orthogonality scores being close to O, by definition, states how faithful a network will be with regard to letting signals propagate without being amplified or attenuated. Therefore, the fact that pruned network with the lowest orthogonality scores tends to yield good generalization errors further validates that our signal propagation perspective is indeed effective for pruning at initialization. Moreover, we test for other nonlinear activation functions (tanh, leaky-relu, selu), and found that the orthogonal initialization consistently outperforms variance scaling methods (see Table 3)
5.2 PRUNING WITHOUT SUPERVISION
So far, we have shown that pruning random networks can be approached from a signal propagation perspective by ensuring faithful connection sensitivity. Another factor that constitutes connection sensitivity is the loss term. At a glance, it is not obvious how informative the supervised loss measured on random network will be for connection sensitivity. In this section, we look into the effect of supervision, by simply replacing the loss computed using ground-trutte labels with different unsupervised surrogate losses as follows: replacing the target distribution using ground-trun labels with uniform distribution (Unif.), and using the averaged output prediction of the network (Pred.; softmax/raw). The results for MLP networks are in Table 4 Even though unsupervised pruning results are not as good as the supervised case, the results are still interesting, especially for the uniform case, in that there was no supervision given. We thus experiment further for the uniform case on other networks, and obtain the following results: 8.25, 11.69, 1 1.01, 8,82 errors (%) for VGG16, 16, ResNet32, ResNet56, ResNetl 10, respectively. Surprisingly, the results are often competitive to that of pruning with supervision (i.e., compare to LDI results in Table Notably, previous pruning algorithms assume the existence of supervision priori. Being the first demonstration, along with the signal propagation perspective, this unsupervised pruning strategy can be useful in scenarios where there are no labels or only weak supervision is available.
To demonstrate further, we also conducted transfer of sparsity experiments such as transferring a pruned network from one task to another (MNIST ++ Fashion-MNIST).. Table 5 shows that, while pruning sults may degrade if sparsity is transferred, or done without supervision, less impact is caused for unsupervised pruning when transferred to a different task (i.e., 0.52 to 0.14 on MNIST, and 1.11 to -0.78 on F- MNIST). This indicates that inductive bias exists in data, affecting transfer and unsupervised prun- ing, and potentially, that "universal" sparse topology might be obtainable if universal data distribu- tion is known (e.g., extremely large dataset in practice). This may help in situations where different tasks from unknown data distribution are to be performed (e.g., continual learning). We also tested two other unsupervised losses, but none performed as well as uniform loss (e.g., Jacobian norms || 5.03, ||J 12: 3.00 vs. Unif.: 2.94), implying that if pruning is to be unsupervised, the uniform loss would better be used, because other unsupervised losses depend on the input data (thus can suffer from inductive bias). Random pruning degrades significantly at high sparsity for all cases.
Table 5: Transfer of sparsity experiment results for LeNet. We prune for R 97% at orthogonal initial- ization, and report gen. errors (average over 10 runs).
Dataset
Error
Error rand
Category
prune
train&test
sup.
unsup.
(A)
Standard
MNIST
MNIST
2.42
2.94
+0.52
15.56
Transfer
F-NNIST
MNIST
2.66 + 2,80
+0.14
18.03
Standard
MNNITTT F-NNIST
11.90 - 13.01
+1.11
24.72
Transfer
MNIST F-MNIST
14.17 + 13.30
-0.78
24.89
9

---

Published as a conference paper at ICLR 2020
5.3 NEURAL ARCHITECTURE SCULPTING
We have shown that pruning at initialization, even when supervision is provided, can be effective based on the signal propagation perspective. This begs the question of whether pruning needs to be limited pre-shaped architectures or not. In other words, what if pruning is applied to an arbitrarily bulky network and is treated as sculpting an architecture? In order to find out, we conduct the following experiments:
take a popular pre-designed architecture (ResNet20 in |H et al.] (2016) as a base network, and consider a range of variants that are originally bigger than the base model, but pruned to have the same number of parameters as the base dense network. Specifically, we consider the following equivalents: (1) the same number of residual blocks, but with larger widths; (2) a reduced number of residual blocks with larger widths: (3) a larger residual block and the same width (see T Appendix B for details). The results are presented in Figure 3
Overall, the sparse equivalents record lower errors than the dense base model. Notice that some models are ex- tremely sparse (e.g., Equivalent pruned for 98.4%). While all networks have the same number of parameters, discovered sparse equivalents outperform the dense refer- ence network. This result is well aligned with recent find- ings in Kalchbrenner et al. (2018): large sparse networks can outperform their small dense counterpart, while en- joying increased computational and memory efficiency via dedicated implementation for sparsity in practice. Also, it seems that pruning wider networks tends to be more effective in producing better model than pruning deeper ones (e.g., Equivalent 1 vs. Equivalent 3). We further note that unlike existing prior works, the sparse networks are discovered by sculpting arbitrarily-desinne architecture, without pretraining nor supervision.
8.0
7.9
7.8
Reference
1.7
Equivalent1
Equivalent2
7.6
Equivalent3
20
40
60
80
100
Sparsity (%)
Figure 3: Neural architecture sculpting results on CIFAR-10 10. We report gener- alization errors (avg. over 5 runs). All networks have the same number of pa- rameters (269k) and trained identically.
DISCUSSION AND FUTURE WORK
In this work, we have approached the problem of pruning neural networks at initialization from a signal propagation perspective. Based on observing the effect of varying the initialization, we found that initial weights have critical impact on connection sensitivity measurements and hence pruning results. This led us to conduct theoretical analysis based on dynamical isometry and a mean field theory, and formally characterize sufficient condition to ensure faithful signal propagation in a given network. Moreover, our analysis on compressed neural networks revealed that signal propagation characteristics of sparse network highly correlates with its trainability, and also that pruning can break dynamical isometry ensured on a network at initialization, resulting in degradation of trainability of the compressed network. To address this, introduced a simple, yet effective data-free method to recover the orthogonality and enhance trainability of the compressed network. Finally, throughout range of validation and extension experiments, we verified that our signal propagation perspective is effective for understanding, improving, and extending the task of pruning at initialization across various settings. We believe that our results on the increased trainability of compressed networks can take us one step towards finding "winning lottery ticket" (i.e., a set of initial weights that given sparse topology can quickly reach a generalization performance that is comparable to the uncompressed network, once trained) suggested in Frankle & Carbin (2019).
We point out, however, that there remains several aspects to consider. While pruning on enforced isometry produces trainable sparse networks, the two-stage orthogonalization process (i.e., prune first and enforce the orthogonality later) can be suboptimal especially at high sparsity level. Also, network weights change during training, which can affect signal propagation characteristics, and therefore, dynamical isometry may not continue to hold over the course of training. We hypothesize that potential key to successful neural network compression is to address the complex interplay be- tween optimization and signal propagation, and it might be immensely beneficial an optimization naturally takes place in the space of isometry. We believe that our signal propagation perspective provides a means to formulate this as an optimization problem by maximizing the trainability of sparse networks while pruning, and we intend explore this direction as a future work.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGMENTS
This work was supported by the ERC grant ERC-2012-AGG 321162-HELIOS, EPSRC grant See- bibyte EP/M013774/1, EPSRC/MURI grant EP/N019474/1 and the Australian Research Council Centre of Excellence for Robotic Vision (project number CE140100016) We would also like to acknowledge the Royal Academy of Engineering and FiveAI, and thank Richard Hartley, Puneet Dokania and Amartya Sanyal for helpful discussions.
REFERENCES
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ICLR, 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. AISTATS, 2010.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. ICML, 2018.
Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. Neural Networks: Tricks of the Trade, 1998.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
Namhoon Lee. Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. 2013.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. NeurIPS, 2017.
Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. The emergence of spectral univer- sality in deep networks. AISTATS, 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. NeurIPS, 2016.
Russell Reed. Pruning algorithms-a survey. Neural Networks, 1993.
Andrew M Saxe, James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. ICLR, 2014.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. ICLR, 2017.
Wojeiech Tarnowski, Piotr Warchot, Stanislaw Jastrzębski, Jacek Tabor, and Maciej A Nowak. Dy- namical isometry is achieved in residual networks in a universal way for any activation function. AISTATS, 2019.
11

---

Published as a conference paper at ICLR 2020
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and a mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. ICML, 2018.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. NeurIPS, 2017.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. ICLR, 2019.
A GRADIENTS IN TERMS OF JACOBIANS
Proposition 1
Let aL/0x denote the error signal and x denote the input signal. Then,
1. the gradients satisfy:
ow = D' xx'11
0x* /öxx is the Jacobian from layer I to the output and @ is the Kronecker product. 2. additionally, for linear networks, i.e., when @ the identity:
where J..K =
Sw J (JON-2x² +a)
(10)
where J = 0x'-1/0x" is the Jacobian from the input to layer I - and a E RN is the constant term that does not depend on x".
Proof. The proof is based on simple algebraic manipulation of the chain rule. The gradient of the loss with respect to the weight matrix W can be written as:
ØL δL 0x Ox
g
OW OxK Ox OW!
(11)
Here, the gradient /0x represented as a matrix of dimension y-size x x-size, For gradients with respect to matrices, their vectorized from is used. Notice,
Ox
Ox Oh' D'
0h
(12)
ow Oh aw ow!
Considering the feed-forward dynamics for particular neuron i,
h
W W
+ b
(13)
0h 1-1
ow! x.
ij
Therefore, using the Kronecker product, we can compactly write:
Ox
(D')T & (xT
(14)
aw
Now, Equation 11 can be written
gw (eJ o (x'-1)T g J.. x-1
(15)
Here, A & TT = (A & B)Th
T is used. Moreover, for linear networks D' = and x!
h for all
I E [1. K}. Therefore, x- can be written as:
x o1 W wW'-o((W'2 w (21-2 (Wl b²) b' 22 + b 1-1
oi @(W 0
b²) +bb 22 + b bb-1)
(16)
1-1
w
II
w +b.-.b'-1.
A=1
k=2
where a is the constant term that does not depend on x Hence, the proof complete.
12

---

Published as a conference paper at ICLR 2020
B EXPERIMENT SETTINGS
Pruning at initialization. By default, we perform pruning at initialization based on connection sensitivity scores as in Lee et al. (2019). When computing connection sensitivity, we always use all examples in the training set to prevent stochasticity by a particular mini-batch. Unless stated otherwise, we set the default sparsity level to be 90% (i.e., 90% of the entire parameters in network is pruned away). For all tested architectures, pruning for such level of sparsity does not lead to large accuracy drop. Additionally, we perform either random pruming (at initialization) or a magnitude based pruning (at pretrained) for comparison purposes. Random pruning refers to pruning parameters randomly and globally for given sparsity level. For the magnitude based pruning. first train model and simply prune parameters globally in single-shot based on the magnitude of the pretrained parameters (i.e., keep the large weights while pruning small ones). follow either variance scaling initialization schemes (i.e., VS-L,, VS-
For initialization methods, G, VS-H, as in LeCun et al. (1998); Glorot & Bengio (2010); He et al. (2013), respectively) or (convolutional) orthogonal initialization schemes (Saxe al. 2014 Xiao et al.] 20188).
Training and evaluation. Throughout experiments, we evaluate pruning results on MNIST, CIFAR-10, and Tmay-emageNet image classification tasks. For training of the pruned sparse net- works, we use SGD with momentum and train up to 80k (for MNIST) or 100k (for CIFAR-10 and Tiny-ImaeeNet) iterations. The initial learning rate is set to be 0.1 and decayed by 1/10 every 20k (MNIST) or 25k (CIFAR-10 and Tiny-ImageNet), The mini-batch size is set to be 100, 128, 200 for MNIST, CIFAR- -10, Tiny-Imagen ImageNet,, respectively. We do not optimize anything specific for particular case, and follow the standard training procedure. For all experiments, we use 10% of train- ing set for the validation set, which corresponds to 5400, 5000, 9000 images for MNIST, CIFAR- 10, Natgee,mgeNet respectively. We evaluate at every 1K iteration, and record the lowest test error. All results are the average of either 10 (for MNIST) or (for CIFAR-10 and Tiny- Finy-ImageNet) runs.
Signal propagation and approximate dynamical isometry. We use the entire training set when computing Jacobian singular values network. order to enforce approximate dynamical isom- etry when specified, given a pruned sparse network, using gradient descent. The learning rate is set to be 0.1 and we perform 10k gradient update steps (although it usually reaches to convergence far before). This process is data-free and thus fast; e.g., depending on the size of the network and the number of update steps, it can take less than a few seconds on modern computer.
optimize for the objective in Equation
Neural architecture sculpting. We provide the model details in Table 6
Table 6: All models (Equivalents 1,2.3) are initially bigger than the base network (ResNet20), by either being wider or deeper, but pruned to have the same number of parameters as the base network (269k). The widening factor (k) refers to the filter multiplier; e.g., for the basic filter size of 16, the widening factor of k=2 will result 32 filters. The block size refers to the number of residual blocks in each block layer; all models have three block layers. More/less number of residual blocks means the network deeper/shallower. The reported generalization errors are averages over runs. We find that the technique of architecture sculpting, pruning randomly initialized neural networks based on our signal propagation perspective even the absence of ground-truth supervision, can be used find models of superior performance under the same parameter budget.
Model eategory
Shape
Widening (k)
Block size
Init.
GT
Sparsity
Error
Base
ResNer20 (e et 2016
3
VS.H
/
0.0
8.046
Equivalent
wider wider wider wider
2
3
LDI
X
74,8
7.618
93.7
6
3
I.DI
97.2
7.708
8
3
LDI
98.4
7.836
Equivalent
wider & shallower
2
2
LDI
60.4
7.776
wider & shallower
4
2
LDI
X
90.1
7,876
wider shallower
2
95.6
Equivalent
deeper
1
S
LDI
X
42.0
7,912
13

---

Published as a conference paper at ICLR 2020
C
SIGNAL PROPAGATION IN SPARSE NETWORKS; ADDITIONAL RESULTS
(K=7)
(K=7)
sinqular values (maan)
1.3
Singuir Volues [mox)]
1.50
Singuler valuees (min)
19
0.33
044
L5
7665
1706
1.25
15:45
1.00
0.4
6.45
0.20
LONC3
0:5
6.00
0.20
3.0
0.00
20
60
50
100
10
60
103
60
6
90
102
40
60
100
Sparsity (96)
5p aparsity (%)
Sparsity 1961
Sparsity (%)
tonh (K-7)
Tanh (K-7)
tanh (K-77
tanh (K-7)
singuler veluee (meean)
SInouu uttr (mhak)
6.20
00.14
2.2
Ring Einglaar valus
2.00
2.4
0.00
0.00
0..0
CD
USCSS
Sparsity
Sparsity 195)
Sparsity
Sparsity
(a) Signal propagation (all statistics)
(K=7)
(K=7)
(K=7)
2.0
Accuraty (%)
Lost
2.5
350
200
205
400
300
150
200
160
403
500
0.0
700
200
505
450
G0
140
200
303
400
500
Iteration
Iteratio
Iteration
Iteration
(b) Training behavior (loss and accuracy)
Figure 4: Full results for (a) signal propagation (all signular value statistics), and (b) training behav- ior (including accuracy) for 7-layer linear and tanh MLP networks. We provide results of LDI-Rand, LDI-Rand-AI, VS-CS, LDI-CS, LDI-CS-AL the linear case for both singular value statistics and training log. We also plot results of LDI-Mag and LDI-Dense on the tanh case for trainability; the training results of non-pruned (LDi-Dense) and magnitude (LD1-Mag) pruning are only reported for the tanh case. because the learning rate had to be lowered for the linear case (otherwise it explodes), which makes the comparison not entirely fair. We provide the singular value statisties for the mag- nitude pruning in Figure 6 to avoid clutter. Also, extended training logs for random and magnitude based pruning are provided separately in Figure to illustrate the difference in convergence speed.
14

---

Published as a conference paper at ICLR 2020
tanh (K=-7
tanh (K=7))
Accuracy US6
66
0.0
5200 12000
15003
20003
25003
30303
5030
:0000
25000
20000
25000 30030
Iteration
Iteration
(a) Training behavior
Figure 5: Extended training log (i.e., Loss and Accuracy) for random (Rand) and magnitude (Mag) pruning. The sparse networks obtained by random or magnitude pruning take a much longer time to train than that obtained by pruning based connection sensitivity. All methods are pruned at the layerwise orthogonal initialization, and trained the same way as before.
linear (K=7)
llnear (K=7)
linear (K=7)
lincar (K=7)
sirguler values (meean)
Singular velves (muak)
singuaa values (min)
-.0
1.
0.73
2.50
0.0
Sparsity (%)
Sparsity (%5
Spansity (%)
Sparsity (56
tant (K=7)
Lanh (K=7)
Lanh (Km7)
tanth (K=7)
J. singular values [meea))
3.0
(min)
2.0
2.5
6.2
3.0
singsler
0.0
30
00
100
20
80
85
200
20
40
80
10
100
20
40
97
100
Sparsity
Sparsity
Sparsity
Sparsity (36
Signal propagation (all statistics; magnitude based pruning)
Figure 6: Signal propagation measurments (all signular value statistics) for the magnitude based pruning (Mag) on the 7-layer linear and tanh MLP networks. As described in the experiment settings in Appendix B the magnitude based pruning is performed on pretrained model. Notice that unlike other cases where pruning done initialization (i.e., using either random or connection sensitivity based pruning methods), the singular value distribution changes abruptly when pruned (i.e., note of the sharp change singular values from to 10% sparsity). Also, the singular values are not concentrated (note of high standard deviations). which explains rather inferior trainability compared to other methods. We conjecture that naively pruning based on the magnitude of parameters a sophisticated tricks such layerwise
single-shot without pruning gradually employing thresholding, can lead to failure of training compressed networks.
retu (K=7)
relu (K=7)
Irelu (K=7)
Irelu (K=7)
Shgulrr ralues (moon))
2.8-
0.12
2.5
1.1
3.13
7.5
3,00
3.5
1.0
Inguler
3.1
10
204
13-3
LE-C2-4
Sparsity (%5)
Iteration
Sparsity (95)
Iteration
Figure 7: Signal propagation and training behavior for ReLU and Leaky-Rel.U activation functions.
They resemble those of the tanh case in Figure 2 and hence the conclusion holds about the same.
15

---

Published as a conference paper at ICLR 2020
VGG16
IDICS
2.0
LDI-CS-IS
1.5
Loss
0.0
5000
10000
15000
20000
25000
30000
35000
40000
Iteration
VGG16
100
Accuracy (%)
40
LDI-Densa
20
LDI-CS
LDI-CS.AIF
LDI-CS-AIS
5000
10000
15000
20000
25000
30000
35000
40000
Iteration
Figure 8: Training performance (loss and accuracy) by different methods for VGG16 CIFAR-IO. To examine the effect of initialization in isolation on the trainability of sparse neural networks, we remove batch normalization (BN) layers for this experiment, BN tends improve training speed as well as generalization performance. As result, enforcing approximate isometry (LDI-CS-AIF) improves the training speed quite dramatically compared the pruned network without isometry (LDI-CS). We also find that even compared to the non-pruned dense network (LDII-Desse which is ensured layerwise dynamical isometry, LDI-CS-AIF trains faster in the early training phase. This result is quite promising and more encouraging than the previous case of MLP (see Figures[land] as it potentially indicates that an underparameterized network (by connection sensitivity pruning) even outperform overparameterized network, at least in the early phase of neural network training. Furthermore, we add results of using the spectral norm in enforcing approximate isometry in Equation 8 (LDI-CS-AIS), and find that also trains faster than the case of broken isometry (LDI-CS), yet not as much as the case of using the Frobenius norm (LDI-CS-AIF).
16


Reviews:
[{"review_id": "SJlF8AaTtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors studied and formalized the effect of initialization to connection-sensitivity-based pruning. The authors first pointed out that a previously studied pruning criterion -- connection sensitivity (CS) -- is a normalized magnitude of gradients. Based on signal propagation theory, to achieve a 'faithful' (with minimal amplification) CS, the gradients must be also faithful. Then by using relation of Jacobians and gradient, the authors proved that orthogonally initial weights guarantees faithful on linear networks and certain distribution property on nonlinear network can achieve layerwise dynamic isometry, which is to ensure faithful signal propagation. Based on these findings, the authors proposed an initialization setup for improving pruning performance, with the goal to ensure dynamic isometry by orthogonal initialization and approximation.\n\nIn summary, this paper used a typical pruning criterion (CS) and used signal propagation theory to studied how initialization effect CS. Although this linking is quite limited, this paper has marched a step in quantitative and theoretical study of initialization in pruning: how pruning affects network performance and why it is. \n\nHere is a question:\nIn Fig.2, nonlinear activation is only shown with tanh. How is training performance when relu is used since relu is more commonly used in modern deep architectures.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJlRgxh3KS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes how signals propagate through randomly initialized neural networks that have undergone a kind of pruning/sparsification. The pruning method utilizes a metric called 'connection sensitivity', which has been used in prior work and which measures the infinitessimal impact of turning off specific parameters. The distribution of singular values in the layer-to-layer Jacobian matrices for pruned networks becomes increasingly pathological as the depth increases. This observation motivates the concept of 'layerwise dynamical isometry' (LDI), a slight generalization of the concept of 'dynamical isometry' that has been studied in prior work. Several methods for approximately obtaining differing amounts of LDI are investigated in a series of in-depth experiments that show a strong correlation between increased signal propagation and improved trainability of sparse networks.\n\nAlthough there have been numerous works studying dynamical isometry as a principle for initializing very deep networks, and many other works studying pruning methods after (and recently before) training, as far as I'm aware there has been no prior work that examines the intersection of these two directions. As such, I found the contributions of this paper to be novel and believe the results will be of interest to practitioners and theorists alike.\n\nAn important contribution of this paper is in identifying that a main difficulty in pruning networks at initialization comes from degradation of signal propagation, leading to poor or impossible training. The numerous well-thought-out experiments provide compelling evidence that the trainability of pruned networks is highly correlated with spectral measures of the networks' Jacobians. \n\nThe authors take this observation a step further by introducing a method for correcting the poor conditioning that can result from pruning. They show that enforcing Approximate Isometry on weights of the pruned connectivity pattern enables the pruned models to train much faster and often achieve better performance.\n\nFinally, the authors look at two natural extensions of their analysis  to designing new high-performing architectures to situations where labels are not present. Overall, I found this paper to have a detailed and thorough experimental analysis and to present nice new perspectives on pruning and signal propagation.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkltFEKDKS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a signal propagation perspective for single-shot network pruning. Particularly, by achieving layerwise dynamical isometry with proper initialization scheme, the paper ensure the reliability of connection sensitivity pruning criteria, thereby improving the pruning results.\n\nI’m inclined to reject this paper because (1) methods of dynamical isometry typically care about trainability of very deep network (e.g., 10,000 layers) and it has little to do with generalization performance which is what we really care about in network pruning; (2) experiments on signal propagation were done in small datasets and fully-connected layers without batch norm, which is not the standard setting we conduct network pruning. It’s unclear whether the analysis would carry over to CNNs with batch norm.\n\nMain argument:\nThis paper attacks single-shot pruning problem from the perspective of signal propagation. By using special initialization scheme, the authors attempt to calibrate the connection sensitivity measurements across layers. The main argument in the paper of dynamical isometry doesn’t has a direct connection with performance of a pruned network since it basically concerns the trainability of very deep neural networks.\n\nMoreover, adopting orthogonal initialization satisfying layerwise dynamical isometry only leads to marginal improvements over standard variance scaling initialization, as shown in Table 2 and 3.\n\nIn terms of potential improvements, I would suggest the authors to first carefully study the relationship between trainability and generalization and also analyze signal propagation of networks with batch norm. Finally, for such an empirical paper, I expect more large-scale experiments conducted.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a strong submission, and I recommend acceptance. The idea is an elegant one: sparsify a network at initialization using a distribution that achieves approximate orthogonality of the Jacobian for each layer. This is well motivated by dynamical isometry theory, and should imply good performance of the pruned network to the extent that the training dynamics are explainable in terms of a linearization around the initial weights. The paper is very well written, and all design decisions are clearly motivated. The experiments are careful, and cleanly demonstrate the effectiveness of the technique. The one shortcoming is that the experiments don't use state-of-the-art modern architectures, even thought that ought to have been easy to try. The architectures differ in ways that could impact the results, so it's not clear to what extent the same principles describe SOTA neural nets. Still, this is overall a very strong submission, and will be of interest to a lot of researchers at the conference.


Decision: Accept (Spotlight)