DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS
Authors: Ilan Price
Nicholas Daultry Ball
Adam Christopher Jones
Samuel Chun Hei Lam
Jared Tanner
Venue: ICLR 2024 poster (2024)
Keywords: Deep neural network
random initialisation
sparsity
gaussian process
Primary area: general machine learning (i.e., none of the above)
TL;DR: Sparsity inducing nonlinear activations are designed, analysed to be stable for training, and shown to be effective with hidden layer sparsity as high as 85%.

Abstract: Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly important as network sizes continue to grow and their application becomes more widespread.  Here we use the large width Gaussian process limit to analyze the behaviour, at random initialization, of nonlinear activations that induce sparsity in the hidden outputs.  A previously unreported form of training instability is proven for arguably two of the most natural candidates for hidden layer sparsification; those being a shifted ReLU ($\phi(x)=\max(0, x-\tau)$ for $\tau\ge 0$) and soft thresholding ($\phi(x)=0$ for $|x|\le\tau$ and $x-\text{sign}(x)\tau$ for $|x|>\tau$).  We show that this instability is overcome by clipping the nonlinear activation magnitude, at a level prescribed by the shape of the associated Gaussian process variance map. Numerical experiments verify the theory and show that the proposed magnitude clipped sparsifying activations can be trained with training and test fractional sparsity as high as 85\% while retaining close to full accuracy.

Published as a conference paper at ICLR 2024
DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS
Ilan Price"'t Nicholas Daultry Ball", Samuel C.H. Lam", Adam C. Jones" & Jared Tanner"
(*) Mathematical Institute, University of Oxford
(3) The Alan Turing Institute
filan.pris price, ni micholas: dau daullarbaalllll samuel lam, adam. c. jones, tanner) @maths cx. ac.uk
ABSTRACT
Inducing and leveraging sparse activations during training and inference is a promising avenue for improving the computational efficiency of deep networks, which is increasingly important as network sizes continue to grow and their appli- cation becomes more widespread. Here we use the large width Gaussian process limit to analyze the behaviour, at random initialization, of nonlinear activations that induce sparsity in the hidden outputs. A previously unreported form of train- ing instability is proven for arguably two of the most natural candidates for hidden layer sparsification; those being a shifted ReLU (Ø(x) max (0, x x-7) for T 2 0) and soft thresholding (Ø(z) = O for |a|| I 7 and x sign(x)r r for |z] > r). We show that this instability overcome by clipping the nonlinear activation magni- tude, at a level prescribed by the shape of the associated Gaussian process variance map. Numerical experiments verify the theory and show that the proposed magni- tude clipped sparsifying activations can be trained with training and test fractional sparsity as high as 85% while retaining close to full accuracy.
INTRODUCTION
Improving the computational efficiency of large deep neural networks is the subject of many lines of work, including network weight pruning and quantization (Blalock al. 2020), adapting network architectures (Bizopoulos & Koutsouris (02T), and inducing low- -iw-dimensional structure in hidden layer outputs through low-rank, sparse, or similar structures (see Sec. 1.2 Two of the key quantities that determine the computational efficiency of the forward pass of deep network are the total number of floating point operations (FLOPs) involved and the total amount of memory required. These quantities primarily influenced by the tensor products of layer weights and layer inputs. The existence of efficient techniques for storing and computing with sparse vectors and matrices makes the sparsification of DNN weights and layer inputs/outputs promising strategy for making the forward passes of deep networks more efficient. In contrast to the extensive literature on weight pruning and quantization, this paper investigates the training of networks with nonlinear activation functions that induce highly sparse hidden layer outputs throughout both training and inference. Sparse hidden layers are especially appealing they naturally combine with the other methods by subselecting only portion of the weight matrix active for that input.
To induce a target sparsity level we will leverage the fact that deep networks randomly initialized with i.i.d. entries generate hidden layers whose entries approach Gaussian distributions with known variance as the network width increases. For survey of the Gaussian process behaviour of deep networks see Roberts et al. (2022) and references therein, and for initial derivations of feedforward networks (Lee al. 2018), CNNs (Xiao et al. 2018 LSTMs and GRUs (Gilboa et al. 2019), RNNs (Chen et 2018), ResNets (Yang & Schoenholz 2017) and extra features like dropout (Schoenholz ct al. [017 Huang et al.) 2020) or batch normalization (Yang et al., 2019) and pruning (Hayou et al. 2021) ). This manuscript considers random initializations of both feedforward networks, abridged
here as DNNs, of width Ni,
N;
W! +
+1,,
==0(h)
for l 1,2,...,4
(1)
i=1

---

Published as a conference paper at ICLR 2024
and one-dimensional CNN: with imput channel length C₁2, and kernel width 2k 1,
Ch Σ (3) 1)((a= = Σ =-
+))
x ==)(h)) for = 1,2, ...,, Li (2)
specifically, with focus on activation functions that induce preseribed substantial fractional sparsity at initialization, which in practice is maintained throughout training and inference on un- seen data. To the best of our knowledge, highly sparsifying activations have not been studied initialization, which may well be due to - as we will show - the failure of DNNs and CNNs train with the intuitive choices of sparsifying activation functions. We analytically identify the source of this failure, which is due to an instability in the variance map of successive layers' Gaus- sian processes. That is, for arguably the most natural sparsifying activations, the Edge-of-Chaos (EoC) initialization Poole et al. (2016) needed to train very deep networks happens to coincide with introducing an instability that typically results exponential growth of the Gaussian process vari- ance with depth. Here, we introduce modified activation functions, using simple magnitude clipping, which decouples EoC initialization from the Gaussian process variance, and demonstrate that this allows training of very deep networks with activation sparsity up to 85%.
1.1
MAIN RESULTS
The Gaussian process model of DNNs (I in the large width limit and CNNs 2 in the large channel limit are characterized by their associated variance. For DNNs of width N; with Gaussian weights Wi, drawn i.i.d. (N0, /Ni) and bias by drawn i.i.d. .N(0, of) it was shown by Lee et al. (2018) that the antiviiivtiion outputs ht approach. ((O, q') where q' can be computed through the 'variance map' or 'length map'
= Vola = o ([(VV==))) (dz)) + oo.
(3)
where (dz) the standard Gaussian measure e e- 12 1122//2m 2T d2, and de | . q'= o. + of. Note that the variance map V-(-) depends on Φ, wo and oz. Typically Vo has single nonzero fixed point, which is essential for the resulting theory?
The Gaussian process model for CNNs (2) in the large channel limit is somewhat more complex due to correlation within h² (a) and h when la - -a' < 2k + 1. Xiao et al. (2018) proved that this Gaussian process characterized by the matrix
Eg = of + 2k
Σ S=-k
E (6th) (a+ B))((1) (a' + B)))
(4)
where expectation is taken over the weights and biases. The full hidden layer vector h then ap- proaches the Gaussian distribution N A x d'aar) where A = 2 /2x+1 and denotes a two- dimensional circulant cross-correlation that accounts for the 2k + overlapping locations. The covariance matrix for the CNN Gaussian process follows similar variance map as (
Il Eg (Da.a' + -a,aa'
(5)
where /111 /--1/
" is the correlation coefficient the inputs at layer - 1 (Poole al. 2016). Assuming that both o' and df converge quickly to q" yields the following iterative correlation map
(o (f p¹ = R 1 9"
+ oî
(6)
HHigher dimensional CNNs follow similarly but with more complex notation, see Lee al. 2018
"There are, however, some notable counter-examples, with ReLU having VRau q + of and conse- quently when 0, all g are fixed points, while when 0 the variance g' increases by of. Conversely ELU with as has VELV(g) < q for all with q" the only fixed point, Murray al. 2022
2

---

Published as a conference paper at ICLR 2024
where u1 =Vq* 21 and u2 =v between two distinct imputs (in the DNN case) or two entries at different locations with overlapping kernels (in the CNN case) evolves from one layer to the next, after both starting with or converging to variance q²
22). which describes how the correlation
Equipped with accurate model of the hidden layer entries it is straightforward to compute the fractional sparsity that a nonlinear activation will induce, and moreover to design activation func- tions o(-) which induce a prescribed sparsity level in the activations. Perhaps the most obvious candidate for such sparsifying activation functions is a shifted ReLU, denoted here as ReLU,, and defined as
x 00
if >
ReLU,(z)
otherwise,
where т > 0 allows greater sparsity than the standard ReLU when r == oF natural sparsifying activation is the Soflohreshold function ST, defined as
The other especially
ST f sign(æ)T, if |z] >
10,
(8)
otherwise,
which is an odd-function variant of the ReLU,; see Figure 1 The appeal of ST, to induce sparsity that it is an optimal denoiser (Donoho 1995) for fixed values with additive Gaussian noise and is
widely used in the compressed sensing community (Foucart & Raubut 2013) to encourage sparsity.
Rell,, T-1
ST.,
CReLU) ou T-
CST, m=1,m=1
Figure 1: From left to right: ReLU (i.e. ReLU, with 7 = 0), ReLU, (r = 1), ST, (T = 1), CReLU,, (r 1, 1), CST,, (T m 1)
Unfortunately, and (8) both suffer from a previously unreported instability of their hidden layer variance map which impedes training DNNs and CNNs with these activation functions. Specif- ically, both and (8) have the property that when initialized on the FoC, their associated (g) has single fixed point q² with Vo(g") = 1, and moreover (4" 0, causing q" to be only stable from the left, see Sec. Ac and Due to the natural stochasticity of q' for finite dimensional networks. even for moderate depths, q' typically obtains value in excess of q" and then diverges exponentially. This phenomenon is unusual as most common activation functions have a unique stable fixed point with (g") 1.
is possible to regain the stability of the fixed point of the Gaussian process variance map by suitable modification of the -parsity -nnducing activation functions Φ. This can be achieved by clipping the magnitude of these activations, (7) and (8); specifically, let
0, if <T
CReLU
if < < if
(9)
m,
and
0, sign(x)r, sign(z)m.
if |z| < if < |x| <
CST
(10)
| T +
as plotted in Figure] I This simple modification is sufficient to guarantee Vi((g)) 1 for bounded. Proof-of-concept experiments in Sec. | verify the efficacy of clipping to retain stability and show a trade-off with m between the stability of training and the expressivity of the network. Moreover, these experiments show that the for DNNs full test accuracy of a standard ReLU network baseline can be retained, or even slightly improved, with activation sparsity as high as 85%. For CNNs similar phenomenon is observed with full accuracy of ReLU baseline being retained for 70% sparsity, while 85% sparsity incurs just a 4% accuracy drop.
This continuous variant of the Foreed Activation Threshold ReLU considered in Kurtz al. 2020
3

---

Published as a conference paper at ICLR 2024
1.2 RELATED WORK
As noted and referenced above, there is a large body of work analysing the infinite-width limit for stable initialization of very deep networks. Complementary to the sparse hidden layer results presented here is the extensive literature on weight pruning (sparsifying the weights), both after a network has been trained, as type of network 'post-pun proocesnnss (see e.g. HHan et al. (2015)) D. and at initialization, such that the weights are sparse during training (Lee et al. [019] Tanaka et al., 2020 Hayou et al., 2020). See Hoefler et al. (202T) for a review of the work on weight spar- sification/pruning. In contrast, sparsification of the layer outputs-often referred to as the layer "activations', or 'feature maps' the case of CNNs-has received relatively little attention. Those few works which do focus on sparsity in the activations tend to propose methods to increase the sparsity of an existing, pre-trained network via fine-tuning. Georgiadis (2019) proposes inducing sparse activations (feature maps) in trrinned CNNs by fine-tuning them with added L1 regu- larisation, penalising the L1 norms of all feature maps. Kurtz et al. (2020) instead fine-tune ReLU networks with Hoyer regularisation, and introduce the FATRRl.U (defined 0 if < T else x) for some threshold r to increase the hidden layer sparsity.
1.3 ORGANISATION OF THE PAPER
Section 2 analyses natural candidates for sparsifying activation functions and 8 to show why very deep networks using these activation functions will almost always fail to train for t > 0. Section 3 analyses the magnitde--lippedd variants (9) and (IO), and shows analytically that this circumvents the most important of the aforementioned inability of the networks to train. Section 4 validates the theory with proof-of-concept numerical experiments training DNNs and CNNs using the proposed activation functions. Finally, Section suggests a number of lines of future work.
2 THE INSTABILITY OF ReLU, AND ST, AS SPARSIFYING ACTIVATION FUNCTIONS
The instability of ReLU, and ST, for > 0 is a consequence of the shape of their associated variance maps V/ (g). (3) when initialized on the FoC. EoC initialization requires that the slope of the correlation map R at =1
X1. : R²,11) ( (d'(V==)))><(a)))
(11)
be equal to 1. X1, determines the stability of the network to small perturbations of the inputs. When X1.e < 1 the network maps perturbations together an exponential rate (this is referred as the ordered regime D. while when X > 1 the perturbations diverge at an exponential rate (the chaotic regime). Having X1. but R())) > p for near I results in a stable convergence of correlations sub-exponential rate, and also helps to prevent exploding vanishing gradients; for details see Yang & Schoenholz (2017) and App. A As noted in Section I.] preserving X1. = 1 depends on q' converging to q², and thus also requires that the variance map (g(g) be sufficiently stable around q² Unfortunately it is impossible for ReLU,, or ST, to meet both of these requirements.
The variance maps Vsl (g) for o ReLU, and ST, are computed in App. B tabulated in Table 1
and plotted in Figure with (oww, σo) on the EoC for different values of T. The key observation is
that for both ReLU, and ST,, the slope V2((g) at the fixed point Vo (g ) = q" is equal to X1. and thus necessarily have that V²((") = 1 on the EoC.
When r = ReLU, is just the standard ReLU function, and ST, collapses to the identity. In both cases, when (aw. ob) lie on the EoC, Vieeu. (q) for all q, making any q a fixed point of VReLU.; see the left-most plot in Figure
For > however, the second derivative V (4*) is strictly positive, causing networks with ReLU, and ST, with to have unstable Gaussian process variance propagation and consequently prove effectively impossible to train for large sparsity. We verify this claim experimentally see App. C for the experimental details and Table 2 for the results.
"The ordered regime X1.6 is stable to perturbations, but for practical training is excessively so, with exponential convergence of all inputs a single point.
4

---

Published as a conference paper at ICLR 2024
(o(g)
X1.0 at 9 =
q
(0(0)
fs 9*)
ReLU,
o ②
(-v vi
o
Vqr (s)
V4*
v7
2v2xg
-1
ST,
20 10
20 a!
2Φ
2y/2m erg (s)
Table 1: Variance and correlation map derivatives for ReLU, and ST, where 2(.) is the CDF of standard normal distribution. The correspondence of V and X1. means that EoC initialization implies that Vi(ag) = for all > Moreover, the second derivative of the variance map V is strictly positive for every 9, resulting in instability of the fixed point Volg*) = q². The simplified expression of d in the final column occurs when X1,
T-
T 0.25
-.5
1.000
2.0
RelU;:
ReLU;
2.0
ReLU;
2.0
ST:: Xt =1
STi:2:
ST 25 =1
ST.: X: =1
1.5
1.5
1.5
g
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0.0
0.0
0.0
0
1
0
0
Figure 2: Variance maps for ReLU, and ST, with (ow, σb) on the EoC, for different values of т.
Here q" = 1 is used to compute ReLU, The dashed line the identity map. 'he curves for ReLU, and ST, overlap exactly for given Note, however, that fixed value of corresponds to substantially different activation sparsities for ReLU, and ST,. Figure 9 in App. C compares V (g) for fixed output sparsities.
In practice, choose t control the 'expected sparsity' (proportion of zeros) after applying different activation functions. Assuming the activation function is applied to a random Gaussian vector with independent entries, each with variance q², we have
P(o(Z) 0), Z ~(0,00)))
(12)
The optimal (smallest) r for DNN with activation function Φ = ReLU, 1 or ST, and activation spar- sity s, denoted as (s, q²), ) is computed in App. | and tabulated Table Substituting (s, into the expression for X1.6 in Table I results the value of og on the EoC. which turns out to be dependent of q" for both ReLU,, and ST,, and which thus makes achieving approximate dynamical isometry (Pennington et al. 2018) with these networks impossible, see App
GAINING STABILITY AND TRAINABILITY VIA CLIPPING
Clipping the maximum output magnitude of ReLU, and ST, allows the EoC condition X1.6 1 to be satisfied while ensuring (q") < 1. The clipped activation functions are denoted as C 7,m and CST,, ,, in (g) and (10) respectively. This allows CReLU, m. and CST T generate hidden layer outputs which are both very sparse and trainable when initialized at the EoC.
The associated variance and correlation maps CReLU and CST are computed and plotted for variety of sparsity factors and clipping levels in App. F. The stability of the fixed point y" when X1. = 1 follows from the expressions for the gradient of their variance map Vg(g) and the slope X1.a:
V'RELL. (g) = 022 ( m
m + - erf
e
+ 2
erf
(13)
2
/2q
/29
VTST..m (q)
2V (q),
(14)
02
m + 7
X1,CRoLU....
erf
erf
(15)
22
229"
5

---

Published as a conference paper at ICLR 2024
X1,CST..m 2X1,CRoLU..
(16)
Crucially,
o exp
(m + T)²
X1,CReLU..m VCRLLU (q")
VC (4"),
(17)
2T
2q*
V2 120 u.
(m + T)2 VCST (1*).
XI,CST,.m = VTTTT. (1*)
exp
(18)
Vxq""
2q
Thus, Vi(a") < at the EoC (Xs 1), making q' locally stable for both & CReLU r.m. or CST The local convergence rate of q' the fixed point q" is determined by Vg(g"). Note, however, that certain values of and induce multiple fixed points, causing q" stable; see Figure4bber as well as the bottom right panels of Figures [0 T in App Another effect of bounding these activation functions that oh the EoC (X1. 1) is no longer independent of q" for given expected sparsity 8, although this unfortunately still does not enable us to achieve approximate dynamical isometry for these activation functions, for details see App.
be globally
1.0
Viln 13
0.6
65 1.000
0.57
0.3
0.6. 0.5
07.
4,9
0.0
0.
085
(mm, 31
05, 125
6., 1.54
9.0
-25
Figure 3: Plots (m) (upper) and (((m) (lower) and at q² = for CReLU (left) and CST,,m (right) with for different fractional sparsities. For (m(m) the horizontal dashed black lines are plotted at 0.5, 0.6, 0.7, 0.8, 0.9.
3.1 CHOOSING THE CLIPPING MAGNITUDE mn FOR CReLU,,, AND CST 7,m
A larger clipping hyperparameter m increases the expressivity of the network by allowing larger activation values and a wider trainable (non-zero gradient) region of its domain. also plays a role in determining the shape of the variance map- cruccrucillly, Vy(4") ) (see Equation (13), as well as the curvature V (g"). Figure 3 show Vy(a") and (q"") as functions of m for different sparsity levels.
As increases, Vy(a") increases and tends from below
Co.
While Vi((") 1 for each m, the curvature (g") initially decreases to negative values and then increases as m increases from 0. Stability for finiteddimensional networks requires Va(g) to be sufficiently stable around q so that the natural stochasticity in remains in the local stable region of Vo(g), meaning that g' does in practice remain approximately equal to q" In particular, for larger values of the curvature (²"(a") can cause Vg(g) curve up after crossing the Vg(g) line and approximately follow it for extended interval before flattening again- resulting in

---

Published as a conference paper at ICLR 2024
4.0
1.74
1.40
061
1.31, 0.4,
0.89
35
2.0
1.30
1.25
1.8
1.5
15
1:35
14
1.0
1.30
05
1.00
1.0
(a) CReLU r.m. 0.85, m chosen such that Vi(44) = 0.9 (orange), with 4"
(b) CST r.m. 0.85, m chosen such that V/24 0.8 (orange) 0.9 (green), with q"
Figure 4: Variance maps for CReLUr,m in (a) and CST r,m. in (b) at 85% sparsity and m chosen such that even though V²((") < the network is unstable to train. For CReLU training fails due to (o(g) being approximately q for a sufficiently large region around q" that the effective value of X1 varies above which results in exploding gradients. For CST training fails for the same reason of effective X1 > 1, which occurs even more dramatically as V²(g) exceeds & and has a second, unstable, fixed point as well another stable fixed point, but with XI substantially greater than 1.
an extended region in which (V((g) is small. Increasing m (and thus V V''((g") )) even further causes a bifurcation, with two new fixed points emerging: a locally unstable one close to the original q¹ and another locally stable one larger value of q. These new fixed points cause a failure of o' to converge to q" in practice instead, q' converges to or hovers at value greater than g", corresponding to X1. > 1, which results in exploding gradients. See Figure (4b) for illustrations of these phenomena which correspond to examples in Sec. where networks fail to train. For this reason, the values of m that have the best values of accuracy and ability to train are as large as possible while (g)) and (g") remain sufficiently small ensure practical stability of q" based on stochasticity of q² in finite-width networks; see Table 2 for examples of parameters and associated test accuracy for CReLU,,, and ST, respectively.
EXPERIMENTS
To confirm the theoretical predictions, we train both feedforward networks (abridged as DNNs) of width 300 and depth 100 using CReLU,,m and CST to classify digits from the MNIST dataset and, similarly, CNNs with 300 channels in each layer and depth 50 are trained to classify images from the CIFARIO dataset. Training such deep DNNs and CNNs allows which aims to improve information propagation at initialization through many layers. The absolute accuracy of the networks is not the focus of these experiments, rather it is the ability retain trainability and approximately the accuracy of standard ReLU networks for sparsities greater than 50%. The networks are initialized the EoC using 9 = 1, before being trained by stochastic gradient descent (SGD) for 200 epochs with learning rate of 10-4 and 10-3 for the DNN and CNN respectively, Each experiment is conducted with sparsity 8 = 0.6, 0.7, 0.8, and 0.85 and m selected to have (g)) = 0.5, 0.7, and 0.9. The results are shown in Table 2 These results should be contrasted with those for ReLU, and ST also shown in Table and described in App.
verify the theory,
Whereas EoC-initialized ReLU, and ST, DNNs failed to train consistently even when initialized at sparsity levels 60% and 50% respectively, the FoC-initialized CReLU-.m and CST 7.m DNNs can maintain full accuracy when initialized with up to 85% sparsity. CReLU,, and CST, CNNs retain full accuracy for sparsity high 70% while 85% sparsity incurs modest loss of accuracy to 66%. Conversely, their non-clipped counterparts fail to train at all at 60% and 70% sparsity respectively. Moreover, in those cases where training is stable and high accuracy is achieved, the starting sparsity level is maintained throughout training, resulting in trained models which exhibit approximately the same high activation sparsity levels when measured the test set.
7

---

Published as a conference paper at ICLR 2024
DNN on MNIST
CNN CIFARIO
Test accuracy std
Test sparsity
Test accuracy
Test sparsity
std
7
m
(g"(
ReLU,
0.50
0.00
N/A
1.0
0.0
0.94
0.002
0.50
0.001
0.70
0.52
0.60
0.25
N/A
1.0
0.12
0.76
0.37
0.49
0.27
0.68
0.6
N/A
ST.
0.5
0.67
N/A
1.0
0.43
0.10
0.00
0.00
0.00
0.1
0.6
0.84
N/A
1.0
0.59
0,10
0,00
0.00
0.00
0.1
0.0
0.7
1.04
N/A
1.0
0.81
0.10
0.00
0.00
0.00
0.1
0.0
CRoLU>..
0.60
0.25
1.22
0.5
-0.44
0.92
0.004
0.60
0.001
0.70
0.61
1.63
0.7
-0.42
0.92
0.003
0.004
0.69
2.25
0.9
-0.19
0.92
0.01
0.60
0.01
0.69
0.60
0.70
0.52
1,05
0.5
40.37
0.93
0.01
0.70
0.002
0.70
0.70
1.45
0.7
-0.31
0.92
0.003
0.70
0.002
0.69
0.69
2.05
0.9
0.84
0.94
1,27
0.7
-0.12
0.93
0.01
0.80
0.01
0.64
0.78
1.85
0.9
0.21
0.94
0.01
0.79
0.02
0.65
0.78
0.85
1.04
0.81
0.5
-0.14
0.78
0.16
0.85
0.004
0.65
0.85
1.17
0.7
0.02
0.65
1.74
0.9
0.41
0.28
0.37
0.79
0.04
0.66
0.83
CST 7,m
0.50
0.67
0,97
0.5
-0.32
0,92
0.004
0.51
0.01
0.69
0.49
1.36
0.7
-0.23
0.92
0.01
0.50
0.004
0.69
0.48
1.96
0.9
0.08
0.92
0.004
0.49
0.01
0.71
0.49
0.84
1,27
0.7
-0.12
0.92
0,004
0.60
0.003
0.67
0.59
1.85
0.9
0.21
0.93
0.004
0.70
1.04
0.81
0.5
.0.14
0.94
0.005
0.70
0.004
0.68
0.69
1.17
0.7
0.02
0.93
0.005
0.71
0.01
0.68
0.67
1.74
0.9
0.41
0.27
0.36
0.41
0.18
0.68
0.68
D.8U
1.28
0.72
0.5
0.00
0.92
0.04
0.80
0.001
U.66
0.79
1.06
0.7
0.23
0.95
0.80
0.01
0.65
0.78
1.61
0.9
0.69
0.11
0.01
0.15
0.01
0.31
0.18
0.85
1.44
0.67
0.5
0.11
0.76
0.16
0.85
0.005
0.63
0.84
1.00
0.7
0.39
0.93
0.01
0.85
0.01
0.63
0.83
1.53
0.9
0.89
0.14
0.04
0.01
0.29
Table Experimental results for ReLU, and ST, with sparsity s up to 0.7, and for CReLU and CST r,m. with sparsity up to 0.85 and m selected to have V²(g") 0.5, 0.7, and 0.9. DNNs have 100 layers and are tested on MNIST with 5 runs and the mean and standard deviation are reported for both test accuracy and average activation sparsity calculated on the test set. For the 50-layer CNN experiments on CIFAR10 single experiment was conducted for each hyperparameter combination.
Upon analysing the 'failure' cases for DNN in which low accuracy is achieved, two failure modes are identified. The first of these corresponds to the analysis and predictions made Section 3.1 For CReLU,,.. 7,m and 0.6, 0.7, and 0.8, all values of Vy(a"): this corresponds to variance map initialization with sufficient symmetry around 9º to ensure that of stays, in practice, very close to q" on average, thus avoiding any exploding gradients, and preserving trainability. Once increases to 0.85, however,
negative small positive values of V (g") for
start to observe the -when m is set such that (g((") = 0.9, then V (²") = 0.41, a high chance of failure in training is observed for the reasons described in Section 3.1 and highlighted by the variance map in Figure 4 Indeed, training fails in four out of five experiments, reflected by the low mean test accuracy and high standard deviation. Figure[5a] plots the gradient norms at each layer for the first 15 steps of training for one of these failed runs. Even at the very first step, the gradients grow as the backwards pass proceeds (up to O(10))) and this reaches O(109) by step +
potential for m to be too
The same trend is observed for DNNs and 0.8 with e tested value of m in these cases, increasing m initially increases test accuracy, but at a certain point,
CST 7m Starting from the lowest
'The reason for the drop back to very low levels at later steps is due to the fact that once the weights (and thus activations) grow large enough, they almost all begin to land in the zero-gradient interval of the activation functions, resulting in no further gradient flow through those layers.
8

---

Published as a conference paper at ICLR 2024
becomes too large, such that the corresponding combination of (g())) and V (q") cause failure in some (and then all) of the five experiments. Figure 5b shows the layerwise gradient norms over the first 15 training steps of a failed run with CST with s 0.85 and m 1.22. Though on step I the gradients remain O(1), very quickly the exploding gradient phenomenon emerges.
contrast, Table shows that CNNs with C exhibit a more stable test accuracy, whereas the analogous tests for CST s.m. show a loss of accuracy for large where Va(a") 0.9 once 0.85 and = 1.53.
Vi"("" 0.69
0.8 and 1.61 well
The second failure mode accounts for the failure to train to high accuracy once starting sparsity reaches 85% for low values of m (the failure when 8 - 0.85 and m large is captured in the first failure mode); specifically, the failure of DNNs to obtain high accuracy when m = 0.81 for C as well as when 0.85 and m S 0.83 for CST,, m. In all these cases, EoC initialization avoids exploding gradients initialization and performance does improve during training for all of the five runs. However, the resulting accuracy is reduced, reaching a maximum of 778% when 0.85 at 0.81, for example. This is discussed further in App. G
0.85 and
One final observation from 2 is that for DNNs the maximum accuracy actually increases with increasing sparsity (up point, and for sufficiently large values of m). Maximum accuracy for ReLU, networks achieved when sparsity is 80% and 85% (accuracy of 94%). and for CST 7,m networks when sparsity is 80% (accuracy of 95%). However, CNNs in contrast do not show the aforementioned improved accuracy as result of sparsity, instead they exhibit more gradual loss of accuracy and train for somewhat larger range m.
10a
14
10
12
12
10'
10
10
Gradient nerm
dove Dollee
Gradient norm
10*
Teaning slep
10-2
10a
10
10"2
4
10-4
2
10-1
10
100
Layer index
Layer Index
(a) CReLU,,... = 0.85, m 1.74, Vi(g") 0.9
(b) CST CST..m:: 0.85, m 1.22, Vi(g") 0.8
Figure 5: Gradient norms per layer for the first steps a failed training run of DNNs on MNIST.
CONCLUSIONS AND AVENUES FOR FUTURE WORK
The analysis here explains why arguably the most natural highly-sparsifying activation functions ReLU, and ST, fail to train for deep DNNs and CNNs. This instability is then overcome with a simple modification in the form of a bound on the absolute value of these functions, yielding stable EoC initializations for the resulting networks, which allows for training very deep networks with these modified activation functions with activation sparsity up to 85%.
There are multiple natural and promising avenues to extend this line of work. Firstly, more com- plete explanation of the unstable training dynamics when is very large and m is small (the second failure mode identified in Section 4. Secondly, extension activation functions from DNNs and CNNs to transformer architectures, and networks with skip connections like ResNet would be important to have the greatest impact in recent applications. Additionally, it would be interesting investigate whether smooth variants of CReLU and or variants, specifically through
EoC analysis for these sparsifying
CST exhibit any preferable properties over their piecewise- controlling higher order moments of the spectrum of the input-output Jacobian. Finally, derivations of formulac for the largest choice of m for which the variance map is stable to the variance g' determined by the network width, input channels other numbers of network parameters.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY
Derivations of the theory in Section 2 and 3 can be found Appendices B and E Experimental
setup and hyperparamters are detailed in Section and Appendices and D
REFERENCES
Paschalis Bizopoulos and Dimitrios Koutsouris. Sparsely activated networks. JEEE Transactions on Neural Networks and Learning Systems, 32(3): 304-1133,,,,, 2021. doi: 10.1109/TNNLS.200 2984514.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state of neural network pruning? In I. Dhillon, D. Papailiopoulos, and V. Sze (eds.). Proceedings of Machine Learning and Systems, volume 2. pp. 129 146, 2020. URL 44144333111166644444224288888883
laaggssmlllyee www.mmllllllolo..m. Paper.
fi wtl.ss/apper//2000
Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of RNNs: Gating enables signal propagation in recurrent neural networks. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 873-882. PMLR, 10-15 Jul 2018. URL BRL.https://prrceediii
mlr mallr.pe www.ss.ppsss.om.m //0/sss/v0/// chen18i......... html
D.L. Donoho. De-noising by soft-threshoiding. IEEE Transactions on Information Theory, 41(3): 613 627, 1995. doi:
Simon Foucart and Holger Rauhut. Mathematical Introduction to Compressive Sensing. Applied and Numerical Harmonic Analysis. Birkhäuser, 2013. ISBN 978-0-8176-49417
Georgios Georgiadis. Accelerating convolutional neural networks via activation map compression. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7078- 7088, 2019.
doi: 10.1109/CVPR22019.0777
Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S. Schoenholz, Ed H. Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of Istms and grus, 2019. URL https:
//arxiv.org abte///b///// 901 08987
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connec- tions for efficient neural network. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL. http:::t/ pprtps::/prooceedings wwwprrpps.cppppper...eeee.e files/ 4024
Soufiane Hayou, Jean-Francois Ton, Arnaud Doucet, and Yee Whye Teh. Robust pruning at initial- ization. In International Conference on Learning Representations, 2020.
Soufiane Hayou, Jean-Francois Ton, Arnaud Doucet, and Yee Whye Teh. Robust pruning ini-
tialization. In International Conference on Learning Representations, 2021. URL https:
//openzeview. wtt.:eew.eettttoommddddxxxx/o/o.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1): 1011000821100000 2021.
Wei Huang, Richard Yi Da Xu, Weitao Du, Yutian Zeng, and Yunce Zhao. Mean field theory for deep dropout networks: Digging up gradient backpropagation deeply. In Giuseppe De Gi- acomo, Alejandro Catalá, Bistra Dilkina, Michela Milano, Senén Barro, Alberto Bugarín, and Jérôme Lang (eds.), ECAI 2020 24th European Conference Artificial Intelligence, 29 August- September 2020, Santiago de Compostela, Spain, August 29 September 8, 2020 Including 10th Conference Prestigious Applications Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial Intelligence and Applications, pp. 1215 1222. IOS Press, 2020. doi: 10.3233/FA1A200221 htRl.tttps://// org 1TT.:/110.333311112220
10

---

Published as a conference paper at ICLR 2024
Mark Kurtz, Justin Kopinsky, Rati Gelashvili, Alexander Matveev, John Carr, Michael Goin, William Leiserson, Sage Moore, Nir Shavit, and Dan Alistarh. Inducing and exploiting activa- tion sparsity for fast inference deep neural networks. In International Conference on Machine Learning, pp. 5533-5543. PMLR, 2020.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and
Yasaman Bahri. Deep neural networks as gaussian processes. In International Confer- ence on Learning Representations, 2018. URL http:ss://ppnneeieww.eeette http:/nee//ooroeoo
1-AAM-00
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. Snip: Single-shot network pruning based connection sensitivity. In International Conference Learning Representations, 2019. URL
atttps://openreviees n Boorumidd-Biigggi
Michael Murray, Vinayak Abrol, and Jared Tanner. Activation function design for deep networks: linearity and effective initialisation. Applied and Compulational Harmonic Analysis, 59:1 17-154, 2022.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learn- ing through dynamical isometry: theory and practice. Advances neural information processing systems, 30, 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp. 1924 1932. PMLR, 2018.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. Advances in neural information processing systems, 29, 2016.
Daniel A. Roberts, Sho Yaida, and Boris Hanin. The Principles of Deep Learning Theory. Cam bridge University Press, 2022.
com
Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017. URL https:
//openzeview.net//EEee wtw.:emmmii-HWUWUS---
H1W1UN9gg
Hidenori Tanaka, Daniel Kunin, Daniel Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information pro- cessing systems, 33:6377-6333 6389, 2020.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and mean field theory of CNNs: How to train 10,000-layer vanilla volutional neural networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 5393-5402. PMLR, 10-15 Jul 2018. URL hallhttps:/// proceediings...... mml.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/pprceeeddnggg/////// new.eeeeedddnnn..nnneseee..e. nwwrins copp://.......ii.....oppoppppr files/ 1tterr/2007/ELle//115556e 2500:7/007..BBssbaaaaaccceeeeees
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. A mean field theory of batch normalization. In International Conference Learning Representa- tions, 2019, URL /ttLittsss//ppprriiiee http::ewweeettttoeeeeeee httt//orummiddd SyyMumnCCF
11

---

Published as a conference paper at ICLR 2024
A GAUSSIAN PROCESS: VARIANCE, COVARIANCE, AND GRADIENT PROPAGATION AT INITIALIZATION
Numerous prior works Poole al. (2016); Schoenholz et al. 2017 Pennington et al. (2017); Mur- ray et al. (2022) have considered how the hyperparameters of and o. and the choice of activation function @ affect signal and information propagation through the network at initialization, as the network's 's trainability via gradient backpropagation. The standard approach in these works is to perform a mean-field analysis in which one considers each layer to have infinite width, that is, N - 0o for all I. In particular (Poole et al. (016) considered the (width-scaa scaled) length of the pre-activations.
well
The shape of both Vi and R2, along with the number of fixed points they admit and their associated stabilities depend on o, oy and or Of particular interest is the slope of this correlation map at p I, which is given by
X1. == R2(1)= = did (d'(V =)) (dzz).
(19)
When X1,6 < 1, the fixed point p² is stable, and any pair of points will asymptotically converge exponentially fast to being exactly aligned, whereas when X1. 1, the fixed point p" is unstable, and another fixed point emerges 0² < 1. In the latter case, even two input points which are arbitrarily close one another will asymptotically decorrelate. The former X1,é
regime is referred to in the literature as the 'ordered' regime, and the latter X1,6 > regime is dubbed the 'chaos' regime. The 'Edge of Chaos' (EoC) is the of points (aw. 0%) which separate these two regions, that is, the set of points (a 0%) such that (qg') = q" and X1. = 1.
It has been proven that initialising a network at the Edge of Chaos improves the depth of information propagation at initialization Yang & Schoenholz (017). Moreover, as shown in Schoenholz et al. (2077). EoC initialization also helps to prevent exploding and vanishing gradients early in training, which otherwise make a network untrainable. To understand why this is the case, consider the backpropagation algorithm with which deep networks are trained via (stochastic) gradient-based optimisation methods. Given a loss function L(x, 8) where = U U (b'), backpropagation calculates the gradients with respect to the weights and biases according to the follow recurrence relations
Oc(x, = 8t = = D'V C(x, 0). OhL
(20)
OC(z, 0) 0 =(D'WNttt))
(21)
ahl
OC(z, 0) -8/,
(22)
06
0L(x, 0) Ow!,
h
(23)
where D' is the diagonal matrix with entries Dú = 0' (ht). In Penningtoo et al. (2018) it was shown that in the widinite-width limit, though error vectors 5 are not Gaussian-like hi, the second moment of the error vectors distribution evolves according to own recurrence relation which depends on the value of X1.ø- Specifically, they show that (given the simplifying assumption that weights used during forward and backward passes are drawn independently from each other) q' := E=))() evolves according to
NL+ N1 X1.
-'
(24)
Thus when X > 1, the gradient norm is expected to increase exponentially over the course of the backward pass (a phenomenon known as 'exploding gradients") ), and conversely shrink exponen- tially (known as 'vanishing gradients") when X1, < 1. Both of these phenomena impede successful training of the network. In contrast, when X1,6 1 (and all layers have the equal width), gradient norm is preserved in expectation from one layer to the next, which allows for meaningful updates to be applied to all parameters.
12

---

Published as a conference paper at ICLR 2024
The X1.6 = condition also emerges from the analysis of the input-output Jacobian using tools from free probability in Pennington et al. (2017). The input-output Jacobian J is given by
OxL
D'W'.
J
(25)
Ox
1=1
The Jacobian is (as expected) closely related to the operations involved in backpropagation and more precisely involves the product of the linear backpropagation operators applied to compute each layer's error vector during the backwards pass (see Equation (2]). In Pennington et al. (2018).
the authors consider the limiting spectral density of JJJ with moment generating function
m
MaJT (2)
(26)
To briefly recap the key points of the analysis in inPeninntoo 2018)). assuming normalised input such that q² = g", then the distribution of D is independent of and the moments of D are given by
Pk =
(27)
The first two of these (μu and μ2z) define the first two moments of the spectrum of JJT, my and m2 according to
my (ouHz)
(x1,0)
(28)
m2 = (1) 2
11
+
- 1- -1)
(29)
112
where $1 is the first moment of the S-transform of WW When W is Gaussian with mean 0 and variance o /N-1 (as in the setup considered here), 81 -1 In Pennington al. (2018) the
authors also consider the case when W is orthogonal, in which case $1 =
From Equation (28 m, (x,,)),, which implies that for large L, the scale of the error vectors
in early layers is likely to blow up or collapse to 0 if X1. > or X1. < 1 respectively. However, while this X1. = 1 condition stops the growth of my as T. grows large, this only tells us about the growth in eгror vector in expectation. In PPennington et al. (2018), the authors' key motivation is investigate the possibility of imposing stronger condition: (approximate) dynamical isometry, where all singular values of the Jacobian are concentrated around one, protecting against even the worst case growth and decay the error signal.
Equations (28 and (2
yield an expression for the variance of the spectrum of JJJ when X1. = 1,
2 JJ = m2 m² = L
11 - 1 $1 16
(30)
The variance of the spectrum of JJJ for both ReLU, and ST, can be readily calculated using (27) in App. which grows linearly with depth as L/(1-s) and Ls/(1- s) for Gaussian and orthogonal weights respectively.
This shows that even with X1.6 = 1, the variance of the Jacobian spectrum grows linearly with depth L for generic μ4 and s1. Whether not this is possible avoid depends on the activation function and weights initialisaton scheme. One case analysed in [PPeninntton et al. (2018), for example, is the HardTanh activation for which (a")
1 as q² 0. Together with Equation 30), this means that while for Gaussian weights, with 81 = - 1, o x L for all g², with orthogonal weights (s) 0) one has that
as
0.
(31)
for fixed L, meaning that one can arbitrarily shrink the variance of the spectrum of the Jacobian by sufficiently shrinking q² They show similarly that it is possible achieve arbitrarily small of for networks with Erf activation functions and orthogonal (but not Gaussian) weights initializations, and also that this is not possible for ReL.U networks (neither with Gaussian nor orthogonal weights).
13

---

Published as a conference paper at ICLR 2024
Empirically, even in the absence of dynamical isometry, EoC initialization yields significant benefits in terms of enabling and speeding up initial training of very deep nets Pennington et al. (2017).
The process for calculating EoC o and of is as follows: (i) select q", say the variance of your (typically already normalised) input data; (ii) use Equation (IT) with X1.6 = 1 to solve for o (ii)
use Equation with (q4)) = q² to solve for of
B VARIANCE AND CORRELATION MAPS FOR ReLU, AND ST,
Recall that the variance map for general activation function ø is the following:
(4(a) o / (((v)=))>><<))) + oz¹
(32)
where (dz) = / 2n dz is the standard normal distribution. We define
(z) = / (dt))
(33)
being the cumulative distribution function (CDF) of standard normal distribution, which is related to the error function by
erf(z) = 22((22)) 1 + 4(V2x)= erf(x) +
(34)
2
1 -1
1
erf e =
+ (x) = /2erft((2:)) 1).
(35)
V2
2
Recall further that ((-2)) = 1 - (22), as well as the following identities:
ty(tt)= cxp(-22/2)
t (dt)) exp(-22/22 + (1-
(36)
2xx
V2n
With these in mind, the variance map for ò ReLU, can be computed as follows:
VRLLU, (4) = 0²
(Vqz 7) + of
1/V9
(g=² -2/ 7²) (dz)
+ +
T/ V
00
1 z7(d2) +2
0 00
2
2² (d((2)
-2
(dz2)
+
T/
T/V9
7/v3
72
T
-
exp
+ 1 - +b
22mq
2q
/
TT exp
72
+ 2
1 +b
T
)
+ o
2t
2q
VQT
72
)
22
(a+
1- +
exp
+ +
2
(37)
2q
Therefore,
ViELU, (4) = 0²
②
T
7(g + 22)
exp
72)
T
exp
72
V4
2q 3/2 2TT
2q
2 22
2q
72
22Rq
exp
29
2/
+b
T
2
+
T
(38)
Vq
Vq
14

---

Published as a conference paper at ICLR 2024
0
0.25
0.5
t=10
0.5
0.5
0.5
0.5
(10))
0.0
0.0
0.0
0.0
-05
1.49,
53 224,
0
2.49,
x1
% -
21
% 6.3.
-1.0
-1.0
-1.0
Figure 6: Correlation maps for ReLU, with (ow, 07) on, and on either side of, the EoC, for different values of except in the case where q² exists for those hyperparameters.
Let q" be fixed point of the variance map, such that VRLLU, (4")= map at 1 is then given by
= q² The slope of the correlation
XI,RLLU, = oh / (d)(v(²z)) 7(dz)
R
= 0 f 7(dz) = = 1 -
T
VieeL,, (q").
(39)
As result, if DNN is initialized at the EoC (i.e, XI,RoLU, = 1), then fixed point of the variance map satisfies ViL (g") XI,RaLU, = 1. Moreover, note that for all q,
T2
ReLU, (4) = oh 22/2m/3/2
exp
0.
(40)
2q
This shows that the fixed point q² unstable when the DNN is initialized at the EoC.
This phenomenon also holds for DNNs with soft thresholding activation function ST In fact,
F/v9
( (VT=- - ([g==>)>>((k)))) ((dz)
VIST, (4) 02
(Vīz + 7(22) +
+ of
7/V4
p
20
(V7z-7)² - 7)² -
+ to
T/V4
2VRLLU, (4)
oh
(41)
and, if g² is a fixed point of the above variance map, then the slope of the corresponding correlation map at p 1 is
0² XIST,
(o'
(Q((VT2)))
7(dz)
JR
(f-
+ .
2²
7(dz) 20
1 + Vq-
(42)
(T/TT)
As result,
VST (g) = 2V (g), X1.ST, = 2X1,ReLU, = (g") VSS (g) 21 2V (q) > 0, (43)
and the fixed point is unstable when such DNN is initialized at the EoC.
Correlation maps R2g (p) for ReLU, and ST, are plotted Figures 6 and 7 for different choices of the hyperparameters (a.,, σb), excluding those cases when > does not exist. Attempting to regain the stability of (q) by setting X1,ReLU, < 1 would result the DNN being unstable to small perturbations. i.c. R(p) being unstable at p 1. Alternatively, having X1,ReLU, > 1 results in the Gaussian process having variance growing exponentially with depth and the associated exponential scaling of the gradient with depth.
15

---

Published as a conference paper at ICLR 2024
0.25
0.5
r=10
0.5
0.5
0.5
0.5
(10)
0.0
0.0
0.0
0.0
-05
0.25,
0.62,
245.1
01
1,25,
x1
0
1.62,,2,200
-1
03
0313.15x
-1.0
-1.0
-1.0
-05
Figure 7: Correlation maps for ST, with (ow. σb) on, and on either side of, the EoC, for different values of except in the case where no q" exists for those hyperparameters.
0.25
1.0
0.5
6.7
12
0.0
8
6.5
8-1
6.1
1.2
2.5
150
3256
0/
0%
Figure 8: Edge of Chaos for ReLU, and ST, for different As expected, the o values for ReLU, are twice those for ST, for the same q². The black dot corresponds to to"" =.
C EXPERIMENTS DEMONSTRATING TRAINING INSTABILITY FOR ReLU, AND ST
Width 300, depth 100 feedforward networks are trained with the above activation functions to per- form image classification on the MNIST dataset. The networks are initialized at the FoC using q" 1, and trained with SGD with 10-4 learning rate for 200 epochs.
When т > (the only exception being ReLU, with s = 0.5, i.e. standard ReLU), the input is normalised to have variance < q² give the best chance of trainability given the instability of g in these cases (in particular the input variance is set to be 0.75). As seen numerically however, in particular at higher sparsities, these measures are insufficient enable trainability of ReLU, and ST, networks.
variety of starting sparsity levels ([50%, 60%. 70%)) are tested for each activation function, and track both their test accuracy, as well as the sparsity of their activations during evaluation, to check to what extent the initial sparsity levels are maintained during training. We plot the Variance maps Vo for each activation function for these settings in Figure 9 For each activation function- hyperparameter combination, we train 5 runs. Table shows the mean and standard deviation the test accuracy and activation sparsity after training, measured the test set.
The results concord with our expectations based at the FoC and variance map analyses above. First, ST, networks fail to train any of the tested sparsities. This reflects the notable instability of g" for ST, with the corresponding values r, as shown in Figure 9 Similarly, ReLU, networks also fail to train
sparsity (and thus 7) grow sufficiently large.
The CNN experiments on the CIFARIO dataset use networks with 50 layers, and 300 channels per layer, and were trained for 200 epochs with SGD and learning rate le-3. A single experiment was run per (activation function, sparsity) pair. Again, both activation functions fail to train at high sparsities.
16

---

Published as a conference paper at ICLR 2024
D ADDITIONAL EXPERIMENTAL IMPLEMENTATION DETAILS
For both MNIST and CIFARIO, 10% of the training set held out the validation Results reported are computed on the test set.
All layers are initialized on the EOC, with a slight modification for the first layer. In our experiments, the first layer is initialized so as to preserve the input variance in the first layer's pre-activations. This was an additional effort to try our best to protect against the instability of q"" for the non-clipped activation functions. The EoC initialization is based on the assumption that the input to the layer are 'post-activations. that is, the incoming vector a Gaussian which has been passed through the activation function. However, the input to the first layer has not been passed through activation function and thus is dense, not sparse multiplication of the inputs by a weight matrix initialized at the EoC would substantially grow the variance, possibly to value larger than q", which would then prevent convergence to with these activation functions.
Experiments
single V100 GPU, and were implemented using Pytorch Lightning.
0
2.0
0.25,
2.0
0.84,
0.32 5 0.7
=
1.5
1.0
0.5
0.5
0.0
1.00
1.75
0.50
1.25 1.50
Figure 9: Variance maps for ReLU, (left) and ST, (right) with (aw: σb) the EoC for different values of sparsity s.
E VARIANCE AND CORRELATION MAPS FOR CReLU, r,m. AND CST
The variance map of CReLU,, r.m. is computed as followed:
(r+m)/
VCReLU,... (g) = o2
(/72-7)2n(d2)-
m²y(dz)
+ +0
7/V9
(r+m)/v3
20 (v) - (2) + + c3(0))
(44)
where
(Vqz - 712 7d2)) = 1 (9 2/qT2+
v1(g) = 3/
- (((4m)/
v2(g)
(vgz 7)2 (d2))
(9 -2-22/944 +))
(T+m)/
vg(g)
m² (d2z)
(45)
(r+m)//V Vq
Notice that v1 (1(2) v)1(g) Tv)2(g) + r"via(g), where
1 2 2r(22),
(1z(9) { 0
011(9) =q
2² (d2x), (2)( = 2
7(22).
(46)
17

---

Published as a conference paper at ICLR 2024
Therefore,
dun - 2² ((d2)
dq
dq
TIV
f.m. TIV
+ a d
2² 7d2))
x² (((())
=
7/VT
72)
73
72
exp
(-29)
+
+
exp
V2Tq
'29
V
2v 22/2
29
do12
2/
xdd22
dp
dy
./V4
1
p 00
5.00 2 (dz)
(dz2)
+ 2 2/ dq
v
7IV
.Iva
72
72
exp
+
exp
V2Tq
v 2T 93/2
2q
du13
d
72
(dz)
exp
dq
dg
T/
2//2 2//2
29
Hence
du
du11
du12
d
dq
dq
dg
dq
22
73
72
exp
+
+
exp
v
22
29
v
2/2xx 22/2
29
T exp
72
T3
72
33
72
exp
+
exp
22
2q
2 q% 92/2
2g
2 2T 9312
2g
1 +
(47)
One could similarly derive
duz
T + m
(T (T+ m) mm)
+ m
(T + mm)
( (T ( m)
exp
+
+
exp
dq
/2Tq t /2Tq
2q
2/22 2m 93/2
29
(T + m)²
7(T + m)
(T + m)
exp
exp
29
222 92/2
29
(2
m)
(T + m)
exp
2// 22/2
2g
m
(T + m)²
( T m
+ m² r m)
(- (T + m)²
V2nq exp
+ 1
exp
(48)
29
/
2 2m 9 9/22
2g
Finally,
dv3
m² + m)
(T m )
exp
(49)
dg
2T2
92/2
2q
80
dVCRoLU....
dv
du2
dv3
+
dg
dg
dq
dq
T
+
m
T
m
(T + m)) 2
+
exp
/
/2mq
2q
m
2
11
erf
22q
m
2
1
erf
22g
22q
exp
(- (T + m)
2q
(50)
18

---

Published as a conference paper at ICLR 2024
The slope of the associated correlation map p 1 for DNNs initialized at the fixed point q" is (r+m)//
XI,CReLU,,,, = =w JT/ 7(dz) = u.
+
+
m
T
V
Vq"
1
+
1
jer
erf
(51)
/24"
2
V24"
Therefore
(m + T)²
XI,C VCRELU,.... (4") +
2x exp
V.RLL. (q"),
(52)
20²*
which makes any fixed points qº of VCReLU>.. at the EoC locally stable, as described in Sec.
For the other activation functions CST the variance map is:
(r+m)/vV m² xd2)+ + / -(r+m)/
r/v
VCST. (g) = ow
(Vqz + + r(2z)
(T+m)/
+
(Vqz - 7dz)) + J(r4m) m² (dz))
to
(r+m)//v
20 2
(Vqz - ((dz)
m² 7d22)
+ +
(r+m)/V4
2VCReLU.,.. (4)
o
(53)
and the slope of the associated correlation map
7/v4
(rtt))/va
X1,CST.. o.
7dzz)
-(F+m/\/
(r+m)//4
= 20
7dz)) 2x1,CRaLU,,.
(54)
J/VT
This yields VCSTT.. (q") 2V (RELU.. (g*), and that X1,CST,. -X1,CReLUr..
ViSST (q") + V202m m VTg
(m (m++7) 7)2
VCST (g"), (55)
exp
24*
which shows that any fixed points d² of VEST the EoC are locally stable as well.
19

---

Published as a conference paper at ICLR 2024
1-
0.5
T=10.
2. 202
02. 3,93, of -22 222
4.38 213
5-5.44.0 2.
1..m
2
127,0 025 n=1
2 7.56, po
0.5
1-100.m
of -
4g 3.l. dj 2i11
3557.0 12
4.33 3.34. 21
0 7.36 -
Figure 10: Variance maps for CReLU,... rmm with (a.., 0b) and either side of the EoC, for different
and m, with 9º' = 1. From top to bottom, the rows correspond to m 1. m = 1.5, and m 2, and from left right the columns correspond to T 0, 0.25, 0.5 and 7
20

---

Published as a conference paper at ICLR 2024
025.0-11
0.5
1=10.
2445.0
3.06. -08
62- 2.46, 022.
213
3.300.0000 2.
0 1.446660 -0
1-10.mm
075.0)
4. 2:15, m11
of 025
og 275. of p
ni
0.5
2,05, 111.
23 225, 92 n11
2.05
Figure 11: Variance maps for CST 7.m with (ow, lbb) and either side of the EoC, for different
and m, with q = 1. From top to bottom, the rows correspond to m 1, m 1.5, and m 2, and from left to right the columns correspond 0, 0.25, 5 0.5 and 1.
21

---

Published as a conference paper at ICLR 2024
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
-0.25
12 1.93,
06 2.335
3.14.
6.36.
2.93.
3.38
02 4.144 m1
6.36. 1.36.
-1.00
-1.0
-0.
0.0
0.5
1.0
-1.00
-1.0
-0.
6.0
0.5
1.0
-1.
3.00
-1.0
-0.5
0.0
0.5
1.0
-1.00
-1.0
-05
0.0
0.5
1.0
0.25
0.25
0.25
0.25
RG0
0.00
0.00
0.00
0.00
-0.25
-0.25
-0.25
-0.25
-0.50
1.311 21
-0.50
1.777
-0.50
21
-0.50
5.50.
-35.x
6.56.)-- x)-1)
333,
3.777
4.5.
7:56.1.>1
-1.00
1.0
-0.5
1.0
-1.0
-0.
1.0
-1.0
-0.5
0.0
1.0
-1.0
-0.5
0.0
0.5
1.00
T 0.0
2.0
1.00
T- 025, m- 2.0
T- 0.5,
2.0
T 1.0,
2.0
0.00
0.00
0.00
0.00
-0.25
-0.5
-0.25
-0.5
-0.00
21
-0.50
1.57 =1
-0.50
2.31
X
-0.50
5.36.
-0.75
31
-0.75
337,
-0.75
4.31
4.31
-0.75
-0-0-.00 1.36.
Figure 12: Correlation maps for CReLU, r.m. with (a... .b) at and on either side of the EoC, for
different and m. From top bottom, the rows correspond to 1, m 1.5, and m and from left to right the columns correspond to 7 0, = 0.25. = 0.5 and 7 = 1. Fixed points p" < 1 (where the correlation map crosses the diagonal R(p) p) are indicated by round markers on the relevant curves.
22

---

Published as a conference paper at ICLR 2024
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.25
0.25
-0.25
-.466
0.46,
0.69,
08 -.07
2.68.
0/ 214.
21
1.69 1.6..)
2.077 X1
3.60 1168..4 m1
4.68.
-1.00
-1.0
-05
0.0
0.5
1.0
-1.00
-1.0
-0.5
6.0
0.5
1.0
-1.
3.000
-1.0
-0.5
0.0
0.5
1.0
-1.00
-1.0
-05
0.0
0.5
1.0
0.25
0.25
0.25
0.25
RG0
0.00
0.00
0.00
0.00
00.5
-0.25
-0.25
-0.25
-0.50
0.15
m. 0.38
-0.50
0.75.
-0.50
m 2.20.
1111
21-1
1.38
-175..
12.88....
2.38
2.75
4:28..3.>1
1.0
-0.
1.0
-1.0
-0.
1.0
-100
-0.5
0.0
0.5
1.0
-10
-0.
0.0
0.5
1.00
0.m
2.0
1.00
T- 0.5,
m- 2.0
T- 0.5,
2.0
T 1.0,
2.0
0.00
0.00
0.00
0.00
-0.55
-0.55
-0.75
-0.7
00.0
0105
-0.50
0.29
0.29.11 X1
-0.50
0.85 x
-0.50
2.18.
-0.75
2.00. x
2.05
-0.75
2.29.1
2.29
-0.75
2.85 m-
-0.75
4.10
Figure 13: Correlation maps for CST, r.m with (a ob) at and on either side of the EoC, for different
and m. From top to bottom, the rows correspond m 1, m 1.5, and 2, and from left to right the columns correspond to 0, t = 0.25, = 0.5 and 1. Fixed points p* < 1 (where the correlation map crosses the diagonal R(p) p) are indicated by round markers on the relevant curves.
23

---

Published as a conference paper at ICLR 2024
F RELATIONSHIP BETWEEN oi AND THE EXPECTED SPARSITY AT THE EOC, CONTROL OF VARIANCE OF THE INPUT-OUTPUT JACOBIAN.
Recall that the 'expected sparsity' with respect to an activation function & could be computed un- der the usual mean-field assumption (that each pre-activation Z follows normal distribution with variance q
(()( =), 0), Z ~ NO
(56)
When studying DNNs with sparsifying activation functions (Ø = ReLU, ST CReLU r.m or 7m),, we would like to find the optimal such that the DNNs achieve the expected sparsity The optimal 7 is related to the and g" as followed:
For ø ReLU, CReLU,,
= ==(,00"" = inn(( 2
|P(fTTP((2) 2,,
= = (s).
(57)
and for 6 ,T, or CST 7,m-
= fo(s, = inf(7 [P =) - ~(,,4""))
=
~(0,,0')}
~~
(8+1)
= -2 (
2 2
(58)
There is simple formula compute oy for initialising the DNNs the EoC (x1. 1) with activation functions @ = ReLU, or ST,. In particular,
* for Φ = ReLU,
(Fols.g))
))
1
- = = (1- -
=(1 -11 + (s))
(59)
1-
and for ST,, using the identity eri
eri (s)
1 Φ (+t)))
V2
f(s 9²)
a = -
1- +
ViT"
1 2
+ +
1 1
(60)
2
Notice that the o above are independent of the variance of the pre-activation q". This is not true for the cases of the clipped activations ò CReLU, or CST, 7, instead, the initialization of at the EoC now depends on g², m, and s. fact.
for CReLU m. using the identity + --1(s) //eerff1//0 1):
-1
m +
2
erf
erf
v 124"
129"
+ /2q= ert
(2ss (2s-1) 11)
erf
erf (erf (22 11)
V29"
2 erf
12)
erf
-1)
-2s +
(61)
/2q"
24

---

Published as a conference paper at ICLR 2024
and for CST, r,m
erf
M7 +
erf
7
'24'
1290
m1 + /24 ert
((0)
-1
erf
erf (erf (8))
V2q"
-1
erf
erf (s)
(62)
/2q*
Despite the fact that that o is now dependent on q", the derivatives of CReLU,,, r.m and CST r.m take only values of zero or one, like those of their non-clipped counterparts, and thus the moments of
the spectra as computed by (27) are independent of k. see Table 3 Once again, for both clipped activation functions, Mk μ for all such that M1/p2 o as was the case for ReLU, and ST,.
For both CReLU (s 2 0.5) and CST 7,m (8 2 0), from Equations (61) and 62 that
as
0,
633)
meaning that Ji grows linearly with with these activation functions, with at least the same rate as their unclipped counterparts.
Nonetheless, the prospect of stable EoC initialization stands substantial improvement over the unclipped variants, and as we show in Section 4 this appears sufficient to enable trainability of very deep networks with sparse activations in practice.
Table Summary statistics for the Jacobian's 's spectrum
pk
Mp² (z)
σ 1
ReLU,
1 +
L (o 81)
G2
1-
1
1
ST,
1 +
L L $1)
1-2
m +
1
CReLU
2 erf
erf
L L - - 81)
22
V2q'
1-
CST
erf
171
erf
L L0 -1-11-10 81)
/2q
V2q"
G I
25

---

Published as a conference paper at ICLR 2024
G FAILURE TO TRAIN FOR HIGH SPARSITY AND LOW MAGNITUDE CLIPPING
The second failure mode discussed in Section appears not to a failure of the initialization, and thus rather suggests the existence of challenges when training networks with such a high level of activation sparsity combined with small values of m. Indeed, inspecting the training loss curves (see e.g. Figure (4) suggests that though training begins in an expected fashion, the problem for those runs exhibiting low accuracy is a combination of slow training, and instability later on in training in these cases, we see the emergence of large spikes training loss, which (to varying degrees) are not fully recovered from. Investigating the source and/or dynamics of the observed training speed and stability issues late in training is beyond the scope of this paper, and we defer it to future work.
1.0
3.0
2.5
2.5
3.5
1.0
Acc
Ace
0.5
25000
50000
75000
100000
125000
150000 175000
25000 50000
75000
159000
125000
150000
175000
Baning Aoss
brining
(a) CST r.m. 0.85, m 0.81
(b) CST 7,m, 0.9, m 0.76
Figure 14: The loss curves during training for each of the 5 runs, for two activation function and hyperparameter configurations with high activation sparsity which achieve low test accuracy as result of slow and/or unstable training. The legend shows the corresponding final test accuracy achieved by each model.
H EXPERIMENTS WITH SHALLOWER NETWORKS
We have repeat the DNN experiments on MNIST from the main paper, but with depth 30 instead of depth 100. The results are shown in Tables and 5
We can see from these new results that the key conclusions from the 100 layer experiments hold true with 30 layers too. While ReLU, and ST, , networks fail train consistently once sparsity reaches 80% and 70% respectively, CReLU and CST T.m networks can train consistently to high accuracy with activation sparsity 90%.
26

---

Published as a conference paper at ICLR 2024
Test Accuracy std
Test Sparsity mean
mean
std
ReLU,
0.70
0.52
0.93
0.01
0.67
0.01
0,80
0.84
0.60
0,46
0.47
0,42
0.85
1.04
0.27
0.37
0.18
0.38
0.90
1.28
0.10
0.00
0.02
0.00
ST,
0.70
1.04
0.27
0.38
0.14
0.31
0.80
1.28
0.10
0.00
0.00
0.00
0.85
1.44
0.10
0.00
0.01
0.00
0.90
1.64
0.27
0.37
0.19
0.38
Table Experimental results for ReLU, and ST, DNNs with 30 layers MNIST, with sparsity
up to 0.9. The mean and standard deviation of 5 runs are reported for both test accuracy and average activation sparsity calculated on the set.
Test Accuracy
Test Sparsity mean
mean
std
std
m
V((*)
V"(a")
CReLU,.. r.
0.70
0.52
1.05
0.5
-0.37
0.91
0.01
0.70
0.01
1.45
0.7
-0.31
0.92
0.00
0.70
0.01
2.05
0.9
-0.04
0.93
0.01
0.69
0.01
0.80
0.84
0.89
0.5
-0.24
0.90
0.02
0.80
0.01
1.27
0.7
-0.12
0.92
0.01
0.80
0.01
1.85
0.9
0.21
0.92
0.01
0.78
0.02
0.85
1.04
0.81
0.5
-0.14
0.90
0.01
0.85
0.01
1.17
0.7
0.02
0.91
0.01
0.84
0.01
1.74
0.9
0.41
0.92
0.01
0.84
0.01
0.90
1.28
0.72
0.5
0.00
0.85
0.06
0.90
0.01
1.06
0.7
0.23
0.90
0.03
0.89
0.01
1.61
0.9
0.69
0.91
0.01
0.74
0.02
CST r.
0.70
1.04
0.81
0.5
-0.14
0.91
0.01
0.70
0.00
1.17
0.7
0.02
0.92
0.01
0.69
0.01
1.74
0.9
0.41
0.93
0.01
0.66
0.02
0.80
1.28
0.72
0.5
0.00
0.90
0.01
0.80
0.01
1.06
0.7
0.23
0.91
0.01
0.79
0.01
1.61
0.9
0.69
0.92
0.01
0.49
0.16
0.85
1.44
0.67
0.5
0.11
0.88
0.02
0.84
0.01
1.00
0.7
0.39
0,91
0.01
0.84
0.01
1.53
0.9
0.89
0.27
0.37
0.44
0.21
0.90
1.64
0,62
0.5
0.28
0.77
0.17
0.90
0.01
0.93
0.7
0.63
0.91
0.01
0.89
0.01
1.44
0.9
1.20
0.11
0.00
0.25
0.01
Table 5: Experimental results for CReLU, and CST 7.m DNNs wth 30 layers on MNIST, with sparsity up to 0.9. The mean and standard deviation of 5 runs reported for both test accuracy and average activation sparsity calculated on the test set.
27


Reviews:
[{"review_id": "I4xAWbXsvi", "reviewer": "Reviewer_TyEd", "summary": "The paper studies the effect of sparsity on the activation function for deep neural network initialization using the existing dynamical isometry and edge of stability theory. In particular, the authors compute the so-called variance map and correlation maps for sparse activating functions, namely, shifted ReLU and soft thresholding, and interpret the shape of these maps, in particular, the values $V'(q^*)$ and $V''(q^*)$ to explain the failure. Then they propose magnitude clipping as a remedy and empirically show that with these magnitude-clipped sparse activation functions, it is possible to train the deep net without losing test accuracy and with high test fractional sparsity.", "questions": "Also, I do not understand the heuristics given in Section 3.1 for how to choose $m$. I understand the dependence of $V'(q^*)$ and $V''(q^*)$ on the magnitude value $m$ is non-trivial from Figure 4. Still, the curves follow regular shapes so maybe it is possible to give simple heuristics based on Figure 4?\n\nI will consider increasing my score based on the author's response to my questions.\n\n------------------------------------------------------------------------------------------------\n\nPost-rebuttal: The authors sufficiently addressed my concerns. I am leaning acceptance.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper introduces two natural classes of activation functions with a tunable parameter $\\tau$. I think understanding what activation function works better for which purpose is an active and fascinating area which I also believe is to appeal to general interest. Sparsity is particularly an important goal to achieve for modern large deep learning. Making sure that the network has non-exploding or non-vanishing gradients at initialization is indeed a sufficient condition for the applicability of an activation function.\n\nThe introduction is well-written. \n\nOn the flip side, I am not entirely sure why sparse ReLU fails to train. Table 2 only shows results for magnitude clipping for CReLU and the usual ReLU with $\\tau=0$.", "weaknesses": "I have three major questions that I wasn't able to resolve by reading the main text only.\n\n1. What is the criterion on $V_\\phi$ for successful training? For example, let's see Figure 2. Which of the shapes are good and expected to train well vs which are the ones that are expected to fail? I see that for $\\tau=1$ the curves have higher curvature. In particular, the blue curve intersects the line $x=y$ at one point where the derivative is non-zero but the curvature is positive. Is this expected to fail because the derivative is non-zero? I found the explanations in the text somehow repetitive and hard to parse. Can the authors explain the criterion on $V_\\phi$ in words just from Figure 2? \n\n2. Can the authors please provide experiments with CReLU $m=0$ as well and also for ReLU in Table 2? Why is there only one row for ReLU?\nAs the table stands now, I am not convinced that sparse activation functions without magnitude clipping fail to train. Is the 'unstable training dynamics' reported in the paper for very large $s$ and small $m$ as claimed in the conclusion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8kkqh69qUm", "reviewer": "Reviewer_auBc", "summary": "The paper aims to encourage high sparsity in the activations of neural networks with the motivation to reduce computational cost. To this end, the authors study the activation dynamics induced by common sparsity inducing nonlinearities such as ReLU and Soft-Thresholding (ST) under random initialization. The authors, via the large width Gaussian process limit of neural networks, discover a training instability for ReLU and ST nonlinearities. They show that the instability can be resolved by clipping the outputs of these nonlinearities. The authors validate the theory through experiments and show that the modification allows training MLPs and CNNs with very sparse activations with no or little reduce in test accuracy.", "questions": "Questions:\n- Is this the first time that the variance map equations are being derived for these non-linearities?\n\nMinor:\n- Plots of Figure 2 are missing axis labels and the plot legends are not readable on printed paper.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The writing is very clear and easy to follow.\n* The theory is elegant.\n* The theory works in practice and the authors effectively demonstrate being able to train neural networks while maintaining high activation sparsity.", "weaknesses": "Minor weaknesses:\n* There could have been a study of the computational efficiency since that is the main motivation of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yDxssEltP9", "reviewer": "Reviewer_qoAp", "summary": "The authors study sparsity inducing non-linear activation functions in neural network training. They provide a theoretical analysis of the instability effects for using shifted ReLU and SoftThreshold when training FeedForward networks and CNNs, motivating their clipped versions of these functions. The authors use the previously introduced EoC initialization and show why training is stable, and becomes unstable with the introduction of a modified ReLU activation function that should induce sparsity.\nTo remedy the instability in training a further modification (clipping) is introduced and proven to be theoretically stable. They then demonstrate the feasibility of their CReLU and CST activation functions for training deep and wide networks on MNIST and CIFAR10, showing only minor degradation in prediction accuracy with tunable levels of activation sparsity.\n\nOverall, the paper is nicely written and relatively easy to follow, despite the math-heavy theoretical section. The argumentation on instability of the shifter ReLU and SoftThreshold seems valid, and the experiments, though not extensive, provide a proof-of-concept of the author’s claim. The idea of clipping the activation functions to achieve stability is as simple as it is effective, providing a valid contribution that can be actually translated into application.  However, a bit more in-depth evaluation of certain aspects would be nice. The findings are interesting, though presentation is lacking, as well as more exploration of the introduced concept.", "questions": "- I don’t fully understand the meaning of Figure 4 and it is not sufficiently discussed in the manuscript.\n- The important EoC concept is not motived and explained enough.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed clipped functions are an easy and natural extension of already existing activation functions.\n- The introduced parameters tau and m are able to control sparsity and training stability as shown theoretically and experimentally.", "weaknesses": "- The stddev in table 2 are a bit odd to me, either a higher numerical precision is needed, or it needs to be explained why the are many cases of zero std.\n- Results on ST are only shown in the appendix. They should be shown and discussed in the main paper, given that they are an essential part in the rest of the manuscript.\n- the higher sparsity regime in table 2 (s=0.85) needs to be explored/explained more, there are interesting things happening.\n- There is no comparison to existing methods, but the authors clearly describe the lack of related research.\n- No source Code provided, implementation details on experiments only in the appendix", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pfyoH65aPs", "reviewer": "Reviewer_FeGe", "summary": "The paper studies how very deep neural networks, including densely connected and convolutional networks, behave at initialization when using sparsity inducing activation functions. The two natural sparsity-inducing functions studied in the paper are the shifted ReLU activation, which is just a ReLU with a fixed bias, and the soft thresholding function, an activation that evaluates to zero in some fixed interval. The main result shows that these activations make the initialization unstable for very deep networks. This instability can be fixed by using a clipped version of these activation functions. The authors show some experiments, demonstrating that deep networks can be trained with a clipped version of the above activation functions, with minor drop in accuracy.", "questions": "See above.\n\n==================\n\nScore is updated after author response.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Using sparse networks, with sparse activations and/or sparse weights, is an important field of study, and it seems that the paper gives some contributions on how the activation function of the network should be adapted to support training networks with sparse activations. This work can potentially serve as a basis for future works on building sparse neural networks.\nThe theoretical analysis of sparse activation functions, their problematic behavior at initialization of deep networks and the solution of clipping the weights is to my understanding novel.", "weaknesses": "The main weakness I find in the paper is that while the motivation for the paper comes from a practical perspective, namely building neural networks with sparse activations that can be implemented more efficiently in practice, it seems that the applicability of the results is not clear. To my understanding, the results only apply for very deep neural networks (the experiments use 100-layer networks). The authors should clarify whether or not their results apply to networks of reasonable depth. Specifically, it would be good to show some experiment for networks of reasonable depth and show how the activation choice affects the behavior. It seems that in this setting depth is only hurting performance, so while it is in theory interesting to analyze how such deep networks should be trained, it seems that the applicability of this method is limited.\n\nThe authors should also discuss the effect of adding residual connections on the stability of the network training. As residual-networks has been the main solution for training very deep networks, the authors should clarify whether their results also apply for residual networks.\n\nAdditionally, it seems that some of the experiments were left out of the main paper, and only appear in the appendix (for example, studying the CST activation and comparing the clipped activations with non-clipped ones). These are key experiments in the paper and should appear in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)