The Effect of Residual Architecture on the Per-Layer Gradient of Deep Networks
Authors: Etai Littwin
Lior Wolf
Venue:  (2020)

Abstract: A critical part of the training process of neural networks takes place in the very first gradient steps post initialization. In this work, we study the connection between the network's architecture and initialization parameters, to the statistical properties of the gradient in random fully connected ReLU networks, through the study of the the Jacobian. We compare three types of architectures: vanilla networks, ResNets and DenseNets. The later two, as we show, preserve the variance of the gradient norm through arbitrary depths when initialized properly,  which prevents exploding or decaying gradients at deeper layers. In addition, we show that the statistics of the per layer gradient norm is a function of the architecture and the layer's size, but surprisingly not the layer's depth. 

This depth invariant result is surprising in light of the literature results that state that the norm of the layer's activations grows exponentially with the specific layer's depth. Experimental support is given in order to validate our theoretical results and to reintroduce concatenated ReLU blocks, which, as we show, present better initialization properties than ReLU blocks in the case of fully connected networks.

Under review as a conference paper at ICLR 2020
THE EFFECT OF RESIDUAL ARCHITECTURE ON THE PER-LAYER GRADIENT OF DEEP NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
A critical part of the training process of neural networks takes place in the very first gradient steps post-inittalization. In this work, study the connection between the network's 's architecture and initialization parameters, to the statistical properties the gradient in random fully connected ReLU networks, through the study of the the per layer Jacobian in finite sized networks. We compare three types of architectures: vanilla networks, ResNets and DenseNets. In our analysis, we show that while the variance of Jacobian squared norm is exponential in depth for ResNets, and polynomial for DenseNets, there exists an initialization strategy for both, such that the norm is preserved through arbitrary depths, preventing exploding or decaying gradients deep networks. We also show that the statistics the per layer Jacobian norm is function of the architecture and the layer's 's size, but surprisingly, not the layer's 's depth.
INTRODUCTION
Understanding the effect of different architectures on the ability to train deep networks has long been a major topic of research. recent line of work has focused on the statistical properties of both the activations and the gradients of different architectures the point of initialization, in order to better understand the effect of different architectural choices on the training dynamics.
well-known hurdle in the ability train deep networks, is the effect of depth the magnitude of the gradients. Specifically, the exploding and vanishing gradient phenomena have been observed in various deep architectures. they are considered main culprit in the failure of training deep networks successfully. A flurry of architectural tricks and initialization techniques have been proposed, in order to mitigate the problem of exploding and vanishing gradients, ranging from residual architectures, different normalization techniques and orthogonal initializations.
In this work, we perform rigorous analysis of the norm, of both the activations and the gradients of each layer, in fully connected architectures at the point of initialization. Our setting is as follows: we assume a fixed input y to an T. layer neural network with weights given by W, and intermediate output for any layer i denoted by yf. Denoting by w/ the weights ij of layer k. we are interested in the quantities:
Oy Fw 1
11'' 11
Σ
representing the norm of the activations of layer and Jacobian with respect to layer k. We consider these quantities random variables at initialization, and, therefore. devote our analysis to deriving expressions for the mean and variance of both I|y' I|2 and ||/ over the random initializations of the weights. Note that the full gradient requires an additional loss term which is independent of the architecture, and so our attention is focused on j While most initialization techniques are designed to maintain the squared length of intermediate activations through the layers, the multiplication of i.idd random weights along input-to-outpun paths in increasingly deep networks might lead to unbounded higher order statistics. In the extreme case, while the mean of activation and gradients remain constant in deep layers, it would be improbable to achieve mean values for any specific realization of the weights, rendering training difficult. Specifically, given some loss function L, the

---

Under review as a conference paper at ICLR 2020
(b)
Figure I:
An illustration of Thm. 1 The activations of the network in (a) are completely different from those of the network in (b), in which all skip connections bypassing layer are removed. However, the moments of the gradient norms at layer k = 2 are exactly the same in both (a) and (b).
norm of the gradient with respect to W is given by:
dc V C| = oy
OC Ty |||
| S
Large fluctuations to the Jacobian norm with respect to small perturbations of the weights will, therefore, cause wildly fluctuating gradients during training. hindering training dynamics.
Our analysis makes use of a duality theorem, which captures an intuitive concept. In residual architectures, there are many alternative paths of varying lengths from input to output. The theorem states that when considering the moments of the gradient at certain layer, one can change the architecture, so all connections which that skip that layer are removed. This is true, despite the activations of the two networks being very different. See Fig. 1
Our contributions are as follows: (i) We rigorously study the effect different architectures, namely vanilla, residual and densely connected architectures, on the mean and variance of Ily and || in finite depth and width networks. (ii) From our forward-backve
norm propagation duality theorem, derive conditions that prevent exploding gradients in each of these architectures. (iii) We prove that for densely connected networks, the commonly used initialization of et al. (2015)
is enough to ensure the boundness of ((]) and var(Ily'11 [I²) for arbitrarily deep networks, shedding new light on the apparent advantage of these architectures in practical applications. (iv) We prove that replacing the standard conv-ReLU blocks with blocks of the concatenated ReLU (CR) activation presented by Shang et al.] (2016). while maintaining the same parameter budget, significantly reduces the fluctuations of both the activations and gradients initialization for any ReLU based architecture. We evaluate the applicability of this simple alteration on large number of UCI datasets, and demonstrate the advantage for CR based networks over their standard counterparts.
RELATED WORK
The effect of architecture and depth on the gradients has been topic of considerable interest recently, leading to several insightful research directions. A major line of work has been focused on the mean field approach. where the propagation of signals in random networks are analyzed in the regime of infinite width (Yang & Schoenholz (2017) Schoenholz et al. (2017) [iaoo et al. (2018) 8). In that case, the pre-activations any layer are approximated by Gaussian distribution with moments depending on the specific layer. For ReI.U networks (without residual connections), controlling the exponential explosion or decay of both the activations and gradients with depth is a simple matter of correctly scaling the initialization parameters. Indeed, it is shown in (Schoenhllz et al. (2017) that classical feed forward networks exhibit exponential behavior on average, when not initialized properly. For the gradients, recursion relation is derived:
dy E
E
(
Oy (
& d'(v9"2)dz
(3)
By!
22T
where the derivative of the activation function and o² denotes the variance of the weights. By initializing with o2 have that Jan-sn' and considering that for the ReLU activation function d' = 1220.
2
Oy 2) Oy!
By"
E
E((
Dy. 1+1 )2).
2

---

Under review as a conference paper at ICLR 2020
Mean field analysis of residual networks is presented in Yang & Schoenholz 2017 where it is shown that ResNets exhibit sub-exponential, or even polynomial dynamics, depending on the activation function. While mean field theory captures the "average" dynamics of random neural networks, it does not take into account the scale of gradient fluctuations that are crucial to the stability of gradient descent, which is the focus of this work. more refined analysis presented in PPnnington al. (2117), considers the full spectrum of the input output Jacobian of infinitely wide networks, given by:
Qy" yy
J10
Exploding or vanishing gradients are then prevented by analyzing conditions that give rise to dy- namical isometry, a state which the squared eigenvalues of Jıo are all concentrated around In (Pennington et al. 2017) it is shown that fully connected ReLU networks are incapable of reaching dynamical isometry, as apposed to sigmoidal networks, while in (Pennington et al. 2018) it is shown that dynamical isometry is achieved in universal manner for a variety of activation functions. Dynamical isometry is a much stronger condition than the Jacobian norm, since it requires access to the entire spectrum. However, in previous work, this is done using various forms of infinite width approximations. Instead, this work focuses on finite depth corrections to the statistics of various quantities concerning the squared Frobenius norm of the per-layer Jacobian (derivative of the output with respect to the weights), and can be viewed as a complementary approach to analyzing the dynamics at the start of training. Our results cannot be obtained using large width approximations, since we seek to characterise the effect of both depth and width on the Jacobian.
A recent line of work, more related to the analysis presented in this paper, deals with the statistical properties of the norm of activations and gradients in random finite width networks. HHanin & Rolnick (2018) tackle two failure modes that are caused by exponential explosion or decay of the norm of intermediate layers. It is shown that for random fully connected vanilla ReLU networks, the variance of the squared norm of the activations exponentially increases, even when initializing with the Fan-in initialization. For ResNets, this failure mode can be overcome by correctly rescaling the residual branches. However, it is not clear how such a rescaling affects the back propagation of gradients. In our work, we prove an equivalence relation between the forward- and back-propagatton of norms in most random ReLU networks. In addition, we present analysis of DenseNets, revealing surprising norm preservation quality that is independent of depth.
Hanin 2018)
explores the conditions at initialization that give risc to the exploding vanishing gradient problem, by analyzing the individual entries of the input-output Jacobian. In the case of a single output network, the layer & to output Jacobian is given by the row vector:
Oyy Oy!
J(k + L) =
i= 1..mx)
oy
with ny denoting the width of layer k. In our work, we take a more direct approach to analyzing the full expression for the Jacobian of random networks given by:
0yy Oy* Sy Ow
0y* (J(k L),
Zw ij
Moreover, the analysis presented by Hanin 2018 does not directly carry over to ResNets or DenseNets.
3 PRELIMINARIES AND NOTATIONS
We make use of the following notations: N denotes a neural network architecture with intermediate outputs y of layer for fixed input y of denotes the i'th component of the vector y'. and m denote the width of the corresponding layers, with n the length of the input vector. || 1 1 is the squared Fuclidean norm, and we assume 012 1. We denote the weight matrix associated with layer by W R= kn; with lower case letters w/y denoting the individual components of W Additional superscripts with are used, when several weight matrices are associated with layer I, for instance in residual, CR and DenseNet architectures. Throughout the paper, we assume that the weights sampled i.i.d from some symmetric distribution with moments B((w(,)") d
3

---

Under review as a conference paper at ICLR 2020
20_. We further identify the following property of distribution A = 2(c2 3(c2))) (for Gaussian distributions, A = 0). The derivative of the output unit y with respect to w/y is given by J/j = Byy ay 0ny with || = Eiy(',y² We will be concerned in our analysis with the quantities E( B(11"(1?), var(Ilv/ll?, E(IJ"1) and var var(11-3 (II-1 where the expectations are taken across the random sampling of the weights, given the fixed input yº
such that Veg,, V,, ej 0, .4-1"
Jw dy' "wit"
Multi Pathway Architectures We compare vanilla connecco network to ResNets He ct al. (2016) and DenseNets Huang et al. (2016) to a network that is built with CR [Shang et al. (2016)
layers. For the standard vanilla fully-connected network, the output for of layer is given by
y = "(W' y²-)),
(8)
where @() denotes the ReLU non-linearity.
For residual architectures, we study models with m layer residual branches of the form
y' + Rm Rm (1'
(9)
where:
" WhnT (..6( o( y'-)) We denote the k'th intermediate output of the l'th residual branch by j WikT (W.2(T @(WITy 11-1),
(10)
(11)
so that, for example
Rm (g- W go g1-1.-11_ Wim @(wim- gf
(12)
Furthermore, since the last layer of any residual branch W does not end with a non- non-linearity, assume cdm = - DenseNets were recently introduced, demonstrating faster training, as well as improved performance on several popular datasets. The main architectual features introduced by DenseNets include the connection of each layer output to all subsequent layers, using concatenation operations instead of summation, such that the weights of layer / multiply the concatenation of the outputs y" ...3//-1 and is of dimensions W' E R For our analysis, we break W into 1-1 weight matrices Wil...Wil-1 E R=** so that the output of layer I is given by of & ( 1-1 -1= W y²")
4 FORWARD-BACKWAR
NORM PROPAGATION DUALITY
In this section, we introduce a link between the propagation of the norm of the activations, and the norm of the Jacobian in different layers in random ReLU networks of finite width. This link will then allow us to study the statistical properties of the gradient in general architectures incorporating residual connections and concatenations with relative ease. Specifically, we would like to establish a connection between the first and second moments of the squared norm of the output || I|2, and those of the Jacobian norm || | 1|2 ?. The family of architectures we consider restricted, so that the ReLU activations are only applied after multiplication with weights. Using path- path-based notation, any unit of the output g. can be decomposed to paths that go through weight matrix W*, denoted by vt (neglecting an additional superscript for ResNets and DenseNets), and paths that skip W denoted by of L..kk Assuming the input v,, 5 = 1. we have:
Itrel
hrd EII
L. wt = vl + Dl
L.K
ΣI KET.
2
(13)
1=1
R 1=1
where the summation is over paths from input to unit i of the output, indexed by nr. with Iral denoting the length of the path indexed by ?t- In non-residual networks, we have hyd L and Σn, sums over I1 n, paths. The term [ [ w Eyy, denotes the product of weights along path Yr multiplied by a binary variable 2y, E [0, 1], indicating whether path 7 is active. Finally, Engy indicates summation over all possible paths that do not include a weight from layer k. In all relevant architectures, the squared norm of the Jacobian entry || 112 is given by:
|| | 112
L. Fw
12
Σ ANNE
Wit
tea
Zays
2
(14)
1=1
4

---

Under review as a conference paper at ICLR 2020
That is, the derivative of output unit with respect to weight wij is given by the sum of all paths that contain wiy divided by why In order to link Jk with y", we make the following definition:
Definition 1. Reduced network: the oulputs of a reduced network N(k) denoted by -.kk) given input v are obtained by removing all connections bypassing weight W from the network N.
Note that for vanilla networks, holds that N(x)) -N, and VocigL, Fay y¹. The following
Lemma links between the moments of v/k) and those of y":
Lemma 1, For any architecture described in Sec. 3 it holds that for m E (2, 4):
(11v()))1*)
-B(1)"AA)
(15)
Note that if network N contains connections bypassing layer k, then for specific realization of the weights, the equality v yf.k does not hold in general, due different activation patterns in N and Nig induced by additional residual connections. In other words, paths that are open in A (indicated by the variables =y,) are not necessarily so in N(a),. and vice versa. Lem. I however, states that the moments of both are equal in the family of considered ReLU networks. The following theorem relates the moments of || with those of Ilv(k) Il:
Theorem 1. For any architecture described in Sec. 3 the following hold:
B (1NMIP))
1 1 (1ottt))))) Ci
var (lu S var(11>* var var((lyte)11) 11)
(16)
(ct)
where ct and of are the second and fourth moments weights in layer A: Thm. indicates that we can compute the moments of the Jacobian norm of layer k, by computing the moments of the output of the reduced network v'ey- Thm. also reveals a surprising property of the gradients in general Relu networks. That is, when the weights are sampled from the same distribution in each layer, the gradient's magnitude for each layer equal in expectation, and depend only the output statistics.
4.1 VANILLA NETWORKS
In order to compare our results for ResNets and DenseNets, we first derive the moments of s/e, in our setting for vanilla architectures. Note that for vanilla networks, the reduced network matches the original, and so Vo V(k) = =y'
Lemma 2. For a vanilla ReLU neural network, as described in Sec. 3 it holds for a 1:
L-1
- - C2 s var (11+4112) < C33 exp
5 |
C1 exp
C4
(17)
2n
n
where C1, C2, C3, C4 are constants that do not depend on L.
By setting a the variance of the Jacobian squared norm || will increase exponentially with E/= 5. ny as noted by Hanin & Rolnick 2018 This can be mitigated by linearly increasing the width of each layer with depth, so that 74 = nl, resulting in constant asymptotic behavior LA. This, however, would require significant resources, since the weight matrices would grow quadratically with depth. For comparison, in DenseNets the variance is maintained with depth, while the number of weights increases linearly.
exp -ーー1
4.2 RESIDUAL NETWORKS
Residual networks have reintroduced the concept of bypass connections, allowing the training of deep and narrow models with relative case. [Hnin & Rolnick (2018) showed shown that the fluctuations of the norm of subsequent layers do not exponentially increase, provided that the weights initialized,
5

---

Under review as a conference paper at ICLR 2020
such that the expected norm E(Ily'I?) is bounded for any layer l. Such an initialization is not constant, and is dependant on the depth of the network L. It is also worth mentioning that the same depth dependent initialization for ResNets was found by Zhang al. (2019) to be crucial for training convolutional ResNets without applying batchnorm, as introduced in the original formulation of ResNets. Recall that in our analysis, consider residual branches with m layers with weights
Wik'', 0 < k < L, 0 < k' < m. and so we will analyze the variance of || =²== Eis (auty
for k' < m. Unlike vanilla networks, the outputs of the reduced network 1/y8) for > 1 are not equal to those of the full network y', and are given by:
Rm Rm(y-111 1+ tz-
1 1
van {
(18)
Rm
1= k
The following theorem states our results for ResNets. Theorem 2. For a ResNet as in Sec, 3 with m 2 2, holds that for O < I, < W m:
Cip" exp
1
LP
C2p²² S var (1Ja s Cspp exp
2
Cap²
(19)
where P1 a ) C2, C3 C4 are constants that do not depend on L, m, and μ
L-1
am (1+am))
The gradients of weights in residual branches are, therefore, sensitive to the depth of each residual branch (large m). However, unlike their vanilla counterparts, by scaling the residual branches, such that Lp is kept constant, ensures that both [(I) and var([] remain bounded as the overall depth of the network grows (increasing L). Note that a similar initialization was proposed in Zhang jet al...)(01) (2019) for training ResNets without batchnorm, albeit for more restricted class of networks, and without an explicit dependency on width. Our result indicates that for narrow and deep networks with residual connections, training without batchnorm requires scaling in both depth and width dependent manner.
4.3 DENSELY CONNECTED NETWORKS
DenseNets have shown improvement over ResNets in terms of accuracy and speed of training on several notable image classification tasks, since their inception. As far as we can ascertain, there have been no theoretical studies on the causes of DenseNets success, or the implications on the training process of its architectural novelties, namely dense connections and feature reuse by concatenation of previous layers. the following, it is shown that the DenseNet architecture, coupled with the standard, depth independent 2_ initialization, gives rise to well-behaving gradients in any depth.
n
By the definition of DenseNets given in Sec.
the reduced architecture outputs B(A) are given by:
Φ E WU'T vf') W ve) 1> k
15 k
v(k) = {
(20)
d o 1'
The following theorem states our results for DenseNets: Theorem 3. For a DenseNet as described in Sec, it holds that for O < k I: L:
2|
(aa exp exp
C22 - war var S 12(a-1) ((((-111) (C exp
-Ca)
(21)
n
n
where C1 C2 Ca, C4 are constants that do not depend L..
It is worth noting the difference between DenseNets, and both vanilla and ResNet architectures. In vanilla architectures, the mean of the activation norm, as well as the gradient norm are exponential in a. However, the variance of the same quantities exponential in Σ n.. mean by correctly setting a = 1 does not prevent the gradients and activations from exploding in deep networks. In ResNets, scaling of the residual branches in order avoid exploding gradients
1. That is, fixing the

---

Under review as a conference paper at ICLR 2020
Empirical a=0.11
Empirical a
Empirical
Empirical a=10
Empirical a-0.3
Empirical a=1.5
Theory a-0.1
Theory a=0.5
Theory a-02
Theory a=1.0
Theory a-0.3
Theory
-2
20
35
45
50
15 20
25 30
40 45
50
(a)
(b)
Figure 2: The variance of the squared norm in log scale of the per layer Jacobian function of (held constant for all the layers) obtained from the simulated results of 200 independent runs, for (a) 40. The x-axis represents the depth
ResNet with m 2 and n 40, and (b) DenseNet with of the network, and y-axis the variance as computed from 200 runs. The dashed lines represent our theoretical predictions for both architectures, and full lines represent the empirical results. All networks were initialized using Gaussian distributions.
requires scaling in a depth dependent fashion. However, DenseNets, the variance of the squared Jacobian norm is in fact polynomial in depth for Gaussian distributions. Constant mean and variance achieved (at least when result is achieved with linear growth rate of the weight complexity, instead of quadratic with vanilla networks.
0) by setting a = 1, for any depth. Moreover, the constant variance
Empirical support We employed Monte Carlo simulations in order to verify theoretical
findings. As shown in Fig. 2 there is an excellent match between the simulations and our derivations.
THE CURIOUS CASE OF CONCATENATED RELU
Our results stated above demonstrate how different architectures and initialization schemes affect the training process the start of training, by analyzing the expected norm and variance of the gradients for each layer, and hold for ReLU based architectures. Since similar analysis can be performed on linear random networks, As it turns out, the ReLU function has a detrimental effect random networks, shown in the following lemma, effectively exacerbating the fluctuations both the activations and the gradients:
interesting to see the effect of the ReLU non-linearity on the outcome.
Lemma 3. Given vanilla linear neural network with wly NN0, _L),, then it holds that:
E (11b"112)
n2L n war (10=
-1) n 2) n n
(22)
Linear networks, of course, have limited capacity, considerably limiting their practical use. Fortu- nately,
the ReLU function to construct computation blocks that behave linear networks at the initialization stage, while not compromising in terms of capacity. This can be achieved by the of CR units. In concatenated ReLUs blocks, the input signal split into two complementary components x and x- such that the output of layer is parameterized by two weight matrices W W and is given by W d(y W12 oi ( An algebraically equivalent way to write the layer's computation is y = W/22 as variant of a residual connection. However, is important note that this equivalence holds for the forward-par -aass only, since deriving by 1 and 1 during the optimization process is not the same as performing an optimization that is based on derivation by Wi 1 and 12 The equivalence at initialization between the CR based architectures and random linear networks is depicted in the following Lemma:
(W"IT + w )o()²-1), which can be viewed
7

---

Under review as a conference paper at ICLR 2020
12
1.0
0.8
06
(a)
(b)
Figure 3: Activations norm per layer, as obtained from simulated results 5000 independent runs. (a) The mean of the squared norm of the activations in each layer (y-axis) function of depth (x-axis) for randomly initialized networks. The red plot for a ReLU network, the blue for CR based network. (b) the y-axis is the standard deviation of the squared norm of the activation. In addition to the blue and red plots, the black and green plots are the theoretical predictions according Lem. 23
Lemma 4. The distribution of the output y/ = W ((r-~1 W12T ó0y' conditioned on y where wiJ. w!} ~ (O, ) equals the distribution of W y where the elements of W are distributed according to an i.i.d normal distribution wt, NOO, m
Using CR, therefore, provides initial conditions identical to random linear networks in terms of the output statistics. However, this equivalence also holds for back propagation. Indeed, from the By )2
construction of W in the lemma, it is easy to see that Du )² =
0g'
) A surprising
Dw!!
property emerges. Lem. shows that similar to vanilla ReLU based networks, CR based ReLU networks have exponential behavior in Σ/= 2. Since CR employ twice as many parameters, then nu for the same budget of parameters (having 74 my for CR) we have Et= 2v2 < 5. hence more efficient than standard ReLU blocks. We can similarly define residual and densely connected architecture comprised of concatenated ReLU blocks. In that case, the conditions of both architectures in terms of output distributions initialization will be identical the linear counterparts of both architectures, given by y W ((gLk-1) W ((- k-1) and y' Ei-1 WIl' "y for ResNets and DenseNets, respectively.
v2
Further evidence for the practical effectiveness of CR layers is provided in the appendix, in which we present experiments the UCI datasets.
CONCLUSIONS
The interplay between architecture and initialization determines much of the early dynamics of training deep networks. In this work, we have demonstrated how careful initialization general residual architectures effects the gradients at the start of training, by rigorous study of the variance of the per layer gradient norm at the point of initialization. Through the use of the duality principle between forward and backward propagation, we have derived the expression for the variance in three architectures: standard networks, ResNets and DenseNets. We show that DenseNets benefit from a simple variance preserving initialization, which is independent of depth, while depth dependent initialization exists for ResNet. We have also shown that CR blocks perform better then ReLU blocks, preservation in deep layers. As future work,
given fixed budget of weights, when it comes we would like to extend our analysis to include normalization techniques, such as batchnorm.
REFERENCES
Ajnrk-Aréé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 151.0225, 2015.
8

---

Under review as a conference paper at ICLR 2020
Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? In S. Bengio. H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Dasa-Bianchi,, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 582-5982119999 2018.
Boris Hanin and David Rolnick. How to start training:
The effect of initialization and architecture. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 571-581. 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Gao Huang, Zhuang Liu, and Kilian Q. Weinberger. Densely connected convolutional networks. In arXiv: /608: 06993, 2016.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in Neurul Information Processing Systems, pp. 972-981, 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Curran Associates, Inc., 2017.
Processing Systems 30, pp-
Jeffrey Pennington, Samuel S. Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In AISTATS, 2018.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In /CLR, 2017.
Wenling Shang, Kihyuk Sohn, Diogo Almeida, and Honglak Lee. Understanding and improving convolutional neural networks via concatenated rectified linear units. In ICML, pp. 2217-2225, 2016.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla con- volutional neural networks. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5393-5402, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Greg Yang and Sam Schoenholz. Mean field residual networks: On the edge of chaos. 2017.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In ICLR, 2019.
PROOFS
We make use the following propositions and definitions in order to prove lemma and theorem 1. For ResNet, we assume = 2 for notation simplicity. The results hold for m >> 2 as well with no additional arguments.
A path 74 from input to output unit defines product of weights along the path denoted by
hall
Ixd
Q,
II
9
(23)
where Iyu| is the length of path % (for vanilla and DenseNets hyl L), &yy is binary variable 2y, E {0, indicating if the path is "active" (i.e, all relevant Relu activations along the path are open), and ,,, are the weights along path 7h and Fr,1 1

---

Under review as a conference paper at ICLR 2020
We generalize the above expression the following way:
1.
Q
and
(24)
where:
E " else
(25)
where $ " indicates if layer I is skipped, Ere the activation variable relevant for weight w That is, if dd = wo, then 2ny = 2 where 2} E {0, 1} is the activation variable unit layer I. Wee denotes generalized weight that can take the value of weight Wy,d Woyd for vanilla layers, or a product of weights T Wyec W for residual layers (Wyr E Will and Whols E W proposition 1. Given random vector w [WI...W]] such that each component is identically and symmetrically distributed iid random variable with moments E(w:") Cms and random binary variable such that z/wo E (0, z/w = 2| then it holds that for > l:
0
m is odd, is even
E
2)
iem
m is even
(26)
2
B
1==
wifh
else
Proof. We have:
/
/ II w p(w)dw
II
w p(w)dw
(27)
1l|==
00|==0
- (w))d + /
1
II (--w;) (i-w:)"((wddr
(28)
uel 2m
ul/am i=1
w zp( zp( ddw / II (-w:) w. ""wp):""""'wd
(29)
2=1
==1
For even m I: it follows that:
c
2
II
u zp(u dd
(30)
1=]
and so:
d
cm
E
W 2)
(31)
2
For odd I, since w is symmetrically distributed, we have that ei, = 0, and so E(II wx"2)
proposition 2. For any paths 7t, 72 it holds that:
0 R(QO))) IT ((())1)))
7r + 72
B(Qx:
(32)
gr 72
Proof. From proposition 2 we have:
L E(Q, = E
9 2....
(33)
10

---

Under review as a conference paper at ICLR 2020
Since the computations done by all considered architectures form markov chain, such that the output of layer y depends only on weights R' = {W' ...W"), using conditional expectation.:
B(Q - (00==(((000000 (E( R(-1)) (1 IE
9 0.g 1. 1.9g. n
(34)
1=1
Note that since both are paths leading to output unit t, the following holds:
E( 10 B B(
LE nt. Lt.Leyg L LE dET. 7 I T.,
Blg 1922.
R²
(35)
T ((0
L
and so it follows from proposition 1:
B(w xt
LE LE LE E =W,eL
Flg//119 Rd 1²
1
L $ n. L n else
= 1 ((gg),)))
(36)
Recursively going through I = 1...1 completes the proof.
proposition 3. For any set of paths Tt ri: 7/. n2, from input to output units and it holds that:
II E (En23.1
(37)
such that:
1 Elg 12
4/ 4,2,11 0.3.1 9-3.1
E
4 = 4-3.d 9,3.1 9-3
(38)
else
Proof.
The proof is practically identical to proposition 3, and will be omitted.
For notation simplicity,
7.1 PROOF OF LEMMA
The proof of lemma 1 stems from the following observation: Since the computations done by all considered architectures form markov chain, such that the output of layer y' depend only weights W'...'' then for any particular realization of the weights, y². and y differ only their activation patterns in layers 2 &o due to the removal of bypassing connections in yk(k). Using Eq. 23
Ixd
E [10 (1mm))))
B
II
Wey a Erys )m)
(39)
1=1
E=1
while:
Wey Woy 2yy
(40)
t=1
kE
Iml
We must therefor prove the expectation is invariant to switch between 2y, and Lyy for layers > k. For m 2, using proposition 2, the expectation factorizes:
E (11))
Σ
Σ
E(Q'))
Σ
II
E (().).)²)
(41)
T MEE
1=1
KK
1=1
11

---

Under review as a conference paper at ICLR 2020
Similarly for 4, from proposition the expectation factorizes:
E (11of
Σ Σ
H (E128.)
(42)
where Syg,u is defined is Eq.
Note that switching the sign of W for all architectures considered will switch the state of both zj. < j S ny and 2,,0 < S ny, and so from proposition 1, the factorized expectations for m 2 and m 4 meet the required invariance.
38
7.2
PROOF OF THEOREM
From the definition the Jacobian for weights W we have using lemma and proposition
auti Ow
1(1))111) = IE(1
11
II 1=1
t= ink
E
I
(43)
(w
1=1
Since by definition w E 7Ł. from proposition 2 the term
I by factorizes:
(wo
L
22
E
II (22))
EE
II
B((Z,,)
II
(44)
k
(wtk )2
(wr )2
E(2t)
E(o
E
II
E(P E
(45)
1ER,1=1
1=1
where the last transition stems from proposition 1. Plugging back into Eq. 43 and summing over ij completes the proof for 2. For m = 4, we have:
Oyli Fwk 1)
Oyte Sui
B(I
E
||
11a)
Σ
E
II 1=1
(46)
(w 2
(wt t
ijkeni ot n2
From proposition 3, for wok
N/ Te ijk 7t: 78 the expectation factorizes:
S (
L II Iy
II (x
E
E
(47)
(w
(wott
(48)
where from proposition and proposition 3, for wok E 7/.723 ijki E ex, np:
1(5.12..)
{u, v # (i.s} {u, v) = (i.j}
(c
E
(w (w
Bis 12
(49)
Note that since oj 2 (c))², it holds that:
B 11 w
E
B(E
E
<
(50)
(no
Puck
(ct)
12

---

Under review as a conference paper at ICLR 2020
Plugging into Eq. 46 it follows:
var var(Ilutt Iia)
(ay((l 11²)
war(11 [([])=1|+)) 1(113+133
(51)
(ch)
Furthermore, it holds that:
L II
5 k (w
-(1}~119))
Σ
Σ
(f12).) (52)
(w (
ijkEt! of uuken!
- -Σ Σ
E
)2
(1,0)= + [(.5))
II bEk,1=1
E (frt3.)) (53)
at in unkEyling
Therefor for large width networks:
S ( " )2
111
E
Σ
Σ
I (u.upAtEs II
E
(54)
tzkEnt of nokenting
Σ
Σ
Σ
E (f (u,o)+(1it
(
(55)
(ct)
st sjkEtt of w
E
(ll 11*)
(56)
(c
(c))2
and so for large width networks:
var( var(1y(()) 113)
11r ~
(57)
(c))2
7.3 PROOF OF LEMMA
Denoting by Z' the diagonal matrix holding in its diagonal the activation variables æf for unit j in layer and so we have:
y z W'y'
(58)
Conditioning on R! (W) TW...Wl-11 and taking expectation:
T.
1 E
( 1. ) yh
(59)
nI-11
1/ 1-1 1
1 E Whi wts 186-1)
(60)
y=1 11.12=1
From proposition 1, it follows that: E ((11~11112211)-
n- E (118--1122)
(61)
nL-1
Recursing through 1...:
E ((110112) = n
(62)
Similarly:
E (11 E. WLZLWLT yl-
(63)
Σ
Vi Ui U
1- L-1 --1 1 E (w whys wh wt
21 Lf 2/2 126--1)
(64)
13

---

Under review as a conference paper at ICLR 2020
From proposition 3:
1
(wh whes w W w Wh (i 212 2 112-11)
2 (65)
E (wt w whisr w whani whese 2 2/2 (/1(R
(1) 3
Fig...
(66)
+ +11 -1 22.
(67)
and so:
(11x111*)
m2c
E 11)
6nt
E
(w/ 1 (01-))
(68)
2
nz
+ n (1.
1)
E
(v/
1- (11-1)2(6 (3 122)
(69)
n2
$1.22
nz(c2 3(c2)))
+
nr (nz +
E (11)4~1111)
(70)
2
nř.-1
n1. A
1o)) +
nr. (nz + +5)
E
(11)-=11(*)
(1
(71)
2
nř.-1
1-1
finally:
(n +5 A)
114) < EE (11*"11*)
(72)
ni_-
(n + 5 + | E (ل
(73)
n²_-1
Recursing through 1...,, and denoting C =
+2(m++5-1A1) n2
and C2
Re. (mg++++|1)
and
n2a
combining with Eq.
62
C
L-1
5
|A
n n
S var (In^112) S C2
L-1
(1+
5+ A
n
(74)
T1
n2
1=1
1=1
Using V12-2-1, 1 U.5x
< + x) S :
Σ 3 |
5-1
L
5 A
C1 exp
C2 S var (100112) S Cs exp
Σ
(75)
2m
For Gaussian distribution we have that A = 0, and So:
var (1==112)
nz (nz + 5)
2)
n² n2
(76)
n2
711
1=1
7.4 PROOF OF THEOREM 2
In the following proof, for the sake of notation simplicity, we omit the notation & in s'ay. and assume that y stands for the reduced network ukky- We have for layer L:
E (1w='112 = E (Ib0- 112) + E --.m IT W Im www.w.oom I.m yf 1.m))
E (1100- 112) + aB - E (110--112) (1 + am)) =E(II+11))
(1+a))
II (1+a")) E (I1x II (1+am)an am
II 1+ am) a am (77) 14k
14

---

Under review as a conference paper at ICLR 2020
B (110111) = E (111 (10 + E (mp)) + 4B
1. y yd -12
+ 2 2 1100--11891--11111 (78)
We now handle each term separately:
E |1 (185-111
1
(+)
(1 (1(((1)4-)) 11
E
(E
(79)
Similarly to the vanilla case with a linear weight layer on top, we have:
2m (1 + 5
|
11 |1 s E 11""~1.""11111""
(80)
7
5 + | ym 11
a 2m (1 +
(81)
E ( g(-1)3))
E
E
Ui
-- 1.m
V12
1m-1.m- 1.m -1
whis Wizja
(82)
a E
1111
_
a
E (1y"~111
(83)
n
n
and:
E
[1 1m
m 12 (10)
112) a= (115 11
(84)
Plugging it all into Eq.
78 and denoting:
2 n2
+ |
3/ = 1 20a
+ a 2m (1+ +
jm
(85)
2
5 - | ym-
ST = + 2a (1 +
+
2m (1 +
(86)
we have after recursing through = L I...:
II 1=k+1
ST S (( s <((I"1))
II
8t
(87)
L-k+1
Since for the reduced architecture siky. the transformation between y² and n² is given by an m layer fully connected network without any residual connections, we can the results from the vanilla case:
L II
5 12 1-1 & E (10+111 S II
L 3 2m (1 Iift
5+ | ym
3T 2m (1 1 1y
(88)
n
Denoting Pt = a² + b+ it follows:
L
L < II
C I 3 PTh S E |100~11
g pT
(89)
1yk
From the definition of 8 it holds for p+ < 1:
B < (1 + p+
z
exp
2p4 2
(90)
and:
8T > (1 + pm)) 2 exp
-0
(91)
and so:
CipTH exp
1 _T
-C2p² - var (1x4112) s Capf exp
2T - -Cyp1
(92)
15

---

Under review as a conference paper at ICLR 2020
For Gaussian distributions, using similar derivation and assigning A we have:
E 1 (185112)
II P
(93)
where:
== 20 + 2
2
2m (1 + n
m
2 2 1 (1
(94)
n
n
and:
p == a 2m (1 - m (1 3 + 2
(95)
n
7.5 PROOF OF THEOREM 3
In the following proof, for the sake of notation simplicity, we omit the notation is in Bkk)= and assume that y stands for the reduced network B(ky We have:
L-1
1-1 2d
L-1 al
Te (110112) = PL = 1E(
:E(1
W
Σ of
Σ
14-1
(96)
L
1=
1=
t=k
4-1 E (1b02110) = -B (B
(v W 2
Σ
(0 (ol
1=
1=
L-1 E0d4T Whll)) (97)
= = (
Σ Whll)
Σ
WylsTwbla 1z
15
Note that the above is a simple Relu block with concatenated yy..// as input. Using the results from the vanilla architecture, and denoting Ct,-- = E (11""12110"
then follows:
(n + 2
L-1
(n + 5 D
L-1
Σ d
Ch I CL.L
Σ
Ch
(98)
nL?
nL?
11
From Eq. 98 it holds that:
n(L 1)2
T-2
n(L 1)2
CL-1L-15 S
Σ
C S
CL-1L-11
(99)
n 5 + A
n n
h,d2=
then follows:
a
5+ |
L-1 Σ
B(1lv*11t) = CL,L =
L7 (1+
Cuu
1112==
a²
(1
5 5+
(C1-1,1-1+
Σ
L-2
Cht +2 Σ
L-2
CL-1.)
n
1111==
1=k
2
5 |A||
(L - 1
To-2
(CI-1,6-1
CL-1,1-1
+2
Σ
CL-11)
(100)
n
07 (
Al)
1=k
(101)
The following also holds:
4-1 E
4-11
Viz222 Chiba
y
Wh
= Σ
C
(102)
4
1=
1=k
16

---

Under review as a conference paper at ICLR 2020
and so:
(n+ 5 |
(L- 11=1)3n
CL. I
(CI-1,1-1+
(103)
nL?
an +5
[
L-2 L-2
+
2a
Σ
Chukks
(104)
L
1
1n+5+121)o*
(L. 1)
(CL-1.1-1+
a² |A]) C1-1,4-1
(105)
nL?
2n(L 1)
a(n + 5 AA] CL-1.L-1)
(106)
(n + 1-55+10 1
C2-1.-1111 (1+ a²(n + |)
(I-
2n [, -011 1)
(107)
nL?
a(n + |
(m+5-14) (1%
(++5+
a(L -(L-1)) a² |AI)
-1--1,
(108)
La
nT.
CL, S C1-1LL-1 (1+
(1
||
((1
a -
a2 -111
(109)
n 5
A
L
nI?
= =CL-1,L-1 11
2/
nLL + 1)2 )
a² - ||
(1+
a
12
(110)
2 5
L
Telescoping through 1 = L - 1...1:
2/
L- k+1
L
(2(5-11))
[+ a -))
Cr.I S 1
(111)
n n A
nlI+a - 1)3
2| 1)
1
2(a - 1)
< exp
exp
exp
(112)
n n 5 |
nil + a 1)2)
LAE
2 ( - 1)
a.2(5 |
exp
exp
(a-
(113)
+ 5 |
nL
2
exp
I,2(a-1)
(114)
n
similar relation holds for the mean:
T--1
I, (11-1E11+ (1-1) 111 u )
a
a
(115)
L
1=k
2
(1 a 1-a1-1 1)2
< (12(a-1)
(116)
1,1-1
L
Finally:
var (1r((1=4113)
CL,L 222
(117)
1 11) (Ci exp
2
(118)
lower bound can easily be derived using Eq. 98 99 and follow the same derivation by flipping the sign of AA:
- - -C2) 5 var((1y < L 2/ (C's (C3exp exp
2|A|L - -Ca) (119)
L (1-1)(C exp
17

---

Under review as a conference paper at ICLR 2020
For Gaussian distributions, using similar derivation and assigning A
we have:
CL,
II ((++
a
a
)
n
a²
(120)
nll + all+a-11 1)2
17
II 1#k
1
a -
a
(121)
12
and so:
var (11(11=411))
(00
(aa2(
+ 5)
II
L
1
5
1)2)
a²
II 1yck
1 a -
1
(122)
nk?
n(1 +
1.2
1
7.6
PROOF OF LEMMA
The proof is identical the Relu case, removing the activation variables Z and hence will be omitted.
7.7
PROOF OF LEMMA 4
Proof. We construct the matrix W' as follows:
2 0 y'- 0
(123)
W
And so y' = W'T w' Note that the elements of W are i.i.d Gaussian random variables w/ (O, m-11
8 EMPIRICAL SUPPORT FOR CONCATENATIVE RELU
Our theoretical results are verified in Fig. 3 As can be seen, the simulations closely match the predictions. The advantage CR over conventional ReL.U networks also clearly visible.
CR has fallen out of flavor and was never shown to be effective outside CNNs. We, therefore, provide empirical support for its effectiveness, which our analysis above points. our experiments, we evaluate the performance of fully connected networks the 40 UCI datasets with more than 1000 samples. The train/test splits were provided by Klambauer al. (2017). In the experiment, we tested three different depths: 8, 16. and 32. Each layer contained employed. learning rate of either 0.1, 0.01, or 0.001 was used for the first 50 epochs and then a learning rate of one tenth for additional 50 epochs. All runs terminated after 100 epochs. Batches were of size
256 hidden neurons. Dropout was not
50.
Following Klambauer et al. (2017). averaging operator with mask size of 5 applied the validation error, and the epoch with the best smoothed validation error was selected. This was done separately for each method, each dataset, and each split. Out of the nine options of depth and learning rate for most methods, the mean validation error across the four splits was used to select the option for which we report the results on the test splits. We compared the following activation functions and architectures: vanilla neural networks (single pathway) with either SELU (Klambauer et al. 2017) or ELU (Clevert et al. 2015) activation functions, ResNets, and CR based architecture. The single pathway ReLU networks were not competitive and the experiments with this architecture were terminated to allow for more experiments with the other architectures. The single pathway experiments were done without batchnorm, is usually done with ELU and SELU. For ResNets, we report results both with batchnorm and without. Batchnorm was applied along the residual pathway, twice, is typically practiced. The first and last layers of the ResNet and CR architecture, i.e., the layers that project from the input and to the output layer, are conventional projections that do not employ multiple pathways. For the baseline architectures, also evaluate using double width, i.e.,
18

---

Under review as a conference paper at ICLR 2020
1
0.9
0.8
0.7
0.6
0.5
ELU
0.4
ELU double width
SELU
0.3
SELU double width
ResNet
0.2
ResNet double width
Resnet w/o bn
0.1
Resnet w/o bn double width
CR
0
0.99
0.98
0.97
0.96
0.95
Figure 4: A Dolan-More profile, based on the accuracy obtained for the UCI experiments. The x-axis the threshold (r). Note that since for accuracy scores, higher is better, whereas typical Dolan-More plots show cost, the axis ordered in reverse. The y y-axis depicts, for given architecture out of the nine, the ratio of datasets in which the obtained accuracy is above the threshold times the maximal accuracy obtained by any architecture.
19

---

Under review as a conference paper at ICLR 2020
Table I: Error rates for each method per dataset. 512 denote the double width settings.
CR
ELU
ELU 512 SELU SELU 512
RES
RES 512
RES
RES 512
w/o bn
w/o bn
semeion
0.102
0.1357
0.118
0.1543
0.1356
0.1192
0.1215
0.121
0.1308
oocytes..._2f 2f
0.0941 0.0918
0.0949
0.0938
0.0892
0.0897
0.0921
0.0944
0.1084
wine-quality-whit white
0.4532 0.4633
0.4488
0.4597
0.4547
0.4521
0.4552
0.4573
0.4645
ringnorm
0.0208 0.0253
0.0251
0,026
0.0222
0.0261
0.0220
0.0205
0.0341
statlog-image
0.0448 0.0483
0.0472
0.042
0.0495
0.0467
0.0639
0.0489
0.0382
cardiotocography--10
0,2090 0.2207
0.212
0.2222
0.2073
0.2147
0.2134
0.2126
0.2141
abalone
0.3408 0.3439
0.3391
0.3476
0.3492
0.3452
0.3479
0.3464
0.344
ozone
0.0295 0.0300
0.0287
0.0339
0.0288
0,0287
0.0312
0.0288
0.0321
thyroid
0.0207 0.0179
0.0183
0.0211
0.0198
0.0192
0.0186
0.018
0.0247
oocyte... 4d
0.1855 0.1946
0.194
0.2052
0.2076
0.2021
0.2006
0.2047
0.2156
adult
0.1515 0.1539
0.1530
0.1559
0.1550
0.1515
0.1509
0.1507
0.1538
chess-krvkp
0.0215 0.0333
0.0257
0.0307
0.0255
0.0205
0.0262
0.0275
0.0178
waveform
0.1403 0.1419
0.1441
0.1468
0.1455
0.1355
0.1399
0.1377
0.1656
plant-texture
0,2614 0.3412
0.2991
0.3022
0.2777
0,2814
0.2704
0.2738
0.3296
statlog-landsat
0.1061 0.1025
0.1011
0.1011
0.0987
0.1053
0.1111
0.1069
0.0960
steel-plates
0,275 0.2933
0.2815
0.2832
0.2776
0.2961
0.2879
0.2881
0.2926
statlog-vehice vehicle
0.2188 0.2115
0.2159
0.2213
0.2233
0.2198
0.2285
0.2164
0.2075
cardiotocograahyy-
0.0967 0.1057
0.1022
0.1023
0.1008
0.0988
0.1053
0.0994
0.1055
page-blocks
0.0324 0.0367
0.0345
0.0367
0.0355
0.0356
0.0343
0.0376
0.0367
wall-following
0.1131 0.1232
0.1272
0.1269
0.1235
0.1204
0.1236
0.1190
0.1192
miniboone
0.0793 0.0751
0.0750
0.0790
0.0790
0.0849
0.0889
0.0772
0.2481
statlog-shuttle
0.0016 0.002
0.0017
0.0015
0.0015
0.0025
0.0025
0.0022
0.0017
mushroom
0.0004 0.0004
0.0003
0.0010
0.0003
0.0010
0.0008
0.0007
0.0001
waveform-noise
0.1464 0.1501
0.1520
0.1626
0.1576
0.1437
0.1421
0.7090
0.1705
twonorm
0,0231 0.027
0.0252
0.0261
0.0258
0.0236
0.0243
0.0241
0.0341
connect-4
0.1442 0.1508
0.1493
0.1483
0.1442
0,1423
0.1431
0.1424
0.1628
letter
0.0488 0.0700
0.0650
0.0606
0.0583
0.0880
0.0778
0.0818
0.0485
titanic
0.2197 0.2148
0.2151
0.2170
0.2118
0.2149
0.2181
0.212
0.2149
statlog-hear
0.1844 0.1939
0.1808
0.2093
0.2113
0.1967
0.2016
0.1794
0.2276
plant-margin
0.2798 0.3330
0.2835
0.3258
0.2918
0.2772
0.2764
0.2951
0.3749
contrac
0.4676 0.4757
0.4716
0.4816
0.4852
0.4606
0.4622
0.4693
0.4889
yeast
0.4239 0.4205
0.4252
0.4333
0.4253
0.4141
0.4199
0.4431
0.4649
bank
0.1121 0.1114
0.1131
0.1180
0.1170
0.1084
0.1161
0.1099
0.1143
hill-valley
0.3667 0.3699
0.3774
0.3700
0.3554
0.3899
0.3657
0.3765
0.4062
optical
0.0299 0.0456
0.0408
0.0416
0.0417
0.0477
0.0475
0.0426
0.0366
plant-shape
0.4127 0.5439
0.4862
0.4300
0.4220
0.4593
0.4581
0.4509
0.4228
led-display
0.2906 0.2929
0.2827
0.2911
0.3060
0.2852
0.2856
0.2781
0.3093
magic
0. 1293 0.1310
0.1331
0.1345
0.1344
0.1375
0.1348
0.1329
0.1332
pendigits
0.0388 0.0446
0.0434
0.0426
0,0432
0,0466
0.0483
0.0467
0.0439
chess-krvk
0.2452 0.2592
0.2426
0.2399
0.2284
0.2956
0.2848
0.2741
0.3289
each layer had 512 hidden neurons. This was done since CR employs weights than the baselines due to the dual pathway architecture.
provide the Dolan-More profile of the methods. In this plot, the x-axis is threshold 7 and the y-axis the ratio, out all datasets, in which the method has reached an accuracy that times the best accuracy for the dataset. It can be seen that the CR method dominates across all threshold values, except for SELU double width at low values of r.
In Fig. 4
9
UCI RESULTS
In Tab. I we provide the full numbers of the UCI experiments in the paper.
20


Reviews:
[{"review_id": "Hyxhc4lAFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.\n\nThe main theoretical results stem from a norm propagation duality that allows computations of certain statistics using simplified architectures. As far as I know, this is a novel way of analyzing multi-pathway architectures, and the method and the implied results will be of interest to deep-learning theorists.\n\nThere are two types of empirical results, one to verify the theoretical predictions, and the other to see if those predictions translate into improved generalization performance on some real-world tasks. Figures 2 and 3 show good agreement with theory. I am less convinced by Fig 4. Unless I missed something, I thought the implication from the theoretical analysis was that multi-pathway models should behave better than single-pathway models at large depth, and for single-pathway models concatenated ReLU may have slightly better large-depth behavior than standard ReLU. Doesn't this suggest that for large depth DenseNet ~= ResNet > CR > ReLU? Is the good performance of CR supposed to be understood as supporting evidence for the theoretical analysis? Or is the point more like \"the theory suggests CR might be interesting, and indeed it performs well, but we leave the details of generalization performance to a future study\"? It would be important to specify the desired interpretation here.\n\nBut I think regardless of the interpretation, the empirical results are not strong enough to stand on their own as support for CR, especially since this type of architecture has been introduced and studied previously (c.f. also the looks-linear setup from [1]). Therefore my evaluation is mostly based on the theoretical contributions, which I think are themselves are probably sufficient for publication.\n\nOne main area for improvement is in the related work. A significant line of work is omitted that studies higher-order statistics of Jacobian matrices in the large-depth, infinite-width regime [2-8], and while this is at infinite width, it work for any non-linearity, so is an important point for comparison. Other kinds of initialization schemes are also relevant for comparison, e.g. [9-10].\n\n[1] Balduzzi, David, et al. \"The shattered gradients problem: If resnets are the answer, then what is the question?.\" Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.\n[2] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. \"Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice.\" Advances in neural information processing systems. 2017.\n[3] Pennington, Jeffrey, Samuel Schoenholz, and Surya Ganguli. \"The emergence of spectral universality in deep networks.\" International Conference on Artificial Intelligence and Statistics. 2018.\n[4] Hayase, Tomohiro. \"Almost Surely Asymptotic Freeness for Jacobian Spectrum of Deep Network.\" arXiv preprint arXiv:1908.03901 (2019).\n[5] Tarnowski, Wojciech, et al. \"Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Function.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.\n[6] Burkholz, Rebekka, and Alina Dubatovka. \"Initialization of ReLUs for Dynamical Isometry.\" arXiv preprint arXiv:1806.06362 (2018).\n[7] Xiao, Lechao, et al. \"Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks.\" International Conference on Machine Learning. 2018.\n[8] Chen, Minmin, Jeffrey Pennington, and Samuel Schoenholz. \"Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks.\" International Conference on Machine Learning. 2018.\n[9]Sutskever, Ilya, et al. \"On the importance of initialization and momentum in deep learning.\" International conference on machine learning. 2013.\n[10] Le, Quoc V., Navdeep Jaitly, and Geoffrey E. Hinton. \"A simple way to initialize recurrent networks of rectified linear units.\" arXiv preprint arXiv:1504.00941 (2015).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylIU-_KFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the mean and variance of the gradient norm at each layer for vanilla feedforward, ResNet and DenseNet, respectively, at the initialization step, which is related with Hanin & Ronick 2018 studying the mean and variance of forward activations. They show that ResNet and DenseNet preserve the variance of the layer gradient norm through depths. In comparison, for the vanilla feedforward network, although the mean of the gradient norm is preserved if  is properly initialized, the variance of the layer gradient norm increases over depths, which may explode or decay the gradient at deeper layers. \n\nThe result presented in the paper is interesting, and the theory and empirical verification have a good match. I  recommend the acceptance after the following points are well addressed.\n\n1. The mean and variance of layer activation norm and gradient norm have been studied in the mean field literatures for example [1] and the paper does not have a good comparison with them.\n2. The experiment of  CONCATENATIVE RELU is not convincing given the small tasks intertwined performance.\n\nMinor presentation flaws:\n1. The sentence in Abstract \"This depth invariant result is surprising in light of the literature results that state that the norm of the layer’s activations grows exponentially with the specific layer’s depth.\" is not clear itself because of no relevant reference.\n2. There are many typos such as \"weather\", \"for of\" in the paper. Please carefully correct them.\n3. The values on the y-axis of Figure 2 and Figure3 are not correctly shown.\n\n[1] Greg Yang and Samuel Schoenholz 2017. Mean Field Residual Networks: On the Edge of Chaos\n\n\n#####after rebuttal period, I found a fatal error in the paper#####\n\nIn (13), the network output $y_t^L$ is decomposed into two parts $y^{L,k}_t$ and $\\hat{y}^{L,k}_t$, where $y^{L,k}_t$ is composed of paths that go through weight matrix $W^k$ and $\\hat{y}^{L,k}_t$ is composed of paths that skip weight matrix $W^k$.\n\nIn (14), the paper derives $J^k:=\\frac{\\partial y_t^L}{\\partial W^k} = \\frac{\\partial y^{L,k}_t}{\\partial W^k}$. However, in fact the other part $\\hat{y}^{L,k}_t$ also has gradient with respect to  $W^k$ even if the path $\\gamma_t$ does not go through $W^k$ because the activation $z_{\\gamma_t}$ is a function of  $W^k$ .\n\nWith this error, all the following claims in the paper are wrong. Thus I vote to reject this paper.  \n\nOne can do experiment to verify  that \"Thm. 1 also reveals a surprising property of the gradients in general Relu networks. That is, when the weights are sampled from the same distribution in each layer, the gradient’s magnitude for each layer are equal in expectation, and depend only on the output statistics.\" is wrong for ResNet.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJl_j5w8tr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the effects of residual and dense net type connections on the moments of per-layer gradients at random initialization. In particular, using duality, bounds on the variance of the square norm of Jacobian (with respect to the randomness of random initialization) are derived for vanilla networks. By noticing that connections that bypass a specific layer does not affect the expected square norm of the gradients of a particular layer, the paper continue to characterize the bounds for residual networks and densely connected networks that have special skip connections. In particular, with properly chosen initialization scales for each layer, the architectures with skip connections can be initialized so that the gradient norm does not explode with increasing depth.\n\nIn the second part of the paper, the topic shifted to a bit unrelated topic, comparing the effects of ReLU vs concatenated ReLU blocks on the norm of layer activations.\n\nThe results in this paper seem to be useful for guidance on choosing proper initialization schemes for different architectures at different depths. I hope the paper could spend more efforts discussing the potential impact of their results. Moreover, some preliminary empirical studies could also help:\n\n- how well do the bounds still hold after the optimization starts to run?\n- does the proposed initialization scheme actually allow training of very deep neural networks?\n\nMinor comments:\n\n- please add x and y labels to the plots\n- please add grid (as in Fig 4) to Fig 2 and 3. Currently it is especially hard to decide the slope for curves in Fig 2b.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the statistics of activation norms and Jacobian norms for randomly-initialized ReLU networks in the presence (and absence) of various types of residual connections. Whereas the variance of the gradient norm grows with depth for vanilla networks, it can be depth-independent for residual networks when using the proper initialization.

Reviewers were positive about the setup, but also pointed out important shortcomings on the current manuscript, especially related to the lack of significance of the measured gradient norm statistics with regards to generalisation, and with some techinical aspects of the derivations. For these reasons, the AC believes this paper will strongly benefit from an extra iteration. 
Decision: Reject