The Benefits of Over-parameterization at Initialization in Deep ReLU Networks
Authors: Devansh Arpit
Yoshua Bengio
Venue:  (2020)
Keywords: deep relu networks
he initialization
norm preserving
gradient preserving

Abstract: It has been noted in existing literature that over-parameterization in ReLU networks generally improves performance. While there could be several factors involved behind this, we prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, it is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information in both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal to the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to be larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for a finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks at initialization.

Under review as a conference paper at ICLR 2020
THE BENEFITS OF OVER-PARAMETERIIZAION AT INI- TIALIZATION IN DEEP RELU NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
It has been noted in existing literature that paraaatteeeezaiionnnn in ReLU networks generally improves performance. While there could be several factors involved behind this, prove some desirable theoretical properties at initialization which may be enjoyed by ReLU networks. Specifically, is known that He initialization in deep ReLU networks asymptotically preserves variance of activations in the forward pass and variance of gradients in the backward pass for infinitely wide networks, thus preserving the flow of information both directions. Our paper goes beyond these results and shows novel properties that hold under He initialization: i) the norm of hidden activation of each layer is equal the norm of the input, and, ii) the norm of weight gradient of each layer is equal to the product of norm of the input vector and the error at output layer. These results are derived using the PAC analysis framework, and hold true for finitely sized datasets such that the width of the ReLU network only needs to larger than a certain finite lower bound. As we show, this lower bound depends on the depth of the network and the number of samples, and by the virtue of being a lower bound, over wwer-parameterized ReLU networks are endowed with these desirable properties. For the aforementioned hidden activation norm property under He initialization, we further extend our theory and show that this property holds for finite width network even when the number of data samples is infinite. Thus we overcome several limitations of existing papers, and show new properties of deep ReLU networks initialization.
INTRODUCTION
Deep rectifier (ReLU) networks are popular in deep learning due to their ease of training and state- of-the-art generalization. This success of deep rectifier networks can be partly attributed good initialization strategies (for example Glorot & Bengio (2010); He et al. (015)). Essentially, good parameter initializations guarantee that there is no exploding or vanishing of information across hidden layers. These properties help gradient descent based optimization methods in navigating the complex non-linear loss landscape of deep networks by initializing them at a good starting point where training begin. Such favorable properties promised by these initialization strategies (in most cases) shown to hold true in asymptotic settings where the network width tends to infinity and/or under striet assumptions made about the distribution of the input data. A detailed account of these existing papers and a contrast between these papers and our work is discussed in section 2
Our paper relaxes the aforementioned assumptions made in previous papers. Further, show novel properties that hold for deep ReLU networks at initialization when using the He initialization scheme (He et al. 2015). Specifically, we show that deep ReLU networks obey the following properties in the forward (Eq. back backward (Eq. 2) pass (see section 3 for notations),
| = ||x||2 VI (1,2,...,
[e(fo(x),))
|| ||o(x,))|l=>
||x||2
V (1,2,.....1
awi
We refer to the above properties as as the the activation norm equality and the gradient norm equality property.

---

Under review as a conference paper at ICLR 2020
Further, we derive finite lower bound the width of the hidden layers for which the above results hold (i.e., the network needs to be sufficiently over over-paramettn perrparamtterzze) in contrast to number of previous papers that assume infinitely wide layers.
We do not make any assumption on the data distribution as done in a number of previous papers that study initialization. Further, results hold for an infinite stream of data for the activation norm equality property and for any finite dataset in the backward pass.
Thus we have relaxed a number of assumptions made in previous research work that focus deriving initialization strategies for deep ReLU networks. Our results showing the connection between activation norm and input norm (and similarly the property for gradients) for deep ReL.U networks can be utilized in further research studies.
RELATION WITH EXISTING WORK
The seminal work of Glorot& & Bengio (010)) studied for the first time a principled way to initialize deep networks to avoid exploding/vanishing gradient problem (EVGP). Their analysis however is done for deep linear networks. The analysis by He et al. (2015) follows the derivation strategy of Glorot & Bengio (010)) except they tailor their derivation for deep ReLU networks. However, both these papers make strong assumption that the dimensions of the input are statistically independent and that the network width is infinite. Our results do not make these assumptions.
Saxe et al. 2013) introduce the notion of dynamical isometry which is achieved when all the singular values of the input-output Jacobian of the network is 1. They show that deep linear networks achieve dynamical isometry when initialized using orthogonal weights and this property allows fast learning in such networks.
Poole et al. (2016) study how the norm of hidden activations evolve when propagating an input through the network. PPennington et al. (2017 7, 2018) study the exploding and vanishing gradient problem in deep ReLU networks using tools from free probability theory. Under the assumption of an infinitely wide network, they show that the average squared singular value of the input-output Jacobian for deep ReLU network is 1 when initialized appropriately. Our paper on the other hand shows that deep ReLU networks are norm preserving maps at appropriate initialization. Further, we show there exists finite lower bound on the width of the network for which the Frobenius norm of the hidden layer-output Jacobian (equivalently the sum of its squared singular values) are equal across all hidden layers.
Hanin& & Rolnick (2018) show that for a fixed input, the variance of the squared norm of hidden layer activations are bounded from above and below for deep ReLU networks to be near the squared norm of the input such that the bound depends on the sum of reciprocal of layer widths of the network. Our paper shows similar result in PAC bound sense but as important difference, results hold even for an infinite stream of data by making the bound depend on the dimensionality of the input.
show that these
Hanin (2018) show that sufficiently wide deep ReLU networks with appropriately initialized weights prevent EVGP in the sense that the fluctuation between the elements of the input-output Jacobian matrix of the network is small. This avoids EVGP because a large fluctuation between the elements of the input-output Jacobian implies large variation in its singular values. Our paper shows that sufficiently wide deep ReLU networks avoid EVGP in the sense that the norm of the gradient for the weights of each layer is roughly equal a fixed quantity that depends on the input and target.
Over- pree-aaratttiizaiionn in deep networks has previously been shown to have advantages, Neyshabur et al. (2014); Arpit et al. (2077) show empirically that wider networks train faster (number of epochs) and have better generalization performance. From a theoretical view point, NNyyshabe et al. (2018) derive a generalization bound for a two layer ReLU network where they show that a wider network has a lower complexity. Lee et al. (2017) show that infinitely wide deep networks act as a Gaussian process. Arora et al. (2018) show that prrmtttratttiiiinooonn in deep linear networks acts as conditioning parameterization in terms of depth is studied. Our analysis complements this line of work by showing another advantage of over-parameetrrizatoo in deep ReLU networks.
the gradient leading to faster convergence, although in this case over-
2

---

Under review as a conference paper at ICLR 2020
3 THEORETICAL RESULTS
Let D = {x;, yi = be N training sample pairs of inputs vectors x; E IR and target vectors y where x; are sampled from a distribution with support X. Define layer deep ReLU network fo(x) = h with the 1th hidden layer's activation given by,
h' Lu' a' = w'n'-i
E (1,2...--L
(3)
where h' E IR are the hidden activations, h is the input to the network and can be one of the input vectors X,, W E [= are the weight matrices, I are the bias vectors which are initialized Os, a' are the pre-activations and 8 bb))=-
Define a loss on the deep network function for any given training data sample (x, y) as.
E {x)
(4)
where f(-) any desired loss function. For instance, E(.) can be log loss for classification problem, in which case fo(x) is transformed using weight matrix to have dimensions equal to the number of classes and the softmax activation is applied to yield class probabilities (i.e., a logistic regression like
model on top of fx(x)). However for our purpose, we do not need to restrict ((.) to specific choice, we only need it to be differentiable. We will make use of the notation,
8(x, y) == {f([o(x).))
(5)
Oar
We organize our theoretical results follows. We first derive the activation norm equality property for finite datasets and then extend these results to infinite dataset setting in section 3.1 We then derive the gradient shown in the main text, are available in the appendix.
equality property for finite datasets in section 3.2 All formal proofs, if not
3.1 ACTIVATION NORM EQUALITY
Consider an L layer deep ReLU network and data x E X. We show in this section that the norm of hidden layer activation of any layer is roughly equal to the norm of the input initialization for all x E X if the network weights are initialized appropriately and the network width is sufficiently large but finite. Specifically we show VI [4] and X,
h 1l2 = |x|12
(6)
To achieve this goal, we start with a very simple result- in expectation, ReLU transformation in each layer preserves the norm of its corresponding input if the weights are sampled appropriately. Evaluating this expectation also helps determining the scale of the random initialization that leads to norm preservation.
Lemma Let ReLU (Ru), where E RD. R E RMXR If R.J Lid. NO,, 2)) then for any fixed vector [|[]] ||u||2.
The proof for the above lemma involves simply computing the expectation analytically by exploting the fact that each dimension of the vector u is a weighted sum of Gaussian random variables. The above result thus shows that for each layer, initializing its weights from i.i.d. Gaussian distribution with 0 mean and 2/ e/fan-out variance (viz. He initialization (He et al. 2015)) preserves the norm of its input in expectation. We now derive lower bound on the width of ReLU layer so that it can preserve the norm of the input for single fixed input with error margin.
Lemma 2
Let ReLU (Ru), where RR R. Rmxn then for any fixed vector u,
Rf 1.d.d. (10 2), and (0,
Pr (I || S (|ll3)) 2 2eepp
-m
+ log
(7)
1+ /1
3

---

Under review as a conference paper at ICLR 2020
The proof of this lemma involves direct application of the Chernoff bounding technique. Now use the above lemma to show that the norm of hidden activations equal the norn of inputs within a specified margin and for finite size dataset for deep ReLU network.
Theorem Let 'D be a fixed dataset with N samples and define L. layer ReLU network (-()) as shown in Eq. 3 such that each weight matrix Wi E [Ruxx--> has its elements sampled as W i..dd NO, m and biases b' are set to zeros. Then for any sample (x, E DD and E [0. 1),
we have that,
Pr ((1 - 1)x1 11,0(x)11 S (1 (1+,)((xx13)
L 2 2N exp
(-ny
( log 1 V1 +
2
(8)
While the statement of the above theorem only talks about the norm of the final output of the network, it equally applies to any hidden layer / as well since the theorem can be applied equivalently to l layer network.
Having proved the activation norm equality property in the finite dataset setting above, we now turn our attention to the case of infinite dataset case. To do so, we first prove a non-trivial result where we use lemma such that this layer preserves the norm of an infinite stream of inputs.
2 to show how a lower bound on the width of an individual ReLU layer can be computed
Lemma 3 Let X be d S dimensional subspace of Th" and R Rmxx, Rfj sid N(O, ) c C 1), and,
c/12 log log(0.5((1 + /1 c/3)) (dlo + log 4 8
m 2
(9)
then with probability at least 1 - 8,
(1 (1=))] < [Ru)] < (1+ u)llull VUE
(10)
where A = mint 3/8" v3d
Proof Sketch: The core idea behind the proof is inspired by lemma 10 of Sarlos (2006). Without any loss of generality, we will show the norm preserving property for any unit vector u the dimensional subspace X R". This is because for any arbitrary length vector [|ReLU(Ru)]]
||u|| RRRLU (Rû)]). The idea then is to define a grid of finite points over X on [-1, 1] with interval size depending e, such that every unit vector â in X' is close enough one of the grid points. Then, if we choose the width of the layer to be large enough to approximately preserve the length of the finite number of grid points, we can guarantee that the length of any arbitrary unit vector approximately remains preserved as well within the derived margin of error. The formal proof can be found in the appendix.
We now extend the above lemma to deep ReLU network and show our main result for the forward pass that the norm of hidden activations equal the norm of input within some distortion margin, for an infinite stream of input data for a sufficiently large (but finite) width deep ReI.U network.
Theorem Define a L layer ReLU network fo(.) as shown in Eq. such that each weight matrix i.dd NO, 2 and biases b' are set zeros. Let my X be a d < dimensional subspace of R.. Wt t.d. N(O. 2,) c E [0. 1), and,
W E xm has its elements sampled as Wt ig
(dlog 2
4L o + log
n4. 2
vl [L]
(1I)
c/12 - log(0.511 / + cc33)
A
then with probubility at least 1 6,
(1 [x=c)|(x13
|bl < (1 12112
X [e [L]
(12)
4

---

Under review as a conference paper at ICLR 2020
Proof: Since the input lies d dimensional subspace of R", apply theorem 3 to the first layer and get the guarantee that the norm of all inputs on the d dimensional subspace are preserved by this layer. Next, we show that since each layer is linear transform followed by pointwise ReLU non-linearity, and the input takes values in a set defined by the d dimensional subspace of R", the output of the first layer will take values in set that is strictly a subset of d dimensional subspace. To see this, let B E wgx denote a matrix with orthonormal columns describing the basis of the subspace X on which the input lies, and let z Rd. Then we have that,
E (Bz|z E TR
(13)
The first layer transforms any input as
= ReLLU = RReLU (w)
(14)
(15)
Denote B' W. Then note that rank(BB) S. d. Let S denote the set of values that h² can take. Then we have that,
S = ( E R2" m
(16)
where R= denote the subset of B where all dimensions take non-negative values. This shows that h takes values in a set that is strictly a subset of a d dimensional subspace of RR
Having proved this for first layer, we can recursively apply this strategy all higher layers since the output of each layer lies on a subset of a subspace. Notice that while doing so, the lower bound on the width of each layer depends only on the subspace dimensionality d. Applying union bound over the result of theorem for L layers proves the claim.
We note that the lower bound width derived above depends two quantities- the depth of the network L. and the dimensionality of the subspace d on which the input lies. Specifically, the lower bound on the width becomes larger for larger input dimensionality d and larger network depth L irrespective of the number of data samples, meaning that a wider network needed as the depth of the network and/or the intrinsic input dimensionality increases.
3.2 GRADIENT NORM EQUALITY
Consider any given loss function {(.) and a data sample (x, y). we show in this section that the norm of gradient for the parameter W of the 7 layer depends only on the input and output. Specifically,
for a wide enough network, the following holds at initialization for all / E {1, 2 L} and Vx E D,
(e(J(()) || (1x(x, y) ||x||2
4Z
(17)
aw
As first step, note that the gradient for parameter W for sample (x, is given,
(x),)) = diag (E([o(x).)) -M,, (h'-1)
(18)
Ow
Dal
where Mn, (h'-1) is a matrix of size n; x m4-1 such that each row is the vector -111 Therefore,
simple algebraic manipulation shows that,
OE (o v) ||r 3l([o(x),)) 112 |
(19)
ow
In the previous section, we showed that for a sufficiently wide network, || = ||x||a VT with high probability. To show that gradient norms of parameters are preserved in the sense shown in Eq. (T7). essentially show that (x) networks.
|| [|ō(x, VI with high probability for sufficiently wide
Dal
Note that | f((ff(())))) ||2 [|0(x, y) by definition. To show the norm is preserved for all layers, 0 we begin by noting that,
de (0(Jo(x),))
Oh'
da 41 ae(Jo(x..))
0al
da!
Oh
0at+1
W4+1 3l(Jo(x),))
=1(a') o
(20)
0a!t1
5

---

Under review as a conference paper at ICLR 2020
where @ is the point-way wise product (or Hadamard product) and 1() is the heaviside step function. The following proposition shows that (.) follows a Bernoulli distribution w.r.t the weights given any fixed imput at the previous layer.
If nenwork weights are sampled i...d. from a Gaussian distribution with mean 0 and biases are 0 at initialization, then conditioned on h'-1, each dimension of 1(a') follows i.i.d. Bernoulli distribution with probability 0.5 at initialization.
Proposition 1
Given this property of (a'), we show below that the transformation of type shown in Eq. 20 is norm preserving in expectation.
Lemma 4 Ler (Ru) where IR", R Rmxn and E R Rf, i.d.d. N(0, pm and
i..dd 2; ~ Bernoullipp). then for any fixed vector u, [[[l]|]]= ||u||?
The proof of this lemma involves analytically computing the expectation of the vector norm by exploiting the fact that each dimension of v a sum of Gaussian random variables multiplied to an independent Bernoulli random variable. This lemma reveals the variance of the 0 mean Gaussian distribution from which the weights must be sampled in order for the vector norm to be preserved in expectation. Since 1 (at) is sampled from a 0.5 probability Bernoulli, we have that the weights must be sampled from Gaussian with variance /m. We now show this property holds for a finite width network.
Lemma Let (Ru) O z, where R", z E R" and R. E RR Lid. Bernoull((0.5) and c E [0, 1), then for any fixed vector u,
If iid. N(O, u.sm)
Z3
Pr (Illv|||=
||l| 2|
S E||lul?)
2
21 2- exp
-m
log
(21)
+
vI++/
The proof of this lemma involves direct application of the Chernoff bounding technique. Having shown that finite width ReLU layer can preserve gradient norm, we now note that we need apply this result to Eq. (20). In this case, we must substitute the matrix R. in the above lemma with the network's weight matrix Wi+1". In the previous subsection, matrix W must be sampled from (0, 2/m(+1) in order for the norm of the input vector to be preserved. However, order for the Jacobian norm to preserved. we require W to be sampled from (0, 2/n;) as per the above lemma. This suggests that if we want the norms to be preserved in the forward and backward pass for single layer simultaneously, network to be close to uniform. The reason we want them to simultaneously hold is because as shown in Eq. 19 in order for the parameter gradient norm to be same for all layers, we need the norm of both the Jacobian [ 1l as well as the hidden activation ||b'='||e ||2 to be preserved throughout the hidden layers. Therefore, assuming the network has a uniform width, we now prove that in deep ReLU networks with He initialization, the norm of weight gradient for each layer is simply a product of norm of the input and norm of the error at output.
showed that each element of the
beneficial for the width of the
Let D be fixed dataset with N samples and define a L layer ReLU network as shown in Eq. such that each weight matrix w e gax" has its elements sampled as Wj i.dd. N0, 2) and biases b' are set to zeros. Then for any sample (x.y) E DD c [0, 1), and for all l E 21,2.... L} with probability at least,
Theorem 3
- 4NLexp (-n ( +log
2
log
(22)
1 VI 11+e)
the following hold true,
[== 1) aE al(fo(x), y) 112
OW < + 1 ()(x,y))l
(23)
'Similar to He et al. 2016 we have assumed that ( in independent from (a') and W/+ initialization.
0a

---

Under review as a conference paper at ICLR 2020
1.2
1.6
1.0
1.4
0.8
1.2
5 11.0
0.6
Mac3CO
0.8
0.4
0.4
0.2
0.2
0.0
0.0
Layer Index (i)
Layer Index (I)
Figure 1: For He initialization, the norm of hidden activation h' roughly equals the norm of input x;
and the norm of weight gradient @W == (f(ff(x).v) roughly equals the product of norm of input and the norm of output error 8 8xx y) when width is sufficiently large. Glorot initialization does not have this property.
2W
2.00
2.00
1.75
1.75
1.50
1.50
1.25
1.00
thecreical
theonet emprinal
ampricaa
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
500
1000
1500 2000 2500 3000 3500 4000 Network width
500
1000 1500 2000 2500 3000 3500 4000 Network width
Figure 2: Tightness of lower bounds network width derived in lemma and lemma 5 shown in the left sub-figure and right sub-figure respectively. See text for more details.
and
(1 (1=|)|x
[h'l (1 +
(24)
We note that even though the theorem relies on the specified independence assumption similar to He al. 2016), show that our predictions hold in practice in the section.
EMPIRICAL VERIFICATION
4.1 NORM PRESERVATION OF ACTIVATION AND GRADIENTS
In this section, we verify the hidden activations have the same norm as input norm ||h ||
(Eq.
|||||
and the parameter gradient norm approximately equal the product input norm and output error
(e((a(x))))
ow? [|
(Eq. 17) for all layer indices for sufficiently wide deep ReLU networks. For this experiment we choose a 10 layer network with 2000 randomly generated input samples in R500 and randomly generated target labels in 29 and cross-entropy loss. We add linear layer ReLU network's outputs to make the final output R We network width from the set {100, 500, 2000, 4060}. We show results for both He initialization (He et al. 2015) which we thcoretically show is optimal, as well as Glorot initialization (Glorot & Bengio 2010) which is not optimal for deep ReLU nets. As can be seen in figure (left), the mean ratio hidden activation norm to the input norm over the dataset is roughly with a small standard deviation for He initializaiton. This approximation becomes better with larger width. On the other hand, Glorot initialization fails at preserving activation norm for deep ReLU nets. A similar result can be seen for parameter gradients norms (figure I (right)). In the figure denote @t(fe(x).x) by ow
norm (xxxy))l=- -
along with softmax activation
aW: Here find for He initialization that the norm of weight gradient for each layer is roughly equal to the product norm of input and norm of error at output, and this approximation becomes stronger for wider networks. Once again Glorot initialization does not have this property.

---

Under review as a conference paper at ICLR 2020
||aW||//|a6, y.ll| ||x||
1.2
He Width-variation 1
1.0
He Width-variation 200
He Width-variation 500
0.8
2
Layer Index (i)
Figure 3: Effect of con-uniformity of width in deep ReLU network on the gradient norm equality property. For each of the three networks, width of each layer is selected independently from 4(1000- v, 1000 + v)), where is the width variation shown in the plot. Gradient norm equality holds more accurately when width variation is smaller.
4.2 TIGHTNESS OF BOUND
In the following experiment verify the tightness of the bound in lemma (f forward pass) and lemma | (for backward pass). To do so, we vary the network width of a one hidden layer ReLU transformation from 500 to 4000, and feed 2000 randomly sampled inputs x through it. For each sample we measure the distortion e defined as,
|b|l |
=
(25)
for the forward pass, and,
8E [e(f)(x).
1
OW
(26)
y)((
||
for the backward pass. Here h is the output of the one hidden layer ReLU transformation. We compute the mean value of for the 2000 examples and plot them against the network width used. We call this the empirical estimate. We simultaneously plot the values of predicted by lemma and and lemma 5 for failure probability pass is shown in ()eft),, As can be seen, our lower bound on width is an over-estimation but becomes tighter for smaller values Thus our proposed bounds can be improved and we leave that as future work.
0.05. We call this the theoretical value. The plot for the forward
similar result can be seen for lemma 5 in figure 2 (right).
4.3 EFFECT OF NON-UNIFORMITY OF WIDTH ON GRADIENT NORM EQUALITY
As discussed in section 3.2, gradient norm equality property holds more accurately when deep networks have more uniform width throughout the layers. To verify this, construct 20 layer deep ReLU network such that the width of each layer determined by independently sampling uniformly between 1000 and 1000 e, where denotes the amount of width variation chosen for particular experiment. Once the network architecture fixed, we initialize the weights with He initialization. We then generate 1000 pairs of input samples and output error similar to the process
|| (FCf
Sw: ||
(6 lla [a-[[x] The mean and standard deviation of this value across samples shown in figure for {I, 200, 500). It can be seen that the ratio is closer to with smaller variance when width variation is small, thus verifying our theoretical prediction.
described in section 4.1 and compute the ratio
CONCLUSION
We derived novel properties that are possessed by deep ReLU networks initialized with He initializa- tion. Specifically, we show that the norm of hidden activations and the norm weight gradients are function of the norm of input data and error at output. While deriving these properties, we relaxed most of the assumptions (such as those on imput distribution and width of network) made by previous work that study weight initialization in deep ReLU networks. Thus work establishes that He initialization optimally preserves the flow of information the forward and backward directions in a stronger setting, and uncovers novel properties.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. arXiv preprint arXiv: 06509, 2018.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. arXiv preprint arXiv: 0739.. 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? arXiv preprint arXiv: 037444, 2018
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. arXiv preprint arXiv: 1803.01799 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sham Kakade and Greg Shakhnarovich. Random projections. 2009. URL http://ttic www.aaa......geeeeeeeeee
Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. arXiv preprint urXiv:/7l 1111.001555 2017.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 1412:661126.6 2014.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over- wer-paraaetrizztioon in generalization of neural networks. arXiv preprint arXiv:1805./2077, 2018
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Advances in neural information processing systems, pp. 4785-4795, 2017.
Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. arXiv preprint arXiv: 2018.
Ben Poole. Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. In Advances in neural information processing systems, pp. 3360-3368. 2016.
Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In Foundations of Computer Science, 2006. FOCS'06, 47th Anmual IEEE Symposium on, pp. 143-152. IEEE, 2006.
Andrew Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv: 13/2.6223 2013.

---

Under review as a conference paper at ICLR 2020
APPENDIX
A PROOFS
A.1 PROOFS FOR FORWARD PASS
Lemma Let v = ReLU (Ru), where u E R" and R E Ruxx Rf and. NOO, m)) then for any fixed vector u, [[[[]]]]] || ||ull?
Proof: Define a, R.Tu u, where R, denotes the ith row of R. Since each element Rij is an independent sample from Gaussian distribution, each a; is essentially weighted sum of these independent random variables. Thus, each o; ~ ~N (0, m |||²)) and independent from one another. Thus each element 1; (a:) ~ ~N (0, m U|l where NR denotes the rectified Normal distribution. Our goal is to compute,
2
B||| E
(27)
mmE
(28)
From the definition of 0i
[u = +
E|Z
(29)
2
2
where Z follows a half-Normal distribution corresponding to the Normal distribution N (0, [|]
Thus [
2 2/122
2
1
Similarly,
m
ma
El = 0.51E
(30)
0.5(var (2)
+ +
(31)
Since var (Z) m
[| 2) we get,
2 + (2
| )²
B 0.5
| (1
(32)
||u||
(33)
m
Thus,
mIEl mIE(u,]
||]|
(34)
which proves the claim.
Lemma Letv= = ReLU (Ru), where u E I", R E Rmxn Rf 3.d N(0, 2), and c E [0, 1),
then for any fixed vector u.
( ( +loog log
2
-|uu|1s e||ull3) 2 2exp -m
1 + (+v)))
(35)
Proof:
Define V
/0.5m |
V. Then we have that each element 03 NR (0,1) and independent
from another since va ReLU (a;) NR (0, [ull?) where NR denotes the rectified Normal distribution. Thus to bound the probability of failure for the R.H.S.,
||u|2
Pr (1lv|]2 2 (1 ()l = Pr
|
2 (1 + +c]lll1))
(36)
0.5m
Pr 2| > (1
(37)

---

Under review as a conference paper at ICLR 2020
Using Chernoff's s bound, we get for any À 0.
Pr (|lll 2 0.5m(1 + <)) Pr (exp(Alli)()) 2 exp(A0.5m(1+ e)))
(38)
Elexp(A A [
(39)
exp(0.5mm 5 + +))
E/exp1 Σ 1=1 A0; )]
(40)
exp(0.5mmX + +e)
M E (exp(Ax;;"))
(41)
(0.5mm + +c)
( 3
exp(0..AA11 + +))
(42)
Denote v))) as the probability distribution of the rectified Normal random variable ü, Then,
Elexp(Ae;
=
(xp(Axi))p(er)
(43)
We know that the mass at = 0 is 0.5 and the density between and = oo follows the Normal distribution. Thus,
Elexp(Ar, 21 0.5 expex)(() +
exp(X6,2
-2,2/2)
(44)
2 Ju
1
V2
0/2
0.5
exp(-
5-11- 2A))
(1-
(45)
2 (1 2/(1
2X)
v 2X)
2
Note that Sn sponding to the Normal distribution
J2
So exp( 2/- (1-) the integral of half Normal distribution corre-
Vn/(1-23) 2X)
(00,, 1/(1 - 2A)). Thus,
1
Elexp(Ar, 0.5 +
(46)
2 2 2A)
Hence, we get,
m
Pr Pr 0.5m(1 e)) S
0.5 (.5
1+
exp(-0.5X(1 + e))
(47)
v(1
2X)
The above failure probability can be bounded be smaller by finding an appropriate value of X. We find that X z 0.5e approximately minimizes the above bound. Substituting this value of X above, we get,
14c
2 0.5m(1 + +c)) S (0.5 (1 vT+e) exp( exp(-=))
(48)
=exp (-m (-m
2
+ log
(49)
1 vI+e)
Thus,
Pr (1lvl < (1 e)[lul 2 exp
(-m + log 1 VI+
(50)
Similarly, to prove the I.H.S. by bounding the probability failure from the other side,
Pr (Iv]] < (1 -e]llll = Pr (- | > -(1 e)]lu
(51)
||u|| | - -(1-)]luul?
= Pr
(52)
0.5m
= Pr
(-1
-0.5m(1 - E))
(53)
2

---

Under review as a conference paper at ICLR 2020
Using Chernoff's bound, we get for any X 0,
Pr (-1 |5|1 -0.5m(1 - c)) = Pr (exp(--X|i1)) -|\p- 2 exp(-20.5mm1
(54)
[
(55)
exp( 0.5mAA1 - c))
E(exp( - Σズ= 2o;;
(56)
cxp( -0.5m - c))
11 Blexp(- -Aoū;
(57)
exp( 0.5mm -c))
1E(exp(-Xit) 26,
(58)
exp(-0.5A(1- - -c))
Performing computations similar to those above to compute the expectation term, we get,
Elexp(-X6E2) 0.5
(59)
2A)
Hence, we get,
m
1
exp(0.5A(1 -c))
Pr Pr S 0.5m(1 - -)) < 0.5 (0.5
14
(60)
v 2x)
Similar to the R.H.S. case, we find that X ss T
0.be
approximately minimizes the failure probability,
Pr Pr ([VV] & 0.5m(1 - -c)) - (0.5 (1 v1) expp((
(61)
/ log
exp
(62)
+ VI
It be shown that,
exp
m
c log
+ VI )) s
2
exp
(-m
(-m
log
1 vT+e
2
(63)
Thus,
exp (-m (1
Pr ( -1- ||| [|
+og log
(64)
1 vI vI+e/
Using union bound, Eq.
50 and (64) hold together with probability,
Pr 1)luu |v/ (1 + ()l z cxp (-m
(1 ( log 1 1ov+++ )
2
(65)
This proves the claim.
Theorem Let D be a fixed dataset with N samples and define a L layer ReLU network as shown in Eq. such that each weight matrix W E R xma- has its elements sampled as W1 Lid. No 2 and biases b' are set zeros. Then for any sample (x,y) E. D and E [0, 1), we have that,
Pr ((1 -) (1x=)) (1 < (1+c)*1[x13) 21 Σ 2N exp
(-nr ( ( + log
2
1 vI +
(66)
Proof: When feed forwardingg a fixed input through the layers of a deep ReLU network, each hidden layer's activation corresponding the given input is also fixed because the network is
3

---

Under review as a conference paper at ICLR 2020
deterministic. Thus applying lemma 2 each layer's transformation, the following holds true for each (1,2,---L),
Pr ((1 (11-e) =)1h 11''11 (1+ e) 1 2 1- eexp (-1
( (" V
(67)
Thus, using union bound, have the lengths of all the layers until layer are simultaneously preserved with probability at least,
- Σ 2exp (-ni
( log
(68)
1 + v1
0=1
Applying union bound again, all the lengths until layer are preserved simultaneously for N inputs with probability,
2N exp (-mV ( + + V1 +
2
(69)
P=1
Finally, note that the following hold true with the above probability,
(1 - 2 < || s (1 +)|x)
(70)
(1 - c1-c)11h11-- & h $ (1+c)
(71)
Substituting ||h' 1 < (1 - |||1 in the R.H.S. of the last equation, and (1 e)|1x|1 < h || in
the L.H.S. of the last equation, we get,
(1 (1=c) < |1²1|2 S (1+)
(72)
Performing substitutions for higher layers similarly yields the claim.
Lemma 3 Let X be d S dimensional subspace of Rh" and R Rmxx Rij c E 1), and,
sid N(0, ).
+ log 4
m 2
dllo
(73)
e/12 - log(0.(111 vI+e/3)
then for all vectors E X, with probability at least 1 -6,
[[ReLU(RR)) - ||u|| S u||u||
(74)
where A = min 3V8
v
Proof: The core idea behind the proof is inspired by lemma 10 of Sarlos (2006). Without any loss of generulity, we will show the norm preserving property for any unit vector u the d dimensional subspace X of R" This is because for any arbitrary length vector
||l|| (ReLU(Rū)) The idea then is to define a grid of finite points over such that every unit vector â in X is close enough one of the grid points. Then, if we choose the width of the layer to be large enough to approximately preserve the length of the finite number of grid points, we essentially guarantee that the length of any arbitrary vector approximately remains preserved.
[ReLU((u)]
To this end, we define a grid G on [_1, with interval of size A minfe/Vd, ve/d). Note the number of points on this grid is (2)² Also, let column vectors of B E &Buxd be the orthonormal basis of X.
We now prove the R.H.S. of the bound in the claim. we consider any unit vector â in X, we can find a point g on the grid G such that ||g|| < L, and it closest in l² norm, and define r' Thus the vector ú can essentially be decomposed as,
g.
ń g
(75)
Also note that since r' lies in the span of X, can represent r' == Br for vector
4

---

Under review as a conference paper at ICLR 2020
Now consider the norm of the vector ú after the ReLU transformation give by [[ū)] Then we have,
(Ri))
(e + +r))
(76)
ReLU ReIU(Rr)))]
(77)
ReLU + ReLU(RR)))
(78)
x (Rg) + |Rr'll
(79)
Similarly, we have,
[ReLU(Rg) |eeLU(R(g+ ú ا))Il
(80)
(Rū) + RRLU ú)))II
(81)
[Rū) + ReLU (
(82)
[[ + Rr'll
(83)
Therefore,
|| (ReLU(gg)) [Rr'l < [ <
ReLU (Rg)ll + ||R]]
(84)
Applying union bound on all the points in G. from lemmal
we know that with probability at least
1
exp (mm + log (+vi++)
(ReLU((g))))
S (1 + +
< 1 +
(85)
S (1 c)²
(86)
This can be substituted in the R.H.S. of Eq. 84 Then,
Now we only need to upper bound |Rr'll []. To this
end, rewrite |Rr'|l |RBr|,
R RB RB >
(87)
1
2 Ird Inal
RB,,
RB,
(88)
V2
V2
Note that V2 R. is a matrix whose entries are sampled from N(, 1). Invoking lemma on the de
terms the above sum, we have that with probability at least 1 2d exp (- m (?2
d d
2
ral - lryl- < RB,
RB
>
2
Iaa
lryl (< B Bj > +e)
(89)
V2
V2
im1
d
d d
Σ
r B +
Ii 11.11slle
(90)
21|r||1 2c ||
(91)
Since r', and hence r is a point inside one of the grid cells containing the origin, its length can be at most the length of the main diagonal of the grid cell. Formally, ||||| S vdA < e, and ||]|] I dA S Ve. Subsituting these inequalities in the above equations, we get,
RBr 2 < 4c2
(92)
Looking back at the R.H.S. of Eq.
huve that with probability at least
(2) exp (-m (i log 21+z))
2d cxp m (e² - e)),
(RLU(RR)))
< 1 + é + 2e
(93)
1 3c
(94)
5

---

Under review as a conference paper at ICLR 2020
To prove the L.H.S. of the claimed bound, we can similarly find point g the grid G such that
lig 2 L, and is closest to ú in E norm, and define r' := -û- - g. Then invoking lemma 2 we know
that with probability at least
( exp (-m ( log (+v1+)))
(Rg))(Rgg]
2 (1 <)
2
(95)
2 c)=
(96)
This can be substituted in the L.H.S. of Eq. 84
We then substitute the previously com- puted upper bound of ||RBr|| once again and have that with probability least
2 (2) exp (-m log (x(+=)) 2d exp
(c² (c)).
1 3e < (Rū) S 11+ +3c
(97)
Scaling û arbitrarily, we equivalently have,
(1- - 31)elll |ReLU (Ru)] < (1 + /u]
(98)
Finally, since,
2
exp -m ( log
))
d exp
(ē=-)))
m
(99)
A
1 /1
We can further lower bound the success probability of Eq. (98) for mathematical ease as,
/1 ))
4
exp
-171
log
(100)
A
Therefore to guarantee success probability of at least we bound,
1 1 (2
2
exp
log
1 v1
+
21
(101)
Rearranging the terms in the equality to get a lower bound on m and rescaling e proves the claim.
A.2 PROOFS FOR BACKWARD PASS
Proposition 1 network weights are sampled i.i.d. from a Gaussian distribution with mean 0 and biases are 0 at initialization, then conditioned on h'-!, each dimension of (aa)) follows an i.i.d. Bernoulli distribution with probubility 0.5 initialization.
Proof: Note that a' = '- at initialization (biases are 0) and Wi are sampled i.i.d. from random distribution with mean Therefore, each dimension at simply a weighted sum of i.i.d. zero mean Gaussian, which is also O mean Gaussian random variable.
To prove the claim, note that the indicator operator applied on a random variable with mean and symmetric distribution will have equal probability mass on both sides of which is the same as a Bernoulli distributed random variable with probability 0.5. Finally, each dimension of a' is i.i.d. simply because all the elements of W are sampled i.i.d., and hence each dimension of al is weighted sum of a different set of i..dd random variables.
Lemma Let (Ru) z, where TR. R E Rmxx and E TR. If 1.i.d. N(O, pm and i.idd
Bernoulli(p), then for any fixed vector [[[[v|i]]= ||u|22
Proof: Define a; = Rd u, where R, denotes the ith row of Since each element Rij is an independent sample from Gaussian distribution, each a, is essentially weighted sum of these independent random variables. Thus, each a; ~ N (0, pm ||ul(() and independent from one another.

---

Under review as a conference paper at ICLR 2020
Our goal is to compute,
T Σ
|||v
(102)
B [2
(103)
i=1
mlEla ||
(104)
mp(evr (a:) + E(a:;
(105)
||u||
(106)
which proves the clain.
Lemma Let v = (Ru)& @ z, where u E R", 2 E RIm,, and R E RR Rig
i.i.d. N(O, 0.bmm )
i.d. Bernoulli(0.5) and E (0,1), 1), then for any fixed vector u.
Pr (1lv|11- ||u||2| s e e|lul?) 2 1 2exp (-m + log
I 1 vI+e))
(107)
Proof:
Define a, = R. u, where R2 denotes the 1th row of Then, each a; ~ N (0, 0.5m ||ul(3) and independent from one another. Define
%0.5mm Then we have that each element āg N (0, 1).
|ul
Define such that U; = -,22. Thus to bound the probability of failure for the R.H.S.,
||u||
Pr (11v|22 2 (1 ()lll))) Pr
| 2 (1+ +e]lull))
(108)
0.5m
2 0.5m(1 +()
(109)
Using Chernoff's s bound, we get for any x> 0,
Pr (I|v|| 2 0.5m(1 + t)) = Pr (exp(A|( exp(A0.5m e)))
(110)
E|exp( A |
<
(111)
exp(0.5mmX1 + +()
A(exp(Ex m A0;22)
(112)
exp(0.5mm + c))
I E, eepp (Ao;;))
(113)
(0.m0.. + +))
Elcxp(Aū; ?)
(114)
exp(0.5XX1 + +))
Denote p(ā;) and p(z)) as the probability distribution of the random variables â; and 2 respectively. Then,
Ejexp(>oū 3)] = - J
(115)
Substituting p(ā;) with standard Normal distribution, we get,
/
ā,²
Elexp(Xú; 2)] Ep(z,)
exp(>a, 22
(116)
V2T a,
21
Ep(z)
exp(
-i (1 - 2Xz,))
(117)
22T
2
Σp(ss)
1
V1-
- 2N
expl
ā,2
(1 - 2xz;))
(118)
Jú
V2T V1 2x
2
21
Σp(e)) ((())
1
(119)
/I 1122M 2\2
7

---

Under review as a conference paper at ICLR 2020
where the last equality holds because the integral of a Gaussian distribution over its domain is 1.
Finally, summing over the Bernoulli random variable 24, we get.
1
(e ?) = (1 0.5) + v1 2A
(120)
Hence, we get,
0.5
Pr (I[V|] 2 0.5m(1 + +c)) 2
0.5
1
exp(- + +
(121)
V(1
2X)
1
0.5
1+
exp(-0.5A(1+ + E))
(122)
v(1 2X)
We find that the above inequality is identical to that in Eq. 447.. Thus À ~ 0.5c Itc approximately
minimizes the above bound as before. Substituting this value of above, we get,
Pr (1 2 0.5m(1 +)) s (0.5 (1+ VI++ exp( exp((-1)) )
(123)
(
= exp
-m
4
+ log
1 1 vI+c
(124)
Thus,
( + log
Pr Pr ([[v13 S (1 + e)[[u| 2 exp
(-m
(125)
+ v1 +
Similarly, to prove the L.H.S. by bounding the probability of failure from the other side,
Pr (1lvl 2 s (1 e)]lu| 12] Pr
| 2 -(1 e)]lu||?
(126)
|||| || 2 -(1 -(1-e)llul?
Pr
(127)
0.5m
= Pr [-|| - 00.mm( -
(128)
Using Chernoff's 's bound, we get any x > 0,
Pr (z(-111 2 01.5m( <)) = Pr (exp(->1\113) > exp(- 1001-100mmm1111 e)))
(129)
(130)
exp(- 00.m-00m - +))
Elexp( Σm Ao;
(131)
exp(-0.5mX{ E))
IT E. E/exp( -x0;2)
(132)
exp(-0.5mL) - E))
Elexp(-XX0 -xE;
(133)
eexp(-0.5x11 -
Performing computations similar to those above compute the expectation term, we gel,
E Blexp(-Xe, 0.5
(134)
(1+2X)
Hence, we get,
0.5
Pr (1|vl12 < 0.5mm1 - -c) <
0.5
1
exp(0.5A(1
(135)
v(1 + 2X)
m
1
0.5
1+
exp(0.5A(1 -c))
(136)
(1 + 2A)
8

---

Under review as a conference paper at ICLR 2020
Similar the R.H.S. case, we find that À 0.5e
1-c approximately minimizes the failure probability,
Pr (1VV] & 0.5m(1 - c)) v 0.5 (1 v 1 expl )
(137)
c - log
= exp
(138)
4
1 VI
can be shown that,
vI )) Sexp
E -
c
exp
m
4
log
(mm
+ log
1 vI+
(139)
Thus,
Pr (Ilvl2 2 (1 (1-c)][u]]
e)|[u| () 2 1 exp -m
c + log
(140)
1 /1 +e
Using union bound, Eq. 725 and (44)) hold together with probability,
(-m ( (
2 1 v1++/ ))=
Pr ((1 ||ul) <
|v| s (1 + +ollu(33
2 - 11--2y 2exp
+ +og
(141)
This proves the claim.
Theorem Let D be a fixed dataset with N samples and define a layer ReLU network as shown in Eq. such that each weight matrix W Raxa has its elements sampled as W/ 1.idd NO 2) and biases b' are set to zeros. for any sample (x,y) E D. E [0, 1), and for all E 21, 2......... L} with probability at least,
4NLexp (-n (n((4
++ log 1 /1 21+e))
(142)
the following hold true,
al(Jo(x).g) OW
(1 - 11=c)111x112 110(x,
(1+c) -10((x
(143)
and
(1 (1-c'|x113 < |h'l (1 (1 [X|
(144)
Proof: From theorem
we know that the following holds for all I,
Pr ((1 c) s Ih's s (1+c) 2 2NL (-n ( + log
2
1 /1 +
(145)
On the other hand, we have that,
dl(fo(x).)) 1(at-1) o (WL" 6(x,y)) daL-1
(146)
From proposition] 1 we know that each element of 1(al-1) follows Bernoulli distribution with probability 0.5. Thus applying lemma to the above equation (under the assumption that 5(x,y) and 1(a') are statistically independent), the following holds for a fixed data sample (x.y).
Pr (1-c)]](xx>)11+ (l(fo(x).)) 0- 12 +=)1((xx,))11) 2 122xpp
(-n
2
log (147)
1 /1
+
Applying union bound on N fixed samples, the the following holds for all N samples,
dl(fo(x),))
(-n ( + log (4
2
Pr (1 (1-c)) (1x))))(xxxx)11
2Ncxp 1 < (1 + o)1(8(x,))11 2 2N exp
h-1
1 + vI +
(148)

---

Under review as a conference paper at ICLR 2020
Similarty.
Pr (1- de(fo(x),))
@e(Fo(x),))
-n1 ("((2tlog
0aL-1 1| <
1 (1 (1+ +)]l
12 2 1 2N exp
+ log
daL-2
Ōa daL-1
1 + vi
(149)
Combining the the above two results and applying union bound, we get, )e(f(xx,,))
Pr (11 (16xx)16(x.)) =
JaL-2 +1+)) 2 1- 4Nexp (-n (-n
( + log
2
1 vI +
(150)
Extending this to all L, layers, we have for all that,
Pr (1 (1=c)==110((xx)) )f(Jo(x),)) 12 (1 (1+0)=-100(( 2 21-2N- Lxp
-n (
+ log
Oal
1 vI+e
(151)
Combining the above result with Eq.
145) using union bound, we get for all
de (x).
Pr (1-c)
|| 121
11 = (1 - ()
118(x
Oal
2 1 4NL exp (-n 6 + +og
2
1 vI +
(152)
Since,
Dl(Jo(x),))
OW 112 [x([o(x),))
2 10
| 112
VI
(153)
da'
we have proved the claim.
Lemma (Corollary 2.1 of Kakade Shakhnarovich (2099) Let u1, U2 E R be any two fixed vectors such that |ui II < 1 and |ua < 1, R E gmx"" be a projection matrix where each element of is drawn i..dd from a standard Gaussian distribution, Raj NO, 1 and any e (0,1/2)
Pr(1 < Ruj, Rug >
U1 U2 {
m 2 et)
2 1 - 4exp
(154)
4
Lemma' Letvi = (Ru O and V2 = (Ruz)Oz, where u1 U2 E R" z E R", and R E R
If Riy s.i.d. [00, 0.5m z:
iid. Bernoulli(O.5) and e [0, 1). then for any fixed vectors un and u2
s.t. ||u||| || < and U12 < 1,
U1 > S 21 - 4exp (-m ( + log 1 v1 2++))
2
Pr( < V1, V2.
(155)
Proof: Applying lemma to vectors u1 + U2 and u u2 we have with probability least -m 1 + log Itvite
1 4exp
(1 <) ||| 12 < 1z Ruj +2 Rua 2 < (11+ (1+)). ||u U2l
(156)
(1 c). | uzll² 2 <
Ruj O Rug 1 < (1 +c)
||u ug u2ll
(157)
Then notice,
4 < V1, V2 = 4< z
Ru Z ① Rug >
(158)
= ||z C Ru; z 3 Rug 2 | O Ru; - z & Ruz
(159)
2 (1 - <) || + U2 2 (1 + e)- ||u - u2ll
(160)
= 4. < in
2 ([|u, + |
(161)
2 4. U1, U1 > 44c
(162)
Equivalently,
< U1 U2 >- < ><
(163)
The other side of the claim can be proved similarly.
10


Reviews:
[{"review_id": "SkxEBvuUcS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers random parameter initialization in neural networks (In particular the initialization presented in He et al.) and develops non-asymptotic bounds for the norms and gradients of neural networks during initialization. The authors show that the norms of the outputs and gradients (for gradients, under a different assumption on the dimension of the matrix) remain constant through the different layers. The results presented differ from previous work in that they give nice concentration bounds for such output and gradient norms. In addition the authors prove results in the case of infinite samples under the assumption that they arise from a finite dimensional space.\n\nOverall the presentation is clean, but the results presented have the following issues.\n\n1.Very similar in nature to previous results, for example the results presented in [1] Theorem 5.4 give very similar concentration results and use very similar mechanisms.\n\n2. All the results for the infinite stream coming from a finite dimensional subspace do not address the fact that the training points usually do not lay in a linear subspace of dimension d << n. Further, there is a stringent requirement on d in such results and they fail to hold for even \"fairly\" small d.\n\n3. In Theorem 2, the argument that the output through a layer of a rank-d linear subspace remains within a rank-d linear subspace does not seem correct, is it possible that it remains in the union of subsets each of which lies in a subspace of rank d?\n\n4. The proofs for the gradients make assumptions that deviate from the initialization previously introduced. The work of Glorot et al. for example discusses the tradeoffs between the two assumptions and suggests the balancing between maintaining the input and output variance distributions.\n\nDue to the following issues I choose to reject this work at this time.\n\nBelow are additional minor typos or issues in the paper.\nOn page 6, the sentence that starts with: \"it is beneficial for the width\": Suggesting to use neural networks of constant width seems a bit impractical.\n\nThe citation of Arpit et al. from 2017 seems possibly wrong since the paper cited discusses memorization and does not focus at the initialization of neural networks.\n\n\nTYPOS:\npage 1 last sentence before equation 1:  back backward? \n\npage 1 last sentence: repeated words: as as the the\n\npage 1 last sentence: property --> properties?\n\npage 2 section 2 paragraph 1: both these papers --> both papers, both of these papers?\n\npage 4 sentence after the proof sketch paragraph: the sentence is difficult to read\n\n[1] \"Stochastic gradient descent optimizes over-parameterized deep relu networks\", 2018, Difan Zou, Yuan Cao, Dongruo Zhou, Quanquan Gu\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkgXIteScr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies initialization techniques for deep ReLU networks from a theoretical standpoint and derives finite layer width concentration bounds to show that with the He initialization scheme, deep ReLU networks preserve the norm of the input sample during a forward pass and the norm of the gradient with respect to the output during a backward pass. The concentration bounds also suggest lower bounds on the width of the ReLU layers. The authors verify their theory with experiments on synthetic data.\n\nWhile I believe the finite sample concentration bounds for networks initialized using the He initialization are valuable, it seems to me that this work is incremental in terms of understanding the He initialization. The techniques used in the paper are also fairly well known Chernoff bounding techniques and concentration of Gaussian random vectors and matrices.\n\nThe authors claims on explaining overparameterization are also overstated in my opinion. While the authors are able to obtain a lower bound on layer widths in order to preserve norms during forward/backward passes at initialization, the lower bound is only dependent on the input dimension and not the size of the dataset, which is the relevant quantity to decide whether a model is under/over parameterized. Even in the authors' bounds for finite datasets, what I can surmise from their results (it would be better to explicitly state it if that is one of the goals of this paper) is that the width of each layer needs to be atleast log(N) where N is the size of the dataset. This is hardly overparameterized. \n\nFurthermore, the authors do not explain how studying the properties of the initialization might help understand generalization at minima. Since gradient descent based techniques seem to prefer solutions that are close to initialization, the analysis in this paper might be a useful starting point in understanding generalization.\n\nThe authors could also consider how adding BatchNorm layers and/or Residual connections affect the He initialization scheme, and whether initialization matters for those techniques. Finite width concentration bounds for initialization of networks using Batchnorm/Residual connections could be useful. \n\nTo summarize, I do not see how the authors claims about explaining overparameterization (even at initialization) can be made. Without those claims the contribution of this paper is incremental and does not warrant publication at this time. I am willing to adjust my score if the claims about overparameterization are stated explicitly and make sense.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkx9eVWAYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper shows that under He initialization and for sufficiently wide network, (1) the norm of the activations of a L layered ReLU is preserved w.r.t the input across layers and (2) the norm of a weight matrix gradient at different layer is only dependent on the norm of the top-layer error and the input, because the norm of back-propagated gradient is approximately preserved. \n\nThe paper is clearly written and easy to read and the proofs are quite straightforward. That being said, the results are not surprising and from my point of view, the overall novelty of this paper is a bit marginal for top-tier conference like ICLR. \n\nThus I vote for rejection. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkxInWuatH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the norm of hidden activation of each layer and the norm of weight gradient of each layer for deep ReLU neural network. By using concentralization property of the random initialization, the paper derives their expected values and high probability range when the network width is sufficiently wide. The results are correct and the paper is easy to follow. However, the result has been given in previous work. I do not recommend the acceptance.\n\nThe result presented in this paper has been covered by a recent work [1]. Please refer to Section 7.1 for the forward part and Section 7.3 for the backward part.\n\n[1] Zeyuan Allen-Zhu, Yuanzhi Li and Zhao Song. A Convergence Theory for Deep Learning via Over-Parameterization", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The article studies benefits of over-parametrization and theoretical properties at initialization in ReLU networks. The reviewers raised concerns about the work being very close to previous works and also about the validity of some assumptions and derivations. Nonetheless, some reviewers mentioned that the analysis might be a starting point in understanding other phenomena and made some suggestions. However, the authors did not provide a rebuttal nor a revision. 
Decision: Reject