On the Effect of the Activation Function
on the Distribution of Hidden Nodes
in a Deep Network
Philip M. Long∗ and Hanie Sedghi∗
Google Brain

Abstract
We analyze the joint probability distribution on the lengths of the vectorsof hidden variables in different layers of a fully connected deep network,when the weights and biases are chosen randomly according to Gaussian
distributions. We show that, if the activation function φ satisfies a minimal
set of assumptions, satisfied by all activation functions that we know thatare used in practice, then, as the width of the network gets large, the “length
process” converges in probability to a length map that is determined as asimple function of the variances of the random weights and biases, and theactivation function φ. We also show that this convergence may fail for φthat violate our assumptions. We show how to use this analysis to choose
the variance of weight initialization, depending on the activation function,
so that hidden variables maintain a consistent scale throughout the network.
Keywords: Initialization, theory, stability.

1

Introduction

The size of the weights of a deep network must be managed delicately. If they
are too large, signals blow up as they travel through the network, leading tonumerical problems, and if they are too small, the signals fade away. The practicalstate of the art in deep learning made a significant step forward due to schemes
for initializing the weights that aimed in different ways at maintaining roughly
the same scale for the hidden variables before and after a layer (LeCun et al.,
1998; Glorot and Bengio, 2010). Later work (He et al., 2015; Poole et al., 2016;
Daniely et al., 2016) took into account the effect of the non-linearities on the length
dynamics of a deep network, informing initialization policies in a more refined way.
An influential theoretical analysis (Poole et al., 2016) considered whether signals tend to blow up or fade away as they propagate through a fully connected
network with the same activation function φ at each hidden node. For a given
input, they studied the probability distribution over the lengths of the vectors of∗

Authors ordered alphabetically.

1

hidden variables, when the weights between nodes are chosen from a zero-mean
Gaussian with variance σw2 /N , and where the biases are chosen from a zero-mean
distribution with variance σb2 . They argued that, in a fully-connected network,
as a width of the network approaches infinity, the (suitably normalized) lengthsof the hidden layers approach a sequence of values, one for each layer, and characterized this length map as a function of φ, σw and σb . This analysis has since
been widely used (Schoenholz et al., 2016; Yang and Schoenholz, 2017; Pennington
et al., 2017; Lee et al., 2018; Xiao et al., 2018; Chen et al., 2018; Pennington et al.,2018; Hayou et al., 2018).
Poole et al. (2016) claimed that their analysis holds for arbitrary non-linearities
φ. In contrast, we show that, for arbitrarily small positive σw , even if σb = 0, forφ(z) = 1/z, the distribution of values of each of the hidden nodes in the second
layer diverges as N gets large. For finite N , each node has a Cauchy distribution,
which already has infinite variance, and as N gets large, the scale parameter of
the Cauchy distribution gets larger, leading to divergence. We also show that thehidden variables in the second layer may not be independent, even for commonlyused φ like the ReLU, contradicting a claim that is part of the analysis of (Pooleet al., 2016).
These observations, together with the wide use of the length map from (Pooleet al., 2016), motivate the search for a new analysis. This note provides such ananalysis for activation functions φ that satisfy the following properties: (a) the
restriction of φ to any finite interval is bounded; (b) as z gets large,1 |φ(z)| ≤exp(o(z 2 )), (c) φ is measurable. We refer to such φ as permissible. Note thatconditions (a) and (c) both hold for any non-decreasing φ.
We show that, for all permissible φ and all σw and σb , as N gets large, thelength process converges in probability to the length map described in (Pooleet al., 2016).
Section 5 describes some simulation experiments verifying some of the findingsof the paper, and illustrating the dependence among the values of the hidden
nodes.
Section 6 describes one way to use our analysis to choose the variance of theweights depending on the activation function so that signals neither blow up nor
vanish as computation flows through a wide and deep network.
Our analysis of the convergence of the length map borrows ideas from (Daniely
et al., 2016), who studied the properties of the mapping from inputs to hidden
representations resulting from random Gaussian initialization. Their theory applies in the case of activation functions with certain smoothness properties, andto a wide variety of architectures. Informally, they showed that, after random
initialization, for wide networks, it is likely that the kernel associated with feature
map computed by the network closely approximates a fixed kernel. Our analysis
treats a wider variety of values of σw and σb , and uses weaker assumptions on φ.
Motivated by Bayesian goals as in (Neal, 1996), Matthews et al. (2018) performed
an analysis in a related setting, characterizing the distribution of kernels arising1

Here o(z 2 ) denotes any function of z that grows strictly more slowly than z 2 , such as z 2−for  > 0.

2

from a random initialization. Their analysis used a “linear envelope” condition on
φ that is stronger than the assumption used here. Alternative but related uses oftheory to guide the choice of weight variances may be found in (Schoenholz et al.,2016; Pennington et al., 2017). Hanin (2018) studied the effect of the widths oflayers and the depth of a fully connected network on the size of the input-output
Jacobian in the case of ReLU activations.

2

Preliminaries

2.1

Notation

For n ∈ N, we use [n] to denote the set {1, 2, . . . , n}. If T is a n × m × p tensor,
then, for i ∈ [n], let Ti,:,: be the matrix A such that Aj,k = Ti,j,k , and define Ti,j,: ,etc., analogously.

2.2

The finite case

Consider a deep fully connected width-N network with D layers. Let W ∈RD×N ×N . An activation function φ maps R to R; we will also use φ to denote thefunction from RN to RN obtained by applying φ componentwise. Computation of
the neural activity vectors x0,: , ..., xD,: ∈ RN and preactivations h1,: , ..., hD,: ∈ RN
proceeds in the standard way as follows:h`,: = W`,:,: x`−1,: + b`,:x`,: = φ(h`,: ),for ` = 1, . . . , D.

We will study the process arising from fixing an arbitrary input x0,: ∈ RN andchoosing the parameters
independently at random: the entries of W are sampled
2
σwfrom Gauss 0, N , and the entries of b from Gauss (0, σb2 ). For each ` ∈ {0, ..., D},P2
define q` = N1 N
i=1 h`,i .
Note that for all ` ≥ 1, all the components of h`,: and x`,: are identically
distributed.

2.3

The wide-network limit

For the purpose of defining a limit, assume that, for a fixed, arbitrary function
χ : N → R,
N , we have x0,: = (χ(1), ..., χ(N )). We also assume that
Pfor finite
2
limN →∞ N1 ∞χ(i)existsand is nonzero. For ` > 0, let x` be a random variable
i=1
whose distribution is the limit of the distribution of x`,1 as N goes to infinity, if
this limit exists (in the sense of “convergence in distribution”). Define h` and q `similarly.

2.4

Total variation distanceIf P and Q are probability distributions, then dRT V (P, Q) = supE P (E) − Q(E),and if p and q are their densities, dT V (P, Q) = 21 |p(x) − q(x)| dx.
3

2.5

Permissible activation functions

Definition 1 An activation function φ is permissible if,• the restriction of φ to any finite interval is bounded• |φ(x)| = exp(o(x2 )) as |x| gets large.2 , and• φ is measurable.
Conditions (b) and (c) ensure that a key integral can be computed. The proof
of Lemma 1 is in Appendix A.
Lemma 1 If φ is permissible, then, for all positive constants c, the function gdefined by g(x) = φ(cx)2 exp(−x2 /2) is integrable.

2.6

Length map

Next we recall the definition of a length map from (Poole et al., 2016); we will
prove that the the length process converges to this length map.
q̃1 , ..., q̃D
P Define
2
.
Then,
for
x
and r̃0 , ..., r̃D recursively as follows. First r̃0 = limN →∞ N1 N
i=1 0,i` > 0,q̃` = σw2 r̃`−1 + σb2
andpr̃` = Ez∈Gauss(0,1) [φ( q̃` z)2 ].
If φ is permissible, then, since φ(cz)2 exp(−z 2 /2) is integrable for all c, we havethat q̃0 , ..., q̃D , r̃0 , ..., r̃D are well-defined finite real numbers.

3

Some surprising behaviors

In this section, we show that, for some activation functions, the probability distribution of hidden nodes can have some surprising properties.

3.1

Failure to converge

In this subsection, we will show that the probability distribution of the hidden
variables may not converge. Our proof will refer to the Cauchy distribution.
Definition 2 A distribution over the reals that, for x0 ∈ R and γ > 0, has adensity f given by f (x) = h 1x−x0 2 i is a Cauchy distribution, denoted by
πγ 1+( γ )Cauchy(x0 , γ). Cauchy(0, 1) is the standard Cauchy distribution.
Lemma 2 ((Hazewinkel, 2013))
Pn If X1 , ..., Xn are i.i.d. random variables with1
a Cauchy distribution, then n i=1 Xi has the same distribution.
2

This condition may be expanded as follows,
limsupx→−∞ log |φ(x)|
= 0.
x2

4

limsupx→∞ log |φ(x)|
x2

=

0 and

Lemma 3 ((Lupton, 1993)) If U and V are zero-mean normally distributed
random variables with the same variance, then U/V has the standard Cauchy
distribution.
The following shows that there is a φ such that the limiting h2 is not defined.
It contradicts a claim made on line 7 of Section A.1 of (Poole et al., 2016).
Proposition 4 For any input function χ with range {−1, 1}, there is an activation function φ such that, for every σw > 0, if σb = 0, then (a) for finite N , h2,1
has infinite variance, and (b) h2,1 diverges as N goes to infinity.
Proof: Consider φ defined byφ(y) =

1/y if y 6= 00
if y = 0.

Fix a value of N and σw > 0, and take σb = 0. Each component of h1,: is a sumof zero-mean Gaussians with variance σw2 /N ; thus, for all i, h1,i ∼ Gauss(0, σw2 ).
Now, almost surely,
h2,1 =NX

W2,1,j φ(h1,j ) =

N
X

j=1

W2,1,j /h1,j .

j=1

By Lemma 3, for each j, W2,1,j /h1,j has a Cauchy distribution, and since(N W2,1,1 ), ..., (N W2,1,N ) ∼ Gauss(0, N σw2 ),recalling that h1,1 , ..., h1,N ∼ Gauss(0, σw2 ), we have that
N W2,1,1 /h1,1 , ..., N W2,1,N /h1,N√
are i.i.d. Cauchy(0, N ). Applying Lemma 2,
N
X

N

1 X
h2,1 =
W2,1,j φ(h2,j ) =N W2,1,j φ(h1,j )
N
j=1
j=1
√
is also Cauchy(0, N ).
√So, for all N , h2,1 is Cauchy(0, N ). Suppose that h2,1 converged in distribution to some distribution P . Since the cdf of P can have at most countably
many discontinuities, we can cover the real line by a countable set of finite-lengthintervals [a√1 , b1 ], [a2 , b2 ], ... whose endpoints are points of continuity for P . Since
Cauchy(0, N ) converges to P in distribution, for any i,
P ([ai , bi ]) ≤ lim

N →∞|bi − ai |
√
= 0.
π N

Thus, the probability assigned by P to the entire real line is 0, a contradiction.

5

3.2

Independence

The following contradicts a claim made on line 8 of Section A.1 of (Poole et al.,2016).
Theorem 5 If φ is either the ReLU or the Heaviside function, then, for every
σw > 0, σb ≥ 0, and N ≥ 2, (h2,1 , ..., h2,N ) are not independent.
Proof: We will show that E[h22,1 h22,2 ] 6= E[h22,1 ]E[h22,2 ], which will imply that h2,1
and h2,2 are not independent.
As mentioned earlier, because each component of h1,: is the dot product of
x0,: with an independent row of W1,:,: plus an independent component of b1,: , thecomponents of h1,: are independent, and since x1,: = φ(h1,: ), this implies that thecomponents of x1,: are independent. Since each row of W1,:,: and each componentof the bias vector has the same distribution, x1,: is i.i.d.
We have


2 
X
E[h22,1 ] = E 
W2,1,i x1,i  + b2,1  i∈[N ]=X

E [W2,1,i W2,1,j x1,i x1,j ] + 2(i,j)∈[N ]2X E [W2,1,i x1,i b2,1 ] + E b22,1 .

i∈[N ]

The components of W2,:,: and x1,: , along with b2,1 , are mutually independent,
so terms in the double sum with i 6= j have zero expectation, and E[h22,1 ] =P 2   2 E
W2,1,i E x1,i + E[b22,1 ]. For a random variable x with the same disi∈[N ]tribution as the components of x1,: , this implies 
E[h22,1 ] = σw2 E x2 + σb2 .
(1)

6

Similarly,
E[h22,1 h22,2 ]

2 
2 
X
X
= E 
W2,1,i x1,i + b2,1  
W2,2,i x1,i + b2,2  i∈[N ]

X=

i∈[N ]

E[W2,1,i W2,1,j W2,2,r W2,2,s x1,i x1,j x1,r x1,s ](i,j,r,s)∈[N ]4

X+2

E[W2,1,i W2,1,j W2,2,r x1,i x1,j x1,r b2,2 ](i,j,r)∈[N ]3

X+2

E[W2,1,i W2,2,r W2,2,s x1,i x1,r x1,s b2,1 ](i,r,s)∈[N ]3+4

X

E[W2,1,i W2,2,r x1,i x1,r b2,1 b2,2 ](i,r)∈[N ]2

X+(i,j)∈[N ]2

+2

X

X

E[W2,1,i W2,1,j x1,i x1,j b22,2 ] +E[W2,2,r W2,2,s x1,r x1,s b22,1 ](r,s)∈[N ]2

E[W2,1,i x1,i b2,1 b22,2 ] + 2i∈[N ]

X

E[W2,2,r x1,r b22,1 b2,2 ]r∈[N ]+ E[b22,1 b22,2 ]
X

=

2
2
E[W2,1,iW2,2,r]E[x21,i ]E[x21,r ] +(i,r)∈[N ]2 ,i6=r+X

2
E[W2,1,i]E[x21,i ]E[b22,2 ] +i∈[N ]

X

2
2
E[W2,1,iW2,2,i]E[x41,i ]i∈[N ]

X

2
E[W2,2,r]E[x21,r ]E[b22,1 ]r∈[N ]+ E[b21,2 b22,2 ](N 2 − N )σw4 E[x2 ]2 N σw4 E[x4 ] 2N σw2 E[x2 ]σb2
+
+
+ σb4
N2
N2
N
σ 4 (E[x4 ] − E[x2 ]2 )= σw4 E[x2 ]2 + w+ 2σw2 σb2 E[x2 ] + σb4 .
N=

Putting this together with (1), we haveE[h22,1 h22,2 ] − E[h22,1 ]E[h22,2 ] =σw4 (E[x4 ] − E[x2 ]2 )
.
N

(2)

Now, we calculate the difference using (2) for the Heaviside and ReLU functions.
Heaviside. Suppose φ is Heaviside function, i.e. φ(z) is the indicator function
for z > 0. In this case, since the components of h1,: are symmetric about 0, thedistribution of x1,: is uniform over {0, 1}N . Thus E[x4 ] = E[x2 ] = 1/2, and so (2)4w
gives E[h22,1 h22,2 ] − E[h22,1 ]E[h22,2 ] = 3σ6= 0.
4NReLU. Next, we consider the case that φ is the ReLU.
that, for all Recalling

R
2
∞
1
−z
i, h1,i ∼ Gauss(0, σw2 ), we have E[x2 ] = √ 2 0 z 2 exp 2σ2 dz. By symmetry
2πσw

7

w

4

this is 12 Ez∼Gauss(0,σw2 ) [z 2 ] = σw2 /2. Similarly, E[x4 ] = 21 Ez∼Gauss(0,σw2 ) [z 4 ] = 3σ2 .
Plugging these into (2) we get that, in the case the φ is the ReLU, that
5σw8
σw4 ((3/2)σw4 − σw4 /4)
2
2
2
2
=
> 0,
E[h2,1 h2,2 ] − E[h2,1 ]E[h2,2 ] =N

4Ncompleting the proof.
Note that, informally, the degree of dependence between pairs of hidden nodes
established in the proof of Theorem 5 approaches 0 as N gets large. On the other
hand, the number of dependent pairs of hidden nodes is Ω(N 2 ).

3.3

Undefined length map

Here, we show, informally, that for φ at the boundary of the second condition in
the definition of permissibility, the recursive formula defining the length map q̃`breaks down. Roughly, this condition cannot be relaxed.
Proposition 6 For any α > 0, if φ is defined by φ(x) = exp(αx2 ), even if allcomponents of all inputs are in {−1, 1}, there exists a σw , σb s.t. q̃` , r̃` is undefinedfor all ` ≥ 2.
Proof: Suppose σw2 + σb2 = 4α1 2 . Then q̃1 = 4α1 2 , so that
 2
 2
Z ∞ p
Z ∞
p 2
1
z
z
1
r̃1 = √φ( q̃1 z) exp −exp(α q̃1 z ) exp −dz = √
dz
2
2
2π −∞
2π −∞
 2
Z ∞
1
z
2
=√
exp(z /2) exp −dz = ∞,
22π −∞and downsteam values of q̃` and r̃` are undefined.

4

Convergence in probability

In this section that the length process q0 , ..., qD converges in probability to thelength map q̃0 , ..., q̃D from (Poole et al., 2016).
Theorem 7 For any permissible φ, σw , σb ≥ 0, any depth D, and any , δ > 0,
there is an N0 such that, for all N ≥ N0 , with probability 1 − δ, for all ` ∈ [D],we have |q` − q̃` | ≤ .
The rest of this section is devoted to proving Theorem 7. Our proof will use
the weak law of large numbers.
Lemma 8 ((Feller, 2008)) For any random variable X with a finite expectation,
and any , δ > 0, there is an N0 such that, for all N ≥ N0 , if X1 , ..., XN are i.i.d.
with the same distribution as X, then!
N
1 X
Pr E[X] −Xi >  ≤ δ.
N i=1
8

In order to divide our analysis into cases, we need the following lemma, whose
proof is in Appendix B.
Lemma 9 If φ is permissible and not zero a.e., for all σw > 0, for all `, q̃` > 0
and r̃` > 0.
We will also need a lemma that shows that small changes in σ lead to small changes
in Gauss(0, σ 2 ).
Lemma 10 (see (Klartag, 2007)) There is an absolute constant C such that,for all σ1 , σ2 > 0,
dT V (Gauss(0, σ12 ), Gauss(0, σ22 )) ≤ C|σ1 − σ2 |.
σ1

The following technical lemma, which shows that tail bounds hold uniformly overdifferent choices of q, is proved in Appendix C.
Lemma 11 If φ is permissible, for all 0R < r ≤ s, for all β > 0, there is√∞an a ≥ 0 such that, for all q ∈ [r, s], a φ( qz)2 exp(−z 2 /2) dz ≤ β andR −a √ 2
φ( qz) exp(−z 2 /2) dz ≤ β.
−∞Armed with these lemmas, we are ready to prove Theorem 7.
First, if φ is zero a.e., or if σw = 0, Theorem 7 follows directly from Lemma 8,
together with a union bound over the layers. Assume for the rest of the proof that
φ(x) is non-zero on a set of positive measure, and that σw > 0, so that q̃` > 0 andr̃` > 0 for all `.
P
2
For each ` ∈ {0, ..., D}, define r` = N1 N
i=1 x`,i .
Our proof of Theorem 7 is by induction. The inductive hypothesis is that, forany , δ > 0 there is an N0 such that, if N ≥ N0 , then, with probability 1 − δ, forall `0 ∈ {1, ..., `}, |q`0 − q̃`0 | ≤  and, for all `0 ∈ {0, ..., `}, and |r`0 − r̃`0 | ≤ .
The base case, where ` = 0, holds because r̃0 is definied to be the limit of r0
as N goes to infinity.
Now for the induction step; choose ` > 0, 0 <  < min{q̃` /4, r̃` } and 0 <δ ≤ 1/2. (Note that these choices are without loss of generality.) Let 0 ∈ (0, )
take a value that will be described later, using quantities from the analysis. By
the inductive hypothesis, whatever the value of 0 , there is an N00 such that, if
N ≥ N00 , then, with probability 1 − δ/2, for all `0 ≤ ` − 1, we have |q`0 − q̃`0 | ≤ 0
and |r`0 − r̃`0 | ≤ 0 . Thus, to establish the inductive step, it suffices to show that,after conditioning on the random choices before the `th layer, if |r`−1 − r̃`−1 | ≤ 0 ,there is an N` such that, if N ≥ N` , then with probability at least 1 − δ/2with respect only to the random choices of W`,:,: and b`,: , that |q` − q̃` | ≤  and|r` − r̃` | ≤ . Given such an N` , the inductive step can be satisfied by letting N0
be the maximum of N00 and N` .
Let us do that. To simplify the notation, for the rest of the proof of the inductive step, let us condition on outcomes of the layers before layer `; all expectations
and probabilities will concern the randomness only in the `th layer. Let us further
assume that |r`−1 − r̃`−1 | ≤ 0 .
9

P
2
Recall that q` = N1 N
i=1 h`,i . Since the values of h`−1,1 , ..., h`−1,N have been
fixed by conditioning, each component of h`,i is obtained by taking the dot-productof x`−1,: = φ(h`−1,: ) with W`,i,: and adding an independent b`,i . Thus, conditionedon h`−1,1 , ..., h`−1,N , we have that h`,1 , ..., h`,N are independent. Also, since x`−1,:
is fixed by conditioning, each h`,i has an identical Gaussian distribution.
Since each component of W and b has zero mean, each h`,i has zero mean.
Choose an arbitrary i ∈ [N ]. Since x`−1,: is fixed by conditioning andW`,i,1 , ..., W`,i,Nand b`,i are independent,
σw2 X 2def
2
2
E[q` ] = E[h`,i ] = σb +x`−1,j = σb2 + σw2 r`−1 = q ` .
N j

(3)

We wish to emphasize the q ` is determined as a function of random outcomes beforethe `th layer, and thus a fixed, nonrandom quantity, regarding the randomizationof the `th layer. By the inductive hypothesis, we have|E[q` ] − q̃` | = |E[h2`,i ] − q̃` | = |q ` − q̃` | = σw2 |r`−1 − r̃`−1 | ≤ 0 σw2 .

(4)

The key consequence of this might be paraphrased by saying that, to establish theportion of the inductive step regarding q` , it suffices for q` to be close to its mean.
Now, we want to prove something similar for r` . We have
N
N
1 X
1 X
2
E[x`,i ] =E[φ(h`,i )2 ] = E[φ(h`,1 )2 ],E[r` ] =
N i=1
N i=1

since, recalling that we have conditioned on previous layers, h`,1 , ..., h`,N are i.i.d.
Since h`,i ∼ Gauss(0, q ` ), we haveE[r` ] = Ez∼Gauss(0,q` ) [φ(z)2 ]p= Ez∼Gauss(0,1) [φ( q ` z)2 ]r Zp1
φ( q ` z)2 exp(−z 2 /2) dz=
2πwhich gives|E[r` ] − r̃` | ≤ Ez∼Gauss(0,q` ) [φ(z)2 ] − Ez∼Gauss(0,q̃` ) [φ(z)2 ] .
Since |q ` − q̃` | ≤ 0 σw2 and we may choose 0 to ensure 0 ≤ 2σq̃`2 , we have q̃` /2 ≤w
q ` ≤ 2q̃` .
For β > 0 and κ ∈ (0, 1/2) to be named later, by Lemma 11, we can choose asuch that, for all q ∈ [q̃` /2, 2q̃` ],
Z −a
Z ∞
√ 2
√
2
φ( qz) exp(−z /2) dz ≤ β/2 andφ( qz)2 exp(−z 2 /2) dz ≤ β/2
−∞

a

10

 2
exp
− z2q dz ≥ 1 − κ. Choose such an a.
−a
Ra
R √
√
We claim that −a φ( qz)2 exp(−z 2 /2) dz − φ( qz)2 exp(−z 2 /2) dz ≤ β
for all q̃` /2 < q ≤ 2q̃` . Choose such a q. We have
Z a
Z
√ 2
√
2
φ( qz) exp(−z /2) dz − φ( qz)2 exp(−z 2 /2) dz
−a
Z −a
Z ∞
√ 2
√
2
φ( qz) exp(−z /2) dz +φ( qz)2 exp(−z 2 /2) dz
=
−∞
a
Z −a

Z ∞
√ 2
√ 2
2
2
φ( qz) exp(−z /2) dz≤ 2 maxφ( qz) exp(−z /2) dz,
1and √2πq

Ra

−∞

a

≤ β.

So now we are trying to bound
Z a
Z a p
p
2
2
φ( q ` z) exp(−z /2) dz −φ( q̃` z)2 exp(−z 2 /2) dz−a−ausing q̃` /2 ≤ q ` ≤ 2q̃` .
Using changes of variables, we have
Z a
Z a p
p
2
2
φ( q ` z) exp(−z /2) dz −φ( q̃` z)2 exp(−z 2 /2) dz
−a
−a




Z a√q`
Z a√q̃`
2
1
z
1
z2
2
2
= √
φ(z) exp −dz − √
φ(z) exp −dz .
q ` −a√q`2q `2q̃`q̃` −a√q̃`√√
Since φ is permissible, φ2 is bounded on [−a 2q̃` , a 2q̃` ]. If P is the distribution
√
√
q
)
on
[−a
q
,
a
q ` ], and P̃ by conditioning
obtained by conditioning
Gauss(0,``√√√Gauss(0, q̃` ) on [−a q̃` , a q̃` ], then if M = 2π supz∈[−a√2q̃` ,a√2q̃` ] φ(z)2 , sinceq ` ≤ 2q̃` ,Z a√q`
Z a√q̃`
2
1
z
1
z2
2
2
√
√
φ(z)
exp(−
)
dz
−
φ(z)
exp(−
) dz
q ` −a√q`2q `2q̃`q̃` −a√q̃`≤ M dT V (P, P̃ ).
But since, for κ < 1/2, conditioning on an event of probability at least 1 − κonly changes a distribution by total variation distance at most 2κ, and therefore,
applying Lemma 10 along with the fact that |q ` − q̃` | ≤ 0 σw2 , for the constant C

11

from Lemma 10, we get
dT V (P, P̃ ) ≤ 4κ + dT V (Gauss(0, q ` ), Gauss(0, q̃` ))√√C| q ` − q̃` |√≤ 4κ +q̃`
C|q ` − q̃` |√ √= 4κ + √| q ` + q̃` | q̃`C0 σw2.
≤ 4κ +
q̃`

Tracing back, we have
Z a
Z a p
p
2
2
φ( q ` z) exp(−z /2) dz −φ( q̃` z)2 exp(−z 2 /2) dz
−a
−a


0 2
C σw
≤ M 4κ +q̃`which implies
Z

Z
p
p
2
2
|E[r` ] − r̃` | ≤φ( q ` z) exp(−z /2) dz − φ( q̃` z)2 exp(−z 2 /2) dz
C0 σw2≤ M 4κ +
+ 2β.
q̃`
n
o
q̃` 

, 13 }, β = 12 , and 0 = min 2 , 2σ2 , 2σq̃`2 , 6CM
this implies |E[r` ]−If κ = min{ 24M2σwww
r̃` | ≤ /2.
Recall that q` is an average of N identically distributed random variables witha mean between 0 and 2q̃` (which is therefore finite) and r` is an average of N
identically distributed random variables, each with mean between 0 and r̃` +/2 ≤2r̃` . Applying the weak law of large numbers (Lemma 8), there is an N` suchthat, if N ≥ N` , with probability at least 1 − δ/2, both |q` − E[q` ]| ≤ /2 and|r` − E[r` ]| ≤ /2 hold, which in turn implies |q` − q̃` | ≤  and |r` − r̃` | ≤ ,
completing the proof of the inductive step, and therefore the proof of Theorem 7.

5

Experiments

Our first experiment fixed x[0, :] = (1, ..., 1), σw = 1, σb = 0, φ(z) = 1/z.
For each N ∈ {10, 100, 1000}, we (a) initialized the weights 100 times,√(b)plotted the histograms of all of the values of h[2, :], along with the Cauchy(0, N )distribution from the proof of Proposition 4, and Gauss(0,√σ 2 ) for σ estimatedfrom the data. Consistent with the theory, the Cauchy(0, N ) distribution fits
the data well.
To illustrate the fact that the values in the second hidden layer are not independent, for N = 1000 and the parameters otherwise as in the other experiment,
12

0.040

0.10
0.08
0.06
0.04
0.02
0.00

−60

−40

−20

0

Activations

20

40

0.012

0.035

Normalized counts/densities

Normalized counts/densities

Normalized counts/densities

0.12

0.030
0.025
0.020
0.015
0.010
0.005
0.000
−200 −150 −100 −50

60

0

50

Activations(a) N = 10

100

150

0.010
0.008
0.006
0.004
0.002
0.000

200

(b) N = 100

−600

−400

−200

0

Activations

200

400

600

(c) N = 1000

Figure 1: Histograms of h[2, :], averaged√ over 100 random initializations, for2 N ∈{10, 100, 1000}, along with Cauchy(0, N ) (shown in red) and Gauss(0, σ ) forσ estimated from the data (shown in green). When we average over multiple
random initializations of the weights, the distribution of the activations matches
the Cauchy distribution, and not the Gaussian.

0.006
0.004
0.002
−400

−200

0

Activations

200

400

0.014
0.012
0.010
0.008
0.006
0.004
0.002
0.000

−600

−400

−200

0

Activations

200

400

0.012
0.010
0.008
0.006
0.004
0.002
0.000

−600

−400

−200

0

Activations

200

400

600

0.008
0.006
0.004
0.002
−600

−400

−200

0

Activations

200

400

0.012
0.010
0.008
0.006
0.004
0.002
−600

−400

−200

0

Activations

200

400

0.012
0.010
0.008
0.006
0.004
0.002
0.000

−600

−400

−200

0

Activations

200

400

600

0.012
0.010
0.008
0.006
0.004
0.002
−600

−400

−200

0

200

400

600

−600

−400

−200

0

200

400

600

−600

−400

−200

0

200

400

600

Activations

0.014
0.012
0.010
0.008
0.006
0.004
0.002
0.000

600

0.014

0.014

0.000

600

0.014

0.000

600

0.014

0.010

0.000

600

Normalized counts/densities

Normalized counts/densities−600

Normalized counts/densities

0.008

0.012

Normalized counts/densities

0.010

0.014

Normalized counts/densities

0.012

0.000

Normalized counts/densities

Normalized counts/densities

0.014

Normalized counts/densities

Normalized counts/densities

we plotted histograms of the values seen in the second layer for nine random initializations of the weights in Figure 2. When some of the values in the first hidden
layer have unusually small magnitude, then the values in the second hidden layer
coordinately tend to be large. Note that this is consistent with Theorem 7 es-

Activations

0.014
0.012
0.010
0.008
0.006
0.004
0.002
0.000

Activations

Figure 2: Histograms of h[2, :] for nine random weight initializations. Plotting
activations separately for different random initializations reveals the dependence
among the activations in a layer.
tablishing convergence in probability for permissible φ, since the φ used in thisexperiment is not permissible.

13activation input variance weight variance
Identity
1
1
ReLU
1/2
2
Heaviside
1/2
2
Exponential
e2
1/e2
Tanh
0.394
2.53
Table 1: Choices of input variance (r0 ) and weight variances (σw2 ) that theory
suggests will promote an invariant that the preactivations maintain a constantscale as computation flows through the network.

6

Maintaining unit scale

In this section, we describe one use of our analysis to guide the design of initialization variances.
Our analysis shows that q1 ≈ σw2 r0 + σb2 , and√q`+1 ≈ σw2 Ez∈Gauss(0,1) [φ( q` z)2 ] + σb2 .
If we achieve
1 ≈ σw2 r0 + σb2
and
1 = σw2 Ez∈Gauss(0,1) [φ(z)2 ] + σb2 ,this will promote q1 ≈ 1, q2 ≈ 1, q3 ≈ 1, and so on. Setting
σw2 = 1/Ez∈Gauss(0,1) [φ(z)2 ], r0 = Ez∈Gauss(0,1) [φ(z)2 ], σb2 = 0
satisfies both. These values are collected from some common activation functionsin Table 1.

7

Conclusion

We have given a rigorous analysis of the limiting value of the distribution of thelengths of the vectors of hidden nodes in a fully connected deep network, anddescribed how to choose the variance of the weights at initialization using thisanalysis for various commonly used activation functions. Our analysis can beeasily applied to other activation functions.
As in earlier work, our analysis concerned a limit in which the input grows
along with the hidden layers. This simplifies the analysis, but it appears not tobe difficult to remove this assumption (see (Matthews et al., 2018)).
After publication of some of this work in preliminary form (Long and Sedghi,2019), elements of its analysis were used in (Novak et al., 2019).
Analysis of the length map in the case of ReLU activations was an importantcomponent of recent analyses of the convergence of deep network training (Zou14

et al., 2018; Allen-Zhu et al., 2019). A non-asymptotic refinement of our analysis would be a step toward generalizing those results to more general activation
functions.

Acknowledgements
We thank Ben Poole, Sam Schoenholz and Jascha Sohl-Dickstein for valuable
conversations, and Jascha and anonymous reviewers for their helpful comments
on earlier versions of this paper.

References
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deeplearning via over-parameterization. In ICML, pages 242–252, 2019.
M. Chen, J. Pennington, and S. S. Schoenholz. Dynamical isometry and a mean
field theory of RNNs: Gating enables signal propagation in recurrent neuralnetworks. arXiv preprint arXiv:1806.05394, 2018.
A. Daniely, R. Frostig, and Y. Singer. Toward deeper understanding of neural networks: The power of initialization and a dual view on expressivity. In Advances
In Neural Information Processing Systems, pages 2253–2261, 2016.
W. Feller. An introduction to probability theory and its applications. John Wiley& Sons, 2008.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward
neural networks. In Proceedings of the thirteenth international conference on
artificial intelligence and statistics, pages 249–256, 2010.
Boris Hanin. Which neural net architectures give rise to exploding and vanishinggradients? In Advances in Neural Information Processing Systems, pages 580–
589, 2018.
S. Hayou, A. Doucet, and J. Rousseau. On the selection of initialization andactivation function for deep neural networks. arXiv preprint arXiv:1805.08266,
2018.
M. Hazewinkel. Cauchy distribution. In Encyclopaedia of Mathematics: Volume
6. Springer Science & Business Media, 2013.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing
human-level performance on imagenet classification. In Proceedings of the IEEE
international conference on computer vision, pages 1026–1034, 2015.
B. Klartag. A central limit theorem for convex sets. Inventiones mathematicae,
168(1):91–131, 2007.
15

Y. A. LeCun, L. Bottou, G. B. Orr, and K. Müller. Efficient backprop. In Neuralnetworks: Tricks of the trade. Springer, 1998.
J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein.
Deep neural networks as gaussian processes. ICLR, 2018.
Philip M Long and Hanie Sedghi. On the effect of the activation function
on the distribution of hidden nodes in a deep network. arXiv preprint
arXiv:1901.02104, 2019.
R. Lupton. Statistics in theory and practice. Princeton University Press, 1993.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, andZoubin Ghahramani. Gaussian process behaviour in wide deep neural networks.
arXiv preprint arXiv:1804.11271, 2018. v2, 8/16/2018.
Radford M Neal. Bayesian learning for neural networks. 1996.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Jiri Hron,
Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Bayesian
deep convolutional networks with many channels are gaussian processes. ICLR,
2019.
J. Pennington, S. Schoenholz, and S. Ganguli. Resurrecting the sigmoid in deeplearning through dynamical isometry: theory and practice. In Advances inneural information processing systems, pages 4785–4795, 2017.
J. Pennington, S. S. Schoenholz, and S. Ganguli. The emergence of spectral
universality in deep networks. arXiv preprint arXiv:1802.09979, 2018.
B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli. Exponential
expressivity in deep neural networks through transient chaos. In Advances inneural information processing systems, pages 3360–3368, 2016.
S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein. Deep information
propagation. arXiv preprint arXiv:1611.01232, 2016.
L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. S. Schoenholz, and J. Pennington. Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer
vanilla convolutional neural networks. arXiv preprint arXiv:1806.05393, 2018.
G. Yang and S. Schoenholz. Mean field residual networks: On the edge of chaos.
In Advances in neural information processing systems, pages 7103–7114, 2017.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Stochastic gradient descent optimizes over-parameterized deep ReLU networks. CoRR,
abs/1811.08888, 2018.

16

A

Proof of Lemma 1= 0 and limsupx→−∞ log x|φ(x)|= 0, we also
Choose c > 0. Since limsupx→∞ log x|φ(x)|22have= 0 and limsupx→−∞ log |φ(cx)|= 0. Thus, there is an a suchlimsupx→∞ log |φ(cx)|x2x22that, for all x 6∈ [−a, a], log |φ(cx)| ≤ x8 , which implies φ(cx)2 ≤ exp
φ is permissible, it is bounded on [−a, a]. Thus, we haveZφ(cx)2 exp(−x2 /2) dx
Z −a
Z a
2
2
=
φ(cx) exp(−x /2)dx +φ(cx)2 exp(−x2 /2)dx
−∞
−a
Z ∞
+
φ(cx)2 exp(−x2 /2)dx
a
!Z
Z
−a

. Sinceaexp(−x2 /4)dx +

≤

x2
4

sup φ(cx)2x∈[−a,a]

−∞

Z ∞
+

exp(−x2 /2)dx−aexp(−x2 /4)dx

a<∞completing the proof.

B

Proof of Lemma 9

The proof is by induction. The base case holds since we have assumed that r̃0 > 0.
To prove the inductive step, we need the following lemma.
Lemma 12 If φ is not zero a.e., then, for all c > 0, Ez∈Gauss(0,1) (φ(cz)2 ) > 0.
Proof: If µ is the Lebesgue measure, sinceµ({x ∈ R : φ2 (cx) > 0}) = lim µ({x : φ2 (cx) > 1/n} ∩ [−n, n]) > 0,n→∞there exists n such that µ({x : φ2 (cx) > 1/n} ∩ [−n, n]) > 0. For such an n, we
have
Ez∈Gauss(0,1) (φ(cz)2 ) ≥1 −n2 /2eµ({x : φ2 (cx) > 1/n} ∩ [−n, n]) > 0.
n

Returning to the proof of Lemma 9, by the inductive hypothesis, r̃`−1 > 0,
which, since σw > 0, implies q̃` > 0. Applying Lemma 12 yields r̃` > 0.

17

C

Proof of Lemma 11= 0 there
is an b such that, for all x ≥ b, log |φ(x)| ≤ x8s ,Since limsupx→∞ log x|φ(x)|2√2which implies φ(x)2 ≤ exp x4s . Now, choose q ∈ [r, s]. For a = b/ r, we then
have
Z ∞
√
φ( qx)2 exp(−x2 /2) dx
a
 2
Z ∞
1
z
2
dz
=√
φ(z) exp −
√
q a q
2q
 2
 2
Z ∞
1
z
z
≤√
exp −
dz
exp
√
q a q
4s
2q
 2
Z ∞
1
z
≤√
dz
exp
−
q a√q
4q
 2
Z ∞
1
z
≤√
exp −
dz.
q b
4q
2

 2
R∞
By increasing b if necessary, we can ensure √1q b exp − z4q dz ≤ β which thengives
R∞ √ 2φ( qx) exp(−x2 /2) dx ≤ β. A symmetric argument yieldsaZ a√φ( qx)2 exp(−x2 /2) dx ≤ β,−∞completing the proof.

18