The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization

Abstract:
Theoretical results show that neural networks can be approximated by Gaussian processes in the infinite-width limit. However, for fully connected networks, it has been previously shown that for any fixed network width, n, the Gaussian approximation gets worse as the network depth, d, increases. Given that modern networks are deep, this raises the question of how well modern architectures, like ResNets, are captured by the infinite-width limit. To provide a better approximation, we study ReLU ResNets in the infinite-depth-and-width limit, where both depth and width tend to infinity as their ratio, d/n, remains constant. In contrast to the Gaussian infinite-width limit, we show theoretically that the network exhibits log-Gaussian behaviour at initialization in the infinite-depth-and-width limit, with parameters depending on the ratio d/n. Using Monte Carlo simulations, we demonstrate that even basic properties of standard ResNet architectures are poorly captured by the Gaussian limit, but remarkably well captured by our log-Gaussian limit. Moreover, our analysis reveals that ReLU ResNets at initialization are hypoactivated: fewer than half of the ReLUs are activated. Additionally, we calculate the interlayer correlations, which have the effect of exponentially increasing the variance of the network output. Based on our analysis, we introduce Balanced ResNets, a simple architecture modification, which eliminates hypoactivation and interlayer correlations and is more amenable to theoretical analysis.

heading: 1 Introduction
text: The characterization of infinite-width dynamics of gradient descent (GD) in terms of the so-called Neural Tangent Kernel (NTK) [1–10] represented a major breakthrough in our understanding of deep learning in the large-width regime. Before the identification of infinite-width limits, the theoretical study of deep learning had long been hindered by the apparent analytical intractability of gradient descent and variants acting on the nonconvex objectives used to train neural networks. Despite this progress, evidence suggests that deep neural networks can outperform their infinite-width limits in practice [11], particularly when the depth of the network is large. These observations motivate the study of other approximations that may close the gap.
Several alternative limits have been proposed. Around the time of the discovery of the NTK limit, mean-field limits were also characterized [12–15], and more recently have been linked with the NTK limit [16]. Yang and Hu [17] describe a family of infinite-width limits indexed by the scaling limits of initial weight variance, weight rescaling, and learning rates. This family includes both the NTK and mean field limits. One motivation for studying these alternative limits is that they yield a notion of feature learning, which provably does not occur in the NTK limit [17].
∗Equal contribution authors. Correspondence: mufan.li@mail.utoronto.ca; nicam@uoguelph.ca; daniel.roy@utoronto.ca.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Despite the variety of these limits, one common feature is that the depth of the network (i.e., the number of its layers) is treated as a constant as the width of the network is allowed to grow. Indeed, for fixed width, n, the Gaussian approximations at initialization [18–23] worsen as the depth, d, increases. While real-world networks are fairly wide, their relative depth is not trivial. Hanin and Nica [24] were the first to compute an infinite-depth-and-width limit for fully connected networks. While the training dynamics of this limit are still not completely understood, we now know that, in the infinite-depth-and-width limit, the neural tangent kernel is random, and the derivative of the kernel is nonzero at initialization. This means training does not correspond to that of a linear model, like it does in the NTK limit [25, 26].
In addition to these theoretical corrections to the Gaussian limit, practitioners have begun to notice that even basic properties of standard neural networks do not line up with those predicted by the infinite-width limit. Perhaps the most basic is that the gradients of the network early in training are not Gaussian, but instead are approximately log-Gaussian [27]. In fact, one should see logGaussian behaviour agrees with the theoretical predictions of Hanin and Nica [24], although there has yet to be a careful empirical comparison made between the precise predictions coming from infinite-depth-and-width models and real world networks.
In practice, however, fully connected networks are not often used without architectural modifications. In particular, residual connections, ushered into widespread use after the description of the ResNet architecture [28], produced very deep architectures that were practically useful with optimization techniques available at the time. Initialization schemes for ResNets have been studied in the infinitewidth limit [19, 29] or with modifications [30–32].
In this work, we consider the infinite-depth-and-width limit of fully connected architectures with residual connections (“Vanilla ResNets”) and standard initializations. Here, the analysis is complicated by the effect of skip connections, which introduce interlayer correlations that have a non-negligible effect in this limit. Surprisingly, we observe a counter-intuitive but fundamental phenomenon, whereby these skip connection cause the network to be hypoactivated, meaning that less than half of the neurons are activated on initialization. This fact undermines key assumptions that underlie other infinite-depth-and-width limit studies of architectures without residual connections or with nonstandard modifications, such as post-activation residual connections.
Hypoactivation is a roadblock that all theoretical research into standard ResNet architectures must contend with—it is an unavoidable property of the standard architecture and the root of many technical difficulties. In order to sidestep this roadblock, we introduce a conjecture that bounds the size of hypoactivation and effect on interlayer correlation. The conjecture is inspired by empirical evidence from Monte Carlo simulations, as well as several simplified analyses that ignore certain technical difficulties. The conjecture introduces what we believe to be the minimal assumption necessary to allow rigorous theoretical work to proceed. It also defines an important open problem in the study of limits of residual architectures.
To demonstrate the utility of the conjecture, we show that it leads to precise predictions. In particular, we prove a limit theorem characterizing the exact marginal distribution of the output at initialization in the infinite-depth-and-width limit, up to order O(dn−2). Our limit result shows that ResNets have log-Gaussian behaviour on initialization, and like fully connected networks [24], the behaviour is determined by the depth-to-width aspect ratio. This corroborates recent empirical observations about deep ResNets [27]. Since real world networks are finite, the question of how well this approximates finite behaviour is of paramount importance. Based on Monte Carlo simulations, we find excellent agreement between our predictions and finite networks (see Figure 1). Moreover, for very deep networks (e.g. d/n = 1) the infinite-depth-and-width prediction is extremely different than the infinite-width prediction. More surprisingly, however, is that even at comparatively small depth-towidth ratio (e.g. d/n = 0.1) the two limits are already significantly different. Furthermore, we also observe that the effects due to hypoactivation and interlayer correlation are non-negligible; these effects are precisely the difference between Vanilla ResNets and so-called “Balanced Resnets” in Figure 1. Perhaps most importantly, we observe that real network outputs exhibit exponentially larger variance than predicted by infinite-width limits. This type of variance at initialization is known to cause exploding and vanishing gradients and other types of training failures [33, 34]. Our result also implies that the output neurons of the network are not independent as predicted by the Gaussian infinite-width limit. See Figure 3.
In order to maintain the same skip connections from layer to layer, but render the activation patterns completely independent from layer to layer on initialization, we introduce the Balanced ResNet architecture, where the sign of each neuron’s activation function is randomized. We demonstrate that this exponentially decreases the variance of the network output on initialization. Moreover, this independence between neurons also makes the model more amenable to theoretical analysis and opens the door to future understanding of network behaviour. Although it is beyond the scope of this paper, a small preliminary empirical study (Appendix C) suggests that standard training regimes are not negatively affected by replacing standard ResNet architectures with Balanced ones.
We summarize our main contributions as follows:
• We identify and characterize a fundamental property of ResNets that we call hypoactivation: less than half of the ReLU neurons are activated. Based on empirical evidence, we formulate a precise minimal conjecture bounding the effect of hypoactivation that permits us to make precise, rigorous estimates for other properties of ResNets.
• We prove a limit theorem which shows that the output of ResNets on initialization exhibits log-Gaussian behaviour with parameterization depending on the depth-to-width ratio d/n.
• We provide empirical evidence from Monte Carlo simulations showing our theory provides more accurate predictions for simple properties of finite networks compared to the predictions made by the Gaussian infinite-width limit.
• We introduce the Balanced ResNet architecture, which corrects the hypoactivation and variance due to layerwise correlation from Vanilla ResNets. We also prove that the output for this architecture is log-Gaussian on initialization with exponentially lower variance. This simple modification can be applied to any neural network that uses ReLU activations.

heading: 2 Main Results
text: In terms of the notation in Table 1, a Vanilla ResNet with fully connected first/last layers and d hidden layers of width n is defined by
z0 := 1√ nin W 0x, z` := αz`−1 +λ
√ 2
n W `ϕ+
( z`−1 ) for 1 ≤ ` ≤ d, zout := 1√
n W outzd . (1)
Notation Description Notation Description nin ∈ N Input dimension nout ∈ N Output dimension
n ∈ N Hidden layer width d ∈ N Number of hidden layers (depth) ϕ+(·) ReLU function
ϕ+(x) = max{x, 0} ϕ−(·) “Domain Flipped” ReLU
ϕ−(x) = max{−x, 0} α ∈ R Skip connection coefficient λ ∈ R+ Feed-forward coefficient x ∈ Rnin Input W 0 ∈ Rnin×n Weight matrix at layer 0 zout ∈ Rnout Network output W out ∈ Rn×nout Weight matrix at final layer. z` ∈ Rn Neurons (pre-activation)
for layer 1 ≤ ` ≤ d W ` ∈ Rn×n Weight matrix at layer 1 ≤ ` ≤ d all weights initialized i.i.d. ∼ N (0, 1)
Table 1: Notation
Note that factors of √
2n−1 in the hidden layer are equivalent to intializing according to the so-called He initialization [35]. Other intializations correspond to changing the coefficient λ. This setup is similar to that of “Stable ResNets” [29], where the infinite-width limit is studied.
In the infinite-depth-and-width limit, the intuition that half of the ReLU units are active (i.e. nonzero) because of symmetry is surprisingly not correct. We find that the following quantities play an important role. We define the average hypoactivation (of layer `) and the total hypoactivation of the network by
h` := E [∥∥ϕ+ (ẑ`)∥∥2]− 1
2 , htotal :=
∑d `=1 h` , (2)
where ẑ` := z`/ ∥∥z`∥∥ and E means expectation over the choice of random network weights on initialization. The average hypoactivation is a measure of how many ReLU neurons are activated in layer `; h` = 0 indicates roughly half of the neurons are active. Counter-intuitively, we observe that in a vanilla ResNet, h` is negative and |h`| = O(1/n), indicating slightly less than half the neurons are active.2 After compounding over d layers, the total hypoactivation is of order htotal = O(d/n) = O(1) in the infinite-depth-and-width limit. As we will see, this effect has a non-trivial contribution.
At the same time, we also find the covariance between the activations of various layers does not vanish in the infinite-depth-and-width limit. This motivates the definition of the total interlayer covariance correction
Itotal := ∑ 1≤` 6=`′≤dCov ( 2 ∥∥ϕ+(ẑ`)∥∥2 , 2‖ϕ+(ẑ`′)‖2) . (3)
As with the hypoactivation, skip connections cause this term to be non-trivial. We formulate Conjecture 5, which contains a precise encapsulation of the behaviour of ẑ` we observe.
Conjecture 5 (Informal). In expectation, the layers ẑ` can be approximated by uniform random variables from the sphere up to a relative error of O(1/n).
The conjecture is well supported by Monte-Carlo simulations (see Figure 4). We provide a more detailed discussion and a precise statement of Conjecture 5 in Section 4. Assuming the conjecture holds, we prove a limit theorem about the distribution of zout. Informally, this says that zout is approximately a log-Gaussian scalar times an independent Gaussian vector
zout ≈ ‖x‖√ nin
( α2 + λ2 ) d 2 exp
( 1 2 N ( −β 2 + 2chtotal, β + c 2Itotal )) ~Z, (4)
where ~Z has iid N (0, 1) entries, and β and c are defined by
β := 2 n + d n · 5λ
4 + 4α2λ2
(α2 + λ2)2 , c :=
λ2
α2 + λ2 . (5)
The precise statement, including asymptotic error bounds, is as follows. Theorem 1. For any choice of hyperparameters nin, nout, n, d, α, λ, and every input x, the output zout at initialization has a marginal distribution which can be written in the form
zout d = ‖x‖√ nin
( α2 + λ2 ) d 2 exp
( 1
2 G
) ~Z, (6)
2For a quantity f = f(n, d) whose dependence on width and depth may be implicit, we use the notation f = O(da/nb) to mean that, for all choice of constants α, λ, r−, r+ > 0, there exists a constant C > 0 such that |f(n, d)| ≤ Cda/nb for all d, n where r− < d/n < r+. This notation will allow us to state precise limit theorems when d, n→∞ with the ratio d/n converging to a constant.
α = λ = 1/
√
2, d/n = 1 are fixed and width n is varied on the x-axis. The error bars indicate 95% confidence interval (CI) of the Monte-Carlo simulation, with truncation at 1e−3 for plotting on log-scale. The balanced ResNet predictions fall within the Monte-Carlo CI and so are not plotted on log scale. Since d/n is fixed, Figure 2b indicates the error in the asymptotic prediction of variance (7) is O(n−1) = O(dn−2) as claimed. On the other hand, the error in mean (7) is O(n−2) = O(dn−3); this is one order smaller than the statement proven in Theorem 1. A possible explanation is that the sub-leading error term in our approximation for G is mean zero.
where ~Z ∈ Rnout is a Gaussian random vector with iid N (0, 1) entries, G = G(n, d, α, λ) is a random variable which is independent of ~Z and whose distribution does not depend on nin, nout or x.
Consider the limit where both the network depth d→∞ and hidden layer width n→∞ in such a way that the ratio d/n converges to a non-zero constant. In this limit, assuming that Conjecture 5 holds, then the random variable G ∈ R has the following asymptotic behaviour:
E [G] = −β 2 + 2chtotal +O
( d
n2
) , Var [G] = β + c2Itotal +O ( d
n2
) , (7)
where β and c are as in (5), and moreover G converges in distribution to a Gaussian random variable with mean and variance given by (7) in this limit.
The main ideas of the proof of Theorem 1 is given in Section 5 and the detailed proof is given in Appendix B. We also provide more explicit formulas for htotal, Itotal below. Proposition 2. Assume Conjecture 5 is true. Then in the same infinite-depth-and-width limit as Theorem 1, the total hypoactivation htotal and total interlayer covariance Itotal obey
htotal = Cα,λ d
n +O
( d
n2
) , Itotal = ∑ 1≤` 6=`′≤d J̄2(θ|`′−`|)− J̄2(π − θ|`′−`|) n +O ( d n2 ) . (8)
Here Cα,λ is a constant depending on α, λ and J̄2(θ) := J2 (θ) /π = 3 sin(θ) cos(θ)/π + (1− θ/π) ( 1 + 2 cos2 θ ) ,
where J2(θ) first appeared in [36], and θk is such that cos(θk) = αk/(α2 + λ2)k/2 .
Remark 3. The Gaussian infinite-width limit predicts that the marginals of zout should have the form of (6) withG being identically zero. As zout depends exponentially onG, the infinite-depth-and-width limit predicts the variance is exponentially larger than the infinite-width limit. See Section 3.1 for a detailed discussion and Figure 3 for verification against finite networks.
Using Monte Carlo simulations, we estimate the constant Cα,λ. The result of Theorem 1 is then compared against finite networks in Figure 2. Full proof can be found Appendix B. The idea of using a scaling parameter α = 1/ √
2 in the skip connections has been noted in empirical papers [37] and also studied under simplified assumptions on the number of activation in each layer [38]. Our result shows that by choosing α2 + λ2 = 1, the prefactor in (6) does not grow with depth thereby enabling deeper networks to be trained.
In the case that α = 0, λ = 1, the architecture reduces to a fully connected network. In this case, log-Gaussian behaviour of the network was discovered in Hanin and Nica [24]. The fully connected case is simpler because the direction vectors ẑ` are always uniformly distributed on the unit sphere and are independent from layer to layer. This means h` = 0 and θn = 0 which greatly simplifies the result of Theorem 1.
Furthermore, the proof easily extends to the case where the coefficients α`, λ` vary from layer to layer; see Appendix A.1 for the general statement. Hayou et al. [29] and Hanin and Rolnick [33] have studied ResNets where α` = 1 in every layer, but λ` is allowed to vary. The prefactor of our result in this case becomes ∏d `=1 (1 2 + λ2`) ≈ exp (∑d `=1 λ 2 ` ) , which is akin to the behaviour found in those papers. Additionally, our result precisely quantifies the log-Gaussian behaviour and its dependence on the sequence λi through the parameters β and c in the infinite-depth-and-width limit.

heading: 2.1 Log-Gaussian Behaviour of Balanced ResNets
text: Given a collection of iid uniformly random signs, s`i ∈ {+,−}, 1 ≤ ` ≤ d, 1 ≤ i ≤ n, a balanced ResNet is defined much like a Vanilla ResNet, except that a random sign is applied preactivation:
z` := αz`−1 + λ
√ 2
n W `ϕs`
( z`−1 ) for 1 ≤ ` ≤ d,
where at each layer the vector function ϕs` : Rn → Rn applies either ϕ+ or ϕ− to the entries according to the random signs s`. More precisely, the i-th component is
ϕs`(z)i := { ϕ+(zi) = max(zi, 0), if s`i = + , ϕ−(zi) = max(−zi, 0), if s`i = − .
An equivalent definition is the entrywise multiplication ϕs`(z) = ϕ+(s` z). Note that the random signs s`i are not trainable parameters; they are frozen on initialization. This same symmetrization was first exploited by Allen-Zhu et al. [39] and Bai and Lee [40] to study a quadratic approximation of the network. We now present a corresponding limiting theorem for Balanced ResNets. Theorem 4. For a balanced ResNet, the same result as Theorem 1 given in (6) still holds, but with the mean and variance of G given simply by
E [G] = −β 2 +O
( d
n2
) , Var [G] = β +O ( d
n2
) . (9)
Balanced ResNets are constructed so that the activation of each neuron is independent of all others due to the random signs s`. This eliminates the hypoactivation and variance terms which complicated the analysis of the Vanilla ResNet and necessitated Conjecture 5. Instead, for Balanced ResNets it is straightforward to compute that for any fixed z, w ∈ Rn we have
E [ ‖ϕs`(z)‖2 ] = ‖z‖2
2 ,Var
[ ‖ϕs`(z)‖2 ] = n∑ i=1 z4i 4 ,Cov [ ‖ϕs`(z)‖2 , ‖ϕs`′ (w)‖ 2 ] = 0. (10)
Even though the layers ẑ`, ẑ` ′
are correlated, because the activation functions ϕs` and ϕs`′ are set to be independent on initialization, the correlation between layers does not induce a correlation on which neurons are activated from layer to layer. This explains why there is no hypoactivation and interlayer correlation correction in Theorem 4 as there is in Theorem 1.

heading: 3 Consequences of Theorems 1 & 4 and Comparison to Infinite-Width Limit

heading: 3.1 Vanishing and Exploding Norms
text: By the basic fact E[exp(N (µ, σ2))] = exp(µ + 12σ2) it follows from Theorems 1 & 4 that, when the inputs x has ‖x‖ = √nin, the mean size scale of any neuron zouti is approximately
E [ (zouti ) 2 ] ≈ { (α2 + λ2)d exp ( 2chtotal + 1 2c 2Itotal ) , for Vanilla ResNets,
(α2 + λ2)d , for Balanced ResNets.
(Note that the terms with β cancel out!) When α2 + λ2 = 1, this is constant for Balanced ResNets. In contrast, Vanilla ResNets have a complicated dependence on the network depth d and width n due to the hypoactivation and correlations terms. This means the behaviour is exp(Cd/n), which is somewhat surprising. A more serious issue is the variance which Theorems 1 & 4 predict to be Var [ (zouti ) 2 ] ≈ { (α2 + λ2)2d ( 3 exp ( β + c2Itotal ) − 1 ) exp ( 4chtotal + c 2Itotal ) , for Vanilla, (α2 + λ2)2d (3 exp (β)− 1) , for Balanced. Since β ≈ Cd/n, the term exp(β) represents exponentially larger variance for deep nets compared to shallow ones of the same width. In contrast, the variance predicted by the infinite width limit does not grow with depth for this model when α2 + λ2 = 1. This effect means the relative sizes of different network outputs can be widely disparate. Unlike problems with the mean, this issue is harder to resolve. For example, normalization methods that divide all neurons by a constant does nothing to address the large relative disparity between two points. Techniques like batch normalization will be skewed by large outliers. This kind of variance is known to obstruct training [33]. The input-output derivative ∂xiz
out has the same type of behaviour as zout itself; a simple proof is given in Appendix B. It is expected that the gradient with respect to the weights ∂W `ijz
out will also have the same qualitative behaviour [27] although more investigation is needed to understand this theoretically. Exponentially large variance for gradients is a manifestation of the vanishing-and-exploding gradient problem [34].
Balanced ResNets suffer from this variance problem less because the interlayer correlation term is zero. Since this variance reduction happens at the exponential scale in, the difference can be significant; for networks with d/n = 1, the contribution is a factor of ≈ e5.5 ≈ 250 for Vanilla ResNets vs. ≈ e2.5 ≈ 10 for Balanced ResNets. See Figure 3 for a comparison of these theoretically predicted properties vs experiments with finite networks.

heading: 3.2 Correlated Output Neurons
text: Since the same random variable G multiplies the entire vector zout, the individual neurons in the output layer are not independent. For example, Theorem 1 and 4 predict that the squared entries have strictly positive correlation given by Corr ( (zouti ) 2 , ( zoutj )2)
= (exp(σ2) − 1)/(3 exp(σ2) − 1) for any two neurons i 6= j where σ2 = Var(G). This tends to 1/3 as d/n grows. The effect of correlated output neurons persists for Balanced ResNet but is reduced again due to the lower variance. This is very different from the infinite-width limit, which predicts that individual neurons should be independent Gaussians. This prediction of the theorem matches finite networks closely; see Figure 3.

heading: 4 Conjecture 5: Hypoactivation and Layerwise Correlations
text: Vanilla ResNets have a subtle asymmetry in the architecture due to the skip connections. Unlike fully connected networks, the distribution of ẑ` := z`/ ∥∥z`∥∥ for ` ≥ 1 is not exactly uniformly distributed
on the unit sphere. Informally speaking, ẑ` can be thought of as a random walk whose variance at each step is proportional to ∥∥ϕ+(ẑ`)∥∥. More randomness is injected when ∥∥ϕ+(ẑ`)∥∥ is large and less when it is small. The net effect is that the walk moves slower when
∥∥ϕ+(ẑ`)∥∥ is small, thereby spending more time in those locations. Consequently, ‖ϕ+(ẑ`)‖ is biased toward smaller values. The size of this effect is limited by entropy; most of the unit sphere Sn−1 has ‖ϕ+(u)‖2 ≈ 1/2 in the sense that for any > 0, the measure of the set { u : |‖ϕ+(u)‖2 − 12 | > 1/n 1 2− }
vanishes as n→∞. To quantify the effect of the bias, we can think of the evolution of ẑ`, ` = 1, 2, . . . as a random walk that takes different step sizes at a different points on the sphere. It is reasonable to expect that the behaviour of this processes will be similar to that of a time changed Brownian motion, which is slowed down at the points where ẑ` takes smaller steps. (Proving this comparison precisely is technically difficult since the parameter n simultaneously plays both the role dimension and the step size of the walk.) Based on this heurstic comparison to time changed Brownian motion and on extensive Monte Carlo simulations we conjecture that the expected size of the hypoactivation effect is only O(1/n) in expectation; Conjecture 5 contains a precise statement.
Even if each layer ẑ` is marginally close to the uniform distribution on the unit sphere, the directions ẑ` and ẑ`+1 are not independent because of the skip connections in the network. As above, the exact behaviour is complicated due to fluctuations in the exact number of neurons which are activated in each layer. However, using the idea that
∥∥ϕ+(ẑ`)∥∥2 = 12 (1 + o(1)), we construct the following approximation. From (16), we have the approximation z`+1/
∥∥z`∥∥ = αẑ` + λg`+1/√n (1 + o(1)) We observe that the norm of RHS is concentrated around √ α2 + λ2 as n→∞, so normalizing this to get ẑ`+1 we have
ẑ`+1 = ( α√
α2 + λ2 ẑ` + λ√ α2 + λ2 g`+1√ n
) (1 + o(1)) .
Iterating this gives the same relationship for ẑ`+k where the first coefficient becomes αk/ √ α2 + λ2 k . As before, based on Monte Carlo simulations, we conjecture that the size of the error is O(1/n) in expectation. We formalize this as a precise statement in Conjecture 5 below. Conjecture 5. The distribution of the unit vector ẑ` = z`/
∥∥z`∥∥ is approximately uniformly distributed from the unit sphere u ∈ Sn−1 in the precise sense that the following asymptotics hold
E [∥∥ϕ+(ẑ`)∥∥2] = E [‖ϕ+(u)‖2](1 +O( 1
n
)) , (11)
Var [∥∥ϕ+(ẑ`)∥∥2] = Var [‖ϕ+(u)‖2](1 +O( 1
n
)) , (12)
where the constants in the big O(·) notation are uniform in `. Moreover, for two layers `, `′, which are k ≥ 1 layers apart |`′ − `| = k, the joint distribution of ẑ`, ẑ`′ is approximately equal to the joint distribution of u, cos(θk)u + sin(θk)g/ √ n where g is a Gaussian vector with iid N (0, 1) entries which is independent of u and θ is such that cos(θk) = αk/(α2 + λ2)k/2 in the sense that the following asymptotics hold
Cov [∥∥ϕ+(ẑ`)∥∥2 ,∥∥∥ϕ+(ẑ`′)∥∥∥2] = Cov [‖ϕ+(u)‖2 ,∥∥∥∥ϕ+(cos(θk)u+ sin(θk)√n g) ∥∥∥∥2 ]( 1 +O ( 1 n )) ,
(13) where the constant in the big O(·) notation is uniform in `, `′.
See Figure 4 for Monte Carlo simulations empirically verifying the conjecture for a fixed depth d, and see Figure 5 for verifying the uniformity in layers `. In particular, we observe that in Figure 5a, we can see the effect of hypoactivation converges rapidly to an equilibrium as the layer ` increases. In fact, we can further verify in Figure 5b that hypoactivation appears to be autoregressive, which implies the convergence is exponentially fast. This motivated the uniformity in layers in Conjecture 5.

heading: 5 Proof Ideas for Theorems 1 & 4
text: A key element of the proof is the following property of Gaussian random matrices. If W which has iid N (0, 1) entries, then for any vector x, we have
Wx d = ‖x‖ g, (14)
where g is a vector whose entries are iid N (0, 1) random variables. Because of the fully connected first and last layer of the network, (14) implies that
z0 d = ‖x‖√ nin g, zout d = ∥∥zd∥∥√ n g′. (15)
Hence G := ln ( ( ∥∥zd∥∥2 /n) · (‖x‖2 /nin)−1 · (α2 + λ2)−d) only depends on n, d, α, λ. (Equiva-
lently, G has the distribution of ln( ∥∥zd∥∥2 /n · (α2 + λ2)−d) when z0 = g.) With this definition, (15) also shows zout is proportional to exp(G/2), establishing the first part of Theorem 1.
From this construction, the essence of the proof is to understand the distribution of ∥∥zd∥∥ when z0 = g.
To understand ∥∥zd∥∥, we look at the ratios ∥∥z`+1∥∥ /∥∥z`∥∥ layer by layer. By using the homogeneity
property of ReLU ϕ+(|c|x) = |c|ϕ+(x), we can divide z`+1 from (1) by ∥∥z`∥∥ to obtain
z`+1 ‖z`‖ = αẑ ` + λ
√ 2
n W `+1ϕ+(ẑ `) d = αẑ` + λ
√ 2
n ∥∥ϕ+(ẑ`)∥∥ g`+1, (16) where g` are iid Gaussian vectors with iid N (0, 1) entries by application of (14). Hence∥∥z`+1∥∥
‖z`‖ d = ∥∥∥∥∥αẑ` + λ √ 2 n ∥∥ϕ+(ẑ`)∥∥ g`+1 ∥∥∥∥∥ d= ∥∥∥∥∥α~e1 + λ √ 2 n ∥∥ϕ+(ẑ`)∥∥ g`+1 ∥∥∥∥∥ .
The last equality follows by applying an orthogonal transformation O such that Oẑ` = ~e1 = (1, 0, 0 . . . , 0)T inside the norm, and observing that Gaussian random vectors are invariant under orthogonal transformations Og` d= g`. Hence we have the telescoping product for ∥∥zd∥∥: ∥∥zd∥∥ = ∥∥z0∥∥ d−1∏
`=0
∥∥z`+1∥∥2 ‖z`‖ d = ∥∥z0∥∥ d−1∏
`=0
∥∥∥∥∥α~e1 + λ √ 2 n ∥∥ϕ+(ẑ`)∥∥ g`+1 ∥∥∥∥∥ . (17)
This shows that ∥∥zd∥∥ is a product of d random variables which are dependent on each other only
through the terms ∥∥ϕ+(ẑ`)∥∥. (Note that ∥∥z0∥∥ is independent of ẑ0 since z0 is Gaussian.) Since∥∥ϕ+(ẑ`)∥∥2 ≈ 1/2 with typical fluctuations on the scale 1/√n, therefore the dependence between terms of (17) is small.
Taking the ln of (17) exhibits ln( ∥∥zd∥∥2 /n) as a sum of these weakly correlated random variables. Here we note that various tail estimates for the same or related quantities have been developed [3, 41], however these estimates are not precise enough to pinpoint the exact limiting distribution. In contrast, we are able to derive the exact limiting distribution via a Central Limit Theorem (CLT) for weakly correlated sums [42]. The proof of Theorem 1 is completed by computing the mean, variance and covariance of terms using Conjecture 5. For Theorem 4, the final calculation is simplified by (10) which shows the terms are uncorrelated. The detailed proof is given in Appendix B.

heading: Acknowledgement
text: We would like to thank Blair Bilodeau, Gintare Karolina Dziugaite, Mahdi Haghifam, Yani A. Ioannou, James Lucas, Jeffrey Negrea, Mengye Ren, and Ekansh Sharma for helpful discussions and draft feedback. We would also like to thank the anonymous NeurIPS reviewers for insightful feedback. In particular, one identified numerous relations to existing work, and another helped us identify the uniformity requirement in Conjecture 5. ML is supported by Ontario Graduate Scholarship and the Vector Institute. MN is supported by an NSERC Discovery Grant. DMR is supported in part by an NSERC Discovery Grant, Ontario Early Researcher Award, and a stipend provided by the Charles Simonyi Endowment.

Review summaries:
The authors study signal propagation in randomly-initialized feedforward residual neural networks with gaussian weights, with architectures consisting of one fully-connected layer per each residual block in the network. Conditional on a certain conjecture about distributions of normalized preactivations in the residual network, they prove theorems that precisely articulate the distribution of the norm of the network output in a limit where the network depth $d$ and width $n$ simultaneously tend to infinity and are directly related, and as a byproduct articulate precise dependences of the mean and variance of this output, as well as the inter-layer correlations of the network, as functions of the depth and width. The authors claim interesting consequences of this conditional analysis relative to predictions one would get by considering a limit where just the width $n$ is taken to be infinite (and then possibly the depth) -- for example, they show that standard ResNets have less than half of their neurons in each layer activated, which can lead to certain unstable output behaviors, and that networks in this limit (in particular when $d > n$ as the limit is being taken) may have exponentially-large output variances, which may be challenging to mitigate (Section 3). The authors justify the conjecture underlying their theoretical results with Monte Carlo simulations and a heuristic argument. 

This paper aims to deal with the problem that the gaussian process approximation of Neural network at initialization will become worse while the network become deeper due to the error accumulation. This paper proposed a central limit theorem in depth to have a log-gaussian limit of the infinite width and depth limit.  The subject of paper is pretty cool, but the assumption in the paper and conclusion doesn't convinced me the new characterization is right and essential.  

The papers studies the infinite width/depth limit of ReLU ResNets, to contrast it to the more usual infinite width/fixed depth limit. They show (under the assumption that a certain conjecture is true) that in the infinite width/depth limit, the limiting distribution of the outputs of the network at initialization (for i.i.d. Gaussian entries) is not Gaussian but log-Gaussian. While this was known for fully-connected networks, proving it for ResNets is more difficult due to dependence between the layers. This is related to the fact that less than half the neurons are active and that there exists correlations between layers of the network (which does not happen for fully-connected networks). These two properties of ResNets however lead to very large variances at initialization which can make training difficult, the authors therefore propose a balanced ResNets (where the activation at each neuron is chosen randomly between the ReLU and the "negative" ReLU) which ensures that half the neurons are active and removes the correlations, however the variance at initialization (though smaller than that of ResNets) can still be large (much larger than what is predicted by the infinite width limit). Finally the results are checked empirically, showing that even for small depth/width ratios (say 0.1) the combined with/depth limit describes the statistics of DNNs more accurately than the fixed depth, infinite with limit.

The paper calculates the forward pass at initialization of resnets in the joint proportional limit of infinite width and depth. In particular, the paper shows that this limit exhibits log-Gaussian behaviors, extending what was previously known for feedforward networks. From here, the paper makes several interesting observations regarding interlayer correlation, enlarged output variance and how to alleviate them. The proof crucially exploits homogeneity of the ReLU activation function, together with an assumed conjecture on the pre-activation vector’s direction being almost uniformly distributed.