The future is log-Gaussian: ResNets and their infinite-depth-and-width limit at initialization
Authors: Mufan Li
Mihai Nica
Daniel M. Roy
Venue: NeurIPS 2021 Poster (2021)
Keywords: ResNet
infinite-depth-and-width
log-Gaussian
hypoactivation
balanced ResNet

Abstract: Theoretical results show that neural networks can be approximated by Gaussian processes in the infinite-width limit. However, for fully connected networks, it has been previously shown that for any fixed network width, $n$, the Gaussian approximation gets worse as the network depth, $d$, increases. Given that modern networks are deep, this raises the question of how well modern architectures, like ResNets, are captured by the infinite-width limit. To provide a better approximation, we study ReLU ResNets in the infinite-depth-and-width limit, where \emph{both} depth and width tend to infinity as their ratio, $d/n$, remains constant. In contrast to the Gaussian infinite-width limit, we show theoretically that the network exhibits log-Gaussian behaviour at initialization in the infinite-depth-and-width limit, with parameters depending on the ratio $d/n$. Using Monte Carlo simulations, we demonstrate that even basic properties of standard ResNet architectures are poorly captured by the Gaussian limit, but remarkably well captured by our log-Gaussian limit. Moreover, our analysis reveals that ReLU ResNets at initialization are hypoactivated: fewer than half of the ReLUs are activated. Additionally, we calculate the interlayer correlations, which have the effect of exponentially increasing the variance of the network output. Based on our analysis, we introduce \emph{Balanced ResNets}, a simple architecture modification, which eliminates hypoactivation and interlayer correlations and is more amenable to theoretical analysis.

The Future is Log-Gaussian: ResNets and Their Infnite-Depth-and-Wiihl Limit at Initialization
Mufan (Bill) Li University of Toronto, Vector Institute
Mihai Nica" University of Guelph, Vector Institute
Daniel M. Roy University of Toronto, Vector Institute
Abstract
Theoretical results show that neural networks can be approximated by Gaussian processes in the infinit-wwidth limit. However, for fully connected networks, it has been previously shown that for any fixed network width, n, the Gaussian approxi- mation gets worse the network depth, d, increases. Given that modern networks are deep, this raises the question of how well modern architectures, like ResNets, are captured by the infinite-width limit. To provide better approximation, limit, where both depth and width
study
ReLU ResNets in the tend to infinity as their ratio, d/n, remains constant. In contrast to the Gaussian infinite-width limit, we show theoretically that the network exhibits log GagSsas behaviour at initialization in the infinite-eptth dentite-dephh-ad--e-d aenth-and-widttee limit, with parameters depending on the ratio d/n. Using Monte Carlo simulations, we demonstrate that even basic properties of standard ResNet architectures are poorly captured by the Gaussian limit, but remarkably well captured by our log-Gaussian limit. Moreover, our analysis reveals that ReLU ResNets initialization are hypoactivated: fewer than half of the ReLUU are activated. Additionally, we calculate the interlayer correlations, which have the effect of exponentially increasing the variance analysis, we introduce Balanced ResNets,
the network output. Based simple architecture modification, which climinates hypoactivation and interlayer correlations and is more amenable to theoretical analysis.
Introduction
The characterization of infinite- width dynamics of gradient descent (GD) in terms of the so-called Neural Tangent Kernel (NTK) [T-TO] represented major breakthrough in our understanding of deep learning in the large-width regime. Before the identification of infinite-width limits, the theoretical study of deep learning had long been hindered by the apparent analytical intractability of gradient descent and variants acting on the nonconvex objectives used train neural networks. Despite this progress, evidence suggests that deep neural networks can outperform their infinite-wide limits in practice [1I). particularly when the depth of the network is large. These observations motivate the study of other approximations that may close the gap.
Several alternative limits have been proposed. Around the time the discovery of the NTK limit, mean-field field limits were also characterized [12 15), and more recently have been linked with the NTK limit [16]. Yang and Hu [17] describe family of indtnite-width limits indexed by the scaling limits of initial weight variance, weight rescaling, and learning rates. This family includes both the NTK and mean field limits. One motivation for studying these alternative limits is that they yield a notion of feature learning, which provably does not occur in the NTK limit [17
"qqual contribution authors. Correspondence: muf an. li@mail. wtwronto. ca; ni www.m@uogueelphccmmmmoou ca; daniel. roybuooooootttooooooooo
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Depch Width Ratio: 0.01
Depth Width Ratio: 0.1
Depth Wsah Ratio:
Vamilla Reshet
Bulanced RosNi
0-
-7.5
-5
-2.5
0
2.5 -75
-2.5
2.5
-1.5
-5
-2.5
2.5
Dassity Prediction
Infimite Depth and Width (Vinilla ResNet) Inlinite Depth and Width (Balincod ReeNet)
Infimite Width
Empirical Histogram
Figure 1: Probability density function of In || for six network configurations on initialization. All networks have n = 100, 72m = Tuu = 10, a A= 1/v2. Depth d varies by column. Top row: Vanilla ReLUU ResNets. Bottom Row: Balanced ResNets (Section 2. which randomize the non- linearities used at each neuron thereby exponentially reducing the variance. The theoretical curves shown are the result of Theorem and Theorem 4 These predictions converge to the prediction when d/n - 0. The details of the simulations and plot here is in Appendix C
width
Despite the variety of these limits, one common feature is that the depth of the network (i.e., the number of its layers) is treated as a constant as the width of the network is allowed to grow. Indeed, for fixed width, the Gaussian approximations at initialization [3] worsen the depth, d, increases. While real-world networks are fairly wide, their relative depth is not trivial. Hanin and feanit--depth-ndd-iidde limit for fully connected networks. While the training dynamics of this limit are still not completely understood, we now know that, in the infinite- siaeeedepthaaaee heepphhanddwiie and-width limit, the neural tangent kernel is random, and the derivative of the kernel is nonzero at initialization. This means training does not correspond to that a linear model,
Nica [4 were the first to compute
like it does in the NTK limit 25 26
In addition to these theoretical corrections to the Gaussian limit, practitioners have begun to notice that even basic properties of standard neural networks do not line up with those predicted by the infinite-widthh limit. Perhaps the most basic is that the gradients of the network early in training not Gaussian, but instead are approximately log- aag-Gausiian [27). In fact, one should log- Gaussian behaviour agrees with the theoretical predictions of Hanin and Nica (24), although there has yet to be a careful empirical comparison made between the precise predictions coming from infinite-den deeeeedepth--aadd aothh-an-wiidee width models and real world networks.
In practice, however, fully connected networks are not often used without architectural modifications. In particular, residual connections, ushered into widespread use after the description of the ResNet architecture (28), produced very deep architectures that were practically useful with optimization techniques available at the time. Initialization schemes for ResNets have been studied in the infinite- or with modifications 32).
width limit 19 29
In this work, we consider the infinite-depth-and-we ual connections ("Vanilla ResNets") and standard initializations. Here, the analysis is complicated by the effect of skip connections, which introduce interlayer correlations that have a non-negligible effect in this limit. Surprisingly. we observe a counter-intuitiv but fundamental phenomenon, whereby these skip connection cause the network to be hypoactivated, meaning that less than half of the neurons are activated on initialization. This fact undermines key assumptions that underlie other -nfnite--eephh-nd--ve limit studies of architectures without residual connections or with nonstandard modifications, such as post-activation residual connections.
limit of fully connected architectures with resid-

---

Hypoactivation is a roadblock that all theoretical research into standard ResNet architectures must contend with-it is an unavoidable property of the standard architecture and the root many technical difficulties. In order to sidestep this roadblock, we introduce conjecture that bounds the size of hypoactivation and effect on interlayer correlation. The conjecture is inspired by empirical evidence from Monte Carlo simulations, well as several simplified analyses that ignore certain technical difficulties. The conjecture introduces what we believe to be the minimal assumption necessary to allow rigorous theoretical work to proceed. It also defines an important open problem in the study of limits of residual architectures.
To demonstrate the utility of the conjecture, we show that it leads to precise predictions. In particular, we prove limit theorem characterizing the exact marginal distribution of the output at initialization in the anfinite-depth-ie aatth--nd-witthh limit, up to order O(dn ?). Our limit result shows that ResNets have aag-Guussian behaviour on initialization, and like fully connected networks (24), the behaviour determined by the depth-to-width aspect ratio. This corroborates recent empirical observations about deep ResNets [27]. Since real world networks are finite, the question of how well this approximates finite behaviour is of paramount importance. Based on Monte Carlo simulations, we find excellent agreement between our predictions and finite networks (see Figure DT. Moreover, for very deep networks (e.g. d/n 1) the mninite-depth-and-wiilel prediction is extremely different than the infinite-widthh prediction. More surprisingly, however, is that even at comparatively small depth-to- width ratio (e.g. d/n 0.1) the two limits are already significantly different. Furthermore, also observe that the effects due to hypoactivation and interlayer correlation are non- nog-neglig;;,e; these effects are precisely the difference between Vanilla ResNets and so-called "Balanced Resnets" in Figure [] Perhaps most importantly, we observe that real network outputs exhibit exponentially larger variance than predicted by infinitewiidth limits. This type of variance at initialization is known to cause exploding and vanishing gradients and other types of training failures
33 34
Our result also
implies that the output neurons of the network are not independent predicted by the Gaussian infinite-width limit. See Figure 3
In order to maintain the same skip connections from layer to layer, but render the activation patterns completely independent from layer to layer on initialization, we introduce the Balanced ResNet architecture, where the sign of each neuron's 's activation function is randomized. We demonstrate that this exponentially decreases the variance the network output on initialization. Moreover, this independence between neurons also makes the model more amenable to theoretical analysis and opens the door to future understanding of network behaviour. Although it is beyond the scope of this paper, small preliminary empirical study (Appendix ( suggests that standard training regimes not negatively affected by replacing standard ResNet architectures with Balanced ones.
We summarize our main contributions follows:
We identify and characterize fundamental property of ResNets that we call hypoactivation: less than half of the ReLU neurons are activated. Based on empirical evidence, we formulate a precise minimal conjecture bounding the effect of hypoactivation that permits us to make precise, rigorous estimates for other properties ResNets.
We prove limit theorem which shows that the output of ResNets on initialization exhibits log-Gaussian behaviour with parameterization depending on the depth-to- width ratio d/n.
We provide empirical evidence from Monte Carlo simulations showing our theory provides more accurate predictions for simple properties of finite networks compared to the predictions made by the Gaussian infinite-width limit.
We introduce the Balanced ResNet architecture, which corrects the hypoactivation and variance due to layerwise correlation from Vanilla ResNets. We also prove that the output for this architecture is Log-Gaussian on initialization with exponentially lower variance. This simple modification can be applied to any neural network that uses ReLU activations.
2 Main Results
In terms of the notation in Table 1, a Vanilla ResNet with fully connected first/last layers and d hidden layers of width n is defined by
Won x, 2º == azz-t + Xv W ++ (1 for sesd, 2 =
az
wou od (1)
Tin
vn
3

---

Notation
Description
Notation E N d E N (P-(-)
Description
Table 1: Notation
Tha E I E EN (+)()
Input dimension
Output dimension
Hidden layer width
Number of hidden layers (depth)
ReI.U function
"Domain Flipped" ReI.U
4p+ = max 2x, 0}
(- = max 0} Feed-forward coefficient
E R x E R
Skip connection coefficient Input
A E R$ W E Rm W E 1 W E 2uxn
Weight matrix at layer Weight matrix at final layer. Weight matrix at layer < é S d all weights initialized i.d. ~ (0,
2m E R
Network output
z' E R
Neurons aprevactivation) for layer S E
Note that factors of 22n in the hidden layer are equivalent to intializing according to the so-called He initialization [35]. Other intializations correspond to changing the coefficient A. This setup is similar to that of "Stable ResNets"
(29), where the infinite width limit is studied.
In the infinite-depth-and-wihte limit, the intuition that half of the ReLU units are active (i.e. nonzero) because of symmetry is surprisingly not correct. We find that the following quantities play an important role. We define the average hypoactivation (of layer 2) and the total hypoactivation of the network by
he = E |1++ (39] 1
hgotal Ei= he
where 26 = ==1|==9| and E means expectation over the choice of random network weights on initialization. The average hypoactivation is measure of how many ReLU neurons are activated in layer e; h4 0 indicates roughly half of the neurons active. Counter-intuitivelly a vanilla ResNet, he is negative and lhel = O(1/n). indicating slightly less than half the neurons are active f After compounding over d layers, the total hypoactivation is of order hotal O(d/n) = O(1) in the infinite-depth-ane aarhch--n--uaa and-width limit. As we will see, this effect has a non-trivial contribution.
observe that in
At the same time, we also find the covariance between the activations of various layers does not vanish in the infinite-depth deeee-depphhddddeeee aadd--ndd-idd width limit. This motivates the definition of the total interlayer covariance correction
honal == Enstor Cov ,2,
As with the hypoactivation, skip connections cause this term to be non-trivial. We formulate Conjecture which contains precise encapsulation of the behaviour of 5 we observe.
Conjecture (Informal). In expectation, the layers ze can be approximated by uniform random variables from the sphere up to a relative error of (1/nn).
The conjecture is well supported by Monte-Carlo simulations (see Figure 4). We provide a more detailed discussion and precise statement of Conjecture 5 in Section 4 Assuming the conjecture holds, we prove a limit thcorem about the distribution of n Informally, this says that aut is approximately a nag-Gassiian scalar times independent Gaussian vector
||||| (a² + x2)
8 + 2chrotal 3 + c² (Looaa) 1) Z
exp
-
.
(4)
/ //lin
2
where Z has iid (0, 1) entries, and 33 and are defined by
2 5X + 2a
(a² + x2)2 c= a x2
(5)
The precise statement, including asymptotic error bounds, is as follows.
Theorem 1. For any choice of hyperparameters Viny Down, n, d, >, and every input the output out at initialization has marginal distribution which can be written in the form
||a|||
exp (1 )/O) Z.
put
(a² +x2)
(6)
Tijn
Forr a quantity f = (n, d) whose dependence on width and depth may be implicit, we use the notation Odd /n") mean that, for all choice of constants a, A,, r+ 0. there exists constant C such that f(n d) < Cd" /n" for all d, n where r- < d/n < r+. This notation will allow us to state precise limit theorems when d, + with the ratio d/ n converging to constant.

---

Mean
Variance
-1.25
0.03
-1.50-
0.01
0.5.
-175
0.003
Slope 2
0.011
Slope
40
80
120
40
80
120
50
100
Empiricall - Prodicted
Vanilla RosNet Balamced ResNet
Prediction Error Linear Fit Vanilla RoaNet
(a) and Var from 6 and prediction.
(b) Prediction error for E and Var of G from
prediction for the random variable G(n, d, o, A) from (6) compared to the results of Theorems] 1 & 4 For these simulations,
Figure 2: Empirical mean and variance and anfnite-depth-and---iie
= 2/v2, d/n = are fixed and width n is varied on the x-axis. The error bars indicate 95% confidence interval (CI) of the Monte-Carlo simulation, with truncation at 1e-3 for plotting on CI and so are not plotted on
log-scale. The balanced ResNet predictions fall within the Monte-Caut log scale. Since d/n is fixed, Figure 2b indicates the error in the asymptotic prediction of variance 77) is O(n = O(dn" 3) as claimed. On the other hand, the error in mean (7) is O(n (() O(dn "); this is one order smaller than the statement proven Theorem ! sub-leading error term in our approximation for G' is mean zero.
possible explanation is that the
where 2 E Ro" is a Gaussian random vector with iid (0 entries, G G(n, is a random variable which is independent of Z and whose distribution does not depend on Thjns Thout or in such
Consider the limit where both the network depth d - and hidden layer width way that the ratio d/n converges to a non-zero constant. In this limit, assuming that Conjecture holds, then the random variable G E R. has the following asymptotic behaviour:
3 2chroul + O
d n2
E [|
Var [G] 3 + c² Leoaal + O
n1
where 3 and are as in 5) and moreover G converges in distribution to a Gaussian random variable with mean and variance given by 7 this limit.
The main ideas of the proof of Theorem is given in Section 5 and the detailed proof is given in Appendix B We also provide more explicit formulas for hoaual votal below.
Proposition Assume Conjecture 5 is true. Then in the same anfinite-detth and-width limit as the total hypoactivation hoota and total interlayer covariance Liout obey
Theorem 7
Ngoral Ca,A -00 d
[2(0
J2(T 0
d
(n²)
Iioral
Σ
n
+ 0
n2
(8)
SSFfe'Sd
Here Ca,y is a constant depending on a., and
Jz( = J2 (0) / 30) cos(0)/T (1- (1 + cos2 0),
where J2(0) first appeared in (36]. and 02 is such that cos(Ox) a /aa2 + /2/k/2
Remark 3. The Gaussian infinite-width limit predicts that the marginals of our should have the form
of (6 with G being identically zero. As gou depends exponentially G. the infinite-depth-and-vie limit predicts the variance is exponentially larger than the infinite-width limit. See Section 3.1 for detailed discussion and Figure for verification against finite networks.
Using Monte Carlo simulations, we estimate the constant Ca, The result of Theorem is then compared against finite networks in Figure 2 Full proof can be found Appendix B
The idea of using a scaling parameter a = 1/v2 the skip connections has been noted empirical papers 377 and also studied under simplified assumptions the number of activation in each layer
(38 Our result shows that by choosing a² + X2 1, the prefactor in (6) does not grow with depth thereby enabling deeper networks to be trained.
5

---

In the case that 0,
0, A= 1, the architecture reduces to fully connected network. In this case, log- Gaussian behaviour of the network was discovered Hanin and Nica (24). The fully connected case is simpler because the direction vectors EL are always uniformly distributed on the unit sphere and are independent from layer to layer. This means he 0 and 0, which greatly simplifies the result of Theorem
Furthermore, the proof easily extends the case where the coefficients az, de vary from layer to layer; see Appendix A.I for the general statement. Hayou et al. (29] and Hanin and Rolnick (33) have studied ResNets where 0g
every layer, but xe is allowed vary. The prefactor of our result in this case becomes [[ (1 + (ス) s exp (Ei >²) which is akin to the behaviour found in those
papers. Additionally, our result precisely quantifies the log- ogg-Gaussian behaviour and its dependence on the sequence X; through the parameters / and in the infinite-eepth-nee aad-th--nn-iddttt limit.
2.1 Log-Gaussian Rehaviour of Balanced ResNets
Given a collection of iid uniformly rundom signs, E ( a balanced ResNet is defined much like Vanilla ResNet, except that random sign is applied preactivation:
2' == + +.
W pue (z for slsd. n
where each layer the vector function fea _ + R" applies either 4 or 4- to the entries according to the random signs s². More precisely, the i-th component is
Pe (z); = (2,) = max 24 0), (z,) 14, 0),
s = + s
An equivalent definition is the entrywise multiplication (z) pp o 2). Note that the random signs of are not trainable parameters; they are frozen on initialization. This same symmetrization was first exploited by Allen-Zhu et al. [39] and Bai and Lee (40) to study quadratic approximation of
the network. We now present corresponding limiting theorem for Balanced ResNets.
Theorem 4. For balanced ResNet, the same result as Theorem 7 given in 6 still holds, but with the mean and variance of G given simply by
8 E [ = +0
d 2 n²
Var [G] = 0 d
Balanced ResNets are constructed so that the activation each neuron independent all others due to the random signs sf. This eliminates the hypoactivation and variance terms which complicated the analysis of the Vanilla ResNet and necessitated Conjecture 5 Instead, for Balanced ResNets straightforward to compute that for any fixed
I2 we have
[1))a(())) ||=1| Var [3)x
o Σ
n
Cov [IIp=((2]]
llpp (w))) 0. (10)
Even though the layers $ 2² are correlated, because the activation functions u and &e are set be independent on initialization, the correlation between layers does induce correlation on which neurons are activated from layer to layer. This explains why there is no hypoactivation and interlayer correlation correction Theorem as there is Theorem
Consequences of Theorems 1 & 4 and Comparison to Infinite-Widtl Limit
3.1 Vanishing and Exploding Norms
By the basic fact E[exp(N((H, xf))] exp(p + 100)) ))
follows from Theorems 1 4 that, when
the inputs has | / /nim the mean size scale of any neuron 2 is approximately
(a2 + x2)dexp (2cl + 2
for Vanilla ResNets, for Balanced ResNets.
E
(2
out ~
(a² + x2)d,

---

Mean
Variance
Correlation
2.25-
1/3
2-
1000
0.3-
1.75
1.5.
100
1.25
0.03-
10.
0.01
0
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Depth to Width Ratio d/n
Empirical
Predicted
Vanilla ResNet Balanced ResNet - Infinite Width
Figure 3: Behaviour of the mean and variance of the typical output (=qa1)² 2 and the correlation between ) (= 2 for two different output neurons. Here n 200, a = X 1/v2 and d/n varies on the r-axis. The infinite width prediction for correlation zero, but due to plotting on log-scale, we replace zero by 0.01 for display.
(Note that the terms with 3 cancel out!) When a² + + X²= =1 this is constant for Balanced ResNets. In contrast, Vanilla ResNets have a complicated dependence on the network depth d and width n due to the hypoactivation and correlations terms. This means the behaviour is exp( (dd/)), which is somewhat surprising. A more serious issue is the variance which Theorems [ predict to be
{ (a² 22)2d (3 exp (3 + c² Twoa) - 1) exp (4chou + e² (uouaa) for Vanilla, Var [+] ~ (a2 22)2d exp (3) 1)
for Balanced.
Since 3 z Cd/n, the term exp(3) represents exponentially larger variance for deep nets compared to shallow ones of the same width. In contrast, the variance predicted by the infinite width limit does not grow with depth for this model when a²² + X² This effect means the relative sizes of different network outputs can be widely disparate. Unlike problems with the mean, this issue is harder to resolve. For example, normalization methods that divide all neurons by a constant does nothing to address the large relative disparity between two points. Techniques like batch normalization will be skewed by large outliers. This kind of variance is known to obstruct training (33]. The input-output derivative 0z,²'" nd has the same type of behaviour - itself: a simple proof is given in Appendix B It is expected that the gradient with respect to the weights 0wg 2 will also have the same qualitative behaviour (27] although more investigation is needed to understand this theoretically. Exponentially large variance for gradients is a manifestation of the vanishing-ande and-exploding gradient problem (34)
Balanced ResNets suffer from this variance problem less because the interlayer correlation term zero. Since this variance reduction happens the exponential scale in, the difference can be significant; for networks with d/n 1, the contribution is factor of Re 5.5 s 250 for Vanilla ResNets vs. 2.5 Re 10 for Balanced ResNets. See Figure for a comparison of these theoretically predicted properties experiments with finite networks.
3.2 Correlated Output Neurons
Since the same random variable G multiplies the entire vector out the individual neurons in the output layer are not independent. For example. Theorem and predict that the squared entries have
strictly positive correlation given by Corr (==!u)² (= = (exp(o2) - 1)/(3 exp(o?) - 1)
for any two neurons + j where o² Var(G). This tends to 1/3 as d/n grows. The effect of correlated output neurons persists for Balanced ResNet but is reduced again due to the lower variance. This is very different from the sdfiite-widie limit, which predicts that individual neurons should be independent Gaussians. This prediction of the theorem matches finite networks closely; Figure
4 Conjecture 5) Hypoactivation and Layerwise Correlations
Vanilla ResNets have subtle asymmetry the architecture due to the skip connections. Unlike fully connected networks, the distribution of of == 2 ||2||| for l > 1 is not exactly uniformly distributed
7

---

(d)
16.3
3--4
3e-4.
1e-2
36-4
le44
1c-4
Slope -2
Slope -2
Slope
3.3.
Slope
3e-5
3e-5
3e.5.
100
70
100
200
100
100
Figure 4: Monte Carlo evidence for Conjecture d = 200, a = A = 1/v2, and n varies on the r-axis. The plots show the quantities (a)
- C(O)) (d)
(b) [Var| |12 Varl (
(c) |Cov (11++()9)1 (4 (+
C(02)). C(0k) is the theoretical covariance formula from the term where 8 - f- in S Note also that the absolute error is expected to be O(n-²) when the theoretical quantity is (n Figures (a) and (b) verify the conjecture in (IT) and (12). and Figures
Cov (1))+ (2
(c) and (d) verify 13 when
and A respectively. For display, clip the bottom edge of
the CI to 2e-5; 5; otherwise the error bar would go down to -oo on the log scale.
on the unit sphere. Informally speaking, 2 can be thought of random walk whose variance at each step proportional to [ More randomness is injected when [(5)( is large and less when it is small. The net effect is that the walk moves slower when (*')] is small, thereby spending more time in those locations. Consequently, I|+e+ (2*) is biased toward smaller values.
The size of this effect is limited by entropy; most of the unit sphere S"-1 has 11 (w)ll ~ 1/2 in the sense that for any > 0, the measure of the set {u (u)]+ 2 1/nt-r] vanishes as n + 00.
To quantify the effect of the bias, we can think of the evolution of 2f, l = 2, random walk that takes different step sizes at different points on the sphere. reasonable to expect that the behaviour of this processes will be similar to that of time changed Brownian motion, which slowed down at the points where g takes smaller steps. (Proving this comparison precisely technically difficult since the parameter simultancously plays both the role dimension and the step size of the walk.) Based on this heurstic comparison to time changed Brownian motion and on extensive Monte Carlo simulations we conjecture that the expected size of the hypoactivation effect only O(1/n) in expectation; Conjecture contains a precise statement.
Even if each layer 3 is marginally close to the uniform distribution the unit sphere, the directions &e and gf+1 are not independent because of the skip connections in the network. As above, the exact the exact number of neurons which are activated in each layer. However, using the idea that (3*))? = (1 + o(1)), we construct the following approximation. From (1)), we have the approximation 2 2=+1/ /|1= a& + >g /v o(1)) We observe that the norm of RHS concentrated around Va² X2 as n oo, so normalizing this to get ge+1 we have
behaviour complicated due fluctuations
getl Va +a=+Xa x2 vn (1+ (1+)
21+
a
Va²
Iterating this gives the same relationship for ghth where the first coefficient becomes a at/va² + x² As before, based Monte Carlo simulations, conjecture that the size of the error is O(1/n) in expectation. We formalize this as a precise statement Conjecture:]beloog
Conjecture 5. The distribution of the unit vector at / ||a*|| approximately uniformly dis- tributed from the wnit sphere u E Sn the precise sense that the following asymptoties hold
[="] =E [[e+((])]]] |1 (1l++(=*)))
E
(11)
(2 (25) = Var [1]]+]]]] (1+0
Var
(12)

---

E# > /n/nn
(2
0.500-
0.495
0.456
0.490
0.485
0.483
20
30
40
50
0
10
30
40
30
Layer
0.480
0,480
0.485
0.490
0.495
Vanilla ResNet Feedforward
(h) Autoregressive
hehaviour
(a) Hypoactivation as function of layer
B (I9) vs (5 I[(
B
Figure 5: Monte-Care simulation for the behaviour of the unit vector 8 as a function layer 5a shows 0 S E d. Here d 50, À v2 Saowws
the mean fraction of neurons which are activated, [ : 2f > 0}] /n and the norm the ReLU E [[]++ (=')] The hypoactivation he is how far this is from 5. At layer
0, 2 is uniformly distributed from a unit sphere, but approaches different steady-state as we go deeper into the network. Figure 55 shows evidence that the process E ( seems to be linear function of the previous layer E [|(+)+()=)))1))) Figure illlusstrtes the dependence as varies.
where the constants in the big O()) notation are uniform in Moreover, for two layers E, f, which are k 2 1 layers apart |e" El = k, the joint distribution of , is approximately equal to the joint distribution of u, cos(O2))u )u + sin(0 / where g is a Gaussian vector with iid N 1) entries which independent of and 0 is such that cos(Ox) a /(a? ( + X2)k/2 in the sense that the following asymptotics hold
Cov [1ee((3)12 (1 {19]
Cov ||l-(w))1 p
sin(Ok) 9)
(1 (13)
vn
where the constant in the big O() notation is uniform in E,
See Figure for Monte Carlo simulations empirically verifying the conjecture for fixed depth d, and see Figure for verifying the uniformity in layers &. In particular, we observe that Figure we can the effect of hypoactivation converges rapidly we can further verify in Figure [5b that hypoactivation appears to be autoregressive, which implics the convergence is exponentially fast. This motivated the uniformity in layers in Conjecture
equilibrium the layer increases. In fact,
Proof Ideas for Theorems 1 &
key element of the proof is the following property of Gaussian random matrices. W which has iid ((, 1) entries, then for any vector x, have
W x
|
(14)
where is a vector whose entries are iid (0, 1) random variables. Because of the fully connected first and last layer of the network, (4) implies that
| V Nin
||2 Vn g'
20
g,
out d
(15)
Hence G == In (11=d12//)) /n) (1|=]] /nta) (a² X2) only depends n, d, o, (Equiva-
lently, G has the distribution of In[[I="|] || 2 /n (a² A2) 4 when 20 g.) With this definition, [1
also shows out is proportional exp(G//i (G/2), establishing the first part of Theorem

---

From this construction, the essence of the proof is to understand the distribution of ||~0|| when 20 = To understand || we look the ratios || 21 layer by layer. By using the homogeneity property of ReLU i ([c| x)) |d| (x), we divide 26+1 from ① by
| obtain
++1 at + vv 2 w (0 a as + + n
2
|+ |+ g
(16)
where of are iid Gaussian vectors with iid NN 1) entries by application of (1 Hence
|| |
_ ar' + x |+ (2 e+1 _ aei + > ( |1
The last equality follows by applying an orthogonal transformation O such that Oz é (1, 0, .)... inside the norm, and observing that Gaussian random vectors are invariant under orthogonal transformations Og
Hence we have the telescoping product for ||=*1]
d-1
||=5|| = ||00||
II t=
αēi + +AV |1 (2
9 e+1
(17)
t=0
This shows that ||=*|] a product of random variables which are dependent on each other only through the terms [| (Note that ||=0|| is independent of 20 since 20 Gaussian.) Since
(**)"""]l" 1/2 with typical fluctuations on the scale 1 1/vn, therefore the dependence between terms of (17) is small.
/n) as sum of these weakly correlated random variables. Here we note that various tail estimates for the same or related quantities have been developed (3] 41J. however these estimates are not precise enough to pinpoint the exact limiting distribution. In contrast, we are able to derive the exact limiting distribution via a Central Limit Theorem (CLT) for
Taking the In of (17 exhibits In( |n[]|=9]]
weakly correlated sums (2) The proof of Theorem I is completed by computing the mean, variance and covariance terms using Conjecture For Theorem 4 the final calculation is simplified by (10)
which shows the terms uncorrelated. The detailed proof is given in Appendix B
Acknowledgement
We would like to thank Blair Bilodeau, Gintare Karolina Dziugaite, Mahdi Haghifam, Yani A. Ioannou, James Lucas, Jeffrey Negrea, Mengye Ren, and Ekansh Sharma for helpful discussions and draft feedback. We would also like to thank the anonymous NeurIPS reviewers for insightful feedback. In particular, one identified numerous relations to existing work, and another helped us identify the uniformity requirement in Conjecture [ ML is supported by Ontario Graduate Scholarship and the Vector Institute. MN is supported by an NSERC Discovery Grant. DMR supported in part by an NSERC Discovery Grant, Ontario Early Researcher Award, and stipend provided by the Charles Simonyi Endowment.
References
[1] A. Jacot, F. Gabriel, and C. Hongler. "Neural tangent kernel: Convergence and generalization in neural networks." Advances in Information Processing Systems (NeurIPS). 2018. arXiv:
1806.07572
[2] S. Du, J. Lee, H. Li, L. Wang, and X. Zhai. "Gradient descent finds global minima of deep neural networks." " Int. Conf. Machine Learning (ICML). PMLR. 2019, pp. 1675-1688.
[3] Z. Allen-Zhu, Y. Li, and Z. Song. "A convergence theory for deep learning via over- parameterization." Int. Conf. Machine Learning (ICML). PMLR. 2019, pp. 242-252.
[4] D. Zou, Y. Cao, D. Zhou, and Q. Gu. "Gradient descent optimizes over-parameterizee deep ReLU networks." In: Machine Learning 109.3 (2020), pp. 467-492.
[5] L. Chizat, E. Oyallon, and Bach. "On Lazy Training in Differentiable Programming." In: Advances in Neural Information Processing Systems 32 (2019), pp. 2937-2947.
10

---

[6] J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. 2019. Xrv: 1990206722
[7] G. Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. 2019. arXiv: 1902
04760
[8]
G. Yang. Tensor programs ii: Neural tangent kernel for any architecture. 2020. arXiv: 2006
14548
[9] S. Arora, S. S. Du, W. Hu, Z. Li, R. Salakhutdinov, and R. Wang- "On exact computation with an infinitely wide neural net." " Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019, pp. 8141-8150.
[10] Z. Chen, Y. Cao, D. Zou, and Q Gu. "How Much Over-parameterization Is Sufficient to Learn Deep Re[LU] Networks?" International Conference on Learning Representations. 2021. URL: https://openrevae
ntw/forumm wdw.rmmmdddddddw.e..m.... _2a6
[11] S. Arora, S. S. Du, Z. Li, R. Salakhutdinov, R. Wang, and D. Yu. Harnessing the power of infinitely wide deep nets small-data tasks. 2019. arXiv: 1910. 01663
[12] G. M. Rotskoff and E. Vanden- Eijnden. Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach. 2018. arXiv: 1805. 00915
[13] L. Chizat and F. Bach. On the Global Convergence of Gradient Descent for Over-parameterized
Models using Optimal Transport. 2018. arXiv: 1805 09545
[14] J. Sirignano and K. Spiliopoulos. Mean Field Analysis of Neural Networks: A Law of Large
Numbers. 2018 arXiv: 1805. L1805001053
[15] S. Mei, A. Montanari, and P.-M Nguyen. "A mean field view of the landscape of two-layer neural networks." In: Proceedings of the National Academy of Sciences 115.33 (2018), E7665- E7671. ISSN: 0027-8424, DOL: 10. 1073/ 1ww3/000...../// pnas. 1886579115
eprint: https: hattps://wwy pnas
org/coni /ooontenn/11//////1/11 33/6///3///66././/. full.pd.
[16] S. Mei, T. Misiakiewicz, and A. Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. arXiv: 1902. 06015
[17] G. Yang and F. J. Hu, "Feature Learning in Infinite- Width Neural Networks." " Int. Conf. Machine Learning (ICML.. 2021. arXiv: 2011 14522.
[18] R. M. Neal. Bayesian learning for neural networks. Vol. 118. Springer Science Business Media, 1995.
[19] G. Yang and S. Schoenholz. "Mean field residual networks: the edge of chaos." Advances in Neural Information Processing Systems. 2017, pp. 2865-2873.
[20] J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein. "Deep Neural Networks as Gaussian Processes." Ini. Conf. Learning Representations (ICLR). 2018.
[21] De Matthews, Hron, M. Rowland, R. Turner, and Z. Ghahramani. "Gaussian process behaviour in wide deep neural networks." Int. Conf. Learning Representations (ICLR). 2018.
[22] R. Novak, L. Xiao, Y. Bahri, J. Lee, G. Yang, J. Hron, D. A. Abolafia, J. Pennington, and
J. Sohl-dickstein. "Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes." Int. Conf. Learning Representations (ICLR). 2018
[23] G. Yang. Tensor programs Wide feedforward or recurrent neural networks of any architecture are gaussian processes. 2019. arXiv: 1910. 12478
[24] B. Hanin and M. Nica. "Products of many large random matrices and gradients in deep neural networks." " In: Communications in Mathematical Physics (2019), pp. 1-36.
[25] B. Hanin and M. Nica. "Finite Depth and Width Corrections to the Neural Tangent Kernel." Inf. Conf. Learning Representations (ICIR). 2019.
[26] M. Seleznova and G. Kutyniok. Analyzing Finite Neural Networks: Can We Trust Neural
Tangent Kernel Theory? 2020. arxiv: 2012. 04477
[27] B. Chmicl, L. Ben- Uri, M. Shkolnik, E. Hoffer, R. Banner, and D. Soudry. "Neural gradients
near-lognormal: improved quantized and sparse training." Int. Conf. Learning Representations URL: httptst//
(ICLR). arXiv: 2006. 08173
[28] K. He. X. Zhang, S. Ren, and J. Sun. "Deep residual learning for image recognition." Pro- ceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770- 778.
11

---

[29] S. Hayou, E. Clerico. B. He, G. Deligiannidis, A. Doucet, and J. Rousseau. "Stable ResNet." Int. Conf. Artificial Intelligence and Statistics (AISTATS). PMLR. 2021, pp. 1324-1332.
[30] H. Zhang, Y. N. Dauphin, and T. Ma. "Fixup Initialization: Residual Learning Without Nor- malization." International Conference on Learning Representations. 2018.
[31] Y. Blumenfeld, D. Gilboa, and D. Soudry. "Beyond signal propagation: feature diversity nec- essary in deep neural network initialization?" International Conference on Machine Learning. PMLR. 2020. pp. 960-969.
[32] D. Arpit, V. Campos Camúñiez, and Y. Bengio. "How to initialize your network? Robust initialization for WeightNorm& & ResNets." Advances in Neural Information Processing Systems
32 (NIPS 2019). 2019, pp. 1-10.
[33] B. Hanin and D. Rolnick. "How to Start Training: The Effect of Initialization and Architecture."
In: Advances in Neural Information Processing Systems 31 (2018).
[34] B. Hanin. "Which neural net architectures give rise exploding and vanishing gradients?"
Vol. 31. 2018.
[35] K. He, X. Zhang, S. Ren, and J. Sun. "Delving deep into rectifiers: Surpassing human- level performance on imagenet classification." Proc. IEEE Int. Conf. Computer Vision. 2015, pp. 1026-1034.
[36] Y. Cho and K. Saul. "Kemel methods for deep learning." Advances in Neural Information Processing Systems (NeurIPS). 2009, pp. 342-350.
[37]
P. Dhariwal and A. Nichol. Diffusion Models Beat GANs on Image Synthexis. 2021. arXiv:
2105. 05233
[38] D. Balduzzi, M. Frean, L. Leary, J. P. Lewis, K. W.-D. Ma, and B. McWilliams. "The Shattered Gradients Problem: If resnets are the answer, then what is the question?" Int. Conf. Machine Learning (ICML). PMLR. 2017, pp. 342-350.
[39] Z. Allen-Zhu, Y. Li, and Y. Liang. "Learning and generalization in overparameterized neural networks, going beyond two layers." " In: Advances in neural information processing systems (2019).
[40] Y. Bai and J. Lee. "Beyond Lincarization: On Quadratic and Higher-Orde Approximation of Wide Neural Networks." " International Conference on Learning Representations. 2019.
[41] S. Buchanan, D. Gilboa, and J. Wright. "Deep Networks and the Multiple Manifold Problem." International Conference on Learning Representations. 2021. URL: /tt::/tttps://oppeeeviss
net/forum? 7an-0mmid----m 0-P d_Q-]
[42] M. H. Neumann. "A central limit theorem for triangular arrays of weakly dependent random variables, with applications in statistics." In: ESAIM: PS 17 (2013), pp. 120-134. DO1: 10 0051/ps/201141
[43] A. J. Stam. "Limit Theorems for Uniform Distributions on Spheres in High- Singh--imessional Fu- clidean Spaces," In: Journal of Applied Probability 19.1 (1982), pp. 221-2221222222222222 ISSN: 00219002.
URL: http:://// / //wwww jsttrr org/ stable/32113333
[44] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne and Q. Zhang. JAX: composable trunsformations of Pytho++NumPP programs. Version 0.2.5. 2018. URL:
com/ google/ jax
[45] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Deto,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. "PyTorch: An Imperative Style, High-Performance Deep Learning Library." In: Advances in Neural Information Processing Systems 32. Ed. by H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, F. Fox, and R. Garnett. Curran Associates, Inc., 2019, pp. 8024 8035. URL: http: http::/ppppp.....pp////p neurips cc/ deep- 1earning
paper/9015 pytorch- an- imperative style- high library. pdf
[46]
C. R. Harris, K. J. Millman, J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, K. Sheppard, T. Reddy, W. Weckesser, H. Abbasi. C. Gohlke, and T. E. Oliphant. "Array programming with NumPy." In: Nature 585.7825 (Sept. 2020), pp. 357-362 DOL: 10. 100.//338118666 020
M. Brett, A. Haldane, J. F. del Río, M. Wiebe, P. Peterson, P. Sérard-Marchann
2649-2 URL.: https:///dot http://iii.oggg//// 10. 11000.138880 1038 020- 228-2
12

---

[47]
Plotnine Contributors. has2kl/plotmine: v0.8.0. Version v0.8.0. Mar. 2021. DOI: 1 5281
Zenodo. 4636791 URL:
5281/2oodo 4636791
[48] H. Wickham, ggplot2: Elegant Graphics for Data Analysis. Springer- New York, 2016.
ISBN: 978-3-419-2427744 URl: htttps:// http:///ppplot2...eeeeersee wwwyyiiiieeeo.om org
[49]
Pandas Contributors. pandax-dev/pandax: Pundas 1.0.3. Version v1.0.3. Mar. 2020, DOr:
10. 0ttps///zenoo..3715222.2 URL: httth::///oooii 1r.o0.orr//...58//eeeodo 3715232
[50]
Anaconda Software Distribution. Version Vers. 2-2.4.0. 2020. URL: https // docs
anaconda. comm/
[51] T. Kluyver, B. Ragan-Kelley. F. Pérez, B. Granger, M. Bussonnier, J. Frederic, K. Kelley, J. Hamrick, Grout, S. Corlay, P. Ivanov, D. Avila, S. Abdalla, and C. Willing. "Jupyter Notebooks a publishing format for reproducible computational workflows." Positioning and Power Academic Publishing: Players, Agents and Agendas. Ed. by F. Loizides and
B. Schmidt. IOS Press. 2016, pp. 87-90.
[52] D. P. Kingma and J. Ba. "Adam: A Method for Stochastic Optimization." ICLR (Poster). 2015.
URL: httt:://axii> org/ abs 1412. 6980
[53] K. Liu, W. Yang. P. Yang. and F. Ducau. pytorch-cifar,
com/ com/kuangliu/
pytorch- cifar 2017.
[54] K. He, X. Zhang, S. Ren, and J. Sun. "Identity mappings in deep residual networks." European conference on computer vision. Springer. 2016, pp. 630-645.
[55] A. Krizhevsky. "Learning Multiple Layers of Features from Tiny Images." In: (2009).
13


Reviews:
[{"review_id": "oLT2s8r6iaR", "reviewer": "Reviewer_SxtD", "summary": "The authors study signal propagation in randomly-initialized feedforward\nresidual neural networks with gaussian weights, with architectures consisting\nof one fully-connected layer per each residual block in the network.\nConditional on a certain conjecture about distributions of normalized\npreactivations in the residual network, they prove theorems that precisely\narticulate the distribution of the norm of the network output in a limit where\nthe network depth $d$ and width $n$ simultaneously tend to infinity and are\ndirectly related, and as a byproduct articulate precise dependences of the mean\nand variance of this output, as well as the inter-layer correlations of the\nnetwork, as functions of the depth and width. The authors claim interesting\nconsequences of this conditional analysis relative to predictions one would get\nby considering a limit where just the width $n$ is taken to be infinite (and\nthen possibly the depth) -- for example, they show that standard ResNets have\nless than half of their neurons in each layer activated, which can lead to\ncertain unstable output behaviors, and that networks in this limit (in\nparticular when $d > n$ as the limit is being taken) may have\nexponentially-large output variances, which may be challenging to mitigate\n(Section 3). The authors justify the conjecture underlying their theoretical\nresults with Monte Carlo simulations and a heuristic argument.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Summary (wrt rating)\n\nThe paper's study of signal propagation properties in simple ResNet\narchitectures is timely, and the conclusions are of potential general interest:\nthe authors demonstrate that infinite-width regime predictions about signal\npropagation in residual nets may not accurately reflect the performance of real\nnetworks, given certain depth dependences that vanish when $n \\to \\infty$\ncompletely (eqn below line 187 is especially unsavory). However, the paper has\nseveral limitations that lead me to rate it as such:\n- Almost all novel theoretical results in the paper hold only conditionally on\n  a certain conjecture about preactivation distributions in the network, which\n  the authors justify intuitively (section 4) but do not prove; this conjecture\n  seems to encapsulate most of the technical difficulty unique to the analysis\n  of ResNet signal propagation relative to the fully-connected case, and it\n  therefore does not seem appropriate to consider the work a contribution to\n  the literature on the theoretical analysis of ResNets in this state.\n- In light of the previous point making most theoretical results in the paper\n  only conjectural, I would expect a stronger practical component to the work\n  in order for it to be suitable for presentation at NeurIPS; however, most\n  theoretical results in the paper are limited to experiments verifying the\n  theoretical predictions (for ResNets with one-layer blocks) using Monte Carlo\n  simulation, rather than verifications on practical architectures or\n  demonstrations of predictions relevant to practice and mitigation strategies\n  (e.g. in this literature connections to the \"vanishing-exploding gradient\n  problem\" are often made; here the discussion in Section 3 seems mostly to\n  survey the theory's predictions rather than interpret it for practice).\n- The contributions do not seem to be sufficiently contextualized in the\n  literature.\n\nIt seems to me that the quality of the work would be greatly enhanced by either\na proof of Conjecture 5 (which would make it acceptable for the work to have a\nminimal practical component), or an expansion of the practical component of the\nwork (as above). I hope the authors will correct me if I have unfairly\ninterpreted the theory, its dependence on Conjecture 5, and its implications.\n\n## Detailed Comments\nI will intermix both strengths and weaknesses in the subsequent discussion.\n\n### Theoretical aspects\n- The mathematical writing of the authors in sections 2 onward is (rather\n  refreshingly) clear and rigorous, making it painless to understand and\n  interpret their theoretical results. \n- It seems to be a severe limitation of the work (as above) that almost all\n  novel theoretical conclusions in the paper depend on the unproven Conjecture\n  5 (as I see it -- the characterization of the moments of the random variable\n  $G$ and its asymptotic behavior via a CLT in Theorem 1; the precise\n  calculations of certain architecture-dependent constants appearing in these\n  moments in Proposition 2). This effectively makes the theoretical results\n  conjectural, only supported by Monte Carlo simulations -- and I am not sure\n  it is possible for such simulations to be adequately convincing at this\n  level, given that the state space one needs to explore is a two-parameter one\n  in (d, n). Although I suppose it could be fine in certain cases to establish\n  and present theorems that depend on a conjecture (e.g. conditional hardness\n  results in TCS; implications of the Riemann hypothesis...), the nature of the\n  conjecture in this paper (a statement about the distribution of the norms of\n  the normalized preactivations across layers of the ResNet) seems to me to\n  encapsulate most of the novel difficulty (i.e., what's different from the\n  well-understood FFNN case) in the analysis of things like the\n  network output norm and the inter-layer correlations in the ResNet context,\n  to the extent that it feels more like the present work skips key aspects of\n  the analysis of the problem compared to these examples of conjectural results\n  in other fields. I hope the authors will correct me if I have been unfair in\n  the way I have construed this issue.\n- The assertions for \"Balanced ResNets\" (Theorem 4), which do not\n  require Conjecture 5, seem to be treated by the authors as \"trivial\" (the\n  proof of Theorem 4 in the appendix is described as such) -- I would\n  appreciate clarification if there are any novel aspects to these proofs (e.g.\n  parts of the proof of Theorem 1 in the appendix that need to be carried over)\n  or whether the proofs become essentially identical to the fully-connected\n  case with this architectural modification. (The former case would speak to\n  the novelty of the authors' work.)\n- Question: does the equidistributional assertion in the proof sketch (eqn\n  (17)) require some additional justification, given that the random variables\n  one is asserting equidistributionality for (the factors in the product) are\n  dependent? I imagine this is benign; I just point it out since I didn't find\n  an additional justification for this manipulation in the proofs in the\n  appendix.\n\n### Practical aspects\n- Somewhat related to the next section, it seems like there is a large existing\n  literature on initialization schemes for ResNets, and it would be of interest\n  to hear the authors discuss any implications of their results relative to\n  these practical methods (and/or dismiss them as not directly relevant, etc.\n  if that is the case). For example, the authors mention Stable ResNet and the\n  work of Hanin and Rolnick in lines 158-163; what is the precise relation to\n  these works? (the authors mention previously that Stable ResNet considers\n  things in the infinite-width setting, which seems good; the authors explain\n  their Theorem 1 as more precise than Hanin and Rolnick's, but it would be\n  nice to see in what sense (given as well that Theorem 1 is conjectural)). \n  Also, what about comparisons to other initialization approaches that involve\n  (as far as I can tell) mitigation strategies for signal propagation issues in\n  ResNets -- e.g. [1-3] below.  It seems like it would be nice to have this\n  material presented in a Related Work section to easily understand how the\n  authors' contribution fits in here (maybe these approaches imagine the widths\n  are infinite, for example, giving evidence to the authors' claim that the\n  \"d/n constant\" setting has been neglected).\n- Question: why have the authors restricted their study to \"vanilla ResNets\",\n  which here means a standard ResNet where each block just has one fully\n  connected layer? It seems in the literature one commonly sees networks\n  studied that admit the possibility of multiple fully connected layers per\n  block; and moreover that these are common features in practical ResNets (I am\n  certainly not an expert here but I think ResNet-50 and so on have this\n  feature?).\n\n### Context\nThe authors should perhaps add a related work section and compare more extensively to\nprior art there -- I feel this would greatly benefit the paper by making the\nnovelty of the authors' contributions clear. I will discuss a few specific axes\nhere below -- perhaps the authors can clarify if they intentionally omitted\ndiscussion of these works because they see them as not directly relevant.\n\n- The identification of \"hypoactivation\" as a key driver of vanilla ResNet\n  signal propagation seems potentially of broad interest (even without rigorous\n  guarantees to characterize it). It seems like this is a novel discovery of\n  the authors -- it feels like additional discussion of relationships to other\n  ResNet initialization schemes + analyses in some kind of a new related work\n  section (as above) would go a long way towards emphasizing the authors' novel\n  insight in this connection.\n- I find the motivation around infinite depth-and-width limits in the\n  introduction to be somewhat incompletely specified. For example, there is a\n  rather large body of work that focuses on nonasymptotic guarantees for\n  training and generalization of FFNNs in the NTK regime, and many of these\n  apply to deep networks -- e.g. the authors cite Allen-Zhu et al., Zou et al.,\n  and Du et al. in the first sentence in connection to the large-width regime,\n  but all of these works allow one to take certain infinite-depth-and-width\n  limits in the theorems they prove and obtain various guarantees in this limit\n  (for training, signal propagation, etc.), and these aspects of these results\n  seem to be overlooked in the discussion starting at line 35. There also may\n  be some relevant omitted works in this connection (e.g. [4-7] below). \n  I find this kind of omission to be surprising given some of the later\n  motivation, e.g. sentence at line 63 about \"finite behaviour\".\n- The remainder of the paragraph starting at line 35 also seems somewhat\n  dubious without additional clarification relative to these works -- e.g. in\n  the sentence starting at line 38 (and somewhat in the remainder of the\n  paragraph), did the authors mean for this discussion to apply only to\n  networks where $d / n$ asymptotically approaches some constant value? This\n  would rule out the applicability of many of the aforementioned works,\n  although some may remain relevant, as the quoted reference here discusses\n  this infinite depth-and-width limit only for diagonal elements of the NTK and\n  not guarantees for the entire training process (for example, I believe that\n  certain technical lemmas in Allen-Zhu et al. apply to regimes where\n  the depth can grow linearly with the width). It might be beneficial to add\n  some additional clarification to this discussion.\n- The \"Balanced ResNets\" proposal seems to be a standard 'trick' in the\n  theoretical deep learning literature in order to introduce theoretical\n  tractability -- for example, I have seen this used prominently (for a\n  slightly different purpose) in [8] below, and that work cites other previous\n  deep learning theory papers. I would not imagine the authors need to credit\n  these works for the idea, as this seems to be even more broadly a standard\n  trick in probability theory, but rather to reference relevant prior art so\n  that this does not seem to be claimed as a unique architectural innovation.\n- It would be nice to have a comparison to [9], which seems relevant --\n  although I cannot fault the authors for not including this as I do not\n  believe the work was available until after the authors submitted their work.\n  Nevertheless, it seems relevant (and possibly an opportunity to emphasize the\n  novelty of the authors' perspective if there is a $d/n$ comparison to be\n  made). [10] also seems like it should be compared to.\n\n\n[1] http://arxiv.org/abs/1901.09321\n\n[2] http://proceedings.mlr.press/v119/blumenfeld20a.html\n\n[3] https://arxiv.org/abs/1906.02341\n\n[4] https://arxiv.org/abs/1904.11955\n\n[5] https://openreview.net/forum?id=fgd7we_uZa6\n\n[6] https://openreview.net/forum?id=O-6Pm_d_Q-\n\n[7] https://arxiv.org/abs/2006.06657\n\n[8] https://openreview.net/forum?id=rkllGyBFPH\n\n[9] http://proceedings.mlr.press/v139/hu21b.html\n\n[10] https://arxiv.org/abs/2001.10460", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "(checklist 1a) The theoretical results in section 2 onwards are completely\nclear, and accurately present all limitations of the results, but I do not\nthink the paper's contributions as framed in the abstract and introduction\naccurately reflect the paper's theoretical results' dependence on the unproven\nConjecture 5. E.g. line 9 \"We show theoretically ...\" in the abstract should\nhave something like \"Assuming a certain conjecture on the distributions of the\npreactivations in the network, ...\" prepended to accurately reflect what the\npaper shows; lines 58-60 in the introduction do not unambiguously state that\n\"find[ing] a simple approximation\" is an empirical/conjectural activity, rather\nthan theoretical\"; line 89 second bullet in relation to the previous bullet is\nsimilar.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "epoOvDLlnGI", "reviewer": "Reviewer_QBeZ", "summary": "The papers studies the infinite width/depth limit of ReLU ResNets, to contrast it to the more usual infinite width/fixed depth limit. They show (under the assumption that a certain conjecture is true) that in the infinite width/depth limit, the limiting distribution of the outputs of the network at initialization (for i.i.d. Gaussian entries) is not Gaussian but log-Gaussian. While this was known for fully-connected networks, proving it for ResNets is more difficult due to dependence between the layers. This is related to the fact that less than half the neurons are active and that there exists correlations between layers of the network (which does not happen for fully-connected networks). These two properties of ResNets however lead to very large variances at initialization which can make training difficult, the authors therefore propose a balanced ResNets (where the activation at each neuron is chosen randomly between the ReLU and the \"negative\" ReLU) which ensures that half the neurons are active and removes the correlations, however the variance at initialization (though smaller than that of ResNets) can still be large (much larger than what is predicted by the infinite width limit). Finally the results are checked empirically, showing that even for small depth/width ratios (say 0.1) the combined with/depth limit describes the statistics of DNNs more accurately than the fixed depth, infinite with limit.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As discussed by the authors in the introduction, it is important to understand the limits of the NTK regime. While a lot of recent work has focused on the active regime for finite depths (especially for shallow networks with the mean-field limit) fewer papers have studied the large depth limit. The fact that less than half of the activations are active at initialization and the correlations between layers was unexpected to me, but the paper does a good job explaining how these arise, and show the impact these quantities have on the distribution of the outputs. The numerical experiments show the relevance of this combined limit even for small depth/width ratios, yielding much better approximations of some statistics of DNNs than the infinite width limit.\n\nThe article is well written, the Conjecture is well discussed and the sketch of proof is very helpful. I am not familiar enough with this line of work to determine whether the Conjecture is reasonable.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The requirements of a Conjecture for the proofs of the main Theorem is of course a weakness, though the Conjecture itself is well discussed and checked empirically. This work is theoretical and has no direct societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UMjE4N8jBh", "reviewer": "Reviewer_xrqc", "summary": "The paper calculates the forward pass at initialization of resnets in the joint proportional limit of infinite width and depth. In particular, the paper shows that this limit exhibits log-Gaussian behaviors, extending what was previously known for feedforward networks. From here, the paper makes several interesting observations regarding interlayer correlation, enlarged output variance and how to alleviate them. The proof crucially exploits homogeneity of the ReLU activation function, together with an assumed conjecture on the pre-activation vector’s direction being almost uniformly distributed.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There have been a lot of interests in analyses of infinite-width networks, but it is possible — such as in NTK limit — that the infinite width limit results in undesirable properties. One escape from this scenario is to consider the joint limit of width and depth (which was possibly first initiated by Hanin, to the best of my knowledge). This paper is a nice and timely contribution to this line of works; the fact that the limiting behavior is log-Gaussian, which is hypothetically more realistic, is interesting.\n\nI have not spotted any technical flaws yet, and the presentation is nice and clean. A criticism, if any, is perhaps on the implication of the result: would correlated neurons at initialization and somewhat bigger (but not exploding) output variance be a bad thing to have? Would eliminating them via the balanced scheme necessarily be a better thing? I’m not yet convinced by the plots in Appendix C, but to be fair, this is certainly beyond the scope of the paper. Another criticism is perhaps on the limitation of the theoretical result itself, which could have been more informative if the backward pass at initialization could be calculated. I do appreciate the efforts to do this kind of exact calculations though, given that this is typically a challenging calculation to do.\n\nI have two specific remarks:\n\n(1) The proof idea in Section 5 is clean, but it lacks an explanation why the case $\\alpha\\neq 0$ is much harder than $\\alpha=0$ and if there is any new technical element to deal with the complication of $\\alpha\\neq 0$. In particular, a more detailed comparison with [22] in terms of technicality would be useful, given that this is a purely theoretical paper.\n\n(2) It seems homogeneity of ReLU is critical to obtain log-Gaussian behavior. My impression is that, following the argument in Section 5,\n\n$$\n\\log \\Vert z^{\\ell+1} \\Vert = \\log \\bigg\\Vert \\alpha e_1 + \\lambda \\sqrt{2/n} \\Vert z^\\ell\\Vert^{-1} \\Big\\Vert\\varphi(\\Vert z^\\ell\\Vert \\hat{z}^\\ell)\\Big\\Vert g^{\\ell+1} \\bigg\\Vert + \\log \\Vert z^\\ell \\Vert\n$$\n\nand as such, when $\\Vert z^\\ell\\Vert$ is not cancelled out inside the first term (e.g. when $\\varphi$ is not homogeneous), the terms do not seem to be weakly dependent. It is then unclear why log-Gaussianity should arise. Would it be possible to comment more on this? I somehow feel this is important, since the title of the paper conveys the sentiment that log-Gaussianity is expected to be universal.\n\nA minor comment: In line 42, “linear model” —> “linear method”.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "1k8gq11knp", "reviewer": "Reviewer_Kh9k", "summary": "This paper aims to deal with the problem that the gaussian process approximation of Neural network at initialization will become worse while the network become deeper due to the error accumulation. This paper proposed a central limit theorem in depth to have a log-gaussian limit of the infinite width and depth limit.\n\nThe subject of paper is pretty cool, but the assumption in the paper and conclusion doesn't convinced me the new characterization is right and essential.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is actually interesting although I provide a weak rejection decision, this paper\n\n+ Provide a new limit of deep resnet and verified via experiment\n\n\nMajor points:\n\n1. the (\\sqrt{lambda^2+sigma^2}^d) in the limit will goes to the infinity which is not a good way to claim the convergence, you'de better to normalize it at left hand side, i.e. Z^{out}/\\sqrt{lambda^2+sigma^2}^d convergence to ....\n\nIn the realistic resnet, we never use the lambda their, i.e. the (\\sqrt{lambda^2+sigma^2}^d) will goes to infinity. If we consider a Neural ODE limit the sigma will also depends on the depth, as an example \\sqrt{lambda^2+1/3}^d --> constant sounds a better limit.\n\n2.Why the log-gaussian process matters for understanding ResNet? I can't find evidence in the paper. \n\nFor the empirical paper you cited they said the gradient is log-gaussian but not the features, here is still a gap for the backward process is strongly correlated with the forward one, how the CLT can be used is not obvious to me.\n\n\n3. I'm not satisfied with the assumption, the O(1/n) error in assumption is uniformly on depth in my mind? I would like the author to clarify this and provide intuition why this can hold?\n\n\n4. Can the assumption be proved via induction like Greg Yang's tensor program papers?\n\n5.We know that infinitely wide NN is a gaussian **process**, this paper only tells us what's the feature distribution of a single data point but failed to answer what's the process. Can the proof generalize to the off-diag term in the correlation matrix of the process? \nI think it relates to quesiton2. If you add a layer norm before the last layer, bot log-gaussian and gaussian will go back to a uniform distribution on the sphere. I wonder whether this paper is playing on the scaling. I think investigating off-diag term can be much helpful\n\n\n\nMinor points:\n1.I can't see the proposed benefit of the balanced ResNet in practice, if this part is deleted, I think no information will lost in my mind.\n\n2.For the experiment, if you can show keep n/d as a constant and the limiting distribution is not changing can be interesting.\n\n\n3.although the proof is not hard, it takes some time to follow it. I hope the author can take some time to organize the prove. [the logic of the proof and lemmas can be put in a section and the detailed calculation of variance can be put later\n\nOther question doesn't relate to the decision\nTo deal with deeper network [1] and their recent book provide a finite depth correction to do this, can their approaches gives a log-gaussian?\n\n\n[1] B. Hanin and M. Nica. “Finite Depth and Width Corrections to the Neural Tangent Kernel.”\nInt. Conf. Learning Representations (ICLR). 2019.\n\n \n\nI believe this paper leads to an interesting direction, but at this point I still find out the paper can't convince me of the conclusion given. \n\n=======================================\n\nAfter rebuttal the reviewer still doesn't convinced me the reason why the assumption is not strong, thus I changed my score to 4.\n\nI think the author need time to\n- distinguish the difference and relationship with the previous techniques\n- clarify why I can trust the assumption is uniform in d", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See main review\n\n\nAdditional point：\nCan authors explain the relationship between your paper and [1]\nI understand [1] is an ICML2021 paper and appears after Neurips deadline thus I'll not take the answer of this question into the decision\n\n[1] On the Random Conjugate Kernel and Neural Tangent Kernel http://proceedings.mlr.press/v139/hu21b/hu21b.pdf\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The papers studies the infinite width and depth limit of ReLU ResNets, in particular the limiting distribution of the outputs of the network, for which under certain conjectured assumptions, is claimed to be log-Gaussian. While the reviewers have mixed opinions, after an extensive discussion between authors and reviewers and among reviewers themselves, most agree on the result being interesting and novel. While there are still some questions on technicality, in particular whether some of the conjectures hold true, in view of encouraging novelty and new insights, the meta-review would recommend acceptance of the paper as a poster to the conference. 
Decision: Accept (Poster)