Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors study signal propagation in randomly-initialized feedforward
residual neural networks with gaussian weights, with architectures consisting
of one fully-connected layer per each residual block in the network.
Conditional on a certain conjecture about distributions of normalized
preactivations in the residual network, they prove theorems that precisely
articulate the distribution of the norm of the network output in a limit where
the network depth $d$ and width $n$ simultaneously tend to infinity and are
directly related, and as a byproduct articulate precise dependences of the mean
and variance of this output, as well as the inter-layer correlations of the
network, as functions of the depth and width. The authors claim interesting
consequences of this conditional analysis relative to predictions one would get
by considering a limit where just the width $n$ is taken to be infinite (and
then possibly the depth) -- for example, they show that standard ResNets have
less than half of their neurons in each layer activated, which can lead to
certain unstable output behaviors, and that networks in this limit (in
particular when $d > n$ as the limit is being taken) may have
exponentially-large output variances, which may be challenging to mitigate
(Section 3). The authors justify the conjecture underlying their theoretical
results with Monte Carlo simulations and a heuristic argument.

main_review: ## Summary (wrt rating)

The paper's study of signal propagation properties in simple ResNet
architectures is timely, and the conclusions are of potential general interest:
the authors demonstrate that infinite-width regime predictions about signal
propagation in residual nets may not accurately reflect the performance of real
networks, given certain depth dependences that vanish when $n \to \infty$
completely (eqn below line 187 is especially unsavory). However, the paper has
several limitations that lead me to rate it as such:
- Almost all novel theoretical results in the paper hold only conditionally on
  a certain conjecture about preactivation distributions in the network, which
  the authors justify intuitively (section 4) but do not prove; this conjecture
  seems to encapsulate most of the technical difficulty unique to the analysis
  of ResNet signal propagation relative to the fully-connected case, and it
  therefore does not seem appropriate to consider the work a contribution to
  the literature on the theoretical analysis of ResNets in this state.
- In light of the previous point making most theoretical results in the paper
  only conjectural, I would expect a stronger practical component to the work
  in order for it to be suitable for presentation at NeurIPS; however, most
  theoretical results in the paper are limited to experiments verifying the
  theoretical predictions (for ResNets with one-layer blocks) using Monte Carlo
  simulation, rather than verifications on practical architectures or
  demonstrations of predictions relevant to practice and mitigation strategies
  (e.g. in this literature connections to the "vanishing-exploding gradient
  problem" are often made; here the discussion in Section 3 seems mostly to
  survey the theory's predictions rather than interpret it for practice).
- The contributions do not seem to be sufficiently contextualized in the
  literature.

It seems to me that the quality of the work would be greatly enhanced by either
a proof of Conjecture 5 (which would make it acceptable for the work to have a
minimal practical component), or an expansion of the practical component of the
work (as above). I hope the authors will correct me if I have unfairly
interpreted the theory, its dependence on Conjecture 5, and its implications.

## Detailed Comments
I will intermix both strengths and weaknesses in the subsequent discussion.

### Theoretical aspects
- The mathematical writing of the authors in sections 2 onward is (rather
  refreshingly) clear and rigorous, making it painless to understand and
  interpret their theoretical results. 
- It seems to be a severe limitation of the work (as above) that almost all
  novel theoretical conclusions in the paper depend on the unproven Conjecture
  5 (as I see it -- the characterization of the moments of the random variable
  $G$ and its asymptotic behavior via a CLT in Theorem 1; the precise
  calculations of certain architecture-dependent constants appearing in these
  moments in Proposition 2). This effectively makes the theoretical results
  conjectural, only supported by Monte Carlo simulations -- and I am not sure
  it is possible for such simulations to be adequately convincing at this
  level, given that the state space one needs to explore is a two-parameter one
  in (d, n). Although I suppose it could be fine in certain cases to establish
  and present theorems that depend on a conjecture (e.g. conditional hardness
  results in TCS; implications of the Riemann hypothesis...), the nature of the
  conjecture in this paper (a statement about the distribution of the norms of
  the normalized preactivations across layers of the ResNet) seems to me to
  encapsulate most of the novel difficulty (i.e., what's different from the
  well-understood FFNN case) in the analysis of things like the
  network output norm and the inter-layer correlations in the ResNet context,
  to the extent that it feels more like the present work skips key aspects of
  the analysis of the problem compared to these examples of conjectural results
  in other fields. I hope the authors will correct me if I have been unfair in
  the way I have construed this issue.
- The assertions for "Balanced ResNets" (Theorem 4), which do not
  require Conjecture 5, seem to be treated by the authors as "trivial" (the
  proof of Theorem 4 in the appendix is described as such) -- I would
  appreciate clarification if there are any novel aspects to these proofs (e.g.
  parts of the proof of Theorem 1 in the appendix that need to be carried over)
  or whether the proofs become essentially identical to the fully-connected
  case with this architectural modification. (The former case would speak to
  the novelty of the authors' work.)
- Question: does the equidistributional assertion in the proof sketch (eqn
  (17)) require some additional justification, given that the random variables
  one is asserting equidistributionality for (the factors in the product) are
  dependent? I imagine this is benign; I just point it out since I didn't find
  an additional justification for this manipulation in the proofs in the
  appendix.

### Practical aspects
- Somewhat related to the next section, it seems like there is a large existing
  literature on initialization schemes for ResNets, and it would be of interest
  to hear the authors discuss any implications of their results relative to
  these practical methods (and/or dismiss them as not directly relevant, etc.
  if that is the case). For example, the authors mention Stable ResNet and the
  work of Hanin and Rolnick in lines 158-163; what is the precise relation to
  these works? (the authors mention previously that Stable ResNet considers
  things in the infinite-width setting, which seems good; the authors explain
  their Theorem 1 as more precise than Hanin and Rolnick's, but it would be
  nice to see in what sense (given as well that Theorem 1 is conjectural)). 
  Also, what about comparisons to other initialization approaches that involve
  (as far as I can tell) mitigation strategies for signal propagation issues in
  ResNets -- e.g. [1-3] below.  It seems like it would be nice to have this
  material presented in a Related Work section to easily understand how the
  authors' contribution fits in here (maybe these approaches imagine the widths
  are infinite, for example, giving evidence to the authors' claim that the
  "d/n constant" setting has been neglected).
- Question: why have the authors restricted their study to "vanilla ResNets",
  which here means a standard ResNet where each block just has one fully
  connected layer? It seems in the literature one commonly sees networks
  studied that admit the possibility of multiple fully connected layers per
  block; and moreover that these are common features in practical ResNets (I am
  certainly not an expert here but I think ResNet-50 and so on have this
  feature?).

### Context
The authors should perhaps add a related work section and compare more extensively to
prior art there -- I feel this would greatly benefit the paper by making the
novelty of the authors' contributions clear. I will discuss a few specific axes
here below -- perhaps the authors can clarify if they intentionally omitted
discussion of these works because they see them as not directly relevant.

- The identification of "hypoactivation" as a key driver of vanilla ResNet
  signal propagation seems potentially of broad interest (even without rigorous
  guarantees to characterize it). It seems like this is a novel discovery of
  the authors -- it feels like additional discussion of relationships to other
  ResNet initialization schemes + analyses in some kind of a new related work
  section (as above) would go a long way towards emphasizing the authors' novel
  insight in this connection.
- I find the motivation around infinite depth-and-width limits in the
  introduction to be somewhat incompletely specified. For example, there is a
  rather large body of work that focuses on nonasymptotic guarantees for
  training and generalization of FFNNs in the NTK regime, and many of these
  apply to deep networks -- e.g. the authors cite Allen-Zhu et al., Zou et al.,
  and Du et al. in the first sentence in connection to the large-width regime,
  but all of these works allow one to take certain infinite-depth-and-width
  limits in the theorems they prove and obtain various guarantees in this limit
  (for training, signal propagation, etc.), and these aspects of these results
  seem to be overlooked in the discussion starting at line 35. There also may
  be some relevant omitted works in this connection (e.g. [4-7] below). 
  I find this kind of omission to be surprising given some of the later
  motivation, e.g. sentence at line 63 about "finite behaviour".
- The remainder of the paragraph starting at line 35 also seems somewhat
  dubious without additional clarification relative to these works -- e.g. in
  the sentence starting at line 38 (and somewhat in the remainder of the
  paragraph), did the authors mean for this discussion to apply only to
  networks where $d / n$ asymptotically approaches some constant value? This
  would rule out the applicability of many of the aforementioned works,
  although some may remain relevant, as the quoted reference here discusses
  this infinite depth-and-width limit only for diagonal elements of the NTK and
  not guarantees for the entire training process (for example, I believe that
  certain technical lemmas in Allen-Zhu et al. apply to regimes where
  the depth can grow linearly with the width). It might be beneficial to add
  some additional clarification to this discussion.
- The "Balanced ResNets" proposal seems to be a standard 'trick' in the
  theoretical deep learning literature in order to introduce theoretical
  tractability -- for example, I have seen this used prominently (for a
  slightly different purpose) in [8] below, and that work cites other previous
  deep learning theory papers. I would not imagine the authors need to credit
  these works for the idea, as this seems to be even more broadly a standard
  trick in probability theory, but rather to reference relevant prior art so
  that this does not seem to be claimed as a unique architectural innovation.
- It would be nice to have a comparison to [9], which seems relevant --
  although I cannot fault the authors for not including this as I do not
  believe the work was available until after the authors submitted their work.
  Nevertheless, it seems relevant (and possibly an opportunity to emphasize the
  novelty of the authors' perspective if there is a $d/n$ comparison to be
  made). [10] also seems like it should be compared to.


[1] http://arxiv.org/abs/1901.09321

[2] http://proceedings.mlr.press/v119/blumenfeld20a.html

[3] https://arxiv.org/abs/1906.02341

[4] https://arxiv.org/abs/1904.11955

[5] https://openreview.net/forum?id=fgd7we_uZa6

[6] https://openreview.net/forum?id=O-6Pm_d_Q-

[7] https://arxiv.org/abs/2006.06657

[8] https://openreview.net/forum?id=rkllGyBFPH

[9] http://proceedings.mlr.press/v139/hu21b.html

[10] https://arxiv.org/abs/2001.10460
limitations_and_societal_impact: (checklist 1a) The theoretical results in section 2 onwards are completely
clear, and accurately present all limitations of the results, but I do not
think the paper's contributions as framed in the abstract and introduction
accurately reflect the paper's theoretical results' dependence on the unproven
Conjecture 5. E.g. line 9 "We show theoretically ..." in the abstract should
have something like "Assuming a certain conjecture on the distributions of the
preactivations in the network, ..." prepended to accurately reflect what the
paper shows; lines 58-60 in the introduction do not unambiguously state that
"find[ing] a simple approximation" is an empirical/conjectural activity, rather
than theoretical"; line 89 second bullet in relation to the previous bullet is
similar.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The papers studies the infinite width/depth limit of ReLU ResNets, to contrast it to the more usual infinite width/fixed depth limit. They show (under the assumption that a certain conjecture is true) that in the infinite width/depth limit, the limiting distribution of the outputs of the network at initialization (for i.i.d. Gaussian entries) is not Gaussian but log-Gaussian. While this was known for fully-connected networks, proving it for ResNets is more difficult due to dependence between the layers. This is related to the fact that less than half the neurons are active and that there exists correlations between layers of the network (which does not happen for fully-connected networks). These two properties of ResNets however lead to very large variances at initialization which can make training difficult, the authors therefore propose a balanced ResNets (where the activation at each neuron is chosen randomly between the ReLU and the "negative" ReLU) which ensures that half the neurons are active and removes the correlations, however the variance at initialization (though smaller than that of ResNets) can still be large (much larger than what is predicted by the infinite width limit). Finally the results are checked empirically, showing that even for small depth/width ratios (say 0.1) the combined with/depth limit describes the statistics of DNNs more accurately than the fixed depth, infinite with limit.
main_review: As discussed by the authors in the introduction, it is important to understand the limits of the NTK regime. While a lot of recent work has focused on the active regime for finite depths (especially for shallow networks with the mean-field limit) fewer papers have studied the large depth limit. The fact that less than half of the activations are active at initialization and the correlations between layers was unexpected to me, but the paper does a good job explaining how these arise, and show the impact these quantities have on the distribution of the outputs. The numerical experiments show the relevance of this combined limit even for small depth/width ratios, yielding much better approximations of some statistics of DNNs than the infinite width limit.

The article is well written, the Conjecture is well discussed and the sketch of proof is very helpful. I am not familiar enough with this line of work to determine whether the Conjecture is reasonable.
limitations_and_societal_impact: The requirements of a Conjecture for the proofs of the main Theorem is of course a weakness, though the Conjecture itself is well discussed and checked empirically. This work is theoretical and has no direct societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper calculates the forward pass at initialization of resnets in the joint proportional limit of infinite width and depth. In particular, the paper shows that this limit exhibits log-Gaussian behaviors, extending what was previously known for feedforward networks. From here, the paper makes several interesting observations regarding interlayer correlation, enlarged output variance and how to alleviate them. The proof crucially exploits homogeneity of the ReLU activation function, together with an assumed conjecture on the pre-activation vector’s direction being almost uniformly distributed.
main_review: There have been a lot of interests in analyses of infinite-width networks, but it is possible — such as in NTK limit — that the infinite width limit results in undesirable properties. One escape from this scenario is to consider the joint limit of width and depth (which was possibly first initiated by Hanin, to the best of my knowledge). This paper is a nice and timely contribution to this line of works; the fact that the limiting behavior is log-Gaussian, which is hypothetically more realistic, is interesting.

I have not spotted any technical flaws yet, and the presentation is nice and clean. A criticism, if any, is perhaps on the implication of the result: would correlated neurons at initialization and somewhat bigger (but not exploding) output variance be a bad thing to have? Would eliminating them via the balanced scheme necessarily be a better thing? I’m not yet convinced by the plots in Appendix C, but to be fair, this is certainly beyond the scope of the paper. Another criticism is perhaps on the limitation of the theoretical result itself, which could have been more informative if the backward pass at initialization could be calculated. I do appreciate the efforts to do this kind of exact calculations though, given that this is typically a challenging calculation to do.

I have two specific remarks:

(1) The proof idea in Section 5 is clean, but it lacks an explanation why the case $\alpha\neq 0$ is much harder than $\alpha=0$ and if there is any new technical element to deal with the complication of $\alpha\neq 0$. In particular, a more detailed comparison with [22] in terms of technicality would be useful, given that this is a purely theoretical paper.

(2) It seems homogeneity of ReLU is critical to obtain log-Gaussian behavior. My impression is that, following the argument in Section 5,

$$
\log \Vert z^{\ell+1} \Vert = \log \bigg\Vert \alpha e_1 + \lambda \sqrt{2/n} \Vert z^\ell\Vert^{-1} \Big\Vert\varphi(\Vert z^\ell\Vert \hat{z}^\ell)\Big\Vert g^{\ell+1} \bigg\Vert + \log \Vert z^\ell \Vert
$$

and as such, when $\Vert z^\ell\Vert$ is not cancelled out inside the first term (e.g. when $\varphi$ is not homogeneous), the terms do not seem to be weakly dependent. It is then unclear why log-Gaussianity should arise. Would it be possible to comment more on this? I somehow feel this is important, since the title of the paper conveys the sentiment that log-Gaussianity is expected to be universal.

A minor comment: In line 42, “linear model” —> “linear method”.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper aims to deal with the problem that the gaussian process approximation of Neural network at initialization will become worse while the network become deeper due to the error accumulation. This paper proposed a central limit theorem in depth to have a log-gaussian limit of the infinite width and depth limit.

The subject of paper is pretty cool, but the assumption in the paper and conclusion doesn't convinced me the new characterization is right and essential.  
main_review: This paper is actually interesting although I provide a weak rejection decision, this paper

+ Provide a new limit of deep resnet and verified via experiment


Major points:

1. the (\sqrt{lambda^2+sigma^2}^d) in the limit will goes to the infinity which is not a good way to claim the convergence, you'de better to normalize it at left hand side, i.e. Z^{out}/\sqrt{lambda^2+sigma^2}^d convergence to ....

In the realistic resnet, we never use the lambda their, i.e. the (\sqrt{lambda^2+sigma^2}^d) will goes to infinity. If we consider a Neural ODE limit the sigma will also depends on the depth, as an example \sqrt{lambda^2+1/3}^d --> constant sounds a better limit.

2.Why the log-gaussian process matters for understanding ResNet? I can't find evidence in the paper. 

For the empirical paper you cited they said the gradient is log-gaussian but not the features, here is still a gap for the backward process is strongly correlated with the forward one, how the CLT can be used is not obvious to me.


3. I'm not satisfied with the assumption, the O(1/n) error in assumption is uniformly on depth in my mind? I would like the author to clarify this and provide intuition why this can hold?


4. Can the assumption be proved via induction like Greg Yang's tensor program papers?

5.We know that infinitely wide NN is a gaussian **process**, this paper only tells us what's the feature distribution of a single data point but failed to answer what's the process. Can the proof generalize to the off-diag term in the correlation matrix of the process? 
I think it relates to quesiton2. If you add a layer norm before the last layer, bot log-gaussian and gaussian will go back to a uniform distribution on the sphere. I wonder whether this paper is playing on the scaling. I think investigating off-diag term can be much helpful



Minor points:
1.I can't see the proposed benefit of the balanced ResNet in practice, if this part is deleted, I think no information will lost in my mind.

2.For the experiment, if you can show keep n/d as a constant and the limiting distribution is not changing can be interesting.


3.although the proof is not hard, it takes some time to follow it. I hope the author can take some time to organize the prove. [the logic of the proof and lemmas can be put in a section and the detailed calculation of variance can be put later

Other question doesn't relate to the decision
To deal with deeper network [1] and their recent book provide a finite depth correction to do this, can their approaches gives a log-gaussian?


[1] B. Hanin and M. Nica. “Finite Depth and Width Corrections to the Neural Tangent Kernel.”
Int. Conf. Learning Representations (ICLR). 2019.

 

I believe this paper leads to an interesting direction, but at this point I still find out the paper can't convince me of the conclusion given. 

=======================================

After rebuttal the reviewer still doesn't convinced me the reason why the assumption is not strong, thus I changed my score to 4.

I think the author need time to
- distinguish the difference and relationship with the previous techniques
- clarify why I can trust the assumption is uniform in d
limitations_and_societal_impact: See main review


Additional point：
Can authors explain the relationship between your paper and [1]
I understand [1] is an ICML2021 paper and appears after Neurips deadline thus I'll not take the answer of this question into the decision

[1] On the Random Conjugate Kernel and Neural Tangent Kernel http://proceedings.mlr.press/v139/hu21b/hu21b.pdf
review_title: null

Meta-review: The papers studies the infinite width and depth limit of ReLU ResNets, in particular the limiting distribution of the outputs of the network, for which under certain conjectured assumptions, is claimed to be log-Gaussian. While the reviewers have mixed opinions, after an extensive discussion between authors and reviewers and among reviewers themselves, most agree on the result being interesting and novel. While there are still some questions on technicality, in particular whether some of the conjectures hold true, in view of encouraging novelty and new insights, the meta-review would recommend acceptance of the paper as a poster to the conference. 
Decision: Accept (Poster)