Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper studies the random covariance matrix of infinite depth and width limit of initialized deep neural networks. Using recent work on deep kernel shaping, the authors show that shaping activation function leads to non-trivial stochastic differential equations (SDE) of neural covariance matrix. Authors show that simulation of finite networks match the distribution predicted by the SDE. 
strengths_and_weaknesses: The paper presents an interesting theoretical framework for understanding simultaneously the large depth + width limit of random neural networks at initialization. Much of conventional large-width limit has focused on fixed depth and large width limit but there is evidence as presented by the authors as well that if one simultaneously takes large width and large depth limit, the behavior of neural networks can be systematically different. 

Interesting contribution of this paper is that even in this seemingly complicated limit, there's tractable theory determining the random covariance matrix in terms of stochastic differential equations. This is allowed by utilizing activation shaping which was used to show that one could train normalization-layer and skip-connection free very deep feed-forward networks.

Paper does a very thorough job of covering related works, with the only exception I see being [Yaida, Non-Gaussian processes and neural networks at finite widths, MSML 2020] mentioned in the Questions section. 

Exposition is clear. The exposition split of starting from unshaped ReLU networks in Section 2 with SDE example in section 2.2 then leading to shaped activation case of Section 3 was quite helpful.  However, to understand the full details of the work, it does require readers to have some prior exposure to modern probability and stochastic processes. Appendix A attempts to provide some background but this serves more of a refresher than an introduction. This may limit the target audience. Having said that, probably simple exposure to stochastic processes and stochastic differential equations may be sufficient to understand the main paper's message. 

Authors did a great job of providing, ipython notebook of the empirical simulations. Empirical results do show solid support for correctness and generality of derived results. One thing to be cautious about is to consider removing unnecessary outputs in the notebook, since there is some identifying information left by mistake. This doesn't seem to be a severe violation of anonymity but needs to be careful for future submissions. 
questions: One limitation I see from the current approach is how to use the current framework to do **inference** In the conventional infinite-width limit, due to deterministic covariance matrix inference leads to conventional Gaussian process inference. With non-deterministic covariances, does the author have ways to perform posterior inference to make predictions? Interesting related work is [Yaida, Non-Gaussian processes and neural networks at finite widths, MSML 2020] (also covered in 6.4 of ref [30] Roberts, Yaida and Hanin 2022) which looks at finite width correction to NNGP and obtain non Gaussian process as correction to the infinite-width NNGP but still retains ways to perform inference as a correction to NNGP inference. 

Also for the first attempt of a new theoretical approach, simplicity is a virtue and I appreciate it. However, it would be beneficial for the readers to see applicability / limitation of neural covariance SDE to general architecture. For example, DKS[37] or TAT[38] both main practical applications are to Convolutional Networks, setup in Section 2 deals just with fully connected networks. Could authors expand more on generalization to more structured architecture? 

What is the practical benefit of having SDEs vs just Markov Chains (as in section 2.3)? I understand there's a nice "trainability" benefit coming from activation shaping and SDE maybe coming along with the choice. But my read from Figure 1 in terms of predictability of the theory of neural covariances, both are as good matching the simulations.

In the setup, neural networks weights are initialized with 0-mean variance 1 Gaussian without any bias terms. Often exploding/vanishing of input norms are phrased in terms of initialization variances of weights and biases(e.g. Schoenholz et al., Deep Information Propagation, ICLR 2017 or [34]) and denoted Chaotic/Critical/Ordered phase. Wonder if the condition in section 4, eq (24) is directly related to the phases if accounted for weight/bias variances. Or should I consider if and only if condition in (24) as totally different conditions due to simultaneous width-depth limit?

Simple Nit: L200 should be either PDE or derivative on $\rho_t$ should be full derivative? Maybe the point is to say it's not a stochastic DEs? From the first read, it was slightly confusing. 

limitations: Mostly a theoretical analysis of deep neural networks, so there's no foreseen negative societal impact. 

Few limitations as raised in the Questions section: (not necessarily needs resolving but would be good if mentioned as future directions etc)
- Methods to perform inferences
- Generalization beyond fully connected networks
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Infinite-depth-and-width limits for the covariances of activation layers are derived for neural networks with ReLU-like and smooth activation functions at initialization (under the iid Gaussian weights assumption). To obtain a nontrivial limit, leaky ReLU-like activations are assumed with the slopes differing by a factor decreasing as $O(n^{-1/2})$. This factor is shown to be critical for this limit. For smooth activation functions, a criterion is obtained on the activation function to ensure that the limiting covariance SDE is nonexplosive. Sigmoidal, tanh, and some classes of softplus activation functions are determined to satisfy this criterion. Some experiments are conducted which show agreement between the distribution of correlations for initialized finite NNs and those of the SDE predictions.
strengths_and_weaknesses: NNGPs and NTKs are among the most useful theoretical tools for analyzing neural networks, and provide a powerful link between the versatile Gaussian process paradigm and deep learning. However, it has been shown time and again that these infinite-width approximations do not mimic the performance of modern neural networks. Here, the authors have instead considered the infinite-depth-and-width regime, and while analogues of NNGPs and NTKs are not yet available, the proposed building blocks are provided here for neural networks at initialization. Comfortingly, the behaviour of these approximations seem to mimic those of finite NNs --- as in double descent theory, the large n,d regime appears to be the correct one. The theoretical analysis is extensive, and the presentation is (mostly) excellent, although I did not have the opportunity to inspect the proofs in detail. 

With this in mind, and given that this work is novel to my knowledge, I believe this is a valuable contribution and am excited by potential follow-up work here. However, on first read, I was having a difficult time drawing conclusions from these particular results. In particular, I believe the numerical results and their details deserve their own section in the main document. To make room for this, I would suggest shrinking Section 2.3 (I found this section to be more confusing than enlightening, although I understand why it is here), and moving the critical exponent results to the Supplementary Material, as I believe they can be easily summarized in a sentence or two. It would also have been nice to see what these covariances could do as part of a GP, but I would not expect any significant improvements over known kernels. 

Some minor points:
- The first paragraph of the abstract is confusing as written; I would strongly suggest rewriting it, starting with 'Recent work...' in the first sentence
- line 72: 'according a' -> 'according to a'
- line 90: 'chains' should probably be capitalized to be consistent
- line 103: 'of, the' -> 'of the'
- line 106: 'Gaussian random matrix' - Define. There should also be explicit mention of the Gaussian weights assumption here.
- line 108: Consider reversing the order of these two equations, as the significance of g is unclear when reading from left-to-right.
- line 119-121: last sentence is unnecessary
- line 161: replace colon with full stop
- line 162-163: last sentence is repetitive; consider removing
- Can different symbols be used to describe the Markov chain and the limiting SDE (e.g. \mathcal{V})? The results are difficult to parse as is.
- line 187: 'they are' -> 'they follow'
- line 232: I'm not sure the exclamation mark is warranted here...
- line 274: Proposition 7 should probably be moved here to avoid confusion
questions: How does the kernel derived from these neural covariance SDEs behave in a GP relative to e.g. the arc-cosine kernel?

Does the typical ReLU setup $s_+ = 1$, $s_- = 0$ not reveal an interesting limiting SDE? This seems like a surprising omission, given that almost any other activation function seems to exhibit nontrivial limiting behaviour. 
limitations: While the authors do not discuss limitations of their own work, I find this to be reasonable, as there are few apparent limitations here. Perhaps the most significant is the focus on initialized networks (requiring all weights to be iid normal), which has surprisingly little discussion in the main document.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper study the random covariance matrix in the shaped infinite-depth-and-width limit to model the layer to layer fluctuations in random feature neural networks.

The authors claim that the modeling of covariance and depth has a significant improvement over the NNGP regime.
strengths_and_weaknesses: Strength

The idea of depth and covariance modeling is intuitive and non-trivial. This paper analyzes infinite-depth-and-width limit by reshaping the activation function, resulting non-trivial SDE structure.

Weakness

Although the “reshaping” idea is theoretically elegant, the practical interpretation is still based on the linearization regime, which is known to underestimate the neural networks.

Besides, the gain from “reshaping” is not significant enough to replace NNGP/NTK (at this moment).
The theoretical modeling is based on the evolution of random features with the depth. However, deep random features are not better than two-layer one in real practice. More interpretation might be needed to justify the significance.

The authors raise some examples such as the Exploding and Vanishing Norms. However, it is an artificial setting which is not even a problem in real practice. A more practical example indicated by this framework is appreciated. 
questions: Is there any further interpretation when incorporating this technique to neural tangent kernel?

limitations: The analysis is still in the linearization regime, which is known underestimate the power of neural networks. The evidence is not strong enough to justify the necessity of “reshaping” in NN literature at this moment.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is theoretical and studies the behavior of neural networks in the shaped infinite depth/width limit. As stated in the appendix, the logits at the end of a feedforward network are conditionally Gaussian at initialization with a random covariance matrix. This work studies that matrix at infinite depth/width.

Current theory does not work at large depths because small fluctuations between layers are disregarded. When properly accounted for, this paper demonstrates that the fluctuations add up so that the covariance matrix is governed by a stochastic differential equation (SDE). This work therefore provides a stronger theoretical understanding about networks at initialization. 

The distribution of the covariance matrix closely follows the distribution predicted using the SDE theory developed here, this is shown by simulating the initialization of many neural networks.
strengths_and_weaknesses: Strengths:

The paper provides a creative theoretical contribution. Furthering our understanding of large neural networks, from a novel angle using SDEs. The work is also tested briefly, leading to positive results like those in Figure 1.

Weaknesses:

The only weakness I can find is in the presentation, which is mainly down to personal preference. In my opinion the paper feels very math heavy, and possibly not as accessible as it could be. The paper may benefit from writing more intuitive statements in the main paper and what their consequences are, essentially walking through the work. Then the more rigorous proofs and statements can be placed in the appendix. For example, equation 2 for the ReLU-like activations is currently:

$\phi(x) = s_{+}\max(x, 0) + s_{-}\min(x, 0) = s_{+}\text{ReLU}(x) - s_{-}\text{ReLU}(-x)$

These are both technically correct, but an easier way of writing may be:

$\phi(x) = s_{+}x ~~ \text{if} ~~ x\geq 0,
~~ \text{or} ~~ 
s_{-}x ~~ \text{if} ~~ x< 0$

Additionally more figures/tables showing visually how well the Covariance SDE works always help.

Minor Typos (do not affect the quality of the paper):

- Line 60: "matches matches" should be "matches"
- Line 246: "Not only do these stopping time rules out" should be "Not only do these stopping times rule out" or "Not only does this stopping time rule out"
questions: - Is there a way this work can be used outside of our understanding? For example to find a valuable initialization scheme?
- Is it possible to extend this to understand training dynamics?
limitations: The societal impact is not addressed. However, this is a theoretical paper and a broader impact statement is not required (though it is always appreciated). 

The limitations are discussed in the introduction. Briefly, the paper only applies to initialization and not training dynamics.
review_title: null

Meta-review: metareview: There is a clear consensus to accept this manuscript.  The results are impressive, and have a nice theoretical orientation that will allow the results to have continued impact as the field advances.  There are some minor errors by the authors in the discussions, which are worth the authors being aware of before submitting their final version.  In particular, they state that:

"The authors of [2,3] considered a bounded activation function in the infinite-width limit with large depth, in which case their variance 
V
ℓ
α
α
 always converged to a finite fixed point when depth is large [2, eq. 3]. Their chaotic and ordered phases are then defined by the behaviour of the correlation fixed point [2, eq. 5], which in turn determines the behaviour of the gradient [2, eq. 16].

In our case, shaping the activation leads to an unbounded function, and consequently the variance 
V
ℓ
α
α
 is not always bounded - even if we take the same limit as [2] (but shaping depends on depth instead like the DKS/TAT papers), in which case we get an ODE with finite time explosion. Intuitively, if we drop the Brownian motion from eq. 18 and consider

d
X
t
=
b
X
t
(
X
t
−
1
)
,
d
t
,
,

which is the logistic ODE, and has a finite time explosion if 
X
0
>
1
 and 
b
>
0
. At the same time, due to shaping, our correlation 
ρ
t
α
β
 will actually be able to avoid the fixed point (i.e., non-degenerate). So the gradient will be well behaved from the perspective of correlations (we are in the critical regime defined by [2]), but it may still explode due to variances exploding.

References
Novak, R., Xiao, L., Lee, J., Bahri, Y., Yang, G., Hron, J., Abolafia, D.A., Pennington, J. and Sohl-Dickstein, J., 2018. Bayesian deep convolutional networks with many channels are gaussian processes. arXiv preprint arXiv:1810.05148. https://arxiv.org/pdf/1810.05148.pdf
Schoenholz, S.S., Gilmer, J., Ganguli, S. and Sohl-Dickstein, J., 2016. Deep information propagation. arXiv preprint arXiv:1611.01232. https://arxiv.org/pdf/1611.01232.pdf
Yang, G. and Schoenholz, S., 2017. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30. https://arxiv.org/pdf/1712.08969.pdf
"

And while [2] states they consider bounded activations, it is not used or necessary and is not used in [3] or subsequent more recent work that discusses the edge of chaos further; see for instance: Activation function design for deep networks: linearity and effective initialisation by Murray et al. and On the impact of the activation function on deep neural networks training by Hayou et al.  

confidence: Certain
Decision: Accept