Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper builds upon the work of Luo et al., 2021 and characterizes the phase diagram for different dynamical regimes in the case of a three-layer neural network at infinite width. Based on appropriately chosen quantities, the paper shows that different layers can be simultaneously in the linear, critical or condensed regime. The proposed phase diagram is backed with several experiments that depict the nature of the learned function and dynamics in these regimes.  
strengths_and_weaknesses: ## Strengths
- I personally find this line of work of characterizing the phase diagram to be quite insightful. It provides a useful perspective to think about neural networks. 

## Weaknesses: 
- The phase diagram partitions the space into four possibilities based on the choice of regime for each layer. However,  more or less all the experiments and results are in the setting when the second hidden layer is in the linear regime. As a result, most of their findings are qualitatively very similar to that observed in Luo et al., 2021. Given their richer characterization of the phase diagram as compared to prior work, the authors nevertheless do not discuss these potential implications: e.g., it would have been understanding to compare what happens in the condensed regime for both hidden layers. As well as, how does condensed regime in a deeper layer while being linear in the lower layer compare to the converse scenario? Further, quantitatively exploring the advantages/disadvantages of being in these different regimes on generalization/optimization could have been worthwhile too. So, I think there are several missed opportunities here and the paper is not properly fleshed out, and overall lacks in contribution.
- On this note, this work is heavily inspired and based on the Luo et al., 2021 -- which in itself is not wrong, but their manner of presentation is misleading. For example, the quantities gamma_3, gamma_2, kappa_3 are essentially identical to the gamma, gamma^\prime, kappa in Luo et al 2021 (gamma_3 is just 1/2 more than gamma since here we have one additional layer). Likewise most of the considered experiments are modeled based on this prior work, and even most of the presented results/figures are quite similar. Again to emphasize, it is absolutely alright to build up on results and experiments from past work, but there should be appropriate acknowledgement --- which are missing here. To give another example, the authors write in section 5.3.1 of regime identification and separation, that "we define the relative change of weights... " but it is the same as Luo et al 2021 and even the results in the figure below are basically the same as in that work. 

### Minor Remarks:
- For Xavier initialization, the values of the gamma_2 and gamma_3 should be interchanged. Also, in the same row for Xavier init, the the value of kappa_3 seems wrong. 
- Writing is a bit sloppy and redundant at times: some lines are repeated multiple times, in the abstract, at the beginning of introduction, and towards the end of introduction. And often there are many grammatical mistakes, so it would be a good idea to fix them as well.
- The figures do not come out well in print.

questions: *Firstly, check out the first point under weaknesses. I would be very interested in looking at additional results related to the mentioned questions therein.*

Besides, there are also some other questions and suggestions that I have related to the experiments:

- Figure 3: Can you make the plot for the similar setting as in Figure 2? It is better to have results in comparable settings. Also, do you see more condensation in the initial layers or towards the output? 

- How do the results in Figure 2 and 3 with increasing number of samples? Do we see that some initialization regimes coalesce when number of samples is large?

- Can you share a similar plot as in Figure 4 for RD(W^[2])?
limitations: NA.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: 
Building on Luo et al. 2021, the current submission presents a detailed numerical study of gradient flow of feedforward neural networks with different scaling parameters used for their initialization.

The experiments reveal that different layers of the three-layer architecture exhibit  changes to their parameters that are:

1. negligible, aka *linear* regime

2. constant wrt. to network size aka *critical* regime

3. increasing with network size, aka *condensed* regime.

Analogously to Luo et al. 2021, the above regimes are revealed to be codimension two bifurcations, controlled by quantities $\gamma_2, \, \gamma_3$ defined in Eq. 9.

By carefully varying $\gamma_2, \, \gamma_3$ for each hidden layer, the authors demonstrate that the weights can evolve separately and in tandem in either of the above three regimes.
strengths_and_weaknesses: To the best of my knowledge, this submission gives new insight into training dynamics of MLPs with more than one hidden layer.
The numerical nature of the study is well complemented by its clever analysis of the bifurcation parameters.
This in turn allows for an elegant and insightful analysis of what I believe is considered a dauntingly difficult problem.

I have not kept up to date with the most recent analyses of the gradient dynamics of shallow NNs, and hence I will defer to other reviewers for estimates of the impact of this work.
Notwithstanding that, I greatly value the elegance and insights that this submission presents.

Minor comments:

1.  The manuscript would benefit from proofreading, especially in regards to small grammatical and stylistic errors.

2. Figure 3: Perhaps a diverging colormap would be better suited for $D(u,v)$?

3. Table 2 and Figure 4: I would like to encourage the authors to report some measure of spread for the experimentally collected quantities.

4. Figures 4 & 5: please label the axis with $\gamma_2, \gamma_3$. I believe the captions indicating the ordinate and abscissa are inconsistent with numerical ranges explored elsewhere, e.g. in figure 6.

5. Figure 3 and 8: mention or indicate graphical the expectation for $D(u,v)$ and $\zeta$ under a Gaussian distribution over the weights
questions: Use of cosine similarity:
Am I correct to assume that the cosine similarity is specific to ReLU is positive homogeneous?
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work studies empirically the effect of initialization scalings on the dynamics of three layer neural networks. The main result is a phase diagram exhibiting quite clearly the presence of different dynamical regimes for the two different layers (condensed and linear).
strengths_and_weaknesses: ## Strengths

- Despite a few typos the paper is well presented and easy to read.
- The phase diagram of figure 1 is neat and a valuable contribution in my opinion. It is well illustrated by the empirical figures of Section 5.
- The empirical investigation is thorough.

## Weaknesses

- The most important part of the paper, where the parameters gamma2 and gamma3 are derived, is rushed and difficult to understand. The authors should clarify their choice as it looks a bit like dark magic right now
- Some important works on phase diagrams are missing in related works, for example the work on disentangling feature and lazy regime by Mario Geiger and colleagues and the series of papers on tensor programs and muparametrization by Greg Yang and colleagues.
questions: - Is the phase diagram of Fig. 1 plotted based on theoretical or empirical lines ? This should be made more clear.
- « For instance, if we want to study the nonlinear effect of a hidden layer, we can set this interested hidden layer in the non-linear regime and all other hidden layers in the linear regime » what do the authors mean by « study the nonlinear effect » ? This is vague and does not match the description of « guidance for hyper parameter tuning » mentioned in the sentence before 
- «  the dynamics of a is not the main concern » : could the authors provide more justification on this ?
- I do not understand the following sentence (« which leads to the initialization parameters... »), and the parameter B in beta2 = B beta3 does not seem to be defined or used anywhere
- Figures 5 and 6 : the caption mentions that the width m is varied across 5 different values but I am confused about how that reflects in the figures : are the results averaged over the different values of m ?
- It would be nice to have a discussion on the impact of the nonlinearity. For example including a phase diagram for Tanh ?
limitations: Yes
review_title: null

Meta-review: metareview: This work studies the effect of initialization scaling on the gradient flow training dynamics of three layer (two hidden layer) MLPs. . Extending Luo et al. 2021's analysis, the authors identify "linear", "critical" and "condensed" regime depending on initialization scaling parameters

Strength pointed out by the authors include, "new insight into training dynamics of MLPS with more than one hidden layer", "elegant and insightful analysis" for a "dauntingly difficult problem" and that "phase diagram is neat and valuable contribution"

Reviewer `PDY9` saw major weakness as treating the second layer as linear regime only, which makes the analysis similar to previous work (Luo et al., 2021). In general, the reviewer believes that the full comprehensive analysis on different phases for each layer is lacking. Although the reviewer did not respond to the AC, this issue has been addressed by author response. Also please do follow the recommendation of the reviewers in terms of improving writing (better description of gamma_2, gamma_3 for example), improving related works, as well as making figures more legible. 

Overall, while the paper is somewhat borderline, there are interesting insights and analysis as the reviewers pointed out without critical issues. I recommend accepting this work at NeurIPS 2022. 

confidence: Less certain
Decision: Accept