Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper present an extension of results obtained in the mid 90's by Saad and Sola, who showed that stochastic gradient dynamic in the context of student/teacher learning of a one hidden layer network for gaussian data can be replaced by ordinary differential equations on the weights *when the dimension of the input is very large*.
They extend this work to number of hidden units that scales with the input dimension and to learning rate that can also scale with the dimension.
The authors aim at bridging the Saad and Sola approach with the mean-field limit of a large number of hidden units which is known to overcome the non-convexity of narrow networks for fixed dimension.


strengths_and_weaknesses: Strengths:

- Scaling of learning rate and number of hidden units in terms of the dimension is a real improvement over the Saad and Sola setting.
- the paper is quite well written.
- So far I checked, the proofs are ok.

Weaknesses:

- The proposed study is limited to the realm of gaussian data and the authors believe the result is applicable more generally but no experiments is made to assess these beliefs. 
- It is not completely clear what are the consequences of these results on practical instances of the problem. Since the scalings are formulated with respect to the dimension, it would be interesting for the reader to make more concrete the practical consequences.
questions: - The authors use a scaling of the learning rate $\gamma$ like $\gamma_0/d^\delta$ with a constant $\gamma_0$ and a parameter $\delta$ that can be either positive or negative. A negative parameter $\delta$ seems a bit unusual. It would be interesting to comment on the practical relevance of this case. I understand that it highlights the fact that one can choose a negative $\delta$ when the scaling of hidden units is polynomial in the dimension. I would like the authors to give more insight on the practical relevance of their result in this situation.

- How could the authors design an experiment to assess that their results is not limited to the gaussian setting? This would give more credit to their work.
limitations: Not applicable.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper considers the behavior of online SGD on a two-layer neural network in high-dimension, for different scaling of the number of neurons and step sizes. The authors consider a student-teacher scenario and gaussian data. In this setting, the test error is fully characterized by the covariance matrix of the student and teacher weights. When the time discretization is small enough, and for finite (continuous) time horizon, they show that the SGD dynamics concentrates on a deterministic ODE (with non-asymptotic bounds). The number of neurons and the step size scale polynomially with the dimension, and the authors identify four different regimes, depending on the ratio of the step size over the number of samples: perfect learning (error at the end vanishes when $d \to \infty$), plateau (proportional to the noise level), bad learning (dominated by noise) and no ODE. Plenty of numerical simulations are provided to illustrate their theoretical results.

strengths_and_weaknesses: Understanding the behavior of SGD in neural networks is central to a theory of deep learning. However only a few results exist on the subject. While this paper avoid the issue of global convergence of the ODE, it provides a simple characterization of different regimes based on the label noise and the scaling of the width and step size. I think this is an interesting work, which the NeurIPS community will enjoy, and I recommend to accept the paper. 
questions: I list below a few comments, questions and concerns.

In the introduction, the link between Saas & Solla ODE and the mean-field PDE is mentioned. However, this is not discussed further in the main text. It would be interesting to define the mean-field dynamics in details: in the mean-field regime, the number of neurons p does not need to grow with the dimension d (as written line 146), but only need to be large enough (for a fixed time horizon). A propagation-of-error analysis show that $\delta << p d  / (d + \log (p))$ is enough for SGD to concentrate on the population gradient flow (perfect learning). This deserves further discussion. More generally, the readers might be more familiar with the NTK scaling or abc-parametrization (Greg Yang 2021) and it would be helpful to compare the scaling in this work to those lines of work.

In the evolution equation of the overlap matrix (between line 112 and 113), there is a multiplicative factor $\| x_\nu \|_2^2 /d$ that is missing in the noise part. While it is true that this quantity concentrates on $1$, this should be analyzed (please propagate in the proofs if need be, I don’t expect it to be difficult or change anything to the story). If I understand correctly, the decomposition in learning and noise parts is related (but not equal) to a bias variance decomposition on the label noise (here the two parts have different scalings, hence this decomposition). However, when label noise is 0, the noise term has a fixed point for perfect learning. I am unclear about its interpretation for 0 noise. Maybe some comments about this would be helpful.

More generally, what is the connection between the ODE and the different regimes of SGD?  For classifying the different regimes, only the scaling of the learning and noise parts are compared. What does the ODE perspective provide? (It is difficult to analyze and simulate) Can it be shown that the risk converges to a value proportional to the noise level for parameters $\kappa \neq 0$ ? 

Contrary to the discussion below Theorem 2.1, I do not think Assumption 3 is mild: for example, it cannot be true if $p > d$.  

Even though this is not the focus of the paper, is there any sense when global convergence occurs in finite time horizon? Should the random start have non-vanishing gradient in the direction of the teacher?

Line 206: *leads
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work draws a phase diagram of SGD for two-layer neural networks in high-dimensional settings by connecting the mean-field regime with an approach by [Saad & Solla]. Specifically, this diagram categorizes learning dynamics depending on the order of network width and learning rate into four types: perfect learning, plateau, bad learning, and no ODEs. 
strengths_and_weaknesses: **Strengths** 

It is well known that the choice of hyperparameters (learning rate, network width, initialization scale, etc.) significantly affects the generalization performance of neural networks trained by SGD. However, theoretical understanding of this phenomenon has been limited (e.g., theory of mean-field regime and neural tangent kernel.) In this sense, this work is interesting because the authors aim to characterize learning behavior more precisely and derive a phase diagram that matches an empirical observation. 

The proof is a non-trivial extension of the previous work [Goldt et al. (2019)]. In particular, this work proved that a trajectory of population risk achieved by SGD converges to a deterministic ODE and investigated an obtained error by relating a trajectory with the limit of ODE. This approach is potentially important to perfectly understand the learning behavior of two-layer neural networks. 

**Weaknesses**

However, I have a concern about Assumption 3 made in Theorem 2.1 which assumes $\overline{\Omega}(t)$ is strictly positive until a time horizon $\tau$. And it is shown that time $\tau$ also terminates the approximation guarantee Eq. (15). Based on these, we can see that unless Assumption 3 holds with $\tau=\infty$, we cannot relate the learning dynamics of SGD with the limit of ODE of $\overline{\Omega}$. Moreover, I think it is difficult to Assumption 3 holds with a long time horizon because this condition implicitly requires linear independence of features in student and teacher models but basically, $W^\nu$ will degenerate by the convergence to the teacher model.


**Other Comments**
The following paper is closely related to this work. [Yang and Hu (2021)] also categorizes hyperparameter settings of the learning rate, network width, and initialization scale into a learning regime that exhibits feature learning property, and derives preferable settings. 

[Yang and Hu (2021)] Greg Yang and Edward J. Hu. Feature Learning in Infinite-Width Neural Networks. ICML, 2021.
questions: My major concern is the third condition in Theorem 2.1. To rigorously justify the claim of the paper, this assumption should hold with a long time horizon, but it seems difficult. See the comment in [Strengths and weaknesses] for detail. 
If my understanding is wrong, I would appreciate it if you could correct me.

limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the behavior of two-layer neural networks trained by online SGD in the high-dimension limit. In a teacher-student setting with Gaussian training data, an ODE system is shown to be the high-dimension limit of the dynamics of an "overlap matrix" that fully governs the behavior of the population risk. With the limit ODE system, different cases regarding the scaling of learning rate and network width with respect to the dimension is studied, and a phase diagram is plotted with the two scalings. The phase diagram consists of four regions---perfect learning, plateau, bad learning, and no ODE. The first three regions are determined by the relative scale of the noise and the learning terms in the dynamics. Although no explicit characterization is made for the dynamics or the limit of the loss, numerical experiments show that the three regions in the phase diagram indeed lead to different behaviors. 
strengths_and_weaknesses: Originality: The ideas of studying high-dimensional limit of neural network training and using "overlap matrix" to characterize the behavior come from previous works. However, analysis in previous works is significantly improved to give the results in the current paper. Studying the impact of the two scaling factors and plotting the phase diagram are also original to this work. These results can deepen our understanding to the training of two-layer neural networks. 

Quality and Clarity: The paper is very well written. All the mathematics and discussions are clear and easy to read.

Significance: The results are theoretically significant because they captures the behavior of the training of neural networks in an extreme case---the high dimensional case. The training process of neural networks is very hard to study, due to its nonlinear nature. Therefore, understanding an extreme case that might not be realistic is still helpful and important. 
questions: 1. In the introduction the authors mentioned to connect the finite dimensional and infinite width case (the mean-field regime) with the infinite dimensional and finite width case. Can the authors make more discussion on the implication of theoretical results in the paper on the mean-field regime? It seems all the analysis in the paper treat the infinite-dimensional case, while in the mean-field analysis the dimension is fixed to be finite. 

2. The neural network models studied in the paper do not have output parameters after the activation function, which is quit different from a usual two-layer neural networks. Does the theory apply to two-layer neural networks with parameters in both layers?
limitations: The authors have honestly discussed the limitations of the work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors rigorously analyze the deterministic dynamical descriptions of one-pass SGD of a two-layer neural network with the interplay between the learning rate $\gamma\propto d^{-\delta}$, the time scale, and the number of hidden units $p\propto d^\kappa$ on a $d$-dimensional Gaussian data as $d\rightarrow \infty$.
strengths_and_weaknesses: ### Strength
- very rigorous approach
- much more generalized settings compared to previous works (e.g., Saad & Solla 1995).

### Weaknesses
- The only weakness I found is the setting the paper considered. 
I'm not sure how the analysis on (i) the specific setting in the paper can help understand the dynamical properties of (ii) a common neural network training.
It would be better for a broad range of community if the authors could provide further implications between the two settings:
- (i) convex, two-layer, global convergence, with teacher-generated labels, limit of high-dimensional data $d\rightarrow \infty$, one-pass, Gaussian data
- (ii) non-convex, multi-layer, no global convergence, without teacher, finite high-dimensional data $d<\infty$, (not one-pass?), general non-Gaussian data

questions: - L17: Why is it a convex problem?
- What does it exactly mean by "the one-pass SGD"?
- L202: What does it mean by "a convergence trade-off"?

limitations: Please provide further limitations when introducing the specific setting considered in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The manuscript proves an ODE-limit (i.e. fluid limit) for the training trajectories of Soft Committee machines in the Teacher/Student setting and when the dimension of the inputs $d \to \infty$. Interestingly, the paper considers the regime when the number of neurons $p \geq 1$ in the hidden layer and the learning rate $\gamma>0$ can scale quite freely with the dimension $d$ of the inputs.
strengths_and_weaknesses: The paper is well written, with a reasonable literature review and a good balance between rigor and intuition. Furthermore, to the best of my knowledge (although I am *not* an expert in this area), the proven results are new. These results offer a comprehensive understanding of the generalization properties of these soft committee machines in the online regime (i.e. infinite training data) and as $d \to \infty$.

As weaknesses, I have only a few relatively minor remarks that are in next section.
questions: 1. The function $\psi$ that is defined as the expected value of the RHS of Eqs. (12) is, I believe, depending on $(p,d,\delta t)$, although I think I understand that there is a well-defined limit under appropriate scaling of $(p,d,\delta t)$. If that is the case, at the cost of slightly more cumbersome notations, it would be more appropriate to reflect this dependency.

2. There is quite a large literature on fluid-limit / ODE limits for analyzing randomized algorithms / Markov chain. Standard references are the book by Ethier & Kurtz, as well as the article "Differential equation approximations for Markov chains" by Darling & Norris, and many other places. I think that it would be worth to slightly expand the literature review on these fluid limits.

3. Once Equation (12) is obtained, at an heuristic level at least, it is very clear what the scaling of $(p,d,\delta t)$ needs to be in order to observe an ODE limit. Indeed, as soon as  $\delta t \to 0$ with $E( \text{(markov chain increments)} ) \asymp \delta t$ and $E( \text{(markov chain increments)}^2 ) \ll \delta t$ one expects a ODE limit when trajectories are appropriately time-rescaled: this already heuristically gives more or less all the results discussed after the statement of the main theorem. Indeed, the proven theorem is very interesting and *much* stronger, but I feel it would be worth mentioning that the conclusions are, in some sense, not entirely surprising once we have Eq. (12).

4. The important discussion of the "learning" and "noise" terms in Eq. (16) [i.e. very bottom of page 5] could be expanded further since it is quite crucial to the existence of plateaus / etc... For example, why is this the case in general that (1) if the noise term vanishes  perfect learning is achievable (2) if the noise term remains as in the $\kappa+\delta = 0$ then perfect learning is not.

5. All the weights of the "second layer" are fixed (and equal): it was not clear to me if it was a technical detail to improve readability (which is entirely fine!), and if the proof/approach could be relatively easily extended to the more general case [eg. as in article (6)]?

**[responses by the authors acknowledged (4/8/2022)]**
Thank you for these clarifications.


limitations: NA.
review_title: null

Meta-review: metareview: The reviewers and I agree that the contributions of the paper are of interest and useful addition to the literature. Therefore, I recommend accepting the paper.

Please consider the reviewers' comments when preparing the camera-ready version.

confidence: Certain
Decision: Accept