Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This article is concerned with convergence guarantees of online stochastic gradient descent for a rather generic class of three layers neural networks (instead of similar analyses that treated two layers). The main results state that in a proper limit of infinite width + vanishing learning rate, the dynamics of online SGD is proven to be tracked thanks a mean-field description in the form of coupled ordinary differential equations. Once this mean-field description at disposal, the main result is obtained: in the infinite width + vanishing learning rate + infinite time (= number of training samples), the generalization error tends to it minimal value for a broad class of models and losses (not necessarily convex, which is a novelty of the work) as well as generic data distribution.

Overall this paper is very well written, enjoyable to read despite the technicality of the results, and understandable even for non-specialists of this line of works (like myself). I did not check the appendices and proofs. In the main part there are no typos, and I have no main concerns to bring about. Yet: I would find interesting to know more details about the differences with the refs Nguyen (2019); Araújo et al. (2019); Sirignano & Spiliopoulos (2019); that is not clear.  Also I would find useful to have some hints about the meaning of the (trained third layer) hypothesis in Theorem 8. Finally I find a bit surprising that there are nor restrictions whatsoever on the data distribution (or I missed that). The authors may comment on that in the final version.

I recommend publication. Even if I'm not a specialist, it is obvious that the authors made a big effort of redaction, that the results are very solid, the proof technique seems original and requires less assumptions than previous works (I liked very much the "idea of proof" part). I have very few doubts about the quality of the paper despite I did not read the proof details, and the fact that I'm not aware of the literature in this specific field.
review_title: ICLR review for "Global Convergence of Three-layer Neural Networks in the Mean Field Regime"

final_score: rating: 7: Good paper, accept
review_content: **Summary**: 
Analysis of neural networks in the mean field regime gains more and more attention as it helps to study the dynamics in the wide regime. The paper extends the recent studies and provides global convergence guarantees for an unregularized feedforward three-layer NN. This is the first time global convergence is established for neural networks of more than two layers in the mean-field regime. 

I find the writing a bit chaotic and overdosed with notations. But, overall, I think the results are significant and will help to extend the line of research in the mean-field regime applied to neural networks. 

**Questions**: 
- In the paper, it is said several times that the convergence result “does not rely critically on convexity”. What do you mean by “critically”? You still assume the convexity. Do you mean that it can be easily relaxed in future works? I think it should be better stated throughout the paper.  

**Minor suggestions**:
- Mean field -> mean-field 
- Section 2.1 “k” is introduced at the very end. Maybe it would be better to add in the beginning, e.g. “the following network at time k”? 
- I would say “W(k) consist of the weights …” instead of “W(k) is the weight with ….” 
- It perturbs me a bit that the difference between NN notations and MF is its boldness. Boldness usually means a vector while in authors’ notations $\mathbf{w}_2$ is an element. But I guess with superscripts or symbols like $\hat w$ it would be heavier in notations... 
- Section 2.2. I would like to see the description of \Omega, F and P  in the beginning. Like “Given a NN \Omega_i would be a space  of…”. Just to connect it from the start and to easify the following read. 
- Definition 2. “The following hold” -> holds 
- Section 4.2 “where V a set…” -> is 
- “Helps avoiding” -> helps to avoid

review_title: Good results 

final_score: rating: 7: Good paper, accept
review_content: This paper studies the behavior of a 3-layer fully connected network when the width of the network is large. The authors define a mean field regime and prove that the behavior of the network under stochastic gradient descent converges to this mean field regime (for any finite time horizon).  This result complements nicely previous works like (Nguyen 2019) that contain an informal derivation of the mean field regime. This transient regime is complemented by a long-term analysis under quite restrictive assumptions (which imply essentially than the mean field regime always converge to the minimizer of the loss function).

I did not check all details of the proof but the approach seems mathematically sound. Once the model is defined, the proof for the finite regime is relatively classical: it relies on Martingale concentration plus Gronwall's lemma. Yet, as always, the devil being in the details and defining the right model and using the right notations is a difficult task. The result of the stationnary regime seems also reasonable but I must admit that the proof of the infinite horizon is not really clear to me. I would have appreciated more pedagogical effort from the authors.

To summarize, the paper seems a nice theoretical contribution. Yet, to me one thing that this paper is missing is an explanation or illustration of how useful is their result to understand the behavior of deep neural networks.

That being said, one major concern that I have about the paper is the link with https://arxiv.org/pdf/2001.11443.pdf The arXiv paper considers a very similar model (but more general as it considers L layers instead of 3). It uses almost the exact same notations and the same structure (overall paper and proofs). I think that the authors should clarify the link between the two papers. Also, if I can admit that the present paper is a resubmission of the arXiv paper, I do not understand why does the current paper focus on 3 layers and not the more general model of the arXiv paper.

review_title: Nice theoretical result. Relation to previous work to be precised

final_score: rating: 7: Good paper, accept
review_content: This paper studies some theoretical properties of three-layer neural networks (NNs) under the mean-field (MF) regime. The authors proposed neuronal embedding in order to study large-width neural networks. Then, the quantitative relation between finite-width NN and the MF limit was clarified. The global convergence of the continuous limit of the stochastic gradient descent (SGD) was proved without assuming the convexity of the loss function. The global convergence of the MF limit is used to establish the optimization efficiency of the neural network with SGD. 

The problem considered in this paper is important. The definitions and the statement of theorems are clearly described. In order to prove the global convergence, the authors assumed the uniform approximation property rather than the convexity of the loss function. This approach is interesting. Though I'm not very familiar with the MF regime, the high-level idea to prove the theorem is well-written. I read some proofs in the appendix, and I found that the description is accessible to a wide range of audiences. Overall, this paper is thought to provide a promising idea to analyze not only three-layer NNs but deep NNs.

Some comments are shown below:
- Neuronal ensemble is introduced to analyze the dynamics of the MF limit. Apart from the MF regime, is there any similar idea of the neuronal ensemble? Showing some references would be beneficial to readers.
- The constant K appears in the upper bound in Theorem 3. Showing a more concrete expression of K is informative. What is the typical K in this case? 
- In Theorem 8 and Corollary 10, the global convergence was proved. If the convexity was also assumed in addition to the assumptions in the theorems, is it possible to derive the convergence rate? 
- In the paper, the global convergence of three-layer NNs was analyzed. What is the main obstacle to investigating the global convergence of multi-layer NNs or deep NNs according to the idea proposed in this paper?

review_title: Overall, this paper is thought to provide a promising idea to analyze not only three-layer NNs but deep NNs.

Meta-review: This paper provides a global convergence guarantee for feedforward three-layer networks trained with SGD in the MF regime. By introducing the novel concept of neuronal embedding of a random initialization procedure, SGD trajectories of large-width networks  are shown to be well approximated by the MF limit, a continuous-time infinite-width limit (Theorem 3). Furthermore, under some additional assumptions the MF limit is shown to converge to the global optimum when the loss is convex (Theorem 8, case 1) and for a generic loss when $y=y(x)$ is a deterministic function of input $x$ (Theorem 8, case 2). The global convergence guarantee presented in this paper is based on less restrictive assumptions compared with existing studies. All the reviewers rated this paper quite positively, with less confidence however, seemingly because of mathematical thickness of the proofs. Although the reviewers did not manage to check every detail of the proofs, they agreed that the reasoning seems mathematically sound as far as they can tell. The authors response adequately addressed minor concerns raised by the reviewers. I am thus glad to recommend acceptance of this paper.

Pros:
- Introduces the idea of a neuronal embedding, which allows establishing relation between SGD on large-width three-layer networks and its MF limit in a quantitative way with a less restrictive setting.
- Provides a global convergence guarantee under the iid initialization, in the sense that if the MF limit converges it attains the global optimum.
- Shows that the global convergence guarantee does not require convexity of the loss when a deterministic function is to be learned.

In particular, the uniform approximation property, rather than the convexity of the loss, plays a crucial role in proving the  global convergence guarantee (it allows translation of the vanishing gradient in expectation at convergence into the almost-sure vanishing gradient), which is a quite original contribution of this paper.
Decision: Accept (Oral)