Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the training of a multi-layer ResNet using mean-field tools, where the training procedure of the ResNet is given in its continuous limit as a partial differential equation. A rigorous proof is given, and the convergence to a global minimum of the cost functional is given.
main_review: 
The proof of this paper seems reasonable and the assumptions and theorems are well-presented. It contributes to an interesting and important research direction, which is to study the convergence of neural ODE/PDEs. However, I have several concerns about this paper.

1. The contributions of this paper are kind of incremental given existing works [1,2]. The main theorems in these papers and this submission are quite similar. In fact, the contributions given in Section 3 are almost identical to the contributions listed in [2], which is concerning. Below Theorem 3.1, it is only discussed how it is nontrivial to extend existing results in the mean-field regime to multilayer networks, but a detailed explanation of how this paper (in terms of the convergence results) differs from [1,2] is missing. 

[1] Lu et. al., A mean field analysis of deep ResNet and beyond: Towards provably optimization via overparameterization from depth. ICML 2020.

[2] Ding et. al., Overparameterization of deep ResNet: zero loss and mean-field analysis. arXiv preprint arXiv:2105.14417.

2. It is also not very clear what is the advantage of this paper compared with prior works on mean-field or NTK. For mean-field, while the mean-field works on two-layer networks are not applicable to multi-layer networks, this paper does not seem to be applicable to any practical multi-layer network architectures either, as this paper requires the network depth to go to infinity. So the authors may need to comment on why studying this infinite limit is interesting. Besides, in (1), is it standard to consider a scaling parameter $1/L$ in each layer? In practice, the residual block is directly added without using this scaling. 

3. My next concern is regarding the comparison to the NTK results. In particular, [3] shows that for two-layer NN, when considering updating the “distribution” of model parameters, the obtained solution still exhibits a “kernel-like” behavior. So I would like the authors to discuss more regarding the advantage of the developed results compared to the kernel models, rather than simply mention “NTK views DNN as a kernel model, a rather limited description, so the estimates obtained through NTK can be far from sharp”. One way to show this is to provide some examples to demonstrate the gap between these two regimes.

[3] Chen et. al.,. A generalized neural tangent kernel analysis for two-layer neural networks. NeurIPS 2020.

4. Following my previous comment, the discussion on the advantages of ResNet should also be included, since I believe the main goal of studying ResNet is because of its superior performance compared to fully-connected NN. In the NTK setting, [4] has shown that adding residual structure can improve the dependency on NN depth compared to FC-NN. Then one question is what can we gain by studying the convergence of ResNet in the mean-field regime?

[4] Frei et. al., Algorithm-dependent generalization bounds for overparameterized deep residual networks. NeurIPS 2019.

5. My last question is whether the mean-field analysis can be adapted to classification problems? It has been widely known this can be done in the NTK regime [5, 6] since the convergence analysis in the NTK regime does not require the loss function to be MSE loss but can be applied to any convex loss function. The authors may also need to discuss this point in the mean-field regime.

[5] Li and Liang, Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data. NeurIPS 2018

[6] Zou et. al., Gradient descent optimizes over-parameterized deep ReLU networks. Machine Learning 2019.


Minor comments

The notations in Section 2, Assumption 6.2, and Remark 6.2 are not very consistent.



summary_of_the_review: While this paper provides some interesting insights into the training of infinitely deep and wide ResNets, it is not well-demonstrated whether the results are novel and significant enough compared to various existing works. Some detailed discussion and comparison may be very helpful to improve the quality of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work analyzes infinitely deep and infinitely wide residual networks. The infinite depth limit is the neural ODE. The infinite width limit is the mean-field limit. The main contribution establishing global convergence of the infinitely wide and deep ResNet. Finally, the global convergence of the infinite limit is translated to the finite but large setup.
main_review: This work analyzes infinitely deep and infinitely wide residual networks. Overall, I found the writing to be quite good, and these types of results are of interest to the community (at least they interest me). I have not been able to check the details of the proofs, but they seem plausible.

The limit of viewing infinitely deep ResNets as an ODE is due to the Neural ODE paper, and further taking the infinite width limit to consider the mean-field regime was considered in [1]. Therefore, the main contribution of this work is in analyzing the training dynamics of regular gradient flow on this network ([1] considered a different, unusual training mechanism).

However, I would like to get more clarity on the incremental novelty. As the authors acknowledge, the paper is very similar to [2]. Since this paper is meant to be a standalone paper distinct from [2], I would like to get some clarification as to what are the specific contributions not present in [2]. 

The results of section 6 seem to be the global convergence of the continuous limit, is the core contribution. The technical challenges that had to be overcome appear to be quite similar to those considered in [3]. I would like to hear from the authors about the novelty of the proof technique compared to that presented and used in [3].


[1] Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, Lexing Ying. A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth, ICML, 2020.
[2] Zhiyan Ding, Shi Chen, Qin Li, Stephen Wright, Overparameterization of deep ResNet: zero loss and mean-field analysis, arXiv, 2021.
[3] Huy Tuan Pham, Phan-Minh Nguyen, Global Convergence of Three-layer Neural Networks in the Mean Field Regime, ICLR, 2021.

summary_of_the_review: Interesting results, but contribution should be clarified.

I am willing to raise the score if the issue of novelty is clarified.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the gradient flow training dynamics for deep and wide ResNet with the mean-field scaling. For a similar setting as (Lu et al., 2020), they report a limiting model with finite-dimensional approximation, and its global convergence result. The authors also justify the well-posedness of the resulting distributional dynamics, which seems only heuristically defined in prior work. 
main_review: This work reports a new mean-field explanation for an infinitely deep and infinitely wide neural network with skip connection and provides global convergence results with the framework of (Chizat & Bach, 2018) under standard assumptions and the universal kernel property exploited in (Lu et al., 2020). My main comments are as follows:

* In P4, the literature review paper list for mean-field limit perspective is not correct, e.g., (Allen-Zhu et al., 2019), (Zhang et al., 2019) are not in that regime.

* In Assumption 4.1, the activation f is required to be C^2. In P6, just below Assumption 4.2, they claim \sigma can be regularized ReLU, which is seems not introduced. Please explain.

* In Theorem 5.1, the final bound seems not uniform for 0<=s<S, which is a much weaker nonasymptotic result than that in (Mei et al., 2018) for two-layer NN.

* The sequential limit, as shown in Theorem E.1 and E.2, is taking limit for L before increasing M, which is similar to the layer-wise scaling used in (Sirignano & Spiliopoulos, 2018). Thus, the M and L would have to grow up at specific speeds simultaneously. So, for curiosity, can we exchange the order of limits of M and L in an asymptotic version of Theorem 3.1?

summary_of_the_review: Overall, I think this paper has useful ideas for understanding the training dynamics of ResNet in the nonlinear regime, and it complements the existing research in the literature.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proves a global convergence result for GD in resnets, whose residual blocks are mean field two-layer neural nets. This is taken with a double limit, where the depth limit is the neural ODE and the width limit is the two-layer MF limit. To prove global convergence of the infinite-depth/width limit, the paper employs previous techniques that assume Sard-type regularities and the accompanying stability argument from Chizat and Bach 2018. 
main_review: Many thanks to the authors for addressing my concerns and correcting the mistake.

As said in my review, there are reasons to believe why the contribution of this paper is a clearly meaningful one on top of prior works in similar resnet mean field setups, namely the removal of the full support assumption on $\rho_{t=\infty}$. Though again the technicality is not necessarily surprising, this is hard work. My final recommendation would have been 7 if possible; in support of this paper, I'm making 8.

==============================

There is a lot of interests in the mean field analysis for neural networks, so the subject of the paper is timely. Before this work, Lu et al 2020 attempted to analyze a very similar problem setup, but had to make the full support assumption of $\rho_{t=\infty}$ for their global convergence result. The current paper fills in this gap via Theorems 6.1 and 6.2, along with Theorem 5.1 which shows that the wide and deep resnet can be approximated well by the corresponding continuous MF model. 

It should be stressed that this assumption on $\rho_{t=\infty}$ is a very strong one (as pointed out in Pham and Nguyen 2021) and is unrealistic in certain situations. Removal of this assumption tends to be highly technical; one can just look at the length of the proof of this paper, compared to that in Lu et al 2020. As such, the paper has good and meaningful technical contents.

In that sense, my overall evaluation is that the paper gives completion to Lu et al 2020, though I am not surprised by the result nor the techniques. In particular, while I cannot check the lengthy proof details, I do not doubt the results, and the arguments in the presented proof are within expectation. Let me go into further details of the global convergence proof of Theorems 6.1 and 6.2:
- Compared to Lu et al 2020, perhaps the one piece that differs is Proposition 6.1, which recognizes that the condition of global optimum can be checked at any residual block, unlike what was done in Lu et al 2020 which sort of integrates over all blocks. The key is the appearance of $g(Z_\rho) - y$ term in the gradient update of every residual block, and any place where this term appears can be exploited. I must say, though, this observation is not new and has been used elsewhere in the mean field literature.
- Since the problem comes down to any of the residual blocks, which are just two-layer MF neural nets, the entire stability argument from Chizat and Bach 2018 can be replicated. Another piece of technical work is in proving the support property of $\rho(s)$ for any time $s$, and this is again a direct application of the topological degree lemma in Chizat and Bach 2018.

To add to this point, there are results which are proven with similar analyses (with the Sard-type regularity and the stability argument) on setups other than two-layer nets; examples include Theorem 50 in Nguyen and Pham 2020 for deep feedforward nets and Theorem 11 in Agazzi and Lu 2020 for temporal difference dynamics. In other words, this analysis can be carried out (with some technical confidence, admittedly) for the right assumptions.

One major issue is that Theorem 5.1 and Theorem 6.2 cannot be merged to claim Theorem 3.1 the main result. As far as I understand, Theorem 5.1 assumes compactly supported initialization distributions. This compact support assumption makes it possible to prove well-posedness and eases various parts of the proof of Theorem 5.1. However it is important NOT to have initializations with a finite support for the global convergence results (Theorem 6.2) to hold. Due to this mismatch, one cannot claim Theorem 3.1. To be fair, the development in Chizat and Bach 2018 also goes along the same line; the issue can be avoided by either:

- assuming a compact domain for the first layer weight with an appropriate version of GD (which Chizat and Bach 2018 covers, but the current paper does not),

- working directly with projection onto the sphere in the 2-homogenous case (which Chizat and Bach 2018 does, but the current paper does not).

This mismatch in the conditions is technical and should be an interesting mathematical problem to fill in, but does not seem to pose any conceptual issue in machine learning. As such, in light of the established literature, I would not evaluate strongly against the paper, but I ask that the paper remove Theorem 3.1 and make a clear acknowledgment of the mismatch.

Another request is that the paper should also make more efforts in discussing the last Sard-type regularity condition in Theorem 6.2. While the paper claims this is a very mild assumption and refers to Remark 4.1, this remark only concerns with the third Sard-type condition and not the last one. In my opinion, it is this last condition that is extremely difficult to verify (the third one can be satisfied for sufficiently smooth activation functions). This is because it involves dealing with points that are at infinity. Again since this assumption is adopted from Chizat and Bach 2018, I would not evaluate strongly against the paper.

Minor comments:
- $\epsilon$ and $\eta$ are not explained in the first page.
- The equation before (3) seems to be missing a factor of $ML$.
- The paper should clarify what “regularized ReLU” is; this is not a standard term.
- The example activations in Remarks 6.1 and 6.2 are somewhat contrived; can the paper give more familiar examples?

References:

L. Chizat, F. Bach, On the global convergence of gradient descent for over-parameterized models using optimal transport, NeurIPS 2018.

Y. Lu, C. Ma, Y. Lu, J. Lu, L. Ying, A mean field analysis of deep resnet and beyond: Towards provable optimization via overparameterization from depth, ICML 2020.

H. T. Pham, P.-M. Nguyen. Global convergence of three-layer neural networks in the mean field regime, ICLR 2021.

P.-M. Nguyen, H. T. Pham. A rigorous framework for the mean field limit of multilayer neural networks, 2020

A. Agazzi, J. Lu, Temporal-difference learning with nonlinear function approximation: lazy training and mean field regime, 2020.

summary_of_the_review: This is a technical paper that removes a strong assumption from the analysis of Lu et al 2020, though the techniques are not surprising. A number of edits are necessary. Though giving a confirmation to a result that is expected, the paper is still a good theoretical contribution, in light of the heavy technical works that are required.
review_title: null

Meta-review: This paper proposes an improved mean-field analysis for multi-player residual networks. Compared with prior works, the proposed analysis removes a full support assumption needed in prior works. The authors have addressed some of the reviewers’ concerns by adding comparisons with the existing analysis of ResNet in the NTK regime, and a more detailed comparison with Ding et al. 2021. While this paper gathers some support from a reviewer, there is still concern that the novelty of this paper is not significant, especially given that the analysis is heavily built upon prior works. I think this paper can benefit from providing a proof sketch to highlight the key difference between the new analysis and existing analyses,  or explicitly demonstrating the key proof technique/technical lemmas that enable the removal of the full support assumption. This paper might be a strong work after careful revision.
Decision: Reject