Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: null
review_content: The reviewer will respect any decision by the AC, but hope this paper will not get rejected only because the careless review by u7jP. One point I want to raise is that the review u7jP chose "3" in the confidence part, while he/she obvious did not read the paper at all. This shows unprofessional and hence this review should be disregarded.
review_title: About the "violation rules"

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the convergence of continuous-time residual networks with linear parameterization in the residual block trained by GD. Due to linear parameterization, the residual block functions are understood to be functions from an RKHS. Gradient of the loss function wrt the parameters (in this case a family of RKHS functions parameterized by time) is derived using standard techniques in optimal control. Then, a functional PL condition is proved for the gradient, which leads to local convergence. To achieve global convergence, two approaches---dimensionality lifting and random Fourier feature approximation---are studied for a special kernel. Final theorem says "global" convergence can be achieved when the network takes random Fourier features, is sufficiently wide, the matrices A and B are appropriately chosen, and the parameter is initialized from 0. 
strengths_and_weaknesses: Originality: The novel parts of this work lie in the functional PL condition for the loss of the model studies, as well as the the approaches to extend local convergence to global convergence in section 5. Though, the reviewer has question regarding the "global convergence" which is left for the questions part. The continuous time residual networks with linear parameterization has been explored in previous works, e.g. in [1]. The local convergence results based on PL condition is also not novel to the current work. 

Quality and clarity: The paper is clear well organized.

Significance: The analysis is theoretical significant because it reveals the good loss landscape for one type of flow-based model. The flow-based model is theoretically interesting because they are infinite-depth limit of deep neural networks, especially ResNets. It is also practically important, with neural ODE as a representative application. 


[1] E W, Ma C, Wu L. Machine learning from a continuous viewpoint, I. Science China Mathematics. 2020 Nov;63(11):2233-66.
questions: 1. To the reviewer's understanding, "global convergence" in optimization means the algorithm converges from any initialization, which does not need to be close to the minimum. However, in section 5 of the paper, convergence to global minimum is proven for a specific initialization, v^0=0. This is not global convergence. Is there a way to improve the results to real global convergence, i.e. the GD converges to global minimum from any (reasonable) initialization? If not the statements in the paper might need to be changed.
limitations: The limitations have been addressed in the paper.
review_title: null

final_score: rating: 1: Very Strong Reject: For instance, a paper with trivial results or unaddressed ethical considerations
review_content: summary: This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.
strengths_and_weaknesses: This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.
questions: This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.
limitations: This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper provided a local Polyak-Lojasiewicz inequality for a kernelized neural ODE-model. This paper's analysis is quite interesting, but still relationship between the proposed work and the previous should be improved. 
strengths_and_weaknesses: This paper provided a local Polyak-Lojasiewicz inequality for a kernelized neural ODE-model. This paper's analysis is quite interesting, but still the relationship between the proposed work and the previous should be improved. 


This paper present a nice Polyak-Lojasiewicz property for the Neural ODE model. I think this paper definitely will make impact in this area. I'm willing to upgrade my score if the paper can make the relationship between the proposed work and the previous more clear.  (detailed points see  Questions)


questions: I guess the authors need to discuss the realtionship between [1], which provided the convergence properties of Gradient Descent for linear ResNets. Can your analysis be considered as iinfiinte dimension analysis of [1]?
[1] M. HARDT AND T. MA, Identity Matters in Deep Learning, arXiv:1611.04231 [cs, stat] (2018). arXiv: 1611.04231.

What's the relationship between your analysis and the Resnet's NTK (which can be written down as an ODE). What's the further degree of freedom increased in your paper. At the same time, this paper is missing reference [2]. I'm curious about the relationship between your paper with [2]. Can the analysis in [2] be considered as a PL analysis?
 
[2] Ding Z, Chen S, Li Q, et al. On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime[J]. arXiv preprint arXiv:2110.02926, 2021.

The assumption 2 is the core assumption to imply the PL condition. Can the analysis in line 256-258 be more concrete?  In there a principled way to construct such RKHS?
limitations: Thi limitation of this paper is two fold in my mind:
- i don't know how strong is assumption 2
- the hidden weight is not trained. [ but i think this model is still more powerful than NTK model.
review_title: null

Meta-review: metareview: The paper presents convergence analysis for ResNet in a certain asymptotic regime, for which the authors are able to establish local Polyak-Lojasiewicz inequality. The analysis sheds new light on the convergence of neural network training. It is a technical sound paper and merits acceptance to the conference. 
confidence: Certain
Decision: Accept