On global convergence of ResNets: From finite to infinite width using linear parameterization
Authors: Raphaël Barboni
Gabriel Peyré
François-Xavier Vialard
Venue: NeurIPS 2022 Accept (2022)
Keywords: Residual Neural Networks
Neural ODEs
Deep Learning

Abstract: Overparameterization is a key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks. Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using on convex optimization technics. To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parameterization while still being nonlinear. Such ResNets admit both infinite depth and width limits, encoding residual blocks in a Reproducing Kernel Hilbert Space (RKHS). In this limit, we prove a local Polyak-Lojasiewicz inequality. Thus, every critical point is a global minimizer and a local convergence result of GD holds, retrieving the lazy regime. In contrast with other mean-field studies, it applies to both parametric and non-parametric cases under an expressivity condition on the residuals. Our analysis leads to a practical and quantified recipe: starting from a universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with high-probability our expressivity condition.

On global convergence of ResNets: From finite to infinite width using linear parameterization
Raphaël Barboni
Gabriel Peyré CNRS and ENS PSL Univ.
ENS PSL Univ.
raphael. barbonidens.fr
sabriel.peyreeem..
François-Xavier Vialard
LIGM, Univ. Gustave Eiffel, CNRS
francois-xavier www.aieeeeaaaa...... vialarddu cellarddu-pem..ee
Abstract
Overparameterization is key factor in the absence of convexity to explain global convergence of gradient descent (GD) for neural networks. Beside the well studied lazy regime, infinite width (mean field) analysis has been developed for shallow networks, using convex optimization techniques. To bridge the gap between the lazy and mean field regimes, we study Residual Networks (ResNets) in which the residual block has linear parameterization while still being nonlinear. Such ResNets admit both infinite depth and width limits, encoding residual blocks in a Reproducing Kernel Hilbert Space (RKHS). In this limit, we prove local Polyak- Lojasiewicz inequality. Thus, every critical point is global minimizer and a local convergence result of GD holds, retrieving the lazy regime. In contrast with other mean-field studies. it applies to both parametric and non paraamameteic cases under an expressivity condition on the residuals. Our analysis leads to a practical and quantified recipe: starting from universal RKHS, Random Fourier Features are applied to obtain a finite dimensional parameterization satisfying with high- probability our expressivity condition.
1 Introduction
State of the art supervised learning methods are based on deep neural networks, sometimes heavily overparameterized, which perfectly fit training data even noisy data while exhibiting good gen- eralization properties. Such a behaviour appears as paradox and questions the established theory of "bias-variance trade-off" (9] That overparameterized model can fit data perfectly comes as no surprise but this capability does not explain the observed generalization properties. Towards a better understanding of it, one first needs to understand the optimization procedure in the parameter space that selects the interpolation map. This question is tightly linked with the parameterization of the space of maps that are explored and state of the art parameterizations have emerged in the past years. One key architecture that is ubiquitous in deep learning are skip connections, heavily used in Residual Neural Networks (ResNets) 25 and has led to state of the art results in supervised leamiing ResNets actually allow one to consider very large number of layers [59].
Continuous models. Passing the limit of infinite depth allows the connection with continuous models (Neural ODE) for which theoretical methods and new algorithms can be designed [T]] (56).
Indeed, the similarities between ResNet architectures and discrete numerical schemes motivated the introduction of a continuous neural ODE
& v(W,
Vt E [0, 1].
(1)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where W E ()), transformation whose output is the residual term. These models correspond to limiting models of a discrete ResNet whose depth D) tends to infinity. Therefore, their study brings a theoretical framework for understanding deep ResNet architectures, and more generally very deep NNs [99 (0). Moreover, their mathematical analysis is facilitated since it allows one to leverage large body of works and tools from analysis and in particular the theory of optimal control [47] Conversely, methods from numerical analysis can bring inspiration for designing new architectures and new optimization algorithms [39].
1,, .."') is the parameter of the model and IR" x R 7 R" is a residual
RKHS parameterization. Most often in the literature studying the training properties of ResNets, the considered residual transformations are Multi-Layer Perceptrons (MLP) [6 24]. Those con- sist in the composition of several trained linear layers alternatively composed with non-linear activation function. A 2-layer MLP with width r reads:
v: (WW U), - Wo(Uz).
where U E IR" and E Rexr are the parameters for the "hidden" and the "visible" layer respec- tively and I2- + R is a non-linear uctivation function applied component-wise. Popular activation functions are for example the ReLU or the Swish function. Provided with these activations, MLPs enjoy a nice universal approximation property as shown in the seminal work of Barron [6].
In contrast, we consider here a setting where the residual term linear w.r.t. the parameters while still being nonlinear w.r.t the inputs. Given feature map y IR9 + R", we consider as space of residuals the space:
V = (U:2 ( Wo E R9X"].
where the matrices W E Rexe are the trained parameters. Compared to Eq. (2) this can be seen as an MLP where the hidden layer is fixed by introducing the feature map y + (U2) for some
feature matrix U. As is standard, the gradient of some loss L w.r.t. W is computed in the sense of the Frobenius metric on the set of matrices:
A W, W' E ROXT, (W,W") = Tr(W WW")
(4)
Such an 2 penalization induces a metric structure on V through the identification <> W in Eq. (3)
Ve,e' E V, (u, d')y == (W, W')
(5)
As finite dimensional space of continuous maps, V has the structure of Reproducing Kernel Hilbert Space (RKHS). Moreover, as pointed out in (5). the space V has natural infinite width limit mean field limit which an infinite dimensional RKHS.
In this paper, we are interested in understanding the convergence properties of Gradient Descent (GD) on a ResNet model for which the residual layers are encoded in possibly infinite- dimensional - vector-valued RKHS V. For V as in EEq. (3) we stress out that, as the metric on V is induced by the one on Roxr, GD on V for this metric is strictly equivalent GD on Raxr with the Frobenius metric. Our model is defined as follows:
Definition 1 (RKHS Neural ODE (RKHS-NODE)). Let V be a RKHS of vector-fields over IR and A E Raxd, B E Rdd x9, Then for v E (²), 1], V) and a data input E Rd, the RKHS-NODE's output F(u, x) = Bz1, where z the solution to the forward problem
2 (2)) and 20
Ax.
The variable will thereafter be called control parameter.
Remark 1. Note that the matrices A and B are fixed and only the control parameter v is trained. However, argue that our approach can be simply adapted to the case where B is trained, fol- lowing for example the proof of (43). Training A seems more challenging as the model is highly non-linear w.E.t this parameter.
Relevance of the RKHS model. The main difference between the model of Detinition 1 and standard ResNets is linearity in the parameters of the residual blocks. As a comparison, 2-layer MLP is nonlinear w.r.t. the parameters of the hidden layers. However, this linearity assumption does not impact the expressivity of the model, but only training dynamic. (i) Indeed, considering V to be Random Features approximation (c.f. Eg. (20)p of some universal RKHS, the residual blocks

---

are as expressive as a 2-layer MLP since both are dense in the space of continuous functions. (ii) Up to the cost of adding a supplementary variable, the dynamical system parameterized by a 2-layer MLP can be expressed as model which is linear w.r.t. its parameters [56] Section 3.2]. Only the training dynamic between these two architectures differs. Also, this assumption of linearity in the parameters also prevents the use of normalization layers. In this direction, [6]] has shown that ResNets without normalization but proper initialization of the weights can lead to robust training the train set than standard ResNets. Finally, the model of Definittio
and similar performance still retains the effect of depth and the nonlinearity w.r.t. the input. Due to composition of these residual blocks the model's 's output is still highly non-linear w.r.t. parameters. For these reasons, consider this model as an important step towards the study of the general case.
In turn, this linearity in parameters naturally leads to an RKHS parameterization which has two important benefits on the theoretical side: (i) Flows of vector-fields as implemented by our model in [Eq. (6) have already been studied theoretically and for applications in image registration prob- lems [58 g 44. Under some regularity assumptions on the considered RKHS V. one can show that the model's output corresponds to the invertible action of diffeomorphism by composition on the input (55). This property was already used in [51] to implement models of Normalizing Flows [29] with applications in generative modeling. (ii) There is an important literature in Machine Learning about Kernel methods [52]. In practice, various sub-sampling methods exist in order to approximate RKHSs with finite-dimensioe
spaces generated by Random Fourier Features
Thereby, leveraging results on the approximation bound for RFF [54] [3], we show that the expressiveness properties of universal kernels, such as the Gaussian kernel, can be efficiently recovered using residuals of the form Eq. (3) with a finite number of neurons.
(RFF) 48 49
To further support the practical applicability and the relevance of our model in comparison with stan- dard architectures, we report in the supplementary material (Appendix A) numerical experiments on MNIST and CIFARIO datasets. They show that predicted by our theory our model can be trained these cases to almost zero loss. But more importantly, they show that our model is able to generalize well the test dataset with performances that are similar to those of classical ResNets.
Supervised learning. consider map F from H x Rd to Rdd for some Hilbert space of parameter H (e.g. the model of Definition with H = [[, 1], V)) and a training dataset consisting on a family of inputs (x')ısigN E (R and target outputs (y") E (R="). Then for every parameter e E H. we define the associated Empirical Risk as:
L(v) = 2N
Σ I5ISN
[FF(,,"'
For simplicity consider here the Euclidean square distance as loss the output space Bed,, but results generalize to any smooth loss satisfying a Sojyak-Ljjsiewicz inequality (O.). e.g. any smooth strongly convex loss.
Remark 2.
Training the model F then amounts finding parameter 8" E arg min L(v). In order to perform such an empirical risk minimization (ERM) task we consider GD on v. For a small step size 77 for some initialization no E HH and for every discrete time step & N, the training dynamic reads:
++1
Note that do not consider any additional regularizing term on the loss. In classical supervised learning one would seek for a minimizer of the "regularized" loss L(u) + XR(v), with X >> 0 a con- stant and R a coercive regularization function. We are here interested in the non regularized setting, i.e. x = O, often used in practice. In this case, the generalization property of the computed map is argued to potentially come from the optimization method that shall select an adequate minimizer of the loss. This implicit regularization depends on the choice of the optimization method 42).
2 Related works and contributions
Recently, several works have addressed the problem of proving convergence of (stochastic) GD in the training of NNs. If the convergence properties of GD are well understood for NNs that are linear
w.r.t. input 24 7 64 it is not the case for non-linear NNs. on the training of "shallow" two layers fully connected NNs and establish convergence of GD in an
33 17, the authors focus
3

---

overparameterized setting where width of the intermediary layer scales polynomially with the size N of the dataset. More recently, with the same setup. [62] showed that the neurons of teacher network are recovered by a student network optimized with GID as long as the width of the student network is higher than the teacher's ' one. Formally, their analysis is similar to ours as the result holds if the loss inequalities
at initialization is already sufficiently low and the proof relies on Polyak-Lojasieww verified by the loss landscape.
Infinite depth. The works of [16 64 32 63 35 12 43 extend those results to arbitrary 35 apply to deep deep NN in the overparameterized setting. Specifically, the results in [16] 2
ResNets. The best result seems to be achieved in 43), with convergence as soon as the last layer has a width m = (2) and at best with linear width. common feature for those works is to rely on the fact that, for a sufficiently high number of parameters, the model can be well approximated by a linear model corresponding to its first order expansion around the initialization. In [15] this phenomenon, called "lazy regime", is attributed to an inappropriate scaling of the parameters. On the other hand, 33 35] refer to this phenomenon as "linear" or "kernel regime" and relate the constancy of the Neural Tangent Kernel (NTK) introduced in [26]. However, in all those works the width of intermediary layers has to depend on the depth D of the network. Therefore, these results do not apply to the training of the model in Eq. (I)) corresponding to the limit D - +00.
Infinite width. The other direction of over-parameterization, analyzed in several works 44 14 40 27 (38 21 46 is to consider the limit of infinitely wide layers. In such a "mean-field" setting, the model is parameterized by the distribution of the parameters at each layer. In [14] 41 40 27 the training dynamic is analyzed as a gradient flow in the Wasserstein space [3]. showing that the only stationary distributions are global minimizers of the empirical risk. In [2T] a similar result is showed for deep NN with arbitrary number of infinitely wide layers. In [3 local linear convergence towards the global optimum is shown for two layers NNs in a teacher-student setup with regularized loss. Finally, 38 analyzes the convergence of continuous ResNets with infinitely wide residual layers and shows that every critical point is a global minimizer of the empirical risk. We stress out that these results only apply infinitely wide NNs. It is not clear if this mean-field limit extends to the parametric setting of MLPs with the Euclidean metric on their parameters. In contrast, a RKHS structure naturally arises when considering a linear parameterization of the residuals. [Asumption I and Assumption 2] can be satisfied both in a parametric setting with a finite number of features and in mean-field setting limit where the residuals are generated by a universal kernel.
Contributions. We show convergence results for GD in the training of RKHS-NODES (see Def- inition T). These correspond infinitely deep continuous ResNets with linear parameterization of the residuals. Our first main contribution, in [Section 4| shows that under some regularity and expressivity assumptions on the residuals, the associated empirical risk satisfies a (local) Polyak- Lojasiewicz [Property 2] A consequence is [Theorm 2 which states global convergence of GD towards global optimum (zero training loss) under the condition that the loss at initialization is already sufficiently low. In the limit where the loss at initialization is arbitrarily small, we recover a linear regime as described in 36. 35). Our second contribution, in Sectionn condition for global convergence can be enforced using suitably chosen first and last linear layers. Thereafter, we show how the assumptions of ohehrem 2] can be satisfied for RKHSs generated by a finite number of Random Features, with high probability over the choice of these features. For any dataset (a',, Df)isI N E (IRd x IRd") ) we conclude in Theorem 3 to convergence of GD towards a global minimum of Eq. (7) with high probability when the width of the layers scales polynomially w.r.t. the size of the dataset N and the inverse input data separation 8-1
shows how this
Finally, we point out that some of our results can be seen as generalization of existing results
concerning convergence of GD for the training of linear NNs 24 how, following the line of our analysis, one can for example recover 664 Theorem 3. However,
7 64) We explain in Appendix E
ie encompasses linear ResNets as special case, we stress that Theorem 2 aapplies to
way larger class of models.
||-|| denotes the Euclidean e norm for vectors and the Frobenius norm for matrices. For matrices the spectral norm is denoted ||-1|2, the smallest (resp. greatest) singular value is denoted Jmin (resp. Fmax) and for symmetric matrices the smallest (resp. greatest) eigenvalue is denoted Amin (resp. Ammax). Given some Hilbert space H. the functional Hilbert space T.2 1], H) is denoted 1.2 (.H) or 1,2 when there is no ambiguity. The notation O (resp. ?) means asymptotically inferior (resp. superior) up to multiplicative constant.
Notations. In what follows

---

3 Analysis of convergence for overparameterized models
In this section, we review methods for analyzing the convergence of overparameterized machine learning models based on [36 35]. We refer to Appendix B/ for detailed proofs of the statements.
As presented above, we consider an optimization over the variable v in some Hilbert space H. with fixed imput and output data, say + F(v) = F( [ Therefore, the empirical risk is function of the parameters v E H. We say that the model is overparametertze whenever the dimension dim(H) of the parameter space is much larger than the dimension of the output space of F(v), here d' N. The RKHS-NODE model defined F in Defiiiitio falls into this category as H the infinite dimensional functional space ( v).
3.1 A (local) Polyak-Lojasiewizn property
When dealing with overparameterized models, one cannot expect the loss to be convex but one expects the model perfectly fit the data, that is to reach the global minimum value of O. In fact, for a sufficient numher of parameters, the loss landscape typically possesses a continuum of infinitely many global minima and is non-convex in any neighbourhood of global minima [36]. One thus rather needs to rely on a set of functional inequalities allowing to control the decrease rate of the loss along GD
Definition 2 (local) Polyak-Lojasiewiez property). Let L H + R. be a differentiable function. We say that L satisfies a (local) Polyak- Lojjsk--jjsiiwicz (PL) property if there exist positive continuous functions m, M. R44 + Rt s.t. for every v E H
2m((l=|))(o)+ < (V(()) < (M(())))LO).
Such functional inequalities have already shown to be relevant for proving convergence guarantees in the training of NNs (22). A first consequence for loss L which satisfies the (local) PL property Detiniitinn 2 is that it does not admit any spurious local minima but only global minima. Also, if the training dynamic is bounded, then m and M are uniformly lower- and upper-boundeds along the dynamic, implying that L decreases at a linear rate. In most cases, m and M are degenerate when ||ا|| + +oc. When the dynamic is not bounded, L can thus decrease 0 slower than a linear rate or even converge towards strictly positive limit.
3.2 Local convergence result
Because of the degeneracy of m and M, it is in general not possible to conclude an unconditional convergence of GD towards a global minimizer of the empirical risk. However, PI. inequalities sufficient to prove convergence when the problem is not too hard to solve, that is when the loss at initialization is not too high. Moreover, when using gradient descent stepping, one needs to make supplementary smoothness assumption on the empirical risk I.. This ensures that the loss decreases each gradient step for a sufficiently small step size.
Definition 3 (Smoothness, Definition 2 of (6) ). Let
be constant. We say that the function L: 7l + R is 8-smooth for every vf E H: [L(x") L(v) (VI(e), u>1 < 9 1ll v1l2.
The local PL property combined with this smoothness assumption then gives local convergence result for the convergence of GD towards a global minimizer of the empirical risk.
Theorem (Theorem of (3)). ). Let L H. + R4 be loss function satisfying a local PL property with local constants m and M. Let S E H. and R 2 0 be such that
2 / |M + R)
VL(1)) S R.
mi[|o"1| + R)
Furthermore, assume that L is 3-soooth within the ball B(o", R). Then for a step size 1] S $-1, GD with initialization o and step size n, converges towards global minimizer of L with a linear convergence rate and inside a ball of radius R. More precisely, for every k 2 0:
1(v²) < (1 10 R)n)* L(e") and 1|e² - 001 < R, Vk V
(10)
5

---

Properties of RKHS-NODE
In this section we analyze the convergence of GD the training of the infinitely deep ResNet model Definition I Note that such model is overparameterized in depth as the parameter space is the infinite dimensional space [ I V) and overparameterization can also come from width when the RKHS is high (or even infinite) dimensional. Therefore, our proof of convergence heavily relies on a PL property verified by the empirical risk.
Recall that we consider the training of deep ResNets with a linear parameterization of the residuals. The of residuals is in Eq. (3) with the metric of Eq. (5) induced by the Frobenius metric (Eq. (4)). This provides V with a RKHS structure 44. whose associated kernel is given for any 2, E I by <(2,=')== = (p(z), (=())) Idg, and whose associated feature map is given by yp
(I) IR' associate each v e V to unique W E grxr, This is satisfied by all the feature maps yp we consider in the following.
Remark 3. The definition of (v. in Eg. (5) requires Span(p
Given training dataset composed of input data points (a*) (s") E (RRYN and of target data points (y') E (Rd"), YN we are interested in the task of minimizing the empirical risk of Eq. (7) by GD over v. Analogously be expressed thanks to a backward equation derived by adjoint sensitivity analysis (7
pack-propagation in discrete NNs architectures, the gradient of L can
Property 1. Let L be the empirical risk in Eq. (7) associated with the quadratic loss. Let K be the kernel function associated with the RKHS V. Then L is differentiable on L², 1], VV), with for every v E (1, 1], V), L - = K (x,,²*) z'),,, where for each index V),
model with a
E [1, zf is the solution of Eq. (6) with initial condition Ax and the adjoint variable p³ the solution to the backward problem:
1 p -De p and p =- N
B 3(2 -').
(11)
4.1 PL property of RKHS-NODE
Following the line of proof sketched in /ecction 3
we show how to derive PL incqualities of the form Eiq. (8) for the empirical loss associated with the RKHS-NODE model. For that purpose we make a few assumptions about the RKHS V. The first concerns its regularity and allows to control the solutions of Eqs. (6) an)
Assumption 1 ((strong) Admissibility), We say that the RKHS V is (strongly) admissible if it is continuously embedded in 22, (R*, IR9). More precisely, there exists constant >> 0 s.t.
V V,
|| + Deell,,so D1llul, S K||u|lvv
(12)
Assuming is embedded in W1.00 (R2, IR9) natural to ensure the regularity of the flow generated by the control parameter [55] 58 and suffices prove convergence of continuous gradient flow on the parameter v. |Assumption is a bit stronger because supplementary smoothness result on the loss landscape is necessary to prove convergence of discrete GD (c.f. Definitio 3). In practice, i can be computed for smooth kernels thanks to Property 4 in Appendix D For example, the RKHS associated with the Gaussian kernel k : r 722 is (strongly) admissible with k = 2 + v3.
The second assumption is related to the expressiveness of V and is weaker form of the classical universality property of RKHSs.
Assumption 2 -N-uiiversalitt). Let K be the kernel function associated with the RKHS V. For family of points (2f)sN E (R9),, define the associated kernel matrix as the block matrix (((());)) == (K(z', (')ssigNN-
More precisely we assume for every 8 >
sup Amax (K((=))) too and X(6-1)= (#)))(RO)
inf (R))) Amin (K((()(()))))) > 0. (13) minagy ||='6
Assumption 2 is required in order to ensure the expressivity of our model, quantified by the con- ditioning of the kernel matrix K and by A and x. The choice of the RKHS V may thus have

---

significant impact on training, In particular, satisfying Assumption 2 requires having V of dimen- sion m 2 N. but it can be satisfied for finite dimensional RKHSs of dimension m S Nº, for example by considering polynomial kernel, or by RKHSs of dimension m 2 poly(N, 4) with high probability on the sampling of random features, as shown in [Section 5 the other hand, even though the existence of À follows from compactness arguments, it seems to be hardly analytically tractable even for classical kernels such as the Gaussian kernel. Therefore, if, in theory, prior knowl- edge of the data distribution might allow to optimize the choice of kernel, we expect the selection of an optimal kernel to be an intractable problem practice. Instead, cross-validation techniques can be used to select suitable kernel.
Remark 4. For RKHS V as in Eq. (3) the properties of V only depend on u. An interesting + feature matrix. In InSSccionn 5 3we show that, when considering the complex activation o : + e e both assumptions can be satisfied with high probability. On the other hand, Assumption 1 is not satisfied when considering
ample is when yp 2 + (U=)) with a an activation function applied component-wise and U a fixed
ReLU due to its non smoohhotssssss O.
Remark 5. Note that A could also be allowed to depend on some parameters, such max || However, as it is a more critical aspect of our analysis, we prefer to highlight the dependency of W.E.t minizj || =21 For all the RKHSs studied here we always have A S N.
The following PL property is satisfied by the risk L. Property 2 is proven in Appendix C.2
Property (RKHS-NODE satisfy PL.. Assume V satisfies Assumption I with k and Assumption with X and A. Let L be the empirical risk in Eq. (7) associated with the RKHS-NODE model of Deftnition 1| Then L satisfies the PL inequalities DfDeniition 2) with m and M given by:
1 M(R) = mmax (B )e
m(R)= = N (B (a (amin(A))22) ( -2xR (14)
where 5 = minszy ||== x²| || is the data separation.
Sketch of proof Assumption 1 can be used to have estimates on the solutions 2' of the forward problem Eq. (6) and on the solutions p² of backward problem [Bq. (11) This gives for every indices i E [1, NJ and every / E [0, 1]:
|
2 2 J miin (A)ll=* - x -eeeellll
where 2' solves Eq. (6) with initial condition Ax", and:
2x |Ill L2 || 1|2 < || S 20 lppi
Moreover using the initial condition pi = - B T (Bzi n² we have:
20 TB
N
20 T2
L(u) S
1 <
L(u).
N
N
Then denoting Pe the vector of stacked pi and using properties of RKHSs, we have for t E [0, 1]:
|VL())1 = Σ (o)){((ff =/pt (pr. K ((=():))).)
ISUSN
where K is the kernel matrix associated with the points (zt);- This last equality gives the result using Assumption and the previous estimates on p'.
Note that the degeneracy of the bounding functions M, m as R + +oo readily appears in Eq. (14) Thus one should not expect these bounds imply global convergence of GD without making any further assumption. Indeed, cases where GD fails to converge towards a global optimizer of the loss are observed in 77. Section 6, with setup corresponding to the model of Definition with as in Eg.. (3) and yo Idge. Also, note that the data separation 8 plays an important role in Property 2 as it intervenes the conditioning of the kernel matrix. In what follows, we always assume the data points to have a data separation
by d 0.
7

---

4.2 Convergence of RKHS-NODE
Thanks to the convergence analysis for overparameterized models detailed in Section 3 our main result follows as a consequence of the previous property. [Theorem 2 proven in Appendix C.3
Theorem 2. Let satisfy Assumption with constant k. and Asssumption with X, A. Let e be some initialization of the control parameter with || RR and assume there exists positive radius R 2 0 s.L.
ma (B") VNA(( ( Be(RR+ miin (B (im (A) 18-111111 en(n+ S R.
(15)
Then, for a sufficiently small step-size > O, GD with step-size converges towards minimizer of the training loss linear rate and inside a ball of radius R. More precisely, for every k 2
L((*) S (1 (1-1))) L(v"), and l1 v V S R,
(16)
where μ === o min (B) )) (
e e-2x(R+Ro),
As Theorem Theorem 2 is a local convergence result in which the condition in Eg. (15) expresses
a threshold between two kinds of behaviours: (i) if L(v") is sufficiently small, the training dynamic
converges towards a global minimizer. The limiting behaviour is when the l.h.s. of (15) tends
0. Because of regularizing effect of GD (i.e. that ||e* e I| I R). the parameter stays in ball of arbitrary small radius R all along the training dynamic. In this limit, we recover "linear" or "kernel" regime where the model is well approximated by its linearization at e [14 35 26). (ii) If L(v") is too large, the result says nothing about the convergence of the GD. However, it is still observed in practice that the training dynamic often converges towards a global minimizer of the loss [60]. Explaining this phenomenon in general setting remains challenging open question.
5 Enforcing convergence with high dimensional embedding and finite width
As Theorem is a local convergence result, it does not allow to conclude a general convergence behaviour of GD in the training of RKHS-NODE In the following, we show how one enforce the hypothesis of Theorem 2 to be verified and prove two global convergence results. The first one relies on suitably choosing matrices A and B in order satisfy [qq (15) and applies in the of infinite width, i.e. with residual layers in a universal RKHS. The second result recovers global convergence in finite width regime, relying on high number r of Random Fourier Features.
For the sake of readability we only consider here the case where belongs to a restricted class of RKHSs and refer to Appendix D) for more general results and complete proofs. For some positive parameter y > 0 we consider the Matérn kernel defined in [57]
21
2v
2v
Vr RR, k(r)
KC,
2T r):
(17)
[(v)
2T
where I' is the Gamma function and Kr, is the modified Bessel function of the second kind. Equiva- lently, is can be defined by its frequency distribution over Rd as:
[ el(e,w) My(W)d with (g(w) C1,
| )
Ve E R9, +(1+=1))
(18)
2v
and Cg.y a normalizing constant. For every q 2 1, such a function known to define a structure of vector-valued RKHS V,, over IR' corresponding to the Sobolev space 2 R*) 52 577.
The associated kernel is given for every 2' R29 by: K,(2, = k[[l= 2 ld Note that is important for this RKHS to depend on the ambient dimension 9. In particular the Sobolev space He (TR9, R2)) is RKHS if and only if it has regularity
q/2. Assuming
2, Mgg further
admits up to 4 finite order moment implying that & is four times differentiable at 0 (28). Then V, satisfies Asssiiss/ssummptinn I with some constant x depending only on v and given by Property 4
3v2
k =K(0) (k()(0) (L0
+
(19)
(v-1)(v
2)
8

---

Also, Vg satisfies Assumption 2 with A S N and À depending a priori on and N.
Note that with this choice of scaling for k and Mg. one recovers the Gaussian kernel k: + e -2 in the limit v + too [57]. Thereafter we consider, V E (2, tool, the case v = +o0 referring to the Gaussian kernel. We also assume that the data distribution is compactly supported. In particular there exists some ro 2 so that every input data z verifies ||>||| I ro-
5.1 Global convergence with high-dimensional lifting
We first show how Eq. (15) can be satisfied by considering appropriate embedding matrices A and B. Doing so, the square distance between the data points, i.e. the model's loss, is preserved whereas the conditioning of the kernel matrix can be controlled.
Proposition 1. Let v E (2, +o0], let (2i, 1SSIN (IR xIR" ) be a dataset with data separation 5 > and let Rt >> 0. There exist 2 1 and matrices A E RRxd B E IRd xq s.1. GD initialized at 0 0 converges lowardy zero-training-lot
optimum in the training of RKHS-NODE.
In particular, Eq. ho5)s with radius R and K, X, A associated with the RKHS Vq.
As shown in the proof in Appendix D. Proposition I sttlll holds for small but non-zero initialization. We present here two ways of obtaining matrices A and B satisfying [.q. (15)
Scaling Consider A a(I,,, O)T R(d+d))xd and B (0.
IR x(("a") for 0.
We show in Appppeddi D. D.1.2 in this setting, the l.h.s. of Eq. (15) scales as (1/aa) and thus The- orem 2) holds for large enough a. Moreover, observe that q d d' is independent of N and and such a regime can easily be implemented in practice. However, it has been shown that, although interpolation of the training data can be achieved as a consequence of a suitable rescaling of the parameters, this "lazy regime" can also lead to bad generalization properties [15].
Ag == q //1/4(Ida, ..., Ida, 0)T E Raxd and Bg == /1//(Idaa 0)..0) E Rdd xq, with Lg/d] copies of Idl in Ag. This choice is motivated by the intention for these matrices to produce high-dimensional lifting, which has been shown to im- prove on the expressivity of ResNets [18). We then show in (Appendix D.I.I that Eq. (15) can be satisfied for g 12(N4 + 8-4 (og)(N)*) We do not expect our condition on q to be optimal as we observe in experiments (see Appendix A) that regime of linear convergence can be obtained for q < N + 5-4 (N) However, we observe that increasing q does improve on the convergence and generalization properties of our model (Fig. 2)
Lifting Consider for q 2 the matrices:
5.2 Global convergence with finite width
In the preceding we showed that, in the case of an RKHS defined by a Matérn kernel, convergence of GD can be ensured for well-chosen matrices A and B. However, for practical implementations, the form of the residual in Eq. (3) forces
consider RKHSs defined by feature maps. A way to overcome this difficulty and to benefit from the properties of a wide range of kernel functions is to consider approximation by Random Fourier Features (RFF) 48 49 More precisely, given
q 2 1, recall the definition of the Matérn kernel k in terms of its frequency distribution (4g over R sid in Eq. (18) and for any sampling w', ..... w" M4y of size r, consider the feature map:
P:EERT (e (a Disis C..
(20)
In other words, considering the complex activation :t + ex applied component wise and U : (w=).... (w") E gaxr the feature matrix, we have (p(2) r-1/2o(U 2). Recall that such
feature map defines a structure of RKHS on V, = (W as a finite-dimensional approximation of the universal RKHS V,, as it is associated with the kernel function (2 z²) = k(2, x') Idg. with:
W Rexr). Such V, can be viewed
h(,,2/) = (e)z))=(((((2))t))))
= Σ c'(=-s/w)) t00 k([l 2'11) a.s.
Given any q 2 1, we show that, with high probability over the choice of features, V., recovers the properties of admissibility and universality V, as soon as r sufficiently high w.r.t g and N. The following is particular case of Proposition 5 in Appendix D.2

---

Proposition 2. Consider any N 2 2 and any E, R > 0. Assume > 4.
(i) Forr 2 (TTq)), ). with probability greater than 1 1- ,, V, satisfies/ Assumption / with is S ki k+
(ii) For 2 ME 10(e222(00 (g + R) + T)), with probability greater than 1 - e , for any where
E (V²) s.I. ||=|| < R and any time E [0, 1]: Amin (K(n((((=))))) 2 Amin ((((()))))))))
(z'); are the solutions to
Eq. (6) and K, K. are the kernel matrices of and respectively.
Sketch of proof for (i). First note that for 4, My admits up -order finite moments and these can be bounded uniformly in (28). Let yp be the feature map of Eq. (20) Then for every 2 E IR, [x))()]] || & 1 so that for every u E Vg. 1101|00 I || Σ | For the differential Dv we have for every E R
1 D+(=)= = VF (wfe ER
in
Then, by the Bienayme-Chebysher inequality, Dz(z)) = +Ej= w (W))T converges in probability (0) Id, +00. Thus, for > 0 and sufficiently high w.r.t. q. a and T, ||Dulll,cc = || S V-k((0) + allllly, with probability greater than --1 The same idea applies to bound ||DRull2,00 and the result follows using that is given by Eq. (19)
we obtain a global convergence result. The- orem 3 states convergence, with high probability over choice of features, of GID towards zero- training-loss optimum for infinitely deep ResNets of finite width.
Finally, combining Proposition 1
and Proposition 2|
Theorem (Global convergence). Assume v > and let (x', .y") E (TRd Red be compactly
supported dataset with input data separation > There exist matrices A Raad and B E Rd xq s.t. for any 0, with probability at least 1 1- T- w.r.t. the choices of features, GD initialized at y O converges towards zero training loss optimum in the training of the RKHS-NODE model Defimition with the feature map yp of Eg.. (20) as soon as 51(T(4" + NNN [og([ All2)).
Proof. Consider R By Proposition we can have A E RRxdd B IRd that Eq. (15)
8v/00 max (B NN miin (B ( for K, À and associated with k. Also, by the proof of Proposition I we can have: X(omin (((++)))) 2 1/2. Taking 1/4 in Proposition 2 the condition in Eg. (15) satis- fied by V. with probability greater than 1- as soon as r 2 ((rgg*+gN"" log( 1 + Al)2r))).
1,
O
Conclusion
We have identified a relevant infinite width limit (RKHS-NODE) for a particular model of ResNet. We showed that GD converges linearly when training this model and that a network's width poly- nomial w.r.t. to the size of the dataset is sufficient to maintain this property. A natural extension of our result study the convergence of GD when also training the hidden layers of the residuals. A first step towards this general case consists in studying the corresponding mean field model where the residuals are parameterized by density distributions over the neurons 14 each residual blocks. Interestingly. such parametrization of the residual blocks is still linear in this measure and thus fits into our framework of linear parameters. However, would require a finer mathematical analysis to obtain similar results.
40
21 for
Potential Negative Societal Impacts. Our work aims at improving the theoretical and practical understanding of deep networks and therefore we do not expect direct negative impact.
Acknowledgements
The work of Gabriel Peyré was supported by the French government under management of Agence Nationale de Recherche as part of the "Investissements d' l'avenir" program, reference 19R19- P3IA-0001 (PRAIRIE 31A Institute) and by the European Research Council (ERC project NORIA).
10

---

References
[1] S. AKIYAMA AND T. SUZUKI, On learnability via gradient method for two-layer relu neural networks in seacher-student setting. arXiv e-prints, (2021), pp. arXiv-2106.
[2] Z. ALLEN-ZHU, Y. LI, AND Z. SONG, A convergence theory for deep learning via over- parameterization, International Conference on Machine Learning, PMLR, 2019, pp. 242- 252.
[3] L. AMBROSIO, N. GIGLI, AND G. SAVARÉ. soudientflowss. in metric spaces and in the space of probability measures, Lectures in mathematics ETH Zürich, (2008).
[4] N. ARONSZAIN, Theory of reproducing kernels, Transactions of the American mathematical society, 68 (1950), pp. 337-404.
[5] F. BACH, Breaking the curse of dimensionality with convex neural networks, The Journal of Machine Learning Research, 18 (2017), pp. 629-681.
[6] A. BARRON, Universal approximation bounds for superpositions of sigmoidal function, IEEE Transactions on Information Theory, 39 (1993), pp. 930-945.
[7] P. BARTLETT, D. HELMBOLD, AND P. LONG, Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks, in Interna- tional Conference on Machine Learning, PMLR, 2018, pp. 521-530.
[8] M. F. BEG, M. I. MILLER, A. TROUVÉ, AND L. YOUNES, Computing large deformation metric mappings via geodesic flows of diffeomorphisms, International journal of computer vi- sion, 61 (2005), pp. 139-157.
[9] M. BELKIN, D. HSU, S. MA, AND S. MANDAL, Reconciling modern machine- prac- fice and the classical bias-variance trade-off, Proceedings of the National Academy of Sci- ences, 116 (2019), pp. 15849-15854.
[10] BOLTE, A. DANILLDISS, O. LEY, AND L. MAZET, Characterizations of Lojasiewicz in- equalitiex: Subgradient flows, talweg, convexity, Transactions of the American Mathematical Society, 362 (2009), pp. 3319-3363.
[11] R. T. Q. CHEN, Y. RUBANOVA, J. BETTENCOURT, AND D. DUVENAUD, Neural ordinary differential equations, Advances in Neural Information Processing Systems, (2018).
[12] Z. CHEN, Y. CAO, D. ZOU, AND Q GU, How much pverpparrmtttiizzaionn is sufficient to learn deep relu networks?, in International Conference on Learning Representations, 2020.
[13] L. CHIZAT, Sparse optimization on measures with over- perapearaeter gradient descent, Math- ematical Programming, (2021), pp. 1-46.
[14] L. CHIZAT AND F. BACH, On the global convergence of gradient descent for over- parameterized models using optimal transport, Advances Neural Information Processing Systems, 31 (2018), pp. 3036-300 0046-3046.
[15] L. CHIZAT, E. OYALLON, AND F. BACH, On lazy training in differentiable programming, in NeurIPS 2019-33rd Conference on Neural Information Processing Systems, 2019, pp. 2937- 2947.
[16] S. DU, J. LEE, H. LI, L. WANG, AND X. ZIIAI, Gradient descent finds global minima of deep neural networks, in International Conference on Machine Learning, PMLR, 2019, pp. 1675- 1685.
[17] S. S. DU, X. ZIAII, B. Poczos, AND A. SINGII, Gradient descent provably optimizes over-parameterized neural networks, International Conference on Learning Representations, 2018.
[18] E. DUPONT, A. DOUCET, AND Y. W. TEH, Augmented neural odes, in Proceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, pp. 3140- 3150.
[19] W. E, J. HAN, AND Q. LI, A mean-field optimal control formulation of deep learning, Re- search the Mathematical Sciences, (2019), 10.
[20] W. E, C. MA, AND I.. WU, The Barron Space and the Flow-Induced Function Spaces for Neural Network Models, Constructive Approximation, (2021).

---

[21] C. FANG, J. LEE, P. YANG, AND T. ZHANG, Modeling from features: mean-field frame- work for praameerreeeeiize deep neural networks, in Conference on Learning Theory, PMLR, 2021, pp. 1887-1936.
[22] S. FREI AND Q. Gu. Proxy convexity: A unified framework for the analysis neural networks trained by gradient descent, in Thirty-Fifth Conference on Neural Information Processing Sys- tems, 2021.
[23] K. HALE, Ordinary differential equations, Dover Publications, Mineola, N.Y, dover ed ed., 2009. OCLC: ocn294885198.
[24] M. HARDT AND T. MA, Identity Matters in Deep Learning, arXiv:1611.0422 [es, stat], (2018), arXiv: 1611.04231.
[25] K. HE, X. ZHANG,, S. REN, AND J. SUN, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[26] A. JACOT, GABRIEL, AND C. HONGLER, Neural tangent kernel: convergence and gener- alization in neural networks (invited paper), in Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, Virtual Italy, June 2021, ACM, pp. 6-6.
[27] A. JAVANMARD, M. MONDELLI, AND A. MONTANARI, Analysis of two-layer neural net- work via displacement convexiry, The Annals of Statistics, 48 (2020).
[28] B. M. G. KIBRIA AND A. JOARDER, A short review of multivariate I-distribution, Journal of Statistical Research ISSN, 40 (2006), pp. 256-422.
[29] KOBYZEV, S. PRINCE, AND M. BRUBAKER, Normalizing Flows: An Introduction and Review of Current Methods, IEEE Transactions on Pattern Analysis and Machine Intelligence, (2020), pp. 1-1.
[30] A. LACOSTE, A. LUCCIONI, V. SCHIIDT, AND T. DANDRES, Quantifying the carbon emis- sions of machine learming, arXiv preprint arXiv: 19 190.09970, (2019).
[31] A. LAFORGIA AND NATALINI, Some inequalities for modified bessel functions, Journal of Inequalities and Applications, 2010 (2010), pp. 1-10.
[32] LEE, L. XIAO, S. SCHOENHOLZ, Y. BAHRI, R. NOVAK, J. SOHL-DICKSTEIN, AND
J. PENNINGTON, Wide neural networks of any depth evolve as linear models under gradi- ent descent, Advances in neural information processing systems, 32 (2019), pp. 3582-8583
[33] Y. LI AND Y. LIANG, Learning overparameterized neural networks via stochastic gradient descent on structured data, Advances in neural information processing systems, (2018).
[34] Y. LI AND Y. YUAN, Convergence analysis of two-layer neural networks with relu activation, Advances in Neural Information Processing Systems, 30 (2017), pp. 597-607.
[35] C. LIU, L. ZHU, AND M. BELKIN, On the linearity of large non-linear models: when and why the tangent kernel is constant, Advances in Neural Information Processing Systems, 33 (2020).
[36]
Loss landscapes and optimization in over-parameterized non non-linear systems and neural networks, arXiv:2003.00333 [cs, math, stat), (2021). arXiv: 2003.00307.
[37] S. LOJASIEWICZ. Sur les trajectoires du gradient d' l'u fonction analytique, Seminari di ge- ometria, 1983 (1982), pp. 115-117.
[38] Y. LU, C. MA, Y. LU, J. LU, AND L. YING, A mean field analysis of deep resnet and be- yond: Towards provably optimization via overparameterization from depth, International Conference on Machine Learning, PMLR, 2020, pp. 6426-6436.
[39] Y. LU, A. ZHONG, Q. LI, AND B. DONG, Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations, in International Conference Machine Learning, PMLR, 2018, pp. 2276-3285.
[40] S. MEI, T. MISIAKIEWICZ, AND A. MONTANARI, Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit, in Conference on Learning Theory, PMIR, 2019, pp. 2388-2464.
[41] S. MEI, A. MONTANARI, AND P.-M. NGUYEN, A mean field view of the landscape of two-layer neural networks, Proceedings of the National Academy of Sciences, 115 (2018), pp. E7665-E7671.
12

---

[42] B. NEYSHABUR, Implicit regularization in deep learning, arXiv preprint arXiv: 170.019999 (2017).
[43] Q. NGUYEN, On the Proof of Global Convergence of Gradient Descent for Deep ReLU Net- works with Linear Widths, arXiv:2101 00111111111222 [es, stat], (2021). arXiv: 2101.09612.
[44] M. NIETHAMMER, Y. HUANG, AND F.-X. VIALARD, Geodesic regression for image time- series, in Medical Image Computing and Computer-Assisted Intervention MICCAI 2011: 14th International Conference, Toronto, Canada, September 18-22, 2011, Proceedings, Part II. G. Fichtinger, A. Martel, and T. Peters, eds., Berlin, Heidelberg, 2011, Springer Berlin Heidelberg, pp. 655-662.
[45] A. PASZKF, S. GROSS, S. CHINTALA, G. CHANAN, E. YANG, Z. DEVITO, Z. LIN, A. DES- MATSON, I.. ANTIGA, AND A. LERER, Automatic differentiation in pytorch, (2017).
[46] H. T. PHAM AND P.-M. NGUYEN, Global convergence of three-layer neural networks in the mean field regime, in International Conference on Learning Representations, 2020.
[47] L. S. PONTRYAGIN, Mathematical theory of optimal processes, CRC press, 1987.
[48] A. RAHIMI AND B. RECHT, Random features for large-scale kernel machines, in Proceed- ings of the 20th International Conference on Neural Information Processing Systems, 2007,
pp. 1177-1184.
[49]
Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning, Advances Neural Information Processing Systems, 21 (2008), pp. 1313-1320.
[50] W. RUDIN, Fourier analysis groups, Courier Dover Publications, 2017.
[51] H. SALMAN, P. YADOLLAHPOUR, T. FLETCHER, AND K. BATMANGHELICH, Deep diffeo- morphic normalizing flows, arXiv e-prints, (2018), pp. arXiv-1810.
[52] B. SCHÖLKOPF, A. SMOLA, F. BACH, ET AL., Learning with kernels: support vector machines, regularization, optimization, and beyond, 2002.
[53] B. SRIPERUMBUDUR AND Z. SZABO, Optimal rates for random fourier features, Advances in Neural Information Processing Systems, 28 (2015), pp. 1144-1152.
[54] D. J. SUTHERLAND AND J. SCHNEIDER, On the error of random features, in Proceed- ings of the Thirty-First Conference on Uncertainty Artificial Intelligence, 2015, pp. 862- 871.
[55] A. TROUVÉ, Diffeomorphisms groups and pattern matching in image analysis, International journal of computer vision, 28 (1998). pp. 213-221.
[56] F.-X. VIALARD, R. KWITT, S. WEI, AND M. NIETHAMMER, A ftrotiingformllll of deep learning, Advances in Neural Information Processing Systems, 33 (2020).
[57] C. K. WILLIAMS AND C. E. RASMUSSEN, Gaussian processes for machine learning, vol. 2, MIT press Cambridge, MA, 2006.
[58] L. YOUNES, Shapes and Diffeomorphisms, vol. 171 of Applied Mathematical Sciences, Springer Berlin Heidelberg, Berlin, Heidelberg, 2010.
[59] S. ZAGORUYKO AND N. KOMODAKIS, Wide residual networks, in British Machine Vision Conference 2016, British Machine Vision Association, 2016.
[60] C. ZHANG, S. BENGIO, M. HARDT, B. RECHT, AND O. VINYALS, Understanding deep learning (still) requires rethinking generalization, Communications of the ACM, 64 (2021),
pp. 107-115.
[61] H. ZHANG, Y. N. DAUPHIN, AND T. MA, Fixup initialization: Residual learning without normalization, in International Conference on Learning Representations, 2018.
[62] M. ZHOU, R. GE, AND C. JIN, A local convergence theory for mildly over-parameterized Iwo-layer neural network, in COLT, 2021.
[63] D. Zou, Y. CAO, D. ZHOU, AND Q. GU, Gradient descent optimizes over-parameterized deep ReLU networks, Machine Learning, 109 (2020), pp. 467-492.
[64] D. Zou, M. LONG, AND Q. GU, On the global convergence of training deep linear resnets, in International Conference on Learning Representations, 2019.
13


Reviews:
[{"review_id": "e-_1BRk1kJX", "reviewer": "Reviewer_jSRi", "summary": "This paper provided a local Polyak-Lojasiewicz inequality for a kernelized neural ODE-model. This paper's analysis is quite interesting, but still relationship between the proposed work and the previous should be improved. ", "questions": "I guess the authors need to discuss the realtionship between [1], which provided the convergence properties of Gradient Descent for linear ResNets. Can your analysis be considered as iinfiinte dimension analysis of [1]?\n[1] M. HARDT AND T. MA, Identity Matters in Deep Learning, arXiv:1611.04231 [cs, stat] (2018). arXiv: 1611.04231.\n\nWhat's the relationship between your analysis and the Resnet's NTK (which can be written down as an ODE). What's the further degree of freedom increased in your paper. At the same time, this paper is missing reference [2]. I'm curious about the relationship between your paper with [2]. Can the analysis in [2] be considered as a PL analysis?\n \n[2] Ding Z, Chen S, Li Q, et al. On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime[J]. arXiv preprint arXiv:2110.02926, 2021.\n\nThe assumption 2 is the core assumption to imply the PL condition. Can the analysis in line 256-258 be more concrete?  In there a principled way to construct such RKHS?", "limitations": "Thi limitation of this paper is two fold in my mind:\n- i don't know how strong is assumption 2\n- the hidden weight is not trained. [ but i think this model is still more powerful than NTK model.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper provided a local Polyak-Lojasiewicz inequality for a kernelized neural ODE-model. This paper's analysis is quite interesting, but still the relationship between the proposed work and the previous should be improved. \n\n\nThis paper present a nice Polyak-Lojasiewicz property for the Neural ODE model. I think this paper definitely will make impact in this area. I'm willing to upgrade my score if the paper can make the relationship between the proposed work and the previous more clear.  (detailed points see  Questions)\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WC4oML3XwH", "reviewer": "Reviewer_u7jP", "summary": "This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.", "questions": "This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.", "limitations": "This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.", "rating": 1, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper violates the NeurIPS submission rules, by putting supplementary materials (page 14-31) into the main paper submission, and will not be reviewed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5WVgiWu93uM", "reviewer": "Reviewer_xZax", "summary": "This paper studies the convergence of continuous-time residual networks with linear parameterization in the residual block trained by GD. Due to linear parameterization, the residual block functions are understood to be functions from an RKHS. Gradient of the loss function wrt the parameters (in this case a family of RKHS functions parameterized by time) is derived using standard techniques in optimal control. Then, a functional PL condition is proved for the gradient, which leads to local convergence. To achieve global convergence, two approaches---dimensionality lifting and random Fourier feature approximation---are studied for a special kernel. Final theorem says \"global\" convergence can be achieved when the network takes random Fourier features, is sufficiently wide, the matrices A and B are appropriately chosen, and the parameter is initialized from 0. ", "questions": "1. To the reviewer's understanding, \"global convergence\" in optimization means the algorithm converges from any initialization, which does not need to be close to the minimum. However, in section 5 of the paper, convergence to global minimum is proven for a specific initialization, v^0=0. This is not global convergence. Is there a way to improve the results to real global convergence, i.e. the GD converges to global minimum from any (reasonable) initialization? If not the statements in the paper might need to be changed.", "limitations": "The limitations have been addressed in the paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: The novel parts of this work lie in the functional PL condition for the loss of the model studies, as well as the the approaches to extend local convergence to global convergence in section 5. Though, the reviewer has question regarding the \"global convergence\" which is left for the questions part. The continuous time residual networks with linear parameterization has been explored in previous works, e.g. in [1]. The local convergence results based on PL condition is also not novel to the current work. \n\nQuality and clarity: The paper is clear well organized.\n\nSignificance: The analysis is theoretical significant because it reveals the good loss landscape for one type of flow-based model. The flow-based model is theoretically interesting because they are infinite-depth limit of deep neural networks, especially ResNets. It is also practically important, with neural ODE as a representative application. \n\n\n[1] E W, Ma C, Wu L. Machine learning from a continuous viewpoint, I. Science China Mathematics. 2020 Nov;63(11):2233-66.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept