Implicit regularization of deep residual networks towards neural ODEs
Authors: Pierre Marion
Yu-Han Wu
Michael Eli Sander
Gérard Biau
Venue: ICLR 2024 spotlight (2024)
Keywords: deep learning theory
residual networks
neural ODEs
optimization
implicit regularization
gradient flow
Primary area: learning theory

Abstract: Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog, neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous models still lacks a solid mathematical foundation. In this article, we take a step in this direction by establishing an implicit regularization of deep residual networks towards neural ODEs, for nonlinear networks trained with gradient flow. We prove that if the network is initialized as a discretization of a neural ODE, then such a discretization holds throughout training. Our results are valid for a finite training time, and also as the training time tends to infinity provided that the network satisfies a Polyak-Łojasiewicz condition. Importantly, this condition holds for a family of residual networks where the residuals are two-layer perceptrons with an overparameterization in width that is only linear, and implies the convergence of gradient flow to a global minimum. Numerical experiments illustrate our results.

Published as a conference paper at ICLR 2024
IMPLICIT REGULARIZATION OF DEEP RESIDUAL NET- WORKS TOWARDS NEURAL ODES
Pierre Marion, Yu-Han Wu'
Michael E. Sander
Gérard Biau
LPSM
DMA
LPSM
Sorbonne Université, CNRS Paris, France
ENS, CNRS
Sorbonne Université, CNRS
Paris, France
Paris, France
ABSTRACT
Residual neural networks are state-of-the-art deep learning models. Their continuous-depth analog. neural ordinary differential equations (ODEs), are also widely used. Despite their success, the link between the discrete and continuous models still lacks solid mathematical foundation. In this article, we take a step in this direction by establishing implicit regularization of deep residual networks towards neural ODEs, for nonlinear networks trained with gradient flow. We prove that if the network is initialized discretization of neural ODE, then such a discretization holds throughout training. Our results are valid for a finite training time, and also the training time tends to infinity provided that the network satisfies a family of residual networks where the residuals are two-layer perceptrons with an overparameterization in width that is only linear, and implies the convergence of gradient flow to a global minimum. Numerical experiments illustrate our results.
condition. Importantly, this condition holds for a
INTRODUCTION
Residual networks are a successful family of deep learning models popularized by breakthrough results in computer vision (He al., 2016b). 'he key idea behind residual networks, namely the presence of skip connections, is now ubiquitous in deep learning, and can be found, for example, Transformer models (Vaswani et al., 2017). The main advantage of skip connections is allow successful training with depth of the order of thousand layers, in contrast to vanilla neural networks, leading to significant performance improvement (e.g.. Wang et al., 2022). This has motivated research on the properties of residual networks in the limit where the depth tends to infinity. One of the main explored directions is the neural ordinary differential equation (ODE) limit (Chen al., 2018).
Before presenting neural ODEs, we first introduce the mathematical formalism of deep residual networks. We consider a single model throughout the paper to simplify the exposition, but most of our results apply to more general models, as will be discussed later. We consider the formulation
1 h = hy + L V Va+1o ( Watthhs): E L-
where is the depth of the network, h2 E IR is the output of the k-th hidden layer, V/8 E R W₂ E m are the weights of the k-th layer, and o is activation function applied clement-wise. Scaling with the square root of the width classical, although it often appears as an equivalent condition the variance at initialization (Glorot & Bengio, 2010; LeCun al., 2012; He et al., 2015). We make the scaling factors explicit to have weights of magnitude (11) independently of the width and the depth. The L scaling factor is less common, but is necessary for the correspondence with neural ODEs to hold. More precisely, if there exist Lipschitz continuous functions V and W such that V₂= = VkkLL) and W₁ W(k/ L), then the residual network (1) converges, as L oo, to the ODE
d.H. (s)
1 m
VisJo W(s)H(s) s e [0, 1].
(2)
ds
"Equal contribution. Correspondence pi eirre.. m on marionemi org.

---

Published as a conference paper at ICLR 2024
where is the continuous-depttl version of the layer index. is important to note that this correspon- dence holds for fixed limiting functions V and W. This is especially true at initialization, for example by setting the VE to zero and the Wi to Gaussian matrices weight-tied across the depth. The initial as (s) = O. Of course, more sophisticated initializations are possible, shown, e.g., in Marion et al. (2022); Sander al. (2022b). However, regardless of an ODE structure at initialization, a more challenging question is that of the structure of the network during and after training. Since the weights are updated during training, there is a priori no guarantee that an ODE limit still holds after training, even if it does at initialization.
residual network is then trivially equal to the neural ODE dH
The question of a potential ODE structure for the trained network is not a mere technical one. In fact, it is important for at least three reasons. First, it gives a precise answer to the question of the connection between (trained) residual networks and neural ODEs, providing more solid ground to a common statement in the community that both can coincide in the large-depth limit (see, e.g., Haber & Ruthotto, 2017; et al., 2019; Dong et al., 2020; Massaroli et al., 2020; Kidger, 2022). Second, opens exciting perspectives for understanding residual networks. Indeed, if trained residual networks are discretizations of neural ODEs, then it is possible to apply results from neural ODEs to the large family of residual networks. In particular, from theoretical point of view, the approximation capabilities of neural ODEs are well understood (Teshima et al., 2020; Zhang et al., 2020) and it is relatively easy to obtain generalization bounds for these models (Hanson & Raginsky, 2022; Marion, training (Chen al., 2018; Sander et al., 2022b) and weight compression (Queiruga et al., 2021). This is important because in practice memory is a bottleneck for training residual networks (Gomez et al., 2017). Finally, our analysis a first step towards understanding the implicit regularization (Neyshabur et al., 2014; Vardi, 2023) of gradient descent for deep residual networks, that is, characterizing the properties of the trained network among all minimizers of the empirical risk.
2023). From a practical standpoint, advantages of neural ODEs include memory-efficien
Throughout the document, it is assumed that the network is trained with gradient flow, which is a continuous analog of gradient descent. The parameters Vi are updated according to an ODE of the I. 2,,
de (t) assumptions are detailed in Section 3), and similarly for Wi. The scaling factor L is the counterpart of the factor 1/L LLL in (I), and prevents vanishing gradients as L tends to infinity. Note that the gradient flow is defined with respect to a time index different from the layer index 8.
form VV22
at (t) for > O, where l is an empirical risk (the exact mathematical context and
Contributions. Our first main contribution (Section 4.1) is to show that neural ODE limit holds after training up to time I, i.e., there exists a function (ss, () such that the residual network converges, as T tends to infinity, to the ODE
d.l
1 V(s, 1)s ds (8) = mm
Va ( Wo.
E [0,
This large-depth limit holds for any finite training time t > 0. However, the convergence of the optimization algorithm tends to infinity, which refer to the tong-time limit to distinguish it from the large-depth limit T. - co, is not guaranteed without further assumptions, due to the non-convexity of the optimization problem. We attack the question (Section 4.2) when the width is large enough by proving a folyak-E.ojssewwic. (PL.) condition, which is now state of the art in analyzing the properties of optimization algorithms for deep neural networks (Liu al., 2022). The main assumption for our PL condition to hold is that the width m of the hidden layers should be greater than some constant times the number of data n. As second main contribution, we show that the PL condition yields the long-ting time convergence of the gradient flow for residual networks with linear overparameterization. Finally, we prove the convergence with high probability the long-time limit, namely the existence of functions V>- and W such that the discrete trajectory defined by the trained residual network (1) converges as both L and t tend to infinity to the solution of the neural ODE (2) with V V.o and W = W... In addition, our approach points out that this limiting ODE interpolates the training data. Finally, our results are illustrated by numerical experiments (Section 5).
2 RELATED WORK
Deep residual networks and neural ODEs. Several works study the large- convergence of residual networks to differential equations, but without considering the training dynamics (Thorpe & van Gennip, 2023; Cohen et al., 2021; Marion et al., 2022; Hayou, 2023). Closer to our setting, Cont
2

---

Published as a conference paper at ICLR 2024
et al. (2022) and Sander et al. (2022b) analyze the dynamies of gradient descent for deep residual networks, as we do, but with significant differences. Cont et al. (2022) consider a l/VZ scaling factor in front of the residual branch, resulting in limit that is not neural ODE. In addition, only W is trained. Furthermore, to obtain convergence in the long-time limit, it is assumed that the data points nearly orthogonal. Sander al. (2022b) prove the existence of ODE limit for trained residual networks, but in the simplified case of a linear activation and under a more restricted setting.
Long-time convergence of wide residual networks. tool to prove long-time convergence of overparameterized neural networks (Liu et al., 2022). These conditions are a relaxation of convexity, and mean that the gradients of the loss with respect to the parameters cannot be small when the loss is large. They have been applied to residual networks with both linear (Bartlett et al., 2018; Wu et al., 2019; Zou et al., 2020) and nonlinear activations (Allen-Zhu et al., 2019; Frei al., 2019; Barboni et al., 2022; Cont al., 2022: Macadn et al., 2022). Building on the proof technique of Nguyen & Mondelli (2020) for non-residual networks, we need only linear overparameterization compares favorably with results requiring polynomial overparameterization (Allen-Zhe -Zh et al., 2019; Barboni et al., 2022) or assumptions on the data, either margin condition (Frei et al., 2019) or sample size smaller than the dimension of the data space (Cont et al., 2022: MacDonald et al., 2022).
Lojjsk-eojssiwiiz conditions are a modern
prove our PL condition, i.e., require m = 9(().. This
Implicit regularization. Our paper can be related to a line of work on the implicit regularization of gradient-based algorithms for residual networks (Neyshabur et al., 2014). We show that the optimization algorithm does not just converge to any residual network that minimizes the empirical risk, but rather the discretization of neural ODE. Note that most implicit regularization results state that the optimization algorithm converges to an interpolator that minimizes some complexity measure, which can be margin (Lyu Li, 2020), a norm (Boursier al., 2022), or a matrix rank (Li et al., 2021). Thus, an interesting next step is to understand if the neural ODE found by gradient flow actually minimizes some complexity measure, and to characterize its generalization properties.
DEFINITIONS AND NOTATION
This section devoted to specifying the setup outlined in Section 1. Proofs are given in the appendix.
Residual network.
A (scaled) residual network of depth E N* is defined by
h/ A A = 1ife + I.
Vi4 W 1h). E IO 1}, /q
L.
FL. F=(x)= = BL
To allow the hidden layers of E IR to have a different dimension than the input x E Rd, we first map x to hg with weight matrix AL E Raxd, We assume that the hidden layers belong to a higher dimensional space than the input and output, i.e., q > max(d, d'). The residual transformations are two-layer perceptrons parameterized by the weight matrices V.-- E R and W. E Rmxxq This is standard in the literature (e.g.. He et al., 2016a; Dupont al., 2019; Barboni et al., 2022). The last weight matrix B² R.dxx x9 maps the last hidden layer the output F'(x)) in Idd Also, o R RR is an element-wise activation function assumed to be C² non-constant, Lipschitz continuous, bounded, and such that o (0) = O. The convenient shorthand 2 = (V,, W is occasionally used, and we denote || | || the sum of the Frobenius norms | r + |[ l|..
Data and loss. The data is sample of pairs (z,, W)isign (X' x )))" where X x y is compact set of TRd x ddd The empirical risk is the mean squared error e = 1 El [F vill?.
Initialization. We initialize A = (Jzaxa, Opti axxa) as the identity matrix in Rdxd concatenated row-wise with the zero matrix in R( to act as a simple projection of the input onto the higher dimensional space R8, and similarly BL (Ogarxig-d")) Ig="xar). The weights V. are initialized to zero and the 7 as weight-tied standard Gaussian matrices, i.e., for all l: E (I,..., D).
3

---

Published as a conference paper at ICLR 2024
W W ~ (N 1) 1) Initializing outer matrices zero is standard practice (Zhang et al., 2019), while taking weight-tied matrices instead of i.id. ones less common. We show Section that it is still possible to learn with this initialization scheme on real world data. As explained in Section 4.3, other initialization choices are possible, provided they correspond to the discretization of Lipschitz continuous function, but focus this one in the main text for simplicity.
Training algorithm. Gradient flow is the limit of gradient descent the learning rate tends to zero. The parameters are set at time = 0 by the initialization, and then evolve according to the ODE
dA= dt (1) OAE (4),
deL
dzk dt (1)= -1, 06L
dBL (1)
ael
-0% (1),
dt
aBt
DBL
(4)
t 1 0,
(4)
for l E {I....,}} In the following, the dependence in is made explicit when necessary, e.g., we write hf((1) instead of lig, and (=;; t) instead of FL(x).
turns out that, without further assumptions, the gradient flow can diverge in finite time, because the dynamics are not (globally) Lipschitz continuous. A common practice (Goodfellow et al., 2016, Section 10. 10.1 1) to consider instead a clipped gradient flow
d di (1) (1)==(--
(- dEL
di (1) - ( OeL K (1): (1
dik
dB² di (1) (1== = JBL - ()), (5)
deL
(1). -
where is a generic notation for bounded Lipschitz continuous operator. For example, clipping each coordinate of the gradient at some >> amounts to taking т as the projection on the ball centered at 0 of radius C' for the lo norm. Clipping ensures that the gradient flow does not diverge, hence the dynamics are well defined, as a consequence of the Fcard-Lindelüö theorem (see Lemma 16).
Proposition The (clipped) gradient flow (5) has a unique solution all > O.
In Section 4.2, we make additional assumptions to prove the long-time convergence of the gradient flow. We then prove that these assumptions ensure that the dynamics of the gradient flow (4) are well defined, eliminating the need for clipping (since in this case we show that the gradients are bounded).
Neural ODE. The neural ODE corresponding the residual network (3) is defined by
H()) Ax 1 Vm F(x) BH(1),
dH ds (s) v(s)o 1 W(s)H(()) 8 E [0. 1], vq
(6)
where x E Rd is the input, E R& the variable of the ODE, V: [0, 1] + R2 and W: [0, 1] + Rmxy are Lipschitz continuous functions, A E Rexd and B E TR' xq are matrices, and the output is F(x) E I²". The following proposition shows that the neural ODE is well defined. In addition, its discretizations of V and W.
output close to the residual network (3) provided the weights
Proposition 2. The neural ODE (6) has unique solution H [0, 1]
+ RR, Consider, moreover, the residual network (3) with A A, V2 = V(k/ L) and W = ((k/ for k E (1,..., L}. and B²_ = B. Then there exists C > O such that, for all L N supzex [[F(()) FF
Clearly, our choices of V and W initialization are discretizations of the Lipschitz continuous (in fact, constant) functions ((s) and W(s) W NN(0,, 1) (m)x)) Thus, Proposition holds at initialization, and the residual network equivalent the trivial ODE did (s) 0. The next section shows that after training we obtain non-trivial dynamics, which still discretize neural ODEs.
4 LARGE-DEPTH LIMIT OF RESIDUAL NETWORKS
We study the large-depth limit of trained residual networks in two settings. In Section 4.1, consider the case of a finite training time. We move in Section 4.2 to the case where the training time tends to infinity, which is tractable under Polyak--.ojssewwicz condition. Proofs given in the appendix.
4

---

Published as a conference paper at ICLR 2024
4.1 CLIPPED GRADIENT FLOW AND FINITE TRAINING TIME
We first consider the case where the neural network is trained with clipped gradient flow (5) on some neural
training time interval [0, T T > This allows us prove large-depth convergence ODE without further assumptions. We emphasize that stopping training after a finite training time is a common technique in practice, referred to as early stopping (Goodfellow et al., 2016, Section 7.8). It is considered as a form of implicit regularization, and our result sheds light on this intuition by showing that the complexity of the trained networks increases with T.
The following proposition is key step in proving the main theorem of this section.
Proposition 3. There exist M, K > O such that, for any t E [0, E N and E {1,.... ,.....,).. L),
max [| (t) (1 (t)l)) < M,
and, for ls E (1.....,LL 1-
K L
max -V
VV
Moreover. with probability at least 1 exp (- 3qm ). the following expressions hold for M and K:
16)
M = TM + 2//qm, K = 3TT""
where Mg is the supremum of s in Frobenius norm, and a and 3 depend on X, y, M, and o.
This proposition ensures that the size of the weights and the difference between successive weights remain bounded throughout training. We can now state the main result, which states the convergence, for any training time in [0, T). of the neural network to neural ODE L + 20. Recall that sequence of functions f converges uniformly over u E U to f if supueu [lf - f(u)ll - 0.
Theorem 4. Consider the residual network (3) with the training dynamics (5). Then the following statements hold as L tends to infinity:
(i) There exist functions A [0,T] Roxd and B [0. T (t) converge uniformly over t E [O.T] to A(t) and B(t).
Rd such that ( and
(ii) There exists Lipschitz continuous functionn 2 : [0, 1] x [0,7] + 12 x R2 such that
zト [ x 0. T) R2xm Rmxg (s, t) + (s, (1) (8)
converges uniformly over E [0 and E [0, to 2 (V,
(iii) Uniformly over s [0, 11, E T), and x X, the hidden layer lot / (1) converges the solution time s of the neural ODE
(10 t) A(t)z
0H
1
(9)
Os t) = _
v(s,t)o
Jor ((s.t)H(sstt) s sE 10,1].
mm
(iv) Uniformly over t [0,T] and x E X, the output Fl converges to (1,)
Let us sketch the proof of statement (ti), which is the cornerstone of the theorem. A first key idea is introduce in (8) the piecewise-constane continuous- depth interpolation Z of the weights, whose ambient space does not depend on I., in contrast to the discrete weight sequence z Since the weights remain bounded during training by Proposition 3, the Arzelà-Ascoli theorem guarantees the existence of an accumulation point for Z We show that the accumulation point is unique because it is the solution of an ODE satisfying the conditions of the Ficard-Lindell theorem. The uniqueness of the accumulation point then implies the existence of limit for the weights
There are two notable byproducts our proof. The first one is an explicit description of the training dynamics of the limiting weights A, B, and 2, the solution of an ODE system, presented in Appendix A.5. The second one, which we now describe, consists of norm bounds on the weights. Proposition bounds the discrete weights and the difference between two consecutive weights
5

---

Published as a conference paper at ICLR 2024
respectively by some M, K > O. The proof of Theorem shows that this bound carries to the continuous weights, in the sense that A(t). ((s, t). W(s, t), and (t) are uniformly bounded by M, and V) and W(-, t) are uniformly Lipschitz continuous with Lipschitz constant K. Formally, this last property means that, for any 8, s' E [0, 1] and E [0,7] 77,
[| t) tll & K|s'
and [ W(s, < K|s' s|.
A key point to obtain this result is that K and M in Proposition 3 are independent of L. This would not be the case if we had naively bounded in Proposition the difference between two successive weight matrices by constant, without taking into account the smoothness of the weights. The boundedness and Lipschitz continuity of the weights are important features because they limit the statistical complexity of neural ODEs (Marion, 2023). More generally, norm-based bounds a common approach in the statistical theory of deep learning (see, e.g.. Bartlett et al., 2017, 7, and references therein). Looking at the formula (7) for M and K, one can diverge exponentially with T. providing an argument in favor of early stopping.
particular that the bounds
Our approach so far characterizes the large-depth limit of the neural network for finite training time T, but two questions remain open. A first challenge is to characterize the value of the loss after training. A second one the long-time limit, i.e., as 7 tends to infinity. To answer these questions, we move to the setting where the width of the network is large enough, which allows us to prove Dolyak-ELojssiwiie (PL. condition and thereby the long-time convergence of the training loss to zero.
provide insight into the convergence of the optimization algorithm in
4.2 CONVERGENCE IN THE LONG-TIME LIMIT FOR WIDE NETWORKS
We now introduce the definition (with the notation Z (VE, ) we (I.....))))))))))))))))))))( of the PL condition: Definition 1. For M, > O, the residual network (3) is said to satisfy the (M, p)-local PL condition around set of parameters (A4, ,BB)) for every set of parameters (AL, -- B²) such that
|| 14 & M,
1
2/11 < M,
| - B² & M,
sup (E.....
one has
de 2 DA
deL Oz
deT. 1/ +
+LE
+
aBL 1|; > ul
k=1
where the loss th is evaluated at the set of parameters (A 2., B²).
The important point is to observe that, under the setup of Section 3 and some additional assumptions, the residual network satisfies the local PL condition of Definition 1.
Proposition Assume that the sample points (Is, ye) are i.i.d. such that ||ælll = /4. Then there exist C1, ..... C1 > 0 (depending only σ) and 8 0 such that, if
7 + d',
m M c
L 3 02 /ug,
then, with probability at least 8, the residual network (3) satisfies the (M, pl-local PL condition around its initialization, with M = ca/ /nq and ca/ ca/(nnnq)
We emphasize that Proposition 5 requires the width m to scale only linearly with the sample size n, which improves on the literature (see Section 2). The other assumptions are mild. Note that our proof shows that the parameter ó is small if n grows at most polynomially with d (see Appendix B.5).
We are now ready to state convergence in the long-tim onng-time and large-deptl limits to global minimum of the empirical risk, when the local PL condition holds and the norm of the targets yi is small enough.
Theorem 6. Consider the residual network (3) with the training dynamics (4), and assume that the assumptions of Proposition 5 hold. Then there exist C, 8 > O such that, if 1 >1 ||y:ll < C, then, with probability as least ô, the gradient flow is well defined on R4, and, for L E R4 and N
C't C't )L
(²) exp
(
(10)
for some C' > depending on a. Moreover, the following statements hold as I. and L. tend infinity:

---

Published as a conference paper at ICLR 2024
(i) There exist matrices Axe E Raxd and Bx Rdd xa such that (t) and B converge Axe and Bx
(ii) There exists a Lipschitz continuous function z0 10, 1] Ruxm x R such that
:0.1 R+ + R x RR x9 (s.t) (s. (11-))44+100
converges uniformly over E [0, 1] 3 (Vox Woo)-
(ii)) Uniformly over 8 [0, 1] and E X, the hidden layer n LLog converges to the solution at time of the neural ODE
(0) A
dH
1 ds (s) vm
(o) (s)()) S E 10.
Vq
(iv) Uniformly over x E X, the output F (x; converges Fxo B (1). Furthermore, Foo (z,) ys for all E (1....,n}.
This theorem proves two important results of separate interest. On the one hand, equation (10) shows the long-time convergence of the gradient flow for deep residual networks under the linear overparameterization assumption 2 Ci of Proposition 5. On the other hand, when both and L tend to infinity, the network converges to a neural ODE that further interpolates the training data. Note that the order which & and L tend infinity does not matter by uniform convergence properties.
4.3 GENERALIZATIONS OTHER ARCHITECTURES AND INITIALIZATION
To simplify the exposition, we have so far considered a particular residual architecture defined in (3). However, most of our results hold for a more general residual network of the form
h h 1 s Z E {0. ........ L 1}.
(11)
where f: R9 x RP R2 is a C2 function such that f(0.-) 0 and f(,, =) is uniformly Lipschitz for z in any compact. All our results are shown in the appendix for this general model, except the PL condition of Proposition 5, which we prove only for the specific setup of Section 3. In particular, the conclusions of Theorem 4 hold for the general model (11), as well as those of Theorem 6 if the network satisfies (M, loocll PL condition with j sufficiently large (see Appendix B for details).
Our network of interest (3) is special case of model (11), and other choices include convolutional layers (or any sparse version of (3)) or Lipschitz continuous version of Transformer (Kim et al., 2021). This latter case is particularly interesting in the light of the literature analyzing Transformer from a neural ODE point of view (Luu al., 2019; Sander et al., 2022a; Geshkovski et al., 2023).
Moreover, the initialization assumption made in Section 3 can also be relaxed to include any so-called smooth initialization of the weights (Marion et al., 2022). A smooth initialization corresponds to taking V (0) and (0) discretizations of some Lipschitz continuous functions Vo= [0. 1] - R and Wo [o, 1] + R=*g x that is, for & {1, ... L L). Vz-(0) Vo # and ( Wo ( ) typical concrete example to let the entries of Vo and Wo be independent Gaussian processes with
expectation zero and squared exponential covariance K (2r, x¹) exp(- 27 exp(-(==p"i) for some e 0. As shown by Proposition 2, a smooth initialization means that the network discretizes a neural ODE.
5 NUMERICAL EXPERIMENTS
We now present numerical experiments to validate our theoretical findings, using both synthetic and real-world data. Our code is available on GitHub (see Appendix E for details and additional plot).
5.1 SYNTHETIC DATA
We consider the residual network (3) with the initialization scheme of Section 3. function is GELU (Hendrycks & Gimpel, 2016), which is smooth approximation of ReI.U:
The activation
7

---

Published as a conference paper at ICLR 2024
max(2, 0). The sample points (z,, Islin follow independent standard Gaussian distributions. The mean-squared error is minimized using full- batch gradient descent. The following experiments exemplify the large-depth (). L oc) and long-time + 00, L finite) limits.
Training iteration
(1,,), V(.
10
50
4
100
8
150
16
10-2
10-3
109
200
32
250
64
10-5
10
10-1
512
1024
450
250
500
500
10
10a
10a
Training iteration
Training iteration
Depth L
Figure I: Left: 1/L convergence of the maximum distance between two successive weight matrices ro x (f)lle). Right: uniform convergence of 2 to its large-depth limit Z. Here, for a matrix-valued function f, |lf denotes So [I((s))]
Large-depth limit. We illustrate key insights of Proposition 3 and Theorem 4, with 7 500. In Figure 1 (left), we plot the maximum distance between two successive weight matrices, i... max (1)
- (t) for different values of and training time T'. We observe a 1/L convergence rate, as predicted by Proposition 3. Moreover, for fixed L, the distance between two successive weight matrices increases with the training time, however at much slower pace than the exponential upper bound on K given in identity (7). Figure (right) depicts the uniform convergence of z to its large-depth limit Z, illustrating statement (ži) of Theorem 4. The function 2 is computed using 2 for I, 214, Note that the convergence is slower for larger training times.
Long-time limit. We now turn to the long-time training setup, training for 80,000 iterations with 64 and m large enough to satisfy the assumptions of Theorem 6. In Figure 2, we plot a specific (randomly-chosee) entry of matrices V. and W across layers, for different training times. This illustrates Theorem 6 in practical setting since, visually, the weights behave Lipschitz continuous function for any training time and converge to a Lipschitz continuous function The loss decays to zero function of training time, also corroborating Theorem 6.
co.
coeft. of (0
coeff. W2(t)
0.60
10
0.0
10-4
0.55
10-7
50000
20
40
60
Layer
Layer &
Training iteration
Figure 2: Left: Randomly-chosen entry of the weight matrices across layers (z-axis) for various training times (lighter color indicates higher training time). Right: Loss against training time.
5.2 REAL-WOL REAL-WORLD DATA
We now investigate the properties of deep residual networks the CIFAR 10 dataset (Krizhevsky, 2009). We deviate from the mathematical model (3) by using convolutions instead of fully connected layers. More precisely, A is replaced by trainable convolutional layer, and the residual layers write h 2 b (o(bn,,
where conv are convolutions and bnl, are batch normalizations (see Appendix E for discussion about normalization). The output of the residual layers is mapped to logits through linear layer B4, We initialize bng 0, and but
8

---

Published as a conference paper at ICLR 2024
and convi, either to weight-tied to ii.d. Gaussian. Table reports the accuracy of the trained network, and whether has Lipschitz continuous (or smooth) weights after training, depending on the activation function and on the initialization scheme. To assess the smoothness of the weights, we simply resort to visual inspection. For example, Figure (left) shows two random entries of the convolutions across layers with GELU and weight-tied initialization: the smoothness is preserved after training. Smooth weights indicate that the residual network discretizes a neural ODE (see, c.g., Proposition 2). On the contrary, if i.i.d. initialization is used, smoothness is not preserved after training. as shown in Figure 3 (right), and the residual network does not discretize neural ODE.
0.050
A coeff. of convi
A coeff. of convl
0.05
0.025
0.000
0.00
0.025
-0.05
50
150
200
250
50
100
150
200
250
Layer
Layer &
Figure 3: Random entries of the convolutions across layers (æ-axis) after training. Left: Weight-tied initialization leads to smooth weights. Right: i.i.d. initialization leads non-smooth weights.
Act. function
Init. scheme
Train Acc.
Test Acc.
Smooth trained weights
Weight-tied i.i.d.
56.5 + 0.1
59.8 + 0.7
Identity
56.1 0.3
59.6 + 0.7
Weight-tied
80.5 + 0.7
79.9 + 0.2
GELU
i..d.
89.8 0.5
85.7 + 0.1
X
Weight-tied
97.4 + 0.6
88.1 + 0.1
X
ReLU
i.i.d.
98.4 0.1
88.4 + 0.5
Table 1: Accuracy and smoothness of the trained weights depending the choice of activation function a and initialization scheme. We display the median over runs and the interquartile range between the first and third quantile. Smooth weights correspond to neural ODE structure.
Table conveys several important messages. First, in accordance with our theory (Theorem 4), we obtain neural ODE structure when using smooth activation function and weight-tied initialization (lines and 3 of Table 1). This is not the case when using the non-smooth ReLU activation and/or i..d. initialization. training with ReLU in a simple setting. Furthermore, the third line of Table I shows that is possible obtain a reasonable accuracy with a neural ODE structure, which, as emphasized in Section also comes with theoretical and practical advantages. Nevertheless, we see an improvement in accuracy in cases corresponding to non-smooth weights, i.e., to a network that does not discretize an ODE.
fact, we prove in Appendix D that the smoothness of the weights is lost when
CONCLUSION
We study the convergence of deep residual networks to neural ODEs. When properly sealed and initialized, residual networks trained with fixed-horizon gradient flow converge to neural ODEs the depth tends to infinity. This result holds for very general architectures. In the case where both training time and depth tend to infinity, convergence holds under local Lojjik--jasiiiiizzz condition. We prove such a condition for a family of deep residual networks with linear overparameterization
The setting of neural ODE-like networks comes with strong guarantees, the cost of some per- formance gap when compared with i.idd initialization as highlighted by the experimental section. Extending the mathematical large-depth study i.i.d. instead of weight-tied initialization is an interesting problem for future research. Previous work suggests that the correct limit object is then a stochastic differential equation (Cohen al., 2021; Cont al., 2022; Marion et al., 2022).

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
P.M. is supported by grant from Région fle-de-France and by MINES Paris PSL. P.M. and Y.-H.W. W. are funded by Google PhD Fellowship. M.S. is supported by the "Investissements d'avenir" program, reference ANRI9-P3IA-000 project NORIA). This work was granted access to the HPC resources of IDRIS under the allocation 2020-[AD01 1012073] made by GENCI.
and by the European Research Council (ERC
REFERENCES
Zeyuan Allen-Zhie Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via overrparameetriizaionn. In K. Chaudhuri and R. Salakhutdinov (eds.). Proceedingy of the 36th International Conference on Machine Learning, volume 97, pp. 242 252. PMLR, 2019.
Vladimir I. Arnold. Ordinary Differential Equationn. Springer, Berlin, 1992.
Raphaël Barboni, Gabriel Peyré. and François-Xavier Vialard. On global convergence of ResNets: From finite to infinite width using linear parameterization. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Nearal Information Processing Systems, volume 35, pp. 16385-16397. Curran Associates, Inc., 2022.
Peter L. Bartlett, Dylan J. Foster, and Matus J. Telgarsky. opectrally-normmalie margin bounds for neural networks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 6240- 6249. Curran Associates, Inc., 2017.
Peter L.. Bartlett, Dave P. Helmbold, and Philip M. Long. Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In J. Dy and A. Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. volume 80, pp. 521-530. PMLR, 2018
Etienne Boursier, Loucas Pillaud- Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20105-200 18. Curran Associates, Inc., 2022.
Ricky T.Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K. Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biancha and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 6572-6583, Curran Associates, Inc., 2018
Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks. In M. Meila and T. Zhang (eds.), Proceedings the 38th International Conference Machine Learning, volume 139, pp. 2039-2048. PMLR, 2021.
Rama Cont, Alain Rossier, and Ren Yuan Xu. Convergence and implicit regularization properties of gradient descent for deep residual networks. 07X16:2204.07261, 2022.
S. De and S. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19964-19975. Curran Associates, Inc., 2020.
Lokenath Debnath and Dambaru Bhatta. Integral Transforms and Their Applications. CRC press, Boca Raton, third edition, 2014.
Chengyu Dong, Liyuan Liu, Zichao Li, and Jingbo Shang. Towards adaptive residual network training: A neural-ODE perspective. In H. Daumé III and A. Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 2616-2626. PMLR, 2020.
Sever S. Dragomir. Some Gronwall Type Inequalities and Applications. Nova Science Publishers, 2003.
10

---

Published as a conference paper at ICLR 2024
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural ODEs. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 3140-3111 40-3150. Curran Associates, Inc., 2019.
Weinan E, Jiequn Han, and Qianxiao Li. A mean-field field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6:10, 2019.
Aleksei F. Filippov. Differential Equctioons with Discontinuous Righthand Sides. Springer, Dordrecht, 1988.
Spencer Frei, Yuan Cao, and Quanquan Gu. Algorithm-dependeee generalization bounds for overpa- rameterized deep residual networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu -ucc E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 14769-14779. Curran Associates, Inc., 2019.
Borjan Geshkovski, Cyril Letrouit, Yury Polyanskiy, and Philippe Rigollet. The emergence of clusters in self-attention dynamics. arXiv:2305.0m 05465, 2023.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Y.W. Teh and M. Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9, pp. 249-256. PMLR, 2010.
Surbhi Goel, Aravind Gollakota, Zhihan Jin, Sushrut Karmalkar, and Adam Klivans. Superpolynomial lower bounds for learning one-layer neural networks using gradient descent. In H. Daumé III and A. Singh (eds.). Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 3587-3596. PMLR, 2020.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual network: Backpropagation without storing activations. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 221 2214-2224. Curran Associates, Inc., 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016
Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34: 014004, 2017.
Joshua Hanson and Maxim Raginsky. Fitting an immersed submanifold to data via Sussmann's 's orbit theorem. In 2022 IEEE 61st Conference on Decision and Control (CDC), pp. 5323-5328, 2022.
Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. Transactions on Machine Learning Research, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance ImageNet classification. In Proceedings 2015 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1026-1034. IEEF, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In B. Leibe, Matas, N. Sebe, and M. Welling (eds.), Computer Vision ECCV 2016, pp. 630-645. Springer International Publishing. 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings 2016 IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778. IEEE, 2016b.
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUSs). arXiv:1606.084/1 2016.
Patrick Kidger. On Neural Ordinary Differential Equations. PhD thesis, University of Oxford, 2022.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The Lipschitz constant of self-attention. In M. Meila and T. Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 5562-5571. PMLR, 2021.
D.P. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
11

---

Published as a conference paper at ICLR 2024
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Béatrice Laurent and Pascal Massart. Adaptive estimation of quadratic functional by model selection. The Annals of Statistics, 28:1302-1333 1338, 2000.
Yann A. LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Rober Müller. Efficient backprop. In G. Montavon, G.B Orr, and K.-R. Müller (eds.), Neural Networks: Tricks of the Trade: Second Edition, pp. 9-4,, Berlin, 2012. Springer.
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2021.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over- wer-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis, 59: 85-116, 2022.
Sergey Loyka. On singular value inequalities for the sum of two matrices. arXiv: 2015.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong. Tao Qin, Liwei Wang. and Tie-Yan Liu. Understanding and improving transformer from multi-particle dynamic system point of view. arXiv:1906.02762, 2019.
Jonathan Luk. Notes on existence and uniqueness theorems for ODEs, 2017. URL: http: /web. stanford edu/-juulllleee jukk/kkkk//u /ma h w 1tte7//xxxteeeneee 170408 .pdf.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020.
Lachlan E. MacDonald, Hemanth Saratchandran, Jack Valmadre, and Simon Lucey. A global analysis of global optimisation. arXiv:2210.05371,, 2022.
Pierre Marion. Generalization bounds for neural ordinary differential equations and deep residual networks. arXiv:2305.066445 2023.
Pierre Marion, Adeline Fermanian, Gérard Biau, and Jean-Philippe Vert. Scaling ResNets in the large-depth regime. arXiv:2206.06922 2022.
Stefano Massaroli, Michael Poli, Jinkyoo Park, Atsushi Yamashita, and Hajime Asama. Dissecting neural ODEs. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3952-3963. Curran Associates, Inc.,
2020.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv:1412.6614, 2014.
Quynh N. Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems, volume 33, pp. 11961-11972. Curran Associates, Inc., 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 8024-8035. Curran Associates, Inc., 2019.
deep learning library.
Alejandro Queiruga, N. Benjamin Erichson, Liam Hodgkinson, and Michael W. Mahoney. Stateful ODE-nets using basis function expansions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21770-21781. Curran Associates, Inc., 2021.
12

---

Published as a conference paper at ICLR 2024
Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transformers with doubly stochastic attention. In G. Camps- Hamps-Valls, F.J.R. Ruiz, and I. Valera (eds.), Proceedings of The 25th International Conference Artificial Intelligence and Statistics, volume 151, pp. 3515-3530, PMLR, 2022a.
Michael E. Sander, Pierre Ablin, and Gabriel Peyré. Do residual neural networks discretize neural ordinary differential equations? In S. Koyejo. Mohamed, Agarwal, D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 36520-36532. Curran Associates, Inc., 2022b.
Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, Isao Ishikawa, and Kenta Oono. Universal approxi- mation property of neural ordinary differential equations. arXivv2012.022144 2020.
Matthew Thorpe and Yves Gennip. Deep limits of residual neural networks. Research in the Mathematical Sciences, 10:6, 2023.
Joel A. Tropp. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12:389 434, 2012.
Gal Vardi, On the implicit bias in deep-learning algorithms. Communications of the ACM, 66:86-93, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, 5998-6008. Curran Associates, Inc., 2017.
Roman Vershynin.
High- Digh-Diiensionl Probability: An Introduction with Applications in Data Science. Cambridge Series Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018
Hongyu Wang, Shuming Ma. Li Dong, Shaohan Huang. Dongdong Zhang, and Furu Wei. DeepNet: Scaling transformers to 1,000 layers. arXiv:2203.005:. 00555, 2022.
Lei Wu, Qingcan Wang, and Chao Ma. Global convergence of gradient descent for deep linear residual networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 13389-13399 Curran Associates, Inc., 2019.
Han Zhang, Xi Gao, Jacob Unterman, and Tom Arodz, Approximation capabilities neural ODEs and invertible residual networks. In H. Daumé III and A. Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 11086-11095. PMLR, 2020.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. In International Conference on Learning Representationss, 2019.
Difan Zou, Philip M. Long, and Quanquan Gu. On the global convergence of training deep linear ResNets. In International Conference on Learning Representations, 2020.
13

---

Published as a conference paper at ICLR 2024
Appendix
Organization of the Appendix. In Section A, give results the general residual network (II), In Section B, these results are instantiated in the specific case of the residual network (3), thus proving the results of the paper. Section C contains some lemmas that are useful for the proofs. We present in Section D counter-example showing that residual network with the ReLU activation can move away from the neural ODE structure during training. Finally, Section E presents some experimental details.
A SOME RESULTS FOR GENERAL RESIDUAL NETWORKS
Lipschitz continuity. Let (2, 1| (). (V. II). and (W.ll II) be generic normed spaces. Then function of two variables W x 7 + W is:
(Globally) Lipschitz Lipsciit continuous if there exists K > 0 such that, for (u, (u', xV,
llg(u, v) - gegl,ll, < K|Iu w' K -'ll-
(ii) Locally Lipschitz continuous in its first variable if, for any compacts E U there exists K > 0 such that, for (u, v), (u', v) E E x E',
E' cY
lg v) - g(vl w) KIlu-ull u'll
Equivalent definitions hold for a function of one variable. Morcover, g(., v) is said to be uniformly Lipschitz continuous for in y if there exists K such that, for (u, v), (u', v) E V x Y,
[1 v) - g(u,, e)ll < K| tau-u" u'Il
and uniformly Lipschitz continuous for in any compact if, for any compact E' C there exists K > 0 such that, for (u, v), (ul, v) u x E',
[1 v) - g(u/, b)ll < K| w'll-
Throughout, refer to Lipschitz continuous function with Lipschitz constant K 0 K- Lipschitz.
Model. As explained in Section 4.3, most of our results are proven for the general residual network
(t(( (t)x (t) -E + E (1E Zt (f)), E
1},
(12)
(2;;) (+)A
where zz) (z)(),.... ZŁ(O) E (IRP)L and f x Ra is C² function such that /(0,-) 0 and f(,,z) is uniformly Lipschitz for in any compact. Let us introduce the backpropagation equations, which are instrumental in the study of the gradient flow dynamics. These equations define the backward state (t)
Bh (t) E IR9 through the backward recurrence
ag
pf((t) (t) (F (x; - pt(() = (k+)(t) + 1 - (1)00))) &E (O....,,-- 1}.
(13)
where 0 E Rax4 stands for the Jacobian matrix of f with respect to its first argument. Similarly, we let 0zf R be the Jacobian matrix of with respect its second argument. For sample (xi, Milisien E (X' x ")"", we let hx,(t) and pk,(t) (t) be, respectively, the hidden layer 1 (t) and the backward state p (1) associated with the i-th input æ;. Denoting the mean squared emro associated
14

---

Published as a conference paper at ICLR 2024
with the sample by eh have, by the chain rule,
0AL (t)
L (t)x
(14)
del
(1)
02 (0), pk,(0), kE (l,,,..,,
(15)
nL
08L
n
QB (1) (F"((;;) - -.)
(16)
n
Initialization. The parameters (t) are initialized to (0) Z (+). where Zi
[0, 1] + IR is a Lipschitz continuous function. Furthermore, we initialize (0) to some matrix iiii E Raxd and B = Binit E Rd xq, Note that this initialization scheme a generalization of the presented in Section 3.
Additional notation. For vector x, ||=||| denotes the Buclidean norm. For matrix A, the operator norm induced by the Euclidean norm is denoted by ||A||]- and the Frobenius norm is denoted by ||A|| Finally, we use the notation A (resp. _ Bd) to denote the function + (+) (resp. I + (1), L + (²))), since the parameters are considered as functions of the training time throughout this appendix.
Overview of Appendix A. First, in Section A.1, we study the case the (clipped) gradient flow (5). We show that the weights and the difference between successive weights are bounded during the entire training. Section A.2 shows similar result for the standard gradient flow (4) under a PI. condition. In Section A.3, we show generalized version of the Arzelå-Ascoll theorem, which allows us to prove the existence of converging subsequence of the weights in the large-depth limit. Section A.4 is devoted to the convergence of the Euler scheme for parameterized ODEs. We then proceed to prove in Section A.5 our main result, i.e., the large-depth convergence of the gradient flow. The key step is to establish the uniqueness of the adherence point of the weights. Finally, in Section A.6, prove the existence of a double limit for the weights and the hidden states when both the depth and the training time tend to infinity.
A.1 THE TRAINED WEIGHTS ARE BOUNDED IN THE FINITE TRAINING-TIME SETUP
Before stating the result, let us introduce the notation f (h, E RXxpxp, which is the third-order tensor of second partial derivatives of f with respect to We endow the space Rexpxp with the operator norm || l|2 induced by the Euclidean norm in IRP and the l|2 norm RRxp, In other words,
||02e (h, 2)122
sup uERP,||u|==1
1222f 2)
where 02, z)u E RRxp is the tensor product of 322 (, =) against u. Similarly, / 2) E Raxpxx denotes the third-order tensor of cross second partial derivatives of f. and the space 12xxxx is endowed with the operator norm || induced by the Euclidean norm in R2 and the || ||2 norm in Roxx
Proposition 7. Consider the residual network (12) initialized as explained Appendix A and trained with the gradient flow (5) on [0, T), for some T E (0, 20). Let
M, max
max AERexa
[[][A),
max ZERP (Z)]l,
max BER x4
(B)]1=))
Mo = max
||A
sup eE(0.1]
|| (s)]l, [iiii and M = Mo + TMM.
Then the gradient flow is well defined on [0, T). and, for t [0,T], L E N and k E [I....,L).
||*(()) M, [|]( & M, and ||B((0]rr & M.
(17)
15

---

Published as a conference paper at ICLR 2024
Moreover, there exist 3, > 0 such that, for E [0, and k fl...., L -11. 1}.
(0) (()l (1224110)
7/ (0)1
3T
cal
I
The following expressions for a and B hold:
a = = K' M( Mº + My) and 3 = 2Ke M(K + xK K K'MMX) ( + My),
where
Mx- = sup ||||| xEX
My = sup Ilylls vEY
K1 = sup
(1arf(h,2) 2)122
(18)
||
E {(h,
E R x IR, , ||ll|| < K1 MMx, ||=|| & M}
(19)
K₂ sup (
|1a2f(h, 2) 11,1 K = max(K1, K2)
K' sup (h,z)EE
max (11@22 (h, 2) |1 12 2) 111))
Proof. The time-independent dynamics
(A 24) (T(
OA ग (-z OeL ozk ) x aB²)
OEL
OeL
defining the gradient flow (5) are locally Lipschitz, continuous, hence the gradient flow is defined on maximal interval [0, max) by the Picard-Lindelüf theorem (see Lemma 16). Let us show by contradiction that maxx = T. Assume that Tmax < T. If this is true, again by the Picard-Lindelöf theorem, we know that the parameters diverge to infinity at Tmax. However, for any E [0, T Tmxx). we have
(14()) & [0)(+) /''
dA di (T)
J. Madr & Mo - TMr M. dr & Mo + [[Mddr
Bounds on B" and 2%4 by M can be shown similarly. This contradicts the divergence of the parameters at t = Tmax. We conclude that the gradient flow is well defined on [0. 7] and that the bounds (17) hold.
It remains to bound the difference
1 (t) 2 (t) We have, fort E 10, T) and & E 1-1).
dZ
d
deL
d
k+1
(1)
dl (t)
L
(t) (t)
1zk (1)
dl
az k+1
|10 (nt, (t), (1) ph,,(t) - (2 (t), (t)) (f)
122 (t), 2 (0) [Ik, (t) - pk (t)|
1=1
+ ||pk- 1, (4) |2 (bE,, Zk, (()) - 22. 22 - (1) 2t (1)1
(20)
1). Furthermore, for t E [0, 1 E (o...., ......LL-.. L- 1}. and i E [I....,n.. nn] 1
1 = (1) (t) + L [h,,, (t), 2 (t))l < (1 + K1
(t)l,,
L
since f(, ((,22t (1)) is K₁ -Lipschitz, where K1 is defined by (18), and f(0,, 2t (1)) = 0. Therefore, for any 1o E {1, (l....,L) L).
(0)1 & ek (1,, Ki (111111))) & Ki MM.
(21)
This bound shows that the pair (1),,(), 2 (1)) belongs to the compact E defined in (19) for every t [0, T), & E (l...., ......, L.). and i [.,...,n], In particular, (102f(h/11. 2/: (t))ll & K, and (12,)(1), (1)) 02 -,,0000(((1110))0 < K' (1)) - + 2 (t)l-
16

---

Published as a conference paper at ICLR 2024
Returning to (20), we obtain
| dZ H+1 di (1)
dip
(0) E K Klpp11(0) - -1Pk:(0)-p-1.1(00)
dl
+ K KK'llpk (000,,,))) -1 (t)l + 112 - 2%
For (I,..., L) and i n.) [1ph,(0) (1))
1
K < T (p),
T
and, similarly,
1
K L
1hk [t)=,,(t]|| & Ke Kc MMx
[|b],(t) - -t) - (0),
&
Thus.
d7k k- 1 (t) dt
dzk
(t)
1
IIE,,
K²
+
K'K
cK MMx + +KIIZZ+ - Z (t)l).
dt
n2
L
L
Moreover, for k E {o......I D} and i E nI.
(001 & (01l + [ors( (t). z (0) & (1P++1 + K Hence
L 11pk+1))0)
(0)] < c 112,,(0)) 2e
(1) (F=(ttt))))) -illl
< 2e M (t) (t)l + llvil) < 2e M M² + My), where we use (17) and (21) for the last inequality. Putting all the pieces together, we obtain
dzb dt (t)
dz
(t) < (t) z (t)l L'
dt
Integrating between 0 and t, we see that
3t + L
1.'' (al2((1) 2 (T)l
(0)1 + (t) (t)l < 2
Applying Grönwall's inequality (see, c.g., Dragomir, 2003), we conclude that | (t) (t)l <
(l1 (0) Z (0)1 + )eT desired.
Remark 1. Clipping is used in our approach to constraint the gradients to live in a ball. It is merely a technical assumption to avoid blow-up of the weights during training. However, in any scenario where we know that the weights do not blow up, clipping is not required. A first example of such scenario is under the Dolyak-Lojasievy flow with momentum instead of vanilla gradient flow. This is setup closer to Adam ( (Kingma & Ba, 2015), which is a very used optimizer in practice. One might then show a similar result to Theorem 4, without clipping, because the gradient updates in the momentun case are bounded by construction.
condition (see below). Another scenario is by using gradient
A.2 THE TRAINED WEIGHTS ARE BOUNDED UNDER THE LOCAL PL CONDITION
Proposition 8. Consider the residual network (12) initialized as explained in Appendix A and trained with the gradient flow (4) on [0, co). Then, for M >> O, there exists μ > 0 such that, if the residual network satisfies the (M, p.) p)-local PL condition (1) around its initialization for any L E No, then:
(i) The gradient flow well defined on R4,, fort E R,, L L LEN" and k E ((.....,.) L},
|| (f)l== < MA. 1|z < Mz, and [[)() MB,
where
MA
||iin 1|2 + M, M2 sup
Izini + M, and Mø | + M.
s0(1,]]
17

---

Published as a conference paper at ICLR 2024
(ii) There exists K > 0 such that, for t R4, E N", and & {l...., L}.
K
(z) Z (t) <
L
(iii) There exists bounded integrable function b: + & E L}.
- R. such that, for E R2,, L N", and
ddL dt ()=2
dzk dt
dBL
max
(t)
(t)
(t) & b(t)
dt
(iv) (t), (t),,, and (t) admit limit uniformly over L No and E fl...., L} as + co.
(v) For t E RR, and L Nr, (t) e (0)
Moreover, the following expression for y hold:
Se sup V (*)(0)
#
My MAMx)
(22)
M LEN-
where
Mx = sup |10||
K1 = sup [ =)]|
zEX
|=
E = {(h, E Rd x RR, |/i|| < MAM, || < Mz}
K₂ = sup (b,+)EE
|10r =]|}, K = max K,, K2).
Proof. Let M 0, μ defined by (22), and assume that the residual network satisfies the (M, 11) PL condition (1) around its initialization for any L E N
The time-independent dynamics
OA - deL
JeL
deL
(A4
2 B
azk O
defining the gradient flow (5) are locally Lipschitz continuous, hence the gradient flow is defined on maximal interval [0, Tmax) by the Picard- theorem (see Lemma 16). Let us show by contradiction that Tmax 00. Assume that Tmax < co. If this is true, again by the Picard- I Lindelöf theorem, we know that the parameters diverge to infinity at T Tmax- In particular, there exist t (0, T max) and lt E (1,..., T.} such that
(t) - (0))l > M or (t) 2 M or [IB B² > M. Let t* E (0, Tmax) be the infimum of such times Then, for t" and (1,..... ......,,,, L}. (t) A (0)] & M and | 2f M and (t) - 1² (0)] M, (23) and, by continuity of A4, BL, and 7/4 these inequalities also hold for t t By definition, this means that the (M, μ) llcal PL condition is satisfied for t & t", and ensures that
LEL yA
OEL
deL ØBC
+
+
OZ
[1)
+
(1)
> pe=
k=1
Therefore, by definition of the gradient flow (4),
deL
dEL
d
deL
dzt
(t)) + dBL (t), dBL
deL
dt (t)
1 (t),
(t)
OzL (t),
(t)
dt
dt
dEL
(t)
||
Σ
1.
dEL
(t)
ØBL (t)
del.
DAL
FF
az
< -u (t).
18

---

Published as a conference paper at ICLR 2024
Thus, by Grönwall's inequality, for t tt",
(t) < c (0)
(24)
Furthermore, by (23) and the definition of MA. MB, Mz. we have, for t & t and & {L, D}.
|A (t)ll & MA. [t)] & Mz, and |IB MB. A quick sean through the proof of Proposition 7 reveals that by similar arguments, we have, for t & t, k E {I...., D). and i E {I....,n),
(1),,,(); (1) E E and 1106-,,0111 & 26 2e 11L,0011 < 2e k Mn MalIF" () yall- Thus, for k {O....,L} L).
2e MB
n
(zat) - will & 2 Ma MBVeL(0) & 26 MBe -5 VE(().
(1.)(()l n
<
n
(25)
where the second inequality is a consequence of the Cauchy-Schwartz inequality. Let us now bound 11 (0)]). We have, for E (I..... L}.
d2k L
|| 1z z (0)1
(1)
dl
0
dt
n (10,(((-11,0,,22(0)) (1) Pk
1
(by (15)).
since (t) E E and @2f 2)1l < K for (h, E E. Therefore, by (25),
12/((0) - z (0) < 2Ke K MB / - < 4Ke" MB (L((0) <
M
2
where the last inequality is consequence of the definition of μ.. Similarly, by (14) and (25),
dAL
||A((*)) A0(0)] <
J dt
(t)
dt
1
(t...
dl
1" e- 4
2 MBMX (2(00)
K MB
(2((0)
P
M
Finally, by (16),
dBL
[B²(+)) B(
(t)
dt
dt
2
((F((x;;)))) - y1 v.),t..(t)) dt
0
MA (L((O) 1'' -
4e MAMX (eL(0)
1
M
2
19

---

Published as a conference paper at ICLR 2024
where the third inequality is consequence of the Sauchy-Schwarr inequality and of the fact that MMMx. By contimuity of A', Z and B", these three bounds contradict the definition of t". We conclude that max = oo and that the parameters stay within a ball of radius M of their initialization, yielding the incqualities, for t E R_, L E N and & {I..... L}.
& (t)l
(t)l( & MA. [t)l( & MB. It < Mz.
This proves statement (i) of the proposition. Moreover, the analysis above show that the derivatives of A_, Z and Bh are bounded by bounded integrable function independent of L and k. This shows (tit), together with the fact that the functions (t), (t), and (t) admit limits Furthermore, the convergence towards their limit is uniform over and k; as we show for example for (I). If we denote by A its limit, and apply the same steps for bounding (I*) (0)]l r. we obtain, for any I. 2 O,
co.
20
dAa dr (T) dl
a - (1)llE &
2e MB MBNx (2((0) f.ee %ea dr
4e Me MBMx
e - (0)
# M c
where the last inequality comes from the definition of μ. The bound is independent of L, proving statement (iv). Statement (v) readily follows from (24).
To complete the proof, it remains to prove statement (ii) by bounding the differences I (t) (t)ll- Now that we know that the weights are bounded, proof of Proposition 7 and show the existence C1, C2 > 0 such that
follow the same steps in the
dZ dl
dzk (1) di
Ci 1 Σ 1pk. (1ll
(1)
(T, + -1-1011))--11111
Using (25), we obtain
dzp k+1
(t)
d2k dt
(t) < 2 Mne - V
C1
+ (21ZZ411 - 2t (t)ll)
dt
I
Integrating between 0 and I, we obtain
(1) - 2 < (0) - 20)1+ + / 2e MBE 4 /eL00 (1 dr
+ 1 26 K MBC /el(0)c211 (T)
C,M
1 (0)1+ 2MxL
1.' 26 MB e (0)C2 (T) (+)
where the second inequality uses the definition of μ. By Grönwall's inequality.
(1 201( < (1 (11) (0) 2 + C.M
exp (L
2e M 77 (e(())2dd))
2MxL
C,M
C.M 2Mx
+ 2Mx L
Exp
again by definition of μ. Finally, since (0) -iiit( k
and Zinit is Lipschitz continuous, this
proves the existence of K > 0 (independent of L, and k) such that (1) (1) 2 (1)l < K. which yields statement (1i).
20

---

Published as a conference paper at ICLR 2024
A.3 GENERALIZED ARZZLA-ASCOR THEOREM
Proposition 9 (Generalized Arzeli--Asoli theorem). Let I C 1.,, be an interval, and LEN, be a family of of functions from to IRP. Define
: x I + Rº, (s.t) + (s, 11o--1)o++1t) (t).
Assume that there exist constant C > 0 and a bounded integrable function b I - IR such that the following statements hold for any I and L E N
(i) For & E {1,.... L 1), 1 (t) (t)l & L.
(ii) For k E {I....,,}}; (t) & C and l| az (t)ll < b(t).
Then there exist a subsequence (z+(L)) LEXT of (z") LEX and a Lipschitz continuous function z" : [0, 1] x + IRP such that 2o(L) tends to (s, 6) uniformly over and
Note that if 7 compact interval, then the existence of (uniformly) convergent subsequence is guaranteed by the standard Arzela-Ascoli theorem. Indeed, the uniform equicontinuity is consequence of assumptions (i) and (fi), while (ti) provides a uniform bound. However, if is not compact, more involved arguments needed.
Proof. Assume, without loss of generality, that is also bounded by C. According to assumption (i). fort E 7 and i.j E (I,...,II) L.),
Cli
(t) - -z,(001 <
L
Also, according to (ii). for 1, 1' and & E [1,....,L).
| - zi (1)) (1)1= f/ d2t
(t)dr) < S91--11.
dr
It follows that, for s E 10 and t, t' EI,
z (s',t') < -z4 (s, - -z, +1) - 2'(5((())
< C 1'1 + C - 11s-1)
L
Therefore, with some simple algebra, we obtain
(s, (-6, < C|L- Cls 8'1 T.
(26)
The statement of the proposition is then a consequence of the next three steps.
There exists a convergent subsequence of (2"(s, (s,t)) LEN First, let ((s,, (x)) = (Qn [0, 1]) x (Q I). By (ii), the sequence (2"(si, f.)) jeN is bounded. It is therefore possible to construct by a diagonal procedure a subsequence (2)(L)) LEN- such that, for each i N, (2o(L) ()) is a convergent sequence.
Let us now show that (z)())(s, LEN converges for any s E [0, 1] and e by proving that is Cauchy sequence in the complete metric space Rº. Let E > 0, s E [o. 1], and t E I. Since (ss, 4))se is dense in [o, 1] x I, there exists some j E such that [8; - a| < and 14 - 4| < e. Then, for I., M E N we have
(s, - ZM(M
< (2, 2o(L) (sg, t;)ll + (s, Z0(M) (sj,t,)l
+ (M), - -z(M) (s,
C < 20 + Ø(L) + (1 [5) 26(M) (s,, 11)l 2Ce ((M)*
C
where we used inequality (26) twice. Since (Zo(L) ty LEN- convergent sequence, it is Cauchy sequence. Thus, the bound can be made arbitrarily small for L, M large enough. This shows that (Z+(1))s. (s. t)) is also a Cauchy sequence. It is therefore convergent, and we denote by 2o (s, 1) its limit.
21

---

Published as a conference paper at ICLR 2024
The function 2d is Lipschitz continuous. By considering (26) for the subsequence ((L) and letting L - oo, we have that, for any s, E [0, 1] and t,t' EI,
(s, - (s 1) & C(|s - + It
(27)
The convergence of (20(L) (s,))) LEN to (s, 1) is uniform over and l. Let > 0, 8 E [0. 1].
and E L. Then, by (26) and (27), it is possible to find 6 > 0 such that, for any s' E [0, 1] and
t', t" E satistying Is' - s"I E 6 and It' t" < 8.
c
(2 - (6", 1']l < - ((L)
and - 26),"(1 <,,
(28)
and
c and
|| (5,())(,,))) -201) ( ||| <
- <
(29)
(())
Furthermore, there exists a finite set fsi.....
fss C [0, 1] such that
10, C U(s: 8,5; + o).
==1
In the sequel, we denote by s" an element of {si...., ss} that is at distance at most 6 from s.
If is unbounded, then, by assumption (it) and since b is integrable, there exists some to >> 0 such that, for t > to.
|| |z Z)LL) to)ll < ((1l d z01
- ()rr)s-n+++00 ()) & S ((r)dr <e. (30)
to dt L(Ot)s
The same inequality holds for 2 by letting tend to infinity. is bounded, we simply let to s.
We may then pick a finite set
c 10, (o] such that
T
[0, fol c U - 5, + 8).
i=1
Two cases may arise depending on the value of t. If E [0, tol. then there exists an element of the set {f,...., (T) } at distance at most 8 from and we denote it by 6*, If > to. we let t to. According to (29) and (30), we then have in both cases that
C and (1, - 20 &
(x,()) - zo( 2) t-)
& 8
+
(31)
xLL)
To conclude, we have to bound the term || (x, - (x, f)ll uniformly over and I. We first have
|| [200() t) (s, t)ll
< (1 (s,t) 20(l) (s, 1*)l + ([ -101)))))))
(s,
+ (,, - =s,(,()
C -120(1) (s, 1*) - 2% (")),
< 28 + (T.)
where the last inequality is a consequence of (31). The last term can be bounded as follows:
(2 (²) - (x,
(12 t²) 20(l)) (st,")+ (2 (1 (s",
+ (s", (s',t)) (s,tt)))
C
2 + ((L) + max
120lL) (s,, 23(
:E(L....s)
22

---

Published as a conference paper at ICLR 2024
by using (28) and the fact that s E {$1,.... 88}. Putting all the pieces together, we finally obtain
20
|z, t) (s), t)ll < 4 +
+
max
(s; tj) (5; (ss,tt)]. ty)
Q(L) :E(L....).EEEELL..
By taking I. large enough, independent of 8 and I, the sum of the last two terms can be made less than Since arbitrary, this concludes the proof.
A consequence of this result is simplified version for sequences of functions only indexed by L and not k; as follows.
Corollary 10. Let I I R+ be interval, and (Z)) be a family of functions from to IPP. Assume that there exist a constant C >> O and a bounded integrable function b I + I. such that, for any I and L N" [t)l & C and d2" (t)ll b(t). Then there exist a subsequence di
(7%()) of (74) and a function 7 I + TR? such that (1) tends to (t) uniformly over t.
A.4 CONSISTENCY OF THE EULER SCHEME FOR PARAMETERIZED ODES
Proposition 11 (Consistency of the Fuler scheme for parameterized ODEs.). Let LEN) be a bounded family of vectors of RR, and let
@ : + +R, 0 ((L-1)+]+1
Assume that there exists θ 10. R a Lipschitz continuous function such that (7) tends to (s)) uniformly over s. Let LEN be sequence of vectors in some compact E Rd converging to a E E. Let R2d x Rp - IRd be C2 function such that g(0, .) = 0 and g(- 0) is uniformly Lipschitz continuous for 9 in any compact of RP. Consider the discrete scheme
utf at
wht1 = u + -
7 0f O2tz), K L0
(32)
Then utenj tends (s) uniformly over E [0, 1}, where U the unique solution of the ODE
(())
dU
(33)
ds (s) = g(U(s), s SE [0,
Moreover, the convergence only depends on the sequence (a LEW) and on its limit a E E through (IIa" -al)
Proof. Let M a bound of the sequence (0% LEN L. By definition OL, the sequence (0") also uniformly bounded by M, and the same is true for O Then the function g(, O(s)) is uniformly Lipschitz for [0, 1]. Furthermore, (U, gU, Os)) is continuous in because g and θ are continuous. Thus the ODE (33) has a unique solution on [0, 1] by the Picard-Lindelöf theorem (see Lemma 16).
Denote by the uniform Lipschitz constant of g(, 0) for ||0|| < M. Since g(0, = 0 and o(s,((s))
is C-Lipschitz, one has
(s) = (l)( < (s))
ds
Therefore, by Grönwall's inequality,
|[((s)) < [O]] = ||a|| exp(C) & DE exp(C'),
where Dg = suprer ||||| co. similar reasoning applies to the discrete scheme (32), using the discrete version of Grönwall's inequality. More precisely, for any & E LO....,-.. 1}.
1 | < ||uE|l| Il + lla(utt u oktiill Dll & (1+ ② |
23

---

Published as a conference paper at ICLR 2024
Thus,
|| < || exp(C) = | < DE exp(C).
Overall, we can consider a restriction of to compact set depending only on M, C, and E, which we will still denote by g with a slight abuse of notation. Since is C', it is therefore bounded and Lipschitz continuous, and still let C be its Lipschitz constant.
For L E N and & E L}. we denote by A the gap between the continuous and the discrete schemes, i.e.,
A
U
k I
uk]
The next step is to recursively bound the size of this gap, first observing that AB = I|a" - -all. We have that
8 s dU
ds (s) = ((()))))((s))
(34)
a Lipschitz continuous function with some Lipschitz constant Č. To this, just note that U itself is Lipschitz continuous in s, since g is bounded, and therefore the function (34) is composition
du is almost everywhere differentiable, and its of Lipschitz continuous functions. In particular, de
L (s) is bounded in the supremum norm by C. As a consequence, for k E {O...., L-1). derivative dxa the Taylor expansion of U on [ &+1] takes the form
U( h +
h -(( i
L -
+
L ds
1 dU k
L
+
(K1))L
k k L
dPU
ds
(s)ds,
where the norm of the remainder term is less than C/L. Therefore,
U k +1 uhtal
A
|
k L
+ 1 (U(1) k (
θ (((
)
+
k (1
(
du
ds (s)ds
kkbb
ut 1
gl 0 1 i\|l
1 (
ut
+ 1
(1
U
k
θ 1:
o
of ok+1
I
+ Juu p1kta//L
(K+1 -s) s)
dU
(s)
ds
KIL
do
< < - 1 + C 0 0/4p + Ō
L
In the last inequality, we used the fact that is C-Lipschitz. Since, by definition, 0 ( ) we obtain, for k E {O...., L
1}.
C A +
1 1
C
- 00 (O' (1=1) L
k.
< 1
L
L
C
C
C - T
<
[,
A +
I
sE(0.1]
sup
|0((s) - (x)) +
I
+
1
< (1
-
C
L
A +
C
L
=E(O.1]
sup
|0((s) (3))
CC. Č
L
+ 1
where Co is the Lipschitz constant of e. By the discrete Grönwall's 's inequality, we deduce that, for k E {O..... L- 1}.
A/41 < (Ab + sup |[O(s) - (')))] Co sE(o.1]
)
e
L
= =(1at - all + sup
|0(s) - 0+ + Ce
+
ė e. T.
(35)
T.
.E(0.1]
24

---

Published as a conference paper at ICLR 2024
This shows that the gaps A converge to uniformly over & E {O...., L) L tends to infinity. We conclude by observing that, for any E [0, 1].
(IU(s) - u r.njl |(() U Lss )
U [.S]]
uta
Cu + fraj:
(36)
where Cr is the Lipschitz constant of U. Both terms converge to zero uniformly over as L tends infinity, Finally, an inspection of our bounds shows that the convergence only depends on (a') EB through ||a_ - -all-
The results of Proposition 11 can be extended without much effort to two other related cases. First, the parameters o5 may depend on some other variable t, as long as all assumptions are verified uniformly over I, Second, these parameters may converge to some limit parameters as both L and I go to infinity. This encapsulated in the following two corollaries.
Corollary 12. Let I C R4 be an interval. Let LENN J be uniformly bounded family of junctionsfrov to RR, and let
Q 0. x + RP, (s,t) 02 ([L-1)a++1(t)
Assume that there exists a Sunction θ: [0, x R such that O² (8, 1) tends to ((s, uniformly over and I, and ( 1) is uniformly Lipschitz continuous for i E T. Let (a+) LEN- be a family of functions from to some compact E. c R2,, uniformly converging to a
F.. Let g Rdd TRP Ra be C¹ function such that g(0, ) = 0 and g(: 0) is uniformly Lipschitz continuous for 0 in any compact of IP. Consider the discrete scheme, for I,
ug (1) (1)
u ( = (t)+ + I ((4 E tO....,,-- L- 1}-
Then w (.) (1) tends to (s,() uniformly over E [0, 1] and E I, where U(,1) is the unique solution of the ODE
=
aU
Øs (s.t) g(U((s, O(s,t)), SE [0,1].
Moreover, the convergence only depends the sequence (a LEN) and on its limit E' through (supter (t) - -a(t)l)LEN..
Corollary 13. Let I C R2 be an interval. Let (ENN be a uniformly bounded family of functions from I TRP, and let
@:: 10.10 - IR,, (s,t) + 0((-11)04+1()))
Assume that there exists a function Ox [0. TRP such that ( 1) tends to O00 (s) uniformly over 3 as L,t- 20, and xx is Lipschitz continuous. Let (L") be family of functions from some compact E C _2, and converging
ax E E as L, co. Let Rd x Re + RR be C³ function such that g(,, -) = 0 and g(: f) is uniformly Lipschitz continuous for in any compact of RP Consider the discrete scheme, fort E I.
(t) = (t)
(+) (t( (t) 1s E 1}.
Then utr (1) lends Us)) uniformly over E [0, as L,l do, where U is the unique solution of the ODE
(0)) aoo
dU
ds (s) g(U((s), (z))),, SE [0.
Moreover, the convergence only depends on the sequence (a") LEN: and on its limit a E through (supter [|a=(() a(t))))xes..
25

---

Published as a conference paper at ICLR 2024
A.5 LARGE-DEPTH CONVERGENCE OF THE GRADIENT FLOW
This section is devoted to proving the main result of Appendix A, namely the large-depth convergence of the gradient flow. The setting we consider encompasses both Section 4.1 (finite training time and clipped gradient flow) and Section 4.2 (arbitrary training time and standard gradient flow). To this end, we consider training interval which is equivalent to the standard gradient flow (4) if ' equals the identity. Note that we do not need to assume the following proof that æ is bounded (but only Lipschitz continuous). Therefore, the proof also holds in the case where π equals the identity.
[0. 7] C +,, for & 0o, and the gradient flow formulation (5),
Theorem 14. Consider the residual network (12) initialized explained in Appendix A and trained with the gradient flow (5) on I = [0, T] C ftr,for some T E (0, oo]. Assume that there exists a unique solution to the gradient flow, such that (IE) and (B") each satisfies the assumptions of Corollary 10, and (ZE LEB" 15kE satisfies the assumptions of Proposition Then the following four statements hold as L tends to infinity:
(i) There exist functions A : I Rexd and B I -> R.d x9 such that (f) and BL converge uniformly over t E T to A(t) and Bt)).
(ii There exists Lipschitz continuous function 2 [0, - TR. such that
z²: [0, 1] IRR (s, (s, = 1(ir-11s(++(t)
converges uniformly over E [0 and E I Z(s,1).
(iil) Uniformly over s E [0, 11, I, and x E X, the hidden layer [ (t) converges to the solution at time s of the neural ODE
H(O,t) t) A(t)x
0H
Ös (s tH,
se 10.1]
(iv) Uniformly over t E and x E X, the output FL (I; converges to (1,)
Proof. According to Proposition 9, there exists a subsequence (Zo(L)) of (Z LEN- and
Lipschitz continuous function 2 [o, 1] x R2 such that g 1) tends to 2 () uniformly over and I. Similarly, by Corollary 10, there exists subsequences of (A") and (B4) (B") LeN- that converge uniformly. With slight abuse of notation, we still denote these subsequences by Φ. and the corresponding limits by A and 1
In the remainder, we prove the uniqueness of the accumulation point A2, B$) by showing that it is the solution of an ODE that satisfies the assumptions of the Picard-Lindelöf theorem. The statements (i) to (iv) then follow easily.
Consider general input (x. X' V, and let (s, h'Le (t) (recall that h is defined by the forward propagation (12)). Corollary 12, with 0f (2((() O 2 a² A x, 9 = f. ensures that H (s, () converges uniformly (over and 1) to H 1) that the solution at time of the ODE
(0, t) A
011 as (s, (s) SE 10.
By inspecting the proof of the corollary, we also have that (h (L) I.N I.NNe (ERG(()) and (Ho())) are uniformly bounded and that (-tt) t) is uniformly Lipschitz continuous for t E 1.
We now turn our attention to the backpropagation recurrence (13), which defines the backward state p (t). First observe that the convergence of HI)) implies that
P ( = 2B)(D) ((BLL) (t)) y) = (t) (0)0))))))
26

---

Published as a conference paper at ICLR 2024
converges uniformly to (B
y) E R Now, let (s, pLLo| (t). We
apply again Corollary 12, this time to the backpropagation recurrence (13), with 0 (h1 2 = (H, 2%), g : (p, (h, 2)) + d₂ f (h, p), and at = ((B)L)) T (BOLL) (1B0L))10(L))) (1,)))(11)) - y) Let
us quickly check that the conditions of the corollary are met:
The sequence (h& ((L))
ITENN,IEEEEN is bounded, as noted previously, and the same holds for (Z)(L) LEN IekEelL) by the assumptions of Theorem 14.
The function (-, is uniformly Lipschitz continuous for i as noted previously, and the same is true for (- t) since 2o is Lipschitz continuous.
The function h ((L) ((o(()) tends to (s, uniformly over and L, seen in the
beginning of the proof. More precisely, we know that H h o(1) [o]L)=|(() tends to
(s) Simple algebra and the fact that two successive iterates of (12) are separated by a distance proportional to 1/L show that both statements are equivalent. Furthermore, (z,) t) tends to (s, t) uniformly over s and as noted above.
The sequence La is uniformly bounded, since BOLL) and H1L((1.)) are. It also converges uniformly to + 2B (1) (1,,
The function is C since f is C2, We clearly have g(0, = 0. Finally, g(, (h, Z)) is uniformly Lipschitz continuous for (h, Z) in any compact since o f is continuous.
Overall, we obtain that POLl) (s, t) converges uniformly (over and f) to (* t), the solution at time of the backward ODE
pd (1,t) (t) (B v)
v)
aP
- (s,t)=th
O. (s, 2 (s, 8 SE [0.
Ōs
Furthermore, the proof of the corollary shows that (po(L)) is uniformly bounded. Now, recall that the gradient flow for (L(L) (t), given by (5) and (15), takes the following form, for t E I and k E (1....,dd ((L)).
oz 2 (1)
T -- Σ Oe 021(1 (1,
(0D (t) PRi (PP
0t
where the i subscript corresponds to the i-th input m4. By definition, for E [0, 1). (x,))ss = ZÓLL)
((o(L)-1)as+1 (t). Thus, the equation above can be rewritten, for E [0, 1] and e I.
(24(1) (s, = - Σ 0zf( (LL) Øl
(1) (10(1)) (4) a(L)
((() 20(2)
P (((())) (1)+))
(37)
The term inside can be rewritten as
(L(())) 1)sl ((L)
[(((L)
1). (s,)) ppotes ((o(L)) - 1)s} +1
Σ Ozf (1
Φ(L)
Since f is C²2 Sz. is locally Lipschitz continuous. Applying the first part of the proof to the specific
case of Fe, know that H.(L) and p.((L) uniformly bounded, and that H,(L)) (s. ts) t) and (L) (s, t)
converge uniformly to H (s, and (s, Therefore, the right-hand side of (37) converges uniformly over and f to
1 Σ 2s, (s))
n
27

---

Published as a conference paper at ICLR 2024
We have just shown the uniform convergence of the derivative in t of (s, t). Furthermore, know that, for E [0, 11. the sequence (t + (s, (s,t) LEN converges to (s...) These two statements imply that 2 is differentiable with respect to t and that, for 8 E [0, 1], its derivative satisfies the ordinary differential equation
(s. a
Σ 0z 20. (...)))
(38)
Moreover, by our initialization scheme,
(s, 0) = (s
(39)
A similar approach reveals that A (t) and (t) are differentiable and that they verify the equations
d dt (t) π
p (0.t)x.))
A*(0) = A
(40)
d
n
dt (f) 7
(
B Binit
(41)
a
The equations (38) to (41) can be seen as an initial value problem whose variables are the function (-,t) [0, 1] + R' and the matrices (t) E 129xd (t) E Rd xq, To complete the proof, it remains to show, using the Picard-Lindelöf theorem (see Lemma 16), that there exists unique solution to this problem. First, note that the space B(O0, 11, RP) of bounded functions from [0, 1] to R2 endowed with the supremum norm is Banach space, which is the proper space in which to apply the Picard-Lindelde theorem. We therefore endow the space of parameters B([0, 1], TRP) Raxd Rd xq with the norm
(1 = sup |2(s)]1 + ||A||2 + B|
=E[0.1]
which makes it a Banach space. We have to show that the mapping
(2, 8 8H π
2()))
(42)
म Σ R.OOTT)
क (-
2 n (BH,(1) -3)((1))))HH(()7))) v))H(()))))
locally Lipschitz continuous with respect to this norm, where we recall that H;(s) in (42) is the solution at time of the initial value problem
H, (0) Ar, dH; (s) ((H,(s), s [0, 1).
(43)
ds
and Pi(s) is the solution at time of the initial value problem
P((1) 2BT (BH,(1) v:)
dP.
(44)
ds (8) (s), (s+)), 8 [0.
To prove that the mapping (42) is locally Lipschitz continuous, we first check that it is well defined. only bounded (and not continuous), the solutions of the initial value
Since 2 assumed problems (43) and (44) are well defined in the sense of the Caratheodory conditions, which are given in Lemma 17.
n). Next, we can show that (2,A, + H; is locally Lipschitz continuous for i E (1....,n) To do this, consider two sets of parameters (Z, B) and (2, A, B) belonging to a compact set D. Let H, and H; denote the corresponding hidden states. As in the proof of Proposition 11, it holds that H, and H; belong some compact set F that depends only D and Let Ky be the Lipschitz
28

---

Published as a conference paper at ICLR 2024
constant of the C' function f on E x D. Then,
[IH((+) - H < (0) H.(O)II + J. dit,
(r) -
dHti dr
(r) (r) dr
dr
< F. (4(O))I + ! (I((H(()),((r))) - -(H
The norm inside the integral can be bounded by
(r,, (r),,
-
< K sup
[[((r) Z(r)ll (,)
re(0.1]
Therefore,
[z(() Z(r)ll + [-" K))l Hr(r)lldr.
(s) - H. & || -1/22|22511 K2 sup 1]
Using Grönwall's inequality, we obtain, for any E 1].
(H) - Hs(s)]l & (1LĀ Al12||=11 + Ky sup r
[z(r) 2(r)l) exp(K)).
This shows that the function (2, + H; is locally Lipschitz continuous. One proves by similar arguments that the function (Z, A, B) P; is locally Lipschitz continuous. Thus, overall, the mapping (42) is locally Lipschitz continuous as a composition of locally Lipschitz continuous functions.
The Picard-I Picard-Lindelöf theorem guarantees the uniqueness of the maximal solution of the initial value Lindelöf
problem (38)-44) in the space [[[0, 1], R2) x Rgx2 x Rd xq Since any accumulation point (Z*, A Bo) is solution belonging to this space, this proves the uniqueness of the accumulation point, which we therefore denote as (2, A, B).
The uniform convergence of (Z", AL B²) to BB is then easily shown by contradiction. Suppose that uniform convergence does not hold. If this is true, then there exists a subsequence that stays at distance similar to the beginning of the proof show the existence of second accumulation point, which is contradiction. This shows the uniform convergence, yielding statements (i) and (ii) of the theorem.
0 from (z, A, B) (in the sense of the uniform norm). Then arguments
Finally, reapplying Corollary 12 with 0% z θ = 2, a² = = f. completes the proof by proving statements (dii and (iv).
Training dynamics of the limiting weights. Interestingly, the proof of Theorem 14 provides us with an explicit description of the evolution of the continuous-dppth limiting weights during training. With the notation of the proof, the continuous weights satisfy the training dynamics:
dA (1)
P.(0,t)x.)
π
dl
11
0z
0t (8,1) = .T( (n(-1 (,((((()),,
i=1
dB (t) = π x-- Σ (()) ())
2
dt
n
where we recall that H, t) is the solution time of the initial value problem
H,(0,)) A())a;
0H, 0s (s, S(H,(, 2(s,)), s (0, 1J.
29

---

Published as a conference paper at ICLR 2024
and tt) is the solution time of the problem
P1( = (B(() (B())HH())) y:)
OP: Øs (8, = O.,(HH,s, !), E(x, (s, 9), 8E (0.
These equations can be thought of as the continuous-depth equivalent of the backpropagation equa- tions.
A.6 EXISTENCE OF THE DOUBLE LIMIT WHEN L, t TEND TO INFINITY
Proposition 15. Consider the residual network (12), and assume that:
(i) (t), LLLe (t), and (t) converge uniformly over N and E [0, 1| as - cc.
(i)) (t), [1.1 (t), and (t) converge uniformly over t R4 and E [0, 1] L
(iii) The loss 8 (t) converges to 0 uniformly over L E No as c0.
Then the following four statements hold as and L tend to infinity:
(i) There exist matrices Agc E Raxd and B E Rdd xa such that A (t) and (t) converge to Aco and Boo.
(ii) There exists Lipschitz continuous function Z : [0, 1] + IPP such that [ (1) converges Z uniformly over E [0, 1].
(iii) Uniformly over & E [0, and x E X, the hidden layer N M.gg (t) converges to the solution at time of the ODE
H(0) A dH
(45)
ds (s) f(H()), sE 10.
(iv) Uniformly over a E the output (x; converges (xo Boe (1). Furthermore, Fxo Fx,(x,) yi for e (1....,n..
Proof. The existence of limits Ax and Ba ( and B²(t) as L and tend to infinity is given by Lemma 19. The same argument applies to Z(.r.) (t), which provides a limit 3oc (s) to the sequence.
Furthermore, following the proof the lemma, we see that the convergence of L (t) to Zx (s) is uniform over E [0, 1). Corollary 13, applied with of 7 Ox Zoc, at A A 9 then ensures that (1..) (t) converges uniformly (over E [0, and X) to H(s) that the solution at time of (45), as L and t tend to infinity. As consequence, px;; converges uniformly over x to (x) L, + oo. Furthermore, recall that
1 Σ
(t)
| (x::t) - -
The left-hand side converges as L, + co to by assumption of the proposition, while the right-hand side converges
(xx)) v1112.
Therefore, Foc (x:) 31 for i (1...., n). and the proof is complete.
30

---

Published as a conference paper at ICLR 2024
B PROOFS OF THE RESULTS OF THE MAIN PAPER
In this section, we prove the results of the main paper. Most of these results follow from those presented in Section A. The only substantial proof is that of Proposition 5, which shows the local PL condition. It uses a result of Nguyen & Mondelli (2020) involving the Hermite transform and the sub- Sab-Gaussian variance proxy, which we define briefly. We refer Debnath & Bhatta (2014, Chapter 17) and Vershynin (2018, Sections 2.5.2 and 3.4.1), respectively, for more detailed explanations.
Hermite transform.
The r-th normalized probabilist's Hermite polynomial is given by
(x)
1 - (11" 12 d"
1 0.
dx
This family of polynomials forms orthonormal basis of square-integrable functions for the inner product
f2) 2 [
f(
Therefore, any function a such that 1 1 (x) 22 < oc can be decomposed on this basis.
v2n The r-th coefficient of this decomposition is denoted by 11 (o).
Sub-Gaussian random vector. A random vector x E Rd is sub-Gaussian with variance proxy Ux 0 if, for every y E Rd of unit norm,
12
(( v\ >1) 2exp
2v
Additional notation. For matrix A, we let Smin and Smax its minimum and maximum singular values, and similarly, Amin and Amax its minimum and maximum eigenvalues (whenever they exist).
Before delving into the proofs, we briefly describe the parts of this section that make use of the specific model (3). The most important one is the proof of Proposition 5, i.e., the proof that the residual the proof of Proposition 3, the
network satisfies the (M, p))llcal PI. condition. Additionally, expressions for M and K are valid only for the specific model (3). Finally, in the proof of Theorem 6, the beginning of the proof reveals that condition (22) Proposition on f can be expressed as condition on the norm of the labels yı- This applies only to the specific model (3). Observe that, if one assumes that the general residual network of Section A satisfies the (M, p)-local PL condition with μ given by (22), then the rest of the proof of Theorem unfolds, and the conclusions of the theorem hold for the general model.
B. PROOF OF PROPOSITION
Proposition 1 is consequence of Proposition 7 with (V,W))
1 Vo Wh).
7m
B.2 PROOF OF PROPOSITION 2
Proposition 11, with 0f = (V/2, W (V, at Ax, g(h, (V, (o
gives the existence and uniqueness of the solution of the neural ODE (6). Moreover, inspecting the proof of Proposition 11, equations (35) gives that, for any input x X, the difference between the last hidden layer hy of the discrete residual network (3) and its continuous counterpart (1) in the neural ODE (6) is bounded by
c sup - sE(0.1]
where C" > is independent of L and x E X, and (s) 0 flit- 1)+ -1- The function e is piecewise-constant interpolation of θ with pieces of length T. Since θ is Lipschitz continuous, the distance between O and O decreases as /t for some C" depending on θ but not on This yields ||4 #(]] < C'(L++G") where C' and C" are independent of L and X. Since
(x)x) Bl and F(x) (1),), the result is proven.
31

---

Published as a conference paper at ICLR 2024
B.3 PROOF OF PROPOSITION 3
We apply Proposition 7 with f(h, (V, W)) V Wh). Recall that the parameters Z v
(V, W)) are considered in Proposition 7 as a vector. In particular, || || + |W]].. Therefore, Proposition 7 shows that, for t E [0, 7). L E N and & ,l,...,, .L..
(Alle & M, (VIIE IIW & M, and [1B < M,
where
M = Mo + TM
Mo max (114400) 11V (0)1l> 11W [0)(()))))
M₁ max
max AEReXd
(A)]FE
max ZEROX" XR
||<(|)||,
BER x2 [T(]]]])
Furthermore, due to our initialization scheme described in Section 3,
||A(())l== = vd. (O)lee O. (W]L & 2/ (0)le] =d
where the third inequality holds with probability at least - exp( 3gm by Lemma 20. Since we 16
take q 2 max(d, d'), this implies that, with high probability, Mo 2/qm. yielding the formula for M in Proposition 3. Finally, the existence of K TTeaT such that the difference between two successive weight matrices is bounded by K/L, as well as the dependence of a and B on X, y. M, and a, follows easily from Proposition 7, given that our initialization scheme ensures that (0) = z (0) for all I. E N' and k E (1...., I.).
B.4 PROOF OF THEOREM 4
By Proposition and the fact that π is bounded, the sequences (A and (B") each satisfy the assumptions of Corollary 10, and (Z() _ı & satisfies the assumptions of Proposi- tion 9. Theorem then follows directly from Theorem 14, by taking, as previously, fh, (V, W)) = Vo Wh).
m
7q
B.5 PROOF OF PROPOSITION 5
We drop the L superscripts for this proof, since fixed. Denote by A, B, VR. Wa parameters sampled according to the initialization scheme of Section 3, which means in particular that V2 0 and W; = W ~ x)) Since, by assumption, the activation function is bounded and not constant, it cannot be a polynomial function. As a consequence, there are infinitely many non-zero coefficients (a) in its Hermite expansion (defined at the beginning of Section B). Throughout, we let 2 be an integer such that 7 (a) is nonzero. We also let K₂ be the Lipschitz constant of a and M, its
)-(a) supremum norm. Now, let A, B, VR, We be parameters at distance at most M min( 32.. 2 from A, B, Vú, Wa in the sense of Definition 1.
It is useful for this proof introduce matrix-valued version of the residual network (3). More specifically, given data matrices x jdx= and y 12d the matrix-valued residual network writes
ho Ax
Vo+10 Wotthe), EE (O...,,, L- 1}.
h = h4 + L
(46)
/
where now h& E R8xx, The loss is equal to e bllbh, - yl and we let The the matrix-valued backward state. Observe that the columns of x are bounded and thus sub-Gaussian. In the sequel, denote by us the sub-Gaussian variance proxy of the columns of /
3r E gex" be
Now that we have introduced the necessary notation, we can proceed to prove some preliminary estimates. Since M & & /2qm, have, for Ie E (1....,n).
|| A < M, | Bli < 2' VRLlE 1 I[ W & < v 22m.
(47)
32

---

Published as a conference paper at ICLR 2024
By Lemma 20, with probability least 1 - exp ( (- 16) ) one has ||W| & V2qm. Together with the previous inequalities, this implies
3 2' | L ||&l & 2qm, (48)
|| < 2, min (B)
|B
2'
where the second inequality is a consequence of Lemma 18, as follows:
5 Smin
(B)
> Smin (B) - B B 1 | BlIF > 2
Let us now bound | || and ||p |. have
||oo || ||x|| & ||A||/|/x|| & 22/on.
(49)
Moreover, by (46), for any k E {O..... 1}. 1}.
Ko || || [[Weille[[bb x(1+ 2v2Ko
b ||r hhllF Ivmm/ where the second inequality is consequence of (48). Therefore, by (49),
|hallr F:
I.
[h || & exp(2v2K,)) holl & 2exp(2vVKK))
(50)
Moving on ||px||r, the chain rule leads
Whet (VV Oo'( 1 Warlhs)). k E L 1}.
Pk = P8+1 +
LVqnm"
where @ denotes the element-wise product. Noting that |o'| < Ko and using (48), we obtain
Kr Lyqm
(W++illF Va-.llleVVVlllllllllee > (1 (1- 2/2Kg |p++1 lI..
||*||e > ||P++IlFF
||p/|P. In addition. It follows that |PulF > exp(-2v 22/222//K K.)
ae BT (Bhr, - y).
Pr. =
Thg
Therefore, by Lemma 18, since d' E.
|| > - smin (B)I - -VlFF 2 Vl.
vn
Collecting bounds, we conclude that, for E ;0....,LI, L},
|pallrr 1
exp((-22 222K2K)
(51)
Vn
A similar proof reveals that, for k {). D}.
|| < Vn cxp cxp(22V2K.VV
Having established these preliminary estimates, our goal in the remainder of the proof is to lower bound the quantity
||
TV. ||f. First note that, by the chain rule, for any k E ft,...,,eo 1}.
de
1
1
P4+10
OV
L
As consequence, when m 3 n, by Lemma 18.
||
O8
1
| IF Smin (0 /q
1 Worzha))
L/m
L Lymn exp( 1ap(-22/KK -22222222 Smin [g
WWethh))
(52)
33

---

Published as a conference paper at ICLR 2024
using (51). Next, by Lemma 18,
Smin (a (7(2 Wk Wh+the) Smin (1 (o xx)) (x)x
- -/ - ( xxxx l1.
Let us first lower bound the first term. Since, by our choice of initialization, A = (Tzaxa, Opto axxa), we have
1 ( WAXx = Smin (W))),
Smin
where W ~ NN ()0(mxx))) and x 1 x E Rdxn has i..d. unitary columns independent of W. vi
Therefore, by Lemma 21, with probability at least exp (- 3m12(o) 2n exp nu.n2
64M2n
Smiu (o (o(
1 (" W (xAxx
Vmnn-(a)
Next,
|o 1 79
Werthes
-0( W Ax
Ko
[| k+1 Wllr|lbelle + || [W]lF|/h, Ax
17
/
Ax Axllr)
+
Clearly,
1
W Wyyhh-1) 1|
4v2Kok exp(2V2Kx)
[bg
Ax|1 F
Vj0
T./m
/7
j=1
by (48) and (50). Also,
(T)
|Ax Ax|| < || -
A||
32/2K=
by (47) and by definition of M. Putting together the two bounds above as well as (47), (48), and (50), we obtain
0
1 h++ Watths) -o ( W Ax
| Ko exxx22/2KK (1+ /Qm
8Kak
Vm ())
I
32
Vngmi
(1)
< Civn + C2
v
16L
32
where Ci = Ko exp(2v2Ko) and C2 = 128C Thus, when Ci &
1
Vmnnr (T), we have
32
(0 v (itttb)))) > Vm
3
C2 /gg L
Smin
())
Vm),(oa)
16
16
for k < Cay
(mx(aa
As a consequence, for k <
Lnd(a) Ca
returning to (52),
Can,(a) Vt. Lvn
8LVR (o) exp(ppp222222 (xp()))xppp222))VE
0v++
letting C3 = exp(-2v2K.) Therefore,
8
dt
0E
Caymy
de 2
+ L
+
>
0A
F
aZ
F
F
(a)
(G)
LPn
(n
e,
202n.nqq
34

---

Published as a conference paper at ICLR 2024
where we used the inequality
[2] 222 for x 3 1. This proves the result, with
(o)2 1) max 220 (xep(44222)
C00CP
max
MT(0))
C2
128K3 exp(2v2Kg)
0 (a)
(0)
11((0) 1
C3 min
32,2KK '2
(o)
17 (o)
C
2C2
K3 exp(6v/Ko)
qm
+ n2 exp (- (-64M3n
(o)
+ 2n exp (-
d
= exp
16
64M3n
20
Remark 2. With appropriate values of and m, the probability of failure 5 can be made as small as
d
E + 2n exp
(53)
2v.n=
for any E > O. This is possible first by choosing T such that 2/r 30 E, then by choosing m such that the first two terms are less than e. Moreover, we refer the interested reader to Goel et al. (2020, Lemmas A.2 and A.9) for quantitative estimates of n- (o) for ReLU and sigmoid activations. Finally, the expression (53) is essentially the same as the one appearing in Nguyen Mondelli (2020, Theorem 3.3), As in this paper, note that this expression is small if n. grows at most polynomially with d, in which case the exponential term in d dominates the polynomial term in n.
B.6 PROOF OF THEOREM 6
By Proposition 5, there exists 8 >> 0 such that, with probability least 1 6, the residual network (3) satisfies the (M, (M)ooall PL condition around its initialization, with
C1
M
and
nq
n//nq'
for C3 and ca depending on σ. Let us now apply Proposition 8 with f(h, (V, Ain V The only assumption of Proposition that requires some care check is that the PL condition holds for the value of μ given by equation (22). Since the (M, 1o-coc PL condition implies the (M, local PL condition for any j E (0, p), the case
Se
C4 max( M K, Mx MAMx) nynq
sup V (L))))
M
LEN-
with Mx. MA. Mn. and K defined in Proposition 8. Due the initialization scheme of Section we have, for any input x E X 1 (0) = (0), hence (=(x)= B" A+))= 0 since > d + d'. As consequence, {²(0)= =
||a:1? Therefore, the condition becomes
1
dd || 64n4g3 max MR MAMx)222K
where we replaced M by its value. Define C to be equal to the constant the right-hand side. Then, & C, we can apply Proposition 8, which gives several guarantees. First, the gradient flow well defined on R,. Moreover, the proposition and the expression of given above yield the bound on the empirical risk. In particular, the empirical risk converges uniformly zero. Furthermore, Proposition 8 shows the uniform convergence of the weights as f + sc. Finally, the proposition ensures that the sequences (A") LeeN and (B") each satisfy the assumptions of Corollary 10, and that (2)) LEN- 11 gkg. satisfies the assumptions of Proposition 9. We can therefore apply Theorem 14, with ʃ defined above and π equal to the identity. This gives the uniform convergence the weights as L - co. The four asymptotic statements of Theorem 6 are then a consequence of Proposition 15.
according to the above, as soon as 1. EE ||
Remark 3. A close examination of the quantities involved in the definition of reveals that it depends only on X, a, n, and q. In particular, it does not depend the dimension m.
35

---

Published as a conference paper at ICLR 2024
C SOME TECHNICAL LEMMAS
We start by recalling the Picard-Lindelis thcorem (see, c.g., Luk, 2017, for a self-contained presenta- tion, and Amold, 1992, for textbook). Lemma 16 (Picard-Lindelöf theorem). I C R4 be an interval. for some E (0, oo]. Consider the initial value problem
[ ((U((), r)dr, s SE L
((s)=Uo
(54)
Rd continuous and locally Lipschitz continuous in its first variable. Then the initial value problem is well defined interval [0, Tmax) C I, i.e. there exists unique maximal solution on this interval. Moreover, if < <. then [s(s) tends to infinity when s tends to Tmax- Finally. if r(rr) is uniformly Lipschitz continuous for in any compact, then Tmax T.
where g I-
We define time-dependent dynamics (54) for generality, but the time- imme-independent case ((s) = Uo + Jo ((U(r)) is also of interest. this case, the existence and uniqueness of the maximal solution holds if is locally Lipschitz continuous, and the solution is defined on / if is Lipschitz continuous. Besides, the first statement of Lemma 16 (existence and uniqueness of the maximal solution) also holds if IR is replaced by any (potentially infiniteddimensional) Banach space.
The next lemma gives conditions for the existence and uniqueness the global solution of the initial value problem (54) when the assumption of continuity of g in its second variable is removed, thereby generalizing the Picard-Lindelle Lindelöf theorem.
Lemma 17 (Caratheodory conditions for the existence and uniqueness of the global solution of an initial value problem). Consider the initial value problem
U(s) = Un + S. g(U(r), S [0.
where g IRd [0, Rd is measurable and the integral is understood in the sense of Lebesgue integration. Assume that g(-.r) r) is uniformly Lipschitz continuous for almost all r E [0, 1], and that g(0, r) 0. Then there exists unique solution to the initial value problem, defined on [0, 1].
Proof. The proof is a consequence of Filippov (1988, Theorems 1, 2, and 4). More specifically, denote by C the uniform Lipschitz constant of g(- r). According Filippov (1988, Theorems 1 and 2), under the conditions of the lemma, there exists a unique maximal solution to the initial value problem. Let us now consider restricted version of the problem, where 9 is defined on D x [0. 1]. with D compact of Rd large enough to contain in its interior the ball of center 0 and radius |10 || (C). There exists a unique maximal solution to this problem as well, also according to Filippov (1988, Theorems 1 and 2), and, according to Filippov (1988, Theorem 4), it is defined until it reaches the boundary of D x [0, 1]. which it reaches at some point (U*, s*). If s* < 1, means that U* is on the boundary of D, and in particular that ||U*||| > ||ol] exp(C). But, on the other hand, for almost every r E [0, 1).
(r)(U()) < lg(0,))) r)ll + (l)(UUr), -g(0,)) CI(U(())). Hence, by Grönwall's inequality, for s < s*.
[|U(s)] < | eXp(C).
Thus, |U*|]| & ||U|| exp(C), which impossible. Hence the maximal solution of the restricted
problem is defined on [0, 1]. Furthermore, the maximal solution of the original problem coincides with the restricted one whenever U ((s) E D, which is the case for every s E [0, 1]. hence the maximal solution is defined on [0, 1].
The next three lemmas recall well-known results from linear algebra, analysis, and random matrix theory. Recall that Smin and Amin denote respectively the minimum singular value and eigenvalue of a matrix.
Lemma 18. Let A, Raxr and B Rr Then
Smin (A A') > smin (A) ||
If 2 r, then |AB|| r Smin (A) Bl]r F. Furthermore, is 2 r, then || 2 ||||| pSmin (B).
36

---

Published as a conference paper at ICLR 2024
Proof. The first statement is consequence of, e.g., Loyka (2015), which establishes that Smin (A A') e Smin(4) (A) Smax (A'), yielding the first inequality since Smax (max((A) (A') = ||A|12 < ||A||p. As for the second one, we have
||B Tr(BB AT Tr(BBT A) Amin A)) Ami (A B)
Since m S T, the rightmost quantity is equal to mmnn (ain(A]] Bl] proving the second statement of the lemma. The third statement is similar.
Lemma 19. Let (EE C E. where E is a Banach space, such that Cxay converges uniformly to eoo.y when x 0o, and converges uniformly to ex,c0 when y - oc. Then there exists E F. such that
lim Ca,y 2y0
lim 6x 40200
lim Coc.y = 6x0-
y2000
Proof. Let > 0. Since Zx converges uniformly to eoo.y + oo, there exists To E 14 such that, for F1, 22 Fy and y E R+,
|ex Crzyyll & E
2"
Similarly, there exists yo E R2 such that, for E R.+ and y/1 U2
yo.
1erm
Cr, &
Hence, for 21, x2 > and y1, > yo.
|e vw2.v2l || & ||em Exi ||ezi. Ex2.12ll E. We conclude that (eow))eR+ E E.
Cauchy sequence, which therefore converges some limit
Lemma 20. Let W RR be a standard Gaussian random matrix. Then, for Mw e V2, with probability at least exp(- (M -1)mm one has [|W|| F & Mw / /
Proof. The quantity [|W]] follows a chi-squared distribution with qm degrees of freedom. Hence, according to Laurent & Massart (2000, Lemma 1), for x > 0,
P([I)] qm 2/qmx + 2x) & exp(-2).
Taking x (M -1)mm
we see that
16
1 2/2 (M 1)gm, 2 2 My 1qm &
2 2qmm
where the bound follows from Mw > V2. Since furthermore 2r & (M 1)4m, obtain
2 + 2x & (My 1)gm,
and thus
P([I > Mi & P([ qm > 2//mm + 2x) & exp(-x)),
yielding the result.
Finally, the last lemma of the section gives a lower bound on the smallest singular value of matrix of the form of (A), where random matrix. The lower bound involves the Hermite transform of which is defined in Section B.
a bounded function applied element wise and belongs a family of
Lemma 21. Let a be a function bounded by some M, > O. Let W E mm be a standard Gaussian random matrix, and X E Rdxx random matrix with i.i.d. unitary columns independent of W. Then, for any integer r > 2, there exists 8 > such that, with probability at least 1 - 8, the smallest singular value of ( X) is greater than / (o), where Ty ())a)) is the r-th coefficient in the Hermite transform of a. Furthermore, the the following expression for 5 holds:
3mmm(or) 2n² exp (- 20n2/+
8 nexp (-64M3n
where C. is the sub- Gaussian variance proxy of the columns of Vdx.
37

---

Published as a conference paper at ICLR 2024
Proof. Denoting by w; the i-th row of W and letting
M, oxX w (w;
our goal is to lower bound the smallest eigenvalue value Amia (M) of M = Ex= M, Observe that
[[M[X)= mB mB (o(x w [o(xx)
mB - (o(Vax) i T ( (((((v))) x)
Letting Amin (E(M[X)) be the smallest eigenvalue of this matrix and > 2 be integer, Nguyen & Mondelli (2020, Lemma 3.4) show that, with probability least 1 2n² exp( 2C12/F) over the matrix X,
Amia (B(M]X)) (o)
(55)
We now apply matrix Chernoff's bound to lower bound with high probability the smallest eigenvalue Amin (M of M conditionally on X, as a function of Auii (B(( By Tropp (2012, Remark 5.3), we have, for E [0, 1].
P(Amin (M) & tAmin (B(M[X)) n exp ( (- (1-
(1-t)) -2-
(B(M[X))
2R(X)
where R(X) is almost sure upper bound the largest eigenvalue of M, which we can take equal to M since the largest eigenvalue of M, is equal to (l,(2,)) & Mnn. Taking t = 1/2,
we obtain, on the event [Amiu (E(M[X)) (a)((r)
PP PP(Min (M) Amnn (E(M(X))
x > 1 nexp (-
3m)
2
64M3n
thus, on the event Danin (E(MIX))
mmn((a)
mn
> nxxp (-
3mmn (a)
Amiu (M)
16
64M3n
Using (55), we obtain
P (Amia( (M)
mn)(o)
1 nexp
3mnf(or)
))
(mmin ( > mnf(a)
64M3n
1 nexp
3mn
(1-2neeex 2x2 (-
2n2
d
64M3n
3mmn
exp (- 2Cn2/r
d
n nexp
2n²
64M3n
D COUNTER-EXAMEE FOR THE RELU CASE.
This section gives proof sketch to illustrate that, with the ReLU activation o a + max(0, x). the smoothness of the weights can be lost during training. More precisely, we show a case where successive weights
distance O ) initialization and at distance (2(1 after training.
For the sake of simplicity, we will assume that the depth is even, and denote as 2L. We place ourselves in one-dimensional setting (i.c., d = 1). The parameters are (wi,... W2r) 2 and the residual network writes as follows, for an input æ R:
ho(t)
1 ho+1(1) ha (1) (wk+ (1), k (O....,21 2L 1}.
2T.
38

---

Published as a conference paper at ICLR 2024
We consider sample consisting of single point (x, Cx) R2 with C (independent of L), and define the empirical risk E(t) (h22(t) Cz)?. The risk is minimized by gradient flow.
The weights are initialized to (0) 1-1 For a E R. we have that (t) >> 0 for all 2L
Note that the argument of a on the odd layers is negative. Therefore, by definition of a. the gradient of the loss with respect to the odd layers is zero and we have, for l E {0.... L-1}. is positive on the even layers, and thus,
E .O...., 2
(2)+ (2k++1(0) (0). On the other hand, the argument
I. hea (t) III (1
22
2L
j=1
As consequence, the gradient flow equation for the even layers is, for is E {1,.... ......,,, L),
dw2k dt
(t)
de
(t) = 2x (c
L II 1+ j=1
w
II J 3f1
L
(1+
W2y (t)
Owyk
2L
2L
Due to the symmetry of these equations for k E (l..... L. and the fact that all the w2s are equal, the parameters on each even layer coincide at all times and are equal to w(t) such that
dw dt (t) 2x (c-
11-(11 (1+
wlll
(1 )
w(())
2L
2L
An analysis of this ODE reveals that wt) tends as
w" satisfying that
w L =C
(1
(56)
2T.
This can be seen by letting yt) C (1 wit) )d and applying Grönwall's inequality to y.
2
Therefore, as t + oo, one has W2++ (t)
- zI and (2x(t) + w*, where (56) implies that
w* > 2 (C). This shows that the final weights are not smooth in the sense that the distance between two successive weights is (2))).
This result contrasts sharply with Proposition 7, which shows that successive weights remain at distance O(+) throughout training. when initialized as a discretization of a Lipschitz continuous function, and with smooth activation function. In fact, Proposition 7 can be generalized to any initialization such that successive weights are at distance O(2)) at initialization, which is the case in the caunter-exampl.. This means that the only broken assumption in our counter-example is the non-smoothness of the activation function. This non-smoothness causes the gradient flow dynamics though the weights initially close each other,
for two successive weights to deviate, because they are separated by the kink of ReLU at zero.
E EXPERIMENTAL DETAILS
Our code is available at
https://github. cww/coom/mc..io. http:/hwww...oommccceeeelaaadaad http:/iccchhaalr........
impl aapliciitregullee zat.ion-eeeett---odee We use Pytorch (Paszke et al., 2019).
Synthetic data. To ease the presentation, we consider the case where 9 d = d', and we do not train the weights A and B", which therefore stay equal to the identity. The sample points (I4, Viligin follow independent standard Gaussian distributions. Note that does not hurt to take and independent since, in this subsection, our focus is on optimization results only and not on statistical aspects.
Large-depth limit. We take n = 100, d = 16, 32. We train for 500 iterations, and set the learning rate to x 10 The scaling of the learning rate with L is the equivalent of the L factor in the gradient flow (4).
Long-time limit. We take n = 50, 16, m 64, = 64, and train for 80,000 iterations with a learning rate of 5L. 10-3
39

---

Published as a conference paper at ICLR 2024
Real-world data. We take L = 256. The first layer is a trainable convolutional layer with kemel size x 5, a stride of 2, a padding of 1, and 16 out channels. We then iterate the residual layers
1 Ne4 1 + bng, ( (bn (cconvf (14)))) E {0................. .0,....,4... L 1,,,,
where convi are convolutions with kemne size 3, stride of 2, and padding of 1. and bn are batch normalizations, as standard residual networks (He et al., 201 The model is trained using stochastic gradient descent on the cross-entropy loss for 180 epochs. The initial learning rate is 4 x 10-2 and is gradually decreased using a cosine learning rate scheduler.
Normalization. The residual layers considered in the real-world case have batch normalization layer (see formula above). We observe empirically that implicit regularization towards a neural ODE still holds in this case. However, these layers as discussed in Section 4.3, some of our results extend to a setting where we only assume that the residual connection is Lipschitz-continnn in particular the case where layer normalizations are added to the architecture, although this should clearly necessitate rigorous and separate mathematical analysis. Finally, note that a connection has been drawn between batch normalization and scaling factors (De & Smith, 2020).
present the models consider. Nevertheless,
function. The intuition suggests that this should include
Additional plot. To complement Figure 3, we display the average (across layers) of the Frobenius norm of the difference between successive weights in the convolutional ResNets after training on CIFAR-10, depending on the initialization strategy. The index i corresponds to the index of the convolution layer. Results are averaged over runs. We that a smooth initialization leads to weights that are in average an order of magnitude smoother than those obtained with an i.i.d. initialization.
10"
Mean A Frob. norm
i=
10-1
-2
Smooth init.
i.idd. init.
Figure 4: Average (across layers) of the Frobenius norm of the difference between two successive weights in the convolutional ResNets after training on CIFAR-10, depending on the initialization strategy.
40


Reviews:
[{"review_id": "24MRJ1UBrw", "reviewer": "Reviewer_NuSB", "summary": "This paper shows that an implicit regularization of deep residual networks towards neural ODEs still holds after training with gradient flow. In particular, this is shown for both large depth (via gradient clipping in finite time) and large time limits (assuming PL condition). Along the way, interesting aspects of the problem are discussed and the results are demonstrated numerically on simple simulations as well as on CIFAR10.", "questions": "Please see above. Besides:\n\n\n- Would it be possible to extend the approach to incorporate layer normalization?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The derived results and the neural ODE limits seem rather interesting. The theoretical analyses seem rigorous and their discussions are quite thorough, though I haven't read the proofs in detail. It could be that I have missed a similar related work in the literature, but I do think this seems to be an important contribution, and thus could spark interesting future work on implicit regularization as well as furthering understanding of residual networks via the toolbox of ODEs. Besides, the paper is, in general, very well written, and nicely presented.", "weaknesses": "- One of the key drawbacks is requiring weight-tying or weight sharing. While results with weight-tied networks don't look bad, still this would have been nice to take care of. Can the authors elaborate on where all and where exactly does it interfere with the proof strategy?\n\n- The other issue is that the linear overparameterization $m > c_1 n$ would, in practice, be somewhat unreasonable and is thus a bit of a stretch. Maybe this is a norm in the literature, but do the authors have some ideas how to go around this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ntzmg8wOsu", "reviewer": "Reviewer_6832", "summary": "This paper shows that the dynamics of a residual connection converges that to of an ODE. \n\nFirst, the authors show that under certain conditions (like the boundedness of the Frobenius norms of the weight matrices) for finite training time as the depth of the network tends to infinity the dynamics of a resnet converge to that of an ODE. This is assuming that the model is trained with clipped gradient descent. \n\nSince here the authors show that the Frobenius norm of the different of the weight matrices is upper bounded by a value that is O(exp(T)), the to show the result for when both T→ \\infty and L → \\infty, the authors also assume that residual network satisfies a PL condition. \n\nTherefore, the authors show that under these condition as T→ \\infty and L→ infty, the dynamics of a residual network tend to an ODE and converges to a function that interpolates the training data.", "questions": "For the experiments on cifar, the authors show the plot of a random coefficient and its value as one goes deeper into the layer. However, it would be more beneficial to see the difference in Frobenius norm of the weight matrices as well.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is very well written and easy to follow, where the results and their implications are clearly stated.\n\nThe authors show weights of a resnet convert to an ODE while considering the dynamics of the network as well, which is new and interesting. \n\nGiven that the result shows that the dynamics of the resnet architecture converges to that of an ODE, and therefore as the authors mention one can utilize existing generalization bounds for neural ODEs etc can under certain conditions now be applied to resnets, given the results that has been shown by the authors.\n\nThe authors also show that their results/assumptions hold very well on their synthetic dataset.\n\nFor the cifar dataset, the authors show that their results hold when using smooth activations and the dynamics may deviate from neural ODE if one uses a non-smooth activation like ReLU.", "weaknesses": "The results are under the condition that the resnet is trained under clipped gradient flow, which is often not used in practice. However, I do understand that the main contribution of the paper is to characterize the behaviour of the model in the limit, so it shouldn’t affect the main contribution of the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9k10nXyEqP", "reviewer": "Reviewer_Hmrt", "summary": "The paper establishes a connection between deep residual networks and neural odes.  Prior works have attempted to provide a theoretical framework for the same, but they come with several shortcomings -- like considered only simplified models with linear activations. The authors train the neural network with gradient flow and show that in the limit of $L \\rightarrow \\infty$, the network converges to a discretization of neural ode.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Most prior works in this domain make simplifying assumptions which prevents their analysis from being directly applicable to practical models. The authors overcome those assumptions", "weaknesses": "N/A\n\nI am giving a score of 6, given my inability to appropriately verify all the proofs in the main text and appendices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FZYCmCxtRQ", "reviewer": "Reviewer_Htsq", "summary": "The paper establishes that initialising the network with an ODE results in preserving smoothness of weights throughout training.  The authors do this through the framework of Polyak-Łojasiewicz condition.", "questions": "Q 1: It seems to me that both fully-connected and convolutional neural networks would be the particular cases where the ODE parameter matrices A, V, W are considered sparse (some or most of the entries are fixed to zero and the rest are optimised according to Equation (5)). The experiment, depicted in Figure 3, suggests that results in Theorem 4 might be expanded to the more generic sparse scenarios.  Might be good if the authors could confirm this observation.\n\nQ2: If it happens that the weights are initialised randomly, for every finite depth of the neural network (as there’s finite time and finite depth), there could be still a Lipschitz-continuous function (ii) over that set (as locally Lipschitz function on a compact set is Lipschitz as per, e.g. https://faculty.etsu.edu/gardnerr/5510/cspace.pdf ). However, what’s appearing in Figure 3 appears to go beyond literal application of Theorem 4: not only the function is compact but also it appears that the Lipshitz constant is preserved to be small during the optimisation. \n\nQ3: Another interesting aspect is that  it would be intuitive to see smaller Lipshitz constant could be beneficial for out-of-distribution recognition. I wonder if the authors could clarify on this aspect? To what extent would the proposed derivation generalise for higher-than-two layers of multilayer perceptrons?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Pros:\n\n- (Significance) It is an important result that shows the impact of initial conditions on the training of ResNets; furthermore, it is crucial that while the paper is mostly theoretical, the settings the authors propose are realistic and can generalise to the practical ResNet models.  I also checked the derivations and could not find any problems with them. \n\n- (Quality) The paper is well written, with thoroughly addressing the reproducibility aspects (through both derivations and the experiments)\n\n- (Clarity) The paper is clearly written, with great attention to detail\n\n- (Originality) While the notion of interpretation of ResNets as a discretisation of (Neural) ODEs has been a widely-discussed topic, it has not been analysed in the opposite direction. The authors address an intriguing question of whether every ResNet corresponds to a (Neural) ODE and show, to my knowledge, previously undocumented, impact of initial conditions on whether the ResNet can be presented as a discretisation of an ODEs", "weaknesses": "Cons: \n\n-  (Significance elements) There are some limitations to the analysis, which are related to implicit generalisation to the convolutional and other sparse scenarios (see questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)