Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Authors: Michael Eli Sander
Pierre Ablin
Gabriel Peyré
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Learning theory
ResNets
Neural ODEs

Abstract: Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog of Residual Neural Networks (ResNets). We investigate whether the discrete dynamics defined by a ResNet are close to the continuous one of a Neural ODE. We first quantify the distance between the ResNet's hidden state trajectory and the solution of its corresponding Neural ODE. Our bound is tight and, on the negative side, does not go to $0$ with depth $N$ if the residual functions are not smooth with depth. On the positive side, we show that this smoothness is preserved by gradient descent for a ResNet with linear residual functions and small enough initial loss. It ensures an implicit regularization towards a limit Neural ODE at rate $\frac1N$, uniformly with depth and optimization time. As a byproduct of our analysis, we consider the use of a memory-free discrete adjoint method to train a ResNet by recovering the activations on the fly through a backward pass of the network, and show that this method theoretically succeeds at large depth if the residual functions are Lipschitz with the input. We then show that Heun's method, a second order ODE integration scheme, allows for better gradient estimation with the adjoint method when the residual functions are smooth with depth. We experimentally validate that our adjoint method succeeds at large depth, and that Heun’s method needs fewer layers to succeed. We finally use the adjoint method successfully for fine-tuning very deep ResNets without memory consumption in the residual layers.

Do Residual Neural Networks discretize Neural Ordinary Differential Equations?
Michael E. Sander ENS, CNRS
Pierre Ablin
Gabriel Peyré ENS, CNRS Paris, France pey.heees.prreees.f..
Université Paris-Dauphing CNRS
Paris, France
Paris, France
mi chael. sander michael..anderGens
pierreablinOgmii..om
Abstract
Neural Ordinary Differential Equations (Neural ODEs) are the continuous analog Residual Neural Networks (ResNets). We investigate whether the discrete dynamics defined by ResNet are close to the continuous one of Neural ODE. We first quantify the distance between the ResNet's hidden state trajectory and the solution of its corresponding Neural ODE. Our bound is tight and, on the negative side, does not go to 0 with depth N if the residual functions are not smooth with depth. On the positive side, we show that this smoothness is preserved by gradient descent for ResNet with linear residual functions and small enough initial loss. It ensures an implicit regularization towards limit Neural ODE at rate N. uniformly with depth and optimization time. As a byproduct of our analysis, we consider the discrete adjoint method to train ResNet by recovering the
use of memory- activations on the fly through backward pass of the network, and show that this method theoretically succeeds large depth if the residual functions are Lipschitz with the input. We then show that Heun's 's method, a second order ODE integration scheme, allows for better gradient estimation with the adjoint method when the residual functions are smooth with depth. We experimentally validate that our adjoint method succeeds at large depth, and that Heun's 's method needs fewer layers to succeed. We finally use the adjoint method successfully for fine-tuning very deep ResNets without memory consumption in the residual layers.
Introduction
Problem setup. Residual Neural Networks (ResNets) 20 21 keep on outperforming state of the art 6). and more generally skip connections are widely used in various range of applications 42 14 ResNet of depth N iterates, starting from 2g Rd In+1 = Fn + f 0
in computer vision 46
and outputs final value IN Rd where is neural network called residual function. In this work, we consider a simple modification of this forward rule by letting explicitly the residual mapping depend on the depth of the network:
Fn+1 = Fn N ( NN)
On the other hand, a Neural ODE S uses a neural network pe(o, (z, s), that takes time into account, parameterise vector field 24 in differential equation, follows,
dr ds (e(r(s),
with
(0)) Z0.
(2)
and outputs final value rx1) Rd. the solution of Eq. 2 The Neural ODE framework enables leam-
ing without storing activations (the x, 's) using the adjoint state method, hence significantly reducing
the memory usage for backpropagation that can be bottleneck during training 43
34
48
[6]
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Neural ODEs also provide a theoretical framework to study deep learning models from the continu- ous viewpoint, using the arsenal of ODE theory
Neural ODE like
Neural ODE unlike
40 25 41). Importantly, they can also be seen as the continuous analog of ResNets. Indeed, con- sider for N an integer, the Euler scheme for solving Eq.
Number of layers
2 with time step VV starting from To and iter-
ating 1 Ii + N. (zn ))) Under mild as- sumptions on ye. this scheme is known to converge to the true solution of Eq. 22 N goes to +.C. Also, if θ = (0% - and pe N) f(,, 0x then the ResNet equation Eq. ① corresponds to a Euler discretization with time step & of Eq. (2). However, for given ResNet with fixed depth N and weights, the activations in Eq. ① can be far from the solution of Eq. (2). This is illustrated in Figure []] where we show that deep ResNet can easily break the topol-
Input of the network
Input of the network
Figure I: Trajectory of ResNets with 300 layers.
Left: learn x -/2 trajectories are smooth and do not intersect. Right: we learn x + =x² trajectories are not smooth and intersect. 2
ogy of the input space, which impossible for a Neural ODE. In this paper, we study the link between ResNets and Neural ODEs. We make the following contributions:
In Section 3] we propose framework to define a set of associated Neural ODEs for a given ResNet. We control the error between the discrete and the continuous trajectory. We show that without additional assumptions on the smoothness with depth of the residual functions, this error does not go to O as N -> oo (Prop. [ However, we show that under some assumptions on the weight initialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depth and training time) approach a Lipschitz function as the depth N of the network goes to infinity, speed x (Prop. 2 and Th. 1.. This result highlights an implicit regularization towards limit Neural ODE.
Section 4 we investigate a simple technique to train ResNets without storing activations. Inspired by the adjoint method, we propose recover the approximated activations during the backward pass by using a reverse-time Euler scheme. We control the error for recovering the activations and gradients with this method. We show that if the residuals of the ResNet are bounded and Lipschitz
continuous, with constants independent of N, then this error scales in O ++)) (Prop. 3 Hence, the adjoint method needs large number of layers to lead to correct gradients (Prop. We then consider a smoothness -eeeenddess reconstruction with Heun's method to bound the error between the true and approximated gradient by a term that depends of the residual functions, hence guaranteeing better approximation when successive weights are close one another (Prop. 5 5nd ().
times the smoothness in depth
In Section 5 on the experimental side, we show that the adjoint method fails when training ResNet 101 on ImageNet. Nevertheless, we empirically show that very deep ResNets pretrained with tied weights (constant weights: 0N" 0 Vn) be refined -using our adjoint method- on CIFAR- 10 and ImageNet by untying their weights, leading to a better test accuracy. Last, but not least, we show using ResNet architecture with heavy downsampling in the first layer that our adjoint method succeeds at large depth and that Heun's method leads to a better behaved training, hence confirming our theoretical results.
2 Background and related work
Neural ODEs. Neural ODEs are class of implicit deep learning models defined by an ODE where a neural network parameterises the vector field 44
24 Given an input To. the output of the model is the solution of the ODE at time 1. From a theoretical viewpoint, the expression capabilities of Neural ODEs have been investigated
40
39
45|
29
36
25 and the Neural ODE framework has been used to better understand the dynamics of more general architectures that
12
include residual connections such as Transformers 38 27.. Experimentaly, Neural ODEs have been successful in a various range of applications, among which physical modelling [18] I1] and generative modeling [ 17.. However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models. Neural ODEs fail to compete with

---

ResNets on ImageNet, and to the best of our knowledge, previous works using Neural ODEs on ImageNet consider weight-tied architectures and only achieves the same accuracy as a ResNet18 [50]. It has also been shown that Neural ODEs sometimes do not admit a continuous interpretation at all
32
Implicit Regularization of ResNets towards ODEs. Recent works have studied the link between ResNets and Neural ODEs. In (9, the authors carry experiments to better understand the scaling behavior of weights in ResNets
function of the depth. They show that under the assumption that for the weights of the ResNets (with 0 < B < 1)
there exists a scaling limit (s) = N lim 0.V [Na]
and if the scale of the ResNet is J with 0 < a < 1 and + B
then the hidden state of the ResNet converges to a solution of a linear ODE. In this paper, we are interested in the case where which seems more natural since it is the scaling that appears in Euler's 's method with step In addition, we do not assume the existence of a scaling limit ((s) = lim 0 [Naj- In subsection
N
we demonstrate the existence of this scaling limit in the linear setting, under some assumptions. The recent work [T0] shows results regarding linear convergence of gradient descent in ResNets and prove the existence of an 2 -Hölder continuous scaling limit as N - 3o with a scaling factor for the residuals in VN which is different from ours ( In contrast, we show that our limit function is Lipschitz continuous, which is a stronger regularity. This is be linked with the recent work of (31,, where the authors show that whereas the VN scaling corresponds to the proper one for standard i.i.d. initializations, マ is the proper scaling for smooth initialization to obtain non-trivial behaviour (other choices lead to explosion to identity [1I). More generally, recent works have proved the convergence of gradient descent training of ResNet when the initial loss is small enough. This include ResNet with finite width but arbitrary large depth [TS] 26 and ResNet with both infinite width and depth [28] 3. These convergence proofs leverage an implicit bias toward weights with small amplitudes. They however leave open the question of convergence of individual weights as depth increases, which we tackle in this work in the linear case. This requires showing an extra bias toward weights with small variations across depth.
3.2
Memory bottleneck in ResNets. Training deep learning models involve graphics processing units (GPUs) where memory is practical bottleneck 43 34 48 Indeed, backpropagation requires to store activations at each layer during the forward pass. Since samples are processed using mini batches, this storage can be important. For instance, with batches of size 128, the memory needed to compute gradients for a ResNet 152 on ImageNet is about 22 GiB. Note that the memory needed to store the parameters of the model is only 220 MiB, which is negligible compared to the memory needed to store the activations. Thus, designing deep invertible architectures where one can recover the activations on the fly during the backpropagation iterations has been an active field in recent years [16] (37 (23.. In this work, we propose to approximate activations using reverse-time Euler scheme, as we detail in the next subsection.
Adjoint Method. Consider a loss function L(IN) for the ResNet equations are
The backpropagation
[O0 1N1) V V., L [1+ Oz - (3)
N
Now, consider a loss function (xx( for the Neural ODE (2). The adjoint state method 35 g gives
VoL = [ delpo(z(s), s)] Lds, - -((s) [0..p((((s) s)] V (2))
(4)
Note that if θ (0.) N- and pe( (., f) {(,, ,N), then Eq. corresponds to Euler discretization with time step + of Eq. (4). The key advantage of using Eq. is that one can recover ass) on the fly by solving the Neural ODE (2) backward in time starting from x(1). This strategy avoids storing the forward trajectory (a(s))=efo.1) and leads to O(1) memory footprint [8]. In this work, we propose to use discrete adjoint method by using a reverse-ting time Euler scheme for approximately recovering the activations in a ResNet (Section 4). Contrarily to other models such as RevNets [16] (architecture change) or Momentum ResNets [37 (forward rule modification) which rely on an exactly invertible forward rule, the proposed method requires no change at all in the network, but gives approximate gradients.
3

---

Notations. For k E N. is the set of functions f: Rd - Rd l times differentiable with continuous k differential. If f E C', (z)())( the differential of at æ evaluated y. For K C. R compact, ||-II a norm and f a continuous function on RR we denote Il suPpek [z)()
3 ResNets as discretization of Neural ODEs
In this section we first show that without further assumptions, the distance between the discrete trajectory and the solution of associated ODEs can be constant with respect to the depth of the network if the residual functions lack smoothness with depth. We then present positive result by studying the linear case where we show that, under some hypothesis (small loss initialization and initial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goes to infinity. We show that this convergence is uniform with depth and optimization time.
3.1 Distance to ODE
We first define associated Neural ODEs for a given ResNet. Definition 1. We say that a neural network pe Rd x R + Rd smoothly interpolates the ResNet Eg. ① if (p is smooth and Vn E {o,... N - }}. po(-, n N (N,0N)
Note that we omit the dependency of ④ in N to simplify notations. For example, for given ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating the residuals, or by interpolating the weights. Indeed, one can interpolate the residuals with (p s) or interpolate the weights with n=11
(n + 1- N 0N + (Ns n: ON+1) when s E (P)(.))) s) = f(,, -==1-Ns)0,, (N for s E n, then both interpolations are identical and one can simply consider (pe(z, s) = f(z, ON), V(z, s).
n
N
N
I 0% does not depend on
IN
N
We now consider any smooth interpolation ye for the ResNet [ and Euler scheme for the Neural ODE (2) with time step N.
Proposition 1 (Approximation error). We suppose that se is C22 and L-Lipschitz with respect to x, uniformly in s. Note that this implies that the solution of Eq. (2) is well defined, unique, C², and that
the trajectory is included in some compact K C IRd. Denote CN = ||0,pe + 0xppelse)) Kx [0.1] Then one has for all n: || 21ll < 2NL c2-1ON if 0 and I|,, (
Cx if =. 2N
For full proof, see appendix A.I Note that this result extends Theorem 3.2 from 44 to the non- autonomous case: our bound depends on 0.pe. Finally, our bound is tight. Indeed, for (e, [|]]]
as b for a, b E Rd, we get 0, + (ayelpe) = a, I. = 0 and [|a((1) xN
2N
The tightness of our bound shows that closeness to the ODE solution is not guar- anteed, because do not know whether CN //N + 0. Indeed, consider first the residual inter- polation (2, s) (nn + Ns) ON + (Ns n)f(x, ON 2N+1)) ) # and the simple case where d = O. We get 0,po (z. s) = (2f 1 - f(r, eN 1) " which corresponds to the discrete derivative. It means that although there is a 1
Implication.
N factor in our bound, the time derivative term without further regularity with depth of the weights, which is the heart of subection 3.2 usually scales with N: Ex O(1). As first example, consider the simple case N
where [(x R,)) n. This gives uN = xo +
(N-1) while the integration of the Neural ODE (2)
2
leads to (1) zo + 7 this shows that weights cannot scale with depth when using the residual interpolation. Now, con- sider the weight interpolation, 0:N (-1)" and suppose I is written f(x, 0) 02. This gives (p((.,)) s) = (2.Ns (2n + 1))² when E [ m+1l. Integrating, we get (11) = while the output of the ResNet xN = 1. Hence ||ر - x(1i|| bounded. Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain Neural ODE in the large depth limit. Intuitively if the weights are initialized close from one another and they are updated using gradient descent, they should stay close from one another during training, since the gradients in two consecutive layers will be similar, as highlighted in Eq. (3). Indeed, we
N because (e 8) = Ns, so the ||æl x(1]] is not small. Intuitively,
is also not small, even though the weights are
see that if on and Tx+1 are close, then Vz, L and V 2 L are close, and then if 0 and @N are close. V x L and V L are also close. In subsection 522 we formalize this intuition and present 4+1

---

positive result for ResNets with linear residual functions. More precisely, we show that with proper initialization, the difference between two successive parameters is in N during the entire training. Furthermore, we show that the weights of the network converge to a smooth function, hence defining a limit Neural ODE.
3.2 Linear Case
As further step towards theoretical understanding of the connections between ResNets and Neural ODEs we investigate the linear setting, where the residual functions are written f(x 0) = Or for any 0 e Rdxd, It corresponds to a deep matrix factorization problem [5T I. As opposed to these previous works, study the infinite depth limit of these linear ResNets with a focus on the learned weights. We show that, the weights are initialized close one to another, then at any training time, the weights stay close one to another (Prop. 2 and importantly, they converge to smooth function of the continuous depth s as N 20 (Th. All the proofs are available in appendix A
Setting. Given training set (Øg, yk [e[n] in Rd, we solve the regression problem of mapping Js to yk with a linear ResNet, i.e. f(x, 0) = Ox, of depth N and parameters (0 0 The ResNet
therefore maps vy. to IT= Ig where IT INN (Ia + AN = (Ia + - ... (Ia + N ). It is trained by minimizing the average errors | IA yk 112. which is equivalent to the deep matrix factorization problem:
a argmin sepp- T. ON = | - -B
where || Tr(AAA*), E is the empirical covariance matrix of the data: Σ == nे Eh= T and B = Σx= k= Uk Σ-1, As is standard, we suppose that E is non degenerated. We denote by M > 0 (resp. m > 0) its largest (resp. smallest) eigenvalue.
BY N and write the gradient V (t) = VoN L I (II - - Intuitively, as N goes to +00, the products VI II and II,.. should converge to some limit, hence we see that N scale as 1. Therefore, we train 0 by the rescaled gradient d flow dt (1) = NV (4) to minimize I and denote e~ = L
Gradient. We denote II = (La +
(La
(IL and IIW. == (Ia + (t), (t)) The chain rule gives N
ON N (Ia + N
&N
N
n
Two continuous variables involved. Our results involve two continuous variables: [0, 1] is the depth of the limit network and corresponds to the time variable in the Neural ODE, whereas / E R, the gradient flow time variable. As is standard in the analysis of convergence of gradient descent for linear networks, we consider the following assumption:
Assumption 1. Suppose that at initialisation one has / (EN(O)
1V2Me and I0 (0) S 1.
Assumption T is the classical assumption in the literature 51 to prove linear convergence of our loss and that the o, (1)'s stay bounded with L. Note that this bounded norm assumption implies that (0) = 0 ( ). This is in contrast with classical initialization scales in the feedforward case where the initialization only depends on width [19]. However this initialization scale is coherent with those of RexNets for which the scale has to depend on depth [47] (31. In addition, the experimental findings in suggest that the weights in ResNets scale in No with 3 O.
We now prove an implicit regularization result showing that if at initialization, in addition to assump- tion 1 the weights are close from one another (O(3) y )), they will stay at distance O / ): the discrete derivative stay in O which is central result consider the infinite depth limit in our Th.
Proposition (Smoothness in depth of the weights). Suppose assumption 7 Suppose that there exists Co > 0 independent of n and N such that 110N ON (0)] < N. Then, Vit E Rt, +oc. Moreover, there exists C 0 Ouch that
|10NN (1)I < J, and 0% (1) admits a limit w
Vt R4, (1) 0N (1)
For full proof, see appendix A.2 The inequality I|oN (t) - @N < C Lipschitz property in depth. Indeed, for s E [0, and R2, let (s 0Nv, (t). Then
N corresponds to a discrete
5

---

our result gives 1/ ("N - UN ( t)1l | C which implies that | (s1, (s2, t)ll S N
Clss $2 C. We now turn to the infinite depth limit N cc. Th. 1 shows that there exists a limit function & such that dhy converges uniformly to v in depth s and optmiizaion time t. Furthermore, this limit is Lipschitz continuous in (s, t). In addition, we show that the ResNet IIN converges the limit Neural ODE defined by v that is preserved along the optimization flow, exhibiting an implicit regularization property of deep linear ResNets towards Neural ODEs.
10 (0) 0, (0)1 S N for some Co > O and that there exists a function Vinit : [0, 1] Raxd such that ( 0) (s)mi in || uniformly in 8 as + oo, at speed -. Then the sequence (Vw NeN uniformly converges (in 11-loo w.r. (s, t)) to a limit vo Lipschitz continuous in ts,t) and 11 - loe = =22). Furthermore, II uniformly converges as N + oG to the mapping Zo - 21 where 21 the solution at time of the Neural ODE dz de (8) (8, ())(s) with initial condition No.
Theorem 1 (Existence of limit map). Suppose assumption
solut
We illustrate Th. in Figure 2 The assumption on the existence of Winit ensures a convergence at speed N Neural ODE optimization time O. Note that for instance, the constant initialization ON (0) 0u
[vx(t,)) - -o(tt..)V(tt..lll
10
10-1
100
Raxd satisfies this hypothesis. In order prove Th.
Depth N
1000
I for which a full proof is presented in appendix A.5
10-3
2000
we first present a useful lemma: the weights of the network have at least one accumulation point.
5000
9000
10-3
Lemma 1 (Existence of limit functions). For [0, 1] and t E R+, let (s,(s, t) = 8% Nx Under the assumptions of Prop. 2 there exists subse- quence Vo(N) and v/ [0, I IR+ + Rdxd Lips- chitz continuous with respect to both parameters and such that Wo(N) + Vo uniformly (in ||-1loo w.r.t (s, t)).
5
Training iteration
Le norm [ - (t. (w.r. depth s) for different training iterations (hori- zontal axis) and different depth . (vertical axis). As predicted by Th. [ this distance goes to as N + +00.
Figure 2:
Lemma proved in appendix AA.3 and gives us the existence a Lipschitz continuous accumulation point, but not the uniqueness nor the convergence speed. For the uniqueness, we show in appendix thh under the assumptions of Th. [ one has that any accumulation point of tha satisfies the limit Neural ODE
0. = ((ba(.)() ()), (ha 0) = winit (-, 0),
and show that F satisfies the hypothesis of the Picard-Lindelóf theorem, hence showing the unique- ness of tb. We finally show that, as intuitively expected, trajectories of the weights of our linear ResNets of depth N and 2N remain close one to each other. This gives the convergence speed in [ See appendix [.pendix/AA.4fooo for a proof.
Lemma 2 (Closeness of trajectories). Suppose asumption
1 |0O ( - 0.N (0) S Ce for some N
C. > 0 and that || (0) - 02N (0)ll = x Then Vt E R4, 110 (t) 02, (t) = ( )
Adjoint Method in Residual Networks
In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to ResNets: their memory free backpropagation thanks to the adjoint method. We consider a ResNet (1 and try to invert it using reverse mode Euler discretization of the Neural ODE (2) when &e is any smooth interpolation of the ResNet. This corresponds to defining nN = xN and iterate for {N - 00:
1
in En+1
N J f(n AN
(6)
We then the approximated activations (ā., (GN)) proxy for the true activations (pN))E(N) to compute gradients without storing the activations:
0N L 10ff
V [1+
- N 0. V (7)
The approximate recovery the activations in Eq. (6) is implementable for any ResNet: there no need for particular architecture or forward rule modification. The drawback is that the recovery is

---

only approximate. We devote the remainder of the section to the study of the corresponding errors and error reduction using second order Heun's 's method. We first show that, if f(., ON) and its derivative are bounded by a constant independent of N, then the error for reconstructing the activations in the backward scheme (6) is O ) Proofs of the theoretical results are appendix A
Error for reconstructing activations. We consider the following assumption:
Assumption 2. There exists constants Cy and Lgg such that VN N, Vn E [N 1], II5 0% Il S
Cy and [ ]l 1 T.g.
Then the error made by reconstructing the activations is in O(2).
Proposition (Reconstruction error). With assumption one has I|x,, - S. I (e 1)C, +0( N²).
Prop. B
shows a slow convergence of the error for recovering activations. This bound does not depend on the discrete derivative (
f(. ( contrarily the errors between the ResNet activations and the trajectory of the interpolating Neural ODE in Prop [ In summary, even though regularity in depth necessary to imply closeness Neural ODE, it not necessary recover activations, and neither gradients, as we now show.
Error in gradients when using the adjoint method. We use the result obtained in Prop. derive a bound O (Nzz) on the error made for computing gradients using formulas (7).
3 to
Proposition 4 (Gradient error). Suppose assumption 2 Suppose addition that S 0) admits Lipschitz constant Lag. Jo 0) admits a Lipschitz constant A, and an upper bound St all of which
are independent of 0. Then one has Vox L VoN Lll = 0 (
For proof, see appendix A.7 where we give the dependency our upper bound as a function of A, Ly, Cy, Ω and Las.
grootthnes-deppender reconstruction with Heun's method. The bounds in Prop. 3 and do not depend on the smoothness with respect to the weights of the f(, NN Only the magnitude of the residuals plays role in the correct recovery of the activations and estimation of the gradient. Hence, there is no apparent benefit of having such a network behave like a Neural ODE. We now turn to Heun's method, a second order integration scheme, and show that in this case smoothness in depth of the network improves activation recovery. A HeunNet [30] of depth with parameters 0N ........ N 1:
iterates for n 0, ......
On = Fn N f ) and In+1 = In 2N
(f(zn, 0N f(yn, 1)+1)))
(8)
These forward iterations can once again be approximately reversed by doing for n N -y,....,
1
Jn 2n+1 N fi ON and H. in+1 2N (f(ūn+1, 2xt2) + f(īn ONN)),
(9)
which also enables approximated backpropagation without storing activations. When discretizing an ODE, Heun's ' method has better O NZ error, hence we expect better recovery than in Prop. 3 Indeed, we have:
Proposition 5 (Reconstruction error Heun's method), Assume assumption 2 Denote by L's the Lipschitz constant of + [ (x, 0 + f(x 1 fxx ON+,), by La the Lispchitz constant of 0 + f(:00) and by T/ that of 0 + OzS(.,0) Let Cy T. Finally, define AN =
1 UN
(64-119, x AN + +0(
max, |10N11 0% 112. Using Heun's method, we have: || - F.lll
This bound is very similar to that in proposition 3 with an additional factor AN Hence, we see that under the condition that A +) the reconstruction error ||<,, - žn is in O (1)) In the linear case, we have proven under some hypothesis in Prop. Prop. that such a condition AN holds during training. Consequently, the smoothness of the weights of HeunNet turns helps it recover the activations, while it is not true for ResNet. This provides better guarantees the error on gradients:
7

---

Proposition 6 (Gradient enror Heun's 's method). Suppose assumption 2 Suppose in addition that 0,f( 8) admits Lipschitz constans, f 8) admits a Lipschitz constant and an upper bound, all of which are independent of O. Then one has V ON L VoN x L|| 0 A
N
Just like with activation, we see that Heun's method allows for a better gradient estimation when the weights smooth with depth. Equivalently, for fixed depth, this proposition indicates that HeunNets have a better estimation of the gradient with the adjoint method than ResNets which ultimately leads to better training and overall better performances by such memory-free model.
5 Experiments
We now present experiments to investigate the applicability of the results presented in this paper. We use Pytorch [33] and Nvidia Tesla V100 GPUs. Our code is available on GitHub] All the experimental details are given in appendix B and we provide recap on ResNet architectures in appendix
Test socrray
80%
Tes oro
60%
mm
1
Gradient
70%
40%
60%
Usual scaling
Adjoing Method
10-7
Scaling I/N
30%
Usual Backprop
NeunNet
50%
25%
2
8
10 12
20
10
60
4
8
16
32
64
Number of blocks per layer
Training epochs
Depth
(b)
Figure 3: (a) Test accuracy CIFAR-10 function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied (3 runs). (b) Failure the adjoint method with ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using ResNet HeunNet. Each point corresponds to parameter.
5.1 Validation of our model with step size 1 NV
The ResNet model the NV term. This makes the model depth aware, and we want to study the impact of this modification on the accuracy on CIFAR and ImageNet. We first train a ResNet-101 [20] on CIFAR-IO and Im- ageNet using the same hyper-parameters Experimental details are in appendix [ and results are summarized in table showing that the explicit addition of the step N does not affect accuracy. In strike contrast, the
is different from the classical ResNet because of
Table 1: Test accuracy (ResNet-101) ResNet-101
Ours
CIFAR-10
95.5 0.1%
95.5 + 0.1%
size I.
ImageNet
77.8%
77.9%
classical ResNet rule without the scaling 1 makes the network behave badly at large depth, while 1
it still works well with our scaling N as shown in Figure (a). On ImageNet, the scaling N also leads to similar test accuracy in the weight tied setting: 72.5% with 4 blocks per layer, 73.2% with blocks per layer and 72% with 16 blocks per layer (mean over 2 runs).
5.2 Adjoint method
New training strategy. Our results in Prop. 3 and 4 assume uniform bounds in N on our residual functions and their derivatives. We also formally proved in the linear setting that these assumptions hold during the whole learning process if the initial loss is small. A natural idea to start from a small loss is to consider a pretrained model.
In addition, we also want our pretrained model verify assumption 2 so we consider the following setup. On CIFAR (resp. ImageNet) we train a ResNet with 4 (resp. 8) blocks in each layer, where weights are tied within each layer. A first observation is that one can transfer these weights to deeper ResNets without significantly affecting the test accuracy of
Table 2: Test accuracy (ResNet)
Before F.T.
After FT
CIFAR-10
95.25 0.2 %
95.65 0.1 %
ImageNet
73.1 %
75.1 %

---

the model: it remains above 94.5% on CIFAR-10 10 and 72% on ImageNet. We then untie the weights of our models and refine them. More precisely, for CIFAR, we then transfer the weights of our model to ResNet with 4, 4, 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method. We display in table 2 the median of the new test accuracy, over runs for the initial pretraining of the model. For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tune the whole model with our adjoint method for the residual layers. Results are summarized in t To the best of our knowledge, this is the first time Neural-ODE like ResNet achieves a test-accuracy of 75.1% on ImageNet.
Failure in usual settings. In Prop. B we showed under assumption 2 that is if the residuals are bounded and Lipschitz, continuous with constant independent of the depth N, then the error for computing the activations backward would scale in 1
N as well as the error for the gradients (Prop. 4 First, this results shows that the architecture needs to be deep enough, because it scales in y= for instance, we fail to train a ResNet- 101 [20] on the ImageNet dataset using the adjoint method on its third layer (depth 23), as shown in Figure (b).
Success at large depth. To further investigate the applicability of the adjoint method for training deeper ResNets, we train a simple ResNet model on the CIFAR data set. First, the input is processed by a 5 x 5 convolution with 16 out channels, and the image is down-sampled to a size 10 x 10. We then apply a batch norm, ReLU and iterate relation (I) where f is a pre-actiii aretvctivation basic block [21]. We con- sider the zero residual initialisation: the last batch norm of each basic block is initialized to zero. We consider differ- ent values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy. We then compare the performance of our model using a ResNet (forward rule (I) or a HeunNet (forward rule 8 y. We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas (6) and (p). We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4 The true backpropagation gives the same curves for the ResNet and the HeunNet. Ap- proximated gradients, however, lead to a large test error at small depth, but give the same performance at large depth, hence confirming our results in Prop. 4 and 6 In addition, at fixed depth, the accuracy when training a HeunNet with the adjoint method is better (or similar at depths 2, 32 and 64) than for the ResNet with the adjoint method. This is to be linked with the two different bounds in Prop. f [6 for the HeunNet, smoothness with depth, which is expected at large depth, according to the theoretical results for the linear case (Prop. 2). implics a faster convergence to the true gradients for the HeunNet than for the ResNet. We finally validate this convergence in Figure (c): the deeper the architecture, the better the approximation on the gradients. In addition, the HeunNet Net approximates the true gradient better than the ResNet.
60%
ResNet +
Adjoint Methad
50%
ResNet Backprop
HeunNet + Adjoint Method
40%
NeunNet Backprop
Test errar
30%
20%
15%
2
8
16
32
64
Depth
Figure 4: Comparison of the best test errors as function of depth when using Euler or Heun's 's discretization method with or without the adjoint method.
Conclusion, limitations and future works
We propose a methodology to analyze how well a ResNet discrctizes a Neural ODE, The positive results predicted by theory in the linear case also observed in practice with real architectures: one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets) using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in the residual layers. However, we also show that for large scale problems such as ImageNet classification from seratch, the adjoint method fails at usual depths.
Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear setting under a small loss initialization. A natural extension would be to study the non-linear case. In addition, the adjoint method time consuming, and an improvement would be to propose a cheaper method than a reverse mode traversal of the architecture for approximating the activations.

---

Acknowledgments
This work was granted access to the HPCC resources of IDRIS under the allocation 2020- (ND011012073) made by GENCI. This work was supported in part by the French government under management of Agence Nationale de Recherche as part of the "Investissements d' d'avenir" program, reference ANRI9 ANR19-P3IA-000 (PRAIRIE 31A Institute). This work was supported in part by the European Research Council (ERC project NORIA), M. S. thanks Mathieu Blondel and Zaccharie Ramzi for helpful discussions.
References
[1] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit accelcration by overparameterization. International Conference on Machine Learning, pages 244-253, PMLR, 2018.
[2] Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
[3] Raphaël Barboni, Gabriel Peyré, and François François-Xavier Xavier Vialard. Global convergence of resnets: From finite to infinite width using linear parameterization, arXiv preprint arXiv:2112.05531,
2021.
[4] Peter Bartlett, Dave Helmbold, and Philip Long. Gradient descent with identity initializa- tion efficiently learns positive definite linear transformations by deep residual networks. In International conference on machine learning, pages 521-530. PMLR, 2018.
[5] Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation machine learning: survey. Journal of machine learning research, 18, 2018.
[6] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems, 34, 2021.
[7] Haim Brezis and Haim Brézis. Functional analysis, Sobolev spaces and partial differential equations, volume 2. Springer, 2011.
[8] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, pages 2018.
[9] Alain-Sam Cohen, Rama Cont, Alain Rossier, and Renyuan Xu. Scaling properties of deep residual networks. In International Conference on Machine Learning, pages 2039-2048. PMLR, 2021.
[10] Rama Cont, Alain Rossier, and Ren Yuan Xu. Convergence and implicit regularizaion properties of gradient descent for deep residual networks. arXiv preprint arXiv:2204.07261, 2022.
[11] Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv preprint arXiv:2003. 0463000000000,,... 2019.
[12] Christa Cuchiero, Martin Larsson, and Josef Teichmann. Deep neural networks, generic universal interpolation, and controlled odes. SIAM Journal on Mathematics of Data Science, 2(3)9901-919, 2020.
[13] Jean-Pierre Demailly. Analyse numérique et équations différenntilles--4eme Ed. EDP sciences, 2016.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint 2020.
10

---

[15] Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In International Conference on Machine Learning, pages 1675-1685. PMIR, 2019.
[16] Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Guyon, U. V. Luxburg. S. Ben- gio, H. Wallach. R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
[17] Will Grathwohl, Ricky TQ Chen, Jesse Bettencourt, Πla Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. arXiv preprint arXiv:1810.01 2018.
[18] Samuel Greydanus. Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In Advances in Neural Information Processing Systems, pages 15353-15363, 2019.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks, 2016.
[22] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456 PMLR. 2015.
[23] Jörn-Henrik Jacobsen, Amoll W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invertible networks. International Conference on Learning Representations, 2018.
[24] Patrick Kidger. On neural differential equations. arXiv preprint arXiv:2202.02433, 2022.
[25] Qianxiao Li, Ting Lin, and Zuowei Shen. Deep learning via dynamical systems: An approxima- tion perspective. arXiv preprint 1rXiv:1912.10388 2019.
[26] Chaoyue Liu, Libin Zhu, and Mikhail Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant. Advances in Neural Information Processing Systems, 33, 2020.
[27] Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv: we... 02762, 2019.
(28) Yiping Lu, Chao Ma, Yulong Lu, Jianfeng Lu, and Lexing Ying. A mean field analysis of deep resnet and beyond: Towards provably optimization via overparameterrization from depth. In International Conference on Machine Learning, pages 6426-6436. PMLR, 2020.
[29] Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations. In International Conference on Machine Learning, pages 3276-3285. PMLR, 2018
[30] Mehrdad Maleki, Mansura Habiba, and Barak A Pearlmutter Heunnet: Extending resnet using heun's 's method. In 2021 32nd Irish Signals and Systems Conference (ISSC), pages 1-6. IEEE, 2021.
[31] Pierre Marion, Adeline Fermanian, Gérard Biau, and Jean Sann-Phlipppe Vert. Scaling resnets in the large-depth regime. arXiv preprint arXiv:2206.066 02220.9699,, 2022.
[32] Katharina Ott, Prateek Katiyar, Philipp Hennig, and Michael Tiemann. Resnet after all: Neural (ode)s )s and their numerical solution. In International Conference on Learning Representations, 2021.
11

---

[33] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Viiioo, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[34] Chao Peng. Xiangyu Zhang. Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters- improve semantic segmentation by global convolutional network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4353-4361, 2017.
[35] Lev Semenovich Pontryagin. Muthematical theory of optimal processes. CRC press, 1987.
[36] Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pages 1-13, 2019.
[37] Michael E. Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Momentum residual neural networks. In Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages Jul 2021.
PMLR, 18-24
[38] Michael E Sander, Pierre Ablin, Mathieu Blondel, and Gabriel Peyré. Sinkformers: Transform- ers with doubly stochastic attention. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 3515-3530. PMLR, 18-24 Jul 2022.
[39] Qi Sun, Yunzhe Tao, and Qiang Du Stochastic training of residual networks: a differential equation viewpoint. arXiv preprint ar2iv:1812.00174. 2018.
[40] Y Teh, Amaud Doucet, and E Dupont. Augmented neural odes. Advances in Neural Information Processing Systems 32 (NIPS 20/9), 32(2019), 2019.
[41] Takeshi Teshima, Koichi Tojo, Masahiro Ikeda, Isao Ishikawa, and Kenta Oono. Universal ap- proximation property of neural ordinary differential equations. arXiv preprint wrXiv:2012.024144 2020.
[42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[43] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska. Supemeurons: Dynamic gpu memory management for training deep neural networks. In Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 41 41-53, 2018
[44] E Weinan. A proposal machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
[45] E Weinan, Jiequn Han, and Qianxiao Li. A mean-field optimal control formulation of deep learning. Research in the Mathematical Sciences, 6(1):10, 2019.
[46] Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint 0tt:::2100.007777 2021.
[47] Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. Advances in neural information processing systems, 30, 2017.
[48] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using sycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.
[49] Juntang Zhuang, Nicha Dvomek, Xiaoxiao Li, Sekhar Tatikonda, Xenophon Papademetris, and James Duncan. Adaptive checkpoint adjoint method for gradient estimation in neural ode. In International Conference Machine Learning, pages 11639-11649, PMLR, 2020.
[50] Juntang Zhuang, Nicha C Dvoreek, Sekhar Tatikonda, and James S Duncan. Mali: A memory efficient and reverse accurate integrator for neural odes. arXiv preprint arXiv:2102.0440 2021.
[51] Difan Zou, Philip M Long, and Quanquan Gu. On the global convergence of training deep linear resnets. arXiv preprint arX1v:2003.01094, 2020.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]. all the assumptions are stated.
(b) Did you include complete proofs of all theoretical results? [Yes]. the proofs are in the supplementary material.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]. appendix
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]. in appendix B
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]. see Figure and table
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] for Pytorch, CIFAR, ImageNet
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "rEOEr6-Yw-Y", "reviewer": "Reviewer_rbYF", "summary": "This paper studies the connection between ResNet and Neural ODE. Specifically, (1). this paper investigates the approximation error between the hidden state trajectory of the Resnet and the solution of the Neural ODE interpolating the hidden state of the Resnet. The approximation error can have a non-zero limit in general, and the authors specify a case (small enough initialization, linear blocks) where the error limit is zero. (2). The paper then views the hidden state trajectory of ResNet as a forward Euler method to solve the corresponding NODE and proposes to use the model obtained by the backward Euler method in the backpropagation, which has a lower memory cost. Such a practice is also transferred to the HeunNet with a theoretically lower approximation error bound. The authors further conduct experiments to verify the theoretical findings.", "questions": "1. Why do we need to save the memory cost for the hidden state, which is relatively small? Specifically, the dimension of one hidden state (size m) is way smaller than the dimension of the corresponding layer’s parameter (size m^2).\n2. What’s the point to approximate the NODE using Resnet (or study this regularization effect)? I expect the authors can comment on the advantage of NODE over ResNet and why we need to study this regularization effect.\n", "limitations": "Please refer to the \"weakness\" and “questions”.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThis paper is well-written\n2.\tThe theoretical analysis is solid, and the authors honestly report the experiment results.\n\nWeakness:\n1.\tThe settings of the theoretical analysis are far from the practice. This paper uses the forward propagation rule x_{t+1}=x_{t} +1/N f(x_t,\\theta_t), where N is the depth of neural networks. However, in the latter analysis (e.g. Proposition 1,2), it is assumed that \\theta_t has bounded norm (or f has bounded Lipschitz parameter with respect to theta_t). Therefore, if we absorb the 1/N factor into f (or more directly into \\theta_t, as in neural networks, f is usually a piece-wise linear function of \\theta_t), which is what we do in practice, the initialization scale of \\theta can be rather small and decreases with the depth, while in practice the initialization scale usually depends on the width. Theorem 1 requires the parameters between adjacent layers to be close, which is also unrealistic.\n2.\tThe experiment results are not promising even compared to the current state of the arts. For example, in table 2, the test accuracy on Imagenet for Resnet using the adjoint method is not high even compared to the vanilla Resnet [1]. Also, the performance of the adjoint method in Figure 4 is worse than the traditional backpropagation.\n\n[1]. https://paperswithcode.com/sota/image-classification-on-imagenet?tag_filter=3\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B7s9hTF8h_S", "reviewer": "Reviewer_9X5i", "summary": "This paper studies the link between Neural ODEs and ResNets from a theoretical point of view.\n\nThe authors show that without extra regularity assumptions on the residual block, a ResNet might not converge to a Neural ODE in the limit when the depth goes to infinity. On the positive side, they then show how this issue can be resolved by assuming additional assumptions on the weight initialiatiation and study the case of linear residual blocks.\n\nAn additional contribution is a numerical technique allowing to perform backpropagation through ResNets similarly to the adjoint method for Neural ODEs; this has the advantage of not requiring to store activations, and therefore comes with constant memory footprint. The disadvantage is given by inaccurate gradients. Error bounds for recovering activations and gradients are provided, and even improved by using a modification of the proposed adjoint method analogue to the Heun's method for differential equations.\n\n\n\n\n", "questions": "How much do the results in section 3.2 depend on the choice of loss function?\n\nWhat are the main theoretical bottlenecks preventing from extending the proofs on section 3.2 from the linear case to the non-linear case? Are there some non-linear activation functions for which this extension could be easier than others?", "limitations": "As mentioned above, the main limitation is that the study is limited to the linear case for the residual block, but the authors clearly mention this in their conclusion.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper sheds some light on the theoretical aspects relating Neural ODEs and ResNets, which has recently become a popular topic in the deep learning community. Several contributions have been provided in this direction prior to this paper, in particular highlighting the failure, in some cases, of ResNets to approximate ODEs in the infinite depth limit. Therefore, the level of originality of the paper is not very high, but the contributions build upon existing ones and the literature seems appropriately cited. The proposed technique based on the adjoint method to train ResNets without storing activations is interesting and yields improvements in performance on CIFAR-10 and ImageNet. \n\nThe main weakness is that the authors only study the linear residual block case.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B0OShvy8jDN", "reviewer": "Reviewer_hARH", "summary": "- The paper examines the relationship between and NeuralODEs and ResNets with $N$ layers whose residual functions have a multiplicative coefficient of $1/N$, i.e. with layers of the type $x_{n+1} = x_n + \\frac{1}{N}f_{\\theta_n}(x_n)$. \n- The paper shows that if the initialised weights of a _linear_ ResNet vary smoothly from one layer to the next and other assumptions at training time $t=0$ are satisfied, as $N\\rightarrow \\infty$, the map learned by the ResNet at any training time $t$ corresponds to the map induced by a (training time-dependent) ODE sending $x(0) \\mapsto x(1)$, where $x(1)$ is the solution of this ODE at time $1$. \n- The paper shows that the smoothness assumption mentioned above is necessary and this convergence is not otherwise guaranteed. \n- The paper investigates how these sort of ResNets can be trained with $O(1)$ memory via a discrete adjoint method producing approximate gradients. It further investigates what the error of this approximation is as a function of $N$ and shows how this error can be reduced by using Heun's method. \n- Finally, the experimental section shows ResNets with rate $1/N$ work similarly well in practice as regular ResNets and study how the proposed discrete adjoint can be used to fine-tune deep ResNets with (initially) tied weights. ", "questions": "- Most importantly, everything in the first bullet point of the weaknesses section. \n- In line 328, what do you mean by \"the classical ResNet forward rule is not compatible with the usual training procedure of CIFAR\"? ", "limitations": "Yes. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strenghts\n\n- Despite the well-known connections between ResNets and NeuralODEs, this paper shows that there are still many interesting aspects to uncover about their relationship. Usually, when this equivalence is discussed, it is typically assumed that $\\theta_n = \\theta$.  This paper investigates this equivalence in the linear regime when the weights $\\theta_n$ vary at each layer $n$, which is much closer to how ResNets are actually constructed and used in practice. Therefore, this improves our understanding of a practically-relevant situation and it is likely that this result will also be relevant to practitioners and not just the theoretically inclined. \n- The proposed discrete adjoint method comes with a useful theoretical analysis providing theoretical guarantees regarding the error potential users should expect. \n- Although not spectacular or surprising, the successful experiments on reproducing the performance of regular ResNets using ResNets with rate $1/N$ are an important result. It shows that this model that is convenient theoretically is also relevant in practice and essentially it does not affect the usual performance we expect from ResNets. This also makes the theory more relevant for practitioners. \n- I like that the paper also explores failure modes in the experimental section for when the assumptions of the results do not hold. \n- In the era of \"large models\", the presented fine-tuning experiments are very relevant and could be a useful tool in the arsenal of anyone training deep ResNets. \n\n# Weaknesses\n\n- I believe the main problem of the paper is the (too) many notational errors or omissions for a theoretical paper. While each in itself is minor, together they accumulate and they do tend to show up in important parts of the paper. So I am adding a list below: \n    1. In the bound of Proposition 1, if $L=0$, then $\\frac{e^L - 1}{L} = \\frac{0}{0}$, which is undefined. So I do not understand the calculations done below where $L=0$. Am I missing something here? \n    2. What does the notation $[\\cdot]$ mean in the definition of $C_N$ (line 142). The $[\\cdot]$ notation is also used in other places. And what does the superscript $K \\times [0, 1]$ signify?\n    3. Please explain the notation $C^k$ for classes of smooth functions. Many people with ML background might not be familiar with it. \n    4. Before Proposition 1, it says ``We now consider a smooth interpolation $\\phi_\\Theta$ for ResNet''. Which interpolation do you actually use? Two were introduced in the paragraph above. Does it matter? \n    5. Below Proposition 1 you mention that the bound is tight. But you show this only for the case when the ODE function depends only on time, which makes L superfluous (ignoring the undefined issue). But is the bound actually tight in general? This question is probably not super relevant for the discussion in the implications paragraph below, but I think it's still an important thing to clarify. \n    6. In line 188, $\\nabla_n^N(t) = \\nabla_{\\theta_n^N}L$, there is no $t$ on the right hand side. Should that be $\\nabla_{\\theta_n^N}L(t)$?\n    7. What is the difference between $L(t)$ and $l^N(t)$. Both are used and as far as I understand they denote the same thing. \n    8. In Assumption 2, what is $D_x$? As far as I can see, $D_x$ is never defined. Should that be $\\partial_x$ instead?\n    9. In Assumption 2, $C_{df}$ is defined and never used. Instead, an $L_f$ shows up, that was never introduced. Should $L_f$ be $C_{df}$?\n    10. Proposition 6, No need to mention $L_{df}$ if it doesn't show up in the final bound. \n    11. Please clarify what you mean by \"tied weights\". I checked many ResNet papers and it doesn't seem to be standard terminology. I take it to mean $\\theta_n = \\theta$. But this can be a confusing term because the smoothness assumption you often employ also \"ties\" the weights of different layers together. Constant weights is a better term IMO. \n- Figure 2 is not particularly useful and could show something much more interesting and visualise it much better. First of all, a sequential colour scale is much more appropriate for what it is trying to show because this shows a progression of training steps. A categorical colouring scale does not convey that information. Second, plotting just a few weights in the matrices loses a lot of information. The plot could show instead the distance from the limit $\\psi$ as a function of the training time or another aggregated metric of interest. Perhaps the distance between consecutive layers and how it evolves over time could be another idea. \n- In the first plot of Figure 3, since this is the median over only 3 runs, it's better to just include all 3 runs (6 runs in total) since it's not a lot. The median drops a lot of information and doesn't say much about the variation. Alternatively, better to show the mean and a confidence interval. \n- In line 377, you mention that \"the HeunNet ... performs better than the ResNet\". This claim is not really backed up by data. It's impossible to deduce that from Figure 4 and we would need to see some numerical data. There are two things that are needed to support that claim: scientific significance (if the difference is just 0.00001%, and it looks very small on the plot, then it's not really better) and statistical significance (what is the standard error?). The claim should either be dropped or more data should be supplied.  \n\n======= EDIT ===========\n\nRaised my score to accept after discussion. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept