RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Neural ODEs.
Neural ODEs are a class of implicit deep learning models defined by an ODE where a neural network parameterises the vector field (Weinan, 2017 ; Chen et al., 2018 ; Teh et al., 2019 ; Sun et al., 2018 ; Weinan et al., 2019 ; Lu et al., 2018 ; Ruthotto and Haber, 2019 ; Kidger, 2022 ) . Given an input x_{0} , the output of the model is the solution of the ODE ( 2 ) at time 1 . From a theoretical viewpoint, the expression capabilities of Neural ODEs have been investigated in (Cuchiero et al., 2020 ; Teshima et al., 2020 ; Li et al., 2019 ) and the Neural ODE framework has been used to better understand the dynamics of more general architectures that include residual connections such as Transformers (Sander et al., 2022 ; Lu et al., 2019 ) .
Experimentaly, Neural ODEs have been successful in a various range of applications, among which physical modelling (Greydanus et al., 2019 ; Cranmer et al., 2019 ) and generative modeling (Chen et al., 2018 ; Grathwohl et al., 2018 ) . However, there are many areas where Neural ODEs have failed to replace ResNets, for instance for building computer vision classification models. Neural ODEs fail to compete with ResNets on ImageNet, and to the best of our knowledge, previous works using Neural ODEs on ImageNet consider weight-tied architectures and only achieves the same accuracy as a ResNet18 (Zhuang et al., 2021 ) .
Implicit Regularization of ResNets towards ODEs.
Recent works have studied the link between ResNets and Neural ODEs. In (Cohen et al., 2021 ) , the authors carry experiments to better understand the
scaling behavior of weights in ResNets as a function of the depth. They show that under the assumption that there exists a scaling limit \theta(s)=N^{\beta}\lim\theta^{N}_{\lfloor Ns\rfloor} for the weights of the ResNets (with 0<\beta<1 ) and if the scale of the ResNet is \frac{1}{N^{\alpha}} with 0<\alpha<1 and \alpha+\beta=1 , then the hidden state of the ResNet converges to a solution of a linear ODE. In this paper, we are interested in the case where \alpha=1 , which seems more natural since it is the scaling that appears in Euler’s method with step \frac{1}{N} .
In addition, we do not assume the existence of a scaling limit \theta(s)=\lim\theta^{N}_{\lfloor Ns\rfloor} . In subsection 3.2 , we demonstrate the existence of this scaling limit in the linear setting, under some assumptions. The recent work (Cont et al., 2022 ) shows results regarding linear convergence of gradient descent in ResNets and prove the existence of an \frac{1}{2} -Hölder continuous scaling limit as N\to\infty with a scaling factor for the residuals in \frac{1}{\sqrt{N}} which is different from ours. In contrast, we show that our limit function is Lipschitz continuous, which is a stronger regularity. We also show that our convergence is uniform in depth and optimization time. More generally, recent works have proved the convergence of gradient descent training of ResNet when the initial loss is small enough. This include ResNet with finite width but arbitrary large depth (Du et al., 2019 ; Liu et al., 2020 ) and ResNet with both infinite width and depth (Lu et al., 2020 ; Barboni et al., 2021 ) . These convergence proofs leverage an implicit bias toward weights with small amplitudes. They however leave open the question of convergence of individual weights as depth increases, which we tackle in this work in the linear case. This requires showing an extra bias toward weights with small variations across depth.
Memory bottleneck in ResNets.
Training deep learning models involve graphics processing units (GPUs) where memory is a practical bottleneck (Wang et al., 2018 ; Peng et al., 2017 ; Zhu et al., 2017 ) . Indeed, backpropagation requires to store activations at each layer during the forward pass. Since samples are processed using mini batches, this storage can be important. For instance, with batches of size 128, the memory needed to compute gradients for a ResNet 152 on ImageNet is about 22 GiB. Note that the memory needed to store the parameters of the model is only 220 MiB, which is negligible compared to the memory needed to store the activations.
Thus, designing deep invertible architectures where one can recover the activations on the fly during the backpropagation iterations has been an active field in recent years (Gomez et al., 2017 ; Sander et al., 2021a ; Jacobsen et al., 2018 ) . In this work, we propose to approximate activations using a reverse-time Euler scheme, as we detail in the next subsection.
Adjoint Method.
Consider a loss function L(x_{N}) for the ResNet ( 1 ). The backpropagation equations (Baydin et al., 2018 ) are

abla_{\theta^{N}_{n-1}}L=\frac{1}{N}[\partial_{\theta}f(x_{n-1},\theta^{N}_{n-1})]^{\top} abla_{x_{n}}L,\quad abla_{x_{n-1}}L=[I+\frac{1}{N}\partial_{x}f(x_{n-1},\theta^{N}_{n-1})]^{\top} abla_{x_{n}}L

Now, consider a loss function L(x(1)) for the Neural ODE ( 2 ). The adjoint state method (Pontryagin, 1987 ; Chen et al., 2018 ) gives

abla_{\Theta}L=-\int_{T}^{0}{\partial_{\Theta}[\varphi_{\Theta}(x(s),s)}]^{\top} abla_{x(s)}L{\mathrm{d}}s,\quad-\dot{ abla}_{x(s)}L(s)=[\partial_{x}{\varphi_{\Theta}(x(s),s)}]^{\top} abla_{x(s)}L

Note that if \Theta=(\theta_{n}^{N})_{n\in[N-1]} and \varphi_{\Theta}(.,\frac{n}{N})=f(.,\theta_{n}^{N}) , then Eq. ( 3 ) corresponds to a Euler discretization with time step \frac{1}{N} of Eq. ( 4 ). The key advantage of using Eq. ( 4 ) is that one can recover x(s) on the fly by solving the Neural ODE ( 2 ) backward in time starting from x(1) . This strategy avoids storing the forward trajectory (x(s))_{s\in[0,1]} and leads to a O(1) memory footprint (Chen et al., 2018 ) . In this work, we propose to use a discrete adjoint method by using a reverse-time Euler scheme for approximately recovering the activations in a ResNet (Section 4 ). Contrarily to other models such as RevNets (Gomez et al., 2017 ) (architecture change) or Momentum ResNets (Sander et al., 2021b ) (forward rule modification) which rely on an exactly invertible forward rule, the proposed method requires no change at all in the network, but gives approximate gradients.
Notations.
For k\in\mathbb{N} , \mathcal{C}^{k} is the set of functions f:\mathbb{R}^{d}\to\mathbb{R}^{d} k times differentiable with continuous k^{th} differential. If f\in\mathcal{C}^{1} , \partial_{x}f(x)[y] is the differential of f at x evaluated in y . For K\subset\mathbb{R}^{d} compact, \|.\| a norm and f a continuous function on \mathbb{R}^{d} , we denote \|f\|^{K}_{\infty}=\sup_{x\in K}\|f(x)\| .
ResNets as discretization of Neural ODEs
In this section we first show that without further assumptions, the distance between the discrete trajectory and the solution of associated ODEs can be constant with respect to the depth of the network if the residual functions lack smoothness with depth. We then present a positive result by studying the linear case where we show that, under some hypothesis (small loss initialization and initial smoothness with depth), the ResNet converges to a Neural ODE as the number of layers goes to infinity. We show that this convergence is uniform with depth and optimization time.
Distance to an ODE
We first define associated Neural ODEs for a given ResNet.

We say that a neural network \varphi_{\Theta}:\mathbb{R}^{d}\times\mathbb{R}\to\mathbb{R}^{d} smoothly interpolates the ResNet Eq. ( 1 ).
if \varphi_{\Theta} is smooth and \forall n\in\{0,...,N-1\} , \varphi_{\Theta}(.,\frac{n}{N})=f(.,\theta_{n}^{N}) .

Note that we omit the dependency of \Theta in N to simplify notations.
For example, for a given ResNet, there are two natural ways to interpolate it with a Neural ODE, either by interpolating the residuals , or by interpolating the weights . Indeed, one can interpolate the residuals with \varphi_{\Theta}(\cdot,s)=(n+1-Ns)f(.,\theta_{n}^{N})+(Ns-n)f(\cdot,\theta_{n+1}^{N}) when s\in[\frac{n}{N},\frac{n+1}{N}] , or interpolate the weights with \varphi_{\Theta}(\cdot,s)=f(\cdot,(n+1-Ns)\theta_{n}^{N}+(Ns-n)\theta_{n+1}^{N}) for s\in[\frac{n}{N},\frac{n+1}{N}] . If \theta_{n}^{N}=\theta^{N} does not depend on n , then both interpolations are identical and one can simply consider \varphi_{\Theta}(x,s)=f(x,\theta^{N}) , \forall(x,s) .

We now consider any smooth interpolation \varphi_{\Theta} for the ResNet ( 1 ) and a Euler scheme for the Neural ODE ( 2 ) with time step \frac{1}{N} .

We suppose that \varphi_{\Theta} is \mathcal{C}^{1} , and L -Lipschitz with respect to x , uniformly in s . Note that this implies that the solution of Eq. ( 2 ) is well defined, unique, \mathcal{C}^{2} , and that the trajectory is included in some compact K\subset\mathbb{R}^{d} . Denote C_{N}\coloneqq\|\partial_{s}\varphi_{\Theta}+\partial_{x}\varphi_{\Theta}[\varphi_{\Theta}]\|^{K\times[0,1]}_{\infty} . Then one has for all n : \|x_{n}-x(\frac{n}{N})\|\leq\frac{e^{L}-1}{2NL}C_{N} if L>0 and \|x_{n}-x(\frac{n}{N})\|\leq\frac{C_{N}}{2N} if L=0 .

For a full proof, see appendix A.1 . Note that this result extends Theorem 3.2 from (Zhuang et al., 2020 ) to the non-autonomous case: our bound depends on \partial_{s}\varphi_{\Theta}. Finally, our bound is tight. Indeed, for \varphi_{\Theta}(x,s)=as+b for a,b\in\mathbb{R}^{d} , we get \partial_{s}\varphi_{\Theta}+\partial_{x}\varphi_{\Theta}[\varphi_{\Theta}]=a , L=0 and \|x(1)-x_{N}\|=\frac{\|a\|}{2N} .
Implication.
The tightness of our bound shows that closeness to the ODE solution is not guaranteed, because we do not know whether C_{N}/N\rightarrow 0 . Indeed, consider first the residual interpolation \varphi_{\Theta}(x,s)=\left((n+1-Ns)f(x,\theta_{n}^{N})+(Ns-n)f(x,\theta_{n+1}^{N})\right))\mathbb{1}_{s\in[\frac{n}{N},\frac{n+1}{N}]} and the simple case where \partial_{x}\varphi_{\Theta}[\varphi_{\Theta}]=0 . We get \partial_{s}\varphi_{\Theta}(x,s)=N(f(x,\theta_{n+1}^{N})-f(x,\theta_{n}^{N}))\mathbb{1}_{s\in[\frac{n}{N},\frac{n+1}{N}]} , which corresponds to the discrete derivative. It means that although there is a \frac{1}{N} factor in our bound, the time derivative term – without further regularity with depth of the weights, which is at the heart of subection 3.2 – usually scales with N : \frac{C_{N}}{N}=O(1) .
As a first example, consider the simple case where f(x,\theta_{n}^{N})=n. This gives x_{N}=x_{0}+\frac{(N-1)}{2} while the integration of the Neural ODE ( 2 ) leads to x(1)=x_{0}+\frac{N}{2} because \varphi_{\Theta}(x,s)=Ns , so the \|x_{N}-x(1)\|=\frac{1}{2} is not small. Intuitively, this shows that weights cannot scale with depth when using the residual interpolation.
Now, consider the weight interpolation, \theta_{n}^{N}=(-1)^{n} and suppose f is written as f(x,\theta)=\theta^{2} . This gives \varphi_{\Theta}(.,s)=(2Ns-(2n+1))^{2} when s\in[\frac{n}{N},\frac{n+1}{N}] . Integrating, we get x(1)=\frac{1}{3} while the output of the ResNet is x_{N}=1 . Hence \|x_{N}-x(1)\|=\frac{2}{3} is also not small, even though the weights are bounded.
Thus, one needs additional regularity assumptions on the weights of the ResNet to obtain a Neural ODE in the large depth limit. Intuitively if the weights are initialized close from one another and they are updated using gradient descent, they should stay close from one another during training,
since the gradients in two consecutive layers will be similar, as highlighted in Eq. ( 3 ).
Indeed, we see that if x_{n} and x_{n+1} are close, then \nabla_{x_{n}}L and \nabla_{x_{n+1}}L are close, and then if \theta_{n}^{N} and \theta_{n+1}^{N} are close, \nabla_{\theta_{n}^{N}}L and \nabla_{\theta_{n+1}^{N}}L are also close. In subsection 3.2 , we formalize this intuition and present a positive result for ResNets with linear residual functions. More precisely, we show that with proper initialization, the difference between two successive parameters is in \frac{1}{N} during the entire training. Furthermore, we show that the weights of the network converge to a smooth function, hence defining a limit Neural ODE.
Linear Case
As a further step towards a theoretical understanding of the connections between ResNets and Neural ODEs we investigate the linear setting, where the residual functions are written f(x,\theta)=\theta x for any \theta\in\mathbb{R}^{d\times d} . It corresponds to a deep matrix factorization problem (Zou et al., 2020 ; Bartlett et al., 2018 ; Arora et al., 2019 , 2018 ) . As opposed to these previous works, we study the infinite depth limit of these linear ResNets with a focus on the learned weights. We show that, if the weights are initialized close one to another, then at any training time, the weights stay close one to another (Prop. 2 ) and importantly, they converge to a smooth function of the continuous depth s as N\to\infty (Th. 1 ). All the proofs are available in appendix A .
Setting.
Given a training set (x_{k},y_{k})_{k\in[n]} in \mathbb{R}^{d} , we solve the regression problem of mapping x_{k} to y_{k} with a linear ResNet, i.e. f(x,\theta)=\theta x , of depth N and parameters (\theta_{1}^{N},\dots,\theta_{N}^{N}) . The ResNet therefore maps x_{k} to \Pi^{N}x_{k} where \Pi^{N}\coloneqq\prod_{n=1}^{N}(I_{d}+\frac{\theta_{n}^{N}}{N})=(I_{d}+\frac{\theta_{N}^{N}}{N})\cdots(I_{d}+\frac{\theta_{1}^{N}}{N}) . It is trained by minimizing the average errors \|\Pi^{N}x_{k}-y_{k}\|_{2}^{2} , which is equivalent to the deep matrix factorization problem:

\operatorname{argmin}_{(\theta_{n}^{N})_{i\in[N-1]}}L(\theta_{1}^{N},\dots,\theta_{N}^{N})\coloneqq\|\Pi^{N}-B\|_{\Sigma}^{2}, (5)

where \|A\|^{2}_{\Sigma}=\mathrm{Tr}(A\Sigma A^{T}) , \Sigma is the empirical covariance matrix of the data: \Sigma\coloneqq\frac{1}{n}\sum_{k=1}^{n}x_{k}x_{k}^{\top} , and B\coloneqq\frac{1}{n}\sum_{k=1}^{n}y_{k}x_{k}^{\top}\Sigma^{-1} . As is standard, we suppose that \Sigma is non degenerated. We denote by M>0 (resp. m>0 ) its largest (resp. smallest) eigenvalue.
Gradient.
We denote \Pi^{N}_{:n}\coloneqq(I_{d}+\frac{\theta_{N}^{N}}{N})\cdots(I_{d}+\frac{\theta_{n+1}^{N}}{N}) and \Pi^{N}_{n:}\coloneqq(I_{d}+\frac{\theta_{n-1}^{N}}{N})\cdots(I_{d}+\frac{\theta_{1}^{N}}{N}) and write the gradient \nabla^{N}_{n}(t)=\nabla_{\theta_{n}^{N}}L(\theta^{N}_{1}(t),..\theta^{N}_{n}(t),..,\theta^{N}_{N}(t)) . The chain rule gives N\nabla^{N}_{n}=\Pi_{:n}^{N\top}(\Pi^{N}-B)\Sigma\Pi_{n:}^{N\top} .
Intuitively, as N goes to +\infty , the products \Pi^{N} , \Pi_{:n}^{N} and \Pi_{n:}^{N} should converge to some limit, hence we see that N\nabla_{n}^{N} scale as 1 . Therefore, we train \theta_{n}^{N} by the rescaled gradient flow \frac{d\theta_{n}^{N}}{dt}(t)=-N\nabla^{N}_{n}(t) to minimize L and denote {\ell}^{N}(t)=L(\theta_{1}^{N}(t),\dots,\theta_{N}^{N}(t)) .
Adjoint Method in Residual Networks
In this section, we focus on a particularly useful feature of Neural ODEs and its applicability to ResNets: their memory free backpropagation thanks to the adjoint method.
We consider a ResNet ( 1 ) and try to invert it using reverse mode Euler discretization of the Neural ODE ( 2 ) when \varphi_{\Theta} is any smooth interpolation of the ResNet.
This corresponds to defining \tilde{x}_{N}=x_{N} and iterate for n\in\{N-1,\dots 0\} :

\tilde{x}_{n}=\tilde{x}_{n+1}-\frac{1}{N}f(\tilde{x}_{n+1},\theta^{N}_{n})

We then use the approximated activations (\tilde{x}_{n})_{n\in[N]} as a proxy for the true activations (x_{n})_{n\in[N]} to compute gradients without storing the activations:

\tilde{ abla}_{\theta^{N}_{n-1}}L=\frac{1}{N}[\partial_{\theta}f(\tilde{x}_{n-1},\theta^{N}_{n-1})]^{\top} abla_{\tilde{x}_{n}}L,\quad abla_{\tilde{x}_{n-1}}L=[I+\frac{1}{N}\partial_{x}f(\tilde{x}_{n-1},\theta^{N}_{n-1})]^{\top} abla_{\tilde{x}_{n}}L

The approximate recovery of the activations in Eq. ( 6 ) is implementable for any ResNet: there is no need for particular architecture or forward rule modification. The drawback is that the recovery is only approximate. We devote the remainder of the section to the study of the corresponding errors and to error reduction using second order Heun’s method.
We first show that, if f(.,\theta^{N}_{n}) and its derivative are bounded by a constant independent of N , then the error for reconstructing the activations in the backward scheme ( 6 ) is O(\frac{1}{N}) . Proofs of the theoretical results are in appendix A .
Error for reconstructing activations.
We consider the following assumption:

There exists constants C_{f} and L_{f} such that \forall N\in\mathbb{N} , \forall n\in[N-1] , \|f(.,\theta_{n}^{N})\|_{\infty}\leq C_{f} and \|{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\partial_{x}}[f(.,\theta^{N}_{n})]\|_{\infty}\leq L_{f} .

Then the error made by reconstructing the activations is in O(\frac{1}{N}) .

With assumption 2 , one has \|x_{n}-\tilde{x}_{n}\|\leq\frac{(e^{L_{f}}-1)C_{f}}{N}+O(\frac{1}{N^{2}}).

Prop. 3 shows a slow convergence of the error for recovering activations.
This bound does not depend on the discrete derivative f(.,\theta_{n+1}^{N})-f(.,\theta_{n}^{N}) , contrarily to the errors between the ResNet activations and the trajectory of the interpolating Neural ODE in Prop 1 . In summary, even though regularity in depth is necessary to imply closeness to a Neural ODE, it is not necessary to recover activations, and neither gradients, as we now show.
Error in gradients when using the adjoint method.
We use the result obtained in Prop. 3 to derive a bound in O(\frac{1}{N^{2}}) on the error made for computing gradients using formulas ( 7 ).

Suppose assumption 2 . Suppose in addition that \partial_{x}f(.,\theta) admits a Lipschitz constant L_{df} , \partial_{\theta}f(.,\theta) admits a Lipschitz constant \Delta , and an upper bound \Omega , all of which are independent of \theta . Then one has \|\tilde{\nabla}_{\theta^{N}_{n}}L-\nabla_{\theta^{N}_{n}}L\|=O(\frac{1}{N^{2}}).

For a proof, see appendix A.7 , where we give the dependency of our upper bound as a function of \Delta,L_{f},C_{f},\Omega and L_{df} .
Experiments
We now present experiments to investigate the applicability of the results presented in this paper. We use Pytorch (Paszke et al., 2017 ) and Nvidia Tesla V100 GPUs. Our code will be open-sourced. All the experimental details are given in appendix B , and we provide a recap on ResNet architectures in appendix C .
Validation of our model with step size \frac{1}{N}
The ResNet model ( 1 ) is different from the classical ResNet because of the \frac{1}{N} term. This makes the model depth aware, and we want to study the impact of this modification on the accuracy on CIFAR and ImageNet.

We first train a ResNet-101 (He et al., 2016a ) on CIFAR-10 and ImageNet using the same hyper-parameters. Experimental details are in appendix B and results are summarized in table 1 , showing that the explicit addition of the step size \frac{1}{N} does not affect accuracy. In strike contrast, the classical ResNet rule without the scaling \frac{1}{N} makes the network behave badly at large depth, while it still works well with our scaling \frac{1}{N} , as shown in Figure 3 (a). On ImageNet, the scaling \frac{1}{N} also leads to similar test accuracy in the weight tied setting: 72.5\% with 4 blocks per layer, 73.2\% with 8 blocks per layer and 72\% with 16 blocks per layer (mean over 2 runs).
New training strategy.
Our results in Prop. 3 and 4 assume uniform bounds in N on our residual functions and their derivatives. We also formally proved in the linear setting that these assumptions hold during the whole learning process if the initial loss is small.
A natural idea to start from a small loss is to consider a pretrained model.

In addition, we also want our pretrained model to verify assumption 2 so we consider the following setup. On CIFAR (resp. ImageNet) we train a ResNet with 4 (resp. 8) blocks in each layer, where weights are tied within each layer.
A first observation is that one can transfer these weights to deeper ResNets without significantly affecting the test accuracy of the model: it remains above 94.5\% on CIFAR-10 and 72\% on ImageNet. We then untie the weights of our models and refine them.
More precisely, for CIFAR, we then transfer the weights of our model to a ResNet with 4 , 4 , 64 and 4 blocks within each layer and fine-tune it only by refining the third layer, using our adjoint method.
We display in table 2 the median of the new test accuracy, over 5 runs for the initial pretraining of the model. For ImageNet, we transfer the weights to a ResNet with 100 blocks per layer and fine-tune the whole model with our adjoint method for the residual layers.
Results are summarized in table 2 . To the best of our knowledge, this is the first time a Neural-ODE like ResNet achieves a test-accuracy of 75.1\% on ImageNet.
Failure in usual settings.
In Prop. 3 we showed under assumption 2 , that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth N , then the error for computing the activations backward would scale in \frac{1}{N} as well as the error for the gradients (Prop. 4 ).
First, this results shows that the architecture needs to be deep enough, because it scales in \frac{1}{N} : for instance, we fail to train a ResNet-101 (He et al., 2016a ) on the ImageNet dataset using the adjoint method on its third layer (depth 23 ), as shown in Figure 3 (b).
Success at large depth.
To further investigate the applicability of the adjoint method for training deeper ResNets, we train a simple ResNet model on the CIFAR data set. First, the input is processed by a 5\times 5 convolution with 16 out channels, and the image is down-sampled to a size 10\times 10 .

We then apply a batch norm, a ReLU and iterate relation ( 1 ) where f is a pre-activation basic block (He et al., 2016b ) . We consider the zero residual initialisation: the last batch norm of each basic block is initialized to zero. We consider different values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy. We then compare the performance of our model using a ResNet (forward rule ( 1 )) or a HeunNet (forward rule ( 8 )).
We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas ( 6 ) and ( 9 )). We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4 . The true backpropagation gives the same curves for the ResNet and the HeunNet.
Approximated gradients, however, lead to a large test error at small depth, but give the same performance at large depth, hence confirming our results in Prop. 4 and 6 . In addition, at fixed depth, the accuracy when training a HeunNet with the adjoint method is better (or similar at depths 2 , 32 and 64 ) than for the ResNet with the adjoint method. This is to be linked with the two different bounds in Prop. 4 and 6 : for the HeunNet, smoothness with depth, which is expected at large depth, according to the theoretical results for the linear case (Prop. 2 ), implies a faster convergence to the true gradients for the HeunNet than for the ResNet. We finally validate this convergence in Figure 3 (c): the deeper the architecture, the better the approximation on the gradients. In addition, the HeunNet approximates the true gradient better than the ResNet.
Conclusion, limitations and future works
We propose a methodology to analyze how well a ResNet discretizes a Neural ODE. The positive results predicted by our theory in the linear case are also observed in practice with real architectures: one can successfully use the adjoint method to train ResNets (or even more effectively HeunNets) using very deep architectures on CIFAR, or fine-tune them on ImageNet, without memory cost in the residual layers. However, we also show that for large scale problems such as ImageNet classification from scratch, the adjoint method fails at usual depths.

Our work provides a theoretical guarantee for the convergence to a Neural ODE in the linear setting under a small loss initialization. A natural extension would be to study the non-linear case. In addition, the adjoint method is time consuming, and an improvement would be to propose a cheaper method than a reverse mode traversal of the architecture for approximating the activations.