Problem setup.
Residual Neural Networks (ResNets) (He et al., 2016a , b ) keep on outperforming state of the art in computer vision (Wightman et al., 2021 ; Bello et al., 2021 ) , and more generally skip connections are widely used in a various range of applications (Vaswani et al., 2017 ; Dosovitskiy et al., 2020 ) .
A ResNet of depth N iterates, starting from x_{0}\in\mathbb{R}^{d} , x_{n+1}=x_{n}+f(x_{n},\theta_{n}^{N}) and outputs a final value x_{N}\in\mathbb{R}^{d} where f is a neural network called residual function. In this work, we consider a simple modification of this forward rule by letting explicitly the residual mapping depend on the depth of the network:

x_{n+1}=x_{n}+\frac{1}{N}f(x_{n},\theta_{n}^{N})

On the other hand, a Neural ODE (Chen et al., 2018 ) uses a neural network \varphi_{\Theta}(x,s) , that takes time s into account, to parameterise a vector field (Kidger, 2022 ) in a differential equation, as follows,

\frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\varphi_{\Theta}(x(s),s)\quadwith\quad x(0)=x_{0}, (2)

and outputs a final value x(1)\in\mathbb{R}^{d} , the solution of Eq.( 2 ).
The Neural ODE framework enables learning without storing activations (the x_{n} ’s) using the adjoint state method, hence significantly reducing the memory usage for backpropagation that can be a bottleneck during training (Wang et al., 2018 ; Peng et al., 2017 ; Zhu et al., 2017 ; Gomez et al., 2017 ) .

Neural ODEs also provide a theoretical framework to study deep learning models from the continuous viewpoint, using the arsenal of ODE theory (Teh et al., 2019 ; Li et al., 2019 ; Teshima et al., 2020 ) .
Importantly, they can also be seen as the continuous analog of ResNets.
Indeed, consider for N an integer, the Euler scheme for solving Eq. ( 2 ) with time step \frac{1}{N} starting from x_{0} and iterating x_{n+1}=x_{n}+\frac{1}{N}\varphi_{\Theta}(x_{n},\frac{n}{N}) . Under mild assumptions on \varphi_{\Theta} , this scheme is known to converge to the true solution of Eq. ( 2 ) as N goes to +\infty . Also, if \Theta=(\theta_{n}^{N})_{i\in[N-1]} and \varphi_{\Theta}(.,\frac{n}{N})=f(.,\theta_{n}^{N}) , then the ResNet equation Eq. ( 1 ) corresponds to a Euler discretization with time step \frac{1}{N} of Eq. ( 2 ). However, for a given ResNet with fixed depth N and weights, the activations in Eq. ( 1 ) can be far from the solution of Eq. ( 2 ).
This is illustrated in Figure 1 where we show that a deep ResNet can easily break the topology of the input space, which is impossible for a Neural ODE.
In this paper, we study the link between ResNets and Neural ODEs.
We make the following contributions:

In Section 3 , we propose a framework to define a set of associated Neural ODEs for a given ResNet. We control the error between the discrete and the continuous trajectory. We show that without additional assumptions on the smoothness with depth of the residual functions, this error does not go to 0 as N\to\infty (Prop. 1 ). However, we show that under some assumptions on the weight initialization, the trained parameters of a deep linear ResNet uniformly (with respect to both depth and training time) approach a Lipschitz function as the depth N of the network goes to infinity, at speed \frac{1}{N} (Prop. 2 and Th. 1 ). This result highlights an implicit regularization towards a limit Neural ODE.

In Section 4 , we investigate a simple technique to train ResNets without storing activations. Inspired by the adjoint method, we propose to recover the approximated activations during the backward pass by using a reverse-time Euler scheme. We control the error for recovering the activations and gradients with this method. We show that if the residuals of the ResNet are bounded and Lipschitz continuous, with constants independent of N , then this error scales in O(\frac{1}{N}) (Prop. 3 ).
Hence, the adjoint method needs a large number of layers to lead to correct gradients (Prop. 4 ). We then consider a smoothness-dependent reconstruction with Heun’s method to bound the error between the true and approximated gradient by a term that depends on \frac{1}{N} times the smoothness in depth of the residual functions, hence guaranteeing a better approximation when successive weights are close one to another (Prop. 5 and 6 ).

In Section 5 , on the experimental side, we show that the adjoint method fails when training a ResNet 101 on ImageNet. Nevertheless, we empirically show that very deep ResNets pretrained with tied weights (constant weights: \theta^{N}_{n}=\theta \forall n ) can be refined -using our adjoint method- on CIFAR-10 and ImageNet by untying their weights, leading to a better test accuracy. Last, but not least, we show using a ResNet architecture with heavy downsampling in the first layer that our adjoint method succeeds at large depth and that Heun’s method leads to a better behaved training, hence confirming our theoretical results.