A.9 Proof of Prop. 6
1) We first control the error made in the gradient with respect to activations. We have the following recursions:

abla_{x_{n}}L=(I+\frac{1}{N}\partial_{x}\varphi_{n}(x_{n+1}))^{\top} abla_{x_{n+1}}L and abla_{\tilde{x}_{n}}L=(I+\frac{1}{N}\partial_{x}\varphi_{n}(\tilde{x}_{n+1}))^{\top} abla_{\tilde{x}_{n+1}}L

Letting r^{\prime}_{n}=\nabla_{x_{n}}L-\nabla_{\tilde{x}_{n}}L , we have

r^{\prime}_{n}=r^{\prime}_{n+1}+\frac{1}{N}\partial_{x}\varphi_{n}(x_{n+1})^{\top}r^{\prime}_{n+1}+\frac{1}{N}\left(\partial_{x}\varphi_{n}(x_{n+1})-\partial_{x}\varphi_{n}(\tilde{x}_{n+1})\right)^{\top} abla_{\tilde{x}_{n+1}}L

Therefore, using the triangle inequality, and letting g a bound on the norm of the gradients \nabla_{\tilde{x}_{n+1}}L and \Delta a Lipschitz constant of \partial_{x}\varphi_{n} , we find

\|r^{\prime}_{n}\|\leq(1+\frac{L_{f}^{\prime}}{N})\|r^{\prime}_{n+1}\|+\frac{1}{N}g\Delta\|x_{n+1}-\tilde{x}_{n+1}\|

The last term is controled with the previous proposition, and we find

\|r^{\prime}_{n}\|\leq(1+\frac{L_{f}^{\prime}}{N})\|r^{\prime}_{n+1}\|+\frac{(e^{L_{f}^{\prime}}-1)C_{f}^{\prime}g\Delta}{L_{f}^{\prime}N^{2}}\Delta_{\theta}^{N}+O(\frac{1}{N^{3}}),

which gives by unrolling:

\|r_{n}^{\prime}\|\leq\frac{(e^{L_{f}^{\prime}}-1)^{2}C_{f}^{\prime}g\Delta}{L_{f}^{\prime 2}N}\Delta_{\theta}^{N}+O(\frac{1}{N^{2}}).

2) We can now control the gradients with respect to parameters. Since Heun’s method involves parameters \theta_{n}^{N} both for the computation of x_{n} and x_{n+1} , the gradient formula is slightly more complicated than for the classical ResNet. It is the sum of two terms, the first one \nabla^{1}_{\theta^{N}_{n}}L corresponding to iteration n and the second one \nabla^{2}_{\theta^{N}_{n}}L corresponding to iteration n-1 .

We have

abla^{1}_{\theta^{N}_{n}}L=\frac{1}{2N}\left(\partial_{\theta}f(x_{n},\theta_{n}^{N})+\frac{1}{N}\partial_{x}f(y_{n},\theta_{n+1}^{N})\partial_{\theta}f(x_{n},\theta_{n}^{N})\right)^{\top} abla_{x_{n}}L

and

abla^{2}_{\theta^{N}_{n}}L=\frac{1}{2N}\left(\partial_{\theta}f(y_{n-1},\theta_{n-1}^{N})\right)^{\top}(I+\frac{1}{N}\partial_{x}f(x_{n-1},\theta_{n-1}^{N}))^{\top} abla_{x_{n-1}}L.

The gradient \nabla_{\theta^{N}_{n}}L is finally

abla_{\theta^{N}_{n}}L= abla^{1}_{\theta^{N}_{n}}L+ abla^{2}_{\theta^{N}_{n}}L.

Overall, these equations map the activations x_{n} and x_{n-1} , and the gradients \nabla_{x_{n-1}}L and \nabla_{x_{n}}L to the gradient \nabla_{\theta_{n}^{N}} , which we rewrite as

abla_{\theta^{N}_{n}}L=\Psi(x_{n},x_{n-1}, abla_{x_{n}}L, abla_{x_{n-1}}L),

where the function \Psi is explicitly defined by the above equations.
With the memory-free backward pass, the gradient is rather estimated as

\tilde{ abla}_{\theta^{N}_{n}}L=\Psi(\tilde{x}_{n},\tilde{x}_{n-1}, abla_{\tilde{x}_{n}}L, abla_{\tilde{x}_{n-1}}L).

The function \Psi is Lispchitz-continuous since all functions involved in its composition are Lipschitz-continuous and the activations belong to a compact set, and its Lipschitz constant scales as \frac{1}{N} . We write its Lipschitz constant as \frac{L_{\Psi}}{N} , and we get:

\| abla_{\theta^{N}_{n}}L-\tilde{ abla}_{\theta^{N}_{n}}L\| =\|\Psi(x_{n},x_{n-1}, abla_{x_{n}}L, abla_{x_{n-1}}L)-\Psi(\tilde{x}_{n},\tilde{x}_{n-1}, abla_{\tilde{x}_{n}}L, abla_{\tilde{x}_{n-1}}L)\| (18) \leq\frac{L_{\Psi}}{N}(\|x_{n}-\tilde{x}_{n}\|+\|x_{n-1}-\tilde{x}_{n-1}\|+\| abla_{x_{n}}L- abla_{\tilde{x}_{n}}L\|+\| abla_{x_{n-1}}L- abla_{\tilde{x}_{n-1}}L\|)

Using the previous propositions, we get:

\| abla_{\theta^{N}_{n}}L-\tilde{ abla}_{\theta^{N}_{n}}L\|=O(\frac{\Delta_{\theta}^{N}}{N^{2}}+\frac{1}{N^{3}}).

∎