A.5 Proof of Th. 1
We first prove the following lemma 3 before proving Th. 1 .

Under the assumptions of Th. 1 , let \sigma be such that \psi_{\sigma(N)}\to\psi_{\sigma} uniformly (in \|.\|_{\infty} w.r.t (s,t) ). Then one has \Pi^{\sigma(N)}(t)\to\Pi(t) uniformly (in t ) where \Pi(t) maps x_{0} to the solution at time 1 of the Neural ODE \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{\sigma}(s,t)x(s) with initial condition x_{0}.

Consider for x_{0}\in\mathbb{R}^{d} with \|x_{0}\|=1 the discrete scheme

x_{n+1}=x_{n}+\frac{1}{\sigma(N)}\theta^{\sigma(N)}_{n}(t)x_{n},

the ODE

\frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{\sigma}(s,t)x(s),

and the Euler scheme with time step \frac{1}{\sigma(N)} for its discretization

y_{n+1}=y_{n}+\frac{1}{\sigma(N)}\psi_{\sigma}(\frac{n}{\sigma(N)},t)y_{n}.

We know by Prop. 1 , since x_{0} has unit norm that

\|x(\frac{n}{\sigma(N)})-y_{n}\|\leq\frac{e^{\frac{1}{2}}-1}{\sigma(N)}\|\partial_{s}\psi_{\sigma}(.,t)+\psi_{\sigma}^{2}(.,t)\|^{{K}\times[0,1]}_{\infty}

where K is a compact that contains all the trajectory starting from any unit norm initial condition.
Since \forall t\in\mathbb{R}_{+} , \|\partial_{s}\psi_{\sigma}(s,t)\|\leq C and \|\psi_{\sigma}(s,t)^{2}\|\leq\frac{1}{2} , there exists \tilde{C}>0 and independent of t such that

\|x(\frac{n}{\sigma(N)})-y_{n}\|\leq\frac{\tilde{C}}{\sigma(N)}

Now, let e_{n}=y_{n}-x_{n} . We have

e_{n+1}=e_{n}(1+\frac{1}{\sigma(N)}\psi_{\sigma}(\frac{n}{\sigma(N)},t))+\frac{1}{\sigma(N)}(\psi_{\sigma}(\frac{n}{\sigma(N)},t)-\psi_{\sigma(N)}(\frac{n}{\sigma(N)},t))x_{n}.

Since \|\theta^{N}_{n}\|\leq\frac{1}{2} and x_{0} has unit norm, there exists M>0 independent of x_{0} such that, \forall n and N , \|x_{n}\|\leq M .
Thus

\|e_{n+1}\|\leq\|e_{n}\|(1+\frac{1}{2\sigma(N)})+\frac{1}{\sigma(N)}\sup_{(s,t)\in[0,1]\times\mathbb{R}_{+}}\|\psi_{\sigma}(s,t)-\psi_{\sigma(N)}(s,t)\|M.

The fact that \sup_{(s,t)\in[0,1]\times\mathbb{R}_{+}}\|\psi_{\sigma}(s,t)-\psi_{\sigma(N)}(s,t)\|\to 0 (uniform convergence of \psi_{\sigma(N)} to \psi_{\sigma} ) along with the discrete Gronwall’s lemma leads to \|e_{n}\|=o(1) independent of t and x_{0} . More precisely,

\sup_{t\in\mathbb{R}_{+},x_{0}\in\mathbb{R}^{d},\|x_{0}\|=1}\|\Pi^{\sigma(N)}(t)x_{0}-\Pi(t)x_{0}\|\to 0

as N\to\infty . We obtain the uniform convergence with t .

∎

We can now prove our Th. 1 .

Consider (\psi_{\sigma(N)})_{N} a sub-sequence of (\psi_{N})_{N} as in lemma 1 that converges to some \psi_{\sigma} .

1) We first prove the uniqueness of the limit.

We want to show that \psi_{\sigma} does not depend on \sigma . This will imply the uniqueness of any accumulation point of the relatively compact sequence (\psi_{N})_{N} and thus its convergence.

We have \forall s\in[0,1] ,

\partial_{t}\psi_{\sigma(N)}(s,t)=-\Pi^{\sigma(N)\top}_{:\lfloor\sigma(N)s\rfloor}(t)(\Pi^{\sigma(N)}(t)-B)\Pi^{\sigma(N)\top}_{\lfloor\sigma(N)s\rfloor:}(t).

As N\to\infty , we have thanks to lemma 3 that the right hand term converges uniformly to

-\Pi^{\top}_{:s}(t)(\Pi(t)-B)\Pi^{\top}_{s:}(t)

where \Pi maps x_{0} to the solution at time 1 of the Neural ODE \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{\sigma}(s,t)x(s) with initial condition x_{0} , \Pi_{:s}(t) maps x_{0} to the solution at time s of the Neural ODE \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{\sigma}(s,t)x(s) with initial condition x_{0} and \Pi_{s:}(t) maps x_{0} to the solution at time 1-s of the Neural ODE \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{\sigma}(s,t)x(s) with initial condition x_{0} .

This uniform convergence makes it possible to consider the limit ODE as N\to\infty :

\partial_{t}\psi_{\sigma}(.,t)=F(\psi_{\sigma}(.,t)),\quad\psi_{\sigma}(.,0)=0_{d\times d} (11)

where \forall s\in[0,1] ,

F(\psi_{\sigma}(s,t))=-\Pi^{\top}_{:s}(t)(\Pi(t)-B)\Pi^{\top}_{s:}(t).

We now show that F is Lipschitz continuous which will guarantee uniqueness through the Picard–Lindelöf theorem. Recall that we have \forall(s,t)\in[0,1]\times\mathbb{R}_{+} :

\|\psi_{\sigma}(s,t)\|\leq\frac{1}{2}.

Let \psi_{1} , \psi_{2} with \|\psi_{1}(s,t)\|\leq\frac{1}{2} and \|\psi_{2}(s,t)\|\leq\frac{1}{2} and \Pi_{1}(t) , \Pi_{2}(t) the corresponding flows.

Let x_{0} in \mathbb{R}^{d} with unit norm, x_{1} (resp. x_{2} ) be the solutions of \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{1}(s,t)x(s) (resp. \frac{{\mathrm{d}}x}{{\mathrm{d}}s}=\psi_{2}(s,t)x(s) ) with initial condition x_{0} . Let y=x_{1}-x_{2} .

One has \Pi_{1}(t)x_{0}=x_{1}(1) and \Pi_{2}(t)x_{0}=x_{2}(1) . One has \dot{y}=\psi_{1}x_{1}-\psi_{2}x_{2}=\psi_{2}y+(\psi_{1}-\psi_{2})x_{1} . Hence, since y(0)=0 , \|y(s)\|\leq\int_{0}^{s}\|\psi_{2}\|\|y\|+\|\psi_{1}-\psi_{2}\|_{\infty}|\|x_{1}\|_{\infty} , we have

\|y(s)\|\leq\frac{1}{2}\|y(s)\|+\|\psi_{1}-\psi_{2}\|_{\infty}.\|\Pi_{1}(t)\|

and since \forall t\in\mathbb{R}_{+} , \|\Pi_{1}(t)\|\leq 2e we get

\|\Pi_{1}(t)x_{0}-\Pi_{2}(t)x_{0}\|=\|y(1)\|\leq\alpha\|\psi_{1}-\psi_{2}\|_{\infty}

for some \alpha>0 . The same arguments go for \Pi_{:s} and \Pi_{s:} .

Since we only consider maps \psi_{\sigma} such that \|\psi_{\sigma}(s,t)\|\leq\frac{1}{2} , this implies that the product is also Lipschitz and thus F is Lipschitz. This guarantees the uniqueness of a solution \psi to the Cauchy problem and we have that \psi_{N}\to\psi uniformly.

2) We now turn to the convergence speed.

We have \|\psi_{2N}-\psi_{N}\|\leq\frac{D}{N} for some D>0 thanks to lemma 2 . For k\in\mathbb{N} , we have that

\|\psi_{2^{k}N}-\psi_{N}\|\leq\sum_{i=0}^{k-1}\|\psi_{2^{i+1}N}-\psi_{2^{i}N}\|\leq\frac{D}{N}\sum_{i=0}^{k-1}\frac{1}{2^{i}}\leq\frac{2D}{N}.

Letting k\to\infty finally gives \|\psi-\psi_{N}\|\leq\frac{2D}{N}.

∎