Two continuous variables involved.
Our results involve two continuous variables: s\in[0,1] is the depth of the limit network and corresponds to the time variable in the Neural ODE, whereas t\in\mathbb{R}_{+} is the gradient flow time variable.
As is standard in the analysis of convergence of gradient descent for linear networks, we consider the following assumption:

Suppose that at initialisation one has \sqrt{{\ell}^{N}(0)}<\frac{m}{4\sqrt{2Me^{3}}} and \|\theta_{n}^{N}(0)\|\leq\frac{1}{4} .

Assumption 1 is the classical assumption in the literature (Zou et al., 2020 ; Barboni et al., 2021 ) to prove linear convergence of our loss and that the \theta^{N}_{n}(t) ’s stay bounded with t . Note that this bounded norm assumption implies that \frac{1}{N}\theta_{n}(0)=O(\frac{1}{N}) . This is in contrast with classical initialization scales in the feedforward case where the initialization only depends on width (He et al., 2015 ) . However this initialization scale is coherent with those of ResNets for which the scale has to depend on depth (Yang and Schoenholz, 2017 ) . In addition, the experimental findings in Cohen et al. ( 2021 ) suggest that the weights in ResNets scale in \frac{1}{N^{\beta}} with \beta>0 .

We now prove an implicit regularization result showing that if at initialization, in addition to assumption 1 , the weights are close from one another ( O(\frac{1}{N}) ), they will stay at distance O(\frac{1}{N}) : the discrete derivative stay in O(\frac{1}{N}) , which is a central result to consider the infinite depth limit in our Th. 1 .

Suppose assumption 1 . Suppose that there exists C_{0}>0 independent of n and N such that \|\theta_{n+1}^{N}(0)-\theta_{n}^{N}(0)\|\leq\frac{C_{0}}{N} . Then, \forall t\in\mathbb{R}_{+} , \|\theta_{n}^{N}(t)\|<\frac{1}{2} , and \theta_{n}^{N}(t) admits a limit \psi_{n}^{N} as {t\to+\infty} . Moreover, there exists C>0 such that \forall t\in\mathbb{R}_{+} , \|\theta_{n+1}^{N}(t)-\theta_{n}^{N}(t)\|\leq\frac{C}{N} .

For a full proof, see appendix A.2 . The inequality \|\theta_{n+1}^{N}(t)-\theta_{n}^{N}(t)\|\leq\frac{C}{N} corresponds to a discrete Lipschitz property in depth. Indeed, for s\in[0,1] and t\in\mathbb{R}_{+} , let \psi_{N}(s,t)=\theta^{N}_{\lfloor Ns\rfloor}(t) . Then our result gives \|\psi_{N}(\frac{n+1}{N},t)-\psi_{N}(\frac{n}{N},t)\|\leq\frac{C}{N} which implies that \|\psi_{N}(s_{1},t)-\psi_{N}(s_{2},t)\|\leq C|s_{1}-s_{2}|+\frac{C}{N} .
We now turn to the infinite depth limit N\to\infty .
Th. 1 shows that there exists a limit function \psi such that \psi_{N} converges uniformly to \psi in depth s and optimization time t . Furthermore, this limit is Lipschitz continuous in (s,t) .
In addition, we show that the ResNet \Pi^{N} converges to the limit Neural ODE defined by \psi that is preserved along the optimization flow, exhibiting an implicit regularization property of deep linear ResNets towards Neural ODEs.

Suppose assumption 1 , \|\theta_{n+1}^{N}(0)-\theta_{n}^{N}(0)\|\leq\frac{C_{0}}{N} for some C_{0}>0 and that there exists a function \psi_{\mathrm{init}}:[0,1]\to\mathbb{R}^{d\times d} such that \psi_{N}(s,0)\to\psi_{\mathrm{init}}(s) in \|.\|_{\infty} uniformly in s as N\to\infty , at speed \frac{1}{N} .
Then the sequence (\psi_{N})_{N\in\mathbb{N}} uniformly converges (in \|.\|_{\infty} w.r.t (s,t) ) to a limit \psi Lipschitz continuous in (s,t) and \|\psi-\psi_{N}\|_{\infty}=O(\frac{1}{N}) . Furthermore, \Pi^{N} uniformly converges as N\to\infty to the mapping x_{0}\to x_{1} where x_{1} is the solution at time 1 of the Neural ODE \frac{{\mathrm{d}}x}{{\mathrm{d}}s}(s)=\psi(s,t)x(s) with initial condition x_{0} .

We illustrate Th. 1 in Figure 2 . The assumption on the existence of \psi_{\mathrm{init}} ensures a convergence at speed \frac{1}{N} to a Neural ODE at optimization time 0 . Note that for instance, the constant initialization \theta_{n}^{N}(0)=\theta_{0}\in\mathbb{R}^{d\times d} satisfies this hypothesis.
In order to prove Th. 1 , for which a full proof is presented in appendix A.5 , we first present a useful lemma: the weights of the network have at least one accumulation point.

For s\in[0,1] and t\in\mathbb{R}_{+} , let \psi_{N}(s,t)=\theta^{N}_{\lfloor Ns\rfloor}(t) . Under the assumptions of Prop. 2 , there exists a subsequence \psi_{\sigma(N)} and \psi_{\sigma}:[0,1]\times\mathbb{R}_{+}\to\mathbb{R}^{d\times d} Lipschitz continuous with respect to both parameters s and t such that \psi_{\sigma(N)}\to\psi_{\sigma} uniformly (in \|.\|_{\infty} w.r.t (s,t) ).

Lemma 1 is proved in appendix A.3 , and gives us the existence of a Lipschitz continuous accumulation point, but not the uniqueness nor the convergence speed.
For the uniqueness, we show in appendix A.5 that, under the assumptions of Th. 1 , one has that any accumulation point of \psi_{\sigma} satisfies the limit Neural ODE

\partial_{t}\psi_{\sigma}(.,t)=F(\psi_{\sigma}(.,t)),\quad\psi_{\sigma}(.,0)=\psi_{\mathrm{init}}(.,0),

and show that F satisfies the hypothesis of the Picard–Lindelöf theorem, hence showing the uniqueness of \psi .
We finally show that, as intuitively expected, trajectories of the weights of our linear ResNets of depth N and 2N remain close one to each other. This gives the convergence speed in Th. 1 . See appendix A.4 for a proof.

Suppose asumption 1 , \|\theta_{n+1}^{N}(0)-\theta_{n}^{N}(0)\|\leq\frac{C_{0}}{N} for some C_{0}>0 and that \|\theta_{n}^{N}(0)-\theta_{2n}^{2N}(0)\|=O(\frac{1}{N}) .
Then \forall t\in\mathbb{R}_{+} , \|\theta_{n}^{N}(t)-{\theta}_{2n}^{2N}(t)\|=O(\frac{1}{N}) .