RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Acknowledgments
This work was granted access to the HPC resources of IDRIS under the allocation 2020-[AD011012073] made by GENCI. This work was supported in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d’avenir” program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute). This work was
supported in part by the European Research Council (ERC project NORIA). M. S. thanks Mathieu Blondel and Zaccharie Ramzi for helpful discussions.
A.1 Proof of Prop. 1
Our proof is inspired by [Demailly, 2016 ] .

We denote h=\frac{1}{N} and s_{n}=nh . We define

\varepsilon_{n}=x(s_{n+1})-x(s_{n})-h\varphi_{\Theta}(x(s_{n}),s_{n}).

We have that \varphi_{\Theta}(x(s_{n}),s_{n})=\dot{x}(s_{n}) .

Taylor’s formula gives

x(s_{n}+h)=x(s_{n})+h\dot{x}(s_{n})+R_{1}(h)

with \|R_{1}(h)\|\leq\frac{1}{2}h^{2}\|\ddot{x}\|_{\infty} .
This implies that

\|\varepsilon_{n}\|\leq\frac{1}{2}h^{2}\|\ddot{x}\|_{\infty}.

The true error we are interested in is the global error e_{n}=x(s_{n})-x_{n} . One has

e_{n+1}-e_{n}=x(s_{n+1})-x(s_{n})+x_{n}-x_{n+1}=\varepsilon_{n}+h(\varphi_{\Theta}(x(s_{n}),s_{n})-\varphi_{\Theta}(x_{n},s_{n})).

Because \varphi_{\Theta} is L -Lipschitz, this gives \|e_{n+1}-e_{n}\|\leq\|\varepsilon_{n}\|+hL\|e_{n}\| and hence

\|e_{n+1}\|\leq(1+hL)\|e_{n}\|+\|\varepsilon_{n}\|.

Because h=\frac{1}{N} , we have

\|e_{n+1}\|\leq(1+\frac{L}{N})\|e_{n}\|+\frac{1}{2N^{2}}\|\ddot{x}\|_{\infty}.

this implies from the discrete Gronwall lemma, since e_{0}=0 that

\|e_{n}\|\leq\frac{e^{L}-1}{2NL}\|\ddot{x}\|_{\infty}.

Note that we have \ddot{x}=\partial_{s}\varphi_{\Theta}+\partial_{x}\varphi_{\Theta}[\varphi_{\Theta}]. This gives the desired result.
∎
A.2 Proof of Prop. 2
Recall that we denote \Pi^{N}=\prod_{n=1}^{N}(I_{d}+\frac{\theta_{n}^{N}}{N}) , \Pi^{N}_{:n}=(I_{d}+\frac{\theta_{N}^{N}}{N})\dots(I_{d}+\frac{\theta_{n+1}^{N}}{N}) and \Pi^{N}_{n:}=(I_{d}+\frac{\theta_{n-1}^{N}}{N})\dots(I_{d}+\frac{\theta_{1}^{N}}{N}) .
We denote \nabla^{N}_{n}=\nabla_{\theta_{n}^{N}}L . One has

N abla^{N}_{n}=\Pi_{:n}^{N\top}(\Pi^{N}-B)\Sigma\Pi_{n:}^{N\top}.

One has as in [Zou et al., 2020 ] that

\sigma^{2}_{max}(\Pi^{N}_{n:})\sigma^{2}_{max}(\Pi^{N}_{:n})\|\Pi-B\|_{\Sigma}^{2}M\geq N^{2}\| abla^{N}_{n}\|^{2}\geq\sigma^{2}_{min}(\Pi^{N}_{n:})\sigma^{2}_{min}(\Pi^{N}_{:n})\|\Pi-B\|_{\Sigma}^{2}m.

where \sigma_{max}(A) (resp. \sigma_{min}(A) ) denotes the largest (resp. smallest) singular value of A .
We first show that \forall t\in\mathbb{R}_{+} , \|\theta_{n}^{N}(t)\|<\frac{1}{2} . Denote

t^{*}=\inf\{t\in\mathbb{R}_{+},\exists n\in[N-1],\|\theta_{n}^{N}(t)\|\geq 1/2\}.

One has that \forall t\in[0,t^{*}] , \sigma^{2}_{min}(\Pi^{N}_{:n})\geq(1-\frac{1}{2N})^{2(N-n)} and \sigma^{2}_{min}(\Pi^{N}_{n:})\geq(1-\frac{1}{2N})^{2(n-1)} which implies that

N^{2}\| abla^{N}_{n}(t)\|^{2}\geq 2(1-\frac{1}{2N})^{2N-2}{\ell}^{N}(t)m\geq\frac{2}{e}{\ell}^{N}(t)m.

Similarly one has N^{2}\|\nabla^{N}_{n}(t)\|^{2}\leq 2e{\ell}^{N}(t)M . To summarize, we have the PL conditions for t\in[0,t^{*}] :

\frac{2}{e}m{\ell}^{N}(t)\leq N^{2}\| abla^{N}_{n}(t)\|^{2}\leq 2eM{\ell}^{N}(t).

As a consequence, one has

\frac{{\mathrm{d}}{\ell}^{N}}{{\mathrm{d}}t}(t)=-N\sum_{n=1}^{N}\| abla^{N}_{n}(t)\|^{2}\leq-\frac{2}{e}m{\ell}^{N}(t)

and thus {\ell}^{N}(t)\leq e^{-\frac{2}{e}mt}{\ell}^{N}(0) .

We have \theta^{N}_{n}(t^{*})=\theta^{N}_{n}(0)+N\int_{0}^{t^{*}}\nabla^{N}_{n} and \|\nabla^{N}_{n}\|\leq\frac{\sqrt{2eM}}{N}\sqrt{{\ell}^{N}} so that

\|\theta^{N}_{n}(t^{*})\|\leq\|\theta^{N}_{n}(0)\|+\sqrt{2eM}\int_{0}^{t^{*}}e^{-\frac{1}{e}mt}\sqrt{{\ell}^{N}(0)}dt<\frac{1}{4}+\frac{1}{4}<1/2.

This is absurd by definition of t^{*} and thus shows that \forall t\in\mathbb{R}_{+},\|\theta_{n}^{N}(t)\|<\frac{1}{2}. We also see that \nabla^{N}_{n} is integrable so that \theta_{n}^{N}(t) admits a limit as t\to\infty .

We now show our main result. Note that we have the relationship (I+\frac{\theta_{n+1}^{N}}{N})^{\top}\nabla_{n+1}=\nabla_{n}(I+\frac{\theta_{n}^{N}}{N})^{\top} so that

abla^{N}_{n+1}- abla^{N}_{n}=(I+\frac{\theta^{N\top}_{n+1}}{N})^{-1}(\frac{ abla^{N}_{n}\theta_{n}^{N\top}-\theta_{n+1}^{N\top} abla_{n}^{N}}{N}).

Because \|(I+A)^{-1}\|\leq 2 if \|A\|\leq\frac{1}{2} this gives \|\nabla^{N}_{n+1}-\nabla^{N}_{n}\|\leq\frac{2}{N}\|\nabla^{N}_{n}\| . Integrating we get

\|\theta^{N}_{n+1}(t)-\theta^{N}_{n}(t)\|\leq\|\theta^{N}_{n+1}(0)-\theta^{N}_{n}(0)\|+2\int_{0}^{t}\| abla^{N}_{n}\|.

This gives

\|\theta^{N}_{n+1}(t)-\theta^{N}_{n}(t)\|\leq O(\frac{1}{N})+\frac{1}{N}2\int_{0}^{t}\sqrt{2eM{\ell}^{N}(0)}e^{-\frac{1}{e}mt}dt=O(\frac{1}{N}),

which is the desired result.

∎
A.6 Proof of Prop. 3
We denote r_{n}=\tilde{x}_{n}-x_{n} .

One has r_{N}=0 and

r_{n}=\tilde{x}_{n+1}-\frac{1}{N}f(\tilde{x}_{n+1},\theta^{N}_{n})-x_{n+1}+\frac{1}{N}f({x}_{n},\theta^{N}_{n}),

that is

r_{n}=r_{n+1}+\frac{1}{N}(f(x_{n+1}-\frac{1}{N}f({x}_{n},\theta^{N}_{n}),\theta^{N}_{n})-f(\tilde{x}_{n+1},\theta^{N}_{n})).

Since

f(x_{n+1}-\frac{1}{N}f({x}_{n},\theta^{N}_{n}),\theta_{n}^{N})=f(x_{n+1},\theta^{N}_{n})-\frac{1}{N}\partial_{x}{f(x_{n+1},\theta_{n}^{N})}[f({x}_{n},\theta^{N}_{n})]+O(\frac{1}{N^{2}})

this gives

r_{n}=r_{n+1}+\frac{1}{N}(f(x_{n+1},\theta^{N}_{n})-f(\tilde{x}_{n+1},\theta^{N}_{n}))-\frac{1}{N^{2}}\partial_{x}{f(x_{n+1},\theta_{n}^{N})}[f({x}_{n},\theta^{N}_{n})]+O(\frac{1}{N^{3}}).

Denoting

K_{N}=\sup_{n\in[N-1]}\|\partial_{x}{f(.,\theta_{n}^{N})}\|^{K}_{\infty}\|f(.,\theta_{n}^{N})\|^{K}_{\infty},

we have the following inequality:

\|r_{n}\|\leq(1+\frac{L_{f}}{N})\|r_{n+1}\|+\frac{1}{N^{2}}K_{N}+O(\frac{1}{N^{3}})

and since r_{N}=0 , the discrete Gronwall lemma leads to \|r_{n}\|\leq\frac{e^{L_{f}}-1}{{L_{f}}N}K_{N}+O(\frac{1}{N^{2}}). In addition, one has K_{N}\leq L_{f}C_{f} so that

\|r_{n}\|\leq\frac{e^{L_{f}}-1}{N}C_{f}+O(\frac{1}{N^{2}}).

∎
A.7 Proof of Prop. 4
1) We first control the error made in the gradient with respect to activations.

Denote

g_{n}= abla_{\tilde{x}_{n}}L- abla_{x_{n}}L.

One has using formulas ( 3 ) and ( 7 ) that

g_{n}=g_{n+1}+\frac{1}{N}(\partial_{x}f(\tilde{x}_{n},\theta^{N}_{n})-\partial_{x}f(x_{n},\theta^{N}_{n}))^{\top} abla_{\tilde{x}_{n+1}}L+\frac{1}{N}\partial_{x}f(x_{n},\theta^{N}_{n})^{\top}g_{n+1}.

Since

\|\partial_{x}f(x_{n},\theta^{N}_{n})^{\top}g_{n+1}\|\leq L_{f}\|g_{n+1}\|

and because

\|(\partial_{x}f(\tilde{x}_{n},\theta^{N}_{n})-\partial_{x}f(x_{n},\theta^{N}_{n}))^{\top} abla_{\tilde{x}_{n+1}}L\|\leq L_{df}\|\tilde{x}_{n}-x_{n}\|g,

where g is a bound on \nabla_{\tilde{x}_{n+1}}L , we conclude by using Prop. 3 and the discrete Gronwall’s lemma.

2) We can now control the gradients with respect to the parameters \theta^{N}_{n} ’s.

Denote

t_{n}=\tilde{ abla}_{\theta^{N}_{n}}L-{ abla}_{\theta_{n}^{N}}L.

We have

Nt_{n}=-[\partial_{\theta}f({x}_{n},\theta_{n}^{N})-\partial_{\theta}f(\tilde{x}_{n},\theta_{n}^{N})]^{\top} abla_{{x}_{n}}L-[\partial_{\theta}f(\tilde{x}_{n},\theta_{n}^{N})]^{\top}g_{n}.

Hence N\|t_{n}\|\leq L_{\theta}\|x_{n}-\tilde{x}_{n}\|g+C_{\theta}\|g_{n}\| where g is a bound on \nabla_{x_{n}}L .

Using our bound on \|g_{n}\| and Prop. 3 we get

N\|t_{n}\|\leq\frac{L_{\theta}(e^{L_{f}}-1)gC_{f}}{N}+\frac{(e^{L_{f}}-1)(e^{L_{f}}-1)C_{f}L_{df}gC_{\theta}}{L_{f}N}+O(\frac{1}{N^{2}})

and thus

t_{n}=O(\frac{1}{N^{2}}).

∎
A.8 Proof of Prop. 5
In the following, we let for short f_{n}(x)=f(x,\theta_{n}^{N}) , and we define

\varphi_{n}(x)=\frac{1}{2}\left(f_{n}(x)+f_{n+1}(x+\frac{1}{N}f_{n}(x))\right) and \psi_{n}(x)=\frac{1}{2}\left(f_{n+1}(x)+f_{n}(x-\frac{1}{N}f_{n+1}(x))\right) (12)

so that Heun’s forward and backward equations are

x_{n+1}=x_{n}+\frac{1}{N}\varphi_{n}(x_{n}) and \tilde{x}_{n}=\tilde{x}_{n+1}-\frac{1}{N}\psi_{n}(\tilde{x}_{n+1}).

We have the following lemma that quantifies the reconstruction error over one iteration:

For x\in\mathbb{R} , we have as N goes to infinity

\psi_{n}(x+\frac{1}{N}\varphi_{n}(x))-\varphi_{n}(x)=\frac{1}{4N}\left(J_{n+1}(x)-J_{n}(x)\right)[f_{n+1}(x)-f_{n}(x)]+O(\frac{1}{N^{2}}),

where J_{n}=\partial_{x}f_{n}(x) is the Jacobian of f_{n} .

As N goes to infinity, we have the following expansions of ( 12 ):

\varphi_{n}(x)=\frac{1}{2}(f_{n}(x)+f_{n+1}(x))+\frac{1}{2N}J_{n+1}(x)[f_{n}(x)]+O(\frac{1}{N^{2}}),

\psi_{n}(x)=\frac{1}{2}(f_{n}(x)+f_{n+1}(x))-\frac{1}{2N}J_{n}(x)[f_{n+1}(x)]+O(\frac{1}{N^{2}}).

As a consequence, we have

\psi_{n}(x+\frac{1}{N}\varphi_{n}(x))= \frac{1}{2}(f_{n}(x)+f_{n+1}(x))-\frac{1}{2N}J_{n}(x)[f_{n+1}(x)] +\frac{1}{4N}(J_{n}(x)[f_{n}(x)+f_{n+1}(x)]+J_{n+1}(x)[f_{n}(x)+f_{n+1}(x)])+O(\frac{1}{N^{2}}).

Putting everything together, we find that the zero-th order in \psi_{n}(x+\frac{1}{N}\varphi_{n}(x))-\varphi_{n}(x) cancels, and that the first order simplifies to \frac{1}{4N}\left(J_{n+1}(x)-J_{n}(x)\right)[f_{n+1}(x)-f_{n}(x)] .
∎

We now turn the the proof of the main proposition:

We let r_{n}=\tilde{x}_{n}-x_{n} the reconstruction error. We have r_{N}=0 , and we find

 r_{n} =\tilde{x}_{n}-x_{n} (13) =\tilde{x}_{n+1}-\frac{1}{N}\psi_{n}(\tilde{x}_{n+1})-x_{n+1}+\frac{1}{N}\varphi_{n}(x_{n}) (14) =r_{n+1}-\frac{1}{N}\left(\psi_{n}(\tilde{x}_{n+1})-\psi_{n}(x_{n+1})\right)-\frac{1}{N}\left(\psi_{n}(x_{n+1})-\varphi_{n}(x_{n})\right)

Using the triangle inequality, and the L_{f}^{\prime}- Lispchitz continuity of \psi_{n} , we get

\|r_{n}\|\leq(1+\frac{L_{f}^{\prime}}{N})\|r_{n+1}\|+\frac{1}{N}\|\psi_{n}(x_{n+1})-\varphi_{n}(x_{n})\|.

The last term is controlled with the previous Lemma 4 :

\|\psi_{n}(x_{n+1})-\varphi_{n}(x_{n})\| \leq\frac{1}{4N}\|\left(J_{n+1}(x_{n})-J_{n}(x_{n})\right)[f_{n+1}(x_{n})-f_{n}(x_{n})]\|+O(\frac{1}{N^{2}}) (16) \leq\frac{C_{f}^{\prime}\Delta_{\theta}^{N}}{N}+O(\frac{1}{N^{2}})

We therefore get the recursion

\|r_{n}\|\leq(1+\frac{L_{f}^{\prime}}{N})\|r_{n+1}\|+\frac{C_{f}^{\prime}\Delta_{\theta}^{N}}{N^{2}}+O(\frac{1}{N^{3}}).

Unrolling the recursion gives,

\|r_{n}\|\leq\frac{(e^{L_{f}^{\prime}}-1)C_{f}^{\prime}}{L_{f}^{\prime}N}\Delta_{\theta}^{N}+O(\frac{1}{N^{2}}).

∎
B.1 CIFAR
For our experiments on CIFAR-10 (training from scratch), we used a batch-size of 128 and we employed SGD with a momentum of 0.9 . The training was done over 200 epochs. The initial learning rate was 0.1 and we used a cosine learning rate scheduler. A constant weight decay was set to 5\times 10^{-4} . Standard inputs preprocessing as proposed in Pytorch [Paszke et al., 2017 ] was performed.

For our finetuning experiment on CIFAR-10, we used a batch-size of 128 and we employed SGD with a momentum of 0.9 . The training was done over 5 epochs. The learning rate was kept constant to 10^{-3} . A constant weight decay was set to 5\times 10^{-4} . Standard inputs preprocessing as proposed in Pytorch was also performed.

For our experiment with our simple ResNet model that processes the input by a 5\times 5 convolution with 16 out channels, we used a batch-size of 256 and we employed SGD with a momentum of 0.9 . The training was done over 90 epochs. The learning rate was set to 10^{-1} and was decayed by a factor 10 every 30 epochs. A constant weight decay was set to 5\times 10^{-4} . Standard inputs preprocessing as proposed in Pytorch was also performed.
B.2 ImageNet
For our experiments on ImageNet (training from scratch), we used a batch-size of 256 and we employed SGD with a momentum of 0.9 . The training was done over 100 epochs. The initial learning rate was 0.1 and was decayed by a factor 10 every 30 epochs. A constant weight decay was set to 10^{-4} . Standard inputs preprocessing as proposed in Pytorch was performed: normalization, random croping of size 224\times 224 pixels, random horizontal flip.

For our finetuning experiment on ImageNet, we used a batch-size of 256 and we employed SGD with a momentum of 0.9 . The training was done over 3 epochs. The learning rate was kept constant to 5\times 10^{-4} . A constant weight decay was set to 10^{-4} . Standard inputs preprocessing as proposed in Pytorch was performed: normalization, random croping of size 224\times 224 pixels, random horizontal flip.
Figure 1: Trajectory of ResNets with 300 layers. Left: we learn x\to\frac{x^{2}}{2} , trajectories are smooth and do not intersect. Right: we learn x\to\frac{-x^{2}}{2} , trajectories are not smooth and intersect.; Trajectory of ResNets with 300 layers. Left: we learn x\to\frac{x^{2}}{2} , trajectories are smooth and do not intersect. Right: we learn x\to\frac{-x^{2}}{2} , trajectories are not smooth and intersect.
Figure 2: L_{2} norm \|\Psi_{N}(t,.)-\Psi(t,.)\|_{2} (w.r.t depth s ) for different training iterations t (horizontal axis) and different depth N (vertical axis). As predicted by Th. 1 , this distance goes to 0 as N\to+\infty .; L_{2} norm \|\Psi_{N}(t,.)-\Psi(t,.)\|_{2} (w.r.t depth s ) for different training iterations t (horizontal axis) and different depth N (vertical axis). As predicted by Th. 1 , this distance goes to 0 as N\to+\infty .
Figure 3: (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.; (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.
Figure 4: Comparison of the best test errors as a function of depth when using Euler or Heun’s discretization method with or without the adjoint method.; Comparison of the best test errors as a function of depth when using Euler or Heun’s discretization method with or without the adjoint method.
Test accuracy (ResNet-101)
Test accuracy (ResNet)