SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we report the results on natural and robust accuracy for the broader range of perturbation budgets for Table 1  $\epsilon \in \{0.01,0.02,0.03,0.04,0.05,0.06\}$  for LeNet5 on MNIST, and  $\epsilon \in \{0.001,0.002,0.003,0.004,0.005,0.006\}$  for VGG16 on CIFAR10. See Table 3 and Table 4 (standard deviation is shown in gray). Additionally we provide results for Projected Gradient Descent attack with L2 norm and 10 steps for VGG16 on CIFAR10 with perturbation budget  $\epsilon \in \{0.1,0.13,0.16,0.2,0.23,0.26,0.3\}$ . See Table 5. The results confirm the same findings as the ones reported in the main paper. Table 6 provides results of comparison between baseline and Algorithm 1 for WideResnet model with depth 16 and width 4 on CIFAR100 for both FGSM and PGD attack with  $L\infty$  norm with perturbation budget  $\epsilon \in \{0.0003,0.0006,0.001,0.0013,0.0016\}$ .
Lemma 2. Let  $\phi : (Y, \| \cdot \|_Y) \to (Z, \| \cdot \|_Z)$  and  $\psi : (X, \| \cdot \|_X) \to (Y, \| \cdot \|_Y)$  be continuous mappings between finite-dimensional Banach spaces, and assume  $\operatorname{cond}(\psi)$  and  $\operatorname{cond}(\phi)$  are well defined. Then the following inequality holds:
$$
\operatorname {c o n d} (\phi \circ \psi) \leq \operatorname {c o n d} (\phi) \operatorname {c o n d} (\psi)
$$
Proof. To prove this lemma, we will pass through the definition of  $R(\phi \circ \psi, x, \delta)$ . Let us recall that it is defined as
$$
R (\phi \circ \psi , x, \delta) = \frac {\| \phi \circ \psi (x + \delta) - \phi \circ \psi (x) \| _ {Z} \| x \| _ {X}}{\| \delta \| _ {X} \| \phi \circ \psi (x) \| _ {Z}}
$$
Now, if  $\psi(x + \delta) - \psi(x) = 0$  for  $\delta \neq 0$ , then  $R(\phi \circ \psi, x, \delta) = 0$ . Since we're interested in the supremum, we can restrict to  $\delta$  such that  $\psi(x + \delta) - \psi(x) \neq 0$ . Moreover, by multiplying and
Table 3: LeNet5 MNIST, FGSM
Rel. perturbation ε	0.0	0.001	0.002	0.003	0.004	0.005	0.006	cr (%)
Baseline	0.9872	0.9836	0.9793	0.9735	0.9655	0.9543	0.9407	0
8.1e-04	5.8e-04	7.7e-04	6.4e-04	5.1e-04	1.0e-03	1.7e-03
Cayley SGD	0.9874	0.9843	0.9804	0.9753	0.9688	0.9602	0.9486	0
3.8e-04	5.2e-04	7.1e-04	4.9e-04	5.1e-04	7.4e-04	1.3e-03
Projected SGD	0.9878	0.9845	0.9807	0.9753	0.968	0.9594	0.9476	0
2.1e-04	5.2e-04	8.6e-04	7.8e-04	1.0e-03	8.1e-04	1.1e-03
CondLR	τ=0.0	0.9883	0.9857	0.9825	0.9783	0.9732	0.9661	0.958
3.2e-04	8.5e-04	4.9e-04	9.9e-04	9.8e-04	4.8e-04	6.0e-04
τ=0.1	0.9877	0.9854	0.9828	0.98	0.9763	0.972	0.9677
4.1e-04	5.5e-04	8.4e-04	1.2e-03	8.9e-04	4.9e-04	1.2e-03
τ=0.5	0.9867	0.9831	0.9802	0.9772	0.9724	0.9675	0.9598
4.4e-04	5.1e-04	6.4e-04	5.1e-04	6.4e-04	1.2e-03	1.2e-03
stief	0.986	0.9838	0.9809	0.9773	0.9721	0.9655	0.9586
2.6e-04	2.4e-04	2.4e-04	2.8e-04	3.6e-04	6.2e-04	1.4e-04
DLRT	0.967	0.9627	0.9573	0.9499	0.939	0.9253	0.9078	50
6.8e-04	6.9e-04	7.5e-04	1.0e-03	6.6e-04	1.1e-03	1.5e-03
vanilla	0.9875	0.983	0.9773	0.97	0.9603	0.9503	0.94	50
5.8e-04	8.2e-04	1.2e-03	3.e-03	4.1e-03	4.2e-03	3.9e-03
SVD prune	0.9883	0.9841	0.9793	0.9733	0.9639	0.9526	0.9414	50
9.6e-04	8.6e-04	1.1e-03	1.4e-03	1.2e-03	1.5e-03
CondLR	τ=0.0	0.9882	0.9846	0.9801	0.9743	0.9676	0.958	0.9452
7.5e-04	2.0e-03	3.4e-03	4.3e-03	6.3e-03	8.e-03	1.2e-02
τ=0.1	0.9877	0.984	0.9795	0.9736	0.966	0.9564	0.9444
5.1e-04	5.3e-04	6.1e-04	1.0e-03	8.1e-04	1.4e-03	1.8e-03
τ=0.5	0.9858	0.982	0.9768	0.97	0.9613	0.9487	0.9342
6.5e-04	8.e-04	8.2e-04	1.3e-03	1.9e-03	3.5e-03	4.0e-03
stief	0.9815	0.9779	0.9729	0.9659	0.9581	0.9499	0.9399
3.8e-04	4.6e-04	1.2e-04	3.7e-04	5.7e-04	4.5e-04	1.0e-03
DLRT	0.9649	0.9596	0.9517	0.9414	0.9281	0.9108	0.8865	80
5.8e-04	9.7e-04	1.7e-03	2.6e-03	4.6e-03	7.6e-03	1.1e-02
vanilla	0.9862	0.9802	0.972	0.9602	0.9464	0.9331	0.9194	80
6.9e-04	9.7e-04	1.9e-03	3.4e-03	3.1e-03	4.1e-03	5.2e-03
SVD prune	0.9864	0.981	0.9737	0.9634	0.9512	0.9392	0.9281	80
8.0e-04	1.1e-03	2.2e-03	3.e-03	4.5e-03	5.1e-03	5.9e-03
dividing by  $\| \psi (x)\| _Y$  we obtain:
$$
R (\phi \circ \psi , x, \delta) = \Big (\frac {\| \phi \circ \psi (x + \delta) - \phi \circ \psi (x) \| _ {Z} \| \psi (x) \| _ {Y}}{\| \psi (x + \delta) - \psi (x) \| _ {Y} \| \phi \circ \psi (x) \| _ {Z}} \Big) \Big (\frac {\| \psi (x + \delta) - \psi (x) \| _ {Y} \| x \| _ {X}}{\| \delta \| _ {X} \| \psi (x) \| _ {Y}} \Big)
$$
Now, if we define  $\operatorname{cond}(f,x,\varepsilon) = \sup_{\delta \neq 0: \| \delta \|_X \leq \varepsilon} R(f,x,\delta)$ , we take the supremum both sides on the set  $\{\delta \in X \mid \| \delta \|_X \leq \varepsilon\}$  and we can upper bound it with the product of the suprema of the two blocks
$$
\sup  _ {\delta \neq 0: \| \delta \| _ {X} \leq \varepsilon} R (\phi \circ \psi , x, \delta) = \operatorname {c o n d} (\psi , x, \varepsilon) \sup  _ {\delta \neq 0: \| \delta \| _ {X} \leq \varepsilon} \left(\frac {\| \phi \circ \psi (x + \delta) - \phi \circ \psi (x) \| _ {Z} \| \psi (x) \| _ {Y}}{\| \psi (x + \delta) - \psi (x) \| _ {Y} \| \phi \circ \psi (x) \| _ {Z}}\right) = \star
$$
Now, letting  $\eta = \psi(x + \delta) - \psi(\delta)$ , we can rewrite the second part of the last equation as a sort of restriction of  $\operatorname{cond}(\phi, \psi(x), \varepsilon)$  to the particular set of perturbation directions of the form  $\eta$ . Thus, we can lower-bound it as:
$$
\begin{array}{l} \star \leq \operatorname {c o n d} (\psi , x, \varepsilon) \sup  _ {\eta \neq 0: \| \eta \| _ {Y} \leq \varepsilon} \left(\frac {\| \phi (\psi (x) + \eta) - \phi (\psi (x)) \| _ {Z} \| \psi (x) \| _ {Y}}{\| \eta \| _ {Y} \| \phi (\psi (x)) \| _ {Z}}\right) = \\ = \operatorname {c o n d} (\psi , x, \varepsilon) \operatorname {c o n d} (\phi , \psi (x), \varepsilon) \\ \end{array}
$$
By hypothesis,  $\operatorname{cond}(\phi)$  and  $\operatorname{cond}(\psi)$  are finite, so we can take the limit  $\varepsilon \downarrow 0$  to get:
$$
\operatorname {c o n d} (\phi \circ \psi , x) \leq \operatorname {c o n d} (\phi , \psi (x)) \operatorname {c o n d} (\psi , x)
$$
Finally, taking the supremum over  $x$  on both members of the last equation, we can upper bound it with the original condition number without constraint on the directions:
$$
\operatorname {c o n d} (\phi \circ \psi) \leq \sup  _ {x} \operatorname {c o n d} (\phi , \psi (x)) \operatorname {c o n d} (\psi , x) \leq \operatorname {c o n d} (\phi) \operatorname {c o n d} (\psi)
$$
and thus conclude.
Table 4: VGG16 Cifar10, FGSM
Rel. perturbation ε	0.0	0.001	0.002	0.003	0.004	0.005	0.006	cr (%)
Baseline	0.9104	0.8397	0.752	0.6631	0.5822	0.5154	0.4592	0
1.3e-03	2.e-03	3.3e-03	2.7e-03	3.9e-03	6.6e-03	8.9e-03
Cayley SGD	0.8962	0.8262	0.7446	0.6624	0.5816	0.5132	0.4529	0
1.7e-03	3.9e-03	3.5e-03	4.1e-03	4.4e-03	4.2e-03	4.3e-03
Projected SGD	0.897	0.8271	0.7455	0.661	0.5832	0.5159	0.4574	0
1.8e-03	3.1e-03	3.0e-03	4.1e-03	5.3e-03	4.2e-03	4.5e-03
CondLR	τ=0.0	0.9099	0.8366	0.7456	0.6564	0.5711	0.4955	0.4292
7.4e-03	9.2e-03	9.8e-03	1.4e-02	1.5e-02	1.8e-02	1.6e-02
τ=0.1	0.9093	0.8291	0.7411	0.6678	0.5985	0.5386	0.4878	50
1.9e-03	3.1e-03	6.e-03	1.2e-02	1.2e-02	8.4e-03	5.6e-03
τ=0.5	0.8997	0.8136	0.7225	0.6577	0.6019	0.5485	0.5017	50
1.0e-03	1.9e-03	5.3e-03	7.5e-03	6.8e-03	7.7e-03	9.1e-03
stief	0.9138	0.8331	0.7322	0.6337	0.546	0.4702	0.4054	50
1.3e-03	3.3e-03	5.4e-03	3.9e-03	3.7e-03	4.1e-03	3.9e-03
DLRT	0.8425	0.7179	0.599	0.5065	0.441	0.3979	0.3691	50
1.6e-03	6.2e-03	8.8e-03	8.3e-03	1.2e-02	1.6e-02	1.8e-02
vanilla	0.8997	0.7962	0.6771	0.5708	0.4886	0.4276	0.3849	50
3.2e-03	4.4e-03	6.1e-03	8.3e-03	9.9e-03	1.0e-02	1.1e-02
SVD prune	0.8992	0.7936	0.673	0.5643	0.4777	0.4161	0.3698	50
3.1e-03	1.7e-03	2.5e-03	7.6e-03	1.1e-02	1.3e-02	1.3e-02
CondLR	τ=0.0	0.9066	0.825	0.7263	0.6289	0.541	0.4633	0.4
2.2e-03	1.8e-03	2.8e-03	3.2e-03	5.2e-03	5.3e-03	5.6e-03
τ=0.1	0.9048	0.8152	0.7123	0.6112	0.5262	0.4574	0.4013	80
1.6e-03	3.8e-03	5.4e-03	6.3e-03	9.1e-03	9.9e-03	8.6e-03
τ=0.5	0.8933	0.7952	0.6823	0.5748	0.4854	0.4177	0.3666	80
1.3e-03	1.4e-03	4.2e-03	6.e-03	5.3e-03	5.3e-03	5.3e-03
stief	0.9067	0.8202	0.7184	0.6185	0.5289	0.4515	0.3861	80
4.4e-04	2.5e-03	5.9e-03	5.5e-03	7.6e-03	8.9e-03	9.3e-03
DLRT	0.8092	0.6916	0.5839	0.4936	0.4178	0.3577	0.3086	80
2.e-02	1.9e-02	2.2e-02	2.5e-02	2.4e-02	2.3e-02	2.2e-02
vanilla	0.881	0.7687	0.6424	0.523	0.4266	0.3533	0.299	80
2.4e-03	5.e-03	7.4e-03	8.5e-03	1.1e-02	1.2e-02	1.2e-02
SVD prune	0.8799	0.7634	0.6357	0.5168	0.4206	0.3474	0.2927	80
3.7e-03	8.2e-03	1.2e-02	1.4e-02	1.2e-02	1.1e-02	1.1e-02
Now, the proof of Proposition 1 follows directly by applying the Lemma above recursively.
Proof of Proposition1. By defining  $T_{i}(z) = W_{i}z$ , we can write the neural network  $f$  as
$$
f (x) = \left(\sigma_ {L} \circ T _ {L} \circ \sigma_ {L - 1} \circ \dots \circ T _ {1}\right) (x)
$$
for nonlinear activation functions  $\sigma_{i}$ . Thus, for  $L = 1$ , the thesis follows directly from Lemma2, as long as  $\operatorname{cond}(\sigma_1) \leq C < \infty$ . For  $L > 1$ , one can unwrap the compositional structure of  $f$  from the left, defining  $\phi(z) = (\sigma_L \circ T_L)(z)$  and  $\psi(x) = (\sigma_{L-1} \circ \dots \circ T_1)(x)$ . Then by using Lemma2 we have that
$$
\operatorname {c o n d} (f) \leq \operatorname {c o n d} (\phi) \operatorname {c o n d} (\psi) \leq \operatorname {c o n d} \left(\sigma_ {L}\right) \operatorname {c o n d} \left(T _ {L}\right) \operatorname {c o n d} (\psi).
$$
Now, since  $\psi$  is a network of depth  $L - 1$ , we can proceed inductively to obtain that  $\operatorname{cond}(f) \leq C \prod_{i} \operatorname{cond}(T_i)$ , with  $C = \Pi_{i=1}^{L} \operatorname{cond}(\sigma_i)$ , and we conclude.
Note that the result above is of interest only if  $\operatorname{cond}(\sigma) = \sup_{x \in \mathcal{X}} \operatorname{cond}(\sigma; x) < \infty$ . When  $\sigma$  is Lipschitz, using the formula
$$
\operatorname {c o n d} (f; x) = \sup  _ {\nu_ {x} \in \partial \sigma (x)} \| \nu_ {x} \| \| x \| \| \sigma (x) \| ^ {- 1},
$$
Table 5: VGG16 Cifar10, PGD
Rel. perturbation ε	0.0	0.1	0.13	0.16	0.2	0.23	0.26	0.3	cr (%)
Baseline	0.9104	0.6704	0.5799	0.5021	0.4222	0.3788	0.347	0.3188	0
1.3e-03	5.4e-03	8.1e-03	9.0e-03	1.4e-02	1.4e-02	1.4e-02	1.4e-02	0
Cayley SGD	0.8962	0.6768	0.5938	0.5109	0.4087	0.3446	0.2921	0.2362	0
1.7e-03	4.4e-03	5.3e-03	7.1e-03	6.9e-03	8.4e-03	8.9e-03	1.0e-02	0
Projected SGD	0.897	0.6764	0.5925	0.5106	0.4086	0.3451	0.2919	0.2364	0
1.8e-03	5.0e-03	6.4e-03	5.6e-03	6.8e-03	9e-03	7.5e-03	7.6e-03	0
CondLR	τ=0.0	0.9099	0.6608	0.562	0.4712	0.3665	0.305	0.2551	0.2078
7.4e-03	1.8e-02	2.4e-02	2.8e-02	2.7e-02	2.2e-02	1.8e-02	1.5e-02	50
0.9093	0.6703	0.6208	0.5915	0.5692	0.5596	0.5528	0.5458	50
1.9e-03	1.6e-02	3.4e-02	5.1e-02	6.5e-02	7.1e-02	7.6e-02	7.9e-02	50
0.8997	0.6484	0.6025	0.5775	0.5603	0.5521	0.5475	0.5425	50
1.0e-03	9.8e-03	1.9e-02	2.6e-02	3.1e-02	3.2e-02	3.3e-02	3.5e-02	50
stief	0.6318	0.5253	0.4266	0.3118	0.2445	0.192	0.1388	50
1.3e-03	6.1e-03	7.0e-03	6.9e-03	7.8e-03	6.4e-03	6.9e-03	7.2e-03	50
vanilla	0.8997	0.5574	0.4511	0.3686	0.2962	0.2628	0.2402	0.2184	50
3.2e-03	1.2e-02	1.5e-02	1.5e-02	1.8e-02	2.0e-02	2.1e-02	2.2e-02	50
SVD prune	0.8992	0.5487	0.4385	0.3523	0.2795	0.2438	0.2206	0.1994	50
3.1e-03	5.4e-03	8.9e-03	1.5e-02	2.0e-02	2.5e-02	2.9e-02	3.2e-02	50
CondLR	τ=0.0	0.9066	0.6324	0.5323	0.4357	0.325	0.2599	0.2096	0.1603
2.2e-03	4.8e-03	6.2e-03	5.3e-03	8.9e-03	1.2e-02	1.5e-02	1.5e-02	80
0.9048	0.61	0.5084	0.4219	0.337	0.2944	0.2655	0.2397	80
1.6e-03	8.7e-03	9.7e-03	1.1e-02	1.1e-02	1.1e-02	1.1e-02	1.1e-02	80
0.8933	0.5745	0.4635	0.372	0.283	0.2382	0.2065	0.1784	80
1.3e-03	9.1e-03	9.7e-03	1.0e-02	1.3e-02	1.3e-02	1.6e-02	1.7e-02	80
stief	0.6153	0.5094	0.4082	0.2924	0.2236	0.1716	0.118	80
4.4e-04	6.7e-03	9.5e-03	1.e-02	9.4e-03	9.1e-03	8.7e-03	7.5e-03	80
vanilla	0.881	0.514	0.397	0.3012	0.2097	0.1629	0.1315	0.1037	80
2.4e-03	1.3e-02	1.5e-02	1.4e-02	1.3e-02	1.4e-02	1.4e-02	1.4e-02	80
SVD prune	0.8799	0.5064	0.3906	0.2957	0.2016	0.1549	0.1222	0.0957	80
3.7e-03	2.e-02	2.1e-02	1.8e-02	1.5e-02	1.2e-02	1.1e-02	1.2e-02	80
Table 6: WRN16-4 Cifar100
Rel. perturbation ε	0.0003	0.0006	0.001	0.0013	0.0016	cr (%)
Baseline, FGSM	0.5845	0.5267	0.4606	0.4187	0.3841	0
τ = 0.1	0.6492	0.5806	0.5009	0.4499	0.4011	50
τ = 0.5	0.6062	0.5442	0.4693	0.4199	0.3774	50
Baseline, PGD10	0.5925	0.5401	0.4713	0.4289	0.3878	0
τ = 0.1	0.6552	0.5918	0.5127	0.4531	0.3972]	50
τ = 0.5	0.6160	0.5547	0.4823	0.4290	0.3810	50
with  $\partial$  being the Clarke's generalized gradient operator [14], we observe below that this is the case for a broad list of activation functions  $\sigma$  and feature spaces  $\mathcal{X}$ .
- LeakyReLU. For  $x \in \mathbb{R}, \alpha > 0$ , let  $\sigma(x) = \max \{0, x\} + \alpha \min \{0, x\}$ . Then any  $\nu_x \in \partial \sigma(x)$  is such that  $\nu_x = 1$  if  $x > 0$ ;  $\nu_x = \alpha$  if  $x < 0$ ;  $\nu_x = [\min(\alpha, 1), \max(\alpha, 1)]$  otherwise. Thus
$$
\operatorname {c o n d} (\sigma) = \sup  _ {x \neq 0} \operatorname {c o n d} (\sigma ; x) = \sup  _ {x \neq 0} \sup  _ {\beta \in \partial \sigma (x)} \frac {| x | | \mathbf {1} _ {x > 0} + \alpha \mathbf {1} _ {x <   0} + \beta \mathbf {1} _ {x = 0} |}{| \max  \{0 , x \} + \alpha \min  \{0 , x \} |} = \max  (\alpha , 1)
$$
- Tanh. For  $x \in \mathbb{R}$ , let  $\sigma(x) = \tanh(x)$ . Then  $\sigma'(x) = \frac{1}{\cosh^2(x)}$  and thus
$$
\operatorname {c o n d} (\sigma) = \sup  _ {x} \frac {| x |}{| \tanh  (x) | | \cosh^ {2} (x) |} = \sup  _ {x} \frac {| 4 x |}{| e ^ {x} - e ^ {- x} | | e ^ {x} + e ^ {- x} |} = 1
$$
Since the maximum of  $\operatorname{cond}(\sigma, x)$  is reached at zero, where the function can be extended by continuity.
- Hardtanh. For  $x \in [-a, a]$  and  $a > 0$ , let  $\sigma(x) = a\mathbf{1}_{x > a} - a\mathbf{1}_{x < -a} + x\mathbf{1}_{x \in [-a, a]}$ . Then, we have that  $\partial \sigma(x)$  coincides with the derivative values in all points but  $x = \pm a$ . In those two points, we have  $\partial \sigma(\pm a) = [0, 1]$ . Thus, for any  $\nu_x \in \partial \sigma(x)$ , we have
$$
\operatorname {c o n d} (\sigma) = \sup  _ {x \in [ - a, a ]} \frac {| \nu_ {x} | | x |}{| \sigma (x) |} \leq \sup  _ {x \in [ - a, a ]} \frac {| x |}{| \sigma (x) |} = a
$$
- Logistic sigmoid. For  $x \in \mathbb{R}$  let  $\sigma(x) = (1 + e^{-x})^{-1}$ . Then  $\sigma'(x) = \sigma(x)(1 - \sigma(x))$  and thus
$$
\operatorname {c o n d} (\sigma ; x) = | x | (1 - \sigma (x)) = | x | e ^ {- x} (1 + e ^ {- x}) ^ {- 1}.
$$
Therefore, when  $x \geq 0$ , we have  $|x|e^{-x} \leq 1 / e$  and  $(1 + e^{-x}) \geq 1$ , thus  $\operatorname{cond}(\sigma; x) \leq 1 / e$ .
- Softplus. For  $x \in \mathbb{R}$ , let  $\sigma(x) = \ln(1 + e^x)$ . Then  $\sigma'(x) = S(x) = (1 + e^{-x})^{-1}$  and  $\operatorname{cond}(\sigma; x) = |x|S(x)\sigma(x)^{-1}$ . Thus, for  $x \geq 0$ , we have  $\operatorname{cond}(\sigma; x) \leq 1$ .
- SiLU. For  $x \in \mathbb{R}$  let  $\sigma(x) = x(1 + e^{-x})^{-1} = xS(x)$ . Then,  $\sigma'(x) = S(x) + xS(x)(1 - S(x))$  and thus for any  $x \geq 0$  we have
$$
\operatorname {c o n d} (\sigma ; x) = | 1 + x (1 - S (x)) | \leq 1 + \frac {1}{e}
$$
In the following, the proof of the main approximation result is presented. We underline that the core part of the proof relies on [31, Theorem 5.2]. For completeness, we repropose here main elements of the argument. We refer the interested reader to [31] and references therein for further details.
Proof. Let  $Y(t)$  be the solution of (5) at time  $t \in [0, \lambda]$ . First, we observe that the projected subflows of  $W(t) = \widetilde{W}(t) + E(t)$  and  $\widetilde{W}(t)$  satisfy the differential equations
$$
\left\{ \begin{array}{l} \dot {Y} = P (Y) \dot {\widetilde {W}} + P (Y) \dot {E}, \\ \widetilde {W} = P \left(\widetilde {W}\right) \widetilde {W}. \end{array} \right.
$$
where  $P(\cdot)$  denotes the orthogonal projection into the tangent space of the low-rank manifold  $\mathcal{M}_r$ . Next, we observe that the following identities hold
$$
(P (Y) - P (\widetilde {W})) \dot {\widetilde {W}} = - (P ^ {\perp} (Y) - P ^ {\perp} (\widetilde {W})) \dot {\widetilde {W}} = - P ^ {\perp} (Y) \dot {\widetilde {W}} = - P ^ {\perp} (Y) ^ {2} \dot {\widetilde {W}}.
$$
where  $P^{\perp}(\cdot) = I - P(\cdot)$  represents the complementary orthogonal projection. The latter implies that
$$
\langle Y - \widetilde {W}, (P (Y) - P (\widetilde {W})) \dot {\widetilde {W}} \rangle = \langle P ^ {\perp} (Y) (Y - \widetilde {W}), (P (Y) - P (\widetilde {W})) \dot {\widetilde {W}} \rangle .
$$
Let  $\gamma = 32\mu (s - \varepsilon)^{-2}$ . It follows from [31, Lemma 4.2] that
$$
\begin{array}{l} \langle Y - \widetilde {W}, \dot {Y} - \dot {\widetilde {W}} \rangle = \langle P ^ {\perp} (Y) (Y - \widetilde {W}), (P (Y) - P (\widetilde {W})) \dot {\widetilde {W}} \rangle + \langle Y - \widetilde {W}, P (Y) \dot {E} \rangle \\ \leq \gamma \| Y - \widetilde {W} \| ^ {3} + \eta \| Y - \widetilde {W} \|. \\ \end{array}
$$
Further, we remind that
$$
\langle Y - \widetilde {W}, \dot {Y} - \dot {\widetilde {W}} \rangle = \frac {1}{2} \frac {d}{d t} \| Y - \widetilde {W} \| ^ {2} = \| Y - \widetilde {W} \| \frac {d}{d t} \| Y - \widetilde {W} \|.
$$
Hence, the error  $e(t) = \| Y(t) - \widetilde{W}(t) \|$  satisfies the differential inequality
$$
\dot {e} \leq \gamma e ^ {2} + \eta , \quad e (0) = 0.
$$
The error  $e(t)$  for  $t \in [0, \lambda]$  admits an upper bound given by the solution of
$$
\dot {z} = \gamma z ^ {2} + \eta , \quad z (0) = 0.
$$
The last differential initial-value problem admits a closed solution given by
$$
z (t) = \sqrt {\eta / \gamma} \tan \left(t \sqrt {\eta \gamma}\right),
$$
where the last term is bounded by  $2t\eta$  for  $t\sqrt{\gamma\eta} \leq 1$ . The proof thus concludes as follows.
$$
\| Y (t) - W (t) \| \leq \| Y (t) - \widetilde {W} (t) \| + \| E (t) \| \leq 2 t \eta + t \eta = 3 t \eta ,
$$
where the last estimate arises by the integral identity  $E(t) = \int_0^t\dot{E} (s)ds$ .