SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Now, let  $\tilde{H}$  be the solution of the initial value problem (5) with another parameter  $\tilde{\theta}$  and with the same initial condition  $\tilde{H}_0 = x$ . Then, for any  $t \geqslant 0$
$$
H _ {t} - \tilde {H} _ {t} = \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} \theta_ {i} (s) f _ {i} (H _ {s}) d s - \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} \tilde {\theta} _ {i} (s) f _ {i} (\tilde {H} _ {s}) d s.
$$
$$
\begin{array}{l} \| H _ {t} - \tilde {H} _ {t} \| = \left\| \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} \left(\theta_ {i} (s) - \tilde {\theta} _ {i} (s)\right) f _ {i} \left(H _ {s}\right) d s + \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} \tilde {\theta} _ {i} (s) \left(f _ {i} \left(H _ {s}\right) - f _ {i} \left(\tilde {H} _ {s}\right)\right) d s \right\| \\ \leqslant \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \theta_ {i} (s) - \tilde {\theta} _ {i} (s) | \| f _ {i} (H _ {s}) \| d s + \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \tilde {\theta} _ {i} (s) | \| f _ {i} (H _ {s}) - f _ {i} (\tilde {H} _ {s}) \| d s \\ \leqslant \int_ {0} ^ {t} \sum_ {i = 1} ^ {m} | \theta_ {i} (s) - \tilde {\theta} _ {i} (s) | \| f _ {i} (H _ {s}) \| d s + K _ {f} \int_ {0} ^ {t} \left(\| H _ {s} - \tilde {H} _ {s} \| \sum_ {i = 1} ^ {m} | \tilde {\theta} _ {i} (s) |\right) d s \\ \leqslant t C \| \theta - \tilde {\theta} \| _ {1, \infty} + K _ {f} R _ {\Theta} \int_ {0} ^ {t} \| H _ {s} - \tilde {H} _ {s} \| d s. \\ \end{array}
$$
Then Gronwall's inequality implies that, for  $t \in [0,1]$ ,
$$
\begin{array}{l} \left\| H _ {t} - \tilde {H} _ {t} \right\| \leqslant t C \| \theta - \tilde {\theta} \| _ {1, \infty} \exp (t K _ {f} R _ {\Theta}) \\ \leqslant M \left(K _ {f} R _ {\Theta} \exp \left(K _ {f} R _ {\Theta}\right) + 1\right) \exp \left(K _ {f} R _ {\Theta}\right) \| \theta - \tilde {\theta} \| _ {1, \infty} \\ \leqslant 2 M K _ {f} R _ {\Theta} \exp (2 K _ {f} R _ {\Theta}) \| \theta - \tilde {\theta} \| _ {1, \infty} \\ \end{array}
$$
since  $1 \leqslant K_f R_\Theta \exp(K_f R_\Theta)$  because  $K_f \geqslant 1$ ,  $R_\Theta \geqslant 1$ .
We first prove the result for  $m = 1$ . Let  $G_{x}$  be an  $\varepsilon /2K_{\Theta}$ -grid of  $[0,1]$  and  $G_{y}$  an  $\varepsilon /2$ -grid of  $[-R_{\Theta},R_{\Theta}]$ . Formally, we can take
$$
G _ {x} = \left\{\frac {k \varepsilon}{2 K _ {\Theta}}, 0 \leqslant k \leqslant \left\lceil \frac {2 K _ {\Theta}}{\varepsilon} \right\rceil \right\} \quad \text {a n d} \quad G _ {y} = \left\{- R _ {\Theta} + \frac {k \varepsilon}{2}, 1 \leqslant k \leqslant \left\lfloor \frac {4 R _ {\Theta}}{\varepsilon} \right\rfloor\right\}
$$
Our cover consists of all functions that start at a point of  $G_y$ , are piecewise linear with kinks in  $G_x$  where each piece has slope  $+K_{\Theta}$  or  $-K_{\Theta}$ . Hence our cover is of size
$$
\mathcal {N} _ {1} (\varepsilon) = \left| G _ {y} \right| 2 ^ {\left| G _ {x} \right|} \leqslant \frac {4 R _ {\Theta}}{\varepsilon} 2 ^ {\frac {2 K _ {\Theta}}{\varepsilon} + 2} = \frac {1 6 R _ {\Theta}}{\varepsilon} 4 ^ {\frac {K _ {\Theta}}{\varepsilon}}.
$$
Now take a function  $f:[0,1]\to \mathbb{R}$  that is uniformly bounded by  $R_{\Theta}$  and  $K_{\Theta}$ -Lipschitz. We construct a cover member at distance  $\varepsilon$  from  $f$  as follows. Choose a point  $y_0$  in  $G_y$  at distance at most  $\varepsilon /2$  from  $f(0)$ . Since  $f(0)\in [-R_{\Theta},R_{\Theta}]$ , this is clearly possible, except perhaps at the end of the interval. To verify that it is possible at the end of the interval, note that  $R_{\Theta}$  is at a distance less than  $\varepsilon /2$  of the last element of the grid, since
$$
R _ {\Theta} - \Big (- R _ {\Theta} + \left\lfloor \frac {4 R _ {\Theta}}{\varepsilon} \right\rfloor \frac {\varepsilon}{2} \Big) = 2 R _ {\Theta} - \left\lfloor \frac {4 R _ {\Theta}}{\varepsilon} \right\rfloor \frac {\varepsilon}{2} \in \Big [ 2 R _ {\Theta} - \frac {4 R _ {\Theta}}{\varepsilon} \frac {\varepsilon}{2}, 2 R _ {\Theta} - \Big (\frac {4 R _ {\Theta}}{\varepsilon} - 1 \Big) \frac {\varepsilon}{2} \Big ] = \Big [ 0, \frac {\varepsilon}{2} \Big ].
$$
Then, among the cover members that start at  $y_0$ , choose the one which is closest to  $f$  at each point of  $G_x$  (in case of equality, pick any one). Let us denote this cover member as  $\tilde{f}$ . Let us show recursively that  $f$  is at  $\ell_{\infty}$ -distance at most  $\varepsilon$  from  $\tilde{f}$ . More precisely, let us first show by induction on  $k$  that for all  $k \in \{0, \dots, \lceil \frac{2K_{\Theta}}{\varepsilon} \rceil \}$ ,
$$
\left| f \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) - \tilde {f} \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) \right| \leqslant \frac {\varepsilon}{2}. \tag {15}
$$
First,  $|f(0) - \tilde{f}(0)| \leqslant \frac{\varepsilon}{2}$ . Then, assume that (15) holds for some  $k$ . Then we have the following inequalities:
$$
\begin{array}{l} \tilde {f} \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) - \varepsilon \leqslant f \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) - \frac {\varepsilon}{2} \quad (\text {b y}) \\ \leqslant f \left(\frac {(k + 1) \varepsilon}{2 K _ {\Theta}}\right) \quad (f \text {i s} K _ {\Theta} \text {- L i p s c h i t z}) \\ \leqslant f \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) + \frac {\varepsilon}{2} \quad (f \text {i s} K _ {\Theta} - \text {L i p s c h i t z}) \\ \leqslant \tilde {f} \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) + \varepsilon \quad (\text {b y}) \\ \end{array}
$$
Moreover, by definition,  $\tilde{f}\left(\frac{(k + 1)\varepsilon}{K_{\Theta}}\right)$  is the closest point to  $f\left(\frac{(k + 1)\varepsilon}{K_{\Theta}}\right)$  among
$$
\left\{\tilde {f} \big (\frac {k \varepsilon}{K _ {\Theta}} \big) - \frac {\varepsilon}{2}, \tilde {f} \big (\frac {k \varepsilon}{K _ {\Theta}} \big) + \frac {\varepsilon}{2} \right\}.
$$
The bounds above show that, among those two points, at least one is at distance no more than  $\varepsilon / 2$  from  $f\left(\frac{(k + 1)\varepsilon}{K_{\Theta}}\right)$ . This shows (15) at rank  $k + 1$ .
To conclude, take now  $x \in [0,1]$ . There exists  $k \in \{0, \dots, \lceil \frac{2K_{\Theta}}{\varepsilon} \rceil \}$  such that  $x$  is at distance at most  $\varepsilon / 4K_{\Theta}$  from  $\frac{k\varepsilon}{2K_{\Theta}}$ . Again, this is clear except perhaps at the end of the interval, where it is also true since
$$
1 - \left\lceil \frac {2 K _ {\Theta}}{\varepsilon} \right\rceil \frac {\varepsilon}{2 K _ {\Theta}} \leqslant 1 - \frac {2 K _ {\Theta}}{\varepsilon} \frac {\varepsilon}{2 K _ {\Theta}} = 0,
$$
meaning that 1 is located between two elements of the grid  $G_{x}$ , showing that it is at distance at most  $\varepsilon /4K_{\Theta}$  from one element of the grid. Then, we have
$$
\begin{array}{l} | f (x) - \tilde {f} (x) | \leqslant \left| f (x) - f \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) \right| + \left| f \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) - \tilde {f} \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) \right| + \left| \tilde {f} \left(\frac {k \varepsilon}{2 K _ {\Theta}}\right) - \tilde {f} (x) \right| \\ \leqslant \frac {\varepsilon}{4} + \frac {\varepsilon}{2} + \frac {\varepsilon}{4}, \\ \end{array}
$$
where the first and third terms are upper-bounded because  $f$  and  $\tilde{f}$  are  $K_{\Theta}$ -Lip, while the second term is upper bounded by (15). Hence  $\| f - \tilde{f} \|_{\infty} \leqslant \varepsilon$ , proving the result for  $m = 1$ .
Finally, to prove the result for a general  $m$ , note that the Cartesian product of  $\varepsilon / m$ -covers for each coordinate of  $\theta$  gives an  $\varepsilon$ -cover for  $\theta$ . Indeed, consider such covers and take  $\theta \in \Theta$ . Since each coordinate of  $\theta$  is uniformly bounded by  $R_{\Theta}$  and  $K_{\Theta}$ -Lipschitz, the proof above shows the existence of a cover member  $\tilde{\theta}$  such that, for all  $i \in \{1, \ldots, m\}$ ,  $\| \theta_i - \tilde{\theta}_i \|_{\infty} \leqslant \varepsilon / m$ . Then
$$
\| \theta - \tilde {\theta} \| _ {1, \infty} = \sup  _ {0 \leqslant t \leqslant 1} \sum_ {i = 1} ^ {m} | \theta_ {i} (t) - \tilde {\theta} _ {i} (t) | \leqslant \sup  _ {0 \leqslant t \leqslant 1} \sum_ {i = 1} ^ {m} \| \theta_ {i} - \tilde {\theta} _ {i} \| _ {\infty} \leqslant \varepsilon .
$$
As a consequence, we conclude that
$$
\mathcal {N} (\varepsilon) \leqslant \left(\mathcal {N} _ {1} \big (\frac {\varepsilon}{m} \big)\right) ^ {m} = \Big (\frac {1 6 m R _ {\Theta}}{\varepsilon} \Big) ^ {m} 4 ^ {\frac {m ^ {2} K _ {\Theta}}{\varepsilon}}.
$$
Taking the logarithm yields the result.
A.4 Proof of Theorem 1
First note that, for any  $\theta \in \Theta$ ,  $x \in \mathcal{X}$  and  $y \in \mathcal{Y}$ ,
$$
\left| \ell \left(F _ {\theta} (x), y\right) \right| \leqslant \left| \ell \left(F _ {\theta} (x), y\right) - \ell (y, y) \right| + \left| \ell (y, y) \right| \leqslant K _ {\ell} \| F _ {\theta} (x) - y \|
$$
since, by assumption,  $\ell$  is  $K_{\ell}$ -Lipschitz with respect to its first variable and  $\ell(y, y) = 0$ . Thus
$$
\left| \ell \left(F _ {\theta} (x), y\right) \right| \leqslant K _ {\ell} \left(\left\| F _ {\theta} (x) \right\| + \| y \|\right) \leqslant K _ {\ell} \left(R _ {\mathcal {X}} + M R _ {\Theta} \exp \left(K _ {f} R _ {\Theta}\right) + R _ {\mathcal {Y}}\right) =: \overline {{M}}
$$
by Proposition 2.
Now, taking  $\delta > 0$ , a classical computation involving McDiarmid's inequality (see, e.g., Wainwright, 2019, proof of thm 4.10) yields that, with probability at least  $1 - \delta$ ,
$$
\mathcal {R} (\widehat {\theta} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + \mathbb {E} \left[ \sup  _ {\theta \in \Theta} | \mathcal {R} (\theta) - \widehat {\mathcal {R}} _ {n} (\theta) | \right] + \frac {\overline {{M}} \sqrt {2}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}.
$$
Denote  $C = 2MK_{f}R_{\Theta}\exp (2K_{f}R_{\Theta})$ . Then we show that  $\mathcal{R}$  and  $\widehat{\mathcal{R}}_n$  are  $CK_{\ell}$ -Lipschitz with respect to  $(\theta, \| \cdot \|_{1,\infty})$ : for  $\theta, \tilde{\theta} \in \Theta$ ,
$$
\begin{array}{l} \left| \mathcal {R} (\theta) - \mathcal {R} (\tilde {\theta}) \right| \leqslant \mathbb {E} \left[ \left| \ell \left(F _ {\theta} (x), y\right) - \ell \left(F _ {\tilde {\theta}} (x), y\right) \right| \right] \\ \leqslant K _ {\ell} \mathbb {E} \left[ \| F _ {\theta} (x) - F _ {\tilde {\theta}} (x) \| \right] \\ \leqslant C K _ {\ell} \| \theta - \tilde {\theta} \| _ {1, \infty}, \\ \end{array}
$$
according to Proposition 2. The proof for the empirical risk is very similar.
Let now  $\varepsilon > 0$  and  $\mathcal{N}(\varepsilon)$  be the covering number of  $\Theta$  endowed with the  $(1, \infty)$ -norm. By Proposition 3,
Take  $\theta^{(1)},\ldots ,\theta^{(\mathcal{N}(\varepsilon))}$  the associated cover elements. Then, for any  $\theta \in \Theta$  , denoting  $\theta^{(i)}$  the cover element at distance at most  $\varepsilon$  from  $\theta$
$$
\begin{array}{l} \left| \mathcal {R} (\theta) - \widehat {\mathcal {R}} _ {n} (\theta) \right| \leqslant \left| \mathcal {R} (\theta) - \mathcal {R} \left(\theta^ {(i)}\right) \right| + \left| \mathcal {R} \left(\theta^ {(i)}\right) - \widehat {\mathcal {R}} _ {n} \left(\theta^ {(i)}\right) \right| + \left| \widehat {\mathcal {R}} _ {n} \left(\theta^ {(i)}\right) - \widehat {\mathcal {R}} _ {n} (\theta) \right| \\ \leqslant 2 C K _ {\ell} \varepsilon + \sup  _ {i \in \{1, \dots , \mathcal {N} (\varepsilon) \}} | \mathcal {R} (\theta^ {(i)}) - \widehat {\mathcal {R}} _ {n} (\theta^ {(i)}) |. \\ \end{array}
$$
$$
\mathbb {E} \Big [ \sup  _ {\theta \in \Theta} | \mathcal {R} (\theta) - \widehat {\mathcal {R}} _ {n} (\theta) | \Big ] \leqslant 2 C K _ {\ell} \varepsilon + \mathbb {E} \Big [ \sup  _ {i \in \{1, \dots , \mathcal {N} (\varepsilon) \}} | \mathcal {R} (\theta^ {(i)}) - \widehat {\mathcal {R}} _ {n} (\theta^ {(i)}) | \Big ].
$$
Recall that a real-valued random variable  $X$  is said to be  $s^2$  sub-Gaussian (Bach, 2023, Section 1.2.1) if for all  $\lambda \in \mathbb{R}$ ,  $\mathbb{E}(\exp (\lambda (X - \mathbb{E}(X))))\leqslant \exp (\lambda^2 s^2 /2)$ . Since  $\widehat{\mathcal{R}}_n(\theta)$  is the average of  $n$  independent random variables, which are each almost surely bounded by  $\overline{M}$ , it is  $\overline{M}^2 /n$  sub-Gaussian, hence we have the classical inequality on the expectation of the maximum of sub-Gaussian random variables (see, e.g., Bach, 2023, Exercise 1.13)
$$
\mathbb {E} \Big [ \sup  _ {i \in \{1, \dots , \mathcal {N} (\varepsilon) \}} | \mathcal {R} (\theta^ {(i)}) - \widehat {\mathcal {R}} _ {n} (\theta^ {(i)}) | \Big ] \leqslant \overline {{M}} \sqrt {\frac {2 \log (2 \mathcal {N} (\varepsilon))}{n}}.
$$
The remainder of the proof consists in computations to put the result in the required format. More precisely, we have
$$
\begin{array}{l} \mathbb {E} \Big [ \sup _ {\theta \in \Theta} | \mathcal {R} (\theta) - \widehat {\mathcal {R}} _ {n} (\theta) | \Big ] \leqslant 2 C K _ {\ell} \varepsilon + \overline {{M}} \sqrt {\frac {2 \log (2 \mathcal {N} (\varepsilon))}{n}} \\ \leqslant 2 C K _ {\ell} \varepsilon + \bar {M} \sqrt {\frac {2 \log (2) + 2 m \log \left(\frac {1 6 m R _ {\Theta}}{\varepsilon}\right) + \frac {2 m ^ {2} K _ {\Theta}}{\varepsilon} \log (4)}{n}} \\ \leqslant 2 C K _ {\ell} \varepsilon + \bar {M} \sqrt {\frac {2 (m + 1) \log \left(\frac {1 6 m R _ {\Theta}}{\varepsilon}\right) + \frac {2 m ^ {2} K _ {\Theta}}{\varepsilon} \log (4)}{n}}. \\ \end{array}
$$
The third step is valid if  $\frac{16mR_{\Theta}}{\varepsilon} \geqslant 2$ . We will shortly take  $\varepsilon$  to be equal to  $\frac{1}{\sqrt{n}}$ , thus this condition holds true under the assumption from the Theorem that  $mR_{\Theta}\sqrt{n} \geqslant 3$ . Hence we obtain
$$
\mathcal {R} \left(\widehat {\theta} _ {n}\right) \leqslant \widehat {\mathcal {R}} _ {n} \left(\widehat {\theta} _ {n}\right) + 2 C K _ {\ell} \varepsilon + \bar {M} \sqrt {\frac {2 (m + 1) \log \left(\frac {1 6 m R _ {\Theta}}{\varepsilon}\right) + \frac {2 m ^ {2} K _ {\Theta}}{\varepsilon} \log (4)}{n}} + \frac {\bar {M} \sqrt {2}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}. \tag {16}
$$
Now denote  $\tilde{B} = 2\overline{M} K_{f}\exp (K_{f}R_{\Theta})$ . Then  $CK_{\ell}\leqslant \tilde{B}$  and  $2\overline{M}\leqslant \tilde{B}$ . Taking  $\varepsilon = \frac{1}{\sqrt{n}}$ , we obtain
$$
\begin{array}{l} \mathcal {R} (\widehat {\theta} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + \frac {2 \tilde {B}}{\sqrt {n}} + \frac {\tilde {B}}{2} \sqrt {\frac {2 (m + 1) \log (1 6 m R _ {\Theta} \sqrt {n})}{n} + \frac {2 m ^ {2} K _ {\Theta} \log (4)}{\sqrt {n}}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}} \\ \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + \frac {2 \tilde {B}}{\sqrt {n}} + \frac {\tilde {B}}{2} \sqrt {\frac {2 (m + 1) \log (1 6 m R _ {\Theta} \sqrt {n})}{n}} + \frac {\tilde {B}}{2} \frac {m \sqrt {2 K _ {\Theta} \log (4)}}{n ^ {1 / 4}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}} \\ \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + \frac {3 \tilde {B}}{2} \sqrt {\frac {2 (m + 1) \log (1 6 m R _ {\Theta} \sqrt {n})}{n}} + \tilde {B} \frac {m \sqrt {K _ {\Theta}}}{n ^ {1 / 4}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}, \\ \end{array}
$$
since  $2 \leqslant 2\sqrt{\log(2)} \leqslant \sqrt{2(m + 1)\log(16mR_{\Theta}\sqrt{n})}$  since  $16mR_{\Theta}\sqrt{n} \geqslant 2$  by the Theorem's assumptions, and  $\sqrt{2\log(4)} \leqslant 2$ . We finally obtain that
$$
\mathcal {R} (\widehat {\theta} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\theta} _ {n}) + 3 \tilde {B} \sqrt {\frac {(m + 1) \log (m R _ {\Theta} n)}{n}} + \tilde {B} \frac {m \sqrt {K _ {\Theta}}}{n ^ {1 / 4}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$
by noting that  $n\geqslant 9\max (m^{-2}R_{\Theta}^{-2},1)$  implies that
$$
\log (1 6 m R _ {\Theta} \sqrt {n}) \leqslant 2 \log (m R _ {\Theta} n).
$$
The result unfolds since the constant  $B$  in the Theorem is equal to  $3\tilde{B}$ .
A.5 Proof of Corollary 1
The corollary is an immediate consequence of Theorem 1. To obtain the result, note that  $m = d^2$ , thus in particular  $\sqrt{m + 1} = \sqrt{d^2 + 1} \leqslant d + 1$ , and besides  $\log (R_{\mathcal{W}}d^{2}n) \leqslant 2\log (R_{\mathcal{W}}dn)$  since  $R_{\mathcal{W}}n \leqslant R_{\mathcal{W}}^2 n^2$  by assumption on  $n$ .
A.6 Proof of Proposition 4
For  $x \in \mathcal{X}$ , let  $(H_k)_{0 \leqslant k \leqslant L}$  be the values of the layers defined by the recurrence (10) with the weights  $\mathbf{W}$  and the input  $H_0 = x$ . We denote by  $\| \cdot \|$  the  $\ell_2$ -norm for vectors and the spectral norm for matrices. Then, for  $k \in \{0, \dots, L - 1\}$ , we have
$$
\| H _ {k + 1} \| \leqslant \| H _ {k} \| + \frac {1}{L} \| W _ {k} \sigma (H _ {k}) \| \leqslant \| H _ {k} \| + \frac {1}{L} \| W _ {k} \| \| \sigma (H _ {k}) \| \leqslant \left(1 + \frac {K _ {\sigma} R _ {\mathcal {W}}}{L}\right) \| H _ {k} \|,
$$
where the last inequality uses that the spectral norm of a matrix is upper-bounded by its  $(1,1)$ -norm and that  $\sigma(0) = 0$ . As a consequence, for any  $k \in \{0, \dots, L\}$ ,
$$
\| H _ {k} \| \leqslant \left(1 + \frac {K _ {\sigma} R _ {\mathcal {W}}}{L}\right) ^ {k} \| H _ {0} \| \leqslant \exp (K _ {\sigma} R _ {\mathcal {W}}) R _ {\mathcal {X}} =: C,
$$
yielding the first claim of the Proposition.
Now, let  $\tilde{H}$  be the values of the layers (10) with another parameter  $\tilde{\mathbf{W}}$  and with the same input  $\tilde{H}_0 = x$ . Then, for any  $k \in \{0, \dots, L - 1\}$ ,
$$
H _ {k + 1} - \tilde {H} _ {k + 1} = H _ {k} - \tilde {H} _ {k} + \frac {1}{L} (W _ {k} \sigma (H _ {k}) - \tilde {W} _ {k} \sigma (\tilde {H} _ {k})).
$$
Hence, using again that the spectral norm of a matrix is upper-bounded by its  $(1,1)$ -norm and that  $\sigma(0) = 0$ ,
$$
\begin{array}{l} \| H _ {k + 1} - \tilde {H} _ {k + 1} \| \leqslant \| H _ {k} - \tilde {H} _ {k} \| + \frac {1}{L} \| W _ {k} (\sigma (H _ {k}) - \sigma (\tilde {H} _ {k})) \| + \frac {1}{L} \| (W _ {k} - \tilde {W} _ {k}) \sigma (\tilde {H} _ {k}) \| \\ \leqslant \left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) \| H _ {k} - \tilde {H} _ {k} \| + \frac {K _ {\sigma}}{L} \| W _ {k} - \tilde {W} _ {k} \| \| \tilde {H} _ {k} \| \\ \leqslant \left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) \| H _ {k} - \tilde {H} _ {k} \| + \frac {C K _ {\sigma}}{L} \| W _ {k} - \tilde {W} _ {k} \|. \\ \end{array}
$$
Then, dividing by  $\left(1 + K_{\sigma}\frac{R_{\mathcal{W}}}{L}\right)^{k + 1}$  and using the method of differences, we obtain that
$$
\begin{array}{l} \frac {\| H _ {k} - \tilde {H} _ {k} \|}{(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}) ^ {k}} \leqslant \| H _ {0} - \tilde {H} _ {0} \| + \frac {C K _ {\sigma}}{L} \sum_ {j = 0} ^ {k - 1} \frac {\| W _ {j} - \tilde {W} _ {j} \|}{(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}) ^ {j + 1}} \\ \leqslant \frac {C K _ {\sigma}}{L} \| \mathbf {W} - \tilde {\mathbf {W}} \| _ {1, 1, \infty} \sum_ {j = 0} ^ {k - 1} \frac {1}{(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}) ^ {j + 1}}. \\ \end{array}
$$
Finally note that
$$
\begin{array}{l} \sum_ {j = 0} ^ {k - 1} \frac {\left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) ^ {k}}{\left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) ^ {j + 1}} = \sum_ {j = 0} ^ {k - 1} \left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) ^ {j} \\ = \frac {L}{K _ {\sigma} R _ {\mathcal {W}}} \left(\left(1 + K _ {\sigma} \frac {R _ {\mathcal {W}}}{L}\right) ^ {k} - 1\right) \\ \leqslant \frac {L}{K _ {\sigma} R _ {\mathcal {W}}} (\exp (K _ {\sigma} R _ {\mathcal {W}}) - 1). \\ \end{array}
$$
$$
\| H _ {k} - \tilde {H} _ {k} \| \leqslant \frac {C}{R _ {\mathcal {W}}} (\exp (K _ {\sigma} R _ {\mathcal {W}}) - 1) \| \mathbf {W} - \tilde {\mathbf {W}} \| _ {1, 1, \infty} \leqslant \frac {R _ {\mathcal {X}}}{R _ {\mathcal {W}}} \exp (2 K _ {\sigma} R _ {\mathcal {W}}) \| \mathbf {W} - \tilde {\mathbf {W}} \| _ {1, 1, \infty}.
$$
A.7 Proof of Proposition 5
For two integers  $a$  and  $b$ , denote respectively  $a / / b$  and  $a\% b$  the quotient and the remainder of the Euclidean division of  $a$  by  $b$ . Then, for  $\mathbf{W} \in \mathbb{R}^{L \times d \times d}$ , let  $\phi(\mathbf{W}) : [0,1] \to \mathbb{R}^{d^2}$  the piecewise-affine function defined as follows:  $\phi(\mathbf{W})$  is affine on every interval  $\left[\frac{k}{L}, \frac{k+1}{L}\right]$  for  $k \in \{0, \dots, L-1\}$ ; for  $k \in \{1, \dots, L\}$  and  $i \in \{1, \dots, d^2\}$ ,
$$
\phi (\mathbf{W})_{i}\Bigl(\frac{k}{L}\Bigr) = \mathbf{W}_{\frac{k}{L},(i / / d) + 1,(i\% d) + 1}  ,
$$
and  $\phi (\mathbf{W})_i(0) = \phi (\mathbf{W})_i(1 / L)$ . Then  $\phi (\mathbf{W})$  satisfies two properties. First, it is a linear function of  $\mathbf{W}$ . Second, for  $\mathbf{W} \in \mathbb{R}^{L \times d \times d}$ ,
$$
\| \phi (\mathbf {W}) \| _ {1, \infty} = \| \mathbf {W} \| _ {1, 1, \infty},
$$
because, for  $x \in [0,1]$ ,  $\phi(\mathbf{W})(x)$  is a convex combination of two vectors that are bounded in  $\ell_1$ -norm by  $\| \mathbf{W} \|_{1,1,\infty}$ , so it is itself bounded in  $\ell_1$ -norm by  $\| \mathbf{W} \|_{1,1,\infty}$ , implying that  $\| \phi(\mathbf{W}) \|_{1,\infty} \leqslant \| \mathbf{W} \|_{1,1,\infty}$ . Reciprocally,
$$
\| \phi (\mathbf {W}) \| _ {1, \infty} = \sup  _ {0 \leqslant t \leqslant 1} \| \phi (\mathbf {W}) (x) \| _ {1} \geqslant \sup  _ {1 \leqslant k \leqslant L} \left\| \phi (\mathbf {W}) \left(\frac {k}{L}\right) \right\| _ {1} = \| \mathbf {W} \| _ {1, 1, \infty}.
$$
Now, take  $\mathbf{W} \in \mathcal{W}$ . The second property of  $\phi$  implies that  $\| \phi(\mathbf{W}) \|_{1,\infty} \leqslant R_{\mathcal{W}}$ . Moreover, each coordinate of  $\phi(\mathbf{W})$  is  $K_{\mathcal{W}}$ -Lipschitz, since the slope of each piece of  $\phi(\mathbf{W})_i$  is at most  $K_{\mathcal{W}}$ . As a consequence,  $\phi(\mathbf{W})$  belongs to
$$
\Theta_ {\mathcal {W}} = \left\{\theta : [ 0, 1 ] \rightarrow \mathbb {R} ^ {d ^ {2}}, \| \theta \| _ {1, \infty} \leqslant R _ {\mathcal {W}} \text {a n d} \theta_ {i} \text {i s} K _ {\mathcal {W}} \text {- L i p s c h i t z f o r} i \in \{1, \dots , d ^ {2} \} \right\}.
$$
Therefore  $\phi (\mathcal{W})$  is a subset of  $\Theta_{\mathcal{W}}$ , thus its covering number is less than the one of  $\Theta_{\mathcal{W}}$ . Moreover,  $\phi$  is clearly injective, thus we can define  $\phi^{-1}$  on its image. Consider an  $\varepsilon$ -cover  $(\theta_1,\dots ,\theta_N)$  of  $(\phi (\mathcal{W}),\| \cdot \|_{1,\infty})$ . Let us show that  $(\phi^{-1}(\theta_1),\ldots ,\phi^{-1}(\theta_N))$  is an  $\varepsilon$ -cover of  $(\mathcal{W},\| \cdot \|_{1,1,\infty})$ : take  $\mathbf{W}\in \mathcal{W}$  and consider  $\theta_{i}$  a cover member at distance less than  $\varepsilon$  from  $\phi (\mathbf{W})$ . Then
$$
\| \mathbf {W} - \phi^ {- 1} (\theta_ {i}) \| _ {1, 1, \infty} = \| \phi (\mathbf {W} - \phi^ {- 1} (\theta_ {i})) \| _ {1, \infty} = \| \phi (\mathbf {W}) - \theta_ {i} \| _ {1, \infty} \leqslant \varepsilon ,
$$
where the second equality holds by linearity of  $\phi$ . Therefore, the covering number of  $(\mathcal{W},\| \cdot \|_{1,1,\infty})$  is upper bounded by the one of  $(\phi (\mathcal{W}),\| \cdot \|_{1,\infty})$ , which itself is upper bounded by the one of  $(\Theta_{\mathcal{W}},\| \cdot \|_{1,\infty})$ , yielding the result by Proposition 3.
The proof structure is the same as the one of Theorem 1, but some constants change. Similarly to (16), we obtain that, if  $\frac{16d^2R_{\mathcal{W}}}{\varepsilon} \geqslant 2$  (which holds true for  $\varepsilon = 1/\sqrt{n}$  and under the assumption of the Theorem),
$$
\mathcal {R} (\widehat {\mathbf {W}} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\mathbf {W}} _ {n}) + 2 C K _ {\ell} \varepsilon + \overline {{M}} \sqrt {\frac {2 (d ^ {2} + 1) \log \left(\frac {1 6 d ^ {2} R _ {\mathcal {W}}}{\varepsilon}\right) + \frac {2 d ^ {4} K _ {\mathcal {W}}}{\varepsilon} \log (4)}{n}} + \frac {\overline {{M}} \sqrt {2}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$