SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\bar {M} = K _ {\ell} \left(R _ {\mathcal {X}} \exp \left(K _ {\sigma} R _ {\mathcal {W}}\right) + R _ {\mathcal {Y}}\right)
$$
$$
C = \frac {R _ {\mathcal {X}}}{R _ {\mathcal {W}}} \exp (2 K _ {\sigma} R _ {\mathcal {W}}).
$$
Finally denote
$$
\tilde {B} = 2 \overline {{M}} \max  \left(\frac {\exp (K _ {\sigma} R _ {\mathcal {W}})}{R _ {\mathcal {W}}}, 1\right).
$$
Then  $CK_{\ell} \leqslant \tilde{B}$  and  $2\overline{M} \leqslant \tilde{B}$ . Taking  $\varepsilon = \frac{1}{\sqrt{n}}$ , we obtain as in the proof of Theorem 1 that
$$
\mathcal {R} (\widehat {\mathbf {W}} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\mathbf {W}} _ {n}) + 3 \tilde {B} \sqrt {\frac {(d ^ {2} + 1) \log (d ^ {2} R _ {\mathcal {W}} n)}{n}} + \tilde {B} \frac {d ^ {2} \sqrt {K _ {\mathcal {W}}}}{n ^ {1 / 4}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}}.
$$
for  $n\geqslant 9R_{\mathcal{W}}^{-1}\max (d^{-4}R_{\mathcal{W}}^{-1},1)$  . Thus
$$
\mathcal {R} (\widehat {\mathbf {W}} _ {n}) \leqslant \widehat {\mathcal {R}} _ {n} (\widehat {\mathbf {W}} _ {n}) + 3 \sqrt {2} \tilde {B} (d + 1) \sqrt {\frac {\log (d R _ {\mathcal {W}} n)}{n}} + \tilde {B} \frac {d ^ {2} \sqrt {K _ {\mathcal {W}}}}{n ^ {1 / 4}} + \frac {\tilde {B}}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$
since  $\sqrt{d^2 + 1} \leqslant d + 1$  and  $R_{\mathcal{W}}n \leqslant R_{\mathcal{W}}^2 n^2$  by assumption on  $n$ . The result unfolds since the constant  $B$  in the Theorem is equal to  $3\sqrt{2}\tilde{B}$ .
A.9 Proof of Corollary 2
$$
A (\mathbf {W}) = \left(\prod_ {k = 1} ^ {L} \left\| I + \frac {1}{L} W _ {k} \right\|\right) \left(\sum_ {k = 1} ^ {L} \frac {\| W _ {k} ^ {T} \| _ {2 , 1} ^ {2 / 3}}{L ^ {2 / 3} \| I + \frac {1}{L} W _ {k} \| ^ {2 / 3}}\right) ^ {3 / 2},
$$
where  $\| \cdot \|_{2,1}$  denotes the  $(2,1)$ -norm defined as the  $\ell_1$ -norm of the  $\ell_2$ -norms of the columns, and  $I$  is the identity matrix (and we recall that  $\| \cdot \|$  denotes the spectral norm). We apply Theorem 1.1 from Bartlett et al. (2017) by taking as reference matrices the identity matrix. The theorem shows that, under the assumptions of the corollary,
$$
\mathbb {P} \Big (\underset {1 \leqslant j \leqslant d} {\arg \max } F _ {\mathbf {W}} (x) _ {j} \neq y \Big) \leqslant \widehat {\mathcal {R}} _ {n} (\mathbf {W}) + C \frac {R _ {\mathcal {X}} A (\mathbf {W}) \log (d)}{\gamma \sqrt {n}} + \frac {C}{\sqrt {n}} \sqrt {\log \frac {1}{\delta}},
$$
where, as in the corollary,  $\widehat{\mathcal{R}}_n(\mathbf{W})\leqslant n^{-1}\sum_{i = 1}^{n}\mathbf{1}_{F\mathbf{W}(x_i)y_i\leqslant \gamma +\max_{j\neq y_i}f(x_i)_j}$  and  $C$  is a universal constant. Let us upper bound  $A(\mathbf{W})$  to conclude. On the one hand, we have
$$
\begin{array}{l} \prod_ {k = 1} ^ {L} \left\| I + \frac {1}{L} W _ {k} \right\| \leqslant \prod_ {k = 1} ^ {L} \left(\| I \| + \frac {1}{L} \| W _ {k} \|\right) \\ \leqslant \prod_ {k = 1} ^ {L} \left(1 + \frac {1}{L} \| W _ {k} \| _ {1, 1}\right) \\ \leqslant \prod_ {k = 1} ^ {L} \left(1 + \frac {1}{L} R _ {\mathcal {W}}\right) \\ \leqslant \exp (R _ {\mathcal {W}}) \\ \end{array}
$$
On the other hand, for any  $k\in \{1,\ldots ,L\}$
$$
\left\| W _ {k} ^ {T} \right\| _ {2, 1} \leqslant \left\| W _ {k} ^ {T} \right\| _ {1, 1} \leqslant R _ {\mathcal {W}},
$$
$$
\left\| I + \frac {1}{L} W _ {k} \right\| \geqslant 1 - \frac {1}{L} \left\| W _ {k} \right\| \geqslant 1 - \frac {R _ {\mathcal {W}}}{L} \geqslant \frac {1}{2},
$$
under the assumption that  $L\geqslant R_{\mathcal{W}}$  . All in all, we obtain that
$$
A (\mathbf {W}) \leqslant \exp (R _ {\mathcal {W}}) \left(2 ^ {2 / 3} L ^ {1 / 3} R _ {\mathcal {W}} ^ {2 / 3}\right) ^ {3 / 2} = 2 R _ {\mathcal {W}} \exp (R _ {\mathcal {W}}) \sqrt {L},
$$
which yields the result.
Our code is available at
https://github.com/PierreMarion23/generalization-ode-resnets.
We use the following model, corresponding to model (10) with additional projections at the beginning and at the end:
$$
\begin{array}{l} H _ {0} = A x \\ H _ {k + 1} = H _ {k} + \frac {1}{L} W _ {k + 1} \sigma (H _ {k}), \quad 0 \leqslant k \leqslant L - 1 \\ F _ {\mathbf {W}} (x) = B H _ {L}, \\ \end{array}
$$
Table 1: Values of the model hyperparameters.
where  $x \in \mathbb{R}^{768}$  is a vectorized MNIST image,  $A \in \mathbb{R}^{d \times 768}$ , and  $B \in \mathbb{R}^{10 \times d}$ . Table 1 gives the value of the hyperparameters.
We use the initialization scheme outlined in Section 4.1: we initialize, for  $k \in \{1, \dots, L\}$  and  $i, j \in \{1, \dots, d\}$ ,
$$
\mathbf {W} _ {k, i, j} = \frac {1}{\sqrt {d}} f _ {i, j} \left(\frac {k}{L}\right),
$$
where  $f_{i,j}$  are independent Gaussian processes with the RBF kernel (with bandwidth equal to 0.1). We refer to Marion et al. (2022) and Sander et al. (2022) for further discussion on this initialization scheme. However,  $A$  and  $B$  are initialized with a more usual scheme, namely with i.i.d.  $\mathcal{N}(0,1 / c)$  random variables, where  $c$  denotes the number of columns of  $A$  (resp.  $B$ ).
In Figure 1a, we repeat training 10 times independently. Each time, we perform 30 epochs, and compute after each epoch both the Lipschitz constant of the weights and the generalization gap. This gives 300 pairs (Lipschitz constant, generalization gap), which each corresponds to one dot in the figure. Furthermore, we report results for two setups: when  $A$  and  $B$  are trained or when they are fixed random matrices.
In Figure 1b,  $A$  and  $B$  are not trained. The reason is to assess the effect of the penalization on  $\mathbf{W}$  for a fixed scale of  $A$  and  $B$ . If we allow  $A$  and  $B$  to vary, then it is possible that the effect of the penalization might be neutralized by a scale increase of  $A$  and  $B$  during training.
For all experiments, we use the standard MNIST datasplit (60k training samples and 10k testing samples). We train using the cross entropy loss, mini-batches of size 128, and the optimizer Adam (Kingma and Ba, 2015) with default parameters and a learning rate of 0.02.
We use PyTorch (Paszke et al., 2019) and PyTorch Lightning for our experiments.
The code takes about 60 hours to run on a standard laptop (no GPU).