SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left| \ell \left(\mathbf {W} _ {2}, \mathbf {z}\right) - \ell \left(\mathbf {W} _ {2} ^ {\prime}, \mathbf {z}\right) \right| \leq K _ {\text {t e m p}} \left(\left\| \left(W _ {2} - W _ {2} ^ {\prime}\right) \right\| _ {F} + \left\| b _ {2} - b _ {2} ^ {\prime} \right\| + \left\| \mathrm {F C N} _ {1} (\mathbf {x}) - \mathrm {F C N} _ {1} ^ {\prime} (\mathbf {x}) \right\|\right).
$$
Exploiting the recursive assumption that  $\mathrm{FCN}_1$  is Lipschitz with respect to its weights  $\mathbf{W}_1$  gives  $\| \mathrm{FCN}_1(\mathbf{x}) - \mathrm{FCN}_1'(\mathbf{x})\| \leq K_1\| \mathbf{W}_1 - \mathbf{W}_1'\|$ .
If we denote by  $(W_{2},b_{2})$  the vector of all concatenated weights, notice that  $\| (W_2 - W_2^{\prime})\| _F + \| b_2 - b_2^\prime \| = \sqrt{(\|(W_2 - W_2^\prime)\|_F + \|b_2 - b_2^\prime\|)^2}\leq \sqrt{2(\|(W_2 - W_2^\prime)\|_F^2 + \|b_2 - b_2^\prime\| ^2)} = \sqrt{2}\| (W_2,b_2) - (W_2^\prime ,b_2^\prime)\|$  (we used that for any real numbers  $a,b,(a + b)^2\leq 2(a^2 +b^2)$ ). We then have:
$$
\begin{array}{l} | \ell (\mathbf {W} _ {2}, \mathbf {z}) - \ell (\mathbf {W} _ {2} ^ {\prime}, \mathbf {z}) | \leq K _ {\text {t e m p}} \max  (\sqrt {2}, K _ {1}) \left(\| (W _ {2}, b _ {2}) - (W _ {2} ^ {\prime}, b _ {2} ^ {\prime}) \| + \| \mathbf {W} _ {1} - \mathbf {W} _ {1} ^ {\prime} \|\right) \\ \leq \sqrt {2} K _ {\text {t e m p}} \max  (\sqrt {2}, K _ {1}) | | \mathbf {W} _ {2} - \mathbf {W} _ {2} ^ {\prime} | |. \\ \end{array}
$$
The last line holds by reusing the same calculation trick. This concludes the proof for  $i = 2$ . Then for  $i = 1$  the same proof holds by replacing  $W_{2}, b_{2}$ ,  $\mathrm{FCN}_2$  by  $W_{1}, b_{1}$ ,  $\mathrm{FCN}_1$  and replacing  $\mathrm{FCN}_1(\mathbf{x})$ ,  $\mathrm{FCN}_1'(\mathbf{x})$  by  $\mathbf{x}$  (we then do not need to assume a recursive Lipschitz behaviour). Therefore the result holds for  $i = 1$ .
We then properly apply a recursive argument by assuming the result at rank  $i - 1$  reusing the same proof at any rank  $i$  by replacing  $W_{2}, b_{2}$ ,  $\mathrm{FCN}_2$  by  $W_{i}, b_{i}$ ,  $\mathrm{FCN}_i$  and  $\mathrm{FCN}_1(\mathbf{x})$ ,  $\mathrm{FCN}_1'(\mathbf{x})$  by  $\mathrm{FCN}_{i-1}(\mathbf{x})$ ,  $\mathrm{FCN}_{i-1}'(\mathbf{x})$ . This concludes the proof.
C.5 Experiments with varying number of priors
The experiments of Section 4 rely on data-dependent priors constructed through the procedure PRIORS LEARNING. We fixed a number of priors  $K$  equal to  $0.2\sqrt{m}$ . This number is an empirical tradeoff between the informativeness of our priors and time-efficient computation. However, there is no theoretical intuition for the value of this parameter (the discussion of Section 3.1 considered  $K = \sqrt{m}$  as a potential tradeoff; see Appendix A). Thus, we gather below the performance of our learning procedures for  $K = \alpha \sqrt{m}$ , where  $\alpha \in \{0, 0.4, 0.6, 0.8, 1\}$  (the case  $\alpha = 0$  being a convention to
denote  $K = 1$ ). The experiments are gathered below, and all remaining hyperparameters (except  $K$ ) are identical to those described in Section 4.
Analysis of our results. First, when considering neural networks, note that for any dataset except SEGMENTATION, LETTER, the performances of our methods are similar or better when considering data-dependent priors (i.e., when  $\alpha > 0$ ). A similar remark holds for the linear models for all datasets except for SATIMAGE, SEGMENTATION, and TICTACTOE. This illustrates the relevance of data-dependent priors. We also remark that there is no value of  $\alpha$ , which provides a better performance on all datasets. For instance, considering neural networks, note that  $\alpha = 1$  gives the better performance (i.e., the smallest  $\Re_{\mu}(h)$ ) for Algorithm 1  $(1/\sqrt{m})$  for the SATIMAGE dataset while, for the same algorithm, the better performance on the SEGMENTATION dataset is attained for  $\alpha = 0.8$ . Sometimes, the number  $K$  does not have a clear influence: on MNIST with NNs, for Algorithm 1  $(1/\sqrt{m})$ , our performances are similar, whatever the value of  $K$ , but still significantly better than ERM. In any case, note that for every dataset, there exists a value of  $K$  and such that our algorithm attains either similar or significantly better performances than ERM on every dataset, which shows the relevance of our learning algorithm to ensure a good generalisation ability. Moreover, there is no obvious choice for the parameters  $\varepsilon$ . For instance, in Tables 2 and 3, for the SEGMENTATION dataset, the parameters  $K = 1$ ,  $\varepsilon = \frac{1}{m}$  are optimal (in terms of test risks) for both models. As  $K = 1$  means that our single prior is data-free, this shows that the intrinsic structure of SEGMENTATION makes it less sensitive to both the information contained in the prior ( $K = 1$  meaning data-free prior) and the place of the prior itself ( $\varepsilon = 1/m$  meaning that we give less weight to the regularisation within our optimisation procedure). On the contrary, in Table Table 1, the YEAST dataset performs significantly better when  $\varepsilon = 1/\sqrt{m} (K = 0.2\sqrt{m})$ , exhibiting a positive impact of our data-dependent priors.
C.6 Experiments on classical regularisation methods
We perform additional experiments to see the performance of the weight decay, i.e., the L2 regularisation on the weights; the results are presented in Table 4. Moreover, notice that the 'distance to initialisation'  $\| \mathbf{w} - \mathbf{w}_0\|$  (where  $\mathbf{w}_0$  is the weights initialized randomly) is a particular case of Algorithm 1 when  $K = 1$  (i.e., we treat the data as a single batch, and the prior is the data-free initialisation); the results are in Tables 2 and 3.
Analysis of our results. This experiment on the weight decay demonstrates that on a few datasets (namely SENSORLESS and YEAST), when our predictors are neural nets, the weight decay regularisation fails to learn while ours succeeds, as shown in Table 1. In general, this table shows that, on most of the datasets, considering data-dependent priors leads to sharper results. This shows the efficiency of our method compared to the 'distance to initialisation' regularisation.
Table 2: Performance of Algorithm 1 compared to ERM on different datasets for neural network models. We consider  $\varepsilon = 1 / m$  and  $\varepsilon = 1 / \sqrt{m}$ , with  $K = \alpha \sqrt{m}$  and  $\alpha \in \{0,0.4,0.6,0.8,1\}$ . We plot the empirical risk  $\Re_S(h)$  with its associated test risk  $\Re_\mu(h)$ .
(a)  $K = 1$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.207	0.207	0.248	0.248
FASHIONMNIST	0.160	0.164	0.158	0.164
LETTER	0.258	0.269	0.268	0.280
MNIST	0.116	0.123	0.085	0.096
MUSHROOMS	0.000	0.000	0.000	0.001
NURSERY	0.705	0.720	0.720	0.736
PENDIGITS	0.704	0.724	0.021	0.037
PHISHING	0.048	0.052	0.038	0.055
SATIMAGE	0.148	0.208	0.147	0.207
SEGMENTATION	0.141	0.176	0.248	0.385
SENSORLESS	0.907	0.911	0.907	0.911
TICTACTOE	0.000	0.042	0.000	0.033
YEAST	0.695	0.712	0.677	0.658
(b)  $K = 0.4\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.167	0.166	0.164	0.164
FASHIONMNIST	0.160	0.164	0.156	0.160
LETTER	0.263	0.275	0.252	0.263
MNIST	0.112	0.120	0.085	0.096
MUSHROOMS	0.000	0.000	0.000	0.000
NURSERY	0.705	0.720	0.706	0.719
PENDIGITS	0.011	0.025	0.010	0.022
PHISHING	0.043	0.053	0.041	0.052
SATIMAGE	0.147	0.178	0.145	0.174
SEGMENTATION	0.345	0.408	0.225	0.416
SENSORLESS	0.075	0.078	0.074	0.077
TICTACTOE	0.000	0.031	0.000	0.019
YEAST	0.450	0.480	0.695	0.712
(c)  $K = 0.6\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.165	0.163	0.165	0.164
FASHIONMNIST	0.158	0.164	0.156	0.160
LETTER	0.259	0.275	0.260	0.267
MNIST	0.112	0.121	0.084	0.094
MUSHROOMS	0.000	0.000	0.000	0.000
NURSERY	0.706	0.719	0.706	0.719
PENDIGITS	0.008	0.023	0.009	0.022
PHISHING	0.043	0.055	0.040	0.050
SATIMAGE	0.138	0.184	0.141	0.174
SEGMENTATION	0.577	0.845	0.145	0.309
SENSORLESS	0.073	0.076	0.073	0.076
TICTACTOE	0.000	0.023	0.000	0.013
YEAST	0.461	0.449	0.410	0.426
(d)  $K = 0.8\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.165	0.164	0.164	0.164
FASHIONMNIST	0.158	0.163	0.156	0.161
LETTER	0.260	0.274	0.260	0.267
MNIST	0.113	0.121	0.083	0.093
MUSHROOMS	0.000	0.000	0.000	0.000
NURSERY	0.706	0.719	0.704	0.721
PENDIGITS	0.011	0.026	0.008	0.020
PHISHING	0.042	0.054	0.048	0.063
SATIMAGE	0.136	0.174	0.128	0.183
SEGMENTATION	0.140	0.463	0.121	0.249
SENSORLESS	0.075	0.079	0.074	0.077
TICTACTOE	0.392	0.301	0.000	0.008
YEAST	0.394	0.422	0.686	0.671
(e)  $K = \sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.167	0.166	0.167	0.167
FASHIONMNIST	0.159	0.164	0.157	0.162
LETTER	0.253	0.269	0.255	0.267
MNIST	0.110	0.119	0.084	0.094
MUSHROOMS	0.000	0.000	0.000	0.000
NURSERY	0.706	0.719	0.703	0.722
PENDIGITS	0.012	0.026	0.012	0.024
PHISHING	0.042	0.050	0.039	0.053
SATIMAGE	0.137	0.176	0.125	0.169
SEGMENTATION	0.162	0.398	0.153	0.387
SENSORLESS	0.075	0.077	0.073	0.076
TICTACTOE	0.000	0.035	0.000	0.008
YEAST	0.415	0.435	0.418	0.450
(f)ERM
Dataset	Rs(h)	Rμ(h)
ADULT	0.165	0.163
FASHIONMNIST	0.163	0.167
LETTER	0.258	0.270
MNIST	0.119	0.127
MUSHROOMS	0.000	0.000
NURSERY	0.706	0.719
PENDIGITS	0.009	0.022
PHISHING	0.046	0.055
SATIMAGE	0.141	0.189
SEGMENTATION	0.174	0.389
SENSORLESS	0.075	0.078
TICTACTOE	0.000	0.023
YEAST	0.644	0.682
Table 3: Performance of Algorithm 1 compared to ERM on different datasets for linear models. We consider  $\varepsilon = 1 / m$  and  $\varepsilon = 1 / \sqrt{m}$ , with  $K = \alpha \sqrt{m}$  and  $\alpha \in \{0,0.4,0.6,0.8,1\}$ . We plot the empirical risk  $\Re_S(h)$  with its associated test risk  $\Re_{\mu}(h)$ .
(a)  $K = 1$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.207	0.207	0.248	0.248
FASHIONMNIST	0.142	0.155	0.126	0.149
LETTER	0.286	0.296	0.286	0.295
MNIST	0.067	0.092	0.069	0.094
MUSHROOMS	0.001	0.001	0.000	0.000
NURSERY	0.788	0.799	0.796	0.804
PENDIGITS	0.049	0.060	0.047	0.057
PHISHING	0.063	0.065	0.057	0.062
SATIMAGE	0.142	0.202	0.136	0.199
SEGMENTATION	0.053	0.151	0.079	0.176
SENSORLESS	0.907	0.911	0.907	0.911
TICTACTOE	0.013	0.021	0.013	0.021
YEAST	0.702	0.720	0.693	0.687
(b)  $K = 0.4\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.166	0.167	0.166	0.167
FASHIONMNIST	0.128	0.150	0.126	0.150
LETTER	0.285	0.296	0.286	0.297
MNIST	0.069	0.089	0.067	0.093
MUSHROOMS	0.001	0.001	0.001	0.001
NURSERY	0.760	0.778	0.769	0.781
PENDIGITS	0.050	0.061	0.048	0.061
PHISHING	0.062	0.067	0.065	0.068
SATIMAGE	0.565	0.773	0.137	0.200
SEGMENTATION	0.058	0.212	0.177	0.382
SENSORLESS	0.220	0.220	0.133	0.134
TICTACTOE	0.378	0.290	0.013	0.021
YEAST	0.488	0.478	0.492	0.478
(c)  $K = 0.6\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.166	0.167	0.166	0.167
FASHIONMNIST	0.127	0.147	0.127	0.150
LETTER	0.288	0.296	0.286	0.296
MNIST	0.067	0.092	0.067	0.093
MUSHROOMS	0.001	0.001	0.001	0.001
NURSERY	0.791	0.802	0.759	0.779
PENDIGITS	0.048	0.061	0.047	0.059
PHISHING	0.062	0.067	0.064	0.068
SATIMAGE	0.146	0.202	0.137	0.199
SEGMENTATION	0.058	0.215	0.058	0.204
SENSORLESS	0.129	0.130	0.130	0.130
TICTACTOE	0.013	0.021	0.013	0.021
YEAST	0.477	0.461	0.478	0.464
(d)  $K = 0.8\sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.166	0.167	0.166	0.167
FASHIONMNIST	0.130	0.149	0.128	0.151
LETTER	0.285	0.296	0.288	0.297
MNIST	0.067	0.091	0.067	0.093
MUSHROOMS	0.001	0.001	0.001	0.001
NURSERY	0.771	0.787	0.758	0.778
PENDIGITS	0.047	0.060	0.047	0.059
PHISHING	0.062	0.066	0.065	0.068
SATIMAGE	0.168	0.216	0.137	0.199
SEGMENTATION	0.053	0.212	0.052	0.204
SENSORLESS	0.129	0.130	0.132	0.132
TICTACTOE	0.013	0.021	0.013	0.021
YEAST	0.476	0.461	0.477	0.460
(e)  $K = \sqrt{m}$
Dataset	Alg. 1 (1/m)	Alg. 1 (1/√m)
RS(h)	Rμ(h)	RS(h)	Rμ(h)
ADULT	0.166	0.167	0.166	0.167
FASHIONMNIST	0.354	0.361	0.127	0.151
LETTER	0.287	0.296	0.288	0.298
MNIST	0.068	0.092	0.065	0.092
MUSHROOMS	0.001	0.001	0.001	0.001
NURSERY	0.795	0.805	0.796	0.805
PENDIGITS	0.050	0.062	0.047	0.059
PHISHING	0.062	0.067	0.065	0.067
SATIMAGE	0.143	0.200	0.137	0.201
SEGMENTATION	0.055	0.210	0.055	0.212
SENSORLESS	0.130	0.130	0.131	0.132
TICTACTOE	0.013	0.021	0.392	0.301
YEAST	0.476	0.456	0.476	0.457
(f)ERM
Dataset	Rs(h)	Rμ(h)
ADULT	0.166	0.167
FASHIONMNIST	0.139	0.153
LETTER	0.287	0.297
MNIST	0.065	0.091
MUSHROOMS	0.001	0.001
NURSERY	0.794	0.807
PENDIGITS	0.052	0.064
PHISHING	0.064	0.067
SATIMAGE	0.148	0.209
SEGMENTATION	0.087	0.232
SENSORLESS	0.134	0.136
TICTACTOE	0.228	0.238
YEAST	0.470	0.427
Table 4: Performance of ERM with weight decay (with the L2 regularisation) for linear and neural network models.
(a) Linear	(b) NN
Dataset	L2 Reg. (1/m)	L2 Reg. (1/√m)	Dataset	L2 Reg. (1/m)	L2 Reg. (1/√m)
R_S(h)	R_μ(h)	R_S(h)	R_μ(h)	R_S(h)	R_μ(h)	R_S(h)	R_μ(h)
ADULT	0.207	0.207	0.248	0.248	ADULT	0.207	0.207	0.248	0.248
FASHIONMNIST	0.141	0.149	0.127	0.150	FASHIONMNIST	0.160	0.166	0.159	0.164
LETTER	0.285	0.295	0.285	0.296	LETTER	0.261	0.275	0.256	0.269
MNIST	0.067	0.092	0.066	0.092	MNIST	0.116	0.125	0.084	0.095
MUSHROOMS	0.001	0.001	0.000	0.000	MUSHROOMS	0.000	0.000	0.000	0.000
NURSERY	0.788	0.799	0.796	0.804	NURSERY	0.704	0.721	0.770	0.788
PENDIGITS	0.049	0.060	0.047	0.057	PENDIGITS	0.009	0.022	0.012	0.026
PHISHING	0.063	0.065	0.057	0.062	PHISHING	0.042	0.050	0.054	0.059
SATIMAGE	0.144	0.203	0.138	0.200	SATIMAGE	0.150	0.215	0.143	0.205
SEGMENTATION	0.058	0.157	0.075	0.177	SEGMENTATION	0.141	0.216	0.198	0.371
SENSORLESS	0.907	0.911	0.907	0.911	SENSORLESS	0.907	0.911	0.907	0.911
TICTACTOE	0.013	0.021	0.013	0.021	TICTACTOE	0.000	0.046	0.000	0.021
YEAST	0.702	0.720	0.693	0.687	YEAST	0.662	0.674	0.693	0.683