SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Importantly, these are the only two properties of  $\max_h$  that are used and, therefore, one could have formulated Algorithm 1 in an (apparently, see below) more general form: instead of introducing a hypothesis-dependent total ordering  $\leq_h$ , one might have considered in its place a hypothesis-dependent preferent and non-associative compression function from many to one example. Mutatis mutandis, Theorem 4.2 and the arguments to prove it would have remained the same. This generalization extends to the context in which multiple examples are selected at a time as suggested in Appendix A.3.
We feel advisable to also notice that the above generalization is indeed a real generalization only in the case of selection of multiple example at a time. Instead, in the context of one example selected at a time the alternative setup illustrated above turns out not to be more general than that of Section 4 (this is why we wrote "apparently" in parenthesis). In fact a converse property holds: any compression function from many examples to one that is preferent and non-associative always defines a total ordering whose notion of max coincides with the compression function itself. This fact follows from the following proposition (see a few lines down for a proof).
Proposition A.10. Any preferent compression function  $\mathbb{W}$  from many examples to one defines a total ordering  $\leq_{\mathbb{W}}$  for which it holds that  $\mathbb{W}(a_1,\ldots ,a_n) = \max_{\mathbb{W}}(a_1,\ldots ,a_n)$ .
Note that, as a byproduct of Proposition A.10, we have that any preferent compression function  $w$  from many examples to one is also non-associative (since  $\max_w$  is non-associative). Therefore, the notion of  $\max$  from a total ordering and that of preferent and non-associative compression function from many examples to one are equivalent. Interestingly, this equivalence crucially relies on the fact that the compression selects only one example, while, provably, the class of preferent and non-associative compression functions offers more freedom than using a total ordering when selecting more than one example at a time. This is important for the development of alternative implementations of Algorithm 2, the version of P2L where multiple examples are selected at a time.
Proof of Proposition A.10. For any  $a$  and  $b$  define  $a \leq_{\mathbb{W}} b$  if  $\mathsf{w}(a, b) = b$  (and thus  $b \leq_{\mathbb{W}} a$  if  $\mathsf{w}(a, b) = a$ ). To show that  $\leq_{\mathbb{W}}$  is indeed an ordering, we have to prove that  $\leq_{\mathbb{W}}$  enjoys the reflexive, antisymmetric, and transitive properties.
a. Since  $\mathfrak{w}$  always selects one element, we have that  $\mathfrak{w}(a,a) = a$ , i.e.,  $a \leq_{\mathfrak{w}} a$ , which is the reflexive property.
b. If  $a \leq_{\mathbb{W}} b$  and  $b \leq_{\mathbb{W}} a$ , then  $\mathsf{w}(a, b) = b$  and  $\mathsf{w}(a, b) = a$ . Thus,  $a \leq_{\mathbb{W}} b$  and  $b \leq_{\mathbb{W}} a \Rightarrow a = b$ , which is the antisymmetric property.
c. Suppose that  $a \leq_{\mathbb{W}} b$  and  $b \leq_{\mathbb{W}} c$ , which corresponds to  $\mathsf{w}(a, b) = b$  and  $\mathsf{w}(b, c) = c$ . Since  $\mathbb{W}$  is preferent, it must be that  $\mathsf{w}(a, b, c) = c$  (indeed,  $\mathsf{w}(a, b, c)$  cannot be  $a$  because  $a$  is not selected by  $\mathbb{W}(a, b)$ ; it cannot be  $b$  either because  $b$  is not selected by  $\mathbb{W}(b, c)$ ). Then, an application of preference again gives  $\mathsf{w}(a, c) = c$ . This means that  $a \leq_{\mathbb{W}} b$  and  $b \leq_{\mathbb{W}} c$  implies that  $a \leq_{\mathbb{W}} c$ , which is the transitive property.
We next prove that the  $\max_{\mathbf{w}}$  operator associated to  $\leq_{\mathbf{w}}$  corresponds indeed to  $\mathbf{w}$ ; that is, it holds that  $a_i \leq_{\mathbf{w}} \mathbf{w}(a_1, \ldots, a_n)$  for all  $i \in \{1, \ldots, n\}$ . The proof is by contrapositive. Suppose thus that  $\mathbf{w}(a_1, \ldots, a_n) = a_i$  for some  $i$ , but  $a_i <_{\mathbf{w}} a_j$  (i.e.,  $a_i \leq_{\mathbf{w}} a_j$  and  $a_i \neq a_j$ ) for some  $j$ . Condition  $a_i <_{\mathbf{w}} a_j$  means that  $\mathbf{w}(a_i, a_j) = a_j$ . By preference, adding another element  $a_k$  to  $a_i$  and  $a_j$  would give  $\mathbf{w}(a_i, a_j, a_k) \neq a_i$ . But then, preference again would give that the compression of  $a_i, a_j$ , and  $a_k$  plus a fourth element cannot be  $a_i$ , and so forth and so on. Thus, proceeding iteratively would eventually lead to  $\mathbf{w}(a_1, \ldots, a_n) \neq a_i$ , which is a contradiction.
B Additional material for the application to MNIST Classification
B.1 Implementation details for SGD and PAC-Bayes
GD & test-set approach. Here we follow a classical test-set approach, whereby a fraction of the dataset is used for training, and the remaining portion is sacrificed to certify the quality of the resulting model. We experiment with different sizes of these fractions. As for training, we use GD with momentum and fix the number of training epochs to 200. To compute the generalization bound, we use the tightest-known Binomial test-set bound $^{10}$  of (Langford, 2005)[Thm 3.3] with a number of samples equal to  $N$  minus those used for training. That is, letting  $\tilde{N}$  be the number of examples out
<sup>10</sup>The Binomial test set bound is tighter then the commonly utilized Chernoff bound.
of  $N$  used for training and  $\tilde{h} = GD(z_1,\ldots ,z_{\tilde{N}})$  be the returned hypothesis, we have that
$$
R (\tilde {h}) \leq \max  \left\{p \in [ 0, 1 ]: \sum_ {j = 0} ^ {k} \binom {N - \tilde {N}} {j} p ^ {j} (1 - p) ^ {N - \tilde {N} - j} \geq \delta , \right. \tag {4}
$$
where  $k$  is the number of  $z_{j}, j = \tilde{N} + 1, \ldots, N$ , misclassified by  $\tilde{h}$
with high confidence  $1 - \delta$ . We also perform a grid search over the following set of parameters: learning rate in [0.001, 0.005, 0.01], momentum in [0.9, 0.95], dropout probability in [0.01, 0.05, 0.1, 0.2]. We then select the combination of parameters that return the lowest generalization bound. $^{11}$
PAC-Bayes approach. Here we learn a model and provide generalization bounds based on the recent results presented in (Clerico et al., 2022; Perez-Ortiz et al., 2021), which build on earlier work in (Dziugaite & Roy, 2017) while providing tighter generalization bounds. Specifically, we utilize the same PAC-Bayes implementation of (Perez-Ortiz et al., 2021) adapted to the considered binary classification problem, with one main difference: instead of training by optimizing a surrogate of the generalization bound as in (Perez-Ortiz et al., 2021), we optimize the exact bound as proposed in (Clerico et al., 2022, Eq 4.a), which provides the sharpest results to date. $^{12}$  As the PAC-Bayes approach requires a prior distribution over the networ weights to start, we split, as it is customary, the training dataset in two and use the first part to pretrain the prior through SGD, while the remaining data are used to obtain the posterior distribution (contrary to P2L and GD+test-set where the final hypothesis consists of a deterministic network, the PAC-Bayes approach is forced to work with stochastic networks). We experiment with different sizes of the data portion to be used to pretrain the prior. In running PAC-Bayes we employ a batch size of 250, 100 prior and posterior training epochs, and  $10^{-6}$  as lower bound for the minimum probability of the softmax output. We also perform a grid search over the remaining parameters: prior standard deviation in [0.01, 0.02, 0.03, 0.04, 0.05, 0.1]; dropout probability in [0.01, 0.05, 0.1, 0.2]; prior/posterior learning rate in [0.001, 0.005, 0.01]; prior/posterior momentum in [0.9, 0.95]. The final bound we present is that in (Perez-Ortiz et al., 2021)[Sec 6.2], by using 10000 Monte Carlo samples from the learned distribution over the models to approximate the average empirical risk. When we experiment with  $\delta = 0.035$  ( $\delta = 0.001$ ), we use a confidence of 0.025 (0.00072) for the KL bound and of 0.01 (0.00028) for Monte Carlo sampling as done in (Perez-Ortiz et al., 2021)[Sec 6.2]. We apply here the same philosophy as described in Footnote 11 regarding not introducing a union bound correction. The post-training performance is evaluated again by sampling the distribution over the network weights, computing the empirical misclassification on the full test dataset, and averaging over 10000 samples of the weights. As a matter of fact, the generalization bounds derived through the PAC-Bayes approach apply precisely to this quantity.
C Some examples when the inner algorithm  $\mathrm{L}$  has already in itself a compression scheme
In this section, also prompted by a reviewer, we perform some synthetic numerical experiments to investigate how P2L performs when the inner algorithm  $L$  already exhibits a compression in itself; specifically, we consider SVM (Support Vector Machine, (Cortes & Vapnik, 1995)) and SVR (Support Vector Regression, (Smola & Scholkopf, 2004)) as inner algorithms for P2L and compare the returned output with that output by SVM and SVR alone. The so-called support vectors returned by SVM and SVR are known to compress the original dataset in a way that is informative for the underlying learning problem. With these experiments, we want to verify whether P2L returns the same or a similar compression.
Formally, we should introduce a union bound correction because we perform a grid search over multiple hyper-parameters. Notably, this correction would enlarge (i.e., worsen) the risk bound. However, as the union bound might be conservative, we do not pursue this approach and give this advantage to both the test-set and, later, the PAC-Bayes approach.
12The research in the field of PAC-Bayes is extremely active, and we would like to acknowledge paper (Wu & Seldin, 2022), which has introduced improved PAC-Bayes bounds based on novel concentration inequalities for random variables with ternary values. However, in a binary setup (as in the present application), (Wu & Seldin, 2022) itself notices that there is no advantage over the existing literature. Thus, to the best of our knowledge, the bounds proposed by (Clerico et al., 2022) are currently the tightest for our problem.
Figure 5: Comparison of learned classifiers and compressed sets returned by SVM and P2L+SVM. Markers  $+$ ,  $\times$  represent examples with labels  $\pm 1$ . The dashed blue and solid red lines  $(-,-,-)$ , which are completely overlapped, represent the classifier returned by SVM and P2L+SVM. The blue triangles and red squares  $(▲,□)$  represent the support vectors returned by SVM and the compressed set  $T$  returned by P2L+SVM.
In a classification problem, we consider datasets with  $N = 1000$  examples  $(x,y)$ , where instances  $x$  are uniformly sampled from  $[-1,1]^2 \subseteq \mathbb{R}^2$  and labels are 1 if  $(x_1 - 0.3)^2 + x_2^2 \leq 0.7$  and  $-1$  otherwise. SVM (either when run over the whole dataset or when called by P2L as inner algorithm) is implemented with kernel  $k(x,\tilde{x}) = x \cdot \tilde{x}' + \| x\|^2 \cdot \| \tilde{x}\|^2$ , for which the two classes defined above are separable. P2L is run by selecting as  $h_0$  the classifier corresponding to setting all the parameters in the SVM parameterization to 1. As for  $\leq_h$ , we consider the order induced by the distance in the lifted feature space of examples from the separating hyperplane corresponding to  $h$ , multiplied by 1 when examples are misclassified and by  $-1$  otherwise (this yields that the worst examples are those misclassified and furthest away from the separating hyperplane). Distance equal to 0 is taken as the threshold for appropriateness, which corresponds to consider a classifier  $h$  appropriate for an example  $(x,y)$  when  $(x,y)$  is not misclassified by  $h$ .
Figure 5 displays the results for two realizations of the considered classification problem, which are representative of what we have observed in multiple trials. Each plot depicts: the examples in the instance domain as markers  $+$  (label 1),  $\times$  (label  $-1$ ); the separation boundaries of the SVM and of the P2L+SVM classifiers (dashed blue and solid red lines, which are completely overlapped); the support vectors returned by SVM (red squares) and the examples in  $T$  returned by P2L+SVM (blue triangles).
In the left panel of Figure 5, the compressed multiset  $T$  returned by P2L+SVM contains all the support vectors of SVM with the addition of few other examples, which are incorporated in  $T$  during the first iterations, while P2L is still exploring the problem. As it is clear, by the definition of support vectors, the classifier returned by P2L+SVM coincides with the classifier obtained by running SVM. In the right panel of Figure 5, only part of the support vectors are also elements of  $T$ . Nonetheless the classifiers obtained with P2L+SVM and SVM are still the same (at least up to a negligible approximation). Hence, it appears that P2L+SVM compensates the deficiency of support vectors with other alternative examples that are anyway capable to recover the SVM classifier. The amount of support vectors appearing in  $T$  depends on the realization of the dataset, but in all our trials the classifier returned by P2L+SVM matches that obtained by running SVM on the same dataset.
As for SVR, we consider the same regression problem as in Section 6 when  $N = 200$ . SVR is run using a radial basis function  $k(x, \tilde{X}) = \exp(-0.1\|x - \tilde{x}\|^2)$  as kernel, tube size equal to 0.1, and coefficient  $C = 1$  ( $C$  modulates the trade-off between the flatness of the predictor and the penalty for deviations from the predictor larger than 0.1). P2L instead starts with a predictor  $h_0$  corresponding to setting all the parameters in the SVR parameterization to 0, while we consider the same total order  $\leq_h$  of Example 3.2 with  $\gamma = 0.1$ . In this context,  $h$  is appropriate for an example  $(x, y)$  if the deviation of the output  $y$  from the prediction corresponding to  $x$  is no more than 0.1.
Figure 6 displays the results obtained for one dataset, which is also representative of other realizations of data. In the plot, examples are represented as black crosses, while the dashed blue and red lines,
Figure 6: Comparison of learned predictors and compressed sets returned by SVR and P2L+SVR. Markers + represent examples. The dashed blue and solid red lines  $(\dashv, -)$ , which are completely overlapped, represent the predictor returned by SVR and P2L+SVR. The red shaded region depicts a tube of radius 0.1 around such predictors. The blue triangles and red squares  $(\triangle, \square)$  represent the support vectors returned by SVR and the compressed set  $T$  returned by P2L+SVR.
which are completely overlapped, depict the predictors returned by SVR and P2L+SVR respectively. The support vectors returned by SVR are displayed by blue triangles, while the examples in  $T$  returned by  $\mathrm{P2L + SVR}$  by red squares.
Similarly to the SVM case, in all our trials the predictor returned by P2L+SVR is identical to the predictor obtained by running SVR on the same dataset. In the present example, however, it is always the case that the compressed multiset  $T$  returned by P2L+SVR contains all the support vectors of SVR, with the addition of few other examples incorporated in  $T$  during the first iterations during which P2L is still exploring the problem.