SlowGuess/NeurIPS_2023_OCR huggingface dataset:
More precisely, we consider a binary version of the MNIST dataset introduced in the seminal work of (Dziugaite & Roy, 2017) for the specific purpose of comparing the generalization guarantees of PAC-Bayes and other approaches, and later employed in, e.g., (Rivasplata et al., 2019). In this problem, the digits 0-4 and 5-9 are mapped to the labels 0 and 1. To classify the inputs, we employ a fully connected feed-forward neural network with three hidden layers each with 600 nodes and ReLu activation functions. The input has 784 nodes and the output has two nodes which are passed to a softmax function. This architecture, employed in the above-cited works, is used across all our experiments.
Figure 2: Top row: Average bounds on the risk (dashed) and misclassification on the test dataset (solid)  $\pm$  one standard deviation for P2L  $\diamond$ , test-set  $\bigcirc$ , and PAC-Bayes  $\triangle$  with a confidence of  $\delta = 0.035$  (left) and  $\delta = 0.001$  (right). The solid markers denote the best bounds achieved and their corresponding post-training performance. Bottom row: Distribution of the upper bounds on the risk and misclassification on the test dataset for the models achieving the best bounds for P2L  $+$ , test-set  $+$ , and PAC-Bayes  $+$ . Precisely, each point in these figures corresponds to the pair (generalization bound, actual misclassification level) achieved in one of the 60 partitions of the MNIST training dataset we use. The means are indicated with a solid diamond, circle, and triangle, respectively.
MNIST consists of a training dataset containing 60000 examples and a test dataset with 10000 examples. In our experiments, we train all models only on 1000 samples from the training dataset at a time. We particularly care of this "small" dataset setting, in which one has to make the most of the data for both training and assessing the generalization. This is a setting of interest to various fields where data are a limited, possibly costly, resource. Specifically, we shuffle the original MNIST training dataset and extract 60 disjoint datasets with 1000 data points each. All approaches we compare are run on the resulting 60 datasets, which are used to both train the network and provide generalization bounds. The full test dataset, containing 10000 examples, is never used in any training phase. It is instead used to evaluate the actual post-training performance of each trained model. We
Our experience ranges from applications to cardiac defibrillation in which patients in out-of-hospital cardiac arrest are classified as being able or not able to positively react to a defibrillation shock (in the latter, alternatives are possible, chiefly administrating a cardiac massage prior to the shock), to applications with spacesuit testing in which suits are artificially bombarded with particles to mimic the condition encountered in the outer space. While in the first example the number of recorded cases is potentially very high (modern defibrillators automatically collect ECG traces), still these data are not made public and easy to secure in large quantities, while in the second example each single data point can be quite costly, suggesting that the dataset has to be kept as small as possible.
now present the specific algorithmic choices made for P2L. Due to space limitations, the details concerning the test-set approach and PAC-Bayes can be found in Appendix B.1.
P2L. As discussed, P2L is fully specified once a learning algorithm  $L$ , an initial hypothesis  $h_0$  and a hypothesis-dependent total ordering  $\leq_h$  with a criterion of stop are defined. For the learning algorithm  $L$ , we choose the widely employed GD (gradient descent) with momentum, which we run for 200 epochs with a learning rate of 0.01, momentum of 0.95, dropout probability of 0.2. For the sake of clarity, we remark that each time GD is called by P2L, GD iterates until convergence (or until reaching the allowed number of epochs) so as to perform the minimization of the empirical risk. As initialization of GD, we take the  $h$  returned at the previous iteration of P2L.<sup>7</sup> Regarding  $h_0$ , it is clear that when starting from an educated guess the choice of the worst example to be inserted in  $T$  tends to be more meaningful, allowing P2L to terminate with a smaller  $T$  and, thus, a better generalization bound. To this purpose, one can use a portion of each training dataset to pretrain  $h_0$  (using GD again), while P2L and, therefore, the ensuing generalization bound rely on the remaining part of the training dataset. In our computation, we experiment with multiple sizes of the pretraining portion. As for  $\leq_h$ , given an hypothesis  $h$ , i.e., a neural network, we use the total order induced by the cross-entropy of its output as discussed in Example 3.1 (ties are broken according to the lexicographic order of the examples' bit representation). This choice ensures that the statistical risk of Definition 4.1 corresponds to the probability of misclassification, and thus a bound on the statistical risk coincides with a bound on the misclassification. The final bound we present on the misclassification is that of the main result in Theorem 4.2, where the number of samples equals  $N$  minus those used to pretrain the initial hypothesis  $h_0$ .
Experimental results. The results are presented in Figure 2 for different values of the confidence. The top row depicts, the average (over the 60 trials) generalization bounds as well as the average misclassification levels on the test dataset (post-training performances) with their dispersion for the three approaches. The values are plotted as functions of the portion of the dataset (called train/pretrain portion) used to train the model in GD+test-set, to pretrain the prior distribution in PAC-Bayes, and to pretrain the initial hypothesis  $h_0$  in P2L. The joint distribution of the returned bound and actual post-training misclassification level for the best models (corresponding to train/pretrain portion equal to 0.5, 0.6, 0.7 for P2L, PAC-Bayes, and test-set approach, respectively) are shown in the bottom row. The average bounds and misclassification levels for these best models are also presented in Table 1 for  $\delta = 0.035$ . For the sake of completeness, Table 1 also reports the average running times of one execution (i.e., for one dataset with 1000 examples) of the three approaches (computational resource: Apple MacBook Pro with M1 Pro CPU and 32Gb of ram).
Table 1: Risk of best models for  $\delta  = {0.035}$  .
Bound on risk	Risk on the test dataset	Difference	Average running time
P2L	0.143	0.072	0.071	2m 1s
Test-set	0.129	0.079	0.050	0m 5s
Pac-Bayes	0.177	0.088	0.089	4m 1s
Conclusions. Four important observations are in order. First, it is evident that, in the present application, the implementation of P2L with GD outperforms the PAC-Bayes approach with respect to both the provided upper bound on the risk and the post-training performance on the test dataset. This is true not just for the best model learned with each approach (cfr. the solid diamonds and the solid triangles in Figure 2), but it holds uniformly across all train/pretrain portions we tried (cfr. the green and red curves). Notably, the model returned by P2L+GD when using only  $10\%$  of the data to pretrain the initial hypothesis  $h_0$  has a risk bound that is comparable to that of the best PAC-Bayes model (whose prior is trained on  $60\%$  of the data) while it achieves a better post-training performance. Second, P2L+GD provides best risk bounds that are comparable to those of GD+test-set approach, albeit slightly inferior (0.143 vs 0.129 for  $\delta = 0.035$  and 0.163 vs 0.155 for  $\delta = 0.001$ ). However, P2L+GD provides better post-training performance uniformly across all train/pretrain portions. As a matter of fact, for any choice of pretrain portion, P2L+GD's performance is equal to that obtained when GD is run on the whole data set ( $N = 1000$ ), for which the test-set approach cannot provide
<sup>7</sup>As an alternative, one could also re-train a model from scratch at every iteration. This implementation of GD can be seen as a slightly different choice of the inner algorithm  $L$  for P2L. The selected implementation of GD is motivated by the ensuing computational advantage, since fewer gradient steps are typically needed to converge, and the fact that this implementation makes  $h$  less sensitive to the addition of one example in  $T$ , which is helpful in later stages of P2L to speed up its termination.
any meaningful bound since no data are left for testing the model. This is a clear indication of the important feature that  $P2L$  utilizes all data to jointly learn a good model and provide a risk bound. Third, the post-training performance of PAC-Bayes and GD+test-set approaches are similar. This suggests that in PAC-Bayes the training of the posterior does not exploit the additional available data to improve the model, but rather to certify it, in a similar vein as in the test-set approach. This fact has also been observed recently in (Lotfi et al., 2022)[Fig 1(a)]. Fourth, as for the tightness of the bounds (i.e., the difference between the upper bounds and risk on the test dataset), GD+test-set approach provides the tightest results, while P2L+GD and PAC-Bayes approaches alternate depending on the specific train/pretrain portion selected.
6 Application to a synthetic regression problem
In this section we apply our methodology to a synthetic regression problem in order to showcase the flexibility of the approach introduced in Section 4.
Specifically, we consider 100 distinct training datasets with  $N = 200$  examples  $(x,y)$  and then another 100 distinct training datasets with  $N = 500$  examples  $(x,y)$ . The input  $x\in [-0.5,1.5]$  is extracted uniformly at random and the output is  $y = f(x) + e$  with  $f(x) = \sin (2.5\pi x) / (2.5\pi x)$  and  $e$  extracted according to a normal distribution with zero mean and standard deviation 0.05, see Figure 3. For each dataset, our objective is twofold: finding a model that fits the data well, and providing a bound on the probability that the model mispredicts the output by more than a threshold  $\gamma = 0.1$  (bounds of this sort matters in relation to many applications).
In pursuing this goal, we compare the generalization bounds and post-training performances attained using P2L and test-set bounds. For illustration purposes, in all our experiments we consider a simple network architecture comprising one input node, one output node, and one hidden layer with six nodes, each equipped with a tanh activation function. To evaluate the post-training performance, we use an additional test dataset with 20000 examples. The value of  $\delta$  is set to 0.035.
Figure 3: An instance of dataset used for regression. The crosses are examples  $(x, y)$ . The black line depicts the function  $f(x) = \sin(2.5\pi x) / (2.5\pi x)$  around which noise is added. The red dashed and blue solid lines represent the fit obtained with a choice of train/pretrain portion equal 0.3 (best) for P2L and test-set approaches. The red shaded region depicts a tube of radius 0.1 around the red model. The risk of the red model is the probability that an unseen example falls outside the shaded region.
P2L. In deploying P2L we use GD as learning algorithm  $L$ , and, similarly to the MNIST example, we experiment with initial hypotheses  $h_0$  trained on different fractions of the training dataset (including the null fraction in which case  $h_0$  is the network with all weights set to 0). We fix the same total order  $\leq h$  of Example 3.2, with  $\gamma = 0.1$ . In this context, the risk introduced in Definition 4.1 measures the probability that the output  $y$  is mispredicted by more than  $\gamma$ , thus giving us a guarantee on the quality of the model. The generalization bound we use is that in Theorem 4.2, where the number of samples equals  $N$  minus those used to pretrain the initial hypothesis  $h_0$ . We use a learning rate of 0.1, momentum of 0.95, 1000 epochs, and no dropout.
Test-set approach. We apportion each dataset in two, with one portion used for training, and the other for deriving the generalization bounds. We run GD for 1000 epochs, perform a grid search over learning rates [0.001, 0.005, 0.01], momentum [0.9, 0.95] (no dropout), and select those giving the best generalization bound. As in the MNIST example (see Appendix B.1), we use the binomial test-set bound of (Langford, 2005)[Thm 3.3] with a number of samples equal to  $N$  minus those used for training.
Experimental results. The first two panels of Figure 4 present the average (over the 100 trials) upper bounds on the risk and the average risk on the test dataset jointly with their dispersion, as a function of the data portion used to train the model (GD+test-set) or to pretrain the initial hypothesis (P2L). The third and fourth panels depict the distribution of the upper bound and of the risk on the test dataset
Figure 4: Top row: Average bounds on the risk (dashed) and risk on the test dataset (solid)  $\pm$  one standard deviation for P2L  $\diamond$  and test-set  $\bigcirc$  approaches, with  $N = 200$  (left) and  $N = 500$  (right). The solid markers denote the best average bounds and the corresponding risks on the test dataset. Bottom row: Empirical distribution of the bound and risk on the test dataset for the data split returning the best average bound for P2L + and test-set +. Their means are indicated with a solid diamond and circle, respectively. Note that the test-set approach is not considered for zero train portion, as it always requires a non-zero amount of data to train the model. Observe that three instances in the test-set case (out of one hundred) in the bottom row violate the upper bound. This is in line with the choice of  $\delta = 0.035$ , suggesting that the bound will fail, on average, on 3.5 datasets out of one hundred.
for the data proportion 0.3, returning the best learned models for both P2L and the test-set approach. Their averages are compared in Table 2, which also includes the average runtime (computational resource: Apple MacBook Pro with M1 Pro CPU and 32Gb of ram).
Table 2: Risk of best models for  $N = {200},\delta  = {0.035}$
Bound on risk	Risk on the test dataset	Difference	Average running time
P2L	0.172	0.064	0.108	1.18s
Test-set	0.250	0.173	0.077	0.21s
Conclusions. First, it appears evident that, in this synthetic regression problem, the application of P2L with GD provides superior results to those achieved by GD with the test-set approach, regardless of the size of the datasets used for training. This holds true jointly for the upper bound and the risk on the test dataset uniformly across all train/pretrain portions (cfr. the red and blue dashed lines, similarly for the solid ones). We ascribe this result to the fact that P2L does not set aside data for testing, and yet it also provides rigorous evaluations of the risk. Second, the test-set approach provides bounds that are closer to the risk it incurs on the test dataset for  $N = 500$ , while this effect is less clear with  $N = 200$ . Third, as expected, when the size of the training dataset grows, both the upper bounds and the risk on the test dataset improve.
To conclude, it is fair to notice that GD does not pursue directly the goal of obtaining a prediction error smaller than  $\gamma$ , and this may justify the large gap in the post-training performance between
the two approaches also when almost all the dataset is used for training in the test-set approach. Although GD is used as inner algorithm  $L$  in P2L, it seems that a good performance with respect to the chosen appropriateness criterion can be obtained thanks to the structure of the meta-algorithm. This is another interesting feature of P2L.
7 Conclusions, Limitations and Future research
We have proposed a novel framework called P2L to provide virtually any learning algorithm with sharp generalization bounds. Our approach is based on making a given learning algorithm into a compression scheme with desirable properties, thus enabling the use of powerful generalization results. Numerical results show that P2L is capable of learning hypotheses with post-training performances and generalization bounds equal or superior to the state of the art.
Computational aspects. While P2L requires learning a hypothesis over a training set of increasing size, and thus might be less efficient than learning the hypothesis only once over the full dataset, it is important to look at it from the appropriate perspective: we are concerned with settings where data is limited or costly to acquire (see, e.g., Footnote 6 and ensuing discussion) while computations are performed off-line and fast execution does not represent a primary concern (a setup also considered in other recent work, e.g., (Foong et al., 2021)).
Source of conservatism. In our framework, as revealed by the proof of Theorem 4.2, the risk is controlled by bounding the probability of change of compression (as defined in Equation (2) in Appendix A.1). Notably, the mismatch between the risk and the probability of change of compression is the only source of conservatism our approach needs to resolve since the upper and lower bounds on the probability of change of compression from Theorem A.4 are extremely tight – see Remark A.9. The magnitude of this conservatism is determined by the choices we make in specializing the proposed meta-algorithm. For a given learning algorithm  $L$ , these choices entail selecting an initial hypothesis  $h_0$  and an hypothesis-dependent total order used to select which data points are fed to the learning algorithm  $L$ . Our experimenting with multiple initial hypotheses (trained with different portions of the dataset) was solely geared at reducing this gap. Indeed, after training  $h_0$ , P2L allows the data to "freely speak", and thus improve the resulting hypothesis, as it can be appreciated from the fact that the misclassification on the test dataset for P2L is constant across all prior/train portions (see Figure 2, top row). This is in stark contrast with the test-set approach and even PAC-Bayes. In the former, data are either used to train the model or to provide a risk bound. In the latter, data employed to train the posterior are effectively used to compute a risk bound as opposed to significantly improving the quality of the prior (see first row in Figure 2). We conclude noting that our choices of the initial hypothesis and total order is but one of many possible. Depending on the specific learning problem, other choices can be made (see for examples the last part of the Introduction). We believe that the overall fact that the theoretical apparatus in our approach clearly identifies the sole source of conservatism will put us and others in the position to build upon the P2L framework beyond this work.
This paper was partly supported by FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, Investment 1.3, Line on Artificial Intelligence); and the EPSRC grant EP/Y001001/1, funded by the International Science Partnerships Fund (ISPF) and UKRI.
Lotfi, S., Finzi, M., Kapoor, S., Potapczynski, A., Goldblum, M., and Wilson, A. G. PAC-Bayes compression bounds so tight that they can explain generalization. arXiv preprint arXiv:2211.13609, 2022.
Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34:20596-20607, 2021.
Perez-Ortiz, M., Rivasplata, O., Shawe-Taylor, J., and Szepesvari, C. Tighter risk certificates for neural networks. Journal of Machine Learning Research, 22:1-40, 2021.
Rivasplata, O., Tankasali, V. M., and Szepesvári, C. PAC-Bayes with backprop. arXiv preprint arXiv:1908.07380, 2019.
Smola, A. and Schölkopf, B. A tutorial on support vector regression. Statistics and Computing, 14: 199-224, 2004.
Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018.
Vapnik, V. and Chervonenkis, A. On the uniform convergence of relative frequencies of events to their probabilities. Theory of probability and its applications, 16:264-280, 1971.
Wu, Y.-S. and Seldin, Y. Split-kl and PAC-Bayes-split-kl inequalities for ternary random variables. In Proceedings of the 35th conference on Advances in Neural Information Processing Systems (NeurIPS 2022), 2022.
Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), 2017.
Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329, 2022.
A Proof of Theorem 4.2 and Additional Results
In this section we prove Theorem 4.2. En route to this goal, we will also present additional results that help better position our contribution. Doing so will allow us to highlight directions where we believe there is room for exciting future work. The section is conceptually divided in three portions: in Appendix A.1 we recall a relevant result from (Campi & Garatti, 2023) along with the necessary mathematical background; this result is then leveraged in Appendix A.2 to prove Theorem 4.2 and other related results; finally, in Appendix A.4 we revisit the P2L framework in the light of some technical details revealed by the proof of Theorem 4.2.
A.1 Preferent Compression Gives Tight Risk Bounds
We begin with introducing two properties of a compression function  $\mathsf{c}$  (preference and non-associativity) and the property of non-concentrated mass from which strong generalization guarantees follow. Recall that a compression function  $\mathsf{c}$  is a map from any multiset  $D$  of elements in  $\mathcal{Z}$  to a submultiset  $\mathsf{c}(D) \subseteq D$ .<sup>8</sup> Throughout, we use  $\mathsf{c}(D, z)$  as a shorthand for  $\mathsf{c}(D \cup \{z\})$ , and likewise for  $\mathsf{c}(D, z_1, \ldots, z_p)$ . Moreover,  $z$  denotes a random example independent of and identically distributed as each  $z_i$ .
Property A.1 (Preference). For any pair of multisets  $V$  and  $D$  of elements of  $\mathcal{Z}$  with  $V \subseteq D$ ,  $\mathsf{c}(D) \neq V \Rightarrow \mathsf{c}(D, z) \neq V, \forall z \in \mathcal{Z}$ .
The preference property asserts that, if the multiset  $V$  was not chosen when compressing  $D$ , it will also not be chosen when compressing the set  $D$  augmented with one more element  $z$ . Under the preference property, it is easy to prove that  $\mathsf{c}(\mathsf{c}(D)) = \mathsf{c}(D)$ , i.e., the compression of a compressed multiset must be the compressed multiset itself.
Property A.2 (Non-associativity). For any multiset  $D$  of elements of  $\mathcal{Z}$  and for any  $z_{1},\ldots ,z_{p}\in \mathcal{Z}$ ,  $p > 1$ ,  $\mathsf{c}(D,z_i) = \mathsf{c}(D)$ ,  $\forall i\in \{1,\dots ,p\} \Longrightarrow \mathsf{c}(D,z_1\dots ,z_p) = \mathsf{c}(D)$ .
The non-associativity property ensures that, if the compression does not change by adding one element at a time, it must also not change when adding all the elements together.[9]
Property A.3 (Non-concentrated mass).  $\mathbb{P}\{z = z\} = 0, \forall z \in \mathcal{Z}$
The property of non-concentrated mass ensures that every point  $z \in \mathcal{Z}$  is drawn with probability zero.
Under these three properties, strong results on the probability of change of compression, formally defined as
$$
\phi (\boldsymbol {D}) = \mathbb {P} \left\{\mathrm {c} (\mathrm {c} (\boldsymbol {D}), \boldsymbol {z}) \neq \mathrm {c} (\boldsymbol {D}) \mid \boldsymbol {D} \right\} \tag {2}
$$
(i.e., change of compression occurs when the compression of  $\mathsf{c}(D)$  augmented with one element is different from  $\mathsf{c}(D)$ ) are established in (Campi & Garatti, 2023).
Theorem A.4 (Thm. 7 in (Campi & Garatti, 2023)). Assume Properties A.1 to A.3. Then, for any  $\delta \in (0,1)$ ,