SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dario Paccagnan
Dept. of Computing  
Imperial College London  
d.paccagnan@imperial.ac.uk
Marco C. Campi
Dip. di Ingegnerira dell'Informazione Università di Brescia marco.campi@unibs.it
Simone Garatti
Dip. di Elettronica Informazione e Bioingegneria
simone garatti@polimi.it
Generalization bounds are valuable both for theory and applications. On the one hand, they shed light on the mechanisms that underpin the learning processes; on the other, they certify how well a learned model performs against unseen inputs. In this work we build upon a recent breakthrough in compression theory (Campi & Garatti, 2023) to develop a new framework yielding tight generalization bounds of wide practical applicability. The core idea is to embed any given learning algorithm into a suitably-constructed meta-algorithm (here called Pick-to-Learn, P2L) in order to instill desirable compression properties. When applied to the MNIST classification dataset and to a synthetic regression problem, P2L not only attains generalization bounds that compare favorably with the state of the art (test-set and PAC-Bayes bounds), but it also learns models with better post-training performance.
Machine learning has achieved remarkable results in the last decades, with successful stories ranging from digit recognition (Dosovitskiy et al., 2020), to protein folding (Jumper et al., 2021), traffic prediction (Li et al., 2017), medical diagnosis (Kononenko, 2001), and beyond. Contrary to that, the understanding of the mechanisms underpinning generalization, and the availability of bounds for its evaluation, is still limited. This is unfortunate because generalization bounds are important both to develop trust in learning methods as well as to allow for a fair comparison among alternatives, which is relevant to hyper-parameter tuning.
Among the various tools that have been introduced to provide generalization bounds, we recall the VC-dimension (Vapnik & Chervonenkis, 1971), Radamacher complexity (Bartlett & Mendelson, 2002), mutual information (Xu & Raginsky, 2017), sharpness (Keskar et al., 2017), compression schemes (Littlestone & Warmuth, 1986; Floyd & Warmuth, 1995; Graepel et al., 2005), and PAC-Bayes approaches (Dziugaite & Roy, 2017; Perez-Ortiz et al., 2021). Each of these frameworks is applicable to specific contexts. For example, the approach based on the VC-theory requires that a model is selected from a class with finite VC-dimension, while compression schemes are applicable provided the existence of an informative subset of data points from which the model can be reconstructed. It has also to be said that the level of precision of the available bounds is highly problem-dependent and they often take a significant margin from the actual performance.
While bounds based on the VC-dimension, and some other aforementioned approaches, are often conservative (resulting in untight, or even vacuous, evaluations), the sharpest approach to evaluate generalization across various learning domains is still that based on test-set bounds (Langford, 2005). However, the test-set approach requires holding out a portion of the training set for testing, which makes it data-inefficient with negative impact on the post-training performance. In view of these considerations, it would be highly desirable to develop a new general-purpose technique able to set the data free in their double role of (i) delivering the information that is needed to learn, while also (ii) providing evaluations on the attained generalization result.
In this paper, we move towards the goal expressed in (i) and (ii). To this aim, we build upon a recent breakthrough in the field of compression schemes, (Campi & Garatti, 2023). Therein, the authors have established new and tight bounds for the so-called probability of change of compression that hold under certain properties: under the property of preference, tight upper bounds have been obtained that link the probability of change of compression to the size of the compressed set<sup>1</sup>; moreover, under a so-called non-associativity property and a property of non-concentrated mass, a lower bound has also been obtained. Hence, under these additional properties, the change of compression is put in sandwich between two bounds, which are shown to rapidly converge one on top of the other as the number of data points increases. The interest of these results in the context of statistical learning lies on the fact that some learning algorithms naturally define a compression scheme and the probability of change of compression can be used to bound the probability of misclassification/misprediction. However, the challenge is that many other learning algorithms (including most algorithms for deep learning) do not exhibit any compression property.
With this paper, we give new, significant, thrust to the above findings. Specifically, we present a meta-algorithm, called Pick-to-Learn (P2L), that incorporates any learning algorithm given as a black box and makes it into a compression scheme, so licensing the use of results from compression to establish generalization bounds in virtually any learning problem. More precisely, the meta-algorithm constructs a loop around the original learning algorithm that elects at each iteration a new data point to be included in the set of those used for training. Independently of the inner algorithm (which can be any algorithm, e.g., GD - Gradient Descent - for regression), we show that the meta-algorithm possesses the properties of preference, non-associativity and non-concentrated mass by which, as mentioned above, one can secure extremely tight bounds on the change of compression. By further linking the change of compression to the probability of misclassification (or misprediction, in regression problems), powerful and informative generalization bounds are obtained without requiring the use of test-sets. When using our approach on various setups, of which we here present the MNIST classification problem and a synthetic regression problem, we find that P2L returns bounds that are comparable, or superior, to those attained via test-set and PAC-Bayes bounds, while it learns models with better post-training performance.
The idea and the theoretical apparatus behind P2L is presented for the first time in this paper, which also provides a complete set of proofs (given in the appendices) to establish the ensuing generalization results. On the other hand, it has to be said that the usage of the meta-algorithm P2L requires the user to choose a rule by which a new data point is selected at each iteration of the external loop, a choice that is problem-dependent and cannot be part of the general theory. Therefore, the present contribution lays the groundwork of a general new methodology and we expect that this methodology will thrive in the coming few years by adapting it to various specific contexts.
While our demonstrative focus in this paper is on classification and regression, P2L bears a promise of applicability to any data-driven learning problem, including large-scale constrained optimization. Interestingly, in recent years, selection rules to iteratively find "core" examples in the training dataset have been studied in various learning problems, see, e.g., (Toneva et al., 2018; Paul et al., 2021; Yang et al., 2022). These works differ from the present contribution both in their motivations (pruning the training dataset is motivated by computational issues) and results (none of these works provide a compression scheme according to the classical definition and certainly they do not enforce any preference property). Nonetheless, these selection rules can be adopted in the external loop of P2L
<sup>1</sup>The notion of preference was known in the machine learning literature before (Campi & Garatti, 2023) where it is often referred to as stability. Under stability, previous contributions, e.g., (Bousquet et al., 2020; Hanneke & Kontorovich, 2021), have provided valuable generalization results and the approach we propose may as well build upon these results; however, the bounds in (Campi & Garatti, 2023) largely improve over all previous findings.
and we envisage a synergy of our results with these methods. This opens exciting perspectives for future research.
Structure of the paper. After introducing the mathematical preliminaries (Section 2), we present the meta-algorithm P2L (Section 3) and its generalization results (Section 4). For the sake of generality, these sections adopt an abstract viewpoint that accommodates various learning frameworks. Applications to MNIST classification and synthetic regression can be found in Sections 5 and 6. Appendix A contains the proof of the main result and various additional results, including an extended version of P2L. Appendix B instead includes some implementation details and additional material for the MNIST application. Finally, Appendix C provides a further numerical experimentation to study the interpretability of the results of P2L. The source code necessary to reproduce all our numerical results can be found in the Supplementary Material, and at at https://github.com/dario-p/P2L.
We use  $z$  to denote an example, which is an element from a generic set  $\mathcal{Z}$ . For instance, in supervised classification,  $z$  is an input-label pair  $(x,y)$  and  $\mathcal{Z} = \mathcal{X} \times \{1,2,\dots,M\}$ ; instead, in supervised regression,  $\mathcal{Z} = \mathcal{X} \times \mathbb{R}$ . We assume to have access to  $N$  examples, which we collect in a dataset  $D$ ; that is,  $D = \{z_{1},z_{2},\ldots,z_{N}\}$ . Throughout, we model  $D$  as a multiset. This is motivated by the fact that P2L's output will not depend on the order of appearance of the examples but it will account for (possibly) repeated observations. With multisets, the set operations  $\cup, \cap, \setminus, \subseteq$  extend from their definitions for sets in an obvious way. All multisets encountered in our mathematical derivations have a finite number of elements and  $|\cdot|$  denotes the cardinality where each element is counted as many times as its multiplicity. The multiset of examples  $D$  is modeled as a realization of  $D = \{z_{1},z_{2},\dots,z_{N}\}$ , where  $z_{1},z_{2},\dots,z_{N}$  are independent and identically distributed (i.i.d.) random elements taking value in  $\mathcal{Z}$  and defined over a probability space  $(\Omega, \mathcal{F}, \mathbb{P})$ .
We wish to utilize  $D$  to construct a hypothesis from a hypothesis space  $\mathcal{H}$ , which can be any generic space. For example, if we are tasked with making predictions on the label/output of previously unseen inputs,  $\mathcal{H}$  can be the space of classifiers/predictors obtained by suitably parameterized neural networks. We assume to be given a learning algorithm  $L$  that maps a list of examples to a hypothesis (this is the inner black-box of P2L). Unlike multisets, lists come with a positional order of their elements and our approach works rigorously with learning algorithms  $L$  whose returned hypothesis either depends on this positional order or does not. Considering a list (as opposed to a multiset) as input to  $L$  gives a setup that embraces the most general perspective. This is important as many commonly employed algorithms, e.g., SGD for the training of neural networks, are of the first type.
3 The Meta-Algorithm P2L
As anticipated, our goal is that of utilizing a given learning algorithm  $L$  as a building block to construct a meta-algorithm (P2L) that induces a compression scheme with desired properties, regardless of whether the initial learning algorithm has such properties. For this, we need two ingredients: (i) an initial hypothesis  $h_0 \in \mathcal{H}$ ; (ii) a suitable criterion of appropriateness, and a corresponding appropriateness threshold. The criterion of appropriateness quantifies to what extent a given hypothesis  $h$  is appropriate for an example  $z \in \mathcal{Z}$ , while the threshold is used to assess – in the form of a yes/no answer – whether the given hypothesis  $h$  is deemed sufficiently appropriate for  $z$ . We formalize the idea of criterion of appropriateness, and the corresponding threshold, by introducing a hypothesis-dependent total order  $\leq_h$  over the extended set  $\mathcal{Z}_{\mathrm{S}} = \mathcal{Z} \cup \{\mathrm{Stop}\}$ , where Stop is an external element that is added to  $\mathcal{Z}$  for algorithmic reasons. Given a hypothesis  $h$  (our meta-algorithm iterates over subsequent choices of  $h$  before exiting and  $\leq_h$  will be used at each iteration with the current  $h$ ),  $\leq_h$  orders the elements in  $\mathcal{Z}$  and, in particular, allows us to determine the element that
2This means that, e.g.,  $\{1,1,2\}$  is the same as  $\{1,2,1\}$ , but both are different from  $\{1,2\}$  because in a multiset, like in a set, there is no concept of order of the elements but account is taken of their multiplicity.
3This is easily achieved using the multiplicity function  $\mu_U$  for a multiset  $U$ , which counts how many times each element of  $\mathcal{Z}$  occurs in  $U$ . Then,  $\mu_{U \cup U'}(z) = \mu_U(z) + \mu_{U'}(z)$ ,  $\mu_{U \cap U'}(z) = \min \left\{\mu_U(z), \mu_{U'}(z)\right\}$ , and  $\mu_{U \setminus U'}(z) = \max \left\{0, \mu_U(z) - \mu_{U'}(z)\right\}$ . Finally,  $U \subseteq U'$  iff  $\mu_U(z) \leq \mu_{U'}(z)$  for all  $z$ .
4 Throughout, boldface denotes random quantities and we tacitly assume that all random quantities that we introduce are measurable.
is the least appropriate. The outside element  $\text{Stop}$  is instead used to model the appropriateness threshold:  $h$  is "enough appropriate" for  $z \in \mathcal{Z}$  if  $z \leq_h \text{Stop}$  while it is not appropriate enough if  $\text{Stop} \le_h z$ . Note that the two conditions cannot be satisfied simultaneously because  $z \neq \text{Stop}$  (recall that  $\text{Stop}$  is an outside element) and  $\le_h$  is a total order. While remarking the generality of this setting, we provide two examples for concreteness and clarity of presentation.
Example 3.1 (Classification). Consider a binary classification problem where  $z = (x, y)$  with  $x \in \mathcal{X}$  and  $y \in \{0, 1\}$ . Commonly employed hypotheses (e.g., neural networks) return the probability that a feature vector is mapped to either of the two classes. A possible criterion measuring the appropriateness of  $h$  for  $z$  is given by the cross-entropy between the label  $y$  and the probability distribution returned by  $h$  in correspondence of  $x$ , (Bishop & Nasrabadi, 2006)[Sec. 4.3.4]. Then, the appropriateness threshold can be used to certify if  $z$  is correctly classified by  $h$  by assessing if the cross-entropy is above or below the threshold  $-\ln(0.5)$ . This setting is captured by the following total order: for  $z_1, z_2 \in \mathcal{Z}$ , we define  $z_1 \leq_h z_2$  if the cross-entropy of  $z_1$  is no bigger than the cross-entropy of  $z_2$  (and ties are broken according to any given rule), while  $\text{Stop} \leq_h z$  if  $z$  has cross-entropy larger than  $-\ln(0.5)$  and  $z \leq_h \text{Stop}$  otherwise.
Example 3.2 (Regression). In regression,  $z = (x,y)$  with  $x \in \mathcal{X}$  and, for example,  $y \in \mathbb{R}$ . In this setting, a hypothesis is typically a predictor that maps  $x$  into an estimate  $\hat{y}(x)$  for  $y$ . A possible criterion measuring the appropriateness of  $h$  for  $z$  is given by  $|y - \hat{y}(x)|$ , while the appropriateness threshold can be specified by requiring that  $|y - \hat{y}(x)| \leq \gamma$  for a given  $\gamma$  for which the predictions are sufficiently informative. In this case, the hypothesis-dependent total order can be defined as follows. For  $z_1, z_2 \in \mathcal{Z}$ ,  $z_1 \leq_h z_2$  when  $|y_1 - \hat{y}(x_1)| \leq |y_2 - \hat{y}(x_2)|$  (when  $|y_1 - \hat{y}(x_1)| = |y_2 - \hat{y}(x_2)|$  but  $z_1 \neq z_2$  the tie can be broken according to any given rule); otherwise, Stop  $\leq_h z$  when  $|y - \hat{y}(x)| > \gamma$  and  $z \leq_h$  Stop when  $|y - \hat{y}(x)| \leq \gamma$ .
We are now ready to introduce the meta-algorithm P2L (Algorithm 1). P2L works by iteratively feeding the learning algorithm  $L$  with a growing list of training examples taken from  $D$ , while terminating when the current hypothesis is deemed sufficiently appropriate (as assessed by the appropriateness threshold) for all the remaining examples that have not been used for training. If this is not the case, P2L appends to the current training examples the example among those not yet used for which the current hypothesis is least appropriate (according to the criterion of appropriateness). P2L returns both the multiset  $T$  of the employed training examples and the final hypothesis  $h$ .
Formally, given  $(L,h_0,\leq h)$ , the meta-algorithm P2L is a map from a dataset  $D$  to two objects: an hypothesis  $h\in \mathcal{H}$  and a multiset  $T\subseteq D$ . We denote this with  $(h,T) = \mathcal{A}(D)$ . P2L is composed of an initialization, and a main loop where three steps are performed. See Algorithm 1, where  $D_{\mathrm{S}} = D\cup \{\mathrm{Stop}\}$  and  $[T]_{\mathcal{A}}$  is the list of the elements in  $T$  with positional order of each element corresponding to the iteration in which that element is selected by P2L.
Algorithm 1  $\mathcal{A}(D)$  - The meta-algorithm P2L
1: Initialize:  $T = \emptyset$ ,  $h = h_0$ ,  $\bar{z} = \max_{h_0}(D_{\mathbf{S}})$   
2: while  $\bar{z} \neq \text{Stop do}$   
3:  $T \gets T \cup \{\bar{z}\}$   
4:  $h \gets L([T]_{\mathcal{A}})$   
5:  $\bar{z} \gets \max_{h}(D_{\mathbf{S}} \setminus T)$   
6: end while  
7: return  $h, T \triangleright$  Hypothesis  $h$  and multiset  $T$
In the initialization, we let the multiset of training examples be empty, the hypothesis  $h$  be  $h_0$ , and compute the maximal element in  $D_{\mathbb{S}} = D \cup \{\text{Stop}\}$  according to  $\leq h_0$ , which we denote with  $\bar{z} = \max_{h_0}(D_{\mathbb{S}})^5$ . In the main loop, we first check if the maximal element is Stop. If so, the meta-algorithm terminates (thus our naming it Stop). Else, the training multiset is augmented with  $\bar{z}$ . We then produce a new hypothesis by running the learning algorithm on  $[T]_{\mathcal{A}}$ , i.e., we set  $h \gets L([T]_{\mathcal{A}})$ . Finally, we compute the maximal element in  $D_{\mathbb{S}} \setminus T$  according to the new hypothesis  $h$ , that is,  $\bar{z} \gets \max_h(D_{\mathbb{S}} \setminus T)$ , and repeat. When the meta-algorithm terminates, it returns both the hypothesis  $h$ , and the multiset  $T$ .
We conclude with two important remarks. First, note that the choice of the learning algorithm  $L$ , ordering  $\leq h$ , and initial hypothesis  $h_0$  are arbitrary. Interestingly, strong generalization guarantees can be secured at this high level of abstraction (Theorem 4.2) and this allows one to tackle multiple learning problems, including both classification and regression (Sections 5 and 6). Second, observe that the generalization guarantees we will provide rely crucially on the fact that the meta-algorithm
Recall that the maximal element is unique.
P2L possesses a key feature: P2L compresses the multiset  $D$  into the multiset  $T$ , and applying P2L to  $D$  or to  $T$  reconstructs the same output.
4 Statistical Risk and Main Result
In this section we provide formal generalization guarantees on the performance of the hypothesis obtained by running the meta-algorithm P2L. Towards this goal, we introduce the notion of statistical risk, which we will then bound.
Definition 4.1 (Statistical Risk). The statistical risk of a given hypothesis  $h \in \mathcal{H}$  is  $R(h) = \mathbb{P}\{\mathrm{Stop} \leq_h z\}$ , where  $z$  is a random example independent of and distributed as each  $z_i$ .
Informally, the statistical risk of a hypothesis  $h$  measures the probability that  $h$  is not appropriate for a new, out-of-sample, example  $z$ . Given the generality of  $\leq_h$ , this notion can embody various specifications. For instance, in the context of Example 3.1 the statistical risk coincides with the probability of misclassification; in the context of Example 3.2 with the probability of misprediction above a level  $\gamma$ .
In the result stated below,  $\pmb{D}$  is seen as a random element (with an unknown probability distribution) so as to capture the variability in the dataset. Correspondingly,  $\pmb{h}$  is a random element and the risk  $R(\pmb{h})$  is a random variable. Our objective is to show that  $R(\pmb{h})$  can be assessed based on the compression level achieved by P2L, i.e., on the cardinality of  $\pmb{T}$ . We aim to make a statement of the form "With high probability  $1 - \delta$  with respect to the  $N$  i.i.d. draws generating the dataset (i.e., with respect to the unknown probability distribution of  $\pmb{D}$ ), the statistical risk is upper bounded by  $\overline{\varepsilon}(|\pmb{T}|, \delta)$ , for a suitable choice of  $\overline{\varepsilon}(\cdot, \cdot)$ . Towards defining  $\overline{\varepsilon}(\cdot, \cdot)$ , we need to introduce the following function  $\Psi_{k,\delta}: (-\infty, 1) \to \mathbb{R}$  indexed by  $k = 0, 1, \ldots, N$  and by the confidence parameter  $\delta \in (0, 1)$ :
$$
\Psi_ {k, \delta} (\varepsilon) = \frac {\delta}{2 N} \sum_ {m = k} ^ {N - 1} \frac {\binom {m} {k}}{\binom {N} {k}} (1 - \varepsilon) ^ {- (N - m)} + \frac {\delta}{6 N} \sum_ {m = N + 1} ^ {4 N} \frac {\binom {m} {k}}{\binom {N} {k}} (1 - \varepsilon) ^ {m - N},
$$
where the first summand evaluates to zero when  $k = N$ . We are now ready to state our main result.
Theorem 4.2 (Bound on Statistical Risk). Let  $(\pmb{h},\pmb{T}) = \mathcal{A}(\pmb{D})$ , the output of  $P2L$ . For any  $\delta \in (0,1)$  it holds that
$$
\mathbb {P} \{R (\boldsymbol {h}) \leq \bar {\varepsilon} (| \boldsymbol {T} |, \delta) \} \geq 1 - \delta , \tag {1}
$$
where, for  $k = 0,1,\dots ,N - 1$ ,  $\overline{\varepsilon} (k,\delta)$  is the unique solution to the equation  $\Psi_{k,\delta}(\varepsilon) = 1$  in the interval  $[k / N,1]$ , while  $\overline{\varepsilon} (N,\delta) = 1$ .
See Appendix A for a proof. Theorem 4.2 unveils a deep connection between the level of compression achieved by  $\mathcal{A}$ , as measured by the cardinality  $|T|$ , and the generalization level of the hypothesis. Quantitatively, the link is dictated by  $\overline{\varepsilon}(k, \delta)$  whose trend is exemplified in Figure 1 for different values of  $\delta$ .
Three important observations are in order. First, higher compression, i.e., a lower  $|T|$ , corresponds to better generalization guarantees. Second, the dependence of  $\overline{\varepsilon}(k,\delta)$  on  $\delta$  is provably logarithmic. Hence, selecting a very small value of  $\delta$ , say  $10^{-6}$ , i.e., asking for  $R(\pmb{h}) \leq \overline{\varepsilon}(|\pmb{T}|,\delta)$  to hold with high confidence, say 0.999999, in practice with certainty, has little cost. Third, we notice that the bound can be applied without knowledge of the probability distribution by which data are generated (distribution-free result).
Figure 1: Function  $\bar{\varepsilon}(k, \delta)$ ,  $\delta = 10^{-6}$ ,  $10^{-4}$ ,  $10^{-2}$ , (top to bottom).  $N = 500$ . The dotted line is  $k / N$ .
5 Application to MNIST Classification
In this section we apply P2L to the MNIST digit recognition problem. Our goal is to compare the posttraining performance and generalization bounds obtained with an implementation of P2L, PAC-Bayes and test-set approaches. The latter two are known to give the best generalization bounds to date.