Title: The Pick-to-Learn Algorithm: Empowering Compression for Tight Generalization Bounds and Improved Post-training Performance
Venue: NeurIPS 2023
Accepted: yes
Keywords: Statistical learning theory

Compression theory

Generalization bounds
Abstract: Generalization bounds are valuable both for theory and applications. On the one hand, they shed light on the mechanisms that underpin the learning processes; on the other, they certify how well a learned model performs against unseen inputs.  In this work we build upon a recent breakthrough in compression theory to develop a new framework yielding tight generalization bounds of wide practical applicability.  The core idea is to embed any given learning algorithm into a suitably-constructed meta-algorithm (here called Pick-to-Learn, P2L) in order to instill desirable compression properties. When applied to the MNIST classification dataset and to a synthetic regression problem, P2L not only attains generalization bounds that compare favorably with the state of the art (test-set and PAC-Bayes bounds), but it also learns models with better post-training performance.

The Pick-to-Learn Algorithm: Empowering Compression for Tight Generalization Bounds and Improved Post-training Performance

Abstract

Generalization bounds are valuable both for theory and applications. On the one hand, they shed light on the mechanisms that underpin the learning processes; on the other, they certify how well a learned model performs against unseen inputs. In this work we build upon a recent breakthrough in compression theory (Campi & Garatti, 2023) to develop a new framework yielding tight generalization bounds of wide practical applicability. The core idea is to embed any given learning algorithm into a suitably-constructed meta-algorithm (here called Pick-to-Learn, P2L) in order to instill desirable compression properties. When applied to the MNIST classification dataset and to a synthetic regression problem, P2L not only attains generalization bounds that compare favorably with the state of the art (test-set and PAC-Bayes bounds), but it also learns models with better post-training performance.

1 Introduction

Machine learning has achieved remarkable results in the last decades, with successful stories ranging from digit recognition (Dosovitskiy et al., 2020), to protein folding (Jumper et al., 2021), traffic prediction (Li et al., 2017), medical diagnosis (Kononenko, 2001), and beyond. Contrary to that, the understanding of the mechanisms underpinning generalization, and the availability of bounds for its evaluation, is still limited. This is unfortunate because generalization bounds are important both to develop trust in learning methods as well as to allow for a fair comparison among alternatives, which is relevant to hyper-parameter tuning.

Among the various tools that have been introduced to provide generalization bounds, we recall the VCdimension (Vapnik & Chervonenkis, 1971), Radamacher complexity (Bartlett & Mendelson, 2002), mutual information (Xu & Raginsky, 2017), sharpness (Keskar et al., 2017), compression schemes (Littlestone & Warmuth, 1986; Floyd & Warmuth, 1995; Graepel et al., 2005), and PAC-Bayes approaches (Dziugaite & Roy, 2017; Perez-Ortiz et al., 2021). Each of these frameworks is applicable to specific contexts. For example, the approach based on the VC-theory requires that a model is selected from a class with finite VC-dimension, while compression schemes are applicable provided the existence of an informative subset of data points from which the model can be reconstructed. It has also to be said that the level of precision of the available bounds is highly problem-dependent and they often take a significant margin from the actual performance.

While bounds based on the VC-dimension, and some other aforementioned approaches, are often conservative (resulting in untight, or even vacuous, evaluations), the sharpest approach to evaluate generalization across various learning domains is still that based on test-set bounds (Langford, 2005). However, the test-set approach requires holding out a portion of the training set for testing, which makes it data-inefficient with negative impact on the post-training performance. In view of these considerations, it would be highly desirable to develop a new general-purpose technique able to set the data free in their double role of (i) delivering the information that is needed to learn , while also (ii) providing evaluations on the attained generalization result .

In this paper, we move towards the goal expressed in (i) and (ii). To this aim, we build upon a recent breakthrough in the field of compression schemes, (Campi & Garatti, 2023). Therein, the authors have established new and tight bounds for the so-called probability of change of compression that hold under certain properties: under the property of preference , tight upper bounds have been obtained that link the probability of change of compression to the size of the compressed set 1 ; moreover, under a so-called non-associativity property and a property of non-concentrated mass , a lower bound has also been obtained. Hence, under these additional properties, the change of compression is put in sandwich between two bounds, which are shown to rapidly converge one on top of the other as the number of data points increases. The interest of these results in the context of statistical learning lies on the fact that some learning algorithms naturally define a compression scheme and the probability of change of compression can be used to bound the probability of misclassification/misprediction. However, the challenge is that many other learning algorithms (including most algorithms for deep learning) do not exhibit any compression property.

With this paper, we give new, significant, thrust to the above findings. Specifically, we present a meta-algorithm , called Pick-to-Learn (P2L), that incorporates any learning algorithm given as a black box and makes it into a compression scheme, so licensing the use of results from compression to establish generalization bounds in virtually any learning problem. More precisely, the meta-algorithm constructs a loop around the original learning algorithm that elects at each iteration a new data point to be included in the set of those used for training. Independently of the inner algorithm (which can be any algorithm, e.g., GD - Gradient Descent - for regression), we show that the metaalgorithm possesses the properties of preference , non-associativity and non-concentrated mass by which, as mentioned above, one can secure extremely tight bounds on the change of compression. By further linking the change of compression to the probability of misclassification (or misprediction, in regression problems), powerful and informative generalization bounds are obtained without requiring the use of test-sets. When using our approach on various setups, of which we here present the MNIST classification problem and a synthetic regression problem, we find that P2L returns bounds that are comparable, or superior, to those attained via test-set and PAC-Bayes bounds, while it learns models with better post-training performance.

The idea and the theoretical apparatus behind P2L is presented for the first time in this paper, which also provides a complete set of proofs (given in the appendices) to establish the ensuing generalization results. On the other hand, it has to be said that the usage of the meta-algorithm P2L requires the user to choose a rule by which a new data point is selected at each iteration of the external loop, a choice that is problem-dependent and cannot be part of the general theory. Therefore, the present contribution lays the groundwork of a general new methodology and we expect that this methodology will thrive in the coming few years by adapting it to various specific contexts.

While our demonstrative focus in this paper is on classification and regression, P2L bears a promise of applicability to any data-driven learning problem, including large-scale constrained optimization. Interestingly, in recent years, selection rules to iteratively find 'core' examples in the training dataset have been studied in various learning problems, see, e.g., (Toneva et al., 2018; Paul et al., 2021; Yang et al., 2022). These works differ from the present contribution both in their motivations (pruning the training dataset is motivated by computational issues) and results (none of these works provide a compression scheme according to the classical definition and certainly they do not enforce any preference property). Nonetheless, these selection rules can be adopted in the external loop of P2L

and we envisage a synergy of our results with these methods. This opens exciting perspectives for future research.

Structure of the paper. After introducing the mathematical preliminaries (Section 2), we present the meta-algorithm P2L (Section 3) and its generalization results (Section 4). For the sake of generality, these sections adopt an abstract viewpoint that accommodates various learning frameworks. Applications to MNIST classification and synthetic regression can be found in Sections 5 and 6. Appendix A contains the proof of the main result and various additional results, including an extended version of P2L. Appendix B instead includes some implementation details and additional material for the MNIST application. Finally, Appendix C provides a further numerical experimentation to study the interpretability of the results of P2L. The source code necessary to reproduce all our numerical results can be found in the Supplementary Material, and at at https://github.com/dario-p/P2L .

2 Mathematical Preliminaries

We use z to denote an example, which is an element from a generic set Z . For instance, in supervised classification, z is an input-label pair ( x, y ) and Z = X × { 1 , 2 , . . . , M } ; instead, in supervised regression, Z = X × R . We assume to have access to N examples, which we collect in a dataset D ; that is, D = { z 1 , z 2 , . . . , z N } . Throughout, we model D as a multiset . 2 This is motivated by the fact that P2L's output will not depend on the order of appearance of the examples but it will account for (possibly) repeated observations. With multisets, the set operations ∪ , ∩ , \ , ⊆ extend from their definitions for sets in an obvious way. 3 All multisets encountered in our mathematical derivations have a finite number of elements and | · | denotes the cardinality where each element is counted as many times as is its multiplicity. The multiset of examples D is modeled as a realization of D = { z 1 , z 2 , . . . , z N } , where z 1 , z 2 , . . . , z N are independent and identically distributed (i.i.d.) random elements taking value in Z and defined over a probability space (Ω , F , P ) . 4

We wish to utilize D to construct a hypothesis from a hypothesis space H , which can be any generic space. For example, if we are tasked with making predictions on the label/output of previously unseen inputs, H can be the space of classifiers/predictors obtained by suitably parameterized neural networks. We assume to be given a learning algorithm L that maps a list of examples to a hypothesis (this is the inner black-box of P2L). Unlike multisets, lists come with a positional order of their elements and our approach works rigorously with learning algorithms L whose returned hypothesis either depends on this positional order or does not. Considering a list (as opposed to a multiset) as input to L gives a setup that embraces the most general perspective. This is important as many commonly employed algorithms, e.g., SGD for the training of neural networks, are of the first type.

3 The Meta-Algorithm P2L

As anticipated, our goal is that of utilizing a given learning algorithm L as a building block to construct a meta-algorithm (P2L) that induces a compression scheme with desired properties, regardless of whether the initial learning algorithm has such properties. For this, we need two ingredients: (i) an initial hypothesis h 0 ∈ H ; (ii) a suitable criterion of appropriateness , and a corresponding appropriateness threshold . The criterion of appropriateness quantifies to what extent a given hypothesis h is appropriate for an example z ∈ Z , while the threshold is used to assess - in the form of a yes/no answer whether the given hypothesis h is deemed sufficiently appropriate for z . We formalize the idea of criterion of appropriateness, and the corresponding threshold, by introducing a hypothesis-dependent total order ≤ h over the extended set Z S = Z ∪ { Stop } , where Stop is an external element that is added to Z for algorithmic reasons. Given a hypothesis h (our meta-algorithm iterates over subsequent choices of h before exiting and ≤ h will be used at each iteration with the current h ), ≤ h orders the elements in Z and, in particular, allows us to determine the element that

glyph[negationslash]

is the least appropriate. The outside element Stop is instead used to model the appropriateness threshold: h is 'enough appropriate' for z ∈ Z if z ≤ h Stop while it is not appropriate enough if Stop ≤ h z . Note that the two conditions cannot be satisfied simultaneously because z = Stop (recall that Stop is an outside element) and ≤ h is a total order. While remarking the generality of this setting, we provide two examples for concreteness and clarity of presentation.

Example 3.1 (Classification) . Consider a binary classification problem where z = ( x, y ) with x ∈ X and y ∈ { 0 , 1 } . Commonly employed hypotheses (e.g., neural networks) return the probability that a feature vector is mapped to either of the two classes. A possible criterion measuring the appropriateness of h for z is given by the cross-entropy between the label y and the probability distribution returned by h in correspondence of x , (Bishop & Nasrabadi, 2006)[Sec. 4.3.4]. Then, the appropriateness threshold can be used to certify if z is correctly classified by h by assessing if the cross-entropy is above or below the threshold -ln(0 . 5) . This setting is captured by the following total order: for z 1 , z 2 ∈ Z , we define z 1 ≤ h z 2 if the cross-entropy of z 1 is no bigger than the cross-entropy of z 2 (and ties are broken according to any given rule), while Stop ≤ h z if z has cross-entropy larger than -ln(0 . 5) and z ≤ h Stop otherwise.

glyph[negationslash]

Example 3.2 (Regression) . In regression, z = ( x, y ) with x ∈ X and, for example, y ∈ R . In this setting, a hypothesis is typically a predictor that maps x into an estimate ˆ y ( x ) for y . A possible criterion measuring the appropriateness of h for z is given by | y -ˆ y ( x ) | , while the appropriateness threshold can be specified by requiring that | y -ˆ y ( x ) | ≤ γ for a given γ for which the predictions are sufficiently informative. In this case, the hypothesis-dependent total order can be defined as follows. For z 1 , z 2 ∈ Z , z 1 ≤ h z 2 when | y 1 -ˆ y ( x 1 ) | ≤ | y 2 -ˆ y ( x 2 ) | (when | y 1 -ˆ y ( x 1 ) | = | y 2 -ˆ y ( x 2 ) | but z 1 = z 2 the tie can be broken according to any given rule); otherwise, Stop ≤ h z when | y -ˆ y ( x ) | > γ and z ≤ h Stop when | y -ˆ y ( x ) | ≤ γ .

We are now ready to introduce the meta-algorithm P2L (Algorithm 1). P2L works by iteratively feeding the learning algorithm L with a growing list of training examples taken from D , while terminating when the current hypothesis is deemed sufficiently appropriate (as assessed by the appropriateness threshold) for all the remaining examples that have not been used for training. If this is not the case, P2L appends to the current training examples the example among those not yet used for which the current hypothesis is least appropriate (according to the criterion of appropriateness). P2L returns both the multiset T of the employed training examples and the final hypothesis h .

Formally, given ( L, h 0 , ≤ h ) , the metaalgorithm P2L is a map from a dataset D to two objects: an hypothesis h ∈ H , and a multiset T ⊆ D . We denote this with ( h, T ) = A ( D ) . P2L is composed of an initialization, and a main loop where three steps are performed. See Algorithm 1, where D S = D ∪ { Stop } and [ T ] A is the list of the elements in T with positional order of each element corresponding to the iteration in which that element is selected by P2L.

glyph[negationslash]

In the initialization, we let the multiset of training examples be empty, the hypothesis h be h 0 , and compute the maximal element in D S = D ∪ { Stop } according to ≤ h 0 , which we denote with ¯ z = max h 0 ( D S ) . 5 In the main loop, we first check if the maximal element is Stop . If so, the metaalgorithm terminates (thus our naming it Stop ). Else, the training multiset is augmented with ¯ z . We then produce a new hypothesis by running the learning algorithm on [ T ] A , i.e., we set h ← L ([ T ] A ) . Finally, we compute the maximal element in D S \ T according to the new hypothesis h , that is, ¯ z ← max h ( D S \ T ) , and repeat. When the meta-algorithm terminates, it returns both the hypothesis h , and the multiset T .

We conclude with two important remarks. First, note that the choice of the learning algorithm L , ordering ≤ h , and initial hypothesis h 0 are arbitrary. Interestingly, strong generalization guarantees can be secured at this high level of abstraction (Theorem 4.2) and this allows one to tackle multiple learning problems, including both classification and regression (Sections 5 and 6). Second, observe that the generalization guarantees we will provide rely crucially on the fact that the meta-algorithm

P2L possesses a key feature: P2L compresses the multiset D into the multiset T , and applying P2L to D or to T reconstructs the same output.

4 Statistical Risk and Main Result

In this section we provide formal generalization guarantees on the performance of the hypothesis obtained by running the meta-algorithm P2L. Towards this goal, we introduce the notion of statistical risk, which we will then bound.

Definition 4.1 (Statistical Risk) . The statistical risk of a given hypothesis h ∈ H is R ( h ) = P { Stop ≤ h z } , where z is a random example independent of and distributed as each z i .

Informally, the statistical risk of a hypothesis h measures the probability that h is not appropriate for a new, out-of-sample, example z . Given the generality of ≤ h , this notion can embody various specifications. For instance, in the context of Example 3.1 the statistical risk coincides with the probability of misclassification; in the context of Example 3.2 with the probability of misprediction above a level γ .

In the result stated below, D is seen as a random element (with an unknown probability distribution) so as to capture the variability in the dataset. Correspondingly, h is a random element and the risk R ( h ) is a random variable. Our objective is to show that R ( h ) can be assessed based on the compression level achieved by P2L, i.e., on the cardinality of T . We aim to make a statement of the form 'With high probability 1 -δ with respect to the N i.i.d. draws generating the dataset (i.e, with respect to the unknown probability distribution of D ), the statistical risk is upper bounded by ε ( | T | , δ ) ', for a suitable choice of ε ( · , · ) . Towards defining ε ( · , · ) , we need to introduce the following function Ψ k,δ : ( -∞ , 1) → R indexed by k = 0 , 1 , . . . , N and by the confidence parameter δ ∈ (0 , 1) :

where the first summand evaluates to zero when k = N . We are now ready to state our main result. Theorem 4.2 (Bound on Statistical Risk) . Let ( h , T ) = A ( D ) , the output of P2L. For any δ ∈ (0 , 1) it holds that

where, for k = 0 , 1 , . . . , N -1 , ε ( k, δ ) is the unique solution to the equation Ψ k,δ ( ε ) = 1 in the interval [ k/N, 1] , while ε ( N,δ ) = 1 .

See Appendix A for a proof. Theorem 4.2 unveils a deep connection between the level of compression achieved by A , as measured by the cardinality | T | , and the generalization level of the hypothesis. Quantitatively, the link is dictated by ε ( k, δ ) , whose trend is exemplified in Figure 1 for different values of δ .

Three important observations are in order. First, higher compression, i.e., a lower | T | , corresponds to better generalization guarantees. Second, the dependence of ε ( k, δ ) on δ is provably logarithmic. Hence, selecting a very small value of δ , say 10 -6 , i.e., asking for R ( h ) ≤ ε ( | T | , δ ) to hold with high confidence, say 0.999999, in practice with certainty , has little cost. Third, we notice that the bound can be applied without knowledge of the probability distribution by which data are generated ( distribution-free result).

5 Application to MNIST Classification

In this section we apply P2L to the MNIST digit recognition problem. Our goal is to compare the posttraining performance and generalization bounds obtained with an implementation of P2L, PAC-Bayes and test-set approaches. The latter two are known to give the best generalization bounds to date.

0

100

200

300

400

500

0

0

.

2

0

.

4

0

.

6

0

.

8

1

k

ε

More precisely, we consider a binary version of the MNIST dataset introduced in the seminal work of (Dziugaite & Roy, 2017) for the specific purpose of comparing the generalization guarantees of PACBayes and other approaches, and later employed in, e.g., (Rivasplata et al., 2019). In this problem, the digits 0-4 and 5-9 are mapped to the labels 0 and 1. To classify the inputs, we employ a fully connected feed-forward neural network with three hidden layers each with 600 nodes and ReLu activation functions. The input has 784 nodes and the output has two nodes which are passed to a softmax function. This architecture, employed in the above-cited works, is used across all our experiments.

0

.

2

0

.

4

0

.

6

0

.

8

0

0

.

1

0

.

2

0

.

3

Train/pretrain portion

Risk

δ

= 0

.

035

0

0

.

1

0

.

2

0

.

3

0

0

.

1

0

.

2

0

.

3

Bound on risk

Risk on the test dataset

0

.

2

0

.

4

0

.

6

0

.

8

0

0

.

1

0

.

2

0

.

3

Train/pretrain portion

Risk

δ

= 0

.

001

0

0

.

1

0

.

2

0

.

3

0

0

.

1

0

.

2

0

.

3

Bound on risk

Risk on the test dataset

MNIST consists of a training dataset containing 60000 examples and a test dataset with 10000 examples. In our experiments, we train all models only on 1000 samples from the training dataset at a time. We particularly care of this 'small' dataset setting, in which one has to make the most of the data for both training and assessing the generalization. This is a setting of interest to various fields where data are a limited, possibly costly, resource. 6 Specifically, we shuffle the original MNIST training dataset and extract 60 disjoints datasets with 1000 data points each. All approaches we compare are run on the resulting 60 datasets, which are used to both train the network and provide generalization bounds. The full test dataset, containing 10000 examples, is never used in any training phase. It is instead used to evaluate the actual post-training performance of each trained model. We

now present the specific algorithmic choices made for P2L. Due to space limitations, the details concerning the test-set approach and PAC-Bayes can be found in Appendix B.1.

P2L. As discussed, P2L is fully specified once a learning algorithm L , an initial hypothesis h 0 , and a hypothesis-dependent total ordering ≤ h with a criterion of stop are defined. For the learning algorithm L , we choose the widely employed GD (gradient descent) with momentum, which we run for 200 epochs with a learning rate of 0 . 01 , momentum of 0 . 95 , dropout probability of 0 . 2 . For the sake of clarity, we remark that each time GD is called by P2L, GD iterates until convergence (or until reaching the allowed number of epochs) so as to perform the minimization of the empirical risk. As initialization of GD, we take the h returned at the previous iteration of P2L. 7 Regarding h 0 , it is clear that when starting from an educated guess the choice of the worst example to be inserted in T tends to be more meaningful, allowing P2L to terminate with a smaller T and, thus, a better generalization bound. To this purpose, one can use a portion of each training dataset to pretrain h 0 (using GD again), while P2L and, therefore, the ensuing generalization bound rely on the remaining part of the training dataset. In our computation, we experiment with multiple sizes of the pretraining portion. As for ≤ h , given an hypothesis h , i.e., a neural network, we use the total order induced by the cross-entropy of its output as discussed in Example 3.1 (ties are broken according to the lexicographic order of the examples' bit representation). This choice ensures that the statistical risk of Definition 4.1 corresponds to the probability of misclassification, and thus a bound on the statistical risk coincides with a bound on the misclassification. The final bound we present on the misclassification is that of the main result in Theorem 4.2, where the number of samples equals N minus those used to pretrain the initial hypothesis h 0 .

Experimental results. The results are presented in Figure 2 for different values of the confidence. The top row depicts, the average (over the 60 trials) generalization bounds as well as the average misclassification levels on the test dataset (post-training performances) with their dispersion for the three approaches. The values are plotted as functions of the portion of the dataset (called train/pretrain portion) used to train the the model in GD+test-set, to pretrain the prior distribution in PAC-Bayes, and to pretrain the initial hypothesis h 0 in P2L. The joint distribution of the returned bound and actual post-training misclassification level for the best models (corresponding to train/pretrain portion equal to 0 . 5 , 0 . 6 , 0 . 7 for P2L, PAC-Bayes, and test-set approach, respectively) are shown in the bottom row. The average bounds and misclassification levels for these best models are also presented in Table 1 for δ = 0 . 035 . For the sake of completeness, Table 1 also reports the average running times of one execution (i.e., for one dataset with 1000 examples) of the three approaches (computational resource: Apple MacBook Pro with M1 Pro CPU and 32Gb of ram).

Conclusions. Four important observations are in order. First, it is evident that, in the present application, the implementation of P2L with GD outperforms the PAC-Bayes approach with respect to both the provided upper bound on the risk and the post-training performance on the test dataset. This is true not just for the best model learned with each approach (cfr. the solid diamonds and the solid triangles in Figure 2), but it holds uniformly across all train/pretrain portions we tried (cfr. the green and red curves). Notably, the model returned by P2L+GD when using only 10% of the data to pretrain the initial hypothesis h 0 has a risk bound that is comparable to that of the best PAC-Bayes model (whose prior is trained on 60% of the data) while it achieves a better post-training performance. Second, P2L+GD provides best risk bounds that are comparable to those of GD+test-set approach, albeit slightly inferior ( 0 . 143 vs 0 . 129 for δ = 0 . 035 and 0 . 163 vs 0 . 155 for δ = 0 . 001 ). However, P2L+GD provides better post-training performance uniformly across all train/pretrain portions. As a matter of fact, for any choice of pretrain portion, P2L+GD's performance is equal to that obtained when GD is run on the whole data set ( N = 1000 ), for which the test-set approach cannot provide

any meaningful bound since no data are left for testing the model. This is a clear indication of the important feature that P2L utilizes all data to jointly learn a good model and provide a risk bound. Third, the post-training performance of PAC-Bayes and GD+test-set approaches are similar. This suggests that in PAC-Bayes the training of the posterior does not exploit the additional available data to improve the model, but rather to certify it, in a similar vein as in the test-set approach. This fact has also been observed recently in (Lotfi et al., 2022)[Fig 1(a)]. Fourth, as for the tightness of the bounds (i.e., the difference between the upper bounds and risk on the test dataset), GD+test-set approach provides the tightest results, while P2L+GD and PAC-Bayes approaches alternate depending on the specific train/pretrain portion selected.

6 Application to a synthetic regression problem

In this section we apply our methodology to a synthetic regression problem in order to showcase the flexibility of the approach introduced in Section 4.

Specifically, we consider 100 distinct training datasets with N = 200 examples ( x, y ) and then another 100 distinct training datasets with N = 500 examples ( x, y ) . The input x ∈ [ -0 . 5 , 1 . 5] is extracted uniformly at random and the output is y = f ( x )+ e with f ( x ) = sin(2 . 5 πx ) / (2 . 5 πx ) and e extracted according to a normal distribution with zero mean and standard deviation 0 . 05 , see Figure 3. For each dataset, our objective is twofold: finding a model that fits the data well, and providing a bound on the probability that the model mispredicts the output by more than a threshold γ = 0 . 1 (bounds of this sort matters in relation to many applications).

In pursuing this goal, we compare the generalization bounds and post-training performances attained using P2L and test-set bounds. For illustration purposes, in all our experiments we consider a simple network architecture comprising one input node, one output node, and one hidden layer with six nodes, each equipped with a tanh activation function. To evaluate the post-training performance, we use an additional test dataset with 20000 examples. The value of δ is set to 0 . 035 .

-

0

.

5

0

0

.

5

1

1

.

5

0

0

.

5

1

y

Examples

y

=

f

(

x

)

P2L

Test-set

x

P2L. In deploying P2L we use GD as learning algorithm L , and, similarly to the MNIST example, we experiment with initial hypotheses h 0 trained on different fractions of the training dataset (including the null fraction in which case h 0 is the network with all weights set to 0 ). We fix the same total order ≤ h of Example 3.2, with γ = 0 . 1 . In this context, the risk introduced in Definition 4.1 measures the probability that the output y is mispredicted by more than γ , thus giving us a guarantee on the quality of the model. The generalization bound we use is that in Theorem 4.2, where the number of samples equals N minus those used to pretrain the initial hypothesis h 0 . We use a learning rate of 0 . 1 , momentum of 0 . 95 , 1000 epochs, and no dropout.

Test-set approach. We apportion each dataset in two, with one portion used for training, and the other for deriving the generalization bounds. We run GD for 1000 epochs, perform a grid search over learning rates [0.001, 0.005, 0.01], momentum [0.9, 0.95] (no dropout), and select those giving the best generalization bound. As in the MNIST example (see Appendix B.1), we use the binomial test-set bound of (Langford, 2005)[Thm 3.3] with a number of samples equal to N minus those used for training.

Experimental results. The first two panels of Figure 4 present the average (over the 100 trials) upper bounds on the risk and the average risk on the test dataset jointly with their dispersion, as a function of the data portion used to train the model (GD+test-set) or to pretrain the initial hypothesis (P2L). The third and fourth panels depict the distribution of the upper bound and of the risk on the test dataset

0

0

.

2

0

.

4

0

.

6

0

.

8

0

0

.

1

0

.

2

0

.

3

0

.

4

Train/pretrain portion

Risk

N

= 200

0

0

.

1

0

.

2

0

.

3

0

.

4

0

0

.

1

0

.

2

0

.

3

0

.

4

Bound on risk

Risk on the test dataset

0

.

2

0

.

4

0

.

6

0

.

8

0

0

.

1

0

.

2

0

.

3

0

.

4

Train/pretrain portion

Risk

N

= 500

0

0

.

1

0

.

2

0

.

3

0

.

4

0

0

.

1

0

.

2

0

.

3

0

.

4

Bound on risk

Risk on the test dataset

for the data proportion 0.3, returning the best learned models for both P2L and the test-set approach. Their averages are compared in Table 2, which also includes the average runtime (computational resource: Apple MacBook Pro with M1 Pro CPU and 32Gb of ram).

Conclusions. First, it appears evident that, in this synthetic regression problem, the application of P2L with GD provides superior results to those achieved by GD with the test-set approach, regardless of the size of the datasets used for training. This holds true jointly for the upper bound and the risk on the test dataset uniformly across all train/pretrain portions (cfr. the red and blue dashed lines, similarly for the solid ones). We ascribe this result to the fact that P2L does not set aside data for testing, and yet it also provides rigorous evaluations of the risk. Second, the test-set approach provides bounds that are closer to the risk it incurs on the test dataset for N = 500 , while this effect is less clear with N = 200 . Third, as expected, when the size of the training dataset grows, both the upper bounds and the risk on the test dataset improve.

To conclude, it is fair to notice that GD does not pursue directly the goal of obtaining a prediction error smaller than γ , and this may justify the large gap in the post-training performance between

the two approaches also when almost all the dataset is used for training in the test-set approach. Although GD is used as inner algorithm L in P2L, it seems that a good performance with respect to the chosen appropriateness criterion can be obtained thanks to the structure of the meta-algorithm. This is another interesting feature of P2L.

7 Conclusions, Limitations and Future research

We have proposed a novel framework called P2L to provide virtually any learning algorithm with sharp generalization bounds. Our approach is based on making a given learning algorithm into a compression scheme with desirable properties, thus enabling the use of powerful generalization results. Numerical results show that P2L is capable of learning hypotheses with post-training performances and generalization bounds equal or superior to the state of the art.

Computational aspects. While P2L requires learning a hypothesis over a training set of increasing size, and thus might be less efficient than learning the hypothesis only once over the full dataset, it is important to look at it from the appropriate perspective: we are concerned with settings where data is limited or costly to acquire (see, e.g., Footnote 6 and ensuing discussion) while computations are performed off-line and fast execution does not represent a primary concern (a setup also considered in other recent work, e.g., (Foong et al., 2021)).

Source of conservatism. In our framework, as revealed by the proof of Theorem 4.2, the risk is controlled by bounding the probability of change of compression (as defined in Equation (2) in Appendix A.1). Notably, the mismatch between the risk and the probability of change of compression is the only source of conservatism our approach needs to resolve since the upper and lower bounds on the probability of change of compression from Theorem A.4 are extremely tight - see Remark A.9. The magnitude of this conservatism is determined by the choices we make in specializing the proposed meta-algorithm. For a given learning algorithm L , these choices entail selecting an initial hypothesis h 0 and an hypothesis-dependent total order used to select which data points are fed to the learning algorithm L . Our experimenting with multiple initial hypotheses (trained with different portions of the dataset) was solely geared at reducing this gap. Indeed, after training h 0 , P2L allows the data to 'freely speak', and thus improve the resulting hypothesis, as it can be appreciated from the fact that the misclassification on the test dataset for P2L is constant across all prior/train portions (see Figure 2, top row). This is in stark contrast with the test-set approach and even PAC-Bayes. In the former, data are either used to train the model or to provide a risk bound. In the latter, data employed to train the posterior are effectively used to compute a risk bound as opposed to significantly improving the quality of the prior (see first row in Figure 2). We conclude noting that our choices of the initial hypothesis and total order is but one of many possible. Depending on the specific learning problem, other choices can be made (see for examples the last part of the Introduction). We believe that the overall fact that the theoretical apparatus in our approach clearly identifies the sole source of conservatism will put us and others in the position to build upon the P2L framework beyond this work.

Acknowledgments

This paper was partly supported by FAIR (Future Artificial Intelligence Research) project, funded by the NextGenerationEU program within the PNRR-PE-AI scheme (M4C2, Investment 1.3, Line on Artificial Intelligence); and the EPSRC grant EP/Y001001/1, funded by the International Science Partnerships Fund (ISPF) and UKRI.

References

Bartlett, P. L. and Mendelson, S. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research , 3:463-482, 2002.

Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning , volume 4. Springer, 2006.

Bousquet, O., Hanneke, S., Moran, S., and Zhivotovskiy, N. Proper learning, Helly number, and an optimal SVM bound. In Proceedings of 33rd Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pp. 582-609, 2020.

Campi, M. C. and Garatti, S. Compression, generalization and learning. Journal of Machine Learning Research , 24(339):1-74, 2023.

Clerico, E., Deligiannidis, G., and Doucet, A. Conditionally gaussian PAC-Bayes. In International Conference on Artificial Intelligence and Statistics , pp. 2311-2329. PMLR, 2022.

Cortes, C. and Vapnik, V. Support vector networks. Machine Learning , 20:273-297, 1995.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.

Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008, 2017.

Floyd, S. and Warmuth, M. Sample compression, learnability, and the Vapnik-Chervonenkis dimension. Machine Learning , 21:269-304, 1995.

Foong, A., Bruinsma, W., Burt, D., and Turner, R. How tight can PAC-Bayes be in the small data regime? Advances in Neural Information Processing Systems , 34:4093-4105, 2021.

Graepel, T., Herbrich, R., and J.Shawe-Taylor. PAC-Bayesian compression bounds on the prediction error of learning algorithms for classification. Machine Learning , 59:55-76, 2005.

Hanneke, S. and Kontorovich, A. Stable sample compression schemes: New applications and an optimal svm margin bound. In Proceedings of the 32nd International Conference on Algorithmic Learning Theory , volume 132 of Proceedings of Machine Learning Research , pp. 697-721, 2021.

Jumper, J., Evans, R., Pritzel, A., Tim Green, M. F., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D. Highly accurate protein structure prediction with alphafold. Nature , 596:583-589, 2021.

Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On large-batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the International Conference on Learning Representations (ICLR) , 2017.

Kononenko, I. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in medicine, Elsevier , 1:89-109, 2001.

Langford, J. Tutorial on practical prediction theory for classification. Journal of machine learning research , 6(3), 2005.

Li, Y., Yu, R., Shahabi, C., and Liu, Y . Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. arXiv preprint arXiv:1707.01926, 2017.

Littlestone, N. and Warmuth, M. Relating data compression and learnability. Unpublished manuscript , 1986.

Lotfi, S., Finzi, M., Kapoor, S., Potapczynski, A., Goldblum, M., and Wilson, A. G. PAC-Bayes compression bounds so tight that they can explain generalization. arXiv preprint arXiv:2211.13609 , 2022.

Paul, M., Ganguli, S., and Dziugaite, G. K. Deep learning on a data diet: Finding important examples early in training. Advances in Neural Information Processing Systems , 34:20596-20607, 2021.

Perez-Ortiz, M., Rivasplata, O., Shawe-Taylor, J., and Szepesvari, C. Tighter risk certificates for neural networks. Journal of Machine Learning Research , 22:1-40, 2021.

Rivasplata, O., Tankasali, V. M., and Szepesvári, C. PAC-Bayes with backprop. arXiv preprint arXiv:1908.07380 , 2019.

Smola, A. and Schölkopf, B. A tutorial on support vector regression. Statistics and Computing , 14: 199-224, 2004.

Toneva, M., Sordoni, A., Combes, R. T. d., Trischler, A., Bengio, Y., and Gordon, G. J. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159 , 2018.

Vapnik, V. and Chervonenkis, A. On the uniform convergence of relative frequencies of events to their probabilities. Theory of probability and its applications , 16:264-280, 1971.

Wu, Y.-S. and Seldin, Y. Split-kl and PAC-Bayes-split-kl inequalities for ternary random variables. In Proceedings of the 35th conference on Advances in Neural Information Processing Systems (NeurIPS 2022) , 2022.

Xu, A. and Raginsky, M. Information-theoretic analysis of generalization capability of learning algorithms. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS) , 2017.

Yang, S., Xie, Z., Peng, H., Xu, M., Sun, M., and Li, P. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09329 , 2022.

Appendix

A Proof of Theorem 4.2 and Additional Results

In this section we prove Theorem 4.2. En route to this goal, we will also present additional results that help better position our contribution. Doing so will allow us to highlight directions where we believe there is room for exciting future work. The section is conceptually divided in three portions: in Appendix A.1 we recall a relevant result from (Campi & Garatti, 2023) along with the necessary mathematical background; this result is then leveraged in Appendix A.2 to prove Theorem 4.2 and other related results; finally, in Appendix A.4 we revisit the P2L framework in the light of some technical details revealed by the proof of Theorem 4.2.

A.1 Preferent Compression Gives Tight Risk Bounds

We begin with introducing two properties of a compression function c ( preference and nonassociativity ) and the property of non-concentrated mass from which strong generalization guarantees follow. Recall that a compression function c is a map from any multiset D of elements in Z to a submultiset c ( D ) ⊆ D . 8 Throughout, we use c ( D,z ) as a shorthand for c ( D ∪ { z } ) , and likewise for c ( D,z 1 , . . . , z p ) . Moreover, z denotes a random example independent of and identically distributed as each z i .

glyph[negationslash]

Property A.1 (Preference) . For any pair of multisets V and D of elements of Z with V ⊆ D , c ( D ) = V = ⇒ c ( D,z ) = V, ∀ z ∈ Z .

glyph[negationslash]

The preference property asserts that, if the multiset V was not chosen when compressing D , it will also not be chosen when compressing the set D augmented with one more element z . Under the preference property, it is easy to prove that c ( c ( D )) = c ( D ) , i.e., the compression of a compressed multiset must be the compressed multiset itself.

Property A.2 (Non-associativity) . For any multiset D of elements of Z and for any z 1 , . . . , z p ∈ Z , p > 1 , c ( D,z i ) = c ( D ) , ∀ i ∈ { 1 , . . . , p } = ⇒ c ( D,z 1 . . . , z p ) = c ( D ) .

The non-associativity property ensures that, if the compression does not change by adding one element at a time, it must also not change when adding all the elements together. 9

Property A.3 (Non-concentrated mass) . P { z = z } = 0 , ∀ z ∈ Z .

The property of non-concentrated mass ensures that every point z ∈ Z is drawn with probability zero.

Under these three properties, strong results on the probability of change of compression, formally defined as glyph[negationslash]

(i.e., change of compression occurs when the compression of c ( D ) augmented with one element is different from c ( D ) ) are established in (Campi & Garatti, 2023).

Theorem A.4 (Thm. 7 in (Campi & Garatti, 2023)) . Assume Properties A.1 to A.3. Then, for any δ ∈ (0 , 1) ,

where ε ( k, δ ) for k = 0 , 1 , . . . , N -1 is the unique solution to Ψ k,δ ( ε ) = 1 in the interval [ k/N, 1] , while ε ( N,δ ) = 1 , and ε ( k, δ ) for k = 0 , 1 , . . . , N is the maximum between 0 and the unique solution to Ψ k,δ ( ε ) = 1 in the interval ( -∞ , k/N ] (function Ψ k,δ ( ε ) is defined in Section 4).

The previous statement asserts that, with probability 1 -δ over the draws of D , the probability of change of compression is contained in an interval with upper and lower extremes given by ε ( k, δ )

and ε ( k, δ ) , where k is evaluated at the cardinality of the compression of the draw. Prompted by one of the reviewers, we gladly provide a concise outline of the proof structure for Theorem A.4 in the following Remark A.5.

Remark A.5 (Gist of the proof of Theorem A.4) . The proof of Theorem A.4 in (Campi & Garatti, 2023) starts by associating any compression scheme that satisfies Properties A.1 to A.3 to a probability measure from which one can compute P { ε ( | c ( D ) | , δ ) ≤ φ ( D ) ≤ ε ( | c ( D ) | , δ ) } and then by characterizing the class of all such probability measures in terms of certain conditions. The next step consists in obtaining an upper bound to P { ε ( | c ( D ) | , δ ) ≤ φ ( D ) ≤ ε ( | c ( D ) | , δ ) } by maximizing this quantity over the class of probability measures that has been previously characterized. The ensuing maximization problem is infinite dimensional and its solution is obtained by duality. Interestingly, no conservatism is introduced at this stage because strong duality holds. See (Campi & Garatti, 2023) for details.

glyph[negationslash]

The interest of Theorem A.4 in the context of statistical learning lies in the fact that, whenever the compression function c ties in with the learning algorithm so that inappropriateness (e.g., misclassification or misprediction) implies change of compression, then the probability of inappropriate examples is dominated by the probability of change of compression and, therefore, it can be upper bounded with high confidence based on (3). See also Section 4 in (Campi & Garatti, 2023). This means that we can prove Theorem 4.2 by exhibiting a compression function that enjoys the result of Theorem A.4 and for which whenever h ( D ) is inappropriate for z , i.e., Stop ≤ h ( D ) z , we also have c ( c ( D ) , z ) = c ( D ) . This is indeed the demonstration path pursued in the next Appendix A.2.

A.2 Proof of main result

Note that Theorem 4.2 does not assume the non-concentrated mass Property A.3. To ease the presentation, however, Theorem 4.2 will be first proven by assuming Property A.3, which allows for a more direct application of Theorem A.4; later, we will show how this extra assumption can be removed.

Suppose thus for the time being that Property A.3 holds true. Throughout this section, we denote with c A the compression function obtained by A when it compresses multiset D into T ⊆ D (recall that ( h, T ) = A ( D ) ). We show that c A enjoys the preference and non-associativity Properties A.1 and A.2. This licenses the use of Theorem A.4 to obtain upper and lower bounds on the probability of change of compression. We shall then relate these bounds on the probability of change of compression to the statistical risk as in Definition 4.1, so closing the proof when Property A.3 is assumed. We shall use A ( D,z ) and c A ( D,z ) as shorthand for A ( D ∪ { z } ) and c A ( D ∪ { z } ) .

We begin by showing that c A ( · ) is preferent.

Lemma A.6. The compression function c A satisfies the preference Property A.1.

glyph[negationslash]

Proof. Consider a multiset D of elements of Z and further let z ∈ Z . Suppose that c A ( D,z ) = c A ( D ) . Our goal is to show that in this case c A ( D,z ) = V for any V ⊆ D , from which Property A.1 immediately follows.

glyph[negationslash]

glyph[negationslash]

Denote by T k , h k and T ′ k , h ′ k the multisets T and hypotheses h constructed in A ( D ) and A ( D,z ) , respectively, at iteration k of the 'while' loop. T 0 = ∅ = T ′ 0 and h 0 = h ′ 0 instead denote the empty multisets T and initial hypotheses in the initialization step. Finally, let D ′ S = D S ∪ { z } (where we recall that D S = D ∪ { Stop } ). Since c A ( D,z ) = c A ( D ) , there must exist a ¯ k ≥ 0 such that the executions of A ( D ) and A ( D,z ) match for k < ¯ k (which implies that T ′ k = T k and h ′ k = h k for k ≤ ¯ k ), but they differ at iteration ¯ k because, for k = ¯ k , the maximal element computed in A ( D ) (line 5 if ¯ k ≥ 1 or line 1 if ¯ k = 0 ) is different from the one computed in A ( D,z ) . That is, max h ¯ k ( D S \ T ¯ k ) = max h ′ ¯ k ( D ′ S \ T ′ ¯ k ) = max h ¯ k ( D ′ S \ T ¯ k ) , where the last equality holds because, as previously noticed, T ′ ¯ k = T ¯ k and h ′ ¯ k = h ¯ k . We claim that it must be that max h ¯ k ( D ′ S \ T ¯ k ) = z and z / ∈ D S \ T ¯ k . Indeed, if this was not the case, it would be that max h ¯ k ( D ′ S \ T ¯ k ) ∈ D S \ T ¯ k , which would imply that max h ¯ k ( D ′ S \ T ¯ k ) = max h ¯ k ( D S \ T ¯ k ) since D S \ T ¯ k ⊆ D ′ S \ T ¯ k . The latter would contradict max h ¯ k ( D S \ T ¯ k ) = max h ¯ k ( D ′ S \ T ¯ k ) . ′ ¯

glyph[negationslash]

glyph[negationslash]

Since max h ¯ k ( D S \ T ¯ k ) = z = Stop , A ( D,z ) executes at least one more iteration after the k -th, and glyph[negationslash]

glyph[negationslash]

z / ∈ D S \ T ¯ k yields that T ′ ¯ k +1 = T ¯ k ∪ { z } contains the element z as many times as it appears in D ∪{ z } , i.e., as many times as it appears in D plus one. Therefore T ′ ¯ k +1 glyph[negationslash]⊆ D and, since the multiset T ′ k is increasing with k , it must be that T ′ k glyph[negationslash]⊆ D for all k ≥ ¯ k +1 . This implies c A ( D,z ) glyph[negationslash]⊆ D ; that is, c A ( D,z ) = V for any V ⊆ D .

We now show that c A is non-associative.

Lemma A.7. The compression function c A satisfies the non-associativity Property A.2.

glyph[negationslash]

Proof. Fix any multiset D of elements of Z and further let z 1 , . . . , z p ∈ Z . We will prove the claim by contrapositive. Towards this goal, assume that c A ( D,z 1 , . . . , z p ) = c A ( D ) . We will then show that there exists an i ∈ { 1 , . . . , p } such that c A ( D,z i ) = c A ( D ) .

glyph[negationslash]

glyph[negationslash]

We first focus on the executions of A ( D ) and A ( D,z 1 , . . . , z p ) . Similarly to the proof of Lemma A.6, denote by T k , h k and T ( p ) k , h ( p ) k the multisets T and hypotheses h constructed in A ( D ) and A ( D,z 1 , . . . , z p ) , respectively, at iteration k ( k = 0 corresponds to the initialization). Finally, let D ( p ) S = D S ∪ { z 1 , . . . , z p } . Since c A ( D,z 1 , . . . , z p ) = c A ( D ) , also in the present context there must exist a ¯ k ≥ 0 such that the executions of A ( D ) and A ( D,z 1 , . . . , z p ) match for k < ¯ k (so that T k = T ( p ) k and h k = h ( p ) k for k ≤ ¯ k ), but they differ at iteration ¯ k because, for k = ¯ k , the maximal element computed in A ( D ) (line 5 if ¯ k ≥ 1 or line 1 if ¯ k = 0 ) is different from the one computed in A ( D,z 1 , . . . , z p ) . That is, max h ¯ k ( D S \ T ¯ k ) = max h ( p ) ¯ k ( D ( p ) S \ T ( p ) ¯ k ) . Then, following an argument identical to that in the proof of Lemma A.6, one observes that it must be that max h ( p ) ¯ k ( D ( p ) S \ T ( p ) ¯ k ) = z i for some i ∈ { 1 , . . . , p } and z i / ∈ D S \ T ¯ k .

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Now compare the executions of A ( D ) and A ( D,z i ) , and denote by T ′ k , h ′ k the multiset T and the hypothesis h constructed in the execution of A ( D,z i ) at iteration k ( k = 0 still corresponds to the initialization). Further, let D ′ S = D S ∪{ z i } . One can first observe that also the executions of A ( D ) and A ( D,z i ) for k < ¯ k must be identical. Notice indeed that: i. T ′ 0 = T 0 = T ( p ) 0 and h ′ 0 = h 0 = h ( p ) 0 ; ii. when T ′ k = T k = T ( p ) k and h ′ k = h k = h ( p ) k for a k < ¯ k , the maximal element selected at iteration k of A ( D ) and A ( D,z i ) is the same since this is so for the maximal element of A ( D ) and A ( D,z 1 , . . . , z p ) and D S \ T k ⊆ D ′ S \ T ′ k ⊆ D ( p ) S \ T ( p ) k ; iii. this in turn gives T ′ k +1 = T k +1 = T ( p ) k +1 and h ′ k +1 = h k +1 = h ( p ) k +1 . At iteration ¯ k , instead, the identity of executions ends, and it must be max h ′ ¯ k ( D ′ S \ T ′ ¯ k ) = z i because z i is maximal over D ( p ) S \ T ( p ) ¯ k and z i ∈ D ′ S \ T ′ ¯ k ⊆ D ( p ) S \ T ( p ) ¯ k (surely z i still belongs to D ′ S \ T ′ ¯ k because T ′ ¯ k = T ¯ k ⊆ D and D ′ S = D ∪ { z i } ∪ { Stop } ). Further, recall that z i / ∈ D S \ T ¯ k . Since A ( D,z i ) executes at least one more iteration after the ¯ k -th (because z i = Stop ) and since the T ′ k 's are increasing, we have that any T ′ k for k ≥ ¯ k +1 is different from any of the T k 's. This implies c A ( D,z i ) = c A ( D ) , thus concluding the proof.

glyph[negationslash]

Given that Properties A.1 and A.2 hold true, and since Property A.3 is assumed, Theorem A.4 applies, yielding P { ε ( | c A ( D ) | , δ ) ≤ φ A ( D ) ≤ ε ( | c A ( D ) | , δ ) } ≥ 1 -δ , where φ A ( D ) = P { c A ( c A ( D ) , z ) = c A ( D ) | D } . As anticipated, to obtain from this the sought result for the statistical risk, it is enough to show that, given D and the hypothesis h returned by A ( D ) , every realization z such that Stop ≤ h z also changes the compression, i.e., c A ( c A ( D ) , z ) = c A ( D ) . As a matter of fact, this implies that for every D the event { Stop ≤ h z } is a sub-event of { c A ( c A ( D ) , z ) = c A ( D ) } , and hence that R ( h ) ≤ φ A ( D ) almost surely. Then, Theorem 4.2's claim readily follows since glyph[negationslash]

Thus, we are left to prove the following lemma.

Lemma A.8. Let D be any multiset of elements of Z , z ∈ Z be any example, and ( h, T ) = A ( D ) . If z is such that Stop ≤ h z , then c A ( c A ( D ) , z ) = c A ( D ) .

glyph[negationslash]

glyph[negationslash]

Proof. We prove the contrapositive and show that, if z is such that c A ( c A ( D ) , z ) = c A ( D ) , then z ≤ h Stop .

Note that ( h, T ) = A ( D ) means that T = c A ( D ) and h = L ([ T ] A ) thanks to line 4 in Algorithm 1. Consider the execution of A ( c A ( D ) ∪ { z } ) . By definition of c A , the condition c A ( c A ( D ) , z ) = c A ( D ) means that A ( c A ( D ) ∪ { z } ) terminates with c A ( D ) = T . Hence, the maximal elements selected by A ( c A ( D ) ∪ { z } ) are the same as those selected by A ( D ) . Further, by the properties of max operators, the elements selected by A ( c A ( D ) ∪ { z } ) and A ( D ) match at each iteration. This implies that A ( c A ( D ) ∪ { z } ) terminates returning h , besides T . For this to occur, it must be that max h (( c A ( D ) ∪{ z }∪{ Stop } ) \ T ) = Stop (line 2). This is equivalent to max h ( { z, Stop } ) = Stop , i.e., z ≤ h Stop .

Having proven Lemma A.8, we have completed the derivation of the main result under the additional assumption that Property A.3 holds true , an assumption that, however, does not appear in the statement of Theorem 4.2. To remove this assumption, and thus complete the proof of Theorem 4.2, we can proceed as follows.

Augment each random element z i with a random variable u i that is uniformly distributed over [0 , 1] and independent of z i . Moreover, define the u i 's so that the random elements ˜ z 1 , ˜ z 2 , . . . , ˜ z N , where ˜ z i = ( z i , u i ) for all i , are independent and identically distributed (i.i.d.). Also, let ˜ z = ( z , u ) be independent of and identically distributed as each ˜ z i . Let ˜ Z = Z × [0 , 1] , which is the space where the ˜ z i 's take value. For any ˜ z = ( z, u ) ∈ ˜ Z and ˜ z ′ = ( z ′ , u ′ ) ∈ ˜ Z define ˜ z ≤ h ˜ z ′ if z ≤ h z ′ and u ≤ u ′ , while define ˜ z ≤ h Stop [resp. Stop ≤ h ˜ z ] if z ≤ h Stop [resp. Stop ≤ h z ]. Also, for any n , define L (˜ z 1 , . . . , ˜ z n ) = L ( z 1 , . . . , z n ) , where ˜ z i = ( z i , u i ) , i = 1 , . . . , n , are elements of ˜ Z .

Consider now the executions of A over multisets of elements of ˜ Z according to the redefinition of ≤ h and L , and let ( ˜ h , ˜ T ) = A ( ˜ D ) , where ˜ D = { ˜ z 1 , . . . , ˜ z N } . As is clear, the non-concentrated mass Property A.3 applies for the ˜ z i 's and an application of the results obtained so far gives P { ˜ R ( ˜ h ) ≤ ε ( | ˜ T | , δ ) } ≥ 1 -δ , where ˜ R ( h ) = P { Stop ≤ h ˜ z } . On the other hand, by virtue of the redefinition of ≤ h and L , one can notice that, for every realization of ˜ D , the execution of A ( ˜ D ) matches step by step that of A ( D ) , with the sole difference that elements in T that are otherwise indistinguishable are incorporated in ˜ T with their ' u ' counterparts (and the highest values are selected first). Thus, the multiset T is equal to the multiset of the ' z ' components of the elements of ˜ T , yielding | ˜ T | = | T | . Moreover, since the addition of elements in T and ˜ T occurs in the same order, we also have that ˜ h = h . Finally, noticing that ˜ R ( h ) = R ( h ) because ˜ z ≤ h Stop ⇐⇒ z ≤ h Stop , one has that

and this concludes the proof of Theorem 4.2.

Remark A.9 (On the tightness of Theorem 4.2) . Although both upper and lower bounds that hold with confidence 1 -δ are available for the probability of change of compression, only the upper bound ports over to the statistical risk, because any realization for which Stop ≤ h z (i.e., a realization that is inappropriate) is also a realization that changes the compression, but the opposite does not hold. Given that the lower and the upper bounds are provably close to each other even for relatively small values of N , this means that ε ( | T | , δ ) is an accurate evaluation of the probability of change of compression, but it can be loose for the statistical risk whenever this deviates from the probability of change of compression. Interestingly, this is the only source of looseness in the evaluation of the risk provided by Theorem 4.2.

A.3 An extension of P2L: adding multiple examples in T simultaneously

A simple extension of the meta-algorithm P2L described in Algorithm 1 consists in adding at every iteration multiple, say R > 1 , examples in T at a time. A possible implementation is then formalized in Algorithm 2 below, where max R h ( U ) returns the R maximal points (the maximum, the 2nd maximum, ..., the R -th maximum) of U according to the total order ≤ h (if U has less than R elements, then the whole U is returned).

glyph[negationslash]

Interestingly, Theorem 4.2 continues to hold true (with no modification in the statement) also when the pair h, T is the output of Algorithm 2 instead of Algorithm 1. To prove this, it is enough to observe that Algorithm 2 is completely equivalent to the following sequential Algorithm 3, which is identical to Algorithm 1, so that only one point is added to T at each iteration, except for the fact that h is updated every R iterations (mod ( iter , R ) is the remainder of the division of iter by R ). This equivalence secures the result because the proof of Theorem 4.2 can be repeated for Algorithm 3 word for word, without any modification.

glyph[negationslash]

As is clear, Algorithm 1 is a particular case of Algorithm 2 since the former is re-obtained from the latter when R = 1 . We notice that adding more than one example to T at every iteration might or might not have a beneficial effect on P2L. Indeed, on one hand, adding more examples simultaneously might result in building a better h earlier on (and save computational time). However, by introducing examples in groups (and thus not allowing for the fine-grained choice of adding them one-by-one) might also result in a larger T , which could worsen the final bound on the risk. In this regard, a modulation of R across iterations seems to be a promising line of investigation, which however is beyond the scope of the present paper and is therefore left for future work.

A.4 A discussion on the P2L framework in retrospect

Aclose inspection of the proofs of the key Lemmas A.6 and A.7 reveals that these two lemmas strongly rely on the fact that the max h operator is a compression function from many elements to just one with the following properties: i) it is preferent (as a matter of fact, for any multiset A and additional element b , either max h ( A ∪ { b } ) = b or max h ( A ∪ { b } ) = max h ( A ) ); and ii) it is non-associative ( max h ( A ∪ { b i } ) = max h ( A ) for all i ∈ { 1 , . . . , p } implies that there is an element ¯ a of A such that c ≤ h ¯ a for all c ∈ A ∪ { b 1 , . . . , b p } and, therefore, max h ( A ∪ { b 1 , . . . , b p } ) = max h ( A ) ).

Importantly, these are the only two properties of max h that are used and, therefore, one could have formulated Algorithm 1 in an (apparently, see below) more general form: instead of introducing a hypothesis-dependent total ordering ≤ h , one might have considered in its place a hypothesisdependent preferent and non-associative compression function from many to one example. Mutatis mutandis , Theorem 4.2 and the arguments to prove it would have remained the same. This generalization extends to the context in which multiple examples are selected at a time as suggested in Appendix A.3.

We feel advisable to also notice that the above generalization is indeed a real generalization only in the case of selection of multiple example at a time. Instead, in the context of one example selected at a time the alternative setup illustrated above turns out not to be more general than that of Section 4 (this is why we wrote 'apparently' in parenthesis). In fact a converse property holds: any compression function from many examples to one that is preferent and non-associative always defines a total ordering whose notion of max coincides with the compression function itself. This fact follows from the following proposition (see a few lines down for a proof).

Proposition A.10. Any preferent compression function w from many examples to one defines a total ordering ≤ w for which it holds that w ( a 1 , . . . , a n ) = max w ( a 1 , . . . , a n ) .

Note that, as a byproduct of Proposition A.10, we have that any preferent compression function w from many examples to one is also non-associative (since max w is non-associative). Therefore, the notion of max from a total ordering and that of preferent and non-associative compression function from many examples to one are equivalent. Interestingly, this equivalence crucially relies on the fact that the compression selects only one example, while, provably, the class of preferent and non-associative compression functions offers more freedom than using a total ordering when selecting more than one example at a time. This is important for the development of alternative implementations of Algorithm 2, the version of P2L where multiple examples are selected at a time.

Proof of Proposition A.10. For any a and b define a ≤ w b if w ( a, b ) = b (and thus b ≤ w a if w ( a, b ) = a ). To show that ≤ w is indeed an ordering, we have to prove that ≤ w enjoys the reflexive , antisymmetric , and transitive properties.

Since w always selects one element, we have that w ( a, a ) = a , i.e., a ≤ w a , which is the reflexive property.

If a ≤ w b and b ≤ w a , then w ( a, b ) = b and w ( a, b ) = a . Thus, a ≤ w b and b ≤ w a = ⇒ a = b , which is the antisymmetric property.

Suppose that a ≤ w b and b ≤ w c , which corresponds to w ( a, b ) = b and w ( b, c ) = c . Since w is preferent, it must be that w ( a, b, c ) = c (indeed, w ( a, b, c ) cannot be a because a is not selected by w ( a, b ) ; it cannot be b either because b is not selected by w ( b, c ) ). Then, an application of preference again gives w ( a, c ) = c . This means that a ≤ w b and b ≤ w c implies that a ≤ w c , which is the transitive property.

glyph[negationslash]

We next prove that the max w operator associated to ≤ w corresponds indeed to w ; that is, it holds that a i ≤ w w ( a 1 , . . . , a n ) for all i ∈ { 1 , . . . , n } . The proof is by contrapositive. Suppose thus that w ( a 1 , . . . , a n ) = a i for some i , but a i < w a j (i.e., a i ≤ w a j and a i = a j ) for some j . Condition a i < w a j means that w ( a i , a j ) = a j . By preference, adding another element a k to a i and a j would give w ( a i , a j , a k ) = a i . But then, preference again would give that the compression of a i , a j , and a k plus a fourth element cannot be a i , and so forth and so on. Thus, proceeding iteratively would eventually lead to w ( a 1 , . . . , a n ) = a i , which is a contradiction.

glyph[negationslash]

B Additional material for the application to MNIST Classification

B.1 Implementation details for SGD and PAC-Bayes

GD & test-set approach. Here we follow a classical test-set approach, whereby a fraction of the dataset is used for training, and the remaining portion is sacrificed to certify the quality of the resulting model. We experiment with different sizes of these fractions. As for training, we use GD with momentum and fix the number of training epochs to 200 . To compute the generalization bound, we use the tightest-known Binomial test-set bound 10 of (Langford, 2005)[Thm 3.3] with a number of samples equal to N minus those used for training. That is, letting ˜ N be the number of examples out

glyph[negationslash]

of N used for training and ˜ h = GD ( z 1 , . . . , z ˜ N ) be the returned hypothesis, we have that

with high confidence 1 -δ . We also perform a grid search over the following set of parameters: learning rate in [0 . 001 , 0 . 005 , 0 . 01] , momentum in [0 . 9 , 0 . 95] , dropout probability in [0 . 01 , 0 . 05 , 0 . 1 , 0 . 2] . We then select the combination of parameters that return the lowest generalization bound. 11

PAC-Bayes approach. Here we learn a model and provide generalization bounds based on the recent results presented in (Clerico et al., 2022; Perez-Ortiz et al., 2021), which build on earlier work in (Dziugaite & Roy, 2017) while providing tighter generalization bounds. Specifically, we utilize the same PAC-Bayes implementation of (Perez-Ortiz et al., 2021) adapted to the considered binary classification problem, with one main difference: instead of training by optimizing a surrogate of the generalization bound as in (Perez-Ortiz et al., 2021), we optimize the exact bound as proposed in (Clerico et al., 2022, Eq 4.a), which provides the sharpest results to date. 12 As the PAC-Bayes approach requires a prior distribution over the networ weights to start, we split, as it is customary, the training dataset in two and use the first part to pretrain the prior through SGD, while the remaining data are used to obtain the posterior distribution (contrary to P2L and GD+test-set where the final hypothesis consists of a deterministic network, the PAC-Bayes approach is forced to work with stochastic networks). We experiment with different sizes of the data portion to be used to pretrain the prior. In running PAC-Bayes we employ a batch size of 250 , 100 prior and posterior training epochs, and 10 -6 as lower bound for the minimum probability of the softmax output. We also perform a grid search over the remaining parameters: prior standard deviation in [0 . 01 , 0 . 02 , 0 . 03 , 0 . 04 , 0 . 05 , 0 . 1] ; dropout probability in [0 . 01 , 0 . 05 , 0 . 1 , 0 . 2] ; prior/posterior learning rate in [0 . 001 , 0 . 005 , 0 . 01] ; prior/posterior momentum in [0 . 9 , 0 . 95] . The final bound we present is that in (Perez-Ortiz et al., 2021)[Sec 6.2], by using 10000 Monte Carlo samples from the learned distribution over the models to approximate the average empirical risk. When we experiment with δ = 0 . 035 ( δ = 0 . 001 ), we use a confidence of 0 . 025 ( 0 . 00072 ) for the KL bound and of 0 . 01 ( 0 . 00028 ) for Monte Carlo sampling as done in (Perez-Ortiz et al., 2021)[Sec 6.2]. We apply here the same philosophy as described in Footnote 11 regarding not introducing a union bound correction. The post-training performance is evaluated again by sampling the distribution over the network weights, computing the empirical misclassification on the full test dataset, and averaging over 10000 samples of the weights. As a matter of fact, the generalization bounds derived through the PAC-Bayes approach apply precisely to this quantity.

C Some examples when the inner algorithm L has already in itself a compression scheme

In this section, also prompted by a reviewer, we perform some synthetic numerical experiments to investigate how P2L performs when the inner algorithm L already exhibits a compression in itself; specifically, we consider SVM (Support Vector Machine, (Cortes & Vapnik, 1995)) and SVR (Support Vector Regression, (Smola & Schölkopf, 2004)) as inner algorithms for P2L and compare the returned output with that output by SVM and SVR alone. The so-called support vectors returned by SVM and SVR are known to compress the original dataset in a way that is informative for the underlying learning problem. With these experiments, we want to verify whether P2L returns the same or a similar compression.

-

1

0

1

-

1

0

1

x

y

-

1

0

1

-

1

0

1

x

y

In a classification problem, we consider datasets with N = 1000 examples ( x, y ) , where instances x are uniformly sampled from [ -1 , 1] 2 ⊆ R 2 and labels are 1 if ( x 1 -0 . 3) 2 + x 2 2 ≤ 0 . 7 and -1 otherwise. SVM (either when run over the whole dataset or when called by P2L as inner algorithm) is implemented with kernel k ( x, ˜ x ) = x · ˜ x ′ + ‖ x ‖ 2 · ‖ ˜ x ‖ 2 , for which the two classes defined above are separable. P2L is run by selecting as h 0 the classifier corresponding to setting all the parameters in the SVM parameterization to 1 . As for ≤ h , we consider the order induced by the distance in the lifted feature space of examples from the separating hyperplane corresponding to h , multiplied by 1 when examples are misclassified and by -1 otherwise (this yields that the worst examples are those misclassified and furthest away from the separating hyperplane). Distance equal to 0 is taken as the threshold for appropriateness, which corresponds to consider a classifier h appropriate for an example ( x, y ) when ( x, y ) is not misclassified by h .

Figure 5 displays the results for two realizations of the considered classification problem, which are representative of what we have observed in multiple trials. Each plot depicts: the examples in the instance domain as markers + (label 1 ), × (label -1 ); the separation boundaries of the SVM and of the P2L+SVM classifiers (dashed blue and solid red lines, which are completely overlapped); the support vectors returned by SVM (red squares) and the examples in T returned by P2L+SVM (blue triangles).

In the left panel of Figure 5, the compressed multiset T returned by P2L+SVM contains all the support vectors of SVM with the addition of few other examples, which are incorporated in T during the first iterations, while P2L is still exploring the problem. As it is clear, by the definition of support vectors, the classifier returned by P2L+SVM coincides with the classifier obtained by running SVM. In the right panel of Figure 5, only part of the support vectors are also elements of T . Nonetheless the classifiers obtained with P2L+SVM and SVM are still the same (at least up to a negligible approximation). Hence, it appears that P2L+SVM compensates the deficiency of support vectors with other alternative examples that are anyway capable to recover the SVM classifier. The amount of support vectors appearing in T depends on the realization of the dataset, but in all our trials the classifier returned by P2L+SVM matches that obtained by running SVM on the same dataset.

As for SVR, we consider the same regression problem as in Section 6 when N = 200 . SVR is run using a radial basis function k ( x, ˜ X ) = exp( -0 . 1 ‖ x -˜ x ‖ 2 ) as kernel, tube size equal to 0 . 1 , and coefficient C = 1 ( C modulates the trade-off between the flatness of the predictor and the penalty for deviations from the predictor larger than 0 . 1 ). P2L instead starts with a predictor h 0 corresponding to setting all the parameters in the SVR parameterization to 0 , while we consider the same total order ≤ h of Example 3.2 with γ = 0 . 1 . In this context, h is appropriate for an example ( x, y ) if the deviation of the output y from the prediction corresponding to x is no more than 0 . 1 .

Figure 6 displays the results obtained for one dataset, which is also representative of other realizations of data. In the plot, examples are represented as black crosses, while the dashed blue and red lines,

-

0

.

4

-

0

.

2

0

0

.

2

0

.

4

0

.

6

0

.

8

1

1

.

2

1

.

4

0

0

.

5

1

x

y

which are completely overlapped, depict the predictors returned by SVR and P2L+SVR respectively. The support vectors returned by SVR are displayed by blue triangles, while the examples in T returned by P2L+SVR by red squares.

Similarly to the SVM case, in all our trials the predictor returned by P2L+SVR is identical to the predictor obtained by running SVR on the same dataset. In the present example, however, it is always the case that the compressed multiset T returned by P2L+SVR contains all the support vectors of SVR, with the addition of few other examples incorporated in T during the first iterations during which P2L is still exploring the problem.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a meta-learning algorithm which compresses a training set $D$ of size $|D|$ into a smaller training set $T\\subset D$ of size $|T|$. The algorithm can be defined for any *base learner*, i.e. any training strategy which outputs a trained hypothesis $h$ when given a training set. The set $T$ is selected in such a way that the performance of the base algorithm trained on $T$ and evaluated on $D$ is better than a predetermined threshold. Relying on some powerful recent results from [1], the authors show that one can obtain tight generalization bounds for the result of the algorithm expressed as a function of the size  ratio $|T|/|D|$: intuitively, if the model is able to compress the dataset $D$ into a much smaller dataset $|T|$ whilst maintaining the performance of the base learner trained on $T$, it must be that the sample size is already sufficient to train the model well. Thus, the results can be interpreted as a middle way between on the one hand  the traditional approach to generalization bounds via direct study of the function class searched by a given algorithm, and on the other hand,  the even simpler *test-set bounds*[2]. The function of the ratio $|T|/|D|$ which gives a bound on the probability of error is a complex implicitly defined function inherited from [1] with a very mild dependence on the failure probability $\\delta$. \n\nThe algorithm functions as follows: start with an initial hypothesis $h_0$, then find the element of the training set $D$ which is the least well explained by $h$ (for instance, the one with the largest loss function value), and include this element in the set $T$. Retrain the hypothesis based on $T$ and repeat the procedure until either every element of the set $D\\setminus T$ has a loss function value below a given predetermined threshold. \n\nThe proof of the generalization bounds is a reasonably direct consequence of Theorem A.4., which is a powerful result from [1] concerning compression schemes. The key is to show that the compression function defined by the algorithm (the function which compresses $D$ into $T$ satisfies the properties of *preference* and *non associativity*, which are proved in Lemmas A.5. and A.6 respectively, and that the failure event which is bounded in Theorem A.4, which is the *probability of change of compression* is implied by the failure of prediction event (this is shown in Lemma A.7.). A technical condition of non concentrated mass which is required to be satisfied for Theorem A.4 to hold, is also removed by a straightforward argument augmenting each datapoint by a continuous random variable which doesn't affect the loss function/order. It is also shown (Proposition A.9) that any compression scheme which outputs a single element and is preferent must correspond to a maximum function with respect to some well defined notion of order. \n\nExperiments on the MNIST dataset and on a synthetic regression dataset demonstrate that the proposed algorithm yields superior generalization performance and tighter bounds than the Pac-Bayesian baseline, and proide slightly inferior bounds compared to a test-set approach. It is still worth noting nonetheless that the test performance is still superior to the test set approach due to a better indirect exploitation of the whole training set in the compression step. \n\n\n==============Post-rebuttal==========\n\nAs seen in the comments below, my discussion with the authors and the extra material promised (especially the experiments in the rebuttal concerning support vectors) have increased my opinion of the paper, resulting in me raising my score to 6. I believe this paper is above the threshold and tackles very interesting questions, with the main downside being a relatively small amount of material and original proofs (which should not necessarily disqualify a paper for publication). \n\n=====\n\n\n**References** \n\n[1] Marco C. Campi, Simone Garatti. Compression, Generalization and Learning. ArXiv 2023.\n[2] Langford, J. and Shapire, R. Tutorial on practical prediction theory for classification. JMLR 2017.\n\nSTRENGTHS:\n1. This is a very interesting direction, not just in terms of providing generalization bounds (which is the main aspect presented in the paper) but also in terms of potential practical interpretation: this is a simple algorithm that, in principle, allows one to compress a dataset into a much smaller one, allowing us to derive interpretable information about which datapoints are key to the training procedure. \n\n2. This is a novel application of a mathematical result from [1] into a machine learning context, and has a distinctly novel flavor compared to many existing generalization bounds. \n\n3. The paper is relatively well written and a pleasure to read. The proofs are clean despite the fact that they must have been a bit annoying to write: It is quite trivial to convince oneself that the results (lemmas A.5, A.6, A.7 and Proposition A.9.) hold, but not so much fun to write down explicitly.\n\nWEAKNESSES:\n1. The results are essentially straightforward applications of existing results from [1], there isn't really any non trivial difficulty that had to be vanquished in the proofs. In short, this paper isn't a lot of work from the authors. \n\n2. The experimental evaluation is still relatively preliminary, there is so much more to explore: Other datasets and architectures, better baselines, and more importantly, the implications in terms of interpretability. For instance, the following follow-up experiments could be performed: \n\n2.1 Compare to more baselines on more datasets (Cifar, other Pac Bayesian bounds). In footnote 7 on page 7, the authors claim that \"Pac-Bayesian approaches have been developed only for linear regression problems\". This is a highly doubtful or (at best misleading) statement. In addition, generalization bounds based on function class capacity could also be evaluated in the synthetic regression dataset. \n\n2.2 (most important) Investigate the interpretability of the results: for instance, it would be interesting and rather key to evaluate the method on a synthetic binary classification dataset with a simple kernel method. It seems that in an ideal situation, the set $T$ should eventually correspond to, or at least strongly overlap, the set of support vectors. Even in the case of MNIST, it would be very interesting to visualize the chosen datapoints and see if they have something qualitatively different from the non-chosen ones. \n\n3 A very big issue is that the algorithm, as presented, is not that applicable in practice. Indeed, to make the algorithm work in the examples considered, the authors needed to pretrain the model on a significant proportion of the training set. In particular, the algorithm cannot select which datapoints in the set used for pretraining are more important than others. Since the algorithms used rely on gradient descent and the authors interpret the call to the base learner as a single gradient step, the initialization is key. It remains to be seen whether the algorithm in its pure form, with a base learner that performs exact empirical risk minimization on $T$, can work in any practical scenario. At the very minimum, it should be checked whether training the network on $T$ from random initialization (rather than pretraining it to obtain $h_0$ and then continuing training with $T$, which is what is done here) yields comparable performance.  \n\n3.2 (related to 3) It seems like the extension to the case where several datapoints are introduced into $T$ simultaneously (perhaps in a hierarchical way) would not be too much to ask in a first submission, since it is key to solving the problem of the overreliance on initialization. \n\n4. I feel like the description of the previous results could be more extensive, for the benefit of the reader. For instance, the following things could improve the paper's reach and interest to a broader audience: \n\n4.1 Explain the gist of the proof of theorem A.4., at least in terms of intuitive reason why the function $\\Psi$ from line 187 appears. \n\n4.2.  Write down the result (theorem 3.3) from [2] which is used to evaluate the test set bounds and discuss it. \n\n4.3 In line 472, the sentence \"See also Section 4 in [1]\" appears to suggest that the connection between the probability of change of compression and the classification/regression error is already established in [1]. If that is the case, how much of the present paper is truly not covered or implied by the discussion in [1]? \n\n5. I feel like a more detailed description of existing literature on training set compression schemes and how they relate to the present method is needed. It is hard to believe that no such literature exists. \n\n\n\n\n**Minor comments/typos**\n\nThe concept of \"probability of change of compression\" is frequently used in the main paper (cf., e.g., line 341, but it is only defined in the appendix (455-457)\n\nLine 28: \"the precision of available bounds is much problem dependent\" ===>  \"the precision of available bounds is highlyproblem dependent\" \n\nline 64: \"is laying the groundwork\" ==> \"lays the groundwork\"\n\nline 110 \"is add to\"  ===> \"is added to\" \n\nLines 114 and 115: \"enough appropriate\" ==> \"appropriate enough\" \n\nLine 175:  \"an hypothesis\" ==> \"a hypothesis\" \n\nLine 280: \"fits well the data\" ==> \"fits the data well\" \n\nlines 480 and 560: I would use \"remove the condition\" instead of \"release the condition\" \n\n\n\n\n\n\n\n\n\n**References** \n\n[1] Marco C. Campi, Simone Garatti. Compression, Generalization and Learning. ArXiv 2023.\n[2] Langford, J. and Shapire, R. Tutorial on practical prediction theory for classification. JMLR 2017.\n\nQUESTIONS:\nDo you think your algorithm will be able to recover support vectors in a simple linear classification problem? \n\nIn Table 1 on page 7, which proportion of the training set is used for pretraining?\n\nLIMITATIONS:\nMostly the reliance on intialization, the limited evaluation of interpretability and the coverage only of the case where a single element is added to the set $T$ at each iteration. See  \"weaknesses\" for more details\".\n\nAssuming there is really no comparable result in the literature (i.e. no generalization bounds expressed in terms of the success of a compression method), this is still a **very interesting paper opening up a new direction**. However, the amount of content included in this first contribution is surgically small.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work elaborates on the recent breakthroughs of Campi&Garatti 2023, by exploiting compression theory results to design a novel meta-algorithm, namely the Pick-To-Learn (P2L) algorithm. This algorithm aims to compress the dataset to a smaller, truly impacting one, this notion of impact being defined through a hypothesis dependent order $\\leq_h$. Authors provably show high-probability generalisation bounds for P2L and experimentally shows that P2L with gradient descent as subroutine yields better theoretical results and experimental performances than both the test-set approach and PAC-Bayes learning.\n\n\nIn conclusion, I am convinced by the P2L algorithm, and it could be enough for acceptance at this point. However, I remain doubtful about the experimental process, see the Questions part. \n\n**References**\nWu et al. 2022 : Split-kl and PAC-Bayes-split-kl Inequalities for Ternary Random Variables\n\nSTRENGTHS:\nI found the theoretical part of this work utterly interesting as it offers a breath of fresh air in the generalisation field (at least up to my knowledge).\n\nWEAKNESSES:\nI have several concerns about the experimental setup (in particular the comparison with PAC-Bayes theory). See the 'Questions' part.\n\nQUESTIONS:\n- To perform their experiments, authors split MNIST in 60 datasets of size 1000 to enlighten how well works P2L when few data are available, which corresponds to many real-life situations. However, there is other situation where a huge amount of data is available. Thus, I am wondering about the performance of all three methods when trained on all, 60000 data simultaneously. Did the authors try to perform this experiment? \n- Appendix B: is the bound 4.a of Clerico et al. truly the tightest? It seems that Wu et al. 2022 improved on this.\n- I understand that the current experiments aim to express the interest of the proposed meta algorithm, but I remain doubtful about the comparison with the PAC-Bayes learning objective. Indeed, the comparison with the test method is meaningful as it shows how P2L improves on classical GD. Why not taking the PAC-Bayes minimisation routine as a subroutine of P2L and compare it to PAC-Bayes without P2L (plotted in green in this work)?\n- Does it exist similar procedures than P2L in the literature which selects meaningful data?\n- What is the time complexity for plotting the bounds and running P2L comparing to other methods?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present a novel framework called P2L, which aims to derive generalization guarantees for black-box supervised learning algorithms. P2L operates as a meta-algorithm that utilizes a learning algorithm to induce a compression scheme. The algorithm relies on two main components: a *criterion of appropriateness* and an *appropriateness threshold*. The criterion of appropriateness serves as a generalization of the loss function, measuring how well a hypothesis describes a given data point. The appropriateness threshold determines when the meta-algorithm terminates by ensuring that the hypothesis is appropriate for all unselected data points. At each step, the algorithm selects the example with the highest loss, according to the criterion of appropriateness, from the set of selected training data ($T$). This example, denoted as $\\bar{z}$, is added to $T$, and the learning algorithm produces a hypothesis ($h$) based on $T$. Using this new hypothesis, a new $\\bar{z}$ is selected, and the process iterates until h becomes appropriate for all examples in the original dataset ($D_S$). An important property of P2L is that running the algorithm on $T$ yields the same hypothesis as running it on the full dataset $D_S$. Consequently, $T$ effectively compresses $D_S$, similar in spirit to dataset distillation or identifying core examples within a dataset. To provide generalization guarantees, a theorem demonstrates that the cardinality of $T$ can be utilized to derive tight upper bounds on a suitable measure of statistical risk. The effectiveness of P2L is evaluated through experiments on binary MNIST and synthetic regression datasets. The results indicate that P2L outperforms the PAC-Bayes bound and performs competitively with using a hold-out set, all without requiring additional data.\n\nSTRENGTHS:\n- To the best of my knowledge, the proposed algorithm is novel and quite elegant (computational complexity notwithstanding). It is another realization of the principle “compression is intelligence”. \n- I find the technical tools used interesting and believe that this framework can potentially lead to many future works.\n- The resulting bounds are tighter than PAC-Bayes bounds.\n- The presentation of the algorithm is very clean and easy to follow. Most technical concepts are explained carefully and illustrated with examples. This is quite rare for theory papers.\n- The algorithm exhibits some very interesting properties that are potentially desirable for learning algorithms. (*\"the misclassification on the test-set for P2L is constant across all prior/train portions\"*)\n\nWEAKNESSES:\n- In contrast to many existing generalization bounds, P2L can only be used for risk certification, that is, it does not offer an immediate way to make the bound tighter (which may improve the performance of the model) nor does it provide an understanding of why particular algorithms or architecture work well. Of course, this is partially due to the fact that P2L is designed to be as general as possible (i.e., black-box) but depending on the goal, the latter is sometimes more important than the risk certification itself.\n- The algorithm seems not particularly efficient which limits its practicality, especially for deep learning with very large datasets. If the model is trained from scratch at every iteration, the complexity could be quadratic in the number of data points (i.e., $N^2$). On the other hand, it seems more suited for applications where the dataset is small. Perhaps it's better to phrase the paper in that direction. If this is not the case, please correct me. The authors do already address this point in the conclusion but note that PAC-Bayes generalization bounds for deep learning often use Gaussian posterior which is easy to sample from and empirically the estimate concentrates quite fast. Furthermore, these bounds can be made deterministic (Nagarajan et al., 2018).\n- Related to the previous point, the paper would benefit from more empirical evaluation such as those in (Lotfi et al., 2022) who provided the SOTA PAC-Bayes bounds for several benchmarks. The field of PAC-Bayes bounds for deep learning has made significant progress in the past couple of years and only having binary MNIST results makes it hard to judge the empirical value of the proposed algorithm.\n- The paper is generally well-written and clear but there are several important clarifications needed (see questions).\n\n**Reference**\n\n- PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization. Lotfi et al., 2022\n- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience. Nagarajan et al., 2018\n\nQUESTIONS:\n- The role of $h_0$ is somewhat unclear to me from the text. What does \"*pre-training $h_0$ on different portions of the dataset, again through GD*\" mean (line 235)? Why is the model not initialized from scratch? In algorithm 1, where is $h_0$ used? Is it only used for selecting the initial $\\bar{z}$?\n- Why did you use GD instead of SGD? Wouldn’t this result in worse models? How do the bounds compare if you use SGD?\n- In the inner loop, does the training have to be done from scratch or can it rely on the hypothesis from the previous iteration? If it is done from scratch, it feels like the algorithm would be extremely impractical since every iteration has to train a new neural network. If it is done starting from the weights from the previous iteration, wouldn’t the weights be prone to overfitting, especially if the problem is non-convex?\n- Is post-training performance just the test performance? If so, could you elaborate on the following statement “*utilizes all data to jointly learn a good model and provide a risk bound*\"? Do you have a theoretical justification for this statement? It feels like this could have some relationship to curriculum learning or boosting.\n- Is there a relationship between $\\gamma$ and the notion of margin? Why does $\\gamma$ not show up in the bound? Is the dependency implicit in $T$?\n- Is the total ordering determined by the most recent hypothesis $h$ or the meta-algorithm? There seem to be some conflicting statements in the paper. “$[T]_A$ *is a list that contains the elements in* $T$ *ordered according to the order in which they are selected by P2L*” (line 150) but in the conclusion *“a hypothesis-dependent total order used to select which data points are fed to the learning algorithm L.”* (line 346). In a similar vein, how is the ordering used in the experiment since GD does not care about the order of the data at all?\n- Does this framework have a relationship to the algorithmic stability framework?\n\nLIMITATIONS:\nSee the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}