How Tight Can PAC-Bayes be in the Small Data Regime?

In this paper , we investigate the question : Given a small number of datapoints , for example N = 30 , how tight can PAC-Bayes and test set bounds be made ? For such small datasets , test set bounds adversely affect generalisation performance by withholding data from the training procedure . In this setting , PAC-Bayes bounds are especially attractive , due to their ability to use all the data to simultaneously learn a posterior and bound its generalisation risk . We focus on the case of i.i.d . data with a bounded loss and consider the generic PAC-Bayes theorem of Germain et al . While their theorem is known to recover many existing PAC-Bayes bounds , it is unclear what the tightest bound derivable from their framework is . For a fixed learning algorithm and dataset , we show that the tightest possible bound coincides with a bound considered by Catoni ; and , in the more natural case of distributions over datasets , we establish a lower bound on the best bound achievable in expectation . Interestingly , this lower bound recovers the Chernoff test set bound if the posterior is equal to the prior . Moreover , to illustrate how tight these bounds can be , we study synthetic one-dimensional classification tasks in which it is feasible to meta-learn both the prior and the form of the bound to numerically optimise for the tightest bounds possible . We find that in this simple , controlled scenario , PAC-Bayes bounds are competitive with comparable , commonly used Chernoff test set bounds . However , the sharpest test set bounds still lead to better guarantees on the generalisation error than the PAC-Bayes bounds we consider . 1 Introduction . Generalisation bounds are of both practical and theoretical importance . Practically , tight bounds provide certificates that algorithms will perform well on unseen data . Theoretically , the bounds and underlying proof techniques can help explain the phenomenon of learning . Among the tightest known bounds are PAC-Bayes ( McAllester , 1999 ) and test set bounds ( Langford , 2002 ) . In this paper , we investigate their numerical tightness when applied to small datasets ( N ≈ 30–60 datapoints ) . The comparison between PAC-Bayes and test set bounds is particularly interesting in this setting as one can not discard data to compute a test set bound without significantly harming post-training performance due to a reduced training set size . PAC-Bayes on the other hand provides valid bounds while using all of the data for learning , since it provides bounds that hold uniformly . The small ∗Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . data setting can also be quite different from the big data setting , as lower-order terms in PAC-Bayes bounds have a non-negligible contribution , and the detailed structure of the bound becomes important . Fortunately , we do not have to study each PAC-Bayes bound separately : remarkably , Germain et al . ( 2009 ) showed that a wide range of bounds can be obtained as special cases of a single generic PAC-Bayes theorem that captures the central ideas of many PAC-Bayes proofs ( see also Bégin et al . ( 2016 ) ) . This theorem has a free parameter : it holds for any convex function , ∆ . By choosing ∆ appropriately , one can recover the well-known bounds of Langford and Seeger ( 2001 ) , Catoni ( 2007 ) and other bounds . We focus on two questions related to this set-up . First , what is the tightest bound achievable by any convex function ∆ ? An answer would characterise the limits of the generic PAC-Bayes theorem , and thereby of a wide range of bounds , by telling us how much improvement could be obtained before new ideas or assumptions are needed . Second , since test set bounds are the de facto standard for larger datasets , but PAC-Bayes has benefits when N is small , we ask : in the small data regime , can PAC-Bayes be tighter than test set bounds ? In Section 3 , Theorem 4 , we show that in the ( artificial ) case when ∆ can be chosen depending on the dataset ( without taking a union bound ) , the tightest version of the generic PAC-Bayes theorem is obtained by one of the Catoni bounds ( Catoni , 2007 ) . In the more realistic case when ∆ must be chosen before sampling the dataset , we do not fully characterise the tightest bound , but in Corollary 3 we lower bound the tightest bound achievable ( in expectation ) with any ∆ . We also provide numerical evidence in Figure 2 that suggests this lower bound can in some cases be attained , by flexibly parameterising a convex function ∆ with a constrained neural network . Interestingly , this lower bound coincides with removing a lower-order term from the Langford and Seeger ( 2001 ) bound ( something that Langford ( 2002 ) conjectured was possible ) , and relaxes to the well-known Chernoff test set bound ( see Theorem 2 below ) when the PAC-Bayes posterior is equal to the prior . In Section 4 , we investigate the tightness of PAC-Bayes and test set bounds in synthetic 1D classification . The goal of this experiment is to find out how tight the bounds could be made in principle . We use meta-learning to adapt all aspects of the bounds and learning algorithms , producing meta-learners that are trained to optimise the value of the bounds on this task distribution . We find that , in this setting , PAC-Bayes can be competitive with the Chernoff test set bound , but is outperformed by the binomial tail test set bound , of which the Chernoff bound is a relaxation . This suggests that , for standard PAC-Bayes to be quantitatively competitive with the best test set bounds on small datasets , a new proof technique leading to bounds that gracefully relax to the binomial tail bound is required . Code to reproduce all experiments can be found at https : //github.com/cambridge-mlg/pac-bayes-tightness-small-data . 2 Background and Related Work . We consider supervised learning . Let X and Y denote the input space and output space , and let Z = X×Y . Assume there is an ( unknown ) probability measure2 D over Z , with the dataset S ∼ DN . Denote the hypothesis space by H ⊆ YX . A learning algorithm is then a map ZN → H. In PACBayes , we also consider maps ZN → M1 ( H ) , where M1 is the set of probability measures on its argument . The performance of a hypothesis h ∈ H is measured by a loss function ℓ : Z ×H → [ 0 , 1 ] . The ( generalisation ) risk of h is RD ( h ) : = E ( x , y ) ∼D [ ℓ ( ( x , y ) , h ) ] and its empirical risk on S is RS ( h ) : = 1 N ∑ ( x , y ) ∈S ℓ ( ( x , y ) , h ) . For Q ∈ M1 ( H ) its ( generalisation Gibbs ) risk is RD ( Q ) : = Eh∼Q [ RD ( h ) ] and its empirical ( Gibbs ) risk is RS ( Q ) : = Eh∼Q [ RS ( h ) ] . In PAC-Bayes , we usually fix a prior P ∈ M1 ( H ) , chosen without reference to S and learn a posterior Q ∈ M1 ( H ) which can depend on S. The KL-divergence between Q and P is defined as KL ( Q‖P ) = ∫ log dQdP dQ if Q ≪ P and ∞ otherwise . Let C denote the set of proper , convex , lower semicontinuous ( l.s.c . ) functions R2 → R ∪ { +∞ } ; if a convex function ’ s domain is a subset of R2 , extend it to all of R2 with the value +∞ . See Appendix C for more details on convex analysis , which we use in Section 3 . Test Set Bounds . Test set bounds rely on a subset of data which is not used to select the hypothesis , called a test set or held-out set . Let S = Strain ∪ Stest , with |S| = N , |Strain| = Ntrain and |Stest| = Ntest . In Theorems 1 and 2 , we assume h is chosen independently of Stest . For the zeroone loss , ℓ ( ( x , y ) , h ) : = ✶ [ h ( x ) 6= y ] , we have that NtestRStest ( h ) is a binomial random variable with parameters ( Ntest , RD ( h ) ) . This leads to the following simple bound , which , for ℓ ∈ { 0 , 1 } , is tight among test set bounds : 2We will colloquially refer to measures on sets without specifying a σ-algebra . We implicitly assume functions are measurable with respect to the σ-algebras on which the relevant measures are defined . Theorem 1 ( Binomial tail test set bound , Langford , 2005 , Theorem 3.3 ) . Let e ( M , k , δ ) : = sup { p : δ ≤ ∑ki=0 ( M i ) pi ( 1−p ) M−i } . For any h ∈ H , ℓ ∈ { 0 , 1 } and δ ∈ ( 0 , 1 ) , Pr ( RD ( h ) ≤ e ( Ntest , NtestRStest ( h ) , δ ) ) ≥ 1− δ . ( 1 ) Often , looser bounds with a simpler form are applied . These can be obtained via the Chernoff method : Theorem 2 ( Chernoff test set bound , Langford , 2005 , Corollary 3.7 ) . For q , p ∈ [ 0 , 1 ] , let kl ( q , p ) : = q log qp + ( 1− q ) log 1−q 1−p . For any h ∈ H , ℓ ∈ [ 0 , 1 ] , and δ ∈ ( 0 , 1 ) , Pr ( kl ( RStest ( h ) , RD ( h ) ) ≤ 1Ntest log 1 δ ) ≥ 1− δ . ( 2 ) PAC-Bayes Bounds . The PAC-Bayes approach bounds the generalisation Gibbs risk of stochastic classifiers , and does not require discarding data , as all the data can be used to choose the posterior , while still obtaining a valid generalisation bound . Since the seminal paper of McAllester ( 1999 ) , a large variety of PAC-Bayes bounds have been derived . Germain et al . ( 2009 ) prove a very general form of the PAC-Bayes theorem which encompasses many of these ( see also Bégin et al . ( 2016 ) and Rivasplata et al . ( 2020 ) ) . Their proof technique consists of a series of inequalities shared by PAC-Bayes proofs ( Jensen ’ s , change of measure , Markov ’ s , supremum over risk3 ) , and reveals their common structure . Thus understanding the properties of this generic theorem can give insight into many PAC-Bayes bounds at once : Theorem 3 ( Generic PAC-Bayes theorem , Bégin et al . ( 2016 ) and Germain et al . ( 2009 ) ) .4 Fix P ∈ M1 ( H ) , ℓ ∈ [ 0 , 1 ] , δ ∈ ( 0 , 1 ) , and ∆ a proper , convex , l.s.c . function [ 0 , 1 ] 2 → R ∪ { +∞ } . Then Pr ( ( ∀Q ) ∆ ( RS ( Q ) , RD ( Q ) ) ≤ 1N [ KL ( Q‖P ) + log I∆ ( N ) δ ] ) ≥ 1− δ , ( 3 ) where I∆ ( N ) : = supr∈ [ 0,1 ] ∑N k=0 ( N k ) rk ( 1− r ) N−keN∆ ( k/N , r ) . Remark 1 . We lose no generality in assuming ∆ ( q , · ) is monotonically increasing for all q ∈ [ 0 , 1 ] , i.e . for any convex ∆ we can define a ∆′ that is monotonically increasing in its second argument and produces a bound that is at least as tight as the bound produced by ∆ . See Appendix D for a proof . Note that the PAC-Bayes bound holds simultaneously for all posteriors Q , and hence is valid even when Q is chosen by minimising the bound . For completeness , we provide a proof of Theorem 3 in Appendix B . Following Germain et al . ( 2009 ) , we briefly recap some of the bounds that can be recovered as special cases ( or looser versions ) of Theorem 3 . Setting ∆ ( q , p ) = Cβ ( q , p ) : = − log ( 1 + p ( e−β − 1 ) ) − βq for β > 0 , we recover the Catoni bounds : Corollary 1 ( Catoni , 2007 , Theorem 1.2.6 ) . For any β > 0 , Pr ( ( ∀Q ) RD ( Q ) ≤ 11−e−β [ 1− exp ( −βRS ( Q ) − 1N ( KL ( Q‖P ) + log 1δ ) ) ] ) ≥ 1− δ . ( 4 ) This specifies a bound for every value of β > 0 . If we instead choose ∆ ( q , p ) = kl ( q , p ) , we obtain the bound of Langford and Seeger ( 2001 ) , also called the PAC-Bayes-kl bound , but with the slightly sharper dependence on N established by Maurer ( 2004 ) : Corollary 2 ( Langford and Seeger , 2001 , Theorem 3 , Maurer , 2004 , Theorem 5 ) . Pr ( ( ∀Q ) kl ( RS ( Q ) , RD ( Q ) ) ≤ 1N [ KL ( Q‖P ) + log 2 √ N δ ] ) ≥ 1− δ . ( 5 ) Corollary 2 is actually very slightly looser than Theorem 3 with ∆ = kl , since Maurer ( 2004 ) upper bounds Ikl ( N ) by 2 √ N using Stirling ’ s formula.5 The Catoni and PAC-Bayes-kl bounds are among the tightest PAC-Bayes bounds known and have been applied in settings where numerical tightness is key , such as obtaining generalisation bounds for stochastic neural networks ( Dziugaite & Roy , 2017 ; Zhou et al. , 2019 ) . Many other bounds can be obtained by loosening these bounds . Applying Pinsker ’ s inequality kl ( q , p ) ≥ 2 ( q − p ) 2 to Equation ( 5 ) yields the “ square-root ” version of the 3The supremum over risk step was introduced in Bégin et al . ( 2016 ) , although for certain ∆ it can be omitted . 4We state a simpler version of their result WLOG , absorbing a free parameter into the function ∆ . 5Maurer ( 2004 ) only proves this bound for N ≥ 8 , but the cases where 1 ≤ N ≤ 7 can be easily verified numerically ( Germain et al. , 2015 , Lemma 19 ) . PAC-Bayes theorem ( McAllester , 1999 , 2003 ) . The “ PAC-Bayes-λ ” ( Thiemann et al. , 2017 ) and “ PAC-Bayes-quadratic ” bounds ( Rivasplata et al. , 2019 ) can be derived as loosened versions of the PAC-Bayes-kl bound using the inequality kl ( q , p ) ≥ ( q − p ) 2/ ( 2p ) , valid for q < p. The “ linear ” bound in McAllester ( 2013 ) can be derived by loosening the Catoni bound using : Cβ ( q , p ) ≤ A =⇒ p ≤ 11−β/2 ( q + 1βA ) , which is valid for β ∈ ( 0 , 2 ) . How Tight Are PAC-Bayes Bounds ? A fundamental question we can ask about a generalisation bound is how tight it is , and whether it can be tightened . Comparing the PAC-Bayes-kl and Chernoff test set bounds when Q = P ( so the PAC-Bayes bound essentially becomes a test set bound ) shows they are identical except for a log ( 2 √ N ) /N on the RHS of the PAC-Bayes-kl bound . Whether this term ( or similar discrepancies between PAC-Bayes and Occam bounds ( Langford , 2002 , Corollary 4.6.2 ) ; see Appendix A ) can be removed has been an open question since Langford ( 2002 , Problem 6.1.2 ) . Maurer ( 2004 ) reduced this term to its current form , improving on work by Langford and Seeger ( 2001 ) . Interestingly , Germain et al . ( 2009 , Proposition 2.1 ) shows that the expression obtained by dropping log ( 2 √ N ) /N from the PAC-Bayes-kl bound is identical to that obtained by illegally6 minimising the Catoni bound with respect to β ; Catoni ( 2007 , Theorem 1.2.8 ) shows that a union bound can be used to , in a legal way , approximately optimise with respect to β at the cost of an additional lower order term . The Chernoff test set bound is itself a looser version of the binomial tail bound , raising the question of whether a PAC-Bayes bound can be found that reduces to the binomial tail bound when Q = P . We provide new insights into these problems in Section 3 . Researchers have also compared PAC-Bayes bounds numerically on actual learning problems . Langford ( 2005 ) and Germain et al . ( 2009 ) were able to obtain reasonable guarantees on small datasets . However , Langford ( 2005 ) found that on datasets with N ≈ 145 , PAC-Bayes was outperformed by test set bounds . Dziugaite and Roy ( 2017 ) , Langford and Caruana ( 2001 ) , and Pérez-Ortiz et al . ( 2021 ) provide non-vacuous bounds for neural networks using PAC-Bayes . Even so , Dziugaite et al . ( 2021 ) states that tighter bounds would be obtained using a test set instead . In Section 4 we find that if the bounds and learning algorithms are optimised for a task distribution , PAC-Bayes can be tight enough to compete with the Chernoff test set bound , but not the binomial tail test set bound .

Summary: This paper proposes to study the best numerical tightness of PAC-Bayes bounds, it comparison with test set bounds. The main motivation given is that in small sample problems one cannot afford to hold out a subset of the data to compute a test set bound.   The main theoretical results are Theorem 4 with its Corollary 3, presented in Sec. 3. In Thm 4 the dataset is fixed, and the best divergence function is sought for the general PAC-Bayes bound, which produces the tightest bound. This turns out to be one of Catoni's bounds. In Cor 3 the data set is no longer fixed, and the authors find that Catoni's bounds can no longer achieve the tightest bound, and the authors conjecture the "optimistic MLS bound" is a tight lower bound, which they show to be a tight lower bound. Apparently the conjecture was previously stated by Langford. While the present paper still leaves this open, it makes some progress in terms of insights. The theoretical investigation is complemented with numerical experiments, which also add insight.  Furthermore, Sec 4. presents extensive experimental work addressing the question of which PAC-Bayes bounds are tightest and how they compare with the test set bound, by optimising the bounds numerically, with respect to the algorithm and the task. Results from the experiments are presented in Figs 3 and 4, and several interesting observations are made, including that PAC-Bayes can be as tight as the test-set bound, and that the "optimistic MLS bound" is indistinguishable from Catoni's. While it is indeed interesting to see that PAC-Bayes bounds can be really tight, of course it doesn't mean they will be tight when applied on some given task and some given algorithm.          The main outcomes of the study are summarised in Sec 5. in the for of 2 open problems: 1. echoes the one of Langford, 2. asks for the possibility that a PAC-Bayes bound be as tight (at its best) as the tightest set set bound.