Towards a Unified Information-Theoretic Framework for Generalization

1 Introduction . In this work , we study the expressiveness of generalization bounds in terms of information-theoretic measures of dependence between the output of a learning algorithm and input data . Informationtheoretic techniques for proving generalization bounds are powerful ; they can provide generalization ∗Part of the work was done the author was an intern at Element AI , a ServiceNow company . †This work was carried out while the author was at ServiceNow . It was finalized at Google Brain . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . bounds that are algorithm-dependent , data-dependent , and distribution-dependent . This approach was initiated by Russo and Zou [ 6 , 7 ] and Xu and Raginsky [ 8 ] and has since been extended by a number of authors [ 9–19 ] . More recently , attention has shifted to whether these techniques can also characterize worst-case ( minimax ) rates for certain learning problems . Let D be an unknown distribution on a space Z , and let H be a set of classifiers . Consider a ( randomized ) learning algorithm A = ( An ) n≥1 that selects an element ĥ in H , based on n i.i.d . samples Sn ∼ D⊗n , i.e. , ĥ = ĥn = An ( Sn ) . The initial focus of this line of work was on the mutual information I ( An ( Sn ) ; Sn ) between the input and the output of a learning problem . This quantity is sometimes referred to as the input–output mutual information ( IOMI ) of an algorithm and denoted by IOMID ( An ) . A natural question is whether the IOMI framework can provide a sharp characterization of the learnability of Vapnik–Chervonenkis ( VC ) classes , for which we have strong generalization guarantees . A negative resolution was provided by Bassily et al . [ 12 ] for the concept class of thresholds in one dimension . Follow up work by Nachum et al . [ 13 ] extended the argument in [ 12 ] , proving the following result : Theorem 1.1 ( Thm . 1 , [ 13 ] ) . For every d ∈ N and every n ≥ 2d2 , there exists a finite input space X and a concept classH ⊆ { 0 , 1 } X of VC-dimension d such that , for all proper and consistent learning algorithmAn , there exists a realizable distribution D such that IOMID ( An ) = Ω ( d log log ( |X |/d ) ) . Livni and Moran [ 18 ] extended this result even further , showing that , for the class of one-dimensional thresholds over { 1 , . . . , m } , m ∈ N,3 for every learning algorithm A there exists a realizable distribution such that either the risk ( population loss ) is large or the IOMID ( An ) scales with the cardinality of the space , m. These results highlight an important limitation of the IOMI framework : given an unbounded input space , for any “ good ” learning algorithm there are always scenarios in which IOMID ( An ) is unbounded . Therefore , the distribution-free learnability of VC classes can not be expressed using the IOMI framework . In this paper , we focus on the “ conditional mutual information ” ( CMI ) framework , proposed by Steinke and Zakynthinou [ 1 ] . In order to reason about the generalization error of a learning algorithm , they introduce a super sample that contains the training sample as a random subset and compute the mutual information between the input and output conditional on the super sample ( formal definitions are provided in Section 2.1 ) . Improvements of this framework and its application in studying the generalization of specific learning algorithms have been studied in [ 20–25 ] . The current paper revolves around the following fundamental question : For which learning problems and learning algorithms is the CMI framework expressive enough to accurately estimate the generalization error ? We will focus in particular on whether we can recover optimal worst case ( minimax ) rates for VC classes satisfying certain properties . The answer to these question provide evidence that the CMI framework provides a unifying framework for studying generalization . For VC classes , Steinke and Zakynthinou [ 1 ] revealed a stark separation between the CMI framework and IOMI framework . They showed the existence of an empirical risk minimization ( ERM ) algorithm whose CMI is no larger than d log n + 2 for learning every VC class of dimension d given n i.i.d . training samples . In contrast to Theorem 1.1 , CMI does not scale with the cardinality of the space . However , the bound on the CMI combined with Steinke and Zakynthinou ’ s CMI-based generalization bound , leads to a bound on the expected excess risk that is suboptimal in some cases by a log n factor . ( For an overview of the known bounds for learning VC classes , please refer to Appendix A . ) The suboptimality of their bound prompted Steinke and Zakynthinou [ 3 ] to conjecture that the CMI bound for proper learners of VC classes can be improved to O ( d ) . Moreover , Steinke and Zakynthinou connected CMI framework to the sample compression framework of [ 26 ] by showing that a sample compressionAn of size k has CMID ( An ) ≤ k log 2n . Their bound for sample compression schemes is also suboptimal in some cases by a log n factor . 1.1 Contributions . In this paper we extend the reach of the CMI framework by demonstrating its unifying nature for obtaining optimal or near-optimal bounds for the expected excess risk of the various algorithms in the realizable setting . 3This concept class can be defined as follows . Let X = { 1 , ... , m } . Let k ∈ N and hk : X → { 0 , 1 } define as hk ( x ) = 1 [ x > k ] . Then , the class of one-dimensional thresholds over { 1 , ... , m } is Hm = { hk|k ∈ N } . 1 . We demonstrate that one can use the CMI framework to express non-trivial ( but sub-optimal ) bounds for every improper learning algorithm that outputs a hypothesis from a class with a bounded VC dimension . This is achieved by an empirical variant of CMI defined by [ 1 ] . 2 . We study the CMI of SVMs for learning half spaces and show that the CMI framework yields optimal bounds on the expected excess risk . Our bound on the CMI of SVM is an application of our general result giving optimal CMI bounds for stable sample compression schemes [ 2 , 27 ] , which improve on CMI bounds for general sample compression schemes [ 1 ] by a log n factor . 3 . In the context of proper learning of VC classes , we exhibit VC classes for which the CMI of any proper learner can not be bounded by any real-valued function of the VC dimension . Then , we consider VC classes with finite star number [ 4 ] , and prove the existence of a learner with bounded CMI . Finally , we show that the release of the set of all consistent classifiers in H has bounded CMI if and only if H has finite star number . 4 . We show that CMI framwork is universal in the realizable setting . More precisely , for every data distribution and consistent learner , the bound on excess risk obtained by the CMI framework vanishes if and only if the excess risk also vanishes as the number training samples diverges . We then show that any learning algorithm with a “ leave-one-out ” bound of order O ( 1/n ) yields an evaluated-CMI bound of order O ( log n ) . As an application , we study the classical one-inclusion graph algorithm of Haussler et al . [ 5 ] for improper learning of VC classes , and provide a nearly optimal bound on its expected excess risk using the CMI framework . We also prove there exists a randomized one-inclusion graph which learns point functions ( singleton ) with bounded CMI . Our results indicate that CMI is a very expressive generalization framework , and one that can tie together existing frameworks . Although most of our results are stated for binary classification in the distribution-free setting , it is interesting to note that the CMI framework is known to provide numerically non-vacuous generalization error guarantees for some modern deep learning models and datasets in the distribution-dependent setting [ 21 , 25 ] . These developments in a range of different problem settings highlight the importance of understanding the expressiveness of the CMI framework . 2 Preliminaries . We consider the problem of binary classification , with inputs in some space X assigned labels in Y = { 0 , 1 } . A concept ( or hypothesis ) class H ⊆ YX is a set of functions h : X → Y . We say H shatters ( x1 , . . . , xm ) ∈ Xm if for all ( y1 , . . . , ym ) ∈ { 0 , 1 } m , there exists h ∈ H , such that , for all i ∈ [ m ] , we have h ( xi ) = yi . The VC dimension ofH , denoted by d , is the largest m ∈ N for which there exists ( x1 , . . . , xm ) ∈ Xm shattered byH . If no such finite m exists , then d =∞ . Let D be a distribution on Z = X × Y . The empirical ( classification ) risk of a classifier h : X → Y on a sample s = ( ( x1 , y1 ) , . . . , ( xn , yn ) ) ∈ Zn is R̂s ( h ) = n−1 ∑ i∈ [ n ] ` ( h , ( xi , yi ) ) , where ` ( h , ( x , y ) ) = 1 [ h ( x ) 6= y ] . Let Sn ∼ Dn , i.e. , let Sn be a sequence of i.i.d . random elements in Z with common distribution D. ( We can view Sn itself as a random element in Zn . ) The risk of h is RD ( h ) = ER̂Sn ( h ) , where E denotes the expectation operator . ( The risk has , of course , no dependence on n due to the data being i.i.d . ) A distribution D is realizable by a class H ⊆ YX if there exists h ∈ H such that RD ( h ) = 0 . A sequence ( ( x1 , y1 ) , . . . , ( xn , yn ) ) is said to be realizable by H , if for some h ∈ H , h ( xi ) = yi for all i ∈ [ n ] = { 1 , . . . , n } . Note that if a distribution is realizable byH , it implies that with probability one over Sn ∼ Dn , the training sample Sn is realizable byH . Let A = ( An ) n≥1 denote a ( potentially randomized ) learning algorithm , which , for any positive integer n , maps Sn to an element of X → Y . We say that A is a proper learner for a class H ⊆ X → Y if the codomain ofAn is a subset ofH for every n. We sayAn is a consistent algorithm ( learner ) if R̂Sn ( An ( Sn ) ) = 0 a.s. Our primary interest in this paper is the expected generalization error of An with respect to D , defined as EGED ( An ) = E [ RD ( An ( Sn ) ) − R̂Sn ( An ( Sn ) ) ] , where we average over both the choice of training sample and the randomness within the algorithm An .

Summary: This work investigates the expressiveness of the CMI framework introduced in [1]. The main contributions of this paper are:  1. The authors show that an empirical variant of CMI can be easily used to express bounds every improper learning.  2. The authors apply their general CMI bound on SVM, and improve previous results by a logarithmic factor.  3. VC classes for which proper learning cannot be bounded by any function of the VC dimension are considered 4. They prove a general reduction showing that the leave-one-out is expressible via the CMI framework.