An Exact Characterization of the Generalization Error for the Gibbs Algorithm

1 Introduction . Evaluating the generalization error of a learning algorithm is one of the most important challenges in statistical learning theory . Various approaches have been developed [ 55 ] , including VC dimensionbased bounds [ 66 ] , algorithmic stability-based bounds [ 16 ] , algorithmic robustness-based bounds [ 72 ] , PAC-Bayesian bounds [ 45 ] , and information-theoretic bounds [ 71 ] . However , upper bounds on generalization error may not entirely capture the generalization ability of a learning algorithm . One apparent reason is the tightness issue , some upper bounds [ 9 ] can be far away from the true generalization error or even vacuous when evaluated in practice . More importantly , existing upper bounds do not fully characterize all the aspects that could influence the generalization error of a supervised learning problem . For example , VC dimension-based bounds depend only on the hypothesis class , and algorithmic stability-based bounds only exploit the properties of the learning algorithm . As a consequence , both methods fail to capture the fact that generalization error depends strongly on the interplay between the hypothesis class , learning algorithm , and the ⇤Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . underlying data-generating distribution , as discussed in [ 71 , 73 ] . This paper overcomes the above limitations by deriving an exact characterization of the generalization error for a specific supervised learning algorithm , namely the Gibbs algorithm . 1.1 Problem Formulation . Let S = { Zi } ni=1 be the training set , where each Zi is defined on the same alphabet Z . Note that Zi is not required to be i.i.d generated from the same data-generating distribution PZ , and we denote the joint distribution of all the training samples as PS . We denote the hypotheses by w 2 W , where W is a hypothesis class . The performance of the hypothesis is measured by a non-negative loss function ` : W ⇥ Z ! R+0 , and we can define the empirical risk and the population risk associated with a given hypothesis w as LE ( w , s ) , 1 n nX i=1 ` ( w , zi ) , and LP ( w , PS ) , EPS [ LE ( w , S ) ] , ( 1 ) respectively . A learning algorithm can be modeled as a randomized mapping from the training set S onto an hypothesis W 2 W according to the conditional distribution PW |S . Thus , the expected generalization error quantifying the degree of over-fitting can be written as gen ( PW |S , PS ) , EPW , S [ LP ( W , PS ) LE ( W , S ) ] , ( 2 ) where the expectation is taken over the joint distribution PW , S = PW |S ⌦ PS . In this paper we focus on the generalization error of the Gibbs algorithm ( a.k.a . Gibbs posterior [ 21 ] ) . The ( ↵ , ⇡ ( w ) , f ( w , s ) ) -Gibbs distribution , which was first proposed by [ 28 ] in statistical mechanics and further investigated by [ 35 ] in information theory , is defined as : P ↵ W |S ( w|s ) , ⇡ ( w ) e ↵f ( w , s ) V ( s , ↵ ) , ↵ 0 , ( 3 ) where ↵ is the inverse temperature , ⇡ ( w ) is an arbitrary prior distribution of W , f ( w , s ) is energy function , and V ( s , ↵ ) , R ⇡ ( w ) e ↵f ( w , s ) dw is the partition function . If P and Q are probability measures over space X , and P is absolutely continuous with respect to Q , the Kullback-Leibler ( KL ) divergence between P and Q is given by D ( PkQ ) , R X log ⇣ dP dQ ⌘ dP . If Q is also absolutely continuous with respect to P , then the symmetrized KL divergence ( a.k.a . Jeffrey ’ s divergence [ 36 ] ) is DSKL ( PkQ ) , D ( PkQ ) +D ( QkP ) . ( 4 ) The mutual information between two random variables X and Y is defined as the KL divergence between the joint distribution and product-of-marginal distribution I ( X ; Y ) , D ( PX , Y kPX ⌦ PY ) , or equivalently , the conditional KL divergence between PY |X and PY averaged over PX , D ( PY |XkPY |PX ) , R X D ( PY |X=xkPY ) dPX ( x ) . By swapping the role of PX , Y and PX ⌦ PY in mutual information , we get the lautum information introduced by [ 49 ] , L ( X ; Y ) , D ( PX ⌦ PY kPX , Y ) . Finally , the symmetrized KL information between X and Y is given by [ 6 ] : ISKL ( X ; Y ) , DSKL ( PX , Y kPX ⌦ PY ) = I ( X ; Y ) + L ( X ; Y ) . ( 5 ) Throughout the paper , upper-case letters denote random variables ( e.g. , Z ) , lower-case letters denote the realizations of random variables ( e.g. , z ) , and calligraphic letters denote sets ( e.g. , Z ) . All the logarithms are the natural ones , and all the information measure units are nats . N ( µ , ⌃ ) denotes the Gaussian distribution with mean µ and covariance matrix ⌃ . 1.2 Contributions . The core contribution of this paper ( see Theorem 1 ) is an exact characterization of the expected generalization error for the Gibbs algorithm in terms of the symmetrized KL information between the input training samples S and the output hypothesis W , as follows : gen ( P↵W |S , PS ) = ISKL ( W ; S ) ↵ . This result highlights the fundamental role of such an information quantity in learning theory that does not appear to have been recognized before . We also discuss some general properties of the symmetrized KL information , which could be used to prove the non-negativity and concavity of the expected generalization error for the Gibbs algorithm . Building upon this result , we further expand our contributions in various directions : • In Section 3 , we tighten existing expected generalization error bound ( see Theorem 2 ) and PAC-Bayesian bound ( see Theorem 3 ) for Gibbs algorithm under i.i.d and sub-Gaussian assumptions by combining our symmetrized KL information characterization with the existing bounding techniques . • In Section 4 ( Proposition 1 and 2 ) , we show how to use our method to characterize the asymptotic behavior of the generalization error for Gibbs algorithm under large inverse temperature limit ↵ ! 1 , where Gibbs algorithm converges to the empirical risk minimization algorithm . Note that existing bounds , such as [ 39 , 52 , 71 ] , become vacuous in this regime . • In Section 5 , we characterize the generalization error of the Gibbs algorithm with datadependent regularizer using symmetrized KL information , which provides some insights on how to reduce the generalization error using regularization . 1.3 Motivations for the Gibbs Algorithm . As we discuss below , the choice of Gibbs algorithm is not arbitrary since it arises naturally in many different applications and is sufficiently general to model many learning algorithms used in practice : Empirical Risk Minimization : The ( ↵ , ⇡ ( w ) , LE ( w , s ) ) -Gibbs algorithm can be viewed as a randomized version of the empirical risk minimization ( ERM ) algorithm if we specify the energy function f ( w , s ) = LE ( w , s ) . As the inverse temperature ↵ ! 1 , the prior distribution ⇡ ( w ) becomes negligible , and the Gibbs algorithm converges to the standard ERM algorithm . Information Risk Minimization : The Gibbs algorithm also arises when conditional KL-divergence is used as a regularizer to penalize over-fitting in the information risk minimization framework . In particular , it is shown in [ 71 , 75 , 76 ] that the solution to the following regularized ERM problem P ? W |S = arg inf PW |S ✓ EPW , S [ LE ( W , S ) ] + 1 ↵ D ( PW |Sk⇡ ( W ) |PS ) ◆ , ( 6 ) corresponds to the ( ↵ , ⇡ ( w ) , LE ( w , s ) ) -Gibbs distribution . The inverse temperature ↵ controls the regularization term and balances between over-fitting and generalization . PAC-Bayesian Bound : The following upper bound on population risk from [ 63 ] holds with probability at least 1 for 0 < < 1 , and 0 < < 2 under distribution PS , EPW |S=s [ LP ( W , PS ) ] EPW |S=s [ LE ( W , s ) ] 1 2 + D ( PW |S=sk⇡ ( W ) ) + log ( 2 p n ) ( 1 2 ) n . ( 7 ) If we fix , ⇡ ( w ) and optimize over PW |S=s , the distribution that minimizes the PAC-Bayes bound in ( 7 ) would be the ( n , ⇡ ( w ) , LE ( w , s ) ) -Gibbs distribution . Similar bounds are proposed in [ 21 , Theorem 1.2.1 ] and [ 65 , Lemma 10 ] , where optimizing over posterior distribution would result in a Gibbs distribution . SGLD Algorithm : The Stochastic Gradient Langevin Dynamics ( SGLD ) can be viewed as the discrete version of the continuous-time Langevin diffusion , and it is defined as follows : Wk+1 = Wk rLE ( Wk , s ) + r 2 ↵ ⇣k , k = 0 , 1 , · · · , ( 8 ) where ⇣k is a standard Gaussian random vector and > 0 is the step size . In [ 51 ] , it is proved that under some conditions on loss function , the conditional distribution PWk|S induced by SGLD algorithm is close to ( ↵ , ⇡ ( W0 ) , LE ( wk , s ) ) -Gibbs distribution in 2-Wasserstein distance for sufficiently large k. Under some conditions on the loss function ` ( w , z ) , [ 22 , 42 ] shows that in the continuous-time Langevin diffusion , the stationary distribution of hypothesis W is the Gibbs distribution . 1.4 Other Related Work . Information-theoretic generalization error bounds : Recently , [ 58 , 71 ] propose to use the mutual information between the input training set and the output hypothesis to upper bound the expected generalization error . However , those bounds are known not to be tight , and multiple approaches have been proposed to tighten the mutual information-based bound . [ 19 ] provides tighter bounds by considering the individual sample mutual information , [ 10 , 11 ] propose using chaining mutual information , and [ 30 , 31 , 62 ] advocate the conditioning and processing techniques . Informationtheoretic generalization error bounds using other information quantities are also studied , such as , f -divergence [ 37 ] , ↵-Rényi divergence and maximal leakage [ 25 , 34 ] , Jensen-Shannon divergence [ 7 ] and Wasserstein distance [ 41 , 56 , 69 ] . Using rate-distortion theory , [ 17 , 18 , 43 ] provide informationtheoretic generalization error upper bounds for model misspecification and model compression . PAC-Bayesian generalization error bounds : First proposed by [ 44 , 45 , 60 ] , PAC-Bayesian analysis provides high probability bounds on the generalization error in terms of KL divergence between the data-dependent posterior induced by the learning algorithm and a data-free prior that can be chosen arbitrarily . There are multiple ways to generalize the standard PAC-Bayesian bounds , including using different information measures other than the KL divergence [ 3 , 8 , 14 , 32 , 48 ] and considering datadependent priors ( prior depends on the training data ) [ 5 , 13 , 21 , 23 , 24 , 53 ] or distribution-dependent priors ( prior depends on data-generating distribution ) [ 20 , 40 , 50 , 54 ] . In [ 27 ] , a more general PACBayesian framework is proposed , which provides a high probability bound on the convex function of the expected population and empirical risk with respect to the posterior distribution , whereas in [ 26 ] the connection between Bayesian inference and PAC-Bayesian theorem is explored by considering Gibbs posterior and negative log loss function . Generalization error of Gibbs algorithm : Both information-theoretic and PAC-Bayesian approaches have been used to bound the generalization error of the Gibbs algorithm . An informationtheoretic upper bound with a convergence rate of O ( ↵/n ) is provided in [ 52 ] for the Gibbs algorithm with bounded loss function , and PAC-Bayesian bounds using a variational approximation of Gibbs posteriors are studied in [ 4 ] . [ 11 , Appendix D ] provides an upper bound on the excess risk of the Gibbs algorithm under sub-Gaussian assumption . [ 39 ] focuses on the excess risk of the Gibbs algorithm and a similar generalization bound with rate of O ( ↵/n ) is provided under sub-Gaussian assumption . Although these bounds are tight in terms of the sample complexity n , they become vacuous when the inverse temperature ↵ ! 1 , hence are unable to capture the behaviour of the ERM algorithm . Our work differs from this body of research in the sense that we provide an exact characterization of generalization error of the Gibbs algorithm in terms of the symmetrized KL information . Our work also further leverages this characterization to tighten the existing expected and PAC-Bayesian generalization error bounds in literature .

Summary: This paper shows the generalization error (actually, the population loss minus the empirical  loss) of the Gibbs algorithm is the symmetric KL divergence between the sample distribution and the Gibbs distribution on hypotheses divided by the inverse temperature.  This formulation is then used to tighten existing generalization bounds on the Gibbs algorithm in several settings.  It allows for improved bounds as the inverse temperature $\alpha$ goes to infinity (and the Gibbs algorithm approaches empirical risk minimization), as existing bounds on the Gibbs algorithm seem to behave as $O( \alpha / n)$.   The author's also consider regularized Gibbs algorithms and use their formulation to deduce when the regularizer will be beneficial.