Stability based Generalization Bounds for Exponential Family Langevin Dynamics

√ n ) dependence . Second , we introduce Exponential Family Langevin Dynamics ( EFLD ) which is a substantial generalization of SGLD and which allows exponential family noise to be used with gradient descent . We establish data-dependent expected stability based generalization bounds for general EFLD . Third , we consider an important new special case of EFLD : noisy sign-SGD , which extends sign-SGD by using Bernoulli noise over { −1 , +1 } , and we establish optimization guarantees for the algorithm . Further , we present empirical results on benchmark datasets to illustrate the our bounds are non-vacuous and quantitatively much sharper than existing bounds . 1 INTRODUCTION . Recent years have seen renewed interest and advances in characterizing generalization performance of learning algorithms in terms of stability , which considers change in performance of a learning algorithm based on change of a single training point ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ; Li et al. , 2020 ; Mou et al. , 2018 ) . For stochastic gradient descent ( SGD ) , Hardt et al . ( 2016 ) established generalization bounds based on uniform stability ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ) , although the analysis needed rather small step sizes ηt = 1/t which is not useful in practice . While improving the analysis for SGD has remained a challenge , advances have been made on noisy SGD algorithms , especially stochastic gradient Langevin dynamics ( SGLD ) ( Welling & Teh , 2011 ; Mou et al. , 2018 ; Li et al. , 2020 ) , which adds Gaussian noise to the stochastic gradients of marginal variance σ2t . In parallel , there has been key developments on related information-theoretic generalization bounds applicable to SGLD type algorithms ( Negrea et al. , 2019 ; Haghifam et al. , 2020 ; Xu & Raginsky , 2017 ; Russo & Zou , 2016 ; Pensia et al. , 2018 ) . While these developments have led to major advances in analyzing generalization of noisy SGD algorithms , they each have certain limitations which leave room for further improvements . Using uniform stability , Mou et al . ( 2018 ) established a bound for SGLD of the form Kn √∑ t η 2 t /σ 2 t which depends on K , the global Lipschitz constant for the loss , and with step size ηt ≤ σt ln 2/K . The bound has a desirable dependency of O ( 1/n ) on the samples , but has an undesirable dependence on K , and the step sizes , bounded by σt/K , are too small . Mou et al . ( 2018 ) also presented another bound which addresses some of these issues , but gets an undesirable O ( 1/ √ n ) sample dependence . By building on the developments of Russo & Zou ( 2016 ) ; Xu & Raginsky ( 2017 ) ; Pensia et al . ( 2018 ) , Negrea et al . ( 2019 ) made advances from the information theoretic perspective and established bounds for SGLD which have the desirable dependence on the norm of gradient incoherence , i.e. , difference in gradients over different mini-batches , avoids dependence on Lipschitz constant K , and is applicable to unbounded sub-Gaussian losses , but have an undesirable O ( 1/ √ n ) sample dependence . Haghifam et al . ( 2020 ) made further advances on the problem from an information theoretic perspective based on the conditional mutual information framework of Steinke & Zakynthinou ( 2020 ) and obtained generalization bounds based on gradient incoherence with O ( 1/n ) sample dependence , but their analysis holds for full batch Langevin dynamics , not mini-batch SGLD . Li et al . ( 2020 ) made advances on such bounds based on the notion of Bayes-stability , by combining ideas from PAC-Bayes bounds into stability , and established a bound of the form cn √∑ t η 2 t ge ( t ) /σ 2 t for bounded losses , where ge ( t ) is the expected gradient norm square at step t. While the bound avoids dependency on the Lipschitz constant K , the dependence on the gradient norm makes such bounds much weaker than the information theoretic bounds of Negrea et al . ( 2019 ) ; Haghifam et al . ( 2020 ) which depend on the norm of gradient incoherence , which are typically orders of magnitude smaller . Further , the analysis of Li et al . ( 2020 ) still needs small step sizes , bounded by σt/K . In this paper , we build on the core strengths of such existing approaches , most notably the O ( 1/n ) sample dependence of stability based bounds ( Mou et al. , 2018 ; Li et al. , 2020 ) and the dependence on gradient incoherence for information theoretic bounds ( Negrea et al. , 2019 ; Haghifam et al. , 2020 ) , and develop a framework ( Section 2 ) for developing generalization bounds for noisy stochastic iterative ( NSI ) algorithms . Our framework considers generalization based on the concept of expected stability , rather than uniform stability ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ; Bousquet et al. , 2020 ; Mou et al. , 2018 ) , which yields distribution dependent generalization bounds and avoids the worst-case setting of uniform stability . Building on Li et al . ( 2020 ) , we show that expected stability of general NSI algorithms can be bounded by the expected Le Cam Style Divergence with dependence on parameter distributions from mini-batches differing by one sample . In Section 3 , we introduce Exponential Family Langevin Dynamics ( EFLD ) , a family of noisy gradient descent algorithms based on exponential family noise . Special cases of EFLD include SGLD and noisy versions of Sign-SGD or quantized SGD algorithms widely used in practice ( Bernstein et al. , 2018a ; b ; Jin et al. , 2020 ; Alistarh et al. , 2017 ) Our main result provides an expected stability based generalization bound applicable to any EFLD algorithm with a O ( 1/n ) sample dependence and a dependence on gradient incoherence , rather than gradient norms . Existing generalization bounds for SGLD ( Li et al. , 2020 ; Negrea et al. , 2019 ) usually use properties of the Gaussian distribution , and our analysis on EFLD illustrates that this was unnecessary . We also consider optimization guarantees for EFLD and establish such results for noisy Sign-SGD and SGLD . Through experiments on benchmark datasets ( Section 4 ) , we illustrate that our bounds are non-vacuous and quantitatively much sharper than existing bounds ( Li et al. , 2020 ; Negrea et al. , 2019 ) . Related work . Uniform stability has been a classical approach for bounding generalization error ( Bousquet & Elisseeff , 2002 ; Bousquet et al. , 2020 ; Feldman & Vondrak , 2018 ; 2019 ) , pioneered by Rogers & Wagner ( 1978 ) ; Devroye & Wagner ( 1979 ) . Beyond the aforementioned work , there has been recent work on differential privacy that analyzes the uniform stability of differentially private SGD ( DP-SGD ) ( Hardt et al. , 2016 ; Bassily et al. , 2020 ) . Beyond uniform stability , informationtheoretic approaches ( Russo & Zou , 2016 ; Xu & Raginsky , 2017 ) that bounds the generalization error by the mutual information between the algorithm input S and the algorithm output w , have been used for deriving generalization bounds for noisy iterative algorithms ( Pensia et al. , 2018 ; Bu et al. , 2019 ) . Along this line of literature , Negrea et al . ( 2019 ) ; Haghifam et al . ( 2020 ) ; Rodrı́guezGálvez et al . ( 2021 ) prove data-dependent generalization bounds dropping dependence on the Lipschitz constant . Further , tighter bounds ( Haghifam et al. , 2020 ; Zhou et al. , 2021 ; Rodrı́guez-Gálvez et al. , 2021 ; Neu , 2021 ; Hellström & Durisi , 2021 ) are proposed based on conditional mutual information ( Steinke & Zakynthinou , 2020 ; Grünwald et al. , 2021 ; Hellström & Durisi , 2020 ) . Due to space limitations , an extended discussion of the related work is deferred to Appendix A . 2 GENERALIZATION BOUNDS WITH EXPECTED STABILITY . In the setting of statistical learning , there is an instance space Z , a hypothesis spaceW , and a loss function ` : W×Z 7→ R+ . LetD be an unknown distribution ofZ and let S ∼ Dn be n i.i.d . draws from D. For any specific hypothesis w ∈ W , the population and empirical loss are respectively given by LD ( w ) , Ez∼D [ ` ( w , z ) ] , and LS ( w ) , 1n ∑n i=1 ` ( w , zi ) . For any distribution P over the hypothesis space , we respectively denote the expected population and empirical loss as LD ( P ) , Ez∼DEw∼P [ ` ( w , z ) ] , and LS ( P ) , 1 n n∑ i=1 Ew∼P [ ` ( w , zi ) ] . ( 1 ) Consider a randomized algorithm A which works with S = { z1 , . . . , zn } ∼ Dn and cre- ates a distribution over the hypothesis space W . For convenience , we will denote the distribution as A ( S ) . The focus of the analysis is to bound the generalization error of A defined as : gen ( A ( S ) ) , LD ( A ( S ) ) − LS ( A ( S ) ) . We will assume A is permutation invariant , i.e. , the ordering of samples in S do not modify A ( S ) , an assumption satisfied by most learning algorithms . We will focus on developing bounds for the expectation ES [ LD ( A ( S ) ) − LS ( A ( S ) ) ] , and discuss high-probability bounds in the Appendix B . 2.1 BOUNDS BASED ON EXPECTED STABILITY . We start our analysis by noting that the expected generalization error can be upper bounded by expected stability based on the Hellinger divergence ( Sason & Verdu , 2016 ; Li et al. , 2020 ) : H2 ( P‖P ′ ) = 12 ∫ w ( √ p ( w ) − √ p′ ( w ) ) 2dw . Proposition 1 . Let Sn ∼ Dn and let S′n be a dataset obtained by replacing zn ∈ Sn with z′n ∼ D. Let A ( Sn ) , A ( S′n ) respectively denote the distributions over the hypothesis space W obtained by running randomized algorithm A on Sn , S′n . Assume that for Sn ∼ Dn , ∀z ∈ Z , EW∼A ( Sn ) [ ` 2 ( W , z ) ] ≤ c0/2 , c0 > 0 . With H ( · , · ) denoting the Hellinger divergence , we have |ESn∼Dn [ LD ( A ( Sn ) ) − LS ( A ( Sn ) ) ] | ≤ c0ESn∼DnEz′n∼D √ 2H2 ( A ( Sn ) , A ( S′n ) ) . ( 2 ) Remark 2.1 . Proposition 1 does not need bounded losses . Just the second moment of ` ( W , z ) , W ∼ A ( Sn ) , Sn ∼ Dn , ∀z ∈ Z need to be bounded . The assumption is satisfied by bounded losses . It is instructive to compare the assumption to that in recent information theoretic bounds ( Haghifam et al. , 2020 ; Xu & Raginsky , 2017 ) , where one assumes ` ( w , Z ) , Z ∼ D , ∀w ∈ W to be sub-Gaussian . Remark 2.2 . The bound in Proposition 1 is in terms of expected stability where we consider ES∼DnEz′n∼D [ · · · ] , an important departure from bounds based on uniform stability ( Elisseeff et al. , 2005 ; Bousquet & Elisseeff , 2002 ; Mou et al. , 2018 ; Bousquet et al. , 2020 ; Feldman & Vondrak , 2018 ; 2019 ) where one considers supS , S′∈Zn , |S\S′|=1 [ · · · ] . Replacing sup by E makes the bounds distribution dependent , and arguably leads to quantitatively tighter bounds . Note that the Hellinger divergence can be bounded by the KL divergence . Proposition 2 . For any distributions P and P ′ , 2H2 ( P , P ′ ) ≤ min { KL ( P , P ′ ) , √ 1 2KL ( P , P ′ ) } .

Summary: This paper studies the generalization of iterative noisy learning algorithms. First of all, the authors provide a new generalization bound based on the Hellinger distance. This bound is basically based on the Hellinger distance between the output of the algorithm when one of the points in the training set is replaced with a fresh sample. They they use this bound to provide a generalization error guarantee for a broad class of noisy iterative algorithm which is defined as W_t = W_{t-1} - eta_t, where eta_t conditioned on the past iterated follows an exponential family distribution which for instance include SGLD. For this broad class, they provide a unified generalization bound that includes SGLD and Noisy Sign-SGD. Finally, the authors provide numerical study of their bound for several benchmarks and show that their bound achieves better estimate of generalization error.