Stability based Generalization Bounds for Exponential Family Langevin Dynamics

√ n ) dependence . Second , we introduce Exponential Family Langevin Dynamics ( EFLD ) which is a substantial generalization of SGLD and which allows exponential family noise to be used with gradient descent . We establish data-dependent expected stability based generalization bounds for general EFLD . Third , we consider an important new special case of EFLD : noisy sign-SGD , which extends sign-SGD by using Bernoulli noise over { −1 , +1 } , and we establish optimization guarantees for the algorithm . Further , we present empirical results on benchmark datasets to illustrate the our bounds are non-vacuous and quantitatively much sharper than existing bounds . 1 INTRODUCTION . Recent years have seen renewed interest and advances in characterizing generalization performance of learning algorithms in terms of stability , which considers change in performance of a learning algorithm based on change of a single training point ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ; Li et al. , 2020 ; Mou et al. , 2018 ) . For stochastic gradient descent ( SGD ) , Hardt et al . ( 2016 ) established generalization bounds based on uniform stability ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ) , although the analysis needed rather small step sizes ηt = 1/t which is not useful in practice . While improving the analysis for SGD has remained a challenge , advances have been made on noisy SGD algorithms , especially stochastic gradient Langevin dynamics ( SGLD ) ( Welling & Teh , 2011 ; Mou et al. , 2018 ; Li et al. , 2020 ) , which adds Gaussian noise to the stochastic gradients of marginal variance σ2t . In parallel , there has been key developments on related information-theoretic generalization bounds applicable to SGLD type algorithms ( Negrea et al. , 2019 ; Haghifam et al. , 2020 ; Xu & Raginsky , 2017 ; Russo & Zou , 2016 ; Pensia et al. , 2018 ) . While these developments have led to major advances in analyzing generalization of noisy SGD algorithms , they each have certain limitations which leave room for further improvements . Using uniform stability , Mou et al . ( 2018 ) established a bound for SGLD of the form Kn √∑ t η 2 t /σ 2 t which depends on K , the global Lipschitz constant for the loss , and with step size ηt ≤ σt ln 2/K . The bound has a desirable dependency of O ( 1/n ) on the samples , but has an undesirable dependence on K , and the step sizes , bounded by σt/K , are too small . Mou et al . ( 2018 ) also presented another bound which addresses some of these issues , but gets an undesirable O ( 1/ √ n ) sample dependence . By building on the developments of Russo & Zou ( 2016 ) ; Xu & Raginsky ( 2017 ) ; Pensia et al . ( 2018 ) , Negrea et al . ( 2019 ) made advances from the information theoretic perspective and established bounds for SGLD which have the desirable dependence on the norm of gradient incoherence , i.e. , difference in gradients over different mini-batches , avoids dependence on Lipschitz constant K , and is applicable to unbounded sub-Gaussian losses , but have an undesirable O ( 1/ √ n ) sample dependence . Haghifam et al . ( 2020 ) made further advances on the problem from an information theoretic perspective based on the conditional mutual information framework of Steinke & Zakynthinou ( 2020 ) and obtained generalization bounds based on gradient incoherence with O ( 1/n ) sample dependence , but their analysis holds for full batch Langevin dynamics , not mini-batch SGLD . Li et al . ( 2020 ) made advances on such bounds based on the notion of Bayes-stability , by combining ideas from PAC-Bayes bounds into stability , and established a bound of the form cn √∑ t η 2 t ge ( t ) /σ 2 t for bounded losses , where ge ( t ) is the expected gradient norm square at step t. While the bound avoids dependency on the Lipschitz constant K , the dependence on the gradient norm makes such bounds much weaker than the information theoretic bounds of Negrea et al . ( 2019 ) ; Haghifam et al . ( 2020 ) which depend on the norm of gradient incoherence , which are typically orders of magnitude smaller . Further , the analysis of Li et al . ( 2020 ) still needs small step sizes , bounded by σt/K . In this paper , we build on the core strengths of such existing approaches , most notably the O ( 1/n ) sample dependence of stability based bounds ( Mou et al. , 2018 ; Li et al. , 2020 ) and the dependence on gradient incoherence for information theoretic bounds ( Negrea et al. , 2019 ; Haghifam et al. , 2020 ) , and develop a framework ( Section 2 ) for developing generalization bounds for noisy stochastic iterative ( NSI ) algorithms . Our framework considers generalization based on the concept of expected stability , rather than uniform stability ( Hardt et al. , 2016 ; Bousquet & Elisseeff , 2002 ; Bousquet et al. , 2020 ; Mou et al. , 2018 ) , which yields distribution dependent generalization bounds and avoids the worst-case setting of uniform stability . Building on Li et al . ( 2020 ) , we show that expected stability of general NSI algorithms can be bounded by the expected Le Cam Style Divergence with dependence on parameter distributions from mini-batches differing by one sample . In Section 3 , we introduce Exponential Family Langevin Dynamics ( EFLD ) , a family of noisy gradient descent algorithms based on exponential family noise . Special cases of EFLD include SGLD and noisy versions of Sign-SGD or quantized SGD algorithms widely used in practice ( Bernstein et al. , 2018a ; b ; Jin et al. , 2020 ; Alistarh et al. , 2017 ) Our main result provides an expected stability based generalization bound applicable to any EFLD algorithm with a O ( 1/n ) sample dependence and a dependence on gradient incoherence , rather than gradient norms . Existing generalization bounds for SGLD ( Li et al. , 2020 ; Negrea et al. , 2019 ) usually use properties of the Gaussian distribution , and our analysis on EFLD illustrates that this was unnecessary . We also consider optimization guarantees for EFLD and establish such results for noisy Sign-SGD and SGLD . Through experiments on benchmark datasets ( Section 4 ) , we illustrate that our bounds are non-vacuous and quantitatively much sharper than existing bounds ( Li et al. , 2020 ; Negrea et al. , 2019 ) . Related work . Uniform stability has been a classical approach for bounding generalization error ( Bousquet & Elisseeff , 2002 ; Bousquet et al. , 2020 ; Feldman & Vondrak , 2018 ; 2019 ) , pioneered by Rogers & Wagner ( 1978 ) ; Devroye & Wagner ( 1979 ) . Beyond the aforementioned work , there has been recent work on differential privacy that analyzes the uniform stability of differentially private SGD ( DP-SGD ) ( Hardt et al. , 2016 ; Bassily et al. , 2020 ) . Beyond uniform stability , informationtheoretic approaches ( Russo & Zou , 2016 ; Xu & Raginsky , 2017 ) that bounds the generalization error by the mutual information between the algorithm input S and the algorithm output w , have been used for deriving generalization bounds for noisy iterative algorithms ( Pensia et al. , 2018 ; Bu et al. , 2019 ) . Along this line of literature , Negrea et al . ( 2019 ) ; Haghifam et al . ( 2020 ) ; Rodrı́guezGálvez et al . ( 2021 ) prove data-dependent generalization bounds dropping dependence on the Lipschitz constant . Further , tighter bounds ( Haghifam et al. , 2020 ; Zhou et al. , 2021 ; Rodrı́guez-Gálvez et al. , 2021 ; Neu , 2021 ; Hellström & Durisi , 2021 ) are proposed based on conditional mutual information ( Steinke & Zakynthinou , 2020 ; Grünwald et al. , 2021 ; Hellström & Durisi , 2020 ) . Due to space limitations , an extended discussion of the related work is deferred to Appendix A . 2 GENERALIZATION BOUNDS WITH EXPECTED STABILITY . In the setting of statistical learning , there is an instance space Z , a hypothesis spaceW , and a loss function ` : W×Z 7→ R+ . LetD be an unknown distribution ofZ and let S ∼ Dn be n i.i.d . draws from D. For any specific hypothesis w ∈ W , the population and empirical loss are respectively given by LD ( w ) , Ez∼D [ ` ( w , z ) ] , and LS ( w ) , 1n ∑n i=1 ` ( w , zi ) . For any distribution P over the hypothesis space , we respectively denote the expected population and empirical loss as LD ( P ) , Ez∼DEw∼P [ ` ( w , z ) ] , and LS ( P ) , 1 n n∑ i=1 Ew∼P [ ` ( w , zi ) ] . ( 1 ) Consider a randomized algorithm A which works with S = { z1 , . . . , zn } ∼ Dn and cre- ates a distribution over the hypothesis space W . For convenience , we will denote the distribution as A ( S ) . The focus of the analysis is to bound the generalization error of A defined as : gen ( A ( S ) ) , LD ( A ( S ) ) − LS ( A ( S ) ) . We will assume A is permutation invariant , i.e. , the ordering of samples in S do not modify A ( S ) , an assumption satisfied by most learning algorithms . We will focus on developing bounds for the expectation ES [ LD ( A ( S ) ) − LS ( A ( S ) ) ] , and discuss high-probability bounds in the Appendix B . 2.1 BOUNDS BASED ON EXPECTED STABILITY . We start our analysis by noting that the expected generalization error can be upper bounded by expected stability based on the Hellinger divergence ( Sason & Verdu , 2016 ; Li et al. , 2020 ) : H2 ( P‖P ′ ) = 12 ∫ w ( √ p ( w ) − √ p′ ( w ) ) 2dw . Proposition 1 . Let Sn ∼ Dn and let S′n be a dataset obtained by replacing zn ∈ Sn with z′n ∼ D. Let A ( Sn ) , A ( S′n ) respectively denote the distributions over the hypothesis space W obtained by running randomized algorithm A on Sn , S′n . Assume that for Sn ∼ Dn , ∀z ∈ Z , EW∼A ( Sn ) [ ` 2 ( W , z ) ] ≤ c0/2 , c0 > 0 . With H ( · , · ) denoting the Hellinger divergence , we have |ESn∼Dn [ LD ( A ( Sn ) ) − LS ( A ( Sn ) ) ] | ≤ c0ESn∼DnEz′n∼D √ 2H2 ( A ( Sn ) , A ( S′n ) ) . ( 2 ) Remark 2.1 . Proposition 1 does not need bounded losses . Just the second moment of ` ( W , z ) , W ∼ A ( Sn ) , Sn ∼ Dn , ∀z ∈ Z need to be bounded . The assumption is satisfied by bounded losses . It is instructive to compare the assumption to that in recent information theoretic bounds ( Haghifam et al. , 2020 ; Xu & Raginsky , 2017 ) , where one assumes ` ( w , Z ) , Z ∼ D , ∀w ∈ W to be sub-Gaussian . Remark 2.2 . The bound in Proposition 1 is in terms of expected stability where we consider ES∼DnEz′n∼D [ · · · ] , an important departure from bounds based on uniform stability ( Elisseeff et al. , 2005 ; Bousquet & Elisseeff , 2002 ; Mou et al. , 2018 ; Bousquet et al. , 2020 ; Feldman & Vondrak , 2018 ; 2019 ) where one considers supS , S′∈Zn , |S\S′|=1 [ · · · ] . Replacing sup by E makes the bounds distribution dependent , and arguably leads to quantitatively tighter bounds . Note that the Hellinger divergence can be bounded by the KL divergence . Proposition 2 . For any distributions P and P ′ , 2H2 ( P , P ′ ) ≤ min { KL ( P , P ′ ) , √ 1 2KL ( P , P ′ ) } .

Summary: The paper under review considers the generalized SDLG algorithms with exponential family noise introduced to the mini-batch gradient descent. The analysis of the error bound is based on the control of the expected stability. Experiments are provided to support the proposed EFLD SGD algorithms.