Boosting the Confidence of Near-Tight Generalization Bounds for Uniformly Stable Randomized Algorithms

1 INTRODUCTION . In many statistical machine learning problems , the ultimate goal is to design a suitable algorithm A : ZN 7→ W that maps a training data set S = { zi } i∈ [ N ] ∈ ZN to a modelA ( S ) in a closed subset W of an Euclidean space such that the following population risk function evaluated at the model is as small as possible : R ( A ( S ) ) : = EZ∼D [ ` ( A ( S ) ; Z ) ] , where ` : W × Z 7→ R+ is a non-negative bounded loss function whose value ` ( w ; z ) measures the loss evaluated at z with parameter w , and D represents a distribution over Z . It is generally the case that the underlying data distribution is unknown , and in this case the training data is usually assumed to be an i.i.d . set , i.e. , S i.i.d.∼ DN . Then , a natural alternative measurement that mimics the computationally intractable population risk is the empirical risk defined by RS ( A ( S ) ) : = EZ∼Unif ( S ) [ ` ( A ( S ) ; Z ) ] = 1 N N∑ i=1 ` ( A ( S ) ; zi ) . The bound on the difference between population and empirical risks is of central interest in understanding the generalization performance of learning algorithm A . Particularly , we hope to derive a suitable law of large numbers , i.e. , a sample size vanishing rate bN such that the generalization bound |RS ( A ( S ) ) −R ( A ( S ) ) | . bN holds with high probability over the randomness of S and potentially the randomness of A as well . Provided that A ( S ) is an almost minimizer of the empirical risk function RS , say RS ( A ( S ) ) . minw∈W RS ( w ) + bN , the generalization bound immediately implies an excess risk bound R ( A ( S ) ) −minw∈W R ( w ) . bN + 1√N due to the standard risk decomposition and Hoeffding ’ s inequality . Therefore , generalization bounds also play a crucial role in understanding the stochastic optimization performance of a learning algorithm . The generalization bounds can be naturally implied by uniform bounds on supw∈W |R ( w ) − RS ( w ) | ( Bartlett et al. , 2006 ; Bottou & Bousquet , 2008 ) . While broadly applicable ( e.g. , to nonconvex problems ) and leading to tight generalization in some specific regimes ( e.g. , margin-based learning ( Kakade et al. , 2009 ) ) , uniform convergence bounds in general cases might suffer from the polynomial dependence on dimensionality and thus are not suitable for high-dimensional models which are ubiquitous in modern machine learning . Alternatively , a powerful proxy for analyzing the generalization bounds is the stability of learning algorithms to changes in the training dataset . Since the seminal work of Bousquet & Elisseeff ( 2002 ) , stability has been extensively demonstrated to beget dimension-independent generalization bounds for deterministic learning algorithms ( Mukherjee et al. , 2006 ; Shalev-Shwartz et al. , 2010 ) , as well as for randomized learning algorithms ( such as bagging and SGD ) ( Elisseeff et al. , 2005 ; Hardt et al. , 2016 ) . So far , the best known results about generalization bounds are offered by approaches based on the notion of uniform stability ( Feldman & Vondrak , 2018 ; 2019 ; Bousquet et al. , 2020 ) . Inspired by these recent breakthrough results , we seek to derive sharper high-probability generalization bounds for randomized learning algorithms with on-average uniform stability . A concrete working example of our study is the widely used stochastic gradient descent ( SGD ) algorithm that carries out the following recursion for all t ≥ 1 with learning rate ηt > 0 : wt : = ΠW ( wt−1 − ηt∇w ` ( wt−1 ; zξt ) ) , ( 1 ) where ξt ∈ [ N ] is a uniform random index of data under with or without replacement sampling , and ΠW is the Euclidean projection operator associated withW . Despite its popularity in the study of stability theory ( Hardt et al. , 2016 ; Zhou et al. , 2019 ; Lei & Ying , 2020 ) , the high probability generalization bounds of SGD are still relatively under explored through the lens of uniform stability . 1.1 PRIOR RESULTS . Let us now briefly review some state-of-the-art generalization bounds for uniformly stable algorithms and their randomized variants . We denote by S .= S′ if a pair of data sets S and S′ differ in a single data point . A randomized learning algorithm A is said to have on-average γN -uniform stability if it satisfies the following uniform bound sup S . =S′ , z∈Z |EA [ ` ( A ( S ) ; z ) ] − EA [ ` ( A ( S′ ) ; z ) ] | ≤ γN . This definition is equivalent to the concept of uniform stability defined over the on-average loss EA [ ` ( A ( S ) ; z ) ] . Suppose that the loss function is Lipschitz and bounded in the interval ( 0 , 1 ] . Then essentially it has been shown in Feldman & Vondrak ( 2019 ) that for any δ ∈ ( 0 , 1 ) , with probability at least 1− δ over S , the on-average generalization error is upper bounded by |EA [ R ( A ( S ) ) −RS ( A ( S ) ) ] | . γN log ( N ) log ( N δ ) + √ log ( 1/δ ) N . ( 2 ) Recently , Bousquet et al . ( 2020 ) derived a slightly improved uniform stability bound that implies |EA [ R ( A ( S ) ) −RS ( A ( S ) ) ] | . γN log ( N ) log ( 1 δ ) + √ log ( 1/δ ) N . ( 3 ) These generalization bounds are near-tight ( up to a logarithmic factor log ( N ) ) in the sense of a lower bound on sum of random functions provided in that paper . While sharp in the dependence on sample size , one common limitation of the above uniform stability implied generalization bounds lies in that these high-probability results only hold in expectation with respect to the internal randomness of algorithm . Further suppose that A has γN -uniform stability with probability at least 1− δ′ for some δ′ ∈ ( 0 , 1 ) over the randomness of A , i.e. , PA { sup S . =S′ , z∈Z | ` ( A ( S ) ; z ) − ` ( A ( S′ ) ; z ) | ≤ γN } ≥ 1− δ′ . ( 4 ) Suppose that the randomness of A is independent of the training set S. Then with probability at least 1− δ − δ′ over S and A , the bound of Bousquet et al . ( 2020 ) naturally implies |R ( A ( S ) ) −RS ( A ( S ) ) | . γN log ( N ) log ( 1 δ ) + √ log ( 1/δ ) N . ( 5 ) Algorithm 1 : Confidence-Boosting for Randomized Learning Input : A randomized learning algorithm A and a training data set S = { zi } i∈ [ N ] i.i.d.∼ DN . Output : Ak∗ ( Sk∗ ) . Uniformly divide S into K disjoint subsets such that S = ⋃ k∈ [ K ] Sk and |Sk| = N K , ∀k ∈ [ K ] . for k = 1 , 2 , ... , K do Estimate Ak ( Sk ) as an output of the randomized algorithm A over subset Sk . end Compute k∗ = arg mink∈ [ K ] ∣∣RS\Sk ( Ak ( Sk ) ) −RSk ( Ak ( Sk ) ) ∣∣ . This is by far the best known generalization bound of randomized stable algorithms that hold with high probability jointly over data and algorithm . The result , however , relies heavily on the highprobability uniform stability condition expressed in equation 4 . For the SGD ( see equation 1 ) with fixed learning rate ηt ≡ η , it is possible to show that γN . η √ T + ηTN and δ ′ = N exp ( −N2 ) in equation 4 ( Bassily et al. , 2020 ) . For SGD with time decaying learning rate that has been widely studied in theory Harvey et al . ( 2019 ) ; Rakhlin et al . ( 2012 ) and applied in practice for training popular deep nets such as ResNet and DenseNet Bengio et al . ( 2017 ) , it is not clear if the condition in equation 4 is still valid for γN and δ′ of interest . On the other hand , it is possible to show ( see the proofs of Corollary 1 and 2 ) that SGD with time decaying learning rate has desirable on-average uniform stability . More specially for randomized learning methods such as bagging ( Breiman , 1996 ) and SGD equation 1 , the randomness of algorithm can be precisely characterized by a vector of i.i.d . parameters ξ = { ξ1 , ... , ξT } which are independent on data S. In such cases , suppose that A ( S ; ξ ) has uniform stability with respect to ξ at any given S , i.e. , supξ .=ξ′ | ` ( A ( S ; ξ ) ) − ` ( A ( S ; ξ′ ) ) | ≤ ρT . Then the high probability bound established in Elisseeff et al . ( 2005 ) shows that with probability at least 1−δ , |R ( A ( S ) ) −RS ( A ( S ) ) | . γN + ( 1 +NγN√ N + √ TρT ) √ log ( 1 δ ) . ( 6 ) Provided that γN . 1N and ρT . 1 T , the above bound shows that the generalization bound scales as O ( 1√ N + 1√ T ) with high probability . However , the above bound will show no guarantee on convergence if γN & 1√N and/or ρT & 1√ T . For example , this is actually the case for SGD with time decaying learning rate ηt = O ( 1t ) on non-convex loss functions in which γN . √ T N and ρT can scale as large as O ( 1 ) . Open problem and motication . Keeping the merits and deficiencies of above recalled prior results in mind , it still remains an open issue if the existing deviation bounds can hold with high confidence for randomized algorithms with on-average uniform stability ( such as SGD with decaying learning rates ) . The goal of the present study is to derive sharper high-probability generalization bounds for randomized algorithms that hold jointly over the randomness of data and algorithm , based on the relatively weaker notion of on-average uniform stability rather than its high probability counterpart . 1.2 OVERVIEW OF OUR RESULTS . The confidence-boosting technique of Schapire ( 1990 ) is a classical meta approach that allows us to boost the dependence of a learning algorithm on the failure probability δ from 1/δ to log ( 1/δ ) , at a certain cost of computational complexity . The fundamental contribution of our work is to reveal that the confidence-boosting trick yields near-tight high probability generalization bounds for uniformly stable randomized learning algorithms . The novelty lies in a refined analysis of the in-expectation first moment generalization error bound for a randomized learning algorithm with onaverage uniform stability , which leads to improved high-probability generalization bounds over the randomness of data and algorithm via confidence-boosting . More specifically , given a randomized learning algorithm A , we consider the following subbagging process over training set : Boosting the Confidence via Subbagging . We independently run A over K disjoint and uniformly divided training subsets { Sk } k∈ [ K ] to obtain solutions { Ak ( Sk ) } k∈ [ K ] . Then we e- valuate the validation error of each candidate solution over its complementary training subset , and output Ak∗ ( Sk∗ ) that has the smallest gap between training error and validation error , i.e. , k∗ = arg mink∈ [ K ] ∣∣RS\Sk ( Ak ( Sk ) ) −RSk ( Ak ( Sk ) ) ∣∣ . Specially when A is deterministic , this reduces to a standard subbagging process , namely a variation of bagging using without-replacement sampling for subsets generation ( see , e.g. , Andonova et al. , 2002 ) . In general , this is essentially a subbagging procedure with greedy model selection for randomized algorithms over multiple disjoint subsets . Here we have assumed without loss of generality that N is a multiplier of K. The considered procedure of confidence-boosting for randomized learning is outlined in Algorithm 1 . Main Results . In what follows , we highlight our main results on the generalization bounds of the output of Algorithm 1 along with the implications for SGD and deterministic algorithms : • General results . Suppose that the loss is Lipschitz and bounded in the range of ( 0 , 1 ] . Our main result in Theorem 1 show that for any δ ∈ ( 0 , 1 ) , setting K log ( 1δ ) yields the following generalization bound of the output of Algorithm 1 that holds with probability at least 1 − δ over S and { Ak } k∈ [ K ] : |R ( Ak∗ ( Sk∗ ) ) −RS ( Ak∗ ( Sk∗ ) ) | . 1 K ( √ γm2 , NK + γm , NK + √ K N ) + √ log ( K/δ ) N , where γm , N and γm2 , N are respectively mean-uniform stability and mean-square-uniform stability bounds introduced in Definition 1 . In contrast to the bound in equation 6 , our bound is not relying on the uniform stability with respect to the random bits of algorithm . • Stronger generalization bounds for SGD via confidence-boosting . We then use our general results to study the benefit of confidence-boosting on the generalization bounds of SGD-w ( SGD via with-replacement sampling as outlined in Algorithm 2 ) . The main results are a series of corollaries of Theorem 1 when substantialized to SGD with smooth ( Corollary 1 ) or non-smooth ( Corollary 2 ) convex loss , and smooth non-convex loss functions ( Corollary 3 ) . For an instance , our result in Corollary 1 showcases that when invoked to SGD-w on smooth convex loss with learning rates ηt = O ( 1√t ) , the generalization bound of the output of Algorithm 1 with K log ( 1 δ ) is upper bounded by |R ( ASGD-w , k∗ ( Sk∗ ) ) −RS ( ASGD-w , k∗ ( Sk∗ ) ) | . √ log ( T ) N + √ T + √ N log ( 1/δ ) N . Compared with theO ( √ T N ) in-expectation bound of smooth convex SGD ( Hardt et al. , 2016 ) , our above bound is competitive in order while it holds with high confidence . • Sharper bounds for uniformly stable algorithms via confidence-boosting . Specially for a deter- ministic learning algorithm A that has γN -uniform stability for each data set size N ≥ 1 , we further show through Corollary 4 that the following bound holds for the output of Algorithm 1 with K = log ( 1δ ) : |R ( A ( Sk∗ ) ) −RS ( A ( Sk∗ ) ) | . γN K + √ log ( K/δ ) N . In the case of γN . 1√N which holds in some popular learning paradigms such as regularized ERM , the above bound implies ( recall K log ( 1δ ) ) |R ( A ( Sk∗ ) ) −RS ( A ( Sk∗ ) ) | . √ log ( 1/δ ) N , which is sharper than the best known result in equation 5 ( under δ′ = 0 ) in the sense of the removal of a log ( N ) factor . This is a side contribution of our work that might be of independent interest towards answering an open question raised by Bousquet et al . ( 2020 ) . In addition to the generalization bounds , we have also derived a high probability excess risk bound for uniformly stable randomized learning with confidence-boosting . More specifically , with a proper modification of the output of Algorithm 1 , we can show that with probability at least 1 − δ over S and { Ak } k∈ [ K ] : R ( Ak∗ ( Sk∗ ) ) − min w∈W R ( w ) . γm , NK + ∆opt + √ log ( K/δ ) N , where ∆opt is the in-expectation empirical risk optimization error as given by equation 7 .

Summary:  This paper derives near-optimal high probability generalization bounds for randomized learning algorithms. To be specific, the authors establish an improved high probability generalization error bounds over the randomness of data and algorithm via confidence boosting method. The main novelty is they establish the in-expectation first moment generalization bound with mean (square) uniform stability for the randomized algorithm and then use subbagging technique to obtain the high-confidence bounds. In particular, their results can handle time-varying stepsize cases when applied to SGD.  The ideas are motivated by the confidence-boosting/bagging ideas in the literature.