Sharper Generalization Bounds for Learning with Gradient-dominated Objective Functions

1 INTRODUCTION . Stochastic optimization has found tremendous applications in training highly expressive machine learning models including deep neural networks ( DNNs ) ( Bottou et al. , 2018 ) , which are ubiquitous in modern learning architectures ( LeCun et al. , 2015 ) . Oftentimes , the models trained in this way have not only very small training errors or even interpolate the training examples , but also surprisingly generalize well to testing examples ( Zhang et al. , 2017 ) . While the low training error can be well explained by the over-parametrization of models and the efficiency of the optimization algorithm in identifying a local minimizer ( Bassily et al. , 2018 ; Vaswani et al. , 2019 ; Ma et al. , 2018 ) , it is still unclear how the highly expressive models also achieve a low testing error ( Ma et al. , 2018 ) . With the recent theoretical and empirical study , it is believed that a joint consideration of the interaction among the optimization algorithm , learning models and training examples is necessary to understand the generalization behavior ( Neyshabur et al. , 2017 ; Hardt et al. , 2016 ; Lin et al. , 2016 ) . The generalization error for stochastic optimization typically consists of an optimization error and an estimation error ( see e.g . Bousquet & Bottou ( 2008 ) ) . Optimization errors arise from the suboptimality of the output of the chosen optimization algorithms , while estimation errors refer to the discrepancy between the testing error and training error at the output model . There is a large amount of literature on studying the optimization error ( convergence ) of stochastic optimization algorithms ( Bottou et al. , 2018 ; Orabona , 2014 ; Karimi et al. , 2016 ; Ying & Zhou , 2017 ; Liu et al. , 2018 ) . In particular , the power of interpolation is clearly justified in boosting the convergence rate of stochastic gradient descent ( SGD ) ( Bassily et al. , 2018 ; Vaswani et al. , 2019 ; Ma et al. , 2018 ) . In contrast , there is far less work on studying estimation errors of optimization algorithms . In a seminal paper ( Hardt et al. , 2016 ) , the fundamental concept of algorithmic stability was used to study the generalization behavior of SGD , which was further improved and extended in Charles & Papailiopoulos ( 2018 ) ; Zhou et al . ( 2018b ) ; Yuan et al . ( 2019 ) ; Kuzborskij & Lampert ( 2018 ) . ∗Corresponding author : Yiming Ying However , these results are still not quite satisfactory in the following three aspects . Firstly , the existing stability bounds in non-convex learning require very small step sizes ( Hardt et al. , 2016 ) and yield suboptimal generalization bounds ( Yuan et al. , 2019 ; Charles & Papailiopoulos , 2018 ; Zhou et al. , 2018b ) . Secondly , majority of the existing work has focused on functions with a uniform Lipschitz constant which can be very large in practical models if not infinite ( Bousquet & Elisseeff , 2002 ; Hardt et al. , 2016 ; Charles & Papailiopoulos , 2018 ; Kuzborskij & Lampert , 2018 ) , e.g. , DNNs . Thirdly , the existing stability analysis fails to explain how the highly expressive models still generalize in an interpolation setting , which is observed for overparameterized DNNs ( Arora et al. , 2019 ; Brutzkus et al. , 2017 ; Bassily et al. , 2018 ; Belkin et al. , 2019 ) . In this paper , we make attempts to address the above three issues using novel stability analysis approaches . Our main contributions are summarized as follows . 1 . We develop general stability and generalization bounds for any learning algorithm to optimize ( non-convex ) β-gradient-dominated objectives . Specifically , we show that the excess generalization error is bounded by O ( 1/ ( nβ ) ) plus the convergence rate of the algorithm , where n is the sample size . This general theorem implies that overfitting will never happen in this case , and generalization would always improve as we increase the training accuracy , which is due to an implicit regularization effect of gradient dominance condition . In particular , we show that interpolation actually improves generalization for highly expressive models . In contrast to the existing discussions based on either hypothesis stability or uniform stability which imply at best a bound of O ( 1/ √ nβ ) , the main idea is to consider a weaker on-average stability measure which allows us to replace the uniform Lipschitz constant in Hardt et al . ( 2016 ) ; Kuzborskij & Lampert ( 2018 ) ; Charles & Papailiopoulos ( 2018 ) with the training error of the best model . 2 . We apply our general results to various stochastic optimization algorithms , and highlight the advantage over existing generalization analysis . For example , we derive an exponential convergence of testing errors for SGD in an interpolation setting , which complements the exponential convergence of optimization errors ( Bassily et al. , 2018 ; Vaswani et al. , 2019 ; Ma et al. , 2018 ) and extends the existing results ( Pillaud-Vivien et al. , 2018 ; Nitanda & Suzuki , 2019 ) from a strongly-convex setting to a non-convex setting . In particular , we show that stochastic variance-reduced optimization outperforms SGD by achieving a significantly faster convergence of testing errors , while this advantage is only shown in terms of optimization errors in the literature ( Reddi et al. , 2016 ; Lei et al. , 2017 ; Nguyen et al. , 2017 ; Zhou et al. , 2018a ; Wang et al. , 2019 ) . 2 RELATED WORK . Algorithmic Stability . We first review the related work on stability and generalization . Algorithmic stability is a fundamental concept in statistical learning theory ( Bousquet & Elisseeff , 2002 ; Elisseeff et al. , 2005 ) , which has a deep connection with learnability ( Shalev-Shwartz et al. , 2010 ; Rakhlin et al. , 2005 ) . The important uniform stability was introduced in Bousquet & Elisseeff ( 2002 ) , where the authors showed that empirical risk minimization ( ERM ) enjoys the uniform stability if the objective function is strongly convex . This concept was extended to study randomized algorithms such as bagging and bootstrap ( Elisseeff et al. , 2005 ) . An interesting trade-off between uniform stability and convergence was developed for iterative optimization algorithms , which was then used to study convergence lower bounds of different algorithms ( Chen et al. , 2018 ) . While generalization bounds based on stability are often stated in expectation , uniform stability was recently shown to guarantee almost optimal high-probability bounds based on elegant concentration inequalities for weakly-dependent random variables ( Maurer , 2017 ; Feldman & Vondrak , 2019 ; Bousquet et al. , 2020 ) . Other than the standard classification and regression setting , uniform stability was very successfully to study transfer learning ( Kuzborskij & Lampert , 2018 ) , PAC-Bayesian bounds ( London , 2017 ) , privacy learning ( Bassily et al. , 2019 ) and pairwise learning ( Lei et al. , 2020b ) . Some other stability measures include the uniform argument stability ( Liu et al. , 2017 ) , hypothesis stability ( Bousquet & Elisseeff , 2002 ) , hypothesis set stability ( Foster et al. , 2019 ) and on-average stability ( Shalev-Shwartz et al. , 2010 ) . An advantage of on-average stability is that it is weaker than the uniform stability and can imply better generalization by exploiting either the strong convexity of the objective function ( Shalev-Shwartz & Ben-David , 2014 , Corollary 13.7 ) or the more relaxed exp-concavity of loss functions ( Koren & Levy , 2015 ; Gonen & Shalev-Shwartz , 2017 ) . Since gradient-dominance condition is another relaxed extension of strong convexity , we use on-average stability to study generalization bounds . Generalization analysis . We now review related work on generalization analysis for stochastic optimization . In a seminal paper ( Hardt et al. , 2016 ) , the authors used the nonexpansiveness of gradient mapping to develop uniform stability bounds for SGD to optimize convex , strongly convex and even non-convex objective functions . This inspired some interesting work on stochastic optimization . An interesting data-dependent stability bound was developed for SGD , a nice property of which is that it shows how the initialization would affect generalization ( Kuzborskij & Lampert , 2018 ) . These stability bounds were integrated into a PAC-Bayesian analysis of SGD , yielding generalization bounds for arbitrary posterior distributions ( London , 2017 ) . Almost optimal generalization bounds were developed for differentially private stochastic convex optimization ( Bassily et al. , 2019 ) . The onaverage variance of stochastic gradients was used to refine the generalization analysis of SGD ( Hardt et al. , 2016 ) in non-convex optimization ( Zhou et al. , 2018b ) . The uniform stability was also studied for SGD implemented in a stagewise manner ( Yuan et al. , 2019 ) and stochastic gradient Langevin dynamics in a non-convex setting ( Li et al. , 2020 ; Mou et al. , 2018 ) . Very recently , the discussions in Hardt et al . ( 2016 ) were extended to tackle non-smooth ( Lei & Ying , 2020 ; Bassily et al. , 2020 ) and non-Lipscthiz functions ( Lei & Ying , 2020 ) . The most related work is Charles & Papailiopoulos ( 2018 ) , where some general hypothesis stability bounds were developed for learning algorithms that converge to optima . A very interesting point is that their bounds depend only on the convergence of the algorithm to a global minimum and the geometry of loss functions around the global minimum . However , their discussion imply at best the slow generalization bounds O ( 1/ √ nβ ) for β-gradientdominated objective functions , and can not explain the benefit of low optimization errors in helping generalization . The underlying reason is that they used the pointwise hypothesis stability and did not consider the smoothness of loss functions . We aim to improve these results by leveraging the weaker on-average stability and smoothness of loss functions . Other than the stability approach , there is interesting generalization analysis of SGD based on either a uniform convergence approach ( Lin et al. , 2016 ) , an integral operator approach ( Lin & Rosasco , 2017 ; Ying & Pontil , 2008 ; Dieuleveut & Bach , 2016 ; Dieuleveut et al. , 2017 ; Mücke et al. , 2019 ) or an information-theoretic approach ( Xu & Raginsky , 2017 ; Negrea et al. , 2019 ; Bu et al. , 2020 ) . 3 MAIN RESULTS . Let ρ be a probability measure defined on a sample spaceZ = X×Y withX ⊆ Rd andY ⊆ R , from which a training dataset S = { z1 , . . . , zn } is drawn independently and identically . The aim is to find a good model w from a model parameter spaceW based on the training dataset S. The performance of a prescribed model w on a single example z can be measured by a nonnegative loss function f ( w ; z ) , where f : W ×Z 7→ R+ . In machine learning we often apply an ( randomized ) algorithm A : ∪nZn 7→ W to S to produce an output modelA ( S ) ∈ W . Oftentimes , the constructed model w would have a small empirical risk FS ( w ) = 1n ∑n i=1 f ( w ; zi ) . However , we are mostly interested in the generalization performance of a model w on testing examples measured by the population ( true ) risk F ( w ) = Ez [ f ( w ; z ) ] , where Ez denotes the expectation with respect to ( w.r.t . ) z . The gap ES , A [ F ( A ( S ) ) −FS ( A ( S ) ) ] between the population risk and empirical risk is called the estimation error , which is due to the approximation of ρ by sampling . Here EA denotes the expectation w.r.t . the randomness of the algorithm A . For example , if A is SGD , then EA denotes the expectation w.r.t . the random indices of training examples selected for the gradient computation . A powerful tool to study the estimation error is the algorithmic stability ( Bousquet & Elisseeff , 2002 ; Elisseeff et al. , 2005 ; Shalev-Shwartz et al. , 2010 ; Hardt et al. , 2016 ) , which measures the sensitivity of the algorithm ’ s output w.r.t . the perturbation of a training dataset . Below we give formal definitions of stability measures , whose connection to generalization is established in Theorem A.1 . Definition 1 ( Uniform Stability ) . A randomized algorithmA has uniform stability if for all datasets S , S̃ ∈ Zn that differ by at most one example , we have supz EA [ f ( A ( S ) ; z ) − f ( A ( S̃ ) ; z ) ] ≤ . The following on-average stability is similar to the average-RO stability in Shalev-Shwartz et al . ( 2010 ) . The difference is we do not use an absolute value . Form ∈ N , we denote [ m ] = { 1 , . . . , m } . Definition 2 ( On-average Stability ) . Let S = { z1 , . . . , zn } and S̃ = { z̃1 , . . . , z̃n } be drawn independently from ρ . For each i ∈ [ n ] , denote S ( i ) = { z1 , . . . , zi−1 , z̃i , zi+1 , . . . , zn } . We say an algorithm A has on-average stability if 1n ∑n i=1 ES , S̃ , A [ f ( A ( S ( i ) ) ; zi ) − f ( A ( S ) ; zi ) ] ≤ . In this paper , we are interested in the excess generalization error F ( A ( S ) ) − F ( w∗ ) , where w∗ ∈ arg minw∈W F ( w ) is the best model with the least testing error ( population risk ) . For this purpose , we introduce some basic assumptions . A basic assumption in non-convex learning is the smoothness of loss functions ( Ghadimi & Lan , 2013 ; Karimi et al. , 2016 ) , meaning the gradients are Lipschitz continuous . Let ‖ · ‖2 denote the Euclidean norm and∇ denote the gradient operator . Assumption 1 ( Smoothness Assumption ) . We assume for all z ∈ Z , the differentiable function w 7→ f ( w ; z ) is L-smooth , i.e. , ‖∇f ( w ; z ) −∇f ( w′ ; z ) ‖2 ≤ L‖w −w′‖2 for all w , w′ ∈ W . Another assumption is the Polyak-Lojasiewicz ( PL ) condition on the objective function , which is common in non-convex optimization ( Zhou et al. , 2018b ; Reddi et al. , 2016 ; Karimi et al. , 2016 ; Wang et al. , 2019 ; Lei et al. , 2017 ) , and was shown to hold true for deep ( linear ) and shallow neural networks ( Hardt & Ma , 2016 ; Charles & Papailiopoulos , 2018 ; Li & Yuan , 2017 ) . Assumption 2 ( Polyak-Lojasiewicz Condition ) . Denote F̂S = infw′∈W FS ( w′ ) . We assume FS satisfies PL or gradient-dominated condition ( in expectation ) with parameter β > 0 , i.e. , ES [ FS ( w ) − F̂S ] ≤ 1 2β ES [ ‖∇FS ( w ) ‖22 ] , ∀w ∈ W. ( 3.1 ) It is worthy of mentioning that our results in this section continue to hold if the global PL condition is relaxed to a local PL condition , i.e. , ( 3.1 ) holds for w in a neighborhood of the minimizer of FS . The existing stability analysis often imposes a bounded gradient assumption below ( Bousquet & Elisseeff , 2002 ; Hardt et al. , 2016 ; Charles & Papailiopoulos , 2018 ; Yuan et al. , 2019 ; Kuzborskij & Lampert , 2018 ) . Indeed , the resulting stability bounds depend on the uniform Lipschitz constant G ( see eq . ( 3.4 ) ) , which can be prohibitively large in practical models , e.g. , DNNs , or even infinite , e.g . least squares regression in an unbounded domain . Assumption 3 ( Bounded Gradient Assumption ) . We assume ‖∇f ( w ; z ) ‖2 ≤ G for all w ∈ W , z ∈ Z and a constant G > 0 . Our main result to be proved in Appendix B removes Assumption 3 and replaces the uniform Lipschitz constant G by the minimal empirical risk F̂S , which is significantly smaller than the Lipschitz constant . Note the assumption L ≤ nβ/4 is mild , and the previous generalization bounds become vacuous as O ( 1 ) ( Yuan et al. , 2019 ; Charles & Papailiopoulos , 2018 ) if this assumption is violated . Theorem 1 ( Main Theorem ) . Let Assumptions 1 , 2 hold and wS = A ( S ) . If L ≤ nβ/4 , then E [ F ( wS ) − F̂S ] ≤ 16LE [ F̂S ] nβ + LE [ FS ( wS ) − F̂S ] 2β . ( 3.2 ) An important implication is as follows . Since E [ F̂S ] ≤ E [ FS ( w ∗ ) ] = F ( w∗ ) and F̂S ≤ FS ( wS ) , Eq . ( 3.2 ) implies an upper bound on the excess generalization error E [ F ( wS ) ] − F ( w∗ ) and E [ F ( wS ) − FS ( wS ) ] = O ( 1 nβ + E [ FS ( wS ) − F̂S ] β ) . ( 3.3 ) The above two terms can be explained as follows . The term O ( 1/ ( nβ ) ) reflects the intrinsic complexity of the problem , while E [ FS ( wS ) − F̂S ] is called the optimization error . An interesting observation is that the overfitting phenomenon would never happen for learning under the PL condition ( analogous to learning with strongly convex objectives where the global minimizer generalizes well ( Bousquet & Elisseeff , 2002 ) ) . Indeed , if the optimization algorithm finds more and more accurate solutions , it achieves the limiting generalization bound O ( 1/ ( nβ ) ) . This shows an important message that optimization can be beneficial to generalization . This seemingly counterintuitive phenomenon is due to the implicit regularization enforced by the PL condition ( analogous to the strong convexity condition ) . Another notable property is that Theorem 1 applies to any algorithm . We can plug any known optimization error bounds into it to immediately get generalization bounds . Remark 1 . We show that our result significantly improves the existing stability analysis . The work ( Charles & Papailiopoulos , 2018 ) showed the pointwise hypothesis stability is controlled by 2G 2 nβ + 2 √ 2G √ E [ FS ( wS ) − F̂S ] /β , which together with the connection between stability and generalization ( cf . ( A.1 ) ) , implies with probability 1− δ that F ( wS ) ≤ FS ( wS ) + ( M2 nδ + 24MG2 nβδ + 24MG √ 2E [ FS ( wS ) − F̂S ] √ βδ ) 1 2 . ( 3.4 ) The above bound requires the bounded gradient assumption ‖∇f ( w ; z ) ‖2 ≤ G and the bounded loss assumption 0 ≤ f ( w ; z ) ≤ M for all w ∈ W and z ∈ Z , which are successfully removed in our generalization analysis . Furthermore , our generalization bound significantly improves ( 3.4 ) . Indeed , assume E [ FS ( wS ) − F̂S ] ≤ 2β for some > 0 , then ( 3.3 ) implies E [ F ( wS ) ] = E [ FS ( wS ) ] +O ( 1 nβ + 2 ) , ( 3.5 ) while ( 3.4 ) becomes F ( wS ) = FS ( wS ) +O ( 1√ nβ + √ ) . To achieve the generalization guarantee O ( 1/ √ nβ ) , the above bound requires the optimization accuracy = O ( 1/ ( nβ ) ) , while our bound ( 3.5 ) only requires the accuracy = O ( 1/ √ nβ ) but gets the significantly better generalization bound 1/ ( nβ ) . We actually develop a better stability bound . Specifically , the pointwise hypothesis stability is bounded by O ( 1 nβ + ) in Charles & Papailiopoulos ( 2018 ) while we show that the on- average stability is bounded by O ( 1 nβ + 2 ) , which is significantly tighter if 1nβ ≤ ≤ 1 ( ignoring constant factors ) . It should be mentioned that Charles & Papailiopoulos ( 2018 ) did not impose a smoothness assumption . However , the smoothness assumption is widely used in non-convex optimization to derive meaningful rates ( Ghadimi & Lan , 2013 ) . As compared to probabilistic bounds in Charles & Papailiopoulos ( 2018 ) , our bounds are stated in expectation . The extension to highprobability bounds will lead to additional O ( 1/ √ n ) term ( Feldman & Vondrak , 2019 ) . Remark 2 ( Bounded gradient assumption ) . Very recently , the bounded gradient assumption was also removed for the stability analysis ( Lei & Ying , 2020 ) . However , their analysis considered SGD applied to convex loss functions . As a comparison , we study stability and generalization in a nonconvex learning setting , and our analysis applies to any stochastic optimization algorithms . Remark 3 . If A is ERM , Theorem 1 immediately implies E [ F ( wS ) − F̂S ] ≤ 16Lnβ E [ F̂S ] . If FS is β-strongly convex and L < nβ/2 , it was shown for ERM that E [ F ( wS ) − F̂S ] ≤ 48Lnβ E [ F̂S ] ( Shalev-Shwartz & Ben-David , 2014 , Corollary 13.7 ) . Their result is extended here from a strongly convex setting to a gradient-dominated setting , and from the particular ERM to any algorithm . As a direct corollary , we can derive the following optimistic bound in the interpolation setting , which is the most intriguing case for over-parameterized or highly expressive DNN models . Corollary 2 . Let Assumptions 1 , 2 hold and wS = A ( S ) . If E [ F̂S ] = 0 and L < nβ/2 , then E [ F ( wS ) ] ≤ L2βE [ FS ( wS ) ] . Remark 4 . Corollary 2 shows a benefit of interpolation in boosting the generalization by achieving a generalization bound O ( ) for any > 0 if we minimize FS sufficiently well . This benefit can not be explained by the existing discussions ( Hardt et al. , 2016 ; Charles & Papailiopoulos , 2018 ) as they imply the same generalization bound O ( 1/ √ nβ ) in the interpolation setting . Although it was observed that interpolation helps in training ( Bassily et al. , 2018 ; Vaswani et al. , 2019 ; Ma et al. , 2018 ; Oymak & Soltanolkotabi , 2020 ; Allen-Zhu et al. , 2019 ; Zou et al. , 2018 ) , it is still largely unclear , as indicated in Ma et al . ( 2018 ) , that how interpolation helps in generalization . Corollary 2 shows new insights on how interpolation from highly expressive models helps generalization . We now move on to the discussion on the critical assumption in Corollary 2 , i.e . L < nβ/2 . According to the proof , the two parameters L and β can be replaced by their local counterparts , i.e. , the smoothness and PL condition related to a particular minimizer w′ of FS ( i ) ( Eqs . ( B.6 ) , ( B.7 ) ) . For example , β can be replaced by 12‖∇FS ( w ′ ) ‖22/ ( FS ( w′ ) − F̂S ) , which can be larger than β . Below are some examples on explaining L/β < n/2 . As we will see , the quantity L/β reflects the complexity of the problem ( related to condition number as shown in Examples 1 , 2 ) . Therefore , the condition L/β < n/2 imposes implicitly a constraint on the complexity of the problems . This explains why the optimization algorithm would never overfit when applied to gradient-dominated objective functions if L/β < n/2 , as shown in Theorem 1 . Example 1 . Let φ : Rd 7→ Rm be a feature map , and ` : R×R 7→ R+ be a loss function which isL ` smooth and σ ` -strongly convex w.r.t . the first argument . Consider f ( w ; z ) = ` ( 〈w , φ ( xi ) 〉 , yi ) with 〈· , ·〉 being an inner product . Then , FS satisfies the PL condition with the parameter σ′min ( ΣS ) σ ` , where ΣS = 1n ∑n i=1 φ ( xi ) φ ( xi ) > is the empirical covariance matrix , A > denotes the transpose of a matrix A and σ′min ( A ) means the minimal non-zero singular value of A . The empirical counterpart ( we have an expectation w.r.t . S in PL condition ) of L/β is of the order of σmax ( ΣS ) /σ′min ( ΣS ) , where σmax ( A ) means the maximal singular value ( we give details in Appendix E.1 ) . Example 2 . Consider neural networks with a single hidden layer with d inputs , m hidden neurons and a single output neuron , for which the prediction function takes the form hv , w =∑m k=1 vkφ ( 〈wk , x〉 ) . Here wk ∈ Rd and vk ∈ R denote the weight of the edges connecting the k-th hidden node to the input and output node , respectively , while φ : R 7→ R is the activation function . Analogous to Arora et al . ( 2019 ) ; Oymak & Soltanolkotabi ( 2020 ) , we fix v = ( v1 , . . . , vm ) > with |vk| = a for some a > 0 and train w = ( w1 , w2 , . . . , wm ) > ∈ Rm×d from S. The loss function then takes the form f ( w ; z ) = ( v > φ ( wx ) −y ) 2 . If we consider the identity activation function , i.e. , φ ( t ) = t , then FS satisfies the PL condition with the parameter σmin ( ΣS ) , where σmin ( A ) denotes the minimal singular value of A and ΣS = 1n ∑n i=1 xix > i . The empirical counterpart of L/β is of the order of σmax ( ΣS ) /σmin ( ΣS ) ( we give details in Appendix E.2 for a general activation function ) . It is possible to get generalization bounds under some other conditions . Since one-point strong convexity condition together with smoothness assumption implies the PL condition ( Yuan et al. , 2019 ) , all our results apply to one-point strongly convex functions . We can also get generalization bounds for objective functions satisfying the quadratic growth condition ( Necoara et al. , 2018 ) , which is weaker than the PL condition . However , we need to impose a realizability condition which was also imposed in Charles & Papailiopoulos ( 2018 ) . The proof of Theorem 3 is given in Section C. Let w ( S ) denote the Euclidean projection of w onto the set of global minimizers of FS inW . Definition 3 ( Quadratic Growth Condition ) . We say FS : W 7→ R satisfies the quadratic growth condition ( in expectation ) with parameter β if E [ FS ( w ) −F̂S ] ≥ β2E [ ‖w−w ( S ) ‖22 ] for all w ∈ W . Theorem 3 . Let Assumption 1 hold and FS satisfy the quadratic growth condition with parameter β . If the problem is realizable , i.e. , E [ F̂S ] = 0 and L ≤ nβ/4 , then E [ F ( wS ) ] ≤ 2Lβ−1E [ FS ( wS ) ] . Finally , we consider any optimization algorithms applied to gradient-dominated and Lipschitz continuous functions . We do not require loss functions to be smooth here . It shows that the excess generalization bound can decay as fast as O ( 1/ ( nβ ) ) if we solve the optimization problem to a sufficient accuracy , which is much better than the generalization bound O ( 1/ √ nβ ) in Charles & Papailiopoulos ( 2018 ) . Recall the analysis in Charles & Papailiopoulos ( 2018 ) requires Assumptions 2 , 3 and a further assumption on boundedness of loss functions . The proof is given in Section C. Theorem 4 . Let Assumptions 2 , 3 hold and wS = A ( S ) . Then the following inequality holds E [ F ( wS ) − F̂S ] ≤ 2G 2 nβ + G ( E [ FS ( wS ) − F̂S ] ) 1 2 √ 2β .

Summary: This paper mainly studies the generalization performance of stochastic algorithms. Compared with previous results which rely on Lipschitz condition, this paper assumes smoothness condition and Polyak-Lojasiewicz Condition, and then prove the excess generalization bound that is a summation of $\frac{1}{n\beta}$ and empirical optimization error. This result looks impressive, not only because the first term looks shaper than previous $\frac{1}{\sqrt{n}}$ of generalization bound , but also it implies optimization benefits generalization, which may help understand some empirical observations in modern machine learning. What's more, authors analyze some common stochastic algorithms as concrete examples to show the corresponding theoretical guarantee.  Besides, the whole paper is well-written and easy to follow.