Title: $L_2$-Uniform Stability of Randomized Learning Algorithms: Sharper Generalization Bounds and Confidence Boosting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Uniform stability

Randomized learning algorithms

Confidence boosting

Generalization bounds

Stochastic gradient methods
Abstract: Exponential generalization bounds with near-optimal rates have recently been established for uniformly stable algorithms~\citep{feldman2019high,bousquet2020sharper}. We seek to extend these best known high probability bounds from deterministic learning algorithms to the regime of randomized learning. One simple approach for achieving this goal is to define the stability for the expectation over the algorithm's randomness, which may result in sharper parameter but only leads to guarantees regarding the on-average generalization error. Another natural option is to consider the stability conditioned on the algorithm's randomness, which is way more stringent but may lead to generalization with high probability jointly over the randomness of sample and algorithm. The present paper addresses such a tension between these two alternatives and makes progress towards relaxing it inside a classic framework of confidence-boosting. To this end, we first introduce a novel concept of $L_2$-uniform stability that holds uniformly over data but in second-moment over the algorithm's randomness. Then as a core contribution of this work, we prove a strong exponential bound on the first-moment of generalization error under the notion of $L_2$-uniform stability. As an interesting consequence of the bound, we show that a bagging-based meta algorithm leads to near-optimal generalization with high probability jointly over the randomness of data and algorithm. We further substantialize these generic results to stochastic gradient descent (SGD) to derive sharper exponential bounds for convex or non-convex optimization with natural time-decaying learning rates, which have not been possible to prove with the existing stability-based generalization guarantees.

L 2 -Uniform Stability of Randomized Learning Algorithms: Sharper Generalization Bounds and Confidence Boosting

Abstract

Exponential generalization bounds with near-optimal rates have recently been established for uniformly stable algorithms (Feldman and Vondrák, 2019; Bousquet et al., 2020). We seek to extend these best known high probability bounds from deterministic learning algorithms to the regime of randomized learning. One simple approach for achieving this goal is to define the stability for the expectation over the algorithm's randomness, which may result in sharper parameter but only leads to guarantees regarding the on-average generalization error. Another natural option is to consider the stability conditioned on the algorithm's randomness, which is way more stringent but may lead to generalization with high probability jointly over the randomness of sample and algorithm. The present paper addresses such a tension between these two alternatives and makes progress towards relaxing it inside a classic framework of confidence-boosting. To this end, we first introduce a novel concept of L 2 -uniform stability that holds uniformly over data but in secondmoment over the algorithm's randomness. Then as a core contribution of this work, we prove a strong exponential bound on the first-moment of generalization error under the notion of L 2 -uniform stability. As an interesting consequence of the bound, we show that a bagging-based meta algorithm leads to near-optimal generalization with high probability jointly over the randomness of data and algorithm. We further substantialize these generic results to stochastic gradient descent (SGD) to derive sharper exponential bounds for convex or non-convex optimization with natural time-decaying learning rates, which have not been possible to prove with the existing stability-based generalization guarantees.

1 Introduction

In many statistical learning problems, we are interested in designing a randomized algorithm A : Z N ×R↦→ W that maps a training data sample S = { Z i } i ∈ [ N ] ∈ Z N with an algorithm's random parameter ξ ∈ R to a model A ( S, ξ ) ∈ W . Here Z and R are some measurable sets, and W is a closed subset of an Euclidean space. The ultimate goal is to find a suitable algorithm such that the following population risk evaluated at the model should be as small as possible:

where Z ∈ Z and ℓ : W×Z ↦→ R + is a non-negative bounded loss function whose value ℓ ( w ; z ) measures the loss evaluated at z with parameter w . It is generally the case that the underlying data distribution is unknown, and in this case the data points Z i are usually assumed to be independent.

Ping Li

Then, a natural alternative measurement that mimics the computationally intractable population risk is the empirical risk given by

The bound on the difference between the population and empirical risks is of central interest in understanding the generalization performance of a learning algorithm. In particular, we hope to derive a suitable law of large numbers, i.e., a sample size vanishing rate b N such that the generalization bound | R S ( A ( S, ξ )) -R ( A ( S, ξ )) | ≲ b N holds with high probability over the randomness of S and hopefully the randomness of ξ as well. Let R ∗ := min w ∈W R ( w ) be the optimal value of the population risk. Conditioned on S , suppose that A ( S, ξ ) is an almost minimizer of the empirical risk R S such that R S ( A ( S, ξ )) -min w ∈W R S ( w ) ≤ ε , then the generalization bound immediately implies an excess risk bound R ( A ( S, ξ )) -R ∗ ≲ b N + 1 √ N + ε based on the standard risk decomposition and Hoeffding's inequality. Therefore, generalization guarantees also play a crucial role in understanding the stochastic optimization performance of a learning algorithm.

A powerful proxy for analyzing the generalization bounds is the stability of learning algorithms to changes in the training dataset. Since the seminal work of Bousquet and Elisseeff (2002), stability has been extensively demonstrated to beget dimension-independent generalization bounds for deterministic learning algorithms (Mukherjee et al., 2006; Shalev-Shwartz et al., 2010), as well as for randomized learning algorithms such as bagging and SGD (Elisseeff et al., 2005; Hardt et al., 2016). So far, the best known results about generalization bounds are offered by approaches based on the notion of uniform stability (Feldman and Vondrák, 2018, 2019; Bousquet et al., 2020; Klochkov and Zhivotovskiy, 2021) which is independent to the underlying distribution of data. For randomized algorithms, the definition of uniform stability can be extended in two natural ways by respectively considering 1) the stability averaged over the algorithm's randomness (Hardt et al., 2016) and 2) the stability conditioned on the algorithm's randomness (Feldman and Vondrák, 2019). The former is simpler to show but typically leads to on-average generalization bounds, while the latter is relatively more stringent but may yield deviation bounds given that the conditional stability holds with high probability over the algorithm's randomness. Between these two extreme cases, however, the generalization behavior of randomized learning algorithm still remains largely under explored.

To address the above mentioned theoretical gap between the current lines of results, we explore the opportunities of deriving exponential generalization bounds for randomized learning algorithms beyond the notions of on-average stability and conditional stability . A concrete working example of our study is the widely used stochastic gradient descent (SGD) algorithm that carries out the following recursion for all t ≥ 1 with learning rate η t > 0 :

where i t ∈ [ N ] is a random index of data under with or without replacement sampling, and Π W is the Euclidean projection operator associated with W . The in-expectation generalization of SGD has been studied under on-average stability (Hardt et al., 2016; Zhou et al., 2022; Lei and Ying, 2020), while the exponential bounds have recently been established given that the stability holds with high probability over the sampling path of SGD (Feldman and Vondrák, 2019; Bassily et al., 2020).

1.1 Prior results

Let us start by briefly reviewing some state-of-the-art exponential generalization bounds under the notion of uniform stability and its randomized variants. We denote by S . = ˜ S if a pair of data sets S and ˜ S differ in a single element. A randomized learning algorithm A is said to have on-average γ N -uniform stability (Elisseeff et al., 2005) if it satisfies the following uniform bound:

This definition is equivalent to the concept of uniform stability defined for the expectation of loss E ξ [ ℓ ( A ( S, ξ ); Z )] . Suppose that the loss function is bounded in the interval [0 , M ] . Then essentially it has been shown in Feldman and Vondrák (2019) that for any δ ∈ (0 , 1) , with probability at least 1 -δ over S , the on-average generalization error is upper bounded by

Bousquet et al. (2020) later derived a slightly improved exponential bound that implies

These bounds are near-tight (up to logarithmic factors) in the sense of an O ( γ N log ( 1 δ ) + √ log(1 /δ ) N ) lower deviation bound on sum of random functions with γ N -uniform stability (Bousquet et al., 2020, Proposition 9). Concerning the excess risk bound, Klochkov and Zhivotovskiy (2021) essentially derived the following result using the sample-splitting techniques of Bousquet et al. (2020):

where ∆ opt := E ξ [ R S ( A ( S, ξ ))] -min w ∈W R S ( w ) represents the in-expectation empirical risk suboptimality, and B is the constant of the generalized Bernstein condition (Koltchinskii, 2006). While sharp in the dependence on sample size, one common limitation of the above uniform stability implied generalization and risk bounds lies in that these high-probability results only hold in expectation with respect to ξ , the internal randomness of algorithm.

Alternatively, consider that A has γ N -uniform stability with probability at least 1 -δ ′ for some δ ′ ∈ (0 , 1) over the random draw of ξ , i.e.,

Suppose that the randomness of A is independent of the training set S . Then the bound of Bousquet et al. (2020) naturally implies that with probability at least 1 -δ -δ ′ over S and ξ ,

This is by far the best known generalization bound of randomized stable algorithms that hold with high probability jointly over the randomness of data and algorithm. The result, however, relies heavily on the high-probability uniform stability as expressed in (6). For the SGD recursion (1) with fixed learning rate η t ≡ η , it is possible to show that γ N ≲ η √ T + ηT N and δ ′ = N exp( -N 2 ) in (6) (Bassily et al., 2020). For SGD with time decaying learning rates, which has been widely studied in theory (Harvey et al., 2019; Rakhlin et al., 2012) and applied in practice for training popular deep nets such as ResNet and DenseNet (Bengio et al., 2017), it is not clear if the condition in (6) is still valid for γ N and δ ′ of interest. Madden et al. (2020) indeed have established a high-probability uniform stability bound for minibatch SGD with learning rates η t ≲ 1 Nt . However, such a fairly conservative choice of learning rates tends to impair the empirical minimization performance of SGD and thus is of limited interest from the perspective of risk minimization.

More specially for randomized learning methods such as bagging (Breiman, 1996) and SGD, the randomness of algorithm can be precisely characterized by a vector of i.i.d. parameters ξ = { i 1 , ..., i t } which are independent on data S . In such cases, assume additionally that A ( S, ξ ) has uniform stability with respect to ξ conditioned on S , i.e., sup ξ . =˜ ξ | ℓ ( A ( S, ξ )) -ℓ ( A ( S, ˜ ξ )) | ≤ ρ T . Then the following exponential bound has been derived by Elisseeff et al. (2005):

Provided that γ N ≲ 1 N and ρ T ≲ 1 T , the above bound shows that the generalization bound scales as O ( 1 √ N + 1 √ T ) with high probability. However, the rate of the above bound is sub-optimal and will show no guarantee on convergence if γ N ≳ 1 √ N and/or ρ T ≳ 1 √ T . As an example, for non-convex SGD with learning rate η t = O ( 1 t ) , it can be shown that γ N ≲ √ T N and ρ T scales as large as O (1) .

Open problem. So far, it still remains open if the exponential generalization bounds for deterministic uniformly stable algorithms might be extended to randomized learning algorithms under the variants of uniform stability tighter than the on-average version (2) but less restrictive than the high-probability

version (6). Particularly, we are interested in the following notion of L 2 -uniform stability (as formally introduced in Definition 1) with parameter γ L 2 ,N :

which represents a second-moment variant of the uniform stability for randomized learning algorithms. For example, as we will shortly show in Section 4 that SGD with practical time-decaying learning rates has L 2 -uniform stability with favorable parameters. The main goal of the present work is to derive sharper exponential generalization bounds for randomized learning algorithms under the notion of L 2 -uniform stability.

1.2 Overview of our contribution

The fundamental contribution of this work is a near-optimal first-moment generalization error bound for L 2 -uniformly stable algorithms, which is summarized in Theorem 1 and highlighted below:

While our first-moment bound above has an identical convergence rate to that of the on-average bound in (4), the former is stronger in the sense that the expectation is taken outside the generalization gap and thus implies the latter where the expectation is taken inside. The key ingredients of our analysis are a set of fine-grained concentration inequalities for randomized functions (Proposition 1) and sums of randomized functions (Proposition 2), which respectively generalize the classic bounded-difference inequalities and a prior result of Bousquet et al. (2020) under the considered L 2 -uniform bounded difference conditions. These generalized concentration inequalities and their proof arguments are novel to our knowledge and should be of independent interests in analyzing randomized functions.

As an important consequence of our main result, we reveal that a bagging-based meta procedure (see Algorithm 1) can be used to boost the confidence of generalization for L 2 -uniformly stable algorithms. More specifically, in the presented bagging procedure we independently run a randomized algorithm A multiple K times over a fraction of the training set to obtain K solutions. Then we evaluate the validation error of these candidate solutions over a holdout training subset, and output the solution that has the smallest training-validation gap. Our result in Theorem 2 shows that for any confidence level δ ∈ (0 , 1) , setting K ≍ log( 1 δ ) yields a near-optimal generalization bound for the selected solution that holds with high probability jointly over the randomness of data and algorithm.

We have substantialized our results to SGD with smooth (Corollary 1) or non-smooth (Corollary 2) convex losses, and smooth non-convex losses (Corollary 3) as well. For an instance, our result in Corollary 1 shows that when invoked to SGD with smooth convex loss and learning rates η t = O ( 1 √ t ) ,

To compare with the O ( T N ) in-expectation bound of smooth convex SGD (Hardt et al., 2016), our bound above for the boosted SGD is comparable in convergence rate while it holds with high probability jointly over the randomness of data and sampling path.

the generalization bound of the output of Algorithm 1 may scale as O ( log( N ) log ( 1 δ ) √ log( T ) N + √ T N ) . √

2 L 2 -Uniform Stability and Generalization

2.1 Notation and definitions

Let us introduce some notation to be used in our analysis. We abbreviate [ N ] := { 1 , ..., N } . Recall that S = { Z i } i ∈ [ N ] is a set of i.i.d. training data. Denote by S ′ = { Z ′ i } i ∈ [ N ] an independent copy of S and we write S ( i ) = { Z 1 , ..., Z i -1 , Z ′ i , Z i +1 , ..., Z N } . For a real-valued random variable Y , its L q -norm for q ≥ 1 is given by ∥ Y ∥ q = ( E [ | Y | q ]) 1 /q . By definition it can be verified that ∀ q ≥ 2 ,

Let h : Z N ↦→ R be some measurable function and consider the random variable h ( S ) = h ( Z 1 , ..., Z N ) . For h ( S ) and any index set I ⊆ [ N ] , we define the following abbreviations:

We say a function f to be G -Lipschitz continuous over W if | f ( w ) -f ( ˜ w ) | ≤ G ∥ w -˜ w ∥ for all w, ˜ w ∈ W , and it is L -smooth if ∥∇ f ( w ) -∇ f ( ˜ w ) ∥ ≤ L ∥ w -˜ w ∥ . For a pair of functions f, f ′ ≥ 0 , we use f ≲ f ′ (or f ′ ≳ f ) to denote f ≤ cf ′ for some universal constant c > 0 .

In the following definition, we formally introduce the concept of L 2 -uniform stability for randomized learning algorithms to be investigated in this work.

Definition 1 ( L 2 -Uniform stability of randomized learning algorithms) . We say a randomized learning algorithm A : Z N ×R↦→ W to have L 2 -uniform stability with parameter γ L 2 ,N ≥ 0 if

Remark 1. By definition the L 2 -uniform stability has a second-moment dependence on the internal randomness of algorithm conditioned on data, while it is invariant to the data distribution. This justifies the name of such a notion of mixed algorithmic stability.

Remark 2. On one hand, by Jensen's inequality the L 2 -uniform stability implies the on-average uniform stability defined in (2) . On the other hand, the second-order form of L 2 -uniform stability is by definition weaker than the high-probability uniform stability in (6) . If the algorithm's randomness ξ can be expressed as a set of i.i.d. random bits, then the L 2 -uniform stability is also weaker than the conditional uniform stability conditioned on data S (Elisseeff et al., 2005).

Throughout this paper, we assume for simplicity that the output models A ( S ( i ) , ξ ) and A ( S, ξ ) share the same internal random bit ξ which is invariant to data. With similar analysis techniques, it is indeed possible to extend Definition 1 and our main results to the general setting where the randomness of algorithm is allowed to be dependent on data, such as in posterior sampling for Bayesian learning.

2.2 Concentration inequalities for randomized functions

We begin by establishing in the following result a group of first- and second-order concentration inequalities (in moments) for randomized functions of independent random variables.

Proposition 1. Let S = { Z 1 , Z 2 , ..., Z N } be a set of independent random variables valued in Z and ξ be a random variable valued in R . Let g : Z N ×R↦→ R be a measurable function that satisfies the following L 2 -bounded-difference condition:

Then for any q ≥ 2 , and

Proof in sketch. Let us consider h ( S ) := E ξ [ | g ( S, ξ ) -E S [ g ( S, ξ )] | ] . The given L 2 -boundeddifference condition implies that h ( S ) has bounded-difference property. Then the desired first-order bound in (11) can be obtained by respectively invoking a moment Efron-Stein inequality (Boucheron et al., 2005, Theorem 2) to upper bound ∥ h ( S ) -E [ h ( S )] ∥ q and a slightly modified Efron-Stein inequality to bound the mean E [ h ( S )] . To prove the second-order concentration bound, we consider the function h ′ ( S ) := E ξ [ ( g ( S, ξ ) -E S [ g ( S, ξ )]) 2 ] , which can be shown to be weakly self-bounding (see Definition 2) under the L 2 -bounded-difference condition. Then the desired bound (12) can be derived by applying the upper tail bound of Boucheron et al. (2005, Theorem 6.19) and lower tail bound of Klochkov and Zhivotovskiy (2021, Proposition 3.1) for weakly self-bounding functions. See Appendix A.2 for a detailed proof of this result.

The moment bound in (11) extends the McDiarmid's (bounded difference) inequality (McDiarmid et al., 1989) to randomized functions with the L 2 -bounded-difference property. The second-order concentration bound in (12) is crucial for proving the moment bound of sums in Proposition 2, as it can be used to sharply control some second-order components involved in the arguments. These generic inequalities are expected to be of independent interests for understanding the first-/second-order concentration behavior of randomized functions.

2.3 A moment inequality for sums of randomized functions

As a key intermediate result, we further establish in the following proposition a moment concentration inequality for sums of randomized functions that satisfy the L 2 -bounded-difference condition. This result extends the moment bound for sums of functions (Bousquet et al., 2020, Theorem 4) to sums of randomized functions.

Proposition 2. Let S = { Z 1 , Z 2 , ..., Z N } be a set of independent random variables valued in Z and ξ be a random variable valued in R . Let g 1 , ..., g N be a set of measurable functions g i : Z N ×R ↦→ R that satisfy the following conditions for any i ∈ [ N ] :

E [ g i ( S, ξ ) | S \ Z i , ξ ] = 0 and | E [ g i ( S, ξ ) | Z i , ξ ] | ≤ M , almost surely;

̸

g i ( S, ξ ) has the following L 2 -bounded-difference property with respect to all variables in S except Z i , i.e., ∀ j = i ,

Then for all q ≥ 2 ,

Proof in sketch. The main idea is inspired by the sample-splitting arguments of Feldman and Vondrák (2019); Bousquet et al. (2020), with some new ingredients developed to handle the first-moment operator taken over the internal randomness of functions. Here we just highlight a fundamental difference, which arises from using a newly developed moment inequality (Lemma 9) for bounding the sums of conditionally independent randomized functions inside each individual data splits. Different from the version of Marcinkiewicz-Zygmund's inequality used in the original analysis of Bousquet et al. (2020), our new bound in Lemma 9 relies on some second-order (over the function's randomness) components which might be tightly bounded by the second-order concentration inequality in Proposition 1. A full proof is provided in Appendix A.3.

Remark 3. For sums of deterministic functions, our result in Proposition 2 reduces to the existing moment bound of Bousquet et al. (2020, Theorem 4) which is known to be near-tight up to logarithmic factors. We comment in passing that the tightness analysis of Bousquet et al. (2020, Proposition 9) for deterministic functions can be more or less straightforwardly extended to randomized functions.

Remark 4. The bound of Proposition 2 would still be valid when the bounded-loss condition | E [ g i ( S, ξ ) | Z i , ξ ] | ≤ M is relaxed to certain sub-Gaussian or sub-exponential stochastic versions.

2.4 Main result on generalization bound

Consequently from Proposition 2, we can now establish our main result on the generalization bound of L 2 -uniformly stable randomized learning algorithms.

Theorem 1. Let A : Z N × R ↦→ W be a randomized learning algorithm that has L 2 -uniform stability with parameter γ L 2 ,N . Assume that the loss function ℓ is valued in [0 , M ] . Then for any δ ∈ (0 , 1) , the following bound holds with probability at least 1 -δ over the draw of S :

Proof. See Appendix A.4 for a proof of this result.

Remark 5. The first-moment bound in Theorem 1 naturally implies the on-average bound in (4) with an identical rate of convergence, though the former is obtained under the relatively stronger notion of L 2 -uniform stability. As we will see shortly that the L 2 -uniform stability can indeed be fulfilled by the popularly applied SGD algorithm and thus Theorem 1 is of practical importance for showcasing sharper generalization performance of SGD. When A is deterministic, our bound reduces to the near-optimal (up to logarithmic factors on sample size and failure tail) generalization bound for uniformly stable algorithms (Bousquet et al., 2020).

In view of the standard risk decomposition, the following excess risk tail bound can be readily obtained via applying Theorem 1 and Hoeffding's inequality:

Here recall that ∆ opt := E ξ [ R S ( A ( S, ξ ))] -min w ∈W R S ( w ) is the sub-optimality of empirical risk minimization. Since the excess risk is by definition non-negative, the above bound can also be obtained under the weaker notion of on-average uniform stability (2) via applying (4). In this sense, the first-moment generalization error bound in Theorem 1 is substantially more challenging to derive than the excess risk bound. Additionally, under the generalized Bernstein condition (Koltchinskii, 2006), the risk bound (13) can be readily improved to (5) by directly applying the corresponding deviation optimal risk bound of Klochkov and Zhivotovskiy (2021) to the on-average loss function E ξ [ ℓ ( A ( S, ξ ); Z )] under on-average uniform stability condition.

3 Boosting the Confidence of Generalization

The confidence-boosting technique of Schapire (1990) is a classic meta approach that allows one to boost the dependence of a learning algorithm on the failure probability δ from 1 /δ to log(1 /δ ) , at a certain cost of computational complexity. In this section, we show an implication of our first-moment bound in Theorem 1 for achieving high-probability generalization jointly over the randomness of data and algorithm, inside a natural framework of confidence-boosting.

3.1 Confidence boosting via bagging

Given a randomized learning algorithm A , we propose to study a bagging based confidence-boosting procedure as outlined in Algorithm 1. In this meta procedure, we independently run the algorithm A for K times over S 1 , a fraction of the training set, to obtain K different candidate solutions { A ( S 1 , ξ k ) } k ∈ [ K ] . Then we evaluate the validation error of these candidate solutions over the holdout training subset S 2 , and cherry pick A ( S 1 , ξ k ∗ ) that has the smallest gap between the training error and validation error, i.e., k ∗ = arg min k ∈ [ K ] | R S 2 ( A ( S 1 , ξ k )) -R S 1 ( A ( S 1 , ξ k )) | . Particularly, consider that the internal randomness of A arises from random sampling with replacement of data points, such as SGD under with-replacement sampling. Then in this setting, the procedure can be regarded as a version of bagging (Breiman, 1996) with a greedy model ensemble scheme, which is invoked to the deterministic counterpart of A with fixed random bits (e.g., SGD with identity permutation) over the training subset S 1 .

3.2 Jointly exponential bounds

The following theorem is our main result about the generalization error bound of the output A ( S 1 , ξ k ∗ ) that holds with high probability over the entire training set S and the random seeds { ξ k } k ∈ [ K ] .

Theorem 2. Suppose that a randomized learning algorithm A : Z N ×R ↦→ W has L 2 -uniform stability with parameter γ L 2 ,N . Assume that the loss function ℓ is valued in [0 , M ] . Then for any δ ∈ (0 , 1) and K ≥ 2 log( 2 δ ) , with probability at least 1 -δ over the randomness of S and { ξ k } k ∈ [ K ] , the output of Algorithm 1 satisfies

Proof in sketch. Based on Theorem 1, we first prove an intermediate result to show that the minimal generalization error of the K outputs satisfies min k ∈ [ K ] | R ( A ( S 1 , ξ k )) -R S 1 ( A ( S 1 , ξ k )) | ≲

used greedy model selection strategy guarantees that the selected A ( S, ξ k ∗ ) mimics the generalization behavior of that best performer among the K candidates, with a slightly expanded log( K/δ ) factor representing the overhead of simultaneously bounding the generalization performance of K different candidate solutions over the holdout validation set. Finally the desired bound follows from the union probability argument. See Appendix B.1 for its full proof.

Remark 6. The bound in Theorem 2 holds with high probability jointly over the randomness of sample and algorithm. Different from the bound in (7) that requires high probability uniform stability, Theorem 2 is valid under a substantially milder notion of L 2 -uniform stability, though at the cost of multiple running of algorithm for confidence boosting. Compared to the bound in (8) that requires certain conditional uniform stability over the random bits of algorithm, our bound has sharper dependence on the uniform stability parameter yet under a weaker notion of stability.

Remark 7. Regarding the scale of the factor 1 / √ µ (1 -µ ) in the bound of Theorem 2, if setting µ = 0 . 01 (i.e., 99% of S are used as S 1 for training), then the factor is around 10 . 05 .

Concerning the excess risk of Algorithm 1, we consider a slightly modified output A ( S 1 , ξ k ∗ ) such that k ∗ = arg min k ∈ [ K ] R S 2 ( A ( S 1 , ξ k )) . Then based on the in-expectation risk bound (13), we can derive the following excess risk bound under the conditions of Theorem 2 using similar arguments:

Again, the above risk bound is still valid under the weaker notion of on-average uniform stability (2).

4 Implications for SGD

This section is devoted to demonstrating the implications of Theorem 1 and Theorem 2 for the widely used SGD algorithm and its confidence-boosted versions as well. We focus on a variant of SGD under with-replacement sampling as outlined in Algorithm 2, which we call A SGD-w . In what follows, we substantialize ξ = { i t } t ∈ [ T ] the sample path of A SGD-w over a given data set, and { ξ k } k ∈ [ K ] the K independent copies of ξ when implemented with bagging as shown in Algorithm 1. Our results can also be extended to the without-replacement variant of SGD and the corresponding results are provided in Appendix D for the sake of completeness.

4.1 Convex optimization with smooth loss

We first present the following lemma that establishes the L 2 -uniform stability of A SGD-w with convex and smooth loss functions, such as logistic loss. See Appendix C.2 for its proof.

Lemma 1. Suppose that the loss function ℓ ( · ; · ) is convex, G -Lipschitz and L -smooth with respect to its first argument. Assume that η t ≤ 2 /L for all t ≥ 1 . Then A SGD-w has L 2 -uniform stability with parameter

Given Lemma 1, we can apply Theorem 1 and Theorem 2 to immediately obtain the following generalization result for A SGD-w and its confidence-boosted version with smooth and convex losses.

Corollary 1. Suppose that the loss function ℓ ( · ; · ) ∈ [0 , M ] is convex, G -Lipschitz and L -smooth with respect to its first argument. Then for any δ ∈ (0 , 1) , it holds with probability at least 1 -δ over the randomness of S that E ξ [ | R ( A SGD-w ( S, ξ )) -R S ( A SGD-w ( S, ξ )) | ] ≲

Moreover, consider Algorithm 1 specified to A SGD-w with learning rate η t ≤ 2 /L and K ≍ log( 1 δ ) . Then with probability at least 1 -δ over the randomness of S and { ξ k } k ∈ [ K ] , it holds that | R ( A SGD-w ( S 1 , ξ k ∗ )) -R S ( A SGD-w ( S 1 , ξ k ∗ )) | ≲

Remark 8. For the conventional choice of η t = 2 L √ t , the high-probability (w.r.t. data) generalization bounds in Corollary 1 for SGD and its confidence boosted version are roughly of scale O ( log( N ) log ( 1 δ ) √ log( T ) N + √ T N ) , which matches the corresponding O ( √ T N ) in-expectation bound of SGD with smooth and convex losses (Hardt et al., 2016).

Combining with the standard in-expectation optimization error bound of convex SGD (see, e.g., Shamir and Zhang, 2013), we can show the following excess risk bound of (modified) Algorithm 1 as a direct consequence of the generic bound (14) to A SGD-w with convex and smooth losses:

where W ∗ := Argmin w ∈W R ( w ) and D ( w,W ∗ ) = min w ∗ ∈ W ∗ ∥ w -w ∗ ∥ . With learning rate η t = 2 L √ t , the right hand side of the above roughly scales as O ( √ log( N ) log ( 1 δ ) log( T ) N + √ T N + log( T ) √ T ) which matches the prior high-probability excess risk bounds of SGD with convex losses (Harvey et al., 2019, Remark 3.7).

4.2 Convex optimization with non-smooth loss

Now we turn to study the case where the loss is convex but not necessarily smooth, such as the hinge loss and absolute loss. We first establish the following lemma about the L 2 -uniform stability parameter of A SGD-w in the considered setting. See Appendix C.3 for its proof.

Lemma 2. Suppose that the loss function ℓ ( · ; · ) is convex and G -Lipschitz with respect to its first argument. Then A SGD-w has L 2 -uniform stability with parameter

With Lemma 2 in place, we can readily apply Theorem 1 and Theorem 2 to establish the following corollary about the generalization bounds of A SGD-w and its confidence-boosted version with convex and non-smooth loss functions.

Corollary 2. Suppose that the loss function ℓ ( · ; · ) ∈ [0 , M ] is convex and G -Lipschitz with respect to its first argument. Then for any δ ∈ (0 , 1) , it holds with probability at least 1 -δ over the randomness of S that E ξ [ | R ( A SGD-w ( S, ξ )) -R S ( A SGD-w ( S, ξ )) | ] ≲

Moreover, consider Algorithm 1 specified to A SGD-w with K ≍ log( 1 δ ) . Then with probability at least 1 -δ over S and { ξ k } k ∈ [ K ] , it holds that | R ( A SGD-w ( S 1 , ξ k ∗ )) -R S ( A SGD-w ( S 1 , ξ k ∗ )) | ≲

Remark 9. For SGD with decaying learning rates η t = 1 √ Nt , Corollary 2 admits high-probability generalization bounds of scale O ( log( N ) log ( 1 δ ) √ log( T ) N + T N 3 + √ log(1 /δ ) N ) . With fixed rates η t ≡ η , Corollary 2 yields deviation bounds of scale O ( η log( N ) log ( 1 δ ) ( √ T + T N ) + √ log(1 /δ ) N ) which matches the near-optimal rate by Bassily et al. (2020, Theorem 3.3).

4.3 Non-convex optimization with smooth loss

We further study the performance of Algorithm 1 for A SGD-w with smooth but not necessarily convex loss functions, such as normalized sigmoid loss (Mason et al., 1999). The following lemma estimates the L 2 -uniform stability of A SGD-w in the considered setting. See Appendix C.4 for its proof.

Lemma 3. Suppose that the loss function ℓ ( · ; · ) is G -Lipschitz and L -smooth with respect to its first argument. Consider η t ≤ 1 /L . Then A SGD-w has L 2 -uniform stability with parameter

where

Based on Lemma 3, we can invoke Theorem 1 and Theorem 2 to show the following generalization result for A SGD-w and its confidence-boosted version with non-convex and smooth loss functions.

Corollary 3. Suppose that the loss function ℓ ( · ; · ) ∈ [0 , M ] is G -Lipschitz and L -smooth with respect to its first argument. Then for any δ ∈ (0 , 1) , it holds with probability at least 1 -δ over the randomness of S that E ξ [ | R ( A SGD-w ( S, ξ )) -R S ( A SGD-w ( S, ξ )) | ] ≲

where u t := η 2 t +2 η t ∑ t -1 τ =1 exp( L ∑ t -1 i = τ +1 η i ) η τ for all t ≥ 1 . Moreover, consider Algorithm 1 specified to A SGD-w with η t ≤ 1 L and K ≍ log( 1 δ ) . Then with probability at least 1 -δ over S and { ξ k } k ∈ [ K ] , it holds that | R ( A SGD-w ( S 1 , ξ k ∗ )) -R S ( A SGD-w ( S 1 , ξ k ∗ )) | ≲

Remark 10. For the decaying learning rates η t = 1 Lνt with arbitrary ν ≥ 1 , the generalization bounds in Corollary 3 are of scale O ( log( N ) log ( 1 δ ) √ T 1 /ν log( T ) νN + √ log(1 /δ ) N ) . For the constant learning rates η t ≡ 1 LT , the bounds are of scale O ( log( N ) log ( 1 δ ) √ log(1 /δ ) N ) .

5 Conclusion

In this paper, we have introduced a novel concept of L 2 -uniform stability for randomized learning algorithms and proved a strong first-moment generalization bound that holds with high probability over training sample. Equipped with this result, we have further developed a bagging based confidenceboosting procedure and shown that it yields near-optimal generalization bounds with high confidence jointly over the randomness of sample and algorithm. The power of our theory has been demonstrated through an application to SGD with time-decaying learning rates, where sharper generalization bounds have been obtained for both convex and non-convex loss functions.

Acknowledgments and Disclosure of Funding

The authors sincerely thank the anonymous reviewers and area chairs for their insightful comments on this paper. The research was conducted while XTY worked for Nanjing University of Information Science and Technology and both authors worked for Baidu Cognitive Computing Lab. The work of XTY is also funded in part by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, and in part by the Natural Science Foundation of China (NSFC) under Grant No. U21B2049, 61936005.

References

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Guha Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems (NeurIPS) , pages 11279-11288, Vancouver, Canada, 2019.

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. In Advances in Neural Information Processing Systems (NeurIPS) , virtual, 2020.

Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning , volume 1. MIT press Cambridge, MA, USA, 2017.

Stéphane Boucheron, Olivier Bousquet, Gábor Lugosi, and Pascal Massart. Moment inequalities for functions of independent random variables. The Annals of Probability , 33(2):514-560, 2005.

Olivier Bousquet and André Elisseeff. Stability and generalization. J. Mach. Learn. Res. , 2:499-526, 2002.

Olivier Bousquet, Yegor Klochkov, and Nikita Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Proceedings of the Conference on Learning Theory (COLT) , pages 610-626, Virtual Event [Graz, Austria], 2020.

Leo Breiman. Bagging predictors. Mach. Learn. , 24(2):123-140, 1996.

Peter Bühlmann. Bagging, boosting and ensemble methods. In Handbook of computational statistics , pages 985-1022. Springer, 2012.

Zachary Charles and Dimitris S. Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 744-753, Stockholmsmässan, Stockholm, Sweden, 2018.

Yuan Shih Chow and Henry Teicher. Probability theory: independence, interchangeability, martingales . Springer Science & Business Media, 2003.

Luc Devroye and Terry J. Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Trans. Inf. Theory , 25(2):202-207, 1979.

André Elisseeff, Theodoros Evgeniou, and Massimiliano Pontil. Stability of randomized learning

algorithms. J. Mach. Learn. Res. , 6:55-79, 2005.

Vitaly Feldman and Jan Vondrák. Generalization bounds for uniformly stable algorithms. In Advances in Neural Information Processing Systems (NeurIPS) , pages 9770-9780, Montréal, Canada, 2018.

Vitaly Feldman and Jan Vondrák. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Proceedings of the Conference on Learning Theory (COLT) , pages 1270-1279, Phoenix, AZ, 2019.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proccedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC) , pages 439-449, Chicago, IL, 2020.

Dylan J. Foster, Spencer Greenberg, Satyen Kale, Haipeng Luo, Mehryar Mohri, and Karthik Sridharan. Hypothesis set stability and generalization. In Advances in Neural Information Processing Systems (NeurIPS) , pages 6726-6736, Vancouver, Canada, 2019.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Proceedings of the 33nd International Conference on Machine Learning (ICML) , pages 1225-1234, New York City, NY, 2016.

Nicholas J. A. Harvey, Christopher Liaw, Yaniv Plan, and Sikander Randhawa. Tight analyses for non-smooth stochastic gradient descent. In Proceedings of the Conference on Learning Theory (COLT) , pages 1579-1613, Phoenix, AZ, 2019.

Matthew Holland. Robustness and scalability under heavy tails, without strong convexity. In Proceedings of the 24th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 865-873, Virtual Event, 2021.

Yegor Klochkov and Nikita Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o (1 /n ) . In Advances in Neural Information Processing Systems (NeurIPS) , pages 5065-5076, virtual, 2021.

Vladimir Koltchinskii. Local rademacher complexities and oracle inequalities in risk minimization. The Annals of Statistics , 34(6):2593-2656, 2006.

Ilja Kuzborskij and Christoph H. Lampert. Data-dependent stability of stochastic gradient descent. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 2820-2829, Stockholmsmässan, Stockholm, Sweden, 2018.

Yunwen Lei and Yiming Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 5809-5819, Virtual Event, 2020.

Yunwen Lei and Yiming Ying. Sharper generalization bounds for learning with gradient-dominated objective functions. In Proceedings of the 9th International Conference on Learning Representations (ICLR) , Virtual Event, Austria, 2021.

Liam Madden, Emiliano DallAnese, and Stephen Becker. High probability convergence and uniform stability bounds for nonconvex stochastic gradient descent. arXiv preprint arXiv:2006.05610 , 2020.

Llew Mason, Jonathan Baxter, Peter L. Bartlett, and Marcus R. Frean. Boosting algorithms as gradient descent. In Advances in Neural Information Processing Systems (NIPS) , pages 512-518, Denver, CO, 1999.

Colin McDiarmid et al. On the method of bounded differences. Surveys in Combinatorics , 141(1): 148-188, 1989.

Nishant A. Mehta. Fast rates with high probability in exp-concave statistical learning. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 1085-1093, Fort Lauderdale, FL, 2017.

Sayan Mukherjee, Partha Niyogi, Tomaso A. Poggio, and Ryan M. Rifkin. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Adv. Comput. Math. , 25(1-3):161-193, 2006.

Dheeraj Nagaraj, Prateek Jain, and Praneeth Netrapalli. SGD without replacement: Sharper rates for general smooth convex functions. In Proceedings of the 36th International Conference on Machine Learning (ICML) , pages 4703-4711, Long Beach, CA, 2019.

David W. Opitz and Richard Maclin. Popular ensemble methods: An empirical study. J. Artif. Intell. Res. , 11:169-198, 1999.

Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning (ICML) , Edinburgh, Scotland, UK, 2012.

William H Rogers and Terry J Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics , pages 506-514, 1978.

Robert E. Schapire. The strength of weak learnability. Mach. Learn. , 5:197-227, 1990.

Mark Schmidt, Nicolas Le Roux, and Francis R. Bach. Convergence rates of inexact proximalgradient methods for convex optimization. In Advances in Neural Information Processing Systems (NIPS) , pages 1458-1466, Granada, Spain, 2011.

Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability, stability and uniform convergence. J. Mach. Learn. Res. , 11:2635-2670, 2010.

Ohad Shamir. Without-replacement sampling for stochastic gradient methods. In Advances in Neural Information Processing Systems (NIPS) , pages 46-54, Barcelona, Spain, 2016.

Ohad Shamir and Tong Zhang. Stochastic gradient descent for non-smooth optimization: Convergence results and optimal averaging schemes. In Proceedings of the 30th International Conference on Machine Learning (ICML) , pages 71-79, Atlanta, GA, 2013.

Giorgio Valentini and Thomas G. Dietterich. Low bias bagged support vector machines. In Proceedings of the Twentieth International Conference (ICML) , pages 752-759, Washington, DC, 2003.

N. Vapnik and A. Ya. Chervonenkis. Theory of Pattern Recognition [in Russian] . Nauka, 1974.

Xiao-Tong Yuan and Ping Li. Stability and risk bounds of iterative hard thresholding. IEEE Trans. Inf. Theory , 68(10):6663-6681, 2022.

Xiao-Tong Yuan and Ping Li. Exponential generalization bounds with near-optimal rates for l q -stable algorithms. In Proceedings of the Eleventh International Conference on Learning Representations (ICLR) , Kigali, Rwanda, 2023.

Tong Zhang. Leave-one-out bounds for kernel methods. Neural Comput. , 15(6):1397-1437, 2003.

Yi Zhou, Yingbin Liang, and Huishuai Zhang. Understanding generalization error of SGD in nonconvex optimization. Mach. Learn. , 111(1):345-375, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers generalization bounds for SGD with a notion of randomization on learning algorithms. The paper considers a $L_2$-uniform stability framework and establishes generalization inequalities under such a framework. Additional algorithms and analysis using confidence-boosting are discussed.\n\nSTRENGTHS:\nThe paper's strengths are summarized in the following criteria.\n\nOriginality: The main comparisons made within the paper focus on comparisons against prior literature established for \"on-average\" uniform stability and \"high probability\" uniform stability. While SGD's under these two forms of uniform stability have been studied extensively, the authors identify an interesting gap in the literature by looking at \"$L_2$\" uniform stability.\n\nQuality and clarity: The paper covers a good amount of motivation and background of SGD theory. Some additional comments regarding quality and clarity can be found in latter sections.\n\nSignificance: Considering that SGD is broadly used in many areas of ML, understanding more about the theory of SGD is quite important. From this perspective, this paper is relevant to the NeurIPS community.\n\nWEAKNESSES:\nWhile the paper covers good amount of material, one of the weaknesses of the paper is its clarity and structure. There were certain parts of the paper that were not easily readable and confusing. It may be helpful to add in formal definitions and clear up typos/confusions:\n\n-  For example, it may be helpful to elaborate in the paper more on specific definitions such as $\\mathcal{Z}$, $\\mathcal{R}$, etc.\n- Another example is the definition of the parameter $\\gamma_{L_2,N}$ in lines 162-164; the words \"for every $N\\geq 1$\" is likely redundant and adds confusion for the reader.\n\nRegarding the main results, considering that an $L_2$ assumption is stronger than an $L_1$ assumption, it does not necessarily make sense to compare Theorem 1 and (4) directly. A better \"comparison\" may be to derive the optimal ordering for the $L_2$ case, i.e., the corresponding lower bound. It is not immediately clear if the known lower bounds from earlier works with different assumptions can be applied to show that Theorem 1 is tight up to a logarithmic factor apart for some special cases (such as when $A$ is deterministic). Is this analysis available?\n\nQUESTIONS:\nPlease refer to the questions listed or implied in the above sections. Additional questions / comments are:\n\n1. Why is the upper bound in Theorem 2 increasing as $K$ increases? One would think that running an algorithm multiple times and selecting the \"best\" output would yield a \"tighter\" result. Can this be elaborated?\n\n2. Theorems 1 and 2 appear to be \"sub-exponential\" bounds with respect to dependency of $\\log(1/\\delta)$. Is it possible to derive with similar methodologies a \"sub-gaussian\" type bound? \n\n3. Are there matching lower bounds (perhaps up to logarithmic factors) under the $L_2$ uniform stability assumption?\n\nBelow are some typos that may need revision:\n\n- the statement in Theorem 2 has an additional period in line 262.\n- \"Jensen's ineuqality\" in line 164\n\nLIMITATIONS:\nLimitations are not addressed in detail.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops new and potentially sharper (as compared to the state of the art) data and randomness dependent (high probability type) generalization bounds using a kind of new and more restrictive notion of algorithmic stability, which the authors call L2-uniform stability. The focus is in particular on generalization analysis for randomized algorithms.\n\nThe main results of the paper are then leveraged to derive very detailed high probability generalization bounds for a certain bagging-based meta algorithmic framework. Those results are subsequently instantiated for SGD, in the same boosting-type scheme.\n\nSTRENGTHS:\nThe paper is well-written and considers an important and challenging question, namely, that of characterizing the generalization performance of randomized algorithms in a \"high-probability\" scheme, which is very much attractive especially from a practical/application perspective.\n\nSection 1 (Intro) are particularly well-written and generally a pleasure to read, and well-suited even for people who are not very familiar with algorithmic generalization.\n\nPropositions 1 and 2 are interesting, and could be potentially of independent interest. This is a plus.\n\nTheorem 1, one of the main and \"backbone\" results, is interesting though somewhat expected, especially because it provides an upper bound on the L1 norm of the estimation error (averaged over algo randomness), rather than the expected estimation error commonly studied in the literature.  However, as pointed out below, the real usefulness of the L2 stability which introduced in the paper might be a bit questionable.\n\nThe results presented in Section 3 for the particular class of meta-algorithms are also interesting and useful. By the way, are the results presented in Sections 3/4 the first ones on the generalization of that type of algorithms?\n\nWEAKNESSES:\nThe L2-type notion stability may be considered \"new\", although it is a reasonable and natural \"norm-Jensenized\" generalization of (2). In terms of contribution, this is probably sufficient, although it may also be considered derivative relative to prior works, at least conceptually.\n\nAll the results presented are for boosted algorithms, which is interesting on the one hand, but it is unclear if the results and the respective techniques produce non-trivial or improved results for standard algorithms, such as (mini-batch) SGD. Maybe the authors would like to elaborate more on this. Also see my comment about Remark 3, below.\n\nAlso, the results in Section 4 are quite weak. In particular, although the bounds are indeed decaying, the decays are extremely slow, like $\\ log{(N)} / \\sqrt{N}$, which is almost constant (so to say). I therefore doubt that such bounds could be of actual interest, apart from strictly theoretical.\n\nQUESTIONS:\nLine 126: What \"sharper\" means here? The bound is the same but with different constants, and with the \"Jensen\" version of the estimation error.\n\nIn Remark 3, although I understand the point, I believe that the word \"tighter\" is not really applicable, because of the different type of stabilities in Theorem 1 and in (4). Maybe you could rephrase. I see, however, that in the appendix the L2 stability of SGD (for instance) is established. I think that the relevant result could be highlighted more explicitly, even if Theorem 1 is in-expectation relative to \\xi.\n\nTitle of Section 4 is misleading, and all algorithms considered are boosted. I would suggest that the authors rephrase, and also state more clearly throughout the paper that the bounds derived for specific algorithms are not for standard schemes such as SGD, but for the boosted versions. \n\nIt seems that Remark 6 has a technical issue: If I am not mistaken, the rate of Hardt et al. in the smooth , Lipschitz, convex case is of order $\\sqrt{T}/N$ for a stepsize equal to $2/(L\\sqrt{T})$, that is for a constant stepsize. In contrast, Remark 6 refers to a decaying root stepsize, which when summed from 1 to T gives a subharmonic number which is strictly larger that $\\sqrt{T}$. If I am correct in the above, please make an effort to revise the results in Section 4 accordingly. See also lines 141-147, where a similar statement is made.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStability is a useful tool for obtaining sharper generalization bounds in learning theory. In prior work, the notion of \"on-average\" stability is as follows. A randomized learning algorithm is stable if for all datasets $S$, adjacent datasets $S'$ (i.e. differing by a single point), and all data points $Z$, the expected difference (taken over the randomness of the algorithm) between the loss incurred by the algorithm at Z when trained on $S$ and $S'$ is bounded.\n\nOn-average stability can be used to provide \"on-average\" risk bounds, where the population risk and the empirical risk of the trained classifier have low expected difference. Note here that this does not guarantee that the empirical risk is close to the population risk -- it rather guarantees that the empirical risk is close to an unbiased estimator of the population risk. Specifically, the quantity $$|\\mathbb{E}_{\\xi}[R(A(S), \\xi) - R_n(A(S), \\xi)]|,$$ is bounded, where the absolute values occur outside the expectation. \n\nThe goal of this is work to, under tighter assumptions, create tighter risk estimation bounds where we can instead bound the absolute difference between the two risks (i.e. proving accuracy) rather than simply providing a \"close-to-unbiased\" estimator. \n\nTo do so, they begin with a tighter notion of stability, where they bound the expected squared difference between the loss incurred at Z when trained on $S$ and $S'$. By simply adding a square here, this eliminates the possibility of \"large differences\" cancelling each other out when the expectation taken, and thus is a much stronger notion of stability. Under this assumption, their main result is a risk estimation bound on $$\\mathbb{E}_{\\xi}|R(A(S), \\xi) - R_n(A(S), \\xi)|.$$ Here the absolute value appears inside the expectation, and this is the crucial difference.\n\nSTRENGTHS:\nThis paper appears to be a significant technical development over prior work. While the notion of stability has been known to lead to better generalization bounds, this paper greatly strengthens the type of generalization bounds that can be obtained for _randomized_ learning algorithms (my understanding is that this paper offers no new insight into deterministic algorithms as their bound essentially reduces to existing bounds).\n\nWEAKNESSES:\nI think that the subtly of the contribution of this paper needs to be spelled out more clearly. As far as I can tell, the crucial difference between this bound and prior bounds is literally upon the placement of the absolute value signs. The quantities $|\\mathbb{E}[R-R_n]|$ and $\\mathbb{E}|R-R_n|$ are substantially different meaning that this difference is important, but nevertheless they visually look very similar to the reader. If my understanding of this paper is correct, then highlighting this difference could be helpful.\n\nQUESTIONS:\nCan you argue more explicitly why your bound is \"sharper than its on-average counterpart?\" I had to read through the algebra for this to become clear to me, and I generally don't like relying on algebra alone due to its sensitivity to typos. \n\nEdit: these concerns have been addressed in the rebuttal.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAlgorithmic stability refers to the sensitivity of learning algorithms to the training set. Roughly speaking, if a learning algorithm is more stable, then its generalization gap (i.e., its training loss is highly indicative of its population loss) can be better controlled. In the literature, various notions of algorithmic stability have been proposed for *randomized* learning algorithms. Two most prominent ones are *on-average uniform stability*, which says that the averaged (over its internal randomness) algorithm is uniformly stable, and *high probability uniform stability,* which says that the algorithm is uniformly stable with high probability over its internal randomness.\n\nThis paper puts forth a new notion of algorithmic stability called *L2-uniform stability*, which can be seen as a second moment version of on-average stability. This notion sits naturally between on-average stability, which is rather weak and thus leads to weaker generalization bounds, and high probability stability, which can be too stringent to satisfy. To demonstrate the utility of this novel notion, the authors show that L2-stability can be used to derive generalization bounds with (exponentially) high confidence, where the randomness is over *both* the training samples and the algorithm’s internal randomness. This is in contrast to the generalization bound derived from on-average stability which only holds *in expectation* with respect to the algorithm’s internal randomness. The authors provide applications of their novel technique by showing qualitatively stronger generalization bounds for SGD with respect to convex loss functions.\n\nSTRENGTHS:\n- **Simple and novel notion with useful applications.** L2-uniform stability is a natural strengthening of on-average uniform stability. Though the modification may seem minor, this novel notion leads to significantly stronger generalization bounds. As mentioned previously, a key deficiency of generalization bounds from on-average stability is that they only hold *in expectation* with respect to the algorithm's randomness. Certainly, generalization bounds that hold *with high probability* over the algorithm's randomness (and also randomness of training samples) are preferable. The first-moment generalization bound from L2-stability, combined with bagging [Breiman, 1996], yields such high probability bounds. The authors further show that SGD in various settings (e.g., for smooth/non-smooth convex losses) is sufficiently L2-stable, leading to sharper generalization bounds.\n\n- **Clear exposition.** The ideas in the paper are presented with great clarity and the overall text, including the technical proof sections, flows seamlessly.\n\nWEAKNESSES:\nN/A\n\nQUESTIONS:\n- Is it possible, perhaps under additional assumptions, to get exponentially high probability generalization bounds from on-average uniform stability? What are some technical barriers that stand in the way of boosting the confidence of generalization bounds from on-average stability?\n\n**Minor comments**\n- Typo in line 56: \"deviatin\"\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new notion of algorithmic stability is introduced for randomized algorithms, which is intermediate to previously studied notions and leads to new strong generalization bounds. The need for a new notion is justified due to known lower bounds under previously studied notions by related to on-average stability, and practical limitations of stronger notions like uniform stability with high probability when applied to SGD. A new intermediate L2-uniform stability is introduced with a goal to fill this gap, and generalization guarantees are obtained under this notion improving over on-average stability bounds. Also boosting is used to give high probability generalization error bounds. Remarkably, the stability of SGD is bounded under this new notion for convex and/or smooth loss functions and the stability constants are derived.\n\nSTRENGTHS:\n- The paper considers a novel and interesting notion of stability, and establishes strong generalization bounds for algorithms that are stable under the defined sense of stability.\n- High probability risk bounds are obtained via boosting.\n- The proposed notion of stability is shown to be satisfied for the popular SGD (stochastic gradient descent) algorithm when the loss function is convex and/or smooth.\n\nWEAKNESSES:\n- No lower bounds are provided, so tightness of the results is not clear. The upper bounds are tighter, but under a stronger assumption than previously studied in Bousquet et al. 2020.\n- The presentation of Section 4 could be improved. What are the novel insights in establishing the bounds on the L2-uniform stability parameters? For implications to SGD, it might be better to present/discuss the lemmas on the L2-uniform stability parameters in the main body, possibly replacing the corresponding corollaries obtained by combination with Theorem 2.\n\nQUESTIONS:\n- How does $\\gamma_{L_2,N}$ compare to $\\gamma_N$ for SGD?\n- Line 75: Should it be $[0,M]$ instead of $(0,M]$?\n- Line 187: Typo: McDiamid’s\n\nLIMITATIONS:\nLimitations of results could be discussed in more detail. For example, in Remark 6, are there terms where the proposed bounds are worse than the in-expectation bounds of Hardt et al. 2016?", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}