Title: Toward Better PAC-Bayes Bounds for Uniformly Stable Algorithms
Venue: NeurIPS 2023
Accepted: yes
Keywords: PAC-Bayesian Bounds

Uniform Stability

Generalization Analysis
Abstract: We give sharper bounds for uniformly stable randomized algorithms in a PAC-Bayesian framework, which improve the existing results by up to a factor of $\sqrt{n}$ (ignoring a log factor), where $n$ is the sample size. The key idea is to bound the moment generating function of the generalization gap using concentration of weakly dependent random variables due to Bousquet et al (2020). We introduce an assumption of sub-exponential stability parameter, which allows a general treatment that we instantiate in two applications: stochastic gradient descent and randomized coordinate descent. Our results eliminate the requirement of strong convexity from previous results, and hold for non-smooth convex problems.

Toward Better PAC-Bayes Bounds for Uniformly Stable Algorithms

Abstract

We give sharper bounds for uniformly stable randomized algorithms in a PACBayesian framework, which improve the existing results by up to a factor of √ n (ignoring a log factor), where n is the sample size. The key idea is to bound the moment generating function of the generalization gap using concentration of weakly dependent random variables due to Bousquet et al (2020). We introduce an assumption of sub-exponential stability parameter, which allows a general treatment that we instantiate in two applications: stochastic gradient descent and randomized coordinate descent. Our results eliminate the requirement of strong convexity from previous results, and hold for non-smooth convex problems.

1 Introduction

Stochastic optimization methods are the workhorses of many modern machine learning problems. A lot of progress has been made in reducing optimization errors with less training time in stochastic settings [6, 21, 41, 46, 64]. One such method is stochastic gradient descent (SGD), which builds a stochastic gradient estimator iteratively based on a randomly sampled example to approximate the gradient for the next iteration. SGD is appealing for large-scale data analysis due to its cheap computational cost, simplicity and efficiency.

Obtaining models that generalize well is the goal in machine learning and we undertake a theoretical analysis of this problem. The generalization behavior of a model can be quantified by the excess risk, which decomposes into two components: the optimization error and the generalization error (or generalization gap). In this paper, we focus on the analysis of the generalization error of stochastic optimization methods. We combine two useful approaches to bound the generalization error, that is algorithmic stability [7, 16] and PAC-Bayes bounds [9, 26, 34, 53]. Our hope is to retain the benefits of both approaches to derive powerful generalization bounds to better understand the behavior of stochastic optimization methods.

Seminal work by Bousquet et al. [7] provided generalization bounds for uniformly stable algorithms in the deterministic case. Sharper generalization bounds were obtained via a moment bound on the generalization gap and a concentration inequality of weakly dependent random variables [8, 18]. Extending stability-based bounds to randomized algorithms is a challenge. In [16], where the stabilitybased bounds for the randomized algorithms were considered, the results hold for fixed distributions. To give bounds uniformly for all distributions, we need to turn to PAC-Bayes analysis. A recent work analyzed stable algorithms in the PAC-Bayes framework and derived generalization bounds that hold for any distribution [32]. However, a comparison between [8] and [32] reveals that the error convergence rate in the randomized case [32] is much slower than that for the deterministic case [8]. In more detail, it was shown that β -uniformly stable ( β is decreasing w.r.t. n , where n is the sample

size) and deterministic algorithms would imply generalization bounds of order ˜ O (1 / √ n + β ) [8, 18] 2 , while PAC-Bayes bounds of order O (1 / √ n + √ nβ ) were developed for randomized algorithms [32]. In [32], the randomness comes from the sampling of hyperparameters such as the index of examples chosen in SGD, and the PAC-Bayes bounds hold for any posterior distribution which may depend on the dataset. It is clear that the PAC-Bayes bounds can be slower than those in [8, 18] by a factor of √ n , which motivates a natural question: can we develop PAC-Bayes bounds for randomized algorithms which match the rate of deterministic algorithms?

This paper explores the above question. We provide sharper PAC-Bayes bounds that hold for uniformly stable randomized algorithms. We adapt a moment bound [8] previously used for stable algorithms in deterministic cases and extend it to the PAC-Bayesian framework, to give bounds that hold for randomized predictors. The PAC-Bayes framework is based on the work of [32]. However, we take a different analysis strategy to control the change in hyperparameters, which is based on an assumption of sub-exponential stability parameter. This general assumption allows us to bound the moment generating function (MGF) of the generalization gap within a high probability domain where our assumption holds (see Appendix A.2). Furthermore, we prove that this assumption holds for both SGD and randomized coordinate descent (RCD). We then illustrate the advantage of our results over existing bounds.

Regarding the convergence rate, our main result improves on the existing PAC-Bayes bounds [32] by a factor of √ n (ignoring a log factor). This improvement holds under weaker conditions under which convergence is not guaranteed in [32]. Our primary technical tool is a moment bound, which we extend to randomized learning algorithms in the PAC-Bayes framework. This allows our bounds to hold for all possible posteriors, not just fixed ones [16] or deterministic algorithms [8, 18]. We need to introduce novel techniques to handle the randomness of the hyperparameter, which is a challenge in the PAC-Bayes analysis (details will be given in Section 3.2).

Regarding assumptions, our result holds without the requirement of hyperparameter stability in previous work [32]. Instead, we introduce a new assumption on the sub-exponential behavior of uniform stability by viewing the uniform stability as a function of the random hyperparameter. Interestingly, it suffices to study the sub-exponential behavior of uniform stability under the prior distribution, which makes the sub-exponential assumption easy to check.

We show that the uniform stability of SGD and RCD have a mixture of sub-Gaussian and subexponential tails that satisfy this assumption. Thus, we remove the strong convexity assumption in the existing analysis of SGD [32] and extend the result to non-smooth problems. Our result also applies to RCD, where the randomness arises from the selected coordinates.

The remainder of the paper is organized as follows. We discuss the basics on PAC-Bayes and stability analysis in Section 2. We present our main results in Section 3, and apply it to SGD and RCD in Section 4. We survey the related work on stability and PAC-Bayesian analysis in Section 5, and conclude the paper in Section 6.

2 Preliminaries

We first introduce some notations. Let X and Y denote the input and output space respectively and let Z = X × Y . We are given a set of training examples of size n , S = { z 1 = ( x 1 , y 1 ) , . . . , z n = ( x n , y n ) } , drawn independently and identically distributed (i.i.d.) from an unknown distribution D on Z . We hope to learn a predictor from a class of hypotheses H to predict unseen new data drawn from D . Let W denote the weight space, W ⊆ R d , and Θ denote a hyperparameter space. A deterministic learning algorithm A : Z n × Θ →H maps the training examples to a hypothesis h w ∈ H determined by w ∈ W . The quality of a hypothesis is measured by a loss function, /lscript : H×Z → [0 , M ] .

The goal of any learning algorithm is to produce a predictor that generalizes well. That means that the learned predictor applied on previously unseen input data from the marginal of D should have a small expected loss. For any θ ∈ Θ , the risk of a predictor returned by the algorithm A is a random variable as a function of S , defined as

Since D is unknown, we have no access to the true risk. Instead, the empirical risk is often used as an approximation

We are interested in how well the empirical risk can estimate the risk. The difference between them is the generalization gap G ( S ; θ ) /defines R ( A ( S ; θ )) -R S ( A ( S ; θ )) . We can upper bound the risk by bounding this difference.

2.1 PAC-Bayes basics

We are interested in randomized algorithms, such as stochastic gradient descent (SGD) and randomized coordinate descent (RCD). In such algorithms, there is an in-built random sampling mechanism that we can think of as a random hyperparameter. In other words, a randomized algorithm may be viewed as a deterministic algorithm with hyperparameters θ that follow a distribution. Therefore, for the analysis that follows, we will explicitly define distributions on Θ . There has been a lot of interest in SGD because it is often used to train a model and its randomness comes from independent sampling of training instances to estimate gradient directions. In this case, the hyperparameters θ ∈ Θ form a random sequence θ = ( θ 1 , ..., θ T ) , where every θ t ∈ [ n ] , for t ∈ [ T ] , is an i.i.d. index of a training point from S . Here [ n ] denotes { 1 , . . . , n } .

Any distribution over the hyperparameter space Θ induces a distribution over the predictors (or weights since we often consider parametric models). In the PAC-Bayes methodology, the common approach is to define a prior distribution on the domain of the weights [13, 22, 34, 53]. However, in the context of randomized algorithms of the kind mentioned above it is more natural to exploit the randomness already present through a distribution on the domain of the hyperparameters.

Hence, we will conduct PAC-Bayesian analysis by focusing on the randomness of the hyperparameters. Let P be any prior distribution on the hyperparameter domain Θ , chosen before seeing the training data. Given a randomized algorithm, we will only need to prove a condition for the prior P on Θ , and in return our generalization guarantees will hold with high probability for all posteriors Q on Θ . In particular, we can assume a simple uniform distribution as a prior for SGD, i.e. SGD with uniform sampling, and derive bounds for data-dependent non-uniform sampling [49, 56, 66]. For example, SGD with importance sampling [66] shows better results than uniform sampling, and there are efforts to learn good sampling distributions [32] in a line of research towards self-certified learning algorithms [13, 44].

The quality of a predictor learned by an algorithm A from the training sample S , is defined as the expected risk w.r.t the PAC-Bayes posterior Q as

Its empirical counterpart, the expected empirical risk w.r.t. Q is

In special cases this expectation can be computed analytically; most often it is approximated either by Monte Carlo sampling or by analytical upper bounds.

PAC-Bayes bounds aim to estimate R ( S, Q ) in terms of R S ( S, Q ) and the divergence between P and Q . A key ingredient that PAC-Bayes bounds rest upon is a change of measure inequality, also known as variational formula. For completeness this is given in Appendix (Lemma A.3).

2.2 Algorithmic stability basics

Another relatively recent framework for generalization analysis is based on the concept of algorithmic stability [7]. The key concept in this framework quantifies how sensitive a learning algorithm is to small perturbations of the training data. There are several notions of algorithmic stability, and the one we use in our work is the following uniform stability [7]. We denote S ∼ S ′ if they are neighboring datasets, i.e., S and S ′ differ by at most a single example.

Definition 1 (Uniform Stability) . For any θ , an algorithm A : S ↦→ A ( S ; θ ) is β θ -uniformly stable w.r.t. a loss function /lscript if ∀ S ∼ S ′ ∈ Z n , ∀ z ∈ Z ,

Recall that A is a randomized algorithm whose randomness is exclusively due to a random draw of θ . Hence, given a fixed instance of θ , the algorithm A becomes a deterministic algorithm. This simple observation helps us reduce the problem from randomized learning to deterministic learning. Based on this observation, the next section presents sharper generalization bounds for randomized algorithms in comparison to previous results [32].

3 Main Results

First we introduce a sub-exponential assumption on the stability parameter. This will allow us to make some key innovations: 1) We will be able to remove one of the assumptions required by [32, Theorem 2] (hyperparameter stability) and only require uniform stability. 2) we will state our result in more general terms, and instantiate it to specific algorithms such as SGD and RCD.

Assumption 1 (Sub-exponential stability) . Let P be a fixed probability distribution on Θ = ∏ T t =1 Θ t . We say that a randomized algorithm with random hyperparameters θ ∼ P satisfies sub-exponential stability if, for any fixed instance of θ it satisfies β θ -uniform stability w.r.t. a loss function /lscript , and there exists c ∈ R such that for any δ ∈ (0 , 1 /n ] , with probability at least 1 -δ over draws of θ ∼ P :

In other words, this assumption says that the deviation of β θ from its mean roughly has a subexponential tail. We will show that c is dominated by E θ ∼ P [ β θ ] in typical stochastic optimization algorithms such as SGD and RCD.

Note that in the above assumption, the probability is made w.r.t. θ ∼ P , which is a prior distribution independent of S . We can choose P to be a uniform distribution. In this case, we will show that SGD satisfies this assumption under mild conditions on the loss functions with very small E θ ∼ P [ β θ ] and c .

Recall that for any fixed instance of θ , the trained model of the learning algorithm A on data set S with hyperparameters θ is a deterministic predictor. Our proof strategy for our main result below is to first fix θ , and apply a recent technique of obtaining sharp bounds [8] to the resulting deterministic algorithm. After that, we deal with the randomness of θ .

In Theorem 1 we give our main result. This bound is useful when we have small E θ ∼ P [ β θ ] and c . We will see two examples in Section 4. We denote B /greaterorsimilar B ′ if there exists a universal constant c 1 > 0 such that B ≥ c 1 B ′ . We use B /lessorsimilar B ′ if there exists a universal constant c 2 > 0 such that B ≤ c 2 B ′ . We use B /equivasymptotic B ′ if B /lessorsimilar B ′ /lessorsimilar B . The proof is given in Appendix A.2.

Theorem 1 (Generalization of sub-exponentially stable randomized algorithms) . Consider a learning algorithm A ( S ; θ ) that satisfies Assumption 1 w.r.t. P and c /lessorsimilar E θ ∼ P [ β θ ] . Assume /lscript ( A ( S ; θ ) , z ) ∈ [0 , M ] . Then for any δ 1 ∈ (0 , 1) the following inequality holds with probability at least 1 -δ 1 uniformly for all Q

To apply Theorem 1, we only need to check the sub-exponential assumption w.r.t. the prior P . Then, we use the PAC-Bayesian analysis to transfer this stability assumption w.r.t. the specific P to a bound holding for all posterior distributions.

3.1 Comparison

Next, we compare our result (Theorem 1) with the previous generalization bounds on randomized algorithms due to [32] and observe the advantages that our approach offers.

To blend the stability into the PAC-Bayes framework, where hyperparameters θ follow a distribution, [32] defined the following new hyperparameter stability assumption for learning algorithms (Definition 2) and obtained the stability-based PAC-Bayes bound given in Theorem 2.

Definition 2 (Hyperparameter Stability) . A learning algorithm A has uniform hyperparameter stability β Θ w.r.t. the loss function /lscript , if

/negationslash where D H ( v , v ′ ) /defines ∑ | v | i =1 I [ v i = v ′ i ] is the Hamming distance and I [ · ] denotes the indicator function, i.e., I [ E ] = 1 if the event E holds and 0 otherwise.

Observe that, uniform stability (Definition 1) concerns the stability of an algorithm with respect to a change in the training set. In contrast, the above Definition 2 requires stability w.r.t. a change in the hyperparameters. Moreover, the approach in [32] requires stability w.r.t. both the loss function and the hyperparameters to derive the following PAC-Bayes bound.

Theorem 2 (Theorem 2 of [32]) . Let A be a randomized learning algorithm and /lscript ( A ( S ; θ ) , z ) ∈ [0 , M ] . Assume A is β θ -uniformly stable w.r.t. loss functions, and β Θ -uniformly stable w.r.t. hyperparamters. Consider the prior P as a fixed probability distribution defined on Θ = ∏ T t =1 Θ t . Then for any n ≥ 1 , T ≥ 1 , and δ ∈ (0 , 1) , the following inequality holds with probability at least 1 -δ over draws of a data set, S ∼ D n , for every posterior Q on Θ

We now compare our bound (Theorem 1) with the bound above (Theorem 2).

In terms of the rate, our bound in Theorem 1 improves the previous result of Theorem 2 by up to a factor of √ n (up to poly-logarithmic factors when the divergence is polylogarithmic in n [32]). To see this, consider the case where in both Theorem 1 and Theorem 2, the term that contains the stability parameter dominates over the KL term. Then, in Theorem 1, for β θ -uniformly stable randomized algorithms, we have

In contrast, Theorem 2 in [32] gives

It is clear that, in the case when ˜ O ( √ n E θ ∼ P [ β θ ]) dominates the KL divergence then having replaced it with ˜ O ( E θ ∼ P [ β θ ]) we improved (3.3) by up to a factor of √ n (ignoring a log term). The prior P can be selected freely if it is independent of S . The posterior Q can depend on the data. Indeed, a strength of the PAC-Bayesian analysis is that it applies to any Q , and therefore it allows to choose a distribution Q in a data-dependent manner. The posterior Q can be optimized to control the divergence between P and Q . Therefore, we typically have a small KL divergence.

Observe also that, in Theorem 1, when E θ ∼ P [ β θ ] /lessorsimilar O ( n -1 2 ) , our generalization bound is ˜ O ( n -1 2 ) , while (3.3) implies a vacuous bound O (1) so generalization is not guaranteed. Furthermore, our bounds require E θ ∼ P [ β θ ] /lessorsimilar 1 / √ n to get almost optimal rates ˜ O (1 / √ n ) . As a comparison, the results in Eq. (3.3) require stronger condition E θ ∼ P [ β θ ] /lessorsimilar 1 /n and β Θ ≤ 1 /T to get the rate O (1 / √ n ) .

In the unlikely case when the KL divergence is the dominating term, for example, D KL ( Q ‖ P ) /greaterorsimilar O ( n ) , then Theorem 2 achieves better result. But in this circumstance, both Theorems 1 and 2 require β θ /lessorsimilar O ( n -3 2 ) to converge at a rate of O ( n -1 2 ) , which we believe this to be also rather uncommon.

In terms of assumptions, we eliminate the hyperparameter stability assumption in [32] at the price of an additional sub-exponential tail assumption on the uniform stability parameter. We will prove later in Section 4 that this general assumption holds for more stochastic optimization methods and even for non-smooth problems. In conclusion, we achieve better results under weaker conditions.

3.2 Proof sketch and challenge in the analysis

We begin by noting that /lscript is applied to the output of A ( S ; θ ) , which depends on the training data and the hyperparameters. Therefore, /lscript ( A ( S ; θ )) could be sensitive to changes in both the dataset and the hyperparameters. To address this issue, Theorem 2 assumes hyperparameter stability, which requires small changes in /lscript ( A ( S ; θ )) when the hyperparameters are perturbed.

In contrast to the previous method, we adopt a different strategy for controlling changes in θ . Our β θ is a random variable w.r.t. θ . We assume a sub-exponential concentration behavior of β θ to control its deviation (Assumption 1). This allows us to control β θ without necessitating hyperparameter stability. Based on this assumption, our proof proceeds by bounding the MGF of the generalization gap. A key challenge is to handle the randomness of β θ . More precisely, for any temporarily fixed θ it has been shown that G ( S ; θ ) is a mixture of sub-Gaussian and sub-exponential random variables when only considering the randomness of S [8]. Then a bound of E S [exp( λG ( S ; θ ))] requires an assumption λ /lessorsimilar 1 /β θ (by Eq. (A.5) on sub-exponential random variables). This constraint makes it challenging since we need to choose an appropriate λ and control the associated E S E θ ∼ P [exp( λG ( S ; θ )] (the random constraint makes the selection of λ difficult). Our key idea to address this problem is to control the MGF of another function H : Z n × Θ ↦→ R defined as follows where Assumption 1 holds everywhere on Ω δ , which is a subset of Θ with probability measure at least 1 -δ . This definition of H has two benefits.

First, H ( S ; θ ) is equal to G ( S ; θ ) with high probability. Therefore, a high-probability bound on H would imply a high-probability bound on G .

Second, H is convenient to control. For any θ ∈ Θ \ Ω δ , the MGF satisfies E S [exp( λH ( S ; θ )] ≤ 1 . For any θ ∈ Ω δ , the term E S [exp( λH ( S ; θ ))] can be controlled under an assumption λ /lessorsimilar 1 / E θ ∼ P [ β θ ] due to the definition of Ω δ which relates β θ with its expectation. A key difference is that the random β θ is replaced by E θ ∼ P [ β θ ] in the constraint of λ , which allows us to choose a deterministic λ uniformly for all θ . A further expectation w.r.t. θ would imply a bound on E S ; θ [exp( λH ( S ; θ ))] , which is needed in the PAC-Bayes analysis.

Finally, we manipulate this MGF bound in the PAC-Bayes framework by the variational formula (Lemma A.3), building upon the framework of [32].

3.3 Bounds in Expectation

Our previous analysis gives high-probability bounds. In this subsection, we give PAC-Bayes bounds in expectation. In this case, we no longer need an assumption on the concentration behavior of β θ around its expectation. The proof is given in Appendix A.3.

Theorem 3. Consider any β θ -uniformly stable algorithm A and M -bounded loss /lscript . For any distribution Q , we have

Remark 1. Under the same assumption, it was shown in Theorem 1 in [32] that

Therefore, our analysis gives a bound of the order O ( 1 / √ n + ( E θ ∼ P [ β 2 θ ] ) 1 2 ) , while Eq. (3.5) gives a bound O ( 1 / √ n + ( E θ ∼ P [ β θ ] ) 1 2 ) . It is known that E θ ∼ P [ β 2 θ ] can be much smaller than E θ ∼ P [ β θ ] . For example, for SGD with t iterations and P being the uniform distribution, it was shown E θ ∼ P [ β θ ] = O ( ηt/n ) and E θ ∼ P [ β 2 θ ] = O ( η 2 t/n ) if the loss function is convex and smooth [27]. In the typical setting with η = O (1 / √ t ) and t /equivasymptotic n (in this setting SGD achieves optimal rates), we have E θ ∼ P [ β θ ] = O (1 / √ n ) and E θ ∼ P [ β 2 θ ] = O (1 /n ) . In this case, our analysis gives PAC-Bayes bounds of the order O (1 / √ n ) , while Eq. (3.5) gives PAC-Bayes bounds of the order O (1 /n 1 4 ) . Therefore, our analysis implies much better PAC-Bayes bounds than that in [32].

4 Applications

We apply our general results to derive PAC-Bayes bounds for two optimization algorithms: Stochastic Gradient Descent (SGD) and Randomized Coordinate Descent (RCD). To this aim, we introduce some necessary definitions. Let ‖ · ‖ 2 denote the Euclidean norm.

Definition 3 (Lipschitz continuity) . We say a loss function /lscript ( · ; z ) is L -Lipschitz if for any w ∈ W and z ∈ Z , we have ‖∇ /lscript ( w ; z ) ‖ 2 ≤ L . This implies that for any w 1 , w 2 ∈ W ,

Definition 4 (Convexity) . Let κ ≥ 0 . We say a loss function /lscript ( · ; z ) is κ -strongly convex if for any w 1 , w 2 ∈ W and z ∈ Z , we have

We say the loss function /lscript is convex if the above inequality holds with κ = 0 .

Definition 5 (Smoothness) . Let α ≥ 0 . We say a loss function /lscript ( · ; z ) is α -smooth if for any w 1 , w 2 ∈ W and z ∈ Z , we have

For the applications, we only need to verify the sub-exponential stability of the algorithm w.r.t. the prior sampling P , which is often chosen to be simple such as the uniform distribution.

4.1 Applications to Stochastic Gradient Descent

SGD is one of the most popular algorithms to solve optimization problems in machine learning due to its simplicity and efficiency. The basic idea is to build a stochastic gradient based on a randomly selected example, which is used to update iterates. Here we consider SGD with a general sampling scheme, where the random index follows from a general distribution. This general SGD has already been considered in the literature to improve the efficiency of SGD with uniform sampling, including importance sampling [66] and Markov chain sampling [54, 58, 63].

Definition 6 (SGD with general sampling) . Let w 1 be an initial point. Let P be a probability measure over [ n ] T and S = { z 1 , . . . , z n } be a training dataset. Let ( i 1 , . . . , i T ) be drawn according to P . At the t -th iteration, SGD with sampling scheme P updates the model by

where { η t } is a positive step-size sequence. If P is the uniform distribution, then we call it SGD with uniform sampling (SGDU).

Now we apply Theorem 1 to develop PAC-Bayes bounds for SGD applied to convex problems, covering both smooth and non-smooth cases. We will show that SGD enjoy sub-exponential stability.

4.1.1 Smooth case

In the following lemma to be proved in Appendix II.1.1, we give stability bounds for SGDU and show it satisfies Assumption 1. Recall that the indicator function I [ · ] is defined in Definition 2.

Lemma 4 (Stability bound) . Let S and S ′ be neighboring datasets. Suppose for all z ∈ Z the loss function is convex, α -smooth and L -Lipschitz. Let { w t } , { w ′ t } be the sequence produced by SGDU on S and S ′ respectively with η t ≤ 2 /α . Then SGDU with t iterations and the hyperparameter θ is β θ -uniformly stable with

If η t = η , then for any δ ∈ (0 , 1) , with probability at least 1 -δ we have

That is, Assumption 1 holds with c = 4 L 2 η ( 1 + ( t/n ) 1 2 ) w.r.t. P .

We can combine the above lemma with Theorem 1 to obtain PAC-Bayes bounds for SGD, whose proof is given in Appendix II.1.1. An interesting property is that generalization bounds for SGD with general sampling can be derived based on the stability analysis for SGD with the uniform sampling. We always let P denote a uniform prior on Θ .

Corollary 5 (Generalization bound) . Assume /lscript is M -bounded, L -Lipschitz, convex and α -smooth. For uniform distribution P , every n ∈ N + and δ ∈ (0 , 1) , with probability at least 1 -δ over draws of a data set, S ∼ D n , for all posterior sampling distribution Q on [ n ] T , SGD with η ≤ 2 /α satisfies

Based on [20, 27], which studied the trade-off between optimization and stability, the recommended choices of parameters are T /equivasymptotic n and η /equivasymptotic 1 / √ T to get a SGD iterate with good generalization behavior. In this setting, the above corollary implies a PAC-Bayes bound ˜ O (1 / √ n ) . London [32] gave PAC-Bayes bounds for SGD under strong convexity and smoothness assumptions.

Corollary 6 (Corollary 1 in [32]) . Suppose /lscript is M -bounded. Let the objective function be κ -strongly convex, L -Lipschitz and α -smooth. Then, for uniform distribution P and any δ ∈ (0 , 1) , with probability at least 1 -δ over draws of a data set, S ∼ D n , SGD with η t = ( κt + α ) -1 and any posterior sampling distribution Q on [ n ] T satisfies

Remark 2. We now compare Corollary 5 and Corollary 6. First, Corollary 6 requires a strong convexity assumption, which is removed in our analysis. Second, our analysis implies PAC-Bayes bounds of the order ˜ O (1 / √ n ) , while Corollary 6 implies bounds of order O (1 / ( √ nκ )) . The strong convexity parameter κ is often very small in both theoretical and empirical analysis. For example, the existing generalization analysis of regularization schemes suggests κ = O ( n -1 2 ) to get an optimal bound (Section 3 in [55]), for which Corollary 6 implies a vacuous bound. By contrast, our bound O (1 / √ n ) is optimal up to a log factor.

˜ Remark 3. Our uniform stability (Definition 1) is slightly different from the uniform stability β Z := sup S ∼ S ′ sup z ∣ ∣ E θ ∼ P [ /lscript ( A ( S ; θ ) , z ) -/lscript ( A ( S ′ ; θ ) , z )] ∣ ∣ in [32] in the sense of taking expectation in different places. The expectation of θ ∼ P is taken outside sup in our case and inside sup in β Z [32]. However, we often have similar upper bounds for E [ β θ ] and β Z . Consider SGD for smooth, Lipschitz and convex problems as an example. It is shown in [20] that β Z ≤ 2 L 2 ∑ t k =1 η k /n , while we can show E [ β θ ] /lessorsimilar L 2 log n ∑ t k =1 η k /n (the proof of Lemma 4) . These two upper bounds are the same order up to a logarithmic factor. Furthermore, lower bounds were established in [65] (Theorem 1) where β Z ≥ L 2 ∑ t k =1 η k /n , which match the existing upper bounds up to a constant factor. This shows that β Z and E [ β θ ] are of similar order.

4.1.2 Non-smooth case

The following lemma shows that SGDU applied to non-smooth problems enjoys the sub-exponential stability. The proof follows the analysis in Section 4.2 in [29] and is given in Appendix II.1.2.

Lemma 7 (Stability bound) . Let S and S ′ be neighboring datasets. Suppose for all z ∈ Z the loss function is convex and L -Lipschitz. Let { w t } , { w ′ t } be the sequence produced by SGDU on S and S ′ respectively with fixed step sizes. Then SGDU with t iterations and the hyperparameter θ is β θ -uniformly stable with β θ = 2 √ eL 2 η ( √ t +max k ∈ [ n ] ∑ t j =1 I [ i j = k ] ) . For any δ ∈ (0 , 1) , with probability at least 1 -δ we have

That is, Assumption 1 holds with c = 4 √ eL 2 η ( 1 + ( t/n ) 1 2 ) w.r.t. P .

Based on the above lemma, we derive the following corollary for the PAC-Bayes bounds of SGD in non-smooth problems.

Corollary 8 (Generalization bound) . Assume /lscript is M -bounded, L -Lipschitz and convex. For any δ ∈ (0 , 1) , with probability at least 1 -δ over draws of a data set, S ∼ D n , for all posterior sampling distribution Q on [ n ] T , SGD with T iterations and η t = η satisfies

Remark 4. If we choose η /equivasymptotic T -3 4 and T /equivasymptotic n 2 , then Corollary 8 gives the PAC-Bayes bounds of order ˜ O (1 / √ n ) . This choice of parameters was suggested in Theorem 7 in [27]. This gives the O (1 / √ n ) optimization bounds to get optimal trade-off between stability and optimization for nonsmooth problems. The analysis in [32] cannot imply PAC-Bayes bounds for non-smooth problems.

4.2 Applications to Randomized Coordinate Descent

In this subsection, we consider RCD; this has not been studied in the PAC-Bayesian literature. RCD is an efficient optimization algorithm that randomly chooses a coordinate to update at each iteration [37]. Here we consider RCD with general sampling scheme, i.e. the coordinate to update follows a general distribution. This scheme has been studied before in the optimization context [1, 66].

Definition 7 (RCD with general sampling) . Let w 1 be an initial point. Let P be a probability measure over [ d ] T and S = { z 1 , . . . , z n } be a training dataset. Let ( i 1 , . . . , i T ) be drawn according to P . At the t -th iteration, RCD with sampling scheme P updates the model by

where { η t } is a step-size sequence, e i is the i -th coordinate vector in R d , and ∇ i g is the derivative of g w.r.t. the i -th coordinate. If P is the uniform distribution, then we call it RCD with uniform sampling (RCDU).

Before giving the generalization bound for RCD, we first introduce coordinate-wise smoothness.

Definition 8 ([37]) . A differentiable function g : W → R has coordinate-wise Lipschitz continuous gradients with parameter ˆ α > 0 , if for all λ ∈ R , w ∈ W , i ∈ [ d ] ,

In Lemma 9, to be proved in Appendix B.2, we develop stability bounds for RCDU with convex and smooth loss functions. In particular, we show the stability follows a sub-exponential distribution.

Lemma 9 (Stability bound) . Let S and S ′ be neighboring datasets. Suppose for all z ∈ Z the loss function /lscript is convex, α -smooth, L -Lipschitz and has coordinate-wise Lipschitz continuous gradients with parameter ˆ α ≥ 0 . Let { w t } , { w ′ t } be the sequence produced by RCDU on S and S ′ respectively with η t ≤ 2 / ˆ α . Then RCD with t iterations is β θ -uniformly stable with where ‖∇ /lscript ( w ; z ) ‖ 1 ≤ L 1 . Furthermore, if η t = η , then for any δ ∈ (0 , 1) the following inequality holds with probability at least 1 -δ over θ ∼ P (the uniform distribution over { ( i 1 , . . . , i t ) : i j ∈ [ d ] } )

Remark 5. Stability bounds of the order O ( ηt/ ( nd )) were developed for RCD in [62] (Theorem 2). Their stability bounds hold in expectation. In contrast, we develop high-probability stability bounds of the order O ( ηt nd + η n ) . We plug the above bounds into Theorem 1, and derive the following PAC-Bayes bounds for RCD.

Corollary 10 (Generalization bound) . Let the assumptions in Corollary 5 hold. We further assume that the gradient is coordinate-wise Lipschitz continuous. When P is the uniform distribution, for any δ ∈ (0 , 1) , with probability at least 1 -δ over draws of S ∼ D n , for all posterior sampling distributions Q on [ d ] T , RCD with the hyperparameter θ and fixed step sizes η ≤ 2 / ˆ α satisfies

According to the above corollary, we can derive PAC-Bayes bounds of the order ˜ O (1 / √ n ) if T = O ( d √ n ) and η = O (1) . These choices of parameters were suggested in Theorem 7 in [62] to balance optimization and stability for RCD.

5 Related Work

Related work on stability The analysis of the generalization error through algorithmic stability is based on the landmark work of [7]. Generalization bounds via stability are algorithm-specific and have been applied to regularization algorithms, such as SVM regression and classification [7, 11, 16, 52]. Pioneering work on stability analysis of SGD was introduced in [20], which motivated much subsequent stability analyses of randomized iterative algorithms [10, 24, 25, 31, 38, 45]. The smoothness assumption in [20] was removed in more recent stability analyses [3, 27] of SGD. Stability bounds showing the benefit of low training errors on generalization were also developed for convex [27, 39, 50], nonconvex [28] and overparameterized models [12, 47, 59]. In recent works, stronger high-probability bounds via uniform stability have been developed. Some breakthroughs have narrowed down the difference between the risk and the empirical risk, leading to faster convergence rates with high probability [8, 17, 18].

Related work on PAC-Bayes bounds The PAC-Bayes theory of generalization dates back to works by [53] and [34] and further improved by [9, 26] and others. PAC-Bayes bounds are upper bounds on the generalization error of randomized learning algorithms, given in terms of data-dependent quantities that we can compute: the empirical error, and a quantity to measure the divergence between the PAC-Bayesian prior and posterior distributions, such as Kullback-Leibler (KL) divergence or the Rényi divergence [4]. The framework was later extended to allow learning the prior from the data, resulting in tighter bounds [2, 14, 15, 35, 42-44, 48]. The sensitivity of learning algorithms to small perturbations in the weights can be analyzed through the properties of a distribution of predictors, which may lead to regularity and improve the generalization bounds [5]. By evaluating the algorithmic stability on all possible outputs, stability of learning algorithms and PAC-Bayes bounds can be combined [32, 33, 35, 36, 40, 48, 57] and applied to randomized learning algorithms such as SGD and SGLD [30, 32, 35, 36]. For PAC-Bayes bounds, usually, the randomness is induced by a distribution on the parameters of a model. In [32], the authors isolated the randomness to view the randomized learning algorithm as deterministic, with hyperparameters following the distribution instead.

6 Conclusions

Under an assumption of a sub-exponential stability parameter, we derive sharper stability-based PAC-Bayes bounds for randomized learning algorithms by utilizing a moment bound. We show that the sub-exponential stability assumption holds for SGD and RCD, for which we develop PACBayes bounds as corollaries. Our results remove the need for the requirements of strong convexity, hyperparameter stability, and even smoothness in previous results.

Limitations . Future work of interest includes exploring other optimization methods that meet the sub-exponential assumption. It would also be interesting to study the quality of bounds obtainable when placing the PAC-Bayes prior on the parameters of a model (as in the classic approach) versus the hyperparametrs of the optimiser of the model (as in this work). Another interesting research direction is to further improve the bound by including additional assumptions. As noted in [23], for the deterministic case, the bounds can be √ n -times faster under a Bernstein condition between expectation and variance of loss functions.

Acknowledgements . The authors are grateful to the anonymous reviewers for their thoughtful comments and constructive suggestions. The work of Yunwen Lei is partially supported by the Research Grants Council of Hong Kong [Project No. 22303723]. The work of Sijia Zhou is funded by CSC and UoB scholarship. AK acknowledges funding by EPSRC Fellowship grant EP/P004245/1.

References

Z. Allen-Zhu, Z. Qu, P. Richtárik, and Y. Yuan. Even faster accelerated coordinate descent using nonuniform sampling. In International Conference on Machine Learning , pages 1110-1119. PMLR, 2016.

A. Ambroladze, E. Parrado-Hernández, and J. Shawe-Taylor. Tighter pac-bayes bounds. Advances in Neural Information Processing Systems , 19, 2006.

R. Bassily, V. Feldman, C. Guzmán, and K. Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. Advances in Neural Information Processing Systems , 33, 2020.

L. Bégin, P. Germain, F. Laviolette, and J.-F. Roy. Pac-bayesian bounds based on the rényi divergence. In Artificial Intelligence and Statistics , pages 435-444. PMLR, 2016.

C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning , pages 1613-1622. PMLR, 2015.

L. Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM Review , 60(2):223-311, 2018.

O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research , 2 (Mar):499-526, 2002.

O. Bousquet, Y. Klochkov, and N. Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference on Learning Theory , pages 610-626, 2020.

O. Catoni. A pac-bayesian approach to adaptive classification. preprint , 840, 2003.

Z. Charles and D. Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference on Machine Learning , pages 744-753, 2018.

J. Chen, H. Chen, X. Jiang, B. Gu, W. Li, T. Gong, and F. Zheng. On the stability and generalization of triplet learning. arXiv preprint arXiv:2302.09815 , 2023.

P. Deora, R. Ghaderi, H. Taheri, and C. Thrampoulidis. On the optimization and generalization of multi-head attention. 2023.

G. K. Dziugaite and D. M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 , 2017.

G. K. Dziugaite and D. M. Roy. Data-dependent pac-bayes priors via differential privacy. In Advances in Neural Information Processing Systems , volume 31, 2018.

G. K. Dziugaite, K. Hsu, W. Gharbieh, G. Arpino, and D. Roy. On the role of data in pac-bayes bounds. In International Conference on Artificial Intelligence and Statistics , pages 604-612. PMLR, 2021.

A. Elisseeff, T. Evgeniou, and M. Pontil. Stability of randomized learning algorithms. Journal of Machine Learning Research , 6(Jan):55-79, 2005.

V. Feldman and J. Vondrak. Generalization bounds for uniformly stable algorithms. In Advances in Neural Information Processing Systems , pages 9747-9757, 2018.

V. Feldman and J. Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly optimal rate. In Conference on Learning Theory , pages 1270-1279, 2019.

D. A. Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning , pages 1225-1234, 2016.

R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems , pages 315-323, 2013.

A. Kabán. Fractional norm regularization: Learning with very few relevant features. IEEE Transactions on Neural Networks and Learning Systems , 24(6):953-963, 2013.

Y. Klochkov and N. Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o (1 /n ) . Advances in Neural Information Processing Systems , 34, 2021.

T. Koren, R. Livni, Y. Mansour, and U. Sherman. Benign underfitting of stochastic gradient descent. In Advances in Neural Information Processing Systems , 2022.

I. Kuzborskij and C. Lampert. Data-dependent stability of stochastic gradient descent. In International Conference on Machine Learning , pages 2820-2829, 2018.

J. Langford and R. Schapire. Tutorial on practical prediction theory for classification. Journal of Machine Learning Research , 6(3), 2005.

Y. Lei and Y. Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In International Conference on Machine Learning , pages 5809-5819, 2020.

Y. Lei and Y. Ying. Sharper generalization bounds for learning with gradient-dominated objective functions. In International Conference on Learning Representations , 2021.

Y. Lei, M. Liu, and Y. Ying. Generalization guarantee of sgd for pairwise learning. Advances in Neural Information Processing Systems , 34:21216-21228, 2021.

J. Li, X. Luo, and M. Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. In International Conference on Learning Representations , 2020.

J. Lin, R. Camoriano, and L. Rosasco. Generalization properties and implicit regularization for multiple passes SGM. In International Conference on Machine Learning , pages 2340-2348, 2016.

B. London. A PAC-bayesian analysis of randomized learning with application to stochastic gradient descent. In Advances in Neural Information Processing Systems , pages 2931-2940, 2017.

B. London, B. Huang, and L. Getoor. Stability and generalization in structured prediction. The Journal of Machine Learning Research , 17(1):7808-7859, 2016.

D. A. McAllester. Some pac-bayesian theorems. Machine Learning , 37(3):355-363, 1999.

W. Mou, L. Wang, X. Zhai, and K. Zheng. Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints. In Conference on Learning Theory , pages 605-638, 2018.

J. Negrea, M. Haghifam, G. K. Dziugaite, A. Khisti, and D. M. Roy. Information-theoretic generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems , 32, 2019.

Y. Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization , 22(2):341-362, 2012.

K. Nikolakakis, F. Haddadpour, D. Kalogerias, and A. Karbasi. Black-box generalization: Stability of zeroth-order learning. Advances in Neural Information Processing Systems , 35:31525-31541, 2022.

K. E. Nikolakakis, F. Haddadpour, A. Karbasi, and D. S. Kalogerias. Beyond lipschitz: sharp generalization and excess risk bounds for full-batch gd. arXiv preprint arXiv:2204.12446 , 2022.

L. Oneto, M. Donini, M. Pontil, and J. Shawe-Taylor. Randomized learning and generalization of fair and private classifiers: From pac-bayes to stability and differential privacy. Neurocomputing , 416:231-243, 2020.

F. Orabona. Simultaneous model selection and optimization through parameter-free stochastic learning. In Advances in Neural Information Processing Systems , pages 1116-1124, 2014.

E. Parrado-Hernández, A. Ambroladze, J. Shawe-Taylor, and S. Sun. Pac-bayes bounds with data dependent priors. The Journal of Machine Learning Research , 13(1):3507-3531, 2012.

M. Pérez-Ortiz, O. Rivasplata, B. Guedj, M. Gleeson, J. Zhang, J. Shawe-Taylor, M. Bober, and J. Kittler. Learning pac-bayes priors for probabilistic neural networks. arXiv preprint arXiv:2109.10304 , 2021.

M. Pérez-Ortiz, O. Rivasplata, J. Shawe-Taylor, and C. Szepesvári. Tighter risk certificates for neural networks. Journal of Machine Learning Research , 22, 2021.

A. Raj, L. Zhu, M. Gurbuzbalaban, and U. Simsekli. Algorithmic stability of heavy-tailed sgd with general loss functions. In International Conference on Machine Learning , pages 28578-28597, 2023.

A. Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In International Conference on Machine Learning , pages 449-456, 2012.

D. Richards and I. Kuzborskij. Stability & generalisation of gradient descent for shallow neural networks without the neural tangent kernel. Advances in Neural Information Processing Systems , 34, 2021.

O. Rivasplata, E. Parrado-Hernández, J. S. Shawe-Taylor, S. Sun, and C. Szepesvári. Pac-bayes bounds for stable algorithms with instance-dependent priors. In Advances in Neural Information Processing Systems , pages 9214-9224, 2018.

F. Salehi, L. E. Celis, and P. Thiran. Stochastic optimization with bandit sampling. arXiv preprint arXiv:1708.02544 , 2017.

M. Schliserman and T. Koren. Stability vs implicit bias of gradient methods on separable data and beyond. In Conference on Learning Theory , pages 3380-3394. PMLR, 2022.

S. Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

S. Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. Journal of Machine Learning Research , 11(Oct):2635-2670, 2010.

J. Shawe-Taylor and R. C. Williamson. A pac analysis of a bayesian estimator. In Proceedings of the Tenth Annual Conference on Computational Learning Theory , pages 2-9, 1997.

S. Smale and D.-X. Zhou. Online learning with markov sampling. Analysis and Applications , 7(01): 87-113, 2009.

K. Sridharan, S. Shalev-Shwartz, and N. Srebro. Fast rates for regularized objectives. In Advances in Neural Information Processing Systems , pages 1545-1552, 2009.

S. U. Stich, A. Raj, and M. Jaggi. Safe adaptive importance sampling. Advances in Neural Information Processing Systems , 30, 2017.

S. Sun, M. Yu, J. Shawe-Taylor, and L. Mao. Stability-based pac-bayes analysis for multi-view learning algorithms. Information Fusion , 86:76-92, 2022.

T. Sun, Y. Sun, and W. Yin. On markov chain gradient descent. Advances in Neural Information Processing Systems , 31, 2018.

H. Taheri and C. Thrampoulidis. Generalization and stability of interpolating neural networks with minimal width. arXiv preprint arXiv:2302.09235 , 2023.

R. Van Handel. Probability in high dimension. Lecture Notes (Princeton University) , 2014.

R. Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

P. Wang, L. Wu, and Y. Lei. Stability and generalization for randomized coordinate descent. In International Joint Conference on Artificial Intelligence , pages 3104-3110, 2021.

P. Wang, Y. Lei, Y. Ying, and D.-X. Zhou. Stability and generalization for markov chain stochastic gradient methods. In Advances in Neural Information Processing Systems , pages 37735-37748, 2022.

B. Woodworth, K. K. Patel, S. Stich, Z. Dai, B. Bullins, B. Mcmahan, O. Shamir, and N. Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334-10343. PMLR, 2020.

Y. Zhang, W. Zhang, S. Bald, V. Pingali, C. Chen, and M. Goswami. Stability of sgd: Tightness analysis and improved bounds. In Uncertainty in Artificial Intelligence , pages 2364-2373. PMLR, 2022.

P. Zhao and T. Zhang. Stochastic optimization with importance sampling for regularized loss minimization. In International Conference on Machine Learning , pages 1-9, 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gave an improved PAC-Bayes bound for uniformly stable randomized algorithms. It introduced new assumptions and proof techniques. The paper also applied the general theorem to concrete applications (SGD and RCD) and got better or novel PAC-Bayes bound for these two algorithms.\n\nSTRENGTHS:\n1. The general PAC-Bayes bound improvement in this paper is good. Although the improvement doesn't always hold, it is sharper in most cases and can be up to $\\sqrt{n}$, which is significant. This result seems to be quite non-trivial.\n2. The authors introduced new assumptions and proof techniques. The newly introduced assumption of sub-exponential stability is interesting and provided a new perspective compared to the previous assumption. The new proof technique is innovative and could be studied and used on proving other problems.\n3. The paper gave concrete applications to the general theorem, which shows the power of such bound. The bound for SGD is sharper and requires no strong convexity and the bound for RCD is completely novel. These results are nice contributions.\n4. The paper is well-written. As someone who doesn't know much about PAC-Bayes, reading the paper doesn't take too much effort. Sufficient background information is given, definitions are clearly and maths are rigorous. Contributions and novelty of the paper is highlighted properly.\n\nWEAKNESSES:\n1. One thing I find confusing is that the prior and posterior are over the hyperparameters but not on the hypothesis class, while usually in the PAC-Bayes framework the prior and the posterior are defined over the hypothesis class. This part should be clarified, I think more background or explanation is needed.\n2. The use of the word \"parameter\" and \"hyperparameter\" is a bit confusing because the meanings seem to be different from its usual meaning. Actually the parameter class $W$ barely shows in the paper and I think it is basically equivalent to $H$ here. From my understanding, \"hyperparameter\" here is just the random seed of the algorithm, but usually it means something like the step size. Maybe changing the names could make it better.\n3. A very tiny typo: line 101 should be \"parameters\" instead of \"parameter\".\n\nQUESTIONS:\n1. In line 79, it says \"given the parameter $w\\in W$\", what does this mean? Shouldn't the algorithm output $w$ instead of given $w$? Or do you mean \"given the parameter class $W$\"?\n2. In line 183-185 you claimed that the benefit of PAC-Bayes bound is that the prior and posterior can be selected and posterior parameters can be optimized? I understand the prior can be selected freely but isn't the posterior depends on $S$ and the algorithm? This part may need more explanation.\n3. If my understanding is correct, in both of your applications (SGD and RCD), even in the generalized version, the sampling distribution is still a fixed distribution across all iterations. Can you extend your result to algorithms where the sampling distribution could be different in every iteration or even depend on the history?\n\nLIMITATIONS:\nThe authors addressed the limitation of their paper properly.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work aims to provide stronger generalization guarantees for stochastic algorithms through the analysis of stability and PAC-Bayes. Such algorithms include SGD and RCD. The work uses a milder assumption in the analysis under the same problem setting as [29]. The results can be extended beyond strong convexity from previous results and hold for non-smooth convex problems.\n\nSTRENGTHS:\nIf the results are correct, they are tighter than a previous work [29] in the same problem setting. Also, they require milder assumptions (uniform stability and sub-exp stability), whereas [29] requires some stability w.r.t. the changes of the data and the hyperparameters. Therefore, although they build from the setting in [29], they require different analysis techniques.\n\nWEAKNESSES:\nThe significance of the results is not very clear. Please see more details in Question 1.\n\nThe soundness of the results is not clear. Please see more details in the next section from point 2 and beyond.\n\nQUESTIONS:\n1. Although it’s a similar setting as [29], it’s unclear why you need PAC-Bayes analysis to guarantee all distributions Q over the hyperparameter space. It doesn’t seem like you are trying to learn some good distributions Q of randomized rule. As far as I can see, you are interested in providing a generalization guarantee for some fixed distribution, regardless of how arbitrary it is. Therefore, isn’t it enough to have results for a fixed distribution like in [15]?\n\n2. I feel there is something missing in the results related to assumption 1. It looks like assumption 1 is a characterization of a randomized algorithm A defined by some distribution P. And it essentially says all deterministic algorithms defined by \\theta in the support of P are nice in the sense of being \\beta_\\theta-uniform stable. Also, the distribution P itself is nice in the sense that the \\beta_\\theta from such P is w.h.p. not too far from the mean with some parameter \\tilde b.\n\na) Is the stability constant \\tilde b in assumption 1 a function of the distribution P? \n\nb) It’s perhaps better to say, in line 136, that it’s uniform stable “w.r.t. a loss function \\ell” since it makes little sense to talk about stability without mentioning the loss. Also, the \\ell\\in[0,M] assumption in line 152 should perhaps be associated with the stability conditions in assumption 1 rather than be put independently.\n\nc) Again, assumption 1 is a characterization of some A defined by some P. Hence, it’s not clear what it means by “a learning algorithm A that satisfies Assumption 1” (in line 152) without mentioning the associated distribution. Also, it’s most likely that other distributions Q won’t follow the same sub-exp stability constant \\tilde b as P (the formula in theorem 1).\n\n3. Do you have any idea why Theorem 1 doesn’t depend on the loss range M? It is neither in the bound nor in the stability constant \\beta_\\theta in your example in Lemma 3.\n\n4. In Lemma 3, the SGDU is explicitly associated with a distribution P=uniform. It’s not clear to me why this randomized algorithm is \\beta_\\theta uniformly stable, which only depends on some specific \\theta.\n\nLIMITATIONS:\nThere were quite some fast-rate results in PAC-Bayes analysis in other applications. There are also some fast-rate analyses for the algorithms addressed in the work but with stronger assumptions on losses. However, due to the “max” function in theorem 1, the results here cannot go beyond O(1/sqrt(n)) convergence rate. Whether a fast rate is possible here can be an interesting question.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors study PAC-Bayesian bounds on the generalization error of uniformly stable algorithms. They consider algorithms in which the randomness comes from a particular hyperparameter $\\theta$, e.g., stochastic gradient descent (SGD), in which the randomness comes from selecting the indices used to determine which samples are utilized in the gradients. This way, they consider algorithms that are stable for every choice of the hyperparamter $\\theta$, but they allow different degrees of stability $\\beta\\_\\theta$ for every parameter. They only require that the stability parameter is at least exponentially close to its mean under some prior distribution $\\mathbb{P}$, that is $\\beta\\_\\theta \\leq \\mathbb{E}\\_{\\vartheta \\sim \\mathbb{P}} [\\beta\\_\\vartheta] + \\tilde{b} \\log(1/\\delta)$ with probability $1 - \\beta$ over the draws of $\\theta$ (sub-exponential condition).\n\nIn this setup, they obtain generic PAC Bayes bounds that scale linearly with the KL divergence between a posterior over the parameters $\\mathbb{Q}$ and the prior $\\mathbb{P}$. These bounds have the same rate as the bounds from Bousquet, Klochkov, and Zhivotovskiy [8, Corollary 8] when the parameters are deterministic. Moreover, compared to the PAC-Bayes bounds from London [29, Theorem 2], they substitute the parameter uniform stability by the sub-exponential condition, which is weaker, and their bound still has a better rate.\n\nIn terms of applications:\n* They show that SGD with a Lipschitz, convex, and smooth loss is stable for all parameters $\\theta$ selecting the indices to determine which samples are utilized in the gradients, and that that the parameters satisfy the sub-exponential condition for a uniform prior distribution. This way, they can obtain a generalization bound with rate $\\tilde{O}(1/\\sqrt{n})$, improving upon London's bounds [9] that require strong convexity and potentially have a worse rate.\n* They show that the smoothness assumption can be lifted and there are still choices of the learning rate and number of iterations under which the rate is $\\tilde{O}(1/\\sqrt{n})$.\n* They show that randomized coordinate descent (RCD) with a Lipschitz, convex, smooth, and coordinate-Lipschitz loss is also stable for all parameters $\\theta$ used to select the coordinate where the gradient is taken, and  that the parameters satisfy the sub-exponential condition for a uniform prior distribution. This way, they obtain bounds of a similar order to Wang, Wu, and Lei [56] that hold with high probability instead than in expectation.\n\nSTRENGTHS:\nIn terms of originality, the main source of originality is the consideration of per hyper-parameter uniform stability + a sub-exponential condition in the stability parameter. The analysis employed for finding the bound for a fixed hyper-parameter is very similar to Bousquet, Klochkov, and Zhivotovskiy [8], but the consideration of a sub-exponential condition on the stability parameter and the introduction of said condition to find generic bounds is novel.\n\nSimilarly, the proofs to show that SGD and RCD are stable with a sub-exponential stability parameter are based on common analyses in the field + standard concentration arguments. However, combining them into this setting is also new as far as I know.\n\nIn general, the idea of introducing a prior and a posterior over the parameters did not seem very relevant in the beginning, as many times the hyper-parameters distribution is independent of the data and one can always have $\\mathbb{Q} = \\mathbb{P}$ with a zero divergence. However, it appeared to be quite clever afterwards since one only needs to prove the sub-exponential condition for the prior and see how far the actual distribution is from that.\n\nThe results seem significant since the control of the sampling is often a drawback in many analyses of SGD and its variants, that have very specific forms of sampling. Also, some of the results obtained are already as good as previous results in the literature when reduced to their setting, or more general, or better.\n\nWEAKNESSES:\nI believe that some things could be clarified. For instance what I mentioned in the strengths about the introduction of a prior and a posterior. Some commentary about that would be helpful. At least for readers like me, who may be wondering why would one introduce this seemingly artificial distribution over the hyper-parameters.\n\nAlso, it would be good to clarify that in Corollaries 4,7, and 9 the prior $\\mathbb{P}$ must be the uniform distribution. Similarly, it would be easier for the reader if in Definition 7 you would use the same letter used fro priors throughout, i.e. $\\mathbb{P}$.\n\nSometimes in the text and in the appendices, references are cited without the specific place where one should look. This difficults the reader experience. For instance, For Lemma A.2 only [55] is mentioned, when it refers to a combination of Propositions 2.5.2 and 2.7.1. I hope the authors go back to their text and look at their citations and try to specify better where the reader should look.\n\nIn Lemma A.1, the authors seem to use the version v1 of the paper [8] in the arxiv. They may want to look at Theorem 4 in the latest version (and the version actually published). They will see that the constants are improved: $3 \\sqrt{2}$ is actually $4$ and $\\sqrt{6}$ is actually $\\sqrt{2}$. \n\nLemma A.3 is not the Donsker-Varadhan formula. The Donsker-Varadhan formula deals with the supremum of the functions $h$ appearing in their equation, finding a dual equivalence. The shown theorem is the Gibbs variational principle (see e.g. Lemma 4.10 of [A]).\n\nThere seem to be some mistakes in the constants in the proof of Theorem 1. Not very important as they seem to be solvable:\n* After line 538 shouldn't it be 36 instead of 72?\n* After line 538, where did the multiplicative factor of 2 in the second exponential?\n* It seems you forgot a + 3 in the right hand side of equation 547. This should also appear in the general statement of the theorem as $(D\\_{\\mathrm{KL}}(\\mathbb{Q} \\Vert \\mathbb{P}) + \\log \\frac{1}{\\delta} + 3)$. Shouldn't it? \n\n**Additional References**\n\nRamon van Handel. \"Probability in High Dimension\". APC 550 Lecture Notes. Princeton University. 2016.\n\nQUESTIONS:\n* I don't understand how you go from the equation after line 593 to the equation after line 540.\n* Could you elaborate on more cases in Remark 2? From what I see, in Remark 2 using the standard $T \\in \\Theta(n)$ and a small learning rate $\\eta \\in O(1/\\sqrt{n})$ yields a reasonable rate of $\\tilde{O}(1/\\sqrt{n})$ as well.  \n\nRepeating here some of the questions in the weaknesses:\n* After line 538 shouldn't it be 36 instead of 72?\n* After line 538, where did the multiplicative factor of 2 in the second exponential?\n* It seems you forgot a + 3 on the right-hand side of equation 547. This should also appear in the general statement of the theorem as $(D\\_{\\mathrm{KL}}(\\mathbb{Q} \\Vert \\mathbb{P}) + \\log \\frac{1}{\\delta} + 3)$. Shouldn't it?\n\nLIMITATIONS:\nThe authors discuss some of the limitations throughout the text and in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors extend the work of London [29] and prove PAC-Bayesian generalization bounds for uniformly stable algorithms. Compared to [29], the present work removes some required assumptions in the work of [29], such as the hyperparameter stability. To do so, they required a new assumption called \"Sub-exponential stability\" and proved Theorem 1 based on this new assumption. They further instantiate their theorem for stochastic gradient descent and randomized coordinate descent.\n\nSTRENGTHS:\n- The hyperparameter stability assumption is not required in the present analysis.\n- The strong convexity assumption in the SGD applications is removed compared to the analysis of [29].\n- To the best of my knowledge, this is the first time that Randomized Coordinate Descent is studied in the literature.\n\nWEAKNESSES:\n- I find the $\\lesssim$ and the $\\widetilde{O}$ notations sometimes confusing (see the comments below).\n- The statements of the theorems of [29] do not seem to be correctly restated.\n- There are no experiment evaluations.\n\nQUESTIONS:\n_Major comments_:  \n\n- **l173**: In the current form of Theorem 2, I could not compare it with Theorem 2 of [29]. Indeed, for me, in [29], their bound includes the term $\\beta_{\\mathcal{Z}}$ while you have instead $\\mathbb{E}\\_{\\theta\\sim\\mathbb{P}}[\\beta\\_\\theta]$. Indeed, by looking at the two definitions, i.e. your Definition 1 and one in [29], the expectation over $\\mathbb{P}$ is not in the same place.  \n- **l179-181**: I am confused with your $\\widetilde{O}(\\cdot)$ notation and your discussion below.\nIndeed, your discussion involves the KL divergence, but it is hidden in the two equations. Hence, the discussion is hard to understand in this current form.  \n- **l270**: In [29], $R\\_S$ does not have to be necessarily $\\gamma$-strongly convex and $\\alpha$-smooth. Indeed, they consider another function (called $F$ in their notations) to be minimized.  \n\n_Minor comments_:  \n\n- **l152**: I would have expected to have $M$ in the bound since you introduced the notation in the statement of the theorem.  \n- **l494**: You cite the wrong Theorem in [8]; you cite an old version of the paper. By the way, the constants are improved in the new version of the theorem.  \n- **l512**: Since [55] is a book, can you cite the section or the theorem of the Lemma for readability?  \n\n_Typos_:  \n\n- **l154**: There is \"..\" at the end of the sentence.  \n- **In Appendix B**: there is some \"$-$\" that are in fact \"$+$\".\n\nLIMITATIONS:\nI do not see any potential negative societal impact of their work since it is theoretical.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-02"}