Title: Stability and Sharper Risk Bounds with Convergence Rate $O(1/n^2)$
Venue: NeurIPS 2024
Accepted: no
Keywords: algorithmic stability

generalization bounds

excess risk bounds;
Abstract: The sharpest known high probability excess risk bounds are up to $O(1/n)$ for empirical risk minimization and projected gradient descent via algorithmic stability [Klochkov and Zhivotovskiy, 2021]. In this paper, we show that high probability excess risk bounds of order up to $O(1/n^2)$ are possible. We discuss how high probability excess risk bounds reach $O(1/n^2)$ under strongly convexity, smoothness and Lipschitz continuity assumptions for empirical risk minimization, projected gradient descent and stochastic gradient descent. Besides, to the best of our knowledge, our high probability results on the generalization gap measured by gradients for nonconvex problems are also the sharpest.

1

Abstract

The sharpest known high probability excess risk bounds are up to O (1 /n ) for empirical risk minimization and projected gradient descent via algorithmic stability [Klochkov and Zhivotovskiy, 2021]. In this paper, we show that high probability excess risk bounds of order up to O (1 /n 2 ) are possible. We discuss how high probability excess risk bounds reach O (1 /n 2 ) under strongly convexity, smoothness and Lipschitz continuity assumptions for empirical risk minimization, projected gradient descent and stochastic gradient descent. Besides, to the best of our knowledge, our high probability results on the generalization gap measured by gradients for nonconvex problems are also the sharpest.

1 Introduction

Algorithmic stability is a fundamental concept in learning theory [Bousquet and Elisseeff, 2002], which can be traced back to the foundational works of Vapnik and Chervonenkis [1974] and has a deep connection with learnability [Rakhlin et al., 2005, Shalev-Shwartz et al., 2010, ShalevShwartz and Ben-David, 2014]. It is not difficult for only providing in-expectation error bounds via stability arguments. However, high probability bounds are beneficial to understand the robustness of optimization algorithms [Bousquet et al., 2020, Klochkov and Zhivotovskiy, 2021] and are much more challenging [Feldman and Vondrak, 2019, Bousquet et al., 2020, Lv et al., 2021]. In this paper, our goal is to improve the high probability risk bounds via algorithmic stability.

Let us start with some standard notations. We have a set of independent and identically distributed observations S = { z 1 , . . . , z n } sampled from a probability measure ρ defined on a sample space Z := X × Y . Based on the training set S , our goal is to build a model h : X ↦→ Y for prediction, where the model is determined by parameter w from parameter space W ⊂ R d . The performance of a model w on an example z can be quantified by a loss function f ( w ; z ) , where f : W×Z ↦→ R + . Then the population risk and the empirical risk of w ∈ W , respectively as

where E z denotes the expectation w.r.t. z .

Let w ∗ ∈ arg min w ∈W F ( w ) be the model with the minimal population risk in W and w ∗ ( S ) ∈ arg min w ∈W F S ( w ) be the model with the minimal empirical risk w.r.t. dataset S . Let A ( S ) be the output of a (possibly randomized) algorithm A on the dataset S . Let ∥ · ∥ 2 denote the Euclidean norm and ∇ g ( w ) denote a subgradient of g at w .

Traditional generalization analysis aims to bound the generalization error F ( A ( S )) -F S ( A ( S )) w.r.t 30 the algorithm A and the dataset S . Based on the technique developed by Feldman and Vondrak [2018, 31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

2019], Bousquet et al. [2020] provide the sharpest high probability bounds of O ( L/ √ n ) , where the loss function f ( · , · ) is bounded by M . No matter how stable the algorithm is, the high probability generalization bound will not be smaller than O ( L/ √ n ) . This is sampling error term scaling as O (1 / √ n ) that controls the generalization error [Klochkov and Zhivotovskiy, 2021].

A frequently used alternative to generalization bounds, that can avoid the sampling error, are the excess risk bounds. The excess risk of algorithm A w.r.t. the dataset S is F ( A ( S )) -F ( w ∗ ) , which is more essential because it considers both generalization error and optimization error. Recently, Klochkov and Zhivotovskiy [2021] provided the best high probability excess risk bounds of order up to O (log n/n ) for empirical risk minimization (ERM) and projected gradient descent (PGD) algorithms via algorithmic stability.

On the other hand, Zhang et al. [2017], Li and Liu [2021], Xu and Zeevi [2024] derived high probability excess risk bounds with O ( 1 /n 2 ) for ERM and stochastic gradient descent (SGD) via uniform convergence when the sample number satisfies n = Ω( d ) , which implied that the rate O ( 1 /n 2 ) is possible. However, the results obtained by the uniform convergence technique are related to the dimension d , which is unacceptable in high-dimensional learning problems. Since stability analysis can yield dimension-free bounds, we naturally have the following question:

Can algorithmic stability provide high probability excess risk bounds with the rate beyond O (1 /n ) ?

The main results of this paper answers this question positively. We provides the first high probability bounds that are dimension-free with the rate O (1 /n 2 ) for ERM, PGD and SGD. Our framework can also be used to solve other stable algorithms.

To this end, we develop the generalization gap measured by gradients. Our bounds under nonconvex setting are tighter than existing works based on both algorithmic stability [Fan and Lei, 2024] and uniform convergence [Xu and Zeevi, 2024]. This is why we can achieve dimension-free excess risk bounds of order O (1 /n 2 ) . In fact, in nonconvex problems, optimization algorithms can only find a local minimizer and we can only obtain optimization error bounds for ∥∇ F S ( A ( S )) ∥ 2 [Ghadimi and Lan, 2013]. Therefore, it is important to study the generalization behavior of A ( S ) measured by gradients. Under Polyak-Lojasiewicz condition, we also obtain sharper results for both generalization bounds of gradients and excess risk bounds. Our route to excess risk bounds can also be applied to various stable algorithms and complex learning scenarios. In this paper, we take ERM, PGD, and SGD as examples to explore the stability of stochastic convex optimization algorithms with strongly convex losses. We provide tighter high probability dimension-free excess risk bounds of up to O (1 /n 2 ) comapring with existing works based on both algorithmic stability [Klochkov and Zhivotovskiy, 2021, Fan and Lei, 2024] and uniform convergence [Zhang et al., 2017, Li and Liu, 2021, Xu and Zeevi, 2024].

Besides, to obtain tighter bounds, we obtain a tighter p -moment bound for sums of vector-valued functions by introducing the optimal Marcinkiewicz-Zygmund's inequality for random variables taking values in a Hilbert space in the proof, which has more potential applications in vector-valued functional data.

This paper is organized as follows. The related work is reviewed in Section 2. In Section 3, we present our main results for stability and generalization. We give applications to ERM, PGD and SGD in Section 4. The conclusion is given in Section 5. All the proofs and additional lemmata are deferred to the Appendix.

2 Related Work

Algorithmic stability. Algorithmic stability is a classical approach in generalization analysis, which 75 can be traced back to the foundational works of [Vapnik and Chervonenkis, 1974]. It gave the 76 generalization bound by analyzing the sensitivity of a particular learning algorithm when changing 77 one data point in the dataset. Modern framework of stability analysis was established by Bousquet 78 and Elisseeff [2002], where they presented an important concept called uniform stability. Since 79 then, a lot of works based on uniform stability have emerged. On one hand, generalization bounds 80 with algorithmic stability have been significantly improved by Feldman and Vondrak [2018, 2019], 81 Bousquet et al. [2020], Klochkov and Zhivotovskiy [2021]. On the other hand, different algorithmic 82 stability measures such as uniform argument stability [Liu et al., 2017, Bassily et al., 2020], on 83 average stability [Shalev-Shwartz et al., 2010, Kuzborskij and Lampert, 2018], hypothesis stability 84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

[Bousquet and Elisseeff, 2002, Charles and Papailiopoulos, 2018], hypothesis set stability [Foster et al., 2019], pointwise uniform stability [Fan and Lei, 2024], PAC-Bayesian stability [Li et al., 2020], locally elastic stability [Deng et al., 2021], collective stability [London et al., 2016] and uniform stability in gradients [Lei, 2023, Fan and Lei, 2024] have been developed. Most of them provided the connection on stability and generalization in expectation. Bousquet and Elisseeff [2002], Elisseeff et al. [2005], Feldman and Vondrak [2018, 2019], Bousquet et al. [2020], Klochkov and Zhivotovskiy [2021], Fan and Lei [2024] considered high probability bounds. However, only Fan and Lei [2024] developed vector-valued bounds (eg: generalization bounds of gradients), which can be the order at most O ( M/ √ n ) and remain improvement.

Uniform convergence. Uniform convergence is another popular approach in statistical learning theory to study generalization bounds [Fisher, 1922, Vapnik, 1999, Van der Vaart, 2000]. The main idea is to bound the generalization gap by its supremum over the whole (or a subset) of the hypothesis space via some space complexity measures, such as VC dimension, covering number and Rademacher complexity. For finite-dimensional problem, Kleywegt et al. [2002] provided that the generalization error is O ( √ d/n ) depended on the sample size n and the dimension of parameters d in high probability. In nonconvex settings, Mei et al. [2018] showed that the empirical of generalization error is O ( √ d/n ) . Xu and Zeevi [2024] developed a novel 'uniform localized convergence' framework using generic chaining for the minimization problem and provided the localized generalization bounds in gradients O ( max { ∥ w -w ∗ ∥ 2 , 1 n } (√ d n + d n )) , which is the optimal result when we only consider the order of n . However, uniform convergence results are related to the dimension d , which is unacceptable in high-dimensional learning problems.

3 Stability and Generalization

To derive sharper generalization bounds of gradients, we need to develop a novel concentration inequality which provide p -moment bound for sums of vector-valued functions. For a real-valued random variable Y , the L p -norm of Y is defined by ∥ Y ∥ p := ( E [ | Y | p ]) 1 p . Similarly, let ∥ · ∥ denote the norm in a Hilbert space H . Then for a random variable X taking values in a Hilbert space, the L p -norm of X is defined by ∥∥ X ∥∥ p := ( E [ ∥ X ∥ p ]) 1 p .

3.1 A Moment Bound for Sums of Vector-valued Functions

Here we present our sharper moment bound for sums of vector-valued functions of n independent variables.

Theorem 1. Let Z = ( Z 1 , . . . , Z n ) be a vector of independent random variables each taking values in Z , and let g 1 , . . . , g n be some functions: g i : Z n ↦→ H such that the following holds for any i ∈ [ n ] :

∥ E [ g i ( Z ) | Z i ] ∥ ≤ M a.s.,

E [ g i ( Z ) | Z [ n ] \{ i } ] = 0 a.s.,

g i satisfies the bounded difference property with β , namely, for any i = 1 , . . . , n , the following inequality holds

Then, for any p ≥ 2 , we have 122

Remark 1. The proof is motivated by Bousquet et al. [2020]. Under the same assumptions, Fan and 123 Lei [2024] also established the following inequality 1 124

It is easy to verify that our result is tighter than result provided by Fan and Lei [2024] for both the first and second term. Comparing Theorem 1 with (2), the larger p is, the tighter our result is relative to (2). In the worst case, when p = 2 , the constant of our first term is 0 . 879 times tighter than (2), and the constant of our second term is 0 . 634 times tighter than (2). This is because we derive the optimal Marcinkiewicz-Zygmund's inequality for random variables taking values in a Hilbert space in the proof.

The improvement of this concentration inequality is meaningful. On one hand, we derive the optimal Marcinkiewicz-Zygmund's inequality for random variables taking values in a Hilbert space. On the other hand, in Section 3.2, we will carefully construct vector-valued functions which satisfies all the assumptions in Theorem 1 and ensures M = 0 at the same time. Under this condition, we can eliminate the first term. When we use Theorem 1 instead of (2) in the whole proofs, at least 0 . 634 times tighter bound can be obtained strictly.

3.2 Sharper Generalization Bounds in Gradients

Let S = { z 1 , . . . , z n } be a set of independent random variables each taking values in Z and S ′ = { z ′ 1 , . . . , z ′ n } be its independent copy. For any i ∈ [ n ] , define S ( i ) = { z i , . . . , z i -1 , z ′ i , z i +1 , . . . , z n } be a dataset replacing the i -th sample in S with another i.i.d. sample z ′ i . We introduce some basic definitions here and we want to emphasize that our main Theorem 2 and Theorem 3 do not need smoothness assumption and PL condition.

Definition 1. Let g : W ↦→ R . Let γ, µ < 0 .

We say g is γ -smooth if

Let g ∗ = min w ∈W g ( w ) . We say g satisfies the Polyak-Lojasiewicz (PL) condition with parameter µ > 0 on W if

Then we define uniform stability in gradients.

Definition 2 (Uniform Stability in Gradients) . Let A be a randomized algorithm. We say A is β -uniformly-stable in gradients if for all neighboring datasets S, S ( i ) , we have

Remark 2. Gradient-based stability was firstly introduced by Lei [2023], Fan and Lei [2024] to describe the generalization performance for nonconvex problems. In nonconvex problems, we can only find a local minimizer by optimization algorithms which may be far away from the global minimizer. Thus the convergence does not make much sense in function values. Instead, the convergence of ∥∇ F S ( A ( S )) ∥ 2 was often studied in the optimization community [Ghadimi and Lan, 2013]. Since the population risk of gradients ∥∇ F ( A ( S )) ∥ 2 can be decomposed as the convergence of ∥∇ F S ( A ( S )) ∥ 2 and the generalization gap ∥∇ F ( A ( S )) - ∇ F S ( A ( S )) ∥ 2 , the generalization analysis of ∥∇ F ( A ( S )) -∇ F S ( A ( S )) ∥ 2 is important, which can be achieved by uniform stability in gradients.

Theorem 2 (Generalization via Stability in Gradients) . Assume for any S and any z , ∥∇ f ( A ( S ); z ) ∥ 2 ≤ M . If A is β -uniformly-stable in gradients, then for any δ ∈ (0 , 1) , the

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

following inequality holds with probability at least 1 -δ 161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

Remark 3. Theorem 2 is a direct application via Theorem 1 where we denote g i ( S ) = E z ′ i [ E Z [ ∇ f ( A ( S ( i ) ) , Z ) ] -∇ f ( A ( S ( i ) ) , z i ) ] and find that g i ( S ) satisfies all the assumptions in Theorem 1. As a comparison, Fan and Lei [2024] also developed high probability bounds under same assumptions, but our bounds are sharper since our moment inequality for sums of vector-valued functions are tighter as we have discussed in Remark 1. Next, we derive sharper generalization bound of gradients under same assumptions.

Theorem 3 (Sharper Generalization via Stability in Gradients) . Assume for any S and any z , ∥∇ f ( A ( S ); z ) ∥ 2 ≤ M . If A is β -uniformly-stable in gradients, then for any δ ∈ (0 , 1) , the following inequality holds with probability at least 1 -δ

Remark 4. Note that the factor in Theorem 2 before 1 / √ n is O ( M √ log ( e/δ ) ) , which depends on the bound of ∥∇ f ( · , · ) ∥ 2 . However, the factor in Theorem 3 before 1 / √ n is O ( √ E Z [ ∥∇ f ( A ( S ); Z ) ∥ 2 2 ] log 1 /δ + β log(1 /δ ) ) , not involving the possibly large term M . As is known, optimization algorithms often provide parameters approaching the optimal solution, which make the term E Z [ ∥∇ f ( A ( S ); Z ) ∥ 2 2 ] much more smaller than M . We will give further reasonable results under more assumptions such as smoothness in Lemma 1 and Lemma 2.

On the other hand, best high probability bounds based on uniform convergence [Xu and Zeevi, 2024] 177 is 178

which is the optimal result when we only consider the order of n . However, uniform convergence 179 results are related to the dimension d , which is unacceptable in high-dimensional learning problems. 180

Note that (4) requires an additional smoothness-type assumption. As a comparison, when f is 181 γ -smoothness, our result in Theorem 3 can be easily derived as 182

√

This result implies that when the uniformly stable in gradients parameter β is smaller than 1 / n , our 183 bound is tighter than (4) and is dimension independent. Note that Theorem 3 holds in nonconvex 184 problems, to the best of our knowledge, this is the sharpest upper bound in both uniform convergence 185 and algorithmic stability analysis. 186

Here we give the proof sketch of Theorem 3, which is motivated by the analysis in Klochkov and 187 Zhivotovskiy [2021]. The key idea is to build vector functions q i ( S ) = h i ( S ) -E S { z i } [ h i ( S )] 188 where we define h i ( S ) = E z ′ i [ E Z [ ∇ f ( A ( S ( i ) ) , Z ) ] -∇ f ( A ( S ( i ) ) , z i ) ] . These functions satisfy 189 all the assumptions in Theorem 1 and ensure the factor M in Theorem 1 to 0 . Then the term O (1 / √ n ) 190 can be eliminated. 191

Lemma1. Let assumptions in Theorem 3 hold. Suppose the function f is γ -smooth and the population 192 risk F satisfies the PL condition with parameter µ . Then for any δ ∈ (0 , 1) , when n ≥ 16 γ 2 log 6 δ µ 2 , 193 with probability at least 1 -δ , we have 194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

Remark 5. The following inequality can be easily derived using triangle inequality and CauchyBunyakovsky-Schwarz inequality:

Above inequality implies that excess risk can be bound by the optimization gradient error ∥∇ F S ( A ( S )) ∥ 2 and uniform stability in gradients β . Note that the assumption F ( w ∗ ) = O (1 /n ) is common and can be found in Srebro et al. [2010], Lei and Ying [2020], Liu et al. [2018], Zhang et al. [2017], Zhang and Zhou [2019]. This is natural since F ( w ∗ ) is the minimal population risk. On the other hand, we can derive that under µ -strongly convex and γ -smooth assumptions for the objective function f , uniform stability in gradients can be bounded of order O (1 /n ) for ERM and PGD. Thus high probability excess risk can be bounded of order up to O ( 1 /n 2 ) under these common assumptions via algorithmic stability. Comparing with current best related work [Klochkov and Zhivotovskiy, 2021], they are insensitive to the stability parameter being smaller than O (1 /n ) and their best rates can only up to O (1 /n ) . Although we involve extra smoothness and PL condition assumptions, these assumptions are also common in optimization community and our work can fully utilize these assumptions.

Besides, we discuss uniform stability in gradients for common algorithms such as ERM, PGD, and SGD in Section 4. Our results can be easily extended to other stable algorithms. Due to smoothness's property to link the uniform stability in gradients with uniform argument stability, many works [Bassily et al., 2020, Feldman and Vondrak, 2019, Hardt et al., 2016] exploring uniform argument stability can also use our framework.

Finally, the population risk of gradients ∥∇ F ( A ( S )) ∥ 2 can be gracefully bounded by the empirical risk of gradients ∥∇ F S ( A ( S )) ∥ 2 under strong growth condition (SGC), that connects the rates at which the stochastic gradients shrink relative to the full gradient Vaswani et al. [2019].

Definition 3 (Strong Growth Condition) . We say SGC holds if

Remark 6. There has been some related work that takes SGC into assumption Solodov [1998], Vaswani et al. [2019], Lei [2023]. Vaswani et al. [2019] has proved that the squared-hinge loss with linearly separable data and finite support features satisfies the SGC. Note that we only suppose this condition holds in Lemma 2.

Lemma 2 (SGC case) . Let assumptions in Theorem 3 hold and suppose SGC holds. Then for any δ > 0 , with probability at least 1 -δ , we have

Remark 7. Lemma 2 build a connection between the population gradient error and the empirical gradient error under Lipschitz, nonconvex, nonsmooth and SGC case and elucidate that the population gradient error can be bounded of up to O (1 /n ) under nonconvex problems.

4 Application

In this section, we analysis stochastic convex optimization with strongly convex losses. The most common setting is where at each round, the learner gets information on f through a stochastic gradient oracle [Rakhlin et al., 2012]. To derive uniform stability in gradients for algorithms, we firstly introduce the strongly convex assumption.

Definition 4. We say g is µ -strongly convex if 232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

4.1 Empirical Risk Minimizer

Empirical risk minimizer is one of the classical approaches for solving stochastic optimization (also referred to as sample average approximation (SAA)) in machine learning community. The following lemma shows the uniform stability in gradient for ERM under µ -strongly convexity and γ -smoothness assumptions.

Lemma 3 (Stability of ERM) . Suppose the objective function f is µ -strongly-convex and γ -smooth. For any w ∈ W and any z , suppose that ∥∇ f ( w ; z ) ≤ M ∥ . Let ˆ w ∗ ( S ( i ) ) be the ERM of F S ( i ) ( w ) that denotes the empirical risk on the samples S ( i ) = { z 1 , ..., z ′ i , ..., z n } and ˆ w ∗ ( S ) be the ERM of F S ( w ) on the samples S = { z 1 , ..., z i , ..., z n } . For any S ( i ) and S , there holds the following uniform stability bound of ERM:

Then, we present the application of our main sharper Theorem 3. In the strongly convex and smooth case, we provide a up to O ( 1 /n 2 ) high probability excess risk guarantee valid for any algorithms depending on the optimal population error F ( w ∗ ) .

Theorem 4. Let assumptions in Theorem 3 and Lemma 3 hold. Suppose the function f is nonnegative. Then for any δ ∈ (0 , 1) , when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , we have

Furthermore, assume F ( w ∗ ) = O ( 1 n ) , we have

Remark 8. Theorem 4 shows that when the objective function f is µ -strongly convex, γ -smooth and nonnegative, high probability risk bounds can even up to O ( 1 /n 2 ) for ERM. The most related work to ours is Zhang et al. [2017]. They also obtain the O ( 1 /n 2 ) -type bounds for ERM by uniform convergence of gradients approach. However, they need the sample number n = Ω( γd/µ ) , which is related to the dimension d . Our risk bounds are dimension independent and only require the sample number n = Ω( γ 2 /µ 2 ) . Comparing with Klochkov and Zhivotovskiy [2021], we add two assumptions, smoothness and F ( w ∗ ) = O (1 /n ) , but our bounds also tighter, from O (1 /n ) to O ( 1 /n 2 ) .

4.2 Projected Gradient Descent

Note that when the objective function f is strongly convex and smooth, the optimization error can be ignored. However, the generalization analysis framework proposed by Klochkov and Zhivotovskiy [2021] does not use smoothness assumption, which only derive high probability excess risk bound of order O (1 /n ) after T = O (log n ) steps under strongly convex and smooth assumptions. In this subsection, we provide sharper risk bound under the same iteration steps, which is because our generalization analysis also fully utilized the smooth assumptions. Here we give the definition of PGD.

Definition 5 (Projected Gradient Descent) . Let w 1 = o ∈ R d be an initial point and { η t } t be a sequence of positive step sizes. PGD updates parameters by

where ∇ F S ( w t ) denotes a subgradient of F w.r.t. w t and Π W is the projection operator onto W .

268

269

270

271

272

273

274

275

276

277

278

279

280

281

Lemma 4 (Stability of Gradient Descent) . Suppose the objective function f is µ -strongly-convex and γ -smooth. For any w ∈ W and any z , suppose that ∥∇ f ( w ; z ) ∥ 2 ≤ M . Let w i t be the output of F S ( i ) ( w ) on t -th iteration on the samples S ( i ) = { z 1 , ..., z ′ i , ..., z n } in running PGD, and w t be the output of F S ( w ) on t -th iteration on the samples S = { z 1 , ..., z i , ..., z n } in running PGD. Let the constant step size η t = 1 /γ . For any S ( i ) and S , there holds the following uniform stability bound of PGD:

Remark 9. The derivations of Feldman and Vondrak [2019] in Section 4.1.2 (See also Hardt et al. [2016] in Section 3.4) imply that if the objective function f is γ -smooth in addition to µ -strongly convexity and M -Lipschitz property, then PGD with the constant step size η = 1 /γ is ( 2 M nµ ) -uniformly argument stable for any number of steps, which means that PGD is ( 2 Mγ nµ ) -uniformlystable in gradients regardless of iteration steps.

Theorem 5. Let assumptions in Theorem 3 and Lemma 3 hold. Suppose the function f is nonnegative. Let { w t } t be the sequence produced by PGD with η t = 1 /γ . Then for any δ ∈ (0 , 1) , when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , we have

Furthermore, assume F ( w ∗ ) = O ( 1 n ) and let T ≍ log n , we have 282

Remark 10. Theorem 5 shows that under the same assumptions as Klochkov and Zhivotovskiy [2021], our bound is O ( F ( w ∗ ) log(1 /δ ) n + log 2 n log 2 (1 /δ ) n 2 ) . Comparing with their bound O ( log n log(1 /δ ) n ) , we are sharper because F ( w ∗ ) is the minimal population risk, which is a common assumption towards sharper risk bounds Srebro et al. [2010], Lei and Ying [2020], Liu et al. [2018], Zhang et al. [2017], Zhang and Zhou [2019].

4.3 Stochastic Gradient Descent

Stochastic gradient descent optimization algorithm has been widely used in machine learning due to its simplicity in implementation, low memory requirement and low computational complexity per iteration, as well as good practical behavior. Here we give the definition of standard SGD.

Definition 6 (Stochastic Gradient Descent) . Let w 1 = o ∈ R d be an initial point and { η t } t be a sequence of positive step sizes. SGD updates parameters by

where ∇ f ( w t ; z i t ) denotes a subgradient of f w.r.t. w t and i t is independently drawn from the uniform distribution over [ n ] := { 1 , 2 , . . . , n } .

Lemma 5 (Stability of SGD) . Suppose the objective function f is µ -strongly-convex and γ -smooth. For any w ∈ W and any z , suppose that ∥∇ f ( w ; z ) ∥ 2 ≤ M . Let w i t be the output of F S ( i ) ( w ) on t -th iteration on the samples S ( i ) = { z 1 , ..., z ′ i , ..., z n } in running PGD and and w t be the output of F S ( w ) on t -th iteration on the samples S = { z 1 , ..., z i , ..., z n } in running SGD. For any S ( i ) and S , there holds the following uniform stability bound of SGD:

where ϵ opt ( w t ) = F S ( w t ) -F S ( ˆ w ∗ ( S )) and ˆ w ∗ ( S ) is the ERM of F S ( w ) .

Next, we introduce a necessary assumption in stochastic optimization theory.

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

Assumption 1. Assume the existence of σ > 0 satisfying 303

where E i t denotes the expectation w.r.t. i t .

Remark 11. Assumption 1 is a standard assumption from the stochastic optimization theory [Nemirovski et al., 2009, Ghadimi and Lan, 2013, Ghadimi et al., 2016, Kuzborskij and Lampert, 2018, Zhou et al., 2018, Bottou et al., 2018, Lei and Tang, 2021], which essentially bounds the variance of the stochastic gradients for dataset S .

Theorem 6. Let assumptions in Theorem 3 and Lemma 5 hold. Suppose Assumption 1 holds and the function f is nonnegative. Let { w t } t be the sequence produced by SGD with η t = η 1 t -θ , θ ∈ (0 , 1) and η 1 < 1 2 γ . Then for any δ ∈ (0 , 1) , when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , we have

Remark 12. When θ < 1 / 2 , we take T ≍ n 2 /θ . When θ = 1 / 2 , we take T ≍ n 4 and when θ > 1 / 2 , we set T ≍ n 2 / (1 -θ ) . Then according to Theorem 6, the population risk of gradient is bounded by O ( log 2 n log 3 (1 /δ ) n 2 + F ( w ∗ ) log 2 (1 /δ ) n ) . To the best of our knowledge, this is the first high probability population gradient bound ∥∇ F ( w t ) ∥ 2 for SGD via algorithmic stability.

Theorem 7. Let Assumptions in Theorem 3 and Lemma 5 hold. Suppose Assumption 1 holds and the function f is nonnegative. Let { w t } t be the sequence produced by SGD with η t = 2 µ ( t + t 0 ) such that t 0 ≥ max { 4 γ µ , 1 } . Then for any δ > 0 , when n ≥ 16 γ 2 log 6 δ µ 2 and T ≍ n 2 , with probability at least 1 -δ , we have

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

Furthermore, assume F ( w ∗ ) = O ( 1 n ) , we have 320

Remark 13. Theorem 7 implies that high probability risk bounds for SGD optimization algorithm can be up to O (1 /n 2 ) and the rate is dimension-free in high-dimensional learning problems. We compare Theorem 7 with most related work. For algorithmic stability, high probability risk bounds in Fan and Lei [2024] is up to O (1 /n ) when choosing optimal iterate number T for SGD optimization algorithm. To the best of knowledge, we are faster than all the existing bounds. The best high probability risk bounds of order O (1 /n 2 ) are given by Li and Liu [2021] via uniform convergence, which require the sample number n = Ω( γd/µ ) depending on dimension d .

5 Conclusion

In this paper, we improve a p -moment concentration inequality for sums of vector-valued functions. By carefully constructing functions, we apply this moment concentration to derive sharper generalization bounds in gradients in nonconvex problems, which can further be used to obtain sharper high probability excess risk bounds for stable optimization algorithms. In application, we study three common algorithms: ERM, PGD, SGD. To the best of our knowledge, we provide the sharpest high probability dimension independent O (1 /n 2 ) -type for these algorithms.

321

322

323

324

325

326

327

328

329

330

331

332

333

334

References

Bassily, V. Feldman, C. Guzmán, and K. Talwar. Stability of stochastic gradient descent on nonsmooth convex 336 losses. In Proceedings of the 34th International Conference on Neural Information Processing Systems 337 (NeurIPS) , volume 33, pages 4381-4391, 2020. 338

Bottou, F. E. Curtis, and J. Nocedal. Optimization methods for large-scale machine learning. SIAM review , 339 60(2):223-311, 2018. 340

Bousquet and A. Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2: 341 499-526, 2002. 342

Bousquet, Y. Klochkov, and N. Zhivotovskiy. Sharper bounds for uniformly stable algorithms. In Conference 343 on Learning Theory , pages 610-626. PMLR, 2020. 344

Charles and D. Papailiopoulos. Stability and generalization of learning algorithms that converge to global 345 optima. In International conference on machine learning , pages 745-754. PMLR, 2018. 346

J. Davis. Gamma function and related functions. Handbook of mathematical functions , 256, 1972. 347

De la Pena and E. Giné. Decoupling: from dependence to independence . Springer Science & Business Media, 348 2012. 349

Deng, H. He, and W. Su. Toward better generalization bounds with locally elastic stability. In International 350 Conference on Machine Learning , pages 2590-2600. PMLR, 2021. 351

Elisseeff, T. Evgeniou, M. Pontil, and L. P. Kaelbing. Stability of randomized learning algorithms. Journal of 352 Machine Learning Research , 6(1), 2005. 353

Fan and Y. Lei. High-probability generalization bounds for pointwise uniformly stable algorithms. Applied 354 and Computational Harmonic Analysis , 70:101632, 2024. 355

Feldman and J. Vondrak. Generalization bounds for uniformly stable algorithms. Advances in Neural 356 Information Processing Systems , 31, 2018. 357

Feldman and J. Vondrak. High probability generalization bounds for uniformly stable algorithms with nearly 358 optimal rate. In Conference on Learning Theory , pages 1270-1279. PMLR, 2019. 359

360

361

362

A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character , 222(594-604): 309-368, 1922.

363

364

365

366

367

368

369

370

J. Foster, S. Greenberg, S. Kale, H. Luo, M. Mohri, and K. Sridharan. Hypothesis set stability and generalization. Advances in Neural Information Processing Systems , 32, 2019.

Ghadimi and G. Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization , 23(4):2341-2368, 2013.

Ghadimi, G. Lan, and H. Zhang. Mini-batch stochastic approximation methods for nonconvex stochastic composite optimization. Mathematical Programming , 155(1):267-305, 2016.

Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Karimi, J. Nutini, and M. Schmidt. Linear convergence of gradient and proximal-gradient methods under the 371 polyak-łojasiewicz condition. In ECML , pages 795-811. Springer, 2016. 372

J. Kleywegt, A. Shapiro, and T. Homem-de Mello. The sample average approximation method for stochastic 373 discrete optimization. SIAM Journal on optimization , 12(2):479-502, 2002. 374

Klochkov and N. Zhivotovskiy. Stability and deviation optimal risk bounds with convergence rate o (1 /n ) . 375 Advances in Neural Information Processing Systems , 34:5065-5076, 2021. 376

Kuzborskij and C. Lampert. Data-dependent stability of stochastic gradient descent. In Proceedings of the 377 35th International Conference on Machine Learning (ICML) , pages 2815-2824. PMLR, 2018. 378

Latała and K. Oleszkiewicz. On the best constant in the khinchin-kahane inequality. Studia Mathematica , 379 109(1):101-104, 1994. 380

Lei. Stability and generalization of stochastic optimization with nonconvex and nonsmooth problems. In The 381 Thirty Sixth Annual Conference on Learning Theory , pages 191-227. PMLR, 2023. 382

Lei and K. Tang. Learning rates for stochastic gradient descent with nonconvex objectives. IEEE Transactions 383 on Pattern Analysis and Machine Intelligence , 43(12):4505-4511, 2021. 384

Lei and Y. Ying. Fine-grained analysis of stability and generalization for stochastic gradient descent. In 385 International Conference on Machine Learning , pages 5809-5819. PMLR, 2020. 386

Li, X. Luo, and M. Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. 387 In International Conference on Learning Representations , 2020. 388

Li and Y. Liu. Improved learning rates for stochastic optimization: Two theoretical viewpoints. arXiv preprint 389 arXiv:2107.08686 , 2021. 390

Liu, X. Zhang, L. Zhang, R. Jin, and T. Yang. Fast rates of erm and stochastic approximation: Adaptive to 391 error bound conditions. Advances in Neural Information Processing Systems , 31, 2018. 392

Liu, G. Lugosi, G. Neu, and D. Tao. Algorithmic stability and hypothesis complexity. In International 393 Conference on Machine Learning , pages 2159-2167. PMLR, 2017. 394

London, B. Huang, and L. Getoor. Stability and generalization in structured prediction. The Journal of 395 Machine Learning Research , 17(1):7808-7859, 2016. 396

Luo and D. Zhang. Khintchine inequality on normed spaces and the application to banach-mazur distance. 397 arXiv preprint arXiv:2005.03728 , 2020. 398

Lv, J. Wang, J. Liu, and Y . Liu. Improved learning rates of a functional lasso-type svm with sparse multi-kernel 399 representation. Advances in Neural Information Processing Systems , 34:21467-21479, 2021. 400

Mei, Y. Bai, and A. Montanari. The landscape of empirical risk for nonconvex losses. The Annals of Statistics , 401 46(6A):2747-2774, 2018. 402

Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic 403 programming. SIAM Journal on optimization , 19(4):1574-1609, 2009. 404

Pinelis. Optimum bounds for the distributions of martingales in banach spaces. The Annals of Probability , 405 pages 1679-1706, 1994. 406

Rakhlin, S. Mukherjee, and T. Poggio. Stability results in learning theory. Analysis and Applications , 3(04): 407 397-417, 2005. 408

Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic 409 optimization. In Proceedings of the 29th International Coference on International Conference on Machine 410 Learning , pages 1571-1578, 2012. 411

Rivasplata, E. Parrado-Hernández, J. S. Shawe-Taylor, S. Sun, and C. Szepesvári. Pac-bayes bounds for stable 412 algorithms with instance-dependent priors. Advances in Neural Information Processing Systems , 31, 2018. 413

414

415

Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

Shalev-Shwartz, O. Shamir, N. Srebro, and K. Sridharan. Learnability, stability and uniform convergence. The Journal of Machine Learning Research , 11:2635-2670, 2010.

Smale and D.-X. Zhou. Learning theory estimates via integral operators and their approximations. Constructive approximation , 26(2):153-172, 2007.

V. Solodov. Incremental gradient algorithms with stepsizes bounded away from zero. Computational Optimization and Applications , 11:23-35, 1998.

Srebro, K. Sridharan, and A. Tewari. Optimistic rates for learning with a smooth loss. arXiv preprint arXiv:1009.3896 , 2010.

W. Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Vapnik and A. Chervonenkis. Theory of Pattern Recognition . 1974.

N. Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks , 10(5):988-999, 1999.

Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of sgd for over-parameterized models and an accelerated perceptron. In The 22nd international conference on artificial intelligence and statistics , pages 1195-1204. PMLR, 2019.

Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. 431 Cambridge university press, 2018. 432

Xu and A. Zeevi. Towards optimal problem dependent generalization error bounds in statistical learning 433 theory. Mathematics of Operations Research , 2024. 434

Zhang and Z.-H. Zhou. Stochastic approximation of smooth and strongly convex functions: Beyond the o(1/t) 435 convergence rate. In Conference on Learning Theory , pages 3160-3179. PMLR, 2019. 436

Zhang, T. Yang, and R. Jin. Empirical risk minimization for stochastic convex optimization: O(1/n)-and 437 o(1/n**2)-type of risk bounds. In Conference on Learning Theory , pages 1954-1979. PMLR, 2017. 438

Zhou, Y. Liang, and H. Zhang. Generalization error bounds with probabilistic guarantee for sgd in nonconvex 439

optimization. arXiv preprint arXiv:1802.06903 , 2018. 440

441

442

443

445

446

447

448

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

A Additional definitions and lemmata

Lemma 6 (Equivalence of tails and moments for random vectors [Bassily et al., 2020]) . Let X be a random variable with

for some a, b ≥ 0 and for any p ≥ 2 . Then for any δ ∈ (0 , 1) we have, with probability at least 1 -δ , 444

Lemma 7 (Vector Bernstein's inequality [Pinelis, 1994, Smale and Zhou, 2007]) . Let { X i } n i =1 be a sequence of i.i.d. random variables taking values in a real separable Hilbert space. Assume that E [ X i ] = µ , E [ ∥ X i -µ ∥ 2 ] = σ 2 , and ∥ X i ∥ ≤ M , ∀ 1 ≤ i ≤ n , then for all δ ∈ (0 , 1) , with probability at least 1 -δ we have

Definition 7 (Weakly self-Bounded Function) . Assume that a, b > 0 . A function f : Z n ↦→ [0 , + ∞ ) 449 is said to be ( a, b ) -weakly self-bounded if there exist functions f i : Z n -1 ↦→ [0 , + ∞ ) that satisfies 450 for all Z n ∈ Z n , 451

Lemma 8 ([Klochkov and Zhivotovskiy, 2021]) . Suppose that z 1 , . . . , z n are independent random variables and the function f : Z n ↦→ [0 , + ∞ ) is ( a, b ) -weakly self-bounded and the corresponding function f i satisfy f i ( Z n ) ≥ f ( Z n ) for ∀ i ∈ [ n ] and any Z n ∈ Z n . Then, for any t > 0 ,

Definition 8. A Rademacher random variable is a Bernoulli variable that takes values ± 1 with probability 1 2 each.

B Proofs of Section 3.1

The proof of Theorem 1 is motivated by Bousquet et al. [2020], which need the MarcinkiewiczZygmund's inequality for random variables taking values in a Hilbert space and the McDiarmid's inequality for vector-valued functions.

Firstly, we derive the optimal constants in the Marcinkiewicz-Zygmund's inequality for random variables taking values in a Hilbert space.

Lemma 9 (Marcinkiewicz-Zygmund's Inequality for Random Variables Taking Values in a Hilbert Space) . Let X 1 , . . . , X n be random variables taking values in a Hilbert space with E [ X i ] = 0 for all i ∈ [ n ] . Then for p ≥ 2 we have

Remark 14. Comparing with Marcinkiewicz-Zygmund's inequality given by Fan and Lei [2024], we provide best constants. Next, we give the proof of Lemma 9.

The Marcinkiewicz-Zygmund's inequality can be proved by using its connection to Khintchine468 Kahane's inequality. Thus, we introduce the best constants in Khintchine-Kahane's inequality for 469 random variables taking values from a Hilbert space here. 470

Lemma 10 (Best constants in Khintchine-Kahane's inequality in Hilbert space [Latała and 471 Oleszkiewicz, 1994, Luo and Zhang, 2020]) . For all p ∈ [2 , ∞ ) and for all choices of Hilbert 472 space H , finite sets of vectors X i , . . . , X n ∈ X ∈ H , and independent Rademacher variables 473 r 1 , . . . , r n , 474

where C p = 2 1 2 { Γ ( p +1 2 ) √ π } 1 p . 475

Proof of Lemma 9. The symmetrization argument goes as follows: Let ( r 1 , . . . , r n ) be i.i.d. with 476 P ( r i = 1) = P ( r i = -1) = 1 / 2 and besides such that r 1 , . . . , r n and ( X 1 , . . . , X n ) are independent. 477 Then by independence and symmetry, according to Lemma 1.2.6 of De la Pena and Giné [2012], 478 conditioning on ( X 1 , . . . , X n ) yields 479

As for the conditional expectation in (7), notice that by independence 480

According to Lemma 10, for v n -almost every x 1 , . . . , x n ∈ R n , where v n := P ◦ ( X 1 , . . . , X n ) -1 481 denotes the distribution of ( X 1 , . . . , X n ) , we have 482

where C = 2 p 2 Γ ( p +1 2 ) √ π and C is optimal. This means that for any constant C ′ such that 483

for all n ∈ N and for each collection of vectors x 1 , . . . , x n , it follows that C ′ ≥ C . 484

From (8) and (9), we can infer that 485

Taking expectations in the above inequalities and (7) yield that 486

To see optimality let the above statement hold for some constants C ′ in place of C . Then if we choose 487 X i := x i r i , 1 ≤ i ≤ n with arbitrary reals vectors x 1 , . . . , x n , it follows that 488

whence we can conclude from (10) that C ′ ≥ C . Thus we obtain that C ′ = C . 489

Notice that by Holder's inequality 490

Plugging (12) into (11), we have 491

where C = 2 p 2 Γ ( p +1 2 ) √ π is a constant. 492

Next, we use the following form of Stirling's formula for the Gamma-function, which follows from 493 (6.1.5), (6.1.15) and (6.1.38) in Davis [1972] to bound the constant C . For every x > 0 , there exists a 494 µ ( x ) ∈ (0 , 1 / (12 x )) such that 495

with g ( p ) = ( 1 + 1 p ) p/ 2 e v ( p ) -1 / 2 , where 0 < v ( p ) < 1 / (6( p +1)) . By Taylor's formula we have 497 that 498

and that for every k ∈ N 0 499

Therefor we obtain with k = 1 that 500

where the last equality follows from elementary calculus. Similarly, 501

Thus 496

502

503

504

505

Thus, we have

which implies that C is strictly smaller than √ 2 e -p/ 2 p p/ 2 for all p ≥ 2 .

Since C = 1 g ( p ) √ 2 e -p/ 2 p p/ 2 and g ( p ) ≥ e -1 4 p , we can obtain that the relative error between C and √ 2 e -p/ 2 p p/ 2 is equal to

using Mean Value Theorem. This implies that the corresponding relative errors between C and 506 √ 2 e -p/ 2 p p/ 2 converge to zero as p tends to infinity. 507

The proof is complete. 508

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

Then we introduce the McDiarmid's inequality for vector-valued functions. We firstly consider real-valued functions, which follows from the standard tail-bound of McDiarmid's inequality and Proposition 2.5.2 in Vershynin [2018].

Lemma 11 (McDiarmid's Inequality for real-valued functions) . Let Z i , . . . , Z n be independent random variables, and f : Z n ↦→ R such that the following inequality holds for any z i , . . . , z i -1 , z i +1 , . . . , z n

Then for any p > 1 we have

To derive the McDiarmid's inequality for vector-valued functions, we need the expected distance between f ( Z 1 , . . . , Z n ) and its expectation.

Lemma 12 ([Rivasplata et al., 2018]) . Let Z i , . . . , Z n be independent random variables, and f : Z n ↦→ H is a function into a Hilbert space H such that the following inequality holds for any z i , . . . , z i -1 , z i +1 , . . . , z n

Then we have

Now, we can easily derive the p -norm McDiarmid's inequality for vector-valued functions which refines from Fan and Lei [2024] with better constants.

Lemma 13 (McDiarmid's inequality for vector-valued functions) . Let Z i , . . . , Z n be independent random variables, and f : Z n ↦→ H is a function into a Hilbert space H such that the following inequality holds for any z i , . . . , z i -1 , z i +1 , . . . , z n

Then for any p > 1 we have 528

Proof of Lemma 13. Define a real-valued function h : Z n ↦→ R as

Wenotice that this function satisfies the increment condition. For any i and z 1 , . . . , z i -1 , z i +1 , . . . , z n , 529 we have 530

Therefore, we can apply Lemma 11 to the real-valued function h and derive the following inequality 531

According to Lemma 12, we know the following inequality E [ h ( Z 1 , . . . , Z n )] ≤ √ nβ . Combing the above two inequalities together and we can derive the following inequality

532

533

The proof is complete. 534

535

536

537

538

539

540

541

542

543

544

545

Then, by the triangle inequality 546

To bound the first term, since ∥ E [ g i | Z i ] ∥ ≤ M , we can check that the vector-valued function 547 f ( Z 1 , . . . , Z n ) = ∑ n i =1 E [ g i | Z i ] satisfies (13) with β = 2 M , and E [ E [ g i | Z i ]] = 0 , applying Lemma 548 13 with β = 2 M , we have 549

Then we start to bound the second term of the right hand side of (14). Observe that 550

551

552

553

554

555

556

557

where the expectation is taken with respect to the variables Z j , j ∈ P l +1 ( i ) \ P l ( i ) . Changing any Z j would change g l i by β . Therefore, we apply Lemma 13 with f = g l i where there are 2 l random variables and obtain a uniform bound

Taking integration over ( Z i , Z [ n ] \ P l +1 ( i ) ) , we have ∥ ∥ ∥ ∥ g l i -g l +1 i ∥ ∥ ∥ ∥ p ≤ ( √ 2 p +1) √ 2 l β as well.

Next, we turn to the sum ∑ i ∈ P l g l i -g l +1 i for any P l ∈ P l . Since g l i -g l +1 i for i ∈ P l depends only on Z i , Z [ n ] \ P l , the terms are independent and zero mean conditioned on Z [ n ] \ P l . Applying Lemma 9, we have for any p ≥ 2 ,

Integrating with respect to ( Z [ n ] \ P l ) and using ∥ ∥ ∥ ∥ g l i -g l +1 i ∥ ∥ ∥ ∥ p ≤ ( √ 2 p +1) √ 2 l β , we have 558

Proof of Theorem 1. For g ( Z 1 , . . . , Z n ) and A ⊂ [ n ] , we write ∥∥ g ∥∥ p ( Z A ) = ( E [ ∥ f ∥ p Z A ]) 1 p . Without loss of generality, we suppose that n = 2 k . Otherwise, we can add extra functions equal to zero, increasing the number of therms by at most two times.

Consider a sequence of partitions P 0 , . . . , P k with P 0 = {{ i } : i ∈ [ n ] } , P k with P n = { [ n ] } , and to get P l from P l +1 we split each subset in P l +1 into two equal parts. We have

We have |P l | = 2 k -l and | P | = 2 l for each P ∈ P l . For each i ∈ [ n ] and l = 0 , . . . , k , denote by P l ( i ) ∈ P l the only set from P l that contains i . In particular, P 0 ( i ) = { i } and P K ( i ) = [ n ] .

For each i ∈ [ n ] and every l = 0 , . . . , k consider the random variables

i.e. conditioned on z i and all the variables that are not in the same set as Z i in the partition P l . In particular, g 0 i = g i and g k i = E [ g i | Z i ] . We can write a telescopic sum for each i ∈ [ n ] ,

Then using triangle inequality over all sets P l ∈ P l , we have 559

Recall that 2 k ≤ n due to the possible extension of the sample. Then we have 560

We can plug the above bound together with (15) into (14), to derive the following inequality 561

The proof is completed. 562 563

564

565

566

567

568

C Proofs of Section 3

Proof of Theorem 2. Let S = { z 1 , . . . , z n } be a set of independent random variables each taking values in Z and S ′ = { z ′ 1 , . . . , z ′ n } be its independent copy. For any i ∈ [ n ] , define S ( i ) = { z i , . . . , z i -1 , z ′ i , z i +1 , . . . , z n } be a dataset replacing the i -th sample in S with another i.i.d. sample z ′ i . Then we can firstly write the following decomposition

We denote that g i ( S ) = E z ′ i [ E Z [ ∇ f ( A ( S ( i ) ) , Z ) ] -∇ f ( A ( S ( i ) ) , z i ) ] , thus we have 569

where the inequality holds from the definition of uniform stability in gradients. 570

According to our assumptions, we get ∥ g i ( S ) ∥ 2 ≤ 2 M and 571

̸

where this equality holds from the fact that z i and Z follow from the same distribution. For any 572 i ∈ [ n ] , any j = i and any z ′′ j , we have 573

where S ( i ) = { z i , . . . , z i -1 , z ′ i , z i +1 , . . . , z n } . Thus, we have verified that three conditions in 574 Theorem 1 are satisfied for g i ( S ) . We have the following result for any p > 2 575

We can combine the above inequality and (16) to derive the following inequality 576

According to Lemma 6 for any δ ∈ (0 , 1) , with probability at least 1 -δ , we have 577

This implies that 578

579

The proof is completed.

Proof of Theorem 3. We can firstly write the following decomposition 580

We denote that h i ( S ) = E z ′ i [ E Z [ ∇ f ( A ( S ( i ) ) , Z ) ] -∇ f ( A ( S ( i ) ) , z i ) ] , we have 581

where the inequality holds from the definition of uniform stability in gradients. 583

̸

Then, for any i = 1 , . . . , n , we define q i ( S ) = h i ( S ) -E S { z i } [ h i ( S )] . It is easy to verify that 584 E S \{ z i } [ q i ( S )] = 0 and E z i [ h i ( S )] = E z i [ q i ( S )] -E z i E S \{ z i } [ q i ( S )] = 0 -0 = 0 . Also, for any 585 j ∈ [ n ] with j = i and z ′′ j ∈ Z , we have the following inequality 586

For the first term ∥ h i ( S ) -h i ( z 1 , . . . , z j -1 , z ′′ j , z j +1 , . . . , z n ) ∥ 2 , it can be bounded by 2 β according 587 to the definition of uniform stability. Similar result holds for the second term ∥ E S \{ z i } [ h i ( S )] -588 E S \{ z i } [ h i (1 , . . . , z j -1 , z ′′ j , z j +1 , . . . , z n )] ∥ 2 according to the uniform stability. By a combina589 tion of the above analysis, we get ∥ q i ( S ) -q i (1 , . . . , z j -1 , z ′′ j , z j +1 , . . . , z n ) ∥ 2 ≤ ∥ h i ( S ) -590 h i (1 , . . . , z j -1 , z ′′ j , z j +1 , . . . , z n ) ∥ 2 ≤ 4 β . 591

Thus, we have verified that three conditions in Theorem 1 are satisfied for q i ( S ) . We have the 592 following result for any p ≥ 2 593

Furthermore, we can derive that 594

which implies that 582

Due to the i.i.d. property between S and S ′ , we know that E S ′ [ ∇ F ( A ( S ′ ))] = E S [ ∇ F ( A ( S ))] . 595 Thus, combined above equality, (17) and (18), we have 596

According to Lemma 6 for any δ ∈ (0 , 1) , with probability at least 1 -δ/ 3 , we have 597

Next, we need to bound the term ∥ E S ′ [ ∇ F S ( A ( S ′ ))] -E S [ ∇ F ( A ( S ))] ∥ 2 . There holds that 598 ∥ E S E S ′ [ ∇ F S ( A ( S ′ ))] ∥ 2 = ∥ E S [ ∇ F ( A ( S ))] ∥ 2 . Then, by the Bernstein inequality in Lemma 7, we 599 obtain the following inequality with probability at least 1 -δ/ 3 , 600

Then using Jensen's inequality, we have 601

Combing (19), (20) with (21), we finally obtain that with probability at least 1 -2 δ/ 3 , 602

Next, since S = { z i , . . . , z n } , we define p = p ( z 1 , . . . , z n ) = E Z [ ∥∇ f ( A ( S ); Z ) ∥ 2 2 ] and p i = 603 p i ( z 1 , . . . , z n ) = sup z i ∈Z p ( z i , . . . , z n ) . So there holds p i ≥ p for any i = 1 , . . . , n and any 604

{ z 1 , . . . , z n } ∈ Z n . Also, there holds that 605

where the first inequality follows from the Jensen's inequality. The second and third inequalities 606 follow from the definition of uniform stability in gradients. The last inequality holds from that 607 ( a + b ) 2 ≤ 2 a 2 +2 b 2 . 608

From (23), we know that p is (8 nβ 2 , 2 nβ 4 ) weakly self-bounded. Thus, by Lemma 8, we obtain that 609 with probability at least 1 -δ/ 3 , 610

where the last inequality follows from that √ ab ≤ a + b 2 for all a, b > 0 . Thus, we have 611

Substituting (24) into (22), we finally obtain that with probability at least 1 -δ 612

According to inequality √ a + b = √ a + √ b for any a, b > 0 , with probability at least 1 -δ , we have 613

The proof is complete. 614

615

Proof of Remark 4. According to the proof in Theorem 3, we have the following inequality that with 616 probability at least 1 -δ 617

Since f ( w ) is γ -smooth, we have 618

Plugging (27) into (26), we have 619

where the second inequality holds because a + b + a + b for any a, b > 0 , which means that 620

The proof is complete. 621

622

Proof of Lemma 1. Inequality (28) implies that 623

When F ( w ) satisfies the PL condition, there holds the following error bound property (refer to 624 Theorem 2 in Karimi et al. [2016]) 625

Thus, we have 626

When n ≥ 16 γ 2 log 6 δ µ 2 , we have 2 γ √ log 6 δ n ≤ µ 2 , then we can derive that 627

This implies that 628

Then, substituting (29) into (28), when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ 629

The proof is complete. 630

Proof of Remark 5. Here we briefly prove the results given in Remark 5. Since F satisfies the PL 631 condition with µ , we have 632

So to bound F ( A ( S )) -F ( A ( S )) , we need to bound the term ∥∇ F ( A ( S )) ∥ 2 . And there holds 633

From Lemma 1, if f is M -Lipschitz and γ -smooth and F satisfies PL condition with µ , for any δ > 0 , 634 when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , there holds 635

where C is a positive constant and the last inequality follows from Lemma 4.1 of Srebro et al. [2010] 636 when f is nonegative and γ -smooth (see (44)). 637

Combing above inequality with (30), (31), we can derive that 638

The proof is complete. 639

640

Proof of Lemma 2. According to the proof in Theorem 3, we have the following inequality with 641 probability at least 1 -δ 642

Since SGC implies that E Z [ ∥∇ f ( w ; Z ) ∥ 2 2 ] ≤ ρ ∥∇ F ( w ) ∥ 2 2 , according to inequalities √ ab ≤ ηa + 1 η b 643 and √ a + b ≤ √ a + √ b for any a, b, η > 0 , we have the following inequality with probability at 644 least 1 -δ 645

which implies that 646

The proof is complete. 647

D Proofs of ERM 648

649

̸

where the first inequality follows from the fact that ˆ w ∗ ( S ( i ) ) is the ERM of F S ( i ) and the second 650 inequality follows from the Lipschitz property. Furthermore, for ˆ w ∗ ( S ( i ) ) , the convexity of f and 651 the strongly-convex property of F S imply that its closest optima point of F S is ˆ w ∗ ( S ) (the global 652 minimizer of F S is unique). Then, there holds that 653

Then we get 654

which implies that ∥ ˆ w ∗ ( S ( i ) ) -ˆ w ∗ ( S ) ∥ 2 ≤ 4 M nµ . Combined with the smoothness property of f we 655 obtain that for any S ( i ) and S 656

The proof is complete. 657

Proof of Theorem 4. Since F is µ -strongly convex, we have 658

So to bound F ( ˆ w ∗ ) -F ( w ∗ ) , we need to bound the term ∥∇ F ( ˆ w ∗ ) ∥ 2 2 . And there holds 659

From Lemma 1, if f is M -Lipschitz and γ -smooth and F S is µ -strongly convex, for any δ > 0 , when 660 n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , there holds 661

̸

where the last inequality follows from Lemma 4.1 of Srebro et al. [2010] when f is nonegative (see 662 (44)) and γ -smooth and ˆ β = ∥∇ f ( ˆ w ∗ ( S ); z ) -∇ f ( ˆ w ∗ ( S ′ ); z ) ∥ 2 . C is a positive constant. 663

From Lemma 3, we have ∥∇ f ( ˆ w ∗ ( S ); z ) -∇ f ( ˆ w ∗ ( S ′ ); z ) ∥ 2 ≤ 4 Mγ nµ . Since ∇ F S ( ˆ w ∗ ) = 0 , we 664 have ∥∇ F S ( ˆ w ∗ ) ∥ 2 = 0 , then we can derive that 665

666

E Proofs of PGD 667

Proof of Theorem 5. According to smoothness assumption and η = 1 /γ , we can derive that 668

According to above inequality and the assumptions that F S is µ -strongly convex, we can prove that 669

which implies that 670

According to the property for γ -smooth for F S and the property for µ -strongly convex for F S , we 671 have 672

which means that µ γ ≤ 1 . 673

Then If η t = 1 /γ , 0 ≤ 1 -µη t < 1 , taking over T iterations, we get 674

Combined (36), the smoothness of F S and the nonnegative property of f , it can be derive that 675

Furthermore, since F is µ -strongly convex, we have 676

So to bound F ( w T +1 ) -F ( w ∗ ) , we need to bound the term ∥∇ F ( w T +1 ) ∥ 2 2 . And there holds 677

From Lemma 1, if f is M -Lipschitz and γ -smooth and F S is µ -strongly convex, for any δ > 0 , when 678 n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ , there holds 679

where the last inequality follows from Lemma 4.1 of Srebro et al. [2010] when f is nonegative and 680 γ -smooth (see (44)) and β = ∥∇ f ( w T +1 ( S ); z ) -∇ f ( w T +1 ( S ′ ); z ) ∥ 2 . C is a positive constant. 681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

From Lemma 4, we have β = ∥∇ f ( w T +1 ( S ); z ) -∇ f ( w T +1 ( S ′ ); z ) ∥ 2 ≤ 2 Mγ nµ . Since ∥∇ F S ( w T +1 ) ∥ 2 = O ( (1 -µ γ ) T ) , then we can derive that

Let T ≍ log n , we have

The proof is complete.

F Proofs of SGD

We first introduce some necessary lemmata on the empirical risk.

Lemma 14 ([Lei and Tang, 2021]) . Let { w t } t be the sequence produced by SGD with η t ≤ 1 2 γ for all t ∈ N . Suppose Assumption 1 hold. Assume for all z , the function w ↦→ f ( w ; z ) is M -Lipschitz and γ -smooth. Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , there holds that

Lemma 15 ([Lei and Tang, 2021]) . Let { w t } t be the sequence produced by SGD with η t = 2 µ ( t + t 0 ) such that t 0 ≥ max { 4 γ µ , 1 } for all t ∈ N . Suppose Assumption 1 hold. Assume for all z , the function w ↦→ f ( w ; z ) is M -Lipschitz and γ -smooth and assume F S satisfies PL condition with parameter µ . Then, for any δ ∈ (0 , 1) , with probability at least 1 -δ , there holds that

Lemma 16 ([Lei and Tang, 2021]) . Let e be the base of the natural logarithm. There holds the following elementary inequalities.

Proof of Lemma 5. We have known that F S ( i ) ( w ) = 1 n ( f ( w ; z ′ i ) + ∑ j = i f ( w ; z j ) ) . We denote ˆ w ∗ ( S ( i ) ) be the ERM of F S ( i ) ( w ) and ˆ w ∗ S be the ERM of F S ( w ) . From Lemma 3, we know that

̸

Also, for w t , the convexity of f and the strongly-convex property implies that its closest optima point 702 of F S is ˆ w ∗ ( S ) (the global minimizer of F S is unique). Then, there holds that 703

Thus we have ∥ w t -ˆ w ∗ ( S ) ∥ 2 ≤ √ 2 ϵ opt ( w t ) µ . A similar relation holds between ˆ w ∗ ( S ( i ) ) and w i t . 704 Combined with the Lipschitz property of f we obtain that for ∀ z ∈ Z , there holds that 705

According to Lemma 15, for any dataset S , the optimization error ϵ opt ( w t ) is uniformly bounded by 706 the same upper bound. Therefore, we write ∥ ∥ ∇ f ( w t ; z ) -∇ f ( w i t ; z ) ∥ ∥ 2 ≤ 2 γ √ 2 ϵ opt ( w t ) µ + 4 Mγ nµ 707 here. 708

The proof is complete. 709

Now We begin to prove Lemma 6. 710

Proof of Lemma 6. If f is L -Lipschitz and γ -smooth and F S is µ -strongly convex. According to 711 Lemma 1, we know that for all w ∈ W and any δ ∈ (0 , 1) , with probability at least 1 -δ/ 2 , when 712 n > 16 γ 2 log 6 δ µ 2 , we have 713

where β t = ∥ ∥ ∇ f ( w t ; z ) -∇ f ( w i t ; z ) ∥ ∥ 2 and C is a positive constant. 714

715

where the second inequality holds from Cauchy-Bunyakovsky-Schwarz inequality and the second 716 inequality satisfies because F S is µ -strongly convex. 717

Plugging (41) into (40), with probability at least 1 -δ/ 2 , when n > 16 γ 2 log 6 δ µ 2 , we have 718

When η t = η 1 t -θ , θ ∈ (0 , 1) , with η 1 ≤ 1 2 β and Assumption 1, according to Lemma 14 and Lemma 719 16, we obtain the following inequality with probability at least 1 -δ/ 2 , 720

On the other hand, when f is nonegative and γ -smooth, from Lemma 4.1 of Srebro et al. [2010], we 721 have 722

Plugging (44), (43) into (42), with probability at least 1 -δ , we derive that 724

When θ < 1 / 2 , we set T ≍ n 2 θ and assume F ( w ∗ ) = O ( 1 n ) , then we obtain the following result 725 with probability at least 1 -δ 726

When θ = 1 / 2 , we set T ≍ n 4 and assume F ( w ∗ ) = O ( 1 n ) , then we obtain the following result with 727 probability at least 1 -δ 728

When θ > 1 / 2 , we set T ≍ n 2 1 -θ and assume F ( w ∗ ) = O ( 1 n ) , then we obtain the following result 729 with probability at least 1 -δ 730

The proof is complete. 731

732

which implies that 723

Proof of Theorem 7. Since F is µ -strongly convex, we have 733

So to bound F ( w T +1 ) -F ( w ∗ ) , we need to bound the term ∥∇ F ( w T +1 ) ∥ 2 2 . And there holds 734

From Lemma 1, if f is L -Lipschitz and γ -smooth and F S is µ -strongly convex, for all w ∈ W and 735 any δ > 0 , when n ≥ 16 γ 2 log 6 δ µ 2 , with probability at least 1 -δ/ 2 , there holds 736

where the last inequality follows from Lemma 4.1 of Srebro et al. [2010] when f is nonegative and 737 γ -smooth (see (44)) and C is a positive constant. Then we can derive that 738

where the second inequality holds from Cauchy-Bunyakovsky-Schwarz inequality and the second 740 inequality satisfies because F S is µ -strongly convex. 741

Plugging (49) into (48), with probability at least 1 -δ/ 2 , when , we have 742

According to the smoothness property of F S and Lemma 15, it can be derived that with propability at 743 least 1 -δ/ 2 744

Substituting (51), (50) into (46), we derive that 745

Further substituting (52) into (45) and choosing T ≍ n 2 , we finally obtain that when n , with 746 probability at least 1 -δ 747

748

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have checked that the abstract and introduction accurately reflect our contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We have clearly stated the required assumptions for each theorem and lemma, and the conditions for the assumptions to hold are also stated in the main text.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We have clearly stated the required assumptions for each theorem and lemma, and all proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper focuses on learning theory.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This paper focuses on learning theory and does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper focuses on learning theory and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper focuses on learning theory and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper focuses on learning theory and does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on learning theory and there is no societal impact of the work performed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper focuses on learning theory and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper focuses on learning theory and does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

1009

1010

1011

1012

1013

1014

1015

1016

1017

1018

1019

1020

1021

1022

1023

1024

1025

1026

1027

1028

1029

1030

1031

1032

1033

1034

1035

1036

1037

1038

1039

1040

1041

1042

1043

1044

1045

1046

1047

1048

1049

1050

1051

1052

1053

1054

1055

1056

1057

1058

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalization measured by gradients via a uniform gradient stability. For $\\beta$-uniformly stable algorithms, the paper gives generalization bounds of order $O(1/n+\\beta+\\sqrt{E_Z[\\|\\nabla f(A(S);Z)\\|_2^2]/n})$, which yields fast rates if $E_Z[\\|\\nabla f(A(S);Z)\\|_2^2$ is small. The paper then uses this generalization measured by gradients to derive generalization error bounds under a PL condition. Applications to empirical risk minimization, gradient descent and stochastic gradient descent are given for strongly convex, smooth and Lipschitz problems.\n\nSTRENGTHS:\nThe paper gives high-probability bounds for generalization gap on gradients, which include the gradient norm at the minimizer. This can imply fast rates in an interpolation setting. Under the PL condition, this gives fast rates of order $O(1/n^2)$.\n\nThe paper provides comprehensive applications to several algorithms such as empirical risk minimization, gradient descent and stochastic gradient.\n\nWEAKNESSES:\nThe high-probability analysis based on uniform stability follows largely from existing work. I do not see enough novelty in the analysis. It would be helpful if the authors can summarize the challenges in the analysis and their novelty. \n\nAs stated in the paper, Theorem 1 and Theorem 2 only improve the existing results by a constant factor. This improvement is not significant. \n\nAs stated in the paper, the generalization by gradients is mostly interesting for nonconvex problems. However, for the applications in Section 4, the paper considers strongly convex problems. Also the results require smoothness and Lipschitz continuity. These assumptions seem to be a bit strong.\n\nThe paper gives fast rates under the case $F(w^*)=O(1/n)$ in Section 4. Note that Section 4 considers strongly convex problems. Then, the objective function should be of the form $F(w)=G(w)+\\mu\\|w\\|^2$, where $G$ is related to loss. Then, if we require $F(w^*)=O(1/n)$, one needs $\\mu\\|w^*\\|^2=O(1/n)$. Suppose we assume $\\|w^*\\|=O(1)$. Then, this requires $\\mu=O(1/n)$. In this case, the generalization bound would be vacuous since $n\\mu=O(1)$.\n\nFor SGD, the computation cost seems to be high. For example, in Theorem 6, the paper requires $T=n^4$ while in Theorem 13 the paper requires $T=n^2$. This high computational cost may not be appealing for large-scale problems.\n\nIn the proof of Lemma 1, the paper uses $\\|\\nabla F(A(S))\\|_2\\geq \\mu\\|A(S)-w^*\\|$. This inequality does not generally hold under a PL condition. Indeed, Theorem 2 in Karimi et al 2016 require $\\|A(S)-w^*\\|$ to be replaced by the distance between $A(S)$ and the set of minimizers.\n\nQUESTIONS:\nThe paper contains various typos. Here are some examples:\n\nLemma 3: $\\|\\nabla f(w;z)\\le M\\|$\n\nBelow eq (25): $\\sqrt{a+b}=\\sqrt{a}+\\sqrt{b}$\n\nBelow Eq (28): Below eq (25): $\\sqrt{a+b}+\\sqrt{a}+\\sqrt{b}$\n\nIn Eq (27), it seems that there is a missing factor of $2$\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

comments: SUMMARY:
The work shows high probability excess risk bounds of $O(1/n^2)$ for several algorithms under strong convexity, smoothness, Lipschitz continuity and low noise assumptions using algorithmic stability.

STRENGTHS:
1. The results of the paper are interesting, showing a risk bound of $O(\frac{1}{n^2})$ using algorithmic stability.

2. The paper uses a novel technique, involving the stability of gradients to demonstrate excess risk bounds and presents applications of this technique in convex optimization.

WEAKNESSES:
1. The problem setup of the paper is not clearly detailed before the technical section, including the assumptions used for proving the results.

2. The presentation of results and related works is somewhat lacking. It would be beneficial if the authors summarized the results from previous work and compared them to the results in the current paper, including the set of assumptions made in each work.

QUESTIONS:
1. In the proof of the risk bound for ERM (line 664), why is $\nabla F_S(w^*) = 0$? This may not hold in constrained optimization. Alternatively, in unconstrained optimization, the results cannot hold since a strongly convex function cannot be Lipschitz.

2. The paper contains several typos in both the analysis and the main text (e.g., line 5, line 228).

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper achieves the high probability excess risk bounds $\\mathcal{O}(1/n^2)$ for empirical risk minimization, projected gradient descent and stochastic gradient descent under strong convexity, smoothness and Lipschitz continuity assumptions.\n\nSTRENGTHS:\nPlease see Summary.\n\nWEAKNESSES:\n1.The paragraph from line 52 to line 65 first analyzes the non-convex problems. Then, it suddenly mentions that this paper explores the stability of stochastic convex optimization algorithms with strongly convex losses in line 61. And, it doesn’t mention its non-convex analysis. It is so confusing. So, I suggest authors rewrite this paragraph to benefit readers’ understanding.\n\n2.The paragraph from line 70 to line 73 is unnecessary since we can not obtain any useful information.\n\n3.In Related Work, it is unnecessary to list the literature related to uniform convergence since it is not helpful for readers’ understanding of the contributions of this paper. It may be better that authors list the detailed literature related to high probability bound.\n\n4.Theorem 1 in this paper is not the sharpest p-moment bound for sums of vector-valued functions. Authors demonstrate their bound is indeed tighter than Theorem 1 of [1]. However, [2] also provided a bound (Theorem 1) that is likely tighter than Theorem 1 in this paper. Note that, [2] also used Marcinkiewicz-Zygmund’s inequality to prove their bound. Besides, the paragraph from line 131 to line 136 states “On the other hand, in Section 3.2, we will carefully construct vector-valued functions which satisfies all the assumptions in Theorem 1 and ensures M = 0 at the same time. Under this condition, we can eliminate the first term.”. This point is also considered in Theorem 1 of [2]. I think that authors just consider the improvement to [1], but omit other related work.\n\n[1]J. Fan and Y. Lei. High-probability generalization bounds for pointwise uniformly stable algorithms. Applied and Computational Harmonic Analysis, 70:101632, 2024.\n\n[2]X. Yuan, P. Li. Exponential generalization bounds with near-optimal rates for $L_q$-stable algorithms. ICLR, 2023.\n\n5.In Section 3.2, authors build some relationships between generalization error and stability parameter $\\beta$. Authors think these relationships are under non-convex, non-smooth, non-PL conditions. These bounds are not the final generalization bounds but the relationships. After giving the stability bounds, the generalization bounds are finally determined. However, in Section 4, authors provide the stability bounds under (strongly) convex and smooth conditions. Therefore, authors didn’t remove (strongly) convex and smooth conditions for generalization analysis.\n\n6.The symbol $M$ is repeatedly used in Theorem 1 and Theorem 2. Therefore, I suggest authors should carefully check their symbol settings.\n\n7.In Remark 4, authors compare their Theorem 3 with the bound in [3]. It is unfair since, as mentioned in the above 5., the bound in [3] is a final result but Theorem 3 is not.\n\n[3]Y. Xu and A. Zeevi. Towards optimal problem dependent generalization error bounds in statistical learning theory. Mathematics of Operations Research, 2024.\n\n8.In line 633, $F(A(S)) - F(A(S))$ is wrong.\n\n9.In line 633, Equation (31) should be an inequality.\n\n10.In line 152, authors state “In nonconvex problems, we can only find a local minimizer by optimization algorithms which may be far away from the global minimizer. Thus the convergence does not make much sense in function values.”. So, they use uniform stability in gradients. However, in Section 4, they provide some uniform stability bounds in gradients for strongly convex problems. It is a paradox.\n\n11.The form of the relationship in Theorem 3 is very normal. The method to obtain an excess risk bound $\\mathcal{O}(1/n^2)$ is very simple. I think other normal generalization results (like [1]) in gradients can derive the excess risk bound $\\mathcal{O}(1/n^2)$. The main contribution of this paper may be the simple method combining PL condition with some usual decompositions as shown in Proof of Remark 5. However, as mentioned in the above 10., the uniform stability in function values is more unreliable than the one in gradients under strongly convex condition.\n\nQUESTIONS:\n1.In line 23, $f$ denotes a loss function. However, what does $g$ denote? Definition 1 and Definition 4 involve this symbol.\n\n2.In line 54, authors state “This is why we can achieve dimension-free excess risk bounds of order $O(1/n^2)$.”. I don’t understand it since the sentence before it is not the reason of it.\n\n3.The paragraph from line 66 to line 69 mentions that “we obtain a tighter p-moment bound ...”. Where is this tighter p-moment bound used? This point is not stated in Abstract. From my perspective, it is valuable to be emphasized.\n\n4.Why do authors give Lemma 2? The population risk of gradients $||\\nabla F(A(S))||_2$ is not included in any result and not cited in any proof in this paper.\n\n5.In line 635, why is the first inequality not consistent with Lemma 1?\n\n6.Why is Theorem 3 better than Theorem 2 under the same conditions? It is better to emphasize the difference between these two theorems, rather than just state a sentence “we derive sharper generalization bound of gradients under same assumptions.”.\n\nLIMITATIONS:\nConsidering the 4. in Weaknesses, I suggest the author reconsider whether their result is the sharpest.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the standard statistical learning setting and derives $\\mathcal{O}(\\frac{1}{n ^ {2}})$ ($n$ denotes the number of samples) high-probability bounds for the excess risk $F(A(S)) - \\inf_{w} F(w)$ ($A$ denotes the algorithm and $S$ denotes the training set) of ERM, PGD, and SGD. The best-known bounds prior to this work were $\\mathcal{O}(\\frac{\\log n}{n})$ for ERM, PGD, that was derived using algorithmic stability, and $\\mathcal{O}(\\frac{1}{n ^ {2}})$  or ERM, SGD that was derived using uniform convergence. However, the latter demanded $n = \\Omega(d)$ samples, thereby introducing the an undesirable dependence on $d$. The current paper shows that it is possible to obtain $\\mathcal{O}(\\frac{1}{n ^ {2}})$ bounds for the algorithms (without any dependence on $d$) under the lens of stability.\n\nSTRENGTHS:\nThe considered problem is interesting and the contributions of the paper show that sharper bounds (at par with uniform convergence, albeit without any dependence on $d$) are possible for ERM, SGD, and PGD for strongly convex and smooth stochastic convex optimization. Below I explain the roadmap taken by the authors in doing so, which also sheds light on some of the other aspects of the paper.\n\n1) The authors first study the generalization gap via gradients, i.e. the quantity $\\||\\nabla F(A(S)) - \\nabla F_S(A(S)\\||$ for the statistical learning setting under the assumption that the function is Lipschitz and the algorithm is uniformly stable in gradients (Theorem 1 and 2). Under the nonconvex setting, the authors obtain dimension-independent bound for the generalization gap via gradients. This bound is subsequently studied under the assumption that the function is smooth and satisfies the Polyak-Lojaseiwicz (PL) condition. The obtained bound is a function of the gradient norm obtained at the end of optimization, i.e. $\\|| \\nabla F_S(A(S) \\||$. \n\n2) To obtain excess risk bound for the algorithms, the (i) authors show that the algorithms are uniformly stable in gradients; (ii) translate the excess risk bound to a bound on the gradient via the PL inequality (recall from the premise that these algorithms are analyzed in the strongly convex and smooth setting of stochastic convex optimization, therefore PL holds vacuously); (iii) use triangle inequality to relate the bound on the gradient norm to the generalization gap, and use the bound on the generalization gap via gradients (see 1 above).\n\nWEAKNESSES:\nI found several typos in the main theorems in section 4. For example, the stability equation in Lemma 4 should be written with respect to the output at the iteration instead of the output of the ERM.  Also, why is the reference in Theorems in section 4 to Theorem 3, instead of Lemma 1? From my understanding (explained in the Strengths above), Lemma 1 is a specific instantiation of Theorem 3 to smooth + PL functions, which is exactly the premise in Section 4. What exactly is $w$ in the bound $F(w) - F(w^\\star)$ in Theorem 5? I expect it to $w_{T + 1}$ (or $w_T$).\n\nI don't see the point of Marcinkiewicz-Zygmund’s inequality with improved constants. The whole paper is about the improved dependence with respect to $n$, so I don't see a point in improving the specific constants in the inequality. \n\nI need some more clarification in lines 204--207. The authors state that Klochkov and Zhivotovskiy obtained $\\mathcal{O}(\\frac{1}{n})$ style bounds for the excess risk. What's the assumption on $f$ considered by them? The authors mention that they can obtain $\\mathcal{O}(\\frac{1}{n ^ {2}})$ bounds with an extra PL and smoothness assumption. Is this something for which Klochkov and Zhivotovskiy, 2021 could only obtain a suboptimal $\\mathcal{O}(\\frac{1}{n})$ bound? Earlier, my interpretation was that this work obtained $\\frac{1}{n}$ bounds for ERM, and PGD for smooth and strongly convex stochastic convex optimization, but lines 204--207 made my understanding unclear. I want to make sure the authors are not invoking extra assumptions to get improved dependence.\n\nAlong similar lines as above, Lines 259--261 seem to be saying inconsistent things (is the assumption just smoothness, or strong convexity and smoothness). I would appreciate clarifications from the authors.\n\nMinor Typos: 1) In definition 1, $\\gamma$ and $\\mu$ should be strictly positive; (2) Line 182: $\\gamma$-smooth instead of $\\gamma$-smoothness.\n\nBased on the authors' responses, I would be happy to revise my assessment of the paper.\n\nQUESTIONS:\nSee the Weaknesses above.\n\nLIMITATIONS:\nYes, the authors have adequately addressed this.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}