SlowGuess/NeurIPS_2023_OCR huggingface dataset:
-  $\mathbb{E}[g_i(S,\xi) \mid S \setminus Z_i, \xi] = 0$  and  $|\mathbb{E}[g_i(S,\xi) \mid Z_i, \xi]| \leq M$ , almost surely;
-  $g_{i}(S,\xi)$  has the following  $L_{2}$ -bounded-difference property with respect to all variables in  $S$  except  $Z_{i}$ , i.e.,  $\forall j \neq i$ ,
$$
\sup  _ {S, Z _ {j} ^ {\prime}} \mathbb {E} _ {\xi} \left[ \left(g _ {i} (S, \xi) - g _ {i} (S ^ {(j)}, \xi)\right) ^ {2} \right] \leq \beta^ {2}.
$$
Then for all  $q \geq 2$ ,
$$
\left\| \mathbb {E} _ {\xi} \left[ \left| \sum_ {i = 1} ^ {N} g _ {i} (S, \xi) \right| \right] \right\| _ {q} \leq 3 M \sqrt {3 N q} + 3 8 N \lceil \log_ {2} N \rceil \beta q.
$$
Proof in sketch. The main idea is inspired by the sample-splitting arguments of Feldman and Vondrak (2019); Bousquet et al. (2020), with some new ingredients developed to handle the first-moment operator taken over the internal randomness of functions. Here we just highlight a fundamental difference, which arises from using a newly developed moment inequality (Lemma 9) for bounding the sums of conditionally independent randomized functions inside each individual data splits. Different from the version of Marcinkiewicz-Zygmund's inequality used in the original analysis of Bousquet et al. (2020), our new bound in Lemma 9 relies on some second-order (over the function's randomness) components which might be tightly bounded by the second-order concentration inequality in Proposition 1. A full proof is provided in Appendix A.3.
Remark 3. For sums of deterministic functions, our result in Proposition 2 reduces to the existing moment bound of Bousquet et al. (2020, Theorem 4) which is known to be near-tight up to logarithmic factors. We comment in passing that the tightness analysis of Bousquet et al. (2020, Proposition 9) for deterministic functions can be more or less straightforwardly extended to randomized functions.
**Remark 4.** The bound of Proposition 2 would still be valid when the bounded-loss condition  $|\mathbb{E}[g_i(S,\xi) \mid Z_i,\xi]| \leq M$  is relaxed to certain sub-Gaussian or sub-exponential stochastic versions.
2.4 Main result on generalization bound
Consequently from Proposition 2, we can now establish our main result on the generalization bound of  $L_{2}$ -uniformly stable randomized learning algorithms.
Theorem 1. Let  $A: \mathcal{Z}^N \times \mathcal{R} \mapsto \mathcal{W}$  be a randomized learning algorithm that has  $L_2$ -uniform stability with parameter  $\gamma_{L_2, N}$ . Assume that the loss function  $\ell$  is valued in  $[0, M]$ . Then for any  $\delta \in (0, 1)$ , the following bound holds with probability at least  $1 - \delta$  over the draw of  $S$ :
$$
\mathbb {E} _ {\xi} \left[ | R (A (S, \xi)) - R _ {S} (A (S, \xi)) | \right] \lesssim \gamma_ {L _ {2}, N} \log (N) \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Proof. See Appendix A.4 for a proof of this result.
Remark 5. The first-moment bound in Theorem 1 naturally implies the on-average bound in (4) with an identical rate of convergence, though the former is obtained under the relatively stronger notion of  $L_{2}$ -uniform stability. As we will see shortly that the  $L_{2}$ -uniform stability can indeed be fulfilled by the popularly applied SGD algorithm and thus Theorem 1 is of practical importance for showcasing sharper generalization performance of SGD. When  $A$  is deterministic, our bound reduces to the near-optimal (up to logarithmic factors on sample size and failure tail) generalization bound for uniformly stable algorithms (Bousquet et al., 2020).
Algorithm 1: Confidence-Boosting for Randomized Learning Algorithms
Input: Randomized learning algorithm  $A$ , data set  $S = \{Z_i\}_{i \in [N]}$ ,  $\mu \in (0,1)$  and  $K \in \mathbb{Z}^+$ .
Output:  $A(S, \xi_{k^*})$ .
Uniformly divide  $S$  into two disjoint subsets  $S_{1}$  and  $S_{2}$  with  $|S_1| = (1 - \mu)N$ ,  $|S_{2}| = \mu N$ .
for  $k = 1,2,\dots,K$  do
Estimate  $A(S_{1},\xi_{k})$  as an output of  $A$  over the subset  $S_{1}$  with random bit  $\xi_{k}$ .
Select the random bit  $k^{*}$  according to  $k^{*} = \arg \min_{k\in [K]}|R_{S_2}(A(S_1,\xi_k)) - R_{S_1}(A(S_1,\xi_k))|$ .
In view of the standard risk decomposition, the following excess risk tail bound can be readily obtained via applying Theorem 1 and Hoeffding's inequality:
$$
\mathbb {E} _ {\xi} \left[ R (A (S, \xi)) - R ^ {*} \right] \lesssim \Delta_ {\text {o p t}} + \gamma_ {\mathrm {L} _ {2}, N} \log (N) \log \left(\frac {1}{\delta}\right) + M \sqrt {\frac {\log (1 / \delta)}{N}}. \tag {13}
$$
Here recall that  $\Delta_{\mathrm{opt}} \coloneqq \mathbb{E}_{\xi} [R_S(A(S, \xi))] - \min_{w \in \mathcal{W}} R_S(w)$  is the sub-optimality of empirical risk minimization. Since the excess risk is by definition non-negative, the above bound can also be obtained under the weaker notion of on-average uniform stability (2) via applying (4). In this sense, the first-moment generalization error bound in Theorem 1 is substantially more challenging to derive than the excess risk bound. Additionally, under the generalized Bernstein condition (Koltchinskii, 2006), the risk bound (13) can be readily improved to (5) by directly applying the corresponding deviation optimal risk bound of Klochkov and Zhivotovsky (2021) to the on-average loss function  $\mathbb{E}_{\xi}[\ell(A(S, \xi); Z)]$  under on-average uniform stability condition.
3 Boosting the Confidence of Generalization
The confidence-boosting technique of Schapire (1990) is a classic meta approach that allows one to boost the dependence of a learning algorithm on the failure probability  $\delta$  from  $1 / \delta$  to  $\log(1 / \delta)$ , at a certain cost of computational complexity. In this section, we show an implication of our first-moment bound in Theorem 1 for achieving high-probability generalization jointly over the randomness of data and algorithm, inside a natural framework of confidence-boosting.
3.1 Confidence boosting via bagging
Given a randomized learning algorithm  $A$ , we propose to study a bagging based confidence-boosting procedure as outlined in Algorithm 1. In this meta procedure, we independently run the algorithm  $A$  for  $K$  times over  $S_{1}$ , a fraction of the training set, to obtain  $K$  different candidate solutions  $\{A(S_1,\xi_k)\}_{k\in [K]}$ . Then we evaluate the validation error of these candidate solutions over the holdout training subset  $S_{2}$ , and cherry pick  $A(S_{1},\xi_{k^{*}})$  that has the smallest gap between the training error and validation error, i.e.,  $k^{*} = \arg \min_{k\in [K]}\left|R_{S_2}(A(S_1,\xi_k)) - R_{S_1}(A(S_1,\xi_k))\right|$ . Particularly, consider that the internal randomness of  $A$  arises from random sampling with replacement of data points, such as SGD under with-replacement sampling. Then in this setting, the procedure can be regarded as a version of bagging (Breiman, 1996) with a greedy model ensemble scheme, which is invoked to the deterministic counterpart of  $A$  with fixed random bits (e.g., SGD with identity permutation) over the training subset  $S_{1}$ .
3.2 Jointly exponential bounds
The following theorem is our main result about the generalization error bound of the output  $A(S_{1}, \xi_{k^{*}})$  that holds with high probability over the entire training set  $S$  and the random seeds  $\{\xi_k\}_{k \in [K]}$ .
Theorem 2. Suppose that a randomized learning algorithm  $A: \mathcal{Z}^N \times \mathcal{R} \mapsto \mathcal{W}$  has  $L_2$ -uniform stability with parameter  $\gamma_{L_2, N}$ . Assume that the loss function  $\ell$  is valued in  $[0, M]$ . Then for any  $\delta \in (0, 1)$  and  $K \geq 2\log\left(\frac{2}{\delta}\right)$ , with probability at least  $1 - \delta$  over the randomness of  $S$  and  $\{\xi_k\}_{k \in [K]}$ , the output of Algorithm 1 satisfies
$$
\left| R \left(A \left(S _ {1}, \xi_ {k ^ {*}}\right)\right) - R _ {S} \left(A \left(S _ {1}, \xi_ {k ^ {*}}\right)\right) \right| \lesssim \gamma_ {L _ {2}, (1 - \mu) N} \log (N) \log \left(\frac {1}{\delta}\right) + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log \left(K / \delta\right)}{N}}.
$$
Algorithm 2:  $A_{\mathrm{SGD - w}}$ : SGD under With-Replacement Sampling
Input :Data set  $S = \{Z_{i}\}_{i\in [N]}$  , step-sizes  $\{\eta_t\}_{t\geq 1}$  , #iterations  $T$  , initialization  $w_0$  Output:  $\bar{w}_T = \frac{1}{T}\sum_{t\in [T]}w_t$  for  $t = 1,2,\dots,T$  do Uniformly randomly sample an index  $i_t\in [N]$  with replacement; Compute  $w_{t} = \Pi_{\mathcal{W}}(w_{t - 1} - \eta_t\nabla_w\ell (w_{t - 1};Z_{i_t}))$    
end
Proof in sketch. Based on Theorem 1, we first prove an intermediate result to show that the minimal generalization error of the  $K$  outputs satisfies  $\min_{k\in [K]}|R(A(S_1,\xi_k)) - R_{S_1}(A(S_1,\xi_k))|\lesssim \gamma_{\mathrm{L}_2,(1 - \mu)N}\log (N)\log \left(\frac{1}{\delta}\right) + \frac{M}{\sqrt{\mu(1 - \mu)}}\sqrt{\frac{\log(1 / \delta)}{N}}$  provided that  $K\gtrsim \log (\frac{1}{\delta})$ . Next we show that the used greedy model selection strategy guarantees that the selected  $A(S,\xi_{k^*})$  mimics the generalization behavior of that best performer among the  $K$  candidates, with a slightly expanded  $\log (K / \delta)$  factor representing the overhead of simultaneously bounding the generalization performance of  $K$  different candidate solutions over the holdout validation set. Finally the desired bound follows from the union probability argument. See Appendix B.1 for its full proof.
Remark 6. The bound in Theorem 2 holds with high probability jointly over the randomness of sample and algorithm. Different from the bound in (7) that requires high probability uniform stability, Theorem 2 is valid under a substantially milder notion of  $L_{2}$ -uniform stability, though at the cost of multiple running of algorithm for confidence boosting. Compared to the bound in (8) that requires certain conditional uniform stability over the random bits of algorithm, our bound has sharper dependence on the uniform stability parameter yet under a weaker notion of stability.
Remark 7. Regarding the scale of the factor  $1 / \sqrt{\mu(1 - \mu)}$  in the bound of Theorem 2, if setting  $\mu = 0.01$  (i.e.,  $99\%$  of  $S$  are used as  $S_{1}$  for training), then the factor is around 10.05.
Concerning the excess risk of Algorithm 1, we consider a slightly modified output  $A(S_{1},\xi_{k^{*}})$  such that  $k^{*} = \arg \min_{k\in [K]}R_{S_2}(A(S_1,\xi_k))$ . Then based on the in-expectation risk bound (13), we can derive the following excess risk bound under the conditions of Theorem 2 using similar arguments:
$$
R \left(A \left(S _ {1}, \xi_ {k ^ {*}}\right)\right) - R ^ {*} \lesssim \Delta_ {\text {o p t}} + \gamma_ {\mathrm {L} _ {2}, (1 - \mu) N} \log (N) \log \left(\frac {1}{\delta}\right) + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log (K / \delta)}{N}}. \tag {14}
$$
Again, the above risk bound is still valid under the weaker notion of on-average uniform stability (2).
4 Implications for SGD
This section is devoted to demonstrating the implications of Theorem 1 and Theorem 2 for the widely used SGD algorithm and its confidence-boosted versions as well. We focus on a variant of SGD under with-replacement sampling as outlined in Algorithm 2, which we call  $A_{\mathrm{SGD - w}}$ . In what follows, we substantially  $\xi = \{i_t\}_{t\in [T]}$  the sample path of  $A_{\mathrm{SGD - w}}$  over a given data set, and  $\{\xi_k\}_{k\in [K]}$  the  $K$  independent copies of  $\xi$  when implemented with bagging as shown in Algorithm 1. Our results can also be extended to the without-replacement variant of SGD and the corresponding results are provided in Appendix D for the sake of completeness.
4.1 Convex optimization with smooth loss
We first present the following lemma that establishes the  $L_{2}$ -uniform stability of  $A_{\mathrm{SGD - w}}$  with convex and smooth loss functions, such as logistic loss. See Appendix C.2 for its proof.
Lemma 1. Suppose that the loss function  $\ell(\cdot; \cdot)$  is convex,  $G$ -Lipschitz and  $L$ -smooth with respect to its first argument. Assume that  $\eta_t \leq 2 / L$  for all  $t \geq 1$ . Then  $A_{SGD - w}$  has  $L_2$ -uniform stability with parameter
$$
\gamma_ {L _ {2}, N} = 2 G ^ {2} \sqrt {1 0 \left(\frac {1}{N} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}\right)}.
$$
Given Lemma 1, we can apply Theorem 1 and Theorem 2 to immediately obtain the following generalization result for  $A_{\mathrm{SGD - w}}$  and its confidence-boosted version with smooth and convex losses.
Corollary 1. Suppose that the loss function  $\ell (\cdot ;\cdot)\in [0,M]$  is convex,  $G$  -Lipschitz and  $L$  -smooth with respect to its first argument. Then for any  $\delta \in (0,1)$ , it holds with probability at least  $1 - \delta$  over the randomness of  $S$  that  $\mathbb{E}_{\xi}\left[|R(A_{SGD - w}(S,\xi)) - R_S(A_{SGD - w}(S,\xi))|\right]\lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\frac {1}{N} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}} + M \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Moreover, consider Algorithm 1 specified to  $A_{SGD - w}$  with learning rate  $\eta_t \leq 2 / L$  and  $K \asymp \log \left(\frac{1}{\delta}\right)$ . Then with probability at least  $1 - \delta$  over the randomness of  $S$  and  $\{\xi_k\}_{k \in [K]}$ , it holds that  $|R(A_{SGD - w}(S_1, \xi_{k^*})) - R_S(A_{SGD - w}(S_1, \xi_{k^*}))| \lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\frac {1}{(1 - \mu) N} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{(1 - \mu) ^ {2} N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}} + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Remark 8. For the conventional choice of  $\eta_t = \frac{2}{L\sqrt{t}}$ , the high-probability (w.r.t. data) generalization bounds in Corollary 1 for SGD and its confidence boosted version are roughly of scale  $\mathcal{O}\bigl (\log (N)\log \bigl (\frac{1}{\delta}\bigr)\sqrt{\frac{\log(T)}{N} + \frac{\sqrt{T}}{N}}\bigr)$ , which matches the corresponding  $\mathcal{O}\bigl (\frac{\sqrt{T}}{N}\bigr)$  in-expectation bound of SGD with smooth and convex losses (Hardt et al., 2016).
Combining with the standard in-expectation optimization error bound of convex SGD (see, e.g., Shamir and Zhang, 2013), we can show the following excess risk bound of (modified) Algorithm 1 as a direct consequence of the generic bound (14) to  $A_{\mathrm{SGD - w}}$  with convex and smooth losses:
$$
\begin{array}{l} R (A _ {\tt S G D - w} (S _ {1}, \xi_ {k ^ {*}})) - R ^ {*} \lesssim G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\frac {1}{(1 - \mu) N} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{(1 - \mu) ^ {2} N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}} \\ + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log (1 / \delta)}{N}} + \frac {D ^ {2} (w _ {0} , W ^ {*}) + G ^ {2} \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2}}{\sum_ {t = 1} ^ {T} \eta_ {t}}, \\ \end{array}
$$
where  $W^{*} \coloneqq \operatorname{Argmin}_{w \in \mathcal{W}} R(w)$  and  $D(w, W^{*}) = \min_{w^{*} \in W^{*}} \| w - w^{*} \|$ . With learning rate  $\eta_t = \frac{2}{L\sqrt{t}}$ , the right hand side of the above roughly scales as  $\mathcal{O}\left(\sqrt{\log(N)}\log\left(\frac{1}{\delta}\right)\frac{\log(T)}{N} + \frac{\sqrt{T}}{N} + \frac{\log(T)}{\sqrt{T}}\right)$  which matches the prior high-probability excess risk bounds of SGD with convex losses (Harvey et al., 2019, Remark 3.7).
4.2 Convex optimization with non-smooth loss
Now we turn to study the case where the loss is convex but not necessarily smooth, such as the hinge loss and absolute loss. We first establish the following lemma about the  $L_{2}$ -uniform stability parameter of  $A_{\mathrm{SGD - w}}$  in the considered setting. See Appendix C.3 for its proof.
Lemma 2. Suppose that the loss function  $\ell (\cdot ;\cdot)$  is convex and  $G$  -Lipschitz with respect to its first argument. Then  $A_{SGD - w}$  has  $L_{2}$  -uniform stability with parameter
$$
\gamma_ {L _ {2}, N} = G ^ {2} \sqrt {4 0 \sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {3 2}{N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}}.
$$
With Lemma 2 in place, we can readily apply Theorem 1 and Theorem 2 to establish the following corollary about the generalization bounds of  $A_{\mathrm{SGD - w}}$  and its confidence-boosted version with convex and non-smooth loss functions.
Corollary 2. Suppose that the loss function  $\ell (\cdot ;\cdot)\in [0,M]$  is convex and  $G$  -Lipschitz with respect to its first argument. Then for any  $\delta \in (0,1)$ , it holds with probability at least  $1 - \delta$  over the randomness of  $S$  that  $\mathbb{E}_{\xi}\left[|R(A_{SGD - w}(S,\xi)) - R_S(A_{SGD - w}(S,\xi))|\right]\lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}} + M \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Moreover, consider Algorithm 1 specified to  $A_{SGD - w}$  with  $K \asymp \log \left(\frac{1}{\delta}\right)$ . Then with probability at least  $1 - \delta$  over  $S$  and  $\{\xi_k\}_{k \in [K]}$ , it holds that  $|R(A_{SGD - w}(S_1, \xi_{k^*})) - R_S(A_{SGD - w}(S_1, \xi_{k^*}))| \lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\sum_ {t = 1} ^ {T} \eta_ {t} ^ {2} + \frac {1}{(1 - \mu) ^ {2} N ^ {2}} \left(\sum_ {t = 1} ^ {T} \eta_ {t}\right) ^ {2}} + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Remark 9. For SGD with decaying learning rates  $\eta_t = \frac{1}{\sqrt{Nt}}$ , Corollary 2 admits high-probability generalization bounds of scale  $\mathcal{O}\big(\log (N)\log \big(\frac{1}{\delta}\big)\sqrt{\frac{\log(T)}{N} + \frac{T}{N^3}} +\sqrt{\frac{\log(1 / \delta)}{N}}\big)$ . With fixed rates  $\eta_t\equiv \eta$ , Corollary 2 yields deviation bounds of scale  $\mathcal{O}\big(\eta \log (N)\log \big(\frac{1}{\delta}\big)(\sqrt{T} +\frac{T}{N}) + \sqrt{\frac{\log(1 / \delta)}{N}}\big)$  which matches the near-optimal rate by Bassily et al. (2020, Theorem 3.3).
4.3 Non-convex optimization with smooth loss
We further study the performance of Algorithm 1 for  $A_{\mathrm{SGD - w}}$  with smooth but not necessarily convex loss functions, such as normalized sigmoid loss (Mason et al., 1999). The following lemma estimates the  $L_{2}$ -uniform stability of  $A_{\mathrm{SGD - w}}$  in the considered setting. See Appendix C.4 for its proof.
Lemma 3. Suppose that the loss function  $\ell (\cdot ;\cdot)$  is  $G$  -Lipschitz and  $L$  -smooth with respect to its first argument. Consider  $\eta_t\leq 1 / L$  . Then  $A_{SGD - w}$  has  $L_{2}$  -uniform stability with parameter
$$
\gamma_ {L _ {2}, N} = 2 G ^ {2} \sqrt {\frac {1}{N} \sum_ {t = 1} ^ {T} \exp \left(3 L \sum_ {\tau = t + 1} ^ {T} \eta_ {\tau}\right) u _ {t}},
$$
$$
u _ {t} := \eta_ {t} ^ {2} + 2 \eta_ {t} \sum_ {\tau = 1} ^ {t - 1} \exp \left(L \sum_ {i = \tau + 1} ^ {t - 1} \eta_ {i}\right) \eta_ {\tau}.
$$
Based on Lemma 3, we can invoke Theorem 1 and Theorem 2 to show the following generalization result for  $A_{\mathrm{SGD - w}}$  and its confidence-boosted version with non-convex and smooth loss functions.
Corollary 3. Suppose that the loss function  $\ell (\cdot ;\cdot)\in [0,M]$  is  $G$  -Lipschitz and  $L$  -smooth with respect to its first argument. Then for any  $\delta \in (0,1)$ , it holds with probability at least  $1 - \delta$  over the randomness of  $S$  that  $\mathbb{E}_{\xi}\left[\left|R(A_{\mathcal{SGD} - w}(S,\xi)) - R_S(A_{\mathcal{SGD} - w}(S,\xi))\right|\right]\lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\frac {1}{N} \sum_ {t = 1} ^ {T} \exp \left(L \sum_ {\tau = t + 1} ^ {T} \eta_ {\tau}\right) u _ {t}} + M \sqrt {\frac {\log (1 / \delta)}{N}},
$$
where  $u_{t} \coloneqq \eta_{t}^{2} + 2\eta_{t}\sum_{\tau = 1}^{t - 1}\exp (L\sum_{i = \tau +1}^{t - 1}\eta_{i})\eta_{\tau}$  for all  $t \geq 1$ . Moreover, consider Algorithm 1 specified to  $A_{SGD - w}$  with  $\eta_t \leq \frac{1}{L}$  and  $K \asymp \log (\frac{1}{\delta})$ . Then with probability at least  $1 - \delta$  over  $S$  and  $\{\xi_k\}_{k\in [K]}$ , it holds that  $|R(A_{SGD - w}(S_1,\xi_{k^*})) - R_S(A_{SGD - w}(S_1,\xi_{k^*}))| \lesssim$
$$
G ^ {2} \log (N) \log \left(\frac {1}{\delta}\right) \sqrt {\frac {1}{(1 - \mu) N} \sum_ {t = 1} ^ {T} \exp \left(L \sum_ {\tau = t + 1} ^ {T} \eta_ {\tau}\right) u _ {t}} + \frac {M}{\sqrt {\mu (1 - \mu)}} \sqrt {\frac {\log (1 / \delta)}{N}}.
$$
Remark 10. For the decaying learning rates  $\eta_t = \frac{1}{L\nu t}$  with arbitrary  $\nu \geq 1$ , the generalization bounds in Corollary 3 are of scale  $\mathcal{O}\big(\log (N)\log \big(\frac{1}{\delta}\big)\sqrt{\frac{T^{1 / \nu}\log(T)}{\nu N}} +\sqrt{\frac{\log(1 / \delta)}{N}}\big)$ . For the constant learning rates  $\eta_t\equiv \frac{1}{LT}$ , the bounds are of scale  $\mathcal{O}\big(\log (N)\log \big(\frac{1}{\delta}\big)\sqrt{\frac{\log(1 / \delta)}{N}}\big)$ .
In this paper, we have introduced a novel concept of  $L_{2}$ -uniform stability for randomized learning algorithms and proved a strong first-moment generalization bound that holds with high probability over training sample. Equipped with this result, we have further developed a bagging based confidence-boosting procedure and shown that it yields near-optimal generalization bounds with high confidence jointly over the randomness of sample and algorithm. The power of our theory has been demonstrated through an application to SGD with time-decaying learning rates, where sharper generalization bounds have been obtained for both convex and non-convex loss functions.
The authors sincerely thank the anonymous reviewers and area chairs for their insightful comments on this paper. The research was conducted while XTY worked for Nanjing University of Information Science and Technology and both authors worked for Baidu Cognitive Computing Lab. The work of XTY is also funded in part by the National Key Research and Development Program of China under Grant No. 2018AAA0100400, and in part by the Natural Science Foundation of China (NSFC) under Grant No. U21B2049, 61936005.