SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In contrast to the previous method, we adopt a different strategy for controlling changes in  $\theta$ . Our  $\beta_{\theta}$  is a random variable w.r.t.  $\theta$ . We assume a sub-exponential concentration behavior of  $\beta_{\theta}$  to control its deviation (Assumption 1). This allows us to control  $\beta_{\theta}$  without necessitating hyperparameter stability. Based on this assumption, our proof proceeds by bounding the MGF of the generalization gap. A key challenge is to handle the randomness of  $\beta_{\theta}$ . More precisely, for any temporarily fixed  $\theta$  it has been shown that  $G(S;\theta)$  is a mixture of sub-Gaussian and sub-exponential random variables when only considering the randomness of  $S$  [8]. Then a bound of  $\mathbb{E}_S[\exp (\lambda G(S;\theta))]$  requires an assumption  $\lambda \lesssim 1 / \beta_{\theta}$  (by Eq. (A.5) on sub-exponential random variables). This constraint makes it challenging since we need to choose an appropriate  $\lambda$  and control the associated  $\mathbb{E}_S\mathbb{E}_{\theta \sim \mathbb{P}}[\exp (\lambda G(S;\theta)]$  (the random constraint makes the selection of  $\lambda$  difficult). Our key idea to address this problem is to control the MGF of another function  $H:\mathcal{Z}^n\times \Theta \mapsto \mathbb{R}$  defined as follows
$$
H (S; \theta) = \left\{ \begin{array}{l l} R (A (S; \theta)) - R _ {S} (A (S; \theta)) & \text {i f} \theta \in \Omega_ {\delta}, \\ 0 & \text {o t h e r w i s e}, \end{array} \right.
$$
where Assumption 1 holds everywhere on  $\Omega_{\delta}$ , which is a subset of  $\Theta$  with probability measure at least  $1 - \delta$ . This definition of  $H$  has two benefits.
- First,  $H(S; \theta)$  is equal to  $G(S; \theta)$  with high probability. Therefore, a high-probability bound on  $H$  would imply a high-probability bound on  $G$ .
- Second,  $H$  is convenient to control. For any  $\theta \in \Theta \setminus \Omega_{\delta}$ , the MGF satisfies  $\mathbb{E}_S[\exp (\lambda H(S;\theta)]\leq 1$ . For any  $\theta \in \Omega_{\delta}$ , the term  $\mathbb{E}_S[\exp (\lambda H(S;\theta))]$  can be controlled under an assumption  $\lambda \lesssim 1 / \mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}]$  due to the definition of  $\Omega_{\delta}$  which relates  $\beta_{\theta}$  with its expectation. A key difference is that the random  $\beta_{\theta}$  is replaced by  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}]$  in the constraint of  $\lambda$ , which allows us to choose a deterministic  $\lambda$  uniformly for all  $\theta$ . A further expectation w.r.t.  $\theta$  would imply a bound on  $\mathbb{E}_{S;\theta}[\exp (\lambda H(S;\theta))]$ , which is needed in the PAC-Bayes analysis.
Finally, we manipulate this MGF bound in the PAC-Bayes framework by the variational formula (Lemma A.3), building upon the framework of [32].
3.3 Bounds in Expectation
Our previous analysis gives high-probability bounds. In this subsection, we give PAC-Bayes bounds in expectation. In this case, we no longer need an assumption on the concentration behavior of  $\beta_{\theta}$  around its expectation. The proof is given in Appendix A.3.
Theorem 3. Consider any  $\beta_{\theta}$ -uniformly stable algorithm  $A$  and  $M$ -bounded loss  $\ell$ . For any distribution  $\mathbb{Q}$ , we have
$$
\mathbb {E} _ {S} \mathbb {E} _ {\theta \sim \mathbb {Q}} [ R (A (S; \theta)) - R _ {S} (A (S; \theta)) ] \leq \left(\chi^ {2} (\mathbb {Q} \| \mathbb {P}) + 1\right) ^ {\frac {1}{2}} \left(\frac {2 M ^ {2}}{n} + 1 6 \mathbb {E} _ {\theta \sim \mathbb {P}} [ \beta_ {\theta} ^ {2} ]\right) ^ {\frac {1}{2}}. \tag {3.4}
$$
Remark 1. Under the same assumption, it was shown in Theorem 1 in [32] that
$$
\mathbb {E} _ {S} \mathbb {E} _ {\theta \sim \mathbb {Q}} [ R (A (S; \theta)) - R _ {S} (A (S; \theta)) ] \leq \left(\chi^ {2} (\mathbb {Q} \| \mathbb {P}) + 1\right) ^ {\frac {1}{2}} \left(\frac {2 M ^ {2}}{n} + 1 2 M \mathbb {E} _ {\theta \sim \mathbb {P}} [ \beta_ {\theta} ]\right) ^ {\frac {1}{2}}. \tag {3.5}
$$
Therefore, our analysis gives a bound of the order  $O\big(1 / \sqrt{n} +\big(\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}^{2}]\big)^{\frac{1}{2}}\big)$ , while Eq. (3.5) gives a bound  $O\big(1 / \sqrt{n} +\big(\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}]\big)^{\frac{1}{2}}\big)$ . It is known that  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}^{2}]$  can be much smaller than  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}]$ . For example, for SGD with  $t$  iterations and  $\mathbb{P}$  being the uniform distribution, it was shown  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}] = O(\eta t / n)$  and  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}^{2}] = O(\eta^{2}t / n)$  if the loss function is convex and smooth [27]. In the typical setting with  $\eta = O(1 / \sqrt{t})$  and  $t\succ n$  (in this setting SGD achieves optimal rates), we have  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}] = O(1 / \sqrt{n})$  and  $\mathbb{E}_{\theta \sim \mathbb{P}}[\beta_{\theta}^{2}] = O(1 / n)$ . In this case, our analysis gives PAC-Bayes bounds of the order  $O(1 / \sqrt{n})$ , while Eq. (3.5) gives PAC-Bayes bounds of the order  $O(1 / n^{\frac{1}{4}})$ . Therefore, our analysis implies much better PAC-Bayes bounds than that in [32].
We apply our general results to derive PAC-Bayes bounds for two optimization algorithms: Stochastic Gradient Descent (SGD) and Randomized Coordinate Descent (RCD). To this aim, we introduce some necessary definitions. Let  $\|\cdot\|_2$  denote the Euclidean norm.
Definition 3 (Lipschitz continuity). We say a loss function  $\ell(\cdot; z)$  is  $L$ -Lipschitz if for any  $\mathbf{w} \in \mathcal{W}$  and  $z \in \mathcal{Z}$ , we have  $\|\nabla \ell(\mathbf{w}; z)\|_2 \leq L$ . This implies that for any  $\mathbf{w}_1, \mathbf{w}_2 \in \mathcal{W}$ ,
$$
\left| \ell \left(\mathbf {w} _ {1}; z\right) - \ell \left(\mathbf {w} _ {2}; z\right) \right| \leq L \| \mathbf {w} _ {1} - \mathbf {w} _ {2} \| _ {2}. \tag {4.1}
$$
Definition 4 (Convexity). Let  $\kappa \geq 0$ . We say a loss function  $\ell(\cdot; z)$  is  $\kappa$ -strongly convex if for any  $\mathbf{w}_1, \mathbf{w}_2 \in \mathcal{W}$  and  $z \in \mathcal{Z}$ , we have
$$
\ell \left(\mathbf {w} _ {1}; z\right) \geq \ell \left(\mathbf {w} _ {2}; z\right) + \left\langle \nabla \ell \left(\mathbf {w} _ {2}; z\right), \mathbf {w} _ {1} - \mathbf {w} _ {2} \right\rangle + \frac {\kappa}{2} \| \mathbf {w} _ {1} - \mathbf {w} _ {2} \| _ {2} ^ {2}. \tag {4.2}
$$
We say the loss function  $\ell$  is convex if the above inequality holds with  $\kappa = 0$ .
Definition 5 (Smoothness). Let  $\alpha \geq 0$ . We say a loss function  $\ell(\cdot; z)$  is  $\alpha$ -smooth if for any  $\mathbf{w}_1, \mathbf{w}_2 \in \mathcal{W}$  and  $z \in \mathcal{Z}$ , we have
$$
\left\| \nabla \ell \left(\mathbf {w} _ {1}; z\right) - \nabla \ell \left(\mathbf {w} _ {2}; z\right) \right\| _ {2} \leq \alpha \| \mathbf {w} _ {1} - \mathbf {w} _ {2} \| _ {2}. \tag {4.3}
$$
For the applications, we only need to verify the sub-exponential stability of the algorithm w.r.t. the prior sampling  $\mathbb{P}$ , which is often chosen to be simple such as the uniform distribution.
4.1 Applications to Stochastic Gradient Descent
SGD is one of the most popular algorithms to solve optimization problems in machine learning due to its simplicity and efficiency. The basic idea is to build a stochastic gradient based on a randomly selected example, which is used to update iterates. Here we consider SGD with a general sampling scheme, where the random index follows from a general distribution. This general SGD has already been considered in the literature to improve the efficiency of SGD with uniform sampling, including importance sampling [66] and Markov chain sampling [54, 58, 63].
Definition 6 (SGD with general sampling). Let  $\mathbf{w}_1$  be an initial point. Let  $\mathbb{P}$  be a probability measure over  $[n]^T$  and  $S = \{z_1, \dots, z_n\}$  be a training dataset. Let  $(i_1, \dots, i_T)$  be drawn according to  $\mathbb{P}$ . At the  $t$ -th iteration, SGD with sampling scheme  $\mathbb{P}$  updates the model by
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} - \eta_ {t} \nabla \ell \left(\mathbf {w} _ {t}; z _ {i _ {t}}\right), \tag {4.4}
$$
where  $\{\eta_t\}$  is a positive step-size sequence. If  $\mathbb{P}$  is the uniform distribution, then we call it SGD with uniform sampling (SGDU).
Now we apply Theorem 1 to develop PAC-Bayes bounds for SGD applied to convex problems, covering both smooth and non-smooth cases. We will show that SGD enjoy sub-exponential stability.
4.1.1 Smooth case
In the following lemma to be proved in Appendix II.1.1, we give stability bounds for SGDU and show it satisfies Assumption 1. Recall that the indicator function  $\mathbb{I}[\cdot ]$  is defined in Definition 2.
Lemma 4 (Stability bound). Let  $S$  and  $S'$  be neighboring datasets. Suppose for all  $z \in \mathcal{Z}$  the loss function is convex,  $\alpha$ -smooth and  $L$ -Lipschitz. Let  $\{\mathbf{w}_t\}, \{\mathbf{w}_t'\}$  be the sequence produced by SGDU on  $S$  and  $S'$  respectively with  $\eta_t \leq 2 / \alpha$ . Then SGDU with  $t$  iterations and the hyperparameter  $\theta$  is  $\beta_{\theta}$ -uniformly stable with
$$
\beta_ {\theta} = 2 L ^ {2} \max  _ {k \in [ n ]} \sum_ {j = 1} ^ {t} \eta_ {j} \mathbb {I} [ i _ {j} = k ].
$$
If  $\eta_t = \eta$ , then for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  we have
$$
\beta_ {\theta} \leq \mathbb {E} _ {\theta \sim \mathbb {P}} [ \beta_ {\theta} ] + 4 L ^ {2} \eta (1 + (t / n) ^ {\frac {1}{2}}) \log (1 / \delta).
$$
That is, Assumption 1 holds with  $c = 4L^2\eta \big(1 + (t / n)^{\frac{1}{2}}\big)$  w.r.t.  $\mathbb{P}$ .
We can combine the above lemma with Theorem 1 to obtain PAC-Bayes bounds for SGD, whose proof is given in Appendix II.1.1. An interesting property is that generalization bounds for SGD with general sampling can be derived based on the stability analysis for SGD with the uniform sampling. We always let  $\mathbb{P}$  denote a uniform prior on  $\Theta$ .
Corollary 5 (Generalization bound). Assume  $\ell$  is  $M$ -bounded,  $L$ -Lipschitz, convex and  $\alpha$ -smooth. For uniform distribution  $\mathbb{P}$ , every  $n \in \mathbb{N}^+$  and  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over draws of a data set,  $S \sim \mathcal{D}^n$ , for all posterior sampling distribution  $\mathbb{Q}$  on  $[n]^T$ , SGD with  $\eta \leq 2 / \alpha$  satisfies
$$
\begin{array}{l} \mathbb {E} _ {\theta \sim \mathbb {Q}} \left[ R (A (S; \theta)) - R _ {S} (A (S; \theta)) \right] \lesssim \\ \left(D _ {\mathrm {K L}} (\mathbb {Q} \| \mathbb {P}) + \log (1 / \delta)\right) \max  \left\{L ^ {2} \eta (T / n + (1 + (T / n) ^ {\frac {1}{2}}) \log (n)) \log n, \frac {M}{\sqrt {n}} \right\}. \\ \end{array}
$$
Based on [20, 27], which studied the trade-off between optimization and stability, the recommended choices of parameters are  $T \asymp n$  and  $\eta \asymp 1 / \sqrt{T}$  to get a SGD iterate with good generalization behavior. In this setting, the above corollary implies a PAC-Bayes bound  $\widetilde{O}(1 / \sqrt{n})$ .
London [32] gave PAC-Bayes bounds for SGD under strong convexity and smoothness assumptions.
Corollary 6 (Corollary 1 in [32]). Suppose  $\ell$  is  $M$ -bounded. Let the objective function be  $\kappa$ -strongly convex,  $L$ -Lipschitz and  $\alpha$ -smooth. Then, for uniform distribution  $\mathbb{P}$  and any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over draws of a data set,  $S \sim \mathcal{D}^n$ ,  $SGD$  with  $\eta_t = (\kappa t + \alpha)^{-1}$  and any posterior sampling distribution  $\mathbb{Q}$  on  $[n]^T$  satisfies
$$
\mathbb {E} _ {\theta \sim \mathbb {Q}} \big [ R (A (S; \theta)) - R _ {S} (A (S; \theta)) \big ] \lesssim \sqrt {\Big (D _ {\mathrm {K L}} (\mathbb {Q} \| \mathbb {P}) + \log (1 / \delta) \Big) \Big (\frac {(M + L ^ {2} / \kappa) ^ {2}}{n} + \frac {L ^ {2}}{\kappa^ {2} T} \Big)}.
$$
Remark 2. We now compare Corollary 5 and Corollary 6. First, Corollary 6 requires a strong convexity assumption, which is removed in our analysis. Second, our analysis implies PAC-Bayes bounds of the order  $\widetilde{O}(1/\sqrt{n})$ , while Corollary 6 implies bounds of order  $O(1 / (\sqrt{n}\kappa))$ . The strong convexity parameter  $\kappa$  is often very small in both theoretical and empirical analysis. For example, the existing generalization analysis of regularization schemes suggests  $\kappa = O(n^{-\frac{1}{2}})$  to get an optimal bound (Section 3 in [55]), for which Corollary 6 implies a vacuous bound. By contrast, our bound  $\widetilde{O}(1/\sqrt{n})$  is optimal up to a log factor.
Remark 3. Our uniform stability (Definition 1) is slightly different from the uniform stability  $\beta_{\mathcal{Z}} := \sup_{S \sim S'} \sup_{z} \left| \mathbb{E}_{\theta \sim \mathbb{P}}[\ell(A(S; \theta), z) - \ell(A(S'; \theta), z)] \right|$  in [32] in the sense of taking expectation in different places. The expectation of  $\theta \sim \mathbb{P}$  is taken outside sup in our case and inside sup in  $\beta_{\mathcal{Z}}$  [32]. However, we often have similar upper bounds for  $\mathbb{E}[\beta_{\theta}]$  and  $\beta_{\mathcal{Z}}$ . Consider SGD for smooth, Lipschitz and convex problems as an example. It is shown in [20] that  $\beta_{\mathcal{Z}} \leq 2L^2 \sum_{k=1}^{t} \eta_k / n$ , while we can show  $\mathbb{E}[\beta_{\theta}] \lesssim L^2 \log n \sum_{k=1}^{t} \eta_k / n$  (the proof of Lemma 4). These two upper bounds are the same order up to a logarithmic factor. Furthermore, lower bounds were established in [65] (Theorem 1) where  $\beta_{\mathcal{Z}} \geq \frac{L}{2} \sum_{k=1}^{t} \eta_k / n$ , which match the existing upper bounds up to a constant factor. This shows that  $\beta_{\mathcal{Z}}$  and  $\mathbb{E}[\beta_{\theta}]$  are of similar order.
4.1.2 Non-smooth case
The following lemma shows that SGDU applied to non-smooth problems enjoys the sub-exponential stability. The proof follows the analysis in Section 4.2 in [29] and is given in Appendix II.1.2.
Lemma 7 (Stability bound). Let  $S$  and  $S'$  be neighboring datasets. Suppose for all  $z \in \mathcal{Z}$  the loss function is convex and  $L$ -Lipschitz. Let  $\{\mathbf{w}_t\}, \{\mathbf{w}_t'\}$  be the sequence produced by SGDU on  $S$  and  $S'$  respectively with fixed step sizes. Then SGDU with  $t$  iterations and the hyperparameter  $\theta$  is  $\beta_{\theta}$ -uniformly stable with  $\beta_{\theta} = 2\sqrt{e} L^{2}\eta\left(\sqrt{t} + \max_{k \in [n]} \sum_{j=1}^{t} \mathbb{I}[i_j = k]\right)$ . For any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  we have
$$
\beta_ {\theta} \leq \mathbb {E} _ {\theta \sim \mathbb {P}} [ \beta_ {\theta} ] + 4 \sqrt {e} L ^ {2} \eta (1 + (t / n) ^ {\frac {1}{2}}) \log (1 / \delta).
$$
That is, Assumption 1 holds with  $c = 4\sqrt{e} L^2 \eta \left(1 + (t / n)^{\frac{1}{2}}\right)$  w.r.t.  $\mathbb{P}$ .
Based on the above lemma, we derive the following corollary for the PAC-Bayes bounds of SGD in non-smooth problems.
Corollary 8 (Generalization bound). Assume  $\ell$  is  $M$ -bounded,  $L$ -Lipschitz and convex. For any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over draws of a data set,  $S \sim \mathcal{D}^n$ , for all posterior sampling distribution  $\mathbb{Q}$  on  $[n]^T$ , SGD with  $T$  iterations and  $\eta_t = \eta$  satisfies
$$
\begin{array}{l} \mathbb {E} _ {\theta \sim \mathbb {Q}} \left[ R (A (S; \theta)) - R _ {S} (A (S; \theta)) \right] \lesssim \\ \left(D _ {\mathrm {K L}} (\mathbb {Q} \| \mathbb {P}) + \log \frac {1}{\delta}\right) \max  \left\{L ^ {2} \eta (\sqrt {T} + T / n + (1 + (T / n) ^ {\frac {1}{2}}) \log n) \log n, \frac {M}{\sqrt {n}} \right\}. \\ \end{array}
$$
Remark 4. If we choose  $\eta \asymp T^{-\frac{3}{4}}$  and  $T \asymp n^2$ , then Corollary 8 gives the PAC-Bayes bounds of order  $\widetilde{O}(1/\sqrt{n})$ . This choice of parameters was suggested in Theorem 7 in [27]. This gives the  $O(1/\sqrt{n})$  optimization bounds to get optimal trade-off between stability and optimization for non-smooth problems. The analysis in [32] cannot imply PAC-Bayes bounds for non-smooth problems.
4.2 Applications to Randomized Coordinate Descent
In this subsection, we consider RCD; this has not been studied in the PAC-Bayesian literature. RCD is an efficient optimization algorithm that randomly chooses a coordinate to update at each iteration [37]. Here we consider RCD with general sampling scheme, i.e. the coordinate to update follows a general distribution. This scheme has been studied before in the optimization context [1, 66].
Definition 7 (RCD with general sampling). Let  $\mathbf{w}_1$  be an initial point. Let  $\mathbb{P}$  be a probability measure over  $[d]^T$  and  $S = \{z_1, \dots, z_n\}$  be a training dataset. Let  $(i_1, \dots, i_T)$  be drawn according to  $\mathbb{P}$ . At the  $t$ -th iteration, RCD with sampling scheme  $\mathbb{P}$  updates the model by
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} - \eta_ {t} \nabla_ {i _ {t}} R _ {S} (\mathbf {w} _ {t}) \mathbf {e} _ {i _ {t}}, \tag {4.5}
$$
where  $\{\eta_t\}$  is a step-size sequence,  $\mathbf{e}_i$  is the  $i$ -th coordinate vector in  $\mathbb{R}^d$ , and  $\nabla_{i}g$  is the derivative of  $g$  w.r.t. the  $i$ -th coordinate. If  $\mathbb{P}$  is the uniform distribution, then we call it RCD with uniform sampling (RCDU).
Before giving the generalization bound for RCD, we first introduce coordinate-wise smoothness.
Definition 8 ([37]). A differentiable function  $g: \mathcal{W} \to \mathbb{R}$  has coordinate-wise Lipschitz continuous gradients with parameter  $\hat{\alpha} > 0$ , if for all  $\lambda \in \mathbb{R}$ ,  $\mathbf{w} \in \mathcal{W}$ ,  $i \in [d]$ ,
$$
g (\mathbf {w} + \lambda \mathbf {e} _ {i}) \leq g (\mathbf {w}) + \lambda \nabla_ {i} g (\mathbf {w}) + \hat {\alpha} \lambda^ {2} / 2.
$$
In Lemma 9, to be proved in Appendix B.2, we develop stability bounds for RCDU with convex and smooth loss functions. In particular, we show the stability follows a sub-exponential distribution.
Lemma 9 (Stability bound). Let  $S$  and  $S'$  be neighboring datasets. Suppose for all  $z \in \mathcal{Z}$  the loss function  $\ell$  is convex,  $\alpha$ -smooth,  $L$ -Lipschitz and has coordinate-wise Lipschitz continuous gradients with parameter  $\hat{\alpha} \geq 0$ . Let  $\{\mathbf{w}_t\}, \{\mathbf{w}_t'\}$  be the sequence produced by RCDU on  $S$  and  $S'$  respectively with  $\eta_t \leq 2/\hat{\alpha}$ . Then RCD with  $t$  iterations is  $\beta_\theta$ -uniformly stable with
$$
\beta_ {\theta} = \frac {L}{n} \max  _ {k \in [ n ]} \sum_ {j = 1} ^ {t} \eta_ {j} | \nabla_ {i _ {j}} \ell (\mathbf {w} _ {j}; z _ {k}) - \nabla_ {i _ {j}} \ell (\mathbf {w} _ {j}; z _ {k} ^ {\prime}) |, \tag {4.6}
$$
where  $\| \nabla \ell (\mathbf{w};z)\| _1\leq L_1$ . Furthermore, if  $\eta_t = \eta$ , then for any  $\delta \in (0,1)$  the following inequality holds with probability at least  $1 - \delta$  over  $\theta \sim \mathbb{P}$  (the uniform distribution over  $\{(i_1,\dots ,i_t):i_j\in [d]\}$ )
$$
\beta_ {\theta} \leq \frac {2 L _ {1} L \eta t}{n d} + \frac {\eta L ^ {2} \log (1 / \delta)}{n} \left(\frac {8}{3} + \sqrt {\frac {3 2 t}{d}}\right).
$$
Remark 5. Stability bounds of the order  $O(\eta t / (nd))$  were developed for RCD in [62] (Theorem 2). Their stability bounds hold in expectation. In contrast, we develop high-probability stability bounds of the order  $O\left(\frac{\eta t}{nd} + \frac{\eta}{n}\right)$ .
We plug the above bounds into Theorem 1, and derive the following PAC-Bayes bounds for RCD.
Corollary 10 (Generalization bound). Let the assumptions in Corollary 5 hold. We further assume that the gradient is coordinate-wise Lipschitz continuous. When  $\mathbb{P}$  is the uniform distribution, for any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$  over draws of  $S \sim \mathcal{D}^n$ , for all posterior sampling distributions  $\mathbb{Q}$  on  $[d]^T$ , RCD with the hyperparameter  $\theta$  and fixed step sizes  $\eta \leq 2 / \hat{\alpha}$  satisfies
$$
\mathbb {E} _ {\theta \sim \mathbb {Q}} \big [ R (A (S; \theta)) - R _ {S} (A (S; \theta)) \big ] \lesssim \Big (D _ {\mathrm {K L}} (\mathbb {Q} \| \mathbb {P}) + \log (1 / \delta) \Big) \max  \Big \{\frac {L L _ {1} \eta T}{n d} \log n, \frac {M}{\sqrt {n}} \Big \}.
$$
According to the above corollary, we can derive PAC-Bayes bounds of the order  $\widetilde{O}(1/\sqrt{n})$  if  $T = O(d\sqrt{n})$  and  $\eta = O(1)$ . These choices of parameters were suggested in Theorem 7 in [62] to balance optimization and stability for RCD.
Related work on stability The analysis of the generalization error through algorithmic stability is based on the landmark work of [7]. Generalization bounds via stability are algorithm-specific and have been applied to regularization algorithms, such as SVM regression and classification [7, 11, 16, 52]. Pioneering work on stability analysis of SGD was introduced in [20], which motivated much subsequent stability analyses of randomized iterative algorithms [10, 24, 25, 31, 38, 45]. The smoothness assumption in [20] was removed in more recent stability analyses [3, 27] of SGD. Stability bounds showing the benefit of low training errors on generalization were also developed for convex [27, 39, 50], nonconvex [28] and overparameterized models [12, 47, 59]. In recent works, stronger high-probability bounds via uniform stability have been developed. Some breakthroughs have narrowed down the difference between the risk and the empirical risk, leading to faster convergence rates with high probability [8, 17, 18].
Related work on PAC-Bayes bounds The PAC-Bayes theory of generalization dates back to works by [53] and [34] and further improved by [9, 26] and others. PAC-Bayes bounds are upper bounds on the generalization error of randomized learning algorithms, given in terms of data-dependent quantities that we can compute: the empirical error, and a quantity to measure the divergence between the PAC-Bayesian prior and posterior distributions, such as Kullback-Leibler (KL) divergence or the Rényi divergence [4]. The framework was later extended to allow learning the prior from the data, resulting in tighter bounds [2, 14, 15, 35, 42-44, 48]. The sensitivity of learning algorithms to small perturbations in the weights can be analyzed through the properties of a distribution of predictors, which may lead to regularity and improve the generalization bounds [5]. By evaluating the algorithmic stability on all possible outputs, stability of learning algorithms and PAC-Bayes bounds can be combined [32, 33, 35, 36, 40, 48, 57] and applied to randomized learning algorithms such as SGD and SGLD [30, 32, 35, 36]. For PAC-Bayes bounds, usually, the randomness is induced by a distribution on the parameters of a model. In [32], the authors isolated the randomness to view the randomized learning algorithm as deterministic, with hyperparameters following the distribution instead.
Under an assumption of a sub-exponential stability parameter, we derive sharper stability-based PAC-Bayes bounds for randomized learning algorithms by utilizing a moment bound. We show that the sub-exponential stability assumption holds for SGD and RCD, for which we develop PAC-Bayes bounds as corollaries. Our results remove the need for the requirements of strong convexity, hyperparameter stability, and even smoothness in previous results.
Limitations. Future work of interest includes exploring other optimization methods that meet the sub-exponential assumption. It would also be interesting to study the quality of bounds obtainable when placing the PAC-Bayes prior on the parameters of a model (as in the classic approach) versus the hyperparameters of the optimiser of the model (as in this work). Another interesting research direction is to further improve the bound by including additional assumptions. As noted in [23], for the deterministic case, the bounds can be  $\sqrt{n}$ -times faster under a Bernstein condition between expectation and variance of loss functions.