SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The second term on the right-hand side of Eq. (11) represents the stability of the gradient with respect to the randomness of the training dataset  $S, S' \sim \mu^n$ , which leads to  $V_{\nabla}$  in Lemma 1 under the SGLD setting. Hereafter, we define  $\mathbb{E}\| \nabla F(W_t, S) - \nabla F(W_t, S')\|^2$  as  $\widetilde{V}_{\nabla t}$ .
By introducing  $\nabla \log \pi (w)$  into  $\mathbb{E}\| \nabla \log \rho_t - \nabla \log \gamma_t\| ^2$  in Eq. (11), we obtain
$$
\begin{array}{l} \frac {\partial \mathrm {K L} (\rho_ {t} | \gamma_ {t})}{\partial t} \leq - \frac {1}{4 \beta} \mathbb {E} \| \nabla \log \rho_ {t} - \nabla \log \pi \| ^ {2} + \frac {1}{2 \beta} \mathbb {E} \| \nabla \log \pi \| ^ {2} + \frac {1}{\beta} \mathbb {E} \nabla \log \rho_ {t} \cdot \nabla \log \gamma_ {t} + \frac {\beta}{2} \widetilde {V} _ {\nabla_ {t}} \\ \leq - \frac {1}{4 \beta c _ {\mathrm {L S}}} \mathrm {K L} (\rho_ {t} | \pi) + \frac {1}{2 \beta} \Omega (\rho_ {t}, \gamma_ {t}, \pi) + \frac {\beta}{2} \widetilde {V} _ {\nabla_ {t}} \\ \leq - \frac {1}{4 \beta c _ {\mathrm {L S}}} \left(\mathrm {K L} \left(\rho_ {t} \mid \gamma_ {t}\right) + \mathbb {E} \log \frac {\gamma_ {t}}{\pi}\right) + \frac {1}{2 \beta} \Omega \left(\rho_ {t}, \gamma_ {t}, \pi\right) + \frac {\beta}{2} \widetilde {V} _ {\nabla_ {t}}, \tag {12} \\ \end{array}
$$
where the first inequality is from the fact that  $-x^{2} \leq -\| x - y\|^{2} / 2 + y^{2}$  for  $x, y \in \mathbb{R}^{d}$  and the second one is from the LSI. We introduced  $\Omega(\rho_{t}, \gamma_{t}, \pi) \coloneqq \mathbb{E}_{\rho_{t}}\|\nabla \log \pi\|^{2} + 2\mathbb{E}_{\rho_{t}}\nabla \log \rho_{t} \cdot \nabla \log \gamma_{t}$  to simplify the notation.
By integrating  $e^{\frac{t}{4\beta c_{\mathrm{LS}}}} \frac{\partial \mathrm{KL}(\rho_t | \gamma_t)}{\partial t}$  in Eq. (12) over  $t \in [0, \eta]$  and rearranging it, we obtain
$$
\begin{array}{l} \mathrm {K L} (\rho_ {\eta} | \gamma_ {\eta}) \leq e ^ {\frac {- \eta}{4 \beta c _ {\mathrm {L S}}}} \mathrm {K L} (\rho_ {0} | \gamma_ {0}) \\ + \int_ {0} ^ {\eta} e ^ {\frac {- (\eta - t)}{4 \beta c _ {\mathrm {L S}}}} \left(\frac {\beta}{2} \widetilde {V} _ {\nabla_ {t}} - \frac {1}{4 \beta c _ {\mathrm {L S}}} \mathbb {E} \log \frac {\gamma_ {t}}{\pi} + \frac {1}{2 \beta} \Omega \left(\rho_ {t}, \gamma_ {t}, \pi\right)\right) d t. \tag {13} \\ \end{array}
$$
In Appendix F, we show that the terms related to  $\pi$  in Eq. (13) can be bounded by using the techniques of Raginsky et al. [29] and Vempala and Wibisono [32].
We next derive an upper bound for the following terms in Eq. (13):  $\mathbb{E}[\log \frac{\gamma_t}{\pi}]$  and  $\Omega (\rho_t,\gamma_t,\pi)$  by using the parametrix method for the FP equation [12, 27], which allows us to expand the FP equation's solution via the heat kernel. On the basis of this expansion, we can upper bound Eq. (13) as
$$
\int_ {0} ^ {\eta} e ^ {- \frac {(\eta - t)}{4 \beta c _ {\mathrm {L S}}}} \left(- \frac {1}{4 \beta c _ {\mathrm {L S}}} \mathbb {E} \log \frac {\gamma_ {t}}{\pi} + \frac {1}{2 \beta} \Omega \left(\rho_ {t}, \gamma_ {t}, \pi\right)\right) \mathrm {d} t \leq \mathcal {O} (\eta). \tag {14}
$$
By combining Eq. (14) with Eq. (13), we obtain the continuous version of Lemma 1.
The same procedure can be used for the SGLD setup. The difference from the continuous Langevin diffusion case is that the discretization errors and the effects of using a stochastic gradient are taken into account, resulting in the appearance of an additional constant  ${}^{4}$  in the above bounds (see Appendix F for details).
3.4 Additional discussion on our bound in terms of stability
We conclude this section by presenting further discussion on our bound in terms of stability with a concrete example.
As shown in Eq. (7), the information-theoretic generalization bound is closely related to the stability in KL divergence under the different training datasets. However, our bound in Theorem 4 incorporates the constant term  $c_{2}$ , which is irrelevant to stability, alongside the stability term  $V_{\nabla}$ . If we can avoid the occurrence of  $c_{2}$ , the resulting upper bound of  $\mathrm{KL}(P_{W_t|S}|P_{W_t|S'})$  would be dominantly expressed by  $V_{\nabla}$ , and as a result, we may obtain a bound where the relationship between generalization and stability is more directly represented.
The problematic constant term  $c_{2}$  arises from  $c_{3}\eta$  in Lemma 1 analyzing the time evolution of stability in KL divergence. Specifically, the term  $c_{3}\eta$  is the byproduct of treating the general dissipative function using LSI. Actually, it is possible to avoid the problematic constant term  $c_{3}\eta$  and derive bounds that are evaluated solely on the basis of stability-related metrics in specific examples, such as strongly convex or bounded (non-convex) losses with  $l_{2}$ -regularization. For simplicity, we show this fact using the following theorem under the Langevin diffusion (LD) setting, where the probability induced by Eq. (9) is expressed as  $P_{W_T|S}$ .
Theorem 5. Suppose that Assumptions 1 and 2 are satisfied and that  $F(w, z)$  is  $R$ -strongly convex  $(0 < R < \infty)$ . Then, for any  $T \in \mathbb{R}_+$ , we have
$$
\frac {\partial \mathrm {K L} (\rho_ {t} | \gamma_ {t})}{\partial t} \leq - \frac {R}{4} \mathrm {K L} (\rho_ {t} | \gamma_ {t}) + \frac {\beta}{2} \mathbb {E} \| \nabla F (W _ {t}, S) - \nabla F (W _ {t}, S ^ {\prime}) \| ^ {2}, \tag {15}
$$
$$
| \operatorname {g e n} (\mu , P _ {W _ {T} | S}; L) | \leq \sqrt {\frac {2 \beta \sigma_ {g} ^ {2}}{n} \int_ {0} ^ {T} e ^ {- \frac {(T - t) R}{4}} \mathbb {E} \| \nabla F (W _ {t} , S) - \nabla F (W _ {t} , S ^ {\prime}) \| ^ {2} \mathrm {d} t}. \tag {16}
$$
A similar bound in Eq. (16) (with  $R$  replaced by  $\lambda / e^{8\beta C}$ ) can be obtained for bounded non-convex losses with  $l_{2}$ -regularization, where  $F(w,z) = F_0(w,z) + \frac{\lambda}{2}\| w\|^2$  ( $0 < \lambda < \infty$ ) and  $F_0(w,z)$  is  $C$ -bounded ( $0 \leq C < \infty$ ). The full proof is summarized in Appendix F.4.
When comparing with Lemma 1, we can see that, in Eq. (15), stability-unrelated constants do not appear in the time evolution of KL divergence at each time step. Therefore, the resulting generalization bound is also independent of such constants. Furthermore, when compared with Theorem 2, which adds up the stability terms at all time steps, our bound is dominated by the stability terms near the final time step, as those at earlier time steps decrease geometrically by  $e^{-\frac{R}{4}}$ . This indicates that the stability around the initial time steps is of lesser importance in evaluating the final generalization performance.
Note that our bounds are closely related to the bound indicated in Proposition 9 of Mou et al. [23], which was also derived by focusing on stability. The bound of Mou et al. [23] primarily assesses
This constant is evaluable (see Vempala and Wibisono [32] or Kinoshita and Suzuki [19]).
generalization errors focusing on the gradient norm near the conclusion of training. In contrast, our bounds evaluate it through the norm of differences in gradients, emphasizing the state in the proximity of training completion. In other words, our bound allows for the evaluation of generalization errors using a stability measure that is more closely related to generalization performance than the gradient norm. This benefit originates from our approach, which tracks the time evolution of MI-related stability in Eq. (15) on the basis of information-theoretic generalization bounds, in contrast to the PAC-Bayes bounds derived from the direct analysis of stability measures as in Mou et al. [23].
4 Generalization analysis for SGLD directly using a training loss
In this section, we consider the setting that the generalization performance is measured by a training loss  $f$  directly as in Eq. (4). We show that this is possible by demonstrating that loss functions of SGLD are sub-exponential under smooth and dissipative assumptions (Section 4.1). On the basis of this fact, we obtain for the first time an information-theoretic generalization bound of SGLD that is similar to Theorem 4. Finally, combining these results with existing optimization error bounds provides an excess risk bound with improved convergence (Section 4.2).
4.1 Smooth and dissipative loss function of SGLD is sub-exponential
To perform an information-theoretic analysis for SGLD, it is necessary to know the tail behavior of  $f(W,Z)$ . Our contribution here is showing that a loss function of SGLD under smooth and dissipative assumptions is sub-exponential.
Theorem 6. Suppose that Assumptions 2, 3 and 4 are satisfied. Let  $P_{W_T} = \mathbb{E}_S[P_{W_T|S}]$  be the marginal distribution of the output obtained using the SGLD algorithm at the  $T$ -th iteration. Then, for any  $\eta \in (0,1 \wedge \frac{m}{5M^2})$  and  $T \in \mathbb{N}$ ,  $f(W_T,Z)$  is sub-exponential under the distribution  $P_{W_T} \otimes \mu$ . That is, there exist positive constants  $\sigma_e^2$  and  $\nu$  w.r.t.  $\{m,\beta,M,b,d,s^2\}^5$  such that
$$
\log \mathbb {E} _ {W _ {T} \otimes Z} \left[ e ^ {\lambda (f (W _ {T}, Z) - \mathbb {E} _ {W _ {T} \otimes Z} [ f (W _ {T}, Z) ])} \right] \leq \frac {\sigma_ {e} ^ {2} \lambda^ {2}}{2} \quad f o r a l l | \lambda | <   \frac {1}{\nu}.
$$
Proof sketch. First, note that under Assumptions 2 and 3, for any  $z \in \mathcal{Z}$ , we obtain
$$
\frac {m}{3} \| w \| ^ {2} - \frac {b}{2} \log 3 \leq f (w, z) \leq \frac {M}{2} \| w \| ^ {2} + M \sqrt {\frac {b}{m}} \| w \| + A, \tag {17}
$$
where  $A$  is a positive constant (see Lemma 8 in Appendix G.1 for its explicit form). By employing Lemma B.2 from Farghly and Rebeschini [10], we can show the following fact: for any  $p \in \mathbb{N}$ ,
$$
\mathbb {E} \| W _ {T} \| _ {2} ^ {2 p} \leq \mathbb {E} \| W _ {0} \| _ {2} ^ {2 p} + c (p), \tag {18}
$$
$$
c (p) := \frac {1}{m} \left(\frac {6}{m}\right) ^ {p - 1} \left(1 + \frac {2 ^ {2 p} p (2 p - 1) d}{m \beta}\right) \left[ \left(2 b + 8 \frac {M ^ {2}}{m ^ {2}} b\right) ^ {p} + 1 + 2 \left(\frac {d}{\beta}\right) ^ {p - 1} (2 p - 1) ^ {p} \right].
$$
This implies that  $W_{T}$  is a sub-Gaussian random variable according to Proposition 2.5.2 in Vershynin [33]. To show the sub-exponential property, we directly upper-bound  $\mathbb{E}_{W_T\otimes Z}[e^{\lambda (f(W_T,Z) - \mathbb{E}_{W_T\otimes Z}[f(W_T,Z)])}]$  by considering the Taylor expansion of the exponential moment and using Eqs. (17) and (18) (see Appendix G.2 for the complete proof).
Remark 1. In previous information-theoretic analysis studies [28, 25, 35], it is often assumed that a loss function  $l(w,Z)$  is sub-Gaussian under the distribution  $\mu$  for all  $w\in \mathcal{W}$ . In contrast, Theorem 6 holds under the distribution  $P_{\mathcal{W}_T}\otimes \mu$ , not conditioned on  $w\in \mathcal{W}$ .
We can interpret the sub-exponential property of SGLD intuitively as follows. Under Assumptions 2 and 3, the loss function grows at most as a quadratic function shown in Eq. (17). The conditional distribution of the parameters follows the Gaussian distribution, and the square of the Gaussian random variable is known as the chi-square  $(\chi^2)$  random variable [34]. According to these facts, we expect that the behavior of the loss function resembles that of the  $\chi^2$ -random variable; therefore, it is sub-exponential since the  $\chi^2$ -distribution is also sub-exponential [34]. Theorem 6 validates this intuition.
5 The explicit form of  $\sigma_e^2$  and  $\nu$  can be seen in Appendix G.2.
4.2 Generalization bounds for SGLD using the same loss for training and evaluation
On the basis of Theorem 6, we can derive the following information-theoretic generalization bound for SGLD even if a surrogate loss is not used. In contrast to Theorem 4, an assumption regarding the tail behavior of a loss function such as Assumption 1 is not necessary.
Corollary 1. Suppose that Assumptions 2, 3, and 4 are satisfied. Then, for any  $\beta \geq 2/m$ ,  $\eta \in (0, 1 \wedge \frac{m}{5M^2} \wedge 4\beta c_{\mathrm{LS}})$ , and  $T \in \mathbb{N}$ , we obtain
$$
| \operatorname {g e n} (\mu , P _ {W _ {T} | S}; F) | \leq \Psi^ {* - 1} \left(\frac {c _ {1}}{n} \left(1 \wedge \frac {\eta T}{4 \beta c _ {\mathrm {L S}}}\right) (V _ {\nabla} + c _ {2})\right),
$$
$$
\Psi^ {* - 1} (y) = \left\{ \begin{array}{l l} \sqrt {2 \sigma_ {e} ^ {2} y} & \text {i f} y \leq \frac {\sigma_ {e} ^ {2}}{2 \nu} \\ \nu y + \frac {\sigma_ {e} ^ {2}}{2 \nu} & \text {o t h e r w i s e} \end{array} \right.,
$$
$c_{1}$  and  $c_{2}$  are the same as in Theorem 4, and  $\sigma_e^2$  and  $\nu$  are the same as in Theorem 6.
Proof sketch. This is the direct consequence of the sub-exponential property from Theorem 6 and the upper bound of MI in Eq. (7) (see Appendix G.3 for the complete proof).  $\square$
Remark 2. Despite the assumptions of Corollary 1 being the same as those made by Farghly and Rebeschini [10] except for the initial distribution and step size, the resulting bound becomes 0 as  $n \to \infty$  without being dependent on inverse stepsize.
We conclude this section by introducing our excess risk bound. Let us define the excess risk as follows:  $\operatorname{Excess}(\mu, P_{W|S}) \coloneqq \mathbb{E}_{W,S}[F_{\mu}(W) - F_{\mu}(w^{*})]$ , where  $w^{*} = \operatorname{argmin}_{w \in \mathcal{W}} F_{\mu}(w)$ . Under this definition, we derive the following upper bound for the excess risk by utilizing Corollary 1.
Corollary 2. Suppose that Assumptions 2, 3, and 4 are satisfied. Then, for any  $\beta \geq 2 / m$ ,  $\eta \in (0,1 \wedge \frac{m}{5M^2} \wedge 4\beta c_{\mathrm{LS}})$ , and  $T \in \mathbb{N}$ , we obtain
$$
\operatorname {E x c e s s} (\mu , P _ {W _ {T} | S}) = \mathcal {O} \left(\sqrt {\frac {(\eta T \wedge 1)}{n}} + e ^ {- \eta T / c _ {L S}} + \sqrt {\eta} + c _ {\mathrm {e r r}}\right),
$$
where  $c_{\mathrm{err}}$  is the positive constant w.r.t.  $\{M, m, b, d, \beta\}$  corresponding to the optimization error.
We show the complete proof in Appendix G.4. In contrast with the existing excess risk studies, our bound does not diverge with increasing  $t$  owing to the time-independent generalization bound in Corollary 1.
5 Related studies and discussion
In this section, we compare our generalization bounds with those in related studies. Table 1 shows the order of each bound value along with its assumptions for a loss function.
SGLD analysis with/without changing losses. The existing generalization error bounds in Table 1 are time-dependent; namely, we need to impose restrictive conditions for the step size  $\eta$  in terms of  $t$  to achieve a generalization bound that decays to zero with increasing sample size [29, 28, 25, 35] (see the right column in Table 1). Some important applications of SGLD do not satisfy these conditions. For instance, the short-run Markov chain Monte Carlo [26] method used in energy-based models [17] adopts SGLD with a fixed step size. Another example is the cyclic SGLD [42] used in deep learning, where the step size is periodically increased or decreased to facilitate escape from local optima.
Farghly and Rebeschini [10] first analyzed the generalization error of SGLD by using smoothness and dissipative assumptions, which are broadly used in sampling and non-convex optimization studies [29, 41, 4, 44]. Their bound is time-independent; the bound does not diverge with time and achieves the order  $\mathcal{O}(n^{-1/2})$ . However, the bound depends on the inverse of step size  $\eta^{-1/2}$  owing to the reflection coupling [9], which results in the unnatural behavior of decreasing  $\eta$  with increasing  $t$ . Farghly and Rebeschini [10] also derived a bound that does not suffer from this problem by assuming the Lipschitz loss function with weight decay; however, these assumptions excessively restrict the
class of loss functions and algorithms. In contrast to these bounds, our bound is time-independent and does not require scaling  $\eta$ ,  $t$ , and  $n$  to achieve  $\mathcal{O}(n^{-1/2})$ .
Time evolution analysis of MI via FP equation. The analysis of SGLD using the FP equation has been successfully used in the convergence analysis of SGLD [32, 19]. These studies present analyses of the discretization errors and convergence properties of the unadjusted Langevin algorithm, SGLD, and variance-reduction SGLD (SVRG-LD) [8], comparing them with the continuous Langevin dynamics through the FP equation.
In generalization error analysis, the FP equation is mainly used to analyze the time evolution of KL divergence appearing in a generalization bound on the basis of the stability approach. Li et al. [20] analyzed the time evolution of the KL divergence between the probability densities of the parameters obtained from two training datasets that differ by only one data point under the bounded loss assumption. Mou et al. [23] also studied the KL divergence and Hellinger divergence, and they derived a generalization error bound on the basis of the PAC-Bayes notion [22]. Our idea is similar to these: we analyze the time evolution of the KL divergence between the probability densities of the parameters obtained from two training datasets. The differences between our approach and other approaches are twofold. First, we do not assume weight decay or Lipschitz continuity but instead derive our analysis assuming smoothness and dissipativity. Second, the KL divergence we analyzed is tighter than that of the PAC-Bayes bound with data-independent prior dealt by Mou et al. [23].
In this paper, we provide a generalization analysis of SGLD, where Gaussian noise is a fundamental assumption for our theoretical results. Thus, it is difficult to extend our analysis to other noisy iterative algorithm variants with a different noise, such as differentially private SGD with Laplace or uniform noise [36]. Another limitation of this study is that we have estimated the sub-exponential parameter roughly with respect to the dimensions of the model parameters. Further investigation of the sub-exponentiality of smooth and dissipative losses, and improvement of the dependence on dimensionality, are crucial for enhancing the practicality of our generalization bounds. The sub-exponential property of a loss function is expected to be helpful in fields other than generalized error analysis. For example, this property opens up room for new theoretical analysis policies that employ useful concentration and transport inequalities [34] in the sampling and optimization context. We hope that the analysis presented in this paper goes beyond generalization analysis and provides valuable insights into understanding the characteristics of machine learning.
<sup>6</sup> The order of the bound in Mou et al. [23] varies with the choice of regularization parameters and decay factors. In this paper, we adopt the order of this bound in Table 1 of Farghly and Rebeschini [10]. For a more comprehensive discussion, we refer to Section 5.2 of Mou et al. [23].
We sincerely appreciate the anonymous reviewers for their insightful feedback. FF was supported by JSPS KAKENHI Grant Number JP23K16948. FF was supported by JST, PRESTO Grant Number JPMJPR22C8, Japan. MF was supported by RIKEN Special Postdoctoral Researcher Program. MF was supported by JST, ACT-X Grant Number JPMJAX210K, Japan.
[1] R. Amit, B. Epstein, S. Moran, and R. Meir. Integral probability metrics PAC-bayes bounds. In Advances in Neural Information Processing Systems, 2022.
[2] D. Bakry, I. Gentil, and M. Ledoux. Analysis and Geometry of Markov Diffusion Operators, volume 348. Springer Science & Business Media, 2013.
[3] Y. Bu, S. Zou, and V. V. Veeravalli. Tightening mutual information-based bounds on generalization error. IEEE Journal on Selected Areas in Information Theory, 1(1):121-130, 2020.
[4] N. H. Chau, É. Moulines, M. Rásonyi, S. Sabanis, and Y. Zhang. On stochastic gradient Langevin dynamics with dependent data streams: The fully nonconvex case. SIAM Journal on Mathematics of Data Science, 3(3):959-986, 2021.
[5] T. Chen, E. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In Proceedings of the 31st International Conference on Machine Learning, volume 32, pages 1683–1691, 2014.
[6] T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley & Sons, 2012.
[7] T. Deck and S. Kruse. Parabolic differential equations with unbounded coefficients - A generalization of the parametrix method. Acta Applicandae Mathematica, 74:71-91, 2002.
[8] K. A. Dubey, S. J. Reddi, S. A. Williamson, B. Poczos, A. J. Smola, and E. P. Xing. Variance reduction in stochastic gradient Langevin dynamics. In Advances in Neural Information Processing Systems, volume 29, pages 1154-1162, 2016.
[9] A. Eberle. Reflection couplings and contraction rates for diffusions. *Probability Theory and Related Fields*, 166:851–886, 2021.
[10] T. Farghly and P. Rebeschini. Time-independent generalization bounds for SGLD in nonconvex settings. In Advances in Neural Information Processing Systems, volume 34, pages 19836-19846, 2021.
[11] V. Feldman, I. Mironov, K. Talwar, and A. Thakurta. Privacy amplification by iteration. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science (FOCS), pages 521-532, 2018.
[12] A. Friedman. *Partial Differential Equations of Parabolic Type*. Courier Dover Publications, 2008.
[13] S. B. Gelfand and S. K. Mitter. Recursive stochastic algorithms for global optimization in  $\mathbb{R}^d$ . SIAM Journal on Control and Optimization, 29(5):999-1018, 1991.
[14] J. K. Hale. Asymptotic Behavior of Dissipative Systems. Mathematical surveys and monographs. American Mathematical Society, 1988.
[15] H. Harutyunyan, M. Raginsky, G. Ver Steeg, and A. Galstyan. Information-theoretic generalization bounds for black-box learning algorithms. In Advances in Neural Information Processing Systems, pages 24670-24682, 2021.
[16] U. G. Haussmann and E. Pardoux. Time reversal of diffusions. The Annals of Probability, 14 (4):1188-1205, 1986.
[17] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771-1800, 2002.
[36] H. Wang, R. Gao, and F. P. Calmon. Generalization bounds for noisy iterative algorithms using properties of additive noise channels. Journal of Machine Learning Research, 24(26):1-43, 2023.
[37] Z. Wang and Y. Mao. On the generalization of models trained with SGD: Information-theoretic bounds and implications. In The Tenth International Conference on Learning Representations, 2022.
[38] Z. Wang and Y. Mao. Tighter information-theoretic generalization bounds from supersamples. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pages 36111-36137, 2023.
[39] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th International Conference on International Conference on Machine Learning, pages 681-688, 2011.
[40] A. Xu and M. Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. In Advances in Neural Information Processing Systems, volume 30, pages 2524-2533, 2017.
[41] P. Xu, J. Chen, D. Zou, and Q. Gu. Global convergence of Langevin dynamics based algorithms for nonconvex optimization. In Advances in Neural Information Processing Systems, volume 31, pages 3126-3137, 2018.
[42] R. Zhang, C. Li, J. Zhang, C. Chen, and A. G. Wilson. Cyclical stochastic gradient MCMC for Bayesian deep learning. In The Eighth International Conference on Learning Representations, 2020.
[43] Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient Langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, volume 65, pages 1980-2022, 2017.
[44] Y. Zhang, Ö. D. Akyildiz, T. Damoulas, and S. Sabanis. Nonasymptotic estimates for stochastic gradient Langevin dynamics under local conditions in nonconvex optimization. Applied Mathematics & Optimization, 87(25), 2021.