Flatness-Aware Stochastic Gradient Langevin Dynamics

## Abstract

Abstract Generalization in deep learning is closely tied to the pursuit of flat minima in the loss landscape, yet classical Stochastic Gradient Langevin Dynamics (SGLD) offers no mechanism to bias its dynamics toward such low-curvature solutions. This work introduces Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD), designed to efficiently and provably seek flat minima in high-dimensional nonconvex optimization problems. At each iteration, fSGLD uses the stochastic gradient evaluated at parameters perturbed by isotropic Gaussian noise, commonly referred to as Random Weight Perturbation (RWP), thereby optimizing a randomized-smoothing objective that implicitly captures curvature information. Leveraging these properties, we prove that the invariant measure of fSGLD stays close to a stationary measure concentrated on the global minimizers of a loss function regularized by the Hessian trace whenever the inverse temperature and the scale of random weight perturbation are properly coupled. This result provides a rigorous theoretical explanation for the benefits of random weight perturbation. In particular, we establish non-asymptotic convergence guarantees in Wasserstein distance with the best known rate and derive an excess-risk bound for the Hessian-trace regularized objective. Extensive experiments on noisy-label and large-scale vision tasks, in both training-from-scratch and fine-tuning settings, demonstrate that fSGLD achieves superior or comparable generalization and robustness to baseline algorithms while maintaining the computational cost of SGD, about half that of SAM. Hessian-spectrum analysis further confirms that fSGLD converges to significantly flatter minima.

## 1 Introduction

Consider the overdamped Langevin dynamics governed by the stochastic differential equation (SDE)

$$ $\mathrm{d}Z_{t}=-\nabla u(Z_{t})\mathrm{d}t+\sqrt{2\beta^{-1}}\mathrm{d}B_{t},$ $$

which admits a unique invariant (Gibbs) measure $\pi_{\beta}(\theta)$ proportional to $\exp(-\beta u(\theta))$, where $\beta>0$ is the inverse temperature and $(B_{t})_{t\geq 0}$ is a $d$-dimensional Brownian motion. As $\beta$ increases, this Gibbs measure concentrates on the global minimizers of $u$, establishing a direct link between Langevin dynamics and global optimization. Building on this property, Stochastic Gradient Langevin Dynamics (SGLD) (Welling & Teh, 2011; Raginsky et al., 2017) was proposed as the Euler-Maruyama discretization of the Langevin SDE in which the exact gradient $\nabla u$ is replaced by a stochastic gradient. SGLD has attracted considerable attention as a prominent optimization algorithm for nonconvex problems, and under mild regularity conditions a series of works has established non-asymptotic global convergence guarantees (Raginsky et al., 2017; Xu et al., 2018; Majka et al., 2020; Chau et al., 2021; Zhang et al., 2023). Despite these elegant theoretical results, SGLD has not become a widely used optimizer in deep learning practice, largely because it lacks an intrinsic mechanism to favor flat minima, which are closely associated to strong generalization.

Alongside advances in SGLD, a separate line of work in deep learning has explored flatter solutions to improve generalization, inspired by the flat minima hypothesis (Hochreiter & Schmidhuber, 1997). As a result, numerous flatness-aware optimization algorithms have been developed,
including Random Weight Perturbation (RWP) (Bisla et al., 2022; Li et al., 2024a), Entropy-SGD (Chaudhari et al., 2017), Sharpness-Aware Minimization (SAM) (Foret et al., 2021) and their variants (Xie et al., 2024; Li et al., 2024b; Tahmasebi et al., 2024; Luo et al., 2024; Chen et al., 2024; Kang et al., 2025; Wei et al., 2025; Liu et al., 2022a, b; Du et al., 2022b; Li et al., 2025). In principle, flatness-aware optimization promotes exploration of flat regions by replacing the standard stochastic gradient with a perturbed gradient. For example, SAM applies a worst-case adversarial perturbation within a local neighborhood, whereas RWP uses symmetric random noise to generate the gradient perturbation and can be viewed as computing the stochastic gradient of a randomized-smoothing objective (Duchi et al., 2012). However, SAM’s min–max formulation requires double gradient evaluations, leading to roughly twice the computational cost of standard SGD. On the theoretical side, recent studies have produced important advances in the analysis of SAM and related flatness-aware optimization methods, yielding valuable insights on generalization bounds, stability, and (local) convergence properties; e.g., see Andriushchenko & Flammarion (2022); Bartlett et al. (2023); Si & Yun (2023); Yu et al. (2024); Khanh et al. (2024); Oikonomou & Loizou (2025); Zhang et al. (2024); Li et al. (2024a).
However, with a few notable exceptions (Ahn et al., 2024; Gatmiry et al., 2024), the global convergence properties of flatness-aware optimization in nonconvex settings, as well as a rigorous theoretical understanding of the role of RWP, remain relatively unexplored.

To address these challenges, we introduce Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD), a principled synthesis of randomized smoothing and Langevin dynamics that efficiently explores flat minima. While randomized-smoothing surrogates are known to encode second-order information such as the Hessian trace, they also contain higher-order remainder terms of which effects are not negligible in high-dimensional nonconvex problems, weakening the intended flatness-aware regularization effect. Our key theoretical contribution is to show that when the two key hyperparameters, the inverse temperature parameter $\beta$ and the perturbation scale $\sigma$, are properly balanced, the invariant measure of fSGLD concentrates on the global minimizers of the true Hessian-trace regularized objective, thereby isolating the genuine flatness-aware regularization effect. This principled coupling is crucial, as it ensures that the global exploration driven by Langevin dynamics is effectively guided across a landscape smoothed by the perturbation noise, steering the process toward genuinely flat regions. In particular, we establish non-asymptotic convergence guarantees in Wasserstein distance and an explicit excess-risk bound for the Hessian-trace-regularized objective, providing the rigorous evidence of the benefits of RWP in nonconvex settings. Our framework bridges and advances the theory and practice of flatness-aware stochastic optimization, opening new avenues to incorporate geometric smoothing into Langevin sampling and paving the way for more effective and principled flatness-regularized learning. To validate these results, we evaluate fSGLD on noisy-label datasets (CIFAR-10N/100N, WebVision) and large-scale vision fine-tuning (ViT-B/16). Extensive experiments demonstrate that fSGLD consistently matches or outperforms baselines including SGD, AdamW, SGLD, and SAM in generalization and robustness while maintaining the computational cost of standard SGD. Notably, using the theoretically prescribed coupling between $\beta$ and $\sigma$ yields substantially better performance than simply fixing a large $\beta$, which is the common SGLD practice. In summary, fSGLD is the first to combine the SGLD framework with the concept of flatness and to provide a global convergence analysis for flatness-aware optimization, thereby advancing the theoretical and practical foundations of both areas.

## 2 Problem Setting and FSGLD Algorithm

##### Notation.

Let $(\Omega,\mathscr{F},\mathbb{P})$ be a fixed
probability space. We denote the probability law of a random variable $Y$ by $\mathscr{L}(Y)$. Fix integers $d,m\geq 1$. Let $I_{d}$ be the identity matrix of dimension $d$. The Euclidean scalar product is denoted by $\langle\cdot,\cdot\rangle$, with $|\cdot|$ standing for the corresponding norm. Let $f:\mathbb{R}^{d}\rightarrow\mathbb{R}$ be a continuously differentiable function, and we denote its gradient by $\nabla f$. For any integer $q\geq 1$, let $\mathscr{P}(\mathbb{R}^{q})$ be the set of probability measures on $\mathscr{B}(\mathbb{R}^{q})$. For $\mu$, $\nu\in\mathscr{P}(\mathbb{R}^{d})$, let $\mathscr{C}(\mu,\nu)$ denote the set of probability measures $\Gamma$ on $\mathscr{B}(\mathbb{R}^{2d})$ such that its respective marginals are $\mu$ and $\nu$. For any $\mu$ and $\nu\in\mathscr{P}(\mathbb{R}^{d})$, the Wasserstein distance of order $p\geq 1$ is defined as

$$ $W_{p}(\mu,\nu)=\left(\inf_{\Gamma\in\mathscr{C}(\mu,\nu)}\int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}}|x-y|^{p}\ \text{d}\Gamma(x,y)\right)^{\frac{1}{p}}.$ (1) $$

For any $\mu$ and $\nu\in\mathscr{P}(\mathbb{R}^{d})$, then Kullbak-Leibler divergence (or relative entropy) between $\mu$ and $\nu$ is defined as

$$ $\begin{split}\text{KL}(\mu||\nu)&=\begin{cases}\int_{\mathbb{R}^{d}}\log\left(\frac{\mathrm{d}\mu}{\mathrm{d}\nu}\right)\mathrm{d}\mu,&\text{if}\ \mu\ll\nu,\\ \infty,&\text{otherwise}.\end{cases}\end{split}$ (2) $$

### 2.1 Intractable Hessian-based Regularization

We consider the following nonconvex stochastic optimization problem:

$$ $\min_{\theta\in\mathbb{R}^{d}}u(\theta):=\min_{\theta\in\mathbb{R}^{d}}\mathbb{E}\bigl[U(\theta,X)\bigr],$ (3) $$

where $u:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is a four-times continuously differentiable function with gradient $h:=\nabla u$, $U:\mathbb{R}^{d}\times\mathbb{R}^{m}\rightarrow\mathbb{R}$ is a measurable function satisfying $\mathbb{E}[|U(\theta,X)|]<\infty$ for all $\theta\in\mathbb{R}^{d}$, and $X$ is a random variable with probability law $\mathscr{L}(X)$. In practice, the gradient $h$ of $u$ is usually unknown and one only has access to its unbiased estimate, i.e. $h(\theta)=\mathbb{E}[\nabla_{\theta}U(\theta,X)]$.

To improve generalization, we incorporate an inductive bias for flatness through a flatness-aware objective. More specifically, instead of optimizing the original objective $u$, we aim to solve the following *Hessian-trace regularized objective*:

$$ $v(\theta):=u(\theta)+\frac{\sigma^{2}}{2}\text{tr}\left(H(\theta)\right),$ (4) $$

where $\text{tr}(H(\theta))$ is the trace of the Hessian of $u$ evaluated at $\theta$ and $\sigma>0$ controls the strength of the sharpness regularization. The global minimizers of this regularized objective $v$ represent a trade-off between low loss from the original objective $u$ and low curvature. For brevity, we will refer to these points as the global flat minima (i.e., $\arg\min_{\theta\in\mathbb{R}^{d}}v(\theta)$). However, computing $\text{tr}\left(H(\theta)\right)$ is expensive in high dimension.

### 2.2 Randomized Smoothing as a Tractable Surrogate

To obtain a tractable alternative to the Hessian-trace regularized objective in [4](https://arxiv.org/html/2510.02174v1#S2.E4), we introduce a Gaussian perturbation $\epsilon\sim\mathscr{N}(0,\sigma^{2}I_{d})$ with $\sigma\in(0,1)$,
independent of $X$, and define the *randomized-smoothing surrogate objective:*

$$ $g_{\epsilon}(\theta):=\mathbb{E}\bigl[u(\theta+\epsilon)\bigr]=\mathbb{E}\bigl[\mathbb{E}_{X}\bigl[U(\theta+\epsilon,X)\bigr]\bigr].$ (5) $$

where the outer expectation is taken with respect to the noise $\epsilon$ and
$\mathbb{E}_{X}[\cdot]$ denotes the conditional expectation given $\epsilon$. This simple surrogate allows us to access curvature information. By Taylor’s theorem, we have

$$ $u(\theta+\epsilon)=u(\theta)+\nabla u(\theta)^{\!\top}\epsilon+\tfrac{1}{2}\epsilon^{\!\top}H(\theta)\epsilon+\mathscr{R}(\theta,\epsilon),$ $$

where $\mathscr{R}(\theta,\epsilon)$ is the remainder term.
Taking the expectation over $\epsilon\sim\mathscr{N}(0,\sigma^{2}I_{d})$ yields the key connection:

$$ $\displaystyle g_{\epsilon}(\theta)$ $\displaystyle=u(\theta)+\tfrac{\sigma^{2}}{2}\text{tr}\left(H(\theta)\right)+\mathbb{E}[\mathscr{R}(\theta,\epsilon)]$ $\displaystyle=v(\theta)+\mathbb{E}[\mathscr{R}(\theta,\epsilon)].$ (6) $$

Thus, optimizing the tractable surrogate $g_{\epsilon}$ introduces the desired inductive bias toward flat minima
by implicitly minimizing the Hessian–trace regularized objective $v$,
provided that the remainder term $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]$ is negligible.

### 2.3 FSGLD Algorithm

To optimize the surrogate objective $g_{\epsilon}$ in [5](https://arxiv.org/html/2510.02174v1#S2.E5), we propose the Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD) algorithm. Formally, let $\theta_{0}$ be an $\mathbb{R}^{d}$-valued random variable representing the initial value, $(X_{k})_{k\in\mathbb{N}}$ be an i.i.d sequence of data, $(\epsilon_{k})_{k\in\mathbb{N}}$ be i.i.d copies of the Gaussian perturbation $\epsilon\sim\mathscr{N}(0,\sigma^{2}I_{d})$, and $(\xi_{k})_{k\in\mathbb{N}}$ be an independent sequence of standard $d$-dimensional Gaussian random variables. We assume that $\theta_{0}$, $(\epsilon_{k})_{k\in\mathbb{N}}$, and $(\xi_{k})_{k\in\mathbb{N}}$ are all mutually independent. Then, the fSGLD algorithm is given by

$$ $\begin{split}\begin{cases}\theta_{0}^{\text{fSGLD}}&:=\theta_{0},\\ \theta_{k+1}^{\text{fSGLD}}&=\theta_{k}^{\text{fSGLD}}-\lambda\nabla_{\theta}U(\theta^{\text{fSGLD}}_{k}+\epsilon_{k+1},X_{k+1})+\sqrt{2\lambda\beta^{-1}}\xi_{k+1},\qquad k\in\mathbb{N}\end{cases}\end{split}$ (7) $$

where $\lambda>0$ is the stepsize, $\beta>0$ is the inverse temperature. We make three important remarks about this update rule. First, the gradient term in [7](https://arxiv.org/html/2510.02174v1#S2.E7) is a unbiased stochastic gradient of $g_{\epsilon}$, as its expectation over both the data $X$ and the perturbation $\epsilon$ recovers the true gradient $\nabla g_{\epsilon}$:

$$ $\begin{split}\nabla g_{\epsilon}(\theta)=\mathbb{E}[\mathbb{E}_{X}[\nabla_{\theta}U(\theta+\epsilon,X)]].\end{split}$ (8) $$

Second, the fSGLD can be interpreted as the standard SGLD for the original objective $u$ combined with RWP. Third, under appropriate conditions, which will be introduced in the next section, the fSGLD algorithm generates a Markov chain that converges to a unique invariant (Gibbs) measure. This measure, denoted by $\pi^{\text{fSGLD}}_{\beta}$, is associated with the randomized-smoothing surrogate objective $g_{\epsilon}$, i.e., $\pi^{\text{fSGLD}}_{\beta}(\theta)\propto\exp(-\beta g_{\epsilon}(\theta))$. The formal convergence guarantees are provided in Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2).

## 3 Theoretical Results

In this section, we present the main theoretical results that rigorously validate the fSGLD algorithm. We begin by stating the formal assumptions for our analysis. We then prove that the invariant measure of fSGLD converges to an ideal target distribution over flat minima when its key hyperparameters $\beta$ and $\sigma$ are properly coupled. Building on this, we derive non-asymptotic convergence guarantees for the fSGLD iterates in both Wasserstein distance and for the excess risk. The logical flow of our theoretical framework is summarized in the schematic illustration in Figure [1](https://arxiv.org/html/2510.02174v1#S3.F1).

Figure: Figure 1: A schematic overview of the theoretical framework of fSGLD. The process begins with the original objective $u(\theta)$ and its associated Gibbs measure $\pi_{\beta}^{\text{SGLD}}$ (left). Randomized smoothing transforms this into a tractable surrogate objective, $g_{\epsilon}(\theta)$, which is the basis for the fSGLD algorithm and its invariant measure, $\pi^{\text{fSGLD}}_{\beta}$ (center). This highlights a key distinction: while the Gibbs measure of standard SGLD, $\pi_{\beta}^{\text{SGLD}}$, is indifferent to the flatness of the minima, the fSGLD framework is designed such that its invariant measure, $\pi^{\text{fSGLD}}_{\beta}$, targets the distribution over the flattest minima. Our ultimate goal is to target the Hessian-trace regularized objective $v(\theta)$ and its corresponding measure $\pi^{\star}_{\beta,\sigma}$, which concentrates on the desired global flat minima (right).
Refer to caption: x1.png

### 3.1 Assumptions

We first state the formal assumptions for our main theoretical results. Specifically, our assumptions impose standard conditions on: (i) moments of the initial parameters, stochastic gradient, and the noise processes; (ii) a Lipschitz condition on the stochastic gradient; and (iii) a dissipativity condition to ensure the stability of the Langevin dynamics.

###### Assumption 1 (Moments of the initial parameter, stochastic gradient, and independence of the data and noise perturbation) .

We assume the initial parameter $\theta_{0}$ has a finite fourth moment, $\mathbb{E}[|\theta_{0}|^{4}]<\infty$, and that we have access to an unbiased stochastic gradient for the original objective $u$, $\mathbb{E}[\nabla_{\theta}U(\theta,X)]=h(\theta)$, where the data sequence $(X_{k})_{k\in\mathbb{N}}$ is is i.i.d. Furthermore, the perturbation noise $(\epsilon_{k})_{k\in\mathbb{N}}\sim\mathscr{N}(0,\sigma^{2}I_{d})$ with $\sigma\in(0,1)$, $(X_{k})_{k\in\mathbb{N}}$, $(\xi_{k})_{k\in\mathbb{N}}\sim\mathscr{N}(0,I_{d})$, and $\theta_{0}$ are mutually independent.

###### Assumption 2 (Lipschitzness) .

There exists $\varphi:\mathbb{R}^{m}\rightarrow[1,\infty)$ with $\mathbb{E}[|(1+|X_{0}|)\varphi(X_{0})|^{4}]<\infty$, and constants $L_{1},L_{2}>0$ such that, for all $x$, $x^{{}^{\prime}}\in\mathbb{R}^{m}$ and $\theta$, $\theta^{{}^{\prime}}\in\mathbb{R}^{d}$,

$$ $\begin{split}|\nabla_{\theta}U(\theta,x)-\nabla_{\theta^{{}^{\prime}}}U(\theta^{{}^{\prime}},x)|&\leq L_{1}\varphi(x)|\theta-\theta^{{}^{\prime}}|,\\ |\nabla_{\theta}U(\theta,x)-\nabla_{\theta}U(\theta,x^{{}^{\prime}})|&\leq L_{2}(\varphi(x)+\varphi(x^{{}^{\prime}}))(1+|\theta|)|x-x^{{}^{\prime}}|,\end{split}$ $$

###### Assumption 3 (Dissipativity) .

There exist a measurable function (symmetric matrix-valued) function $A:\mathbb{R}^{m}\rightarrow\mathbb{R}^{d\times d}$ and a measurable function $\hat{b}:\mathbb{R}^{m}\rightarrow\mathbb{R}$ such that for any $x\in\mathbb{R}^{m}$, $y\in\mathbb{R}^{d}$, $\langle y,A(x)y\rangle\geq 0$ and for all $\theta\in\mathbb{R}^{d}$ and $x\in\mathbb{R}^{m}$,

$$ $\langle\nabla_{\theta}U(\theta,x),\theta\rangle\geq\langle\theta,A(x)\theta\rangle-\hat{b}(x).$ $$

The smallest eigenvalue of $\mathbb{E}[A(X_{0})]$ is a positive real number $\bar{a}>0$ and $\mathbb{E}[\hat{b}(X_{0}))]=\bar{b}>0$.

Note that this dissipativity condition is a standard requirement for analysis of SGLD in the literature; e.g., see Raginsky et al. (2017); Xu et al. (2018); Deng et al. (2020a, b, 2022); Futami & Fujisawa (2023). In particular, our version in Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) follows the more general formulation of Zhang et al. (2023), which allows for dependency on the data $X$. Moreover, several direct consequences of these assumptions, which are useful for our subsequent analysis, are detailed in Appendix [B](https://arxiv.org/html/2510.02174v1#A2).

###### Assumption 1 (Moments of the initial parameter, stochastic gradient, and independence of the data and noise perturbation) .

###### Assumption 2 (Lipschitzness) .

###### Assumption 3 (Dissipativity) .

### 3.2 Target Gibbs Measure for Global Flat Minima

Our analysis begins by defining the ideal target distribution which concentrates on the global flat minima. The natural choice is the Gibbs measure associated with $v$, which we define as $\pi^{\star}_{\beta,\sigma}$:

$$ $\pi^{\star}_{\beta,\sigma}(\mathrm{d}\theta)\propto\exp(-\beta v(\theta))\mathrm{d}\theta.$ (9) $$

By construction, as the inverse temperature $\beta\rightarrow\infty$, this measure concentrates on the global flat minima.

The central question is whether the invariant measure of fSGLD, $\pi^{\text{fSGLD}}_{\beta}$, converges to this ideal Gibbs measure $\pi^{\star}_{\beta,\sigma}$. For these two Gibbs measures to align, the remainder term $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]$ in [6](https://arxiv.org/html/2510.02174v1#S2.E6) must be negligible. In high-dimensional nonconvex problems, this is a non-trivial condition, as higher-order terms can be substantial and unpredictable, potentially corrupting the intended regularization effect. For this reason, in the low-temperature limit ($\beta\rightarrow\infty$), a careful interplay between the inverse temperature $\beta$ and the noise scale $\sigma$ becomes essential. Please refer to Appendix [A](https://arxiv.org/html/2510.02174v1#A1) for the formal relationship between two Gibbs measures $\pi^{\text{fSGLD}}_{\beta}$ and $\pi^{\star}_{\beta,\sigma}$.

The following proposition shows that when the perturbation scale $\sigma$ and inverse temperature $\beta$ are properly coupled, the invariant measure of fSGLD converges to the ideal target measure in the Wasserstein distance of order two and in KL divergence.

###### Proposition 3.1 .

Let Assumptions [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold, and let $\sigma=\beta^{-\frac{1+\eta}{4}}$ for $\eta>0$. Then

$$ $\begin{split}\lim_{\beta\rightarrow\infty}\text{KL}(\pi_{\beta}^{\text{fSGLD}}||\pi_{\beta,\sigma}^{\star})=0,\quad\text{and}\quad\lim_{\beta\rightarrow\infty}W_{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star})&=0.\end{split}$ (10) $$

The proof of Proposition [3.1](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem1) is postponed to Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2). This proposition rigorously shows how RWP induces the desired Hessian-trace regularization effect through a theoretically-prescribed coupling of the two key hyperparameters, $\sigma$ and $\beta$. As demonstrated in our experiments, this coupling yields meaningful improvements in generalization.

###### Proposition 3.1 .

### 3.3 Convergence Guarantees for fSGLD

Having established that fSGLD correctly targets the ideal distribution for flat minima, our first main result provides non-asymptotic error bounds on the Wasserstein-1 and -2 distances between the law of the $k$-th fSGLD iterate $\mathscr{L}(\theta_{k}^{\text{fSGLD}})$ and the target Gibbs measure $\pi^{\star}_{\beta,\sigma}$. All proofs for the results in this section are provided in Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2).

###### Theorem 3.2 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold, and let $\sigma=\beta^{-\frac{1+\eta}{4}}$ for $\eta>0$. Then, there exist constants $\dot{c},D_{1},D_{2},D_{3},\underline{D}>0$ such that, for every $\beta>0$, for $0<\lambda\leq\lambda_{\text{max}}$, and $k\in\mathbb{N}$,

$$ $\begin{split}W_{1}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})&\leq D_{1}e^{-\dot{c}\lambda k/2}(1+\mathbb{E}[|\theta_{0}|^{4}])+(D_{2}+D_{3})\sqrt{\lambda}+\underline{D},\end{split}$ (11) $$

where $\dot{c}$, $D_{1}$, $D_{2}$, $D_{3}$ are given explicitly in the Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2), $\lambda_{\text{max}}$ is given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), and $\underline{D}=\mathscr{O}(\beta^{-\eta})$ whose expression is explicitly given in the Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2).

###### Corollary 3.3 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold, and let $\sigma=\beta^{-\frac{1+\eta}{4}}$ for $\eta>0$. Then, there exists constants $\dot{c},D_{4},D_{5},D_{6},\underline{D}>0$ such that, for every $\beta>0$, $0<\lambda\leq\lambda_{\text{max}}$, and $k\in\mathbb{N}$,

$$ $\begin{split}W_{2}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})&\leq D_{4}e^{-\dot{c}\lambda k/4}(\mathbb{E}[|\theta_{0}|^{4}]+1)+(D_{5}+D_{6})\lambda^{1/4}+\underline{D},\end{split}$ (12) $$

where $D_{4}$, $D_{5}$, $D_{6}$ are given explicitly in the Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2), $\lambda_{\text{max}}$ is given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), and $\dot{c}$, $\underline{D}$ are the same as in Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2).

###### Remark 3.4 .

The constant $\underline{D}$ on the right-hand side of [11](https://arxiv.org/html/2510.02174v1#S3.E11) and [12](https://arxiv.org/html/2510.02174v1#S3.E12) vanishes as $\beta\to\infty$. Moreover, the remainder terms on the right-hand side of [11](https://arxiv.org/html/2510.02174v1#S3.E11) and [12](https://arxiv.org/html/2510.02174v1#S3.E12) can be made arbitrarily small by choosing $\lambda$ sufficiently small. We emphasize that Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2) and Corollary [3.3](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem3) recover the best known convergence results for SGLD under comparable assumptions, see e.g. Zhang et al. (2023). A detailed complexity analysis of these bounds is provided in Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2).

While the previous results guarantee convergence from a sampling perspective, our final result analyzes fSGLD as an optimizer. The following theorem provides a non-asymptotic bound on the expected excess risk with respect to the Hessian-trace regularized objective $v$.

###### Theorem 3.5 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold, and let $\sigma=\beta^{-\frac{1+\eta}{4}}$ for $\eta>0$. Then, there exist constants $\dot{c}$, $D_{1}^{\#}$, $D_{2}^{\#}$, $D_{3}^{\#}>0$ such that, for every $\beta>0$, $0<\lambda\leq\lambda_{\text{max}}$, $k\in\mathbb{N}$,

$$ $\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\inf_{\theta\in\mathbb{R}^{d}}v(\theta)\leq D_{1}^{\#}e^{-\dot{c}\lambda k/4}+D_{2}^{\#}\lambda^{1/4}+D_{3}^{\#},$ (13) $$

where $D_{1}^{\#}$, $D_{2}^{\#}$, and $D_{3}^{\#}$ are given explicitly in the Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2), and $\dot{c}$ is the same as in Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2).

This result provides a rigorous guarantee that fSGLD finds global flat minima by effectively solving the Hessian-trace regularized objective.

###### Remark 3.6 .

The constant $D_{3}^{\#}=O\left((d/\beta)\log((\beta/d+1))+\beta^{-(1+\eta)}\right)$ vanishes as $\beta\rightarrow\infty$. Furthermore, the remainder terms on the right-hand side of [13](https://arxiv.org/html/2510.02174v1#S3.E13) can be made arbitrarily small by choosing $\lambda$ sufficiently small. A detailed complexity analysis for the bound in [13](https://arxiv.org/html/2510.02174v1#S3.E13) is provided in Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2).

###### Theorem 3.2 .

###### Corollary 3.3 .

###### Remark 3.4 .

###### Theorem 3.5 .

###### Remark 3.6 .

## 4 Numerical Experiments

The code for all the experiments is available at [https://github.com/youngsikhwang/Flatness-aware-SGLD](https://github.com/youngsikhwang/Flatness-aware-SGLD)

### 4.1 Experimental Setup

##### Datasets.

We evaluate our method on three challenging noisy label datasets including CIFAR-10N and CIFAR-100N (Wei et al., 2022), and WebVision (Li et al., 2017). CIFAR-10N and CIFAR-100N include real-world annotation errors introduced by human annotators, offering realistic yet standardized benchmarks for noisy label learning. For CIFAR-10N, we use the aggregate noise setting. WebVision is a large-scale, in-the-wild benchmark, consisting of more than 2.4 million images with labels automatically collected from Google and Flickr based on the 1,000 ImageNet ILSVRC2012 categories. Following standard protocol Li et al. (2020); Ortego et al. (2021); Li et al. (2022), we use the first 50 classes from its Google image subset and report Top-1 (WV-1) and Top-5 (WV-5) accuracy on the official validation set.

##### Models.

We use ResNet-34 and ResNet-50 for training from scratch. For fine-tuning experiments, we use the pre-trained ViT-B/16 (Dosovitskiy et al., 2021) architecture, which has been trained on the ImageNet-1K (Deng et al., 2009) dataset as the backbone on CIFAR-10N and CIFAR-100N.

##### Baselines and Implementation Details.

We compare fSGLD against four baselines: SGD with momentum, AdamW (Loshchilov & Hutter, 2019), SGLD (Welling & Teh, 2011), and SAM (Foret et al., 2021). To ensure a fair comparison, all optimizer hyperparameters are tuned using Optuna (Akiba et al., 2019) with 20 trials of Bayesian optimization. For each optimizer, the search spaces were carefully chosen to include previously reported optimal hyperparameters from the literature, ensuring that all baselines are strongly tuned. For fSGLD, we search for the optimal noise scale $\sigma$, while the inverse temperature $\beta$ is determined by our theoretically-prescribed coupling, $\beta=\sigma^{-4/(1+\eta)}$ with $\eta=0.01$. For experiments with training from scratch, all experiments are trained for 150 epochs with a batch size of 128. The learning rate decays by a factor of 0.1 in the 50th and 100th epochs. For fine tuning, models are trained for 75 epochs with a batch size of 128, decaying the rate by a factor of 0.1 at the 50th epoch. The detailed hyperparameter search spaces for each optimizer and experimental settings are provided in Appendix [D.1](https://arxiv.org/html/2510.02174v1#A4.SS1).

### 4.2 Empirical performance on real-world noisy label datasets

**Table 1: Performance comparison on ResNet-34 and ResNet-50. Results are reported as mean$\pm$std over five different random seeds. Within each model block, the best result is bold and the second-best is underlined. WV-1/WV-5 denote Top-1/Top-5 accuracy on WebVision. The wall-clock time per iteration (s/iter) measured on CIFAR-10N for each model architecture.**
| Model | Optimizer | CIFAR-10N | CIFAR-100N | WV-1 | WV-5 | (s/iter) |
| --- | --- | --- | --- | --- | --- | --- |
| ResNet-34 | SGD | $89.31_{\pm 0.84}$ | $58.47_{\pm 0.20}$ | $71.87_{\pm 0.44}$ | $89.33_{\pm 0.30}$ | 22.0 |
| AdamW | $89.25_{\pm 0.66}$ | $56.77_{\pm 0.47}$ | $68.69_{\pm 0.32}$ | $87.01_{\pm 0.24}$ | 22.5 |  |
| SAM | $91.53_{\pm 0.22}$ | $59.18_{\pm 0.33}$ | 73.49±0.36 | 90.32±0.31 | 41.3 |  |
| SGLD | $88.77_{\pm 0.51}$ | $57.33_{\pm 0.36}$ | $70.87_{\pm 0.67}$ | $88.06_{\pm 0.30}$ | 22.2 |  |
| fSGLD ($\beta$-$\sigma$ coupled) | 91.72±0.20 | 62.02±0.29 | 73.55±0.27 | $89.86_{\pm 0.12}$ | 23.7 |  |
|  | fSGLD ($\beta$ fixed) | 91.56±0.19 | 61.55±0.45 | $73.23_{\pm 0.34}$ | 90.63±0.38 | 23.7 |
| ResNet-50 | SGD | $89.41_{\pm 0.26}$ | $57.52_{\pm 0.17}$ | $71.11_{\pm 0.59}$ | $88.31_{\pm 0.40}$ | 31.9 |
| AdamW | $89.26_{\pm 0.31}$ | $57.28_{\pm 0.90}$ | $69.92_{\pm 0.67}$ | $87.97_{\pm 0.34}$ | 32.3 |  |
| SAM | 90.88±0.49 | $59.01_{\pm 0.60}$ | $72.52_{\pm 0.46}$ | $89.53_{\pm 0.44}$ | 60.7 |  |
| SGLD | $88.89_{\pm 0.40}$ | $56.90_{\pm 0.65}$ | $69.43_{\pm 0.40}$ | $87.17_{\pm 0.22}$ | 32.1 |  |
| fSGLD ($\beta$-$\sigma$ coupled) | 91.26±0.08 | 62.08±0.45 | 73.31±0.50 | 90.07±0.20 | 34.1 |  |
|  | fSGLD ($\beta$ fixed) | $90.72_{\pm 0.29}$ | 61.56±1.08 | 72.87±0.64 | 89.59±0.41 | 34.1 |

##### Training from scratch.

We first evaluate the performance of all optimizers when training ResNet models from scratch. Table [1](https://arxiv.org/html/2510.02174v1#S4.T1) presents the results across all dataset-architecture combinations. Our proposed method, fSGLD ($\beta$-$\sigma$ coupled), consistently achieves the best or second-best performance on every benchmark. Notably, on the CIFAR-100N dataset which presents significant challenges due to its higher noise ratio and larger number of classes, fSGLD significantly outperforms all baselines.

In terms of computational cost, the wall-clock time per iteration (s/iter) shows that fSGLD has a training speed comparable to standard optimizers like SGD, AdamW, and SGLD. In contrast, SAM incurs nearly double the computational overhead due to its min-max formulation requiring two gradient evaluations per step. This highlights a key advantage of our method: fSGLD matches or surpasses SAM’s strong performance with a computational budget similar to standard SGD.

##### Fine-tuning.

We also evaluate performance in the fine-tuning setting, using a pre-trained ViT-B/16 model on CIFAR-10N and CIFAR-100N. The results are presented in Table [2](https://arxiv.org/html/2510.02174v1#S4.T2). Our method, fSGLD ($\beta$-$\sigma$ coupled), consistently outperforms standard optimizers like SGD and SGLD, and achieves performance competitive with or superior to SAM at roughly half the computational overhead.

**Table 2: Fine-tuning performance comparison on ViT-B/16.**
| Model | ViT-B/16 |  |  |
| --- | --- | --- | --- |
| Dataset | CIFAR-10N | CIFAR-100N | (s/epoch) |
| SGD | 94.64 | 71.80 | 343.2 |
| AdamW | 95.57 | 72.30 | 344.5 |
| SAM | 96.75 | 74.66 | 656.7 |
| SGLD | 94.13 | 71.36 | 344.8 |
| fSGLD ($\beta$ fixed) | 96.70 | 75.16 | 345.8 |
| fSGLD ($\beta$-$\sigma$ coupled) | 96.72 | 75.18 | 345.8 |

### 4.3 Ablation Study: The effect of the β \beta - σ \sigma coupling

To empirically validate our theoretical claim, we examine the effect of the theoretically-prescribed $\beta$-$\sigma$ coupling. We compare fSGLD ($\beta$-$\sigma$ coupled) against fSGLD ($\beta$ fixed) which reflects a common heuristic of setting a large, fixed $\beta$ for optimization. The results, summarized in Table [1](https://arxiv.org/html/2510.02174v1#S4.T1) and Table [2](https://arxiv.org/html/2510.02174v1#S4.T2), show that the coupled version consistently outperforms the fixed version in all settings, with the single exception of the WV-5 metric on ResNet-34. This provides strong empirical evidence that our theoretically-prescribed coupling is crucial for improving performance.

### 4.4 Sensitivity analysis

Figure: Table 3: Performance with respect to the number of random perturbations $n$ used in fSGLD.
Refer to caption: sensitive.png

While our fSGLD algorithm uses a single perturbation per iteration $(n=1)$, we examine how performance is affected by using multiple perturbations, which can provide a more accurate estimation of the Hessian trace. As shown in Table [3](https://arxiv.org/html/2510.02174v1#S4.T3), increasing $n$ can improve accuracy, but this comes at a nearly linear increase in computational cost. Remarkably, fSGLD already achieves strong performance with just a single perturbation, making $n=1$ a practical and efficient choice.

Next, we analyze the effect of the perturbation scale $\sigma$, as illustrated in Figure [2](https://arxiv.org/html/2510.02174v1#S4.F2). The performance on CIFAR-10N remains stable and robust across a wide range of small to moderate values of $\sigma$. However, performance degrades sharply when $\sigma$ becomes excessively large, as the strong perturbations begin to destabilize the training process.

### 4.5 Hessian spectrum

Figure: (a) SGD
Refer to caption: sgd_distribution.png

To empirically verify our theoretical insight that fSGLD finds flat minima by implicitly regularizing the Hessian trace, we analyze the curvature of the loss landscape at the solutions found by SGD, SAM, fSGLD. Note that we use the best hyperparameter configuration for each optimizer.

We compute two standard measures of sharpness for a ResNet-34 trained on CIFAR-10N: the maximum eigenvalue ($\lambda_{\max}$) of the Hessian and its trace ($\text{tr}(H(\theta))$). Since exact computation is intractable, we estimate the top 50 eigenvalues using the Lanczos algorithm (Lin et al., 2016; Ghorbani et al., 2019) and approximate the trace with Hutchinson’s method (Avron & Toledo, 2011; Ubaru et al., 2017). Detailed settings are described in Appendix [D.2](https://arxiv.org/html/2510.02174v1#A4.SS2).

The results, presented in Figure [3](https://arxiv.org/html/2510.02174v1#S4.F3), confirm our hypothesis. fSGLD converges to solutions with a significantly smaller maximum eigenvalue and Hessian trace compared to standard SGD. Remarkably, the degree of flatness achieved by fSGLD is comparable to SAM in terms of $\lambda_{\max}$ and even lower in terms of $\text{tr}(H)$. This result is achieved at roughly half the computational cost of SAM. These results empirically validate our theoretical analysis, confirming that the proposed algorithm effectively promotes convergence to flatter minima.

## 5 Related Work and Discussions

We review the most relevant literature on SAM, RWP, Hessian-based optimization, and SGLD.

##### Flat Minima and Generalization.

Empirical studies (Keskar et al., 2017; Jastrzkebski et al., 2017; Jiang et al., 2020) and theoretical analyses (Dziugaite & Roy, 2017; Neyshabur et al., 2017) consistently show that flatter minima are strongly correlated with better generalization in deep neural networks. However, elucidating precise notions of sharpness and their relationship to generalization remains an open and active area of research (Andriushchenko & Flammarion, 2022; Ding et al., 2024; Wen et al., 2023; Tahmasebi et al., 2024).

##### SAM and RWP.

The success of SAM (Foret et al., 2021) has produced a wide range of follow-up work to improve its efficiency, effectiveness, and applicability.
Extensions include algorithmic improvements to approximate the inner maximization more efficiently (Liu et al., 2022a; Du et al., 2022a; Kwon et al., 2021; Xie et al., 2024; Li et al., 2024b; Chen et al., 2024; Kang et al., 2025). Beyond these, several Hessian-based regularization approaches have explored flatness from a different angle. For example, Zhang et al. (2024) propose Noise-Stability Optimization, and Li et al. (2024a) studies random weight perturbation with explicit Hessian penalties. Both works focus on PAC-Bayes generalization bounds and local convergence to stationary points,
providing algorithm-agnostic guarantees about the perturbed loss rather than the training dynamics of a specific optimizer.
By contrast, we show that the invariant measure of fSGLD yields global, non-asymptotic convergence guarantees and an explicit link between random weight perturbation and Hessian-trace regularization.
Lastly, the concept of using noise for regularization was formalized through the framework of randomized smoothing (Duchi et al., 2012), and our work makes this connection explicit for Langevin dynamics,
differing fundamentally from explicit Hessian-penalty methods that rely on costly approximations (Sankar et al., 2021).

##### SGLD and its Convergence Rate.

Following the seminal works of Welling & Teh (2011); Raginsky et al. (2017), numerous variants of SGLD have been developed to improve its practical performance, such as variance reduction techniques (Kinoshita & Suzuki, 2022; Dubey et al., 2016; Huang & Becker, 2021), preconditioned SGLD (Li et al., 2016), replica exchange SGLD (Dong & Tong, 2021; Deng et al., 2020a). A parallel line of research has focused on its theoretical properties, particularly its non-asymptotic convergence rate. Early results (Raginsky et al., 2017; Xu et al., 2018) showed convergence in the Wasserstein-2 distance at a rate dependent on the number of iterations. More recently, the state-of-the art analyses have established convergence rates of $O(\lambda^{1/2})$ in Wasserstein-1 and $O(\lambda^{1/4})$ in Wasserstein-2 distance (Zhang et al., 2023). Our convergence rates are consistent with these best-known results. However, a crucial distinction is that prior work proves convergence to the minimizers of the original objective $u$, whereas our guarantees are for convergence to global flat minima.

## 6 Conclusion and Limitations

In this work, we introduced Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD), a novel algorithm that synthesizes randomized smoothing with Langevin dynamics to efficiently target flat minima. By evaluating the gradient at parameters perturbed by Gaussian noise, a technique known as Random Weight Perturbation (RWP), fSGLD optimizes a surrogate objective that provably incorporates Hessian trace information without explicit computation.

Our main theoretical contribution is a rigorous non-asymptotic analysis of this process. We establish convergence guarantees in Wasserstein distance and provide the explicit excess risk bound for this class of flatness-aware optimizers. Crucially, our theory shows that the desired regularization effect emerges from a precise coupling of the noise scale $\sigma$ and the inverse temperature $\beta$.

Empirically, fSGLD demonstrates superior or competitive performance against strong baselines, including SAM, on challenging noisy-label and fine-tuning benchmarks. These gains are achieved at a computational cost comparable to standard SGD, roughly half that of SAM. Our analysis of the Hessian spectrum further confirms that fSGLD converges to significantly flatter minima, providing a direct validation of its mechanism. Ultimately, our work provides one of the provable links between an efficient algorithmic design (RWP within SGLD) and quantifiable generalization benefits, bridging the gap between heuristic flatness-seeking methods and rigorous convergence theory.

### 6.1 Limitations and Future Directions.

Applying fSGLD to diffusion-based generative models is a particularly promising direction; investigating whether its bias towards flatter regions of the loss landscape can lead to more diverse or higher-quality samples is a compelling open question.
On the theoretical side, we leave for future work the extension of our analysis to the case where $u$ is semiconvex (i.e., its gradient is one-sided Lipschitz), rather than satisfying Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2).

##### Acknowledgments.

This work has received funding from the Ministry of Trade, Industry and Energy (MOTIE) and Korea Institute for Advancement of Technology (KIAT) through the International Cooperative R&D program (No.P0025828). This work has been partially supported by project MIS 5154714 of the National Recovery and Resilience Plan Greece 2.0 funded by the
European Union under the NextGenerationEU Program.

## References

- Ahn et al. (2024)
Kwangjun Ahn, Ali Jadbabaie, and Suvrit Sra.
How to escape sharp minima with random perturbations.
In *Proceedings of the 41st International Conference on Machine
Learning*, 2024.
- Akiba et al. (2019)
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.
Optuna: A next-generation hyperparameter optimization framework.
In *Proceedings of the 25th ACM SIGKDD international conference
on knowledge discovery & data mining*, pp. 2623–2631, 2019.
- Andriushchenko & Flammarion (2022)
Maksym Andriushchenko and Nicolas Flammarion.
Towards understanding sharpness-aware minimization.
In *International conference on machine learning*, pp. 639–668. PMLR, 2022.
- Avron & Toledo (2011)
Haim Avron and Sivan Toledo.
Randomized algorithms for estimating the trace of an implicit
symmetric positive semi-definite matrix.
*Journal of the ACM (JACM)*, 58(2):1–34,
2011.
- Bartlett et al. (2023)
Peter L. Bartlett, Philip M. Long, and Olivier Bousquet.
The dynamics of sharpness-aware minimization: bouncing across ravines
and drifting towards wide minima.
*J. Mach. Learn. Res.*, 24(1), 2023.
ISSN 1532-4435.
- Bisla et al. (2022)
Devansh Bisla, Jing Wang, and Anna Choromanska.
Low-pass filtering sgd for recovering flat optima in the deep
learning optimization landscape.
In *International Conference on Artificial Intelligence and
Statistics*, pp. 8299–8339. PMLR, 2022.
- Bolley & Villani (2005)
François Bolley and Cédric Villani.
Weighted csiszár-kullback-pinsker inequalities and applications
to transportation inequalities.
In *Annales de la Faculté des sciences de Toulouse:
Mathématiques*, volume 14, pp. 331–352, 2005.
- Chau et al. (2019)
Huy N Chau, Chaman Kumar, Miklós Rásonyi, and Sotirios Sabanis.
On fixed gain recursive estimators with discontinuity in the
parameters.
*ESAIM: Probability and Statistics*, 23:217–244,
2019.
- Chau et al. (2021)
Ngoc Huy Chau, Éric Moulines, Miklos Rásonyi, Sotirios Sabanis, and
Ying Zhang.
On stochastic gradient langevin dynamics with dependent data streams:
The fully nonconvex case.
*SIAM Journal on Mathematics of Data Science*, 3(3):959–986, 2021.
- Chaudhari et al. (2017)
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi,
Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina.
Entropy-SGD: Biasing gradient descent into wide valleys.
In *International Conference on Learning Representations*, 2017.
- Chen et al. (2024)
Simiao Chen, Xiaoge Deng, Dongpo Xu, Tao Sun, and Dongsheng Li.
Decentralized stochastic sharpness-aware minimization algorithm.
*Neural Networks*, 176:106325, 2024.
ISSN 0893-6080.
- Deng et al. (2009)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.
Imagenet: A large-scale hierarchical image database.
In *2009 IEEE conference on computer vision and pattern
recognition*, pp. 248–255. Ieee, 2009.
- Deng et al. (2020a)
Wei Deng, Qi Feng, Liyao Gao, Faming Liang, and Guang Lin.
Non-convex learning via replica exchange stochastic gradient mcmc.
In *International Conference on Machine Learning*, pp. 2474–2483. PMLR, 2020a.
- Deng et al. (2020b)
Wei Deng, Guang Lin, and Faming Liang.
A contour stochastic gradient Langevin dynamics algorithm for
simulations of multi-modal distributions.
*Advances in neural information processing systems*,
33:15725–15736, 2020b.
- Deng et al. (2022)
Wei Deng, Siqi Liang, Botao Hao, Guang Lin, and Faming Liang.
Interacting contour stochastic gradient Langevin dynamics.
In *International Conference on Learning Representations*, 2022.
- Ding et al. (2024)
Lijun Ding, Dmitriy Drusvyatskiy, Maryam Fazel, and Zaid Harchaoui.
Flat minima generalize for low-rank matrix recovery.
*Information and Inference: A Journal of the IMA*, 13(2), 2024.
- Dong & Tong (2021)
Jing Dong and Xin T Tong.
Replica exchange for non-convex optimization.
*Journal of Machine Learning Research*, 22(173):1–59, 2021.
- Dosovitskiy et al. (2021)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn,
Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg
Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at
scale.
In *International Conference on Learning Representations*, 2021.
- Du et al. (2022a)
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick
Siow Mong Goh, and Vincent Tan.
Efficient sharpness-aware minimization for improved training of
neural networks.
In *International Conference on Learning Representations*,
2022a.
- Du et al. (2022b)
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Tan, and Joey Tianyi Zhou.
Sharpness-aware training for free.
In *Advances in Neural Information Processing Systems*,
volume 35, pp. 23439–23451, 2022b.
- Dubey et al. (2016)
Kumar Avinava Dubey, Sashank J. Reddi, Sinead A Williamson, Barnabas Poczos,
Alexander J Smola, and Eric P Xing.
Variance reduction in stochastic gradient langevin dynamics.
In *Advances in Neural Information Processing Systems*,
volume 29, 2016.
- Duchi et al. (2012)
John C Duchi, Peter L Bartlett, and Martin J Wainwright.
Randomized smoothing for stochastic optimization.
*SIAM Journal on Optimization*, 22(2):674–701, 2012.
- Dziugaite & Roy (2017)
Gintare Karolina Dziugaite and Daniel M. Roy.
Computing nonvacuous generalization bounds for deep (stochastic)
neural networks with many more parameters than training data.
In *Proceedings of the Thirty-Third Conference on Uncertainty in
Artificial Intelligence (UAI)*, pp. 884–893, 2017.
- Foret et al. (2021)
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur.
Sharpness-aware minimization for efficiently improving
generalization.
In *International Conference on Learning Representations*, 2021.
- Futami & Fujisawa (2023)
Futoshi Futami and Masahiro Fujisawa.
Time-independent information-theoretic generalization bounds for
sgld.
*Advances in Neural Information Processing Systems*,
36:8173–8185, 2023.
- Gatmiry et al. (2024)
Khashayar Gatmiry, Zhiyuan Li, Sashank J. Reddi, and Stefanie Jegelka.
Simplicity bias via global convergence of sharpness minimization.
In *Forty-first International Conference on Machine Learning*,
2024.
- Ghorbani et al. (2019)
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.
An investigation into neural net optimization via hessian eigenvalue
density.
In *International Conference on Machine Learning*, pp. 2232–2241. PMLR, 2019.
- Hochreiter & Schmidhuber (1997)
Sepp Hochreiter and Jürgen Schmidhuber.
Flat minima.
*Neural computation*, 9(1):1–42, 1997.
- Huang & Becker (2021)
Zhishen Huang and Stephen Becker.
Stochastic gradient langevin dynamics with variance reduction.
In *2021 International Joint Conference on Neural Networks
(IJCNN)*, pp. 1–8. IEEE, 2021.
- Jastrzkebski et al. (2017)
Stanisław Jastrzkebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja
Fischer, Yoshua Bengio, and Amos Storkey.
Three factors influencing minima in sgd.
*arXiv preprint arXiv:1711.04623*, 2017.
- Jiang et al. (2020)
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy
Bengio.
Fantastic generalization measures and where to find them.
In *International Conference on Learning Representations*, 2020.
- Kang et al. (2025)
Helei Kang, Yiming Jiang, Jinlan Liu, and Dongpo Xu.
Sharpness-aware minimization method with momentum acceleration for
deep neural networks.
*Knowledge-Based Systems*, 326:113967, 2025.
ISSN 0950-7051.
- Keskar et al. (2017)
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy,
and Ping Tak Peter Tang.
On large-batch training for deep learning: Generalization gap and
sharp minima.
In *International Conference on Learning Representations*, 2017.
- Khanh et al. (2024)
Pham Duy Khanh, Hoang-Chau Luong, Boris S. Mordukhovich, and Dat Ba Tran.
Fundamental convergence analysis of sharpness-aware minimization.
In *Advances in Neural Information Processing Systems*,
volume 37, pp. 13149–13182, 2024.
- Kinoshita & Suzuki (2022)
Yuri Kinoshita and Taiji Suzuki.
Improved convergence rate of stochastic gradient langevin dynamics
with variance reduction and its application to optimization.
In *Advances in Neural Information Processing Systems*, 2022.
- Kodge (2024)
Sangamesh Kodge.
MiniWebVision, February 2024.
URL [https://github.com/sangamesh-kodge/Mini-WebVision](https://github.com/sangamesh-kodge/Mini-WebVision).
- Kwon et al. (2021)
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi.
Asam: Adaptive sharpness-aware minimization for scale-invariant
learning of deep neural networks.
In *International conference on machine learning*, pp. 5905–5914, 2021.
- Li et al. (2016)
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin.
Preconditioned stochastic gradient langevin dynamics for deep neural
networks.
In *Proceedings of the AAAI conference on artificial
intelligence*, volume 30, 2016.
- Li et al. (2020)
Junnan Li, Richard Socher, and Steven C.H. Hoi.
DivideMix: Learning with noisy labels as semi-supervised learning.
In *International Conference on Learning Representations*, 2020.
- Li et al. (2022)
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu.
Selective-supervised contrastive learning with noisy labels.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)*, pp. 316–325, June 2022.
- Li et al. (2024a)
Tao Li, Qinghua Tao, Weihao Yan, Yingwen Wu, Zehao Lei, Kun Fang, Mingzhen He,
and Xiaolin Huang.
Revisiting random weight perturbation for efficiently improving
generalization.
*Transactions on Machine Learning Research*, 2024a.
ISSN 2835-8856.
- Li et al. (2024b)
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang.
Friendly sharpness-aware minimization.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pp. 5631–5640, 2024b.
- Li et al. (2025)
Tian Li, Tianyi Zhou, and Jeff Bilmes.
Tilted sharpness-aware minimization.
In *Forty-second International Conference on Machine Learning*,
2025.
- Li et al. (2017)
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, Jesse Berent, Abhinav Gupta,
Rahul Sukthankar, and Luc Van Gool.
Webvision challenge: Visual learning and understanding with web data.
In *Proceedings of the IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) Workshops*, pp. 2243–2251, 2017.
- Lin et al. (2016)
Lin Lin, Yousef Saad, and Chao Yang.
Approximating spectral densities of large matrices.
*SIAM review*, 58(1):34–65, 2016.
- Liu et al. (2022a)
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You.
Towards efficient and scalable sharpness-aware minimization.
In *Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition (CVPR)*, pp. 12360–12370, 2022a.
- Liu et al. (2022b)
Yong Liu, Siqi Mai, Minhao Cheng, Xiangning Chen, Cho-Jui Hsieh, and Yang You.
Random sharpness-aware minimization.
In *Advances in Neural Information Processing Systems*,
volume 35, pp. 24543–24556, 2022b.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
In *International Conference on Learning Representations*, 2019.
- Luo et al. (2024)
Haocheng Luo, Tuan Truong, Tung Pham, Mehrtash Harandi, Dinh Phung, and Trung
Le.
Explicit eigenvalue regularization improves sharpness-aware
minimization.
*Advances in Neural Information Processing Systems*,
37:4424–4453, 2024.
- Majka et al. (2020)
Mateusz B. Majka, Aleksandar Mijatović, and Łukasz Szpruch.
Nonasymptotic bounds for sampling algorithms without log-concavity.
*Annals of Applied Probability*, 30(4):1534–1581, 2020.
- Neyshabur et al. (2017)
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro.
Exploring generalization in deep learning.
*Advances in neural information processing systems*, 30, 2017.
- Oikonomou & Loizou (2025)
Dimitris Oikonomou and Nicolas Loizou.
Sharpness-aware minimization: General analysis and improved rates.
In *The Thirteenth International Conference on Learning
Representations*, 2025.
- Ortego et al. (2021)
Diego Ortego, Eric Arazo, Paul Albert, Noel E O’Connor, and Kevin McGuinness.
Multi-objective interpolation training for robustness to label noise.
In *Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition*, pp. 6606–6615, 2021.
- Raginsky et al. (2017)
Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky.
Non-convex learning via stochastic gradient langevin dynamics: a
nonasymptotic analysis.
In *Proceedings of the 2017 Conference on Learning Theory*,
volume 65, pp. 1674–1703. PMLR, 2017.
- Sankar et al. (2021)
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N
Balasubramanian.
A deeper look at the hessian eigenspectrum of deep neural networks
and its applications to regularization.
In *Proceedings of the AAAI Conference on Artificial
Intelligence*, volume 35, pp. 9481–9488, 2021.
- Si & Yun (2023)
Dongkuk Si and Chulhee Yun.
Practical sharpness-aware minimization cannot converge all the way to
optima.
In *Advances in Neural Information Processing Systems*,
volume 36, pp. 26190–26228, 2023.
- Tahmasebi et al. (2024)
Behrooz Tahmasebi, Ashkan Soleymani, Dara Bahri, Stefanie Jegelka, and Patrick
Jaillet.
A universal class of sharpness-aware minimization algorithms.
In *Proceedings of the 41st International Conference on Machine
Learning*, volume 235, 2024.
- Ubaru et al. (2017)
Shashanka Ubaru, Jie Chen, and Yousef Saad.
Fast estimation of tr(f(a)) via stochastic lanczos quadrature.
*SIAM Journal on Matrix Analysis and Applications*, 38(4):1075–1099, 2017.
- Wei et al. (2022)
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu.
Learning with noisy labels revisited: A study using real-world human
annotations.
In *International Conference on Learning Representations*, 2022.
- Wei et al. (2025)
Zheng Wei, Xingjun Zhang, and Zhendong Tan.
Unifying and revisiting sharpness-aware minimization with
noise-injected micro-batch scheduler for efficiency improvement.
*Neural Networks*, 185:107205, 2025.
ISSN 0893-6080.
- Welling & Teh (2011)
Max Welling and Yee W Teh.
Bayesian learning via stochastic gradient langevin dynamics.
In *Proceedings of the 28th international conference on machine
learning*, pp. 681–688, 2011.
- Wen et al. (2023)
Kaiyue Wen, Zhiyuan Li, and Tengyu Ma.
Sharpness minimization algorithms do not only minimize sharpness to
achieve better generalization.
In *Thirty-seventh Conference on Neural Information Processing
Systems*, 2023.
- Xie et al. (2024)
Wanyun Xie, Fabian Latorre, Kimon Antonakopoulos, Thomas Pethick, and Volkan
Cevher.
Improving SAM requires rethinking its optimization formulation.
In *Forty-first International Conference on Machine Learning*,
2024.
- Xu et al. (2018)
Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu.
Global convergence of langevin dynamics based algorithms for
nonconvex optimization.
In *Advances in Neural Information Processing Systems*,
volume 31, 2018.
- Yu et al. (2024)
Runsheng Yu, Youzhi Zhang, and James Kwok.
Improving sharpness-aware minimization by lookahead.
In *Proceedings of the 41st International Conference on Machine
Learning*, volume 235, 2024.
- Zhang et al. (2024)
Hongyang R. Zhang, Dongyue Li, and Haotian Ju.
Noise stability optimization for finding flat minima: A hessian-based
regularization approach.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
- Zhang et al. (2023)
Ying Zhang, Ömer Deniz Akyildiz, Theodoros Damoulas, and Sotirios Sabanis.
Nonasymptotic estimates for stochastic gradient langevin dynamics
under local conditions in nonconvex optimization.
*Applied Mathematics & Optimization*, 87(2):25, 2023.

## Appendix A Relationship between π β , σ ⋆ \pi_{\beta,\sigma}^{\star} and π β fSGLD \pi_{\beta}^{\text{fSGLD}}

We derive the relationship between the target measure $\pi_{\beta,\sigma}^{\star}$ and the invariant measure $\pi_{\beta}^{\text{fSGLD}}$ of the fSGLD algorithm, which will be used to prove Proposition [3.1](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem1), Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2), and Corollary [3.3](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem3). By Taylor’s theorem, we obtain

$$ $\begin{split}u(\theta+\epsilon)=u(\theta)+\nabla u(\theta)^{T}\epsilon+\frac{1}{2}\epsilon^{T}H(\theta)\epsilon+\mathscr{R}(\theta,\epsilon),\end{split}$ (14) $$

where $\mathscr{R}(\epsilon)$ denotes the remainder term. Taking the expectation over $\epsilon\sim\mathscr{N}(0,\sigma^{2}I_{d})$ in [14](https://arxiv.org/html/2510.02174v1#A1.E14), we have

$$ $\begin{split}g_{\epsilon}(\theta)&=u(\theta)+\frac{1}{2}\mathbb{E}[\epsilon^{T}H_{u}(\theta)\epsilon]+\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\\ &=u(\theta)+\frac{1}{2}\text{tr}\left(H_{u}(\theta)\cdot\mathbb{E}[\epsilon^{T}\epsilon]\right)+\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\\ &=v(\theta)+\mathbb{E}[\mathscr{R}(\theta,\epsilon)],\end{split}$ (15) $$

where

$$ $v(\theta)=u(\theta)+\frac{\sigma^{2}}{2}\text{tr}\left(H_{u}(\theta)\right),$ $$

and

$$ $\begin{split}\mathbb{E}[\mathscr{R}(\theta,\epsilon)]&=\frac{1}{6}\sum_{i,j,k=1}^{d}\frac{\partial^{3}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}}(\theta)\ \mathbb{E}[\epsilon_{i}\epsilon_{j}\epsilon_{k}]+\frac{1}{24}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)\ \mathbb{E}[\epsilon_{i}\epsilon_{j}\epsilon_{k}\epsilon_{l}]\\ &=\frac{1}{24}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)\ \sigma^{4}(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk}),\end{split}$ (16) $$

where $\delta_{ij}$ denotes the Kronecker delta. Let the normalization constant of $\pi_{\beta}^{\text{fSGLD}}$ be given by

$$ $Z_{\beta}:=\int_{\mathbb{R}^{d}}e^{-\beta g_{\epsilon}(\theta)}\ \mathrm{d}\theta,$ (17) $$

and let the normalization constant of $\pi_{\beta,\sigma}^{\star}$ be given by

$$ $Z_{\beta,\sigma}:=\int_{\mathbb{R}^{d}}e^{-\beta v(\theta)}\ \mathrm{d}\theta.$ (18) $$

Using [15](https://arxiv.org/html/2510.02174v1#A1.E15), [17](https://arxiv.org/html/2510.02174v1#A1.E17), and [18](https://arxiv.org/html/2510.02174v1#A1.E18), we obtain

$$ $\begin{split}\pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta)&=Z^{-1}_{\beta}\exp(-\beta g_{\epsilon}(\theta))\ \mathrm{d}\theta\\ &=Z^{-1}_{\beta}Z_{\beta,\sigma}\exp(-\beta\ \mathbb{E}[\mathscr{R}(\theta,\epsilon)])\ \pi_{\beta,\sigma}^{\star}(\mathrm{d}\theta).\end{split}$ (19) $$

## Appendix B Additional results for Section 3.1

This section collects several technical remarks and direct consequences of the assumptions presented in Section [3.1](https://arxiv.org/html/2510.02174v1#S3.SS1).

###### Remark B.1 .

By Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1) and [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), the gradient $h(\theta)=\mathbb{E}[\nabla_{\theta}U(\theta,X)]$ for all $\theta\in\mathbb{R}^{d}$, is well-defined. In addition, one obtains for all $\theta$, $\theta^{{}^{\prime}}\in\mathbb{R}^{d}$,

$$ $|h(\theta)-h(\theta^{{}^{\prime}})|\leq L_{1}\mathbb{E}[\varphi(X_{0})]|\theta-\theta^{{}^{\prime}}|.$ $$

As a consequence of Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), one obtains, for fixed $\widetilde{\epsilon}\in\mathbb{R}^{d}$,

$$ $\begin{split}|\nabla_{\theta}U(\theta+\widetilde{\epsilon},x)-\nabla_{\theta^{{}^{\prime}}}U(\theta^{{}^{\prime}}+\widetilde{\epsilon},x)|&\leq L_{1}\varphi(x)|\theta-\theta^{{}^{\prime}}|,\\ |\nabla_{\theta}U(\theta+\widetilde{\epsilon},x)-\nabla_{\theta}U(\theta+\widetilde{\epsilon},x^{{}^{\prime}})|&\leq L_{2}(\varphi(x)+\varphi(x^{{}^{\prime}}))(1+|\theta+\widetilde{\epsilon}|)|x-x^{{}^{\prime}}|.\end{split}$ $$

Also, Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) implies

$$ $|\nabla_{\theta}U(\theta+\widetilde{\epsilon},x)|\leq L_{1}\varphi(x)|\theta|+L_{2}\bar{\varphi}(x)(1+|\widetilde{\epsilon}|)+\widetilde{G}(\widetilde{\epsilon}),$ $$

where $\bar{\varphi}(x):=(\varphi(x)+\varphi(0))|x|$, and $\widetilde{G}(\widetilde{\epsilon}):=|\nabla_{\theta^{{}^{\prime}}}U(\widetilde{\epsilon},0)|$.

###### Remark B.2 .

By Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), one obtains a dissipativity condition of $h$, i.e., for any $\theta\in\mathbb{R}^{d}$, $\langle\nabla h(\theta),\theta\rangle\geq\bar{a}|\theta|^{2}-\bar{b}$. Let $\zeta\in(0,\bar{a}L_{1}^{-2}(\mathbb{E}[\varphi^{2}(X_{0})])^{-1})$. As a consequence of Assumptions [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), one obtains, for any $\theta\in\mathbb{R}^{d}$

$$ $\begin{split}\langle\nabla g_{\epsilon}(\theta),\theta\rangle\geq a|\theta|^{2}-b,\end{split}$ (20) $$

where

$$ $\begin{split}a&:=\bar{a}-\zeta L^{2}_{1}\mathbb{E}[\varphi^{2}(X_{0})]>0,\\ b&:=(2\zeta)^{-1}\sigma^{2}d+4\zeta L^{2}_{2}\mathbb{E}[\bar{\varphi}^{2}(X_{0})](1+\sigma^{2}d)+2\zeta\mathbb{E}[\widetilde{G}^{2}(\epsilon)]+\bar{b}>0,\end{split}$ (21) $$

and $\widetilde{G}$ and $\bar{\varphi}$ are given in Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1).

###### Proof of Remark B.2 .

Using Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) and Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1), and Young’s inequality, one obtains, for fixed $\widetilde{\epsilon}\in\mathbb{R}^{d}$

$$ $\begin{split}\langle\nabla_{\theta}U(\theta+\widetilde{\epsilon},x),\theta\rangle&=\langle\nabla_{\theta}U(\theta+\widetilde{\epsilon},x),\theta+\widetilde{\epsilon}\rangle-\langle\nabla_{\theta}U(\theta+\widetilde{\epsilon},x),\widetilde{\epsilon}\rangle\\ &\geq\langle\theta+\widetilde{\epsilon},A(x)\theta+\widetilde{\epsilon}\rangle-\hat{b}(x)-\zeta 2^{-1}|\nabla_{\theta}U(\theta+\widetilde{\epsilon},x)|^{2}-(2\zeta)^{-1}|\widetilde{\epsilon}|^{2}\\ &\geq\langle\theta,(A(x)-\zeta L^{2}_{1}\varphi^{2}(x))\theta\rangle+\langle\theta,A(x)\widetilde{\epsilon}\rangle+\langle\widetilde{\epsilon},A(x)\theta\rangle+\langle\widetilde{\epsilon},A(x)\widetilde{\epsilon}\rangle\\ &\qquad-4\zeta L^{2}_{2}\bar{\varphi}^{2}(x)(1+|\widetilde{\epsilon}|^{2})-2\zeta\widetilde{G}^{2}(\widetilde{\epsilon})-\hat{b}(x)-(2\zeta)^{-1}|\widetilde{\epsilon}|^{2}.\end{split}$ (22) $$

Therefore,

$$ $\begin{split}\nabla g_{\epsilon}(\theta)&=\mathbb{E}[\mathbb{E}_{X}[\nabla_{\theta}U(\theta+\epsilon,X)]]\\ &\geq(\bar{a}-\zeta L^{2}_{1}\mathbb{E}[\varphi^{2}(X_{0})])|\theta|^{2}+(\bar{a}-(2\zeta)^{-1})\sigma^{2}d-4\zeta L^{2}_{2}\mathbb{E}[\bar{\varphi}^{2}(X_{0})](1+\sigma^{2}d)\\ &\qquad-2\zeta\mathbb{E}[\widetilde{G}^{2}(\epsilon)]-\bar{b}\\ &\geq a|\theta|^{2}-b,\end{split}$ $$

where $a$ and $b$ are defined in [21](https://arxiv.org/html/2510.02174v1#A2.E21).

∎

###### Remark B.3 .

Controlling the remainder term $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]$ could in principle require very strong smoothness assumptions such as globally bounded fourth-order derivatives to ensure uniform control of higher-order terms. These are not standard in SGLD analyses, and our approach does not impose any such extra conditions, Instead, by leveraging only the dissipativity conditon (Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3)) together with local Lipschitz continuity (Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2)), we establish all convergence results without any global $C^{4}$ boundedness or similar strong regularity. This distinction highlights a key theoretical contribution of our work: rigorous non-asymptotic analysis for nonconvex high-dimensional objectives under significantly weaker and more realistic assumptions.

###### Lemma B.4 .

Let Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Then $\pi_{\beta,\sigma}^{\star}$ has finite second moments.

###### Proof of Lemma B.4 .

As a consequence of Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), $\nabla v(\theta)$ is Lipschitz continuous. Let $\bar{\zeta}\in(0,4\bar{a}\sigma^{-2})$. Using Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), and Young’s inequality, one obtains

$$ $\begin{split}\langle\nabla v(\theta),\theta\rangle&=\langle\nabla u(\theta),\theta\rangle+\frac{\sigma^{2}}{2}\langle\nabla\left(\text{tr}\left(H_{u}(\theta)\right)\right),\theta\rangle\\ &\geq\left(\bar{a}-\frac{\bar{\zeta}\sigma^{2}}{4}\right)|\theta|^{2}-\bar{b}-\frac{\sigma^{2}}{4\bar{\zeta}}|\nabla\left(\text{tr}\left(H_{u}(\theta)\right)\right)|^{2},\end{split}$ $$

which implies that $\nabla v(\theta)$ is dissipative.
Therefore, $\pi_{\beta,\sigma}^{\star}$ has finite second moment.
∎

###### Remark B.1 .

###### Remark B.2 .

###### Proof of Remark B.2 .

###### Remark B.3 .

###### Lemma B.4 .

###### Proof of Lemma B.4 .

## Appendix C Overview of the non-asymptotic Wasserstein analysis and error bound for the expected
excess risk

In this section, we derive the results introduced in Sections [3.2](https://arxiv.org/html/2510.02174v1#S3.SS2) and [3.3](https://arxiv.org/html/2510.02174v1#S3.SS3). We begin by presenting the framework behind these two sections.

The ‘data’ process $(X_{k})_{k\in\mathbb{N}}$ in [7](https://arxiv.org/html/2510.02174v1#S2.E7) is adapted to a given filtration $(\mathscr{X}_{k})_{k\in\mathbb{N}}$ representing the flow of past information, and we denote the sigma-algebra of $\cup_{k\in\mathbb{N}}\mathscr{X}_{k}$ by $\mathscr{X}_{\infty}$. In addition, we assume that $\theta_{0}$ , $\mathscr{X}_{\infty}$, $(\epsilon_{k})_{k\in\mathbb{N}}$, and $(\xi_{k})_{k\in\mathbb{N}}$ are all independent among themselves.

We define

$$ $\lambda_{\text{max}}:=\min\left\{\frac{\min\{a,a^{\frac{1}{3}}\}}{16(1+L_{1})^{2}(\mathbb{E}[(1+\varphi(X_{0}))^{4}])^{1/2}},\frac{1}{a}\right\},$ (23) $$

where $L_{1}$, $\varphi$ and $a$ are defined in Assumptions [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and Remark [B.2](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem2), respectively.

### C.1 Auxiliary processes

We start by defining the process $(Z_{t}^{\text{fSGLD}})_{t\in\mathbb{R}_{+}}$ as the solution of the flatness Langevin SDE

$$ $\begin{split}Z_{0}^{\text{fSGLD}}&:=\theta_{0}\in\mathbb{R}^{d},\\ \mathrm{d}Z_{t}^{\text{fSGLD}}&:=-\nabla g_{\epsilon}(Z_{t}^{\text{fSGLD}})\mathrm{d}t+\sqrt{2\beta^{-1}}\ \mathrm{d}B_{t},\end{split}$ (24) $$

where $B_{t}$ is a standard $d$-dimensional Brownian motion. Denote by $(\mathscr{F}_{t})_{t\geq 0}$ the natural filtration of $(B_{t})_{t\geq 0}$ and by $\Sigma_{\theta_{0}}$ the sigma-algebra generated by $\theta_{0}$, and we assume that $(\mathscr{F}_{t})_{t\geq 0}$ is independent of $\mathscr{X}_{\infty}\vee\Sigma_{\theta_{0}}$. Furthermore, denote by $\mathscr{F}_{\infty}$ the sigma-algebra of $\bigcup_{t\geq 0}\mathscr{F}_{t}$.

###### Remark C.1 .

By Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1), SDE [24](https://arxiv.org/html/2510.02174v1#A3.E24) has a unique solution adapted to $(\mathscr{F}_{t})_{t\in\mathbb{R}_{+}}$.

To facilitate the convergence analysis, we introduce another process. For each $\lambda>0$, $Z_{t}^{\lambda,\text{fSGLD}}:=Z^{\text{fSGLD}}_{\lambda t}$, $t\in\mathbb{R}_{+}$, and let $\widetilde{B}^{\lambda}_{t}:=B_{\lambda t}/\sqrt{\lambda}$, $t\geq 0$. We observe that $(\widetilde{B}_{t})_{t\geq 0}$ is a Brownian motion and

$$ $\begin{split}Z_{0}^{\lambda,\text{fSGLD}}&:=\theta_{0}\\ \mathrm{d}Z_{t}^{\lambda,\text{fSGLD}}&=-\lambda\nabla g_{\epsilon}(Z_{t}^{\lambda,\text{fSGLD}})\ \mathrm{d}t+\sqrt{2\lambda\beta^{-1}}\ \mathrm{d}\widetilde{B}^{\lambda}_{t}.\end{split}$ $$

The natural filtration of $(\widetilde{B}_{t})_{t\geq 0}$ is denoted by $(\mathscr{F}^{\lambda}_{t})_{t\geq 0}$ with $\mathscr{F}^{\lambda}_{t}:=\mathscr{F}_{\lambda t}$, $t\in\mathbb{R}_{+}$. For a positive real number $a$, we denote its integer part by $\lfloor a\rfloor$. Then, we define $(\bar{\theta}_{t}^{\text{fSGLD}})_{t\in\mathbb{R}_{+}}$, the continuous-time interpolation of fSGLD [7](https://arxiv.org/html/2510.02174v1#S2.E7), as

$$ $\begin{split}\bar{\theta}_{0}^{\mathrm{fSGLD}}&:=\theta_{0},\\ \mathrm{d}\bar{\theta}_{t}^{\mathrm{fSGLD}}&=-\lambda\nabla_{\theta}U(\bar{\theta}_{\lfloor t\rfloor}^{\mathrm{fSGLD}}+\epsilon_{\lceil t\rceil},X_{\lceil t\rceil})\ \mathrm{d}t+\sqrt{2\lambda\beta^{-1}}\mathrm{d}\widetilde{B}_{t}.\end{split}$ (25) $$

At grid-points, we note that the law of the interpolated process is the same as the law
of the fSGLD algorithm [7](https://arxiv.org/html/2510.02174v1#S2.E7), i.e. $\mathscr{L}(\bar{\theta}^{\text{fSGLD}}_{k})=\mathscr{L}(\theta_{k}^{\text{fSGLD}})$, for each $k\in\mathbb{N}$. Moreover, we introduce the following continuous-time process $(\Phi_{t}^{s,u,\lambda,\text{fSGLD}})_{t\geq s}$, which is beneficial for our analysis, and define it as the solution of the following SDE

$$ $\begin{split}\Phi_{s}^{s,u,\lambda,\text{fSGLD}}&:=v\in\mathbb{R}^{d}\\ \mathrm{d}\Phi_{t}^{s,u,\lambda,\text{fSGLD}}&:=-\lambda\nabla g_{\epsilon}(\Phi_{t}^{s,u,\lambda,\text{fSGLD}})\ \mathrm{d}t+\sqrt{2\lambda\beta^{-1}}\ \mathrm{d}\widetilde{B}^{\lambda}_{t}.\end{split}$ $$

###### Definition C.2 .

Fix $k\in\mathbb{N}$. For any $t\geq kT$, define $\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}:=\Phi_{t}^{kT,\bar{\theta}^{\text{fSGLD}}_{kT},\lambda,\text{fSGLD}}$, where $T:=\lfloor 1/\lambda\rfloor$.

In other words, $\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}$ in Definition [C.2](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem2) is a process started from the value of the continuous-time interpolation fSGLD process [25](https://arxiv.org/html/2510.02174v1#A3.E25) at
time $kT$ and run until time $t\geq kT$ with the continuous-time flatness Langevin dynamics.

###### Remark C.1 .

###### Definition C.2 .

### C.2 Proofs of the results in Sections 3.2 and 3.3

To prove Proposition [3.1](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem1), Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2), and Corollary [3.3](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem3), we will use the following results in Corollary [C.3](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem3) and Lemma [C.4](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem4) below.

###### Corollary C.3 .

(Bolley & Villani, 2005, Corollary 2.3)
For any two Borel probability measures $\mu$ and $\nu$ with finite second moments, one obtains

$$ $W_{2}(\mu,\nu)\leq C_{\nu}\left[\sqrt{\text{KL}(\mu||\nu)}+\left(\frac{\text{KL}(\mu||\nu)}{2}\right)^{1/4}\right],$ $$

where

$$ $C_{\nu}:=2\inf_{\widetilde{\kappa}>0}\left(\frac{1}{\widetilde{\kappa}}\left(\frac{3}{2}+\log\int_{\mathbb{R}^{d}}e^{\widetilde{\kappa}|\theta|^{2}}\ \nu(\mathrm{d}\theta)\right)\right)^{1/2}.$ (26) $$

###### Lemma C.4 .

Let Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Then, the following set

$$ $A:=\left\{\theta\in\mathbb{R}^{d}:|\theta|\leq\sqrt{\frac{b}{a}}\right\},$ (27) $$

contains all the minimizers of $u(\theta)$, $v(\theta)$, and $g_{\epsilon}(\theta)$, where $a$ and $b$ are given in [21](https://arxiv.org/html/2510.02174v1#A2.E21).

###### Proof of Lemma C.4 .

Let $\theta^{\star}_{g_{\epsilon}}$, $\theta^{\star}_{u}$, $\theta^{\star}_{v}$ be a minimizer of $g_{\epsilon}(\theta)$, $u(\theta)$, and $v(\theta)$, respectively. By Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), we have

$$ $\begin{split}0=\langle\nabla v(\theta^{*}_{v}),\theta^{\star}_{v}\rangle=\langle\nabla v(\theta^{*}_{u}),\theta^{\star}_{u}\rangle=\langle\nabla u(\theta^{\star}_{u}),\theta^{\star}_{u}\rangle\geq\overline{a}|\theta^{\star}_{u}|^{2}-\overline{b},\end{split}$ (28) $$

which implies

$$ $|\theta^{\star}_{u}|=|\theta^{\star}_{v}|\leq\sqrt{\frac{\overline{b}}{\overline{a}}}\leq\sqrt{\frac{b}{a}}.$ $$

Due to Remark [B.2](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem2), we have

$$ $\begin{split}0=\langle\nabla g_{\epsilon}(\theta^{\star}_{g_{\epsilon}}),\theta^{\star}_{g_{\epsilon}}\rangle\geq a|\theta^{\star}_{g_{\epsilon}}|^{2}-b,\end{split}$ (29) $$

which implies

$$ $|\theta^{\star}_{g_{\epsilon}}|\leq\sqrt{\frac{b}{a}}.$ $$

∎

###### Proof of Proposition 3.1 .

Using [19](https://arxiv.org/html/2510.02174v1#A1.E19) with [16](https://arxiv.org/html/2510.02174v1#A1.E16), we have

$$ $\begin{split}\text{KL}(\pi_{\beta}^{\text{fSGLD}}||\pi_{\beta,\sigma}^{\star})&=\int_{\mathbb{R}^{d}}\log\left(\frac{\pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta)}{\pi_{\beta,\sigma}^{\star}(\mathrm{d}\theta)}\right)\ \pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta)\\ &=\int_{\mathbb{R}^{d}}\log\left(Z^{-1}_{\beta}Z_{\beta,\sigma}\exp(-\beta\ \mathbb{E}[\mathscr{R}(\theta,\epsilon)])\right)\pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta)\\ &=\log\left(\frac{Z_{\beta,\sigma}}{Z_{\beta}}\right)-\beta\int_{\mathbb{R}^{d}}\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\ \pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta).\end{split}$ (30) $$

We focus on the first term on the right-hand side of [30](https://arxiv.org/html/2510.02174v1#A3.E30). We denote the complementary set of $A$ in Lemma [C.4](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem4) by $A^{c}$. Using [17](https://arxiv.org/html/2510.02174v1#A1.E17) and [18](https://arxiv.org/html/2510.02174v1#A1.E18), one obtains

$$ $\begin{split}\log\left(\frac{Z_{\beta,\sigma}}{Z_{\beta}}\right)&=\log\left(\frac{\int_{A}e^{-\beta v(\theta)}d\theta+\int_{A^{c}}e^{-\beta v(\theta)}d\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}d\theta+\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta)}d\theta}\right)\\ &=\log\left(\frac{\frac{\int_{A}e^{-\beta v(\theta)}d\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}d\theta}+\frac{\int_{A^{c}}e^{-\beta v(\theta)}d\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}d\theta}}{1+\frac{\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta)}d\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}d\theta}}\right).\end{split}$ (31) $$

We provide a bound on the first term of the numerator in [31](https://arxiv.org/html/2510.02174v1#A3.E31), i.e., $\frac{\int_{A}e^{-\beta v(\theta)}\mathrm{d}\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta))}\mathrm{d}\theta}$. By the extreme value theorem, there exists a constant $C_{A}>0$:

$$ $|g_{\epsilon}(\theta)-v(\theta)|\leq C_{A}\sigma^{4},\quad\forall\ \theta\in A,$ (32) $$

where $C_{A}$ is the bound of $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]$ over $\theta\in A$ in [16](https://arxiv.org/html/2510.02174v1#A1.E16). This leads to

$$ $e^{-C_{A}\beta\sigma^{4}}\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta\leq\int_{A}e^{-\beta v(\theta)}\mathrm{d}\theta\leq e^{C_{A}\beta\sigma^{4}}\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta,$ $$

which implies

$$ $e^{-C_{A}\beta\sigma^{4}}\leq\frac{\int_{A}e^{-\beta v(\theta)}\mathrm{d}\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}\leq e^{C_{A}\beta\sigma^{4}}.$ (33) $$

We provide a bound on the second term of the numerator on the right-hand side of [31](https://arxiv.org/html/2510.02174v1#A3.E31), i.e., $\frac{\int_{A^{c}}e^{-\beta v(\theta)}\mathrm{d}\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}$. We note that, for any $\theta\in A^{c}$, there exists $\delta_{K}>0$ such that

$$ $v(\theta)>v(\theta_{v}^{*})+\delta_{v},\quad\text{for}\ \theta\in A^{c}.$ $$

For $0<\beta_{0}<\beta$, we have

$$ $-(\beta-\beta_{0})v(\theta)<-(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v}),$ $$

which implies

$$ $\begin{split}\int_{A^{c}}e^{-\beta v(\theta)}\mathrm{d}\theta&\leq e^{-(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\int_{A^{c}}e^{-\beta_{0}v(\theta)}\mathrm{d}\theta.\end{split}$ (34) $$

By [16](https://arxiv.org/html/2510.02174v1#A1.E16) and the extreme value theorem, we obtain

$$ $\begin{split}\exp(-\beta\ \mathbb{E}[\mathscr{R}(\theta,\epsilon)])&=\exp\left(-\frac{\beta}{24}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)\ \sigma^{4}(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})\right)\\ &\geq\exp(-\beta\tilde{c}_{A}\sigma^{4})\\ &:=\exp(-\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)]),\end{split}$ (35) $$

where $\tilde{c}_{A}$ in the inequality denotes the bound of the fourth derivative of $u$ over $\theta\in A$, and $\mathbb{E}[\mathscr{R}_{A}(\epsilon)])=O(\sigma^{4})$. Using [35](https://arxiv.org/html/2510.02174v1#A3.E35), we have

$$ $\begin{split}\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta&\geq\exp(-\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A}e^{-\beta v(\theta)}\ \mathrm{d}\theta\\ &=\exp(-\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{\infty}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)\\ &\geq\exp(-\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right).\end{split}$ (36) $$

Combining [34](https://arxiv.org/html/2510.02174v1#A3.E34) and [36](https://arxiv.org/html/2510.02174v1#A3.E36), we obtain

$$ $\begin{split}\frac{\int_{A^{c}}e^{-\beta v(\theta)}\mathrm{d}\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}&\leq\frac{\exp(\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A^{c}}e^{-\beta_{0}v(\theta)}d\theta}{e^{(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)}.\end{split}$ (37) $$

We provide a bound on the ratio in the denominator on the right-hand side of [31](https://arxiv.org/html/2510.02174v1#A3.E31). By Taylor’s theorem,

$$ $g_{\epsilon}(\theta)=g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})+\frac{1}{2}(\theta-\theta^{\star}_{g_{\epsilon}})^{T}\nabla^{2}g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})(\theta-\theta^{\star}_{g_{\epsilon}})+R_{2}(\theta),$ (38) $$

where $R_{2}(\theta)$ is the remainder term accounting for the residual error above second order. By the extreme value theorem, there exists $\overline{m}$, $\overline{M}>0$ such that

$$ $\begin{split}\overline{m}\leq e^{-\beta R_{2}(\theta)}\leq\overline{M},\quad\forall\ \theta\in A.\end{split}$ (39) $$

Using [38](https://arxiv.org/html/2510.02174v1#A3.E38) and [39](https://arxiv.org/html/2510.02174v1#A3.E39), one obtains

$$ $\begin{split}\int_{A}e^{-\beta g_{\epsilon}(\theta)}d\theta&=e^{-\beta g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}\int_{A}e^{-\frac{\beta}{2}(\theta-\theta^{\star}_{g_{\epsilon}})^{T}\nabla^{2}g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})(\theta-\theta^{\star}_{g_{\epsilon}})-\beta R_{2}(\theta)}d\theta\\ &\leq\overline{M}e^{-\beta g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}\left(\frac{2\pi}{\beta}\right)^{d/2}\frac{1}{\sqrt{\text{det}\nabla^{2}g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}}.\end{split}$ (40) $$

Using [40](https://arxiv.org/html/2510.02174v1#A3.E40), it follows

$$ $\begin{split}1+\frac{\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}{\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}\geq 1+\frac{\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}{\overline{M}e^{-\beta g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}\frac{1}{\sqrt{\text{det}\nabla^{2}g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}}}\geq 1.\end{split}$ $$

Thus,

$$ $\begin{split}&\frac{1}{1+\frac{\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}{\int_{\mathbb{R}^{d}}\int_{A}e^{-\beta g_{\epsilon}(\theta)}\mathrm{d}\theta}}\leq\frac{1}{1+\frac{\int_{A^{c}}e^{-\beta g_{\epsilon}(\theta))}\mathrm{d}\theta}{\overline{M}e^{-\beta g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}\frac{1}{\sqrt{\text{det}\nabla^{2}g_{\epsilon}(\theta^{\star}_{g_{\epsilon}})}}}}\leq 1.\end{split}$ (41) $$

Using [41](https://arxiv.org/html/2510.02174v1#A3.E41), [37](https://arxiv.org/html/2510.02174v1#A3.E37), and [33](https://arxiv.org/html/2510.02174v1#A3.E33) in [31](https://arxiv.org/html/2510.02174v1#A3.E31) yields

$$ $\begin{split}\log\left(\frac{Z_{\beta,\sigma}}{Z_{\beta}}\right)&\leq\log\left(e^{C_{A}\beta\sigma^{4}}+\frac{\exp(\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A^{c}}e^{-\beta_{0}v(\theta)}d\theta}{e^{(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)}\right).\end{split}$ (42) $$

We can bound [30](https://arxiv.org/html/2510.02174v1#A3.E30) using [42](https://arxiv.org/html/2510.02174v1#A3.E42), so that

$$ $\begin{split}\text{KL}(\pi_{\beta}^{\text{fSGLD}}||\pi_{\beta,\sigma}^{\star})&\leq\log\left(e^{C_{A}\beta\sigma^{4}}+\frac{\exp(\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A^{c}}e^{-\beta_{0}v(\theta)}d\theta}{e^{(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)}\right)\\ &\quad-\beta\int_{\mathbb{R}^{d}}\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\ \pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta).\end{split}$ (43) $$

Since $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]=O(\sigma^{4})$, $\mathbb{E}[\mathscr{R}_{A}(\epsilon)])=O(\sigma^{4})$, and $\sigma^{4}=\beta^{-(1+\eta)}$, one obtains

$$ $\lim_{\beta\rightarrow\infty}\text{KL}(\pi_{\beta}^{\text{fSGLD}}||\pi_{\beta,\sigma}^{\star})=0.$ (44) $$

We apply Corollary [C.3](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem3) with $\widetilde{\kappa}=1$, to prove the asymptotic convergence in Wasserstein distance of order two between $\pi_{\beta}^{\text{fSGLD}}$ and $\pi_{\beta,\sigma}^{\star}$. First, we provide a bound on the constant $C_{\pi_{\beta,\sigma}^{\star}}$ in Corollary [C.3](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem3) using $-\log x\leq x+1$ for all $x>0$,

$$ $\begin{split}C^{2}_{\pi_{\beta,\sigma}^{\star}}&=6-4\log\left(Z_{\beta,\sigma}\right)+\log\left(\int_{\mathbb{R}^{d}}e^{|\theta|^{2}-\beta v(\theta)}\ \mathrm{d}\theta\right)\\ &\leq 10+4Z_{\beta,\sigma}+\int_{\mathbb{R}^{d}}e^{|\theta|^{2}-\beta v(\theta)}\ \mathrm{d}\theta.\end{split}$ (45) $$

By Assumption [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), the Hessian $H(\theta)$ contained in $v(\theta)$ is bounded. For $\theta\in A$, we can control the last integral on the right-hand side of [45](https://arxiv.org/html/2510.02174v1#A3.E45) using [33](https://arxiv.org/html/2510.02174v1#A3.E33), Remark [C.4](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem4), and [39](https://arxiv.org/html/2510.02174v1#A3.E39), i.e.,

$$ $\begin{split}\int_{A}e^{|\theta|^{2}-\beta v(\theta)}\ \mathrm{d}\theta\leq e^{\frac{b}{a}-\frac{\beta\sigma^{2}}{2}\text{tr}\left(H(\theta)\right)}|\text{vol}(A)|.\end{split}$ (46) $$

For $\theta\in A^{c}$ and $\widetilde{c}\in(0,1)$, we have, by Assumption [3](https://arxiv.org/html/2510.02174v1#Thmassumption3),

$$ $\begin{split}u(\theta)&=u(\widetilde{c}\theta)+\int_{\widetilde{c}}^{1}\langle\theta,\nabla u(t\theta)\rangle\ \mathrm{d}t\\ &\geq u(\theta_{u}^{\star})+\int_{\widetilde{c}}^{1}t^{-1}\langle t\theta,\nabla u(t\theta)\rangle\ \mathrm{d}t\\ &\geq u(\theta_{u}^{\star})+\int_{\widetilde{c}}^{1}t^{-1}(\bar{a}|t\theta|^{2}-\bar{b})\ \mathrm{d}t\\ &\geq\frac{\bar{a}(1-\widetilde{c}^{2})}{2}|\theta|^{2}+\bar{b}\log\widetilde{c}+u(\theta_{u}^{\star})\\ &=\bar{c}|\theta|^{2}+\bar{p},\end{split}$ (47) $$

where $\bar{c}:=\frac{\bar{a}(1-\widetilde{c}^{2})}{2}>0$, and $\bar{p}:=\bar{b}\log\widetilde{c}+u(\theta_{u}^{\star})$.
For any $\theta\in A^{c}$, there exists $\delta_{u}>0$ such that

$$ $u(\theta)>u(\theta_{u}^{*})+\delta_{u},\quad\text{for}\quad\theta\in A^{c}.$ $$

For any $\beta_{0}\in(\frac{1}{\bar{c}},\beta)=(\frac{2}{\bar{a}(1-\widetilde{c}^{2})},\beta)$, we have

$$ $-(\beta-\beta_{0})u(\theta)<-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u}),\quad\text{for}\quad\theta\in A^{c}.$ (48) $$

Using [47](https://arxiv.org/html/2510.02174v1#A3.E47), [48](https://arxiv.org/html/2510.02174v1#A3.E48), and $\beta_{0}>\frac{1}{\bar{c}}$, one obtains

$$ $\begin{split}\int_{A^{c}}e^{|\theta|^{2}-\beta u(\theta)}d\theta&\leq e^{-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u})}\int_{A^{c}}e^{|\theta|^{2}-\beta_{0}u(\theta)}d\theta\\ &\leq e^{-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u})+\beta_{0}\bar{p}}\int_{A^{c}}e^{(1-\beta_{0}\bar{c})|\theta|^{2}}d\theta\\ &\leq e^{-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u})+\beta_{0}\bar{p}}\left(\frac{\pi}{\beta_{0}\bar{c}-1}\right)^{\frac{d}{2}}.\end{split}$ (49) $$

Plugging [46](https://arxiv.org/html/2510.02174v1#A3.E46) and [49](https://arxiv.org/html/2510.02174v1#A3.E49) in [45](https://arxiv.org/html/2510.02174v1#A3.E45) yields

$$ $\begin{split}C^{2}_{\pi_{\beta,\sigma}^{\star}}&\leq 10+4Z_{\beta,\sigma}\\ &\quad+e^{-\frac{\beta\sigma^{2}}{2}\text{tr}\left(H(\theta)\right)}\left(e^{\frac{b}{a}}|\text{vol}(A)|+e^{-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u})+\beta_{0}\bar{p}}\left(\frac{\pi}{\beta_{0}\bar{c}-1}\right)^{\frac{d}{2}}\right).\end{split}$ (50) $$

Therefore, applying Corollary [C.3](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem3) with [50](https://arxiv.org/html/2510.02174v1#A3.E50) and taking the limit for $\beta\rightarrow\infty$ as in [44](https://arxiv.org/html/2510.02174v1#A3.E44), we arrive at

$$ $\lim_{\beta\rightarrow\infty}W_{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star})=0.$ (51) $$

∎

We use the following triangle inequality to establish a non-asymptotic bound for $W_{1}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})$:

$$ $\begin{split}W_{1}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})&\leq W_{1}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))+W_{1}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))\\ &\quad+W_{1}(\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}),\pi_{\beta}^{\text{fSGLD}})+W_{1}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star}).\end{split}$ (52) $$

We control the four terms on the right-hand side of [52](https://arxiv.org/html/2510.02174v1#A3.E52) separately. The bounds for the first three terms follow directly from Zhang et al. (2023), with Zhang et al. (2023, Assumptions 1–3) replaced by Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3). For completeness, we reproduce these proofs here to make the convergence analysis of fSGLD self-contained.

We define, for each $p\geq 1$, the Lyapunov function $\widetilde{V}_{p}$ by $\widetilde{V}_{p}(\theta):=(1+|\theta|^{2})^{p/2}$, $\theta\in\mathbb{R}^{d}$, and similarly $\widetilde{v}_{p}(\omega):=(1+\omega^{2})^{p/2}$, for any real $\omega\geq 0$. These functions are twice continuously differentiable and

$$ $\sup_{\theta}(|\nabla\widetilde{V}_{p}(\theta)|/\widetilde{V}_{p}(\theta))<\infty,\qquad\lim_{|\theta|\rightarrow\infty}(|\nabla\widetilde{V}_{p}(\theta)|/\widetilde{V}_{p}(\theta))=0.$ (53) $$

Let $\mathscr{P}_{\widetilde{V}_{p}}$ denote
the set of $\mu\in\mathscr{P}(\mathbb{R}^{d})$ satisfying $\int_{\mathbb{R}^{d}}\widetilde{V}_{p}(\theta)\ \mu(\mathrm{d}\theta)<\infty$. Then, we define a functional that plays a central role in establishing the convergence rate in the Wasserstein-1 distance. For $\mu,\nu\in\mathscr{P}_{\widetilde{V}_{2}}$, let

$$ $w_{1,2}(\mu,\nu):=\inf_{\Gamma\in\mathscr{C}(\mu,\nu)}\int_{\mathbb{R}^{d}}\int_{\mathbb{R}^{d}}[1\land|\theta-\theta^{{}^{\prime}}|](1+\widetilde{V}_{2}(\theta)+\widetilde{V}_{2}(\theta^{{}^{\prime}}))\ \Gamma(\mathrm{d}\theta,\mathrm{d}\theta^{{}^{\prime}}).$ (54) $$

Moreover, it holds that $W_{1}(\mu,\nu)\leq w_{1,2}(\mu,\nu)$.

###### Proposition C.5 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Let $(\tilde{Z}_{t}^{fSGLD})_{t\in\mathbb{R}_{+}}$ be the solution of [24](https://arxiv.org/html/2510.02174v1#A3.E24) with initial condition $\tilde{Z}_{0}^{fSGLD}=\tilde{\theta}_{0}$ which is independent of $\mathscr{F}_{\infty}$ and satisfies $\mathbb{E}[|\tilde{\theta}_{0}|^{2}]<\infty$. Then,

$$ $w_{1,2}(\mathscr{L}(Z_{t}^{fSGLD}),\mathscr{L}(\tilde{Z}_{t}^{fSGLD}))\leq\hat{c}e^{-\dot{c}t}w_{1,2}(\mathscr{L}(\theta_{0}),\mathscr{L}(\tilde{\theta}_{0})),$ $$

where the constants $\dot{c}$ and $\hat{c}$ are given in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6).

###### Proof.

From Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1), one can deduce

$$ $\begin{split}|\nabla_{\theta}g_{\epsilon}(\theta)-\nabla_{\theta^{{}^{\prime}}}g_{\epsilon}(\theta^{{}^{\prime}})|&\leq\mathbb{E}[|\nabla_{\theta}u(\theta+\epsilon)-\nabla_{\theta^{{}^{\prime}}}u(\theta^{{}^{\prime}}+\epsilon)]\\ &\leq L_{1}\mathbb{E}[\varphi(X_{0})]|\theta-\theta^{{}^{\prime}}|.\end{split}$ (55) $$

The rest of the proof follows using Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), [55](https://arxiv.org/html/2510.02174v1#A3.E55), Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14), and [53](https://arxiv.org/html/2510.02174v1#A3.E53) in Zhang et al. (2023, Proof of Proposition 4.6).
∎

The constants $\dot{c}$ and $\hat{c}$ from Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5) are given in an explicit form.

###### Lemma C.6 .

The contraction constant $\dot{c}>0$ in Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5) is given by

$$ $\dot{c}:=\min\left\{\bar{\phi},\bar{c}(2),4\tilde{c}(2)\varepsilon\bar{c}(2)/2\right\}/2$ (56) $$

where $\bar{c}(2)=a/2$, $\tilde{c}(2)=(3/2)av_{2}(\bar{M}_{2})$ with $\bar{M}_{2}$ given in Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14), $\bar{\phi}$ is given by

$$ $\bar{\phi}:=\left(\bar{r}\sqrt{8\pi/(\beta L_{1}\mathbb{E}[\varphi(X_{0})])}\exp\left(\left(\bar{r}\sqrt{\beta L_{1}\mathbb{E}[\varphi(X_{0})]/8}+\sqrt{8/(\beta L_{1}\mathbb{E}[\varphi(X_{0})])}\right)^{2}\right)\right)^{-1},$ (57) $$

and moreover, $\varepsilon>0$ can be chosen such that the following inequality is satisfied

$$ $\varepsilon\leq 1\wedge\left(4\tilde{c}(2)\sqrt{2\beta\pi/(L_{1}\mathbb{E}[\varphi(X_{0})])}\int_{0}^{\tilde{r}}\exp\left(s\sqrt{\beta L_{1}\mathbb{E}[\varphi(X_{0})]/8}+\sqrt{8/(\beta L_{1}\mathbb{E}[\varphi(X_{0})])}\right)^{2}\mathrm{d}s\right)^{-1},$ (58) $$

where $\tilde{r}:=2\sqrt{2\tilde{c}(2)/\bar{c}(2)-1}$ and $\bar{r}:=2\sqrt{4\tilde{c}(2)(1+\bar{c}(2))/\bar{c}(2)-1}$. The constant $\hat{c}>0$ is given by $\hat{c}:=2(1+\bar{r})\exp(\beta L_{1}\mathbb{E}[\varphi(X_{0})]\bar{r}^{2}/8+2\bar{r})/\varepsilon$.

###### Proof.

This follows by adapting the arguments of Zhang et al. (2023, Proof of Lemma 4.11) to the flatness Langevin SDE [24](https://arxiv.org/html/2510.02174v1#A3.E24), using [55](https://arxiv.org/html/2510.02174v1#A3.E55) together with Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14).
∎

From the definition of $\lambda_{\text{max}}$ given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), it follows that $0<\lambda\leq\lambda_{\text{max}}\leq 1$, and hence $1/2<\lambda T\leq 1$. We now proceed to bound the first term in [52](https://arxiv.org/html/2510.02174v1#A3.E52).

###### Lemma C.7 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda<\lambda_{\text{max}}$ given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), $t\in(kT,(k+1)T]$,

$$ $W_{2}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))\leq\sqrt{\lambda}\left(e^{-ak/4}\bar{D}_{2,1}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\bar{D}_{2,2}\right)^{1/2},$ $$

where

$$ $\begin{split}\bar{D}_{2,1}&:=4e^{4L_{1}^{2}\mathbb{E}[\varphi^{2}(X_{0})]}(L_{1}^{2}\mathbb{E}[\varphi^{2}(X_{0})]\bar{\psi}_{Y}+\bar{\psi}_{Z}),\\ \bar{D}_{2,2}&:=4e^{4L_{1}^{2}\mathbb{E}[\varphi^{2}(X_{0})]}(L_{1}^{2}\mathbb{E}[\varphi^{2}(X_{0})]\widetilde{\psi}_{Y}+\widetilde{\psi}_{Z}),\end{split}$ (59) $$

with $\bar{\psi}_{Y}$, $\widetilde{\psi}_{Y}$ given in [81](https://arxiv.org/html/2510.02174v1#A3.E81), and $\bar{\psi}_{Z}$, $\widetilde{\psi}_{Z}$ given in [82](https://arxiv.org/html/2510.02174v1#A3.E82).

###### Proof.

This follows by applying Lemma [C.16](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem16) together with the argument used in Zhang et al. (2023, Proof of Lemma 4.7). We summarize the main steps in the following.
Using synchronous coupling together with [25](https://arxiv.org/html/2510.02174v1#A3.E25), Definition [C.2](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem2), Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1), and it follows that for any $t\in(kT,(k+1)T]$,

$$ $\begin{split}\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|&\leq\lambda\left|\int_{kT}^{t}\left[\nabla_{\theta}U(\bar{\theta}_{\lfloor s\rfloor}^{\mathrm{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})-\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})\right]\mathrm{d}s\right|\\ &\leq\lambda\left|\int_{kT}^{t}\left[\nabla_{\theta}U(\bar{\theta}_{\lfloor s\rfloor}^{\mathrm{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|\\ &\quad+\lambda\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|\\ &\leq\lambda L_{1}\int_{kT}^{t}\varphi(X_{\lceil s\rceil})\left|\bar{\theta}_{\lfloor s\rfloor}^{\mathrm{fSGLD}}-\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}\right|\mathrm{d}s\\ &\qquad+\lambda\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|.\end{split}$ (60) $$

Squaring both sides of [60](https://arxiv.org/html/2510.02174v1#A3.E60) and taking expectations, we obtain using Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1)

$$ $\begin{split}\mathbb{E}\left[\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|^{2}\right]&\leq 2\lambda L_{1}^{2}\int_{kT}^{t}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\mathbb{E}\left[\left|\bar{\theta}_{\lfloor s\rfloor}^{\mathrm{fSGLD}}-\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}\right|^{2}\right]\mathrm{d}s\\ &\quad+2\lambda^{2}\mathbb{E}\left[\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|^{2}\right].\end{split}$ $$

From $\lambda T\leq 1$ and Lemma [C.16](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem16), we get

$$ $\begin{split}&\mathbb{E}\left[\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|^{2}\right]\\ &\leq 4\lambda L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\int_{kT}^{t}\mathbb{E}\left[\left|\bar{\theta}_{\lfloor s\rfloor}^{\mathrm{fSGLD}}-\bar{\theta}_{s}^{\mathrm{fSGLD}}\right|^{2}\right]\mathrm{d}s\\ &\quad+4\lambda L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\int_{kT}^{t}\mathbb{E}\left[\left|\bar{\theta}_{s}^{\mathrm{fSGLD}}-\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}\right|^{2}\right]\mathrm{d}s\\ &\quad+2\lambda^{2}\mathbb{E}\left[\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|^{2}\right]\\ &\leq 4\lambda L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right](e^{-\lambda akT}\bar{\psi}_{Y}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\widetilde{\psi}_{Y})\\ &\quad+4\lambda L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\int_{kT}^{t}\mathbb{E}\left[\left|\bar{\theta}_{s}^{\mathrm{fSGLD}}-\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}\right|^{2}\right]\mathrm{d}s\\ &\quad+2\lambda^{2}\mathbb{E}\left[\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|^{2}\right].\end{split}$ (61) $$

We now bound the last term in [61](https://arxiv.org/html/2510.02174v1#A3.E61) by splitting the final integral. Let $kT+N<t\leq kT+N+1$ with $N+1\leq T,N\in\mathbb{N}$. It follows that

$$ $\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|=\left|\sum_{n=1}^{N}I_{n}+R_{N}\right|,$ $$

where $I_{n}:=\int_{kT+(n-1)}^{kT+n}[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})]\mathrm{d}s$, and

$R_{N}:=\int_{kT+N}^{t}[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+N+1},X_{kT+N+1})]\mathrm{d}s$.
Squaring both sides, we obtain

$$ $\displaystyle\left|\sum_{n=1}^{N}I_{n}+R_{N}\right|^{2}=\sum_{n=1}^{N}|I_{n}|^{2}+2\sum_{n=2}^{N}\sum_{j=1}^{n-1}\langle I_{n},I_{j}\rangle+2\sum_{n=1}^{N}\langle I_{n},R_{N}\rangle+|R_{N}|^{2}.$ $$

Let $\mathscr{H}_{\epsilon}$ denote the sigma-algebra generated by $\epsilon$. We define the filtration $\mathscr{J}_{t}=\mathscr{F}^{\lambda}_{\infty}\vee\mathscr{X}_{\lfloor t\rfloor}\vee\mathscr{H}_{\lfloor\epsilon\rfloor}$ and we take expectations of both sides. Observe that for any $n=2,\dots,N$, $j=1,\dots,n-1$,

$$ $\displaystyle\mathbb{E}\left[\langle I_{n},I_{j}\rangle\right]$ $\displaystyle=\mathbb{E}\left[\mathbb{E}[\langle I_{n},I_{j}\rangle|\mathscr{J}_{kT+j}]\right]$ $\displaystyle=\mathbb{E}\left[\mathbb{E}\left[\left\langle\int_{kT+(n-1)}^{kT+n}[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})]\mathrm{d}s,\right.\right.\right.$ $\displaystyle\hskip 50.00008pt\left.\left.\left.\left.\int_{kT+(j-1)}^{kT+j}[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+j},X_{kT+j})]\mathrm{d}s\right\rangle\right|\mathscr{J}_{kT+j}\right]\right]$ $\displaystyle=\mathbb{E}\left[\left\langle\int_{kT+(n-1)}^{kT+n}\mathbb{E}\left[\left.\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\right|\mathscr{J}_{kT+j}\right]\mathrm{d}s,\right.\right.$ $\displaystyle\hskip 50.00008pt\left.\left.\int_{kT+(j-1)}^{kT+j}[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+j},X_{kT+j})]\mathrm{d}s\right\rangle\right]=0.$ $$

By the same reasoning, $\mathbb{E}\langle I_{n},R_{N}\rangle=0$ for all $1\leq n\leq N$. Combining these results, we can bound the last term on the right-hand side of [61](https://arxiv.org/html/2510.02174v1#A3.E61) using Lemma [C.17](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem17)

$$ $\begin{split}&2\lambda^{2}\mathbb{E}\left[\left|\int_{kT}^{t}\left[\nabla g_{\epsilon}(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}+\epsilon_{\lceil s\rceil},X_{\lceil s\rceil})\right]\mathrm{d}s\right|^{2}\right]\\ &=2\lambda^{2}\sum_{n=1}^{N}\mathbb{E}\left[|I_{n}|^{2}\right]+2\lambda^{2}\mathbb{E}\left[|R_{N}|^{2}\right]\\ &\leq 4e^{-a\lambda kT/2}\lambda(\bar{\psi}_{Z}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\widetilde{\psi}_{Z}).\end{split}$ $$

Consequently, [61](https://arxiv.org/html/2510.02174v1#A3.E61) is bounded as follows

$$ $\begin{split}\mathbb{E}\left[\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|^{2}\right]&\leq 4\lambda L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\int_{kT}^{t}\mathbb{E}\left[\left|\bar{\theta}_{s}^{\mathrm{fSGLD}}-\bar{\Phi}_{s}^{\lambda,k,\text{fSGLD}}\right|^{2}\right]\mathrm{d}s\\ &\quad+4e^{-a\lambda kT/2}\lambda(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\bar{\psi}_{Y}+\bar{\psi}_{Z})\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]\\ &\quad+4\lambda(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\widetilde{\psi}_{Y}+\widetilde{\psi}_{Z}).\end{split}$ $$

Applying Grönwall’s inequality yields

$$ $\begin{split}\mathbb{E}\left[\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|^{2}\right]&\leq\lambda e^{4L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]}\left[4e^{-a\lambda kT/2}(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\bar{\psi}_{Y}+\bar{\psi}_{Z})\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]\right.\\ &\hskip 80.00012pt\left.+4(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\widetilde{\psi}_{Y}+\widetilde{\psi}_{Z})\right].\end{split}$ $$

Finally, we obtain using $\lambda T\geq 1/2$,

$$ $\begin{split}W^{2}_{2}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))&\leq\mathbb{E}\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}\right|^{2}\\ &\leq\lambda(e^{-an/4}\bar{C}_{2,1}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\bar{C}_{2,2}),\end{split}$ (62) $$

where

$$ $\begin{split}&\bar{D}_{2,1}:=4e^{4L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]}(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\bar{\psi}_{Y}+\bar{\psi}_{Z}),\\ &\bar{D}_{2,2}:=4e^{4L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]}(L_{1}^{2}\mathbb{E}\left[\varphi^{2}(X_{0})\right]\widetilde{\psi}_{Y}+\widetilde{\psi}_{Z}).\end{split}$ $$

∎

The bound for the second term on the right-hand side of [52](https://arxiv.org/html/2510.02174v1#A3.E52) is established in the following lemma.

###### Lemma C.8 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda<\lambda_{\text{max}}$ given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), $t\in(kT,(k+1)T]$,

$$ $\begin{split}W_{1}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))&\leq\sqrt{\lambda}(e^{-\dot{c}k/2}\bar{D}_{2,3}\mathbb{E}[\widetilde{V}_{4}(\theta_{0})]+\bar{D}_{2,4}),\end{split}$ $$

where

$$ $\begin{split}\bar{D}_{2,3}&=\hat{c}\left(1+\frac{2}{\dot{c}}\right)(e^{a/2}\bar{D}_{2,1}+12),\\ \bar{D}_{2,4}&=\frac{\hat{c}}{1-\exp\left(-\dot{c}\right)}(\bar{D}_{2,2}+12c_{3}(\lambda_{\text{max}}+a^{-1})+9\widetilde{v}_{4}(\bar{M}_{4})+15),\end{split}$ (63) $$

with $\bar{D}_{2,1}$, $\bar{D}_{2,2}$ given in [59](https://arxiv.org/html/2510.02174v1#A3.E59), $\hat{c}$, $\dot{c}$ given in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6), $c_{3}$ given in [80](https://arxiv.org/html/2510.02174v1#A3.E80), and $\bar{M}_{4}$ given in Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14).

###### Proof.

This follows by applying Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5), Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7), Corollary [C.13](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem13), and Lemma [C.15](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem15) together with the arguments in Zhang et al. (2023, Proof of Lemma 4.8).
∎

Adapting the reasoning of Lemma [C.8](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem8), we establish a non-asymptotic $W_{2}$ bound between $\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})$ and $\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}})$, presented in the next corollary.

###### Corollary C.9 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda<\lambda_{\text{max}}$ given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), $t\in(kT,(k+1)T]$,

$$ $\begin{split}&W_{2}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))\leq\lambda^{1/4}(e^{-\dot{c}/4}\bar{D}^{\star}_{2,3}(\mathbb{E}[\widetilde{V}_{4}(\theta_{0})])^{1/2}+\bar{D}^{\star}_{2,4}),\end{split}$ $$

where

$$ $\begin{split}\bar{D}^{\star}_{2,3}&:=\sqrt{2\hat{c}}(1+4/\dot{c})(e^{a/8}\bar{D}^{1/2}_{2,1}+2\sqrt{2}),\\ \bar{D}^{\star}_{2,4}&:=\frac{\sqrt{2\hat{c}}}{1-\exp\left(-\dot{c}/2\right)}(\bar{D}^{1/2}_{2,2}+2\sqrt{2c_{3}}(\lambda_{\text{max}}+a^{-1})^{1/2}+\sqrt{3}\widetilde{v}^{1/2}_{4}(\bar{M}_{4})+\sqrt{15}),\end{split}$ (64) $$

with $\bar{D}_{2,1}$, $\bar{D}_{2,2}$ given in [59](https://arxiv.org/html/2510.02174v1#A3.E59), $\hat{c}$, $\dot{c}$ given in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6), $c_{3}$ given in [80](https://arxiv.org/html/2510.02174v1#A3.E80), and $\bar{M}_{4}$ given in Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14).

###### Proof.

This follows using Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5), Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7), Corollary [C.13](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem13), and Lemma [C.15](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem15) in Zhang et al. (2023, Proof of Corollary 4.9).
∎

We can now derive a non-asymptotic bound for the first three terms on the right-hand side of [52](https://arxiv.org/html/2510.02174v1#A3.E52) in $W_{1}$ distance.

###### Theorem C.10 .

Let Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Then, there exist constants $\dot{c},D_{1},D_{2},D_{3}>0$ such that, for every $\beta>0$, for $0<\lambda<\lambda_{\text{max}}$, any $t\in(kT,(k+1)T]$, and $k\in\mathbb{N}$,

$$ $\begin{split}&W_{1}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))+W_{1}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))+W_{1}(\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}),\pi_{\beta}^{\text{fSGLD}})\\ &\leq D_{1}e^{-\dot{c}\lambda k/2}(1+\mathbb{E}[|\theta_{0}|^{4}])+(D_{2}+D_{3})\sqrt{\lambda},\end{split}$ $$

where

$$ $\begin{split}D_{1}&:=2e^{\dot{c}/2}\left[(\lambda^{1/2}_{\text{max}}(\bar{D}^{1/2}_{2,1}+\bar{D}^{1/2}_{2,2}+\bar{D}_{2,3}+\bar{D}_{2,4})+\hat{c})+\hat{c}\left(1+\int_{\mathbb{R}^{d}}\widetilde{V}_{2}(\theta)\pi_{\beta,\sigma}(\mathrm{d}\theta)\right)\right]\\ &=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}}}\right)\right),\\ D_{2}&:=\bar{D}^{1/2}_{2,1}+\bar{D}^{1/2}_{2,2}=O\left(1+\sqrt{\frac{d}{\beta}}\right),\\ D_{3}&:=\bar{D}_{2,3}+\bar{D}_{2,4}=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}}}\right)\right),\end{split}$ (65) $$

with $\hat{c}$, $\dot{c}$ given in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6), $\bar{D}_{2,1}$, $\bar{D}_{2,2}$ given in [59](https://arxiv.org/html/2510.02174v1#A3.E59) (Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7)), $\bar{D}_{2,3}$, $\bar{D}_{2,4}$ given in [63](https://arxiv.org/html/2510.02174v1#A3.E63) (Lemma [C.8](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem8)), $D_{\star}>0$ is independent of $d$, $\beta$, $k$.

###### Proof.

Using Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7), and Lemma [C.8](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem8) in Zhang et al. (2023, Proof of Lemma 4.10), we obtain for $t\in(kT,(k+1)T]$,

$$ $\begin{split}&W_{1}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))+W_{1}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))\\ &\leq(\bar{D}^{1/2}_{2,1}++\bar{D}^{1/2}_{2,2}+\bar{D}_{2,3}+\bar{D}_{2,4})\sqrt{\lambda}[(e^{-\dot{c}k/2}\mathbb{E}[\widetilde{V}_{4}(\theta_{0})]+1)],\end{split}$ (66) $$

where $\bar{D}_{2,1}$, $\bar{D}_{2,2}$ are given in [59](https://arxiv.org/html/2510.02174v1#A3.E59) (Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7)), and $\bar{D}_{2,3}$, $\bar{D}_{2,4}$ are given in [63](https://arxiv.org/html/2510.02174v1#A3.E63) (Lemma [C.8](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem8)).
The remainder of the proof follows by applying [66](https://arxiv.org/html/2510.02174v1#A3.E66) and Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5) in Zhang et al. (2023, Proof of Theorem 2.4).
∎

An analogous result to Theorem [C.10](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem10) holds in Wasserstein-2 distance, as stated in the next corollary.

###### Corollary C.11 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Then, there exists constants $\dot{c},D_{4},D_{5},D_{6}>0$ such that, for every $\beta>0$, $0<\lambda\leq\lambda_{\text{max}}$, any $t\in(kT,(k+1)T]$, and $k\in\mathbb{N}$,

$$ $\begin{split}&W_{2}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))+W_{2}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))+W_{2}(\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}),\pi_{\beta}^{\text{fSGLD}})\\ &\leq D_{4}e^{-\dot{c}\lambda k/4}(\mathbb{E}[|\theta_{0}|^{4}]+1)+(D_{5}+D_{6})\lambda^{1/4},\end{split}$ $$

where

$$ $\begin{split}D_{4}&:=2(\lambda_{\text{max}}^{1/2}(\bar{D}^{1/2}_{2,1}+\bar{D}^{1/2}_{2,2})+\lambda_{\text{max}}^{1/4}(\bar{D}^{\star}_{2,3}+\bar{D}^{\star}_{2,4})+\sqrt{2}\hat{c}^{1/2})\\ &\quad+\sqrt{2}\hat{c}^{1/2}\left(1+\int_{\mathbb{R}^{d}}\widetilde{V}_{2}(\theta)\pi_{\beta}^{\text{fSGLD}}(\mathrm{d}\theta)\right)\\ &=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)\right)\\ D_{5}&:=\lambda_{\text{max}}^{1/4}\bar{D}^{1/2}_{2,1}+\lambda_{\text{max}}^{1/4}\bar{D}^{1/2}_{2,2}=O\left(1+\sqrt{\frac{d}{\beta}}\right)\\ D_{6}&:=\bar{D}^{\star}_{2,3}+\bar{D}^{\star}_{2,4}=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)\right),\end{split}$ (67) $$

where $\hat{c}$, $\dot{c}$ given in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6), $\bar{D}_{2,1}$, $\bar{D}_{2,2}$ given in [59](https://arxiv.org/html/2510.02174v1#A3.E59) (Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7)), $\bar{D}^{\star}_{2,3}$, $\bar{D}^{\star}_{2,4}$ given in [64](https://arxiv.org/html/2510.02174v1#A3.E64) (Corollary [C.9](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem9)), $D_{\star}>0$ is independent of $d$, $\beta$, $k$.

###### Proof.

This follows by applying Lemma [C.7](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem7), Corollary [C.9](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem9), and Proposition [C.5](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem5) in Zhang et al. (2023, Proof of Corollary 2.5).
∎

###### Proof of Theorem 3.2 .

Using [52](https://arxiv.org/html/2510.02174v1#A3.E52) and Theorem [C.10](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem10), we get

$$ $\begin{split}&W_{1}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})\\ &\leq D_{1}e^{-\dot{c}\lambda k/2}(1+\mathbb{E}[|\theta_{0}|^{4}])+(D_{2}+D_{3})\sqrt{\lambda}+W_{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star}).\end{split}$ (68) $$

The last term on the right-hand side of [68](https://arxiv.org/html/2510.02174v1#A3.E68) can be controlled as done in the proof of Proposition [3.1](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem1) using the bounds [43](https://arxiv.org/html/2510.02174v1#A3.E43) and [50](https://arxiv.org/html/2510.02174v1#A3.E50) in Corollary [C.3](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem3), and $\sigma^{4}=\beta^{-(1+\eta)}$ for $\eta>0$, arriving at

$$ $\begin{split}&W_{2}^{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star})\\ &\leq\left[20+8Z_{\beta,\sigma}+2e^{-\frac{\text{tr}\left(H(\theta)\right)}{2\beta^{(1+\eta)/2}}}\left(e^{\frac{b}{a}}|\text{vol}(A)|+e^{-(\beta-\beta_{0})(u(\theta_{u}^{*})+\delta_{u})+\beta_{0}\bar{p}}\left(\frac{\pi}{\beta_{0}\bar{c}-1}\right)^{\frac{d}{2}}\right)\right]\\ &\quad\times\Bigg[\log\left(e^{C_{A}\beta^{-\eta}}+\frac{\exp(\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A^{c}}e^{-\beta_{0}v(\theta)}d\theta}{e^{(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)}\right)\\ &\qquad\quad-\beta\int_{\mathbb{R}^{d}}\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\ \pi_{\beta}^{\text{fSGLD}}(\theta)\ \mathrm{d}\theta\\ &\qquad+\frac{1}{\sqrt{2}}\Bigg(\log\left(e^{C_{A}\beta^{-\eta}}+\frac{\exp(\beta\ \mathbb{E}[\mathscr{R}_{A}(\epsilon)])\int_{A^{c}}e^{-\beta_{0}v(\theta)}d\theta}{e^{(\beta-\beta_{0})(v(\theta_{v}^{*})+\delta_{v})}\left(|\text{vol}(A)|+\int_{A}\sum_{i=1}^{n}\frac{(-\beta v(\theta))^{i}}{i!}\mathrm{d}\theta\right)}\right)\\ &\qquad\qquad\qquad\quad-\beta\int_{\mathbb{R}^{d}}\mathbb{E}[\mathscr{R}(\theta,\epsilon)]\ \pi_{\beta}^{\text{fSGLD}}(\theta)\ \mathrm{d}\theta\Bigg)^{1/2}\Bigg].\end{split}$ (69) $$

Since $\mathbb{E}[\mathscr{R}(\theta,\epsilon)]=O(\sigma^{4})$ and $\mathbb{E}[\mathscr{R}_{A}(\epsilon)])=O(\sigma^{4})$, the square root of the right-hand side of [69](https://arxiv.org/html/2510.02174v1#A3.E69), which we denote by $\underline{D}$, is $O(\beta^{-\eta})$. The bound [11](https://arxiv.org/html/2510.02174v1#S3.E11) follows by using [69](https://arxiv.org/html/2510.02174v1#A3.E69) in [68](https://arxiv.org/html/2510.02174v1#A3.E68).
In addition, for any $\bar{\delta}>0$, if we choose $\lambda$, $k$ and $\beta$ such that $\lambda\leq\lambda_{\text{max}}$, and

$$ $\begin{split}D_{1}e^{-\dot{c}\lambda k/2}(1+\mathbb{E}[|\theta_{0}|^{4}])\leq\frac{\bar{\delta}}{3},\qquad(D_{2}+D_{3})\sqrt{\lambda}\leq\frac{\bar{\delta}}{3},\qquad\underline{D}\leq\frac{\bar{\delta}}{3},\end{split}$ $$

then $W_{1}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})\leq\bar{\delta}$. This yields $\beta\geq\left(3\underline{D}^{0}/\bar{\delta}\right)^{\frac{1}{\eta}}$ where $\underline{D}^{0}$ contains the remaining terms on the right-hand side of the bound in $W_{2}$ in [69](https://arxiv.org/html/2510.02174v1#A3.E69), $\lambda\leq\frac{\bar{\delta}^{2}}{9(D_{2}+D_{3})^{2}}\wedge\lambda_{\text{max}}$, and $\lambda k\geq\frac{2}{\dot{c}}\ln\left(\frac{3D_{1}(1+\mathbb{E}[|\theta_{0}|^{4}])}{\bar{\delta}}\right)$. From [65](https://arxiv.org/html/2510.02174v1#A3.E65), it follows that

$$ $k\geq\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}^{2}\dot{c}}\left(1+\frac{1}{(1-e^{-\dot{c}})^{2}}\right)\ln\left(\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}}\left(1+\frac{1}{1-e^{-\dot{c}}}\right)\right).$ $$

∎

###### Proof of Corollary 3.3 .

Using triangle inequality and Corollary [C.11](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem11), we get, for any $t\in(kT,(k+1)T]$, and $k\in\mathbb{N}$,

$$ $\begin{split}W_{2}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})&\leq W_{2}(\mathscr{L}(\bar{\theta}_{t}^{\text{fSGLD}}),\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}))+W_{2}(\mathscr{L}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}),\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}))\\ &\quad+W_{2}(\mathscr{L}(Z_{t}^{\lambda,\text{fSGLD}}),\pi_{\beta}^{\text{fSGLD}})+W_{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star})\\ &\leq D_{4}e^{-\dot{c}\lambda k/4}(\mathbb{E}[|\theta_{0}|^{4}]+1)+(D_{5}+D_{6})\lambda^{1/4}+W_{2}(\pi_{\beta}^{\text{fSGLD}},\pi_{\beta,\sigma}^{\star}).\end{split}$ (70) $$

Similarly as done in the proof of Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2), we can use [69](https://arxiv.org/html/2510.02174v1#A3.E69) to control the last term in [70](https://arxiv.org/html/2510.02174v1#A3.E70). This leads to [12](https://arxiv.org/html/2510.02174v1#S3.E12). In addition, for any $\bar{\delta}>0$, $\lambda$, $k$ and $\beta$ such that $\lambda\leq\lambda_{\text{max}}$, and

$$ $D_{4}e^{-\dot{c}\lambda k/4}(\mathbb{E}[|\theta_{0}|^{4}]+1)\leq\frac{\bar{\delta}}{3},\qquad(D_{5}+D_{6})\lambda^{1/4}\leq\frac{\bar{\delta}}{3},\qquad\underline{D}\leq\frac{\bar{\delta}}{3},$ $$

then $W_{2}(\mathscr{L}(\theta_{k}^{\text{fSGLD}}),\pi_{\beta,\sigma}^{\star})\leq\bar{\delta}$. This yields $\beta\geq\left(3\underline{D}^{0}/\bar{\delta}\right)^{\frac{1}{\eta}}$ where $\underline{D}^{0}$ is the same as in the proof of Theorem [3.2](https://arxiv.org/html/2510.02174v1#S3.Thmtheorem2), $\lambda\leq\frac{\bar{\delta}^{4}}{81(D_{5}+D_{6})^{4}}\wedge\lambda_{\max}$, and $\lambda k\geq\frac{4}{\dot{c}}\ln\left(\frac{3D_{4}(1+\mathbb{E}[|\theta_{0}|^{4}])}{\bar{\delta}}\right)$. From [67](https://arxiv.org/html/2510.02174v1#A3.E67), it follows that

$$ $k\geq\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}^{4}\dot{c}}\left(1+\frac{1}{(1-e^{-\dot{c}/2})^{4}}\right)\ln\left(\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)\right).$ $$

∎

###### Proof of Theorem 3.5 .

We begin by decomposing the expected excess risk using the random variable $Z^{\text{fSGLD}}_{\infty}$ , for which
$\mathscr{L}(Z^{\text{fSGLD}}_{\infty})=\pi_{\beta}^{\text{fSGLD}}$, and obtain

$$ $\begin{split}&\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\inf_{\theta\in\mathbb{R}^{d}}g_{\epsilon}(\theta)\\ &=(\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\mathbb{E}[g_{\epsilon}(Z_{\infty})])+(\mathbb{E}[g_{\epsilon}(Z_{\infty})]-\inf_{\theta\in\mathbb{R}^{d}}g_{\epsilon}(\theta)).\end{split}$ (71) $$

We proceed by controlling the two terms on the right-hand side of [71](https://arxiv.org/html/2510.02174v1#A3.E71) separately.
By using Raginsky et al. (2017, Lemma 3.5), Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1) with $\sigma^{2}=\beta^{-\frac{1+\eta}{2}}$ for $\eta>0$, Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12), and Corollary [C.11](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem11), the first term on the RHS of [71](https://arxiv.org/html/2510.02174v1#A3.E71) can be bounded by

$$ $\begin{split}\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\mathbb{E}[g_{\epsilon}(Z_{\infty})]\leq D_{1}^{\#}e^{-\dot{c}\lambda k/4}+D_{2}^{\#}\lambda^{1/4},\end{split}$ (72) $$

where

$$ $\begin{split}D_{1}^{\#}&:=D_{4}(L_{1}\mathbb{E}[\varphi(X_{0})](\mathbb{E}[|\theta_{0}|^{2}]+c_{1}(\lambda_{\text{max}}+a^{-1}))+L_{2}\mathbb{E}[\bar{\varphi}(X_{0})](1+d\beta^{-(1+\eta)/2})+\mathbb{E}[\widetilde{G}(\epsilon)])\\ &\qquad\times(\mathbb{E}[|\theta_{0}|^{4}]+1),\\ D_{2}^{\#}&:=(D_{5}+CD_{6})\\ &\qquad\times(L_{1}\mathbb{E}[\varphi(X_{0})](\mathbb{E}[|\theta_{0}|^{2}]+c_{1}(\lambda_{\text{max}}+a^{-1}))+L_{2}\mathbb{E}[\bar{\varphi}(X_{0})](1+d\beta^{-(1+\eta)/2})+\mathbb{E}[\widetilde{G}(\epsilon)]),\end{split}$ (73) $$

with $\dot{c}$ given in [56](https://arxiv.org/html/2510.02174v1#A3.E56), $D_{4}$, $D_{5}$, $D_{6}$ given in [67](https://arxiv.org/html/2510.02174v1#A3.E67), and $c_{1}$ given in [79](https://arxiv.org/html/2510.02174v1#A3.E79). The second term on the RHS of [71](https://arxiv.org/html/2510.02174v1#A3.E71) can be controlled via Raginsky et al. (2017, Proposition 3.4), which leads to

$$ $\mathbb{E}[g_{\epsilon}(Z_{\infty})]-\inf_{\theta\in\mathbb{R}^{d}}g_{\epsilon}(\theta)\leq D_{\diamond}^{\#},$ (74) $$

where

$$ $D_{\diamond}^{\#}:=\frac{d}{2\beta}\log\left(\frac{eL_{1}\mathbb{E}[\varphi(X_{0})]}{a}\left(\frac{b\beta}{d}+1\right)\right).$ (75) $$

Using the estimates from [72](https://arxiv.org/html/2510.02174v1#A3.E72) and [74](https://arxiv.org/html/2510.02174v1#A3.E74) in [71](https://arxiv.org/html/2510.02174v1#A3.E71), we obtain

$$ $\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\inf_{\theta\in\mathbb{R}^{d}}g_{\epsilon}(\theta)\leq D_{1}^{\#}e^{-\dot{c}\lambda k/4}+D_{2}^{\#}\lambda^{1/4}+D_{\diamond}^{\#}.$ (76) $$

Applying [15](https://arxiv.org/html/2510.02174v1#A1.E15) on the LHS of [76](https://arxiv.org/html/2510.02174v1#A3.E76), along with [16](https://arxiv.org/html/2510.02174v1#A1.E16), and choosing $\sigma^{4}=\beta^{-(1+\eta)}$, it follows that

$$ $\begin{split}\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\inf_{\theta\in\mathbb{R}^{d}}v(\theta)\leq D_{1}^{\#}e^{-\dot{c}\lambda k/4}+D_{2}^{\#}\lambda^{1/4}+D_{3}^{\#},\end{split}$ (77) $$

where

$$ $\begin{split}D_{1}^{\#}&=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)(1+d\beta^{-(1+\eta)/2})\right),\\ D_{2}^{\#}&=O\left(e^{D_{\star}(1+d/\beta)(1+\beta)}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)(1+d\beta^{-(1+\eta)/2})\right),\\ D_{3}^{\#}&:=D_{\diamond}^{\#}+\frac{\beta^{-(1+\eta)}}{24}\inf_{\theta\in\mathbb{R}^{d}}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})\\ &=O\left((d/\beta)\log(D_{\star}(\beta/d+1))+\beta^{-(1+\eta)}\right),\end{split}$ (78) $$

with $D_{\star}>0$ a constant independent of $d$, $\beta$, $k$.
In addition, for $\bar{\delta}>0$, if we choose $\beta$ such that $D^{\#}_{3}\leq\bar{\delta}/3$, then choose $\lambda$ such that $\lambda\leq\lambda_{\max}$ and $D^{\#}_{2}\lambda^{1/4}\leq\bar{\delta}/3$, and choose $k$ such that $D^{\sharp}_{1}e^{-\dot{c}\lambda k/4}\leq\bar{\delta}/3$, we obtain

$$ $\mathbb{E}[g_{\epsilon}(\theta_{k}^{\text{fSGLD}})]-\inf_{\theta\in\mathbb{R}^{d}}v(\theta)\leq\bar{\delta}.$ $$

This yields

$$ $\begin{split}\beta&\geq\beta_{\bar{\delta}}\vee\frac{9d}{2\bar{\delta}}\log\left(\frac{eL_{1}\mathbb{E}[\varphi(X_{0})]}{ad}\left(b+1\right)\left(d+1\right)\right)\\ &\qquad\vee\left[\frac{3}{8\bar{\delta}}\inf_{\theta\in\mathbb{R}^{d}}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})\right]^{\frac{1}{1+\eta}},\end{split}$ $$

where $\beta_{\bar{\delta}}$ is the root of the function $f^{\sharp}(\beta)=\frac{\log\left(\beta+1\right)}{\beta}-\frac{2\bar{\delta}}{9d}$, with $\beta>0$. Since

$$ $\begin{split}D^{\sharp}_{3}&\leq\frac{d}{2\beta}\log\left(\frac{eL_{1}\mathbb{E}[\varphi(X_{0})]}{ad}\left(b+1\right)\left(d+1\right)\left(\beta+1\right)\right)\\ &\qquad+\frac{\beta^{-(1+\eta)}}{24}\inf_{\theta\in\mathbb{R}^{d}}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk}),\end{split}$ $$

we can ensure $D^{\sharp}_{3}\leq\bar{\delta}/3$ by imposing

$$ $\begin{split}&\frac{d}{2\beta}\log\left(\frac{eL_{1}\mathbb{E}[\varphi(X_{0})]}{ad}\left(b+1\right)\left(d+1\right)\right)\leq\frac{\bar{\delta}}{9},\qquad\frac{d}{2\beta}\log\left(\beta+1\right)\leq\frac{\bar{\delta}}{9},\\ &\frac{\beta^{-(1+\eta)}}{24}\inf_{\theta\in\mathbb{R}^{d}}\sum_{i,j,k,l=1}^{d}\frac{\partial^{4}u}{\partial\theta_{i}\partial\theta_{j}\partial\theta_{k}\partial\theta_{l}}(\theta)(\delta_{ij}\delta_{kl}+\delta_{ik}\delta_{jl}+\delta_{il}\delta_{jk})\leq\frac{\bar{\delta}}{9}.\end{split}$ $$

Moreover, one can verify that $\lambda\leq\frac{\bar{\delta}^{4}}{81(D^{\sharp}_{2})^{4}}\wedge\lambda_{\max}$, and $\lambda k\geq\frac{4}{\dot{c}}\ln\frac{3D^{\sharp}_{1}}{\bar{\delta}}$, where $\dot{c}$ is given explicitly in Lemma [C.6](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem6). This leads to

$$ $\begin{split}k&\geq\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}^{4}\dot{c}}\left(1+\frac{1}{(1-e^{-\dot{c}/2})^{4}}\right)\left(1+d\beta^{-(1+\eta)/2}\right)^{4}\\ &\qquad\times\ln\left(\frac{D_{\star}e^{D_{\star}(1+d/\beta)(1+\beta)}}{\bar{\delta}}\left(1+\frac{1}{1-e^{-\dot{c}/2}}\right)\left(1+d\beta^{-(1+\eta)/2}\right)\right).\end{split}$ $$

∎

###### Corollary C.3 .

###### Lemma C.4 .

###### Proof of Lemma C.4 .

###### Proof of Proposition 3.1 .

###### Proposition C.5 .

###### Proof.

###### Lemma C.6 .

###### Proof.

###### Lemma C.7 .

###### Proof.

###### Lemma C.8 .

###### Proof.

###### Corollary C.9 .

###### Proof.

###### Theorem C.10 .

###### Proof.

###### Corollary C.11 .

###### Proof.

###### Proof of Theorem 3.2 .

###### Proof of Corollary 3.3 .

###### Proof of Theorem 3.5 .

### C.3 Auxiliary Results

We present auxiliary results required for the convergence analysis in Appendix [C.2](https://arxiv.org/html/2510.02174v1#A3.SS2). Their proofs follow the same lines as Zhang et al. (2023), with Zhang et al. (2023, Assumptions 1–3) replaced by Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3). For completeness, we include their statements to make the convergence analysis of fSGLD self-contained.

###### Lemma C.12 (Moment bounds of 25 ) .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda\leq\lambda_{\text{max}}$ given in [23](https://arxiv.org/html/2510.02174v1#A3.E23), $k\in\mathbb{N}$, $t\in(k,k+1]$,

$$ $\mathbb{E}\left[|\bar{\theta}^{fSGLD}_{t}|^{2}\right]\leq(1-a\lambda(t-k))(1-a\lambda)^{k}\ \mathbb{E}[|\theta_{0}|^{2}]+c_{1}(\lambda_{\text{max}}+a^{-1}),$ $$

where $a$ and $b$ are given in Remark [B.2](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem2), and

$$ $c_{1}:=c_{0}+2d\beta^{-1},\quad c_{0}:=2b+8\lambda_{\max}L_{2}^{2}\mathbb{E}\left[\bar{\varphi}^{2}(X_{0})\right](1+\sigma^{2}d)+4\lambda_{\max}\mathbb{E}[\widetilde{G}^{2}(\epsilon)].$ (79) $$

Moreover, $\sup_{t>0}\mathbb{E}[|\bar{\theta}^{fSGLD}_{t}|^{2}]\leq\mathbb{E}[|\theta_{0}|^{2}]+c_{1}(\lambda_{\text{max}}+a^{-1})<\infty$. By a similar argument, one obtains

$$ $\mathbb{E}\left[|\bar{\theta}^{fSGLD}_{t}|^{4}\right]\leq(1-a\lambda(t-k))(1-a\lambda)^{k}\ \mathbb{E}[|\bar{\theta}^{fSGLD}_{0}|^{4}+c_{3}(\lambda_{\text{max}}+a^{-1}),$ $$

where

$$ $\begin{split}M&:=\max\{(8ba^{-1}+48a^{-1}\lambda_{\max}(L_{2}^{2}\mathbb{E}\left[\bar{\varphi}^{2}(X_{0})\right](1+\sigma^{2}d)+\mathbb{E}[\widetilde{G}^{2}(\epsilon)]))^{1/2},\\ &\hskip 40.00006pt(128a^{-1}\lambda_{\max}^{2}(L_{2}^{3}\mathbb{E}\left[\bar{\varphi}^{3}(X_{0})\right]\mathbb{E}[(1+|\epsilon|)^{3}]+\mathbb{E}[\widetilde{G}^{3}(\epsilon)]))^{1/3}\},\\ c_{2}&:=4bM^{2}+152(1+\lambda_{\max})^{3}\\ &\quad\times\left((1+L_{2})^{4}\mathbb{E}\left[(1+\bar{\varphi}(X_{0}))^{4}\right]\mathbb{E}[(1+|\epsilon|)^{4}]+\mathbb{E}[(1+\widetilde{G}(\epsilon))^{4}]\right)(1+M)^{2},\\ c_{3}&:=(1+a\lambda_{\text{max}})c_{2}+12d^{2}\beta^{-2}(\lambda_{\text{max}}+9a^{-1}).\end{split}$ (80) $$

Moreover, this implies $\sup_{t>0}\mathbb{E}[|\bar{\theta}^{fSGLD}_{t}|^{4}]<\infty$.

###### Proof.

This follows along the same lines as Zhang et al. (2023, Lemma 4.2) under our own Assumptions [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2), and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3), and using the estimates in Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1) and [B.2](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem2).
∎

Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12) provides a uniform fourth-moment bound for the process $(\bar{\theta}^{fSGLD}_{t})_{t\geq 0}$ which in turn yields a uniform bound for $\widetilde{V}_{4}(\bar{\theta}^{fSGLD}_{t})$, as given in the next corollary.

###### Corollary C.13 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda<\lambda_{\text{max}}$, $k\in\mathbb{N}$, $t\in(k,k+1]$,

$$ $\mathbb{E}[\widetilde{V}_{4}(\bar{\theta}^{fSGLD}_{t})]\leq 2(1-a\lambda)^{\lfloor t\rfloor}\mathbb{E}[\widetilde{V}_{4}(\bar{\theta}^{fSGLD}_{0})]+2c_{3}(\lambda_{\text{max}}+a^{-1})+2,$ $$

where $c_{3}$ is given in Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12).

###### Proof.

This follows from the definition of the Lyapunov function $\widetilde{V}_{4}$ together with Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12).
∎

We establish a drift condition for the flatness Langevin SDE [24](https://arxiv.org/html/2510.02174v1#A3.E24), which will be instrumental in deriving moment bounds for the continuous-time proces $\bar{\Phi}^{\lambda,k,\ \text{fSGLD}}_{t}$ in Lemma [C.15](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem15).

###### Lemma C.14 .

(Chau et al., 2021, Lemma 3.5)
Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. Then, for each $p\geq 2$, $\theta\in\mathbb{R}^{d}$,

$$ $\Delta\widetilde{V}_{p}(\theta)\beta^{-1}-\langle\nabla g_{\epsilon}(\theta),\nabla\widetilde{V}_{p}(\theta)\rangle\leq-\bar{c}(p)\widetilde{V}_{p}(\theta)+\tilde{c}(p),$ $$

where $\bar{c}(p):=ap/4$ and $\tilde{c}(p):=(3/4)ap\ \widetilde{v}_{p}(\bar{M}_{p})$ with $\bar{M}_{p}:=(1/3+4b/(3a)+4d/(3a\beta)+4(p-2)/(3a\beta))^{1/2}$.

###### Lemma C.15 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $0<\lambda<\lambda_{\text{max}}$, $t\geq kT$, with $k\in\mathbb{N}$, the following inequality holds

$$ $\mathbb{E}[\widetilde{V}_{2}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})]\leq e^{-\lambda ta/2}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+c_{1}(\lambda_{\text{max}}+a^{-1})+3\widetilde{v}_{2}(\bar{M}_{2})+1,$ $$

where $c_{1}$ is given in Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12). In addition, the following inequality holds

$$ $\mathbb{E}[\widetilde{V}_{4}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})]\leq 2e^{-a\lambda t}\mathbb{E}[\widetilde{V}_{4}(\bar{\theta}^{\text{fSGLD}}_{0})]+3\widetilde{v}_{4}(\bar{M}_{4})+2c_{3}(\lambda_{\text{max}}+a^{-1})+2,$ $$

where $\bar{M}_{2}$ and $\bar{M}_{4}$ are given in Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14), and $c_{3}$ is given in Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12).

###### Proof.

This follows by applying Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12), Corollary [C.13](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem13), and Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14) in Zhang et al. (2023, Proof of Lemma 4.5).
∎

###### Lemma C.16 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold, and let $\lambda_{\text{max}}$ be given in [23](https://arxiv.org/html/2510.02174v1#A3.E23). Then, for any $t>0$,

$$ $\mathbb{E}\left[|\bar{\theta}_{\lfloor t\rfloor}^{\text{fSGLD}}-\bar{\theta}_{t}^{\text{fSGLD}}|^{2}\right]\leq\lambda\left[e^{-\lambda a\lfloor t\rfloor}\bar{\psi}_{Y}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\widetilde{\psi}_{Y}\right],$ $$

where

$$ $\begin{split}\bar{\psi}_{Y}&:=2\lambda_{\text{max}}L_{1}^{2}\mathbb{E}[\varphi^{2}(X_{0})],\\ \widetilde{\psi}_{Y}&:=2c_{1}L_{1}^{2}\lambda_{\text{max}}\mathbb{E}[\varphi^{2}(X_{0})](\lambda_{\text{max}}+a^{-1})+4\lambda_{\text{max}}L_{2}^{2}\mathbb{E}[\bar{\varphi}^{2}(X_{0})]+4\lambda_{\text{max}}\mathbb{E}[\widetilde{G}^{2}(\epsilon)]+2d\beta^{-1},\end{split}$ (81) $$

with $c_{1}$ given in Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12).

###### Proof.

This follows by applying Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1) and Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12) in Zhang et al. (2023, Proof of Lemma A.2).
∎

###### Lemma C.17 .

Let Assumption [1](https://arxiv.org/html/2510.02174v1#Thmassumption1), [2](https://arxiv.org/html/2510.02174v1#Thmassumption2) and [3](https://arxiv.org/html/2510.02174v1#Thmassumption3) hold. For any $t\in(kT,(k+1)T]$, with $k,N\in\mathbb{N}$ and $n=1,\dots,N+1$, where $N+1\leq T$, one obtains

$$ $\begin{split}\mathbb{E}[|\nabla g_{\epsilon}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})|^{2}]\leq e^{-a\lambda t/2}\bar{\psi}_{Z}\mathbb{E}[\widetilde{V}_{2}(\theta_{0})]+\widetilde{\psi}_{Z},\end{split}$ $$

where

$$ $\begin{split}\bar{\psi}_{Z}&=8L_{2}^{2}\mathbb{E}[(\varphi(X_{0})+\varphi(\mathbb{E}[X_{0}]))^{2}|X_{0}-\mathbb{E}[X_{0}]|^{2}],\\ \widetilde{\psi}_{Z}&=8L_{2}^{2}E[(\varphi(X_{0})+\varphi(\mathbb{E}[X_{0}]))^{2}|X_{0}-\mathbb{E}[X_{0}]|^{2}](3\widetilde{v}_{2}(\bar{M}_{2})+c_{1}(\lambda_{\text{max}}+a^{-1})+1+\sigma^{2}d),\end{split}$ (82) $$

with $\bar{M}_{2}$ and $c_{1}$ given in Lemma [C.14](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem14) and Lemma [C.12](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem12), respectively.

###### Proof.

We adapt the Zhang et al. (2023, Proof of Lemma A.1). First, we define the filtration $\mathscr{J}_{t}=\mathscr{F}^{\lambda}_{\infty}\vee\mathscr{X}_{\lfloor t\rfloor}\vee\mathscr{H}_{\lfloor\epsilon\rfloor}$. Then, the result follows by an application of Lemma [C.18](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem18), Remark [B.1](https://arxiv.org/html/2510.02174v1#A2.Thmtheorem1), and Lemma [C.15](https://arxiv.org/html/2510.02174v1#A3.Thmtheorem15)

$$ $\begin{split}&\mathbb{E}\left[\left|\nabla g_{\epsilon}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\right|^{2}\right]\\ &=\mathbb{E}\left[\mathbb{E}\left[\left.\left|\nabla g_{\epsilon}(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}})-\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\right|^{2}\right|\mathscr{J}_{kT}\right]\right]\\ &=\mathbb{E}\Bigg[\mathbb{E}\Bigg[\Bigg.\Bigg|\mathbb{E}\Bigg[\Bigg.\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\Bigg|\mathscr{J}_{kT}\Bigg]\\ &\qquad\qquad\qquad-\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\Bigg|^{2}\Bigg|\mathscr{J}_{kT}\Bigg]\Bigg]\\ &\leq 4\mathbb{E}\Bigg[\mathbb{E}\Bigg[\Bigg.\Bigg|\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},X_{kT+n})\\ &\qquad\qquad\qquad-\nabla_{\theta}U(\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}+\epsilon_{kT+n},\mathbb{E}\left[\left.X_{kT+n}\right|\mathscr{J}_{kT}\right])\Bigg|^{2}\Bigg|\mathscr{J}_{kT}\Bigg]\Bigg]\\ &\leq 8L_{2}^{2}\mathbb{E}\left[(\varphi(X_{0})+\varphi(\mathbb{E}[X_{0}]))^{2}|X_{0}-\mathbb{E}[X_{0}]|^{2}\right]\left(\sigma^{2}d+\mathbb{E}\left[\left(1+\left|\bar{\Phi}_{t}^{\lambda,k,\text{fSGLD}}\right|^{2}\right)\right]\right)\\ &\leq 8L_{2}^{2}\mathbb{E}\left[(\varphi(X_{0})+\varphi(\mathbb{E}[X_{0}]))^{2}|X_{0}-\mathbb{E}[X_{0}]|^{2}\right]\\ &\qquad\times\left(e^{-\lambda ta/2}\mathbb{E}[V_{2}(\theta_{0})]+c_{1}(\lambda_{\text{max}}+a^{-1})+3\widetilde{v}_{2}(\bar{M}_{2})+1+\sigma^{2}d\right).\end{split}$ $$

∎

In the next lemma, $L^{p}$ denotes the usual space of $p$-integrable real-valued random variables for $1\leq p<\infty$.

###### Lemma C.18 .

Let $\mathscr{F},\mathscr{X},\mathscr{H}\subset\mathscr{M}$ be sigma-algebras. Let $X,Y$ be $\mathbb{R}^{d}$-valued random vectors in $L^{p}$ for any $p\geq 1$ such that $Y$ is measurable with respect to $\mathscr{F}\vee\mathscr{X}\vee\mathscr{H}$. Then,

$$ $\mathbb{E}^{1/p}\left[\left.\left|X-\mathbb{E}[X|\mathscr{\mathscr{F}\vee\mathscr{X}\vee\mathscr{H}}]\right|^{p}\right|\mathscr{X}\vee\mathscr{H}\right]\leq 2\mathbb{E}^{1/p}\left[\left.\left|X-Y\right|^{p}\right|\mathscr{X}\vee\mathscr{H}\right].$ $$

###### Proof.

This follows by applying Chau et al. (2019, Lemma 6.1) to $\mathscr{F}\vee\mathscr{N}$, where the sigma-algebra $\mathscr{N}:=\mathscr{X}\vee\mathscr{H}$.
∎

###### Lemma C.12 (Moment bounds of 25 ) .

###### Proof.

###### Corollary C.13 .

###### Proof.

###### Lemma C.14 .

###### Lemma C.15 .

###### Proof.

###### Lemma C.16 .

###### Proof.

###### Lemma C.17 .

###### Proof.

###### Lemma C.18 .

###### Proof.

## Appendix D Experimental details

### D.1 Details for Section 4.2

#### D.1.1 Software and hardware environments

We conduct all experiments with Python 3.10.9 and Pytorch 1.13.1, CUDA 11.6.2, NVIDIA Driver 510.10 on Ubuntu 22.04.1 LTS server which equipped with AMD Ryzen Threadripper PRO 5975WX, NVIDIA A100 GPUs.

#### D.1.2 Implementation details

We follow standard data preprocessing and augmentation strategies as adopted in prior work (Li et al., 2017; Wei et al., 2022) on noisy-label benchmarks. For CIFAR-10N and CIFAR-100N, we apply random cropping with padding, random horizontal flipping, and normalization using dataset-specific statistics. For WebVision, we follow the preprocessing protocol of Kodge (2024).

Regarding model architectures, we employ the CIFAR-specific variants of ResNet-34 and ResNet-50 when training on CIFAR-10N and CIFAR-100N, where the first convolution layer is replaced by a $3\times 3$ kernel with stride $1$ (instead of the $7\times 7$ stride-$2$ convolution and max pooling used in ImageNet models) to accommodate the smaller $32\times 32$ resolution. For WebVision, we adopt the standard ResNet implementations as provided for ImageNet-scale data.

For both training-from-scratch and fine-tuning experiments, we use the same hyperparameter search spaces. Table [4](https://arxiv.org/html/2510.02174v1#A4.T4) summarizes the ranges considered for each optimizer. We do not employ any early stopping or pruning strategy during the Optuna-based hyperparameter tuning, ensuring that each trial is fully evaluated to its final epoch. We performed the same number of hyper-parameter trials for all methods so that the search-space exploration budget (number of trials) was identical.
Because each SAM update requires two gradient evaluations, this design implies that, for the same number of trials and training epochs, SAM consumed roughly twice the wall-clock compute time of the other baselines. Thus our tuning protocol is at least as favorable to SAM as to the proposed fSGLD, ensuring that our reported improvements are not due to weaker tuning of SAM.

For SGLD and fSGLD ($\beta$ fixed), we set a large inverse temperature $\beta=10^{14}$. This follows the common heuristic of using a near-zero temperature to minimize exploration when employing Langevin Dynamics as a optimizer for a given objective. For fSGLD ($\beta$-$\sigma$ coupled), we leverage our theoretical analysis as a practical tuning strategy. We only search for the optimal perturbation scale $\sigma$ and then deterministically set $\beta$ via our theoretically-derived relationship, $\beta=\sigma^{-4/(1+\eta)}$ with $\eta=0.01$. This is a practical choice, as a larger $\eta$ would cause $\beta$ to become too small, allowing the Langevin noise term to overwhelm the gradient term and turning the dynamics into a near-random exploration. A small $\eta$ thus ensures stable optimization. This principled approach significantly simplifies the search space.

**Table 4: Hyperparameter search spaces for different optimizers.**
| Optimizer | Learning rate | Momentum | Weight decay | Other hyperparameters |
| --- | --- | --- | --- | --- |
| SGD | $10^{[-2,0]}$ | $\{0.1,0.9\}$ | $5\times 10^{-4}$ | – |
| AdamW | $10^{[-4,-2]}$ | – | $10^{-2}$ | $[\beta_{1},\beta_{2}]\in\{[0.8,0.95],[0.99,0.999]\}$ |
| SGLD | $10^{[-2,0]}$ | – | $5\times 10^{-4}$ | $\beta=10^{14}$ |
| SAM | $10^{[-2,0]}$ | $\{0.1,0.9\}$ | $5\times 10^{-4}$ | $\rho\in 10^{[-3,-1]}$ |
| fSGLD ($\beta$ fixed) | $10^{[-2,0]}$ | – | $5\times 10^{-4}$ | $\beta=10^{14},\;\sigma\in 10^{[-3,-2]}$ |
| fSGLD ($\beta$-$\sigma$ coupled) | $10^{[-2,0]}$ | – | $5\times 10^{-4}$ | $\beta=\sigma^{-4/1.01},\;\sigma\in 10^{[-3,-2]}$ |

### D.2 Details for Section 4.5

For the Hessian spectrum analysis, we use the best-performing ResNet-34 model trained on CIFAR-10N under each optimizer setting.
Given a trained network $f_{\theta}$ and loss function $L$, we compute Hessian-vector products (HVPs) by applying automatic differentiation to the scalar product $\nabla_{\theta}L^{\top}v$ for a random vector $v$.
For eigenvalue computation, we adopt the Lanczos algorithm (Lin et al., 2016) as implemented in scipy.sparse.linalg.eigsh, which allows us to approximate the top-$k$ eigenvalues without explicitly forming the Hessian.
In all reported results, we compute up to the top 50 eigenvalues.
As a complementary measure of curvature, we estimate the trace of the Hessian using Hutchinson’s stochastic estimator (Avron & Toledo, 2011) with Rademacher random vectors:

$$ $\mathrm{tr}(H(\theta))\approx\frac{1}{m}\sum_{i=1}^{m}z_{i}^{\top}H(\theta)z_{i},\quad z_{i}\sim\mathrm{Unif}\{\pm 1\}^{d},$ $$

where $m=1000$ in our experiments and $d$ denote the number of model parameters.

The analysis is conducted on the CIFAR-10N, where we randomly subsample at most 1,000 examples to reduce computational overhead. Eigenvalue computations are performed with a tolerance of $10^{-4}$ and a maximum of 500 iterations for the Lanczos solver.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: * This work targets the online setting in the theory and the multi-pass setting in the experiments. How does the theory compare in the multi-pass setting, and how do the experiments compare in the online setting?
* Is the computational cost truly comparable to SGD? Does the RWP and Langevvin noise in fSGLD not necessitate a smaller step-size than standard SGD, thus requiring more steps to converge?
* How does early stopping affect the experiments and the curvature of the resulting solutions?
* How does the bound compare to standard SGLD bounds? Which error terms are brought about by the RWP estimate?
soundness: 3
strengths: * Addresses a classical yet current problem in machine learning: tractable flatness-aware optimisation.
* Provides a rigorous mathematical proof of convergence.
* Using its theory, the authors identify a good rate of coupling between SGLD temperature and the scale of Laplacian regularisation. They then show that this coupling works well in their experiments.
* Not only obtains convergence bounds but also provides excess risk bounds.
* The experiments consider a realistic setting.
summary: The authors propose an algorithm for flatness-aware optimisation in non-convex settings. Their method combines stochastic gradient Langevin dynamics (SGLD) and random weight perturbations. They propose using SGLD applied to a Laplacian-regularised modification of the objective, estimating the Laplacian through random weight perturbations. They compare this method against standard optimisation techniques and SAM, the most popular flatness-aware optimiser, demonstrating improved results.
weaknesses: * The dependence on dimension in the bound appears poor (exponential in $d$), requiring the step size to decay exponentially with $d$. While common in the SGLD literature (and thus not a fundamental flaw in novelty), the authors should mention this limitation in the main body, as dimension-dependence is critical for the large-scale settings targeted in the experiments.
* A large portion of the proof appears to be borrowed from previous papers on SGLD, making it difficult to isolate the novel theoretical contributions. I would suggest including a discussion on what is new in the proof technique or, if the technique is standard, a statement clarifying that it is adapted from prior work.
* In general, there is little discussion surrounding the theoretical results, making them somewhat opaque to the theoretical audience this paper will likely appeal to. For example, it would be helpful to decompose the error terms in the bound (e.g., from convergence, discretisation, and objective estimation). In particular, what is the error contributed by the RWP estimator, and how does it scale with dimension?
* There is a mismatch between the theory (online setting) and the experiments (multi-pass setting). While this may not invalidate the results, it is a detail that the authors should disclose and briefly discuss.
* The authors do not appear to use early stopping, which is common for the baselines considered. It would be interesting to know if early stopping interacts with the curvature of the final minimisers. A plot of the train curve for these methods might help with aleviating this concern.
* It is difficult to find details in the paper due to imprecise referencing to the appendix. For example, the constants $D_1, \dots, D_7$ in the main theoretical statements are critical for understanding dimension and temperature dependence. The authors should reference the specific equations (e.g., (65) and (67)) where these are defined, rather than referencing the entire appendix section. Similarly, I struggled to find the "compelxity analysis" referenced in the main body.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: The idea of fSGLD comes from random weight perturbation. The main contribution of this work lies in that it provides theoretical guarantees on how fSGLD indeed produces an effect of sharpness penalty, measured by trace of the Hessian. Regarding this, I have the following main questions:

1. The Assumptions 2-3 are quite strong,in particular, the Assumption 3. It would be very helpful to provide concrete (nonconvex) examples that satisfy those conditions. I don’t see a meaningful example for Assumption 3 except for linear models. If I understand correctly, networks with more than two layers would typically violate Assumption 2. 
2. Proposition 3.1 seems rather weak as $\alpha$ and $\beta$ are tied together. In particular, the equivalence only holds as $\beta \to \infty$, in which case $\alpha\to 0$, meaning that the sharpness penalty is asymptotic zero. In this sense, Proposition 3.1 does not reveal a direct connection between fSGLD and flat minimizer.  
3. I find the condition $\alpha = o(\beta^{-1/4})$ not intuitive. Why is the exponent $1/4$ instead of $1/6$ or so? I tried to dig into the proof, and find that equation (16) only keeps order-4 components in the Taylor expansion – why do the higher order terms disappear here? One selling point of this paper (Line 71) is the promise of dealing with higher order terms. However, I don’t quite understand why order-6 and higher terms disappear – is it part of the assumption? 
4. Theorem 3.2 and Corollary 3.3 have the similar issue, where the convergence only holds in the strong asymptotic sense that requires $\alpha\to 0$, thus their implication to finding a flat minimizer is limited. 
5. The proof of Theorem 3.2 and Corollary 3.3 seems mainly using techniques from Zhang et al 2023. It would be helpful to clarify the novelty here. 

Besides, I have a couple of minor suggestions. 

6. Line 126. Computing the trace of Hessian does not need to be expensive in high dimensions. For example, one can avoid dimension-dependent cost by doing the trick of commuting gradient and linear operators and using monte carlo. 
7. In Theorem 3.2 and Corollary 3.3, certain quantities are called “constants”, e.g., D with underline. However, these quantities go to zero in order for the bounds to be useful, which means they are not constants. Additionally, it would be better to make $\dot{c}, D_1, D_2, D_3$ etc explicit in the main paper – at least clarify which given constants they depend on. 

8. Line206, "is is i.i.d." -> "is i.i.d."
soundness: 2
strengths: See below.
summary: This paper proposes a new method, flatness-aware stochastic Langevin dynamics (fSGLD), as an efficient method for finding flat minimizer in nonconvex problems. Specifically, fSGLD is a variant of the typical SGLD (SGD with an additive Gaussian noise), with the modification that the gradient is evaluated at the current weight perturbed by a Gaussian noise. Then this paper proves that, in a certain asymptotic sense, fSGLD converges to the stationary distribution of SGLD that applied to the original objective with sharpness penalty. This paper further provides empirical verifications on the effectiveness of fSGLD in CIFAR-type problems, matching or outperforming benchmark methods such as SGD, AdamW, and SAM.
weaknesses: See below.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: (1) In Assumption 2 and throughout the rest of the paper, $x^{'}$ should be $x'$ and $\theta^{'}$ should be $\theta'$.

(2) You stated 1-Wasserstein convergence result in Theorem 3.2 and then the 2-Wasserstein convergence result as a corollary in Corollary 3.3. I thought 2-Wasserstein upper bounds the 1-Wasserstein distance. Intuitively, I expect the bound in Corollary 3.3. trivially provides a 1-Wasserstein bound for Theorem 3.2. If you can obtain tighter 2-Wasserstein bound by establishing 1-Wasserstein bound first, it would be helpful if you can add some discussions and explanations.

(3) Flat minima phenomenon says that SGD favors flat local minima that lead to better generalization. In designing fSGLD, it seems that you are making the global minimum flatter. Does fSGLD also make local minima flatter or it will only lead to a flatter global minimum and if that is the case, will that help you with generalization performance?
soundness: 3
strengths: (1) fSGLD is a novel algorithm that is naturally motivated by flat minima phenomenon and Random Weight Perturbation from the literature.

(2) There is solid theoretical analysis that establishes non-asymptotic Wasserstein convergence guarantees for the proposed algorithm. With the presence of Random Weight Perturbation, the analysis is quite sophisticated.

(3) Numerical experiments are extensive and illustrative.
summary: It is well known that the generalization property is closely associated with flat minima phenomenon, that says that flat local minima lead to better generalization performance. This work proposes and studies Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD) that optimizes a randomized-smooth objective that implicitly captures curvature information by adding Random Weight Perturbation (RWP). 

More specifically, consider $\min_{\theta\in\mathbb{R}^{d}}u(\theta)$, with $u(\theta)=\mathbb{E}[U(\theta,X)]$. One approach is to replace $u(\theta)$ by $v(\theta):=u(\theta)+\frac{\sigma^{2}}{2}\text{tr}(H(\theta))$, where $H(\theta)$ is the Hessian of $u$ at $\theta$. However, $\text{tr}(H(\theta))$ can be expensive to compute. If one takes $g_{\epsilon}(\theta)=\mathbb{E}[u(\theta+\epsilon)]$, where $\epsilon\sim\mathcal{N}(0,\sigma^{2}I_{d})$, then $g_{\epsilon}(\theta)\approx u(\theta)+\frac{\sigma^{2}}{2}\text{tr}(H(\theta))$, which motivates the author(s) of the paper to propose fSGLD:
\begin{equation*}
\theta_{k+1}=\theta_{k}-\lambda\nabla_{\theta}U(\theta_{k}+\epsilon_{k+1},X_{k+1})+\sqrt{2\lambda\beta^{-1}}\xi_{k+1}.
\end{equation*}
Intuitively, $\theta_{k}$ will converge to a stationary distribution that is close to $e^{-\beta v(\theta)}$ when $\sigma$ and $\lambda$ are small, and the Gibbs distribution will concentrate around the global minimizer of $v(\theta)$ when $\beta$ is large. The paper establishes non-asymptotic convergence guarantees in Wasserstein distance with the best known rate and derive an excess-risk bound for the Hessian-trace regularized objective. Extensive numerical experiments are also provided.
weaknesses: (1) One major weakness I see is that there does not seem to be any discussion comparing the iteration complexity or non-asymptotic convergence bounds for fSGLD vs SGLD theoretically. If you can have some theoretical result, or even just some double-well example for which you can show theoretically that fSGLD can outperform SGLD, that will strengthen the paper. Based on your current result, if you view SGLD as a special case of fSGLD as $\sigma\rightarrow 0$, you can perhaps directly compare fSGLD vs SGLD. 

(2) Proposition 3.1. seems to be an asymptotic result. In the proof, it seems most steps (if not are) are non-asymptotic. Also, since Theorem 3.2 and Corollary 3.3. are non-asymptotic, I guess Proposition 3.1. can also be made non-asymptotic. It would be really nice if you can make Proposition 3.1. non-asymptotic. If the bound is too complicated, you can highlight the big O dependence in the statement of Proposition 3.1. and refer to the proof section for the explicit expressions.

---

confidence: 4
contribution: 2
initial_rating: 0
presentation: 2
questions: Here is my derivation of the SDE for Equation 7:

### Expected value of the increments

We consider the fSGLD iteration

\begin{equation}
\theta_{k+1}
= \theta_k - \lambda  \nabla_\theta U(\theta_k + \epsilon_{k+1}, X_{k+1}) + \sqrt{2\lambda \beta^{-1}} \xi_{k+1},
\end{equation}

where
$ \epsilon_{k+1} \sim \mathcal{N}(0,\sigma^2 I_d) $,  $\xi_{k+1} \sim \mathcal{N}(0,I_d)$, and all variables are independent.

The increment is

\begin{equation}
\Delta \theta_k = \theta_{k+1} - \theta_k = - \lambda  \nabla_\theta U(\theta_k + \epsilon_{k+1}, X_{k+1}) + \sqrt{2\lambda \beta^{-1}}\ xi_{k+1}.
\end{equation}

Conditioning on $\theta_k$:

\begin{equation}
\mathbb{E}[\Delta \theta_k] = - \lambda G(\theta_k),
\end{equation}

where

\begin{equation}
G(\theta) := \mathbb{E}\left[ \nabla_\theta U(\theta + \epsilon, X) \right].
\end{equation}

---

### Covariance of the increments

Define the zero-mean stochastic component

\begin{equation}
\delta_{k+1} := \nabla_\theta U(\theta_k + \epsilon_{k+1}, X_{k+1}) - G(\theta_k), \qquad \mathbb{E}[\delta_{k+1}] = 0.
\end{equation}

Then

\begin{equation}
\Delta \theta_k = -\lambda G(\theta_k) - \lambda \delta_{k+1} + \sqrt{2\lambda \beta^{-1}} \xi_{k+1}.
\end{equation}

Since $\delta_{k+1}$ and $\xi_{k+1}$ are independent and centered,

\begin{equation}
\operatorname{Cov}(\Delta \theta_k) = \lambda^2  \mathbb{E}[\delta_{k+1}\delta_{k+1}^\top \mid \theta_k] + 2\lambda \beta^{-1} I_d.
\end{equation}

Define

\begin{equation}
\Sigma(\theta) := \operatorname{Cov}\left[  \nabla_\theta U(\theta + \epsilon, X)\right],
\end{equation}

so that

\begin{equation}
\operatorname{Cov}(\Delta \theta_k \mid \theta_k) = \lambda^2 \Sigma(\theta_k) + 2\lambda \beta^{-1} I_d.
\end{equation}

---

### Limiting SDE

The continuous-time limit satisfies

$$
d\theta_t = - \lambda \mathbb{E}\left[\nabla_\theta U(\theta_t + \epsilon, X)\right] dt + \sqrt{2\beta^{-1} I_d + \operatorname{Cov}\left[    \nabla_\theta U(\theta_t + \epsilon, X)\right]} dB_t,
$$

where $B_t$ is a standard $d$-dimensional Brownian motion.

Probably, if we expand the gradient of $U$ with Taylor, we get that the additional covariance term I derived scales like $\sigma^2 H H^{\top}$ where H is the Hessian of U. But I did not work the calculations out, sorry.
soundness: 1
strengths: 1. **Clear theoretical contribution.**  
The paper rigorously establishes that a specific coupling between the inverse temperature $\beta$ and the perturbation scale $\sigma$ guarantees that the invariant measure of fSGLD concentrates on flat minima.

2. **Strong theoretical guarantees.**  
The paper provides non-asymptotic convergence bounds in Wasserstein distance and an excess-risk bound, matching the best known rates for SGLD under comparable assumptions.

3. **Empirical validation consistent with the theory.**  
The experiments demonstrate that the theoretically prescribed $\beta$–$\sigma$ coupling consistently improves generalization and stability across some noisy-label and fine-tuning tasks, and in general, that your method is better than the others in most cases. Importantly, you tuned the hyperparameters fairly.
summary: The paper advances the literature by establishing a formal coupling between the Langevin temperature and random perturbation scale that provably links stochastic weight perturbations to convergence toward globally flat minima.
weaknesses: 1. **Questionable premise on flatness and generalization.**  
   Contrary to the claim made in the Abstract, I do not find the assertion that generalization in deep learning is tied to the pursuit of flatter minima convincing. For instance, Adam is known to converge to sharper minima than SGD, yet AdamW remains the default optimizer in large-scale LLM training. In such settings, convergence is not even expected, let alone a meaningful notion of selecting flatter or sharper minima. Moreover, [1] empirically challenges the link between generalization and flatness across a broad range of architectures and training setups, undermining the central motivation of the paper.

2. **Misleading explanation for the lack of adoption of SGLD.**  
   The Introduction attributes the limited use of SGLD in practice to its lack of an intrinsic bias toward flat minima. I find this reasoning unconvincing. The primary reason SGLD (and SGD-like samplers) are not used in modern deep learning is that they fail to effectively optimize highly nonconvex, large-scale architectures such as transformers and LLMs. The discussion on flatness is likely secondary or even irrelevant in this context.

3. **Missing discussion of implicit regularization in SAM-like algorithms.**  
   Several recent works have shown that SAM and its variants implicitly regularize toward flatter regions (See Thm 3.2 and Thm 3.5, and their discussions in [2]), not necessarily by minimizing the Hessian trace directly, but through implicit regularization of the objective function and by how gradient noise interacts with local curvature: sharp minima tend to amplify noise, effectively promoting escape toward flatter areas. The paper should discuss this implicit regularization mechanism and compare it conceptually to that of fSGLD. Furthermore, it would be valuable to explore whether a similar stochastic-curvature coupling could be incorporated into fSGLD to improve its dynamics.

4. **Limited novelty of the core method.**  
   Random Weight Perturbation (RWP) is a well-established technique, and SGLD is a classical stochastic optimization method. The proposed algorithm essentially combines two well-known components without introducing a genuinely new conceptual element. While the theoretical analysis is rigorous, the algorithmic contribution feels incremental given the existing literature.

5. **Insufficient experimental relevance.**  
   The experimental setup does not reflect the scale or complexity of contemporary deep learning practice. Evaluations on noisy-label datasets and ViT fine-tuning are limited in scope. To substantiate claims of improved generalization and practical relevance, the method should be tested on modern architectures such as small LLMs (e.g., NanoGPT).

**IMPORTANTLY:**  
Are you certain that the SDE presented in your Equation 24 — which is meant to model the dynamics of the discrete-time optimizer defined in Equation 7 — is derived correctly? Can you rigorously justify this correspondence in the sense of *weak approximations*, as proposed in [3]?  

1. The **expected value of the increments** appears to be **correct**, as it corresponds to the expectation of  $ \theta_{k+1} - \theta_k$ thereby yielding the appropriate drift term.  

2. The **covariance of the increments**, however, appears to be **incorrect**. While the contribution from the stochastic term $\xi$ is included, the covariance arising from the injected perturbations $\epsilon$ is **not accounted for**, which is problematic. Preliminary calculations suggest that this additional covariance term scales with $\sigma$ and may also depend on the Hessian of $U$, resembling the stochastic–curvature interaction observed in the analysis of SAM in [2].  

This is a **major concern** and must be **thoroughly addressed and clarified**.  
Please see my derivation below — I would be glad to be proven wrong.

---

**In summary:**

1. The stated motivation, linking generalization to flatness, is weak and empirically disputable.  
2. The comparison with SAM and related implicit-regularization methods is underdeveloped.  
3. The proposed method is conceptually incremental, combining two classical ideas without introducing a fundamentally new mechanism.  
4. The experimental validation is not sufficient to support claims of superiority over AdamW, especially given recent optimizers like SOAP, Muon, or Scion that demonstrate stronger empirical evidence on large-scale models.

**Most importantly, there seems to be a mistake in the derivation of the SDE of the method. I provide my derivation below.**

---

**[1]** *A Modern Look at the Relationship between Sharpness and Generalization.*  
Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, Nicolas Flammarion.

**[2]** *An SDE for Modeling SAM: Theory and Insights.*  
Enea Monzio Compagnoni, Luca Biggio, Antonio Orvieto, Frank Norbert Proske, Hans Kersting, Aurélien Lucchi.  

**[3]** *Stochastic Modified Equations and Adaptive Stochastic Gradient Algorithms.*  
Qi Li, Cheng Tai, Weinan E.


Decision: Reject