Title: A Dynamical System View of Langevin-Based Non-Convex Sampling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Non-Convex Sampling

Langevin Dynamics

Dynamical Systems
Abstract: Non-convex sampling is a key challenge in machine learning, central to non-convex optimization in deep learning as well as to approximate probabilistic inference. Despite its significance, theoretically there remain some important challenges: Existing guarantees suffer from the drawback of lacking guarantees for the last-iterates, and little is known beyond the elementary schemes of stochastic gradient Langevin dynamics. To address these issues, we develop a novel framework that lifts the above issues by harnessing several tools from the theory of dynamical systems. Our key result is that, for a large class of state-of-the-art sampling schemes, their last-iterate convergence in Wasserstein distances can be reduced to the study of their continuous-time counterparts, which is much better understood. Coupled with standard assumptions of MCMC sampling, our theory immediately yields the last-iterate Wasserstein convergence of many advanced sampling schemes such as mirror Langevin, proximal, randomized mid-point, and Runge-Kutta methods.

A Dynamical System View of Langevin-Based Non-Convex Sampling

Abstract

Non-convex sampling is a key challenge in machine learning, central to non-convex optimization in deep learning as well as to approximate probabilistic inference. Despite its significance, theoretically there remain some important challenges: Existing guarantees suffer from the drawback of lacking guarantees for the lastiterates , and little is known beyond the elementary schemes of stochastic gradient Langevin dynamics. To address these issues, we develop a novel framework that lifts the above issues by harnessing several tools from the theory of dynamical systems. Our key result is that, for a large class of state-of-the-art sampling schemes, their last-iterate convergence in Wasserstein distances can be reduced to the study of their continuous-time counterparts, which is much better understood. Coupled with standard assumptions of MCMC sampling, our theory immediately yields the last-iterate Wasserstein convergence of many advanced sampling schemes such as mirror Langevin, proximal, randomized mid-point, and Runge-Kutta methods.

1 Introduction

Many modern learning tasks involve sampling from a high-dimensional density 𝜋 / 𝑒 GLYPH<0> 𝑓 , where 𝑓 is a non-convex potential representing, for instance, the loss function of a deep neural network. To this end, an approach that has found wide success is to discretize the continuous-time Langevin diffusion

where 𝐵 𝑡 is a Brownian motion [57]. The idea behind this approach is that, since 𝜋 is the stationary distribution of (LD), one can expect a similar behavior for discretizations of (LD). Such a framework has inspired numerous sampling schemes with per-iteration costs as cheap as stochastic gradient descent, which are particularly suitable for large-scale approximate probabilistic inference and Bayesian learning [2, 54, 57]. Moreover, several works have noticed that these Langevin-based schemes provide deep insights about minimizing 𝑓 using stochastic oracles [22, 48], which serves as an important step toward explaining the empirical success of training deep neural networks.

The convergence of Langevin-based non-convex sampling has therefore attracted significant interest from both practitioners and theoreticians, whose intense study has led to a plethora of new guarantees; see related work for details. Despite such impressive progress, several challenges remain for the fully non-convex setup:

The convergence is typically given on the averaged iterates instead of the more natural last iterates [4, 54]. This is especially problematic from the perspective of understanding the minimization of 𝑓 , as in practice, the last iterates of an optimization algorithm play the most pivotal role for downstream tasks.

An additional notable drawback of the current theory is its predominant focus on the basic EulerMaruyama discretization of (LD) (see, e.g., [4, 20, 54]). As a result, the convergence analysis of more advanced sampling schemes remains largely unexplored in the fully non-convex regime [1, 24, 25, 34, 53, 61].

§ Contributions and Approaches. To overcome the aforementioned challenges, our main contribution, from a high level, can be succinctly summarized as:

Under mild assumptions, we prove that the iterates of a broad range of Langevin-based sampling schemes converge to the continuous-time (LD) in Wasserstein distance . ( ★ )

Combining ( ★ ) with classical results on Langevin diffusion [45] immediately yields the last-iterate convergence in Wasserstein distances for a wide spectrum of sampling schemes, thus resolving all the challenges mentioned above. To illustrate this point, we state a simple version of our main result.

Theorem (Informal) . Suppose we discretize (LD) as

with step-sizes f 𝛾 𝑘 g 𝑘 2 ℕ and i.i.d. standard Gaussians f 𝜉 𝑘 g 𝑘 2 ℕ . Then, under an easy-to-verify condition on the bias (see (5) in Assumption 3), f 𝑥 𝑘 g 𝑘 2 ℕ converges in Wasserstein distance to 𝜋 . In addition, these conditions are satisfied by many advanced sampling schemes.

This result is achieved via a new dynamical perspective to study Langevin-based sampling. More specifically,

We introduce the Picard process , which is the sampling analogue of Picard's method of successive approximations for solving ODEs [17]. Contrary to most existing analyses, the Picard process allows us to completely bypass the use of relative entropy, which is the culprit for the appearance of averaged iterates [20].

Using the Picard process, we will prove that the iterates of various Langevin-based schemes generate a so-called Wasserstein asymptotic pseudotrajectory (WAPT) for the continuous-time (LD). The main motivation for considering WAPT is to connect Langevin-based schemes to the dynamical system theory of Benaïm and Hirsch [7], which works for metric spaces and is last-iterate by design, and therefore particularly suitable for our purpose.

Finally, under standard stability assumptions in the literature [39, 51], we show how a tandem of our WAPT result and dynamical system theory yields the desirable convergence of various existing schemes, as well as motivates more efficient algorithms that enjoy the same rigorous guarantees.

§ Related work. There is a vast literature on structured non-convex sampling, where one imposes extra assumptions on the target density. Under these conditions, one can derive non-asymptotic rates for Langevin-based schemes [13, 15, 33, 35, 36, 41, 48, 56, 60, 63]. Our work is orthogonal to these works as we study generic non-convex sampling, an NP-hard problem whose convergence is asymptotic at best.

Most relevant to our paper are the works [4, 8, 20, 29, 54], which study the asymptotic convergence of Langevin-based schemes under minimal regularity assumptions on 𝑓 . Compared to their results, our guarantees either improve upon existing ones or are incomparable; see Section 5.4 for a more detailed comparison.

2 The Langevin-Robbins-Monro Template

We consider the following general template for sampling algorithms: Starting from an initial point, the iterates f 𝑥 𝑘 g 𝑘 2 ℕ follow the recursion

where 𝛾 𝑘 's are step sizes, 𝑣 is a vector field, 𝑍 𝑘 's are (random or deterministic) perturbations, 𝜎 is the state-dependent diffusion matrix, and 𝜉 𝑘 's are i.i.d. standard Gaussian random variables. In the sequel, we will further decompose the perturbation as 𝑍 𝑘 = 𝑈 𝑘 , 𝑏 𝑘 , where 𝑈 𝑘 is the (zero-mean)

noise and 𝑏 𝑘 is the bias . We call this recursion the Langevin-Robbins-Monro (LRM) template, as it is reminiscent of the Robbins-Monro template for stochastic approximation [50].

The generality of the LRM template allows us to capture many existing algorithms and suggests ways to design new ones. For illustration purposes, we showcase instances of (LRM) with the following examples. Other examples (SGLD and proximal) are provided in Appendix A. In the first three examples, the vector field 𝑣 in (LRM) is GLYPH<0>r 𝑓 and 𝜎 GLYPH<17> 1 .

Example 1. The Randomized Mid-Point Method [24, 53] is an alternative discretization scheme to Euler-Maruyama and has been proposed for both overdamped and underdamped Langevin diffusion. For the overdamped case, its iterates are

where f 𝛼 𝑘 g are i.i.d. and uniformly distributed in » 0 GLYPH<150> 1 … , 𝜉 𝑘 GLYPH<150> 𝜉 0 𝑘 are standard Gaussian random variables with cross-variance p 𝛼 𝑘 𝐼 , and e r 𝑓 is a noisy evaluation of r 𝑓 . To cast (RMM) in the LRM template, we set 𝑈 𝑘 , 1 B e r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' and 𝑏 𝑘 , 1 B r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' . GLYPH<3>

Example 2. Inspecting the update rule of (RMM), we see that it requires two gradient oracle calls at each iteration. Inspired by the optimistic gradient methods in optimization and online learning [16, 47, 49], we propose to 'recycle' the past gradients:

where f 𝛼 𝑘 g , 𝜉 𝑘 GLYPH<150> 𝜉 0 𝑘 , and e r 𝑓 are the same as in (RMM). This is again an LRM scheme with 𝑈 𝑘 , 1 B e r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' and 𝑏 𝑘 , 1 B r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' .

Notice that (ORMM) requires one gradient oracle, thereby reducing the per-iteration cost of (RMM) by 2. To our knowledge, the scheme (ORMM) is new. GLYPH<3>

Example 3. In addition to the simple (stochastic) Euler-Maruyama discretization in (SGLD), there exists a class of more sophisticated discretization methods of (LD) known as higher-order integrators. The Stochastic Runge-Kutta method [34] is an example of an order 1 GLYPH<149> 5 integrator, with iterates

where 𝜉 𝑘 , 1 and 𝜉 0 𝑘 , 1 are independent standard Gaussian random variables, and 𝑐 1 GLYPH<150> 𝑐 2 GLYPH<150> 𝑐 3 are suitably chosen integrator constants. This algorithm is an LRM scheme with 𝑈 𝑘 , 1 B 1 2 ' e r 𝑓 ' ℎ 1 ' GLYPH<0>r 𝑓 ' ℎ 1 '' , 1 2 ' e r 𝑓 ' ℎ 2 '' GLYPH<0> r 𝑓 ' ℎ 2 '' and 𝑏 𝑘 , 1 B 1 2 'r 𝑓 ' ℎ 1 ' , r 𝑓 ' ℎ 2 '' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' .

Example 4. The Mirror Langevin algorithm [1, 25, 61], which is the sampling analogue of the celebrated mirror descent scheme in optimization [5, 43], uses a strongly convex function 𝜙 to adapt to a favorable local geometry. In the dual space (i.e., the image of r 𝜙 ), its iterates follow

where 𝜙 GLYPH<3> is the Fenchel dual of 𝜙 [52]. In our framework, (ML) fits into (LRM) by taking 𝑣 = GLYPH<0>r 𝑓 GLYPH<14> r 𝜙 GLYPH<3> and 𝜎 = 'r 2 𝜙 GLYPH<3> ' GLYPH<0> 1 GLYPH<157> 2 . Additionally, one can also consider a stochastic version of (ML) with noisy evaluations of r 𝑓 .

3 Technique Overview: A Dynamical System Perspective

The goal of our paper is to provide last-iterate guarantees for the general LRM schemes introduced in Section 2. There are two equivalent, commonly considered, ways of characterizing the dynamics of the iterates of an LRM scheme. The first one is to view the iterates f 𝑥 𝑘 g 𝑘 2 ℕ as a random trajectory in ℝ 𝑑 , which is perhaps the most natural way of describing a sampling algorithm. The second way is to view the distributions f 𝜌 𝑘 g 𝑘 2 ℕ of f 𝑥 𝑘 g 𝑘 2 ℕ as a deterministic trajectory in the Wasserstein space .

With these two characterizations in mind, in this section, we will devise a new framework based on the dynamical system theory and present its high-level ideas.

To understand our novelty, it is important to contrast our framework to the existing Wasserstein viewpoint towards Langevin-based sampling algorithms. Following the seminal work of Otto [44], one can view a sampling algorithm as the discretization of a class of well-studied dynamical systemsgradient flows . This viewpoint suggests using Lyapunov arguments, which has become the predominant approach in much prior work.

Despite its appealing nature, in the rest of this section, we will argue that Lyapunov analysis of gradient flows is in fact not suited for studying generic non-convex sampling. In particular, we will show how our new framework is motivated to overcome the several important limitations of gradient flow analysis. Finally, we give a high-level overview of the techniques used in our paper.

§ Langevin Diffusion as Gradient Flows. We denote by 𝜌 𝑡 the probability density of 𝐿 𝑡 in (LD), and consider the continuous curve 𝑡 ↦! 𝜌 𝑡 in the Wasserstein space 𝕎 2 . In their seminal works, Jordan et al. [27] and Otto [44] discover that this curve is the (exact) gradient flow of the relative entropy functional; that is, defining the functional 𝐹 : 𝜌 ↦! 𝐷 KL GLYPH<0> 𝜌 k 𝑒 GLYPH<0> 𝑓 GLYPH<1> , one has 𝜕 𝑡 𝜌 𝑡 = GLYPH<0> grad 𝐹 ' 𝜌 𝑡 ' , where ' grad ' is the gradient in the Wasserstein sense. This gradient flow viewpoint of (LD) thus provides a clear link between sampling in ℝ 𝑑 and optimization in 𝕎 2 . Indeed, this suggests that the relative entropy is a natural choice for the Lyapunov function of the discrete-time sampling algorithm, which is a prominent approach for analyzing sampling algorithms in recent years [4, 21, 58].

Although the gradient flow viewpoint has led to a sequence of breakthroughs, it has a few important shortcomings:

(a) The usual Lyapunov-type analysis for sampling algorithms focuses on bounding the change in relative entropy across iterations. This is extremely challenging when one considers more advanced sampling algorithms, as one has to understand the effect of the additive bias and noise of the algorithm on the change of relative entropy. Crucially, this makes the Lyapunov analysis applicable only to the simple Euler-Maruyama discretization of (LD), 2 i.e., 𝑥 𝑘 , 1 = 𝑥 𝑘 GLYPH<0> 𝛾 𝑘 , 1 r 𝑓 ' 𝑥 𝑘 ' , √︁ 2 𝛾 𝑘 , 1 𝜉 𝑘 , 1 , and fails to capture more advanced and biased sampling schemes such as Examples 1-4. Even for the simple (SGLD), the presence of stochastic gradients significantly complicates the Lyapunov analysis and requires extra assumptions such as convexity [21] or uniform spectral gap [48].

(b) This gradient flow-based analysis often requires an extra averaging step to decrease the relative entropy (see, e.g., [4]). This is the main reason why many existing works provide guarantees only on the averaged iterates ( ¯ 𝜌 𝑘 B 1 𝑘 P 𝑘 𝑖 = 1 𝜌 𝑖 ) instead of the last ones ( 𝜌 𝑘 ).

In this paper, we overcome these limitations by introducing a new perspective, whose two ingredients are as follows.

§ Wasserstein Asymptotic Pseudotrajectories. Anotion that will play a pivotal role in our analysis is the Wasserstein asymptotic pseudotrajectory (WAPT), which is a measure of 'asymptotic closeness' in the Wasserstein sense, originally defined by Benaïm and Hirsch [7] for metric spaces:

Definition 1 (Wasserstein asymptotic pseudotrajectory) . We say the stochastic process ' 𝑋 𝑡 ' 𝑡 GLYPH<21> 0 is a Wasserstein asymptotic pseudotrajectory (WAPT) of the SDE

Here, Φ ' 𝑡 ' 𝑠 is the solution of the SDE at time 𝑠 initialized at 𝑋 𝑡 , and 𝑊 2 is the 2-Wasserstein distance.

Despite the seemingly convoluted definition, WAPT can be intuitively understood as follows: Let f 𝑥 𝑘 g 𝑘 2 ℕ be the iterates of a sampling scheme. Then, (1) simply posits that for sufficiently large

if, for all 𝑇 ¡ 0 ,

Dynamical

Perspective

Dynamics

via WAPT

(Theorem 1)

Dissipativity

(Theorem 3)

Weak

Dissipativity

(Theorem 4)

Stability

(Eq. (10))

+

Last-Iterate Convergence

in

𝕎

2

(Theorem 2)

𝑚 , one cannot distinguish between the 'tail' iterates f 𝑥 𝑘 g 𝑘 GLYPH<21> 𝑚 versus the SDE solution starting at 𝑥 𝑚 , up to arbitrarily small error measured in terms of the Wasserstein distance. Since we are only interested in the asymptotic behavior of 𝑥 𝑘 , these controls on the tail iterates will suffice to conclude the last-iterate convergence. 3

Importantly, from the perspective of WAPT, the Langevin diffusion (LD) (or more generally, Φ ' 𝑡 ' 𝑠 ) is simply viewed as a generic dynamical system and not as a gradient flow. In particular, relative entropy will play no role throughout our analysis , thereby resolving issue (b).

§ Langevin-Robbins-Monro Schemes. We have seen that the LRM template in Section 2 is capable of capturing a broad range of existing and new algorithms in a unified way. To resolve the remaining issue (a), we will further rely on the LRM template: for proving that (LRM) generates a WAPT of the corresponding SDE, we show that the key condition (1) in WAPT can be reduced to checking an easy-to-verify bound on the perturbation terms 𝑍 𝑘 .

To achieve this, the most important step in our proof, which distinguishes our analysis from all existing works in non-convex sampling, is the construction of the so-called Picard process , the natural generalization of the Picard's successive approximation method [17] from ordinary differential equations to stochastic differential equations. In the stochastic approximation literature, similar techniques have been successfully applied to study optimization and games in various settings such as on Riemannian or primal-dual spaces [26, 28, 37]. The application to sampling has also been previously explored by Bubeck et al. [11], Chau et al. [12] in different contexts. What distinguishes our work from the existing literature is the advantage of generalizing the Picard process to encompass a vastly wider class of algorithms, specifically the LRM schemes. Moreover, the integration of the Picard process with the theory of WAPT plays a pivotal role in our analysis, and both of these aspects present original contributions.

§ Framework overview. To conclude, for proving last-iterate convergence, we proceed as follows:

For a given LRM scheme f 𝑥 𝑘 g 𝑘 2 ℕ , we first construct a continuous-time trajectory ' 𝑋 𝑡 ' 𝑡 GLYPH<21> 0 via interpolating the iterates (see (3)).

We prove that ' 𝑋 𝑡 ' constitutes a WAPT of the SDE (see Theorem 1). This step relies heavily on the construction of the aforementioned Picard process.

By invoking the dynamical system theory of Benaïm and Hirsch [7], the convergence of LRM schemes reduces to simply checking the stability condition (Theorem 2). In the Wasserstein space, this condition translates into boundedness of the second moments of the iterates f 𝑥 𝑘 g , for which there is a plethora of approaches; we present two such methods in Section 5.

Fig. 1 depicts a high-level overview of the ingredients needed in our framework, and their corresponding theorems.

4 The Dynamics of Langevin-Robbins-Monro Schemes

In this section, we view (LRM) as a noisy and biased discretization of (LD). To make this analogy precise, let ' 𝐵 𝑡 ' 𝑡 GLYPH<21> 0 be a Brownian motion defined on a filtered probability space with filtration

' ℱ 𝑡 ' 𝑡 GLYPH<21> 0 , and define 𝜏 𝑘 = P 𝑘 𝑛 = 1 𝛾 𝑛 to be the effective time that has elapsed at the iteration 𝑘 . Using the Brownian motion, we can rewrite (LRM) as

assuming that the filtration satisfies 𝑍 𝑘 2 ℱ 𝜏 𝑘 . 4 The (continuous-time) interpolation ' 𝑋 𝑡 ' 𝑡 GLYPH<21> 0 of f 𝑥 𝑘 g 𝑘 2 ℕ is then defined as the adapted process

In addition, for a fixed 𝑡 , consider the Brownian motion ' 𝐵 ' 𝑡 ' 𝑠 ' 𝑠 GLYPH<21> 0 where 𝐵 ' 𝑡 ' 𝑠 B 𝐵 𝑡 , 𝑠 GLYPH<0> 𝐵 𝑡 , and define the Langevin flow ' Φ ' 𝑡 ' 𝑠 ' 𝑠 GLYPH<21> 0 as the (strong) solution of (SDE) initialized at 𝑋 𝑡 . It is important to note that Φ ' 𝑡 ' and 𝑋 are synchronously coupled by sharing the same Brownian motion.

4.1 Technical Assumptions and Requirements

We now introduce the basic technical assumptions and discuss their generality.

Assumption 1. The vector field 𝑣 is 𝐿 -Lipschitz, and satisfies h 𝑥GLYPH<150> 𝑣 ' 𝑥 'i GLYPH<20> 𝐶 𝑣 ' 1 , k 𝑥 k' for some 𝐶 𝑣 ¡ 0 . Moreover, 𝜎 is 𝐿 -Lipschitz and is bounded in Hilbert-Schmidt norm.

Lipschitzness of 𝑣 is a standard assumption and is also required to ensure the existence of a unique strong solution of (SDE). The second assumption on the vector field is exceedingly weak and when 𝑣 = GLYPH<0>r 𝑓 , is satisfied even for distributions without moments. The assumptions on diffusion coefficient 𝜎 are already satisfied when 𝜎 GLYPH<17> 1 , and we show that it holds for practical schemes such as Example 4.

Assumption 2. The Robbins-Monro summability conditions hold: P 1 𝑘 = 1 𝛾 𝑘 = 1 and P 1 𝑘 = 1 𝛾 2 𝑘 GLYPH<159> 1 . Moreover, for some constant 𝑃 to be defined in (20) , we have

The Robbins-Monro step size conditions are standard in the non-convex sampling literature [4, 20, 29, 30]. For (4), it can be verified that condition is satisfied even for slowly-decreasing step sizes such as 𝛾 𝑘 / ' p 𝑘 log 𝑘 ' GLYPH<0> 1 , which hence is not restrictive.

Assumption 3. The noises f 𝑈 𝑘 g 𝑘 2 ℕ form a martingale difference sequence, i.e., 𝔼 » 𝑈 𝑘 , 1 j 𝑈 𝑘 … = 0 , and have uniformly bounded second moments. In addition, the bias terms satisfy

A martingale difference sequence is more general than an i.i.d. sequence, allowing the noise to be state-dependent. The bias condition (5) simply states that the bias shall not overpower the signal 𝑣 ' 𝑥 𝑘 ' , and, as we show later, is satisfied by all our examples.

4.2 From Discrete to Continuous: LRM Schemes and WAPTs

We are now in a position to state our main theorems. Our first result below establishes a precise link between the discrete-time (LRM) and the continuous-time (SDE).

Theorem 1. Under Assumptions 1-3, the interpolation (3) of an LRM scheme is a Wasserstein asymptotic pseudotrajectory of (SDE) .

§ Sketch of the Proof for Theorem 1. The proof of this theorem is heavily based on the notion of the Picard process and iterate moment bounds. The complete proof can be found in Appendix C.

§ Step 1: The Picard Process. For a fixed 𝑡 ¡ 0 , recall the construction of the interpolation (3) and the Langevin flow. Central to our analysis is the Picard process , defined as

The Picard process is adapted and is (synchronously) coupled with the Langevin flow and the interpolation. We think of the Picard process as one step of the Picard iteration for successive approximations to solve ODEs. This means, intuitively, that its trajectory should be close to the original interpolation, as well as to that of the Langevin flow, playing the role of a 'bridge'.

Fix 𝑇 ¡ 0 . For 𝑠 2 » 0 GLYPH<150>𝑇 … , we decompose the distance between the interpolation 𝑋 𝑡 in (3) and the Langevin flow as

We now bound each term of the decomposition. By synchronous coupling of the processes, Lipschitzness of 𝑣 , and Itô isometry, Lemma 3 bounds the first term as

This will be suitable for later use of Grönwall's lemma.

§ Step 2: Accumulated Noise and Bias. For the rest of the proof, we need some extra notation. Define 𝑚 ' 𝑡 ' B sup f 𝑘 GLYPH<21> 0 : 𝜏 𝑘 GLYPH<20> 𝑡 g and the piecewise-constant process 𝑋 𝑡 B 𝑥 𝑚 ' 𝑡 ' . Going back to the second term of (7), observe that

where Δ 𝑍 ' 𝑡GLYPH<150> 𝑠 ' is the accumulated noise and bias from time 𝑡 to time 𝑡 , 𝑠 . It is expected that k Δ 𝑍 ' 𝑡GLYPH<150> 𝑠 'k eventually becomes negligible, since the step size becomes small. The next lemma confirms this intuition.

Lemma 1. Suppose Assumptions 1-3 hold. Then, for any fixed 𝑇 ¡ 0 we have

§ Step 3: Gradient Moment Bounds. Based on (9) and Lemma 1, bounding the distance between the Picard process and the interpolation essentially reduces to bounding how much the discrete algorithm 'moves' during one iteration in expectation. This, in turn, depends on how large the moments of k 𝑣 ' 𝑥 𝑘 'k grow per iteration, which is controlled by the following lemma:

Lemma2. Let f 𝑥 𝑘 g 𝑘 2 ℕ be the iterates of (LRM) and suppose Assumptions 1-3 hold. Then, 𝔼 k 𝑥 𝑘 k 2 = O' 1 GLYPH<157> 𝛾 𝑘 , 1 ' . This in turn implies 𝔼 k 𝑣 ' 𝑥 𝑘 'k 2 = O' 1 GLYPH<157> 𝛾 𝑘 , 1 ' and 𝔼 k 𝑏 𝑘 , 1 k 2 = O' 𝛾 𝑘 , 1 ' .

Using this lemma and Lemma 1 we can obtain 𝐴 𝑡 : = sup 0 GLYPH<20> 𝑠 GLYPH<20> 𝑇 𝔼 k 𝑋 𝑡 , 𝑠 GLYPH<0> 𝑌 ' 𝑡 ' 𝑠 k 2 ! 0 as 𝑡 ! 1 , which shows that the Picard process gets arbitrarily close to the interpolation as 𝑡 !1 .

§ Step 4: Concluding the Proof. Let us go back to the decomposition (7). Taking expectation and using (8) and Grönwall's lemma, we obtain 𝔼 »k 𝑋 𝑡 , 𝑠 GLYPH<0> Φ ' 𝑡 ' 𝑠 k 2 … GLYPH<20> 4 𝐴 𝑡 exp ' 𝑇 2 𝐿 2 ' GLYPH<150> Thus,

As we coupled 𝑋 𝑡 , 𝑠 and Φ ' 𝑡 ' 𝑠 in a specific way (via synchronizing the Brownian motions), we directly get an upper bound on the Wasserstein distance. GLYPH<4>

5 Last-Iterate Convergence of Sampling Schemes

In this section we focus on last-iterate convergence of LRM schemes in Wasserstein space. We first explore the interplay between the convergence of WAPTs and stability . We then show that the existing stability results for simple Euler-Maruyama discretization of the Langevin diffusion can be extended, with little to no extra assumptions, to the class of LRM schemes in Section 2. This in turn readily implies the last-iterate convergence of a wide class of LRM schemes.

5.1 From WAPTs to Convergence in 𝕎 2

Since convergence of the distribution of 𝑥 𝑘 to 𝜋 in Wasserstein distance implies convergence of the second moments of 𝑥 𝑘 to that of 𝜋 [3], convergence in the Wasserstein space should at least require:

It turns out that, for WAPTs, the exceedingly weak necessary condition (10) is also sufficient :

Theorem 2. Let ' 𝑋 𝑡 ' be a Wasserstein asymptotic pseudotrajectory of the Langevin diffusion (LD) generated by an LRM scheme f 𝑥 𝑘 g via (3) . Then 𝑊 2 ' 𝑥 𝑘 GLYPH<150> 𝜋 ' ! 0 if and only if (10) holds.

Proof. The proof relies on the structure of compact sets in the Wasserstein space and limit-set theorems for dynamical systems [7]. Specifically, the closure of bounded subsets of 𝕎 2 is compact [3], so condition (10) implies that ' law ' 𝑋 𝑡 '' 𝑡 GLYPH<21> 0 is pre-compact in 𝕎 2 . Moreover, Assumption 1 implies that the Langevin flow is globally integrable. Thus, ' law ' 𝑋 𝑡 '' 𝑡 GLYPH<21> 0 is a pre-compact WAPT of a globally integrable flow, and we can apply the limit-set theorem for metric spaces [7, Theorem 0.1] to conclude that the limit-set of ' law ' 𝑋 𝑡 '' 𝑡 is an internally chain transitive (ICT) set .

Next, we show that for the case of the Langevin flow, the only ICT set is f 𝜋 g , implying the desired convergence of our theorem. To see this, define 𝑉 'GLYPH<1>' = 𝐷 KL 'GLYPH<1> j 𝜋 ' . It can be observed that 𝑉 is a Lyapunov function for (LD), whose value is strictly decreasing along the flow (as the time derivative of 𝑉 along the flow is negative of the relative Fisher information, which is strictly positive for all measures other than 𝜋 ). Thus, all requirements of [6, Prop. 6.4] are satisfied, showing that the only point in the ICT set is 𝜋 . This also shows the uniqueness of the stationary distribution of (LD). GLYPH<4>

Remark. From the proof of Theorem 1, we observe that the supremum of the Wasserstein distance between ' 𝑋 GLYPH<1> ' » 𝑡GLYPH<150> 𝑡 , 𝑇 … and ' Φ ' 𝑡 ' GLYPH<1> ' » 0 GLYPH<150>𝑇 … typically scales exponentially with 𝑇 , which is common for weak approximation error in the literature, see [40]. Despite the exponential dependence on 𝑇 , the convergence of the last iterate is assured by Theorem 2 without a need of a uniform control in 𝑇 . This is primarily attributed to the adoption of a dynamical system viewpoint and the application of corresponding tools, effectively harnessing the paradigm established by Benaïm and Hirsch.

Theorems 1-2 in tandem thus show that, as long as an LRM scheme satisfies Assumptions 1-3 and the moment condition (10), the desirable last-iterate convergence in 𝕎 2 is immediately attained. Therefore, in the rest of this section, we turn our focus to establishing (10) for LRM schemes.

5.2 Bounded Moments of LRM Schemes

There is a long history of study on conditions that ensure (10) for iterative algorithms, which has culminated in the so-called dissipativity properties. We consider two such examples below.

Assumption 4 (Dissipativity) . There exist constants 𝛼 ¡ 0 and 𝛽 GLYPH<21> 0 such that

Under Assumption 4, it is classical that (10) holds for the simple Euler-Maruyama discretization of (LD) with deterministic or stochastic gradient oracles [23, 29, 30, 38, 48, 51, 54]. These studies, however, cannot handle non-zero bias , which, as seen in Examples 1-3, is crucial for incorporating more advanced sampling schemes.

To this end, our next result shows that for a wide class of LRM schemes, the stability (10) essentially comes for free under Assumption 4. The proof is provided in Appendix D.

Theorem 3. Let 𝑣 be a vector field satisfying Assumptions 1 and 4 and 𝜎 be a diffusion coefficient satisfying Assumption 1, and let f 𝑥 𝑘 g be an LRM scheme. Assume that lim 𝑘 !1 𝛾 𝑘 = 0 , sup 𝑘 𝔼 k 𝑈 𝑘 k 2 GLYPH<159> 1 , and the bias satisfies (5) . Then, the stability condition (10) holds for f 𝑥 𝑘 g .

A weaker notion of dissipativity that has been studied in the literature is:

Assumption 5 (Weak dissipativity) . There exist constants 𝛼 ¡ 0 , 𝜅 2 ' 0 GLYPH<150> 1 … , and 𝛽 GLYPH<21> 0 such that

When 𝜅 = 1 , Assumption 5 is simply Assumption 4. As opposed to Assumption 4, which requires quadratic growth of 𝑓 outside a compact set (when 𝑣 = GLYPH<0>r 𝑓 ), Assumption 5 only entails superlinear growth and therefore is considerably weaker.

For Euler-Maruyama discretization of (LD) with deterministic gradients, [20] prove that Assumption 5 is sufficient to guarantee bounded moments of the iterates. As for a generic LRM scheme, we consider the following general condition on the bias terms, which will suffice to cover all our examples in Section 2: For some constant 𝑐 ,

where 𝑈 0 𝑘 , 1 is an extra noise term, and 𝜉 0 𝑘 , 1 is a standard Gaussian independent of the noises and 𝜉 𝑘 . The price to pay with the weaker Assumption 5, however, is that we need to assume sub-Gaussianity of the noise. For a proof, see Appendix D.

Theorem 4. Let 𝜋 / 𝑒 GLYPH<0> 𝑓 be the target distribution, where 𝑣 = GLYPH<0>r 𝑓 satisfies Assumptions 1 and 5, and let f 𝑥 𝑘 g be an LRM scheme. Assume that lim 𝑛 !1 𝛾 𝑘 = 0 , the noises 𝑈 𝑘 and 𝑈 0 𝑘 are sub-Gaussian, and the bias term of f 𝑥 𝑘 g satisfies (11) . Then, (10) holds for f 𝑥 𝑘 g in when (i) 𝜎 GLYPH<17> 1 , or (ii) 𝑓 is Lipschitz and the LRM follows the Mirror Langevin algorithm (Example 4).

5.3 Examples of Convergent LRM Schemes

We now illustrate the use of Theorems 1-4 on our examples in Section 2.

Proposition 1. Under Assumption 1 and noise with uniformly bounded second moments, the following holds for Examples 1-6: (i) The bias has the form (11) and satisfies (5) , (ii) As a result, under Assumptions 2 and 3, Examples 1-6 produce iterates that generate a WAPT of (SDE) . (iii) Under the additional conditions of Theorem 3 or Theorem 4, Examples 1-6 enjoy last-iterate convergence to the target distribution in Wasserstein distance.

5.4 Comparison to Existing Work

We now give a more detailed comparison of our results to existing literature; a summary is given in Table 1, and additional comparison with prior works can be found in Appendix B.

§ Guarantees for LRM Schemes. Lamberton and Pages [29] and Lemaire [30] study the simple Euler-Maruyama discretization of (LD) with deterministic gradients (i.e., 𝑈 𝑘 = 𝑏 𝑘 = 0 ) and establish the weak convergence of the average iterates under a moment condition that is slightly weaker than (10). 5 Their analysis is further extended by [54] to incorporate stochastic gradients. Later, the last-iterate convergence of the simple Euler-Maruyama discretization of (LD) is studied by [20], who prove the convergence in the total variation distance under Assumption 5. Another work on a similar setting as [20] is [8], where the convergence criterion is given in an integral probability metric (IPM) [42] of the form 𝑑 B ' 𝜇GLYPH<150> 𝜈 ' B sup 𝜑 2B j 𝔼 𝜇 𝜑 GLYPH<0> 𝔼 𝜈 𝜑 j for a certain class of test functions B that is known to imply weak convergence, but not convergence in total variation or Wasserstein distances.

Compared to these results, our guarantees possess the following desirable features:

The convergence is always on the last iterates instead of the average iterates.

As we tolerate biased algorithms, the class of LRM schemes we consider is significantly more general than the ones in existing work.

Finally, we note that our results are incomparable to the recent work of Balasubramanian et al. [4], who derive the same result as in [29, 30], i.e., average-iterate, weak convergence, deterministic Euler-Maruyama discretization. A remarkable feature of the analysis in [4] is that it does not require any bounded moments, and, in particular, their bounds can be applied to target distributions with unbounded variance. However, the downside of [4] is that, in the presence of 𝑈 𝑘 and 𝑏 𝑘 , their analysis produces a bound that does not vanish as 𝑘 !1 ; see [4, Theorem 15]. In contrast, our framework can tolerate quite general 𝑈 𝑘 and 𝑏 𝑘 , gives stronger guarantees ( 𝑊 2 vs. weak convergence; last-iterate vs. average-iterate).

§ On Analysis Techniques. While, to our knowledge, our framework is significantly different from previous works on sampling, we acknowledge that similar ideas of creating an auxiliary process in-between the iterates and the continuous-time flow is not entirely new and has been touched upon in the literature, e.g., [10, 12]. That being said, our specific approach in building the Picard process and its development into a wider array of algorithms, i.e., Langevin-Robbins-Monro schemes, undoubtedly plays a pivotal role in our analysis. Moreover, the integration of the Picard process with the theory of asymptotic pseudo-trajectories offers dual benefits to our study, and we view these as our unique contributions to this area of research.

Furthermore, the novel Picard process gives a significant advantage in all of our results. The work of [8] also hinges on dynamical system theory-related ideas. Yet, missing the critical step of the Picard process has seemingly resulted in much weaker findings compared to our work. This observation is not meant as a critique; rather, it merely highlights the potency of the unique method we have integrated into our study.

6 Concluding Remarks

In this paper, we provided a new, unified framework for analyzing a wide range of sampling schemes, thus laying the theoretical ground for using them in practice, as well as motivating new and more efficient sampling algorithms that enjoy rigorous guarantees. We built on the ideas from dynamical system theory, and gave a rather complete picture of the asymptotic behavior of many first-order sampling algorithms. In short, our results help with the following:

Validating existing methods: Methods like mirror Langevin and randomized mid-point currently lack even asymptotic guarantees in fully non-convex scenarios, such as sampling from neural network-defined distributions. Our work fills this gap by offering the first rigorous justification for these schemes, supporting practitioners in utilizing these methods confidently.

Facilitating new algorithm design: Our work motivates novel sampling methods through a straightforward verification of Assumptions 1-3. An illustrative instance involves the randomized mid-point method and Runge-Kutta integrators, wherein a substantial 50% reduction in computation per iteration can be achieved without compromising convergence by simply recycling past gradients, shown in Example 2. The balance between the benefits of saving gradient oracles and potential drawbacks remains an open question, necessitating case-by-case practical evaluation. Nevertheless, our theory provides a flexible algorithmic design template that extends beyond the current literature's scope.

While our WAPT result holds under very mild conditions, a severe limitation of our current framework is that it only applies to Langevin-based algorithms, whereas there exist numerous practical sampling schemes, such as Metropolis-Hastings, that are not immediately linked to (LD). We believe that this restriction arises as an artifact of our analysis, as the WAPT framework can in principle be applied equally well to any continuous-time dynamics. Lifting such constraint is an interesting future work.

Acknowledgments and Disclosure of Funding

This work was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program grant agreement No 815943. YPH acknowledges funding through an ETH Foundations of Data Science (ETH-FDS) postdoctoral fellowship.

References

Kwangjun Ahn and Sinho Chewi. Efficient constrained sampling via the mirror-langevin algorithm. Advances in Neural Information Processing Systems , 34:28405-28418, 2021.

Sungjin Ahn, Anoop Korattikara, and Max Welling. Bayesian posterior sampling via stochastic gradient fisher scoring. In Proceedings of the 29th International Coference on International Conference on Machine Learning , pages 1771-1778, 2012.

Luigi Ambrosio, Nicola Gigli, and Giuseppe Savaré. Gradient flows: in metric spaces and in the space of probability measures . Springer Science & Business Media, 2005.

Krishna Balasubramanian, Sinho Chewi, Murat A Erdogdu, Adil Salim, and Shunshi Zhang. Towards a theory of non-log-concave sampling: first-order stationarity guarantees for langevin monte carlo. In Conference on Learning Theory , pages 2896-2923. PMLR, 2022.

Amir Beck and Marc Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Michel Benaïm. Dynamics of stochastic approximation algorithms. In Seminaire de probabilites XXXIII , pages 1-68. Springer, 1999.

Michel Benaïm and Morris W Hirsch. Asymptotic pseudotrajectories and chain recurrent flows, with applications. Journal of Dynamics and Differential Equations , 8(1):141-176, 1996.

Michel Benaïm, Florian Bouguet, and Bertrand Cloez. Ergodicity of inhomogeneous markov chains through asymptotic pseudotrajectories. The Annals of Applied Probability , 27(5):30043049, 2017.

Espen Bernton. Langevin monte carlo and jko splitting. In Conference On Learning Theory , pages 1777-1798. PMLR, 2018.

Sébastien Bubeck, Ronen Eldan, and Joseph Lehec. Sampling from a log-concave distribution with projected langevin monte carlo. arXiv preprint arXiv:1507.02564 , 2015.

Sébastien Bubeck, Ronen Eldan, and Joseph Lehec. Sampling from a log-concave distribution with projected langevin monte carlo. Discrete & Computational Geometry , 59:757-783, 2018.

Ngoc Huy Chau, Éric Moulines, Miklos Rásonyi, Sotirios Sabanis, and Ying Zhang. On stochastic gradient langevin dynamics with dependent data streams: the fully non-convex case, 2021.

Xiang Cheng, Niladri S Chatterji, Yasin Abbasi-Yadkori, Peter L Bartlett, and Michael I Jordan. Sharp convergence rates for langevin dynamics in the nonconvex setting. arXiv preprint arXiv:1805.01648 , 2018.

Sinho Chewi. Log-concave sampling, 2023.

Sinho Chewi, Murat A Erdogdu, Mufan Bill Li, Ruoqi Shen, and Matthew Zhang. Analysis of langevin monte carlo from poincar n 'e to log-sobolev. arXiv preprint arXiv:2112.12662 , 2021.

Chao-Kai Chiang, Tianbao Yang, Chia-Jung Lee, Mehrdad Mahdavi, Chi-Jen Lu, Rong Jin, and Shenghuo Zhu. Online optimization with gradual variations. In Conference on Learning Theory , pages 6-1. JMLR Workshop and Conference Proceedings, 2012.

Earl A Coddington and Norman Levinson. Theory of ordinary differential equations . Tata McGraw-Hill Education, 1955.

Arnak S Dalalyan and Avetik G Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. arXiv preprint arXiv:1710.00095 , 2017.

Arnak S. Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic Langevin diffusions. Bernoulli , 26(3):1956 - 1988, 2020. doi: 10.3150/19-BEJ1178. URL https://doi.org/10.3150/19-BEJ1178 .

Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted langevin algorithm. The Annals of Applied Probability , 27(3):1551-1587, 2017.

Alain Durmus, Szymon Majewski, and Bła˙ zej Miasojedow. Analysis of langevin monte carlo via convex optimization. The Journal of Machine Learning Research , 20(1):2666-2711, 2019.

Murat A Erdogdu, Lester Mackey, and Ohad Shamir. Global non-convex optimization with discretized diffusions. Advances in Neural Information Processing Systems , 31, 2018.

J Hale. Asymptotic behavior of dissipative systems. In Aerican Mathematical Society , 1988.

Ye He, Krishnakumar Balasubramanian, and Murat A Erdogdu. On the ergodicity, bias and asymptotic normality of randomized midpoint sampling method. Advances in Neural Information Processing Systems , 33:7366-7376, 2020.

Ya-Ping Hsieh, Ali Kavis, Paul Rolland, and Volkan Cevher. Mirrored langevin dynamics. Advances in Neural Information Processing Systems , 31, 2018.

Ya-Ping Hsieh, Panayotis Mertikopoulos, and Volkan Cevher. The limits of min-max optimization algorithms: Convergence to spurious non-critical sets. In International Conference on Machine Learning , pages 4337-4348. PMLR, 2021.

Richard Jordan, David Kinderlehrer, and Felix Otto. The variational formulation of the fokkerplanck equation. SIAM journal on mathematical analysis , 29(1):1-17, 1998.

Mohammad Reza Karimi, Ya-Ping Hsieh, Panayotis Mertikopoulos, and Andreas Krause. The dynamics of riemannian robbins-monro algorithms. arXiv preprint arXiv:2206.06795 , 2022.

Damien Lamberton and Gilles Pages. Recursive computation of the invariant distribution of a diffusion. Bernoulli , pages 367-405, 2002.

Vincent Lemaire. Estimation récursive de la mesure invariante d'un processus de diffusion. PhD thesis, Université de Marne la Vallée, 2005.

Ruilin Li, Molei Tao, Santosh S. Vempala, and Andre Wibisono. The mirror langevin algorithm converges with vanishing bias, 2021.

Ruilin Li, Hongyuan Zha, and Molei Tao. Sqrt(d) dimension dependence of langevin monte carlo. In The International Conference on Learning Representations , 2022.

Xuechen Li, Yi Wu, Lester Mackey, and Murat A Erdogdu. Stochastic runge-kutta accelerates langevin monte carlo and beyond. Advances in neural information processing systems , 32, 2019.

Xuechen Li, Denny Wu, Lester Mackey, and Murat A. Erdogdu. Stochastic Runge-Kutta Accelerates Langevin Monte Carlo and Beyond, February 2020.

Yi-An Ma, Niladri S Chatterji, Xiang Cheng, Nicolas Flammarion, Peter L Bartlett, and Michael I Jordan. Is there an analog of nesterov acceleration for gradient-based mcmc? Bernoulli , 27(3):1942-1992, 2021.

Mateusz B Majka, Aleksandar Mijatovi´ c, and Łukasz Szpruch. Nonasymptotic bounds for sampling algorithms without log-concavity. The Annals of Applied Probability , 30(4):15341581, 2020.

Panayotis Mertikopoulos, Ya-Ping Hsieh, and Volkan Cevher. Learning in games from a stochastic approximation viewpoint. arXiv preprint arXiv:2206.03922 , 2022.

Sean P Meyn and Richard L Tweedie. Stability of markovian processes iii: Foster-lyapunov criteria for continuous-time processes. Advances in Applied Probability , 25(3):518-548, 1993.

Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability . Springer Science &Business Media, 2012.

Grigori N Milstein and Michael V Tretyakov. Stochastic numerics for mathematical physics , volume 39. Springer, 2004.

Wenlong Mou, Nicolas Flammarion, Martin J Wainwright, and Peter L Bartlett. Improved bounds for discretization of langevin diffusions: Near-optimal rates without convexity. Bernoulli , 28(3):1577-1601, 2022.

Alfred Müller. Integral probability metrics and their generating classes of functions. Advances in Applied Probability , 29(2):429-443, 1997.

AS Nemirovsky and DB Yudin. Problem Complexity and Method Efficiency in Optimization . J. Wiley @ Sons, New York, 1983.

Felix Otto. The geometry of dissipative evolution equations: the porous medium equation. Communications in Partial Differential Equations , 26, 2001.

Grigorios A Pavliotis. Stochastic processes and applications: diffusion processes, the FokkerPlanck and Langevin equations , volume 60. Springer, 2014.

Marcelo Pereyra. Proximal markov chain monte carlo algorithms. Statistics and Computing , 26 (4):745-760, 2016.

Leonid Denisovich Popov. A modification of the arrow-hurwicz method for search of saddle points. Mathematical notes of the Academy of Sciences of the USSR , 28(5):845-848, 1980.

Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory , pages 1674-1703. PMLR, 2017.

Sasha Rakhlin and Karthik Sridharan. Optimization, learning, and games with predictable sequences. Advances in Neural Information Processing Systems , 26, 2013.

Herbert Robbins and Sutton Monro. A Stochastic Approximation Method. The Annals of Mathematical Statistics , 22(3):400 - 407, 1951. doi: 10.1214/aoms/1177729586. URL https: //doi.org/10.1214/aoms/1177729586 .

Gareth O Roberts and Richard L Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli , pages 341-363, 1996.

Ralph Tyrell Rockafellar. Convex analysis . Princeton university press, 2015.

Ruoqi Shen and Yin Tat Lee. The randomized midpoint method for log-concave sampling. Advances in Neural Information Processing Systems , 32, 2019.

Yee Whye Teh, Alexandre H Thiery, and Sebastian J Vollmer. Consistency and fluctuations for stochastic gradient langevin dynamics. Journal of Machine Learning Research , 17, 2016.

Belinda Tzen, Anant Raj, Maxim Raginsky, and Francis Bach. Variational principles for mirror descent and mirror langevin dynamics, 2023.

Santosh Vempala and Andre Wibisono. Rapid convergence of the unadjusted langevin algorithm: Isoperimetry suffices. Advances in neural information processing systems , 32, 2019.

Max Welling and Yee Whye Teh. Bayesian Learning via Stochastic Gradient Langevin Dynamics. In ICML , page 8, 2011.

Andre Wibisono. Sampling as optimization in the space of measures: The langevin dynamics as a composite optimization problem. In Conference on Learning Theory , pages 2093-3027. PMLR, 2018.

Andre Wibisono. Proximal langevin algorithm: Rapid convergence under isoperimetry. arXiv preprint arXiv:1911.01469 , 2019.

Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems , 31, 2018.

Kelvin Shuangjian Zhang, Gabriel Peyré, Jalal Fadili, and Marcelo Pereyra. Wasserstein control of mirror langevin monte carlo. In Conference on Learning Theory , pages 3814-3841. PMLR, 2020.

Kelvin Shuangjian Zhang, Gabriel Peyré, Jalal Fadili, and Marcelo Pereyra. Wasserstein control of mirror langevin monte carlo. In Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 3814-3841. PMLR, 09-12 Jul 2020. URL https: //proceedings.mlr.press/v125/zhang20a.html .

Difan Zou, Pan Xu, and Quanquan Gu. Sampling from non-log-concave distributions via variance-reduced gradient langevin dynamics. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 2936-2945. PMLR, 2019.

A Further Examples of LRM Schemes

Example 5. The classic Stochastic Gradient Langevin Dynamics [57] iterates as

where e r 𝑓 is the gradient of the negative log-likelihood of a random batch of the data. (SGLD) fits the LRM template by setting 𝑈 𝑘 , 1 B e r 𝑓 ' 𝑥 𝑘 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' , and 𝑏 𝑘 , 1 B 0 . GLYPH<3>

Example 6. The Proximal Langevin Algorithm [9, 46, 59] is defined via

This algorithm is implicit, and it is assumed that one can solve (PLA) for 𝑥 𝑘 , 1 . By setting 𝑏 𝑘 , 1 B r 𝑓 ' 𝑥 𝑘 , 1 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' and 𝑈 𝑘 , 1 B 0 , we see that this algorithm also follows the LRM template. GLYPH<3>

B Additional Related Work

Our paper studies the behavior of a wide range of Langevin-based sampling algorithms proposed in the literature in the asymptotic setting under minimal assumptions. This allows us to give last-iterate guarantees in Wasserstein distance. As stressed in Section 1, our goal is not to provide non-asymptotic rates in this general setting as the problem is inherently NP-Hard. However, given more assumptions and structures on the potential 𝑓 , there is a plethora of works which prove convergence rates for the last iterates in Wasserstein distance. In this appendix, we provide additional backgraound for these works and the methods used in the literature.

A powerful framework for quantifying the global discretization error of a numerical algorithm is the mean-square analysis framework [40]. This framework furnishes a general recipe for controlling short and long-term integration errors. For sampling, this framework has been applied to prove convergence rates for Langevin Monte-Carlo (the Euler-Maruyama discretization of (LD)) in the strongly-convex setting [32, 34]. Similar to our work, the convergence obtained in these works is last-iterate and in Wasserstein distance. One of the essential ingredients in the latter work is the contraction property of the SDE, which is ensured by the strong convexity assumption. This, in turn, implies strong non-asymptotic convergence guarantees.

It is an interesting future direction to study the combination of the Mean-Squared analysis together with the Picard process and its applicability to more sophisticated algorithms (such as LRM schemes with bias and noise), as well as non-convex potentials.

As explained in Section 3, one of the main themes in proving error bounds for sampling is the natural relation between sampling and optimization in the Wasserstein space. This point of view, when applied to strongly-convex potentials, has produced numerous non-asymptotic guarantees; see [14, 18] for a recent account and the references therein. Note that strong convexity is crucial for the analysis used in the aforementioned work. Moreover, the error bounds for biased and noisy discretizations do not decrease with the step-size or iteration count; see [18, Theorem 4, Eqn. (14)]. This means that while the bound is non-asymptotic, it does not automatically result in an asymptotic convergence. Finally, we stress that these approaches are orthogonal to our techniques: We view a sampling algorithm as a (noisy and biased) discretization of a dynamical system (and not necessarily a gradient flow), and use tools from dynamical system theory to provide asymptotic convergence results.

C Proofs for Section 4

C.1 Proof of Theorem 1

In this appendix, we bring the detailed proof of Theorem 1. Recall that we interpolate the iterates of the LRM scheme f 𝑥 𝑘 g as

Moreover, for a fixed 𝑡 ¡ 0 , we considered the Brownian motion 𝐵 ' 𝑡 ' 𝑠 = 𝐵 𝑡 , 𝑠 GLYPH<0> 𝐵 𝑡 , and constructed two important processes: the Langevin flow defined via

and the Picard process (6) constructed as

Let us fix 𝑇 ¡ 0 , and for 𝑠 2 » 0 GLYPH<150>𝑇 … decompose the distance between the interpolation and the Langevin flow as

where we have used k 𝑎 , 𝑏 k 2 GLYPH<20> 2 k 𝑎 k 2 , 2 k 𝑏 k 2 . We now bound each term of this decomposition. Notice that due to the synchronous coupling of the processes, the Brownian motion cancels out in the differences.

The first term controls how close the Picard process is to the Langevin flow, and is bounded in the following lemma.

Lemma 3. For fixed 𝑡GLYPH<150>𝑇 ¡ 0 and 0 GLYPH<20> 𝑠 GLYPH<20> 𝑇 , the distance of the Picard process and the Langevin flow is bounded as

Proof. By the auxiliary Lemma 4 below, Lipschitzness of 𝑣GLYPH<150>𝜎 , Itô isometry (see, e.g., [62]) and 𝑠 GLYPH<20> 𝑇 , we have

For the rest of the proof, we need to define the continuous-time piecewise-constant processes 𝑋 ' 𝜏 𝑘 , 𝑠 ' = 𝑋 𝑘 , 𝛾 ' 𝜏 𝑘 , 𝑠 ' = 𝛾 𝑘 , 1 , 𝑍 ' 𝜏 𝑘 , 𝑠 ' = 𝑍 𝑘 , 1 , and 𝑍 ' 𝜏 𝑘 , 𝑠 ' = 𝔼 » 𝑍 𝑘 , 1 j ℱ 𝜏 𝑘 , 𝑠 … , for 0 GLYPH<20> 𝑠 GLYPH<159> 𝛾 𝑘 , 1 . Also, let 𝑚 ' 𝑡 ' = sup f 𝑘 GLYPH<21> 0 : 𝜏 𝑘 GLYPH<20> 𝑡 g so that 𝜏 𝑚 ' 𝑡 ' GLYPH<20> 𝑡 GLYPH<159> 𝜏 𝑚 ' 𝑡 ', 1 .

To bound the second term in (7), we have seen that

where Δ 𝑍 ' 𝑡GLYPH<150> 𝑠 ' plays the role of accumulated noise and bias from time 𝑡 to 𝑡 , 𝑠 , and is defined as

with 𝑘 = 𝑚 ' 𝑡 , 𝑠 ' and 𝑛 = 𝑚 ' 𝑡 ' . We therefore have

For bounding the term inside the integral, we have

We have used the fact that

Notice that since conditional expectation is a projection in 𝐿 2 , we have 𝔼 k 𝑍 ' 𝑢 'k 2 GLYPH<20> 𝔼 k 𝑍 ' 𝑢 'k 2 . Using this fact, along with boundedness of 𝜎 'GLYPH<1>' by 𝐶 𝜎 , and Lemma 2 we get

for some constant 𝐶 ¡ 0 . Plugging this estimate into (14) after taking expectation yields

Taking supremum over 𝑠 2 » 0 GLYPH<150>𝑇 … and noticing that the right-hand-side is independent of 𝑠 and 𝛾 𝑘 ! 0 , together with Lemma 1 yields

showing that the Picard process gets arbitrary close to the original interpolation, as 𝑡 !1 .

Let us return to the decomposition (7). By taking expectation and using (8) and (15) we obtain

where in the last line we have used the Grönwall lemma. Thus,

Recall that the Wasserstein distance between 𝑋 𝑡 , 𝑠 and Φ ' 𝑡 ' 𝑠 is the infimum over all possible couplings between them, having the correct marginals. As Φ ' 𝑡 ' 𝑠 has the same marginal as the Langevin diffusion started from 𝑋 𝑡 at time 𝑠 , and the synchronous coupling of the interpolation and the Langevin flow produces a specific coupling between them, we directly get

which implies as desired.

GLYPH<4>

C.2 Auxiliary Lemmas

Lemma 1. Suppose Assumptions 1-3 hold. Then, for any fixed 𝑇 ¡ 0 we have

Proof. Define Δ 𝑏 and Δ 𝑈 the same way as in (13). By Cauchy-Schwarz we have

where the last inequality comes from P 𝑘 GLYPH<0> 1 𝑖 = 𝑛 𝛾 𝑖 , 1 GLYPH<20> 𝑠 , 𝑡 , 𝑠 GLYPH<0> 𝜏 𝑘 GLYPH<20> 𝛾 𝑘 , 1 , 𝑡 GLYPH<0> 𝜏 𝑛 GLYPH<20> 𝛾 𝑛 , 1 , and 𝛾 𝑘 , 1 GLYPH<20> 𝛾 𝑛 , 1 . Noticing that conditional expectation is a contraction in 𝐿 2 and letting 𝑘 0 = 𝑚 ' 𝑡 , 𝑇 ' , we get

Now, invoking Lemma 2 yields

As 𝑡 !1 , the last quantity vanishes, since 𝛾 𝑛 ! 0 .

For the noise we have

Taking expectations and then sup, we get

Since f 𝑈 𝑖 g is a martingale difference sequence, we have that GLYPH<8>P 𝑘 GLYPH<0> 1 𝑖 = 𝑛 𝛾 𝑖 , 1 𝑈 𝑖 , 1 GLYPH<9> 𝑘¡𝑛 is a martingale. Thus, by the boundedness of the second moments of 𝑈 𝑖 , we get

Hence,

GLYPH<4>

Lemma2. Let f 𝑥 𝑘 g 𝑘 2 ℕ be the iterates of (LRM) and suppose Assumptions 1-3 hold. Then, 𝔼 k 𝑥 𝑘 k 2 = O' 1 GLYPH<157> 𝛾 𝑘 , 1 ' . This in turn implies 𝔼 k 𝑣 ' 𝑥 𝑘 'k 2 = O' 1 GLYPH<157> 𝛾 𝑘 , 1 ' and 𝔼 k 𝑏 𝑘 , 1 k 2 = O' 𝛾 𝑘 , 1 ' .

Proof. Without loss of generality, suppose 𝑣 has a stationary point at 0 . We repeatedly use the fact that 𝔼 k 𝑣 ' 𝑥 𝑘 'k 2 GLYPH<20> 𝐿 2 𝔼 k 𝑥 𝑘 k 2 . Moreover, by Assumption 1 we have h 𝑣 ' 𝑥 ' GLYPH<150> 𝑥 i GLYPH<20> 𝐶 𝑣 'k 𝑥 k , 1 ' , and k 𝜎 ' 𝑥 'k 2 𝐹 GLYPH<20> 𝐶 𝜎 .

Define 𝑎 𝑘 B 𝔼 k 𝑥 𝑘 k 2 . We have

By Assumption 3, there is some 𝐶 𝑏 ¡ 0 such that 𝔼 k 𝑏 𝑘 , 1 k 2 GLYPH<20> 𝐶 𝑏 ' 𝛾 2 𝑘 , 1 𝑎 𝑘 , 𝛾 𝑘 , 1 ' , and we have

Moreover, as p 𝑝 , 𝑞 GLYPH<20> p 𝑝 , p 𝑞 , we have

Plugging the bounds from (17) and (18) into (16) gives

where

The exact values of 𝑃 , 𝑄 , and 𝑅 are irrelevant, and we only need upper bounds for them. Assuming that 𝛾 𝑘 , 1 GLYPH<159> 1 for all 𝑘 , we replace the three quantities by

Now, define ℎ 𝑘 = 𝛾 2 𝑘 , 1 𝑎 𝑘 . The recursion (19) in terms of ℎ 𝑘 becomes

We now prove that there exists some 𝑀 ¡ 0 so that ℎ 𝑘 GLYPH<20> 𝑀𝛾 𝑘 , 1 by induction. Suppose it is the case for 𝑘 , and we prove it for 𝑘 , 1 . Using the induction hypothesis we get

For the last to be less than 𝑀𝛾 𝑘 , 2 , we have to verify or equivalently,

This is a quadratic equation in 𝑀 , and for this inequality to hold, we prove that the leading coefficient is negative, and the largest root is bounded above by some constant not depending on 𝑛 .

Negativity of the leading coefficient is equivalent to

which is implied by our assumption on the step size.

The larger root of the equation is

By our assumption on the step size that

we get that the larger root is smaller than

Letting 𝑀 : = 𝑄 , p 𝑅 gives the desired result.

The second argument of the lemma follows from Assumption 3 and the first result of the lemma. GLYPH<4>

Lemma 4. For a vector valued function 𝑔 2 𝐿 2 ' ℝ ; ℝ 𝑑 ' , one has

D Proofs for Section 5

D.1 Proof of Theorem 3

For brevity, let us write ℱ 𝑘 instead of ℱ 𝜏 𝑘 . Opening up k 𝑥 𝑘 , 1 k 2 = k 𝑥 𝑘 , 𝛾 𝑘 , 1 f 𝑣 ' 𝑥 𝑘 ' , 𝑍 𝑘 , 1 g , p 𝛾 𝑘 , 1 𝜎 ' 𝑥 𝑘 ' 𝜉 𝑘 , 1 k 2 and ignoring every term that is zero-mean under 𝔼 »GLYPH<1> j ℱ 𝑘 … , we get

Recalling (5) in Assumption 3, we have for some 𝐶 ¡ 0

Without loss of generality, assume 𝛾 𝑘 GLYPH<20> 1 and 𝔼 k 𝑥 𝑘 k 2 GLYPH<21> 1 (so that GLYPH<0> 𝔼 k 𝑥 𝑘 k 2 GLYPH<1> 2 GLYPH<21> 𝔼 k 𝑥 𝑘 k 2 ) for all 𝑘 . Then, k 𝑣 ' 𝑥 𝑘 'k 2 GLYPH<20> 𝐿 2 k 𝑥 𝑘 k 2 , together with Assumption 4 and the Cauchy-Schwartz inequality on the last term of (21), implies

for some constants 𝐶 1 GLYPH<150>𝐶 2 GLYPH<150>𝐶 3 depending on 𝐿GLYPH<150> 𝐶GLYPH<150> 𝜎GLYPH<150> 𝛼GLYPH<150> 𝛽 , and 𝑑 . Since 𝛾 𝑘 ! 0 , there exist ˜ 𝛼GLYPH<150> ˜ 𝛽 ¡ 0 and 𝑘 0 such that, for all 𝑘 GLYPH<21> 𝑘 0 ,

A simple induction yields which concludes the proof.

D.2 Proof of Theorem 4 for Constant Diffusion

Before proceeding, we need a lemma which can be distilled from [20, Proposition 8]:

Lemma 5. Suppose r 𝑓 is 𝐿 -Lipschitz. Fix 𝑥 2 ℝ 𝑑 and 𝛾 ¡ 0 , let ˜ 𝑥 , = 𝑥 GLYPH<0> 𝛾 r 𝑓 ' 𝑥 ' , √︁ 2 𝛾𝜉 . Then

Let ˜ 𝑥 𝑘 , 1 B 𝑥 𝑘 GLYPH<0> 𝛾 𝑘 , 1 r 𝑓 ' 𝑥 𝑘 ' , √︁ 2 𝛾 𝑘 , 1 𝜉 𝑘 , 1 so that 𝑥 𝑘 , 1 GLYPH<0> 𝑥 𝑘 = ˜ 𝑥 𝑘 , 1 GLYPH<0> 𝑥 𝑘 GLYPH<0> 𝛾 𝑘 , 1 ' 𝑈 𝑘 , 1 , 𝑏 𝑘 , 1 ' . Conditioned on 𝑥 𝑘 GLYPH<150>𝑈 𝑘 , 1 GLYPH<150>𝑈 0 𝑘 , 1 GLYPH<150> 𝜉 0 𝑘 , 1 , and using the 𝐿 -Lipschitzness of r 𝑓 , we get

Let 𝛿 2 ' 0 GLYPH<150> 1 ' . Since

we have

Invoking (11) an denoting 𝑐 0 , GLYPH<0> 𝐿𝛾 2 𝑘 , 1 , 1 GLYPH<1> GLYPH<1> 𝑐 , we get

where,

Recalling that √︁ 2 𝛾 𝑘 , 1 𝜉 𝑘 , 1 = ˜ 𝑥 𝑘 , 1 GLYPH<0> 𝑥 𝑘 , 𝛾 𝑘 , 1 r 𝑓 ' 𝑥 𝑘 ' , we have 𝛾 𝑘 , 1 k 𝜉 𝑘 , 1 k 2 GLYPH<20> k ˜ 𝑥 𝑘 , 1 GLYPH<0> 𝑥 𝑘 k 2 , 𝛾 2 𝑘 , 1 kr 𝑓 ' 𝑥 𝑘 'k 2 , and thus

where 𝐴 0 𝑘 = 𝐴 𝑘 , 𝑐 0 𝛾 2 𝑘 , 1 kr 𝑓 ' 𝑥 𝑘 'k 2 . Lemma 5 then implies

where 𝐴 00 𝑘 = 𝐴 0 𝑘 GLYPH<0> 𝛾 𝑘 , 1 4 kr 𝑓 ' 𝑥 𝑘 'k 2 .

We now take the expectation over 𝑥 𝑘 GLYPH<150>𝑈 𝑘 , 1 GLYPH<150>𝑈 0 𝑘 , 1 GLYPH<150> 𝜉 0 𝑘 , 1 (in other words, we are now only conditioning on 𝑥 𝑘 ). Set 𝜖 , ' 1 GLYPH<0> 𝛾 𝑘 , 1 𝐿 0 ' GLYPH<0> 1 2 GLYPH<0> 1 ¡ 0 . Since 𝑈 𝑘 , 1 GLYPH<150>𝑈 0 𝑘 , 1 GLYPH<150> 𝜉 0 𝑘 , 1 are sub-Gaussian and since 𝛾 𝑘 ! 0 , for 𝑘 sufficiently large we have

To summarize, we have shown that, conditioned on 𝑥 𝑘 ,

A simple induction à la [20, Lemma 1 & Proposition 8] then concludes the proof.

D.3 Proof of Theorem 4 for Mirror Langevin

Here, we bring the proof of Theorem 4 for the case of Example 4 and without noise. The proof for the noisy case is the same as in Appendix D.2.

Define

where 𝜉 is a standard Gaussian random variable. Let 𝑈 ' 𝑥 ' = 𝑓 'r 𝜙 GLYPH<3> ' 𝑥 '' . For a fixed 𝑥 , we have

GLYPH<4>

Notice that we have which implies

Thus, the integral, after the change of variable from 𝜉 to 𝑥 , becomes

with 𝐶 = ' 4 𝜋𝛾 ' 𝑑 GLYPH<157> 2 √︁ det r 2 𝜙 GLYPH<3> ' 𝑥 ' GLYPH<0> 1 . Now we use the smoothness of 𝑓 :

On the other hand, we have

Notice that in (37), the colored terms cancel out, and what we are left with is

As, by our assumption, r 2 𝜙 GLYPH<3> is bounded from above and below, we get the exact form as in Lemma 5. The rest of the proof is the same as in Appendix D.2. GLYPH<4>

D.4 Proof of Proposition 1

In this section, we prove that Examples 1-6 satisfy our bias conditions, which, as we have seen in Section 5, implies Proposition 1. For brevity, we write ℱ 𝑘 for ℱ 𝜏 𝑘 .

§ Proof for Example 1. For randomized mid-point method, by replacing e r 𝑓 ' 𝑥 𝑘 ' and e r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' with r 𝑓 ' 𝑥 𝑘 ' , 𝑈 0 𝑘 , 1 and r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' , 𝑈 𝑘 , 1 respectively, we have

where f 𝛼 𝑘 g are i.i.d. and uniformly distributed in » 0 GLYPH<150> 1 … , f 𝑈 𝑘 g and f 𝑈 0 𝑘 g are noises in evaluating r 𝑓 at the corresponding points, and 𝜉 𝑘 GLYPH<150> 𝜉 0 𝑘 are independent standard Gaussians.

Notice that the Lipschitzness of r 𝑓 , and the fact that 𝛼 𝑘 GLYPH<20> 1 implies that the bias term 𝑏 𝑘 , 1 B r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' satisfies

§ Proof for Example 2. Recall that the new algorithm Optimistic Randomized Mid-Point Method has the iterates

where f 𝛼 𝑘 g , 𝜉 𝑘 GLYPH<150> 𝜉 0 𝑘 , and e r 𝑓 are the same as in (RMM), and the noise and bias are 𝑈 𝑘 , 1 B e r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' and 𝑏 𝑘 , 1 B r 𝑓 ' 𝑥 𝑘 , 1 GLYPH<157> 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' . We have

Similar to the proof for Example 6, notice that kr 𝑓 ' 𝑥 𝑘 GLYPH<0> 1 2 'k 2 GLYPH<20> 2 kr 𝑓 ' 𝑥 𝑘 GLYPH<0> 1 2 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 'k 2 , 2 kr 𝑓 ' 𝑥 𝑘 'k 2 . As 𝛾 𝑘 ! 0 , one can assume that 2 𝐿 2 𝛾 2 𝑘 , 1 GLYPH<159> 1 2 , and we get

§ Proof for Example 3. The iterates of stochastic Runge-Kutta Langevin algorithm is as follows:

where 𝜉 𝑘 , 1 and 𝜉 0 𝑘 , 1 are independent standard Gaussian random variables independent of 𝑥 𝑘 , and 𝑈 𝑘 , 1 and 𝑈 0 𝑘 , 1 are noise in the evaluation of 𝑓 .

Observe that

We have

and

We thus have

as desired.

GLYPH<4>

§ Proof for Example 4. Suppose 𝜙 is a Legendre function [52] for ℝ 𝑑 , and consider the iterates

where 𝜙 GLYPH<3> is the Fenchel dual of 𝜙 , that is, 𝜙 GLYPH<3> ' 𝑥 ' = sup 𝑦 2 ℝ 𝑑 'h 𝑥GLYPH<150> 𝑦 i GLYPH<0> 𝜙 ' 𝑦 '' . Also recall that [52]

Let 𝑣 = GLYPH<0>r 𝑓 GLYPH<14> r 𝜙 GLYPH<3> and 𝜎 = 'r 2 𝜙 GLYPH<3> ' GLYPH<0> 1 GLYPH<157> 2 . First, we mention what our assumptions imply on 𝑓 :

The Lipschitzness of 𝑣 corresponds to a similar condition in [31, A2]:

The Lipschitzness of 𝜎 in Frobenius norm corresponds to modified self-concordance in [31, A1]:

Boundedness of 𝜎 in Hilbert-Schmidt norm implies

Dissipativity and weak-dissipativity of 𝑣 corresponds to the conditions below, respectively:

If 𝑓 and 𝜙 satisfy the conditions above, then the mirror Langevin algorithm Example 4 fits into the (LRM) scheme.

Remark. Note that this version of Mirror Langevin cannot handle the case where 𝑒 GLYPH<0> 𝑓 is supported on a compact domain; in that case, the Hessian of 𝜙 has to blow up near the boundary, and will not satisfy our boundedness assumption. The version of mirror Langevin we consider in this paper, though, can be thought as an adaptive conditioning method for densities supported on ℝ 𝑑 . This setting has also been studied in the literature, see [55].

§ Proof for Example 6. The iterates of (PLA) follow

We mentioned that the bias term is 𝑏 𝑘 , 1 = r 𝑓 ' 𝑥 𝑘 , 1 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 ' . Now it remains to prove that it satisfies the conditions (5) and (11). We have

Now, notice that kr 𝑓 ' 𝑥 𝑘 , 1 'k 2 GLYPH<20> 2 kr 𝑓 ' 𝑥 𝑘 , 1 ' GLYPH<0> r 𝑓 ' 𝑥 𝑘 'k 2 , 2 kr 𝑓 ' 𝑥 𝑘 'k 2 . As 𝛾 𝑘 ! 0 , one can assume that 2 𝐿 2 𝛾 2 𝑘 , 1 GLYPH<159> 1 2 , and we get

which implies

𝔼 »k 𝑏 𝑘 , 1 k 2 j ℱ 𝑘 … GLYPH<20> 4 𝐿 2 𝛾 2 𝑘 , 1 kr 𝑓 ' 𝑥 𝑘 'k 2 , 8 𝐿 2 𝑑𝛾 𝑘 , 1 = O' 𝛾 2 𝑘 , 1 kr 𝑓 ' 𝑥 𝑘 'k 2 , 𝛾 𝑘 , 1 ' GLYPH<150> as desired. GLYPH<4>

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a general framework for studying the convergence of last-iterate, noisy, and possibly biased Langevin-like discrete time approximations to the continuous Langevin flow for sampling from a distribution.\n\nSTRENGTHS:\nThis paper is (with one small quibble which I will explain below) very well written and well explained. Using the machinery of asymptotic pseudotrajectories, it establishes asymptotic convergence of a wide range of discrete time sampling schemes to the Boltzman distribution $e^{-f}$. The advantage of the pseudotrajectory framework is that it replaces standard \"Cauchy-type\" convergence (which requires checking that an infinitely long tail of iterates converge) with a weaker notion of convergence that only requires \"finite length\" tails. \n\nThe upshot is that, due to cited source [6], this weaker sense of convergence is sufficient for convergence to a limit point of the original flow $\\Phi$, provided that $\\Phi$ has a unique fixed point. This allows for much more freedom in the analysis, and hence encompasses a wider range of conditions. The technique also has advantages of discretization-based approaches as explained by the author.\n\nWEAKNESSES:\nMy biggest complain with the paper is that perhaps the most interesting part of the framework is the 2-line proof of Theorem 2 relegated to the appendix. This argument relies on  the limit-set characterization of [6] to show that asymptotic pseudotrajectories are sufficient to ensure convergence to the fixed point of the Langevin flow, namely the correct Boltzman distribution. I think the authors should explain this point in the body, and explain which theorem of [6] is being invoked (I suspect Thm 0.1), and how it's conditions are met. That way, the reader understands not simply how the proof framework of the paper works, but why we should believe that is somehow the \"correct\" one.\n\nQUESTIONS:\nWhat is the theorem of [6] being invoked here? Can the authors explain its conditions and why they are met.\n\nLIMITATIONS:\nThis work is purely asymptotic, and provides no rates of convergence.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors offer theoretical guarantees on the convergence of the last iterate of a very generic class of sampling methods to the stationary  distribution for a very large classe of sampling schemes in non-convex settings. This is achieved by showing that a large class of discrete sampling schemes can be mapped to continuous time ones and then to a Wasserstein asymptotic pseudo-trajectory. The distribution at large times of this process then converges to the stationary one under quite general assumptions on the Langevin noise and an opportune annealing scheme for the learning rate.\n\nSTRENGTHS:\nThe paper addresses the interesting and actively researched problem of sampling non-convex, high dimensional densities. It explores a general framework that can be specialised to a wide range of sampling schemes, thereby providing a valuable guarantee for practical sampling scenarios. \n\nNotably, the authors employ a novel proof technique, which, to the best of my knowledge, is unique in the context of sampling problems. This innovative approach contributes to the originality of the paper and distinguishes it from existing research in the field, which mostly study these problems as gradient flows. \n\nFurthermore, the paper exhibits exceptional writing quality, being highly readable, well-structured, and accessible to a broad audience. These qualities enhance its overall impact and make it easier for readers to grasp the main idea.\n\nConsidering these strengths, I strongly believe that this paper is of high caliber. It offers significant contributions to the field, with its practical relevance, original proof technique, and excellent presentation.\n\nWEAKNESSES:\nI think this is a solid paper without major weaknesses, so I just have a minor remark.\nThe main result of this paper is showing that at very large times the sampling schemes converge to the desired distribution. It would be interesting to relax this condition and obtain bounds on the Wasserstein distance for a large (but finite) number of steps.\n\nQUESTIONS:\n1. Within this framework, is it possible to extract how the typical time to convergence scales with the size of the system? (for example in the sense of obtaining a lower bound on the number of steps T(epsilon) to obtain a distance between the distribution after T steps and the stationary one less than epsilon).\n2. Related to the previous point: can you say something quantitative on the performance of ORMM beyond requiring less gradient calls?\n3. Can you comment on the case where sigma(x_k)=1/beta_k in the definition of LRM, where beta_k is positively diverging as k increase?\n\nLIMITATIONS:\nThis paper contains no applications or experiments, as it's expected by a paper of this kind.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work studies when a discretized Langevin dynamics under the Robbins-Monro-type stepsizes can converge to the Gibbs distribution. The paper obtains asymptotic results with very mild assumptions, and the framework not only includes Euler discretization, but many other sampling schemes as well, such as mirror Langevin, proximal, randomized mid-point and Runge-Kutta methods. The analysis builds upon constructing a continuous-time trajectory via interpolating the iterates, Wasserstein asymptotic pseudotrajectory and checking the stability condition by invoking the dynamical system theory.\n\nSTRENGTHS:\nThe paper is built upon solid mathematical analysis and it is a nice contribution to the vast literature of Langevin algorithms in machine learning. The builds provides a unified framework for the asymptotic guarantees under the Robbins-Monro scheme.\n\nWEAKNESSES:\nEven though the mathematical theory is nice, the results may not have too much practical importance. The author(s) emphasized that this work is about asymptotic analysis instead of the non-asymptotic analysis. However, because the results are of asymptotic nature, it not clear to me what insights the results can provide concerning the schemes such as mirror Langevin, proximal, randomized mid-point and Runge-Kutta methods, because you only have asymptotic guarantees, it is impossible to use these results to compare these algorithms with more classic and basic Euler discretization of Langevin algorithms. \n\nAlso, I find the discussion of existing literature less satisfactory. Some of the claims and statements may not be that accurate. For example, on page 1, ``Existing guarantees suffer from the drawback of lacking guarantees for the last-iterates’’  ``the convergence is typically given on the averaged iterates instead of the more natural last iterates’’. To the best of my knowledge, there are numerous works on Langevin algorithms in the past decade and most of them are about last iterates guarantees in Wasserstein, KL or other distances; see e.g. Dalalyan and Karagulyan (2019), Dalalyan and Riou-Durand (2020), Ma et al. (2019), Ma et al. (2021), Raginsky et al. (2017), Gao et al. (2022). ``and little is known beyond the elementary schemes of stochastic gradient Langevin dynamics.’’ This is not accurate either. There have been many studies in the literature about underdamped Langevin, high-order Langevin, non-reversible Langevin and other variants of SGLD such as Dalalyan and Riou-Durand (2020), Hu et al. (2020), Ma et al. (2021), Mou et al. (2021), Gao et al. (2022).\n\nQUESTIONS:\n(1)\tPage 4. ``The usual Lyapunov-type analysis for sampling algorithms focuses on bounding the change in relative entropy across iterations…” ``this makes the Lyapunov analysis applicable only to the simple Euler-Maruyama discreteization of (LD)’’ I am not too sure whether these two statements are accurate. Lyapunov functions are often used in analyis of Langevin algorithms, to show uniform bounds on the moments, e.g. Raginsky et al. (2017), in the coupling methods, e.g. Dalalyan and Riou-Durand (2020). It is definitely applicable beyond the Euler-Maruyama scheme, e.g. Dalalyan and Riou-Durand (2020) uses the disretization proposed in Cheng et al. (2018) to analyze kinetic Langevin dynamics.\n\n(2)\tOne technical point I would like to see more discussions is that in equation (1) in your Definition 1, it is for fixed $T>0$. Actually the dependence on $T$ can be exponential in $T$, which is quite common for weak approximation error in the literature. However, in order for the Langevin algorithm to converge to the Gibbs distribution, one often needs uniform-in-time guarantees, and would you need $T\\rightarrow\\infty$ in order to obtain Theorem 2?\n\n(3)\tTheorem 2 is a very nice and clean result. But I am surprised that you only need assumption (10) which is an assumption on the discretized dynamics only. The reason I am asking is that it seems to me that Assumptions 1-3 alone do not guarantee that the continuous-time Langevin SDE has a unique stationary distribution. If Theorem 2 holds, that means assumption (10) can imply that the continuous-time Langevin SDE has a unique stationary distribution? The existence of $\\pi$ is necessary for Theorem 2 to hold.\n\nLIMITATIONS:\nI did not see such discussions about limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives a unified asymptotic analysis of a broad class of stochastic algorithms that encompasses several variants of the Langevin algorithm. In particular, it can handle issues of inexact gradients, bias, noise, and problems beyond gradient-based algorithms. The key technique is the introduction of an intermediate process, termed the *Picard process*, which fits between the iterates and the continuous-time\n\nSTRENGTHS:\nThe main strength of this paper is the unified nature of the convergence results. The general framework encompasses a fairly large variety of Langevin-type algorithms, and likely a decent class of problems beyond Langevin algorithms. It also gives a clean, unified analysis.\n\nWEAKNESSES:\nMy only major criticism is that the paper seems to overstate the novelty of the analytic methods. In particular, I have not seen the specific Picard process defined here, but several works introduce related processes that fit between the iterates and the continuous time process. This enables similar triangle-inequality based convergence proofs. For example (but not limited to):\n\nChau, Ngoc Huy, et al. \"On stochastic gradient langevin dynamics with dependent data streams: The fully nonconvex case.\" SIAM Journal on Mathematics of Data Science 3.3 (2021): 959-986.\n\nBubeck, Sébastien, Ronen Eldan, and Joseph Lehec. \"Sampling from a log-concave distribution with projected Langevin Monte Carlo.\" Discrete & Computational Geometry 59 (2018): 757-783.\n\nAdditionally, last iterate convergence guarantees are not particularly rare. Both works cited above give last-iterate bounds from corresponding stationary distributions. Many of the works citing these papers do as well. \n\nOn a minor note, there are some confusing notations.\n* $b_k$ is used for the bias, but then gets re-defined in the proof of Lemma 2.\n* Using $\\sigma$ for both the diffusion matrix and the variance bound on the gradient noise is mildly confusing.\n* At the end of the proof of Lemma 1, it should be the limit as $n\\to\\infty$, instead of $k\\to \\infty$\n\nQUESTIONS:\n* Can you give examples beyond Langevin-type algorithms for which you can apply the  method?\n* Can you get some more quantitative guarantees beyond asymptotic convergence?\n\nLIMITATIONS:\nThese are adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}