Title: A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers
Venue: NeurIPS 2024
Accepted: yes
Keywords: Proximal samplers

Complexity of heavy-tailed sampling

Restricted Gaussian oracle

Restricted Stable oracle
Abstract: We study the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees i.e., samplers that require only $\mathcal{O}(\log(1/\varepsilon))$ versus $\Omega(\text{poly}(1/\varepsilon))$ iterations to output a sample which is $\varepsilon$-close to the target in $\chi^2$-divergence. Our results are presented for proximal samplers that are based on Gaussian versus stable oracles. We show that proximal samplers based on the Gaussian oracle have a fundamental barrier in that they necessarily achieve only low-accuracy guarantees when sampling from a class of heavy-tailed targets. In contrast, proximal samplers based on the stable oracle exhibit high-accuracy guarantees, thereby overcoming the aforementioned limitation. We also prove lower bounds for samplers under the stable oracle and show that our upper bounds cannot be fundamentally improved.

A Separation in Heavy-Tailed Sampling: Gaussian vs. Stable Oracles for Proximal Samplers

Abstract

We study the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees i.e., samplers that require only O (log(1 /ε )) versus Ω( poly (1 /ε )) iterations to output a sample which is ε -close to the target in χ 2 -divergence. Our results are presented for proximal samplers that are based on Gaussian versus stable oracles. We show that proximal samplers based on the Gaussian oracle have a fundamental barrier in that they necessarily achieve only low-accuracy guarantees when sampling from a class of heavy-tailed targets. In contrast, proximal samplers based on the stable oracle exhibit high-accuracy guarantees, thereby overcoming the aforementioned limitation. We also prove lower bounds for samplers under the stable oracle and show that our upper bounds cannot be fundamentally improved.

1 Introduction

The task of sampling from heavy-tailed targets arises in various domains such as Bayesian statistics [GJPS08, GLM18], machine learning [CDV09, BZ17, N¸ SR19, ¸ SZTG20, DKTZ20], robust statistics [KN04, JR07, Kam18, YŁR22], multiple comparison procedures [GBH04, GB09], and study of geophysical systems [SP15, QM16, PBEM23]. This problem is particularly challenging when using gradient-based Markov Chain Monte Carlo (MCMC) algorithms due to diminishing gradients, which occurs when the tails of the target density decay at a slow (e.g. polynomial) rate. Indeed, canonical algorithms like Langevin Monte Carlo (LMC) have been empirically observed to perform poorly [LWME19, HMW21, HFBE24] when sampling from such heavy-tailed targets.

Several approaches have been proposed in the literature to overcome these limitations of LMC and related algorithms. The predominant ones include (i) transformation-based approaches, where a diffeomorphic (invertible) transformation is used to first map the heavy-tailed density to a light-tailed one so that a light-tailed sampling algorithm can be used [JG12, YŁR22, HBE24], (ii) discretizing general Itô diffusions with non-standard Brownian motion that have heavy-tailed densities as their equilibrium density [EMS18, LWME19, HFBE24], and (iii) discretizing stable-driven stochastic differential equations [ZZ23]. However, the few theoretical results available on the analysis of algorithms based on approaches (i) and (ii) provide only low-accuracy heavy-tailed samplers; such algorithms require poly (1 /ε ) iterations to obtain a sample that is ε -close to the target in a reasonable metric of choice. Furthermore, quantitative complexity guarantees for the sampling approach used in (iii) are not yet available; thus, existing comparisons are mainly based on empirical studies.

In stark contrast, when the target density is light-tailed it is well-known that algorithms like proximal samplers based on Gaussian oracles and the Metropolis Adjusted Langevin Algorithm (MALA) have high-accuracy guarantees; these algorithms require only polylog (1 /ε ) iterations to obtain a sample which is ε -close to the target in some metric. See, for example, the works by [DCWY19, LST21b,

Table 1: Separation for Proximal Samplers: Gaussian vs. practical Stable oracles ( α =1 ): Upper and lower iteration complexity bounds to generate an ε -accurate sample in χ 2 -divergence from the generalized Cauchy target densities with degrees of freedom ν , i.e. π ν ∝ (1 + | x | 2 ) -( d + ν ) / 2 . Here, ˜ Ω , ˜ O hide constants depending on ν and polylog ( d, 1 /ε ) . For the proximal sampler with a general α -Stable oracle (Algorithm 2), the upper bound for ν ∈ (0 , 1) is O (log(1 /ε )) when α = ν . The lower bounds are from Corollary 2 via 2TV 2 ≤ χ 2 .

WSC22a, CCSW22, CG23]. Specifically, [LST21b] analyzed the proximal sampling algorithm to sample from a class of strongly log-concave densities and obtained high-accuracy guarantees. [CCSW22] established similar high-accuracy guarantees for the proximal sampler to sample from target densities that satisfy a certain functional inequality, covering a range of light-tailed densities with exponentially fast tail decay (e.g. log-Sobolev and Poincaré inequalities). However, it is not clear if the proximal sampler achieves the same desirable performance when the target is not light-tailed.

In light of existing results, in this work, we first consider the following question:

Q1. What are the fundamental limits of proximal samplers under the Gaussian oracle when sampling from heavy-tailed targets?

To answer this question, we construct lower bounds showing that Gaussian-based samplers necessarily require poly (1 /ε ) iterations to sample from a class of heavy-tailed targets. These results complement the lower bounds on the complexity of sampling from heavy-tailed densities using the LMC algorithm established in [MHFH + 23]. With this lower bound in hand, we next consider the following question:

Q2. Is it possible to design high-accuracy samplers for heavy-tailed targets?

We answer this in the affirmative by constructing proximal samplers that are based on stable oracles (see Definition 1 and Algorithm 2) by leveraging the fractional heat-flow corresponding to a class of stable-driven SDEs. We analyze the complexity of this algorithm when sampling from heavy-tailed densities that satisfy a fractional Poincaré inequality, and establish that they require only log (1 /ε ) iterations. Together, our answers to Q1 and Q2 provide a clear separation between samplers based on Gaussian and stable oracles. Our contributions can be summarized as follows.

Lower bounds for the Gaussian oracle : In Section 2, we focus on Q1 and establish in Theorems 1 and 2 respectively that the Langevin diffusion and the proximal sampler based on the Gaussian oracle necessarily have a fundamental barrier when sampling from heavy-tailed densities. Our proof technique builds on [Hai10], and provides a novel perspective for obtaining algorithmdependent lower bounds for sampling, which may be of independent interest.

A proximal sampler based on the stable oracle: In Section 3, we introduce a proximal sampler based on the α -stable oracle, which fundamentally relies on the exact implementations of the fractional heat flow that correspond to a stable-driven SDE. Here, the parameter α determines the allowed class of heavy-tailed targets which could be sampled with high-accuracy. In Theorem 3 and Proposition 1, we provide upper bounds on the iteration complexity that are of smaller order than the corresponding lower bounds established for the Gaussian oracle. We provide a rejectionsampling based implementation of the α -stable oracle for the case α = 1 and prove complexity upper bounds in Corollary 3. Finally, in Theorem 4, considering a sub-class of Cauchy-type targets, we prove lower bounds showing that our upper bounds cannot be fundamentally improved.

An illustration of our results for Cauchy target densities, π ν ∝ (1 + | x | 2 ) -( d + ν ) / 2 where ν is the degrees of freedom, is provided in Table 1. We specifically consider the practical version of the stable proximal sampler with α = 1 (i.e., Algorithm 2 with the stable oracle implemented by Algorithm 3), and show that it always outperforms the Gaussian proximal sampler (Algorithm 1). Indeed, when ν ≥ 1 , the separation between these algorithms is obvious. In the case ν ∈ (0 , 1) , Algorithm 2 & 3 has a poly( 1 /ε ) complexity, nevertheless, it still improves the complexity of the Gaussian proximal sampler by a factor of ε . Wealso show via lower bounds (in Section 3.4) that the poly( 1 /ε ) complexity for Algorithm 2 & 3, when ν ∈ (0 , 1) , can only be improved up to certain factors. We remark that for the ideal proximal sampler (Algorithm 2), the upper bound when ν ∈ (0 , 1) is also O (log(1 /ε )) . These results demonstrate a clear separation between Gaussian and stable proximal samplers.

Related works. We first discuss works analyzing the complexity of heavy-tailed sampling as characterized by a functional inequality assumption. [CDV09] analyzed the connection between sampling

algorithms for a class of s -concave densities satisfying a certain isoperimetry condition related to weighted Poincaré inequalities. [HFBE24] undertook a mean-square analysis of discretization of a specific Itô diffusion that characterizes a class of heavy-tailed densities satisfying a weighted Poincaré inequality. [ALPW22] and [ALPW23] analyzed the complexity of pseudo-marginal MCMC algorithms and the random-walk Metropolis algorithm respectively, under weak Poincaré inequalities. As mentioned before, [MHFH + 23] showed lower bounds for the LMC algorithm when the target density satisfies a weak Poincaré inequality. [HBE24] and [YŁR22] analyzed a transformation based approach for heavy-tailed sampling under conditions closely related to the same functional inequality. This transformation methodology is also used to demonstrate asymptotic exponential ergodicity for other sampling algorithms like the bouncy particle sampler and the zig-zag sampler, in the heavytailed settings [DBCD19, DGM20, BRZ19]. These works provide only low-accuracy guarantees for heavy-tailed sampling and do not consider the use of weak Fractional Poincaré inequalities.

Recent years have witnessed a significant focus on (strongly) log-concave sampling, leading to an extensive body of work that is challenging to encapsulate succinctly. In the context of (strongly) logconcave or light-tailed distributions, a plethora of non-asymptotic investigations have been conducted on LMC variations, including advanced integrators [SL19, LWME19, HBE20], underdamped LMC [CCBJ18, EGZ19, CLW23, DRD20], and MALA [DCWY19, LST20, CLA + 21, WSC22b]. Outside the realm of log-concavity, the dissipativity assumption, which regulates the growth of the potential, has been used in numerous studies to derive convergence guarantees [DM17, RRT17, EMS18, EH21, MFWB22, EHZ22, BCE + 22].

While research on upper bounds of sampling algorithms' complexity has advanced considerably, the exploration of lower bounds is still nascent. [CGL + 22] explored the query complexity of sampling from strongly log-concave distributions in one-dimensional settings. [LZT22] established lower bounds for LMC in sampling from strongly log-concave distributions. [CBL22] presented lower bounds for sampling from strongly log-concave distributions with noisy gradients. [GLL20] focused on lower bounds for estimating normalizing constants of log-concave densities. Contributions by [LST21a] and [WSC22b] provide lower bounds in the metropolized algorithm category, including Langevin and Hamiltonian Monte Carlo, in strongly log-concave contexts. Finally, [CGLL22] contributed to lower bounds in Fisher information for non-log-concave sampling.

2 Lower Bounds for Sampling with the Gaussian Oracle

In this section, we focus on Q1 for both the Langevin diffusion (in continuous time) and the proximal sampler (in discrete time), where both procedures have the target density as their invariant measures. Our results below illustrate the limitation of the Gaussian oracle 1 for heavy-tailed sampling in both continuous and discrete time, showing that the phenomenon is not because of the discretization effect, but is inherently related to the use of Gaussian oracles.

Langevin diffusion. We first start with the overdamped Langevin diffusion (LD):

LD achieves high-accuracy 'sampling' in continuous time, i.e. a polylog(1 /ε ) convergence rate in the light-tailed setting. We make the following dissipativity-type assumption.

Assumption 1. The target density is given by π X ( x ) ∝ exp( -V ( x )) , where V : R d → R satisfies

Remark 1. The upper bound on ⟨ x, ∇ V ( x ) ⟩ ensures that V grows at most logarithmically in | x | . Consequently, π X is heavy-tailed and in fact does not satisfy a Poincaré inequality. The lower bound on ⟨ x, ∇ V ( x ) ⟩ is only needed for deriving the dimension dependency in our guarantees. If one is only interested in the ε dependency, this condition can be replaced with 0 ≤ ⟨ x, ∇ V ( x ) ⟩ .

A classical example of a density satisfying the above assumption is the generalized Cauchy density with degrees of freedom ν = ν 1 = ν 2 > 0 , where the potential is given by

The following result, proved in Appendix A, provides a lower bound on the performance of LD.

Theorem 1. Suppose π X ∝ exp( -V ) satisfies Assumption 1. Let X t be the solution of the Langevin diffusion, and µ t := Law( X t ) . Then, for any δ > 0 ,

where κ δ := 1 ∨ 2 d + ν 2 ∨ ν 2 (1+ δ ) ( d + ν 2 ) δ , C δ ( µ 0 ) := 1 d + ν 2 E [(1 + | X 0 | 2 ) γ ] 1 /γ with γ = κ δ ( d + ν 2 ) / 2 , and C ν 1 ,ν 2 is a constant depending only on ν 1 and ν 2 .

If we assume | X 0 | ≤ O ( √ d ) for simplicity, then by choosing δ = 2 ln ln t ν 2 ln t ∧ 2 ln ln d ( ν 2 -ν 1 ) ln d , we obtain

Thus, LD requires at least T = ˜ Ω ν 1 ,ν 2 ( d ν 1 -ν 2 ν 2 (1 /ε ) 2 /ν 2 ) to reach ε error in total variation. While this bound may be small in high dimensions when ν 2 > ν 1 , for the canonical model of Cauchy-type potentials with ν 2 = ν 1 = ν , it will be independent of dimension, as stated by the following result. Note that Assumption 1 can also cover a general scaling by replacing | x | with c | x | for some constant c , which would introduce a multiplicative factor of 1 /c 2 for the lower bound on T . This is expected as e.g., mixing to the Gibbs potential c 2 | x | 2 can be faster than mixing to | x | 2 by a factor of 1 /c 2 .

Corollary 1. Consider the generalized Cauchy density π X ν ∝ exp( -V ν ) where V ν is as in (1) . Let X t be the solution of the Langevin diffusion, and µ t := Law( X t ) . For simplicity, assume the initialization satisfies | X 0 | ≤ O ( √ d ) . Then, achieving TV( π X ν , µ T ) ≤ ε requires T ≥ ˜ Ω ν ( ε -2 ν ) .

The above lower bound implies that LD is a low-accuracy 'sampler' for this target density in the sense that it depends polynomially on 1 /ε ; this dependence gets worse with smaller ν as the tails get heavier. It is worth highlighting the gap between the upper bound of [MHFH + 23, Corollary 8], which is ˜ O ( 1 /ε 4 /ν ) , and the lower bound in Corollary 1.

Gaussian proximal sampler. In the remainder of this section, we prove that the Gaussian proximal sampler, described in Algorithm 1, also suffers from a poly(1 /ε ) rate when the target density is heavy-tailed. In each iteration of Algorithm 1, the first step involves sampling a standard Gaussian random variable y k centered at the current iterate x k with variance ηI ; this is a one-step isotropic Brownian random walk. Alternatively, since the Fokker-Planck equation of the standard Brownian motion is the classical heat equation, this step could also be interpreted as an exact simulation of the heat flow; see, for example, [CG03] and [Wib18]. Specifically, the density of y k is the solution to the heat flow at time η with the initial condition being the density of x k . The second step is called the restricted Gaussian oracle (RGO) as coined by [LST21b]; under which ( x k , y k ) is a reversible Markov chain whose stationary density has x -marginal π X .

Assumption 2. For some ν 2 ≥ ν 1 ≥ 0 , the target π X ( x ) ∝ exp( -V ( x )) with V : R d → R satisfies

The first condition above also appears in Assumption 1 and the second condition implies the upper bound of Assumption 1; thus, the above assumption is stronger. Note that the generalized Cauchy measure (1) satisfies this assumption with ν 1 = ν 2 = ν . Under Assumption 2, we state the following lower bound on the Gaussian proximal sampler and defer its proof to Appendix A.

Theorem 2. Suppose π X ∝ exp( -V ) satisfies Assumption 2. Let x k denote the k th iterate of the Gaussian proximal sampler (Algorithm 1) with step η and let ρ X k := Law( x k ) . Then, for any δ > 0 ,

where κ δ , C δ ( µ 0 ) , and C ν 1 ,ν 2 are defined in Theorem 1.

Above, assuming | X 0 | ≤ O ( √ d ) with the same choice of δ as in Theorem 1 yields TV( π X ν , ρ X k ) ≥ ˜ Ω ν 1 ,ν 2 ( d ν 1 -ν 2 2 ( kη ) -ν 2 2 ) . Note that in order for the RGO step to be efficiently implementable, we need to have a sufficiently small η . The state-of-the-art implementation of RGO requires a step size of order η = ˜ O (1 / ( Ld 1 / 2 )) when V has L -Lipschitz gradients [FYC23]. With this choice of step size, the above lower bound requires at least N = ˜ Ω ν 1 ,ν 2 ( Ld 1 / 2+( ν 1 -ν 2 ) /ν 2 (1 /ε ) 2 /ν 2 ) iterations. The assumptions in Theorem 2 once again cover the canonical examples of generalized Cauchy densities, where we have L = d + ν , which simplifies the lower bound as follows.

Corollary 2. Consider the generalized Cauchy density π X ν ∝ exp( -V ν ) where V ν is as in (1) . Let x k denote the k th iterate of the Gaussian proximal sampler, and define ρ X k := Law( x k ) , and choose the step size η = ˜ O (1 / ( Ld 1 / 2 )) . If we assume | X 0 | ≤ O ( √ d ) for simplicity, then achieving TV( π X ν , ρ X N ) ≤ ε requires N ≥ ˜ Ω ν ( d 3 2 ε -2 ν ) iterations.

We emphasize that the above lower bound is of order poly (1 /ε ) as advertised. Thus, the RGO-based proximal sampler can only yield a low-accuracy guarantee in this setting.

3 Stable Proximal Sampler and the Restricted α -Stable Oracle

Having characterized the limitations of Gaussian oracles for heavy-tailed sampling, thereby answering Q1 , in what follows, we will focus on Q2 and construct proximal samplers based on the α -stable oracle, and prove that they achieve high-accuracy guarantees when sampling from heavy-tailed targets. First, we provide a basic overview of α -stable processes and fractional heat flows.

Isotropic α -stable process. For t ≥ 0 , let X ( α ) t be the isotropic stable Lévy process in R d , starting from x ∈ R d , with the index of stability α ∈ (0 , 2] , defined uniquely via its characteristic function E x e i ⟨ ξ,X ( α ) t -x ⟩ = e -t | ξ | α . When α = 2 , X (2) t is a scaled Brownian motion, and when 0 < α < 2 , it becomes a pure Lévy jump process in R d . The transition density of X ( α ) t is then given by

where the second equation above is the inverse Fourier transform of the characteristic function, thus returns the density. The transition kernel and the density in (2) have closed-form expressions for the special cases α = 1 , 2 . In particular, when α = 1 , p (1) t reduces to a Cauchy density with degrees of freedom ν = 1 , i.e. p (1) t ( y ) ∝ ( | y | 2 + t 2 ) -( d +1) / 2 . We finally note that the isotropic stable Lévy process X ( α ) t displays self-similarity like the Brownian motion; the processes X ( α ) at and a 1 /α X ( α ) t have the same distribution. This property is crucial in the development of the stable proximal sampler.

Fractional heat flow. The equation ∂ t u ( t, x ) = -( -∆) α/ 2 u ( t, x ) with the condition u (0 , x ) = u 0 ( x ) is an extension of the classical heat flow, and is referred to as the fractional heat flow. Here, -( -∆) α/ 2 is the fractional Laplacian operator with α ∈ (0 , 2] , which is the infinitesimal generator of the isotropic α -stable process. For α = 2 , it reduces to the standard Laplacian operator ∆ .

Stable proximal sampler. Let π ( x, y ) be a joint density such that π ( x, y ) ∝ π X ( x ) p ( α ) ( η ; x, y ) , where π X is the target and p ( α ) ( η ; x, y ) is the transition density of the α -stable process, introduced in (2). It is easy to verify that (i) the X -marginal of π is π X , (ii) the conditional density of Y given X is π Y | X ( ·| x ) = p ( α ) ( η ; x, · ) , (iii) the Y -marginal is π Y = π X ∗ p ( α ) η , i.e. π Y is obtained by evolving π X along the α -fractional heat flow for time η , and (iv) the conditional density of X given Y is π X | Y ( ·| y ) ∝ π X ( · ) p ( α ) ( η ; · , y ) . Based on these, we introduce the following stable oracle.

Definition 1 (Restricted α -Stable Oracle) . Given y ∈ R d , an oracle that outputs a random vector distributed according to π X | Y ( ·| y ) , is called the Restricted α -Stable Oracle (R α SO).

Note that when α = 2 , the R α SO reduces to the RGO of [LST21b]. The Stable Proximal Sampler (Algorithm 2) with parameter α is initialized at a point x 0 ∈ R d and performs Gibbs sampling on the joint density π . In each iteration, the first step involves sampling an isotropic α -stable random vector y k centered at the current iterate x k , which is a one-step isotropic α -stable random walk. This could also be interpreted as an exact simulation of the fractional heat flow. Indeed, due to the relation between the fractional heat flow and the isotropic stable process, the density of y k is exactly the solution to the α -fractional heat flow at time η with the initial condition being the density of x k .

When α = 2 , the first step reduces to an isotropic Brownian random walk and a simulation of the classical heat flow. The second step calls the R α SO at the point y k .

3.1 Convergence guarantees

We next provide convergence guarantees for the stable proximal sampler in χ 2 -divergence assuming access to the R α SO. Similar results for a practical implementation are presented in Section 3.2. To proceed, we introduce the fractional Poincaré inequality, first introduced in [WW15] to characterize a class of heavy-tailed densities including the canonical Cauchy class.

Definition 2 (Fractional Poincaré Inequality) . For ϑ ∈ (0 , 2) , a probability density µ satisfies a ϑ -fractional Poincaré inequality (FPI) if there exists a positive constant C FPI ( ϑ ) such that for any function ϕ : R d → R in the domain of E ( ϑ ) µ , we have

where E ( ϑ ) µ is a non-local Dirichlet form associated with µ defined as

̸

Remark 2. FPI is a weaker condition than Assumption 2. In fact, any density satisfying the first 2 conditions in Assumption 2 satisfies ϑ -FPI for all ϑ < ν 1 [WW15, Theorem 1.1]. In Proposition 2, we show that as ϑ → 2 -, FPI becomes equivalent to the standard Poincaré inequality.

In the sequel, ρ X k denotes the law of x k , ρ Y k denotes the law of y k , and ρ k = ρ X,Y k is the joint law of ( x k , y k ) . We provide the following convergence guarantee under an FPI, proved in Appendix B.2.

Theorem 3. Assume that π X satisfies the α -FPI with parameter C FPI ( α ) for α ∈ (0 , 2) . For any step size η > 0 and initial density ρ X 0 , the k th iterate of Algorithm 2, with parameter α , satisfies

As a consequence of Remark 2 and Proposition 2, we recover the result in [CCSW22, Theorem 4], by letting α → 2 -. While our results in Theorem 3 are based on Algorithm 2 which requires exact calls to R α SO, the next result, proved in Appendix B.3, shows that even with an inexact implementation of R α SO, the error accumulation is at most linear, and Algorithm 2 still converges quickly.

Proposition 1. Suppose the R α SO in Algorithm 2 is implemented inexactly, i.e. there exists a positive constant ε TV such that TV(˜ ρ X | Y k ( ·| y ) , ρ X | Y k ( ·| y )) ≤ ε TV for all y ∈ R d and k ≥ 1 , where ˜ ρ X | Y k ( ·| y ) is the density of the inexact R α SO sample conditioned on y . Let ˜ ρ X k be the density of the output of the k th step of Algorithm 2 with the inexact R α SO and ρ X k be the density of the output of k th step Algorithm 2 with the exact R α SO. Then, for all k ≥ 0 ,

Further, if ˜ ρ X 0 = ρ X 0 , for any K ≥ K 0 , we get TV(˜ ρ K X , π X ) ≤ ε , if ε TV ≤ ε/ 2 K , where the constant K 0 = (1 + C FPI ( α ) η -1 ) log ( χ 2 (˜ ρ X 0 | π X ) /ε 2 ) with C FPI ( α ) being the α -FPI parameter of π X .

3.2 A practical implementation of R α SO

In the sequel, we introduce a practical implementation of R α SO when α = 1 . For this, we consider the case when the target density π X ∝ e -V satisfies the 1 -FPI with parameter C FPI (1) . A more thorough implementation of R α SO for other values of α will be investigated in future work.

Assumption 3. There exist constants β, L > 0 such that for any minimizer x ∗ ∈ arg min y ∈ R d V ( y ) and for all x ∈ R d , V satisfies V ( x ) -V ( x ∗ ) ≤ L | x -x ∗ | β .

Algorithm 3 provides an exact implementation of R α SO for α = 1 via rejection sampling. Inputs to this algorithm are the intermediate points y k in the stable proximal sampler (Algorithm 2). Note that Algorithm 3 requires a global minimizer of V , which is always assumed to exist, which guarantees that the acceptance probability is non-trivial. It generates proposals with density p (1) ( η ; · , y ) and utilizes that p (1) is a Cauchy density and Cauchy random vectors can be generated via ratios between a Gaussian random vector and square-root of a χ 2 random variable. Finally, the accept-reject step ensures that the output x has density π X | Y ( ·| y ) ∝ e -V p (1) ( η ; · , y ) . This makes Algorithm 3 a zeroth-order algorithm requiring only access to function evaluations of V . Under Assumption 3, by choosing a small step-size, we can control the expected number of rejections in Algorithm 3. We now state the iteration complexity of our stable proximal sampler with this R α SO implementation in the following result, whose proof is provided in Appendix B.3.

Corollary 3. Assume V satisfies Assumption 3. If we choose the step-size η = Θ( d -1 2 L -1 β ) , then Algorithm 3 implements the R α SO with α = 1 , with the expected number of zeroth-order calls to V of order E [exp( L | y k | β )] . Further assume π X satisfies 1 -FPI with parameter C FPI (1) . Suppose we run Algorithm 2 with R α SO implemented for with α = 1 by Algorithm 3. Then, to return a sample which is ε -close in χ 2 -divergence to the target, the expected number of iterations required by Algorithm 2 is

Note that the above result provides a high-accuracy guarantee for the implementable version of the stable proximal sampler (Algorithm 3) for a class of heavy-tailed targets, overcoming the fundamental barrier established in Theorem 2 for the Gaussian proximal sampler (i.e., Algorithm 1). A numerical illustration of this improvement is provided in Appendix D by sampling from student-t distributions.

Remark 3. (1) Finding a global minimizer of the potential V can be hard, which could be avoided if a lower bound on the potential V is available; see Appendix B.3. (2) A trivial bound for E [exp( L | y k | β )] is exp( LM ) for M = E π X [ | X | β ] + χ 2 ( ρ X 0 | π X ) E π X [ | X | 2 β ] 1 2 . Since our main focus is high vs low accuracy samplers, deriving a sharper bound is beyond the scope of the current paper.

3.3 Illustrative examples

To illustrate our results, we now apply the proximal algorithms to sample from Cauchy densities and discuss the complexity of both the ideal sampler (Algorithm 2) in which we can choose any α ∈ (0 , 2) and the implementable version with α = 1 (Algorithm 3). For the ideal sampler, we can choose α ≤ ν for any degrees of freedom ν > 0 , and apply Theorem 3 since π ν satisfies a α -FPI [WW15].

Corollary 4. For any ν > 0 , consider the generalized Cauchy target π ν ∝ exp( -V ν ) with V ν defined in (1) . For the stable proximal sampler with parameter α ∈ (0 , 2) and α ≤ ν (i.e., Algorithm 2), suppose we set the step-size η ∈ (0 , 1) and draw the initial sample from the standard Gaussian density. Then, the number of iterations required by Algorithm 2 to produce an ε -accurate sample in χ 2 -divergence is O ( C FPI ( α ) η -1 log( d/ε )) , where C FPI ( α ) is the α -FPI parameter of π ν .

For the implementable sampler, since the parameter α is fixed to be 1 , whether a suitable FPI is satisfied or not depends on the degrees of freedom ν . Specifically, when ν ≥ 1 , 1 -FPI is satisfied and Corollary 5 applies. When ν ∈ (0 , 1) , on the other hand, 1 -FPI is not satisfied. To tackle this issue, we prove convergence guarantees for the proximal sampler under a weak fractional Poincaré inequality; the next corollary, proved in Appendix B.4, summarizes these results.

Corollary 5. For the Cauchy target π ν ∝ exp( -V ν ) where V ν is defined in (1) , we consider Algorithm 2 with α = 1 , a standard Gaussian initialization, and R α SO implemented by Algorithm 3.

When ν ≥ 1 , if we set the step-size η = Θ ( d -1 2 ( d + ν ) -4 ) , the expected number iterations required by Algorithm 2 to output a sample which is ε -close in χ 2 -divergence to the target is of order O ( C FPI (1) d 1 2 ( d + ν ) 4 log( d/ε ) ) , where C FPI (1) is the 1 -FPI parameter of π ν .

When ν ∈ (0 , 1) , if we set the step-size η = Θ ( d -1 2 ( d + ν ) -4 ν ) , the expected number of iterations required by Algorithm 2, to output a sample which is ε -close in χ 2 -divergence to the target is of order ˜ O ( max { c 1 ν d 1 2 ν + 4 ν 2 , cd 1 2 + 4 ν ε -1 ν +1 }) , where c is the positive constant given in (16) . Here, ˜ O hides the polylog factors on d and 1 /ε .

The stable proximal sampler (Algorithm 2) is a high accuracy sampler for the class of generalized Cauchy targets, as long as α ≤ ν , meaning that it achieves log( 1 /ε ) iteration complexity. The improvement from poly( 1 /ε ) to log( 1 /ε ) separates the stable proximal sampler and the Gaussian proximal sampler in the task of heavy-tailed sampling. When we use the rejection-sampling implementation with parameter α = 1 (Algorithm 3), iteration complexity goes through a phase transition as the tails get heavier. When the generalized Cauchy density has a finite mean ( ν > 1 ), we achieve a high-accuracy sampler with log( 1 /ε ) iteration complexity. However, without a finite mean (i.e., ν ∈ (0 , 1) ), the algorithm becomes a low-accuracy sampler with poly( 1 /ε ) complexity. Even in this low-accuracy regime, the implementable stable proximal sampler outperforms the Gaussian one, as originally highlighted in Table 1. Last, we claim that the poly( 1 /ε ) complexity of Algorithms 2 and 3 is not due to a loose analysis, as we show poly(1 /ε ) lower bounds in the following section.

3.4 Lower bounds for the stable proximal sampler

Wenow study lower bounds on the stable proximal sampler to sample from the class of target densities satisfying Assumption 2, which includes the generalized Cauchy target. Recall that Assumption 2 implies the FPI used in Theorem 3. The result below, proved in Appendix C, complements Theorem 3, showing the impossibility of achieving log(1 /ε ) rates for a sufficiently large α .

Theorem 4. Suppose π X ∝ exp( -V ) with V satisfying Assumption 2 and ν 2 ( d + ν 2 ) d + ν 1 < α ≤ 2 . Let x k denote the k th iterate of Algorithm 2 with parameter α and step size η , and let ρ X k := Law( x k ) . Then for any τ ∈ ( ν 2 ( d + ν 2 ) d + ν 1 , α ) , and g ( d, ν 1 , ν 2 , τ ) = ν 2 / { τ ( d + ν 1 ) -ν 2 ( d + ν 2 ) } , we have

where C ν 1 ,ν 2 ,α is a constant depending only on ν 1 , ν 2 , α , and m ( α ) τ is the τ th absolute moment of the α -stable random variable with density p ( α ) 1 defined in (2) .

Remark 4. The parameter τ in Theorem 4 can be chosen arbitrarily close to α . Specifically, if we assume | X 0 | ≤ O ( √ d ) , then with the choice of τ = α -( log(log d ) log d ∧ log log( η -1 ) log( η -1 ) ) , we have

where ˜ Ω hides polylog ( d/η ) factors.

The τ th absolute moment of the α -stable random variable depends on the choice of α and the dimension d . It is hard to find an explicit formula of m ( α ) τ in general. An explicit formula is only available in some special cases, such as α = 1 , 2 . Specializing Theorem 4 for the generalized Cauchy potential (i.e., ν 1 = ν 2 ) we obtain the following explicit result.

Corollary 6. Let α ∈ (0 , 2] . Suppose π ν ∝ exp( -V ν ) where V ν ( x ) is as in (1) for some ν ∈ (0 , α ) . Let ( x k ) k ≥ 0 be the output of Algorithm 2 with parameter α and step-size η > 0 , and ρ X k := Law( x k ) for all k ≥ 0 . Then for any τ ∈ ( ν, α ) ,

where m ( α ) τ is the τ th absolute moment of the α -stable random variable with density p ( α ) 1 as in (2) .

For the rejection sampling implementation in Algorithm 3, α = 1 and m (1) τ = Θ( d τ 2 ) for all τ < 1 (see Appendix B.1). Notice that to implement the R α SO in the Stable proximal sampler efficiently, we need a sufficiently small step-size η . When the target potential satisfies Assumption 3, i.e. V is β -Hölder continuous with parameter L , we require η = Θ( d -1 2 L -1 β ) to ensure R α SO can be implemented with O (1) queries. Therefore, if we choose η = Θ( d -1 2 L -1 β ) , the minimum number of iterations we need to get an ε -error in TV is

For the generalized Cauchy potential with ν ∈ (0 , 1) , we have β = ν/ 4 and L = ( d + ν ) /ν , which leads to the following corollary.

Corollary 7. Suppose π X ν ∝ exp( -V ν ) is the generalized Cauchy density with ν ∈ (0 , 1) . Let x k denote the k -th iterate of the stable proximal sampler with α = 1 (Algorithm 3), and ρ X k := Law( x k ) . If we choose the step size η = Θ( L -4 ν d -1 2 ) where L = d + ν ν is the ν/ 4 -Hölder constant of V ν , and assume, for simplicity, | x 0 | ≤ O ( √ d ) , then, TV( π X ν , ρ X N ) ≤ ε requires N ≥ Ω ν,τ ( d τ +8 τ/ν 2+ τ ε -2( τ -ν ) ν (2+ τ ) ) , for any τ ∈ ( ν, 1) . Further, by choosing τ = max( ν, 1 -log(log( d/ε )) log( d/ε ) ) , we obtain

The above result shows that when implementing the R α SO in Algorithm 2 with Algorithm 3, to sample from generalized Cauchy targets with ν ∈ (0 , 1) , we can at best have an iteration complexity of order poly (1 /ε ) , matching the upper bounds in Corollary 5 up to certain factors.

4 Overview of Proof Techniques

Lower bounds. We build on the techniques developed in [Hai10]. Let µ t denotes the law of LD along its trajectory. To proceed, we need some G : R d → R for which we can upper bound µ t ( G ) := ∫ G d µ t , and some f : R d → R that satisfies π X ( G ≥ y ) ≥ f ( y ) for all y ∈ R + . After finding the candidates G and f , Lemma 1 in Appendix A guarantees TV( π X , µ t ) ≥ sup y ∈ R + f ( y ) -µ t ( G ) /y. This technique relies on choosing G such that it has heavy tails under π X leading to a large f ( y ) , while having light tails along the trajectory, thus small µ t ( G ) . By picking G = exp( κV ) with κ ≥ 1 , one can immediately observe that π X ( G ) = ∞ , thus G indeed has heavy tails under π X .

To control µ t ( G ) along the trajectory, one can use the generator of LD to bound ∂ t µ t ( G ) . Recall the generator of LD, L LD ( · ) = ∆( · ) - ⟨∇ V, ∇·⟩ . Therefore, with a choice of G = exp( κV ) , controlling ∂ t µ t ( G ) requires bounding the first and second derivatives of V . To avoid making extra assumptions for V in the analysis of LD, we instead construct G based on a surrogate potential ˜ V ( x ) = d + ν 2 2 ln(1 + | x | 2 ) , which is an upper bound to the potential V . We then estimate f based on this surrogate potential in Lemma 2, and control the growth of µ t ( G ) in Lemma 3. Combined with Lemma 1, this leads to the proof of Theorem 1, with the details provided in Appendix A.

For the Gaussian proximal sampler, bounding ρ X k ( G ) requires controlling the expectation of G along the forward and backward heat flow. For the particular choice of G = exp( κV ) , we show in Lemma 4 that the growth of ρ X k ( G ) can be controlled only by considering a forward heat flow with the corresponding generator L HF = 1 2 ∆ . Therefore, given additional estimates on the second derivatives of V , we bound the growth of ρ X k ( G ) in Lemma 5. Once this bound is achieved, we can invoke Lemma 1 to finish the proof of Theorem 2.

Upper bounds. Our upper bound analysis builds on that by [CCSW22] in the specific ways discussed next. We consider the change in χ 2 divergence when we apply the two operations to the law ρ X k to the iterates and the target π X : ( i ) evolving the two densities along the α -fractional heat flow for time η and ( ii ) applying the R α SO to the resulting densities. For the step (i), it is required to show that the solution along the fractional heat flow of the stable proximal sampler at any time, satisfies FPI. To show this, ( a ) the convolution property of the FPI is proved in Lemma 6, and ( b ) the FPI parameter for the stable process follows from [Cha04, Theorem 23]. In Proposition 3, it is then shown that the χ 2 divergence decays exponentially fast along the fractional heat flow under the assumption of FPI. The aforementioned results enable us to prove the exponential decay of χ 2 divergence along the fractional heat flow under FPI in Proposition 3. To deal with the step (ii) above, we use the data processing inequality; see Proposition 3. These two steps together, enable us to derive the stated upper bounds for the stable proximal sampler.

5 Discussion

We showed the limitations of Gaussian proximal samplers for high-accuracy heavy-tailed sampling, and proposed and analyzed stable proximal samplers, establishing that they are indeed high-accuracy algorithms. We now list a few important limitations and problems for future research: (i) It is important to develop efficiently implementable versions of the stable proximal sampler for all values of α ∈ (0 , 2) , and characterize their complexity in terms of problem parameters, (ii) Gaussian

proximal samplers can be interpreted as a proximal point method for approximating the entropic regularized Wasserstein gradient flow of the KL objective [CCSW22]. This leads to the question, can we provide a variational intepreration of the stable proximal sampler? A potential approach is to leverage the results by [Erb14] on gradient flow interpretation of jump processes corresponding to the fractional heat equation, (iii) It is possible to use a non-standard Itô process in the proximal sampler (in place of the α -stable diffusion); see, for example, [EMS18, LWME19, HFBE24]. With this modification, it is interesting to examine the rates under weighted Poincaré inequalities that also characterize heavy-tailed densities. There are two difficulties to overcome here: ( a ) How to generate an exact non-standard Itô process? ( b ) How to implement the corresponding Restricted non-standard Gaussian Oracle, which requires the zeroth order information of the transition density of the Itô process? In certain cases, non-standard Itô diffusion can be interpreted as a Brownian motion on an embedded sub-manifold; thus, the approach in [GLL + 23] might be useful.

References

[ALPW22] Christophe Andrieu, Anthony Lee, Sam Power, and Andi Q Wang, Comparison of Markov chains via weak Poincaré inequalities with application to pseudo-marginal MCMC , The Annals of Statistics 50 (2022), no. 6, 3592-3618.

[ALPW23] , Weak Poincaré Inequalities for Markov chains: Theory and Applications , arXiv preprint arXiv:2312.11689 (2023).

[App09] David Applebaum, Lévy processes and stochastic calculus , Cambridge university press, 2009.

[BCE + 22] Krishnakumar Balasubramanian, Sinho Chewi, Murat A Erdogdu, Adil Salim, and Shunshi Zhang, Towards a theory of non-log-concave sampling: First-order stationarity guarantees for Langevin Monte Carlo , Conference on Learning Theory, PMLR, 2022, pp. 2896-2923.

[BHJ08] Krzysztof Bogdan, Wolfhard Hansen, and Tomasz Jakubowski, Time-dependent Schrödinger perturbations of transition densities , Studia Mathematica 189 (2008), no. 3, 235-254.

[BRZ19] Joris Bierkens, Gareth O Roberts, and Pierre-André Zitt, Ergodicity of the zigzag process , The Annals of Applied Probability 29 (2019), no. 4, 2266-2301.

[BZ17] Maria-Florina F Balcan and Hongyang Zhang, Sample and computationally efficient learning algorithms under s -concave distributions , Advances in Neural Information Processing Systems 30 (2017).

[CBL22] Niladri S Chatterji, Peter L Bartlett, and Philip M Long, Oracle lower bounds for stochastic gradient sampling algorithms , Bernoulli 28 (2022), no. 2, 1074-1092.

[CCBJ18] Xiang Cheng, Niladri S Chatterji, Peter L Bartlett, and Michael I Jordan, Underdamped Langevin MCMC: A non-asymptotic analysis , Conference on learning theory, PMLR, 2018, pp. 300-323.

[CCSW22] Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono, Improved analysis for a proximal algorithm for sampling , Conference on Learning Theory, PMLR, 2022, pp. 2984-3014.

[CDV09] Karthekeyan Chandrasekaran, Amit Deshpande, and Santosh Vempala, Sampling s-concave functions: The limit of convexity based isoperimetry , International Workshop on Approximation Algorithms for Combinatorial Optimization, Springer, 2009, pp. 420-433.

[CG03] Eric A Carlen and Wilfrid Gangbo, Constrained steepest descent in the 2-Wasserstein metric , Annals of mathematics (2003), 807-846.

[CG23] Yuansi Chen and Khashayar Gatmiry, A Simple Proof of the Mixing of MetropolisAdjusted Langevin Algorithm under Smoothness and Isoperimetry , arXiv preprint arXiv:2304.04095 (2023).

[CGL + 22] Sinho Chewi, Patrik R Gerber, Chen Lu, Thibaut Le Gouic, and Philippe Rigollet, The query complexity of sampling from strongly log-concave distributions in one dimension , Proceedings of Thirty Fifth Conference on Learning Theory, vol. 178, PMLR, 2022, pp. 2041-2059.

[CGLL22] Sinho Chewi, Patrik Gerber, Holden Lee, and Chen Lu, Fisher information lower bounds for sampling , arXiv preprint arXiv:2210.02482 (2022).

[Cha04] Djalil Chafaï, Entropies, convexity, and functional inequalities, on Φ -entropies and Φ -sobolev inequalities , Journal of Mathematics of Kyoto University 44 (2004), no. 2, 325-363.

[CLA + 21] Sinho Chewi, Chen Lu, Kwangjun Ahn, Xiang Cheng, Thibaut Le Gouic, and Philippe Rigollet, Optimal dimension dependence of the Metropolis-Adjusted Langevin Algorithm , Conference on Learning Theory, PMLR, 2021, pp. 1260-1300.

[CLW23] Yu Cao, Jianfeng Lu, and Lihan Wang, On explicit L 2 -convergence rate estimate for underdamped Langevin dynamics , Archive for Rational Mechanics and Analysis 247 (2023), no. 5, 90.

[DBCD19] George Deligiannidis, Alexandre Bouchard-Côté, and Arnaud Doucet, Exponential ergodicity of the Bouncy Particle Sampler , Annals of Statistics 47 (2019), no. 3.

[DCWY19] Raaz Dwivedi, Yuansi Chen, Martin J Wainwright, and Bin Yu, Log-concave sampling: Metropolis-Hastings algorithms are fast , Journal of Machine Learning Research 20 (2019), no. 183, 1-42.

[DGM20] Alain Durmus, Arnaud Guillin, and Pierre Monmarché, Geometric ergodicity of the Bouncy Particle Sampler , Annals of applied probability 30 (2020), no. 5, 2069-2098.

[DKTZ20] Ilias Diakonikolas, Vasilis Kontonis, Christos Tzamos, and Nikos Zarifis, Learning halfspaces with Massart noise under structured distributions , Conference on Learning Theory, PMLR, 2020, pp. 1486-1513.

[DM17] Alain Durmus and Éric Moulines, Nonasymptotic convergence analysis for the unadjusted Langevin algorithm , The Annals of Applied Probability 27 (2017), no. 3, 1551-1587 (en).

[DRD20] Arnak S Dalalyan and Lionel Riou-Durand, On sampling from a log-concave density using kinetic Langevin diffusions , Bernoulli 26 (2020), no. 3, 1956-1988.

[EGZ19] Andreas Eberle, Arnaud Guillin, and Raphael Zimmer, Couplings and quantitative contraction rates for Langevin dynamics , The Annals of Probability 47 (2019), no. 4, 1982-2010.

[EH21] Murat A Erdogdu and Rasa Hosseinzadeh, On the convergence of Langevin Monte Carlo: The interplay between tail growth and smoothness , Conference on Learning Theory, PMLR, 2021, pp. 1776-1822.

[EHZ22] Murat A Erdogdu, Rasa Hosseinzadeh, and Shunshi Zhang, Convergence of Langevin Monte Carlo in chi-squared and Rényi divergence , International Conference on Artificial Intelligence and Statistics, PMLR, 2022, pp. 8151-8175.

[EMS18] Murat A Erdogdu, Lester Mackey, and Ohad Shamir, Global non-convex optimization with discretized diffusions , Advances in Neural Information Processing Systems 31 (2018).

[Erb14] Matthias Erbar, Gradient flows of the entropy for jump processes , Annales de l'IHP Probabilités et statistiques, vol. 50, 2014, pp. 920-945.

[FYC23] Jiaojiao Fan, Bo Yuan, and Yongxin Chen, Improved dimension dependence of a proximal algorithm for sampling , arXiv preprint arXiv:2302.10081 (2023).

[GB09] Alan Genz and Frank Bretz, Computation of multivariate normal and t-probabilities , vol. 195, Springer Science & Business Media, 2009.

[GBH04] Alan Genz, Frank Bretz, and Yosef Hochberg, Approximations to multivariate t integrals with application to multiple comparison procedures , Recent Developments in Multiple Comparison Procedures, Institute of Mathematical Statistics, 2004, pp. 24-32.

[GJPS08] Andrew Gelman, Aleks Jakulin, Maria Grazia Pittau, and Yu-Sung Su, A weakly informative default prior distribution for logistic and other regression models , The annals of applied statistics 2 (2008), no. 4, 1360-1383.

[GLL20] Rong Ge, Holden Lee, and Jianfeng Lu, Estimating normalizing constants for logconcave distributions: Algorithms and lower bounds , Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, 2020, pp. 579-586.

[GLL + 23] Sivakanth Gopi, Yin Tat Lee, Daogao Liu, Ruoqi Shen, and Kevin Tian, Algorithmic aspects of the log-Laplace transform and a non-Euclidean proximal sampler , arXiv preprint arXiv:2302.06085 (2023).

[GLM18] Joyee Ghosh, Yingbo Li, and Robin Mitra, On the use of Cauchy prior distributions for Bayesian logistic regression , Bayesian Analysis 13 (2018), no. 2, 359-383.

[Hai10] Martin Hairer, Convergence of Markov processes , Lecture notes (2010).

[HBE20] Ye He, Krishnakumar Balasubramanian, and Murat A Erdogdu, On the ergodicity, bias and asymptotic normality of randomized midpoint sampling method , Advances in Neural Information Processing Systems 33 (2020), 7366-7376.

[HBE24] , An analysis of Transformed Unadjusted Langevin Algorithm for Heavy-tailed Sampling , IEEE Transactions on Information Theory (2024).

[HFBE24] Ye He, Tyler Farghly, Krishnakumar Balasubramanian, and Murat A Erdogdu, Meansquare analysis of discretized Itô diffusions for heavy-tailed sampling , Journal of Machine Learning Research (to appear) (2024).

[HMW21] Lu-Jing Huang, Mateusz B Majka, and Jian Wang, Approximation of heavy-tailed distributions via stable-driven SDEs , Bernoulli 27 (2021), no. 3, 2040-2068.

[JG12] Leif T Johnson and Charles J Geyer, Variable transformation to obtain geometric ergodicity in the Random-Walk Metropolis algorithm , The Annals of Statistics 40 (2012), no. 6, 3050-3076.

[JR07] Søren Jarner and Gareth Roberts, Convergence of heavy-tailed Monte Carlo Markov Chain algorithms , Scandinavian Journal of Statistics 34 (2007), no. 4, 781-815.

[Kam18] Kengo Kamatani, Efficient strategy for the Markov chain Monte Carlo in highdimension with heavy-tailed target probability distribution , Bernoulli 24 (2018), no. 4B, 3711-3750.

[KN04] Samuel Kotz and Saralees Nadarajah, Multivariate t-distributions and their applications , Cambridge University Press, 2004.

[Kwa17] Mateusz Kwa´ snicki, Ten equivalent definitions of the fractional Laplace operator , Fractional Calculus and Applied Analysis 20 (2017), no. 1, 7-51.

[LST20] Yin Tat Lee, Ruoqi Shen, and Kevin Tian, Logsmooth gradient concentration and tighter runtimes for Metropolized Hamiltonian Monte Carlo , Conference on learning theory, PMLR, 2020, pp. 2565-2597.

[LST21a] , Lower bounds on Metropolized sampling methods for well-conditioned distributions , Advances in Neural Information Processing Systems 34 (2021), 18812-18824.

[LST21b] , Structured logconcave sampling with a Restricted Gaussian Oracle , Conference on Learning Theory, PMLR, 2021, pp. 2993-3050.

[LWME19] Xuechen Li, Yi Wu, Lester Mackey, and Murat A Erdogdu, Stochastic Runge-Kutta accelerates Langevin Monte Carlo and beyond , Advances in neural information processing systems 32 (2019).

[LZT22] Ruilin Li, Hongyuan Zha, and Molei Tao, Sqrt(d) Dimension Dependence of Langevin Monte Carlo , The International Conference on Learning Representations, 2022.

[MFWB22] Wenlong Mou, Nicolas Flammarion, Martin J Wainwright, and Peter L Bartlett, Improved bounds for discretization of Langevin diffusions: Near-optimal rates without convexity , Bernoulli 28 (2022), no. 3, 1577-1601.

[MHFH + 23] Alireza Mousavi-Hosseini, Tyler K. Farghly, Ye He, Krishna Balasubramanian, and Murat A. Erdogdu, Towards a Complete Analysis of Langevin Monte Carlo: Beyond Poincaré Inequality , Proceedings of Thirty Sixth Conference on Learning Theory, vol. 195, 2023, pp. 1-35.

[Nol20] John P Nolan, Univariate stable distributions , Springer, 2020.

[N¸ SR19] Than Huy Nguyen, Umut ¸ Sim¸ sekli, and Gaël Richard, Non-asymptotic analysis of Fractional Langevin Monte Carlo for non-convex optimization , International Conference on Machine Learning, 2019, pp. 4810-4819.

[PBEM23] Mathieu Le Provost, Ricardo Baptista, Jeff D Eldredge, and Youssef Marzouk, An adaptive ensemble filter for heavy-tailed distributions: Tuning-free inflation and localization , arXiv preprint arXiv:2310.08741 (2023).

[QM16] Di Qi and Andrew J Majda, Predicting fat-tailed intermittent probability distributions in passive scalar turbulence with imperfect models through empirical information theory , Communications in Mathematical Sciences 14 (2016), no. 6, 1687-1722.

[RRT17] Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky, Non-convex learning via stochastic gradient Langevin dynamics: A nonasymptotic analysis , Conference on Learning Theory, PMLR, 2017, pp. 1674-1703.

[SL19] Ruoqi Shen and Yin Tat Lee, The randomized midpoint method for log-concave sampling , Advances in Neural Information Processing Systems 32 (2019).

[SP15] Prashant D Sardeshmukh and Cécile Penland, Understanding the distinctively skewed and heavy tailed character of atmospheric and oceanic probability distributions , Chaos: An Interdisciplinary Journal of Nonlinear Science 25 (2015), no. 3.

[ ¸ SZTG20] Umut ¸ Sim¸ sekli, Lingjiong Zhu, Yee Whye Teh, and Mert Gurbuzbalaban, Fractional underdamped Langevin dynamics: Retargeting SGD with momentum under heavytailed gradient noise , International Conference on Machine Learning, 2020, pp. 89708980.

[Wib18] Andre Wibisono, Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem , Conference on Learning Theory, PMLR, 2018, pp. 2093-3027.

[WSC22a] Keru Wu, Scott Schmidler, and Yuansi Chen, Minimax mixing time of the Metropolisadjusted Langevin algorithm for log-concave sampling , The Journal of Machine Learning Research 23 (2022), no. 1, 12348-12410.

[WSC22b] , Minimax Mixing Time of the Metropolis-Adjusted Langevin Algorithm for Log-Concave Sampling , Journal of Machine Learning Research 23 (2022), no. 270, 1-63.

[WW15] Feng-Yu Wang and Jian Wang, Functional inequalities for stable-like Dirichlet forms , Journal of Theoretical Probability 28 (2015), no. 2, 423-448.

[YŁR22] Jun Yang, Krzysztof Łatuszy´ nski, and Gareth Roberts, Stereographic Markov Chain Monte Carlo , arXiv preprint arXiv:2205.12112 (2022).

[ZZ23] Xiaolong Zhang and Xicheng Zhang, Ergodicity of supercritical SDEs driven by α -stable processes and heavy-tailed sampling , Bernoulli 29 (2023), no. 3, 1933-1958.

A Lower Bound Proofs for the Langevin Diffusion and the Gaussian Proximal Sampler

While research on upper bounds of sampling algorithms' complexity has advanced considerably, the exploration of lower bounds is still nascent. [CGL + 22] explored the query complexity of sampling from strongly log-concave distributions in one-dimensional settings. [LZT22] established lower bounds for LMC in sampling from strongly log-concave distributions. [CBL22] presented lower bounds for sampling from strongly log-concave distributions with noisy gradients. [GLL20] focused on lower bounds for estimating normalizing constants of log-concave densities. Contributions by [LST21a] and [WSC22b] provide lower bounds in the metropolized algorithm category, including Langevin and Hamiltonian Monte Carlo, in strongly log-concave contexts. Finally, [CGLL22] contributed to lower bounds in Fisher information for non-log-concave sampling. In what follows, we take a different approach and rely on the arguments developed in [Hai10].

We begin by stating the following result which drives our lower bound strategy.

Lemma 1 ([Hai10, Theorem 5.1]) . Suppose µ and ν are probability measures on R d . Consider some G : R d → R + and f : R + → R + satisfying µ ( G ≥ y ) ≥ f ( y ) for all y ∈ R + . Then,

In particular, suppose Id · f : R + ∋ y ↦→ yf ( y ) ∈ R + is a bijection, then

for any m ≥ ∫ G d ν .

Proof. By the definition of total variation and Markov's inequality, for any y > 0

When Id · f is invertible, choosing y = (Id · f ) -1 (2 m ) implies yf ( y ) = 2 m and yields the desired result.

To apply Lemma 1 when the target density satisfies Assumption 1, we need to establish tail lower bounds for this density, which we do so via the following lemma. In the following, let ω d := π d/ 2 Γ(( d +2) / 2) denote the volume of the unit d -ball.

Lemma 2. Suppose π X ( x ) ∝ exp( -V ( x )) satisfies Assumption 1. Then, for all R > 0 ,

When focusing on dependence on R and d , we obtain,

where C ν 1 = 2 1 -ν 1 / 2 e -ν 1 (1+ ν 1 )Γ( ν 1 / 2) .

Proof. Without loss of generality assume V (0) = 0 . Via Assumption 1, we have the estimates for V ,

and similarly

Consequently, using the spherical coordinates,

Next, using the lower bound established on V and spherical coordinates, we obtain,

where B denotes the beta function. Plugging back into our tail lower bound, we obtain,

Moreover, by [MHFH + 23, Lemma 32] we have

which completes the proof.

Another element of Lemma 1 is controlling the growth of E [ G ( X t )] throughout the process. The following lemma achieves such control under the Langevin diffusion.

Lemma 3. Suppose ( X t ) t ≥ 0 is the solution to the Langevin diffusion starting at X 0 with the corresponding potential V ( x ) satisfying Assumption 1. Let G ( x ) = exp( κ ˜ V ( x )) where ˜ V ( x ) = d + ν 2 2 ln(1 + | x | 2 ) and κ ≥ 2 d + ν 2 ∨ 1 . Then,

Proof. Recall the generator of the Langevin diffusion L ( · ) = ∆ · -⟨∇ V , ∇·⟩ . Then,

Integrating the above inequality completes the proof.

.

With the above lemmas in hand, we are ready to present the proof of Theorem 1.

Proof of Theorem 1. To apply Lemma 1 we choose G ( x ) = exp( κ ˜ V ( x )) where ˜ V ( x ) = d + ν 2 2 ln(1+ | x | 2 ) with κ ≥ 1 ∨ 2 d + ν 2 . By Lemma 2 we have

Moreover, define and notice that

Using the fact that

we have

where ˜ C ν 1 ,ν 2 = C ν 1 e -1+ ν 2 /d 8 . By plugging in the value of y ∗ from (3), we obtain,

Thus for sufficiently large C ′ ν 1 ,ν 2 , there exists C ′′ ν 1 ,ν 2 such that

Choosing κ according to the statement of the theorem completes the proof.

In order to prove a similar theorem for the Gaussian proximal sampler, we control the growth of E [ G ( x k )] for the iterates of the proximal sampler via the following lemmas.

with g (0) := E [ G ( X 0 )] . Then by Lemma 3 we have E [ G ( X t )] ≤ g ( t ) and we can invoke Lemma 1 to obtain

where we used the fact that 1 + x ≤ e x for all x ∈ R and g ( t ) is non-decreasing in t . Choose

for a sufficiently large constant C ′ ν 1 ,ν 2 ≥ 1 . For simplicity, let

Lemma 4. Suppose ( x k , y k ) k are the iterates of the Gaussian proximal sampler with step size η and target density π X ∝ exp( -V ) for some V : R d → R . Let G ( x ) = exp( κV ( x )) with κ ≥ 1 . Then, for every k ≥ 0 ,

where z ∼ N (0 , I d ) is sampled independently from x k .

Proof. Recall that π X | Y ( x | y ) ∝ exp ( -V ( x ) -| x -y | 2 2 η ) . Therefore,

where z 1 ∼ N (0 , I d ) . Furthermore,

Therefore,

Recall y k = x k + √ ηz 2 where z 2 ∼ N (0 , I d ) is independent from x k . By the towering property of conditional expectation,

where z ∼ N (0 , I d ) is independent from x k , which completes the proof.

In order to provide a more refined control over E [ G ( x k )] , we need additional assumptions on V . In particular, when considering the generalized Cauchy density, we arrive at the following lemma.

Lemma 5. Suppose ( x k , y k ) k are the iterates of the Gaussian proximal sampler with step size η and target density π X ∝ exp( -V ) satisfies

for all x ∈ R d . Let G ( x ) = exp( κV ( x )) with κ ≥ 1 ∨ 2 d + ν 2 . Then, for every k ≥ 0 ,

Proof. From Lemma 4, we have

where z ∼ N (0 , I d ) is independent from x k . Consider the Brownian motion starting at x k , denoted by Z t = B t + x k where ( B t ) is a standard Brownian motion in R d . Notice that the generator for the

process d Z t = d B t is L = 1 ∆ . Therefore,

Integrating the above inequality yields

The proof is complete by noticing that Z 0 = x k and Z t = x k + √ 2 ηz for t = 2 η .

Proof of Theorem 2. Notice that the statements of Lemmas 3 and 5 are virtually the same by changing t to 2 kη . Using this fact, the rest of the proof follows exactly the same as the proof of Theorem 1.

B Proofs for the Stable Proximal Sampler

B.1 Preliminaries

In this section, we introduce additional preliminaries on the isotropic α -stable process, the fractional Poincaré-type inequalities, the fractional Laplacian and the fractional heat flow.

The Lévy process is a stochastic process that is stochastically continuous with independent and stationary increments. Due to the stochastic continuity, the Lévy processes have càdlàg trajectories, which allows jumps in the paths. A Lévy process Y t is uniquely determined by a triple ( b, A, ν ) through the following Lévy-Khinchine formula: for all t ≥ 0 and ξ ∈ R d ,

where b ∈ R d is a drift vector. A ∈ R d × d is the covariance matrix of the Brownian motion in the Lévy-Itô decomposition[App09, Thereom 2.4.16] and ν is the Lévy measure related to the jump parts in the Lévy-Itô decomposition.

The rotationally invariant(isotropic) stable process is a special case for the Le´ vy process when b = 0 , A = 0 and ν is the measure given by

Based on the Lévy-Khinchine formula (4), if we initialize the process at x ∈ R d , its characteristic function is given by

The index of stability α ∈ (0 , 2] determines the tail-heaviness of the densities: the smaller is α , the heavier is the tail. The parameter t in (6) measures the spread of X t around the center. When α = 2 , the stable process pertains to the Brownian motion running with a time clock twice as fast as the standard one and hence it has continuous paths. When α ∈ (0 , 2) , the stable process paths contain discontinuities, which are often referred as jumps. At each fixed time, unlike the Brownian motion, the α -stable process density only has a finite p th -moment for p < α , i.e.

When d = 1 , the fractional absolute moment formula for m ( α ) p can be derived explicitly, see [Nol20, Chapter 3.7]. When d > 1 , the explicit formula for m ( α ) p is only known in some special cases. For example, when α = 1 , m (1) p = Γ(( d + p ) / 2)Γ((1 -p ) / 2) Γ( d/ 2)Γ(1 / 2) for all p < 1 . Another good property of α -stable process is the self-similarity. By examining the characteristic functions, it is easy to verify that the isotropic α -stable process is self-similar with the Hurst index 1 /α , i.e. X ( α ) at and a 1 /α X ( α ) t have the same distribution. Or equivalently, p ( α ) t ( x ) = t -d α p ( α ) 1 ( t -1 α x ) for all x ∈ R d and t > 0 .

The fractional Laplacian operator in R d of order α is denoted by -( -∆) α/ 2 for α ∈ (0 , 2] . It was introduced as a non-local generalization of the Laplacian operator to model various physical phenomenons. In [Kwa17], ten equivalent definitions of the fractional Laplacian operator are introduced. Here we recall two of them:

(a) Distributional definition: For all Schwartz functions ϕ defined on R d , we have

(b) Singular integral definition: For a limit in the space L p ( R d ) , p ∈ [1 , ∞ ) , we have

where B r is the unit ball with radius r centered at the origin.

The fractional Laplacian can be understood as the infinitesimal generator of the stable Le´ vy process. More explicitly, the semigroup defined by the transition probability p ( α ) t in (2) has the infinitesimal generator -( -∆) α/ 2 , i.e. the density function p ( α ) t satisfies the following equation in the sense of distribution, [BHJ08]:

(7) is usually referred as the α -fractional heat flow. When α = 2 , -( -∆) α/ 2 is the Laplacian operator and (7) becomes the heat flow.

Proposition 2 (From FPI to PI) . When ϑ → 2 -, the ϑ -FPI reduces to the classical Poincaré inequality with Dirichlet form E µ ( ϕ ) = ∫ |∇ ϕ ( x ) | 2 d x for any smooth bounded ϕ : R d → R d .

Proof. It suffices to prove that E ( ϑ ) µ ( ϕ ) converges to E µ ( ϕ ) as ϑ → 2 -for any smooth function ϕ . Recall the definition of E ( ϑ ) µ ( ϕ ) :

̸

where c d,ϑ = O (2 -ϑ ) as ϑ → 2 -. Now we rewrite the inside integral in E ( ϑ ) µ ( ϕ ) and split the integral region into a centered unit ball, denoted as B 1 , and its complement:

̸

For I 2 , we have

As a result, the term in E ( ϑ ) µ ( ϕ ) that is induced by I 2 satisfies

̸

For I 1 , we have when ϑ > 1 ,

where ∥ ϕ ∥ C i ( R d ) := sup x ∈ R d | ϕ ( i ) ( x ) | for i = 1 , 2 . As a result, the term in E ( ϑ ) µ ( ϕ ) that is induced by I 1 satisfies

̸

Therefore we have E ( ϑ ) µ ( ϕ ) → c d,ϑ ∫ R d ∫ B 1 |⟨∇ ϕ ( y ) ,z ⟩| 2 | z | d + ϑ µ ( y )d z d y as ϑ → 2 -. Last, we prove the limit is equivalent to 2 E µ ( ϕ ) . For i = j , we have

̸

where ˜ z k = z k for all k = j and ˜ z j = -z j . Therefore, ∫ B 1 ∂ i ϕ ( y ) ∂ j ϕ ( y ) z i z j d z = 0 . As a result,

and the proof follows from c d,ϑ π d 2 (2 -ϑ )Γ( d 2 +1) → 2 as ϑ → 2 -.

B.2 χ 2 convergence under FPI

In this section, we study the decaying property of χ 2 -divergence from ρ X k to π X , where ρ X k is the law of x k . In the following analysis, we denote ρ k = ρ X,Y k as the law of ( x k , y k ) , ρ Y k the law of y k . We will analyze the two steps in the stable proximal sampler separately.

Step 1. In the following proposition, we study the decay of χ 2 -divergence in step 1.

Proposition 3. Assume that π X satisfies the α -FPI with parameter C FPI ( α ) , then for each k ≥ 0 ,

Proof of Proposition 3. For the simplicity of notations, we will write p ( α ) and p ( α ) t as p and p t respectively in this proof. Since x k ∼ ρ X k and y k | x k ∼ p ( η ; x, · ) , we have

Therefore, we can view ρ Y k as ρ X k evolving along the following factional heat flow

That is if ˜ ρ 0 = ρ X k , then ˜ ρ η = ρ Y k . Similarly, since π Y = π X ∗ p η , if ˜ ρ 0 = π X , then ˜ ρ η = π Y . For any t ∈ [0 , η ] , define π X t = π X ∗ p t and ρ X t = ρ X k ∗ p t . The derivative of ϕ -divergence from ρ X t to π X t can be calculated as

where in the second identity we used the distributional definition of the fractional Laplacian. Next according to the singular integral definition of fractional Laplacian, we have

where B r = { x ∈ R d : | x | ≤ r } and c d,α is given in (5). With (8), we have

When ϕ ( r ) = ( r -1) 2 , ∫ R d ϕ ( ρ t X π t X ) π t X d x = χ 2 ( ρ X t | π t X ) and we have

According to [Cha04, Theorem 23], p t satisfies α -FPI with parameter t for all t ∈ (0 , η ] . Since π X also satisfies the α -FPI with parameter C FPI ( α ) , Lemma 6 implies that π t X = π X ∗ p t satisfies the α -FPI with parameter C FPI ( α ) + η for all t ∈ (0 , η ] . Therefore we have

Last, according to Gronwall's inequality we have

Step 2. In this step, we study the decay of χ 2 -divergence in step 2. building on the work by [CCSW22]. According to the R α SO, we have ρ X k +1 ( x ) = ∫ R d π X | Y ( x | y ) ρ Y k ( y )d y . Also notice that π X ( x ) = ∫ R d π X | Y ( x | y ) π Y ( y )d y . According to the data processing inequalities, χ 2 divergence won't increase after step 2, i.e. χ 2 ( ρ X k +1 | π X ) ≤ χ 2 ( ρ Y k | π Y ) .

Combining our results in Step 1 and Step 2 , we prove Theorem 3.

Lemma 6. Let µ 1 , µ 2 be two probability densities satisfying the ϑ -FPI with parameters C 1 , C 2 respectively. Then µ 1 ∗ µ 2 satisfies the ϑ -FPI with parameter C 1 + C 2 .

Proof of Lemma 6. Let X,Y be two independent random variables such that X ∼ µ 1 and Y ∼ µ 2 . Then X + Y ∼ µ 1 ∗ µ 2 . According to variance decomposition, we have for any function ϕ ,

Since X ∼ µ 1 and µ 1 satisfies the ϑ -FPI with parameter C 1 , we have

̸

therefore we have

̸

Since Y ∼ µ 2 and µ 2 satisfies the ϑ -FPI with parameter C 2 , we have

̸

̸

where the last inequality follows from Jensen's inequality. Combining (9) and (10), we have

̸

̸

where the second inequality follows from Fatou's lemma.

B.3 Implementation of the Stable Proximal Sampler

In this section we discuss the implementation of the R α SO step in our stable proximal sampler. We introduce an exact implementation of the R α SO step without optimizing the target potential and the proofs for Corollary 3 and Proposition 1.

Rejection sampling without optimization . Suppose a uniform lower bound of the target potential is known, i.e. there is a constant C Low such that inf x ∈ R d V ( x ) ≥ C Low > -∞ , R α SO at each step can be implemented exactly via a rejection sampler with proposals ˜ x k +1 following p ( α ) η ( · -y k ) and the acceptance probability exp( -V (˜ x k +1 ) + C Low ) . Then the expected number of rejections, N , satisfies

Without loss of generality, we assume x ∗ = 0 , which always hold if we translate the potential V by V (0) . Then we have

̸

̸

where the second inequality follows from Assumption 3 and the last inequality follows from the proof of Corollary 3. With the above estimation, we can pick η = Θ( C 1 β Low d -1 2 L -1 β ) and the expected number of rejections satisfies log N = O ( C Low + LM ) with M = E π X [ | X | β ] + χ 2 ( ρ X 0 | π X ) E π X [ | X | 2 β ] 1 2 .

Proof of Corollary 3. The expected number of iterations conditioned on y k in the rejection sampling is

WLOG, assume x ∗ = 0 . Since V satisfies Assumption 3, we have

Therefore, when η = Θ( d -1 2 L -1 β ) , the expected number of rejections N is of order E [exp( L | y k | β ] . Since π X satisfies a 1 -FPI with parameter C FPI (1) , according to [Cha04], p t satisfies the 1 -FPI with parameter η for any t ∈ (0 , η ) . Last it follows from Theorem 9 that for any η > 0 , to achieve a ε -accuracy in χ 2 divergence, we need to perform the stable proximal sampler K steps with

Proof of Proposition 1. For all k ≥ 0 , we have

where the last two inequalities follow from the data processing inequality. Therefore, TV(˜ ρ X k , ρ X k ) ≤ kε TV +TV(˜ ρ X 0 , ρ X 0 ) for all k ≥ 1 .

Next, the iteration complexity of Algorithm 2 with an inexact R α SO can be obtained from Proposition 1. Since ˜ ρ X 0 = ρ X 0 , according to Pinsker's inequality, we have

For any ε > 0 and any K satisfies

if the R α SO can be implemented inexactly with ε TV ≤ ε 2 K , the density of the K th iterate of Algorithm 2 is ε -close to the target in the total variation distance, i.e. TV(˜ ρ K X , π X ) ≤ ε .

B.4 Convergence under Weak Fractional Poincaré Inequality

Our main result for Algorithm 2 in Theorem 3 is proved under the assumption the target satisfying α -FPI. Furthermore, for the rejection-sampling based implementation of the R α SO in Algorithm 3, the parameter α is set to be 1 . In order to use Theorem 3 for the case of generalized Cauchy targets, one has to check if the α -FPI is satisfied or not, which depends on the degrees of freedom parameter ν of the generalized Cauchy desity. Specifically, when ν ≥ 1 , 1 -FPI is satisfied and we hence have Corollary 5, part (i) based on Theorem 3. When ν ∈ (0 , 1) , 1 -FPI is not satisfied and hence Theorem 3 no longer applies.

To tackle this issue, we now introduce a generalization of Theorem 3 to the case when the target satisfies a weak version of Fractioanl Poincaré inequality (wFPI) and provide convergence guarantees for the stable proximal sampler in χ 2 -divergence.

Definition 3 (weak Fractional Poincaré Inequality) . For ϑ ∈ (0 , 2) , a probability density µ satisfies a ϑ -weak fractional Poincaré inequality if there exists a decreasing function β WFPI ( ϑ ) : R + → R + such that for any ϕ : R d → R in the domain of E ( ϑ ) µ with µ ( ϕ ) = 0 , we have

where E ( ϑ ) µ is a non-local Dirichlet form associated with µ defined as

̸

The wFPI is satisfied by any probability density that is locally bounded, and is hence extremely general. Setting the parameter r = 0 , wFPI reduces to FPI with C FPI ( ϑ ) = β WFPI ( ϑ ) (0) .

Theorem 5. Assume that π X satisfies the α -wFPI with parameter β WFPI ( α ) ( r ) for some α ∈ (0 , 2) . Then for any step size η > 0 and initial condition ρ X 0 such that R ∞ ( ρ X 0 | π X ) < ∞ , the k th iterate of the stable proximal sampler with parameter α (Algorithm 2) satisfies

The proof of Theorem 5 follows the same two-step analysis as it is introduced in the beginning of Section B.2. The convergence property corresponding to Step 1 is stated in the following Proposition.

Proposition 4. Assume that π X satisfies the α -wFPI with parameter β WFPI ( α ) for some α ∈ (0 , 2) , then for each k ≥ 0 , r > 0 ,

Proof of Proposition 4. In the stable proximal sampler with parameter α , we have ρ Y k = ρ X k ∗ p ( α ) η and π Y = π X ∗ p ( α ) η . Therefore we can view ρ Y k and π Y as ρ X k and π X evolving along the fractional

heat flow by time η respectively. For any t ∈ [0 , η ] , define π X t = π X ∗ p ( α ) t and ρ X t = ρ X k ∗ p ( α ) t . We have

According to [Cha04, Theorem 23], p ( α ) t satisfies α -FPI with parameter η for all t ∈ (0 , η ] . According to Lemma 7, π X t satisfies the α -wFPI with β WFPI ( α ) ( r ) + η . Therefore we get

where the last inequality follows from the definition of Renyi-divergence and the data processing inequality. Last, (11) follows from Gronwall's inequality.

Proof of Theorem 5. According to Proposition 4, the χ 2 decaying property in step 1 of the algorithm is as follows,

In step 2, we have ρ X k +1 = ρ Y k ∗ π X | Y and π X = π Y ∗ π X | Y . Therefore according to the data processing inequality, we get

where the last inequality follows from the data processing inequality. Last, apply the above iterative relation k times and we prove (11).

Lemma 7. Let µ 1 be a probability density on R d satisfying the ϑ -wFPI with parameter β WFPI ( ϑ ) ( r ) . Let µ 2 be a probability density on R d satisfying the ϑ -FPI with parameter C FPI ( ϑ ) . Then µ 1 ∗ µ 2 satisfies ϑ -wFPI with parameter β WFPI ( ϑ ) ( r ) + C FPI ( ϑ ) .

Proof of Lemma 7. Let X,Y be two independent random variables such that X ∼ µ 2 and Y ∼ µ 1 . According to variance decomposition, we have for any function ϕ such that µ 1 ∗ µ 2 ( ϕ ) = 0 ,

Var µ 1 ∗ µ 2 ( ϕ ) = Var ( ϕ ( X + Y )) = E [ Var ( ϕ ( X + Y ) | Y )] + Var ( E [ ϕ ( X + Y ) | Y ]) . Since X ∼ µ 2 and µ 2 satisfies the ϑ -FPI with parameter C FPI ( ϑ ) , we have

̸

Since Y ∼ µ 1 and µ 1 satisfies the ϑ -wFPI with parameter β WFPI ( ϑ ) , following the proof of Lemma 6, we have

Var ( E [ ϕ ( X + Y ) | Y ])

̸

̸

where the last inequality follows from the fact that µ 1 ∗ µ 2 ( ϕ ) = 0 and the convexity ∥·∥ ∞ . Combining (12) and (14), we have

̸

̸

B.5 Proofs for the Generalized Cauchy Examples

In this section, we provide proofs for the two corollaries in Section 3.2.

Proof of Corollary 4. According to [WW15, Corollary 1.2], π ν satisfies α -FPI with parameter C FPI ( ϑ ) for any α ≤ min(2 , ν ) . Therefore it follows from Theorem 3 that

According to [MHFH + 23, Corollary 22], when ρ X 0 = N (0 , I d ) and d ≥ 2 , R ∞ ( ρ X 0 | π ν ) ≤ ln(2 ν/ 2 Γ( ν/ 2)) + ln( d + ν 2 e ) which implies χ 2 ( ρ X 0 | π ν ) = Θ( d ) . Therefore Corollary 4 follows from (15) and η ∈ (0 , 1) .

Proof of Corollary 5. We prove the two part in the Corollary separately:

(i) When ν ≥ 1 , according to [WW15, Corollary 1.2] π ν satisfies the 1 -FPI with parameter C FPI (1) . Corollary 3 applies with L = 4( d + ν ) and β = 1 / 4 and the iteration complexity of Algorithm 2 is of order O ( C FPI (1) d 1 2 ( d + ν ) 4 ln( χ 2 ( ρ X 0 | π ν ) /ε ) ) .

(ii) When ν ∈ (0 , 1) , according to [WW15, Corollary 1.2], there exists a positive constant c such that π ν satisfies the 1 -wFPI with parameter

Theorem 5 implies that

For any ε > 0 and k ≥ 1 , pick r = exp ( -2 νR ∞ ( ρ X 0 | π ν ) ) c ν ε ν ( k +1) ν η ν , we have χ 2 ( ρ X k | π ν ) ≤ ε if

Corollary 3 applies with L = ( d + ν ) /ν and β = ν/ 4 . Therefore, by choosing η = Θ( d -1 2 ( d + ν ) -4 ν ) , the iteration complexity in Algorithm 2 is of order

̸

C Proofs for the Lower Bounds on the Stable Proximal Sampler

In this section we introduce the proofs for the lower bounds for the stable proximal sampler with parameter α when the target is the generalized Cauchy density with degrees of freedom strictly smaller than α . The lower bound is proved following the idea introduced in Section 2.

Lemma 8. Suppose ( x k , y k ) k are the iterates of the stable proximal sampler with parameter α , step size η and target density π X ∝ exp( -V ) for some V : R d → R . Let G ( x ) = exp( κV ( x )) with κ ∈ (0 , 1) . Then, for every k ≥ 0 ,

where z k , with density p ( α ) 1 , is sampled independently from x k .

Proof of Lemma 8. Recall that π X | Y ( x | y ) ∝ π X ( x ) p ( α ) ( η ; x, y ) . We have

where Z y k = ∫ π X ( x ) p ( α ) η ( x -y k )d x = E [ π X ( y k + η 1 α z k ) | y k ] and z k is the α -stable random vector with density p ( α ) 1 , which is independent to y k , x k . Let T : R + → R be T ( r ) = r -κ . Since κ ∈ (0 , 1) , T is convex and r ↦→ rT ( r ) is concave. According to the fact that G ( x ) = T ( π X )( x ) and Jensen's inequality, we have

Since T is convex, apply Jensen's inequality again and we get

where z ′ k is the α -stable random vector with density p ( α ) 1 , which is independent to x k , z k and the last identity follows from the self-similarity of α -stable process with ¯ z k ∼ p ( α ) 1 which is independent to x k .

Lemma 9. Suppose ( x k , y k ) k are the iterates of the stable proximal sampler with parameter α , step size η and target density π X ∝ exp( -V ) satisfies

for some ν 2 ∈ (0 , α ) and for all x ∈ R d . Let G ( x ) = exp( κV ( x )) with

Then, for every k ≥ 0 and for all r > 0 ,

where m ( α ) κ ( d + ν 2 ) = E [ | z k | κ ( d + ν 2 ) ] with z k being an α -stable random vector with density p ( α ) 1 . Moreover, for every N ≥ 0 ,

where ≲ is hiding a uniform positive constant factor.

Proof of Lemma 9. Without loss of generality assume V (0) = 0 . Then, we have that,

where the first inequality follows from the Young's inequality and m ( α ) κ ( d + ν 2 ) = E [ | z k | κ ( d + ν 2 ) ] with z k being an α -stable random vector with density p ( α ) 1 . (17) follows from Lemma 8. Furthermore, by induction we have

Pick r = 2 κ ( d + ν 2 ) N and (18) is proved.

Proof of Theorem 4. To apply Lemma 1, we choose G ( x ) = exp( κV ( x )) with κ ∈ ( ν 2 ( d + ν 2 ) -1 , α ( d + ν 2 ) -1 ) ⊂ (0 , 1) . Without loss of generality assume V (0) = 0 . Via Assumption 1, we have the estimates for V ,

We then invoke Lemma 1 and Lemma 9 to obtain

The fact that κ ∈ ( ν 2 ( d + ν 1 ) -1 , α ( d + ν 2 ) -1 ) ensures that the supremum on the right side is always positive. In particular, picking y such that

we obtain that

where ≳ is hiding a uniform positive constant factor. Therefore, for any α ∈ ( ν 2 ( d + ν 2 ) d + ν 1 , 2] and δ ∈ (0 , α -ν 2 ( d + ν 2 ) d + ν 1 ) , we can choose κ = α -δ d + ν 2 ∈ ( ν 2 d + ν 1 , α d + ν 2 ) and get that

TV(

ρ

X

N

, π

)

Theorem 4 then follows by taking τ = α -δ .

X

Position

15

20

0.35

0.30

GaussianProximalSampler

StableProximalSampler

istance

20.0

17.5

15.0

Average

10

GaussianProximalSampler

StableProximalSampler

0.25

0.20

serstei

12.5

10.0

StableProximalSampler

D

0.15

7.5

5

0

0.10

0.05

Was

S

S

5.0

2.5

0.0

0

20

40

60

80

100

0.00

-5

0

5

10

15

20

25

30

0

20

40

60

80

100

Stens

Position

Stens

C.1 Further Discussions on Lower bounds of the stable proximal sampler

To derive a lower bound for the stable proximal sampler with parameter α , it is worth mentioning that there is an extra difficulty applying our method when ν ≥ α . Recall that when ν ∈ (0 , α ) , π ν has heavier tail than ρ X k does. Therefore, when we apply

to study the lower bound, it suffices to derive a lower bound on π ν ( G ≥ y ) , and an upper bound on ρ X k ( G ≥ y ) which is smaller than the lower bound on π ν ( G ≥ y ) . Deriving these bounds is not too hard: the lower bound can be obtained by looking at an explicit integral against π ν directly and the upper bound is derived based on the fractional absolute moment accumulation of the isotropic α -stable random variables along the stable proximal sampler.

However, when ν ≥ α , we expect that ρ X k has heavier tail than π ν . Therefore, to apply (19), we need to find an upper bound on π ν ( G ≥ y ) , and a lower bound on ρ X k ( G ≥ y ) which is smaller than the upper bound on π ν ( G ≥ y ) . Notice that ρ X k ( G ≥ y ) is a quantity varying along the trajectory of the stable proximal sampler. Deriving a lower bound along the trajectory is essentially more challenging than deriving an upper bound.

In order to derive a satisfying lower bound in this case, it hence remains to characterize the stable proximal sampler as an approximation of an appropriate gradient flow, just as that the Browniandriven proximal sampler can be interpret as the entropy-regularized JKO scheme in [CCSW22]; see also Section 5. To understand this kind of gradient flow approximations itself is an interesting future work as it may help us to understand and characterize the class of MCMC samplers that utilize heavy-tail samples to approximate lighter-tail target densities, which is non-standard compared to commonly used MCMC samplers such as ULA, MALA, etc.

D Numerical Illustrations

In this section, we present numerical results that illustrate the improved performance of the proximal sampler with stable oracles ( α = 1 ) compared to that with Gaussian oracles. We first sample from the one-dimensional student-t distribution with center zero and 4 degrees of freedom by running the proximal samplers with different oracles in parallel for 100 times. Each individual chain is run for 100 iterations with step-size η = 0 . 1 . Figures 1,2,3 present the convergence results for different initializations x 0 = 20 , 5 , -5 respectively. In each figure, the first column shows the means and variances of the iterates along the trajectories; the center column shows the histograms of the last iterates and the target density (red curve); the last column shows the convergence of Wasserstein2 distance along the trajectories. We also sample from the two-dimensional student-t distribution with center at the origin and 4 degrees of freedom by running the proximal samplers with different oracles in parallel for 30 times. Each individual chain is run for 20 iterations with step-size η = 0 . 1 with the initialization at x 0 = [5 , 1] . In Figure 4, we present the convergence results, the first column showing the means and variances of the first-coordinates along the trajectories, the center column showing the histograms of first-coordinate in the last iterates and the first-coordinate marginal density of the target distribution (red curve), and the last column showing the convergence of Wasserstein2 distance along the trajectories.

1.0

5

GaussianProximalSampler

GaussianProximalSampler

WassersteinDistance

5

GaussianProximalSampler

Position

4

StableProximalSampler

0.8

StableProximalSampler

4

StableProximalSampler

3

Average

2

Densit

m

1

0.4

2

0

0.2

1

0

20

40

60

80

100

0.0

-4

-2

0

2

4

6

8

10

0

20

40

60

80

100

Stens

Position

Stens

1.0

2

GaussianProximalSampler

GaussianProximalSampler

WassersteinDistance

5

Position

StableProximalSampler

0.8

StableProximalSampler

4

0

m

AverageR

-2

0.4

2

0.2

1

0

20

40

60

80

100

0.0

-8

-6

4

2

0

2

4

6

0

0

20

40

60

80

100

Stens

Position

Stens

10

1.0

6

Position

8

StableProximalSampler

GaussianProximalSampler

0.8

StableProximalSampler

GaussianProximalSampler

Wasserstein Distance

5

GaussianProximalSampler

6

4

4

Average

3

2

0.4

2

0

-2

0.2

0.0

2.5

5.0

7.5

10.0

12.5 15.017.5

0.0

-8

-6

-4

-2

0

2

4

6

8

0

0.0

2.5

5.0

7.5

10.0

12.5 15.017.5

Stens

Position

Stens

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claim in the abstract and introduction is the separation result between Gaussian and Proximal Sampler. The rest of the sections are exactly stating (and proving) the aforementioned separation result.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please see Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are listed in the respective theorem. The (correct) proofs are provided in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have read the Ethics Guideline and followed it in the paper preperation.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work is primarily theoretical.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper investigates the complexity of sampling from heavy-tailed distributions and presents a distinction between obtaining high-accuracy and low-accuracy guarantees. It analyzes two types of proximal samplers: those based on Gaussian oracles and those based on stable oracles. The main findings are that Gaussian oracle-based samplers can only achieve low-accuracy guarantees when sampling from heavy-tailed distributions, while stable oracle-based samplers can achieve high-accuracy guarantees. Additionally, the paper establishes lower bounds for samplers using the stable oracle, indicating that the presented upper bounds are optimal and cannot be fundamentally improved.

STRENGTHS:
1. The problem is well-motivated and interesting. 
2. Designed the algorithms and derived the upper bounds and lower bounds for different settings. 
3. The authors also provided insightful discussion.
4. The authors provided solid theoretical proof for the results.

WEAKNESSES:
There is no experiment to verify the theoretical findings.

QUESTIONS:
1. Can you give an example in the real-world to motivate your problem?
2. Is it possible to run some experiments to verify your results?

LIMITATIONS:
There is no experiment.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-04

comments: SUMMARY:
This paper studies the problem of heavy-tailed sampling. First, the paper shows that while the gaussian proximal samplers are efficient for light-tailed targets, they are not accurate for heavy-tailed ones; the paper develops a lower bounds for the Gaussian proximal samplers, which reveals a fundamental challenge in heavy-tailed settings.

Then, the paper proceeds to develop a novel samplers based on restricted alpha-stable oracle; the insight is to replace the standard heat equation in gaussian oracle with a fractional heat flow. The paper proves that under suitable conditions the proposed sampler is efficient for heavy-tailed targets. Additionally, the paper proposes a practical implementation for a particular case of alpha=1.

STRENGTHS:
- Novel theoretical analysis for the gaussian oracle sampler, which provides a new insight to developing sampling algorithms

- A novel methodology for heavy-tailed sampling

WEAKNESSES:
- The paper is purely theoretical and lacks experimental evaluation; it would be nice to at least have a toy illustration for the implementable algorithm 2+3 in the alpha=1 case.

- As the authors discussed in Sec5, the current paper does not present implementable algorithms for general alpha values in (0,2).

QUESTIONS:
- I wonder if the efficiency rejection sampling efficiency in Alg.3 has been taken into account of the sampler's theoretical complexity and practical complexity?

- Maybe I am missing this -- what is the impact of alpha?

LIMITATIONS:
Most of the limitations have been touched upon in sec 5. Otherwise see the weakness comments.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

comments: SUMMARY:
The paper focus on studying the complexity of heavy-tailed sampling and present a separation result in terms of obtaining high-accuracy versus low-accuracy guarantees. Their results are presented for proximal samplers that are based on Gaussian versus stable oracles. Authors show that proximal samplers based on the Gaussian oracle have a fundamental barrier in that they necessarily achieve only low-accuracy guarantees when sampling from a class of heavy-tailed targets. In contrast, proximal samplers based on the stable oracle exhibit high-accuracy guarantees, thereby overcoming the aforementioned limitation. They also prove lower bounds for samplers under the stable oracle and show that our upper bounds cannot be fundamentally improved.

STRENGTHS:
Although I am not an expert in this field, I find this work quite interesting. The authors provide new material and support their statements with proofs.

WEAKNESSES:
The paper is not tested in any way on a numerical experiment. I am convinced that a paper presented at this type of conference should be both motivated by a real-world application and tested numerically, e.g., on a near-real-world formulation of the problem.

**After a rebuttal process**, the authors agreed with this weakness and promised to add the experiments to the final version of the paper.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide a lower bound for sampling from heavy tailed distributions under the Gaussian oracle of order $O(\\textup{poly}(1/\\varepsilon))$. They then propose an alternative proximal sampling algorithm using the $\\alpha$-stable oracle that achieves a convergence rate of $O(\\log(1/\\varepsilon))$ for heavy-tailed distributions satisfying a fractional Poincare inequality. They then provide a practical implementation of the stable proximal sampler, and lower bounds on its convergence rate.\n\nSTRENGTHS:\n- This work presents a very nice combination of results showing a separation in the performance of stable and Gaussian proximal samplers. The combination of lower and upper bounds separating the two methods makes the work a particularly interesting contribution.\n\n- The addition of a practical implementation of the stable proximal sampler is nice to have, demonstrating that it is viable in practice.\n\n- The work is generally clearly presented and the authors are clear about their contributions.\n\n- Overall, I consider this to be a very sound piece of theoretical work.\n\nWEAKNESSES:\nI have no major concerns about this paper. The presentation is somewhat dense in places, though this is mostly just a consequence of it being a very technical paper and not a flaw as such. If the authors want to make the claim that practicioners should use the stable proximal sampler in applied settings, then they may want to provide empirical evidence of its performance compared to the Gaussian proximal sampler. However, I understand that this is not the main purpose of this theoretical paper.\n\nQUESTIONS:\nI have no clarifications to request.\n\nLIMITATIONS:\nThe authors provide an adequate discussion of the limitations of their methods in the final section, and I foresee no additional negative impacts of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper studies the complexity of sampling heavy-tailed distributions. It provides lower bounds on the complexity of Gaussian-based samplers for a class of heavy-tailed targets. Then, the paper constructs proximal samplers based on stable oracles, which improve the sampling complexity.

STRENGTHS:
* This paper is well-written. The background of sampling and the research problems regarding sampling complexity are clearly introduced. The contributions of the lower bound on Gaussian-based samplers for heavy-tailed targets and the improved complexity using stable oracles are clearly presented.
* The paper is technically sound. The definitions and assumptions are discussed clearly, and the theoretical results are supported by proof sketches.

WEAKNESSES:
The contribution of the paper could be improved with empirical experiments to evaluate the sampling algorithms and their complexity.

QUESTIONS:
* Is there any intuition that a Gaussian-based sampler has lower accuracy for heavy-tailed targets than for non-heavy-tailed targets?
* How would a Gaussian-based sampler compare with a stable oracle for not heavy-tailed targets?

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13