Lower Bounds on Metropolized Sampling Methods for Well-Conditioned Distributions

1 Introduction . Sampling from a continuous distribution in high dimensions is a fundamental problem in algorithm design . As sampling serves as a key subroutine in a variety of tasks in machine learning [ AdFDJ03 ] , statistical methods [ RC99 ] , and scientific computing [ Liu01 ] , it is an important undertaking to understand the complexity of sampling from families of distributions arising in applications . The more restricted problem of sampling from a particular distribution family we call “ wellconditioned ” has garnered a substantial amount of recent research effort from the algorithmic learning and statistics communities . This specific family is interesting for a number of reasons . First , it is practically relevant : Bayesian methods have found increasing use in machine learning applications [ Bar12 ] , and many distributions in these settings are well-conditioned , such as multivariate Gaussians , mixture models with small separation , and densities arising from Bayesian logistic regression with a Gaussian prior [ DCWY18 ] . Moreover , for several widely-used sampler implementations in popular packages [ Aba16 , CGH+17 ] , such as the Metropolis-adjusted Langevin algorithm ( MALA ) and Hamiltonian Monte Carlo ( HMC ) , the target having a small condition number is in some sense a minimal assumption for known provable guarantees ( discussed more thoroughly in Section 1.2 ) . Finally , the highly-documented success of first-order ( gradient ) methods in optimization [ Bec17 ] , which are particularly favorable in the well-conditioned setting , has driven interest in connections between optimization and sampling . Exploring this connection has been highly fruitful : since seminal work of [ JKO98 ] , demonstrating that the Langevin dynamics which MALA and HMC discretize has an interpretation as gradient descent on density space , a flurry of work [ Dal17 , CCBJ18 , DCWY18 , DR18 , DM19 , DMM19 , CDWY19 , CV19 , SL19 , MMW+19 , LST20a , LST20b , CLA+20 ] has ob- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tained improved upper bounds for the mixing of various Langevin discretizations for well-conditioned sampling . Many of these works draw inspiration from first-order optimization . On the other hand , demonstrating lower bounds on the complexity of sampling tasks ( in the wellconditioned regime or otherwise ) has proven remarkably challenging . To our knowledge , there are few unconditional lower bounds for sampling tasks . This is in stark contrast to the theory of optimization , where there are matching upper and lower bounds for a variety of problems and query models , such as convex optimization under first-order oracle access [ Nes03 ] . This gap in the algorithmic theory of sampling is our primary motivation : we aim to answer the following more restricted question . What is the complexity of the popular sampling methods , MALA and HMC , for sampling well-conditioned distributions ? The problem we study is still less general than unconditional query lower bounds for sampling , in that our lower bounds are algorithm-specific ; we characterize the performance of particular sampling algorithms on a distribution family . However , we believe asking this question , and developing an understanding of it , is an important first step towards a theory of complexity for sampling . On the one hand , algorithm-specific lower bounds highlight their weaknesses , pinpointing bottlenecks in attaining faster rates ; this is useful for algorithm designers , as it clarifies what key barriers must be overcome . On the other hand , the hard instances which arise in designing lower bounds may have important structural properties , paving the way to stronger and more general bounds . For these reasons , we focus on characterizing the complexity of MALA and HMC ( see Section 2 for algorithm definitions ) , which are often the samplers of choice in practice , by lower bounding their performance when they are used to sample from densities proportional to exp ( −f ( x ) ) , where f : Rd → R has a finite condition number κ < ∞.1 We call such a density “ well-conditioned. ” Finally , we explicitly assume throughout that κ = O ( d4 ) , as otherwise in light of our lower bounds the general-purpose logconcave samplers of [ LV07 , JLLV20 , Che21 ] are preferable . 1.1 Our results . Our primary contribution is a nearly-tight characterization of the performance of MALA for sampling from two high-dimensional distribution families without a warm start assumption : well-conditioned Gaussians , and the more general family of well-conditioned densities . We prove the following two lower bounds on MALA ’ s complexity , which is a one-parameter algorithm ( for a given target distribution ) depending only on step size . We fix a scale [ 1 , κ ] on the problem , as otherwise nonscale-invariance can be exploited to give more trivial lower bounds ( cf . Appendix B , supplement ) . Theorem 1 . For every step size , there is a target Gaussian on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of MALA is Ω ( κ √ d√ log d ) . Theorem 2 . For every step size , there is a target density on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of MALA is Ω ( κdlog d ) . To give more context , MALA is a Metropolis-adjusted Markov chain , which performs updates preserving the stationary distribution . It can be derived by applying a filter on the Euler discretization of the Langevin dynamics , a stochastic differential equation with stationary density ∝ exp ( −f ( x ) ) : dxt = −∇f ( xt ) dt+ √ 2dWt , where Wt is Brownian motion . Such Metropolis-adjusted methods typically provide total variation distance guarantees , and attain logarithmic dependence on the target accuracy.2 The mixing of such chains is governed by their relaxation time , also known as the inverse spectral gap . However , in continuous space , it is not always clear how to relate the relaxation time and mixing time , the time it takes to reach total variation 1e to the stationary distribution from a given start ( we 1f has condition number κ if it is L-smooth and µ-strongly convex ( has all second derivatives in the range [ µ , L ] ) , where κ = L µ ; we overload this terminology and say the density itself has condition number κ . 2This contrasts with a different family of unadjusted discretizations , which are analyzed by coupling them with the stochastic differential equation they simulate ( see e.g . [ Dal17 , CCBJ18 ] for examples ) , at the expense of a polynomial dependence on the target accuracy ; we focus on Metropolis-adjusted discretizations in this work . choose 1e to match the literature , but any constant smaller than 1 will do ) . There is extensive research on when these two quantities are relateable [ BGL14 ] , but typically these arguments are tailored to chain-specific properties , causing relaxation time bounds to not be entirely satisfactory in some cases . We complement Theorems 1 and 2 with a mixing time lower bound from an exponentially warm start . Theorem 3 . For every step size , there is a target density on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that MALA initialized at an exp ( d ) -warm start requires Ω ( κd log2 d ) iterations to reach e−1 total variation distance to the stationary distribution . We remark that Theorem 3 is the first mixing time lower bound for discretizations of the Langevin dynamics we are aware of : other related bounds have primarily been on relaxation times [ CV19 , LST20a , CLA+20 ] . It is unknown how to obtain a starting distribution for a general κ-conditioned distribution with warmness better than κd ( obtained byN ( x∗ , 1LI ) where L is the smoothness and x ∗ is the mode ) .3 A line of work [ DCWY18 , CDWY19 , LST20a ] analyzed the performance of MALA under this start , culminating in a mixing time of Õ ( κd ) , where Õ hides logarithmic factors in κ , d , and the target accuracy . On the other hand , a recent work [ CLA+20 ] demonstrated that MALA obtains a mixing time scaling as Õ ( poly ( κ ) √ d ) , initialized at a polynomially warm start,4 and further showed that such a mixing time is tight ( in its dependence on d ) . They posed as an open question whether it was possible to obtain Õ ( poly ( κ ) d1−Ω ( 1 ) ) mixing from an explicit start . We address this question via Theorem 3 , showing the Õ ( κd ) rate of [ LST20a ] for MALA applied to a κ-conditioned density is nearly tight . To prove Theorems 1-3 , in each case we exhibit an exp ( −d ) -sized set according to the stationary measure where either the chain can not move in poly ( d ) steps , or must choose a very small step size . Beyond exhibiting a mixing bound , this demonstrates the subexponential warmness assumption in [ CLA+20 ] is necessary for their improved bound . To our knowledge , this is the first nearly-tight characterization of a specific sampler ’ s performance , and improves bounds of [ CLA+20 , LST20a ] . It also implies going beyond Õ ( κd ) mixing for general well-conditioned densities requires subexponential warmness . The lower bound statement of Theorem 3 is warmness-sensitive , and is of the following ( somewhat non-standard ) form : for β = exp ( d ) , we provide a lower bound on the quantity inf algorithm parameters sup starts of warmness≤β densities in target family mixing time of algorithm . In other words , we are allowed to choose both the hard density and starting distribution adaptively based on the algorithm parameters ( in the case of MALA , our choices are in response to the step size ) . We note that this type of lower bound is compatible with standard conductance-based upper bound analyses , which typically only depend on the starting distribution through the warmness parameter . We further study the multi-step generalization of MALA , known as Hamiltonian Monte Carlo with a leapfrog integrator ( which we refer to as HMC ) . In addition to a step size η , HMC is parameterized by a number of steps per iteration K , making K gradient queries in every step to perform a discretization of the Langevin dynamics , before applying a Metropolis filter . Multi-step HMC has the potential of attaining overall runtime gains by effectively taking longer steps . It was recently shown [ CDWY19 ] that under higher derivative bounds , balancing η and K more carefully depending on problem parameters could break the apparent κd barrier of MALA , even from an exponentially warm start . It is natural to ask if there is a stopping point for improving HMC . We demonstrate that HMC can not obtain a better relaxation time than Õ ( κ √ dK−1 ) for any K , even when the target is a Gaussian . Since every HMC step requires K gradients , this suggests Ω̃ ( κ √ d ) queries are necessary . Theorem 4 . For every step size and count , there is a target Gaussian on Rd whose negative logdensity always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of HMC is Ω ( κ √ d K √ log d ) . In Appendix B of the supplement , we also lower bound how much increasing K can improve HMC in the in-between range κ √ d to κd . In particular , we demonstrate that if K ≤ dc for some constant c , then K-step HMC can only improve the relaxation time of Theorem 4 by roughly K2 , showing that 3The warmness of a distribution is the worst-case ratio between measures it and the stationary assign to a set . 4As discussed , it is currently unknown how to obtain such a warm start generically . to truly go beyond a κd relaxation time by a dΩ ( 1 ) factor , the step size must scale polynomially with the dimension . Our mixing lower bound technique in Theorem 3 does not directly extend to give a complementary lower bound for Theorem 4 for all K , but we defer this to interesting future work .

Summary: This paper proves lower bounds on the performance of two important sampling algorithms, the Metropolis-adjusted Langevin algorithm (MALA) and Hamiltonian Monte Carlo (HMC), for strongly log-concave and log-smooth target distributions. The paper first proves upper bounds on the spectral gap for these algorithms, assuming that the target is either Gaussian or more generally log-concave. These results are then used to prove lower bounds on the mixing time of MALA for exponential warm starts, which are perhaps the clearest and most compelling results obtained. Therefore, I will summarize this result in more detail.  Suppose that the target distribution is strongly log-concave and log-smooth in dimension d and with condition number κ. Previous algorithmic works have shown that MALA mixes after dκ steps from an exponentially warm start (which can be obtained by initializing at a Gaussian with appropriate variance). On the other hand, the recent work by Chewi et al. shows that MALA mixes after poly(κ) d^{½} steps under a polynomially warm start (and it is unknown how to achieve such a warm start in practice). This current paper sheds light on this situation by constructing a hard target distribution such that when initialized at a particular exp(d) warm start, MALA requires κd steps to mix.