Lower Bounds on Metropolized Sampling Methods for Well-Conditioned Distributions

1 Introduction . Sampling from a continuous distribution in high dimensions is a fundamental problem in algorithm design . As sampling serves as a key subroutine in a variety of tasks in machine learning [ AdFDJ03 ] , statistical methods [ RC99 ] , and scientific computing [ Liu01 ] , it is an important undertaking to understand the complexity of sampling from families of distributions arising in applications . The more restricted problem of sampling from a particular distribution family we call “ wellconditioned ” has garnered a substantial amount of recent research effort from the algorithmic learning and statistics communities . This specific family is interesting for a number of reasons . First , it is practically relevant : Bayesian methods have found increasing use in machine learning applications [ Bar12 ] , and many distributions in these settings are well-conditioned , such as multivariate Gaussians , mixture models with small separation , and densities arising from Bayesian logistic regression with a Gaussian prior [ DCWY18 ] . Moreover , for several widely-used sampler implementations in popular packages [ Aba16 , CGH+17 ] , such as the Metropolis-adjusted Langevin algorithm ( MALA ) and Hamiltonian Monte Carlo ( HMC ) , the target having a small condition number is in some sense a minimal assumption for known provable guarantees ( discussed more thoroughly in Section 1.2 ) . Finally , the highly-documented success of first-order ( gradient ) methods in optimization [ Bec17 ] , which are particularly favorable in the well-conditioned setting , has driven interest in connections between optimization and sampling . Exploring this connection has been highly fruitful : since seminal work of [ JKO98 ] , demonstrating that the Langevin dynamics which MALA and HMC discretize has an interpretation as gradient descent on density space , a flurry of work [ Dal17 , CCBJ18 , DCWY18 , DR18 , DM19 , DMM19 , CDWY19 , CV19 , SL19 , MMW+19 , LST20a , LST20b , CLA+20 ] has ob- 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . tained improved upper bounds for the mixing of various Langevin discretizations for well-conditioned sampling . Many of these works draw inspiration from first-order optimization . On the other hand , demonstrating lower bounds on the complexity of sampling tasks ( in the wellconditioned regime or otherwise ) has proven remarkably challenging . To our knowledge , there are few unconditional lower bounds for sampling tasks . This is in stark contrast to the theory of optimization , where there are matching upper and lower bounds for a variety of problems and query models , such as convex optimization under first-order oracle access [ Nes03 ] . This gap in the algorithmic theory of sampling is our primary motivation : we aim to answer the following more restricted question . What is the complexity of the popular sampling methods , MALA and HMC , for sampling well-conditioned distributions ? The problem we study is still less general than unconditional query lower bounds for sampling , in that our lower bounds are algorithm-specific ; we characterize the performance of particular sampling algorithms on a distribution family . However , we believe asking this question , and developing an understanding of it , is an important first step towards a theory of complexity for sampling . On the one hand , algorithm-specific lower bounds highlight their weaknesses , pinpointing bottlenecks in attaining faster rates ; this is useful for algorithm designers , as it clarifies what key barriers must be overcome . On the other hand , the hard instances which arise in designing lower bounds may have important structural properties , paving the way to stronger and more general bounds . For these reasons , we focus on characterizing the complexity of MALA and HMC ( see Section 2 for algorithm definitions ) , which are often the samplers of choice in practice , by lower bounding their performance when they are used to sample from densities proportional to exp ( −f ( x ) ) , where f : Rd → R has a finite condition number κ < ∞.1 We call such a density “ well-conditioned. ” Finally , we explicitly assume throughout that κ = O ( d4 ) , as otherwise in light of our lower bounds the general-purpose logconcave samplers of [ LV07 , JLLV20 , Che21 ] are preferable . 1.1 Our results . Our primary contribution is a nearly-tight characterization of the performance of MALA for sampling from two high-dimensional distribution families without a warm start assumption : well-conditioned Gaussians , and the more general family of well-conditioned densities . We prove the following two lower bounds on MALA ’ s complexity , which is a one-parameter algorithm ( for a given target distribution ) depending only on step size . We fix a scale [ 1 , κ ] on the problem , as otherwise nonscale-invariance can be exploited to give more trivial lower bounds ( cf . Appendix B , supplement ) . Theorem 1 . For every step size , there is a target Gaussian on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of MALA is Ω ( κ √ d√ log d ) . Theorem 2 . For every step size , there is a target density on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of MALA is Ω ( κdlog d ) . To give more context , MALA is a Metropolis-adjusted Markov chain , which performs updates preserving the stationary distribution . It can be derived by applying a filter on the Euler discretization of the Langevin dynamics , a stochastic differential equation with stationary density ∝ exp ( −f ( x ) ) : dxt = −∇f ( xt ) dt+ √ 2dWt , where Wt is Brownian motion . Such Metropolis-adjusted methods typically provide total variation distance guarantees , and attain logarithmic dependence on the target accuracy.2 The mixing of such chains is governed by their relaxation time , also known as the inverse spectral gap . However , in continuous space , it is not always clear how to relate the relaxation time and mixing time , the time it takes to reach total variation 1e to the stationary distribution from a given start ( we 1f has condition number κ if it is L-smooth and µ-strongly convex ( has all second derivatives in the range [ µ , L ] ) , where κ = L µ ; we overload this terminology and say the density itself has condition number κ . 2This contrasts with a different family of unadjusted discretizations , which are analyzed by coupling them with the stochastic differential equation they simulate ( see e.g . [ Dal17 , CCBJ18 ] for examples ) , at the expense of a polynomial dependence on the target accuracy ; we focus on Metropolis-adjusted discretizations in this work . choose 1e to match the literature , but any constant smaller than 1 will do ) . There is extensive research on when these two quantities are relateable [ BGL14 ] , but typically these arguments are tailored to chain-specific properties , causing relaxation time bounds to not be entirely satisfactory in some cases . We complement Theorems 1 and 2 with a mixing time lower bound from an exponentially warm start . Theorem 3 . For every step size , there is a target density on Rd whose negative log-density always has Hessian eigenvalues in [ 1 , κ ] , such that MALA initialized at an exp ( d ) -warm start requires Ω ( κd log2 d ) iterations to reach e−1 total variation distance to the stationary distribution . We remark that Theorem 3 is the first mixing time lower bound for discretizations of the Langevin dynamics we are aware of : other related bounds have primarily been on relaxation times [ CV19 , LST20a , CLA+20 ] . It is unknown how to obtain a starting distribution for a general κ-conditioned distribution with warmness better than κd ( obtained byN ( x∗ , 1LI ) where L is the smoothness and x ∗ is the mode ) .3 A line of work [ DCWY18 , CDWY19 , LST20a ] analyzed the performance of MALA under this start , culminating in a mixing time of Õ ( κd ) , where Õ hides logarithmic factors in κ , d , and the target accuracy . On the other hand , a recent work [ CLA+20 ] demonstrated that MALA obtains a mixing time scaling as Õ ( poly ( κ ) √ d ) , initialized at a polynomially warm start,4 and further showed that such a mixing time is tight ( in its dependence on d ) . They posed as an open question whether it was possible to obtain Õ ( poly ( κ ) d1−Ω ( 1 ) ) mixing from an explicit start . We address this question via Theorem 3 , showing the Õ ( κd ) rate of [ LST20a ] for MALA applied to a κ-conditioned density is nearly tight . To prove Theorems 1-3 , in each case we exhibit an exp ( −d ) -sized set according to the stationary measure where either the chain can not move in poly ( d ) steps , or must choose a very small step size . Beyond exhibiting a mixing bound , this demonstrates the subexponential warmness assumption in [ CLA+20 ] is necessary for their improved bound . To our knowledge , this is the first nearly-tight characterization of a specific sampler ’ s performance , and improves bounds of [ CLA+20 , LST20a ] . It also implies going beyond Õ ( κd ) mixing for general well-conditioned densities requires subexponential warmness . The lower bound statement of Theorem 3 is warmness-sensitive , and is of the following ( somewhat non-standard ) form : for β = exp ( d ) , we provide a lower bound on the quantity inf algorithm parameters sup starts of warmness≤β densities in target family mixing time of algorithm . In other words , we are allowed to choose both the hard density and starting distribution adaptively based on the algorithm parameters ( in the case of MALA , our choices are in response to the step size ) . We note that this type of lower bound is compatible with standard conductance-based upper bound analyses , which typically only depend on the starting distribution through the warmness parameter . We further study the multi-step generalization of MALA , known as Hamiltonian Monte Carlo with a leapfrog integrator ( which we refer to as HMC ) . In addition to a step size η , HMC is parameterized by a number of steps per iteration K , making K gradient queries in every step to perform a discretization of the Langevin dynamics , before applying a Metropolis filter . Multi-step HMC has the potential of attaining overall runtime gains by effectively taking longer steps . It was recently shown [ CDWY19 ] that under higher derivative bounds , balancing η and K more carefully depending on problem parameters could break the apparent κd barrier of MALA , even from an exponentially warm start . It is natural to ask if there is a stopping point for improving HMC . We demonstrate that HMC can not obtain a better relaxation time than Õ ( κ √ dK−1 ) for any K , even when the target is a Gaussian . Since every HMC step requires K gradients , this suggests Ω̃ ( κ √ d ) queries are necessary . Theorem 4 . For every step size and count , there is a target Gaussian on Rd whose negative logdensity always has Hessian eigenvalues in [ 1 , κ ] , such that the relaxation time of HMC is Ω ( κ √ d K √ log d ) . In Appendix B of the supplement , we also lower bound how much increasing K can improve HMC in the in-between range κ √ d to κd . In particular , we demonstrate that if K ≤ dc for some constant c , then K-step HMC can only improve the relaxation time of Theorem 4 by roughly K2 , showing that 3The warmness of a distribution is the worst-case ratio between measures it and the stationary assign to a set . 4As discussed , it is currently unknown how to obtain such a warm start generically . to truly go beyond a κd relaxation time by a dΩ ( 1 ) factor , the step size must scale polynomially with the dimension . Our mixing lower bound technique in Theorem 3 does not directly extend to give a complementary lower bound for Theorem 4 for all K , but we defer this to interesting future work .

Summary: The paper proves a mixing time lower bound for Metropolized Langevin (MALA) on distributions with smooth and strongly convex potentials, starting at an exponential warm start. The lower bound matches the best known upper bounds. It also proves a mixing time lower bound for HMC on the class of Gaussians with condition number kappa from an exponential warm start.