Title: Sampling from Structured Log-Concave Distributions via a Soft-Threshold Dikin Walk
Venue: NeurIPS 2023
Accepted: yes
Keywords: Logconcave sampling

Dikin walk

Markov chain Monte Carlo

interior point methods
Abstract: Given a Lipschitz or smooth convex function $f:K \to \mathbb{R}^d$ for a bounded polytope $K:=${ $\theta \in \mathbb{R}^d: A\theta \leq b$}, where $A\in  \mathbb{R}^{m\times d}$ and $b \in \mathbb{R}^m$, we consider the problem of sampling from the log-concave distribution $\pi(\theta) \propto e^{-f(\theta)}$ constrained to $K$. Interest in this problem derives from its applications to  Bayesian inference and differential privacy. We present  a generalization of the Dikin walk  to this setting that requires at most $O((md + d L^2 R^2) \times md^{\omega-1} \log(\frac{w}{\delta}))$ arithmetic operations to sample from $\pi$ within error $\delta>0$ in the total variation distance from a $w$-warm start. Here $L$ is the Lipschitz constant of $f$, $K$ is contained in a ball of radius $R$ and contains a ball of smaller radius $r$, and $\omega \approx 2.37$ is the matrix-multiplication constant. This improves on the running time of prior works for a range of structured settings important for the aforementioned inference and privacy applications. Technically, we depart from previous Dikin walks by adding a soft-threshold regularizer derived from the Lipschitz or smoothness properties of $f$  to a barrier function for $K$ that allows our version of the Dikin walk to propose updates  that have a high Metropolis acceptance ratio for $f$, while at the same time remaining inside the polytope $K$.

Sampling from Structured Log-Concave Distributions via a Soft-Threshold Dikin Walk

Abstract

Given a Lipschitz or smooth convex function f : K → R d for a bounded polytope K := { θ ∈ R d : Aθ ≤ b } , where A ∈ R m × d and b ∈ R m , we consider the problem of sampling from the log-concave distribution π ( θ ) ∝ e -f ( θ ) constrained to K . Interest in this problem derives from its applications to Bayesian inference and differential privacy. We present a generalization of the Dikin walk to this setting that requires at most O (( md + dL 2 R 2 ) × md ω -1 log( w δ )) arithmetic operations to sample from π within error δ > 0 in the total variation distance from a w -warm start. Here L is the Lipschitz constant of f , K is contained in a ball of radius R and contains a ball of smaller radius r , and ω ≈ 2 . 37 is the matrix-multiplication constant. This improves on the running time of prior works for a range of structured settings important for the aforementioned inference and privacy applications. Technically, we depart from previous Dikin walks by adding a soft-threshold regularizer derived from the Lipschitz or smoothness properties of f to a barrier function for K that allows our version of the Dikin walk to propose updates that have a high Metropolis acceptance ratio for f , while at the same time remaining inside the polytope K .

1 Introduction

We consider the problem of sampling from a log-concave distribution supported on a polytope: Given a polytope K := { θ ∈ R d : Aθ ≤ b } , where A ∈ R m × d and b ∈ R m , and a convex function f : K → R , output a sample θ ∈ K from the distribution π ( θ ) ∝ e -f ( θ ) . Our interest in this problem derives from its applications to Bayesian inference and differentially private optimization. In Bayesian inference, the ability to sample from π ( θ ) ∝ e -f ( θ ) allows one to compute Bayesian confidence intervals and other statistics for the Bayesian posterior distribution of many machine learning models (see e.g. [17, 20, 6]). In differentially private optimization, sampling from the 'exponential mechanism' [36] allows one to get optimal utility bounds for the problem of minimizing f under ε -differential privacy [2].

The instances of the polytope-constrained sampling problem that arise in these applications are more general than the two well-studied special cases- the uniform density case ( f ≡ 0) and the unconstrained case ( K = R d ). However, they still have more structure than the case of a general log-concave function supported on an arbitrary convex body. For instance, in Bayesian Lasso logistic regression, f ( θ ) = ∑ n i =1 /lscript ( θ ; x i ), where /lscript is the logistic loss and x i are datapoints with ‖ x i ‖ 2 ≤ 1, and K = { θ ∈ R d : ‖ θ ‖ 1 ≤ O (1) } ; see [44, 43, 26, 47]. Since the logistic function is both O (1)-smooth and O (1)-Lipschitz, f is both O ( n )-Lipschitz and O ( n )-smooth, and K is defined by O ( d ) inequalities and contained in a ball of radius O (1).

To obtain an ε -differentially private mechanism for the Lasso logistic regression problem, using the exponential mechanism, the goal is to sample from exp( -ε R ∑ n i =1 /lscript ( θ ; x i )), where /lscript is the logistic loss and K is contained in a ball of radius R . Thus, the log-density is both

β -smooth and L -Lipschitz for β = L = nε / R = O ( d ) if n = d and ε < 1, since R = O (1). Another example is a result of [28] that reduces the problem of ε -differentially private lowrank approximation of a symmetric p × p matrix to a constrained sampling problem where f is linear (and thus 0-smooth) of dimension d = p 2 and K is the Gelfand-Tsetlin polytope (a generalization of the probability simplex). Here K has d inequalities and diameter O ( √ d ).

Importantly, when sampling from the exponential mechanism in privacy applications, sampling with total variation (TV) bounds-the case that has received the most attention-is insufficient to guarantee ε -differential privacy, the strongest notion of differential privacy; see [14]. Instead, one requires bounds in the stronger infinity-distance metric d ∞ ( ν, π ) := sup θ ∈ K | log( ν ( θ ) / π ( θ ) ) | . A recent work [35] showed how to convert samples within O ( δ )-TV distance from continuous log-Lipschitz densities π , into samples with O ( ε )-infinity-distance bounds, but it requires the TV distance δ to be very small -roughly δ = O ( εe -d -LR ), raising the question of designing Markov chains whose runtime bounds also have a low-order dependence on log 1 / δ . Thus, for the aforementioned applications to Bayesian inference and privacy which give rise to structured instances of sampling from log-concave densities over polytopes, it is desirable to design sampling algorithms that have a low-order polynomial dependence not only on the parameters d, L, R, β , but also on log 1 / δ .

Main related works. A line of work has developed algorithms for sampling in the general setting when K is an arbitrary convex body given by a membership oracle [16, 1, 15, 33, 34]. [33] use the 'hit-and-run' framework to give an algorithm to sample from a log-concave distribution π ∝ e -f on a convex body K which also contains a ball of radius r with TV-error δ > 0 in O ( d 2 ( R / r ) 2 log 2 ( wdR / ( rδ ) ) log 3 ( w / δ ))( T f + ˆ T K )) arithmetic operations from a w -warm start (their Theorem 1.1) and O ( d 3 ( R / r ) 2 ) log 5 ( dR 2 / ( δr ) ( T f + ˆ T K )) arithmetic operations from a cold start (Corollary 1.2). Here T f is the time required to evaluate f and ˆ T K is the time for a membership oracle query. Here, a distribution ν is w -warm for w ≥ 1 w.r.t. the stationary distribution π if sup z ∈ K ν ( z ) / π ( z ) ≤ w .

[38] give a 'Dikin-walk'-based algorithm to sample from any log-concave π ∝ e -f on K where f is L -Lipschitz or β -smooth. The Dikin walk Markov chain was introduced in [24] in the special case where f ≡ 0 (see also [37]). Their runtime is O (( d 5 + d 3 L 2 R 2 ) log( w / δ )( T f + md ω -1 )) arithmetic operations, where ω = 2 . 37 · · · is the matrix-multiplication constant. From a cold start, their runtime is O (( d 5 + d 3 L 2 R 2 ) log( 1 / δ ) ( d log( R / r ) + M +log( 1 / δ ))( T f + md ω -1 ), where M := log(max θ ∈ K e f ( a ) -f ( θ ) ). Their bounds when f is β -smooth are the same, but with each L 2 term replaced with β . We discuss additional related work in Appendix A.

Our contributions. We present a Markov chain sampling algorithm that generates samples from an L -log-Lipschitz or β -log-smooth log-concave distribution π ∝ e -f on an R -bounded polytope K given by m inequalities, with an error bounded in the TV distance (Algorithm 1 and Theorem 2.1). Our algorithm requires O (( md + dL 2 R 2 ) log( w / δ ))( T f + md ω -1 ) arithmetic operations to sample with TV error O ( δ ) from π ∝ e -f when f is L -Lipschitz, and O (( md + dβR 2 ) log( w / δ ))( T f + md ω -1 ) arithmetic operations in the setting where f is β -smooth, where T f is the number of arithmetic operations to compute the value of f and md ω -1 is the number of arithmetic operations to compute the Hessian of the log-barrier of K .

In comparison to [33], we improve dependence of the running time on the parameters R / r and log 1 / δ . In comparison to [38], we improve the dependence on d while retaining the same dependence on 1 / δ . Our result directly implies faster runtimes with improved dependence on the dimension d for structured inference problems such as Bayesian Lasso logistic regression where e.g. R / r = Ω( √ d ) (Corollary B.1). Moreover, plugging our algorithm into the TV-toinfinity distance bound converter of [35], we obtain an algorithm to sample from a logconcave density constrained to a polytope with error bounded by infinity distance that improves upon prior work of [2]; see Corollary C.1 and the subsequent discussion. Corollary C.1, along with the exponential mechanism [36], allows us to obtain faster runtime bounds with improved

dependence on d for applications to differentially private empirical risk minimization [2] and matrix approximation [28]; see Corollary D.1 and the subsequent discussion.

Technically, our algorithm is a Markov chain inspired by the Dikin walk [24, 37], whose steps are determined by a barrier function that generalizes the log-barrier function by adding a 'soft-threshold' /lscript 2 -norm regularizer. The regularized barrier allows our Markov chain to take larger steps, while still retaining a high acceptance probability on Lipschitz or smooth log-densities f -allowing our Markov chain to sample from these distributions with a faster runtime. A key technical step in obtaining our results is to show that our self-concordant barrier function is the limit of an infinite sequence of log-barrier functions for K 'padded' with additional redundant inequalities. This allows us to leverage well-known properties of the log-barrier to bound the acceptance probability and mixing time of our Markov chain.

While /lscript 2 regularization is optimal for classes of functions f which are Lipschitz or smooth in the /lscript 2 -norm, for other classes of functions (e.g., Lipschitz in the /lscript 1 -norm), /lscript 2 regularization may not be optimal. Moreover, it remains open to obtain runtime bounds for the Dikin walk that do not require f to be Lipschitz or smooth, and/or depend polynomially on log R . This leads to the question of whether one can design other tractable self-concordant barriers to obtain further runtime improvements for sampling log-concave distributions on a polytope; we discuss this in Appendix F.

2 Results

Our main result (Theorem 2.1 and Algorithm 1) is a Markov chain algorithm that generates TV-error bounded samples from L -log-Lipschitz or β -log-smooth log-concave distributions on a polytope. As explained later in this section, Theorem 2.1 often results in the fastest known algorithm for some of the applications to Bayesian inference and differentially private optimization mentioned in the introduction.

Notation. In the following, f is L -Lipschitz or β -smooth for some L, β > 0. T f denotes the number of arithmetic operations to evaluate f , T K is the number of arithmetic operations to compute the Hessian of the log-barrier function, ˆ T K the operations for a membership oracle query, and ˜ T K the operations to compute a projection oracle for K . When K is a polytope K = { θ ∈ R d : Aθ ≤ b } given by A ∈ R m × d and b ∈ R m , one has T K = O ( md ω -1 ), ˆ T K = O ( md ), and ˜ T K = O ( md ω -1 ). For every j ∈ { 1 , . . . , m } , we denote the j 'th row of A by a j and the j 'th entry of b by b j . When comparing runtimes, we often assume for simplicity that T f = Θ( d 2 ) unless otherwise stated, which is the case, e.g., in logistic regression with n = Θ( d ) datapoints. For any two distributions µ, ν on R d , we denote their total variation distance by ‖ µ -ν ‖ TV := sup S ⊆ R d | µ ( S ) -ν ( S ) | . For θ ∈ R d , t > 0, denote the ball of radius t at θ by B ( θ, t ) := { z ∈ R d : ‖ z -θ ‖ 2 ≤ t } where ‖ · ‖ 2 is the Euclidean norm. Denote the interior of any S ⊆ R d by Int( S ) := { θ ∈ S : B ( θ, t ) ⊆ S for some t > 0 } . α > 0 is a step-size hyperparameter shared by our algorithm and the original Dikin walk of [24, 37], γ > 0 is the hyperparameter in [38], and η > 0 a hyperparameter for our algorithm's regularizer.

Theorem 2.1 (Sampling with TV bounds via a soft-threshold Dikin Walk)

There exists an algorithm (Algorithm 1) which, given δ, R > 0 and either L > 0 or β > 0 , A ∈ R m × d , b ∈ R m that define a polytope K := { θ ∈ R d : Aθ ≤ b } such that K is contained in a ball of radius R and has nonempty interior, an oracle for the value of a convex function f : K → R d , where f is either L -Lipschitz or β -smooth, and an initial point sampled from a distribution supported on K which is w -warm with respect to π ∝ e -f for some w > 0 , outputs a point from a distribution µ where ‖ µ -π ‖ TV ≤ δ . Moreover, this algorithm takes O (( md + dL 2 R 2 ) log( w / δ )) × ( T f + T K ) arithmetic operations in the setting where f is L -Lipschitz, or O (( md + dβR 2 ) log( w / δ )) × ( T f + T K ) arithmetic operations when f is β -smooth, where T f is the number of operations to evaluate f and T K = O ( md ω -1 ) .

Theorem 2.1 improves on the previous bound of ([33]; Theorem 1.1) of O ( d 2 ( R / r ) 2 log 2 ( wdR / ( rδ ) ) log 3 ( w / δ ))( T f + ˆ T K ) arithmetic operations, for sampling from ∝ e -f

1

with TV error O ( δ ) from a w -warm start, by a factor of min( d 2 -ω ( R / r ) 2 , d 3 -ω / ( r 2 L 2 ) ) log 4 ( w / δ ) when f is L -Lipschitz on a polytope K defined by m = O ( d ) inequalities (see Table 1). Thus, ˆ T K = O ( md ) and if, e.g., we also have LR = O ( √ d ) and R / r = √ d , the improvement is d 3 -ω arithmetic operations. 1 When f is β -smooth, the improvement is min( d 2 -ω ( R / r ) 2 , d 3 -ω / ( r 2 β )) log 4 ( w / δ ) arithmetic operations.

When a warm start is not provided, Algorithm 1 takes at most O (( md + dL 2 R 2 ) × ( d log( R / r )+ M +log( 1 / δ )))( T f + T k ) arithmetic operations when f is L -Lipschitz (or O (( md + dβR 2 ) × ( d log( R / r ) + M +log( 1 / δ )))( T f + T k ) arithmetic operations when f is β -smooth), since an e d log( R / r )+ M -warm start can be obtained by sampling uniformly from the ball B ( a, r ) ⊆ K , where M = log(max θ ∈ K e f ( a ) -f ( θ ) ) ≤ LR . In comparison, the work of [33] for the hit-andrun algorithm gives a bound of O ( d 3 ( R / r ) 2 ) log 5 ( dR 2 / ( δr ) )( T f + ˆ T K ) arithmetic operations to sample with TV error O ( δ ) without a warm start. Thus, when we are not given a warm start and π ∝ e -f is constrained to a polytope K defined by m = O ( d ) inequalities, Theorem 2.1 improves on the bounds of [33] for the hit-and-run algorithm by a factor of min( d 2 -ω ( R / r ) 2 , d 3 -ω / ( rL 2 ) ) log 4 ( 1 / δ ) arithmetic operations in the setting when f is L -Lipschitz, and min( d 2 -ω ( R / r ) 2 , d 3 -ω / ( rβ ) ) log 4 ( 1 / δ ) when f is β -smooth. On the other hand, we note that the results of [33] apply more generally when π is a log-concave distribution on a convex body, while our bounds for the soft-threshold Dikin walk (Theorem 2.1) apply to the setting where π is a log-Lipschitz (or log-smooth) log-concave distribution on a polytope. In the example of Bayesian Lasso logistic regression, our algorithm takes O ( d 3+ ω log( d / δ )) arithmetic operations without a warm start since f is both β -smooth and L -Lipschitz, with β = L = n = m = d , and R = 1 and r = 1 / √ d (Corollary B.1 in Appendix B); this improves by a factor of d 3 -ω log 4 ( d / δ ) arithmetic operations on the bound of d 6 log 5 ( d / δ ) arithmetic operations for the hit-and-run algorithm of [33]. This is because, while it only takes ˆ T K = O ( d ) arithmetic operations to establish membership in the /lscript 1 ball, computing the function f in logistic regression requires T f = nd = d 2 arithmetic operations.

We also compare to the bound of O (( d 5 + d 3 L 2 R 2 ) log( w / δ ))( T f + T k ) arithmetic operations to sample from an L -log-Lipschitz density on a polytope defined by m inequalities with TV error O ( δ ) from a w -warm start of [38]. 2 We note that while this bound has a larger dependence on d (and L ) than the bounds for the hit-and-run algorithm, the dependence on log 1 / δ is much smaller- by a factor of log 4 1 / δ - which can allow for faster runtimes when the goal is to sample with infinity-distance error. We explain this in detail below. Theorem 2.1 improves on the bound of [38] bound by a factor of d 2 arithmetic operations if m = O ( d ),

√

and d 3 if we also have that LR = O ( d ). When f is instead β -smooth their bound is O (( d 5 + d 3 βR 2 ) log( w / δ ))( T f + T k ), and our improvement on this bound is d 2 arithmetic operations if m = O ( d ), and d 3 if we also have that βR 2 = O ( d ).

We also note that while many works, e.g. [24, 37, 27, 8], give faster bounds for the Dikin walk and its variants than the bounds in [38], these only apply in the special case when π is the uniform distribution on K . The proof of Theorem 2.1 appears in Appendix E. We give an overview of the main ideas in the proof of Theorem 2.1 in Section 3. In Appendix F, we give an axiomatic approach to arrive at our barrier function and discuss possible extensions.

Infinity-distance sampling. In applications of sampling to differentially private optimization [36, 23, 2, 18, 28], bounds in the total variation (TV) distance are insufficient to guarantee 'pure' ε -differential privacy, and one instead requires bounds in the infinitydistance d ∞ ( ν, π ) := sup θ ∈ K | log( ν ( θ ) / π ( θ ) ) | ; see e.g. [14]. [35] give an algorithm that converts samples from TV bounds to those bounded in infinity-distance. Namely, given ε > 0 and a sample from a distribution µ within TV distance δ ≤ O ( ε ( R ( d log( R / r )+ LR ) 2 / εr ) -d e -LR ) of π , this post-processing algorithm outputs a sample within infinity-distance O ( ε ) from π . Plugging the TV bounds from our Theorem 2.1 into their Theorem 2.2 gives a faster algorithm to sample from a log-concave and log-Lipschitz (or log-smooth) distribution constrained to a polytope K , with O ( ε ) error in d ∞ (Corollary C.1 in Appendix C). In particular, Corollary C.1 gives a bound of O (( md + dL 2 R 2 ) × [ LR + d log( Rd + LRd / rε )])( T f + T K ) arithmetic operations to sample within O ( ε ) error in d ∞ from a log-concave distribution ∝ e -f constrained to a polytope K from a cold start when f is L -Lipschitz (when f is also β -smooth, the bound is O (( md + dβR 2 ) × [ LR + d log( Rd + LRd / rε )])( T f + T K )). This improves on the bound of O (( m 2 d 3 + m 2 dL 2 R 2 ) × [ LR + d log( Rd + LRd / rε )])( T f + T K ) arithmetic operations in Theorem 2.1 of [35] by a factor of d 3 . Moreover, it further improves on the bound of O (( md 9 + md 5 L 4 R 4 )( 1 / ε 2 ) × polylog( 1 / ε , 1 / r , R, L, d ))( T f + ˜ T K ) operations implied by [2]. Corollary C.1 improves on this bound by a factor of d 9 -ω / ε 2 when, e.g, each function evaluation takes T f = O ( d 2 ) operations and m = O ( d ) as may be the case in privacy applications.

Differentially private optimization. A randomized mechanism h : D n → R is ε -differentially private ( ε -DP) if for any datasets x, x ′ ∈ D which differ by a single datapoint, and any S ⊆ R , we have P ( h ( x ) ∈ S ) ≤ e ε P ( h ( x ′ ) ∈ S ); see [14]. ε -differential privacy is the strongest notion of differential privacy, holds several advantages over weaker notions of differential privacy (see Remark 2.2), and has been widely studied in the literature [14, 25, 2].

In the application of the exponential mechanism to ε -DP low-rank approximation of a p × p symmetric matrix M [28] (see also [19]), one wishes to sample within infinity distance O ( ε ) from a log-linear distribution ∝ e -f on the Gelfand-Tsetlin polytope K ⊆ R d (which generalizes the probability simplex), where d = p 2 , and where K has m = d inequalities with diameter R = O ( √ d ). In this application, the log-linear density f is (trivially) 0-smooth and d 2 σ 1 -Lipschitz, where σ 1 := ‖ M ‖ 2 is the spectral norm of M . Thus, when applied to the mechanism of [28], our algorithm takes d 4 . 5+ ω σ 1 log( 1 / ε ) arithmetic operations. This improves by a factor of d 3 on the bound of O ( d 7 . 5+ ω σ 1 ) arithmetic operations implied by [35, 38], and improves by a factor of d 11 . 5 σ 3 1 / ε 2 on the bound of O ( d 16 σ 4 1 / ε 2 ) implied by [2].

Consider the problem of finding an (approximate) minimum ˆ θ of an empirical risk function f : K × D n → R under the constraint that ˆ θ is ε -differentially private, where f ( θ, x ) := ∑ n i =1 /lscript i ( θ, x i ). We assume that the /lscript i ( · , x ) are ˆ L -Lipschitz for all x ∈ D n , i ∈ N , for some given ˆ L > 0. In this setting, [2] show the minimum ERM utility bound under the constraint that ˆ θ is pure ε -DP, E ˆ θ [ f ( ˆ θ, x )] -min θ ∈ K f ( θ, x ) = Θ( d ˆ LR / ε ), is achieved if one samples ˆ θ from the exponential mechanism π ∝ exp( -ε 2 ˆ LR f ) with infinity-distance error at most O ( ε ). Plugging Corollary C.1 into the exponential mechanism, we obtain a faster algorithm for a pure ε -DP mechanism which achieves the minimum expected risk (Corollary D.1). Specifically, the runtime bound implied by Corollary D.1 is O (( md + dn 2 ε 2 ) × ( εn + d log( nRd / ( rε ) )( T f + T K )) arithmetic operations if each /lscript i is ˆ L -Lipschitz (or O (( md + dn ˆ β ˆ L Rε ) × ( εn + d log( nRd / ( rε ) )( T f + T K )) if f is also β -Lipschitz). This improves upon the bound of

O (( d 10 ( 1 / ε 2 ) + ε 2 n 4 d 6 ) × polylog( nRd / rε )))( T f + ˜ T K ) arithmetic operations in [2] by a factor of max( d 10 -ω / ( ε 2 m ) , nd 5 ( 1 / ε )), when the /lscript i are ˆ L -Lipschitz on a polytope K and f can be evaluated in T f = O ( nd ) operations. And it improves by a factor of (at least) md on the bound of O (( m 2 d 3 + m 2 dn 2 ε 2 )( εn + d )log 2 ( nRd / ( rε ) )) × md ω -1 ) operations obtained in [35].

For instance, when applying the exponential mechanism to Lasso logistic regression, if e.g. n = d and ε < 1, our algorithm requires O ( d 3+ ω ) arithmetic operations, improving by d 3 on the bound of d 6+ ω operations implied by [35] and by roughly d 9 -ω on the bound of O ( d 12 ) operations implied by [2]. In another example, when training a support vector machine with hinge loss and Lasso constraints under ε -DP, our algorithm requires O ( d 4+ ω ) arithmetic operations, improving by a factor of d 2 on the bound of d 6+ ω operations implied by [35] and by d 8 -ω on the bound of O ( d 12 ) implied by [2]. See Appendix D for details.

Remark 2.2 (Weaker notions of differential privacy) ε -DP holds several practical advantages over weaker notions of differential privacy (DP), including ( ε, δ ) -DP- a notion of differential privacy where the privacy of the mechanism is allowed to fail with probability O ( δ ) . E.g., when group privacy- privacy of subsets of k individuals- must be preserved, any pure ε -DP mechanism is also kε -DP. In contrast, ( ε, δ ) -DP only implies ( ε, ke ( k -1) ε δ ) -DP for subsets of k individuals- the failure probability grows exponentially with k .

3 Overview of proof of Theorem 2.1

Suppose we are given any polytope K = { θ ∈ R d : Aθ ≤ b } defined by m inequalities, and a convex f : K → R d which is L -Lipschitz (or β -smooth) and given by an oracle which returns f ( θ ) at any θ ∈ K . Our goal is to sample from π ∝ e -f on K within any TV error δ > 0, in a number of arithmetic operations and oracle calls that has a dependence on the dimension d that is a lower-order polynomial than currently available bounds for sampling from log-Lipschitz (or log-smooth) log-concave distributions, and is logarithmic in 1 / δ .

Extending the Dikin walk to sample from log-concave distributions on K . As a first step, we begin by attempting to generate samples from π via the (Gaussian) Dikin walk, by extending the standard analysis given in e.g. [24, 37] for the special case when π is uniform on K to the more general case where π is a log-Lipschitz log-concave on K .

In the special case where π is the uniform distribution on K , from any point θ in the interior of K , the Dikin walk proposes updates z = θ + √ αH -1 ( θ ) ξ where ξ ∼ N (0 , I d ) and H ( θ ) = ∇ 2 ϕ ( θ ) is the Hessian of the log-barrier function ϕ ( θ ) = -∑ m j =1 log( b j -a /latticetop j θ ) for K = { θ ∈ R d : Aθ ≤ b } , and α > 0 is a scalar hyperparameter. To ensure that the stationary distribution of the Dikin walk is the uniform distribution on K , if a proposed update falls in the interior of K , it is accepted with probability

determined by the metropolis rule; otherwise, it is rejected. The use of the log-barrier is to ensure that the steps proposed by the Dikin walk remain inside the polytope K w.h.p.

The hyperparameter α is chosen as large as possible while still ensuring the proposed steps remain in K and are accepted w.h.p. On the one hand, since the covariance matrix αH -1 ( θ ) of the proposed updates is proportional to α , larger choices of α allow the walk to propose larger update steps. On the other hand, if α is too large, the proposed steps may fall outside the polytope and be rejected w.h.p. To see how to choose α , note that for any θ ∈ Int( K ), the Dikin ellipsoid D θ = { w : ( w -θ ) /latticetop H -1 ( θ )( w -θ ) ≤ 1 } is contained in K . Thus, standard Gaussian concentration inequalities which guarantee ‖ ξ ‖ 2 = O ( √ d ) w.h.p. imply θ + √ αH -1 ( θ ) ξ is in K w.h.p. if α ≤ O ( 1 / d ). Moreover, using properties of log-barrier functions, one can show the term det( H ( z )) / det( H ( θ )) in the acceptance ratio is also Ω(1) for α = O ( 1 / d ), as is done in [24, 37]. To see why, Lemma 4.3 of [45] implies that, if H ( θ ) is the Hessian of a log-barrier function for K , its log-determinant V ( θ ) = log(det( H ( θ ))) satisfies

Thus, if α ≤ 1 / d , the proposed update z = θ + √ αH -1 ( θ ) ξ has variance Ω(1) in the direction ∇ V ( θ ), and (by Gaussian concentration), ( θ -z ) /latticetop ∇ V ( θ ) ≤ O (1) w.h.p. This implies V ( z ) -V ( θ ) = log det( H ( z )) -log det( H ( θ )) = Ω(1), and hence det( H ( z )) / det( H ( θ )) = Ω(1).

In [38], the Dikin walk is applied to the more general problem of sampling from a L -logLipschitz (or β -log-smooth) log-concave π ∝ e -f on K (the problem of interest in this paper). To guarantee the walk has the correct stationary distribution π , the Metropolis acceptance probability of the proposed updates z = θ + √ γH -1 ( θ ) ξ , where γ is a hyperparameter, gains an additional factor e -f ( z ) / e -f ( θ ) . To ensure this acceptance probability remains Ω(1), they modify the scalar step size γ such that w.h.p. the walk takes steps where f changes by O (1). To see how to choose γ , note that since f is L -Lipschitz, e f ( z ) -f ( θ ) = Ω(1) if the Euclidean distance ‖ z -θ ‖ 2 is O ( 1 / L ). This can be shown to occur w.h.p. if γ = O ( 1 / ( LR ) 2 ), since the fact that the Dikin ellipsoid is in K ⊆ B (0 , R ) implies that the eigenvalues of H ( θ ) are all ≤ R 2 and hence the variance γv /latticetop H -1 ( θ ) v of the proposed step is ≤ O ( 1 / ( dL 2 ) ) in any given direction v ∈ R d (where v is a unit vector). Thus, it is sufficient for them to choose γ = min( 1 / d , 1 / ( LR ) 2 ) to ensure the proposed step remains in K and is accepted w.h.p.

On the one hand, to ensure the Markov chain proposes steps that change f by an amount at most O (1) for any L -Lipschitz f , it is necessary and sufficient to ensure that from any point θ ∈ Int( K ), the Markov chain makes updates which fall w.h.p. inside a Euclidean ball B ( θ, 1 / L ) of radius 1 / L centered at θ . This is because the Lipschitz condition on f : ‖ f ( θ ) -f ( z ) ‖ 2 ≤ L ‖ θ -z ‖ 2 for all θ, z ∈ K holds w.r.t. the Euclidean norm ‖ · ‖ 2 . On the other hand, to ensure that the Markov chain remains inside the polytope K , it is sufficient to propose steps that lie inside the Dikin ellipsoid D θ = { w : ( w -θ ) /latticetop H -1 ( θ )( w -θ ) ≤ 1 } centered at θ . Roughly, the scalar step size γ is chosen such that this ellipsoid is contained inside the Euclidean ball B ( θ, 1 / L ), as this guarantees that w.h.p. the steps proposed by the walk will both remain in K and will also not change the value of f by more than O (1).

However, at many points θ the Dikin ellipsoid D θ is such that the ratio of the largest to smallest eigenvalues of H -1 ( θ ) may be very large (this ratio can grow arbitrarily large as θ approaches a face of the polytope). Thus, roughly speaking, modifying the covariance matrix of the Dikin walk by a scalar constant ( γH -1 ( θ )) can cause the Dikin walk to propose steps whose variance in some directions is much smaller than is required for either of the two goals: staying inside K and staying inside the ball B ( θ, 1 / L ) defined by the Lipschitz condition on f . This suggests that modifying the log-barrier function for K by a scalar multiple may not be the most efficient way of extending the Dikin walk to the problem of sampling from a general L -log-Lipschitz (or β -log-smooth) log-concave distribution on K , and that one may be able to obtain faster runtimes by making other modifications to the barrier function.

Asoft-threshold regularized Dikin walk. Before we introduce our soft-threshold Dikin walk, we first note that even in the special case where π is the uniform distribution on K , the analysis in [38] does not recover the bounds given in [24, 37] for this special case, as [38] use a different runtime analysis geared to time-varying distributions studied in that paper. Namely, [38] imply a bound of O ( m 2 d 3 log( ω / δ )) steps to sample from a uniform distribution on K , while [24, 37] show a bound of O ( md log( ω / δ )). For this reason, we first extend the analysis of the Gaussian Dikin walk given in [24, 37] for the special case of uniform π , to the more general problem of sampling from an L -log-Lipschitz or β -log-smooth log-concave density. The analysis in [24, 37] uses the cross-ratio distance metric. More specifically, if for any distinct u, v ∈ Int( K ) we let p, q be the endpoints of the chord in K passing through u and v such that the four points lie in the order p, u, v, q , the cross-ratio distance is

One can show that for any u, v ∈ Int( K ), σ 2 ( u, v ) ≥ ( 1 / ( mγ -1 ) ) ‖ u -v ‖ 2 γ -1 H ( u ) (see e.g. [24, 37]). Thus, as the usual Dikin walk takes steps that have roughly identity covariance matrix I d with respect to the local norm ‖ u ‖ γ -1 H ( θ ) := √ u /latticetop γ -1 H ( θ ) u , for γ = min( 1 / d , ( 1 / ( LR ) 2 )), the bound we would obtain on the number of steps until the Dikin walk is within TV error δ from π is O (∆ -1 log( ω / δ )) = O (( md + mL 2 R 2 ) log( w / δ )) steps from a w -warm start.

To obtain even faster bounds, we would ideally like to allow the Dikin walk to take larger steps by choosing a larger γ , closer to the value 1 / d that is sufficient to ensure an Ω(1) acceptance probability in the special case when π is uniform. Unfortunately, if e.g. LR ≥ d , reducing γ from 1 / d to ( 1 / ( LR ) 2 ) may be necessary to ensure the variance of the Dikin walk steps is O ( 1 / ( dL 2 ) ) in every direction, and hence that the acceptance probability is Ω(1).

To get around this problem, we introduce a new variant of the Dikin walk Markov chain for sampling from any L -log-Lipschitz (or β -log-smooth) log-concave distributions on a polytope K , which generalizes the Dikin walk introduced in [24] for sampling from π in the special case when π is the uniform distribution on K . The main difference between our Dikin walk and the usual Dikin walk of [24] (and of [38]) is that our Dikin walk regularizes the Hessian α -1 H ( θ ) of the log-barrier for K by adding a 'soft-threshold' regularization term η -1 I d proportional to the identity matrix, where η is a hyperparameter and α is the same hyperparameter appearing in the original Dikin walk of [24]. Since the log-barrier Hessian α -1 H ( θ ) and regularization term η -1 I d have different scalar hyperparameters α, η , we can set α and η independently from each other: roughly, α -1 is chosen to be the largest value such that the Dikin ellipsoid defined by the matrix α -1 H ( θ ) remains inside K , while η -1 is independently chosen to be the largest value such that, with high probability, the steps proposed by our Markov chain remain inside the ball B ( θ, 1 / L ) defined by the Lipschitz condition on f . Roughly, the addition of the soft-threshold regularization term to the logbarrier Hessian allows us to reduce the variance of the proposed steps of the Dikin walk only in those directions where a choice of α = 1 / d would cause the variance to be greater than 1 / ( dL 2 ) while leaving the variance in other directions unchanged.

More specifically, the steps proposed by our soft-threshold Dikin walk are Gaussian with mean 0 and covariance matrix Φ -1 ( θ ) := ( α -1 H ( θ ) + η -1 I d ) -1 . The addition of the softthreshold regularization term η -1 I d allows us to ensure that the largest eigenvalues of the covariance matrix Φ -1 ( θ ) are ≤ O ( 1 / ( dL 2 ) ), without reducing (by more than a constant factor) the eigenvalues which were already ≤ O ( 1 / ( dL 2 ) ). This allows our Dikin walk to take larger steps, while still ensuring these steps are accepted w.h.p. by the Metropolis accept/reject rule for f . Taking larger steps allows our Dikin walk to converge more quickly to the target distribution π ( θ ) ∝ e -f ( θ ) on K .

The (inverse) covariance matrix Φ( θ ) of the steps proposed by our Dikin walk is the Hessian of the function ψ ( θ ) = α -1 ϕ ( θ ) + η -1 ‖ θ ‖ 2 where ϕ ( θ ) = -∑ m j =1 log( b j -a /latticetop j θ ) is the logbarrier for K . The modified function ψ ( θ ) can be seen to also be a self-concordant barrier for K . In the special case where π is the uniform distribution, L = 0 and η -1 = 0, and our 'soft-threshold' Dikin walk recovers the original walk of [24]. Thus, our walk generalizes the original Dikin walk to the problem of sampling from a general L -log-Lipschitz (or β -logsmooth) log-concave distribution on a polytope.

Bounding the number of Markov chain steps. Setting η = 1 / ( dL 2 ) ensures the variance v /latticetop αH -1 ( θ ) v in any given unit-vector direction v of the proposed update z -θ of our Markov chain is at most O ( 1 / ( dL 2 ) ), and hence the term e f ( z ) -f ( θ ) in the Metropolis acceptance rule is Ω(1) with high probability (Lemma E.5). Moreover, we also show that, if we choose α = 1 / d , the other terms in the Metropolis acceptance rule are also Ω(1) (Lemmas E.8, E.9). While the proofs of these lemmas follow roughly the same outline as in the special case of the original Dikin walk where π is uniform (e.g., [24, 37]), our bound on the determinantal term detΦ( z ) / detΦ( θ ) must deal with additional challenges, which we discuss in the next subsection.

To bound the number of steps required by our Markov chain, we first bound the cross-ratio distance σ ( u, v ) between any u, v ∈ Int( K ) by the local norm ‖ u -v ‖ Φ( u ) (Lemma E.2):

Using (3) together with the isoperimetric inequality for the cross-ratio distance (Theorem 2.2 of [32]), we show that, if the acceptance probability of our Markov chain is Ω(1) at each step, then the number of steps for our Markov chain to obtain a sample within a TV distance of δ from π is O (∆ -1 log( ω / δ )) = O (( md + dL 2 R 2 ) log( w / δ )) from an w -warm start,

where ∆ = Ω( 1 / (2 mα -1 +2 η -1 R 2 ) ). In particular, if m = O ( d ) and LR > d , this improves on the bound we would get for the basic Dikin walk by a factor of d .

Bounding the determinantal term in the acceptance probability. For our mixing time bound to hold, we still need to show that det(Φ( z )) / det(Φ( θ )) is Ω(1) w.h.p. We would ideally like to follow the general approach previous works [24, 37] use to show the term det( H ( z )) / det( H ( θ )) in the basic Dikin walk is Ω(1) w.h.p., which relies on the property of logbarrier functions in Inequality (1). Unfortunately, as Φ( θ ) is not the Hessian of a log-barrier function for any system of inequalities defining K , we cannot directly apply (1) to Φ( θ ).

To get around this problem, we show that, while Φ( θ ) is not the Hessian of a log-barrier function, it is in fact the limit of a sequence of matrices H i ( θ ), i ∈ N , where each matrix H i ( θ ) in this sequence is the Hessian of a (different) log-barrier function for K . Specifically, for every j ∈ N , we consider the matrices A j = [ A /latticetop , I d , . . . , I d ] /latticetop where A is concatenated with ( m j -1) / d copies of the identity matrix I d , and m j = m + /floorleft αη -1 j 2 /floorright d . And we consider the vectors b j = ( b /latticetop , j 1 /latticetop , . . . , j 1 /latticetop ) /latticetop , where b is concatenated with ( m j -1) / d copies of the vector j 1 , where 1 = (1 , . . . , 1) /latticetop ∈ R d is the all-ones vector. Then, for large enough j , K = { θ ∈ R d : A j θ ≤ b j } , and the Hessian of the corresponding log-barrier functions is

Using this fact (4), we show that, for every θ ∈ int( K ), every z ∈ 1 2 D θ , and every sequence { z j } ∞ j =1 ⊆ 1 2 D θ such that z j → z , we have (Lemma E.7),

Moreover, since each H j is the Hessian of a log-barrier function for K , we have that (1) does hold for H j and hence (from the work of [24, 37]) that det( H j ( z j )) / det( H j ( θ )) = Ω(1) w.h.p. for all j ∈ N , if we set z j = θ + α 1 / 2 H -1 / 2 j ( θ ) ξ , ξ ∼ N (0 , I d ), and choose α = 1 / d . Thus, (5) implies that det(Φ( z )) / det(Φ( θ )) = Ω(1) w.h.p. as well (Lemma E.9), and hence the acceptance probability of the soft-threshold Dikin walk is Ω(1) at each step.

Bounding the number of arithmetic operations. Since the acceptance probability is Ω(1), from the above discussion, the number of steps for our Markov chain to obtain a sample within TV distance δ > 0 from π is O (( md + L 2 R 2 ) log( w / δ )) from a w -warm start.

Each time our Markov chain proposes a step θ +Φ -1 / 2 ( θ ) ξ , it must first sample a Gaussian vector ξ ∼ N (0 , I d ) which takes O ( d ) arithmetic operations. It must then compute the log-barrier Hessian H ( θ ), and invert the matrix Φ( θ ) = α -1 H ( θ ) + η -1 I d .

Since H ( θ ) = C ( θ ) C ( θ ) /latticetop , where C ( θ ) is a d × m matrix with columns c j ( θ ) = a j / ( a /latticetop j θ -b j ) for all j ∈ [ m ], we can compute H ( θ ) in md ω -1 arithmetic operations. And since Φ( θ ) is a d × d matrix, computing Φ -1 / 2 ( θ ) can be accomplished in d ω arithmetic operations by computing the singular value decomposition (SVD) of Φ. Next, we compute the acceptance probability min ( e -f ( z ) √ det(Φ( z )) e -f ( θ ) √ det(Φ( θ )) e ‖ z -θ ‖ 2 Φ( θ ) -‖ θ -z ‖ 2 Φ( z ) , 1 ) . The determinants can be computed in O ( d ω )

arithmetic operations via the SVD. Evaluating f ( z ) , f ( θ ) takes two calls to the oracle for f .

Thus, from a w -warm start, the soft-threshold Dikin walk takes at most O (( md + dL 2 R 2 ) × log( w / δ )) Markov chain steps to obtain a sample from π with total variation error δ > 0, where each step takes O ( md ω -1 ) arithmetic operations, and one function evaluation.

Remark 3.1 (Optimality of /lscript 2 -regularization) For the class of functions considered in our paper, we conjecture that an /lscript 2 -regularizer that does not depend on θ is optimal. This is because we consider the class of functions which are L -Lipschitz or β -smooth with respect to the /lscript 2 -norm, and our bound on L or β does not depend on θ . Moreover, we only have access to the function θ through an oracle which returns the value of f at any given point θ but does not tell us how f changes at nearby points.

Algorithm 1: Soft-threshold Dikin walk

In Theorem 2.1, we set the step size hyperparameters α = 1 / (10 5 d ) and η = 1 / (10 4 dL 2 ) if f is L -Lipschitz, and the number of steps to be T = 10 9 ( 2 mα -1 + η -1 R 2 ) × log( w / δ ) . When f is β -smooth (but not necessarily Lipschitz), we instead set α = 1 / (10 5 d ) and η = 1 / (10 4 dβ ) .

In many applications, a bound on L or β can be calculated analytically, allowing one to set α, η as above. This includes, e.g., applications to training Bayesian or differentially private logistic regression models (or other generalized linear models such as support vector machines). When a bound on L or β is not known, one can in practice set α, η by hand such that the average acceptance probability is ≥ Ω(1).

4 Conclusions, limitations, and future work

Our result improves on the runtime bounds of a line of previous work, for the problem of sampling from several classes of log-Lipschitz or log-smooth log-concave distributions on a polytope (see Table 1). To the best of our knowledge, this is the first result to introduce regularized barrier functions that simultaneously take into account the geometry of both the constraint polytope and the Lipschitz or smoothness property of a target logconcave function. These barrier functions may be of independent interest for sampling or optimization.

On the other hand, we note that [27] give an implementation of the Dikin walk in the special case where f is constant, where the (average) cost of computing the Hessian matrix of the log-barrier of the polytope K := { θ ∈ R d : Aθ ≤ b } at each step of the walk is improved to roughly O ( d 2 +nnz( A )) arithmetic operations, where nnz( A ) denotes the number of non-zero entries of A . Whether this improvement in the per-step computation time can be achieved for the problem of computing the regularized barrier functions used in our algorithm, in the more general setting where f is L -Lipschitz or β -smooth, is an interesting open problem.

Moreover, we note that our bounds are polynomial in L or β , yet there are algorithms for sampling from log-concave distributions ∝ e -f which do not assume f is L -Lipschitz or β -smooth. Thus, another interesting open problem is whether one can obtain runtime bounds for a version of the Dikin walk which do not require f to be Lipschitz or smooth.

Our results have applications to Bayesian inference and differentially private optimization. Bayesian inference can lead to algorithms with better generalization properties and quantification of uncertainty, and differential privacy guarantees are important to protecting the privacy of individuals in medical and other sensitive datasets. Thus, we believe our results will have positive societal impacts, and do not anticipate any negative impacts to society.

Acknowledgments. NV was supported in part by an NSF CCF-2112665 award. OM was supported in part by an NSF CCF-2104528 award and a Google Research Scholar award.

References

David Applegate and Ravi Kannan. Sampling and integration of near log-concave functions. In Proceedings of the twenty-third annual ACM symposium on Theory of computing , pages 156-163, 1991.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pages 464-473. IEEE, 2014.

Nicolas Brosse, Alain Durmus, Éric Moulines, and Marcelo Pereyra. Sampling from a log-concave distribution with compact support with proximal Langevin Monte Carlo. In Conference on Learning Theory , pages 319-342. PMLR, 2017.

Sébastien Bubeck and Ronen Eldan. The entropic barrier: a simple and optimal universal self-concordant barrier. In Conference on Learning Theory , pages 279-279. PMLR, 2015.

Sebastien Bubeck, Ronen Eldan, and Joseph Lehec. Finite-time analysis of projected Langevin Monte Carlo. In Advances in Neural Information Processing Systems , pages 1243-1251. Citeseer, 2015.

Bradley P Carlin and Siddhartha Chib. Bayesian model choice via markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Methodological) , 57(3):473-484, 1995.

Apostolos Chalkis, Vissarion Fisikopoulos, Marios Papachristou, and Elias Tsigaridas. Truncated log-concave sampling for convex bodies with reflective Hamiltonian Monte Carlo. ACM Transactions on Mathematical Software , 2023.

Yuansi Chen, Raaz Dwivedi, Martin J Wainwright, and Bin Yu. Vaidya walk: A sampling algorithm based on the volumetric barrier. In 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 1220-1227. IEEE, 2017.

Arnak S Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic Langevin diffusions. Bernoulli , 26(3):1956-1988, 2020.

John Duchi. Derivations for linear algebra and optimization. Berkeley, California , 3:2325-5870, 2007.

Alain Durmus, Szymon Majewski, and Blazej Miasojedow. Analysis of Langevin Monte Carlo via convex optimization. J. Mach. Learn. Res. , 20:73-1, 2019.

Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin algorithm. Annals of Applied Probability , 27(3):1551-1587, 2017.

Raaz Dwivedi, Yuansi Chen, Martin J Wainwright, and Bin Yu. Log-concave sampling: Metropolis-hastings algorithms are fast! In Conference on Learning Theory , pages 793-797. PMLR, 2018.

Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3-4):211-407, 2014.

Alan Frieze and Ravi Kannan. Log-Sobolev inequalities and sampling from log-concave distributions. The Annals of Applied Probability , 9(1):14-26, 1999.

Alan Frieze, Ravi Kannan, and Nick Polson. Sampling from log-concave distributions. The Annals of Applied Probability , pages 812-837, 1994.

Dani Gamerman and Hedibert F Lopes. Markov chain Monte Carlo: stochastic simulation for Bayesian inference . CRC press, 2006.

Arun Ganesh and Kunal Talwar. Faster differentially private samplers via Rényi divergence analysis of discretized Langevin MCMC. Advances in Neural Information Processing Systems , 33, 2020.

Rong Ge, Holden Lee, Jianfeng Lu, and Andrej Risteski. Efficient sampling from the Bingham distribution. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato, editors, Algorithmic Learning Theory, 16-19 March 2021, Virtual Conference, Worldwide , volume 132 of Proceedings of Machine Learning Research , pages 673-685. PMLR, 2021.

Andrew Gelman and Donald B Rubin. Markov chain Monte Carlo methods in biostatistics. Statistical methods in medical research , 5(4):339-355, 1996.

Sivakanth Gopi, Yin Tat Lee, and Daogao Liu. Private convex optimization via exponential mechanism. In Po-Ling Loh and Maxim Raginsky, editors, Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 1948-1989. PMLR, 02-05 Jul 2022.

David Lee Hanson and Farroll Tim Wright. A bound on tail probabilities for quadratic forms in independent random variables. The Annals of Mathematical Statistics , 42(3):1079-1083, 1971.

Moritz Hardt and Kunal Talwar. On the geometry of differential privacy. In Proceedings of the forty-second ACM symposium on Theory of computing , pages 705-714, 2010.

Ravi Kannan and Hariharan Narayanan. Random walks on polytopes and an affine interior point method for linear programming. Mathematics of Operations Research , 37(1):1-20, 2012.

Michael Kapralov and Kunal Talwar. On differentially private low rank approximation. In Proceedings of the twenty-fourth annual ACM-SIAM symposium on Discrete algorithms , pages 1395-1414. SIAM, 2013.

Sun Mi Kim, Yongdai Kim, Kuhwan Jeong, Heeyeong Jeong, and Jiyoung Kim. Logistic lasso regression for the diagnosis of breast cancer using clinical demographic data and the bi-rads lexicon for ultrasonography. Ultrasonography , 37(1):36, 2018.

Aditi Laddha, Yin Tat Lee, and Santosh Vempala. Strong self-concordance and sampling. In Proceedings of the 52nd annual ACM SIGACT symposium on theory of computing , pages 1212-1222, 2020.

Jonathan Leake, Colin S McSwiggen, and Nisheeth K Vishnoi. A polynomial-time algorithm and applications for matrix sampling from Harish-Chandra-Itzykson-Zuber densities. In ACM symposium on theory of computing STOC , 2021.

Yin Tat Lee and Aaron Sidford. Solving linear programs with Sqrt(rank) linear system solves. arXiv preprint arXiv:1910.08033 , 2019.

Yin Tat Lee and Santosh S Vempala. Convergence rate of Riemannian Hamiltonian Monte Carlo and faster polytope volume computation. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1115-1121, 2018.

László Lovász and Miklós Simonovits. Random walks in a convex body and an improved volume algorithm. Random structures & algorithms , 4(4):359-412, 1993.

László Lovász and Santosh Vempala. Hit-and-run is fast and fun. preprint, Microsoft Research , 2003.

László Lovász and Santosh Vempala. Fast algorithms for logconcave functions: Sampling, rounding, integration and optimization. In Foundations of Computer Science, 2006. FOCS'06. 47th Annual IEEE Symposium on , pages 57-68. IEEE, 2006.

László Lovász and Santosh Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358, 2007.

Oren Mangoubi and Nisheeth K. Vishnoi. Sampling from log-concave distributions with infinity-distance guarantees. In Advances in Neural Information Processing Systems , 2022.

Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE, 2007.

Hariharan Narayanan. Randomized interior point methods for sampling and optimization. The Annals of Applied Probability , 26(1):597-641, 2016.

Hariharan Narayanan and Alexander Rakhlin. Efficient sampling from time-varying log-concave distributions. The Journal of Machine Learning Research , 18(1):4017-4045, 2017.

Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994.

Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub-gaussian concentration. Electronic Communications in Probability , 8(82):1-9, 2013.

Sushant Sachdeva and Nisheeth K Vishnoi. The mixing time of the Dikin walk in a polytope-a simple proof. Operations Research Letters , 44(5):630-634, 2016.

Adil Salim and Peter Richtárik. Primal dual interpretation of the proximal stochastic gradient langevin algorithm. Advances in Neural Information Processing Systems , 33:3786-3796, 2020.

Mervyn J Silvapulle and Pranab Kumar Sen. Constrained statistical inference: Order, inequality, and shape constraints . John Wiley & Sons, 2011.

Guo-Liang Tian, Man-Lai Tang, Hong-Bin Fang, and Ming Tan. Efficient methods for estimating constrained parameters with applications to regularized (lasso) logistic regression. Computational statistics & data analysis , 52(7):3528-3542, 2008.

Pravin M Vaidya and David S Atkinson. A technique for bounding the number of iterations in path following algorithms. In Complexity in Numerical Optimization , pages 462-489. World Scientific, 1993.

Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. In Conference on Learning Theory , pages 2093-3027. PMLR, 2018.

Yuan Zhang, Jonathan N Hofmann, Mark P Purdue, Shili Lin, and Swati Biswas. Logistic bayesian lasso for genetic association analysis of data from complex sampling designs. Journal of Human Genetics , 62(9):819-829, 2017.

Contents

A Additional discussion with related work

The problem of sampling from the uniform distribution when K is a polytope given by a set of inequalities, and the distance to π is measured in total-variation (TV) distance has been widely studied; see, e.g., [24, 37, 30, 8, 27]. Also widely studied is the problem of sampling from an unconstrained log-concave distribution. One direction of study includes works that require the log-density π to be L -log-Lipschitz or β -log-smooth on all R d for some L, β > 0, and give bounds on the distance to π in terms of TV distance [13], Wasserstein distance [12, 9, 42], and Kullback-Leibler (KL) divergence [46, 11].

Further, [5, 3] provide versions of the Langevin dynamics for sampling from a log-concave distribution π ∝ e -f on K where f is L -Lipschitz and β -smooth and K is given by a projection oracle- when, e.g., K is contained in a ball of radius O (1), it contains a ball of radius Ω(1), and L = β = O (1), [5] give a bound of roughly O ( d 9 / δ 22 ) gradient and projection oracle calls to sample from π with TV error δ > 0, while [3] give a bound of roughly O ( d 5 / δ 6 ) gradient and projection oracle calls. [21] provide an algorithm for sampling from distributions ∝ e -f where f is both L -Lipschitz and µ -strongly convex (that is, f ( θ ) is the sum of a convex function ˆ f ( θ ) and the quadratic function ( µ / 2 ) ‖ θ ‖ 2 for some µ > 0) on a convex body K in roughly ˜ O (( L 2 / µ ) log 2 ( d / δ )) membership and function oracle calls (see also [7], who require f to be µ -strongly convex for µ > 0 and β -smooth). However, the algorithms of [21, 7] do not apply to the more general setting where f is convex and L -Lipschitz (or β -smooth), but not necessarily µ -strongly convex for any µ > 0.

Detailed comparison to [38]. Compared to [38], in our version of the Dikin walk, z is sampled from a Gaussian with covariance matrix

where α -1 and η -1 are hyper-parameters chosen to be α -1 ≈ d 2 and η -1 ≈ dL 2 if f is L -Lipschitz (or η -1 ≈ dβ if f is L -smooth). The 'soft-threshold' regularization term η -1 I d prevents the Markov chain from taking steps where the value of f decreases by more than O (1) w.h.p., ensuring that the term e f ( θ ) -f ( z ) in the acceptance probability is Ω(1). The soft-threshold regularizer is chosen to be a multiple of the identity matrix I d since the Lipschitz condition on f is rotationally invariant- it bounds the derivative of f by the same amount L in each direction (the same is true for the second derivative of f if f is β -smooth). This in turn allows our choice of scaling α -1 for the 'Dikin ellipsoid' term H ( z )- which is not in general rotationally invariant, and determined only by the geometry of the polytope rather than the geometry of the function f - to be independent of L (or β ). This is in contrast to the Dikin walk in [38] where the scaling parameter for H ( z ) must depend on L (or β ) to ensure an Ω(1) acceptance probability, which allows our Markov chain to propose steps with a larger variance than the Dikin walk in [38] in directions which are not the largest eigenvector of H -1 ( z ).

The (inverse) covariance matrix α -1 H ( z ) + η -1 I d of our soft-threshold Dikin walk updates is the Hessian of the function ψ ( θ ) = α -1 ϕ ( θ ) + η -1 ‖ θ ‖ 2 2 . This barrier function can be seen to be a Hessian of a self-concordant barrier. On the other hand, it is not the Hessian of a logarithmic-barrier function for any polytope defined by any set of inequalities. This prevents us from directly applying the analysis of the Dikin walk in the special case where f ≡ 0 [37]-which relies on properties of log-barrier functions- to our soft-threshold Dikin walk on Lipschitz or smooth f . To get around this problem, we show that, while ψ ( θ ) is not a log-barrier function of any polytope K , it is the limit of a sequence of log-barrier functions ˆ ψ 1 , ˆ ψ 2 , . . . where ˆ ψ i ( θ ) → ψ ( θ ) uniformly in θ as i → ∞ . See Section 3 for a detailed overview of the proof.

An open problem is to obtain runtime bounds for the Dikin walk that do not require f to be Lipschitz or smooth, and/or that depend polynomially on log R . This leads to the question of whether one can design other tractable self-concordant barriers to obtain further runtime improvements for sampling log-concave distributions on a polytope. We discuss possible extensions in Appendix F.

Explanation of bounds in [38]. [38] consider a version of the Dikin walk for sampling from a log-concave distribution π ∝ e -f on a polytope K contained in a ball of radius R > 0 (and also more generally on any convex body with a self-concordant barrier function). Their version of the Dikin walk has proposed updates z = α 1 2 H -1 2 ( θ ) ξ where ξ ∼ N (0 , I d ), and acceptance probability given by 1 2 × min ( e -f ( z ) √ det( H ( z )) e -f ( θ ) √ det( H ( θ )) , 1 ) . Here H is the hessian of a barrier function for the convex body K , which may be the log-barrier or another barrier function. For a step size of α 1 2 = O (min( 1 d , 1 LR )) (Condition 4 in Section 3.1 of their paper; note that R is considered a constant in [38] and therefore does appear explicitly in their bounds) 3 , they obtain a bound on the conductance φ of their version of the Dikin walk (Lemma 4 of their paper):

where κ is the self-concordance parameter of the barrier function; for the log-barrier κ = m , although there are other barrier functions for which κ = O ( d ). Plugging Inequality (6) into Corollary 1.5 of [31] implies that their Dikin walk can obtain a sample from π with O ( δ ) TV error in roughly φ -2 log( w δ ) = O (( d 5 + d 3 L 2 R 2 ) log( w δ )) steps from a w -warm start.

The number of arithmetic operations per step is md ω -1 and the number of arithmetic operations to obtain a sample with TV error O ( δ ) from π is O (( md 4+ ω + md 2+ ω L 2 R 2 ) log( w δ )). If one also has that K contains a ball of radius r > 0, from a cold start one can find an initial point which is e d log( R r )+ LR -warm by sampling from the uniform distribution on this ball, and hence their bound is O (( md 4+ ω + md 2+ ω L 2 R 2 )( d log( R r ) + LR +log( 1 δ ))).

B Application to Bayesian Lasso logistic regression

The following corollary follows directly from Theorem 2.1.

Corollary B.1 There exists an algorithm which, given δ, c > 0 and a dataset { x i } n i =1 ⊆ R d s.t. ‖ x i ‖ 2 ≤ 1 ∀ i ∈ [ n ] , and letting f ( θ ) = ∑ n i =1 /lscript ( θ /latticetop x i ) where /lscript ( s ) = 1 1+ e -s is the logistic loss, K = { θ ∈ R d : ‖ θ ‖ 1 ≤ c } is the Lasso constraint, and π ∝ e -f has support K , outputs a point from a distribution µ s.t. ‖ µ -π ‖ TV ≤ δ , in O (( d 2 + ndc 2 )( d ω +1 + nd 2 ) log( d δ )) arithmetic operations.

C Infinity-distance sampling

In applications of sampling to differentially private optimization [36, 23, 2, 18, 28], bounds in the total variation (TV) distance are insufficient to guarantee 'pure' differential privacy, and one instead requires bounds in the infinity-distance d ∞ ( ν, π ) := sup θ ∈ K | log( ν ( θ ) / π ( θ ) ) | ; see e.g. [14]. [35] give an algorithm that converts samples from TV bounds to those bounded in infinity-distance. Namely, given ε > 0 and a sample from a distribution µ within TV distance δ ≤ O ( ε ( R ( d log( R / r )+ LR ) 2 / εr ) -d e -LR ) of π , this post-processing algorithm outputs a sample from a distribution ν with infinity distance d ∞ ( ν, π ) ≤ ε from π . Plugging the TV bounds from our Theorem 2.1 into their Theorem 2.2 gives a faster algorithm to sample from a log-concave and log-Lipschitz distribution constrained to a polytope K , with O ( ε ) error in d ∞ . In particular, Corollary C.1 improves the bound of O (( m 2 d 3 + m 2 dL 2 R 2 ) × [ LR + d log( Rd + LRd / ( rε ) )])( T f + T K ) arithmetic operations in their Theorem 2.1. The proof is identical to that of how their Theorem 2.2 implies their Theorem 2.1, and we refer the reader to their paper.

Corollary C.1 (Sampling with infinity-distance guarantees) There exists an algorithm which, given ε, L, r, R > 0 , A ∈ R m × d , b ∈ R m (and possibly β > 0 ), that define a polytope K := { θ ∈ R d : Aθ ≤ b } contained in a ball of radius R , a point

a ∈ R d such that K contains a ball B ( a, r ) of smaller radius r , and an oracle for the value of a convex function f : K → R d , where f is L -Lipschitz (or is both L -Lipschitz and β -smooth), and defining π to be the distribution π ∝ e -f , outputs a point from a distribution ν such that d ∞ ( ν, π ) < ε . Moreover, this algorithm takes O ( ( md + dL 2 R 2 ) × [ LR + d log ( Rd + LRd rε )]) ( T f + T K ) arithmetic operations, where T f is the number of arithmetic operations to evaluate f and T K = O ( md ω -1 ) . If f is also β -Lipschitz, the number of operations is O ( ( md + dβR 2 ) × [ LR + d log ( Rd + LRd rε )]) ( T f + T K ) .

Corollary C.1 further improves the dependence on the dimension d over [2, 35]. Specifically, [2] implies a bound of O (( md 9 + md 5 L 4 R 4 )( 1 / ε 2 ) × polylog( 1 / ε , 1 / r , R, L, d ))( T f + ˜ T K ) arithmetic operations, where T f is the number of operations to compute f and ˜ T K the operations to compute a projection oracle for K . Corollary C.1 improves on this bound by a factor of roughly d 10 -ω / ( ε 2 m ) when, e.g, each function evaluation takes T f = O ( d 2 ) operations. For example, when m = O ( d ), as may be the case in privacy applications, the improvement is d 9 -ω / ε 2 . Moreover, it improves by d 3 on the bound of O (( m 2 d 3 + m 2 d min( L 2 R 2 , βR 2 )) × [ LR + d log( ( Rd + LRd ) / ( rε ) )]( T f + T K ) arithmetic operations obtained in [35].

D Differentially private optimization

A randomized mechanism h : D n → R is said to be ε -differentially private ( ε -DP) if for any datasets x, x ′ ∈ D which differ by a single datapoint, and any S ⊆ R , we have that P ( h ( x ) ∈ S ) ≤ e ε P ( h ( x ′ ) ∈ S ); see [14]. In the application of the exponential mechanism to ε -DP low-rank approximation of a p × p symmetric matrix M [28] (see also [19]), one wishes to sample within infinity distance O ( ε ) from a log-linear distribution ∝ e -f on the Gelfand-Tsetlin polytope K ⊆ R d (which generalizes the probability simplex), where d = p 2 , and where K has m = d inequalities with diameter R = O ( √ d ). In this application, the log-linear density f is (trivially) 0-smooth and d 2 σ 1 -Lipschitz, where σ 1 := ‖ M ‖ 2 is the spectral norm of M . Thus, when applied to the mechanism of [28], our algorithm takes d 4 . 5+ ω σ 1 log( 1 / ε ) arithmetic operations. This improves by a factor of d 3 on the runtime bound of O ( d 7 . 5+ ω σ 1 ) arithmetic operations implied by [35, 38], and improves by a factor of d 11 . 5 σ 3 1 / ε 2 on the bound of O ( d 16 σ 4 1 / ε 2 ) arithmetic operations for the bound in [2].

Consider the problem of finding an (approximate) minimum ˆ θ of an empirical risk function f : K × D n → R under the constraint that ˆ θ is ε -differentially private, where f ( θ, x ) := ∑ n i =1 /lscript i ( θ, x i ). We assume that the /lscript i ( · , x ) are ˆ L -Lipschitz for all x ∈ D n , i ∈ N , for some given ˆ L > 0. In this setting, [2] show the minimum ERM utility bound under the constraint that ˆ θ is pure ε -DP, E ˆ θ [ f ( ˆ θ, x )] -min θ ∈ K f ( θ, x ) = Θ( d ˆ LR / ε ), is achieved if one samples ˆ θ from the exponential mechanism π ∝ exp( -ε 2 ˆ LR f ) with infinity-distance error at most O ( ε ). Plugging Corollary C.1 into the exponential mechanism, we obtain a faster algorithm for a pure ε -DP mechanism which achieves the minimum expected risk (Corollary D.1).

Corollary D.1 (Differentially private empirical risk minimization) There exists an algorithm which, given ε, ˆ L, r, R > 0 , A ∈ R m × d , b ∈ R m (and possibly ˆ β > 0 ) that define a polytope K := { θ ∈ R d : Aθ ≤ b } contained in a ball of radius R and containing a ball of smaller radius r , and an empirical risk function f ( θ, x ) := ∑ n i =1 /lscript i ( θ, x i ) , where each /lscript i : K → R is ˆ L -Lipschitz (and possibly also ˆ β -smooth ), outputs a random point ˆ θ ∈ K which is pure ε -differentially private and satisfies E ˆ θ [ f ( ˆ θ, x )] -min θ ∈ K f ( θ, x ) ≤ O ( d ˆ LR / ε ) . Moreover, this algorithm takes O (( md + dn 2 ε 2 ) × ( εn + d log( nRd / ( rε ) )( T f + T K )) arithmetic operations if each /lscript i is ˆ L -Lipschitz (or O (( md + dn ˆ β ˆ L Rε ) × ( εn + d log( nRd / ( rε ) )( T f + T K )) if f is also β -Lipschitz), where T f is the number of operations to evaluate f and T K = O ( md ω -1 ) .

Corollary D.1 improves on [2, 35]. In particular, it improves upon the bound of O (( d 10 ( 1 / ε 2 )+ ε 2 n 4 d 6 ) × polylog( nRd / rε )))( T f + ˜ T K ) arithmetic operations in [2] by a factor of roughly max( d 10 -ω / ( ε 2 m ) , nd 5 ( 1 / ε )) arithmetic operations, in the setting where the /lscript i are ˆ L -Lipschitz

on a polytope K and f can be evaluated in T f = O ( nd ) operations ( ˜ T K is the number of operations to compute a projection oracle for K ). And it improves by a factor of (at least) md on the bound of O (( m 2 d 3 + m 2 dn 2 ε 2 ) × ( εn + d )log 2 ( nRd / ( rε ) )) × md ω -1 ) arithmetic operations obtained in [35]. The proof is identical to that of how their Theorem 2.2 implies their Corollary 2.4, and we refer the reader to their paper. For instance, when applying the exponential mechanism to the Lasso logistic regression problem, each loss /lscript i is both ˆ β -smooth and ˆ L -Lipschitz for ˆ β = ˆ L = 1 and R = O (1). Thus, if n = d and ε < 1, for this problem our algorithm requires O ( d 3+ ω ) arithmetic operations, which improves by a factor of d 3 on the bound of d 6+ ω arithmetic operations implied by Corollary 2.4 in [35] and by roughly d 9 -ω on the bound of O ( d 12 ) arithmetic operations implied by [2]. In another example, when training a support vector machine model with hinge loss and Lasso constraints under ε -differential privacy, one has that /lscript i is ˆ L -Lipschitz but not smooth, for ˆ L = 1, and R = O (1). Thus, if n = d and ε < 1, our algorithm requires O ( d 4+ ω ) arithmetic operations, which improves by a factor of d 2 on the bound of d 6+ ω operations implied by Corollary 2.4 in [35] and by roughly d 8 -ω on the bound of O ( d 12 ) implied by [2].

E Proof of Theorem 2.1

E.1 Bounding the number of arithmetic operations

In the following, we assume the hyperparameters α, η satisfy α ≤ 1 10 5 d log( 1 ˆ δ ) for some choice of ˆ δ > 0, and either η ≤ 1 10 4 dL 2 (in the setting where f is L -Lipschitz) or η ≤ 1 10 4 dβ (in the setting where f is β -smooth). To obtain our main result (Theorem 2.1), we set ˆ δ = 1 100 . To obtain improved runtime bounds in the setting where A is sparse, we will set ˆ δ = 1 100( md + dL 2 R 2 ) log( w δ ) δ when f is L -Lipschitz, and ˆ δ = 1 100( md + dβR 2 ) log( w δ ) δ when f is β -smooth.

Lemma E.1 Each iteration of Algorithm 1 can be implemented in O ( md ω -1 ) arithmetic operations plus O (1) calls to the oracle for the value of f .

Proof: We go through each step of Algorithm 1 and add up the number of arithmetic operations and oracle calls for each step:

Line 4 samples a d -dimensional Gaussian random vector ξ ∼ N (0 , I d ), which can be performed in O ( d ) arithmetic operations.

At each iteration, Algorithm 1 computes the matrix H ( w ) at w = θ (line 5) and w = z (line 9):

Computing H ( w ) at any w ∈ R d can be accomplished in md ω -1 operations as follows:

Define C ( w ) to be the d × m matrix where each column c j ( w ) = a j a /latticetop j w -b j for all j ∈ [ m ]. Then

Since C ( w ) is a d × m matrix the product C ( w ) C ( w ) /latticetop can be computed in md ω -1 arithmetic operations. Thus, Lines 5 and 9 of Algorithm 1 can each be computed in md ω -1 arithmetic operations.

Since Lines 6 and 10 compute a sum of two d × d matrices, Lines 6 and 10 can each be performed in d 2 arithmetic operations.

Line 7 computes the proposed update

/negationslash

Computing Φ( θ ) -1 2 can be performed by taking the singular value decomposition of the matrix Φ( θ ), and then inverting and taking the square root of its eigenvalues. This can be accomplished in d ω arithmetic operations since Φ( θ ) is a d × d matrix. Once Φ( θ ) -1 2 is computed, the computation θ +Φ( θ ) -1 2 ξ can be performed in d 2 arithmetic operations. Thus Line 7 can be computed in O ( d ω ) ≤ O ( md ω -1 ) arithmetic operations.

Line 8 requires determining whether z ∈ K . This can be accomplished in O ( md ) arithmetic operations, by checking whether the inequality Az ≤ b is satisfied.

Line 11 requires computing the determinant det(Φ( w )) and f ( w ) for w = θ and w = z . Computing det(Φ( w )) can be accomplished by computing the singular value decomposition of det(Φ( w )), and then taking the product of the resulting singular values to compute the determinant. Since Φ( w ) is a d × d matrix, computing the singular value decomposition can be done in d ω arithmetic operations. Computing f ( w ) for any w ∈ R d can be accomplished in one call to the oracle for the value of f . Thus, Line 11 can be computed in O ( d ω ) ≤ O ( md ω -1 ) arithmetic operations, plus two calls to the oracle for the value of f .

Therefore, adding up the number of arithmetic operations and oracle calls from all the different steps of Algorithm 1, we get that each iteration of Algorithm 1 can be computed in O ( md ω -1 ) arithmetic operations plus O (1) calls to the oracle for the value of f .

E.2 Bounding the step size

Definition E.1 (Cross-ratio distance) Let u, v ∈ Int( K ) . If u = v , let p, q be the endpoints of the chord in K which passes through u and v such that the four points lie in the order p, u, v, q . Define

if u = v , and σ ( u, v ) = 0 if u = v .

For convenience, we define the cross-ratio distance between any two subsets S 1 , S 2 ⊆ K as

And for any S ⊆ R d and any density ν : R d → R we define the induced measure:

Definition E.2 For any positive-definite matrix M ∈ R d × R d , and any u ∈ R d , we define

Lemma E.2 For any u, v ∈ Int( K ) , we have

/negationslash

Proof: Let p, q be the endpoints of the chord in K which passes through u and v such that the four points lie in the order p, u, v, q . Then

Thus, we have

Lemma E.3 For any u, v ∈ Int( K ) such that ‖ u -v ‖ Φ( u ) ≤ 1 2 α 1 / 2 we have that

Proof:

Thus,

Therefore, for all w ∈ R d we have

Thus,

E.3 Bounding the acceptance probability

Lemma E.4 Let θ ∈ Int( K ) . Then the acceptance ratio of satisfies

By Lemmas E.5, E.9, and E.8 we have that

Lemma E.5 Let θ ∈ int( K ) . Then

Proof: Since z ∼ N ( θ, Φ( θ ) -1 ), we have that

for some ξ ∼ N (0 , I d ).

Since α -1 H ( θ ) + η -1 I d /followsequal η -1 I d , and H ( θ ) and I d are both positive definite, we have that ηI d /followsequal ( α -1 H ( θ ) + η -1 I d ) -1 . (8)

Thus,

Now, since ξ ∼ N (0 , I d ), by the Hanson-wright concentration inequality for the χ -distribution [22, 40], we have that

Thus, Equations (9) and (10) together imply that

Now, in the setting where f is L -Lipschitz, we have

Therefore,

where the last inequality holds by Inequality (11), since η ≤ 1 10 4 dL 2 .

Moreover, in the setting where f is differentiable and β -Lipschitz, we have that, since z -θ is a multivariate Gaussian random variable,

If ( z -θ ) /latticetop ∇ f ( θ ) ≤ 0, we have that

Therefore,

where the last Inequality holds by Inequality (11), since η ≤ 1 10 4 dβ and since the Gaussian distribution is symmetric about the d -1-dimensional hyperplane ( z -θ ) /latticetop ∇ f ( θ ) = 0.

Lemma E.6 For any θ, z ∈ Int( K ) such that ‖ θ -z ‖ Φ( θ ) ≤ 1 4 α 1 / 2 , we have that

The proof of Lemma E.6 is an adaptation of the proof Lemma 3 in [41] to our Markov chain's 'soft' barrier function, and follows roughly along the lines of that proof.

Proof: Since ‖ θ -z ‖ Φ( θ ) ≤ 1 4 α 1 / 2 , by Lemma E.3 we have that

From Inequality (12) we have that

Thus, denoting by λ i ( M ) the i th-largest eigenvalue of any matrix M ∈ R d × R d , we have by Inequality (12) that

Now the KL-divergence between any to multivariate Gaussian distributions with any means µ 1 , µ 2 ∈ R d and any covariance matrices Σ 1 , Σ 2 ∈ R d satisfies (see e.g. Section 9 of [10] or Fact 5 in [41])

Therefore we have that

where the first inequality is Pinsker's inequality, the second inequality holds because log( 1 t ) ≤ 1 t -1 for all t>0, the fourth inequality holds because (1 + t ) 2 + 1 (1+ t ) 2 -2 ≤ 6 t 2 for all t ∈ [ -1 4 , 1 4 ], and the fifth inequality holds since t 2 is convex for t ∈ R .

For every j ∈ N , let m j = m + /floorleft αη -1 j 2 /floorright d . Consider the matrices A j = [ A /latticetop , I d , . . . , I d ] /latticetop where A is concatenated with m j -1 d copies of the identity matrix I d . And consider the vectors b j = ( b /latticetop , j 1 /latticetop , . . . , j 1 /latticetop ), where b is concatenated with m j -1 d copies of the vector j 1 , where 1 = (1 , . . . , 1) /latticetop ∈ R d is the all-ones vector. Then K = { θ ∈ R d : A j θ ≤ b j } , and the Hessian of the corresponding log-barrier functions is

Lemma E.7 For all w ∈ int( K ) we have that

uniformly in w . Moreover, for any θ ∈ int( K ) , any z ∈ 1 2 D θ and any sequence { z j } ∞ j =1 ⊆ 1 2 D θ such that lim j →∞ z j = z uniformly in z , we have that

uniformly in w , and that uniformly in z , and that

uniformly in z .

Proof:

Now, since w ∈ K ⊆ B (0 , R ), we have that

Thus, and

Thus, (19) (21), and (22) together imply that

where Inequalities (21) and (22) imply that the convergence to the limit is uniform in w . This proves Equation (15).

Moreover, since { z k } ∞ k =1 ⊆ 1 2 D θ , and D θ ⊆ K , we also have that

Therefore,

where the convergence of the limit in the fourth equality holds uniformly in z by (24) and the fact that lim j →∞ z j = z . Thus, we have that

uniformly in z . This proves Equation (17).

Moreover, since the determinant is a polynomial in the entries of the matrix, Inequality (23) implies that

uniformly in w .

We also have, plugging Inequality (20) into Equation (19),

Thus, Inequality (27) implies that

Thus, Inequality (28) implies that

Thus, since H ( w ) is positive definite, Inequality (29) together with inequalities (21) and (22) imply that

uniformly in w . This proves Equation (16).

Moreover, Inequality (28) implies that

for some c 1 > 0 which does not depend on j .

Inequality (27) also implies that

for some c 2 > 0 which does not depend on either j or w .

Thus, Inequalities (26), (30), and (31) together imply that for any θ, z ∈ int( K ) we have that

where the limit converges uniformly in z . This proves Equation (18).

Lemma E.8 Let ξ ∼ N (0 , I d ) and let θ ∈ int( K ) . Then with probability at least 1 -1 100 ˆ δ we have that and

Proof: Let z = θ + α 1 2 H -1 2 ( θ ) ξ . Then since ξ ∼ N (0 , I d ), by the Hanson-Wright concentration inequality [22, 40], we have that

And hence, since α -1 / 2 ‖ z -θ ‖ H ( θ ) = ‖ H 1 2 ( θ ) H -1 2 ( θ ) ξ ‖ 2 = ‖ ξ ‖ 2 , that

Hence,

Thus, since α ≤ log( 1 ˆ δ ) 100 d we have that

Lemma E.9 Consider any θ ∈ int( K ) , and ξ ∼ N (0 , I d ) . Let z = θ +(Φ( θ )) -1 2 ξ . Then

Proof: Let z j = θ + α 1 2 H -1 2 j ( θ ) ξ for all j ∈ N . Since H j ( θ ) /followsequal H ( θ ) for all j ∈ N , we have that z j = θ + α 1 2 H -1 2 j ( θ ) ξ ∈ 1 2 D θ whenever θ + α 1 2 H ( θ ) -1 2 ξ ∈ 1 2 D θ . Let E be the event that ‖ ξ ‖ 2 ≤ 10 √ d and that { z j } ∞ j =1 ⊆ 1 2 D θ . Thus, by Lemma E.8, we have that

Moreover, by Equation (16) of Lemma E.7 we have that lim j →∞ H -1 j ( θ ) = α -1 Φ -1 ( θ ), which implies that

uniformly in ξ (and hence uniformly in z = θ + Φ -1 2 ( θ ) ξ ) whenever the event E occurs. Therefore, by Equation (18) of Lemma E.7 we have that

uniformly in z whenever the event E occurs.

Since, for each j ∈ N , H j is the Hessian of a log-barrier function for K , by last line in the proof of Proposition 6 in [41], we have that, for all t ∈ (0 , 1 2 ] and all γ ∈ (0 , 1],

Setting t = 1 100 ˆ δ , we have

since α ≤ 1 400 2 d log( 1 ˆ δ ) .

Thus,

and

Inequalities (34) and (37) together imply that

Moreover, Equation (36) implies that there exists some number N ∈ N such that

Hence,

This proves Inequality (32).

Moreover, since, for each j ∈ N , H j is the Hessian of a log-barrier function for K , by Proposition 7 in [41], for all t ∈ (0 , 1 2 ], all α > 0 such that √ αd ≤ t 20 log( 11 t ) -3 2 , and all j ∈ N , we have that

Thus, Equations (41) and (34) imply that

since α ≤ 1 10 5 d .

By Equation (17) of Lemma E.7, Equation (35) implies that

uniformly in z , whenever the event E occurs. Thus, Equation (43) implies that

uniformly in z (and hence in ξ = Φ 1 2 ( θ )( z -θ )) whenever event E occurs. Thus, Equation (44) implies that there exists a number M ∈ N such that

Thus,

This proves Inequality (33).

E.4 Bounding the conductance

Lemma E.10 For any θ, z ∈ int( K ) we have that

Proof: First, we note that,

By Lemma E.6, the middle term on the RHS of (46) satisfies

Plugging in our choice of hyperparameter α = 1 10 5 d , Inequality (47) simplifies to

Thus, if we can show that ‖ P θ -N ( θ, Φ -1 ( θ )) ‖ TV ≤ 1 5 for all θ ∈ int( K ), we will have that ‖ P θ -P z ‖ TV ≤ 29 30 whenever ‖ θ -z ‖ Φ( θ ) ≤ 1 2 , as desired.

To bound the other two terms on the RHS of (46), we observe that

where is the acceptance ratio.

By Lemma E.5, we have that P z ∼ N ( θ, Φ -1 ( θ )) ( π ( z ) π ( θ ) ≥ 99 100 ) ≥ 99 10 . Therefore,

where the term with the exponent is bounded by Inequality (33) of Lemma E.9, and the other two terms are bounded by Lemmas E.8 and E.9. Therefore,

Plugging Inequalities (48) and (50) into (46), we obtain that ‖ P θ -P z ‖ TV ≤ 29 30 whenever ‖ θ -z ‖ Φ( θ ) ≤ 1 2 , as desired.

We recall the following isoperimetric inequality for a log-concave distribution on a convex body, which uses the cross-ratio distance:

Lemma E.11 (Isoperimetric inequality for cross-ratio distance (Theorem 2.2 of [32]))

Let π : R d → R be a log-concave density, with support on a convex body K . Then for any partition of R d into measurable sets S 1 , S 2 , S 3 , the induced measure π /star satisfies

For any θ ∈ Int( K ), define the random variable Z θ to be the step taken by the Markov chain in Algorithm 1 from the point θ . That is, set z = θ +Φ( θ ) -1 2 ξ where ξ ∼ N (0 , I d ). If z ∈ K , set Z θ = z with probability min ( π ( z ) √ det(Φ( z )) π ( θ ) √ det(Φ( θ )) exp ( ‖ z -θ ‖ 2 Φ( θ ) -‖ θ -z ‖ 2 Φ( z ) ) , 1 ) . Else, set z = θ .

For any θ , S ⊆ R d , define the one-step transition probability of the soft-threshold Dikin walk Markov chain to be

The next proposition shows that the soft-threshold Dikin walk Markov chain is reversible and π is a stationary distribution of this Markov chain:

Proposition E.12 (Reversibility and stationary distribution) For any S 1 , S 2 ⊆ K we have that

Proof: Let ρ θ ( z ) := √ det(Φ( θ )) (2pi) d 2 e -1 2 ( θ -z ) /latticetop Φ( θ )( θ -z ) for any θ, z ∈ Int( K ) be the density of the N ( θ, Φ( θ ) -1 ) distribution.

Define the conductance φ of the Markov chain to be

Lemma E.13 The conductance φ satisfies

Proof: The proof follows the general format for conductance proofs for geometric Markov chains. Let S 1 ⊆ K and let S 2 = K \ S 1 . Without loss of generality, we may assume that π ( S 1 ) ≤ 1 2 (since otherwise we could just swap the names ' S 1 ' and ' S 2 '). Let

and let

By Proposition E.12 we have that

Moreover, by Lemma E.10, for any u, v ∈ Int( K ) we have that

Thus, on the one hand, by Lemma E.2, Inequality (53) implies that for any u, v ∈ Int( K ),

On the other hand, Equations (51) imply that, for any u ∈ S ′ 1 , v ∈ S ′ 2 we have that

Thus, Inequalities (54) and (55) together imply that

Moreover by Lemma E.11 we have that

First, we assume that both π /star ( S ′ 1 ) ≥ 1 4 π /star ( S 1 ) and π /star ( S ′ 2 ) ≥ 1 4 π /star ( S 2 ). In this case we have

Now suppose that instead either π /star ( S ′ 1 ) < 1 4 π /star ( S 1 ) or π /star ( S ′ 2 ) < 1 4 π /star ( S 2 ). If π /star ( S ′ 1 ) < 1 4 π /star ( S 1 ) then we have

Similarly, if π /star ( S ′ 2 ) < 1 4 π /star ( S 2 ) we have

Therefore, Inequalities (58), (59), and (60) together imply that

for every partition S 1 ∪ S 2 = K . Hence, Inequality (61) implies that

Definition E.3 We say that a distribution ν is w -warm for some w ≥ 1 with respect to the stationary distribution π if sup z ∈ K ν ( z ) π ( z ) ≤ w .

Lemma E.14 (Corollary 1.5 of [31]) Suppose that µ 0 is the initial distribution of a lazy reversible Markov chain with conductance φ > 0 and stationary distribution π , and let µ i be the distribution of this Markov chain after i ≥ 0 steps. Suppose that µ 0 is w -warm with respect to π for some w ≥ 1 . Then for all i ≥ 0 we have

Lemma E.15 Let δ > 0 . Suppose that f : K → R is either L -Lipschitz (or has β -Lipschitz gradient). Suppose that θ 0 ∼ µ 0 where µ 0 is a w -warm distribution with respect to π ∝ e -f with support on K . Let µ denote the distribution of the output of Algorithm 1 with hyperparameters α ≤ 1 10 5 d and η ≤ 1 10 4 dL 2 (or η ≤ 1 10 4 dβ ), if it is run for T iterations. Then if T ≥ 10 9 ( 2 mα -1 + η -1 R 2 ) × log( w δ ) we have that

Proof: By Lemma E.13 we have that the conductance φ of the Markov chain in Algorithm 1 satisfies and hence that

Proof: [of Theorem 2.1]

Total variation bound. Recall that we have set the step size parameters α = 1 10 5 d and either η = 1 10 4 dL 2 (if f is L -Lipschitz) or η = 1 10 4 dβ (if f is β -smooth). Thus, after running Algorithm 1 for T = 10 9 ( 2 mα -1 + η -1 R 2 ) × log( w δ ) iterations, by Lemma E.15 we have that the distribution µ of the output of Algorithm 1 satisfies

Bounding the number of operations. Moreover, by Lemma E.1 we have that each iteration of Algorithm 1 can be implemented in O ( md ω -1 ) arithmetic operations plus O (1) calls to the oracle for the value of f . Thus, the number of iterations T taken by Algorithm 1 is O (( md + dL 2 R 2 ) × log( w δ )) iterations in the setting where f is L -Lipschitz and O (( md + dβR 2 ) × log( w δ )) iterations in the setting where f is β -smooth, where each iteration takes one function evaluation and md ω -1 arithmetic operations.

F Extension to general barrier functions?

From any point θ , our algorithm proposes a step with Gaussian distribution N ( θ, 1 d ( ∇ 2 g ( θ )) -1 ), where g is the following barrier function

where ϕ ( θ ) is a barrier function for the polytope K , and the parameter ˆ η -1 = Ω( 1 L 2 ) in the setting where f is guaranteed to be L -Lipschitz and ˆ η -1 = Ω( 1 β ) in the setting where f is guaranteed to be β -smooth. Thus, most of the probability mass of the Gaussian distribution concentrates in the Dikin ellipsoid ˆ D θ = θ + { w : w /latticetop ( ∇ 2 g ( θ )) -1 w ≤ 1 } for the barrier function g . For simplicity, in our main result, we assume that ϕ ( θ ) := -∑ m j =1 log( b j -a /latticetop j θ ) (which has self-concordance parameter m ), however, we can in principle choose any barrier function ϕ for the polytope K , such as the entropic barrier [4] or the Lee-Sidford Barrier [29] which have self-concordance parameter roughly ν = d .

To arrive at our barrier function from a more axiomatic approach, we first consider the definition of ν -self concordant barrier function:

Definition F.1 ( ν -self-concordant barrier function for K ) We say that g is a ν -selfconcordant barrier function if g : Int( K ) → R and g satisfies the following conditions:

Convex and differentiable barrier: g is convex and third-order differentiable, and g ( x ) → + ∞ as x → ∂K .

Self-concordance: ∇ 3 g ( x )[ h, h, h ] ≤ 2( ∇ 2 g ( x )[ h, h ]) 3 / 2 for all h ∈ R d (this ensures that the Hessian of the barrier function does not change too much each time the Dikin walk takes a step from x to z , that is, 1 2 ∇ 2 g ( z ) /precedesequal ∇ 2 g ( x ) /precedesequal 2 ∇ 2 g ( z ) )

g is ν -self concordant: h /latticetop ∇ g ( x ) ≤ √ νh /latticetop ∇ 2 g ( x ) h for every x ∈ Int( K ) , h ∈ R d

The fact that our barrier function (63) satisfies parts (1) and (2) of Definition F.1 follows from the fact that ϕ satisfies Definition F.1 and that ∇ 3 ( θ /latticetop θ ) = 0. We discuss the selfconcordance parameter ν for which our barrier function satisfies (63) below.

To ensure that the steps z ∼ N ( θ, ( 1 d ∇ 2 g ( θ )) -1 ) proposed by our Dikin walk Markov chain arising from the barrier function g has an Ω(1) acceptance ratio e f ( z ) -f ( θ ) , we require that the function g satisfies the following property. This property says that at least 1 4 of the volume of the Dikin ellipsoid ˆ D θ is contained in the sublevel set { z ∈ K : f ( θ ) < f ( θ ) + 2 } where the value of f does not increase by more than 2.

Property F.1 Dikin ellipsoid mostly contained in sublevel set: At every θ ∈ Int( K ) , the Dikin Ellipsoid, satisfies

When designing a barrier function g , there is a trade-off between choosing g such that the self-concordance parameter ν is small, while at the same time ensuring that Property F.1 holds. Roughly, to make the parameter ν as small as possible, we would like the Dikin ellipsoid to be large relative to the Hilbert-distance metric for the convex body K (This is because, by Proposition 2.3.2(iii) of [39], any ν -self concordant function g satisfies ( h /latticetop ∇ 2 g ( x ) h ) -1 2 ≤ | h | x ≤ (1+3 ν )( h /latticetop ∇ 2 g ( x ) h ) -1 2 , for any h ∈ R d where | h | x := sup { α > 0 : x ± αh ∈ { z ∈ K } ). On the other hand, if we make the Dikin ellipsoid too large (with respect to the sublevel set { z ∈ K : f ( z ) < f ( x ) + 2 } ) then Property F.1 will not be satisfied, and the steps proposed by the Dikin walk will have a very low acceptance probability. Setting the hyperparameter ˆ η = 1 L 2 when f is L -Lipschitz or ˆ η = 1 β when f is β smooth ensures that our barrier function in (63) satisfies Property F.1.

In the special case where f is the log-density of the uniform distribution ( f ≡ 0), or when f is any linear function, we have that f is β -smooth for β = 0. Thus, our barrier function (63) we use to encode the geometry of f on K is the same as the barrier function ϕ for the polytope K . This is because, since the level sets of linear functions f are half-planes, any ellipsoid centered at θ has at least half of its volume in the sublevel set { z ∈ K : f ( z ) ≤ f ( θ ) } ⊆ { z ∈ K : f ( z ) < f ( θ ) + 2 } , satisfying Property F.1.

The following lemma shows that our barrier function in (63) is ν -self concordant with ν = O ( ν ′ +ˆ ηR 2 ), where ν ′ is the self-concordance parameters of ϕ (which is ν ′ = m if we choose ϕ to be log-barrier function). Thus, our barrier function in (63) is ν = O ( ν ′ + L 2 R 2 ) selfconcordant in the setting where f is L -Lipschitz, and ν = O ( ν ′ + βR 2 ) in the setting where f is β -smooth:

Lemma F.2 Suppose that φ ( x ) is a ν ′ -self concordant barrier function for a convex body K ⊂ R d where B (0 , r ) ⊆ K ⊆ B (0 , R ) for some R > r > 0 . Let g ( x ) = φ ( x ) + α 2 x /latticetop x for some α > 0 . Then g is ν -self concordant for ν = 4 ν ′ +4 αR 2 .

The proof of Lemma F.2 is given in Appendix F.1. The polynomial dependence of ν on LR or βR 2 is a necessary feature of any objective function satisfying Definition F.1 and Property F.1. In appendix F.2, we give explicit examples of classes of objective functions f and polytopes K for which the minimum value of ν depends polynomially on LR (and classes of smooth functions f where ν depends polynomially on √ βR .

An open problem is whether one can design versions of the Dikin walk which sample from log-concave distributions with a runtime that depends only on the dimension d , and is independent of L , R or β and which are invariant to affine transformations. The difficulty in achieving this lies in the fact that (e.g., in the setting where f is L -Lipschitz), on the one hand, the level sets of f where most of the probability mass of ∝ e -f concentrates may have a diameter roughly RL times smaller than the diameter R of K . Thus, to have an Ω(1) acceptance probability the Dikin walk may need to take steps that are roughly RL times smaller than the diameter of K . On the other hand, the isoperimetric inequality currently used to bound the mixing time of the Dikin walk uses a metric- the cross-ratio distance for K -which, roughly speaking, defines distances between steps by how quickly these steps approach the boundary of K . Thus, measured in the cross-ratio distance, the steps proposed by the Dikin walk are of size roughly proportional to 1 RL , and mixing time bounds obtained with this isoperimetric inequality thus depend polynomially on RL . To obtain mixing time bounds independent of the R,L,β one may need to show a new isoperimetric inequality which is based on a different metric that encodes the geometry of all of the level sets of f -rather than just the geometry of its support K .

F.1 Proof of Lemma F.2

Proof: [Proof of Lemma F.2] For any x ∈ int( K ), we have

Thus, for any h ∈ R d ,

Thus,

where the third inequality holds since φ is ν ′ self-concordant. Thus, plugging in ν = 4 ν ′ + 4 αR 2 to (65), we get that

F.2 Lower bounds for self-concordance parameter

Lower bounds on ν for worst-case L -Lipschitz f . Consider the L -Lipschitz function f ( θ ) = L ‖ θ ‖ 2 constrained to the convex body K = R 2 √ d [ -1 , 1] d which is contained in a ball of radius R . In this case, any barrier function g which satisfies Property F.1 has Dikin Ellipsoid ˆ D θ which is contained in the ball B (0 , 8 L ). Thus, we have that ν ≥ Ω( RL ). (This is because, by Proposition 2.3.2(iii) of [39], any ν -self concordant function g satisfies ( h /latticetop ∇ 2 g ( θ ) h ) -1 2 ≤ | h | θ ≤ (1 + 3 ν )( h /latticetop ∇ 2 g ( θ ) h ) -1 2 , for any h ∈ R d where | h | θ := sup { α > 0 : θ ± αh ∈ { z ∈ K } . Thus, at θ = 0 and choosing h = (1 , · · · , 1) we have

( ) ).

Since there exists a convex body K for which any self-concordant barrier function satisfying Definition F.1 has self-concordance parameter at least ν ≥ d , for any L, R > 0 there exists a function f and convex body K ⊂ B (0 , R ) such that the self-concordance parameter of every barrier function satisfying both Definition F.1 and Property F.1 is at least ν ≥ Ω(max( d, LR )).

Lower bounds on ν for worst-case β -smooth f . Consider the β -smooth function f ( θ ) = 1 2 βθ /latticetop θ constrained to the convex body K = R 2 √ d [ -1 , 1] d . At θ = 0, any ellipsoid ˆ D θ satisfying Property F.1 is contained in the ball 8 √ β B (0 , 1). Thus, we have that ν ≥ R 8 √ β √ β = Ω( √ βR ) (This is because at θ = 0 we have | h | θ ‖ h ‖ 2 = R and ( h /latticetop ∇ 2 g ( θ ) h ) -1 2 ≤ O ( 1 √ β ) and thus, by Proposition 2.3.2(iii) of [39], we have ν ≥ Ω ( | h | θ ( h /latticetop ∇ 2 g ( θ ) h ) -1 2 ) ≥ √ βR ). Thus, for any β, R > 0 there exists a function f and convex body K ⊂ B (0 , R ) such that the selfconcordance parameter of every barrier function satisfying both Definition F.1 and Property F.1 is at least ν ≥ max( d, Ω( βR )).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of sampling from a log-concave distribution, parametrized by $\\exp(f)$ where $f$ is $L$-Lipschitz, over a $d$-dimensional polytope specified by $m$ halfspaces. Assuming the polytope is contained in a ball of radius $R$, the authors develop an algorithm that generates a sample from a distribution at most $\\delta$ TV distance away from the target distribution, in time $O((md+dL^2R^2)\\cdot md^{\\omega-1}\\log(w/\\delta))$ where $\\omega\\approx 2.37$ and the initial point is $w$-warm start. Their approach is an extension from Dikin walk by introducing a soft-threshold regularization term. This result has a wide range of applications in differential private optimization. \n\nIt is worth noting that Dikin walk was previously adapted for sampling from a uniform distribution over a polytope, and the iteration count and arithmetic operations per iteration are nearly tight [LLV20]. For sampling from a log-concave distribution, hit-and-run has its runtime depends on the dimension $d$, but under certain regimes, it is slower than the proposed algorithm in this paper. \n\n[LLV20]: Laddha, Lee and Vempala. Strong self-concordance and sampling. STOC'20.\n\nSTRENGTHS:\nUnder many regimes, this paper provides the state-of-the-art result for sampling from a log-concave distribution over a polytope. Moreover, their algorithm is conceptually easy to understand and should be easy to implement, as it is a variant of the standard Dikin walk for sampling from a uniform distribution. The soft-threshold regularization term is not completely novel, but prior work gets a highly sub-optimal dependence on the self-concordance parameter $\\nu$, at least for log-barrier. \n\nConsequently, this algorithm also leads to many applications, such as differential private empirical risk minimization and Bayesian Lasso logit regression.\n\nWEAKNESSES:\nThere are several weaknesses of this paper to highlight.\n\n1. Restriction to log-barrier. Note that the $md$ iteration bound comes from the log-barrier is $m$-self-concordance. It is well-known that if $m\\gg d$, better barrier functions with better parameters exist, such as the hybrid barrier and Lewis weights barrier. However, if one carefully examines the techniques employed in this paper, they are tuned very specified towards log-barrier. In particular, the authors prove that in the limit, sampling from the regularized Gaussian distribution over a polytope whose constraint contains infinitely many copies of identities will converge to sample from the uniform distribution over another polytope. This enables them to utilize prior rich works on Dikin walk for uniform sampling over polytopes to conclude the proof. Unfortunately, this approach does not scale with more complicated barrier functions. Take the volumetric function as an example, the Hessian itself reweights all constraints based on their leverage scores, and duplicating a constraint infinitely many times will make the leverage score goes to 0, invalidating the construction completely. I think it will be much more interesting if the proposed framework can generalize to other barrier functions.\n\n2. Tightness of arithmetic operations. The algorithm takes $O(md^{\\omega-1})$ to form the Hessian matrix. This does not seem to be tight. I think the results in this paper will be much stronger if, at least for log-barrier, the arithmetic operations per iteration are tight (say nearly linear in $md$). \n\n3. Dependence on $L$ and $R$. The Dikin walk is inherently a second-order method, so one would expect a better dependence on $L$ and $R$, hopefully polylog dependence. Notably, the Dikin walk for sampling from uniform distribution over a polytope does not depend on $R$, as the walk exploits the local geometry of the Dikin ellipsoid. Hit-and-run can achieve such a result only if the body is in isotropic position [CE22]. The quadratic dependence on $R$ weakens the result significantly in my opinion. Authors should discuss whether the dependence on $R$ can be improved.\n\nDespite the above concerns, I think this paper is technically solid and has many interesting applications.\n\n[CE22]: Yuansi Chen and Ronen Eldan. Hit-and-run via localization schemes. 2022.\n\nQUESTIONS:\nSee weakness. I'm willing to raise score if authors can address my concerns.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a sampling method for log-concave distribution with bounded polytope. The methods can be used in privacy preservation. The theoretical results show the guarantee of the sampling methods with some accepted error magnitude. The comparisons for other methods are also detailedly proposed, showing the improvement of the proposed one.\n\nSTRENGTHS:\nPros:\n1. The work focuses on an important problem in the privacy field--sampling and also provides some ways for us to adopt their methods.\n2. The theoretical results show the effectiveness of the proposed methods in magnitude.\n3. Some interesting discussion and delicate application in the appendix is provided for better practical usage.\n\nWEAKNESSES:\nCons:\n1. Work in the main context is not friendly to the readers. Maybe authors can use more paras to provide different information. The current version is not so friendly for me to read at least. Some more important Lemma/Corollary can also be organized, I believe. I think the design of the func for the barrier should also be clearly stated at first, but not in the Algorithms.\n2. Some technical contributions can also be provided to better differentiate yours and the Dikin walk with simply applying a new regularizer.\n3. Some numerical studies can be provided to better show the effectiveness of the methods.\n\nQUESTIONS:\n1. can you provide some insights about how your algorithm can speed up the sampling. I just think that is because compared with the Dikin walk, the designed regularizer supports fast computing, but I am not sure.\n2. Also, I can not figure out how much improvement for the dimension $d$ in the real problem. Some small examples can be provides to both show the constant in the magnitude and $d$'s improvement in some cases.\n3. I am not sure about the adjustment of the line height is allowed.\n4. Maybe we also need to consider the time cost for computation of the Lip/smooth constant for $f$\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of sampling from distribution of the form $\\pi(\\theta) \\propto e^{-f(\\theta)}$, restricted to a polytope $K$. Here, $f$ is either Lipschitz or smooth convex. To this end, the authors propose to use _Dikin walk_ Markov chain (Kannan and Narayanan, 2012), which was originally proposed as a sampler for the uniform distribution on polytopes. Given that $K = \\{\\theta : A\\theta \\leq B\\}$ with $A=(a_1,\\ldots, a_n), B=(b_1,\\ldots,b_n)$, the Dikin walk proposes the update $z = \\theta + \\sqrt{\\alpha H^{-1}(\\theta)}\\xi$ where $\\xi\\sim N(0,I_d)$, $H$ is the Hessian matrix of the log-barrier function $\\varphi(\\theta) = -\\sum_{j=1}^m \\log(b_j-a_j^{\\perp}\\theta)$, and $\\alpha$ is a hyperparameter. If the proposed update is in the interior of $K$, it is accepted with a certain probability, otherwise it is rejected. To ensure that the acceptance probability is $\\Omega(1)$ while allowing the Dikin walk to make sufficiently large steps, the authors propose to add the scaled Hessian $\\alpha H(\\theta)$ by a regularizing term $\\eta^{-1}I_d$, in order to \"round up\" the set of proposals; then, we can adjust $\\alpha$ and $\\eta$ so that the set of proposals can be fitted in $K$ with high probability. The authors have done a runtime analysis and show that their method can be run in $\\tilde{O}(md^{\\omega+1})$ in order to sample with small TV error, where $m$ is the dimension of the polytope and $\\omega$ is the matrix multiplication constant. In particular, when $m = O(d)$ (e.g. $A$ is full rank), it can be run in $\\tilde{O}(d^{\\omega+2})$.\n\nSTRENGTHS:\n- The authors did an excellent job on literature review, with complete runtime analysis of previous methods. The time for warm-start and the evaluation of $f$ are also taken into consideration.\n- The authors provide detailed explanation on Dikin walk and its limitations.\n- The algorithm seems simple enough to implement with a few lines of code.\n- The authors provide some motivations for sampling on polytopes in the area of differential privacy.\n- I appreciate the overview of the proof of the sampling guarantee in Section 3.\n\nWEAKNESSES:\n- The authors mention that the hyperparameter $\\alpha$ alone is insufficient for the Dikin walk to make large steps due to the differences in the geometry of the proposals at each point $\\theta$. So I think it is better to add a regularizer that depends on $\\theta$ that allows us to remove the hyperparameter $\\eta$. Have the authors considered such approach? (In other words, can we somehow make $\\eta$ depend on $\\theta$?)\n- The comparison of the runtimes are nice, but I think some experiments on simulated data would make it more convincing that the proposed method is better than the previous ones.\n- The authors might want to also discuss the following paper by Chalkis, Fisikopoulos, Papachristou and Tsigaridas: \"Truncated Log-concave Sampling for Convex Bodies with Reflective Hamiltonian Monte Carlo\".\n- In the introduction, the authors need to be more explicit about the matrix multiplication constant $\\omega$. Outside readers might not know that $\\omega \\leq 2.373$, making it harder to compare the numbers in Table 1.\n\nQUESTIONS:\n- There are some notations that should be introduced in the Notation section. For example, $a_j$ and $b_j$. $\\Phi(\\theta)$ and $\\Phi(z)$ are used at the beginning of Section 3 but I could not find their definitions anywhere before this section.\n- Maybe I have missed it, but there should be a mention at the beginning on a lower bound of $R/r$ so that it is easier to make comparisons of the numbers in Table 1.\n\nLIMITATIONS:\nThe authors have mentioned a limitation of the proposed method that it can only be applied to Lipschitz or $\\beta$-smooth $f$. Also, finding a better regularizing term is also a possible research direction.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies the question of efficient sampling from a log-concave distribution from a constrained polytope. The main idea is to use the well known Dikin's random walk with a soft threshold. The threshold is important to ensure a far more efficient convergence to the desired distribution. This soft threshold ensures that the acceptance ratio is high while also ensuring that the walk remains inside the polytope.

STRENGTHS:
The paper gives a definite improvement over the best-known algorithm in the run time for both Lipschitz and smooth loss function. This has direct implications in making DP algorithm for convex optimization more efficient. Just on the strength of the result of the algorithm, I favor acceptance. I have read the proof and did not find any issue with any of it.

WEAKNESSES:
The only weakness of the paper is that the writing can be improved.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07