Title: Log-concave Sampling from a Convex Body with a Barrier: a Robust and Unified Dikin Walk
Venue: NeurIPS 2024
Accepted: yes
Keywords: Log-concave sampling

Dikin walk
Abstract: We consider the problem of sampling from a $d$-dimensional log-concave distribution $\pi(\theta) \propto \exp(-f(\theta))$ for $L$-Lipschitz $f$, constrained to a convex body (described by $n$ hyperplanes) equipped with a barrier function, contained in a ball of radius $R$ with a $w$-warm start. 

We propose a \emph{robust} sampling framework that computes spectral approximations to the Hessian of the barrier functions in each iteration. We prove that for the polytope constraints, sampling with the Lee-Sidford barrier function mixes within $\widetilde O((d^2+dL^2R^2)\log(w/\delta))$ steps with a per step cost of $\widetilde O(nd^{\omega-1})$, where $\omega\approx 2.37$ is the fast matrix multiplication exponent. Compared to the prior work of Mangoubi and Vishnoi, our approach gives faster mixing time as we are able to design a generalized soft-threshold Dikin walk beyond log-barrier.

We further extend our result to show how to sample from a $d$-dimensional spectrahedron, the constrained set of a semidefinite program, specified by the set $\{x\in \mathbb{R}^d: \sum_{i=1}^d x_i A_i \succeq C \}$ where $A_1,\ldots,A_d, C$ are $n\times n$ real symmetric matrices. We design a walk that mixes in $\widetilde O((nd+dL^2R^2)\log(w/\delta))$ steps with a per iteration cost of $\widetilde O(n^\omega+n^2d^{3\omega-5})$. We improve the mixing time bound of prior best Dikin walk due to Narayanan and Rakhlin that mixes in $\widetilde O((n^2d^3+n^2dL^2R^2)\log(w/\delta))$ steps.

Log-concave Sampling from a Convex Body with a Barrier: a Robust and Unified Dikin Walk

Abstract

We consider the problem of sampling from a d -dimensional log-concave distribution π ( θ ) ∝ exp( -f ( θ )) for L -Lipschitz f , constrained to a convex body with an efficiently computable self-concordant barrier function, contained in a ball of radius R with a w -warm start.

We propose a robust sampling framework that computes spectral approximations to the Hessian of the barrier functions in each iteration. We prove that for polytopes that are described by n hyperplanes, sampling with the Lee-Sidford barrier function mixes within ˜ O (( d 2 + dL 2 R 2 ) log( w/δ )) steps with a per step cost of ˜ O ( nd ω -1 ) where ω ≈ 2 . 37 is the fast matrix multiplication exponent. Compared to the prior work of Mangoubi and Vishnoi, our approach gives faster mixing time as we are able to design a generalized soft-threshold Dikin walk beyond log-barrier.

, We further extend our result to show how to sample from a d -dimensional spectrahedron, the constrained set of a semidefinite program, specified by the set { x ∈ R d : ∑ d i =1 x i A i ⪰ C } where A 1 , . . . , A d , C are n × n real symmetric matrices. We design a walk that mixes in ˜ O (( nd + dL 2 R 2 ) log( w/δ )) steps with a per iteration cost of ˜ O ( n ω + n 2 d 3 ω -5 ) . We improve the mixing time bound of prior best Dikin walk due to Narayanan and Rakhlin that mixes in ˜ O (( n 2 d 3 + n 2 dL 2 R 2 ) log( w/δ )) steps.

1 Introduction

Given a convex body, generate samples from the body according to structured densities is a fundamental problem in computer science and machine learning. It has extensive applications in constrained convex optimization [Lovász and Vempala, 2006, Narayanan, 2016], differentially private learning [Wang et al., 2015, Lin et al., 2024] and online optimization [Narayanan and Rakhlin, 2017]. A central theme in the theory of sampling is to leverage stochasticity and reduce per iteration costs without having to proportionally increase the number of iterations. This theme has played out in continuous optimization for both first and second order methods. For the first order methods, focus is on reducing the variance of the gradient estimators [Johnson and Zhang, 2013, Shalev-Shwartz and Zhang, 2013, Defazio et al., 2014]. For the second order methods, matrix sketching is often used to

MIT CSAIL

reduce computation and storage related to the Hessian matrix [Lee et al., 2019, Jiang et al., 2020b, 2021, Song and Yu, 2021, Qin et al., 2023].

In Markov chain Monte Carlo (MCMC), much of the progress is on the first order methods. Nonasymptotic analyses are performed for the stochastic gradient Langevin algorithms and their variance reduced extensions [Dubey et al., 2016, Raginsky et al., 2017, Brosse et al., 2018, Chatterji et al., 2018, Zou et al., 2018, Dalalyan and Karagulyan, 2019, Li et al., 2019, Ding and Li, 2021]. For second order methods, there has been a paucity when it comes to applying the sketching techniques.

In this paper, we focus on sampling from a log-concave distribution constrained to a d -dimensional convex body K that is described by n constraints, where second order information is proven essential for capturing the geometry of the convex body and consequently for achieving fast convergence rates [Narayanan, 2016, Narayanan and Rakhlin, 2017, Chen et al., 2018, Laddha et al., 2020, Mangoubi and Vishnoi, 2023, 2024]. In particular, we associate the convex body with a selfconcordant barrier function [Nesterov and Nemirovskii, 1994, Renegar, 1988, Vaidya, 1989] and utilize the Hessian matrix H ( x ) of the barrier function in the sampling algorithm. We then use the Hessian matrix to propose samples and compute the probability to accept or reject the proposed samples. We note that this problem involves subtleties beyond the scope of constrained optimization. In continuous optimization, which focuses on finding the descent directions, unbiased estimators with reasonable variance oftentimes suffice [Vaidya, 1989, Lee et al., 2015, Huang et al., 2022]. In MCMC, on the other hand, the target probability distribution needs to be maintained along the entire trajectory. This poses significant challenges to speed up the sampling algorithms. In the scenario of uniform sampling over a polytope, Laddha et al. [2020] shows that for a simple logarithmic barrier 1 , an unbiased estimator in fact suffices. However, more complicated barriers such as the Lee-Sidford barrier [Lee and Sidford, 2014, 2019] can only be approximately computed 2 , an unbiased estimator is extremely hard to be obtained. Moreover, for sampling from more sophisticated log-concave distributions and convex bodies, a more general approach is needed.

We therefore propose to obtain a high precision estimator to the desired acceptance rate with an improved per step running time and without sacrificing the rapid mixing time. In particular, we ask the following question:

Can we significantly reduce per iteration cost, while preserving the convergence rate of the log-concave sampling algorithms over various convex bodies?

We answer the above question in the affirmative. To this end, we present a slew of results regarding log-concave sampling. For polytopes, we provide a walk that mixes in ˜ O ( d 2 + dL 2 R 2 ) steps 3 with per iteration cost ˜ O ( nd ω -1 ) . Prior state-of-the-art results are due to Mangoubi and Vishnoi [2023, 2024], for which their walks mix in ˜ O ( nd + dL 2 R 2 ) with a per iteration cost 4 ˜ O (nnz( A ) + d 2 ) . Our walk mixes faster whenever n ≥ d . Our result partially resolves an open problem in Mangoubi and Vishnoi [2023] as they asked whether it's possible to design a Dikin walk whose mixing time is only depends on d and independent of L, R . We remove the L and R dependence on the dominating term d 2 , hence for the case of sampling from uniform distribution ( L = 0 ), we recover the state-of-art result of Laddha et al. [2020] which mixes in ˜ O ( d 2 ) steps with a per iteration cost ˜ O ( nd ω -1 ) .

We also consider sampling from convex bodies beyond polytopes. In semidefinite programming (SDP), one often focuses on the dual program of an SDP, where the constraint set is defined as a spectrahedron K = { x ∈ R d : ∑ d i =1 x i A i ⪰ C } 5 for n × n symmetric matrices A i and C [Jiang et al., 2020a, Huang et al., 2022]. We propose a walk that samples from a log-concave distribution over a spectrahedron K using the Hessian information of the log-barrier. The walk mixes in ˜ O ( nd + dL 2 R 2 ) steps. For log-barrier of an SDP, explicitly forming the Hessian matrix takes a prohibitively large

O ( n ω d + n 2 d ω -1 ) time. We utilize our robust sampling framework to approximately compute this Hessian via tensor-based sketching techniques, achieving a runtime of ˜ O ( n ω + n 2 d 3 ω -5 ) . As long as n ≥ d 3 ω -4 ω -2 (which is a usual setting for SDP, where d ≪ n ), this provides a speedup.

1.1 Related works

In this work, we focus on Dikin walk [Kannan and Narayanan, 2012], a refined variant of ball walk [Lovász and Simonovits, 1993]. Roughly speaking, ball walk progresses by moving to a random point in the ball centered at the current point with the obvious downside that when the convex body is flat, ball walk progresses slowly. Dikin walk overcomes this problem by instead moving to a random point in a good ellipse centered at the current point, in an effort to better utilize the local geometry of the convex body.

For sampling over polytopes, a number of Dikin walks use the ellipse induced by the log-barrier functions [Kannan and Narayanan, 2012]. The work of Laddha et al. [2020] shows that uniform sampling over a polytope can mix in O ( νd ) steps, where ν is the self-concordant parameter of the barrier function 6 . Going beyond uniform distributions, Narayanan and Rakhlin [2017] proposes a walk that samples from a log-concave distribution on a polytope in ˜ O ( ν 2 d ) steps. This bound is later improved by Mangoubi and Vishnoi [2023], in which they show that for logarithmic barrier, a mixing time of ˜ O ( nd + dL 2 R 2 ) is attainable, where L is the Lipschitz constant of f and R is the radius of the ball that contains K . A more recent work by Kook and Vempala [2024] has obtained better bound when the function f is α -relative strongly-convex and the density π is β -Lipschitz. They manage to obtain a mixing bound of O ( νdβ log( w/δ )) . However, their algorithm requires stronger assumptions on f and hence is incomparable to our result. In most previous works, the focus has been on improving the mixing time, rather than on per iteration costs.

For any convex body, it is well-known that a universal barrier with ν = d exists [Nesterov and Nemirovskii, 1989, Lee and Yue, 2021], however it is computationally hard to construct the Hessian of the universal barrier. In short, the universal barrier requires to compute the volume of the polar set of a high dimensional body, which is even hard to approximate deterministically [Furedi and Barany, 1986, Nesterov and Nemirovskii, 1994]. The seminal work of Lee and Sidford [Lee and Sidford, 2014] presents a nearly-universal barrier with ν = O ( d log 5 n ) and the Hessian can be approximated in O ( nd ω -1 ) time. The Lee-Sidford barrier was originally designed to solve linear programs in O ( √ d ) iterations, and it has been leveraged lately for Dikin walks with rapid mixing time. For uniform sampling, Chen et al. [2018] gives an analysis with a walk that mixes in ˜ O ( d 2 . 5 ) steps. Subsequently, Laddha et al. [2020] improves the mixing to ˜ O ( d 2 ) steps. In a pursuit to better leverage the local geometry, Lee and Vempala [2017] proposes a walk relying on Riemannian metric that mixes in ˜ O ( nd 3 / 4 ) steps. The mixing rate is later improved to ˜ O ( nd 2 / 3 ) via Riemannian Hamiltonian Monte Carlo (RHMC) with log-barrier [Lee and Vempala, 2018] and ˜ O ( n 1 / 3 d 4 / 3 ) with a mixed Lee-Sidford and log-barrier [Gatmiry et al., 2024]. In this work, we show that log-concave sampling can also leverage Lee-Sidford barrier to obtain a mixing time of ˜ O ( d 2 + dL 2 R 2 ) . In comparison, the hit-and-run algorithm [Lovász and Vempala, 2007] mixes in ˜ O ( d 2 R 2 /r 2 ) steps where r is the radius of the inscribed ball inside K . For the regime where L = O (1) and r = O (1) , our walk mixes strictly faster than that of hit-and-run.

We note that for sampling over more general convex bodies, a recent work [Chen and Eldan, 2022] proves that for uniform sampling over an isotropic convex body, the mixing time bound is ˜ O ( d 2 /ψ 2 d ) where ψ d is the KLS constant [Kannan et al., 1995]. However, it is unclear how to generalize their result to non-isotropic convex bodies or log-concave densities.

1.2 Our results

Our results concern log-concave sampling from polytopes and spectrahedra. For polytopes, we state the result in its full generality.

Theorem 1.1 (Robust sampling for log-concave distribution over polytopes) . Let δ ∈ (0 , 1) and R ≥ 1 . Given a constraint matrix A ∈ R n × d with a vector b ∈ R n , let K := { x ∈ R d : Ax ≤ b } be the corresponding polytope. Suppose K is enclosed in a ball of radius R with non-empty interior. Let f : K → R be an L -Lipschitz, convex function with an evaluation oracle. Finally, let π be the distribution such that π ∝ e -f .

Suppose we are given an initial point from K that is sampled from a w -warm start distribution 7 with respect to π for some w > 0 , then there is an algorithm (Algorithm 1) that outputs a point from a distribution µ where TV( µ, π ) ≤ δ .

Let g : K → R be a ν -self-concordant barrier function such that in time C g , a spectral approximation ˜ H g of the Hessian of g denoted by H g can be computed satisfying

for ϵ = Θ(1 /d ) . Then, Algorithm 1 takes at most

Markov chain steps. It uses O (1) function evaluations and an extra C g + d ω time at each step.

Let us pause and make some remarks regarding Theorem 1.1. As long as the Hessian matrix can be approximately generated with an error inversely depends on d , then our algorithm is guaranteed to converge. Moreover, if the approximate Hessian can be generated more efficiently, then this directly implies an improvement of our algorithm. On the mixing time side, Theorem 1.1 is nearly-optimal up to polylogarithmic factors and the dependence on L and R . In particular, the νd mixing time bound is also achieved by Laddha et al. [2020] in the case of uniform sampling. We instantiate the meta result of Theorem 1.1 into the following theorem.

Theorem 1.2 (Robust sampling with nearly-universal barrier) . Under the conditions of Theorem 1.1, let g be the Lee-Sidford barrier with ν = O ( d log 5 n ) . Then, we have

The algorithm takes at most

The Lee-Sidford barrier [Lee and Sidford, 2014, 2019] is the first polynomial-time computable barrier with a nearly-optimal self-concordance parameter. Several prior works [Chen et al., 2018, Laddha et al., 2020] utilize this barrier for uniform sampling. For log-concave sampling, the walk of Kook and Vempala [2024] requires strong-convexity-like assumption on f in order to attain an ˜ O ( d 2 ) mixing time. Our work is the first to obtain an ˜ O ( d 2 ) mixing time for log-concave sampling over polytopes, when L and R are small.

Table 1: Comparison of algorithms for sampling from a log-concave density e -f over a d -dimensional polytope with n constraints, where f is L -Lipschitz. We omit ˜ O ( · ) to mixing time and per iteration cost. We assume the evaluation of f can be done in unit time. The first row is the hit-and-run walk. Our algorithm has the fastest mixing time among all Dikin walks, and for L = O (1) , r = O (1) , our walk mixes faster than hit-and-run.

Our next result is closely related to semidefinite programming. Given the the set K = { x ∈ R d : ∑ d i =1 x i A i ⪰ C } for symmetric n × n matrices A 1 , . . . , A d , C , we consider sampling from a log-concave distribution over K . We use the popular log-barrier for K studied in Nesterov and Nemirovskii [1994], Jiang et al. [2020a], Huang et al. [2022]. To further utilize our robust sampling framework, we develop a novel approach for generating a spectral approximation of the Hessian. Compared to Jiang et al. [2020a], Huang et al. [2022] in which the Hessian is maintained in a very careful way in conjunction with the interior point method, our spectral approximation does not require to control the changes over iterations and is highly efficient when n ≫ d , which is a popular regime for semidefinite programs.

Theorem 1.3 (Robust sampling for log-concave distribution over spectrahedra) . Let δ ∈ (0 , 1) and R ≥ 1 . Given a collection of symmetric matrices A 1 , . . . , A d ∈ R n × n and a target symmetric matrix C ∈ R n × n , let K := { x ∈ R d : ∑ d i =1 x i A i ⪰ C } be the corresponding spectrahedron. Suppose K is enclosed in a ball of radius R with non-empty interior. Let f : K → R be an L -Lipschitz, convex function with an evaluation oracle. Finally, let π be the distribution such that π ∝ e -f .

Suppose we are given an initial point from K that is sampled from a w -warm start distribution with respect to π for some w > 0 , then there is an algorithm (Algorithm 1) that outputs a point from a distribution µ where TV( µ, π ) ≤ δ .

Let g : K → R be the logarithmic barrier function with ν = n . There is an algorithm that uses

Markov chain steps. In each step it uses O (1) function evaluations and an extra cost of

Prior work due to Narayanan and Rakhlin [2017] provides a walk that mixes in ˜ O ( n 2 d 3 + n 2 dL 2 R 2 ) steps and each step could be implemented in O ( dn ω + n 2 d ω -1 ) time, our algorithm improves upon both mixing and per iteration cost. On the front of per iteration cost, computing the Hessian of the log-barrier of K exactly takes time O ( dn ω + n 2 d ω -1 ) , for the regime where n ≫ d the dominating term is dn ω . Note that whenever n ≫ d 3 ω -4 ω -2 , our algorithm is more efficient than that of exact computation. This is a common regime in semidefinite program in which the number of constraints d is small compared to the size of the primal solution.

2 Technical Overview

Before diving into our techniques, we first illustrate an informal version of our algorithm. The algorithm itself is a variant of the Dikin walk [Kannan and Narayanan, 2012] called the soft-threshold Dikin walk [Mangoubi and Vishnoi, 2023]. Starting with a w -warm start point, the algorithm approximately computes the Hessian of the barrier function, then proposes a new point z from a Gaussian with mean at the current point x and variance ˜ Φ( x ) -1 , where ˜ Φ is a multiple of the approximate Hessian plus a multiple of the identity. The proposal is then accepted via a Metropolis filter. Note that compared to the standard Dikin walk framework, our algorithm crucially allows the Hessian to be approximated with a good precision. This means any improvements on generating spectral approximation leads to runtime improvement of our algorithm in a black-box fashion.

Algorithm 1 Sampler for log-concave distribution π ∝ exp( -f ) over convex body with a barrier function g . A is the description of the convex set, function f is an L -Lipschitz concave function and g is a ν -self-concordant barrier function, x 0 ∈ K is a w -warm start point, and δ is the total variance distance between our Markov chain and the target distribution. Let K ⊆ R d enclosed by a ball of radius R .

2.1 Soft-threshold Dikin walk for log-concave sampling via a barrier-based argument

For uniform sampling from structured convex bodies such as polytopes, Kannan and Narayanan [2012] introduce a refined ball walk that utilizes a self-concordant barrier function associated with the convex body. To extend the Dikin walk framework for log-concave sampling, Narayanan and Rakhlin [2017] adds an additional coefficient exp( -f ( x )) exp( -f ( z )) to the acceptance probability of Metropolis filter. This achieves a sub-optimal mixing time bound of ˜ O ( ν 2 d ) . By introducing a soft-threshold regularizer to the Hessian, Mangoubi and Vishnoi [2023] proves that for log-barrier function, it is possible to approach the optimality with an ˜ O ( nd + dL 2 R 2 ) mixing time. Unfortunately, the method used by Mangoubi and Vishnoi [2023] is specialized to log-barrier, making it hard to generalize to other barriers. They show that under proper scaling, the true Hessian H ( x ) under a sequence of polytopes converges to the regularized Hessian under the given polytope in the limit. They define the j -th polytope as

the number of copies of I d and j · 1 d is a function of j . While this polytope-based method works well for log-barrier, it functions poorly for more intricate barriers, such as volumetric barrier and Lee-Sidford barrier. Take volumetric barrier as an example, recall that H vol ( x ) = A ⊤ Σ( S ( x ) -1 A ) S ( x ) -2 A where Σ( S ( x ) -1 A ) is the statistical leverage score [Drineas et al., 2006, Spielman and Srivastava, 2011]. Leverage score is a numerical quantity that measures how important a row of a matrix is, compared to other rows . This means that, if one duplicates a row infinitely many times, the row will be assigned a score of 0. More concretely, suppose we are given an n × d matrix with n identical rows, then the leverage score of each row is d n . Taking n →∞ , it's easy to see that all rows have score 0, and the Hessian of volumetric barrier will zero out the rows contributed to the

infinitely many copies of identities. Thus, for volumetric barrier, extra constraints of the polytope sequence vanish. On the other hand, the Mangoubi and Vishnoi [2023] construction heavily relies on the infinitely many occurrences of I d for it to converge to the regularized Hessian.

To circumvent this issue and provide a simpler argument, we develop a proof strategy that is barrierbased , instead of polytope-based. We example a class of popular barrier functions for polytopes that can be classified as weighted log-barrier functions [Vaidya, 1989]. Roughly speaking, these functions take in the form of g ( x ) = -∑ n i =1 w i log( S ( x ) i ) , with w = 1 n being the log-barrier, w = σ ( S ( x ) -1 A ) being the volumetric barrier [Vaidya, 1989] and if we choose w to be a proper power of the Lewis weights, then the barrier is precisely the Lee-Sidford barrier [Lee and Sidford, 2019, Laddha et al., 2020]. These weights are stable, meaning that if two points are close in certain local norm induced by proper ellipses given by these weights, then their weights are also close [Lee and Sidford, 2019]. This property proves to be particularly useful when proving the mixing rate of our Dikin walk. In addition, we carve out three sufficient conditions for a weighed log-barrier barrier function to work for log-concave distribution, and has ˜ O ( νd ) mixing rate:

ν -symmetry. The Hessian H ( x ) is ν -symmetry, that is, for any x ∈ K , E x (1) ⊆ K ∩ (2 x -K ) ⊆ E x ( √ ν ) where E x ( r ) = { y ∈ R d : ( y -x ) ⊤ H ( x )( y -x ) ≤ r 2 } . We show that ν = ν for log-barrier, volumetric barrier and Lee-Sidford barrier for polytopes, and log-barrier for spectrahedra.

Bounded local norm. The variance term ∥ H ( x ) -1 / 2 · ∇ log det( H ( x )) ∥ 2 2 ≤ d poly log n . We show this condition holds for all barriers of interest in this paper.

Convexity of regularized barrier. The function F ( x ) = log det( H ( x ) + I d ) is convex at x for any x ∈ K . We note that for log-barrier, volumetric barrier and Lee-Sidford barrier, it has been shown that log det( H ( x )) is indeed convex at x . We further prove that this still holds when a copy of identity is blended in.

We want to highlight that our approach is more robust and generic than that of Mangoubi and Vishnoi [2023], as it solely depends on the structure of the Hessian matrix. Our argument should be treated as a generalization of Sachdeva and Vishnoi [2016], in which they utilize the bounded local norm and convexity of log det( H ( x )) for log-barrier, together with the Gaussianity to conclude that the difference between log det are small. We go beyond log-barrier for polytopes and uniform sampling.

2.2 Approximation preserves the mixing time

The core premise of our algorithm is we allow the Hessian to be approximated, so that each step can be implemented efficiently. This poses significant challenges in proving mixing, as even if we have a good approximation of one-step in the Markov chain, the approximate chain does not necessarily mixes as fast as the original chain. For example, if every step is approximated within ϵ -TV distance to the original Markov chain, then in T steps, the TV distance between the resulting distribution under approximate walk and the distribution under original walk could be as large as Tϵ . This means we have to take ϵ very small ( O ( T -1 mix ) ) to guarantee same convergence property as original chain, which is unacceptable. A related issue is that the stationary distribution under approximate walk may not be the same as the target distribution. Therefore we need a way to control the mixing properties of the approximate walk. We resolve this problem by exactly computing the acceptance probability under approximate walk. That is, after proposing the next step z from x, ˜ Φ( x ) , we sample ̂ Φ( z ) , and accept with probability min { 1 , π ( z ) ̂ G z ( x ) π ( x ) ˜ G x ( z ) } (where ˜ G x ( z ) is the probability of proposing z starting from x under the approximate walk). Recall that in the exact walk, we accept with probability min { 1 , π ( z ) G z ( x ) π ( x ) G x ( z ) } (where G x ( z ) is the probability of proposing z stating from x under the exact walk). If we use this acceptability for the approximate walk, then we only have one-step approximation guarantee, and will suffer from the problem mentioned previously.

By using this modified acceptance probability, we can prove reversibility of the approximate walk, and that stationary distribution of the approximate walk is indeed our target distribution. For the mixing rate, we bound the conductance of the approximate walk, which requires us to prove the following properties:

The proposed step is accepted with decent probability.

Starting from two points moderately close to each other (i.e., ∥ x -z ∥ Φ( x ) ≤ ϵ ), the approximate steps taken P x and P z are close in TV distance (i.e., TV( P x , P z ) ≤ 1 -ϵ ′ ) for some absolute constants ϵ, ϵ ′ > 0 .

We are able to prove these properties by comparison with the original chain. That is, assuming the original chain satisfies properties (1)(2), then the approximate chain also satisfies the same properties, as long as the approximation is good enough. This enables us to prove these key properties in a hassle-free way. To prove these comparison results, we need to prove, for example, ˜ G x ( z ) ≈ G x ( z ) . Because both G x ( z ) and ˜ G x ( z ) have nice factorizations

we only need to prove det(Φ( x )) ≈ det( ˜ Φ( x )) and exp( -1 2 ∥ x -z ∥ 2 Φ( x ) ) ≈ exp( -1 2 ∥ x -z ∥ 2 ˜ Φ( x ) ) separately. Both of these can be handled via Φ( x ) ≈ ˜ Φ( x ) , using our approximation procedure.

2.3 Approximation of Lewis weights for rapid mixing

Given our robust and generic Dikin walk framework, we realize it with the Lee-Sidford barrier whose complexity ν = d log 5 n . This ensures the walk mixes in ˜ O ( d 2 + dL 2 R 2 ) steps, improving upon the log-barrier-based walk of Mangoubi and Vishnoi [2023, 2024] that mixes in ˜ O ( nd + dL 2 R 2 ) steps, as the log-barrier has complexity ν = n . To implement the Lee-Sidford barrier, it is imperative to compute the ℓ p (for p > 0 ) Lewis weights, which is defined as the following convex program:

and the ℓ p Lewis weights is a vector w p ∈ R d where ( w p ) i = a ⊤ i M ∗ a i and M ∗ is the optimum of Program (1). Solving the program exactly is not efficient, and a long line of works [Cohen and Peng, 2015, Lee and Sidford, 2014, 2019, Cohen et al., 2019, Fazel et al., 2022, Jambulapati et al., 2022] provide fast algorithms that approximate all weights up to (1 ± ϵ ) -factor. The Hessian of Lee-Sidford barrier has the form (up to scaling) H Lewis ( x ) = A ⊤ S ( x ) -1 W p ( S ( x ) -1 A ) 1 -2 /p S ( x ) -1 A where W p ( S ( x ) -1 A ) is the diagonal matrix of ℓ p Lewis weights with respect to S ( x ) -1 A . When clear from context, we use W p as a shorthand for W p ( S ( x ) -1 A ) . As W p could be approximated in ˜ O ( nd ω -1 ) time, we could then perform subsequent operations exactly to form the approximate Hessian. We instead provide a spectral approximation procedure that runs in ˜ O (nnz( A ) + T mat ( d, d 3 , d )) given the approximate Lewis weights where T mat ( a, b, c ) is the time of multiplying an a × b matrix with a b × c matrix and T mat ( d, d 3 , d ) ≈ d 4 . 2 . This is important as even though it doesn't lead to direct runtime improvement, if approximate Lewis weights can be computed in ˜ O (nnz( A ) + poly( d )) time, then we obtain a per iteration cost upgrade in a black-box fashion. Our approach is based on spectral sparsification via random sampling: given a matrix B ∈ R n × d , one can construct a matrix ˜ B ∈ R s × d consists of re-scaled rows of B , such that (1 -ϵ ) · B ⊤ B ⪯ ˜ B ⊤ ˜ B ⪯ (1 + ϵ ) · B ⊤ B . If we sample according to the leverage scores [Drineas et al., 2006, Spielman and Srivastava, 2011], one can set s = O ( ϵ -2 d log d ) . In our case, we set B = W 1 / 2 -1 /p p S ( x ) -1 A , and perform the sampling. However, computing leverage score exactly is as expensive as forming the Hessian exactly, which requires O ( nd ω -1 ) time. To speed up the crucial step, we use randomized sketching technique to reduce the row count from n to poly( d ) , then approximately estimate all leverage scores with this small matrix. One can either use the sparse embedding matrix [Nelson and Nguyên, 2013] to approximate these scores in time O (nnz( A ) log n + T mat ( d, ϵ -2 d, d )) , or use the subsampled randomized Hadamard transform [Lu et al., 2013] in time O ( nd log n + T mat ( d, ϵ -2 d, d )) . We remark that approximating leverage scores for subsampling has many applications in numerical linear algebra, such as low rank approximation [Boutsidis et al., 2016, Song et al., 2017, 2019]. Utilizing this framework, we provide an algorithm that approximates the Hessian of Lee-Sidford barrier in time ˜ O ( nd ω -1 ) + ˜ O (nnz( A ) + T mat ( d, d 3 , d )) , as advertised.

2.4 Approximation of log-barrier Hessian over a spectrahedron

For a spectrahedron induced by the dual semidefinite program, it is also natural to define its log-barrier with its corresponding Hessian being

where A ∈ R d × n 2 with the i -th row being vec( A i ) for n × n symmetric matrix A i , and S ( x ) = ∑ d i =1 x i A i -C where C is also an n × n symmetric matrix. Nesterov and Nemirovskii [1994] shows that by smartly arranging the organization of H log ( x ) , it can be computed exactly in time O ( dn ω + d ω -1 n 2 ) . Beyond exact computation, Jiang et al. [2020a], Huang et al. [2022] present algorithms that approximately maintain the Hessian matrix under low-rank updates. These maintenance approaches are crafted towards solving semidefinite programs in which the trajectory can be carefully controlled. For our soft-threshold Dikin walk, though the proposal generates a point that is relatively close to the starting point, due to its Gaussianity nature, much less structure can be exploited and thus maintained. Instead, we propose a maintenance-free approach that uses randomized sketching to generate a spectral approximation of H log ( x ) .

To illustrate the algorithm, let's first define a matrix

It is not hard to see that H log ( x ) = BB ⊤ . Due to the Kronecker product structure of ( S ( x ) -1 / 2 ⊗ S ( x ) -1 / 2 ) , it is natural to consider sketches for Kronecker product of matrices [Diao et al., 2018, 2019, Ahle et al., 2020, Song et al., 2021]. Following the standard procedure for using sketch to generate spectral approximation, we can choose a TensorSRHT matrix [Ahle et al., 2020, Song et al., 2021] T and compute R := ( S ( x ) -1 / 2 ⊗ S ( x ) -1 / 2 ) T then form A R . This is unfortunately, not efficient enough, as multiplying A with R might be too slow. To further optimize the runtime efficiency, we use a more intrinsic approach based on the matrix T and B . It is well-known that the i -th row of B can be first computed as S ( x ) -1 / 2 A i S ( x ) -1 / 2 then vectorize. If we can manage to apply the sketch in a row-by-row fashion, then fast matrix multiplication can be utilized for even faster algorithms. We recall that T = P · ( HD 1 ⊗ HD 2 ) where H is the Hadamard matrix and P is a row sampling matrix. To compute a row, we can first apply HD 1 and HD 2 individually to S ( x ) -1 / 2 in ˜ O ( n 2 ) time since H is a Hadamard matrix. We can then form two matrices X and Y with rows being the corresponding sampled rows from HD 1 and HD 2 . Finally, we compute XA i Y ⊤ to form one row. Using fast matrix multiplication, we can form each row in T mat ( d 3 , n, n ) = O ( n 2 d 3( ω -2) ) time. Applying this procedure to d rows leads to an overall O ( n 2 d 3 ω -5 ) time, which beats the O ( dn ω ) time as long as n ≫ d . Our runtime is somewhat slow when n is not that larger than d , this is due to we have to set ϵ = O (1 /d ) . For more popular regimes where it suffices to set ϵ = O (1) , our algorithm outperforms exact computation as long as n ≥ d .

3 Applications

There are many applications of our Dikin walk, such as differentially private learning, simulated annealing and regret minimization. For a more comprehensive overview of the applications, we refer readers to Narayanan and Rakhlin [2017].

Differentially private learning. Let X m = { x 1 , . . . , x m } denote an m -point dataset, associate a convex loss function ℓ i to point x i , the learning problem attempts to find an optimal parameter θ ∗ ∈ K that minimizes ℓ ( θ ) = ∑ m i =1 ℓ i ( θ ) . We could further enforce privacy constraints to learning: we say a randomized algorithm M : X m → R is ϵ -DP if for any datasets X,X ′ ∈ X m differ by a single point and for any S ⊆ R , Pr[ M ( X ) ⊆ S ] ≤ e ϵ Pr[ M ( X ′ ) ⊆ S ] . The differentially private learning seeks to solve the learning problem under DP guarantees, and it has been shown that if one allows for approximate DP , then it can be achieved via the exponential mechansim and sampling from the distribution π ( θ ) ∝ e -ℓ ( θ ) [Wang et al., 2015]. If K is a polytope or spectrahedron, and in addition ℓ is L -Lipschitz, one can implement our walk to obtain an ˜ O ( d 2 + dL 2 R 2 ) mixing for polytopes or ˜ O ( nd + dL 2 R 2 ) mixing for spectrahedra. Via a standard reduction from TV distance bound to infinity distance bound [Mangoubi and Vishnoi, 2022], our walk can also be adapted for a pure DP guarantee [Lin et al., 2024].

Simulated annealing for convex optimization. Let f : R d → R be an L -Lipschitz and convex function, we consider the problem of minimizing f over K when we can only access the function value of f . This is also referred to as zeroth-order convex optimization. One could solve the problem via the simulated annealing framework , in which one needs to sample from the distribution π ( x ) ∝ e -f ( x ) /T where T is the temperature parameter [Kalai and Vempala, 2006]. Our walk can be adapted to the framework for polytopes, it mixes in ˜ O ( d 2 . 5 + d 1 . 5 L 2 R 2 ) steps and for spectrahedra, it mixes in ˜ O ( nd 1 . 5 + d 1 . 5 L 2 R 2 ) steps, improving upon prior best algorithms that mix in min { d 5 . 5 + d 3 . 5 L 2 R 2 , nd 1 . 5 + d 1 . 5 L 2 R 2 } steps for polytopes and n 2 d 3 . 5 + n 2 d 1 . 5 L 2 R 2 steps for spectrahedra [Narayanan and Rakhlin, 2017, Mangoubi and Vishnoi, 2023].

Online convex optimization. Consider the following online optimization problem: let K be a convex set that we could choose our actions from and let ℓ 1 , . . . , ℓ T be a sequence of unknown convex cost functions with ℓ t : K → R . The goal is to design a good strategy that chooses from K so that the total cost is small, compared to the offline optimal cost, in which one could choose the strategy after seeing the entire sequence of ℓ t 's. The algorithm works as follows: at round t , we choose a distribution µ t -1 supported on K and play the action Y t ∼ µ t -1 . Our goal is to minimize the expected regret, defined as Reg T ( U ) = E [ ∑ T t =1 ℓ t ( Y t ) -∑ T t =1 ℓ t ( U )] with respect to all randomized strategies defined by distribution p U . It turns out if one sets s t ( x ) = η ∑ t s =1 ℓ t ( x ) where η > 0 is a step size parameter and sets µ t ∝ e -s t , then this update rule reflects the multiplicative weights update algorithm [Arora et al., 2012]. Moreover, one could prove a nearly-optimal regret bound of O ( √ T ) if the KL divergence between µ 0 and all possible p U 's are bounded. The algorithmic problem of sampling from µ t is equivalent to the log-concave sampling problem we study in this paper, and we could efficiently generate strategies with good regrest, similar to Narayanan and Rakhlin [2017]. Let ℓ t 's be 1-Lipschitz linear functions, Narayanan and Rakhlin [2017] uses their time-varying Dikin walk to achieve a regret of O ( d 2 . 5 √ T ) , while we could significantly improve this bound to O ( d √ T ) .

4 Conclusion

In this paper, we design a class of error-robust Dikin walks for sampling from a log-concave and log-Lipschitz density over a convex body. The key features of our walks are that their mixing time depends linearly on the complexity of self-concordant barrier function of the convex body, and they allow computationally intensive quantities to be approximated rather than computed exactly. For polytopes, our walk mixes in ˜ O ( d 2 + dL 2 R 2 ) steps with a per iteration cost of ˜ O ( nd ω -1 ) , and for spectrahedra, our walk mixes in ˜ O ( nd + dL 2 R 2 ) steps with a per iteration cost of ˜ O ( n ω + n 2 d 3 ω -5 ) . For polytopes, our walk is the first successful adaptation of the Lee-Sidford barrier for log-concave sampling under the minimal assumption that f is Lipschitz, improving upon the mixing of prior works [Narayanan and Rakhlin, 2017, Mangoubi and Vishnoi, 2023, 2024]. For spectrahedra, we improve the mixing of Narayanan and Rakhlin [2017] from n 2 d 3 + n 2 dL 2 R 2 to nd + dL 2 R 2 , for the term that depends on only n and d , we obtain an improvement of nd 2 , and for the term that depends on L and R , we shave off the quadratic dependence on n . Moreover, we adapt our error-robust framework and present a sketching algorithm for approximating the Hessian matrix in ˜ O ( n ω + n 2 d 3 ω -5 ) time. Our results have deep implications, as it could be leveraged for differentially private learning, convex optimization and regret minimization.

While our framework offers the fastest mixing Dikin walk for log-concave sampling, its mixing time has a rather unsatisfactory dependence on the radius of the bounding box, R . It would be interesting if one could design a walk that does not depend on R . We also note that we require the density of be log-Lipschitz rather than Lipschitz. Kook and Vempala [2024] shows that if f in addition satisfies the relative strongly-convex property , then there exists a walk whose mixing does not depend on R and one only needs the density of be Lipschitz. Another important direction is to further improve the mixing rate of sampling over a spectrahedron using barrier functions such as volumetric barrier and hybrid barrier [Anstreicher, 2000], while maintaining an small per iteration cost. Finally, extending RHMC to log-concave sampling would be essentially, as it has the potential to give the fastest mixing walk by utilizing Riemannian metrics instead of Euclidean. As our work is theoretical in nature, we don't foresee any potential negative societal impact. It has the potential positive impact to reduce energy consumption and carbon emission when deployed in practice.

References

Thomas D Ahle, Michael Kapralov, Jakob BT Knudsen, Rasmus Pagh, Ameya Velingker, David P Woodruff, and Amir Zandieh. Oblivious sketching of high-degree polynomial kernels. In Proceedings of the Fourteenth Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 141-160, 2020.

Josh Alman and Virginia Vassilevska Williams. A refined laser method and faster matrix multiplication. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 522-539. SIAM, 2021.

Kurt M Anstreicher. The volumetric barrier for semidefinite programming. Mathematics of Operations Research , 25(3):365-380, 2000.

Sanjeev Arora, Elad Hazan, and Satyen Kale. The multiplicative weights update method: a metaalgorithm and applications. Theory of Computing , 8(1):121-164, 2012.

Christos Boutsidis, David P Woodruff, and Peilin Zhong. Optimal principal component analysis in distributed and streaming models. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing (STOC) , pages 236-249, 2016.

Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. Advances in Neural Information Processing Systems , 31, 2018.

Niladri Chatterji, Nicolas Flammarion, Yian Ma, Peter Bartlett, and Michael Jordan. On the theory of variance reduction for stochastic gradient monte carlo. In International Conference on Machine Learning , pages 764-773. PMLR, 2018.

Yuansi Chen and Ronen Eldan. Hit-and-run mixing via localization schemes, 2022.

Yuansi Chen, Raaz Dwivedi, Martin J. Wainwright, and Bin Yu. Fast mcmc sampling algorithms on polytopes. J. Mach. Learn. Res. , 2018.

Michael B. Cohen and Richard Peng. Lp row sampling by lewis weights. In Proceedings of the Forty-Seventh Annual ACM Symposium on Theory of Computing , STOC '15, 2015.

Michael B Cohen, Ben Cousins, Yin Tat Lee, and Xin Yang. A near-optimal algorithm for approximating the john ellipsoid. In Conference on Learning Theory , pages 849-873. PMLR, 2019.

Arnak S Dalalyan and Avetik Karagulyan. User-friendly guarantees for the langevin monte carlo with inaccurate gradient. Stochastic Processes and their Applications , 129(12):5278-5311, 2019.

Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems , 27, 2014.

Huaian Diao, Zhao Song, Wen Sun, and David Woodruff. Sketching for kronecker product regression and p-splines. In International Conference on Artificial Intelligence and Statistics , pages 12991308. PMLR, 2018.

Huaian Diao, Rajesh Jayaram, Zhao Song, Wen Sun, and David Woodruff. Optimal sketching for kronecker product regression and low rank approximation. Advances in neural information processing systems , 32:4737-4748, 2019.

Zhiyan Ding and Qin Li. Langevin monte carlo: random coordinate descent and variance reduction. J. Mach. Learn. Res. , 22:205-1, 2021.

Petros Drineas, Michael W. Mahoney, and S. Muthukrishnan. Sampling algorithms for l2 regression and applications. In Proceedings of the Seventeenth Annual ACM-SIAM Symposium on Discrete Algorithm , SODA '06, page 1127-1136, USA, 2006. Society for Industrial and Applied Mathematics.

Ran Duan, Hongxun Wu, and Renfei Zhou. Faster matrix multiplication via asymmetric hashing. In FOCS , 2023.

Kumar Avinava Dubey, Sashank J Reddi, Sinead A Williamson, Barnabas Poczos, Alexander J Smola, and Eric P Xing. Variance reduction in stochastic gradient langevin dynamics. Advances in neural information processing systems , 29, 2016.

Maryam Fazel, Yin Tat Lee, Swati Padmanabhan, and Aaron Sidford. Computing lewis weights to high precision. In Proceedings of the 2022 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 2723-2742. SIAM, 2022.

Z Furedi and I Barany. Computing the volume is difficult. In Proceedings of the Eighteenth Annual ACM Symposium on Theory of Computing , STOC '86, page 442-447, New York, NY, USA, 1986. Association for Computing Machinery.

François Le Gall and Florent Urrutia. Improved rectangular matrix multiplication using powers of the coppersmith-winograd tensor. In Proceedings of the Twenty-Ninth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA '18, page 1029-1046, 2018.

Francois Le Gall. Faster rectangular matrix multiplication by combination loss analysis. In Proceedings of the Thirty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA'24, 2024.

Khashayar Gatmiry, Jonathan Kelner, and Santosh S. Vempala. Sampling polytopes with riemannian hmc: Faster mixing via the lewis weights barrier. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 1796-1881. PMLR, 30 Jun-03 Jul 2024.

Roger A. Horn and Charles R. Johnson. Matrix Analysis . Cambridge University Press, 1990.

Baihe Huang, Shunhua Jiang, Zhao Song, Runzhou Tao, and Ruizhe Zhang. Solving sdp faster: A robust ipm framework and efficient implementation. In FOCS , 2022.

Arun Jambulapati, Yang P. Liu, and Aaron Sidford. Improved iteration complexities for overconstrained p-norm regression. In Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing , STOC 2022, 2022.

Haotian Jiang, Tarun Kathuria, Yin Tat Lee, Swati Padmanabhan, and Zhao Song. A faster interior point method for semidefinite programming. In 2020 IEEE 61st annual symposium on foundations of computer science (FOCS) , pages 910-918. IEEE, 2020a.

Haotian Jiang, Yin Tat Lee, Zhao Song, and Sam Chiu-wai Wong. An improved cutting plane method for convex optimization, convex-concave games, and its applications. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 944-953, 2020b.

Shunhua Jiang, Zhao Song, Omri Weinstein, and Hengjie Zhang. A faster algorithm for solving general lps. In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing , pages 823-832, 2021.

Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems , 26, 2013.

William B Johnson and Joram Lindenstrauss. Extensions of lipschitz mappings into a hilbert space. Contemporary mathematics , 26(189-206):1, 1984.

Adam Tauman Kalai and Santosh Vempala. Simulated annealing for convex optimization. Mathematics of Operations Research , 31(2):253-266, 2006.

Kannan, L. Lovász, and M. Simonovits. Isoperimetric problems for convex bodies and a localization lemma. Discrete Comput. Geom. , 1995.

Ravindran Kannan and Hariharan Narayanan. Random walks on polytopes and an affine interior point method for linear programming. Mathematics of Operations Research , 37(1):1-20, 2012.

Yunbum Kook and Santosh S. Vempala. Gaussian cooling and Dikin walks: The interior-point method for logconcave sampling. In Shipra Agrawal and Aaron Roth, editors, Proceedings of Thirty Seventh Conference on Learning Theory , volume 247 of Proceedings of Machine Learning Research , pages 3137-3240. PMLR, 30 Jun-03 Jul 2024.

Aditi Laddha, Yin Tat Lee, and Santosh Vempala. Strong self-concordance and sampling. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing (STOC) , pages 1212-1222, 2020.

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , pages 1302-1338, 2000.

Yin Tat Lee and Aaron Sidford. Path finding methods for linear programming: Solving linear programs in õ(sqrt(rank)) iterations and faster algorithms for maximum flow. In 55th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2014, Philadelphia, PA, USA, October 18-21, 2014 , pages 424-433, 2014.

Yin Tat Lee and Aaron Sidford. Solving linear programs with sqrt (rank) linear system solves. arXiv preprint arXiv:1910.08033 , 2019.

Yin Tat Lee and Santosh S. Vempala. Geodesic walks in polytopes. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , STOC 2017, 2017.

Yin Tat Lee and Santosh S Vempala. Convergence rate of riemannian hamiltonian monte carlo and faster polytope volume computation. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pages 1115-1121, 2018.

Yin Tat Lee and Man-Chung Yue. Universal barrier is n -self-concordant. Mathematics of Operations Research , 46(3):1129-1148, 2021.

Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In Foundations of Computer Science (FOCS), 2015 IEEE 56th Annual Symposium on , pages 1049-1065. IEEE, 2015.

Yin Tat Lee, Zhao Song, and Qiuyi Zhang. Solving empirical risk minimization in the current matrix multiplication time. In Conference on Learning Theory , pages 2140-2157. PMLR, 2019.

Zhize Li, Tianyi Zhang, Shuyu Cheng, Jun Zhu, and Jian Li. Stochastic gradient hamiltonian monte carlo with variance reduction for bayesian inference. Machine Learning , 108(8):1701-1727, 2019.

Yingyu Lin, Yian Ma, Yu-Xiang Wang, Rachel Emily Redberg, and Zhiqi Bu. Tractable MCMC for private learning with pure and gaussian differential privacy. In The Twelfth International Conference on Learning Representations , 2024.

László Lovász and Miklós Simonovits. Random walks in a convex body and an improved volume algorithm. Random structures & algorithms , 4(4):359-412, 1993.

László Lovász and Santosh Vempala. Hit-and-run is fast and fun. preprint, Microsoft Research , 2003.

László Lovász and Santosh Vempala. Simulated annealing in convex bodies and an O ∗ ( n 4 ) volume algorithm. Journal of Computer and System Sciences , 72(2):392-417, 2006.

László Lovász and Santosh Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms , 30(3):307-358, 2007.

Yichao Lu, Paramveer Dhillon, Dean P Foster, and Lyle Ungar. Faster ridge regression via the subsampled randomized hadamard transform. In Advances in neural information processing systems , pages 369-377, 2013.

Oren Mangoubi and Nisheeth Vishnoi. Sampling from log-concave distributions with infinity-distance guarantees. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 12633-12646. Curran Associates, Inc., 2022.

Oren Mangoubi and Nisheeth K Vishnoi. Sampling from structured log-concave distributions via a soft-threshold dikin walk. In Advances in Neural Information Processing Systems , NeurIPS'23, 2023.

Oren Mangoubi and Nisheeth K. Vishnoi. Faster sampling from log-concave densities over polytopes via efficient linear solvers. In The Twelfth International Conference on Learning Representations , 2024.

Hariharan Narayanan. Randomized interior point methods for sampling and optimization. Ann. Appl. Probab. , 2016.

Hariharan Narayanan and Alexander Rakhlin. Efficient sampling from time-varying log-concave distributions. The Journal of Machine Learning Research , 18(1):4017-4045, 2017.

Jelani Nelson and Huy L Nguyên. OSNAP: Faster numerical linear algebra algorithms via sparser subspace embeddings. In 54th Annual IEEE Symposium on Foundations of Computer Science (FOCS) , pages 117-126. IEEE, 2013.

Yurii Nesterov and Arkadii Nemirovskii. Self-concordant functions and polynomial-time methods in convex programming. USSR Academy of Sciences , 1989.

Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994.

Lianke Qin, Zhao Song, Lichen Zhang, and Danyang Zhuo. An online and unified algorithm for projection matrix vector multiplication with application to empirical risk minimization. In AISTATS , 2023.

Maxim Raginsky, Alexander Rakhlin, and Matus Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis. In Conference on Learning Theory , pages 1674-1703. PMLR, 2017.

James Renegar. A polynomial-time algorithm, based on newton's method, for linear programming. Math. Program. , 1988.

Sushant Sachdeva and Nisheeth K Vishnoi. The mixing time of the dikin walk in a polytope-a simple proof. Operations Research Letters , 44(5):630-634, 2016.

Shai Shalev-Shwartz and Tong Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. Journal of Machine Learning Research , 14(1), 2013.

Zhao Song and Zheng Yu. Oblivious sketching-based central path method for solving linear programming problems. In 38th International Conference on Machine Learning (ICML) , 2021.

Zhao Song, David P Woodruff, and Peilin Zhong. Low rank approximation with entrywise l1-norm error. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 688-701, 2017.

Zhao Song, David P Woodruff, and Peilin Zhong. Relative error tensor low rank approximation. In Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms , pages 2772-2789. SIAM, 2019.

Zhao Song, David Woodruff, Zheng Yu, and Lichen Zhang. Fast sketching of polynomial kernels of polynomial degree. In International Conference on Machine Learning , pages 9812-9823. PMLR, 2021.

Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing , 40(6):1913-1926, 2011.

Joel A Tropp. Improved analysis of the subsampled randomized hadamard transform. Advances in Adaptive Data Analysis , 3(01n02):115-126, 2011.

Pravin M Vaidya. A new algorithm for minimizing convex functions over convex sets. In 30th Annual Symposium on Foundations of Computer Science (FOCS) , pages 338-343. IEEE Computer Society, 1989.

Santosh Vempala. Geometric random walks: a survey. Combinatorial and computational geometry , 52(573-612):2, 2005.

Yu-Xiang Wang, Stephen Fienberg, and Alex Smola. Privacy for free: Posterior sampling and stochastic gradient monte carlo. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2493-2502, Lille, France, 07-09 Jul 2015. PMLR.

Virginia Vassilevska Williams. Multiplying matrices faster than coppersmith-winograd. In Proceedings of the forty-fourth annual ACM symposium on Theory of computing (STOC) , pages 887-898. ACM, 2012.

Virginia Vassilevska Williams, Yinzhan Xu, Zixuan Xu, and Renfei Zhou. New bounds for matrix multiplication: from alpha to omega. In Proceedings of the Thirty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms , SODA'24, 2024.

David P. Woodruff. Sketching as a tool for numerical linear algebra. Foundations and Trends in Theoretical Computer Science , 10(1-2):1-157, 2014.

Difan Zou, Pan Xu, and Quanquan Gu. Stochastic variance-reduced hamilton monte carlo methods. In International Conference on Machine Learning , pages 6028-6037. PMLR, 2018.

Appendix

Roadmap. Since there are many technical details in the appendix, we provide a roadmap. The appendix can be divided into 5 parts: the first part conveys some preliminary information and states the sufficient conditions we require barrier functions to have, in Section A and B. The second part proves the mixing time of the Dikin walk when barrier functions satisfy the conditions, and the proofs are divided into Section C and D. The next part focuses on the runtime complexity of sampling from polytopes, including how to generate a spectral approximation of the Hessian, approximate leverage scores and Lewis weights to high precision, in Section E and how to incorporate these algorithmic prototypes to implement an efficient sampling algorithm in Section F. We dedicate Section G to study the log-barrier for sampling from a spectrahedron, as it is relatively less explored before. Finally, we prove the convexity of the function log det( H ( x ) + I d ) for log-barrier, volumetric barrier and in extension Lee-Sidford barrier in Section H and I.

A Preliminaries

For any positive integer, we use [ n ] to denote the set { 1 , 2 , · · · , n } . For a vector x ∈ R n , we use ∥ x ∥ 2 to denote its ℓ 2 norm, i.e., ∥ x ∥ 2 := ( ∑ n i =1 x 2 i ) 1 / 2 . We use ∥ x ∥ 1 to denote its ℓ 1 norm, ∥ x ∥ 1 := ∑ n i =1 | x i | . We use ∥ x ∥ ∞ to denote its ℓ ∞ norm, i.e., ∥ x ∥ ∞ := max i ∈ [ n ] | x i | . For a random variable X , we use E [ X ] to denote its expectation. We use Pr[ · ] to denote probability.

We use 0 d to denote a lengthd vector where every entry is 0 . We use 1 d to denote a lengthd vector where every entry is 1 . We use I d to denote an identity matrix which has size d × d or simply I when dimension is clear from context.

For a matrix A , we use A ⊤ to denote the transpose of matrix A . For a square and non-singular matrix A , we use A -1 to denote the inverse of matrix A . For a real square matrix A , we say it is positive definite (PD, i.e., A ≻ 0 ) if for all vectors x ∈ R n (except for 0 n ), we have x ⊤ Ax > 0 . For a real square matrix A , we say it is positive semi-definite(PSD, i.e., A ⪰ 0 ) if for all vectors x ∈ R n , we have x ⊤ Ax ≥ 0 . For a square matrix, we use det( A ) to denote the determinant of matrix A . For a matrix A , we use ∥ A ∥ to denote its spectral norm, use ∥ A ∥ F to denote its Frobenius norm, i.e., ∥ A ∥ F := ( ∑ n i =1 ∑ d j =1 A 2 i,j ) 1 / 2 and use nnz( A ) to denote the number of nonzero entries in A .

Given a function f : K → R , we say it's convex if for any x, y ∈ K , f ( x ) ≥ f ( y ) + ∇ f ( y ) ⊤ ( x -y ) . We say it's L -Lipschitz if | f ( x ) -f ( y ) | ≤ L · ∥ x -y ∥ 2 for a fixed parameter L > 0 .

For two distributions P 1 and P 2 , we use TV( P 1 , P 2 ) to denote the total variation (TV) distance between P 1 and P 2 .

For a convex body K ⊆ R d , we use Int( K ) to denote the interior of K .

We use ⊗ to denote Kronecker product. We use A ⊗ S B to denote A ⊗ I + I ⊗ B . Given a matrix A , we use vec( A ) to denote its vectorization, since we will always apply vec( · ) to a symmetric matrix, whether the vetorization is row-major or column-major doesn't matter. We use ◦ to denote Hadamard or element-wise product.

For two vectors x, y ∈ R d , we use ⟨ x, y ⟩ = x ⊤ y to denote the standard Euclidean inner product over R d , and for two symmetric matrices A,B ∈ R d × d , we use ⟨ A,B ⟩ = tr[ A ⊤ B ] to denote the trace inner product.

In Section A.1, we present several definitions and notations related to fast matrix multiplication. In Section A.2, we provide some backgrounds about convex geometry. In Section A.3, we state several basic probability tools. In Section A.4, we state several basic facts related to trace, and Kronecker product. In Section A.5, we present the standard notion about self-concordance barrier.

A.1 Fast matrix multiplication

Definition A.1 (Fast matrix multiplication) . Given three positive integers a, b, c , we use T mat ( a, b, c ) to denote the time of multiplying an a × b matrix with another b × c matrix.

For convenience, we also define the ω ( · , · , · ) function as follows:

Definition A.2 (Fast matrix multiplication, an alternative notation) . Given x, y, z , we use d ω ( x,y,z ) to denote the time of multiplying a d x × d y matrix with another d y × d z matrix.

Lemma A.3 (Gall and Urrutia [2018]) . We have the following bounds of ω ( · , · , · ) :

ω (1 , 1 , 1) = ω ,

ω (1 , 3 , 1) = 4 . 199712 (see Table 3 in Gall and Urrutia [2018]).

Here, ω denotes the exponent of matrix multiplication, currently ω ≈ 2 . 373 [Williams, 2012, Gall and Urrutia, 2018, Alman and Williams, 2021, Duan et al., 2023, Williams et al., 2024, Gall, 2024].

A.2 Convex geometry

We define B ( x, R ) ⊂ R d as

Define polytope K as

Define spectrahedron K as

A.3 Probability tools

We state some useful concentration inequalities.

where A 1 , . . . , A d , C ∈ R n × n are symmetric matrices.

We will often use the notion of a Dikin ellipsoid :

Definition A.4 (Dikin ellipsoid) . We define the Dikin ellipsoid D θ ⊂ R d as follows

where H ( θ ) ∈ R d × d is the Hessian of self-concordant barrier at θ /

We define the standard cross-ratio distance (see Definition E.1 in Mangoubi and Vishnoi [2023] for example).

̸

Definition A.5 (Cross-ratio distance) . Let u, v ∈ K . If u = v , let p, q be two endpoints of the chord in K which passes through u and v such that the four points lie in the order of p, u, v, q , let

We additionally set σ ( u, v ) = 0 for u = v .

For two subsets S 1 , S 2 ⊆ K , we define

We state the standard isoperimetric inequality for cross-ratio distance.

Lemma A.6 (Lovász and Vempala [2003]) . Let π : R d → R be a log-concave density, with support on a convex body K . Then for any partition of R d into measurable sets S 1 , S 2 , S 3 , the induced measure π ∗ satisfies

Lemma A.7 (Lemma 1 on page 1325 of Laurent and Massart [2000]) . Let X ∼ X 2 k be a chi-squared distributed random variable with k degrees of freedom. Each random variable has zero mean and σ 2 variance. Then

Lemma A.8 (Matrix Chernoff bound [Tropp, 2011]) . Let X 1 , . . . , X s be independent copies of a symmetric random matrix X ∈ R d × d with E [ X ] = 0 , ∥ X ∥ ≤ γ almost surely and ∥ E [ X ⊤ X ] ∥ ≤ σ 2 . Let W = 1 s ∑ i ∈ [ s ] X i . For any ϵ ∈ (0 , 1) ,

A.4 Basic algebra facts

Lemma A.9. Let A and B be m × m matrices. Then we have

tr[ AB ] = tr[ BA ] .

If A is symmetric, then tr[ AB ] = tr[ AB ⊤ ] .

If A and B are PSD, then ⟨ A,B ⟩ ≥ 0 , and ⟨ A,B ⟩ = 0 if and only if AB = 0 m × m .

If A ⪰ 0 and B ⪰ C , then ⟨ A,B ⟩ ≥ ⟨ A,C ⟩ .

Lemma A.10. Let A,B,C,D be conforming matrices. Then

( A ⊗ B )( C ⊗ D ) = AC ⊗ BD .

( A ⊗ S B )( C ⊗ S D ) = 1 2 ( AC ⊗ S BD + AD ⊗ S BC ) .

( A ⊗ B ) ⊤ = A ⊤ ⊗ B ⊤ .

If A and B are non-singular, then A ⊗ B is non-singular, and ( A ⊗ B ) -1 = A -1 ⊗ B -1 .

vec( ABC ) = ( C ⊤ ⊗ A ) · vec( B ) .

A.5 Self-concordant barrier

We provide a standard definition about self-concordance barrier,

Definition A.11 (Self-concordant barrier) . A real-valued function F : Int( K ) → R , is a regular self-concordant barrier if it satisfies the conditions stated below. For convenience, if x ̸∈ Int( K ) , we define F ( x ) = ∞ .

(Convex, Smooth) F is a convex thrice continuously differentiable function on Int( K ) .

(Barrier) For every sequence of points { x i } ∈ Int( K ) converging to a point x ̸∈ Int( K ) , lim i →∞ f ( x i ) = ∞ .

(Differential Inequalities) For all h ∈ R d and all x ∈ Int( K ) , the following inequalities hold.

(a) D 2 F ( x )[ h, h ] is 2 -Lipschitz continuous with respect to the local norm, which is equivalent to

(b) F ( x ) is ν -Lipschitz continuous with respect to the local norm defined by F ,

We call the smallest positive integer ν for which this holds, the self-concordance parameter of the barrier.

B Sufficient Conditions for Log-concave Sampling

To prove that our algorithm works for log-concave distribution, we state three key assumptions (see Section B.1) which are the sufficient conditions to prove the mixing rate of log-concave distribution, for different barrier functions.

B.1 Conditions: ν -symmetry, convexity and bounded local norm

We state three key conditions in order to lower bound the conductance for log-concave sampling.

Assumption B.1. Given a convex body K , let H : K → R d × d be a self-concordant matrix function, we assume that

(i) ν -symmetry. For any x ∈ K , we have E x (1) ⊆ K ∩ (2 x - K ) ⊆ E x ( √ ν ) where E x ( r ) = { y ∈ R d : ( y -x ) ⊤ H ( x )( y -x ) ≤ r 2 } .

(ii) Convexity. Let F : K → R be defined as F ( x ) = log(det( H ( x ) + I d )) , then F is convex in x .

(iii) Bounded local norm. Let ∇ log(det( H ( x ))) denote the gradient of log(det( H ( x ))) in x . For any x ∈ K , we have

For the ν -symmetry assumption, we will prove that for barriers of the concern (log-barrier, LeeSidford barrier for polytopes and log-barrier for spectrahedra), ν = ν . This characteristic is proved in Laddha et al. [2020] for polytopes, and we prove that for spectrahedra in Section G. For convexity, due to the large amount of calculations, we defer to Section H and I. For bounded local norm, the results for polytopes are similarly shown in Laddha et al. [2020], and we will prove for log-barrier over spectrahedra.

B.2 Local PSD approximation for any self-concordant matrix function

In this section, we prove a generalization of the matrix function self-concordance to regularized matrix function. When H is Hessian of the log barrier, this fact was proved in [Mangoubi and Vishnoi, 2023, Lemma E.3]. We generalize this fact to general barriers.

Lemma B.2. Let α ∈ (0 , 1) . Let H : K → R be a self-concordant matrix function and Φ( u ) := α -1 H ( u ) + η -1 I d . For any u, v ∈ K such that ∥ u -v ∥ Φ( u ) ≤ 1 α 1 / 2 we have

Proof. By [Laddha et al., 2020, Lemma 1.1], we have

Because H ( u ) ⪯ α · Φ( u ) , we get

Using Φ( u ) = α -1 H ( u ) + η -1 I d again we get

This completes the proof.

B.3 Bounded local norm for regularized Hessian function

We note that the bounded local norm condition generally holds for standard barrier functions, but in our core argument, we will instead rely on the bounded local norm of the regularized Hessian function . We prove that the bounded local norm condition implies the bounded local norm on the regularized Hessian function.

Lemma B.3. Let H : K → R d × d be a matrix function, define F ( x ) = log det( H ( x ) + L 2 I d ) for L ∈ R , then we have the following inequality:

Proof. Let H ( x ) = U Λ U ⊤ be the eigendecomposition of the matrix H ( x ) , let y = U ⊤ ∇ H ( x ) , then

the given condition implies

As L 2 ≥ 0 , we can expand the regularized squared ℓ 2 norm as

This completes the proof.

C Key Tools for Robust Sampling

In this section, we list several tools which can be shared in the core proofs of all the main theorems. This section is organized as follows. In Section C.1, we provide a tool for Gaussian concentration. In Section C.2, we show that PSD approximation implies the determinant approximation (by losing a factor of d ). One of the major idea in this work is instead of using exact accept probability, we will only need to use approximate accept probability. In Section C.3, we prove several useful robust properties for approximate accept probability. In Section C.4, we present a general lemma for bounding the TV distance between exact process and approximate process. In Section C.5, we provide some specific choices for parameters and then bound the TV distance. In Section C.6, we provide a definition which will be used in proof of lower bound on conductance. In Section C.7, we show how to lower bound the conductance by cross ratio.

C.1 Gaussian concentration lemma

Lemma C.1. Let ξ ∼ N (0 , I d ) , then for any t > √ 2 d , we have

Proof. We consider the squared ℓ 2 norm of ξ : ∥ ξ ∥ 2 2 = ∑ d i =1 ξ 2 i , which is a χ 2 distribution with degree of freedom d . By Lemma A.7, we know that

set k 2 = ( t 2 -d ) / 8 , we obtain the desired bound.

C.2 Spectral approximation to determinant approximation

Lemma C.2. Let ϵ Φ ∈ (0 , 1) . Given four psd matrices satify the following conditions

Then we have

Proof. Since ˜ Φ( x ) ∈ (1 ± ϵ Φ )Φ( x ) , we have the following spectral bound:

we can take the determinant of the middle term:

Thus, we complete the proof.

C.3 Approximate accept probability

We start with some definitions:

Definition C.3. We define

Definition C.4. We define

We will use P x , ˜ P x as a shorthand for the above notations, and we call P x the exact process and ˜ P x the approximate process .

Lemma C.5. Fix x and ˜ Φ( x ) . Suppose ˜ Φ( x ) is ( ϵ H , δ H ) -good approximation to Φ( x ) . Then

Proof. Note that

Because ˜ Φ( x ) is a good approximation to Φ( x ) , by Lemma C.2 we have

By choosing step size sufficiently small, we have ∥ u -x ∥ 2 Φ( x ) < d with probability 0 . 9999 . Therefore

Then

For ϵ H d small enough, we have

and and therefore

So

This completes the proof.

C.4 TV distance between exact process and approximate process

LemmaC.6 (TV distance between exact process and approximate process) . Let ˜ Φ denote the (1 ± ϵ H ) approximation of Φ . Let δ H denote the failure probability. Let ϵ p be defined as Lemma C.5. Let P x denote the exact process. Let ˜ P x denote the approximate process. If x ∈ K , then

Proof. We say ˜ Φ( x ) is good if it is an ( ϵ H , δ H ) -approximation to Φ( x ) , and ˜ Φ( x ) is bad otherwise. We can upper bound TV( P x , ˜ P x ) as follows:

where the third step follows from TV( · , · ) ≤ 1 , the last step follows from Lemma E.3.

It remains to compute the second term in the above equation. By Pinsker's inequality, when ˜ Φ( x ) is good, we have

where the first step follows from Pinsker inequality.

Now, we can upper bound TV( P x , ˜ P x ) in the following sense,

The forth step follows from definition of ϵ p

, the fifth step follows from Eq. (2).

C.5 TV distance between exact and approximate process: instantiation

Lemma C.7. If dϵ H < 0 . 001 and δ H < 0 . 001 . Then, we have

Proof. We can upper bound it as follows

where the first step follows from Lemma C.6, the second step follows from choice ϵ H , δ H , and the last step follows from Lemma C.5.

C.6 Lower bound on conductance, definition

Definition C.8. Let β ∈ (0 , 0 . 1) denote some fixed constant. Let S 1 ⊂ K and S 2 := K\ S 1 and π ( S 1 ) ≤ 1 / 2 . We define S ′ 1 and S ′ 2 as follows

C.7 Lower bound on conductance, lemma

LemmaC.9 (Lower bound the conductance by cross ratio) . Let S ′ 1 and S ′ 2 be defined as Definition C.8. The conductance ϕ satisfies

Proof. The proof follows the general format for conductance proofs for geometric Markov chains (see e.g. Section 5 of Vempala [2005]).

Let S 1 ⊆ K and let S 2 = K\ S 1 . Let S ′ 1 and S ′ 2 be defined as Definition C.8. We define S ′ 3

̸

Case 1. First, we assume that both π ⋆ ( S ′ 1 ) ≥ 1 4 π ⋆ ( S 1 ) and π ⋆ ( S ′ 2 ) ≥ 1 4 π ⋆ ( S 2 ) . In this case we have

where the first step follows from Eq. (3), the second step follows from Definition C.8, the third step follows from Eq. (4), the fifth step follows from one of π ⋆ ( S ′ 1 ) and π ⋆ ( S ′ 2 ) is at least 1 / 2 .

Case 2. Now suppose that instead either π ⋆ ( S ′ 1 ) < 1 4 π ⋆ ( S 1 ) or π ⋆ ( S ′ 2 ) < 1 4 π ⋆ ( S 2 ) .

Case 2a.

where the first step follows from Eq. (3), and the third step follows from Definition C.8.

Case 2b. Similarly, if π ⋆ ( S ′ 2 ) < 1 4 π ⋆ ( S 2 ) we have

where the first step follows from Eq. (3), and the third step follows from Definition C.8.

Therefore, Eq. (5), (6), and (7) together imply that

for every partition S 1 2

Hence, Eq. (8) implies that

D Correctness for General Barrier Functions with Regularization

In Section D.1, we show how to bound the density ratios. In Section D.2, we show how to bound the determinant. In Section D.3, we show how to bound the local norm. In Section D.4, we explain how to bound the TV distance between two exact distributions. In Section D.5, we show how to bound the TV distance between two approximate processes. In Section D.6, we show how to bound the TV distance between two Gaussians. In Section D.7, we prove reversibility and stationary distribution. In Section D.8, we prove a lower bound on cross ratio distance. In Section D.9, we relate the cross ratio distance to local norm by utilizing ν -symmetry.

Algorithm 2 Our algorithm for sampling from polytope with log-barrier (formal version of Algorithm 1). As the only differences between log-barrier for polytopes and others are the choice of ν and how to generate the approximate the Hessian, we only present this algorithm.

D.1 Bounding the density ratio

Lemma D.1 (Bounding the density ratio) . Let x ∈ Int( K ) . Suppose f is L -Lipschitz and η ≤ 1 / (10 dL 2 ) , then

If in addition, f is β -smooth and η ≤ 1 / (10 dβ ) , then we further have

Proof. Proof of Part 1. Since z ∼ N ( x, Φ( x ) -1 ) , we have that

for some ξ ∼ N (0 , I d ) .

Since α -1 H ( x ) + η -1 I d ⪰ η -1 I d , and H ( x ) and I d are both positive definite, we have that

Thus, we can upper bound ∥ z -x ∥ 2 as follows:

where the third step follows from Eq. (9),

Recall that ξ is sampled from N (0 , I d ) , using Lemma C.1, we can show

Combining the above two equations, we have

Using η ≤ 1 / (80 dL 2 ) , we have

Since f is L -Lipschitz, then we have

Therefore,

where the last inequality holds by Eq. (11).

Proof of Part 2. Moreover, in the setting where f is differentiable and β -smooth, we have that, since z -x is a multivariate Gaussian random variable,

If ( z -x ) ⊤ ∇ f ( x ) ≤ 0 , we have that

Therefore,

where the last Inequality holds by Eq. (10) since η ≤ 1 20 dβ .

Lemma D.2. Let ∥ x -z ∥ Φ( x ) < 0 . 001 . Let η ≤ 0 . 01 /L 2 . Then we have

where the second step follows from Φ( x ) ⪰ η -1 I d , the last step follows from η ≤ 0 . 01 /L 2 .

D.2 Bounding the determinant

Lemma D.3 (Bounding the determinant) . Consider any x ∈ Int( K ) , and ξ ∼ N (0 , I d ) . Let z = x +(Φ( x )) -1 / 2 ξ . Then

Proof. First, note that

thus, it suffices to consider the function F ( x ) = log(det( H ( x ) + L 2 · I d )) and bound F ( z ) -F ( x ) . By Assumption (ii), we know that F ( x ) is convex. Thus,

Proof. We have

We know that where ξ ∼ N (0 , I d ) .

Thus,

By property of Gaussian distribution, we know that

is a Gaussian with mean 0 and variance ∥ (Φ( x )) -1 / 2 ∇ F ( x ) ∥ 2 2 .

By definition of Φ and Assumption (iii) , we have

Thus, rescaling the function and applying the standard concentration inequality, we complete the proof.

Lemma D.4 (Bounding the determinant, shifted) . Let α < 0 . 001 /d . For fixed x, z ∈ K such that ∥ z -x ∥ Φ( x ) < 0 . 001 . Then, we have

Proof. Let F ( x ) = log det( H ( x ) + L 2 · I d ) . Then

where the second step is because F ( x ) is convex (by Assumption (ii)).

We have

where the second step is by Cauchy-Schwarz, the third step is by assumption ∥ z -x ∥ Φ( x ) ≤ 0 . 001 , the fourth step is by α < 0 . 001 /d and Assumption (iii).

D.3 Bounding the local norm

Lemma D.5 (Bounding the local norm) . Consider any x ∈ Int( K ) , and ξ ∼ N (0 , I d ) . Let η ≤ 0 . 001 /d . Let z = x +(Φ( x )) -1 / 2 ξ . Then

Proof Sketch. Our proof is a generalization of [Sachdeva and Vishnoi, 2016, Proposition 7], where they prove the result for non-regularized log-barrier. Note that their proof also works for regularized barriers because two points are close in the Φ( x ) norm indicates that they are close in the H ( x ) norm. The only difference is that when computing the Gaussian polynomial as in Sachdeva and Vishnoi [2016], we need to handle non-uniform weights instead of the uniform weights of log-barrier. This could also be handled by observing that if two points are close in H ( x ) norm, then their weights are close in the sense that ∥ w p ( x ) -1 ( w p ( x ) -w p ( z )) ∥ ∞ ≤ c p where c p < 1 is some small constant that depends on p (see [Lee and Sidford, 2019, Lemma 34]). One could then check that the argument of [Sachdeva and Vishnoi, 2016, Proposition 7] is robust under small perturbation to the weights, and conclude similar conclusions.

Lemma D.6 (Bounding the local norm, shifted) . Consider any x ∈ Int( K ) , and ξ ∼ N (0 , I d ) . Let η ≤ 0 . 001 /d . Let ∥ z -x ∥ Φ( x ) < 0 . 001 . Let u = x +(Φ( x )) -1 / 2 ξ . Then

and therefore

Proof. Consider the following inequalities:

where the second step follows from Lemma B.2, in the last step w ∼ N (0 , ∥ Φ( x ) 1 / 2 ( x -z ) ∥ 2 2 ) . First, using η is small, we can show ∥ u -x ∥ 2 Φ( x ) < 0 . 01 with probability 0 . 9999 .

Second, using ∥ z -x ∥ Φ( x ) < 0 . 001 and Gaussian concentration, we can show that

Put them together and union bound, we obtain the desired result.

Lemma D.7 (Bounding the local norm, shifted) . Consider any x ∈ Int( K ) , and ξ ∼ N (0 , I d ) . Let η ≤ 0 . 001 /d . Let ∥ z -x ∥ Φ( x ) < 0 . 001 . Let u = x +(Φ( x )) -1 / 2 ξ . We have

Proof. Let u ′ ∼ z +Φ( z ) -1 / 2 ξ ′ where ξ ′ ∼ N (0 , I d ) . We have

So

Proof. We define

Then where

̸

where first step is by union bound, second step is by Lemma D.5, Lemma D.6, and by TV( u, u ′ ) ≤ 0 . 001 when ∥ x -z ∥ Φ( x ) ≤ 0 . 001 (Lemma D.10).

D.4 Bounding TV distance between exact processes

Recall that we define P x ( S ) = Pr[ x ∈ S ] . We prove a bound between exact distributions of two points x and z .

Lemma D.8 (TV distance between the exact distribution) . For any x, z ∈ K such that ∥ x -z ∥ Φ( x ) ≤ 0 . 001 , we have that

For convenience, we define

Let θ ∈ (0 , 1) . We have the following:

where the sixth step follows from Eq. (12), and the last step follows from choice of θ and θ 0 . Let a i = e -b i . It suffices to prove that for some θ ∈ (0 , 1) , we have

Suppose for some τ ∈ [1 , 5] we can show that

Then by union bound, we can know that

where the second step follows from e -τ = θ ≥ 0 . 01 , and the last step follows from θ 1 = θ 2 = θ 3 = 0 . 99 . In the following, we will establish the three bounds of interest.

Part 1. By definition, we have

Using Lemma D.7 (on the term 0 . 5 ∥ u -z ∥ 2 Φ( z ) -0 . 5 ∥ u -x ∥ 2 Φ( u ) ) and Lemma D.5 (on the term 0 . 5 ∥ u -x ∥ 2 Φ( u ) -0 . 5 ∥ u -x ∥ 2 Φ( x ) ), we have

Part 2. By definition, for a 2 we have

Then, combining the above three equations, we can get the following result:

Thus,

Part 3.

By Lemma D.1 (on the term f ( u ) -f ( x ) ) and Lemma D.2 (on the term f ( x ) -f ( z ) ), we have

By Lemma D.5 (on the term 0 . 5 ∥ u -x ∥ 2 Φ( u ) -0 . 5 ∥ u -x ∥ 2 Φ( x ) ) and Lemma D.6 (on the term 0 . 5 ∥ z -u ∥ 2 Φ( u ) -0 . 5 ∥ u -x ∥ 2 Φ( u ) ), we have

Thus, combining the above three equations, we obtain the following:

D.5 Bounding TV distance between approximate processes

Lemma D.9 (Robust version of Lemma E.10 in Mangoubi and Vishnoi [2023]) . If δ H < 0 . 001 , dα < 0 . 001 , dϵ H < 0 . 001 and for any x, z ∈ K , ∥ x -z ∥ Φ( x ) < 0 . 001 , then we have

Proof. We can upper bound TV( ˜ P x , ˜ P y ) as follows:

First step is by triangle inequality. Second step is by Lemma D.8. The third step is by Lemma C.7.

D.6 Bounding TV distance between Gaussians

Lemma D.10 (Lemma E.6 in Mangoubi and Vishnoi [2023]) . For any x, z ∈ K such that ∥ x -z ∥ Φ( x ) ≤ 0 . 001 , we have

Further, if αd < 0 . 001 , then we have

Lemma D.11 (Robust version of Lemma D.10) . For any x, z ∈ K such that ∥ x -z ∥ Φ( x ) ≤ 1 4 α 1 / 2 , with probability at least 1 -1 / 1000 , we have

Proof. By triangle inequality,

Second term is bounded using Lemma D.10. First and third term are bounded using similar ways. We have

Thus, we complete the proof.

D.7 Reversibility and stationary distribution

For any x ∈ K , we define the random variable Z x to be the step taken by the Markov chain in Algorithm 2 from the point x , that is, set z = x + ˜ Φ( x ) -1 / 2 ξ where ξ ∼ N (0 , I d ) . If z ∈ K , set Z x = z with probability

Else, we set z = x .

We provide a modified version of Proposition E.12 in Mangoubi and Vishnoi [2023] where our Markov chain differs from theirs.

Lemma D.12 (Reversibility and stationary distribution) . For any S 1 , S 2 ⊆ K we have that

Proof. Let a, b be two i.i.d. random variables (one can view a and b as random coins to generate the corresponding sparsifers) such that ˜ Φ a,x is a function of a, x and ̂ Φ b,y is a function of b, y .

Let p a,x ( · ) be pdf of N (0 , ˜ Φ -1 a,x ) and p b,y ( · ) be the pdf of N (0 , ̂ Φ -1 b,y ) . Because a and b are i.i.d, they are interchangeable, i.e., ( a, b ) has the same distribution as ( b, a ) .

The first and fifth steps are by definition of ˜ P x . The second and fourth steps are by changing order of integration. The third step is by interchangeability of a and b .

This proves reversibility and stationary distribution.

D.8 Lower bound on cross ratio distance

Definition D.13. We define

For simplicity, we use F to denote F ( n, α, β, R ) .

Lemma D.14. Let S ′ 1 and S ′ 2 be defined as Definition C.8. Le F be defined as Definition D.13. We have

Proof. Using Lemma D.17, we have that for any u, v ∈ K ,

We will try to prove σ ( u, v ) > 1000 F in the following. To do that, we will prove it by making a contradiction.

Suppose σ ( u, v ) ≤ 1000 F , then we have to require ∥ u -v ∥ Φ( u ) ≤ 0 . 001 .

Once we had that ∥ u -v ∥ Φ( u ) ≤ 0 . 001 , using Lemma D.9, for any u, v ∈ K we have that

On the other hand, Definition C.8 implies that, for any u ∈ S ′ 1 , v ∈ S ′ 2 we have that

where the last step follows from 1 -2 β > 0 . 99 .

Thus, we obtain a contradiction, which means that

D.9 Bounding cross ratio distance through ν -symmetry

In Lemma E.2 of Mangoubi and Vishnoi [2023], they only prove the cross ratio distance bound for log-barrier function. Here, we generalize it to arbitrary barrier function. The key concept our proof relies on is the notion of ν -symmetry (Assumption (i)).

Lemma D.15. Let H ∈ K → R d × d be a ν -symmetric function, and u, v ∈ K . Consider the chord that passes through u, v and intersects with K . Let p, q be the two endpoints of the chord, with the order p, u, v, q . Then, we have

Proof. Suppose we can show that p ∈ K ∩ (2 u -K ) , then it naturally follows that p ∈ E u ( √ ν ) and ∥ p -u ∥ H ( u ) ≤ √ ν .

p ∈ K trivially follows by the construction of the chord. To see p ∈ 2 u -K , it is enough to show that for some y ∈ K , we have p = 2 u -y . We claim y = 2 u -p is such a choice. To see 2 u -p ∈ K , we need to show that 2 Au -Ap ≤ b . We partition the constraints into two sets.

Case 1. Suppose for a i , we have a ⊤ i u ≤ a ⊤ i p , then 2 a ⊤ i u ≤ 2 a ⊤ i p ≤ a ⊤ i p + b i , therefore, 2 a ⊤ i u -a ⊤ i p ≤ b i .

Case 2. Suppose otherwise, a ⊤ i u > a ⊤ i p , then consider

We shall show that d 1 ≤ d 2 via a geometric argument. Consider the projection of both p and u onto the hyperplane a i x = b i , and the two chords that pass through p and u respectively, orthogonal to the hyperplane. Let us denote them with l p , l u respectively. Observe that l p and l u are parallel, and ∥ l p -l u ∥ 2 = ( a ⊤ i u -b i ) -( a ⊤ i p -b i ) = d 1 . It should then be obvious that d 1 ≤ d 2 : let proj( u ) denote the orthogonal projection of u onto the hyperplane, then

since ∥ l p -proj( p ) ∥ 2 ≥ 0 , we have that d 1 ≤ d 2 , and consequently,

This proves that 2 u -p ∈ K , and we can conclude that p ∈ 2 u - K , which yields our desired result.

Remark D.16. The above argument can be extended to spectrahedra and more general convex sets.

Let u, v ∈ K be two arbitrary points, and consider the chord that passes through u, v with two endpoints being p, q . Note that p, q ∈ ∂ K .

Lemma D.17. For any u, v ∈ K , let H be any matrix function satisfying Assumption (i), (ii) and (iii). We assume that K is contained in ball of radius R and has nonempty interior.

For any parameter α ∈ (0 , 1) , η ∈ (0 , 1) , we define matrix function Φ( u ) := α -1 · H ( u ) + η -1 I d . Then, we have

Proof. Without loss of generality, we can assume that

We can lower bound σ 2 ( u, v ) as follows:

where the second step is by ∥ p -q ∥ 2 ≥ max {∥ p -u ∥ 2 , ∥ v -q ∥ 2 } , the third step is by ∥ v -q ∥ 2 ≤ ∥ u -q ∥ 2 , the fourth step is by ∥ p -q ∥ 2 ≥ ∥ p -u ∥ 2 , the fifth step follows from ∥ p -u ∥ 2 ≤ ∥ u -q ∥ 2 (see Eq. (15)), sixth step is by max { A,B } ≥ 0 . 5 · ( A + B ) , where the seventh step follows from ∥ p -q ∥ 2 ≤ R and Eq. (16). The eighth step is due to Lemma D.15.

It remains to show Eq. (16). Due to the fact that p, u, v are on the same chord, vector u -v and p -u are on the same direction. This means we can write u -v = c · ( p -u ) for some c ∈ R , and

This completes the proof.

E Computing Approximate Lewis Weights and Leverage Scores

In Section E.1, we present an algorithm to approximate the Hessian. In Section E.2, we show how to compute leverage score to low precision. In Section E.3, we show how to compute Lewis weights to high precision.

E.1 Subsampling to approximate the Hessian

Algorithm 3 Subsampling Algorithm

1: procedure SUBSAMPLE( A ∈ R n × d , b ∈ R n , x, n, d, ϵ H ∈ (0 , 1) )

2: ϵ σ ← 0 . 001

4: Form B ∈ R n × d where i -th row of B is a i · ( ⟨ a i , x ⟩ -b i ) -1

3: m ← ϵ -2 H d log d

5: Compute the O (1 ± ϵ σ ) -approximation to the leverage score of B

6: Sample a matrix ˜ B ∈ R m × d such that (1 -ϵ H ) B ⊤ B ⪯ ˜ B ⊤ ˜ B ⪯ (1 + ϵ H ) B ⊤ B

7: return ˜ B ⊤ ˜ B

8: end procedure

In this section, we provide a sparsification result for H ( w ) using leverage score sampling.

To better monitor the whole process, it is useful to write H ( w ) as A ⊤ S ( w ) -2 A , where A ∈ R n × d is the constraint matrix and S ( w ) is a diagonal matrix with S ( w ) i = a ⊤ i w -b i . The sparsification process is then sample the rows from the matrix S ( w ) -1 A .

Definition E.1. Let B ∈ R n × d be a full rank matrix. We define the leverage score of the i -th row of B as

where b i is the i -th row of B .

Definition E.2 (Sampling process) . For any w ∈ K , let H ( w ) = A ⊤ S ( w ) -2 A . Let p i ≥ β · σ i ( S -1 ( w ) A ) d , suppose we sample with replacement independently for s rows of matrix S ( w ) -1 A , with probability p i of sampling row i for some β ≥ 1 . Let i ( j ) denote the index of the row sampled in the j -th trial. Define the generated sampling matrix as

Lemma E.3 (Sample using Matrix Chernoff) . Let ϵ, δ ∈ (0 , 1) be precision and failure probability parameters, respectively. Suppose ˜ H ( w ) is generated as in Definition E.2, then with probability at least 1 -δ , we have

Moreover, the number of rows s = Θ( β · ϵ -2 d log( d/δ )) .

Proof. The proof will be designing a family of random matrices X . Let y i = ( A ⊤ S ( w ) -2 A ) -1 / 2 S ( w ) -1 i,i · a i be the i -th sampled row and set Y i = 1 p i y i y ⊤ i . Let X i = Y i -I d . Note that

Also, the norm of y i connects directly to the leverage score:

We use i ( j ) to denote the index of row that has been sampled during j -th trial.

Unbiased Estimator. Note that

Bound on ∥ X ∥ . To bound ∥ X ∥ , we provide a bound for any ∥ X i ∥ as follows:

Bound on ∥ E [ X ⊤ X ] ∥ . We compute the spectral norm of the covariance matrix:

the spectral norm is then

Put things together. Set γ = 1 + d β and σ 2 = d β -1 , we apply Matrix Chernoff Bound as in Lemma A.8:

where we choose s = Θ( β · ϵ -2 d log( d/δ )) . Finally, we notice that

Therefore, we can conclude the desired result via ∥ W ∥ ≥ ϵ .

E.2 Computing leverage score

In this section, we provide an algorithm that approximates leverage scores in near-input sparsity time. Our algorithm makes use of the sparse embedding matrix of Nelson and Nguyên [2013] that has O (log( n/ϵ )) nonzero entries per column.

Lemma E.4 (Approximate leverage scores) . Let A ∈ R n × d , there exists an algorithm that runs in time ˜ O ( ϵ -1 (nnz( A ) + ϵ -1 d ω )) that outputs a vector ˜ w 2 ( A ) ∈ R n , such that ˜ w 2 ( A ) ≈ ϵ w 2 ( A ) .

Proof. We follow an approach of Woodruff [2014], but instead we use a high precision sparse embedding matrix Nelson and Nguyên [2013] and prove a two-sided bound on leverage score.

Let S be a sparse embedding matrix with r = O ( ϵ -2 d poly log( d/ ( ϵδ ))) rows and each column has O ( ϵ -1 log( nd/ϵ )) nonzero entries. We first compute SA in time ˜ O ( ϵ -1 nnz( A )) , then compute the QR decomposition SA = QR in time ˜ O ( ϵ -2 d ω ) . Note that R ∈ R d × d hence R -1 can be computed in O ( d ω ) time.

Now, let G ∈ R d × t matrix with t = O ( ϵ -2 log( n/δ )) , each entry of G is i.i.d. N (0 , 1 /t ) random variables. Set q i = ∥ e ⊤ i AR -1 G ∥ 2 2 for all i ∈ [ n ] . We argue q i is a good approximation to ( w 2 ( A )) i .

First, with failure probability at most δ/n , we have that q i ≈ ϵ ∥ e ⊤ i AR -1 ∥ 2 2 via Johnson-Lindenstrauss lemma [Johnson and Lindenstrauss, 1984]. Now, it suffices to argue that ∥ e ⊤ i AR -1 ∥ 2 2 approximates ∥ e ⊤ i U ∥ 2 2 well, where U ∈ R n × d is the left singular vectors of A . To see this, first observe that for any x ∈ R d ,

where the last step is due to Q has orthonormal columns. This means that all singular values of AR -1 are in the range [1 -ϵ, 1+ ϵ ] . Now, since U is an orthonormal basis for the column space of A , AR -1 and U has the same column space (since R is full rank). This means that there exists a change of basis matrix T ∈ R d × d with AR -1 T = U . Our goal is to provide a bound on all singular values of T . For the upper bound, we claim the largest singular value is at most 1 + 2 ϵ , to see this, suppose for the contradiction that the largest singular is larger than 1 + 2 ϵ and let v be its corresponding (unit) singular vector. Since the smallest singular value of AR -1 is at least 1 -ϵ , we have

however, recall AR -1 T = U , therefore ∥ AR -1 Tv ∥ 2 2 = ∥ Uv ∥ 2 2 = ∥ v ∥ 2 2 = 1 , a contradiction. One can similarly establish a lower bound of 1 -2 ϵ . Hence, the singular values of T are in the range of [1 -2 ϵ, 1 + 2 ϵ ] . This means that

as desired. Scaling ϵ to ϵ/ 2 yields the approximation result.

Now, regarding the running time of computing q i , note that we can first multiply R -1 with G in time ˜ O ( ϵ -2 d 2 ) , this gives a matrix of size d × t . Multiplying this matrix with A takes ˜ O ( ϵ -1 nnz( A )) time. Hence, the overall time for computing ˜ w 2 ( A ) = q ∈ R n is ˜ O ( ϵ -1 nnz( A ) + ϵ -2 d ω ) .

Remark E.5. It suffices to choose ϵ = 1 10 for generating approximate leverage scores and subsampling the target matrix H ( x ) .

E.3 Computing Lewis weights

Before stating the main technical tool for this section, we want to make several remarks regarding the Lewis weights.

Since its introduction by Cohen and Peng [Cohen and Peng, 2015] as an algorithmic tool for ℓ p row sampling, Lewis weights, as its natural formulation follows from a convex program, is not known to be solved exactly in polynomial time . All known algorithms [Cohen and Peng, 2015, Lee and Sidford, 2019, Fazel et al., 2022, Cohen et al., 2019, Jambulapati et al., 2022] can only compute ϵ -approximate Lewis weights .

Definition E.6 ( ℓ p Lewis weights) . Let A ∈ R n × d . The ℓ p Lewis weights of A is a vector w p ( A ) ∈ R n satisfying

where W p ( A ) ∈ R n × n is the diagonal matrix that puts w p ( A ) on the diagonal, σ : R n × d → R n is the operation that outputs all leverage scores of input matrix.

We are now posed to state a main tool from Lee and Sidford [2019].

Theorem E.7 (Theorem 46 in Lee and Sidford [2019]) . Let P = { x : Ax > b } denote the interior of non-empty polytope for non-degenerate A ∈ R n × d . There is an O ( d log 5 n ) -self concordant barrier ψ defined by ℓ p Lewis weight with p = Θ(log n ) satisfying

where A x = diag( Ax -b ) and w x is the ℓ p Lewis weight of the matrix A x . Furthermore, we can compute or update the w x , ∇ ψ ( x ) and ∇ 2 ψ ( x ) as follows:

Initial Weight: For any x ∈ R d , one can compute a vector ˜ w x such that (1 -ϵ ) w x ≤ ˜ w x ≤ (1 + ϵ ) w x in O ( nd ω -1 / 2 · log 3 n log( n/ϵ )) time.

Update Weight and Compute Gradient/Hessian: Given a vector ˜ w x such that ˜ w x = (1 ± 1 100 ) w x for any y with ∥ x -y ∥ A ⊤ x W x A x ≤ c log 2 n for some small constant c > 0 , we can compute ˜ w y , v and H such that ˜ w y = (1 ± ϵ ) w y ,

in O ( nd ω -1 · log n · log( n/ϵ ))

F Fast Approximate Sampling from Polytopes: Complexity

In this section, we provide the runtime analysis for sampling from polytopes via log-barrier and Lee-Sidford barrier. The result for log-barrier is in Section F.1 and for Lee-Sidford barrier is in Section F.2.

F.1 Log-barrier in nearly-linear time

Throughout this section, we let α ∈ (0 , 1 / (10 5 d )) and η ∈ (0 , 1 / (20 dL 2 )) .

Lemma F.1. If g is log-barrier function with ν = n , then each iteration of Algorithm 2 can be implemented in O (nnz( A ) + T mat ( d, d 3 , d )) time plus O (1) calls to the oracle of f .

Proof. We go through each step of Algorithm 2 and add up the time and oracle calls for each step:

We first need to sample a d -dimensional Gaussian random vector ξ ∼ N (0 , I d ) , which can be performed in O ( d ) time.

At each iteration, by the definition of log-barrier function, we can write the Hessian as follows:

We define define matrix B ∈ R n × d as follows

where s i = ( ⟨ a i , x ⟩ -b i ) , ∀ i ∈ [ n ] . Then we can write H ( x ) = B ⊤ B ∈ R d × d .

For matrix B , we can compute a constant approximation of its leverage score, then we can sample according to leverage, and generate a diagonal matrix D such that

This step takes ˜ O (nnz( A ) + d ω (1 , 3 , 1) ) time.

Computing ˜ Φ( x ) can be done in d 2 time and computing the vector z

requires to invert the matrix Φ( x ) , taking the square root and multiplying with a vector ξ . Inversion and square root can be performed in time O ( d ω ) by computing its spectral decomposition, and the matrix-vector product can be done in time O ( d 2 ) . Hence, the overall time of this step is O ( d ω ) .

To determine whether z ∈ K , one can simplify verify Az ≤ b , in time O (nnz( A )) .

We also need to compute ̂ H ( z ) and ̂ Φ( z ) . This is similar to computing ˜ H ( x ) and ˜ Φ( x ) .

We then need to compute the determinant det( ˜ Φ( x )) and det( ̂ Φ( z )) which can be done in O ( d ω ) time via a spectral decomposition.

We then need O (1) oracle queries to the function values of f .

Therefore, adding up the number of arithmetic operations and oracle calls from all the different steps of Algorithm 2, we get that each iteration of Algorithm 2 can be computed in ˜ O (nnz( A )+ T mat ( d, d 3 , d )) arithmetic operations plus O (1) calls to the oracle of f .

Remark F.2. We note a recent work Mangoubi and Vishnoi [2024] has provided an algorithm that runs in nnz( A ) + d 2 time per iteration, by adapting techniques from Laddha et al. [2020]. Their per iteration cost improvement only works for log-barrier, and they could only manage to obtain an ˜ O ( nd + dL 2 R 2 ) mixing time. Our framework on the other hand generalizes to other barriers such as Lee-Sidford barrier, which provides a nearly-optimal mixing rate. We also adapt the framework to sampling from spectrahedra.

F.2 Lee-Sidford barrier via approximation scheme

Lemma F.3. If g is the Lee-Sidford barrierfunction with ν = d log 5 n , then, each iteration of Algorithm 2 can be implemented in ˜ O ( nd ω -1 ) time plus O (1) calls to the oracle of f .

Proof. For Lee-Sidford barrier, it suffices to use H ( x ) = A ⊤ x W x A x as where W x is the ℓ p Lewis weights for p = Θ(log m ) . However, as Lewis weights cannot be computed exactly, we shall use the algorithm of Lee and Sidford [2019] to compute an ϵ -approximation.

To invoke Theorem E.7, we need to make sure that z and x satisfy ∥ z -x ∥ A ⊤ x W x A x ≤ c log 2 n . We can achieve so by scaling down α by a factor of log 2 n . This only blows up the convergence by ˜ O (1) factor, so it is acceptable.

The runtime analysis is then similar to Lemma F.1. Computing the initial Lewis weights takes ˜ O ( nd ω -1 / 2 log(1 /ϵ )) time owing to Theorem E.7. As the algorithm requires ˜ O ( d 2 ) iterations, this time can be amortized. For each iteration, it needs to query the Lewis weights data structure that outputs an ϵ -Lewis weights in ˜ O ( nd ω -1 log(1 /ϵ )) time. As we will choose ϵ as O (1 /d ) , this time is ˜ O ( nd ω -1 ) . We can then use this approximate Hessian to progress the algorithm. All subsequent operations take O ( d ω ) time. Thus, the cost per iteration is ˜ O ( nd ω -1 ) .

G Approximate Sampling from a Spectrahedron

We present an algorithm that efficiently and approximately samples from a spectrahedron, which is a popular convex body utilized by semidefinite programs.

G.1 Definitions

Definition G.1. Let A 1 , . . . , A d ∈ R n × n be a collection of symmetric matrices and C ∈ R n × n be symmetric. We define the corresponding spectrahedron as

We define the log-barrier for spectrahedron and its corresponding Hessian.

Definition G.2 (Nesterov and Nemirovskii [1994]) . Let K be a spectrahedron described by symmetric matrices { A 1 , . . . , A d } ⊆ R n × n and C ∈ R n × n . The log barrier ϕ log ( x ) is defined as

and its corresponding Hessian is defined as

where S ( x ) := ∑ d i =1 x i A i -C ∈ R n × n and A ∈ R d × n 2 and the i -th row of A is the vectorization of A i ∈ R n × n .

To compute H log ( x ) , it is handy to define a matrix B ∈ R d × n 2 .

Definition G.3. We define a matrix B ∈ R d × n 2 as B = A ( S ( x ) -1 / 2 ⊗ S ( x ) -1 / 2 ) . Consequently, H log ( x ) = BB ⊤ .

We state a useful lemma for computing the matrix B .

Lemma G.4. Let matrix B ∈ R d × n 2 be defined as in Def. G.3. Then, the i -th row of B can be computed as vec( S ( x ) -1 / 2 A i S ( x ) -1 / 2 ) .

Proof. The proof relies on a simple fact of Kronecker product and vectorization:

which is the definition of the i -th row of B .

G.2 n -symmetry of H log for spectrahedra

In this section, we prove that H log is n -symmetry for spectrahedra.

Lemma G.5. H log ( x ) is n -symmetry, that is, for any x ∈ K ,

Proof. Pick a point y ∈ E x (1) , then we know that ( y -x ) ⊤ A ( S ( x ) -1 ⊗ S ( x ) -1 ) A ⊤ ( y -x ) ≤ 1 . We note that the set K∩ (2 x -K ) can be characterized as the set of all points u ∈ R d such that

These two conditions imply

and henceforth

We will prove y satisfies these conditions. Note that y ∈ E x (1) implies that

if we let M denote ∑ d i =1 ( y i -x i ) S ( x ) -1 / 2 A i S ( x ) -1 / 2 , then the above condition implies that M ⊤ M ⪯ I n , meaning all the eigenvalues of M lie between [ -1 , 1] , thus we have shown that E x (1) ⊆ K ∩ (2 x -K ) .

For the other direction, we know that

where we use the fact that -I n ⪯ M ⪯ I n hence all eigenvalues of M lie between [ -1 , 1] , and the squared eigenvalues of M have its magnitude at most 1. We then use the squared Frobenius norm is equivalent to squared ℓ 2 norm of singular values of M , as desired.

G.3 Bounded local norm of H log ( x )

We prove that H log ( x ) has the bounded local norm property.

Lemma G.6. Let H log ( x ) ∈ R d × d be defined as in Def. G.2. Let F ( x ) = log det( H log ( x )) . Then, F ( x ) is convex in x .

Proof. The proof is by observe that the function F ( x ) is actually the volumetric barrier function for the SDP spectrahedron. Thus, the function F is convex, for more details, see Nesterov and Nemirovskii [1994].

Lemma G.7. Let F ( x ) be defined as in Lemma G.6. Then, we have

where P = B ⊤ H ( x ) -1 B .

Proof. For simplicity, we let H to denote H log . For each coordinate j ∈ [ d ] , we have

examine the following term:

where P ∈ R n 2 × n 2 is the projection of B ⊤ , i.e., P = B ⊤ H ( x ) -1 B . Thus,

This completes the proof.

Lemma G.8. The matrix function H ( x ) = A ( S -1 ⊗ S -1 ) A ⊤ ∈ R d × d has the following bounded norm property: for any direction h ∈ R d ,

Proof. Let x t = x + t · h for some fixed vector h ∈ R d , S t = ∑ m i =1 x t,i A i -C ∈ R n × n , A t = A ( S -1 / 2 t ⊗ S -1 / 2 t ) ∈ R d × n 2 , P t = A ⊤ t ( A t A ⊤ t ) -1 A t ∈ R n 2 × n 2 and Σ t = ∑ n i =1 ( e ⊤ i ⊗ I n ) P t ( e i ⊗ I n ) = ∑ n i =1 ( I n ⊗ e ⊤ i ) P t ( I n ⊗ e i ) . Note that for any matrix M ∈ R n × n , tr[ P t · ( M ⊗ I n )] = tr[ P t · ( I n ⊗ M )] = tr[Σ t · M ] . Also note that tr[Σ t ] = tr[ P t ] = d .

Let H t = A ⊤ ( S -1 t ⊗ S -1 t ) A ∈ R d × d . We have

the fifth step is by for PSD matrix M , we have MPM ⪯ M 2 where P is a projection matrix, the sixth step is by the below derivation, the seventh step is by tr[ P t · ( M 2 ⊗ S I n )] = 2 tr[ M Σ t M ] , the eighth step is by for PSD matrix M , 2 M ⊗ S I n ⪰ M ⊗ M , the ninth step is by Hölder's inequality where ⟨ A,B ⟩ ≤ ∥ A ∥ · ∥ B ∥ s 1 for ∥ B ∥ s 1 be its Schatten-1 norm. If B is PSD, ∥ B ∥ 1 = tr[ B ] , the tenth step is by for PSD matrix A , ∥ A ∥ ≤ tr[ A ] .

Note that

Thus, and we have

plug in the above derivation we obtain the desired result. Finally, we send t → 0 and conclude the proof.

Lemma G.9. Let H ( x ) be the matrix function A ( S -1 ⊗ S -1 ) A ⊤ , then we have

Proof. We have

where the fourth step is by tr[ M ] ≤ √ d · ∥ M ∥ F for d × d PSD matrix M , the fifth step follows from Lemma G.8, the last step follows from ∥ u ∥ H ( x ) = 1 .

G.4 Fast Hessian approximation

We need a particular type of sketch for Kronecker product of matrices.

Definition G.10 ( TensorSRHT [Song et al., 2021]) . The TensorSRHT Π : R n × R n → R s is defined as S := 1 √ s P · ( HD 1 ⊗ HD 2 ) , where each row of P ∈ { 0 , 1 } s × n 2 contains only one 1 at a random coordinate and one can view P as a sampling matrix. H is a n × n Hadamard matrix, and D 1 , D 2 are two n × n independent diagonal matrices with diagonals that are each independently set to be a Rademacher random variable (uniform in {-1 , 1 } ).

Lemma G.11 (Lemma 2.12 in Song et al. [2021]) . Let Π be a TensorSRHT matrix defined in Definition G.10. If s = O ( ϵ -2 d log 3 ( nd/ ( ϵδ ))) , then for any orthonormal basis U ∈ R n 2 × d , we have that with probability at least 1 -δ , the singular values of Π U lie in the range of [1 -ϵ, 1 + ϵ ] .

The following result provides an efficient embedding for B .

LemmaG.12. Let B ∈ R d × n 2 be defined as in Def. G.3, ϵ ∈ (0 , 1 / 10) denote an accuracy parameter and δ ∈ (0 , 1 / 10) denote a failure probability.

Let Π ∈ R s × n 2 be a TensorSRHT matrix with s = Θ( ϵ -2 d log 3 ( nd/ ( ϵδ ))) , then we have

Moreover, Π B ⊤ can be computed in time

Proof. The correctness part follows directly from Lemma G.11. It remains to argue for the running time. We need to unravel the construction of both S and B (see Definition G.2). Recall that Π = 1 √ s P · ( HD 1 ⊗ HD 2 ) and

since P is a row sampling matrix, the product can be computed as follows:

First compute HD 1 S ( x ) -1 / 2 and HD 2 S ( x ) -1 / 2 . Since H is a Hadamard matrix, this step can be carried out in O ( n 2 log n ) time.

Applying P to the vector can be interpreted as sampling s coordinates from the matrix HD 2 S ( x ) -1 / 2 A i S ( x ) -1 / 2 D 1 H ⊤ . Let ( i 1 , j 1 ) , . . . , ( i s , j s ) denote the coordinates sampled by P . We form two matrices X,Y ∈ R s × n where the k -th row of X is ( HD 2 S ( x ) -1 / 2 ) i k , ∗ and the k -th row of Y is ( HD 1 S ( x ) -1 / 2 ) j k , ∗ . It is easy to verify that the ( i k , j k ) -th entry of XA i Y ⊤ ∈ R s × s is the corresponding entry of HD 2 S ( x ) -1 / 2 A i S ( x ) -1 / 2 D 1 H ⊤ . This step therefore takes T mat ( n, n, s ) time.

As we need to apply the second step to d rows, the total runtime is

Remark G.13. For comparison, compute B straightforwardly using the Kronecker product identity takes O ( dn ω ) time. As long as the row count s is independent of n , Lemma G.12 approximately computes B in time ˜ O ( n 2 · poly( d )) . For the regime where d ≪ n , our algorithm is much more efficient.

H Convexity of Regularized Log-Barrier Function

In this section, we prove the convexity of log det( H log ( x ) + I d ) for both polytopes and spectrahedra.

H.1 Convexity of log det( H log ( x ) + I d ) : polytopes

We prove that for H ( x ) = ∑ n i =1 a i a ⊤ i ( a ⊤ i x -b i ) 2 , the Hessian of the log barrier, we have the convexity.

Definition H.1 (Ridge leverage score) . Let B ∈ R n × d , we define the λ -ridge leverage score of B as

If λ = 1 , we abbrieviate as ˜ σ i ( B ) .

For convenient of the proof, we also define

We also define ̂ σ λ,i ( B ) as follows:

Fact H.2. Let X be R d × d be invertible and symmetric. Then we have

Lemma H.3. Let X ∈ R d × d be invertible and symmetric. Then,

to compute ∂ det X , recall the adjugate of X , denoted by adj( X ) where X -1 = 1 det X adj( X ) , and it remains to show that ∂ det X = adj( X ) , this can be derived using cofactor expansion of a matrix, which states that fix a row i , we have

Proof. The proof is by chain rule:

by product rule,

and consequently,

Proof. First, we know that

where ∂ adj i,k ∂X i,j = 0 for all k since the cofactor of i, k is the principal minor by removing row i and column k . Therefore,

The goal of this section is to prove the d × d matrix ∇ 2 F ( x ) ≻ 0 . We start with computing ∇ F ( x ) ∈ R d .

Lemma H.4. We define s x,i = a ⊤ i x -b i for each i ∈ [ n ] . Let S x denote a diagonal matrix such that S x = diag( s x ) . Let H ( x ) = A ⊤ S -2 x A ∈ R d × d . Let F ( x ) = log det( H ( x ) + I d ) , then

For each j ∈ [ d ] , we can write ∂F ∂x j as follows:

where the forth step follows from Eq. (19), the fifth step follows from tr[ AB ] = tr[ BA ] , and the last step follows from ˜ σ .

We use chain rule:

Fact H.5. We have the following partial derivative:

Proof. We analyze the partial derivative term

where the second step follows from Fact H.2.

Proof. Note that

therefore,

Thus we complete the proof.

Lemma H.7. Let F ( x ) = log det( H ( x ) + I d ) , then ∇ 2 F ( x ) ⪰ 0 .

Thus, F ( x ) is convex.

Proof. Note that

Plug in Eq. (20) into Eq. (21), we have

and

:=

B

1

+

B

2

+

B

3

.

where the first step follows from Fact H.6, the last step follows from

It is obvious that B 1 ≻ 0 and B 2 ≻ 0 .

It is not hard to see that

Thus we have B 3 ⪰ 0 and we complete the proof.

H.2 Convexity of log det( H log ( x ) + I d ) : spectrahedra

In this section, we prove that log det( H log ( x ) + I d ) is a convex function on x . Combined with the bounded variance property, we provide an algorithm that samples from the SDP spectrahedron.

Definition H.8. We define the ridge-projection matrix P ∈ R n 2 × n 2 as

We define projection matrix P ∈ R n 2 × n 2 as follows

Remark H.9. It is not hard to see that P is a PSD matrix. Note that the matrix H + I d is PSD, since H is PSD, therefore ( H + I d ) -1 is also PSD. Thus, the ridge-projection is PSD, as its two 'arms' are symmetric.

Lemma H.10. Let P ∈ R n 2 × n 2 be the projection matrix and let P ∈ R n 2 × n 2 be the ridge-projection matrix defined in Def. H.8. Then, we have P ⪯ P .

Proof. Let H = U Σ U ⊤ be its eigendecomposition, we need to show that

where the last step follows from for any non-negative number λ , (1 + λ ) -1 ≤ λ -1 .

Let us expand the LHS:

The following lemma is a generalization of Theorem 4.1 of Anstreicher [2000].

Lemma H.11. Let H log ( x ) ∈ R d × d be defined as Definition G.6. Let F ( x ) = log det( H log ( x )+ I d ) . Then, F ( x ) is convex in x .

Proof. Using standard algebra and chain rule computations, we have

For simplicity, we drop x . The Q,R,T ∈ R d × d can be defined as follows

Thus, F ( x ) is convex.

Lemma H.12. For any x , if S ( x ) ≻ 0 , then we have

Part 1. Q ⪰ 0 ;

Part 2. T ⪰ 0 ;

Part 3. T ⪯ 1 2 ( Q + R ) ;

Part 4. ∇ 2 F ( x ) ⪰ 0 ;

Part 5. R ⪯ Q ;

Part 6. ∇ 2 F ( x ) ⪯ 3 Q ( x ) .

̸

Proof. Proof of Part 1 and 2. Let ξ ∈ R d and ξ = 0 .

Then we have

where B = B ( ξ ) = ∑ d i =1 ξ i A i , and B = S -1 / 2 BS -1 / 2 .

Similarly, we have

We can rewrite ξ ⊤ Tξ as follows:

It is obvious that I d , P and B 2 are all PSD matrices.

Using Part 3 of Lemma A.9 and Part 6 of Lemma A.10, we have

Since ξ is arbitrary, then we know that

Proof of Part 4. We have

Using Part 4 of Lemma A.9, we know

where the second step follows from T ⪰ 0 , the last step follows from Part 5.

Which is

Since it's arbitrary ξ , then we have

Proof of Part 6. We have

Proof of Part 3. Note that P is a projection matrix, and P ⪯ P by Lemma H.10.

where the last step follows from Part 2 of Lemma A.10.

Applying Part 1 of Lemma A.9, we have

The above equation implies the following

Note that, here ξ is arbitrary, thus we know that

Proof of Part 5. Let v i be orthonormal eigenvectors of B with corresponding eigenvalues λ i .

Then using [Horn and Johnson, 1990, Theorem 4.4.5], B 2 ⊕ I has orthonormal eigenvectors v i ⊗ v j with corresponding eigenvalues 1 2 ( λ 2 i + λ 2 j ) , while (see Horn and Johnson [1990]) B ⊗ B has the same eigenvectors v i ⊗ v j , with corresponding eigenvalues λ i λ j .

It then follows from ( λ i -λ j ) 2 for each i, j that

H.3 Discussion

It is well-known that for popular barrier functions, such as volumetric barrier and Lee-Sidford barrier [Lee and Sidford, 2014, 2019], the function log det H ( x ) is convex in x . However, given a PSD matrix function H ( x ) for which log det H ( x ) is convex in x , it is generally not true that log det( H ( x ) + I d ) is convex.

Fact H.13 (Folklore) . Suppose that H ( x ) is a positive semi-definite matrix for x in the domain. Suppose log det H ( x ) is convex. It is possible log det( H ( x ) + I ) is not convex.

Proof. If log det( H ( x )+ I ) is convex in x , then for any fixed x ∈ K , v ∈ R d , log det( H ( x + tv )+ I d ) is convex in t (for t sufficiently close to 0 ).

For t sufficiently closely to 0 , define f i ( t ) := λ i ( H ( x + tv )) . Then

and the condition that log det H ( x ) is convex is equivalent to

Now

The condition that log det( H ( x ) + I d ) is convex is equivalent to

Suppose d = 2 , f 1 (0) = 1 , f ′ 1 (0) = 0 , f ′′ 1 (0) = 1 , f 2 (0) = 4 , f ′ 2 (0) = 0 , f ′′ 2 (0) = -3 . Then

This gives an example for which log det H ( x ) is convex in a region but log det( H ( x ) + I d ) is not.

I Convexity of Regularized Volumetric Barrier Function

In this section, we prove the convexity of log det of the regularized volumetric barrier function. This is crucial, as the convexity proof for regularized Lee-Sidford barrier is identical up to replacing leverage score by Lewis weights.

I.1 Definitions

Definition I.1. Let A ∈ R n × d . Let b ∈ R n . For each i ∈ [ n ] , we define

For each i ∈ [ n ] , we define

For each i ∈ [ n ] , for each l ∈ [ n ]

Definition I.2. We define H ( x ) ∈ R d × d as follows

Definition I.3. For each j ∈ [ d ] , we define P j ( x ) ∈ R n × n as follows

For each j ∈ [ d ] , for each k ∈ [ d ] , we define P j,k ( x ) ∈ R n × n as follows:

In the previous sections, we mainly use notation σ ∗ , ∗ ( A x ) . For simplicity, from this section, we will use notation Q ( x ) instead.

Definition I.4. We define Q ( x ) ∈ R n × n as follows

Proof. The proof is straightforward from definition.

We list a number of standard calculus results.

Fact I.6. We define the following quantities:

Let s x = Ax -b ∈ R n ;

Let S x = diag( s x ) ∈ R n × n denote the diagonal matrix by putting s x on the diagonal;

Let A ∗ ,j denote the j -th column of matrix A ∈ R n × d ;

Let A x = S -1 x A ;

Let a ⊤ x,i denote the i -th row of A x for each i ∈ [ n ] .

Then, we have for each j ∈ [ d ]

Part 1.

Part 2.

Part 3.

Fact I.5. We have

Definition I.7. We define ˜ Σ( A x ) ∈ R n × n and ˜ σ ∗ , ∗ ( A x ) ∈ R n × n as

Definition I.8. We define F ( x ) as follows:

Definition I.9. We define ∂ ∂x ∂F ∂x ⊤ ∈ R d × d to be

where

I.2 Gradient of σ i,j

Fact I.10 (First derivative of leverage score) . We define the following quantities:

For each j ∈ [ d ] , let A ∗ ,j ∈ R n denote j -th column of A ;

Let A x, ∗ ,j ∈ R n denote the j -th column of A x ∈ R n × d ;

Let Σ( A x ) ∈ R n × n denote a diagonal matrix where ( i, i ) -th entry is σ i,i ( A x ) ;

Let σ ∗ ,i ( A x ) ∈ R n denote a column vector of σ ∗ , ∗ ( A x ) ∈ R n × n .

Then we have for each j ∈ [ d ]

Part 1. For each i ∈ [ n ] ,

Part 2. For each i ∈ [ n ] , l ∈ [ n

Proof. We know

For the first term in the above, we have

For the second term, we have

Similarly, we can prove for ∂σ i,l ( A x ) ∂x j .

I.3 Gradient of σ

Lemma I.11. Let σ and Σ be defined as Definition I.1. Then, we have

Part 1

Part 2.

Part 3.

Proof. It follows from Fact I.10.

I.4 Gradient for H ( x )

Lemma I.12. Recall the definitions of the following quantities:

Let H ( x ) ∈ R d × d be defined as Definition I.2.

For each j ∈ [ d ] , let P j ( x ) ∈ R n × n be defined as Definition I.3.

For each j ∈ [ d ] , for each k ∈ [ d ] , let P j,k ( x ) ∈ R n × n be defined as Definition I.3.

Then, we have

Part 1. For each j ∈ [ d ]

Proof. Proof of Part 1. We have

Proof of Part 2. Then, we have

where the first step follows from Part 1 , the second step follows from the sum rule. For the first term of Eq. (22), we have

For the second term of Eq. (22), we have

where

By combining Eq. (22), Eq. (23), and Eq. (24), we have

Thus, we complete the proof.

I.5 Hessian of H ( x )

Lemma I.13. Recall the definitions of following quantities:

Then, we have

Part 1. For each j ∈ [ d ] and for each k ∈ [ d ]

where we define some local d × d size matrix variables

Proof. By Part 2 of Lemma I.12, we can show that

where B 2 can be decomposed further as

where the second step follows from simple algebra, the third step follows from the definition of diag( · ) , the fourth step follows from simple algebra, and the last step follows from the definition of diag( · ) . For B 3 , consider the following:

where the second step follows from simple algebra, the third step follows from the definition of diag( · ) , the fourth step follows from simple algebra, and the last step follows from the definition of diag( · ) .

Finally for B 4 ,

Eventually, we can show Hessian is C 1 + · · · + C 5 . The reason is following:

For the term C 1 , we have

where B 2 , 2 is the second term of B 2 , B 3 , 2 is the second term of B 3 and B 4 , 5 is the last term of B 4 . For the term C 2 , we have

For the term C 3 , we have

For the term C 4 , we have

For the term C 5 , we have

I.6 Gradient and Hessian of F ( x )

Lemma I.14. Let F ( x ) ∈ R be defined as Definition I.8. Then, we have

Part 1.

Part 2

Proof. Proof of Part 1. We have

where the first step follows from the definition of F ( x ) (see Definition I.8), the third step follows from the basic calculus rule, and the last step follows from ∂I d ∂x j = 0 .

Proof of Part 2. We have

where the first step follows from Part 1 , the second step follows from the product rule, and the last step follows from simple algebra.

Using standard algebraic tools in literature [Lee and Sidford, 2019], we can show that,

Lemma I.15. We can rewrite it as follows and

Thus, the function F is convex in x .

I.7 Quantity I for Hessian of F

where

Proof. It comes from P

i,j and the following lemma.

Lemma I.17. Let H 1 ∈ R d × d be Hessian where each entry is

Then, we have

Proof. We can rewrite ( H 1 ) j,k as follows

where the first step follows from the Lemma statement, the second step follows from the fact that all of ( H ( x ) + I d ) -1 , diag( A x, ∗ ,j ) , Σ( A x ) , and diag( A x, ∗ ,k ) are diagonal matrices, the third step follows from the definition of ˜ σ ∗ , ∗ (see Definition I.7), and the last step follows from the definition of ˜ Σ( A x ) (see Definition I.7).

Thus, we have

Lemma I.18. Let H 2 be defined as

Then, we have

Proof. We have

where the first step follows from the Lemma statement, the second step follows from the definition of ˜ Σ( A x ) (see Definition I.7), the third step follows from the definition of 1 ⊤ n , and the last step follows from the definition of Σ( A x ) .

Thus, we have

Then, we

Proof. Then, we have

where the first step follows from the Lemma statement, the second step follows from the definition of ˜ Σ( A x ) (see Definition I.7), and the last step follows from the property of diag( · ) .

Thus, we have

Then, we

Proof. This proof is very similar to Lemma I.19, so we omit the details here.

Lemma I.21. Let H 5 be defined as

Then, we have

Proof. We have

where the first step follows from the Lemma statement, the second step follows from the definition of ˜ Σ( A x ) (see Definition I.7), the third step follows from the definition of tr[ · ] , and the last step follows from the fact that ˜ Σ( A x ) is a diagonal matrix (see Definition I.7).

Thus, we have

I.8 Quantity II for Hessian of F

Recall that

=

=

-

2

diag(Σ(

+2

A

A

⊤

x

A

⊤

x

A

diag((

⊤

x

Q

diag(2

Q

Lemma I.22. For the second item, we have

Proof. The proof follows by combining the following lemmas.

Lemma I.23. Let ( G 1 ) j,k

Proof. We can show

Thus, we have

◦

2

◦

x

2

(

x

)

)

A

(

x

x,

∗

,j

)

-

-

)

A

Σ(

4Σ(

x

A

A

x

x

))

))

A

A

x, x,

∗

,j

∗

)

,j

)

A

A

x

Then, we have

Then, we have

The proofs of G 2 , G 3 , G 4 are similar to G 1 . So we omit the details here.

Lemma I.24. Let ( G 2 ) j,k

( G 2 ) j,k = tr[( H + I d ) -1 A ⊤ x diag( Q ◦ 2 ( x ) A x, ∗ ,j ) A x ( H + I d ) -1 A ⊤ x diag(Σ( A x ) A x, ∗ ,k ) A x ] Then, we have

x

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main result of this paper is a class of Dikin walks for log-concave sampling from convex bodies with self-concordant barrier functions. The abstract and introduction reflect the complexity of these walks, and the remainder of the paper is dedicated to prove these assertions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in the last paragraph of Section 4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are clearly stated in the statement of the theorems: convex body contained in a ball of radius R , density is log-Lipschitz with parameter L , the walk starts from a w -warm starting point. See Theorem 1.1, 1.2 and 1.3. The proofs of Theorem 1.1 and 1.2 results can be found in Section C and D for mixing time, and Section F for per iteration cost. Proofs of Theorem 1.3 can be found in Section G.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully read Code of Ethics and the paper is adhered to those principles.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The potential societal impact is discussed in the last paragraph of Section 4.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The results in this paper are theoretical with no experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The authors propose a sampling framework from a convex body with a barrier.

STRENGTHS:
The main strength of the paper lies at the novelty and the significance of the results, especially that of Theorem 1.1.

WEAKNESSES:
See questions below.

QUESTIONS:
Page 1:
Can you please motivate the problem a bit better? For a broader CS audience? :)

Page 2:
Can you please give some intuition about the benefit of using the Hessian in the design of your walk?

Page 3:
Can you provide further details about the hardness of constructing the Hessian of the universal barrier?

Page 4:
I do not understand Lines 154 -- 156.

Page 5:
Line 15:
Why is this probability 0.5 \min\{\tau, 1\}?

Page 6:
Lines 196, 202:
Can you please give some intuition about the symmetry and convexity properties? :)

Page 7:
Line 240:
I do not understand this math display.

Can you please elaborate on Equation (1)?

Page 8:
Lines 285 -- 288:
I do not understand this sentence :)

Can you please give some details about TensorSRHT?

Page 9:
Can you please provide a more extensive future work? Thanks!

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-06-26

comments: SUMMARY:
The paper gives a Markov chain Monte Carlo algorithm for estimating the probability of a high dimensional polytope under a log-concave probability distribution. The algorithm improves the mixing time of previous results, while maintaining the best known per-iteration cost. More specifically, in conjunction with a known self-concordant barrier function, they use a cheap spectral approximation of the Hessian matrix required in an iteration, and they modify the Markov chain to converge more quickly despite the approximation.  A second result applies similar ideas to spectrahedrons, and there the improvement over past results is both the asymptotic number of iterations and the asymptotic computational complexity of each iteration.

STRENGTHS:
This is an important and fundamental problem that has been researched extensively recently, and the result improves upon the best known algorithms.

WEAKNESSES:
The improvement is somewhat incremental, but nonetheless significant.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-15

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors give improved mixing times for a Markov chain whose goal is to approximately sample from distributions whose densities are proportional to $\\exp(-f(x))$, where $f$ is $L$-Lipschitz and convex, restricted to a convex set defined by the intersection of $n$ hyperplanes that lies in a Euclidean ball of radius $R$. The Markov chain itself is an approximate variant of the Dikin walk that tolerates using spectral approximations to the Hessian instead of the exact Hessian for drawing the next sample.\n\nThe main result is as follows. Given a subroutine to $1/d$-spectrally approximate the Hessian of the barrier that runs in time $C_g$ and given that the barrier itself is $\\nu$-self concordant, the authors give a Markov chain whose iteration complexity to converge to a $\\delta$-TV-approximate distribution to the target is $\\widetilde{O}(\\nu d + d L^2 R^2)\\log(1/\\delta)$. Each step has runtime $C_g + d^{\\omega}$. \n\nThis general framework recovers some of the best known mixing time bounds for special cases of the studied problem, including sampling from the uniform distribution over a convex body. Moreover, for the setting considered, as far as I can tell, this is the first guarantee whose iteration complexity does not depend on the number of constraints defining the polytope (this seems to get replaced by the self-concordance parameter of the barrier, which can be made to be $\\widetilde{O}(d)$ using more sophisticated barriers than the log-barrier).\n\nThe ideas also transfer to giving mixing times for the problem of sampling uniformly from the constraint set of a covering SDP. Again, a key part of the contribution is that the walk can tolerate using spectral approximations to the Hessian of the barrier (in this case, the log det barrier) instead of requiring it exactly. \n\nThe main technical insight is that there are a few properties that one needs the barrier to satisfy in order to give the $O(\\nu d)$ dependence in the mixing time. They consist of a symmetry assumption, a bounded local norm condition, and that the barrier is convex under regularizing it with an identity matrix (see page 6). Under these assumptions, the authors prove that their variant of the Dikin walk satisfies the promised mixing time (the argument is executed in Appendices B, C, D).\n\nSTRENGTHS:\nThe problem addressed in this work is an important one to advance our understanding of the geometry of polytopes. The conditions under which mixing times can be obtained are pretty general, and as the authors show, only depend on certain natural properties of the barrier function that is being used. The results also imply significant quantitative improvements over prior work (in particular the result about uniformly sampling from polytopes with an iteration complexity that is independent of the number of constraints when $n$ is much larger than $d$).\n\nWEAKNESSES:\nThere are a few questions I have (see below).\n\nBesides that, a very minor comment -- in Section 2.3, the Lewis weight optimization program as written is not actually convex. But, there is a convex formulation for all $p > 0$ given in [LS19].\n\nQUESTIONS:\nCan one obtain better mixing times under more structural assumptions on the convex set (e.g. symmetry, some notion of uniform convexity, etc)? Or is this an open question?\n\nDo you think it is possible to get runtime improvements by intelligently recycling Hessian computations? Then, the amortized runtime of finding each approximate Hessian could improve over the stated $C_g + d^{\\omega}$. In particular, self concordance implies some kind of Hessian stability (for any two nearby points, the Hessians at those points are spectrally close) -- so, could you hope that the Hessians of the barrier aren't changing too much between two consecutive steps?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}