SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the application of the exponential mechanism to  $\varepsilon$ -DP low-rank approximation of a  $p\times p$  symmetric matrix  $M$  [28] (see also [19]), one wishes to sample within infinity distance  $O(\varepsilon)$  from a log-linear distribution  $\propto e^{-f}$  on the Gelfand-Tsetlin polytope  $K\subseteq \mathbb{R}^d$  (which generalizes the probability simplex), where  $d = p^2$ , and where  $K$  has  $m = d$  inequalities with diameter  $R = O(\sqrt{d})$ . In this application, the log-linear density  $f$  is (trivially) 0-smooth and  $d^{2}\sigma_{1}$ -Lipschitz, where  $\sigma_{1}\coloneqq \| M\|_{2}$  is the spectral norm of  $M$ . Thus, when applied to the mechanism of [28], our algorithm takes  $d^{4.5 + \omega}\sigma_1\log (1 / \varepsilon)$  arithmetic operations. This improves by a factor of  $d^3$  on the bound of  $O(d^{7.5 + \omega}\sigma_1)$  arithmetic operations implied by [35, 38], and improves by a factor of  $d^{11.5}\sigma_1^3 /\varepsilon^2$  on the bound of  $O(d^{16}\sigma_1^4 /\varepsilon^2)$  implied by [2].
Consider the problem of finding an (approximate) minimum  $\hat{\theta}$  of an empirical risk function  $f:K\times \mathcal{D}^n\to \mathbb{R}$  under the constraint that  $\hat{\theta}$  is  $\varepsilon$ -differentially private, where  $f(\theta ,x)\coloneqq \sum_{i = 1}^{n}\ell_{i}(\theta ,x_{i})$ . We assume that the  $\ell_i(\cdot ,x)$  are  $\hat{L}$ -Lipschitz for all  $x\in \mathcal{D}^n$ ,  $i\in \mathbb{N}$ , for some given  $\hat{L} >0$ . In this setting, [2] show the minimum ERM utility bound under the constraint that  $\hat{\theta}$  is pure  $\varepsilon$ -DP,  $\mathbb{E}_{\hat{\theta}}[f(\hat{\theta},x)] - \min_{\theta \in K}f(\theta ,x) = \Theta (d\hat{L} R / \varepsilon)$ , is achieved if one samples  $\hat{\theta}$  from the exponential mechanism  $\pi \propto \exp (-\frac{\varepsilon}{2\hat{L}R} f)$  with infinity-distance error at most  $O(\varepsilon)$ . Plugging Corollary C.1 into the exponential mechanism, we obtain a faster algorithm for a pure  $\varepsilon$ -DP mechanism which achieves the minimum expected risk (Corollary D.1). Specifically, the runtime bound implied by Corollary D.1 is  $O((md + dn^{2}\varepsilon^{2})\times (\varepsilon n + d\log^{(nRd / (r\varepsilon))}(T_f + T_K))$  arithmetic operations if each  $\ell_i$  is  $\hat{L}$ -Lipschitz (or  $O((md + dn^{\frac{\beta}{L}}R\varepsilon)\times (\varepsilon n + d\log^{(nRd / (r\varepsilon))}(T_f + T_K))$  if  $f$  is also  $\beta$ -Lipschitz). This improves upon the bound of
$O((d^{10}(1 / \varepsilon^2) + \varepsilon^2 n^4 d^6)\times \mathrm{polylog}(^{nRd / r_{\varepsilon}})))(T_f + \tilde{T}_K)$  arithmetic operations in [2] by a factor of  $\max (d^{10 - \omega} / (\varepsilon^2 m),nd^5 (1 / \varepsilon))$  , when the  $\ell_{i}$  are  $\hat{L}$  -Lipschitz on a polytope  $K$  and  $f$  can be evaluated in  $T_{f} = O(nd)$  operations. And it improves by a factor of (at least)  $md$  on the bound of  $O((m^{2}d^{3} + m^{2}dn^{2}\varepsilon^{2})(\varepsilon n + d)\log^{2}\bigl {(}^{nRd / (r_{\varepsilon})}\bigr))\times md^{\omega -1})$  operations obtained in [35].
For instance, when applying the exponential mechanism to Lasso logistic regression, if e.g.  $n = d$  and  $\varepsilon < 1$ , our algorithm requires  $O(d^{3 + \omega})$  arithmetic operations, improving by  $d^3$  on the bound of  $d^{6 + \omega}$  operations implied by [35] and by roughly  $d^{9 - \omega}$  on the bound of  $O(d^{12})$  operations implied by [2]. In another example, when training a support vector machine with hinge loss and Lasso constraints under  $\varepsilon$ -DP, our algorithm requires  $O(d^{4 + \omega})$  arithmetic operations, improving by a factor of  $d^2$  on the bound of  $d^{6 + \omega}$  operations implied by [35] and by  $d^{8 - \omega}$  on the bound of  $O(d^{12})$  implied by [2]. See Appendix D for details.
Remark 2.2 (Weaker notions of differential privacy)  $\varepsilon$ -DP holds several practical advantages over weaker notions of differential privacy (DP), including  $(\varepsilon, \delta)$ -DP- a notion of differential privacy where the privacy of the mechanism is allowed to fail with probability  $O(\delta)$ . E.g., when group privacy—privacy of subsets of  $k$  individuals—must be preserved, any pure  $\varepsilon$ -DP mechanism is also  $k\varepsilon$ -DP. In contrast,  $(\varepsilon, \delta)$ -DP only implies  $(\varepsilon, ke^{(k-1)\varepsilon}\delta)$ -DP for subsets of  $k$  individuals—the failure probability grows exponentially with  $k$ .
3 Overview of proof of Theorem 2.1
Suppose we are given any polytope  $K = \{\theta \in \mathbb{R}^d : A\theta \leq b\}$  defined by  $m$  inequalities, and a convex  $f: K \to \mathbb{R}^d$  which is  $L$ -Lipschitz (or  $\beta$ -smooth) and given by an oracle which returns  $f(\theta)$  at any  $\theta \in K$ . Our goal is to sample from  $\pi \propto e^{-f}$  on  $K$  within any TV error  $\delta > 0$ , in a number of arithmetic operations and oracle calls that has a dependence on the dimension  $d$  that is a lower-order polynomial than currently available bounds for sampling from log-Lipschitz (or log-smooth) log-concave distributions, and is logarithmic in  $^1/\delta$ .
Extending the Dikin walk to sample from log-concave distributions on  $K$ . As a first step, we begin by attempting to generate samples from  $\pi$  via the (Gaussian) Dikin walk, by extending the standard analysis given in e.g. [24, 37] for the special case when  $\pi$  is uniform on  $K$  to the more general case where  $\pi$  is a log-Lipschitz log-concave on  $K$ .
In the special case where  $\pi$  is the uniform distribution on  $K$ , from any point  $\theta$  in the interior of  $K$ , the Dikin walk proposes updates  $z = \theta + \sqrt{\alpha H^{-1}(\theta)} \xi$  where  $\xi \sim N(0, I_d)$  and  $H(\theta) = \nabla^2 \varphi(\theta)$  is the Hessian of the log-barrier function  $\varphi(\theta) = -\sum_{j=1}^{m} \log (b_j - a_j^\top \theta)$  for  $K = \{\theta \in \mathbb{R}^d : A\theta \leq b\}$ , and  $\alpha > 0$  is a scalar hyperparameter. To ensure that the stationary distribution of the Dikin walk is the uniform distribution on  $K$ , if a proposed update falls in the interior of  $K$ , it is accepted with probability
$$
\min  \left(\left(\frac {\sqrt {\det  (H (z))}}{\sqrt {\det  (H (\theta))}}\right) e ^ {\| z - \theta \| _ {H (\theta)} ^ {2} - \| \theta - z \| _ {H (z)} ^ {2}}, 1\right)
$$
determined by the metropolis rule; otherwise, it is rejected. The use of the log-barrier is to ensure that the steps proposed by the Dikin walk remain inside the polytope  $K$  w.h.p.
The hyperparameter  $\alpha$  is chosen as large as possible while still ensuring the proposed steps remain in  $K$  and are accepted w.h.p. On the one hand, since the covariance matrix  $\alpha H^{-1}(\theta)$  of the proposed updates is proportional to  $\alpha$ , larger choices of  $\alpha$  allow the walk to propose larger update steps. On the other hand, if  $\alpha$  is too large, the proposed steps may fall outside the polytope and be rejected w.h.p. To see how to choose  $\alpha$ , note that for any  $\theta \in \operatorname{Int}(K)$ , the Dikin ellipsoid  $D_{\theta} = \{w : (w - \theta)^{\top}H^{-1}(\theta)(w - \theta) \leq 1\}$  is contained in  $K$ . Thus, standard Gaussian concentration inequalities which guarantee  $\| \xi \|_2 = O(\sqrt{d})$  w.h.p. imply  $\theta + \sqrt{\alpha H^{-1}(\theta)}\xi$  is in  $K$  w.h.p. if  $\alpha \leq O(1 / d)$ . Moreover, using properties of log-barrier functions, one can show the term  $\frac{\operatorname*{det}(H(z))}{\operatorname*{det}(H(\theta))}$  in the acceptance ratio is also  $\Omega(1)$  for  $\alpha = O(1 / d)$ , as is done in [24, 37]. To see why, Lemma 4.3 of [45] implies that, if  $H(\theta)$  is the Hessian of a log-barrier function for  $K$ , its log-determinant  $V(\theta) = \log (\operatorname*{det}(H(\theta)))$  satisfies
$$
\left(\nabla V (\theta)\right) ^ {\top} [ H (\theta) ] ^ {- 1} \nabla V (\theta) \leq O (d) \quad \forall \theta \in \operatorname {I n t} (K). \tag {1}
$$
Thus, if  $\alpha \leq 1 / d$ , the proposed update  $z = \theta + \sqrt{\alpha H^{-1}(\theta)} \xi$  has variance  $\Omega(1)$  in the direction  $\nabla V(\theta)$ , and (by Gaussian concentration),  $(\theta - z)^{\top} \nabla V(\theta) \leq O(1)$  w.h.p. This implies  $V(z) - V(\theta) = \log \det(H(z)) - \log \det(H(\theta)) = \Omega(1)$ , and hence  $\frac{\operatorname*{det}(H(z))}{\operatorname*{det}(H(\theta))} = \Omega(1)$ .
In [38], the Dikin walk is applied to the more general problem of sampling from a  $L$ -log-Lipschitz (or  $\beta$ -log-smooth) log-concave  $\pi \propto e^{-f}$  on  $K$  (the problem of interest in this paper). To guarantee the walk has the correct stationary distribution  $\pi$ , the Metropolis acceptance probability of the proposed updates  $z = \theta + \sqrt{\gamma H^{-1}(\theta)} \xi$ , where  $\gamma$  is a hyperparameter, gains an additional factor  $e^{-f(z)} / e^{-f(\theta)}$ . To ensure this acceptance probability remains  $\Omega(1)$ , they modify the scalar step size  $\gamma$  such that w.h.p. the walk takes steps where  $f$  changes by  $O(1)$ . To see how to choose  $\gamma$ , note that since  $f$  is  $L$ -Lipschitz,  $e^{f(z) - f(\theta)} = \Omega(1)$  if the Euclidean distance  $\|z - \theta\|_2$  is  $O(1/L)$ . This can be shown to occur w.h.p. if  $\gamma = O(1/(LR)^2)$ , since the fact that the Dikin ellipsoid is in  $K \subseteq B(0,R)$  implies that the eigenvalues of  $H(\theta)$  are all  $\leq R^2$  and hence the variance  $\gamma v^\top H^{-1}(\theta)v$  of the proposed step is  $\leq O(1/(dL^2))$  in any given direction  $v \in \mathbb{R}^d$  (where  $v$  is a unit vector). Thus, it is sufficient for them to choose  $\gamma = \min(1/d, 1/(LR)^2)$  to ensure the proposed step remains in  $K$  and is accepted w.h.p.
On the one hand, to ensure the Markov chain proposes steps that change  $f$  by an amount at most  $O(1)$  for any  $L$ -Lipschitz  $f$ , it is necessary and sufficient to ensure that from any point  $\theta \in \operatorname{Int}(K)$ , the Markov chain makes updates which fall w.h.p. inside a Euclidean ball  $B(\theta, 1/L)$  of radius  $1/L$  centered at  $\theta$ . This is because the Lipschitz condition on  $f$ :  $\| f(\theta) - f(z) \|_2 \leq L \| \theta - z \|_2$  for all  $\theta, z \in K$  holds w.r.t. the Euclidean norm  $\| \cdot \|_2$ . On the other hand, to ensure that the Markov chain remains inside the polytope  $K$ , it is sufficient to propose steps that lie inside the Dikin ellipsoid  $D_\theta = \{w : (w - \theta)^\top H^{-1}(\theta)(w - \theta) \leq 1\}$  centered at  $\theta$ . Roughly, the scalar step size  $\gamma$  is chosen such that this ellipsoid is contained inside the Euclidean ball  $B(\theta, 1/L)$ , as this guarantees that w.h.p. the steps proposed by the walk will both remain in  $K$  and will also not change the value of  $f$  by more than  $O(1)$ .
However, at many points  $\theta$  the Dikin ellipsoid  $D_{\theta}$  is such that the ratio of the largest to smallest eigenvalues of  $H^{-1}(\theta)$  may be very large (this ratio can grow arbitrarily large as  $\theta$  approaches a face of the polytope). Thus, roughly speaking, modifying the covariance matrix of the Dikin walk by a scalar constant  $(\gamma H^{-1}(\theta))$  can cause the Dikin walk to propose steps whose variance in some directions is much smaller than is required for either of the two goals: staying inside  $K$  and staying inside the ball  $B(\theta, 1/L)$  defined by the Lipschitz condition on  $f$ . This suggests that modifying the log-barrier function for  $K$  by a scalar multiple may not be the most efficient way of extending the Dikin walk to the problem of sampling from a general  $L$ -log-Lipschitz (or  $\beta$ -log-smooth) log-concave distribution on  $K$ , and that one may be able to obtain faster runtimes by making other modifications to the barrier function.
A soft-threshold regularized Dikin walk. Before we introduce our soft-threshold Dikin walk, we first note that even in the special case where  $\pi$  is the uniform distribution on  $K$ , the analysis in [38] does not recover the bounds given in [24, 37] for this special case, as [38] use a different runtime analysis geared to time-varying distributions studied in that paper. Namely, [38] imply a bound of  $O(m^2 d^3 \log (\omega / \delta))$  steps to sample from a uniform distribution on  $K$ , while [24, 37] show a bound of  $O(md \log (\omega / \delta))$ . For this reason, we first extend the analysis of the Gaussian Dikin walk given in [24, 37] for the special case of uniform  $\pi$ , to the more general problem of sampling from an  $L$ -log-Lipschitz or  $\beta$ -log-smooth log-concave density. The analysis in [24, 37] uses the cross-ratio distance metric. More specifically, if for any distinct  $u, v \in \operatorname{Int}(K)$  we let  $p, q$  be the endpoints of the chord in  $K$  passing through  $u$  and  $v$  such that the four points lie in the order  $p, u, v, q$ , the cross-ratio distance is
$$
\sigma (u, v) := \frac {\| u - v \| _ {2} \times \| p - q \| _ {2}}{\| p - u \| _ {2} \times \| v - q \| _ {2}}. \tag {2}
$$
One can show that for any  $u, v \in \mathrm{Int}(K)$ ,  $\sigma^2(u, v) \geq \left( \frac{1}{m\gamma^{-1}} \right) \| u - v \|_{\gamma^{-1}H(u)}^2$  (see e.g. [24, 37]). Thus, as the usual Dikin walk takes steps that have roughly identity covariance matrix  $I_d$  with respect to the local norm  $\| u \|_{\gamma^{-1}H(\theta)} \coloneqq \sqrt{u^\top \gamma^{-1}H(\theta)u}$ , for  $\gamma = \min(1/d, (1/(LR)^2))$ , the bound we would obtain on the number of steps until the Dikin walk is within TV error  $\delta$  from  $\pi$  is  $O(\Delta^{-1} \log(\omega/\delta)) = O((md + mL^2R^2) \log(w/\delta))$  steps from a  $w$ -warm start.
To obtain even faster bounds, we would ideally like to allow the Dikin walk to take larger steps by choosing a larger  $\gamma$ , closer to the value  $1 / d$  that is sufficient to ensure an  $\Omega(1)$  acceptance probability in the special case when  $\pi$  is uniform. Unfortunately, if e.g.  $LR \geq d$ , reducing  $\gamma$  from  $1 / d$  to  $(1 / (LR)^2)$  may be necessary to ensure the variance of the Dikin walk steps is  $O(1 / (dL^2))$  in every direction, and hence that the acceptance probability is  $\Omega(1)$ .
To get around this problem, we introduce a new variant of the Dikin walk Markov chain for sampling from any  $L$ -log-Lipschitz (or  $\beta$ -log-smooth) log-concave distributions on a polytope  $K$ , which generalizes the Dikin walk introduced in [24] for sampling from  $\pi$  in the special case when  $\pi$  is the uniform distribution on  $K$ . The main difference between our Dikin walk and the usual Dikin walk of [24] (and of [38]) is that our Dikin walk regularizes the Hessian  $\alpha^{-1}H(\theta)$  of the log-barrier for  $K$  by adding a "soft-threshold" regularization term  $\eta^{-1}I_d$  proportional to the identity matrix, where  $\eta$  is a hyperparameter and  $\alpha$  is the same hyperparameter appearing in the original Dikin walk of [24]. Since the log-barrier Hessian  $\alpha^{-1}H(\theta)$  and regularization term  $\eta^{-1}I_d$  have different scalar hyperparameters  $\alpha, \eta$ , we can set  $\alpha$  and  $\eta$  independently from each other: roughly,  $\alpha^{-1}$  is chosen to be the largest value such that the Dikin ellipsoid defined by the matrix  $\alpha^{-1}H(\theta)$  remains inside  $K$ , while  $\eta^{-1}$  is independently chosen to be the largest value such that, with high probability, the steps proposed by our Markov chain remain inside the ball  $B(\theta, 1/L)$  defined by the Lipschitz condition on  $f$ . Roughly, the addition of the soft-threshold regularization term to the log-barrier Hessian allows us to reduce the variance of the proposed steps of the Dikin walk only in those directions where a choice of  $\alpha = 1/d$  would cause the variance to be greater than  $1/(dL^2)$  while leaving the variance in other directions unchanged.
More specifically, the steps proposed by our soft-threshold Dikin walk are Gaussian with mean 0 and covariance matrix  $\Phi^{-1}(\theta) \coloneqq (\alpha^{-1}H(\theta) + \eta^{-1}I_d)^{-1}$ . The addition of the soft-threshold regularization term  $\eta^{-1}I_d$  allows us to ensure that the largest eigenvalues of the covariance matrix  $\Phi^{-1}(\theta)$  are  $\leq O(1/(dL^2))$ , without reducing (by more than a constant factor) the eigenvalues which were already  $\leq O(1/(dL^2))$ . This allows our Dikin walk to take larger steps, while still ensuring these steps are accepted w.h.p. by the Metropolis accept/reject rule for  $f$ . Taking larger steps allows our Dikin walk to converge more quickly to the target distribution  $\pi(\theta) \propto e^{-f(\theta)}$  on  $K$ .
The (inverse) covariance matrix  $\Phi (\theta)$  of the steps proposed by our Dikin walk is the Hessian of the function  $\psi (\theta) = \alpha^{-1}\varphi (\theta) + \eta^{-1}\| \theta \| ^2$  where  $\varphi (\theta) = -\sum_{j = 1}^{m}\log (b_{j} - a_{j}^{\top}\theta)$  is the log-barrier for  $K$ . The modified function  $\psi (\theta)$  can be seen to also be a self-concordant barrier for  $K$ . In the special case where  $\pi$  is the uniform distribution,  $L = 0$  and  $\eta^{-1} = 0$ , and our "soft-threshold" Dikin walk recovers the original walk of [24]. Thus, our walk generalizes the original Dikin walk to the problem of sampling from a general  $L$ -log-Lipschitz (or  $\beta$ -log-smooth) log-concave distribution on a polytope.
Bounding the number of Markov chain steps. Setting  $\eta = 1 / (dL^2)$  ensures the variance  $v^\top \alpha H^{-1}(\theta)v$  in any given unit-vector direction  $v$  of the proposed update  $z - \theta$  of our Markov chain is at most  $O\big(1 / (dL^2)\big)$ , and hence the term  $e^{f(z) - f(\theta)}$  in the Metropolis acceptance rule is  $\Omega(1)$  with high probability (Lemma E.5). Moreover, we also show that, if we choose  $\alpha = 1/d$ , the other terms in the Metropolis acceptance rule are also  $\Omega(1)$  (Lemmas E.8, E.9). While the proofs of these lemmas follow roughly the same outline as in the special case of the original Dikin walk where  $\pi$  is uniform (e.g., [24, 37]), our bound on the determinantal term  $\frac{\operatorname*{det}\Phi(z)}{\operatorname*{det}\Phi(\theta)}$  must deal with additional challenges, which we discuss in the next subsection.
To bound the number of steps required by our Markov chain, we first bound the cross-ratio distance  $\sigma(u,v)$  between any  $u,v\in \operatorname{Int}(K)$  by the local norm  $\| u - v\|_{\Phi (u)}$  (Lemma E.2):
$$
\sigma^ {2} (u, v) \geq \left(\frac {1}{2 m} \sum_ {i = 1} ^ {m} \frac {\left(a _ {i} ^ {\top} (u - v)\right) ^ {2}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}}\right) + \frac {1}{2} \frac {\| u - v \| _ {2} ^ {2}}{R ^ {2}} \geq \frac {1}{2 m \alpha^ {- 1} + 2 \eta^ {- 1} R ^ {2}} \| u - v \| _ {\Phi (u)} ^ {2}. \tag {3}
$$
Using (3) together with the isoperimetric inequality for the cross-ratio distance (Theorem 2.2 of [32]), we show that, if the acceptance probability of our Markov chain is  $\Omega(1)$  at each step, then the number of steps for our Markov chain to obtain a sample within a TV distance of  $\delta$  from  $\pi$  is  $O(\Delta^{-1}\log(\omega/\delta)) = O((md + dL^2R^2)\log(w/\delta))$  from an  $w$ -warm start,
where  $\Delta = \Omega \big(1 / (2m\alpha^{-1} + 2\eta^{-1}R^2)\big)$ . In particular, if  $m = O(d)$  and  $LR > d$ , this improves on the bound we would get for the basic Dikin walk by a factor of  $d$ .
Bounding the determinantal term in the acceptance probability. For our mixing time bound to hold, we still need to show that  $\frac{\operatorname*{det}(\Phi(z))}{\operatorname*{det}(\Phi(\theta))}$  is  $\Omega(1)$  w.h.p. We would ideally like to follow the general approach previous works [24, 37] use to show the term  $\frac{\operatorname*{det}(H(z))}{\operatorname*{det}(H(\theta))}$  in the basic Dikin walk is  $\Omega(1)$  w.h.p., which relies on the property of log-barrier functions in Inequality (1). Unfortunately, as  $\Phi(\theta)$  is not the Hessian of a log-barrier function for any system of inequalities defining  $K$ , we cannot directly apply (1) to  $\Phi(\theta)$ .
To get around this problem, we show that, while  $\Phi (\theta)$  is not the Hessian of a log-barrier function, it is in fact the limit of a sequence of matrices  $H_{i}(\theta)$ ,  $i\in \mathbb{N}$ , where each matrix  $H_{i}(\theta)$  in this sequence is the Hessian of a (different) log-barrier function for  $K$ . Specifically, for every  $j\in \mathbb{N}$ , we consider the matrices  $A^{j} = [A^{\top},I_{d},\ldots ,I_{d}]^{\top}$  where  $A$  is concatenated with  $(m_j - 1) / d$  copies of the identity matrix  $I_{d}$ , and  $m_j = m + \lfloor \alpha \eta^{-1}j^2\rfloor d$ . And we consider the vectors  $b^{j} = (b^{\top},j\mathbf{1}^{\top},\dots ,j\mathbf{1}^{\top})^{\top}$ , where  $b$  is concatenated with  $(m_j - 1) / d$  copies of the vector  $j\mathbf{1}$ , where  $\mathbf{1} = (1,\dots ,1)^{\top}\in \mathbb{R}^{d}$  is the all-ones vector. Then, for large enough  $j$ ,  $K = \{\theta \in \mathbb{R}^{d}:A^{j}\theta \leq b^{j}\}$ , and the Hessian of the corresponding log-barrier functions is
$$
{H _ {j} (\theta)} {= \sum_ {i = 1} ^ {m ^ {j}} \frac {a _ {i} ^ {j} (a _ {i} ^ {j}) ^ {\top}}{((a _ {i} ^ {j}) ^ {\top} \theta - b _ {i} ^ {j}) ^ {2}} = H (\theta) + \lfloor \alpha \eta^ {- 1} j ^ {2} \rfloor \sum_ {i = 1} ^ {d} \frac {e _ {i} e _ {i} ^ {\top}}{(e _ {i} ^ {\top} \theta - j) ^ {2}}.} {(4)}
$$
Using this fact (4), we show that, for every  $\theta \in \mathrm{int}(K)$ , every  $z \in \frac{1}{2} D_{\theta}$ , and every sequence  $\{z_j\}_{j=1}^{\infty} \subseteq \frac{1}{2} D_{\theta}$  such that  $z_j \to z$ , we have (Lemma E.7),
$$
\lim  _ {j \rightarrow \infty} \frac {\det  \left(H _ {j} \left(z _ {j}\right)\right)}{\det  \left(H _ {j} (\theta)\right)} = \frac {\det  (\Phi (z))}{\det  (\Phi (\theta))}. \tag {5}
$$
Moreover, since each  $H_{j}$  is the Hessian of a log-barrier function for  $K$ , we have that (1) does hold for  $H_{j}$  and hence (from the work of [24, 37]) that  $\frac{\operatorname*{det}(H_j(z_j))}{\operatorname*{det}(H_j(\theta))} = \Omega(1)$  w.h.p. for all  $j \in \mathbb{N}$ , if we set  $z_{j} = \theta + \alpha^{1/2} H_{j}^{-1/2}(\theta) \xi$ ,  $\xi \sim N(0, I_d)$ , and choose  $\alpha = 1/d$ . Thus, (5) implies that  $\frac{\operatorname*{det}(\Phi(z))}{\operatorname*{det}(\Phi(\theta))} = \Omega(1)$  w.h.p. as well (Lemma E.9), and hence the acceptance probability of the soft-threshold Dikin walk is  $\Omega(1)$  at each step.
Bounding the number of arithmetic operations. Since the acceptance probability is  $\Omega(1)$ , from the above discussion, the number of steps for our Markov chain to obtain a sample within TV distance  $\delta > 0$  from  $\pi$  is  $O((md + L^2 R^2) \log(w / \delta))$  from a  $w$ -warm start.
Each time our Markov chain proposes a step  $\theta +\Phi^{-1 / 2}(\theta)\xi$ , it must first sample a Gaussian vector  $\xi \sim N(0,I_d)$  which takes  $O(d)$  arithmetic operations. It must then compute the log-barrier Hessian  $H(\theta)$ , and invert the matrix  $\Phi (\theta) = \alpha^{-1}H(\theta) + \eta^{-1}I_{d}$ .
Since  $H(\theta) = C(\theta)C(\theta)^{\top}$ , where  $C(\theta)$  is a  $d \times m$  matrix with columns  $c_{j}(\theta) = \frac{a_{j}}{(a_{j}^{\top}\theta - b_{j})}$  for all  $j \in [m]$ , we can compute  $H(\theta)$  in  $md^{\omega - 1}$  arithmetic operations. And since  $\Phi(\theta)$  is a  $d \times d$  matrix, computing  $\Phi^{-1/2}(\theta)$  can be accomplished in  $d^{\omega}$  arithmetic operations by computing the singular value decomposition (SVD) of  $\Phi$ . Next, we compute the acceptance probability  $\min \left(\frac{e^{-f(z)}\sqrt{\operatorname*{det}(\Phi(z))}}{e^{-f(\theta)}\sqrt{\operatorname*{det}(\Phi(\theta))}} e^{\|z - \theta\|_{\Phi(\theta)}^{2} - \|z - \theta - z\|_{\Phi(z)}^{2}}, 1\right)$ . The determinants can be computed in  $O(d^{\omega})$  arithmetic operations via the SVD. Evaluating  $f(z)$ ,  $f(\theta)$  takes two calls to the oracle for  $f$ .