SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A randomized mechanism  $h: \mathcal{D}^n \to \mathcal{R}$  is said to be  $\varepsilon$ -differentially private ( $\varepsilon$ -DP) if for any datasets  $x, x' \in \mathcal{D}$  which differ by a single datapoint, and any  $S \subseteq \mathcal{R}$ , we have that  $\mathbb{P}(h(x) \in S) \leq e^{\varepsilon} \mathbb{P}(h(x')) \in S$ ; see [14]. In the application of the exponential mechanism to  $\varepsilon$ -DP low-rank approximation of a  $p \times p$  symmetric matrix  $M$  [28] (see also [19]), one wishes to sample within infinity distance  $O(\varepsilon)$  from a log-linear distribution  $\propto e^{-f}$  on the Gelfand-Tsetlin polytope  $K \subseteq \mathbb{R}^d$  (which generalizes the probability simplex), where  $d = p^2$  and where  $K$  has  $m = d$  inequalities with diameter  $R = O(\sqrt{d})$ . In this application, the log-linear density  $f$  is (trivially) 0-smooth and  $d^2\sigma_1$ -Lipschitz, where  $\sigma_1 := \|M\|_2$  is the spectral norm of  $M$ . Thus, when applied to the mechanism of [28], our algorithm takes  $d^{4.5 + \omega}\sigma_1\log(1/\varepsilon)$  arithmetic operations. This improves by a factor of  $d^3$  on the runtime bound of  $O(d^{7.5 + \omega}\sigma_1)$  arithmetic operations implied by [35, 38], and improves by a factor of  $d^{11.5}\sigma_1^3/\varepsilon^2$  on the bound of  $O\left(d^{16}\sigma_1^4/\varepsilon^2\right)$  arithmetic operations for the bound in [2].
Consider the problem of finding an (approximate) minimum  $\hat{\theta}$  of an empirical risk function  $f:K\times \mathcal{D}^n\to \mathbb{R}$  under the constraint that  $\hat{\theta}$  is  $\varepsilon$ -differentially private, where  $f(\theta ,x)\coloneqq \sum_{i = 1}^{n}\ell_{i}(\theta ,x_{i})$ . We assume that the  $\ell_i(\cdot ,x)$  are  $\hat{L}$ -Lipschitz for all  $x\in \mathcal{D}^n$ ,  $i\in \mathbb{N}$ , for some given  $\hat{L} >0$ . In this setting, [2] show the minimum ERM utility bound under the constraint that  $\hat{\theta}$  is pure  $\varepsilon$ -DP,  $\mathbb{E}_{\hat{\theta}}[f(\hat{\theta},x)] - \min_{\theta \in K}f(\theta ,x) = \Theta (d\hat{L} R / \varepsilon)$ , is achieved if one samples  $\hat{\theta}$  from the exponential mechanism  $\pi \propto \exp (-\frac{\varepsilon}{2\hat{L}R} f)$  with infinity-distance error at most  $O(\varepsilon)$ . Plugging Corollary C.1 into the exponential mechanism, we obtain a faster algorithm for a pure  $\varepsilon$ -DP mechanism which achieves the minimum expected risk (Corollary D.1).
Corollary D.1 (Differentially private empirical risk minimization) There exists an algorithm which, given  $\varepsilon, \hat{L}, r, R > 0$ ,  $A \in \mathbb{R}^{m \times d}$ ,  $b \in \mathbb{R}^m$  (and possibly  $\hat{\beta} > 0$ ) that define a polytope  $K := \{\theta \in \mathbb{R}^d : A\theta \leq b\}$  contained in a ball of radius  $R$  and containing a ball of smaller radius  $r$ , and an empirical risk function  $f(\theta, x) := \sum_{i=1}^{n} \ell_i(\theta, x_i)$ , where each  $\ell_i: K \to \mathbb{R}$  is  $\hat{L}$ -Lipschitz (and possibly also  $\hat{\beta}$ -smooth), outputs a random point  $\hat{\theta} \in K$  which is pure  $\varepsilon$ -differentially private and satisfies  $\mathbb{E}_{\hat{\theta}}[f(\hat{\theta}, x)] - \min_{\theta \in K} f(\theta, x) \leq O(d\hat{L}R/\varepsilon)$ . Moreover, this algorithm takes  $O((md + dn^2\varepsilon^2) \times (\varepsilon n + d\log^{(nRd/(r\varepsilon))(T_f + T_K)}))$  arithmetic operations if each  $\ell_i$  is  $\hat{L}$ -Lipschitz (or  $O((md + dn\frac{\hat{\beta}}{\hat{L}}R\varepsilon) \times (\varepsilon n + d\log^{(nRd/(r\varepsilon))(T_f + T_K)}))$  if  $f$  is also  $\beta$ -Lipschitz), where  $T_f$  is the number of operations to evaluate  $f$  and  $T_K = O(md^{\omega-1})$ .
Corollary D.1 improves on [2, 35]. In particular, it improves upon the bound of  $O((d^{10}(1 / \varepsilon^2) + \varepsilon^2 n^4 d^6)\times \mathrm{polylog}(^{nRd} / r\varepsilon)))(T_f + \tilde{T}_K)$  arithmetic operations in [2] by a factor of roughly  $\max \bigl (d^{10 - \omega} / (\varepsilon^2 m),nd^5\bigl (1 / \varepsilon \bigr)\bigr)$  arithmetic operations, in the setting where the  $\ell_{i}$  are  $\hat{L}$ -Lipschitz
on a polytope  $K$  and  $f$  can be evaluated in  $T_{f} = O(nd)$  operations ( $\tilde{T}_{K}$  is the number of operations to compute a projection oracle for  $K$ ). And it improves by a factor of (at least)  $md$  on the bound of  $O((m^{2}d^{3} + m^{2}dn^{2}\varepsilon^{2}) \times (\varepsilon n + d)\log^{2}(nRd / (r\varepsilon))) \times md^{\omega - 1})$  arithmetic operations obtained in [35]. The proof is identical to that of how their Theorem 2.2 implies their Corollary 2.4, and we refer the reader to their paper. For instance, when applying the exponential mechanism to the Lasso logistic regression problem, each loss  $\ell_{i}$  is both  $\hat{\beta}$ -smooth and  $\hat{L}$ -Lipschitz for  $\hat{\beta} = \hat{L} = 1$  and  $R = O(1)$ . Thus, if  $n = d$  and  $\varepsilon < 1$ , for this problem our algorithm requires  $O(d^{3 + \omega})$  arithmetic operations, which improves by a factor of  $d^{3}$  on the bound of  $d^{6 + \omega}$  arithmetic operations implied by Corollary 2.4 in [35] and by roughly  $d^{9 - \omega}$  on the bound of  $O(d^{12})$  arithmetic operations implied by [2]. In another example, when training a support vector machine model with hinge loss and Lasso constraints under  $\varepsilon$ -differential privacy, one has that  $\ell_{i}$  is  $\hat{L}$ -Lipschitz but not smooth, for  $\hat{L} = 1$ , and  $R = O(1)$ . Thus, if  $n = d$  and  $\varepsilon < 1$ , our algorithm requires  $O(d^{4 + \omega})$  arithmetic operations, which improves by a factor of  $d^{2}$  on the bound of  $d^{6 + \omega}$  operations implied by Corollary 2.4 in [35] and by roughly  $d^{8 - \omega}$  on the bound of  $O(d^{12})$  implied by [2].
E Proof of Theorem 2.1
E.1 Bounding the number of arithmetic operations
In the following, we assume the hyperparameters  $\alpha, \eta$  satisfy  $\alpha \leq \frac{1}{10^5d}\log\left(\frac{1}{\delta}\right)$  for some choice of  $\hat{\delta} > 0$ , and either  $\eta \leq \frac{1}{10^4dL^2}$  (in the setting where  $f$  is  $L$ -Lipschitz) or  $\eta \leq \frac{1}{10^4d\beta}$  (in the setting where  $f$  is  $\beta$ -smooth). To obtain our main result (Theorem 2.1), we set  $\hat{\delta} = \frac{1}{100}$ . To obtain improved runtime bounds in the setting where  $A$  is sparse, we will set  $\hat{\delta} = \frac{1}{100(md + dL^2R^2)\log\left(\frac{w}{\delta}\right)}\delta$  when  $f$  is  $L$ -Lipschitz, and  $\hat{\delta} = \frac{1}{100(md + d\beta R^2)\log\left(\frac{w}{\delta}\right)}\delta$  when  $f$  is  $\beta$ -smooth.
Lemma E.1 Each iteration of Algorithm 1 can be implemented in  $O(md^{\omega - 1})$  arithmetic operations plus  $O(1)$  calls to the oracle for the value of  $f$ .
Proof: We go through each step of Algorithm 1 and add up the number of arithmetic operations and oracle calls for each step:
1. Line 4 samples a  $d$ -dimensional Gaussian random vector  $\xi \sim N(0, I_d)$ , which can be performed in  $O(d)$  arithmetic operations.
2. At each iteration, Algorithm 1 computes the matrix  $H(w)$  at  $w = \theta$  (line 5) and  $w = z$  (line 9):
$$
H (w) = \sum_ {j = 1} ^ {m} \frac {a _ {j} a _ {j} ^ {\top}}{(a _ {j} ^ {\top} w - b _ {j}) ^ {2}}.
$$
Computing  $H(w)$  at any  $w \in \mathbb{R}^d$  can be accomplished in  $md^{\omega - 1}$  operations as follows:
Define  $C(w)$  to be the  $d \times m$  matrix where each column  $c_{j}(w) = \frac{a_{j}}{a_{j}^{\top}w - b_{j}}$  for all  $j \in [m]$ . Then
$$
H (w) = C (w) C (w) ^ {\top}.
$$
Since  $C(w)$  is a  $d \times m$  matrix the product  $C(w)C(w)^{\top}$  can be computed in  $md^{\omega - 1}$  arithmetic operations. Thus, Lines 5 and 9 of Algorithm 1 can each be computed in  $md^{\omega - 1}$  arithmetic operations.
3. Since Lines 6 and 10 compute a sum of two  $d \times d$  matrices, Lines 6 and 10 can each be performed in  $d^2$  arithmetic operations.
4. Line 7 computes the proposed update
$$
z = \theta + \Phi (\theta) ^ {- \frac {1}{2}} \xi .
$$
Computing  $\Phi (\theta)^{-\frac{1}{2}}$  can be performed by taking the singular value decomposition of the matrix  $\Phi (\theta)$ , and then inverting and taking the square root of its eigenvalues. This can be accomplished in  $d^{\omega}$  arithmetic operations since  $\Phi (\theta)$  is a  $d\times d$  matrix. Once  $\Phi (\theta)^{-\frac{1}{2}}$  is computed, the computation  $\theta +\Phi (\theta)^{-\frac{1}{2}}\xi$  can be performed in  $d^{2}$  arithmetic operations. Thus Line 7 can be computed in  $O(d^{\omega})\leq O(md^{\omega -1})$  arithmetic operations.
5. Line 8 requires determining whether  $z \in K$ . This can be accomplished in  $O(md)$  arithmetic operations, by checking whether the inequality  $Az \leq b$  is satisfied.
6. Line 11 requires computing the determinant  $\operatorname{det}(\Phi(w))$  and  $f(w)$  for  $w = \theta$  and  $w = z$ . Computing  $\operatorname{det}(\Phi(w))$  can be accomplished by computing the singular value decomposition of  $\operatorname{det}(\Phi(w))$ , and then taking the product of the resulting singular values to compute the determinant. Since  $\Phi(w)$  is a  $d \times d$  matrix, computing the singular value decomposition can be done in  $d^{\omega}$  arithmetic operations. Computing  $f(w)$  for any  $w \in \mathbb{R}^{d}$  can be accomplished in one call to the oracle for the value of  $f$ . Thus, Line 11 can be computed in  $O(d^{\omega}) \leq O(md^{\omega - 1})$  arithmetic operations, plus two calls to the oracle for the value of  $f$ .
Therefore, adding up the number of arithmetic operations and oracle calls from all the different steps of Algorithm 1, we get that each iteration of Algorithm 1 can be computed in  $O(md^{\omega - 1})$  arithmetic operations plus  $O(1)$  calls to the oracle for the value of  $f$ .
E.2 Bounding the step size
Definition E.1 (Cross-ratio distance) Let  $u, v \in \operatorname{Int}(K)$ . If  $u \neq v$ , let  $p, q$  be the endpoints of the chord in  $K$  which passes through  $u$  and  $v$  such that the four points lie in the order  $p, u, v, q$ . Define
$$
\sigma (u, v) := \frac {\| u - v \| _ {2} \times \| p - q \| _ {2}}{\| p - u \| _ {2} \times \| v - q \| _ {2}}
$$
if  $u\neq v$  , and  $\sigma (u,v) = 0$  if  $u = v$
For convenience, we define the cross-ratio distance between any two subsets  $S_{1}, S_{2} \subseteq K$  as
$$
\sigma \left(S _ {1}, S _ {2}\right) = \min  \left\{\sigma (u, v): u \in S _ {1}, v \in S _ {2} \right\}.
$$
And for any  $S \subseteq \mathbb{R}^d$  and any density  $\nu : \mathbb{R}^d \to \mathbb{R}$  we define the induced measure:
$$
\nu^ {\star} (S) = \int_ {z \in S} \nu (z) \mathrm {d} z.
$$
Definition E.2 For any positive-definite matrix  $M \in \mathbb{R}^d \times \mathbb{R}^d$ , and any  $u \in \mathbb{R}^d$ , we define
$$
\left\| u \right\| _ {M} := \sqrt {u ^ {\top} M u}.
$$
Lemma E.2 For any  $u, v \in \operatorname{Int}(K)$ , we have
$$
\sigma (u, v) \geq \frac {1}{\sqrt {2 m \alpha^ {- 1} + \eta^ {- 1} R ^ {2}}} \| u - v \| _ {\Phi (u)}.
$$
Proof: Let  $p, q$  be the endpoints of the chord in  $K$  which passes through  $u$  and  $v$  such that the four points lie in the order  $p, u, v, q$ . Then
$$
\begin{array}{l} \sigma^ {2} (u, v) = \left(\frac {\| u - v \| _ {2} \times \| p - q \| _ {2}}{\| p - u \| _ {2} \times \| v - q \| _ {2}}\right) ^ {2} \\ \geq \max  \left(\frac {\| u - v \| _ {2} ^ {2}}{\| p - u \| _ {2} ^ {2}}, \frac {\| u - v \| _ {2} ^ {2}}{\| u - q \| _ {2} ^ {2}}, \frac {\| u - v \| _ {2} ^ {2}}{\| p - q \| _ {2} ^ {2}}\right) \\ = \max  \left(\max  _ {i \in [ m ]} \frac {\left(a _ {i} ^ {\top} (u - v)\right) ^ {2}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}}, \frac {\| u - v \| _ {2} ^ {2}}{\| p - q \| _ {2} ^ {2}}\right) \\ \geq \frac {1}{2} \max _ {i \in [ m ]} \frac {(a _ {i} ^ {\top} (u - v)) ^ {2}}{(a _ {i} ^ {\top} u - b _ {i}) ^ {2}} + \frac {1}{2} \frac {\| u - v \| _ {2} ^ {2}}{\| p - q \| _ {2} ^ {2}} \\ \geq \left(\frac {1}{2 m} \sum_ {i = 1} ^ {m} \frac {\left(a _ {i} ^ {\top} (u - v)\right) ^ {2}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}}\right) + \frac {1}{2} \frac {\| u - v \| _ {2} ^ {2}}{R ^ {2}} \\ = (u - v) ^ {\top} \left(\frac {1}{2 m \alpha^ {- 1}} \times \alpha^ {- 1} \sum_ {i = 1} ^ {m} \frac {(a _ {i} a _ {i} ^ {\top}) ^ {2}}{(a _ {i} ^ {\top} u - b _ {i}) ^ {2}} + \frac {1}{2 R ^ {2} \eta^ {- 1}} \times \eta^ {- 1} I _ {d}\right) (u - v) \\ \geq \frac {1}{2 m \alpha^ {- 1} + 2 \eta^ {- 1} R ^ {2}} (u - v) ^ {\top} \Phi (u) (u - v) \\ = \frac {1}{2 m \alpha^ {- 1} + 2 \eta^ {- 1} R ^ {2}} \| u - v \| _ {\Phi (u)} ^ {2}. \\ \end{array}
$$
Lemma E.3 For any  $u, v \in \operatorname{Int}(K)$  such that  $\| u - v\|_{\Phi (u)} \leq \frac{1}{2\alpha^{1 / 2}}$  we have that
$$
\left(1 - \alpha^ {1 / 2} \| u - v \| _ {\Phi (u)}\right) ^ {2} \Phi (v) \preceq \Phi (u) \preceq \left(1 + \alpha^ {1 / 2} \| u - v \| _ {\Phi (u)}\right) ^ {2} \Phi (v).
$$
$$
\begin{array}{l} \| u - v \| _ {\Phi (u)} ^ {2} = \alpha^ {- 1} \sum_ {i = 1} ^ {m} \frac {\left(a _ {i} ^ {\top} (u - v)\right) ^ {2}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}} + \eta^ {- 1} \| u - v \| ^ {2} \\ \geq \alpha^ {- 1} \max  _ {i \in [ m ]} \frac {\left(a _ {i} ^ {\top} (u - v)\right) ^ {2}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}}. \\ \end{array}
$$
$$
\left. \left| \left(a _ {i} ^ {\top} u - b _ {i}\right) - \left(a _ {i} ^ {\top} v - b _ {i}\right) \right| \leq \alpha^ {1 / 2} \| u - v \| _ {\Phi (u)} \left| a _ {i} ^ {\top} u - b _ {i} \right| \quad \forall i \in [ m ]. \right.
$$
Therefore, for all  $w\in \mathbb{R}^d$  we have
$$
\begin{array}{l} w ^ {\top} \left[ (1 - \alpha^ {1 / 2} \| u - v \| _ {\Phi (u)}) ^ {2} \alpha^ {- 1} \sum_ {i = 1} ^ {m} \frac {a _ {i} a _ {i} ^ {\top}}{(a _ {i} ^ {\top} v - b _ {i}) ^ {2}} + \eta^ {- 1} I _ {d} \right] w \\ \leq w ^ {\top} \left[ \alpha^ {- 1} \sum_ {i = 1} ^ {m} \frac {a _ {i} a _ {i} ^ {\top}}{\left(a _ {i} ^ {\top} u - b _ {i}\right) ^ {2}} + \eta^ {- 1} I _ {d} \right] w \\ \leq w ^ {\top} \left[ \left(1 + \alpha^ {1 / 2} \| u - v \| _ {\Phi (u)}\right) ^ {2} \alpha^ {- 1} \sum_ {i = 1} ^ {m} \frac {a _ {i} a _ {i} ^ {\top}}{\left(a _ {i} ^ {\top} v - b _ {i}\right) ^ {2}} + \eta^ {- 1} I _ {d} \right] w. \\ \end{array}
$$
E.3 Bounding the acceptance probability
Lemma E.4 Let  $\theta \in \operatorname{Int}(K)$ . Then the acceptance ratio of satisfies
$$
\mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\frac {\pi (z) \sqrt {\det (\Phi (z))}}{\pi (\theta) \sqrt {\det (\Phi (\theta))}} \times \exp \left(\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| \theta - z \| _ {\Phi (z)} ^ {2}\right) \times \mathbb {1} \{z \in K \} \geq \frac {3}{1 0}\right) \geq \frac {1}{3}.
$$
By (33) of Lemma E.9, we have
$$
\mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| z - \theta \| _ {\Phi (z)} ^ {2} \geq - \frac {2}{5 0}\right) \geq \frac {9 8}{1 0 0}. \tag {7}
$$
By Lemmas E.5, E.9, and E.8 we have that
$$
\mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\frac {\pi (z) \sqrt {\operatorname * {d e t} (\Phi (z))}}{\pi (\theta) \sqrt {\operatorname * {d e t} (\Phi (\theta))}} \times \exp \left(\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| \theta - z \| _ {\Phi (z)} ^ {2}\right) \times \mathbb {1} \{z \in K \} \geq \frac {3}{1 0}\right)
$$
$$
\begin{array}{l} \geq \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\left\{\frac {\pi (z)}{\pi (\theta)} \geq \frac {1}{2} \right\} \cap \left\{\frac {\sqrt {\det  (\Phi (z))}}{\sqrt {\det  (\Phi (\theta))}} \geq \frac {4 8}{5 0} \right\} \right. \\ \left. \cap \left\{\exp \left(\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| \theta - z \| _ {\Phi (z)} ^ {2}\right) \geq 0. 9 6 \right\} \cap \{z \in K \}\right) \\ = 1 - \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\left\{\frac {\pi (z)}{\pi (\theta)} \geq \frac {1}{2} \right\} ^ {c} \cup \left\{\frac {\sqrt {\det  (\Phi (z))}}{\sqrt {\det  (\Phi (\theta))}} \geq \frac {4 8}{5 0} \right\} ^ {c} \right. \\ \left. \cup \left\{\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| \theta - z \| _ {\Phi (z)} ^ {2} \geq - \frac {2}{5 0} \right\} ^ {c} \cup \{z \in K \} ^ {c}\right) \\ \geq 1 - \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\left\{\frac {\pi (z)}{\pi (\theta)} \geq \frac {1}{2} \right\} ^ {c}\right) - \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\left\{\frac {\sqrt {\det  (\Phi (z))}}{\sqrt {\det  (\Phi (\theta))}} \geq \frac {4 8}{5 0} \right\} ^ {c}\right) \\ - \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\| z - \theta \| _ {\Phi (\theta)} ^ {2} - \| \theta - z \| _ {\Phi (z)} ^ {2} <   - \frac {2}{5 0}\right) - \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\{z \in K \} ^ {c}\right) \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {\text {L e m m a s E . 5 , E . 9 , E . 8 , E q . 7}} {\geq} 1 - \frac {6}{1 0} - \frac {2}{1 0 0} - \frac {2}{1 0 0} - \frac {1}{1 0 0} \\ \geq \frac {1}{3}. \\ \end{array}
$$
Lemma E.5 Let  $\theta \in \operatorname{int}(K)$ . Then
$$
\mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\frac {\pi (z)}{\pi (\theta)} \geq \frac {9 9}{1 0 0}\right) \geq \frac {9 9}{1 0 0}.
$$
Proof: Since  $z \sim N(\theta, \Phi(\theta)^{-1})$ , we have that
$$
z = \theta + \Phi (\theta) ^ {- \frac {1}{2}} \xi = \theta + \left(\alpha^ {- 1} H (\theta) + \eta^ {- 1} I _ {d}\right) ^ {- \frac {1}{2}} \xi
$$
for some  $\xi \sim N(0, I_d)$ .
Since  $\alpha^{-1}H(\theta) + \eta^{-1}I_d\succeq \eta^{-1}I_d$ , and  $H(\theta)$  and  $I_{d}$  are both positive definite, we have that
$$
\eta I _ {d} \succeq (\alpha^ {- 1} H (\theta) + \eta^ {- 1} I _ {d}) ^ {- 1}. \tag {8}
$$
$$
\begin{array}{l} \left\| z - \theta \right\| _ {2} = \left\| \left(\alpha^ {- 1} H (\theta) + \eta^ {- 1} I _ {d}\right) ^ {- \frac {1}{2}} \xi \right\| _ {2} \\ = \sqrt {\xi^ {\top} \left(\alpha^ {- 1} H (\theta) + \eta^ {- 1} I _ {d}\right) ^ {- 1} \xi} \\ \begin{array}{c} \text {E q . (8)} \\ \leq \sqrt {\xi^ {\top} \eta I _ {d} \xi} \end{array} \\ = \sqrt {\eta} \| \xi \| _ {2}. \tag {9} \\ \end{array}
$$
Now, since  $\xi \sim N(0, I_d)$ , by the Hanson-wright concentration inequality for the  $\chi$ -distribution [22, 40], we have that
$$
\mathbb {P} \left(\| \xi \| _ {2} > t\right) \leq e ^ {- \frac {t ^ {2} - d}{8}} \quad \forall t > \sqrt {2 d}. \tag {10}
$$
Thus, Equations (9) and (10) together imply that
$$
\mathbb {P} \left(\| z - \theta \| _ {2} > \sqrt {\eta} \sqrt {4 0 d}\right) \leq e ^ {- \frac {2 9 d}{8}} <   \frac {1}{1 0 0}. \tag {11}
$$
Now, in the setting where  $f$  is  $L$ -Lipschitz, we have
$$
\frac {\pi (z)}{\pi (\theta)} = e ^ {- (f (z) - f (\theta))} \geq e ^ {- L \| z - \theta \| _ {2}}.
$$
$$
\begin{array}{l} \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\frac {\pi (z)}{\pi (\theta)} \geq \frac {9 9}{1 0 0}\right) \geq \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(e ^ {- L \| z - \theta \| _ {2}} \geq \frac {9 9}{1 0 0}\right) \\ = \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\| z - \theta \| _ {2} \leq \frac {\log (\frac {1 0 0}{9 9})}{L}\right) \\ \geq \begin{array}{c} 9 9 \\ \hline 1 0 0 \end{array} \\ \end{array}
$$
where the last inequality holds by Inequality (11), since  $\eta \leq \frac{1}{10^4 dL^2}$ .
Moreover, in the setting where  $f$  is differentiable and  $\beta$ -Lipschitz, we have that, since  $z - \theta$  is a multivariate Gaussian random variable,
$$
\mathbb {P} ((z - \theta) ^ {\top} \nabla f (\theta) \leq 0) = \frac {1}{2}.
$$
If  $(z - \theta)^{\top}\nabla f(\theta)\leq 0$  , we have that
$$
\begin{array}{l} f (z) - f (\theta) \leq (z - \theta) ^ {\top} \nabla f (\theta) + \beta \| z - \theta \| _ {2} ^ {2} \\ \leq \beta \| z - \theta \| _ {2} ^ {2}. \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\frac {\pi (z)}{\pi (\theta)} \geq \frac {9 9}{1 0 0}\right) \\ \geq \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\left\{\frac {\pi (z)}{\pi (\theta)} \geq \frac {9 9}{1 0 0} \right\} \cap \left\{(z - \theta) ^ {\top} \nabla f (\theta) \leq 0 \right\}\right) \\ \geq \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(e ^ {- \beta \| z - \theta \| _ {2} ^ {2}} \geq \frac {9 9}{1 0 0} | (z - \theta) ^ {\top} \nabla f (\theta) \leq 0\right) \times \mathbb {P} ((z - \theta) ^ {\top} \nabla f (\theta) \leq 0) \\ = \mathbb {P} _ {z \sim N (\theta , \Phi^ {- 1} (\theta))} \left(\| z - \theta \| _ {2} \leq \frac {\sqrt {\log (2)}}{\sqrt {\beta}} \Big | (z - \theta) ^ {\top} \nabla f (\theta) \leq 0\right) \times \mathbb {P} ((z - \theta) ^ {\top} \nabla f (\theta) \leq 0) \\ \geq \frac {9 9}{1 0 0} \times \frac {1}{2}, \\ \end{array}
$$
where the last Inequality holds by Inequality (11), since  $\eta \leq \frac{1}{10^4d\beta}$  and since the Gaussian distribution is symmetric about the  $d - 1$ -dimensional hyperplane  $(z - \theta)^{\top}\nabla f(\theta) = 0$ .
Lemma E.6 For any  $\theta, z \in \operatorname{Int}(K)$  such that  $\| \theta - z \|_{\Phi(\theta)} \leq \frac{1}{4\alpha^{1/2}}$ , we have that
$$
\| N (\theta , \Phi^ {- 1} (\theta)) - N (z, \Phi^ {- 1} (z)) \| _ {\mathrm {T V}} ^ {2} \leq 3 d \alpha \| \theta - z \| _ {\Phi (\theta)} ^ {2} + \frac {1}{2} \| \theta - z \| _ {\Phi (\theta)} ^ {2}
$$
The proof of Lemma E.6 is an adaptation of the proof Lemma 3 in [41] to our Markov chain's "soft" barrier function, and follows roughly along the lines of that proof.
Proof: Since  $\| \theta - z \|_{\Phi(\theta)} \leq \frac{1}{4\alpha^{1/2}}$ , by Lemma E.3 we have that
$$
\left(1 - \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} \Phi (z) \preceq \Phi (\theta) \preceq \left(1 + \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} \Phi (z). \tag {12}
$$
From Inequality (12) we have that
$$
\left(1 - \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} I _ {d} \preceq \Phi (z) ^ {- \frac {1}{2}} \Phi (\theta) \Phi (z) ^ {- \frac {1}{2}} \preceq \left(1 + \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} I _ {d}.
$$
Thus, denoting by  $\lambda_{i}(M)$  the  $i$ th-largest eigenvalue of any matrix  $M\in \mathbb{R}^d\times \mathbb{R}^d$ , we have by Inequality (12) that
$$
\left(1 - \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} \leq \lambda_ {i} \left(\Phi (z) ^ {- \frac {1}{2}} \Phi (\theta) \Phi (z) ^ {- \frac {1}{2}}\right) \leq \left(1 + \alpha^ {1 / 2} \| \theta - z \| _ {\Phi (\theta)}\right) ^ {2} \quad \forall i \in [ d ]. \tag {13}
$$
Now the KL-divergence between any to multivariate Gaussian distributions with any means  $\mu_1, \mu_2 \in \mathbb{R}^d$  and any covariance matrices  $\Sigma_1, \Sigma_2 \in \mathbb{R}^d$  satisfies (see e.g. Section 9 of [10] or Fact 5 in [41])
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(N \left(\mu_ {1}, \Sigma\right), N \left(\mu_ {2}, \Sigma\right)\right) \\ = \frac {1}{2} \left(\operatorname {T r} \left(\Sigma_ {1} ^ {- 1} \Sigma_ {2}\right) - d + \log \left(\frac {\det \left(\Sigma_ {1}\right)}{\det \left(\Sigma_ {2}\right)}\right) + \left(\mu_ {1} - \mu_ {2}\right) ^ {\top} \Sigma_ {1} ^ {- 1} \left(\mu_ {1} - \mu_ {2}\right)\right). \tag {14} \\ \end{array}
$$
Therefore we have that
$$
\| N (\theta , \Phi^ {- 1} (\theta)) - N (z, \Phi^ {- 1} (z) \| _ {\mathrm {T V}} ^ {2} \overset {\text {P i n s k e r ' s I n e q u a l i t y}} {\leq} 2 D _ {\mathrm {K L}} \left(N (\theta , \Phi^ {- 1} (\theta)), N (z, \Phi^ {- 1} (z))\right)
$$