SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Thus, from a  $w$ -warm start, the soft-threshold Dikin walk takes at most  $O((md + dL^2 R^2) \times \log^{(w / \delta)})$  Markov chain steps to obtain a sample from  $\pi$  with total variation error  $\delta > 0$ , where each step takes  $O(md^{\omega - 1})$  arithmetic operations, and one function evaluation.
Remark 3.1 (Optimality of  $\ell_2$ -regularization) For the class of functions considered in our paper, we conjecture that an  $\ell_2$ -regularizer that does not depend on  $\theta$  is optimal. This is because we consider the class of functions which are  $L$ -Lipschitz or  $\beta$ -smooth with respect to the  $\ell_2$ -norm, and our bound on  $L$  or  $\beta$  does not depend on  $\theta$ . Moreover, we only have access to the function  $\theta$  through an oracle which returns the value of  $f$  at any given point  $\theta$  but does not tell us how  $f$  changes at nearby points.
Algorithm 1: Soft-threshold Dikin walk
Input:  $m,d\in \mathbb{N}$ $A\in \mathbb{R}^{m\times d}$ $b\in \mathbb{R}^m$  , which define  $K\coloneqq \{\theta \in \mathbb{R}^d:A\theta \leq b\}$  Input: Oracle returning the value of a convex  $f:K\to \mathbb{R}$  . Initial point  $\theta_0\in \mathrm{Int}(K)$    
1 Hyperparameters:  $\alpha >0$ $\eta >0$  , and  $T\in \mathbb{N}$    
2 Set  $\theta \gets \theta_{0}$    
3 for  $i = 1,\ldots ,T$  do   
4 Sample a point  $\xi \sim N(0,I_d)$    
5 Set  $H(\theta)\leftarrow \sum_{j = 1}^{m}\frac{a_ja_j^\top}{(a_j^\top\theta - b_j)^2}$    
6 Set  $\Phi (\theta)\gets \alpha^{-1}H(\theta) + \eta^{-1}I_d$    
7 Set  $z\gets \theta +\Phi (\theta)^{-1 / 2}\xi$    
8 if  $z\in \operatorname {Int}(K)$  then   
9 Set  $H(z)\leftarrow \sum_{j = 1}^{m}\frac{a_ja_j^\top}{(a_j^\top z - b_j)^2}$    
10 Set  $\Phi (z)\gets \alpha^{-1}H(z) + \eta^{-1}I_d$    
11 Accept  $z$  with probability  $\frac{1}{2}\min \left(\frac{e^{-f(z)}\sqrt{\operatorname*{det}(\Phi(z))}}{e^{-f(\theta)}\sqrt{\operatorname*{det}(\Phi(\theta))}}\times e^{\| z - \theta \|_{\Phi (\theta)}^2 -\| \theta -z\|_{\Phi (z)}^2},1\right)$    
12 else   
13 Reject  $z$    
14 end   
15 end   
16 Output  $\theta$
In Theorem 2.1, we set the step size hyperparameters  $\alpha = \frac{1}{(10^5d)}$  and  $\eta = \frac{1}{(10^4dL^2)}$  if  $f$  is  $L$ -Lipschitz, and the number of steps to be  $T = 10^{9}\left(2m\alpha^{-1} + \eta^{-1}R^{2}\right)\times \log (w / \delta)$ . When  $f$  is  $\beta$ -smooth (but not necessarily Lipschitz), we instead set  $\alpha = \frac{1}{(10^5d)}$  and  $\eta = \frac{1}{(10^4d\beta)}$ .
In many applications, a bound on  $L$  or  $\beta$  can be calculated analytically, allowing one to set  $\alpha, \eta$  as above. This includes, e.g., applications to training Bayesian or differentially private logistic regression models (or other generalized linear models such as support vector machines). When a bound on  $L$  or  $\beta$  is not known, one can in practice set  $\alpha, \eta$  by hand such that the average acceptance probability is  $\geq \Omega(1)$ .
4 Conclusions, limitations, and future work
Our result improves on the runtime bounds of a line of previous work, for the problem of sampling from several classes of log-Lipschitz or log-smooth log-concave distributions on a polytope (see Table 1). To the best of our knowledge, this is the first result to introduce regularized barrier functions that simultaneously take into account the geometry of both the constraint polytope and the Lipschitz or smoothness property of a target logconcave function. These barrier functions may be of independent interest for sampling or optimization.
On the other hand, we note that [27] give an implementation of the Dikin walk in the special case where  $f$  is constant, where the (average) cost of computing the Hessian matrix of the log-barrier of the polytope  $K \coloneqq \{\theta \in \mathbb{R}^d : A\theta \leq b\}$  at each step of the walk is improved to roughly  $O(d^{2} + \mathrm{nnz}(A))$  arithmetic operations, where  $\mathrm{nnz}(A)$  denotes the number of non-zero entries of  $A$ . Whether this improvement in the per-step computation time can be achieved for the problem of computing the regularized barrier functions used in our algorithm, in the more general setting where  $f$  is  $L$ -Lipschitz or  $\beta$ -smooth, is an interesting open problem.
Moreover, we note that our bounds are polynomial in  $L$  or  $\beta$ , yet there are algorithms for sampling from log-concave distributions  $\propto e^{-f}$  which do not assume  $f$  is  $L$ -Lipschitz or  $\beta$ -smooth. Thus, another interesting open problem is whether one can obtain runtime bounds for a version of the Dikin walk which do not require  $f$  to be Lipschitz or smooth.
Our results have applications to Bayesian inference and differentially private optimization. Bayesian inference can lead to algorithms with better generalization properties and quantification of uncertainty, and differential privacy guarantees are important to protecting the privacy of individuals in medical and other sensitive datasets. Thus, we believe our results will have positive societal impacts, and do not anticipate any negative impacts to society.
Acknowledgments. NV was supported in part by an NSF CCF-2112665 award. OM was supported in part by an NSF CCF-2104528 award and a Google Research Scholar award.
[1] David Applegate and Ravi Kannan. Sampling and integration of near log-concave functions. In Proceedings of the twenty-third annual ACM symposium on Theory of computing, pages 156-163, 1991.
[2] Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464-473. IEEE, 2014.
[3] Nicolas Brosse, Alain Durmus, Éric Moulines, and Marcelo Pereyra. Sampling from a log-concave distribution with compact support with proximal Langevin Monte Carlo. In Conference on Learning Theory, pages 319-342. PMLR, 2017.
[4] Sébastien Bubeck and Ronen Eldan. The entropic barrier: a simple and optimal universal self-concordant barrier. In Conference on Learning Theory, pages 279-279. PMLR, 2015.
[5] Sebastien Bubeck, Ronen Eldan, and Joseph Lehec. Finite-time analysis of projected Langevin Monte Carlo. In Advances in Neural Information Processing Systems, pages 1243-1251. Citeseer, 2015.
[6] Bradley P Carlin and Siddhartha Chib. Bayesian model choice via markov chain monte carlo methods. Journal of the Royal Statistical Society: Series B (Methodological), 57(3):473-484, 1995.
[7] Apostolos Chalkis, Vissarion Fisikopoulos, Marios Papachristou, and Elias Tsigaridas. Truncated log-concave sampling for convex bodies with reflective Hamiltonian Monte Carlo. ACM Transactions on Mathematical Software, 2023.
[8] Yuansi Chen, Raaz Dwivedi, Martin J Wainwright, and Bin Yu. Vaidya walk: A sampling algorithm based on the volumetric barrier. In 2017 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 1220-1227. IEEE, 2017.
[9] Arnak S Dalalyan and Lionel Riou-Durand. On sampling from a log-concave density using kinetic Langevin diffusions. Bernoulli, 26(3):1956-1988, 2020.
[10] John Duchi. Derivations for linear algebra and optimization. Berkeley, California, 3:2325-5870, 2007.
[11] Alain Durmus, Szymon Majewski, and Blazej Miasojedow. Analysis of Langevin Monte Carlo via convex optimization. J. Mach. Learn. Res., 20:73-1, 2019.
[12] Alain Durmus and Eric Moulines. Nonasymptotic convergence analysis for the unadjusted Langevin algorithm. Annals of Applied Probability, 27(3):1551-1587, 2017.
[13] Raaz Dwivedi, Yuansi Chen, Martin J Wainwright, and Bin Yu. Log-concave sampling: Metropolis-hastings algorithms are fast! In Conference on Learning Theory, pages 793-797. PMLR, 2018.
[14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407, 2014.
[15] Alan Frieze and Ravi Kannan. Log-Sobolev inequalities and sampling from log-concave distributions. The Annals of Applied Probability, 9(1):14-26, 1999.
[16] Alan Frieze, Ravi Kannan, and Nick Polson. Sampling from log-concave distributions. The Annals of Applied Probability, pages 812-837, 1994.
[34] László Lovász and Santosh Vempala. The geometry of logconcave functions and sampling algorithms. Random Structures & Algorithms, 30(3):307-358, 2007.
[35] Oren Mangoubi and Nisheeth K. Vishnoi. Sampling from log-concave distributions with infinity-distance guarantees. In Advances in Neural Information Processing Systems, 2022.
[36] Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07), pages 94-103. IEEE, 2007.
[37] Hariharan Narayanan. Randomized interior point methods for sampling and optimization. The Annals of Applied Probability, 26(1):597-641, 2016.
[38] Hariharan Narayanan and Alexander Rakhlin. Efficient sampling from time-varying log-concave distributions. The Journal of Machine Learning Research, 18(1):4017-4045, 2017.
[39] Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming. SIAM, 1994.
[40] Mark Rudelson and Roman Vershynin. Hanson-wright inequality and sub-gaussian concentration. Electronic Communications in Probability, 8(82):1-9, 2013.
[41] Sushant Sachdeva and Nisheeth K Vishnoi. The mixing time of the Dikin walk in a polytope—a simple proof. Operations Research Letters, 44(5):630-634, 2016.
[42] Adil Salim and Peter Richtárik. Primal dual interpretation of the proximal stochastic gradient Langevin algorithm. Advances in Neural Information Processing Systems, 33:3786-3796, 2020.
[43] Mervyn J Silvapulle and Pranab Kumar Sen. Constrained statistical inference: Order, inequality, and shape constraints. John Wiley & Sons, 2011.
[44] Guo-Liang Tian, Man-Lai Tang, Hong-Bin Fang, and Ming Tan. Efficient methods for estimating constrained parameters with applications to regularized (lasso) logistic regression. Computational statistics & data analysis, 52(7):3528-3542, 2008.
[45] Pravin M Vaidya and David S Atkinson. A technique for bounding the number of iterations in path following algorithms. In Complexity in Numerical Optimization, pages 462-489. World Scientific, 1993.
[46] Andre Wibisono. Sampling as optimization in the space of measures: The Langevin dynamics as a composite optimization problem. In Conference on Learning Theory, pages 2093-3027. PMLR, 2018.
[47] Yuan Zhang, Jonathan N Hofmann, Mark P Purdue, Shili Lin, and Swati Biswas. Logistic bayesian lasso for genetic association analysis of data from complex sampling designs. Journal of Human Genetics, 62(9):819-829, 2017.
1 Introduction 1
2 Results 3
3 Overview of proof of Theorem 2.1 6
4 Conclusions, limitations, and future work 10
A Additional discussion with related work 15
B Application to Bayesian Lasso logistic regression 16
C Infinity-distance sampling 16
D Differentially private optimization 17
E Proof of Theorem 2.1 18
E.1 Bounding the number of arithmetic operations 18
E.2 Bounding the step size 19
E.3 Bounding the acceptance probability 21
E.4 Bounding the conductance 29
F Extension to general barrier functions? 33
F.1 Proof of Lemma F.2 34
F.2 Lower bounds for self-concordance parameter 35
A Additional discussion with related work
The problem of sampling from the uniform distribution when  $K$  is a polytope given by a set of inequalities, and the distance to  $\pi$  is measured in total-variation (TV) distance has been widely studied; see, e.g., [24, 37, 30, 8, 27]. Also widely studied is the problem of sampling from an unconstrained log-concave distribution. One direction of study includes works that require the log-density  $\pi$  to be  $L$ -log-Lipschitz or  $\beta$ -log-smooth on all  $\mathbb{R}^d$  for some  $L, \beta > 0$ , and give bounds on the distance to  $\pi$  in terms of TV distance [13], Wasserstein distance [12, 9, 42], and Kullback-Leibler (KL) divergence [46, 11].
Further, [5, 3] provide versions of the Langevin dynamics for sampling from a log-concave distribution  $\pi \propto e^{-f}$  on  $K$  where  $f$  is  $L$ -Lipschitz and  $\beta$ -smooth and  $K$  is given by a projection oracle- when, e.g.,  $K$  is contained in a ball of radius  $O(1)$ , it contains a ball of radius  $\Omega (1)$ , and  $L = \beta = O(1)$ , [5] give a bound of roughly  $O(d^{9} / \delta^{22})$  gradient and projection oracle calls to sample from  $\pi$  with TV error  $\delta >0$ , while [3] give a bound of roughly  $O(d^{5} / \delta^{6})$  gradient and projection oracle calls. [21] provide an algorithm for sampling from distributions  $\propto e^{-f}$  where  $f$  is both  $L$ -Lipschitz and  $\mu$ -strongly convex (that is,  $f(\theta)$  is the sum of a convex function  $\hat{f} (\theta)$  and the quadratic function  $(\mu /2)\| \theta \| ^2$  for some  $\mu >0$ ) on a convex body  $K$  in roughly  $\tilde{O} ((L^2 /\mu)\log^2 (d / \delta))$  membership and function oracle calls (see also [7], who require  $f$  to be  $\mu$ -strongly convex for  $\mu >0$  and  $\beta$ -smooth). However, the algorithms of [21, 7] do not apply to the more general setting where  $f$  is convex and  $L$ -Lipschitz (or  $\beta$ -smooth), but not necessarily  $\mu$ -strongly convex for any  $\mu >0$ .
Detailed comparison to [38]. Compared to [38], in our version of the Dikin walk,  $z$  is sampled from a Gaussian with covariance matrix
$$
(\alpha^ {- 1} H (z) + \eta^ {- 1} I _ {d}) ^ {- 1},
$$
where  $\alpha^{-1}$  and  $\eta^{-1}$  are hyper-parameters chosen to be  $\alpha^{-1} \approx d^2$  and  $\eta^{-1} \approx dL^2$  if  $f$  is  $L$ -Lipschitz (or  $\eta^{-1} \approx d\beta$  if  $f$  is  $L$ -smooth). The "soft-threshold" regularization term  $\eta^{-1}I_d$  prevents the Markov chain from taking steps where the value of  $f$  decreases by more than  $O(1)$  w.h.p., ensuring that the term  $e^{f(\theta) - f(z)}$  in the acceptance probability is  $\Omega(1)$ . The soft-threshold regularizer is chosen to be a multiple of the identity matrix  $I_d$  since the Lipschitz condition on  $f$  is rotationally invariant-- it bounds the derivative of  $f$  by the same amount  $L$  in each direction (the same is true for the second derivative of  $f$  if  $f$  is  $\beta$ -smooth). This in turn allows our choice of scaling  $\alpha^{-1}$  for the "Dikin ellipsoid" term  $H(z)$ -- which is not in general rotationally invariant, and determined only by the geometry of the polytope rather than the geometry of the function  $f$ -- to be independent of  $L$  (or  $\beta$ ). This is in contrast to the Dikin walk in [38] where the scaling parameter for  $H(z)$  must depend on  $L$  (or  $\beta$ ) to ensure an  $\Omega(1)$  acceptance probability, which allows our Markov chain to propose steps with a larger variance than the Dikin walk in [38] in directions which are not the largest eigenvector of  $H^{-1}(z)$ .
The (inverse) covariance matrix  $\alpha^{-1}H(z) + \eta^{-1}I_d$  of our soft-threshold Dikin walk updates is the Hessian of the function  $\psi (\theta) = \alpha^{-1}\varphi (\theta) + \eta^{-1}\| \theta \| _2^2$ . This barrier function can be seen to be a Hessian of a self-concordant barrier. On the other hand, it is not the Hessian of a logarithmic-barrier function for any polytope defined by any set of inequalities. This prevents us from directly applying the analysis of the Dikin walk in the special case where  $f\equiv 0$  [37]–which relies on properties of log-barrier functions– to our soft-threshold Dikin walk on Lipschitz or smooth  $f$ . To get around this problem, we show that, while  $\psi (\theta)$  is not a log-barrier function of any polytope  $K$ , it is the limit of a sequence of log-barrier functions  $\hat{\psi}_1,\hat{\psi}_2,\ldots$  where  $\hat{\psi}_i(\theta)\rightarrow \psi (\theta)$  uniformly in  $\theta$  as  $i\to \infty$ . See Section 3 for a detailed overview of the proof.
An open problem is to obtain runtime bounds for the Dikin walk that do not require  $f$  to be Lipschitz or smooth, and/or that depend polynomially on  $\log R$ . This leads to the question of whether one can design other tractable self-concordant barriers to obtain further runtime improvements for sampling log-concave distributions on a polytope. We discuss possible extensions in Appendix F.
Explanation of bounds in [38]. [38] consider a version of the Dikin walk for sampling from a log-concave distribution  $\pi \propto e^{-f}$  on a polytope  $K$  contained in a ball of radius  $R > 0$  (and also more generally on any convex body with a self-concordant barrier function). Their version of the Dikin walk has proposed updates  $z = \alpha^{\frac{1}{2}}H^{-\frac{1}{2}}(\theta)\xi$  where  $\xi \sim N(0, I_d)$ , and acceptance probability given by  $\frac{1}{2} \times \min \left(\frac{e^{-f(z)}\sqrt{\operatorname*{det}(H(z))}}{e^{-f(\theta)}\sqrt{\operatorname*{det}(H(\theta))}}, 1\right)$ . Here  $H$  is the hessian of a barrier function for the convex body  $K$ , which may be the log-barrier or another barrier function. For a step size of  $\alpha^{\frac{1}{2}} = O(\min(\frac{1}{d}, \frac{1}{LR}))$  (Condition 4 in Section 3.1 of their paper; note that  $R$  is considered a constant in [38] and therefore does appear explicitly in their bounds), they obtain a bound on the conductance  $\phi$  of their version of the Dikin walk (Lemma 4 of their paper):
$$
\phi \geq \frac {\alpha^ {\frac {1}{2}}}{\kappa \sqrt {d}}, \tag {6}
$$
where  $\kappa$  is the self-concordance parameter of the barrier function; for the log-barrier  $\kappa = m$ , although there are other barrier functions for which  $\kappa = O(d)$ . Plugging Inequality (6) into Corollary 1.5 of [31] implies that their Dikin walk can obtain a sample from  $\pi$  with  $O(\delta)$  TV error in roughly  $\phi^{-2}\log \left(\frac{w}{\delta}\right) = O((d^5 + d^3 L^2 R^2)\log \left(\frac{w}{\delta}\right))$  steps from a  $w$ -warm start.
The number of arithmetic operations per step is  $md^{\omega - 1}$  and the number of arithmetic operations to obtain a sample with TV error  $O(\delta)$  from  $\pi$  is  $O((md^{4 + \omega} + md^{2 + \omega}L^{2}R^{2})\log (\frac{w}{\delta}))$ . If one also has that  $K$  contains a ball of radius  $r > 0$ , from a cold start one can find an initial point which is  $e^{d\log (\frac{R}{r}) + LR}$ -warm by sampling from the uniform distribution on this ball, and hence their bound is  $O((md^{4 + \omega} + md^{2 + \omega}L^{2}R^{2})(d\log (\frac{R}{r}) + LR + \log (\frac{1}{\delta})))$ .
B Application to Bayesian Lasso logistic regression
The following corollary follows directly from Theorem 2.1.
Corollary B.1 There exists an algorithm which, given  $\delta, c > 0$  and a dataset  $\{x_i\}_{i=1}^n \subseteq \mathbb{R}^d$  s.t.  $\| x_i \|_2 \leq 1 \forall i \in [n]$ , and letting  $f(\theta) = \sum_{i=1}^{n} \ell(\theta^\top x_i)$  where  $\ell(s) = \frac{1}{1 + e^{-s}}$  is the logistic loss,  $K = \{\theta \in \mathbb{R}^d : \| \theta \|_1 \leq c\}$  is the Lasso constraint, and  $\pi \propto e^{-f}$  has support  $K$ , outputs a point from a distribution  $\mu$  s.t.  $\| \mu - \pi \|_{\mathrm{TV}} \leq \delta$ , in  $O((d^2 + ndc^2)(d^{\omega + 1} + nd^2) \log(\frac{d}{\delta}))$  arithmetic operations.
C Infinity-distance sampling
In applications of sampling to differentially private optimization [36, 23, 2, 18, 28], bounds in the total variation (TV) distance are insufficient to guarantee "pure" differential privacy, and one instead requires bounds in the infinity-distance  $\mathrm{d}_{\infty}(\nu, \pi) \coloneqq \sup_{\theta \in K} |\log(\nu(\theta)/\pi(\theta))|$ ; see e.g. [14]. [35] give an algorithm that converts samples from TV bounds to those bounded in infinity-distance. Namely, given  $\varepsilon > 0$  and a sample from a distribution  $\mu$  within TV distance  $\delta \leq O\left(\varepsilon\left(R(d\log(R/r)+LR)^2/\varepsilon r\right)^{-d}e^{-LR}\right)$  of  $\pi$ , this post-processing algorithm outputs a sample from a distribution  $\nu$  with infinity distance  $\mathrm{d}_{\infty}(\nu, \pi) \leq \varepsilon$  from  $\pi$ . Plugging the TV bounds from our Theorem 2.1 into their Theorem 2.2 gives a faster algorithm to sample from a log-concave and log-Lipschitz distribution constrained to a polytope  $K$ , with  $O(\varepsilon)$  error in  $\mathrm{d}_{\infty}$ . In particular, Corollary C.1 improves the bound of  $O((m^2 d^3 + m^2 dL^2 R^2) \times [LR + d\log(^{Rd+LRd/(r\varepsilon)})])(T_f + T_K)$  arithmetic operations in their Theorem 2.1. The proof is identical to that of how their Theorem 2.2 implies their Theorem 2.1, and we refer the reader to their paper.
Corollary C.1 (Sampling with infinity-distance guarantees) There exists an algorithm which, given  $\varepsilon, L, r, R > 0$ ,  $A \in \mathbb{R}^{m \times d}$ ,  $b \in \mathbb{R}^m$  (and possibly  $\beta > 0$ ), that define a polytope  $K := \{\theta \in \mathbb{R}^d : A\theta \leq b\}$  contained in a ball of radius  $R$ , a point
3There is a typo on page 7 of [38]: "R" should be the radius of the smallest ball containing  $K$ , not the radius of the largest ball contained in  $K$ .
$a \in \mathbb{R}^d$  such that  $K$  contains a ball  $B(a,r)$  of smaller radius  $r$ , and an oracle for the value of a convex function  $f: K \to \mathbb{R}^d$ , where  $f$  is  $L$ -Lipschitz (or is both  $L$ -Lipschitz and  $\beta$ -smooth), and defining  $\pi$  to be the distribution  $\pi \propto e^{-f}$ , outputs a point from a distribution  $\nu$  such that  $\mathrm{d}_{\infty}(\nu, \pi) < \varepsilon$ . Moreover, this algorithm takes  $O\left((md + dL^2 R^2) \times [LR + d\log\left(\frac{Rd + LRd}{r\varepsilon}\right)]\right)(T_f + T_K)$  arithmetic operations, where  $T_f$  is the number of arithmetic operations to evaluate  $f$  and  $T_K = O(md^{\omega - 1})$ . If  $f$  is also  $\beta$ -Lipschitz, the number of operations is  $O\left((md + d\beta R^2) \times [LR + d\log\left(\frac{Rd + LRd}{r\varepsilon}\right)]\right)(T_f + T_K)$ .
Corollary C.1 further improves the dependence on the dimension  $d$  over [2, 35]. Specifically, [2] implies a bound of  $O((md^9 + md^5 L^4 R^4)(1 / \varepsilon^2) \times \mathrm{polylog}(1 / \varepsilon, 1 / r, R, L, d))(T_f + \tilde{T}_K)$  arithmetic operations, where  $T_f$  is the number of operations to compute  $f$  and  $\tilde{T}_K$  the operations to compute a projection oracle for  $K$ . Corollary C.1 improves on this bound by a factor of roughly  $d^{10 - \omega} / (\varepsilon^2 m)$  when, e.g., each function evaluation takes  $T_f = O(d^2)$  operations. For example, when  $m = O(d)$ , as may be the case in privacy applications, the improvement is  $d^{9 - \omega} / \varepsilon^2$ . Moreover, it improves by  $d^3$  on the bound of  $O((m^2 d^3 + m^2 d\min(L^2 R^2, \beta R^2)) \times [LR + d\log((^{Rd + LRd}) / (r\varepsilon)])(T_f + T_K)$  arithmetic operations obtained in [35].
D Differentially private optimization