SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arun Ganesh Google Research
Mahdi Haghifam*  
University of Toronto, Vector Institute
Thomas Steinke Google DeepMind
Abhradeep Thakurta  
Google DeepMind
Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than first-order methods like gradient descent. In this work, we investigate the prospect of using the second-order information from the loss function to accelerate DP convex optimization. We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak [NP06], and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss. We then design a practical second-order DP algorithm for the unconstrained logistic regression problem. We theoretically and empirically study the performance of our algorithm. Empirical results show our algorithm consistently achieves the best excess loss compared to other baselines and is  $10 - 40 \times$  faster than DP-GD/DP-SGD for challenging datasets.
Many machine learning tasks reduce to a convex optimization problem. More precisely, given a dataset  $S_{n} = (z_{1},\ldots ,z_{n})\in \mathcal{Z}^{n}$ , a closed, convex set  $\mathcal{W}\subseteq \mathbb{R}^d$ , and a loss function  $f:\mathcal{W}\times \mathcal{Z}\to \mathbb{R}$  such that, for every  $z\in \mathcal{Z}$ ,  $f(w,z)$  is a convex function in  $w$ , our goal is to compute an approximation to  $\arg \min_{w\in \mathcal{W}}\left(\ell (w,S_n)\triangleq \frac{1}{n}\sum_{i\in [n]}f(w,z_i)\right)$ . In this paper, we are interested in the problem of designing optimization algorithms in the scenario that the dataset  $S_{n}$  contains private information. Differential privacy (DP) [DMNS06] is a formal standard for privacy-preserving data analysis that provides a framework for ensuring that the output of an analysis on the data does not leak this private information. This problem is known as private convex optimization: Design an algorithm  $\mathcal{A}: \mathcal{Z}^n\to \mathcal{W}$  that is both DP and ensures low excess loss  $\triangleq \ell (\mathcal{A}(S_n),S_n) - \min_{w\in \mathcal{W}}\ell (w,S_n)$ .
The predominant algorithm for private convex optimization is DP (stochastic) gradient descent (DP-GD/DP-SGD). This is a first-order iterative method. I.e., we start with an initial value  $w_{0}$  and iteratively update it using the gradient of the loss  $\nabla_{w_t}\ell (w_t,S_n)$  following the update rule  $w_{t + 1} = w_{t} - \eta \cdot (\nabla_{w_{t}}\ell (w_{t},S_{n}) + \xi_{t})$ , where  $\eta >0$  is a constant and  $\xi_{t}$  is Gaussian noise to ensure privacy. The number of iterations  $T$  also determines the amount of noise at each iteration, i.e., the scale of  $\xi_{t}$  is proportional to  $\sqrt{T}$  due to the composition of DP. Note that we assume  $\| \nabla_{w_t}\ell (w_t,S_n)\| \leq 1$ .
One of the major drawbacks of DP-(S)GD is slow convergence. The choice of  $(\eta, T)$  exhibits a tradeoff in terms of the excess loss: if  $\eta \cdot T$  is small, the algorithm cannot reach the optimal solution; on the other hand, the magnitude of noise at each iteration is  $\eta \cdot \sqrt{T}$ , which cannot be too large. Therefore, to maximize  $\eta \cdot T$  and minimize  $\eta \cdot \sqrt{T}$ , implementations of DP-(S)GD err on the side of
*This work was carried out while the author was an intern at Google Research, Brain Team. m.haghifam@northeastern.edu, {arunganesh, steinke, athakurta} @google.com
large  $T$  and small  $\eta$ , which results in a long, slow path to convergence. This fact has been shown theoretically as well: for the class of  $\beta$ -smooth convex functions, the optimal instantiations of DP-GD use a step size of  $\max \{1 / \sqrt{n}, \sqrt{d} / \varepsilon n\}$  [BFTG19] while in the non-private setting the stepsize for GD is set to  $1 / \beta$ . Smaller step size requires more steps (i.e. more iterations) to converge. This slowness is exacerbated by the facts that (1) DP-SGD requires large batch sizes for good performance [PHKX+23] and (2) the hyperparameter tuning of DP-(S)GD, and generally DP algorithms, is a challenging task [PS22]. Can we design a DP optimization algorithm which accelerates DP-(S)GD by choosing the step size dynamically based on the local geometry of the loss function?
We draw inspiration from the non-private optimization literature: To address the slow convergence of GD and of first-order methods in general, a class of algorithms based on preconditioning the gradient using second-order information has been developed [Nes98; NW99]. This class of algorithms is based on successively minimizing a quadratic approximation of the function, i.e.,  $w_{t + 1} = w_t + \Delta_t$  where  $\Delta_t = \arg \min_{\Delta}\{\ell (w_t,S_n) + \langle \nabla \ell (w_t,S_n),\Delta \rangle +\frac{1}{2}\langle H_t\cdot \Delta ,\Delta \rangle \} = -(H_t)^{-1}\nabla \ell (w_t,S_n)$ . Here,  $H_{t}$  is a scaling matrix which provides curvature information about the loss  $\ell (\cdot ,S_n)$  at  $w_{t}$ . For instance, Newton's method uses the Hessian  $H_{t} = \nabla^{2}\ell (w_{t},S_{n})$ . Second-order algorithms significantly improve over the convergence speed of GD, and key to their success is that at each step they automatically tune the stepsize along each dimension based on the local curvature.
In this paper, our goal is to accelerate DP convex optimization. In particular, the current paper revolves around the following questions: Can the second-order information accelerate private convex optimization while achieving optimal excess error? What is the best way to privatize second-order information, e.g., the Hessian matrix? How does the achievable privacy-utility-routine tradeoff compare with first-order methods such as DP-GD? We show that second-order information can accelerate DP optimization while achieving excess loss that matches or improves on DP-GD. Our main contributions are both theoretical and empirical:
1.1 Provably Optimal Algorithm for Strongly Convex Functions
Newton's method is a second-order optimization technique that is well-known for its rapid convergence for strongly convex and smooth functions in non-private optimization. Specifically, to achieve an excess loss of  $\alpha$ , the method only requires  $O(\log \log (1 / \alpha))$  iterations, which is provably faster than the convergence rate of any first-order method. One natural question is whether it is possible to design a second-order DP convex optimization algorithm that can achieve the optimal minmax excess error err $^{\mathrm{opt}}$  in  $O(\log \log (1 / \mathrm{err}^{\mathrm{opt}}))$  iterations? We provide an affirmative answer to this question in Section 4 by designing a second-order DP algorithm based on the cubic regularized Newton's method of Nesterov and Polyak [NP06]. At each step  $t$ , we compute a cubic upper bound  $\ell (w + \Delta ,S_n)\leq \ell (w,S_n) + \langle \nabla_w\ell (w,S_n),\Delta \rangle +\frac{1}{2}\bigl {\langle}\nabla_w^2\ell (w,S_n)\cdot \Delta ,\Delta \bigr {\rangle} + O\bigl (\| \Delta \| ^3\bigr)$ . We can minimize this cubic upper bound using any DP convex optimization subroutine; the minimizer becomes the next iterate  $w_{t + 1}$ . Since the cubic is a universal upper bound, our algorithm converges globally
1.2 Fast Practical Algorithms for DP Logistic Regression
DP logistic regression is a popular approach for private classification, with DP-GD/DP-SGD being the predominant class of algorithms for this task. As we numerically show, DP-GD/DP-SGD exhibit slow convergence for this task (See Figure 1). In Section 5, we develop a practical algorithm that injects carefully designed noise into Newton's update rule as follows:
$$
w _ {t + 1} = w _ {t} - \Psi \left(\nabla_ {w _ {t}} ^ {2} \ell \left(w _ {t}, S _ {n}\right)\right) ^ {- 1} \cdot \left(\nabla_ {w _ {t}} \ell \left(w _ {t}, S _ {n}\right) + \xi_ {t, 1}\right) + \xi_ {t, 2}. \tag {1}
$$
In particular, we inject noise twice:  $\xi_{t,1}$  privatizes the gradient and  $\xi_{t,2}$  privatizes the direction. The function  $\Psi$  modifies the Hessian to ensure that the eigenvalues are not too small; this is essential for bounding the sensitivity and, hence, the scale of  $\xi_{t,2}$ . We consider two types of modification based on eigenvalue clipping and eigenvalue adding. For eigenvalue clipping,  $\Psi (\nabla_{w_t}^2\ell (w_t,S_n))$  replaces the eigenvalues  $\lambda_{i}$  of  $\nabla_{w_t}^2\ell (w_t,S_n)$  with  $\max \{\lambda_i,\lambda_0\}$ , where  $\lambda_0 > 0$  is a carefully chosen constant. For eigenvalue adding,  $\Psi (\nabla_{w_t}^2\ell (w_t,S_n)) = \nabla_{w_t}^2\ell (w_t,S_n) + \lambda_0I$ . Using  $\Psi$  we can control the sensitivity and still have fast convergence, since important curvature information is generally
Figure 1: Excess loss versus runtime of DP-GD & our algorithms.
contained in the larger eigenvalues/vectors of the Hessian. We prove the local convergence of the update rule (1) in Section 5.3 and perform a thorough empirical evaluation Section 6. We demonstrate that our algorithm outperforms existing baselines on a variety of benchmarks.
Ensuring Global Convergence. One limitation of the update rule in Equation (1) is it does not converge globally (even without noise added for DP). That is, if the initial point  $w_0$  is too far from the optimal solution, then the iterates may diverge. To address this problem, we propose a variant of Newton's update rule where we replace the Hessian with a different form of second-order information which gives a Quadratic Upperbound (QU) on the logistic loss. This is guaranteed to converge globally, like the cubic Newton approach. And we show numerically that this algorithm converges almost as fast as the regular Newton's method in the private setting. Figure 1 shows the convergence speed of our algorithms and DP-GD in terms of real wall time for the task of logistic regression on the Covertype dataset for  $(\varepsilon, \delta) = (1, (\text{num. samples})^{-2})$ -DP. Despite DP-GD having a lower per-iteration cost, our algorithm is  $30 \times$  faster than DP-GD and achieves better excess loss.
Stochastic Minibatch Variant. We also show that our algorithms naturally extend to the minibatch setting where gradient and second-order information are computed on a subset of samples. We numerically compare it with DP-SGD and show that it has faster convergence.
DP optimization is a well-studied topic [e.g., SCS13; MRTZ17; ACGM+16; STU17; WLKC+17; INST+19; STT20; SSTT21; GTU22; GLL22; BFTG19; BST14]. Most similar to our work, Avella-Medina, Bradshaw, and Loh [ABL21] consider second-order methods for DP convex optimization. We provide a detailed comparison between our results and theirs in Remark 4.5 and Section 6 showing that our algorithms relax restrictive assumptions and provide better excess error for logistic regression.
There are numerous non-private second-order optimization methods in the literature. The choice of method depends primarily on the values of  $n$  and  $d$ . When  $n$  is large, several works consider various sampling techniques for constructing second-order information, see [RM19; XYRRM16; Erd15; EM15]. When  $d$  is large, various methods are proposed in the literature for efficient approximation of the Hessian matrix, see [ABH17; Erd15; EM15; XYRRM16; GKLR19]. There is also a family of algorithms based on the estimation of the curvature from the change in gradients. These algorithms are generally known as quasi-Newton methods stemming from the seminal BFGS algorithm [JM23].
Let  $d \in \mathbb{N}$ . For a vector  $x \in \mathbb{R}^d$ ,  $\|x\|$  denotes the  $\ell_2$  norm of  $x$ . Let  $n, m \in \mathbb{N}$ . For a matrix  $A \in \mathbb{R}^{n \times m}$ ,  $\|A\| = \sup_{x \in \mathbb{R}^m: \|x\| \leq 1} \|Ax\|$  denotes the operator norm, and  $\|A\|_F \triangleq \sqrt{\operatorname{trace}(A^T \cdot A)}$  denotes the Frobenius norm of  $A$  where trace denotes the trace operator.  $I_d \in \mathbb{R}^{d \times d}$  denotes the identity matrix.  $\langle \cdot, \cdot \rangle$  denotes the standard inner product in  $\mathbb{R}^d$ . For a convex and closed subset  $\mathcal{W} \subseteq \mathbb{R}^d$ , let  $\Pi_{\mathcal{W}}: \mathbb{R}^d \to \mathcal{W}$  be the Euclidean projection operator, given by  $\Pi_{\mathcal{W}}(x) = \arg \min_{y \in \mathcal{W}} \|y - x\|_2$ . For a (measurable) space  $\mathcal{R}$ ,  $\mathcal{M}_1(\mathcal{R})$  denotes the set of all probability measures on  $\mathcal{R}$ . Note that the statements in the paper about random variables hold almost surely. We will skip such declarations to aid readability. Let  $\mathcal{Z}$  be the data and let  $\mathcal{W} \subseteq \mathbb{R}^d$  be the parameter space. Let  $f: \mathcal{W} \times \mathcal{Z} \to \mathbb{R}$  be a loss function. Throughout the paper, we assume  $f$  is doubly continuous, a convex function in  $w$ , and  $\mathcal{W}$  is a closed and convex set. We say (1)  $f$  is  $\mathrm{L}_0$ -Lipschitz iff there exists  $\mathrm{L}_0 \in \mathbb{R}$  such that  $\forall z \in \mathcal{Z}, \forall w, v \in \mathcal{W}: |f(w, z) - f(v, z)| \leq \mathrm{L}_0 \|w - v\|$ , (2)  $f$  is  $\mathrm{L}_1$ -smooth iff there exists  $\mathrm{L}_1 \in \mathbb{R}$  such that  $\forall z \in \mathcal{Z}, \forall w, v \in \mathcal{W}: \| \nabla f(w, z) - \nabla f(v, z)\| \leq \mathrm{L}_1 \|w - v\|$ , (3)  $f$  has a  $\mathrm{L}_2$ -Lipschitz Hessian iff there exists  $\mathrm{L}_2 \in \mathbb{R}$  such that  $\forall z \in \mathcal{Z}, \forall w, v \in \mathcal{W}: \| \nabla^2 f(w, z) - \nabla^2 f(v, z)\| \leq \mathrm{L}_2 \|w - v\|$ , (4)  $f$  is a strongly convex iff for all  $w, v \in \mathcal{W}$  and  $z \in \mathcal{Z}$  we have  $f(v, z) \geq f(w, z) + (\nabla f(w, z), v - w) + \frac{\mu}{2} \|v - w\|^2$ .
3.1 Zero-Concentrated DP
For our privacy analysis, we use concentrated differential privacy [DR16; BS16], as it provides a simpler composition theorem – the privacy parameter  $\rho$  adds up when we compose.
Definition 3.1 ([BS16, Def. 1.1]). A randomized mechanism  $\mathcal{A}:\mathcal{Z}^n\to \mathcal{M}_1(\mathcal{R})$  is  $\rho$ -zCDP, iff, for every neighbouring dataset (i.e., addition or removal)  $S_{n}\in \mathcal{Z}^{n}$  and  $S_n^\prime \in \mathcal{Z}^n$ , and for every  $\alpha \in (1,\infty)$ , it holds  $\mathrm{D}_{\alpha}(\mathcal{A}(S_n)\| \mathcal{A}(S_n'))\leq \rho \alpha$ , where  $\mathrm{D}_{\alpha}(\mathcal{A}_n(S_n)\| \mathcal{A}_n(S_n'))$  is the  $\alpha$ -Renyi divergence between  $\mathcal{A}_n(S_n)$  and  $\mathcal{A}_n(S_n')$ .
We should think of  $\rho \approx \varepsilon^2$ : to attain  $(\varepsilon, \delta)$ -DP, it suffices to set  $\rho = \frac{\varepsilon^2}{4\log(1 / \delta) + 4\varepsilon}$  [BS16, Lem. 3.5].
Lemma 3.2 ([BS16, Prop. 1.3]). Assume we have a randomized mechanism  $\mathcal{A}:\mathcal{Z}\to \mathcal{M}_1(\mathcal{R})$  that satisfies  $\rho$ -zCDP, then for every  $\delta >0$ ,  $\mathcal{A}$  is  $(\rho +2\sqrt{\rho\log(1 / \delta)},\delta)$ -DP.
4 Optimal Algorithm for the Class of Strongly Convex Functions
In this section, we present a DP variant of the cubic-regularized Newton's method of Nesterov and Polyak [NP06]. To motivate the idea behind our algorithm, we revisit DP gradient descent (DP-GD) for the class of  $\mathrm{L}_0$ -Lipschitz and  $\mathrm{L}_1$ -smooth convex loss functions.
Let  $\{w_t^{\mathrm{GD}}\}_{t\in [T]}$  be the iterates of DP-GD. The smoothness of  $\ell$  lets us construct a global quadratic upper bound on the function [Nes98, Thm. 2.1.5] as follows  $\forall w\in \mathcal{W}$  and  $S_{n}\in \mathcal{Z}^{n}$ :
$$
\ell (w, S _ {n}) \leq q _ {t} (w) \triangleq \ell \left(w _ {t} ^ {\mathrm {G D}}, S _ {n}\right) + \left\langle \nabla \ell \left(w _ {t} ^ {\mathrm {G D}}, S _ {n}\right), w - w _ {t} ^ {\mathrm {G D}} \right\rangle + \frac {\mathrm {L} _ {1}}{2} \left\| w - w _ {t} ^ {\mathrm {G D}} \right\| ^ {2}. \tag {2}
$$
Then, DP-GD can be seen as a two-step process:
(Step I)  $v_{t + 1} = \underset {v}{\arg \min}q_t(v) = w_t^{\mathrm{GD}} - \mathrm{L}_1^{-1}\nabla \ell (w_t^{\mathrm{GD}},S_n),$  (Step II)  $w_{t + 1}^{\mathrm{GD}} = \Pi_{\mathcal{W}}(v_{t + 1} + \mathrm{L}_1^{-1}\xi_t)$
where  $\xi_{t} = \mathcal{N}(0,\sigma^{2}I_{d})$  with  $\sigma^2 = \frac{\mathrm{L}_0^2}{2\rho n^2}$  so that  $w_{t + 1}^{\mathrm{GD}}$  satisfies  $\rho$ -zCDP [BS16, Lem. 2.5]. That is, in each iteration of DP-GD, we find a minimum of the quadratic upper bound  $q_{t}(w)$  and then project back to  $\mathcal{W}$ . (In the unconstrained setting where  $\mathcal{W} = \mathbb{R}^d$  we do not need the second projection step.)
Consider the class of  $\mathbf{L}_2$ -Lipschitz Hessian convex loss functions. Nesterov and Polyak [NP06, Lem. 1] show that we can construct a global cubic upper bound exploiting the second-order information (i.e., Hessian) as follows: for all  $w$  and  $w_t$ ,  $\ell(w, S_n) \leq \phi_t(w)$  where
$$
\phi_ {t} (w) \triangleq \ell \left(w _ {t}, S _ {n}\right) + \left\langle \nabla \ell \left(w _ {t}, S _ {n}\right), w - w _ {t} \right\rangle + \frac {1}{2} \left\langle \nabla^ {2} \ell \left(w _ {t}, S _ {n}\right) \left(w - w _ {t}\right), w - w _ {t} \right\rangle + \frac {\mathrm {L} _ {2}}{6} \| w - w _ {t} \| ^ {3}. \tag {3}
$$
Their non-private algorithm is based on the exact minimization of  $\phi_t(w)$ , i.e., the next iterate is  $w_{t + 1} = \arg \min \phi_t(w)$ . Note that  $\arg \min \phi_t(w)$  does not admit a closed form solution, as opposed to the quadratic upper bound (2). Similar to the intuition for DP-GD on smooth loss functions (2), our algorithms in this section are based on privately minimizing  $\phi_t(w)$  at each iteration. Our algorithm is shown in Algorithm 1. In each iteration the algorithm makes an oracle call to obtain  $(\ell (w_{t},S_{n}),\nabla \ell (w_{t},S_{n}),\nabla^{2}\ell (w_{t},S_{n}))$ . Then, the algorithm calls an efficient DPSolver for privately optimizing the cubic upper bound (3). The privacy analysis of Algorithm 1 is a direct application of the composition property of zCDP [BS16, Lemma 2.3]; the output of DPSolver at each iteration satisfies  $\rho /T - zCDP$  where  $\rho$  is the total privacy budget and  $T$  is the total number of iterations.
Remark 4.1. DPSolver in Algorithm 1 does not affect the oracle complexity of Algorithm 1, as it is applied to the proxy loss  $\phi_t(w)$ , rather than the underlying loss  $\ell(w, S_n)$ .
Algorithm 1 Meta Algorithm
1: Input: training set  $S_{n}\in \mathcal{Z}^{n}$  , privacy budget  $\rho$  -zCDP, initialization  $w_0\in \mathcal{W}$  , number of iterations  $T$
2: for  $t = 0, \dots, T - 1$  do
3: Query  $\ell(w_t, S_n)$ ,  $\nabla \ell(w_t, S_n)$ ,  $\nabla^2 \ell(w_t, S_n)$
4: Construct  $\phi_t(w)$  from Equation (3)
5:  $w_{t + 1} = \mathsf{DP}\mathsf{S o l v e r}(\phi_t(w),\rho /T,w_t)$
6: Output  $w_{T}$ .
Algorithm 2 DPSolver
1: Input: function  $\phi : \mathcal{W} \to \mathbb{R} : \phi(\theta) = \ell + \langle g, \theta - \theta_0 \rangle + \frac{1}{2} \langle H(\theta - \theta_0), (\theta - \theta_0) \rangle + \frac{\mathrm{L}_2}{6} \| \theta - \theta_0 \|^3$ , privacy budget  $\tilde{\rho}$ -zCDP, initialization  $\theta_0$ .
2:  $N = \frac{2\tilde{\rho}(\mathrm{L}_0 + \mathrm{L}_1D + \mathrm{L}_2D^2)^2n^2}{(\mathrm{L}_0 + \mathrm{L}_1D)^2d},\sigma^2 = \frac{N(\mathrm{L}_0 + \mathrm{L}_1D)^2}{2\tilde{\rho}}$
3: for  $i = 0, \dots, N - 1$  do
4:  $\eta_{i} = \frac{2}{\mu(i + 2)}$
5:  $\operatorname{grad}_i = g + H(\theta_i - \theta_0) + \frac{\mathrm{L}_2}{2} \| \theta_i - \theta_0\| (\theta_i - \theta_0).$
6:  $\theta_{i + 1} = \Pi_{\mathcal{W}}(\theta_i - \eta_i(\operatorname{grad}_i + \mathcal{N}(0, \sigma^2 I_d)))$
7: Return  $\sum_{i=0}^{N-1} \frac{2i}{N(N+1)} \theta_i$
Theorem 4.2. Let  $f$  be a  $\mathbb{L}_0$ -Lipschitz,  $\mathbb{L}_1$ -smooth,  $\mathbb{L}_2$ -Lipschitz Hessian, and  $\mu$ -strongly convex function. Also, assume that  $\mathcal{W} \subseteq \mathbb{R}^d$  has finite diameter  $D$ . Let  $w^\star = \arg \min_{w \in \mathcal{W}} \ell(w, S_n)$ . Then, for every  $\rho > 0$ ,  $\beta \in (0, 1)$ , and  $S_n \in \mathcal{Z}^n$  for sufficiently large  $n$ , by setting the number of iterations in Algorithm 1 to
$$
T = \Theta \Bigl(\frac{\sqrt{\mathsf{L}_{2}}}{\mu^{3 / 4}} (\ell (w_{0},S_{n}) - \ell (w^{\star},S_{n}))^{\frac{1}{4}} + \log \log \bigl(\frac{n\sqrt{\rho}}{\sqrt{\log(1 / \beta)d}}\bigr)\Bigr),
$$
and using Algorithm 2 as DPSolver, we have the following: The output of Algorithm 1, i.e.,  $w_{T}$ , satisfies  $\rho - zCDP$  and with probability at least  $1 - \beta$
$$
\ell \left(w _ {T}, S _ {n}\right) - \ell \left(w ^ {\star}, S _ {n}\right) \leq \tilde {O} \left(\frac {d \left(\mathrm {L} _ {0} + \mathrm {L} _ {1} D\right) ^ {2} \log (1 / \beta)}{\mu \rho n ^ {2}} \cdot \left(\frac {\mathrm {L} _ {2} ^ {2} \mathrm {L} _ {0} D}{\mu^ {3}}\right) ^ {\frac {1}{4}}\right)
$$
Remark 4.3. The lower bound on the excess error of any DP algorithm for the class of strongly convex functions [BST14, Thm. 5.5] implies that the achievable excess error in Theorem 4.2 is optimal in terms of the dependence on  $d$ ,  $\rho$ , and  $n$ . Also, the oracle complexity of our algorithm is an exponential improvement over the oracle complexity of first-order methods [STU17].
Remark 4.4. The proof of Theorem 4.2 suggests that Algorithm 1 has two phases. First, while  $w_{t}$  is far from  $w^{\star}$ , the convergence rate is  $1 / T^{4}$ . Second, when  $w_{t}$  is close to  $w^{\star}$ , the algorithm exhibits the convergence rate of  $\exp (\exp (-T))$ . Notice that Algorithm 1 is agnostic to this transition in the sense that we do not have an explicit switching step in Algorithm 1 and Algorithm 2. It is also interesting to note that the transition happens when  $\| w_{t} - w^{\star}\| \leq 3\mu /4\mathrm{L}_{2}$ .
Remark 4.5 (Comparison with [ABL21].). In [ABL21, §4], the authors propose a DP variant of Newton's method. Their main idea is to add independent noise directly to the Hessian matrix and the gradient vector using the Gaussian mechanism. They also require that the Hessian be a rank-1 matrix. The issue with adding noise directly to a full-rank Hessian matrix is that the noise scales with the dimension  $d$ , which can lead to a suboptimal excess loss. In contrast, our algorithm has a global convergence without placing restrictions on the rank of the Hessian matrix or the initialization.  $\triangleleft$  Remark 4.6. We showed in Theorem 4.2 that our algorithm has an exponentially smaller oracle complexity than the first-order methods in terms of the dependence to  $n$ . For the class of convex, smooth, Lipschitz, and strongly convex, [ZZMW17] proposes a first-order algorithm with an oracle complexity of  $T_{1} = \Theta \left( \sqrt{\mathsf{L}_{1}} / \sqrt{\mu} + \log(n) \right)$ . It is important to note that the constant term in  $T_{1}$  differs from our result, making a direct comparison challenging. It is an interesting question to develop a second-order DP algorithm with a smaller oracle complexity than both the algorithms proposed in [ZZMW17] and ours in Algorithm 1.
Remark 4.7. The cubic Newton method has a non-private convergence rate of  $T^{-2}$  for the class of convex (but not strongly convex) functions [NP06, Thm. 4]. We leave it as an open question whether there exists a DPSolver such that Algorithm 1 achieves an optimal excess error and oracle complexity for convex functions. However, this can be achieved by a DP variant of the first-order accelerated Nesterov's method [Nes98; NJLS09; GL12]; see Appendix A.2.
5 DP Logistic Regression using Second-Order Information