SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The main limitation of our cubic Newton's method (Algorithm 1) is that each iteration requires solving a nontrivial subproblem. So, despite low oracle complexity, it is computationally expensive. Moreover, many loss functions, such as logistic loss, are not strongly convex in the unconstrained setting. In this section, we aim to develop a fast second-order algorithm for unconstrained logistic regression avoiding this issue. In many real-world classification tasks, the logistic loss is the loss of choice. The logistic loss is a convex surrogate of the 0-1 loss, and satisfies many regularity conditions that give rise to various practical optimization algorithms [Bac10; Erd15; KSJ18]. Also, note that our results in this section can readily be extended to the class of smooth and convex GLMs.
First, we recall the logistic loss function. Let  $d \in \mathbb{N}$  and  $\mathcal{Z} = \mathcal{B}^d(1) \times \{-1, 1\}$  be the dimension and data space, where  $\mathcal{B}^d(1) = \{x \in \mathbb{R}^d : \|x\| \leq 1\}$  is the unit ball in  $\mathbb{R}^d$ . Let  $f_{\mathrm{LL}}: \mathbb{R}^d \times \mathcal{Z} \to \mathbb{R}$  denote the logistic loss function defined as
$$
f _ {\mathrm {L L}} (w, (x, y)) = \log \left(1 + \exp (- y \cdot \langle w, x \rangle)\right). \tag {4}
$$
The gradient and Hessian of  $f_{\mathrm{LL}}$  are given by
$$
\nabla_ {w} f _ {\mathrm {L L}} (w, (x, y)) = \frac {- x y}{1 + \exp (y \langle w , x \rangle)}, \quad \nabla_ {w} ^ {2} f _ {\mathrm {L L}} (w, (x, y)) = \frac {x x ^ {\top}}{(\exp (- \frac {\langle w , x \rangle}{2}) + \exp (\frac {\langle w , x \rangle}{2})) ^ {2}}. \tag {5}
$$
Newton's method [BV04, §9.5] is based on successively minimizing a local second-order Taylor approximation on the function. Newton's method does not guarantee a global convergence [JT16]; the reason is that the second-order Taylor approximation of the logistic loss can greatly underestimate the function. Next we show that it is possible to obtain a quadratic global upper bound on the logistic loss function. We will use this to develop an algorithm that converges globally.
Lemma 5.1. For every  $v\in \mathbb{R}^d$ $x\in \mathbb{R}^d$ $w\in \mathbb{R}^d$  , and  $y\in \{-1, + 1\}$  , we have
$$
f _ {\mathrm {L L}} (w, (x, y)) \leq f _ {\mathrm {L L}} (v, (x, y)) + \langle \nabla f _ {\mathrm {L L}} (v, (x, y)), w - v \rangle + \frac {1}{2} \langle H _ {q u} (v, (x, y)) (w - v), w - v \rangle ,
$$
where  $H_{qu}(v,(x,y))\triangleq \frac{\tanh(\langle x,v\rangle / 2)}{2\langle x,v\rangle} xx^{\top}\in \mathbb{R}^{d\times d}$
Remark 5.2. Since  $f_{\mathrm{LL}}$  is  $\frac{1}{4}$ -smooth, we can construct a simpler global quadratic upper-bound as follows [Nes98, Thm. 2.1.5]:  $f_{\mathrm{LL}}(w,(x,y)) \leq f_{\mathrm{LL}}(v,(x,y)) + \langle \nabla f_{\mathrm{LL}}(v,(x,y)), w - v \rangle + \frac{1}{8} \| w - v \|^2$ . Lemma 5.1 is tighter than this, since  $H_{\mathrm{qu}}(v,(x,y)) \preceq \frac{1}{4} I_d$ ; see Appendix B.2. Remark 5.3. The second-order Taylor approximation and our upper bound in Lemma 5.1 both provide a quadratic approximation of the logistic loss. In the remainder of the paper, we write  $H(v,(x,y))$  to refer to both  $\nabla^2 f_{\mathrm{LL}}(v,(x,y))$  and  $H_{\mathrm{qu}}(v,(x,y))$ . We refer to  $H(v,(x,y))$  as the second-order information (SOI) and to  $H_{\mathrm{qu}}$  as quadratic upperbound SOI. Finally, notice both  $\nabla^2 f_{\mathrm{LL}}(v,(x,y))$  and  $H_{\mathrm{qu}}(v,(x,y))$  are PSD rank-1 matrices, with maximum eigenvalue  $\leq \frac{1}{4} \| x \|^2 \leq \frac{1}{4}$ .
5.1 Algorithm Description
We are given a dataset  $S_{n} = ((x_{1},y_{1}),\ldots ,(x_{n},y_{n})) \in (\mathcal{B}^{d}(1)\times \{-1, + 1\})^{n}$  and we aim to minimize  $\ell_{\mathrm{LL}}(w,S_n)\triangleq \frac{1}{n}\sum_{i\in [n]}f_{\mathrm{LL}}(w,(x_i,y_i))$ . Our algorithm iteratively minimizes a quadratic approximation of  $\ell_{\mathrm{LL}}(w,S_n)$ . Consider
$$
q _ {t} (w) \triangleq \ell_ {\mathrm {L L}} \left(w _ {t}, S _ {n}\right) + \left\langle \nabla \ell_ {\mathrm {L L}} \left(w _ {t}, S _ {n}\right), w - w _ {t} \right\rangle + \frac {1}{2} \left\langle H \left(w _ {t}, S _ {n}\right) \left(w - w _ {t}\right), \left(w - w _ {t}\right) \right\rangle , \tag {6}
$$
where  $H(w_{t},S_{n})\triangleq \frac{1}{n}\sum_{i\in [n]}H(w_{t},(x_{i},y_{i}))$ . In the non-private setting the next iterate is set to  $w_{t + 1} = \arg \min_{w}q_{t}(w) = w_{t} - H(w_{t},S_{n})^{-1}\nabla \ell_{\mathrm{LL}}(w_{t},S_{n})$ . To develop a private variant of Newton's method, we need to characterize the sensitivity of this update rule. Our key observation is that the directions corresponding to small eigenvalues of  $H(w_{t},S_{n})$  are more sensitive than the directions corresponding to large eigenvalues. To overcome this issue, we modify the eigenvalues of  $H(w_{t},S_{n})$  to ensure a minimum eigenvalue  $\geq \lambda_0$ , where  $\lambda_0 > 0$  is a carefully chosen constant. We show how to adaptively tune  $\lambda_0$  in Section 5.2. This procedure yields the desired stability with respect to neighbouring datasets. Formally, the modification operator is defined as follows:
Definition 5.4. Let  $A \in \mathbb{R}^{d \times d}$  be a positive semi-definite (PSD) matrix and  $\lambda_0 \geq 0$ . Define
$$
\Psi_ {\lambda_ {0}} (A, \mathsf {c l i p}) = \sum_ {i = 1} ^ {d} \max  \{\lambda_ {0}, \lambda_ {i} \} u _ {i} u _ {i} ^ {\top}, \quad \Psi_ {\lambda_ {0}} (A, \mathsf {a d d}) = \sum_ {i = 1} ^ {d} (\lambda_ {i} + \lambda_ {0}) u _ {i} u _ {i} ^ {\top} = A + \lambda_ {0} I _ {d}.
$$
where  $A = \sum_{i=1}^{d} \lambda_i u_i u_i^\top$  is the eigendecomposition of  $A$ -i.e.,  $0 \leq \lambda_1 \leq \dots \leq \lambda_d$  are the eigenvalues and  $u_1, \ldots, u_d \in \mathbb{R}^d$  are the eigenvectors, which satisfy  $\forall i \neq j \|u_i\| = 1 \wedge \langle u_i, u_j \rangle = 0$ .
Algorithm 3 describes our algorithm. First, we state the privacy guarantee of Algorithm 3 whose proof can be found in Appendices B.3 and B.4.
Theorem 5.5. Assume in Algorithm 3 we choose  $\text{add}$  for the SOI modification. Then, for every training set  $S_{n} \in (\mathbb{R}^{d} \times \{-1, +1\})^{n}$ ,  $w_{0} \in \mathcal{W}$ ,  $\lambda_{0} > 0$ ,  $T \in \mathbb{N}$ ,  $\rho \in \mathbb{R}_{+}$ , and  $\theta \in (0,1)$ , by setting  $\sigma_{1} = \frac{\sqrt{T}}{n\sqrt{2\rho(1 - \theta)}}$  and  $\sigma_{2} = \frac{\sqrt{T}}{(4n\lambda_{0}^{2} + \lambda_{0})\sqrt{2\rho\theta}}$ ,  $w_{T}$  satisfies  $\rho - zCDP$ .
Theorem 5.6. Assume in Algorithm 3, we choose clip for the SOI modification. Then, for every training set  $S_{n} \in (\mathbb{R}^{d} \times \{-1, +1\})^{n}$ ,  $w_{0} \in \mathcal{W}$ ,  $\lambda_{0} > 0$ ,  $T \in \mathbb{N}$ ,  $\rho \in \mathbb{R}_{+}$ , and  $\theta \in (0,1)$  such that  $n > \frac{1}{4\lambda_{0}}$ , by setting  $\sigma_{1} = \frac{\sqrt{T}}{n\sqrt{2\rho(1 - \theta)}}$  and  $\sigma_{2} = \frac{\sqrt{T}}{(4n\lambda_{0}^{2} - \lambda_{0})\sqrt{2\rho\theta}}$ ,  $w_{T}$  satisfies  $\rho - zCDP$ .
Algorithm 3 Newton Method with Double noise
1: Inputs: training set  $S_{n}\in \mathcal{Z}^{n},\lambda_{0} > 0,\theta \in (0,1)$  privacy budget  $\rho$  -zCDP, initialization  $w_{0}$  , number of iterations  $T$  ,SOI modification  $\in$  {clip,add}.
2: Set  $\sigma_{1} = \frac{\sqrt{T}}{n\sqrt{2\rho(1 - \theta)}}$
3: if SOI modification = Add then
4:  $\sigma_{2} = \frac{\sqrt{T}}{(4n\lambda_{0}^{3} + \lambda_{0})\sqrt{2\rho\theta}}$
5: else if SOI modification = Clip then
6:  $\sigma_{2} = \frac{\sqrt{T}}{(4n\lambda_{0}^{2} - \lambda_{0})\sqrt{2\rho\theta}}$
7: for  $t = 0, \dots, T - 1$  do
8: Query  $\nabla f(w_{t},S_{n})$  and  $H(w_{t},S_{n})$
9:  $\dot{H}_t = \Psi_{\lambda_0}(H(w_t, S_n), \text{SOI modification})$
10:  $\tilde{g}_t = \nabla f_{\mathrm{LL}}(w_t,S_n) + \mathcal{N}(0,\sigma_1^2 I_d)$
11:  $w_{t + 1} = w_t - \tilde{H}_t^{-1}\tilde{g}_t + \mathcal{N}(0,\| \tilde{g}_t\| ^2\sigma_2^2 I_d)$
12: Output  $w_{T}$ .
Our DP algorithm differs from the non-private Newton's method in three ways: (1) We first privatize the gradient by adding noise. (2) We modify  $H(w_{t}, S_{n})$  to ensure its eigenvalues are not too small. And (3) we add a second noise to the update computed using the noised gradient and modified second-order information (SOI).
Notice that Algorithm 3 has four variations based on the SOI and the modification of SOI, namely, Hess-clip, Hess-add, QU-clip, and QU-add which refer to using Hessian and clip, Hessian and add, quadratic upper bound (See Lemma 5.1) and clip, and quadratic upper bound and add, respectively.
Remark 5.7 (Generalization of Algorithm 3). In this section our main focus is on DP logistic regression, and the privacy guarantees hold for
the logistic loss. Nevertheless, in Appendix B.6, we present a generalization of Algorithm 3 whose privacy guarantee holds for every convex, doubly differentiable, Lipschitz, and smooth loss function without any constraints on the rank of Hessian. The main technical challenge for sensitivity analysis is proving the approximate Lipschitzness of  $\Psi$  in the operator norm (See Lemma B.7). This demonstrates that our algorithm is more general than objective perturbation [CMS11; KST12; INST+19] and the private damped Newton's method [ABL21] which both require a low-rank Hessian.
5.2 Private and Adaptive Selection of Minimum Eigenvalue
One of the hyperparameters of Algorithm 3 is the minimum eigenvalue  $\lambda_0$ . There exists a tradeoff for choosing  $\lambda_0$ . We ideally want the modification to be as small as possible, so that the SOI is preserved. However, decreasing  $\lambda_0$  increases  $\sigma_2$  and we add more noise. To deal with this problem, we propose a heuristic rule for an adaptive, private, and time-varying selection of the minimum eigenvalue. We wish to find  $\lambda_{0,t}$  that minimizes expected loss at the next iteration, for which we have the quadratic approximation (6). More formally, we compute  $\lambda_{0,t}$  as  $\arg \min_{\lambda} \mathbb{E}[q_t(w_t - \Psi_{\lambda}(H(w_t,S_n),\mathrm{SOI~modification})\tilde{g}_t + \| \tilde{g}_t\| \sigma_2(\lambda)\cdot \xi)]$  where  $q_{t}$  is given in (6) and  $\xi \sim \mathcal{N}(0,I_d)$ . We show in Appendix B.5 that an approximate minimizer is  $\lambda_{0,t} \propto \left(\frac{\mathrm{trace}(H_t(w_t,S_n))}{n^2\times\mathrm{privacy~budget~for~the~direction}}\right)^{\frac{1}{3}}$ . Note that  $\lambda_{0,t}$  depends on the data through trace  $(H(w_{t},S_{n}))$  which has sensitivity  $1 / 4n$ , so it can be estimated privately. In Appendix B.5, we provide the algorithmic description of a variant of Algorithm 3 with an adaptive and private minimum eigenvalue. In particular, we divide the privacy budget at each iteration into three parts: (1) privatizing the gradient; (2) estimating the trace of SOI; and (3) privatizing the direction. We use this variant for our numerical experiments in Section 6.
5.3 Convergence Results for Algorithm 3
In this section, we provide data-dependent convergence guarantees for Algorithm 3. We express these guarantees in terms of the conditional expectation  $\mathbb{E}_t[\cdot ] = \mathbb{E}\left[\cdot |\{w_i\}_{i\in [t]}\right]$  and they can be easily extended to obtain high probability bounds. Before presenting the results, we introduce a notation. For a dataset  $S_{n} = ((x_{1},y_{1}),\ldots ,(x_{n},y_{n}))\in (\mathbb{R}^{d}\times \{-1, + 1\})^{n}$ , let  $V\in \mathbb{R}^{d\times d}$  denote the orthogonal projection matrix on the linear subspace spanned by  $\{x_1,\dots ,x_n\}$ . For every vector  $u\in \mathbb{R}^d$ , define  $\| u\| _V\triangleq \sqrt{u^\top V u}$ . This norm naturally arises since for every  $w\in \mathbb{R}^d$  we have  $\ell_{\mathrm{LL}}(w,S_n) - \ell_{\mathrm{LL}}(w^\star ,S_n)\leq \frac{1}{8}\| w - w^\star \| _V^2$  where  $w^{\star} = \arg \min \ell_{\mathrm{LL}}(w,S_n)$  (See Appendix B.7).
5.3.1 Local Convergence Guarantee of Hess-clip and Hess-add
Theorem 5.8. Let  $S_{n}$  denote the dataset and rank denote the dimension of the linear subspace spanned by  $\{x_{1},\ldots ,x_{n}\}$ . Let  $\lambda_{\min,t}$  be the smallest non-zero eigenvalue of  $\nabla^2\ell_{\mathrm{LL}}(w_t,S_n)$  and  $\rho$  be
Figure 2: Privacy-Utility tradeoff on different datasets.
the privacy budget (in  $zCDP$ ) per iteration. Then,
$$
\mathbb {E} _ {t} \left[ \| w _ {t + 1} - w ^ {\star} \| _ {V} ^ {2} \right] \leq \nu_ {1, t} ^ {2} \| w _ {t} - w ^ {\star} \| _ {V} ^ {2} + 2 \nu_ {1, t} \nu_ {2, t} \| w _ {t} - w ^ {\star} \| _ {V} ^ {3} + \nu_ {2, t} ^ {2} \| w _ {t} - w ^ {\star} \| _ {V} ^ {4} + \Delta ,
$$
where the coefficients are given by
$$
\nu_ {1, t} = 1 - \frac {\tilde {\lambda} _ {m i n , t}}{\lambda_ {0}} + \frac {\sqrt {\mathrm {r a n k}}}{\left(4 n \lambda_ {0} ^ {2} - \lambda_ {0}\right) \sqrt {2 \rho \theta}}, \quad \nu_ {2, t} = \frac {0 . 0 5}{\tilde {\lambda} _ {m i n , t}}, \quad \Delta = O \left(\frac {\mathrm {r a n k}}{\rho (1 - \theta) n ^ {2}} \frac {1}{(\tilde {\lambda} _ {m i n , t}) ^ {2}}\right). \quad (7)
$$
Here,  $\tilde{\lambda}_{\min,t} = \begin{cases} \min\{\lambda_{\min,t}, \lambda_0\} & \text{for Hess-clip,} \\ \lambda_{\min,t} + \lambda_0 & \text{for Hess-add,} \end{cases}$  depends on the modification procedure.
This type of convergence is known as composite convergence, as it is a combination of linear and quadratic rates, and has been observed in the convergence analysis of several quasi-Newton's methods [EM15; Erd15; RM16; XYRRM16].
Remark 5.9.  $\lambda_{\min,t}$  is the smallest non-zero eigenvalue of  $\nabla^2\ell_{\mathrm{LL}}(w_t,S_n)$ . Therefore, for sufficiently large  $n$  we have  $0 < \nu_{1,t} < 1$ . It shows Algorithm 3 with Hessian as SOI is, in expectation, a descent algorithm locally given  $\| w_t - w^\star\|$  is sufficiently larger than  $\Delta$ . Roughly speaking, Theorem 5.8 guarantees a linear convergence to a ball around the optimum whose radius is given by  $\Delta$ . We also observe the linear rate in Figure 3. Moreover, the error due to the privacy, i.e.,  $\Delta$  in Equation (7), is proportional to the rank of the feature vectors which is always smaller than  $d$ . These interesting properties is due to the convergence analysis with respect to  $\| \cdot \|_V$ .
Remark 5.10. The coefficients of the convergence in Equation (7) depend on the iteration step which is an undesirable aspect of the results. In Lemma B.11, we prove that  $|\lambda_{\min ,t} - \lambda_{\min}^{\star}| \leq 0.1\| w_t - w^\star \|_V$  where  $\lambda_{\mathrm{min}}^{\star}$  is the smallest non-zero eigenvalue of  $\nabla^2\ell_{\mathrm{LL}}(w^\star, S_n)$ . Therefore, the coefficients can be well-approximated by their analogous values evaluated at the optimum.
5.3.2 Global Convergence Guarantee of QU-clip and QU-add
We also establish a global convergence guarantee for QU-clip and QU-add. Due to the space the formal statement and proof are deferred to Appendix B.9. Roughly speaking, under the assumption of local strong convexity at the optimum [Bac14], QU-clip and QU-add converge globally: this is intuitive since QU-clip and QU-add are based on minimizing a global upper bound on the function.
In this section, we evaluate the performance of our algorithm (Algorithm 3 with the adaptive minimum eigenvalue selection from Section 5.2) for the problem of binary classification using logistic regression. For brevity, many of the details behind our implementation and more experimental results are deferred to Appendix C.
The setup of the experiments is as follows: **Baseline1- DP-(S)GD:** The update rule is  $w_{t+1} = w_t - \eta \nabla \ell(w_t, S_n) + \xi$  where  $\xi$  is a Gaussian noise [SCS13; BST14; ACGM+16]. Since the logistic loss is 1-Lipschitz, we do not need gradient clipping. The Lipschitzness parameter controls the variance of the Gaussian random vector. To draw a fair comparison and show the advantage of using second-order information, we chose the stepsize to be equal to the inverse smoothness. **Baseline2-
Figure 3: Comparison with DP-GD Oracle where at each iteration the stepsize tuned non-privately.
T*DP-GD/T*ours	T*ours(sec)
ε = 0.01	ε = 0.1	ε = 1	ε = 10	min(T*ours) (sec.)	max(T*ours) (sec.)
ala	4.87×	2.95×	5.09×	30.59×	2.45	4.2
synthetic	2.90×	2.90×	5.19×	11.61×	0.18	0.21
adult	12.08×	11.84×	22.17×	38.16×	6.81	8.07
covertype	24.19×	19.85×	35.70×	36.20×	2.93	3.58
Table 1: Comparison between the run time of our algorithm and DP-GD in terms of the ratio  $T_{\mathrm{DP - GD}}^{\star} / T_{\mathrm{our}}^{\star}$ . The last two columns show the minimum and maximum run time of our algorithm.
Approximate Objective Perturbation (AOP): AOP is built on objective perturbation [CMS11; KST12]. Objective perturbation consists of a two-stage process: (1) perturbing the objective function by adding a random linear term and (2) outputting the minimum of the perturbed objective. Releasing such a minimum is sufficient for achieving DP guarantees [CMS11; KST12], but only if we can find the exact minimum of the perturbed objective. AOP extends objective perturbation to permit using an approximate minimum of the perturbed objective [INST+19; INST+]. Notice AOP is not an iterative optimization algorithm. Baseline3- Damped Newton Method [ABL21]: The algorithm in [ABL21] is a variant of damped Newton's method with the assumption that the Hessian of loss function is rank-1, which holds for the logistic loss. Their algorithm is based on adding two i.i.d. noises to the Hessian and the gradient:  $w_{t + 1} = w_t - \eta_tH_{\mathrm{noisy},t}(w_t,S_n)^{-1}\tilde{g}_t$ , where  $\eta_t$  is the stepsize,  $H_{\mathrm{noisy},t}(w_t,S_n) = \nabla^2\ell_{\mathrm{LL}}(w_t,S_n) + \Xi_t$  and  $\tilde{g}_t = \nabla \ell_{\mathrm{LL}}(w_t,S_n) + \xi_t$ . Here  $\Xi_t$  and  $\xi_t$  are carefully chosen Gaussian noise. With  $\eta_t = 1$ , our experiments show that their algorithm is not converging. We use the strategy suggested in [ABL21, Page 22] and set  $\eta_t = \log (1 + \beta_t) / \beta_t$  where  $\beta_t = \left\| \nabla^2\ell_{\mathrm{LL}}(w_t,S_n)^{-1}\nabla \ell_{\mathrm{LL}}(w_t,S_n)\right\|$ . This stepsize selection makes the algorithm non-private, however, it serves as a good baseline. Datasets: We conducted experiments on six publicly available datasets: a1a, Adult, covertype, synthetic, fashion-MNIST, and protein (Appendix C includes fashion-MNIST and protein results). The synthetic dataset is generated as follows: Fix  $d\in \mathbb{N}$  and  $w^{\star}\in \mathbb{R}^{d}$ . Then, (1) the feature vectors  $\{x_i\in \mathbb{R}^d : i\in [n]\}$  are independent and sampled uniformly at random from the unit sphere in  $\mathbb{R}^d$ , (2) for the  $i$ -th datapoint the label is +1 with probability  $(1 + \exp (-\langle x_i,w^{\star}\rangle))^{-1}$  and -1 otherwise. Privacy Notion: The privacy notion for our experiments is  $(\epsilon ,\delta = (\mathrm{num.~of~samples})^{-2})$ -DP. Next, we present the results.
6.2 Privacy-Utility-Run Time Tradeoff
We study the tradeoff for our algorithm and compare it with other baselines for a broad range of  $\varepsilon \in \{0.01, \dots, 10\}$ . We non-privately tune the total number of iterations of the iterative algorithms and report the best achievable excess error in Figure 2. As can be seen our algorithm almost always achieves the best excess loss for a broad range of  $\epsilon$ . Also, Figure 2 shows that damped private Newton method of [ABL21] achieves a low excess loss only for large  $\epsilon$ . Figure 2 indicates that DP-GD and our algorithm are the best in terms of excess loss. In Table 1, we compare the run time of DP-GD and our algorithm, i.e., the computational time in seconds for achieving the excess loss in Figure 2. As can be seen, for many challenging datasets, our algorithm is  $10 - 40 \times$  faster than DP-GD. Our experiments are run on CPU. We also remark that each step of Algorithm 3, i.e., computing gradient and SOI, is heavily parallelizable implying that the run time of Algorithm 3 can be made much smaller by an efficient implementation. Also, the reported numbers in Figure 2 and Table 1 correspond to Hess-clip.
Figure 4: Minibatch Variant of Our Algorithm and Comparison with DP-SGD
6.3 Second Order Information vs Optimal Stepsize
In non-private convex optimization, the key to the success of second-order optimization algorithms is that the second-order information acts as a preconditioner, and the same performance cannot be attained by optimally tuning the stepsize for GD algorithm. To investigate whether the same holds for our algorithms, we consider the following variant of DP-GD. Let  $\tilde{g}_t$  denote the perturbed gradient obtained by adding a Gaussian random vector to  $\nabla \ell_{\mathrm{LL}}(w_t,S_n)$ . Instead of a constant stepsize, the stepsize at iteration  $t$  is chosen based on  $\eta_t = \arg \min_{\eta \geq 0}\ell_{\mathrm{LL}}(w_t - \eta \tilde{g}_t)$ . Notice this variant is obviously not DP. We refer to this variant as DP-GD-Oracle. The comparison with DP-GD-Oracle lets us answer the following question: Could we have just computed a single number, i.e., stepsize, to achieve the same performance as our second-order optimization algorithms which require computing a  $d\times d$  matrix? In Figure 3, we compare the convergence speed of our algorithms with DP-GD-Oracle in low- and high-privacy regimes. Figure 3 shows our algorithms converge faster than DP-GD-Oracle which is not even a DP algorithm. Figure 3 confirms the expectation that as the privacy budget increases the difference between our algorithms and DP-GD-Oracle increases since we can use more curvature information.
6.4 Minibatch Variant of Our Algorithm and Comparison with DP-SGD
So far we have considered full-batch algorithms that compute first- and second-order information on the entire dataset. We extend Algorithm 3 to the minibatch setting, where, at each iteration, the gradient and SOI matrix are computed using a subsample of the data points. In Appendix C.1 we provide a formal algorithmic description of the minibatch version of Algorithm 3 along with its privacy proof. Then, we compare the convergence speed and excess loss with DP-SGD.
DP-SGD is faster than DP-GD, but to achieve good privacy and utility, we need large batches [PHKX+23, Fig. 2]. This is in stark contrast with non-private SGD, where larger batch sizes yield diminishing returns [ZLNM+19]. In particular, to achieve the best excess loss we need to select the batch size as large as possible. We select the batch size of DP-SGD so that the achievable excess loss will be close to the full batch versions. Specifically, we select  $\frac{\text{batch size DP-SGD}}{\text{number of samples}} \approx 0.02$  and tune the number of iterations of DP-SGD to obtain the best result. Figure 4 shows the progress of different algorithm versus run time. Obviously, for a fixed run time DP-SGD performs more iterations compared to our algorithms. Nevertheless, our algorithms achieve the same excess error as DP-GD with  $8 - 10 \times$  faster run time over all the datasets while the batch sizes of our algorithms are larger than that of DP-SGD. We observe that the variations of our algorithms based on the adding operator performs better in the minibatch setting. This can be attributed to the smaller  $\sigma_{2}$  for the adding operator in Algorithm 3. In summary, the comparison between privacy-utility-wall time tradeoff of the subsampled variant of our algorithm and DP-SGD is similar to their full-batch counterparts.
We showed that second-order methods can be used in the DP setting both for improving worst-case convergence guarantees and designing faster practical algorithms. We believe our results open up many directions: A limitation of our algorithms is that the cost of forming and inverting the Hessian can be prohibitive when  $d$  is large. In the non-private setting, a line of research tries to address this limitation by constructing an approximation to SOI such that the update is efficient, yet still provides sufficient SOI [EM15; Erd15; XYRRM16; ABH17]. It would be interesting to investigate how the ideas developed in our paper could be incorporated into these methods.