Title: Faster Differentially Private Convex Optimization via Second-Order Methods
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

logistic regression

optimization

Newton's method

second-order methods
Abstract: Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than first-order methods like gradient descent. In this work, we investigate the prospect of using the second-order information from the loss function to accelerate DP convex optimization. We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak, and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss. We then design a practical second-order DP algorithm for the unconstrained logistic regression problem. We theoretically and empirically study the performance of our algorithm. Empirical results show our algorithm consistently achieves the best excess loss compared to other baselines and is 10-40x faster than DP-GD/DP-SGD for challenging datasets.

Faster Differentially Private Convex Optimization via Second-Order Methods

Abstract

Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than first-order methods like gradient descent. In this work, we investigate the prospect of using the second-order information from the loss function to accelerate DP convex optimization. We first develop a private variant of the regularized cubic Newton method of Nesterov and Polyak [NP06], and show that for the class of strongly convex loss functions, our algorithm has quadratic convergence and achieves the optimal excess loss. We then design a practical second-order DP algorithm for the unconstrained logistic regression problem. We theoretically and empirically study the performance of our algorithm. Empirical results show our algorithm consistently achieves the best excess loss compared to other baselines and is 10 -40 × faster than DP-GD/DP-SGD for challenging datasets.

1 Introduction

Many machine learning tasks reduce to a convex optimization problem. More precisely, given a dataset S n = ( z 1 , . . . , z n ) ∈ Z n , a closed, convex set W ⊆ R d , and a loss function f : W×Z → R such that, for every z ∈ Z , f ( w,z ) is a convex function in w , our goal is to compute an approximation to arg min w ∈W ( glyph[lscript] ( w,S n ) glyph[defines] 1 n ∑ i ∈ [ n ] f ( w,z i ) ) . In this paper, we are interested in the problem of designing optimization algorithms in the scenario that the dataset S n contains private information. Differential privacy (DP) [DMNS06] is a formal standard for privacy-preserving data analysis that provides a framework for ensuring that the output of an analysis on the data does not leak this private information. This problem is known as private convex optimization : Design an algorithm A : Z n →W that is both DP and ensures low excess loss glyph[defines] glyph[lscript] ( A ( S n ) , S n ) -min w ∈W glyph[lscript] ( w,S n ) .

The predominant algorithm for private convex optimization is DP (stochastic) gradient descent (DP-GD/DP-SGD). This is a first-order iterative method. I.e., we start with an initial value w 0 and iteratively update it using the gradient of the loss ∇ w t glyph[lscript] ( w t , S n ) following the update rule w t +1 = w t -η · ( ∇ w t glyph[lscript] ( w t ,S n )+ ξ t ) , where η > 0 is a constant and ξ t is Gaussian noise to ensure privacy. The number of iterations T also determines the amount of noise at each iteration, i.e., the scale of ξ t is proportional to √ T due to the composition of DP. Note that we assume ‖∇ w t glyph[lscript] ( w t , S n ) ‖ ≤ 1 .

One of the major drawbacks of DP-(S)GD is slow convergence . The choice of ( η, T ) exhibits a tradeoff in terms of the excess loss: if η · T is small, the algorithm cannot reach the optimal solution; on the other hand, the magnitude of noise at each iteration is η · √ T , which cannot be too large. Therefore, to maximize η · T and minimize η · √ T , implementations of DP-(S)GD err on the side of

Abhradeep Thakurta Google DeepMind

large T and small η , which results in a long, slow path to convergence. This fact has been shown theoretically as well: for the class of β -smooth convex functions, the optimal instantiations of DP-GD use a step size of max { 1 / √ n, √ d/εn } [BFTG19] while in the non-private setting the stepsize for GD is set to 1 /β . Smaller step size requires more steps (i.e. more iterations) to converge. This slowness is exacerbated by the facts that (1) DP-SGD requires large batch sizes for good performance [PHKX+23] and (2) the hyperparameter tuning of DP-(S)GD, and generally DP algorithms, is a challenging task [PS22]. Can we design a DP optimization algorithm which accelerates DP-(S)GD by choosing the step size dynamically based on the local geometry of the loss function?

We draw inspiration from the non-private optimization literature: To address the slow convergence of GD and of first-order methods in general, a class of algorithms based on preconditioning the gradient using second-order information has been developed [Nes98; NW99]. This class of algorithms is based on successively minimizing a quadratic approximation of the function, i.e., w t +1 = w t +∆ t where ∆ t = arg min ∆ { glyph[lscript] ( w t , S n ) + 〈∇ glyph[lscript] ( w t , S n ) , ∆ 〉 + 1 2 〈 H t · ∆ , ∆ 〉} = -( H t ) -1 ∇ glyph[lscript] ( w t , S n ) . Here, H t is a scaling matrix which provides curvature information about the loss glyph[lscript] ( · , S n ) at w t . For instance, Newton's method uses the Hessian H t = ∇ 2 glyph[lscript] ( w t , S n ) . Second-order algorithms significantly improve over the convergence speed of GD, and key to their success is that at each step they automatically tune the stepsize along each dimension based on the local curvature.

In this paper, our goal is to accelerate DP convex optimization. In particular, the current paper revolves around the following questions: Can the second-order information accelerate private convex optimization while achieving optimal excess error ? What is the best way to privatize second-order information , e.g., the Hessian matrix? How does the achievable privacy-utility-runtime tradeoff compare with first-order methods such as DP-GD? We show that second-order information can accelerate DP optimization while achieving excess loss that matches or improves on DP-GD. Our main contributions are both theoretical and empirical:

1.1 Provably Optimal Algorithm for Strongly Convex Functions

Newton's method is a second-order optimization technique that is well-known for its rapid convergence for strongly convex and smooth functions in non-private optimization. Specifically, to achieve an excess loss of α , the method only requires O (log log(1 /α )) iterations, which is provably faster than the convergence rate of any first-order method. One natural question is whether it is possible to design a second-order DP convex optimization algorithm that can achieve the optimal minmax excess error err opt in O (log log(1 / err opt )) iterations? We provide an affirmative answer to this question in Section 4 by designing a second-order DP algorithm based on the cubic regularized Newton's method of Nesterov and Polyak [NP06]. At each step t , we compute a cubic upper bound glyph[lscript] ( w +∆ ,S n ) ≤ glyph[lscript] ( w,S n )+ 〈∇ w glyph[lscript] ( w,S n ) , ∆ 〉 + 1 2 〈 ∇ 2 w glyph[lscript] ( w,S n ) · ∆ , ∆ 〉 + O ( ‖ ∆ ‖ 3 ) . We can minimize this cubic upper bound using any DP convex optimization subroutine; the minimizer becomes the next iterate w t +1 . Since the cubic is a universal upper bound, our algorithm converges globally

1.2 Fast Practical Algorithms for DP Logistic Regression

DP logistic regression is a popular approach for private classification, with DP-GD/DP-SGD being the predominant class of algorithms for this task. As we numerically show, DP-GD/DP-SGD exhibit slow convergence for this task (See Figure 1). In Section 5, we develop a practical algorithm that injects carefully designed noise into Newton's update rule as follows:

In particular, we inject noise twice: ξ t, 1 privatizes the gradient and ξ t, 2 privatizes the direction. The function Ψ modifies the Hessian to ensure that the eigenvalues are not too small; this is essential for bounding the sensitivity and, hence, the scale of ξ t, 2 . Weconsider two types of modification based on eigenvalue clipping and eigenvalue adding . For eigenvalue clipping, Ψ( ∇ 2 w t glyph[lscript] ( w t ,S n )) replaces the eigenvalues λ i of ∇ 2 w t glyph[lscript] ( w t ,S n ) with max { λ i , λ 0 } , where λ 0 > 0 is a carefully chosen constant. For eigenvalue adding, Ψ( ∇ 2 w t glyph[lscript] ( w t ,S n )) = ∇ 2 w t glyph[lscript] ( w t ,S n ) + λ 0 I . Using Ψ we can control the sensitivity and still have fast convergence, since important curvature information is generally

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

10

-

2

10

-

1

Excess Loss

covertype,

(

ε, δ

)

=(1,

n

-

2

)-DP

DP-GD

Ours - Hessian with Clipping

Ours - QU with Clipping

Best Excess Loss

Wall Time (sec)

contained in the larger eigenvalues/vectors of the Hessian. We prove the local convergence of the update rule (1) in Section 5.3 and perform a thorough empirical evaluation Section 6. We demonstrate that our algorithm outperforms existing baselines on a variety of benchmarks.

Ensuring Global Convergence. One limitation of the update rule in Equation (1) is it does not converge globally (even without noise added for DP). That is, if the initial point w 0 is too far from the optimal solution, then the iterates may diverge. To address this problem, we propose a variant of Newton's update rule where we replace the Hessian with a different form of second-order information which gives a Quadratic Upperbound (QU) on the logistic loss. This is guaranteed to converge globally , like the cubic Newton approach. And we show numerically that this algorithm converges almost as fast as the regular Newton's method in the private setting. Figure 1 shows the convergence speed of our algorithms and DP-GD in terms of real wall time for the task of logistic regression on the Covertype dataset for ( ε, δ ) = (1 , ( num. samples ) -2 ) -DP. Despite DP-GD having a lower per-iteration cost, our algorithm is 30 × faster than DP-GD and achieves better excess loss.

Stochastic Minibatch Variant. We also show that our algorithms naturally extend to the minibatch setting where gradient and second-order information are computed on a subset of samples. We numerically compare it with DP-SGD and show that it has faster convergence.

2 Related Work

DP optimization is a well-studied topic [e.g., SCS13; MRTZ17; ACGM+16; STU17; WLKC+17; INST+19; STT20; SSTT21; GTU22; GLL22; BFTG19; BST14]. Most similar to our work, AvellaMedina, Bradshaw, and Loh [ABL21] consider second-order methods for DP convex optimization. We provide a detailed comparison between our results and theirs in Remark 4.5 and Section 6 showing that our algorithms relax restrictive assumptions and provide better excess error for logistic regression.

There are numerous non-private second-order optimization methods in the literature. The choice of method depends primarily on the values of n and d . When n is large, several works consider various sampling techniques for constructing second-order information, see [RM19; XYRRM16; Erd15; EM15]. When d is large, various methods are proposed in the literature for efficient approximation of the Hessian matrix, see [ABH17; Erd15; EM15; XYRRM16; GKLR19]. There is also a family of algorithms based on the estimation of the curvature from the change in gradients. These algorithms are generally known as quasi-Newton methods stemming from the seminal BFGS algorithm [JM23].

3 Preliminaries

Let d ∈ N . For a vector x ∈ R d , ‖ x ‖ denotes the glyph[lscript] 2 norm of x . Let n, m ∈ N . For a matrix A ∈ R n × m , ‖ A ‖ = sup x ∈ R m : ‖ x ‖≤ 1 ‖ Ax ‖ denotes the operator norm, and ‖ A ‖ F glyph[defines] √ trace ( A T · A ) denotes the Frobenius norm of A where trace denotes the trace operator. I d ∈ R d × d denotes the identity matrix. 〈· , ·〉 denotes the standard inner product in R d . For a convex and closed subset W ⊆ R d , let Π W : R d → W be the Euclidean projection operator, given by Π W ( x ) = arg min y ∈W ‖ y -x ‖ 2 . For a (measurable) space R , M 1 ( R ) denotes the set of all probability measures on R . Note that the statements in the paper about random variables hold almost surely. We will skip such declarations to aid readability. Let Z be the data and let W ⊆ R d be the parameter space. Let f : W × Z → R be a loss function. Throughout the paper, we assume f is doubly continuous, a convex function in w , and W is a closed and convex set. We say (1) f is L 0 -Lipschitz iff there exists L 0 ∈ R such that ∀ z ∈ Z , ∀ w,v ∈ W : | f ( w,z ) -f ( v, z ) | ≤ L 0 ‖ w -v ‖ , (2) f is L 1 -smooth iff there exists L 1 ∈ R such that ∀ z ∈ Z , ∀ w,v ∈ W : ‖∇ f ( w,z ) -∇ f ( v, z ) ‖ ≤ L 1 ‖ w -v ‖ , (3) f has a L 2 -Lipschitz Hessian iff there exists L 2 ∈ R such that ∀ z ∈ Z , ∀ w,v ∈ W : ∥ ∥ ∇ 2 f ( w,z ) -∇ 2 f ( v, z ) ∥ ∥ ≤ L 2 ‖ w -v ‖ , (4) f is µ -strongly convex iff for all w,v ∈ W and z ∈ Z we have f ( v, z ) ≥ f ( w,z ) + 〈∇ f ( w,z ) , v -w 〉 + µ 2 ‖ v -w ‖ 2 .

3.1 Zero-Concentrated DP

For our privacy analysis, we use concentrated differential privacy [DR16; BS16], as it provides a simpler composition theorem - the privacy parameter ρ adds up when we compose.

Definition 3.1 ([BS16, Def. 1.1]) . A randomized mechanism A : Z n → M 1 ( R ) is ρ -zCDP, iff, for every neighbouring dataset (i.e., addition or removal) S n ∈ Z n and S ′ n ∈ Z n , and for every α ∈ (1 , ∞ ) , it holds D α ( A ( S n ) ‖A ( S ′ n )) ≤ ρα, where D α ( A n ( S n ) ‖A n ( S ′ n )) is the α -Renyi divergence between A n ( S n ) and A n ( S ′ n ) .

We should think of ρ ≈ ε 2 : to attain ( ε, δ ) -DP, it suffices to set ρ = ε 2 4 log(1 /δ )+4 ε [BS16, Lem. 3.5].

Lemma 3.2 ([BS16, Prop. 1.3]) . Assume we have a randomized mechanism A : Z → M 1 ( R ) that satisfies ρ -zCDP, then for every δ > 0 , A is ( ρ +2 √ ρ log(1 /δ ) , δ ) -DP.

4 Optimal Algorithm for the Class of Strongly Convex Functions

In this section, we present a DP variant of the cubic-regularized Newton's method of Nesterov and Polyak [NP06]. To motivate the idea behind our algorithm, we revisit DP gradient descent (DP-GD) for the class of L 0 -Lipschitz and L 1 -smooth convex loss functions.

Let { w GD t } t ∈ [ T ] be the iterates of DP-GD. The smoothness of glyph[lscript] lets us construct a global quadratic upper bound on the function [Nes98, Thm. 2.1.5] as follows ∀ w ∈ W and S n ∈ Z n :

Then, DP-GD can be seen as a two-step process:

where ξ t = N (0 , σ 2 I d ) with σ 2 = L 2 0 2 ρn 2 so that w GD t +1 satisfies ρ -zCDP [BS16, Lem. 2.5]. That is, in each iteration of DP-GD, we find a minimum of the quadratic upper bound q t ( w ) and then project back to W . (In the unconstrained setting where W = R d we do not need the second projection step.)

Consider the class of L 2 -Lipschitz Hessian convex loss functions. Nesterov and Polyak [NP06, Lem. 1] show that we can construct a global cubic upper bound exploiting the second-order information (i.e., Hessian) as follows: for all w and w t , glyph[lscript] ( w,S n ) ≤ φ t ( w ) where

Their non-private algorithm is based on the exact minimization of φ t ( w ) , i.e., the next iterate is w t +1 = arg min φ t ( w ) . Note that arg min φ t ( w ) does not admit a closed form solution, as opposed to the quadratic upper bound (2). Similar to the intuition for DP-GD on smooth loss functions (2), our algorithms in this section are based on privately minimizing φ t ( w ) at each iteration. Our algorithm is shown in Algorithm 1. In each iteration the algorithm makes an oracle call to obtain ( glyph[lscript] ( w t , S n ) , ∇ glyph[lscript] ( w t , S n ) , ∇ 2 glyph[lscript] ( w t , S n )) . Then, the algorithm calls an efficient DPSolver for privately optimizing the cubic upper bound (3). The privacy analysis of Algorithm 1 is a direct application of the composition property of zCDP [BS16, Lemma 2.3]; the output of DPSolver at each iteration satisfies ρ/T -zCDP where ρ is the total privacy budget and T is the total number of iterations.

Remark 4.1 . DPSolver in Algorithm 1 does not affect the oracle complexity of Algorithm 1, as it is applied to the proxy loss φ t ( w ) , rather than the underlying loss glyph[lscript] ( w,S n ) . glyph[triangleleft]

Algorithm 2 DPSolver

Theorem 4.2. Let f be a L 0 -Lipschitz, L 1 -smooth, L 2 -Lipschitz Hessian, and µ -strongly convex function. Also, assume that W ⊆ R d has finite diameter D . Let w glyph[star] = arg min w ∈W glyph[lscript] ( w,S n ) . Then, for every ρ > 0 , β ∈ (0 , 1) , and S n ∈ Z n for sufficiently large n , by setting the number of iterations in Algorithm 1 to

and using Algorithm 2 as DPSolver , we have the following: The output of Algorithm 1, i.e., w T , satisfies ρ -zCDP and with probability at least 1 -β

Remark 4.3 . The lower bound on the excess error of any DP algorithm for the class of strongly convex functions [BST14, Thm. 5.5] implies that the achievable excess error in Theorem 4.2 is optimal in terms of the dependence on d , ρ , and n . Also, the oracle complexity of our algorithm is an exponential improvement over the oracle complexity of first-order methods [STU17]. glyph[triangleleft]

Remark 4.4 . The proof of Theorem 4.2 suggests that Algorithm 1 has two phases. First, while w t is far from w glyph[star] , the convergence rate is 1 /T 4 . Second, when w t is close to w glyph[star] , the algorithm exhibits the convergence rate of exp(exp( -T )) . Notice that Algorithm 1 is agnostic to this transition in the sense that we do not have an explicit switching step in Algorithm 1 and Algorithm 2. It is also interesting to note that the transition happens when ‖ w t -w glyph[star] ‖ ≤ 3 µ/ 4 L 2 . glyph[triangleleft]

Remark 4.5 (Comparison with [ABL21].) . In [ABL21, §4], the authors propose a DP variant of Newton's method. Their main idea is to add independent noise directly to the Hessian matrix and the gradient vector using the Gaussian mechanism. They also require that the Hessian be a rank-1 matrix . The issue with adding noise directly to a full-rank Hessian matrix is that the noise scales with the dimension d , which can lead to a suboptimal excess loss. In contrast, our algorithm has a global convergence without placing restrictions on the rank of the Hessian matrix or the initialization. glyph[triangleleft]

Remark 4.6 . We showed in Theorem 4.2 that our algorithm has an exponentially smaller oracle complexity than the first-order methods in terms of the dependence to n . For the class of convex, smooth, Lipschitz, and strongly convex, [ZZMW17] proposes a first-order algorithm with an oracle complexity of T 1 = Θ ( √ L 1 / √ µ +log( n ) ) . It is important to note that the constant term in T 1 differs from our result, making a direct comparison challenging. It is an interesting question to develop a second-order DP algorithm with a smaller oracle complexity than both the algorithms proposed in [ZZMW17] and ours in Algorithm 1. glyph[triangleleft]

Remark 4.7 . The cubic Newton method has a non-private convergence rate of T -2 for the class of convex (but not strongly convex) functions [NP06, Thm. 4]. We leave it as an open question whether there exists a DPSolver such that Algorithm 1 achieves an optimal excess error and oracle complexity for convex functions. However, this can be achieved by a DP variant of the first-order accelerated Nesterov's method [Nes98; NJLS09; GL12]; see Appendix A.2. glyph[triangleleft]

5 DP Logistic Regression using Second-Order Information

The main limitation of our cubic Newton's method (Algorithm 1) is that each iteration requires solving a nontrivial subproblem. So, despite low oracle complexity, it is computationally expensive. Moreover, many loss functions, such as logistic loss, are not strongly convex in the unconstrained setting. In this section, we aim to develop a fast second-order algorithm for unconstrained logistic regression avoiding this issue. In many real-world classification tasks, the logistic loss is the loss of choice. The logistic loss is a convex surrogate of the 0-1 loss, and satisfies many regularity conditions that give rise to various practical optimization algorithms [Bac10; Erd15; KSJ18]. Also, note that our results in this section can readily be extended to the class of smooth and convex GLMs.

First, we recall the logistic loss function. Let d ∈ N and Z = B d (1) ×{-1 , 1 } be the dimension and data space, where B d (1) = { x ∈ R d : ‖ x ‖ ≤ 1 } is the unit ball in R d . Let f LL : R d ×Z → R denote the logistic loss function defined as

The gradient and Hessian of f LL are given by

Newton's method [BV04, §9.5] is based on successively minimizing a local second-order Taylor approximation on the function. Newton's method does not guarantee a global convergence [JT16]; the reason is that the second-order Taylor approximation of the logistic loss can greatly underestimate the function. Next we show that it is possible to obtain a quadratic global upper bound on the logistic loss function. We will use this to develop an algorithm that converges globally.

Lemma 5.1. For every v ∈ R d , x ∈ R d , w ∈ R d , and y ∈ {-1 , +1 } , we have

Remark 5.2 . Since f LL is 1 4 -smooth, we can construct a simpler global quadratic upper-bound as follows [Nes98, Thm. 2.1.5]: f LL ( w, ( x, y )) ≤ f LL ( v, ( x, y )) + 〈∇ f LL ( v, ( x, y )) , w -v 〉 + 1 8 ‖ w -v ‖ 2 . Lemma 5.1 is tighter than this, since H qu ( v, ( x, y )) glyph[precedesorcurly] 1 4 I d ; see Appendix B.2. glyph[triangleleft] Remark 5.3 . The second-order Taylor approximation and our upper bound in Lemma 5.1 both provide a quadratic approximation of the logistic loss. In the remainder of the paper, we write H ( v, ( x, y )) to refer to both ∇ 2 f LL ( v, ( x, y )) and H qu ( v, ( x, y )) . We refer to H ( v, ( x, y )) as the second-order information (SOI) and to H qu as quadratic upperbound SOI. Finally, notice both ∇ 2 f LL ( v, ( x, y )) and H qu ( v, ( x, y )) are PSD rank-1 matrices, with maximum eigenvalue ≤ 1 4 ‖ x ‖ 2 ≤ 1 4 . glyph[triangleleft]

5.1 Algorithm Description

We are given a dataset S n = (( x 1 , y 1 ) , . . . , ( x n , y n )) ∈ ( B d (1) × {-1 , +1 } ) n and we aim to minimize glyph[lscript] LL ( w,S n ) glyph[defines] 1 n ∑ i ∈ [ n ] f LL ( w, ( x i , y i )) . Our algorithm iteratively minimizes a quadratic approximation of glyph[lscript] LL ( w,S n ) . Consider

where H ( w t , S n ) glyph[defines] 1 n ∑ i ∈ [ n ] H ( w t , ( x i , y i )) . In the non-private setting the next iterate is set to w t +1 = arg min w q t ( w ) = w t -H ( w t , S n ) -1 ∇ glyph[lscript] LL ( w t , S n ) . To develop a private variant of Newton's method, we need to characterize the sensitivity of this update rule. Our key observation is that the directions corresponding to small eigenvalues of H ( w t , S n ) are more sensitive than the directions corresponding to large eigenvalues . To overcome this issue, we modify the eigenvalues of H ( w t , S n ) to ensure a minimum eigenvalue ≥ λ 0 , where λ 0 > 0 is a carefully chosen constant. We show how to adaptively tune λ 0 in Section 5.2. This procedure yields the desired stability with respect to neighbouring datasets. Formally, the modification operator is defined as follows:

Definition 5.4. Let A ∈ R d × d be a positive semi-definite (PSD) matrix and λ 0 ≥ 0 . Define

where A = ∑ d i =1 λ i u i u glyph[latticetop] i is the eigendecomposition of A - i.e., 0 ≤ λ 1 ≤ ·· · ≤ λ d are the eigenvalues and u 1 , . . . , u d ∈ R d are the eigenvectors, which satisfy ∀ i = j ‖ u i ‖ = 1 ∧〈 u i , u j 〉 = 0 .

glyph[negationslash]

Algorithm 3 describes our algorithm. First, we state the privacy guarnatee of Algorithm 3 whose proof can be found in Appendices B.3 and B.4.

Theorem 5.5. Assume in Algorithm 3 we choose add for the SOI modification. Then, for every training set S n ∈ ( R d ×{-1 , +1 } ) n , w 0 ∈ W , λ 0 > 0 , T ∈ N , ρ ∈ R + , and θ ∈ (0 , 1) , by setting σ 1 = √ T n 2 ρ (1 θ ) and σ 2 = √ T (4 nλ 2 0 + λ 0 ) √ 2 ρθ , w T satisfies ρ -zCDP.

Theorem 5.6. Assume in Algorithm 3, we choose clip for the SOI modification. Then, for every training set S n ∈ ( R d ×{-1 , +1 } ) n , w 0 ∈ W , λ 0 > 0 , T ∈ N , ρ ∈ R + , and θ ∈ (0 , 1) such that n > 1 4 λ 0 , by setting σ 1 = √ T n √ 2 ρ (1 -θ ) and σ 2 = √ T (4 nλ 2 0 -λ 0 ) √ 2 ρθ , w T satisfies ρ -zCDP.

Algorithm 3 Newton Method with Double noise

Our DP algorithm differs from the non-private Newton's method in three ways: (1) We first privatize the gradient by adding noise. (2) We modify H ( w t , S n ) to ensure its eigenvalues are not too small. And (3) we add a second noise to the update computed using the noised gradient and modified second-order information (SOI).

Notice that Algorithm 3 has four variations based on the SOI and the modification of SOI, namely, Hess -clip , Hess -add , QU -clip , and QU -add which refer to using Hessian and clip, Hessian and add, quadratic upper bound (See Lemma 5.1) and clip, and quadratic upper bound and add, respectively.

Remark 5.7 (Generalization of Algorithm 3) . In this section our main focus is on DP logistic regression, and the privacy guarantees hold for the logistic loss. Nevertheless, in Appendix B.6, we present a generalization of Algorithm 3 whose privacy guarantee holds for every convex, doubly differentiable, Lipschitz, and smooth loss function without any constraints on the rank of Hessian . The main technical challenge for sensitivity analysis is proving the approximate Lipschitzness of Ψ in the operator norm (See Lemma B.7). This demonstrates that our algorithm is more general than objective perturbation [CMS11; KST12; INST+19] and the private damped Newton's method [ABL21] which both require a low-rank Hessian. glyph[triangleleft]

5.2 Private and Adaptive Selection of Minimum Eigenvalue

One of the hyperparameters of Algorithm 3 is the minimum eigenvalue λ 0 . There exists a tradeoff for choosing λ 0 . We ideally want the modification to be as small as possible, so that the SOI is preserved. However, decreasing λ 0 increases σ 2 and we add more noise. To deal with this problem, we propose a heuristic rule for an adaptive, private, and time-varying selection of the minimum eigenvalue. We wish to find λ 0 ,t that minimizes expected loss at the next iteration, for which we have the quadratic approximation (6). More formally, we compute λ 0 ,t as arg min λ E [ q t ( w t -Ψ λ ( H ( w t , S n ) , SOI modification )˜ g t + ‖ ˜ g t ‖ σ 2 ( λ ) · ξ )] where q t is given in (6) and ξ (0 , I d ) . We show in Appendix B.5 that an approximate minimizer is which has sensitivity 1 / 4 n rithmic description of a variant of Algorithm 3 with an adaptive and private minimum eigenvalue. In particular, we divide the privacy budget at each iteration into three parts: (1) privatizing the gradient; (2) estimating the trace of SOI; and (3) privatizing the direction. We use this variant for our numerical experiments in Section 6.

∼ N λ 0 ,t ∝ ( trace ( H t ( w t ,S n )) n 2 × privacy budget for the direction ) 1 3 . Note that λ 0 ,t depends on the data through trace ( H ( w t , S n )) , , so it can be estimated privately. In Appendix B.5, we provide the algo-

5.3 Convergence Results for Algorithm 3

In this section, we provide data-dependent convergence guarantees for Algorithm 3. We express these guarantees in terms of the conditional expectation E t [ · ] = E [ ·|{ w i } i ∈ [ t ] ] and they can be easily extended to obtain high probability bounds. Before presenting the results, we introduce a notation. For a dataset S n = (( x 1 , y 1 ) , . . . , ( x n , y n )) ∈ ( R d ×{-1 , +1 } ) n , let V ∈ R d × d denote the orthogonal projection matrix on the linear subspace spanned by { x 1 , . . . , x n } . For every vector u ∈ R d , define ‖ u ‖ V glyph[defines] √ u glyph[latticetop] V u . This norm naturally arises since for every w ∈ R d we have glyph[lscript] LL ( w,S n ) -glyph[lscript] LL ( w glyph[star] , S n ) ≤ 1 8 ‖ w -w glyph[star] ‖ 2 V where w glyph[star] = arg min glyph[lscript] LL ( w,S n ) (See Appendix B.7).

5.3.1 Local Convergence Guarantee of Hess -clip and Hess -add

Theorem 5.8. Let S n denote the dataset and rank denote the dimension of the linar subspace spanned by { x 1 , . . . , x n } . Let λ min ,t be the smallest non-zero eigenvalue of ∇ 2 glyph[lscript] LL ( w t , S n ) and ρ be

10

-

2

10

-

1

10

0

10

1

10

-

4

10

-

3

10

-

2

10

-

1

Excess Loss

synthetic

DP-GD

AOP [INSTTW19]

Damped Newton [ABL21]

Ours

10

-

2

10

-

1

10

0

10

1

10

-

3

10

-

2

10

-

1

Excess Loss

a1a

10

-

2

10

-

1

10

0

10

1

10

-

2

10

-

1

Excess Loss

covertype

10

-

2

10

-

1

10

0

10

1

10

-

2

10

-

1

Excess Loss

adult

ε

-DP

ε

-DP

ε

-DP

ε

-DP

the privacy budget (in zCDP) per iteration. Then,

where the coefficients are given by

This type of convergence is known as composite convergence , as it is a combination of linear and quadratic rates, and has been observed in the convergence analysis of several quasi-Newton's methods [EM15; Erd15; RM16; XYRRM16].

Remark 5.9 . λ min ,t is the smallest non-zero eigenvalue of ∇ 2 glyph[lscript] LL ( w t , S n ) . Therefore, for sufficiently large n we have 0 < ν 1 ,t < 1 . It shows Algorithm 3 with Hessian as SOI is, in-expectation, a descent algorithm locally given ‖ w t -w glyph[star] ‖ is sufficiently larger than ∆ . Roughly speaking, Theorem 5.8 guarantees a linear convergence to a ball around the optimum whose radius is given by ∆ . We also observe the linear rate in Figure 3. Moreover, the error due to the privacy, i.e., ∆ in Equation (7), is proportional to the rank of the feature vectors which is always smaller than d . These interesting properties is due to the convergence analysis with respect to ‖·‖ V . glyph[triangleleft]

Remark 5.10 . The coefficients of the convergence in Equation (7) depend on the iteration step which is an undesirable aspect of the results. In Lemma B.11, we prove that | λ min ,t -λ glyph[star] min | ≤ 0 . 1 ‖ w t -w glyph[star] ‖ V where λ glyph[star] min is the smallest non-zero eigenvalue of ∇ 2 glyph[lscript] LL ( w glyph[star] , S n ) . Therefore, the coefficients can be well-approximated by their analogous values evaluated at the optimum. glyph[triangleleft]

5.3.2 Global Convergence Guarantee of QU -clip and QU -add

We also establish a global convergence guarantee for QU -clip and QU -add . Due to the space the formal statement and proof are deferred to Appendix B.9. Roughly speaking, under the assumption of local strong convexity at the optimum [Bac14], QU -clip and QU -add converge globally: this is intuitive since QU -clip and QU -add are based on minimizing a global upper bound on the function.

6 Numerical Results

In this section, we evaluate the performance of our algorithm (Algorithm 3 with the adaptive minimum eigenvalue selection from Section 5.2) for the problem of binary classification using logistic regression . For brevity, many of the details behind our implementation and more experimental results are deferred to Appendix C.

6.1 Setup

The setup of the experiments is as follows: Baseline1- DP-(S)GD : The update rule is w t +1 = w t -η ∇ glyph[lscript] ( w t , S n ) + ξ where ξ is a Gaussian noise [SCS13; BST14; ACGM+16]. Since the logistic loss is 1 -Lipschitz, we do not need gradient clipping. The Lipschitzness parameter controls the variance of the Gaussian random vector. To draw a fair comparison and show the advantage of using second-order information, we chose the stepsize to be equal to the inverse smoothness. Baseline2-

0

2

4

6

8

10

12

14

Iteration

10

-

3

10

-

2

10

-

1

Excess Loss

synthetic-

ε

=1.0-DP

DP-GD

HESS-ADD

QU-ADD

HESS-CLIP

QU-CLIP

0

2

4

6

8

10

12

14

Iteration

10

-

3

10

-

2

10

-

1

Excess Loss

synthetic-

ε

=3.0-DP

0

2

4

6

8

10

12

14

Iteration

10

-

1

Excess Loss

covertype-

ε

=1.0-DP

0

2

4

6

8

10

12

14

Iteration

10

-

1

Excess Loss

covertype-

ε

=3.0-DP

Approximate Objective Perturbation (AOP) : AOP is built on objective perturbation [CMS11; KST12]. Objective perturbation consists of a two-stage process: (1) perturbing the objective function by adding a random linear term and (2) outputting the minimum of the perturbed objective. Releasing such a minimum is sufficient for achieving DP guarantees [CMS11; KST12], but only if we can find the exact minimum of the perturbed objective. AOP extends objective perturbation to permit using an approximate minimum of the perturbed objective [INST+19; INST+]. Notice AOP is not an iterative optimization algorithm. Baseline3- Damped Newton Method [ABL21] : The algorithm in [ABL21] is a variant of damped Newton's method with the assumption that the Hessian of loss function is rank-1, which holds for the logistic loss. Their algorithm is based on adding two i.i.d. noises to the Hessian and the gradient: w t +1 = w t -η t H noisy ,t ( w t , S n ) -1 ˜ g t , where η t is the stepsize, H noisy ,t ( w t , S n ) = ∇ 2 glyph[lscript] LL ( w t , S n ) + Ξ t and ˜ g t = ∇ glyph[lscript] LL ( w t , S n ) + ξ t . Here Ξ t and ξ t are carefully chosen Gaussian noise. With η t = 1 , our experiments show that their algorithm is not converging. We use the strategy suggested in [ABL21, Page 22] and set η t = log(1 + β t ) /β t where β t = ∥ ∥ ∇ 2 glyph[lscript] LL ( w t , S n ) -1 ∇ glyph[lscript] LL ( w t , S n ) ∥ ∥ . This stepsize selection makes the algorithm nonprivate , however, it serves as a good baseline. Datasets: We conducted experiments on six publicly available datasets: a1a, Adult, covertype, synthetic, fashion-MNIST, and protein (Appendix C includes fashion-MNIST and protein results). The synthetic dataset is generated as follows: Fix d ∈ N and w glyph[star] ∈ R d . Then, (1) the feature vectors { x i ∈ R d : i ∈ [ n ] } are independent and sampled uniformly at random from the unit sphere in R d , (2) for the i -th datapoint the label is +1 with probability (1 + exp( -〈 x i , w glyph[star] 〉 )) -1 and -1 otherwise. Privacy Notion: The privacy notion for our experiments is ( glyph[epsilon1], δ = ( num. of samples ) -2 ) -DP. Next, we present the results.

6.2 Privacy-Utility-Run Time Tradeoff

We study the tradeoff for our algorithm and compare it with other baselines for a broad range of ε ∈ { 0 . 01 , . . . , 10 } . We non-privately tune the total number of iterations of the iterative algorithms and report the best achievable excess error in Figure 2. As can be seen our algorithm almost always achieves the best excess loss for a broad range of glyph[epsilon1] . Also, Figure 2 shows that damped private Newton method of [ABL21] achieves a low excess loss only for large glyph[epsilon1] . Figure 2 indicates that DP-GD and our algorithm are the best in terms of excess loss. In Table 1, we compare the run time of DP-GD and our algorithm, i.e., the computational time in seconds for achieving the excess loss in Figure 2. As can be seen, for many challenging datasets, our algorithm is 10 -40 × faster than DP-GD. Our experiments are run on CPU. We also remark that each step of Algorithm 3, i.e., computing gradient and SOI, is heavily parallelizable implying that the run time of Algorithm 3 can be made much smaller by an efficient implementation. Also, the reported numbers in Figure 2 and Table 1 correspond to Hess -clip .

0

.

00

0

.

01

0

.

02

0

.

03

0

.

04

Time (sec)

10

-

2

10

-

1

Excess Loss

synthetic,

ε

=3-DP

HESS-ADD,

(

B

grad

, B

hess

)

=(2000,1000)

HESS-CLIP,

(

B

grad

, B

hess

)

=(2000,1000)

QU-CLIP,

(

B

grad

, B

hess

)

=(2000,1000)

QU-ADD,

(

B

grad

, B

hess

)

=(2000,1000)

DP-SGD,

B

grad

=300

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Time (sec)

10

-

2

10

-

1

Excess Loss

a1a,

ε

=3-DP

HESS-ADD,

(

B

grad

, B

hess

)

=(4000,2000)

HESS-CLIP,

(

B

grad

, B

hess

)

=(4000,2000)

QU-CLIP,

(

B

grad

, B

hess

)

=(4000,2000)

QU-ADD,

(

B

grad

, B

hess

)

=(4000,2000)

DP-SGD,

B

grad

=2000

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Time (sec)

10

-

1

Excess Loss

adult,

ε

=3-DP

HESS-ADD,

(

B

grad

, B

hess

)

=(3000,1000)

HESS-CLIP,

(

B

grad

, B

hess

)

=(3000,1000)

QU-CLIP,

(

B

grad

, B

hess

)

=(3000,1000)

QU-ADD,

(

B

grad

, B

hess

)

=(3000,1000)

DP-SGD,

B

grad

=1000

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Time (sec)

10

-

2

10

-

1

Excess Loss

covertype,

ε

=3-DP

HESS-ADD,

(

B

grad

, B

hess

)

=(4000,2000)

HESS-CLIP,

(

B

grad

, B

hess

)

=(4000,2000)

QU-CLIP,

(

B

grad

, B

hess

)

=(4000,2000)

QU-ADD,

(

B

grad

, B

hess

)

=(4000,2000)

DP-SGD,

B

grad

=1000

6.3 Second Order Information vs Optimal Stepsize

In non-private convex optimization, the key to the success of second-order optimization algorithms is that the second-order information acts as a preconditioner, and the same performance cannot be attained by optimally tuning the stepsize for GD algorithm. To investigate whether the same holds for our algorithms, we consider the following variant of DP-GD. Let ˜ g t denote the perturbed gradient obtained by adding a Gaussian random vector to ∇ glyph[lscript] LL ( w t , S n ) . Instead of a constant stepsize, the stepsize at iteration t is chosen based on η t = arg min η ≥ 0 glyph[lscript] LL ( w t -η ˜ g t ) . Notice this variant is obviously not DP . We refer to this variant as DP-GD-Oracle . The comparison with DP-GD-Oracle lets us answer the following question: Could we have just computed a single number, i.e., stepsize, to achieve the same performance as our second-order optimization algorithms which require computing a d × d matrix? In Figure 3, we compare the convergence speed of our algorithms with DP-GD-Oracle in low- and high-privacy regimes. Figure 3 shows our algorithms converge faster than DP-GD-Oracle which is not even a DP algorithm. Figure 3 confirms the expectation that as the privacy budget increases the difference between our algorithms and DP-GD-Oracle increases since we can use more curvature information.

6.4 Minibatch Variant of Our Algorithm and Comparison with DP-SGD

So far we have considered full-batch algorithms that compute first- and second-order information on the entire dataset. We extend Algorithm 3 to the minibatch setting, where, at each iteration, the gradient and SOI matrix are computed using a subsample of the data points. In Appendix C.1 we provide a formal algorithmic description of the minibatch version of Algorithm 3 along with its privacy proof. Then, we compare the convergence speed and excess loss with DP-SGD.

DP-SGD is faster than DP-GD, but to achieve good privacy and utility, we need large batches [PHKX+23, Fig. 2]. This is in stark contrast with non-private SGD, where larger batch sizes yield diminishing returns [ZLNM+19]. In particular, to achieve the best excess loss we need to select the batch size as large as possible. We select the batch size of DP-SGD so that the achievable excess loss will be close to the full batch versions. Specifically, we select batch size DP-SGD number of samples ≈ 0 . 02 and tune the number of iterations of DP-SGD to obtain the best result. Figure 4 shows the progress of different algorithm versus run time. Obviously, for a fixed run time DP-SGD performs more iterations compared to our algorithms. Nevertheless, our algorithms achieve the same excess error as DP-GD with 8 -10 × faster run time over all the datasets while the batch sizes of our algorithms are larger than that of DP-SGD . We observe that the variations of our algorithms based on the adding operator performs better in the minibatch setting. This can be attributed to the smaller σ 2 for the adding operator in Algorithm 3. In summary, the comparison between privacy-utility-wall time tradeoff of the subsampled variant of our algorithm and DP-SGD is similar to their full-batch counterparts.

7 Conclusion and Limitations

We showed that second-order methods can be used in the DP setting both for improving worst-case convergence guarantees and designing faster practical algorithms. We believe our results open up many directions: A limitation of our algorithms is that the cost of forming and inverting the Hessian can be prohibitive when d is large. In the non-private setting, a line of research tries to address this limitation by constructing an approximation to SOI such that the update is efficient, yet still provides sufficient SOI [EM15; Erd15; XYRRM16; ABH17]. It would be interesting to investigate how the ideas developed in our paper could be incorporated into these methods.

Acknowledgments

The authors would like to thank Murat Erdogdu, Jalaj Upadhyay, and Mohammad Yaghini for helpful discussions. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute www.vectorinstitute.ai/partners .

References

[Erd15] M. A. Erdogdu. 'Newton-Stein method: A second order method for GLMs via Stein's lemma'. Advances in Neural Information Processing Systems 28 (2015).

[GKLR19] R. Gower, D. Kovalev, F. Lieder, and P. Richtárik. 'RSN: randomized subspace Newton'. Advances in Neural Information Processing Systems 32 (2019).

[GL12] S. Ghadimi and G. Lan. 'Optimal Stochastic Approximation Algorithms for Strongly Convex Stochastic Composite Optimization I: A Generic Algorithmic Framework'. SIAM Journal on Optimization 22.4 (2012), pp. 1469-1492. eprint: https://doi. org/10.1137/110848864 .

[GLL22] S. Gopi, Y. T. Lee, and D. Liu. 'Private Convex Optimization via Exponential Mechanism'. In: Proceedings of Thirty Fifth Conference on Learning Theory . Ed. by P.-L. Loh and M. Raginsky. Vol. 178. Proceedings of Machine Learning Research. PMLR, Feb. 2022, pp. 1948-1989.

[GTU22] A. Ganesh, A. Thakurta, and J. Upadhyay. 'Langevin diffusion: An almost universal algorithm for private Euclidean (convex) optimization'. arXiv preprint arXiv:2204.01585 (2022).

[GV13] G. H. Golub and C. F. Van Loan. Matrix computations . JHU press, 2013.

[HLR19] N. J. Harvey, C. Liaw, and S. Randhawa. 'Simple and optimal highprobability bounds for strongly-convex stochastic gradient descent'. arXiv preprint arXiv:1909.00843 (2019).

[INST+] R. Iyengar, J. P. Near, D. Song, O. Thakkar, et al. Differentially Private Convex Optimization Benchmark . URL: https://github.com/sunblaze- ucb/dpmlbenchmark .

[INST+19] R. Iyengar, J. P. Near, D. Song, O. Thakkar, et al. 'Towards practical differentially private convex optimization'. In: 2019 IEEE Symposium on Security and Privacy (SP) . 2019.

[JM23] Q. Jin and A. Mokhtari. 'Non-asymptotic superlinear convergence of standard quasiNewton methods'. Mathematical Programming 200.1 (2023), pp. 425-473.

[JNGKJ19] C. Jin, P. Netrapalli, R. Ge, S. M. Kakade, and M. I. Jordan. 'A short note on concentration inequalities for random vectors with subgaussian norm'. arXiv preprint arXiv:1902.03736 (2019).

[JT16] F. Jarre and P. L. Toint. 'Simple examples for the failure of Newton's method with line search for strictly convex minimization'. Mathematical Programming 158.1 (2016), pp. 23-34.

[Kat73] T. Kato. 'Continuity of the map S →| S | for linear operators'. Proceedings of the Japan Academy 49.3 (1973), pp. 157-160.

[KSJ18] S. P. Karimireddy, S. U. Stich, and M. Jaggi. 'Global linear convergence of Newton's method without strong-convexity or Lipschitz gradients'. arXiv preprint arXiv:1806.00413 (2018).

[KST12] D. Kifer, A. Smith, and A. Thakurta. 'Private Convex Empirical Risk Minimization and High-dimensional Regression'. In: Proceedings of the 25th Annual Conference on Learning Theory . Ed. by S. Mannor, N. Srebro, and R. C. Williamson. Vol. 23. Proceedings of Machine Learning Research. Edinburgh, Scotland: PMLR, 25-27 Jun 2012, pp. 25.1-25.40.

[Lyu22] X. Lyu. 'Composition Theorems for Interactive Differential Privacy'. arXiv preprint arXiv:2207.09397 (2022).

[Mir75] L. Mirsky. 'A trace inequality of John von Neumann'. Monatshefte für mathematik 79.4 (1975), pp. 303-306.

[MRTZ17] H. B. McMahan, D. Ramage, K. Talwar, and L. Zhang. 'Learning differentially private recurrent language models'. arXiv preprint arXiv:1710.06963 (2017).

[MTZ19] I. Mironov, K. Talwar, and L. Zhang. 'Renyi differential privacy of the sampled Gaussian mechanism'. arXiv preprint arXiv:1908.10530 (2019).

[Nes98] Y. Nesterov. 'Introductory lectures on convex programming volume i: Basic course'. Lecture notes 3.4 (1998), p. 5.

[NJLS09] A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. 'Robust Stochastic Approximation Approach to Stochastic Programming'. SIAM Journal on Optimization 19.4 (2009), pp. 1574-1609. eprint: https://doi.org/10.1137/070704277 .

[NP06] Y. Nesterov and B. T. Polyak. 'Cubic regularization of Newton method and its global performance'. Mathematical Programming 108.1 (2006), pp. 177-205.

[NW99] J. Nocedal and S. J. Wright. Numerical optimization . Springer, 1999.

[PHKX+23] N. Ponomareva, H. Hazimeh, A. Kurakin, Z. Xu, et al. 'How to dp-fy ml: A practical guide to machine learning with differential privacy'. arXiv preprint arXiv:2303.00654 (2023).

[PS22] N. Papernot and T. Steinke. 'Hyperparameter Tuning with Renyi Differential Privacy'. In: International Conference on Learning Representations . 2022.

[RM16] F. Roosta-Khorasani and M. W. Mahoney. 'Sub-sampled newton methods ii: Local convergence rates'. arXiv preprint arXiv:1601.04738 (2016).

[RM19] F. Roosta-Khorasani and M. W. Mahoney. 'Sub-sampled Newton methods'. Mathematical Programming 174 (2019), pp. 293-326.

[SCS13] S. Song, K. Chaudhuri, and A. D. Sarwate. 'Stochastic gradient descent with differentially private updates'. In: 2013 IEEE global conference on signal and information processing . IEEE. 2013, pp. 245-248.

[SSTT21] S. Song, T. Steinke, O. Thakkar, and A. Thakurta. 'Evading the curse of dimensionality in unconstrained private glms'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2021, pp. 2638-2646.

[Ste98] G. W. Stewart. Matrix algorithms: volume 1: basic decompositions . SIAM, 1998.

[STT20] S. Song, O. Thakkar, and A. Thakurta. 'Characterizing Private Clipped Gradient Descent on Convex Generalized Linear Problems'. arXiv preprint arXiv:2006.06783 (2020).

[STU17] A. Smith, A. Thakurta, and J. Upadhyay. 'Is interaction necessary for distributed private learning?' In: 2017 IEEE Symposium on Security and Privacy (SP) . IEEE. 2017, pp. 58-77.

[VZ22] S. Vadhan and W. Zhang. 'Concurrent Composition Theorems for Differential Privacy'. arXiv preprint arXiv:2207.08335 (2022).

[WLKC+17] X. Wu, F. Li, A. Kumar, K. Chaudhuri, et al. 'Bolt-on Differential Privacy for Scalable Stochastic Gradient Descent-based Analytics'. In: Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD . Ed. by S. Salihoglu, W. Zhou, R. Chirkova, J. Yang, and D. Suciu. 2017.

[XRV17] H. Xiao, K. Rasul, and R. Vollgraf. 'Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms'. arXiv preprint arXiv:1708.07747 (2017).

[XYRRM16] P. Xu, J. Yang, F. Roosta, C. Ré, and M. W. Mahoney. 'Sub-sampled Newton methods with non-uniform sampling'. Advances in Neural Information Processing Systems 29 (2016).

[ZLNM+19] G. Zhang, L. Li, Z. Nado, J. Martens, et al. 'Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model'. Advances in neural information processing systems 32 (2019).

[ZZMW17] J. Zhang, K. Zheng, W. Mou, and L. Wang. 'Efficient private ERM for smooth objectives'. In: Proceedings of the 26th International Joint Conference on Artificial Intelligence . 2017, pp. 3922-3928.

Peer reviews:
comments: SUMMARY:
This paper studies DP convex optimization by using second-order methods. The authors present a private version of the cubic regularized Newton method and prove it faster than the first-order methods under strongly convex case. They also provide efficient second-order method for solving the DP logistic regression problems.

STRENGTHS:
This is the very first paper to study the second-order methods for DP convex optimization. The author present algorithms for both general strongly convex functions and logistic regressions which are very novel. The Algorithm 3 for DP logistic regression consider the special structure of the object function which is very interesting.

WEAKNESSES:
The algorithm for solving the general strongly convex functions seems straight forward. It is a combination of cubic-regularized Newton and DPGD. The requirement of solving subproblem by DPGD at each iteration is not satisfying and may cause the total computation cost even worse than the first-order method.  Can the author present the total number of the iterations (including the inner loop) and compare it with the first-order DP method?

Although the author present efficient method for solving the DP logistic regression, Algorithm 3 seems quite different from the meta algorithm for the general strongly convex optimization problems. Can the authors show the connection between the Algorithm 3 and Algorithm1,2?

QUESTIONS:
See weakness part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
This work investigates the use of second-order methods in differentially private convex optimization for machine learning. The authors propose a private variant of the regularized cubic Newton method and demonstrate its quadratic convergence and optimal excess loss for strongly convex loss functions. They also design a practical second-order differentially private algorithm for unconstrained logistic regression, which outperforms other baselines in terms of excess loss and is significantly faster than DP-GD/DP-SGD, achieving a speedup of 10-40 times.

STRENGTHS:
This paper is good as it has been a really hard to use second-order information while preserving DP property. The contribution is novel and the proof seems to be correct.

WEAKNESSES:
1. Maybe extend to use laplace noise? 
2. The presentation of the paper is a little bit awkward. For example, the formal DP theorems and proof of Alg. 3 only presents in Appendix. It is really hard to find the corresponding reference. This paper might need to be rewritten.
3. Experiments:
    1. Compare the true running time. It takes a lot of time to compute the CLIP and ADD operations as it needs to compute SVD. So instead of seeing how many steps it needs, please present the true computational time to show the true running time improvement compared with DP-SGD and DP-GD.
    2. Compare memory usage. SVD needs tons of memory.
    3. DP-SGD brings randomness and sometimes accelerates the training procedure. Please add a comparison with DP-SGD.
    4. Compare with different learning rates (I know there does not exist a learning rate in Alg. 3 but a learning rate exists in DP-(S)GD). It would be interesting to see the impact of different learning rates (10, 0.1, or 0.01).

QUESTIONS:
See Weakness.

LIMITATIONS:
See Weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the use of second-order methods in differentially private optimization. In particular, two algorithms are proposed. One is based on cubic regularized Newton and works for strongly-convex functions. The other one is designed specifically for the logistic regression problem. Numerical experiments on the logistic regression are provided.\n\nSTRENGTHS:\nThe topic of using second-order methods in differentially private (DP) optimization is under-explored. In this sense, the paper can be a good starting point and introduce the idea to the DP community. The paper is well-written and easy to follow, with theoretical results that do not exist before and are supported by numerical experiments.\n\nWEAKNESSES:\n1. The theoretical justification of the benefit of using second-order methods is not strong enough. For the considered smooth strongly-convex setting, first-order methods can already achieve the same rate in linear time $O(n)$ [arXiv:1703.09947, arXiv:1802.05251, arXiv:2005.04763, arXiv:2102.05855, arXiv:2206.00363]. As a comparison, since the subproblem of the cubic regularized Newton cannot be efficiently solved due to nonsmoothness, the total gradient complexity of the proposed method is $NT\\sim n^2$. The main improvement is mostly on the oracle complexity but with more assumptions on the second-order information. Even for the oracle complexity, the proposed algorithm achieves $\\sqrt{L_2}/\\mu^{3/4}+\\log\\log(n)$, which is hard to compare with the one achieved by first-order methods (e.g., $(\\sqrt{L_1}/\\sqrt{\\mu})\\log(n)$ for private versions of the Nesterov's accelerated gradient descent [arXiv:2102.04704, arXiv:2206.00363]). I am also wondering if an improved lower-bound is possible with second-order information.\n\n2. The convergence analysis of the logistic regression case seems to be not complete. I don't find (also in the appendix) any specific rate, complexity, and comparison with first-order methods. The proposed method requires the local strong convexity assumption at the optimum. Does it hold for the logistic regression loss?\n\n3. The proposed method requires computing the inverse of the second-order information, which could be hard for large-scale experiments. Also, all the numerical experiments in this paper use small models and datasets, where computing this inverse does not cost too much. Given that DP-GD already completes the task within 1 minute, such a 10-40$\\times$ improvement is not considered to be so surprising.\n\n4. What does $T^*$ mean in Table 1, or what is the stopping criterion to define $T^*$? Why do Figure 3 & 4 only show the first 10 steps or 0.6s? I guess the noise added for the proposed method is computed according to the theory. Are there any numerical privacy accounting methods used in the experiments to justify the effectiveness of the added noises? In case the privacy analysis is wrong and less noise is added than what is required, this might not be a fair comparison.\n\n5. Minors: It might be good to also summarize related works for non-DP second-order optimization algorithms. What does privacy budget for direction mean in $\\lambda_{0,t}$ in line 249.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of differentially-private convex optimization and discusses how second-order information can be utilized to accelerate the optimization process while also achieving the optimal excess error, with DP involved. The main contributions of this paper are the two proposed algorithms, and the corresponding analyses; 1) A second-order DP algorithm that is based on the cubic-regularized Newton's method - for a specific class of convex functions, 2) A second-order DP algorithm for the logistic regression problem.\n\nSTRENGTHS:\nThe paper considers an important problem which studies how the performance of a differentially-private convex optimization process can be enhanced with the use of second-order information. The paper is well organized and rigorous proofs have been provided for the claims made. The first algorithm presented in this paper (DP variant of the cubic-regularized Newton's Method) achieves the optimal excess loss and quadratic convergence. The proposed second-order DP algorithm for logistic regression achieves equal or better excess losses and lower computational times compared to the existing DP algorithms, based on the experimental results.\n\nWEAKNESSES:\nThe authors could clearly state their contributions and remark on the significance of the contributions, as in some cases (for example algorithms 1 and 2), it seems like the authors have simply incorporated DP into existing non-private results. The main paper lacks justification on the privacy guarantees. The authors could comment on how the stated DP guarantees are achieved by the selected noise parameters.\n\nQUESTIONS:\n1) For a given set of inputs in algorithm 3, how can one determine whether the SOI modification is \"add\" or \"clip\"? The reader could benefit from a clear description of this part (lines 3-6) in algorithm 3.\n2) Do the authors assume any characteristics (i.e., any specific distribution or any statistical characteristics) on the dataset $S_n$ in either of the algorithms?\n\nLIMITATIONS:\nThe authors have clearly stated the limited types of functions which the proposed algorithms can be applied on, and have also commented on possible extensions to other classes of functions.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an algorithm that utilizes second-order information to enhance the speed of private convex optimization, concurrently ensuring optimization excess error is minimal.\n\nThe outcomes of the experiments indicate that employing second-order information can expedite Differential Privacy (DP) optimization. In addition, it achieves an excess loss that either equals or surpasses that of first-order methods like DP Gradient Descent (DP-GD).\n\nSTRENGTHS:\nThe strength of this paper are\n\n1) The application of second-order methods to convex optimization is challenging research area. While some progress has been made, it's still uncertain whether second-order methods can be as practical as first-order methods. This paper unveils a novel second-order approach for Differential Privacy (DP) optimization, which demonstrates optimal efficiency for strongly convex functions.\n\n2) The authors offer an analysis of both the local convergence assurance associated with Hess-clip and Hess-add, as well as the global convergence guarantee for QU-clip and QU-add.\n\n3) The numerical findings indicate that the proposed method outperforms DP Gradient Descent (DP-GD) substantially, demonstrating a speed that is 10 to 40 times quicker for the datasets tested.\n\nWEAKNESSES:\nThe weakness the paper include\n1) The potential impact of this work remains uncertain to me. While it presents strong results in the field of Differential Privacy (DP) private machine learning, it raises the question: can these results or concepts be applied more broadly?\n\n2) The mini-batch version offers interesting numerical results, however, the loss it produces is notably higher compared to the full-batch version.\n\nQUESTIONS:\nOn page 1, \"One of the major drawbacks of DP-(S)GD is slow convergence. We argue that the main reason for this is the difficulty of choosing the hyperparameters (/eta; T).\" Please explain why it is that. How about gradients?\n \nFor minibatch version, how to make sure the second-order information is still relevant/meaningful from one batch to another batch?\n\nLIMITATIONS:\nYes, I think so.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of private convex optimisation using second order information where the main motivation is the acceleration of private convex optimisation problem as DP-SGD shows slow convergence. This is done in two parts. Firstly, they introduce the privatised version of the cubic method of Nesterov and Polyak when the loss function is strongly convex. In this step, the privatisation is done by solving the optimisation problem given by the global cubic upper-bound privately (DP-Solver). They analyse the algorithm and provide convergence guarantees. \n\nSince the cubic method is computationally expensive, for the rest of the paper the authors focus on a method inspired by Newton's method with its specific application to logistic regression. Specifically, they provide a global upper bound for the logistic loss and for which the optimisation the upper bound has a closed form. They then privatise this method by privatising the steps of the optimisation algorithm. They do this in two steps. First by adding the proper amount of noise to the information from gradient and then adding noise at the update step.\n\nFinally, they empirically compare the performance of their proposed method for logistic regression to DP-(S)GD and objective perturbation.\n\nSTRENGTHS:\n1. Convex optimisation appears in many settings in Machine Learning and given the increasing attention to privacy, this is a timely problem to study.\n\n2. The combination of ideas to use a method similar to newton's method while making sure the upper bound is global like in the cubic method is nice.\n\n3.  The convergence analysis for the proposed algorithms.\n\n4. The details of the experiments are explained thoroughly and for the most part, they seem fair.\n\nWEAKNESSES:\n1. While private convex optimisation is an important problem and the techniques are interesting, I feel the scope of the paper is a bit limited as it only allows us to use the results for logistic regression in a private setting. For any other method that uses convex optimisation, like kernel methods, the user would need to calculate the sensitivity of the queries in algorithm 3 at which point the method being faster than DP-SGD might not be justification enough for this method.\n\n\n2. One of the main motivations for this paper is the slow convergence of DP-SGD. While the authors provide references to other bottlenecks for DP-SGD like the batch size and hyper-parameter tuning, there are no references for the slow-convergence of DP-SGD and the main thing supporting this claim in the paper is Figure 1.\n\nQUESTIONS:\n1. The upper-bound for the convergence analysis for  ($(lw) - l(w^*)$) in both Theorem 1 and Theorem 2 is done w.r.t $w^*$ which is the optimal w not attainable in the case where $\\rho>0$ as given by the lower-bound given in [BST14].  Does it not make more sense to provide the upper bound w.r.t the set of parameters $w$ which are achievable under privacy constraints?\n\n2. Throughout the paper you mention that your proposed method is 10-40 times faster but looking at Table 1 this is the case for $\\epsilon = 10$, otherwise the range is something like 3-40. Is there a reason for mentioning 10-40?\n\n3. [KSJ18] show that the Newton's method globally converges under some conditions which hold for logistic regression. Given this result, why do we need to build the global upper bound of lemma  5.1?\n\n4. Is the idea for building global upper  bounds for newton's something done for the first time in your paper? If not, it might be a good idea to add some references.\n\n\n\n\n\n\nMinor comments:\n\n1. You can add other examples where convex optimisation appears in ML. Some examples are kernel regression and extreme learning machines.\n\n2. I think adding some explanation and details to Theorem 4.2 might help the reader to digest the theorem a bit better i.e. What is the value of $w$ for which the transition between the convergence rates happen? My intuition is that the privacy constraint makes a set of parameters centred around $w^*$)admissible and while $w$ is outside of this range the convergence rate is slower and once we are within those parameters the convergence is much much faster. \n\n3. For theorem 5.6, it might help to mention that the relationship between the semi-norm and the $\\ell^2$ norm for ease of understanding.\n\nLIMITATIONS:\nIt is similar to the comment I make in the weaknesses. The authors have not clearly mentioned that while the method is faster than DP-SGD, it requires an expert to get the algorithm up and running for other convex optimisation problems.\n\nAdditionally, it is known that the computational complexity of Newton's method scales badly with dimension d which I think should be mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}