Title: Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

Non-convex optimization

Stationary points

Exponential Mechanism
Abstract: We reconsider the challenge of non-convex optimization under differential privacy constraint. Building upon the previous variance-reduced algorithm SpiderBoost, we propose a novel framework that employs two types of gradient oracles: one that estimates the gradient at a single point and a more cost-effective option that calculates the gradient difference between two points. Our framework can ensure continuous accuracy of gradient estimations and subsequently enhances the rates of identifying second-order stationary points.
Additionally, we consider a more challenging task by attempting to locate the global minima of a non-convex objective via the exponential mechanism without almost any assumptions. Our preliminary results suggest that the regularized exponential mechanism can effectively emulate previous empirical and population risk bounds, negating the need for smoothness assumptions for algorithms with polynomial running time. Furthermore, with running time factors excluded, the exponential mechanism demonstrates promising population risk bound performance, and we provide a nearly matching lower bound.

Private (Stochastic) Non-Convex Optimization Revisited: Second-Order Stationary Points and Excess Risks

Abstract

We reconsider the challenge of non-convex optimization under differential privacy constraint. Building upon the previous variance-reduced algorithm SpiderBoost, we propose a novel framework that employs two types of gradient oracles: one that estimates the gradient at a single point and a more cost-effective option that calculates the gradient difference between two points. Our framework can ensure continuous accuracy of gradient estimations and subsequently enhances the rates of identifying second-order stationary points. Additionally, we consider a more challenging task by attempting to locate the global minima of a nonconvex objective via the exponential mechanism without almost any assumptions. Our preliminary results suggest that the regularized exponential mechanism can effectively emulate previous empirical and population risk bounds, negating the need for smoothness assumptions for algorithms with polynomial running time. Furthermore, with running time factors excluded, the exponential mechanism demonstrates promising population risk bound performance, and we provide a nearly matching lower bound.

1 Introduction

Differential privacy [19] is a standard privacy guarantee for training machine learning models. Given a randomized algorithm A : P ∗ → R , where P is a data domain and R is a range of outputs, we say A is ( ε, δ ) -differentially private (DP) for some ε ≥ 0 and δ ∈ [0 , 1] if for any neighboring datasets D , D ′ ∈ P ∗ that differ in at most one element and any R⊆ R , the distribution of the outcome of the algorithm, e.g., pair of models trained on the respective datasets, are similar:

Smaller ε and δ imply the distributions are closer; hence, an adversary accessing the trained model cannot tell with high confidence whether an example x was in the training dateset. Given this measure of privacy, we consider the problem of optimizing a non-convex loss while ensuring a desired level of privacy. In particular, suppose we are given a dataset D = { z 1 , . . . , z n } drawn i.i.d. from underlying distribution P . Each loss function f ( · ; z ) : K → R is G -Lipschitz over the convex set K ⊂ R d of diameter D . Let the population risk function be F P ( x ) := E z ∼P [ f ( x ; z )] and the empirical risk function be F D ( x ) := 1 n ∑ z ∈D f ( x ; z ) . We also denote F S ( x ) := 1 | S | ∑ z ∈ S f ( x ; z ) for S ⊆ D .

Daogao Liu

Our focus is in minimizing non-convex (empirical and population) risk functions, which may have multiple local minima. Since finding the global optimum of a non-convex function can be challenging, an alternative goal in the field is to find stationary points: A first-order stationary point is a point with a small gradient of the function, and a second-order stationary point is a first-order stationary point where additionally the function has a positive or nearly positive semi-definite Hessian. As first order stationary points can be saddle points or even a local maximum, we focus on the problem of finding a second order stationary point, i.e., a local minimum, privately. Existing works in finding approximate SOSP privately only give guarantees for the empirical function F D . We improve upon the state-of-the-art result for empirical risk minimization and give the first guarantee for the population function F P . This requires standard assumptions on bounded Lipschitzness, smoothness, and Hessian Lipschitzness, which we make precise in Section 2 and in Assumption 3.1.

Compared to finding a local minimum, finding a global minimum can be extremely challenging. We also present two methods, polynomial and exponential time, that outperform existing guarantees measured in excess risks for respective computational complexities. Our primary results are succinctly summarized in Table 1.

Related Work. We propose a novel and simple framework based on SpiderBoost [52], and its private version [2] that achieves the current best rate for finding the first order stationary point privately. We discuss the primary difference between our framework and theirs, that is their algorithms only promise small gradient estimation errors on average, but our framework can ensure small estimation errors consistently throughout all the iterations, and the motivation behind this briefly.

In SGD and its variants, the typical approach involves obtaining an estimation ∆ t of the gradient ∇ f ( x t ) . In the stochastic variance-reduced algorithm SpiderBoost [52, 2], it queries the gradient O 1 ( x t ) ≈ ∇ f ( x t ) directly every q steps with some oracle O 1 , and for the other q -1 steps within each period, it queries the gradient difference between two steps, that is O 2 ( x t , x t -1 ) ≈ ∇ f ( x t ) -∇ f ( x t -1 ) , and maintain ∆ t = ∆ t -1 + O 2 ( x t , x t -1 ) . The contrast between these two types of oracles can be perceived as O 1 being more accurate but also more costly, in terms of computation or privacy budget, although our framework does not strictly necessitate this assumption.

As SpiderBoost queries O 1 every q steps, the error on the estimation may accumulate and ‖ ∆ t -∇ f ( x t ) ‖ can become large. Despite this, as demonstrated in [2], these estimations can, on average, suffice to find a private FOSP. However, such large deviations pose a challenge when scrutinizing behavior near a saddle point. For instance, when the current point is a saddle point, but the current estimation is unsatisfactory, it becomes uncertain whether the algorithm can escape the saddle point. It could be argued that average good estimations could achieve a SOSP, but to the best of our knowledge, there is no existing result addressing this concern.

A plausible solution to this challenge is to maintain high-quality gradient estimations throughout all iterations, a feat accomplished by our framework. We believe this feature holds promise for improving the outcomes of various other optimization problems, thus enhancing the overall appeal and significance of our work.

1.1 Main Results

SOSP. One of our main contributions is a refined optimization framework (Algorithm 1), predicated on the variance-reduced SpiderBoost [52], which guarantees consistently accurate gradient estimations. By integrating this framework with private gradient oracles, we achieve improved error rates for privately identifying SOSP of both empirical and population risks.

Advances in private non-convex optimization have focused on finding a first-order stationary point (FOSP), whose performance is measured in ( i ) the norm of the empirical gradient at the solution x , i.e., ‖∇ F D ( x ) ‖ , and ( ii ) the norm of the population gradient, i.e., ‖∇ F P ( x ) ‖ . We survey the recent progress in the appendix in detail.

Definition 1.1 (First-order stationary point) . We say x ∈ R d is a First-Order Stationary Point (FOSP) of g : R d → R iff ∇ g ( x ) = 0 . x is an α -FOSP of g, if ‖∇ g ( x ) ‖ 2 ≤ α .

Since FOSP can be a saddle point or a local maxima, finding a second-order stationary point is desired. Exact second-order stationary points can be extremely challenging to find [25]. Instead, progress is commonly measured in terms of how well the solution approximates an SOSP.

Definition 1.2 (Second-order stationary point, [1]) . We say a point x ∈ R d is a Second-Order Stationary Point (SOSP) of a twice differentiable function g : R d → R iff ‖∇ g ( x ) ‖ 2 = 0 and ∇ 2 g ( x ) glyph[followsequal] 0 . We say x ∈ R d is an α -SOSP for ρ -Hessian Lipschitz function g , if ‖∇ g ( x ) ‖ 2 ≤ α ∧ ∇ 2 g ( x ) glyph[followsequal] - √ ραI .

On the empirical risk F D , the SOTA on privately finding α -SOSP is by [46, 48], which achieves α = ˜ O (min { ( √ d/n ) 1 / 2 , ( d/n ) 4 / 7 } ) . In Theorem 4.2, we show that applying the proposed Algorithm 1 achieves a rate bounded by α = ˜ O (( √ d/n ) 2 / 3 ) , which improves over the SOTA in all regime. 2 There remains a factor ( √ d/n ) -1 / 6 gap to a known lower bound of α = Ω( √ d/n ) that holds even if finding only an α -FOSP [2]. On the population risk F P , applying Algorithm 1 with appropriate private gradient oracles is the first private algorithm to guarantee finding an α -SOSP with α = ˜ O ( n -1 / 3 + ( √ d/n ) 3 / 7 ) in Theorem 4.6. There is a gap to a known lower bound of α = Ω(1 / √ n + √ d/nε ) that holds even if finding only an α -FOSP [2].

Minimizing Excess Risk. In addition to the optimization framework, we present sampling-based algorithms designed to identify a private solution x priv ∈ R d that minimizes both the excess empirical risk: E [ F D ( x priv )] -min x ∈K F D ( x ) , and the excess population risk: E [ F P ( x priv )] -min x ∈K F P ( x ) . Here, the expectation is over the randomness of the solution x priv and the drawing of the training date over P . Our method is different from [46], which Gradient Langevin Dynamics and achieves in polynomial time a bound of O ( d √ log(1 /δ ) / ( ε 2 log n )) for both excess empirical and population risks with a need for the smoothness assumption. In Table 1 we omit excess empirical risk, as the bounds align with those of the population risk. We introduce a sampling-based algorithm from the exponential mechanism, which runs in polynomial time and achieves excess empirical and population risks bounded by O ( d √ log(1 /δ ) / ( ε log( nd ))) with improved dependence on ε (Theorem 5.6). Crucially, it achieves these results without the need for the smoothness assumption required by [46].

In the case of permitting an exponential running time, [23] demonstrated ˜ O ( d/ ( εn )) upper bound for non-convex excess empirical risks alongside a nearly matching lower bound. However, establishing a tight bound for the excess population risk remained an unresolved problem. We address this open question by providing nearly matching upper and lower bounds of ˜ Θ( d/ ( εn ) + √ d/n ) for the excess population risk (Theorem 5.8).

1.2 Our Techniques

Stationary Points. In our framework, we deviate from the traditional approach of querying O 1 once every q steps. Instead, we introduce a novel but simple method of monitoring the total drift we make, that is drift t = ∑ t i = τ t ‖ x i -x i -1 ‖ 2 2 , where τ t represents the last timestamp when we employed O 1 . As we are considering smooth functions, the maximum error to estimate ∇ f ( x t ) -∇ f ( x t -1 )

is proportional to ‖ x t -x t -1 ‖ 2 . If the value drift t is small, we know the current estimation should still be good enough, eliminating the need for an expensive fresh estimation from O 1 . Conversely, when drift t is large, the gradient estimation error may be substantial, necessitating a query to O 1 and thus obtaining ∆ t = O 1 ( x t ) . To effectively manage the total cost, it is crucial to set an appropriate threshold to decide when the drift is significant. A smaller threshold would ensure more accurate estimations but might incur higher costs due to more frequent queries to O 1 .

Our aim is to bound the total occurrences of the event that drift t is large, which leads to querying O 1 . A crucial observation is that, if drift t increases rapidly, then the gradient norms are large and hence function values decrease quickly, which we know does not happen frequently under the standard assumption that the function is bounded.

In our framework, we assume O 1 ( x ) is an unbiased estimation of ∇ f ( x ) , and O 1 ( x ) -∇ f ( x ) is Norm-SubGaussian (Definition 2.2), and similarly O 2 ( x, y ) is an unbiased estimation of ∇ f ( x ) -∇ f ( y ) whose error is also Norm-SubGaussian. In the empirical case, we can simply add Gaussian noises with appropriately chosen variances to the gradients of the empirical function ∇ F D for simplicity, and one can choose a smaller batch size to reduce the computational complexity. In the population case, we draw samples from the dataset without replacement to avoid dependence issues, and add the Gaussian noises to the sampled gradients. Hence we only need the gradient oracle complexity to be linear in the size of dataset for the population case.

Minimizing Excess Risk. Our polynomial time approach harnesses the power of the Log-Sobolev Inequality (LSI) and the classic Stroock perturbation lemma. The previous work of [39] shows that if the density exp ( -βF D ( x ) -r ( x ) ) satisfies the LSI for some regularizer r , then sampling a model x from this density is DP with an appropriate ( ε, δ ) . If r is a µ strongly convex function, then the density proportional to exp( -r ) satisfies LSI with constant 1 /µ , and exp( -βF D ( x ) -r ( x )) satisfies LSI with constant exp(max x,y | F D ( x ) -F D ( y ) | ) /µ by the Stroock perturbation lemma. Our bound on the empirical risk follows from choosing the appropriate inverse temperature β and regularizer r to satisfy ( ε, δ ) -DP. The final bound on the population risk also follows from LSI, which bounds the stability of the sample drawn from the respective distribution.

When running time is not a priority, we employ an exponential mechanism over a discretization of K to establish the upper bound. The empirical risk bound derives from [9], and we leverage the concentration of sums of bounded random variables to bound the maximum difference over the discretizations between the empirical and population risk. We show this is nearly tight by reductions from selection to non-convex Lipschitz optimization of [23].

1.3 Organization

In Section 2, we present necessary definitions and backgrounds for our work. In Section 3, we construct the optimization framework, with guarantees on finding the SOSP with two different kinds of SubGaussian gradient oracles. It's crucial to note that this framework focuses solely on optimization and does not pertain to privacy. Section 4 explores the pursuits of finding the SOSP privately by constructing private SubGaussian gradient oracles and seamlessly integrating them into the existing framework. We bound the private excess bounds in Section 5. For other preliminaries, all omitted proofs and some further discussions on related work can be found in the Appendix.

2 Preliminaries

Throughout the paper, if not stated explicitly, the norm ‖ · ‖ means the glyph[lscript] 2 norm.

Definition 2.1 (Lipschitz, Smoothness and Hessian Lipschitz) . Given a function f : K → R , we say f is G -Lipschitz, if for all x 1 , x 2 ∈ K , | f ( x 1 ) -f ( x 2 ) | ≤ G ‖ x 1 -x 2 ‖ , we say a function f is M -smooth, if for all x 1 , x 2 ∈ K , ‖∇ f ( x 1 ) -∇ f ( x 2 ) ‖ ≤ M ‖ x 1 -x 2 ‖ . and we say the function f is ρ -Hessian Lipschitz, if for all x 1 , x 2 ∈ K , we have ‖∇ 2 f ( x 1 ) -∇ 2 f ( x 2 ) ‖ ≤ ρ ‖ x 1 -x 2 ‖ .

Definition 2.2 (SubGaussian, and Norm-SubGaussian) . A random vector x ∈ R d is SubGaussian ( SG( ζ ) ) if there exists a positive constant ζ such that E e 〈 v,x -E x 〉 ≤ e ‖ v ‖ 2 ζ 2 / 2 , ∀ v ∈ R d . x ∈ R d is norm-SubGaussian ( nSG( ζ ) ) if there exists ζ such that Pr[ ‖ x -E x ‖ ≥ t ] ≤ 2 e -t 2 2 ζ 2 , ∀ t ∈ R .

Fact 2.3. For a Gaussian θ ∼ N (0 , σ I ) , θ is SG( σ ) and nSG( σ d )

2 d √ .

Lemma 2.4 (Hoeffding type inequality for norm-subGaussian, [30]) . Let x 1 , · · · , x k ∈ R d be random vectors, and for each i ∈ [ k ] , x i | F i -1 is zero-mean nSG( ζ i ) where F i is the corresponding filtration. Then there exists an absolute constant c such that for any δ > 0 , with probability at least

3 Convergence to Stationary Points: Framework

We present the optimization framework for finding SOSP in this section. It's important to emphasize that this framework is dedicated exclusively to optimization concerns, with privacy considerations being outside of its purview. The results about SOSP throughout the paper follows the assumptions of [46].

Assumption 3.1. Any function drawn from P is G -Lipschitz, ρ -Hessian Lipschitz, and M -smooth, almost surely, and the risk is upper bounded by B .

As discussed before, we define two different kinds of gradient oracles, one for estimating the gradient at one point and the other for estimating the gradient difference at two points.

For a G -Lipschitz and M -smooth function F :

We say O 1 is a first kind of ζ 1 norm-subGaussian Gradient oracle if given x ∈ R , O ( x ) satisfies E O 1 ( x ) = ∇ F ( x ) and O 1 ( x ) -∇ F ( x ) is nSG( ζ 1 ) .

Definition 3.2 (SubGaussian gradient oracles) . d

We say O 2 is a second kind of ζ 2 norm-subGaussian stochastic Gradient oracle if given x, y ∈ R d , O 2 ( x, y ) satisfies that E O 2 ( x, y ) = ∇ F ( x ) -∇ F ( y ) and O 2 ( x, y ) -( ∇ F ( x ) -∇ F ( y )) is nSG( ζ 2 ‖ x -y ‖ ) .

Note that we should assume M ≥ √ ρα to make finding a second-order stationary point strictly more challenging than finding a first-order stationary point. We use smin ( · ) to denote the smallest eigenvalue of a matrix.

Algorithm 1 Stochastic Spider

Inspired by [2] who adapted the SpiderBoost algorithm for finding private FOSPs, we give a framework based on the SpiderBoost in Algorithm 1. Our analysis of Algorithm 1 hinges on three key properties we establish in this section: ( i ) ∇ t remains consistently close to the true gradient ∇ F ( x t ) with high probability; ( ii ) the algorithm is capable of escaping the saddle point with high probability, and ( iii ) a large drift implies significant decrease in the function value, which enables us to limit the number of queries to the more accurate but costlier first kind of gradient oracle O 1 .

Lemma 3.3. For any 0 ≤ t ≤ T and letting τ t ≤ t be the largest integer such that drift τ t is set to be 0, with probability at least 1 -ω/T , for some universal constant C > 0 , we have

Hence with probability at least 1 -ω , we know for each t ≤ T , ‖∇ t -∇ F ( x t ) ‖ 2 ≤ γ 2 / 16 , where γ 2 := 16 C ( ζ 2 2 κ +4 ζ 2 1 ) · log( Td/ω ) and κ is a parameter we can choose in the algorithm.

As shown in Lemma 3.3, the error on the gradient estimation for each step is bounded with high probability. Then by adding the Gaussian noise in Line 8, we can show the algorithm can escape the saddle point efficiently based on previous results.

Lemma 3.4 (Essentially from [46]) . Under Assumption 3.1, run SGD iterations x t +1 = x t -η ∇ t , with step size η = 1 /M . Suppose x 0 is a saddle point satisfying ‖∇ F ( x 0 ) ‖ ≤ α and smin ( ∇ 2 F ( x 0 )) ≤ - √ ρα , α = γ log 3 ( dBM/ρω ) . If ∇ 0 = ∇ F ( x 0 ) + ζ 1 + ζ 2 where ‖ ζ 1 ‖ ≤ γ , ζ 2 ∼ N (0 , γ 2 d log( d/ω ) I d ) , and ‖∇ t - ∇ F ( x t ) ‖ ≤ γ for all t ∈ [Γ] , with probability at least 1 -ω · log(1 /ω ) , one has F ( x Γ ) -F ( x 0 ) ≤ -Ω ( γ 3 / 2 √ ρ log 3 ( dMB ) ) , where Γ = M log( dMB ργω ) √ ργ .

We discuss this lemma in the Appendix in more details. The next lemma is standard, showing how large the function values can decrease in each step.

Lemma3.5. By setting η = 1 /M , we have F ( x t +1 ) ≤ F ( x t )+ η ‖∇ t ‖·‖∇ F ( x t ) -∇ t ‖-η 2 ‖∇ t ‖ 2 . Moreover, with probability at least 1 -ω , for each t ≤ T such that ‖∇ F ( x t ) ‖ ≥ γ , we have

With the algorithm designed to control the drift term, the guarantee for Stochastic Spider to find the second order stationary point is stated below:

Lemma 3.6. Suppose O 1 and O 2 are ζ 1 and ζ 2 norm-subGaussian respectively. If one sets γ = O (1) √ ( ζ 2 2 κ +4 ζ 2 1 ) · log( Td/ω ) , with probability at least 1 -ω , at least one point in the output set { x 1 , · · · , x T } of Algorithm 1 is α -SOSP, where

As mentioned before, we can bound the number of occurrences where the drift gets large and hence bound the total time we query the oracle of the first kind.

Lemma 3.7. Under the event that ‖∇ t -∇ F ( x t ) ‖ ≤ γ/ 4 for all t ∈ [ T ] and our parameter settings, letting K = { t ∈ [ T ] : drift t ≥ κ } be the set of iterations where the drift is large, we know | K | ≤ O ( Bη κ + Tγ 2 η 2 /κ ) = O ( Bη log 4 ( dMB ργω ) /κ ) .

4 Private SOSP

We adopt the framework before and get our main results on finding SOSP privately by constructing private gradient oracles in this section. Finding SOSP for empirical risk function F D and for population risk function F P are discussed in Subsection 4.1 and Subsection 4.2 respectively.

4.1 Convergence to the SOSP of the Empirical Risk

We use Stochastic Spider to improve the convergence to α -SOSP of the empirical risk, and aim at getting α = ˜ O ( d 1 / 3 /n 2 / 3 ) . We use the full-batch size for simplicity, and use the gradient oracles

where g 1 ∼ N (0 , σ 2 1 I d ) and g 2 ∼ N (0 , σ 2 2 ‖ x -y ‖ 2 2 I d ) are added to ensure privacy by Gaussian mechanism (in Appendix).

Before stating the formal results, note that by Lemma 3.6, the framework can only guarantee the existence of an α -SOSP in the outputted set. In order to find the SOSP privately from the set, we adopt the well-known AboveThreshold algorithm, whose pseudo-code can be found in Algorithm 2 in the Appendix. Algorithm 2 is a slight modification of the well-known AboveThreshold algorithm in [20], and we get the following guarantee immediately.

Lemma 4.1. Algorithm 2 is ( ε, 0) -DP. Given the point set { x 1 , · · · , x T } and S of size n as the input, ( i ) if it outputs any point x i , then with probability at least 1 -ω , we know

( ii ) if there exists a α -SOSP point x ∈ { x i } i ∈ [ T ] , then with probability at least 1 -ω , Algorithm 2 will output one point.

Choosing the appropriate noise scales for the Gaussian added in Equation (2) and running Algorithm 1 can get a private set of points which contains at least one good SOSP. Then we can run Algorithm 2 to find the good SOSP in the set privately. The formal guarantee is stated below:

Theorem 4.2 (Empirical) . For ϕ = O (1) , use Equation (2) as gradient oracles with κ = G 4 / 3 B 1 / 3 M 5 / 3 ( √ d n √ ϕ ) 2 / 3 , σ 1 = G √ Bη/κ log 2 ( dMB/ω ) n √ ϕ , σ 2 = M √ BM/α 2 1 log 5 ( dMB/ω ) n √ ϕ . Running Algorithm 1, outputting the set { x i } i ∈ [ T ] if the total time to query O 1 is bounded by O ( Bη log 4 ( dMB ργω ) /κ ) , otherwise outputting a set of T arbitrary points is ( ϕ/ 2) -zCDP. With probability at least 1 -ω , at least one point in the output set is α 1 -SOSP of F D with

Moreover, if we run Algorithm 2 with inputs { x i } i ∈ [ T ] , D , B, M, G, ρ, α 1 , and ε = √ ϕ , with probability at least 1 -ω , we can get an α 2 -SOSP of F D with α 2 = O ( α 1 + G log( n/Gω ) n √ ϕ + M log( dBGM/ρω ) n √ ϕ √ ρ √ α 1 ) . The whole procedure is ϕ -zCDP.

For more generality, we state the above theorem in terms of zCDP. ϕ -zCDP implies ( ϕ + 2 √ ϕ log(1 /δ ) , δ ) -DP, which gives the bounds in Table 1 (see Appendix for details).

Remark 4.3. It's worth noting that the cost of gradient computation can be reduced by utilizing smaller batch sizes. However, our work does not focus on optimizing this aspect.

4.2 Convergence to the SOSP of the Population Risk

This subsection aims at getting an α -SOSP for F P (the population function). Differing from the stochastic oracles used for empirical function F D , we do not use full batch in the oracle. As an alternative, we draw fresh samples from D without replacement with a smaller batch size:

where S 1 and S 2 are sets of size of b 1 and b 2 respectively drawn from D without replacement , g 1 ∼ N (0 , σ 2 1 I d ) and g 2 ∼ N (0 , σ 2 2 ‖ x -y ‖ 2 2 · I d ) are added for privacy guarantee. These gradient oracles satisfy the following.

Claim 4.4. The gradient oracles O 1 and O 2 constructed in Equation (3) are a first kind of O ( L √ log d √ b 1 + √ dσ 1 ) norm-subGaussian gradient oracle and second kind of O ( M √ log d √ b 2 + √ dσ 2 ) norm-subGaussian gradient oracle respectively.

Recall that in the empirical case, we use Algorithm 2 to choose the SOSP for F D . But in the population case, we need to find SOSP for F P , and what we have are samples from P . We need the following technical results to help us find the SOSP from the set, which follows from Hoeffding inequality for norm-subGaussians (Lemma 2.4) and Matrix Bernstein inequality (in the Appendix).

Lemma 4.5. Fix a point x ∈ R d . Given a set S of m samples drawn i.i.d. from the distribution P , then we know with probability at least 1 -ω , we have

By choosing the appropriate noise scales σ 1 and σ 2 to ensure the privacy guarantee, we can bound the population bound similar to the empirical bound with these tools.

Theorem 4.6 (Population) . Divide the dataset D into two disjoint datasets D 1 and D 2 of size glyph[ceilingleft] n/ 2 glyph[ceilingright] and glyph[floorleft] n/ 2 glyph[floorright] respectively. Set b 1 = nκ Bη , b 2 = nα 2 1 BM , σ 1 = G b 1 √ ϕ , σ 2 = M b 2 √ ϕ and κ = max( G 4 / 3 B 1 / 3 log 1 / 3 d M 5 / 3 n -1 / 3 , ( GB 2 / 3 M 5 / 3 ) 6 / 7 ( √ d n √ ϕ ) 4 / 7 ) in Equation (3) and use them as gradient oracles. Running Algorithm 1 with D 1 , and outputting the set { x i } i ∈ [ T ] if the total time to query O 1 is bounded by O ( Bη log 4 ( dMB ργω ) /κ ) , otherwise outputting a set of T arbitrary points, is ( ϕ/ 2) -zCDP, and with probability at least 1 -ω , at least one point in the output is α 1 -SOSP of F P with

Moreover, if we run Algorithm 2 with inputs { x i } i ∈ [ T ] , D 2 , B, M, G, ρ, α 1 , ε = √ ϕ with probability at least 1 -ω , Algorithm 2 can output an α 2 -SOSP of F P with α 2 = O ( α 1 + M log( ndBGM/ρω ) √ ρ min( n √ ϕ,n 1 / 2 ) √ α 1 + G ( log( n/Gω ) n √ ϕ + log( d/ω ) √ n ) ) . The whole procedure is ϕ -zCDP.

5 Bounding the Private Excess Risk

In this section, we shift our focus from 'second-order' guarantees to 'zeroth-order' guarantees, and consider the problem of getting good private risk bounds without convexity.

5.1 Polynomial Time Approach

If we want the algorithm to be efficient and implementable in polynomial time, to our knowledge the only known bound is O ( d log(1 /δ ) ε 2 log n ) in [46] for smooth functions. [46] used Gradient Langevin Dynamics, a popular variant of SGD to solve this problem, and prove the privacy by advanced composition [31]. We generalize the exponential mechanism to the non-convex case and implement it without a smoothness assumption.

First recall the Log-Sobolev inequality: We say a probability distribution π satisfies LSI with constant C LSI if for all f : R d → R , E π [ f 2 log f 2 ] -E π [ f 2 ] log E π [ f 2 ] ≤ 2 C LSI E π ‖∇ f ‖ 2 2 . A well-known result ([40]) says if f is µ -strongly convex, then the distribution proportional to exp( -f ) satisfies LSI with constant 1 /µ . Recall the results from previous results [39] about LSI and DP:

Theorem 5.1 ([39]) . Sampling from exp( -βF ( x ; D ) -r ( x )) for some public regularizer r is ( ε, δ ) -DP, where ε ≤ 2 Gβ n √ C LSI √ 1 + 2 log(1 /δ ) , and C LSI is the worst LSI constant.

We can apply the classic perturbation lemma to get the new LSI constant in the non-convex case. Suppose we add a regularizer µ 2 ‖ x ‖ 2 , and try to sample from exp( -β ( F ( x ; D ) + µ 2 ‖ x ‖ 2 )) .

Lemma 5.2 (Stroock perturbation) . Suppose π satisfies LSI with constant C LSI ( π ) . If 0 < c ≤ d π ′ d π ≤ C , then C LSI ( π ′ ) ≤ C c C LSI ( π ) .

Lemma 5.3 is a more general version of Theorem 3.4 in [23] and can be used to bound the empirical risk.

Lemma 5.3. Let π ( x ) ∝ exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 2 )) . Then for βGD > d , we know

We now turn to bound the generalization error, and use the notion of uniform stability:

Lemma 5.4 (Stability and Generalization [10]) . Given a dataset D = { s i } i ∈ [ n ] drawn i.i.d. from some underlying distribution P , and given any algorithm A , suppose we randomly replace a sample s in D by an independent fresh one s ′ from P and get the neighoring dataset D ′ , then E D , A [ F P ( A ( D )) -F D ( A ( D ))] = E D ,s ′ , A [ f ( A ( D ); s ′ )) -f ( A ( D ′ ); s ′ ))] , where A ( D ) is the output of A with input D .

As each function f (; s ′ ) is G -Lipschitz, it suffices to bound the W 2 distance of A ( D ) and A ( D ′ ) . If A is sampling from the exponential mechanism, letting π D ∝ exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 )) and π D ′ ∝ exp( -β ( F D ′ ( x ) + µ 2 ‖ x ‖ 2 )) , it suffices to bound the W 2 distance between π D and π D ′ . The following lemma can bound the generalization risk of the exponential mechanism under LSI:

Lemma 5.5 (Generalization error bound) . Let π D ∝ exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 2 )) . Then we have E D ,x ∼ π D [ F P ( x ) -F D ( x )] ≤ O ( G 2 exp( βGD ) ) .

We get the following results:

Theorem 5.6 (Risk bound) . We are given ε, δ ∈ (0 , 1 / 2) . Sampling from exp( -β ( F D ( x )+ 2 ‖ x ‖ 2 )) with β = O ( ε log( nd ) GD √ log(1 /δ )) ) , µ = d D 2 β is ( ε, δ ) -DP. The empirical risk and population risk are

µ 2 d · log log( n ) √ log(1 /δ )

Implementation There are multiple existing algorithms that can sample efficiently from density with LSI, under mild assumptions. For example, when the functions are smooth or weakly smooth, one can turn to the Langevin Monte Carlo [16], and [36]. The algorithm in [46] also requires mild smoothness assumptions. We discuss the implementation of non-smooth functions in bit more details, which is more challenging.

We can adopt the rejection sampler in [26], which is based on the alternating sampling algorithm in [35]. Both [35] and [26] are written in the language of log-concave and strongly log-concave densities, but their results hold as long as LSI holds. By combining them together, we can get the following risk bounds. The details of the implementation can be found in Appendix D.3.

Theorem 5.7 (Implementation, risk bound) . For ε, δ ∈ (0 , 1 / 2) , there is an ( ε, 2 δ ) -DP efficient sampler that can achieve the empirical and population risks O ( GD d · log log( n ) √ log(1 /δ ) ε log( nd ) ) . Moreover, in expectation, the sampler takes ˜ O ( nε 3 log 3 ( d ) √ log(1 /δ ) / ( GD ) ) function values query and some Gaussian random variables restricted to the convex set K in total.

5.2 Exponential Time Approach

In [23], it is shown that sampling from exp( -εn GD F D ( x )) is ε -DP, and a nearly tight empirical risk bound of ˜ O ( DGd nε ) is achieved for convex functions. It is open what is the bound we can get for non-convex DP-SO.

Upper Bound Given exponential time we can use a discrete exponential mechanism as considered in [9]. We recap the argument and extend it to DP-SO. The proof is based on a simple packing argument, and can be found in the Appendix.

Theorem 5.8. There exists an ε -DP differentially private algorithm that achieves a population risk of O ( GD ( d log( εn/d ) / ( εn ) + √ d log( εn/d ) / ( √ n ) )) .

Lower Bound Results in [23] imply that the first term of ˜ O ( GDd/εn ) is tight, even if we relax to approximate DP with δ > 0 . A reduction from private selection problem shows the ˜ O ( √ d/n ) generalization term is also nearly-tight (Theorem 5.11). In the selection problem, we have k coins, each with an unknown probability p i . Each coin is flipped n times such that { x i,j } j ∈ [ n ] , each x i,j i.i.d. sampled from Bern( p i ) , and we want to choose a coin i with the smallest p i . The risk of choosing i is p i -min i ∗ p i ∗ .

Theorem 5.9. Any algorithm for the selection problem has excess population risk ˜ Ω( √ log k n ) .

This follows from a folklore result on the selection problem (see e.g. [5]). We can combine this with the following reduction from selection to non-convex optimization:

Theorem 5.10 (Restatement of results in [23]) . If any ( ε, δ ) -DP algorithm for selection has risk R ( k ) , where R ( k ) is a function with k as variables, then any ( ε, δ ) -DP algorithm for minimizing 1-Lipschitz losses over B d (0 , 1) (the d -dimensional unit ball) has risk R (2 Θ( d ) ) .

From this and the aforementioned lower bounds in empirical non-convex optimization we get the following:

Theorem 5.11. For ε ≤ 1 , δ ∈ [2 -Ω( n ) , 1 /n 1+Ω(1) ] , any ( ε, δ ) -DP algorithm for minimizing 1 -Lipschitz losses over B d (0 , 1) has excess population risk max { Ω( d log(1 /δ ) / ( εn )) , ˜ Ω( √ d/n ) } .

6 Discussion

In this paper, we gave improved bounds for finding SOSPs under differential privacy, as well as points with low population risk. We discuss some potential follow-up questions here.

First, there is still a gap between our upper bounds and the lower bounds of [2]. Closing this gap is an interesting question and may lead to novel technical insights about private optimization. To obtain upper bounds for finding SOSP, both our work and the work of [2] uses the oracle of the second kind defined in Definition 3.2. Second, in typical private optimization work, only oracles of the first kind are used. It is possible that the oracle of the second kind is useful in private optimization problems besides finding stationary points, either in theory or practice. In particular, since this oracle has lower sensitivity, it may allow us to tolerate a higher noise level / lower privacy budget across multiple iterations. Third, we privatize the SpiderBoost in the work, and there are other versions of variance-reduced algorithms like Spiker and SARAH. It is interesting if our ideas can be used to privatize those algorithms, and compare their practical performance. Lastly, our polynomial-time excess population risk bounds have a O (1 / log n ) dependence on the dataset size, whereas for convex losses standard results have a max { 1 / √ n, 1 /εn } dependence. The stronger dependence is achievable under LSI, but the practical settings in which LSI holds without convexity holding seems limited. It remains an open question to find a practical assumption weaker than convexity that allows us to achieve better dependence on n .

References

Naman Agarwal, Zeyuan Allen-Zhu, Brian Bullins, Elad Hazan, and Tengyu Ma. Finding approximate local minima faster than gradient descent. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing , pages 1195-1199, 2017.

Raman Arora, Raef Bassily, Tomás González, Cristóbal Guzmán, Michael Menart, and Enayat Ullah. Faster rates of convergence to stationary points in differentially private optimization. arXiv preprint arXiv:2206.00846 , 2022.

Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in l1 geometry. In International Conference on Machine Learning , pages 393-403. PMLR, 2021.

Hilal Asi, Daniel Asher Nathan Levy, and John Duchi. Adapting to function difficulty and growth conditions in private optimization. In Advances in Neural Information Processing Systems , 2021.

Mitali Bafna and Jonathan Ullman. The price of selection in differential privacy. In Satyen Kale and Ohad Shamir, editors, Proceedings of the 2017 Conference on Learning Theory , volume 65 of Proceedings of Machine Learning Research , pages 151-168. PMLR, 07-10 Jul 2017.

Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses. arXiv preprint arXiv:2006.06914 , 2020.

Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems , pages 11279-11288, 2019.

Raef Bassily, Cristóbal Guzmán, and Michael Menart. Differentially private stochastic optimization: New results in convex and non-convex settings. Advances in Neural Information Processing Systems , 34:9317-9329, 2021.

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In Proc. of the 2014 IEEE 55th Annual Symp. on Foundations of Computer Science (FOCS) , pages 464-473, 2014.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Martin Hirt and Adam Smith, editors, Theory of Cryptography , pages 635-658, Berlin, Heidelberg, 2016. Springer Berlin Heidelberg.

Yair Carmon, Arun Jambulapati, Yujia Jin, Yin Tat Lee, Daogao Liu, Aaron Sidford, and Kevin Tian. Resqueing parallel and private stochastic convex optimization. arXiv preprint arXiv:2301.00457 , 2023.

Kamalika Chaudhuri and Claire Monteleoni. Privacy-preserving logistic regression. Advances in neural information processing systems , 21, 2008.

Kamalika Chaudhuri, Claire Monteleoni, and Anand D Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(Mar):1069-1109, 2011.

Yongxin Chen, Sinho Chewi, Adil Salim, and Andre Wibisono. Improved analysis for a proximal algorithm for sampling. In Conference on Learning Theory , pages 2984-3014. PMLR, 2022.

Sinho Chewi, Murat A Erdogdu, Mufan Li, Ruoqi Shen, and Shunshi Zhang. Analysis of langevin monte carlo from poincare to log-sobolev. In Conference on Learning Theory , pages 1-2. PMLR, 2022.

Rishav Chourasia, Jiayuan Ye, and Reza Shokri. Differential privacy dynamics of langevin diffusion and noisy gradient descent. In Advances in Neural Information Processing Systems , 2021.

Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC) , pages 265-284, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends R © in Theoretical Computer Science , 9(3-4):211-407, 2014.

Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal nonconvex optimization via stochastic path-integrated differential estimator. Advances in Neural Information Processing Systems , 31, 2018.

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in linear time. In Proc. of the Fifty-Second ACM Symp. on Theory of Computing (STOC'20) , 2020.

Arun Ganesh, Abhradeep Thakurta, and Jalaj Upadhyay. Langevin diffusion: An almost universal algorithm for private euclidean (convex) optimization. arXiv preprint arXiv:2204.01585 , 2022.

Changyu Gao and Stephen J Wright. Differentially private optimization for smooth nonconvex erm. arXiv preprint arXiv:2302.04972 , 2023.

Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. In Conference on learning theory , pages 797-842. PMLR, 2015.

Sivakanth Gopi, Yin Tat Lee, and Daogao Liu. Private convex optimization via exponential mechanism. In Conference on Learning Theory , pages 1948-1989. PMLR, 2022.

Sivakanth Gopi, Yin Tat Lee, Daogao Liu, Ruoqi Shen, and Kevin Tian. Private convex optimization in general norms. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 5068-5089. SIAM, 2023.

Roger Iyengar, Joseph P Near, Dawn Song, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Towards practical differentially private convex optimization. In 2019 IEEE Symposium on Security and Privacy (SP) , 2019.

Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In International conference on machine learning , pages 1724-1732. PMLR, 2017.

Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736 , 2019.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1376-1385, Lille, France, 07-09 Jul 2015. PMLR.

Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. In International Conference on Machine Learning , pages 10633-10660. PMLR, 2022.

Daniel Kifer, Adam Smith, and Abhradeep Thakurta. Private convex empirical risk minimization and high-dimensional regression. In Conference on Learning Theory , pages 25-1, 2012.

Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth erm and sco in subquadratic steps. Advances in Neural Information Processing Systems , 34, 2021.

Yin Tat Lee, Ruoqi Shen, and Kevin Tian. Structured logconcave sampling with a restricted gaussian oracle. In Conference on Learning Theory , pages 2993-3050. PMLR, 2021.

Jiaming Liang and Yongxin Chen. A proximal algorithm for sampling from non-smooth potentials. In 2022 Winter Simulation Conference (WSC) , pages 3229-3240. IEEE, 2022.

Songtao Lu, Meisam Razaviyayn, Bo Yang, Kejun Huang, and Mingyi Hong. Finding secondorder stationary points efficiently in smooth nonconvex linearly constrained optimization problems. Advances in Neural Information Processing Systems , 33:2811-2822, 2020.

Frank McSherry and Kunal Talwar. Mechanism design via differential privacy. In 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07) , pages 94-103. IEEE, 2007.

Kentaro Minami, HItomi Arai, Issei Sato, and Hiroshi Nakagawa. Differential privacy without sensitivity. Advances in Neural Information Processing Systems , 29, 2016.

Felix Otto and Cédric Villani. Generalization of an inequality by talagrand and links with the logarithmic sobolev inequality. Journal of Functional Analysis , 173(2):361-400, 2000.

Shuang Song, Kamalika Chaudhuri, and Anand D Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing , pages 245-248. IEEE, 2013.

Shuang Song, Thomas Steinke, Om Thakkar, and Abhradeep Thakurta. Evading the curse of dimensionality in unconstrained private glms. In International Conference on Artificial Intelligence and Statistics , pages 2638-2646. PMLR, 2021.

Kunal Talwar, Abhradeep Guha Thakurta, and Li Zhang. Nearly optimal private lasso. Advances in Neural Information Processing Systems , 28, 2015.

Hoang Tran and Ashok Cutkosky. Momentum aggregation for private non-convex erm. In Advances in Neural Information Processing Systems , 2022.

Joel A Tropp. An introduction to matrix concentration inequalities. Foundations and Trends R © in Machine Learning , 8(1-2):1-230, 2015.

Di Wang, Changyou Chen, and Jinhui Xu. Differentially private empirical risk minimization with non-convex loss functions. In International Conference on Machine Learning , pages 6526-6535. PMLR, 2019.

Di Wang and Jinhui Xu. Differentially private empirical risk minimization with smooth nonconvex loss functions: A non-stationary view. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 1182-1189, 2019.

Di Wang and Jinhui Xu. Escaping saddle points of empirical risk privately and scalably via dp-trust region method. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 90-106. Springer, 2020.

Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems , 30, 2017.

Lingxiao Wang, Bargav Jayaraman, David Evans, and Quanquan Gu. Efficient privacypreserving stochastic nonconvex optimization. arXiv preprint arXiv:1910.13659 , 2019.

Yongqiang Wang and Tamer Ba¸ sar. Decentralized nonconvex optimization with guaranteed privacy and accuracy. Automatica , 150:110858, 2023.

Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. Advances in Neural Information Processing Systems , 32, 2019.

Yi Xu, Rong Jin, and Tianbao Yang. First-order stochastic algorithms for escaping from saddle points in almost linear time. Advances in neural information processing systems , 31, 2018.

Xiaodong Yang, Huishuai Zhang, Wei Chen, and Tie-Yan Liu. Normalized/clipped sgd with perturbation for differentially private non-convex optimization. arXiv preprint arXiv:2206.13033 , 2022.

Qiuchen Zhang, Jing Ma, Jian Lou, and Li Xiong. Private stochastic non-convex optimization with improved utility rates. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence , 2021.

Yingxue Zhou, Xiangyi Chen, Mingyi Hong, Zhiwei Steven Wu, and Arindam Banerjee. Private stochastic non-convex optimization: Adaptive algorithms and tighter generalization bounds. arXiv preprint arXiv:2006.13501 , 2020.

A Other Preliminary

Definition A.1 (Laplace distribution) . We say X ∼ Lap( b ) if X has density f ( X = x ) = 1 2 b exp( -| x | b ) .

In the analysis we use zero-concentrated differential privacy (zCDP):

Definition A.2 (zCDP) . A mechanism M is ϕ -zCDP if for all α > 1 and neighboring D , D ′ :

where R α denotes the α -Rényi divergence.

Theorem A.3 (zCDP composition, [11]) . The (adaptive) composition of k mechanisms that satisfy ϕ 1 -zCDP, ϕ 2 -zCDP, . . . , ϕ k -zCDP respectively is ( ϕ 1 + ϕ 2 + . . . + ϕ k ) -zCDP.

Lemma A.4. [Conversion between ( ε, δ ) -DP and zCDP, [11]] A mechanism that is ( ε, 0) -DP is also ε 2 / 2 -zCDP. A mechanism that is ϕ -zCDP is also ( ε, δ ) -DP for any δ > 0 and ε = ϕ +2 √ ϕ log(1 /δ ) .

Theorem A.5 (zCDP of Gaussian Mechanism, [11]) . Given a randomized algorithm A : P ∗ → R d , let ∆ 2 f = max neighboring D , D ′ ‖A ( D ) - A ( D ′ ) ‖ 2 , then releasing A ( D ) plus noise N (0 , σ 2 ) is (∆ 2 f ) 2 2 σ 2 -zCDP.

Theorem A.6 (Matrix Bernstein inequality, [45]) . Consider a sequence { X i } i ∈ m of independent, mean-zero, symmetric d × d random matrices. If for each matrix X i , we know ‖ X i ‖ op ≤ M , then for all t ≥ 0 , we have Pr [ ‖ ∑ i ∈ [ m ] X i ‖ op ≥ t ] ≤ d exp ( -t 2 2( σ 2 + Mt/ 3) ) , where σ 2 = ‖ ∑ i ∈ [ m ] E X 2 i ‖ op .

B Omitted Proof of Section 3

B.1 Proof of Lemma 3.3

Lemma 3.3. For any 0 ≤ t ≤ T and letting τ t ≤ t be the largest integer such that drift τ t is set to be 0, with probability at least 1 -ω/T , for some universal constant C > 0 , we have

Hence with probability at least 1 -ω , we know for each t ≤ T , ‖∇ t -∇ F ( x t ) ‖ 2 ≤ γ 2 / 16 , where γ 2 := 16 C ( ζ 2 2 κ +4 ζ 2 1 ) · log( Td/ω ) and κ is a parameter we can choose in the algorithm.

Proof. If drift τ t = 0 happens, we use the first kind oracle to query the gradient, and hence ∇ τ t -∇ F ( x τ t ) is zero-mean and nSG(2 ζ 1 ) . If t = τ t , Equation (1) holds by the property of normsubGaussian.

For each τ t +1 ≤ i ≤ t , conditional on ∇ i -1 , we know ∆ i -( ∇ F ( x i ) -F ( x i -1 )) is zero-mean and nSG( ζ 2 ‖ x i -x i -1 ‖ ) . Note that

Equation (1) follows from Lemma 2.4.

We know drift t -1 = ∑ t i = τ t +1 ‖ x i -x i -1 ‖ 2 ≤ κ almost surely by the design of the algorithm. By union bound, we know with probability at least 1 -ω , for each t ∈ [ T ] ,

B.2 Discussion of Lemma 3.4

Lemma 3.4 (Essentially from [46]) . Under Assumption 3.1, run SGD iterations x t +1 = x t -η ∇ t , with step size η = 1 /M . Suppose x 0 is a saddle point satisfying ‖∇ F ( x 0 ) ‖ ≤ α and smin ( ∇ 2 F ( x 0 )) ≤ - √ ρα , α = γ log 3 ( dBM/ρω ) . If ∇ 0 = ∇ F ( x 0 ) + ζ 1 + ζ 2 where ‖ ζ 1 ‖ ≤ γ , ζ 2 ∼ N (0 , γ 2 d log( d/ω ) I d ) , and ‖∇ t - ∇ F ( x t ) ‖ ≤ γ for all t ∈ [Γ] , with probability at least 1 -ω · log(1 /ω ) , one has F ( x Γ ) -F ( x 0 ) ≤ -Ω ( γ 3 / 2 √ ρ log 3 ( dMB ργω ) ) , where Γ = M log( dMB ργω ) √ ργ .

We briefly recap the proof of Lemma 3.4 in [46]. One observation between the decreased function value, and the distance solutions moved is stated below:

Lemma B.1 (Lemma 11, [46]) . For each t ∈ [Γ] , we know

The difference between our algorithm and the DP-GD in [46] is the noise on the gradient. Note that with high probability, ∑ i ∈ [Γ] ‖∇ i -∇ F ( x t ) ‖ 2 2 in our algorithm is controlled and small, and hence does not change the other proofs in [46]. Hence if F ( x 0 ) -F ( x Γ ) is small, i.e., the function value does not decrease significantly, we know x t is close to x 0 .

Let B x ( r ) be the unit ball of radius r around point x . Denote the ( x ) Γ the point x Γ after running SGD mentioned in Lemma 3.4 for Γ steps beginning at x . With this observation, denote B γ ( x 0 ) := { x | x ∈ B x 0 ( ηα ) , Pr[ F (( x ) Γ ) -F ( x ) ≥ -Φ] ≥ ω } . [46] demonstrates the following lemma:

Lemma B.2. If ‖∇ F ( x 0 ) ‖ ≤ α and smin ( ∇ 2 F ( x 0 )) ≤ - √ ργ , then the width of B γ ( x 0 ) along the along the minimum eigenvector of ∇ 2 F ( x 0 ) is at most ωηγ log(1 /ω ) √ 2 π d .

The intuition is that if two different points x 1 , x 2 ∈ B x 0 ( ηα ) , and x 1 -x 2 is large along the minimum eigenvector, then with high probability, the distance between ‖ ( x 1 ) Γ -( x 2 ) Γ ‖ will be large, and either ‖ ( x 1 ) Γ -x 1 ‖ or ‖ ( x 2 ) Γ -x 2 ‖ is large, and hence either F ( x 1 ) -F (( x 1 ) Γ ) or F ( x 2 ) -F (( x 2 ) Γ ) is large. The Lemma 3.4 follows from Lemma B.2 by using the Gaussian ζ 2 to kick off the point.

B.3 Proof of Lemma 3.5

Lemma3.5. By setting η = 1 /M , we have F ( x t +1 ) ≤ F ( x t )+ η ‖∇ t ‖·‖∇ F ( x t ) -∇ t ‖-η 2 ‖∇ t ‖ 2 . Moreover, with probability at least 1 -ω , for each t ≤ T such that ‖∇ F ( x t ) ‖ ≥ γ , we have

Proof. By the assumption on smoothness, we know

By Lemma 3.3, with probability at least 1 -ω , for each t ∈ [ T ] we have ‖∇ F ( x t ) -∇ t ‖ 2 ≤ γ/ 4 . Hence we know if ∇ F ( x t ) ≥ γ , we have

B.4 Proof of Lemma 3.6

Lemma 3.6. Suppose O 1 and O 2 are ζ 1 and ζ 2 norm-subGaussian respectively. If one sets γ = O (1) √ ( ζ 2 2 κ +4 ζ 2 1 ) · log( Td/ω ) , with probability at least 1 -ω , at least one point in the output set

{ x 1 , · · · , x T } of Algorithm 1 is α -SOSP, where

Proof. By Lemma 3.5, we know if the gradient ‖∇ F ( x t ) ‖ ≥ γ , then with high probability that F ( x t +1 ) -F ( x t ) ≤ -ηγ 2 / 6 . By Lemma 3.4, if x t is a saddle point (with small gradient norm but the Hessian has a small eigenvalue), then with high probability that F ( x Γ+ t ) -F ( x t ) ≤ -Ω( γ 3 / 2 √ ρ log 3 ( dMB ργω ) ) , and the function values decrease Ω ( γ 2 M log 4 ( dMB ργω ) ) on average for each step.

Recall the assumption that the risk is upper bounded by B , by our setting T = Ω ( BM γ 2 log 4 ( dMB ργω ) ) , the statement is proved.

B.5 Proof of Lemma 3.7

Lemma 3.7. Under the event that ‖∇ t -∇ F ( x t ) ‖ ≤ γ/ 4 for all t ∈ [ T ] and our parameter settings, letting K = { t ∈ [ T ] : drift t ≥ κ } be the set of iterations where the drift is large, we know | K | ≤ O ( Bη κ + Tγ 2 η 2 /κ ) = O ( Bη log 4 ( dMB ργω ) /κ ) .

Proof. By Lemma 3.5, if ‖ F ( x t ) ‖ ≥ γ , we know F ( x t +1 ) -F ( x t ) ≤ -η ‖∇ t ‖ 2 / 6 , and F ( x t +1 ) -F ( x t ) ≤ ηγ 2 otherwise. Index the items in K = { t 1 , t 2 , · · · , t | K | } such that t i < t i +1 . We know

Recall by the assumption that max y F ( y ) -min x F ( x ) ≤ B . And hence -B ≤ F ( x t | L | ) -F ( x t 1 ) ≤ -| K | 6 η κ + Tγ 2 η , and we know

C Appendix for Section 4

The pseudocode of Algorithm 2 is stated below:

Algorithm 2 AboveThreshold

1: Input: Aset of points { x i } T i =1 , dataset S , parameters of objective function B,M,G,ρ , objective error α

3: for i = 1 , · · · , T do

2: Set ̂ T 1 = α +Lap( 4 G nε ) + 16 log(2 T/ω ) G nε , ̂ T 2 = - √ ρα +Lap( 4 M nε ) -16 log(2 T/ω ) M nε

4: if ‖∇ F S ( x i ) ‖ +Lap( 8 G nε ) ≤ ̂ T 1 ∧ smin ( ∇ 2 F S ( x i )) + Lap( 8 M nε ) ≥ ̂ T 2 then

5: Output: x i

6: Halt

7: end if

8: end for

C.1 Proof of Theorem 4.2

Theorem 4.2 (Empirical) . For ϕ = O (1) , use Equation (2) as gradient oracles with κ = G 4 / 3 B 1 / 3 M 5 / 3 ( √ d n √ ϕ ) 2 / 3 , σ 1 = G √ Bη/κ log 2 ( dMB/ω ) n √ ϕ , σ 2 = M √ BM/α 2 1 log 5 ( dMB/ω ) n √ ϕ . Running Algorithm 1, outputting the set { x i } i ∈ [ T ] if the total time to query O 1 is bounded by O ( Bη log 4 ( dMB ργω ) /κ ) ,

otherwise outputting a set of T arbitrary points is ( ϕ/ 2) -zCDP. With probability at least 1 -ω , at least one point in the output set is α 1 -SOSP of F D with

Moreover, if we run Algorithm 2 with inputs { x i } i ∈ [ T ] , D , B, M, G, ρ, α 1 , and ε = √ ϕ , with probability at least 1 -ω , we can get an α 2 -SOSP of F D with α 2 = O ( α 1 + G log( n/Gω ) n √ ϕ + M log( dBGM/ρω ) n √ ϕ √ ρ √ α 1 ) . The whole procedure is ϕ -zCDP.

Proof. The privacy guarantee can be proved by zCDP composition (Lemma A.3) Gaussian Mechanism (Theorem A.5) and Lemma 3.7. Specifically, by Assumption 3.1 and our settings of parameters, we know the sensitivity of O 1 and O 2 are bounded by G n and M ‖ x -y ‖ n respectively, and querying O 1 and O 2 each time are G 2 n 2 σ 2 1 -zCDP and M 2 n 2 σ 2 2 -zCDP respectively. We can apply the advanced composition to prove the privacy guarantee of the whole algorithm. As the total number of iterations T is fixed, all queries made to O 2 satisfy TM 2 n 2 σ 2 2 -zCDP. It suffices to bound the total time to query O 1 , which is guaranteed in the statement. That is the total times to query O 1 is T 1 := O ( Bη log 4 ( dMB ρω ) /κ ) , so the queries to O 1 satisfy T 1 G 2 n 2 σ 2 1 -zCDP. If the time exceeds O ( Bη log 4 ( dMB ρω ) /κ ) , then we will output a set of arbitrary points which does not occur additional privacy cost. The overall privacy guarantee is ( G 2 T 1 n 2 σ 2 1 + M 2 T n 2 σ 2 2 ) -zCDP. To get ( ϕ/ 2) -zCDP, as desired we can choose:

As for the utility, we know the O 1 and O 2 constructed in Equation (2) are first kind of σ 1 √ d and second kind of σ 2 √ d norm-subGaussian gradient oracle by Fact 2.3. Hence by Lemma 3.6, the utility α 1 satisfies that

By Lemma 3.7, with probability at least 1 -ω , the total time to query O 1 is controlled and the final output will not be arbitrary points. Choosing the best κ demonstrates the bound on α 1 . The bound for α 2 follows from the value of α 1 and Lemma 4.1. Combining the two items in Lemma 4.1, we know with probability at least 1 -ω , the output point x of Algorithm 2 satisfies that

Hence we know x is an α 2 -SOSP for α 2 stated in the statement.

C.2 Proof of Claim 4.4

Claim 4.4. The gradient oracles O 1 and O 2 constructed in Equation (3) are a first kind of O ( L √ log d √ b 1 + √ dσ 1 ) norm-subGaussian gradient oracle and second kind of O ( M √ log d √ b 2 + √ dσ 2 ) norm-subGaussian gradient oracle respectively.

Proof. For the oracle O 1 , we know for each z ∈ S 1 , E z ∼P [ ∇ f ( x, z )] = ∇ F P ( x ) and ∇ f ( x, z ) -∇ F P ( x ) is nSG( L ) due to the Lipschitzness assumption. The statement follows from Fact 2.3 and Lemma 2.4. As for the O 2 , the statement follows similarly with the smoothness assumption.

C.3 Proof of Lemma 4.5

Lemma 4.5. Fix a point x ∈ R d . Given a set S of m samples drawn i.i.d. from the distribution P , then we know with probability at least 1 -ω , we have

Proof. As for any s ∈ S , ∇ f ( x ; s ) -∇ F P ( x ) is zero-mean nSG( G ) . Then the Hoeffding inequality for norm-subGuassians (Lemma 2.4) demonstrates with probability at least 1 -ω/ 2 , we have ‖∇ F S ( x ) -∇ F P ( x ) ‖ 2 ≤ O ( G log( d/ω ) √ m ) .

As for the other term, we know for any s ∈ S, E [ ∇ 2 f ( x ; s ) -∇ 2 F P ( x )] = 0 , and ‖∇ 2 f ( x ; s ) -∇ 2 F P ( x ) ‖ op ≤ 2 M almost surely. Hence applying Matrix Bernstein inequality (Theorem A.6) with σ 2 = 4 M 2 m,t = O ( √ mM log( d/ω ) ) , we know with probability at least 1 -ω/ 2 , ‖∇ 2 F S ( x ) -∇ 2 F P ( x ) ‖ op ≤ t/m .

Applying the Union bound completes the proof.

C.4 Proof of Theorem 4.6

Theorem 4.6 (Population) . Divide the dataset D into two disjoint datasets D 1 and D 2 of size glyph[ceilingleft] n/ 2 glyph[ceilingright] and glyph[floorleft] n/ 2 glyph[floorright] respectively. Set b 1 = nκ Bη , b 2 = nα 2 1 BM , σ 1 = G b 1 √ ϕ , σ 2 = M b 2 √ ϕ and κ = max( G 4 / 3 B 1 / 3 log 1 / 3 d M 5 / 3 n -1 / 3 , ( GB 2 / 3 M 5 / 3 ) 6 / 7 ( √ d n √ ϕ ) 4 / 7 ) in Equation (3) and use them as gradient oracles. Running Algorithm 1 with D 1 , and outputting the set { x i } i ∈ [ T ] if the total time to query O 1 is bounded by O ( Bη log 4 ( dMB ργω ) /κ ) , otherwise outputting a set of T arbitrary points, is ( ϕ/ 2) -zCDP, and with probability at least 1 -ω , at least one point in the output is α 1 -SOSP of F P with

Moreover, if we run Algorithm 2 with inputs { x i } i ∈ [ T ] , D 2 , B, M, G, ρ, α 1 , ε = √ ϕ with probability at least 1 -ω , Algorithm 2 can output an α 2 -SOSP of F P with α 2 = O ( α 1 + M log( ndBGM/ρω ) √ ρ min( n √ ϕ,n 1 / 2 ) √ α 1 + G ( log( n/Gω ) n √ ϕ + log( d/ω ) √ n ) ) . The whole procedure is ϕ -zCDP.

Proof. Recall that we draw the samples to construct the gradient oracles (Equation 3) without replacement, and we should have all samples to be fresh to avoid dependency, and hence we need

which is satisfied by the procedure in the statement, as if the total time to query the O 1 exceeds the threshold, the algorithm fails and outputs a set of arbitrary points. As we never reuse a sample, the privacy guarantee follows directly from the Gaussian Mechanism [20]. Specifically, the sensitivity of querying O 1 and O 2 are bounded by G/b 1 and M ‖ x -y ‖ /b 2 respectively, and querying O 1 and O 2 are ( ϕ/ 2) -zCDP by Theorem A.5.

The Norm-subGaussian parameters of the oracles follow from Claim 4.4. By lemma 3.6, we have

Setting κ = max( G 4 / 3 B 1 / 3 log 1 / 3 d M 5 / 3 ( n ) -1 / 3 , ( GB 2 / 3 M 5 / 3 ) 6 / 7 ( √ d n √ ϕ ) 4 / 7 ) , we get

Then we use the other half fresh samples D 2 to find the point in the set by Algorithm 2. By Lemma 4.1 and Lemma 4.5, we know with probability at least 1 -ω , for some large enough constant C > 0 , the output point x of Algorithm 2 satisfies that

Hence we know x is an α 2 -SOSP for α 2 stated in the statement. The privacy guarantee follows from Lemmas A.3, A.4, and 4.1.

D Omitted proof of Section 5

D.1 Proof of Lemma 5.5

Lemma 5.5 (Generalization error bound) . Let π D ∝ exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 2 )) . Then we have E D ,x ∼ π D [ F P ( x ) -F D ( x )] ≤ O ( G 2 exp( βGD ) nµ ) .

Proof. We know how to bound the KL divergence by LSI:

LSI can lead to Talagrand transportation inequality [Theorem 1 in [40]], i.e.,

The generalization error is bounded by O ( C LSI G 2 β/n ) . Using Holley-Stroock perturbation, we know C LSI ( π D ) ≤ exp( βGD ) βµ and hence the W 2 distance between π D and π D ′ can be bounded by O ( G exp( βGD ) nµ ) . The statement follows the Lipschitzness constant and Lemma 5.4.

D.2 Proof of Theorem 5.6

Theorem 5.6 (Risk bound) . We are given ε, δ ∈ (0 , 1 / 2) . Sampling from exp( -β ( F D ( x )+ µ 2 ‖ x ‖ 2 2 )) with β = O ( ε log( nd ) GD √ log(1 /δ )) ) , µ = d D 2 β is ( ε, δ ) -DP. The empirical risk and population risk are bounded by O ( GD d · log log( n ) √ log(1 /δ ) ε log( nd ) ) .

Proof. Denote π ( x ) ∝ exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 2 )) . By Lemma 5.2, we know C LSI ( π ) ≤ 1 βµ · exp( βGD ) . Plugging in the parameters and applying Theorem 5.1, we get

and hence prove the privacy guarantee.

As for the empirical risk bound, by Lemma 5.3, we know

and we know

Replacing the value of β achieves the empirical risk bound.

As for the population risk, we have

We can bound E x ∼ π [ F P ( x ) -F D ( x )] ≤ O ( G 2 exp( βGD ) nµ ) ≤ O ( GDε log( n ) n 1 -c d √ log(1 /δ ) ) by Lemma 5.5 for an arbitrarily small constant c > 0 . Hence the empirical risk is dominated term compared to E x ∼ π [ F P ( x ) -F D ( x )] , and we complete the proof.

D.3 Implementation

We rewrite them below: Let ̂ F ( x ) := F ( x ) + r ( x ) where r ( x ) is some regularizer, and F = E i ∈ I f i is the expectation of a family of G -Lipschitz functions.

Algorithm 3 AlternateSample, [35]

1: Input: Function ̂ F , initial point x 0 ∼ π 0 , step size η

2: for t ∈ [ T ] do

3: y t ← x t -1 + √ ηζ where ζ ∼ N (0 , I d )

4: Sample x t ← exp( -̂ F ( x ) -1 2 η ‖ x -y t ‖ 2 2 )

5: end for

6: Output: x T

Theorem D.1 (Guarantee of Algorithm 3, [15]) . Let K ⊂ R d be a convex set of diameter D , and ̂ F : K → R , and π ∝ exp( -̂ F ) satisfies LSI with constant C LSI . Then set η ≥ 0 , we have

where R q ( π ′ , π ) is the q -th order of Renyi divergence between π ′ and π .

To get a sample from exp( -̂ F ( x ) -1 2 η ‖ x -y t ‖ 2 2 ) , we use the rejection sampler from [26], whose guarantee is stated below:

Lemma D.2 (Rejection Sampler, [26]) . If the step size η glyph[lessorsimilar] G -2 log -1 (1 /δ inner ) and the inner accuracy δ inner ∈ (0 , 1 / 2) , there is an algorithm that can return a random point x that has δ inner total variation distance to the distribution proportional to exp( -̂ F ( x ) -1 2 η ‖ x -y ‖ 2 2 ) . Moreover, the algorithm accesses O (1) different f i function values and O (1) samples from the density proportional to exp( -r ( x ) -1 2 η ‖ x -y ‖ 2 2 ) .

Combining Theorem 5.6, Theorem D.1 and Lemma D.2, we can get the following implementation of the exponential mechanism for non-smooth functions.

Theorem 5.7 (Implementation, risk bound) . For ε, δ ∈ (0 , 1 / 2) , there is an ( ε, 2 δ ) -DP efficient sampler that can achieve the empirical and population risks O ( GD d · log log( n ) √ log(1 /δ ) ε log( nd ) ) . Moreover, in expectation, the sampler takes ˜ O ( nε 3 log 3 ( d ) √ log(1 /δ ) / ( GD ) ) function values query and some Gaussian random variables restricted to the convex set K in total.

Proof. By Theorem 5.6, it suffices to get a good sample from π with density proportional to exp( -β ( F D ( x ) + µ 2 ‖ x ‖ 2 2 )) where β = O ( ε log( nd ) GD √ log(1 /δ )) ) , µ = d D 2 β . Set q = 1 , which gives that R q ( · , · ) is the KL-divergence. Suppose we let x 0 is drawn from density proportional to exp( -β 2 µ ‖ x ‖ 2 2 ) , then the KL divergence between π 0 and π is bounded by exp( qβGD ) .

Now let π ( i ) T be the distribution we get over x T from Algorithm 3 if we use an exact sampler for i iterations, then the sampler of Lemma D.2 for the remaining T -i iterations. The output of Algorithm 3 that we actually get is π (0) T . Note that C LSI ≤ D 2 n , and η glyph[lessorsimilar] β -2 G -2 log -1 (2 T/δ ) . Setting

we get δ inner = δ/ 2 T in Lemma D.2 and that R 1 ( π ( T ) T , π ) ≤ δ 2 / 8 . This implies the total variation distance between π ( T ) T and π is at most δ/ 2 by Pinsker's inequality. Furthermore, by the postprocessing inequality, the total variation distance between π ( i ) T and π ( i +1) T is at most δ/ 2 T for all i . Then by triangle inequality the total variation distance between π (0) T and π is at most δ .

D.4 Proof of Theorem 5.8

Theorem 5.8. There exists an ε -DP differentially private algorithm that achieves a population risk of O ( GD ( d log( εn/d ) / ( εn ) + √ d log( εn/d ) / ( √ n ) )) .

Proof. We pick a maximal packing P of O (( D/r ) d ) points, such that every point in K is distance at most r from some point in P . By G -Lipschitzness, the risk of any point in P for the DP-ERM/SCO problems over K are at most Gr plus the risk of the same point for DP-ERM/SCO over P . The exponential mechanism over P gives a DP-ERM risk bound of O ( GD εn log | P | ) . Next, note that the empirical loss of each point in P is the average of n random variables in [0 , GD ] wlog. the expected maximum difference between the empirical and population loss of any point in P is

So, O ( GD √ log | P | √ n ) . Putting it all together we get a DP-SCO expected risk bound of:

This is approximately minimized by setting r = Dd/εn . This gives a bound of:

E Conclusion

We present a novel framework that can improve upon the state-of-the-art rates for locating secondorder stationary points for both empirical and population risks. We also examine the utilization of the exponential mechanism to attain favorable excess risk bounds for both a polynomial time sampling approach and an exponential time sampling approach. Despite the progress made, several interesting questions remain. There is still a gap between the upper and lower bounds for finding stationary points. As noted in [2], it is quite challenging to beat the current ( √ d n ) 2 / 3 empirical upper bound, and overcoming this challenge may require the development of new techniques. A potential avenue for improving the population rate for SOSP could be combining our drift-controlled framework with the tree-based private SpiderBoost algorithm in [2]. Additionally, it is worth exploring if it is possible to achieve better excess risk bounds within polynomial time, and what the optimal risk bound could be.

F Extended related work

In the convex setting, it is feasible to achieve efficient algorithms with good risk guarantees. In turn, differentially private empirical risk minimization (DP-ERM) [13, 14, 17, 28, 33, 9, 43, 41, 42] and differentially private stochastic optimization [4, 7, 6, 22, 34, 3, 32, 26, 23, 12, 27] have been two of

the most extensively studied problems in the DP literature. Most common approaches are variants of DP-SGD [14] or the exponential mechanism [38].

As for the non-convex optimization, due to the intrinsic challenges in minimizing general non-convex functions, most of the previous works [49, 50, 47, 46, 56, 42, 44, 54, 2, 51, 24] adopted the gradient norm as the accuracy metric rather than risk. Instead of minimizing the gradient norm discussed before, [8] tried to minimize the stationarity gap of the population function privately, which is defined as Gap F P ( x ) := max y ∈K 〈∇ F P ( x ) , x -y 〉 , which requires K to be a bounded domain. There are also some different definitions of the second order stationary point. We refer the readers to [37] for more details.

The risk bound achieved by algorithms with polynomial running time is weak and requires n glyph[greatermuch] d to be meaningful. Many previous works consider minimizing risks of non-convex functions under stronger assumptions, such as, Polyak-Lojasiewicz condition [49, 55], Generalized linear model (GLM) [46] and weakly convex functions [8].

In the (non-private) classic stochastic optimization, there is a long line of influential works on finding the first and second-order stationary points for non-convex functions, [1, 29, 21, 53, 18].

First order stationary points. Progress towards privately finding a first-order stationary point is measured in ( i ) the norm of the empirical gradient at the solution x , i.e., ‖∇ F D ( x ) ‖ , and ( ii ) the norm of the population gradient, i.e., ‖∇ F P ( x ) ‖ . We summarize compare these first-order guarantees achieved by Algorithm 1 with previous algorithms in Table 2:

Second order stationary points. We say a point x is a Second-Order Stationary Point (SOSP), or a local minimum of a twice differentiable function g if ‖∇ g ( x ) ‖ 2 = 0 and smin ( ∇ 2 g ( x )) ≥ 0 . Exact second-order stationary points can be extremely challenging to find [25]. Instead, it is common to measure the progress in terms of how well the solution approximates an SOSP.

Definition F.1 (approximate-SOSP, [1]) . We say x ∈ R d is an α -second order stationary point ( α -SOSP) for ρ -Hessian Lipschitz function g , if

Existing works in finding approximate SOSP privately give guarantees for the empirical function F D . We improve upon the state-of-the-art result and give the first guarantee for the population function F P , which is summarized in Table 3.

Peer reviews:
comments: SUMMARY:
This paper provides a private algorithm for (Stochastic) Non-Convex Optimization using gradients, differences between two successive gradients, and the exponential mechanism for privacy (added to the gradient). It leverages the SpiderBoost algorithm and its private version to find (privately) stationary points. The main challenge addressed by the paper is to keep the quality of the gradient estimations at all steps.

STRENGTHS:
- simple modification of  SpiderBoost to maintain the quality of the gradient estimations.
- interesting properties of the algorithm, i.e., the algorithm is capable of escaping the saddle point  with high probability, and a large drift implies significant decrease in the function value, limiting the number of gradient queries.

WEAKNESSES:
- there is something weird about Algorithm 1. drift_t is set to zero in lines 7 and 10, but then overwritten in line 14.

QUESTIONS:
See above

LIMITATIONS:
- traditional hypotheses on these types of results
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

comments: SUMMARY:
This paper addresses the problem of private stochastic optimization for general non-convex functions. For smoothed functions, the authors propose a combination of the Gaussian mechanism and the variance reduction technique (spider algorithm) to find the SOSP of the loss function. Their results for the empirical risk improve upon previous state-of-the-art bounds by a factor of $n\varepsilon$ and their results for the population risk are new in the field of private optimization literature. For general Lipschitz non-convex problems, the authors establish improved bounds for bounding the excess risk using exponential mechanisms. Notably, they provide a new exponentially-time algorithm that matches the proposed lower bound.

STRENGTHS:
The paper is well-written and clear in its claims. The authors propose a novel approach to private stochastic optimization for both smooth and non-convex loss functions by combining existing private mechanisms with the Spider algorithm. They provide new results for both upper and lower bounds on the excess population risk, which are technically solid and contribute to the literature on private stochastic optimization. Overall, the results presented in the paper make a valuable contribution to the field.

WEAKNESSES:
1. The paper makes a significant contribution to improving the bounds for private optimization, but there is still a gap between the existing upper and lower bounds that needs to be explored in future work.
2. While the paper focuses on theoretical studies, the authors could consider conducting experimental studies to validate their theoretical findings

QUESTIONS:
I have no further questions.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a non-convex optimization framework that integrates two types of gradient oracle and an application of the exponential mechanism to find the global minima of non-convex objectives, requiring minimal assumptions and showcasing remarkable population risk bound performance. The paper also highlights the ability to emulate previous empirical and population risk bounds effectively, removing the necessity for smoothness assumptions.\n\nSTRENGTHS:\nThis paper presents robust, theoretical insights into a clearly defined problem, and its logical flow makes it accessible and easy to comprehend. \n\nIt commendably surveys and incorporates existing related literature.\n\nWEAKNESSES:\nThis paper appears to conclude abruptly, with the discourse ending unexpectedly following a theorem. The absence of a conclusion or summary weakens the paper's overall structure.\n\nThere is an absence of numerical experiments. \n\nAdditionally, the authors seem to have relied solely on Advanced Composition for privacy accounting. There are, however, various alternative privacy accounting methods available that could potentially improve the overall performance. For instance, the 'Better Privacy Accounting' method [1], utilization of GDP [2], or the application of tighter composition theorems [3] might be considered for future research and improvements.\n\n[1]: Altschuler, J., & Talwar, K. (2022). Privacy of noisy stochastic gradient descent: More iterations without more privacy loss. Advances in Neural Information Processing Systems, 35, 3788-3800.\n\n[2] Liu, Y., Sun, K., Jiang, B., & Kong, L. (2022). Identification, amplification and measurement: A bridge to gaussian differential privacy. Advances in Neural Information Processing Systems, 35, 11410-11422.\n\n[3] Kairouz, P., Oh, S., & Viswanath, P. (2015, June). The composition theorem for differential privacy. In International conference on machine learning (pp. 1376-1385). PMLR.\n\nQUESTIONS:\nHow does the performance of the Stochastic Spider, as presented in this paper, compare to that of the BoostSpider in practical applications? While the theoretical results indicate a rate improvement, is this enhancement accompanied by an increase in constants and an impact on finite sample performances?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies differentially private second-order optimization problems, where the loss function is assumed to be second-order smooth. The objective is to identify second-order stationary points (SOSP) while ensuring privacy guarantees. The authors present results for both empirical risk and population risk, with notable advancements over existing state-of-the-art outcomes. Specifically, the empirical risk minimization (ERM) bound improves from the state-of-art rate, while the population risk result stands as the first of its kind in the literature on differentially private SOSP optimization. Furthermore, the paper contributes a lower bound analysis for private SOSP optimization, further strengthening its significance in this domain.\n\nSTRENGTHS:\nThe strength of this paper lies in its highly comprehensive study of private second-order optimization. The authors study both empirical risk minimization and population risk, and the obtained bounds improves from the state-of-the-art results in both case. The inclusion of a lower bound also closes the gap of this problem.\n\nWEAKNESSES:\nThe authors can enhance the writing by providing additional discussion regarding the algorithm employed, the obtained results, and the insights derived from their findings. Furthermore, providing more detailed explanations for the technical results cited from previous works, such as Lemma 3.4 and Alg.2, would help clarify any confusion and prevent ambiguity.\n\nQUESTIONS:\nI have a few questions regarding the general SOSP framework (Algorithm 1):\n- What role does $\\text{frozen}_t$ play?\n- line 8: why is an additional noise $g_t$ added to the private oracle $\\mathcal{O}_1(x_t)$?\n- Since Alg. 1 only serves as a non-private SOSP optimization framework, is it possible to choose other variance-reduction algorithms (e.g., [1])? If not, what is the key characteristic that distinguishes SpiderBoost from other optimizers in the private setting?\n\nAlso, Section 5 focuses on private excess risk minimization, which seems to deviate from the previous discussion about SOSP. I'm wondering how Sec. 5 relates to the previous part of the paper?\n\nReference:\n1. Ashok Cutkosky, Francesco Orabona. Momentum-Based Variance Reduction in Non-Convex SGD.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of non-convex optimization under differential privacy constraints. The goal of the paper is to find first and second order stationary points of the function. The proposed algorithm which is inspired by SpiderBoost [1] is a sampling based method that generates a private estimator that minimizes both excess empirical and population risks. \n\nThe authors studied the cases: when only polynomial run time is allowed, and when the exponential run time is permitted. In the first case, their proposed algorithm outperforms the previous best and tightens the gap between the provided upper bound and the already existing lower bound. For the case when exponential run time is allowed, the paper provides a nearly matching upper and lower bound. \n\n[1] Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. Advances in Neural Information Processing Systems, 32, 2019.\n\nSTRENGTHS:\nThe paper is a collection of elegant results. From a technical point of view, the paper is solid. It enhances advanced techniques to fulfill its objective. While the paper outperforms previous best algorithm [2], with polynomial run time it relaxes the smoothness assumption that has been assumed in [2]. \n\n[2] Di Wang, Changyou Chen, and Jinhui Xu. Differentially private empirical risk minimization with non-convex loss functions. In International Conference on Machine Learning, pages 6526–6535. PMLR, 2019.\n\nWEAKNESSES:\nFor a reader who is not fluent with the literature, the paper is not easy to follow. Several insights left to be unexplained. The paper is quite dense without enough provided intuition on the proposed algorithm. The authors did a good job in mentioning the similarities with precious work, but they did not discuss their technical novelties. Some expressions and notation are left undefined (ex. W_2 at line 308). \n\nAs a side note, I think P at line 18 should be P*.\n\nQUESTIONS:\n1. What is the main technical novelty of the work?\n\n2. Is the proposed algorithm a privatized version of SpiderBoost? if yes, except the added gaussian noise what are the additional elements that are added? And can one similarly construct a privatized version of other algorithms like Spider or SARAH?\n\n3. Can you please provide an intuition that why your technique could relax the smoothness assumption?\n\n4. Except the relaxed smoothness assumption, can you please provide an intuition what would be the difference in the perfromance if one used Spider or SARAH instead of SpiderBoost?\n\nLIMITATIONS:\nThe limitation of the work is the optimality gap between the upper and lower bound, where the authors mentioned this and referred to it as an open problem.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}