Differentially Private Stochastic Optimization: New Results in Convex and Non-Convex Settings

Abstract:  Raef Bassily, Cristóbal Guzmán, Michael Menart

Reviews and author responses:

This paper studies differentially private stochastic optimization under different settings and develop several novel results. The authors first consider convex and nonsmooth generalized linear models, for which they develop an algorithm with optimal generalization bounds and nearly linear time. As a comparison, the existing results require super-linear training time. Then, they move on to nonconvex and smooth problems under $\ell_1$ and $\ell_p$ setting, and developed private algorithms with linear time based on stochastic Frank-Wolfe algorithm. Finally, they develop the first private algorithms for nonconvex and nonsmooth problems.   The paper contains many theoretical results on differentially private stochastic optimization under various settings. These results seem to be novel and interesting. I have some queries on the analysis and results and would increase my scores if the authors can address my queries.

The paper contains a nice table to summarize the main results. While this table is useful to understand the contribution of the paper, it lacks of a comparison with related work. It would be helpful if the authors can summarize the state of the art in the table so that readers can clearly understand the improvement over existing work.

According to Table 1, the bound in Theorem 10 has two terms. The first term is a decreasing function of $d$. This appears a bit surprising since it is expected that a larger dimension would always increase the difficulty of learning. It is more natural that both terms grow as $d$. It would be helpful if the authors can give some explanation.

According to Table 1, the rates in Theorem 10 are independent of $p$. Therefore, it is not clear to me the advantage of using different norm in the learning setting. Theorem 10 suggests that algorithms in different $\ell_p$ setting would behave similarly.

In Lemma 1, the authors use the advanced composition. In the literature, people have developed another method to handle composition called Moments Accountant [1]. As indicated in [1], MA can outperform the advanced composition by a log(T) factor in the variance of random noise. Can the results in this paper be further improved if using MA?

In the proof of Lemma 4, the meaning of $\tau$ seems not given. The authors mention that standard guarantees of the bisection give the bound $h_\beta(\tilde{u})-h_\beta(u^*)\leq\tau 2^{-T}$. This bound is not quite clear to me. It would be helpful if the authors can give some details.

In Algorithm 3, the authors resort to PSGD in [FKT20]. I would suggest the authors to present PSGD in the paper to make it self-contained. In the first equation of Page 16, $F_{\beta,D}(w_K)-F_{\beta,D}(w_\beta^*)$ should be $F_{\beta,D}(\tilde{w}_K)-F_{\beta,D}(w_\beta^*)$. Furthermore, it is not clear how the first inequality holds. It seems that the authors omit several steps in the deduction. It would be helpful if the authors can give some details.

In Line 548, the authors remove the subscript V in the inner product. It is not clear to me how this subscript V in the inner product can be removed. I would suggest the authors to give some explanation.

In Theorem 5, the authors should mention that they consider convex problems.
 
The meaning of the abbreviation DP-SCO is not given.

Line 189: in error to due to

Line 283: normed spaced

[1] Deep Learning with Differential Privacy This is a theory paper and I do not see societal impacts.

 Thank you for your detailed response. I have also read other reviews.  My concerns are well addressed and I would increase my score. 

 Thank you for your detailed review. The typos/omissions you mention will be fixed. With regards to the rest of the comments:

* On using $\ell_p$ norms: This is not really our choice. Problems are naturally embedded in a certain geometry (e.g., sparsity oriented problems have diameter and Lipschitzness bounded in $\ell_1$ norm), and our algorithms use that geometry to the best extent possible. Notice that changing $p$ would change simultaneously the radius, Lipschitzness, and other problem parameters.  Solving a stochastic optimization problem in an $\ell_p$ setting using an optimal algorithm for the $\ell_2$ setting would lead to  significantly worse (by dimension-dependent factors) accuracy/convergence guarantees.  On the other hand, using a method adapted to the  $\ell_p$ geometry (e.g., mirror descent in the convex non-private setting) can lead to improvements in accuracy of order polynomial in the dimension (see, e.g., Sec 5.1 in https://www2.isye.gatech.edu/~nemirovs/MLOptChapterI.pdf and Ch. 4 in https://arxiv.org/pdf/1405.4980.pdf). In summary, despite the bounds may look similar, achieving such bounds requires a different structure/tuning for the algorithm.  

* Comparison to Other Work: Direct comparison to existing work is not straightforward as most of the settings we explore are, to the best of our knowledge, new. With that said, [ZCH+20], which considers smooth unconstrained optimization in $\ell_2$ setup (whereas we consider smooth constrained optimization in $\ell_2$ setup) is the most directly comparable. In this case, their algorithm achieves better sample complexity with worse oracle complexity. Specifically, to obtain an $\alpha$ approximate stationary point, [ZCH+20] has oracle complexity $\alpha^{-4}\sqrt{d}$ whereas our algorithm achieves  $\max(\alpha^{-5/2}\sqrt{d}, \alpha^{-10/3}d^{-1/3})$ (furthermore, following your next comment we have been able to further refine this oracle complexity to $\max(\alpha^{-5/2}\sqrt{d}, \alpha^{-3})$). We will make this comparison more explicit  in the paper. 

* Theorem 10: Thank you for this observation. The bound we obtained only decreases w.r.t. $d$ up to a certain point: namely, $d=n^{1/3}\varepsilon^2$. After that point it strictly grows  with $d$. . This behavior of the upper bound implies that it can be further refined in the following way: if the dimension is lower than $n^{1/3}\varepsilon^2$, then we can embed the problem in $n^{1/3}\varepsilon^2$ dimensions, and use the upper bound obtained for such dimension. If the dimension is larger than $n^{1/3}\varepsilon^2$, then we use our upper bound as is. Notice that in the latter range, the upper bound is $O(\frac{\kappa^{2/3}(d\tilde\kappa)^{1/5}}{(n\varepsilon)^{2/5}})$. Hence, we easily get the following, more refined upper bound:  $O(\frac{\kappa^{2/3}\tilde\kappa^{1/5}}{n^{1/3}}+\frac{\kappa^{2/3}(d\tilde\kappa)^{1/5}}{(n\varepsilon)^{2/5}})$, which no longer has dependence on $d$ in the denominator. We will update the paper to include this refinement of the bound. 
Regarding why our upper bound had a dimension term in the denominator, this is a result of the parameter optimization we perform (sensitivity and number of rounds become dimension dependent because of the use of the Gaussian mechanism).

* Moments Accountant: Our results in Section 5 implicitly use the Moments Accountant through the analysis of [AFKT21]. The rest of our algorithms can be grouped into two categories: those which use parallel composition and those which apply advanced composition to pure DP mechanisms. The Moments Accountant will not lead to further improvement in either of these cases. 

* Bisection Method: $\tau$ here is a bound on the initial excess loss. The convergence is derived from the fact that bisection shrinks the distance to the minimizer at an exponential rate, which is a well-known fact in one-dimensional convex optimization. We will elaborate on this to clarify this argument.

* Regarding removing the subscript V in the inner product, as stated in the text, this step follows from the fact that the gradient lies in the span of feature vectors. Over this subspace, VV^T acts as the identity transformation, and can thus be removed.

* Inequality at the top of page 16: This inequality simply comes from the convergence guarantees of PSGD and the bound on the expectation of the norm of the Gaussian noise. We will update this section to make this derivation easier to follow.


 Thank you for the suggestion. We will do this to facilitate reading, and also provide more effort on motivating and explaining the most technical parts of the paper. We are considering shortening Section 3 in order to give more room to Sections 4 and 5, but we are open to more specific suggestions should you have any. 

 We would like to thank the reviewer for the positive assessment and constructive feedback. 
To improve the clarity of presentation, we are considering shortening the length of Section 3 and giving more room to discuss parameter optimization and analysis done in Sections 4 and 5, which are arguably the most challenging ones. We are also open to take more specific directions from the reviewer, if they see other ways of improving the presentation.


The paper studies the stochastic optimization problem with differential privacy in both the convex and non-convex setting. The main contributions of the paper are algorithms that achieve optimal excess risk and have faster running times in a number of settings. The first setting is the non-smooth convex setting. Here a challenging open problem is to obtain nearly-linear running time, and the paper makes progress towards this goal by providing such an algorithm for the special case of generalized linear losses in the ell_2 setting. The paper also provides an algorithm for generalized linear losses in the ell_1 setting as well. The next setting considered is the smooth non-convex setting, where the main contributions are linear time algorithms for ell_2, ell_1, and more generally ell_p with 1 <= p <= 2. The final setting is weakly convex functions, where the paper gives the first algorithm with privacy for ell_p with 1 <= p <= 2.  Significance: The stochastic optimization problem with privacy is a canonical problem in the area, and it has received considerable attention. This work makes several novel contributions to this line of work, and it makes progress towards fast algorithms in both the convex and non-convex setting.

Novelty/originality: There seems to be sufficient novelty in both the algorithms and the analysis. The paper builds on techniques from both the prior work on private stochastic optimization as well as non-private optimization. However, adapting and extending the techniques poses several challenges in both the algorithms and the analysis.  

Clarity: The paper could be made more accessible by providing more discussion and intuition for the algorithms and the analysis. Providing a discussion on the parameter choices made by the algorithms and an interpretation of the convergence results would be helpful as well. Yes

The paper provides a series of results on differentially private optimization. The covered settings include convex (nonsmooth) generalized linear losses and nonconvex (nonsmooth) losses.
Contributions:
The provided convergence rates are new for the consider settings.  The paper provides a few algorithms to achieve new or improved convergence rates for a few settings of private optimization. 
Strengths:
1. Some convergence rates have a milder dependency on dimension compared with existing results in the literature. 
2. The convergence results on non-smooth settings seem to be new.

Weaknesses:
1. The paper is hard to read, there are too many results filled in the paper which leaves little space for informative discussions on intuitions and connections to existing results. It might be better to put some less important results in Appendix and provide more discussions in the main paper.
 Yes.

Meta-review: This paper studies private algorithms for stochastic convex and non-convex optimization when the losses of interest come from Generalized Linear Models. For this case, they show that faster algorithms are possible for the non-smooth convex case for ell_2, and better rates are possible for the l_1 case, compared to the corresponding results for convex non-GLM losses. The author also study non-convex losses and there they show that under various constraints, one can privately find approximately stationary points.
The paper pushes forward the research in this important are of private stochastic optimization. The authors are encouraged to go over the reviews and incorporate corrections, as well as the relevant clarifications that came in the rebuttal. I recommend this paper be accepted.

Decision: accept