Private Non-smooth ERM and SCO in Subquadratic Steps

3/2 d1/8 + N 2 d ) gradient queries , which is achieved with the help of subsampling and smoothing the function via convolution . Combining this result with the iterative localization technique of Feldman et al . [ FKT20 ] , we achieve the optimal excess population loss for the SCO problem with O ( min { N5/4d1/8 , N3/2 d1/8 } ) gra- dient queries . Our work makes progress towards resolving a question raised by Bassily et al . [ BFGT20 ] , giving first algorithms for private SCO with subquadratic steps . In a concurrent work , Asi et al . [ AFKT21 ] gave other algorithms for private ERM and SCO with subquadratic steps . 1 Introduction . Privacy has become an important consideration for learning algorithms dealing with sensitive data . Over the past decade , differential privacy , introduced in the seminal work of [ DMNS06 ] , has established itself as the defacto notion of privacy for machine learning problems . In this paper , we revisit Empirical Risk Minimization ( ERM ) and Stochastic Convex Optimization ( SCO ) problems , two fundamental problems in statistics and machine learning , in differential privacy setting . In the ERM problem , we are given a family of convex functions { f ( · , x ) } x∈Ξ over a bounded closed convex set K ⊂ Rd of diameter D , a data set S = { x1 , · · · , xN } drawn from some unknown distribution P over the universe Ξ , and the objective is to minimize F̂ ( ω ) : = 1 N ∑ xi∈S f ( ω , xi ) over ω ∈ K , The excess empirical loss with respect to a solution ω is defined by F̂ ( ω ) − F̂ ∗ , where F̂ ∗ = minω∈K F̂ ( ω ) . A closely related but more general problem is SCO , where the objective is to minimize F ( ω ) : = E x∼P f ( ω , x ) over ω ∈ K , We refer F ( ω ) − F ∗ as the excess population loss , where F ( ω ) = Ex∼P f ( ω , x ) and F ∗ = minω∈K F ( ω ) . Differentially private convex optimization has been studied extensively for over a decade now [ CM08 , RBHT09 , CMS11 , KST12 , JT14 , TTZ14 , BST14 , TTZ15 , KJ16 , WLK+17 , FTS17 , ∗Algorithms Group , Microsoft Research ( MSR ) Redmond . jakul @ microsoft.com †University of Washington and Microsoft Research . Supported by NSF awards CCF-1749609 , DMS1839116 , DMS-2023166 , Microsoft Research Faculty Fellowship , Sloan Research Fellowship , Packard Fellowships.yintat @ uw.edu ‡University of Washington . Part of the work was done while visiting Shanghai Qi Zhi Institute.dgliu @ uw.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ZZMW17 , WYX17 , INS+19 ] . Both DP-ERM and DP-SCO for smooth convex functions are well understood in the sense that we know ( near ) linear time algorithms that achieve optimal loss ; we refer the readers to [ WYX17 , FKT20 ] for more details . However , for the more general non-smooth convex loss functions our understanding is not yet complete , which is the focus of this paper . Our algorithms for DP-SCO build upon our improvements for the DP-ERM problem , and hence we begin with DP-ERM problem . A summary of the state-of-the-art results of DP-ERM and our contributions for the non-smooth convex loss functions is given in Table 1 . We will discuss the concurrent work [ AFKT21 ] separately at the end of the introduction , and the following discussion is only limited to the previous work . Nε } ) and of Strongly Convex function is Ω ( min { G2µ , G 2d µN2ε2 } ) [ BST14 ] . [ KST12 ] used objective perturbation method to design a DP-algorithm with O ( GD √ d log ( 1/δ ) √ Nε ) excess empirical risk . This result was improved significantly by [ BST14 ] , who first showed a lower bound of Ω ( min { GD , GD √ d Nε } ) on the excess empirical risk for DP-ERM . Further , they gave an algorithm with excess empirical risk O ( GD log 3 2 ( N/δ ) √ d log ( 1/δ ) Nε ) , which is sub-optimal by a factor of log 3 2 ( N/δ ) . Their algorithm is based on a modification of SGD by adding Gaussian noise to the gradients to make it differentially private . The privacy analysis proceeds via amplification by sampling and the strong composition theorem . Roughly speaking , the logarithmic blowup in the excess empirical risk is due to two reasons : 1 ) The strong composition theorem requires that at each step one needs to add Gaussian noise with a larger variance ; 2 ) They used sub-optimal convergence rate O ( log T/ √ T ) for T -step SGD . However , getting the optimal bounds with small gradient complexity for non-smooth case turns out to be a more difficult problem . This was noted by [ WYX17 ] , who raised it as an important open problem . This question was answered in [ BFTT19 ] , who gave an algorithm with almost optimal excess empirical risk . To achieve this , [ BFTT19 ] first consider the smooth case , and give an improved privacy analysis via the Moments Accountant technique proposed by [ ACG+16 ] . They extend their result to non-smooth case by applying Moreau-Yosida envelope technique ( a.k.a . Moreau envelope smoothing ) [ Nes05 ] to make the function smooth . However , this technique is computationally inefficient and leads to O ( N4.5 ) -gradient computations for the whole algorithm . This limitation was overcome in a recent work of [ BFGT20 ] who gave the optimal excess empirical risk guarantee with O ( N2 ) -gradient computations . The privacy analysis of this result also used Moments Accountant method , and they used the standard online-to-batch conversion technique [ CBCG04 ] to prove the high-probability and expectation bound on the excess empirical error of SGD . As we can see from Table 1 , all the previously known results ( except the concurrent work [ AFKT21 ] ) achieving near optimal excess empirical risk bounds require at least O ( N2 ) -gradient computations . As Table 2 shows , a similar situation arises in Stochastic Convex Optimization ( SCO ) , which is a closely related problem compared to ERM . Many results for SCO [ BST14 , BFTT19 , BFGT20 ] are directly based on ERM ; that is , solving the ERM and analyzing the generalization error . The first non-trivial result for general convex loss functions achieving excess population loss of O ( GD ( d 1/4 √ N + √ d Nε ) ) was given by [ BST14 ] , who showed the result by first solving the ERM prob- lem and bounding the generalization error . They used the result on universal convergence directly , namely , bounding supω∈K E [ F ( ω ) −F̂ ( ω ) ] . But this method has its limitations ; For example , [ Fel16 ] showed that lower bound of universal convergence is Ω ( √ d/N ) for some ( not necessarily convex ) loss functions . Later , [ BFTT19 ] , [ FKT20 ] and [ BFGT20 ] obtained near optimal excess population loss with significantly better running times ( gradient complexity ) . The privacy analysis in these papers relied on recent advances in the privacy techniques such as the Moments Accountant method [ ACG+16 ] , Rényi differential privacy ( RDP ) [ Mir17 ] and the Privacy Amplification by Iteration [ FMTT18 ] and other fast stochastic convex optimization algorithms such as [ JNN19 ] . The excess population loss bound in most of these works followed by solving a ( phased ) convex ( regularized ) ERM problem and then appealing to the uniform stability property [ HRS16 ] or the iterative localization approach [ FKT20 ] to do the generalization error analysis . N2ε2 ) ) [ BST14 ] . Despite these impressive improvements , as the Table 2 suggests , the previous algorithms that achieve the optimal excess population loss still require O ( N2 ) -gradient computations . Indeed , [ BFGT20 ] write that “ Proving that quadratic running time is necessary for general non-smooth DP-SCO is a very interesting open problem ... ” . Understanding if the lower bound is the right answer to the above questions or one can design algorithms with subquadratic gradient complexity is the main motivation that spurred our work . 1.1 Our Contributions . The first contribution of this paper is to show that we can obtain subquadratic gradient complexity bound for ERM when the dimension is super constant . In particular , for the important regime of over-parameterization ( d ≥ N ) , we achieve a bound of O ( N1+3/8 ) . Let Kr = { y | y = ω + z , ω ∈ K , z ∈ Rd , ∥z∥r ≤ r } . We now state the result formally . Theorem 1.1 ( DP-ERM ) . Suppose K ⊂ Rd is a closed convex set and { f ( · , x ) } x∈Ξ is a family of G-Lipschitz and convex functions over Kr , where r = D √ d log ( 1/δ ) εN 4 . For ε , δ ≤ 1/2 , given any sample set S consists of N samples from Ξ and arbitrary initial point ω0 ∈ K , we have a ( ε , δ ) -differentially private algorithm A which takes O ( εN 3 2 d1/8 log1/4 ( 1/δ ) + ε2N2 d log ( 1/δ ) ) gradient queries and outputs ωT such that E [ F̂ ( ωT ) − F̂ ∗ ] = O ( GD √ d log ( 1/δ ) εN ) , where D = ∥ω∗ − ω0∥2 , F̂ ( ω ) = 1N ∑ xi∈S f ( ω , xi ) , F̂ ∗ = minω∈K F̂ ( ω ) , and the expectation is taken over the randomness of the algorithm . 4We only need consider the non-trivial case when √ d log ( 1/δ ) εN ≤ 1 , or any feasible solution is good enough . This means that r = O ( D ) , which is a mild assumption . Moreover , if { f ( · , x ) } x∈Ξ is also µ-strongly convex over Kr , we can meet the same gradient query complexity and get a solution ωT such that E [ F̂ ( ωT ) − F̂ ∗ ] = O ( G2d log ( 1/δ ) µε2N2 ) . The main contribution of the paper is to use the above result to obtain a better gradient complexity for the SCO problem , answering the open problem in [ BFGT20 ] . Combining our private ERM algorithm with the iterative localization technique [ FKT20 ] , we give the first algorithm achieving optimal error with strictly sub-quadratic steps for all dimensions . Theorem 1.2 ( DP-SCO ) . Suppose ε , δ ≤ 12 and sample set S consists of N samples drawn i.i.d from a distribution P over Ξ . Suppose { f ( · , x ) } x∈Ξ is convex and G-Lipschitz with respect to ℓ2 norm and convex over Kr , where r = D √ d log ( 1/δ ) εN , there is an ( ε , δ ) -differentially private algorithm taking O ( N +min { √εN5/4d1/8 , εN 3/2 d1/8 log1/4 ( 1/δ ) } ) gradient queries to get a solution ωT such that E [ F ( ωT ) − F ( ω∗ ) ] = O ( GD ( 1√N + √ d log ( 1/δ ) Nε ) . Moreover , if { f ( · , x ) } x∈Ξ is also µ-strongly convex over Kr , we can meet the same gradient query complexity and get a solution ωT such that E [ F ( ωT ) − F ( ω∗ ) ] = O ( G2 µ ( d log ( 1/δ ) ε2N2 + 1 N ) ) . Finally , we note that with straightforward modifications , our results can also capture the regularized ERM and SCO , where there is one more simple ( and convex ) function h ( ω ) added to the objective function and the objective function takes the form 1N ∑ xi∈S f ( ω , xi ) + h ( ω ) , which show up often in the previous work such as [ RBHT09 , KST12 , WYX17 , INS+19 ] . 1.2 Our Techniques Most of the previous works [ BST14 , BFTT19 , BFGT20 ] that achieve near optimal bounds for ERM and SCO are based on adaptations of SGD to make it differentially private . The information theoretic lower bound of Ω ( 1/ √ T ) for T -step SGD may be one of the important reasons why we can not get subquadratic gradient complexity for non-smooth convex ERM easily . Consider the algorithm in [ BFGT20 ] as an example . It needs to add Gaussian noise v ∼ N ( 0 , σ2Id×d ) with σ2 = G 2 log ( 1/δ ) ε2 to each gradient . By a standard analysis of SGD , we can only show an excess empirical risk of Θ ( D √ dσ2√ T ) , which requires us to set T = Ω ( N2 ) to get ideal bound , thus hitting the quadratic barrier . We deviate from the above approaches for designing private algorithms for non-smooth functions . First notice that the gradient complexity O ( εN 3 2 d1/8 log1/4 ( 1/δ ) + ε 2N2 d log ( 1/δ ) ) in Theorem 1.1 is the same for both strongly convex and general non-smooth functions ; same holds for DP-SCO . This is not a coincidence ; We prove that if we can achieve optimal empirical risk ( population loss ) for one case , then we can achieve optimal empirical risk ( population loss ) for another with the same privacy guarantee and gradient complexity . The Figure 1 illustrates the relationship among these different problems . Our result for the general convex non-smooth case is obtained by providing a reduction to the strongly convex non-smooth case . Thus , our task becomes designing better algorithms for the strongly convex non-smooth functions . Rather than using SGD , we let the objective function take convolution with a sphere kernel to make it smooth . We then use the accelerated stochastic approximation algorithm in [ GL12 ] for solving strongly convex stochastic optimization problems . However , this is not enough , as the required noise that needs to be added to the gradients to make the algorithm private is too large to get subquadratic gradient complexity , even if we use the tighter Moments Accountant technique [ ACG+16 ] . We overcome this by increasing the batch size to an appropriate value . Combining these ideas together , we show that the amount of noise we add can be reduced to achieve the optimal excess empirical loss , and we get the gradient complexity of O ( max { N3/2/d1/8 , N2/d } ) . For SCO , we get the gradient complexity of O ( min { N5/4d1/8 , N3/2/d1/8 } ) via application of the iterative localization approach of Feldman et al [ FKT20 ] . The intuition behind iterative localization is using private ERM to solve regularized objective functions which have low sensitivity , iteration by iteration . Each iteration reduces the distance to an approximate minimizer by a multiplicative factor , so after logarithmic number of phases we are done .

Summary: This paper addresses the question of the gradient complexity of differentially private (DP) convex empirical risk minimization (ERM) and stochastic convex optimization (SCO). It provides substantial improvements compared to most work in the area, and it provides the best running times on some ranges of parameters.   The main results in this work are:  1. An Accelerated Stochastic Gradient method with randomized smoothing, and therefore applicable to nonsmooth losses (Lemmata 3.4 and 3.5) 2. A two-way reduction between convex and strongly convex ERM (Lemmata 3.6 and 3.7) 3. A two-way reduction between convex and strongly convex SCO (Lemmata 3.7 and 5.5)  These results are used in combination to show that both DP-ERM and DP-SCO can be solved to optimal rates with subquadratic number of gradient evaluations (with respect to sample size). This resolves an open question from a recent work of Bassily et al. (BFGT20). On the other hand, a concurrent work by Asi et al. (AFKT21) provides subquadratic algorithms in the same setting, but the current submission provides faster rates when $d\leq N^{4/3}$ (this range is interesting, as when $d$ gets closer to $N^2$ the rates for DP-ERM and DP-SCO become vacuous.