Private Non-smooth ERM and SCO in Subquadratic Steps

3/2 d1/8 + N 2 d ) gradient queries , which is achieved with the help of subsampling and smoothing the function via convolution . Combining this result with the iterative localization technique of Feldman et al . [ FKT20 ] , we achieve the optimal excess population loss for the SCO problem with O ( min { N5/4d1/8 , N3/2 d1/8 } ) gra- dient queries . Our work makes progress towards resolving a question raised by Bassily et al . [ BFGT20 ] , giving first algorithms for private SCO with subquadratic steps . In a concurrent work , Asi et al . [ AFKT21 ] gave other algorithms for private ERM and SCO with subquadratic steps . 1 Introduction . Privacy has become an important consideration for learning algorithms dealing with sensitive data . Over the past decade , differential privacy , introduced in the seminal work of [ DMNS06 ] , has established itself as the defacto notion of privacy for machine learning problems . In this paper , we revisit Empirical Risk Minimization ( ERM ) and Stochastic Convex Optimization ( SCO ) problems , two fundamental problems in statistics and machine learning , in differential privacy setting . In the ERM problem , we are given a family of convex functions { f ( · , x ) } x∈Ξ over a bounded closed convex set K ⊂ Rd of diameter D , a data set S = { x1 , · · · , xN } drawn from some unknown distribution P over the universe Ξ , and the objective is to minimize F̂ ( ω ) : = 1 N ∑ xi∈S f ( ω , xi ) over ω ∈ K , The excess empirical loss with respect to a solution ω is defined by F̂ ( ω ) − F̂ ∗ , where F̂ ∗ = minω∈K F̂ ( ω ) . A closely related but more general problem is SCO , where the objective is to minimize F ( ω ) : = E x∼P f ( ω , x ) over ω ∈ K , We refer F ( ω ) − F ∗ as the excess population loss , where F ( ω ) = Ex∼P f ( ω , x ) and F ∗ = minω∈K F ( ω ) . Differentially private convex optimization has been studied extensively for over a decade now [ CM08 , RBHT09 , CMS11 , KST12 , JT14 , TTZ14 , BST14 , TTZ15 , KJ16 , WLK+17 , FTS17 , ∗Algorithms Group , Microsoft Research ( MSR ) Redmond . jakul @ microsoft.com †University of Washington and Microsoft Research . Supported by NSF awards CCF-1749609 , DMS1839116 , DMS-2023166 , Microsoft Research Faculty Fellowship , Sloan Research Fellowship , Packard Fellowships.yintat @ uw.edu ‡University of Washington . Part of the work was done while visiting Shanghai Qi Zhi Institute.dgliu @ uw.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ZZMW17 , WYX17 , INS+19 ] . Both DP-ERM and DP-SCO for smooth convex functions are well understood in the sense that we know ( near ) linear time algorithms that achieve optimal loss ; we refer the readers to [ WYX17 , FKT20 ] for more details . However , for the more general non-smooth convex loss functions our understanding is not yet complete , which is the focus of this paper . Our algorithms for DP-SCO build upon our improvements for the DP-ERM problem , and hence we begin with DP-ERM problem . A summary of the state-of-the-art results of DP-ERM and our contributions for the non-smooth convex loss functions is given in Table 1 . We will discuss the concurrent work [ AFKT21 ] separately at the end of the introduction , and the following discussion is only limited to the previous work . Nε } ) and of Strongly Convex function is Ω ( min { G2µ , G 2d µN2ε2 } ) [ BST14 ] . [ KST12 ] used objective perturbation method to design a DP-algorithm with O ( GD √ d log ( 1/δ ) √ Nε ) excess empirical risk . This result was improved significantly by [ BST14 ] , who first showed a lower bound of Ω ( min { GD , GD √ d Nε } ) on the excess empirical risk for DP-ERM . Further , they gave an algorithm with excess empirical risk O ( GD log 3 2 ( N/δ ) √ d log ( 1/δ ) Nε ) , which is sub-optimal by a factor of log 3 2 ( N/δ ) . Their algorithm is based on a modification of SGD by adding Gaussian noise to the gradients to make it differentially private . The privacy analysis proceeds via amplification by sampling and the strong composition theorem . Roughly speaking , the logarithmic blowup in the excess empirical risk is due to two reasons : 1 ) The strong composition theorem requires that at each step one needs to add Gaussian noise with a larger variance ; 2 ) They used sub-optimal convergence rate O ( log T/ √ T ) for T -step SGD . However , getting the optimal bounds with small gradient complexity for non-smooth case turns out to be a more difficult problem . This was noted by [ WYX17 ] , who raised it as an important open problem . This question was answered in [ BFTT19 ] , who gave an algorithm with almost optimal excess empirical risk . To achieve this , [ BFTT19 ] first consider the smooth case , and give an improved privacy analysis via the Moments Accountant technique proposed by [ ACG+16 ] . They extend their result to non-smooth case by applying Moreau-Yosida envelope technique ( a.k.a . Moreau envelope smoothing ) [ Nes05 ] to make the function smooth . However , this technique is computationally inefficient and leads to O ( N4.5 ) -gradient computations for the whole algorithm . This limitation was overcome in a recent work of [ BFGT20 ] who gave the optimal excess empirical risk guarantee with O ( N2 ) -gradient computations . The privacy analysis of this result also used Moments Accountant method , and they used the standard online-to-batch conversion technique [ CBCG04 ] to prove the high-probability and expectation bound on the excess empirical error of SGD . As we can see from Table 1 , all the previously known results ( except the concurrent work [ AFKT21 ] ) achieving near optimal excess empirical risk bounds require at least O ( N2 ) -gradient computations . As Table 2 shows , a similar situation arises in Stochastic Convex Optimization ( SCO ) , which is a closely related problem compared to ERM . Many results for SCO [ BST14 , BFTT19 , BFGT20 ] are directly based on ERM ; that is , solving the ERM and analyzing the generalization error . The first non-trivial result for general convex loss functions achieving excess population loss of O ( GD ( d 1/4 √ N + √ d Nε ) ) was given by [ BST14 ] , who showed the result by first solving the ERM prob- lem and bounding the generalization error . They used the result on universal convergence directly , namely , bounding supω∈K E [ F ( ω ) −F̂ ( ω ) ] . But this method has its limitations ; For example , [ Fel16 ] showed that lower bound of universal convergence is Ω ( √ d/N ) for some ( not necessarily convex ) loss functions . Later , [ BFTT19 ] , [ FKT20 ] and [ BFGT20 ] obtained near optimal excess population loss with significantly better running times ( gradient complexity ) . The privacy analysis in these papers relied on recent advances in the privacy techniques such as the Moments Accountant method [ ACG+16 ] , Rényi differential privacy ( RDP ) [ Mir17 ] and the Privacy Amplification by Iteration [ FMTT18 ] and other fast stochastic convex optimization algorithms such as [ JNN19 ] . The excess population loss bound in most of these works followed by solving a ( phased ) convex ( regularized ) ERM problem and then appealing to the uniform stability property [ HRS16 ] or the iterative localization approach [ FKT20 ] to do the generalization error analysis . N2ε2 ) ) [ BST14 ] . Despite these impressive improvements , as the Table 2 suggests , the previous algorithms that achieve the optimal excess population loss still require O ( N2 ) -gradient computations . Indeed , [ BFGT20 ] write that “ Proving that quadratic running time is necessary for general non-smooth DP-SCO is a very interesting open problem ... ” . Understanding if the lower bound is the right answer to the above questions or one can design algorithms with subquadratic gradient complexity is the main motivation that spurred our work . 1.1 Our Contributions . The first contribution of this paper is to show that we can obtain subquadratic gradient complexity bound for ERM when the dimension is super constant . In particular , for the important regime of over-parameterization ( d ≥ N ) , we achieve a bound of O ( N1+3/8 ) . Let Kr = { y | y = ω + z , ω ∈ K , z ∈ Rd , ∥z∥r ≤ r } . We now state the result formally . Theorem 1.1 ( DP-ERM ) . Suppose K ⊂ Rd is a closed convex set and { f ( · , x ) } x∈Ξ is a family of G-Lipschitz and convex functions over Kr , where r = D √ d log ( 1/δ ) εN 4 . For ε , δ ≤ 1/2 , given any sample set S consists of N samples from Ξ and arbitrary initial point ω0 ∈ K , we have a ( ε , δ ) -differentially private algorithm A which takes O ( εN 3 2 d1/8 log1/4 ( 1/δ ) + ε2N2 d log ( 1/δ ) ) gradient queries and outputs ωT such that E [ F̂ ( ωT ) − F̂ ∗ ] = O ( GD √ d log ( 1/δ ) εN ) , where D = ∥ω∗ − ω0∥2 , F̂ ( ω ) = 1N ∑ xi∈S f ( ω , xi ) , F̂ ∗ = minω∈K F̂ ( ω ) , and the expectation is taken over the randomness of the algorithm . 4We only need consider the non-trivial case when √ d log ( 1/δ ) εN ≤ 1 , or any feasible solution is good enough . This means that r = O ( D ) , which is a mild assumption . Moreover , if { f ( · , x ) } x∈Ξ is also µ-strongly convex over Kr , we can meet the same gradient query complexity and get a solution ωT such that E [ F̂ ( ωT ) − F̂ ∗ ] = O ( G2d log ( 1/δ ) µε2N2 ) . The main contribution of the paper is to use the above result to obtain a better gradient complexity for the SCO problem , answering the open problem in [ BFGT20 ] . Combining our private ERM algorithm with the iterative localization technique [ FKT20 ] , we give the first algorithm achieving optimal error with strictly sub-quadratic steps for all dimensions . Theorem 1.2 ( DP-SCO ) . Suppose ε , δ ≤ 12 and sample set S consists of N samples drawn i.i.d from a distribution P over Ξ . Suppose { f ( · , x ) } x∈Ξ is convex and G-Lipschitz with respect to ℓ2 norm and convex over Kr , where r = D √ d log ( 1/δ ) εN , there is an ( ε , δ ) -differentially private algorithm taking O ( N +min { √εN5/4d1/8 , εN 3/2 d1/8 log1/4 ( 1/δ ) } ) gradient queries to get a solution ωT such that E [ F ( ωT ) − F ( ω∗ ) ] = O ( GD ( 1√N + √ d log ( 1/δ ) Nε ) . Moreover , if { f ( · , x ) } x∈Ξ is also µ-strongly convex over Kr , we can meet the same gradient query complexity and get a solution ωT such that E [ F ( ωT ) − F ( ω∗ ) ] = O ( G2 µ ( d log ( 1/δ ) ε2N2 + 1 N ) ) . Finally , we note that with straightforward modifications , our results can also capture the regularized ERM and SCO , where there is one more simple ( and convex ) function h ( ω ) added to the objective function and the objective function takes the form 1N ∑ xi∈S f ( ω , xi ) + h ( ω ) , which show up often in the previous work such as [ RBHT09 , KST12 , WYX17 , INS+19 ] . 1.2 Our Techniques Most of the previous works [ BST14 , BFTT19 , BFGT20 ] that achieve near optimal bounds for ERM and SCO are based on adaptations of SGD to make it differentially private . The information theoretic lower bound of Ω ( 1/ √ T ) for T -step SGD may be one of the important reasons why we can not get subquadratic gradient complexity for non-smooth convex ERM easily . Consider the algorithm in [ BFGT20 ] as an example . It needs to add Gaussian noise v ∼ N ( 0 , σ2Id×d ) with σ2 = G 2 log ( 1/δ ) ε2 to each gradient . By a standard analysis of SGD , we can only show an excess empirical risk of Θ ( D √ dσ2√ T ) , which requires us to set T = Ω ( N2 ) to get ideal bound , thus hitting the quadratic barrier . We deviate from the above approaches for designing private algorithms for non-smooth functions . First notice that the gradient complexity O ( εN 3 2 d1/8 log1/4 ( 1/δ ) + ε 2N2 d log ( 1/δ ) ) in Theorem 1.1 is the same for both strongly convex and general non-smooth functions ; same holds for DP-SCO . This is not a coincidence ; We prove that if we can achieve optimal empirical risk ( population loss ) for one case , then we can achieve optimal empirical risk ( population loss ) for another with the same privacy guarantee and gradient complexity . The Figure 1 illustrates the relationship among these different problems . Our result for the general convex non-smooth case is obtained by providing a reduction to the strongly convex non-smooth case . Thus , our task becomes designing better algorithms for the strongly convex non-smooth functions . Rather than using SGD , we let the objective function take convolution with a sphere kernel to make it smooth . We then use the accelerated stochastic approximation algorithm in [ GL12 ] for solving strongly convex stochastic optimization problems . However , this is not enough , as the required noise that needs to be added to the gradients to make the algorithm private is too large to get subquadratic gradient complexity , even if we use the tighter Moments Accountant technique [ ACG+16 ] . We overcome this by increasing the batch size to an appropriate value . Combining these ideas together , we show that the amount of noise we add can be reduced to achieve the optimal excess empirical loss , and we get the gradient complexity of O ( max { N3/2/d1/8 , N2/d } ) . For SCO , we get the gradient complexity of O ( min { N5/4d1/8 , N3/2/d1/8 } ) via application of the iterative localization approach of Feldman et al [ FKT20 ] . The intuition behind iterative localization is using private ERM to solve regularized objective functions which have low sensitivity , iteration by iteration . Each iteration reduces the distance to an approximate minimizer by a multiplicative factor , so after logarithmic number of phases we are done .

Summary: The paper studies the empirical risk minimization and stochastic convex optimization problems under differential privacy. Previous works give private algorithms for ERM that achieve nearly-optimal utility in the convex and Lipschitz as well as the strongly convex settings but have gradient complexity that is at least quadratic in the number of samples. Thus a natural open question that has been raised in prior works is whether one can obtain an algorithm with a sub-quadratic gradient complexity. The current paper answers this question affirmatively in the regime where the dimension is a super-constant function of the number of samples. Using techniques developed in prior work, the paper extends the ERM algorithm to the more general SCO problem.