Title: Faster Algorithms for User-Level Private Stochastic Convex Optimization
Venue: NeurIPS 2024
Accepted: yes
Keywords: differential privacy

user-level privacy

stochastic convex optimization

private optimization
Abstract: We study private stochastic convex optimization (SCO) under user-level differential privacy (DP) constraints. In this setting, there are $n$ users (e.g., cell phones), each possessing $m$ data items (e.g., text messages), and we need to protect the privacy of each user's entire collection of data items. Existing algorithms for user-level DP SCO are impractical in many large-scale machine learning scenarios because: (i) they make restrictive assumptions on the smoothness parameter of the loss function and require the number of users to grow polynomially with the dimension of the parameter space; or (ii) they are prohibitively slow, requiring at least $(mn)^{3/2}$ gradient computations for smooth losses and $(mn)^3$ computations for non-smooth losses. To address these limitations, we provide novel user-level DP algorithms with state-of-the-art excess risk and runtime guarantees, without stringent assumptions. First, we develop a linear-time algorithm with state-of-the-art excess risk (for a non-trivial linear-time algorithm) under a mild smoothness assumption. Our second algorithm applies to arbitrary smooth losses and achieves optimal excess risk in $\approx (mn)^{9/8}$ gradient computations. Third, for non-smooth loss functions, we obtain optimal excess risk in $n^{11/8} m^{5/4}$ gradient computations. Moreover, our algorithms do not require the number of users to grow polynomially with the dimension.

Faster Algorithms for User-Level Private Stochastic Convex Optimization ∗

Abstract

We study private stochastic convex optimization (SCO) under user-level differential privacy (DP) constraints. In this setting, there are n users (e.g., cell phones), each possessing m data items (e.g., text messages), and we need to protect the privacy of each user's entire collection of data items. Existing algorithms for user-level DP SCO are impractical in many large-scale machine learning scenarios because: (i) they make restrictive assumptions on the smoothness parameter of the loss function and require the number of users to grow polynomially with the dimension of the parameter space; or (ii) they are prohibitively slow, requiring at least ( mn ) 3 / 2 gradient computations for smooth losses and ( mn ) 3 computations for non-smooth losses. To address these limitations, we provide novel user-level DP algorithms with state-of-the-art excess risk and runtime guarantees, without stringent assumptions. First, we develop a linear-time algorithm with state-of-the-art excess risk (for a non-trivial linear-time algorithm) under a mild smoothness assumption. Our second algorithm applies to arbitrary smooth losses and achieves optimal excess risk in ≈ ( mn ) 9 / 8 gradient computations. Third, for non-smooth loss functions, we obtain optimal excess risk in n 11 / 8 m 5 / 4 gradient computations. Moreover, our algorithms do not require the number of users to grow polynomially with the dimension.

1 Introduction

The increasing ubiquity of machine learning (ML) systems in industry and society has sparked serious concerns about the privacy of the personal data used to train these systems. Much work has shown that ML models may violate individuals' privacy by leaking their sensitive training data [SSSS17, LLL + 24a, LLL + 24b]. For instance, large language models (LLMs) are vulnerable to black-box attacks that extract individual training examples [CTW + 21]. Differential privacy (DP) [DMNS06] prevents ML models from leaking their training data.

The classical definition of differential privacyitem-level differential privacy [DMNS06]-is illsuited for many modern applications. Item-level DP ensures that the inclusion or exclusion of any one training example has a negligible impact on the model's outputs. If each person (a.k.a. user) contributes only one piece of training data , then item-level DP provides a strong guarantee that each user's data cannot be leaked. However, in many modern ML applications, such as training LLMs on users' data in federated learning, each user contributes a large number of training examples [XZ24].

In such scenarios, the privacy protection that item-level DP provides for each user is insufficiently weak.

User-level differential privacy is a stronger privacy notion that addresses the above shortcoming of item-level DP. Informally, user-level DP ensures that the inclusion or exclusion of any one user's entire training data ( m samples) has a negligible impact on the model's outputs. Thus, user-level DP provides a strong guarantee that no user's data can be leaked, even when users contribute many training examples.

A fundamental problem in (private) machine learning is stochastic convex optimization (SCO): given a data set D = ( Z 1 , . . . , Z n ) from n i.i.d. users, each possessing m i.i.d. samples from an unknown distribution Z i ∼ P m our goal is to approximately minimize the expected population loss

Here, f : X × Z → R is a loss function (e.g., cross-entropy loss), X ⊂ R d is the parameter domain, and Z is the data universe. We require that the output of the optimization algorithm A : Z mn →X satisfies user-level DP (Definition 1.3). We measure the accuracy of A by its excess (population) risk

Given the practical importance of user-level DP SCO, it is unsurprising that many prior works have studied this problem. The work of [LSA + 21] initiated this line of work, and provided an excess risk lower bound of Ω(1 / √ nm + √ d/ ( εn √ m )) , where ε is the privacy parameter. However, their upper bound was suboptimal and required strong assumptions. The work of [BS23] gave an algorithm that achieves optimal risk for β -smooth losses with β < ( n/ √ md ∧ n 3 / 2 / ( d √ m )) , provided that n ≥ √ d/ε and m ≤ max( √ d, nε 2 / √ d ) . These assumptions are restrictive in large-scale applications with a large number of examples per user m or when the number of model parameters d is large. For example, in deep learning, we often have d ≫ n and an enormous smoothness parameter β ≫ 1 . Moreover, their algorithm requires mn 3 / 2 gradient evaluations, making it slow when the number of users n is large. 2 The work of [GKK + 23] gave another user-level DP algorithm that only requires n ≥ log( d ) /ε , but unfortunately their algorithm does not run in polynomial-time.

To address the deficiencies of previous works on user-level DP SCO, the recent work [AL24] provided an algorithm that achieves optimal excess risk in polynomial-time, while also only requiring n ≥ log( md ) /ε users. Moreover, their algorithm also works for non-smooth losses. The drawback of [AL24] is that it is even slower than the algorithm of [BS23]: for β -smooth losses, their algorithm requires β · ( nm ) 3 / 2 gradient evaluations; for non-smooth losses, their algorithm requires ( nm ) 3 evaluations.

Evidently, the runtime requirements and parameter restrictions of existing algorithms for user-level DP SCO are prohibitive in many important ML applications . Thus, an important question is:

Question 1. Can we develop faster user-level DP algorithms that achieve optimal excess risk without restrictive assumptions ?

Contribution 1. We give a positive answer to Question 1, providing a novel algorithm that achieves optimal excess risk using max { β 1 / 4 ( nm ) 9 / 8 , β 1 / 2 n 1 / 4 m 5 / 4 } gradient computations for β -smooth loss functions, with any β < ∞ (Theorem 3.2). For non-smooth loss functions, our algorithm achieves optimal excess risk using n 11 / 8 m 5 / 4 gradient evaluations for non-smooth loss functions (Theorem 4.1). Our runtime bounds dominate those of all prior works in every applicable parameter regime, by polynomial factors in n, m , and d . Moreover, our results only require n 1 -o (1) ≥ log( d ) /ε users. See Table 1 for a comparison of our results vs. prior works. For example, for non-smooth loss functions, our optimal algorithm is faster than the previous state-of-the-art [AL24] by a multiplicative factor of n 13 / 8 m 7 / 4 . For smooth loss functions, our optimal algorithm is faster than [AL24] by a factor of ( nm ) 3 / 8 β 3 / 4 (in the typical parameter regime when n 7 ≥ m ).

Linear-Time Algorithms The 'holy grail' of DP SCO is a linear-time algorithm with optimal excess risk, which is unimprovable both in terms of runtime and accuracy. In the item-level DP setting, such algorithms are known to exist for smooth loss functions [FKT20, ZTOH22]. [AL24] posed an interesting open question: is there a user-level DP algorithm that achieves optimal excess risk in linear time for smooth functions? For our second contribution, we make progress towards answering this question.

Existing techniques for user-level DP SCO are not well-suited for linear-time algorithms. Indeed, the only prior non-trivial linear-time algorithm is the user-level LDP algorithm of [BS23, Algorithm 5]. 3 Their algorithm can achieve excess risk ≈ 1 / √ nmε + √ d/ ( √ nmε ) . Unfortunately, however, their algorithm requires a very stringent assumption on the smoothness parameter β < √ n 3 / ( md 3 ) , which is unlikely to hold for large-scale ML problems. Further, the result of [BS23] requires the number of users queried in each round to grow polynomially with the dimension d , and it assumes m<d<n . These assumptions severely limit the applicability of [BS23, Algorithm 5] in practical ML scenarios . This leads us to:

Question 2. Can we develop a linear-time userlevel DP algorithm with state-of-the-art excess risk, without restrictive assumptions ?

Contribution 2. Weanswer Question 2 affirmatively in Theorem 2.1: under a very mild requirement on the smoothness parameter β < √ nmd , our novel linear-time algorithm achieves excess risk of ≈ 1 / √ nmε + √ d/ ( √ nmε ) . Moreover, our algorithm does not require the number of users to grow polynomially in the dimension d , and our result holds for any values of m,d, and n . Thus, our algorithm has excess risk matching that of [BS23], but is much more widely applicable.

1.1 Techniques

We develop novel techniques and algorithms to achieve new state-of-the-art results in user-level DP SCO. Before discussing our techniques, let us review the key ideas from prior works that we build on.

The goal of prior works [BS23, AL24] was to develop user-level analogs of DP-SGD [BFTT19], which is optimal in the item-level setting. To do so, they observed that each user i 's gradient 1 m ∑ m j =1 ∇ f ( x, Z i,j ) lies in a ball of radius ≈ 1 / √ m around the population gradient ∇ F ( x ) with high probability, if the data is i.i.d ( Z i ∼ P m ). Consequently, if the data is i.i.d., then replacing one user Z i ∈ D by another user Z ′ i ∈ D ′ will not change the empirical gradient ∇ F D ( x ) by too much: ∥∇ F D ( x ) - ∇ F D ′ ( x ) ∥ ≲ 1 / ( n √ m ) with high probability. Thus, one would hope for a method to privatize ∇ F D ( x ) by adding noise that scales with 1 / ( n √ m ) -rather than 1 /n -which would allow for optimal excess risk. [AL24] devised such a method, which was inspired by FriendlyCore [TCK + 22]. Their method privately detects and removes 'outlier' user gradients, and then adds noise to the average of the 'inlier' user gradients. This outlier-removal procedure ensures privacy with noise scaling with 1 / ( n √ m ) , provided n ≳ 1 /ε . Moreover, when the data is i.i.d., no

outliers will be removed with high probability, leading to a nearly unbiased estimator of the empirical gradient.

Our algorithms apply variations of the outlier-removal idea of [AL24] in novel ways.

Our linear-time Algorithm 1 takes a different approach to outlier removal, compared to prior works. Instead of removing outlier gradients , we aim to detect and remove outlier SGD iterates . 4 The high-level idea of our algorithm is to partition the n users into C ≈ 1 /ε groups, with each group containing ≈ nε users. For each group of users, we run T ≈ mnε steps of online SGD using the samples in this group and obtain the average iterate of each group: { ˜ x j } C j =1 . We then privately identify and remove the outlier iterates from { ˜ x j } C j =1 . In order to successfully do so, we need to argue that if we run online SGD independently on user Z and user Z ′ to obtain ˜ x and ˜ x ′ respectively, then ∥ ˜ x -˜ x ′ ∥ ≲ η √ T with high probability, where η is the SGD step size. We prove such a stability bound in Lemma 2.3, which we hope will be of independent interest. By repeating the above process log( n ) times and using iterative localization [FKT20], we obtain our state-of-the-art linear-time result.

Our second algorithm, Algorithm 3, builds on [AL24] in a different way. In Algorithm 3, we apply an outlier-removal procedure to users' gradients. However, unlike [AL24], we draw random minibatches of users in each iteration and apply outlier-removal to these minibatches. To make this procedure private while also achieving optimal excess risk, we combine AboveThreshold [DR14] with privacy amplification by subsampling [BBG18]. We then develop an accelerated [GL12] user-level DP algorithm that solves a carefully chosen sequence of regularized ERM problems, and applies localization in the spirit of [KLL21, AFKT21]. An obstacle that arises when we try to extend the ERM-based localization framework to the user-level DP setting is getting a tight bound on the variance of our minibatch stochastic gradient estimator that scales with 1 /m . We overcome this obstacle in Lemma 3.5, by appealing to the stability of user-level DP [BS23]. To handle non-smooth loss functions, we apply randomized smoothing to our accelerated algorithm.

1.2 Preliminaries

We consider loss functions f : X × Z → R , where X is a convex parameter domain and Z is a data universe. Let P be an unknown data distribution and F ( x ) := E z ∼ P [ f ( x, z )] be the population loss function. Denote F ∗ := min x ∈X F ( x ) . The SCO problem is min x ∈X F ( x ) . Let ∥ · ∥ denote the ℓ 2 norm. Π X ( u ) := argmin x ∈X ∥ u -x ∥ 2 denotes projection onto X .

Assumptions and Notation. Function g : X → R is L -Lipschitz if | g ( x ) -g ( x ′ ) | ≤ L ∥ x -x ′ ∥ 2 for all x, x ′ ∈ X . Function g : X → R is β -smooth if g is differentiable and has β -Lipschitz gradient: ∥∇ g ( x ) - ∇ g ( x ′ ) ∥ 2 ≤ β ∥ x -x ′ ∥ 2 . Function g : X → R is µ -strongly convex if g ( αx +(1 -α ) x ′ ) ≤ αg ( x ) + (1 -α ) g ( x ′ ) -α (1 -α ) µ 2 ∥ x -x ′ ∥ 2 for all α ∈ [0 , 1] and all x, x ′ ∈ X . If µ = 0 , we say g is convex .

Assumption 1.1. 1. The convex set X is compact with ∥ x -x ′ ∥ ≤ R for all x, x ′ ∈ X .

The loss function f ( · , z ) is L -Lipschitz and convex for all z ∈ Z .

In all of the paper except for Section 4 , we will also assume:

Assumption 1.2. The loss function f ( · , z ) is β -smooth for all z ∈ Z .

Denote a ∧ b := min( a, b ) . For functions f and g of input parameters θ , we write f ≲ g if there is an absolute constant C > 0 such that f ( θ ) ≤ Cg ( θ ) for all permissible values of θ . We use ˜ O to hide logarithmic factors. Write a ≤ poly ( b ) if there exists some large J > 1 for which a ≤ b J .

Differential Privacy.

Definition 1.3 (User-Level Differential Privacy) . Let ε ≥ 0 , δ ∈ [0 , 1) . Randomized algorithm A : Z nm →X is ( ε, δ ) -user-level differentially private (DP) if for any two datasets D = ( Z 1 , . . . , Z n ) and D ′ = ( Z ′ 1 , . . . , Z ′ n ) that differ in one user's data (say Z i = Z ′ i but Z j = Z ′ j for j = i ), we have

̸

̸

for all measurable subsets S ⊂ X .

Definition 1.3 prevents any adversary from learning much more about an individual's data set than if that data had not been used for training. Appendix A contains the necessary background on DP.

1.3 Roadmap

We begin with our state-of-the-art linear-time algorithm in Section 2. In Section 3, we present our error-optimal algorithm with state-of-the-art runtime for smooth loss functions. Section 4 extends our fast optimal algorithm to non-smooth loss functions. We conclude in Section 5 with a discussion and guidance on future research directions stemming from our work.

2 A state-of-the-art linear-time algorithm for user-level DP SCO

In this section, we develop a new algorithm (Algorithm 1) for user-level DP SCO that runs in linear time and has state-of-the-art excess risk, without requiring any impractical assumptions. The algorithm can be seen as a user-level DP variation of the localized phased SGD of [FKT20]: we execute a sequence of SGD trajectories with geometrically decaying step sizes, shrinking both the expected distance to the population minimizer and the privacy noise over a logarithmic number of phases.

In each phase i , we first re-set algorithmic parameters and draw a disjoint set of n i users D i ⊂ D (lines 4-5). We further partition D i into C disjoint subsets { D i,j } C j =1 . For each j ∈ [ C ] , we pool together all of the n i m samples in D i,j and run one-pass online SGD on D i,j with initial point x i -1 given to us from the previous phase. Next, in lines 10-20, we privately detect and remove 'outliers' from { ˜ x i,j } C j =1 . That is, our goal is to privately select a subset S i ⊂ { ˜ x i,j } C j =1 , such that for any two points ˜ x i,j , ˜ x i,j ′ ∈ S i , ∥ ˜ x i,j -˜ x i,j ′ ∥ ≤ τ i = ˜ O ( η i L √ T i ) . This will enable us to add noise scaling with τ i in line 22, rather than with the much larger worst-case sensitivity (that scales linearly with T i ). In order to privately select such a subset S i , we first compute (and privatize) the concentration score for { ˜ x i,j } C j =1 in line 10. A small concentration score indicates that outlier removal is doomed to fail and we must halt the algorithm to avoid breaching the privacy constraint. A large concentration score indicates that { ˜ x i,j } C j =1 is nearly τ i -concentrated and we may proceed with outlier removal in lines 12-15.

Theorem 2.1 (Privacy and utility of Algorithm 1 - Informal) . Let ε ≤ 10 , n 1 -o (1) ≳ log( n/δ ) ε , β ≤ ( L/R ) √ dmnε , and m ≲ poly ( n ) . Then, Algorithm 1 is ( ε, δ ) -user-level DP. Further,

The gradient complexity of Algorithm 1 is ≤ nm .

Remark 2.2 (State-of-the-art excess risk in linear time, without the restrictive assumptions) . Under the assumptions that β < ( Lε 3 /R ) √ n 3 /md 3 and m ≤ d/ε 2 ≤ n , [BS23] gave a linear-time algorithm with similar excess risk to Algorithm 1. However, their assumptions are very restrictive in practice: For example, in the canonical regime n ≈ d , their assumption on β rules out essentially every (non-linear) loss function. By contrast, our result holds even if the smoothness parameter is huge ( β ≈ √ nmd ) and we only require a logarithmic number of users. Thus, our algorithm and result is applicable to many practical ML problems.

To prove that Algorithm 1 is private, we essentially argue that for any phase i , the ℓ 2 -sensitivity of ˜ x i is upper bounded by ˜ O ( τ i /C ) with probability at least 1 -δ/ 2 . The argument goes roughly as follows: First, the Laplace noise added to s i ( τ i ) ensures that s i ( τ i ) is ε/ 4 -user-level DP. Now, it suffices to assume ̂ s i ( τ i ) ≥ 4 C/ 5 , since otherwise the algorithm halts and outputs 0 independently of the data. Next, conditional on the high probability event that the Laplace noise is smaller than ˜ O (1 /ε ) , we know that ̂ s i ( τ i ) ≥ 4 C/ 5 = ⇒ s i ( τ i ) ≥ 2 C/ 3 with high probability by our choice of C . In this case, an argument along the lines of [AL24, Lemma 3.5] shows that ˜ x i has sensitivity bounded by ˜ O ( τ i /C ) with probability at least 1 -δ/ 2 . See Appendix B for the detailed proof.

To prove the excess risk bound in Algorithm 1, the key step is to show that if the data is i.i.d., then with high probability, no points are removed from { ˜ x i,j } C j =1 during the outlier-removal phase

(i.e. |S i | = C ). If |S i | = C holds, then we can use the convergence guarantees of SGD and the localization arguments in [FKT20] to establish the excess risk guarantee. In order to prove that |S i | = C with high probability, we need the following novel stability lemma:

Lemma 2.3. Assume f ( · , z ) is convex, L -Lipschitz, and β -smooth on X with η ≤ 1 /β . Let ˜ x ← SGD ( D,η,T,x 0 ) and ˜ y ← SGD ( D ′ , η, T, x 0 ) be two independent runs of projected SGD, where D,D ′ ∼ P N are i.i.d. Then, with probability at least 1 -ζ , we have

We prove Lemma 2.3 via induction on t , using non-expansiveness of gradient descent on smooth losses [HRS16], subgaussian concentration bounds, and a union bound.

Lemma 2.3 implies that if the data is i.i.d., then the following events hold with high probability: ∥ ˜ x i,j -˜ x i,j ′ ∥ ≤ τ i for all j, j ′ ∈ [ C i ] and hence s i ( τ i ) = C . Further, conditional on s i ( τ i ) = C , we know that ̂ s i ( τ i ) ≥ 4 C/ 5 with high probability, so that the algorithm does not halt. Also, ∥ ˜ x i,j -˜ x i,j ′ ∥ ≤ τ i for all j, j ′ implies p i,j = 1 for all j and hence |S i | = C for all j . The detailed excess risk proof is provided in Appendix B.

Challenges of getting optimal excess risk in linear time: In the item-level DP setting, there are several (nearly) linear time algorithms that achieve optimal excess risk for smooth DP SCO

under mild smoothness conditions, such as snowball SGD [FKT20], phased SGD [FKT20], and phased ERM with output perturbation [ZTOH22]. Extending these approaches into optimal nearly linear-time user-level DP algorithms is challenging. First, the user-level DP implementation of output perturbation in [GKK + 23] is computationally inefficient. Second, snowball SGD relies on privacy amplification by iteration , which does not extend nicely to the user-level DP case due to instability of the outlier-detection procedure in [AL24]. Specifically, since amplification by iteration intermediate only provides DP for the last iterate x T but not the intermediate iterates x t ( t < T ) , the sensitivity of the concentration score function is not O (1) , which impairs DP outlier-detection. A similar instability issue arises if one tries to naively extend phased SGD to be user-level DP by applying [AL24] to user gradients. This issue motivates our Algorithm 1, which extends phased SGD in an alternative way: by applying outlier-detection/removal to the SGD iterates instead of the gradients, we can control the sensitivity of the concentration score and thus prove that our algorithm is DP. However, since the bound in Lemma 2.3 scales polynomially with T (and we believe this dependence on T is necessary), Algorithm 1 adds excessive noise and has suboptimal excess risk. We believe that obtaining optimal risk in linear time will require a fundamentally different user-level DP mean estimation procedure that does not suffer from the instability issue.

3 An optimal algorithm with ≈ ( mn ) 9 / 8 gradient complexity for smooth losses

In this section, we provide an algorithm that achieves optimal excess risk using ≈ ( mn ) 9 / 8 stochastic gradient evaluations. Our Algorithm 3 is inspired by the item-level accelerated phased ERM algorithm of [KLL21]. It applies iterative localization [FKT20] to the user-level DP accelerated minibatch SGD Algorithm 2. Algorithm 2 is a user-level DP variation of the accelerated minibatch SGD of [GL12, GL13].

Our Algorithm 2 applies a DP outlier-removal procedure to the users' gradients in each iteration. We use Above Threshold [DR14] to privatize the concentration scores s ( t ) i and determine whether or not most of the gradients of users in minibatch D t i are 2 τ -close to each other. If ̂ s t i ≥ ̂ ∆ i , indicating that the gradients of users in D t i are nearly 2 τ -concentrated, then we proceed with outlier removal in lines 8-12. We then invoke privacy amplification by subsampling [BBG18] and the advanced composition theorem [KOV15] to privatize the average of the 'inlier' gradients with additive Gaussian noise. By properly choosing algorithmic parameters, we obtain the following results, proved in Appendix C:

Theorem 3.1 (Privacy of Algorithm 3) . Let ε ≤ 10 , q > 0 such that n 1 -q > 100 log(20 nmde ε /δ ) ε (1 -(1 / 2) q ) . Then, Algorithm 3 is ( ε, δ ) -DP.

Theorem 3.2 (Utility & runtime of Algorithm 3 - Informal) . Let ε ≤ 10 and δ < 1 / ( mn ) . Then, Algorithm 3 yields optimal excess risk:

The gradient complexity of this algorithm is upper bounded by

If n = d , ε = 1 , and βR = L then the gradient complexity bound in Theorem 3.2 simplifies to ( mn ) 9 / 8 + n 1 / 4 m 5 / 4 . Typically, n 7 ≥ m , so that the dominant term in this bound is ( mn ) 9 / 8 .

Remark 3.3 (State-of-the-art runtime) . The gradient complexity bound in Theorem 3.2 is superior to the runtime bounds of all existing near-optimal algorithms by polynomial factors in n, m , and d [BS23, GKK + 23, AL24]. Note that while the mn 3 / 2 gradient complexity bound of [BS23] may appear to be better than β 1 / 4 ( nm ) 9 / 8 in certain parameter regimes (e.g. m>n 3 or β ≫ nm ), this is not the case: the result of [BS23] requires m<n and β < √ n/m .

Remark 3.4 (Mild assumptions) . Note that Theorems 3.1 and 3.2 do not require any bound on the smoothness parameter β , and only require the number of users to grow logarithmically: n 1 -o (1) ≥ ˜ Ω(1 /ε ) . Contrast this with the results of previous works (e.g. [BS23]).

Achallenge in proving Theorem 3.2 is getting a tight bound on the variance of the the noisy minibatch stochastic gradients ̂ g t i that are used in Algorithm 2 (lines 12-14). Conditional on S t i = D t i , it is easy to obtain a variance bound of the form E ∥ ̂ g t i -∇ ̂ F i ( x t i ) ∥ 2 ≲ dσ 2 i + L 2 K i , since we are sampling K i users uniformly at random. However, this bound is too weak to obtain Theorem 3.2, since it does not scale with m . To prove Theorem 3.2, we need the following stronger result:

Lemma 3.5 (Variance Bound for Algorithm 2) . Let δ ≤ 1 / ( nm ) , ε ≲ 1 . Denote ˜ F i ( x ) := 1 n i ∑ Z i,j ∈ D i ̂ F ( x, Z i,j ) . Then, conditional on S t i = D t i for all i ∈ [ l ] , t ∈ [ T i ] , we have

for all i ∈ [ l ] , t ∈ [ T i ] , where the expectation is over both the random i.i.d. draw of D = ( Z 1 , . . . , Z n ) ∼ P nm and the randomness in Algorithm 3.

The difficulty in proving Lemma 3.5 comes from the fact that the iterates x t i and the data D are not independent. To overcome this difficulty, we use the stability of user-level DP [BS23] to argue that for all Z ∈ D i , ∇ ̂ F ( x t i -1 , Z ) is ≈ L/ √ m -close to ∇ F ( x t i -1 ) with high probability, since x t i -1 is user-level DP. A detailed proof is given in Appendix C.

Remark 3.6 (Strongly convex losses: Optimal excess risk with state-of-the-art runtime) . If f ( · , z ) is µ -strongly convex, then Algorithm 3 can be combined with the meta-algorithm of [FKT20, Section 5.1] to obtain optimal excess risk

with the same gradient complexity stated in Theorem 3.2. This improves over the previous state-ofthe-art gradient complexity ≈ β ( mn ) 3 / 2 of [AL24].

4 An optimal algorithm with subquadratic gradient complexity for non-smooth losses

In this section, we extend our accelerated algorithm from the previous section to non-smooth loss functions. To accomplish this with minimal computational cost, we apply randomized (convolution) smoothing [YNS12, DBW12] to approximate non-smooth f by a β -smooth ˜ f . We can then apply Algorithm 3 to ˜ f . Since convolution smoothing is by now a standard optimization technique, we defer the details and proof to Appendix D.

Theorem 4.1 (Privacy and utility of smoothed Algorithm 3 for non-smooth loss - informal) . Let ε ≤ 10 , δ < 1 / ( mn ) , and q > 0 such that n 1 -q > 100 log(20 nmde ε /δ ) ε (1 -(1 / 2) q ) . Then, applying Algorithm 3 to the smooth approximation of f yields optimal excess risk:

The gradient complexity of this algorithm is upper bounded by

Remark 4.2 (State-of-the-art gradient complexity) . The only previous polynomial-time algorithm that can achieve optimal excess risk for non-smooth loss functions is due to [AL24]. The algorithm of [AL24] required ( nm ) 3 +( mn ) 2 √ d gradient evaluations. Thus, the gradient complexity of the smoothed version of Algorithm 3 offers a significant improvement over the previous state-of-the-art . For example, if ε = 1 , then our algorithm is faster than the previous state-of-the-art by a multiplicative factor of at least n 13 / 8 m 7 / 4 .

5 Conclusion

In this paper, we developed new user-level DP algorithms with improved runtime and excess risk guarantees for stochastic convex optimization without the restrictive assumptions made in prior

works. Our accelerated Algorithm 3 achieves optimal excess risk for both smooth and non-smooth loss functions, with significantly smaller computational cost than the previous state-of-the-art. Our linear-time Algorithm 1 achieves state-of-the-art excess risk under much milder, more practical assumptions than existing linear-time approaches.

Our work paves the way for several intriguing future research directions. First, the question of whether there exists a linear-time algorithm that can attain the user-level DP lower bound for smooth losses remains open. In light of our improved gradient complexity bound ( ≈ ( nm ) 9 / 8 ), we are now optimistic that the answer to this question is 'yes.' We believe that our novel techniques will be key to the development of an optimal linear-time algorithm. Specifically, utilizing Lemma 2.3 to apply outlier removal to the iterates instead of the gradients appears to be pivotal. Second, the study of user-level DP SCO has been largely limited to approximate ( ε, δ ) -DP. What rates are achievable under the stronger notion of pure ε -user-level DP? Third, it would be useful to develop fast and optimal algorithms that are tailored to federated learning environments [MRTZ18, GLZW24], where only a small number of users may be available to communicate with the server in each iteration. We hope our work inspires and guides further research in this exciting and practically important area.

References

[AFKT21] Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in ℓ 1 geometry. In ICML , 2021.

[AL24] Hilal Asi and Daogao Liu. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In International Conference on Artificial Intelligence and Statistics , pages 4240-4248. PMLR, 2024.

[ALT24] Hilal Asi, Daogao Liu, and Kevin Tian. Private stochastic convex optimization with heavy tails: Near-optimality from simple reductions. arXiv preprint arXiv:2406.02789 , 2024.

[BBG18] Borja Balle, Gilles Barthe, and Marco Gaboardi. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018.

[BFTT19] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Thakurta. Private stochastic convex optimization with optimal rates. In Advances in Neural Information Processing Systems , volume 32, 2019.

[BS23] Raef Bassily and Ziteng Sun. User-level private stochastic convex optimization with optimal rates. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 1838-1851. PMLR, 23-29 Jul 2023.

[CTW + 21] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium , volume 6, pages 2633-2650, 2021.

[DBW12] John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization , 22(2):674-701, 2012.

[DMNS06] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

[DR14] Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy , volume 9. Now Publishers, Inc., 2014.

[FKT20] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing , pages 439-449, 2020.

[GKK + 23] Badih Ghazi, Pritish Kamath, Ravi Kumar, Pasin Manurangsi, Raghu Meka, and Chiyuan Zhang. On user-level private convex optimization. In International Conference on Machine Learning , pages 11283-11299. PMLR, 2023.

[GL12] Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization , 22(4):1469-1492, 2012.

[GL13] Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization, ii: Shrinking procedures and optimal algorithms. SIAM Journal on Optimization , 23(4):2061-2089, 2013.

[GLZW24] Changyu Gao, Andrew Lowy, Xingyu Zhou, and Stephen J Wright. Private heterogeneous federated learning without a trusted server revisited: Error-optimal and communication-efficient algorithms for convex losses. arXiv preprint arXiv:2407.09690 , 2024.

[HRS16] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning , volume 48 of Proceedings of Machine Learning Research , pages 1225-1234, New York, New York, USA, 20-22 Jun 2016. PMLR.

[JNG + 19] Chi Jin, Praneeth Netrapalli, Rong Ge, Sham M Kakade, and Michael I Jordan. A short note on concentration inequalities for random vectors with subgaussian norm. arXiv preprint arXiv:1902.03736 , 2019.

[KLL21] Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth erm and sco in subquadratic steps. Advances in Neural Information Processing Systems , 34:4053-4064, 2021.

[KOV15] Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy, 2015.

[LJCJ17] Lihua Lei, Cheng Ju, Jianbo Chen, and Michael I Jordan. Non-convex finite-sum optimization via scsg methods. In Proceedings of the 31st International Conference on Neural Information Processing Systems , pages 2345-2355, 2017.

[LLL + 24a] Zhuohang Li, Andrew Lowy, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, Bradley Malin, and Ye Wang. Analyzing inference privacy risks through gradients in machine learning. arXiv preprint arXiv:2408.16913 , 2024.

[LLL + 24b] Andrew Lowy, Zhuohang Li, Jing Liu, Toshiaki Koike-Akino, Kieran Parsons, and Ye Wang. Why does differential privacy with large epsilon defend against practical membership inference attacks? arXiv preprint arXiv:2402.09540 , 2024.

[LR22] Andrew Lowy and Meisam Razaviyayn. Private stochastic optimization with large worst-case lipschitz parameter, 2022.

[LSA + 21] Daniel Levy, Ziteng Sun, Kareem Amin, Satyen Kale, Alex Kulesza, Mehryar Mohri, and Ananda Theertha Suresh. Learning with user-level privacy. Advances in Neural Information Processing Systems , 34:12466-12479, 2021.

[LUW24] Andrew Lowy, Jonathan Ullman, and Stephen Wright. How to make the gradients small privately: Improved rates for differentially private non-convex optimization. In Forty-first International Conference on Machine Learning , 2024.

[McS09] Frank D McSherry. Privacy integrated queries: an extensible platform for privacypreserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data , pages 19-30, 2009.

[MRTZ18] Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. In International Conference on Learning Representations (ICLR) , 2018.

[SSSS17] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

[SSSSS09] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In COLT , volume 2, page 5, 2009.

[TCK + 22] Eliad Tsfadia, Edith Cohen, Haim Kaplan, Yishay Mansour, and Uri Stemmer. Friendlycore: Practical differentially private aggregation. In International Conference on Machine Learning , pages 21828-21863. PMLR, 2022.

[Ull17] Jonathan Ullman. CS7880: rigorous approaches to data privacy, 2017.

[XZ24] Zheng Xu and Yanxiang Zhang. Advances in private training for production on-device language models. https://research.google/blog/ advances-in-private-training-for-production-on-device-language-models/ , 2024. Google Research Blog.

[YNS12] Farzad Yousefian, Angelia Nedi´ c, and Uday V Shanbhag. On stochastic gradient and subgradient methods with adaptive steplength sequences. Automatica , 48(1):56-67, 2012.

[ZTOH22] Liang Zhang, Kiran K Thekumparampil, Sewoong Oh, and Niao He. Bring your own algorithm for optimal differentially private stochastic minimax optimization. Advances in Neural Information Processing Systems , 35:35174-35187, 2022.

Appendix

A More Preliminaries

A.1 Tools from Differential Privacy

Additive Noise Mechanisms Additive noise mechanisms privatize a query by adding noise to its output, with the scale of the noise calibrated to the sensitivity of the query.

Definition A.1 (Sensitivity) . Given a function q : Z N → R k and a norm ∥·∥ p on R k , the ℓ p -sensitivity of q is defined as

where the supremum is taken over all pairs of datasets that differ in one user's data.

Definition A.2 (Laplace Distribution) . We say X ∼ Lap( b ) if the density of X is f ( X = x ) = 1 2 b exp( -| x | b ) .

Definition A.3 (Laplace Mechanism) . Let ε > 0 . Given a function q : Z N → R k on R k with ℓ 1 -sensitivity ∆ , the Laplace Mechanism M is defined by

where { Y i } k i =1 are i.i.d., Y i ∼ Lap ( ∆ ε ) .

Lemma A.4 (Privacy of Laplace Mechanism [DR14]) . The Laplace Mechanism is ε -DP.

Definition A.5 (Gaussian Mechanism) . Let ε > 0 , δ ∈ (0 , 1) . Given a function q : Z N → R k with ℓ 2 -sensitivity ∆ , the Gaussian Mechanism M is defined by

where G ∼ N k ( 0 , σ 2 I k ) and σ 2 = 2∆ 2 log(2 /δ ) ε 2 .

Lemma A.6 (Privavcy of Gaussian Mechanism [DR14]) . The Laplace Mechanism is ( ε, δ ) -DP.

Advanced Composition If we adaptively query a data set T times, then the privacy guarantees of the T -th query is still DP and the privacy parameters degrade gracefully:

Lemma A.7 (Advanced Composition Theorem [DR14]) . Let ε ≥ 0 , δ, δ ′ ∈ [0 , 1) . Assume A 1 , · · · , A T , with A t : Z n × X → X , are each ( ε, δ ) -DP ∀ t = 1 , · · · , T . Then, the adaptive composition A ( D ) := A T ( D , A T -1 ( D , A T -2 ( X, · · · ))) is ( ε ′ , T δ + δ ′ ) -DP for

Privacy Amplification by Subsampling

Lemma A.8 ([Ull17]) . Let M : Z M → X be ( ε, δ ) -DP. Let M ′ : Z N → X that first selects a random subsample D ′ of size M from the data set D ∈ Z N and then outputs M ( D ′ ) . Then, M ′ is ( ε ′ , δ ′ ) -DP, where ε ′ = ( e ε -1) M N and δ ′ = δM N .

AboveThreshold: AboveThreshold algorithm [DR14] which is a key tool in differential privacy to identify whether there is a query q i : Z → R in a stream of queries q 1 , . . . , q T that is above a certain threshold ∆ . The AboveThreshold Algorithm 4 has the following guarantees:

Lemma A.9 ([DR14], Theorem 3.24) . Let γ > 0 and α = 8 log(2 T/γ ) ε , k ∈ [ T +1] . AboveThreshold is ( ε, 0) -DP. Moreover, with probability at least 1 -γ , for all t ≤ k , we have:

if a t = ⊤ , then q t ( D ) ≥ ∆ -α ; and

if a t = ⊥ , then q t ( D ) ≤ ∆+ α .

Algorithm 4: AboveThreshold

A.2 SubGaussian and Norm-SubGuassian Random Vectors

Definition A.10. Let ζ > 0 . We say a random vector X is SubGaussian ( SG( ζ ) ) with parameter ζ if E [ e ⟨ v,X -E X ⟩ ] ≤ e ∥ v ∥ 2 ζ 2 / 2 for any v ∈ R d . Random vector X ∈ R d is Norm-SubGaussian with parameter ζ ( nSG( ζ ) ) if P [ ∥ X -E X ∥ ≥ t ] ≤ 2 e -t 2 2 ζ 2 for all t > 0 .

Theorem A.11 (Hoeffding-type inequality for norm-subGaussian, [JNG + 19]) . Let X 1 , · · · , X k ∈ R d be random vectors, and let F i = σ ( x 1 , · , x i ) for i ∈ [ k ] be the corresponding filtration. Suppose for each i ∈ [ k ] , X i | F i -1 is zero-mean nSG( ζ i ) . Then, there exists an absolute constant c > 0 , for any γ > 0 ,

Lemma A.12 ([JNG + 19]) . There exists an absolute constant c , such that if X is nSG( ζ ) , then for any fixed unit vector v ∈ R d , ⟨ v, X ⟩ is cζ norm-SubGaussian.

B Proof of Theorem 2.1

Theorem B.1 (Formal statement of Theorem 2.1) . Suppose n 1 -q ≥ (100 / (1 -1 / 2 q )) log( n/δ ) /ε for some small q > 0 , and m ≤ n J for some large J > 0 . Choose p = J +3 / 2 and η = R/ ( L √ dmnε ) . in Algorithm 1. Then, Algorithm 1 is ( ε, δ ) -user-level DP and achieves excess risk

using nm gradient evaluations, provided β ≤ ( L/R ) √ dmnε .

The gradient complexity is clear by inspection of the algorithm: The number of stochastic gradients computed during the algorithm is

Next, we will prove the privacy statement in Theorem B.1. The following lemma ensures that if the Laplace noise added in Algorithm 1 is sufficiently small and outlier detection succeeds, then the sensitivity of ˜ x i is ˜ O ( τ i /C ) with high probability.

Lemma B.2. [AL24, Slight modification of Lemma 3.5] Let i ∈ [ l ] and ζ > 0 . Suppose D i and D ′ i differ in the data of one user and we are in phase i of Algorithm 1. Let E i be the event that the Laplace noise added to the concentration score s i ( τ i ) for D i has absolute value less than 2 C/ 15 and define E ′ i similarly for data D ′ i . Denote a i := 1 ( ̂ s i ( τ i ) ≥ 4 C/ 5) and a ′ i := 1 ( ̂ s ′ i ( τ i ) ≥ 4 C/ 5) , where ̂ s i ( τ i ) and ̂ s ′ i ( τ i ) are the noisy concentration scores that we get when running phase i of Algorithm 1 on neighboring D i and D ′ i , respectively. Then, conditional on a i = a ′ i and E i ⋂ E ′ i , there is a coupling Γ i over ˜ x i and ˜ x ′ i such that for ( y i , y ′ i ) drawn from Γ i , we have

with probability at least 1 -ζ .

With Lemma B.2 in hand, we proceed to prove that Algorithm 1 is ( ε, δ ) -user-level DP:

̸

Proof of Theorem B.1 - Privacy. Privacy: Since the { D i } l i =1 are disjoint, parallel composition of DP [McS09] implies that it suffices to prove that phase i is ( ε, δ ) -user-level-DP for any fixed i and fixed x i -1 . To that end, let D and D ′ be adjacent datasets differing in the data of one user, say Z i, 1 = Z ′ i, 1 without loss of generality. We will show that the outputs of phase i when run on D and D ′ , x i := x i ( D ) and x ′ i := x i ( D ′ ) respectively, are ( ε, δ ) -indistinguishable.

Let E i be the event that the Laplace noise added in phase i (for data set D ) has absolute value less than 2 C/ 15 and define E ′ i analogously for data set D ′ . Note that E i and E ′ i are independent and P ( E i , E ′ i ) ≥ 1 -δ/ 10 e ε . Denote ζ := δ/ 10 e ε . Let a i := 1 ( ̂ s i ( τ i ) ≥ 4 C/ 5) and a ′ i := 1 ( ̂ s ′ i ( τ i ) ≥ 4 C/ 5) , where ̂ s i ( τ i ) and ̂ s ′ i ( τ i ) are the noisy concentration scores that we get when running phase i of Algorithm 1 on neighboring D i and D ′ i , respectively. By Lemma B.2 and our choice of C , we know that, conditional on E i ⋂ E ′ i and on a i = a ′ i , there exists a coupling Γ over (˜ x i , ˜ x ′ i ) such that for ( y i , y ′ i ) drawn from Γ , we have

with probability at least 1 -ζ .

Note that the sensitivity of s i is less than or equal to 2 . Thus, by the privacy guarantees of the Laplace mechanism (Lemma A.4), we have

for any b ∈ { 0 , 1 } . Further, this implies

By the bound (1), the privacy guarantee of the Gaussian mechanism (Lemma A.6), our choice of σ i , and independence of the Laplace and Gaussian noises that we add in Algorithm 1, we have

for any event O ⊂ X .

Moreover, since the algorithm halts and returns x i = 0 if a i = 0 , we know that

for any event O ⊂ X .

Therefore,

where ( i ) follows from inequalities (3), (4), and (5). Thus, x i is ( ε, δ ) -user-level-DP. This completes the privacy proof.

Next, we turn to the excess risk proof. The following lemma is immediate from [FKT20, Lemma 4.5]:

Lemma B.3. Let η i ≤ 1 /β . Then, for any y ∈ X and all i, j , we have

The next novel lemma is crucial in our analysis:

Lemma B.4 (Re-statement of Lemma 2.3) . Assume f ( · , z ) is convex, L -Lipschitz, and β -smooth on X with η ≤ 1 /β . Let ˜ x ← SGD ( D,η,T,x 0 ) and ˜ y ← SGD ( D ′ , η, T, x 0 ) be two independent runs of projected SGD, where D,D ′ ∼ P N are i.i.d. Then, with probability at least 1 -ζ , we have

Proof. Let g t := ∇ f ( x t , z t ) for z t drawn uniformly from D without replacement and g ′ t := ∇ f ( y t , z ′ t ) for z ′ t drawn uniformly from D ′ without replacement. Let F ( x ) := E z ∼ P [ f ( x, z )] .

We will prove that ∥ x t -y t ∥ ≲ ηL √ T log( dT/ζ ) with probability at least 1 -ζ/t for all t ∈ [ T ] . Note that this implies the lemma. We proceed by induction. The base case, when t = 0 , is trivially true since x 0 = y 0 . For the inductive hypothesis, suppose there is an absolute constant c > 0 such that with probability at least 1 -tζ/T , we have

∀ i ≤ t . Then, for the inductive step, we have by non-expansiveness of projection onto convex sets, that

where ( i ) follows from the non-expansive property of gradient descent on smooth convex function for η ≤ 1 /β [HRS16].

Define a t := -2 η ⟨ x t -η ∇ F ( x t ) -( y t -η ∇ F ( y t )) , g t -∇ F ( x t ) -g ′ t + ∇ F ( y t ) ⟩ . By Inequality (6) and the inductive hypothesis, we obtain

It remains to bound ∑ t i =1 a i . Note that E [ a i | a 1 , · · · , a i -1 ] = 0 , and by Lemma A.12 we know there is a constant c > 0 such that a i is nSG( cηL ∥ x i -y i ∥ ) for all i . Hence by Theorem A.11, we know

Conditional on the event that ∥ x i -y i ∥ ≤ c √ log( dT/ζ ) ηL √ i for all i ≤ t (which happens with probability 1 -tζ/T by the inductive hypothesis), we know

Hence we know

Combining the above pieces completes the inductive step, showing that ∥ x t +1 -y t +1 ∥ ≤ c √ ( t +1)log( dT/ζ ) ηL +2 ηL with probability at least 1 -( t +1) ζ/T . This completes the proof.

By combining Lemmas 2.3 and B.3 with the localization proof technique of [FKT20], we can now prove the excess risk guarantee of Theorem B.1:

Proof of Theorem 2.1 - Excess risk. Excess Risk: First, we will argue that ˜ x i = 1 C ∑ C j =1 ˜ x i,j for all i with high probability ≥ 1 -3 /nm . Lemma 2.3 implies that

for all i ∈ [ l ] , j, j ′ ∈ [ C ] with probability at least 1 -1 /nm . Thus, s i ( τ i ) = C with probability at least 1 -1 /nm . Now, conditional on s i ( τ i ) = C , we have ̂ s i ( τ i ) ≥ 4 C/ 5 for all i with probability at least 1 -1 /nm by Laplace concentration and a union bound. Moroever, if ∥ ˜ x i,j -˜ x i,j ′ ∥ ≤ τ i for all j, j ′ , then p i,j = 1 for all j and hence there are no outliers: S i = { ˜ x i,j } j ∈ [ C ] . By a union bound, we conclude that S i = { ˜ x i,j } j ∈ [ C ] and hence ˜ x i = 1 S i ∑ D i,j ∈S i ˜ x i,j for all i with probability at least ≥ 1 -3 /nm . By the law of total expectation and Lipschitz continuity, it suffices to condition on this high probability good event that ˜ x i = 1 C ∑ C j =1 ˜ x i,j for all i : the total expected excess risk can only be larger than the conditional excess risk by an additive factor of at most 3 LR/nm .

Now, conditional on ˜ x i = 1 S i ∑ D i,j ∈S i ˜ x i,j , Lemma B.3 and Jensen's inequality implies

Next, let x ∗ 0 := x ∗ = argmin x ∈X F ( x ) , and write

Plugging in the prescribed algorithmic parameters completes the excess risk proof.

C Proofs of Results in Section 3

Theorem C.1 (Re-statement of Theorem 3.1) . Let ε ≤ 10 , q > 0 such that n 1 -q > 100 log(20 nmde ε /δ ) ε (1 -(1 / 2) q ) . Then, Algorithm 3 is ( ε, δ ) -DP.

We require the following lemma, which is a direct consequence of [AL24, Lemma 3.5]:

Lemma C.2. Consider Algorithm 2. Let D ′ i and D ′ i be two data sets that differ in the data of one user. Let E i = {| v t i | ≤ K i / 20 ∀ t ∈ [ T i ] ∩ | ξ i | ≤ K i / 20 } . Define E ′ i similarly for independent draws of random Laplace noise: E ′ i = {| ( v t i ) ′ | ≤ K i / 20 ∀ t ∈ [ T i ] ∩ | ξ ′ i | ≤ K i / 20 } . Let a t i = 1 ( ̂ s t i ( D i ) ≥ 4 K i / 5) and b t i = 1 ( ̂ s t i ( D ′ i ) ≥ 4 K i / 5) denote the concentration scores in iteration t . Then, conditional on E i ⋂ E ′ i and conditional on a t i = b t i , there exists a coupling Γ t i over g t i ( D i ) and g t i ( D ′ i ) such that for ( h, h ′ ) drawn from Γ i , we have

with probability at least 1 -ζ .

Proof of Theorem C.1. Note that our assumption on n 1 -q being sufficiently large implies that n i ≳ log( nmd/δ ) ε for all i ∈ [ l ] . By parallel composition [McS09] and post-processing, it suffices to show that { ̂ g t i } T i t =1 satisfies ( ε, δ ) -user-level DP for any i ∈ [ l ] . To that end, fix any i ∈ [ l ] and let D and D ′ be adjacent datasets that differ in the data of one user such that D i = D ′ i . We will show that { ̂ g t i ( D ) } T i t =1 and { ̂ g t i ( D ′ ) } T i t =1 are ( ε, δ ) -indistinguishable, which will imply that Algorithm 3 is ( ε, δ ) -user-level DP.

̸

Let E i = {| v t i | ≤ K i / 20 ∀ t ∈ [ T i ] ∩ | ξ i | ≤ K i / 20 } . Define E ′ i similarly for independent draws of random Laplace noise: E ′ i = {| ( v t i ) ′ | ≤ K i / 20 ∀ t ∈ [ T i ] ∩ | ξ ′ i | ≤ K i / 20 } . Our choice of K i ≥ 500 log( nmde ε /δ ) ε ensures that

by Laplace concentration and a union bound. Let ζ := δ/ (10 T i e ε ) .

Let a t i = 1 ( ̂ s t i ( D ) ≥ 4 K i / 5) and b t i = 1 ( ̂ s t i ( D ′ ) ≥ 4 K i / 5) . Note that if a t i = b t i = 0 , then ̂ g t i ( D ) = 0 = ̂ g t i ( D ′ ) .

Conditional on the good event that a t i = b t i for all t and conditional on E i ⋂ E ′ i , we can bound the ℓ 2 -sensitivity of g t i with high probability, via Lemma C.2 and a union bound:

for all t ∈ [ T i ] with probability at least 1 -T i ζ = 1 -δ/ (10 e ε ) .

Note that { ̂ s t i ( D ) } T i t =1 and { ̂ s t i ( D ′ ) } are ε/ 2 -indistinguishable by the DP guarantees of AboveThreshold in Lemma A.9, since the sensitivity of s t i is upper bounded by 2 . Therefore,

for any v ∈ { 0 , 1 } T i .

Now, by the sensitivity bound (8), the privacy guarantee of the Gaussian mechanism (Lemma A.6) and our choice of σ i , the advanced composition theorem (Lemma A.7), and privacy amplification by subsampling (Lemma A.8), we have

for any event O ⊂ X . Here we also used the fact that K i ≥ n i ε √ T i .

For short-hand, write { a t i } T i t =1 = 1 if a t i = 1 for all t ∈ [ T i ] and { a t i } T i t =1 = 0 if a t i = 0 for some t ∈ [ T i ] ; similarly for b t i . Then since the algorithm halts and returns { ̂ g t i ( D ) } T i t =1 = 0 if { a t i } T i t =1 = 0 , we know that

for any event O ⊂ X .

Combining the above pieces, we have

where ( i ) follows from inequalities (9), (10), and (11). Thus, { ̂ g t i ( D ) } T i t =1 is ( ε, δ ) -user-level-DP, which implies the result.

Theorem C.3 (Formal statement of Theorem 3.2) . Let ε ≤ 10 and δ < 1 / ( mn ) . Then, choosing λ = L R ( 1 √ nm + √ d εn √ m ) and p ≥ 3 q +2 . 5 + log n ( √ m ) in Algorithm 3 yields optimal excess risk:

The gradient complexity of this algorithm is upper bounded by

We will need the following bound on the excess empirical risk of accelerated (noisy) SGD for the proof of Theorem C.3:

Lemma C.4. [GL13, Proposition 7] Let x T be computed by T steps of (multi-stage) Accelerated Minibatch SGD on λ -strongly convex, β -smooth ̂ F with unbiased stochastic gradient estimator g t such that E ∥ g t -∇ ̂ F ( x t ) ∥ 2 ≤ V 2 for all t ∈ [ T ] . Then,

Remark C.5 . As noted in Lemma C.4, we technically need to call a multi-stage implementation of Algorithm 2 in line 7 of Algorithm 3 (as in [GL13]) to get the desired excess risk bound for minimizing the regularized ERM problem in each iteration. For improved readability, we omitted these details in the main body.

Next, we obtain a bound on the variance of the noisy stochastic minibatch gradient estimator ̂ g t i in Algorithm 2, which can then be plugged in for V 2 in Lemma C.4.

Lemma C.6 (Re-statement of Lemma 3.5) . Let δ ≤ 1 / ( nm ) , ε ≲ 1 . Denote ˜ F i ( x ) := 1 n i ∑ Z i,j ∈ D i ̂ F ( x, Z i,j ) . Then, conditional on S t i = D t i for all i ∈ [ l ] , t ∈ [ T i ] , we have

for all i ∈ [ l ] , t ∈ [ T i ] , where the expectation is over both the random i.i.d. draw of D = ( Z 1 , . . . , Z n ) ∼ P nm and the randomness in Algorithm 3.

Proof. By [LJCJ17, Lemma A.1], we know that, conditional on the draw of the data D i and for fixed x t i -1 , the variance of the minibatch estimator of the gradient of the empirical loss is

This also implies

Recall ˜ F i ( x ) := 1 n i m ∑ z ∈ D i f ( x, z ) is the empirical loss of D i .

Now, for any fixed x and any Z ∈ D i , Hoeffding's inequality implies that

with probability at least 1 -γ , where ̂ F ( x, Z ) := 1 m ∑ z ∈ Z f ( x, z ) is user Z 's empirical loss. Thus, by the stability of user-level DP (see [BS23, Theorem 3.4]), for any i ∈ [ l ] , t ∈ [ t ] , we have that

for all Z ∈ D with probability at least 1 -γ ′ = n ( e 2 ε γ + δ ) , since x t i -1 is ( ε, δ ) -user-level DP. To make γ ′ ≲ 1 /m , we choose γ = 1 /mn and use the assumptions that ε ≲ 1 and δ ≤ 1 /mn . Thus, for any fixed i, t we have

for all Z ∈ D with probability at least 1 -1 /m , which implies

by Jensen's inequality, where ̂ F D ( x ) is the empirical loss over the entire data set D .

Plugging the above bounds into (12) then yields

completing the proof.

We are now ready to prove Theorem C.3:

Proof of Theorem C.3. Excess risk: Note that the assumption in the theorem ensures that K i ≤ n i for all i . By similar arguments to those used in [AL24, Proposition 3.7], we will show that with high probability ≥ 1 -2 / ( nm ) , for all i ∈ [ l ] , t ∈ [ T i ] , S t i = D t i and hence g t i is an unbiased estimator of ∇ ̂ F D t i ( x t i ) . To show this, first note that for any γ > 0 and any fixed x ,

with probability at least 1 -γ/K i by Hoeffding's inequality (see [AL24, Lemma 4.3]). Next, we invoke the stability of differential privacy to show that for all t ∈ [ T i ] , ( q t ( Z t i, 1 ) , . . . , q t ( Z t i,K i )) is τ -concentrated (i.e. there exists q ∗ ∈ R d such that ∥ q t ( Z t i,j -q ∗ ∥ ≤ τ ) with probability at least 1 -T i ( e 2 ε γ + δ ) (see [BS23, Theorem 4.3]). By a union bound and the choice of

γ = 1 / [( nm ) 5 / 4 log( ndm ) e 2 ε ] , we have that ( q t ( Z t i, 1 ) , . . . , q t ( Z t i,K i )) is τ -concentrated for all i ∈ [ l ] , t ∈ [ T i ] with probability at least 1 -1 /nm . Now, τ -concentration of ( q t ( Z t i, 1 ) , . . . , q t ( Z t i,K i )) implies s t i ( τ ) = K i . Further, s t i ( τ ) = K i implies ̂ s t i ( τ ) ≥ 4 K i / 5 with probability at least 1 -ζ if K i ≥ 500 log( nm/ζ ) , by Laplace concentration and a union bound. Next, note that τ -concentration of ( q t ( Z t i, 1 ) , . . . , q t ( Z t i,K i )) implies p t i ( Z t i,j ) = 1 for all j ∈ [ K i ] and S t i = D t i . Thus, S t i = D t i for all i, t with probability at least 1 -1 /nm . Setting ζ = 1 /nm and using a union bound shows that with probability at least 1 -2 /nm , we have S t i = D t i and g t i = ∇ ̂ F D t i ( x t i ) for all i, t .

Now, Lemma C.4 implies that, if the outlier-removal procedure in Algorithm 2 leads to an unbiased gradient estimator g t i (line 12) for all t ∈ [ T i = ˜ Θ(1 + √ β/λ i )] , then

where V 2 i = max t ∈ [ t i ] E ∥ ̂ g t i - ∇ ̂ F i ( x t i ) ∥ 2 ≲ dσ 2 i + log( ndm ) L 2 K i m (unconditionally, after taking expectation over the random draw of D ∼ P nm , by Lemma 3.5). We have shown that the event GOOD := { g t i = ∇ ̂ F D t i ( x t i ) for all i ∈ [ l ] , t ∈ [ T i ] } occurs with probability at least 1 -2 /nm . We will condition on GOOD for the rest of the proof: note that the Lipschitz assumption implies that the total (unconditional) excess risk will only by larger than the conditional (on GOOD) excess risk by an additive factor of at most 2 LR/ √ nm .

By stability of regularized ERM (see [SSSSS09]), we have

for all i , where x ∗ i := argmin x ̂ F i ( x ) . By strong convexity and (14), we have

Thus,

Now, letting x ∗ 0 := x ∗ = argmin x ∈X F ( x ) and hiding logarithmic factors, we have:

where the first inequality used (15) and Lipschitz continuity, the second inequality used (17).

Note that K i T i ≥ n i . Further, our choice of sufficiently large p makes λ l large enough that L √ dτ √ T l log(1 /δ ) λ l εn l ≤ LR √ d εn √ m . Therefore, upper bounding the sum by it's corresponding geometric series gives us

Plugging in λ completes the excess risk proof.

Gradient Complexity: The gradient complexity is ∑ l i =1 T i K i m . Plugging in the prescribed choices of T i and K i completes the proof.

D Details on the non-smooth algorithm and the proof of Theorem 4.1

For any loss function f ( · , z ) , we define the convolution function f r ( · , z ) := f ( · , z ) ∗ n r where n r is the uniform density in the ℓ 2 ball of radius r centered at the origin in R d . Specifically, n r ( y ) = Γ( d 2 +1) π d 2 r d for ∥ y ∥ ≤ r , and n r ( y ) = 0 otherwise. For simplicity, we omit the dependence on z in the following Lemma:

Lemma D.1 (Randomized Smoothing, [YNS12, DBW12]) . For any r > 0 , let X r := X + { x ∈ R d : ∥ x ∥ ≤ r } . If f is convex and L -Lipschitz over X r , then the convolution function f r has the following properties:

f r ( x ) ≤ f ( x ) ≤ f r ( x ) + Lr , for all x ∈ X .

f r is L -Lipschitz and convex.

f r is L √ d r -smooth.

For random variables y ∼ n r , we have E y [ ∇ f ( x + y )] = ∇ f r ( x ) .

The following lemma can be easily seen from the proofs of Theorems 3.1 and 3.2:

Lemma D.2 (Privacy and utility of Algorithm 3 for general K i , T i ) . Let ε ≤ 10 , q > 0 such that n 1 -q > 100 log(20 nmde ε /δ ) ε (1 -(1 / 2) q ) .

If K i ≳ n i ε √ T i + log( nmde ε /δ ) ε , then Algorithm 3 is ( ε, δ ) -user-level DP.

If T i K i ≥ n i and T i ≳ (1 + √ β/λ i ) log( ndm ) for all i , then Algorithm 3 achieves optimal excess risk.

Theorem D.3 (Formal statement of Theorem 4.1) . Let ε ≤ 10 , δ < 1 / ( mn ) , and q > 0 such that n 1 -q > 100 log(20 nmde ε /δ ) ε (1 -(1 / 2) q ) . Suppose that for any z , f ( , z ) is convex and L -Lipschitz over X r for X r := X + { x ∈ R d : ∥ x ∥ ≤ r } where r = √ d εn √ m R . Then, running Algorithm 3 with functions { f r ( x ; z ) } z ∈D yields optimal excess risk:

The gradient complexity of this algorithm is upper bounded by

Proof. By Lemma D.1 and our choice of r , we have | f r ( x, z ) -f ( x, z ) | ≤ Lr = O ( LR √ d εn √ m ) . Set λ = 1 √ mn . Then we know that

Further, F r is β -smooth for β ≤ L R εn √ m . Set T i = (1 + √ β/λ i ) log( ndm ) = 1 + n 3 / 4 i m 1 / 2 ε 1 / 2 log( ndm ) and K i = n i ε √ T i + log( nmde ε /δ ) ε . Then Lemma D.2 implies that Algorithm 3 is ( ε, δ ) user-level DP, and yields the excess risk bound

as desired. The number of gradient evaluations is

This completes the proof.

E Limitations

Our work weakens the assumptions on the smoothness parameter and the number of users that are needed for user-level DP SCO. Nevertheless, our results still require certain assumptions that may not always hold in practice. For example, we assume convexity of the loss function. In deep learning scenarios, this assumption does not hold and our algorithms should not be used. Thus, user-level DP non-convex optimization is an important direction for future research [LUW24]. Furthermore, the assumption that the loss function is convex and uniformly Lipschitz continuous may not hold in certain applications, motivating the future study of user-level DP stochastic optimization with heavy tails [LR22, ALT24].

Our algorithms are also faster than the previous state-of-the-art, including a linear-time Algorithm 1 with state-of-the-art excess risk. However, our error-optimal accelerated Algorithm 3 runs in superlinear time. Thus, in certain applications where a linear-time algorithm is needed due to strict computational constraints, Algorithm 1 should be used instead.

F Broader Impacts

Our work on differentially private optimization for machine learning advances the field of privacypreserving ML by developing techniques that protect the privacy of individuals (users) who contribute data. The significance of privacy cannot be overstated, as it is a fundamental right enshrined in various legal systems worldwide. However, the implications of our work extend beyond its intended benefits, and it is essential to consider both potential positive and negative impacts.

Positive Impacts:

Enhanced Privacy Protections: By incorporating differential privacy into machine learning models, we can provide strong privacy guarantees for individuals, mitigating the risk of personal data being exposed or misused.

Ethical Data Utilization: DP ML enables organizations to leverage data while adhering to ethical standards and privacy regulations, fostering trust among users and stakeholders.

Broad Applications: The techniques we develop can be applied across diverse domains, including healthcare, finance, and social sciences, where sensitive data is prevalent. This broad applicability can drive innovations while maintaining privacy.

Educational Advancement: Our research contributes to the growing body of knowledge in privacy-preserving technologies, serving as a valuable resource for future studies and fostering an environment of continuous improvement in privacy practices.

Potential Negative Impacts:

Misuse by Corporations and Governments: There is a risk that our algorithms could be exploited by entities to justify the unauthorized collection of personal data under the guise of privacy compliance. Vigilant oversight and clear regulatory frameworks are necessary to prevent such abuses.

Decreased Model Accuracy: While DP ML provides privacy benefits, it can also lead to reduced model accuracy compared to non-private models. This trade-off might have adverse consequences, such as less accurate medical diagnoses or flawed economic forecasts. For example, an overly optimistic prediction of environmental impacts due to lower accuracy could be misused to weaken environmental protections.

While recognizing the potential for misuse and the trade-offs involved, we firmly believe that the advancement and dissemination of differentially private machine learning algorithms offer a net benefit to society. By addressing privacy concerns head-on and advocating for responsible use, we aim to contribute positively to the field of machine learning and uphold the fundamental right to privacy. Through ongoing research, collaboration, and education, we strive to enhance both the capabilities and ethical foundations of machine learning technologies.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We state our results in Sections 2-4 and prove them in the Appendix.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions are provided in theorem statements and proofs are provided in the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms to the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is a theoretical paper without experiments.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revisits the user-level private stochastic convex optimization (SCO) problem, where each user can possess multiple data points. The contributions of this paper are in two aspects:\n\n1. They propose a linear-time algorithm that attains the same excess risk as the prior linear-time algorithm, but with a milder assumption on the smooth parameter $\\beta$ and does not require the number of users $n$ to depend on dimension $d$, making the algorithm more applicable in practice.\n2. They propose an algorithm that achieves optimal excess risk with improved gradient complexity compared to previous work in both smooth and non-smooth settings.\n\nThe key insight behind the linear-time algorithm is to remove outlier SGD iterates instead of gradients. To achieve this, they prove a stability argument (Lemma 2.3), the utility then follows by the localization arguments in [FKT20].\n\nThe optimal algorithm is inspired by the item-level accelerated phased ERM algorithm of [KLL21], which is then applied to a user-level gradient outlier-removal procedure. The non-smooth case is done by a standard randomized smoothing technique.\n\nSTRENGTHS:\n1. DP SCO is a fundamental problem in private machine learning. In many applications, each user can hold more than one data point. Studying user-level DP SCO is of great importance.\n2. The proposed algorithms improve the previous algorithms in various settings, relaxing the assumptions and providing a better run time.\n3. The writing of this paper is clear.\n\nWEAKNESSES:\nThe linear-time algorithm only has a suboptimal excess risk, while the optimal algorithm has a gradient complexity higher than linear.\n\nQUESTIONS:\nIn the conclusion, it is stated that whether a linear-time algorithm with optimal risk exists for smooth losses is an open question. What about non-smooth losses? If one has a linear-time algorithm for smooth losses, does the randomized smoothing give a linear-time algorithm for non-smooth losses?\n\nLIMITATIONS:\nSuccessfully discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new mechanisms for private SCO with user-level DP.  Their approach extends prior work on this topic, and reduces the gradient complexity while attaining optimal excess risk.  Three algorithms are proposed (1) a linear time algorithm that pushes state of the art (but does not achieve optimal excess risk), (2) a (m n)^{9/8} time algorithm that does achieve optimal excess risk (improving over prior work which required (m n)^(3 / 2) gradient evaluation, and (3) an optimal algorithm for non-smooth losses that requires < (m n)^2 gradient computations.  Extensive theoretical results give expressions for the expected excess risk under various assumptions for each mechanism.\n\nSTRENGTHS:\n* The paper improves state of the art on an important problem, reducing gradient complexity while also relaxing assumptions. \n* The theoretical results are strong, and clearly stated, presented, and discussed.\n* The work is well-motivated and the authors make a compelling story.\n\nWEAKNESSES:\n* One of main claims is not supported by the evidence given.  The work is motivated by \"modern ML applications, such as training LLMs...\" and it claims that \"our algorithm and result is applicable to many practical ML problems.\"  However, looking at Algorithms 1-2 it is not clear that this statement is true.  I believe modern LLMs are pretty big (> 1B parameters) and typically saturate GPU memory, and Alg 1 requires storing C different settings of those parameters to compute the concentration scores -- is that feasible?  From what I can tell, for reasonable values of n, m, epsilon, and delta, C ~ 500.  \n\n* The proposed algorithms seem quite complicated.  Is that complexity necessary?  The paper is missing a description + summarization of simpler approaches and their limitations.  E.g., what about plain-old DP-SGD?  What happens when you try applying optimal item-level DP mechanisms to the user-level setting (with appropriate modifications).  What are those modifications (I can think of a few natural ways to extend them).  It would be nice if your mechanism reduced to a known optimal item-level DP mechanism when m=1.\n\nQUESTIONS:\n* In what ways does your algorithm exploit the user-structure of the problem (the fact that each user holds m items)?\n\n* The noise multiplier seems humongous, am I missing something?  It's hard to believe this algorithm doesn't diverge --- do you have any empirical evidence (even on synthetic data satisfying all your required assumptions) to confirm / augment your theorem statements?  Note I have not checked your proofs at all.\n\nepsilon = 10\ndelta = 1e-6\nstep size = 1\nn = 10 million\nm = 100\nd = 1 billion\nL = 1\n--> C = 475\n--> T = ? >= 100\n--> tau = 400,000\n--> sigma = 250,000\n\nLIMITATIONS:\nLimitations are discussed in the appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

comments: SUMMARY:
This paper considers stochastic convex optimization under user-level differential privacy.

Algorithm 1 achieves the previous state-of-the-art excess risk of the linear-time user-level DP algorithm with milder assumptions. The algorithm is based on item-level DP-SGD algorithms. Previous paper AL24 shows that when the data of one user changes, the empirical gradient will not change too much, hence they remove outlier gradients to ensure privacy. Instead of removing outlier gradients, this paper removes outlier SGD iterates. This technique relies on a novel stability bound of SGD iterates proved in this paper.

Algorithm 3 improves the run-time for both beta-smooth and non-smooth loss functions. It applies outlier-gradient removal to random mini-batches and implements previous subroutines to reach a good performance.

STRENGTHS:
This paper makes concrete improvements from the previous work. Algorithm 3 has a faster run-time under milder assumptions while achieving optimal excess risk and user-level DP. Algorithm 1 improves the previous state-of-the-art excess risk of the linear-time user-level DP algorithm. Novel techniques and analysis are put forward which may be of independent interest.

WEAKNESSES:
Section 1.1 may contain more information about the intuition of the novel techniques (e.g., what may be the key reason that outlier-iterate removal is better than outlier-gradient removal?).
Contribution 2 of this paper lacks a comparison table to previous results.

QUESTIONS:
The questions are asked in the 'Weaknesses' part.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes new algorithms for stochastic convex optimization under user level differential privacy. This paper improves the computation complexity. The first algorithm achieves linear time complexity with suboptimal risk bound (the risk is SOTA among all linear time algorithms). The second algorithm achieves optimal risk with suboptimal time complexity (the time complexity is SOTA among all risk-optimal algorithms).\n\nSTRENGTHS:\nThe authors have addressed an important problem. The method improves over Bassily et al. ICML 2023 and Liu et al. AISTATS 2024 on the time complexity. Moreover, this paper removes the assumption n\\geq \\sqrt{d}. I think that this paper makes a solid and interesting contribution.\n\nWEAKNESSES:\n1. I feel that the second algorithm (Section 3 in the paper) is hard to follow. I do not understand the main ideas of the design. \n\n2. Based on my own understanding, it seems that this paper requires $\\epsilon<1$, since it divides users into $1/\\epsilon$ groups. When $\\epsilon>1$, there is only one group. Following the remainder of this paper, I feel that the bound with $\\epsilon>1$ may not be optimal. Please correct me if it is wrong.\n\n3.In item-level case, optimal rates with linear time has been achieved in\n\nFeldman et al. Private stochastic convex optimization: optimal rates in linear time. STOC 2020.\n\nAlthough this paper has cited the above paper, it would be better if the authors can provide more explanations on why achieving linear time with optimal rik is hard for user-level cases. In other words, why does the methods in Feldman et al. can not be simply extended to user-level case. My feeling is that using something like two-stage approach (such as Levy et al. Learning with user level privacy. NeurIPS 2021), the item-level methods can be converted to user-level ones with optimal rate.\n\n\nMinor issue: The statements of three algorithms are not with the same font. The algorithm 2 is different with algorithm 1 and 3.\n\nIn general, I think that the paper indeed makes a solid contribution. However, the writing needs to be further improved and details need to be further polished.\n\nQUESTIONS:\n1. Can the authors provide more intuitive explanations of the design of algorithms, especially the second one? \n\n2. I wonder what will be the risk bound for strong convex loss functions. The case with strong convex loss function has been discussed in \n\nKamath et al. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. ICML 2022.\n\nThis holds for item-level case. I wonder if one can derive the user-level counterparts.\n\n3. Can the authors please provide more discussions on why it is hard to achieve linear time with optimal risk?\n\nLIMITATIONS:\nThe authors have addressed limitations well in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}