Title: Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions
Venue: NeurIPS 2024
Accepted: yes
Keywords: Stochastic Convex Optimization

Heavy-Tailed Distributions

Differential Privacy
Abstract: We study the problem of differentially private stochastic convex optimization (DP-SCO) with heavy-tailed gradients, where we assume a $k^{\text{th}}$-moment bound on the Lipschitz constants of sample functions, rather than a uniform bound. We propose a new reduction-based approach that enables us to obtain the first optimal rates (up to logarithmic factors) in the heavy-tailed setting, achieving error $G_2 \cdot \frac 1 {\sqrt n} + G_k \cdot (\frac{\sqrt d}{n\epsilon})^{1 - \frac 1 k}$ under $(\epsilon, \delta)$-approximate differential privacy, up to a mild $\textup{polylog}(\frac{1}{\delta})$ factor, where $G_2^2$ and $G_k^k$ are the $2^{\text{nd}}$ and $k^{\text{th}}$ moment bounds on sample Lipschitz constants, nearly-matching a lower bound of [LR23].
We then give a suite of private algorithms for DP-SCO with heavy-tailed gradients improving our basic result under additional assumptions, including an optimal algorithm under a known-Lipschitz constant assumption, a near-linear time algorithm for smooth functions, and an optimal linear time algorithm for smooth generalized linear models.

Private Stochastic Convex Optimization with Heavy Tails: Near-Optimality from Simple Reductions

Abstract

We study the problem of differentially private stochastic convex optimization (DP-SCO) with heavy-tailed gradients, where we assume a k th -moment bound on the Lipschitz constants of sample functions, rather than a uniform bound. We propose a new reduction-based approach that enables us to obtain the first optimal rates (up to logarithmic factors) in the heavy-tailed setting, achieving error G 2 · 1 √ n + G k · ( √ d nε ) 1 -1 k under ( ε, δ ) -approximate differential privacy, up to a mild polylog ( 1 δ ) factor, where G 2 2 and G k k are the 2 nd and k th moment bounds on sample Lipschitz constants, nearly-matching a lower bound of [LR23]. We further give a suite of private algorithms in the heavy-tailed setting which improve upon our basic result under additional assumptions, including an optimal algorithm under a known-Lipschitz constant assumption, a near-linear time algorithm for smooth functions, and an optimal linear time algorithm for smooth generalized linear models.

1 Introduction

Differentially private stochastic convex optimization (DP-SCO), where an algorithm aims to minimize a population loss given samples from a distribution, is a fundamental problem in statistics and machine learning. In this problem, given n samples from a distribution P over a sample space S , our goal is to privately find an approximate minimizer ˆ x ∈ X ⊂ R d for the population loss

where f ( · ; s ) is a convex function for all s ∈ S . The quality of an algorithm is measured by the excess population loss of its output ˆ x , that is F P (ˆ x ) -min x ⋆ ∈X F P ( x ⋆ ) .

Extensive research efforts have been devoted to DP-SCO, resulting in important progress over the past few years [BFTT19, FKT20, AFKT21, BGN21, ALD21, KLL21]. In an important milestone, [BFTT19] developed optimal algorithms (in terms of the excess population loss) for DP-SCO under a uniform Lipschitz assumption (i.e., where every f ( · ; s ) is assumed to have the same Lipschitz bound), and [FKT20] followed this result with efficient and optimal algorithms that run in linear time for smooth functions. DP-SCO has also been explored in other notable settings, including developing faster algorithms for non-smooth settings [AFKT21, KLL21, CJJ + 23], different geometries imposed on the solution space [AFKT21, BGN21, GLL + 23], and different notions of privacy [ALD21].

Most existing results in DP-SCO are based on the assumption that the function f ( · ; s ) is uniformly G -Lipschitz for all s ∈ S . This assumption is convenient for private algorithm design, because it allows us to straightforwardly bound the sensitivity of iterates of private algorithms, i.e., how far a pair of iterates defined via algorithms induced by neighboring datasets drift apart. Under the uniform Lipschitz assumption, the DP-SCO problem is relatively well-understood, as optimal and efficient

algorithms exist (sometimes requiring additional regularity assumptions) [BFTT19, FKT20]. 2 Stateof-the-art SCO algorithms satisfying ( ε, δ ) -differential privacy (Definition 1) in the uniform Lipschitz setting result in excess population loss

where D is the diameter of X . However, the assumption of uniformly G -Lipschitz gradients is strong, and may be violated in real-life applications where the distribution in question has heavy tails (see e.g. discussion in [ACG + 16]). As a simple motivating example, consider mean estimation, where each f ( · ; s ) = 1 2 ∥· -s ∥ 2 , so the minimizer of F P is the population mean. The uniform Lipschitz requirement amounts to P having a bounded support over X , whereas an algorithm that can handle heavy tails only posits the weaker assumption that P has bounded k -th moments. However, as pointed out by [WXDX20], many real-world datasets [MM97, BDFS07, IIW15], especially those from biomedicine and finance, are usually unbounded or even heavy-tailed. As a result, existing algorithms for DP-SCO may have overly pessimistic performance bounds when G is large or even unbounded, necessitating the search for new private algorithms handling heavy-tailed gradients.

Motivated by this weakness of existing DP-SCO analyses, several papers studied the problem of DP-SCO with heavy-tailed gradients [WXDX20, ADF + 21, KLZ22, LR23], formally defined in Definition 4. Rather than assuming uniformly Lipschitz gradients, this line of work builds on the more realistic assumption that the norm of the gradients has bounded k th -moments. In particular, [ADF + 21] studied heavy-tailed private optimization for the related empirical loss, while [WXDX20] initiated an analogous study for the population loss. More recently, [KLZ22, LR23] also proposed algorithms to solve the heavy-tailed DP-SCO problem based on clipped stochastic gradient methods.

Despite the significant progress made in addressing heavy-tailed DP-SCO, it remains notably less understood compared to the uniformly Lipschitz setting. As a benchmark, under a notion called ρ -concentrated differential privacy (CDP, see Definition 3), which translates to ( ε, δ ) -DP for ρ ≈ ε 2 log -1 ( 1 δ ) , [LR23] established that the best excess population loss achievable scales as

where G j j is the j th moment bound on the Lipschitz constant of sampled functions, see Definition 4. Note that as k →∞ , the rate in (2) recovers the uniform Lipschitz rate in (1).

Unfortunately, existing works on heavy-tailed DP-SCO assume stringent conditions on problem parameters and are suboptimal in the general case. For example, [KLZ22] requires the loss functions to be uniformly smooth with various parameter bounds in order to guarantee optimal rates, while the recent work [LR23] obtains a suboptimal rate scaling as 3 G 2 D · 1 √ n + G k D · ( √ d n √ ρ ) 1 -2 k , which is worse than (2) by polynomial factors in the dimension for any constant k .

1.1 Our contributions

Motivated by the suboptimality of existing results for heavy-tailed DP-SCO, we develop the first algorithm for this problem, which achieves the optimal rate (2) up to logarithmic factors with no additional assumptions. Along the way, we give several simple reduction-based tools for overcoming technical barriers encountered by prior works. To state our results (deferring a formal problem statement to Definition 1), we assume that for some k ≥ 2 and all j ∈ [ k ] , we have

Our results hold in several settings and are based on different reductions which allow us to apply strategies for DP-SCO from the uniform Lipschitz setting.

Near-optimal rates for heavy-tailed DP-SCO (Section 3). We design an algorithm for the k -heavy-tailed DP-SCO problem, which satisfies ρ -CDP 4 and attains near-optimal excess loss

This matches the lower bounds recently proved by [KLZ22, LR23] for ρ -concentrated DP algorithms up to logarithmic factors, stated in (2). Standard conversions from CDP to ( ε, δ ) -DP imply that our algorithm also obtains loss ≈ G 2 D · √ 1 n + G k D · ( √ d log 3 (1 /δ ) nε ) 1 -1 k under this parameterization. We note that our bound (3) holds with high probability ≥ 1 -δ , whereas the lower bound (2) is for an error which holds only in expectation (see Theorem 13, [LR23]). Our lossiness in (3) is due to a natural sample-splitting strategy used to boost our failure probability, and we conjecture that (3) may be optimal in the high-probability error bound regime.

As in [LR23], to establish our result we begin by deriving utility guarantees for a clipped stochastic gradient descent subroutine on an empirical loss, where clipping ensures privacy but induces bias, parameterized by a dataset-dependent quantity b 2 D defined in (26). We give a standard analysis of this subroutine in Proposition 1, a variant of which (with slightly different parameterizations) also appeared as Lemma 27, [LR23]. However, the key technical barrier encountered by the [LR23] analysis, when converting to population risk, was bounding E b 2 D over the sampled dataset, which naïvely depends on the 2 k th moment of gradients. This either incurs an overhead depending on G 2 k , or in the absence of such a bound (which is not given under the problem statement), leads to the aforementioned suboptimal rate in [LR23] losing a factor of ( √ d n √ ρ ) 1 k in the utility. We give a further discussion of natural strategies and barriers towards directly bounding E b 2 D in Appendix G.

Where we depart from the strategy of [LR23] is in the use of a new population-level localization framework we design (see Algorithm 2), inspired by similar localization techniques in prior work [FKT20] (discussed in more detail in Section 1.2). This strategy allows us to use constant-success probability bounds on the quantity b D (which also bound b 2 D ), which are easy to achieve depending only on G k rather than G 2 k via Markov's inequality. This bypasses the need in [LR23] for bounding E b 2 D . The motivation for population-level localization is that we wish to aggregate empirical solutions to multiple datasets, some of which have small b D , and others which do not. However, each dataset has a different empirical minimizer, so it is unclear how to argue about convergence if we apply the empirical localization. Instead, we aggregate solutions close to the population-level minimizer and share them across datasets via a simple geometric aggregation technique, showing that it suffices for a constant fraction of datasets to have this desirable property for us to carry out our population-level localization argument. We formally state our main result achieving the rate (3) as Theorem 1.

Interestingly, as a straightforward corollary of our new localization framework, we achieve a tight rate for high-probability stochastic convex optimization under a bounded-variance gradient estimator parameterization, perhaps the most well-studied formulation of SCO. To our knowledge, this result was only first achieved very recently by [CH24]. 5 However, we find it a promising proof-of-concept that our new framework directly yields the same result. For completeness, we include a derivation in Appendix E (see Theorem 5) as a demonstration of the utility of our framework.

Optimal rates with known Lipschitz constants (Appendix B). We next consider the known Lipschitz setting, where each sample function f ( · ; s ) arrives with a value L s which is an overestimate of its Lipschitz constant, such that E L j s is bounded for all j ∈ [ k ] (see Assumption 2). As motivation, consider the problem of learning a generalized linear model (GLM), where f ( · ; s ) = σ ( ⟨· , s ⟩ ) for a known convex activation function σ . Typically, the Lipschitz constant for f ( · ; s ) is simply the Lipschitz constant of σ times ∥ s ∥ , which can be straightforwardly calculated. Thus, for GLMs, our known Lipschitz heavy-tailed assumption amounts to moment bounds on the distribution P .

Our second result, Theorem 2, shows a natural strategy obtains optimal rates in this known Lipschitz setting, eliminating logarithmic factors from Theorem 1. As mentioned previously, this result holds for the important family of GLMs. Our algorithm is based on a straightforward reduction to the uniformly Lipschitz setting: after simply iterating over the input samples, and replacing samples whose Lipschitz constant exceeds a given threshold with a new dummy sample, we show existing Lipschitz DP-SCO algorithms then obtain the optimal heavy-tailed excess population loss (2). Despite the simplicity of this result, to the best of our knowledge, it was not previously known.

Efficient algorithms for smooth functions (Appendices C and D). Finally, we propose algorithms with improved query efficiency for general smooth functions or smooth GLMs, with moderate smoothness bounds. Our strategy is to analyze the stability of clipped-DP-SGD in the smooth heavytailed setting, and use localization-based reductions to transform a stable algorithm into a private one [FKT20]. This results in linear-time algorithms for the smooth case with near-optimal rates. In order to prove the privacy of our smooth, heavy-tailed algorithm, we analyze a careful interplay of our clipped stochastic gradient method with the sparse vector technique (SVT) [DNR + 09, DR14]. At a high level, our use of SVT comes from the fact that under clipping, smooth gradient steps no longer enjoy the type of contraction guarantees applicable in the uniform Lipschitz setting (see Fact 3), so we must take care to not clip too often. The SVT is then used to ensure privacy of our count of how many clipping operations were used. In Appendix F, we provide a simple counterexample showing that the noncontractiveness of contractive steps after applying clipping is inherent. Our general smooth heavy-tailed DP-SCO result is stated as Theorem 3.

We believe the use of SVT within an optimization algorithm to ensure privacy may be of independent interest, as it is one of few such instances that have appeared in the private optimization literature to our knowledge; it is inspired by a simpler application of this technique carried out in [AL24].

On the other hand, we make the simple observation that for GLMs, clipping cannot make a contractive gradient step noncontractive, by taking advantage of the fact that the derivative of f ( x ; s ) = σ ( ⟨ x, s ⟩ ) is a multiple of s for any x ∈ X (see Lemma 14). We use this observation to give a straightforward adaptation of the smooth algorithm in [FKT20] to the heavy-tailed setting, proving Theorem 4, which attains both a linear gradient query complexity and the optimal rate (2).

1.2 Prior work

The best-known rates for heavy-tailed DP-SCO were recently achieved by [KLZ22, LR23]. As discussed previously, their results do not provide the same optimality guarantees as our Theorem 1. The rate achieved by [LR23] is polynomially worse than the optimal loss (2) for any constant k . On the other hand, the work of [KLZ22] uses a different assumption on the gradients than Assumption 1, which is arguably more nonstandard: in particular, they require that the k th -order central moments of each coordinate ∇ j f ( x ; s ) is bounded. Moreover, their algorithms require each sample function f ( · ; s ) to be β -smooth, and the final rates have a strong dependence on the condition number κ = β λ where λ is the strong convexity parameter (see Appendix C in [LR23] for additional discussion).

Our result in the heavy-tailed setting assuming β -smoothness of sample functions, Theorem 3, is most directly related to Theorem 15 of [LR23]. These two results respectively require

omitting logarithmic factors in our bound for simplicity, to obtain near-optimal rates. These regimes are different and not generally comparable. However, we find it potentially useful that our upper bound on β grows as more samples are taken, whereas the [LR23] bound degrades with larger n . It is worth mentioning that [LR23]'s Theorem 15 shaves roughly one logarithmic factor in the error bound from our Theorem 3. On the other hand, Theorem 3 actually requires a looser condition than mentioned above (see (20)), which can improve its guarantees in a wider range of parameters.

Finally, we briefly contextualize our population-level localization framework in regard to previous localization schemes proposed by [FKT20]. The two localization schemes in [FKT20] (see Sections 4.1 and 5.1 of that work) both follow the same strategy of gradually improving distance bounds to a minimizer in phases. However, their implementation is qualitatively different than our Algorithm 2, preventing their direct application in our algorithm. For instance, Section 4.1 of [FKT20] does not use strong convexity and, therefore cannot take advantage of generalization bounds afforded to

strongly convex losses (see discussion in [SSSS09]). On the other hand, the scheme in Section 5.1 of [FKT20] serves a different purpose than Algorithm 2, aiming to solve strongly convex optimization by reducing it to non-strongly convex optimization; our Algorithm 2, on the other hand, directly targets non-strongly convex optimization as its goal. We view our approach as complementary to these prior frameworks and are optimistic it will find further utility in applications.

2 Preliminaries

General notation. We use [ d ] to denote the set { i ∈ N | i ≤ d } . We use sign( x ) ∈ {± 1 } to denote the sign for x ∈ R , with sign(0) = 1 . We use N ( µ, Σ ) to denote the multivariate normal distribution of specified mean and covariance. We denote the all-ones and all-zeroes vectors of dimension d by ✶ d and 0 d . We use ∥·∥ to denote the Euclidean ( ℓ 2 ) norm. We use I d to denote the identity matrix on R d . We use B ( C ) to denote the ℓ 2 ball of radius C , and for x ∈ R d , B ( x, C ) is used to denote { x ′ ∈ R d | ∥ x ′ -x ∥ ≤ C } . For a set X ⊆ R d , we let diam ( X ) := sup x,x ′ ∈X ∥ x -x ′ ∥ , and we let Π X ( x ) denote the Euclidean projection of x to X , i.e. argmin x ′ ∈X ∥ x ′ -x ∥ , which exists and is unique when X is compact. We use f X to denote the restriction of a function f to X , i.e.

For x ∈ R d , we use Π C ( x ) as shorthand for Π B ( C ) ( x ) , i.e. Π C ( X ) denotes the clipped vector x · min( C ∥ x ∥ , 1) . We say two datasets D , D ′ are neighboring if they differ in one entry, and |D| = |D ′ | . We say x ∈ X is an ε -approximate minimizer to f : X → R if f ( x ) -inf x ⋆ ∈X f ( x ⋆ ) ≤ ε . For two densities µ, ν on the same probability space, and α > 1 , we define the α -Rényi divergence

For an event E on a probability space clear from context, we let I E denote the 0 -1 indicator of E . We say f : X → R is L -Lipschitz if | f ( x ) -f ( x ′ ) | ≤ L ∥ x -x ′ ∥ for all x, x ′ ∈ X ; if f is differentiable and convex, an equivalent characterization is ∥∇ f ( x ) ∥ ≤ L for all x ∈ X . We say f : X → R is µ -strongly convex if f ( λx ′ +(1 -λ ) x ) ≤ λf ( x ′ )+(1 -λ ) f ( x ) -µλ (1 -λ ) 2 ∥ x -x ′ ∥ 2 for all x, x ′ ∈ X . We say differentiable f : X → R is β -smooth if for all x, x ′ ∈ X , ∥∇ f ( x ) -∇ f ( x ′ ) ∥ ≤ β ∥ x -x ′ ∥ .

Differential privacy. We begin with a definition of standard differential privacy.

Definition 1 (Differential privacy) . Let ε ≥ 0 , δ ∈ [0 , 1] . We say a mechanism (randomized algorithm) M : S n → Ω satisfies ( ε, δ ) -differential privacy (alternatively, M is ( ε, δ ) -DP) if for any neighboring D , D ′ ∈ S n , and any S ⊆ Ω , Pr[ M ( D ) ∈ S ] ≤ exp( ε ) Pr [ M ( D ′ ) ∈ S ] + δ .

More generally, for random variables X,Y ∈ Ω satisfying Pr[ X ∈ S ] ≤ exp( ε ) Pr[ Y ∈ S ] + δ for all S ⊆ Ω , we say that X,Y are ( ε, δ ) -indistinguishable.

Throughout the paper, other notions of differential privacy will frequently be useful for our accounting of privacy loss in our algorithms. For example, we define the following variants of DP.

Definition 2 (Rényi DP) . Let α > 1 , ε ≥ 0 . We say a mechanism M : S n → Ω satisfies ( α, ε ) -Rényi differential privacy (RDP) if for any neighboring D , D ′ ∈ S n , D α ( M ( D ) ∥M ( D ′ )) ≤ ε .

Definition 3 (CDP) . Let ρ ≥ 0 . We say a mechanism M : S n → Ω satisfies ρ -concentrated differential privacy (alternatively, M satisfies ρ -CDP) if for any neighboring D , D ′ ∈ S n , and any α ≥ 1 , D α ( M ( D ) ∥M ( D ′ )) ≤ αρ .

For an extended discussion of RDP and CDP and their properties, we refer the reader to [BS16, Mir17, BDRS18]. We summarize the main facts about these notions we use here.

Lemma 1 ([Mir17]) . RDP has the following properties.

(Composition): Let M 1 : S n → Ω satisfy ( α, ε 1 ) -RDP and M 2 : S n × Ω → Ω ′ satisfy ( α, ε 2 ) -RDP for any input in Ω . Then the composition of M 2 and M 1 , i.e. the randomized algorithm which takes D to M 2 ( D , M 1 ( D )) , satisfies ( α, ε 1 + ε 2 ) -RDP.

(RDP to DP): If M satisfies ( α, ε ) -RDP, it satisfies ( ε + 1 α -1 log 1 δ , δ ) -DP for all δ ∈ (0 , 1) .

(Gaussian mechanism): Let f : S n → R d be an L -sensitive randomized function for L ≥ 0 , i.e. for any neighboring D , D ′ , we have ∥ f ( D ) -f ( D ′ ) ∥ ≤ L . Then for any σ > 0 , the mechanism which outputs f ( D ) + ξ for ξ ∼ N ( 0 d , σ 2 I d ) satisfies L 2 2 σ 2 -CDP.

Private SCO. Throughout the paper, we study the problem of private stochastic convex optimization (SCO) with heavy-tailed gradients. We first define the assumptions used in our algorithms.

Assumption 1 ( k -heavy-tailed distributions) . Let X ⊂ R d be a compact, convex set. Let P be a distribution over a sample space S , such that each s ∈ S induces a continuously-differentiable, convex, L s -Lipschitz loss function f ( · ; s ) : X → R , 6 where L s := max x ∈X ∥∇ f ( x ; s ) ∥ is unknown. For k ∈ N satisfying k ≥ 2 , we say P satisfies the k -heavy tailed assumption if, for a sequence of monotonically nondecreasing { G j } j ∈ [ k ] , 7 we have E s ∼P [ L j s ] ≤ G j j < ∞ for all j ∈ [ k ] .

In Appendix B, we consider a variant of Assumption 1 where we have explicit access to upper bounds on the Lipschitz constants L s , formalized in Assumption 2. Our goal is to approximately optimize a population loss over sample functions satisfying Assumptions 1 or 2, formalized in the following.

Definition 4 ( k -heavy-tailed private SCO) . In the k -heavy-tailed private SCO problem, X ⊂ R d is a compact, convex set with diam ( X ) = D . Further, P is a distribution over a sample space S satisfying Assumption 1. Our goal is to design an algorithm which provides an approximate minimizer in expectation to the population loss, F P ( x ) := E s ∼P [ f ( x ; s )] , subject to satisfying differential privacy. We say such an algorithm queries N sample gradients if it queries ∇ f ( x ; s ) for N different pairs ( x, s ) ∈ X × S . If P further satisfies Assumption 2, we call the corresponding problem the known Lipschitz k -heavy-tailed private SCO problem.

We first observe the following consequence of Assumption 1, deferring a proof to Appendix A.

Lemma 2. Let P be a distribution over S satisfying Assumption 1. Then F P is G 1 -Lipschitz.

We require the following claim which bounds the bias of clipped heavy-tailed distributions.

Fact 1 ([BD14], Lemma 3) . Let k > 1 and X ∈ R d be a random vector with E [ ∥ X ∥ k ] ≤ G k . Then,

We also use the following standard claim on geometric aggregation.

Fact 2 ([KLL + 23], Claim 1) . Let S := { x i } i ∈ [ k ] ⊂ R d have the property that for (unknown) z ∈ R d , |{ i ∈ [ k ] | ∥ x i -z ∥ ≤ R }| ≥ 0 . 51 k for some R ≥ 0 . There is an algorithm Aggregate which runs in time O ( dk 2 ) and outputs x ∈ S such that ∥ x -z ∥ ≤ 3 R .

Finally, given a dataset D ∈ S ∗ of arbitrary size, and λ ≥ 0 , we use the following shorthand to denote the regularized empirical risk minimization (ERM) objective corresponding to the dataset:

When λ = 0 , we simply denote the function above by F D ( x ) := 1 |D| ∑ s ∈D f ( x ; s ) .

3 Heavy-Tailed Private SCO

In this section, we obtain near-optimal algorithms for the problem in Definition 4 using a new population-level localization framework, combined with geometric aggregation for boosting weak subproblem solvers to succeed with high probability (Fact 2). Our algorithm's main ingredient, in Section 3.1, is a clipped DP-SGD subroutine for privately minimizing a regularized ERM subproblem, under a condition on a randomly sampled dataset holding with constant probability. Next, in Section 3.2 we show that our algorithm from Section 3.1 returns points near the minimizer of a

regularized loss function over the population, using generalization arguments. Finally, we develop our population-level localization scheme in Section 3.3, and combine it with our subproblem solver to give our overall method for heavy-tailed private SCO. Several proofs and a generalization to strongly convex functions (Corollary 3) are deferred to Appendix A.

3.1 Strongly convex DP-ERM solver

We give a parameterized subroutine for minimizing a DP-ERM objective F D ,λ ( x ) associated with a dataset D and a regularization parameter λ ≥ 0 (recalling the definition (5)). In this section only, for notational convenience we identify elements of D with [ n ] where n := |D| , so we will also write

i.e. we let f i ( · ) := f ( · ; s ) where s ∈ D is the element identified with i ∈ [ n ] . Our subroutine is a clipped DP-SGD algorithm (Algorithm 1), which only clips the heavy-tailed portion of ∇ F D ,λ (i.e. the sample gradients), and leaves both the regularization and additive noise unchanged. The utility of Algorithm 1 is parameterized by the following function of the dataset:

In other words, b D denotes the maximum bias incurred by the clipped gradient of F D when compared to the true gradient, over points in X ; note the maximum is achieved as X is compact.

We are now ready to state our algorithm, Clipped -DP -SGD , as Algorithm 1.

8 Return: ˆ x ← ∑ 0 ≤ t<T ( t +4) x t ∑ 0 ≤ t<T ( t +4)

We provide the following guarantee on Clipped -DP -SGD , by modifying an analysis of [LSB12].

Proposition 1. Let ρ ≥ 0 , and ˆ x be the output of Clipped -DP -SGD with η t ← 4 λ ( t +1) for all 0 ≤ t < T , σ 2 ← 2 C 2 T n 2 ρ , and T ≥ max( n, n 2 ρ d ) . Clipped -DP -SGD satisfies ρ -CDP, and

For ease of use of Proposition 1, we now provide a simple bound on b D which holds with constant probability from a dataset drawn from a distribution satisfying Assumption 1.

Lemma 3. Let D ∼ P n , where P is a distribution over S satisfying Assumption 1. With probability at least 4 5 , denoting b D as in (26) , we have

We therefore have the following corollary of Proposition 1 and Lemma 3.

Corollary 1. Let D ∼ P n , where P is a distribution over S satisfying Assumption 1, and let x ⋆ D ,λ := argmin x ∈X F D ,λ ( x ) , following (5) . If we run Clipped -DP -SGD with parameters in Proposition 1 and C ← G k · ( 25 n 2 ρ 32 d ) 1 2 k , Clipped -DP -SGD is ρ -CDP, and there is a universal constant C erm such that with probability ≥ 3 5 over the randomness of D and Clipped -DP -SGD , ˆ x , the output of Clipped -DP -SGD , satisfies

Clipped -DP -SGD queries at most max( n 2 , n 3 ρ d ) sample gradients (using samples in D ).

Proof. Condition on the conclusion of Lemma 3, which holds with probability 4 5 . Therefore, Markov's inequality shows that with probability at least 3 5 , after a union bound with Proposition 1,

where we used strong convexity in the first inequality, and plugged in our choice of C in the last. The conclusion follows by rearranging the above display, and using √ a 2 + b 2 ≤ a + b for a, b ∈ R ≥ 0 .

3.2 Localizing regularized population loss minimizers

Here, we use generalization arguments from the SCO literature to show how that our algorithm Clipped -DP -SGD from Section 3.1 acts as an oracle which, with constant probability, returns a point near the minimizer of a regularized population loss. We begin with a standard helper statement.

Lemma 4. Let λ ≥ 0 , let P be a distribution over S satisfying Assumption 1, let ¯ x ∈ X where X ⊂ R d is compact and convex, and let

Next, we apply a result on generalization due to [LR23] to bound the expected distance between a restricted empirical regularized minimizer and the minimizer of the population variant in (7).

Lemma 5. Let λ ≥ 0 , let D ∼ P n where P is a distribution over S satisfying Assumption 1, and let ¯ x ∈ X where X ⊂ R d is compact and convex. Following notation (4) , (5) , let

and let x ⋆ λ, ¯ x be defined as in (7) . Then with probability ≥ 0 . 95 over the randomness of D ∼ P n ,

Corollary 2. Let D ∼ P n , where P is a distribution over S satisfying Assumption 1, and let ¯ x ∈ X where X ⊂ R d is compact and convex. Let λ ≥ 0 and define x ⋆ λ, ¯ x as in (7) . There is a ρ -CDP algorithm A which queries max( n 2 , n 3 ρ d ) sample gradients (using samples in D ). With probability 0 . 55 over the randomness of A and D , A returns ˆ x satisfying, for a universal constant C reg-pop ,

Proof. Condition on the conclusion of Lemma 5 holding for our dataset, which loses 0 . 05 in the failure probability. Next, consider the guarantee of Corollary 1, when applied to the truncated and shifted functions, ˜ f ( x ; s ) ← f B (¯ x,r ) ( x -¯ x ; s ) , where r is set as in Lemma 5. It shows that with probability 3 5 , ∥ ˆ x + ¯ x -y ∥ = O ( G k λ ( √ d n √ ρ ) 1 -1 k + √ λr √ n ) , for the point ˆ x returned by the algorithm, and y the exact minimizer of the empirical loss restricted to B (¯ x, r ) . Therefore, the conclusion follows by overloading ˆ x ← ˆ x +¯ x , applying the triangle inequality with the conclusions of Corollary 1 and 2, and taking a union bound over their failure probabilities.

3.3 Population-level localization

In this section, we provide a generic population-level localization scheme for stochastic convex optimization, which may be of broader interest. Our localization scheme is largely patterned off of the analogous localization methods developed by [FKT20], but directly argues about contraction to population-level regularized minimizers (as opposed to empirical minimizers), which makes it compatible with our framework in Section 3.1 and 3.2, specificially the guarantees of Corollary 2.

Algorithm 2: Population -Localize ( x 0 , P , λ, I )

1 Input: Initial point x 0 ∈ X , distribution P over samples in S , for X , S inducing a k -heavy-tailed DP-SCO problem as in Definition 4, with a population loss F P := E s ∼S [ f ( · ; s )] , λ ≥ 0 , I ∈ N 2 for i ∈ [ I ] do

3 λ i ← λ · 32 i

4 x i ← any point satisfying

5 end

6 Return: x I

We briefly discuss the role of the hyperparameters λ, ∆ in Algorithm 2 for clarity. The parameter ∆ scales with the error guarantee of our regularized ERM solver; in particular, it will be determined by the bound in Corollary 2. The parameter λ specifies an initial regularization amount that will later be tuned to trade off the terms in the following Proposition 2.

Proposition 2. Following notation of Algorithm 2, let x ⋆ := argmin x ∈X F P ( x ) . Then,

In particular, choosing λ to optimize this bound, we have

Proof. We denote x ⋆ 0 := x ⋆ throughout the proof. First, we expand

Moreover, for each i ∈ [ I ] , since x ⋆ i minimizes F P ( x ) + λ i 2 ∥ x -x i -1 ∥ 2 ,

Combining the above two displays, and using that F P is G 1 -Lipschitz (Lemma 2), we have

where we used the diameter bound assumption diam ( X ) = D , as in Definition 4.

In particular, note that Corollary 2 shows that by using n samples from P and a CDP budget of ρ , with constant probability, we can satisfy the requirement (8) with ∆4 i = O ( G k ( √ d n √ ρ ) 1 -1 k + G 2 √ n ) . By plugging this guarantee into the aggregation subroutine in Fact 2, we have our SCO algorithm.

Theorem 1. Consider an instance of k -heavy-tailed private SCO, following notation in Definition 4, let x ⋆ := argmin x ∈X F P ( x ) , and let ρ ≥ 0 , δ ∈ (0 , 1) . Algorithm 2 using Algorithm 3 in Line 5 is a ρ -CDP algorithm which draws D ∼ P n , queries C sco max( n 2 , n 3 ρ d ) sample gradients (using samples in D ) for a universal constant C sco , and outputs x ∈ X satisfying, with probability ≥ 1 -δ ,

4 Conclusion

In this work, we consider the DP-SCO with heavy-tailed gradients. When the k -th moments of gradients are bounded, we propose the population-level localization framework and attain nearoptimal excess loss G 2 D · √ log( 1 δ ) n + G k D · ( √ d log( 1 δ ) n √ ρ ) 1 -1 k with probability at least 1 -δ and satisfy ρ -CDP. We can achieve a tight rate for high-probability SCO under a bounded-variance gradient estimator parameterization by applying the population-level localization framework. Moreover, we improve this basic result under additional assumptions, including an optimal algorithm under a known-Lipschitz constant assumption, a near-linear time algorithm for smooth functions, and an optimal linear time algorithm for smooth generalized linear models, with interesting techniques adapted to each setting.

It leaves many intriguing open problems in this direction. For example, can we design near-linear time algorithms for non-smooth functions? Can the population-level localization framework be applied to solve other problems? Can we establish a high-probability lower bound or eliminate the additional logarithmic term if we are only concerned with the excess bound in expectation? Can we evaluate the algorithm's performance through numerical simulations or real-world datasets? We leave these questions for future research.

References

[ACG + 16] Martin Abadi, Andy Chu, Ian Goodfellow, Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 23rd Annual ACM Conference on Computer and Communications Security (CCS) , pages 308-318, 2016.

[ACJ + 21] Hilal Asi, Yair Carmon, Arun Jambulapati, Yujia Jin, and Aaron Sidford. Stochastic bias-reduced gradient methods. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021 , pages 1081010822, 2021.

[ADF + 21] Hilal Asi, John Duchi, Alireza Fallah, Omid Javidbakht, and Kunal Talwar. Private adaptive gradient methods for convex optimization. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 383-392, 2021.

[AFKT21] Hilal Asi, Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: Optimal rates in ℓ 1 geometry. In Proceedings of the 38th International Conference on Machine Learning (ICML) , 2021.

[AL24] Hilal Asi and Daogao Liu. User-level differentially private stochastic convex optimization: Efficient algorithms with optimal rates. In International Conference on Artificial Intelligence and Statistics, 2024 , volume 238 of Proceedings of Machine Learning Research , pages 4240-4248. PMLR, 2024.

[ALD21] Hilal Asi, Daniel Levy, and John Duchi. Adapting to function difficulty and growth conditions in private optimization. In Proceedings of the 34nd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS) , volume 34, pages 19069-19081, 2021.

[BD14] Rina Foygel Barber and John C. Duchi. Privacy: A few definitional aspects and consequences for minimax mean-squared error. In 53rd IEEE Conference on Decision and Control, CDC 2014 , pages 1365-1369. IEEE, 2014.

[BDFS07] Atanu Biswas, Sujay Datta, Jason P Fine, and Mark R Segal. Statistical advances in the biomedical science. (No Title) , 2007.

[BDRS18] Mark Bun, Cynthia Dwork, Guy N. Rothblum, and Thomas Steinke. Composable and versatile privacy via truncated CDP. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing, STOC 2018 , pages 74-86. ACM, 2018.

[BFTT19] Raef Bassily, Vitaly Feldman, Kunal Talwar, and Abhradeep Thakurta. Private stochastic convex optimization with optimal rates. In Proceedings of the 32nd Annual Conference on Advances in Neural Information Processing Systems (NeurIPS) , pages 11282-11291, 2019.

[BGN21] Raef Bassily, Cristobal Guzman, and Anupama Nandi. Non-euclidean differentially private stochastic convex optimization. arXiv:2103.01278 [cs.LG] , 2021.

[BS16] Mark Bun and Thomas Steinke. Concentrated differential privacy: Simplifications, extensions, and lower bounds. In Theory of Cryptography - 14th International Conference, TCC 2016-B, Proceedings, Part I , volume 9985 of Lecture Notes in Computer Science , pages 635-658, 2016.

[CH24] Yair Carmon and Oliver Hinder. The price of adaptivity in stochastic convex optimization. CoRR , abs/2402.10898, 2024.

[CJJ + 23] Yair Carmon, Arun Jambulapati, Yujia Jin, Yin Tat Lee, Daogao Liu, Aaron Sidford, and Kevin Tian. Resqueing parallel and private stochastic convex optimization. In 64th IEEE Annual Symposium on Foundations of Computer Science, FOCS 2023 , pages 2031-2058. IEEE, 2023.

[DDXZ21] Damek Davis, Dmitriy Drusvyatskiy, Lin Xiao, and Junyu Zhang. From low probability to high confidence in stochastic convex optimization. J. Mach. Learn. Res. , 22:49:149:38, 2021.

[DNR + 09] Cynthia Dwork, Moni Naor, Omer Reingold, Guy N. Rothblum, and Salil P. Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing, STOC 2009 , pages 381-390. ACM, 2009.

[DR14] Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9(3 & 4):211-407, 2014.

[FKT20] Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM on the Theory of Computing , pages 439-449, 2020.

[GLL + 23] Sivakanth Gopi, Yin Tat Lee, Daogao Liu, Ruoqi Shen, and Kevin Tian. Private convex optimization in general norms. In Proceedings of the 2023 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA) , pages 5068-5089. SIAM, 2023.

[HRS16] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In ICML , pages 1225-1234, 2016.

[HS16] Daniel J. Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. J. Mach. Learn. Res. , 17:18:1-18:40, 2016.

[IIW15] Marat Ibragimov, Rustam Ibragimov, and Johan Walden. Heavy-tailed distributions and robustness in economics and finance , volume 214. Springer, 2015.

[JST24] Arun Jambulapati, Aaron Sidford, and Kevin Tian. Closing the computational-query depth gap in parallel stochastic convex optimization. In The Thirty Seventh Annual Conference on Learning Theory, COLT 2024 , Proceedings of Machine Learning Research. PMLR, 2024.

[KLL21] Janardhan Kulkarni, Yin Tat Lee, and Daogao Liu. Private non-smooth empirical risk minimization and stochastic convex optimization in subquadratic steps. arXiv preprint arXiv:2103.15352 , 2021.

[KLL + 23] Jonathan A. Kelner, Jerry Li, Allen X. Liu, Aaron Sidford, and Kevin Tian. Semirandom sparse recovery in nearly-linear time. In The Thirty Sixth Annual Conference on Learning Theory, COLT 2023 , volume 195 of Proceedings of Machine Learning Research , pages 2352-2398. PMLR, 2023.

[KLZ22] Gautam Kamath, Xingtu Liu, and Huanyu Zhang. Improved rates for differentially private stochastic convex optimization with heavy-tailed data. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 10633-10660, 2022.

[Lia24] Jiaming Liang. Variance reduction and low sample complexity in stochastic optimization via proximal point method. CoRR , abs/2402.08992, 2024.

[LR23] Andrew Lowy and Meisam Razaviyayn. Private stochastic optimization with large worst-case lipschitz parameter: Optimal rates for (non-smooth) convex losses and extension to non-convex losses. In International Conference on Algorithmic Learning Theory , volume 201 of Proceedings of Machine Learning Research , pages 986-1054. PMLR, 2023.

[LSB12] Simon Lacoste-Julien, Mark Schmidt, and Francis R. Bach. A simpler approach to obtaining an o(1/t) convergence rate for the projected stochastic subgradient method. CoRR , abs/1212.2002, 2012.

[Mir17] Ilya Mironov. Rényi differential privacy. In 30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017 , pages 263-275. IEEE Computer Society, 2017.

[MM97] Benoit B Mandelbrot and Benoit B Mandelbrot. The variation of certain speculative prices . Springer, 1997.

[Sch14] Rolf Schneider. Convex bodies: the Brunn-Minkowski theory . Number 151. Cambridge university press, 2014.

[SSSS09] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In COLT 2009 - The 22nd Conference on Learning Theory , 2009.

[SSSSS09] Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Stochastic convex optimization. In Proceedings of the Twenty Second Annual Conference on Computational Learning Theory , 2009.

[SZ23] Aaron Sidford and Chenyi Zhang. Quantum speedups for stochastic optimization. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023 , 2023.

[WXDX20] Di Wang, Hanshen Xiao, Srinivas Devadas, and Jinhui Xu. On differentially private stochastic convex optimization with heavy-tailed data. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 10081-10091, 2020.

A Deferred proofs from the main body

A.1 Deferred proofs from Section 2

Lemma 2. Let P be a distribution over S satisfying Assumption 1. Then F P is G 1 -Lipschitz.

Proof. This follows from the derivation

A.2 Deferred proofs from Section 3

Proposition 1. Let ρ ≥ 0 , and ˆ x be the output of Clipped -DP -SGD with η t ← 4 λ ( t +1) for all 0 ≤ t < T , σ 2 ← 2 C 2 T n 2 ρ , and T ≥ max( n, n 2 ρ d ) . Clipped -DP -SGD satisfies ρ -CDP, and

Proof. For the privacy claim, note that each call to Line 3 is a postprocessing of a 2 C n -sensitive statistic of the dataset D , since neighboring databases can only change 1 n ∑ i ∈ [ n ] Π C ( ∇ f i ( x t )) by 2 C n in the ℓ 2 norm, via the triangle inequality. Therefore, applying the first and third parts of Lemma 1 shows that after T iterations, the CDP of the mechanism is at most T · 2 C 2 n 2 σ 2 ≤ ρ .

We next prove the utility claim. For each 0 ≤ t ≤ T , denote

where all expectations are only over randomness used by the algorithm, and not the randomness in sampling D . First-order optimality applied to the definition of x t +1 implies, for all 0 ≤ t < T ,

Adding ⟨ g t -ˆ g t -ξ t , x t -x ⋆ ⟩ to both sides and rearranging shows

In the first line, we used strong convexity of the function λ 2 ∥ x ∥ 2 , and in the last line, we used ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 +2 ∥ b ∥ 2 and the definitions of C and b D . Next, adding λ 2 ( ∥ x t ∥ 2 -∥ x t +1 ∥ 2 ) to both sides above and taking expectations over the first t iterations yields

where we used the Fenchel-Young inequality to bound b D ∥ x t -x ⋆ ∥ ≤ b 2 D λ + λ 4 ∥ x t -x ⋆ ∥ 2 . Now, plugging in our step size schedule η t = 4 λ ( t +1) , multiplying by t +4 , and rearranging shows

where we plugged in the choice of σ 2 and T ≥ n 2 ρ d , so C 2 ≤ σ 2 d 2 . Summing the above for 0 ≤ t < T , using that all iterates and x ⋆ lie in B ( r ) , and dividing by Z := ∑ 0 ≤ t<T ( t +4) ≥ T 2 2 , shows

Lemma 3. Let D ∼ P n , where P is a distribution over S satisfying Assumption 1. With probability at least 4 5 , denoting b D as in (26) , we have

Proof. For every s ∈ S let x ⋆ ( s ) := argmax x ∈X ∥∇ f ( x ; s ) -Π C ( ∇ f ( x ; s )) ∥ 2 . Then, we have

The last line used independence of samples, used Fact 1 on the random vector ∇ f ( x ⋆ ( s ); s ) , and applied Assumption 1 with the definition of x ⋆ ( s ) . The conclusion uses Markov's inequality.

Lemma 4. Let λ ≥ 0 , let P be a distribution over S satisfying Assumption 1, let ¯ x ∈ X where X ⊂ R d is compact and convex, and let

Then ∥

Proof. Let r := ∥ ¯ x -x ⋆ ∥ . By strong convexity and the definition of x ⋆ λ, ¯ x ,

Here, we used that F P is G 1 -Lipschitz (Lemma 2), and rearranging yields the conclusion.

Lemma 5. Let λ ≥ 0 , let D ∼ P n where P is a distribution over S satisfying Assumption 1, and let ¯ x ∈ X where X ⊂ R d is compact and convex. Following notation (4) , (5) , let

and let x ⋆ λ, ¯ x be defined as in (7) . Then with probability ≥ 0 . 95 over the randomness of D ∼ P n ,

Proof. For each f ( x ; s ) , define a restricted variant ˜ f ( x ; s ) := f B (¯ x,r ) ( x ; s ) , and let ˜ F P := E s ∼S ˜ f ( · ; s ) . Similarly, define ˜ F D to be the restricted variant of the empirical loss F D . Because ˜ F P is pointwise larger than F P and x ⋆ λ, ¯ x ∈ B (¯ x, r ) by Lemma 4, it is clear that

and y is the minimizer of the empirical (restricted) variant of the above display. Moreover, each of the regularized functions ˜ f ( x ; s ) + λ 2 ∥ x -¯ x ∥ 2 has a Lipschitz constant at most λr = 2 G 1 larger than its unregularized counterpart in X ∩ B (¯ x, r ) , so these functions satisfy the moment bound in Assumption 1 for j = 2 with a bound of 2 G 2 2 +8 G 2 1 . Now, applying Proposition 29, [LR23] yields

The first equality used that x ⋆ λ, ¯ x is independent of sampling D , and the second used ˆ x is the empirical risk minimizer. The conclusion follows from Markov's inequality and strong convexity.

Theorem 1. Consider an instance of k -heavy-tailed private SCO, following notation in Definition 4, let x ⋆ := argmin x ∈X F P ( x ) , and let ρ ≥ 0 , δ ∈ (0 , 1) . Algorithm 2 using Algorithm 3 in Line 5 is a ρ -CDP algorithm which draws D ∼ P n , queries C sco max( n 2 , n 3 ρ d ) sample gradients (using samples in D ) for a universal constant C sco , and outputs x ∈ X satisfying, with probability ≥ 1 -δ ,

Proof. Throughout, we assume that 1 δ is at least a large enough constant (where lossiness can be absorbed into C sco), and that n is at least a sufficiently large constant multiple of log 1 δ (because the entire range of F P is ≤ G 2 D ). We first handle the case where 1 δ is larger than polylog ( n ) , deferring the case of small 1 δ to the end of the proof. Let I, J ∈ N be chosen such that

which is achievable with I = O (log n ) and J = O (log log n δ ) = O (log 1 δ ) . Let m := n J , and assume without loss that m is a power of 2 , which we can guarantee by discarding ≤ 1 2 our samples, losing a constant factor in the claim. For each i ∈ [ I ] , let m i := m 2 i . We subdivide D into J portions, each with m samples, and subdivide each portion into I parts each with m i samples. For j ∈ [ J ] and i ∈ [ I ] , we denote the samples corresponding to the i th part of the j th portion by D j i , so

̸

Next, we show how to implement Line 5 in Algorithm 2, for an iteration i ∈ [ I ] , by calling Algorithm 3 with appropriate parameters. Let n ← m i , ρ ← ρ , and initialize Algorithm 3 with the dataset ∪ j ∈ [ J ] D j i and R := ∆4 i λ i , where

By Corollary 2, each independent run outputs x j i ∈ X satisfying, with probability 0 . 55 ,

Therefore, by a Chernoff bound, with probability ≥ 1 -δ I , at least 0 . 51 J of the copies satisfy the above bound, so Fact 2 yields x i satisfying ∥ x i -x ⋆ i ∥ ≤ R = ∆4 i λ i with the same probability. Union bounding over all I iterations of Algorithm 2 yields the failure probability, and so we obtain the claim from Proposition 2, after plugging in n = O ( m log( 1 δ )) , since the dominant term is D ∆ . The privacy proof follows from the first part of Lemma 1 since for each pair of neighboring databases, exactly one of the datasets D j i are neighboring, and Corollary 2 guarantees privacy of the empirical risk minimization algorithm using that dataset; privacy for all other datasets used is immediate from postprocessing properties of privacy. The gradient complexity comes from aggregating all of the IJ calls to Corollary 2, where we recall the sample sizes decay geometrically.

Finally, if 1 δ is smaller than polylog ( n ) , for the i th iteration of Algorithm 2 we instead set J i ∈ [400 log( I δ i ) , 500 log( I δ i )] where δ i := δ 2 i . Then we subdivide a consecutive batch of n 2 i samples into J i portions, and follow the above proof. It is straightforward to check that (9) still holds with the new value of m i = ⌊ n 2 i J i ⌋ because the 4 i factor growth on the right-hand side continues to outweigh the change in m i . The error bound follows from Proposition 2, and the privacy proof is identical.

A.3 Strongly convex heavy-tailed private SCO via localization

Finally, by following the template of standard localization reductions in the literature (see e.g. Theorem 5.1, [FKT20] or Lemma 5.5, [KLL21]), Theorem 1 obtains an improved rate when all sample functions are strongly convex. For completeness, we state this result below.

Corollary 3. In the setting of Theorem 1, suppose f ( x ; s ) is µ -strongly convex for all s ∈ S . There is an algorithm which draws D ∼ P n , queries C sco max( n 2 , n 3 ρ d ) sample gradients (using samples in D ) for a universal constant C sco , and outputs x ∈ X satisfying, with probability ≥ 1 -δ ,

Proof. This is immediate from the development in Section 5.1 (and the proof of Theorem 5.1) of [FKT20], but we mention one slight difference here. Our guarantees in Theorem 1 do not scale with the initial distance bound to the function minimizer, and instead scale with the domain size, which makes it less directly compatible with the standard localization framework in [FKT20]. However, because Theorem 1 holds with high probability, we also have explicit bounds on the domain size via function error, as seen in the proof of Theorem 5.1 in [FKT20], so we can explicitly truncate our domain to have smaller domain without removing the minimizer. With this modification, the claim follows directly from Theorem 5.1 in [FKT20].

B Optimal Algorithms in the Known Lipschitz Setting

Compared to the standard Lipschitz setting (i.e. the ∞ -heavy-tailed private SCO problem), our algorithm in Section 3 has two downsides: it pays a polylogarithmic overhead in the utility, and it requires an extra aggregation step. In this section, assuming we are in the known Lipschitz k -heavytailed setting (see the following Assumption 2, and Definition 4), we provide a simple reduction to the standard Lipschitz setting, resulting in optimal rates.

Assumption 2 (Known Lipschitz k -heavy-tailed distributions) . In the setting of Assumption 1, suppose that for each s ∈ S we know a value L s ≥ L s . For k ∈ N satisfying k ≥ 2 , we say P satisfies the known Lipschitz k -heavy tailed assumption if, for a sequence of monotonically nondecreasing { G j } j ∈ [ k ] , we have E s ∼P [ L j s ] ≤ G j j < ∞ for all j ∈ [ k ] .

Note that Assumption 2 clearly implies Assumption 1, but gives us additional access to Lipschitz overestimates with bounded moments. We require some additional definitions used throughout the section. First, we augment S with a designated element s 0 ̸∈ S , and define

We also define a truncated distribution parameterized by C ≥ 0 , where we use f ( · ; s 0 ) in place of sample functions with large Lipschitz overestimates, following notation of Assumption 2:

We denote S 0 := S ∪ { s 0 } , and for D ∈ S n , the dataset D C ∈ S n 0 replaces all s ∈ D satisfying L s > C with s 0 . We additionally provide a second reduction in the known Lipschitz heavy-tailed setting, when all sample functions are assumed to be µ -strongly convex. Because our treatments of these cases are slightly different, we use different notation when µ = 0 and µ > 0 , for convenience of exposition. Fixing an arbitrary point ¯ x ∈ X , for µ > 0 , instead of using the constant 0 function as in (10), we define a strongly convex alternative f ( · ; s µ ) , for a designated element s µ :

The truncated distribution parameterized by C ≥ µD , is defined in a similar way:

We denote S µ := S ∪ { s µ } , and for D ∈ S n , the dataset D C µ ∈ S n µ replaces every s ∈ D such that L s > C with s µ . Our focus on the regime C ≥ µD 4 is motivated by the following well-known claim.

Lemma 6. Let X ⊆ R d be compact and convex satisfying diam ( X ) = D , and suppose f : X → R is L -Lipschitz and µ -strongly convex. Then, L ≥ µD 4 .

Proof. Let x ⋆ := argmin x ∈X f ( x ) . By strong convexity, for all x ∈ X ,

Now, choose x such that ∥ x -x ⋆ ∥ ≥ D 2 . To see this is always possible, let x, x ′ ∈ X realize ∥ x -x ′ ∥ = D ; then at least one of x, x ′ must have distance ≥ D 2 from x ⋆ by the triangle inequality. The conclusion follows by rearranging after using our choice of x .

In other words, if C < µD 4 then no sample function will survive the truncation in (13). Finally, we parameterize the performance of algorithms in the standard Lipschitz setting.

Definition 5 (Lipschitz private SCO algorithm) . We say A is an L -Lipschitz private SCO algorithm if it takes input ( D , ρ, X ) , where D ∈ S n is drawn i.i.d. from P , a distribution over S where every s ∈ S induces L -Lipschitz f ( · ; s ) over X ⊂ R d , A ( D , ρ, X ) ∈ X , and A satisfies ρ -CDP. We denote

where F P ( x ) := E s ∼P f ( x ; s ) , and denote the number of sample gradients queried by A by N ( A ) . Moreover, if each Lipschitz function f (; s ) is µ -strongly convex over the convex domain X , we say A is an L -Lipschitz, µ -strongly convex private SCO algorithm, and define Err ( A ) , N ( A ) as before.

With this notation in place, we state our reduction.

We begin with a simple bound relating F C P , F C,µ P and F P .

Lemma 7. Let F P be defined as in Definition 4, where P satisfies Assumption 2, and define F C P as in (11) . Then, F P -F C P is G k k ( k -1) C k -1 -Lipschitz, and F P -F C,µ P is G k k ( k -1) C k -1 + 4 G k +1 k C k -Lipschitz.

Algorithm 4: KnownLipReduction ( D , C, µ, ρ, X , A )

1 Input: Dataset D ∈ S n , clip threshold C ∈ R ≥ 0 , strong convexity parameter µ ∈ R ≥ 0 , privacy parameter ρ ∈ R > 0 , domain X ∈ R d , C -Lipschitz private SCO algorithm A (if µ = 0 ), or C -Lipschitz µ -strongly convex private SCO algorithm A (if µ > 0 )

2 if µ = 0 then

3 Return: A ( D C , ρ, X )

4 end

5 else

6 Return: A ( D C µ , ρ, X )

7 end

Proof. For s ∈ S , let π ( s ) := s 0 if L s > C , and otherwise let π ( s ) := s . For any x, x ′ ∈ X , we have

In the second line, we used that π ( s ) = s unless L s > C , in which case f ( · ; π ( s )) = 0 uniformly. The last line used the definition of L s and Fact 1 with X ← L s , recalling Assumption 2.

Next, we analyze F C,µ P . Overloading π ( s ) := s µ if L s > C , and letting π ( s ) := s otherwise,

In the third line, we used that µD ≤ 4 G 1 ≤ 4 G k by Lemma 6 and Lemma 2 to show that f ( · ; s µ ) is 4 G k -Lipschitz over X . Finally, the last line used Markov's inequality to bound E [ I L s >C ] .

Using Lemma 7, we provide a straightforward analysis of Algorithm 4.

Proposition 3. Consider an instance of known-Lipschitz k -heavy-tailed private SCO (Definition 4), and let ρ ≥ 0 . If A is a C -Lipschitz private SCO algorithm (Definition 5) and µ = 0 , Algorithm 4 using A is a ρ -CDP algorithm which outputs x ∈ X satisfying

Further, if f ( · ; s ) is µ -strongly convex for all s ∈ S and A is a C -Lipschitz, µ -strongly convex private SCO algorithm for µ > 0 , Algorithm 4 using A is a ρ -CDP algorithm which outputs x ∈ X satisfying

In either case, Algorithm 4 queries N ( A ) sample gradients (using samples in D ).

Proof. For the first utility claim, letting x ⋆,C := argmin x ∈X F C P ( x ) , we have

where the first inequality used the definition of x ⋆,C and Lemma 7, and the second used the definition of Err and diam ( X ) = D . For the second claim, we first have

by the definition of Err ( A ) and µ -strong convexity of F C,µ P , so that E [ ∥ ∥ x -x ⋆,C ∥ ∥ ] ≤ ( 2 µ Err ( A )) 1 / 2 by Jensen's inequality. Moreover, we also have

where we use optimality of x ⋆,C in the second inequality, and Lemma 7 in the third. Combining,

and then the claim follows by substituting this bound into (14).

We can use any existing optimal algorithms for DP-SCO to instantiate our reduction. In particular, we can use the algorithm of [FKT20], denoted by A Lip , which has the following guarantees. For simplicity of exposition, we focus on the case where our functions do not possess additional regularity properties e.g. smoothness, and we also focus on the simplest A Lip which attains the optimal utility bound. Because of the generality of our reduction, however, improvements can be made by using more structured or faster subroutines as A Lip , such as the smooth DP-SCO algorithms of [FKT20] or the Lipschitz DP-SCO algorithms of e.g. [AFKT21, KLL21, CJJ + 23], which are more query-efficient, sometimes at the cost of logarithmic factors in the utility (in the case of [CJJ + 23]).

Proposition 4. Let P be a distribution over S such that f ( · ; s ) is L -Lipschitz and convex for all s ∈ S . There exists a constant C Lip such that given D ∼ S n , the algorithm A Lip is ρ -CDP and outputs x priv such that, for a universal constant C Lip , letting x ⋆ := argmin x ∈X F P ( x ) ,

and A Lip queries ≤ C Lip max( n 2 , n 3 ρ d ) sample gradients (using samples in D ), where G 2 is defined as in Assumption 1. Moreover, if f ( · ; s ) is µ -strongly convex for all s ∈ S , then

and A Lip queries ≤ C Lip max( n 2 , n 3 ρ d ) sample gradients (using samples in D ).

Proof. This follows from developments in [FKT20], but we briefly explain any discrepancies. The µ = 0 case applies Theorem 4.8 in [FKT20], where for simplicity we consider the full-batch variant

which does not subsample. 8 Moreover, Theorem 4.8 in [FKT20] is stated with a dependence on L rather than G 2 on the n -1 / 2 term, but inspecting the proof shows it only uses a second moment bound. The µ > 0 case follows from Theorem 5.1 of [FKT20], using Theorem 4.8 as a subroutine.

We are now ready to present our main result in this section, using our reduction with A Lip .

Theorem 2. Consider an instance of known-Lipschitz k -heavy-tailed private SCO (Definition 4), let ρ ≥ 0 , and let x ⋆ := argmin x ∈X F P ( x ) . Algorithm 4 with C ← G k ( n √ ρ √ d ) 1 k using A Lip in Proposition 4 is a ρ -CDP algorithm which outputs x ∈ X satisfying, for a universal constant C HT ,

querying ≤ C HT max( n 2 , n 3 ρ d ) sample gradients (using samples in D ). Further, if f ( · ; s ) is µ -strongly convex for all s ∈ S , Algorithm 4 with C ← G k ( n 2 ρ d ) 1 2 k using A Lip in Proposition 4 is a ρ -CDP algorithm which outputs x ∈ X satisfying

querying ≤ C HT max( n 2 , n 3 ρ d ) sample gradients (using samples in D ).

Proof. Throughout the proof, assume without loss of generality that d ≤ n 2 ρ , as otherwise all stated bounds are vacuous since the additive function value range over X is at most G 1 D ≤ 4 G 2 1 µ by Lemma 6 and Lemma 2. This also implies that C ≥ G k in either case.

In the µ = 0 case, Proposition 3 and the guarantees of A Lip in Proposition 4 imply that

where the last inequality follows from our choice of C . Next, we consider µ > 0 . Proposition 3 and the guarantees of A Lip in Proposition 4 for this case imply that, assuming C Lip ≥ 2 without loss,

where we used C ≥ G k to simplify bounds, and applied our choice of C .

C Fast Algorithms for Smooth Functions

In this section, we develop a linear-time algorithm for the smooth setting where we additionally assume f ( · ; s ) is β -smooth for all s ∈ S . Our algorithm attains nearly-optimal rates for a sufficiently small value of β , and is based on the localization framework of [FKT20]. To apply this framework, we show that a variant of clipped DP-SGD (see Algorithm 5) is stable in the heavy-tailed setting with high probability. We then ensure that stability holds for any input dataset (not necessarily sampled from a distribution P ), by using the sparse vector technique [DR14] to verify that the number of clipped gradients is not too large. In Appendix C.1, we provide some standard preliminary results from the literature. We use these results in Appendix C.2, where we state our algorithm in full as Algorithm 7 and analyze it in Theorem 3, the main result of this section.

C.1 Helper tools

First, we state a standard bound on the contractivity of smooth gradient descent iterations.

Fact 3 (Lemma 3.7, [HRS16]) . Let f : X → R be β -smooth, and let η ≤ 2 β . Then for any x, x ′ ∈ X ,

Next, we provide a standard utility bound on a one-pass SGD algorithm using clipped gradients.

Algorithm 5: OnePass -Clipped -SGD ( D , C, η, T, X , x 0 )

1 Input: Dataset D = { s t } t ∈ [ T ] ∈ S T , clip threshold C ∈ R ≥ 0 , step size η ∈ R ≥ 0 , iteration count T ∈ N , domain X ⊂ B ( x 0 , D ) for x 0 ∈ X

2 for 0 ≤ t < T do

3 x t +1 ← argmin x ∈X { η ⟨ Π C ( ∇ f ( x t ; s t +1 )) , x ⟩ + 1 2 ∥ x -x t ∥ 2 }

4 end

5 Return: ˆ x ← 1 T ∑ 0 ≤ t<T x t

Lemma 8. Consider an instance of k -heavy-tailed private SCO, following notation in Definition 4, and let u ∈ X be independent of D . Assuming D ∼ P T i.i.d., Algorithm 5 outputs ˆ x ∈ X satisfying

Proof. To simplify notation, let g t := ∇ f ( x t ; s t +1 ) for all 0 ≤ t < T , and let ˆ g t := T C ( g t ) . Because s t +1 ∼ P is independent of x t , we have that E g t = ∇ F P ( x t ) . Therefore, in iteration t ,

where all expectations are conditional on the first t iterations of the algorithm, and taken over the randomness of s t +1 . In the third line, we used the first-order optimality condition on x t +1 , applied Fact 1 to bound E ∥ g t -ˆ g t ∥ , and used

Summing across all iterations and dividing by T yields the result upon iterating expectations.

We also note the following straightforward generalization of Lemma 8 to the case of randomized clipping thresholds, which is used in our later development.

Corollary 4. For C, ˆ C ≥ 0 and g ∈ R d , define the operation

If Algorithm 5 is run with Π C ( ∇ f ( x t ; s t +1 )) replaced by Π C, ˆ C t ( ∇ f ( x t ; s t +1 )) where ˆ C t is independent of s t +1 and satisfies ˆ C t ≥ C 2 for all 0 ≤ t < T , then following notation in Lemma 8,

Proof. For a fixed iteration 0 ≤ t < T , the calculation (16) changes in two ways. First, in place of the variance bound (16) (which used ∥ ˆ g t ∥ ≤ ∥ g t ∥ deterministically), when using the modified clipping operators we require the modified deterministic bound

which follows because ∥ ˆ g t ∥ ̸ = ∥ g t ∥ (which implies C = ∥ ˆ g t ∥ ) only if ∥ g t ∥ ≥ C 2 . Moreover, in place of the bias bound E ∥ g t -ˆ g t ∥ ≤ G k k ( k -1) C k -1 which followed from Fact 1, we instead have

The conclusion follows by adjusting these constants appropriately in Lemma 8.

Next, for R,τ ≥ 0 , we let BLap ( R,τ ) denote the bounded Laplace distribution with scale parameter R and truncation threshold τ be defined as the conditional distribution of ξ ∼ Lap( R ) on the event | ξ | ≤ τ (recall that Lap( R ) has a density function ∝ exp( -1 R | ξ | ) ). It is a standard calculation that

so that the total variation distance between Lap( R ) and BLap ( R,R log( 1 δ )) is δ . We hence have the following bounded generalization of the privacy given by the Laplace mechanism.

Lemma 9. Let ε, δ ∈ (0 , 1) . If S ( D ) ∈ R is a ∆ -sensitive statistic of the dataset D , i.e. for neighboring datasets D , D ′ we have that | S ( D ) -S ( D ′ ) | ≤ ∆ , then the bounded Laplace mechanism which outputs S ( D ) + ξ where ξ ∼ BLap ( ∆ ε , τ ) for any τ ≥ ∆ ε log( 4 δ ) satisfies ( ε, δ ) -DP.

Proof. For notational simplicity, let A denote the Laplace mechanism (which samples ξ ∼ Lap( ∆ ε ) instead of BLap ( ∆ ε , τ ) ), let A denote the bounded Laplace mechanism, and let E ⊆ R be an event in the outcome space. By standard guarantees on ( ε, 0) -DP of A (e.g. Theorem 3.6, [DR14]),

for any neighboring datasets, where we used exp( ε ) ≤ 3 and that the total variation distance between ( A ( D ) , A ( D )) and ( A ( D ′ ) , A ( D ′ )) are bounded by δ 4 by (17).

We also use the sparse vector technique (SVT) [DR14], which has been used recently in private optimization in the user-level setting [AL24]. Given an input dataset D = { s i } i ∈ [ n ] ∈ S n , SVT takes a stream of queries q 1 , q 2 , . . . , q T : D → R in an online manner. We assume each q i is ∆ -sensitive, i.e. | q i ( D ) -q i ( D ′ ) | ≤ ∆ for neighboring datasets D , D ′ ∈ S n . One notable difference is that our SVT algorithm will use the bounded Laplace mechanism, rather than the Laplace mechanism, but this distinction is handled similarly to Lemma 9. We provide a guarantee on this variant of SVT in Lemma 10, and pseudocode is provided as Algorithm 6.

Lemma 10. Let δ, ε ∈ (0 , 1) and suppose

Algorithm 6 outputs a sequence of answers { a i ∈ {⊥ , ⊤}} i ∈ [ k ] for some k ∈ [ T ] , and is ( ε, δ ) -DP.

Proof. The proof is analogous to Lemma 9. Let A denote SVT run with Laplace noise in place of bounded Laplace noise (i.e. τ = ∞ ), and let A denote SVT run with bounded Laplace noise. We first claim that A is ( ε, δ 5 ) -DP, which is immediate from Theorem 3.23 and Theorem 3.20 in [DR14].

Next, by a union bound on all of the ≤ 2 T random variables sampled, the total variation distance between ( A ( D ) , A ( D )) for any dataset D is bounded by δ 5 . Then, for neighboring datasets D , D ′ and some event E in the outcome space, repeating the calculation (18),

C.2 Algorithm statement and analysis

In this section, we present the full details of our algorithm (see Algorithm 7) and prove its corresponding guarantees, separating out the privacy analysis and utility analysis.

The following theorem summarizes the guarantees of Algorithm 7.

Theorem 3. Consider an instance of k -heavy-tailed private SCO, following notation in Definition 4, and let x ⋆ := argmin x ∈X F P ( x ) , and ε, δ ∈ (0 , 1) . Algorithm 7 run with parameters

is ( ε, δ ) -DP and outputs x I that satisfies, for a universal constant C smooth ,

assuming f ( · ; s ) is β -smooth for all s ∈ S , where

We now proceed to prove Theorem 3.

̸

Privacy proof overview. We first overview the structure of our privacy proof. Consider two neighboring datasets D , D ′ that differ on a single sample s i,j 0 = s ′ i,j 0 . The core argument used to prove privacy is controlling the total number of times when gradients are clipped, so we introduce the variable ' count .' Note that we have ∥ x i,j 0 +1 -x ′ i,j 0 +1 ∥ = O ( Cη ) due to the clip operation. If no clip ever happened afterward, then we know ∥ x i,n i -x ′ i,n i ∥ ≤ ∥ x i,j 0 +1 -x ′ i,j 0 +1 ∥ = O ( Cη ) due to our smoothness assumption (see Fact 3), which means the algorithm is private. When count is not too large, we can still bound the sensitivity between ∥ x i,n i -x ′ i,n i ∥ by O ( Cη ) . However, when the value of count is larger, there is a risk that the sensitivity of x i,n i is not bounded as before, and hence we halt the algorithm when count exceeds some appropriate cutoff point ˆ c i .

One subtle difference between our algorithm and standard uses of SVT is that we add Laplace noise to the cutoff point c to obtain a randomized cutoff ˆ c i . This is because the sensitivity of the count increment at the j th 0 iteration of phase i is bounded by one, even though ∥∇ f ( x i,j 0 ; s i,j 0 ) ∥ -∥∇ f ( x ′ i,j 0 ; s ′ i,j 0 ) ∥ can be arbitrarily large. The guarantees of the bounded Laplace mechanism imply that the noise added in ˆ c i hence suffices to privatize count .

̸

In summary, we can control the sensitivity between ∥ x i,j -x ′ i,j ∥ for all j due to the termination condition in Line 18 and our use of bounded Laplace noise, and hence can control the sensitivity of the query for ∥∇ f ( x i,j ; s i,j ) ∥ - ∥∇ f ( x ′ i,j ; s ′ i,j ) ∥ for all j = j 0 . By adding Laplace noise on the cutoff c , we handle the issue of the sensitivity of the j th 0 query ∥∇ f ( x i,j 0 ; s i,j 0 ) ∥ being unbounded. If the algorithm succeeds and returns x k , we know the sensitivity ∥ x i,n i -x ′ i,n i ∥ is O ( Cη i ) and the privacy guarantee follows from the Gaussian mechanism. If the algorithm fails and outputs ⊥ , the privacy guarantee follows from the bounded Laplace noise on the cutoff point and the guarantees of SVT.

̸

Privacy proof. We now provide our formal privacy analysis following this overview. To fix notation in the remainder of the privacy proof, we consider running Algorithm 7 on two neighboring datasets D , D ′ that differ on a single sample s i,j 0 = s ′ i,j 0 , for some i ∈ [ I ] . By standard postprocessing properties of differential privacy, it suffices to argue that the i th phase (i.e. the run of the loop in Lines 3 to 25 corresponding to this value of i ) is private, so we fix i ∈ [ I ] in the following discussion.

We let { x i,j } j ∈ [ n i ] and { x ′ i,j } j ∈ [ n i ] be the iterates of the i th phase of Algorithm 7 using D and D ′ , and we let Y i,j and Y ′ i,j be the respective 0 -1 indicator variables that count increases by 1 in iteration j . We also let count j and count ′ j denote the values of count at the end of the j th iteration, and abusing notation we let ˆ c i , ˆ c ′ i be the values of ˆ c i in the i th phase when using D or D ′ respectively. Finally, we denote x i := 1 n i ∑ j ∈ [ n i ] x i,j and let x ′ i denote the average iterate using D ′ similarly.

We first bound the sensitivity between the iterates { x i,j } j ∈ [ n i ] and { x ′ i,j } j ∈ [ n i ] in the following lemma, assuming count j and count j ′ are bounded. The proof is deferred to Appendix H.

̸

Lemma 11. Let t ∈ [ n i ] , and suppose that 192 η i βc ≤ 1 and C ≥ 8 ω i log( 30 n i δ ) . If count t < ˆ c i , count ′ t < ˆ c ′ i , and Y i,j = Y ′ i,j for all j < t with j = j 0 , then

Using this bound on the sensitivity, we are now ready to prove privacy of the algorithm.

Lemma 12. Algorithm 7 is ( ε, δ ) -DP if it is run with parameters satisfying

Proof. Recall our assumption that D and D ′ only differ in s i,j 0 , the j th 0 sample used in the i th phase of the algorithm. The privacy of all phases of the algorithm other than phase i is immediate from postprocessing properties of DP, so it suffices to argue that phase i is ( ε, δ ) -DP. Note also that the conditions of Lemma 11 are met after reparameterizing δ ← δ 4 . We split our privacy argument into two cases, depending on whether the algorithm terminates on Line 18 or Line 26.

Termination on Line 18. We begin with the case where the algorithm outputs ⊥ . We introduce some simplifying notation. For iterations S ⊆ [ n i ] , define W S := { Y i,j } j ∈ S to be the 0 -1 indicator variables for whether count incremented on iterations j ∈ S (when run on D ), and define [ W ] S :=

∑ j ∈ S Y i,j to be their sum. Similarly, define W ′ S and [ W ′ ] S for when the algorithm is run on D ′ . Observe that the algorithm outputs ⊥ iff the following event occurs:

The right-hand side -[ W ] [ j 0 -1] is independent of whether the dataset used was D or D ′ , so it suffices to argue about the privacy loss of the random variables Y i,j 0 -ˆ c i and W [ n i ] \ [ j 0 ] as a function of the dataset used. First, Y i,j 0 -c is clearly a 1 -sensitive statistic, so Lemma 9 implies Y i,j 0 -ˆ c i is ( ε 3 , δ 3 ) -indistinguishable as a function of the dataset used. Next, conditioning on the value of Y i,j 0 -ˆ c i , the random variable W [ n i ] \ [ j 0 ] is an instance of Algorithm 6 run with a fixed threshold ˆ c i -Y i,j 0 -[ W ] [ j 0 -1] ≤ 2 c , where we rename the output variables {⊥ , ⊤} to { 0 , 1 } . Moreover, Lemma 11 and smoothness of each sample function implies that the sensitivity of each query ∥∇ f ( · ; s i,j ) ∥ is bounded by ∆ := 6 Cη i β . Therefore, Lemma 10 shows that W [ n i ] \ [ j 0 ] is ( ε 3 , δ 3 ) -indistinguishable, where we note that we adjusted constants appropriately in ω and the failure probabilities everywhere. By basic composition of DP, this implies Y i,j 0 -ˆ c i + [ W ] [ n i ] \ [ j 0 ] (a postprocessing of Y i,j 0 -ˆ c i and W [ n i ] \ [ j 0 ] | Y i,j 0 -ˆ c i ) is ( 2 ε 3 , 2 δ 3 ) -DP, as required.

Termination on Line 26. Finally, we argue about the privacy when the algorithm does not terminate on Line 18. As before, the sensitivity of ¯ x i is bounded by 6 Cη i via Lemma 11 and the triangle inequality, conditioned on a ( 2 ε 3 , 2 δ 3 ) -indistinguishable event (i.e. the values of Y i,j 0 -ˆ c i and W [ n i ] \ [ j 0 ] | Y i,j 0 -ˆ c i ). Then x i is ( ε 3 , δ 3 ) -indistinguishable by standard bounds on the Gaussian mechanism (Theorem A.1, [DR14]), which completes the proof upon applying basic composition.

Utility proof. The utility proof follows the standard analysis of localized SGD algorithms and a specialized analysis of clipped SGD (Corollary 4). We first state a utility guarantee in each phase.

Lemma 13. Following notation in Algorithm 7, fix i ∈ [ I ] , and suppose D ∼ P n i.i.d. where P satisfies Assumption 1. For any x ∈ X , if C ≥ 8 ω i log( 30 n i δ ) and c 4 ≥ max( n · ( 2 G k C ) k , 6 log( n )) ,

Proof. By Markov's inequality, E s ∼P [ I L s > C 2 ] ≤ ( 2 G k C ) k , so the total number of expected samples with L s > C 2 is at most c 4 . Hence by applying a Chernoff bound,

Conditional on E , the algorithm will not halt (i.e., return ⊥ ) and is running one-pass clipped-SGD (Algorithm 5) using the modified clipping operation defined in the precondition in Corollary 4. Then, the statement follows from Corollary 4 as follows: letting E c denote the complement of E ,

where we used that F P is G 1 ≤ G 2 -Lipschitz by Lemma 2.

Combining our privacy and utility guarantees, we are ready to prove this section's main theorem.

Proof of Theorem 3. For simplicity, let ¯ x 0 := x ⋆ and ζ 0 := x 0 -x ⋆ , so ∥ ζ 0 ∥ ≤ D by assumption. Also, suppose that n is a power of 2 , as the adjustment on Line 2 only affects n (and hence the guarantees) by constant factors. The privacy claim follows immediately from Lemma 12 assuming its preconditions are met, which we verify at the end of the proof. By applying Lemma 13 in each phase i ∈ [ I ] to x ← x i , assuming its preconditions are met, we have

In the first inequality, we used G 1 ≤ G 2 -Lipschitzness of F P by Lemma 2, and in the second inequality, we pulled out the i = 1 term and adjusted indices, and bounded I ≤ n and used Jensen's inequality to bound ( E ∥ ζ I ∥ ) 2 ≤ E ∥ ζ I ∥ 2 = σ 2 I d . Now using that η i n i and η i are geometrically decaying sequences, we continue bounding the above display using our choice of C :

Next, plugging in our choice of

√

we have the claimed utility bound upon simplifying, and using that G 2 σ I d is a low-order term. We now verify our parameters satisfy the conditions in Lemma 12 and Lemma 13, which concludes the proof. First, it is straightforward to check that both sets of conditions are implied by

given that we chose ω = 18 ε √ 2 c log( 15 δ ) ≤ c √ ε . Indeed, C ≥ 8 ω i log( 30 n i δ ) ⇐ = 2 ηβω log( 30 n δ ) ≤ 1 which is subsumed by the first condition in (22). Clearly, c ≥ 26 ε log( 15 n δ ) , giving the third condition in (22). Next, a direct computation with the definition of η 2 in (21) yields

Now because C depends inversely on η ≤ η 2 defined in (21), the second condition in (22) holds:

Finally, the first condition in (22) now follows from our upper bound on β .

D Improved Smoothness Bounds for Generalized Linear Models

In this section, we give an improved algorithm for heavy-tailed private SCO when the sample functions f ( x ; s ) are instances of a smooth generalized linear model (GLM). That is, we assume the sample space S ⊆ R d , and that for a convex function σ : R → R ,

We also assume that all f ( x ; s ) are β -smooth. Observe that

so that for all x ∈ X , ∇ f ( x ; s ) are all scalar multiples of the same vector s . We prove that under this assumption, clipped gradient descent steps can only improve contraction, in contrast to Fact 16.

Lemma 14. Let s, s ′ ∈ R and let x, x ′ , g ∈ R d . Assume that

Then for any C ≥ 0 , letting t := sign( s ) min( | s | , C ) and t ′ := sign( s ′ ) min( | s ′ | , C ) , we have

Proof. Note that the premise is impossible unless sign( s -s ′ ) = sign( ⟨ x -x ′ , g ⟩ ) . Without loss of generality, assume they are both nonnegative, else we can negate s, s ′ , g . In this case,

Now, observe that t -t ′ ≤ s -s ′ and sign( t -t ′ ) = sign( s -s ′ ) , for any value of C ≥ 0 . Therefore, t -t ′ ≤ 2 ⟨ v,g ⟩ ∥ g ∥ 2 as well, and we can reverse the above chain of implications.

Note that the premise of Lemma 14 is exactly an instance of Fact 3 where ∇ f ( x ) and ∇ f ( x ′ ) are scalar multiples of the same direction, which is the case for GLMs by (24). Hence, Lemma 14 shows the contraction property in Fact 3 is preserved after clipping gradients (again, for GLMs).

We can now directly combine Lemma 8 and our contraction results, used to analyze the stability of Algorithm 5, with the iterative localization framework of [FKT20], Section 4.

Algorithm 8: OnePass -Clipped -DP -SGD ( D , n, X , x 0 , ρ )

Theorem 4. Consider an instance of k -heavy-tailed private SCO, following notation in Definition 4, let x ⋆ := argmin x ∈X F P ( x ) , and let ρ ≥ 0 . Further, assume that for a convex function σ , the sample functions f ( x ; s ) satisfy (23) for all s ∈ S ⊆ R d . Finally, assume f ( x ; s ) is β -smooth for all s ∈ S ,

where β ≤ max( √ n 2 · G 2 D , n · ( d n 2 ρ ) k -1 2 k · G k D ) . Algorithm 8 is a ρ -CDP algorithm which draws D ∼ P n , queries n sample gradients (using samples in D ), and outputs x I ∈ X satisfying

.

Proof. We begin with the privacy claim. Consider neighboring datasets D , D ′ , and suppose the datasets differ on the j th entry such that s j ∈ D i (if the differing entry is not in ∪ i ∈ [ I ] D i , Algorithm 8 clearly satisfies 0 -CDP). Let ¯ x i and ¯ x ′ i be the outputs of Line 8 when run with the same initialization x i -1 , and neighboring D i , D ′ i . By the assumption on β , since η i ≤ η for all i ∈ [ I ] , we can apply Fact 3 and Lemma 14 (recalling the characterization (24)) to show ∥ ¯ x i -¯ x ′ i ∥ ≤ 2 η i C i with probability 1 . Therefore, by our choice of σ i and the first and third parts of Lemma 1, the whole algorithm is ρ -CDP regardless of which D i contained the differing sample, since all other calls to OnePass -Clipped -SGD are 0 -CDP as we can couple all randomness used by the calls.

Next, we prove the utility claim. For simplicity, let ¯ x 0 := x ⋆ and ξ 0 := x 0 -x ⋆ , so ∥ ξ 0 ∥ ≤ D by assumption. By applying Lemma 8 for all i ∈ [ I ] with x 0 ← x i -1 and u ← ¯ x i -1 , we have

where the second line applied Lemma 2, the third used Jensen's inequality to bound E [ ∥ x I -¯ x I ∥ ] 2 ≤ E [ ∥ x I -¯ x I ∥ 2 ] and our assumption k ≥ 2 , and the last used the geometric decay of the different parameters. Finally, by plugging in our choices of C, η , we have

We can also check that the final summand is a low-order term, by using η ≤ 1 n · ( n 2 ρ 32 d ) k -1 2 k · 2 k +1 2 k D G k :

The conclusion follows by adjusting n , since Algorithm 8 is run with a sample count in [ n 2 , n ] .

E High-probability stochastic convex optimization

In this section, to highlight another application of our population-level localization framework, we show that it obtains improved high-probability guarantees for the following standard bounded-variance estimator parameterization of SCO in the non-private setting.

Definition 6 (Stochastic convex optimization) . Let X ⊂ R d be compact and convex, with diam ( X ) = D . In the stochastic convex optimization (SCO) problem, there is a convex function f : X → R , and we have query access to a stochastic oracle g : X → R d satisfying, for all x ∈ X ,

For a convex function ψ : X → R , our goal in SCO is to optimize the composite function f + ψ .

For instance, one can set ψ to the constant zero function to recover the non-composite variant of SCO. We include the composite variant of Definition 6 as it is a standard extension in the SCO literature, under the assumption that the function ψ is 'simple.' The specific notion of simplicity we use is that ψ : X → R admits an efficient proximal oracle (Definition 7).

Definition 7 (Proximal oracle) . Let X ⊂ R d be compact and convex. We say O is a proximal oracle for a convex function ψ : X → R if for any inputs v ∈ R d , η ∈ R ≥ 0 , O ( v ) returns

In Theorem 5, we give an algorithm which uses n queries to each of g and a proximal oracle for ψ , and achieves an error bound for f + ψ of

with probability ≥ 1 -δ . Similar rates are straightforward to derive using martingale concentration when the estimator g is assumed to satisfy heavier tail bounds, such as a sub-Gaussian norm. To our knowledge, the rate (25) was first attained recently by [CH24], who also proved a matching lower bound. Our Theorem 5 gives an alternative route to achieving this error bound. As was the case in several recent works in the literature [HS16, DDXZ21, Lia24] who studied high-probability variants of stochastic convex optimization, our Theorem 5 is based on using geometric aggregation techniques within a proximal point method framework (in our case, using Fact 2 within Algorithm 2). However, these aforementioned prior works all assume additional smoothness bounds on the function f .

We use the following standard result in the literature as a key subroutine.

Lemma 15 (Lemma 1, [ACJ + 21]) . In the setting of Definition 6, assume ψ is λ -strongly convex, let x ⋆ := argmin x ∈X f ( x ) + ψ ( x ) , and let T ∈ N . There is an algorithm which queries the stochastic oracle g and a proximal oracle for ψ each T times, and produces ¯ x satisfying, with probability ≥ 4 5 ,

We combine Lemma 15 with Proposition 2 to obtain the following high-probability SCO algorithm.

Theorem 5. Consider an instance of SCO, following notation in Definition 6, let n ∈ N , x ⋆ := argmin x ∈X f ( x ) + ψ ( x ) , and δ ∈ (0 , 1 2 ) . There is an algorithm using n queries to g and a proximal oracle for ψ and outputs x ∈ X satisfying, for a universal constant C sco , with probability ≥ 1 -δ ,

Proof. Assume without loss of generality that 1 δ is a sufficiently large constant (else we can adjust the constant factor C sco), and that n is sufficiently larger than log 1 δ (else the result holds because the range of the function is bounded by GD ). We instantiate Proposition 2 with F P ← f + ψ , I ← 1 2 log 2 n , and in each phase i ∈ [ I ] of Algorithm 2, we let n i := n 2 i . In the remainder of the proof, we describe how to implement (8) in the i th phase, where F P ← f + ψ , splitting into cases.

If 1 δ is bounded by polylog ( n ) and n is sufficiently large, suppose that n is a power of 4 , else we can use fewer queries and lose a constant factor in the guarantee. Then we can use a batch of n i consecutive queries, divided into 48 log( 1 δ i ) portions, where δ i := δ 2 i . We then use Lemma 15 on each portion of queries, with f ← f and ψ ← ψ + λ i 2 ∥· -x i -1 ∥ 2 ; it is straightforward to see that Definition 7 generalizes to give a proximal oracle for this new ψ . A Chernoff bound shows that at least 3 5 of the portions will return a point satisfying the bound in Lemma 15 except with probability δ i , so Fact 2 returns us a point at distance at most 90 G λ √ T from x ⋆ i , where

(accounting for rounding error). Therefore, (8) holds with

for sufficiently large C sco. Proposition 2 then implies that Algorithm 2 outputs x satisfying

where we use that G 1 ≤ G by Jensen's inequality and our second moment bound in Definition 6. The failure probability follows from a union bound because we ensured that ∑ i ∈ [ I ] δ i ≤ δ .

Finally, if 1 δ is larger than polylog ( n ) , then we let I, J ∈ N be chosen such that

which is achievable with I = O (log n ) and J = O (log log n δ ) = O (log 1 δ ) . Let m := n J , and assume without loss that m is a power of 2 , which we can guarantee by discarding ≤ 1 2 our queries, losing a constant factor in the error bound. The remainder of the proof follows identically to the first part of this proof, where we union bound over I phases, the i th of which uses J batches of m 2 i unused queries. Again we may apply Lemma 15 and Fact 2 with T = m 2 i , so (8) holds with

except with probability δ I . The conclusion then follows from Proposition 2.

F Non-contraction of truncated contractive steps

In this section, we demonstrate that a natural conjecture related to the performance of clipped private gradient algorithms in the smooth setting is false. We state this below as Conjecture 1. To motivate it, suppose v is the difference between a current pair of coupled iterates of a private gradient algorithm instantiated on neighboring datasets, and suppose the differing sample function has already been encountered. If we take a coupled gradient step in a sufficiently smooth function, Fact 3 shows that the step is a contraction. However, to preserve privacy in the heavy-tailed setting, it is natural to ask whether such a contractive step remains contractive after the gradients are clipped, i.e. the statement of Conjecture 1 (which gives the freedom for C to be lower bounded).

Conjecture 1. Let ∥ v ∥ 2 ≤ C for a sufficiently large constant C , and let ∥ v -( g -h ) ∥ ≤ ∥ v ∥ . Let g ′ = Π 1 ( g ) and h ′ = Π 1 ( h ) . 9 Then, ∥ v -( g ′ -h ′ ) ∥ ≤ C .

We strongly refute Conjecture 1, by disproving it for any C ≥ 0 . We remark that Lemma 16 does not necessarily rule out this approach to designing heavy-tailed DP-SCO algorithms in the smooth regime, but demonstrates an obstacle if additional structure of gradients is not exploited.

Lemma 16. Conjecture 1 is false for any choice of C ≥ 0 .

Proof. We give a 2 -dimensional counterexample. Let

Observe that

It is easy to verify ∥ v -( g -h ) ∥ = C at this point. Moreover,

For C ≥ 0 , the first coordinate of this vector is already less than -C .

G Non-decay of empirical squared bias

In this section, we present an obstacle towards a natural approach to improving the logarithmic terms in our algorithm in Section 3. We follow the notation of Section 3.1, i.e. for samples { i ≡ s i } i ∈ [ n ] ∼ P n , we define sample functions f i ≡ f ( · ; s i ) , and let

A basic bottleneck with known approaches following SCO-to-ERM reductions is that they require a strongly convex ERM solver as a primitive, due to known barriers to generalization in SCO without strong convexity (see e.g. discussion in [SSSSS09]). This poses an issue in the heavy-tailed setting, because standard analyses of strongly convex clipped SGD (see e.g. our Proposition 1) appear to suffer a dependence on b 2 D in the utility bound, which upon taking expectations requires bounding

Recall from Lemma 3 that it is straightforward to bound E b D ≤ G k k C k -1 , due to Fact 1. Bounding E b 2 D is more problematic; in [LR23], requiring this bound resulted in a dependence on G 2 k as opposed to G k (see the proof of Theorem 31), which we avoid (up to a polylogarithmic overhead) via our population-level localization strategy. We now present an alternative strategy to bound (27), avoiding a G 2 k dependence. Observe that, by using ( a + b + c ) 2 ≤ 3( a 2 + b 2 + c 2 ) ,

We focus on T 1 , as T 3 can be bounded by similar means (as truncation can only improve moment bounds), and T 2 ≤ G 2 k k C 2( k -1) via Fact 1. Hence, if we can show that T 1 = O ( G 2 2 n ) under the moment bound assumption in Assumption 1, we can avoid the logarithmic factors lost by our population localization approach. We suggest the following conjecture as an abstraction of this bound.

Conjecture 2. Let P be a distribution over S . For each x ∈ X , let g ( x ; s ) ∈ R d be a random vector, indexed by s ∼ S , satisfying E s ∼P [ g ( x ; s )] = 0 d and E s ∼P [sup x ∈X ∥ g ( x ; s ) ∥ 2 ] ≤ 1 . Finally for S ∼ P n and x ∈ X , let g ( x ; S ) := 1 n ∑ s ∈ S g ( x ; s ) . Then,

Note that the bound in Conjecture 2 exactly corresponds to T 1 in (28), after rescaling all sample gradients by 1 G 2 , and centering them by subtracting ∇ F P ( x ) . Hence, if Conjecture 2 is true, it would yield the following desirable bound in (28):

Moreover, it is simple to prove a bound of O (1) on the right-hand side of Conjecture 2, and as n →∞ it is reasonable to suppose g ( x ; S ) → 0 d for all x ∈ X . Nonetheless, we refute Conjecture 2 in full generality with a simple 1 -dimensional example.

Lemma 17. Conjecture 2 is false.

Proof. Let S = [0 , 1] and let P be the uniform distribution over S . Let X index a set of random g ( x ; · ) : [0 , 1] → [0 , 1] which are nonzero at finitely many points. 10 Then E s ∼P g ( x ; s ) = 0 for all x ∈ X , and g ( x ; s ) 2 ≤ 1 for all x ∈ X , s ∈ S . However, for any S ∈ [0 , 1] n , we have

While Lemma 17 does not rule out the approach suggested in (28) (or other approaches) to improve the analysis of strongly convex ERM solvers in heavy-tailed settings, it presents an obstacle to applying the natural decomposition strategy in (28). To overcome Lemma 17, one must either use more structure about the index set X or the iterates encountered by the algorithm, or consider a different decomposition strategy for bounding the squared empirical bias.

H Proof of Lemma 11

In this section, we prove Lemma 11. We first require the following standard fact (see e.g. [Sch14]).

Fact 4. Let X ⊆ R d be a convex set. Then for any x, y ∈ R d , we have

We now set up some notation. Let { ψ j : X → X} j ∈ [ T ] and { ϕ j : X → X} j ∈ [ T ] be two sequences of operations. We say that an operation pair ( ψ, ϕ ) is contractive if for any two points x, y ∈ X ,

We say an operation pair ( ψ, ϕ ) is ( C, ζ ) -contractive if for any x, y where ∥ x -y ∥ ≤ C , we have

Let ψ j ( x ) = ψ j ◦ ψ j -1 ◦ . . . ◦ ψ 1 ( x ) , and define ϕ j similarly, for all j ∈ [ T ] .

We prove Lemma 11 as a consequence of the following more general result.

Lemma 18. Let x 0 = x ′ 0 ∈ X , and consider two sequences of operations { ψ j : X → X} j ∈ [ T ] and { ψ ′ j : X → X} j ∈ [ T ] satisfying the following conditions, for c := ⌊ C ζ ⌋ .

For at least T -c -1 indices j ∈ [ T ] , ( ψ j , ϕ j ) is contractive.

At most one operation pair, ( ψ k , ψ k ) , is ( ∞ , C ) -contractive.

For at most c indices j ∈ [ T ] , ( ψ j , ϕ j ) is (2 C, ζ ) -contractive.

Then for all j ∈ [ T ] , we have that ∥ ψ j ( x 0 ) -ϕ j ( y 0 ) ∥ ≤ 2 C .

Proof. Define ∆ j := ∥ ψ j ( x 0 ) -ϕ j ( x ′ 0 ) ∥ for all j ∈ [ T ] . Let a j ≤ c be the total number of (2 C, ζ ) -contractive operation pairs ( ψ i , ϕ i ) where i ≤ j , and let b j be the 0 -1 indicator variable for k ≤ j . We use induction to show that ∆ j ≤ a j ζ + b j C . When j = 1 , the claim holds. Now if the claim holds for j -1 , then ∆ j -1 ≤ a j -1 ζ + b j -1 C ≤ 2 C . Hence, by definition,

which completes our induction. This also implies ∆ T ≤ 2 C as claimed.

Proof of Lemma 11. Throughout the following proof, note that ˆ c i ≤ 2 c deterministically (due to our use of BLap ( 3 ε , c ) noise), and under the stated parameter bounds,

Let { g i,j = Π C ( ∇ f ( x i,j ; s i,j )) } j ∈ [ n i ] and { g ′ i,j = Π C ( ∇ f ( x ′ i,j ; s ′ i,j )) } be the two truncated gradient sequences in the i th phase corresponding to the two datasets, and let { x i,j } j ∈ [ n i ] and { x ′ i,j } j ∈ [ n i ] be the corresponding iterate sequences. We set the operation sequences ψ j ( x ) := Π X ( x -η i g i,j ) and ϕ j ( x ) := Π X ( x -η i g ′ i,j ) . We bound the contractivity of these operation pairs and apply Lemma 18.

First, note that because count t , count ′ t < ˆ c i ≤ 2 c , the operation pair ( ψ j , ϕ j ) is an identical untruncated gradient mapping for at least t -2 c -1 indices j ∈ [ t ] . Because we assume each sample function f ( · ; s ) is β -smooth, it follows that for these indices j ∈ [ t ] , the operation pair ( ψ j , ϕ j ) is contractive, by applying Fact 3, Fact 4, and η i β ≤ 1 .

Next, recall the assumption that the datasets D , D ′ differ in the j th 0 sample only. Because ∥ g i,j 0 ∥ ≤ 9 C 8 + C 4 ≤ 11 C 8 by assumption, and similarly ∥ g ′ i,j 0 ∥ ≤ 11 C 8 , it follows that the operation pair ( ψ j 0 , ϕ j 0 ) is ( ∞ , 3 Cη i ) -contractive by applying the triangle inequality and Fact 4.

For all remaining indices j ∈ [ t ] , count t and count ′ t both incremented (under the assumption that Y i,j = Y ′ i,j for these indices). We claim that ( ψ j , ϕ j ) is (6 η i C, 12 η 2 i Cβ ) -contractive for these iterations. To see this, we bound

The first line used Fact 4, the second used the triangle inequality, and the last used Fact 3, Fact 4, and the fact that ∥∇ f ( x i,j ; s i,j ) -∇ f ( x ′ i,j ; s i,j ) ∥ ≤ 6 η i Cβ by smoothness, when ∥ x i,j -x ′ i,j ∥ ≤ 6 Cη i .

Finally, it suffices to apply Lemma 18 with C ← 3 Cη i , ζ ← 12 η 2 i Cβ , and c ← 2 c , which we can check meets the conditions of Lemma 18 under the stated parameter bounds.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide new algorithms with improved guarantees for heavy-tailed private SCO in several settings, which is what we claim in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses places where the main result is lossy, providing an appendix section dedicated towards improving it to be optimal. It also compares its results to another similar bound in the literature, discussing regimes where our bound is weaker.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide a full set of verifiable details for all of our theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: The paper does not include experiments requiring code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: The paper does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We read the ethics guidelines and believe we meet them.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We do not believe our paper poses a significant negative societal impact, as it is about making existing learning algorithms differentially private under less stringent distributional assumptions, which we do not foresee being used in any significant malicious cases. We do believe that our algorithms can have positive societal impacts, but do not wish to overclaim to this effect because our results are primarily theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release data or models.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
The paper investigates the problem of differentially private stochastic convex optimization (SCO) under the heavy-tailed setting and achieves a nearly optimal rate of $G_2 \cdot \frac{1}{\sqrt{n}}+G_k\left(\frac{\sqrt{d}}{n \varepsilon}\right)^{1-\frac{1}{k}}$. Specifically, it first provides results using Clipped-DP-SGD in the differentially private empirical risk minimization (DP-ERM) framework and then utilizes generalization techniques to offer similar results in the population case. Finally, it explores the heavy-tailed DPSCO.

STRENGTHS:
1. The paper provides a nearly optimal bound for DPSCO in the heavy-tailed setting.

WEAKNESSES:
1. Section 3 is not very clear. The connection between Sections 3.1 and 3.2 is not well explained.
2. The presentation needs improvement. One of the most important parts, Population-level Localization, is placed on the last page and is only briefly introduced and discussed. For example, how to choose parameters like $\lambda$ in Algorithm 2 and what $\Delta 4^i$ in Equation 8 represents should be explained.

QUESTIONS:
NA

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses differentially private stochastic convex optimization (DP-SCO) with heavy-tailed gradients, where previous assumptions of uniform Lipschitz constants are relaxed to bounded k-th moments. The authors introduce a new reduction-based framework that adapts strategies from the uniform Lipschitz setting, enabling optimal rates up to logarithmic factors under (ε,δ)-approximate differential privacy. They propose several algorithms, including an optimal algorithm for known Lipschitz constants, a near-linear time algorithm for smooth functions, and an optimal linear-time algorithm for smooth generalized linear models. A novel population-level localization framework is also presented, overcoming technical barriers and providing robust bounds on excess population loss without stringent gradient moment assumptions. This work advances the theoretical and practical understanding of DP-SCO with heavy-tailed gradients, outperforming previous approaches in handling real-world data challenges.\n\nSTRENGTHS:\n1. The paper introduces a novel reduction-based framework for handling heavy-tailed gradients in differentially private stochastic convex optimization (DP-SCO). This innovative approach enables the achievement of near-optimal rates and overcomes limitations of previous methods, marking a significant advancement in the field.\n2. The proposed method bypasses the need for bounding $\\mathbb{E}b_{\\mathcal{D}}^2$\n\nWEAKNESSES:\n1. Referring to Corollary 2 in Line 4 of Algorithm 3 reduces readability. To enhance clarity, it would be beneficial to specify the algorithm $\\mathcal{A}$ (from Corollary 2) clearly, at least within the proof. While leaving the optimization oracle as a black box might help generalize the framework, it makes the algorithm harder to follow if the instantiations are not highlighted.\n2. The motivation of the population-level localization could be further clarified.\n\nQUESTIONS:\n1. To solve population-level localization, could one use the same (or slightly adjusted) method for the localization of the empirical minimizer?\n2. How does the sample split parameter $J$ affect the utility of Algorithm 3?\n3. Should $G_2^2$ and $G_k^k$ on line 7 be $G_2$ and $G_k$?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-16"}

comments: SUMMARY:
.The paper studies DP-SGD under the assumption that the gradients have heavy-tailed phenomenon. This has been recently motivated and studied a lot by recent works. The authors claim to achieve optimal rate for this problem.

STRENGTHS:
They obtain the first optimal rates (up to logarithmic factors) in the heavy-tailed setting, achieving error that depends on $\frac{G_1}{\sqrt{n}} + G_k(\sqrt{d}/nk)^{1-1/k}$ under approximate-DP guarantee. 

They additionally study this problem under well-studied assumptions: Lipschitz constant assumption, smooth convex functions, and smooth generalized linear model.

I haven't checked the proof, but the paper is definitely an accept considering it solves a problem in this domain.

WEAKNESSES:
N/A

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper gives three main results.\n\nThe first is a nearly optimal (losing a few logarithmic factors) excess loss rate for differentially private stochastic convex optimization (DP-SCO) when the gradient norms have $k$ bounded moments. In particular, they achieve the optimal excess loss under $\\rho$-concentrated differential privacy (CDP) of\n\n$\\frac{G_2 D}{\\sqrt{n}} + G_kD \\cdot \\left(\\frac{\\sqrt{d}}{n\\sqrt{\\rho}} \\right)^{1-1/k}$\n\nwhere $G_i$ denotes the uniform upper bound on the $i$th moment of the gradients and $D$ is the diameter of the domain. This improves over the prior state of the art [LR23] by a factor of $G_kD \\cdot \\left(\\frac{\\sqrt{d}}{n\\sqrt{\\rho}} \\right)^{1/k}$. This improvement is increasingly pronounced as we decrease the number of gradient moments that we assume are bounded, which is probably the important regime in this work.\n\nAs I understand, the main technical innovation is a \"population-level localization framework\", through which the authors are able to control the population excess risk without having to argue about the variance of the bias term arising from gradient clipping.\n\nThe second main result is an algorithm that achieves the optimal excess risk when each sample function arrives with a known Lipschitz constant. This yields algorithms for privately learning a generalized linear model with the optimal excess risk. This follows from a clean reduction to the case where we know that our losses are uniformly Lipschitz.\n\nThe third main result is an improved query complexity for optimizing smooth functions. The algorithm follows from an application of the sparse vector technique.\n\nSTRENGTHS:\nThe algorithms seem very natural, the analysis looks clean, and the results yield quantitative improvements over prior work. Therefore, I think this work is an important contribution to private convex optimization.\n\nThe ideas are cleanly explained and at least on a surface level make sense to a complete outsider to the field (such as myself).\n\nWEAKNESSES:\nCan't really think of anything significant. A natural criticism I anticipate is that the improvement over [LR23] is pretty minor when $k$ is large, but given that the authors get the optimal result and [LR23] doesn't, this doesn't seem like a real issue.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of differentially private stochastic convex optimization with heavy-tailed gradients. This paper points out that in typical optimization research, the assumption of uniformly G-Lipschitz, while convenient for bounding sensitivity, does not always hold. Based on this weakness, the authors studied k-heavy-tailed DP-SCO. The author obtains near-optimal algorithms that lie in the clipped DP-SGD subroutine and ensure the private minimization of a regularized ERM problem under k-heavy-tailed DP-SCO conditions. This method yields points that closely approximate the minimizer of population loss, verified through Markov’s inequality.\n\nUpdates post response:\n\n- I find the lack of experiments a bit underwhelming, and the authors' response didn't really convince me why this should not be part of this paper. Data privacy is a highly practical question that should be of strong interest to practitioners. The lack of practical implications is thus concerning.\n\n- The paper does not have a conclusion section, which is very strange for a NeurIPS paper. I have been reviewing NeurIPS submissions for many years; It is really awkward for a paper to not have a conclusions section. Not having a conclusions section also means that the authors did not carefully think about the \"Limitations\" of their work, which is a requirement as per NeurIPS submission guidelines.\n\n- Lastly, I wonder what are some of the broader impacts/implications of this work. It would be helpful if the authors could add some discussions about this. Again, I feel like this would require a proper \"conclusions\" section added to the paper.\n\n- While reading through the paper I noticed many notational inconsistencies and concepts that have not been clearly explained (please see concrete examples under the list of questions below), I suggest the authors to carefully proof read their paper and making improvements on explaining their work. This will help broaden the impact of this research.\n\nDisclaimer: Please be aware that due to the rather short reviewing time, I have not been able to check all the proof details in the appendix.\n\nSTRENGTHS:\n- This paper provides a comprehensive analysis of prior work, enabling readers to quickly understand the limitations of previous research on DP-SCO and why the authors believe this work is necessary.\n- This paper conducts solid theoretical research and has a profound understanding of the k-heavy-tailed DP-SCO problem. The entire paper is filled with rigorous derivations, providing a theoretical foundation for future research.\n\nWEAKNESSES:\n- Experiments could be added to validate the proposed algorithm for readers to better understand the key takeaways.\n\n- The paper's organization could be improved, and the contribution can be better highlighted.\n\nQUESTIONS:\n- Why is the k-heavy-tailed assumption more common than the uniformly G-Lipschitz assumption? Intuitively, the uniform is indeed a weaker assumption, but I would like more direct evidence to prove the necessity of using the k-heavy-tailed assumption.\n\n- What are the comparisons between the bounds under the k-heavy-tailed assumption with those of prior work? Please comment on this\n\n- Is it possible to provide the convergence analysis of the proposed algorithm?\n\n- From equation (1) to equation (2), what's the difference between $G$ and $G_2$?\n\n- In line 45, can you give a specific example of what you mean by \"heavy-tailed gradients\"? When might readers expect this condition to hold in practice (e.g., in the training of a neural network)?\n\n- While citing a paper, can you please properly cite the author's name(s) using \\cite{}, as opposed to things like [LR23]?\n\n- In the introduction, can you please clarify exactly under what notion of DP your results will hold? Note: I'm seeing multiple versions of DP definitions in section 2, page 5, thus prompting this question.\n\n- Is Theorem 1 (at the bottom of page 9) supposed the main result in your paper?\n\n- What are some of the broader impacts/implications of your work? Please comment on this.\n\nLIMITATIONS:\nLimitations are discussed on page 32.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-19"}