Adapting to Function Difficulty and Growth Conditions in Private Optimization
Year: 2022

Abstract:
We develop algorithms for private stochastic convex optimization that adapt to the hardness of the specific function we wish to optimize. While previous work provide worst-case bounds for arbitrary convex functions, it is often the case that the function at hand belongs to a smaller class that enjoys faster rates. Concretely, we show that for functions exhibiting κ-growth around the optimum, i.e., f(x) ≥ f(x) + λκ−1‖x− x‖2 for κ > 1, our algorithms improve upon the standard √ d/nε privacy rate to the faster ( √ d/nε) κ κ−1 . Crucially, they achieve these rates without knowledge of the growth constant κ of the function. Our algorithms build upon the inverse sensitivity mechanism, which adapts to instance difficulty [2], and recent localization techniques in private optimization [25]. We complement our algorithms with matching lower bounds for these function classes and demonstrate that our adaptive algorithm is simultaneously (minimax) optimal over all κ ≥ 1 + c whenever c = Θ(1).

text: √ d/nε privacy rate to the faster ( √ d/nε) κ κ−1 . Crucially, they achieve
these rates without knowledge of the growth constant κ of the function. Our algorithms build upon the inverse sensitivity mechanism, which adapts to instance difficulty [2], and recent localization techniques in private optimization [25]. We complement our algorithms with matching lower bounds for these function classes and demonstrate that our adaptive algorithm is simultaneously (minimax) optimal over all κ ≥ 1 + c whenever c = Θ(1).

heading: 1 Introduction
text: Stochastic convex optimization (SCO) is a central problem in machine learning and statistics, where for a sample space S, parameter space X ⊂ Rd, and a collection of convex losses {F (·; s) : s ∈ S}, one wishes to solve
minimize x∈X f(x) := ES∼P [F (x;S)] = ∫ S F (x; s)dP (s) (1)
using an observed dataset S = Sn1 iid∼ P . While as formulated, the problem is by now fairly well-understood [12, 38, 29, 10, 37], it is becoming clear that, because of considerations beyond pure statistical accuracy—memory or communication costs [45, 26, 13], fairness [23, 28], personalization or distributed learning [35]—problem (1) is simply insufficient to address modern learning problems. To that end, researchers have revisited SCO under the additional constraint that the solution preserves the privacy of the provided sample [22, 21, 1, 16, 19]. A waypoint is Bassily et al. [7], who provide a private method with optimal convergence rates for the related empirical risk minimization problem, with recent papers focus on SCO providing (worst-case) optimal rates in various settings: smooth convex functions [8, 25], non-smooth functions [9], non-Euclidean geometry [5, 4] and under more stringent privacy constraints [34].
Yet these works ground their analyses in worst-case scenarios and provide guarantees for the hardest instance of the class of problems they consider. Conversely, they argue that their algorithms are optimal in a minimax sense: for any algorithm, there exists a hard instance on which the error achieved by the algorithm is equal to the upper bound. While valuable, these results are pessimistic—the exhibited hard instances are typically pathological—and fail to reflect achievable performance.
∗Equal contribution. Author order determined by coin toss.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this work, we consider the problem of adaptivity when solving (1) under privacy constraints. Importantly, we wish to provide private algorithms that adapt to the hardness of the objective f . A loss function f may belong to multiple problem classes, each exhibiting different achievable rates, so a natural desideratum is to attain the error rate of the easiest sub-class. As a simple vignette, if one gets an arbitrary 1-Lipschitz convex loss function f , the worst-case guarantee of any ε-DP algorithm is Θ(1/ √ n+d/(nε)). However, if one learns that f exhibits some growth property—say f is 1-strongly convex—the regret guarantee improves to the faster Θ(1/n + (d/(nε))2) rate with the appropriate algorithm. It is thus important to provide algorithms that achieves the rates of the “easiest” class to which the function belongs [32, 46, 18].
To that end, consider the nested classes of functions Fκ for κ ∈ [1,∞] such that, if f ∈ Fκ then there exists λ > 0 such that for all x ∈ X ,
f(x)− inf x′∈X f(x′) ≥ λ κ ‖x− x?‖κ2 .
For example, strong convexity implies growth with parameter κ = 2. This growth assumption closely relates to uniform convexity [32] and the Polyak-Kurdyka-Łojasiewicz inequality [11], and we make these connections precise in Section 2. Intuitively, smaller κ makes the function much easier to optimize: the error around the optimal point grows quickly. Objectives with growth are widespread in machine learning applications: among others, the `1-regularized hinge loss exhibits sharp growth (i.e. κ = 1) while `1- or `∞-constrained κ-norm regression —i.e. s = (a, b) ∈ Rd×R and F (x; s) = |b − 〈a, x〉|κ—has κ-growth for any κ integer greater than 2 [43]. In this work, we provide private adaptive algorithms that adapt to the actual growth of the function at hand.
We begin our analysis by examining Asi and Duchi’s inverse sensitivity mechanism [2] on ERM as a motivation. While not a practical algorithm, it achieves instance-optimal rates for any onedimensional function under mild assumptions, quantifying the best bound one could hope to achieve with an adaptive algorithm, and showing (in principle) that adaptive private algorithms can exist. We first show that for any function with κ-growth, the inverse sensitivity mechanism achieves privacy cost (d/(nε))κ/(κ−1); importantly, without knowledge of the function class Fκ, that f belongs to. This constitutes grounding and motivation for our work in three ways: (i) it validates our choice of sub-classes Fκ as the privacy rate is effectively controlled by the value of κ, (ii) it exhibits the rate we wish to achieve with efficient algorithms on Fκ and (iii) it showcases that for easier functions, privacy costs shrink significantly—to illustrate, for κ = 5/4 the privacy rate becomes (d/(nε))5.
We continue our treatment of problem (1) under growth in Section 4 and develop practical algorithms that achieve the rates of the inverse sensitivity mechanism. Moreover, for approximate (ε, δ)differential privacy, our algorithms improve the rates, achieving roughly ( √ d/(nε))κ/(κ−1). Our algorithms hinge on a reduction to SCO: we show that by solving a sequence of increasingly constrained SCO problems, one achieves the right rate whenever the function exhibits growth at the optimum. Importantly, our algorithm only requires a lower bound κ ≤ κ (where κ is the actual growth of f ).
We provide optimality guarantees for our algorithms in Section 5 and show that both the inverse sensitivity and the efficient algorithms of Section 4 are simultaneously minimax optimal over all classes Fκ whenever κ = 1 + Θ(1) and d = 1 for ε-DP algorithms. Finally, we prove that in arbitrary dimension, for both pure- and approximate-DP constraints, our algorithms are also simultaneously optimal for all classes Fκ with κ ≥ 2. On the way, we provide results that may be of independent interest to the community. First, we develop optimal algorithms for SCO under pure differential privacy constraints, which, to the best of our knowledge, do not exist in the literature. Secondly, our algorithms and analysis provide high-probability bounds on the loss, whereas existing results only provide (weaker) bounds on the expected loss. Finally, we complete the results of Ramdas and Singh [40] on (non-private) optimization lower bounds for functions with κ-growth by providing information-theoretic lower bounds (in contrast to oracle-based lower bounds that rely on observing only gradient information) and capturing the optimal dependence on all problem parameters (namely d, L and λ).

heading: 1.1 Related work
text: Convex optimization is one of the best studied problems in private data analysis [16, 19, 41, 7]. The first papers in this line of work mainly study minimizing the empirical loss, and readily establish that
the (minimax) optimal privacy rates are d/nε for pure ε-DP and √ d log(1/δ)/nε for (ε, δ)-DP [16, 7]. More recently, several works instead consider the harder problem of privately minimizing the population loss [8, 25]. These papers introduce new algorithmic techniques to obtain the worst-case optimal rates of 1/ √ n+ √ d log(1/δ)/nε for (ε, δ)-DP. They also show how to improve this rate to the faster 1/n + d log(1/δ)/(nε)2 in the case of 1-strongly convex functions. Our work subsumes both of these results as they correspond to κ = ∞ and κ = 2 respectively. To the best of our knowledge, there has been no work in private optimization that investigates the rates under general κ-growth assumptions or adaptivity to such conditions.
In contrast, the optimization community has extensively studied growth assumptions [40, 32, 15] and show that on these problems, carefully crafted algorithms improves upon the standard 1/ √ n for convex functions to the faster (1/ √ n)κ/(κ−1). [32] derives worst-case optimal (in the first-order oracle model) gradient algorithms in the uniformly convex case (i.e. κ ≥ 2) and provides technique to adapt to the growth κ, while [40], drawing connections between growth conditions and active learning, provides upper and lower bounds in the first-order stochastic oracle model. We complete the results of the latter and provide information-theoretic lower bounds that have optimal dependence on d, λ and n—their lower bound only holding for λ inversely proportional to d1/2−1/κ, when κ ≥ 2. Closest to our work is [15] who studies instance-optimality via local minimax complexity [14]. For one-dimensional functions, they develop a bisection-based instance-optimal algorithm and show that on individual functions of the form t 7→ κ−1|t|κ, the local minimax rate is (1/ √ n)κ/(κ−1).

heading: 2 Preliminaries
text: We first provide notation that we use throughout this paper, define useful assumptions and present key definitions in convex analysis and differential privacy.
Notation. n typically denotes the sample size and d the dimension. Throughout this work, x refers to the optimization variable, X ⊂ Rd to the constraint set and s to elements (S when random) of the sample space S. We usually denote by F : X × S→ R the (convex) loss function and for a dataset S = (s1, . . . , sn) ⊂ S, we define the empirical and population losses
fS(x) := 1
n ∑ i≤n F (x; si) and f(x) := ES∼P [F (x;S)].
We omit the dependence on P as it is often clear from context. We reserve ε, δ ≥ 0 for the privacy parameters of Definition 2.1. We always take gradients with respect to the optimization variable x. In the case that F (·; s) is not differentiable at x, we override notation and define ∇F (x; s) = argming∈∂F (x;s) ‖g‖2, where ∂F (x; s) is the subdifferential of F (·; s) at x. We use A for (potentially random) mechanism and Sn1 as a shorthand for (S1, . . . , Sn). For p ≥ 1, ‖ · ‖p is the standard `p-norm, Bdp(R) is the corresponding d-dimensional p-ball of radius R and p? is the dual of p, i.e. such that 1/p? + 1/p = 1. Finally, we define the Hamming distance between datasets dHam(S,S ′) := infσ∈Sn 1{si 6= s′σ(i)}, where Sn is the set of permutations over sets of size n.
Assumptions. We first state standard assumptions for solving (1). We assume that X is a closed, convex domain such that diam2(X ) = supx,y∈X ‖x − y‖2 ≤ D < ∞. Furthermore, we assume that for any s ∈ S, F (·; s) is convex and L-Lipschitz with respect to ‖ · ‖2. Central to our work, we define the following κ-growth assumption. Assumption 1 (κ-growth). Let x? = argminx∈X f(x). For a loss F and distribution P , we say that (F, P ) has (λ, κ) growth for κ ∈ [1,∞] and λ > 0, if the population function satisfies
for all x ∈ X , f(x)− f(x?) ≥ λ κ ‖x− x?‖κ2 .
In the case where P̂ is the empirical distribution on a finite dataset S, we refer to (λ, κ)-growth of (F, P̂ ) as κ-growth of the empirical function fS .
Uniform convexity and Kurdyka-Łojasiewicz inequality. Assumption 1 is closely related to two fundamental notions in convex analysis: uniform convexity and the Kurdyka-Łojasiewicz inequality.
Following [39], we say that h : Z ⊂ Rd → R is (σ, κ)-uniformly convex with σ > 0 and κ ≥ 2 if
for all x, y ∈ Z, h(y) ≥ h(x) + 〈∇h(x), y − x〉+ σ κ ‖x− y‖κ2 .
This immediately implies that (i) sums (and expectations) preserve uniform convexity (ii) if f is uniformly convex with λ and κ, then it has (λ, κ)-growth. This will be useful when constructing hard instances as it will suffice to consider (λ, κ)-uniformly convex functions which are generally more convenient to manipulate. Finally, we point out that, in the general case that κ ≥ 1, the literature refers to Assumption 1 as the Kurdyka-Łojasiewicz inequality [11] with, in their notation, ϕ(s) = (κ/λ)1/κs1/κ. Theorem 5-(ii) in [11] says that, under mild conditions, Assumption 1 implies the following inequality between the error and the gradient norm for all x ∈ X
f(x)− inf x′∈X
f(x′) ≤ e
λ 1 κ−1 ‖∇f(x)‖
κ κ−1 2 , (2)
This is a key result in our analysis of the inverse sensitivity mechanism of Section 3.
Differential privacy. We begin by recalling the definition of (ε, δ)-differential privacy. Definition 2.1 ([22, 21]). A randomized algorithm A is (ε, δ)-differentially private ((ε, δ)-DP) if, for all datasets S,S ′ ∈ Sn that differ in a single data element and for all events O in the output space of A, we have
Pr (A(S) ∈ O) ≤ e Pr (A(S ′) ∈ O) + δ.
We use the following standard results in differential privacy.
Lemma 2.1 (Composition [20, Thm. 3.16]). If A1, . . . ,Ak are randomized algorithms that each is (ε, δ)-DP, then their composition (A1(S), . . . ,Ak(S)) is (kε, kδ)-DP.
Next, we consider the Laplace mechanism. We will let Z ∼ Lapd(σ) denote a d-dimensional vector Z ∈ Rd such that Zi iid∼ Lap(σ) for 1 ≤ i ≤ d.
Lemma 2.2 (Laplace mechanism [20, Thm. 3.6]). Let h : Sn → Rd have `1-sensitivity ∆, that is, supS,S′∈Sn:dHam(S,S′)≤1 ‖h(S) − h(S
′)‖1 ≤ ∆. Then the Laplace mechanism A(S) = h(S) + Lapd(σ) with σ = ∆/ε is ε-DP.
Finally, we need the Gaussian mechanism for (ε, δ)-DP.
Lemma 2.3 (Gaussian mechanism [20, Thm. A.1]). Let h : Sn → Rd have `2-sensitivity ∆, that is, supS,S′∈Sn:dHam(S,S′)≤1 ‖h(S)− h(S
′)‖2 ≤ ∆. Then the Gaussian mechanism A(S) = h(S) + N(0, σ2Id) with σ = 2∆ log(2/δ)/ε is (ε, δ)-DP.
Inverse sensitivity mechanism. Our goal is to design private optimization algorithms that adapt to the difficulty of the underlying function. As a reference point, we turn to the inverse sensitivity mechanism of [2] as it enjoys general instance-optimality guarantees. For a given function h : Sn → T ⊂ Rd that we wish to estimate privately, define the inverse sensitivity at x ∈ T
lenh(S;x) = inf S′ {dHam(S ′,S) : h(S ′) = x}, (3)
that is, the inverse sensitivity of a target parameter y ∈ T at instance S is the minimal number of samples one needs to change to reach a new instance S ′ such that h(S ′) = y. Having this quantity, the inverse sensitivity mechanism samples an output from the following probability density
πAinv(S)(x) ∝ e −εlenh(S;x). (4)
The inverse sensitivity mechanism preserves ε-DP and enjoys instance-optimality guarantees in general settings [2]. In contrast to (worst-case) minimax optimality guarantees which measure the performance of the algorithm on the hardest instance, these notions of instance-optimality provide stronger per-instance optimality guarantees.

heading: 3 Adaptive rates through inverse sensitivity for ε-DP
text: To understand the achievable rates when privately optimizing functions with growth, we begin our theoretical investigation by examining the inverse sensitivity mechanism in our setting. We show that, for instances that exhibit κ-growth of the empirical function, the inverse sensitivity mechanism privately solves ERM with excess loss roughly (d/nε) κ κ−1 .
In our setting, we use a gradient-based approximation of the inverse sensitivity mechanism to simplify the analysis, while attaining similar rates. Following [3] with our function of interest h(S) := argminx∈X fS(x), we can lower bound the inverse sensitivity lenh(S;x) ≥ n‖∇fS(x)‖2/2L under natural assumptions. We define a ρ-smoothed version of this quantity which is more suitable to continuous domains
GρS(x) = inf y∈X :‖y−x‖2≤ρ ‖∇fS(y)‖2,
and define the ρ-smooth gradient-based inverse sensitivity mechanism
πAgr−inv(S)(x) ∝ e −εnGρS(x)/2L. (5)
Note that while exactly sampling from the un-normalized density πAgr−inv(S) is computationally intractable, analyzing its performance is an important step towards understanding the optimal rates for the family of functions with growth that we study in this work. The following theorem demonstrates the adaptivity of the inverse sensitivity mechanism to the growth of the underlying instance. We defer the proof to Appendix A. Theorem 1. Let S = (s1, . . . , sn) ∈ Sn, F (x; s) be convex, L-Lipschitz for all s ∈ S. Let x? = argminx∈X fS(x) and assume x
? is in the interior of X . Assume that fS(x) has κ-growth (Assumption 1) with κ ≥ κ > 1. For ρ > 0, the ρ-smooth inverse sensitivity mechanism Agr−inv (5) is ε-DP, and with probability at least 1− β the output x̂ = Agr−inv(S) has
fS(x̂)−min x∈X
fS(x) ≤ 1
λ 1 κ−1
( 2L(log(1/β) + d log(D/ρ))
nε
) κ κ−1
+ Lρ.
Moreover, setting ρ = (L/λ) 1 κ−1 (d/nε) κ κ−1 , we have
fS(x̂)−min x∈X
fS(x) ≤ 1
λ 1
κ−1 Õ
( Ld
nε
) κ κ−1
.
The rates of the inverse sensitivity in Theorem 1 provide two main insights regarding the landscape of the problem with growth conditions. First, these conditions allow to improve the worst-case rate d/nε to (d/nε) κ κ−1 for pure ε-DP and therefore suggest a better rate ( √ d log(1/δ)/nε) κ κ−1 is possible for approximate (ε, δ)-DP. Moreover, the general instance-optimality guarantees of this mechanism [2] hint that these are the optimal rates for our class of functions. In the sections to come, we validate the correctness of these predictions by developing efficient algorithms that achieve these rates (for pure and approximate privacy), and prove matching lower bounds which demonstrate the optimality of these algorithms.

heading: 4 Efficient algorithms with optimal rates
text: While the previous section demonstrates that there exists algorithms that improve the rates for functions with growth, we pointed out that Agr−inv was computationally intractable in the general case. In this section, we develop efficient algorithms—e.g. that are implementable with gradient-based methods—that achieve the same convergence rates. Our algorithms build on the recent localization techniques that Feldman et al. [25] used to obtain optimal rates for DP-SCO with general convex functions. In Section 4.1, we use these techniques to develop private algorithms that achieve the optimal rates for (pure) DP-SCO with high probability, in contrast to existing results which bound the expected excess loss. These results are of independent interest.
In Section 4.2, we translate these results into convergence guarantees on privately optimizing convex functions with growth by solving a sequence of increasingly constrained SCO problems—the highprobability guarantees of Section 4.1 being crucial to our convergence analysis of these algorithms.

heading: 4.1 High-probability guarantees for convex DP-SCO
text: We first describe our algorithm (Algorithm 1) then analyze its performance under pure-DP (Proposition 1) and approximate-DP constraints (Proposition 2). Our analysis builds on novel tight generalization bounds for uniformly-stable algorithms with high probability [24]. We defer the proofs to Appendix B.
Algorithm 1 Localization-based Algorithm Require: Dataset S = (s1, . . . , sn) ∈ Sn, constraint set X , step size η, initial point x0, privacy
parameters (ε, δ); 1: Set k = dlog ne and n0 = n/k 2: for i = 1 to k do 3: Set ηi = 2−4iη 4: Solve the following ERM over Xi = {x ∈ X : ‖x− xi−1‖2 ≤ 2Lηin0}:
Fi(x) = 1
n0 in0∑ j=1+(i−1)n0 F (x; sj) + 1 ηin0 ‖x− xi−1‖22
5: Let x̂i be the output of the optimization algorithm 6: if δ = 0 then 7: Set ζi ∼ Lapd(σi) where σi = 4Lηi √ d/εi 8: else if δ > 0 then 9: Set ζi ∼ N(0, σ2i ) where σi = 4Lηi √ log(1/δ)/ε

heading: 10: Set xi = x̂i + ζi
text: 11: return the final iterate xk
Proposition 1. Let β ≤ 1/(n + d), diam2(X ) ≤ D and F (x; s) be convex, L-Lipschitz for all s ∈ S. Setting
η = D
L min
( 1√
n log(1/β) ,
ε
d log(1/β) ) then for δ = 0, Algorithm 1 is ε-DP and has with probability 1− β
f(x)− f(x?) ≤ LD ·O (√ log(1/β) log3/2 n√
n + d log(1/β) log n nε
) .
Similarly, by using a different choice for the parameters and noise distribution, we have the following guarantees for approximate (ε, δ)-DP.
Proposition 2. Let β ≤ 1/(n + d), diam2(X ) ≤ D and F (x; s) be convex, L-Lipschitz for all s ∈ S. Setting
η = D
L min
( 1√
n log(1/β) , ε√ d log(1/δ) log(1/β)
) ,
then for δ > 0, Algorithm 1 is (ε, δ)-DP and has with probability 1− β
f(x)− f(x?) ≤ LD ·O (√ log(1/β) log3/2 n√
n +
√ d log(1/δ) log(1/β) log n
nε
) .
Before presenting our algorithms for functions with growth, we remark that the exact calculation of the ERM solution in step 5 of Algorithm 1 is not necessary; we chose it to clarify the main algorithmic ideas. However, as long as the returned solution x̂i is sufficiently accurate, say Fi(x̂i)− minx?i∈X Fi(x ? i ) ≤ ∆, we have that ‖x̂i − x?i ‖2 ≤ √ 2∆/λi where λi = 1/(ηin0). This implies that as long as ∆ ≤ Ln , the sensitivity of x̂i is at most twice the sensitivity of the exact ERM solution x?i and hence multiplying the noise σi by a factor of 2 is sufficient to guarantee privacy. As the set Xi is convex, finding sufficiently accurate x̂i can be done efficiently using standard optimization methods for minimizing convex functions over convex domains.

heading: 4.2 Algorithms for DP-SCO with growth
text: Building on the algorithms of the previous section, we design algorithms that recover the rates of the inverse sensitivity mechanism for functions with growth, importantly without knowledge of the value of κ. Inspired by epoch-based algorithms from the optimization literature [31, 29], our algorithm iteratively applies the private procedures from the previous section. Crucially, the growth assumption allows to reduce the diameter of the domain after each run, hence improving the overall excess loss by carefully choosing the hyper-parameters. We provide full details in Algorithm 2.
Algorithm 2 Epoch-based algorithms for κ-growth Require: Dataset S = (s1, . . . , sn) ∈ Sn, convex set X , initial point x0, number of iterations T ,
privacy parameters (ε, δ); 1: Set n0 = n/T and D0 = diam2(X ) 2: if δ = 0 then 3: Set η0 = D02L min ( 1√
n0 log(n0) log(1/β) , εd log(1/β) ) 4: else if δ > 0 then 5: η0 = D0 2L min ( 1√
n0 log(n0) log(1/β) , ε√ d log(1/δ) log(1/β) ) 6: for i = 0 to T − 1 do 7: Let Si = (s1+(i−1)n0 , . . . , sin0) 8: Set Di = 2−iD0 and ηi = 2−iη0 9: Set Xi = {x ∈ X : ‖x− xi‖2 ≤ Di}
10: Run Algorithm 1 on dataset Si with starting point xi, privacy parameter (ε, δ), domain Xi (with diameter Di), step size ηi 11: Let xi+1 be the output of the private procedure 12: return xT
The following theorem summarizes our main upper bound for DP-SCO with growth in the pure privacy model, recovering the rates of the inverse sensitivity mechanism in Section 3. We defer the proof to Appendix B.3. Theorem 2. Let β ≤ 1/(n+ d), diam2(X ) ≤ D and F (x; s) be convex, L-Lipschitz for all s ∈ S. Assume that f has κ-growth (Assumption 1) with κ ≥ κ > 1. Setting T = ⌈ 2 logn κ−1 ⌉ , Algorithm 2 is ε-DP and has with probability 1− β
f(xT )−min x∈X f(x) ≤ 1 λ 1 κ−1 · Õ
( L √
log(1/β)√ n + Ld log(1/β) nε(κ− 1)
) κ κ−1
,
where Õ hides logarithmic factors depending on n and d.
Sketch of the proof. The main challenge of the proof is showing that the iterate achieves good risk without knowledge of κ. Let us denote byD ·ρ the error guarantee of Proposition 1 (or Proposition 2 for approximate-DP). At each stage i, as long as x? = argminx∈X f(x) belongs to Xi, the excess loss is of order Di · ρ and thus decreases exponentially fast with i. The challenge is that, without knowledge of κ, we do not know the index i0 (roughly log2 n κ−1 ) after which x
? /∈ Dj for j ≥ i0 and the regret guarantees become meaningless with respect to the original problem. However, in the stages after i0, as the constraint set becomes very small, we upper bound the variations in function values f(xj+1) − f(xj) and show that the sub-optimality cannot increase (overall) by more than O(Di0 · ρ), thus achieving the optimal rate of stage i0.
Moreover, we can improve the dependence on the dimension for approximate (ε, δ)-DP, resulting in the following bounds. Theorem 3. Let β ≤ 1/(n+ d), diam2(X ) ≤ D and F (x; s) be convex, L-Lipschitz for all s ∈ S. Assume that f has κ-growth (Assumption 1) with κ ≥ κ > 1. Setting T = ⌈ 2 logn κ−1 ⌉ and δ > 0,
Algorithm 2 is (ε, δ)-DP and has with probability 1− β
f(xT )−min x∈X f(x) ≤ 1 λ 1 κ−1 · Õ
( L √
log(1/β)√ n
+ L √ d log(1/δ) log(1/β)
nε(κ− 1)
) κ κ−1
,
where Õ hides logarithmic factors depending on n and d.

heading: 5 Lower bounds
text: In this section, we develop (minimax) lower bounds for the problem of SCO with κ-growth under privacy constraints. Note that taking ε → ∞ provides lower bound for the unconstrained minimax risk. For a sample space S and collection of distributions P over S, we define the function class Fκ(P) as the set of convex functions from Rd → R that are L-Lipschitz and has κ-growth (Assumption 1). We define the constrained minimax risk [6]
Mn(X ,P,Fκ, ε, δ) := inf x̂n∈Aε,δ sup (F,P )∈Fκ×P
E [ f(x̂n(S n 1 ))− inf
x′∈X f(x′)
] , (6)
where A ,δ is the collection of (ε, δ)-DP mechanisms from Sn to X . When clear from context, we omit the dependency onP of the function class and simply writeFκ. We also forego the dependence on δ when referring to pure-DP constraints, i.e. Mn(X ,P,Fκ, ε, δ = 0) =: Mn(X ,P,Fκ, ε). We now proceed to prove tight lower bounds for ε-DP in Section 5.1 and (ε, δ)-DP in Section 5.2.

heading: 5.1 Lower bounds for pure ε-DP
text: Although in Section 4 we show that the same algorithm achieves the optimal upper bounds for all values of κ > 1, the landscape of the problem is more subtle for the lower bounds and we need to delineate two different cases to obtain tight lower bounds. We begin with κ ≥ 2, which corresponds to uniform convexity and enjoys properties that make the problem easier (e.g., closure under summation or addition of linear terms). The second case, 1 < κ < 2, corresponds to sharper growth and requires a different hard instance to satisfy the growth condition.
κ-growth with κ ≥ 2. We begin by developing lower bounds under pure DP for κ ≥ 2 Theorem 4 (Lower bound for ε-DP, κ ≥ 2). Let d ≥ 1, X = Bd2(R), S = {±ej}j≤d, κ ≥ 2 and n ∈ N. Let P be the set of distributions on S. Assume that
2κ−1 ≤ L λ 1 Rκ−1 ≤ 2κ−1
√ 96n and nε ≥ 1√
3
The following lower bound holds
Mn(X ,P,Fκ, ) ≥ 1
λ 1 κ−1 Ω̃
(( L√ n ) κ (κ−1) + ( Ld nε ) κ κ−1 ) . (7)
First of all, note that L ≥ λ2κRκ−1 is not an overly-restrictive assumption. Indeed, for an arbitrary (λ, κ)-uniformly convex and L-Lipschitz function, it always holds that L ≥ λ2R
κ−1. This is thus equivalent to assuming κ = Θ(1). Note that when κ 1, the standard n−1/2 +d/(nε) lower bound holds. We present the proof in Appendix C.1.1 and preview the main ideas here.
Sketch of the proof. Our lower bounds hinges on the collections of functions F (x; s) := aκ−1‖x‖κ2 + b〈x, s〉 for a, b ≥ 0 to be chosen later. These functions are [39, Lemma 4] κ-uniformly convex for any s ∈ S and in turn, so is the population function f . We proceed as follows, we first prove an information-theoretic (non-private) lower bound (Theorem 8 in Appendix C.1.1) which provides the statistical term in (7). With the same family of functions, we exhibit a collection of datasets and prove by contradiction that if an estimator were to optimize below a certain error it would have violated ε-DP—this yields a lower bound on ERM for our function class (Theorem 9 in Appendix C.1.1). We conclude by proving a reduction from SCO to ERM in Proposition 4.
κ-growth with κ ∈ (1, 2]. As the construction of the hard instance is more intricate for κ < 2, we provide a one-dimensional lower bound and leave the high-dimensional case for future work. In this case we directly obtain the result with a private version of Le Cam’s method [44, 42, 6], however with a different family of functions.
The issue with the construction of the previous section is that the function does not exhibit sharp growth for κ < 2. Indeed, the added linear function shifts the minimum away from 0 where the function is differentiable and as a result it locally behaves as a quadratic and only achieves growth κ = 2. To establish the lower bound, we consider a different sample function F that has growth exactly 1 on one side and κ on the other side. This yields the following Theorem 5 (Lower bound for ε-DP, κ ∈ (1, 2] ). Let d = 1, S = {−1,+1}, κ ∈ (1, 2], λ = 1, L = 2, and n ∈ N. There exists a collection of distributions P such that, whenever nε ≥ 1/ √ 3, it holds that
Mn([−1, 1],P,Fκd=1, ) = Ω {( 1√ n ) κ κ−1 + ( 1 nε ) κ κ−1 } . (8)

heading: 5.2 Lower bounds under approximate privacy constraints
text: We conclude our treatment by providing lower bounds but now under approximate privacy constraints, demonstrating the optimality of the risk bound of Theorem 3. We prove the result via a reduction: we show that if one solves ERM with κ-growth with error ∆, this implies that one solves arbitrary convex ERM with error φ(∆). Given that a lower bound of Ω( √ d/(nε)) holds for ERM, a lower bound of φ−1( √ d/(nε)) holds for ERM with κ-growth. However, for this reduction to hold, we require that κ ≥ 2. Furthermore, we consider κ to be roughly a constant—in the case that κ is too large, standard lower bounds on general convex functions apply. Theorem 6 (Private lower bound for (ε, δ)-DP). Let κ ≥ 2 such that κ = Θ(1), X = Bd2(D). Let d ≥ 1 and S = {±1/ √ d}d. Assume that nε = Ω( √ d), then for any (ε, δ) mechanism A, there exists λ > 0, F and S ⊂ S such that
E[fS(A(S))]− inf x′∈X fS(x ′) ≥ Ω̃  1 λ 1 κ−1 ( L √ d nε ) κ κ−1 . Theorem 6 implies that the same lower bound (up to logarithmic factors) applies to SCO via the reduction of [8, Appendix C]. Before proving the theorem, let us state (and prove in Appendix C.2) the following reduction: if an (ε, δ)-DP algorithm achieves excess error (roughly) ∆ on ERM for any function with κ-growth, there exists an (ε, δ)-DP algorithm that achieves error ∆(κ−1)/κ for any convex function. We construct the latter by iteratively solving ERM problems with geometrically increasing ‖ · ‖κ2 -regularization towards the previous iterate to ensure the objective has κ-growth. Proposition 3 (Solving ERM with κ-growth implies solving any convex ERM). Let κ ≥ 2. Assume there exists an ( , δ) mechanism A such that for any L-Lipschitz loss G on Y and dataset S such that gS(x) := 1n ∑ s∈S G(x; s) exhibits (λ, κ)-growth, the mechanism achieves excess loss
E[gS(A(S, G,Y))]− inf y′∈Y
gS(y ′) ≤ 1
λ 1 κ−1 ∆(n,L, , δ).
Then, we can construct an (ε, δ)-DP mechanism A′ such that for any L-Lipschitz loss f , the mechanism achieves excess loss
E[fS(A′(S))]− inf x′∈X fS(x ′) ≤ O
( D[∆(n,L, /k, δ/k)] κ−1 κ ) ,
where k is the smallest integer such that k ≥ log
[ κ 1 κ−1 L κ κ−1
22κ−3∆(n,L,ε/k,δ/k)
] .
With this proposition, the proof of the theorem directly follows as Bassily et al. [7] prove a lower bound Ω( √ d/(nε)) for ERM with (ε, δ)-DP.
Discussion
In this work, we develop private algorithms that adapt to the growth of the function at hand, achieving the convergence rate corresponding to the “easiest” sub-class the function belongs to. However, the picture is not yet complete. First of, there are still gaps in our theoretical understanding, the most interesting one being κ = 1. On these functions, appropriate optimization algorithms achieve linear convergence [43] and raise the question, can we achieve exponentially small privacy cost in our setting? Finally, while our optimality guarantees are more fine-grained than the usual minimax results over convex functions, they are still contigent on some predetermined choice of sub-classes. Studying more general notions of adaptivity is an important future direction in private optimization.

heading: Acknowledgments
text: The authors would like to thank Karan Chadha and Gary Cheng for comments on an early version of the draft. HA, DL and JCD were supported NSF under CAREER Award CCF-1553086 and HDR 1934578 (Stanford Data Science Collaboratory), Office of Naval Research YIP Award N00014-192288 and the Stanford DAWN Consortium.
Competing Interests
JCD has a consulting relationship with Apple. HS has spent internships at Apple during the 2019, 2020 and 2021 summers. DL has spent an internship at Google during the summer of 2020.

Review summaries:
This paper studies the SCO with loss function that satisfies growth conditions in the differential privacy model. Firstly, it considers the empirical risk function and provide an algorithm based on the inverse sensitivity mechanism. Then they show an efficient algorithm and the lower bounds of the problem in the both $\epsilon$ and $(\epsilon, \delta)$-DP models.

This paper presents a differentially private mechanism for releasing data and an optimisation algorithm that can use the released data and minimise a convex function. While there are many such algorithms in prior work, the current manuscript provides an algorithm that is adaptive and has convergence rate that improves with the "growth" of the function around the optimum point. Furthermore, the authors present matching lower bounds, which show that their algorithm is close to optimal.