Title: Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners
Venue: NeurIPS 2023
Accepted: yes
Keywords: differential privacy

empirical risk minimization

objective perturbation
Abstract: In the arena of privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) has outstripped the objective perturbation mechanism in popularity and interest. Though unrivaled in versatility, DP-SGD requires a non-trivial privacy overhead (for privately tuning the model’s hyperparameters) and a computational complexity which might be extravagant for simple models such as linear and logistic regression. This paper revamps the objective perturbation mechanism with tighter privacy analyses and new computational tools that boost it to perform competitively with DP-SGD on unconstrained convex generalized linear problems.

Improving the Privacy and Practicality of Objective Perturbation for Differentially Private Linear Learners

Abstract

In the arena of privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) has outstripped the objective perturbation mechanism in popularity and interest. Though unrivaled in versatility, DP-SGD requires a non-trivial privacy overhead (for privately tuning the model's hyperparameters) and a computational complexity which might be extravagant for simple models such as linear and logistic regression. This paper revamps the objective perturbation mechanism with tighter privacy analyses and new computational tools that boost it to perform competitively with DP-SGD on unconstrained convex generalized linear problems.

1 Introduction

The rise of deep neural networks has transformed the study of differentially private learning no less than any other area of machine learning. Differentially private stochastic gradient descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) has thus gained widespread appeal as a versatile framework for privately training deep learning models.

How does DP-SGD fare on simpler models such as linear and logistic regression? The verdict is unclear. Clearly an algorithm capable of privately optimizing non-convex functions represented by millions of parameters is up to the computational task of fitting a linear model. A more pressing concern is that DP-SGD is up to too much. Look, for example, at the algorithm's computational complexity: DP-SGD requires O ( n 2 ) steps to achieve the optimal excess risk bounds for DP convex empirical risk minimization (Bassily et al., 2014).

DP-SGD furthermore takes after its non-private counterpart in sensitivity to hyperparameters. A poor choice of learning rate or batch size, for instance, could lead to suboptimal performance or slow convergence. There are well-established procedures for hyperparameter optimization that typically involve evaluating the performance of the model trained using different sets of candidate hyperparameters. But with privacy constraints, there is a catch: tuning hyperparameters requires multiple passes over the training dataset and thereby constitutes a privacy cost.

At best, existing work tends to circumvent this obstacle by optimistically assuming the availability of a public auxiliary dataset for hyperparameter tuning. More often the procedure for private hyperparameter selection is left largely to the reader's imagination. Only recently have Liu & Talwar (2019) and subsequently Papernot & Steinke (2022) studied how to obtain tighter privacy loss bounds for this task beyond standard composition theorems.

In the meantime, objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012) has been to some extent shelved as a historical curiosity. Sifting through the literature, we find that opinions are divided: some tout objective perturbation as "[o]ne of the most effective algorithms for differentially private learning and optimization" (Neel et al., 2020), whereas other works (Wang et al., 2017) dismiss

objective perturbation as being impractical and restrictive. Some empirical evaluations (Yu et al., 2019; McKenna et al., 2021) suggest that DP-SGD often achieves better utility in practice than does objective perturbation; others (Iyengar et al., 2019) report the opposite.

Our goal in this paper is to lay some of this debate to rest and demonstrate that for generalized linear problems in particular, objective perturbation can outshine DP-SGD.

1.1 Our Contributions

We establish an improved ( ϵ, δ ) -DP bound for objective perturbation via privacy profiles, a modern tool for privacy accounting that bounds the hockey-stick divergence. The formula can be computed numerically using only calls to Gaussian CDFs. We further obtain a dominating pair of distributions as defined by Zhu et al. (2022) which enables tight composition and amplification by subsampling of the privacy profiles.

We present a novel Rényi differential privacy (RDP) (Mironov, 2017) analysis of the objective perturbation mechanism. Using this analysis, we show empirically that objective perturbation performs competitively against DP-SGD with 'honestly' 1 -tuned hyperparameters. The tightest analyses to date of private hyperparameter tuning are the RDP bounds derived in Papernot & Steinke (2022). This tool allows us to empirically evaluate objective perturbation against DP-SGD on a level playing field (Section 5).

We fix a decade-old oversight in the privacy analysis of objective perturbation. Existing literature overlooks a nuanced argument in the privacy analysis of objective perturbation, which requires a careful treatment of the dependence between the noise vector and the private minimizer. Without assuming GLM structure, the privacy bound of objective perturbation is subject to a dimensional dependence that has gone unacknowledged in previous work 2 .

We introduce computational tools that expand the applicability of objective perturbation to a broader range of loss functions. The privacy guarantees of objective perturbation require the loss function to have bounded gradient. Our proposed framework extends the Approximate Minima Perturbation framework of Iyengar et al. (2019) to take any smooth loss function as a blackbox, then algorithmically ensure that it has bounded gradient. We also provide a computational guarantee O ( n log n ) on the running time of this algorithm, in contrast to the O ( n 2 ) complexity of DP-SGD for achieving information-theoretic limits.

1.2 A Short History of DP Learning

Differentially private learning dates back to Chaudhuri et al. (2011), which extended the output perturbation method of Dwork et al. (2006) to classification algorithms and also introduced objective perturbation. In its first public appearance, objective perturbation required gamma-distributed noise; Kifer et al. (2012) provided a refined analysis of the mechanism with Gaussian noise, which is the entry point into our work.

Differentially private stochastic gradient descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) brought DP into the fold of modern machine learning, allowing private training of models with arbitrarily complex loss landscapes that can scale to enormous datasets. DP-SGD adds Gaussian noise at every iteration to an aggregation of clipped gradients, and thus privacy analysis for DP-SGD often boils down to finding tight composition bounds (of the subsampled Gaussian mechanism).

The initial version of DP-SGD based on the standard strong composition (Bassily et al., 2014) is not quite practical, but that has changed, thanks to a community-wide effort in developing modern numerical privacy accounting tools in the past few years. These include the moments accountant that composes Renyi DP functions (Abadi et al., 2016; Wang et al., 2019; Mironov et al., 2019) and the Fourier accountant (also known as PLV or PLD accountants) that directly compose the privacy profiles (Sommer et al., 2019; Koskela et al., 2020; Gopi et al., 2021; Zhu et al., 2022). It is safe to conclude that the numerically computed privacy loss of DP-SGD using these modern tools is now very precise.

Because DP-SGD releases each intermediate model, the algorithm can stop after any number of iterations and simply accumulates privacy loss as it goes. In contrast, the privacy guarantees of objective perturbation hold only when the output of the mechanism is the exact minima of the perturbed objective. This requirement is at odds with practical convex optimization frameworks which typically use first-order methods to search for an approximate solution.

To remedy this, Iyengar et al. (2019) proposed an approach to approximately minimize a perturbed objective function while maintaining privacy. Approximate Minima Perturbation (AMP) was introduced as a tractable alternative to objective perturbation whose privacy guarantees permit the output to be an approximate (rather than exact) solution to the perturbed minimization problem. In this paper we extend AMP to a broader range of loss functions; Algorithm 1 can be viewed as a special case of AMP with a transformation of the loss function.

2 Preliminaries

2.1 Differential Privacy

Differential privacy (DP) (Dwork et al., 2006) offers provable privacy protection by restricting how much the output of a randomized algorithm can leak information about a single data point.

DP requires a notion of how to measure similarity between datasets. We say that datasets Z and Z ′ are neighboring datasets (denoted Z ≃ Z ′ ) if they differ by exactly one datapoint z , i.e. Z ′ = Z ∪{ z } or Z ′ = Z \ { z } for some data entry z .

Definition 2.1 (Differential privacy) . A mechanism M : Z → R satisfies ( ϵ, δ )-differential privacy if for all neighboring datasets Z, Z ′ ∈ Z and output sets S ⊆ R ,

When δ > 0 , M satisfies approximate DP . When δ = 0 , M satisfies the stronger notion of pure DP .

We say that M is tightly ( ϵ, δ )-DP if there is no δ ′ < δ for which M would be ( ϵ, δ ′ )-DP.

In what follows, we overview two different styles of achieving DP guarantees: one via hockey-stick divergence, and the other via Rényi divergence.

2.1.1 DP via hockey-stick divergence

Definition 2.2 (Hockey-stick divergence) . Denote [ x ] + = max { 0 , x } for x ∈ R . For α > 0 the hockey-stick divergence H α from a distribution P to a distribution Q is defined as

Now (with some abuse of notation) we will discuss how to bound the hockey-stick divergence between distributions M ( Z ) and M ( Z ′ ) via the concept of privacy profiles .

Definition 2.3 (Privacy profiles Balle et al., 2018) . The privacy profile δ M ( ϵ ) of a mechanism M is defined as

Tight ( ϵ, δ ) -DP bounds can then be obtained as follows.

Lemma 2.4 (Zhu et al., 2022, Lemma 5) . Mechanism M satisfies ( ϵ, δ ) -DP if any only if δ ≥ δ M ( ϵ ) .

Dominating pairs of distributions are useful for bounding the hockey-stick divergence H e ϵ ( M ( Z ) ||M ( Z ′ ) ) accurately and, in particular, for obtaining tight bounds for compositions.

Definition 2.5 (Zhu et al. 2022) . A pair of distributions ( P, Q ) is a dominating pair of distributions for mechanism M : Z → R if for all neighboring datasets Z and Z ′ and for all α > 0 ,

2.1.2 DP via Rényi divergence

Definition 2.6. (Rényi divergence.) Let α > 0 . For α = 1 , the Rényi divergence D α from distribution P to distribution Q is defined as

̸

When α = 1 , Rényi divergence reduces to the Kullback-Leibler (KL) divergence:

Rényi differential privacy (RDP) is a relaxation of pure DP ( δ = 0 ) based on Rényi divergence.

Definition 2.7 (Rényi differential privacy) . A mechanism M : Z → R satisfies ( α, ϵ )-Rényi differential privacy if for all neighboring datasets Z, Z ′ ∈ Z ,

RDP implies ( ϵ, δ ) -DP for any 0 < δ ≤ 1 with ϵ = min α> 1 { ϵ ( α ) + log(1 /δ ) / ( α -1) } . Tighter but more complex conversion formulae were derived by Balle et al. (2020) and Canonne et al. (2020), which we adopt numerically in our experiments whenever approximate DP is needed.

2.2 Differentially Private Empirical Risk Minimization

We have a dataset Z ∈ Z and a loss function ℓ ( θ ; z ) ; we want to solve problems of the form

where z = ( x, y ) ∈ X × Y is a data point and r ( θ ) is a regularization term. The feature space is X ⊆ R d and the label space is Y ⊆ R . We will assume that || x || 2 ≤ 1 and | y | ≤ 1 .

This work focuses on unconstrained convex generalized linear models (GLMs): we require that ℓ ( θ ) and r ( θ ) are convex and twice-differentiable and that Θ = R d . The loss function is assumed to have GLM structure of the form ℓ ( θ ; z ) = f ( x T θ ; y ) .

Objective Perturbation Construct the perturbed objective function by sampling b ∼ N (0 , σ 2 I d ) :

The objective perturbation mechanism (ObjPert) outputs ˆ θ P ( Z ) = arg min θ ∈ Θ L P ( θ ; Z, b ) .

Theorem 2.8 (DP guarantees of objective perturbation (Kifer et al., 2012)) . Let ℓ ( θ ; z ) be convex and twice-differentiable such that ||∇ ℓ ( θ ; z ) || 2 ≤ L and ∇ 2 ℓ ( θ ; z ) ≺ βI d for all θ ∈ Θ and z ∈ X × Y . √

Then objective perturbation satisfies ( ϵ, δ )-DP when λ ≥ 2 β ϵ and σ ≥ L 8 log(2 /δ )+4 ϵ ϵ .

Differentially Private Gradient Descent DP-SGD is a differentially private version of stochastic gradient descent which ensures privacy by clipping the per-example gradients at each iteration before aggregating them and adding noise to the result. The update rule at iteration t is given by

where η t is the learning rate at iteration t , B t is the current batch at iteration t , σ is the noise scale, and clip is a function that bounds the norm of the per-example gradients.

3 Analytical Tools

Existing privacy guarantees of the objective perturbation mechanism (Chaudhuri et al., 2011; Kifer et al., 2012) pre-date modern privacy accounting tools such as Rényi differential privacy and privacy profiles. In this section, we present two new privacy analyses of objective perturbation: an ( ϵ, δ ) -DP bound based on privacy profiles, and an RDP bound.

3.1 Approximate DP Bound

Theorem 3.1 (Approximate DP guarantees of objective perturbation for GLMs) . Consider a loss function ℓ ( θ ; z ) = f ( x T θ ; y ) with GLM structure. Suppose that f is β -smooth and ||∇ ℓ ( θ ; z ) || 2 ≤ L for all θ ∈ Θ and z ∈ X × Y . Fix λ > β . Let ϵ ≥ 0 and let ˜ ϵ = ϵ -log ( 1 -β λ ) , ˆ ϵ = ϵ -log ( 1 -β λ ) -L 2 2 σ 2 , and let P and Q be the density functions of N ( L, σ 2 ) and N (0 , σ 2 ) , respectively. Objective perturbation satisfies ( ϵ , δ ( ϵ ) ) -DP for

Notice that we can express (3.1) analytically using (B.1). To obtain the bound (3.1) we repeatedly use the fact that the privacy loss random variable (PLRV) determined by the distributions N (1 , σ 2 ) and N (0 , σ 2 ) is distributed as N ( 1 2 σ 2 , 1 σ 2 ) . As the upper bound (3.1) is obtained using a PLRV that is a certain scaled and shifted half-normal distribution, we can also find certain scaled and shifted half-normal distributions P and Q which give the dominating pair of distributions for the objective perturbation mechanism such that the hockey-stick divergence between P and Q is exactly the upper bound (3.1) for all ϵ (shown in the appendix).

3.2 Rényi Differential Privacy Bound

If our sole objective is to obtain the tightest possible approximate DP bounds for objective perturbation, we can stop at Theorem 3.1! Directly calculating the privacy profiles of objective perturbation using the hockey-stick divergence, as in the previous section, will achieve this goal (until more privacy accounting tools come along).

In this section we turn instead to Rényi differential privacy, a popular relaxation of pure differential privacy ( δ = 0 ) which avoids the 'catastrophic privacy breach' possibility permitted by approximate DP ( δ > 0 ). Below, we present an RDP guarantee for objective perturbation.

Theorem 3.2 (RDP guarantees of objective perturbation for GLMs) . Consider a loss function ℓ ( θ ; z ) = f ( x T θ ; y ) with GLM structure. Suppose that f is β -smooth and ||∇ ℓ ( θ ; z ) || 2 ≤ L for all θ ∈ Θ and z ∈ X × Y . Fix λ > β . Objective perturbation satisfies ( α, ϵ )-RDP for any α > 1 with

For α = 1 , the RDP bound holds with

One of our main motivations for improving the privacy analysis of objective perturbation comes from the observation that it can be competitive to DP-SGD when the privacy cost of hyperparameter tuning is included in the privacy budget. As the tightest results for DP hyperparameter tuning are stated in terms of RDP (Papernot & Steinke, 2022), in our experiments we use RDP bounds of objective perturbation to get a clear understanding of the differences in the privacy-utility trade-offs between these two approaches.

Remark 3.3 . Privacy profile and RDP bounds (such as Theorems 3.1 and 3.2) are unified in the sense that they are both based on a certain bound of the PLRV ϵ Z,Z ′ (Definition K.4) for a fixed pair of datasets Z, Z ′ . From Definitions 2.2 and 2.7 we see that for ϵ ∈ R , the hockey-stick divergence is

and for α > 1 we have that the Rényi divergence is

3.3 Distance to Optimality

How close to optimal are the bounds of Theorems 3.1 and 3.2 ? We can in fact show that the Gaussian mechanism is a special case of the objective perturbation mechanism - thereby providing a lower bound on its approximate DP and RDP.

Example 3.4 . Consider the loss function ℓ ( θ ; x ) = x T θ and choose neighboring datasets X = { x } and X ′ = ∅ , for some x ∈ R d . Fix λ > 0 and sample b ∼ N (0 , σ 2 I d ) . Then the objective perturbation mechanism solves

Observe that ˆ θ P ( X ) ∼ N ( -1 λ x, σ 2 λ 2 I d ) and ˆ θ P ( X ′ ) ∼ N (0 , σ 2 λ 2 I d ) . Following the problem setting described in Theorem 2.8, we have that || x || 2 = ||∇ ℓ ( θ ; x ) || 2 ≤ L . In this case, objective perturbation reduces directly to the Gaussian mechanism with sensitivity ∆ f = L λ and noise scale σ λ . Corollary 3.5 . As a consequence of Example 3.4 and the scaling invariance of the hockey-stick divergence, for all α > 1 we have the following:

The argument works the same for the Rényi divergence D α which is similarly scaling-invariant. Corollary 3.5 implies that we can measure the tightness of the bounds given in Theorems 3.1 and 3.2 by comparing them to the tight bounds of the Gaussian mechanism (B.1) with sensitivity ∆ f = L and noise scale σ .

This means that in Figure 1, the hockey-stick divergence of the Gaussian mechanism is a lower bound on the hockey-stick divergence for objective perturbation. While our hockey-stick divergence bound is unsurprisingly a bit tighter than the RDP bound for objective perturbation, we see that both significantly improve over the classic ( ϵ, δ ) -DP bounds of Kifer et al. (2012).

0.2

0.4

0.6

0.8

1.0

1.2

10

11

10

9

10

7

10

5

10

3

10

1

10 1

RDP for ObjPert

HS for ObjPert

HS for GM

0.2

0.4

0.6

0.8

1.0

1.2

10

11

10

9

10

7

10

5

10

3

10

1

10 1

RDP for ObjPert

HS for ObjPert

HS for GM

Remark 3.6 . The RDP and approximate DP bounds in this section require a careful analysis of the dependence between the noise vector b and the private minimizer θ P . In the appendix, we show how the GLM assumption simplifies this issue.

4 Computational Tools

In this section we present Algorithm 1, which extends the Approximate Minima Perturbation of Iyengar et al. (2019) to handle loss functions with unbounded gradient.

Approximate minima The privacy guarantees of objective perturbation hold only when its output is an exact minimizer of the perturbed objective. Approximate Minima Perturbation (AMP) (Iyengar et al., 2019) addresses this issue by finding an approximate minimizer to the perturbed objective, then privately releases this approximate minimizer with the Gaussian mechanism.

Gradient clipping DP-SGD requires no a priori bound on the gradient of the loss function; at each iteration, the algorithm clips the per-example gradients above a pre-specified threshold in order to bound the gradient norms. We extend this same technique to objective perturbation.

Given a loss function ℓ ( θ ; z ) and a clipping threshold C , we can transform the gradient of ℓ ( θ ; z ) as follows:

Then we can define the aggregation of clipped gradients as ∇L C ( θ ; Z ) = ∑ z ∈ Z ∇ ℓ C ( θ ; z ) .

The aggregation of clipped gradients ∇L C ( θ ; Z ) corresponds to an implicit "clipped-gradient" objective function L C ( θ ; Z ) . For convex GLMs, Song et al. (2020) define this function precisely and show that it retains the convexity and GLM structure of the original objective function L ( θ ; Z ) . We furthermore demonstrate that this function retains the same bound β on the Lipschitz smoothness (Theorem E.3).

Algorithm 1 extends the privacy guarantees of AMP (Iyengar et al., 2019) to loss functions with unbounded gradient. Notice that for smooth loss functions with gradient norm bounded by L , we can set C = L in order to recover Approximate Minima Perturbation.

Algorithm 1 Computational Objective Perturbation

Input: dataset Z ; noise levels σ, σ out ; β -smooth loss function ℓ ( · ) ; regularization strength λ ; gradient norm threshold τ ; clipping threshold C .

Construct the set of clipped-gradient loss functions { ℓ C ( θ ; z ) : z ∈ Z } .

Sample b ∼ N (0 , σ 2 I d ) .

Let L P C ( θ ; Z, b ) = ∑ z ∈ Z ℓ C ( θ ; z ) + λ 2 || θ || 2 2 + b T θ .

Output: ˜ θ P = ˜ θ + N (0 , σ 2 out I d ) .

Solve for ˜ θ such that ||∇L P C ( ˜ θ ; Z ) || 2 ≤ τ .

Theorem 4.1 (RDP guarantees of Algorithm 1) . Consider a loss function ℓ ( θ ; z ) = f ( x T θ ; y ) with GLM structure, such that f is β -smooth. Fix λ > β . Algorithm 1 satisfies ( α, ϵ )-RDP for any α > 1 with

Remark 4.2 . Gradient clipping aside, our proof of Theorem 4.1 takes a different tack than the proof of Theorem 1 (for AMP) in Iyengar et al. (2019). We observe that Algorithm 1 is essentially an adaptive composition of the objective perturbation mechanism and the Gaussian mechanism. We can write ˜ θ = θ P +( ˜ θ -θ P ) to see that we are releasing two quantities: θ P (with objective perturbation) and the difference ˜ θ -θ P (with the Gaussian mechanism). Algorithm 1 stops iterating only after the gradient norm ||∇L P C ( ˜ θ ; Z ) || 2 is below the threshold τ . This along with the λ -strong convexity of the objective function ∇L P C ( θ ; Z ) ensures a bound on the ℓ 2 -sensitivity of the difference ˜ θ -θ P , so that we can apply the Gaussian mechanism.

4.1 Computational Guarantee

To achieve the optimal excess risk bounds for DP-ERM in the convex setting, DP-SGD clocks in at a hefty O ( n 2 ) gradient evaluations (Bassily et al., 2014). It has been an open problem to obtain optimal DP-ERM algorithms that runs in subquadratic time (Kulkarni et al., 2021). One of our contributions is to show that when we further restrict to smooth GLM-losses (so ObjPert is applicable) Algorithm 1 can achieve the same optimal rate with only O ( n log n ) gradient evaluations.

A formal claim and proof that Algorithm 1 - with appropriately chosen parameters - achieves the optimal rate is deferred to Appendix H. The analysis is largely the same as those in (Kifer et al., 2012) but with the bug fixed (details in Appendix G) by adding a GLM assumption.

The improved computational complexity is due to that we can apply any off-the-shelf optimization algorithms to solve Step 4 of Algorithm 1. Observe that L P ( θ ; Z, b ) has a finite-sum structure, we can employ the Stochastic Averaged Gradient (SAG) method (Schmidt et al., 2017) which halts in O ( n log n ) with high probability. Details are provided in Appendix F.

5 Empirical Evaluation

In this section we evaluate Algorithm 1 against two baselines: 'dishonest' DP-SGD and 'honest' DP-SGD. Dishonest DP-SGD does not account for the privacy cost of hyperparameter tuning; honest DP-SGD follows the private selection algorithm and RDP bounds from Papernot & Steinke (2022).

Our experimental design includes some guidelines in order to make it a fair fight. One of the strengths of Algorithm 1 that we advocate for is its blackbox optimization. Whereas DP-SGD consumes privacy budget for testing each set of hyperparameter candidates, an advantage of approximate minima perturbation is that the privacy guarantees are independent of the choice of optimizer used to solve for ˜ θ . We can therefore test out any number of optimization hyperparameters for Algorithm 1 at no additional privacy cost , provided that these parameters are independent of the privacy guarantee (e.g. learning rate, batch size). More specifically, once the loss function is perturbed with the noise b in Algorithm 1, any ˜ θ that satisfies the convergence guarantees with the tolerance parameter τ will have the RDP-guarantees of Theorem 4.1 and therefore we are free to also carry out tuning of the optimization algorithm without an additional privacy cost.

Because we are interested in measuring the effect of the privacy cost of hyperparameter tuning, we tune only the learning rate which does not affect the privacy guarantee of the base algorithm. This isolates the effect of hyperparameter tuning as we will need to appeal to Papernot & Steinke (2022) to get valid DP bounds for DP-SGD, but Algorithm 1 enjoys hyperparameter tuning 'for free'.

The following table summarizes the optimization-related parameters for all three methods.

The choice of C = √ 2 is a natural value for logistic regression in that ||∇ ℓ ( θ, z ) || ≤ √ 2 for all θ, z due to data-preprocessing and the bias term. Dishonest DP-SGD selects s = 10 learning rate candidates evenly log-spaced from the range of values between 10 -8 and 10 -1 . Honest DP-SGD selects learning rate candidates from the same range of values, but with granularity determined by a random variable K sampled from the Poisson distribution Poisson( µ ) . We select µ so that with 90% probability, K is larger than the grid size s used for dishonest DP-SGD, resulting in µ ≈ 15 . 4 .

We use the Adam optimizer for both honest and dishonest DP-SGD. For Algorithm 1 we use the L-BFGS optimizer whose second-order behavior allows us to get within a smaller distance to optimal (as required by Algorithm 1).

For DP-SGD we set the subsampling ratio such that the expected batch size is 256 and we run for 60 "epochs". We calculate the number of iterations as T = 60 · num_batches , where num_batches is the number of batches in the training dataset (we pass the train loader through the Opacus privacy engine, so the size of each batch is random). To calibrate the scale of the noise for DP-SGD, we use the analytical moments accountant (Wang et al., 2019) with Poisson sampling (Zhu & Wang, 2019; Mironov et al., 2019).

The parameters specific to AMP are σ out , the noise scale for the output perturbation step; and τ , the gradient norm bound. A larger τ will improve our computational cost, though our approximate minimizer ˜ θ will be farther away from the true minimizer θ P . We can achieve a smaller τ by choosing a larger σ out , but this will mean that our release of ˜ θ will be noisier. In our experiments we fix τ = 0 . 01 and σ out = 0 . 15 .

The privacy parameters of objective perturbation are the noise scale σ and the regularization strength λ . Balancing these parameters is a classic exercise in bias-variance trade-off. A larger σ will allow us to use less regularization, but if σ is too large then we risk adding too much noise to the objective function and hurting utility.

Our strategy is to find the smallest possible λ such that σ isn't too large. To quantify when σ is 'too large', we use the Gaussian mechanism as a reference point: the noise scale σ for objective perturbation shouldn't be too much larger than the noise scale σ G for the Gaussian mechanism. Let's say that the Gaussian mechanism with noise scale σ G satisfies ( ϵ, δ )-DP, then we want our σ for ( ϵ, δ ) -DP objective perturbation to satisfy σ ≤ fσ G for some small constant factor f . In our experiments, we set f = 1 . 3 .

For objective perturbation, we can thus select the privacy parameters σ and λ using fixed values (e.g., ϵ, δ, σ G ) that are independent of the data. Likewise, the choices of σ out and τ are fixed across all datasets. This is noteworthy since σ, λ, σ out and τ each have an effect on the privacy guarantee, outside of the blackbox algorithm. Tuning these parameters on the data would require us to use the private selection algorithm.

10 -1

10 0

10 1

Privacy-

/epsilon1

70

80

90

Test Accuracy

Synthetic-L

DP-Adam (dishonest)

Algorithm 1

DP-Adam (honest)

10 -1

10 0

10 1

Privacy-

/epsilon1

76

78

80

82

84

Test Accuracy

Adult

DP-Adam (dishonest)

Algorithm 1

DP-Adam (honest)

10 -1

10 0

10 1

Privacy-

/epsilon1

50

60

70

80

90

Test Accuracy

Synthetic-H

DP-Adam (dishonest)

Algorithm 1

DP-Adam (honest)

10 -1

10 0

10 1

Privacy-

/epsilon1

60

70

80

90

Test Accuracy

DP-Adam (dishonest)

Algorithm 1

DP-Adam (honest)

We evaluate our methods for binary classification on the Adult, Synthetic-L, Synthetic-H and Gisette datasets provided by Iyengar et al. (2019). We normalize each row x i to have unit ℓ 2 -norm. Note

that the assignment x i ← x i || x i || 2 doesn't require expending any privacy budget as each data point is transformed only by its own per-sample norm.

Results from Figure 5 in numerical format for the low-dimensional datasets, Synthetic-L and Adult. For ϵ = 0 . 1 , these can be cross-referenced with the results in Fig. 3 from (Iyengar et al., 2019).

The experimental results shown in Figure 5, Table 1 and Table 2 paint a consistent picture. While dishonest DP-SGD is clearly the best-performing algorithm, when we account for the cost of hyperparameter tuning then Algorithm 1 can typically best honest DP-SGD. This effect is especially pronounced under small ϵ , for which diverting some of the limited privacy budget to hyperparameter tuning could be more impactful.

Is it fair? Our experimental design aims to fairly compare ObjPert to DP-SGD. One limitation, however, is that the state-of-the-art tools for private hyperparameter tuning from Papernot & Steinke (2022) are RDP bounds - and RDP is not state-of-the-art for DP-SGD privacy accounting. At this moment, the tighest privacy accounting tool for DP-SGD is the PRV accountant from Gopi et al. (2021), which is the counterpart to our privacy profiles analysis for ObjPert (Theorem 3.1). Unfortunately, even though dishonest DP-SGD would benefit from using the PRV accountant, for private hyperparameter tuning we would then have to use the sub-optimal private selection bounds for approximate DP from Liu & Talwar (2019). In our experiments we therefore use RDP-based privacy accounting for both ObjPert and DP-SGD. Comparing DP-SGD with numerical composition against ObjPert with Theorem 3.1 will have to wait until more private selection tools are available.

One might also object that by tuning only the learning rate for DP-SGD, we didn't explore the full range of hyperparameters relevant to DP-SGD's performance. While tuning additional hyperparameters such as the batch size and number of epochs could benefit dishonest DP-SGD, it would likely worsen the privacy-utility tradeoff for honestly-tuned DP-SGD due to the increased privacy cost of the hyperparameter tuning algorithm from Papernot & Steinke (2022).

6 Conclusion

One point that we really wanted to drive home is that while DP-SGD works extraordinarily well across a wide variety of problem settings, it's not necessarily the best solution for every problem setting. But at the same time, DP-SGD has received the benefit of an enormous amount of attention that other DP learning algorithms haven't received. A goal of our paper was to hone in on a particular problem setting and give a different algorithm the same star treatment.

Objective perturbation now boasts two new privacy analyses. One is an improved ( ϵ, δ ) -DP analysis based on bounding the hockey-stick divergence. The other is an RDP analysis which allows us to fairly compare objective perturbation against DP-SGD - the workhorse of differentially private learning -with honest hyperparameter tuning. We've also expanded the approximate minima perturbation algorithm of Iyengar et al. (2019) in order to encompass a broader range of loss functions which need not have bounded gradient. Our algorithm moreover can be used in conjunction with SVRG to guarantee a running time of O ( n log n ) to achieve the optimal excess risk bounds, improving on the O ( n 2 ) computational guarantee of DP-SGD.

Acknowledgments

The work was partially supported by NSF Award #2048091.

References

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pp. 308-318, 2016.

Agarwal, N., Kale, S., Singh, K., and Thakurta, A. Differentially private and lazy online convex optimization. In The Thirty Sixth Annual Conference on Learning Theory , pp. 4599-4632. PMLR, 2023.

Balle, B. and Wang, Y.-X. Improving the gaussian mechanism for differential privacy: Analytical calibration and optimal denoising. In International Conference on Machine Learning , pp. 394-403. PMLR, 2018.

Balle, B., Barthe, G., and Gaboardi, M. Privacy amplification by subsampling: Tight analyses via couplings and divergences. In Advances in Neural Information Processing Systems , pp. 6277-6287, 2018.

Balle, B., Barthe, G., Gaboardi, M., Hsu, J., and Sato, T. Hypothesis testing interpretations and renyi differential privacy. In International Conference on Artificial Intelligence and Statistics , pp. 2496-2506. PMLR, 2020.

Bassily, R., Smith, A., and Thakurta, A. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science , pp. 464-473. IEEE, 2014.

Bun, M., Dwork, C., Rothblum, G. N., and Steinke, T. Composable and versatile privacy via truncated cdp. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing , pp. 74-86, 2018.

Canonne, C. L., Kamath, G., and Steinke, T. The discrete gaussian for differential privacy. Advances in Neural Information Processing Systems , 33:15676-15688, 2020.

Chaudhuri, K., Monteleoni, C., and Sarwate, A. D. Differentially private empirical risk minimization. Journal of Machine Learning Research , 12(3), 2011.

Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pp. 265-284. Springer, 2006.

Gopi, S., Lee, Y. T., and Wutschitz, L. Numerical composition of differential privacy. In Advances in Neural Information Processing Systems , 2021.

Iyengar, R., Near, J. P., Song, D., Thakkar, O., Thakurta, A., and Wang, L. Towards practical differentially private convex optimization. In 2019 IEEE Symposium on Security and Privacy (SP) , pp. 299-316. IEEE, 2019.

Kairouz, P., McMahan, B., Song, S., Thakkar, O., Thakurta, A., and Xu, Z. Practical and private (deep) learning without sampling or shuffling. In International Conference on Machine Learning , pp. 5213-5225. PMLR, 2021.

Kifer, D., Smith, A., and Thakurta, A. Private convex empirical risk minimization and highdimensional regression. In Conference on Learning Theory , pp. 25-1. JMLR Workshop and Conference Proceedings, 2012.

Koskela, A., Jälkö, J., and Honkela, A. Computing tight differential privacy guarantees using FFT. In International Conference on Artificial Intelligence and Statistics , pp. 2560-2569. PMLR, 2020.

Koskela, A., Jälkö, J., Prediger, L., and Honkela, A. Tight differential privacy for discrete-valued mechanisms and for the subsampled gaussian mechanism using FFT. In International Conference on Artificial Intelligence and Statistics , pp. 3358-3366. PMLR, 2021.

Kulkarni, J., Lee, Y . T., and Liu, D. Private non-smooth erm and sco in subquadratic steps. Advances in Neural Information Processing Systems , 34:4053-4064, 2021.

Liu, J. and Talwar, K. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing , pp. 298-309, 2019.

McKenna, R., Paskov, H., and Talwar, K. A practitioners guide to differentially private convex optimization. 2021.

Mironov, I. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF) , pp. 263-275. IEEE, 2017.

Mironov, I., Talwar, K., and Zhang, L. Rényi differential privacy of the sampled Gaussian mechanism. arXiv preprint arXiv:1908.10530 , 2019.

Mohapatra, S., Sasy, S., He, X., Kamath, G., and Thakkar, O. The role of adaptive optimizers for honest private hyperparameter selection. In Proceedings of the aaai conference on artificial intelligence , volume 36, pp. 7806-7813, 2022.

Neel, S., Roth, A., Vietri, G., and Wu, S. Oracle efficient private non-convex optimization. In International Conference on Machine Learning , pp. 7243-7252. PMLR, 2020.

Nguyen, L. M., van Dijk, M., Phan, D. T., Nguyen, P. H., Weng, T.-W., and Kalagnanam, J. R. Finite-sum smooth optimization with sarah. Computational Optimization and Applications , 82(3): 561-593, 2022.

Papernot, N. and Steinke, T. Hyperparameter tuning with Rényi differential privacy. In International Conference on Learning Representations , 2022.

Redberg, R. and Wang, Y.-X. Privately publishable per-instance privacy. Advances in Neural Information Processing Systems , 34, 2021.

Schmidt, M., Le Roux, N., and Bach, F. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017.

Sommer, D. M., Meiser, S., and Mohammadi, E. Privacy loss classes: The central limit theorem in differential privacy. Proceedings on Privacy Enhancing Technologies , 2019(2):245-269, 2019.

Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing , pp. 245-248. IEEE, 2013.

Song, S., Steinke, T., Thakkar, O., and Thakurta, A. Evading curse of dimensionality in unconstrained private glms via private gradient descent. arXiv preprint arXiv:2006.06783 , 2020.

Wang, D., Ye, M., and Xu, J. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems , 30, 2017.

Wang, Y.-X., Balle, B., and Kasiviswanathan, S. P. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics , pp. 1226-1235. PMLR, 2019.

Yu, D., Zhang, H., Chen, W., Liu, T.-Y., and Yin, J. Gradient perturbation is underrated for differentially private convex optimization. arXiv preprint arXiv:1911.11363 , 2019.

Zhu, Y. and Wang, Y.-X. Poisson subsampled Rényi differential privacy. In International Conference on Machine Learning , pp. 7634-7642, 2019.

Zhu, Y., Dong, J., and Wang, Y.-X. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics , pp. 4782-4817. PMLR, 2022.

Appendix

Table of Contents

A Notation

Denote the following:

˜ θ satisfies ||∇L P ( ˜ θ ) || 2 ≤ τ ,

We take ||X || to be the size of the data domain X , i.e. ||X || = sup x ∈X || x || . For conciseness of presentation we sometimes drop the dataset Z from the notation, e.g. we abbreviate L ( θ ; Z ) as L ( θ ) .

Sometimes (especially in the proofs that follow) we will abuse notation by overloading a function with its output, e.g. θ P is the output of objective perturbation and θ P ( Z ) is the objective perturbation mechanism.

B Warm-up: Gaussian Mechanism

We will get started by reviewing the RDP and privacy profile bounds for the Gaussian mechanism. Once warmed up, we then present the RDP and privacy profile bounds for objective perturbation in Sections C and D.

Consider the Gaussian mechanism defined by M ( Z ) = f ( Z ) + N (0 , σ 2 I d ) , for a function f : Z → R d with global sensitivity ∆ f = max Z ≃ Z ′ || f ( Z ) -f ( Z ′ ) || 2 .

B.1 Privacy profile of the Gaussian Mechanism

Analytic Gaussian mechanism (Balle & Wang, 2018).

Let P and Q be the density functions of N (∆ f , σ 2 ) and N (0 , σ 2 ) , respectively. Then ( P, Q ) is a dominating pair of distributions for M , and M is tightly ( ϵ, δ ( ϵ )) -DP for

where Φ denotes the CDF of the standard univariate Gaussian distribution.

We can analytically express a tight upper bound for the Gaussian mechanism above, but in general numerical methods are needed to evaluate the hockey-stick divergence for dominating pairs of distributions. This is discussed with more detail in Section D.

B.2 RDP Analysis of the Gaussian Mechanism

Theorem B.1 (RDP guarantees of the Gaussian mechanism) . The Gaussian mechanism M satisfies ( α, ϵ )-RDP for α > 1 and ϵ = ∆ 2 f α 2 σ 2 .

C RDP analysis of objective perturbation

In this section we present the proof of Theorem 3.2, one of our main privacy results: an RDP bound on the objective perturbation mechanism.

Proof of Theorem 3.2. Recall from Definition 2.7 that the objective perturbation mechanism ˆ θ P : Z ∗ → R d satisfies ϵ ( α ) -Rényi differential privacy if for all neighboring datasets Z and Z ′ ,

Assume that Z ∈ Z n and construct Z ′ ∈ Z n + by adding a datapoint z to Z . Note that this convention (while convenient for writing down the PLRV of objective perturbation) comes with loss of generality. As a consequence of asymmetry 3 , the upper bound on the RDP must satisfy

We will calculate the Rényi divergence D α ( ˆ θ P ( Z ) || ˆ θ P ( Z ′ ) ) of objective perturbation under a change of measure:

Let R ( θ P ) := Pr [ ˆ θ P ( Z )= θ P ] Pr [ ˆ θ P ( Z ′ )= θ P ] be shorthand for the probability density ratio at output θ P . Then

Denote J ( θ ; Z ) = ∑ z ∈ Z ℓ ( θ ; z ) + λ 2 || θ || 2 2 and µ ( θ, Z, z ) = x T ( ∇ 2 J ( θ ; Z ) ) -1 x .

From Lemma K.9 and the λ -strong convexity of J ( θ ; Z ) , we can bound µ ( θ, Z, z ) by || x || 2 2 λ ≤ 1 λ .

We also know from the L -Lipschitzness of ℓ ( θ ; z ) and the β -smoothness of f ( x T θ ; y ) that ||∇ ℓ ( θ ; z ) || 2 ≤ L and f ′′ ( x T θ ; y ) ≤ β for all θ ∈ R d and z = ( x, y ) ∈ Z .

Then using the GLM assumption, from Redberg & Wang (2021) we can bound the absolute value of the log-probability ratio for any θ P as

It is more challenging to find a data-independent bound for the third term due to the the shared dependence on θ P .

Recall that b ∼ N (0 , σ 2 I d ) is the noise vector in the perturbed objective. By first-order conditions at the minimizer θ P ,

If θ P were fixed (or if θ P were independent to b ), the quantity ∇ J ( θ P ; Z ) T ∇ ℓ ( θ P ; z ) = -b T ∇ ℓ ( θ P ; z ) would have been distributed as a univariate Gaussian N (0 , σ 2 ||∇ ℓ ( θ P ; z ) || 2 2 ) . Unfortunately in our case θ P is a random variable, and consequently we don't have the tools to understand the distribution of ∇ J ( θ P ; Z ) T ∇ ℓ ( θ P ; z ) for an arbitrary loss function.

But using the GLM assumption on the loss function, we can write

Observe that x is fixed w.r.t. b so that -b T x ∼ N (0 , σ 2 ∥ x ∥ 2 2 ) , and f ′ ( x T θ, y ) is a scalar. So while this scalar is a random variable that still depends on b in a complicated way, the worst possible dependence can be more easily quantified without incurring additional dimension dependence.

By the L -Lipschitz assumption and | ab | ≤ | a || b | , we obtain the following bound:

This is much better! By first-order conditions we can then see

Now we can bound

Plugging in this bound on ∣ ∣ log R ( θ P ) ∣ ∣ ,

Let p σ be the probability density function of b ∼ N (0 , σ 2 I d ) , and p Θ the probability density function of θ P ∼ ˆ θ P ( Z ) . We know from Lemmas K.6 and K.7 that ∂θ = ∣ ∣ ∣ ∣ det ∂θ ∂b ∣ ∣ ∣ ∣ ∂b , and

We also know from Lemma K.8 that ∣ ∣ ∣ ∣ det ∂θ ∂b ∣ ∣ ∣ ∣ · ∣ ∣ ∣ ∣ det ∂b ∂θ ∣ ∣ ∣ ∣ = 1 .

Using the change of variables P T 2 2

In the last line, we applied our assumption that ∥ x ∥ ≤ 1 and the fact that the MGF of a half-normal R.V. increases monotonically when its scale parameter gets larger.

The above bound holds for the reverse Rényi divergence D α ( ˆ θ P ( Z ′ ) || ˆ θ P ( Z )) . Observe that

This is because log Pr [ ˆ θ P ( Z ′ )= θ P ] Pr [ ˆ θ P ( Z )= θ P ] = -log R ( θ P ) ≤ | log R ( θ P ) | . If we use the change of variables b = -∇ J ( θ P ; Z ′ ) for the reverse direction, the above calculation works out identically (the difference is that p Θ and the bijection between b and θ P are different under Z and Z ′ -but the determinant of the mapping cancels out with its inverse just the same).

We've shown max ( D α ( ˆ θ P ( Z ) || ˆ θ P ( Z ′ ) ) , D α ( ˆ θ P ( Z ′ ) || ˆ θ P ( Z ) ) ) ≤ ϵ ( α ) for any neighboring datasets Z and Z ′ , where

C.1 Linearized RDP Bound for Objective Perturbation

In our calculation of the RDP for objective perturbation, we needed to take an absolute value of the privacy loss random variable in order to handle negative values. But in doing so we end up with a quantity that depends on the moments of the half-normal distribution rather than those of the normal distribution, which gives us a looser bound. Can we avoid having to make this compromise? In this section we demonstrate that a linearization of the first-order conditions on the perturbed and unperturbed objective functions provides a more precise analysis of the PLRV of objective perturbation, translating to a tighter RDP bound in some regimes.

Recall that the objective perturbation mechanism is given by

where b ∼ N (0 , σ 2 I d ) .

From the non-linearized RDP calculation, we know that for any neighboring datasets Z and Z ′ ,

where θ P is the output of the objective perturbation mechanism given the noise vector b . We can write

The first term t 1 = b T ∇ ℓ ( θ ∗ λ ) is a univariate Gaussian t 1 ∼ N (0 , σ 2 ||∇ ℓ ( θ ∗ λ ) || 2 2 ) because θ ∗ λ is fixed w.r.t. b . We can bound the second term t 2 = b T [ ∇ ℓ ( θ P ) -∇ ℓ ( θ ∗ λ ) ] using our assumptions on the loss function ℓ ( θ ) .

By assumption, the loss function has GLM structure ℓ ( θ ; z ) = f ( x T θ ; y ) . We can therefore write

We have furthermore assumed that the function f is β -smooth, so that for any z = ( x, y ) and θ P , θ ∗ λ ∈ R d we have

We will next apply Taylor's Theorem to rewrite θ P -θ ∗ λ .

Recall that θ P is the minimizer of the perturbed objective:

and θ ∗ λ is the minimizer of the (non-private) regularized objective:

Parameterize the line segment between θ P and θ ∗ λ by t ∈ [0 , 1] , i.e. the line segment is t ( θ P -θ ∗ λ )+ θ P . By Taylor's Theorem, there exists θ ′ = t ′ ( θ P -θ ∗ λ ) + θ P for some t ′ ∈ [0 , 1] such that

By first-order conditions on Equations C.2 and C.3,

Then subtracting Equation C.5 from Equation C.4, we have that

Again applying Taylor's theorem, there exists θ ′′ = t ′′ ( θ P -θ ∗ λ ) + θ P for some t ′′ ∈ [0 , 1] ) such that

Putting together Equations C.6 and C.7 we then have

So we now have

Note that since e x > 0 for all x ∈ R , we have that E [ | e x | ] = E [ e x ] .

Let a := 1 σ 2 b T ∇ ℓ ( θ ∗ ) and c := 1 σ 2 b T [ ∇ ℓ ( θ P ) -∇ ℓ ( θ ∗ ) ] . Then by Holder's inequality,

Above, we've applied the assumption that || x || 2 ≤ 1 and the fact that the MGF of a normal R.V. increases monotonically when its scale parameter gets larger (Lemma K.11). By Lemma K.9, we

2

Note that our approach below is agnostic to the relationship between | z 1 | and | z 2 | ; in reality, they depend on each other through the noise vector b . Again applying the assumption || x || 2 2 ≤ 1 and Lemma K.11 (while not forgetting that the random variables z 1 and z 2 depend on each other through b ), we get

So altogether, for p, q such that 1 p + 1 q = 1 , we get

D Hockey-stick Divergence Analysis of Objective Perturbation

D.1 Further Details on Hockey-stick Divergence Analysis

Using dominating pairs of distributions (Def. 2.5) for all the individual mechanisms in an adaptive composition, we can obtain accurate ( ϵ, δ ) -bounds for the whole composition. For this end we need the following result.

Theorem D.1 (Zhu et al. 2022) . If ( P, Q ) dominates M and ( P ′ , Q ′ ) dominates M ′ for all inputs of M ′ , then ( P × P ′ , Q × Q ′ ) dominates the adaptive composition M◦M ′ .

To get the hockey-stick divergence from P × P ′ to Q × Q ′ into an efficiently computable form, we express it using so called privacy loss random variables (recall Def. K.4). If P and Q are probability density functions, the privacy loss function L P/Q is defined as

and the privacy loss random variable (PLRV) ω P/Q as

The δ ( ϵ ) -bounds can be represented using the following representation that involves the PLRV. Theorem D.2 (Gopi et al. 2021) . We have:

Moreover, if ω P/Q is the PLRV for the pair of distributions ( P, Q ) and ω P ′ /Q ′ the PLRV for the pair of distributions ( P ′ , Q ′ ) , then the PLRV for the pair of distributions ( P × P ′ , Q × Q ′ ) is given by ω P/Q + ω P ′ /Q ′ .

By Theorem D.2, to computing accurate ( ϵ, δ ) -bounds for compositions, it suffices that we can evaluate integrals of the form E s ∼ ω 1 + ... + ω k [1 -e ϵ -s ] + . For this we can use the Fast Fourier Transform (FFT)-based method by Koskela et al. (2021), where the distribution of each PLRV is truncated and placed on an equidistant numerical grid over an interval [ -L, L ] , where L > 0 is a pre-defined parameter. The distributions for the sums of the PLRVs are given by convolutions of the individual distributions and can be evaluated using the FFT algorithm. By a careful error analysis the error incurred by the numerical method can be bounded and an upper δ ( ϵ ) -bound obtained. For accurately carrying out this numerical computation one could also use, for example, the FFT-based method proposed by Gopi et al. (2021).

D.2 Proof of Theorem 3.1

Before giving a proof to Thm. 3.1, we first give the following bound which is a hockey-stick equivalent of the moment-generating function bound given in Thm. 3.2.

Lemma D.3 . Let ϵ ∈ R and let the objective perturbation mechanism ˆ θ P be defined as in Section 2.2. Let ||∇ ℓ ( θ ; z ) || 2 ≤ L and ∇ 2 ℓ ( θ ; z ) ≺ βI d for all θ ∈ Θ and z ∈ X ×Y . Then, for any neighboring datasets Z and Z ′ , we have:

Proof. The proof goes analogously to the proof of Thm. 3.2. Let Z and Z ′ be any neighboring datasets. Following the proof of Thm. 3.2, denote the privacy loss

By Thm. D.2 and by using the reasoning of the proof of Thm. 3.2 for the moment-generating function, we have

where the inequalities follow from the fact that the function f ( s ) = [1 -e ϵ -s ] + is monotonically increasing function w.r.t. s for all ϵ ∈ R and from the bound for | R ( θ ) | used in the proof of Thm. 3.2.

Proof of Theorem 3.1. We use Lemma D.3 and simply upper bound the right-hand side of the inequality (D.2).

We first show that if ∥ x ∥ ≤ 1 , then for all ϵ ∈ R

where the first equality follows from the fact that for s ≥ ˆ ϵ , the density function of the half-normal random variable is positive and 2 times the density of the corresponding normal distribution. The inequality follows from Lemma D.8, as

where the inequality follows from Lemma D.8. Inequalities (D.5) and (D.6) together give (D.4). Then, we show that for all ϵ ∈ R ,

Continuing from (D.5), by change of variables, we see that for ˆ ϵ ≥ 0 ,

where ˜ ϵ = ϵ -∣ ∣ ∣ log ( 1 -β λ )∣ ∣ ∣ , P is the density function of N ( L, σ 2 ) and Q the density function of N (0 , σ 2 ) . This follows from the fact that the PLRV determined by the pair ( P, Q ) is distributed as N ( L 2 2 σ 2 , L 2 σ 2 ) .

Continuing from (D.6), by change of variables (used after the third equality sign), we see that for ˆ ϵz 0 ,

where we again use the fact that the PLRV of the Gaussian mechanism with sensitivity L and noise scale σ is distributed as N ( L 2 2 σ 2 , L 2 σ 2 ) .

D.3 Dominating Pairs of Distributions for the Objective Perturbation Mechanism

From Lemma D.3 and the inequality (D.4) we have that for all ϵ ∈ R

Thus, if we have distributions P and Q such that for all ϵ ∈ R

then the pair ( P, Q ) is a dominating pair of distributions for the objective perturbation mechanism. Then, by Theorem D.2, we can use this distribution ω also to compute ( ϵ, δ ) -bounds for compositions involving the objective perturbation mechanism. We give such a pair of distribution ( P, Q ) explicitly in Lemma D.6 below.

In the following, we denote the density of a discrete probability mass by a Dirac delta function and use the indicator function for the continuous part of the density. The following result is a straightforward calculation.

Lemma D.4 . Let σ > 0 . Let P be the density function of ∣ ∣ N ( 0 , σ 2 )∣ ∣ , i.e.,

where 1 A ( x ) denotes the indicator function, i.e., 1 [0 , ∞ )] ( x ) = 1 if x ≥ 0 , else 1 [0 , ∞ )] ( x ) = 0 . Let L > 0 and let Q be a density function, where part of the mass of P is shifted to -∞ :

where

Then, we have that the PLRV ω , is distributed as

Proof. As P has its support on [0 , ∞ ) , we need to consider the values of the privacy loss function log P ( x ) Q ( x ) only on [0 , ∞ ) . We have, for all x ≥ 0 ,

Since x ∼ P , we see that L σ 2 · x ∼ ∣ ∣ ∣ N ( 0 , L 2 σ 2 )∣ ∣ ∣ and the claim follows.

Remark D.5 . In Lemma D.4, instead of shifting part of the mass of P to -∞ when forming Q , we could place this mass anywhere on the negative real axis. This would not affect the PLRV ω .

We can shift the PLRV ω given by Lemma D.4 by scaling the distribution Q . We get the following.

Lemma D.6 . Let σ > 0 and L > 0 . Suppose P is the density function given in Eq. (D.7) and Q the density function

where

Then, the PLRV ω determined by P and Q is distributed as

Proof. Showing this goes as the proof of Lemma D.4. We just now have that for all x ≥ 0 :

As a corollary of Lemma D.6 and Thm. 2.4, we have:

Lemma D.7 . Let k ∈ Z + and let for each i ∈ [ k ]

such that ω i 's are independent. Then, the k -wise adaptive composition of ˆ θ ( Z ) is ( ϵ, δ ( ϵ )) -DP for

D.3.1 Numerical Evaluation of ( ϵ, δ ) -Bounds for Compositions

Figure 3 shows the result of applying the FFT-based numerical method of Koskela et al. (2021) for evaluating the expression (D.11). We compare the resulting approximate DP bounds to those obtained from the RDP bounds combined with standard composition results (Mironov, 2017).

Notice that we could also carry out tighter accounting of the approximative minima perturbation (Section 4) by adding the PLRVs of the Gaussian mechanism to the total PLRV, similarly as RDP parameters of the Gaussian mechanism are added to the RDP guarantees of the objective perturbation mechanism (Theorem 4.1). Adding the Gaussian PLRV to the total PLRV using convolutions is straightforward using the method of Koskela et al. (2021).

D.4 Auxiliary Lemma

For Theorem 3.1, we need the following auxiliary result.

Lemma D.8 . Denote f µ,σ 2 ( x ) the density function of the normal distribution N ( µ, σ 2 ) and let c ≥ µ . Let g ( x ) be a non-negative differentiable non-decreasing function on [ c, ∞ ) . Then, if σ 1 ≤ σ 2 ,

Proof. By integration by parts, we have

where Φ µ,σ 2 ( x ) denotes the cdf of N ( µ, σ 2 ) . A simple calculation shows that for all x ∈ R ,

Thus, Φ µ,σ 2 ( c ) is a non-increasing function of σ for all c ≥ µ . Furthermore, the first term in (D.12) is a non-decreasing function of σ since g ( c ) is non-negative and the second term is a non-decreasing function of σ , since g ′ ( x ) is non-negative for all x ∈ [ c, ∞ ) .

0.2

0.4

0.6

0.8

1.0

1.2

1.4

10

15

10

13

10

11

10

9

10

7

10

5

10

3

10

1

RDP

k

=1

RDP

k

=2

RDP

k

=3

RDP

k

=4

Hockey-stick

k

=1

Hockey-stick

k

=2

Hockey-stick

k

=3

Hockey-stick

k

=4

E RDP guarantee of Algorithm 1

In what follows, we will present a (corrected) privacy guarantee for Approximate Minima Perturbation (i.e., Algorithm 1 without gradient clipping). We will then demonstrate that the 'clipped-gradient' function ℓ C ( θ ) not only bounds the per-example gradient norm by C , but also preserves other properties (i.e., β -smoothness and GLM structure) required for the privacy guarantees stated in Theorem 3.2.

E.1 Privacy Guarantee for Approximate Minima Perturbation

The proof of the privacy guarantee for Approximate Minima Perturbation (Iyengar et al., 2019), i.e. Algorithm 1 without gradient clipping, falls prey to the same trap as previous work on objective perturbation. In particular, we see that there is a mistake in Lemma IV.1, with the assertion that 'we get the statement of the lemma from the guarantees of the Gaussian mechanism.' The Gaussian mechanism is inapplicable in Lemma IV.1 for similar reasons as discussed in Section G.

The proof of Theorem E.1 corrects this issue. We state it in terms of RDP, but it can also extend to approximate DP and other DP variants.

Algorithm 2 Approximate Minima Perturbation (Iyengar et al., 2019)

Input: dataset Z ; noise levels σ, σ out ; β -smooth loss function ℓ ( · ) with Lipschitz constant L ; regularization strength λ ; gradient norm threshold τ .

Sample b ∼ N (0 , σ 2 I d ) .

Output ˜ θ P = ˜ θ + N (0 , σ 2 out I d ) .

Solve for ˜ θ such that ||∇L P C ( ˜ θ ; Z ) || 2 ≤ τ .

Theorem E.1 (RDP guarantees of Approximate Minima Perturbation) . Consider the Approximate Minima Perturbation algorithm which satisfies ( α, ϵ )-RDP for any α > 1 with

Proof. Sample b ∼ N (0 , σ 2 I d ) and let L P ( θ ; Z, b ) := ∑ z ∈ Z ℓ ( θ ; z ) + λ 2 ∥ θ ∥ 2 2 + b T θ , i.e., the perturbed and regularized objective function used by ObjPert.

Let θ P = arg min L P ( θ ; Z, b ) . From Chaudhuri et al. (2011); Kifer et al. (2012) we know that there is a bijection b ( θ P ; Z ) from the output θ P to the noise vector b .

Consider a blackbox algorithm θ A ( Z, b ) which returns θ such that ||∇L P ( θ ; Z, b ) || ≤ τ . Define query

where θ A ( Z ) is an abbreviation for θ A ( Z, b ( θ P ; Z )) .

We assume that q can recover b from the input θ P via the bijection b ( θ P ; Z ) , and hence has access to the perturbed objective function L P ( θ ; Z, b ) .

Notice that since L P is λ -strongly convex, by applying the Cauchy-Schwarz inequality and by Definition K.3 we see that for any θ 1 , θ 2 ,

Algorithm θ A ( Z, b ) guarantees that its output θ satisfies ||∇L P ( θ ) || 2 ≤ τ and by first-order conditions on the perturbed objective function, ∇L P ( θ P ; Z, b ) = 0 . It follows that for any dataset Z and θ P ,

Since the algorithm θ A ( Z, b ) guarantees that ∥ θ A -θ P ∥ 2 ≤ γ/λ , then conditioning on θ P , q ( Z, θ P ) has a global sensitivity bounded by 2 γ/λ since

Now, the algorithm that first draws b then outputs θ A ( Z, b ) + N (0 , σ 2 I d ) is equivalent to

First run ObjPert that returns θ P .

Release ˆ ∆ = q ( Z, θ P ) + N (0 , σ 2 I d ) .

Return θ P + ˆ ∆ .

This is adaptive composition of ObjPert with the Gaussian mechanism. The third step is post processing.

The privacy guarantee stated in Theorem E.1 is thus achieved by combining the results of Theorem 3.2 (RDP of ObjPert), Theorem B.1 (RDP of the Gaussian mechanism) with ∆ q = 2 τ λ , and Lemma K.5 (adaptive composition for RDP mechanisms).

E.2 The 'Clipped-Gradient' Function

The RDP guarantees of objective perturbation (stated in Theorem 3.2) require several assumptions on the loss function ℓ ( θ ; Z ) . If we can demonstrate that these properties are satisfied by the 'clippedgradient' loss function ℓ C ( θ ; Z ) , then the rest of the proof of Theorem 4.1 (the privacy guarantee of Algorithm 1) will follow directly from that.

In particular, we need to show:

That ℓ C ( θ ; z ) retains the convex GLM structure of the original function ℓ ( θ ; z ) .

That ℓ C ( θ ; z ) satisfies ||∇ ℓ C ( θ ; z ) || 2 ≤ C for any θ, z .

That ℓ C ( θ ; z ) has the same β -smoothness parameter as the original function ℓ ( θ ; z ) .

That even though ℓ C ( θ ; z ) is not twice-differentiable everywhere, the privacy guarantees of objective perturbation (whose proof involves a Jacobian mapping) still hold.

We will begin by stating a result from Song et al. (2020).

Theorem E.2 (Song et al., 2020, Lemma 5.1) . Let f : R → R be any convex function and let C ∈ R + be any positive value. For any non-zero x ∈ R d , define

If U L is non-empty, let u L = sup U L ; otherwise u L = -∞ . If U H is non-empty, let u H = inf U H ; otherwise u H = ∞ . For any non-zero x ∈ R d , let

Define u x ( θ ) = x T θ . Then the following holds.

1. f C is convex.

The first two desired properties of ℓ C ( θ ; z ) , i.e. GLM structure and gradient norm bound C , follow directly from the above theorem. Next, we will prove the third property of β -smoothness.

Theorem E.3 . For a data point z = ( x, y ) , consider a function f such that ℓ ( θ ; z ) = f ( x T θ ; y ) . Suppose that f ( x T θ ; y ) satisfies β -smoothness. Then the 'clipped-gradient' function f C ( x T θ ; y ) defined in Lemma 5.1 of Song et al. (2020) also satisfies β -smoothness.

Proof. Because f is β -smooth by assumption, we know that f ( u ) satisfies β -smoothness for all u ∈ ( u L , u H ) . When u ∈ ( -∞ , u L ) or when u ∈ ( u H , ∞ ) , the function f C ( u ) is linear in u and thus is 0 -smooth (hence satisfying β -smoothness).

Lastly, the proof of objective perturbation (see, e.g., Theorem 9 of Chaudhuri et al. (2011)) requires that the loss function be twice-differentiable. Even though ℓ C ( θ ; z ) is not twice-differentiable everywhere, the privacy guarantees of objective perturbation still hold. To show this, we can invoke Corollary 13 of Chaudhuri et al. (2011). This corollary assumes the Huber loss; for brevity, we will leave it as an exercise for the reader to verify that the proof also carries through for the 'clippedgradient' loss.

F Computational Guarantee of Algorithm 1

In this section, we provide a computational guarantee to Algorithm 1 in terms of the number of gradient evaluations on individual loss functions to compute the approximate minimizer for achieving (up to a constant) the information-theoretical limit.

Let f ( θ ) := ∑ i ℓ i ( θ ) + λ 2 ∥ θ ∥ 2 + b T θ , i.e., the perturbed and regularized objective function used by ObjPert. Let θ ∗∗ be the output returned by the blackbox algorithm θ A ( · ) described in Section E. Note the deviation from the notation used in the previous section.

Iyengar et al. (2019) proposed a procedure that keeps checking the gradients in an iterative optimization algorithm and stops when the gradient is smaller than τ . This always ensures that ∥∇ f ( θ ∗∗ ) ∥ ≤ τ .

And using tools from the next section, it can be proven that it implies that ∥ θ ∗∗ -θ ∗ ∥ 2 ≤ τ/λ as was previously stated. But how many iterations it takes for this to happen for specific algorithms was not explicitly considered.

F.1 Tools from convex optimization

We will need a few tools from convex optimization.

Firstly, under our assumption that ℓ i is β -smooth, f is nβ + λ -smooth and λ -strongly convex. Let L := nβ + λ as a shorthand.

By L -smoothness (gradient Lipschitzness), and the optimality of θ ∗ , we have that for any θ

By λ -strong convexity, we get

By strong convexity also implies that which implies that

which is the quantity used to establish the global sensitivity of q ( D,θ ∗ ) as we talked about earlier. (F.2) and (F.3) sandwich ∥ θ -θ ∗ ∥ in between by

F.2 Computational bounds for Stopping at small gradient

(F.1) and (F.2) together provides bounds for ∥∇ f ( θ ) ∥ using either objective function or argument convergence (in square ℓ 2 .)

Standard convergence results are often parameterized in terms of either suboptimality f ( θ ) -f ∗ or argument ∥ θ -θ ∗ ∥ 2 . In the following we instantiate specific convergence bounds for deriving computation guarantees.

Gradient Descent. If we run gradient descent with learning rate 1 /L for T iterations from θ 0 , then

This happens deterministically (with no randomness, or failure probability).

One may ask why are we not running a fixed number of iterations and directly applying the bound to ∥ θ T -θ ∗ ∥ in order to control the ℓ 2 sensitivity. That works fine, except that we have an unconstrained problem and θ ∗ can be anywhere, thus there might not be a fixed parameter T to provide a required bound for all input θ ∗ . We also do not know where θ ∗ is during the actual execution of the algorithm and thus cannot compute ∥ θ 0 -θ ∗ ∥ directly.

The 'gradient-norm check' as a stopping condition from Iyengar et al. (2019) is nice because it always ensures DP for any θ ∗ (at a price of sometimes running for a bit longer).

To ensure ∥∇ f ( θ ) ∥ ≤ γ , the number of iterations

Since each gradient computation requires n incremental gradient evaluation, under the regime that λ is independent of n , under the choice that λ = 1 /ϵ independent to n from the standard calibration, the total number of is therefore O ( n 2 log n ) for achieving γ ≤ n -v for any constant v > 0 .

The quadratic runtime is not ideal, but it can be improved using accelerated gradient descent which gives a convergence bound of

This would imply a computational guarantee of O ( n 1 . 5 log n ) . The overall computation bound depends on ∥ θ ∗ ∥ which is random (due to objective perturbation). The dependence on ∥ θ ∗ ∥ is only logarithmic though.

Finite Sum and SAG. The result can be further improved if we uses stochastic gradient methods. However, the sublinear convergence of the standard SGD or its averaged version makes the application of the above conversion rules somewhat challenging.

By taking advantage of the finite sum structure of f ( θ ) one can obtain faster convergence.

First of all, the finite sum structure says that f ( θ ) = ∑ n i =1 f i ( θ ) . In our case, we can split the regularization and linear perturbation to the n data points, i.e.,

Check that it satisfies β + λ/n smoothness.

There is a long list of methods that satisfy the faster convergence for finite sum problems, e.g., SAG, SVRG, SAGA, SARAH and so on (see, e.g., Nguyen et al., 2022, for a recent survey). Specifically, Stochastic Averaged Gradient (Schmidt et al., 2017) (and similarly others with slightly different parameters) satisfies

Therefore, by (F.2), we have

Note that each iteration costs just one incremental gradient evaluation, so to ensure E [ ∥∇ f ( θ T ) ∥ 2 ] ≤ γ 2 , the computational complexity is on the order of

This is O ( n log n ) runtime to any γ = n -s for a constant s > 0 .

On the other hand, the main difference from the gradient descent result is that we only get convergence in expectation. By Markov's inequality

which implies high probability convergence naturally.

Theorem F.1 . Assume λ ≥ β . The algorithm that runs SAG and checks the stopping condition ∥∇ f ( θ T ) ∥ ≤ γ after every n iteration will terminate with probability at least 1 -δ in less than

incremental gradient evaluations, where C is a universal constant.

How to set γ to achieve information-theoretic limit? The lower bounds for convex and smooth losses in differentially private ERM are well-known (Bassily et al., 2014) and it is known that among GLMs, θ ∗ from ObjPert achieves the lower bound with appropriate choices of λ, σ . Notably, λ ≍ d/ϵ for achieving an ( ϵ, δ ) -DP.

Let ˆ θ = θ T + N (0 , γ 2 2 λ 2 ρ I ) be the final output.

By the nG Lipschitzness of ∑ i ℓ i , with high probability over the Gaussian mechanism, we have that

where ρ is the zCDP parameter for the Gausssian mechanism chosen to match the large α part of the ObjectivePerturbation's RDP bound, which increases the overall RDP by αρ .

To conclude, the above results imply that the computationally efficient objective perturbation achieves the optimal rate under the same RDP guarantee with an algorithm that terminates in O ( n log n ) time with high probability.

G The GLM Bug

G.1 Discussion

Limiting our main results to generalized linear models might appear restrictive - but we argue that the GLM assumption is not specific to our paper, but rather has been lurking in the objective perturbation literature for some time now.

Let's first take a look at Section 3.3.2 of Chaudhuri et al. (2011): Lemma 10 requires that the matrix E have rank at most 2, but this is not necessarily true without assuming GLM structure. This is used to bound the determinant of the Jacobian, and corresponds to the first term of our bound in Theorem 3.2.

It is a similar story for bounding the log ratio / difference between the noise vector densities under neighboring datasets, corresponding to the second and third terms of our bound in Theorem 3.2. Let's also revisit this line from the proof of Lemma 17 of the Kifer et al. (2012) paper: "Note that Γ is independent of the noise vector." This is not true without assuming GLM structure! (In their proof, Γ is the difference between the noise vectors under neighboring distributions. From first-order conditions at the minimizer of the perturbed objective, we can see that Γ = ∇ ℓ ( θ P ) , where θ P is a function of the noise vector b .

In fact, to our knowledge, Iyengar et al. (2019) was the first work to acknowledge the GLM assumption on objective perturbation. But their privacy proof also fails to handle the dependence on the noise vector! In Theorems 3.2 and 3.1, we have included a careful analysis including a discussion on how the GLM assumption removes this dependence.

G.2 RDP bound for non-GLMs

In this section we generalize the RDP bound for objective perturbation to a general class of smooth convex losses.

We first state the following bound from Theorem 6

Theorem G.1 .

where µ j = λ j u T j ( ∇ b ( ˆ θ P ; D ) ∓ ∑ j -1 k =1 λ k u k u T k ) -1 u j according to the eigendecomposition ∇ 2 ℓ ( θ ; z ) = ∑ d k =1 λ k u k u T k .

Theorem G.2 (RDP bound for non-GLMS.) . Let ||∇ ℓ ( θ ; z ) || 2 ≤ L and ∇ 2 ℓ ( θ ; z ) ≺ βI d for all θ ∈ Θ and z ∈ X × Y . Objective perturbation satisfies ( α, ϵ )-RDP for any α > 1 with

Proof. The above follows from Theorem 6 in (Redberg & Wang, 2021) applied to the analysis of Theorem 3.2. We find that

We now need to bound

So

H Excess Empirical Risk of Algorithm 1

Our goal in this section is to find a bound on the excess empirical risk:

Theorem H.1 . Let ˜ θ P be the output of Algorithm 1 and θ ∗ = arg min L ( θ ) the minimizer of the loss function L ( θ ) = ∑ n i =1 ℓ ( θ ; z i ) . Denote ||X || as the diameter of the set X . We have

Proof. Following the proof of Theorem 2 from Iyengar et al. (2019) (itself adapted from Kifer et al. (2012)), we write

By the λ -strong convexity of L P , for any ˜ θ, θ ∗ we have

By first-order conditions, ∇L P ( θ P ) = 0 . Applying the Cauchy-Schwarz inequality along with our stopping criteria on the gradient norm, we then have

Let b 2 ∼ N (0 , σ 2 out I d ) . Because L is nL -Lipschitz continuous, we have

Recall that b ∼ N (0 , σ 2 I d ) . Using the first-order condition ∇ J ( θ P ) = -b and the Cauchy-Schwarz inequality, we have

Observe also that

So

Altogether we then have

The optimal choice of λ would then be λ ≍ dL || θ ∗ || 2 ϵ . The optimal choice of τ is discussed in Section F.

H.1 Generalized Linear Model

With some additional assumptions and restrictions, we can get a tighter bound on E [ L λ ( θ P ) ] -L λ ( θ ∗ λ ) .

Wewill assume GLM structure on ℓ ( · ) , i.e. ℓ ( θ ; z ) = f ( x T θ ; y ) . Wewill further assume boundedness: c ≤ f ( x T θ ; y ) ≤ C for some universal constants c, C ∈ R . Applying Taylor's Theorem (in an argument similar to Equation C.8), we can show that for some θ ′′ ∈ [ θ P , θ ∗ λ ]

To bound the expectation of L ( θ P ) -L ( θ ∗ ) , we can write

Observe that

and that by optimality ( θ ∗ λ = arg min θ ∈ R d L λ ( θ ) ) , we have

By Taylor's Theorem, for some θ ′ ∈ [ θ P , θ ∗ λ ] we can write

The last equality is due to optimality conditions. We can then bound

where b ∼ N (0 , σ 2 I d ) . Then taking the expectation,

Note that by the GLM assumption, the eigendecomposition of ∇ 2 L ∗ λ ( θ ) can be written as X T Λ( θ ) X . Then plugging in from Equation H.1 and using the boundedness assumption on the loss f ,

Then in expectation,

I Distance to Optimality

Consider the mechanism M ( Z ) = f ( Z ) + N (0 , σ 2 I d ) , for a function f : Z → R d with sensitivity ∆ f = L . From Balle & Wang (2018), we know that for any neighboring datasets Z and Z ′ , the privacy loss random variable of this mechanism is distributed as N ( ∆ 2 Z,Z ′ 2 σ 2 , ∆ 2 Z,Z ′ σ 2 ) . Maximizing the Rényi divergence D α ( M ( Z ) || M ( Z ′ )) over all neighboring datasets Z ≃ Z ′ shows that the RDP for the Gaussian mechanism can be written as

Thus the main deviations between the RDP bound for objective perturbation and that of the Gaussian mechanism are 1) the leading term (a function of β and λ that vanishes as we increase the regularization) and 2) the moment-generating function of the half-normal (instead of normal) distribution.

Figure I plots the Rényi divergence ϵ ( α ) := D α ( M ( Z ) || M ( Z ′ )) for the Gaussian mechanism ("normal"), the objective perturbation mechanism ("ObjPert"), and the mechanism 4 obtained by adding noise from the half-normal distribution ("half-normal").

We consider several different regimes of interest by varying the noise scale σ and the regularization strength λ . There are several takeaways to observe:

The difference between the RDP for the half-normal mechanism and the RDP for the objective perturbation mechanism is due entirely to the leading term of the bound given in Theorem 3.2, which vanishes as λ increases (Figures 4b and 4c). For smaller λ there is a constant "start-up" gap between the half-normal and Objpert RDP curves (best displayed in Figure 4b) which disappears for larger α , where the moments of the half-normal distribution overwhelm the contribution of the leading term of the objective perturbation RDP.

As σ increases (e.g. between Figures 4c and 4a, and between Figures 4d and 4b), the half-normal curve - and therefore also the ObjPert curve - doesn't converge with the normal curve until larger α .

J Bridging the Gap between Objective Perturbation and DP-SGD

J.1 RDP of Objective Perturbation vs DP-SGD

DP-SGD (for example, n 2 rounds of sampled Gaussian mechanism with Poisson sampling probability 1 /n ) is known to experience a phase transition in its RDP curve: for smaller α , amplification by

10 1

10 2

10 3

alpha

10

4

10

3

10

2

10

1

divergence

half-normal

normal

(a)

σ

= 100

, λ

= 10

10 1

10 2

10 3

alpha

10

2

10

1

10 0

divergence

half-normal

normal

(b)

σ

= 10

, λ

= 1

10 1

10 2

10 3

alpha

10

2

10

1

10 0

divergence

half-normal

normal

(c)

σ

= 10

, λ

= 10

10 1

10 2

10 3

alpha

10 0

10 1

10 2

divergence

half-normal

normal

(d)

σ

= 1

, λ

= 1

Figure 4

sampling is effective, and the RDP of a DP-SGD mechanism behaves like a Gaussian mechanism with ϵ ( α ) = O ( α 2 σ 2 ) , then it leaps up at a certain α and begins converging to ϵ ( α ) = O ( n 2 α 2 σ 2 ) that does not benefit from sampling at all (Wang et al., 2019; Bun et al., 2018). In contrast, the RDP curve for objective perturbation defined by Theorem 3.2 converges to the RDP curve of the Gaussian mechanism ϵ ( α ) = O ( α 2 σ 2 ) after a certain point. Whereas DP-SGD offers stronger privacy parameters for small α , objective perturbation is stronger for large α , which offers stronger privacy protection for lower-probability events (see, e.g., Mironov, 2017, Proposition 10).

10 1

10 2

10 3

10 4

10 5

10 6

10

3

10

1

10 1

10 3

(

)

GM

DP-SGD

(a)

σ

=

σ

GM

=

100;

p

=

0

.

1

, σ

DP-SGD

= 10

.

10 1

10 2

10 3

10 4

10 5

10 6

10

3

10

2

10

1

10 0

10 1

10 2

10 3

(

)

GM

DP-SGD

(b)

σ

=

σ

GM

=

40;

p

=

0

.

5

, σ

DP-SGD

= 20

.

J.2 A Spectrum of DP Learning Algorithms

We can also connect Algorithm 1 to differentially private follow-the-regularized-leader (DP-FTRL) (Kairouz et al., 2021), which uses a tree-based aggregation algorithm to privately release the gradients of the loss function as a prefix sum. This approach provides a competitive privacy/utility tradeoff without relying on privacy amplification or shuffling, which is often not possible in distributed settings. DP-FTRL differs from DP-SGD by adding correlated rather than independent noise at each iteration; Algorithm 1, in contrast, differs from both by adding identical noise at each iteration.

K Technical Lemmas & Definitions

K.1 Convex Optimization

We will give a short review of relevant concepts from convex optimization.

Definition K.1 ( ℓ 2 -Lipschitz continuity) . A function f : Θ → R is L -Lipschitz w.r.t. the ℓ 2 -norm over Θ ⊆ R d if for all θ 1 , θ 2 ∈ Θ , the following holds: | f ( θ 1 ) -f ( θ 2 ) | ≤ L || θ 1 -θ 2 || 2 .

Definition K.2 ( β -smoothness) . A differentiable function f : Θ → R is β -smooth over Θ ⊆ R d if its gradient ∇ f is β -Lipschitz, i.e. |∇ f ( θ 1 ) -∇ f ( θ 2 ) | ≤ β || θ 1 -θ 2 || 2 for all θ 1 , θ 2 ∈ Θ .

Definition K.3 (Strong convexity) . A differentiable function f : Θ → R is λ -strongly convex over Θ ⊆ R d if for all θ 1 , θ 2 ∈ Θ : f ( θ 1 ) ≥ f ( θ 2 ) + ∇ f ( θ 2 ) T ( θ 1 -θ 2 ) + λ 2 || θ 1 -θ 2 || 2 2 .

K.2 Differential Privacy

Definition K.4 (Privacy loss random variable) . Let Pr [ M ( Z ) = θ ] denote the probability density of the random variable M ( Z ) at output θ . For a fixed pair of neighboring datasets Z and Z ′ , the privacy loss random variable (PLRV) of mechanism M : Z → Θ is defined as

for the random variable θ ∼ M ( Z ) .

Lemma K.5 (Adaptive composition (RDP) (Mironov, 2017)) . Let M 1 : Z → R 1 be ( α, ϵ 1 ) -RDP and M 2 : R 1 ×Z → R 2 be ( α, ϵ 2 ) -RDP. Then the mechanism M = ( m 1 , m 2 ) , where m 1 ∼ M 1 ( Z ) and m 2 ∼ M 2 ( Z, m 1 ) , satisfies ( α, ϵ 1 + ϵ 2 ) -RDP

Lemma K.6 (Change of coordinates) . Consider the map g : X → Y between X ⊆ R d and Y ⊆ R d that transforms y = g ( x ) . Then

where ∣ ∣ ∣ ∣ det ∂y ∂x ∣ ∣ ∣ ∣ is the absolute value of the determinant of the Jacobian of the map g :

Lemma K.7 (Change of variables for probability density functions.) . Let g be a strictly monotonic function. Then for y = g ( x ) ,

Lemma K.8 . Let A be an invertible matrix. Then det A -1 = 1 det A .

Lemma K.9 (Maximum Rayleigh quotient) . For any symmetric matrix A ∈ R d × d ,

where λ max is the largest eigenvalue of A .

Lemma K.10 (Bound on the expected norm of multivariate Gaussian with mean 0.) . Let x = N (0 , σ 2 I d ) . Then

Lemma K.11 (Gaussian MGF) . Let X ∼ N ( µ, σ 2 ) for µ ∈ R , σ ∈ R > 0 . The moment generating function of order t is then MGF X ( t ) = e µt + 1 2 σ 2 t 2 . So for any t ∈ R ≥ 0 and σ 1 < σ 2 ,

where X 1 ∼ N ( µ, σ 2 1 ) and X 2 ∼ N ( µ, σ 2 2 ) .

Definition K.12 (Holder's Inequality) . Let X,Y be random variables satisfying E [ | X | p ] < ∞ , E [ | X | q ] < ∞ for p > 1 and 1 p + 1 q = 1 . Then

Definition K.13 (Rényi Divergence) . Let P, Q be distributions with probability density functions P ( x ) , Q ( x ) . The Rényi divergence of order α > 1 between P and Q is given by

Lemma K.14 (MGF inequality for identical distributions) . Let x, y, z ∼ N (0 , σ 2 ) . Then

Proof. Observe that for any a, b ∈ R , it holds that 2 ab ≤ a 2 + b 2 . Applying this along with the Cauchy-Schwarz inequality,

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new privacy analysis for the objective perturbation method for learning differentially private linear models. The new analysis makes use of gradient clipping (similar to DP-SGD) and is shown to have a competitive performance with respect to DP-SGD, while exhibiting better behavior for hyperparameter tuning.\n\nSTRENGTHS:\nThe problem that this paper sets out to address is very interesting and can potentially have (minor) practical implications.\n\nWEAKNESSES:\nMy main issue with this paper is the presentation: \nMany vague sentences, several terms are not properly defined, confusing notation, and several careless mistakes in mathematical definitions. \nI'll be more specific in the following:\n1. The first item in the contribution (line 47) states that the PLRV of the objective perturbation mechanism is \"almost\" Gaussian. The authors failed to elaborate on this point in the draft and it seems to be left to the reader's imagination. The second item (line 53): \"We establish even tighter ($\\epsilon, \\delta$)-DP\". Tighter compared to what. The fourth item (line 64) discusses \"weakly\" differentiable loss functions. What is a weakly differentiable function? \n2. The following terms were not defined: $\\epsilon(\\alpha)$, $L(\\theta, Z)$ (line 219),  \"privacy profile\" (i.e., Lemma  2.6 is very hard to understand without a precise definition of $\\delta(\\epsilon)$),  $\\tilde\\theta^P_T$ (in line 229), the set $\\mathcal X$ in Theorem 4.2, and also $\\hat \\theta^P(Z)$ and $\\hat \\theta^P(Z')$ in Corollary 3.4 (I suspect the authors meant $\\hat \\theta^P(X)$ and $\\hat \\theta^P(X')$ as defined in line 190). Finally, the identity $\\ell(\\theta, z) = \\ell(x^T\\theta, y)$ in line 141 is very confusing; again I think the authors meant $\\ell(\\theta, z) = f(x^T\\theta, y)$ for some functions $f$. \n 3. There are three ways a dataset is referred to in this paper: $Z\\in \\mathcal Z^n$, $Z\\in \\mathcal Z$, and $z\\in Z^n$!! The notation $\\mathbb E [e^{|\\mathcal N (0, \\sigma^2)|}]$ is not a proper mathematical notation! The authors should have clarified that $\\mathcal M(Z)$ denotes both a random variable (i.e., mechanism's output) and its distribution.\n4. The definition of R\\'enyi divergence in line 108 is WRONG (and the mistake was repeated more than 3 times throughout!!)  Another example: It is stated in line 225 that $\\tilde \\theta^P = \\theta^P + (\\tilde \\theta - \\theta^P)$, which implies $\\tilde \\theta^P = \\tilde \\theta$ (which is clearly wrong).\n\nThe second weakness is Algorithm 1: First of all, $\\gamma$ is one of the inputs but hasn't been used! Moreover, the definition of $\\tilde \\theta$ is cryptic. This is  the definition:  Solve for $\\tilde \\theta$  such that $|| \\nabla \\mathcal L_C^P(\\tilde \\theta; Z)||_2\\leq \\tau$. For a given $\\tau$, we might have (infinitely many) different such $\\tilde \\theta$.  \n\nThe third weakness of this work is the presentation of the main results. It is rather difficult to see the significance of the results compared to the existing privacy analyses of the objective perturbation method. The authors failed to put their results in the context of the known results, so the readers can appreciate them. In particular, it is particularly unclear how Algorithm 1 extends Iyengar et al.'s AMP.\n\nQUESTIONS:\nThe authors compared their results for the objective perturbation mechanism with the rather old results for DP-SGD (namely, Bassily et al., 2014). For the convex loss functions, there many more recent privacy analyses for DP-SGD in the literature. For instance, it would be important and insightful if the authors could compare their results with Altschuler and Talwar's result \"Privacy of Noisy Stochastic Gradient Descent: More Iterations without More Privacy Loss\".\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper revisited a differentially private learning algorithm: objective perturbation. The authors gave an RDP analysis of objective perturbation and extended AMP to losses with unbounded gradients using clipped loss function.\n\nSTRENGTHS:\nThe main contributions of the paper are (1) RDP and HK analysis of objective perturbation; (2) an improved version of AMP which extends to losses with unbounded gradients.\n\nI really enjoyed reading the manuscript. The authors did a fantastic job telling the story why objective perturbation should regain attention from the community.\n\nThe analysis of HK and RDP bound, despite not challenging, are good contribution. Another contribution is that the authors found an unacknowledged subtlety in the analysis of objective perturbation and pointed out GLM assumption solved the issue.\n\nThe extended AMP is a direct combination of clipped loss function and AMP. Again, not too technically challenging, but solid contribution.\n\nAnother contribution is that the authors showed that objective perturbation is computationally more efficient than DP-SGD.\n\nWEAKNESSES:\n1. Compared to the pretty theory and story-telling, the evaluation is somewhat weak. First, objective perturbation does not consistently show better performance than honest DP-SGD. Second, even in cases where objective perturbation wins, the winning is marginal. Third, objective perturbation is only evaluated on 2 toy datasets. More realistic datasets are preferred.\n2. The 3rd bullet point in the contribution list emphasized that there is neglected subtlety in the analysis of objective perturbation which might lead to dimensional dependence of PLRV and the paper fixed the issue without touching the topic in detail except a pointer to appendix in remark 3.5. I think this contribution deserves more space in the main text. \n3. typo: line 306 \"boudning\" -> \"bounding\"\n\nQUESTIONS:\nPlease add more experiments if possible\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper revisits objective perturbation as a viable alternative to DP-SGD with tighter analysis and illustrating it competitiveness on a few problems.

STRENGTHS:
This paper explores the interesting direction of fairly comparing objective perturbation and DP-SGD specifically. The paper has multiple strengths

- Paper presents a novel analysis of objective perturbation using RDP, and extending it to an even tighter bound
- It also empirically verifies that its competitive against DP-SGD, especially on lower epsilons, while having the advantage of "free" hyperparam tuning
- Introduces an algorithm to expand the applicability of objective perturbation to a broad range of losses

WEAKNESSES:
Here are a few weaknesses:

- Empirical evaluation of this method can be improved, I had following questions in this regard
  - DP-SGD is known to work best with large batch sizes, I would suggest increasing batch size for for the baselines -- even better is to have a plot with varying batch sizes. Similar argument can made for number of iterations as well. 
  - Even though there is a good reason to use L-BFGS with Alg 1, including an ablation when using Adam with Alg 1 would be useful as a comparison.
-  While not a major one for publication but a still a weakness: the function class that analysis applies to for objective perturbation is limited compared to DP-SGD.
- Another weakness is Alg 1's either matches or performs worse against DP-SGD on epsilons > 0.1

QUESTIONS:
Please see "weaknesses" section

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper derives new privacy analysis of the objective perturbation mechanism. Novel bounds are given for Rényi differential privacy and approximate differential privacy, that are tighter than existing bounds. A practical variant of objective perturbation (where the problem does not have to be solved exactly) is proposed, and evaluated numerically.\n\nSTRENGTHS:\n1. The RDP analysis of objective perturbation is an interesting and original contribution, and the proposed analyses improve over existing results on the privacy of objective perturbation.\n2. A practical variant of the method is proposed, with a convergence criterion that allows tuning the hyperparameter of the underlying optimization sub-routine for free (in terms of privacy).\n3. Empirical results are quite convincing, matching the utility of DP-SGD without requiring a heavy tuning procedure.\n\nWEAKNESSES:\n1. The proposed method is compared with DP-SGD for non-smooth functions [1], which is a different setting. In the smooth setting, DP-SGD and its variants converge faster, which makes the argument that the proposed Objective Perturbation method is more computationally efficient vacuous. (In general, I don't find the discussion on computational cost is so useful, and it could be mostly removed to incorporate more discussion on other results.)\n2. Some literature on differentially private generalized models is missing. In particular, [2] could be a good baseline and is not mentioned in the paper.\n3. No closed-form on the value of $\\sigma$ required to obtain a given RDP or DP guarantee is provided. This makes the comparison with existing results a bit difficult.\n\nIn terms of structure, the paper is not too convincing: a large part of the contributions is deferred to the appendix, and references to appendix in the main text are vague. In general, results are presented without much discussions. In particular, Theorem 3.1 (central result of the paper) is presented without details on the comparison with Theorem 2.8; and Figure 1 is not discussed (and lacks a bit of details for proper understanding). It also seems that a title is missing between lines 184 and 185\n\nSome statements are also not very clear: the regularization $r(\\theta)$ disappears in the definition of $J^P$ (page 4); $\\hat \\theta$ in Corollary 3.4 seems to be the one from Example 3.3 but it is not stated; the problem considered in Figure 1 is not stated.\n\n[1] Private empirical risk minimization: Efficient algorithms and tight error bounds, Raef Bassily et al., 2014.\n[2] Differentially Private Generalized Linear Models Revisited, Raman Arora et al., 2022.\n\nQUESTIONS:\n1. Theorem 2.8 is stated for a fixed $\\epsilon$, while other theorems fix $\\sigma$ and express the privacy parameters ($\\alpha, \\epsilon$ or $\\delta$) as a function of $\\sigma$. What is the reason? It seems that unifying this would significantly help the discussion (e.g. by stating Theorem 2.8 for fixed value of $\\sigma$). Conversely, is it possible to give a closed-form for $\\sigma$ in Theorems 3.1 and 3.2?\n2. In Figure 1, it seems that the gap between Kifer's results and RDP/HS for ObjPert reduces as $\\sigma$ increases. Is it the case, and is it coherent with the theory? Are there some regimes where both analyses give equivalent results?\n3. Why is the comparison with DP-SGD with \"honest\"-tuning so important? It seems that empirical results are competitive even with the tuned DP-SGD, which hints that tuning is not so fundamental in the comparison between the two mechanisms.\n4. It is a bit weird to talk about \"DP-SGD with ADAM optimizer\" (e.g., this variant is not mentioned in [1]): what does it means? Why not compare with the usual DP-SGD?\n5. In general, SGD is known not to be the best optimizer for GLMs. Another approach, closely related to Objective Perturbation, that could be better, is *Output* Perturbation. Why not compare with Output Perturbation? Can the ideas in this paper be used to improve Output Perturbation? \n\n[1] Private empirical risk minimization: Efficient algorithms and tight error bounds, Raef Bassily et al., 2014.\n\nLIMITATIONS:\nThe paper is mostly limited to GLMs, which is clearly stated by the authors. The baselines considered for comparing computational cost are a bit weak (e.g., DP-SGD from [1] does not assume smoothness), which reduces the strength of the argument on the computational cost.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a practical perspective on privacy for unconstrained convex generalized linear problems.  While DP-SGD has been the workhorse for differential privacy in deep learning and prediction/classification problems, the authors suggest that objective perturbation might be more suitable for simpler models.  (Also to provide a more 'honest' privacy mechanism.)\nThe authors provide privacy bounds for objective perturbation based on Renyi DP.  Using the hockey-stick divergence, the authors provide a tighter $(\\epsilon,\\delta)$-DP bound. The authors also introduce computational tools to avoid the necessity of bounded gradient.\n\nThe paper is clear and enjoyable to read.  The authors perform a good analysis of DP in linear settings, and provide usable practical tools.  While the numerical results do not demonstrate a significant improvement with the proposed algorithm, the importance to using objective perturbation or looking beyond DP-SGD in practice is a valuable analysis.\n\nSTRENGTHS:\nThe paper is technically sound - the authors carry out a good analysis and propose an objective perturbation method using various frameworks such Renyi DP and the hockey-stick divergence.\nThe paper makes a meaningful contribution to the practical applicability of DP.\n\nWEAKNESSES:\nA gradient clipping parameter is used in the proposed algorithm - authors should explain why this does not lead a more 'dishonest' privacy mechanism.\n\nThe paragraph that explains the tuning of privacy parameter of objective perturbation (starting line 287, page 9) is not very clear.  How is it that the tuning is data-independent?\n\nThe results on the Adult dataset show that the proposed algorithm performs close to dishonest DP-SGD, but actually the accuracy values of all three methods are quite close.  For the synthetic dataset on the other hand, the algorithm doesn't show an improved performance, even over the honest method. Why is that?  Experiments on more datasets would be necessary to support the importance of objective perturbation.\n\nQUESTIONS:\nA gradient clipping parameter is used in the proposed algorithm - authors should explain why this does not lead a more 'dishonest' privacy mechanism.\n\nThe paragraph that explains the tuning of privacy parameter of objective perturbation (starting line 287, page 9) is not very clear.  How is it that the tuning is data-independent?\n\nThe results on the Adult dataset show that the proposed algorithm performs close to dishonest DP-SGD, but actually the accuracy values of all three methods are quite close.  For the synthetic dataset on the other hand, the algorithm doesn't show an improved performance, even over the honest method. Why is that?  Experiments on more datasets would be necessary to support the importance of objective perturbation.\n\nLIMITATIONS:\nThe authors have not addressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-28"}