SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rachel Redberg  
UC Santa Barbara
Antti Koskela  
Nokia Bell Labs
Yu-Xiang Wang  
UC Santa Barbara
In the arena of privacy-preserving machine learning, differentially private stochastic gradient descent (DP-SGD) has outstripped the objective perturbation mechanism in popularity and interest. Though unrivaled in versatility, DP-SGD requires a non-trivial privacy overhead (for privately tuning the model's hyperparameters) and a computational complexity which might be extravagant for simple models such as linear and logistic regression. This paper revamps the objective perturbation mechanism with tighter privacy analyses and new computational tools that boost it to perform competitively with DP-SGD on unconstrained convex generalized linear problems.
The rise of deep neural networks has transformed the study of differentially private learning no less than any other area of machine learning. Differentially private stochastic gradient descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) has thus gained widespread appeal as a versatile framework for privately training deep learning models.
How does DP-SGD fare on simpler models such as linear and logistic regression? The verdict is unclear. Clearly an algorithm capable of privately optimizing non-convex functions represented by millions of parameters is up to the computational task of fitting a linear model. A more pressing concern is that DP-SGD is up to too much. Look, for example, at the algorithm's computational complexity: DP-SGD requires  $O(n^{2})$  steps to achieve the optimal excess risk bounds for DP convex empirical risk minimization (Bassily et al., 2014).
DP-SGD furthermore takes after its non-private counterpart in sensitivity to hyperparameters. A poor choice of learning rate or batch size, for instance, could lead to suboptimal performance or slow convergence. There are well-established procedures for hyperparameter optimization that typically involve evaluating the performance of the model trained using different sets of candidate hyperparameters. But with privacy constraints, there is a catch: tuning hyperparameters requires multiple passes over the training dataset and thereby constitutes a privacy cost.
At best, existing work tends to circumvent this obstacle by optimistically assuming the availability of a public auxiliary dataset for hyperparameter tuning. More often the procedure for private hyperparameter selection is left largely to the reader's imagination. Only recently have Liu & Talwar (2019) and subsequently Papernot & Steinke (2022) studied how to obtain tighter privacy loss bounds for this task beyond standard composition theorems.
In the meantime, objective perturbation (Chaudhuri et al., 2011; Kifer et al., 2012) has been to some extent shelved as a historical curiosity. Sifting through the literature, we find that opinions are divided: some tout objective perturbation as "[o]ne of the most effective algorithms for differentially private learning and optimization" (Neel et al., 2020), whereas other works (Wang et al., 2017) dismiss
objective perturbation as being impractical and restrictive. Some empirical evaluations (Yu et al., 2019; McKenna et al., 2021) suggest that DP-SGD often achieves better utility in practice than does objective perturbation; others (Iyengar et al., 2019) report the opposite.
Our goal in this paper is to lay some of this debate to rest and demonstrate that for generalized linear problems in particular, objective perturbation can outshine DP-SGD.
- We establish an improved  $(\epsilon, \delta)$ -DP bound for objective perturbation via privacy profiles, a modern tool for privacy accounting that bounds the hockey-stick divergence. The formula can be computed numerically using only calls to Gaussian CDFs. We further obtain a dominating pair of distributions as defined by Zhu et al. (2022) which enables tight composition and amplification by subsampling of the privacy profiles.
- We present a novel Rényi differential privacy (RDP) (Mironov, 2017) analysis of the objective perturbation mechanism. Using this analysis, we show empirically that objective perturbation performs competitively against DP-SGD with "honestly"  $^{1}$ -tuned hyperparameters. The tightest analyses to date of private hyperparameter tuning are the RDP bounds derived in Papernot & Steinke (2022). This tool allows us to empirically evaluate objective perturbation against DP-SGD on a level playing field (Section 5).
- We fix a decade-old oversight in the privacy analysis of objective perturbation. Existing literature overlooks a nuanced argument in the privacy analysis of objective perturbation, which requires a careful treatment of the dependence between the noise vector and the private minimizer. Without assuming GLM structure, the privacy bound of objective perturbation is subject to a dimensional dependence that has gone unacknowledged in previous work<sup>2</sup>.
- We introduce computational tools that expand the applicability of objective perturbation to a broader range of loss functions. The privacy guarantees of objective perturbation require the loss function to have bounded gradient. Our proposed framework extends the Approximate Minima Perturbation framework of Iyengar et al. (2019) to take any smooth loss function as a blackbox, then algorithmically ensure that it has bounded gradient. We also provide a computational guarantee  $O(n \log n)$  on the running time of this algorithm, in contrast to the  $O(n^2)$  complexity of DP-SGD for achieving information-theoretic limits.
1.2 A Short History of DP Learning
Differentially private learning dates back to Chaudhuri et al. (2011), which extended the output perturbation method of Dwork et al. (2006) to classification algorithms and also introduced objective perturbation. In its first public appearance, objective perturbation required gamma-distributed noise; Kifer et al. (2012) provided a refined analysis of the mechanism with Gaussian noise, which is the entry point into our work.
Differentially private stochastic gradient descent (DP-SGD) (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) brought DP into the fold of modern machine learning, allowing private training of models with arbitrarily complex loss landscapes that can scale to enormous datasets. DP-SGD adds Gaussian noise at every iteration to an aggregation of clipped gradients, and thus privacy analysis for DP-SGD often boils down to finding tight composition bounds (of the subsampled Gaussian mechanism).
The initial version of DP-SGD based on the standard strong composition (Bassily et al., 2014) is not quite practical, but that has changed, thanks to a community-wide effort in developing modern numerical privacy accounting tools in the past few years. These include the moments accountant that composes Renyi DP functions (Abadi et al., 2016; Wang et al., 2019; Mironov et al., 2019) and the Fourier accountant (also known as PLV or PLD accountants) that directly compose the privacy profiles (Sommer et al., 2019; Koskela et al., 2020; Gopi et al., 2021; Zhu et al., 2022). It is safe to conclude that the numerically computed privacy loss of DP-SGD using these modern tools is now very precise.
"Honest" hyperparameter tuning is a term coined by Mohapatra et al. (2022).
2The concurrent work of Agarwal et al. (2023) has independently identified this bug as well.
Because DP-SGD releases each intermediate model, the algorithm can stop after any number of iterations and simply accumulates privacy loss as it goes. In contrast, the privacy guarantees of objective perturbation hold only when the output of the mechanism is the exact minima of the perturbed objective. This requirement is at odds with practical convex optimization frameworks which typically use first-order methods to search for an approximate solution.
To remedy this, Iyengar et al. (2019) proposed an approach to approximately minimize a perturbed objective function while maintaining privacy. Approximate Minima Perturbation (AMP) was introduced as a tractable alternative to objective perturbation whose privacy guarantees permit the output to be an approximate (rather than exact) solution to the perturbed minimization problem. In this paper we extend AMP to a broader range of loss functions; Algorithm 1 can be viewed as a special case of AMP with a transformation of the loss function.
Differential privacy (DP) (Dwork et al., 2006) offers provable privacy protection by restricting how much the output of a randomized algorithm can leak information about a single data point.
DP requires a notion of how to measure similarity between datasets. We say that datasets  $Z$  and  $Z'$  are neighboring datasets (denoted  $Z \simeq Z'$ ) if they differ by exactly one datapoint  $z$ , i.e.  $Z' = Z \cup \{z\}$  or  $Z' = Z \setminus \{z\}$  for some data entry  $z$ .
Definition 2.1 (Differential privacy). A mechanism  $\mathcal{M}:\mathcal{Z}\to \mathcal{R}$  satisfies  $(\epsilon ,\delta)$ -differential privacy if for all neighboring datasets  $Z$ $\dot{Z}^{\prime}\in \mathcal{Z}$  and output sets  $S\subseteq \mathcal{R}$ ,
$$
\Pr \left[ \mathcal {M} (Z) \in S \right] \leq e ^ {\epsilon} \Pr \left[ \mathcal {M} \left(Z ^ {\prime}\right) \in S \right] + \delta .
$$
When  $\delta > 0$ ,  $\mathcal{M}$  satisfies approximate  $DP$ . When  $\delta = 0$ ,  $\mathcal{M}$  satisfies the stronger notion of pure  $DP$ .
We say that  $\mathcal{M}$  is tightly  $(\epsilon, \delta)$ -DP if there is no  $\delta' < \delta$  for which  $\mathcal{M}$  would be  $(\epsilon, \delta')$ -DP.
In what follows, we overview two different styles of achieving DP guarantees: one via hockey-stick divergence, and the other via Rényi divergence.
2.1.1 DP via hockey-stick divergence
Definition 2.2 (Hockey-stick divergence). Denote  $[x]_{+} = \max \{0, x\}$  for  $x \in \mathbb{R}$ . For  $\alpha > 0$  the hockey-stick divergence  $H_{\alpha}$  from a distribution  $P$  to a distribution  $Q$  is defined as
$$
H _ {\alpha} (P | | Q) = \int [ P (x) - \alpha \cdot Q (x) ] _ {+} \mathrm {d} x.
$$
Now (with some abuse of notation) we will discuss how to bound the hockey-stick divergence between distributions  $\mathcal{M}(Z)$  and  $\mathcal{M}(Z^{\prime})$  via the concept of privacy profiles.
Definition 2.3 (Privacy profiles Balle et al., 2018). The privacy profile  $\delta_{\mathcal{M}}(\epsilon)$  of a mechanism  $\mathcal{M}$  is defined as
$$
\delta_ {\mathcal {M}} (\epsilon) := \max  _ {Z \simeq Z ^ {\prime}} H _ {\mathrm {e} ^ {\epsilon}} \left(\mathcal {M} (Z) | | \mathcal {M} \left(Z ^ {\prime}\right)\right).
$$
Tight  $(\epsilon, \delta)$ -DP bounds can then be obtained as follows.
Lemma 2.4 (Zhu et al., 2022, Lemma 5). Mechanism  $\mathcal{M}$  satisfies  $(\epsilon, \delta)$ -DP if any only if  $\delta \geq \delta_{\mathcal{M}}(\epsilon)$ .
Dominating pairs of distributions are useful for bounding the hockey-stick divergence  $H_{\mathrm{e}^{\epsilon}}(\mathcal{M}(Z)||\mathcal{M}(Z'))$  accurately and, in particular, for obtaining tight bounds for compositions.
Definition 2.5 (Zhu et al. 2022). A pair of distributions  $(P, Q)$  is a dominating pair of distributions for mechanism  $\mathcal{M}: \mathcal{Z} \to \mathcal{R}$  if for all neighboring datasets  $Z$  and  $Z'$  and for all  $\alpha > 0$ ,
$$
H _ {\alpha} \left(\mathcal {M} (Z) | | \mathcal {M} \left(Z ^ {\prime}\right)\right) \leq H _ {\alpha} (P | | Q).
$$
2.1.2 DP via Renyi divergence
Definition 2.6. (Rényi divergence.) Let  $\alpha > 0$ . For  $\alpha \neq 1$ , the Rényi divergence  $D_{\alpha}$  from distribution  $P$  to distribution  $Q$  is defined as
$$
D _ {\alpha} (P | | Q) = \frac {1}{\alpha - 1} \log \mathbb {E} _ {x \sim Q} \left[ \left(\frac {P (x)}{Q (x)}\right) ^ {\alpha} \right].
$$
When  $\alpha = 1$ , Rényi divergence reduces to the Kullback-Leibler (KL) divergence:
$$
D _ {1} (P | | Q) = \mathbb {E} _ {x \sim P} \left[ \log \left(\frac {P (x)}{Q (x)}\right) \right].
$$
Rényi differential privacy (RDP) is a relaxation of pure DP ( $\delta = 0$ ) based on Rényi divergence.
Definition 2.7 (Rényi differential privacy). A mechanism  $\mathcal{M}: \mathcal{Z} \to \mathcal{R}$  satisfies  $(\alpha, \epsilon)$ -Rényi differential privacy if for all neighboring datasets  $Z, Z' \in \mathcal{Z}$ ,
$$
D _ {\alpha} \left(\mathcal {M} (Z) \mid \mid \mathcal {M} \left(Z ^ {\prime}\right)\right) \leq \epsilon ,
$$
RDP implies  $(\epsilon, \delta)$ -DP for any  $0 < \delta \leq 1$  with  $\epsilon = \min_{\alpha > 1} \{\epsilon(\alpha) + \log(1/\delta) / (\alpha - 1)\}$ . Tighter but more complex conversion formulae were derived by Balle et al. (2020) and Canonne et al. (2020), which we adopt numerically in our experiments whenever approximate DP is needed.
2.2 Differentially Private Empirical Risk Minimization
We have a dataset  $Z \in \mathcal{Z}$  and a loss function  $\ell(\theta; z)$ ; we want to solve problems of the form
$$
\hat {\theta} = \underset {\theta \in \Theta} {\arg \min } \sum_ {z \in Z} \ell (\theta ; z) + r (\theta),
$$
where  $z = (x,y)\in \mathcal{X}\times \mathcal{Y}$  is a data point and  $r(\theta)$  is a regularization term. The feature space is  $\mathcal{X}\subseteq \mathbb{R}^d$  and the label space is  $\mathcal{V}\subseteq \mathbb{R}$ . We will assume that  $||x||_2\leq 1$  and  $|y|\leq 1$ .
This work focuses on unconstrained convex generalized linear models (GLMs): we require that  $\ell(\theta)$  and  $r(\theta)$  are convex and twice-differentiable and that  $\Theta = \mathbb{R}^d$ . The loss function is assumed to have GLM structure of the form  $\ell(\theta; z) = f(x^T\theta; y)$ .
Objective Perturbation Construct the perturbed objective function by sampling  $b \sim \mathcal{N}(0, \sigma^2 I_d)$ :
$$
\mathcal {L} ^ {P} (\theta ; Z, b) = \sum_ {z \in Z} \ell (\theta ; z) + \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} + b ^ {T} \theta .
$$
The objective perturbation mechanism (ObjPert) outputs  $\hat{\theta}^P (Z) = \arg \min_{\theta \in \Theta}\mathcal{L}^P (\theta ;Z,b)$ .
Theorem 2.8 (DP guarantees of objective perturbation (Kifer et al., 2012)). Let  $\ell(\theta; z)$  be convex and twice-differentiable such that  $\|\nabla \ell(\theta; z)\|_2 \leq L$  and  $\nabla^2 \ell(\theta; z) \prec \beta I_d$  for all  $\theta \in \Theta$  and  $z \in \mathcal{X} \times \mathcal{Y}$ .
Then objective perturbation satisfies  $(\epsilon, \delta)$ -DP when  $\lambda \geq \frac{2\beta}{\epsilon}$  and  $\sigma \geq \frac{L\sqrt{8\log(2/\delta) + 4\epsilon}}{\epsilon}$ .
Differentially Private Gradient Descent DP-SGD is a differentially private version of stochastic gradient descent which ensures privacy by clipping the per-example gradients at each iteration before aggregating them and adding noise to the result. The update rule at iteration  $t$  is given by
$$
\theta_ {t + 1} = \theta_ {t} - \eta_ {t} \left(\sum_ {z \in B _ {t}} \operatorname {c l i p} \left(\nabla \ell \left(\theta_ {t}; z\right)\right) + \mathcal {N} \left(0, \sigma^ {2} I _ {d}\right)\right),
$$
where  $\eta_{t}$  is the learning rate at iteration  $t$ ,  $B_{t}$  is the current batch at iteration  $t$ ,  $\sigma$  is the noise scale, and  $\mathrm{clip}$  is a function that bounds the norm of the per-example gradients.
3 Analytical Tools
Existing privacy guarantees of the objective perturbation mechanism (Chaudhuri et al., 2011; Kifer et al., 2012) pre-date modern privacy accounting tools such as Rényi differential privacy and privacy profiles. In this section, we present two new privacy analyses of objective perturbation: an  $(\epsilon, \delta)$ -DP bound based on privacy profiles, and an RDP bound.
3.1 Approximate DP Bound
Theorem 3.1 (Approximate DP guarantees of objective perturbation for GLMs). Consider a loss function  $\ell(\theta; z) = f(x^T\theta; y)$  with GLM structure. Suppose that  $f$  is  $\beta$ -smooth and  $||\nabla \ell(\theta; z)||_2 \leq L$  for all  $\theta \in \Theta$  and  $z \in \mathcal{X} \times \mathcal{Y}$ . Fix  $\lambda > \beta$ . Let  $\epsilon \geq 0$  and let  $\tilde{\epsilon} = \epsilon - \log\left(1 - \frac{\beta}{\lambda}\right)$ ,  $\hat{\epsilon} = \epsilon - \log\left(1 - \frac{\beta}{\lambda}\right) - \frac{L^2}{2\sigma^2}$ , and let  $P$  and  $Q$  be the density functions of  $\mathcal{N}(L, \sigma^2)$  and  $\mathcal{N}(0, \sigma^2)$ , respectively. Objective perturbation satisfies  $(\epsilon, \delta(\epsilon))$ -DP for
$$
\delta (\epsilon) = \left\{ \begin{array}{l l} 2 \cdot H _ {e ^ {\hat {\epsilon}}} (P | | Q), & \text {i f} \hat {\epsilon} \geq 0, \\ \left(1 - \mathrm {e} ^ {\hat {\epsilon}}\right) + \mathrm {e} ^ {\hat {\epsilon}} \cdot 2 \cdot H _ {e ^ {\frac {L ^ {2}}{\sigma^ {2}}}} (P | | Q), & \text {o t h e r w i s e .} \end{array} \right. \tag {3.1}
$$
Notice that we can express (3.1) analytically using (B.1). To obtain the bound (3.1) we repeatedly use the fact that the privacy loss random variable (PLRV) determined by the distributions  $\mathcal{N}(1,\sigma^2)$  and  $\mathcal{N}(0,\sigma^2)$  is distributed as  $\mathcal{N}\left(\frac{1}{2\sigma^2},\frac{1}{\sigma^2}\right)$ . As the upper bound (3.1) is obtained using a PLRV that is a certain scaled and shifted half-normal distribution, we can also find certain scaled and shifted half-normal distributions  $P$  and  $Q$  which give the dominating pair of distributions for the objective perturbation mechanism such that the hockey-stick divergence between  $P$  and  $Q$  is exactly the upper bound (3.1) for all  $\epsilon$  (shown in the appendix).
3.2 Rényi Differential Privacy Bound
If our sole objective is to obtain the tightest possible approximate DP bounds for objective perturbation, we can stop at Theorem 3.1! Directly calculating the privacy profiles of objective perturbation using the hockey-stick divergence, as in the previous section, will achieve this goal (until more privacy accounting tools come along).
In this section we turn instead to Rényi differential privacy, a popular relaxation of pure differential privacy  $(\delta = 0)$  which avoids the "catastrophic privacy breach" possibility permitted by approximate DP  $(\delta > 0)$ . Below, we present an RDP guarantee for objective perturbation.
Theorem 3.2 (RDP guarantees of objective perturbation for GLMs). Consider a loss function  $\ell(\theta; z) = f(x^T\theta; y)$  with GLM structure. Suppose that  $f$  is  $\beta$ -smooth and  $||\nabla \ell(\theta;z)||_2 \leq L$  for all  $\theta \in \Theta$  and  $z \in \mathcal{X} \times \mathcal{Y}$ . Fix  $\lambda > \beta$ . Objective perturbation satisfies  $(\alpha, \epsilon)$ -RDP for any  $\alpha > 1$  with
$$
\epsilon = - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {X \sim \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {(\alpha - 1) | X |} \right].
$$
For  $\alpha = 1$ , the RDP bound holds with
$$
\epsilon = - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L ^ {2}}{2 \sigma^ {2}} + \log \mathbb {E} _ {X \sim \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {| X |} \right].
$$
One of our main motivations for improving the privacy analysis of objective perturbation comes from the observation that it can be competitive to DP-SGD when the privacy cost of hyperparameter tuning is included in the privacy budget. As the tightest results for DP hyperparameter tuning are stated in terms of RDP (Papernot & Steinke, 2022), in our experiments we use RDP bounds of objective perturbation to get a clear understanding of the differences in the privacy-utility trade-offs between these two approaches.
Remark 3.3. Privacy profile and RDP bounds (such as Theorems 3.1 and 3.2) are unified in the sense that they are both based on a certain bound of the PLRV  $\epsilon_{Z,Z^{\prime}}$  (Definition K.4) for a fixed pair of datasets  $Z, Z^{\prime}$ . From Definitions 2.2 and 2.7 we see that for  $\epsilon \in \mathbb{R}$ , the hockey-stick divergence is
$$
H _ {\mathrm {e} ^ {\epsilon}} \big (\mathcal {M} (Z) | | \mathcal {M} (Z ^ {\prime}) \big) = \mathbb {E} _ {\theta \sim \mathcal {M} (Z)} \left[ 1 - \mathrm {e} ^ {\epsilon - \epsilon_ {Z, Z ^ {\prime}} (\theta)} \right] _ {+},
$$
and for  $\alpha > 1$  we have that the Rényi divergence is
$$
D _ {\alpha} \big (\mathcal {M} (Z) | | \mathcal {M} (Z ^ {\prime}) \big) = \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta \sim \mathcal {M} (Z)} \left[ \mathrm {e} ^ {\alpha \epsilon_ {Z, Z ^ {\prime}} (\theta)} \right].
$$
3.3 Distance to Optimality
How close to optimal are the bounds of Theorems 3.1 and 3.2? We can in fact show that the Gaussian mechanism is a special case of the objective perturbation mechanism — thereby providing a lower bound on its approximate DP and RDP.
Example 3.4. Consider the loss function  $\ell(\theta; x) = x^T\theta$  and choose neighboring datasets  $X = \{x\}$  and  $X' = \emptyset$ , for some  $x \in \mathbb{R}^d$ . Fix  $\lambda > 0$  and sample  $b \sim \mathcal{N}(0, \sigma^2 I_d)$ . Then the objective perturbation mechanism solves
$$
\hat {\theta} ^ {P} (X) = \underset {\theta \in \mathbb {R} ^ {d}} {\arg \min} x ^ {T} \theta + \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} + b ^ {T} \theta = - \frac {1}{\lambda} (x + b),
$$
$$
\hat {\theta} ^ {P} (X ^ {\prime}) = \underset {\theta \in \mathbb {R} ^ {d}} {\arg \min } \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} + b ^ {T} \theta = - \frac {1}{\lambda} b.
$$
Observe that  $\hat{\theta}^P (X)\sim \mathcal{N}(-\frac{1}{\lambda} x,\frac{\sigma^2}{\lambda^2} I_d)$  and  $\hat{\theta}^{P}(X^{\prime})\sim \mathcal{N}(0,\frac{\sigma^{2}}{\lambda^{2}} I_{d})$  . Following the problem setting described in Theorem 2.8, we have that  $||x||_2 = ||\nabla \ell (\theta ;x)||_2\leq L$  . In this case, objective perturbation reduces directly to the Gaussian mechanism with sensitivity  $\Delta_f = \frac{L}{\lambda}$  and noise scale  $\frac{\sigma}{\lambda}$  Corollary 3.5. As a consequence of Example 3.4 and the scaling invariance of the hockey-stick divergence, for all  $\alpha >1$  we have the following:
$$
H _ {\alpha} \big (\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} (Z ^ {\prime}) \big) \geq H _ {\alpha} \big (\mathcal {N} (0, \frac {\sigma^ {2}}{\lambda^ {2}}) | | \mathcal {N} (\frac {L}{\lambda}, \frac {\sigma^ {2}}{\lambda^ {2}}) \big) = H _ {\alpha} \big (\mathcal {N} (0, \sigma^ {2}) | | \mathcal {N} (L, \sigma^ {2}) \big).
$$
The argument works the same for the Rényi divergence  $D_{\alpha}$  which is similarly scaling-invariant. Corollary 3.5 implies that we can measure the tightness of the bounds given in Theorems 3.1 and 3.2 by comparing them to the tight bounds of the Gaussian mechanism (B.1) with sensitivity  $\Delta_f = L$  and noise scale  $\sigma$ .
This means that in Figure 1, the hockey-stick divergence of the Gaussian mechanism is a lower bound on the hockey-stick divergence for objective perturbation. While our hockey-stick divergence bound is unsurprisingly a bit tighter than the RDP bound for objective perturbation, we see that both significantly improve over the classic  $(\epsilon, \delta)$ -DP bounds of Kifer et al. (2012).
Figure 1: Comparison of different  $(\epsilon, \delta)$ -bounds for objective perturbation: the  $(\epsilon, \delta)$ -bound by Kifer et al. (2012) given in Thm. 2.8, the RDP bound of Thm. 3.2, the approximate DP bound of Thm. 3.1 using the hockey-stick divergence and the approximate DP lower bound obtained using the hockey-stick divergence and Cor. 3.5. Left:  $\sigma = 5.0$ ,  $\beta = 1.0$  and  $\lambda = 20.0$ . Right:  $\sigma = 10.0$ ,  $\beta = 1.0$  and  $\lambda = 5.0$ .
Remark 3.6. The RDP and approximate DP bounds in this section require a careful analysis of the dependence between the noise vector  $b$  and the private minimizer  $\theta^P$ . In the appendix, we show how the GLM assumption simplifies this issue.
4 Computational Tools
In this section we present Algorithm 1, which extends the Approximate Minima Perturbation of Iyengar et al. (2019) to handle loss functions with unbounded gradient.