SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Approximate minima The privacy guarantees of objective perturbation hold only when its output is an exact minimizer of the perturbed objective. Approximate Minima Perturbation (AMP) (Iyengar et al., 2019) addresses this issue by finding an approximate minimizer to the perturbed objective, then privately releases this approximate minimizer with the Gaussian mechanism.
Gradient clipping DP-SGD requires no a priori bound on the gradient of the loss function; at each iteration, the algorithm clips the per-example gradients above a pre-specified threshold in order to bound the gradient norms. We extend this same technique to objective perturbation.
Given a loss function  $\ell(\theta; z)$  and a clipping threshold  $C$ , we can transform the gradient of  $\ell(\theta; z)$  as follows:
$$
\nabla \ell_ {C} (\theta ; z) = \left\{ \begin{array}{l l} \nabla \ell (\theta ; z) & \text {i f} | | \nabla \ell (\theta ; z) | | _ {2} \leq C, \\ \frac {C}{| | \nabla \ell (\theta ; z) | | _ {2}} \nabla \ell (\theta ; z) & \text {i f} | | \nabla \ell (\theta ; z) | | _ {2} > C. \end{array} \right.
$$
Then we can define the aggregation of clipped gradients as  $\nabla \mathcal{L}_C(\theta; Z) = \sum_{z \in Z} \nabla \ell_C(\theta; z)$ .
The aggregation of clipped gradients  $\nabla \mathcal{L}_C(\theta; Z)$  corresponds to an implicit "clipped-gradient" objective function  $\mathcal{L}_C(\theta; Z)$ . For convex GLMs, Song et al. (2020) define this function precisely and show that it retains the convexity and GLM structure of the original objective function  $\mathcal{L}(\theta; Z)$ . We furthermore demonstrate that this function retains the same bound  $\beta$  on the Lipschitz smoothness (Theorem E.3).
Algorithm 1 extends the privacy guarantees of AMP (Iyengar et al., 2019) to loss functions with unbounded gradient. Notice that for smooth loss functions with gradient norm bounded by  $L$ , we can set  $C = L$  in order to recover Approximate Minima Perturbation.
Algorithm 1 Computational Objective Perturbation
Input: dataset  $Z$ ; noise levels  $\sigma, \sigma_{\mathrm{out}}$ ;  $\beta$ -smooth loss function  $\ell(\cdot)$ ; regularization strength  $\lambda$ ; gradient norm threshold  $\tau$ ; clipping threshold  $C$ .
1. Construct the set of clipped-gradient loss functions  $\{\ell_C(\theta;z):z\in Z\}$ .
2. Sample  $b\sim \mathcal{N}(0,\sigma^2 I_d)$
3. Let  $\mathcal{L}_C^P (\theta ;Z,b) = \sum_{z\in Z}\ell_C(\theta ;z) + \frac{\lambda}{2} ||\theta ||_2^2 +b^T\theta .$
4. Solve for  $\tilde{\theta}$  such that  $||\nabla \mathcal{L}_C^P (\tilde{\theta};Z)||_2\leq \tau$
Output:  $\tilde{\theta}^P = \tilde{\theta} +\mathcal{N}(0,\sigma_{\mathrm{out}}^2 I_d)$
Theorem 4.1 (RDP guarantees of Algorithm 1). Consider a loss function  $\ell(\theta; z) = f(x^T\theta; y)$  with GLM structure, such that  $f$  is  $\beta$ -smooth. Fix  $\lambda > \beta$ . Algorithm 1 satisfies  $(\alpha, \epsilon)$ -RDP for any  $\alpha > 1$  with
$$
\epsilon \leq - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {C ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {X \sim \mathcal {N} \left(0, \frac {C ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {(\alpha - 1) | X |} \right] + \frac {2 \tau^ {2} \alpha}{\sigma_ {o u t} ^ {2} \lambda^ {2}}.
$$
Remark 4.2. Gradient clipping aside, our proof of Theorem 4.1 takes a different tack than the proof of Theorem 1 (for AMP) in Iyengar et al. (2019). We observe that Algorithm 1 is essentially an adaptive composition of the objective perturbation mechanism and the Gaussian mechanism. We can write  $\tilde{\theta} = \theta^{P} + (\tilde{\theta} -\theta^{P})$  to see that we are releasing two quantities:  $\theta^P$  (with objective perturbation) and the difference  $\tilde{\theta} -\theta^{P}$  (with the Gaussian mechanism). Algorithm 1 stops iterating only after the gradient norm  $||\nabla \mathcal{L}_C^P (\tilde{\theta};Z)||_2$  is below the threshold  $\tau$ . This along with the  $\lambda$ -strong convexity of the objective function  $\nabla \mathcal{L}_C^P (\theta ;Z)$  ensures a bound on the  $\ell_2$ -sensitivity of the difference  $\tilde{\theta} -\theta^{P}$ , so that we can apply the Gaussian mechanism.
4.1 Computational Guarantee
To achieve the optimal excess risk bounds for DP-ERM in the convex setting, DP-SGD clocks in at a hefty  $O(n^{2})$  gradient evaluations (Bassily et al., 2014). It has been an open problem to obtain optimal DP-ERM algorithms that runs in subquadratic time (Kulkarni et al., 2021). One of our contributions is to show that when we further restrict to smooth GLM-losses (so ObjPert is applicable) Algorithm 1 can achieve the same optimal rate with only  $O(n\log n)$  gradient evaluations.
A formal claim and proof that Algorithm 1 — with appropriately chosen parameters — achieves the optimal rate is deferred to Appendix H. The analysis is largely the same as those in (Kifer et al., 2012) but with the bug fixed (details in Appendix G) by adding a GLM assumption.
The improved computational complexity is due to that we can apply any off-the-shelf optimization algorithms to solve Step 4 of Algorithm 1. Observe that  $\mathcal{L}^P (\theta ;Z,b)$  has a finite-sum structure, we can employ the Stochastic Averaged Gradient (SAG) method (Schmidt et al., 2017) which halts in  $O(n\log n)$  with high probability. Details are provided in Appendix F.
In this section we evaluate Algorithm 1 against two baselines: "dishonest" DP-SGD and "honest" DP-SGD. Dishonest DP-SGD does not account for the privacy cost of hyperparameter tuning; honest DP-SGD follows the private selection algorithm and RDP bounds from Papernot & Steinke (2022).
Our experimental design includes some guidelines in order to make it a fair fight. One of the strengths of Algorithm 1 that we advocate for is its blackbox optimization. Whereas DP-SGD consumes privacy budget for testing each set of hyperparameter candidates, an advantage of approximate minima perturbation is that the privacy guarantees are independent of the choice of optimizer used to solve for  $\tilde{\theta}$ . We can therefore test out any number of optimization hyperparameters for Algorithm 1 at no additional privacy cost, provided that these parameters are independent of the privacy guarantee (e.g. learning rate, batch size). More specifically, once the loss function is perturbed with the noise  $b$  in Algorithm 1, any  $\tilde{\theta}$  that satisfies the convergence guarantees with the tolerance parameter  $\tau$  will have the RDP-guarantees of Theorem 4.1 and therefore we are free to also carry out tuning of the optimization algorithm without an additional privacy cost.
Because we are interested in measuring the effect of the privacy cost of hyperparameter tuning, we tune only the learning rate which does not affect the privacy guarantee of the base algorithm. This isolates the effect of hyperparameter tuning as we will need to appeal to Papernot & Steinke (2022) to get valid DP bounds for DP-SGD, but Algorithm 1 enjoys hyperparameter tuning "for free".
The following table summarizes the optimization-related parameters for all three methods.
Dishonest DP-SGD	Honest DP-SGD	Algorithm 1
clipping	C = √2	C = √2	C = √2
learning rate	log(10-8, 10-1)	log(10-8, 10-1)	linear(.08, .5)
grid size	s = 10	K ~ Poisson(μ) s.t. Pr [K > s] = 0.9	s = 10
optimizer	Adam	Adam	L-BFGS
convergence	after T iterations	after T iterations	||∇L(θ)||2 ≤ τ
The choice of  $C = \sqrt{2}$  is a natural value for logistic regression in that  $||\nabla \ell(\theta, z)|| \leq \sqrt{2}$  for all  $\theta, z$  due to data-preprocessing and the bias term. Dishonest DP-SGD selects  $s = 10$  learning rate candidates evenly log-spaced from the range of values between  $10^{-8}$  and  $10^{-1}$ . Honest DP-SGD selects learning rate candidates from the same range of values, but with granularity determined by a random variable  $K$  sampled from the Poisson distribution Poisson  $(\mu)$ . We select  $\mu$  so that with  $90\%$  probability,  $K$  is larger than the grid size  $s$  used for dishonest DP-SGD, resulting in  $\mu \approx 15.4$ .
We use the Adam optimizer for both honest and dishonest DP-SGD. For Algorithm 1 we use the L-BFGS optimizer whose second-order behavior allows us to get within a smaller distance to optimal (as required by Algorithm 1).
For DP-SGD we set the subsampling ratio such that the expected batch size is 256 and we run for 60 "epochs". We calculate the number of iterations as  $T = 60 \cdot \text{num\_batches}$ , where num_batches is the number of batches in the training dataset (we pass the train loader through the Opacus privacy engine, so the size of each batch is random). To calibrate the scale of the noise for DP-SGD, we use the analytical moments accountant (Wang et al., 2019) with Poisson sampling (Zhu & Wang, 2019; Mironov et al., 2019).
The parameters specific to AMP are  $\sigma_{out}$ , the noise scale for the output perturbation step; and  $\tau$ , the gradient norm bound. A larger  $\tau$  will improve our computational cost, though our approximate minimizer  $\tilde{\theta}$  will be farther away from the true minimizer  $\theta^P$ . We can achieve a smaller  $\tau$  by choosing a larger  $\sigma_{out}$ , but this will mean that our release of  $\tilde{\theta}$  will be noisier. In our experiments we fix  $\tau = 0.01$  and  $\sigma_{out} = 0.15$ .
The privacy parameters of objective perturbation are the noise scale  $\sigma$  and the regularization strength  $\lambda$ . Balancing these parameters is a classic exercise in bias-variance trade-off. A larger  $\sigma$  will allow us to use less regularization, but if  $\sigma$  is too large then we risk adding too much noise to the objective function and hurting utility.
Our strategy is to find the smallest possible  $\lambda$  such that  $\sigma$  isn't too large. To quantify when  $\sigma$  is "too large", we use the Gaussian mechanism as a reference point: the noise scale  $\sigma$  for objective perturbation shouldn't be too much larger than the noise scale  $\sigma_{G}$  for the Gaussian mechanism. Let's say that the Gaussian mechanism with noise scale  $\sigma_{G}$  satisfies  $(\epsilon, \delta)$ -DP, then we want our  $\sigma$  for  $(\epsilon, \delta)$ -DP objective perturbation to satisfy  $\sigma \leq f\sigma_{G}$  for some small constant factor  $f$ . In our experiments, we set  $f = 1.3$ .
For objective perturbation, we can thus select the privacy parameters  $\sigma$  and  $\lambda$  using fixed values (e.g.,  $\epsilon, \delta, \sigma_G$ ) that are independent of the data. Likewise, the choices of  $\sigma_{out}$  and  $\tau$  are fixed across all datasets. This is noteworthy since  $\sigma, \lambda, \sigma_{out}$  and  $\tau$  each have an effect on the privacy guarantee, outside of the blackbox algorithm. Tuning these parameters on the data would require us to use the private selection algorithm.
Figure 2: Comparison of Algorithm 1 against honest and dishonest DP-SGD baselines, varying  $\epsilon \in \{0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 4.0, 8.0\}$  and fixing  $\delta = 10^{-5}$ . On all three methods, we train the model for each learning rate on its grid (see Table 5) and report the test accuracy for the best learning rate on the grid. Results are averaged over 10 trials and the error bars on both sides of the mean values depict 1.96 times the standard error, giving the asymptotic  $95\%$  coverage.
We evaluate our methods for binary classification on the Adult, Synthetic-L, Synthetic-H and Gisette datasets provided by Iyengar et al. (2019). We normalize each row  $x_{i}$  to have unit  $\ell_2$ -norm. Note
that the assignment  $x_{i} \gets \frac{x_{i}}{||x_{i}||_{2}}$  doesn't require expending any privacy budget as each data point is transformed only by its own per-sample norm.
Table 1: Synthetic-L
Dishonest	Alg 1	Honest
ε = 0.1	93.85%	90.05%	77.34%
ε = 1	95.25	94.50%	93.15%
ε = 8	95.43%	95.30%	95.17 %
Table 2: Adult
Dishonest	Alg 1	Honest
ε = 0.1	81.61%	81.37%	78.32%
ε = 1	83.54%	83.18%	82.40%
ε = 8	84.19%	83.99%	83.66%
Results from Figure 5 in numerical format for the low-dimensional datasets, Synthetic-L and Adult. For  $\epsilon = 0.1$ , these can be cross-referenced with the results in Fig. 3 from (Iyengar et al., 2019).
The experimental results shown in Figure 5, Table 1 and Table 2 paint a consistent picture. While dishonest DP-SGD is clearly the best-performing algorithm, when we account for the cost of hyperparameter tuning then Algorithm 1 can typically best honest DP-SGD. This effect is especially pronounced under small  $\epsilon$ , for which diverting some of the limited privacy budget to hyperparameter tuning could be more impactful.
Is it fair? Our experimental design aims to fairly compare ObjPert to DP-SGD. One limitation, however, is that the state-of-the-art tools for private hyperparameter tuning from Papernot & Steinke (2022) are RDP bounds — and RDP is not state-of-the-art for DP-SGD privacy accounting. At this moment, the tighest privacy accounting tool for DP-SGD is the PRV accountant from Gopi et al. (2021), which is the counterpart to our privacy profiles analysis for ObjPert (Theorem 3.1). Unfortunately, even though dishonest DP-SGD would benefit from using the PRV accountant, for private hyperparameter tuning we would then have to use the sub-optimal private selection bounds for approximate DP from Liu & Talwar (2019). In our experiments we therefore use RDP-based privacy accounting for both ObjPert and DP-SGD. Comparing DP-SGD with numerical composition against ObjPert with Theorem 3.1 will have to wait until more private selection tools are available.
One might also object that by tuning only the learning rate for DP-SGD, we didn't explore the full range of hyperparameters relevant to DP-SGD's performance. While tuning additional hyperparameters such as the batch size and number of epochs could benefit dishonest DP-SGD, it would likely worsen the privacy-utility tradeoff for honestly-tuned DP-SGD due to the increased privacy cost of the hyperparameter tuning algorithm from Papernot & Steinke (2022).
One point that we really wanted to drive home is that while DP-SGD works extraordinarily well across a wide variety of problem settings, it's not necessarily the best solution for every problem setting. But at the same time, DP-SGD has received the benefit of an enormous amount of attention that other DP learning algorithms haven't received. A goal of our paper was to hone in on a particular problem setting and give a different algorithm the same star treatment.
Objective perturbation now boasts two new privacy analyses. One is an improved  $(\epsilon, \delta)$ -DP analysis based on bounding the hockey-stick divergence. The other is an RDP analysis which allows us to fairly compare objective perturbation against DP-SGD — the workhorse of differentially private learning — with honest hyperparameter tuning. We've also expanded the approximate minima perturbation algorithm of Iyengar et al. (2019) in order to encompass a broader range of loss functions which need not have bounded gradient. Our algorithm moreover can be used in conjunction with SVRG to guarantee a running time of  $O(n \log n)$  to achieve the optimal excess risk bounds, improving on the  $O(n^2)$  computational guarantee of DP-SGD.
The work was partially supported by NSF Award #2048091.
Liu, J. and Talwar, K. Private selection from private candidates. In Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing, pp. 298-309, 2019.
McKenna, R., Paskov, H., and Talwar, K. A practitioners guide to differentially private convex optimization. 2021.
Mironov, I. Rényi differential privacy. In 2017 IEEE 30th computer security foundations symposium (CSF), pp. 263-275. IEEE, 2017.
Mironov, I., Talwar, K., and Zhang, L. Rényi differential privacy of the sampled Gaussian mechanism. arXiv preprint arXiv:1908.10530, 2019.
Mohapatra, S., Sasy, S., He, X., Kamath, G., and Thakkar, O. The role of adaptive optimizers for honest private hyperparameter selection. In Proceedings of the aaai conference on artificial intelligence, volume 36, pp. 7806-7813, 2022.
Neel, S., Roth, A., Vietri, G., and Wu, S. Oracle efficient private non-convex optimization. In International Conference on Machine Learning, pp. 7243-7252. PMLR, 2020.
Nguyen, L. M., van Dijk, M., Phan, D. T., Nguyen, P. H., Weng, T.-W., and Kalagnanam, J. R. Finite-sum smooth optimization with sarah. Computational Optimization and Applications, 82(3): 561-593, 2022.
Papernot, N. and Steinke, T. Hyperparameter tuning with Rényi differential privacy. In International Conference on Learning Representations, 2022.
Redberg, R. and Wang, Y.-X. Privately publishable per-instance privacy. Advances in Neural Information Processing Systems, 34, 2021.
Schmidt, M., Le Roux, N., and Bach, F. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162:83-112, 2017.
Sommer, D. M., Meiser, S., and Mohammadi, E. Privacy loss classes: The central limit theorem in differential privacy. Proceedings on Privacy Enhancing Technologies, 2019(2):245-269, 2019.
Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 245-248. IEEE, 2013.
Song, S., Steinke, T., Thakkar, O., and Thakurta, A. Evading curse of dimensionality in unconstrained private glms via private gradient descent. arXiv preprint arXiv:2006.06783, 2020.
Wang, D., Ye, M., and Xu, J. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017.
Wang, Y.-X., Balle, B., and Kasiviswanathan, S. P. Subsampled rényi differential privacy and analytical moments accountant. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1226-1235. PMLR, 2019.
Yu, D., Zhang, H., Chen, W., Liu, T.-Y., and Yin, J. Gradient perturbation is underrated for differentially private convex optimization. arXiv preprint arXiv:1911.11363, 2019.
Zhu, Y. and Wang, Y.-X. Poisson subsampled Rényi differential privacy. In International Conference on Machine Learning, pp. 7634-7642, 2019.
Zhu, Y., Dong, J., and Wang, Y.-X. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pp. 4782-4817. PMLR, 2022.
A Notation 14
B Warm-up: Gaussian Mechanism 14
B.1 Privacy profile of the Gaussian Mechanism 14
B.2 RDP Analysis of the Gaussian Mechanism 14
C RDP analysis of objective perturbation 15
C.1 Linearized RDP Bound for Objective Perturbation 17
D Hockey-stick Divergence Analysis of Objective Perturbation 19
D.1 Further Details on Hockey-stick Divergence Analysis 19
D.2 Proof of Theorem 3.1 20
D.3 Dominating Pairs of Distributions for the Objective Perturbation Mechanism 22
D.4 Auxiliary Lemma 23
E RDP guarantee of Algorithm 1 24
E.1 Privacy Guarantee for Approximate Minima Perturbation 24
E.2 The "Clipped-Gradient" Function 25
F Computational Guarantee of Algorithm 1 26
F.1 Tools from convex optimization 27
F.2 Computational bounds for Stopping at small gradient 27
G The GLM Bug 29
G.1 Discussion 29
G.2 RDP bound for non-GLMs 29
H Excess Empirical Risk of Algorithm 1 30
H.1 Generalized Linear Model 31
I Distance to Optimality 32
J Bridging the Gap between Objective Perturbation and DP-SGD 32
J.1 RDP of Objective Perturbation vs DP-SGD 32
J.2 A Spectrum of DP Learning Algorithms 34
K Technical Lemmas & Definitions 34
K.1 Convex Optimization 34
K.2 Differential Privacy 34
Denote the following:
$\mathcal{L}(\theta) = \sum_{i = 1}^{n}\ell_{i}(\theta),$
$\mathcal{L}_{\lambda}(\theta) = \mathcal{L}(\theta) + \frac{\lambda}{2} ||\theta||_2^2,$
$\mathcal{L}^P (\theta) = \mathcal{L}(\theta) + \frac{\lambda}{2} ||\theta ||_2^2 +b^T\theta ,$ $b\sim \mathcal{N}(0,\sigma^2 I_d)$
$\theta^{*} = \operatorname *{arg  min}_{\theta \in \mathbb{R}^{d}}\mathcal{L}(\theta),$
$\theta_{\lambda}^{*} = \operatorname *{arg  min}_{\theta \in \mathbb{R}^{d}}\mathcal{L}_{\lambda}(\theta),$
$\theta^P = \operatorname *{arg  min}_{\theta \in \mathbb{R}^d}\mathcal{L}^P (\theta),$
-  $\tilde{\theta}$  satisfies  $||\nabla \mathcal{L}^P (\tilde{\theta})||_2\leq \tau$
$\tilde{\theta}^P = \tilde{\theta} + b_2, b_2 \sim \mathcal{N}(0, \sigma_{out}^2 I_d)$ .
We take  $||\mathcal{X}||$  to be the size of the data domain  $\mathcal{X}$ , i.e.  $||\mathcal{X}|| = \sup_{x\in \mathcal{X}}||x||$ . For conciseness of presentation we sometimes drop the dataset  $Z$  from the notation, e.g. we abbreviate  $\mathcal{L}(\theta;Z)$  as  $\mathcal{L}(\theta)$ .
Sometimes (especially in the proofs that follow) we will abuse notation by overloading a function with its output, e.g.  $\theta^P$  is the output of objective perturbation and  $\theta^P(Z)$  is the objective perturbation mechanism.
B Warm-up: Gaussian Mechanism
We will get started by reviewing the RDP and privacy profile bounds for the Gaussian mechanism. Once warmed up, we then present the RDP and privacy profile bounds for objective perturbation in Sections C and D.
Consider the Gaussian mechanism defined by  $\mathcal{M}(Z) = f(Z) + \mathcal{N}(0, \sigma^2 I_d)$ , for a function  $f: \mathcal{Z} \to \mathbb{R}^d$  with global sensitivity  $\Delta_f = \max_{Z \simeq Z'} ||f(Z) - f(Z')||_2$ .
B.1 Privacy profile of the Gaussian Mechanism
Analytic Gaussian mechanism (Balle & Wang, 2018).
Let  $P$  and  $Q$  be the density functions of  $\mathcal{N}(\Delta_f, \sigma^2)$  and  $\mathcal{N}(0, \sigma^2)$ , respectively. Then  $(P, Q)$  is a dominating pair of distributions for  $\mathcal{M}$ , and  $\mathcal{M}$  is tightly  $(\epsilon, \delta(\epsilon))$ -DP for
$$
\delta (\epsilon) = H _ {e ^ {\epsilon}} (P | | Q) = \Phi \left(- \frac {\epsilon \sigma}{\Delta_ {f}} + \frac {\Delta_ {f}}{2 \sigma}\right) - e ^ {\epsilon} \Phi \left(- \frac {\epsilon \sigma}{\Delta_ {f}} - \frac {\Delta_ {f}}{2 \sigma}\right), \tag {B.1}
$$
where  $\Phi$  denotes the CDF of the standard univariate Gaussian distribution.
We can analytically express a tight upper bound for the Gaussian mechanism above, but in general numerical methods are needed to evaluate the hockey-stick divergence for dominating pairs of distributions. This is discussed with more detail in Section D.
B.2 RDP Analysis of the Gaussian Mechanism
Theorem B.1 (RDP guarantees of the Gaussian mechanism). The Gaussian mechanism  $\mathcal{M}$  satisfies  $(\alpha, \epsilon)$ -RDP for  $\alpha > 1$  and  $\epsilon = \frac{\Delta_f^2 \alpha}{2 \sigma^2}$ .
C RDP analysis of objective perturbation
In this section we present the proof of Theorem 3.2, one of our main privacy results: an RDP bound on the objective perturbation mechanism.
Proof of Theorem 3.2. Recall from Definition 2.7 that the objective perturbation mechanism  $\hat{\theta}^P: \mathcal{Z}^* \to \mathbb{R}^d$  satisfies  $\epsilon(\alpha)$ -Rényi differential privacy if for all neighboring datasets  $Z$  and  $Z'$ ,
$$
\mathcal {D} _ {\alpha} \left(\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \epsilon (\alpha).
$$
Assume that  $Z \in \mathcal{Z}^n$  and construct  $Z' \in \mathcal{Z}^{n+}$  by adding a datapoint  $z$  to  $Z$ . Note that this convention (while convenient for writing down the PLRV of objective perturbation) comes with loss of generality. As a consequence of asymmetry<sup>3</sup>, the upper bound on the RDP must satisfy
$$
\left. \right. \max  \left(D _ {\alpha} \left(\hat {\theta} ^ {P} (Z) \mid \mid \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right), D _ {\alpha} \left(\hat {\theta} ^ {P} \left(Z ^ {\prime}\right) \mid \mid \hat {\theta} ^ {P} (Z)\right)\right) \leq \epsilon (\alpha).
$$
We will calculate the Rényi divergence  $D_{\alpha}\big(\hat{\theta}^{P}(Z) \mid \mid \hat{\theta}^{P}(Z^{\prime})\big)$  of objective perturbation under a change of measure:
$$
D _ {\alpha} (P | | Q) = \frac {1}{\alpha - 1} \log \mathbb {E} _ {x \sim Q} \left[ \left(\frac {P (x)}{Q (x)}\right) ^ {\alpha} \right] = \frac {1}{\alpha - 1} \log \mathbb {E} _ {x \sim P} \left[ \left(\frac {P (x)}{Q (x)}\right) ^ {\alpha - 1} \right].
$$