SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $R(\theta^P) \coloneqq \frac{\operatorname*{Pr}\left[\hat{\theta}^P(Z) = \theta^P\right]}{\operatorname*{Pr}\left[\hat{\theta}^P(Z') = \theta^P\right]}$  be shorthand for the probability density ratio at output  $\theta^P$ . Then
$$
\begin{array}{l} D _ {\alpha} \left(\hat {\theta} ^ {P} (Z) \mid \mid \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) = \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ R \left(\theta^ {P}\right) ^ {(\alpha - 1)} \right] \\ = \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {\log \left[ R \left(\theta^ {P}\right) ^ {(\alpha - 1)} \right]} \right] \\ = \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {(\alpha - 1) \log R (\theta^ {P})} \right] \\ \leq \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \tilde {\theta} ^ {P} (Z)} \left[ e ^ {(\alpha - 1) | \log R (\theta^ {P}) |} \right]. \\ \end{array}
$$
Denote  $J(\theta; Z) = \sum_{z \in Z} \ell(\theta; z) + \frac{\lambda}{2} ||\theta||_2^2$  and  $\mu(\theta, Z, z) = x^T \left( \nabla^2 J(\theta; Z) \right)^{-1} x$ .
From Lemma K.9 and the  $\lambda$ -strong convexity of  $J(\theta; Z)$ , we can bound  $\mu(\theta, Z, z)$  by  $\frac{||x||_2^2}{\lambda} \leq \frac{1}{\lambda}$ .
We also know from the  $L$ -Lipschitzness of  $\ell(\theta;z)$  and the  $\beta$ -smoothness of  $f(x^T\theta;y)$  that  $||\nabla \ell(\theta;z)||_2 \leq L$  and  $f''(x^T\theta;y) \leq \beta$  for all  $\theta \in \mathbb{R}^d$  and  $z = (x,y) \in \mathcal{Z}$ .
Then using the GLM assumption, from Redberg & Wang (2021) we can bound the absolute value of the log-probability ratio for any  $\theta^P$  as
$$
\begin{array}{l} \left| \log \frac {\Pr \left[ \hat {\theta} ^ {P} (Z) = \theta^ {P} \right]}{\Pr \left[ \hat {\theta} ^ {P} \left(Z ^ {\prime}\right) = \theta^ {P} \right]} \right| \leq \left| - \log \left(1 - f ^ {\prime \prime} \left(x ^ {T} \theta ; y\right) \mu \left(\theta^ {P}, Z, z\right)\right) - \frac {1}{2 \sigma^ {2}} | | \nabla \ell \left(\theta^ {P}; z\right) | | _ {2} ^ {2} - \frac {1}{\sigma^ {2}} \nabla J \left(\theta^ {P}; Z\right) ^ {T} \nabla \ell (\theta ; z) \right| \\ \leq \left| - \log \left(1 - f ^ {\prime \prime} \left(x ^ {T} \theta^ {P}; y\right) \mu \left(\theta^ {P}, Z, z\right)\right) \right| + \frac {1}{2 \sigma^ {2}} | | \nabla \ell \left(\theta^ {P}; z\right) | | _ {2} ^ {2} + \frac {1}{\sigma^ {2}} \left| \nabla J \left(\theta^ {P}; Z\right) ^ {T} \nabla \ell \left(\theta^ {P}; z\right) \right| \\ \leq \left| - \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\sigma^ {2}} \left| \nabla J (\theta^ {P}; Z) ^ {T} \nabla \ell (\theta^ {P}; z) \right|. \\ \end{array}
$$
It is more challenging to find a data-independent bound for the third term due to the shared dependence on  $\theta^P$ .
3This is in contrast to the symmetry of the Gaussian mechanism  $\mathcal{M}(Z) = f(Z) + \mathcal{N}(0,\sigma^2 I_d)$ , in which case  $\epsilon (\alpha)$  can be calculated exactly as  $D_{\alpha}\big(\mathcal{N}(0,\sigma^{2}I_{d})||\mathcal{N}(\Delta_{f},\sigma^{2}I_{d})\big) = \frac{\alpha\Delta_{f}^{2}}{2\sigma^{2}} = D_{\alpha}\big(\mathcal{N}(\Delta_{f},\sigma^{2}I_{d})||\mathcal{N}(0,\sigma^{2}I_{d})\big)$ .
Recall that  $b \sim \mathcal{N}(0, \sigma^2 I_d)$  is the noise vector in the perturbed objective. By first-order conditions at the minimizer  $\theta^P$ ,
$$
b = - \nabla J (\theta^ {P}; Z).
$$
If  $\theta^P$  were fixed (or if  $\theta^P$  were independent to  $b$ ), the quantity  $\nabla J(\theta^P; Z)^T \nabla \ell(\theta^P; z) = -b^T \nabla \ell(\theta^P; z)$  would have been distributed as a univariate Gaussian  $\mathcal{N}(0, \sigma^2 || \nabla \ell(\theta^P; z)||_2^2)$ . Unfortunately in our case  $\theta^P$  is a random variable, and consequently we don't have the tools to understand the distribution of  $\nabla J(\theta^P; Z)^T \nabla \ell(\theta^P; z)$  for an arbitrary loss function.
But using the GLM assumption on the loss function, we can write
$$
\nabla J (\theta ; Z) ^ {T} \nabla \ell (\theta ; z) = - b ^ {T} x f ^ {\prime} (x ^ {T} \theta , y).
$$
Observe that  $x$  is fixed w.r.t.  $b$  so that  $-b^{T}x\sim \mathcal{N}(0,\sigma^{2}\| x\|_{2}^{2})$ , and  $f^{\prime}(x^{T}\theta ,y)$  is a scalar. So while this scalar is a random variable that still depends on  $b$  in a complicated way, the worst possible dependence can be more easily quantified without incurring additional dimension dependence.
By the  $L$ -Lipschitz assumption and  $|ab| \leq |a||b|$ , we obtain the following bound:
$$
\begin{array}{l} \left| \nabla J \left(\theta^ {P}; Z\right) ^ {T} \nabla \ell \left(\theta^ {P}; z\right) \right| = \left| f ^ {\prime} \left(x ^ {T} \theta^ {P}; y\right) \nabla J \left(\theta^ {P}; Z\right) ^ {T} x \right| \\ \leq L \left| \nabla J \left(\theta^ {P}; Z\right) ^ {T} x \right|. \\ \end{array}
$$
This is much better! By first-order conditions we can then see
$$
L \left| \nabla J \left(\theta^ {P}; Z\right) ^ {T} x \right| = L \left| b ^ {T} x \right| = \left| \mathcal {N} \left(0, \sigma^ {2} \| x \| ^ {2} L ^ {2}\right) \right| \sim \text {H a l f - N o r m a l} (\sigma L \| x \|).
$$
Now we can bound
$$
\begin{array}{l} \left| \log R \left(\theta^ {P}\right) \right| \leq \left| - \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\sigma^ {2}} \left| \nabla J \left(\theta^ {P}; Z\right) ^ {T} \nabla \ell \left(\theta^ {P}; z\right) \right| \\ \leq \left| - \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {L}{\sigma^ {2}} \left| \nabla J (\theta^ {P}; Z) ^ {T} x \right|. \\ \end{array}
$$
Plugging in this bound on  $\left|\log R(\theta^P)\right|$ ,
$$
\begin{array}{l} D _ {\alpha} \left(\hat {\theta} ^ {P} (Z) \mid \mid \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {\left(\alpha - 1\right) \left| \log R \left(\theta^ {P}\right) \right|} \right] \\ \leq \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {(\alpha - 1) \left[ \left| - \log (1 - \frac {\beta}{\lambda}) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} \right]} e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} \left| \nabla J (\theta^ {P}; Z) ^ {T} x \right|} \right] \\ = \left| - \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} | \nabla J (\theta^ {P}; Z) ^ {T} x |} \right]. \\ \end{array}
$$
Let  $p_{\sigma}$  be the probability density function of  $b \sim \mathcal{N}(0, \sigma^2 I_d)$ , and  $p_{\Theta}$  the probability density function of  $\theta^P \sim \hat{\theta}^P(Z)$ . We know from Lemmas K.6 and K.7 that  $\partial \theta = \left| \det \frac{\partial \theta}{\partial b} \right| \partial b$ , and  $p_{\Theta}(\theta) = \left| \det \frac{\partial b}{\partial \theta} \right| p_{\sigma}(b)$ .
We also know from Lemma K.8 that  $\left|\operatorname{det}\frac{\partial\theta}{\partial b}\right| \cdot \left|\operatorname{det}\frac{\partial b}{\partial\theta}\right| = 1$ .
Using the change of variables  $b = -\nabla J(\theta^P; Z)$  and  $b^T x = u \sim \mathcal{N}(0, \sigma^2 ||x||_2^2)$ , we have
$$
\begin{array}{l} \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z)} \left[ e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} | \nabla J (\theta^ {P}; Z) ^ {T} x |} \right] = \int_ {\mathbb {R} ^ {d}} e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} | \nabla J (\theta^ {P}; Z) ^ {T} x |} p _ {\Theta} (\theta^ {P}) \partial \theta \\ = \int_ {\mathbb {R} ^ {d}} e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} | b ^ {T} x |} \left| \det  \frac {\partial b}{\partial \theta} \right| p _ {\sigma} (b) \left| \det  \frac {\partial \theta}{\partial b} \right| \partial b \\ = \int_ {\mathbb {R} ^ {d}} e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} \left| b ^ {T} x \right|} p _ {\sigma} (b) \partial b \\ = \mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(\alpha - 1) \cdot \frac {L}{\sigma^ {2}} \left| b ^ {T} x \right|} \right] \\ = \mathbb {E} _ {u \sim \mathcal {N} (0, \sigma^ {2} | | x | | _ {2} ^ {2}} \left[ e ^ {(\alpha - 1) \left| \frac {L}{\sigma^ {2}} u ^ {2} \right|} \right] \\ \leq \mathbb {E} _ {\zeta \sim \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {(\alpha - 1) | \zeta |} \right]. \\ \end{array}
$$
In the last line, we applied our assumption that  $\| x\| \leq 1$  and the fact that the MGF of a half-normal R.V. increases monotonically when its scale parameter gets larger.
The above bound holds for the reverse Rényi divergence  $D_{\alpha}(\hat{\theta}^{P}(Z^{\prime})||\hat{\theta}^{P}(Z))$ . Observe that
$$
D _ {\alpha} (\hat {\theta} ^ {P} (Z ^ {\prime}) | | \hat {\theta} ^ {P} (Z)) \leq \frac {1}{\alpha - 1} \log \mathbb {E} _ {\theta^ {P} \sim \hat {\theta} ^ {P} (Z ^ {\prime})} \left[ e ^ {(\alpha - 1) | \log R (\theta^ {P}) |} \right].
$$
This is because  $\log \frac{\operatorname*{Pr}\left[\hat{\theta}^P(Z') = \theta^P\right]}{\operatorname*{Pr}\left[\hat{\theta}^P(Z) = \theta^P\right]} = -\log R(\theta^P)\leq |\log R(\theta^P)|$  . If we use the change of variables  $b = -\nabla J(\theta^{P};Z^{\prime})$  for the reverse direction, the above calculation works out identically (the difference is that  $p_{\Theta}$  and the bijection between  $b$  and  $\theta^P$  are different under  $Z$  and  $Z^{\prime}$  — but the determinant of the mapping cancels out with its inverse just the same).
We've shown  $\max \left(D_{\alpha}\Big(\hat{\theta}^{P}(Z)\parallel \hat{\theta}^{P}(Z^{\prime})\Big),D_{\alpha}\Big(\hat{\theta}^{P}(Z^{\prime})\parallel \hat{\theta}^{P}(Z)\Big)\right)\leq \epsilon (\alpha)$  for any neighboring datasets  $Z$  and  $Z^{\prime}$ , where
$$
\epsilon (\alpha) = - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L}{2 \sigma^ {2}} + \mathbb {E} \left[ e ^ {(\alpha - 1) \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|} \right].
$$
C.1 Linearized RDP Bound for Objective Perturbation
In our calculation of the RDP for objective perturbation, we needed to take an absolute value of the privacy loss random variable in order to handle negative values. But in doing so we end up with a quantity that depends on the moments of the half-normal distribution rather than those of the normal distribution, which gives us a looser bound. Can we avoid having to make this compromise? In this section we demonstrate that a linearization of the first-order conditions on the perturbed and unperturbed objective functions provides a more precise analysis of the PLRV of objective perturbation, translating to a tighter RDP bound in some regimes.
Recall that the objective perturbation mechanism is given by
$$
\hat {\theta} ^ {P} (Z) = \sum_ {i = 1} ^ {n} \ell (\theta ; z _ {i}) + \frac {\lambda}{2} | | \theta | | _ {2} ^ {2} + b ^ {T} \theta , \tag {C.1}
$$
where  $b\sim \mathcal{N}(0,\sigma^2 I_d)$
From the non-linearized RDP calculation, we know that for any neighboring datasets  $Z$  and  $Z'$ ,
$$
D _ {\alpha} \left(\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \left| - \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {b \sim \mathcal {N} \left(0, \sigma^ {2} I _ {d}\right)} \left[ e ^ {\left(\alpha - 1\right) b ^ {T} \nabla \ell \left(\theta^ {P}\right)} \right],
$$
where  $\theta^P$  is the output of the objective perturbation mechanism given the noise vector  $b$ . We can write
$$
b ^ {T} \nabla \ell (\theta^ {P}) = b ^ {T} \nabla \ell (\theta_ {\lambda} ^ {*}) + b ^ {T} \left[ \nabla \ell (\theta^ {P}) - \nabla \ell (\theta_ {\lambda} ^ {*}) \right].
$$
The first term  $t_1 = b^T \nabla \ell(\theta_{\lambda}^*)$  is a univariate Gaussian  $t_1 \sim \mathcal{N}(0, \sigma^2 || \nabla \ell(\theta_{\lambda}^*)||_2^2)$  because  $\theta_{\lambda}^*$  is fixed w.r.t.  $b$ . We can bound the second term  $t_2 = b^T [\nabla \ell(\theta^P) - \nabla \ell(\theta_{\lambda}^*)]$  using our assumptions on the loss function  $\ell(\theta)$ .
By assumption, the loss function has GLM structure  $\ell (\theta ;z) = f(x^T\theta ;y)$ . We can therefore write
$$
\begin{array}{l} b ^ {T} \left[ \nabla \ell \left(\theta^ {P}\right) - \nabla \ell \left(\theta_ {\lambda} ^ {*}\right) \right] = b ^ {T} \left[ f ^ {\prime} \left(x ^ {T} \theta^ {P}; y\right) x - f ^ {\prime} \left(x ^ {T} \theta_ {\lambda} ^ {*}; y\right) x \right] \\ = b ^ {T} x \left(f ^ {\prime} \left(x ^ {T} \theta^ {P}; y\right) - f ^ {\prime} \left(x ^ {T} \theta_ {\lambda} ^ {*}; y\right)\right) \\ \end{array}
$$
We have furthermore assumed that the function  $f$  is  $\beta$ -smooth, so that for any  $z = (x,y)$  and  $\theta^P, \theta_\lambda^* \in \mathbb{R}^d$  we have
$$
\begin{array}{l} \left| f ^ {\prime} \left(x ^ {T} \theta^ {P}; y\right) - f ^ {\prime} \left(x ^ {T} \theta_ {\lambda} ^ {*}; y\right) \right| \leq \beta \left| x ^ {T} \theta^ {P} - x ^ {T} \theta_ {\lambda} ^ {*} \right| \\ = \beta \left| x ^ {T} \left[ \theta^ {P} - \theta_ {\lambda} ^ {*} \right] \right|. \\ \end{array}
$$
We will next apply Taylor's Theorem to rewrite  $\theta^P -\theta_\lambda^*$
Recall that  $\theta^P$  is the minimizer of the perturbed objective:
$$
\theta^ {P} = \arg \min  \left(\sum_ {i = 1} ^ {n} \ell \left(\theta ; z _ {i}\right) + \frac {\lambda}{2} \left\| \theta \right\| _ {2} ^ {2} + b ^ {T} \theta\right); \tag {C.2}
$$
and  $\theta_{\lambda}^{*}$  is the minimizer of the (non-private) regularized objective:
$$
\theta_ {\lambda} ^ {*} = \arg \min  \left(\sum_ {i = 1} ^ {n} \ell \left(\theta ; z _ {i}\right) + \frac {\lambda}{2} \| \theta \| _ {2} ^ {2}\right). \tag {C.3}
$$
Parameterize the line segment between  $\theta^P$  and  $\theta_{\lambda}^{*}$  by  $t \in [0,1]$ , i.e. the line segment is  $t(\theta^P - \theta_{\lambda}^*) + \theta^P$ . By Taylor's Theorem, there exists  $\theta' = t'(\theta^P - \theta_{\lambda}^*) + \theta^P$  for some  $t' \in [0,1]$  such that
$$
\nabla \ell (\theta^ {P}) - \nabla \ell (\theta_ {\lambda} ^ {*}) = \nabla^ {2} \ell (\theta^ {\prime}) (\theta^ {P} - \theta_ {\lambda} ^ {*}).
$$
By first-order conditions on Equations C.2 and C.3,
$$
\nabla \mathcal {L} \left(\theta^ {P}\right) + \lambda \theta^ {P} + b = 0; \tag {C.4}
$$
$$
\nabla \mathcal {L} \left(\theta_ {\lambda} ^ {*}\right) + \lambda \theta_ {\lambda} ^ {*} = 0. \tag {C.5}
$$
Then subtracting Equation C.5 from Equation C.4, we have that
$$
\nabla \mathcal {L} \left(\theta^ {P}\right) - \nabla \mathcal {L} \left(\theta_ {\lambda} ^ {*}\right) + \lambda \left(\theta^ {P} - \theta_ {\lambda} ^ {*}\right) + b = 0. \tag {C.6}
$$
Again applying Taylor's theorem, there exists  $\theta'' = t''(\theta^P - \theta_\lambda^*) + \theta^P$  for some  $t'' \in [0,1]$  such that
$$
\nabla \mathcal {L} \left(\theta^ {P}\right) - \nabla \mathcal {L} \left(\theta_ {\lambda} ^ {*}\right) = \nabla^ {2} \mathcal {L} \left(\theta^ {\prime \prime}\right) \left(\theta^ {P} - \theta_ {\lambda} ^ {*}\right). \tag {C.7}
$$
Putting together Equations C.6 and C.7 we then have
$$
\theta^ {P} - \theta_ {\lambda} ^ {*} = - \left(\nabla^ {2} \mathcal {L} \left(\theta^ {\prime \prime}\right) + \lambda I _ {d}\right) ^ {- 1} b. \tag {C.8}
$$
So we now have
$$
\begin{array}{l} b ^ {T} \left[ \nabla \ell \left(\theta^ {P}\right) - \nabla \ell \left(\theta_ {\lambda} ^ {*}\right) \right] \leq \beta \left| b ^ {T} x \right| \left| x ^ {T} \left(\theta^ {P} - \theta_ {\lambda} ^ {*}\right) \right| \\ \leq \beta \left| b ^ {T} x \right| \left| x ^ {T} \left(\nabla^ {2} \mathcal {L} \left(\theta_ {\lambda} ^ {*}\right) + \lambda I _ {d}\right) ^ {- 1} b \right|. \tag {C.9} \\ \end{array}
$$
Note that since  $e^x > 0$  for all  $x \in \mathbb{R}$ , we have that  $\mathbb{E}[|e^x|] = \mathbb{E}[e^x]$ .
Let  $a \coloneqq \frac{1}{\sigma^2} b^T \nabla \ell(\theta^*)$  and  $c \coloneqq \frac{1}{\sigma^2} b^T \left[ \nabla \ell(\theta^P) - \nabla \ell(\theta^*) \right]$ . Then by Holder's inequality,
$$
\begin{array}{l} \mathbb {E} \left[ e ^ {(\alpha - 1) a} e ^ {(\alpha - 1) c} \right] = \mathbb {E} \left[ \left| e ^ {(\alpha - 1) a} e ^ {(\alpha - 1) c} \right| \right] \\ \leq \mathbb {E} \left[ \left| e ^ {(\alpha - 1) a} \right| ^ {p} \right] ^ {\frac {1}{p}} \mathbb {E} \left[ \left| e ^ {(\alpha - 1) c} \right| ^ {q} \right] ^ {\frac {1}{q}} \\ = \mathbb {E} \left[ e ^ {(p \alpha - p) a} \right] ^ {\frac {1}{p}} \mathbb {E} \left[ e ^ {(q \alpha - q) c} \right] ^ {\frac {1}{q}}. \\ \end{array}
$$
By the GLM assumption,  $b^{T}\nabla \ell (\theta_{\lambda}^{*};z) = f^{\prime}(x^{T}\theta_{\lambda}^{*};y)b^{T}x$  . Then
$$
\begin{array}{l} \mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(p \alpha - p) \frac {1}{\sigma^ {2}} b ^ {T} \nabla \ell (\theta^ {P})} \right] = \mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(p \alpha - p) \frac {1}{\sigma^ {2}} f ^ {\prime} (x ^ {T} \theta_ {\lambda} ^ {*}; y) b ^ {T} x} \right] \\ = \mathbb {E} _ {u _ {1} \sim \mathcal {N} \left(0, f ^ {\prime} \left(x ^ {T} \theta_ {\lambda} ^ {*} ; y\right) ^ {2} | | x | | _ {2} ^ {2} \frac {1}{\sigma^ {2}}\right)} \left[ e ^ {(p \alpha - p) u _ {1}} \right] \\ \leq \mathbb {E} _ {u _ {2} \sim \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {(p \alpha - p) u _ {2}} \right]. \\ \end{array}
$$
Above, we've applied the assumption that  $||x||_2 \leq 1$  and the fact that the MGF of a normal R.V. increases monotonically when its scale parameter gets larger (Lemma K.11). By Lemma K.9, we have that  $x^T\left(\nabla^2\mathcal{L}(\theta_\lambda^*) + I_d\right)^{-2}x \leq \frac{||x||_2^2}{\lambda^2}$ . From C.9 we also have
$$
\mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(q \alpha - q) \frac {1}{\sigma^ {2}} b ^ {T} [ \nabla \ell (\theta^ {P}) - \nabla \ell (\theta_ {\lambda} ^ {*}) ]} \right] \leq \mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(q \alpha - q) \beta | b ^ {T} x | | x ^ {T} (\nabla^ {2} \mathcal {L} (\theta^ {*}) + \lambda I _ {d}) ^ {- 1} b |} \right]
$$
Define  $z_{1} \coloneqq b^{T}x$  and  $z_{2} \coloneqq x^{T}\left(\nabla^{2}\mathcal{L}(\theta_{\lambda}^{*}) + \lambda I_{d}\right)^{-1}b$ , and observe
$$
z _ {1} \sim \mathcal {N} \left(0, \sigma^ {2} | | x | | _ {2} ^ {2}\right),
$$
$$
z _ {2} \sim \mathcal {N} \left(0, \sigma^ {2} x ^ {T} \left(\nabla^ {2} \mathcal {L} \left(\theta_ {\lambda} ^ {*}\right) + I _ {d}\right) ^ {- 2} x\right).
$$
Note that our approach below is agnostic to the relationship between  $|z_1|$  and  $|z_2|$ ; in reality, they depend on each other through the noise vector  $b$ . Again applying the assumption  $||x||_2^2 \leq 1$  and Lemma K.11 (while not forgetting that the random variables  $z_1$  and  $z_2$  depend on each other through  $b$ ), we get
$$
\begin{array}{l} \mathbb {E} _ {b \sim \mathcal {N} (0, \sigma^ {2} I _ {d})} \left[ e ^ {(q \alpha - q) \beta \left| b ^ {T} x \right| \left| x ^ {T} (\nabla^ {2} \mathcal {L} (\theta_ {\lambda} ^ {*}) + \lambda I _ {d}) ^ {- 1} b \right|} \right] = \mathbb {E} _ {z _ {1}, z _ {2}} \left[ e ^ {(q \alpha - q) \beta | z _ {1} | | z _ {2} |} \right] \\ \leq \mathbb {E} _ {z _ {3} \sim \mathcal {N} (0, \sigma^ {2}), z _ {4} \sim \mathcal {N} (0, \frac {\sigma^ {2}}{\lambda^ {2}})} \left[ e ^ {(q \alpha - q) \beta | z _ {3} | | z _ {4} |} \right] \\ = \mathbb {E} _ {z _ {3} \sim \mathcal {N} (0, \sigma^ {2}), z _ {5} \sim \mathcal {N} (0, \sigma^ {2})} \left[ e ^ {(q \alpha - q) \frac {\beta}{\lambda} | z _ {3} | | z _ {5} |} \right] \\ \leq \mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} \left[ e ^ {(q \alpha - q) \frac {\beta}{\lambda} z ^ {2}} \right]. \\ \end{array}
$$
So altogether, for  $p, q$  such that  $\frac{1}{p} + \frac{1}{q} = 1$ , we get
$$
\begin{array}{l} D _ {\alpha} \left(\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \\ - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {u \sim \mathcal {N} (0, \frac {L ^ {2}}{\sigma^ {2}})} \left[ e ^ {(p \alpha - p) u} \right] ^ {\frac {1}{p}} \mathbb {E} _ {z \sim \mathcal {N} (0, \sigma^ {2})} \left[ e ^ {(q \alpha - q) \frac {\beta}{\lambda} z ^ {2}} \right] ^ {\frac {1}{q}}. \\ \end{array}
$$
D Hockey-stick Divergence Analysis of Objective Perturbation
D.1 Further Details on Hockey-stick Divergence Analysis
Using dominating pairs of distributions (Def. 2.5) for all the individual mechanisms in an adaptive composition, we can obtain accurate  $(\epsilon, \delta)$ -bounds for the whole composition. For this end we need the following result.
Theorem D.1 (Zhu et al. 2022). If  $(P, Q)$  dominates  $\mathcal{M}$  and  $(P', Q')$  dominates  $\mathcal{M}'$  for all inputs of  $\mathcal{M}'$ , then  $(P \times P', Q \times Q')$  dominates the adaptive composition  $\mathcal{M} \circ \mathcal{M}'$ .
To get the hockey-stick divergence from  $P \times P'$  to  $Q \times Q'$  into an efficiently computable form, we express it using so-called privacy loss random variables (recall Def. K.4). If  $P$  and  $Q$  are probability density functions, the privacy loss function  $\mathcal{L}_{P/Q}$  is defined as
$$
\mathcal {L} _ {P / Q} (x) = \log \frac {P (x)}{Q (x)}
$$
and the privacy loss random variable (PLRV)  $\omega_{P / Q}$  as
$$
\omega_ {P / Q} = \mathcal {L} _ {P / Q} (x), \quad x \sim P (x).
$$
The  $\delta (\epsilon)$ -bounds can be represented using the following representation that involves the PLRV.
Theorem D.2 (Gopi et al. 2021). We have:
$$
H _ {\mathrm {e} ^ {\epsilon}} (P \| Q) = \underset {x \sim P} {\mathbb {E}} \left[ 1 - \mathrm {e} ^ {\epsilon - \mathcal {L} _ {P / Q} (x)} \right] _ {+} = \underset {s \sim \omega_ {P / Q}} {\mathbb {E}} \left[ 1 - \mathrm {e} ^ {\epsilon - s} \right] _ {+}. \tag {D.1}
$$
Moreover, if  $\omega_{P / Q}$  is the PLRV for the pair of distributions  $(P,Q)$  and  $\omega_{P^{\prime} / Q^{\prime}}$  the PLRV for the pair of distributions  $(P^{\prime},Q^{\prime})$ , then the PLRV for the pair of distributions  $(P\times P^{\prime},Q\times Q^{\prime})$  is given by  $\omega_{P / Q} + \omega_{P^{\prime} / Q^{\prime}}$ .
By Theorem D.2, to computing accurate  $(\epsilon, \delta)$ -bounds for compositions, it suffices that we can evaluate integrals of the form  $\mathbb{E}_{s \sim \omega_1 + \ldots + \omega_k} [1 - \mathrm{e}^{\epsilon - s}]_+$ . For this we can use the Fast Fourier Transform (FFT)-based method by Koskela et al. (2021), where the distribution of each PLRV is truncated and placed on an equidistant numerical grid over an interval  $[-L, L]$ , where  $L > 0$  is a pre-defined parameter. The distributions for the sums of the PLRVs are given by convolutions of the individual distributions and can be evaluated using the FFT algorithm. By a careful error analysis the error incurred by the numerical method can be bounded and an upper  $\delta(\epsilon)$ -bound obtained. For accurately carrying out this numerical computation one could also use, for example, the FFT-based method proposed by Gopi et al. (2021).
Before giving a proof to Thm. 3.1, we first give the following bound which is a hockey-stick equivalent of the moment-generating function bound given in Thm. 3.2.
Lemma D.3. Let  $\epsilon \in \mathbb{R}$  and let the objective perturbation mechanism  $\hat{\theta}^P$  be defined as in Section 2.2. Let  $||\nabla \ell (\theta ;z)||_2\leq L$  and  $\nabla^2\ell (\theta ;z)\prec \beta I_d$  for all  $\theta \in \Theta$  and  $z\in \mathcal{X}\times \mathcal{Y}$ . Then, for any neighboring datasets  $Z$  and  $Z^{\prime}$ , we have:
$$
H _ {e ^ {\epsilon}} \left(\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \mathbf {E} _ {s \sim \omega} [ 1 - e ^ {\epsilon - s} ] _ {+}, \tag {D.2}
$$
where  $\omega \sim \left|\log \left(1 - \frac{\beta}{\lambda}\right)\right| + \frac{L^2}{2\sigma^2} +\left|\mathcal{N}\left(\frac{\|x\|^2L^2}{\sigma^2}\right)\right|$ .