SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. The proof goes analogously to the proof of Thm. 3.2. Let  $Z$  and  $Z'$  be any neighboring datasets. Following the proof of Thm. 3.2, denote the privacy loss
$$
R (\theta) := \frac {\Pr \left[ \hat {\theta} ^ {P} (Z) = \theta \right]}{\Pr \left[ \hat {\theta} ^ {P} \left(Z ^ {\prime}\right) = \theta \right]}.
$$
By Thm. D.2 and by using the reasoning of the proof of Thm. 3.2 for the moment-generating function, we have
$$
\begin{array}{l} H _ {e ^ {\epsilon}} \big (\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} (Z ^ {\prime}) \big) = \mathbf {E} _ {\theta \sim \hat {\theta} ^ {P} (Z)} [ 1 - e ^ {\epsilon - \log R (\theta)} ] _ {+} \\ \leq \mathbf {E} _ {\theta \sim \hat {\theta} ^ {P} (Z)} [ 1 - e ^ {\epsilon - | \log R (\theta) |} ] + \tag {D.3} \\ \leq \mathbf {E} _ {s \sim \left| \mathcal {N} (\frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}) \right|} [ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} ] _ {+}, \\ \end{array}
$$
where the inequalities follow from the fact that the function  $f(s) = [1 - e^{\epsilon - s}]_+$  is monotonically increasing function w.r.t.  $s$  for all  $\epsilon \in \mathbb{R}$  and from the bound for  $|R(\theta)|$  used in the proof of Thm. 3.2.
Proof of Theorem 3.1. We use Lemma D.3 and simply upper bound the right-hand side of the inequality (D.2).
We first show that if  $\| x\| \leq 1$ , then for all  $\epsilon \in \mathbb{R}$
$$
\mathbf {E} _ {s \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} \left| \mathcal {N} \left(0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}\right) \right|} [ 1 - e ^ {\epsilon - s} ] _ {+} \leq \mathbf {E} _ {s \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|} [ 1 - e ^ {\epsilon - s} ] _ {+}. \tag {D.4}
$$
Denote  $\hat{\epsilon} = \epsilon -\left|\log \left(1 - \frac{\beta}{\lambda}\right)\right| - \frac{L^2}{2\sigma^2}$ . Consider first the case  $\hat{\epsilon}\geq 0$ . Then, we have:
$$
\begin{array}{l} \mathbf {E} _ {s \sim \left| \mathcal {N} (0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}) \right|} [ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} ] _ {+} = 2 \cdot \mathbf {E} _ {s \sim \mathcal {N} (0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\hat {\epsilon} - s} ] _ {+} \\ \leq 2 \cdot \mathbf {E} _ {s \sim \mathcal {N} (0, \frac {L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\hat {\epsilon} - s} ] + \\ = \mathbf {E} _ {s \sim \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|} \left[ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} \right] +, \tag {D.5} \\ \end{array}
$$
where the first equality follows from the fact that for  $s \geq \hat{\epsilon}$ , the density function of the half-normal random variable is positive and 2 times the density of the corresponding normal distribution. The inequality follows from Lemma D.8, as
$$
\mathbf {E} _ {s \sim \mathcal {N} (0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\hat {\epsilon} - s} ] _ {+} = \int_ {\hat {\epsilon}} ^ {\infty} f _ {0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}} (x) \left(1 - \mathrm {e} ^ {\hat {\epsilon} - x}\right)   \mathrm {d} x.
$$
Next, consider the case  $\hat{\epsilon} < 0$ . Then:
$$
\begin{array}{l} \mathbf {E} _ {s \sim \left| \mathcal {N} (0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}) \right|} \left[ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} \right] _ {+} = 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {\| x \| ^ {2} L ^ {2}}{\sigma^ {2}}} (x) (1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - x}) \mathrm {d} x \\ \leq 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) \left(1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - x}\right) d x \\ = \mathbf {E} _ {s \sim \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|} \left[ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} \right] +. \tag {D.6} \\ \end{array}
$$
where the inequality follows from Lemma D.8. Inequalities (D.5) and (D.6) together give (D.4).
Then, we show that for all  $\epsilon \in \mathbb{R}$
$$
\mathbf{E}_{s\sim \left|\log \left(1 - \frac{\beta}{\lambda}\right)\right| + \frac{L^{2}}{2\sigma^{2}}\left|\mathcal{N}(0,\frac{L^{2}}{\sigma^{2}})\right|} [1 - e^{\epsilon -s}]_{+} = \left\{ \begin{array}{ll}2\cdot H_{e^{\hat{\epsilon}}}\big(P||Q\big), & \text{if}\hat{\epsilon}\geq 0,\\ (1 - \mathrm{e}^{\hat{\epsilon}}) + \mathrm{e}^{\hat{\epsilon}}\cdot 2\cdot H_{\frac{L^{2}}{e^{\sigma^{2}}}}\big(P||Q\big), & \text{otherwise}. \end{array} \right.
$$
Continuing from (D.5), by change of variables, we see that for  $\hat{\epsilon} \geq 0$
$$
\begin{array}{l} 2 \cdot {\bf E} _ {s \sim \mathcal {N} (0, \frac {L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - s} ] _ {+} = 2 \cdot {\bf E} _ {s \sim \mathcal {N} (\frac {L ^ {2}}{2 \sigma^ {2}}, \frac {L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - s} ] _ {+} \\ = 2 \cdot H _ {e ^ {\varepsilon}} (P | | Q), \\ \end{array}
$$
where  $\tilde{\epsilon} = \epsilon -\left|\log \left(1 - \frac{\beta}{\lambda}\right)\right|$ ,  $P$  is the density function of  $\mathcal{N}(L,\sigma^2)$  and  $Q$  the density function of  $\mathcal{N}(0,\sigma^2)$ . This follows from the fact that the PLRV determined by the pair  $(P,Q)$  is distributed as  $\mathcal{N}\left(\frac{L^2}{2\sigma^2},\frac{L^2}{\sigma^2}\right)$ .
Continuing from (D.6), by change of variables (used after the third equality sign), we see that for  $\hat{\epsilon} z0$
$$
\begin{array}{l} 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) \left(1 - e ^ {\epsilon - \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| - \frac {L ^ {2}}{2 \sigma^ {2}} - x}\right) d x \\ = 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) d x - 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) \left(1 - e ^ {\hat {\epsilon} - x}\right) d x \\ = \left(1 - \mathrm {e} ^ {\hat {\epsilon}}\right) \cdot 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) \mathrm {d} x + \mathrm {e} ^ {\hat {\epsilon}} \cdot 2 \cdot \int_ {0} ^ {\infty} f _ {0, \frac {L ^ {2}}{\sigma^ {2}}} (x) \left(1 - e ^ {- x}\right) \mathrm {d} x \\ = \left(1 - \mathrm {e} ^ {\hat {\epsilon}}\right) + \mathrm {e} ^ {\hat {\epsilon}} \cdot 2 \cdot \int_ {\frac {L ^ {2}}{2 \sigma^ {2}}} ^ {\infty} f _ {\frac {L ^ {2}}{2 \sigma^ {2}}, \frac {L ^ {2}}{\sigma^ {2}}} (x) \left(1 - e ^ {\frac {L ^ {2}}{2 \sigma^ {2}} - x}\right) \mathrm {d} x \\ = (1 - \mathrm {e} ^ {\hat {\epsilon}}) + \mathrm {e} ^ {\hat {\epsilon}} \cdot 2 \cdot \mathbf {E} _ {s \sim \mathcal {N} (\frac {L ^ {2}}{2 \sigma^ {2}}, \frac {L ^ {2}}{\sigma^ {2}})} [ 1 - e ^ {\frac {L ^ {2}}{2 \sigma^ {2}} - s} ] _ {+} \\ = (1 - \mathrm {e} ^ {\hat {\epsilon}}) + \mathrm {e} ^ {\hat {\epsilon}} \cdot 2 \cdot H _ {\underset {e ^ {2 \sigma^ {2}}} {L ^ {2}}} (P | | Q), \\ \end{array}
$$
where we again use the fact that the PLRV of the Gaussian mechanism with sensitivity  $L$  and noise scale  $\sigma$  is distributed as  $\mathcal{N}\left(\frac{L^2}{2\sigma^2},\frac{L^2}{\sigma^2}\right)$ .
D.3 Dominating Pairs of Distributions for the Objective Perturbation Mechanism
From Lemma D.3 and the inequality (D.4) we have that for all  $\epsilon \in \mathbb{R}$
$$
H _ {e ^ {\epsilon}} \left(\hat {\theta} ^ {P} (Z) | | \hat {\theta} ^ {P} \left(Z ^ {\prime}\right)\right) \leq \mathbf {E} _ {\omega \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|} [ 1 - e ^ {\epsilon - \omega} ] +.
$$
Thus, if we have distributions  $P$  and  $Q$  such that for all  $\epsilon \in \mathbb{R}$
$$
H _ {e ^ {\epsilon}} \left(P | | Q\right) = \mathbf {E} _ {\omega \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \left| \mathcal {N} (0, \frac {L ^ {2}}{\sigma^ {2}}) \right|} \left[ 1 - e ^ {\epsilon - \omega} \right] _ {+},
$$
then the pair  $(P, Q)$  is a dominating pair of distributions for the objective perturbation mechanism. Then, by Theorem D.2, we can use this distribution  $\omega$  also to compute  $(\epsilon, \delta)$ -bounds for compositions involving the objective perturbation mechanism. We give such a pair of distributions  $(P, Q)$  explicitly in Lemma D.6 below.
In the following, we denote the density of a discrete probability mass by a Dirac delta function and use the indicator function for the continuous part of the density. The following result is a straightforward calculation.
Lemma D.4. Let  $\sigma > 0$ . Let  $P$  be the density function of  $\left|\mathcal{N}\left(0,\sigma^2\right)\right|$ , i.e.,
$$
P (x) = \frac {2}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {e} ^ {\frac {- x ^ {2}}{2 \sigma^ {2}}} \mathbf {1} _ {[ 0, \infty) ]} (x), \tag {D.7}
$$
where  $\mathbf{1}_A(x)$  denotes the indicator function, i.e.,  $\mathbf{1}_{[0,\infty)[}(x) = 1$  if  $x\geq 0$ , else  $\mathbf{1}_{[0,\infty)[}(x) = 0$ . Let  $L > 0$  and let  $Q$  be a density function, where part of the mass of  $P$  is shifted to  $-\infty$ :
$$
Q (x) = Q (- \infty) \cdot \delta_ {- \infty} (x) + \frac {2}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {e} ^ {\frac {- (t + L) ^ {2}}{2 \sigma^ {2}}} \mathbf {1} _ {[ 0, \infty) ]} (x), \tag {D.8}
$$
$$
Q (- \infty) = 1 - \int_ {0} ^ {\infty} \frac {2}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {e} ^ {\frac {- (t + L) ^ {2}}{2 \sigma^ {2}}} \mathrm {d} x.
$$
Then, we have that the PLRV  $\omega$
$$
\omega = \log \frac {P (x)}{Q (x)}, \quad x \sim P, \tag {D.9}
$$
is distributed as
$$
\omega \sim \frac {L ^ {2}}{2 \sigma^ {2}} + \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|.
$$
Proof. As  $P$  has its support on  $[0, \infty)$ , we need to consider the values of the privacy loss function  $\log \frac{P(x)}{Q(x)}$  only on  $[0, \infty)$ . We have, for all  $x \geq 0$ ,
$$
\log \frac {P (x)}{Q (x)} = \frac {L}{\sigma^ {2}} \cdot x + \frac {L ^ {2}}{2 \sigma^ {2}}.
$$
Since  $x \sim P$ , we see that  $\frac{L}{\sigma^2} \cdot x \sim \left|\mathcal{N}\left(0, \frac{L^2}{\sigma^2}\right)\right|$  and the claim follows.
Remark D.5. In Lemma D.4, instead of shifting part of the mass of  $P$  to  $-\infty$  when forming  $Q$ , we could place this mass anywhere on the negative real axis. This would not affect the PLRV  $\omega$ .
We can shift the PLRV  $\omega$  given by Lemma D.4 by scaling the distribution  $Q$ . We get the following.
Lemma D.6. Let  $\sigma > 0$  and  $L > 0$ . Suppose  $P$  is the density function given in Eq. (D.7) and  $Q$  the density function
$$
Q (x) = Q (- \infty) \cdot \delta_ {- \infty} (x) + \mathrm {e} ^ {- \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right|} \cdot \frac {2}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {e} ^ {\frac {- (t - L) ^ {2}}{2 \sigma^ {2}}} \mathbf {1} _ {[ 0, \infty) ]} (x), \tag {D.10}
$$
$$
Q (- \infty) = \left(1 - \mathrm {e} ^ {- \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right|} \cdot \int_ {0} ^ {\infty} \frac {2}{\sqrt {2 \pi \sigma^ {2}}} \mathrm {e} ^ {\frac {- (t - L) ^ {2}}{2 \sigma} ^ {2}} \mathrm {d} x\right).
$$
Then, the PLRV  $\omega$  determined by  $P$  and  $Q$  is distributed as
$$
\omega \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \left| \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right) \right|.
$$
Proof. Showing this goes as the proof of Lemma D.4. We just now have that for all  $x \geq 0$ :
$$
\log \frac {P (x)}{Q (x)} = \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {L}{\sigma^ {2}} \cdot x.
$$
As a corollary of Lemma D.6 and Thm. 2.4, we have:
Lemma D.7. Let  $k \in \mathbb{Z}_{+}$  and let for each  $i \in [k]$
$$
\omega_ {i} \sim \left| \log \left(1 - \frac {\beta}{\lambda}\right) \right| + \frac {L ^ {2}}{2 \sigma^ {2}} + \left| \mathcal {N} (0, \frac {L ^ {2}}{\sigma^ {2}}) \right|,
$$
such that  $\omega_{i}$ 's are independent. Then, the  $k$ -wise adaptive composition of  $\hat{\theta}(Z)$  is  $(\epsilon, \delta(\epsilon))$ -DP for
$$
\delta (\epsilon) = \mathbf {E} _ {s \sim \omega_ {1} + \dots + \omega_ {k}} [ 1 - e ^ {\epsilon - s} ] _ {+}. \tag {D.11}
$$
D.3.1 Numerical Evaluation of  $(\epsilon, \delta)$ -Bounds for Compositions
Figure 3 shows the result of applying the FFT-based numerical method of Koskela et al. (2021) for evaluating the expression (D.11). We compare the resulting approximate DP bounds to those obtained from the RDP bounds combined with standard composition results (Mironov, 2017).
Notice that we could also carry out tighter accounting of the approximative minima perturbation (Section 4) by adding the PLRVs of the Gaussian mechanism to the total PLRV, similarly as RDP parameters of the Gaussian mechanism are added to the RDP guarantees of the objective perturbation mechanism (Theorem 4.1). Adding the Gaussian PLRV to the total PLRV using convolutions is straightforward using the method of Koskela et al. (2021).
D.4 Auxiliary Lemma
For Theorem 3.1, we need the following auxiliary result.
Lemma D.8. Denote  $f_{\mu, \sigma^2}(x)$  the density function of the normal distribution  $\mathcal{N}(\mu, \sigma^2)$  and let  $c \geq \mu$ . Let  $g(x)$  be a non-negative differentiable non-decreasing function on  $[c, \infty)$ . Then, if  $\sigma_1 \leq \sigma_2$
$$
\int_ {c} ^ {\infty} f _ {\mu , \sigma_ {1} ^ {2}} (x) \cdot g (x) \mathrm {d} x \leq \int_ {c} ^ {\infty} f _ {\mu , \sigma_ {2} ^ {2}} (x) \cdot g (x) \mathrm {d} x.
$$
Proof. By integration by parts, we have
$$
\int_ {c} ^ {\infty} f _ {\mu , \sigma^ {2}} (x) \cdot g (x) \mathrm {d} x = - \Phi_ {\mu , \sigma^ {2}} (c) \cdot g (c) - \int_ {c} ^ {\infty} \Phi_ {\mu , \sigma^ {2}} (x) \cdot g ^ {\prime} (x) \mathrm {d} x, \tag {D.12}
$$
where  $\Phi_{\mu, \sigma^2}(x)$  denotes the cdf of  $\mathcal{N}(\mu, \sigma^2)$ . A simple calculation shows that for all  $x \in \mathbb{R}$ ,
$$
\frac {\partial}{\partial \sigma} \Phi_ {\mu , \sigma^ {2}} (x) = - \frac {x - \mu}{\sigma^ {2}} f _ {\mu , \sigma^ {2}} (x).
$$
Thus,  $\Phi_{\mu,\sigma^2}(c)$  is a non-increasing function of  $\sigma$  for all  $c \geq \mu$ . Furthermore, the first term in (D.12) is a non-decreasing function of  $\sigma$  since  $g(c)$  is non-negative and the second term is a non-decreasing function of  $\sigma$ , since  $g'(x)$  is non-negative for all  $x \in [c,\infty)$ .
Figure 3: Comparison of our RDP bound (implied  $(\epsilon, \delta)$ -DP bound) and our numerical PLRV bound (D.11) for different numbers of compositions  $k$ , when  $\sigma = 8.0$ ,  $\beta = 1.0$  and  $\lambda = 10.0$ .
E RDP guarantee of Algorithm 1
In what follows, we will present a (corrected) privacy guarantee for Approximate Minima Perturbation (i.e., Algorithm 1 without gradient clipping). We will then demonstrate that the "clipped-gradient" function  $\ell_C(\theta)$  not only bounds the per-example gradient norm by  $C$ , but also preserves other properties (i.e.,  $\beta$ -smoothness and GLM structure) required for the privacy guarantees stated in Theorem 3.2.
E.1 Privacy Guarantee for Approximate Minima Perturbation
The proof of the privacy guarantee for Approximate Minima Perturbation (Iyengar et al., 2019), i.e. Algorithm 1 without gradient clipping, falls prey to the same trap as previous work on objective perturbation. In particular, we see that there is a mistake in Lemma IV.1, with the assertion that "we get the statement of the lemma from the guarantees of the Gaussian mechanism." The Gaussian mechanism is inapplicable in Lemma IV.1 for similar reasons as discussed in Section G.
The proof of Theorem E.1 corrects this issue. We state it in terms of RDP, but it can also extend to approximate DP and other DP variants.
Algorithm 2 Approximate Minima Perturbation (Iyengar et al., 2019)
Input: dataset  $Z$ ; noise levels  $\sigma, \sigma_{\mathrm{out}}$ ;  $\beta$ -smooth loss function  $\ell(\cdot)$  with Lipschitz constant  $L$ ; regularization strength  $\lambda$ ; gradient norm threshold  $\tau$ .
Sample  $b\sim \mathcal{N}(0,\sigma^2 I_d)$
Let  $\mathcal{L}^P (\theta ;Z) = \sum_{z\in Z}\ell (\theta ;z) + \frac{\lambda}{2} ||\theta ||_2^2 +b^T\theta$
Solve for  $\tilde{\theta}$  such that  $||\nabla \mathcal{L}_C^P (\tilde{\theta};Z)||_2\leq \tau$
Output  $\tilde{\theta}^P = \tilde{\theta} +\mathcal{N}(0,\sigma_{\mathrm{out}}^2 I_d)$
Theorem E.1 (RDP guarantees of Approximate Minima Perturbation). Consider the Approximate Minima Perturbation algorithm which satisfies  $(\alpha, \epsilon)$ -RDP for any  $\alpha > 1$  with
$$
\epsilon \leq - \log \left(1 - \frac {\beta}{\lambda}\right) + \frac {L ^ {2}}{2 \sigma^ {2}} + \frac {1}{\alpha - 1} \log \mathbb {E} _ {X \sim \mathcal {N} \left(0, \frac {L ^ {2}}{\sigma^ {2}}\right)} \left[ e ^ {(\alpha - 1) | X |} \right] + \frac {\left(\frac {2 \tau}{\lambda}\right) ^ {2} \alpha}{2 \sigma_ {o u t} ^ {2}}.
$$
Proof. Sample  $b \sim \mathcal{N}(0, \sigma^2 I_d)$  and let  $\mathcal{L}^P(\theta; Z, b) \coloneqq \sum_{z \in Z} \ell(\theta; z) + \frac{\lambda}{2} \| \theta \|_2^2 + b^T\theta$ , i.e., the perturbed and regularized objective function used by ObjPert.
Let  $\theta^P = \arg \min \mathcal{L}^P (\theta ;Z,b)$ . From Chaudhuri et al. (2011); Kifer et al. (2012) we know that there is a bijection  $b(\theta^P;Z)$  from the output  $\theta^P$  to the noise vector  $b$ .
Consider a blackbox algorithm  $\theta^A (Z,b)$  which returns  $\theta$  such that  $||\nabla \mathcal{L}^{P}(\theta ;Z,b)||\leq \tau$ . Define query
$$
q (Z, \theta^ {P}) = \theta^ {A} (Z) - \theta^ {P}.
$$
where  $\theta^A (Z)$  is an abbreviation for  $\theta^A (Z,b(\theta^P;Z))$
We assume that  $q$  can recover  $b$  from the input  $\theta^P$  via the bijection  $b(\theta^P; Z)$ , and hence has access to the perturbed objective function  $\mathcal{L}^P(\theta; Z, b)$ .
Notice that since  $\mathcal{L}^P$  is  $\lambda$ -strongly convex, by applying the Cauchy-Schwarz inequality and by Definition K.3 we see that for any  $\theta_1, \theta_2$ ,
$$
\left| \left| \nabla \mathcal {L} ^ {P} (\theta_ {1}) - \nabla \mathcal {L} ^ {P} (\theta_ {2}) \right| \right| _ {2} | | \theta_ {1} - \theta_ {2} | | _ {2} \geq \left(\nabla \mathcal {L} ^ {P} (\theta_ {1}) - \nabla \mathcal {L} ^ {P} (\theta_ {2})\right) ^ {T} (\theta_ {1} - \theta_ {2}) \geq \lambda | | \theta_ {1} - \theta_ {2} | | _ {2} ^ {2}.
$$
Algorithm  $\theta^A (Z,b)$  guarantees that its output  $\theta$  satisfies  $||\nabla \mathcal{L}^{P}(\theta)||_{2}\leq \tau$  and by first-order conditions on the perturbed objective function,  $\nabla \mathcal{L}^P (\theta^P;Z,b) = 0$ . It follows that for any dataset  $Z$  and  $\theta^P$ ,
$$
\left| \left| \theta^ {A} (Z) - \theta^ {P} \right| \right| _ {2} \leq \frac {\tau}{\lambda},
$$
Since the algorithm  $\theta^A (Z,b)$  guarantees that  $\| \theta^{A} - \theta^{P}\|_{2}\leq \gamma /\lambda$ , then conditioning on  $\theta^P,q(Z,\theta^P)$  has a global sensitivity bounded by  $2\gamma /\lambda$  since
$$
\| q (Z, \theta^ {P}) - q (Z ^ {\prime}, \theta^ {P}) \| \leq \| (\theta^ {A} (Z) - \theta^ {P}) - (\theta^ {A} (Z ^ {\prime}) - \theta^ {P}) \| _ {2} \leq \| (\theta^ {A} (Z) - \theta^ {P}) \| + \| (\theta^ {A} (Z ^ {\prime}) - \theta^ {P}) \| \leq \frac {2 \gamma}{\lambda}.
$$
Now, the algorithm that first draws  $b$  then outputs  $\theta^A (Z,b) + \mathcal{N}(0,\sigma^2 I_d)$  is equivalent to
- First run ObjPert that returns  $\theta^P$ .
- Release  $\hat{\Delta} = q(Z, \theta^P) + \mathcal{N}(0, \sigma^2 I_d)$ .
- Return  $\theta^P + \hat{\Delta}$ .
This is adaptive composition of ObjPert with the Gaussian mechanism. The third step is post processing.
The privacy guarantee stated in Theorem E.1 is thus achieved by combining the results of Theorem 3.2 (RDP of ObjPert), Theorem B.1 (RDP of the Gaussian mechanism) with  $\Delta_q = \frac{2\tau}{\lambda}$ , and Lemma K.5 (adaptive composition for RDP mechanisms).
E.2 The "Clipped-Gradient" Function
The RDP guarantees of objective perturbation (stated in Theorem 3.2) require several assumptions on the loss function  $\ell (\theta ;Z)$ . If we can demonstrate that these properties are satisfied by the "clipped-gradient" loss function  $\ell_C(\theta ;Z)$ , then the rest of the proof of Theorem 4.1 (the privacy guarantee of Algorithm 1) will follow directly from that.
In particular, we need to show:
1. That  $\ell_{C}(\theta ;z)$  retains the convex GLM structure of the original function  $\ell (\theta ;z)$
2. That  $\ell_C(\theta ;z)$  satisfies  $||\nabla \ell_C(\theta ;z)||_2\leq C$  for any  $\theta ,z$
3. That  $\ell_C(\theta;z)$  has the same  $\beta$ -smoothness parameter as the original function  $\ell(\theta;z)$ .
4. That even though  $\ell_C(\theta; z)$  is not twice-differentiable everywhere, the privacy guarantees of objective perturbation (whose proof involves a Jacobian mapping) still hold.
We will begin by stating a result from Song et al. (2020).
Theorem E.2 (Song et al., 2020, Lemma 5.1). Let  $f: \mathbb{R} \to \mathbb{R}$  be any convex function and let  $C \in \mathbb{R}_+$  be any positive value. For any non-zero  $x \in \mathbb{R}^d$ , define
$$
U _ {L} = \left\{u: g <   - \frac {C}{| | x | | _ {2}} \forall g \in \partial f (u) \right\},
$$
$$
U _ {H} = \left\{u: g > \frac {C}{| | x | | _ {2}} \forall g \in \partial f (u) \right\}.
$$
If  $U_{L}$  is non-empty, let  $u_{L} = \sup U_{L}$ ; otherwise  $u_{L} = -\infty$ . If  $U_{H}$  is non-empty, let  $u_{H} = \inf U_{H}$ ; otherwise  $u_{H} = \infty$ . For any non-zero  $x \in \mathbb{R}^{d}$ , let
$$
f _ {C} (u) = \left\{ \begin{array}{l l} - \frac {C}{| | x | | _ {2}} \left(u - u _ {L}\right), & f o r u \in (- \infty , u _ {L}) \\ f (u; y), & f o r u \in \left(u _ {L}, u _ {H}\right) \\ \frac {C}{| | x | | _ {2}} \left(u - u _ {H}\right), & f o r u \in \left(u _ {H}, \infty\right) \end{array} \right.
$$
Define  $u_{x}(\theta) = x^{T}\theta$ . Then the following holds.
1.  $f_{C}$  is convex.
2. Let  $\ell (\theta ;(x,y)) = f(u_x(\theta);y)$  for any  $\theta ,z = (x,y)$ . Then we have
$$
\partial_ {\theta} \ell_ {C} (\theta ; z) = \left\{\min  \left\{1, \frac {C}{| | u _ {x} (\theta) | | _ {2})} \right\} \cdot u: u \in \partial_ {\theta} \ell (\theta ; z) \right\}.
$$
The first two desired properties of  $\ell_C(\theta; z)$ , i.e. GLM structure and gradient norm bound  $C$ , follow directly from the above theorem. Next, we will prove the third property of  $\beta$ -smoothness.
Theorem E.3. For a data point  $z = (x,y)$ , consider a function  $f$  such that  $\ell (\theta ;z) = f(x^T\theta ;y)$ . Suppose that  $f(x^{T}\theta ;y)$  satisfies  $\beta$ -smoothness. Then the "clipped-gradient" function  $f_{C}(x^{T}\theta ;y)$  defined in Lemma 5.1 of Song et al. (2020) also satisfies  $\beta$ -smoothness.
Proof. Because  $f$  is  $\beta$ -smooth by assumption, we know that  $f(u)$  satisfies  $\beta$ -smoothness for all  $u \in (u_L, u_H)$ . When  $u \in (-\infty, u_L)$  or when  $u \in (u_H, \infty)$ , the function  $f_C(u)$  is linear in  $u$  and thus is 0-smooth (hence satisfying  $\beta$ -smoothness).
Lastly, the proof of objective perturbation (see, e.g., Theorem 9 of Chaudhuri et al. (2011)) requires that the loss function be twice-differentiable. Even though  $\ell_C(\theta; z)$  is not twice-differentiable everywhere, the privacy guarantees of objective perturbation still hold. To show this, we can invoke Corollary 13 of Chaudhuri et al. (2011). This corollary assumes the Huber loss; for brevity, we will leave it as an exercise for the reader to verify that the proof also carries through for the "clipped-gradient" loss.
F Computational Guarantee of Algorithm 1