SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We consider several different regimes of interest by varying the noise scale  $\sigma$  and the regularization strength  $\lambda$ . There are several takeaways to observe:
1. The difference between the RDP for the half-normal mechanism and the RDP for the objective perturbation mechanism is due entirely to the leading term of the bound given in Theorem 3.2, which vanishes as  $\lambda$  increases (Figures 4b and 4c). For smaller  $\lambda$  there is a constant "start-up" gap between the half-normal and Objpert RDP curves (best displayed in Figure 4b) which disappears for larger  $\alpha$ , where the moments of the half-normal distribution overwhelm the contribution of the leading term of the objective perturbation RDP.
2. As  $\sigma$  increases (e.g. between Figures 4c and 4a, and between Figures 4d and 4b), the half-normal curve – and therefore also the ObjPert curve – doesn't converge with the normal curve until larger  $\alpha$ .
J Bridging the Gap between Objective Perturbation and DP-SGD
J.1 RDP of Objective Perturbation vs DP-SGD
DP-SGD (for example,  $n^2$  rounds of sampled Gaussian mechanism with Poisson sampling probability  $1 / n$ ) is known to experience a phase transition in its RDP curve: for smaller  $\alpha$ , amplification by
4More formally, we say that the "half-normal" mechanism is  $\mathcal{M}(Z) = f(Z) + |\mathcal{N}(0,\sigma^2)|$  , where  $f:\mathcal{Z}\to$ $\mathbb{R}$  is the function we wish to release.
(a)  $\sigma = 100, \lambda = 10$
(b)  $\sigma = 10, \lambda = 1$
(c)  $\sigma = 10, \lambda = 10$
(d)  $\sigma = 1, \lambda = 1$
Figure 4
sampling is effective, and the RDP of a DP-SGD mechanism behaves like a Gaussian mechanism with  $\epsilon (\alpha) = O\left(\frac{\alpha}{2\sigma^2}\right)$ , then it leaps up at a certain  $\alpha$  and begins converging to  $\epsilon (\alpha) = O\left(\frac{n^2\alpha}{2\sigma^2}\right)$  that does not benefit from sampling at all (Wang et al., 2019; Bun et al., 2018). In contrast, the RDP curve for objective perturbation defined by Theorem 3.2 converges to the RDP curve of the Gaussian mechanism  $\epsilon (\alpha) = O\left(\frac{\alpha}{2\sigma^2}\right)$  after a certain point. Whereas DP-SGD offers stronger privacy parameters for small  $\alpha$ , objective perturbation is stronger for large  $\alpha$ , which offers stronger privacy protection for lower-probability events (see, e.g., Mironov, 2017, Proposition 10).
(a)  $\sigma_{\mathrm{ObjPert}} = \sigma_{\mathrm{GM}} = 100; p = 0.1, \sigma_{\mathrm{DP-SGD}} = 10.$
Figure 5: RDP curves of objective perturbation and DP-SGD.
(b)  $\sigma_{\mathrm{ObjPert}} = \sigma_{\mathrm{GM}} = 40; p = 0.5, \sigma_{\mathrm{DP-SGD}} = 20.$
J.2 A Spectrum of DP Learning Algorithms
We can also connect Algorithm 1 to differentially private follow-the-regularized-leader (DP-FTRL) (Kairouz et al., 2021), which uses a tree-based aggregation algorithm to privately release the gradients of the loss function as a prefix sum. This approach provides a competitive privacy/utility tradeoff without relying on privacy amplification or shuffling, which is often not possible in distributed settings. DP-FTRL differs from DP-SGD by adding correlated rather than independent noise at each iteration; Algorithm 1, in contrast, differs from both by adding identical noise at each iteration.
K Technical Lemmas & Definitions
K.1 Convex Optimization
We will give a short review of relevant concepts from convex optimization.
Definition K.1 ( $\ell_2$ -Lipschitz continuity). A function  $f:\Theta \to \mathbb{R}$  is  $L$ -Lipschitz w.r.t. the  $\ell_2$ -norm over  $\Theta \subseteq \mathbb{R}^d$  if for all  $\theta_{1},\theta_{2}\in \Theta$ , the following holds:  $|f(\theta_1) - f(\theta_2)|\leq L||\theta_1 - \theta_2||_2$
Definition K.2 ( $\beta$ -smoothness). A differentiable function  $f: \Theta \to \mathbb{R}$  is  $\beta$ -smooth over  $\Theta \subseteq \mathbb{R}^d$  if its gradient  $\nabla f$  is  $\beta$ -Lipschitz, i.e.  $|\nabla f(\theta_1) - \nabla f(\theta_2)| \leq \beta ||\theta_1 - \theta_2||_2$  for all  $\theta_1, \theta_2 \in \Theta$ .
Definition K.3 (Strong convexity). A differentiable function  $f: \Theta \to \mathbb{R}$  is  $\lambda$ -strongly convex over  $\Theta \subseteq \mathbb{R}^d$  if for all  $\theta_1, \theta_2 \in \Theta$ :  $f(\theta_1) \geq f(\theta_2) + \nabla f(\theta_2)^T (\theta_1 - \theta_2) + \frac{\lambda}{2} ||\theta_1 - \theta_2||_2^2$ .
K.2 Differential Privacy
Definition K.4 (Privacy loss random variable). Let  $\operatorname*{Pr}\left[\mathcal{M}(Z) = \theta \right]$  denote the probability density of the random variable  $\mathcal{M}(Z)$  at output  $\theta$ . For a fixed pair of neighboring datasets  $Z$  and  $Z'$ , the privacy loss random variable (PLRV) of mechanism  $\mathcal{M}:\mathcal{Z}\to \Theta$  is defined as
$$
\epsilon_ {Z, Z ^ {\prime}} (\theta) = \log \frac {\operatorname * {P r} [ \mathcal {M} (Z) = \theta ]}{\operatorname * {P r} [ \mathcal {M} (Z ^ {\prime}) = \theta ]},
$$
for the random variable  $\theta \sim \mathcal{M}(Z)$
Lemma K.5 (Adaptive composition (RDP) (Mironov, 2017)). Let  $\mathcal{M}_1: \mathcal{Z} \to \mathcal{R}_1$  be  $(\alpha, \epsilon_1)$ -RDP and  $\mathcal{M}_2: \mathcal{R}_1 \times \mathcal{Z} \to \mathcal{R}_2$  be  $(\alpha, \epsilon_2)$ -RDP. Then the mechanism  $\mathcal{M} = (m_1, m_2)$ , where  $m_1 \sim \mathcal{M}_1(Z)$  and  $m_2 \sim \mathcal{M}_2(Z, m_1)$ , satisfies  $(\alpha, \epsilon_1 + \epsilon_2)$ -RDP
Lemma K.6 (Change of coordinates). Consider the map  $g: \mathcal{X} \to \mathcal{Y}$  between  $\mathcal{X} \subseteq \mathbb{R}^d$  and  $\mathcal{Y} \subseteq \mathbb{R}^d$  that transforms  $y = g(x)$ . Then
$$
\partial y = \left| \det  \frac {\partial y}{\partial x} \right| \partial x,
$$
where  $\left|\operatorname{det}\frac{\partial y}{\partial x}\right|$  is the absolute value of the determinant of the Jacobian of the map  $g$ :
$$
\frac {\partial y}{\partial x} = \left[ \begin{array}{c c c} \partial y _ {1} / \partial x _ {1} & \ldots & \partial y _ {1} / \partial x _ {d} \\ \vdots & \ddots & \vdots \\ \partial y _ {d} / \partial x _ {1} & \ldots & \partial y _ {d} / \partial x _ {d} \end{array} \right].
$$
Lemma K.7 (Change of variables for probability density functions.). Let  $g$  be a strictly monotonic function. Then for  $y = g(x)$ ,
$$
p _ {X} (x) = \left| \det  \frac {\partial y}{\partial x} \right| p _ {Y} (y).
$$
Lemma K.8. Let  $A$  be an invertible matrix. Then  $\det A^{-1} = \frac{1}{\det A}$ .
Lemma K.9 (Maximum Rayleigh quotient). For any symmetric matrix  $A \in \mathbb{R}^{d \times d}$ ,
$$
\max  _ {v \in \mathbb {R} ^ {d}} \frac {v ^ {T} A v}{v ^ {T} v} = \lambda_ {m a x},
$$
where  $\lambda_{max}$  is the largest eigenvalue of  $A$ .
Lemma K.10 (Bound on the expected norm of multivariate Gaussian with mean 0.).
Let  $x = \mathcal{N}(0, \sigma^2 I_d)$ . Then
$$
\mathbb {E} \left[ | | x | | _ {2} \right] \leq \sigma \sqrt {d}.
$$
Lemma K.11 (Gaussian MGF). Let  $X \sim \mathcal{N}(\mu, \sigma^2)$  for  $\mu \in \mathbb{R}$ ,  $\sigma \in \mathbb{R}_{>0}$ . The moment generating function of order  $t$  is then  $MGF_X(t) = e^{\mu t + \frac{1}{2}\sigma^2 t^2}$ . So for any  $t \in \mathbb{R}_{\geq 0}$  and  $\sigma_1 < \sigma_2$ ,
$$
M G F _ {X _ {1}} (t) \leq M G F _ {X _ {2}} (t),
$$
where  $X_{1}\sim \mathcal{N}(\mu ,\sigma_{1}^{2})$  and  $X_{2}\sim \mathcal{N}(\mu ,\sigma_2^2)$
Definition K.12 (Holder's Inequality). Let  $X, Y$  be random variables satisfying  $\mathbb{E}[|X|^p] < \infty$ ,  $\mathbb{E}[|X|^q] < \infty$  for  $p > 1$  and  $\frac{1}{p} + \frac{1}{q} = 1$ . Then
$$
\mathbb {E} \left[ | X Y | \right] \leq \mathbb {E} \left[ | X | ^ {p} \right] ^ {\frac {1}{p}} \mathbb {E} \left[ | X | ^ {q} \right] ^ {\frac {1}{q}}.
$$
Definition K.13 (Rényi Divergence). Let  $P, Q$  be distributions with probability density functions  $P(x), Q(x)$ . The Rényi divergence of order  $\alpha > 1$  between  $P$  and  $Q$  is given by
Lemma K.14 (MGF inequality for identical distributions). Let  $x, y, z \sim \mathcal{N}(0, \sigma^2)$ . Then
$$
\mathbb {E} _ {x, y} \left[ e ^ {t | z _ {1} | | z _ {2} |} \right] \leq \mathbb {E} _ {z} \left[ e ^ {t z ^ {2}} \right].
$$
Proof. Observe that for any  $a, b \in \mathbb{R}$ , it holds that  $2ab \leq a^2 + b^2$ . Applying this along with the Cauchy-Schwarz inequality,
$$
\begin{array}{l} \mathbb {E} _ {x, y} \left[ e ^ {t | x | | y |} \right] \leq \mathbb {E} \left[ e ^ {\frac {t}{2} x ^ {2}} e ^ {\frac {t}{2} y ^ {2}} \right] \\ \leq \sqrt {\mathbb {E} _ {x} \left[ e ^ {\frac {t}{2} x ^ {2}} \right] \mathbb {E} _ {y} \left[ e ^ {\frac {t}{2} y ^ {2}} \right]} \\ = \mathbb {E} _ {z} \left[ e ^ {t z ^ {2}} \right]. \\ \end{array}
$$