SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuition for how  $\beta$ D-Bayes provides DP estimation is provided in Figure 5 which shows the influence of adding an observation  $y$  at distance  $|y - \mu|$  from the posterior mean  $\mu$  when updating using a Gaussian distribution under KLD-Bayes and  $\beta$ D-Bayes. Here, influence is defined by [53] as the Fisher-Rao divergence between the posterior with or without that observation and can be easily estimated using an MCMC sample from the posterior without the observation. The influence of observations under KLD-Bayes is steadily increasing, making the posterior sensitive to extreme observations and therefore leaking their information. Under  $\beta$ D-Bayes, the influence initially increases before being maximised at a point depending on the value of  $\beta$ , before decreasing to 0. Therefore, each observation has bounded influence on the posterior, allowing for DP estimation.
Figure 5: The influence of adding an observation  $y$  at distance  $|y - \mu|$  from the posterior mean  $\mu$  on the posterior conditioned on a sample of 1000 points from a  $\mathcal{N}(0,1)$  when fitting a  $\mathcal{N}(\mu, \sigma^2)$ .
Before proving the paper's results, we first define all notation used.
For likelihood  $f(\cdot ;\theta)$ , prior  $\pi (\theta)$  and data  $D = \{D_{i}\}_{i = 1}^{n}$  from data generating process  $g$ , the posterior, Gibbs posterior parameters by  $w > 0$  and  $\beta \mathrm{D}$ -Bayes posterior for  $\beta >1$  are given by
$$
\begin{array}{l} \pi (\theta | D) \propto \pi (\theta) \prod_ {i = 1} ^ {n} f (D _ {i}; \theta) \\ \pi^ {w \log f} (\theta | D) \propto \pi (\theta) \exp \left\{w \sum_ {i = 1} ^ {n} f (D _ {i}; \theta) \right\} \\ \pi^ {\ell^ {(\beta)}} (\theta | D) \propto \pi (\theta) \exp \left\{- \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i}, f (\cdot ; \theta) \right\} \\ \end{array}
$$
$$
\ell^ {(\beta)} (D, f (\cdot ; \theta)) = - \frac {1}{\beta - 1} f (D; \theta) ^ {\beta - 1} + \frac {1}{\beta} \int f (\overline {{D}}; \theta) ^ {\beta} d \overline {{D}}.
$$
The population loss minimising parameters are
$$
\theta_ {0} ^ {\log \mathrm {f}} := \underset {\theta \in \Theta} {\arg \min } \int \log f (D; \theta) g (D) d D = \underset {\theta \in \Theta} {\arg \min } \operatorname {K L D} (g | | f (\cdot ; \theta))
$$
$$
\theta_ {0} ^ {\ell^ {(\beta)}} := \underset {\theta \in \Theta} {\arg \min } \int \ell^ {(\beta)} (D, f (\cdot ; \theta)) g (D) d D = \underset {\theta \in \Theta} {\arg \min } D _ {B} ^ {(\beta)} (g | | f (\cdot ; \theta))
$$
and their sample estimates
$$
\hat {\theta} _ {n} ^ {\log \mathrm {f}} := \underset {\theta \in \Theta} {\arg \min } \sum_ {i = 1} ^ {n} \log f (D _ {i}; \theta)
$$
$$
\hat {\theta} _ {n} ^ {\ell^ {(\beta)}} := \arg \min  _ {\theta \in \Theta} \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i}, f (\cdot ; \theta))
$$
If the model is well specified for the data generating process  $g$  then we write  $g(\cdot) = f(\cdot; \theta_0)$  and  $\theta_0 = \arg \min_{\theta \in \Theta} \mathrm{KLD}(g||f(\cdot; \theta)) = \arg \min_{\theta \in \Theta} D_B^{(\beta)}(g||f(\cdot; \theta))$ .
Lastly, our asymptotic results require the definitions of the Hessian and gradient cross-product matrices
$$
\hat {H} _ {n} ^ {\ell (\beta)} := \left(\frac {\partial}{\partial \theta_ {i} \partial \theta_ {j}} \frac {1}{n} \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i}, f (\cdot ; \theta_ {0} ^ {\ell (\beta)}))\right) _ {i, j}
$$
$$
H _ {0} ^ {\ell^ {(\beta)}} := \left. \left(\frac {\partial}{\partial \theta_ {i} \partial \theta_ {j}} \mathbb {E} _ {D} \left[ \ell^ {(\beta)} \left(D, f (\cdot ; \theta_ {0} ^ {\ell^ {(\beta)}})\right) \right]\right) _ {i, j} \right. \tag {5}
$$
$$
K _ {0} ^ {\ell^ {(\beta)}} := \left. \right.\left(\frac {\partial}{\partial \theta_ {i}} \mathbb {E} _ {D} \left[ \ell^ {(\beta)} \left(D; f (\cdot ; \theta_ {0} ^ {\ell^ {(\beta)}})\right)\right] \frac {\partial}{\partial \theta_ {j}} \mathbb {E} _ {D} \left[ \ell^ {(\beta)} \left(D; f (\cdot ; \theta_ {0} ^ {\ell^ {(\beta)}})\right)\right]\right) _ {i, j}. \tag {6}
$$
A.2 Bernstein-von Mises theorem for  $\beta$ D-Bayes
The general Bernstein-von Mises theorem for generalised posteriors [Theorem 4; 65] can be applied to the  $\beta$ D-Bayes posterior. We first state the required Condition 3 before stating the result.
Condition 3 (Assumptions of Theorem 4 of [65] for the  $\beta$ D-Bayes). Fix  $\theta_0^{\ell^{(\beta)}} \in \mathbb{R}^p$  and let prior  $\pi(\theta)$  is continuous at  $\theta_0^{\ell^{(\beta)}}$  with  $\pi(\theta_0^{\ell^{(\beta)}}) > 0$ . Let  $f_n^{(\beta)}: \mathbb{R}^p \to \mathbb{R}$  with  $f_n^{(\beta)}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell^{(\beta)}(D_i, f(:, \theta)$  for  $n \in \mathbb{N}$  and assume:
(1)  $f_{n}^{(\beta)}$  can be represented as
$$
f _ {n} ^ {(\beta)} (\theta) = f _ {n} ^ {(\beta)} (\hat {\theta} _ {n} ^ {\ell^ {(\beta)}}) + \frac {1}{2} (\theta - \hat {\theta} _ {n} ^ {\ell^ {(\beta)}}) ^ {T} \hat {H} _ {n} ^ {\ell^ {(\beta)}} (\theta - \hat {\theta} _ {n} ^ {\ell^ {(\beta)}}) + r _ {n} ^ {(\beta)} (\theta - \hat {\theta} _ {n} ^ {\ell^ {(\beta)}})
$$
where  $\hat{\theta}_n^{\ell(\beta)} \in \mathbb{R}^p$  and  $\hat{\theta}_n^{\ell(\beta)} \to \theta_0^{\ell(\beta)}$ , with  $\hat{H}_n^{\ell(\beta)} \to H_0^{\ell(\beta)}$  for positive definite  $H_0^{\ell(\beta)}$ , and  $r_n^{(\beta)}: \mathbb{R}^p \to \mathbb{R}$  has the following property: there exist  $\epsilon_0, c_0 > 0$  such that for all  $n$  sufficiently large, for all  $x \in B_{\epsilon_0}(0)$ , we have  $|r_n^{(\beta)}(x)| \leq c_0 |x|^3$ .
(2) For any  $\epsilon > 0$ ,  $\lim \inf_{n} \inf_{\theta \in B_{\epsilon}(\hat{\theta}_{n}^{\ell(\beta)})^{c}} (f_{n}^{(\beta)}(\theta) - f_{n}^{(\beta)}(\hat{\theta}_{n}^{\ell(\beta)})) > 0$ , where  $B_{r}(x_{0}) = \{x \in \mathbb{R}^{D} : |x - x_{0}| < r\}$
Condition 3 (1) requires that the  $\beta$ D-Bayes loss can be approximated by a quadratic form and (2) requires that as  $n$  grows the  $\beta$ D-Bayes loss is uniquely minimized at  $\hat{\theta}_n^{\ell^{(\beta)}}$ .
Theorem 3 (Theorem 4 of [65] for the  $\beta$ D-Bayes). Assume Condition 3. Define  $\pi_n^{\ell^{(\beta)}} := \pi^{\ell^{(\beta)}}(\theta | D = \{D_1, \ldots, D_n\})$ . Then
(i) The  $\beta$ D-Bayes posterior concentrates on  $\theta_0^{(\beta)}$
$$
\int_ {B _ {\varepsilon} \left(\theta_ {0} ^ {(\beta)}\right)} \pi_ {n} ^ {\ell^ {(\beta)}} (\theta) d \theta \underset {n \rightarrow \infty} {\longrightarrow} 1 \tag {7}
$$
where  $B_{r}(x_{0}) = \{x\in \mathbb{R}^{p}:|x - x_{0}| <   r\}$
(ii) The  $\beta$ D-Bayes posterior is asymptotically Gaussian
$$
\int \left| \tilde {\pi} _ {n} ^ {\ell (\beta)} (\phi) - \mathcal {N} _ {p} \left(\phi ; 0, \left(H _ {0} ^ {\ell (\beta)}\right) ^ {- 1}\right)\right| d \phi \underset {n \rightarrow \infty} {\longrightarrow} 0 \tag {8}
$$
where  $\tilde{\pi}_n^{\ell^{(\beta)}}$  denotes the density of  $\sqrt{n} (\tilde{\theta} -\hat{\theta}_n^{\ell^{(\beta)}})$  when  $\tilde{\theta}\sim \pi_n^{\ell^{(\beta)}},\mathcal{N}_p(x;\mu ,\Sigma)$  denotes the  $p$ -dimensional multivariate Gaussian distribution with mean vector  $\mu$  and covariance matrix  $\boldsymbol{\Sigma}$
Proof. The result is proved as a direct application of Theorem 4 of [65] with  $f_{n}(\theta) = \frac{1}{n}\sum_{i=1}^{n}\ell^{(\beta)}(D_{i},f(\cdot;\theta))$  being the  $\beta$ D-Bayes loss function.
Theorem 3 shows that the  $\beta$ D-Bayes posterior concentrates on  $\theta_0^{(\beta)}$  and converges to a Gaussian distribution centred around the  $\beta$ D minimising parameter  $\theta_0^{\ell^{(\beta)}}$  in total variation distance.
A.3 Proofs
A.3.1 Proof of Lemma 1
Lemma 1 (Bounded sensitivity of the  $\beta$ D-Bayes loss). Under Condition 1 the sensitivity of the  $\beta$ D-Bayes-loss for any  $\beta > 1$  is  $|\ell^{(\beta)}(D, f(\cdot; \theta)) - \ell^{(\beta)}(D', f(\cdot; \theta))| \leq \frac{M^{\beta - 1}}{\beta - 1}$ .
Proof. By (4), for  $\beta > 1$
$$
\begin{array}{l} \left| \ell^ {(\beta)} (D, f (\cdot ; \theta)) - \ell^ {(\beta)} \left(D ^ {\prime}, f (\cdot ; \theta)\right) \right| = \frac {1}{\beta - 1} \left(f \left(D ^ {\prime}; \theta\right) ^ {\beta - 1} - f (D; \theta) ^ {\beta - 1}\right) \\ \leq \max  _ {D} \frac {1}{\beta - 1} f (D; \theta) ^ {\beta - 1} \\ \leq \frac {M ^ {\beta - 1}}{\beta - 1} \\ \end{array}
$$
A.3.2 Proof of Theorem 1
Theorem 1 (Differential privacy of the  $\beta$ D-Bayes posterior). Under Condition 1, a draw  $\tilde{\theta}$  from the  $\beta$ D-Bayes posterior  $\pi^{(\beta)}(\theta|D)$  in (3) is  $(\frac{2M^{\beta - 1}}{\beta - 1},0)$ -differentially private.
Proof. Define  $D = \{D_1, \ldots, D_n\}$ ,  $D' = \{D_1', \ldots, D_n'\}$  and let  $j$  be the index such that  $D_j \neq D_j'$  with  $D_i = D_i'$  for all  $i \neq j$ . Firstly, the normalising constant of the  $\beta$ -D-Bayes posterior combining (3) with (4) is
$$
P ^ {(\beta)} (D) := \int \pi (\theta) \exp \left\{- \sum_ {i = 1} ^ {n} \ell^ {(\beta)} \left(\theta , D _ {i}\right) \right\} d \theta \tag {9}
$$
$$
\begin{array}{l} \log \frac {\pi^ {(\beta)} (\theta | D)}{\pi^ {(\beta)} (\theta | D ^ {\prime})} = \sum_ {i = 1} ^ {n} \ell^ {(\beta)} \left(D _ {i} ^ {\prime}, f (\cdot ; \theta)\right) - \sum_ {i = 1} ^ {n} \ell^ {(\beta)} \left(D _ {i}, f (\cdot ; \theta)\right) + \log \frac {P ^ {(\beta)} (D ^ {\prime})}{P ^ {(\beta)} (D)} \\ = \ell^ {(\beta)} (D _ {j} ^ {\prime}; f (\cdot ; \theta)) - \ell^ {(\beta)} (D _ {j}; f (\cdot ; \theta)) + \log \frac {P ^ {(\beta)} (D ^ {\prime})}{P ^ {(\beta)} (D)} \\ \end{array}
$$
Now, by Condition 1 and Lemma 1,
$$
\ell^ {(\beta)} \left(D _ {j} ^ {\prime}; f (\cdot ; \theta)\right) - \ell^ {(\beta)} \left(D _ {j}; f (\cdot ; \theta)\right) \leq \frac {M ^ {\beta - 1}}{\beta - 1},
$$
$$
\begin{array}{l} P ^ {(\beta)} (D ^ {\prime}) = \int \exp \left\{- \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i} ^ {\prime}, f (\cdot ; \theta)) \right\} \pi (\theta) d \theta \\ = \int \exp \left\{\ell^ {(\beta)} (D _ {j}, f (\cdot ; \theta)) - \ell^ {(\beta)} (D _ {j} ^ {\prime}, f (\cdot ; \theta)) - \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i}, f (\cdot ; \theta)) \right\} \pi (\theta) d \theta \\ \leq \exp \left\{\frac {M ^ {\beta - 1}}{\beta - 1} \right\} \int \exp \left\{- \sum_ {i = 1} ^ {n} \ell^ {(\beta)} (D _ {i}, f (\cdot ; \theta)) \right\} \pi (\theta) d \theta , \\ \end{array}
$$
which combined provides that
$$
\log \frac {\pi (\theta | D)}{\pi (\theta | D ^ {\prime})} \leq 2 \frac {M ^ {\beta - 1}}{\beta - 1}.
$$
Theorem 2 (Consistency of  $\beta$ D-Bayes sampling). Under Conditions 3,
1. a posterior sample  $\tilde{\theta} \sim \pi^{\ell^{(\beta)}}(\theta | \tilde{D})$  is a consistent estimator of  $\theta_0^{\ell^{(\beta)}}$ .
2. if data  $D_{1},\ldots ,D_{n}\sim g(\cdot)$  were generated such that there exists  $\theta_0$  with  $g(D) = f(D;\theta_0)$ , then  $\tilde{\theta}\sim \pi^{\ell^{(\beta)}}(\theta |D)$  for all  $1 < \beta < \infty$  is consistent for  $\theta_0$ .
Proof. For Part 1), define  $B_r(x_0) = \{x \in \mathbb{R}^p : |x - x_0| < r\}$ . Theorem 3 Part (i) proves that
$$
\int_ {B _ {\varepsilon} \left(\theta_ {0} ^ {(\beta)}\right)} \pi_ {n} ^ {\ell^ {(\beta)}} (\theta) d \theta \underset {n \rightarrow \infty} {\longrightarrow} 1 \tag {10}
$$
for all  $\varepsilon > 0$  and  $\pi_n^{\ell^{(\beta)}} \coloneqq \pi^{\ell^{(\beta)}}(\theta|D = \{D_1, \ldots, D_n\})$ . This is enough to show that for  $\tilde{\theta} \sim \pi^{\ell^{(\beta)}}(\theta|D) \to \theta_0^{\ell^{(\beta)}}$  in probability.
For Part 2), note that if  $g(D) = f(D; \theta_0)$ , then for all  $1 < \beta < \infty$
$$
\begin{array}{l} \theta_ {0} ^ {\ell^ {(\beta)}} := \underset {\theta \in \Theta} {\arg \min } \mathbb {E} _ {g} \left[ \ell^ {(\beta)} (D; f (\cdot ; \theta)) \right] \\ = \operatorname * {a r g   m i n} _ {\theta \in \Theta} D _ {B} ^ {(\beta)} (g | | f (\cdot ; \theta)) = \theta_ {0}. \\ \end{array}
$$
A.3.4 Proof of Proposition 1
Proposition 1 (Asymptotic efficiency). Under Conditions 3,  $\tilde{\theta} \sim \pi^{\ell^{(\beta)}}(\theta|D)$  is asymptotically distributed as  $\sqrt{n} (\tilde{\theta} - \theta_0^{\ell^{(\beta)}}) \stackrel{\text{weakly}}{\longrightarrow} \mathcal{N}(0, (H_0^{\ell^{(\beta)}})^{-1} K_0^{\ell^{(\beta)}}(H_0^{\ell^{(\beta)}})^{-1} + (H_0^{\ell^{(\beta)}})^{-1})$ , where  $K_0^{\ell^{(\beta)}}$  and  $H_0^{\ell^{(\beta)}}$  are the gradient cross-product and Hessian matrices for the  $\beta D$  loss and are defined in (5) and (6).
Proof. Let  $\tilde{\theta} \sim \pi^{\ell^{(\beta)}}(\theta | D)$ . By Theorem 3 Part (ii),
$$
\sqrt {n} (\tilde {\theta} - \hat {\theta} _ {n} ^ {\ell (\beta)}) \to \mathcal {N} (0, (H _ {0} ^ {\ell (\beta)}) ^ {- 1}).
$$
By the asymptotic normality of  $\hat{\theta}_n^{\ell^{(\beta)}}$  [10], we have that
$$
\sqrt {n} \left(\hat {\theta} _ {n} ^ {\ell (\beta)} - \theta_ {0} ^ {\ell (\beta)}\right)\rightarrow^ {D} \mathcal {N} \left(0, \left(H _ {0} ^ {\ell (\beta)}\right) ^ {- 1} K _ {0} ^ {\ell (\beta)} \left(H _ {0} ^ {\ell (\beta)}\right) ^ {- 1}\right)
$$
$$
K _ {0} ^ {\ell^ {(\beta)}} := \left. \right.\left(\frac {\partial}{\partial \theta_ {i}} \mathbb {E} _ {D} \left[ \ell^ {(\beta)} (D; f (\cdot ; \theta_ {0} ^ {\ell^ {(\beta)}})) \right] \frac {\partial}{\partial \theta_ {j}} \mathbb {E} _ {D} \left[ \ell^ {(\beta)} (D; f (\cdot ; \theta_ {0} ^ {\ell^ {(\beta)}})) \right]\right) _ {i, j}.
$$
The result then comes from the asymptotic independence of  $\tilde{\theta} -\hat{\theta}_n^{\ell (\beta)}$  and  $\hat{\theta}_n^{\ell (\beta)}$  [see e.g. 81].
A.3.5 Proof of Proposition 3
Lemma 1 considers the sensitivity of the  $\beta$ D-Bayes loss to the change in one observation. DP-MCMC methods require the sensitivity to the change in the parameter which is provided by Lemma 2.
Lemma 2 (Bounded parameter sensitivity of the  $\beta$ D-Bayes loss). Under Condition 1 the parameter sensitivity of the  $\beta$ D-Bayes-loss for any  $\beta > 1$  is  $|\ell^{(\beta)}(D, f(:, \theta)) - \ell^{(\beta)}(D, f(:, \theta'))| \leq \frac{2\beta - 1}{\beta(\beta - 1)} M^{\beta - 1}$ .
$$
\begin{array}{l} \left| \ell^ {(\beta)} (D, f (\cdot ; \theta)) - \ell^ {(\beta)} (D, f (\cdot ; \theta^ {\prime})) \right| \\ = \left| - \frac {1}{\beta - 1} f (D; \theta) ^ {\beta - 1} + \frac {1}{\beta} \int f (\bar {D}; \theta) ^ {\beta} d \bar {D} + \frac {1}{\beta - 1} f (D; \theta^ {\prime}) ^ {\beta - 1} - \frac {1}{\beta} \int f (\bar {D}; \theta^ {\prime}) ^ {\beta} d \bar {D} \right| \\ \leq \max  _ {\theta , \theta^ {\prime}} \left\{\frac {1}{\beta} \int f (\bar {D}; \theta) ^ {\beta} d \bar {D} + \frac {1}{\beta - 1} f (D; \theta^ {\prime}) ^ {\beta - 1} \right\} \\ \leq \max  _ {\theta , \theta^ {\prime}} \left\{\frac {1}{\beta} \mathbb {E} _ {f (\bar {D}; \theta)} [ f (\bar {D}; \theta) ^ {\beta - 1} ] + \frac {1}{\beta - 1} M ^ {\beta - 1} \right\} \\ \leq \frac {1}{\beta} M ^ {\beta - 1} + \frac {1}{\beta - 1} M ^ {\beta - 1} \\ = \frac {2 \beta - 1}{\beta (\beta - 1)} M ^ {\beta - 1} \\ \end{array}
$$
Proof. Algorithm 1 of [83] (Assumption A1), Algorithm 1 of [73] and Algorithm 2 of [85] (Assumptions 1 and 2) require a posterior whose log-likelihood has bounded parameter sensitivity. For  $\beta$  D-Bayes posterior, this requires the  $\beta$  D-Bayes-loss to have bounded parameter sensitivity which is provided by Condition 1 and Lemma 2.
Algorithm 1 of [56] requires a posterior whose log-likelihood has bounded gradient. For  $\beta$  D-Bayes posterior, this requires  $\beta$  D-Bayes-loss to have bounded gradient. Assuming we can interchange integration and differentiation, this requires
$$
\begin{array}{l} \left| \left| \nabla_ {\theta} \ell^ {(\beta)} (D; \theta) \right| \right| _ {\infty} = \left| \left| \nabla_ {\theta} f (D; \theta) \times f (D; \theta) ^ {\beta - 2} - \int \nabla_ {\theta} f (\overline {{D}}; \theta) \times f (\overline {{D}}; \theta) ^ {\beta - 1} d \overline {{D}} \right| \right| _ {\infty} \\ = \left| \left| \nabla_ {\theta} f (D; \theta) \times f (D; \theta) ^ {\beta - 2} - \int \nabla_ {\theta} f (\bar {D}; \theta) \times f (\bar {D}; \theta) ^ {\beta - 2} \times f (\bar {D}; \theta) d \bar {D} \right| \right| _ {\infty} \\ \leq \max  \left\{G ^ {(\beta)}, G ^ {(\beta)} M \right\}, \\ \end{array}
$$
as  $\left|\left|\nabla_{\theta}f(D;\theta)\times f(D;\theta)^{\beta -2}\right|\right|_{\infty}\leq G^{(\beta)}$  by Condition 2.
We show that Condition 2 is satisfied for logistic regression.
Example 3 (Satisfying Condition 2 for logistic regression). For Logistic regression introduced in Example 1
$$
\begin{array}{l} f (y; \theta , X) = \frac {1}{(1 + \exp (- (2 y - 1) X \theta))} \\ \nabla_ {\theta} f (y; \theta , X) = \frac {(2 y - 1) X}{(1 + \exp (- X \theta)) ^ {2}} \\ f (y; \theta , X) ^ {\beta - 2} = (1 + \exp (- (2 y - 1) X \theta)) ^ {2 - \beta} \\ \end{array}
$$
$$
\begin{array}{l} \left| \left| \nabla_ {\theta} f (y; \theta , X) \times f (y; \theta , X) ^ {\beta - 2} \right| \right| _ {\infty} = \left| \left| \frac {(2 y - 1) X (1 + \exp (- (2 y - 1) X \theta)) ^ {2 - \beta}}{(1 + \exp (- (2 y - 1) X \theta)) ^ {2}} \right| \right| _ {\infty} \\ = \left| \left| \frac {(2 y - 1) X}{(1 + \exp (- (2 y - 1) X \theta)) ^ {\beta}} \right| \right| _ {\infty} \\ \leq | | X | | _ {\infty}. \\ \end{array}
$$
Satisfying Condition 2 for logistic regression requires bounding of the features, a similar requirement to the DP methods of [81, 66, 18].
A.4 Related work
Here, we would like to extend our discussion of two important areas within the related work.
A.4.1 Differentially private logistic regression
Extensions to  $L_{1}$ -sensitivity Our presentation of the object perturbation of Chaudhuri et al. [18] in Section 2 differs slightly from its original presentation. Corollary 8 [18] assumes that  $\|\nabla_{\theta}\ell(D_i, f(\cdot; \theta))\| < 1$  which allows the bounding of the  $L_{2}$ -sensitivity of the empirical risk minimiser (1). Then adding multivariate Laplace noise  $z' \sim \nu(z) \propto \exp\left(-\frac{n\lambda\epsilon}{2}||z'||_2\right)$  provides DP estimation. For logistic regression,  $\|\nabla_{\theta}\ell(D_i, f(\cdot; \theta))\| < 1$  requires that the features  $X$  are such that  $||X||_2 < 1$ .
In order to provide a fairer comparison with [66], who require that  $|\nabla_{\theta_j}\ell (D_i,f(\cdot ;\theta)| < 1, j = 1,\ldots ,p$ , which for logistic regression requires  $|X_{j}| < 1$  for  $j = 1,\dots ,p$ , we add univariate Laplace noise to each parameter estimate requiring us to bound the  $L_{1}$  sensitivity of the empirical risk minimiser (1). This is a natural extension of the result of Chaudhuri et al. [18], but for completeness, we provide a proof that the approach presented in Section 2 is  $(\epsilon ,0)$ -DP.
Theorem 4 (Modification of Output-Perturbation Chaudhuri et al. [18]). Consider empirical risk minimisation for a  $p$  dimensional parameter  $\theta \in \Theta \subseteq \mathbb{R}^p$ :
$$
\hat {\theta} (D) := \underset {\theta \in \Theta} {\arg \min } \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (D _ {i}, f (\cdot ; \theta)) + \lambda R (\theta)
$$
where  $\ell(D_i, f(\cdot; \theta))$  is convex and differentiable loss function with  $|\nabla_{\theta_j} \ell(D_i, f(\cdot; \theta)| < 1$ ,  $j = 1, \ldots, p$ ,  $R(\theta)$  is a differentiable and 1-strongly convex regulariser, and  $\lambda > 0$  is the regularisation weight. Then releasing
$$
\tilde {\theta} = \hat {\theta} (D) + z, \mathbb {R} ^ {p} \in z = (z _ {1}, \dots , z _ {p}) \stackrel {i i d} {\sim} \mathcal {L} \left(0, \frac {2}{n \lambda \epsilon}\right),
$$
where  $\mathcal{L}(\mu, s)$  is a Laplace distribution with density  $f(z) = \frac{1}{2s}\exp\{-|z - \mu|/s\}$ , is  $(\epsilon, 0)$ -DP.
Proof. Fix datasets  $D_{1}$  and  $D_{2} = \{D_{1} \setminus D_{k}\} \cup D_{l}$ . Then define
$$
\theta_ {k} := \hat {\theta} (D _ {k}) = \underset {\theta \in \Theta} {\arg \min } G _ {k} (\theta)
$$
$$
G _ {k} (\theta): = \frac {1}{n} \sum_ {D \in D _ {k}} \ell (D, f (.; \theta) + \lambda R (\theta)
$$
$$
\begin{array}{l} g (\theta) := G _ {1} (\theta) - G _ {2} (\theta) \\ = \frac {1}{n} \ell (D _ {k}, f (\cdot ; \theta) - \frac {1}{n} \ell (D _ {l}, f (\cdot ; \theta) \\ \end{array}
$$
Lemma 7 of Chaudhuri et al. [18] then proves that
$$
\left| \left| \theta_ {1} - \theta_ {2} \right| \right| _ {2} \leq \frac {1}{\lambda} \max  _ {\theta \in \Theta} \left| \left| \nabla_ {\theta} g (\theta) \right| \right| _ {2}
$$
Now, as  $|\nabla_{\theta_j}\ell (D_i,f(\cdot ;\theta)| <   1,j = 1,\ldots ,p$
$$
\begin{array}{l} | | \nabla_ {\theta} g (\theta) | | _ {2} = \frac {1}{n} | | \nabla_ {\theta} \ell (D _ {k}, f (\cdot ; \theta) - \nabla_ {\theta} \ell (D _ {l}, f (\cdot ; \theta) | | _ {2} \\ \leq \frac {2}{n} \max  _ {D} | | \nabla_ {\theta} \ell (D, f (\cdot ; \theta) | | _ {2} \\ = \frac {2}{n} \max  _ {D} \sqrt {\sum_ {j = 1} ^ {p} \nabla_ {\theta_ {j}} \ell (D , f (\cdot ; \theta)} \\ \leq \frac {2 \sqrt {p}}{n} \\ \end{array}
$$
The original result of Chaudhuri et al. [18] assumed that  $||\nabla_{\theta}\ell (D_i,f(\cdot ;\theta)||_2 < 1$  and they obtained  $||\nabla_{\theta}g(\bar{\theta})||_2 < \frac{2}{n}$ . Instead, we require a weaker element wise bound that  $|\nabla_{\theta_j}\ell (D_i,f(\cdot ;\theta)| < 1$  and note that
$$
\left| \left| \theta_ {1} - \theta_ {2} \right| \right| _ {2} \leq \frac {2 \sqrt {p}}{\lambda} \Rightarrow \left| \left| \theta_ {1} - \theta_ {2} \right| \right| _ {1} \leq \frac {2}{n \lambda}.
$$
Therefore by Proposition 1 of [26] we have that  $\tilde{\theta} = \hat{\theta}(D) + (z_1, \ldots, z_p)$  with  $z_j \sim \mathcal{L}\left(0, \frac{2}{n\epsilon\lambda}\right)$  is  $(\epsilon, 0)$ -DP.
Note, 1-strong convexity of an  $L_{2}$  type regulariser requires that  $R(\theta) = \frac{1}{2} ||\theta||_2^2 = \frac{1}{2}\sum_{j=1}^{p}\theta_j^2$ . This parametrisation is the default implementation in sklearn.
Consistency Chaudhuri et al. [18] propose a regularised DP logistic regression, solving (1). The optimisation problem in (1) adds the regulariser to the average loss and as a result, the impact of the regulariser does not diminish as  $n \to \infty$ . Even though the scale of the Laplace noise decreases as  $n$  grows, Chaudhuri et al. [18] consistently estimate a parameter that is not the data generating parameter. Alternatively, one could choose a regulariser  $\lambda' := \frac{\lambda}{n}$  whose influence decreases as  $n$  grows. This would allow for unbiased inference as  $n \to \infty$  (assuming a Bayesian model with corresponding prior distribution), but the  $n$  cancels in the scale of the Laplace noise and therefore the perturbation scale does not decrease in  $n$ , and the estimator is inconsistent. Choosing instead  $\lambda' := \frac{\lambda}{n^r}$  with  $0 < r < 1$ , would help in constructing unbiased and consistent estimators. In our experiments, we did not find this choice to help.
A.4.2 Differentially private Monte Carlo methods
Wang et al. [81] propose using Stochastic Gradient Langevin Dynamics [SGLD; 82] with a modified burn-in period and bounded step-size to provide DP sampling when the log-likelihood has bounded gradient. Li et al. [56] improve upon [81], taking advantage of the moments accountant [1] to allow for a larger step-size and faster mixing for non-convex target posteriors. Foulds et al. [27] extend their privatisation of sufficient statistics to a Gibbs sampling setting where the conditional posterior distribution for a Gibbs update is from the exponential family. Yildirim and Ermis [83] use the penalty algorithm which adds noise to the log of the Metropolis-Hastings acceptance probability. Heikkilä et al. [37] use Barker's acceptance test [8, 76] and provide RDP guarantees. Raïsa et al. [73] derive DP-HMC also using the penalty algorithm. Zhang and Zhang [85] propose a random batch size implementation of Metropolis-Hastings for a general proposal distribution that takes advantage of the inherent randomness of Metropolis-Hastings and is asymptotically exact. Lastly, Awan and Rao [7] consider DP rejection sampling.
A.5 Attack optimality
The privacy attacks outlined in Section 4 require the calculation of