SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We do this in the following set-up: Given a target  $z$ , we initialize our reconstruction from uniform noise and optimize with the gradient-based reconstruction attack introduced in Section 2 to produce  $\hat{z}$ . We mark  $\hat{z}$  as a successful reconstruction of  $z$  if  $\frac{1}{d}\sum_{i=1}^{d}\mathbb{I}[|z[i] - \hat{z}[i]| < \delta] \geq \rho$ , where  $\rho \in [0,1]$ ,  $d$  is the data dimensionality, and we set  $\delta = \frac{32}{255}$  in our experiments. If  $\rho = 1$  this means we mark the reconstruction as successful if  $\| \hat{z} - z \|_{\infty} < \delta$ , and for  $\rho < 1$ , then at least a fraction  $\rho$  values in  $\hat{z}$  must be within an  $\ell_{\infty}$  ball of radius  $\delta$  from  $z$ . Under the assumption the adversary has no background knowledge of the target point, with  $\delta = \frac{32}{255}$  and a uniform prior, the prior probability of reconstruction is given by  $(2\times 32 / 256)^{d\rho}$  — if  $\rho = 1$ , for MNIST and CIFAR-10, this means the prior probability of a successful reconstruction is  $9.66\times 10^{-473}$  and  $2.96\times 10^{-1850}$ , respectively.
We plot the reconstruction upper bound compared to the attack success for different values of  $\rho$  in Figure 12. We also visualize the quality of reconstructions for different values of  $\rho$ . Even for  $\rho = 0.6$ , where  $40\%$  of the reconstruction pixels can take any value, and the remaining  $60\%$  are within an absolute value of  $\frac{32}{255}$  from the target, one can easily identify that the reconstructions look visually similar to the target.
Figure 12: Comparison of reconstruction success under a very small prior. The prior probability of success for MNIST and CIFAR-10 are  $9.66 \times 10^{-473}$  and  $2.96 \times 10^{-1850}$ , respectively.
J Discussion on related work
Here, we give a more detailed discussion of relevant related work over what is surfaced in Section 1 and Section 2.
DP and reconstruction. By construction, differential privacy bounds the success of a membership inference attack, where the aim is to infer if a point  $z$  was in or out of the training set. While the connection between membership inference and DP is well understood, less is known about the relationship between training data reconstruction attacks and DP. A number of recent works have begun to remedy this in the context of models trained with DP-SGD by studying the value of  $\epsilon$  required to thwart training data reconstruction attacks (Bhowmick et al., 2018; Balle et al., 2022; Guo et al., 2022a,b; Stock et al., 2022). Of course, because differential privacy bounds membership inference, it will also bound ones ability to reconstruct training data; if one cannot determine if  $z$  was used in training, they will not be able to reconstruct that point. These works are interested in both formalizing training data reconstruction attacks, and quantifying the necessary  $\epsilon$  required to bound its success. Most of these works share a common finding – the  $\epsilon$  value needed for this bound is much larger than the value required to protect against membership inference attacks (< 10 in practice). If all other parameters in  $q\sqrt{T\log(\frac{1}{\delta})} / \varepsilon$  remain fixed, one can see that a larger value of  $\epsilon$  reduces the scale of noise we add to gradients, which in turn results in models that achieve smaller generalization error than models trained with DP-SGD that protect against membership inference.
The claim that a protection against membership inference attacks also protects against training data reconstruction attacks glosses over many subtleties. For example, if  $z$  was not included in training it could still have a non-zero probability of reconstruction if samples that are close to  $z$  were included in training. Balle et al. (2022) take the approach of formalizing training reconstruction attacks in a
Bayesian framework, where they compute a prior probability of reconstruction, and then find how much more information an adversary gains by observing the output of DP-SGD.
Balle et al. (2022) use an average-case definition of reconstruction over the output of a randomized mechanism. In contrast, Bhowmick et al. (2018) define a worst-case formalization, asking when should an adversary not be able to reconstruct a point of interest regardless of the output of the mechanism. Unfortunately, such worst-case guarantees are not attainable under DP-relaxations like  $(\epsilon, \delta)$ -DP and RDP, because the privacy loss is not bounded; there is a small probability that the privacy loss will be high.
Stock et al. (2022) focus on bounding reconstruction for language tasks. They use the probability preservation guarantee from RDP to derive reconstruction bounds, showing that the length of a secret within a piece of text itself provides privacy. They translate this to show a smaller amount of DP noise is required to protect longer secrets.
While Balle et al. (2022) propose a Bayesian formalization for reconstruction error, Guo et al. (2022a) propose a frequentist definition. They show that if  $M$  is  $(2,\epsilon)$ -RDP, then the reconstruction MSE is lower bounded by  $\sum_{i=1}^{d} \frac{\mathrm{diam}_i(\mathcal{Z})^2}{4d(e^\epsilon - 1)}$ , where  $\mathrm{diam}_i(\mathcal{Z})$  is the diameter of the space  $\mathcal{Z}$  in the  $i$ th dimension.
Gradient inversion attacks. The early works of Wang et al. (2019) and Zhu et al. (2019) showed that one can invert single image representation from gradients of a deep neural network. Zhu et al. (2019) actually went beyond this and showed one can jointly reconstruct both the image and label representation. The idea is that given a target point  $z$ , a loss function  $\ell$ , and an observed gradient (wrt to model parameters  $\theta$ )  $g_{z} = \nabla_{\theta}\ell (\theta ,z)$ , to construct a  $\hat{z}$  such that  $\hat{z} = \arg \min_{z^{\prime}}\| g_{z^{\prime}} - g_{z}\|$ . The expectation is that images that have similar gradients will be visually similar. By optimizing the above objective with gradient descent, Zhu et al. (2019) showed that one can construct visually accurate reconstruction on standard image benchmark datasets like CIFAR-10.
Jeon et al. (2021); Yin et al. (2021); Jin et al. (2021); Huang et al. (2021); Geiping et al. (2020) proposed a number of improvements over the reconstruction algorithm used in Zhu et al. (2019): they showed how to reconstruct multiple training points in batched gradient descent, how to optimize against batch normalization statistics, and incorporate priors into the optimization procedure, amongst other improvements.
The aforementioned attacks assumed an adversary has access to gradients through intermediate model updates. Balle et al. (2022) instead investigate reconstruction attacks when adversary can only observe a model after it has finished training, and propose attacks against (parametric) ML models under this threat model. However, the attack they construct is computationally demanding as it involves retraining thousands of models. This computational bottleneck is also a factor in Haim et al. (2022), who also investigate training data reconstruction attacks where the adversary has access only to final model parameters.
K More experiments on the effect of DP-SGD hyperparameters
We extend on our investigation into the effect that DP-SGD hyperparameters have on reconstruction. We begin by varying the clipping norm parameter,  $C$ , and measure the effect on reconstruction. Following this, we replicate our results from Section 4.3 (the effect hyperparameters have on reconstruction at a fixed  $\epsilon$ ) across different values of  $\epsilon$  and prior sizes,  $|\pi|$ .
K.1 Effect of clipping norm
If we look again at our attack set-up in Algorithm 2, we see that in essence we are either summing a set of samples only from a Gaussian centred at zero or a Gaussian centred at  $C^2$ . If the gradient of the target point is not clipped, then this will reduce the sum of gradients when the target is included in a batch, as the Gaussian will be centred at a value smaller than  $C^2$ . This will increase the probability that the objective is not maximized by the target point.
We demonstrate how this changes the reconstruction success probability by training a model for 100 steps with a clipping norm of 0.1 or 1, and measuring the average gradient norm of all samples over all steps. Results are shown in Figure 13. We see at  $C = 0.1$ , our attack is tight to the upper bound,
(a) Average gradient norm (over all samples and steps) for different values of  $\epsilon$  at  $C = 0.1$  and  $C = 1$ .
(b) Reconstruction success probability for different values of  $\epsilon$  at  $C = 0.1$  and  $C = 1$ .
Figure 13: Comparison of how reconstruction success is changes with the clipping norm,  $C$ . We see that if examples have a gradient norm smaller than  $C$ , and so are not clipped, reconstruction success probability becomes smaller.
and the average gradient norm is 0.1 for all values of  $\epsilon$ ; all individual gradients are clipped. When  $C = 1$ , the average gradient norm decreases from 0.9 at  $\epsilon = 1$  to 0.5 at  $\epsilon = 40$ , and we see a larger gap between upper and lower bounds. The fact that some gradients may not be clipped is not taken into account by our theory used to compute upper bounds, and so we conjecture that the reduction is reconstruction success is a real effect rather than a weakness of our attack.
We note that these findings chime with work on individual privacy accounting (Feldman & Zrnic, 2021; Yu et al., 2022; Ligett et al., 2017; Redberg & Wang, 2021). An individual sample's privacy loss is often much smaller than what is accounted for by DP bounds. These works use the gradient norm of an individual sample to measure the true privacy loss, the claim is that if the gradient norm is smaller than the clipping norm, the amount of noise added is too large, as the DP accountant assumes all samples are clipped. Our experiments support the claim that there is a disparity in privacy loss between samples whose gradients are and are not clipped.
K.2 More results on the effect of DP-SGD hyperparameters at a fixed  $\epsilon$
In Section 4.3, we demonstrated that the success of a reconstruction attack cannot be captured only by the  $(\epsilon, \delta)$  guarantee, when  $\epsilon = 4$  and the size of the prior,  $\pi$ , is set to ten. We now observe how these results change across different  $\epsilon$  and  $|\pi|$ , where we again fix the number of updates to  $T = 100$ ,  $C = 1$ , vary  $q \in [0.01, 0.99]$ , and adjust  $\sigma$  accordingly.
Firstly, in Figure 14, we measure the upper and lower bound ((improved) prior-aware attack) on the probability of successful reconstruction across different  $q$ . In all settings, we observe smaller reconstruction success at smaller  $q$ , where the largest fluctuations in reconstruction success are for larger values of  $\epsilon$ . We visualise this in another way by plotting  $\sigma$  against  $q$  and report the upper bound in Figure 15. Note that the color ranges in Figure 15 are independent across subfigures.
L Estimating  $\kappa$  from samples
Here, we discuss how to estimate the base probability of reconstruction success,  $\kappa$ , if the adversary can only sample from the prior distribution.
Let  $\hat{\pi}$  be the empirical distribution obtained by taking  $N$  independent samples from the prior and  $\hat{\kappa} = \kappa_{\hat{\pi},\rho}(\eta)$  be the corresponding parameter for this discrete approximation to  $\pi$  - this can be computed using the methods sketched in Section 3. Then we have the following concentration bound.
Proposition 6. With probability  $1 - e^{-N\tau^2\kappa /2}$  we have
$$
\kappa \leq \frac {\hat {\kappa}}{1 - \tau} .
$$
The proof is given in Appendix M.
(a)  $\epsilon = 1, |\pi| = 2$ .
(b)  $\epsilon = 4,|\pi | = 2$
(c)  $\epsilon = 8, |\pi| = 2$ .
(d)  $\epsilon = 16, |\pi| = 2$ .
(e)  $\epsilon = 1,|\pi | = 10$
(i)  $\epsilon = 1, |\pi| = 100$ .
Figure 14: How the upper bound and (improved) prior-aware attack change as a function of  $q$  at a fixed value of  $\epsilon$  and prior size,  $|\pi|$ . The amount of privacy leaked through a reconstruction at a fixed value of  $\epsilon$  can change with different  $q$ .
(f)  $\epsilon = 4, |\pi| = 10$ .
(j)  $\epsilon = 4, |\pi| = 100$ .
(g)  $\epsilon = 8,|\pi | = 10$
(k)  $\epsilon = 8, |\pi| = 100$ .
(h)  $\epsilon = 16, |\pi| = 10$ .
(1)  $\epsilon = 16, |\pi| = 100$ .
M Proofs
Throughout the proofs we make some of our notation more succinct for convenience. For a probability distribution  $\omega$  we write  $\omega(E) = \mathbb{P}_{\omega}[E]$ , and rewrite  $\mathcal{B}_{\kappa}(\mu, \nu) = \sup \{\mathbb{P}_{\mu}[E] : E \text{ s.t. } \mathbb{P}_{\nu}[E] \leq \kappa\}$  as  $\sup_{\nu(E) \leq \kappa} \mu(E)$ . Given a distribution  $\omega$  and function  $\phi$  taking values in  $[0,1]$  we also write  $\omega(\phi) = \mathbb{E}_{X \sim \omega}[\phi(X)]$ .
M.1 Proof of Theorem 2
We say that a pair of distributions  $(\mu, \nu)$  is testable if for all  $\kappa \in [0,1]$  we have
$$
\inf  _ {\nu (\phi) \leq \kappa} (1 - \mu (\phi)) = \inf  _ {\nu (E) \leq \kappa} (1 - \mu (E)),
$$
where the infimum on the left is over all  $[0,1]$ -valued measurable functions and the one on the right is over measurable events (i.e.  $\{0,1\}$ -valued functions). The Neyman-Pearson lemma (see e.g. Lehmann & Romano (2005)) implies that this condition is satisfied whenever the statistical hypothesis problem of distinguishing between  $\mu$  and  $\nu$  admits a uniformly most powerful test. For example, this is the case for distributions on  $\mathbb{R}^d$  where the density ratio  $\mu/\nu$  is a continuous function.
Theorem 7 (Formal version of Theorem 2). Fix  $\pi$  and  $\rho$ . Suppose that for every fixed dataset  $D_{-}$  there exists a distribution  $\mu_{D_{-}}$  such that  $\sup_{z\in supp(\pi)}\mathcal{B}_{\kappa}(\mu_{D_z},\nu_{D_-})\leq \mathcal{B}_{\kappa}(\mu_{D_+},\nu_{D_-})$  for all  $\kappa \in [0,1]$ . If the pair  $(\mu ,\nu)$  is testable, then  $M$  is  $(\eta ,\gamma)$ -ReRo with
$$
\gamma = \sup_{D_{-}}\sup_{\nu_{D_{-}}(E)\leq \kappa_{\pi ,\rho}(\eta)}\mu_{D_{-}}(E)  .
$$
The following lemma from Dong et al. (2019) will be useful.
Lemma 8. For any  $\mu$  and  $\nu$ , the function  $\kappa \mapsto \inf_{\nu(\phi) \leq \kappa}(1 - \mu(\phi))$  is convex in  $[0,1]$ .
Lemma 9. For any testable pair  $(\mu, \nu)$ , the function  $\kappa \mapsto \sup_{\nu(E) \leq \kappa} \mu(E)$  is concave.
Proof. By the testability assumption we have
$$
\begin{array}{l} \sup  _ {\nu (E) \leq \kappa} \mu (E) = \sup  _ {\nu (E) \leq \kappa} \mu (E) \\ = \sup  _ {\nu (E) \leq \kappa} (1 - \mu (\bar {E})) \\ = 1 - \inf_{\nu (E)\leq \kappa}\mu (\bar{E}) \\ = 1 - \inf_{\nu (E)\leq \kappa}(1 - \mu (E)) \\ = 1 - \inf  _ {\nu (\phi) \leq \kappa} \left(1 - \mu (\phi)\right). \\ \end{array}
$$
Concavity now follows from Lemma 8.
Proof of Theorem 7. Fix  $D_{-}$  and let  $\kappa = \kappa_{\pi, \rho}(\eta)$  throughout. Let also  $\nu = \nu_{D_{-}}$ ,  $\mu_z = \mu_{D_z}$ ,  $\nu^* = \nu_{D_-}^*$  and  $\mu^* = \mu_{D_-}^*$ .
Expanding the probability of successful reconstruction, we get:
$$
\begin{array}{l} \mathbb {P} _ {Z \sim \pi , W \sim M (D \cdot \cup \{Z \})} [ \rho (Z, R (W)) \leq \eta ] = \mathbb {E} _ {Z \sim \pi} \mathbb {P} _ {W \sim M (D \cdot \cup \{Z \})} [ \rho (Z, R (W)) \leq \eta ] \\ = \mathbb {E} _ {Z \sim \pi} \mathbb {E} _ {W \sim M (D _ {-} \cup \{Z \})} \mathbb {I} [ \rho (Z, R (W)) \leq \eta ] \\ = \mathbb {E} _ {Z \sim \pi} \mathbb {E} _ {W \sim \mu_ {Z}} \mathbb {I} [ \rho (Z, R (W)) \leq \eta ] \\ = \mathbb {E} _ {Z \sim \pi} \mathbb {E} _ {W \sim \nu} \left[ \frac {\mu_ {Z} (W)}{\nu (W)} \mathbb {I} [ \rho (Z, R (w)) \leq \eta ] \right]. \\ \end{array}
$$
Now fix  $z \in \operatorname{supp}(\pi)$  and let  $\kappa_z = \mathbb{P}_{W \sim \nu}[\rho(z, R(W)) \leq \eta]$ . Using the assumption on  $\mu^*$  we get:
$$
\begin{array}{l} \mathbb {E} _ {W \sim \nu} \left[ \frac {\mu_ {z} (W)}{\nu (W)} \mathbb {I} [ \rho (z, R (w)) \leq \eta ] \right] \leq \sup  _ {\nu (E) \leq \kappa_ {z}} \mathbb {E} _ {W \sim \nu} \left[ \frac {\mu_ {z} (W)}{\nu (W)} \mathbb {I} [ W \in E ] \right] \quad (\text {B y}) \\ = \sup  _ {\nu (E) \leq \kappa_ {z}} \mathbb {E} _ {W \sim \mu_ {z}} \left[ \mathbb {I} [ W \in E ] \right] \\ = \sup  _ {\nu (E) \leq \kappa_ {z}} \mu_ {z} (E) \\ \leq \sup  _ {\nu^ {*} (E) \leq \kappa_ {z}} \mu^ {*} (E). \quad (\text {B y} \\ \end{array}
$$
Finally, using Lemma 9 and Jensen's inequality on the following gives the result:
$$
\begin{array}{l} \mathbb {E} _ {Z \sim \pi} [ \kappa_ {Z} ] = \mathbb {E} _ {Z \sim \pi} \mathbb {P} _ {W \sim \nu} [ \rho (Z, R (W)) \leq \eta ] \\ = \mathbb {E} _ {W \sim \nu} \mathbb {P} _ {Z \sim \pi} [ \rho (Z, R (W)) \leq \eta ] \\ \leq \mathbb {E} _ {W \sim \nu} \kappa \\ = \kappa . \\ \end{array}
$$
M.2 Proof of Corollary 4
Here we prove Corollary 4. We will use the following shorthand notation for convenience:  $\mu = \mathcal{N}(B(T,q),\sigma^2 I)$  and  $\nu = \mathcal{N}(0,\sigma^2 I)$ . To prove our result, we use the notion of  $TV_{a}$ .
Definition 10 (Mahloujifar et al. (2022)). For two probability distributions  $\omega_{1}(\cdot)$  and  $\omega_{2}(\cdot)$ ,  $TV_{a}$  is defined as
$$
T V _ {a} \left(\omega_ {1}, \omega_ {2}\right) = \int \left| \omega_ {1} (x) - a \cdot \omega_ {2} (x) \right| d x.
$$
Now we state the following lemma borrowed from Mahloujifar et al. (2022).
Lemma 11 (Theorem 6 in Mahloujifar et al. (2022)). Let  $\nu_{D}$ ,  $\mu_{D_z}$  be the output distribution of  $DP$ -SGD applied to  $D_{-}$  and  $D_{z}$  respectively, with noise multiplier  $\sigma$ , sampling rate  $q$ . Then we have
$$
T V _ {a} \left(\nu_ {D _ {-}}, \mu_ {D _ {z}}\right) \leq T V _ {a} (\nu , \mu).
$$
Now, we state the following lemma that connects  $TV_{a}$  to blow-up function.
Lemma 12 (Lemma 21 in Zhu et al. (2022).). For any pair of distributions  $\omega_{1},\omega_{2}$  we have
$$
\sup  _ {\omega_ {1} (E) \leq \kappa} \omega_ {2} (E) = \inf  _ {a > 1} \min  \left\{0, a \cdot \kappa + \frac {T V _ {a} \left(\omega_ {1} , \omega_ {2}\right) + 1 - a}{2}, \frac {2 \kappa + T V _ {a} \left(\omega_ {1} , \omega_ {2}\right) + a - 1}{2 a} \right\}
$$
Since  $TV_{a}(\nu_{D_{z}}, \mu_{D_{z}})$  is bounded by  $TV_{a}(\nu, \mu)$  for all  $a$ , therefore we have
$$
\sup_{\nu_{D_{z}}(E)\leq \kappa}\mu_{D_{z}}(E)\leq \sup_{\nu (E)\leq \kappa}\mu (E)  .
$$
M.3 Proof of Proposition 6
Recall  $\kappa = \sup_{z_0\in \mathcal{Z}}\mathbb{P}_{Z\sim \pi}[\rho (Z,z_0)\leq \eta ]$  and  $\hat{\kappa} = \sup_{z_0\in \mathcal{Z}}\mathbb{P}_{Z\sim \hat{\pi}}[\rho (Z,z_0)\leq \eta ]$ . Let  $\kappa_z = \mathbb{P}_{Z\sim \pi}[\rho (Z,z)\leq \eta ]$  and  $\hat{\kappa}_z = \mathbb{P}_{Z\sim \hat{\pi}}[\rho (Z,z)\leq \eta ]$ . Note  $\hat{\kappa}_z$  is the sum of  $N$  i.i.d. Bernoulli random variables and  $\mathbb{E}_{\hat{\pi}}[\hat{\kappa}_z] = \kappa_z$ . Then, using a multiplicative Chernoff bound, we see that for a fixed  $z$  the following holds with probability at least  $1 - e^{-N\tau_z^2\kappa /2}$ :
$$
\kappa_ {z} \leq \frac {\hat {\kappa} _ {z}}{1 - \tau}.
$$
Applying this to  $z^{*} = \arg \sup_{z_{0}\in \mathcal{Z}}\mathbb{P}_{Z\sim \pi}[\rho (Z,z_{0})\leq \eta ]$  we get that the following holds with probability at least  $1 - e^{-N\tau^2\kappa /2}$ :
$$
\kappa = \kappa_ {z ^ {*}} \leq \frac {\hat {\kappa} _ {z ^ {*}}}{1 - \tau} \leq \frac {\hat {\kappa}}{1 - \tau}.
$$
M.4 Proof of Proposition 5
Let  $z = \frac{(r_{N'}' + r_{N'-1}')}{2}$ . Let  $E_1$  be the event that  $|\mathbb{P}[r > z] - \kappa| \geq \tau$ . By applying Chernoff-Hoefding bound we have  $\mathbb{P}[E_1] \leq 2e^{-2N\tau^2}$ . Now note that since  $\mu$  is a Gaussian mixture, we can write  $\mu = \sum_{i \in [2^T]} a_i \mu_i$  where each  $\mu_i$  is a Gaussian  $\mathcal{N}(c_i, \sigma)$  where  $|c_i|_2 \leq \sqrt{T}$ . Now let  $r_i = \mu_i(W) / \nu(W)$ . By holder, we have  $\mathbb{E}[r^2] \leq \sum a_i \mathbb{E}[r_i^2]$ . We also now that  $\mathbb{E}[r_i^2] \leq e^T$ , therefore,  $\mathbb{E}[r^2] \leq e^T$ . Now let  $E_2$  be the event that  $|\mathbb{E}[r \cdot I(r > z)] - \gamma'| \geq \tau$ . Since the second moment of  $r$  is bounded, the probability of  $E_2$  goes to zero as  $N$  increases. Therefore, almost surely we have
$$
\sup_{\nu (E)\leq \kappa -\tau}\mu (E) - \tau \leq \lim_{N\to \infty}\gamma^{\prime}\leq \sup_{\nu (E)\leq \kappa +\tau}\mu (E) + \tau .
$$
Now by pushing  $\tau$  to 0 and using the fact that  $\mu$  and  $\nu$  are smooth we have
$$
\lim_{N\to \infty}\gamma^{\prime} = \lim_{\tau \to 0}\sup_{\nu (E)\leq \kappa +\tau}\mu (E) + \tau = \sup_{\nu (E)\leq \kappa}\mu (E).
$$
(b)  $\epsilon = 1, |\pi| = 10$ .
(c)  $\epsilon = 1, |\pi| = 100$ .
(d)  $\epsilon = 2,|\pi | = 2$
(e)  $\epsilon = 2, |\pi| = 10$ .
(f)  $\epsilon = 2, |\pi| = 100$ .
(g)  $\epsilon = 4, |\pi| = 2$ .
(h)  $\epsilon = 4, |\pi| = 10$ .
(i)  $\epsilon = 4, |\pi| = 100$ .
(j)  $\epsilon = 8, |\pi| = 2$ .
(k)  $\epsilon = 8, |\pi| = 10$ .
(1)  $\epsilon = 8, |\pi| = 100$ .
(m)  $\epsilon = 16, |\pi| = 2$ .
(p)  $\epsilon = 32, |\pi| = 2$ .
Figure 15: How the upper bound changes as a function of  $q$  and  $\sigma$  at a fixed value of  $\epsilon$  and prior size,  $|\pi|$ , and setting  $T = 100$ . The probability of a successful reconstruction can vary widely with different values of  $q$ . For example, at  $\epsilon = 32$  and  $|\pi| = 100$ , at  $q = 0.01$  the upper bound is 0.4 and at  $q = 0.99$  it is 1. Note that the color ranges are independent across subfigures.
(n)  $\epsilon = 16, |\pi| = 10$ .
(q)  $\epsilon = 32, |\pi| = 10$ .
(o)  $\epsilon = 16, |\pi| = 100$ .
(r)  $\epsilon = 32, |\pi| = 100$ .