SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Going forward in our investigation we will assume the adversary has access to more "reliable" side knowledge: their prior is a uniform distribution over a finite set of candidate points  $\{z_1, \ldots, z_n\}$ , one of which corresponds to the true target point. This setting is favorable towards the adversary: the points in the prior represent a shortlist of candidates the adversary managed to narrow down using side knowledge about the target. In this case it is also reasonable to use the error function  $\rho(z, z') = \mathbb{I}[z \neq z']$  since the adversary's goal is to identify which point from the prior is the target. As DP assumes the adversary knows (and can influence) all but one record in the training set, the assumption that the adversary has prior knowledge about the target is aligned with the DP threat model. The main distinction between membership inference and reconstruction with a uniform prior is that in the former the adversary (implicitly) managed to narrow down the target point to two choices, while in the latter they managed to narrow down the target to  $n$  choices. This enables us to smoothly interpolate between the exceedingly strong DP threat model of membership inference (where the goal is to infer a single bit) and a relaxed setting where the adversary's side knowledge is less refined: here  $n$  controls the amount of prior knowledge the adversary is privy to, and requires the adversary to infer  $\log(n)$  bits to achieve a successful reconstruction.
The discrete prior setup, we argue, provides a better alignment between the success of reconstruction attacks and what actually constitutes privacy leakage. In particular, it allows us to move away from (approximate) verbatim reconstruction as modelled, e.g., by an  $\ell_2$  reconstruction criteria success, and model more interesting situations. For example, if the target image contains a car, an attacker might be interested in the digits of the license plate, not the pixels of the image of the car and its background. Thus, if the license plate contains 4 digits, the attacker's goal is to determine which of the possible 10,000 combinations was present in the in the training image.
In Appendix I, we also conduct experiments where the adversary has less background knowledge about the target point, and so the prior probability of reconstruction is extremely small (e.g.  $\mathcal{O}\big((1 / 256)^{32\times 32\times 3}\big)$  for a CIFAR-10 image).
3.2 Reconstruction Robustness of DP-SGD
ReRo bounds from blow-up functions We now state a novel reconstruction robustness bound that, instead of using the (R)DP guarantees of the mechanism, is directly expressed in terms of its output distributions. The bound depends on the blow-up function between two distributions  $\mu$  and  $\nu$ :
$$
\mathcal {B} _ {\kappa} (\mu , \nu) = \sup  \left\{\mathbb {P} _ {\mu} [ E ]: E \text {s . t .} \mathbb {P} _ {\nu} [ E ] \leq \kappa \right\}. \tag {3}
$$
In particular, let  $\nu_{D_{-}} = M(D_{-})$  denote the output distribution of the mechanism on a fixed dataset  $D_{-}$  and  $\mu_{D_z} = M(D_{-}\cup \{z\})$  for any potential target  $z$ . Then we have the following (see Appendix M.1 for the full statement).
Theorem 2 (Informal). Fix  $\pi$  and  $\rho$ . Suppose that for every fixed dataset  $D_{-}$  there exists a pair of distributions  $\mu_{D_{-}}^{*}, \nu_{D_{-}}^{*}$  such that  $\sup_{z \in supp(\pi)} \mathcal{B}_{\kappa}(\mu_{D_z}, \nu_{D_{-}}) \leq \mathcal{B}_{\kappa}(\mu_{D_{-}}^{*}, \nu_{D_{-}}^{*})$  for all  $\kappa \in [0,1]$ . Then  $M$  is  $(\eta, \gamma)$ -ReRo with  $\gamma = \sup_{D_{-}} \mathcal{B}_{\kappa_{\pi, \rho}(\eta)}(\mu_{D_{-}}^{*}, \nu_{D_{-}}^{*})$ .
This result basically says that the probability of successful reconstruction for an adversary that does observe the output of the mechanism can be bounded by the maximum probability under  $\mu_{D_{-}}^{*}$  over all events that have probability  $\kappa_{\pi, \rho}(\eta)$  under  $\nu_{D_{-}}^{*}$ , when we take the worst-case setting over all fixed datasets  $D_{-}$  and all target points  $z$  in the support of the prior. If  $M$  satisfies  $(\epsilon, 0)$ -DP (i.e.  $(\infty, \epsilon)$ -RDP), then  $\mathbb{P}_{\nu_{D_{-}}}[E] \leq e^{\epsilon} \mathbb{P}_{\mu_{D_{z}}}[E]$  for any event  $E$ , in which case Theorem 2 gives with  $\gamma = \kappa_{\pi, \rho}(\eta) e^{\epsilon}$ . This recovers the case  $\alpha = \infty$  of the bound from Balle et al. (2022) stated above.
Remark 3. The blow-up function is tightly related to the notion of trade-off function defined in Dong et al. (2019). Precisely, the trade-off function between two probability distributions  $\mu$  and  $\nu$  is defined as  $\mathcal{T}_{\kappa}(\mu, \nu) = \inf \{\mathbb{P}_{\mu}[E] : E \text{ s.t. } \mathbb{P}_{\nu}[E] \leq \kappa\}$ . The trade-off function is usually introduced in terms of Type I and Type II errors; it defines the smallest Type II error achievable given constraint on the maximum Type I error, and we have the following relationship:  $\mathcal{T}_{\kappa}(\mu, \nu) = 1 - \mathcal{B}_{\kappa}(\mu, \nu)$ .
ReRo for DP-SGD Next we apply this bound directly to DP-SGD without an intermediate appeal to its RDP guarantees; in Section 4.1 we will empirically demonstrate that in practice this new bound is vastly superior to the bound in Balle et al. (2022).
Let  $M$  be the mechanism that outputs all the intermediate gradients produced by running DP-SGD for  $T$  steps with noise multiplier  $\sigma$  and sampling rate  $q$ . Our analysis relies on the observation from Mahloujifar et al. (2022) that comparing the output distributions of  $M$  on two neighbouring datasets can be reduced to comparing the  $T$ -dimensional Gaussian distribution  $\nu_{T,\sigma} = \mathcal{N}(0,\sigma^2 I)$  with the Gaussian mixture  $\mu_{T,\sigma,q} = \sum_{w \in \{0,1\}^T} \mathbb{P}[B(q,T) = w] \mathcal{N}(w,\sigma^2 I)$ , where  $B(q,T)$  is a random binary vector obtained by sampling each coordinate independently from a Bernoulli distribution with success probability  $q$ .
Corollary 4. For any prior  $\pi$  and error function  $\rho$ ,  $M$  is  $(\eta, \gamma)$ -ReRo with  $\gamma = \mathcal{B}_{\kappa_{\pi, \rho}(\eta)}(\mu_{T, \sigma, q}, \nu_{T, \sigma})$ .
We defer proofs and discussion to Appendix M. We note that the concurrent work of Guo et al. (2022b) also provides an upper bound on the success of training data reconstruction under DP. We compare this to our own upper bound in Appendix H.
3.3 Upper bound estimation via reconstruction robustness
To estimate the bound in Corollary 4 in practice we first estimate the baseline probability  $\kappa = \kappa_{\pi, \rho}(\eta)$  and then the blow-up  $\gamma = \mathcal{B}_{\kappa}(\mu_{T,\sigma,q}, \nu_{T,\sigma})$ .
Estimating  $\kappa$ . All our experiments deal with a uniform discrete prior  $\pi = \frac{1}{n}\sum_{i}\mathbf{1}_{z_i}$  over a finite set of points  $\{z_1,\ldots ,z_n\}$ . In this case, to compute  $\kappa$  it suffices to find the  $\rho$ -ball of radius  $\eta$  containing the largest number of points from the prior. If one is further interested in exact reconstruction (i.e.  $\rho (z,z^{\prime}) = \mathbb{I}[z\neq z^{\prime}]$  and  $\eta < 1$ ) then we immediately have  $\kappa = 1 / n$ . Although this already covers the setting from our experiments, we show in Appendix L that more generally it is possible to estimate  $\kappa$  only using samples from the prior. We use this to calculate  $\kappa$  for some non-uniform settings where we relax the condition to be on a close reconstruction instead of exact reconstruction.
Estimating  $\gamma$ . Note that it is easy to both sample and evaluate the density of  $\mu = \mu_{T,\sigma,q}$  and  $\nu = \nu_{T,\sigma}$ . Any time these two operations are feasible we can estimate  $\mathcal{B}_{\kappa}(\mu, \nu)$  using a non-parametric approach to find the event  $E$  that maximizes Equation (3). The key idea is to observe that, as long as  $\mu/\nu < \infty$ , then a change of measure gives  $\mathbb{P}_{\mu}[E] = \mathbb{E}_{W \sim \nu}\left[\frac{\mu(W)}{\nu(W)} \mathbb{I}[W \in E]\right]$ . For fixed  $E$  this expectation can be approximated using samples from  $\nu$ . Furthermore, since we are interested in the event of probability less than  $\kappa$  under  $\nu$  that maximizes the expectation, we can take  $N$  samples from  $\nu$  and keep the  $\kappa N$  samples that maximize the ratio  $\mu/\nu$  to obtain a suitable candidate for the maximum event. This motivates the Monte-Carlo approximation of  $\gamma$  in Algorithm 1.
Proposition 5. For  $\mu = \mu_{T,\sigma,q}$  and  $\nu = \nu_{T,\sigma}$  we have  $\lim_{N\to \infty}\hat{\gamma} = \mathcal{B}_{\kappa}(\mu, \nu)$  almost surely.
Algorithm 1 Estimating  $\gamma$
Input: Prior probability  $\kappa$ , number of samples  $N$ , sampling and density evaluation access to  $\nu$  and  $\mu$ .
Output: Estimate  $\hat{\gamma}$  of reconstruction robustness probability.
Sample  $w_{1},\ldots ,w_{N}$  independently from  $\nu$  Calculate the ratios  $r_i = \mu (w_i) / \nu (w_i)$
Sort the ratios in decreasing order:  $r_1' \geq \dots \geq r_N'$
Let  $N^{\prime} = \lceil \kappa N\rceil$  and return  $\hat{\gamma} = \frac{1}{N^{\prime}}\sum_{i = 1}^{N^{\prime}}r_{i}^{\prime}$
Algorithm 2 Prior-aware attack
Input: Discrete prior  $\pi = \{z_1,\dots ,z_n\}$  Model parameters  $\{\theta_{1},\theta_{1},\ldots ,\theta_{T}\}$  ,Privatized gradients (with known gradients subtracted)  $\{\bar{g}_1,\dots ,\bar{g}_T\}$  Observations:  $\mathcal{O}\gets \{\}$
Output: Reconstruction guess  $\hat{z} \in \pi$
for  $i\in [1,2,\ldots ,n]$  do
$$
\mathcal {O} [ i ] \leftarrow \sum_ {t = 1} ^ {T} \left\langle \operatorname {c l i p} _ {C} \left(\nabla_ {\theta} \ell \left(\theta_ {t}, z _ {i}\right)\right), \bar {g} _ {t} \right\rangle
$$
$i\gets \arg \max \mathcal{O}$
return  $\hat{z}\gets \pi [i]$
4This is satisfied by  $\mu = \mu_{T,\sigma ,q}$  and  $\nu = \nu_{T,\sigma}$
Estimation guarantees Although Proposition 4 is only asymptotic, we can provide confidence intervals around this estimate using two types of inequalities. Algorithm 1 has two points of error, the gap between the empirical quantile and the population quantile, and the error from mean estimation  $\hat{\gamma}$ . The first source of error can be bounded using the DKW inequality (Dvoretzky et al., 1956) which provides a uniform concentration for each quantile. In particular, we can show that the error of quantile estimation is at most  $2e^{-2n\eta^2}$ , where  $n$  is the number of samples and  $\eta$  is the error in calculation of the quantile. Specifically, with a million samples, we can make sure that with probability 0.999 the error of quantile estimation is less than 0.002, and we can make this smaller by increasing the number of samples. We can account for the second source of error with Bennett's inequality, that leverages the bounded variance of the estimate. In all our bounds, we can show that the error of this part is also less than 0.01 with probability 0.999. We provide an analysis on the computational cost of estimating  $\gamma$  in Appendix A.
3.4 Lower bound estimation via a prior-aware attack
We now present a prior-aware attack whose success can be directly compared to the reconstruction upper bound from Corollary 4.
In addition to the uniform prior  $\pi = \frac{1}{n}\sum_{i}\mathbf{1}_{z_i}$  from which we assume the target is sampled from, our prior-aware attack has access to the same information as the gradient-based attack from Section 2: all the privatized gradients (and therefore, intermediate models) produced by DP-SGD, and knowledge of the fixed dataset  $D_{-}$  that can be used to remove all the known datapoints from the privatized gradients. The attack is given in Algorithm 2. For simplicity, in this section we present and experiment with the version of the attack corresponding to the full-batch setting in DP-SGD (i.e.  $q = 1$ ). We present an extension of the algorithm to the mini-batch setting in Section 4.3.
The rationale for the attack is as follows. Suppose, for simplicity, that all the gradients are clipped so they have norm exactly  $C$ . If  $z^{*}$  is the target sampled from the prior, then  $\bar{g}_t \sim \mathrm{clip}_C(\nabla_\theta \ell(\theta_t, z^*)) + \mathcal{N}(0, C^2 \sigma^2 I)$ . Then the inner products  $\langle \mathrm{clip}_C(\nabla_\theta \ell(\theta_t, z_i)), \bar{g}_t \rangle$  follow a distribution  $\mathcal{N}(C^2, C^4 \sigma^2)$  if  $z_i = z^*$  and  $\mathcal{N}(A, C^4 \sigma^2)$  for some  $A < C^2$  otherwise (assuming no two gradients are in the exact same direction). In particular,  $A \approx 0$  if the gradients of the different  $z_i$  in the prior are mutually orthogonal. Thus, finding the  $z_i$  that maximizes this sum of inner products is likely to produce a good guess for the target point. This attack gives a lower bound on the probability of successful reconstruction, which we can compare with the upper bound estimate derived in Section 3.3.
4 Experimental Evaluation
We now evaluate both our upper bounds for reconstruction success and our empirical privacy attacks (which gives us lower bounds on reconstruction success). We show that our attack has a success probability nearly identical to the bound given by our theory. We conclude this section by inspecting how different variations on our threat models change both the upper bound and the optimality of our new attack.
4.1 Attack and Bound Evaluation on CIFAR-10
We now evaluate our upper bound (Corollary 4) and prior-aware reconstruction attack (Algorithm 2) on full-batch DP-SGD, and compare it to the gradient-based (prior-oblivious) attack optimizing Equation (1) and the RDP upper bound obtained from previous work (Equation (2)). To perform our evaluation on CIFAR-10 models with relatively high accuracy despite being trained with DP-SGD, we follow De et al. (2022) and use DP-SGD to fine-tune the last layer of a WideResNet model (Zagoruyko & Komodakis, 2016) pre-trained on ImageNet. In this section and throughout the paper, we use a prior with uniform support over 10 randomly selected points unless we specify the contrary. Further experimental details are deferred to Appendix A.
The results of our evaluation are presented in Figure 2, which reports the upper and lower bounds on the probability of successful reconstruction produced by the four methods at three different values of  $(\epsilon, 10^{-5})$ -DP. The first important observation is that our new ReRo bound for DP-SGD is a significant improvement over the RDP-based bound obtained in Balle et al. (2022), which gives a trivial bound for  $\epsilon \in \{10, 100\}$  in this setting. On the other hand, we also observe that the prior-aware attack substantially improves over the prior-oblivious
gradient-based attack<sup>5</sup>, which Section 2 already demonstrated is stronger than previous model-based attacks – a more thorough investigation of how the gradient-based attack compares to the prior-aware attack is presented in Appendix D for varying sizes of the prior distribution.
We will use the prior-aware attack exclusively in following experiments due to its superiority over the gradient-based attack. Finally, we observe that our best empirical attack and theoretical bound are very close to each other in all settings. We conclude there are settings where the upper bound is nearly tight and the prior-based attack is nearly optimal.
Now that we have established that our attacks work well on highly accurate large models, our remaining experiments will be performed on MNIST which are significantly more efficient to run.
4.2 Effects of the prior size
We now investigate how the bound and prior-aware attack are affected by the size of the prior in Figure 3. As expected, both the upper and lower bound to the probability that we can correctly infer the target point decreases as the prior size increases. We also observe a widening gap between the upper bound and attack for larger values of  $\epsilon$  and prior size. However, this bound is still relatively tight for  $\epsilon < 10$  and a prior size up to  $2^{10}$ . Further experiments regarding how the prior affects reconstruction are given in Appendix E.
4.3 Effect of DP-SGD Hyperparameters
Given that we have developed an attack that is close to optimal in a full-batch setting, we are now ready to inspect the effect DP-SGD hyper-parameters controlling its privacy guarantees have on reconstruction success. First, we observe how the size of mini-batch affects reconstruction success, before measuring how reconstruction success changes with differing hyperparameters at a fixed value of  $\epsilon$ . Further experiments that measure the effect of the clipping norm,  $C$ , are deferred to Appendix K.
Effect of sampling rate  $q$ . All of our experiments so far have been in the full-batch ( $q = 1$ ) setting. We now measure how mini-batching affects both the upper and lower bounds by reducing the data sampling probability  $q$  from 1 to 0.02 (see Appendix A for additional experimental details).
The attack presented in Algorithm 2 assumes that the gradient of  $z^{*}$  is present in every privatized gradient. Since this is no longer true in mini-batch DP-SGD, we design an improved attack by factoring in the knowledge that only a fraction  $q$  of privatized gradients will contain the gradient of the target  $z^{*}$ . This is
Figure 2: Comparison of success of reconstruction attacks against reconstruction upper bounds on CIFAR-10 with test accuracy at  $\epsilon = 1, 10$ , and 100 equal to  $61.88\%$ ,  $89.09\%$ , and  $91.27\%$ , respectively. Note, the best model we could train at  $\epsilon = \infty$  with our experimental settings reaches  $93.91\%$ .
Figure 3: How the size of the prior  $\pi$  affects reconstruction success probability for a range of different values of  $\epsilon$ .
Figure 4: We compare how the attack from Algorithm 2 compares with the improved prior-aware attack when  $q = 0.02$ .
$\langle \mathrm{clip}_C(\nabla_\theta \ell (\theta_t,z_i)),\bar{g}_t\rangle$  across all  $T$  iterations,
To measure the success probability of the gradient-based attack we take the output of the optimization process, find the closest point in the support of the prior, and declare success if that was the actual target. This process is averaged over 10,000 randomly constructed priors and samples from each prior.
(a) Upper bound.
(b) (Improved) Prior-aware attack.
Figure 5: We plot  $\sigma$  against  $q$  at  $(4,10^{-5})$ -DP at a fixed  $T = 100$  and the corresponding upper bound (Figure 5a) and (improved) prior-aware attack (Figure 5b). Reconstruction is less successful at smaller  $q$  in comparison to large  $q$  for the same value of  $\epsilon$ .
and computing the score using only the top  $qT$  values. Pseudo-code for this improved prior-aware attack variant is given in Appendix F.
Results are shown in Figure 4, where we observe the improved attack is fairly tight to the upper bound, and also a large difference between the previous and improved attack.
Effect of DP-SGD hyperparameters at a fixed  $\epsilon$ . In our final experiment, we fix  $\epsilon$  and investigate if and how the reconstruction upper bound and lower bound (through the improved prior-aware attack) change with different DP-SGD hyperparameters. In Figure 5, we fix  $\epsilon = 4$  and run DP-SGD for  $T = 100$  iterations with  $C = 1$  while varying  $q \in [0.01, 0.99]$ . In each setting we tune  $\sigma$  to satisfy  $(4, 10^{-5})$ -DP. For experimental efficiency, we use fewer values of  $q$  for the attack than the theoretical bound. Surprisingly, we find that both the upper bound and lower bound change with different hyperparameter configurations. For example, at  $q = 0.01$  the upper bound on successful reconstruction is  $\approx 0.20$  (and the attack is  $\approx 0.15$ ), while at  $q = 0.99$  it is  $\approx 0.35$  (and the attack is  $\approx 0.32$ ). Generally, an increase in  $T$  or  $q$  increases the upper bound. This suggests that an  $(\epsilon, \delta)$ -DP guarantee alone is not able to fully specify the probability that a reconstruction attack is successful. We conduct a more extensive evaluation for different  $\epsilon$  and prior sizes in Appendix K.2.
In this work, we investigated training data reconstruction bounds on DP-SGD. We developed upper bounds on the success that an adversary can reconstruct a training point  $z^{*}$  under DP-SGD. In contrast to prior work that develop reconstruction upper bounds based on DP or RDP analysis of DP-SGD, our upper bounds are directly based on the parameters of the algorithm. We also developed new reconstruction attacks, specifically against DP-SGD, that obtain lower bounds on the probability of successful reconstruction, and observe they are close to our upper bounds. Our experiments show that both our upper and lower bounds are superior to previously proposed bounds for reconstruction in DP-SGD. Our investigations further showed that the  $\epsilon$  parameter in DP (or RDP) cannot solely explain robustness to reconstruction attacks. In other words, one can have the same value of  $\epsilon$  for two different algorithms while achieving stronger robustness against reconstruction attacks in one over the other. This suggests that a more fine-grained analysis of algorithms against specific privacy attacks can lead to superior privacy guarantees and also opens up the possibility of better hyperparameter selection for specific privacy concerns.
The authors would like to thank Leonard Berrada and Taylan Cemgil for their thoughtful feedback on an earlier version of this work.
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.