SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Song, S., Chaudhuri, K., and Sarwate, A. D. Stochastic gradient descent with differentially private updates. In 2013 IEEE global conference on signal and information processing, pp. 245-248. IEEE, 2013.
Stock, P., Shilov, I., Mironov, I., and Sablayrolles, A. Defending against reconstruction attacks with rényi differential privacy. arXiv preprint arXiv:2202.07623, 2022.
Tirumala, K., Markosyan, A. H., Zettlemoyer, L., and Aghajanyan, A. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv preprint arXiv:2205.10770, 2022.
Tramér, F. and Boneh, D. Differentially private learning needs better features (or much more data). In International Conference on Learning Representations (ICLR), 2021.
Tramér, F., Terzis, A., Steinke, T., Song, S., Jagielski, M., and Carlini, N. Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202.12219, 2022.
Wang, Z., Song, M., Zhang, Z., Song, Y., Wang, Q., and Qi, H. Beyond inferring class representatives: User-level privacy leakage from federated learning. In IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 2512-2520. IEEE, 2019.
Ye, J. and Shokri, R. Differentially private learning needs hidden state (or much faster convergence). arXiv preprint arXiv:2203.05363, 2022.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF), pp. 268-282. IEEE, 2018.
Yin, H., Mallya, A., Vahdat, A., Alvarez, J. M., Kautz, J., and Molchanov, P. See through gradients: Image batch recovery via gradconversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346, 2021.
Yu, D., Kamath, G., Kulkarni, J., Yin, J., Liu, T.-Y., and Zhang, H. Per-instance privacy accounting for differentially private stochastic gradient descent. arXiv preprint arXiv:2206.02617, 2022.
Zagoruyko, S. and Komodakis, N. Wide residual networks. In *British Machine Vision Conference* 2016. British Machine Vision Association, 2016.
Zhao, B., Mopuri, K. R., and Bilen, H. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020.
Zhu, L., Liu, Z., and Han, S. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019.
Zhu, Y., Dong, J., and Wang, Y.-X. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pp. 4782-4817. PMLR, 2022.
Table 1: Hyperparameter settings for each experiment. The set of examples that make up the prior  $(\pi)$ , including the target  $(z^{*})$ , and the other examples in the training set  $(D_{-})$  are always drawn from the same data distribution, except for the experiment performed in Figure 9.
Dataset	Experiment	Clipping norm C	Sampling probability q	Update steps T	Model architecture θ	Training dataset size (|Dt| + 1)	Prior size
CIFAR-10	Figure 1b, Figure 6b, Figure 12b	1	1	100	WRN-28-10	500	-
Figure 2	1	1	100	WRN-28-10	500	10
Figure 1a, Figure 6a, Figure 12a	0.1	1	100	MLP (784 → 10 → 10)	1,000	-
Figure 3	0.1	1	100	MLP (784 → 10 → 10)	1,000	21 - 211
Figure 4	0.1	0.02	1,000	MLP (784 → 10 → 10)	500	10
Figure 5	1	0.01-0.99	100	MLP (784 → 10 → 10)	1,000	10
MNIST	Figure 7	0.1	1	100	MLP (784 → 10, 100, 1000 → 10)	1,000	10
Figure 8	0.1	1	100	MLP (784 → 10 → 10)	1,000	21, 23, 27
Figure 9	0.1	1	100	MLP (784 → 10 → 10)	1,000	10
Figure 10	0.1	1	100	MLP (784 → 10 → 10)	5, 129, 1,000	10
Figure 11	1.0	1	-	-	-	10
Figure 13	0.1, 1	1	100	MLP (784 → 10 → 10)	1,000	10
Figure 14	1	0.01-0.99	100	MLP (784 → 10 → 10)	1,000	2, 10, 100
Figure 15	1	0.01-0.99	100	MLP (784 → 10 → 10)	1,000	2, 10, 100
We detail the experimental settings used throughout the paper, and specific hyperparameters used for the various attacks we investigate. The exact configurations for each experiment are given in Table 1. We vary many experimental hyperparameters to investigate their effect on reconstruction, however, the default setting is described next.
For MNIST experiments we use a two layer MLP with hidden width 10 and eLU activations. The attacks we design in this work perform equally well on all common activation functions, however it is well known that the model-based attack (Balle et al., 2022) performs poorly on piece-wise linear activations like ReLU. We set  $|D_{-}| = 999$  (and so the training set size is  $|D_{-} \cup \{z^{*}\}| = 1,000$ ) and train with full-batch DP-SGD for  $T = 100$  steps. For each  $\epsilon$ , we select the learning rate by sweeping over a range of values between 0.001 and 100; we do not use any momentum in optimization. We set  $C = 0.1$ ,  $\delta = 10^{-5}$  and adjust the noise scale  $\sigma$  for a given target  $\epsilon$ . The accuracy of this model is over  $90\%$  for  $\forall \epsilon \geq 10$ , however we emphasize that our experiments on MNIST are meant to primarily investigate the tightness of our reconstruction upper bounds. We set the size of the prior  $\pi$  to ten, meaning the baseline probability of successful reconstruction is  $10\%$ .
For the CIFAR-10 dataset, we use a Wide-ResNet (Zagoruyko & Komodakis, 2016) model with 28 layers and width factor 10 (denoted as WRN-28-10), group normalization, and eLU activations. We align with the set-up of De et al. (2022), who fine-tune a WRN-28-10 model from ImageNet to CIFAR-10. However, because the model-based attack is highly expensive, we only fine-tune the final layer. We set  $|D_{-}| = 499$  (and so the training set size is  $|D_{-} \cup \{z^{*}\}| = 500$ ) and train with full-batch DP-SGD for  $T = 100$  steps; again we sweep over the choice of learning rate for each value of  $\epsilon$ . We set  $C = 1$ ,  $\delta = 10^{-5}$  and adjust the noise scale  $\sigma$  for a given target  $\epsilon$ . The accuracy of this model is over  $89\%$  for  $\forall \epsilon \geq 10$ , which is close to the state-of-the-art results given by De et al. (2022), who achieve  $94.2\%$  with the same fine-tuning setting at  $\epsilon = 8$  (with a substantially larger training set size). Again, we set the size of the prior  $\pi$  to ten, meaning the baseline probability of successful reconstruction is  $10\%$ .
For the gradient-based and model-based attack we generate 1,000 reconstructions and for prior-aware attack experiments we generate 10,000 reconstructions from which we estimate a lower bound for probability of successful reconstruction. That is, for experiments in Section 2 repeat the attack 1,000 times for targets randomly sampled from base dataset (MNIST or CIFAR-10), and for all other experiments we repeat the attack 10,000 times for targets randomly sampled from the prior, which is itself sampled from the base dataset (MNIST or CIFAR-10). We now give experimental details specific to the various attacks used throughout the paper. Note that for attack results, we report  $95\%$  confidence intervals around our lower bound estimate, however, in many cases these intervals are so tight it renders them invisible to the eye.
Model-based attack details. For the model-based attack given by Balle et al. (2022), we train  $40K$  shadow models, and as stated above, construct a test set by training a further 1,000 models on 1,000 different targets (and  $D_{-}$ ) from which we evaluate our reconstructions. We use the same architecture for the RecoNN network and optimization hyperparameters as described in the MNIST and CIFAR-10 experiments in Balle et al. (2022), and refer the interested reader there for details.
(a) Examples of reconstructions on MNIST.
(b) Examples of reconstructions on CIFAR-10.
Figure 6: We give qualitative examples of reconstructions in Figure 6a and Figure 6b for the gradient-based reconstruction attack described in Section 2.
Gradient-based attack details. Our optimization hyperparameters are the same for both MNIST and CIFAR-10. We initialize a  $\hat{z}$  from uniform noise and optimize it with respect to the loss given in Equation (1) for 1M steps of gradient descent with a learning rate of 0.01. We found that the loss occasionally diverges and it is useful to have random restarts of the optimization process; we set the number of random restarts to five. Note we assume that the label of  $z^{*}$  is known to the adversary. This is a standard assumption in training data reconstruction attacks on federated learning, as Zhao et al. (2020) demonstrated the label of the target can be inferred given access to gradients. If we did not make this assumption, we can run the attack be exhaustively searching over all possible labels. For the datasets we consider, this would increase the cost of the attack by a factor of ten. We evaluate the attack using the same 1,000 targets used to evaluate the model-based attack.
Prior-aware attack details. The prior-aware attacks given in Algorithm 2 (and in Algorithm 3) have no specific hyper-parameters that need to be set. As stated, the attack proceeds by summing the inner-product defined in Section 3.4 over all training steps for each sample in the prior and selecting the sample that maximizes this sum as the reconstruction. One practical note is that we found it useful to normalize privatized gradients such that the privatized gradient containing the target will be sampled from a Gaussian with unit mean instead of  $C^2$ , which will be sensitive to choice of  $C$  and can lead to numerical precision issues.
Estimating  $\gamma$  details. As described in Section 3,  $\nu$  is instantiated as  $\mathcal{N}(0,\sigma^2 I)$ , a  $T$ -dimensional isotropic Gaussian distribution with zero mean, and  $\mu$  is given by  $\sum_{w\in \{0,1\}^T}p(w)\mathcal{N}(w,\sigma^2 I)$ , a mixture of  $T$ -dimensional isotropic Gaussian distributions with means in  $\{0,1\}^T$  sampled according to  $B(q,T)$ . Throughout all experiments, we use 1M independent Gaussian samples to compute the estimation of  $\gamma$  given by the procedure in Algorithm 1, and because we use a discrete prior of size  $|\pi|$ , the base probability of reconstruction success,  $\kappa$ , is given as  $1 / |\pi|$ . Estimating  $\gamma$  is cheap; setting  $T = 1$  and using a 2.3 GHz 8-Core Intel Core i9 CPU it takes 0.002s to estimate with 10,000 samples. This increases to 0.065s, 0.196s, and 2.084s with 100,000, 1M, and 10M samples. The estimate sensitivity is also small; for the experimental parameters used in Figure 2 at  $\epsilon = 10$ , over 1,000 different calls to Algorithm 1 the standard deviation of  $\gamma$  estimates is  $< 0.019$  using 10,000 samples, and  $< 0.0017$  using 1M samples.
B Visualization of reconstruction attacks on MNIST and CIFAR-10
In Figure 6, we give a selection of examples for the gradient-based reconstruction attack presented in Section 2 and plotted in Figure 1.
C Does the model size make a difference to the prior-aware attack?
Our results on MNIST and CIFAR-10 suggest that the model size does not impact the tightness of our reconstruction attack (lower bound on probability of success); the MLP model used for MNIST has 7,960 trainable parameters, while the WRN-28-10 model used for CIFAR-10 has 36.5M. We systematically evaluate the impact of the model size on our prior-aware attack by increasing the size of the MLP hidden layer by factors of ten, creating models with 7,960, 79,600, and 796,000 parameters. Results are given in Figure 7, where we observe almost no difference in terms of attack success between the different model sizes.
(a) Width: 10.
(b) Width: 100.
(c) Width: 1000.
Figure 7: Comparison of model sizes on reconstruction by varying the hidden layer width in a two layer MLP.
(a) Prior size is 2.
Figure 8: Comparison of prior-aware and gradient-based attack for different prior sizes.
(b) Prior size is 8.
(c) Prior size is 128.
D Comparing the gradient-based attack with the prior-aware attack
Our experiments have mainly been concerned with measuring how DP affects an adversary's ability to infer which point was included in training, given that they have access to all possible points that could have been included, in the form of a discrete prior. This experimental set-up departs from Figure 1, where we assumed the adversary does not have access to a prior set, and so cannot run the prior-aware attack as described in Algorithm 2. Following on from results in Section 4.1, we transform these gradient-based attack experimental findings into a probability of successful reconstruction by running a post-processing conversion, allowing us to measure how the assumption of adversarial access to the discrete prior affects reconstruction success. We run the post-processing conversion in the following way: Given a target sample  $z^{*}$  and a reconstruction  $\hat{z}$  found through optimizing the gradient based loss in Equation (1), we construct a prior consisting of  $z^{*}$  and  $n - 1$  randomly selected points from the MNIST dataset, where  $n = 10$ . We then measure the  $L_{2}$  distance between  $\hat{z}$  and every point in this constructed prior, and assign reconstruction a success if the smallest distance is with respect to  $z^{*}$ . For each target  $z^{*}$ , we repeat this procedure 1,000 times, with different random selections of size  $n - 1$ , and overall report the average reconstruction success over 1,000 different targets.
This allows us to compare the gradient-based attack (which is prior "unaware") directly to our prior-aware attack. Results are shown in Figure 8, where we vary the size of the prior between 2, 8, and 128. In all cases, we see an order of magnitude difference between the gradient-based and prior-aware attack in terms of reconstruction success. This suggests that if we assume the adversary does not have prior knowledge of the possible set of target points, the minimum value of  $\epsilon$  necessary to protect against reconstruction attacks increases.
E Effects of the threat model and prior distribution on reconstruction
The ability to reconstruct a training data point will naturally depend on the threat model in which the security game is instantiated. So far, we have limited our investigation to align with the standard adversary assumptions in the DP threat model. We have also limited ourselves to a setting where the prior is sampled from the same base distribution as  $D_{-}$ . These choices will change the performance of our attack, which is what we measure next.
Prior type. We measure how the choice of prior affects reconstruction in Figure 9. We train models when the prior is from the same distribution as the rest of the training set (MNIST), and when the prior is sampled random noise. Note, because the target point  $z^{*}$  is included in the prior, this means
Figure 9: Comparison of how the choice of prior,  $\pi$ , affects reconstruction success. The prior is selected from a set of examples sampled from MNIST or uniform noise (that has the same intra-sample distance statistics as the MNIST prior).
we measure how reconstruction success changes when we change the distribution the target was sampled from. One may expect that the choice of prior to make a difference to reconstruction success if the attack relies on distinguishability between  $D_{-}$  and  $z^{*}$  with respect to some function operating on points and model parameters (e.g. the difference in loss between points in  $D_{-}$  and  $z^{*}$ ). However, we see that there is little difference between the two; both are close to the upper bound.
On reflection, this is expected as our objective is simply the sum of samples from a Gaussian, and so the choice of prior may impact our probability of correct inference if this choice affects the probability that a point will be clipped, or if points in the prior have correlated gradients. We explore how different values of clipping,  $C$ , can change reconstruction success probability in Appendix K.
Knowledge of batch gradients. The DP threat model assumes the adversary has knowledge of the gradients of all samples other than the target  $z^{*}$ . Here, we measure how important this assumption is to our attack. We compare the prior-aware attack (which maximizes  $\sum_{t=1}^{T} \langle \mathrm{clip}_C(\nabla_{\theta_t} \ell(z_i)), \bar{g}_t \rangle$ ) against the attack that selects the  $z_i$  maximizing  $\sum_{t=1}^{T} \langle \mathrm{clip}_C(\nabla_{\theta_t} \ell(z_i)), g_t \rangle$ , where the adversary does not subtract the known gradients from the objective.
In Figure 10, we compare, in a full-batch setting, when  $|D_{-}|$  is small (set to 4), and see the attack does perform worse when we do not deduct known gradients. However, the effect is more pronounced as  $|D_{-}|$  becomes larger, the attack completely fails when setting it to 128. This is somewhat expected, as with a larger number of samples in a batch it is highly likely there are gradients correlated with the  $z^{*}$  target gradient, masking out its individual contribution and introducing noise into the attack objective.
F Improved prior-aware attack algorithm
As explained in Section 4.3, the prior-aware attack in Algorithm 2 does not account for the variance introduced into the attack objective in mini-batch DP-SGD, and so we design a more efficient attack specifically for the mini-batch setting. We give the pseudo-code for this improved prior-aware attack in Algorithm 3.
Figure 10: In line with the DP threat model, our attack in Algorithm 2 assumes the adversary can subtract known gradients from the privatized gradient. We measure what effect removing this assumption has on reconstruction success probability. When the size of the training set is small, removing this assumption has a minor effect, while reconstruction success drops to random with a larger training set size.
Algorithm 3 Improved prior-aware attack
Input: Discrete prior  $\pi = \{z_1,\dots ,z_n\}$  , Model parameters  $\{\theta_{1},\theta_{1},\ldots ,\theta_{T}\}$  , Privatized gradients (with known gradients subtracted)  $\{\bar{g}_1,\dots ,\bar{g}_T\}$  , sampling probability  $q$  , function that takes the top  $qT$  values from a set of observed gradients  $top_{qT}$    
Observations:  $\mathcal{O}\gets \{\}$    
Output: Reconstruction guess  $\hat{z}\in \pi$    
for  $i\in [1,2,\dots ,n]$  do   
 $\mathcal{R}\leftarrow \{\}$    
for  $t\in [1,2,\dots ,T]$  do   
 $\mathcal{R}[t]\leftarrow \langle \mathrm{clip}_C(\nabla_{\theta_t}\ell (\theta_t,z_i)),\bar{g}_t\rangle$    
end for   
 $\mathcal{R}\gets top_{qT}(\mathcal{R})$ $\mathcal{O}[i]\gets sum(\mathcal{R})$    
end for   
 $\hat{i}\gets \arg \max \mathcal{O}$    
return  $\hat{z}\gets \pi [\hat{i} ]$
G Alternative variant of the prior-aware attack
Here, we state an alternative attack that uses the log-likelihood to find out which point in the prior set is used for training. Assume we have  $T$  steps with clipping threshold  $C = 1$ , noise  $\sigma$ , and the sampling rate is  $q$ .
Let  $\bar{g}_1, \ldots, \bar{g}_T$  be the observed gradients minus the gradient of the examples that are known to be in the batch and let  $l_1, \ldots, l_T$  be the  $\ell_2$  norms of these gradients.
For each example  $z$  in the prior set let  $g_1^z, \ldots, g_T^z$  be the clipped gradient of the example on the intermediate model. Also let  $l_1^z, \ldots, l_T^z$  be the  $\ell_2$  norms of  $(\bar{g}_1 - g_1^z), \ldots, (\bar{g}_T - g_T^z)$ .
Now we describe the optimal attack based on  $l_{i}^{z}$ . For each example  $z$ , calculate the following:  $s_{z} = \sum_{i\in [T]}\ln (1 - q + qe^{\frac{-(l_{i}^{z})^{2} + l_{i}^{2}}{2\sigma^{2}}})$ . It is easy to observe that this is the log probability of outputting the steps conditioned on  $z$  being used in the training set. Then since the prior is uniform over the prior set, we can choose the  $z$  with maximum  $s_{z}$  and report that as the example in the batch.
In fact, this attack could be extended to the non-uniform prior by choosing the example that maximizes  $s_z \cdot p_z$ , where  $p_z$  is the original probability of  $z$ .
H Comparison with Guo et al. (2022b)
Figure 11: Comparison of our upper bound on advantage (Equation (4)) with Guo et al. (2022b) as function of  $\sigma$  for a uniform prior of size ten. We use a single step of DP-SGD with no mini-batch subsampling, and use 100,000 samples for Monte-Carlo approximation.
Table 2: Comparison of our upper bound on advantage (Equation (4)) with Guo et al. (2022b) and the Guo et al. (2022b) Monte-Carlo approximation (abbreviated to MC) as function of  $\sigma$  for a uniform prior size of ten and one hundred.
Prior size	Method	Advantage upper bound
σ
0.5	1	1.5	2	2.5	3
10	Guo et al. (2022b)	0.976	0.593	0.380	0.274	0.213	0.174
Guo et al. (2022b) (MC)	0.771	0.397	0.257	0.184	0.144	0.118
Ours	0.737	0.322	0.189	0.128	0.099	0.080
100	Guo et al. (2022b)	0.861	0.346	0.195	0.131	0.097	0.076
Guo et al. (2022b) (MC)	0.549	0.210	0.120	0.081	0.062	0.049
Ours	0.362	0.077	0.035	0.024	0.018	0.012
Recently, Guo et al. (2022b) have analyzed reconstruction of discrete training data. They note that DP bounds the mutual information shared between training data and learned parameters, and use Fano's inequality to convert this into a bound on reconstruction success. In particular, they define the advantage of the adversary as
$$
\mathbf {A d v} := \frac {p _ {\text {a d v e r s a r y s u c c e s s}} - p _ {\pi} ^ {\max }}{1 - p _ {\pi} ^ {\max }} \in [ 0, 1 ]. \tag {4}
$$
where  $p_{\pi}^{\max}$  is the maximum sampling probability from the prior,  $\pi$ , and  $p_{\text{adversary success}}$  is the probability that the adversary is successful at inferring which point in the prior was included in training. They then bound the advantage by lower bounding the adversary's error  $t := 1 - p_{\text{adversary success}}$  and by appealing to Fano's inequality they show this can be done by finding the smallest  $t \in [0,1]$  satisfying
$$
\begin{array}{l} f (t) := H (\pi) - I (\pi ; w) + t \log t + (1 - t) \log (1 - t) \\ - t \log (| \pi | - 1) \leq 0, \\ \end{array}
$$
where  $w$  is output of the private mechanism,  $H(\pi)$  is the entropy of the prior, and  $I(\pi; w)$  is the mutual information between the prior and output of the private mechanism. For an  $(\alpha, \epsilon)$ -RDP mechanism,  $I(\pi; w) \leq \epsilon$ , and so  $I(\pi; w)$  can be replaced by  $\epsilon$  in Equation (5). However, Guo et al. (2022b) show that for the Gaussian mechanism, this can improved upon either by using a Monte-Carlo approximation of  $I(\pi; w)$  — this involves approximating the KL divergence between a Gaussian and a Gaussian mixture — or by showing that  $I(\pi; w) \leq -\sum_{i=1}^{|\pi|} p_{\pi}^{i} \log \left(p_{\pi}^{i} + (1 - p_{\pi}^{i}) \exp \left(\frac{-\Delta^{2}}{2\sigma^{2}}\right)\right)$ ,
where  $\Delta$  is the sensitivity of the mechanism, and  $p_{\pi}^{i}$  is the probability of selecting the  $i$ th element from the prior. We use a uniform prior in all our experiments and so  $H(\pi) = -\log \left(\frac{1}{|\pi|}\right)$  and  $p_{\pi}^{i} = p_{\pi}^{\max} = \frac{1}{|\pi|}$ .
We convert our bound on success probability to advantage and compare with the Guo et al. (2022b) upper bound (and its Monte-Carlo approximation) in Figure 11 and Table 2, and note our bound is tighter.
I Experiments with very small priors
Our experiments in Section 3 and Section 4.3 were conducted with an adversary who has side information about the target point. Here, we reduce the amount of background knowledge the adversary has about the target, and measure how this affects the reconstruction upper bound and attack success.