SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jamie Hayes*
jamhay@google.com
Saeed Mahloujifar*†
saeedm@meta.com
Borja Balle
bballe@google.com
Differentially private training offers a protection which is usually interpreted as a guarantee against membership inference attacks. By proxy, this guarantee extends to other threats like reconstruction attacks attempting to extract complete training examples. Recent works provide evidence that if one does not need to protect against membership attacks but instead only wants to protect against training data reconstruction, then utility of private models can be improved because less noise is required to protect against these more ambitious attacks. We investigate this further in the context of DP-SGD, a standard algorithm for private deep learning, and provide an upper bound on the success of any reconstruction attack against DP-SGD together with an attack that empirically matches the predictions of our bound. Together, these two results open the door to fine-grained investigations on how to set the privacy parameters of DP-SGD in practice to protect against reconstruction attacks. Finally, we use our methods to demonstrate that different settings of the DP-SGD parameters leading to the same DP guarantees can result in significantly different success rates for reconstruction, indicating that the DP guarantee alone might not be a good proxy for controlling the protection against reconstruction attacks.
Machine learning models can and do leak training data (Ippolito et al., 2022; Kandpal et al., 2022; Carlini et al., 2019; Yeom et al., 2018; Song & Shmatikov, 2019; Tirumala et al., 2022). If the training data contains private or sensitive information, this can lead to information leakage via a variety of different privacy attacks (Balle et al., 2022; Carlini et al., 2022; Fredrikson et al., 2015; Carlini et al., 2021). Perhaps the most commonly studied privacy attack, membership inference (Homer et al., 2008; Shokri et al., 2017), aims to infer if a sample was included in the training set, which can lead to a privacy violation if inclusion in the training set is in and of itself sensitive. Membership inference leaks a single bit of information about a sample – whether that sample was or was not in the training set – and so any mitigation against this attack also defends against attacks that aim to reconstruct more information about a sample, such as training data reconstruction attacks (Balle et al., 2022; Carlini et al., 2021; Zhu et al., 2019).
Differential privacy (DP) (Dwork et al., 2006) provides an effective mitigation that provably bounds the success of any membership inference attack, and so consequently any training data reconstruction attack. The strength of this mitigation is controlled by a privacy parameter  $\epsilon$  which, informally, represents the number of bits that can be leaked about a training data sample, and so  $\epsilon$  must be small to guarantee the failure of a membership inference attack (Sablayrolles et al., 2019; Nasr et al., 2021). Unfortunately, training machine learning models with DP in the small  $\epsilon$  regime usually produces models that perform significantly worse than their non-private counterpart (Tramér & Boneh, 2021; De et al., 2022).
†Work done at Princeton University.
Membership inference may not always be the privacy attack that is of most concern. For example, in social networks, participation is usually public; recovering privately shared photos or messages from a model trained on social network data is the privacy violation. These kinds of attacks are referred to as training data reconstruction attacks, and have been successfully demonstrated against a number of machine learning models including language models (Carlini et al., 2021; Mireshghallah et al., 2022), generative models (Somepalli et al., 2022; Carlini et al., 2023), and image classifiers (Balle et al., 2022; Haim et al., 2022). Recent work (Bhowmick et al., 2018; Balle et al., 2022; Guo et al., 2022a,b; Stock et al., 2022) has begun to provide evidence that if one is willing to forgo protection against membership inference, then the  $\epsilon$  regime that protects against training data reconstruction is far larger, as predicted by the intuitive reasoning that successful reconstruction requires a significant number of bits about an individual example to be leaked by the model. This also has the benefit that models trained to protect against training data reconstruction but not membership inference do not suffer as large a drop in performance, as less noise is added during training. Yet, the implications of choosing a concrete  $\epsilon$  for a particular application remain unclear since the success of reconstruction attacks can vary greatly depending on the details of the threat model, the strength of the attack, and the criteria of what constitutes a successful reconstruction.
In this paper we re-visit the question of training data reconstruction against image classification models trained with DP-SGD (Song et al., 2013; Abadi et al., 2016), the workhorse of differentially private deep learning. We choose to concentrate our analysis on DP-SGD because state-of-the-art results are almost exclusively obtained with DP-SGD or other privatized optimizers (De et al., 2022; Cattan et al., 2022; Mehta et al., 2022). Our investigation focuses on attacks performed under a strong threat model where the adversary has access to intermediate gradients and knowledge of all the training data except the target of the reconstruction attacks. This threat model is consistent with the privacy analysis of DP-SGD (Abadi et al., 2016) and the informed adversary implicit in the definition of differential privacy (Nasr et al., 2021; Balle et al., 2022), and implies that conclusions about the impossibility of attacks in this model will transfer to weaker, more realistic threat models involving real-world attackers. Our investigation focuses on three main questions. 1) How do variations in the threat model (e.g. access to gradients and side knowledge available to the adversary) affect the success of reconstruction attacks? 2) Is it possible to obtain upper bounds on reconstruction success for DP-SGD that match the best known attacks and thus provide actionable insights into how to tune the privacy parameters in practice? 3) Does the standard DP parameter  $\epsilon$  provide enough information to characterize vulnerability against reconstruction attacks?
- We illustrate how changes in the threat model for reconstruction attacks against image classification models can significantly influence their success by comparing attacks with access to the final model parameters, access to intermediate gradients, and access to prior information.
- We obtain a tight upper bound on the success of any reconstruction attack against DP-SGD with access to intermediate gradients and prior information. Tightness is shown by providing an attack whose reconstruction success closely matches our bound's predictions.
- We provide evidence that the DP parameter  $\epsilon$  is not sufficient to capture the success of reconstruction attacks on DP-SGD by showing that different configurations of DP-SGD's hyperparameters leading to the same DP guarantee lead to different rates of reconstruction success.
2 Training Data Reconstruction in DP-SGD
We start by introducing DP-SGD, the algorithm we study throughout the paper, and then discuss reconstruction attacks with access to either only the final trained model, or all intermediate gradients. Then we empirically compare both attacks and show that gradient-based attacks are more powerful than model-based attacks, and we identify a significant gap between the success of the best attack and a known lower bound. As a result, we propose the problem of closing the gap between theoretical bounds and empirical reconstruction attacks.
Differential privacy and DP-SGD. Differential privacy (Dwork et al., 2006) formalizes the idea that data analysis algorithms whose output does not overly depend on any individual input data point
can provide reasonable privacy protections. Formally, we say that a randomized mechanism  $M$  satisfies  $(\epsilon, \delta)$ -differential privacy (DP) if, for any two datasets  $D$ ,  $D'$  that differ by one point, and any subset  $S$  of the output space we have  $P[M(D) \in S] \leq e^{\epsilon} P[M(D') \in S] + \delta$ . Informally, this means that DP mechanisms bound evidence an adversary can collect (after observing the output) about whether the point where  $D$  and  $D'$  differ was used in the analysis. For the bound to provide a meaningful protection against an adversary interested in this membership question it is necessary to take  $\epsilon \approx 1$  and  $\delta \leq 1 / |D|$ .
A differentially private version of stochastic gradient descent useful for training ML models can be obtained by bounding the influence of any individual sample in the trained model and masking it with noise. The resulting algorithm is called DP-SGD (Abadi et al., 2016) and proceeds by iteratively updating parameters with a privatized gradient descent step. Given a sampling probability  $q$ , current model parameters  $\theta$  and a loss function  $\ell(\theta, \cdot)$ , the privatized gradient  $g$  is obtained by first creating a mini-batch  $B$  including each point in the training dataset with probability  $q$ , summing the  $L_{2}$ -clipped gradients<sup>3</sup> for each point in  $B$ , and adding Gaussian noise with standard deviation  $\sigma \cdot C$  to all coordinates of the gradient:  $g \gets \sum_{z \in B} \mathrm{clip}_C(\nabla_\theta \ell(\theta, z)) + \mathcal{N}(0, C^2 \sigma^2 I)$ . Running DP-SGD for  $T$  training steps yields an  $(\epsilon, \delta)$ -DP mechanism with  $\epsilon \approx \frac{q\sqrt{T\log(1/\delta)}}{\sigma}$  (Abadi et al., 2016) - in practice, tighter numerical bounds on  $\epsilon$  are often used (Gopi et al., 2021; Google DP Team, 2022). For analytic purposes it is often useful to consider alternatives to  $(\epsilon, \delta)$ -DP to capture the differences between distributions  $M(D)$  and  $M(D')$ . Rényi differential privacy (RDP) (Mironov, 2017) is one such alternative often used in the context of DP-SGD. It states that the mechanism is  $(\alpha, \epsilon)$ -RDP for  $\alpha > 1$  and  $\epsilon \geq 0$  if  $\mathbb{E}_{W \sim M(D')} \left[ \left( \frac{\mathbb{P}[M(D) = W]}{\mathbb{P}[M(D') = W]} \right)^{\alpha} \right] \leq e^{(\alpha - 1)\epsilon}$ . In particular,  $T$  iterations of full-batch DP-SGD (i.e.  $q = 1$ ) with noise multiplier  $\sigma$  give  $(\alpha, \frac{\alpha T}{2\sigma^2})$ -RDP for every  $\alpha > 1$ .
Ultimately, we are interested in understanding how the privacy guarantee of DP-SGD affects reconstruction attacks and, in particular, whether  $\epsilon \gg 1$  still provides some protection against these more ambitious attacks. The first step is to understand what the most idoneous threat model to investigating this question is, and then to instantiate a powerful attack in that model.
2.1 Comparing reconstruction attacks under intermediate and final model access
In the case of DP-SGD, the privacy guarantee for the final model is obtained by analyzing the privacy loss incurred by releasing the  $T$  private gradients used by the algorithm. Thus, the guarantee applies both to the intermediate gradients and the final model (by virtue of the post-processing property of DP). It has been shown through membership inference attacks that the guarantee obtained through this analysis for the collection of intermediate gradients can be numerically tight (Nasr et al., 2021). However, in some specific settings amenable to mathematical treatment, it has also been shown that the final model produced by DP-SGD can enjoy a stronger DP guarantee than the collection of all intermediate gradients (Ye & Shokri, 2022; Altschuler & Talwar, 2022).
Although these observations apply the standard membership formulation of DP, they motivate an important question for trying to understand the implications of DP guarantees for reconstruction attacks: how does access to intermediate gradients affect the success of reconstruction attacks against DP-SGD? We investigate this question by introducing and comparing the success of model-based and gradient-based attacks. We assume the adversary receives the output of a DP mechanism  $M(D)$  and all the input data in  $D$  except for one point  $z^{*}$ ; the adversary's goal is then to produce a reconstruction  $\hat{z}$  of the unknown point. This adversary is referred to as the informed adversary in Balle et al. (2022).
Model-based training data reconstruction. Suppose we consider the output of DP-SGD to be only the final model  $\theta = M(D)$ . Under the informed adversary threat model, Balle et al. (2022) propose a reconstruction attack in which the adversary uses their knowledge of  $M$  and  $D_{-} = D\setminus \{z^{*}\}$  to train a reconstructor neural network (RecoNN) capable of mapping model parameters  $\theta$  to reconstructed points  $\hat{z}$ . The training dataset for this network consists of model-point pairs  $(\theta^i,z_i)$  where the  $z_{i}$  are auxiliary points representing side knowledge the adversary might possess about the distribution of the target point  $z^{*}$ , and the so-called shadow model  $\theta^i$  is obtained by applying  $M$  to the dataset  $D_{-}\cup \{z_{i}\}$  obtained by replacing  $z^{*}$  by  $z_{i}$  in the original dataset. Despite its computational cost, this attack is effective in reconstructing complete training examples from image classification models
3The clipping operation is defined as  $\mathrm{clip}_C(v) = v / \max \{1,\| v\| /C\}$
(a) Average MSE between targets and reconstructions on MNIST.
(b) Average MSE between targets and reconstructions on CIFAR-10.
Figure 1: Comparison of model-based and gradient-base reconstruction attacks. We run the attacks over 1K different images for both MNIST and CIFAR-10 for a range of  $\epsilon$ . In Figure 1a and Figure 1b, we plot the average MSE between reconstruction and target images for both attacks. To help calibrate a reader's interpretation of good and bad reconstructions we plot a nearest neighbor (NN) baseline, marking the average  $L_{2}$ -distance between NN images over the entire dataset, and a baseline corresponding to the average distance between random uniform vectors. We also plot a lower bound for MSE given by Guo et al. (2022a). We give qualitative examples of reconstructions in Appendix B.
trained without DP on MNIST and CIFAR-10, but fails to produce correct reconstructions when the model is trained with any reasonable setting of DP parameters.
Gradient-based training data reconstruction. Suppose now that the adversary gets access to all the intermediate gradients  $(g_{1},\ldots ,g_{T}) = M(D)$  produced by DP-SGD when training model  $\theta$ . We can instantiate a reconstruction attack for this scenario by leveraging gradient inversion attacks found in the federated learning literature (Yin et al., 2021; Huang et al., 2021; Jeon et al., 2021; Jin et al., 2021; Zhu et al., 2019; Geiping et al., 2020). In particular, we use the gradients produced by DP-SGD to construct a loss function  $\mathcal{L}(z)$  for which the target point  $z^{*}$  would be optimal if the gradients contained no noise, and then optimize it to obtain our candidate reconstruction. An important difference between our attack and previous works in the federated learning setting is that we use gradients from multiple steps to perform the attack and only attempt to recover a single point.
More formally, we start by removing from each privatized gradient  $g_{t}$  the gradients of the known points in  $D$ . For simplicity, in this section we only consider the full batch case ( $q = 1$ ) where every training point in  $D$  is included in every gradient step. Thus, we set  $\bar{g}_{t} = g_{t} - \sum_{z\in D_{-}}\mathrm{clip}_{C}(\nabla_{\theta}\ell (\theta_{t},z))$ , where  $\theta_t$  are the model parameters at step  $t$ . Note these can be inferred from the gradients and knowledge of the model initialization, which we also assume is given to the adversary. Similar (but not identical) to Geiping et al. (2020), we use the loss  $\mathcal{L}(z) = \sum_{t = 1}^{T}\mathcal{L}_{t}(z)$  with
$$
\mathcal {L} _ {t} (z) = - \left\langle \operatorname {c l i p} _ {C} \left(\nabla_ {\theta} \ell \left(\theta_ {t}, z\right)\right), \bar {g} _ {t} \right\rangle + \| \operatorname {c l i p} _ {C} \left(\nabla_ {\theta} \ell \left(\theta_ {t}, z\right)\right) - \bar {g} _ {t} \| _ {1}. \tag {1}
$$
This loss was selected after exhaustively trying all other losses suggested in the aforementioned literature; empirically, Equation (1) performed best.
Comparing reconstruction attacks. We now compare the success of model-based and gradient-based reconstruction attacks against classification models trained with DP-SGD on MNIST and CIFAR-10. We refer to Appendix A for experimental details.
Following Balle et al. (2022), we report the mean squared error (MSE) between the reconstruction and target as a function of  $\epsilon$  for both attacks. Results are shown in Figure 1, where we include two baselines to help calibrate how MSE corresponds to good and bad reconstructions. Firstly, we include a threshold representing the average distance between target points and their nearest neighbor in the remaining of the dataset. An MSE below this line indicates a near-perfect reconstruction. Secondly, we include a threshold representing the average distance between random uniform vectors of the same data dimensionality. An MSE above or close to this line indicates a poor reconstruction.
We make the following observations. First, the gradient-based attack outperforms the model-based attack by orders of magnitude at larger  $\epsilon$  values. Second, the best attack starts to perform well on
MNIST and CIFAR-10 between  $10^{2} < \epsilon < 10^{3}$  and  $10^{3} < \epsilon < 10^{4}$  respectively. This indicates that attack success is affected by the complexity of the underlying data, including its dimension and geometry. Finally, the attacks give an upper bound for reconstruction error (MSE), however it is not clear if this bound is tight (i.e., whether a better attack could reduce MSE). Guo et al. (2022a) report a lower bound for MSE of the form  $\frac{1}{4}(e^{\epsilon} - 1)$ , which is very far from the upper bound.
The experiment above illustrates how a change in threat model can make a significant difference in the success of reconstruction attacks. In particular, it shows that the attack from Balle et al. (2022) is far from optimal, potentially because of its lack of information about intermediate gradients. Thus, while optimal attacks for membership inference are known (Nasr et al., 2021) and can be used to empirically evaluate the strength of DP guarantees (e.g. for auditing purposes (Tramère et al., 2022)), the situation is far less clear in the case of reconstruction attacks. Gradient-based attacks improve over model-based attacks, but are they (near) optimal? Optimality of attacks is important for applications where one would like to calibrate the privacy parameters of DP-SGD to provide a demonstrable, pre-specified amount of protection against any reconstruction attacks.
3 Bounding Training Data Reconstruction
We will now formalize the reconstruction problem further and then provide bounds on the success probability of reconstruction attacks against DP-SGD. We will also develop improvements to the reconstruction attack introduced in Section 2 from which to benchmark the tightness of our bound.
3.1 Threat models for reconstruction
Balle et al. (2022) introduce a formal definition for reconstruction that attempts to capture the success probability of any reconstruction attack against a given mechanism. The definition involves an informed adversary with knowledge of the mechanism  $M$ , the fixed part of the training dataset  $D_{*}$ , and a prior distribution  $\pi$  from which the target  $z^{*}$  is sampled – the prior encodes the adversary's side knowledge about the target.
Definition 1. Let  $\pi$  by a prior over target points and  $\rho$  a reconstruction error function. A randomized mechanism  $M$  is  $(\eta, \gamma)$ -ReRo (reconstruction robust) with respect to  $\pi$  and  $\rho$  if for any fixed dataset  $D_{-}$  and any reconstruction attack  $R$  we have  $\mathbb{P}_{Z \sim \pi, w \sim M(D_{-} \cup \{Z\})}[\rho(Z, R(w)) \leq \eta] \leq \gamma$ .
Note that the output of the mechanism does not necessarily need to be final model parameters – indeed, the definition also applies to attacks operating on intermediate gradients when those are included in the output of the mechanism. Balle et al. (2022) also proved that any  $(\alpha, \epsilon)$ -RDP mechanism is  $(\eta, \gamma)$ -ReRo with  $\gamma = (\kappa_{\pi, \rho}(\eta) \cdot e^{\epsilon})^{\frac{\alpha - 1}{\alpha}}$ , where  $\kappa_{\pi, \rho}(\eta) = \sup_{z_0 \in \mathcal{Z}} \mathbb{P}_{Z \sim \pi}[\rho(Z, z_0) \leq \eta]$ . In particular, using the RDP guarantees of DP-SGD, one obtains that in the full-batch case, running the algorithm for  $T$  iterations with noise multiplier  $\sigma$  is  $(\eta, \gamma)$ -ReRo with  $\gamma$  bounded by
$$
\exp \left(- \max  \left\{0, \sqrt {\log \frac {1}{\kappa_ {\pi , \rho} (\eta)}} - \sqrt {\frac {T}{2 \sigma^ {2}}} \right\} ^ {2}\right). \tag {2}
$$
The quantity  $\kappa_{\pi, \rho}(\eta)$  can be thought of as the prior probability of reconstruction by an adversary who outputs a fixed candidate  $z_0$  based only on their knowledge of the prior (i.e. without observing the output of the mechanism).
Instantiating threat models within the  $(\eta, \gamma)$ -ReRo framework In Section 2.1, we described two variants of a reconstruction attack, one where the adversary has access to intermediate model updates (gradient-based reconstruction) and one where the adversary has access to the final model only (model-based reconstruction). In the language of  $(\eta, \gamma)$ -ReRo, both the gradient-based and model-based reconstruction attacks introduced in Section 2 take as arguments:  $\pi$  - prior information about the target point,  $D_{-}$  - the training dataset excluding the target point,  $w$  - the output of the mechanism  $M$ , and side information about DP-SGD - such as hyperparameters used in training and how to the model was initialized. The model-based reconstruction attack assumes that  $w = \theta_T$ , the parameters of the final model, whereas for gradient-based attacks,  $w = (g_1, \dots, g_T)$ , and so the adversary has access to all intermediate privatized model parameter updates.
The gradient-based attack optimizing Equation (1) does not make use of any potential prior knowledge the adversary might have about the target point  $z^{*}$ , beyond input bounds (the input is bound between [0,1]) and the dimensionality of target  $(28 \times 28$  for an MNIST digit). On the other hand, the model-based attack makes use of a prior in the form of the auxiliary points  $z_{i}$  used in the construction of shadow models; these points represent the adversary's knowledge about the distribution from which the target points  $z^{*}$  is sampled.