SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Google DeepMind steinke@google.com
Milad Nasr*
Google DeepMind srxzr@google.com
Matthew Jagielski*
Google DeepMind jagielski@gmail.com
We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple training examples independently. We analyze this using the connection between differential privacy and statistical generalization, which avoids the cost of group privacy. Our auditing scheme requires minimal assumptions about the algorithm and can be applied in the black-box or white-box setting. We demonstrate the effectiveness of our framework by applying it to DP-SGD, where we can achieve meaningful empirical privacy lower bounds by training only one model. In contrast, standard methods would require training hundreds of models.
Differential privacy (DP) [DMNS06] provides a quantifiable privacy guarantee by ensuring that no person's data significantly affects the probability of any outcome. Formally, a randomized algorithm  $M$  satisfies  $(\varepsilon, \delta)$ -DP if, for any pair of inputs  $x, x'$  differing only by the addition or removal of one person's data and any measurable  $S$ , we have
$$
\mathbb {P} [ M (x) \in S ] \leq e ^ {\varepsilon} \cdot \mathbb {P} [ M \left(x ^ {\prime}\right) \in S ] + \delta . \tag {1}
$$
A DP algorithm is accompanied by a mathematical proof giving an upper bound on the privacy parameters  $\varepsilon$  and  $\delta$ . In contrast, a privacy audit provides an empirical lower bound on the privacy parameters. Privacy audits allow us to assess the tightness of the mathematical analysis [JUO20; NHSBTJCT23] or, if the lower and upper bounds are contradictory, to detect errors in the analysis or in the algorithm's implementation [DWWZK18; BGDCTV18; TTSSJC22].
Typically, privacy audits obtain a lower bound on the privacy parameters directly from the DP definition (1). That is, we construct a pair of inputs  $x$ ,  $x'$  and a set of outcomes  $S$  and we estimate the probabilities  $\mathbb{P}[M(x) \in S]$  and  $\mathbb{P}[M(x') \in S]$ . However, estimating these probabilities requires running  $M$  hundreds of times. This approach to privacy auditing is computationally expensive, which raises the question Can we perform privacy auditing using a single run of the algorithm  $M$ ?
Our approach: The DP definition (1) considers adding or removing a single person's data to or from the dataset. We consider multiple people's data and the dataset independently includes or excludes each person's data point. Our analysis exploits the parallelism of multiple independent data points in a single run of the algorithm in lieu of multiple independent runs. This approach is commonly used as an unproven heuristic in prior work [MEMPST21; ZBWTSRPNK22].
Our auditing procedure operates as follows. We identify  $m$  data points (i.e., training examples or "canaries") to either include or exclude and we flip  $m$  independent unbiased coins to decide which of them to include or exclude. We then run the algorithm on the randomly selected dataset. Based
*Reverse alphabetical author order. Full version: https://arxiv.org/abs/2305.08846
on the output of the algorithm, the auditor "guesses" whether or not each data point was included or excluded (or it can abstain from guessing for some data points). We obtain a lower bound on the privacy parameters from the fraction of guesses that were correct.
Intuitively, if the algorithm is  $(\varepsilon, 0)$ -DP, then the auditor can correctly guess each inclusion/exclusion coin flip with probability  $\leq \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$ . That is, the highest possible accuracy is attained by randomized response [War65]. Thus DP implies a high-probability upper bound on the fraction of correct guesses and, conversely, the fraction of correct guesses implies a high-probability lower bound on the privacy parameters.
Our analysis: Naively, analyzing the addition or removal of multiple data elements would rely on group privacy; but this does not exploit the fact that the data items were included or excluded independently. Instead, we leverage the connection between DP and generalization [DFHPRR15b; DFHPRR15a; BNSSSU16; RRST16; JLNRSMS19; SZ20]. Our main theoretical contribution is an improved analysis of this connection that is tailored to yield nearly tight bounds in our setting.
Informally, if we run a DP algorithm on i.i.d. samples from some distribution, then, conditioned on the output of the algorithm, the samples are still "close" to being i.i.d. samples from that distribution. There is some technicality in making this precise, but, roughly speaking, we show that including or excluding  $m$  data points independently for one run is essentially as good as having  $m$  independent runs (as long as  $\delta$  is small).
Our results: As an application of our new auditing framework, we audit DP-SGD training on a WideResNet model, trained on the CIFAR10 dataset across multiple configurations. Our approach successfully achieves an empirical lower bound of  $\varepsilon \geq 1.8$ , compared to a theoretical upper bound of  $\varepsilon \leq 4$  in the white-box setting. The  $m$  examples we insert for auditing (known in the literature as "canaries") do not significantly impact the accuracy of the final model (less than a  $5\%$  decrease in accuracy) and our procedure only requires a single end-to-end training run. Such results were previously unattainable in the setting where only one model could be trained.
2 Our Auditing Procedure
Algorithm 1 Auditor with One Training Run
1: Data:  $x \in \mathcal{X}^n$  consisting of  $m$  auditing examples (a.k.a. canaries)  $x_1, \dots, x_m$  and  $n - m$  non-auditing examples  $x_{m + 1}, \dots, x_n$ .
2: Parameters: Algorithm to audit  $\mathcal{A}$ , number of examples to randomize  $m$ , number of positive  $k_{+}$  and negative  $k_{-}$  guesses.
3: For  $i \in [m]$ , sample  $S_i \in \{-1, +1\}$  uniformly and independently. Set  $S_i = 1$  for all  $i \in [n] \setminus [m]$ .
4: Partition  $x$  into  $x_{\mathrm{IN}} \in \mathcal{X}^{n_{\mathrm{IN}}}$  and  $x_{\mathrm{OUT}} \in \mathcal{X}^{n_{\mathrm{OUT}}}$  according to  $S$ , where  $n_{\mathrm{IN}} + n_{\mathrm{OUT}} = n$ . Namely, if  $S_i = 1$ , then  $x_i$  is in  $x_{\mathrm{IN}}$ ; and, if  $S_i = -1$ , then  $x_i$  is in  $x_{\mathrm{OUT}}$ .
5: Run  $\mathcal{A}$  on input  $x_{\mathrm{IN}}$  with appropriate parameters, outputting  $w$ .
6: Compute the vector of scores  $Y = (\mathrm{SCORE}(x_i, w) : i \in [m]) \in \mathbb{R}^m$ .
7: Sort the scores  $Y$ . Let  $T \in \{-1, 0, +1\}^m$  be  $+1$  for the largest  $k_+$  scores and  $-1$  for the smallest  $k_-$  scores. (i.e.,  $T \in \{-1, 0, +1\}^m$  maximizes  $\sum_{i}^{m} T_i \cdot Y_i$  subject to  $\sum_{i}^{m} |T_i| = k_+ + k_-$  and  $\sum_{i}^{m} T_i = k_+ - k_-$ .)
8: Return:  $S \in \{-1, +1\}^m$  indicating the true selection and the guesses  $T \in \{-1, 0, +1\}^m$ .
We present our auditing procedure in Algorithm 1. We independently include each of the first  $m$  examples with  $50\%$  probability and exclude it otherwise. When applied to DP-SGD, our approach is applicable to both "white-box" auditing, where the adversary can access to all intermediate values of the model weights (as in the federated learning setting), and "black-box" auditing, where the adversary only sees the final model weights (or can only query the final model, as in the centralized setting). In both cases we simply compute a "score" for each example and "guess" whether the example is included or excluded based on these scores. Specifically, we guess that the examples with the  $k_{+}$  highest scores are included and the examples with the  $k_{-}$  lowest scores are excluded,
2We consider the add/remove notion of DP, so changing one  $S_{i}$  results in neighbouring inputs  $x_{\mathrm{IN}}$ . Our methods readily extend to the replacement notion of DP; see the full version.
and we abstain from guessing for the remaining  $m - k_{+} - k_{-}$  auditing examples; the setting of these parameters will depend on the application.
Note that we only randomize the first  $m$  examples  $x_{1},\dots ,x_{m}$  (which we refer to as "auditing examples" or "canaries"); the last  $n - m$  examples  $x_{m + 1},\dots ,x_{n}$  are always included and, thus, we do not make any guesses about them. To get the strongest auditing results we would set  $m = n$ , but we usually want to set  $m < n$ . For example, computing the score of all  $n$  examples may be computationally prohibitive, so we only compute the scores of  $m$  examples. We may wish to artificially construct  $m$  examples to be easy to identify (i.e., canaries), but also include  $n - m$  "real" examples to ensure that  $\mathcal{A}$  still produces a useful model. (I.e., having more training examples improves the performance of the model.)
The score function is arbitrary and will depend on the application. For black-box auditing, we use the loss of the final model  $w$  on the example  $x_{i}$ -i.e.,  $\mathrm{SCORE}(x_i,w) = -\mathrm{loss}(w,x_i)$ . For white-box auditing,  $\mathrm{SCORE}(x_i,w) = \sum_t^\ell \left\langle w^{t - 1} - w^t,\nabla_{w^{t - 1}}\mathrm{loss}(w^{t - 1},x_i)\right\rangle$  is the sum of the inner products of updates with the (clipped) gradients of the loss on the example.
Intuitively, the vector of scores  $Y$  should be correlated with the true selection  $S$ , but too strong a correlation would violate DP. This is the basis of our audit. Specifically, the auditor computes  $T$  from  $Y$  which is a "guess" at  $S$ . By the postprocessing property of DP, the guesses  $T$  are a differentially private function of the true  $S$ , which means that they cannot be too accurate.
Why abstain from guessing? The guesses are binary ( $T_{i} = 1$  for IN,  $T_{i} = -1$  for OUT). Abstaining ( $T_{i} = 0$ ) allows us to capture uncertainty. That is, on some examples we will not be sure whether they are IN or OUT. By abstaining on these uncertain examples, we can ensure higher accuracy for the remaining guesses. This yields better auditing results in practice.
To obtain a lower bound on the DP parameters we show that DP implies a high-probability upper bound on the number of correct guesses  $W \coloneqq \sum_{i}^{m} \max \{0, T_i \cdot S_i\}$  of our auditing procedure (Algorithm 1). Note that  $\max \{0, T_i \cdot S_i\}$  is simply 1 if the guess was correct - i.e.,  $T_i = S_i$  and 0 otherwise. The observed value of  $W$  then yields a high-probability lower bound on the DP parameters. To be more precise, we have the following guarantee.
Theorem 3.1 (Main Result). Let  $(S,T)\in \{-1, + 1\} ^m\times \{-1,0, + 1\} ^m$  be the output of Algorithm 1. Assume the algorithm to audit  $\mathcal{A}$  satisfies  $(\varepsilon ,\delta)$  -DP.Let  $r\coloneqq k_{+} + k_{-} = \| T\|_{1}$  be the number of guesses. Then, for all  $v\in \mathbb{R}$
$$
\mathbb {P} \left[ \sum_ {i} ^ {m} \max  \{0, T _ {i} \cdot S _ {i} \} \geq v \right] \leq \underset {\check {W} \leftarrow \operatorname {B i n o m i a l} \left(r, \frac {e ^ {\varepsilon}}{e ^ {\varepsilon} + 1}\right)} {\mathbb {P}} [ \check {W} \geq v ] + \delta \cdot m \cdot \alpha , \tag {2}
$$
$$
\alpha = \max  _ {i \in [ m ]} \frac {2}{i} \cdot \underset {\check {W} \leftarrow \operatorname {B i n o m i a l} \left(r, \frac {e ^ {\varepsilon}}{e ^ {\varepsilon} + 1}\right)} {\mathbb {P}} [ v > \check {W} \geq v - i ]. \tag {3}
$$
If we ignore  $\delta$  for the moment, Theorem 3.1 says that the number of correct guesses is stochastically dominated by Binomial  $\left(r,\frac{e^{\varepsilon}}{e^{\varepsilon} + 1}\right)$ , where  $r = k_{+} + k_{-}$  is the total number of guesses. This binomial distribution is precisely the distribution of correct guesses we would get if  $T$  was obtained by independently performing  $(\varepsilon ,0)$ -DP randomized response on  $r$  bits of  $S$ . In other words, the theorem says that  $(\varepsilon ,0)$ -DP randomized response is exactly the worst-case algorithm in terms of the number of correct guesses. In particular, this means the theorem is tight (when  $\delta = 0$ ).
The binomial distribution is well-concentrated. In particular, for all  $\beta \in (0,1)$ , we have
$$
\underset {\check {W} \leftarrow \operatorname {B i n o m i a l} \left(r, \frac {e ^ {\varepsilon}}{e ^ {\varepsilon} + 1}\right)} {\mathbb {P}} \left[ \check {W} \geq \underbrace {\frac {r \cdot e ^ {\varepsilon}}{e ^ {\varepsilon} + 1} + \sqrt {\frac {1}{2} \cdot r \cdot \log (1 / \beta)}} _ {= v} \right] \leq \beta . \tag {4}
$$
There is an additional  $O(\delta)$  term in the guarantee (2). The exact expression (3) is somewhat complex. It is always  $\leq 2m\delta$ , since  $\alpha \leq 2$ , but it is much smaller than this for reasonable parameter values. In particular, for  $v$  as in Equation 4 with  $\beta \leq 1 / r^4$ , we have  $\alpha \leq O(1 / r)$ .
Theorem 3.1 gives us a hypothesis test: If  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -DP, then the number of correct guesses  $W$  is  $\leq \frac{r \cdot e^{\varepsilon}}{e^{\varepsilon} + 1} + O(\sqrt{r})$  with high probability. Thus, if the observed number of correct guesses  $v$  is larger than this, we can reject the hypothesis that  $\mathcal{A}$  satisfies  $(\varepsilon, \delta)$ -DP. We can convert this hypothesis test into a confidence interval (i.e., a lower bound on  $\varepsilon$ ) by finding the largest  $\varepsilon$  that we can reject at a desired level of confidence.
Proof of Theorem 3.1: Due to space limitations, the proof is deferred to the full version. But we outline the main ideas: First note that  $S \in \{-1, +1\}^m$  is uniform and  $T$  is a DP function of  $S$ . Consider the distribution of  $S$  conditioned on  $T = t$ . If  $\mathcal{A}$  is pure  $(\varepsilon, 0)$ -DP, then we can easily analyze the conditional distribution using Bayes' law [DFHPRR15a; RRST16; JLNRSMS19] to conclude that each guess has probability  $\leq \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$  of being correct. Furthermore, this holds even if we condition on the other guesses, which allows us to inductively prove that the number of correct guesses is stochastically dominated by Binomial  $\left(r, \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}\right)$ . Handling approximate DP  $(\delta > 0)$  introduces additional complexity – some outputs  $T$  are "bad" in the sense that the conditional distribution of  $S_i$  could be arbitrary. Fortunately, such bad outputs are rare [KS14]. What we can show is that the number of correct guesses is stochastically dominated by  $\check{W} + F(T)$ , where  $\check{W} \gets \text{Binomial} \left(r, \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}\right)$  is as before and  $F(T) \in \{0, 1, \dots, m\}$  indicates how many of these bad events happened. We do not know the exact distribution of  $F(T)$ , but we do know  $\mathbb{E}[F(T)] \leq 2m\delta$ , which suffices to prove our result. Equation 3 comes from looking for the worst-case  $F(T)$ ; essentially the worst case is  $\mathbb{P}[F(T) = i] = 2m\delta / i$  and  $\mathbb{P}[F(T) = 0] = 1 - 2m\delta / i$  for some  $i \in [m]$ .
Figure 1: Lower bound on the privacy parameter  $\varepsilon$  given by Theorem 3.1 with  $95\%$  confidence as the number of correct guesses changes. The total number of examples and guesses is 100. For comparison, we plot the ideal  $\varepsilon$  that gives  $100 \cdot \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$  correct guesses.
Figure 2: Lower bound on the privacy parameter  $\varepsilon$  given by Theorem 3.1 with  $95\%$  confidence as the number of correct guesses changes. The total number of examples and guesses is 1000 (with no abstentions). Here we plot the ideal  $\varepsilon$  on the horizontal axis, so that the number of correct guesses is  $1000 \cdot \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$ .
Our theoretical analysis also gives novel implications for generalization from DP, improving on the results of Jung, Ligett, Neel, Roth, Sharifi-Malvajerdi, and Shenfeld [JLNRSMS19]. In addition, we obtain bounds on the mutual information  $I(S; M(S)) \leq \frac{1}{8} \varepsilon^2 m \log e + \delta m \log 2$ , where  $S \in \{0, 1\}^m$  is uniformly random and  $M: \{0, 1\}^m \to \mathcal{V}$  satisfies  $(\varepsilon, \delta)$ -DP, which improves on the results of Steinke and Zakynthinou [SZ20]. See the full version for more details.
Experiment Setup Our contributions are focused on improved analysis of an existing privacy attack, and are therefore orthogonal to the design of an attack. As a result, we rely on the experimental setup of the recent auditing procedure of Nasr, Hayes, Steinke, Balle, Tramér, Jagielski, Carlini, and Terzis [NHSBTJCT23].
We run DP-SGD on the CIFAR-10 dataset with Wide ResNet (WRN-16) [ZK16], following the experimental setup of Nasr et al. [NHSBTJCT23]. Our experiments reach  $76\%$  test accuracy at  $(\varepsilon = 8, \delta = 10^{-5})$ -DP, which is comparable with the state-of-the-art [DBHSB22]. Unless specified
Figure 3: Effect of the number of auditing examples  $(m)$  in the white-box setting. By increasing the number of the auditing examples we are able to achieve tighter empirical lower bounds.
Figure 4: Effect of the number of additional examples  $(n - m)$  in the white-box setting. Importantly, adding additional examples does not impact the auditing results in the white-box setting.
otherwise, all lower bounds are presented with  $95\%$  confidence. Following prior work, we refer to the setting where the adversary has access to all intermediate steps as "white-box" and when the adversary can only see the last iteration as "black-box." We experiment with both settings.
Algorithm 1 summarizes our approach for auditing DP-SGD. The results are converted into lower bounds on the privacy parameters using Theorem 3.1.
We also experiment with both the gradient and input attacks proposed by Nasr et al. [NHSBTJCT23]. In particular, for the gradient attack we use the strongest attack they proposed – the “Dirac canary” approach – which sets all gradients to zero except at a single random index. In our setting where we need to create multiple auditing examples (canaries) we make sure the indices selected in our experiments do not have any repetitions. To compute the score for gradient space attacks, we use the dot product between the gradient update and auditing gradient. When auditing in input space, we leverage two different types of injected examples as:
1. Mislabeled example: We select a random subset of the test set and randomly relabel them (ensuring the new label is not the same as the original label).
2. In-distribution example: We select a random subset of the test set.
For input space audits, we use the loss of the input example as the score. In our experiments we report the attack with the highest lower bound.
In our experiments, we evaluate different values of  $k_{+}$  and  $k_{-}$  and report the best auditing results. Since changing the number of guesses  $r = k_{+} + k_{-}$  versus number of abstentions  $m - r$  is potentially testing multiple hypothesis on the same data, we should reduce the reported confidence value of our results. However, common practice is to simply ignore this minor issue [ZBWTSRPNK22; MSS22]. Fortunately, our main theorem can be extended to account for the dynamic choice of the number of guesses (see Corollary 5.8 in the full version).
4.1 Gradient Space attacks
We start with the strongest attack: We assume white-box access – i.e., the auditor sees all intermediate iterates of DP-SGD (which is a reasonable assumption in the federated learning setting) – and that the auditor can insert examples with arbitrary gradients into the training procedure. First, we evaluate the effect of the number of the auditing example on the tightness. Figure 3 demonstrates that as the number of examples increases, the auditing becomes tighter. However, the impact of the additional examples eventually diminishes. Intriguingly, adding more non-auditing training examples (resulting in a larger  $n$  compared to  $m$ ) does not seem to influence the tightness of the auditing, as depicted in Figure 4. This can be primarily due to the fact that gradient attacks proposed in prior studies can generate near-worst-case datasets, irrespective of the presence of other data points.
Figure 5: Effect of number of iterations in the white-box setting. Increasing the number of the steps (while keeping the same overall privacy by increasing the added noise) will not effect the auditing results.
Figure 6: Auditing CIFAR10 SoTA in white-box setting using gradient attacks. Our auditing framework can achieve meaningful empirical privacy lower bounds for SoTA models.
Another parameter that might affect the auditing results is the number of iterations  $\ell$  in the DP-SGD algorithm. As shown in Figure 5 we compare the extreme setting of having one iteration to multiple iterations and we do not observe any significant difference in the auditing when auditing for the equivalent privacy guarantees (by increasing the noise). The results confirm the tightness of composition and that the number of iterations does not have significant effect on auditing in white-box setting.
Now we directly use the parameters used in the training CIFAR10 models. Figure 6 summarizes results for the CIFAR10 models. We used  $m = 5000$  and all of the training dataset from CIFAR10 ( $n = 50,000$ ) for the attack. We were able to achieve  $76\%$  accuracy for  $\varepsilon = 8$  ( $\delta = 10^{-5}$ , compared to  $78\%$  when not auditing). We are able to achieve an empirical lower bound of 0.7, 1.2, 1.8, 3.5 for theoretical epsilon of 1, 2, 4, 8 respectively. While our results are not as tight as the prior works, we only require a single run of training which is not possible using the existing techniques. In the era of exponentially expanding machine learning models, the computational and financial costs of training these colossal architectures even once are significant. Expecting any individual or entity to shoulder the burden of training such models thousands of times for the sake of auditing or experimental purposes is both unrealistic and economically infeasible. Our method offers a unique advantage by facilitating the auditing of these models, allowing for an estimation of privacy leakage in a white-box setting without significantly affecting performance.
4.2 Input Space Attacks
Now we evaluate the effect of input space attacks in the black-box setting. In this attack, the auditor can only insert actual images into the training procedure and cannot control any of the aspects of the training. Then, the adversary can observe the final model as mentioned in Algorithm 1. This is the weakest attack setting.
For simplicity we start with the setting where  $m = n$ ; in other words, all of the examples used to train the model are randomly included or excluded and can be used for auditing. Figure 7 illustrates the result of this setting. As we see from the figure, unlike the white-box attack we do not observe a monotonic relationship between the number of auditing examples and the tightness of the auditing. Intuitively, when the number of auditing examples are low then we do not have enough observations to have high confidence lower bounds for epsilon. On the other hand, when the number of auditing examples are high, the model does not have enough capacity to "memorize" all of the auditing examples which reduces the tightness of the auditing. However, this can be improved by designing better black-box attacks which we reiterate in the next section.
We also evaluate the effect of adding additional training data to the auditing in Figure 8. We see that adding superfluous training data significantly reduces the effectiveness of auditing. The observed reduction in auditing effectiveness with the addition of more training data could be attributed to
Figure 7: Effect of the number of auditing examples  $(m)$  in the black-box setting. Black-box auditing is very sensitive to the number of auditing examples.
Figure 8: Effect of the number of additional examples on auditing  $(n - m)$  in the black-box setting. By increasing the number of additional examples, the auditing results get significantly looser.