SlowGuess/NeurIPS_2023_OCR huggingface dataset:
several factors. One interpretation could be that the theoretical privacy analysis in a black-box setting tends to be considerably more loose when the adversary is constrained to this setting. This could potentially result in an overestimation of the privacy bounds. Conversely, it is also plausible that the results are due to the weak black-box attacks and can be improved in the future.
The goal of privacy auditing is to empirically estimate the privacy provided by an algorithm, typically to accompany a formal privacy guarantee. Early work on auditing has often been motivated by trying to identify bugs in the implementations of differentially private data analysis algorithms [DWWZK18; BGDCTV18].
Techniques for auditing differentially private machine learning typically rely on conducting some form of membership inference attack [SSSS17]; these attacks are designed to detect the presence or absence of an individual example in the training set. Essentially, a membership inference attack which achieves some true positive rate (TPR) and false positive rate (FPR) gives a lower bound on the privacy parameter  $\varepsilon \geq \log_e(\mathrm{TPR / FPR})$  (after ensuring statistical validity of the TPR and FPR estimates).
Jayaraman and Evans [JE19] use standard membership inference attacks to evaluate different privacy analysis algorithms. Jagielski, Ullman, and Oprea [JUO20] consider inferring membership of worst-case "poisoning" examples to conduct stronger membership inference attacks and understand the tightness of privacy analysis. Nasr, Song, Thakurta, Papernot, and Carlini [NSTPC21] measure the tightness of privacy analysis under a variety of threat models, including showing that the DP-SGD analysis is tight in the threat model assumed by the standard DP-SGD analysis.
Improvements to auditing have been made in a variety of directions. For example, Nasr, Hayes, Steinke, Balle, Tramér, Jagielski, Carlini, and Terzis [NHSBTJCT23] and Maddock, Sablayrolles, and Stock [MSS22] take advantage of the iterative nature of DP-SGD, auditing individual steps to understand privacy of the end-to-end algorithm. Improvements have also been made to the basic statistical techniques for estimating the  $\varepsilon$  parameter, for example by using Log-Katz confidence intervals [LMFLZWRFT22], Bayesian techniques [ZBWTSRPNK22], or auditing algorithms in different privacy definitions [NHSBTJCT23].
Andrew, Kairouz, Oh, Oprea, McMahan, and Suriyakumar [AKOOMS23] build on the observation that, when performing membership inference, analyzing the case where the data is not included does
<sup>3</sup>Shokri, Stronati, Song, and Shmatikov [SSSS17] coined the term "membership inference attack" and were the first to apply such attacks to machine learning systems. However, similar attacks were developed for applications to genetic data [HSRDTMPSNC08; SOJH09; DSSUV15] and in cryptography [BS98; Tar08].
not require re-running the algorithm; instead we can re-sample the excluded data point; if the data points are i.i.d. from a nice distribution, this permits closed-form analysis of the excluded case. This gives a method for estimating the privacy parameters of an algorithm in a single run, but it does not guarantee that the estimate is either a lower or upper bound. In a similar vein, independent and concurrent work by Pillutla, Andrew, Kairouz, McMahan, Oprea, and Oh [PAKMOO23] provides a rigorous privacy auditing scheme that re-uses training runs to improve efficiency. Rather than consider a single pair of inputs differing by a single example, they consider multiple pairs of neighbouring inputs, but these pairs are overlapping in the sense that a single input dataset may appear in multiple pairs. Thus each training run can be re-used to estimate the privacy parameters of multiple pairs simultaneously or, rather, a single lower bound on the privacy parameters is computed over a (random) combination of the pairs. This is significantly more efficient than the naïve approach, but still requires more than one run.
A recent heuristic proposed to improve the efficiency of auditing is performing membership inference on multiple examples simultaneously. This heuristic was proposed by Malek Esmaeili, Mironov, Prasad, Shilov, and Tramer [MEMPST21], and evaluated more rigorously by Zanella-Béguelin, Wutschitz, Tople, Salem, Ruhle, Paverd, Naseri, and Kopf [ZBWTSRPNK22]. However, this heuristic is not theoretically justified, as the TPR and FPR estimates are not based on independent samples. In our work, we provide a proof of the validity of this heuristic. In fact, with this proof, we show for the first time that standard membership inference attacks, which attack multiple examples per training run, can be used for auditing analysis; prior work using these attacks must make an independence assumption. As a result, auditing can take advantage of progress in the membership inference field [CCNSTT22; WBKBGGG22].
Our theoretical analysis builds on the connection between DP and generalization [DFHPRR15b; DFHPRR15a; BNSSSU16; RRST16; JLNRSMS19; SZ20]. Our approach can be viewed as a contrapositive to "privacy implies generalization." That is, "failure to generalize implies non-privacy." Our auditing scheme can also be viewed as a "reconstruction attack" [DN03; DSSU17]. That is, the auditor's guesses  $T$  can be viewed as a reconstruction of the private coins  $S$ . And it is known that DP prevents reconstruction [De12]
Our main contribution is showing that we can audit the differential privacy guarantees of an algorithm with a single run. In contrast, prior methods require hundreds – if not thousands – of runs, which is computationally prohibitive for all but the simplest algorithms and machine learning models. Our experimental results demonstrate that in practical settings our methods are able to give meaningful lower bounds on the privacy parameter  $\varepsilon$ .
However, while we win on computational efficiency, we lose on tightness of our lower bounds. We now illustrate the limitations of our approach and discuss the extent to which this is inherent, and what lessons we can learn.
But, first, we illustrate that our method can give tight lower bounds. In Figure 9, we consider an idealized setting where the number of guesses changes and the fraction that are correct is fixed at  $\frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$  for  $\varepsilon = 4$ -i.e.,  $98.2\%$  of guesses are correct. This is the maximum expected fraction of correct guesses compatible with (4,0)-DP. In this setting the lower bound on  $\varepsilon$  does indeed come close to 4. With 10,000 guesses we get  $\varepsilon \geq 3.87$  with  $95\%$  confidence.
Note that the lower bound in Figure 9 improves as we increase the number of guesses. This is simply accounting for sampling error – to get a lower bound with  $95\%$  confidence, we must underestimate to account for the fact that the number of correct guesses may have been inflated by chance. As we get more guesses, the relative size of chance deviations reduces.
Limitations: Next we consider a different idealized setting - one that is arguably more realistic - where our method does not give tight lower bounds. Suppose  $S_{i} \in \{-1, + 1\}$  indicates whether example  $i \in [n]$  is included or excluded. In Figure 10, we consider Gaussian noise addition. That is, we release a sample from  $\mathcal{N}(S_i,4)$ . (In contrast, Figure 9 considers randomized response on  $S_{i}$ .) A tight analysis of the Gaussian mechanism [BW18, Theorem 8] gives an upper bound of
4The number of correct guesses is rounded down to an integer (which results in the lines being jagged). There are no abstentions.
Figure 9: Comparison of upper and lower bounds for idealized setting with varying number of guesses. The fraction of correct guesses is always  $\frac{e^{\varepsilon}}{e^{\varepsilon} + 1} \approx 0.982$  for  $\varepsilon = 4$ .
Figure 11: Same setting as Figure 10 with varying  $\delta$  and confidence, but number of guesses fixed to  $r = 1,500$  of which 1,429 are correct out of  $m = 100,000$  auditing examples.
Figure 10: Comparison of upper and lower bounds for idealized setting with varying number of guesses. For each example  $i \in [m]$ , we release  $S_{i} + \xi_{i}$ , where  $\xi_{i} \gets \mathcal{N}(0,4)$  and  $S_{i} \in \{-1, +1\}$  is independently uniformly random and indicates whether the sample is included/excluded. For the upper bound, we compute the exact  $(4.38, 10^{-5})$ -DP guarantee for the Gaussian mechanism. For the lower bound, we plot the bound of Theorem 3.1 with  $95\%$  confidence for varying numbers of guesses  $r$ . Total of  $m = 100,000$  randomized examples; we guess  $T_{i} = +1$  for the largest  $r/2$  scores and  $T_{i} = -1$  for the smallest  $r/2$  scores; we guess  $T_{i} = 0$  for the remaining  $m - r$  examples. The number of correct guesses is set to  $\lceil r \cdot \mathbb{P}[S_i = +1 | S_i + \xi_i > c] \rceil$ , where  $c$  is a threshold such that  $\mathbb{P}[S_i + \xi_i > c] = \frac{r}{2m}$ .
$(4.38, 10^{-5})$ -DP. Unlike for randomized response, abstentions matter here. We consider 100,000 examples, each of which has a score sampled from  $\mathcal{N}(S_i, 4)$ , where  $S_i \in \{-1, +1\}$  is uniformly random. We pick the largest  $r/2$  scores and guess  $S_i = +1$ . Similarly we guess  $S_i = -1$  for the smallest  $r/2$  scores. We abstain for the remaining  $100,000 - r$  examples. If we make more guesses (i.e., increase  $r$ ), then we start making guesses on examples which are less "in the tails" of each distribution, which provide less signal for the attack, making the attack accuracy go down along with our lower bound. We must trade off between more guesses being less accurate on average and more guesses having smaller relative sampling error.
In Figure 10, the highest value of the lower bound is  $\varepsilon \geq 2.675$  for  $\delta = 10^{-5}$ , which is attained by 1439 correct guesses out of 1510. In contrast, the upper bound is  $\varepsilon = 4.38$  for  $\delta = 10^{-5}$ . To get a matching upper bound of  $\varepsilon \leq 2.675$  we would need to set  $\delta = 0.0039334$ . In other words, the gap between the upper and lower bounds is a factor of  $393 \times$  in  $\delta$ .
Figure 11 considers the same idealized setting as Figure 10, but we fix the number of guesses to 1,500 out of 100,000 (of which 1,429 are correct); instead we vary  $\delta$  and the confidence level.
Are these limitations inherent? Figures 10 & 11 illustrate the limitations of our approach. They also hint at the cause: The number of guesses versus abstentions, the  $\delta$  parameter, and the confidence all have a large effect on the tightness of our lower bound.
Our theoretical analysis is fairly tight; there is little room to improve Theorem 3.1. We argue that the inherent problem is a mismatch between "realistic" DP algorithms and the "pathological" DP algorithms for which our analysis is nearly tight. This mismatch makes our lower bound much more sensitive to  $\delta$  than it "should" be.
To be concrete about what we consider pathological, consider  $M: \{-1, +1\}^m \to \{-1, 0, +1\}^m$  defined by Algorithm 2. This algorithm satisfies  $(\varepsilon, \delta)$ -DP and makes  $r$  guesses with  $m - r$  abstentions. In the  $X = 1$  case, the expected fraction of correct guesses is  $\frac{m\delta}{r\beta} + \left(1 - \frac{m\delta}{r\beta}\right) \cdot \frac{e^{\varepsilon}}{e^{\varepsilon} + 1}$ . This is higher than the average fraction of correct guesses, but if we want confidence  $1 - \beta$  in our lower bound, we must consider this case, as  $X = 1$  happens with probability  $\beta$ .
Intuitively, the contribution from  $\delta$  to the fraction of correct guesses should be negligible. However, we see that  $\delta$  is multiplied by  $m / r\beta$ . That is to say, in the settings we consider,  $\delta$  is multiplied by a factor on the order of  $100\times$  or  $1000\times$ , which means  $\delta = 10^{-5}$  makes a non-negligible contribution to the fraction of correct guesses.
Algorithm 2 Pathological Algorithm
1: Input:  $s \in \{-1, +1\}^m$   
2: Parameters:  $r \in [m]$ ,  $\varepsilon, \delta \geq 0$ ,  $\beta \in [0,1]$ . Assume  $0 < m\delta \leq r\beta$ .  
3: Select  $U \subset [m]$  with  $|U| = r$  uniformly random.  
4: Set  $T_i = 0$  for all  $i \notin U$ .  
5: Sample  $X \leftarrow$  Bernoulli( $\beta$ ).  
6: if  $X = 1$  then  
7: for  $i \in U$  do  
8: Independently sample  $T_i \in \{-1, +1\}$  with  $\mathbb{P}[T_i = s_i] = \frac{m\delta}{r\beta} + \left(1 - \frac{m\delta}{r\beta}\right) \cdot \frac{e^\varepsilon}{e^\varepsilon + 1}$ .  
9: end for  
10: else if  $X = 0$  then  
11: for  $i \in U$  do  
12: Independently sample  $T_i \in \{-1, +1\}$  with  $\mathbb{P}[T_i = s_i] = \frac{e^\varepsilon}{e^\varepsilon + 1}$ .  
13: end for  
14: end if  
15: Output:  $T \in \{-1, 0, +1\}^m$ .
It is tempting to try to circumvent this problem by simply setting  $\delta$  to be very small. However, as shown in Figure 11, the corresponding upper bound on  $\varepsilon$  also increases as  $\delta \to 0$ .
Unfortunately, there is no obvious general way to rule out algorithms that behave like Algorithm 2. The fundamental issue is that the privacy losses of the  $m$  examples are not independent; nor should we expect them to be, but they shouldn't be pathologically dependent either.
Directions for further work: Our work highlights several questions for further exploration:
- Improved attacks: Our experimental evaluation uses existing attack methods. Any improvements to membership inference attacks could be combined with our results to yield improved privacy auditing.
One limitation of our attacks is that some examples may be "harder" than others and the scores we compute do not account for this. When we have many runs, we can account for the hardness of individual examples [CCNSTT22], but in our setting it is not obvious how to do this.
- Algorithm-specific analyses: Our methods are generic – they can be applied to essentially any DP algorithm. This is a strength, but there is also the possibility that we could obtain stronger results by exploiting the structure of specific algorithms. A natural example of such structure is the iterative nature of DP-SGD. That is, we can view one run of DP-SGD as the composition of multiple independent DP algorithms which are run sequentially.
- Multiple runs & multiple examples: Our method performs auditing by including or excluding multiple examples in a single training run, while most prior work performs multiple training runs with a single example example included or excluded. Can we get the best of both worlds? If we use multiple examples and multiple runs, we should be able to get tighter results with fewer runs.
- Other measures of privacy: Our theoretical analysis is tailored to the standard definition of differential privacy. But there are other definitions of differential privacy such as Rényi DP. And, in particular, many of the upper bounds are stated in this language. Hence it would make sense for the lower bounds also to be stated in this language.
- Beyond lower bounds: Privacy auditing produces empirical lower bounds on the privacy parameters. In contrast, mathematical analysis produces upper bounds. Both are necessarily conservative, which leaves a large gap between the upper and lower bounds. A natural question is to find some middle ground – an estimate which is neither a lower nor upper bound, but provides some meaningful estimate of the "true" privacy loss. However, it is unclear what kind of guarantee such an estimate should satisfy, or what interpretation the estimate should permit.
[TSZ20] T. Steinke and L. Zakynthinou. “Reasoning about generalization via conditional mutual information”. In: Conference on Learning Theory. PMLR. 2020, pp. 3437–3452. URL: https://arxiv.org/abs/2001.09122 (cit. on pp. 2, 4, 8).
[Tar08] G. Tardos. "Optimal probabilistic fingerprint codes". In: Journal of the ACM (JACM) 55.2 (2008), pp. 1-24 (cit. on p. 7).
[TTSSJC22] F. Tramer, A. Terzis, T. Steinke, S. Song, M. Jagielski, and N. Carlini. "Debugging differential privacy: A case study for privacy auditing". In: arXiv preprint arXiv:2202.12219 (2022). URL: https://arxiv.org/abs/2202.12219 (cit. on p. 1).
[War65] S. L. Warner. "Randomized response: A survey technique for eliminating evasive answer bias". In: Journal of the American Statistical Association 60.309 (1965), pp. 63-69. URL: https://www.jstor.org/stable/2283137 (cit. on p. 2).
[WBKBGGG22] Y. Wen, A. Bansal, H. Kazemi, E. Borgnia, M. Goldblum, J. Geiping, and T. Goldstein. "Canary in a Coalmine: Better Membership Inference with Ensemble Adversarial Queries". In: arXiv preprint arXiv:2210.10750 (2022) (cit. on p. 8).
[ZBWTSRPNK22] S. Zanella-Béguelin, L. Wutschitz, S. Tople, A. Salem, V. Rühle, A. Paverd, M. Naseri, and B. Köpf. "Bayesian estimation of differential privacy". In: arXiv preprint arXiv:2206.05199 (2022) (cit. on pp. 1, 5, 7, 8).
[ZK16] S. Zagoruyko and N. Komodakis. "Wide residual networks". In: arXiv preprint arXiv:1605.07146 (2016) (cit. on p. 4).