Title: Privacy Auditing with One (1) Training Run
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy

membership inference attacks

privacy auditing
Abstract: We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple training examples independently. We analyze this using the connection between differential privacy and statistical generalization, which avoids the cost of group privacy. Our auditing scheme requires minimal assumptions about the algorithm and can be applied in the black-box or white-box setting. We demonstrate the effectiveness of our framework by applying it to DP-SGD, where we can achieve meaningful empirical privacy lower bounds by training only one model. In contrast, standard methods would require training hundreds of models.

Privacy Auditing with One (1) Training Run

Abstract

We propose a scheme for auditing differentially private machine learning systems with a single training run. This exploits the parallelism of being able to add or remove multiple training examples independently. We analyze this using the connection between differential privacy and statistical generalization, which avoids the cost of group privacy. Our auditing scheme requires minimal assumptions about the algorithm and can be applied in the black-box or white-box setting. We demonstrate the effectiveness of our framework by applying it to DP-SGD, where we can achieve meaningful empirical privacy lower bounds by training only one model . In contrast, standard methods would require training hundreds of models.

1 Introduction

Differential privacy (DP) [DMNS06] provides a quantifiable privacy guarantee by ensuring that no person's data significantly affects the probability of any outcome. Formally, a randomized algorithm M satisfies ( ε, δ ) -DP if, for any pair of inputs x, x ′ differing only by the addition or removal of one person's data and any measurable S , we have

A DP algorithm is accompanied by a mathematical proof giving an upper bound on the privacy parameters ε and δ . In contrast, a privacy audit provides an empirical lower bound on the privacy parameters. Privacy audits allow us to assess the tightness of the mathematical analysis [JUO20; NHSBTJCT23] or, if the lower and upper bounds are contradictory, to detect errors in the analysis or in the algorithm's implementation [DWWZK18; BGDCTV18; TTSSJC22].

Typically, privacy audits obtain a lower bound on the privacy parameters directly from the DP definition (1). That is, we construct a pair of inputs x, x ′ and a set of outcomes S and we estimate the probabilities P [ M ( x ) ∈ S ] and P [ M ( x ′ ) ∈ S ] . However, estimating these probabilities requires running M hundreds of times. This approach to privacy auditing is computationally expensive, which raises the question Can we perform privacy auditing using a single run of the algorithm M?

1.1 Our Contributions

Our approach: The DP definition (1) considers adding or removing a single person's data to or from the dataset. We consider multiple people's data and the dataset independently includes or excludes each person's data point. Our analysis exploits the parallelism of multiple independent data points in a single run of the algorithm in lieu of multiple independent runs. This approach is commonly used as an unproven heuristic in prior work [MEMPST21; ZBWTSRPNK22].

Our auditing procedure operates as follows. We identify m data points (i.e., training examples or 'canaries') to either include or exclude and we flip m independent unbiased coins to decide which of them to include or exclude. We then run the algorithm on the randomly selected dataset. Based

on the output of the algorithm, the auditor 'guesses' whether or not each data point was included or excluded (or it can abstain from guessing for some data points). We obtain a lower bound on the privacy parameters from the fraction of guesses that were correct.

Intuitively, if the algorithm is ( ε, 0) -DP, then the auditor can correctly guess each inclusion/exclusion coin flip with probability ≤ e ε e ε +1 . That is, the highest possible accuracy is attained by randomized response [War65]. Thus DP implies a high-probability upper bound on the fraction of correct guesses and, conversely, the fraction of correct guesses implies a high-probability lower bound on the privacy parameters.

Our analysis: Na¨ ıvely, analyzing the addition or removal of multiple data elements would rely on group privacy; but this does not exploit the fact that the data items were included or excluded independently. Instead, we leverage the connection between DP and generalization [DFHPRR15b; DFHPRR15a; BNSSSU16; RRST16; JLNRSMS19; SZ20]. Our main theoretical contribution is an improved analysis of this connection that is tailored to yield nearly tight bounds in our setting.

Informally, if we run a DP algorithm on i.i.d. samples from some distribution, then, conditioned on the output of the algorithm, the samples are still 'close' to being i.i.d. samples from that distribution. There is some technicality in making this precise, but, roughly speaking, we show that including or excluding m data points independently for one run is essentially as good as having m independent runs (as long as δ is small).

Our results: As an application of our new auditing framework, we audit DP-SGD training on a WideResNet model, trained on the CIFAR10 dataset across multiple configurations. Our approach successfully achieves an empirical lower bound of ε ≥ 1 . 8 , compared to a theoretical upper bound of ε ≤ 4 in the white-box setting. The m examples we insert for auditing (known in the literature as 'canaries') do not significantly impact the accuracy of the final model (less than a 5% decrease in accuracy) and our procedure only requires a single end-to-end training run. Such results were previously unattainable in the setting where only one model could be trained.

2 Our Auditing Procedure

Algorithm 1 Auditor with One Training Run

1: Data: x ∈ X n consisting of m auditing examples (a.k.a. canaries) x 1 , · · · , x m and n -m non-auditing examples x m +1 , · · · , x n .

2: Parameters: Algorithm to audit A , number of examples to randomize m , number of positive k + and negative k -guesses.

3: For i ∈ [ m ] , sample S i ∈ {-1 , +1 } uniformly and independently. Set S i = 1 for all i ∈ [ n ] \ [ m ] .

4: Partition x into x IN ∈ X n IN and x OUT ∈ X n OUT according to S , where n IN + n OUT = n . Namely, if S i = 1 , then x i is in x IN; and, if S i = -1 , then x i is in x OUT.

5: Run A on input x IN with appropriate parameters, outputting w .

7: Sort the scores Y . Let T ∈ {-1 , 0 , +1 } m be +1 for the largest k + scores and -1 for the smallest k -scores. (I.e., T ∈ {-1 , 0 , +1 } m maximizes ∑ m i T i · Y i subject to ∑ m i | T i | = k + + k -and ∑ m i T i = k + -k -.)

6: Compute the vector of scores Y = ( SCORE ( x i , w ) : i ∈ [ m ]) ∈ R m .

8: Return: S ∈ {-1 , +1 } m indicating the true selection and the guesses T ∈ {-1 , 0 , +1 } m .

We present our auditing procedure in Algorithm 1. We independently include each of the first m examples with 50% probability and exclude it otherwise. 2 When applied to DP-SGD, our approach is applicable to both 'white-box' auditing, where the adversary can access to all intermediate values of the model weights (as in the federated learning setting), and 'black-box' auditing, where the adversary only sees the final model weights (or can only query the final model, as in the centralized setting). In both cases we simply compute a 'score' for each example and 'guess' whether the example is included or excluded based on these scores. Specifically, we guess that the examples with the k + highest scores are included and the examples with the k -lowest scores are excluded,

and we abstain from guessing for the remaining m -k + -k -auditing examples; the setting of these parameters will depend on the application.

Note that we only randomize the first m examples x 1 , · · · , x m (which we refer to as 'auditing examples' or 'canaries'); the last n -m examples x m +1 , · · · , x n are always included and, thus, we do not make any guesses about them. To get the strongest auditing results we would set m = n , but we usually want to set m < n . For example, computing the score of all n examples may be computationally prohibitive, so we only compute the scores of m examples. We may wish to artificially construct m examples to be easy to identify (i.e., canaries), but also include n -m 'real' examples to ensure that A still produces a useful model. (I.e., having more training examples improves the performance of the model.)

The score function is arbitrary and will depend on the application. For black-box auditing, we use the loss of the final model w on the example x i - i.e., SCORE ( x i , w ) = -loss ( w,x i ) . For white-box auditing, SCORE ( x i , w ) = ∑ glyph[lscript] t 〈 w t -1 -w t , ∇ w t -1 loss ( w t -1 , x i ) 〉 is the sum of the inner products of updates with the (clipped) gradients of the loss on the example.

Intuitively, the vector of scores Y should be correlated with the true selection S , but too strong a correlation would violate DP. This is the basis of our audit. Specifically, the auditor computes T from Y which is a 'guess' at S . By the postprocessing property of DP, the guesses T are a differentially private function of the true S , which means that they cannot be too accurate.

Why abstain from guessing? The guesses are binary ( T i = 1 for IN, T i = -1 for OUT). Abstaining ( T i = 0 ) allows us to capture uncertainty. That is, on some examples we will not be sure whether they are IN or OUT. By abstaining on these uncertain examples, we can ensure higher accuracy for the remaining guesses. This yields better auditing results in practice.

3 Theoretical Analysis

To obtain a lower bound on the DP parameters we show that DP implies a high-probability upper bound on the number of correct guesses W := ∑ m i max { 0 , T i · S i } of our auditing procedure (Algorithm 1). Note that max { 0 , T i · S i } is simply 1 if the guess was correct - i.e., T i = S i -and 0 otherwise. The observed value of W then yields a high-probability lower bound on the DP parameters. To be more precise, we have the following guarantee.

Theorem 3.1 (Main Result) . Let ( S, T ) ∈ {-1 , +1 } m ×{-1 , 0 , +1 } m be the output of Algorithm 1. Assume the algorithm to audit A satisfies ( ε, δ ) -DP. Let r := k + + k -= ‖ T ‖ 1 be the number of guesses. Then, for all v ∈ R ,

where

If we ignore δ for the moment, Theorem 3.1 says that the number of correct guesses is stochastically dominated by Binomial ( r, e ε e ε +1 ) , where r = k + + k -is the total number of guesses. This binomial distribution is precisely the distribution of correct guesses we would get if T was obtained by independently performing ( ε, 0) -DP randomized response on r bits of S . In other words, the theorem says that ( ε, 0) -DP randomized response is exactly the worst-case algorithm in terms of the number of correct guesses. In particular, this means the theorem is tight (when δ = 0 ).

The binomial distribution is well-concentrated. In particular, for all β ∈ (0 , 1) , we have

There is an additional O ( δ ) term in the guarantee (2). The exact expression (3) is somewhat complex. It is always ≤ 2 mδ , since α ≤ 2 , but it is much smaller than this for reasonable parameter values. In particular, for v as in Equation 4 with β ≤ 1 /r 4 , we have α ≤ O (1 /r ) .

Theorem 3.1 gives us a hypothesis test: If A is ( ε, δ ) -DP, then the number of correct guesses W is ≤ r · e ε e ε +1 + O ( √ r ) with high probability. Thus, if the observed number of correct guesses v is larger than this, we can reject the hypothesis that A satisfies ( ε, δ ) -DP. We can convert this hypothesis test into a confidence interval (i.e., a lower bound on ε ) by finding the largest ε that we can reject at a desired level of confidence.

Proof of Theorem 3.1: Due to space limitations, the proof is deferred to the full version. But we outline the main ideas: First note that S ∈ {-1 , +1 } m is uniform and T is a DP function of S . Consider the distribution of S conditioned on T = t . If A is pure ( ε, 0) -DP, then we can easily analyze the conditional distribution using Bayes' law [DFHPRR15a; RRST16; JLNRSMS19] to conclude that each guess has probability ≤ e ε e ε +1 of being correct. Furthermore, this holds even if we condition on the other guesses, which allows us to inductively prove that the number of correct guesses is stochastically dominated by Binomial ( r, e ε e ε +1 ) . Handling approximate DP ( δ > 0 ) introduces additional complexity - some outputs T are 'bad' in the sense that the conditional distribution of S i could be arbitrary. Fortunately, such bad outputs are rare [KS14]. What we can show is that the number of correct guesses is stochastically dominated by ˇ W + F ( T ) , where ˇ W ← Binomial ( r, e ε e ε +1 ) is as before and F ( T ) ∈ { 0 , 1 , · · · , m } indicates how many of these bad events happened. We do not know the exact distribution of F ( T ) , but we do know E [ F ( T )] ≤ 2 mδ , which suffices to prove our result. Equation 3 comes from looking for the worst-case F ( T ) ; essentially the worst case is P [ F ( T ) = i ] = 2 mδ/i and P [ F ( T ) = 0] = 1 -2 mδ/i for some i ∈ [ m ] .

50

60

70

80

90

100

Correct Guesses (100 total, 0 abstentions)

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

95%   Lower Bound

=0

=0.001

=0.01

Ideal

0

1

2

3

4

5

6

7

Ideal

--

exp( )

exp( ) + 1

fraction correct guesses

0

2

4

6

95%   Lower Bound

=0

=10 3

=10 2

Ideal

Our theoretical analysis also gives novel implications for generalization from DP, improving on the results of Jung, Ligett, Neel, Roth, Sharifi-Malvajerdi, and Shenfeld [JLNRSMS19]. In addition, we obtain bounds on the mutual information I ( S ; M ( S )) ≤ 1 8 ε 2 m log e + δm log 2 , where S ∈ { 0 , 1 } m is uniformly random and M : { 0 , 1 } m → Y satisfies ( ε, δ ) -DP, which improves on the results of Steinke and Zakynthinou [SZ20]. See the full version for more details.

4 Experiments

Experiment Setup Our contributions are focused on improved analysis of an existing privacy attack, and are therefore orthogonal to the design of an attack. As a result, we rely on the experimental setup of the recent auditing procedure of Nasr, Hayes, Steinke, Balle, Tram` er, Jagielski, Carlini, and Terzis [NHSBTJCT23].

We run DP-SGD on the CIFAR-10 dataset with Wide ResNet (WRN-16) [ZK16], following the experimental setup of Nasr et al. [NHSBTJCT23]. Our experiments reach 76% test accuracy at ( ε = 8 , δ = 10 -5 ) -DP, which is comparable with the state-of-the-art [DBHSB22]. Unless specified

0

1

2

3

4

5

6

Theoretical Epsilon

0

1

2

3

4

5

6

Empirical Epsilon

Theoretical

m=5000

m=1000

m=100

m=50

m=10

0

1

2

3

4

5

6

Theoretical Epsilon

0

1

2

3

4

5

6

Empirical Epsilon

Theoretical

n

m

=0

n

m

=10 3

n

m

=10 4

otherwise, all lower bounds are presented with 95% confidence. Following prior work, we refer to the setting where the adversary has access to all intermediate steps as 'white-box' and when the adversary can only see the last iteration as 'black-box.' We experiment with both settings.

Algorithm 1 summarizes our approach for auditing DP-SGD. The results are converted into lower bounds on the privacy parameters using Theorem 3.1.

Wealso experiment with both the gradient and input attacks proposed by Nasr et al. [NHSBTJCT23]. In particular, for the gradient attack we use the strongest attack they proposed - the 'Dirac canary' approach - which sets all gradients to zero except at a single random index. In our setting where we need to create multiple auditing examples (canaries) we make sure the indices selected in our experiments do not have any repetitions. To compute the score for gradient space attacks, we use the dot product between the gradient update and auditing gradient. When auditing in input space, we leverage two different types of injected examples as:

Mislabeled example : We select a random subset of the test set and randomly relabel them (ensuring the new label is not the same as the original label).

In-distribution example : We select a random subset of the test set.

For input space audits, we use the loss of the input example as the score. In our experiments we report the attack with the highest lower bound.

In our experiments, we evaluate different values of k + and k -and report the best auditing results. Since changing the number of guesses r = k + + k -versus number of abstentions m -r is potentially testing multiple hypothesis on the same data, we should reduce the reported confidence value of our results. However, common practice is to simply ignore this minor issue [ZBWTSRPNK22; MSS22]. Fortunately, our main theorem can be extended to account for the dynamic choice of the number of guesses (see Corollary 5.8 in the full version).

4.1 Gradient Space attacks

We start with the strongest attack: We assume white-box access - i.e., the auditor sees all intermediate iterates of DP-SGD (which is a reasonable assumption in the federated learning setting) - and that the auditor can insert examples with arbitrary gradients into the training procedure. First, we evaluate the effect of the number of the auditing example on the tightness. Figure 3 demonstrates that as the number of examples increases, the auditing becomes tighter. However, the impact of the additional examples eventually diminishes. Intriguingly, adding more non-auditing training examples (resulting in a larger n compared to m ) does not seem to influence the tightness of the auditing, as depicted in Figure 4. This can be primarily due to the fact that gradient attacks proposed in prior studies can generate near-worst-case datasets, irrespective of the presence of other data points.

0

1

2

3

4

5

6

Theoretical Epsilon

0

1

2

3

4

5

6

Emprical Epsilon

Theoretical

One Step

100 Steps

500 Steps

1000 Steps

1

2

3

4

5

6

7

8

Theoretical Epsilon

1

2

3

4

5

6

7

8

Empirical Epsilon

Theoretical

Empirical

Another parameter that might affect the auditing results is the number of iterations glyph[lscript] in the DPSGD algorithm. As shown in Figure 5 we compare the extreme setting of having one iteration to multiple iterations and we do not observe any significant difference in the auditing when auditing for the equivalent privacy guarantees (by increasing the noise). The results confirm the tightness of composition and that the number of iterations does not have significant effect on auditing in white-box setting.

Now we directly use the parameters used in the training CIFAR10 models. Figure 6 summarizes results for the CIFAR10 models. We used m = 5000 and all of the training dataset from CIFAR10 ( n = 50 , 000 ) for the attack. We were able to achieve 76% accuracy for ε = 8 ( δ = 10 -5 , compared to 78% when not auditing). We are able to achieve an empirical lower bound of 0 . 7 , 1 . 2 , 1 . 8 , 3 . 5 for theoretical epsilon of 1 , 2 , 4 , 8 respectively. While our results are not as tight as the prior works, we only require a single run of training which is not possible using the existing techniques. In the era of exponentially expanding machine learning models, the computational and financial costs of training these colossal architectures even once are significant. Expecting any individual or entity to shoulder the burden of training such models thousands of times for the sake of auditing or experimental purposes is both unrealistic and economically infeasible. Our method offers a unique advantage by facilitating the auditing of these models, allowing for an estimation of privacy leakage in a white-box setting without significantly affecting performance.

4.2 Input Space Attacks

Now we evaluate the effect of input space attacks in the black-box setting. In this attack, the auditor can only insert actual images into the training procedure and cannot control any of the aspects of the training. Then, the adversary can observe the final model as mentioned in Algorithm 1. This is the weakest attack setting.

For simplicity we start with the setting where m = n ; in other words, all of the examples used to train the model are randomly included or excluded and can be used for auditing. Figure 7 illustrates the result of this setting. As we see from the figure, unlike the white-box attack we do not observe a monotonic relationship between the number of auditing examples and the tightness of the auditing. Intuitively, when the number of auditing examples are low then we do not have enough observations to have high confidence lower bounds for epsilon. On the other hand, when the number of auditing examples are high, the model does not have enough capacity to 'memorize' all of the auditing examples which reduces the tightness of the auditing. However, this can be improved by designing better black-box attacks which we reiterate in the next section.

We also evaluate the effect of adding additional training data to the auditing in Figure 8. We see that adding superfluous training data significantly reduces the effectiveness of auditing. The observed reduction in auditing effectiveness with the addition of more training data could be attributed to

1

2

3

4

5

6

7

8

Theoretical Epsilon

0

1

2

3

4

5

6

7

8

Empirical Epsilon

Theoretical

m

=500

m

=1000

m

=10000

1

2

3

4

5

6

7

8

Theoretical Epsilon

0

1

2

3

4

5

6

7

8

Empirical Epsilon

Theoretical

n

m

=0

n

m

=1000

n

m

=10000

several factors. One interpretation could be that the theoretical privacy analysis in a black-box setting tends to be considerably more loose when the adversary is constrained to this setting. This could potentially result in an overestimation of the privacy bounds. Conversely, it is also plausible that the results are due to the weak black-box attacks and can be improved in the future.

5 Related Work

The goal of privacy auditing is to empirically estimate the privacy provided by an algorithm, typically to accompany a formal privacy guarantee. Early work on auditing has often been motivated by trying to identify bugs in the implementations of differentially private data analysis algorithms [DWWZK18; BGDCTV18].

Techniques for auditing differentially private machine learning typically rely on conducting some form of membership inference attack [SSSS17]; 3 these attacks are designed to detect the presence or absence of an individual example in the training set. Essentially, a membership inference attack which achieves some true positive rate (TPR) and false positive rate (FPR) gives a lower bound on the privacy parameter ε ≥ log e ( TPR / FPR ) (after ensuring statistical validity of the TPR and FPR estimates).

Jayaraman and Evans [JE19] use standard membership inference attacks to evaluate different privacy analysis algorithms. Jagielski, Ullman, and Oprea [JUO20] consider inferring membership of worstcase 'poisoning' examples to conduct stronger membership inference attacks and understand the tightness of privacy analysis. Nasr, Song, Thakurta, Papernot, and Carlini [NSTPC21] measure the tightness of privacy analysis under a variety of threat models, including showing that the DP-SGD analysis is tight in the threat model assumed by the standard DP-SGD analysis.

Improvements to auditing have been made in a variety of directions. For example, Nasr, Hayes, Steinke, Balle, Tram` er, Jagielski, Carlini, and Terzis [NHSBTJCT23] and Maddock, Sablayrolles, and Stock [MSS22] take advantage of the iterative nature of DP-SGD, auditing individual steps to understand privacy of the end-to-end algorithm. Improvements have also been made to the basic statistical techniques for estimating the ε parameter, for example by using Log-Katz confidence intervals [LMFLZWRFT22], Bayesian techniques [ZBWTSRPNK22], or auditing algorithms in different privacy definitions [NHSBTJCT23].

Andrew, Kairouz, Oh, Oprea, McMahan, and Suriyakumar [AKOOMS23] build on the observation that, when performing membership inference, analyzing the case where the data is not included does

not require re-running the algorithm; instead we can re-sample the excluded data point; if the data points are i.i.d. from a nice distribution, this permits closed-form analysis of the excluded case. This gives a method for estimating the privacy parameters of an algorithm in a single run, but it does not guarantee that the estimate is either a lower or upper bound. In a similar vein, independent and concurrent work by Pillutla, Andrew, Kairouz, McMahan, Oprea, and Oh [PAKMOO23] provides a rigorous privacy auditing scheme that re-uses training runs to improve efficiency. Rather than consider a single pair of inputs differing by a single example, they consider multiple pairs of neighbouring inputs, but these pairs are overlapping in the sense that a single input dataset may appear in multiple pairs. Thus each training run can be re-used to estimate the privacy parameters of multiple pairs simultaneously or, rather, a single lower bound on the privacy parameters is computed over a (random) combination of the pairs. This is significantly more efficient than the na¨ ıve approach, but still requires more than one run.

A recent heuristic proposed to improve the efficiency of auditing is performing membership inference on multiple examples simultaneously. This heuristic was proposed by Malek Esmaeili, Mironov, Prasad, Shilov, and Tramer [MEMPST21], and evaluated more rigorously by ZanellaB´ eguelin, Wutschitz, Tople, Salem, R¨ uhle, Paverd, Naseri, and K¨ opf [ZBWTSRPNK22]. However, this heuristic is not theoretically justified, as the TPR and FPR estimates are not based on independent samples. In our work, we provide a proof of the validity of this heuristic. In fact, with this proof, we show for the first time that standard membership inference attacks, which attack multiple examples per training run, can be used for auditing analysis; prior work using these attacks must make an independence assumption. As a result, auditing can take advantage of progress in the membership inference field [CCNSTT22; WBKBGGG22].

Our theoretical analysis builds on the connection between DP and generalization [DFHPRR15b; DFHPRR15a; BNSSSU16; RRST16; JLNRSMS19; SZ20]. Our approach can be viewed as a contrapositive to 'privacy implies generalization.' That is, 'failure to generalize implies non-privacy.' Our auditing scheme can also be viewed as a 'reconstruction attack' [DN03; DSSU17]. That is, the auditor's guesses T can be viewed as a reconstruction of the private coins S . And it is known that DP prevents reconstruction [De12]

6 Discussion

Our main contribution is showing that we can audit the differential privacy guarantees of an algorithm with a single run. In contrast, prior methods require hundreds - if not thousands - of runs, which is computationally prohibitive for all but the simplest algorithms and machine learning models. Our experimental results demonstrate that in practical settings our methods are able to give meaningful lower bounds on the privacy parameter ε .

However, while we win on computational efficiency, we lose on tightness of our lower bounds. We now illustrate the limitations of our approach and discuss the extent to which this is inherent, and what lessons we can learn.

But, first, we illustrate that our method can give tight lower bounds. In Figure 9, we consider an idealized setting where the number of guesses changes and the fraction that are correct is fixed at e ε e ε +1 for ε = 4 - i.e., 98.2% of guesses are correct. 4 This is the maximum expected fraction of correct guesses compatible with (4 , 0) -DP. In this setting the lower bound on ε does indeed come close to 4 . With 10,000 guesses we get ε ≥ 3 . 87 with 95% confidence.

Note that the lower bound in Figure 9 improves as we increase the number of guesses. This is simply accounting for sampling error - to get a lower bound with 95% confidence, we must underestimate to account for the fact that the number of correct guesses may have been inflated by chance. As we get more guesses, the relative size of chance deviations reduces.

Limitations: Next we consider a different idealized setting - one that is arguably more realistic where our method does not give tight lower bounds. Suppose S i ∈ {-1 , +1 } indicates whether example i ∈ [ n ] is included or excluded. In Figure 10, we consider Gaussian noise addition. That is, we release a sample from N ( S i , 4) . (In contrast, Figure 9 considers randomized response on S i .) A tight analysis of the Gaussian mechanism [BW18, Theorem 8] gives an upper bound of

10 1

10 2

10 3

10 4

10 5

10 6

Number of Guesses (98% are correct)

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Lower Bound on

Ideal

95% Lower Bound (

=10 5 )

10

7

10

5

10

3

10

1

0

1

2

3

4

5

6

99% Lower Bound

95% Lower Bound

75% Lower Bound

10 1

10 2

10 3

10 4

Number of Guesses (out of 100000 examples)

0

1

2

3

4

for  =1

e

05

95% Lower Bound

(4 . 38 , 10 -5 ) -DP. Unlike for randomized response, abstentions matter here. We consider 100,000 examples, each of which has a score sampled from N ( S i , 4) , where S i ∈ {-1 , +1 } is uniformly random. We pick the largest r/ 2 scores and guess S i = +1 . Similarly we guess S i = -1 for the smallest r/ 2 scores. We abstain for the remaining 100 , 000 -r examples. If we make more guesses (i.e., increase r ), then we start making guesses on examples which are less 'in the tails' of each distribution, which provide less signal for the attack, making the attack accuracy go down along with our lower bound. We must trade off between more guesses being less accurate on average and more guesses having smaller relative sampling error.

In Figure 10, the highest value of the lower bound is ε ≥ 2 . 675 for δ = 10 -5 , which is attained by 1439 correct guesses out of 1510 . In contrast, the upper bound is ε = 4 . 38 for δ = 10 -5 . To get a matching upper bound of ε ≤ 2 . 675 we would need to set δ = 0 . 0039334 . In other words, the gap between the upper and lower bounds is a factor of 393 × in δ .

Figure 11 considers the same idealized setting as Figure 10, but we fix the number of guesses to 1,500 out of 100,000 (of which 1,429 are correct); instead we vary δ and the confidence level.

Are these limitations inherent? Figures 10 & 11 illustrate the limitations of our approach. They also hint at the cause: The number of guesses versus abstentions, the δ parameter, and the confidence all have a large effect on the tightness of our lower bound.

Our theoretical analysis is fairly tight; there is little room to improve Theorem 3.1. We argue that the inherent problem is a mismatch between 'realistic' DP algorithms and the 'pathological' DP algorithms for which our analysis is nearly tight. This mismatch makes our lower bound much more sensitive to δ than it 'should' be.

To be concrete about what we consider pathological, consider M : {-1 , +1 } m → {-1 , 0 , +1 } m defined by Algorithm 2. This algorithm satisfies ( ε, δ ) -DP and makes r guesses with m -r abstentions. In the X =1 case, the expected fraction of correct guesses is mδ rβ + ( 1 -mδ rβ ) · e ε e ε +1 . This is higher than the average fraction of correct guesses, but if we want confidence 1 -β in our lower bound, we must consider this case, as X =1 happens with probability β .

Intuitively, the contribution from δ to the fraction of correct guesses should be negligible. However, we see that δ is multiplied by m/rβ . That is to say, in the settings we consider, δ is multiplied by a factor on the order of 100 × or 1000 × , which means δ = 10 -5 makes a non-negligible contribution to the fraction of correct guesses.

Algorithm 2 Pathological Algorithm

1: Input: s ∈ {-1 , +1 } m 2: Parameters: r ∈ [ m ] , ε, δ ≥ 0 , β ∈ [0 , 1] . Assume 0 < mδ ≤ rβ . 3: Select U ⊂ [ m ] with | U | = r uniformly random. 4: Set T i = 0 for all i / ∈ U . 5: Sample X ← Bernoulli ( β ) . 6: if X = 1 then 7: for i ∈ U do 8: Independently sample T i ∈ {-1 , +1 } with P [ T i = s i ] = mδ rβ + ( 1 -mδ rβ ) · e ε e ε +1 . 9: end for 10: else if X = 0 then 11: for i ∈ U do 12: Independently sample T i ∈ {-1 , +1 } with P [ T i = s i ] = e ε e ε +1 . 13: end for 14: end if 15: Output: T ∈ {-1 , 0 , +1 } m .

It is tempting to try to circumvent this problem by simply setting δ to be very small. However, as shown in Figure 11, the corresponding upper bound on ε also increases as δ → 0 .

Unfortunately, there is no obvious general way to rule out algorithms that behave like Algorithm 2. The fundamental issue is that the privacy losses of the m examples are not independent; nor should we expect them to be, but they shouldn't be pathologically dependent either.

Directions for further work: Our work highlights several questions for further exploration:

Improved attacks: Our experimental evaluation uses existing attack methods. Any improvements to membership inference attacks could be combined with our results to yield improved privacy auditing.

One limitation of our attacks is that some examples may be 'harder' than others and the scores we compute do not account for this. When we have many runs, we can account for the hardness of individual examples [CCNSTT22], but in our setting it is not obvious how to do this.

Algorithm-specific analyses: Our methods are generic - they can be applied to essentially any DP algorithm. This is a strength, but there is also the possibility that we could obtain stronger results by exploiting the structure of specific algorithms. A natural example of such structure is the iterative nature of DP-SGD. That is, we can view one run of DP-SGD as the composition of multiple independent DP algorithms which are run sequentially.

Multiple runs & multiple examples: Our method performs auditing by including or excluding multiple examples in a single training run, while most prior work performs multiple training runs with a single example example included or excluded. Can we get the best of both worlds? If we use multiple examples and multiple runs, we should be able to get tighter results with fewer runs.

Other measures of privacy: Our theoretical analysis is tailored to the standard definition of differential privacy. But there are other definitions of differential privacy such as R´ enyi DP. And, in particular, many of the upper bounds are stated in this language. Hence it would make sense for the lower bounds also to be stated in this language.

Beyond lower bounds: Privacy auditing produces empirical lower bounds on the privacy parameters. In contrast, mathematical analysis produces upper bounds. Both are necessarily conservative, which leaves a large gap between the upper and lower bounds. A natural question is to find some middle ground - an estimate which is neither a lower nor upper bound, but provides some meaningful estimate of the 'true' privacy loss. However, it is unclear what kind of guarantee such an estimate should satisfy, or what interpretation the estimate should permit.

References

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper improves the computational efficiency of auditing differentially private machine learning systems by connecting differential privacy and statistical generalization. The authors propose the first 1-round scheme compared to the standard solutions with hundreds of training rounds. \n\nThe auditing procedure operates as follows. 1) Randomly identify $m$ data points to either include or exclude. 2) Run the algorithm on the randomly selected dataset. 3) Given the algorithm's output, the auditor “guesses” whether or not each data point was included or excluded. \n\nThe main theoretical contribution is an improved analysis that is tailored to yield tight bounds. The auditing scheme requires minimal assumptions. The authors theoretically justified the improved efficiency of auditing via membership inference on multiple examples simultaneously. The paper shows that standard membership inference attacks can be used for auditing analysis, i.e., exploiting the parallelism of multiple independent data points in a single run of the algorithm in lieu of multiple independent runs.\n\n\nIn experiments, the authors audited DP-SGD training on a WideResNet model, trained on the CIFAR10 dataset across multiple configurations. The experiments contain both gradient and input attacks. The experimental results confirm the contributions claimed by the authors before.\n\nSTRENGTHS:\n+ The paper gives the first scheme of one-round privacy auditing, which is computationally efficient. This is a totally new perspective, jumping out of prior literature.\n+ Theoretical foundation has been well built, i.e., Theorem 3.1. The formulation of privacy auditing bridges guess and privacy, which can be adopted in future instantiations/applications.\n+ The experimental results are convincing, i.e., confirming the paper's contribution.\n+ The writing is excellent. The authors clearly introduce the scheme, and meantime, show their deep insights and creative knowledge. At least, I feel I learned a lot in this paper.\n+ Personally speaking, I think this paper will motivate many future works.\n\nWEAKNESSES:\nI did not find the weakness.\n\nQUESTIONS:\nThe paper is self-contained. The details in the supplementary are very sufficient.\n\nLIMITATIONS:\nThe authors studied the limitation comprehensively.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper gives a simple version of a differential privacy (DP) auditing, and the proposed method is related to the recent works (G. Andrew, P. Kairouz, S. Oh, A. Oprea, H. B. McMahan, and V. Suriyaku- mar. “One-shot Empirical Privacy Estimation for Federated Learning”, 2023 and S. Zanella-Beguelin, L. Wutschitz, S. Tople, A. Salem, V. Ruhle, A. Paverd, M. Naseri, and B. Kopf. “Bayesian estimation of differential privacy”, 2022). There is also a theoretically rigorous analysis for the proposed method.\n\nAndrew et al. (2023) work seems very much related (does also one shot auditing) however more heuristic. This paper provides confidence intervals for the epsilons, whereas it seems to be an open problem whether the empirical epsilons by Andrew et al. (2023) can be rigorously connected to the theoretical guarantees.\n\nThe idea in the proposed method is simple: certain amount of data (canaries) is held out for auditing such that each sample is randomly included in the training, and in the end the total number of correct guesses (whether sample was or wasn't in the training data) will give the empirical epsilons (high-probability lower bounds). This ratio can be connected to the theoretical guarantees via Theorem 3.1. The main result is the analysis of the method (Theorem 3.1). The experiments are very similar to those used in the recent auditing paper by Nasr, Hayes, Steinke, Balle, Tram`er, Jagielski, Carlini, and Terzis.\n\nA pathological example in the shows the tightness of the epsilons give by this auditing method, and also includes some discussion about the looseness of the bounds in realistic situations.\n\nSTRENGTHS:\n- The idea is simple, the analysis sound and the paper is easy to read. All in all I think fits well to NeurIPS\n- The analysis is tight in a sense that there are pathological DP algorithms for which the empirical epsilons are tight.\n- Contribution seems clear: puts one shot auditing on a more rigorous footing (more rigorous than the previously proposed approaches)\n\nWEAKNESSES:\n- It remains a bit unclear from the paper, how does it compare to other auditing methods? It would be nice to see how it compares to e.g. to the empirical epsilons given in [NHSBTJCT23]. Especially in case of black box attacks there seems to be a big gap. Also would be interesting to see how the result compare to the results given by the one shot method of [AKOOMS23]. Even if the epsilons reported here are on a more rigorous footing than those of [AKOOMS23], I would guess that in practice one might end up using the method of [AKOOMS23] if it tends to give much more realistic epsilons.\n\n- It still remains a bit unclear how useful this would be in practice. Especially in the black box setting the results look weak: already with 10000 samples (all used for auditing) or small amount of additional samples (not used for auditing), the empirical epsilons are really far from the theoretical ones. \n\n- One small weakness is perhaps certain lack of originality: the main contribution is the mathematical analysis, the main ideas behind this auditing method seem to be laid out already in the recent works [ZBWTSRPNK22] and [AKOOMS23].\n\nQUESTIONS:\nIn the white box setting, your experiments show that increasing number of data samples increases the epsilon lower bound. In Figure 3 you have Cifar-10 results up to $m=5000$ samples. Did you try for $m>5000$? How close can you get? Why not to report results for $m>5000$?\n\nIn the white box setting, where you insert the canaries and get the best results:  how expensive is the attack itself? I mean if you construct the canaries at each step, how expensive it is compared to the model training? \n\nThis approach is for auditing an ML system, and then it seems reasonable to use a small amount of samples and report the epsilon lower bounds. What if the protection gets really strong for realistic training data sizes, are those epsilons meaningful anymore? Would there be a way to make the method dataset/model specific?\n\nTypo, p.5:\n\"First, we evaluate the effect of the number of the auditing example\"\n\nLIMITATIONS:\nYes, there is an extensive discussion in the end about the limitations of this approach. The extended version (supplements) has also a discussion about possible directions for future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The authors propose a scheme for auditing differentially private machine learning models with a single training run (instead of thousands as have been used so far).

STRENGTHS:
1. limitations clearly explained and illustrated
2. paper is really well structured (except for related work; see weaknesses)
3. important impact within DP auditing

WEAKNESSES:
1. It would help the reader if the related work was moved to before the empirical evaluation. Methods from the empirical evaluation are adapted from related work so more explanations earlier on might help with understanding
2. the comparison with related work falls a bit short. it would be nice to whether and when the heuristic proposed by MEMPST21 is not practically applicable. If they provide a better bound that holds true in most scenarios (even if there is no mathematical guarantee for it), it might be preferable to a loose bound with mathematical guarantee.

QUESTIONS:
1.  in ll 123 f., the authors say that the design of the attack is orthogonal to the analysis of the privacy attack. I wonder whether this claim is necessarily always true. Design methodology introduced for attacks with 1000s of runs might suffer from a high variance that could be maybe be highly detrimental for a single training run?
2. Do the authors believe that the methodology could be extended to multiple training runs to have a tradeoff between either 1000 or 1 runs?

LIMITATIONS:
the limitations have been adequately addressed and illustrated
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a one shot approach for auditing privacy. Their approach is as follows: given n independent input samples and a (dp) algorithm to audit, they divide the $n$ input samples into two groups $X$ and $Y$ of size $m$ and $n-m$ respectively. Then they randomly select a partition of the first part $(X_1,X_2)$, and only use $X_1 \\cup Y$ as the input of the dp algorithm. After that, they choose / design an application dependent score function that helps decide if each member of $X$ appeared in the input of the algorithm or not. The accuracy of this decision function implies a lower bound on the privacy of the algorithm. \n\nTheir main technical observation is that if $X$ is partitioned through a poisson sampling procedure, then running the auditing algorithm for one run is as good having $m$ independent runs. Their algorithm works both in the case where the final result of the algorithm is released and in the case where the intermediate steps are also released. In the end they decide whether the algorithm could be $(\\varepsilon, \\delta)$-dp or not by observing that the number correct guesses has to be less than $r \\cdot e^\\varepsilon / (e^\\varepsilon+1) + O(\\sqrt{r})$ with high probability, where $r< m$ is the total number of guesses.\n\nSTRENGTHS:\nThe experiments for gradient space attack suggest that their lower bounds are within constant factor of the theoretical upper bounds, in both the case where the adversary has access to all of the model iterates and in the case where the adversary only has access to the final model.\n\nThe review of related work is very good and well written.\n\nTheir empirical lower bounds converge to analytical bounds in some cases if delta is sufficiently small and the number of examples goes to infinity.\n\nWEAKNESSES:\nTheir approach does not seem to extend well to the case where delta is not very small.\n\nIn the setting where the adversary only can audit the input space (as opposed to the gradient space), their results do not seem to be very strong. This is important because in practice the algorithm has to be private with respect to the input space and not the gradient space.\n\nQUESTIONS:\nIn line 51 to 57, the authors mention that such results were previously unattainable in the setting where only one model could be trained. On a conceptual level, what do you think was the roadblock that previous work could not overcome and is overcome in this work?\n\nIt's not clear to me in the experiments whether the \"meaningful\" metric to consider is $n-m$ or $n / m$.\n\nThere's no other auditing baseline presented to compare to in Figure 6. How do your results compare to previous work?\n\nYour empirical results suggest that the lower bounds converge to analytical bounds in some cases if delta is sufficiently small and the number of examples go to infinity. Is there an analytical explanation for this?\n\nSomething that's a bit confusing to me is that in Figure 10, the empirical lower bound on epsilon decays after some point. Can you explain why this is happening?\n\nLIMITATIONS:\nThe authors have discussed the limitations of their work adequately.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of auditing differentially private machine learning systems. They propose a procedure which does so in one training run -- the key is the ability to include/exclude multiple data items in the run, as well as a novel analysis via leveraging connections between DP and generalization. They conduct experiments auditing DP-SGD obtaining meaningful empirical privacy lower bounds.\n\nSTRENGTHS:\n1. The topic of the paper is important and timely.  As the authors put, a privacy audit allows to assess the tightness of mathematical analysis and detect errors in analysis and implementation. As differentially private machine learning systems are being deployed increasingly, the research has potential to be impactful in these areas.\n\n2. The authors make interesting contributions, with strong theoretical underpinnings which directly lead to improved results in practice. In my limited experience, such results aren't are rather rare and I greatly enjoyed this aspect of the paper. Moreover, the idea of membership inference with multiple inclusion/exclusion of data items, has been used as a heuristic in the past, so this paper gives a mathematical justification for the heuristic.\n\n3. The paper is very well-written -- it is insightful and to the point. The paper also contains an extended discussion (Section 6) on the limitation of this work (and related approaches).\n\nWEAKNESSES:\n1. In some parts of the paper, I found the writing to be too dense. This is especially true for Section 6. I encourage the authors to revise this part.\n\n2. While the authors provide extensive experimental results testing various aspects of the procedure, I found the section to be poorly organized. I encourage the authors to provide an overview of what is to come in the beginning of the experiments section.\n\nQUESTIONS:\n1. Why the particular choice of score function for white-box attacks? Did the authors try other choices before settling on this?\n2. What is the intuition behind allowing abstention?\n\nLIMITATIONS:\nI greatly appreciate that the authors provided an extensive discussion of the limitations. However, as suggested above, the writing is too dense in these parts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}