Title: Unleashing the Power of Randomization in Auditing Differentially Private ML
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential privacy auditing

multiple canaries

randomization

lifting

adaptive confidence intervals
Abstract: We present a rigorous methodology for auditing differentially private machine learning by adding multiple carefully designed examples called canaries. We take a first principles approach based on three key components. First, we introduce Lifted Differential Privacy (LiDP) that expands the definition of differential privacy to handle randomized datasets. This gives us the freedom to design randomized canaries. Second, we audit LiDP by trying to distinguish between the model trained with $K$ canaries versus $K-1$ canaries in the dataset, leaving one canary out. By drawing the canaries i.i.d., LiDP can leverage the symmetry in the design and reuse each privately trained model to run multiple statistical tests, one for each canary. Third, we introduce novel confidence intervals that take advantage of the multiple test statistics by adapting to the empirical higher-order correlations. Together, this new recipe demonstrates significant improvements in sample complexity, both theoretically and empirically, using synthetic and real data. Further, recent advances in designing stronger canaries can be readily incorporated in the new framework.

Unleashing the Power of Randomization in Auditing Differentially Private ML

Abstract

We present a rigorous methodology for auditing differentially private machine learning algorithms by adding multiple carefully designed examples called canaries. We take a first principles approach based on three key components. First, we introduce Lifted Differential Privacy (LiDP) which expands the definition of differential privacy to handle randomized datasets. This gives us the freedom to design randomized canaries. Second, we audit LiDP by trying to distinguish between the model trained with K canaries versus K -1 canaries in the dataset, leaving one canary out. By drawing the canaries i.i.d., LiDP can leverage the symmetry in the design and reuse each privately trained model to run multiple statistical tests, one for each canary. Third, we introduce novel confidence intervals that take advantage of the multiple test statistics by adapting to the empirical higher-order correlations. Together, this new recipe demonstrates significant improvements in sample complexity, both theoretically and empirically, using synthetic and real data. Further, recent advances in designing stronger canaries can be readily incorporated into the new framework.

1 Introduction

Differential privacy (DP), introduced in [22], has gained widespread adoption by governments, companies, and researchers by formally ensuring plausible deniability for participating individuals. This is achieved by guaranteeing that a curious observer of the output of a query cannot be confident in their answer to the following binary hypothesis test: did a particular individual participate in the dataset or not? For example, introducing sufficient randomness when training a model on a certain dataset ensures a desired level of differential privacy. This in turn ensures that an individual's sensitive information cannot be inferred from the trained model with high confidence. However, calibrating the right amount of noise can be a challenging process. It is easy to make mistakes in implementing a DP mechanism due to intricacies involving micro-batching, sensitivity analysis, and privacy accounting. Even with correct implementation, there are several incidents of published DP algorithms with miscalculated privacy guarantees that falsely report higher levels of privacy [16, 34, 40, 47, 57, 58]. Data-driven approaches to auditing a mechanism for a violation of a claimed privacy guarantee can significantly mitigate the danger of unintentionally leaking sensitive data.

Popular approaches for auditing privacy share three common components [e.g. 31-33, 45, 69]. Conceptually, these approaches are founded on the definition of DP and involve producing counterexamples that potentially violate the DP condition. Algorithmically, this leads to the standard recipe of injecting a single carefully designed example, referred to as a canary , and running a statistical hypothesis test for its presence from the outcome of the mechanism. Analytically, a high-confidence bound on the DP condition is derived by calculating the confidence intervals of the corresponding Bernoulli random variables from n independent trials of the mechanism.

Recent advances adopt this standard approach and focus on designing stronger canaries to reduce the number of trials required to successfully audit DP [e.g., 31-33, 39, 45, 69]. However, each independent trial can be as costly as training a model from scratch; refuting a false claim of ( ε, δ ) -DP with a minimal number of samples is of utmost importance. In practice, standard auditing can require training on the range of thousands to hundreds of thousands of models [e.g., 60]. Unfortunately, under the standard recipe, we are fundamentally limited by the 1 / √ n sample dependence of the Bernoulli confidence intervals.

Contributions. We break this 1 / √ n barrier by rethinking auditing from first principles.

Lifted DP : We propose to audit an equivalent definition of DP, which we call Lifted DP in § 3.1. This gives an auditor the freedom to design a counter-example consisting of random datasets and rejection sets. This enables adding random canaries, which is critical in the next step. Theorem 3 shows that violation of Lifted DP implies violation of DP, justifying our framework.

Auditing Lifted DP with Multiple Random Canaries : We propose adding K > 1 canaries under the alternative hypothesis and comparing it against a dataset with K -1 canaries, leaving one canary out. If the canaries are deterministic, we need K separate null hypotheses; each hypothesis leaves one canary out. Our new recipe overcomes this inefficiency in § 3.2 by drawing random canaries independently from the same distribution. This ensures the exchangeability of the test statistics, allowing us to reuse each privately trained model to run multiple hypothesis tests in a principled manner. This is critical in making the confidence interval sample efficient.

Adaptive Confidence Intervals : Due to the symmetry of our design, the test statistics follow a special distribution that we call eXchangeable Bernoulli (XBern) . Auditing privacy boils down to computing confidence intervals on the average test statistic over the K canaries included in the dataset. If the test statistics are independent, the resulting confidence interval scales as 1 / √ nK . However, in practice, the dependence is non-zero and unknown. We propose a new and principled family of confidence intervals in § 3.3 that adapts to the empirical higher-order correlations between the test statistics. This gives significantly smaller confidence intervals when the actual dependence is small, both theoretically (Proposition 4) and empirically.

Numerical Results : We audit an unknown Gaussian mechanism with black-box access and demonstrate (up to) 16 × improvement in the sample complexity. We also show how to seamlessly lift recently proposed canary designs in our recipe to improve the sample complexity on real data.

2 Background

We describe the standard recipe to audit DP. Formally, we adopt the so-called add/remove definition of differential privacy; our constructions also seamlessly extend to other choices of neighborhoods as we explain in § B.2.

Definition 1 (Differential privacy) . A pair of datasets, ( D 0 , D 1 ) , is said to be neighboring if their sizes differ by one and the datasets differ in one entry: | D 1 \ D 0 | + | D 0 \ D 1 | = 1 . A randomized mechanism A : Z ∗ →R is said to be ( ε, δ ) -Differentially Private (DP) for some ε ≥ 0 and δ ∈ [0 , 1] if it satisfies P A ( A ( D 1 ) ∈ R ) ≤ e ε P A ( A ( D 0 ) ∈ R ) + δ , which is equivalent to

for all pairs of neighboring datasets, ( D 0 , D 1 ) , and all measurable sets, R ⊂ R , of the output space R , where P A is over the randomness internal to the mechanism A . Here, Z ∗ is a space of datasets.

For small ε and δ , one cannot infer from the output whether a particular individual is in the dataset or not with a high success probability. For a formal connection, we refer to [35]. This naturally leads to a standard procedure for auditing a mechanism A claiming ( ε, δ ) -DP: present ( D 0 , D 1 , R ) ∈ Z ∗ ×Z ∗ ×R that violates Eq. (1) as a piece of evidence. Such a counter-example confirms that an adversary attempting to test the participation of an individual will succeed with sufficient probability, thus removing the potential for plausible deniability for the participants.

Standard Recipe: Adding a Single Canary. When auditing DP model training (using e.g. DPSGD [1, 54]), the following recipe is now standard for designing a counter-example ( D 0 , D 1 , R ) [31-33, 45, 69]. A training dataset D 0 is assumed to be given. This ensures that the model under scrutiny matches the use-case and is called a null hypothesis . Next, under a corresponding alternative hypothesis , a neighboring dataset D 1 = D 0 ∪{ c } is constructed by adding a single carefully-designed

example c ∈ Z , known as a canary . Finally, Eq. (1) is evaluated with a choice of R called a rejection set . For example, one can reject the null hypothesis (and claim the presence of the canary) if the loss on the canary is smaller than a fixed threshold; R is a set of models satisfying this rejection rule.

Bernoulli Confidence Intervals. Once a counter-example ( D 0 , D 1 , R ) is selected, we are left to evaluate the DP condition in Eq. (1). Since the two probabilities in the condition cannot be directly evaluated, we rely on the samples of the output from the mechanism, e.g., models trained with DP-SGD. This is equivalent to estimating the expectation, P ( A ( D ) ∈ R ) for D ∈ { D 0 , D 1 } , of a Bernoulli random variable, I ( A ( D ) ∈ R ) , from n i.i.d. samples. Providing high confidence intervals for Bernoulli distributions is a well-studied problem with several off-the-shelf techniques, such as Clopper-Pearson, Jeffreys, Bernstein, and Wilson intervals. Concretely, let ˆ P n ( A ( D ) ∈ R ) denote the empirical probability of the model falling in the rejection set in n independent runs. The standard intervals scale as | P ( A ( D 0 ) ∈ R ) -ˆ P n ( A ( D 0 ) ∈ R ) | ≤ C 0 n -1 / 2 and | P ( A ( D 1 ) ∈ R ) -ˆ P n ( A ( D 1 ) ∈ R ) | ≤ C 1 n -1 / 2 for constants C 0 and C 1 independent of n . If A satisfies a claimed ( ε, δ ) -DP in Eq. (1), then the following finite-sample lower bound holds with high confidence:

Auditing ( ε, δ ) -DP amounts to testing the violation of this condition. This is fundamentally limited by the n -1 / 2 dependence of the Bernoulli confidence intervals. Our goal is to break this barrier.

Notation. While the DP condition is symmetric in ( D 0 , D 1 ) , we use D 0 , D 1 to refer to specific hypotheses. For symmetry, we need to check both conditions: Eq. (1) and its counterpart with D 0 , D 1 interchanged. We omit this second condition for notational convenience. We use the shorthand [ k ] := { 1 , 2 , . . . , k } . We refer to random variables by boldfaced letters (e.g. D is a random dataset).

Related Work. We provide a detailed survey in Appendix A. A stronger canary (and its rejection set) can increase the RHS of (1). The resulting hypothesis test can tolerate larger confidence intervals, thus requiring fewer samples. This has been the focus of recent breakthroughs in privacy auditing in [32, 39, 41, 45, 60]. They build upon membership inference attacks [e.g. 12, 53, 68] to measure memorization. Our aim is not to innovate on this front. Instead, our framework can seamlessly adopt recently designed canaries and inherit their strengths as demonstrated in § 5 and § 6.

Random canaries have been used in prior work, but for making the canary out-of-distribution in a computationally efficient manner. No variance reduction is achieved by such random canaries. Adding multiple (deterministic) canaries has been explored in literature but for different purposes. [32, 39] include multiple copies of the same canary to make the canary easier to detect while paying for group privacy since the paired datasets differ in multiple entries (see § 3.2 for a detailed discussion).

[42, 69] propose adding multiple distinct canaries to reuse each trained model for multiple hypothesis tests. However, each canary is no stronger than a single canary case, and the resulting auditing suffers from group privacy. When computing the lower bound on ε , however, group privacy is ignored and the test statistics are assumed to be independent without rigorous justification. [2] avoids group privacy in the federated scenario where the adversary has the freedom to return a canary gradient update of choice. The prescribed random gradient shows good empirical performance. The confidence interval is not rigorously derived. Our recipe for injecting multiple canaries without a group privacy cost with rigorous confidence intervals can be incorporated into these works to give provable lower bounds.

In an independent and concurrent work, Steinke et al. [56] also consider auditing with randomized canaries that are Poisson-sampled, i.e., each canary is included or excluded independently with equal probability. Their recipe involves computing an empirical lower bound by comparing the accuracy (rather than the full confusion matrix as in our case) from the possibly dependent guesses with the worst-case randomized response mechanism. This allows them to use multiple dependent observations from a single trial to give a high probability lower bound on ε . Their confidence intervals, unlike the ones we give here, are non-adaptive and worst-case.

3 A New Framework for Auditing DP Mechanisms with Multiple Canaries

We define Lifted DP, a new definition of privacy that is equivalent to DP ( § 3.1). This allows us to define a new recipe for auditing with multiple random canaries, as opposed to a single deterministic canary in the standard recipe, and reuse each trained model to run multiple correlated hypothesis tests

in a principled manner ( § 3.2). The resulting test statistics form a vector of dependent but exchangeable indicators (which we call an eXchangeable Bernoulli or XBern distribution), as opposed to a single Bernoulli distribution. We leverage this exchangeability to give confidence intervals for the XBern distribution that can potentially improve with the number of injected canaries ( § 3.3). The pseudocode of our approach is provided in Algorithm 1.

3.1 From DP to Lifted DP

To enlarge the design space of counter-examples, we introduce an equivalent definition of DP.

Definition 2 (Lifted differential privacy) . Let P denote a joint probability distribution over ( D 0 , D 1 , R ) where ( D 0 , D 1 ) ∈ Z ∗ ×Z ∗ is a pair of random datasets that are neighboring (as in the standard definition of neighborhood in Definition 1) with probability one and let R ⊂ R denote a random rejection set. We say that a randomized mechanism A : Z ∗ →R satisfies ( ε, δ ) -Lifted Differential Privacy (LiDP) for some ε ≥ 0 and δ ∈ [0 , 1] if, for all P independent of A , we have

In Appendix A.3, we discuss connections between Lifted DP and other existing extensions of DP, such as Bayesian DP and Pufferfish, that also consider randomized datasets. The following theorem shows that LiDP is equivalent to the standard DP, which justifies our framework of checking the above condition; if a mechanism A violates the above condition then it violates ( ε, δ ) -DP.

Theorem 3. A randomized algorithm A is ( ε, δ ) -LiDP iff A is ( ε, δ ) -DP.

A proof is provided in Appendix B. In contrast to DP, LiDP involves probabilities over both the internal randomness of the algorithm A and the distribution P over ( D 0 , D 1 , R ) . This gives the auditor greater freedom to search over a lifted space of joint distributions over the paired datasets and a rejection set; hence the name Lifted DP. Auditing LiDP amounts to constructing a randomized (as emphasized by the boldface letters) counter-example ( D 0 , D 1 , R ) that violates (3) as evidence.

3.2 From a Single Deterministic Canary to Multiple Random Canaries

Our strategy is to turn the LiDP condition in Eq. (3) into another condition in Eq. (4) below; this allows the auditor to reuse samples, running multiple hypothesis tests on each sample. This derivation critically relies on our carefully designed recipe that incorporates three crucial features: ( a ) binary hypothesis tests between pairs of stochastically coupled datasets containing K canaries and K -1 canaries, respectively, for some fixed integer K , ( b ) sampling those canaries i.i.d. from the same distribution, and ( c ) choice of rejection sets, where each rejection set only depends on a single left-out canary. We introduce the following recipe, also presented in Algorithm 1.

We fix a given training set D and a canary distribution P canary over Z . This ensures that the model under scrutiny is close to the use case. Under the alternative hypothesis, we train a model on a randomized training dataset D 1 = D ∪ { c 1 , . . . , c K } , augmented with K random canaries drawn i.i.d. from P canary . Conceptually, this is to be tested against K leave-one-out (LOO) null hypotheses. Under the k th null hypothesis for each k ∈ [ K ] , we construct a coupled dataset, D 0 ,k = D ∪ { c 1 , . . . , c k -1 , c k +1 , . . . c K } , with K -1 canaries, leaving the k th canary out. This coupling of K -1 canaries ensures that ( D 0 ,k , D 1 ) is neighboring with probability one. For each left-out canary, the auditor runs a binary hypothesis test with a choice of a random rejection set R k . We restrict R k to depend only on the canary c k that is being tested and not the index k . For example, R k can be the set of models achieving a loss on the canary c k below a predefined threshold τ .

The goal of this LOO construction is to reuse each trained private model to run multiple tests such that the averaged test statistic has a smaller variance for a given number of models . Under the standard definition of DP, one can still use the above LOO construction but with fixed and deterministic canaries. This gives no variance gain because evaluating P ( A ( D 0 ,k ) ∈ R k ) in Eq. (1) or its averaged counterpart (1 /K ) ∑ K k =1 P ( A ( D 0 ,k ) ∈ R k ) requires training one model to get one sample from the test statistic I ( A ( D 0 ,k ) ∈ R k ) . The key ingredient in reusing trained models is randomization.

We build upon the LiDP condition in Eq. (3) by noting that the test statistics are exchangeable for i.i.d. canaries. Specifically, we have for any k ∈ [ K ] that P ( A ( D 0 ,k ) ∈ R k ) = P ( A ( D 0 ,K ) ∈ R K ) = P ( A ( D 0 ,K ) ∈ R ′ j ) for any canary c ′ j drawn i.i.d. from P canary and its corresponding

Algorithm 1 Auditing Lifted DP

Input: Sample size n , number of canaries K , number of null tests m , DP mechanism A , training set D , canary generating distribution P canary, threshold τ , failure probability β , privacy δ ∈ [0 , 1] .

1: for i = 1 , . . . , n do

2: Randomly generate K + m canaries { c 1 , . . . , c K , c ′ 1 , . . . , c ′ m } i.i.d. from P canary .

4: Train two models θ 0 ←A ( D 0 ) and θ 1 ←A ( D 1 )

3: D 0 ← D ∪ { c 1 , . . . , c K -1 } , D 1 ← D ∪ { c 1 , . . . , c K }

6: Set p 1 ← XBernLower ( { x ( i ) } i ∈ [ n ] , β/ 2 ) and p 0 ← XBernUpper ( { y ( i ) } i ∈ [ n ] , β/ 2 )

5: Record test statistics x ( i ) ← ( I ( f θ 1 ( c k ) < τ ) ) K k =1 and y ( i ) ← ( I ( f θ 0 ( c ′ j ) < τ ) ) m j =1

7: Return ˆ ε n ← log ( ( p 1 -δ ) / p 0 ) and a guarantee that P ( ε < ˆ ε n ) ≤ β .

rejection set R ′ j that are statistically independent of D 0 ,K . Therefore, we can rewrite the right side of Eq. (3) using m i.i.d. test canaries c ′ 1 , . . . , c ′ m ∼ P canary and a single trained model A ( D 0 ,K ) as

Checking this condition is sufficient for auditing LiDP and, via Theorem 3, for auditing DP. For each model trained on D 1 , we record the test statistics of K (correlated) binary hypothesis tests. This is denoted by a random vector x = ( I ( A ( D 1 ) ∈ R k )) K k =1 ∈ { 0 , 1 } K , where R k is a rejection set that checks for the presence of the k th canary. Similar to the standard recipe, we train n models to obtain n i.i.d. samples x (1) , . . . , x ( n ) ∈ { 0 , 1 } K to estimate the left side of (4) using the empirical mean:

where the subscript one in ˆ µ 1 denotes that this is the empirical first moment. Ideally, if the K tests are independent, the corresponding confidence interval is smaller by a factor of √ K . In practice, the K tests are correlated and the size of the confidence interval depends on their correlation. We derive principled confidence intervals that leverage the empirically measured correlations in § 3.3. We can define y ∈ { 0 , 1 } m and its mean ˆ ν 1 analogously for the null hypothesis. We provide pseudocode in Algorithm 1 as an example guideline for applying our recipe to auditing DP training, where f θ ( z ) is the loss evaluated on an example z for a model θ . XBernLower() and XBernUpper() respectively return the lower and upper adaptive confidence intervals from § 3.3. We instantiate Algorithm 1 with concrete examples of canary design in § 5.

Our Recipe vs. Multiple Deterministic Canaries. An alternative with deterministic canaries would be to test between D 0 with no canaries and D 1 with K canaries such that we can get K samples from a single trained model on D 0 , one for each of the K test statistics { I ( A ( D 0 ) ∈ R k ) } K k =1 . However, due to the fact that D 1 and D 0 are now at Hamming distance K , this suffers from group privacy; we are required to audit for a much larger privacy leakage of ( Kε, (( e Kε -1) / ( e ε -1)) δ ) -DP. Under the (deterministic) LOO construction, this translates into (1 /K ) ∑ K k =1 P ( A ( D 1 ) ∈ R k ) ≤ e Kε (1 /K ) ∑ K k =1 P ( A ( D 0 ) ∈ R k ) + (( e Kε -1) / ( e ε -1)) δ , where if one canary violates the group privacy condition then the average also violates it. We can reuse a single trained model to get K test statistics in the above condition, but each canary is distinct and not any stronger than the one from ( ε, δ ) -DP auditing. One cannot obtain stronger counterexamples without sacrificing the sample gain. For example, we can repeat the same canary K times as proposed in [32]. This makes it easier to detect the canary, making a stronger counter-example, but there is no sample gain as we only get one test statistic per trained model. With deterministic canaries, there is no way to avoid this group privacy cost while our recipe does not incur it.

3.3 From Bernoulli Intervals to Higher-Order Exchangeable Bernoulli (XBern) Intervals

The LiDP condition in Eq. (4) critically relies on the canaries being sampled i.i.d. and the rejection set only depending on the corresponding canary. For such a symmetric design, auditing boils down to deriving a Confidence Interval (CI) for a special family of distributions that we call Exchangeable Bernoulli (XBern) . We derive the CI for the alternate hypothesis, i.e., the left side of Eq. (4). The CI under the null hypothesis is analogous.

z

1

z

2

z

1

+

z

2

z

1

+

z

2

+

σ

ξ

z

1

+

z

2

+

c

+

σ

ξ

µ

µ

µ

Computing the Bernstein C.I. by Root-Finding

|

µ

-

ˆ

µ

|

a

√

µ

(1

-

µ

) +

b

10

-

3

10

-

2

10

-

1

Failure Probability

β

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Coefficient

Leading Coefficient of Wilson vs.

Bernstein

Bernstein

Wilson

̂

Recall that x k := I ( A ( D 1 ) ∈ R k ) denotes the test statistic for the k th canary. By the symmetry of our design, x ∈ { 0 , 1 } K is an exchangeable random vector that is distributed as an exponential family. Further, the distribution of x is fully defined by a K -dimensional parameter ( µ 1 , . . . , µ K ) where µ glyph[lscript] is the glyph[lscript] th moment of x . We call this family XBern. Specifically, this implies permutation invariance of the higher-order moments: E [ x j 1 · · · x j glyph[lscript] ] = E [ x k 1 · · · x k glyph[lscript] ] for any distinct sets of indices ( j l ) l ∈ [ glyph[lscript] ] and ( k l ) l ∈ [ glyph[lscript] ] for any glyph[lscript] ≤ K . For example, µ 1 := E [(1 /K ) ∑ K k =1 x k ] , which is the LHS of (4).

Using samples from this XBern, we aim to derive a CI on µ 1 around the empirical mean ˆ µ 1 in (5). Bernstein's inequality applied to our test statistic m 1 := (1 /K ) ∑ K k =1 x k gives, w.p. at least 1 -β . Bounding Var( m 1 ) ≤ µ 1 (1 -µ 1 ) since m 1 ∈ [0 , 1] a.s. and numerically solving the above inequality for µ 1 gives a CI that scales as 1 / √ n -see Figure 2 (left). We call this the 1 st -order Bernstein bound , as it depends only on the 1 st moment µ 1 . Our strategy is to measure the (higher-order) correlations between x k 's to derive a tighter CI that adapts to the given instance. This idea applies to any standard CI. We derive and analyze the higher order Bernstein intervals here, and experiments use Wilson intervals from Appendix C; see also Figure 2 (right).

Concretely, we can leverage the 2 nd order correlation by expanding

Ideally, when the second order correlation µ 2 -µ 2 1 = E [ x 1 x 2 ] -E [ x 1 ] E [ x 2 ] equals 0 , we have Var( m 1 ) = µ 1 (1 -µ 1 ) /K , a factor of K improvement over the worst-case. Our higher-order CIs adapt to the actual level of correlation of x by further estimating the 2 nd moment µ 2 from samples. Let µ 2 be the first-order Bernstein upper bound on µ 2 such that P ( µ 2 ≤ µ 2 ) ≥ 1 -β . On this event,

Combining this with (6) gives us the 2 nd -order Bernstein bound on µ 1 , valid w.p. 1 -2 β . Since µ 2 glyph[lessorsimilar] ˆ µ 2 +1 / √ n where ˆ µ 2 is the empirical estimate of µ 2 , the 2 nd -order bound scales as where constants and log factors are omitted. Thus, our 2 nd -order CI can be as small as 1 / √ nK + 1 /n 3 / 4 (when ˆ µ 2 1 ≈ µ 2 ) or as large as 1 / √ n (in the worst-case). With small enough correlations of | ˆ µ 2 -ˆ µ 2 1 | = O (1 /K ) , this suggests a choice of K = O ( √ n ) to get CI of 1 /n 3 / 4 . In practice, the correlation is controlled by the design of the canary. For the Gaussian mechanism with random canaries, the correlation indeed empirically decays as 1 /K , as we see in § 4. Thus, the CI decreases monotonically with K . This is also true for the CI under the null hypothesis with K replaced by m .

However, there is a qualitative difference between the null and alternate hypotheses. Estimation under the alternate hypothesis incurs a larger bias as the number K of canaries increases - this is due

√

2

9

2

11

2

13

2

15

Number of trials

n

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

16

×

gain

4

×

gain

ε

= 2

.

0

, K

=

n, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

1

.

45

1

.

50

1

.

55

1

.

60

1

.

65

1

.

70

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 4096

, d

= 10

4

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

2

1

.

3

1

.

4

1

.

5

1

.

6

1

.

7

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 4096

, K

=

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

√

to the additional randomness from adding more canaries into the training process, as illustrated in Figure 1. The optimal choice of K balances the bias and variance. On the other hand, estimation under the null hypothesis incurs no bias and a larger number m of test canaries always helps. We return to the bias-variance tradeoff in § 4.

Higher-order intervals. We can recursively apply this method by expanding the variance of higherorder statistics, and derive higher-order Bernstein bounds. The next recursion uses empirical 3 rd and 4 th moments ˆ µ 3 and ˆ µ 4 to get the 4 th -order Bernstein bound which scales as

Ideally, when the 4 th -order correlation is small enough, | ˆ µ 4 -ˆ µ 2 2 | = O (1 /K ) , (along with the 2 nd-order correlation, ˆ µ 2 -ˆ µ 2 1 ) this 4 th-order CI scales as 1 /n 7 / 8 with a choice of K = O ( n 3 / 4 ) improving upon the 2 nd-order CI of 1 /n 3 / 4 . We can recursively derive even higher-order CIs, but we find in § 4 that the gains diminish rapidly. In general, the glyph[lscript] th order Bernstein bounds achieve CIs scaling as 1 /n (2 glyph[lscript] -1) / 2 glyph[lscript] with a choice of K = O ( n ( glyph[lscript] -1) /glyph[lscript] ) . This shows that the higher-order CI decreases in the order glyph[lscript] of the correlations used; we refer to Appendix C for details.

Proposition 4. For any positive integer glyph[lscript] that is a power of two and K = glyph[ceilingleft] n ( glyph[lscript] -1) /glyph[lscript] glyph[ceilingright] , suppose we have n samples from a K -dimensional XBern distribution with parameters ( µ 1 , . . . , µ K ) . If all glyph[lscript] ′ th -order correlations scale as 1 /K , i.e., | µ 2 glyph[lscript] ′ -µ 2 glyph[lscript] ′ | = O (1 /K ) , for all glyph[lscript] ′ ≤ glyph[lscript] and glyph[lscript] ′ is a power of two, then the glyph[lscript] th -order Bernstein bound is | µ 1 -ˆ µ 1 | = O (1 /n (2 glyph[lscript] -1) / (2 glyph[lscript] ) ) .

4 Simulations: Auditing the Gaussian Mechanism

Setup. We consider a simple sum query q ( D ) = ∑ z ∈ D z over the unit sphere Z = { z ∈ R d : ‖ z ‖ 2 = 1 } . We want to audit a Gaussian mechanism that returns q ( D ) + σ ξ with standard Gaussian ξ ∼ N (0 , I d ) and σ calibrated to ensure ( ε, δ ) -DP. We assume black-box access, where we do not know what mechanism we are auditing and we only access it through samples of the outcomes. A white-box audit is discussed in § A.1. We apply our new recipe with canaries sampled uniformly at random from Z . Following standard methods [e.g. 23], we declare that a canary c k is present if c glyph[latticetop] k A ( D ) > τ for a threshold τ learned from separate samples. For more details and additional results, see Appendix E. A broad range of values of K (between 32 and 256 in Figure 3 middle) leads to good performance in auditing LiDP. We use K = √ n (as suggested by our analysis in (8)), m = K test canaries, and the 2 nd -order Wilson estimator (as gains diminish rapidly afterward) as a reliable default setting.

Sample Complexity Gains. In Figure 3 (left), the proposed approach of injecting K canaries with the 2 nd order Wilson interval (denoted 'LiDP +2 nd -Order Wilson') reduces the number of trials, n , needed to reach the same empirical lower bound, ˆ ε , by 4 × to 16 × , compared to the baseline of injecting a single canary (denoted 'DP+Wilson'). We achieve ˆ ε n = 0 . 85 with n = 4096 (while the baseline requires n = 65536 ) and ˆ ε n = 0 . 67 with n = 1024 (while the baseline requires n = 4096 ).

√

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

-

0

.

10

-

0

.

05

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Difference of empirical

ˆ

ε

ε

= 4

.

0

, n

= 4096

, d

= 10

4

K

=

√

n

Var. Gain

Net Gain

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Correlation Measure

ε

= 2

.

0

, n

= 4096

, d

= 10

5

1

/K

|

ˆ

µ

2

1

-

ˆ

µ

2

|

|

ˆ

µ

2

2

-

ˆ

µ

4

|

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

Moment statistics

ε

= 1

.

0

, n

= 65536

, K

=

n

1

/d

|

ˆ

µ

2

1

-

ˆ

µ

2

|

|

ˆ

µ

2

2

-

ˆ

µ

4

|

Number of Canaries and Bias-Variance Tradeoffs. In Fig. 3 (middle), LiDP auditing with 2 nd /4 th -order CIs improve with increasing canaries up to a point and then decreases. This is due to a bias-variance tradeoff, which we investigate further in Figure 4 (left). Let ˆ ε ( K,glyph[lscript] ) denote the empirical privacy lower bound with K canaries using an glyph[lscript] th order interval, e.g., the baseline is ˆ ε (1 , 1) . Let the bias from injecting K canaries and the variance gain from glyph[lscript] th -order interval respectively be

In Figure 4 (left), the gain ∆ Bias ( K ) from bias is negative and gets worse with increasing K ; when testing for each canary, the K -1 other canaries introduce more randomness that makes the test more private and hence lowers the ˆ ε (see also Figure 1. The gain ∆ Var ( K ) in variance is positive and increases with K before saturating. This improved 'variance' of the estimate is a key benefit of our framework. The net improvement ˆ ε ( K,glyph[lscript] ) -ˆ ε (1 , 1) is a sum of these two effects. This trade-off between bias and variance explains the concave shape of ˆ ε in K .

Next, we see from Figure 5 that a larger number m of test canaries always helps, i.e., it does not incur a similar bias-variance tradeoff. Indeed, this is because larger m does not lead to any additional bias, as discussed in § 3.3. However, the gains quickly saturate, so m = K is a reliable default, especially for K ≥ 16 or so; we refer to Appendix E for more plots and details.

Correlation between Canaries. Based on (8), this improvement in the variance can further be examined by looking at the term | ˆ µ 2 -ˆ µ 2 1 | that leads to a narrower 2 nd -order Wilson interval. The log-log plot of this term in Figure 4 (middle) is nearly parallel to the dotted 1 /K line (slope = -0 . 93 ), meaning that it decays roughly as 1 /K (note that the log-log plot of y = cx a is a straight line with slope a ). This indicates that we get close to a 1 / √ nK confidence interval as desired. Similarly, we get that | ˆ µ 4 -ˆ µ 2 2 | decays roughly as 1 /K (slope = -1 . 05 ). However, the 4 th -order estimator offers only marginal additional improvements in the small ε regime (see Appendix E). Thus, the gain diminishes rapidly in the order of our estimators.

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

m

=

K

Gaussian,

ε

= 2

, n

= 4096

, K

= 64

, d

= 10

4

2

8

2

9

2

10

2

11

2

12

2

13

2

14

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

8

×

gain

Laplace,

ε

= 16

, K

= 1024

, d

= 10

4

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

075

0

.

100

0

.

125

0

.

150

0

.

175

0

.

200

0

.

225

0

.

250

Empirical lower bound

ˆ

ε

K

=

√

n

Laplace,

ε

= 16

, n

= 4096

, d

= 10

4

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

Effect of Dimension on the Bias and Variance. As the dimension d increases, the LiDP-based lower bound becomes tighter monotonically as we show in Figure 3 (right). This is due to both the bias gain, as in (10), improving (less negative) and the variance gain improving (more positive). With increasing d , the variance of our estimate reduces because the canaries become less correlated. Guided by Eq. (8,9), we measure the relevant correlation measures | ˆ µ 2 -ˆ µ 2 1 | and | ˆ µ 4 -ˆ µ 2 2 | in Figure 4 (right). Both decay approximate as 1 /d (slope = -1 . 06 and -1 . 00 respectively, ignoring the outlier at d = 10 6 ). This suggests that, for Gaussian mechanisms, the corresponding XBern distribution resulting from our recipe behaves favorably as d increases.

Auditing Other Mechanisms. The proposed LiDP-based auditing recipe is agnostic to the actual privacy mechanism. We audit a Laplace mechanism with canaries drawn uniformly at random over the unit glyph[lscript] 1 ball in Figure 4 (center and right). The results are qualitatively similar to those of the Gaussian mechanism, leading to an eight-fold improvement in the sample complexity over the baseline.

5 Lifting Existing Canary Designs

Several prior works [e.g., 32, 41, 45, 60], focus on designing stronger canaries to improve the lower bound on ε . We provide two concrete examples of how to lift these canary designs to be compatible with our framework while inheriting their strengths. We refer to Appendix D for further details.

We impose two criteria on the distribution P canary over canaries for auditing LiDP. First, the injected canaries are easy to detect, so that the probabilities on the left side of (4) are large and those on the right side are small. Second, a canary c ∼ P canary, if included in the training of a model θ , is unlikely to change the membership of θ ∈ R c ′ for an independent canary c ′ ∼ P canary . Existing canary designs already impose the first condition to audit DP using (1). The second condition ensures that the canaries are uncorrelated, allowing our adaptive CIs to be smaller, as we discussed in § 3.3.

Data Poisoning via Tail Singular Vectors. ClipBKD [32] adds as canaries the tail singular vector of the input data (e.g., images). This ensures that the canary is out of distribution, allowing for easy detection. We lift ClipBKD by defining the distribution P canary as the uniform distribution over the p tail singular vectors of the input data. If p is small relative to the dimension d of the data, then they are still out of distribution, and hence, easy to detect. For the second condition, the orthogonality of the singular vectors ensures the interaction between canaries is minimal, as measured empirically.

Random Gradients. The approach of [2] samples random vectors (of the right norm) as canary gradients, assuming a grey-box access where we can inject gradients. Since random vectors are nearly orthogonal to any fixed vector in high dimensions, their presence is easy to detect with a dot product. Similarly, any two i.i.d. canary gradients are roughly orthogonal, leading to minimal interactions.

6 Experiments

We compare the proposed LiDP auditing recipe relative to the standard one for DP training of machine learning models. Our code is available online. 1

Setup. Wetest with two classification tasks: FMNIST [65] is a 10-class grayscale image classification dataset, while Purchase-100 is a sparse tabular dataset with 600 binary features and 100 classes [19, 53]. We train a linear model and a multi-layer perceptron (MLP) with 2 hidden layers using DP-SGD [1] to achieve ( ε, 10 -5 ) -DP with varying values of ε . The training is performed using cross-entropy for a fixed epoch budget and a batch size of 100 . We refer to Appendix F for details.

Auditing. We audit the LiDP using the two types of canaries from § 5: data poisoning and random gradient canaries. We vary the number K of canaries and the number n of trials. We track the empirical lower bound obtained from the Wilson family of confidence intervals. We compare this with auditing DP, which coincides with auditing LiDP with K = 1 canary. We audit only the final model in all the experiments.

Sample Complexity Gain. Table 1 shows the reduction in the sample complexity from auditing LiDP. For each canary type, auditing LiDP is better 11 out of the 12 settings considered. The average improvement (i.e., the harmonic mean over the table) for data poisoning is 2 . 3 × , while for random

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

3

.

14

×

Purchase-MLP + Data Poisoning,

ε

= 2

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

2

.

69

×

FMNIST-Linear + Random Gradient,

ε

= 2

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

Purchase-MLP + Data Poisoning,

ε

= 2

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

05

0

.

10

0

.

15

0

.

20

Empirical lower bound

ˆ

ε

FMNIST-Linear + Random Gradient,

ε

= 4

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

gradients, it is 3 . 0 × . Since each trial is a full model training run, this improvement can be quite significant in practice. This improvement can also be seen visually in Figure 6 (left two).

Number of Canaries. We see from Figure 6 (right two) that LiDP auditing on real data behaves similarly to Figure 3 in § 4. We also observe the bias-variance tradeoff in the case of data poisoning (center right). The choice K = √ n is competitive with the best value of K , validating our heuristic. Overall, these results show that the insights from § 4 hold even with the significantly more complicated DP mechanism involved in training models privately.

Additional Comparisons. We show in § F.4 that our LiDP auditing with generally outperforms the Bayesian auditing approach of Zanella-B´ eguelin et al. [69]. This also suggests an interesting future research direction: adapt Bayesian credible intervals for LiDP auditing to get the best of both worlds.

7 Conclusion

We introduce a new framework for auditing differentially private learning. Diverging from the standard practice of adding a single deterministic canary, we propose a new recipe of adding multiple i.i.d. random canaries. This is made rigorous by an expanded definition of privacy that we call LiDP. We provide novel higher-order confidence intervals that can automatically adapt to the level of correlation in the data. We empirically demonstrate that there is a potentially significant gain in sample dependence of the confidence intervals, achieving a favorable bias-variance tradeoff.

Although any rigorous statistical auditing approach can benefit from our framework, it is not yet clear how other popular approaches [e.g. 12] for measuring memorization can be improved with randomization. Bridging this gap is an important practical direction for future research. It is also worth considering how our approach can be adapted to audit the diverse definitions of privacy in machine learning [e.g. 28].

Broader Impact. Auditing private training involves a trade-off between the computational cost and the tightness of the guarantee. This may not be appropriate for all practical settings. For deployment in production, it is worth further studying approaches with minimal computational overhead [e.g. 2, 12].

References

M. Abadi, A. Chu, I. J. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep Learning with Differential Privacy. In Proc. of the 2016 ACM SIGSAC Conf. on Computer and Communications Security (CCS'16) , pages 308-318, 2016.

G. Andrew, P. Kairouz, S. Oh, A. Oprea, H. B. McMahan, and V. Suriyakumar. One-shot Empirical Privacy Estimation for Federated Learning. arXiv preprint arXiv:2302.03098 , 2023.

¨ O. Askin, T. Kutta, and H. Dette. Statistical Quantification of Differential Privacy: A Local Approach. In 2022 IEEE Symposium on Security and Privacy (SP) , pages 402-421. IEEE, 2022.

B. Balle, G. Barthe, M. Gaboardi, J. Hsu, and T. Sato. Hypothesis Testing Interpretations and Renyi Differential Privacy. In The 23rd International Conference on Artificial Intelligence and Statistics , volume 108, pages 2496-2506. PMLR, 2020.

G. Barthe, B. K¨ opf, F. Olmedo, and S. Zanella Beguelin. Probabilistic Relational Reasoning for Differential Privacy. ACM SIGPLAN Notices , 47(1):97-110, 2012.

G. Barthe, G. Danezis, B. Gr´ egoire, C. Kunz, and S. Zanella-Beguelin. Verified Computational Differential Privacy with Applications to Smart Metering. In 2013 IEEE 26th Computer Security Foundations Symposium , pages 287-301. IEEE, 2013.

G. Barthe, M. Gaboardi, E. J. G. Arias, J. Hsu, C. Kunz, and P.-Y. Strub. Proving differential privacy in Hoare logic. In 2014 IEEE 27th Computer Security Foundations Symposium , pages 411-424. IEEE, 2014.

R. Bhaskar, A. Bhowmick, V. Goyal, S. Laxman, and A. Thakurta. Noiseless Database Privacy. In International Conference on the Theory and Application of Cryptology and Information Security , pages 215-232. Springer, 2011.

B. Bichsel, S. Steffen, I. Bogunovic, and M. Vechev. DP-Sniper: Black-Box Discovery of Differential Privacy Violations using Classifiers. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 391-409. IEEE, 2021.

M. Bun and T. Steinke. Concentrated Differential Privacy: Simplifications, Extensions, and Lower Bounds. In Theory of Cryptography Conference , pages 635-658. Springer, 2016.

M. Bun, J. Ullman, and S. Vadhan. Fingerprinting Codes and the Price of Approximate Differential Privacy. SIAM Journal on Computing , 47(5):1888-1938, 2018.

N. Carlini, C. Liu, ´ U. Erlingsson, J. Kos, and D. Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In USENIX Security Symposium , volume 267, 2019.

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 2021) , 2021.

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer. Membership Inference Attacks from First Principles. In IEEE Symposium on Security and Privacy (SP) , pages 1519-1519, Los Alamitos, CA, USA, May 2022. IEEE Computer Society. doi: 10.1109/SP46214.2022.00090. URL https://doi.ieeecomputersociety.org/10.1109/SP46214.2022.00090 .

N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying Memorization Across Neural Language Models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=TatRHT_1cK .

Y. Chen and A. Machanavajjhala. On the privacy properties of variants on the sparse vector technique. arXiv preprint arXiv:1508.07306 , 2015.

Z. Ding, Y. Wang, G. Wang, D. Zhang, and D. Kifer. Detecting violations of differential privacy. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security , pages 475-489, 2018.

K. Dixit, M. Jha, S. Raskhodnikova, and A. Thakurta. Testing Lipschitz Property over Product Distribution and its Applications to Statistical Data Privacy. In Theory of Cryptography Conference , pages 418-436. Springer, 2013.

W. C. DMDave, Todd B. Acquire valued shoppers challenge, 2014. URL https://kaggle. com/competitions/acquire-valued-shoppers-challenge .

R. M. Dudley. Uniform Central Limit Theorems , volume 142. Cambridge university press, 2014.

C. Dwork and G. N. Rothblum. Concentrated Differential Privacy. arXiv preprint arXiv:1603.01887 , 2016.

C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating Noise to Sensitivity in Private Data Analysis. In Proc. of the Third Conf. on Theory of Cryptography (TCC) , pages 265-284, 2006. URL http://dx.doi.org/10.1007/11681878_14 .

C. Dwork, A. Smith, T. Steinke, J. Ullman, and S. Vadhan. Robust Traceability from Trace Amounts. In 2015 IEEE 56th Annual Symposium on Foundations of Computer Science , pages 650-669. IEEE, 2015.

M. Gaboardi, A. Haeberlen, J. Hsu, A. Narayan, and B. C. Pierce. Linear Dependent Types for Differential Privacy. In Proceedings of the 40th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages , pages 357-370, 2013.

A. C. Gilbert and A. McMillan. Property Testing for Differential Privacy. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 249-258. IEEE, 2018.

C. Guo, B. Karrer, K. Chaudhuri, and L. van der Maaten. Bounding Training Data Reconstruction in Private (Deep) Learning. In Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 8056-8071, 17-23 Jul 2022.

Y. Han, J. Jiao, and T. Weissman. Minimax estimation of divergences between discrete distributions. IEEE Journal on Selected Areas in Information Theory , 1(3):814-823, 2020.

A. Hannun, C. Guo, and L. van der Maaten. Measuring Data Leakage in Machine-Learning Models with Fisher Information. In UAI , pages 760-770, 2021.

J. Hayes, S. Mahloujifar, and B. Balle. Bounding Training Data Reconstruction in DP-SGD, 2023.

N. Homer, S. Szelinger, M. Redman, D. Duggan, W. Tembe, J. Muehling, J. V. Pearson, D. A. Stephan, S. F. Nelson, and D. W. Craig. Resolving individuals contributing trace amounts of DNA to highly complex mixtures using high-density SNP genotyping microarrays. PLoS genetics , 4(8):e1000167, 2008.

S. L. Hyland and S. Tople. An Empirical Study on the Intrinsic Privacy of SGD. arXiv preprint arXiv:1912.02919 , 2019.

M. Jagielski, J. Ullman, and A. Oprea. Auditing differentially private machine learning: How private is private SGD? Advances in Neural Information Processing Systems , 33:22205-22216, 2020.

B. Jayaraman and D. Evans. Evaluating Differentially Private Machine Learning in Practice. In USENIX Security Symposium , pages 1895-1912, 2019.

M. Johnson. fix prng key reuse in differential privacy example. https://github.com/ google/jax/pull/3646 , 2020.

P. Kairouz, S. Oh, and P. Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385. PMLR, 2015.

D. Kifer and A. Machanavajjhala. Pufferfish: A framework for mathematical privacy definitions. ACM Transactions on Database Systems (TODS) , 39(1):1-36, 2014.

T. Kutta, ¨ O. Askin, and M. Dunsche. Lower Bounds for R´ enyi Differential Privacy in a Black-Box Setting. arXiv preprint arXiv:2212.04739 , 2022.

X. Liu and S. Oh. Minimax Optimal Estimation of Approximate Differential Privacy on Neighboring Databases. In Advances in Neural Information Processing Systems , volume 32, 2019.

F. Lu, J. Munoz, M. Fuchs, T. LeBlond, E. Zaresky-Williams, E. Raff, F. Ferraro, and B. Testa. A General Framework for Auditing Differentially Private Machine Learning. In NeurIPS , 2022.

M. Lyu, D. Su, and N. Li. Understanding the sparse vector technique for differential privacy. arXiv preprint arXiv:1603.01699 , 2016.

S. Maddock, A. Sablayrolles, and P. Stock. CANIFE: Crafting canaries for empirical privacy measurement in federated learning. In ICLR , 2023.

M. Malek Esmaeili, I. Mironov, K. Prasad, I. Shilov, and F. Tramer. Antipodes of label differential privacy: Pate and alibi. Advances in Neural Information Processing Systems , 34: 6934-6945, 2021.

F. D. McSherry. Privacy integrated queries: an extensible platform for privacy-preserving data analysis. In Proceedings of the 2009 ACM SIGMOD International Conference on Management of data , pages 19-30. ACM, 2009.

I. Mironov. R´ enyi Differential Privacy. In 30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017 , pages 263-275. IEEE Computer Society, 2017.

M. Nasr, S. Songi, A. Thakurta, N. Papernot, and N. Carlini. Adversary Instantiation: Lower Bounds for Differentially Private Machine Learning. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 866-882. IEEE, 2021.

M. Nasr, J. Hayes, T. Steinke, B. Balle, F. Tram` er, M. Jagielski, N. Carlini, and A. Terzis. Tight Auditing of Differentially Private Machine Learning. arXiv preprint arXiv:2302.07956 , 2023.

M. Park. Bug-fix. https://github.com/mijungi/vips_code/commit/ 4e32042b66c960af618722a43c32f3f2dda2730c , 2018.

N. Ponomareva, H. Hazimeh, A. Kurakin, Z. Xu, C. Denison, H. B. McMahan, S. Vassilvitskii, S. Chien, and A. Thakurta. How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy. arXiv Preprint , 2023.

M. A. Rahman, T. Rahman, R. Lagani` ere, N. Mohammed, and Y. Wang. Membership Inference Attack against Differentially Private Deep Learning Model. Trans. Data Priv. , 11(1):61-79, 2018.

J. Reed and B. C. Pierce. Distance makes the types grow stronger: a calculus for differential privacy. In ACM Sigplan Notices , volume 45, pages 157-168. ACM, 2010.

I. Roy, S. T. Setty, A. Kilzer, V . Shmatikov, and E. Witchel. Airavat: Security and privacy for MapReduce. In NSDI , volume 10, pages 297-312, 2010.

S. Sankararaman, G. Obozinski, M. I. Jordan, and E. Halperin. Genomic privacy and limits of individual detection in a pool. Nature genetics , 41(9):965-967, 2009.

R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

S. Song, K. Chaudhuri, and A. D. Sarwate. Stochastic gradient descent with differentially private updates. In 2013 IEEE Global Conference on Signal and Information Processing , pages 245-248. IEEE, 2013.

S. Song, Y. Wang, and K. Chaudhuri. Pufferfish privacy mechanisms for correlated data. In Proceedings of the 2017 ACM International Conference on Management of Data , pages 1291-1306, 2017.

T. Steinke, M. Nasr, and M. Jagielski. Privacy auditing with one (1) training run. arXiv preprint arXiv:2305.08846 , 2023.

T. Stevens, I. C. Ngong, D. Darais, C. Hirsch, D. Slater, and J. P. Near. Backpropagation Clipping for Deep Learning with Differential Privacy. arXiv preprint arXiv:2202.05089 , 2022.

F. Tramer. Tensorflow privacy issue #153: Incorrect comparison between privacy amplification by iteration and DP-SGD. https://github.com/tensorflow/privacy/issues/153 , 2020.

F. Tram` er, R. Shokri, A. San Joaquin, H. Le, M. Jagielski, S. Hong, and N. Carlini. Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security , CCS '22, page 2779-2792, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450394505. doi: 10.1145/3548606.3560554. URL https://doi.org/10.1145/3548606.3560554 .

F. Tramer, A. Terzis, T. Steinke, S. Song, M. Jagielski, and N. Carlini. Debugging differential privacy: A case study for privacy auditing. arXiv preprint arXiv:2202.12219 , 2022.

A. Triastcyn and B. Faltings. Bayesian Differential Privacy for Machine Learning. In International Conference on Machine Learning , pages 9583-9592. PMLR, 2020.

M. C. Tschantz, D. Kaynar, and A. Datta. Formal verification of differential privacy for interactive systems. Electronic Notes in Theoretical Computer Science , 276:61-79, 2011.

S. Vadhan. The complexity of differential privacy. In Tutorials on the Foundations of Cryptography , pages 347-450. Springer, 2017.

G. Valiant and P. Valiant. Estimating the unseen: improved estimators for entropy and other properties. Journal of the ACM (JACM) , 64(6):1-41, 2017.

H. Xiao, K. Rasul, and R. Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. arXiv Preprint , 2017.

B. Yang, I. Sato, and H. Nakagawa. Bayesian Differential Privacy on Correlated Data. In Proceedings of the 2015 ACM SIGMOD international conference on Management of Data , pages 747-762, 2015.

J. Ye, A. Maddi, S. K. Murakonda, V. Bindschaedler, and R. Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. arXiv preprint arXiv:2111.09679 , 2021.

S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pages 268-282. IEEE, 2018.

S. Zanella-B´ eguelin, L. Wutschitz, S. Tople, A. Salem, V. R¨ uhle, A. Paverd, M. Naseri, and B. K¨ opf. Bayesian Estimation of Differential Privacy. arXiv preprint arXiv:2206.05199 , 2022.

Appendix

Table of Contents

A Related Work

Prior to [17], privacy auditing required some access to the description of the mechanism. [57, 24, 43, 50, 51, 62] provide platforms with a specific set of functions to be used to implement the mechanism, where the end-to-end privacy of the source code can be automatically verified. Closest to our setting is the work of [18], where statistical testing was first proposed for privacy auditing, given access to an oracle that returns the exact probability measure of the mechanism. However, the guarantee is only for a relaxed notion of DP from [8] and the run-time depends super-linearly on the size of the output domain.

The pioneering work of [17] was the first to propose practical methods to audit privacy claims given a black-box access to a mechanism. The focus was on simple queries that do not involve any training of models. This is motivated by [16, 40], where even simple mechanisms falsely reported privacy guarantees. For such mechanisms, sampling a large number, say 500 , 000 in [17], of outputs is computationally easy, and no effort was made in [17] to improve the statistical trade-off. However, for private learning algorithms, training such a large number of models is computationally prohibitive. The main focus of our framework is to improve this statistical trade-off for auditing privacy. We first survey recent breakthroughs in designing stronger canaries, which is orthogonal to our main focus.

A.1 Auditing Private Machine Learning with Strong Canaries

Recent breakthroughs in auditing are accelerated by advances in privacy attacks, in particular membership inference. An attacker performing membership inference would like to determine if a particular data sample was part of the training set. Early work on membership inference [11, 23, 30, 52] considered algorithms for statistical methods, and more recent work demonstrates black-box attacks on ML algorithms [14, 33, 53, 67, 68]. [33] compares different notions of privacy by measuring privacy empirically for the composition of differential privacy [35], concentrated differential privacy [10, 21], and R´ enyi differential privacy [44]. This is motivated by [49], which compares different DP mechanisms by measuring the success rates of membership inference attacks. [31] attempts to measure the intrinsic privacy of stochastic gradient descent (without additional noise as in DP-SGD) using canary designs from membership inference attacks. Membership inference attacks have been shown to have higher success when the adversary can poison the training dataset [59].

A more devastating privacy attack is the extraction or reconstruction of the training data, which is particularly relevant for generative models such as large language models (LLMs). Several papers showed that LLMs tend to memorize their training data [13, 15], allowing an adversary to prompt the generative models and extract samples of the training set. The connection between the ability of an attacker to perform training data reconstruction and DP guarantees has been shown in recent work [26, 29].

When performing privacy auditing, a stronger canary design increases the success of the adversary in the distinguishing test and improves the empirical privacy bounds. The resulting hypothesis test can tolerate larger confidence intervals and requires less number of samples. Recent advances in privacy auditing have focused on designing such stronger canaries. [32] designs data poisoning canaries, in the direction of the lowest variance of the training data. This makes the canary out of distribution, making it easier to detect. [45] proposes attack surfaces of varying capabilities. For example, a gradient attack canary returns a gradient of choice when accessed by DP-SGD. It is shown that, with more powerful attacks, the canaries become stronger and the lower bounds become higher. [60] proposes using an example from the baseline training dataset, after changing the label, and introduces a search procedure to find a strong canary. More recently, [46] proposes a significantly improved auditing scheme for DP-SGD under a white-box access model where ( i ) the auditor knows that the underlying mechanism is DP-SGD with a spherical Gaussian noise with unknown variance, and ( ii ) all intermediate models are revealed. Since each coordinate of the model update provides an independent sample from the same Gaussian distribution, sample complexity is dramatically improved. [41] proposes CANIFE, a novel canary design method that finds a strong data poisoning canary adaptively under the federated learning scenario.

Prior work in this space shows that privacy auditing can be performed via privacy attacks, such as membership inference or reconstruction, and strong canary design results in better empirical privacy bounds. We emphasize that our aim is not to innovate on optimizing canary design for performing

privacy auditing. Instead, our framework can seamlessly adopt recently designed canaries and inherit their strengths as demonstrated in § 5 and § 6.

A.2 Improving Statistical Trade-offs in Auditing

Eq. (2) points to two orthogonal directions that can potentially improve the sample dependence: designing stronger canaries and improving the sample dependence of the confidence intervals. The former was addressed in the previous section. There is relatively less work in improving the statistical dependence, which our framework focuses on.

Given a pair of neighboring datasets, ( D 0 , D 1 ) , and for a query with discrete output in a finite space, the statistical trade-off of estimating privacy parameters was studied in [25] where a plug-in estimator is shown to achieve an error of O ( √ de 2 ε /n ) , where d is the size of the discrete output space. [38] proposes a sub-linear sample complexity algorithm that achieves an error scaling as √ de ε /n log n , based on polynomial approximation of a carefully chosen degree to optimally trade-off bias and variance motivated by [27, 64]. Similarly, [37] provides a lower bound for auditing R´ enyi differential privacy. [9] trains a classifier for the binary hypothesis test and uses the classifier to design rejection sets. [3] proposes local search to find the rejection set efficiently. More recently, [69] proposes numerical integration over a larger space of false positive rate and true positive rate to achieve better sample complexity of the confidence region in the two-dimensional space. Our framework can be potentially applied to this confidence region scenario, which is an important future research direction.

A.3 Connections to Other Notions of Differential Privacy

Similar to Lifted DP, a line of prior works [36, 55, 61, 66] generalizes DP to include a distributional assumption over the dataset. However, unlike Lifted DP, they are motivated by the observation that DP is not sufficient for preserving privacy when the samples are highly correlated. For example, upon releasing the number of people infected with a highly contagious flu within a tight-knit community, the usual Laplace mechanism (with sensitivity one) is not sufficient to conceal the likelihood of one member getting infected when the private count is significantly high. One could apply group differential privacy to hide the contagion of the entire population, but this will suffer from excessive noise. Ideally, we want to add a noise proportional to the expected size of the infection. Pufferfish privacy, introduced in [36], achieves this with a generalization of DP that takes into account prior knowledge of a class of potential distributions over the dataset. A special case of ε -Pufferfish with a specific choice of parameters recovers a special case of our Lifted DP in Eq. (3) with δ = 0 [36, Section 3.2], where a special case of Theorem 3 has been proven for pure DP [36, Theorem 3.1]. However, we want to emphasize that our Lifted DP is motivated by a completely different problem of auditing differential privacy and is critical to breaking the barriers in the sample complexity.

B Properties of Lifted DP and Further Details

B.1 Equivalence Between DP and LiDP

In contrast to the usual ( ε, δ ) -DP in Eq. (1), the probability in LiDP is over both the internal randomness of the algorithm A and the distribution P over the triplet ( D 0 , D 1 , R ) . Since we require that the definition holds only for lifted distributions P that are independent of the algorithm A , it is easy to show its equivalence to the usual notion of ( ε, δ ) -DP.

Theorem 3. A randomized algorithm A is ( ε, δ ) -LiDP iff A is ( ε, δ ) -DP.

Proof. Suppose A is ( ε, δ ) -LiDP. Fix a pair of neighboring datasets D 0 , D 1 and an outcome R ⊂ R . Define P D 0 ,D 1 ,R as the point mass on ( D 0 , D 1 , R ) , i.e.,

d P D 0 ,D 1 ,R ( D ′ 0 , D ′ 1 , R ′ ) = ✶ ( D ′ 0 = D 0 , D ′ 1 = D 1 , R ′ = R ) , so that P A , P D 0 ,D 1 ,R ( A ( D 0 ) ∈ R ) = P ( A ( D 0 ) ∈ R ) and similarly for D 1 . Then, applying the definition of ( ε, δ ) -LiDP w.r.t. the distribution P D 0 ,D 1 ,R gives

Since this holds for any neighboring datasets D 0 , D 1 and outcome set R , we get that A is ( ε, δ ) -DP.

Conversely, suppose that A is ( ε, δ ) -DP. For any distribution P over pairs of neighboring datasets D 0 , D 1 and outcome set R , we have by integrating the DP definition

Next, we use the law of iterated expectation to get

where ( ∗ ) followed from the independence of A and P . Plugging this and the analogous expression for P ( A ( D 1 ) ∈ R ) into (11) gives us that A is ( ε, δ ) -LiDP.

B.2 Auditing LiDP with Different Notions of Neighborhood

We describe how to modify the recipe of § 3 for other notions of neighborhoods of datasets. The notion of the neighborhood in Definition 1 is also known as the 'add-or-remove' neighborhood.

Replace-one Neighborhood. Two datasets D,D ′ ∈ Z ∗ are considered neighboring if | D | = | D ′ | and | D \ D ′ | = | D ′ \ D | = 1 . Roughly speaking, this leads to privacy guarantees that are roughly twice as strong as the add-or-remove notion of the neighborhood in Definition 1, as the worst-case sensitivity of the operation is doubled. We refer to [48, 63] for more details. Just like Definition 1, Definition 2 can also be adapted to this notion of neighborhood.

Auditing LiDP with Replace-one Neighborhood. The recipe of § 3.2 can be slightly modified for this notion of neighborhood. The main difference is that the null hypothesis must now use K canaries as well, with one fresh canary.

The alternative hypothesis is the same - we train a model on a randomized training dataset D 1 = D ∪ { c 1 , . . . , c K } augmented with K random canaries drawn i.i.d. from P canary. Under the j th null hypothesis for each j ∈ [ K ] , we construct a coupled dataset D 0 ,j = D ∪{ c 1 , . . . , c j -1 , c ′ j , c j +1 , . . . , c K } , where c ′ j is a fresh canary drawn i.i.d. from P canary . This coupling ensures that ( D 0 ,j , D 1 ) are neighboring with probability one. We restrict the rejection region R j to now depend only on c j and not in the index j , e.g., we test for the present of c j . 2

Note the symmetry of the setup. Testing for the presence of c j in D 0 ,j is exactly identical to testing for the presence of c ′ j in D 1 . Thus, we can again rewrite the LiDP condition as

Note the subtle difference between (4) and (12): both sides depend only on A ( D 1 ) and we have completely eliminated the need to train models on K -1 canaries.

From here on, the rest of the recipe is identical to § 3 and Algorithm 1; we construct XBern confidence intervals for both sides of (12) and get a lower bound ˆ ε .

C Confidence Intervals for Exchangeable Bernoulli Means

We give a rigorous definition of the multivariate Exchangeable Bernoulli (XBern) distributions and derive their confidence intervals. We also give proof of the correctness of the confidence intervals.

Definition 5 (XBern Distributions) . A random vector ( x 1 , . . . , x K ) ∈ { 0 , 1 } K is said to be distributed as XBern K ( µ 1 , . . . , µ K ) if:

x 1 , . . . , x K is exchangeable, i.e., the vector ( x 1 , . . . , x K ) is identical in distribution to ( x π (1) , . . . , x π ( K ) ) for any permutation π : [ K ] → [ K ] , and,

for each glyph[lscript] = 1 , . . . , K , we have E [ m glyph[lscript] ] = µ glyph[lscript] , where

We note that the XBern K distribution is fully determined by its K moments µ 1 , . . . , µ K . For K = 1 , XBern 1 ( µ 1 ) = Bernoulli ( µ 1 ) is just the Bernoulli distribution.

The moments m glyph[lscript] satisfy a computationally efficient recurrence.

Proposition 6. Let x ∼ XBern K ( µ 1 , . . . , µ K ) . We have the following for all 1 ≤ glyph[lscript] ≤ K m 1 :

Computing the first glyph[lscript] moments can thus be done in O ( K + glyph[lscript] ) time rather than O ( K glyph[lscript] ) by naively computing the sum in (13).

Proof of Proposition 6. We show that this holds for any fixed vector ( x 1 , . . . , x K ) ∈ { 0 , 1 } K and their corresponding moments m 1 , . . . , m K as defined in (13). Define the sums s 1 := ∑ K j =1 x j = Km 1 and

We can compute s glyph[lscript] by a counting argument: each term in the sum is non-zero only if all glyph[lscript] of the x j 1 , . . . , x j glyph[lscript] are non-zero. The number of non-zero terms is equal to number of ways of selecting glyph[lscript] items from a total of ∑ K j =1 x j = s 1 items, i.e.,

Combining Equations (15) and (16) completes the proof.

Notation. In this section, we are interested in giving confidence intervals on the mean µ 1 of a XBern K ( µ 1 , . . . , µ K ) random variable from n i.i.d. observations:

We will define the confidence intervals using the empirical mean

as well as the higher-order moments for glyph[lscript] ∈ [ K ] ,

C.1 Non-Asymptotic Confidence Intervals

We start by giving non-asymptotic confidence intervals for the XBern distributions based on the Bernstein bound, recalled below [e.g. 20, Thm. 1.3.2].

Algorithm 2 First-Order Bernstein Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

1: Set µ 1 as the unique solution of x ∈ [0 , ˆ µ 1 ] such that

Output: Confidence intervals [ µ 1 , µ 1 ] such that P ( µ 1 < µ 1 ) ≤ β and P ( µ 1 > µ 1 ) ≤ β .

if it exists, else set µ 1 = 0 .

2: Set µ 1 as the unique solution of x ∈ [ ˆ µ 1 , 1] such that

if it exists, else set µ 1 = 1 .

3: return µ 1 , µ 1 .

Lemma 7 (Bernstein inequality) . Let x 1 , . . . , x n be independent random variables with E [ x i ] = 0 , Var( x i ) ≤ σ 2 , and | x i | ≤ b almost surely. Let µ 1 := (1 /n ) ∑ n i =1 x i . For any t > 0 , we have

Equivalently, we have with probability at least 1 -β that

The last simplification is obtained by using √ a + b ≤ √ a + √ b for scalars a, b ≥ 0 .

C.1.1 First-Order Bernstein Intervals

The first-order Bernstein interval only depends on the empirical mean µ 1 and is given in Algorithm 2.

Proposition 8. Consider Algorithm 2 with inputs n i.i.d. samples x (1) , . . . , x ( n ) i.i.d. ∼ XBern K ( µ 1 , . . . , µ K ) for some K ≥ 1 . Then, its outputs µ 1 , µ 1 satisfy P ( µ 1 ≤ µ 1 ) ≥ 1 -β and P ( µ 1 ≥ µ 1 ) ≥ 1 -β .

Proof. Applying Bernstein's inequality to m 1 := (1 /K ) ∑ K j =1 x j , we have with probability 1 -β that

where we used that Var( m 1 ) ≤ µ 1 (1 -µ 1 ) since m 1 ∈ [0 , 1] a.s. We see from Figure 2 that µ 1 is that largest value of µ 1 that satisfies the above inequality, showing that it is an upper confidence bound. Similarly, we get that µ 1 is a valid lower confidence bound.

Algorithm 3 Second-Order Bernstein Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

Output: Confidence intervals [ µ 1 , µ 1 ] such that P ( µ 1 < µ 1 ) ≤ β and P ( µ 1 > µ 1 ) ≤ β .

3: Set µ 2 as the unique solution of x ∈ [ ˆ µ 2 , 1] such that

2: Set ˆ µ glyph[lscript] = (1 /n ) ∑ n i =1 m ( i ) glyph[lscript] for glyph[lscript] = 1 , 2 .

if it exists, else set µ 2 = 1 .

4: Set µ 1 as the unique solution of x ∈ [0 , ˆ µ 1 ] such that

if it exists, else set µ 1 = 0 .

5: Set µ 1 as the unique solution of x ∈ [ ˆ µ 1 , 1] such that

if it exists, else set µ 1 = 1 .

6: return µ 1 , µ 1 .

C.1.2 Second-Order Bernstein Intervals

The second-order Bernstein interval only depends on the first two empirical moments µ 1 and µ 2 . It is given in Algorithm 3. The algorithm is based on the calculation

where we used x 2 j = x j since it is an indicator.

Proposition 9. Consider Algorithm 3 with inputs n i.i.d. samples x (1) , . . . , x ( n ) i.i.d. ∼ XBern K ( µ 1 , . . . , µ K ) for some K ≥ 2 . Then, its outputs µ 1 , µ 1 satisfy P ( µ 1 ≥ µ 1 ) ≥ 1 -β and P ( µ 1 ≤ µ 1 ) ≥ 1 -β and P ( µ 1 ≤ µ 1 ≤ µ 1 ) ≤ 1 -3 β 2 .

Proof. Algorithm 3 computes the correct 2nd moment m ( i ) 2 due to Proposition 6. Applying Bernstein's inequality to m 2 , we get P ( µ 2 ≤ µ 2 ) ≥ 1 -β/ 2 (see also the proof of Proposition 8). Next, from Bernstein's inequality applied to m 1 , we leverage (17) to say that with probability at least 1 -β/ 2 , we have

Together with the result on µ 2 , we have with probability at least 1 -β that

Algorithm 4 Fourth-Order Bernstein Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

Output: Confidence intervals [ µ 1 , µ 1 ] such that P ( µ 1 < µ 1 ) ≤ β and P ( µ 1 > µ 1 ) ≤ β .

3: For glyph[lscript] = 3 , 4 , set µ glyph[lscript] as the unique solution of x ∈ [ ˆ µ glyph[lscript] , 1] such that

2: Set ˆ µ glyph[lscript] = (1 /n ) ∑ n i =1 m ( i ) glyph[lscript] for glyph[lscript] = 1 , 2 , 3 , 4 .

4: Set µ as the unique solution, if it exists, of x [ ˆ µ , 1] such that

if it exists, else set µ glyph[lscript] = 1 . 2 ∈ 2

where σ 2 2 ( · , · , · ) is as defined in (18). Else set µ 2 = 1 . 5: Set µ 1 as the unique solution of x ∈ [0 , ˆ µ 1 ] such that

6: Set µ as the unique solution of x [ ˆ µ , 1] such that

if it exists, else set µ 1 = 0 . 1 ∈ 1

if it exists, else set µ = 1

7: return , 1 .

1 . µ 1 µ

We can verify that the output µ 1 is the largest value of µ 1 ≤ 1 that satisfies the above inequality. Similarly, µ 1 is obtained as a lower Bernstein bound on m 1 with probability at least 1 -β . By the union bound, µ 1 ≤ µ 1 ≤ µ 1 holds with probability at least 1 -3 β/ 2 , since we have three invocations of Bernstein's inequality, each with a failure probability of β/ 2 .

C.1.3 Fourth-Order Bernstein Intervals

The fourth-order Bernstein interval depends on the first four empirical moments µ 1 , . . . , µ 4 . It is given in Algorithm 4. The derivation of the interval is based on the calculation

Proposition 10. Consider Algorithm 4 with inputs n i.i.d. samples x (1) , . . . , x ( n ) i.i.d. ∼ XBern K ( µ 1 , . . . , µ K ) for some K ≥ 4 . Then, its outputs µ 1 , µ 1 satisfy P ( µ 1 ≥ µ 1 ) ≥ 1 -β and P ( µ 1 ≤ µ 1 ) ≥ 1 -β and P ( µ 1 ≤ µ 1 ≤ µ 1 ) ≤ 1 -5 β 4 .

Algorithm 5 First-Order Wilson Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

Output: Asymptotic confidence intervals [ µ 1 , µ 1 ] such that lim n →∞ P ( µ 1 < µ 1 ) ≤ β and

1: Set µ 1 < µ 1 as the roots of the quadratic in x :

2: return µ 1 , µ 1 .

Algorithm 6 Second-Order Wilson Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

Output: Asymptotic confidence intervals [ µ 1 , µ 1 ] such that lim n →∞ P ( µ 1 < µ 1 ) ≤ β and lim n →∞ P ( µ 1 > µ 1 ) ≤ β .

2: Set ˆ µ glyph[lscript] = (1 /n ) ∑ n i =1 m ( i ) glyph[lscript] for glyph[lscript] = 1 , 2 . 3: Let µ 2 be the larger root of the quadratic in x :

4: Set µ 1 < µ 1 as the roots of the quadratic in x :

5: return µ 1 , µ 1 .

Proof. Algorithm 4 computes the correct moments m ( i ) glyph[lscript] for glyph[lscript] ≤ 4 due to Proposition 6. Applying Bernstein's inequality to m 3 and m 4 , we get P ( µ glyph[lscript] ≤ µ glyph[lscript] ) ≥ 1 -β/ 4 for glyph[lscript] = 3 , 4 (see also the proof of Proposition 8).

Next, from Bernstein's inequality applied to m 2 , we have with probability at least 1 -β/ 4 that

Combining this with the results on µ 3 , µ 4 with the union bound, we get with probability at least 1 -3 β/ 4 that

Finally, plugging this into a Bernstein bound on m 1 using the variance calculation from (17) (also see the proof of Proposition 9) completes the proof.

C.2 Asymptotic Confidence Intervals

We derive asymptotic versions of the Algorithms 2 to 4 using the Wilson confidence interval.

The Wilson confidence interval is a tightening of the constants for the Bernstein confidence interval

where Z β is the (1 -β ) -quantile of the standard Gaussian. Essentially, this completely eliminates the 1 /n term, while the coefficient of the 1 / √ n term improves from √ 2 log(1 /β ) to Z β -see Figure 2.

Algorithm 7 Fourth-Order Wilson Intervals

Input: Random vectors x (1) , . . . , x ( n ) ∼ XBern K ( µ 1 , . . . , µ K ) with unknown parameters, failure probability β ∈ (0 , 1) .

Output: Asymptotic confidence intervals [ µ 1 , µ 1 ] such that lim n →∞ P ( µ 1 < µ 1 ) ≤ β and lim P ( µ > µ ) β .

3: For glyph[lscript] = 3 , 4 , let µ glyph[lscript] be the larger root of the quadratic in x :

2: Set ˆ µ glyph[lscript] = (1 /n ) ∑ n i =1 m ( i ) glyph[lscript] for glyph[lscript] = 1 , 2 , 3 , 4 .

4: Let µ 2 be the larger root of the quadratic in x :

5: Set µ 1 < µ 1 as the roots of the quadratic in x :

6: return µ 1 , µ 1 .

The Wilson approximation holds under the assumption that ( µ 1 -ˆ µ 1 ) / √ Var( m 1 ) /n d ≈ N (0 , 1) and using a Gaussian confidence interval. This can be formalized by the central limit theorem.

Lemma 11 (Lindeberg-L´ evy Central Limit Theorem) . Consider a sequence of independent random variables y (1) , y (2) , . . . with finite moments E [ y ( i ) ] = µ < ∞ and E ( y ( i ) -µ ) 2 = σ 2 < ∞ for each i . Then, the empirical mean ˆ µ ( n ) 1 = (1 /n ) ∑ n i =1 y ( i ) based on n samples satisfies for all t ∈ R . Consequently, we have,

The finite moment requirement above is satisfied in our case because all our random variables are bounded between 0 and 1 . The asymptotic confidence interval implied by (19) is known as the Wilson confidence interval . It states that

We give the Wilson-variants of Algorithms 2 to 4 respectively in Algorithms 5 to 7. Apart from the fact that the Wilson intervals are tighter, we can also solve the equations associated with the Wilson intervals in closed form as they are simply quadratic equations (i.e., without the need for numerical root-finding). The following proposition shows their correctness.

Proposition 12. Consider n i.i.d. samples x (1) , . . . , x ( n ) i.i.d. ∼ XBern K ( µ 1 , . . . , µ K ) as inputs to Algorithms 5 to 7. Then, their outputs µ 1 , µ 1 satisfy lim n →∞ P ( µ 1 ≤ µ 1 ) ≥ 1 -β and lim n →∞ P ( µ 1 ≥ µ 1 ) ≥ 1 -β and lim n →∞ P ( µ 1 ≤ µ 1 ≤ µ 1 ) ≤ 1 -Cβ if

(a) K ≥ 1 and C = 2 for Algorithm 5,

(b) K ≥ 2 and C = 3 / 2 for Algorithm 6, and

(c) K ≥ 4 and C = 5 / 4 for Algorithm 7.

We omit the proof as it is identical to those of Propositions 8 to 10 except that it uses the Wilson interval from Lemma 11 rather than the Bernstein interval.

C.3 Scaling of Higher-Order Bernstein Bounds (Proposition 4)

We now re-state and prove Proposition 4.

Proposition 4. For any positive integer glyph[lscript] that is a power of two and K = glyph[ceilingleft] n ( glyph[lscript] -1) /glyph[lscript] glyph[ceilingright] , suppose we have n samples from a K -dimensional XBern distribution with parameters ( µ 1 , . . . , µ K ) . If all glyph[lscript] ′ th -order correlations scale as 1 /K , i.e., | µ 2 glyph[lscript] ′ -µ 2 glyph[lscript] ′ | = O (1 /K ) , for all glyph[lscript] ′ ≤ glyph[lscript] and glyph[lscript] ′ is a power of two, then the glyph[lscript] th -order Bernstein bound is | µ 1 -ˆ µ 1 | = O (1 /n (2 glyph[lscript] -1) / (2 glyph[lscript] ) ) .

Proof. We are given n samples from an XBern distribution x ∈ { 0 , 1 } K with parameters ( µ 1 , . . . , µ K ) , where µ glyph[lscript] := E [ m glyph[lscript] ] with

By exchangeability, it also holds that µ glyph[lscript] = E [ x 1 · · · x glyph[lscript] ] . Assuming a confidence level 1 -β < 1 and K = glyph[ceilingleft] n ( glyph[lscript] -1) /glyph[lscript] glyph[ceilingright] , the 1 st-order Bernstein bound gives

where σ 2 glyph[lscript] := Var( m glyph[lscript] ) . Expanding σ 2 glyph[lscript] , it is easy to show that it is dominated by the 2 glyph[lscript] th-order correlation | µ 2 glyph[lscript] -µ 2 glyph[lscript] | :

Note that our Bernstein confidence interval does not use the fact that the higher-order correlations are small. We only use that assumption to bound the resulting size of the confidence interval in the analysis. Applying the assumption that all the higher order correlations are bounded by 1 /K , i.e., | µ 2 glyph[lscript] ′ -µ 2 glyph[lscript] ′ | = O (1 /K ) , we get that | ˆ µ 2 glyph[lscript] ′ -ˆ µ 2 glyph[lscript] ′ | = O (1 /K + √ σ 2 2 glyph[lscript] ′ /n ) . Applying this recursively into (20), we get that

for any glyph[lscript] that is a power of two. For an glyph[lscript] th -order Bernstein bound, we only use moment estimates up to glyph[lscript] and bound σ 2 glyph[lscript] ≤ 1 . The choice of K = n ( glyph[lscript] -1) /glyph[lscript] gives the desired bound: | µ 1 -ˆ µ 1 | = O (1 /n (2 glyph[lscript] -1) / (2 glyph[lscript] ) ) .

D Canary Design for Lifted DP: Details

The canary design employed in the auditing of the usual ( ε, δ ) -DP can be easily extended to create distributions over canaries to audit LiDP. We give some examples for common classes of canaries.

Setup. We assume a supervised learning setting with a training dataset D train = { ( x i , y i ) } N i =1 and a held-out dataset D val = { ( x i , y i ) } N + N ′ i = N +1 of pairs of input x i ∈ X and output y i ∈ Y . We then aim to minimize the average loss

where L ( z, θ ) is the loss incurred by model θ on input-output pair z = ( x, y ) .

In the presence of canaries c 1 , . . . , c k , we instead aim to minimize the objective where L canary ( c, θ ) is the loss function for a canary c -this may or may not coincide with the usual loss L .

Goals of Auditing DP. The usual practice is to set D 0 = D train and D 1 = D 0 ∪{ c } and R ≡ R c for a canary c . Recall from the definition of ( ε, δ ) -DP in (1), we have for some class C of canaries. The goal then is to find canaries c that approximate the sup over C . Since this goal is hard, one usually resorts to finding canaries whose effect can be 'easy to detect' in some sense.

Goals of Auditing LiDP. Let P denote a probability distribution over a set

We sample K canaries c 1 , . . . , c K i.i.d. ∼ P canary and set canary C of allowed canaries.

From the definition of ( ε, δ ) -LiDP in (3), we have for some class C of canaries. The goal then is to approximate the distribution P canary for each choice of the canary set C . Since this is hard, we will attempt to define a distribution over canaries that are easy to detect (similar to the case of auditing DP). Following the discussion in § 3.3, auditing LiDP benefits the most when the canaries are uncorrelated. To this end, we will also impose the restriction that a canary c ∼ P canary, if included in the training of a model θ , is unlikely to change the membership of θ ∈ R c ′ for an i.i.d. canary c ′ ∼ P canary that is independent of c .

We consider two choices of the canary set C (as well as the outcome set R c and the loss L canary): data poisoning, and random gradients.

D.1 Data Poisoning

We describe the data poisoning approach known as ClipBKD [32] that is based on using the tail singular vectors of the input data matrix and its extension to auditing LiDP.

Let X = ( x glyph[latticetop] 1 ; · · · ; x glyph[latticetop] N ) ∈ R N × d denote the matrix with the datapoints x i ∈ R d as rows. Let X = ∑ min { N,d } i =1 σ i u i v glyph[latticetop] i be the singular value decomposition of X with σ 1 ≤ σ 2 ≤ · · · be the singular values arranged in ascending order. Let Y denote the set of allowed labels.

For this section, we take the set of allowed canaries C = { αv 1 , αv 2 , . . . , αv min { N,d } } × Y as the set of the right singular vector of X scaled by a given factor α > 0 together with any possible target from Y . We take L canary ( c, θ ) = L ( c, θ ) to be the usual loss function, and the output set R c to be the loss-thresholded set

for some threshold τ .

Auditing DP. The ClipBKD approach [32] uses a canary with input αv 1 , the singular vector corresponding to the smallest singular value, scaled by a parameter α > 0 . The label is taken as y glyph[star] ( αv 1 ) , where is the target that has the highest loss on input x under the empirical risk minimizer θ glyph[star] 0 = arg min θ ∈R F ( θ ) . Since a unique θ glyph[star] 0 is not guaranteed for deep nets nor can we find it exactly,

10

1

10

2

10

3

10

4

10

5

10

6

Sample size

n

10

-

3

10

-

2

Width of Confidence Interval

Lower Confidence Interval

Clopper-Pearson

Jeffreys

Wilson

10

1

10

2

10

3

10

4

10

5

10

6

Sample size

n

10

-

3

10

-

2

10

-

1

Width of Confidence Interval

Upper Confidence Interval

we train 100 models with different random seeds and pick the class y that yields the highest average loss over these runs.

Auditing LiDP. We extend ClipBKD to define a probability distribution over a given number p of canaries. We take

i.e., P canary is the uniform distribution over the p singular vectors corresponding to the smallest singular values.

D.2 Random Gradients

The update poisoning approach of [2] relies of supplying gradients c ∼ Uniform ( B R (0 , r )) that are uniform on the Euclidean ball of a given radius r . This is achieved by setting the loss of the canary as

is the desired vector c .

The set R c is a threshold of the dot product

for a given threshold τ . This set is analogous to the loss-based thresholding of (23) in that both can be written as L canary ( c, θ ) ≤ τ .

Auditing DP and LiDP. The random gradient approach of [2] relies on defining a distribution P canary ≡ Uniform ( B R (0 , r )) over canaries. It can be used directly to audit LiDP.

E Simulations with the Gaussian Mechanism: Details and More Results

Here, we give the full details and additional results of auditing the Gaussian mechanism with synthetic data in § 4.

E.1 Experiment Setup

Fix a dimension d and a failure probability β ∈ (0 , 1) . Suppose we have a randomized algorithm A that returns a noisy sum of its inputs with the goal of simulating the Gaussian mechanism. Concretely, the input space Z = { z ∈ R d : ‖ z ‖ 2 ≤ 1 } is the unit ball in R d . Given a finite set D ∈ Z ∗ , we sample a vector ξ ∼ N (0 , σ 2 I d ) of a given variance σ 2 and return

√

2

9

2

11

2

13

2

15

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, K

=

n, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, K

= 64

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, K

= 256

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, K

= 1024

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, K

=

√

n, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, K

= 64

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, K

= 256

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, K

= 1024

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

6

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, K

=

√

n, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, K

= 64

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, K

= 256

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

0

.

8

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, K

= 1024

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

3

.

50

3

.

75

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, K

=

√

n, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

3

.

50

3

.

75

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, K

= 64

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

3

.

50

3

.

75

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, K

= 256

, d

= 10

5

2

9

2

11

2

13

2

15

Number of trials

n

1

.

75

2

.

00

2

.

25

2

.

50

2

.

75

3

.

00

3

.

25

3

.

50

3

.

75

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, K

= 1024

, d

= 10

5

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

To isolate the effect of the canaries, we set our original dataset D = { 0 d } as a singleton with the vector of zeros in R d . Since we are in the blackbox setting, we do not assume that this is known to the auditor.

Since we add ξ ∼ N (0 , σ 2 I d ) , it follows that the operation A ( · ) is ( α, α/ (2 σ 2 ) ) -RDP for every α > 1 . Thus, A ( · ) is ( ε δ , δ ) -DP where

DP Upper Bound. The non-private version of our function computes the sum D ↦→ ∑ z ∈ D z . where each z ∈ D is a canary. Hence, the glyph[lscript] 2 sensitivity of the operation is ∆ 2 = max x ∈ D ‖ x ‖ 2 = 1 , as stated in § 4.

Auditing LiDP. We follow the recipe of Algorithm 1. We set the rejection region R = R τ ( c K ) as a function of the canary c K that differs between D 0 and D 1 , where based on [4, Thm. 21]. This can be shown to be bounded above by 1 σ √ 2 log 1 δ + 1 2 σ 2 [44, Prop. 3]. By Theorem 3, it follows that the operation A ( · ) is also ( ε δ , δ ) -LiDP.

and τ ∈ R is a tuned threshold.

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 1024

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

250

0

.

275

0

.

300

0

.

325

0

.

350

0

.

375

0

.

400

0

.

425

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 4096

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

34

0

.

36

0

.

38

0

.

40

0

.

42

0

.

44

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 16384

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

40

0

.

41

0

.

42

0

.

43

0

.

44

0

.

45

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 65536

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

40

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 1024

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 4096

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

78

0

.

80

0

.

82

0

.

84

0

.

86

0

.

88

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 16384

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

84

0

.

85

0

.

86

0

.

87

0

.

88

0

.

89

0

.

90

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 65536

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

1

.

15

1

.

20

1

.

25

1

.

30

1

.

35

1

.

40

1

.

45

1

.

50

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 1024

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

1

.

50

1

.

55

1

.

60

1

.

65

1

.

70

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 4096

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

1

.

66

1

.

68

1

.

70

1

.

72

1

.

74

1

.

76

1

.

78

1

.

80

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 16384

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

1

.

74

1

.

76

1

.

78

1

.

80

1

.

82

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 65536

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

2

.

40

2

.

45

2

.

50

2

.

55

2

.

60

2

.

65

2

.

70

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 1024

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

3

.

0

3

.

1

3

.

2

3

.

3

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 4096

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

3

.

40

3

.

45

3

.

50

3

.

55

3

.

60

3

.

65

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 16384

, d

= 10

5

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

3

.

575

3

.

600

3

.

625

3

.

650

3

.

675

3

.

700

3

.

725

3

.

750

3

.

775

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 65536

, d

= 10

5

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

We evaluate empirical privacy auditing methods by how large the lower bound ˆ ε is - the higher the lower bound, the better the confidence interval.

Methods Compared. An empirical privacy auditing method is defined by the type of privacy auditing (DP or LiDP) and the type of confidence intervals. We compare the following auditing methods:

DP + Wilson : We audit the usual ( ε, δ ) -DP with K = 1 canary. This corresponds exactly to auditing LiDP with K = 1 . We use the 1st-Order Wilson confidence intervals for a fair comparison with the other LiDP auditing methods. This performs quite similarly to the other intervals used in the literature, cf. Figure 7.

LiDP + 1st-Order Wilson : We audit LiDP with K canaries with the 1st-Order Wilson confidence interval. This method cannot leverage the shrinking of the confidence intervals from higher-order estimates.

LiDP + 2nd/4th-Order Wilson : Weaudit LiDP with K > 1 canaries using the higher-order Wilson confidence intervals. Unless stated otherwise, we use m = K test canaries.

Parameters of the Experiment. We vary the following parameters in the experiment:

Number of trials n ∈ { 2 8 , 2 10 , · · · , 2 16 } .

Dimension d ∈ { 10 2 , 10 3 , . . . , 10 6 } .

Number of canaries k ∈ { 1 , 2 , 2 2 , . . . , 2 10 } .

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

225

0

.

250

0

.

275

0

.

300

0

.

325

0

.

350

0

.

375

0

.

400

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

34

0

.

36

0

.

38

0

.

40

0

.

42

0

.

44

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

37

0

.

38

0

.

39

0

.

40

0

.

41

0

.

42

0

.

43

0

.

44

0

.

45

Empirical lower bound

ˆ

ε

ε

= 1

.

0

, n

= 65536

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

40

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 65536

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

0

1

.

1

1

.

2

1

.

3

1

.

4

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

2

1

.

3

1

.

4

1

.

5

1

.

6

1

.

7

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

1

1

.

2

1

.

3

1

.

4

1

.

5

1

.

6

1

.

7

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

0

1

.

2

1

.

4

1

.

6

1

.

8

Empirical lower bound

ˆ

ε

ε

= 4

.

0

, n

= 65536

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

8

2

.

0

2

.

2

2

.

4

2

.

6

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

8

2

.

0

2

.

2

2

.

4

2

.

6

2

.

8

3

.

0

3

.

2

3

.

4

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

1

.

0

1

.

5

2

.

0

2

.

5

3

.

0

3

.

5

Empirical lower bound

ˆ

ε

ε

= 8

.

0

, n

= 65536

, K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

DP upper bound ε ∈ { 1 , 2 , 4 , 8 } .

We fix the DP parameter δ = 10 -5 and the failure probability β = 0 . 05 .

Tuning the threshold τ . For each confidence interval scheme, we repeat the estimation of the lower bound ˆ ε ( τ ) for a grid of thresholds τ ∈ Γ on a holdout set of n trials. We fix the best threshold τ ∗ = arg max τ ∈ Γ ˆ ε ( τ ) that gives the largest lower bound ˆ ε ( τ ) from the grid Γ . We then fix the threshold τ ∗ and report numbers over a fresh set of n trials.

Randomness and Repetitions. We repeat each experiment 25 times (after fixing the threshold) with different random seeds and report the mean and standard error.

E.2 Additional Experimental Results

Effect of the Number m of Test Canaries. In Figure 13, we vary the number m of test canaries while keeping the number K of training canaries fixed. We see that the 2nd and 4th-order Wilson estimators lead to monotonically improving lower bound ˆ ε as m increases. However, these improvements soon flatten out between m = 8 and m = 32 depending on other parameters. In particular, the default value of m = K is nearly or equal to the best value of m for K ≥ 16 .

We give additional experimental results, expanding on the plots shown in Figures 3 and 4:

Figure 8 shows the effect of varying the number of trials n , similar to Figure 3 (left).

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 1

.

0

, n

= 1024

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 1

.

0

, n

= 4096

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 1

.

0

, n

= 16384

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 1

.

0

, n

= 65536

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 2

.

0

, n

= 1024

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 2

.

0

, n

= 4096

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 2

.

0

, n

= 16384

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

10

-

2

Correlation Measure

ε

= 2

.

0

, n

= 65536

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

7

10

-

5

10

-

3

Correlation Measure

ε

= 4

.

0

, n

= 1024

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

7

10

-

5

10

-

3

Correlation Measure

ε

= 4

.

0

, n

= 4096

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

9

10

-

7

10

-

5

10

-

3

Correlation Measure

ε

= 4

.

0

, n

= 16384

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

10

10

-

8

10

-

6

10

-

4

10

-

2

Correlation Measure

ε

= 4

.

0

, n

= 65536

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

12

10

-

10

10

-

8

10

-

6

10

-

4

10

-

2

Correlation Measure

ε

= 8

.

0

, n

= 1024

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

11

10

-

9

10

-

7

10

-

5

10

-

3

Correlation Measure

ε

= 8

.

0

, n

= 4096

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

12

10

-

10

10

-

8

10

-

6

10

-

4

10

-

2

Correlation Measure

ε

= 8

.

0

, n

= 16384

, d

= 10

5

2

2

2

4

2

6

2

8

2

10

Number of canaries

K

10

-

13

10

-

11

10

-

9

10

-

7

10

-

5

10

-

3

Correlation Measure

ε

= 8

.

0

, n

= 65536

, d

= 10

5

1

/K

ˆ

µ

2

1

ˆ

µ

2

ˆ

µ

2

2

ˆ

µ

4

|

-

|

|

-

|

Figure 9 shows the effect of varying the number of canaries k , similar to Figure 3 (middle).

Figure 10 shows the effect of varying the data dimension d , similar to Figure 3 (right).

Figure 11 shows the effect of varying the number of canaries k on the moment estimates, similar to Figure 4 (right).

Figure 12 shows the effect of varying the data dimension d on the moment estimates, similar to Figure 4 (right).

We observe that the insights discussed in § 4 hold across a wide range of the parameter values. In addition, we make the following observations.

The benefit of higher-order confidence estimators. We see from Figures 8 to 10 that the higherorder Wilson estimators lead to larger relative improvements at smaller ε . On the other hand, they perform similarly at large ε (e.g., ε = 8 ) to the lower-order estimators.

4th-Order Wilson vs. 2nd-Order Wilson. We note that the 4th-order Wilson interval outperforms the 2nd-order Wilson interval at ε = 1 , while the opposite is true at large ε = 8 . At intermediate values of ε , both behave very similarly. We suggest the 2nd-order Wilson interval as a default because it is nearly as good as or better than the 4th-order variant across the board, but is easier to implement.

√

√

|

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 1

.

0

, n

= 1024

, K

=

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

Moment statistics

ε

= 1

.

0

, n

= 4096

, K

=

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

Moment statistics

ε

= 1

.

0

, n

= 16384

, K

=

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

Moment statistics

ε

= 1

.

0

, n

= 65536

, K

=

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 2

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 2

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 2

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 2

.

0

, n

= 65536

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 4

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 4

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 4

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 4

.

0

, n

= 65536

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 8

.

0

, n

= 1024

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 8

.

0

, n

= 4096

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 8

.

0

, n

= 16384

, K

=

√

n

10

2

10

3

10

4

10

5

10

6

Dimension

d

10

-

8

10

-

7

10

-

6

10

-

5

10

-

4

10

-

3

Moment statistics

ε

= 8

.

0

, n

= 65536

, K

=

√

n

1

/d

ˆ

µ

2

1

ˆ

µ

2

ˆ

µ

2

2

ˆ

µ

4

-

|

|

-

|

4

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

ε

= 2

.

0

, n

= 4096

, K

= 2

, d

= 10

4

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

ε

= 2

.

0

, n

= 4096

, K

= 4

, d

= 10

4

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

ε

= 2

.

0

, n

= 4096

, K

= 16

, d

= 10

4

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

ε

= 2

.

0

, n

= 4096

, K

= 64

, d

= 10

4

2

1

2

3

2

5

2

7

2

9

Number of test canaries

m

ε

= 2

.

0

, n

= 4096

, K

= 256

, d

= 10

m

=

K

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

F Experiments: Details and More Results

We describe the detailed experimental setup here.

F.1 Training Details: Datasets, Models

We consider two datasets, FMNIST and Purchase-100. Both are multiclass classification datasets trained with the cross-entropy loss using stochastic gradient descent (without momentum) for a fixed epoch budget.

√

√

FMNIST : FMNIST or FashionMNIST [65] is a classification of 28 × 28 grayscale images of various articles of clothing into 10 classes. It contains 60K train images and 10K test images. The dataset is available under the MIT license. We experiment with two models: a linear model and a multi-layer perceptron (MLP) with 2 hidden layers of dimension 256 each. We train each model for 30 epochs with a batch size of 100 and a fixed learning rate of 0.02 for the linear model and 0.01 for the MLP.

Purchase-100 : The Purchase dataset is based on Kaggle's 'acquire valued shoppers' challenge [19] that records the shopping history of 200K customers. The dataset is available publicly on Kaggle but the owners have not created a license as far as we could tell. We use the preprocessed version of [53] 3 where the input is a 600-dimensional binary vector encoding the shopping history. The classes are obtained by grouping the records into 100 clusters using k -means. We use a fixed subsample of 20K training points and 5K test points. The model is a MLP with 2 hidden layers of 256 units each. It is trained for 100 epochs with a batch size of 100 and a fixed learning rate of 0.05.

F.2 DP and Auditing Setup

We train each dataset-model pair with DP-SGD [1]. The noise level is calibrated so that the entire training algorithm satisfies ( ε, δ ) -differential privacy with ε varying and δ = 10 -5 fixed across all experiments. We tune the per-example gradient clip norm for each dataset, model, and DP parameter ε so as to maximize the validation accuracy.

Auditing Setup. We follow the LiDP auditing setup described in Appendix B.2. Recall that auditing LiDP with K = 1 canaries and corresponds exactly with auditing DP. We train n ∈ { 125 , 250 , 500 , 1000 } trials, where each trial corresponds to a model trained with K canaries in each run. We try two methods for canary design (as well as their corresponding rejection regions), as discussed in Appendix D: data poisoning and random gradients.

With data poisoning for FMNIST, we define the canary distribution P canary as the uniform distribution over the last p = 284 principal components (i.e., principal components 500 to 784 ). For Purchase100, we use the uniform distribution over the last p = 300 principal components (i.e., principal components 300 to 600 ).

For both settings, we audit only the final model , assuming that we do not have access to the intermediate models. This corresponds to the blackbox auditing setting for data poisoning and a graybox setting for random gradient canaries.

F.3 Miscellaneous Details

Hardware. We run each job on an internal compute cluster using only CPUs (i.e., no hardware accelerators such as GPUs were used). Each job was run with 8 CPU cores and 16G memory.

F.4 Additional Experimental Results

F.4.1 Comparison to Bayesian Auditing of DP

We compare the proposed LiDP auditing approach with the Bayesian approach of Zanella-B´ eguelin et al. [69]. This approach constructs a posterior distribution over the true and false positive rates by updating the non-informative Jeffreys priors using empirical observations. They use Bayesian credible intervals (as opposed to confidence intervals in our case) computed using numerical quadrature.

We compare the proposed LiDP auditing with K = m> 1 with the Bayesian approach of [69] with K = m = 1 in Figure 14 using the code open-sourced by the authors. 4 Owing to the increased runtime in the numerical estimation of the ε of the Bayesian approach, we tune its threshold τ to separate the null and alternate hypotheses (cf. line 5 of Algorithm 1) for the Clopper-Pearson interval.

We observe that the LiDP auditing approach generally provides improved estimates when compared to this baseline. Note that the reason behind the improvement for the Bayesian approach of [69] is by capture the joint distribution of the true and false positive rates, while the improvement behind LiDP

2

0

2

1

2

2

2

3

2

4

2

5

DP upper bound

ε

0

.

0

0

.

2

0

.

4

0

.

6

Empirical

ˆ

ε

FashionMNIST-Linear + Data Poisoning

2

0

2

1

2

2

2

3

2

4

2

5

DP upper bound

ε

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical

ˆ

ε

FashionMNIST-Linear + Random Gradient

2

0

2

1

2

2

2

3

2

4

2

5

DP upper bound

ε

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical

ˆ

ε

FashionMNIST-MLP + Data Poisoning

2

0

2

1

2

2

2

3

2

4

2

5

DP upper bound

ε

0

.

0

0

.

5

1

.

0

1

.

5

2

.

0

Empirical

ˆ

ε

FashionMNIST-MLP + Random Gradient

DP + 1st-Order Wilson

DP + Bayesian Credible Interval

LiDP + 2nd-Order Wilson

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

0

.

175

Empirical lower bound

ˆ

ε

FMNIST-Linear

ε

= 1

/ Data Canary

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

FMNIST-MLP

ε

= 4

/ Data Canary

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

Purchase-100-MLP

ε

= 2

/ Data Canary

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Empirical lower bound

ˆ

ε

Purchase-100-MLP /

ε

= 16

/ Grad. Canary

DP + Wilson

DP + Bernstein

LiDP + 2nd-Order Wilson

LiDP + 2nd-Order Bernstein

auditing is by a clever use of randomized tests. Both sources of improvements are orthogonal to each other. This suggests an interesting future research direction: adapt Bayesian credible intervals for LiDP auditing to get the best of both worlds.

F.4.2 Comparison of Asymptotic vs. Non-asymptotic Confidence Intervals

Wecompare the asymptotic Wilson intervals (used primarily in the experiments) to the non-asymptotic Bernstein intervals (used primarily for the theory) in Figure 15. The results show that the Wilson intervals give larger empirical lower bounds in general with all other parameters remaining the same. This is as expected because the Wilson intervals are an asymptotic tightening of the Bernstein intervals as shown in Figure 2.

F.4.3 Experiments Across Different Parameter Settings

We give the following plots to augment Table 1 and Figure 6:

Figure 16: plot of the test accuracy upon adding K canaries to the training process.

Figure 18: experimental results for FMNIST MLP model.

Figure 17: experimental results for FMNIST linear model.

Figure 19: experimental results for Purchase MLP model.

2 1

2 3

2 5

Number of canaries

K

0.828

0.830

0.832

0.834

0.836

0.838

Test Accuracy

FMNIST-Linear + Data Poisoning

2 1

2 3

2 5

Number of canaries

K

0.830

0.835

0.840

0.845

0.850

Test Accuracy

FMNIST-MLP + Data Poisoning

2 1

2 3

2 5

Number of canaries

K

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Test Accuracy

Purchase-100-MLP + Data Poisoning

2 1

2 3

2 5

2 7

2 9

Number of canaries

K

0.828

0.830

0.832

0.834

0.836

0.838

Test Accuracy

FMNIST-Linear + Random Gradient

2 1

2 3

2 5

2 7

2 9

Number of canaries

K

0.830

0.835

0.840

0.845

0.850

Test Accuracy

FMNIST-MLP + Random Gradient

2 1

2 3

2 5

2 7

Number of canaries

K

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0.55

Test Accuracy

Purchase-100-MLP + Random Gradient

=1.0

=2.0

=4.0

=8.0

=16.0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

0

.

12

0

.

14

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

Linear Model on FMNIST with Data Poisoning Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

7

2

8

2

9

2

10

Number of trials

n

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

0

.

175

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

Linear Model on FMNIST with Random Gradient Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

Empirical lower bound

ˆ

ε

DP

ε

= 32

Linear Model on FMNIST with Data Poisoning Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

0

.

35

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

1

2

3

2

5

2

7

Number of canaries

K

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 32

Linear Model on FMNIST with Random Gradient Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

0

.

30

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

MLP Model on FMNIST with Data Poisoning Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

01

0

.

02

0

.

03

0

.

04

0

.

05

0

.

06

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

0

.

175

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

MLP Model on FMNIST with Random Gradient Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

000

0

.

025

0

.

050

0

.

075

0

.

100

0

.

125

0

.

150

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

0

0

.

1

0

.

2

0

.

3

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

0

.

60

Empirical lower bound

ˆ

ε

DP

ε

= 32

MLP Model on FMNIST with Data Poisoning Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

05

0

.

10

0

.

15

0

.

20

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

1

2

3

2

5

2

7

2

9

Number of canaries

K

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

0

.

60

Empirical lower bound

ˆ

ε

DP

ε

= 32

MLP Model on FMNIST with Random Gradient Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

02

0

.

04

0

.

06

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

6

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

MLP Model on Purchase-100 with Data Poisoning Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

02

0

.

04

0

.

06

0

.

08

0

.

10

0

.

12

Empirical lower bound

ˆ

ε

DP

ε

= 1

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Empirical lower bound

ˆ

ε

DP

ε

= 4

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Empirical lower bound

ˆ

ε

DP

ε

= 8

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Empirical lower bound

ˆ

ε

DP

ε

= 16

.

0

2

7

2

8

2

9

2

10

Number of trials

n

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

.

25

1

.

50

1

.

75

Empirical lower bound

ˆ

ε

DP

ε

= 32

.

0

MLP Model on Purchase-100 with Random Gradient Canary

DP + Wilson

LiDP + 2nd-Order Wilson

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

02

0

.

04

0

.

06

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

0

.

20

0

.

25

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

30

0

.

35

0

.

40

0

.

45

0

.

50

0

.

55

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

7

0

.

8

0

.

9

1

.

0

1

.

1

1

.

2

Empirical lower bound

ˆ

ε

DP

ε

= 32

MLP Model on Purchase-100 with Data Poisoning Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

00

0

.

05

0

.

10

0

.

15

Empirical lower bound

ˆ

ε

DP

ε

= 1

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

20

0

.

25

0

.

30

0

.

35

0

.

40

0

.

45

Empirical lower bound

ˆ

ε

DP

ε

= 2

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

Empirical lower bound

ˆ

ε

DP

ε

= 4

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Empirical lower bound

ˆ

ε

DP

ε

= 8

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

0

.

4

0

.

6

0

.

8

1

.

0

1

.

2

Empirical lower bound

ˆ

ε

DP

ε

= 16

2

0

2

1

2

2

2

3

2

4

2

5

2

6

Number of canaries

K

1

.

4

1

.

5

1

.

6

1

.

7

1

.

8

1

.

9

Empirical lower bound

ˆ

ε

DP

ε

= 32

MLP Model on Purchase-100 with Random Gradient Canary

K

=

√

n

DP + Wilson

LiDP + 1st-Order Wilson

LiDP + 2nd-Order Wilson

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe use of canaries is a very intuitive strategy to evaluate the DP mechanism (which can subsequently attract a broader non-DP specialist audience to this area). The paper is structured well, has comprehensive details on both the method and the corresponding experimental results and shows clear motivation and conclusions. This work presents methodology for auditing of DP trained models through the introduction of Lifted DP and canary-based auditing techniques.\n\nSTRENGTHS:\nThe use of canaries is a very intuitive strategy to evaluate the DP mechanism (which can subsequently attract a broader non-DP specialist audience to this area). The paper is structured well, has comprehensive details on both the method and the corresponding experimental results and shows clear motivation and conclusions.\n\nWEAKNESSES:\nWhile the method show promise, the evaluation results are not very representative: FMNIST is a rather small dataset and so is Purchase-100. Given that the models associated with these are also rather small too, I do not see how well can this method scale to larger architectures and more complex datasets (especially as the canary-generation process can heavily depend on it).\n\nIt is not very clear to me what the advantage is over the work of https://arxiv.org/abs/2305.08846. Here model auditing can be performed in a single training run, rendering other methods of auditing significantly less efficient. I would like the authors to comment on why their work is a substantial contribution given the existence of this method.\n\nQUESTIONS:\nWhat is the dependency with epsilon level? There does not seem to be a clear trend of how well the method performs with more noise being added. I would like the authors to discuss this further and hypothesise what the trend is and why.\n\nCould the authors elaborate on what they mean by ‘poisoning’ canaries? What is the methodology like on non-image modalities?\n\nLIMITATIONS:\nI described most of the limitations in the weaknesses section, but to summarise: A) it is not clear how the method performs in non-trivial settings (e.g. ImageNet), B) the improvements compared to the work I linked above are not clear. \nOverall, this work shows promise, but I would like the authors to clarify the points I raised above before I can recommend acceptance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a principled improvement over existing privacy auditing techniques. Privacy auditing through the binary hypothesis testing formulation implies gathering samples on the success of the adversary when trying to correctly guess the membership game against a mechanism $M$. All auditing techniques involve switching to a high-probability argument via confidence intervals to be able to provide lower bounds. These CIs converge with a rate of $O(\\frac{1}{\\sqrt{N}})$, where N is the number of auditing samples. The goal of this paper is to improve this convergence rate by testing multiple (possibly correlated) samples at a time, providing an improvement for the convergence of up to $O(\\frac{1}{\\sqrt{NK}})$, where K is the number of inserted canaries. Their proposed technique is as follows: 1. Run the mechanism $M$ on $D \\cup ( c_1, .., c_K ) $ and $D \\cup (c_1, .., c_{K-1}) $, where the canaries are iid samples drawn from a fixed canary distribution. Through a clever randomisation argument, using the property that the canaries are iid from the same distribution, the authors propose testing multiple iid canaries by testing other iid canaries by exchanging the missing $c_K$ with other iid canaries. By doing this, the authors can test $K$ test statistics for the alternative hypothesis and $M$ test statistics for the null hypothesis. Now, the goal is to integrate this new information into the confidence interval. The authors do not make an independence assumption as in previous work and derive a tailored confidence interval that accounts for the possible correlation between the K tests. Their analysis shows how their confidence interval converges with a rate of $O(\\frac{1}{\\sqrt{N\n}})$ when the statistics are correlated to a rate of $O(\\frac{1}{\\sqrt{NK}})$ when the statistics are independent.\n\nSTRENGTHS:\n* The paper has a nice structure, it solves a clearly stated problem known in the privacy auditing community, for which they introduce all the needed formalism and they make the necessary experiments to back up their results.\n* The authors propose an algorithm that could be used as a drop-in replacement for other auditing techniques.\n* Formally accounting for the correlation between hypotheses fills an existing gap in auditing literature when multiple samples are used for auditing purposes. All the literature I am aware of makes an independence assumption, which might be strong in some cases. This work recovers that case but accounts for others as well, framing and formalising in a principled way the multisample testing problem.\n\nWEAKNESSES:\n* in the experimental section, the authors show a bias/variance tradeoff when auditing a mechanism M that computes a noisy sum/mean. The same argument does not hold against differentially private machine learning models, and it would be interesting to show if the proposed technique can achieve tightness when auditing a (possibly simple) machine learning pipeline.\n\nQUESTIONS:\n* The proposed auditing technique is not tailored for the Gaussian mechanism but for any mechanism $M$. For example, one could audit a Laplace Mechanism or a Subsampled Gaussian Mechanism with it. I think the authors could stress this out for the reader, as improving auditing sample complexity in the general case is a truly desirable property, and this work does that. Would the authors want to add some details or employ their technique to audit more diverse mechanisms?\n* The bias/variance argument in section 4 holds against a mechanism that averages over a list of gradients, as it is easy to observe how inserting noisy samples will yield privacy amplification in that scenario. Yet, when we audit a machine learning model with input poisoning (so, actual samples need to be inserted, not the gradients), the relationship is not so clear anymore. How many sample canaries could we use for auditing purposes against a machine-learning model? Is there any scenario in which we can hope to audit correctly by training only two models, for example? (n=1, k=?) What is the performance in that scenario?\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the sample complexity (in terms of sampled models) of auditing differential privacy. It proposes a new approach to usage of canaries in auditing via hypothesis test, breaking an existing sample complexity barrier by making use of multiple, randomized canaries. These randomized canaries are used to audit a new, but equivalent formulation of approximate differential privacy called lifted DP, which states a privacy condition for neighboring datasets and rejection sets sampled from a distribution. Sampled models are reused to compute test statistics for each of the multiple canaries included in the dataset, and confidence intervals for the average test statistic are computed using techniques introduced in the paper, tailored to the distribution of test statistics produced by the new auditing method. Empirical evaluation of the proposed method is also given.\n\nSTRENGTHS:\nThe paper makes progress in improving the sample complexity of DP auditing by introducing new techniques that should be compatible with future progress in strengthening canaries. The usage of multiple, randomized canaries is innovative and novel tools were used to adapt their usage to existing hypothesis testing frameworks for auditing. The empirical results in Section 4 and 6 were quite helpful in interpreting the expected sample complexity improvements from various choices of K and order of confidence interval.\n\nWEAKNESSES:\nThe sample complexity improvements seem notable, but I'm curious about how large of a bottleneck sample complexity is to practical auditing efforts. I'd expect it to be a significant obstacle, but additional discussion of the extent to which sample complexity impedes privacy auditing could help further motivate this work.\n \nNotes:\npg 5 - \"in practice, it depends on how correlated the K test are\"\nIn the caption of figure 3 - \"provides significant gain in the require number of trials to achieve a desired level of lower bound\"\npg 9 - \"For deployment in production, it worth further studying approaches with minimal computational overhead\"\n\nQUESTIONS:\nSee comments in weaknesses.\n\nLIMITATIONS:\nYes, the authors adequately addressed potential societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a methodology for calculating lower bounds for the parameter $\\\\varepsilon$ of an $(\\\\varepsilon,\\\\delta)$-differentially private (DP) mechanism. When applied to machine learning algorithms, the methodology can be used to audit the privacy guarantees of popular training algorithms such as DP-SGD.\n\nThe paper builds on existing methods that audit DP guarantees by testing the presence of a canary in the input of the mechanism. Given an $(\\\\varepsilon,\\\\delta)$-DP mechanism $\\\\mathcal{A}$, two neighboring datasets $D_0$ and $D_1 = D_0 \\\\cup \\\\{c\\\\}$ and a measurable set $R$, a lower bound for $\\\\varepsilon$ is given by\n\n$$\n   \\\\varepsilon \\\\geq\n   \\\\log (\\\\mathbb{P}(\\\\mathcal{A}(D_1) \\\\in R) - \\\\delta) -\n   \\\\log \\\\mathbb{P}(\\\\mathcal{A}(D_0) \\\\in R) .\n$$\n\nA lower bound for $\\\\varepsilon$ can be derived from a lower bound $\\\\underline{\\\\boldsymbol{p}}_1 \\\\leq \\\\mathbb{P}(\\\\mathcal{A}(D_1) \\\\in R)$ and an upper bound $\\\\overline{\\\\boldsymbol{p}}_0 \\\\geq \\\\mathbb{P}(\\\\mathcal{A}(D_0) \\\\in R)$ estimated from samples. This lower bound can be made tighter by choosing a canary $c$ whose presence in the training dataset can be easily tested with a rejection set $R$. However, using Bernoulli confidence intervals computed from $n$ samples, the bounds $\\\\underline{\\\\boldsymbol{p}}_1, \\\\overline{\\\\boldsymbol{p}}_0$ are loose by a factor of $1/\\\\sqrt{n}$. Since each sample requires evaluating $\\\\mathcal{A}$ on a different dataset $D_0$, decreasing this factor is expensive.\n\nThe key idea of the paper is to instead audit a probabilistic lifting of DP (LiDP) using counterexamples sampled from a joint distribution $( \\\\boldsymbol{D_0},  \\\\boldsymbol{D_1}, \\\\boldsymbol{R})$. While entirely equivalent to DP, thanks to the exchangeability of random i.i.d canaries  LiDP allows reusing samples $\\\\mathcal{A}(D \\\\cup \\\\{c_1,\\\\ldots,c_K\\\\})$ and $\\\\mathcal{A}(D \\\\cup \\\\{c_1,\\\\ldots,c_{K-1}\\\\})$ to gather multiple correlated test statistics $\\\\boldsymbol{x}_k$, one for each canary $c_k$. A lower bound for $\\\\varepsilon$ can be then calculated similarly as for DP from bounds of the mean of these statistics. The paper analyzes the correlation between the statistics to derive higher-order exact (Bernstein) and asymptotic (Wilson) confidence intervals. Importantly, the authors prove that $\\\\ell$-th order bounds are loose by a factor of $n^{(1-2\\\\ell)/(2\\\\ell)}$ when choosing $K = \\\\lceil n^{(\\\\ell-1)/\\\\ell} \\\\rceil$ and so e.g. second-order bounds reduce the number of samples required to attain a given confidence because their looseness decreases as $1/n^{3/2}$ rather than $1/n^{1/2}$ for first-order bounds (and previous methods auditing DP).\n\nThe authors evaluate the methodology on a Gaussian mechanism showing a 4x gain in sample efficiency when using second-order confidence intervals with $n=1000$ compared to a baseline using Bernoulli confidence intervals. They also evaluate the method on linear and 2-layer MLP classifiers trained with DP-SGD on FMNIST and Purchase-100 using either random or clipping-aware poisoned canaries [31]. This evaluation shows an average improvement in sample efficiency of up to 3x also with $n=1000$.\n\nThe supplemental material includes proofs of the results in the paper, algorithmic descriptions of the methodology, a derivation of asymptotic Wilson intervals, and comprehensive ablation studies varying the privacy budget $\\\\varepsilon$, the number of canaries $K$, samples $n$, and dimension (for the Gaussian mechanism).\n\nSTRENGTHS:\n1. A novel method for auditing differential privacy that can be combined with existing canary design strategies and improve sample efficiency.\n\n2. First formal analysis of previously used heuristics reusing trained models to obtain multiple samples.\n\n3. Technically solid theoretical foundations. Detailed proofs. Derivation of exact and asymptotic confidence intervals.\n\n4. Great high-level intuition for why the method improves sample efficiency and the reasons for the bias/variance trade-off in selecting the number of canaries to use.\n\n5. Extensive algorithmic descriptions of the method in the supplemental material that make the paper fairly self-contained and enable reproducibility. Authors promise to open-source their code to replicate results.\n\nWEAKNESSES:\n1. The evaluation on ML scenarios uses very simple models: a linear model and MLPs with 2 hidden layers with 256 units (269k and 245k parameters for FMNIST and Purchase-100, respectively).\n\n2. Evaluation baselines are limited to approaches using Bernoulli confidence intervals and not to more recent approaches using credible regions [66] which also claim improved sample efficiency.\n\nQUESTIONS:\n1. Using the form of Bernstein's bound I am familiar with (Dudley, Richard M. Uniform central limit theorems. Vol. 142. CUP, 2014) would give a slightly tighter bound than in Equation (6). The bound in the paper seems to use $\\\\sqrt{a+b} \\\\leq \\\\sqrt{a} + \\\\sqrt{b}$ to simplify it, making it looser. Can you clarify whether you use this simplification, or else provide a detailed derivation?\n\n2. In your evaluation, do you set $m = K$ in Algorithm 1?\n\n3. There seems to be a qualitative difference between $\\\\underline{\\\\boldsymbol{p}}_1$ and $\\\\overline{\\\\boldsymbol{p}}_0$ in Algorithm 1 that is not discussed in the paper. The former is calculated from statistics $\\\\boldsymbol{x}^{(i)}$ using canaries $c_1,\\\\ldots,c_K$ inserted into the training set while the latter is calculated from statistics $\\\\boldsymbol{y}^{(i)}$ depending on canaries $c'_1,\\\\ldots,c'_m$ not in the training set that are independent of the output of the mechanism (the statistics themselves are correlated because they all use the same model). The bias-variance tradeoffs that you discuss in §4 seem to apply only to $c_k$. Are there tradeoffs that make increasing the number of out-canaries $m$ not always beneficial?\n\n### Details\n\n- The paper presents exact Bernstein confidence intervals but then uses asymptotic Wilson intervals exclusively in the evaluation. The two are only compared in Figure 2. I would have liked to see an additional comparison in at least some selected empirical results for bounds on $\\varepsilon$.\n\n- In Eq.1 $D,D'$ should be $D_0,D_1$. This has been fixed in the PDF in the supplemental material.\n\n- l.119: \"the canary has the freedom\" => \"the **adversary** has the freedom\"?\n\n- l.197: \"applying probabilistic method\" => \"applying **a heuristic** method\"?\n\n- Figure 1 is not referenced. I think it is supposed to support the analysis in the paragraph starting at line 271.\n\n- In Figure 3 caption: \"require number\" => \"required number\"\n\n- l.243: \"find §4 that\" => \"find **in** §4 that\"\n\n- l.346: \"it worth\" => \"it **is** worth\"\n\n- l.710: $k$ => $K$\n\n- In Algorithm 3 in steps 4 and 5, and in Algorithm 4 in steps 5 and 6, I believe that there is a missing $\\frac{K-1}{K}$ multiplying $\\overline{\\boldsymbol{\\mu}}_2$ as in Equation (17).\n\nLIMITATIONS:\nThe authors discuss some limitations of the auditing methodology throughout the paper and discuss the need to balance the tightness/computation trade-off in practice.\n\nIt could be a good idea to complement this with a discussion of how the lower bounds for $\\\\varepsilon$ depend on the canary design and detection strategy, how to interpret the gap between lower bounds and the theoretical guarantee of a mechanism, and the impact of deciding whether a mechanism provides enough privacy based on a lower bound alone.\n\nI also believe that given the size and variety of the ML models used in the evaluation, the paper should discuss how the results may extrapolate to other architectures and larger number of parameters. Figure 4 (right) shows the method may benefit as the dimension increases; it would be great if this effect could be backed with experimental results on ML models.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-19"}