SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Krishna Pillutla<sup>1</sup>  
H. Brendan McMahan<sup>1</sup>  
<sup>1</sup>Google Research 2Northeast
Galen Andrew<sup>1</sup>, Alina Oprea<sup>1,2</sup>, Bern University
Peter Kairouz $^{1}$   
Sewoong Oh $^{1,3}$ $^{3}$ University of Washington
We present a rigorous methodology for auditing differentially private machine learning algorithms by adding multiple carefully designed examples called canaries. We take a first principles approach based on three key components. First, we introduce Lifted Differential Privacy (LiDP) which expands the definition of differential privacy to handle randomized datasets. This gives us the freedom to design randomized canaries. Second, we audit LiDP by trying to distinguish between the model trained with  $K$  canaries versus  $K - 1$  canaries in the dataset, leaving one canary out. By drawing the canaries i.i.d., LiDP can leverage the symmetry in the design and reuse each privately trained model to run multiple statistical tests, one for each canary. Third, we introduce novel confidence intervals that take advantage of the multiple test statistics by adapting to the empirical higher-order correlations. Together, this new recipe demonstrates significant improvements in sample complexity, both theoretically and empirically, using synthetic and real data. Further, recent advances in designing stronger canaries can be readily incorporated into the new framework.
Differential privacy (DP), introduced in [22], has gained widespread adoption by governments, companies, and researchers by formally ensuring plausible deniability for participating individuals. This is achieved by guaranteeing that a curious observer of the output of a query cannot be confident in their answer to the following binary hypothesis test: did a particular individual participate in the dataset or not? For example, introducing sufficient randomness when training a model on a certain dataset ensures a desired level of differential privacy. This in turn ensures that an individual's sensitive information cannot be inferred from the trained model with high confidence. However, calibrating the right amount of noise can be a challenging process. It is easy to make mistakes in implementing a DP mechanism due to intricacies involving micro-batching, sensitivity analysis, and privacy accounting. Even with correct implementation, there are several incidents of published DP algorithms with miscalculated privacy guarantees that falsely report higher levels of privacy [16, 34, 40, 47, 57, 58]. Data-driven approaches to auditing a mechanism for a violation of a claimed privacy guarantee can significantly mitigate the danger of unintentionally leaking sensitive data.
Popular approaches for auditing privacy share three common components [e.g. 31-33, 45, 69]. Conceptually, these approaches are founded on the definition of DP and involve producing counterexamples that potentially violate the DP condition. Algorithmically, this leads to the standard recipe of injecting a single carefully designed example, referred to as a canary, and running a statistical hypothesis test for its presence from the outcome of the mechanism. Analytically, a high-confidence bound on the DP condition is derived by calculating the confidence intervals of the corresponding Bernoulli random variables from  $n$  independent trials of the mechanism.
Recent advances adopt this standard approach and focus on designing stronger canaries to reduce the number of trials required to successfully audit DP [e.g., 31-33, 39, 45, 69]. However, each independent trial can be as costly as training a model from scratch; refuting a false claim of  $(\varepsilon, \delta)$ -DP with a minimal number of samples is of utmost importance. In practice, standard auditing can require training on the range of thousands to hundreds of thousands of models [e.g., 60]. Unfortunately, under the standard recipe, we are fundamentally limited by the  $1/\sqrt{n}$  sample dependence of the Bernoulli confidence intervals.
Contributions. We break this  $1 / \sqrt{n}$  barrier by rethinking auditing from first principles.
1. Lifted DP: We propose to audit an equivalent definition of DP, which we call Lifted DP in §3.1. This gives an auditor the freedom to design a counter-example consisting of random datasets and rejection sets. This enables adding random canaries, which is critical in the next step. Theorem 3 shows that violation of Lifted DP implies violation of DP, justifying our framework.
2. Auditing Lifted DP with Multiple Random Canaries: We propose adding  $K > 1$  canaries under the alternative hypothesis and comparing it against a dataset with  $K - 1$  canaries, leaving one canary out. If the canaries are deterministic, we need  $K$  separate null hypotheses; each hypothesis leaves one canary out. Our new recipe overcomes this inefficiency in §3.2 by drawing random canaries independently from the same distribution. This ensures the exchangeability of the test statistics, allowing us to reuse each privately trained model to run multiple hypothesis tests in a principled manner. This is critical in making the confidence interval sample efficient.
3. Adaptive Confidence Intervals: Due to the symmetry of our design, the test statistics follow a special distribution that we call  $eXchangeable$  Bernoulli (XBern). Auditing privacy boils down to computing confidence intervals on the average test statistic over the  $K$  canaries included in the dataset. If the test statistics are independent, the resulting confidence interval scales as  $1 / \sqrt{nK}$ . However, in practice, the dependence is non-zero and unknown. We propose a new and principled family of confidence intervals in §3.3 that adapts to the empirical higher-order correlations between the test statistics. This gives significantly smaller confidence intervals when the actual dependence is small, both theoretically (Proposition 4) and empirically.
4. Numerical Results: We audit an unknown Gaussian mechanism with black-box access and demonstrate (up to)  $16 \times$  improvement in the sample complexity. We also show how to seamlessly lift recently proposed canary designs in our recipe to improve the sample complexity on real data.
We describe the standard recipe to audit DP. Formally, we adopt the so-called add/remove definition of differential privacy; our constructions also seamlessly extend to other choices of neighborhoods as we explain in §B.2.
Definition 1 (Differential privacy). A pair of datasets,  $(D_0, D_1)$ , is said to be neighboring if their sizes differ by one and the datasets differ in one entry:  $|D_1 \setminus D_0| + |D_0 \setminus D_1| = 1$ . A randomized mechanism  $\mathcal{A}: \mathcal{Z}^* \to \mathcal{R}$  is said to be  $(\varepsilon, \delta)$ -Differentially Private (DP) for some  $\varepsilon \geq 0$  and  $\delta \in [0,1]$  if it satisfies  $\mathbb{P}_{\mathcal{A}}(\mathcal{A}(D_1) \in R) \leq e^\varepsilon \mathbb{P}_{\mathcal{A}}(\mathcal{A}(D_0) \in R) + \delta$ , which is equivalent to
$$
\varepsilon \geq \log (\mathbb {P} (\mathcal {A} (D _ {1}) \in R) - \delta) - \log \mathbb {P} (\mathcal {A} (D _ {0}) \in R), \tag {1}
$$
for all pairs of neighboring datasets,  $(D_0,D_1)$ , and all measurable sets,  $R\subset \mathcal{R}$ , of the output space  $\mathcal{R}$ , where  $\mathbb{P}_{\mathcal{A}}$  is over the randomness internal to the mechanism  $\mathcal{A}$ . Here,  $\mathcal{Z}^*$  is a space of datasets.
For small  $\varepsilon$  and  $\delta$ , one cannot infer from the output whether a particular individual is in the dataset or not with a high success probability. For a formal connection, we refer to [35]. This naturally leads to a standard procedure for auditing a mechanism  $\mathcal{A}$  claiming  $(\varepsilon, \delta)$ -DP: present  $(D_0, D_1, R) \in \mathcal{Z}^* \times \mathcal{Z}^* \times \mathcal{R}$  that violates Eq. (1) as a piece of evidence. Such a counter-example confirms that an adversary attempting to test the participation of an individual will succeed with sufficient probability, thus removing the potential for plausible deniability for the participants.
Standard Recipe: Adding a Single Canary. When auditing DP model training (using e.g. DP-SGD [1, 54]), the following recipe is now standard for designing a counter-example  $(D_0, D_1, R)$  [31-33, 45, 69]. A training dataset  $D_0$  is assumed to be given. This ensures that the model under scrutiny matches the use-case and is called a null hypothesis. Next, under a corresponding alternative hypothesis, a neighboring dataset  $D_1 = D_0 \cup \{c\}$  is constructed by adding a single carefully-designed
example  $c \in \mathcal{Z}$ , known as a canary. Finally, Eq. (1) is evaluated with a choice of  $R$  called a rejection set. For example, one can reject the null hypothesis (and claim the presence of the canary) if the loss on the canary is smaller than a fixed threshold;  $R$  is a set of models satisfying this rejection rule.
Bernoulli Confidence Intervals. Once a counter-example  $(D_0, D_1, R)$  is selected, we are left to evaluate the DP condition in Eq. (1). Since the two probabilities in the condition cannot be directly evaluated, we rely on the samples of the output from the mechanism, e.g., models trained with DP-SGD. This is equivalent to estimating the expectation,  $\mathbb{P}(\mathcal{A}(D) \in R)$  for  $D \in \{D_0, D_1\}$ , of a Bernoulli random variable,  $\mathbb{I}(\mathcal{A}(D) \in R)$ , from  $n$  i.i.d. samples. Providing high confidence intervals for Bernoulli distributions is a well-studied problem with several off-the-shelf techniques, such as Clopper-Pearson, Jeffreys, Bernstein, and Wilson intervals. Concretely, let  $\hat{\mathbb{P}}_n(\mathcal{A}(D) \in R)$  denote the empirical probability of the model falling in the rejection set in  $n$  independent runs. The standard intervals scale as  $|\mathbb{P}(\mathcal{A}(D_0) \in R) - \hat{\mathbb{P}}_n(\mathcal{A}(D_0) \in R)| \leq C_0 n^{-1/2}$  and  $|\mathbb{P}(\mathcal{A}(D_1) \in R) - \hat{\mathbb{P}}_n(\mathcal{A}(D_1) \in R)| \leq C_1 n^{-1/2}$  for constants  $C_0$  and  $C_1$  independent of  $n$ . If  $\mathcal{A}$  satisfies a claimed  $(\varepsilon, \delta)$ -DP in Eq. (1), then the following finite-sample lower bound holds with high confidence:
$$
\varepsilon \geq \hat {\varepsilon} _ {n} = \log \left(\hat {\mathbb {P}} _ {n} (\mathcal {A} (D _ {1}) \in R) - \frac {C _ {1}}{\sqrt {n}} - \delta\right) - \log \left(\hat {\mathbb {P}} _ {n} (\mathcal {A} (D _ {0}) \in R) + \frac {C _ {0}}{\sqrt {n}}\right). \tag {2}
$$
Auditing  $(\varepsilon, \delta)$ -DP amounts to testing the violation of this condition. This is fundamentally limited by the  $n^{-1/2}$  dependence of the Bernoulli confidence intervals. Our goal is to break this barrier.
Notation. While the DP condition is symmetric in  $(D_0, D_1)$ , we use  $D_0, D_1$  to refer to specific hypotheses. For symmetry, we need to check both conditions: Eq. (1) and its counterpart with  $\bar{D}_0, D_1$  interchanged. We omit this second condition for notational convenience. We use the shorthand  $[k] := \{1, 2, \dots, k\}$ . We refer to random variables by boldfaced letters (e.g.  $D$  is a random dataset).
Related Work. We provide a detailed survey in Appendix A. A stronger canary (and its rejection set) can increase the RHS of (1). The resulting hypothesis test can tolerate larger confidence intervals, thus requiring fewer samples. This has been the focus of recent breakthroughs in privacy auditing in [32, 39, 41, 45, 60]. They build upon membership inference attacks [e.g. 12, 53, 68] to measure memorization. Our aim is not to innovate on this front. Instead, our framework can seamlessly adopt recently designed canaries and inherit their strengths as demonstrated in §5 and §6.
Random canaries have been used in prior work, but for making the canary out-of-distribution in a computationally efficient manner. No variance reduction is achieved by such random canaries. Adding multiple (deterministic) canaries has been explored in literature but for different purposes. [32, 39] include multiple copies of the same canary to make the canary easier to detect while paying for group privacy since the paired datasets differ in multiple entries (see §3.2 for a detailed discussion).
[42, 69] propose adding multiple distinct canaries to reuse each trained model for multiple hypothesis tests. However, each canary is no stronger than a single canary case, and the resulting auditing suffers from group privacy. When computing the lower bound on  $\varepsilon$ , however, group privacy is ignored and the test statistics are assumed to be independent without rigorous justification. [2] avoids group privacy in the federated scenario where the adversary has the freedom to return a canary gradient update of choice. The prescribed random gradient shows good empirical performance. The confidence interval is not rigorously derived. Our recipe for injecting multiple canaries without a group privacy cost with rigorous confidence intervals can be incorporated into these works to give provable lower bounds.
In an independent and concurrent work, Steinke et al. [56] also consider auditing with randomized canaries that are Poisson-sampled, i.e., each canary is included or excluded independently with equal probability. Their recipe involves computing an empirical lower bound by comparing the accuracy (rather than the full confusion matrix as in our case) from the possibly dependent guesses with the worst-case randomized response mechanism. This allows them to use multiple dependent observations from a single trial to give a high probability lower bound on  $\varepsilon$ . Their confidence intervals, unlike the ones we give here, are non-adaptive and worst-case.
3 A New Framework for Auditing DP Mechanisms with Multiple Canaries
We define Lifted DP, a new definition of privacy that is equivalent to DP (§3.1). This allows us to define a new recipe for auditing with multiple random canaries, as opposed to a single deterministic canary in the standard recipe, and reuse each trained model to run multiple correlated hypothesis tests
in a principled manner (§3.2). The resulting test statistics form a vector of dependent but exchangeable indicators (which we call an eXchangeable Bernoulli or XBern distribution), as opposed to a single Bernoulli distribution. We leverage this exchangeability to give confidence intervals for the XBern distribution that can potentially improve with the number of injected canaries (§3.3). The pseudocode of our approach is provided in Algorithm 1.
3.1 From DP to Lifted DP
To enlarge the design space of counter-examples, we introduce an equivalent definition of DP.
Definition 2 (Lifted differential privacy). Let  $\mathcal{P}$  denote a joint probability distribution over  $(D_0, D_1, R)$  where  $(D_0, D_1) \in \mathcal{Z}^* \times \mathcal{Z}^*$  is a pair of random datasets that are neighboring (as in the standard definition of neighborhood in Definition 1) with probability one and let  $R \subset \mathcal{R}$  denote a random rejection set. We say that a randomized mechanism  $\mathcal{A}: \mathcal{Z}^* \to \mathcal{R}$  satisfies  $(\varepsilon, \delta)$ -Lifted Differential Privacy (LiDP) for some  $\varepsilon \geq 0$  and  $\delta \in [0,1]$  if, for all  $\mathcal{P}$  independent of  $\mathcal{A}$ , we have
$$
\mathbb {P} _ {\mathcal {A}, \mathcal {P}} (\mathcal {A} (\boldsymbol {D} _ {1}) \in \boldsymbol {R}) \leq e ^ {\varepsilon} \mathbb {P} _ {\mathcal {A}, \mathcal {P}} (\mathcal {A} (\boldsymbol {D} _ {0}) \in \boldsymbol {R}) + \delta . \tag {3}
$$
In Appendix A.3, we discuss connections between Lifted DP and other existing extensions of DP, such as Bayesian DP and Pufferfish, that also consider randomized datasets. The following theorem shows that LiDP is equivalent to the standard DP, which justifies our framework of checking the above condition; if a mechanism  $\mathcal{A}$  violates the above condition then it violates  $(\varepsilon, \delta)$ -DP.
Theorem 3. A randomized algorithm  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -LiDP iff  $\mathcal{A}$  is  $(\varepsilon, \delta)$ -DP.
A proof is provided in Appendix B. In contrast to DP, LiDP involves probabilities over both the internal randomness of the algorithm  $\mathcal{A}$  and the distribution  $\mathcal{P}$  over  $(D_0, D_1, R)$ . This gives the auditor greater freedom to search over a lifted space of joint distributions over the paired datasets and a rejection set; hence the name Lifted DP. Auditing LiDP amounts to constructing a randomized (as emphasized by the boldface letters) counter-example  $(D_0, D_1, R)$  that violates (3) as evidence.
3.2 From a Single Deterministic Canary to Multiple Random Canaries
Our strategy is to turn the LiDP condition in Eq. (3) into another condition in Eq. (4) below; this allows the auditor to reuse samples, running multiple hypothesis tests on each sample. This derivation critically relies on our carefully designed recipe that incorporates three crucial features:  $(a)$  binary hypothesis tests between pairs of stochastically coupled datasets containing  $K$  canaries and  $K - 1$  canaries, respectively, for some fixed integer  $K$ ,  $(b)$  sampling those canaries i.i.d. from the same distribution, and  $(c)$  choice of rejection sets, where each rejection set only depends on a single left-out canary. We introduce the following recipe, also presented in Algorithm 1.
We fix a given training set  $D$  and a canary distribution  $P_{\mathrm{canary}}$  over  $\mathcal{Z}$ . This ensures that the model under scrutiny is close to the use case. Under the alternative hypothesis, we train a model on a randomized training dataset  $D_{1} = D \cup \{c_{1}, \ldots, c_{K}\}$ , augmented with  $K$  random canaries drawn i.i.d. from  $P_{\mathrm{canary}}$ . Conceptually, this is to be tested against  $K$  leave-one-out (LOO) null hypotheses. Under the  $k^{\mathrm{th}}$  null hypothesis for each  $k \in [K]$ , we construct a coupled dataset,  $\pmb{D}_{0,k} = D \cup \{c_{1}, \ldots, c_{k-1}, c_{k+1}, \ldots, c_{K}\}$ , with  $K - 1$  canaries, leaving the  $k^{\mathrm{th}}$  canary out. This coupling of  $K - 1$  canaries ensures that  $(\pmb{D}_{0,k}, \pmb{D}_{1})$  is neighboring with probability one. For each left-out canary, the auditor runs a binary hypothesis test with a choice of a random rejection set  $\pmb{R}_{k}$ . We restrict  $\pmb{R}_{k}$  to depend only on the canary  $c_{k}$  that is being tested and not the index  $k$ . For example,  $\pmb{R}_{k}$  can be the set of models achieving a loss on the canary  $c_{k}$  below a predefined threshold  $\tau$ .
The goal of this LOO construction is to reuse each trained private model to run multiple tests such that the averaged test statistic has a smaller variance for a given number of models. Under the standard definition of DP, one can still use the above LOO construction but with fixed and deterministic canaries. This gives no variance gain because evaluating  $\mathbb{P}(\mathcal{A}(D_{0,k})\in R_k)$  in Eq. (1) or its averaged counterpart  $(1 / K)\sum_{k = 1}^{K}\mathbb{P}(\mathcal{A}(D_{0,k})\in R_k)$  requires training one model to get one sample from the test statistic  $\mathbb{I}(\mathcal{A}(D_{0,k})\in R_k)$ . The key ingredient in reusing trained models is randomization.
We build upon the LiDP condition in Eq. (3) by noting that the test statistics are exchangeable for i.i.d. canaries. Specifically, we have for any  $k \in [K]$  that  $\mathbb{P}(\mathcal{A}(\pmb{D}_{0,k}) \in \pmb{R}_k) = \mathbb{P}(\mathcal{A}(\pmb{D}_{0,K}) \in \pmb{R}_K) = \mathbb{P}(\mathcal{A}(\pmb{D}_{0,K}) \in \pmb{R}_j')$  for any canary  $c_j'$  drawn i.i.d. from  $P_{\text{canary}}$  and its corresponding
Algorithm 1 Auditing Lifted DP
Input: Sample size  $n$ , number of canaries  $K$ , number of null tests  $m$ , DP mechanism  $\mathcal{A}$ , training set  $D$ , canary generating distribution  $P_{\mathrm{canary}}$ , threshold  $\tau$ , failure probability  $\beta$ , privacy  $\delta \in [0,1]$ .
1: for  $i = 1, \dots, n$  do
2: Randomly generate  $K + m$  canaries  $\{c_1, \ldots, c_K, c_1', \ldots, c_m'\}$  i.i.d. from  $P_{\text{canary}}$ .
3:  $\pmb{D}_0\gets D\cup \{\pmb {c}_1,\dots ,\pmb{c}_{K - 1}\} ,\pmb{D}_1\gets D\cup \{\pmb {c}_1,\dots ,\pmb {c}_K\}$
4: Train two models  $\pmb{\theta}_0\gets \mathcal{A}(\pmb {D}_0)$  and  $\pmb {\theta}_1\gets \mathcal{A}(\pmb {D}_1)$
5: Record test statistics  $\pmb{x}^{(i)}\gets \left(\mathbb{I}(f_{\pmb{\theta}_1}(\pmb {c}_k) <   \tau)\right)_{k = 1}^K$  and  $\pmb{y}^{(i)}\gets \left(\mathbb{I}(f_{\pmb{\theta}_0}(\pmb {c}_j') <   \tau)\right)_{j = 1}^m$
6: Set  $\underline{p}_1 \gets \mathrm{XBernLower}\big(\{\boldsymbol{x}^{(i)}\}_{i \in [n]}, \beta / 2\big)$  and  $\overline{p}_0 \gets \mathrm{XBernUpper}\big(\{\boldsymbol{y}^{(i)}\}_{i \in [n]}, \beta / 2\big)$
7: Return  $\hat{\varepsilon}_n \gets \log \left((\underline{p}_{-1} - \delta) / \overline{p}_0\right)$  and a guarantee that  $\mathbb{P}(\varepsilon < \hat{\varepsilon}_n) \leq \beta$ .
rejection set  $\pmb{R}_j'$  that are statistically independent of  $D_{0,K}$ . Therefore, we can rewrite the right side of Eq. (3) using  $m$  i.i.d. test canaries  $c_1', \ldots, c_m' \sim P_{\text{canonical}}$  and a single trained model  $\mathcal{A}(D_{0,K})$  as
$$
\frac {1}{K} \sum_ {k = 1} ^ {K} \mathbb {P} _ {\mathcal {A}, \mathcal {P}} (\mathcal {A} (\boldsymbol {D} _ {1}) \in \boldsymbol {R} _ {k}) \leq \frac {e ^ {\varepsilon}}{m} \sum_ {j = 1} ^ {m} \mathbb {P} _ {\mathcal {A}, \mathcal {P}} (\mathcal {A} (\boldsymbol {D} _ {0, K}) \in \boldsymbol {R} _ {j} ^ {\prime}) + \delta . \tag {4}
$$
Checking this condition is sufficient for auditing LiDP and, via Theorem 3, for auditing DP. For each model trained on  $D_{1}$ , we record the test statistics of  $K$  (correlated) binary hypothesis tests. This is denoted by a random vector  $\pmb{x} = (\mathbb{I}(\mathcal{A}(D_1)\in R_k))_{k = 1}^K\in \{0,1\} ^K$ , where  $R_{k}$  is a rejection set that checks for the presence of the  $k^{\mathrm{th}}$  canary. Similar to the standard recipe, we train  $n$  models to obtain  $n$  i.i.d. samples  $x^{(1)},\ldots ,x^{(n)}\in \{0,1\} ^K$  to estimate the left side of (4) using the empirical mean:
$$
\hat {\boldsymbol {\mu}} _ {1} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{K} \sum_ {k = 1} ^ {K} \boldsymbol {x} _ {k} ^ {(i)} \in [ 0, 1 ], \tag {5}
$$
where the subscript one in  $\hat{\mu}_1$  denotes that this is the empirical first moment. Ideally, if the  $K$  tests are independent, the corresponding confidence interval is smaller by a factor of  $\sqrt{K}$ . In practice, the  $K$  tests are correlated and the size of the confidence interval depends on their correlation. We derive principled confidence intervals that leverage the empirically measured correlations in §3.3. We can define  $\pmb{y} \in \{0, 1\}^m$  and its mean  $\hat{\nu}_1$  analogously for the null hypothesis. We provide pseudocode in Algorithm 1 as an example guideline for applying our recipe to auditing DP training, where  $f_{\theta}(z)$  is the loss evaluated on an example  $z$  for a model  $\theta$ . XBernLower() and XBernUpper() respectively return the lower and upper adaptive confidence intervals from §3.3. We instantiate Algorithm 1 with concrete examples of canary design in §5.