SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our Recipe vs. Multiple Deterministic Canaries. An alternative with deterministic canaries would be to test between  $D_0$  with no canaries and  $D_1$  with  $K$  canaries such that we can get  $K$  samples from a single trained model on  $D_0$ , one for each of the  $K$  test statistics  $\{\mathbb{I}(\mathcal{A}(D_0) \in R_k)\}_{k=1}^K$ . However, due to the fact that  $D_1$  and  $D_0$  are now at Hamming distance  $K$ , this suffers from group privacy; we are required to audit for a much larger privacy leakage of  $(K\varepsilon, ((e^{K\varepsilon} - 1) / (e^{\varepsilon} - 1))\delta)$ -DP. Under the (deterministic) LOO construction, this translates into  $(1 / K)\sum_{k=1}^{K}\mathbb{P}(\mathcal{A}(D_1) \in R_k) \leq e^{K\varepsilon}(1 / K)\sum_{k=1}^{K}\mathbb{P}(\mathcal{A}(D_0) \in R_k) + ((e^{K\varepsilon} - 1) / (e^{\varepsilon} - 1))\delta$ , where if one canary violates the group privacy condition then the average also violates it. We can reuse a single trained model to get  $K$  test statistics in the above condition, but each canary is distinct and not any stronger than the one from  $(\varepsilon, \delta)$ -DP auditing. One cannot obtain stronger counterexamples without sacrificing the sample gain. For example, we can repeat the same canary  $K$  times as proposed in [32]. This makes it easier to detect the canary, making a stronger counter-example, but there is no sample gain as we only get one test statistic per trained model. With deterministic canaries, there is no way to avoid this group privacy cost while our recipe does not incur it.
3.3 From Bernoulli Intervals to Higher-Order Exchangeable Bernoulli (XBern) Intervals
The LiDP condition in Eq. (4) critically relies on the canaries being sampled i.i.d. and the rejection set only depending on the corresponding canary. For such a symmetric design, auditing boils down to deriving a Confidence Interval (CI) for a special family of distributions that we call Exchangeable Bernoulli (XBern). We derive the CI for the alternate hypothesis, i.e., the left side of Eq. (4). The CI under the null hypothesis is analogous.
Figure 1: Bias illustration: Consider the sum query  $z_{1} + z_{2}$  with 2 inputs. Its DP version produces a point in the blue circle w.h.p. due to the noise  $\xi \sim \mathcal{N}(0, I)$  scaled by  $\sigma$ . When auditing with a random canary  $c$ , it contributes additional randomness (red disc) leading to a smaller effective privacy parameter  $\varepsilon$ .
Figure 2: Left: The Bernstein CI  $[\underline{\mu},\overline{\mu} ]$  can be found by equating the two sides of Bernstein's inequality in Eq. (6). Right: The asymptotic Wilson CI is a tightening of the Bernstein CI with a smaller  $1 / \sqrt{n}$  coefficient (shown here) and no  $1 / n$  term.
Recall that  $\pmb{x}_k \coloneqq \mathbb{I}(\mathcal{A}(\pmb{D}_1) \in \pmb{R}_k)$  denotes the test statistic for the  $k^{\mathrm{th}}$  canary. By the symmetry of our design,  $\pmb{x} \in \{0, 1\}^K$  is an exchangeable random vector that is distributed as an exponential family. Further, the distribution of  $\pmb{x}$  is fully defined by a  $K$ -dimensional parameter  $(\mu_1, \dots, \mu_K)$  where  $\mu_\ell$  is the  $\ell^{\mathrm{th}}$  moment of  $\pmb{x}$ . We call this family XBern. Specifically, this implies permutation invariance of the higher-order moments:  $\mathbb{E}[x_{j_1} \cdots x_{j_\ell}] = \mathbb{E}[x_{k_1} \cdots x_{k_\ell}]$  for any distinct sets of indices  $(j_l)_{l \in [\ell]}$  and  $(k_l)_{l \in [\ell]}$  for any  $\ell \leq K$ . For example,  $\mu_1 \coloneqq \mathbb{E}[(1 / K) \sum_{k=1}^{K} x_k]$ , which is the LHS of (4).
Using samples from this XBern, we aim to derive a CI on  $\mu_{1}$  around the empirical mean  $\hat{\mu}_{1}$  in (5). Bernstein's inequality applied to our test statistic  $m_{1} := (1 / K)\sum_{k = 1}^{K}\pmb{x}_{k}$  gives,
$$
\left| \hat {\mu} _ {1} - \mu_ {1} \right| \leq \sqrt {\frac {2 \log (2 / \beta)}{n} \operatorname {V a r} \left(\boldsymbol {m} _ {1}\right)} + \frac {2 \log (2 / \beta)}{3 n}, \tag {6}
$$
w.p. at least  $1 - \beta$ . Bounding  $\operatorname{Var}(m_1) \leq \mu_1(1 - \mu_1)$  since  $m_1 \in [0, 1]$  a.s. and numerically solving the above inequality for  $\mu_1$  gives a CI that scales as  $1 / \sqrt{n}$  — see Figure 2 (left). We call this the  $1^{\text{st}}$ -order Bernstein bound, as it depends only on the  $1^{\text{st}}$  moment  $\mu_1$ . Our strategy is to measure the (higher-order) correlations between  $\pmb{x}_k$ 's to derive a tighter CI that adapts to the given instance. This idea applies to any standard CI. We derive and analyze the higher order Bernstein intervals here, and experiments use Wilson intervals from Appendix C; see also Figure 2 (right).
Concretely, we can leverage the  $2^{\mathrm{nd}}$  order correlation by expanding
$$
\operatorname {V a r} \left(\boldsymbol {m} _ {1}\right) = \frac {1}{K} \left(\mu_ {1} - \mu_ {2}\right) + \left(\mu_ {2} - \mu_ {1} ^ {2}\right) \quad \text {w h e r e} \quad \mu_ {2} := \frac {1}{K (K - 1)} \sum_ {k _ {1} <   k _ {2} \in [ K ]} \mathbb {E} \left[ \boldsymbol {x} _ {k _ {1}} \boldsymbol {x} _ {k _ {2}} \right].
$$
Ideally, when the second order correlation  $\mu_{2} - \mu_{1}^{2} = \mathbb{E}[\pmb{x}_{1}\pmb{x}_{2}] - \mathbb{E}[\pmb{x}_{1}]\mathbb{E}[\pmb{x}_{2}]$  equals 0, we have  $\mathrm{Var}(\pmb{m}_1) = \mu_1(1 - \mu_1) / K$ , a factor of  $K$  improvement over the worst-case. Our higher-order CIs adapt to the actual level of correlation of  $\pmb{x}$  by further estimating the  $2^{\mathrm{nd}}$  moment  $\mu_{2}$  from samples. Let  $\overline{\mu}_2$  be the first-order Bernstein upper bound on  $\mu_{2}$  such that  $\mathbb{P}(\mu_2\leq \overline{\mu}_2)\geq 1 - \beta$ . On this event,
$$
\operatorname {V a r} \left(\boldsymbol {m} _ {1}\right) \leq \frac {1}{K} \left(\mu_ {1} - \bar {\boldsymbol {\mu}} _ {2}\right) + \left(\bar {\boldsymbol {\mu}} _ {2} - \mu_ {1} ^ {2}\right). \tag {7}
$$
Combining this with (6) gives us the  $2^{\mathrm{nd}}$ -order Bernstein bound on  $\mu_{1}$ , valid w.p.  $1 - 2\beta$ . Since  $\overline{\mu}_2 \lesssim \hat{\mu}_2 + 1 / \sqrt{n}$  where  $\hat{\mu}_2$  is the empirical estimate of  $\mu_{2}$ , the  $2^{\mathrm{nd}}$ -order bound scales as
$$
\left| \mu_ {1} - \hat {\mu} _ {1} \right| \lesssim \sqrt {\frac {1}{n K}} + \sqrt {\frac {1}{n} \left| \hat {\mu} _ {2} - \hat {\mu} _ {1} ^ {2} \right|} + \frac {1}{n ^ {3 / 4}}, \tag {8}
$$
where constants and log factors are omitted. Thus, our  $2^{\mathrm{nd}}$ -order CI can be as small as  $1 / \sqrt{nK} + 1 / n^{3/4}$  (when  $\hat{\mu}_1^2 \approx \mu_2$ ) or as large as  $1 / \sqrt{n}$  (in the worst-case). With small enough correlations of  $|\hat{\mu}_2 - \hat{\mu}_1^2| = O(1 / K)$ , this suggests a choice of  $K = O(\sqrt{n})$  to get CI of  $1 / n^{3/4}$ . In practice, the correlation is controlled by the design of the canary. For the Gaussian mechanism with random canaries, the correlation indeed empirically decays as  $1 / K$ , as we see in §4. Thus, the CI decreases monotonically with  $K$ . This is also true for the CI under the null hypothesis with  $K$  replaced by  $m$ .
However, there is a qualitative difference between the null and alternate hypotheses. Estimation under the alternate hypothesis incurs a larger bias as the number  $K$  of canaries increases — this is due
DP + Wilson
LiDP + 1st-Order Wilson
$\star$  LiDP + 4th-Order Wilson
Figure 3: Left: For Gaussian mechanisms, the proposed LiDP-based auditing with  $K$  canaries provides a significant gain in the required number of trials to achieve a desired level of lower bound  $\hat{\varepsilon}$  on the privacy. Center: Increasing the number of canaries trades off the bias and the variance, with our prescribed  $K = \sqrt{n}$  achieving a good performance. Right: Increasing the dimension makes the canaries less correlated, thus achieving smaller confidence intervals, and larger  $\hat{\varepsilon}$ . We shade the standard error over 25 repetitions.
to the additional randomness from adding more canaries into the training process, as illustrated in Figure 1. The optimal choice of  $K$  balances the bias and variance. On the other hand, estimation under the null hypothesis incurs no bias and a larger number  $m$  of test canaries always helps. We return to the bias-variance tradeoff in §4.
Higher-order intervals. We can recursively apply this method by expanding the variance of higher-order statistics, and derive higher-order Bernstein bounds. The next recursion uses empirical  $3^{\mathrm{rd}}$  and  $4^{\mathrm{th}}$  moments  $\hat{\mu}_3$  and  $\hat{\mu}_4$  to get the  $4^{\mathrm{th}}$ -order Bernstein bound which scales as
$$
\left| \mu_ {1} - \hat {\mu} _ {1} \right| \lesssim \sqrt {\frac {1}{n K}} + \sqrt {\frac {1}{n} \left| \hat {\mu} _ {2} - \hat {\mu} _ {1} ^ {2} \right|} + \frac {1}{n ^ {3 / 4}} \left| \hat {\mu} _ {4} - \hat {\mu} _ {2} ^ {2} \right| ^ {1 / 4} + \frac {1}{n ^ {7 / 8}}. \tag {9}
$$
Ideally, when the  $4^{\mathrm{th}}$ -order correlation is small enough,  $|\hat{\mu}_4 - \hat{\mu}_2^2| = O(1 / K)$ , (along with the 2nd-order correlation,  $\hat{\mu}_2 - \hat{\mu}_1^2$ ) this 4th-order CI scales as  $1 / n^{7/8}$  with a choice of  $K = O(n^{3/4})$  improving upon the 2nd-order CI of  $1 / n^{3/4}$ . We can recursively derive even higher-order CIs, but we find in §4 that the gains diminish rapidly. In general, the  $\ell^{\mathrm{th}}$  order Bernstein bounds achieve CIs scaling as  $1 / n^{(2\ell - 1)/2\ell}$  with a choice of  $K = O(n^{(\ell - 1)/\ell})$ . This shows that the higher-order CI decreases in the order  $\ell$  of the correlations used; we refer to Appendix C for details.
Proposition 4. For any positive integer  $\ell$  that is a power of two and  $K = \lceil n^{(\ell -1) / \ell}\rceil$ , suppose we have  $n$  samples from a  $K$ -dimensional XBern distribution with parameters  $(\mu_1,\ldots ,\mu_K)$ . If all  $\ell^{\prime \prime \prime}$ -order correlations scale as  $1 / K$ , i.e.,  $|\mu_{2\ell '} - \mu_{\ell '}^2 | = O(1 / K)$ , for all  $\ell^{\prime}\leq \ell$  and  $\ell^\prime$  is a power of two, then the  $\ell^{th}$ -order Bernstein bound is  $|\mu_1 - \hat{\mu}_1| = O(1 / n^{(2\ell -1) / (2\ell)})$ .
4 Simulations: Auditing the Gaussian Mechanism
Setup. We consider a simple sum query  $q(D) = \sum_{z \in D} z$  over the unit sphere  $\mathcal{Z} = \{z \in \mathbb{R}^d : \|z\|_2 = 1\}$ . We want to audit a Gaussian mechanism that returns  $q(D) + \sigma \xi$  with standard Gaussian  $\xi \sim \mathcal{N}(0, I_d)$  and  $\sigma$  calibrated to ensure  $(\varepsilon, \delta)$ -DP. We assume black-box access, where we do not know what mechanism we are auditing and we only access it through samples of the outcomes. A white-box audit is discussed in §A.1. We apply our new recipe with canaries sampled uniformly at random from  $\mathcal{Z}$ . Following standard methods [e.g. 23], we declare that a canary  $c_k$  is present if  $c_k^\top \mathcal{A}(D) > \tau$  for a threshold  $\tau$  learned from separate samples. For more details and additional results, see Appendix E. A broad range of values of  $K$  (between 32 and 256 in Figure 3 middle) leads to good performance in auditing LiDP. We use  $K = \sqrt{n}$  (as suggested by our analysis in (8)),  $m = K$  test canaries, and the  $2^{\mathrm{nd}}$ -order Wilson estimator (as gains diminish rapidly afterward) as a reliable default setting.
Sample Complexity Gains. In Figure 3 (left), the proposed approach of injecting  $K$  canaries with the  $2^{\text{nd}}$  order Wilson interval (denoted "LiDP +2nd-Order Wilson") reduces the number of trials,  $n$ , needed to reach the same empirical lower bound,  $\hat{\varepsilon}$ , by  $4 \times$  to  $16 \times$ , compared to the baseline of injecting a single canary (denoted "DP+Wilson"). We achieve  $\hat{\varepsilon}_n = 0.85$  with  $n = 4096$  (while the baseline requires  $n = 65536$ ) and  $\hat{\varepsilon}_n = 0.67$  with  $n = 1024$  (while the baseline requires  $n = 4096$ ).
Figure 4: Left: Separating the effects of bias and variance in auditing LiDP; cf. definition (10). Center & Right: The correlations between the test statistics of the canaries decrease with  $K$  and  $d$ , achieving smaller CIs.
Number of Canaries and Bias-Variance Tradeoffs. In Fig. 3 (middle), LiDP auditing with  $2^{\mathrm{nd}} / 4^{\mathrm{th}}$ -order CIs improve with increasing canaries up to a point and then decreases. This is due to a bias-variance tradeoff, which we investigate further in Figure 4 (left). Let  $\hat{\varepsilon}(K,\ell)$  denote the empirical privacy lower bound with  $K$  canaries using an  $\ell^{\mathrm{th}}$  order interval, e.g., the baseline is  $\hat{\varepsilon}(1,1)$ . Let the bias from injecting  $K$  canaries and the variance gain from  $\ell^{\mathrm{th}}$ -order interval respectively be
$$
\Delta \operatorname {B i a s} (K) := \hat {\varepsilon} (K, 1) - \hat {\varepsilon} (1, 1), \quad \text {a n d} \quad \Delta \operatorname {V a r} (K, \ell) := \hat {\varepsilon} (K, \ell) - \hat {\varepsilon} (K, 1). \tag {10}
$$
In Figure 4 (left), the gain  $\Delta \mathrm{Bias}(K)$  from bias is negative and gets worse with increasing  $K$ ; when testing for each canary, the  $K - 1$  other canaries introduce more randomness that makes the test more private and hence lowers the  $\hat{\varepsilon}$  (see also Figure 1). The gain  $\Delta \mathrm{Var}(K)$  in variance is positive and increases with  $K$  before saturating. This improved "variance" of the estimate is a key benefit of our framework. The net improvement  $\hat{\varepsilon}(K,\ell) - \hat{\varepsilon}(1,1)$  is a sum of these two effects. This trade-off between bias and variance explains the concave shape of  $\hat{\varepsilon}$  in  $K$ .
Next, we see from Figure 5 that a larger number  $m$  of test canaries always helps, i.e., it does not incur a similar bias-variance tradeoff. Indeed, this is because larger  $m$  does not lead to any additional bias, as discussed in §3.3. However, the gains quickly saturate, so  $m = K$  is a reliable default, especially for  $K \geq 16$  or so; we refer to Appendix E for more plots and details.
Correlation between Canaries. Based on (8), this improvement in the variance can further be examined by looking at the term  $|\hat{\mu}_2 - \hat{\mu}_1^2|$  that leads to a narrower  $2^{\mathrm{nd}}$ -order Wilson interval. The log-log plot of this term in Figure 4 (middle) is nearly parallel to the dotted  $1 / K$  line (slope = -0.93), meaning that it decays roughly as  $1 / K$  (note that the log-log plot of  $y = cx^a$  is a straight line with slope  $a$ ). This indicates that we get close to a  $1 / \sqrt{nK}$  confidence interval as desired. Similarly, we get that  $|\hat{\mu}_4 - \hat{\mu}_2^2|$  decays roughly as  $1 / K$  (slope = -1.05). However, the  $4^{\mathrm{th}}$ -order estimator offers only marginal additional improvements in the small  $\varepsilon$  regime (see Appendix E). Thus, the gain diminishes rapidly in the order of our estimators.
Figure 5: Left: There is no bias-variance tradeoff with the number  $m$  of test canaries: larger  $m$  always gives a better empirical  $\hat{\varepsilon}$  but the results saturate quickly. This experiment is for the Gaussian mechanism. Center & Right: The proposed LiDP-based auditing procedure gives significant improvements similar to Figure 3 when auditing the Laplace mechanism with canaries sampled randomly from the unit  $\ell_1$  ball.
Effect of Dimension on the Bias and Variance. As the dimension  $d$  increases, the LiDP-based lower bound becomes tighter monotonically as we show in Figure 3 (right). This is due to both the bias gain, as in (10), improving (less negative) and the variance gain improving (more positive). With increasing  $d$ , the variance of our estimate reduces because the canaries become less correlated. Guided by Eq. (8,9), we measure the relevant correlation measures  $|\hat{\mu}_2 - \hat{\mu}_1^2|$  and  $|\hat{\mu}_4 - \hat{\mu}_2^2|$  in Figure 4 (right). Both decay approximate as  $1 / d$  (slope  $= -1.06$  and  $-1.00$  respectively, ignoring the outlier at  $d = 10^6$ ). This suggests that, for Gaussian mechanisms, the corresponding XBern distribution resulting from our recipe behaves favorably as  $d$  increases.
Auditing Other Mechanisms. The proposed LiDP-based auditing recipe is agnostic to the actual privacy mechanism. We audit a Laplace mechanism with canaries drawn uniformly at random over the unit  $\ell_1$  ball in Figure 4 (center and right). The results are qualitatively similar to those of the Gaussian mechanism, leading to an eight-fold improvement in the sample complexity over the baseline.
5 Lifting Existing Canary Designs
Several prior works [e.g., 32, 41, 45, 60], focus on designing stronger canaries to improve the lower bound on  $\varepsilon$ . We provide two concrete examples of how to lift these canary designs to be compatible with our framework while inheriting their strengths. We refer to Appendix D for further details.
We impose two criteria on the distribution  $P_{\mathrm{canary}}$  over canaries for auditing LiDP. First, the injected canaries are easy to detect, so that the probabilities on the left side of (4) are large and those on the right side are small. Second, a canary  $c \sim P_{\mathrm{canary}}$ , if included in the training of a model  $\theta$ , is unlikely to change the membership of  $\theta \in R_{c'}$  for an independent canary  $c' \sim P_{\mathrm{canary}}$ . Existing canary designs already impose the first condition to audit DP using (1). The second condition ensures that the canaries are uncorrelated, allowing our adaptive CIs to be smaller, as we discussed in §3.3.
Data Poisoning via Tail Singular Vectors. ClipBKD [32] adds as canaries the tail singular vector of the input data (e.g., images). This ensures that the canary is out of distribution, allowing for easy detection. We lift ClipBKD by defining the distribution  $P_{\text{canary}}$  as the uniform distribution over the  $p$  tail singular vectors of the input data. If  $p$  is small relative to the dimension  $d$  of the data, then they are still out of distribution, and hence, easy to detect. For the second condition, the orthogonality of the singular vectors ensures the interaction between canaries is minimal, as measured empirically.
Random Gradients. The approach of [2] samples random vectors (of the right norm) as canary gradients, assuming a grey-box access where we can inject gradients. Since random vectors are nearly orthogonal to any fixed vector in high dimensions, their presence is easy to detect with a dot product. Similarly, any two i.i.d. canary gradients are roughly orthogonal, leading to minimal interactions.
We compare the proposed LiDP auditing recipe relative to the standard one for DP training of machine learning models. Our code is available online.<sup>1</sup>
Setup. We test with two classification tasks: FMNIST [65] is a 10-class grayscale image classification dataset, while Purchase-100 is a sparse tabular dataset with 600 binary features and 100 classes [19, 53]. We train a linear model and a multi-layer perceptron (MLP) with 2 hidden layers using DP-SGD [1] to achieve  $(\varepsilon, 10^{-5})$ -DP with varying values of  $\varepsilon$ . The training is performed using cross-entropy for a fixed epoch budget and a batch size of 100. We refer to Appendix F for details.
Auditing. We audit the LiDP using the two types of canaries from §5: data poisoning and random gradient canaries. We vary the number  $K$  of canaries and the number  $n$  of trials. We track the empirical lower bound obtained from the Wilson family of confidence intervals. We compare this with auditing DP, which coincides with auditing LiDP with  $K = 1$  canary. We audit only the final model in all the experiments.
Sample Complexity Gain. Table 1 shows the reduction in the sample complexity from auditing LiDP. For each canary type, auditing LiDP is better 11 out of the 12 settings considered. The average improvement (i.e., the harmonic mean over the table) for data poisoning is  $2.3 \times$ , while for random
<https://github.com/google-research/federated/tree/master/lidp_auditing>
Dataset / Model	C.I./ (Wilson)	Data Poisoning Canary	Random Gradient Canary
ε = 2	ε = 4	ε = 8	ε = 16	ε = 2	ε = 4	ε = 8	ε = 16
FMNIST / Linear	2nd-Ord.	0.68	3.31	2.55	4.38	2.69	3.34	5.98	5.06
4th-Ord.	0.42	2.68	2.29	3.76	2.66	2.70	7.75	4.19
FMNIST / MLP	2nd-Ord.	4.80	1.46	2.95	1.60	4.62	2.88	2.33	5.46
4th-Ord.	4.42	2.49	2.37	1.30	3.95	2.81	2.02	4.60
Purchase / MLP	2nd-Ord.	3.14	1.06	1.41	9.28	1.41	0.71	4.30	2.84
4th-Ord.	2.84	1.09	1.36	6.99	1.29	0.42	4.35	2.38
Table 1: The (multiplicative) improvement in the sample complexity from auditing LiDP with  $K = 16$  canaries compared to auditing DP with  $n = 1000$  trials. We determine this factor by linearly interpolating/extrapolating  $\hat{\varepsilon}_n$ ; cf. Figure 6 (left) for a visual representation of these numbers. For instance, an improvement of 3.31 means LiDP needs  $n \approx 1000 / 3.31 \approx 302$  trials to reach the same empirical lower bound that DP reaches at  $n = 1000$ .
Figure 6: Left two: LiDP-based auditing with  $K > 1$  canaries achieves the same lower bound  $\hat{\varepsilon}$  on the privacy loss with fewer trials. Right two: LiDP auditing is robust to  $K$ ; the prescribed  $K = \sqrt{n}$  is a reliable default.
gradients, it is  $3.0 \times$ . Since each trial is a full model training run, this improvement can be quite significant in practice. This improvement can also be seen visually in Figure 6 (left two).
Number of Canaries. We see from Figure 6 (right two) that LiDP auditing on real data behaves similarly to Figure 3 in §4. We also observe the bias-variance tradeoff in the case of data poisoning (center right). The choice  $K = \sqrt{n}$  is competitive with the best value of  $K$ , validating our heuristic. Overall, these results show that the insights from §4 hold even with the significantly more complicated DP mechanism involved in training models privately.
Additional Comparisons. We show in §F.4 that our LiDP auditing with generally outperforms the Bayesian auditing approach of Zanella-Béguelin et al. [69]. This also suggests an interesting future research direction: adapt Bayesian credible intervals for LiDP auditing to get the best of both worlds.
We introduce a new framework for auditing differentially private learning. Diverging from the standard practice of adding a single deterministic canary, we propose a new recipe of adding multiple i.i.d. random canaries. This is made rigorous by an expanded definition of privacy that we call LiDP. We provide novel higher-order confidence intervals that can automatically adapt to the level of correlation in the data. We empirically demonstrate that there is a potentially significant gain in sample dependence of the confidence intervals, achieving a favorable bias-variance tradeoff.
Although any rigorous statistical auditing approach can benefit from our framework, it is not yet clear how other popular approaches [e.g. 12] for measuring memorization can be improved with randomization. Bridging this gap is an important practical direction for future research. It is also worth considering how our approach can be adapted to audit the diverse definitions of privacy in machine learning [e.g. 28].
Broader Impact. Auditing private training involves a trade-off between the computational cost and the tightness of the guarantee. This may not be appropriate for all practical settings. For deployment in production, it is worth further studying approaches with minimal computational overhead [e.g. 2, 12].
We acknowledge Lang Liu for helpful discussions regarding hypothesis testing and conditional probability, Matthew Jagielski for help in debugging the experiments with data poisoning canaries, and Zeba Islam for help in simplifying proofs involving combinatorics.
A Related Work 16
A.1 Auditing Private Machine Learning with Strong Canaries 16
A.2 Improving Statistical Trade-offs in Auditing 17
A.3 Connections to Other Notions of Differential Privacy 17
B Properties of Lifted DP and Further Details 17
B.1 Equivalence Between DP and LiDP 17
B.2 Auditing LiDP with Different Notions of Neighborhood 18