SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 11 (Previous studies on the hardness of privacy verification). Several studies [16, 8] have shown that DP verification is an NP-hard problem. However, these works consider the setting where the input description of the DP mechanism is its corresponding randomized Boolean circuits. Some other works [18] show that DP verification is impossible, but this assertion is proved for the black-box setting where the verifier can only query the mechanism. Our work gets around this barrier by providing the description of the PRV of the mechanism as input to the verifier.
4.1 DPV through an MC Estimator for  $\delta_Y(\varepsilon)$
Recall that most of the recently proposed DP accountants are essentially different techniques for estimating the expectation
$$
\delta_ {Y = \sum_ {i = 1} ^ {k} Y _ {i}} (\varepsilon) = \mathbb {E} _ {Y} \left[ \left(1 - e ^ {\varepsilon - Y}\right) _ {+} \right]
$$
where each  $Y_{i}$  is the privacy loss random variable  $Y_{i} = \log \left(\frac{P_{i}(t)}{Q_{i}(t)}\right)$  for  $t \sim P_{i}$ , and  $(P_{i}, Q_{i})$  is a pair of dominating distribution for individual mechanism  $\mathcal{M}_{i}$ . In the following text, we denote the product distribution  $\pmb{P} := P_{1} \times \ldots \times P_{k}$  and  $\pmb{Q} := Q_{1} \times \ldots \times Q_{k}$ . Recall from Lemma 5 that  $(\pmb{P}, \pmb{Q})$  is a pair of dominating distributions for the composed mechanism  $\mathcal{M}$ . For notation simplicity, we denote a vector  $\pmb{t} := (t^{(1)}, \ldots, t^{(k)})$ .
Monte Carlo (MC) technique is arguably one of the most natural and widely used techniques for approximating expectations. Since  $\delta_Y(\varepsilon)$  is an expectation in terms of the PRV  $Y$ , one can apply MC-based technique to estimate it. Given an MC estimator for  $\delta_Y(\varepsilon)$ , we construct a  $\mathrm{DPV}(\mathcal{M},\varepsilon ,\delta^{\mathrm{est}})$  as shown in Algorithm 2 (instantiated by the Simple MC estimator introduced in Section 4.2). Specifically, we first obtain an estimate  $\widehat{\delta}$  from an MC estimator for  $\delta_Y(\varepsilon)$ . The estimate  $\delta^{\mathrm{est}}$  passes the test if
$\widehat{\delta} < \frac{\delta^{\mathrm{est}}}{\tau} - \Delta$ , and fails otherwise. The parameter  $\Delta \geq 0$  here is an offset that allows us to conveniently controls the  $\tau$ -relaxed false positive rate. We will discuss how to set  $\Delta$  in Section 4.4.
In the following contents, we first present two constructions of MC estimators for  $\delta_Y(\varepsilon)$  in Section 4.2. We then discuss the condition for which our MC-based DPV achieves a certain target FP rate in Section 4.3. Finally, we discuss the utility guarantee for the MC-based DPV in Section 4.4.
Algorithm 2 DPV  $(\mathcal{M},\varepsilon ,\delta^{\mathrm{est}})$  with Simple MC Estimator and Offset Parameter  $\Delta$
1: Obtain i.i.d. samples  $\{\pmb{t}_i\}_{i=1}^m$  from  $P$ .
2: Compute  $\widehat{\delta} = \frac{1}{m}\sum_{i=1}^{m}(1 - e^{\varepsilon - y_i})_+$  with PRV samples  $y_i = \log\left(\frac{P(t_i)}{Q(t_i)}\right), i = 1 \ldots m.$
3: if  $\widehat{\delta} < \frac{\delta^{\mathrm{est}}}{\tau} - \Delta$  then return True.
4: else return False.
4.2 Constructing MC Estimator for  $\delta_Y(\varepsilon)$
In this section, we first present a simple MC estimator that applies to any mechanisms where we can derive and sample from the dominating distribution pairs. Given the importance of Poisson Subsampled Gaussian mechanism for privacy-preserving machine learning, we further design a more advanced and specialized MC estimator for it based on the importance sampling technique.
Simple Monte Carlo Estimator. One can easily sample from  $Y$  by sampling  $t \sim P$  and output  $\log \left( \frac{P(t)}{Q(t)} \right)$ . Hence, a straightforward algorithm for estimating (2) is the Simple Monte Carlo (SMC) algorithm, which directly samples from the privacy random variable  $Y$ . We formally define it here.
Definition 12 (Simple Monte Carlo (SMC) Estimator). We denote  $\widehat{\delta}_{MC}^{m}(\varepsilon)$  as the random variable of SMC estimator for  $\delta_{Y}(\varepsilon)$  with  $m$  samples, i.e.,  $\widehat{\delta}_{MC}^{m}(\varepsilon) := \frac{1}{m}\sum_{i=1}^{m}(1 - e^{\varepsilon - y_i})_+$  for  $y_1, \ldots, y_m$  i.i.d. sampled from  $Y$ .
Importance Sampling Estimator for Poisson Subsampled Gaussian (Overview). As  $\delta_Y(\varepsilon)$  is usually a tiny value ( $10^{-5}$  or even cryptographically small), it is likely that by naive sampling from  $Y$ , almost all of the samples in  $\{(1 - e^{\varepsilon - y_i})_+\}_{i=1}^m$  are just 0s! That is, the i.i.d. samples  $\{y_i\}_{i=1}^m$  from  $Y$  can rarely exceed  $\varepsilon$ . To further improve the sample efficiency, one can potentially use more advanced MC techniques such as Importance Sampling or MCMC. However, these advanced tools usually require additional distributional information about  $Y$  and thus need to be developed case-by-case.
Poisson Subsampled Gaussian mechanism is the main workhorse behind the DP-SGD algorithm [1]. Given its important role in privacy-preserving ML, we derive an advanced MC estimator for it based on the Importance Sampling technique. Importance Sampling (IS) is a classic method for rare event simulation [39]. It samples from an alternative distribution instead of the distribution of the quantity of interest, and a weighting factor is then used for correcting the difference between the two distributions. The specific design of alternative distribution is complicated and notation-heavy, and we defer the technical details to Appendix C. At a high level, we construct the alternative sampling distribution based on the exponential tilting technique and derive the optimal tilting parameter such that the corresponding IS estimator approximately achieves the smallest variance. Similar to Definition 12, we use  $\widehat{\delta}_{\mathrm{IS}}^{m}$  to denote the random variable of importance sampling estimator with  $m$  samples.
4.3 Bounding FP Rate
We now discuss the FP guarantee for the DPV instantiated by  $\widehat{\delta}_{\mathrm{MC}}^m$  and  $\widehat{\delta}_{\mathrm{IS}}^m$  we developed in the last section. Since both estimators are unbiased, by Law of Large Number, both  $\widehat{\delta}_{\mathrm{MC}}^m$  and  $\widehat{\delta}_{\mathrm{IS}}^m$  converge to  $\delta_Y(\varepsilon)$  almost surely as  $m\to \infty$ , which leads a DPV with perfect accuracy. Of course,  $m$  cannot go to  $\infty$  in practice. In the following, we derive the required amount of samples  $m$  for ensuring that  $\tau$ -relaxed false positive rate is smaller than  $\delta^{\mathrm{est}} / \tau$  for  $\widehat{\delta}_{\mathrm{MC}}^m$  and  $\widehat{\delta}_{\mathrm{IS}}^m$ . We use  $\widehat{\delta}_{\mathrm{MC}}$  (or  $\widehat{\delta}_{\mathrm{IS}}$ ) as an abbreviation for  $\widehat{\delta}_{\mathrm{MC}}^1$  (or  $\widehat{\delta}_{\mathrm{IS}}^1$ ), the random variable for a single draw of sampling. We state the theorem for  $\widehat{\delta}_{\mathrm{MC}}^m$ , and the same result for  $\widehat{\delta}_{\mathrm{IS}}^m$  can be obtained by simply replacing  $\widehat{\delta}_{\mathrm{MC}}$  with  $\widehat{\delta}_{\mathrm{IS}}$ . We use FP to denote the FP rate for DPV implemented by SMC estimator.
Theorem 13. Suppose  $\mathbb{E}\left[\left(\widehat{\delta}_{\mathcal{MC}}\right)^2\right] \leq \nu$ . DPV instantiated by  $\widehat{\delta}_{\mathcal{MC}}^m$  has bounded  $\tau$ -relaxed false positive rate  $\mathrm{FP}_{\mathcal{MC}}(\varepsilon, \delta^{\mathrm{est}}; \tau) \leq \delta^{\mathrm{est}} / \tau$  with  $m \geq \frac{2\nu}{\Delta^2} \log (\tau / \delta^{\mathrm{est}})$ .
The proof is based on Bennett's inequality and is deferred to Appendix D. This result suggests that, to improve the computational efficiency of MC-based DPV (i.e., tighten the number of required samples), it is important to tightly bound  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2]$  (or  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$ ), the second moment of  $\widehat{\delta}_{\mathrm{MC}}$  (or  $\widehat{\delta}_{\mathrm{IS}}$ ).
Bounding the Second-Moment of MC Estimators (Overview). For clarity, we defer the notation-heavy results and derivation of the upper bounds for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2]$  and  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS}})^2]$  to Appendix E. Our high-level idea for bounding  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2]$  is through the RDP guarantee for the composed mechanism  $\mathcal{M}$ . This is a natural idea since converting RDP to upper bounds for  $\delta_Y(\varepsilon)$  - the first moment of  $\widehat{\delta}_{\mathrm{MC}}$  - is a well-studied problem [30, 9, 3]. Bounding  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS}})^2]$  is highly technically involved.
4.4 Guaranteeing Utility
Overall picture so far. Given the proposed privacy parameter  $(\varepsilon, \delta^{\mathrm{est}})$ , a tolerable degree of underestimation  $\tau$ , and an offset parameter  $\Delta$ , one can now compute the number of samples  $m$  required for the MC-based DPV such that  $\tau$ -relaxed FP rate to be  $\leq \delta^{\mathrm{est}} / \tau$  based on the results from Section 4.3 and Appendix E. We have not yet discussed the selection of the hyperparameter  $\Delta$ . An appropriate  $\Delta$  is important for the utility of MC-based DPV. That is, when  $\delta^{\mathrm{est}}$  is not too smaller than  $\delta_Y(\varepsilon)$ , the probability of being rejected by DPV should stay negligible. If we set  $\Delta \to \infty$ , the DPV simply rejects everything, which achieves 0 FP rate (and with  $m = 0$ ) but is not useful at all!
Formally, the utility of a DPV is quantified by the  $\rho$ -relaxed false negative (FN) rate (Definition 8). While one may be able to bound the FN rate through concentration inequalities, a more convenient
way is to pick an appropriate  $\Delta$  such that  $\mathsf{FN}_{\mathsf{DPV}}$  is approximately smaller than  $\mathsf{FP}_{\mathsf{DPV}}$ . After all,  $\mathsf{FP}_{\mathsf{DPV}}$  already has to be a small value  $\leq \delta^{\mathrm{est}} / \tau$  for privacy guarantee. The result is stated informally in the following (holds for both  $\widehat{\delta}_{\mathsf{MC}}$  and  $\widehat{\delta}_{\mathsf{IS}}$ ), and the involved derivation is deferred to Appendix F.
Theorem 14 (Informal). When  $\Delta = 0.4(1 / \tau -1 / \rho)\delta^{\mathrm{est}}$ , then  $\mathsf{FN}_{\mathcal{MC}}(\varepsilon ,\delta^{\mathrm{est}};\rho)\lesssim \mathsf{FP}_{\mathcal{MC}}(\varepsilon ,\delta^{\mathrm{est}};\tau)$
Therefore, by setting  $\Delta = 0.4(1 / \tau -1 / \rho)\delta^{\mathrm{est}}$ , one can ensure that  $\mathsf{FN}_{\mathsf{MC}}(\varepsilon ,\delta^{\mathrm{est}};\rho)$  is also (approximately) upper bounded by  $\Theta (\delta^{\mathrm{est}} / \tau)$ . Moreover, in Appendix, we empirically show that the FP rate is actually a very conservative bound for the FN rate. Both  $\tau$  and  $\rho$  are selected based on the tradeoff between privacy, utility, and efficiency.
The pseudocode of privacy verification for DP-SGD is summarized in Appendix G.
5 Monte Carlo Accountant of Differential Privacy
The Monte Carlo estimators  $\widehat{\delta}_{\mathrm{MC}}$  and  $\widehat{\delta}_{\mathrm{IS}}$  described in Section 4.2 are used for implementing DP verifiers. One may already realize that the same estimators can also be utilized to directly implement a DP accountant which estimates  $\delta_{Y}(\varepsilon)$ . It is important to note that with the EVR paradigm, DP accountants are no longer required to derive a strict upper bound for  $\delta_{Y}(\varepsilon)$ . We refer to the technique of estimating  $\delta_{Y}(\varepsilon)$  using the MC estimators as Monte Carlo accountant.
Finding  $\varepsilon$  for a given  $\delta$ . It is straightforward to implement MC accountant when we fix  $\varepsilon$  and compute for  $\delta_Y(\varepsilon)$ . In practice, privacy practitioners often want to do the inverse: finding  $\varepsilon$  for a given  $\delta$ , which we denote as  $\varepsilon_Y(\delta)$ . Similar to the existing privacy accounting methods, we use binary search to find  $\varepsilon_Y(\delta)$  (see Algorithm 3). Specifically, after generating PRV samples  $\{y_i\}_{i=1}^m$ , we simply need to find the  $\varepsilon$  such that  $\frac{1}{m}\sum_{i=1}^{m}(1 - e^{\varepsilon - y_i})_+ = \delta$ . We do not
Algorithm 3 MC Accountant for  $\varepsilon_{Y}(\delta)$
1: Obtain PRV samples  $\{y_i\}_{i=1}^m$  with either Simple MC or Importance Sampling.
2: Binary search  $\varepsilon$  such that  $\frac{1}{m}\sum_{i = 1}^{m}(1 - e^{\varepsilon -y_i})_+ = \delta .$
3: Return  $\varepsilon$ .
need to generate new PRV samples for different  $\varepsilon$  we evaluate during the binary search; hence the additional binary search is computationally efficient.
Number of Samples for MC Accountant. Compared with the number of samples required for achieving the FP guarantee in Section 4.3, one may be able to use much fewer samples to obtain a decent estimate for  $\delta_Y(\varepsilon)$ , as the sample complexity bound derived based on concentration inequality may be conservative. Many heuristics for guiding the number of samples in MC simulation have been developed (e.g., Wald confidence interval) and can be applied to the setting of MC accountants.
Compared with FFT-based and CLT-based methods, MC accountant exhibits the following strength:
(1) Accurate  $\delta_Y(\varepsilon)$  estimation in all regimes. As we mentioned earlier, the state-of-the-art FFT-based method [19] fails to provide meaningful bounds due to computational limitations when the true value of  $\delta_Y(\varepsilon)$  is small. In contrast, the simplicity of the MC accountant allows us to accurately estimate  $\delta_Y(\varepsilon)$  in all regimes.
(2) Short clock runtime & Easy GPU acceleration. MC-based techniques are well-suited for parallel computing and GPU acceleration due to their nature of repeated sampling. One can easily utilize PyTorch's CUDA functionality (e.g., torch Randn(size=(k, m)). CUDA(*)*sigma+mu) to significantly boost the computational efficiency for sampling from common distributions such as Gaussian. In Appendix H, we show that when using one NVIDIA A100 GPU, the runtime time of sampling Gaussian mixture  $(1 - q)\mathcal{N}(0, \sigma^2) + q\mathcal{N}(1, \sigma^2)$  can be improved by  $10^{3}$  times compared with CPU-only scenario.
(3) Efficient online privacy accounting. When training ML models with DP-SGD or its variants, a privacy practitioner usually wants to compute a running privacy leakage for every training iteration, and pick the checkpoint with the best utility-privacy tradeoff. This involves estimating  $\delta_{Y^{(i)}}(\varepsilon)$  for every  $i = 1,\dots ,k$ , where  $Y^{(i)}\coloneqq \sum_{j = 1}^{i}Y_{j}$ . We refer to such a scenario as online privacy accounting<sup>2</sup>. MC accountant is especially efficient for online privacy accounting. When estimating
Note that this is different from the scenario of privacy odometer [36], as here the privacy parameter of the next individual mechanism is not adaptively chosen.
$\delta_{Y^{(i)}}(\varepsilon)$ , one can re-use the samples previously drawn from  $Y_{1}, \ldots, Y_{i-1}$  that were used for estimating privacy loss at earlier iterations.
These advantages are justified empirically in Section 6 and Appendix H.
In this section, we conduct numerical experiments to illustrate (1) EVR paradigm with MC verifiers enables a tighter privacy analysis, and (2) MC accountant achieves state-of-the-art performance in privacy parameter estimation.
6.1 EVR vs Upper Bound
To illustrate the advantage of the EVR paradigm compared with directly using a strict upper bound for privacy parameters, we take the current state-of-the-art DP accountant, the FFT-based method from [19] as the example.
EVR provides a tighter privacy guarantee.
Recall that in Figure 1, FFT-based method provides vacuous bound when the ground-truth  $\delta_Y(\varepsilon) < 10^{-10}$ . Under the same hyperparameter setting, Figure 3 (a) shows the privacy bound of the EVR paradigm where the  $\delta^{\mathrm{est}}$  are FFT's estimates. We use the Importance Sampling estimator  $\widehat{\delta}_{\mathrm{IS}}$  for DP verification. We experiment with different values of  $\tau$ . A higher value of  $\tau$  leads to tighter privacy guarantee but longer runtime. For fair comparison, the EVR's output distribution needs to be almost indistinguishable from the original mechanism. We set  $\rho = (1 + \tau) / 2$  and set  $\Delta$  according to the heuristic from Theorem 14. This guarantees that, as long as the estimate of  $\delta^{\mathrm{est}}$  from FFT is not a big underestimation (i.e., as long as  $\delta^{\mathrm{est}} \geq \rho \delta_Y(\varepsilon)$ ), the failure probability of the EVR paradigm is
Figure 3: Privacy analysis and runtime of the EVR paradigm. The settings are the same as Figure 1. For (a), when  $\tau >0.9$ , the curves are indistinguishable from 'Exact'. For fair comparison, we set  $\rho = (1 + \tau) / 2$  and set  $\Delta$  according to Theorem 14, which ensures EVR's failure probability of the order of  $\delta$ . For (b), the runtime is estimated on an NVIDIA A100-SXM4-80GB GPU.
negligible  $(O(\delta_Y(\varepsilon)))$ . The 'FFT-EVR' curve in Figure 3 (a) is essentially the 'FFT-est' curve in Figure 1 scaled up by  $1 / \tau$ . As we can see, EVR provides a significantly better privacy analysis in the regime where the 'FFT-upp' is unmeaningful ( $\delta < 10^{-10}$ ).
EVR incurs little extra runtime. In Figure 3 (b), we plot the runtime of the Importance Sampling verifier in Figure 3 (b) for different  $\tau \geq 0.9$ . Note that for  $\tau > 0.9$ , the privacy curves are indistinguishable from 'Exact' in Figure 3 (a). The runtime of EVR is determined by the number of samples required to achieve the target  $\tau$ -relaxed FP rate from Theorem 13. Smaller  $\tau$  leads to faster DP verification. As we can see, even when  $\tau = 0.99$ , the runtime of DP verification in the EVR is  $< 2$  minutes. This is attributable to the sample-efficient IS estimator and GPU acceleration.
EVR provides better privacy-utility tradeoff for Privacy-preserving ML with minimal time consumption. To further underscore the superiority of the EVR paradigm in practical applications, we illustrate the privacy-utility tradeoff curve when finetuning on CIFAR100 dataset with DP-SGD. As shown in Figure 4, the EVR paradigm provides a lower test error across all privacy budget  $\varepsilon$  compared with the traditional upper bound method. For instance, it achieves around  $7\%$  (relative) error reduction when  $\varepsilon = 0.6$ . The runtime time required for privacy verification is less than  $< 10^{-10}$  seconds for all  $\varepsilon$ , which is negligible compared to the training time. We provide additional experimental results in Appendix H.
6.2 MC Accountant
We evaluate the MC Accountant proposed in Section 5. We focus on privacy accounting for the composition of Poisson Subsampled Gaussian mechanisms, the algorithm behind the famous DP-SGD algorithm [1]. The mechanism is specified by the noise magnitude  $\sigma$  and subsampling rate  $q$ .
Settings. We consider two practical scenarios of privacy accounting: (1) Offline accounting which aims at estimating  $\delta_{Y^{(k)}}(\varepsilon)$ , and (2) Online accounting which aims at estimating  $\delta_{Y^{(i)}}(\varepsilon)$  for all  $i = 1, \dots, k$ . For space constraint, we only show the results of online accounting here, and defer the results for offline accounting to Appendix H. Metric: Relative Error. To easily and fairly evaluate the performance of privacy parameter estimation, we compute the almost exact (yet computationally expensive) privacy parameters as the ground-truth value. The ground-truth value allows us to compute the relative error of an estimate of privacy leakage. That is, if the corresponding ground-truth of an estimate  $\widehat{\delta}$  is  $\delta$ , then the relative error  $r_{\mathrm{err}} = |\widehat{\delta} - \delta| / \delta$ . Implementation. For MC accountant, we use the IS estimator described in Section 4.2. For baselines, in addition to the FFT-based and CLT-based method we mentioned earlier, we also examine AFA [44] and GDP accountant [7]. For a fair comparison, we adjust the number of samples for MC accountant so that the runtime of MC accountant and FFT is comparable. Note that we compared with the privacy parameter estimates instead of upper bounds from the baselines. Detailed settings for both MC accountant in Appendix H.
Results for Online Accounting: MC accountant is both more accurate and efficient. Figure 5 (a) shows the online accounting results for  $(\sigma, \delta, q) = (1.0, 10^{-9}, 10^{-3})$ . As we can see, MC accountant outperforms all of the baselines in estimating  $\varepsilon_{Y}(\delta)$ . The sharp decrease in FFT at approximately 250 steps is due to the transition of FFT's estimates from underestimating before this point to overestimating after. Figure 5 (b) shows that MC accountant is around 5 times faster than FFT, the baseline with the best performance in (a). This showcases the MC accountant's efficiency and accuracy in online setting.
7 Conclusion & Limitations
Figure 4: Utility-privacy tradeoff curve for fine-tuning ImageNet-pretrained BEiT [4] on CIFAR100 when  $\delta = 10^{-5}$ . We follow the training procedure from [34].
Figure 5: Experiment for Composing Subsampled Gaussian Mechanisms in the Online Setting. (a) Compares the relative error in approximating  $k \mapsto \varepsilon_Y(\delta)$ . The error bar for MC accountant is the variance taken over 5 independent runs. Note that the y-axis is in the log scale. (b) Compares the cumulative runtime for online privacy accounting. We did not show AFA [44] as it does not terminate in 24 hours.
This paper tackles the challenge of deriving provable privacy leakage upper bounds in privacy accounting. We present the estimate-verify-release (EVR) paradigm which enables the safe use of privacy parameter estimate. Limitations. Currently, our MC-based DP verifier and accountant require known and efficiently sampleable dominating pairs and PRV for the individual mechanism. Fortunately, this applies to commonly used mechanisms such as Gaussian mechanism and DP-SGD. Generalizing MC-based DP verifier and accountant to other mechanisms is an interesting future work.
This work was supported in part by the National Science Foundation under grants CNS-2131938, CNS-1553437, CNS-1704105, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, and Princeton E-affiliates Award, Amazon-Virginia Tech Initiative in Efficient and Robust Machine Learning, and Princeton's Gordon Y. S. Wu Fellowship. We are grateful to anonymous reviewers at NeurIPS for their valuable feedback.