SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318, 2016.
[2] Wael Alghamdi, Juan Felipe Gomez, Shahab Asoodeh, Flavio Calmon, Oliver Kosut, and Lalitha Sankar. The saddle-point method in differential privacy. In Proceedings of the 40th International Conference on Machine Learning, pages 508-528, 2023.
[3] Shahab Asoodeh, Jiachun Liao, Flavio P Calmon, Oliver Kosut, and Lalitha Sankar. Three variants of differential privacy: Lossless conversion and applications. IEEE Journal on Selected Areas in Information Theory, 2(1):208-222, 2021.
[4] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021.
[5] Gilles Barthe and Federico Olmedo. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In International Colloquium on Automata, Languages, and Programming, pages 49-60. Springer, 2013.
[6] Benjamin Bichsel, Samuel Steffen, Ilija Bogunovic, and Martin Vechev. Dp-sniper: Black-box discovery of differential privacy violations using classifiers. In 2021 IEEE Symposium on Security and Privacy (SP), pages 391-409. IEEE, 2021.
[7] Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review, 2020(23), 2020.
[8] Mark Bun, Marco Gabordi, and Ludmila Glinskih. The complexity of verifying boolean programs as differentially private. In 2022 IEEE 35th Computer Security Foundations Symposium (CSF), pages 396-411. IEEE, 2022.
[9] Clément L Canonne, Gautam Kamath, and Thomas Steinke. The discrete gaussian for differential privacy. Advances in Neural Information Processing Systems, 33:15676-15688, 2020.
[10] Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84(1):3-37, 2019.
[11] Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. Proceedings on Privacy Enhancing Technologies, 4:552-570, 2022.
[12] Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing, pages 371-380, 2009.
[13] Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
[14] Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3-4):211-407, 2014.
[15] Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science, pages 51-60. IEEE, 2010.
[34] Ashwinee Panda, Xinyu Tang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal. Dp-raft: A differentially private recipe for accelerated fine-tuning. arXiv preprint arXiv:2212.04486, 2022.
[35] Rachel Redberg, Yuqing Zhu, and Yu-Xiang Wang. Generalized ptr: User-friendly recipes for data-adaptive algorithms with differential privacy. In International Conference on Artificial Intelligence and Statistics, pages 3977-4005. PMLR, 2023.
[36] Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. Privacy odometers and filters: Pay-as-you-go composition. Advances in Neural Information Processing Systems, 29, 2016.
[37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022.
[38] Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. arXiv preprint arXiv:2305.08846, 2023.
[39] Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics, 2(1):54-60, 2010.
[40] Salil Vadhan. The complexity of differential privacy. Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich, pages 347-450, 2017.
[41] Hua Wang, Sheng Gao, Huanyu Zhang, Milan Shen, and Weijie J Su. Analytical composition of differential privacy via the edgeworth accountant. arXiv preprint arXiv:2206.04236, 2022.
[42] Jiachen T Wang, Saeed Mahloujifar, Shouda Wang, Ruoxi Jia, and Prateek Mittal. Renyi differential privacy of propose-test-release and applications to private and robust machine learning. Advances in Neural Information Processing Systems, 35:38719-38732, 2022.
[43] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104-12113, 2022.
[44] Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics, pages 4782-4817. PMLR, 2022.
In this section, we review the related works in privacy accounting, privacy verification, privacy auditing, and we also discuss the connection between our EVR paradigm and the famous Propose-Test-Release (PTR) paradigm.
Privacy Accounting. Early privacy accounting techniques such as Advanced Composition Theorem [15] only make use of the privacy parameters of the individual mechanisms, which bounds  $\delta_{\mathcal{M}}(\varepsilon)$  in terms of the privacy parameter  $(\varepsilon_i,\delta_i)$  for each  $\mathcal{M}_i,i = 1,\dots ,k$  . The optimal bound for  $\delta_{\mathcal{M}}(\varepsilon)$  under this condition has been derived [22, 31]. However, the computation of the optimal bound is #P-hard in general. Bounding  $\delta_{\mathcal{M}}(\varepsilon)$  only in terms of  $(\varepsilon_i,\delta_i)$  is often sub-optimal for many commonly used mechanisms [31]. This disadvantage has spurred many recent advances in privacy accounting by making use of more statistical information from the specific mechanisms to be composed [1, 30, 24, 7, 23, 25, 19, 44, 17, 11, 41, 2]. All of these works can be described as approximating the expectation in (2) when  $Y = \sum_{i = 1}^{k}Y_{i}$  . For instance, the line of [24, 7, 23, 25, 19, 17, 11] discretize the domain of each  $Y_{i}$  and use Fast Fourier Transform (FFT) in order to speed up the approximation of  $\delta_Y(\varepsilon)$  . [44] tracks the characteristic function of the privacy loss random variables for the composed mechanism and still requires discretization when the mechanisms do not have closed-form characteristic functions. The line of [7, 41] uses Central Limit Theorem (CLT) to approximate the distribution of  $Y = \sum_{i = 1}^{k}Y_{i}$  as Gaussian distribution and uses the finite-sample bound to derive the strict upper bound for  $\delta_Y(\varepsilon)$  . We also note that [29] also uses Monte Carlo approaches to calculate optimal membership inference bounds. They use a similar Simple MC estimator as the one in Section 4.2. Although their Monte Carlo approach is similar, their error analysis only works for large values of  $\delta$ $(\delta \approx 0.5)$  as they use sub-optimal concentration bounds.
Privacy Verification. As we mentioned in Remark 11, some previous works have also studied the problem of privacy verification. Most of the works consider either "white-box setting" where the input description of the DP mechanism is its corresponding randomized Boolean circuits [16, 8]. Some other works consider an even more stringent "black-box setting" where the verifier can only query the mechanism [18, 26, 6, 20, 28]. In contrast, our MC verifier is designed specifically for those mechanisms where the PRV can be derived, which includes many commonly used mechanisms such as the Subsampled Gaussian mechanism.
Privacy verification via auditing. Several heuristics have tried to perform DP verification, forming a line of work called auditing differential privacy [21, 33, 27, 32, 38]. Specifically, these techniques can verify a claimed privacy parameter by computing a lower bound for the actual privacy parameter, and comparing that with the claimed privacy parameter. The input description of mechanism  $\mathcal{M}$  for DPV, in this case, is a black-box oracle  $\mathcal{M}(\cdot)$ , where the DPV makes multiple queries to  $\mathcal{M}(\cdot)$  and estimates the actual privacy leakage. Privacy auditing techniques can achieve  $100\%$  accuracy when  $\delta^{\mathrm{est}} > \delta_Y(\varepsilon)$  (or  $0\ \rho$ -FN rate for any  $\rho \leq 1$ ), as the computed lower bound is guaranteed to be smaller than  $\delta^{\mathrm{est}}$ . However, when  $\delta^{\mathrm{est}}$  lies between  $\delta_Y(\varepsilon)$  and the computed lower bound, the DP verification will be wrong. Moreover, such techniques do not have a guarantee for the lower bound's tightness.
Remark 15 (Connection between our EVR paradigm and the Propose-Test-Release (PTR) paradigm [12]). PTR is a classic differential privacy paradigm introduced over a decade ago by [12], and is being generalized in [35, 42]. At a high level, PTR checks if releasing the query answer is safe with a certain amount of randomness (in a private way). If the test is passed, the query answer is released; otherwise, the program is terminated. PTR shares a similar underlying philosophy with our EVR paradigm. However, they are fundamentally different in terms of implementation. The verification step in EVR is completely independent of the dataset. In contrast, the test step in PTR measures the privacy risks for the mechanism  $\mathcal{M}$  on a specific dataset  $D$ , which means that the test itself may cause additional privacy leakage. One way to think about the difference is that EVR asks "whether  $\mathcal{M}$  is private", while PTR asks "whether  $\mathcal{M}(D)$  is private".
B Proofs for Privacy
Proof. For any mechanism  $\mathcal{M}$ , we denote  $A$  as the event that  $\delta^{\mathrm{est}} \geq \tau \delta_Y(\varepsilon)$ , and indicator variable  $B = \mathbb{1}[\mathrm{DPV}(\mathcal{M}, \varepsilon, \delta^{\mathrm{est}}, \tau) = \mathrm{True}]$ . Note that event  $A$  implies  $\mathcal{M}$  is  $(\delta^{\mathrm{est}} / \tau)$ -DP.
Thus, we know that
$$
\Pr [ B = 1 | \bar {A} ] \leq \mathbb {F P} _ {\mathrm {D P V}} (\varepsilon , \delta^ {\text {e s t}}; \tau) \tag {1}
$$
For notation simplicity, we also denote  $p_{\mathsf{FP}} \coloneqq \operatorname*{Pr}[B = 1|\bar{A}]$ , and  $p_{\mathsf{TP}} \coloneqq \operatorname*{Pr}[B = 1|A]$ .
For any possible event  $S$
$$
\begin{array}{l} \operatorname * {P r} _ {\mathcal {M} ^ {\mathrm {a u g}}} \left[ \mathcal {M} ^ {\mathrm {a u g}} (D) \in S \right] \\ = \Pr_ {\mathcal {M}} [ \mathcal {M} (D) \in S | B = 1 ] \Pr [ B = 1 ] + I [ \bot \in S ] \Pr [ B = 0 ] \\ = \Pr_ {\mathcal {M}} [ \mathcal {M} (D) \in S | B = 1, A ] \Pr [ B = 1 | A ] I [ A ] + \Pr_ {\mathcal {M}} [ \mathcal {M} (D) \in S | B = 1, \bar {A} ] \Pr [ B = 1 | \bar {A} ] I [ \bar {A} ] \\ + I [ \bot \in S ] \Pr [ B = 0 ] \\ \leq \left(e ^ {\varepsilon} \operatorname * {P r} _ {\mathcal {M}} [ \mathcal {M} (D ^ {\prime}) \in S | B = 1, A ] + \frac {\delta^ {\mathrm {e s t}}}{\tau}\right) \operatorname * {P r} [ B = 1 | A ] I [ A ] \\ + \Pr_ {\mathcal {M}} [ \mathcal {M} (D) \in S | B = 1, \bar {A} ] \Pr [ B = 1 | \bar {A} ] I [ \bar {A} ] \\ + I [ \bot \in S ] \Pr [ B = 0 ] \\ \leq \left(e ^ {\varepsilon} \operatorname * {P r} _ {\mathcal {M}} [ \mathcal {M} (D ^ {\prime}) \in S | B = 1, A ] + \frac {\delta^ {\mathrm {e s t}}}{\tau}\right) p _ {\mathsf {T P}} I [ A ] + p _ {\mathsf {F P}} I [ \bar {A} ] + I [ \bot \in S ]   \Pr [ B = 0 ] \\ \leq e ^ {\varepsilon} \left(\operatorname * {P r} _ {\mathcal {M}} [ \mathcal {M} (D ^ {\prime}) \in S | B = 1, A ] p _ {\mathsf {T P}} I [ A ] + \operatorname * {P r} _ {\mathcal {M}} [ \mathcal {M} (D ^ {\prime}) \in S | B = 1, \bar {A} ] p _ {\mathsf {F P}} I [ \bar {A} ] + I [ \bot \in S ] \Pr [ B = 0 ]\right) \\ + \frac {\delta^ {\mathrm {e s t}}}{\tau} p _ {\mathsf {T P}} I [ A ] + p _ {\mathsf {F P}} I [ \bar {A} ] \\ \leq e ^ {\varepsilon} \operatorname * {P r} _ {\mathcal {M} ^ {\text {a u g}}} \left[ \mathcal {M} ^ {\text {a u g}} \left(D ^ {\prime}\right) \in S \right] + \max  \left(\frac {\delta^ {\text {e s t}} p _ {\text {T P}}}{\tau}, p _ {\text {F P}}\right) \\ \end{array}
$$
where in the first inequality, we use the definition of differential privacy. Therefore,  $\mathcal{M}^{\mathrm{aug}}$  is  $\left(\varepsilon ,\max \left(\frac{\delta^{\mathrm{est}}p_{\mathrm{FP}}}{\tau},p_{\mathrm{FP}}\right)\right)$  -DP. By assumption of  $p_{\mathrm{FP}}\leq \mathsf{FP}_{\mathrm{DPV}}(\varepsilon ,\delta^{\mathrm{est}};\tau)\leq \delta^{\mathrm{est}} / \tau$  , we reach the conclusion.
C Importance Sampling via Exponential Tilting
Notation Review. Recall that most of the recently proposed DP accountants are essentially different techniques for estimating the expectation
where each  $Y_{i}$  is the privacy loss random variable  $Y_{i} = \log \left(\frac{P_{i}(t)}{Q_{i}(t)}\right)$  for  $t\sim P_i$ , and  $(P_{i},Q_{i})$  is a pair of dominating distribution for individual mechanism  $\mathcal{M}_i$ . In the following text, we denote the product distribution  $\pmb {P}\coloneqq P_1\times \ldots \times P_k$  and  $\pmb {Q}\coloneqq Q_1\times \ldots \times Q_k$ . Recall from Lemma 5 that  $(\pmb {P},\pmb {Q})$  is a pair of dominating distributions for the composed mechanism  $\mathcal{M}$ . For notation simplicity, we denote a vector  $\pmb {t}\coloneqq (t^{(1)},\dots ,t^{(k)})$ . We slightly abuse the notation and write  $y(t;P,Q)\coloneqq \log \left(\frac{P(t)}{Q(t)}\right)$ . Note that  $y(t;\pmb {P},\pmb {Q}) = \sum_{i = 1}^{k}y(t^{(i)};P_{i},Q_{i})$ . When the context is clear, we omit the dominating pairs and simply write  $y(t)$ .
Dominating Distribution Pairs for Poisson Subsampled Gaussian Mechanisms. The dominating distribution pair for Poisson Subsampled Gaussian Mechanisms is a well-known result.
Lemma 16. For Poisson Subsampled Gaussian mechanism with sensitivity  $C$ , noise variance  $C^2 \sigma^2$  and subsampling rate  $q$ , one dominating pair  $(P, Q)$  is  $Q := \mathcal{N}(0, \sigma^2)$  and  $P := (1 - q) \mathcal{N}(0, \sigma^2) + q \mathcal{N}(1, \sigma^2)$ .
Proof. See Appendix B of [19].
That is,  $Q$  is just a 1-dimensional standard Gaussian distribution, and  $P$  is a convex combination between standard Gaussian and a Gaussian centered at 1.
Remark 17 (Dominating pair supported on higher dimensional space). The cost of our approach would not increase (in terms of the number of samples) even if the dominating pair is supported in a high dimensional space. For Monte Carlo estimate, we can see from Hoeffding's inequality that the expected error rate of estimation is independent of the dimension of the support set of dominating distribution pairs. This means the number of samples we need to ensure a certain confidence interval is independent of the dimension. However, we should also note that although the number of samples does not change, the sampling process itself might be more costly for higher dimensional spaces.
C.1 Importance Sampling for the Composition of Poisson Subsampled Gaussian Mechanisms
Importance Sampling (IS) is a classic method for rare event simulation. It samples from an alternative distribution instead of the distribution of the quantity of interest, and a weighting factor is then used for correcting the difference between the two distributions. Specifically, we can re-write the expression for  $\delta_Y(\varepsilon)$  as follows:
$$
\begin{array}{l} \delta_ {Y} (\varepsilon) = \mathbb {E} _ {Y} \left[ (1 - e ^ {\varepsilon - Y}) _ {+} \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P} ^ {\prime}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} \frac {\boldsymbol {P} (\boldsymbol {t})}{\boldsymbol {P} ^ {\prime} (\boldsymbol {t})} \right] \tag {2} \\ \end{array}
$$
where  $P'$  is the alternative distribution up to the user's choice. From Equation (2), one can construct an unbiased importance sampling estimator for  $\delta_Y(\varepsilon)$  by sampling from  $P'$ . In this section, we develop a  $P'$  for estimating  $\delta_Y(\varepsilon)$  when composing identically distributed Poisson subsampled Gaussian mechanisms, which is arguably the most important DP mechanism nowadays due to its application in differentially private stochastic gradient descent.
Exponential tilting is a common way to construct alternative sampling distribution for IS. The exponential tilting of a distribution  $P$  is defined as
$$
P _ {\theta} (t) := \frac {e ^ {\theta t}}{M _ {P} (\theta)} P (t)
$$
where  $M_P(\theta) \coloneqq \mathbb{E}_{t \sim P}[e^{\theta t}]$  is the moment generating function for  $P$ . Such a transformation is especially convenient for distributions from the exponential family. For example, for normal distribution  $\mathcal{N}(\mu, \sigma^2)$ , the tilted distribution is  $\mathcal{N}(\mu + \theta \sigma^2, \sigma^2)$ , which is easy to sample from.
Without the loss of generality, we consider Poisson Subsampled Gaussian mechanism with sensitivity 1, noise variance  $\sigma^2$ , and subsampling rate  $q$ . Recall from Lemma 16 that the dominating pair in this case is  $Q \coloneqq \mathcal{N}(0,\sigma^2)$  and  $P \coloneqq (1 - q)\mathcal{N}(0,\sigma^2) + q\mathcal{N}(1,\sigma^2)$ . For notation simplicity, we denote  $P_0 \coloneqq \mathcal{N}(1,\sigma^2)$ , and thus  $P = (1 - q)Q + qP_0$ . Since each individual mechanism is the same,  $P = P \times \ldots \times P$  and  $Q = Q \times \ldots \times Q$ . The exponential tilting of  $P$  with parameter  $\theta$  is  $P_{\theta} \coloneqq (1 - q)\mathcal{N}(\theta \sigma^2,\sigma^2) + q\mathcal{N}(1 + \theta \sigma^2,\sigma^2)$ . We propose the following importance sampling estimator for  $\delta_Y(\varepsilon)$  based on exponential tilting.
Definition 18 (Importance Sampling Estimator for Subsampled Gaussian Composition). Let the alternative distribution
$$
\boldsymbol {P} ^ {\prime} := \boldsymbol {P} _ {\theta} = (P, \ldots , \underbrace {P _ {\theta}} _ {i \text {t h d i m}}, \ldots , P), i \sim \operatorname {U n i f} ([ k ])
$$
with  $\theta = 1/2 + \sigma^2 \log \left( \frac{\exp(\varepsilon) - (1 - q)}{q} \right)$ . Given a random draw  $t \sim P_{\theta}$ , an unbiased sample for  $\delta_Y(\varepsilon)$  is  $(1 - e^{\varepsilon - y(t;P,Q)})_+ \left( \frac{1}{k} \sum_{i=1}^{k} \frac{P_{\theta}(t_i)}{P(t_i)} \right)^{-1}$ . We denote  $\widehat{\delta}_{IS,\theta}^m(\varepsilon)$  as the random variable of the corresponding importance sampling estimator with  $m$  samples.
We defer the formal justification of the choice of  $\theta$  to Appendix C.2. We first give the intuition for why we choose such an alternative distribution  $P_{\theta}$ .
Intuition for the alternative distribution  $P_{\theta}$ . It is well-known that the variance of the importance sampling estimator is minimized when the alternative distribution
$$
\boldsymbol {P} ^ {\prime} (\boldsymbol {t}) \propto \left(1 - e ^ {\varepsilon - y (\boldsymbol {t})}\right) _ {+} \boldsymbol {P} (\boldsymbol {t})
$$
The distribution of each privacy loss random variable  $y(t; P, Q)$ ,  $t \sim P$  is light-tailed, which means that for the rare event where  $y(\pmb{t}) = \sum_{i=1}^{k} y(t^{(i)}) > \varepsilon$ , it is most likely that there is only one outlier  $t^*$  among all  $\{t^{(i)}\}_{i=1}^{k}$  such that  $y(t^*)$  is large (which means that  $y(\pmb{t})$  is also large), and all the rest of  $y(t^{(i)})$ s are small. Hence, a reasonable alternative distribution can just tilt the distribution of a randomly picked  $t^{(i)}$ , and leave the rest of  $k-1$  distributions to stay the same. Moreover,  $\theta$  is selected to approximately minimize the variance of  $\widehat{\delta}_{\mathrm{IS},\theta}$  (detailed in Appendix C.2). An intuitive way to see it is that  $y(\theta) = \varepsilon$ , which significantly improves the probability where  $y(t) \geq \varepsilon$  while also accounting for the fact that  $P(t)$  decays exponentially fast as  $t$  increases.
We also empirically verify the advantage of the IS estimator over the SMC estimator. The orange curve in Figure 6 (a) shows the empirical estimate of  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$  which quantifies the variance of  $\widehat{\delta}_{\mathrm{IS},\theta}$ . Note that  $\theta = 0$  corresponds to the case of  $\widehat{\delta}_{\mathrm{MC}}$ . As we can see,  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$  drops quickly as  $\theta$  increases, and eventually converges. We can also see that the  $\theta$  selected by our heuristic in Definition 18 (marked as red  $*$ ) approximately corresponds to the lowest point of  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$ . This validates our theoretical justification for the selection of  $\theta$ .
Figure 6: We examine the properties of MC-based DP verifiers for Poisson Subsampled mechanism. We set  $q = 10^{-3}$ ,  $\sigma = 0.6$ ,  $\varepsilon = 1.5$ ,  $k = 100$ .  $\delta_{Y}(\varepsilon) \approx 7.7 \times 10^{-6}$  in this case. (a) Plot for the upper bound and empirical estimate of  $\mathbb{E}[\widehat{\delta}_{\mathrm{MC}}^2]$  and  $\mathbb{E}[\widehat{\delta}_{\mathrm{IS},\theta}^2]$ . The upper bounds are computed by Corollary 22 (for  $\mathbb{E}[\widehat{\delta}_{\mathrm{MC}}^2]$ ) and Theorem 23 (for  $\mathbb{E}[\widehat{\delta}_{\mathrm{IS},\theta}^2]$ ). Note that  $\theta = 0$  corresponds to  $\widehat{\delta}_{\mathrm{MC}}$ . The red star indicates the second moment for the value of  $\theta$  selected by our heuristic in Definition 18. The blue star indicates the  $\theta$  that minimizes the analytical bound. (b) Empirical estimate of the rejection probability  $\operatorname*{Pr}[\widehat{\delta}_{\mathrm{IS},\theta}^{m} > \delta^{\mathrm{est}} / \tau - \Delta]$  scaled with the number of samples  $m$ . We set  $\delta^{\mathrm{est}} = 0.8\delta_{Y}(\varepsilon)$ ,  $\tau = 10^{-5} / \delta^{\mathrm{est}}$ , and we set  $\Delta$  following the heuristic proposed in Section 4.4.
C.2 Justification of the Heuristic of Choosing Exponential Tilting Parameter  $\theta$
The variance of the IS estimator proposed in Definition 18 is given by
$$
\begin{array}{l} \mathbb {E} _ {\pmb {t} \sim \pmb {P} _ {\theta}} \left[ \left(1 - e ^ {\varepsilon - y (\pmb {t}; \pmb {P}, \pmb {Q})}\right) _ {+} ^ {2} \left(\frac {\pmb {P} (\pmb {t})}{\pmb {P} _ {\theta} (\pmb {t})}\right) ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {\boldsymbol {P} (\boldsymbol {t})}{\boldsymbol {P} _ {\theta} (\boldsymbol {t})}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{k} \sum_ {i = 1} ^ {k} \frac {P _ {\theta} (t _ {i})}{P (t _ {i})}\right) ^ {- 1} \right] \\ = k M _ {P} (\theta) \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right] \\ \end{array}
$$
Let  $S(\theta) \coloneqq kM_P(\theta)\mathbb{E}_{\pmb{t} \sim \pmb{P}}\left[\left(1 - e^{\varepsilon - y(\pmb{t};\pmb{P},\pmb{Q})}\right)_+^2\left(\frac{1}{\sum_{i=1}^k e^{\theta t_i}}\right)\right]$ . We aim to find  $\theta$  that minimizes  $S(\theta)$ .
$$
M _ {P} (\theta) = (1 - q) e ^ {\frac {1}{2} \sigma^ {2} \theta^ {2}} + q e ^ {\theta + \frac {1}{2} \sigma^ {2} \theta^ {2}} \tag {3}
$$