SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To simplify the notation, let  $b(\pmb{t}) \coloneqq \left(1 - e^{\varepsilon - y(\pmb{t})}\right)_+^2 \prod_{i=1}^k P_0(t_i)$ .
$$
\begin{array}{l} \frac {\partial}{\partial \theta} S (\theta) = \left[ (1 - q) e ^ {\frac {1}{2} \sigma^ {2} \theta^ {2}} \left(\sigma^ {2} \theta\right) + q e ^ {\theta + \frac {1}{2} \sigma^ {2} \theta^ {2}} \left(1 + \sigma^ {2} \theta\right) \right] \int \dots \int b (\boldsymbol {t}) \left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {- 1} d \boldsymbol {t} (4) \\ - \left[ (1 - q) e ^ {\frac {1}{2} \sigma^ {2} \theta^ {2}} + q e ^ {\theta + \frac {1}{2} \sigma^ {2} \theta^ {2}} \right] \int \dots \int b (\boldsymbol {t}) \frac {\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}} t _ {i}}{\left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {2}} d \boldsymbol {t} (5) \\ \end{array}
$$
By setting  $\frac{\partial}{\partial\theta} S(\theta) = 0$  and simplify the expression, we have
$$
\frac {(1 - q + q e ^ {\theta}) (\sigma^ {2} \theta) + q e ^ {\theta}}{1 - q + q e ^ {\theta}} = \frac {\int \dots \int b (\boldsymbol {t}) \frac {\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}} t _ {i}}{\left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {2}} d \boldsymbol {t}}{\int \dots \int b (\boldsymbol {t}) \left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {- 1} d \boldsymbol {t}} \tag {6}
$$
As we mentioned earlier,  $b(t) > 0$  only when  $y(t) = \sum_{i=1}^{k} y(t^{(i)}) > \varepsilon$ , and for such an event it is most likely that there is only one outlier  $t^*$  among all  $\{t^{(i)}\}_{i=1}^{k}$  such that  $y(t^*) \approx \varepsilon$ , and all the rest of  $y(t^{(i)}) \approx 0$ . Therefore, a simple but surprisingly effective approximation for the RHS of (6) is
$$
\frac {\int \dots \int b (\boldsymbol {t}) \frac {\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}} t _ {i}}{\left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {2}} d \boldsymbol {t}}{\int \dots \int b (\boldsymbol {t}) \left(\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}\right) ^ {- 1} d \boldsymbol {t}} \approx \frac {b (t) e ^ {- \theta t} t}{b (t) e ^ {- \theta t}} = t \tag {7}
$$
for  $t$  s.t.  $y(t) = \varepsilon$ . This leads to an approximate solution
$$
\theta^ {*} = \frac {1}{2 \sigma^ {2}} + \log \left(\left(\exp (\varepsilon) - (1 - q)\right) / q\right) \tag {8}
$$
D Sample Complexity for Achieving Target False Positive Rate
To derive the sample complexity for achieving a DP verifier with a target false positive rate, we use Bennett's inequality.
Lemma 19 (Bennett's inequality). Let  $X_{1}, \ldots, X_{n}$  be independent real-valued random variables with finite variance such that  $X_{i} \leq b$  for some  $b > 0$  almost surely for all  $1 \leq i \leq n$ . Let  $\nu \geq \sum_{i=1}^{n} \mathbb{E}[X_{i}^{2}]$ . For any  $t > 0$ , we have
$$
\Pr \left[ \sum_ {i = 1} ^ {n} X _ {i} - \mathbb {E} [ X _ {i} ] \geq t \right] \leq \exp \left(- \frac {\nu}{b ^ {2}} h \left(\frac {b t}{\nu}\right)\right) \tag {9}
$$
where  $h(x) = (1 + x)\log (1 + x) - x$  for  $x > 0$ .
Proof. For any  $\mathcal{M}$  s.t.  $\delta^{\mathrm{est}} < \tau \delta_Y(\varepsilon)$ , we have
$$
\begin{array}{l} \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {m} (\varepsilon ; Y) <   \delta^ {\text {e s t}} / \tau - \Delta \right] \leq \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {m} (\varepsilon ; Y) <   \delta_ {Y} (\varepsilon) - \Delta \right] \\ = \Pr \left[ \frac {1}{m} \sum_ {i = 1} ^ {m} \left(\widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - \delta_ {Y} (\varepsilon)\right) <   - \Delta \right] \\ = \Pr \left[ \sum_ {i = 1} ^ {m} \left(\delta_ {Y} (\varepsilon) - \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)}\right) > m \Delta \right] \tag {10} \\ \end{array}
$$
Since  $\widehat{\pmb{\delta}}_{\mathsf{MC}}\in [-1,0]$ , the condition in Bennett's inequality is satisfied with  $b\to 0^{+}$ . Hence, (10) can be upper bounded by
$$
\begin{array}{l} (1 0) \leq \lim  _ {b \rightarrow 0 ^ {+}} \exp \left(- \frac {m \nu}{b ^ {2}} h \left(\frac {b \Delta}{\nu}\right)\right) \\ = \exp \left(- \frac {m \Delta^ {2}}{2 \nu}\right) \\ \end{array}
$$
By setting  $\exp \left(-\frac{m\Delta^2}{2\nu}\right) \leq \delta^{\mathrm{est}} / \tau$ , we have
$$
m \geq \frac {2 \nu}{\Delta^ {2}} \log (\tau / \delta^ {\mathrm {e s t}})
$$
E Proofs for Moment Bound
E.1 Overview
As suggested by Theorem 13, a good upper bound for  $\mathbb{E}\left[\left(\widehat{\pmb{\delta}}_{\mathrm{MC}}\right)^2\right]$  (or  $\mathbb{E}\left[\left(\widehat{\pmb{\delta}}_{\mathrm{IS}}\right)^2\right]$ ) is important for the computational efficiency of MC-based DPV.
We upper bound the higher moment of  $\widehat{\delta}_{\mathbb{MC}}$  through the RDP guarantee for the composed mechanism  $\mathcal{M}$ . This is a natural idea since converting RDP to upper bounds for  $\delta_Y(\varepsilon)$  - the first moment of  $\widehat{\delta}_{\mathbb{MC}}$  - is a well-studied problem [30, 9, 3]. Recall that the RDP guarantee for  $\mathcal{M}$  is equivalent to a bound for  $M_{Y}(\lambda)\coloneqq \mathbb{E}[e^{\lambda Y}]$  for  $\mathcal{M}$ 's privacy loss random variable  $Y$  for any  $\lambda \geq 0$ .
Lemma 20 (RDP-MGF bound conversion [30]). If a mechanism  $\mathcal{M}$  is  $(\alpha, \varepsilon_{\mathrm{R}}(\alpha))$ -RDP, then  $M_{Y}(\lambda) \leq \exp (\lambda \varepsilon_{\mathrm{R}}(\lambda +1))$ .
We convert an upper bound for  $M_Y(\cdot)$  into the following guarantee for the higher moment of  $\widehat{\pmb{\delta}}_{\mathbb{MC}} = (1 - e^{\varepsilon -Y})_+$ .
Theorem 21. For any  $u \geq 1$ , we have
$$
\mathbb {E} \left[ \left(\widehat {\boldsymbol {\delta}} _ {\mathcal {M C}}\right) ^ {u} \right] = \mathbb {E} \left[ (1 - e ^ {\varepsilon - Y}) _ {+} ^ {u} \right] \leq \min  _ {\lambda \geq 0} M _ {Y} (\lambda) e ^ {- \varepsilon \lambda} \frac {u ^ {u} \lambda^ {\lambda}}{(u + \lambda) ^ {u + \lambda}}
$$
The proof is deferred to Appendix E.2. The basic idea is to find the smallest constant  $c$  such that  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^u ]\leq cM_Y(\lambda)$ . By setting  $u = 1$ , our result recovers the RDP-DP conversion from [9]. By setting  $u = 2$ , we obtain the desired bound for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2 ]$ .
Corollary 22.  $\mathbb{E}[(\widehat{\pmb{\delta}}_{MC})^2 ]\leq \min_{\lambda \geq 0}M_Y(\lambda)e^{-\varepsilon \lambda}\frac{4\lambda^\lambda}{(\lambda + 2)^{\lambda + 2}}.$
Corollary 22 applies to any mechanisms where the RDP guarantee is available, which covers a wide range of commonly used mechanisms such as (Subsampled) Gaussian or Laplace mechanism. We also note that one may be able to further tighten the above bound similar to the optimal RDP-DP conversion in [3]. We leave this as an interesting future work.
Next, we derive the upper bound for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2 ]$  for Poisson Subsampled Gaussian mechanism.
Theorem 23. For any positive integer  $\lambda$ , and for any  $a, b \geq 1$  s.t.  $1/a + 1/b = 1$ , we have  $\mathbb{E}[(\widehat{\pmb{\delta}}_{IS,\theta})^2] \leq kM_P(\theta)\left(\mathbb{E}[\widehat{\pmb{\delta}}_{MC}^{2a}]\right)^{1/a} \cdot \left(b\theta e^{-\lambda\varepsilon} \int [r(\lambda,x)]^k e^{-b\theta x} dx\right)^{1/b}$  where  $r(\lambda,x)$  is an upper bound for  $\operatorname{Pr}_{t \sim P}[\max_i t_i \leq x, y(t) \geq \varepsilon]$  detailed in Appendix E.3.
The proof is based on applying Hölder's inequality to the expression of  $\mathbb{E}[(\widehat{\pmb{\delta}}_{\mathrm{IS},\theta})^2]$ , and then bound the part where  $\theta$  is involved:  $\mathbb{E}_{t\sim P}\left[\left(\frac{1}{k}\sum_{i = 1}^{k}\frac{P_{\theta}(t_i)}{P(t_i)}\right)^{-1}\right]$ . We can bound  $\mathbb{E}[\widehat{\pmb{\delta}}_{\mathrm{MC}}^{2a}]$  through Theorem 21.
Figure 6 (a) shows the analytical bound from Corollary 22 and Theorem 23 compared with empirically estimated  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2]$  and  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$ . As we can see, the analytical bound for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$  for relatively large  $\theta$  is much smaller than the bound for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{MC}})^2]$  (i.e.,  $\theta = 0$  in the plot). Moreover, we find that the  $\theta$  which minimizes the analytical bound (the blue '*) is close to the  $\theta$  selected by our heuristic (the red ''). For computational efficiency, one may prefer to use  $\theta$  that minimizes the analytical bound. However, the heuristically selected  $\theta$  is still useful when one simply wants to estimate  $\delta_Y(\varepsilon)$  and does not require the formal, analytical guarantee for the false positive rate. We see such a scenario when we introduce the MC accountant in Section 5. We also note that such a discrepancy (and the gap between the analytical bound and empirical estimate) is due to the use of Hölder's inequality in bounding  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$ . Further tightening the bound for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2]$  is important for future work.
E.2 Moment Bound for Simple Monte Carlo Estimator
$$
\mathbb {E} [ (\widehat {\boldsymbol {\delta}} _ {\mathcal {M C}}) ^ {u} ] = \mathbb {E} \left[ (1 - e ^ {\varepsilon - Y}) _ {+} ^ {u} \right] \leq \min  _ {\lambda \geq 0} M _ {Y} (\lambda) e ^ {- \varepsilon \lambda} \frac {u ^ {u} \lambda^ {\lambda}}{(u + \lambda) ^ {u + \lambda}}
$$
$$
\begin{array}{l} \mathbb {E} [ (1 - e ^ {\varepsilon - Y}) _ {+} ^ {u} ] = \int (1 - e ^ {\varepsilon - x}) _ {+} ^ {u} P (x) \mathrm {d} x \\ = M _ {Y} (\lambda) \int (1 - e ^ {\varepsilon - x}) _ {+} ^ {u} e ^ {- \lambda x} \frac {P (x) e ^ {\lambda x}}{\mathbb {E} [ e ^ {\lambda Y} ]} \mathrm {d} x \\ = M _ {Y} (\lambda) \mathbb {E} _ {x \sim P _ {\theta}} [ (1 - e ^ {\varepsilon - x}) _ {+} ^ {u} e ^ {- \lambda x} ] \\ = M _ {Y} (\lambda) e ^ {- \lambda \varepsilon} \mathbb {E} _ {x \sim P _ {\theta}} [ (1 - e ^ {\varepsilon - x}) _ {+} ^ {u} e ^ {(\varepsilon - x) \lambda} ] \\ \end{array}
$$
where  $P_{\lambda}(x) \coloneqq \frac{P(x)e^{\lambda x}}{\mathbb{E}[e^{\lambda Y}]}$  is the exponential tilting of  $P$ . Define  $f(x, \lambda) \coloneqq (1 - e^{-x})_{+}^{u}e^{-x\lambda}$ . When  $x \leq 0$ ,  $f(x, \lambda) = 0$ . When  $x > 0$ , the derivative of  $f$  with respect to  $x$  is
$$
\frac {\partial f (x , \lambda)}{\partial x} = e ^ {- x \lambda} (1 - e ^ {- x}) ^ {u - 1} [ e ^ {- x} (u + \lambda) - \lambda ]
$$
It is easy to see that the maximum of  $f(x, \lambda)$  is achieved at  $x^{*} = \log \left(\frac{u + \lambda}{\lambda}\right)$ , and we have
$$
\begin{array}{l} \max  _ {x} f (x, \lambda) = \left(\frac {u}{u + \lambda}\right) ^ {u} \left(\frac {\lambda}{u + \lambda}\right) ^ {\lambda} \\ = \frac {u ^ {u} \lambda^ {\lambda}}{(u + \lambda) ^ {u + \lambda}} \\ \end{array}
$$
Overall, we have
$$
\mathbb {E} [ (1 - e ^ {\varepsilon - Y}) _ {+} ^ {u} ] \leq M _ {Y} (\lambda) e ^ {- \varepsilon \lambda} \frac {u ^ {u} \lambda^ {\lambda}}{(u + \lambda) ^ {u + \lambda}}
$$
E.3 Moment Bound for Importance Sampling Estimator
In this section, we first prove two possible upper bounds for  $\mathbb{E}[(\widehat{\delta}_{\mathrm{IS},\theta})^2 ]$  in Theorem 24 and Theorem 26. We then combine these two bounds in Theorem 23 via Holder's inequality.
Theorem 24. For any  $\theta \geq 1 / \sigma^2$ , we have
$$
\mathbb {E} \left[ \left(\widehat {\boldsymbol {\delta}} _ {I S, \theta}\right) ^ {2} \right] \leq M _ {P} (\theta) \left[ \frac {1}{k} \left(\frac {\varepsilon}{q} + k\right) \right] ^ {- \theta \sigma^ {2}} e ^ {- \theta / 2} \mathbb {E} \left[ \left(\widehat {\boldsymbol {\delta}} _ {N C}\right) ^ {2} \right]
$$
$$
\begin{array}{l} \mathbb {E} _ {\pmb {t} \sim \pmb {P} _ {\theta}} \left[ \left(1 - e ^ {\varepsilon - y (\pmb {t}; \pmb {P}, \pmb {Q})}\right) _ {+} ^ {2} \left(\frac {\pmb {P} (\pmb {t})}{\pmb {P} _ {\theta} (\pmb {t})}\right) ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {\boldsymbol {P} (\boldsymbol {t})}{\boldsymbol {P} _ {\theta} (\boldsymbol {t})}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{k} \sum_ {i = 1} ^ {k} \frac {P _ {\theta} (t _ {i})}{P (t _ {i})}\right) ^ {- 1} \right] \\ = M _ {P} (\theta) \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {k}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right] \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {\pmb {t} \sim \pmb {P}} \left[ \left(1 - e ^ {\varepsilon - y (\pmb {t}; \pmb {P}, \pmb {Q})}\right) _ {+} ^ {2} \left(\frac {k}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {k}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \\ \end{array}
$$
Lemma 25. When  $y(t; \mathbf{P}, \mathbf{Q}) = \sum_{i=1}^{k} \log \left(1 - q + q e^{(2t_i - 1) / (2\sigma^2)}\right) \geq \varepsilon$  and  $\theta \sigma^2 \geq 1$ , we have
$$
\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}} \geq k \left[ \frac {1}{k} \left(\frac {\varepsilon}{k} + k\right) e ^ {1 / (2 \sigma^ {2})} \right] ^ {\theta \sigma^ {2}}
$$
Proof. Since  $\log (1 + x) \leq x$ , we have
$$
\begin{array}{l} \varepsilon \leq \sum_ {i = 1} ^ {k} \log \left(1 - q + q e ^ {(2 t _ {i} - 1) / (2 \sigma^ {2})}\right) \\ \leq \sum_ {i = 1} ^ {k} q \left(e ^ {(2 t _ {i} - 1) / (2 \sigma^ {2})} - 1\right) \\ \end{array}
$$
$$
\sum_ {i = 1} ^ {k} e ^ {t _ {i} / \sigma^ {2}} \geq \left(\frac {\varepsilon}{q} + k\right) e ^ {1 / (2 \sigma^ {2})}
$$
$$
\begin{array}{l} \sum_ {i = 1} ^ {k} e ^ {t _ {i} \theta} = \sum_ {i = 1} ^ {k} \left(e ^ {t _ {i} / \sigma^ {2}}\right) ^ {\theta \sigma^ {2}} = k \left[ \frac {1}{k} \sum_ {i = 1} ^ {k} \left(e ^ {t _ {i} / \sigma^ {2}}\right) ^ {\theta \sigma^ {2}} \right] \\ \geq k \left[ \frac {1}{k} \sum_ {i = 1} ^ {k} \left(e ^ {t _ {i} / \sigma^ {2}}\right) \right] ^ {\theta \sigma^ {2}} \\ \geq k \left[ \frac {1}{k} \left(\frac {\varepsilon}{q} + k\right) e ^ {1 / (2 \sigma^ {2})} \right] ^ {\theta \sigma^ {2}} \\ = k \left[ \frac {1}{k} \left(\frac {\varepsilon}{q} + k\right) \right] ^ {\theta \sigma^ {2}} e ^ {\theta / 2} \\ \end{array}
$$
where the first inequality is due to Jensen's inequality.
By Lemma 25, we have
$$
\begin{array}{l} (1 1) \leq M _ {P} (\theta) \frac {1}{\left[ \frac {1}{k} \left(\frac {\varepsilon}{q} + k\right) \right] ^ {\theta \sigma^ {2}} e ^ {\theta / 2}} \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \right] \\ = M _ {P} (\theta) \left[ \frac {1}{k} \left(\frac {\varepsilon}{q} + k\right) \right] ^ {- \theta \sigma^ {2}} e ^ {- \theta / 2} \mathbb {E} [ (\widehat {\boldsymbol {\delta}} _ {\mathbb {M C}}) ^ {2} ] \\ \end{array}
$$
Theorem 26. For any positive integer  $\lambda$ , we have
$$
\mathbb {E} \left[ \left(\widehat {\boldsymbol {\delta}} _ {I S, \theta}\right) ^ {2} \right] \leq k M _ {P} (\theta) \theta e ^ {- \lambda \varepsilon} \int [ r (\lambda , x) ] ^ {k} e ^ {- \theta x} d x \tag {12}
$$
where  $r(\lambda, x)$  is an upper bound for the MGF of privacy loss random variable of truncated Gaussian mixture  $P|_{\leq x}$ .
Proof. Similar to Theorem 24, the goal is to bound
$$
k M _ {P} (\boldsymbol {\theta}) \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right]
$$
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \\ \leq \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}} I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \\ \leq \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \frac {1}{e ^ {\theta t _ {\max }}} I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \tag {13} \\ \end{array}
$$
where  $t_{\mathrm{max}} \coloneqq \max_i t_i$
Further note that
$$
\begin{array}{l} (1 3) = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ e ^ {- \theta t _ {\max }} I [ y (\boldsymbol {t}) \geq \varepsilon ] \right] (13) \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ e ^ {- \theta t _ {\max }} | y (\boldsymbol {t}) \geq \varepsilon \right] \Pr_ {\boldsymbol {t} \sim \boldsymbol {P}} [ y (\boldsymbol {t}) \geq \varepsilon ] \\ = \left(\int e ^ {- \theta x} \mathrm {d} _ {\boldsymbol {t} \sim \boldsymbol {P}} [ t _ {\max } \leq x ]\right) \underset {\boldsymbol {t} \sim \boldsymbol {P}} {\operatorname * {P r}} [ y (\boldsymbol {t}) \geq \varepsilon ] \\ = - \left(\int_ {\boldsymbol {t} \sim \boldsymbol {P}} \Pr_ {\boldsymbol {t} \sim \boldsymbol {P}} [ t _ {\max } \leq x | y (\boldsymbol {t}) \geq \varepsilon ] \mathrm {d} e ^ {- \theta x}\right) \Pr_ {\boldsymbol {t} \sim \boldsymbol {P}} [ y (\boldsymbol {t}) \geq \varepsilon ] (14) \\ = \theta \int_ {\boldsymbol {t} \sim \boldsymbol {P}} \Pr_ {\left[ t _ {\max } \leq x, y (\boldsymbol {t}) \geq \varepsilon \right]} e ^ {- \theta x} d x (15) \\ \end{array}
$$
where (14) is obtained through integration by parts.
Now, as we can see from (15), the question reduces to bound  $\operatorname{Pr}_{t \sim P}\left[t_{\max} \leq x, y(t) \geq \varepsilon\right]$  for any  $x \in \mathbb{R}$ . It might be easier to write
$$
\operatorname * {P r} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ t _ {\max } \leq x, y (\boldsymbol {t}) \geq \varepsilon \right] = \operatorname * {P r} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ y (\boldsymbol {t}) \geq \varepsilon | t _ {\max } \leq x \right] \operatorname * {P r} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ t _ {\max } \leq x \right]
$$
and we know that
$$
\begin{array}{l} \Pr_ {t \sim P} [ t _ {\max } \leq x ] = \left(\Pr_ {t \sim P} [ t \leq x ]\right) ^ {k} (16) \\ = \left((1 - q) \Phi (x; 0, \sigma^ {2}) + q \Phi (x; 1, \sigma^ {2})\right) ^ {k} (17) \\ \end{array}
$$
as all  $t_i$ s are i.i.d. random samples from  $P$ , where  $\Phi(\cdot; \mu, \sigma^2)$  is the CDF of Gaussian distribution with mean  $\mu$  and variance  $\sigma^2$ .
It remains to bound the conditional probability  $\operatorname{Pr}_{\pmb{t} \sim \pmb{P}}[y(\pmb{t}) \geq \varepsilon | t_{\max} \leq x]$ , it may be easier to see it in this way:
$$
\begin{array}{l} \operatorname * {P r} _ {\boldsymbol {t} \sim \boldsymbol {P}} [ y (\boldsymbol {t}) \geq \varepsilon | t _ {\max } \leq x ] \\ = \Pr_ {\boldsymbol {t} \sim \boldsymbol {P}} [ y (\boldsymbol {t}) \geq \varepsilon | t _ {1} \leq x, \dots , t _ {k} \leq x ] \\ = \Pr_ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \sum_ {i = 1} ^ {k} y (t _ {i}) \geq \varepsilon | y (t _ {1}) \leq y (x), \dots , y (t _ {k}) \leq y (x) \right] \\ \leq \frac {\mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ e ^ {\lambda \sum_ {i = 1} ^ {k} y (t _ {i})} \geq e ^ {\lambda \varepsilon} | y (t _ {1}) \leq y (x) , \dots , y (t _ {k}) \leq y (x) \right]}{e ^ {\lambda \varepsilon}} \\ \end{array}
$$
where the last step is due to Chernoff bound which holds for any  $\lambda > 0$ . Now we only need to bound the moment generating function for  $y(t) = \log (1 - q + qe^{\frac{2t - 1}{2\sigma^2}})$ ,  $t \sim P|_{\leq x}$ , where  $P|_{\leq x}$  is
the truncated distribution of  $P$ . We note that this is equivalent to bounding the Rényi divergence for truncated Gaussian mixture distribution.
Recall that  $P = (1 - q)\mathcal{N}(0,\sigma^2) + q\mathcal{N}(1,\sigma^2)$ . For any  $\lambda$  that is a positive integer, we have
$$
\begin{array}{l} \mathbb {E} _ {t \sim P} \left[ e ^ {\lambda y (t)} I [ t \leq x ] \right] \\ = \frac {1}{\sqrt {2 \pi} \sigma} \left[ (1 - q) \int_ {- \infty} ^ {x} \left(1 - q + q e ^ {\frac {2 t - 1}{2 \sigma^ {2}}}\right) ^ {\lambda} e ^ {- \frac {t ^ {2}}{2 \sigma^ {2}}} d t + q \int_ {- \infty} ^ {x} \left(1 - q + q e ^ {\frac {2 t - 1}{2 \sigma^ {2}}}\right) ^ {\lambda} e ^ {- \frac {(t - 1) ^ {2}}{2 \sigma^ {2}}} d t \right] \\ = \frac {1}{2} \left(q e ^ {- \frac {1}{2 \sigma^ {2}}}\right) ^ {\lambda} \sum_ {i = 0} ^ {\lambda} \binom {\lambda} {i} \tilde {q} ^ {i} \left[ (1 - q) e ^ {\frac {(\lambda - i) ^ {i}}{2 \sigma^ {2}}} \left(\operatorname {e r f} \left(\frac {x - (\lambda - i)}{\sqrt {2} \sigma}\right) + 1\right) + q e ^ {\frac {(\lambda - i + 1) ^ {i} - 1}{2 \sigma^ {2}}} \left(\operatorname {e r f} \left(\frac {x - (\lambda - i + 1)}{\sqrt {2} \sigma}\right) + 1\right) \right] \tag {18} \\ \end{array}
$$
where  $\tilde{q} \coloneqq \frac{(1 - q)\exp(1 / (2\sigma^2))}{q}$ . Note that the above expression can be efficiently computed. Denote the above results as  $r(\lambda, x) \coloneqq (18)$ . Hence
$$
\mathbb {E} _ {t \sim P} \left[ e ^ {\lambda y (t)} | t \leq x \right] = \frac {r (\lambda , x)}{\Pr_ {t \sim P} [ t \leq x ]} \tag {19}
$$
Now we have
$$
\begin{array}{l} \operatorname * {P r} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ y (\boldsymbol {t}) \geq \varepsilon | t _ {\max } \leq x \right] \leq \frac {\mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ e ^ {\lambda \sum_ {i = 1} ^ {k} y (t _ {i})} \geq e ^ {\lambda \varepsilon} | t _ {\max } \leq x \right]}{e ^ {\lambda \varepsilon}} \\ = \frac {\left[ r (\lambda , x) \right] ^ {k}}{e ^ {\lambda \varepsilon} \left(\operatorname * {P r} _ {t \sim P} [ t \leq x ]\right) ^ {k}} \\ \end{array}
$$
Plugging this bound into (15), we have
$$
\begin{array}{l} (1 3) = \theta \int_ {\boldsymbol {t} \sim \boldsymbol {P}} \Pr_ {t \leq x, y (\boldsymbol {t}) \geq \varepsilon} e ^ {- \theta x} d x \\ \leq \theta e ^ {- \lambda \varepsilon} \int [ r (\lambda , x) ] ^ {k} e ^ {- \theta x} \mathrm {d} x \\ \end{array}
$$
which leads to the final conclusion.
Remark 27. In practice, we can further improve the bound by moving the minimum operation inside the integral:
$$
\mathbb {E} \left[ \left(\widehat {\boldsymbol {\delta}} _ {I S, \theta}\right) ^ {2} \right] \leq k M _ {P} (\theta) \theta e ^ {- \lambda \varepsilon} \int \left[ \min  _ {\lambda} r (\lambda , x) \right] ^ {k} e ^ {- \theta x} d x \tag {20}
$$
Of course, this bound will be less efficient to compute.
Theorem 23 (Generalizing Theorem 24 and Theorem 26 via Holder's inequality). For any positive integer  $\lambda$ , and for any  $a, b \geq 1$  s.t.  $1/a + 1/b = 1$ , we have  $\mathbb{E}[(\widehat{\pmb{\delta}}_{IS,\theta})^2] \leq kM_P(\theta)\left(\mathbb{E}[\widehat{\pmb{\delta}}_{MC}^{2a}]\right)^{1/a}$  ( $b\theta e^{-\lambda\varepsilon} \int [r(\lambda,x)]^k e^{-b\theta x} dx$ )<sup>1/b</sup> where  $r(\lambda,x)$  is an upper bound for the MGF of privacy loss random variable of truncated Gaussian mixture  $P|_{\leq x}$  defined in (18).
Proof. Note that Theorem 24 and Theorem 26 can both be viewed as two special cases of Hölder's inequality: for any  $a, b \geq 1$  s.t.  $\frac{1}{a} + \frac{1}{b} = 1$ , we have