SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\pmb {t} \sim \pmb {P}} \left[ \left(1 - e ^ {\varepsilon - y (\pmb {t}; \pmb {P}, \pmb {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2} \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \\ \leq \mathbb {E} _ {\pmb {t} \sim \pmb {P}} \left[ \left(1 - e ^ {\varepsilon - y (\pmb {t}; \pmb {P}, \pmb {Q})}\right) _ {+} ^ {2 a} \right] ^ {1 / a} \mathbb {E} _ {\pmb {t} \sim \pmb {P}} \left[ \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) ^ {b} I [ y (\pmb {t}; \pmb {P}, \pmb {Q}) \geq \varepsilon ] \right] ^ {1 / b} \\ \end{array}
$$
Theorem 24 corresponds to the case where  $a = 1$ , and Theorem 26 corresponds to the case where  $b = 1$ . We can actually tune the parameters  $a$  and  $b$  to see if we can obtain any better bounds, as we have
$$
\mathbb {E} _ {\boldsymbol {t} \sim P} \left[ \left(1 - e ^ {\varepsilon - y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q})}\right) _ {+} ^ {2 a} \right] \leq \mathbb {E} \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {2 a} \right] \tag {21}
$$
$$
\mathbb {E} _ {\boldsymbol {t} \sim \boldsymbol {P}} \left[ \left(\frac {1}{\sum_ {i = 1} ^ {k} e ^ {\theta t _ {i}}}\right) ^ {b} I [ y (\boldsymbol {t}; \boldsymbol {P}, \boldsymbol {Q}) \geq \varepsilon ] \right] \leq b \theta e ^ {- \lambda \varepsilon} \int [ r (\lambda , x) ] ^ {k} e ^ {- b \theta x} d x \tag {22}
$$
Simply combining the above inequalities leads to the final conclusion.
F Proofs for Utility
F.1 Overview
While one may be able to bound the false negative rate through similar techniques that we bound the false positive rate, i.e., applying the concentration inequalities, the guarantee may be loose. As a formal, strict guarantee for  $\mathsf{FN}_{\mathsf{DPV}}$  is not required, we provide a convenient heuristic of picking an appropriate  $\Delta$  such that  $\mathsf{FN}_{\mathsf{DPV}}$  is approximately smaller than  $\mathsf{FP}_{\mathsf{DPV}}$ .
For any mechanism  $\mathcal{M}$  such that  $\delta^{\mathrm{est}} > \rho \delta_Y(\varepsilon)$ , we have
$$
\begin{array}{l} \Pr_ {\mathsf {D P V}} \left[ \mathsf {D P V} (\mathcal {M}, \varepsilon , \delta^ {\text {e s t}}) = \mathsf {F a l s e} \right] \\ = \Pr \left[ \hat {\delta} _ {\mathrm {M C}} ^ {m} > \delta^ {\text {e s t}} / \tau - \Delta \right] \\ = \Pr \left[ \widehat {\delta} _ {\mathrm {M C}} ^ {m} - \delta_ {Y} (\varepsilon) > \delta^ {\text {e s t}} / \tau - \delta_ {Y} (\varepsilon) - \Delta \right] \\ \leq \Pr \left[ \widehat {\delta} _ {\mathbb {M C}} ^ {m} - \delta_ {Y} (\varepsilon) > \left((1 / \tau - 1 / \rho) \delta^ {\mathrm {e s t}} - \Delta\right) \right] \\ \end{array}
$$
and in the meantime, if  $\delta^{\mathrm{est}} < \tau \delta_Y(\varepsilon)$ , we have
$$
\Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} <   \delta^ {\mathrm {e s t}} / \tau - \Delta \right] \leq \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta_ {Y} (\varepsilon) <   - \Delta \right]
$$
Our main idea is to find  $\Delta$  such that
$$
\begin{array}{l} \Pr \left[ \widehat {\delta} _ {\mathbb {M C}} ^ {m} - \delta_ {Y} (\varepsilon) > \left((1 / \tau - 1 / \rho) \delta^ {\text {e s t}} - \Delta\right) \right] \\ \lesssim \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {m} - \delta_ {Y} (\varepsilon) <   - \Delta \right] \\ \end{array}
$$
In this way, we know that  $\mathsf{FN}_{\mathsf{MC}}(\varepsilon ,\delta^{\mathrm{est}};\rho)$  is upper bounded by  $\Theta (\delta^{\mathrm{est}} / \tau)$  for the same amount of samples we discussed in Section 4.3.
Observe that  $\widehat{\delta}_{\mathrm{MC}}$  (or  $\widehat{\delta}_{\mathrm{IS},\theta}$  with a not too large  $\theta$ ) is typically a highly asymmetric distribution with a significant probability of being zero, and the probability density decreases monotonically for higher values. Under such conditions, we prove the following results:
Theorem 28 (Informal). When  $m \geq \frac{2\nu}{\Delta^2} \log(\tau / \delta^{\mathrm{est}})$ , we have
$$
\Pr \left[ \widehat {\boldsymbol {\delta}} _ {N C} ^ {m} - \delta_ {Y} (\varepsilon) <   - \Delta \right] \gtrsim \Pr \left[ \widehat {\boldsymbol {\delta}} _ {N C} ^ {m} - \delta_ {Y} (\varepsilon) > \frac {3}{2} \Delta \right]
$$
The proof is deferred to Appendix F.2. Therefore, by setting  $\Delta = 0.4(1 / \tau -1 / \rho)\delta^{\mathrm{est}}$ , one can ensure that  $\mathsf{FN}_{\mathsf{MC}}(\varepsilon ,\delta^{\mathrm{est}};\rho)$  is also (approximately) upper bounded by  $\Theta (\delta^{\mathrm{est}} / \tau)$ . We empirically verify the effectiveness of such a heuristic by estimating the actual false negative rate. As we can see from Figure 6 (b), the dashed curve is much higher than the two solid curves, which means that the false negative rate is a very conservative bound.
Remark 29 (For the halting case). In this section, we develop techniques for ensuring the false negative rate (i.e., the rejection probability) is around  $O(\delta)$  when the proposed privacy parameter  $\delta^{\mathrm{est}}$  is close to the true  $\delta$ . In the experiment, we use the state-of-the-art FFT accountant to produce  $\delta^{\mathrm{est}}$ , which is very accurate as we can see from Figure 1. Hence, the rejection probability in the experiment is around  $O(\delta)$ , which means the probability of rejection is close to the probability of catastrophic failure for privacy.
If one is still concerned that the rejection probability is too large, we can further reduce the probability as follows: we run two instances of EVR paradigm simultaneously; if both of the instances are passed, we randomly pick one and release the output. If either one of them is passed, we release the passed instance. It only fails when both of the instances fail. By running two instances of the EVR paradigm in parallel, the false positive rate (i.e., the final  $\delta$ ) will be only doubled, but the probability of rejection will be squared.
We can also introduce a variant of our EVR paradigm that better deals with the failure case: whenever we face the "rejection", we run a different mechanism  $\mathcal{M}'$  that is guaranteed to be  $(\varepsilon, \delta^{\mathrm{est}})$ -DP
(e.g., by adjusting the subsampling rate and/or noise multiplier in DP-SGD). Moreover, we use FFT accountant to obtain a strict privacy guarantee upper bound  $(\varepsilon, \delta^{\mathrm{upp}})$  for the original mechanism  $\mathcal{M}$ , where  $\delta^{\mathrm{est}} < \delta^{\mathrm{upp}}$ . We use  $p_{\mathrm{FN}}$  and  $p_{\mathrm{FP}}$  to denote the false negative and false positive rate of the privacy verifier used in EVR paradigm. If the original mechanism  $\mathcal{M}$  is indeed  $(\varepsilon, \delta^{\mathrm{est}})$ -DP, then for any subset  $S$ , for this augmented EVR paradigm  $\mathcal{M}^{\mathrm{aug}}$  we have
$$
\begin{array}{l} \Pr \left[ \mathcal {M} ^ {\text {a u g}} (D) \in S \right] = p _ {\mathbb {F N}} \Pr \left[ \mathcal {M} (D) \in S \right] + \left(1 - p _ {\mathbb {F N}}\right) \Pr \left[ \mathcal {M} ^ {\prime} (D) \in S \right] \\ \leq p _ {\mathsf {F N}} \left(e ^ {\varepsilon} \Pr \left[ \mathcal {M} \left(D ^ {\prime}\right) \in S \right] + \delta^ {\text {e s t}}\right) + \left(1 - p _ {\mathsf {F N}}\right) \left(e ^ {\varepsilon} \Pr \left[ \mathcal {M} ^ {\prime} \left(D ^ {\prime}\right) \in S \right] + \delta^ {\text {e s t}}\right) \\ \leq e ^ {\varepsilon} \Pr [ \mathcal {M} ^ {\mathrm {a u g}} (D ^ {\prime}) \in S ] + \delta^ {\mathrm {e s t}} \\ \end{array}
$$
If the original mechanism  $\mathcal{M}$  is not  $(\varepsilon, \delta^{\mathrm{est}})$ -DP, then we have
$$
\begin{array}{l} \Pr \left[ \mathcal {M} ^ {\text {a u g}} (D) \in S \right] = p _ {\mathbb {F P}} \Pr \left[ \mathcal {M} (D) \in S \right] + \left(1 - p _ {\mathbb {F P}}\right) \Pr \left[ \mathcal {M} ^ {\prime} (D) \in S \right] \\ \leq p _ {\mathbb {F P}} \left(e ^ {\varepsilon} \Pr \left[ \mathcal {M} \left(D ^ {\prime}\right) \in S \right] + \delta^ {\text {u p p}}\right) + \left(1 - p _ {\mathbb {F P}}\right) \left(e ^ {\varepsilon} \Pr \left[ \mathcal {M} ^ {\prime} \left(D ^ {\prime}\right) \in S \right] + \delta^ {\text {e s t}}\right) \\ \leq e ^ {\varepsilon} \Pr [ \mathcal {M} ^ {\mathrm {a u g}} (D ^ {\prime}) \in S ] + \delta^ {\mathrm {e s t}} + p _ {\mathrm {F P}} (\delta^ {\mathrm {u p p}} - \delta^ {\mathrm {e s t}}) \\ \end{array}
$$
Hence, this augmented EVR algorithm will be  $(\varepsilon, \delta^{\mathrm{est}} + p_{\mathrm{FP}}(\delta^{\mathrm{upp}} - \delta^{\mathrm{est}}))$ , and if  $p_{\mathrm{FP}}$  is around  $\delta^{\mathrm{est}}$ , then this extra factor  $p_{\mathrm{FP}}(\delta^{\mathrm{upp}} - \delta^{\mathrm{est}})$  will be very small. We can also adjust the privacy guarantee for  $\mathcal{M}'$  such that the privacy guarantees for the two cases are the same, which can further optimize the final privacy cost.
F.2 Technical Details
In this section, we provide theoretical justification for the heuristic of setting  $\Delta = 0.4(1 / \tau - 1 / \rho) \delta^{\mathrm{est}}$ .
For notation convenience, throughout this section, we talk about  $\widehat{\delta}_{\mathrm{MC}}$ , but the same argument also applies to  $\widehat{\delta}_{\mathrm{IS},\theta}$  with a not too large  $\theta$ , unless otherwise specified. We use  $\widehat{\delta}_{\mathrm{MC}}(x)$  to denote the density of  $\widehat{\delta}_{\mathrm{MC}}$  at  $x$ . Note that  $\widehat{\delta}_{\mathrm{MC}}(0) = \infty$ .
We make the following assumption about the distribution of  $\widehat{\delta}_{\mathrm{MC}}$
Assumption 30.  $\operatorname*{Pr}[\widehat{\delta}_{MC} = 0]\geq 1 / 2.$
While intuitive, this assumption is hard to analyze for the case of Subsampled Gaussian mechanism. Therefore, we instead provide a condition for which the assumption holds for Pure Gaussian mechanism.
Lemma 31. Fix  $\varepsilon, \sigma$ . When  $k / (2\sigma^2) \leq \varepsilon$ , Assumption 30 holds.
Proof. The PRV for the composition of  $k$  Gaussian mechanism is  $\mathcal{N}\left(\frac{k}{2\sigma^2},\frac{k}{\sigma^2}\right)$ .
$$
\Pr \left[ \left(1 - e ^ {\varepsilon - Y}\right) _ {+} = 0 \right] = \Pr \left[ Y \leq \varepsilon \right]
$$
which is clearly  $\geq 1 / 2$  when  $k / (2\sigma^2)\leq \varepsilon$
We also empirically verify this assumption for Subsampled Gaussian mechanism as in Figure 7. As we can see, the  $\theta$  selected by our heuristic (the red star) has  $\operatorname{Pr}[\widehat{\delta}_{\mathrm{IS},\theta} = 0] \approx 1/2$  which matches our principle of selecting  $\theta$ . The  $\theta$  minimizes the analytical bound (which we are going to use in practice) achieves  $\operatorname{Pr}[\widehat{\delta}_{\mathrm{IS},\theta} = 0] \approx 0.88 \gg 0.5$ .
Figure 7: We empirically estimate  $\operatorname*{Pr}[\widehat{\delta}_{\mathrm{IS},\theta} = 0]$  for the case of Poisson Subsampled Gaussian mechanism. We set  $q = 10^{-3}$ ,  $\sigma = 0.6$ ,  $\varepsilon = 1.5$ ,  $k = 100$ .  $\delta_{Y}(\varepsilon)\approx 7.7\times 10^{-6}$  in this case. The red star indicates the second moment for the value of  $\theta$  selected by our heuristic in Definition 18. The blue star indicates the  $\theta$  that minimizes the analytical bound.
Our goal is to show that  $\operatorname*{Pr}[\widehat{\delta}_{\mathbb{MC}}^m -\delta_Y(\varepsilon) < - \Delta^* ]\gtrsim \operatorname*{Pr}[\widehat{\delta}_{\mathbb{MC}}^m -\delta_Y(\varepsilon) > \Delta^* ]$  for large  $m$ . For notation simplicity, we denote  $\delta \coloneqq \mathbb{E}[\widehat{\delta}_{\mathbb{MC}}] = \delta_Y(\varepsilon)$  in the remaining of the section. We also denote
$$
p _ {0} := \Pr \left[ \hat {\delta} _ {\mathrm {M C}} = 0 \right]
$$
$$
p _ {(0, 1)} := \Pr [ 0 <   \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} <   \delta ]
$$
$$
p _ {(1, 2)} := \Pr [ \delta \leq \widehat {\delta} _ {\mathrm {M C}} \leq 2 \delta ]
$$
$$
p _ {(2, \infty)} := \Pr [ \widehat {\boldsymbol {\delta}} _ {\mathcal {M C}} \geq 2 \delta ]
$$
Note that  $p_0 + p_{(0,1)} + p_{(1,2)} + p_{(2,\infty)} = 1$ . We also write  $\widehat{\delta}_{\mathbb{MC}}|_{(a,b)}$  and  $\widehat{\delta}_{\mathbb{MC}}|_{[a,b)}$  to indicate the truncated distribution of  $\widehat{\delta}_{\mathbb{MC}}$  on  $(a,b)$  and  $[a,b)$ , respectively.
We first construct an alternative random variable  $\widetilde{\delta}_{\mathrm{MC}}$  with the following distribution:
$$
\widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} = \left\{ \begin{array}{l l} 2 \delta - x \text {f o r} x \sim \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} | _ {\geq 2 \delta} & \mathrm {w . p .} p _ {(2, \infty)} \\ x \text {f o r} x \sim \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} | _ {(0, \delta)} & \mathrm {w . p .} p _ {(0, 1)} \\ 2 \delta - x \text {f o r} x \sim \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} | _ {(0, \delta)} & \mathrm {w . p .} p _ {(0, 1)} \\ x \text {f o r} x \sim \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} | _ {\geq 2 \delta} & \mathrm {w . p .} p _ {(2, \infty)} \\ \delta & \mathrm {w . p .} 1 - 2 \left(p _ {(0, 1)} + p _ {(2, \infty)}\right) \end{array} \right. \tag {23}
$$
This is a valid probability due to Assumption 30, as  $p_{(0,1)} + p_{(2,\infty)} \leq 1 - p_0 \leq 1 / 2$ . The distribution of  $\widetilde{\delta}_{\mathrm{MC}}$  is illustrated in Figure 8. Note that  $\widetilde{\delta}_{\mathrm{MC}}$  is a symmetric distribution with  $\mathbb{E}[\widetilde{\delta}_{\mathrm{MC}}] = \delta$ .
Figure 8: Illustration for the density function of  $\widetilde{\delta}_{\mathrm{MC}}$  (black curve indicates the density curve for  $\widehat{\delta}_{\mathrm{MC}}$ , and red curve indicates the density curve for  $\widehat{\delta}_{\mathrm{IS},\theta}$ ).
Similar to the notation of  $\widehat{\delta}_{\mathrm{MC}}^m$ , we write  $\widetilde{\delta}_{\mathrm{MC}}^m \coloneqq \frac{1}{m} \sum_{i=1}^{m} \widetilde{\delta}_{\mathrm{MC}}^{(i)}$ . We show an asymmetry result for  $\widetilde{\delta}_{\mathrm{MC}}^m$  in terms of  $\delta$ .
Lemma 32. For any  $\Delta^{*}\in \mathbb{R}$ , we have
$$
\Pr \left[ \widetilde {\boldsymbol {\delta}} _ {M C} ^ {m} - \delta <   - \Delta^ {*} \right] \geq \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {M C} ^ {m} - \delta > \Delta^ {*} \right] \tag {24}
$$
Proof. Note that the above argument holds trivially when  $m = 0, 1$ . For  $m \geq 2$ , we use the induction argument. Suppose we have
$$
\Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m - 1} - \delta <   - \Delta^ {*} \right] \geq \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m - 1} - \delta > \Delta^ {*} \right] \tag {25}
$$
for any  $\Delta^{*}\in \mathbb{R}$  . That is,
$$
\Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta <   - (m - 1) \Delta^ {*} \right] \geq \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta > (m - 1) \Delta^ {*} \right] \tag {26}
$$
for any  $\Delta^{*}\in \mathbb{R}$
$$
\begin{array}{l} \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta <   - \Delta^ {*} \right] = \Pr \left[ \sum_ {i = 1} ^ {m} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - m \delta <   - m \Delta^ {*} \right] \\ = \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(i)} - (m - 1) \delta + \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(m)} - \delta <   - m \Delta^ {*} \right] \\ = \int \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta <   - m \Delta^ {*} - x \right] \Pr [ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(m)} - \delta = x ] d x \\ \geq \int \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta > m \Delta^ {*} + x \right] \Pr [ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(m)} - \delta = x ] d x \\ = \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(i)} - (m - 1) \delta - (\widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(m)} - \delta) > m \Delta^ {*} \right] \\ = \int \Pr \left[ x - \left(\widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(m)} - \delta\right) > m \Delta^ {*} \right] \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta = x \right] \mathrm {d} x \\ = \int \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(m)} - \delta <   - m \Delta^ {*} + x \right] \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {(i)} - (m - 1) \delta = x \right] d x \\ \geq \int \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(m)} - \delta > m \Delta^ {*} - x \right] \Pr \left[ \sum_ {i = 1} ^ {m - 1} \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {(i)} - (m - 1) \delta = x \right] \mathrm {d} x \\ = \left[ \sum_ {i = 1} ^ {m} \widetilde {\delta} _ {\mathbb {M C}} ^ {(i)} - m \delta > m \Delta^ {*} \right] \\ = \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta > \Delta^ {*} \right] \\ \end{array}
$$
where the two inequalities are due to the induction assumption.
Now we come back and analyze  $\widehat{\delta}_{\mathrm{MC}}^{m}$  by using Lemma 32.
$$
\begin{array}{l} \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \leq - \Delta^ {*} \right] \\ = \Pr \left[ \tilde {\delta} _ {\mathrm {M C}} ^ {m} - \tilde {\delta} _ {\mathrm {M C}} ^ {m} + \tilde {\delta} _ {\mathrm {M C}} ^ {m} - \delta \leq - \Delta^ {*} \right] \\ \geq \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \leq c \right] \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} + \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \leq - \Delta^ {*} | \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \leq c \right] \\ \geq \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathcal {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathcal {M C}} ^ {m} \leq c \right] \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathcal {M C}} ^ {m} - \delta \leq - \Delta^ {*} - c \right] \\ \geq \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \leq c \right] \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + c \right] \\ \end{array}
$$
$$
\begin{array}{l} \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + c \right] \\ = \Pr \left[ \widetilde {\delta} _ {\mathrm {M C}} ^ {m} - \widehat {\delta} _ {\mathrm {M C}} ^ {m} + \widehat {\delta} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + c \right] \\ \geq \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \geq - c \right] \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} + \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + c | \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \geq - c \right] \\ \geq \Pr \left[ \widetilde {\delta} _ {\mathrm {M C}} ^ {m} - \widehat {\delta} _ {\mathrm {M C}} ^ {m} \geq - c \right] \Pr \left[ \widehat {\delta} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + 2 c \right] \\ \end{array}
$$
$$
\Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \leq - \Delta^ {*} \right] \geq \left(\Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} \leq c \right]\right) ^ {2} \Pr \left[ \widehat {\boldsymbol {\delta}} _ {\mathrm {M C}} ^ {m} - \delta \geq \Delta^ {*} + 2 c \right] \tag {27}
$$
for any  $\Delta^{*}\in \mathbb{R}$ . Note that the above argument does not require  $\widetilde{\delta}_{\mathrm{MC}}^m$  and  $\widetilde{\delta}_{\mathrm{MC}}^m$  to be correlated. To maximize  $\operatorname*{Pr}\left[\widehat{\delta}_{\mathrm{MC}}^m -\widetilde{\delta}_{\mathrm{MC}}^m\leq c\right]$ , we can sample  $\widetilde{\delta}_{\mathrm{MC}}^m$  for a given  $\widehat{\delta}_{\mathrm{MC}}^m$  as follows: for each  $\widehat{\delta}_{\mathrm{MC}}^{(i)}$ ,
1. If  $\widehat{\delta}_{\mathrm{MC}}^{(i)} > 0$ , then let  $\widetilde{\delta}_{\mathrm{MC}}^{(i)} = \widehat{\delta}_{\mathrm{MC}}^{(i)}$ .
2. If  $\widehat{\delta}_{\mathbb{MC}}^{(i)} = 0$ , then with probability  $p_{(2,\infty)} / p_0$ , output  $2\delta - x$  for  $x \sim \widehat{\delta}_{\mathbb{MC}}|_{(2,\infty)}$ ; with probability  $p_{(0,1)} / p_0$  output  $2\delta - x$  for  $x \sim \widehat{\delta}_{\mathbb{MC}}|_{(0,1)}$ ; with probability  $1 - (p_{(0,1)} + p_{(2,\infty)}) / p_0$  output  $\delta$ .
Denote the random variable  $\delta_{\mathrm{diff}} := \widehat{\delta}_{\mathrm{MC}} - \widetilde{\delta}_{\mathrm{MC}}$ . It is not hard to see that  $\mathbb{E}[\delta_{\mathrm{diff}}^2] \leq \mathbb{E}[\widehat{\delta}_{\mathrm{MC}}^2] + \delta^2 \leq 2\mathbb{E}[\widehat{\delta}_{\mathrm{MC}}^2]$ . By Bennett's inequality, with  $m \geq \frac{2\nu}{\Delta^2}\log (\tau /\delta^{\mathrm{est}})$ , we have
$$
\begin{array}{l} \Pr \left[ \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {m} - \widetilde {\boldsymbol {\delta}} _ {\mathbb {M C}} ^ {m} \leq c \right] \gtrsim 1 - \frac {\tau}{\delta^ {\text {e s t}}} \exp \left(- \frac {c ^ {2}}{\Delta^ {2}}\right) \\ = 1 - O \left(\frac {\tau}{\delta^ {\mathrm {e s t}}}\right) \\ \end{array}
$$
if we set  $c = O(\Delta)$ . Let  $c = \frac{1}{4}\Delta$ , then by setting  $-\Delta^{*} = -\Delta$  and  $\Delta^{*} + 2c = (1 / \tau - 1 / \rho)\delta^{\mathrm{est}} - \Delta$ , we have  $\Delta = 0.4(1 / \tau - 1 / \rho)\delta^{\mathrm{est}}$ .
To summarize, when we set  $\Delta$  with the heuristic, we have
$$
\begin{array}{l} \Pr \left[ \widehat {\delta} _ {\mathbb {M C}} ^ {m} - \delta \geq \Delta^ {*} + 2 c \right] \lesssim \frac {\tau / \delta^ {\text {e s t}}}{\left(1 - O \left(\frac {\tau}{\delta^ {\text {e s t}}}\right)\right) ^ {2}} \\ \approx \tau / \delta^ {\mathrm {e s t}} \\ \end{array}
$$
G Pseudocode for Privacy Accounting for DP-SGD with EVR Paradigm
In this section, we outline the steps of privacy accounting for DP-SGD with our EVR paradigm. Recall from Lemma 16 that for subsampled Gaussian mechanism with sensitivity  $C$ , noise variance  $C^2 \sigma^2$ , and subsampling rate  $q$ , one dominating pair  $(P, Q)$  is  $Q := \mathcal{N}(0, \sigma^2)$  and  $P := (1 - q) \mathcal{N}(0, \sigma^2) + q \mathcal{N}(1, \sigma^2)$ . Hence, for DP-SGD with  $k$  iterations, the dominating pair is the product distribution  $P := P_1 \times \ldots \times P_k$  and  $Q := Q_1 \times \ldots \times Q_k$  where each  $P_i$  and  $Q_i$  follow the same distribution as  $P$  and  $Q$ .<sup>3</sup>
Algorithm 4 Privacy Accounting for DP-SGD with EVR Paradigm