SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jiachen T. Wang
Saeed Mahloujifar
sfar@princeton.edu
Tong Wu
tongwu@princeton.edu
Ruoxi Jia
Bounding privacy leakage over compositions, i.e., privacy accounting, is a key challenge in differential privacy (DP). However, the privacy parameter  $(\varepsilon$  or  $\delta$ ) is often easy to estimate but hard to bound. In this paper, we propose a new differential privacy paradigm called estimate-verify-release (EVR), which tackles the challenges of providing a strict upper bound for the privacy parameter in DP compositions by converting an estimate of privacy parameter into a formal guarantee. The EVR paradigm first verifies whether the mechanism meets the estimated privacy guarantee, and then releases the query output based on the verification result. The core component of the EVR is privacy verification. We develop a randomized privacy verifier using Monte Carlo (MC) technique. Furthermore, we propose an MC-based DP accountant that outperforms existing DP accounting techniques in terms of accuracy and efficiency. MC-based DP verifier and accountant is applicable to an important and commonly used class of DP algorithms, including the famous DP-SGD. An empirical evaluation shows the proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.
The concern of privacy is a major obstacle to deploying machine learning (ML) applications. In response, ML algorithms with differential privacy (DP) guarantees have been proposed and developed. For privacy-preserving ML algorithms, DP mechanisms are often repeatedly applied to private training data. For instance, when training deep learning models using DP-SGD [1], it is often necessary to execute sub-sampled Gaussian mechanisms on the private training data thousands of times.
A major challenge in machine learning with differential privacy is privacy accounting, i.e., measuring the privacy loss of the composition of DP mechanisms. A privacy accountant takes a list of mechanisms, and returns the privacy parameter  $(\varepsilon$  and  $\delta)$  for the composition of those mechanisms. Specifically, a privacy accountant is given a target  $\varepsilon$  and finds the smallest achievable  $\delta$  such that the composed mechanism  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -DP (we can also fix  $\delta$  and find  $\varepsilon$ ). We use  $\delta_{\mathcal{M}}(\varepsilon)$  to denote the smallest achievable  $\delta$  given  $\varepsilon$ , which is often referred to as optimal privacy curve in the literature.
Training deep learning models with DP-SGD is essentially the adaptive composition for thousands of sub-sampled Gaussian Mechanisms. Moment Accountant (MA) is a pioneer solution for privacy loss calculation in differentially private deep learning [1]. However, MA does not provide the optimal  $\delta_{\mathcal{M}}(\varepsilon)$  in general [44]. This motivates the development of more advanced privacy accounting techniques that outperforms MA. Two major lines of such works are based on Fast Fourier Transform (FFT) (e.g., [19] and Central Limit Theorem (CLT) [7, 41]. Both techniques can provide an estimate
Estimate-Verify-Release (EVR) Paradigm
Figure 2: An overview of our EVR paradigm. EVR converts an estimated  $(\varepsilon, \delta)$  provided by a privacy accountant into a formal guarantee. Compared with the original mechanism, the EVR has an extra failure mode that does not output anything when the estimated  $(\varepsilon, \delta)$  is rejected. We show that the MC-based verifier we proposed can achieve negligible failure probability  $(O(\delta))$  in Section 4.4.
as well as an upper bound for  $\delta_{\mathcal{M}}(\varepsilon)$  though bounding the worst-case estimation error. In practice, only the upper bounds for  $\delta_{\mathcal{M}}(\varepsilon)$  can be used, as differential privacy is a strict guarantee.
Motivation: estimates can be more accurate than upper bounds. The motivation for this paper stems from the limitations of current privacy accounting techniques in providing tight upper bounds for  $\delta_{\mathcal{M}}(\varepsilon)$ . Despite outperforming MA, both FFT- and CLT-based methods can provide ineffective bounds in certain regimes [19, 41]. We demonstrate such limitations in Figure 1 using the composition of Gaussian mechanisms. For FFT-based technique [19], we can see that although it outperforms MA for most of the regimes, the upper bounds (blue dashed curve) are worse than that of MA when  $\delta < 10^{-10}$  due to computational limitations (as discussed in [19]'s Appendix A; also see Remark 6 for a discussion of why the regime of  $\delta < 10^{-10}$  is important). The CLT-based techniques (e.g., [41]) also produce sub-optimal upper bounds (red dashed curve) for the entire range of  $\delta$ . This is primarily due to the small number of mechanisms used ( $k = 1200$ ), which does not meet the requirements for CLT bounds to converge (similar phenomenon observed in [41]). On the other hand, we can see that the estimates of  $\delta_{\mathcal{M}}(\varepsilon)$  from both FFT and CLT-based techniques, which estimate the parameters rather than providing an upper bound, are in fact very close to the ground truth (the three curves overlapped in Figure 1). However, as we mentioned earlier, these accurate estimations cannot be used in practice, as we cannot prove that they do not underestimate  $\delta_{\mathcal{M}}(\varepsilon)$ . The
Figure 1: Results of estimating/bounding  $\delta_{\mathcal{M}}(\varepsilon)$  for the composition of 1200 Gaussian mechanisms with  $\sigma = 70$  - 'upp' means upper bound and -est' means estimate. Curves of Exact', FFTest, and CLT-est' are overlapped. The groundtruth curve (Exact') for pure Gaussian mechanism can be computed analytically [19].
dilemma raises an important question: can we develop new techniques that allow us to use privacy parameter estimates instead of strict upper bounds in privacy accounting?1
This paper gives a positive answer to it. Our contributions are summarized as follows:
Estimate-Verify-Release (EVR): a DP paradigm that converts privacy parameter estimate into a formal guarantee. We develop a new DP paradigm called Estimate-Verify-Release, which augments a mechanism with a formal privacy guarantee based on its privacy parameter estimates. The basic idea of EVR is to first verify whether the mechanism satisfies the estimated DP guarantee, and release the mechanism's output if the verification is passed. The core component of the EVR paradigm is privacy verification. A DP verifier can be randomized and imperfect, suffering from both false positives (accept an underestimation) and false negatives (reject an overestimation). We show that EVR's privacy guarantee can be achieved when privacy verification has a low false negative rate.
A Monte Carlo-based DP Verifier. For an important and widely used class of DP algorithms including Subsampled Gaussian mechanism (the building block for DP-SGD), we develop a Monte
<sup>1</sup>We note that this not only brings benefits for the regime where  $\delta < 10^{-10}$ , but also for the more common regime where  $\delta \approx 10^{-5}$ . See Figure 3 for an example.
Carlo (MC) based DP verifier for the EVR paradigm. We present various techniques that ensure the DP verifier has both a low false positive rate (for privacy guarantee) and a low false negative rate (for utility guarantee, i.e., making the EVR and the original mechanism as similar as possible).
A Monte Carlo-based DP Accountant. We further propose a new MC-based approach for DP accounting, which we call the  $MC$  accountant. It utilizes similar MC techniques as in privacy verification. We show that the MC accountant achieves several advantages over existing privacy accounting methods. In particular, we demonstrate that MC accountant is efficient for online privacy accounting, a realistic scenario for privacy practitioners where one wants to update the estimate on privacy guarantee whenever executing a new mechanism.
Figure 2 gives an overview of the proposed EVR paradigm as well as this paper's contributions.
2 Privacy Accounting: a Mean Estimation/Bounding Problem
In this section, we review relevant concepts and introduce privacy accounting as a mean estimation/bounding problem.
Symbols and notations. We use  $D, D' \in \mathbb{N}^{\mathcal{X}}$  to denote two datasets with an unspecified size over space  $\mathcal{X}$ . We call two datasets  $D$  and  $D'$  adjacent (denoted as  $D \sim D'$ ) if we can construct one by adding/removing one data point from the other. We use  $P, Q$  to denote random variables. We also overload the notation and denote  $P(\cdot), Q(\cdot)$  the density function of  $P, Q$ .
Differential privacy and its equivalent characterizations. Having established the notations, we can now proceed to formally define differential privacy.
Definition 1 (Differential Privacy [13]). For  $\varepsilon, \delta \geq 0$ , a randomized algorithm  $\mathcal{M}:\mathbb{N}^{\mathcal{X}}\to \mathcal{Y}$  is  $(\varepsilon, \delta)$ -differentially private if for every pair of adjacent datasets  $D\sim D^{\prime}$  and for every subset of possible outputs  $E\subseteq \mathcal{V}$ , we have  $\operatorname*{Pr}_{\mathcal{M}}[\mathcal{M}(D)\in E]\leq e^{\varepsilon}\operatorname*{Pr}_{\mathcal{M}}[\mathcal{M}(D^{\prime})\in E] + \delta$
One can alternatively define differential privacy in terms of the maximum possible divergence between the output distribution of any pair of  $\mathcal{M}(D)$  and  $\mathcal{M}(D')$ .
Lemma 2 ([5]). A mechanism  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -DP iff  $\sup_{D \sim D'} \mathsf{E}_{e^{\varepsilon}}(\mathcal{M}(D) \| \mathcal{M}(D')) \leq \delta$ , where  $\mathsf{E}_{\gamma}(P \| Q) := \mathbb{E}_{o \sim Q}[(\frac{P(o)}{Q(o)} - \gamma)_+]$  &  $(a)_+ := \max(a, 0)$ .
$\mathsf{E}_{\gamma}$  is usually referred as Hockey-Stick (HS) Divergence in the literature. For every mechanism  $\mathcal{M}$  and every  $\varepsilon \geq 0$ , there exists a smallest  $\delta$  such that  $\mathcal{M}$  is  $(\varepsilon, \delta)$ -DP. Following the literature [44, 2], we formalize such a  $\delta$  as a function of  $\varepsilon$ .
Definition 3 (Optimal Privacy Curve). The optimal privacy curve of a mechanism  $\mathcal{M}$  is the function  $\delta_{\mathcal{M}}: \mathbb{R}^{+} \to [0,1]$  s.t.  $\delta_{\mathcal{M}}(\varepsilon) := \sup_{D \sim D'} \mathsf{E}_{e^{\varepsilon}}(\mathcal{M}(D) \| \mathcal{M}(D'))$ .
Dominating Distribution Pair and Privacy Loss Random Variable (PRV). It is computationally infeasible to find  $\delta_{\mathcal{M}}(\varepsilon)$  by computing  $\mathsf{E}_{e^{\varepsilon}}(\mathcal{M}(D)\| \mathcal{M}(D'))$  for all pairs of adjacent dataset  $D$  and  $D'$ . A mainstream strategy in the literature is to find a pair of distributions  $(P,Q)$  that dominates all  $(\mathcal{M}(D),\mathcal{M}(D'))$  in terms of the Hockey-Stick divergence. This results in the introduction of dominating distribution pair and privacy loss random variable (PRV).
Definition 4 ([44]). A pair of distributions  $(P, Q)$  is a pair of dominating distributions for  $\mathcal{M}$  under adjacent relation  $\sim$  if for all  $\gamma \geq 0$ ,  $\sup_{D \sim D'} \mathsf{E}_{\gamma}(\mathcal{M}(D) \| \mathcal{M}(D')) \leq \mathsf{E}_{\gamma}(P \| Q)$ . If equality is achieved for all  $\gamma \geq 0$ , then we say  $(P, Q)$  is a pair of tightly dominating distributions for  $\mathcal{M}$ . Furthermore, we call  $Y := \log \left(\frac{P(o)}{Q(o)}\right)$ ,  $o \sim P$  the privacy loss random variable (PRV) of  $\mathcal{M}$  associated with dominating distribution pair  $(P, Q)$ .
Zhu et al. [44] shows that all mechanisms have a pair of tightly dominating distributions. Hence, we can alternatively characterize the optimal privacy curve as  $\delta_{\mathcal{M}}(\varepsilon) = \mathsf{E}_{e^{\varepsilon}}(P\| Q)$  for the tightly dominating pair  $(P,Q)$ , and we have  $\delta_{\mathcal{M}}(\varepsilon)\leq \mathsf{E}_{e^{\varepsilon}}(P\| Q)$  if  $(P,Q)$  is a dominating pair that is not necessarily tight. The importance of the concept of PRV comes from the fact that we can write  $\mathsf{E}_{e^{\varepsilon}}(P\| Q)$  as an expectation over it:  $\mathsf{E}_{e^{\varepsilon}}(P\| Q) = \mathbb{E}_Y\left[\left(1 - e^{\varepsilon -Y}\right)_+\right]$ . Thus, one can bound  $\delta_{\mathcal{M}}(\varepsilon)$  by first identifying  $\mathcal{M}$ 's dominating pair distributions as well as the associated PRV  $Y$ , and then computing this expectation. Such a formulation allows us to bound  $\delta_{\mathcal{M}}(\varepsilon)$  without enumerating over
all adjacent  $D$  and  $D'$ . For notation convenience, we denote  $\delta_Y(\varepsilon) \coloneqq \mathbb{E}_Y\left[(1 - e^{\varepsilon - Y})_+\right]$ . Clearly,  $\delta_{\mathcal{M}} \leq \delta_Y$ . If  $(P, Q)$  is a tightly dominating pair for  $\mathcal{M}$ , then  $\delta_{\mathcal{M}} = \delta_Y$ .
Privacy Accounting as a Mean Estimation/Bounding Problem. Privacy accounting aims to estimate and bound the optimal privacy curve  $\delta_{\mathcal{M}}(\varepsilon)$  for adaptively composed mechanism  $\mathcal{M} = \mathcal{M}_1 \circ \dots \circ \mathcal{M}_k(D)$ . The adaptive composition of two mechanisms  $\mathcal{M}_1$  and  $\mathcal{M}_2$  is defined as  $\mathcal{M}_1 \circ \mathcal{M}_2(D) := (\mathcal{M}_1(D), \mathcal{M}_2(D, \mathcal{M}_1(D))),$  in which  $\mathcal{M}_2$  can access both the dataset and the output of  $\mathcal{M}_1$ . Most of the practical privacy accounting techniques are based on the concept of PRV, centered on the following result.
Lemma 5 ([44]). Let  $(P_j, Q_j)$  be a pair of tightly dominating distributions for mechanism  $\mathcal{M}_j$  for  $j \in \{1, \ldots, k\}$ . Then  $(P_1 \times \dots \times P_k, Q_1 \times \dots \times Q_k)$  is a pair of dominating distributions for  $\mathcal{M} = \mathcal{M}_1 \circ \dots \circ \mathcal{M}_k$ , where  $\times$  denotes the product distribution. Furthermore, the associated privacy loss random variable is  $Y = \sum_{i=1}^{k} Y_i$  where  $Y_i$  is the PRV associated with  $(P_i, Q_i)$ .
Lemma 5 suggests that privacy accounting for DP composition can be cast into a mean estimation/bounding problem where one aims to approximate or bound the expectation in (2) when  $Y = \sum_{i=1}^{k} Y_i$ . Note that while Lemma 5 does not guarantee a pair of tightly dominating distributions for the adaptive composition, it cannot be improved in general, as noted in [10]. Hence, all the current privacy accounting techniques work on  $\delta_Y$  instead of  $\delta_{\mathcal{M}}$ , as Lemma 5 is tight even for non-adaptive composition. Following the prior works, in this paper, we only consider the practical scenarios where Lemma 5 is tight for the simplicity of presentation. That is, we assume  $\delta_Y = \delta_{\mathcal{M}}$  unless otherwise specified.
Most of the existing privacy accounting techniques can be described as different techniques for such a mean estimation problem. Example-1: FFT-based methods. This line of works (e.g., [19]) discretizes the domain of each  $Y_{i}$  and use Fast Fourier Transform (FFT) to speed up the approximation of  $\delta_Y(\varepsilon)$ . The upper bound is derived through the worst-case error bound for the approximation.
Example-2: CLT-based methods. [7, 41] use CLT to approximate the distribution of  $Y = \sum_{i=1}^{k} Y_i$  as Gaussian distribution. They then use CLT's finite-sample approximation guarantee to derive the upper bound for  $\delta_Y(\varepsilon)$ .
Remark 6 (The Importance of Privacy Accounting in Regime  $\delta < 10^{-10}$ ). The regime where  $\delta < 10^{-10}$  is of significant importance for two reasons. (1)  $\delta$  serves as an upper bound on the chance of severe privacy breaches, such as complete dataset exposure, necessitating a "cryptographically small" value, namely,  $\delta < n^{-\omega(1)}$  [14, 40]. (2) Even with the oft-used yet questionable guideline of  $\delta \approx n^{-1}$  or  $n^{-1.1}$ , datasets of modern scale, such as JFT-3B [43] or LAION-5B [37], already comprise billions of records, thus rendering small  $\delta$  values crucial. While we acknowledge that it requires a lot of effort to achieve a good privacy-utility tradeoff even for the current choice of  $\delta \approx n^{-1}$ , it is important to keep such a goal in mind.
3 Estimate-Verify-Release
As mentioned earlier, upper bounds for  $\delta_Y(\varepsilon)$  are the only valid options for privacy accounting techniques. However, as we have demonstrated in Figure 1, both FFT- and CLT-based methods can provide overly conservative upper bounds in certain regimes. On the other hand, their estimates for  $\delta_Y(\varepsilon)$  can be very close to the ground truth even though there is no provable guarantee. Therefore, it is highly desirable to develop new techniques that enable the use of privacy parameter estimates instead of overly conservative upper bounds in privacy accounting.
We tackle the problem by introducing a new paradigm for constructing DP mechanisms, which we call Estimate-Verify-Release (EVR). The key component of the EVR is an object called DP verifier (Section 3.1). The full EVR paradigm is then presented in Section 3.2, where the DP verifier is utilized as a building block to guarantee privacy.
3.1 Differential Privacy Verifier
We first formalize the concept of differential privacy verifier, the central element of the EVR paradigm. In informal terms, a DP verifier is an algorithm that attempts to verify whether a mechanism satisfies a specific level of differential privacy.
Definition 7 (Differential Privacy Verifier). We say a differentially private verifier  $DPV(\cdot)$  is an algorithm that takes the description of a mechanism  $\mathcal{M}$  and proposed privacy parameter  $(\varepsilon, \delta^{\mathrm{est}})$  as input, and returns True  $\leftarrow DPV(\mathcal{M}, \varepsilon, \delta^{\mathrm{est}})$  if the algorithm believes  $\mathcal{M}$  is  $(\varepsilon, \delta^{\mathrm{est}})$ - $DP$  (i.e.,  $\delta^{\mathrm{est}} \geq \delta_Y(\varepsilon)$  where  $Y$  is the PRV of  $\mathcal{M}$ ), and returns False otherwise.
A differential privacy verifier can be imperfect, suffering from both false positives (FP) and false negatives (FN). Typically, FP rate is the likelihood for DPV to accept  $(\varepsilon, \delta^{\mathrm{est}})$  when  $\delta^{\mathrm{est}} < \delta_Y(\varepsilon)$ . However,  $\delta^{\mathrm{est}}$  is still a good estimate for  $\delta_Y(\varepsilon)$  by being a small (e.g.,  $< 10\%$ ) underestimate. To account for this, we introduce a smoothing factor,  $\tau \in (0,1]$ , such that  $\delta^{\mathrm{est}}$  is deemed "should be rejected" only when  $\delta^{\mathrm{est}} \leq \tau \delta_Y(\varepsilon)$ . A similar argument can be put forth for FN cases where we also introduce a smoothing factor  $\rho \in (0,1]$ . This leads to relaxed notions for FP/FN rate:
Definition 8. We say a DPV's  $\tau$ -relaxed false positive rate at  $(\varepsilon, \delta^{\mathrm{est}})$  is
$$
\mathsf {F P} _ {D P V} (\varepsilon , \delta^ {\mathrm {e s t}}; \tau) := \sup  _ {\mathcal {M}: \delta^ {\mathrm {e s t}} <   \tau \delta_ {Y} (\varepsilon)} \Pr_ {D P V} \left[ D P V (\mathcal {M}, \varepsilon , \delta^ {\mathrm {e s t}}) = \mathsf {T r u e} \right]
$$
We say a DPV's  $\rho$ -relaxed false negative rate at  $(\varepsilon, \delta^{\mathrm{est}})$  is
$$
\mathsf {F N} _ {D P V} (\varepsilon , \delta^ {\text {e s t}}; \rho) := \sup  _ {\mathcal {M}: \delta^ {\text {e s t}} > \rho \delta_ {Y} (\varepsilon)} \Pr_ {D P V} \left[ D P V (\mathcal {M}, \varepsilon , \delta^ {\text {e s t}}) = \mathsf {F a l s e} \right]
$$
Privacy Verification with DP Accountant. For a composed mechanism  $\mathcal{M} = \mathcal{M}_1\circ \dots \circ \mathcal{M}_k$ , a DP verifier can be easily implemented using any existing privacy accounting techniques. That is, one can execute DP accountant to obtain an estimate or upper bound  $(\varepsilon ,\hat{\delta})$  of the actual privacy parameter. If  $\delta^{\mathrm{est}} < \hat{\delta}$ , then the proposed privacy level is rejected as it is more private than what the DP accountant tells; otherwise, the test is passed. The input description of a mechanism  $\mathcal{M}$ , in this case, can differ depending on the DP accounting method. For Moment Accountant [1], the input description is the upper bound of the moment-generating function (MGF) of the privacy loss random variable for each individual mechanism. For FFT and CLT-based methods, the input description is the cumulative distribution functions (CDF) of the dominating distribution pair of each individual  $\mathcal{M}_i$ .
3.2 EVR: Ensuring Estimated Privacy with DP Verifier
We now present the full paradigm of EVR. As suggested by the name, it contains three steps: (1) Estimate: A privacy parameter  $(\varepsilon, \delta^{\mathrm{est}})$  for  $\mathcal{M}$  is estimated, e.g., based on a privacy auditing or accounting technique. (2) Verify: A DP verifier DPV is used for validating whether mechanism  $\mathcal{M}$  sat
isfits  $(\varepsilon, \delta^{\mathrm{est}})$ -DP guarantee. (3) Release: If DP verification test is passed, we can execute  $\mathcal{M}$  as usual; otherwise, the program is terminated immediately. For practical utility, this rejection probability needs to be small when  $(\varepsilon, \delta^{\mathrm{est}})$  is an accurate estimation. The procedure is summarized in Algorithm 1.
Given estimated privacy parameter  $(\varepsilon, \delta^{\mathrm{est}})$ , we have the privacy guarantee for the EVR paradigm:
Theorem 9. Algorithm 1 is  $(\varepsilon, \delta^{\mathrm{est}} / \tau)$ -DP for any  $\tau > 0$  if  $\mathsf{FP}_{DPP}(\varepsilon, \delta^{\mathrm{est}}; \tau) \leq \delta^{\mathrm{est}} / \tau$ .
We defer the proof to Appendix B. The implication of this result is that, for any estimate of the privacy parameter, one can safely use it as a DPV with a bounded false positive rate would enforce differential privacy. However, this is not enough: an overly conservative DPV that satisfies 0 FP rate but rejects everything would not be useful. When  $\delta^{\mathrm{est}}$  is accurate, we hope the DPV can also achieve a small false negative rate so that the output distributions of EVR and  $\mathcal{M}$  are indistinguishable. We discuss the instantiation of DPV in Section 4.
4 Monte Carlo Verifier of Differential Privacy
As we can see from Section 3.2, a DP verifier (DPV) that achieves a small FP rate is the central element for the EVR framework. In the meanwhile, it is also important that DPV has a low FN rate in
Algorithm 1 Estimate-Verify-Release (EVR) Framework
1: Input:  $\mathcal{M}$ : mechanism.  $D$ : dataset.  $(\varepsilon, \delta^{\mathrm{est}})$ : an estimated privacy parameter for  $\mathcal{M}$ .
2: if  $\mathrm{DPV}(\mathcal{M},\varepsilon ,\delta^{\mathrm{est}})$  outputs True then Execute  $\mathcal{M}(D)$ .
3: else Print  $\bot$ .
order to maintain the good utility of the EVR when the privacy parameter estimate is accurate. In this section, we introduce an instantiation of DPV based on the Monte Carlo technique that achieves both a low FP and FN rate, assuming the PRV is known for each individual mechanism.
Remark 10 (Mechanisms where PRV can be derived). PRV can be derived for many commonly used DP mechanisms such as the Laplace, Gaussian, and Subsampled Gaussian Mechanism [24, 19]. In particular, our DP verifier applies for DP-SGD, one of the most important application scenarios of privacy accounting. Moreover, the availability of PRV is also the assumption for most of the recently developed privacy accounting techniques (including FFT- and CLT-based methods). The extension beyond these commonly used mechanisms is an important future work in the field.