Title: A Randomized Approach to Tight Privacy Accounting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy; Privacy Accounting
Abstract: Bounding privacy leakage over compositions, i.e., privacy accounting, is a key challenge in differential privacy (DP). However, the privacy parameter ($\varepsilon$ or $\delta$) is often easy to estimate but hard to bound. In this paper, we propose a new differential privacy paradigm called estimate-verify-release (EVR), which tackles the challenges of providing a strict upper bound for the privacy parameter in DP compositions by converting an *estimate* of privacy parameter into a formal guarantee. The EVR paradigm first verifies whether the mechanism meets the *estimated* privacy guarantee, and then releases the query output based on the verification result. The core component of the EVR is privacy verification. We develop a randomized privacy verifier using Monte Carlo (MC) technique. Furthermore, we propose an MC-based DP accountant that outperforms existing DP accounting techniques in terms of accuracy and efficiency. MC-based DP verifier and accountant is applicable to an important and commonly used class of DP algorithms, including the famous DP-SGD. An empirical evaluation shows the proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.

A Randomized Approach to Tight Privacy Accounting

Abstract

Bounding privacy leakage over compositions, i.e., privacy accounting, is a key challenge in differential privacy (DP). However, the privacy parameter ( ε or δ ) is often easy to estimate but hard to bound. In this paper, we propose a new differential privacy paradigm called estimate-verify-release (EVR), which tackles the challenges of providing a strict upper bound for the privacy parameter in DP compositions by converting an estimate of privacy parameter into a formal guarantee. The EVR paradigm first verifies whether the mechanism meets the estimated privacy guarantee, and then releases the query output based on the verification result. The core component of the EVR is privacy verification. We develop a randomized privacy verifier using Monte Carlo (MC) technique. Furthermore, we propose an MC-based DP accountant that outperforms existing DP accounting techniques in terms of accuracy and efficiency. MC-based DP verifier and accountant is applicable to an important and commonly used class of DP algorithms, including the famous DP-SGD. An empirical evaluation shows the proposed EVR paradigm improves the utility-privacy tradeoff for privacy-preserving machine learning.

1 Introduction

The concern of privacy is a major obstacle to deploying machine learning (ML) applications. In response, ML algorithms with differential privacy (DP) guarantees have been proposed and developed. For privacy-preserving ML algorithms, DP mechanisms are often repeatedly applied to private training data. For instance, when training deep learning models using DP-SGD [1], it is often necessary to execute sub-sampled Gaussian mechanisms on the private training data thousands of times.

A major challenge in machine learning with differential privacy is privacy accounting , i.e., measuring the privacy loss of the composition of DP mechanisms. A privacy accountant takes a list of mechanisms, and returns the privacy parameter ( ε and δ ) for the composition of those mechanisms. Specifically, a privacy accountant is given a target ε and finds the smallest achievable δ such that the composed mechanism M is ( ε, δ ) -DP (we can also fix δ and find ε ). We use δ M ( ε ) to denote the smallest achievable δ given ε , which is often referred to as optimal privacy curve in the literature.

Training deep learning models with DP-SGD is essentially the adaptive composition for thousands of sub-sampled Gaussian Mechanisms. Moment Accountant (MA) is a pioneer solution for privacy loss calculation in differentially private deep learning [1]. However, MA does not provide the optimal δ M ( ε ) in general [44]. This motivates the development of more advanced privacy accounting techniques that outperforms MA. Two major lines of such works are based on Fast Fourier Transform (FFT) (e.g., [19] and Central Limit Theorem (CLT) [7, 41]. Both techniques can provide an estimate

Estimate-Verify-Release (EVR)Paradigm

Pass

Execute M

PrivacyAccountant

(e,8)estimate

How private is M?

Is M is (E,0)-DP ?

FFT-basedMethods

MC-basedVerifier(ours)

Terminate

CLT-basedMethods

MCAccountant(ours)

Reject

program

000

as well as an upper bound for δ M ( ε ) though bounding the worst-case estimation error. In practice, only the upper bounds for δ M ( ε ) can be used , as differential privacy is a strict guarantee.

Motivation: estimates can be more accurate than upper bounds. The motivation for this paper stems from the limitations of current privacy accounting techniques in providing tight upper bounds for δ M ( ε ) . Despite outperforming MA, both FFT- and CLT-based methods can provide ineffective bounds in certain regimes [19, 41]. We demonstrate such limitations in Figure 1 using the composition of Gaussian mechanisms. For FFT-based technique [19], we can see that although it outperforms MA for most of the regimes, the upper bounds (blue dashed curve) are worse than that of MA when δ < 10 -10 due to computational limitations (as discussed in [19]'s Appendix A; also see Remark 6 for a discussion of why the regime of δ < 10 -10 is important). The CLT-based techniques (e.g., [41]) also produce sub-optimal upper bounds (red dashed curve) for the entire range of δ . This is primarily due to the small number of mechanisms used ( k = 1200 ), which does not meet the requirements for CLT bounds to converge (similar phenomenon observed in [41]). On the other hand, we can see that the estimate s of δ M ( ε ) from both FFT and CLT-based techniques, which estimate the parameters rather than providing an upper bound, are in fact very close to the ground truth (the three curves overlapped in Figure 1). However, as we mentioned earlier, these accurate estimations cannot be used in practice, as we cannot prove that they do not underestimate δ M ( ε ) . The

Exact

GDP-Edgeworth-est

FFT-upp

FFT-est

GDP-Edgeworth-upp

MA

100

10-2

10-4

10-6

6

10-8

10-10

10-12

2.0

2.5

3.0

3.5

dilemma raises an important question: can we develop new techniques that allow us to use privacy parameter estimates instead of strict upper bounds in privacy accounting? 1

This paper gives a positive answer to it. Our contributions are summarized as follows:

Estimate-Verify-Release (EVR): a DP paradigm that converts privacy parameter estimate into a formal guarantee. We develop a new DP paradigm called Estimate-Verify-Release , which augments a mechanism with a formal privacy guarantee based on its privacy parameter estimates. The basic idea of EVR is to first verify whether the mechanism satisfies the estimated DP guarantee, and release the mechanism's output if the verification is passed. The core component of the EVR paradigm is privacy verification . A DP verifier can be randomized and imperfect, suffering from both false positives (accept an underestimation) and false negatives (reject an overestimation). We show that EVR's privacy guarantee can be achieved when privacy verification has a low false negative rate.

A Monte Carlo-based DP Verifier. For an important and widely used class of DP algorithms including Subsampled Gaussian mechanism (the building block for DP-SGD), we develop a Monte

Carlo (MC) based DP verifier for the EVR paradigm. We present various techniques that ensure the DP verifier has both a low false positive rate (for privacy guarantee) and a low false negative rate (for utility guarantee, i.e., making the EVR and the original mechanism as similar as possible).

A Monte Carlo-based DP Accountant. We further propose a new MC-based approach for DP accounting, which we call the MC accountant . It utilizes similar MC techniques as in privacy verification. We show that the MC accountant achieves several advantages over existing privacy accounting methods. In particular, we demonstrate that MC accountant is efficient for online privacy accounting , a realistic scenario for privacy practitioners where one wants to update the estimate on privacy guarantee whenever executing a new mechanism.

Figure 2 gives an overview of the proposed EVR paradigm as well as this paper's contributions.

2 Privacy Accounting: a Mean Estimation/Bounding Problem

In this section, we review relevant concepts and introduce privacy accounting as a mean estimation/bounding problem.

Symbols and notations. We use D,D ′ ∈ N X to denote two datasets with an unspecified size over space X . We call two datasets D and D ′ adjacent (denoted as D ∼ D ′ ) if we can construct one by adding/removing one data point from the other. We use P, Q to denote random variables. We also overload the notation and denote P ( · ) , Q ( · ) the density function of P, Q .

Differential privacy and its equivalent characterizations. Having established the notations, we can now proceed to formally define differential privacy.

Definition 1 (Differential Privacy [13]) . For ε, δ ≥ 0 , a randomized algorithm M : N X → Y is ( ε, δ ) -differentially private if for every pair of adjacent datasets D ∼ D ′ and for every subset of possible outputs E ⊆ Y , we have Pr M [ M ( D ) ∈ E ] ≤ e ε Pr M [ M ( D ′ ) ∈ E ] + δ .

One can alternatively define differential privacy in terms of the maximum possible divergence between the output distribution of any pair of M ( D ) and M ( D ′ ) .

Lemma 2 ([5]) . A mechanism M is ( ε, δ ) -DP iff sup D ∼ D ′ E e ε ( M ( D ) ∥M ( D ′ )) ≤ δ , where E γ ( P ∥ Q ) := E o ∼ Q [( P ( o ) Q ( o ) -γ ) + ] & ( a ) + := max( a, 0) .

E γ is usually referred as Hockey-Stick (HS) Divergence in the literature. For every mechanism M and every ε ≥ 0 , there exists a smallest δ such that M is ( ε, δ ) -DP. Following the literature [44, 2], we formalize such a δ as a function of ε .

Definition 3 (Optimal Privacy Curve) . The optimal privacy curve of a mechanism M is the function δ M : R + → [0 , 1] s.t. δ M ( ε ) := sup D ∼ D ′ E e ε ( M ( D ) ∥M ( D ′ )) .

Dominating Distribution Pair and Privacy Loss Random Variable (PRV). It is computationally infeasible to find δ M ( ε ) by computing E e ε ( M ( D ) ∥M ( D ′ )) for all pairs of adjacent dataset D and D ′ . A mainstream strategy in the literature is to find a pair of distributions ( P, Q ) that dominates all ( M ( D ) , M ( D ′ )) in terms of the Hockey-Stick divergence. This results in the introduction of dominating distribution pair and privacy loss random variable (PRV).

Definition 4 ([44]) . A pair of distributions ( P, Q ) is a pair of dominating distributions for M under adjacent relation ∼ if for all γ ≥ 0 , sup D ∼ D ′ E γ ( M ( D ) ∥M ( D ′ )) ≤ E γ ( P ∥ Q ) . If equality is achieved for all γ ≥ 0 , then we say ( P, Q ) is a pair of tightly dominating distributions for M . Furthermore, we call Y := log ( P ( o ) Q ( o ) ) , o ∼ P the privacy loss random variable (PRV) of M associated with dominating distribution pair ( P, Q ) .

Zhu et al. [44] shows that all mechanisms have a pair of tightly dominating distributions. Hence, we can alternatively characterize the optimal privacy curve as δ M ( ε ) = E e ε ( P ∥ Q ) for the tightly dominating pair ( P, Q ) , and we have δ M ( ε ) ≤ E e ε ( P ∥ Q ) if ( P, Q ) is a dominating pair that is not necessarily tight. The importance of the concept of PRV comes from the fact that we can write

E

e

ε

(

P

∥

Q

)

as an expectation over it:

E

e

ε

(

P

∥

Q

) =

E

Y

1

-

e

+

. Thus, one can bound

δ

M

(

ε

)

by first identifying M 's dominating pair distributions as well as the associated PRV Y , and then computing this expectation. Such a formulation allows us to bound δ M ( ε ) without enumerating over

[ (

ε

-

Y

)

]

all adjacent D and D ′ . For notation convenience, we denote δ Y ( ε ) := E Y [ ( 1 -e ε -Y ) + ] . Clearly, δ M ≤ δ Y . If ( P, Q ) is a tightly dominating pair for M , then δ M = δ Y .

Privacy Accounting as a Mean Estimation/Bounding Problem. Privacy accounting aims to estimate and bound the optimal privacy curve δ M ( ε ) for adaptively composed mechanism M = M 1 ◦ · · · ◦ M k ( D ) . The adaptive composition of two mechanisms M 1 and M 2 is defined as M 1 ◦ M 2 ( D ) := ( M 1 ( D ) , M 2 ( D, M 1 ( D ))) , in which M 2 can access both the dataset and the output of M 1 . Most of the practical privacy accounting techniques are based on the concept of PRV, centered on the following result.

Lemma 5 ([44]) . Let ( P j , Q j ) be a pair of tightly dominating distributions for mechanism M j for j ∈ { 1 , . . . , k } . Then ( P 1 ×··· × P k , Q 1 ×··· × Q k ) is a pair of dominating distributions for M = M 1 ◦· · · ◦ M k , where × denotes the product distribution. Furthermore, the associated privacy loss random variable is Y = ∑ k i =1 Y i where Y i is the PRV associated with ( P i , Q i ) .

Lemma 5 suggests that privacy accounting for DP composition can be cast into a mean estimation/bounding problem where one aims to approximate or bound the expectation in (2) when Y = ∑ k i =1 Y i . Note that while Lemma 5 does not guarantee a pair of tightly dominating distributions for the adaptive composition, it cannot be improved in general, as noted in [10]. Hence, all the current privacy accounting techniques work on δ Y instead of δ M , as Lemma 5 is tight even for non-adaptive composition. Following the prior works, in this paper, we only consider the practical scenarios where Lemma 5 is tight for the simplicity of presentation. That is, we assume δ Y = δ M unless otherwise specified.

Most of the existing privacy accounting techniques can be described as different techniques for such a mean estimation problem. Example-1: FFT-based methods. This line of works (e.g., [19]) discretizes the domain of each Y i and use Fast Fourier Transform (FFT) to speed up the approximation of δ Y ( ε ) . The upper bound is derived through the worst-case error bound for the approximation.

Example-2: CLT-based methods. [7, 41] use CLT to approximate the distribution of Y = ∑ k i =1 Y i as Gaussian distribution. They then use CLT's finite-sample approximation guarantee to derive the upper bound for δ Y ( ε ) .

Remark 6 ( The Importance of Privacy Accounting in Regime δ < 10 -10 ) . The regime where δ < 10 -10 is of significant importance for two reasons. (1) δ serves as an upper bound on the chance of severe privacy breaches, such as complete dataset exposure, necessitating a 'cryptographically small' value, namely, δ < n -ω (1) [14, 40]. (2) Even with the oft-used yet questionable guideline of δ ≈ n -1 or n -1 . 1 , datasets of modern scale, such as JFT-3B [43] or LAION-5B [37], already comprise billions of records, thus rendering small δ values crucial. While we acknowledge that it requires a lot of effort to achieve a good privacy-utility tradeoff even for the current choice of δ ≈ n -1 , it is important to keep such a goal in mind.

3 Estimate-Verify-Release

As mentioned earlier, upper bounds for δ Y ( ε ) are the only valid options for privacy accounting techniques. However, as we have demonstrated in Figure 1, both FFT- and CLT-based methods can provide overly conservative upper bounds in certain regimes. On the other hand, their estimates for δ Y ( ε ) can be very close to the ground truth even though there is no provable guarantee. Therefore, it is highly desirable to develop new techniques that enable the use of privacy parameter estimates instead of overly conservative upper bounds in privacy accounting.

We tackle the problem by introducing a new paradigm for constructing DP mechanisms, which we call Estimate-Verify-Release (EVR). The key component of the EVR is an object called DP verifier (Section 3.1). The full EVR paradigm is then presented in Section 3.2, where the DP verifier is utilized as a building block to guarantee privacy.

3.1 Differential Privacy Verifier

Wefirst formalize the concept of differential privacy verifier , the central element of the EVR paradigm. In informal terms, a DP verifier is an algorithm that attempts to verify whether a mechanism satisfies a specific level of differential privacy.

Definition 7 (Differential Privacy Verifier) . We say a differentially private verifier DPV ( · ) is an algorithm that takes the description of a mechanism M and proposed privacy parameter ( ε, δ est ) as input, and returns True ← DPV ( M , ε, δ est ) if the algorithm believes M is ( ε, δ est ) -DP (i.e., δ est ≥ δ Y ( ε ) where Y is the PRV of M ), and returns False otherwise.

A differential privacy verifier can be imperfect, suffering from both false positives (FP) and false negatives (FN). Typically, FP rate is the likelihood for DPV to accept ( ε, δ est ) when δ est < δ Y ( ε ) . However, δ est is still a good estimate for δ Y ( ε ) by being a small (e.g., <10%) underestimate. To account for this, we introduce a smoothing factor, τ ∈ (0 , 1] , such that δ est is deemed 'should be rejected' only when δ est ≤ τδ Y ( ε ) . A similar argument can be put forth for FN cases where we also introduce a smoothing factor ρ ∈ (0 , 1] . This leads to relaxed notions for FP/FN rate:

Definition 8. We say a DPV 's τ -relaxed false positive rate at ( ε, δ est ) is

We say a DPV 's ρ -relaxed false negative rate at ( ε, δ est ) is

Privacy Verification with DP Accountant. For a composed mechanism M = M 1 ◦ . . . ◦M k , a DP verifier can be easily implemented using any existing privacy accounting techniques. That is, one can execute DP accountant to obtain an estimate or upper bound ( ε, ˆ δ ) of the actual privacy parameter. If δ est < ˆ δ , then the proposed privacy level is rejected as it is more private than what the DP accountant tells; otherwise, the test is passed. The input description of a mechanism M , in this case, can differ depending on the DP accounting method. For Moment Accountant [1], the input description is the upper bound of the moment-generating function (MGF) of the privacy loss random variable for each individual mechanism. For FFT and CLT-based methods, the input description is the cumulative distribution functions (CDF) of the dominating distribution pair of each individual M i .

3.2 EVR: Ensuring Estimated Privacy with DP Verifier

We now present the full paradigm of EVR. As suggested by the name, it contains three steps: (1) Estimate: A privacy parameter ( ε, δ est ) for M is estimated, e.g., based on a privacy auditing or accounting technique. (2) Verify: A DP verifier DPV is used for validating whether mechanism M sat-

Algorithm 1 Estimate-Verify-Release (EVR) Framework

1: Input: M : mechanism. D : dataset. ( ε, δ est ) : an estimated privacy parameter for M .

2: if DPV ( M , ε, δ est ) outputs True then Execute M ( D ) .

3: else Print ⊥ .

isfies ( ε, δ est ) -DP guarantee. (3) Release: If DP verification test is passed, we can execute M as usual; otherwise, the program is terminated immediately. For practical utility, this rejection probability needs to be small when ( ε, δ est ) is an accurate estimation. The procedure is summarized in Algorithm 1.

Given estimated privacy parameter ( ε, δ est ) , we have the privacy guarantee for the EVR paradigm:

We defer the proof to Appendix B. The implication of this result is that, for any estimate of the privacy parameter, one can safely use it as a DPV with a bounded false positive rate would enforce differential privacy. However, this is not enough: an overly conservative DPV that satisfies 0 FP rate but rejects everything would not be useful. When δ est is accurate, we hope the DPV can also achieve a small false negative rate so that the output distributions of EVR and M are indistinguishable. We discuss the instantiation of DPV in Section 4.

4 Monte Carlo Verifier of Differential Privacy

As we can see from Section 3.2, a DP verifier ( DPV ) that achieves a small FP rate is the central element for the EVR framework. In the meanwhile, it is also important that DPV has a low FN rate in

order to maintain the good utility of the EVR when the privacy parameter estimate is accurate. In this section, we introduce an instantiation of DPV based on the Monte Carlo technique that achieves both a low FP and FN rate, assuming the PRV is known for each individual mechanism.

Remark 10 ( Mechanisms where PRV can be derived ) . PRV can be derived for many commonly used DP mechanisms such as the Laplace, Gaussian, and Subsampled Gaussian Mechanism [24, 19]. In particular, our DP verifier applies for DP-SGD, one of the most important application scenarios of privacy accounting. Moreover, the availability of PRV is also the assumption for most of the recently developed privacy accounting techniques (including FFT- and CLT-based methods). The extension beyond these commonly used mechanisms is an important future work in the field.

Remark 11 ( Previous studies on the hardness of privacy verification ) . Several studies [16, 8] have shown that DP verification is an NP-hard problem. However, these works consider the setting where the input description of the DP mechanism is its corresponding randomized Boolean circuits. Some other works [18] show that DP verification is impossible, but this assertion is proved for the black-box setting where the verifier can only query the mechanism. Our work gets around this barrier by providing the description of the PRV of the mechanism as input to the verifier.

4.1 DPV through an MC Estimator for δ Y ( ε )

Recall that most of the recently proposed DP accountants are essentially different techniques for estimating the expectation

where each Y i is the privacy loss random variable Y i = log ( P i ( t ) Q i ( t ) ) for t ∼ P i , and ( P i , Q i ) is a pair of dominating distribution for individual mechanism M i . In the following text, we denote the product distribution P := P 1 × . . . × P k and Q := Q 1 × . . . × Q k . Recall from Lemma 5 that ( P , Q ) is a pair of dominating distributions for the composed mechanism M . For notation simplicity, we denote a vector t := ( t (1) , . . . , t ( k ) ) .

Monte Carlo (MC) technique is arguably one of the most natural and widely used techniques for approximating expectations. Since δ Y ( ε ) is an expectation in terms of the PRV Y , one can apply MC-based technique to estimate it. Given an MC estimator for δ Y ( ε ) , we construct a DPV ( M , ε, δ est ) as shown in Algorithm 2 (instantiated by the Simple MC estimator introduced in Section 4.2). Specifically, we first obtain an estimate ̂ δ from an MC estimator for δ Y ( ε ) . The estimate δ est passes the test if

Algorithm 2 DPV ( M , ε, δ est ) with Simple MC Estimator and Offset Parameter ∆ .

1: Obtain i.i.d. samples { t i } i m =1 from P .

2: Compute ̂ δ = 1 m ∑ m i =1 (1 -e ε -y i ) + with PRV samples y i = log ( P ( t i ) Q ( t i ) ) , i = 1 . . . m .

3: if ̂ δ < δ est τ -∆ then return True .

4: else return False .

̂ δ < δ est τ -∆ , and fails otherwise. The parameter ∆ ≥ 0 here is an offset that allows us to conveniently controls the τ -relaxed false positive rate. We will discuss how to set ∆ in Section 4.4.

In the following contents, we first present two constructions of MC estimators for δ Y ( ε ) in Section 4.2. We then discuss the condition for which our MC-based DPV achieves a certain target FP rate in Section 4.3. Finally, we discuss the utility guarantee for the MC-based DPV in Section 4.4.

4.2 Constructing MC Estimator for δ Y ( ε )

In this section, we first present a simple MC estimator that applies to any mechanisms where we can derive and sample from the dominating distribution pairs. Given the importance of Poisson Subsampled Gaussian mechanism for privacy-preserving machine learning, we further design a more advanced and specialized MC estimator for it based on the importance sampling technique.

Simple Monte Carlo Estimator. One can easily sample from Y by sampling t ∼ P and output log ( P ( t ) Q ( t ) ) . Hence, a straightforward algorithm for estimating (2) is the Simple Monte Carlo (SMC) algorithm, which directly samples from the privacy random variable Y . We formally define it here.

Definition 12 (Simple Monte Carlo (SMC) Estimator) . We denote ̂ δ m MC ( ε ) as the random variable of SMC estimator for δ Y ( ε ) with m samples, i.e., ̂ δ m MC ( ε ) := 1 m ∑ m i =1 (1 -e ε -y i ) + for y 1 , . . . , y m i.i.d. sampled from Y .

Importance Sampling Estimator for Poisson Subsampled Gaussian (Overview). As δ Y ( ε ) is usually a tiny value ( 10 -5 or even cryptographically small), it is likely that by naive sampling from Y , almost all of the samples in { (1 -e ε -y i ) + } i m =1 are just 0s! That is, the i.i.d. samples { y i } i m =1 from Y can rarely exceed ε . To further improve the sample efficiency, one can potentially use more advanced MC techniques such as Importance Sampling or MCMC. However, these advanced tools usually require additional distributional information about Y and thus need to be developed case-by-case.

Poisson Subsampled Gaussian mechanism is the main workhorse behind the DP-SGD algorithm [1]. Given its important role in privacy-preserving ML, we derive an advanced MC estimator for it based on the Importance Sampling technique. Importance Sampling (IS) is a classic method for rare event simulation [39]. It samples from an alternative distribution instead of the distribution of the quantity of interest, and a weighting factor is then used for correcting the difference between the two distributions. The specific design of alternative distribution is complicated and notation-heavy, and we defer the technical details to Appendix C. At a high level, we construct the alternative sampling distribution based on the exponential tilting technique and derive the optimal tilting parameter such that the corresponding IS estimator approximately achieves the smallest variance. Similar to Definition 12, we use ̂ δ m IS to denote the random variable of importance sampling estimator with m samples.

4.3 Bounding FP Rate

We now discuss the FP guarantee for the DPV instantiated by ̂ δ m MC and ̂ δ m IS we developed in the last section. Since both estimators are unbiased, by Law of Large Number, both ̂ δ m MC and ̂ δ m IS converge to δ Y ( ε ) almost surely as m →∞ , which leads a DPV with perfect accuracy. Of course, m cannot go to ∞ in practice. In the following, we derive the required amount of samples m for ensuring that τ -relaxed false positive rate is smaller than δ est /τ for ̂ δ m MC and ̂ δ m IS . We use ̂ δ MC (or ̂ δ IS ) as an abbreviation for ̂ δ 1 MC (or ̂ δ 1 IS ), the random variable for a single draw of sampling. We state the theorem for ̂ δ m MC , and the same result for ̂ δ m IS can be obtained by simply replacing ̂ δ MC with ̂ δ IS . We use FP MC to denote the FP rate for DPV implemented by SMC estimator.

The proof is based on Bennett's inequality and is deferred to Appendix D. This result suggests that, to improve the computational efficiency of MC-based DPV (i.e., tighten the number of required samples), it is important to tightly bound E [( ̂ δ MC ) 2 ] (or E [( ̂ δ IS ,θ ) 2 ] ), the second moment of ̂ δ MC (or ̂ δ IS ).

Bounding the Second-Moment of MC Estimators (Overview). For clarity, we defer the notationheavy results and derivation of the upper bounds for E [( ̂ δ MC ) 2 ] and E [( ̂ δ IS ) 2 ] to Appendix E. Our high-level idea for bounding E [( ̂ δ MC ) 2 ] is through the RDP guarantee for the composed mechanism M . This is a natural idea since converting RDP to upper bounds for δ Y ( ε ) - the first moment of ̂ δ MC -is a well-studied problem [30, 9, 3]. Bounding E [( ̂ δ IS ) 2 ] is highly technically involved.

4.4 Guaranteeing Utility

Overall picture so far. Given the proposed privacy parameter ( ε, δ est ) , a tolerable degree of underestimation τ , and an offset parameter ∆ , one can now compute the number of samples m required for the MC-based DPV such that τ -relaxed FP rate to be ≤ δ est /τ based on the results from Section 4.3 and Appendix E. We have not yet discussed the selection of the hyperparameter ∆ . An appropriate ∆ is important for the utility of MC-based DPV . That is, when δ est is not too smaller than δ Y ( ε ) , the probability of being rejected by DPV should stay negligible. If we set ∆ →∞ , the DPV simply rejects everything, which achieves 0 FP rate (and with m = 0 ) but is not useful at all!

Formally, the utility of a DPV is quantified by the ρ -relaxed false negative (FN) rate (Definition 8). While one may be able to bound the FN rate through concentration inequalities, a more convenient

way is to pick an appropriate ∆ such that FN DPV is approximately smaller than FP DPV . After all, FP DPV already has to be a small value ≤ δ est /τ for privacy guarantee. The result is stated informally in the following (holds for both ̂ δ MC and ̂ δ IS ), and the involved derivation is deferred to Appendix F.

Theorem 14 (Informal) . When ∆ = 0 . 4 (1 /τ -1 /ρ ) δ est , then FN MC ( ε, δ est ; ρ ) ⪅ FP MC ( ε, δ est ; τ ) .

Therefore, by setting ∆ = 0 . 4 (1 /τ -1 /ρ ) δ est , one can ensure that FN MC ( ε, δ est ; ρ ) is also (approximately) upper bounded by Θ( δ est /τ ) . Moreover, in Appendix, we empirically show that the FP rate is actually a very conservative bound for the FN rate. Both τ and ρ are selected based on the tradeoff between privacy, utility, and efficiency.

The pseudocode of privacy verification for DP-SGD is summarized in Appendix G.

5 Monte Carlo Accountant of Differential Privacy

The Monte Carlo estimators ̂ δ MC and ̂ δ IS described in Section 4.2 are used for implementing DP verifiers. One may already realize that the same estimators can also be utilized to directly implement a DP accountant which estimates δ Y ( ε ) . It is important to note that with the EVR paradigm, DP accountants are no longer required to derive a strict upper bound for δ Y ( ε ) . We refer to the technique of estimating δ Y ( ε ) using the MC estimators as Monte Carlo accountant .

Finding ε for a given δ . It is straightforward to implement MC accountant when we fix ε and compute for δ Y ( ε ) . In practice, privacy practitioners often want to do the inverse: finding ε for a given δ , which we denote as ε Y ( δ ) . Similar to the existing privacy accounting methods, we use binary search to find ε Y ( δ ) (see Algorithm 3). Specifically, after generating PRV samples { y i } i m =1 , we simply need to find the ε such that 1 m ∑ m i =1 (1 -e ε -y i ) + = δ . We do not

Algorithm 3 MCAccountant for ε Y ( δ ) .

1: Obtain PRV samples { y i } i m =1 with either Simple MC or Importance Sampling.

2: Binary search ε such that 1 m ∑ m i =1 (1 -e ε -y i ) + = δ . 3: Return ε .

need to generate new PRV samples for different ε we evaluate during the binary search; hence the additional binary search is computationally efficient.

Number of Samples for MC Accountant. Compared with the number of samples required for achieving the FP guarantee in Section 4.3, one may be able to use much fewer samples to obtain a decent estimate for δ Y ( ε ) , as the sample complexity bound derived based on concentration inequality may be conservative. Many heuristics for guiding the number of samples in MC simulation have been developed (e.g., Wald confidence interval) and can be applied to the setting of MC accountants.

Compared with FFT-based and CLT-based methods, MC accountant exhibits the following strength:

(1) Accurate δ Y ( ε ) estimation in all regimes. As we mentioned earlier, the state-of-the-art FFTbased method [19] fails to provide meaningful bounds due to computational limitations when the true value of δ Y ( ε ) is small. In contrast, the simplicity of the MC accountant allows us to accurately estimate δ Y ( ε ) in all regimes.

(2) Short clock runtime & Easy GPU acceleration. MC-based techniques are well-suited for parallel computing and GPU acceleration due to their nature of repeated sampling. One can easily utilize PyTorch's CUDA functionality (e.g., torch.randn(size=(k,m)).cuda()*sigma+mu ) to significantly boost the computational efficiency for sampling from common distributions such as Gaussian. In Appendix H, we show that when using one NVIDIA A100 GPU, the runtime time of sampling Gaussian mixture (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) can be improved by 10 3 times compared with CPU-only scenario.

(3) Efficient online privacy accounting . When training ML models with DP-SGD or its variants, a privacy practitioner usually wants to compute a running privacy leakage for every training iteration, and pick the checkpoint with the best utility-privacy tradeoff. This involves estimating δ Y ( i ) ( ε ) for every i = 1 , . . . , k , where Y ( i ) := ∑ i j =1 Y j . We refer to such a scenario as online privacy accounting 2 . MCaccountant is especially efficient for online privacy accounting. When estimating

δ Y ( i ) ( ε ) , one can re-use the samples previously drawn from Y 1 , . . . , Y i -1 that were used for estimating privacy loss at earlier iterations.

These advantages are justified empirically in Section 6 and Appendix H.

6 Numerical Experiments

In this section, we conduct numerical experiments to illustrate (1) EVR paradigm with MC verifiers enables a tighter privacy analysis, and (2) MCaccountant achieves state-of-the-art performance in privacy parameter estimation.

6.1 EVR vs Upper Bound

To illustrate the advantage of the EVR paradigm compared with directly using a strict upper bound for privacy parameters, we take the current state-of-the-art DP accountant, the FFT-based method from [19] as the example.

EVR provides a tighter privacy guarantee. Recall that in Figure 1, FFT-based method provides vacuous bound when the ground-truth δ Y ( ε ) < 10 -10 . Under the same hyperparameter setting, Figure 3 (a) shows the privacy bound of the EVR paradigm where the δ est are FFT's estimates. We use the Importance Sampling estimator ̂ δ IS for DP verification. We experiment with different values of τ . A higher value of τ leads to tighter privacy guarantee but longer runtime. For fair comparison, the EVR's output distribution needs to be almost indistinguishable from the original mechanism. We set ρ = (1+ τ ) / 2 and set ∆ according to the heuristic from Theorem 14. This guarantees that, as long as the estimate of δ est from FFT is not a big underestimation (i.e., as long as δ est ≥ ρδ Y ( ε ) ), the failure probability of the EVR paradigm is

10-8

102

10-9

FFT-EVR(T=0.99)

10-10

S

FFT-EVR(T=0.95)

6

itim

101

FFT-EVR(T=0.9)

10-11

Exact

FFT-upp

R

in

10-12

FFT-EVR(T=0.9)

FFT-EVR (T = 0.6)

100

10-13

FFT-EVR(T=0.3)

(a)

2.8

3.0

3.2

3.4

3.6

(b)

2.8

3.0

3.2

3.4

3.6

E

E

negligible ( O ( δ Y ( ε )) ). The 'FFT-EVR' curve in Figure 3 (a) is essentially the 'FFT-est' curve in Figure 1 scaled up by 1 /τ . As we can see, EVR provides a significantly better privacy analysis in the regime where the 'FFT-upp' is unmeaningful ( δ < 10 -10 ).

EVR incurs little extra runtime. In Figure 3 (b), we plot the runtime of the Importance Sampling verifier in Figure 3 (b) for different τ ≥ 0 . 9 . Note that for τ > 0 . 9 , the privacy curves are indistinguishable from 'Exact' in Figure 3 (a). The runtime of EVR is determined by the number of samples required to achieve the target τ -relaxed FP rate from Theorem 13. Smaller τ leads to faster DP verification. As we can see, even when τ = 0 . 99 , the runtime of DP verification in the EVR is < 2 minutes. This is attributable to the sample-efficient IS estimator and GPU acceleration.

EVR provides better privacy-utility tradeoff for Privacy-preserving ML with minimal time consumption. To further underscore the superiority of the EVR paradigm in practical applications, we illustrate the privacy-utility tradeoff curve when finetuning on CIFAR100 dataset with DP-SGD. As shown in Figure 4, the EVR paradigm provides a lower test error across all privacy budget ε compared with the traditional upper bound method. For instance, it achieves around 7% (relative) error reduction when ε = 0 . 6 . The runtime time required for privacy verification is less than < 10 -10 seconds for all ε , which is negligible compared to the training time. We provide additional experimental results in Appendix H.

6.2 MCAccountant

We evaluate the MC Accountant proposed in Section 5. We focus on privacy accounting for the composition of Poisson Subsampled Gaussian mechanisms, the algorithm behind the famous DP-SGD algorithm [1]. The mechanism is specified by the noise magnitude σ and subsampling rate q .

Settings. We consider two practical scenarios of privacy accounting: (1) Offline accounting which aims at estimating δ Y ( k ) ( ε ) , and (2) Online accounting which aims at estimating δ Y ( i ) ( ε ) for all i = 1 , . . . , k . For space constraint, we only show the results of online accounting here, and defer the results for offline accounting to Appendix H. Metric: Relative Error. To easily and fairly evaluate the performance of privacy parameter estimation, we compute the almost exact (yet computationally expensive) privacy parameters as the ground-truth value. The ground-truth value allows us to compute the relative error of an estimate of privacy leakage. That is, if the corresponding ground-truth of an estimate ̂ δ is δ , then the relative error r err = | ̂ δ -δ | /δ . Implementation. For MC accountant, we use the IS estimator described in Section 4.2. For baselines, in addition to the FFT-based and CLT-based method we mentioned earlier, we also examine AFA [44] and GDP accountant [7]. For a fair comparison, we adjust the number of samples for MC accountant so that the runtime of MC accountant and FFT is comparable. Note that we compared with the privacy parameter estimates instead of upper

15

14

13

Error

12

Test

11

10

FFT-upp

FFT-EVR(T=0.95)

9

0.4

0.6

0.8

1.0

1.2

bounds from the baselines. Detailed settings for both MC accountant and the baselines are provided in Appendix H.

Results for Online Accounting: MC accountant is both more accurate and efficient. Figure 5 (a) shows the online accounting results for ( σ, δ, q ) = (1 . 0 , 10 -9 , 10 -3 ) . As we can see, MC accountant outperforms all of the baselines in estimating ε Y ( δ ) . The sharp decrease in FFT at approximately 250 steps is due to the transition of FFT's estimates from underestimating before this point to overestimating after. Figure 5 (b) shows that MC accountant is around 5 times faster than FFT, the baseline with the best performance in (a). This showcases the MC accountant's efficiency and accuracy in online setting.

7 Conclusion & Limitations

FFT

GDP

GDP-Edgeworth

MC (IS)

RelativeErrorinEstimatingε

CumulativeRuntime

100

104

103

Error

10-2

102

Relative

10-4

Time

101

100

10-6

10-1

10-2

10-8

0

250

500

750

1000

0

250

500

750

1000

(a)

Number of Compositionsk

(b)

Number ofCompositionsk

This paper tackles the challenge of deriving provable privacy leakage upper bounds in privacy accounting. We present the estimate-verify-release (EVR) paradigm which enables the safe use of privacy parameter estimate. Limitations. Currently, our MC-based DP verifier and accountant require known and efficiently samplable dominating pairs and PRV for the individual mechanism. Fortunately, this applies to commonly used mechanisms such as Gaussian mechanism and DP-SGD. Generalizing MC-based DP verifier and accountant to other mechanisms is an interesting future work.

Acknowledgments

This work was supported in part by the National Science Foundation under grants CNS-2131938, CNS-1553437, CNS-1704105, the ARL's Army Artificial Intelligence Innovation Institute (A2I2), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, and Princeton E-ffiliates Award, Amazon-Virginia Tech Initiative in Efficient and Robust Machine Learning, and Princeton's Gordon Y. S. Wu Fellowship. We are grateful to anonymous reviewers at NeurIPS for their valuable feedback.

References

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pages 308-318, 2016.

Wael Alghamdi, Juan Felipe Gomez, Shahab Asoodeh, Flavio Calmon, Oliver Kosut, and Lalitha Sankar. The saddle-point method in differential privacy. In Proceedings of the 40th International Conference on Machine Learning , pages 508-528, 2023.

Shahab Asoodeh, Jiachun Liao, Flavio P Calmon, Oliver Kosut, and Lalitha Sankar. Three variants of differential privacy: Lossless conversion and applications. IEEE Journal on Selected Areas in Information Theory , 2(1):208-222, 2021.

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Gilles Barthe and Federico Olmedo. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In International Colloquium on Automata, Languages, and Programming , pages 49-60. Springer, 2013.

Benjamin Bichsel, Samuel Steffen, Ilija Bogunovic, and Martin Vechev. Dp-sniper: Black-box discovery of differential privacy violations using classifiers. In 2021 IEEE Symposium on Security and Privacy (SP) , pages 391-409. IEEE, 2021.

Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J Su. Deep learning with gaussian differential privacy. Harvard data science review , 2020(23), 2020.

Mark Bun, Marco Gaboardi, and Ludmila Glinskih. The complexity of verifying boolean programs as differentially private. In 2022 IEEE 35th Computer Security Foundations Symposium (CSF) , pages 396-411. IEEE, 2022.

Clément L Canonne, Gautam Kamath, and Thomas Steinke. The discrete gaussian for differential privacy. Advances in Neural Information Processing Systems , 33:15676-15688, 2020.

Jinshuo Dong, Aaron Roth, and Weijie J Su. Gaussian differential privacy. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(1):3-37, 2019.

Vadym Doroshenko, Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Connect the dots: Tighter discrete approximations of privacy loss distributions. Proceedings on Privacy Enhancing Technologies , 4:552-570, 2022.

Cynthia Dwork and Jing Lei. Differential privacy and robust statistics. In Proceedings of the forty-first annual ACM symposium on Theory of computing , pages 371-380, 2009.

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference , pages 265-284. Springer, 2006.

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science , 9(3-4):211-407, 2014.

Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science , pages 51-60. IEEE, 2010.

Marco Gaboardi, Kobbi Nissim, and David Purser. The complexity of verifying loop-free programs as differentially private. In 47th International Colloquium on Automata, Languages, and Programming (ICALP 2020) . Schloss Dagstuhl-Leibniz-Zentrum für Informatik, 2020.

Badih Ghazi, Pritish Kamath, Ravi Kumar, and Pasin Manurangsi. Faster privacy accounting via evolving discretization. In International Conference on Machine Learning , pages 7470-7483. PMLR, 2022.

Anna C Gilbert and Audra McMillan. Property testing for differential privacy. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) , pages 249-258. IEEE, 2018.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. Numerical composition of differential privacy. Advances in Neural Information Processing Systems , 34:11631-11642, 2021.

Daniele Gorla, Louis Jalouzot, Federica Granese, Catuscia Palamidessi, and Pablo Piantanida. On the (im) possibility of estimating various notions of differential privacy. arXiv preprint arXiv:2208.14414 , 2022.

Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems , 33:22205-22216, 2020.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. The composition theorem for differential privacy. In International conference on machine learning , pages 1376-1385. PMLR, 2015.

Antti Koskela and Antti Honkela. Computing differential privacy guarantees for heterogeneous compositions using fft. CoRR , abs/2102.12412, 2021.

Antti Koskela, Joonas Jälkö, and Antti Honkela. Computing tight differential privacy guarantees using fft. In International Conference on Artificial Intelligence and Statistics , pages 2560-2569. PMLR, 2020.

Antti Koskela, Joonas Jälkö, Lukas Prediger, and Antti Honkela. Tight differential privacy for discrete-valued mechanisms and for the subsampled gaussian mechanism using fft. In International Conference on Artificial Intelligence and Statistics , pages 3358-3366. PMLR, 2021.

Xiyang Liu and Sewoong Oh. Minimax optimal estimation of approximate differential privacy on neighboring databases. Advances in neural information processing systems , 32, 2019.

Fred Lu, Joseph Munoz, Maya Fuchs, Tyler LeBlond, Elliott V Zaresky-Williams, Edward Raff, Francis Ferraro, and Brian Testa. A general framework for auditing differentially private machine learning. In Advances in Neural Information Processing Systems , 2022.

Yun Lu, Yu Wei, Malik Magdon-Ismail, and Vassilis Zikas. Eureka: A general framework for black-box differential privacy estimators. Cryptology ePrint Archive , 2022.

Saeed Mahloujifar, Alexandre Sablayrolles, Graham Cormode, and Somesh Jha. Optimal membership inference bounds for adaptive composition of sampled gaussian mechanisms. arXiv preprint arXiv:2204.06106 , 2022.

Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF) , pages 263-275. IEEE, 2017.

Jack Murtagh and Salil Vadhan. The complexity of computing the optimal composition of differential privacy. In Theory of Cryptography Conference , pages 157-175. Springer, 2016.

Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning. arXiv preprint arXiv:2302.07956 , 2023.

Milad Nasr, Shuang Songi, Abhradeep Thakurta, Nicolas Papernot, and Nicholas Carlin. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on security and privacy (SP) , pages 866-882. IEEE, 2021.

Ashwinee Panda, Xinyu Tang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal. Dp-raft: A differentially private recipe for accelerated fine-tuning. arXiv preprint arXiv:2212.04486 , 2022.

Rachel Redberg, Yuqing Zhu, and Yu-Xiang Wang. Generalized ptr: User-friendly recipes for data-adaptive algorithms with differential privacy. In International Conference on Artificial Intelligence and Statistics , pages 3977-4005. PMLR, 2023.

Ryan M Rogers, Aaron Roth, Jonathan Ullman, and Salil Vadhan. Privacy odometers and filters: Pay-as-you-go composition. Advances in Neural Information Processing Systems , 29, 2016.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. arXiv preprint arXiv:2305.08846 , 2023.

Surya T Tokdar and Robert E Kass. Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics , 2(1):54-60, 2010.

Salil Vadhan. The complexity of differential privacy. Tutorials on the Foundations of Cryptography: Dedicated to Oded Goldreich , pages 347-450, 2017.

Hua Wang, Sheng Gao, Huanyu Zhang, Milan Shen, and Weijie J Su. Analytical composition of differential privacy via the edgeworth accountant. arXiv preprint arXiv:2206.04236 , 2022.

Jiachen T Wang, Saeed Mahloujifar, Shouda Wang, Ruoxi Jia, and Prateek Mittal. Renyi differential privacy of propose-test-release and applications to private and robust machine learning. Advances in Neural Information Processing Systems , 35:38719-38732, 2022.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104-12113, 2022.

Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. Optimal accounting of differential privacy via characteristic function. In International Conference on Artificial Intelligence and Statistics , pages 4782-4817. PMLR, 2022.

A Extended Related Work

In this section, we review the related works in privacy accounting, privacy verification, privacy auditing, and we also discuss the connection between our EVR paradigm and the famous ProposeTest-Release (PTR) paradigm.

Privacy Accounting. Early privacy accounting techniques such as Advanced Composition Theorem [15] only make use of the privacy parameters of the individual mechanisms, which bounds δ M ( ε ) in terms of the privacy parameter ( ε i , δ i ) for each M i , i = 1 , . . . , k . The optimal bound for δ M ( ε ) under this condition has been derived [22, 31]. However, the computation of the optimal bound is #P-hard in general. Bounding δ M ( ε ) only in terms of ( ε i , δ i ) is often sub-optimal for many commonly used mechanisms [31]. This disadvantage has spurred many recent advances in privacy accounting by making use of more statistical information from the specific mechanisms to be composed [1, 30, 24, 7, 23, 25, 19, 44, 17, 11, 41, 2]. All of these works can be described as approximating the expectation in (2) when Y = ∑ k i =1 Y i . For instance, the line of [24, 7, 23, 25, 19, 17, 11] discretize the domain of each Y i and use Fast Fourier Transform (FFT) in order to speed up the approximation of δ Y ( ε ) . [44] tracks the characteristic function of the privacy loss random variables for the composed mechanism and still requires discretization when the mechanisms do not have closed-form characteristic functions. The line of [7, 41] uses Central Limit Theorem (CLT) to approximate the distribution of Y = ∑ k i =1 Y i as Gaussian distribution and uses the finite-sample bound to derive the strict upper bound for δ Y ( ε ) . We also note that [29] also uses Monte Carlo approaches to calculate optimal membership inference bounds. They use a similar Simple MC estimator as the one in Section 4.2. Although their Monte Carlo approach is similar, their error analysis only works for large values of δ ( δ ≈ 0 . 5) as they use sub-optimal concentration bounds.

Privacy Verification. As we mentioned in Remark 11, some previous works have also studied the problem of privacy verification. Most of the works consider either 'white-box setting' where the input description of the DP mechanism is its corresponding randomized Boolean circuits [16, 8]. Some other works consider an even more stringent 'black-box setting' where the verifier can only query the mechanism [18, 26, 6, 20, 28]. In contrast, our MC verifier is designed specifically for those mechanisms where the PRV can be derived, which includes many commonly used mechanisms such as the Subsampled Gaussian mechanism.

Privacy verification via auditing. Several heuristics have tried to perform DP verification, forming a line of work called auditing differential privacy [21, 33, 27, 32, 38]. Specifically, these techniques can verify a claimed privacy parameter by computing a lower bound for the actual privacy parameter, and comparing that with the claimed privacy parameter. The input description of mechanism M for DPV , in this case, is a black-box oracle M ( · ) , where the DPV makes multiple queries to M ( · ) and estimates the actual privacy leakage. Privacy auditing techniques can achieve 100% accuracy when δ est > δ Y ( ε ) (or 0 ρ -FN rate for any ρ ≤ 1 ), as the computed lower bound is guaranteed to be smaller than δ est . However, when δ est lies between δ Y ( ε ) and the computed lower bound, the DP verification will be wrong. Moreover, such techniques do not have a guarantee for the lower bound's tightness.

Remark 15 (Connection between our EVR paradigm and the Propose-Test-Release (PTR) paradigm [12]) . PTR is a classic differential privacy paradigm introduced over a decade ago by [12], and is being generalized in [35, 42]. At a high level, PTR checks if releasing the query answer is safe with a certain amount of randomness (in a private way). If the test is passed, the query answer is released; otherwise, the program is terminated. PTR shares a similar underlying philosophy with our EVR paradigm. However, they are fundamentally different in terms of implementation. The verification step in EVR is completely independent of the dataset. In contrast, the test step in PTR measures the privacy risks for the mechanism M on a specific dataset D , which means that the test itself may cause additional privacy leakage. One way to think about the difference is that EVR asks 'whether M is private', while PTR asks 'whether M ( D ) is private'.

B Proofs for Privacy

Theorem 9. Algorithm 1 is ( ε, δ est /τ ) -DP for any τ > 0 if FP DPV ( ε, δ est ; τ ) ≤ δ est /τ .

Proof. For any mechanism M , we denote A as the event that δ est ≥ τδ Y ( ε ) , and indicator variable B = ✶ [ DPV ( M , ε, δ est ; τ ) = True ] . Note that event A implies M is ( δ est /τ ) -DP. Thus, we know that

For notation simplicity, we also denote p FP := Pr[ B = 1 | ¯ A ] , and p TP := Pr[ B = 1 | A ] . For any possible event S ,

where in the first inequality, we use the definition of differential privacy. Therefore, M aug is ( ε, max ( δ est p TP τ , p FP )) -DP. By assumption of p FP ≤ FP DPV ( ε, δ est ; τ ) ≤ δ est /τ , we reach the conclusion.

C Importance Sampling via Exponential Tilting

Notation Review. Recall that most of the recently proposed DP accountants are essentially different techniques for estimating the expectation

where each Y i is the privacy loss random variable Y i = log ( P i ( t ) Q i ( t ) ) for t ∼ P i , and ( P i , Q i ) is a pair of dominating distribution for individual mechanism M i . In the following text, we denote the product distribution P := P 1 × . . . × P k and Q := Q 1 × . . . × Q k . Recall from Lemma 5 that ( P , Q ) is a pair of dominating distributions for the composed mechanism M . For notation simplicity, we denote a vector t := ( t (1) , . . . , t ( k ) ) . We slightly abuse the notation and write y ( t ; P, Q ) := log ( P ( t ) Q ( t ) ) .

Note that y ( t ; P , Q ) = ∑ k i =1 y ( t ( i ) ; P i , Q i ) . When the context is clear, we omit the dominating pairs and simply write y ( t ) .

Dominating Distribution Pairs for Poisson Subsampled Gaussian Mechanisms. The dominating distribution pair for Poisson Subsampled Gaussian Mechanisms is a well-known result.

Lemma 16. For Poisson Subsampled Gaussian mechanism with sensitivity C , noise variance C 2 σ 2 , and subsampling rate q , one dominating pair ( P, Q ) is Q := N (0 , σ 2 ) and P := (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) .

That is, Q is just a 1-dimensional standard Gaussian distribution, and P is a convex combination between standard Gaussian and a Gaussian centered at 1 .

Remark 17 (Dominating pair supported on higher dimensional space) . The cost of our approach would not increase (in terms of the number of samples) even if the dominating pair is supported in a high dimensional space. For Monte Carlo estimate, we can see from Hoeffding's inequality that the expected error rate of estimation is independent of the dimension of the support set of dominating distribution pairs. This means the number of samples we need to ensure a certain confidence interval is independent of the dimension. However, we should also note that although the number of samples does not change, the sampling process itself might be more costly for higher dimensional spaces.

C.1 Importance Sampling for the Composition of Poisson Subsampled Gaussian Mechanisms

Importance Sampling (IS) is a classic method for rare event simulation. It samples from an alternative distribution instead of the distribution of the quantity of interest, and a weighting factor is then used for correcting the difference between the two distributions. Specifically, we can re-write the expression for δ Y ( ε ) as follows:

where P ′ is the alternative distribution up to the user's choice. From Equation (2), one can construct an unbiased importance sampling estimator for δ Y ( ε ) by sampling from P ′ . In this section, we develop a P ′ for estimating δ Y ( ε ) when composing identically distributed Poisson subsampled Gaussian mechanisms , which is arguably the most important DP mechanism nowadays due to its application in differentially private stochastic gradient descent.

Exponential tilting is a common way to construct alternative sampling distribution for IS. The exponential tilting of a distribution P is defined as

where M P ( θ ) := E t ∼ P [ e θt ] is the moment generating function for P . Such a transformation is especially convenient for distributions from the exponential family. For example, for normal distribution N ( µ, σ 2 ) , the tilted distribution is N ( µ + θσ 2 , σ 2 ) , which is easy to sample from.

Without the loss of generality, we consider Poisson Subsampled Gaussian mechanism with sensitivity 1, noise variance σ 2 , and subsampling rate q . Recall from Lemma 16 that the dominating pair in this case is Q := N (0 , σ 2 ) and P := (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) . For notation simplicity, we denote P 0 := N (1 , σ 2 ) , and thus P = (1 -q ) Q + qP 0 . Since each individual mechanism is the same, P = P × . . . × P and Q = Q × . . . × Q . The exponential tilting of P with parameter θ is P θ := (1 -q ) N ( θσ 2 , σ 2 ) + q N (1 + θσ 2 , σ 2 ) . We propose the following importance sampling estimator for δ Y ( ε ) based on exponential tilting.

Definition 18 (Importance Sampling Estimator for Subsampled Gaussian Composition) . Let the alternative distribution

with θ = 1 / 2 + σ 2 log ( exp( ε ) -(1 -q ) q ) . Given a random draw t ∼ P θ , an unbiased sample for δ Y ( ε ) is ( 1 -e ε -y ( t ; P , Q ) ) + ( 1 k ∑ k i =1 P θ ( t i ) P ( t i ) ) -1 . We denote ̂ δ m IS ,θ ( ε ) as the random variable of the corresponding importance sampling estimator with m samples.

We defer the formal justification of the choice of θ to Appendix C.2. We first give the intuition for why we choose such an alternative distribution P θ .

Intuition for the alternative distribution P θ . It is well-known that the variance of the importance sampling estimator is minimized when the alternative distribution

The distribution of each privacy loss random variable y ( t ; P, Q ) , t ∼ P is light-tailed, which means that for the rare event where y ( t ) = ∑ k i =1 y ( t ( i ) ) > ε , it is most likely that there is only one outlier t ∗ among all { t ( i ) } k i =1 such that y ( t ∗ ) is large (which means that y ( t ) is also large), and all the rest of y ( t ( i ) ) s are small. Hence, a reasonable alternative distribution can just tilt the distribution of a randomly picked t ( i ) , and leave the rest of k -1 distributions to stay the same. Moreover, θ is selected to approximately minimize the variance of ̂ δ IS ,θ (detailed in Appendix C.2). An intuitive way to see it is that y ( θ ) = ε , which significantly improves the probability where y ( t ) ≥ ε while also accounting for the fact that P ( t ) decays exponentially fast as t increases.

We also empirically verify the advantage of the IS estimator over the SMC estimator. The orange curve in Figure 6 (a) shows the empirical estimate of E [( ̂ δ IS ,θ ) 2 ] which quantifies the variance of ̂ δ IS ,θ . Note that θ = 0 corresponds to the case of ̂ δ MC . As we can see, E [( ̂ δ IS ,θ ) 2 ] drops quickly as θ increases, and eventually converges. We can also see that the θ selected by our heuristic in Definition 18 (marked as red '*') approximately corresponds to the lowest point of E [( ̂ δ IS ,θ ) 2 ] . This validates our theoretical justification for the selection of θ .

1.0

AnalyticalBound

10-2

Probability

Empirical Est.

0.8

10-4

0.6

MC (θ=0)

25

10-6

Rejection

0.4

IS-Bound-Optimal (0 =7.5)

IS-Heuristic(0=9.54)

10-8

0.2

0.0

(a)

0.0

2.5

5.0

7.5

10.0

(b)

103

105

107

θ

C.2 Justification of the Heuristic of Choosing Exponential Tilting Parameter θ

The variance of the IS estimator proposed in Definition 18 is given by

Let S ( θ ) := kM P ( θ ) E t ∼ P [ ( 1 -e ε -y ( t ; P , Q ) ) 2 + ( 1 ∑ k i =1 e θt i )] . We aim to find θ that minimizes S ( θ ) .

Note that

To simplify the notation, let b ( t ) := ( 1 -e ε -y ( t ) ) 2 + ∏ k i =1 P 0 ( t i ) .

By setting ∂ ∂θ S ( θ ) = 0 and simplify the expression, we have

As we mentioned earlier, b ( t ) > 0 only when y ( t ) = ∑ k i =1 y ( t ( i ) ) > ε , and for such an event it is most likely that there is only one outlier t ∗ among all { t ( i ) } k i =1 such that y ( t ∗ ) ≈ ε , and all the rest of y ( t ( i ) ) ≈ 0 . Therefore, a simple but surprisingly effective approximation for the RHS of (6) is

for t s.t. y ( t ) = ε . This leads to an approximate solution

D Sample Complexity for Achieving Target False Positive Rate

To derive the sample complexity for achieving a DP verifier with a target false positive rate, we use Bennett's inequality.

Lemma 19 (Bennett's inequality) . Let X 1 , . . . , X n be independent real-valued random variables with finite variance such that X i ≤ b for some b > 0 almost surely for all 1 ≤ i ≤ n . Let ν ≥ ∑ n i =1 E [ X 2 i ] . For any t > 0 , we have

where h ( x ) = (1 + x ) log(1 + x ) -x for x > 0 .

Theorem 13. Suppose E [ ( ̂ δ MC ) 2 ] ≤ ν . DPV instantiated by ̂ δ m MC has bounded τ -relaxed false positive rate FP MC ( ε, δ est ; τ ) ≤ δ est /τ with m ≥ 2 ν ∆ 2 log( τ/δ est ) .

Proof. For any M s.t. δ est < τδ Y ( ε ) , we have

Since ̂ δ MC ∈ [ -1 , 0] , the condition in Bennett's inequality is satisfied with b → 0 + . Hence, (10) can be upper bounded by

By setting exp ( -m ∆ 2 2 ν ) ≤ δ est /τ , we have

E Proofs for Moment Bound

E.1 Overview

As suggested by Theorem 13, a good upper bound for E [ ( ̂ δ MC ) 2 ] (or E [ ( ̂ δ IS ) 2 ] ) is important for the computational efficiency of MC-based DPV .

We upper bound the higher moment of ̂ δ MC through the RDP guarantee for the composed mechanism M . This is a natural idea since converting RDP to upper bounds for δ Y ( ε ) - the first moment of ̂ δ MC -is a well-studied problem [30, 9, 3]. Recall that the RDP guarantee for M is equivalent to a bound for M Y ( λ ) := E [ e λY ] for M 's privacy loss random variable Y for any λ ≥ 0 .

Lemma 20 (RDP-MGF bound conversion [30]) . If a mechanism M is ( α, ε R ( α )) -RDP, then M Y ( λ ) ≤ exp( λε R ( λ +1)) .

We convert an upper bound for M Y ( · ) into the following guarantee for the higher moment of ̂ δ MC = (1 -e ε -Y ) + .

Theorem 21. For any u ≥ 1 , we have

The proof is deferred to Appendix E.2. The basic idea is to find the smallest constant c such that E [( ̂ δ MC ) u ] ≤ cM Y ( λ ) . By setting u = 1 , our result recovers the RDP-DP conversion from [9]. By setting u = 2 , we obtain the desired bound for E [( ̂ δ MC ) 2 ] .

Corollary 22 applies to any mechanisms where the RDP guarantee is available, which covers a wide range of commonly used mechanisms such as (Subsampled) Gaussian or Laplace mechanism. We also note that one may be able to further tighten the above bound similar to the optimal RDP-DP conversion in [3]. We leave this as an interesting future work.

Next, we derive the upper bound for E [( ̂ δ IS ,θ ) 2 ] for Poisson Subsampled Gaussian mechanism.

Theorem 23. For any positive integer λ , and for any a, b ≥ 1 s.t. 1 /a + 1 /b = 1 , we have E [( ̂ δ IS ,θ ) 2 ] ≤ kM P ( θ ) ( E [ ̂ δ 2 a MC ] ) 1 /a · ( bθe -λε ∫ [ r ( λ, x )] k e -bθx dx ) 1 /b where r ( λ, x ) is an upper bound for Pr t ∼ P [max i t i ≤ x, y ( t ) ≥ ε ] detailed in Appendix E.3.

The proof is based on applying Hölder's inequality to the expression of E [( ̂ δ IS ,θ ) 2 ] , and then bound the part where θ is involved: E t ∼ P [ ( 1 k ∑ k i =1 P θ ( t i ) P ( t i ) ) -1 ] . We can bound E [ ̂ δ 2 a MC ] through Theorem 21.

Figure 6 (a) shows the analytical bound from Corollary 22 and Theorem 23 compared with empirically estimated E [( ̂ δ MC ) 2 ] and E [( ̂ δ IS ,θ ) 2 ] . As we can see, the analytical bound for E [( ̂ δ IS ,θ ) 2 ] for relatively large θ is much smaller than the bound for E [( ̂ δ MC ) 2 ] (i.e., θ = 0 in the plot). Moreover, we find that the θ which minimizes the analytical bound (the blue '*') is close to the θ selected by our heuristic (the red '*'). For computational efficiency, one may prefer to use θ that minimizes the analytical bound. However, the heuristically selected θ is still useful when one simply wants to estimate δ Y ( ε ) and does not require the formal, analytical guarantee for the false positive rate. We see such a scenario when we introduce the MC accountant in Section 5. We also note that such a discrepancy (and the gap between the analytical bound and empirical estimate) is due to the use of Hölder's inequality in bounding E [( ̂ δ IS ,θ ) 2 ] . Further tightening the bound for E [( ̂ δ IS ,θ ) 2 ] is important for future work.

E.2 Moment Bound for Simple Monte Carlo Estimator

Theorem 21. For any u ≥ 1 , we have

Proof.

where P λ ( x ) := P ( x ) e λx E [ e λY ] is the exponential tilting of P . Define f ( x, λ ) := (1 -e -x ) u + e -xλ . When x ≤ 0 , f ( x, λ ) = 0 . When x > 0 , the derivative of f with respect to x is

It is easy to see that the maximum of f ( x, λ ) is achieved at x ∗ = log ( u + λ λ ) , and we have

E.3 Moment Bound for Importance Sampling Estimator

In this section, we first prove two possible upper bounds for E [( ̂ δ IS ,θ ) 2 ] in Theorem 24 and Theorem 26. We then combine these two bounds in Theorem 23 via Holder's inequality.

Theorem 24. For any θ ≥ 1 /σ 2 , we have

Overall, we have

Proof.

Note that

Proof. Since log(1 + x ) ≤ x , we have

Hence,

Hence,

where the first inequality is due to Jensen's inequality.

which concludes the proof.

Theorem 26. For any positive integer λ , we have

where r ( λ, x ) is an upper bound for the MGF of privacy loss random variable of truncated Gaussian mixture P | ≤ x .

Proof. Similar to Theorem 24, the goal is to bound

Note that

where t max := max i t i .

Further note that

where (14) is obtained through integration by parts.

Now, as we can see from (15), the question reduces to bound Pr t ∼ P [ t max ≤ x, y ( t ) ≥ ε ] for any x ∈ R . It might be easier to write

and we know that

as all t i s are i.i.d. random samples from P , where Φ( · ; µ, σ 2 ) is the CDF of Gaussian distribution with mean µ and variance σ 2 .

It remains to bound the conditional probability Pr t ∼ P [ y ( t ) ≥ ε | t max ≤ x ] , it may be easier to see it in this way:

where the last step is due to Chernoff bound which holds for any λ > 0 . Now we only need to bound the moment generating function for y ( t ) = log(1 -q + qe 2 t -1 2 σ 2 ) , t ∼ P | ≤ x , where P | ≤ x is

the truncated distribution of P . We note that this is equivalent to bounding the Rényi divergence for truncated Gaussian mixture distribution.

Recall that P = (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) . For any λ that is a positive integer, we have

where ˜ q := (1 -q ) exp(1 / (2 σ 2 )) q . Note that the above expression can be efficiently computed. Denote the above results as r ( λ, x ) := (18). Hence

Now we have

Plugging this bound into (15), we have

which leads to the final conclusion.

Remark 27. In practice, we can further improve the bound by moving the minimum operation inside the integral:

Of course, this bound will be less efficient to compute.

Theorem 23 (Generalizing Theorem 24 and Theorem 26 via Holder's inequality) . For any positive integer λ , and for any a, b ≥ 1 s.t. 1 /a +1 /b = 1 , we have E [( ̂ δ IS ,θ ) 2 ] ≤ kM P ( θ ) ( E [ ̂ δ 2 a MC ] ) 1 /a · 1 /b

( bθe -λε ∫ [ r ( λ, x )] k e -bθx dx ) where r ( λ, x ) is an upper bound for the MGF of privacy loss random variable of truncated Gaussian mixture P | ≤ x defined in (18).

Proof. Note that Theorem 24 and Theorem 26 can both be viewed as two special cases of Hölder's inequality: for any a, b ≥ 1 s.t. 1 a + 1 b = 1 , we have

Theorem 24 corresponds to the case where a = 1 , and Theorem 26 corresponds to the case where b = 1 . We can actually tune the parameters a and b to see if we can obtain any better bounds, as we have

and

Simply combining the above inequalities leads to the final conclusion.

F Proofs for Utility

F.1 Overview

While one may be able to bound the false negative rate through similar techniques that we bound the false positive rate, i.e., applying the concentration inequalities, the guarantee may be loose. As a formal, strict guarantee for FN DPV is not required, we provide a convenient heuristic of picking an appropriate ∆ such that FN DPV is approximately smaller than FP DPV .

For any mechanism M such that δ est > ρδ Y ( ε ) , we have

and in the meantime, if δ est < τδ Y ( ε ) , we have

Our main idea is to find ∆ such that

In this way, we know that FN MC ( ε, δ est ; ρ ) is upper bounded by Θ( δ est /τ ) for the same amount of samples we discussed in Section 4.3.

Observe that ̂ δ MC (or ̂ δ IS ,θ with a not too large θ ) is typically a highly asymmetric distribution with a significant probability of being zero, and the probability density decreases monotonically for higher values. Under such conditions, we prove the following results:

Theorem 28 (Informal) . When m ≥ 2 ν ∆ 2 log( τ/δ est ) , we have

The proof is deferred to Appendix F.2. Therefore, by setting ∆ = 0 . 4 (1 /τ -1 /ρ ) δ est , one can ensure that FN MC ( ε, δ est ; ρ ) is also (approximately) upper bounded by Θ( δ est /τ ) . We empirically verify the effectiveness of such a heuristic by estimating the actual false negative rate. As we can see from Figure 6 (b), the dashed curve is much higher than the two solid curves, which means that the false negative rate is a very conservative bound.

Remark 29 (For the halting case) . In this section, we develop techniques for ensuring the false negative rate (i.e., the rejection probability) is around O ( δ ) when the proposed privacy parameter δ est is close to the true δ . In the experiment, we use the state-of-the-art FFT accountant to produce δ est , which is very accurate as we can see from Figure 1. Hence, the rejection probability in the experiment is around O ( δ ) , which means the probability of rejection is close to the probability of catastrophic failure for privacy.

If one is still concerned that the rejection probability is too large, we can further reduce the probability as follows: we run two instances of EVR paradigm simultaneously; if both of the instances are passed, we randomly pick one and release the output. If either one of them is passed, we release the passed instance. It only fails when both of the instances fail. By running two instances of the EVR paradigm in parallel, the false positive rate (i.e., the final δ ) will be only doubled, but the probability of rejection will be squared.

We can also introduce a variant of our EVR paradigm that better deals with the failure case: whenever we face the 'rejection', we run a different mechanism M ′ that is guaranteed to be ( ε, δ est ) -DP

(e.g., by adjusting the subsampling rate and/or noise multiplier in DP-SGD). Moreover, we use FFT accountant to obtain a strict privacy guarantee upper bound ( ε, δ upp ) for the original mechanism M , where δ est < δ upp . We use p FN and p FP to denote the false negative and false positive rate of the privacy verifier used in EVR paradigm. If the original mechanism M is indeed ( ε, δ est ) -DP, then for any subset S , for this augmented EVR paradigm M aug we have

If the original mechanism M is not ( ε, δ est ) -DP, then we have

Hence, this augmented EVR algorithm will be ( ε, δ est + p FP ( δ upp -δ est )) , and if p FP is around δ est , then this extra factor p FP ( δ upp -δ est ) will be very small. We can also adjust the privacy guarantee for M ′ such that the privacy guarantees for the two cases are the same, which can further optimize the final privacy cost.

F.2 Technical Details

In this section, we provide theoretical justification for the heuristic of setting ∆ = 0 . 4 (1 /τ -1 /ρ ) δ est .

For notation convenience, throughout this section, we talk about ̂ δ MC , but the same argument also applies to ̂ δ IS ,θ with a not too large θ , unless otherwise specified. We use ̂ δ MC ( x ) to denote the density of ̂ δ MC at x . Note that ̂ δ MC (0) = ∞ .

We make the following assumption about the distribution of ̂ δ MC .

While intuitive, this assumption is hard to analyze for the case of Subsampled Gaussian mechanism. Therefore, we instead provide a condition for which the assumption holds for Pure Gaussian mechanism.

Lemma 31. Fix ε, σ . When k/ (2 σ 2 ) ≤ ε , Assumption 30 holds.

Proof. The PRV for the composition of k Gaussian mechanism is N ( k 2 σ 2 , k σ 2 ) .

which is clearly ≥ 1 / 2 when k/ (2 σ 2 ) ≤ ε .

We also empirically verify this assumption for Subsampled Gaussian mechanism as in Figure 7. As we can see, the θ selected by our heuristic (the red star) has Pr[ ̂ δ IS ,θ = 0] ≈ 1 / 2 which matches our principle of selecting θ . The θ minimizes the analytical bound ( which we are going to use in practice ) achieves Pr[ ̂ δ IS ,θ = 0] ≈ 0 . 88 ≫ 0 . 5 .

1.0

0

0.8

S

0.6

0.4

0.0

2.5

5.0

7.5

10.0

θ

Our goal is to show that Pr[ ̂ δ m MC -δ Y ( ε ) < -∆ ∗ ] ⪆ Pr[ ̂ δ m MC -δ Y ( ε ) > ∆ ∗ ] for large m . For notation simplicity, we denote δ := E [ ̂ δ MC ] = δ Y ( ε ) in the remaining of the section. We also denote

Note that p 0 + p (0 , 1) + p (1 , 2) + p (2 , ∞ ) = 1 . We also write ̂ δ MC | ( a,b ) and ̂ δ MC | [ a,b ) to indicate the truncated distribution of ̂ δ MC on ( a, b ) and [ a, b ) , respectively.

We first construct an alternative random variable ˜ δ MC with the following distribution:

This is a valid probability due to Assumption 30, as p (0 , 1) + p (2 , ∞ ) ≤ 1 -p 0 ≤ 1 / 2 . The distribution of ˜ δ MC is illustrated in Figure 8. Note that ˜ δ MC is a symmetric distribution with E [ ˜ δ MC ] = δ .

28 -1

?

28

Similar to the notation of ̂ δ m MC , we write ˜ δ m MC := 1 m ∑ m i =1 ˜ δ ( i ) MC . We show an asymmetry result for ˜ δ m MC in terms of δ .

Lemma 32. For any ∆ ∗ ∈ R , we have

Proof. Note that the above argument holds trivially when m = 0 , 1 . For m ≥ 2 , we use the induction argument. Suppose we have

for any ∆ ∗ ∈ R . That is,

for any ∆ ∗ ∈ R .

where the two inequalities are due to the induction assumption.

Now we come back and analyze ̂ δ m MC by using Lemma 32.

Similarly,

Overall, we have

for any ∆ ∗ ∈ R . Note that the above argument does not require ̂ δ m MC and ˜ δ m MC to be correlated. To maximize Pr [ ̂ δ m MC -˜ δ m MC ≤ c ] , we can sample ˜ δ m MC for a given ̂ δ m MC as follows: for each ̂ δ ( i ) MC ,

If ̂ δ ( i ) MC > 0 , then let ˜ δ ( i ) MC = ̂ δ ( i ) MC .

If ̂ δ ( i ) MC = 0 , then with probability p (2 , ∞ ) /p 0 , output 2 δ -x for x ∼ ̂ δ MC | (2 , ∞ ) ; with probability p (0 , 1) /p 0 output 2 δ -x for x ∼ ̂ δ MC | (0 , 1) ; with probability 1 -( p (0 , 1) + p (2 , ∞ ) ) /p 0 output δ .

Denote the random variable δ diff := ̂ δ MC -˜ δ MC . It is not hard to see that E [ δ 2 diff ] ≤ E [ ̂ δ 2 MC ] + δ 2 ≤ 2 E [ ̂ δ 2 MC ] . By Bennett's inequality, with m ≥ 2 ν ∆ 2 log( τ/δ est ) , we have

if we set c = O (∆) . Let c = 1 4 ∆ , then by setting -∆ ∗ = -∆ and ∆ ∗ +2 c = (1 /τ -1 /ρ ) δ est -∆ , we have ∆ = 0 . 4 (1 /τ -1 /ρ ) δ est .

To summarize, when we set ∆ with the heuristic, we have

G Pseudocode for Privacy Accounting for DP-SGD with EVR Paradigm

In this section, we outline the steps of privacy accounting for DP-SGD with our EVR paradigm. Recall from Lemma 16 that for subsampled Gaussian mechanism with sensitivity C , noise variance C 2 σ 2 , and subsampling rate q , one dominating pair ( P, Q ) is Q := N (0 , σ 2 ) and P := (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) . Hence, for DP-SGD with k iterations, the dominating pair is the product distribution P := P 1 × . . . × P k and Q := Q 1 × . . . × Q k where each P i and Q i follow the same distribution as P and Q . 3

Algorithm 4 Privacy Accounting for DP-SGD with EVR Paradigm

1: Privacy Parameters for DP-SGD: k - number of mechanisms to be composed, σ - noise multiplier, q - subsampling rate, C - clipping ratio.

2: 3: // Step 1: Estimate Privacy Parameter. 4: Obtain a privacy parameter estimate ( ε, δ est ) from a DP accountant (e.g., FFT/GDP/MC accountant). Set the smoothing factor τ = 0 . 99 (one can also adjust the value of τ according to the privacy guarantee and runtime requirements). 5: 6: // Step 2: Verify Privacy Parameter. 7: 8: // Step 2.1: Derive the required amount of samples for privacy verification. 9: Compute the number of required samples m according to Theorem 13, where the second moment upper bound ν is computed according to Corollary 22 for Simple MC estimator, or Theorem 23 for Importance Sampling estimator. Set ρ = (1 + τ ) / 2 and set ∆ according to Theorem 14 for utility guarantee. 10: 11: // Step 2.2: Estimate privacy parameter with MC estimator. 12: If use Simple MC estimator: Obtain i.i.d. samples { t i } i m =1 from P . Compute simple MC estimate ̂ δ = 1 m ∑ m i =1 (1 -e ε -y i ) + with PRV samples y i = log ( P ( t i ) Q ( t i ) ) , i = 1 . . . m . 13: If use Importance Sampling estimator: compute MC estimate according to Definition 18. 14: 15: // Step 3: Release according to verification result. 16: if ̂ δ < δ est τ -∆ then release the result of DP-SGD. 17: else terminate program.

H Experiment Settings & Additional Results

H.1 GPU Acceleration for MC Sampling

MC-based techniques are well-suited for parallel computing and GPU acceleration due to their nature of repeated sampling. One can easily utilize PyTorch's CUDA functionality, e.g.,

, to significantly boost the computational efficiency. Figure 9 (a) shows that when using a NVIDIA A100-SXM4-80GB GPU, the execution time of sampling Gaussian mixture ( (1 -q ) N (0 , σ 2 ) + q N (1 , σ 2 ) ) can be improved by 10 3 times compared with CPU-only scenario. Figure 9 (b) shows the predicted runtime for different target false positive rates for k = 1000 . We vary σ and set the target false positive rate as the smallest s × 10 -r that is greater than δ Y ( ε ) , where s ∈ { 1 , 5 } and r is positive integer. We set δ est = 0 . 8 δ Y ( ε ) , and τ, ρ, ∆ are set as the heuristics introduced in the previous sections. The runtime is predicted by the number of required samples for the given false positive rate. As we can see, even when we target at 10 -10 false positive rate (which means that ∆ ≈ 10 -10 ), the clock time is still acceptable (around 3 hours).

106

GPU

109

CPU

104

107

Time

Time

102

105

Clock

Clock7

100

103

10-2

101

107

109

1011

1013

10-10

10-7

10-4

(a)

(b)

# Gaussian Mixture Sample

6

H.2 Experiment for Evaluating EVR Paradigm

H.2.1 Settings

For Figure 1 & Figure 3, the FFT-based method has hyperparameter being set as ε error = 10 -3 , δ error = 10 -10 . For the GDP-Edgeworth accountant, we use the second-order expansion and uniform bound, following [41].

For Figure 4 (as well as Figure 11), the BEiT [4] is first self-supervised pretrained on ImageNet-1k and then trained finetuned on ImageNet-21k, following the state-of-the-art approach in [34]. For DP-GD training, we set σ as 28.914, clipping norm as 1, learning rate as 2, and we train for at most 60 iterations, and we only finetune the last layer on CIFAR-100.

H.2.2 Additional Results

k → ε Y ( k ) ( δ ) curve. We show additional results for a more common setting in privacy-preserving machine learning where one set a target δ and try to find ε Y ( k ) ( δ ) for different k , the number of individual mechanisms in the composition. We use Y ( k ) to stress that the PRV Y is for the composition of k mechanisms. Such a scenario can happen when one wants to find the optimal stopping iteration for training a differentially private neural network.

Figure 10 shows such a result for Poisson Subsampled Gaussian where we set subsampling rate 0.01, σ = 2 , and δ = 10 -5 . We set ε error = 10 -1 , δ error = 10 -10 for the FFT method. The estimate in this case is obtained by fixing δ est = τδ and find the corresponding estimate for ε through FFT-based method [19]. As we can see, the EVR paradigm achieves a much tighter privacy analysis compared with the upper bound derived by FFT-based method. The runtime of privacy verification in this case is < 15 minutes for all k s.

0.7

0.6

0.5

0.4

3

0.3

0.2

FFT-upp

0.1

FFT-EVR(T=0.95)

0.0

0

250

500

750

1000

k

Privacy-Utility Tradeoff. We show additional results for the privacy-utility tradeoff curve when finetuning ImageNet-pretrained BEiT on CIFAR100 dataset with DP stochastic gradient descent (DPSGD). For DP-SGD training, we set σ as 5.971, clipping norm as 1, learning rate as 0.2, momentum as 0 . 9 , batch size as 4096, and we train for at most 360 iterations (30 epochs). We only finetune the last layer on CIFAR-100.

As shown in Figure 11 (a), the EVR paradigm provides a better utility-privacy tradeoff compare with the traditional upper bound method. In Figure 11 (b), we show the runtime of DP verification when ρ = (1 + τ ) / 2 and we set ∆ according to Theorem 14 (which ensures EVR's failure probability is negligible). The runtime is estimated on an NVIDIA A100-SXM4-80GB GPU. As we can see, it only takes a few minutes for privacy verification, which is short compared with hours of model training.

35

FFT-upp

14

FFT-EVR(T=0.95)

FFT-EVR(T=0.95)

30

(%)

(min)

12

25

Error

10

ntime

e

20

Test

Runt

8

15

6

10

4

(a)

0.3

0.4

0.5

0.6

0.7

0.8

(b)

0.2

0.4

0.6

0.8

1.0

E

E

H.3 Experiment for Evaluating MC Accountant

H.3.1 Settings

Evaluation Protocol. We use Y ( k ) to stress that the PRV Y is for the composition of k Poisson Subsampled Gaussian mechanisms. For the offline setting, we make the following two kinds of plots: (1) the relative error in approximating ε ↦→ δ Y ( k ) ( ε ) (for fixed k ), and (2) the relative error in k ↦→ ε Y ( k ) ( δ ) (for fixed δ ), where ε Y ( k ) ( δ ) is the inverse of δ Y ( k ) ( ε ) from (2). For the online setting, we make the following two kinds of plots: (1) the relative error in approximating k ↦→ ε Y ( k ) ( δ ) (for fixed δ ), and (2) k ↦→ cumulative time for privacy accounting until k th iteration.

MCAccountant. We use the importance sampling technique with the tilting parameter being set according to the heuristic described in Definition 18.

Baselines. We compare MC accountant against the following state-of-the-art DP accountants with the following settings:

The state-of-the-art FFT-based approach [19]. The setting of ε error and δ error is specified in the next section.

CLT-based GDP accountant [7].

GDP-Edgeworth accountant with second-order expansion and uniform bound.

The Analytical Fourier Accountant based on characteristic function [44], with double quadrature approximation as this is the practical method recommended in the original paper.

H.3.2 Additional Results for Online Accounting

Figure 12 and 13 show the online accounting results for ( σ, δ, q ) = (0 . 5 , 10 -5 , 10 -3 ) and ( σ, δ, q ) = (0 . 5 , 10 -13 , 10 -3 ) , respectively. For the setting of ( σ, δ, q ) = (0 . 5 , 10 -5 , 10 -3 ) , we can see that the MC accountant achieves a comparable performance with a shorter runtime. For the setting of ( σ, δ, q ) = (0 . 5 , 10 -13 , 10 -3 ) , we can see that the MC accountant achieves significantly better performance compared to the state-of-the-art FFT accountant (and again, with a shorter runtime). This showcases the MC accountant's efficiency and accuracy in online setting.

FFT

GDP

GDP-Edgeworth

MC (IS)

Relative Error in Estimating ε

CumulativeRuntime

100

104

103

Error

10-2

102

Relative

Time

101

10-4

100

10-1

10-6

10-2

(a)

0

250

500

750

1000

(b)

0

250

500

750

1000

Number of Compositions k

Number of Compositionsk

FFT

GDP

GDP-Edgeworth

MC (IS)

RelativeErrorinEstimatingε

CumulativeRuntime

104

10°

103

10-1

Error

10-2

102

Relative

10-3

Time

e

101

10-4

10°

10-5

10-1

10-6

10-2

0

250

500

750

1000

0

250

500

750

1000

Number of Compositions k

Number of Compositions k

H.3.3 Additional Results for Offline Accounting

In this experiment, we set the number of samples for MC accountant as 10 7 , and the parameter for FFT-based method as ε error = 10 -3 , δ error = 10 -10 . The parameters are controlled so that the MC accountant is faster than FFT-based method, as shown in Table 1. Figure 14 (a) shows the offline accounting results for ε ↦→ δ Y ( k ) ( ε ) when we set ( σ, q, k ) = (0 . 5 , 10 -3 , 1000) . As we can see, the performance of MC accountant is comparable with the state-of-the-art FFT method. In Figure 15 (a), we decreases q to 10 -5 . Compared against baselines, MC approximations are significantly more accurate for larger ε , compared with the FFT accountant. Figure 14 (b) shows the offline accounting results for k ↦→ ε Y ( δ ) when we set ( σ, q, δ ) = (0 . 5 , 10 -3 , 10 -5 ) . Similarly, MC accountant performs comparably as FFT accountant. However, when we decrease q to 10 -5 and δ to 10 -14 (Figure 15 (b)), MC accountant significantly outperforms FFT accountant. This illustrates that MC accountant performs well in all regimes, and is especially more favorable when the true value of δ Y ( ε ) is tiny.

AFA

FFT

GDP

GDP-Edgeworth

MC (IS)

RelativeErrorinEstimating6

RelativeErrorinEstimatingε

102

100

Error

100

Error

10-1

Relative

Relative

10-2

10-2

10-3

10-4

10-4

0.0

2.5

5.0

7.5

10.0

200

400

600

800

1000

(a)

E

(b)

AFA

FFT

GDP

GDP-Edgeworth

MC (IS)

RelativeErrorinEstimating

RelativeErrorinEstimatingε

102

102

101

101

Error

100

Error

100

Relative

10-1

Relative

10-1

10-2

10-2

10-3

10-3

10-4

10-4

(a)

0

1

2

3

4

(b)

200

400

600

800

1000

E

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the problem of bounding the privacy loss for a composition of DP mechanisms. This problem is well-studied in the literature and the particular setting here is when the mechanisms are Gaussian or have Gaussian sub-routines but the privacy loss is measured through the original (approximate) DP definition. This is particularly difficult for Gaussians because of the sub-exponential tail and other work has created new definitions better suited to the Gaussian mechanism. While the privacy loss random variable may be challenging to work with under the original DP definition, it can still be efficiently sampled from, so an estimate of the expected privacy loss can be computed efficiently. The authors show that these estimates are quite accurate with high probability and leverage this guarantee to turn the estimate into a formal privacy bound (or failure to run with tiny probability). As a result, for certain settings the authors give improved privacy composition bounds over the previous work.\n\nSTRENGTHS:\n1. Clever new technique to use composition privacy loss estimates instead by rejecting the estimate with low probability.\n\n2. Significant technical work that combines a variety of techniques.\n\n3. Their work extends nicely to the subsampled Gaussian mechanism for DP-SGD.\n\n4. Empirical testing run in a variety of settings for both Gaussian mechanism and DP-SGD.\n\n5. Improves upon previous work for DP composition in a reasonably important setting.\n\nWEAKNESSES:\nRemark 6 feels exaggerated. For the settings considered in this paper, the delta parameter is just a result of concentration bounds and sub-exponential tails of Gaussians, not the probability of a catastrophic data breach. Unless the authors know of examples with delta <= 10^{-10}, I've only seen practical applications with delta at minimum 10^{-7} and 10^{-6} is most common. Of course there are privacy advocates that will always push for more privacy, but they are also likely to further push for \"pure\" differential privacy in which composition is easy and Laplace mechanism must be used instead.\n\n$\\textit{The authors pointed to the discussion here I missed, so strike this comment}$\nThere are privacy definitions better suited to Gaussians that have become very common both in practice and in the literature that allow for easier analysis and make this more difficult accounting less necessary (though still interesting).\n\n$\\textit{The authors adequately addressed this in the rebuttal}$\nThe improvement in epsilon for composition of the Gaussian mechanism over the previous work is quite small (<1%) for their empirical study (figure 5).\n\nQUESTIONS:\n$\\textit{All questions were answered by the authors}$\n\nWhat's the runtime for the 'Exact' method compared to FFT or EVR? It was a bit confusing why that couldn't be used given that it's the ideal privacy parameter to output.\n\nFrom the experimental result with DP-SGD, it looks like MC beats FFT even with more standard delta settings. Does this hold more generally? Why was this not the result emphasized in the intro instead of the gaussian mechanism composition? Are the curves in figure 4 identical but shifted? So basically using MC accounting you compute a better epsilon at each epoch?\n\nLIMITATIONS:\nThe limitations were appropriately discussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

comments: SUMMARY:
Authors introduce a new privacy accounting method to characterize the privacy loss random variable. The work reduce the classical privacy accounting problem into mean estimation problem following the previous work and give a Monte Carlo solution.  The work provides detailed analysis of the proposed method and its utility and also give some common distribution to show the effectiveness and performance theoretically. The numerical studies also show the correctness and practicability of the method in the real privacy accounting tasks.

STRENGTHS:
Pros: 
1. The analysis for the proposed method is detailed and the writing is friendly to follow with a detailed preliminary.
2. The proposed tool works better than the compared existing accounting tools like CLT and FFT-based methods.
3. The fast speed and the online implementation show some potential for the method used in privacy accounting applications.

WEAKNESSES:
Actually, I have reviewed the paper in ICML. I think the paper has fixed almost the problem in the former cycle. The only problem I still keep is about whether the method will suffer from the dimension curse when deriving the prv samples for a general distribution.

QUESTIONS:
/

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a privacy accounting method called estimate-verify-release (EVR), whose basic principle is to convert an estimate of a privacy parameter into a formal privacy guarantee. The mechanism works by verifying whether the estimated privacy guarantee holds, and then releasing the query output depending on the verification result. The paper develops a Monte-Carlo-based verifier for this paradigm. The overall accountant is broadly applicable and is shown to give a tighter privacy-utility tradeoff than existing baselines.\n\nSTRENGTHS:\nThe mechanism is applicable broadly, in particular to important DP algorithms such as DP-SGD (that is, the subsampled Gaussian mechanism). It exploits the fact that existing work provides good privacy loss estimates and converts these estimates into a formal paradigm for ensuring differential privacy. The method is shown to beat a strong baseline, namely the FFT-based accountant from [19].\n\nWEAKNESSES:\nThe paper places a lot of emphasis on the fact that we can't naively use an estimated privacy parameter as the truth, because DP is a strict guarantee, and this makes perfect sense. But then in the analysis and implementation of the accountant there are some steps of the new accountant that are not made completely rigorous, such as the number of Monte Carlo samples. Or, in Theorem 13 there is a nu parameter that is not known. So my suggestion is to write a fully formal version of the accountant for DP-SGD in the main body to show that the paradigm can indeed be applied fully rigorously.\n\nIn addition, it would be good to see at least one more experiment showing the same comparison as Figure 5. In Figure 5 the gains of MC over FFT are not clear for a small number of compositions. So it would be good to see if the comparison of Figure 5 is robust and generalizes to different problem settings.\n\nA few minor comments:\n- In the Conclusion, you say \"allowing safe privacy parameter estimates *without* provable assurance\"?\n- The Figures don't seem to be in vector format and are blurry if zoomed in.\n- Typo in Line 132, \"privacy loss random variable *is* ...\" (right now there's no verb in the main clause)\n- In Line 180, say where rho takes values.\n\nQUESTIONS:\nNothing at the moment.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors propose EVR framework for privacy accounting. The core idea is to estimate the privacy budget, verify whether the budget is approximately met, and then decide whether to release the result or halt. The workhorse is a Monte-Carlo verifier (also used as an accountant through binary search). The empirical evaluation shows significant improvement over existing techniques, especially in the large epsilon/small delta regime.

STRENGTHS:
1. The authors propose a novel framework for privacy accounting: EVR. By trading off a small probability that the program halts with privacy budget, the authors manage to get a tighter privacy profile curve at larger epsilon/smaller delta regime.
2. The authors verify the proposed framework empirically and showcase the advantage of EVR.

WEAKNESSES:
1. One downside of the proposed approach is that there is a probability that the mechanism halts with a privacy budget cost. To get a tighter epsilon, you need to take the risk that you get nothing. Although this seems to be reasonable trade-off, it can be a problem in practical use case. More discussion should be put on this.

QUESTIONS:
Any idea on how to deal with halting in practical usage?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07