Title: Optimal privacy guarantees for a relaxed threat model: Addressing sub-optimal adversaries in differentially private machine learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Differential Privacy

Membership Inference Attack

Hypothesis Testing

Data Reconstruction Attack

Security
Abstract: Differentially private mechanisms restrict the membership inference capabilities of powerful (optimal) adversaries against machine learning models. Such adversaries are rarely encountered in practice. In this work, we examine a more realistic threat model relaxation, where (sub-optimal) adversaries lack access to the exact model training database, but may possess related or partial data. We then formally characterise and experimentally validate adversarial membership inference capabilities in this setting in terms of hypothesis testing errors. Our work helps users to interpret the privacy properties of sensitive data processing systems under realistic threat model relaxations and choose appropriate noise levels for their use-case.

Optimal privacy guarantees for a relaxed threat model: Addressing sub-optimal adversaries in differentially private machine learning

Abstract

Differentially private mechanisms restrict the membership inference capabilities of powerful (optimal) adversaries against machine learning models. Such adversaries are rarely encountered in practice. In this work, we examine a more realistic threat model relaxation, where (sub-optimal) adversaries lack access to the exact model training database, but may possess related or partial data. We then formally characterise and experimentally validate adversarial membership inference capabilities in this setting in terms of hypothesis testing errors. Our work helps users to interpret the privacy properties of sensitive data processing systems under realistic threat model relaxations and choose appropriate noise levels for their use-case.

1 Introduction

Machine learning (ML) offers a powerful set of techniques for addressing complex problems across various fields, including fields where sensitive data is processed, e.g. medicine or finance. However, the use of sensitive data in ML comes with privacy challenges: It is known that ML models memorise their training data [1], and that some degree of memorisation may be necessary for the best model performance [2]. When a model memorises private information, it can reveal this information when attacked by adversaries. A very important class of attack is Membership Inference (MI) [3], which aims to determine whether the data of a specific individual was used to train a machine learning model. MI can lead to the disclosure of highly sensitive information about the individual, e.g. that

they are part of a database of cancer patients which was used to train a predictive medical ML system. The importance of studying MI is furthermore derived from the fact that a successful MI attack signifies that the model is also vulnerable to all other attacks such as data reconstruction or attribute inference ('most significant bit' property [4]).

Differential Privacy (DP) [5] is a formal framework and collection of techniques to furnish objective privacy guarantees for ML model training. It is a natural counterpart to MI, and it is possible to interpret DP entirely through the MI lens, which is known as the hypothesis testing interpretation of DP [6, 7]. Given a database D and a database D ′ which differ in exactly one record, hypothesis testing DP quantifies the trade-offs between the Type-I and Type-II errors of an adversary using a hypothesis test to determine whether a model was trained on D or on D ′ . This hypothesis test is considered to be optimal in the sense of the Neyman-Pearson (NP) lemma [8], which makes the DP Threat Model (DPTM) very strong: It assumes that the adversary has full access to the training database, knowledge of the specifications of the DP mechanism and can even deeply manipulate the process of ML model training itself to their advantage [9]. Although this has the benefit that DP is a true 'worst-case' guarantee and thus holds for all weaker adversaries, it may be unnecessarily conservative in practice. For instance, the assumption that the adversary has full access to the training data may be unrealistic in settings where this data is tightly access-restricted. As a consequence, previous works also considered threat models with weaker adversaries, i.e. Relaxed Threat Models (RTMs), and experimentally showed that the empirical protection against MI offered by DP mechanisms is much stronger than the upper bound predicted by the DP analysis [9, 10, 11]. While such empirical investigations provide valuable initial insights, we contend that an additional formal analysis of specific, practically relevant RTMs can greatly benefit stakeholders in conducting a thorough privacy analysis of ML systems. For example, a 'privacy certificate' for an ML model can include both (1) the MI risk under the DPTM (which holds when 'all bets are off') but also (2) the more realistic MI risk under whichever RTM is applicable to the actual model training conditions.

In this work, we study MI in the context of a specific RTM with high practical relevance, in which the adversary has incomplete access to the model's training data . An example of this RTM is the training of a predictive ML model on medical data which is generated and stored at a single hospital and never released, whereas the model is shared with third parties. Here, an MI adversary must resort to leveraging auxiliary data from the same distribution as the hospital's target database. A similar attack ( offline MI ) was studied in [12, 13, 14], and also does not use the full database for the attack (albeit for reasons of computational efficiency). As MI is equivalent to a hypothesis testing problem, the aforementioned works demonstrate that the Type-I and Type-II errors (i.e. the false positive/false negative rates) of offline MI attacks are higher than MI attacks assuming the DPTM. In this work, we formally derive and validate exact bounds on the error rates of MI attacks against DP noise mechanisms in the aforementioned RTM (which we will refer to as 'the RTM' from now on).

Contributions Our contributions are as follows: (1) Since MI can be fully characterised using hypothesis testing, we begin by formalising the RTM as a hypothesis testing security game and show that the RTM adversary is strictly sub-optimal in terms of error rates compared to the DPTM adversary; (2) Next, we formally derive the 'best sub-optimal' hypothesis testing MI strategy in the RTM; (3) We then analyse the error rates associated with this strategy against common additive noise mechanisms used in DP under the RTM. (4) We express these error rates using trade-off functions, which parameterise all achievable Type-I/Type-II error combinations, similar to the approach taken by [7] in defining f-DP. (5) Additionally, we introduce a new metric, λ -Detection Resilience ( λ -DR), which allows us to compare the hypothesis testing capabilities of the RTM and DPTM adversaries, despite their differing threat models. (6) We conclude our study by empirically auditing the proposed bounds and demonstrating accuracy advantages in the training of deep neural networks with DP-SGD.

2 Background

Related work Our work contributes to the ongoing research in three distinct, yet related areas of privacy-preserving ML. Firstly, we aim to augment empirical investigations which use MI to establish lower bounds on the privacy guarantees of ML systems, with formal guarantees. For instance, [9] experimentally evaluate several types of adversaries against DP ML workflows, [10, 11] present empirical privacy analyses of DP-SGD [15], while [16] focus on federated learning. Such empirical privacy auditing techniques have found their way into software tools such as ML-Doctor [17] or ML

Privacy Meter [14]. As mentioned above, our RTM considers an adversary performing MI without access to the (complete) model's training data. This type of MI attack has been studied in several prior works, most notably by Watson et al. [13], Ye et al. [14] and Carlini et al. [12], who find the adversary's hypothesis testing capabilities to be substantially diminished compared to the DPTM. The second line of work related to our paper are formal analyses of MI in the DPTM, such as [18], who consider a Bayesian adversary or [4], who bypass the DP analysis and directly study the properties of the underlying distributions. Moreover, [19] and [20] are rooted in hypothesis testing theory like our work, but only consider threat models equivalent to the DPTM, while [21] consider restricted adversaries, albeit not through the hypothesis testing lens. Lastly, we adopt the recent development of using functions (rather than numbers) to parameterise privacy guarantees, which is becoming common practice in the DPTM [7, 22, 23], but has not yet been used to study threat model relaxations. Our paper unifies these strands of work by introducing a formal guarantee in the RTM, which allows for a more nuanced analysis of the privacy risks associated with machine learning models.

Differential privacy DP [5] is a stability condition on randomised mechanisms requiring that their outcomes should be approximately equally likely when the data of a single individual is added or removed from the input database. Throughout this work, we will consider the scenario of ML model training, where DP is realised through an additive noise mechanism . We consider a deterministic model training function q . The global sensitivity ∆ > 0 of q is defined as ∆ = sup D,D ′ : D ≃ D ′ ∥ q ( D ) -q ( D ′ ) ∥ where the sup is taken over all pairs of neighbouring databases D ≃ D ′ which differ in a single record. 2 ∆ is measured in some norm, which we will indicate by subscript wherever relevant (e.g. ∆ 1 ). An additive noise mechanism (or just mechanism) M maps databases and probability distributions to a (model) parameter space Θ . Its outputs are of the form θ = q ( X ) + z, z ∼ Z , where Z is a noise distribution. Since q and the database are considered deterministic, θ is a random variable induced by the randomness of Z . Unless otherwise indicated, M will be realised through distributions Z ( µ, ξ ) supported on the real line which admit a density. We assume that ξ is fixed and public information and that Z is shift invariant. In other words, M ( D ) = ( q ( D ) + Z (0 , ξ )) ∼ Z ( q ( D ) , ξ ) and M ( D ′ ) = ( q ( D ′ ) + Z (0 , ξ )) ∼ Z ( q ( D ′ ) , ξ ) .

In the hypothesis testing interpretation of DP [6, 7, 24], the adversary attempts to determine whether θ was trained on D or D ′ using a classical/frequentist hypothesis test. The adversary considers the hypotheses H 0 : θ ∼ M ( D ) vs. H 1 : θ ∼ M ( D ′ ) and their reverse, i.e. H 0 : θ ∼ M ( D ′ ) vs. H 1 : θ ∼ M ( D ) . H 0 and H 1 are called the null and alternative hypothesis, respectively. The aim of the hypothesis testing problem is to distinguish between M ( D ) and M ( D ′ ) using a pair of decision rules (tests). Let ϕ, ϕ ′ be tests with Type-I error rate α ϕ = E M ( D ) ( ϕ ) and Type-II error rate β ϕ = 1 -E M ( D ′ ) ( ϕ ) (identical for ϕ ′ ). The adversary's goal will be to maximise their power 1 -β at a fixed level α , since the fundamental trade-off in hypothesis testing is that α and β cannot be minimised simultaneously [25]. All achievable α/β pairs for a given adversary/test combination are expressed by a pair of functions on the unit square, called the trade-off functions T and T -1 .

T is defined through the test ϕ : M ( D ) vs. M ( D ′ ) as T ( M ( D ) , M ( D ′ ))( α ) = inf ϕ { β ϕ : α ϕ ≤ α } , whereas T -1 = T ( M ( D ′ ) , M ( D )) is defined using ϕ ′ . The inf is taken over the set of tests which can be constructed in a given threat model, in this case the DPTM. For mechanisms with symmetric noise density, T = T -1 , allowing us to consider only a single trade-off function. In general, one takes the symmetrisation/convexification of T and T -1 , denoted C( T, T -1 ) which creates a single, symmetric trade-off function. Since trade-off functions fully characterise the adversary's error rates, they can be used to compare the privacy guarantees of different mechanisms and/or different threat models. In particular, the closer the trade-off function's graph is to the line β ( α ) = 1 -α , i.e. the off-diagonal of the unit square, the stronger the implied privacy guarantee of the mechanism is. The notion of expressing DP guarantees based on a comparison of C( T, T -1 ) with a 'reference' trade-off function f is called f-DP [7]:

Definition 1. A mechanism M satisfies f -DP, if ∀ D,D ′ : D ≃ D ′ it holds that C( T, T -1 )( α ) ≥ f ( α ) ∀ α ∈ [0 , 1] for a reference trade-off function f (see Figure 2 of [7] for an example).

A pair of distributions Z ( µ 1 , ξ ) , Z ( µ 2 , ξ ) is called a dominating pair [22] for M if, ∀ D,D ′ : D ≃ D ′ , C( T, T -1 )( α ) ≥ C( T ( Z ( µ 1 , ξ ) , Z ( µ 2 , ξ )) , T ( Z ( µ 2 , ξ ) , Z ( µ 1 , ξ )))( α ) ∀ α ∈ [0 , 1] .

For example, the dominating pair for the Gaussian Mechanism (GM) is ( N (0 , σ 2 ) , N (∆ 2 , σ 2 )) and for the Laplace Mechanism (LM), it is (Lap(0 , b ) , Lap(∆ 1 , b )) . The dominating pair distributions thus exhibit the greatest effect size (in this case ∆ 1 / b or ∆ 2 / σ ), allowing for the construction of the trade-off functions whose graphs are farthest from the off-diagonal of the unit square. In other words: the construction of the trade-off functions is determined by the hypothesis tests with the greatest power 1 -β under a specific threat model, evaluated at the dominating pair distributions.

In the DPTM, H 0 and H 1 are both simple hypotheses of the form H 0 : θ ∈ Θ 0 ⊂ Θ vs. H 1 : θ ∈ Θ 1 ⊂ Θ with ∥ Θ 0 ∥ = ∥ Θ 1 ∥ = 1 and Θ 0 ∩ Θ 1 = ∅ . The reason is that the distributions of M ( D ) and M ( D ′ ) are fully specified because the adversary can evaluate q ( D ) and q ( D ′ ) . The NP lemma [8] states that, for simple vs. simple hypothesis tests such as the ones in the DPTM, the uniformly most powerful (UMP) level α tests ϕ and ϕ ′ (with the highest power 1 -β at a specified level α over the entire range of θ , in this case q ( D ) or q ( D ′ ) ) can be constructed by thresholding the (log-) Likelihood Ratio (LRs) Λ / Λ ′ of the mechanism's outputs θ under D and D ′ :

where c ∈ R is the critical value. The trade-off functions in the DPTM are then constructed from the distributions of Λ / Λ ′ evaluated at the dominating pairs under the respective H 0 and H 1 (also called privacy loss distributions [23]). Since the DPTM adversary enjoys the optimality properties of the NP lemma, we will refer to this adversary as A NP .

3 Hypothesis testing in the RTM

The trade-off function's form for a given threat model depends only on the test statistic's distributions under H 0 and H 1 . Our strategy for the rest of the paper is thus: (1) Derive the optimal tests the adversary can construct in the RTM; (2) Specify the test statistic distributions for these tests, and (3) Construct the trade-off functions for a given mechanism using these test statistics, which will allow us to tightly bound the adversary's error rates and compare them to the DPTM. We first recall a general result, specifying how trade-off functions are constructed from test statistics.

Definition 2 (Trade-off function construction) . Let P, Q be the distributions of a test statistic under H 0 and H 1 , respectively and let Φ P/Q denote their cumulative distribution function (CDF), Ψ P/Q their survival function (SF) and Φ -1 P/Q , Ψ -1 P/Q their respective inverses (iCDF , iSF). Then, we define:

For example, in experimental studies, empirical proxies of the test statistic distributions are evaluated by training shadow models [3]. Then, the empirical trade-off functions are constructed, which parameterise the error rates of the empirical adversary under a specific threat model [11, 12, 14]. We will utilise this strategy later to validate our theoretical bounds empirically. For our formal analysis on the other hand, we will derive exact expressions for T and T -1 .

RTM security game For reasons which will soon become clear, we will refer to the adversary under our RTM as a sub-optimal adversary ( A SO ) and use a formal security game, similar to the one used to define the DPTM (see e.g. Section 3.1. of [4]), which proceeds between a neutral/trusted curator C and A SO . We consider a single-round, non-interactive protocol.

Step 2: C flips a bit b . If b = 0 , they fix X = D . If b = 1 , they choose a singleton { x ∗ } ⊂ D ′ , where D ⊂ D ′ and D ∩ D ′ = { x ∗ } and fix X = D ∪ { x ∗ } . They then train a model θ ∈ Θ on X using q and M ( θ may also be a gradient) and send θ to A SO ;

Step 1: The adversary A SO constructs a database D = { x 1 , . . . x n } ⊆ D , decides on a training function q , a DP mechanism M and sends them to the curator C ;

Step 3: A SO decides if H 0 : θ ∼ M ( D ) or H 1 : θ ∼ M ( D ′ ) ;

The key difference between the RTM and the DPTM is step 2: A SO has no access to the point x ∗ which may be added by C , and by extension, no knowledge of the exact value of q ( D ′ ) . In step 3, A SO can therefore only decide θ ∼ M ( D ′ ) by rejecting that θ ∼ M ( D ) , but not by directly confirming that θ ∼ M ( D ′ ) .

Step 4: If A SO is correct, the MI attack is successful, privacy is breached and the game is won.

The hypothesis testing capabilities of the RTM adversary are identical to those of an offline MI adversary . Offline MI attacks are used in previous works to audit the privacy of ML model training [12, 13, 14] and decide whether x ∗ is part of the training data or not without actively training shadow models on databases containing it. Equivalently, offline MI is MI by exclusion (i.e. inferring θ ∼ M ( D ′ ) by rejecting H 0 : θ ∼ M ( D ) ). In offline MI, this is done for reasons of practicality and computational efficiency [12], while in the RTM, it by design. Nonetheless, the resulting error rates are identical and our bounds thus can be used to characterise offline MI.

RTM: Formal analysis M is an additive noise mechanism based on the distribution Z , thus H 0 : θ ∼ M ( D ) is equivalent to H 0 : θ ∼ Z ( q ( D ) , ξ ) . Since q ( D ) is computable by A SO through access to D , the likelihood L ( θ | Z ( q ( D ) , ξ )) is also computable and H 0 is a simple hypothesis of the form θ ∈ Θ 0 , ∥ Θ 0 ∥ = 1 , exactly like the DPTM. However, the likelihood L ( θ | Z ( q ( D ′ ) , ξ )) is not computable without access to D ′ and H 0 is a composite hypothesis of the form θ ∈ Θ \ Θ 0 with ∥ Θ \ Θ 0 ∥ > 1 because it depends on the unknown value of q ( D ′ ) . The following is a direct consequence of the NP lemma, which holds only for simple vs. simple hypotheses [25].

Lemma 1. In the RTM, no UMP level α test exists for all possible values of q ( D ′ ) , i.e. ∀ θ ∈ Θ \ Θ 0 . Thus, H 0 vs. H 1 (and H 1 vs. H 0 ) are each not decidable by a single test.

This implies that A SO is necessarily weaker than A NP , i.e. cannot achieve the same power 1 -β at a given level α for all values of the parameter of interest; to thus establish the optimal tests in this sub-optimal setting , A SO must consider specific value ranges of q ( D ′ ) separately. In the following, we will call two tests equivalent if they have the same power at a level α for all values of the tested parameter. Moreover, we will limit our description to H 0 : θ ∼ M ( D ) vs. H 1 : θ ∼ M ( D ′ ) , as H 1 vs. H 0 is handled identically.

Lemma 2. The composite LR test H 0 : θ ∼ Z ( q ( D ) , ξ ) vs. H 1 : θ ∼ Z ( q ( D ′ ) , ξ ) is equivalent to letting θ ∼ Z ( µ, ξ ) and simultaneously conducting two individual one-sided LR tests r and r ′ with null hypothesis µ = 0 and alternative hypotheses -∆ ≤ µ < 0 for r and 0 < µ ≤ ∆ for r ′ , where µ = q ( D ′ ) -q ( D ) and ∆ is the global sensitivity of q .

Thus, A SO can leverage known facts about q and M to limit the value ranges for the alternative hypothesis, but, as q ( D ′ ) is unknown, must 'split the problem' into two individual tests. The individual tests are then UMP level α under the following pre-condition.

Lemma 3. The individual LR tests r and r ′ are UMP level α if the noise distribution Z has the monotone likelihood ratio property (MLRP). The Laplace Mechanism (LM), Gaussian Mechanism (GM) and the Poisson-Subsampled Gaussian Mechanism (SGM) all have the MLPR. Moreover, the power of both tests is maximised when ∥ µ ∥ = ∥ q ( D ′ ) -q ( D ) ∥ = ∆ .

This lemma intuitively states that, while there is an optimality condition associated with the tests that can be constructed in the RTM, it is not the same optimality condition as the NP lemma guarantees in the DPTM. In particular, although A SO can construct a pair of tests which are optimal (UMP level α ), they are only optimal for specific parameter value ranges rather than for all possible parameter values. The lemma also states that it suffices to consider the tests µ = 0 vs. µ = ± ∆ when constructing the trade-off functions, as these correspond to the highest effect size at a fixed ξ . In other words, dominating pairs remain dominating pairs in the RTM. Next, we provide two convenience lemmata, which allow A SO to conduct both tests simultaneously.

Lemma 4. If the density of Z is additionally symmetric about the location parameter, the simultaneous individual LR tests r and r ′ are equivalent to performing a single test using the magnitude of the observation as a test statistic, i.e. letting ∥ θ ∥ ∼ ∥Z ( µ, ξ ) ∥ and testing H 0 : ∥ µ ∥ = 0 vs. H 1 : ∥ µ ∥ > 0 . The power of this 'combined' test is maximised at ∥ µ ∥ = ∆ .

Lemma 5. For a (not necessarily symmetric) Z the Generalised Likelihood Ratio Test (GLRT) using the test statistic log L ( θ |Z (ˆ µ,ξ )) L ( θ |Z (0 ,ξ )) , where ˆ µ is the maximum likelihood estimate of µ is equivalent to simultaneously conducting r and r ′ . The GLRT's power is maximised at ˆ µ = ∆ .

We stress that, while the two tests are UMP level α and can be 'combined' into one test, the combined test is not UMP level α over the whole range (by Lemma 1). To construct both trade-off functions, the steps above are either repeated for H 1 vs. H 0 , or the trade-off function for H 0 vs. H 1 is inverted directly, i.e. in closed form or numerically.

λ -Detection resilience Next, we introduce a measure for A SO 's optimal achievable error rates under the RTM, inspired by f-DP: For each mechanism, we create a pair of trade-off functions j and j -1 . These parameterise the error rates of the UMP level α tests against the dominating pair distributions of a given mechanism M in the RTM. Next, we 'unify' the functions by symmetrisation/convexification to obtain a single, symmetric trade-off function J : α ↦→ β , which expresses all optimal error rates of A SO against M . Finally, we compare J to a reference trade-off function λ . If J lies on or above λ , A SO cannot achieve lower β for any α in the task of detecting the presence of x ∗ in D ′ by observing θ , and we say that M satisfies λ -Detection Resilience ( λ -DR).

Definition 3. Let r ∈ R be a test for M ( D ) vs. M ( D ′ ) and r ′ ∈ R be a test for M ( D ′ ) vs. M ( D ) , where R is the set of LR tests which are computable in the RTM. To construct J , fix a level α , then obtain j = inf r ∈ R { β r : α r ≤ α } and j -1 = inf r ′ ∈ R { β r ′ : α r ′ ≤ α } . Then compute the symmetrisation/convexification J = C( j, j -1 ) . We say that a mechanism M satisfies λ -DR, if for some reference function λ and ∀ D,D ′ : D ≃ D ′ , it holds that J ( α ) ≥ λ ( α ) ∀ α ∈ [0 , 1] .

We remark that computing the symmetrisation/convexification C( j, j -1 ) is only a convenience measure to circumvent the necessity to work with both j and j -1 separately. The concrete algorithm for computing C for any trade-off function pair can be found in Definition E.1 of the Appendix to [7].

Since A SO is weaker than A DP , we expect the trade-off functions characterising their error rates to lie on or above the trade-off functions for the same mechanism under the DPTM (see Figure 1). However, it is important to remark that λ -DR is not a DP guarantee , although the opposite holds true.

Lemma 6. If a mechanism satisfies f -DP, it also satisfies f ′ -DR, with f ′ ( α ) ≥ f ( α ) ∀ α ∈ [0 , 1] .

Moreover, since λ -DR is defined conditional on a restriction of A SO 's background knowledge, it is not resilient to post-processing through auxiliary information. However, a weaker condition holds:

Lemma 7. If a mechanism satisfies f -DP and f ′ -DR with f ′ ≥ f , arbitrary post-processing can only deteriorate privacy up to the f -DP guarantee.

It follows that λ -DR is not closed under adaptive composition . In fact, no privacy relaxation assuming restricted background knowledge is closed under adaptive composition [26]. However, mechanismspecific guarantees can be given for non-adaptive composition , where the data/mechanism parameters are fixed in advance and all intermediate models/gradients are released to A SO . This covers the standard DP-SGD setting and is studied below.

4 Mechanism-specific analysis

Since precisely quantifying DR for a specific noise mechanism requires knowledge of the RTM trade-off functions, we now provide methods to compute j and j -1 for various DP mechanisms. The construction of J = C( j, j -1 ) is independent of the specific form of j, j -1 . By definition for all j, j -1 : j (0) = j -1 (0) = 1 , j (1) = j -1 (1) = 0 and α ∈ [0 , 1] .

Laplace Mechanism The LM with scale b on a function q with global ℓ 1 -sensitivity ∆ 1 outputs q ( X ) + Lap(0 , b ) . The LM has a symmetric noise density, thus, the magnitude distributions can be used as test statistics according to Lemma 4. Under H 0 , the test statistic follows an exponential distribution, while under H 1 , it follows the (uncommon) folded Laplace distribution [27]. Both distributions admit closed-form expressions for their CDF, SF, iCDF and iSF, thus the trade-off functions can also be characterised in closed form.

Theorem 1. Let µ 1 = ∆ 1 / b . The LM satisfies C( j Lap ( α ) , j -1 Lap ( α )) -DR with:

and

Gaussian Mechanism The GM with covariance matrix σ 2 I d on a d -dimensional function q with global ℓ 2 sensitivity ∆ 2 outputs q ( X ) + N (0 , σ 2 I d ) . Like the LM, it also has a symmetric noise density, thus the (squared) magnitude distributions can be used as a test statistic. Let χ 2 d (0 , σ 2 ) be the chi-squared distribution with d degrees of freedom, i.e. the distribution of ∥N (0 , σ 2 I d ) ∥ 2 2 . Let χ 2 d ( µ 2 2 , σ 2 ) be the noncentral chi-squared distribution with d degrees of freedom and noncentrality parameter µ 2 2 = ∆ 2 2 / σ 2 , i.e. the distribution of ∥N ( ν , σ 2 I d ) ∥ 2 2 with ∥ ν ∥ 2 = ∆ . As in Definition 2, Φ , Ψ , Φ -1 and Ψ -1 denote the CDF, SF, iCDF and iSF of the subscripted distribution, respectively.

Theorem 2. Let µ 2 = ∆ 2 / σ . The GM satisfies C( j GM ( α ) , j -1 GM ( α )) -DR with:

These functions have no general analytic representation, but are easy to evaluate numerically. However, an analytic representation is available for j ( α ) at d = 1 . We discuss the importance of d = 1 below.

Corollary 1. For d = 1 , j ( α ) admits the following closed-form representation:

Next, we study non-adaptive composition, where a database and mechanism parameters are fixed ahead of time, and all intermediate results are released to A SO . This setting is often encountered, e.g. in standard DP-SGD.

Lemma 8. Let GM a , GM b be GMs with noise variances σ 2 a I d , σ 2 b I d on functions with sensitivities ∆ 2 a , ∆ 2 b , respectively. Then, the non-adaptively composed mechanism GMC has trade-off functions:

with κ c = (∆ 2 a +∆ 2 b ) 2 / σ 2 a + σ 2 b and σ 2 c = σ 2 a + σ 2 b / 4 .

The GM moreover exhibits specific asymptotic behaviour when the query dimensionality d and/or the number of non-adaptive compositions increase.

Theorem 3 (Blessing of dimensionality in the RTM) . Consider a GM on a function with sensitivity ∆ 2 and noise variance σ 2 I d such that ∆ 2 / σ ≪ 1 . Let µ 2 = ∆ 2 / σ . As d and/or as the number of non-adaptive compositions N increase, j GM and j -1 GM tend to the common form:

As d increases for fixed N , j c and j -1 c become symmetric and eventually both tend to Φ N (0 , 1) ( Ψ -1 N (0 , 1) ( α ) ) = 1 -α , i.e. the Type-I and Type-II errors are equal and A SO 's test is no better than random guessing. Intuitively, this can be understood by considering that the magnitude of θ (i.e. the test statistic) is influenced less by its individual components as dimensionality increases [28]. Therefore, at d > 1 , the GM provides progressively stronger λ -DR guarantees than for scalar queries. However, as the adversary can leverage their knowledge of the model to design a database which influences only a single coordinate of the weight/gradient vector, we mostly use d = 1 as a worst-case scenario below. In practice, such a worst-case database may be unrealistic, and the aforementioned result can thus be applied to obtain tighter bounds on MI success in specific scenarios. We note that, in the DPTM, the trade-off function of the GM is always independent of d and has the form

f

⊗

N

GM

= Φ

N

(0

,

1)

Ψ

-

1

N

(0

,

1)

(

α

)

µ

,

µ

=

N

∆

2

/

σ

,

N

denotes

N

-fold composition.

The similarity between f GM and Equation (7) is a consequence of the central limit theorem-like phenomenon [7]. Despite the similarity between the functional forms, we stress that this does not mean that the asymptotic λ -DR guarantee implies a GDP guarantee, although the converse holds.

-

(

)

3

⊗

√

Poisson-Subsampled Gaussian Mechanism (SGM) The SGM is relevant in private deep learning [15]. On a d -dimensional function q with global ℓ 2 -sensitivity ∆ 2 , the SGM with diagonal covariance matrix σ 2 I d samples a mask S ∼ Ber ( p ) d , where S ∈ { 0 , 1 } d and Ber ( p ) denotes a Bernoulli distribution with probability p and outputs q ( X ) + S ⊙ Z , Z ∼ N (0 , σ 2 I d ) , where ⊙ is the Hadamard product. The dominating pairs are ( N (0 , σ 2 ) , (1 -p ) N (0 , σ 2 ) + p N (∆ 2 , σ 2 )) and ((1 -p ) N (∆ 2 , σ 2 ) + p N (0 , σ 2 ) , N (∆ 2 , σ 2 )) [22]. Although we have shown in Lemma 3 that the SGM satisfies the MLRP, the test statistic distributions under H 0 and H 1 do not have a tractable form. We thus proceed numerically: First, we instantiate j GM and j -1 GM for the standard GM as shown above, then use the following general result to amplify the trade-off functions directly.

Lemma 9. Let T ( A,B )( α ) be a trade-off function between two general distributions A,B representing mechanism outputs. The trade-off functions for the sub-sampled mechanisms are given by T ( A, (1 -p ) A + pB )( α ) = pT ( A,B )( α ) + (1 -p )(1 -α ) and by its inverse.

We thus obtain the amplified functions j p and j -1 p from which we compute J SGM = C( j p , j -1 p ) . It follows that the SGM satisfies J SGM -DR. This subsampling technique can also be applied to other mechanisms [22, 23]. An example for the LM is shown in the Appendix. Composition of mechanisms other than the GM is also handled numerically through direct composition of the test statistics as described in [7], Definition 3.1. ff.

5 Experimental evaluation

RTM/DPTM trade-off function comparisons Figure 1 shows trade-off functions for the LM, GM and SGM in low privacy (low noise) and high privacy (high noise) settings to compare the error rates in the RTM to the DPTM. The pair of asymmetric DPTM trade-off functions j, j -1 , is combined into a single symmetric function J = C( j, j -1 ) (symmetrisation/convexification). We also plot the (symmetrified) trade-off function f for the same mechanism under the DPTM for reference. All RTM curves ( j, j -1 , J ) are closer to the off-diagonal than f , indicating that A SO has lower MI attack success than A DP . The 'blessing of dimensionality' is shown in subfigure (f) : at d = 30 , stronger λ -DR is preserved compared to d = 1 . In the DPTM, the GM enjoys no such privacy amplification.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

Lap

j

Lap

j

-

1

Lap

J

Lap

(a) LM, high privacy.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

Lap

j

Lap

j

-

1

Lap

J

Lap

(b) LM, low privacy

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

GM

j

GM

j

-

1

GM

J

GM

(c) GM, high privacy

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

GM

j

GM

j

-

1

GM

J

GM

(d) GM, low privacy

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

SGM

jp

j

-

1

p

J

SGM

(e) SGM

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

f

GM

, d

= 30

J

GM

, d

= 1

J

GM

, d

= 30

(f) GM, high-dimensional

Asymptotic behaviour and power function of the GM Figure 2a shows the asymptotic behaviour of a GM with ∆ 2 / σ = 1 / 5 , d = 300 and N = 150 . The computed trade-off functions become symmetric, approach the off-diagonal and match the plot of the analytic form in Equation (7) exactly, even at a modest number of compositions. Figure 2b plots the power 1 -β of an MI adversary vs. the ∆ 2 / σ -ratio at a fixed α = 10 -3 for the GM and Figure 2c for the SGM assuming the worst case of a one-dimensional learning task and a sampling rate of 0 . 3 . The curve for the DPTM in 2c was created using the formula from [29]. As anticipated, the adversary has lower power throughout in the RTM compared to the DPTM, and the loss of power is amplified through subsampling.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

j

GM

j

-

1

GM

Analytic

(a) GM: Asymptotic behaviour

0

1

2

∆

2

/σ

10

-

3

10

-

2

10

-

1

1

-

β

RTM

DPTM

(b) GM: Power vs.

∆

2

/

σ

0

.

0

0

.

5

1

.

0

∆

2

/σ

10

-

3

10

-

2

1

-

β

RTM

DPTM

(c) SGM: Power vs.

∆

2

/

σ

MI attacks on DP ML models To test the tightness of our theoretical bounds, we now perform an empirical audit using state-of-the-art offline MI attacks on ML models trained with DP. Recall that, despite the nominal difference in the threat model, the offline MI adversary and the RTM adversary compute the same tests and thus have the same capabilities. Each auditing run produces a pair of empirical trade-off functions shown in green and blue. These are compared to the theoretical bound, shown in orange. If the empirical curves crossed below the theoretical one, this would indicate a lack of tightness in our theoretical bounds. All results are shown as averages with standard deviation across 1024 runs for a single step of a one-dimensional synthetic binary classification task, since we are interested in the worst-case scenario under the RTM. Figure 3a shows the results of the LiRa offline MI attack introduced in [12], where A SO trains shadow models on D and fits a Gaussian likelihood to their confidence scores. At ∆ 2 / σ = 1 , this logit space auditing technique already nearly perfectly matches the theoretical bound. A further improvement is achieved by implementing the very recent auditing technique presented in [11], also at ∆ 2 / σ = 1 . Here, a worst-case gradient with magnitude ∆ ('Dirac canary') is inserted and the hypothesis test takes place in gradient space instead of logit space. As seen in Figure 3b, the theoretical curve is matched exactly by the empirical one.

So far, we have assumed that A SO can leverage their knowledge of the model to design a worstcase database which influences only a single coordinate of the gradient. This is useful for auditing purposes, but does not necessarily mirror real-life training. We therefore also repeat the gradient space attack against a linear neural network with a 50 -dimensional latent space trained on the diabetes dataset [30]. Figure 3c shows that, when the adversary cannot reduce the effective dimensionality of the gradient, the hypothesis test loses substantial power compared to d = 1 (dashed line) at ∆ 2 / σ = 2 and the trade-off functions become symmetric, exactly verifying Theorem 3.

Additional results for auditing the LM and SGM are shown in the Appendix.

Benefits for ML model training We next demonstrate that the tighter privacy bounds of the RTM translate to tangible improvements in terms of accuracy/privacy trade-offs in deep learning applications. We performed experiments on three classification tasks: CIFAR-10 using ResNet-9, ImageNet using ResNet-18, and Stanford SNLI using a BERT transformer. In each case, we fixed a final privacy guarantee (in terms of ( ε, δ ( ε )) or equivalently ( α, β ( α )) ) for the DPTM and then trained a set of models up to this privacy guarantee. We then computed the DP-SGD noise scale corresponding to the same hypothesis testing privacy guarantee for the RTM, and re-trained the models for the same number of steps. For all datasets, the enhanced privacy guarantees of the RTM allowed us to obtain models with higher out-of-sample accuracy (e.g. up to 14% higher on CIFAR-10 at ε ≈ 1 ) for the nominally same membership inference guarantee by lowering the DP-SGD noise scale. These results are illustrated in Figure 4 and experimental details can be found in the Appendix.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

Bound

D

vs.

D

′

D

′

vs.

D

(a)

Logit space

attack

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

Bound

D

vs.

D

′

D

′

vs.

D

(b)

Gradient space

attack

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

Bound

d

= 1

D

vs.

D

′

D

′

vs.

D

(c) High-dimensional latent space

1

250

500

750

1000

Step

0

.

3

0

.

4

0

.

5

0

.

6

0

.

7

0

.

8

Top-1 Accuracy

RTM

DPTM

(a) CIFAR-10

1

100

200

300

400

Step

0

.

45

0

.

50

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

Top-1 Accuracy

DPTM

RTM

(b) Stanford SNLI

1

10

20

30

40

50

60

70

Epoch

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

Top-1 Accuracy

DPTM

RTM

(c) ImageNet

Improved bounds on data reconstruction attacks As discussed above, successfully defending against MI attacks implies that all weaker attacks such as data reconstruction will also be unsuccessful. In the Appendix, we demonstrate that, in the RTM, data reconstruction adversaries also suffer a diminished success rate compared to the DPTM, as measured in terms of Reconstruction Robustness [31, 32], which can be bounded directly through its relationship to hypothesis testing [33].

6 Conclusion

In this work, we consider a threat model relaxation of high practical relevance, where the adversary does not have access to the candidate record in question when performing an MI attack. Our results provide a complete characterisation of the adversary's optimal error rates under the restrictions of the RTM, including at a fixed low Type-I error rate which is particularly meaningful for privacysensitive applications such as medical or financial data [12]. Moreover, they allow for a fair and direct comparison between the privacy properties of noise mechanisms under different threat models. Our results can thus guide individuals who train ML systems on sensitive data in the selection of the appropriate noise magnitude to better balance model accuracy vs. privacy protection.

We acknowledge the following limitations. We focused on additive noise mechanisms and the global model of DP in this work. An extension of our results to local DP, mechanisms, mechanisms with discrete outputs and private selection mechanisms is a natural next step. Moreover, our analysis considers the case of non-adaptive composition, which applies to some, but not all, relevant ML tasks. An investigation of adaptive composition scenarios (considering potential correlations between mechanism outputs) would be of interest to expand the scope of our guarantees. Last but not least, we regard the analysis of adversaries who are even more limited in their background knowledge (e.g. no knowledge of the sensitivity, noise magnitude or noise type) as a promising avenue for future work.

References

Nicholas Carlini et al. 'The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks.' In: USENIX Security Symposium . Vol. 267. 2019.

Vitaly Feldman. 'Does learning require memorization? a short tale about a long tail'. In: Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing . 2020, pp. 954-959.

Reza Shokri et al. 'Membership inference attacks against machine learning models'. In: 2017 IEEE symposium on security and privacy (S&P) . IEEE. 2017, pp. 3-18.

Saeed Mahloujifar et al. 'Optimal Membership Inference Bounds for Adaptive Composition of Sampled Gaussian Mechanisms'. In: arXiv preprint arXiv:2204.06106 (2022).

Cynthia Dwork et al. 'Calibrating noise to sensitivity in private data analysis'. In: Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3 . Springer. 2006, pp. 265-284.

Larry Wasserman and Shuheng Zhou. 'A statistical framework for differential privacy'. In: Journal of the American Statistical Association 105.489 (2010), pp. 375-389.

Jinshuo Dong, Aaron Roth, and Weijie Su. 'Gaussian Differential Privacy'. In: Journal of the Royal Statistical Society (2021).

Jerzy Neyman and Egon Sharpe Pearson. 'IX. On the problem of the most efficient tests of statistical hypotheses'. In: Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character 231.694-706 (1933), pp. 289-337.

Milad Nasr et al. 'Adversary instantiation: Lower bounds for differentially private machine learning'. In: 2021 IEEE Symposium on Security and Privacy (SP) . IEEE. 2021, pp. 866-882.

Matthew Jagielski, Jonathan Ullman, and Alina Oprea. 'Auditing differentially private machine learning: How private is private SGD?' In: Advances in Neural Information Processing Systems 33 (2020), pp. 22205-22216.

Milad Nasr et al. 'Tight Auditing of Differentially Private Machine Learning'. In: arXiv preprint arXiv:2302.07956 (2023).

Nicholas Carlini et al. 'Membership inference attacks from first principles'. In: 2022 IEEE Symposium on Security and Privacy (SP) . IEEE. 2022, pp. 1897-1914.

Lauren Watson et al. 'On the importance of difficulty calibration in membership inference attacks'. In: arXiv preprint arXiv:2111.08440 (2021).

Jiayuan Ye et al. 'Enhanced membership inference attacks against machine learning models'. In: Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security . 2022, pp. 3093-3106.

Martin Abadi et al. 'Deep learning with differential privacy'. In: Proceedings of the 2016 ACM SIGSAC conference on computer and communications security . 2016, pp. 308-318.

Samuel Maddock, Alexandre Sablayrolles, and Pierre Stock. 'CANIFE: Crafting Canaries for Empirical Privacy Measurement in Federated Learning'. In: arXiv preprint arXiv:2210.02912 (2022).

Yugeng Liu et al. ' { ML-Doctor } : Holistic Risk Assessment of Inference Attacks Against Machine Learning Models'. In: 31st USENIX Security Symposium (USENIX Security 22) . 2022, pp. 4525-4542.

Alexandre Sablayrolles et al. 'White-box vs black-box: Bayes optimal strategies for membership inference'. In: International Conference on Machine Learning . PMLR. 2019, pp. 55585567.

Zuxing Li, Tobias J Oechtering, and Deniz Gündüz. 'Privacy against a hypothesis testing adversary'. In: IEEE Transactions on Information Forensics and Security 14.6 (2018), pp. 15671581.

Benedetta Tondi, Mauro Barni, and Neri Merhav. 'Detection games with a fully active attacker'. In: 2015 IEEE International Workshop on Information Forensics and Security (WIFS) . IEEE. 2015, pp. 1-6.

Kamalika Chaudhuri, Jacob Imola, and Ashwin Machanavajjhala. 'Capacity bounded differential privacy'. In: Advances in Neural Information Processing Systems 32 (2019).

Yuqing Zhu, Jinshuo Dong, and Yu-Xiang Wang. 'Optimal accounting of differential privacy via characteristic function'. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2022, pp. 4782-4817.

Sivakanth Gopi, Yin Tat Lee, and Lukas Wutschitz. 'Numerical composition of differential privacy'. In: Advances in Neural Information Processing Systems 34 (2021), pp. 11631-11642.

Peter Kairouz, Sewoong Oh, and Pramod Viswanath. 'The composition theorem for differential privacy'. In: International conference on machine learning . PMLR. 2015, pp. 1376-1385.

E.L. Lehmann and Joseph P. Romano. Testing Statistical Hypotheses . 4th ed. Singapore: Springer Nature, 2022. ISBN: 978-3-030-70578-7.

Damien Desfontaines and Balázs Pejó. 'Sok: differential privacies'. In: Proceedings on privacy enhancing technologies 2020.2 (2020), pp. 288-313.

Yucui Liu and Tomasz J Kozubowski. 'A folded Laplace distribution'. In: Journal of Statistical Distributions and Applications 2.1 (2015), pp. 1-17.

H. Urkowitz. 'Energy detection of unknown deterministic signals'. In: Proceedings of the IEEE 55.4 (1967), pp. 523-531.

Zhiqi Bu et al. 'Deep learning with gaussian differential privacy'. In: Harvard data science review 2020.23 (2020), pp. 10-1162.

Bradley Efron et al. 'Least angle regression'. In: The Annals of Statistics 32.2 (2004), pp. 407499.

Borja Balle, Giovanni Cherubin, and Jamie Hayes. 'Reconstructing training data with informed adversaries'. In: 2022 IEEE Symposium on Security and Privacy (SP) . IEEE. 2022, pp. 11381156.

Jamie Hayes, Saeed Mahloujifar, and Borja Balle. 'Bounding Training Data Reconstruction in DP-SGD'. In: arXiv preprint arXiv:2302.07225 (2023).

Georgios Kaissis et al. 'Bounding data reconstruction attacks with the hypothesis testing interpretation of differential privacy'. In: arXiv preprint arXiv:2307.03928 (2023).

Samuel Karlin and Herman Rubin. 'The theory of decision procedures for distributions with monotone likelihood ratio'. In: The Annals of Mathematical Statistics (1956), pp. 272-299.

Antonio De Maio, Steven M Kay, and Alfonso Farina. 'On the invariance, coincidence, and statistical equivalence of the GLRT, Rao test, and Wald test'. In: IEEE Transactions on Signal Processing 58.4 (2009), pp. 1967-1979.

Jess Marcum. 'A statistical theory of target detection by pulsed radar'. In: IRE Transactions on Information Theory 6.2 (1960), pp. 59-267.

Helena Klause et al. 'Differentially private training of residual networks with scale normalisation'. In: arXiv preprint arXiv:2203.00324 (2022).

Soham De et al. 'Unlocking high-accuracy differentially private image classification through scale'. In: arXiv preprint arXiv:2204.13650 (2022).

Jacob Devlin et al. 'Bert: Pre-training of deep bidirectional transformers for language understanding'. In: arXiv preprint arXiv:1810.04805 (2018).

Alexey Kurakin et al. 'Toward training at imagenet scale with differential privacy'. In: arXiv preprint arXiv:2201.12328 (2022).

Qinqing Zheng et al. 'Sharp composition bounds for Gaussian differential privacy via Edgeworth expansion'. In: International Conference on Machine Learning . PMLR. 2020, pp. 1142011435.

Alexandre Lacoste et al. 'Quantifying the Carbon Emissions of Machine Learning'. In: arXiv preprint arXiv:1910.09700 (2019).

7 Supplementary Material

7.1 Additional results

Subsampling and auditing the Laplace Mechanism As mentioned in the section on the SGM, numerical subsampling can also be applied to mechanisms other than the GM. Figure 5a shows the trade-off function curves for the LM without subsampling in orange and the subsampled curve for ∆ 1 / b = 1 , p = 0 . 3 in blue. Similar to the GM, the MI error rates in the RTM (and thus the privacy guarantees) are amplified with sub-sampling.

Figure 5b complements the results on empirical privacy auditing for the LM from the main manuscript. The gradient space auditing technique [11] is again applied to a one-dimensional learning task as described in the main manuscript, however the LM is used instead of the GM. Once more, the empirical privacy audit (green and blue curves) exactly matches the theoretical bounds (shown in orange). As in the main manuscript, curves are averages and standard deviations of 1024 repetitions.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

J

Lap

jp

Lap

j

-

1

p

Lap

J

(a) Subsampled LM (SLM)

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

Bound

D

vs.

D

′

D

′

vs.

D

(b) LM: Empirical auditing

Auditing the SGM Figure 6 shows additional results on privacy auditing when the SGM is used. Here, a single step of DP-SGD with ∆ 2 / σ 2 = 1 . 5 and p = 0 . 3 is audited using the gradient space auditing technique from [11] as described above. Once more, the empirical privacy audit (green and blue curves) exactly matches the theoretical bounds (shown in orange). As in the main manuscript, curves are averages and standard deviations of 1024 repetitions.

0

.

0

0

.

5

1

.

0

α

0

.

0

0

.

5

1

.

0

β

Bound

D

vs.

D

′

D

′

vs.

D

Bounding data reconstruction As discussed in the main manuscript, successfully bounding MI implies an ability of the mechanism to also bound all weaker attacks. It is thus possible to translate our MI bounds into a direct and tight bound the probability of a successful data reconstruction attack by using the Reconstruction Robustness (ReRo) framework [31, 32]. In brief, the goal of the ReRo adversary is to obtain a successful reconstruction (from model weights/gradients), defined as a reconstruction loss ≤ η , for example mean squared error or perceptual loss. The ReRo framework models the adversary's prior knowledge as the probability κ ( η ) of a successful reconstruction before/without observing the model weights/gradients. Satisfying ReRo then requires the probability of reconstruction after observing the model weights/gradients to be ≤ γ . In summary, a privacy mechanism satisfies ( η, γ ) -ReRo against an adversary, if, given a prior probability κ ( η ) of successful reconstruction (i.e. of reconstruction error ≤ η ), the probability of reconstruction after seeing the mechanism's outputs (e.g. model weights or gradients) is no larger than γ . In recent work, [33] show that, if the trade-off function of a privacy mechanism is known, the probability of reconstruction γ can be computed directly, allowing us to extend our theoretical membership inference guarantees to tight bounds on the probability of success of reconstruction attacks. Concretely, it is shown that, if a privacy mechanism's trade-off function f is known, then γ = 1 -f ( κ ( η )) . Since λ -DR is expressed in terms of trade-off functions, the relaxed threat model we study extends naturally to this type of attack. We conducted experiments under (highly) pessimistic assumptions about the adversary: We assume that the adversary has a prior probability of successful reconstruction κ ( η ) = 0 . 1 (a more optimistic baseline would be a uniform prior over the entire dataset, for example ≈ 10 -6 for ImageNet). Moreover, following the pessimistic assumption made above, i.e. that the adversary can design a database which influences only a single entry in the weight/gradient vector, we assumed d = 1 for the construction of all trade-off functions in the experiments. We then analysed the DP-SGD applications mentioned above (CIFAR-10, ImageNet and Stanford SNLI), whereby we used the Edgeworth expansion technique introduced in [33] to estimate the ReRo upper bound. Even under these permissive assumptions, our findings, detailed in the Figure below, show that the relaxed threat model adversary has a substantially lower probability of success in reconstructing model inputs compared to the DP adversary. For details on the computation of ε , see below.

0.5

10

20

40

60

ε

(

δ

= 10

-

5

)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prob. recon. (

γ

)

DPTM

RTM

(a) CIFAR-10

0.5

10

20

30

40

50

ε

(

δ

= 1

.

8

·

10

-

6

)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prob. recon. (

γ

)

DPTM

RTM

(b) Stanford SNLI

1

20

40

60

80

ε

(

δ

= 10

-

6

)

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Prob. recon. (

γ

)

DPTM

RTM

(c) ImageNet

7.2 Proofs

Lemma 1. In the RTM, no UMP level α test exists for all possible values of q ( D ′ ) , i.e. ∀ θ ∈ Θ \ Θ 0 . Thus, H 0 vs. H 1 (and H 1 vs. H 0 ) are each not decidable by a single test.

Proof. The claim follows immediately from the Neyman-Pearson Lemma [8]. For a detailed treatment, we refer to Section 3.2. of [25].

Lemma 2. The composite LR test H 0 : θ ∼ Z ( q ( D ) , ξ ) vs. H 1 : θ ∼ Z ( q ( D ′ ) , ξ ) is equivalent to letting θ ∼ Z ( µ, ξ ) and simultaneously conducting two individual one-sided LR tests r and r ′ with null hypothesis µ = 0 and alternative hypotheses -∆ ≤ µ < 0 for r and 0 < µ ≤ ∆ for r ′ , where µ = q ( D ′ ) -q ( D ) and ∆ is the global sensitivity of q .

Proof. We will first show that testing H 0 : Z ( q ( D ) , ξ ) vs. H 1 : Z ( q ( D ′ ) , ξ ) is equivalent to testing Z (0 , ξ ) vs. Z ( q ( D ′ ) -q ( D ) , ξ ) . We know that Z ( µ, ξ ) is a shift invariant probability measure which admits a density. Thus, for any c, θ ∈ R :

The LR for Z ( q ( D ) , ξ ) vs. Z ( q ( D ′ ) , ξ ) is:

Since the LR may be evaluated at an θ , we can evaluate it at θ + q ( D ) , since q ( D ) is a known constant:

Therefore, we have shown that testing Z ( q ( D ) , ξ ) vs. Z ( q ( D ′ ) , ξ ) is equivalent to testing Z (0 , ξ ) vs. Z ( q ( D ′ ) -q ( D ) , ξ ) , as the LR for any θ under both hypothesis testing problems will be the same. Now, let µ = q ( D ′ ) -q ( D ) . Since A SO does not know the value of q ( D ′ ) , they can only deduct that ∥ µ ∥ ≤ ∆ . Thus, the two cases µ > 0 and µ < 0 must be considered simultaneously and individually. The claim follows. Observe that the proof is identical for the reversed test H 0 : Z ( q ( D ′ ) -q ( D ) , ξ ) vs. H 1 : Z (0 , ξ ) , i.e. with the null and alternative hypothesis switched.

Lemma 3. The individual LR tests r and r ′ are UMP level α if the noise distribution Z has the monotone likelihood ratio property (MLRP). The Laplace Mechanism (LM), Gaussian Mechanism (GM) and the Poisson-Subsampled Gaussian Mechanism (SGM) all have the MLPR. Moreover, the power of both tests is maximised when ∥ µ ∥ = ∥ q ( D ′ ) -q ( D ) ∥ = ∆ .

Proof. The existence of an UMP level α LR test for distributions with the MLRP is justified by the Karlin-Rubin Theorem [34], which states that, when the null hypothesis is simple, the alternative hypothesis is one-sided and the LR is a monotone function of the parameter of interest (MLRP), the LR test is UMP level α ∀ θ ∈ Θ \ Θ 0 .

More precisely, the MLRP implies that the LR is non-decreasing or non-increasing in the parameter of interest (in this case θ ) of a test. We must consider the cases µ < 0 and µ > 0 individually for each mechanism. Throughout, let P = Z (0 , ξ ) and Q = Z ( µ, ξ ) . We will replace the placeholder Z with the distribution of interest as required and consider both tests P vs. Q and Q vs. P .

For the Laplace Mechanism (LM), we have:

For µ > 0 , log P ( θ ) Q ( θ ) is non-increasing in θ and for µ < 0 it is non-decreasing in θ . The opposite holds for log Q ( θ ) P ( θ ) by symmetry.

For the Gaussian Mechanism (GM), we have:

For µ > 0 , log P ( θ ) Q ( θ ) is non-increasing in θ and for µ < 0 , it is non-decreasing in θ . Again, the opposite holds for log Q ( θ ) P ( θ ) by symmetry. Observe that the LRs are independent of dimensionality for the (S)GM, hence it suffices to consider the scalar case.

Finally, for the Poisson-Subsampled Gaussian mechanism (SGM), we have:

The computation here is slightly more involved. Taking the derivatives, we have:

Both numerators are positive when µ is positive and negative otherwise. For d d θ log P ( θ ) Q ( θ ) , the denominator is negative when µ is positive, while for d d θ log Q ( θ ) P ( θ ) , it is negative. Thus, when µ is positive, log P ( θ ) Q ( θ ) is non-increasing in θ and non-decreasing otherwise. Conversely, log Q ( θ ) P ( θ ) is non-decrasing with µ positive and non-increasing otherwise.

So far, we have proven that (1) the LM, GM and SGM have the MLRP and (2) thus, by Karlin-Rubin, there exists an UMP level α test for Z (0 , ξ ) vs. Z ( µ, ξ ) as well as for the reverse Z ( µ, ξ ) vs. Z (0 , ξ ) for both r ′ ( µ > 0 ) and r ( µ < 0 ). We note that the proof could have also been achieved by using the fact that the Laplace and Gaussian distributions with known scale are part of a specific one-parameter exponential family, which has the MLRP property. Compare [25], Section 4.2. It remains to show that the power is maximised for µ = ∆ or µ = -∆ , i.e. that the power increases with the effect size, in this case ∆ / ξ , with ξ assumed fixed.

First, observe that we can reduce our workload by half as, by shift invariance, the tests r ′ : Z (0 , ξ ) vs. Z ( µ, ξ ) and r : Z (0 , ξ ) vs. Z ( -µ, ξ ) as well as their reverses are all equivalent. It remains to show that the power function (i.e. 1 -β ( µ ) at a fixed α ) is maximised at µ = ∆ . In fact, it suffices to show that the power function is strictly increasing as a function of µ , since ∆ is the largest value µ can take. The power function is a re-parameterised version of the trade-off function for the one-sided UMP level α test (note that this is not the combined test but one of the individual component tests the combined test is equivalent to performing the two component tests simultaneously and necessarily loses power as it represents multiple testing which increases the false discovery rate). Fix a level α .

For the LM, we have [7]:

It is easily verified that 1 -β (Lap(0 , b ) , Lap( µ, b ) | α ) is strictly increasing in µ ∀ α . For the GM, we have [7]:

where Φ , Ψ -1 are the CDF and and iSF, respectively. Recall that the function is defined on [0 , 1] × [0 , 1] . Taking the derivative with respect to µ and observing that Ψ -1 ( x ) = Φ -1 (1 -x ) we obtain:

where g is the PDF of the standard normal. The second term simplifies to:

Thus, we obtain:

Since σ is a positive constant and g ( x ) > 0 ∀ x , the derivative is negative for all values of µ . Thus, β ( N (0 , σ 2 ) , N ( µ, σ 2 ) | α ) is decreasing in µ and thus 1 -β is increasing in µ as required.

Finally, for the SGM, we can invoke Lemma 9 below and re-use the proof for the GM. Fix 0 < p < 1 . Lemma 9, specialised to the SGM, states:

The second term is independent of µ , thus the claim follows from the proof for the GM.

In conclusion, we have shown that, in the RTM, the adversary must conduct two pairs of individual and simultaneous UMP level α LR tests whose power is maximised when µ = ± ∆ , i.e. at the dominating pair distributions [22]. This conforms to the intuition that the power of the test should increase when the distributions are 'further apart', i.e. when the effect size increases. Moreover, identically to the DPTM, it allows us to consider only the dominating pairs when constructing the trade-off functions, since no other test between the distributions of the mechanism on the two databases can have higher power at the same level α than the test(s) between the dominating pair(s). We have thus established that the optimal tests the sub-optimal adversary A SO can construct against the aforementioned noise mechanisms in the RTM are UMP level α tests. These tests are necessarily less powerful than the uniformly most powerful tests that A NP can construct because they are only optimal for a subset of the values of q ( D ′ ) at a time, whereas the uniformly most powerful tests of A NP are optimal for all values of q ( D ′ ) simultaneously.

Lemma 4. If the density of Z is additionally symmetric about the location parameter, the simultaneous individual LR tests r and r ′ are equivalent to performing a single test using the magnitude of the observation as a test statistic, i.e. letting ∥ θ ∥ ∼ ∥Z ( µ, ξ ) ∥ and testing H 0 : ∥ µ ∥ = 0 vs. H 1 : ∥ µ ∥ > 0 . The power of this 'combined' test is maximised at ∥ µ ∥ = ∆ .

Proof. Observe that H 1 can be split up into two hypotheses: H 1 a : µ > 0 and H 1 b : µ < 0 . Now, we can conduct two simultaneous tests E 1 : H 0 vs. H 1 a and E 2 : H 0 vs. H 1 b . The LRs are:

For E 1 : Λ 1 ( θ ) = log L ( θ |H 1 a ) L ( θ |H 0 ) .

For E 2 : Λ 2 ( θ ) = log L ( θ |H 1 b ) L ( θ |H 0 ) .

The rejection region for these tests will be { Λ 1 > c 1 } and { Λ 2 > c 2 } , where c 1 , c 2 are the critical values. The distributions for both Λ 1 and Λ 2 under H 0 are identical. It thus suffices to consider the numerator of the LR, which, from the MLRP, is monotone in θ . Cancelling the logarithms, we may thus simplify and consider Λ 1 = θ and Λ 2 = -θ . To maintain the same level α for all tests, the critical values should be equal: c = ˆ c 1 = ˆ c 2 , where the hat denotes the new critical values after simplification. We remark for completeness that this will render the tests unbiased. Since these tests are UMP and unbiased, they are also called UMP unbiased (i.e. the tests with the greatest power in the set of unbiased tests). For further details, we refer to [25], Section 4. For the combined test, the rejection region is thus: { Λ 1 > ˆ c 1 } ∪ { Λ 2 > ˆ c 2 } = { θ > c } ∪ {-θ > c } = {| θ | > c } . This proves the first claim. The claim that the power is maximised when ∥ µ ∥ = ∆ follows from the previous observation and the proof to Lemma 3. For the reverse tests, the proof is identical.

Lemma 5. For a (not necessarily symmetric) Z the Generalised Likelihood Ratio Test (GLRT) using the test statistic log L ( θ |Z (ˆ µ,ξ )) L ( θ |Z (0 ,ξ )) , where ˆ µ is the maximum likelihood estimate of µ is equivalent to simultaneously conducting r and r ′ . The GLRT's power is maximised at ˆ µ = ∆ .

Proof. For the concrete problem µ = 0 vs ∥ µ | > 0 , the GLRT test statistic can be written:

For a single observation θ (i.e. absent other information) the MLE for the mean ˆ µ is equal to the observed value of θ . Thus, the GLRT becomes:

Compare the GLRT to the two simultaneous tests:

r : The LR test statistic is Λ 1 ( θ ) = log L ( θ |Z ( µ,ξ )) L ( θ |Z (0 ,ξ )) for µ > 0 ;

r ′ : The LR test statistic is Λ 2 ( θ ) = log L ( θ |Z ( µ,ξ )) L ( θ |Z (0 ,ξ )) for µ < 0 .

Observe that these are identical to Λ( θ ) when the restricted MLE is used, provided the same critical value is chosen. Thus, the tests are equivalent at the same level α . The claim that the power is maximised when ˆ µ = ∆ then follows from the previous lemma. For the reverse tests, the proof is identical. We note for completeness that the same result could have been achieved using the Wald or Rao tests, which are derived differently, but are statistically equivalent to the GLRT [35].

Lemma 6. If a mechanism satisfies f -DP, it also satisfies f ′ -DR, with f ′ ( α ) ≥ f ( α ) ∀ α ∈ [0 , 1] .

Proof. The lemma is essentially a re-stating of Lemma 1 in the language of f-DP. Since a UMP level α test does not exist for all values of q ( D ′ ) , any test that A SO constructs is less powerful for at least a subset of the parameter space, and thus the resulting trade-off function in the RTM will be on or above (dominating) the trade-off function in the DPTM. An alternative way to look at this is through the post-processing guarantee of f-DP. As an example, consider using the test statistic ∥ θ ∥ as discussed above. As θ is the output of a mechanism, we can consider it a post-processing step on M ( D/D ′ ) . For f-DP, it holds that [7]:

for any trade-off function T . The claim follows by setting the LHS to f ′ , which is the trade-off function constructed using the test statistic ∥ θ ∥ in the RTM and the RHS to f , which is the trade-off function of the (NP) UMP level α test in the DPTM.

Lemma 7. If a mechanism satisfies f -DP and f ′ -DR with f ′ ≥ f , arbitrary post-processing can only deteriorate privacy up to the f -DP guarantee.

Proof. This is essentially the contrapositive to Lemma 6. Since f-DP is resilient to post-processing, the worst deterioration of the λ -DR-guarantee can never exceed the f-DP guarantee.

Before proceeding with the proofs about the closed-form/numerical expressions of the trade-off functions, we re-state Definition 2 and provide a brief derivation.

Definition 2 (Trade-off function construction) . Let P, Q be the distributions of a test statistic under H 0 and H 1 , respectively and let Φ P/Q denote their cumulative distribution function (CDF), Ψ P/Q their survival function (SF) and Φ -1 P/Q , Ψ -1 P/Q their respective inverses (iCDF , iSF). Then, we define:

Proof. The proof is standard. We reproduce it here for the purpose of being self-contained. Consider a one-tailed test for a parameter θ . From basic hypothesis testing theory, given a test statistic Λ( θ ) and a critical value c , α = Pr (Λ( θ ) > c |H 0 ) and β = Pr (Λ( θ ) < c |H 1 ) . Letting P be the distribution of the test statistic under H 0 and Q its distribution under H 0 , we have:

Since the trade-off function is T : α ↦→ β ( α ) , we solve the RHS for c and have:

T -1 follows by inverting T . The same form manifests by considering T -1 the trade-off function for the test with the null and alternative hypotheses exchanged (the reverse test). The role of P and Q will be played by the test statistics for the various mechanisms below. We note that, since the functions are inverses of each other, we may arbitrarily designate one as T and the other as T -1 .

Theorem 1. Let µ 1 = ∆ 1 / b . The LM satisfies C( j Lap ( α ) , j -1 Lap ( α )) -DR with:

and

Proof. j ( α ) is the trade-off function corresponding to the test r : Lap(0 , b ) vs. Lap(∆ 1 , b ) and j -1 ( α ) is the trade-off function for r ′ : Lap(∆ 1 , b ) vs. Lap(0 , b ) . Since the density of the Laplace distribution is symmetric, we can invoke Lemma 4 and use the absolute value of the observation as a test statistic.

For r , under H 0 , the test statistic ∥ θ ∥ follows an exponential distribution with scale parameter b . Under H 1 , the test statistic ∥ θ ∥ follows a folded Laplace (FoldL) distribution [27] with mean ∆ 1 and scale parameter b . For r ′ , the hypotheses and distributions are reversed. To construct the trade-off functions, we require the CDF Φ , and iCDF Φ -1 . From these, we can construct Ψ( x ) = 1 -Φ( x ) and Ψ -1 ( x ) = Φ -1 (1 -x ) . All are available for both distributions in closed form. We have:

and

j and j -1 then follow by substitution in the expressions for the trade-off functions from Definition 2 above:

Theorem 2. Let µ 2 = ∆ 2 / σ . The GM satisfies C( j GM ( α ) , j -1 GM ( α )) -DR with:

Proof. The proof is similar to Theorem 1. j ( α ) is the trade-off function corresponding to the test r : N (0 , σ 2 ) . vs. N (∆ 2 , σ 2 ) and j -1 ( α ) corresponds to the test r ′ : N (∆ 2 , σ 2 ) vs. r : N (0 , σ 2 ) . The Gaussian density is also symmetric, hence we can appeal to Lemma 4 and use the magnitude of θ as the test statistic once more. In fact, we will use the squared magnitude for numerical reasons, which does not change the monotonicity behaviour of the test statistics.

For r , under H 0 , the test statistic ∥ θ ∥ 2 follows a central chi-squared distribution with d degrees of freedom scaled by σ 2 , i.e. χ 2 d (0 , σ 2 ) . Under H 1 , the test statistic ∥ θ ∥ 2 follows a noncentral chi-squared distribution with d degrees of freedom and noncentrality parameter µ 2 2 = ∆ 2 2 / σ 2 , i.e. χ 2 d ( µ 2 2 , σ 2 ) . For r ′ , the hypotheses and distributions are reversed. To construct the trade-off functions, we require the CDF Φ , and iCDF Φ -1 . From these, we can construct Ψ( x ) = 1 -Φ( x ) and Ψ -1 ( x ) = Φ -1 (1 -x ) as above. For the central chi-squared distribution, Φ has an analytic form, but Φ -1 does not. For the non-central chi-squared distribution, Φ is expressed in terms of the Marcum Q-function [36], while Φ -1 once again has no analytic form. However, high-precision numerical implementations of all the aforementioned functions are widely available. We thus express j and j -1 in terms of these abstract functions and implement them numerically throughout. The concrete expressions follow directly from Definition 4. Moreover, for d = 1 , a closed form is available for j , which is shown in the upcoming Corollary.

Corollary 1. For d = 1 , j ( α ) admits the following closed-form representation:

Proof. For this proof, it helps to picture Definition 2 with the role of the test statistic played once again by ∥ θ ∥ 2 . Thus, α = Pr ( ∥ θ ∥ 2 > c 2 |H 0 ) and β = Pr ( ∥ θ ∥ 2 < c 2 |H 1 ) . We can take the square of the critical value since the magnitude is always non-negative and we may choose the critical value arbitrarily; it will also simplify the computations below. Recall that d = 1 because θ is now a scalar.

From the proof to Theorem 2, we have: α ( c 2 ) = Ψ χ 2 1 (0 ,σ 2 ) ( c 2 ) and β ( c 2 ) = Φ χ 2 1 ( µ 2 2 ,σ 2 ) ( c 2 ) . Ψ χ 2 1 (0 ,σ 2 ) admits an analytical form:

where γ is the lower incomplete gamma function, Γ the gamma function and erf , erfc are the error function and complementary error function of the Gaussian distribution, respectively. We can now exploit the following pattern:

so the term in Equation (31) can be written as 2Ψ N (0 , 1) ( c σ ) . Since Ψ N (0 , 1) is invertible, we have that c = σ Ψ -1 N (0 , 1) ( α 2 ) .

Similarly, Φ χ 2 1 ( µ 2 2 ,σ 2 ) ( c 2 ) admits a closed-form representation. Recall that µ 2 2 = ∆ 2 2 / σ 2 .

where Q M 1 2 is the Marcum Q-function of order 1 2 [36].

Substituting the expression for c from above, we obtain:

The Marcum Q-function of order 1 2 also admits a closed form:

Using the pattern 1 2 erfc ( k √ 2 ) = Ψ N (0 , 1) ( k ) as above, we rewrite Equation (38) as:

Finally, we substitute the arguments from Equation 37 and obtain:

which is the desired form and completes the proof.

Lemma 8. Let GM a , GM b be GMs with noise variances σ 2 a I d , σ 2 b I d on functions with sensitivities ∆ 2 a , ∆ 2 b , respectively. Then, the non-adaptively composed mechanism GMC has trade-off functions:

Proof. Let θ a , θ b ∈ R d be uncorrelated mechanism outputs on two independent queries. Since A SO passively observes both outputs before testing, the test statistic becomes:

Of note, the reason why the mean of the observations is the correct choice here follows from Lemma 5, as the MLE of the mean of a normal distribution for > 1 observation is the sample average. By the additive properties of isotropic Gaussian noise, the distributions of the observations thus are:

Under H 0 , the test statistic is thus distributed as:

while under H 1 it is distributed as:

From here, the proof continues identically to the proof to Theorem 2 above, from which the claim follows. The generalisation to > 2 compositions follows inductively. In particular, for N homogeneous compositions (i.e. with identical sensitivity ∆ 2 and noise variance σ 2 ), the test statistic under H 0 is distributed as:

while under H 1 it is distributed as:

which follows directly from the above. We will use this property in the following proof.

Theorem 3 (Blessing of dimensionality in the RTM) . Consider a GM on a function with sensitivity ∆ 2 and noise variance σ 2 I d such that ∆ 2 / σ ≪ 1 . Let µ 2 = ∆ 2 / σ . As d and/or as the number of non-adaptive compositions N increase, j GM and j -1 GM tend to the common form:

Proof. To prove the claim, we will appeal to the central limit theorem (CLT). We will use the facts that the mean of the central chi-squared distribution with d degrees of freedom is d and its variance is 2 d . The noncentral chi-squared distribution with d degrees of freedom and noncentrality κ has mean d + κ and variance 2 d +4 κ . Under the CLT, as N → ∞ and/or as d → ∞ , we thus have convergence in distribution as follows. Letting σ 2 N = δ (compare Equations (47) and (48)):

We now proceed as in the proofs above and consider the test statistic under the null and alternative hypotheses and a critical value c We have:

and

For the trade-off function, we substitute to obtain:

Substituting κ ← N ∆ 2 2 / σ 2 (i.e. the degrees of freedom, compare Equation (48)) and separating the terms, we obtain:

Letting µ 2 = ∆ 2 / σ , we obtain the form of the LHS of Equation (7).

To obtain the RHS of Equation (7), we further massage Equation (55). Concretely, we let κ / d := ζ and Taylor expand the equation around ζ = 0 to obtain:

When d is large, O ( ζ 2 ) vanishes and √ d 2 dominates the term in the parentheses, yielding:

Finally, recursively substituting ζ ← κ / d and κ ← N ∆ 2 2 / σ 2 , we get:

Letting ∆ 2 2 / σ 2 = µ 2 2 , we obtain the RHS of Equation (7). The fact that the two trade-off functions become symmetrical follows this equation's functional form, which corresponds to the trade-off function between two Gaussians with the same variance, i.e. f GM .

Lemma 9. Let T ( A,B )( α ) be a trade-off function between two general distributions A,B representing mechanism outputs. The trade-off functions for the sub-sampled mechanisms are given by T ( A, (1 -p ) A + pB )( α ) = pT ( A,B )( α ) + (1 -p )(1 -α ) and by its inverse.

Proof. Of note, this fact was first observed in [7] and also proved in [22]. We provide a more concise proof here for the purpose of self-containedness. Let Φ , Ψ and their inverses be defined for A and B as above. By Definition 2, we have:

where T is a generic trade-off function, in our case j or j -1 . We want to show that for a mixture of distributions A and B , with mixture coefficient p ∈ (0 , 1) :

Define the CDF of the mixture distribution C as Φ C = (1 -p )Φ A + p Φ B . Now, we have:

The inverse T -1 is not always easy to construct and, in practice, is usually computed numerically. We have thus shown that we can construct the trade-off functions of a sub-sampled mechanism directly from the trade-off functions of the original mechanism, even if none of them has an analytical form.

7.3 Experimental details

Privacy auditing experiments All experiments were conducted on a single workstation computer equipped with 64 GB of RAM and a 12-core Intel Xeon CPU with a Thermal Design Power of 105 Wand implemented in the Python programming language. For the synthetic learning tasks, a dataset of 1025 samples with one feature each, split evenly across two classes, was generated using scikit-learn 's make_classification method. Of these, 1024 samples were used for shadow model training and a single example ( x ∗ ) was held out and its label flipped, similarly to the data poisoning technique of [12] to maximise the point's impact on the model's confidence. This process was repeated for 1024 iterations, and shadow models were trained. No fixed random seed was used. Shadow models were trained in pytorch using full-batch DP gradient descent for the experiments in the main manuscript and Poisson sampling for the experiment in the Appendix, with a sample-wise ℓ 2 norm bound of 1 , a noise scale of 1 and implemented using a single linear layer with a single unit. Models were trained to convergence on the training database, at which point the training loss was < 0 . 1 . In every case, we considered a single step of model training, i.e. disregarded all previous model updates. For the logit-space auditing, logits were transformed by applying the logistic sigmoid activation function, converted to confidence scores and a Gaussian likelihood fitted as described in the Likelihood Ratio Attack Section IV.C of [12]. For executing the offline MI attacks, we followed Algorithm 1 of [12] with the modifications described to implement offline MI. In brief, likelihoods were fitted to the so-called IN models, i.e. the ones the adversary is assumed to have access to in the RTM. The models' confidence was then measured on the unseen sample x ∗ to determine its membership status by implementing the one-sided hypothesis test described in [12] for offline MI. To compute the trade-off functions, the test statistic was evaluated at 50 critical values using scikit-learn 's ROC functionality, as the ROC and the trade-off function are complements of each other. Results are reported as averages and standard deviations over the β at each critical value α , whereby the critical values were chosen identically for each repetition. For the gradient space auditing, the gradients of the final training iteration, when norms were lowest, were used as a reference 'database' ( D ). To these, a gradient with a magnitude of ∆ 2 , i.e. equal to the clipping threshold was added to simulate the maximal effect of x ∗ , similar to the Dirac canary technique described in [11]. Noise was then added to all gradient samples. Then, the magnitude of the gradients was used as a test statistic as discussed in Appendix A.B. of [12], and as proven to correspond to the optimal hypothesis test under the RTM in Lemma 4. Similar to above, the test statistic was evaluated at 50 critical value settings and the ROCs were computed. For the high-dimensional experiment, we utilised the diabetes dataset [30], a 10 -dimensional regression dataset with 442 samples. The same strategy was followed as in the gradient space auditing section above, but this time, a model with a latent space dimensionality of 50 was implemented. The noise multiplier was set to 0 . 5 . Once again, a single Dirac canary gradient was added to the gradient 'database' and the ROCs were computed as above. The experiments did not require a graphics processing unit and required approximately 60 minutes of net computation time on a CPU. At an assumed carbon efficiency of 0 . 432 kg/kWh, the computation of these experiments caused approximately 0 . 04 kg of CO 2 equivalent emissions.

Deep learning with DP-SGD CIFAR10 was trained from scratch using the ResNet9 architecture from [37] and a subset of the techniques presented in [38] ( 2 augmentations per draw but no expo-

nential moving average on the weights) to (1 , 10 -5 ) -DP. We used a sampling rate of 4 096 / 50 000 , a noise scale of 10 . 5 and trained for 1 000 steps in total. For SNLI, we fine-tuned the final 3 layers of a 105 M-parameter pre-trained BERT Transformer [39] to (1 , 1 . 8 · 10 -6 ) -DP using a sampling rate of 4 096 / 549 361 , a noise scale of 1 . 15 and 405 steps. For ImageNet, we fine-tuned the final 12 layers of a ResNet18 architecture pre-trained on Places365 following the exact hyperparameters from [40] to (10 , 10 -6 ) -DP. For DP accounting, we used the Rényi-DP accountant from the Google Differential Privacy library, whereas for acounting for privacy loss in the RTM, we computed the product distributions of the test statistics as described in [7], Definition 3.1. ff. using direct numerical integration to 50 digits of precision using the mpmath library. To determine the noise scale for training the models under the RTM, we adapted the technique described in [41]. In brief, we determined the upper and lower noise scale values for which the RTM trade-off functions ( λ -functions) optimally 'sandwich' the f -DP trade-off function and then chose the more pessimistic of the two noise scales to train the model, keeping all other hyperparameters constant. As a point of comparison, if (hypothetically) one would convert the RTM noise scales to the DPTM in terms of ε , this would result in ε -values of ≈ 6 , 2 . 5 and 19 at the same δ for CIFAR-10, SNLI and ImageNet, respectively. We remark that this conversion does not imply that these models satisfy DP for these ε -values, just that a hypothetical ( ε, δ ) -analogue of λ -DR would achieve these ε -values. The true worst-case ( ε, δ ) -DP guarantees remain the same in both cases. The experiments were performed on a GPU cluster using NVidia RTX A6000 GPUs and required a total of 212 GPU hours. At the carbon efficiency assumed above, the computation of these experiments caused approximately 27 . 5 kg of CO 2 equivalent emissions. All emission computations were computed as described in [42].

7.4 Ethics statement

Our study focuses on the relaxation of the differential privacy (DP) threat model and provides formal bounds on membership hypothesis testing error rates. We acknowledge that our work has positive and potential negative ethical implications, and we strive to address these concerns as thoroughly as possible.

Positive implications of our study include the following:

Improved privacy guarantees and higher utility in machine learning models, which may encourage privacy sceptics to adopt DP.

Acknowledging that the standard DP threat model may not always be applicable in practice and providing a more realistic alternative.

Enhancing our understanding of privacy dynamics in systems by introducing relaxations incrementally, which can demonstrate how privacy guarantees improve as the adversary's power decreases.

Providing formal bounds, which offer a more rigorous understanding of the sub-optimal adversary's membership inference capabilities in the relaxed threat model.

Despite these positive implications, we recognise potential drawbacks and have identified mitigations for each:

Our guarantees may not hold if the assumptions of the relaxed threat model are violated. To mitigate this, the relaxed threat model guarantee should only be reported alongside the full DP guarantee, which holds in the worst case.

If stakeholders or users misunderstand the circumstances under which the relaxed threat model's guarantees break down, they may come away with a false sense of security. To address this, we ensure a comprehensive understanding of the threat model through formal characterisation and clear communication of the exact guarantees provided by the mechanism in the relaxed threat model, and the situations in which they don't hold, e.g. post-processing.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a notion of privacy called Detection Resilience, which quantifies success of a membership inference adversary in a relaxed threat model in which the adversary does not have access to the target example. This is in contrast to the standard strong adversary model in Differential Privacy that has this knowledge. The paper provides a series of theoretical results regarding the relaxed threat model, showing how to characterize the optimal adversary under the relaxation, which for common privacy mechanisms boils down to one-sided hypothesis testing. The trade-off curve of the most powerful test can be used to describe privacy similarly to f-privacy. Next, the paper provides expressions that enable to numerically compute the trade-off curves for Laplace, Gaussian, and Subsampled Gaussian mechanisms. Finally, using synthetic setups the paper visualizes the difference between the trade-off curves in the relaxed model and f-privacy, and shows that the trade-off curves of empirical membership inference attacks are close to those predicted by the theory.\n\nSTRENGTHS:\n**A great step towards useful provable privacy guarantees in DP.** The pessimistic nature of privacy guarantees afforded by DP has been a critical barrier to its deployment. The paper proposes a novel notion of privacy in a relaxation of the standard \"strong adversary\" model. Thus, it should enable to reduce the noise added during private training while keeping reasonable privacy guarantees, i.e. the guarantees within the relaxed model. *This is the first paper that provides such actionable relaxation* that I am aware of. The prior notions of privacy that incorporate background knowledge rely on assumptions about the data distribution, whereas the notion proposed in the paper does not.\n\nWEAKNESSES:\n- W1. **The threat model does not correspond to relevant privacy threats.** The proposed relaxation of the adversarial model in DP, although is a great step towards provable guarantees for realistic attack scenarios, does not model an actually practically relevant threat model. This is because it assumes the adversary does not know the challenge example. To the contrary, in many standard models of practical membership inference attacks the adversary starts with a challenge example they want to attack. For example, this is the case in the \"offline attack\" by Carlini et al. [12] that is cited as the motivation.\n\n  Instead of a realistic setting, I believe the proposed approach models something that is amenable to the analysis using the decision-theoretic tools in a natural extension of the hypothesis testing interpretation of DP (this is not to say this is something bad on its own).\n\n- W2. **No results with DP-SGD.** The paper seems to make most of the steps needed to evaluate privacy-guarantees under composition, e.g., DP-SGD yet does not quite go there. Without these experiments, it is impossible to gauge the utility gains of relaxing the threat model in most machine learning applications.\n\nNitpicks:\n- Clarity: \n  - The contributions paragraph could be interpreted as that the paper studies any relaxed threat models.\n  - The text does not provide sufficient context for Lemmas 1-3. It would be clearer to exactly define UMP in the context of any given Lemma.\n- DR is a rather confusing name\n- Line 92, 499, 508: Should be double bars to denote the norm?\n- Line 251: Unclear writing\n- Lines 103: The signature of the $\\mathcal{M}$ function in the definition of the hypotheses is inconsistent with the notation right above.\n\nQUESTIONS:\n* Line 280 \"[In adaptive composition] a database and mechanism parameters are fixed ahead of time, and all intermediate results are released to $A_{SO}$, or –equivalently– the components the mechanism’s outcomes are independent of one another.\" Are these two settings well-known to be equivalent to adaptive composition? Is there a citation?\n* If I understand the former setting (\"all intermediate results are released to $A_{SO}$\"), the composition results are applicable to standard analyses of DP-SGD. Is this right, or are there fundamental barriers to computing the DR guarantees of DP-SGD using the proposed techniques?\n* How exactly is the membership inference attack in the experimental section set up? Does it use shadow models? How many? What are the error bars over in Figure 3?\n\nLIMITATIONS:\n- The paper's title reads as an overclaim. The content covers optimal guarantees to **one** particular relaxed threat model, thus \"Optimal privacy guarantees against sub-optimal adversaries\" is a stretch.\n- The paper does not describe what are the particular privacy harms that are protected in the strong adversary model of DP that are not protected by the relaxed notion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the context of membership inference attacks, the authors consider a relaxation of the differential privacy threat model (RTM) wherein the adversary lacks access to the targeted data record. By adopting the binary hypothesis testing formalism commonly employed in the DP literature, the authors prove that the RTM adversary is weaker than the standard DP adversary. Additionally, they prove that it is possible to design uniformly most powerful tests under the monotone likelihood ratio property. Since the Laplace, Gaussian, and Poisson-subsampled Gaussian mechanisms satisfy this property, the authors are able to find the optimal error rates for these three mechanisms under RTM. Finally, the authors include numerical illustrations using state-of-the-art membership inference attacks.\n\nSTRENGTHS:\nIn order to incorporate privacy into real-life applications, it is necessary to calibrate privacy mechanisms against practical threats that relax the differential privacy assumptions. The present paper explores this research direction.\n\nWEAKNESSES:\nThe problem addressed in this paper is of significant importance; however, the paper falls short in delivering substantial results as many of them appear to be straightforward and intuitive. While some proofs may be laborious, a significant portion of the results can be attributed to classical results in statistics. The introduction of the concept of \"lambda-Detection Resilience\" adds unnecessary complexity, and the results derived from this notion (Lemmas 6 and 7) lack depth. Furthermore, the provided illustrations are elementary and do not contribute significantly to the overall understanding. Collectively, both the theoretical and empirical contributions lack the desired impact.\n\nQUESTIONS:\nWhile studying additive mechanisms serves as a valuable initial assessment, the emphasis on these mechanisms alone appears somewhat restricted in scope. To enhance the potential impact of the paper, it would be beneficial for the authors to analyze more complex privacy mechanisms. While conducting such an analysis analytically may present considerable challenges, it may be feasible to develop a principled approach that combines theoretical and empirical methods to address this objective.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper studies differentially private machine learning (DPML) against sub-optimal membership inferences attacks (MIAs), i.e., the attack cannot access the exact training data. Similar to DPML against the worst-case MIAs, the paper formally characterizes membership inference capabilities in terms of hypothesis testing errors and validates the hypothesis in synthetic datasets.

STRENGTHS:
+ The paper is well motivated and well-written 
+ Well-formed theoretical results

WEAKNESSES:
- Only evaluations on synthetic datasets

QUESTIONS:
I do not have any serious comments on the motivation and theoretical results. My only concern is that the proposed DP mechanism is only evaluated on synthetic datasets and does not test in practical settings. Specifically, what are the results on the real-world dataset and applications that are commonly used in the existing DP literature? Whether it is possible to compare the proposed DP with other DP mechanisms against the worst-case adversary? 

Could the proposed DP be generalizable to defend against our privacy attacks such as model inversion/data reconstruction attacks? Any comments or thoughts? Regarding this, I would suggest the authors to clearly state "membership inference adversaries" in the title instead of "adversaries" in general.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a relaxation of the DP threat model in which the adversary's goal is to perform membership inference without access to the target example. The goal of this threat model is to reflect the offline setting explored in Carlini et al. 2022, Watson et al. 2021, and Ye et al 2022. It is possible to characterize the performance of the optimal adversary in this threat model, similar to the Neyman Pearson optimality in the full DP threat model. The paper computes these optimal performances for a variety of practical additive noise mechanisms for DP.\n\nSTRENGTHS:\nThe paper approaches a well motivated goal - relaxing the DP threat model to understand the power of weaker, more realistic, adversaries.\n\nIn their threat model, they are able to derive the optimal attack for a number of mechanisms.\n\nTheir techniques allow a model trainer to \"fall back\" on the provable DPTM guarantees, even if an adversary is more powerful than the RTM guarantee would permit.\n\nThe paper is well written, and is careful about a lot of points, e.g. DR is not robust to postprocessing and does not compose. It is also very well cited.\n\nWEAKNESSES:\nI find the threat model difficult to motivate. At its core, the threat model is defined in a way where the adversary does not learn any private information from succeeding in the game. All it learns is that some example has been added, but no information about this example aside from membership.\n\nThe paper suggests a connection to offline MI attacks, but this doesn't seem right. In offline MI, the adversary still knows the example, they just are unable to fully compute the distribution of M(D'). The paper shows that in their threat model, the adversary can only use the magnitude of the noise, not the direction, as they don't know which direction the modification could have come from. However, in offline MI, the adversary *does* know the direction, because they know the example.\n\nThe experimental results are performed on very small datasets or small dimension synthetic data.\n\nQUESTIONS:\nThe threat model seems more directly applicable to understanding the performance of reconstruction attacks rather than MI attacks, since reconstruction attacks also match your threat model's assumption that the target example is unknown to the adversary. Do you think your results could be applied to those attacks?\n\nLIMITATIONS:\nThese are all well noted in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}