Title: PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining
Venue: NeurIPS 2024
Accepted: yes
Keywords: privacy

auditing

machine learning

differential privacy

membership inference attack
Abstract: We present PANORAMIA, a privacy leakage measurement framework for machine learning models that relies on membership inference attacks using generated data as non-members. By relying on generated non-member data, PANORAMIA eliminates the common dependency of privacy measurement tools on in-distribution non-member data. As a result, PANORAMIA does not modify the model, training data, or training process, and only requires access to a subset of the training data. We evaluate PANORAMIA on ML models for image and tabular data classification, as well as on large-scale language models.

PANORAMIA : Privacy Auditing of Machine Learning Models without Retraining

Abstract

We present PANORAMIA , a privacy leakage measurement framework for machine learning models that relies on membership inference attacks using generated data as non-members. By relying on generated non-member data, PANORAMIA eliminates the common dependency of privacy measurement tools on in-distribution nonmember data. As a result, PANORAMIA does not modify the model, training data, or training process, and only requires access to a subset of the training data. We evaluate PANORAMIA on ML models for image and tabular data classification, as well as on large-scale language models.

1 Introduction

Training Machine Learning (ML) models with Differential Privacy (DP) Dwork et al. (2006), such as with DP-SGD Abadi et al. (2016), upper-bounds the worst-case privacy loss incurred by the training data. In contrast, privacy auditing aims to empirically lower-bound the privacy loss of a target ML model or algorithm. In practice, privacy audits usually rely on the link between DP and the performance of membership inference attacks (MIA) Wasserman & Zhou (2010); Kairouz et al. (2015); Dong et al. (2019). At a high level, DP implies an upper-bound on the performance of MIAs, thus creating a high-performance MIA implies a lower-bound on the privacy loss. Auditing schemes have proven valuable in many settings, such as to audit DP implementations Nasr et al. (2023), or to study the tightness of DP algorithms Nasr et al. (2021); Lu et al. (2023); Steinke et al. (2023). Typical privacy audits rely on retraining the model several times, each time guessing the membership of one sample Jagielski et al. (2020); Carlini et al. (2022a); Zanella-Béguelin et al. (2022), which is computationally prohibitive, requires access to the target model (entire) training data as well as control over the training pipeline.

To circumvent these concerns, Steinke et al. (2023) proposed an auditing recipe (called O(1)) requiring only one training run (which could be the same as the actual training) by randomly including/excluding several samples (called auditing examples) into the training dataset of the target model. Later, the membership of the auditing examples are guessed for privacy audit. However, O(1) faces a few challenges in certain setups. First, canaries, which are datapoints specially crafted to be easy to detect when added to the training set Nasr et al. (2023); Lu et al. (2023); Steinke et al. (2023), cannot be employed as auditing examples when measuring the privacy leakage for data that a contributor actually puts into the model, and not a worst case data point .

This matches a setting in which individual data contributors ( e.g. , a hospital in a cross-site Federated Learning (FL) setting or a user of a service that trains ML models on users' data) measure the leakage of their own ( i.e. , known) partial training data in the final trained model. Second, O(1) also relies on the withdrawal of real data from the model to construct non-member in-distribution data. This is problematic in situations in which ML model owners need to conduct post-hoc audits, in which case it is too late for removal Negoescu et al. (2023). Moreover, in-distribution audits require much more data, thus withholding many data points (typically more than the test set size) and reducing model utility. This brings us to the question: Given an instance of a machine learning model as a target, can we perform post-hoc estimation of the privacy loss with regards to a known member subset of the target model training dataset?

Our contributions. We propose PANORAMIA , a new scheme for Privacy Auditing with NO Retraining by using Artificial data for Membership Inference Attacks . More precisely, we consider an auditor with access to a subset of the training data and introduce a new alternative for accessing non-members: using synthetic datapoints from a generative model trained on the member data, unlocking the limit on non-member data. PANORAMIA uses this generated data, together with known members, to train and evaluate a MIA attack on the target model to audit (§3). We also adapt the theory of privacy audits, and show how PANORAMIA can estimate the privacy loss (though not a lower-bound) of the target model with regards to the known member subset (§4). An important benefit of PANORAMIA is to perform privacy loss measurements with (1) no retraining the target ML model ( i.e. , we audit the end-model, not the training algorithm), (2) no alteration of the model, dataset, or training procedure, and (3) only partial knowledge of the training set. We evaluate PANORAMIA on CIFAR10 models and observe that overfitted models, larger models, and models with larger DP parameters have higher measured privacy leakage. We also demonstrate the applicability of our approach on the GPT-2 based model ( i.e. , WikiText dataset) and CelebA models.

2 Background and Related Work

DP is the established privacy definition in the context of ML models, as well as for data analysis in general. We focus on the pure DP definition to quantify privacy loss with well-understood semantics. In a nutshell, DP is a property of a randomized mechanism (or computation) from datasets to an output space O , noted M : D → O . It is defined over neighboring datasets D,D ′ , differing by one element x ∈ X (we use the add/remove neighboring definition), which is D ′ = D ∪ { x } . Formally:

Definition 1 (Differential Privacy Dwork et al. (2006)) . A mechanism M : D → O is ϵ -DP if for any two neighbouring datasets D , D ′ ∈ D , and for any measurable output subset O ⊆ O it holds that:

Since the neighbouring definition is symmetric, so is the DP definition, and we also have that P [ M ( D ′ ) ∈ O ] ≤ e ε P [ M ( D ) ∈ O ] . Intuitively, ϵ upper-bounds the worst-case contribution of any individual example to the distribution over outputs of the computation ( i.e. , the ML model learned). More formally, ϵ is an upper-bound on the privacy loss incurred by observing an output o , defined as ∣ ∣ ln ( P [ M ( D )= o ] P [ M ( D ′ )= o ] ) ∣ ∣ , which quantifies how much an adversary can learn to distinguish D and D ′ based on observing output o from M . A smaller ϵ hence means higher privacy.

DP, MIA and privacy audits. To audit a DP training algorithm M that outputs a model f , one can perform a MIA on datapoint x , trying to distinguish between a neighboring training sets D and D ′ = D ∪{ x } . The MIA can be formalized as a hypothesis test to distinguish between H 0 = D and H 1 = D ′ using the output of the computation f . Wasserman & Zhou (2010); Kairouz et al. (2015); Dong et al. (2019) show that any such test at significance level α (False Positive Rate or FPR) has power (True Positive Rate or TPR) bounded by e ϵ α . In practice, one repeats the process of training model f with and without x in the training set, and uses a MIA to guess whether x was included. If the MIA has TPR > e ϵ FPR, the training procedure that outputs f is not ϵ -DP. This is the building block of most privacy audits Jagielski et al. (2020); Nasr et al. (2021); Zanella-Béguelin et al. (2022); Lu et al. (2023); Nasr et al. (2023). Appendix F provides further context, and discusses other related work that is not directly relevant to our contribution.

Averaging over data instead of models with O(1). The above result bounds the success rate of MIAs when performed over several retrained models , on two alternative datasets D and D ′ . Steinke et al. (2023) show that it is possible to average over data when several data points independently

tr

tr

D out

D in

D out

D

f

Generative

Model

G

D G

b

PANORAMIA

b(precision, recall)

PANORAMIA(precision, recall)

Trained

ML

Model

f

f

(

D in

),

f

(

D out

)

split

te

te

D in

differ between D and D ′ . Let x 1 ,m be the m data points independently included in the training set, and s 1 ,m ∈ { 0 , 1 } m be the vector encoding inclusion. T 0 ,m ∈ R m represents any vector of guesses, with positive values for inclusion in the training set (member), negative values for non-member, and zero for abstaining. Then, if the training procedure is ϵ -DP, Proposition 5.1 in Steinke et al. (2023) bounds the performance of guesses from T with:

In other words, an audit (MIA) T that can guess membership better than a Bernoulli random variable with probability e ϵ 1+ e ϵ refutes an ϵ -DP claim. In this work we build on this result, extending the algorithm (§3) and theoretical analysis (§4) to enable the use of generated data for non-members, which breaks the independence between data points.

3 PANORAMIA

Figure 1 summarizes the end-to-end PANORAMIA privacy measurement. The measurement starts with a target model f , and a subset of its training data D f from distribution D . For instance, D and D f could be the distribution and dataset of one participant in an FL training procedure that outputs a final model f . The privacy measurement then proceeds in two phases.

Phase 1: In the first phase, PANORAMIA uses a subset of the known training data D G ⊂ D f to train a generative model G . The goal of the generative model G is to match the training data distribution D as closely as possible, which is formalized in Definition 3 (§4). Using the generative model G , we can synthesize non-member data, which corresponds to data that was not used in the training of target model f . Hence, we now have access to an independent dataset of member data D in = D f \ D G , and a synthesized dataset of non-member data D out ∼ G , of size m = | D in | .

Phase 2: In the second phase, we leverage D in and D out to audit the privacy leakage of f using a MIA. To this end, we split D in , D out into training and testing sets, respectively called D tr in , D tr out and D te in , D te out . We use the training set to train a MIA (called PANORAMIA in Figure 1), a binary classifier that predicts whether a given datapoint is a member of D f , the training set of the target model f . This MIA classifier makes its prediction based on both a training example x , as well as information from applying the target model f to the input, such as the loss of the target model when applied to this example loss ( f ( x )) (see §5, Appendix C for details). We use the test set to measure the MIA performance, using the precision at different recall values.

Previous results linking the performance of a MIA on several data-points to ϵ -DP bounds rely on independence between members and non-members. This intuitively means that there is no information about membership in x itself. When the auditor controls the training process this independence is enforced by construction, by adding data points to the training set based on an independent coin flip. In PANORAMIA , we do not have independence between membership and x , as all non-members

Algorithm 1 PANORAMIA

Input: Target ML model f , audit set size m , confidence 1 -β

Phase 1: 1: Split D f in D G , D tr in , D te in , with | D te in | = m 2: Train generator G on D G 3: Generate D tr out , D te out of size | D tr in | , | D te in | Phase 2: Train the baseline and MIA: 1: Label D tr in as members, and D tr out as non-members 2: Train b to predict labels using x ∈ D tr in ∪ D tr out 3: Train MIA to predict labels using x ∈ D tr in ∪ D tr out and f ( x ) Measure privacy leakage (see §4): 1: Sample s ∼ Bernoulli ( 1 2 ) m ▷ Def.2 2: Create audit set X = s · D te in +(1 -s ) D te out 3: Score each audit point for membership, creating t b ≜ b ( X ) ∈ R m + and t a ≜ MIA ( X ) ∈ R m + 4: Set v b ub ( c, t ) ≜ sup { v : β b ( m,c,v, t ) ≤ β 2 } ▷ Prop.1 5: c lb = max t,c 1 { t b ≥ t } · s ≤ v b ub ( c, 1 { t b ≥ t } ) 6: Set v a ub ( c, ϵ, t ) ≜ sup { v : β a ( m,c,ϵ, v, t ) ≤ β 2 } ▷ Prop.2 7: { c + ϵ } lb = max t,c,ϵ 1 { t a ≥ t } · s ≤ v a ub ( c, ϵ, 1 { t a ≥ t } ) Return ˜ ϵ ≜ { c + ϵ } lb -c lb

come from the generator G ̸ = D . As a result, there are two ways to guess membership and have high MIA precision: either by using f to detect membership ( i.e. , symptomatic of privacy leakage) or by detecting generated data ( i.e. , not a symptom of privacy leakage). To measure the privacy leakage, we compare the results of the MIA to that of a baseline classifier b that guesses membership based exclusively on x , without access to f . The stronger this baseline, the better the removal of the effect of synthesized data detection. Algorithm 1 summarizes the entire procedure. In the next section, we demonstrate how to relate the difference between the baseline b and the MIA performance to the privacy loss ϵ .

4 Quantifying Privacy Leakage with PANORAMIA

We formalize a privacy game as follows: construct a sequence of auditing samples, in which s i is sampled independently to choose either the real ( x in ∈ X m training points from D if s i = 1 ) or generated ( x gen ∈ X m generated points from G if s 1 = 0 ) data point at each index i .

Definition 2 (Privacy game) .

The key difference between Definition 2 and the privacy game of Steinke et al. (2023) lies in how we create the audit set. Instead of sampling non-members independently and removing them from the training set (which requires modifyig the training set and the model), Definition 2 starts from a set of known members (after the fact), and pairs each point with a non-member (generated i.i.d. from the generator distribution). For each pair, we then flip a coin to decide which point in the pair will be shown to the auditor for testing, thereby creating the test task of our privacy measurement.

The level of achievable success in this game quantifies the privacy leakage of the target model f . More precisely, we follow an analysis inspired by that of Steinke et al. (2023), but require several key technical changes to support auditing with no retraining using generated non-member data.

4.1 Formalizing the Privacy Measurements as a Hypothesis Test

We first need a notion of quality for our generator:

Definition 3 ( c -closeness) . For all c > 0 , we say that a generative model G is c -close for data distribution D if:

The smaller c , the better the generator, as G assigns a probability to real data that cannot be much smaller than that of the real data distribution. We make three important remarks.

Remark 1: c -closeness is a strong requirement but it is achievable, at least for a large c . For instance, a uniform random generator is c -close to any distribution D , with c < ∞ . Of course c would be very large, and the challenge in PANORAMIA is to create a generator that empirically has a small enough c . Remark 2: Weshow in Appendix E that we can relax this definition to hold only with high probability (following the ( ϵ, δ ) -DP relaxation). Empirical results under this relaxation are very similar. We also show how to measure ( ϵ, δ ) -DP, instead of ϵ -DP on which we focus here.

Remark 3: Our definition of closeness is very similar to that of DP. This is not a coincidence as we will use this definition to be able to reject claims of both c -closeness for the generator and ϵ -DP for the algorithm that gave the target model. We further note that, contrary to the DP definition, c -closeness is one-sided as we only bound P G from below. Intuitively, this means that the generator has to produce high-quality samples ( i.e. , samples likely under D ) with high enough probability but it does not require that all samples are good though. This one-sided measure of closeness is enabled by our focus on detecting members ( i.e. , true positives) as opposed to members and non-members, and puts less stringent requirements on the generator.

Using Definition 3, we can formulate the hypothesis test on the privacy game of Definition 2 that underpins our approach:

H : generator G is c -close, and target model f is ϵ -DP .

Note that, in a small abuse of language, we will often refer to f as ϵ -DP to say that f is the output of an ϵ -DP training mechanism. That is, B ( s, x ) = { b ( x 1 ) , b ( x 2 ) , . . . , b ( x m ) } .

We define two key mechanisms: B ( s, x ) : { 0 , 1 } m ×X m → R m + outputs a (potentially randomized) non-negative score for the membership of each datapoint x i , based on x i only; A ( s, x, f ) : { 0 , 1 } m × X m ×F → R m + outputs a (potentially randomized) non-negative score for the membership of each datapoint, with the guess for index i depending on x ≤ i and target model f . We can construct a statistical test for each part of the hypothesis separately.

Proposition 1. Let G be c -close, S and X be the random variables for s and x from Definition 2, and T b ≜ B ( S, X ) be the vector of guesses from the baseline. Then, for all v ∈ R and all t in the support of T :

Now that we have a test to reject a claim that the generator G is c -close for the data distribution D , we turn our attention to the second part of H which claims that the target model f is the result of an ϵ -DP mechanism.

Proposition 2. Let G be c -close, S and X be the random variables for s and x from Definition 2, f be ϵ -DP, and T a ≜ A ( S, X, f ) be the guesses from the membership audit. Then, for all v ∈ R and all t in the support of T :

We can now provide a test for hypothesis H , by applying a union bound over Propositions 1 and 2: Corollary 1. Let H be true, T b ≜ B ( S, X ) , and T a ≜ A ( S, X, f ) . Then:

To make things more concrete, let us instantiate Corollary 1 as we do in PANORAMIA . Our baseline ( B above) and MIA ( A above) classifiers return a membership guesses T a,b ∈ { 0 , 1 } m , with 1 corresponding to membership. Let us call r a,b ≜ ∑ i t a,b i the total number of predictions, and tp a,b ≜ ∑ i t a,b i · s i the number of correct membership guesses (true positives). We also call the precision prec a,b ≜ tp a,b r a,b . Using the following tail bound on the sum of Bernoulli random variables for simplicity and clarity (we use a tighter bound in practice, but this one is easier to read),

we can reject H at confidence level β by setting β a = β b = β 2 and if either prec b ≥ e c 1+ e c + √ log(2 /β ) 2 r b or prec a ≥ e c + ϵ 1+ e c + ϵ + √ log(2 /β ) 2 r a .

4.2 Quantifying Privacy Leakage and Interpretation

In an privacy measurement, we would like to quantify ϵ , not just reject a given ϵ claim. We use the hypothesis test from Corollary 1 to compute a confidence interval on c and ϵ . To do this, we first define an ordering between ( c, ϵ ) pairs, such that if ( c 1 , ϵ 1 ) ≤ ( c 2 , ϵ 2 ) , the event ( i.e. , set of observations for T a,b , S ) for which we can reject H ( c 2 , ϵ 2 ) is included in the event for which we can reject H ( c 1 , ϵ 1 ) . That is, if we reject H for values ( c 2 , ϵ 2 ) based on audit observations, we also reject values ( c 1 , ϵ 1 ) based on the same observations.

We define the following order to fit this assumption, based on the hypothesis test from Corollary 1:

This lexicographic order over hypotheses ensures that, when using the joint hypothesis test from Corollary 1 to construct confidence intervals, we always reject hypotheses with a low value of c incompatible with the baseline performance. Formally, this yields the following confidence intervals:

Corollary 2. For all β ∈ ]0 , 1] , m , and observed t a , t b , call v a ub ( c, ϵ ) ≜ sup { v : β a ( m,c,ϵ, v, t a ) ≤ β } and v b ( c ) ≜ sup { v : β b ( m,c,v, t b ) ≤ β } . Then:

Proof. Apply Lemma 4.7 from Steinke et al. (2023) with the ordering from Equation 2 and the test from Corollary 1.

The lower confidence interval for ( c, ϵ ) at confidence 1 -β is the largest ( c, ϵ ) pair that cannot be rejected using Corollary 1 with false rejection probability at most β . Hence, for a given confidence level 1 -β , PANORAMIA computes ( c lb , ˜ ϵ ) , the largest value for ( c, ϵ ) that it cannot reject. Note that Corollaries 1 and 2 rely on a union bound between two tests, one for c and one for c + ϵ . We can thus consider each test separately (at level β/ 2 ). In practice, we use the whole precision/recall curve to achieve tighter bounds: each level of recall ( i.e. , threshold on t a,b to predict membership) corresponds to a bound on the precision, which we can compare to the empirical value. Separately for each test, we pick the level of recall yielding the highest lower-bound (Lines 4-7 in the last section of Algorithm 1). For all bounds to hold together, we apply a union bound over all tests at each level of recall. Specially, if we have m test datapoints, we have at most m different values of recall, and the union bound corresponds to β ← β/m (notice in Equation (1) that the bounds depend on √ log(1 /β ) , so this is not too costly). We next discuss the semantics of returned measurement values ( c lb , ˜ ϵ ) .

Measurement semantics. Corollary 2 gives us a lower-bound for ( c, ϵ ) , based on the ordering from Equation (2). To understand the value ˜ ϵ returned by PANORAMIA , we need to understand what the hypothesis test rejects. Rejecting H means either rejecting the claim about c , or the claim about c + ϵ (which is the reason for the ordering in Equation 2). With Corollary 2, we hence get both a lower-bound c lb on c , and { c + ϵ } lb on c + ϵ . Unfortunately, ˜ ϵ ≜ max { 0 , { c + ϵ } lb -c lb } , which is the value PANORAMIA returns, does not imply a lower-bound on ϵ . Instead, we can claim that

' PANORAMIA could not reject a claim of c -closeness for G , and if this claim is tight, then f cannot be the output of a mechanism that is ˜ ϵ -DP'.

While this is not as strong a claim as typical lower-bounds on ϵ -DP from prior privacy auditing works, we believe that this measure is useful in practice. Indeed, the ˜ ϵ measured by PANORAMIA is a quantitative privacy measurement, that is accurate (close to a lower-bound on ϵ -DP) when the baseline performs well (and hence c lb is tight). Thus, when the baseline is good, ˜ ϵ can be interpreted as (close to) a lower bound on (pure) DP. PANORAMIA opens a new capability, measuring privacy leakage of a trained model f without access or control of the training pipeline, with a meaningful and practically useful measurement, as we empirically show next.

5 Evaluation

We instantiate PANORAMIA on target models for four tasks from three data modalities 1 . For image classification , we consider the CIFAR10 Krizhevsky (2009), and CelebA Liu et al. (2015) datasets, with varied target models: a four-layers CNN O'Shea & Nash (2015), a ResNet101 He et al. (2015), a Wide ResNet-28-10 Zagoruyko & Komodakis (2017b), a Vision Transformer (ViT)-small Dosovitskiy et al. (2021), and a DP ResNet18 He et al. (2015) trained with DP-SGD (Abadi et al., 2016) using Opacus (Yousefpour et al., 2021) at different values of ϵ . We use StyleGAN2 Karras et al. (2020) for G . For language models , we fine-tune small GPT-2 Radford et al. (2019) on the WikiText train dataset Merity et al. (2016) (we take a subset of the documents in WikiText-103). G is again based on small GPT-2, and then fine-tuned on D G . We generate samples using topk sampling Holtzman et al. (2019) and a held-out prompt dataset D prompt G ⊂ D G . Finally, we conduct experiments on classification on tabular data . However for this data modality, PANORAMIA did not detect any meaningful leakage. More precisely, we have observed significant variance in the audit values returned by PANORAMIA which make the results inconclusive. Nonetheless, we present the results obtained for this task and discuss this issue further in Appendix D.6. Table 1 summarizes the tasks, models, and performance. More details on the settings, the models used, and implementation details are provided in Appendix C.

5.1 Baseline Design and Evaluation

Our MIA is a loss-based attack, which uses an ML model taking as input both a datapoint x and the value of the loss of target model f on point x . Appendix C details the architectures used for the attack model for each data modality. Recall from §4.2 the importance of having a tight c lb for our measure ˜ ϵ to be close to a lower-bound on ϵ -DP. To increase the performance of our baseline b , we mimic the role of the target model f 's loss in the MIA using a helper model h trained on synthetic data, which adds a loss-based feature to b .

This new feature can be viewed as side information about the data distribution. Table 2 shows the c lb value for different designs for h . The best performance is consistently when h is trained on synthetic data before being used as a feature to train the

Baseline model

c

lb

CIFAR-10 Baseline

D

tr

h

=

gen

2

.

44

±

0

.

19

CIFAR-10 Baseline

D

tr

h

=

real

2

.

21

±

0.17

CIFAR-10 Baselineno helper

1

.

25

±

0

.

24

CelebA Baseline

D

tr

h

=

gen

2

.

05

±

0

.

21

CelebA Baseline

D

tr

h

=

real

1

.

97

±

0

.

22

CelebA Baselineno helper

0

.

947

±

0

.

25

D

tr

h

=

gen

3

.

31

±

0

.

15

D

tr

h

=

real

3

.

26

±

0

.

14

WikiText Baselineno helper

3

.

11

±

0

.

15

500

5000

10000

15000

20000

25000

30000

Train Dataset Size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Measurement of

clb

Baseline-CIFAR10

b . Indeed, such a design reaches a c lb up to 1 . 19 larger that without any helper (CIFAR10) and 0 . 23 higher than when training on real non-member data, without requiring access to real non-member data , a key requirement in PANORAMIA . We adopt this design in all the following experiments. In Figure 2 we show that the baseline has enough training data (vertical dashed line) to reach its best performance. Appendix D.1, shows similar results on GPT-2 as well as different architectures that we tried for the helper model. All these pieces of evidence confirm the strength of our baseline.

5.2 Main Privacy Measurement Results

We run PANORAMIA on models with different degrees of over-fitting by varying the number of epochs, (see the final accuracy on Table 1) for each data modality. More over-fitted models are known to leak more information about their training data due to memorization (Yeom et al., 2018; Carlini et al., 2019, 2022a). To show the privacy loss measurement power of PANORAMIA , we compare it with two strong approaches to lower-bounding privacy loss. First, we compare PANORAMIA with the O(1)

(b) CNN, CelebA

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0.5

0.6

0.7

0.8

0.9

1.0

Precision

Baseline

PANORAMIA GPT-2_E150

PANORAMIA GPT-2_E75

PANORAMIA GPT-2_E37

Theoretical maximum precision

(c) GPT-2, WikiText

1.0

1.0

0.9

0.9

Precision

0.8

Precision

0.8

0.7

Baseline-CIFAR10

0.7

0.6

PANORAMIA:ResNet101_E100

PANORAMIA:ResNet101_E50

0.6

PANORAMIA:MultiLabel CNN_E100

Baseline-CelebA

0.5

Theoreticalmaximumprecision

PANORAMIA:ResNet101E20

0.5

Theoreticalmaximumprecision

PANORAMIA:MultiLabel CNN_E50

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Measurement for

clb

or {

c

+ }

lb

clb

{

c

+ }

lb

ResNet101-E100

{

c

+ }

lb

ResNet101-E50

clb

ResNet101-E20

Empirical maximum value

(a) ResNet101, CIFAR10

(b) CNN, CelebA

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0

1

2

3

4

5

6

Measurement for

c

lb or {

c

+ }lb

c

lb

{

c

+ }lb GPT-2_E150

{

c

+ }lb GPT-2_E75

{

c

+ }lb GPT-2_E37

(c) GPT-2, WikiText

4.0

3.5

+

3.0

CIb

2.5

2.0

1.5

CIb

1.0

（c+ε）ibMultiLabelCNN_E100

(C+ε）ibMultiLabel CNN_E50

0.5

Empirical maximum value

0.0

8.0

0.2

0.4

0.6

0.8

1.0

Recall

audit (Steinke et al., 2023) (in input space, without canaries in a black box setting), which uses a loss threshold MIA to detect both members and non-members. Second, we use a hybrid between O(1) and PANORAMIA , in which we use real non-member data instead of generated data with our ML-based MIA (called RM;RN for Real Members; Real Non-members). In both cases c lb = 0 , but it requires control over training data inclusion and a large set of non-member data. The privacy loss measured by these techniques gives a target that we hope PANORAMIA to detect.

Figure 3 shows the precision of b and PANORAMIA at different levels of recall, and Figure 4 the corresponding value of { c + ϵ } lb (or c lb for b ). Dashed lines show the maximum value of { c + ϵ } lb (or c lb) achieved (Fig. 4) (returned by PANORAMIA ), and the precision implying these values at different recalls (Fig. 3). Table 3 summarizes those { c + ϵ } lb (or c lb) values, as well as the ϵ measured by existing approaches. We make two key observations.

First, the best prior method (whether RM;RN or O(1)) measures a larger privacy loss ( ˜ ϵ ≤ ϵ ). Overall, these results empirically confirm the strength of b , as we do not seem to spuriously assign differences between G and D to our privacy loss proxy ˜ ϵ . Wealso note that O(1) tends to perform better, due to its ability to rely on non-member detection, which improves the power of the statistical test at equal data sizes. Such tests are not available in PANORAMIA given our one-sided closeness definition for G (see §4), and we keep that same one-sided design for RM;RN for the sake of comparison.

Second, the values of ˜ ϵ measured by PANORAMIA are close to those of the methods against which we compared. In particular, despite a more restrictive adversary model ( i.e. , no non-member data, no control over the training process, and no shadow model training), PANORAMIA is able to detect meaningful amounts of privacy loss, comparable to that of stateof-the-art methods.

For instance, on a non-overfitted CIFAR-

10 ResNet101 model (E20), PANORAMIA detects a privacy loss of 0 . 33 , while using real non-member (RM;RN) data yields 0 . 42 , and controlling the training process O (1) gets 0 . 45 . The relative gap gets even closer on models that reveal more about their training data. Indeed, for the most over-fitted model (E100), ˜ ϵ = 1 . 16 is very close to RM;RN ( ϵ = 1 . 22 ) and O(1) ( ϵ = 1 . 41 ). This also confirms that the leakage detected by PANORAMIA on increasingly overfitted models does augment, which is confirmed by prior state-of-the-art methods. For instance, NLP models' ˜ ϵ goes from 0 . 22 to 2 . 40 ( 2 . 82 to 5 . 73 for O(1)). Appendix D.6 details results on tabular data, in which PANORAMIA is not able to detect significant privacy loss.

Privacy Measurement of ϵ -DP Models:

Another avenue to varying privacy leakage is by training Differentially-Private (DP) models with different values of ϵ . We evaluate PANORAMIA on DP ResNet-18 models on CIFAR10, with ϵ values shown on Table 3 and Figure 5. The hyper-parameters were tuned independently for the highest train accuracy (see Appendix D.9 for more results and discussion). As the results show, neither PANORAMIA nor O(1) detect privacy loss on the most private models ( ϵ = 1 , 2 ). At higher values of ϵ = 10 , 15 ( i.e. , less private models) and ϵ = ∞ ( i.e. , non-private model) PANORAMIA does detect an increasing level of privacy leakage with ˜ ϵ ϵ =10 < ˜ ϵ ϵ =15 < ˜ ϵ ϵ = ∞ . In this regime, the O(1) approach detects a larger, though comparable, amount of privacy loss.

for various ϵ values. We also evaluate PANORAMIA on DP (fine-tuned) large language models (see Appendix D.9 and Table 9 for details). We fine-tune GPT2-Small target models with DP-SGD on the WikiText dataset, with various values of ϵ . Neither PANORAMIA nor O(1) are able to measure a positive privacy loss.

5.3 Leveraging More Data to Improve Privacy Measurements

PANORAMIA can leverage much more data for its measurement (up to the whole training set size for training and testing the MIA), while in our setting O(1) is limited by the size of the test set (for non-members). As we have seen in Figure 6(b), PANORAMIA can leverage larger test set size to measure higher privacy loss values than O(1), up to 2 . 62 on the ResNet101 trained for 100 epochs, versus 2 . 23 for O(1). It is also important to note that at small amounts of data O(1) measures a larger privacy loss ( ˜ ϵ ≤ ϵ ). These findings provide additional empirical evidence for the robustness of b , indicating that we are not erroneously attributing differences between G and D to our privacy loss proxy ˜ ϵ . However, in the case of text data (on the GPT-2 trained for 150 epochs), Figure 6(d), we hit an upper-bound on the power of the hypothesis test which estimates { c + ϵ } lb using 20 k test samples. The maximum measurement for ˜ ϵ we can achieve is around 2 . 59 with a 20 k test set, as opposed to O(1), which reaches 5 . 73 with 10 k test set size.

500

10000

20000

30000

40000

Test Dataset Size m

1.0

1.5

2.0

2.5

3.0

3.5

4.0

measurement of   or

O(1)-WRN-28-10_E150

O(1)-ML-based-WRN-28-10_E150

Panoramia-WRN-28-10_E150

500

10000

20000

30000

40000

Test Dataset Size m

0.5

1.0

1.5

2.0

2.5

3.0

measurement of   or

O(1)-ML-based-ResNet101_E100

Panoramia-ResNet101_E100

12

4

6

10

15

20

Theoretical Epsilon

0

2

4

6

8

10

Empirical

or

m=1000

m=10000

O(1) m=10000

O(1) (ML based MIA) m=10000

m=20000

O(1) m=20000

Theoretical

1000

5000

10000

15000

20000

Test Dataset Size m

2.0

2.5

3.0

3.5

4.0

4.5

5.0

5.5

measurement of   or

PANORAMIA-GPT2_E150

O(1)-GPT2_E150

Figure 6: Effect of increasing test set size on PANORAMIA 's privacy measurement for our target models. In the case of the image dataset, increasing the number of auditing examples allows us to achieve tighter empirical measurements for privacy leakage despite a restricted adversary. For the case of the language modeling task, when increasing test set size m , we hit an upper bound on the power of the hypothesis test due to test dataset size, and do not see significant improvement in privacy measurement in this case.

Increasing Test Set Size for DP models: With an increase in the test set size m , we demonstrate that we can achieve tighter bounds on the privacy measurement ˜ ϵ . Figure 6(c) shows that as m increases, PANORAMIA (solid lines) can measure higher privacy leakage for various ϵ -DP ResNet18 models. The reason for this is that PANORAMIA can leverage a much higher test set size (up to the whole training set size for training and testing the MIA), in comparison to O(1) in our setting (due to being limited by the number of real non-member data points available).

6 Impact, Limitations and Future Directions

We have introduced a new approach to quantify the privacy leakage from ML models, in settings in which individual data contributors (such as a hospital in a cross-site FL setting or a user of a text auto-complete service) measure the leakage of their own, known partial training data in the final trained model. Our approach does not introduce new attack capabilities (it would likely benefit from future progress on MIA performance though), but can help model providers and data owners assess the privacy leakage incurred by their data, due to inclusion in ML models training datasets. Consequently, we believe that any impact from our proposed work is likely to be positive.

However, PANORAMIA suffers from a major limitation: its privacy measurement is not a lower-bound on the privacy loss (see details in Section 4.2). Providing a proper lower-bound ϵ lb on privacy leakage in this setting is an important avenue for future work. A promising direction is to devise a way to measure or enforce an upper-bound on c , thereby yielding ϵ lb via { c + ϵ } lb -c ub . Despite this shortcoming, we believe that the new measurement setting we introduce in this work is important. Indeed, recent work has shown that MIA evaluation on foundation models suffers from a similar lack of non-member data (since all in distribution data is included in the model) Das et al. (2024); Duan et al. (2024); Meeus et al. (2024). The theory we develop in this paper provides a meaningful step towards addressing privacy measurements in this setting, and provides a more rigorous approach to privacy benchmarks for such models. We also demonstrate that PANORAMIA 's privacy measurements can also be empirically valuable for instance for providing improved measurements with more data (Figure 6).

Acknowledgments

Mathias Lécuyer is grateful for the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) [reference number RGPIN-2022-04469], as well as a Google Research Scholar award. Sébastien Gambs is supported by the Canada Research Chair program, a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) as well as the DEEL Project CRDPJ 537462-18 funded by the NSERC and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ). This research was enabled by computational support provided by the Digital Research Alliance of Canada (alliancecan.ca). Hadrien Lautraite is supported by National Bank of Canada and a bursary from the Canada Research Chair in Privacy-preserving and Ethical Analysis of Big Data..

References

Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security , pp. 308-318, 2016.

Andrew, G., Kairouz, P., Oh, S., Oprea, A., McMahan, H. B., and Suriyakumar, V. One-shot empirical privacy estimation for federated learning, 2023.

Carlini, N., Liu, C., Erlingsson, Ú., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19) , pp. 267-284, 2019.

Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21) , 2021.

Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 2022 IEEE Symposium on Security and Privacy (SP) (SP) , pp. 15191519, Los Alamitos, CA, USA, may 2022a. IEEE Computer Society. doi: 10.1109/SP46214.2022. 00090. URL https://doi.ieeecomputersociety.org/10.1109/SP46214.2022.00090 .

Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP) , pp. 1897-1914. IEEE, 2022b.

Das, D., Zhang, J., and Tramèr, F. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.16201 , 2024.

Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. arXiv preprint arXiv:1905.02383 , 2019.

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxiv.org/abs/ 2010.11929 .

Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841 , 2024.

Dwork, C., McSherry, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference . Springer, 2006.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015.

Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 , 2019.

Jagielski, M., Ullman, J., and Oprea, A. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems , 33:22205-22216, 2020.

Jayaraman, B. and Evans, D. E. Evaluating differentially private machine learning in practice. In USENIX Security Symposium , 2019.

Kairouz, P., Oh, S., and Viswanath, P. The composition theorem for differential privacy. In International conference on machine learning . PMLR, 2015.

Karras, T., Aittala, M., Hellsten, J., Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data, 2020.

Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https://api. semanticscholar.org/CorpusID:18268744 .

Li, X., Tramer, F., Liang, P., and Hashimoto, T. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679 , 2021.

Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Lu, F., Munoz, J., Fuchs, M., LeBlond, T., Zaresky-Williams, E., Raff, E., Ferraro, F., and Testa, B. A general framework for auditing differentially private machine learning, 2023.

Maddock, S., Sablayrolles, A., and Stock, P. Canife: Crafting canaries for empirical privacy measurement in federated learning, 2023.

McKenna, R., Miklau, G., and Sheldon, D. Winning the nist contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2108.04978 , 2021.

Meeus, M., Jain, S., Rei, M., and de Montjoye, Y.-A. Inherent challenges of post-hoc membership inference for large language models. arXiv preprint arXiv:2406.17975 , 2024.

Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843 , 2016.

Nasr, M., Shokri, R., and Houmansadr, A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP) , pp. 739-753. IEEE, 2019.

Nasr, M., Songi, S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP) , pp. 866-882, 2021. doi: 10.1109/SP40001.2021.00069.

Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagielski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning, 2023.

Negoescu, D. M., Gonzalez, H., Orjany, S. E. A., Yang, J., Lut, Y., Tandra, R., Zhang, X., Zheng, X., Douglas, Z., Nolkha, V., et al. Epsilon*: Privacy metric for machine learning models. arXiv preprint arXiv:2307.11280 , 2023.

O'Shea, K. and Nash, R. An introduction to convolutional neural networks, 2015.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Steinke, T., Nasr, M., and Jagielski, M. Privacy auditing with one (1) training run. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Suri, A., Zhang, X., and Evans, D. Do parameters reveal more than loss for membership inference? In High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning .

Wasserman, L. and Zhou, S. A statistical framework for differential privacy. Journal of the American Statistical Association , 2010.

Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF) , pp. 268-282. IEEE, 2018.

Yousefpour, A., Shilov, I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode, G., and Mironov, I. Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv:2109.12298 , 2021.

Zagoruyko, S. and Komodakis, N. Wide residual networks, 2017a.

Zagoruyko, S. and Komodakis, N. Wide residual networks, 2017b. URL https://arxiv.org/ abs/1605.07146 .

Zanella-Béguelin, S., Wutschitz, L., Tople, S., Salem, A., Rühle, V ., Paverd, A., Naseri, M., Köpf, B., and Jones, D. Bayesian estimation of differential privacy, 2022.

Zarifzadeh, S., Liu, P. C.-J. M., and Shokri, R. Low-cost high-power membership inference by boosting relativity. 2023.

A Notations

Table 4 summarizes the main notations used in the paper.

B Proofs

For both Proposition 1 and Proposition 2, we state the proposition again for convenience before proving it.

B.1 Proof of Proposition 1

Proposition 1. Let G be c -close, and T b ≜ B ( S, X ) be the guess from the baseline. Then, for all v ∈ R and all t in the support of T :

Proof. Notice that under our baseline model B ( s, x ) = { b ( x 1 ) , b ( x 2 ) , . . . , b ( x m ) } , and given that the X i are i.i.d., we have that: S <i ⊥ ⊥ T b <i | X <i , since T b i = B ( S, X ) i 's distribution is entirely determined by X i ; and S ≤ i ⊥ ⊥ T b >i | X <i since the X i are sampled independently from the past.

We study the distribution of S given a fixed prediction vector t b , one element i ∈ [ m ] at a time:

The first equality uses the independence remarks at the beginning of the proof, the second relies Bayes' rule, while the third and fourth that S i is sampled i.i.d from a Bernoulli with probability half, and X i i.i.d. conditioned on S i . The last inequality uses Definition 3 for c -closeness.

Using this result and the law of total probability to introduce conditioning on X ≤ i , we get that:

and hence that:

We can now proceed by induction: assume inductively that W m -1 ≜ ∑ m -1 i =1 T b i · S i is stochastically dominated (see Definition 4.8 in Steinke et al. (2023)) by W ′ m -1 ≜ ∑ m -1 i =1 T b i · S ′ i , in which S ′ ∼ Bernoulli ( e c 1+ e c ) m -1 . Setting W 1 = W ′ 1 = 0 makes it true for m = 1 . Then, conditioned on W m -1 and using Eq. 3, T b m · S m = T m · 1 { S m = 1 } is stochastically dominated by T b m · Bernoulli ( e c 1+ e c ) . Applying Lemma 4.9 from Steinke et al. (2023) shows that W m is stochastically dominated by W ′ m , which proves the induction and implies the proposition's statement.

B.2 Proof of Proposition 2

Proposition 2. Let G be c -close, f be ϵ -DP, and T a ≜ A ( S, X, f ) be the guess from the membership audit. Then, for all v ∈ R and all t in the support of T :

Proof. Fix some t a ∈ R m + . We study the distribution of S one element i ∈ [ m ] at a time:

The first equality uses Bayes' rule. The first inequality uses the decomposition:

and the fact that A ( s, x, f ) is ϵ -DP w.r.t. s and hence that:

This inequality might seem surprising at first, as we know from the privacy game (Definition 2) that S is independent of f , that is S ⊥ ⊥ f . However, these two quantities are not conditionally independent when conditioning on X (where X is the test set for the MIA/baseline), (i.e., S ̸⊥ ⊥ f | X ). This is because X i is either a member or non-member based on S i . So when S i = 1 , then f was trained on X i , which makes f and S i dependent if membership can be detected through f . If f is ϵ -DP, we can bound this dependency using DP properties, which is what we do in Equation (4).

The second inequality uses that:

As in Proposition 1, applying the law of total probability to introduce conditioning on X ≤ i yields:

and we can proceed by induction. Assume inductively that W m -1 ≜ ∑ m -1 i =1 T a i · S i is stochastically dominated (see Definition 4.8 in Steinke et al. (2023)) by W ′ m -1 ≜ ∑ m -1 i =1 T a i · S ′ i , in which S ′ ∼ Bernoulli ( e c + ϵ 1+ e c + ϵ ) m -1 . Setting W 1 = W ′ 1 = 0 makes it true for m = 1 . Then, conditioned on W m -1 and using Eq. 5, T a m · S m = T a m · 1 { S m = 1 } is stochastically dominated by T a m · Bernoulli ( e c + ϵ 1+ e c + ϵ ) . Applying Lemma 4.9 from Steinke et al. (2023) shows that W m is stochastically dominated by W ′ m , which proves the induction and implies the proposition's statement.

C Experimental Details

Hereafter, we provide the details about the datasets and models trained in our experiments.

C.1 Image data

Target models. We audit target models with the following architectures: a Multi-Label Convolutional Neural Network (CNN) with four layers O'Shea & Nash (2015), and the ResNet101 He et al. (2015) as well as a Wide ResNet-28-10 Zagoruyko & Komodakis (2017b), and a Vision Transformer (ViT)-small Dosovitskiy et al. (2021). We also include in our analysis, differentially-private models for ResNet18 He et al. (2015) and WideResNet-16-4 Zagoruyko & Komodakis (2017a) models as targets, with ϵ = 1 , 2 , 4 , 6 , 10 , 15 , 20 . The ResNet-based models are trained on CIFAR10 using 50 k images Krizhevsky (2009) of 32 x 32 resolution. For ResNet101 CIFAR10-based classification models, we use a training batch size of 32; for WideResNet-28-10, we use batch size 128. The details for training DP models are given in D.9. The associated test accuracies and epochs are mentioned in Table 1. The Multi-Label CNN is trained on 200 k images of CelebA Liu et al. (2015) of 128 x 128 resolution, training batch-size 32, to predict 40 attributes associated with each image.

Generator. For both image datasets, we use StyleGAN2 Karras et al. (2020) to train the generative model G from scratch on D G , and produce non-member images. For CIFAR10 dataset, we use a 10 , 000 out of 50 , 000 images from the training data of the target model to train the generative model. For the CelebA dataset, we select 35 , 000 out of 200 , 000 images from the training data of the target model to train the generative model. Generated images will in turn serve as non-members for performing the MIAs. Figure 7 shows examples of member and non-member images used in our experiments. In the case of CelebA, we also introduce a vanilla CNN as a classifier or filter to distinguish between fake and real images, and remove any poor-quality images that the classifier detects with high confidence. The data used to train this classifier was the same data used to train StyleGAN2, which ensures that the generated high-resolution images are of high quality.

MIA and Baseline training. For the MIA, we follow a loss-based attack approach: PANORAMIA takes as input raw member and non-member data points for training along with the loss values the target model f attributes to these data points. More precisely, the training set of PANORAMIA is:

In §4.2, we discussed the importance of having a tight c lb so that our measure, ˜ ϵ , becomes close to a lower-bound on ϵ -DP, which requires a strong baseline. To strengthen our baseline, we introduce the helper model h , which helps the baseline model b by supplying additional features ( i.e. , embeddings) that can be viewed as side information about the data distribution. The motivation is that h 's features might differ between samples from D and D ′ , enhancing the performance of the baseline classifier. This embedding model h is similar in design to f (same task and architecture) but is trained on synthetic data that is close in distribution to the real member data distribution.

The helper model has the same classification task and architecture as the target model. To train it, we generate separate sets of training and validation data using our generator. For image data, the synthetic samples do not have a label. We thus train a labeler model, also a classifier with the same task, on the same dataset used to train the generator. We use the labeler model to provide labels for our synthetic samples (training and validation sets above). We train the helper model on the resulting training set, and select hyperparameters on the validation set.

Whether for the baseline or MIA, we use side information models ( h and f , respectively) by concatenating the loss of h ( x ) and f ( x ) to the final feature representation (more details are provided later) before a last layer of the MIA/Baseline makes the membership the prediction. Since we need labels to compute the loss, we label synthetic images with a Wide ResNet-28-2 in the case of CIFAR10, and a Multi-Label CNN of similar architecture as the target model in the case of CelebA labeling. For both instances, we used a subset of the data, that was used to train the respective generative models, to train the 'labeler' classifiers as well. The labeler used to train the helper model for image data has 80.4% test accuracy on the CIFAR10 real data test set. The rationale for this approach is to augment the baseline with a model providing good features (here for generated data) to balance the good features provided to the MIA by f outside of the membership information. In practice, the labeled generated data seems enough to provide such good features, despite the fact that the labeler is not extremely accurate. We studied alternative designs (e.g., a model trained on non-member data, no helper model) in Appendix D.1, Table 5, and the helper model trained on the synthetic data task performs best (while not requiring non-member data, which is a key point).

We use two different modules for each MIA and Baseline training. More precisely, the first module optimizes image classification using a built-in Pytorch ResNet101 classifier. The second module, in the form of a multi-layer perceptron, focuses on classifying member and non-member labels via loss values attributed to these data points by f as input for the loss module of MIA and losses of e to the baseline b respectively. We then stack the scores of both image and loss modules into a logistic regression task (as a form of meta-learning) to get the final outputs for member and non-member data points by MIA and baseline b . When training the baseline and MIA models, we use a validation set to select hyper-parameters and training stopping time that maximize the lower bounds (effectively maximizing the c lb or ˜ ϵ lb for the Baseline and MIA respectively) on the validation set. The MIA and baseline are trained on 4500 data samples (half members and half generated non-members). The test dataset consists of 10000 samples, again half members and half generated non-members. The actual and final number of members and non-members that ended up in the test set depends on the Bernoulli samples in our auditing game. We repeat the training process over 5 times, each time independently resampling the training dataset for the MIA and baseline (keeping the train dataset size fixed). We report all our results over a 95% confidence interval and report the standard deviation for our results in Table 3. The compute resources used for MIA and Baseline training were mainly running all experiments on cloud-hosted VMs, using 1 v100l GPU, 4 nodes per task, and 32G memory requested for each job on the cloud cluster. The time to run the MIA and baseline attack pipeline was around 10 hours including hyperparameter tuning on ϵ lb and c lb respectively.

C.2 Language Modeling

Target model. The target model (a small GPT-2 Radford et al. (2019)) task is causal language modeling (CLM), a self-supervised task in which the model predicts the next word in a given sequence of words ( i.e. , context), which is done on a subset of WikiText-103 dataset Merity et al. (2016), a collection of Wikipedia articles. While the GPT-2 training dataset is undisclosed, Radford et al. (2019) do state that they did not include any Wikipedia document.

One common standard pre-processing in CLM is to break the tokenized training dataset into chunks of equal sizes to avoid padding of the sequences. To achieve this, the entire training dataset is split into token sequences of fixed length, specifically 64 tokens per sequence, refer hereafter as 'chunks'.

(a) Real

(b) Synthetic

(c) Real

(d) Synthetic

Each of these chunks serves as an individual sample within our training dataset. Therefore, in the experiments, the membership inference will be performed for a chunk. In other words, we adopt an example-level differential privacy (DP) viewpoint, where an example is considered a sequence of tokens that constitute a row in a batch of samples. This chunk-based strategy offers additional benefits in synthetic sample generation. By maintaining uniformity in the length of these synthetic samples, which also consists of 64 tokens, it is possible to effectively mitigate any distinguishability between synthetic and real samples based solely on their length. We also consider the possibility of weak correlations between chunks (e.g., being from the same article). To mitigate this, we perform a form of stratified sampling as a pre-processing step. From each document, we include only the first 50 chunks and discard the rest. We select 2,113 documents in total, resulting in 105,650 chunks in our dataset.

We split the dataset into train and test with a 90:10 ratio for the target model (We insist on following the typical training pipelines in machine learning, since we plan to conduct post-hoc audits with PANORAMIA ). The test dataset provides real non-member samples, allowing comparison to the O(1) auditor and another version of our approach which uses real non-member data. The training dataset ( D f ) contains 95,085 samples, while the test dataset consists of 10,565 samples.

We train the target model for 200 epochs in total. Figure 8 displays how the training and validation cross-entropy loss changes throughout the training, when we audit models from epochs 37 (best generalization), 75, and 150. To check how well the target model generalizes, we look at the crossentropy loss (on a validation set), which is the only metric in a causal language model task to report (or the perplexity Radford et al. (2019) which conveys the same information).

0

25

50

75

100

125

150

175

200

Epoch

0

1

2

3

4

Loss

Train Loss

Validation Loss

The compute resources used for training the target models are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the target model for 200 epochs required about 13 hours.

Generator. The G is a GPT-2 fine-tuned using a CLM task on dataset D G , a subset of D f . To create synthetic samples with G , we use topk sampling Holtzman et al. (2019) method in an auto-regressive manner while keeping the generated sequence length fixed at 64 tokens. To make G generate samples less like the real members it learned from (and hopefully, more like real non-members), we introduce more randomness in the generation process in the following manner. We choose a top_k value of

200, controlling the number of available tokens for sampling. In addition, we balance the value of top_k, making it not too small to avoid repetitive generated texts but also not too large to maintain the quality. Finally, we fix the temperature parameter that controls the randomness in the softmax function at one. Indeed, a larger value increases the entropy of the distribution over tokens resulting in a more diverse generated text.

However, the quality of synthetic text depends on the prompts used. To mitigate this issue, we split the D G dataset into two parts: D train G and D prompt G . G is fine-tuned on D train G , while D prompt G is used for generating prompts. Numerically, the size of the D train G is 35% of the D f and the size of D prompt G is 15% of D f . During generation, we sample from D prompt G , which is a sequence of length 64. We feed the prompt into the generator and request the generator to generate a suffix of equivalent length. In our experiments, for each prompt, we make 8 synthetic samples, supplying a sufficient number of synthetic non-member samples for different parts of the pipeline of PANORAMIA , including training the helper model, and constructing D tr out , D te out . Overall, we generate 92276 synthetic samples out of which we use 53502 of them for training the helper model and keep the rest for audit purposes. Note that this generation approach does not inherently favor our audit scheme, and shows how PANORAMIA can leverage existing, public generative models with fine tuning to access high-quality generators at reasonable training cost.

We also perform a sanity check on the quality of the generated data. We visualize the loss distribution of in-distribution members, in-distribution non-members, and synthetic data under both the target and helper models, aiming for synthetic data that behaves similarly to in-distribution non-member data. Figure 14 visualizes these distributions for our target and helper models.

The compute resources used for fine-tuning the generator model, and synthetic text generation are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the GPT2-small model took approximately 1.5 hours, and generating the synthetic samples required about 1 hour.

Baseline & MIA. For both the baseline and PANORAMIA classifiers, we employ a GPT-2 based sequence classification model. In this setup, GPT-2 extracts features ( i.e. , hidden vectors) directly from the samples. We concatenate a vector of features extracted using the helper h (for the baseline) or target model f (for the MIA) to this representation. This vector of features is the loss sequence produced in a CLM task for the respective model. These two sets of extracted features are then concatenated before being processed through a feed-forward layer to generate the logit values required for binary classification (distinguishing between members and non-members).

In the main results in Table 3, for both the baseline and PANORAMIA classifiers, the training and validation sets consist of 20000 and 2000 samples, respectively, with equal member non-member distribution. The test set consists of 10000 samples, in which the actual number of members and non-members that ended up in the test set depends on the Bernoulli samples in the auditing game. The helper model is fine-tuned for 60 epochs on synthetic samples and we pick the model that has the lowest validation loss throughout training, generalizing well.

As the GPT-2 component of the classifiers can effectively minimize training loss without achieving strong generalization, regularization is applied to the classifier using weight decay. Additionally, the optimization process is broken into two phases. In the first phase, we exclusively update the parameters associated with the target (in PANORAMIA ) or helper model (in the baseline) to ensure that this classifier has the opportunity to focus on these specific features. In the second phase, we optimize the entire model as usual. We train both the baseline and Membership Inference Attack (MIA) models with 5 different seeds. The randomness is over: 1. The data split into train/validation/test sets for both the baseline and MIA classifiers. Each classifier is retrained for every seed. 2. The coin flips in our auditing game (see Definition 2). For a fixed seed mentioned above, we employ an additional set of 5 seeds to account for the randomness in the training algorithm (such as initialization of the model, and batch sampling). We then train a classifier for each of these seeds. From the 5 models generated for each latter seed, we select the one with the largest c lb value (for the baseline) or { c + ϵ } lb (for MIA), determined over the validation set. We calculate the 95% confidence interval of our measurements using the t-score.

The compute resources used for MIA and Baseline training were 1 GPU with 32gb memory, running on cloud-hosted VMs. The compute time depends on the training set size, for the largest training size, it took about 3 hours to train the baseline or MIA.

The compute resources used for fine-tuning the helper model are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, it took about 4 hours to fine-tune the helper model.

C.3 Tabular data

Target f and the helper model h . The Target f and the helper h model are both Multi-Layer Perceptron with four hidden layers containing 150, 100, 50, and 10 neurons respectively. Both models are trained using a learning rate of 0.0005 over 100 training epochs. In the generalized case and to obtain the embedding, we retain the model's parameters that yield the lowest loss on a validation set, which typically occurs at epoch 10 (MLPE_10). For the overfitted scenario, we keep the model's state after the 100 training epochs (MLP_E100). The non overfitted model and the overfitted one achieved an accuracy of 86% and 82% respectively.

Generator. We use the MST method McKenna et al. (2021) which is a differentialy-private method to generate synthetic data. However, as we do not need Differential Privacy for data generation we simply set the value for ε to 1000 . Synthetic data generators can sometimes produce bad-quality samples. Those out-of-distribution samples can affect our audit process (the detection between real and synthetic data being due to bad quality samples rather than privacy leakage). To circumvent this issue, we train an additional classifier to distinguish between real data from D G and additional synthetic data (not used in the audit). We use this classifier to remove from the audit data synthetic data synthetic samples predicted as synthetic with high confidence.

Baseline and MIA. To distinguish between real and synthetic data, we use the Gradient Boosting model and conduct a grid search to find the best hyperparameters.

C.4 Comparison with Privacy Auditing with One (1) Training Run: Experimental Details

We implement the black-box auditor version of O(1) approach Steinke et al. (2023). This method assigns a membership score to a sample based on its loss value under the target model. They also subtract the sample's loss under the initial state (or generally, a randomly initialized model) of the target model, helping to distinguish members from non-members even more. In our instantiation of the O(1) approach, we only consider the loss of samples on the final state of the target model. Moreover, in their audit, they choose not to guess the membership of every sample. This abstention has an advantage over making wrong predictions as it does not increase their baseline. Roughly speaking, their baseline is the total number of correct guesses achieved by employing a randomized response ( ϵ, 0) mechanism, for those samples that O(1) auditor opts to predict. We incorporate this abstention approach in our implementation by using two thresholds, t + and t -. More precisely, samples with scores below t + are predicted as members, those above t -as non-members, and the rest are abstained from prediction. We check for different combinations of t + and t -and report the highest ϵ among them. This involves performing multiple tests on the same evidence, each with a separate confidence level of 95%. To ensure that these tests hold collectively, we apply a union bound and adjust the significance level accordingly. The total number of separate tests depends on the different number of guesses that O(1) can make. We also set δ to 0 and use a confidence interval of 0.05 in their test. In PANORAMIA , for each hypothesis test (whether for c lb or { c + ϵ } lb), we stick to a 0.025 confidence interval for each one, adding up to an overall confidence level of 0.05.

D Results: Additional Experiments and Detailed Discussion

D.1 Baseline Strength Evaluation on Text and Tabular datasets

The basis of our privacy measurement directly depends on how well our baseline classifier distinguishes between real member data samples and synthetic non-member data samples generated by our generative model. As mentioned in Section 5.1 and Appendix C.1, in order to increase the performance of our baseline b , we mimic the role of the target model f 's loss in the MIA using a helper model h , which adds a loss-based feature to b . To assess the strength of the baseline classifier, we train the baseline's helper model h on two datasets, synthetic data, and real non-members. We also train a baseline without any helper model. This enables us to identify which setup allows the baseline classifier to distinguish the member and non-member data the best in terms of the highest c lb value ( c lb is reported for each case in table 5). For initial analysis, we keep a sub-set of real data as non-members to train a helper model with them, for conducting this ablation study on the baseline.

Our experiments confirm that the baseline with loss values from a helper model trained on synthetic data performs better compared to a helper model trained on real non-member data. This confirms the strength of our framework in terms of not having a dependency of using real non-members. Therefore, we choose this setting for our baseline helper model. In addition, we also experiment with different helper model architectures in the case of CIFAR10, looking for the best baseline performance among them. For the WikiText dataset, the c lb reported in Table 5 differs from that in Table 3 because we had less data available for this experiment. We withheld half of the real samples to train the helper model with real samples in its training set.

For each data modality setting, we also train the baseline classifier on an increasing dataset size of real member and synthetic non-member data points to gauge the trend of the baseline performance as the training dataset size increases (see Figure 2 and Figure 9). This experiment allows us to aim for a worst-case empirical lower bound on how close the generated non-member data is to the real member data distribution, i.e. what would be the largest c lb we can reach as we increase the training set size. Subsequently, we can compare the c lb reported in table 3 (corresponding to the vertical dashed lines in Figure 2 and Figure 9) to the largest c lb we can achieve in this experiment. In the following, we explain why we chose that number of training samples for our baseline in the main results.

On the WikiText dataset, we increase the baseline's training set size up to 50k, while fixing the validation and test sets sizes to 2000 and 10000, respectively). Figure 9 shows the changes of c lb while varying the training set size. The vertical dashed line specified the number of training samples we chose for our main auditing results reported in Table 3.

Now, let us discuss why we don't choose the largest training set size available for the baseline. The measurement for c lb depends not only on the training set size but also on the test set size. Therefore, we need to balance the sizes of the train and test sets. We conducted another experiment where we varied the train set size and, instead of fixing the test set size, allocated the remaining available data to the test set (keeping in mind that our data for the audit is limited by the number of real members allocated for audit purposes).Figure 9(b) depicts the trend of changes. As expected, we observed an initial increase in c lb as the training set size increased (and the number of test data is sufficient to capture that). However, it decreased toward the end because of the smaller test set size. For a train set size of 40k, we found the best trade-off between train and test set sizes, which justifies our choice of 40k for the training set size in the main results. The same argument applies for other data modalities.

Let us make one last point. It is also important to note that the MIA is expected to show similar behavior if we use a larger train set size or test set size. Hence, the gap between the performance of MIA and baseline (which is our privacy measurement) is not expected to be affected (see Appendix D.4).

The experiments are repeated five times, and we provide a 95% confidence interval. For the image data modality, the randomness lies in the training data provided to the classifiers, while the test set remains fixed throughout. For the text data modality, both the train and test sets change across the five runs (the test set size is fixed to 10k).

D.2 Further Analysis on General Privacy Measurement Results

The value of ˜ ϵ for each target model is the gap between its corresponding dashed line and the baseline one in Figure 4. This allows us to compute values of ˜ ϵ reported in Table 3. It is also interesting to note

2000

10000

20000

30000

40000

50000

Training Dataset Size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

clb

Baseline-WikiText

(a) c lb as the size of the training set increases for baseline (while the test set size is 10k), trained to classify real (WikiText) and synthetic data points. The vertical dashed line indicates the number of samples we used to train the baseline in our main results, reported in Table 3.

2000 10000

20000

30000

40000

50000

60000

70000

Training Dataset Size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

clb

Baseline-WikiText

(b) c lb as the size of the training set increases for baseline, while the test set size is changing at the same time. The number of test samples for a given train set size is specified by the total number of samples we have for the audit ( D in), which is limited by the number of real members.

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0

2000

4000

6000

8000

Number of Predictions

Baseline

PANORAMIA: ResNet101_E20

PANORAMIA: ResNet101_E50

PANORAMIA: ResNet101_E100

Number of true positives

(a) Number of predictions

of ResNet101 on CIFAR10

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0

2000

4000

6000

8000

Number of Predictions

Baseline

PANORAMIA: CNN_E50

PANORAMIA: CNN_E100

Number of true positives

(b) Number of predictions

of Multi-Label CNN on the

CelebA

0.0

0.2

0.4

0.6

0.8

1.0

Recall

0

2000

4000

6000

8000

10000

Number of Predictions

Baseline

Number of True Positives

PANORAMIA GPT-2_E150

PANORAMIA GPT-2_E75

PANORAMIA GPT-2_E37

(c) Number of predictions

of GPT-2 model on Wiki-

Text

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Recall

0

2000

4000

6000

8000

Number of Predictions

Baseline

PANORAMIA MLP E100

PANORAMIA MLP E10

Number of true positive

(d) Number of predictions

of an MLP on the Adult

dataset.

that the maximum value of { c + ϵ } lb typically occurs at low recall. Even if we do not use the same metric (precision-recall as opposed to TPR at low FPR) this is coherent with the findings from Carlini et al. (2022b). We can detect more privacy leakage (higher { c + ϵ } lb which leads to higher ˜ ϵ ) when making a few confident guesses about membership rather than trying to maximize the number of members found ( i.e. , recall). Figure 10, decomposes the precision in terms of the number of true positives and the number of predictions for a direct mapping to the propositions 1 and 2. These are non-negligible values of privacy leakage, even though the true value is likely much higher.

D.3 PANORAMIA performance with increasing Test and Train Size m

PANORAMIA : Increasing Test Set Size: We compare PANORAMIA with the O(1) audit (Steinke et al., 2023) (in input space, without canaries), which uses a loss threshold MIA. As shown in Figure 6(b) and Figure 6(c) in the CIFAR10 image dataset case, both for non-private and Differentially Private (DP) models, PANORAMIA can leverage much more data for its measurement (up to the whole training set size for training and testing the MIA), while in our setting O(1) is limited by the size of the test set (for non-members). We vary test size from m = 500 , up to 35 k . The experiment is repeated 10 times with resampling of the test set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence.

D.4 PANORAMIA with Increasing Train Dataset Size

We also conduct experiments to show the effect of increasing training dataset size on the performance of PANORAMIA (as well as the baseline). In the CIFAR10 image dataset case, Figure 11, we see PANORAMIA performance slowly increase as the train dataset size increases. It reaches its max performance at dataset sizes 4500 -5000 , 10 k, 20 k -30 k and slowly begins to stabilize at the higher dataset sizes. We vary train size from m = 500 , up to 30 k . The experiment is repeated 5 times with resampling of the train set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence.

500

5000

10000

15000

20000

25000

30000

Train Dataset Size

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Measurement of

clb

or { +

c

}

lb

Baseline-CIFAR10

Panoramia-ResNet101_E100

D.5 PANORAMIA in cases of no privacy leakage

As illustrated in Figure 4(c) and Table 3, there exist instances in which the c lb is as large as { c + ϵ } lb . For instance, for GPT-2_E37, the upper bound in the confidence interval measured for c lb is larger than the lower bound of the measurement for { c + ϵ } lb . Although the measured ˜ ϵ is positive on average, but the lower bound of the interval is negative. For such cases, we don't claim any detection of privacy leakage. This implies that, under certain conditions, the baseline could outperform the MIA in PANORAMIA . We can see three possible reasons for this behavior. (1) The synthetic data might not be of high enough quality. If there is an evident distinction between real and synthetic samples, the baseline cannot be fooled reducing the power of our audit. (2) The MIA in PANORAMIA is not powerful enough. (3) The target model's privacy leakage is indeed low. For GPT-2_E150 and GPT-2_E75 in Figure 4(c), we can claim that given the target model's leakage level, the synthetic data and the MIA can perform the audit to a non-negligible power. For GPT-2_37, we need to examine the possible reasons more closely. Focusing on generated data, in the ideal case we would like the synthetic data to behave just like real non-member data on Figure 14(a). If this was the case, { c + ϵ } lb would go up (since the MIA uses this loss to separate members and non-members). Nonetheless, predicting changes in c lb remains challenging, as analogous loss distributions do not guarantee indistinguishability between data points themselves (also, we cannot necessarily predict how the loss distribution changes under the helper model h in Figure 14(d)). Hence, reason (1) which corresponds to low generative data quality, seems to be a factor in the under-performance of PANORAMIA . Moreover, in the case GPT-2_37, we observe that real members and non-members have very similar loss distributions. Hence, factor (2) also seems to be at play, and a stronger MIA that does not rely exclusively on the plot may increase auditing power. Finally, the helper model h seems to have a more distinct loss distribution when compared to GPT-2_37 (see Figure 14(d)).

D.6 PANORAMIA with Tabular data

As we can see in Table 6, our proposed method PANORAMIA cannot detect any meaningful privacy leakage in the case of a small MLP classifier trained on tabular data.

As shown in Figure 12(b) there is no significant difference between the values ( c + ε ) lb c lb returned by PANORAMIA . As a result we get ˜ ε = 0

Average c_lb vs. Recall with 95% Confidence Interval

1.0

Mean c_Ib

95% Confidence Interval

2.0

Mean (c+eps)_Ib MLP 10

95% Confidence Interval

Mean (c+eps)_Ib MLP 100

95% Confidence Interval

0.9

1.5

recision

0.8

qI">

1.0

0.7

baseline model

0.5

0.6

PANORAMIAMLP_E100

PANORAMIA MLP_E10

Theoreticalmaximumprecision

0.5

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

(a) precision vs recall, for tabular one instance of the tabular data classification audit.

Average Recall

(b) Average { c + ϵ } lb (or c lb) vs recall with 95% confidence, for tabular data classification. Randomness comes from resampling of the MIA's train and test set as well as retraining the MIA

The high precision at low recall in the baseline as can be seen in Figure 12(a) illustrates the fact that some real data can easily be distinguished from the synthetic ones. Instead of poor quality synthetic data , we are dealing with real observation that are actually 'looking to real'. Investigating those observations reveals that they have extreme or rare values in one of their features. Indeed, the Adult dataset contains some extreme values ( e.g. , in the capital_gain column), such extreme values are rarely reproduced by the generative model. This leads to the baseline accurately predicting such observations as real with high confidence that translates into a high c lb . Moreover, the audit values maximum values ( c + ε ) lb and c lb being achieved for a relatively small number of guesses corresponding to extreme values, the audits results vary a lot depending on the sampling or not of such extreme values in the audit test set. We believe this is part of the reason why we observe an high variance in the audit results leading to no significant privacy leakage detected by PANORAMIA .

This is a potential drawback of our auditing scheme, while the presence of extreme or rare values in the dataset usually allows for good auditing results it can have a negative impact on PANORAMIA 's success.

D.7 Privacy Auditing of Overfitted ML Models

Methodology. Varying the number of training epochs for the target model to induce overfitting is known to be a factor in privacy loss Yeom et al. (2018); Carlini et al. (2022a). As discussed in Section 5.2, since these different variants of target models share the same dataset and task, PANORAMIA can compare them in terms of privacy leaking.

To verify if PANORAMIA will indeed attribute a higher value of ˜ ϵ to more overfitted models, we train our target models for varying numbers of training epochs. The final train and test accuracies are reported in Table 1. For GPT-2 (as a target model), we report the train and validation loss as a measure of overfitting in Figure 8. Figures 13 and 14 show how the gap between member data points ( i.e. , data used to train the target models) and non-member data points (both real as well as

generated non-members) increases as the degree of overfitting increases, in terms of loss distributions. We study the distribution of losses since these are the features extracted from the target model f or helper model h , to pass respectively to PANORAMIA and the baseline classifier. The fact that the loss distributions of member data become more separable from non-member data for more overfitted models is a sign that the corresponding target model could leak more information about its training data. We thus run PANORAMIA on each model, hereafter presenting the results obtained.

0.0

0.1

0.2

0.3

0.4

0.5

Losses

0

5

10

15

20

25

Frequency (normalized)

real member losses

fake non-member losses

real non-member losses

(a) CelebA Multi-Label CNN_E30.

0.0

0.1

0.2

0.3

0.4

0.5

Losses

0

5

10

15

20

25

Frequency (normalized)

real member losses

fake non-member losses

real non-member losses

(b) CelebA Multi-Label CNN_E100; an overfit model.

Results. In Figure 3, we observe that more training epochs ( i.e. , more overfitting) lead to better precision-recall trade-offs and higher maximum precision values. Our results are further confirmed by Figure 10 with PANORAMIA being able to capture the number of member data points (positive predictions) better than the baseline b .

In Table 3, we further demonstrate that our audit output ˜ ϵ orders the target models in terms of privacy leakage: higher the degree of overfitting, more memorization and hence a higher ˜ ϵ returned by PANORAMIA . From our experiments, we consistently found that as the number of epochs increased, the value of ˜ ϵ also increased. For instance, in the WikiText dataset, since we observe { c + ϵ } GPT-2_E150 lb > { c + ϵ } GPT-2_E75 lb > { c + ϵ } GPT-2_E37 lb , we would have { c + ϵ } GPT-2_E150 lb -c lb > { c + ϵ } GPT-2_E75 lb -c lb > { c + ϵ } GPT-2_E37 lb -c lb, which leads to ˜ ϵ GPT-2_E150 > ˜ ϵ GPT-2_E75 > ˜ ϵ GPT-2_E37. Our experiment is coherent with the intuition that more training epochs lead to more over-fitting, leading to more privacy leakage measured with a higher value of ˜ ϵ .

4.0

3.5

+

3.0

or

CIb

2.5

Measurement for

2.0

CIb

PANORAMIA:ResNet101

1.0

PANORAMIA:ResNet50

0.5

PANORAMIA:WideResNet-28-2

empiricalmaximumvalue

0.0

0.0

0.2

0.4

0.6

0.8

1.0

Recall

1

2

3

4

5

6

Loss values

0.0

0.2

0.4

0.6

0.8

Frequency (normalized)

Real Members

Synthetic Non-members

Real non-members

(a) GPT-2_E37

1

2

3

4

5

6

Loss values

0.0

0.2

0.4

0.6

0.8

1.0

Frequency (normalized)

Real Members

Synthetic Non-members

Real non-members

(b) GPT-2_E75

0

1

2

3

4

5

6

Loss values

0.0

0.2

0.4

0.6

0.8

1.0

Frequency (normalized)

Real Members

Synthetic Non-members

Real non-members

(c) GPT-2_E150

1

2

3

4

5

6

Loss values

0.0

0.2

0.4

0.6

0.8

1.0

Frequency (normalized)

Real Members

Synthetic Non-members

Real non-members

(d) Helper model

h

D.8 Models of varying complexity:

Carlini et al. (2021) have shown that larger models tend to have bigger privacy losses. To confirm this, we also audit ML models with varying numbers of parameters, from a ≈ 4 M parameters Wide ResNet28-2, to a 25 . 5 M parameters ResNet-50, and a 44 . 5 M parameters ResNet-101. Figure 15 shows that PANORAMIA does detect increasing privacy leakage, with ˜ ϵ wide -resnet ≤ ˜ ϵ resnet 50 ≤ ˜ ϵ resnet 101 .

D.9 Privacy Auditing of Differentially-Private ML Models

Methodology.

For the image data modality, we evaluate the performance of PANORAMIA on differentially-private ResNet-18 and Wide-ResNet-16-4 models on the CIFAR10 dataset under different target privacy budgets ( ϵ ) with δ = 10 -5 and the non-private ( ϵ = ∞ ) cases. The models are trained using the DP-SGD algorithm (Abadi et al., 2016) using Opacus (Yousefpour et al., 2021), which we tune for the highest train accuracy on learning rate lr , number of epochs e , batch size bs and maximum ℓ 2 clipping norm (C) for the largest final accuracy.

For the text data modality, we evaluate the performance of PANORAMIA on the differentially private GPT-2-Small model, which has been fine-tuned on the WikiText dataset (the preprocessing steps are identical to those in Appendix C.2). As with the image modality, we use various target privacy budgets ( ϵ ) with δ = 10 -5 , as well as non-private settings ( ϵ = ∞ ). We train the models using the DP-SGD algorithm Abadi et al. (2016), incorporating Ghost clipping Li et al. (2021) for calculating

the per-example gradient. We minimize the validation loss by tuning hyperparameters such as the learning rate ( lr ), number of epochs ( e ), batch size ( bs ), and clipping norm ( C ). The lowest validation loss values obtained are 3 . 79 , 3 . 7 , and 3 . 65 for ϵ values of 1, 3, and 10, respectively. For the non-private GPT-2-Small model, the lowest validation loss achieved is 3 . 341 .

The noise multiplier σ is computed given ϵ , number of epochs and batch size. We use both PANORAMIA and O(1) (Steinke et al., 2023) privacy loss measurements under the pure ϵ -DP analysis.

Results. For the image data modality, tables 7 and 8 summarizes the auditing results of PANORAMIA on different DP models. For ResNet-18, we observe that at ϵ = 1 , 2 , 4 , 6 (more private models) PANORAMIA detects no privacy loss, whereas at higher values of ϵ = 10 , 15 , 20 (less private models) and ϵ = ∞ (a non-private model) PANORAMIA detects an increasing level of privacy loss with ˜ ϵ ϵ =10 < ˜ ϵ ϵ =15 < ˜ ϵ ϵ =20 < ˜ ϵ ϵ = ∞ , suggesting a higher value of ϵ correspond to higher ˜ ϵ . We observe a similar pattern with Wide-ResNet-16-4, in which no privacy loss is detected at ϵ = 1 , 2 and higher privacy loss is detected at ϵ = 10 , 15 , 20 , ∞ . We also compare the auditing performance of PANORAMIA with that of O(1) (Steinke et al., 2023), with the conclusion drawn by these two methods being comparable. For both ResNet-18 and Wide-ResNet-16-4, O(1) reports values close to 0 (almost a random guess between members and non-members) for ϵ < 10 DP models, and higher values for ϵ = 10 , 15 , 20 , ∞ DP models. The results suggest that PANORAMIA is potentially an effective auditing tool for DP models that has comparable performance with O(1) and can generalize to different model structures.

For the text data modality, Table 9 summarizes the auditing results of PANORAMIA on differentially private (fine-tuned) Large Language Models. In all cases, neither PANORAMIA not O(1) Steinke et al. (2023) do not detect any privacy loss.

Discussion. In the image data modality, we observe that the auditing outcome ( ˜ ϵ values for PANORAMIA and ϵ for O(1)) can be different for DP models with the same ϵ values (Table 10). We hypothesize that the auditing results may relate more to level of overfitting than the target ϵ values in trained DP models. The difference between train and test accuracies could be a possible indicator that has a stronger relationship with the auditing outcome. We also observe that O(1) shows a faster increase in ϵ for DP models with higher targeted ϵ values. We believe it depends on the actual ratio of the correct and total number of predicted samples, since O(1) considers both true positives and true negatives while PANORAMIA considers true positives only. We leave these questions for future work.

In the text data modality, the membership inference attacks (MIAs) relying on loss thresholds in privacy audits, without retraining or access to canaries, may not be sufficiently strong to effectively classify members and non-members.

E Generator Closeness γ -relaxation

Definition 4 ( ( c, γ ) -closeness) . Let the γ -approximate maximum divergence between generative model G and data distribution D be,

For all c > 0 we say G is ( c, γ ) -close to D if D γ ∞ ( D∥G ) ≤ c , i.e.

E.1 Pure DP case

First we prove a similar result as Lemma 5.5 (Steinke et al., 2023) but with the one-sided inequality of ( c, γ ) -closeness and the case when we additionally observe output from a pure-DP mechanism.

Lemma 1. Let G , D be probability distributions over Y , fix c, γ ≥ 0 . Suppose for all measurable S ⊂ Y we have e -c ( D ( S ) -γ ) ≤ G ( S ) . Then ∃ γ ′ ∈ [0 , γ ] and D ′ , D ′′ , G ′ , G ′′ such that the following properties are satisfied.

D , G can be expressed as a convex combination D = (1 -γ ′ ) D ′ + γ ′ D ′′ , G = (1 -γ ′ ) G ′ + γ ′ G ′′ ;

for all measurable S ⊂ Y , we have e -c D ′ ( S ) ≤ G ′ ( S ) ;

there exists measurable S ⊂ Y such that D ′′ ( S ) = 1 , ∀ S ′ ⊂ S, D ( S ′ ) ≥ G ( S ′ ) .

Let M denote an arbitrary ϵ -DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P , Q be probability distributions over Z which are distributions of M ( Y ) when Y is sampled from D and G respectively. Fix ϵ ≥ 0 , suppose for all measurable T ⊂ Z we have P ( T ) ≤ e ϵ Q ( T ) and Q ( T ) ≤ e ϵ P ( T ) . When additionally observing the output from a DP mechanism we have the following properties. Let ( D , P ) , ( G , Q ) be the joint distribution over ( Y , Z ) which we assume D ⊥ ⊥ P , G ⊥ ⊥ Q .

( D , P ) , ( G , Q ) can be expressed as a convex combination DP = ((1 -γ ′ ) D ′ + γ ′ D ′′ ) P , GQ = ((1 -γ ′ ) G ′ + γ ′ G ′′ ) Q ;

for all measurable S ⊂ Y , T ⊂ Z , we have e -c e -ϵ D ′ ( S ) P ( T ) ≤ G ′ ( S ) Q ( T ) ;

there exists measurable S ⊂ Y , T ⊂ Z such that D ′′ ( S ) P ( T ) = 1 , ∀ S ′ ⊂ S, T ′ ⊂ T, D ( S ′ ) P ( T ) ≥ G ( S ′ ) P ( T ) .

Proof. For the edge cases of γ = 0 , the results hold with γ ′ = 0 , D ′ = D , G ′ = G ; similarly when γ = 1 , the results hold with γ ′ = 1 , D ′′ = D , G ′′ = G , D ′ = G ′ . Let c ′ ∈ [0 , c ] , γ 1 , γ 2 ∈ (0 , 1) , define distribution G ′ , G ′′ , D ′ , D ′′ as follows. For all points y ∈ Y ,

By construction (1 -γ 1 ) D ′ + γ 1 D ′′ = D , (1 -γ 2 ) G ′ + γ 2 G ′′ = G , so the first property is satisfied, and D ′′ ( y ) is supported on S = { y ∈ Y : D ( y ) > e c ′ G ( y ) } so the third property is implied. If 0 < γ 1 = γ 2 = γ ′ ≤ γ , for all y ∈ Y we have,

as required for the second property. Following the same as in Lemma 5.5 (Steinke et al., 2023), we can ensure 0 < γ 1 = γ 2 ≤ γ by appropriately setting ϵ 1 , ϵ 2 ∈ [0 , ϵ ] . We can use the same decomposition of D , G to prove the second part of the Lemma. If assuming D ⊥ ⊥ P and G ⊥ ⊥ Q , the joint distributions can be decomposed as f D , P = f P · f D and f G , Q = f Q · f G . Therefore by construction, DP = ((1 -γ ′ ) D ′ + γ ′ D ′′ ) P , GQ = ((1 -γ ′ ) G ′ + γ ′ G ′′ ) Q . D ′′ P is supported on ( S, T ) = { y ∈ Y , M ( y ) ∈ Z : D ( y ) P ( M ( y )) > e c ′ G ( y ) P ( M ( y )) } . For all y ∈ Y , M ( y ) ∈ Z we have,

where Q ( M ( y )) P ( M ( y )) ≥ e -ϵ since M is an ϵ -DP mechanism.

Similar to Lemma 5.6 (Steinke et al., 2023), next we prove a Bayesian version of Lemma 1 which is used to prove the main result afterwards.

Lemma 2. Let G , D be probability distributions over Y , fix c, γ ≥ 0 , suppose for all measurable S ⊂ Y we have e -c ( D ( S ) -γ ) ≤ G ( S ) . Then there exists a randomized function E D , G : Y → { 0 , 1 } with the following properties. Suppose X ∼ Bernoulli( 1 2 ) , if X = 1 sample Y ∼ D , and if X = 0 sample Y ∼ G . Then for all y ∈ Y we have,

,

Let M denote an arbitrary ϵ -DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P , Q be probability distributions over Z which are distributions of M ( Y ) when Y is sampled from D and G respectively. Fix ϵ ≥ 0 , suppose for all measurable T ⊂ Z we have P ( T ) ≤ e ϵ Q ( T ) and Q ( T ) ≤ e ϵ P ( T ) . Let ( D , P ) , ( G , Q ) denote the joint distributions over ( Y , Z ) which we assume D ⊥ ⊥ P , G ⊥ ⊥ Q . Then there exists a randomized function E ( D , P ) , ( G , Q ) : ( Y , Z ) →{ 0 , 1 } such that for all y ∈ Y , M ( y ) ∈ Z we have,

Proof. We apply the decomposition from Lemma 1 and denote D ′ , D ′′ , G ′ , G ′′ , γ ′ as is. We define E D , G : Y → { 0 , 1 } by,

For any y ∈ Y we have,

The first equality uses that P [ A,B | C ] = P [ A | B,C ] P [ B | C ] . The second equality holds since we assume the internal randomness of E is independent of everything else. The third equality uses the Bayes' rule. The fourth equality uses the definition of the randomized function E . The fifth equality uses the decomposition of Lemma 1. After rearranging and use the properties from Lemma 1 we obtain the first inequality, then the rest follows from simplifications. Furthermore we have the

following,

We follow a similar procedure to prove the second part of the Lemma. We define E ( D , P ) , ( G , Q ) : ( Y , Z ) →{ 0 , 1 } by,

Since we define the success probability with the same expression, the expected success of the randomized function directly follows,

Next we use Lemma 2 to prove the following Proposition which is used to form statistical tests on hypothesis H .

Proposition 3. Let G be ( c, γ ) -close, f be output of an ϵ -DP mechanism, T b ≜ B ( S, X ) be the guess from the baseline, T a ≜ A ( S, X, f ) be the guess from the membership audit. Let T b , T a ∈ [0 , 1] m be bounded. Then, for all v ∈ R and all t in the support of T :

where F is independent from S ′ , F ( T b ) , F ( T a ) is supported on { 0 , 1 , . . . , m } and E T b ,F [ F ( T b )] = E T a ,F [ F ( T a )] ≤ 2 mγ .

Proof. Let B ( s ≤ i ) denote the distribution on [0 , 1] m obtained by conditioning B on past S <i = s <i and X ≤ i = x ≤ i . By Lemma 2, for all t b ∈ [0 , 1] m we have

Using the law of total probability we get,

Applying the union bound on S i we can bound the expectation of success events by,

By induction, for k ∈ [ m ] define ˜ W k ( s, t b ) := ∑ k i =1 t b i · s i · E B ( s <i, 1 ) ,B ( s <i, 0 ) ( t b ) , then ˜ W k ( S, t b ) is stochastically dominated by ˇ W k ( t b ) := ∑ k i =1 t b i · ˇ S i ( t ) where ˇ S i ( t ) ∼ Bernoulli( e c 1+ e c ) . Let the failing events F be,

then we have W m ( s, t b ) := ∑ m i =1 t b i · s i = ˜ W m ( s, t b ) + F ( s, t b ) is stochastically dominated by ˇ W m ( T b ) + F ( S, T b ) , where the expectation of the failing events is bounded by,

Similar to Lemma 5.5 in Steinke et al. (2023) since E [ F ( S, T b )] is independent of S we could have F ( T b ) = F ( S, T b ) for S drawn from an appropriate distribution. The proof of the second part of the Proposition is essentially the same except with the following result from Lemma 2,

Similar to Theorem 5.2 (Steinke et al., 2023) we could compute for the optimal distribution F ( t b ) and F ( t a ) by formulating it into a linear programming problem. Similar to Corollary 1, to test Hypothesis H together we would apply Union bound on the two parts of the hypothesis as in Proposition 3.

E.2 Approximate DP case

In this section we consider the case where we have a ( c, γ ) -close generator and audit for an approximate DP mechanism. We modify the results to account for the failing events from these two sources of randomness. We start by proving a similar version of Lemma 2 where we use both the decomposition of G , D from Lemma 1 and the decomposition of P , Q from Lemma 5.5 (Steinke et al., 2023).

Lemma 3. Let G , D be probability distributions over Y , fix c, γ ≥ 0 , suppose for all measurable S ⊂ Y we have e -c ( D ( S ) -γ ) ≤ G ( S ) . Let M denote an arbitrary ( ϵ, δ ) -DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P , Q be probability distributions over Z which are distributions of M ( Y ) when Y is sampled from D and G respectively. Fix ϵ, δ ≥ 0 , suppose for all measurable T ⊂ Z we have P ( T ) ≤ e ϵ Q ( T ) + δ and Q ( T ) ≤ e ϵ P ( T ) + δ . Let ( D , P ) , ( G , Q ) denote the joint distributions over ( Y , Z ) which we assume D ⊥ ⊥ P , G ⊥ ⊥ Q . Then there exists a randomized function E ( D , P ) , ( G , Q ) : ( Y , Z ) → { 0 , 1 } such that for all y ∈ Y , M ( y ) ∈ Z we have,

Proof. For ease of notation we shorthand F = M ( Y ) , E = E ( D , P ) , ( G , Q ) , D ′ = D ′ ( y ) , G ′ = G ′ ( y ) , D ′′ = D ′′ ( y ) , G ′′ = G ′′ ( y ) , P ′ = P ′ ( f ) , Q ′ = Q ′ ( f ) , P ′′ = P ′′ ( f ) , Q ′′ = Q ′′ ( f ) below. We define E : ( Y , Z ) →{ 0 , 1 } by,

For any y ∈ Y , M ( y ) ∈ Z we have,

Using the assumption that D ⊥ ⊥ P , G ⊥ ⊥ Q , we account for the expectation of success events as follows,

Note that we can rewrite the success probability as,

Then we have,

Next we use Lemma 3 to prove the following Proposition to form the statistical tests on hypothesis H for the ( ϵ, δ ) -DP version.

Proposition 4. Let G be ( c, γ ) -close, f be output of an ( ϵ, δ ) -DP mechanism, T b ≜ B ( S, X ) be the guess from the baseline, T a ≜ A ( S, X, f ) be the guess from the membership audit. Let T b , T a ∈ [0 , 1] m be bounded. Then, for all v ∈ R and all t in the support of T :

where F is independent from S ′ , F ( T b ) , F ( T a ) is supported on { 0 , 1 , . . . , m } and E T b ,F [ F ( T b )] ≤ 2 mδ , E T a ,F [ F ( T a )] ≤ 2 m ( γ + δ -γδ ) .

Proof. The first part of the result (baseline test) exactly follows the proof of Proposition 3. For the approximate DP case, the proof of the auditor test follows similar steps except for the bound on the expectation of the failing events,

Similar to the pure DP case we could optimize for the optimal distribution of F ( t b ) and F ( t a ) by solving a linear program (Theorem 5.2 in Steinke et al. (2023)) and apply Union bound as in Corollary 1 to test for Hypothesis H .

E.3 Evaluations

We empirically evaluate the auditing performance of PANORAMIA under different relaxed level of the generator for both the pure and approximate DP cases. We observe that the overall performance is similar to that of no relaxation. We observe smaller ˜ ε (which is analogous to a looser lower bound) as we increase the relaxation level, and in the more extreme cases where we allow many failing events (e.g. when relaxation > 1e-3) we would fail to detect meaningful privacy leakage as ˜ ε = 0 .

F Extended Related Work

Most related privacy auditing works measure the privacy of an ML model by lower-bounding its privacy loss. This usually requires altering the training pipeline of the ML model, either by injecting canaries that act as outliers Carlini et al. (2019) or by using data poisoning attack mechanisms to search for worst-case memorization Jagielski et al. (2020); Nasr et al. (2021). MIAs are also increasingly used in privacy auditing, to estimate the degree of memorization of member data by an ML algorithm by resampling the target algorithm M to bound P ( M | in ) P ( M | out ) Jayaraman & Evans (2019). The auditing procedure usually involves searching for optimal neighboring datasets D,D ′ and sampling the DP outputs M ( D ) , M ( D ′ ) , to get a Monte Carlo estimate of ϵ . This approach raises important challenges. First, existing search methods for neighboring inputs, involving enumeration or symbolic search, are impossible to scale to large datasets, making it difficult to find optimal dataset pairs. In addition, Monte Carlo estimation requires up to thousands of costly model retrainings to bound ϵ with high confidence. Consequently, existing approaches for auditing ML models predominantly require the re-training of ML models for every (batch of) audit queries, which is computationally expensive in large-scale systems Jagielski et al. (2020); Zanella-Béguelin et al. (2022); Lu et al. (2023). This makes privacy auditing computationally expensive and gives an estimate by averaging over models, which might not reflect the true guarantee of a specific pipeline deployed in practice.

Nonetheless, improvements to auditing have been made in a variety of directions. For example, Nasr et al. (2023) and Maddock et al. (2023) have taken advantage of the iterative nature of DP-SGD, auditing individual steps to understand the privacy of the end-to-end algorithm. Andrew et al. (2023) leverage the fact that analyzing MIAs for non-member data does not require re-running the algorithm. Instead, it is possible to re-sample the non-member data point: if the data points are i.i.d. from an asymptotically Gaussian distribution with mean zero and variance 1 /d , this enables a closed-form analysis of the non-member case.

Recently, Steinke et al. (2023) proposed a novel scheme for auditing differential privacy with O (1) training rounds. This approach enables privacy audits using multiple training examples from the

same model training, if examples are included in training independently (which requires control over the training phase, and altering the target model). They demonstrate the effectiveness of this new approach on DP-SGD, in which they achieve meaningful empirical privacy lower bounds by training only one model (the strongest results are achieved by including canaries in the training set though, which is not possible in our setting), whereas standard methods would require training hundreds of models. Our work builds closely on the theory from Steinke et al. (2023), but introduces a baseline model to account for distribution shifts. The key difference that enables our work to account for member/non-member distribution shifts lies in how we create the audit set in our privacy game (Definition 2). In Steinke et al. (2023), the audit set is fixed, and data points are randomly assigned to member or non-member by a Bernoulli random variable S . Members are actually used in training the target model f , while non-members are not (so assignment happens before training). In our framework, we take a set of known members (after the fact), and pair each point with a non-member(generated i.i.d. from the generator distribution). We then flip S to sample which data point of each pair will be shown to the 'auditor' (MIA/baseline) for testing, thereby creating the test task of our privacy measurement. When we replace our generated data with in-distribution independent non-members ( PANORAMIA RM;RN), we exactly enforce the same independence as Steinke et al. (2023) (and as a result have c = 0 , and ˜ ϵ is a lower-bound on ϵ ), except that we 'waste' some member data by drawing our auditing game after the fact. The difference in analysis accounts for distribution shifts when our non-members are generated.

The field of MIA without direct connections to privacy leakage measurement has also seen a lot of recent activity, with new proposals to improve the strength of MIAs in various settings (Carlini et al., 2022b; Nasr et al., 2019; Suri et al.; Zarifzadeh et al., 2023). While it is not the focus of this paper, an interesting avenue for future work is to port ideas from these new MIAs to improve PANORAMIA 's MIA and baseline. Such transfer of MIA progress could lead to more powerful measurements with PANORAMIA . More closely related to our proposal, a recent set of works on Das et al. (2024); Duan et al. (2024); Meeus et al. (2024) MIAs on foundation models has observed that current evaluations are limited due a lack of availability of non-member data. Indeed, foundation models typical include all known data at the time of their training, and there is no well known public set of in-distribution non-member data. Evaluation tasks typical rely on more recent data, which is known to consist in non-members, but suffers from distribution shifts. Das et al. (2024); Duan et al. (2024); Meeus et al. (2024) all show that such a shift invalidates MIA evaluations by showing that 'blind attacks' (i.e., our baselines) perform better than proposed MIAs. One could apply our proposed framework to such a setting: when the baseline in our framework performs better than the MIA, PANORAMIA returns a privacy loss measurement of zero instead of misleading measurements of MIA performance. We thus believe that PANORAMIA is an important step towards a theory for rigorous evaluations of membership inference for MIA models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and Introduction sections highlight the motivation and contributions we make; We provide, to the best of our ability, comprehensive experimental results in Section 5 to justify our claims and contributions. Section 4 provides the theory behind our experiments, whilst Appendix B details proofs.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss these in Section 4.2 and Section 6 of the paper.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we present sections assumptions, theory, and formal proofs in 3, 4 and Appendix B, E provides respectively.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We attach the code as supplementary material in a zip file and also include the main experimental details in Section 5 as well as a detailed experimental setup, hyperparameters, compute details, and model architecture details along with other full necessary information in Appendix C to ensure reproducibility. Section 3 also includes the algorithm as well as the system design diagram detailing our privacy measurement framework.

We also updated the paper and experiments according to the reviewers' helpful feedback.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We specify the details of the code we release in the zip file we attach. The code is primarily the PANORAMIA privacy measurement pipeline. We include structured files specifying run instructions, licenses etc. Training details are within the code itself and also highlighted in Appendix C.

Now we also updated the paper with an open source github code link given under the experimental section of the main paper.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Section 5 and Table 1 give key experimental details, whilst the full experimental setup is highlighted in Appendix C, as well as the zip file of code attached.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We repeated all experiments that involved training data resampling 5 times, and repeated experiments that involved test data resampling over 10 times. All our figures and plots in the core body of the paper report results with error bars, with all results reported at a 95% confidence interval. We also report the standard deviation in the tables in Section 5.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information on compute resources for each modality in Appendix C. We also updated compute times to run the PANORAMIA pipeline in C for each data modality.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and conform to them in every respect.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes] .

Justification: We discuss these in Section 6; overall, our approach can only help model providers and owners to assess the privacy of risk of their data and is not applicable as a privacy attack. As a consequence, we believe that any impact from our proposed work is likely to be positive.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer:[NA] .

Justification: Our paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: All papers, code, and asset owners and authors have been properly cited to the best of our knowledge.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes] .

Justification: We specify the details of the code we release in the zip file we attach. The code is primarily the PANORAMIA privacy measurement pipeline. We include structured files specifying run instructions, licenses, etc. Training details are within the code itself and also highlighted in Appendix C.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing, nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer:[NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to estimate a lower bound on the (pure) Differential Privacy parameter $\\epsilon$ for already trained machine learning models using a post-hoc empirical evaluation based on conducting Membership Inference Attacks. Building on the analysis by Steinke et al., 2023, for auditing DP with $O(1)$ training runs, this paper tweaks the setting by using synthetically generated canary data points that closely resemble the training data population, rather than inserting or omitting pathologically crafted canaries that degrade the ML model's performance. This approach is useful because if the canary distribution and the training distribution are (nearly) identical, an already trained model can be considered as the output of the training algorithm on a random partition of the combined training and canary datasets. This means any post-hoc auditing analysis based on membership inference on a randomly and independently selected training dataset yields a valid lower bound on $\\epsilon$. To get such a canary distribution, the authors suggest training a synthetic data generation model to create a canary dataset to get a distribution similar to the population. Additionally, the authors extend Steinke et al.'s analysis to situations where the synthetic data distribution is $c$-close to the true population in a DP-like divergence, although the estimator does not technically yield a lower bound in this case. Through extensive empirical evaluations, the paper demonstrates that the proposed estimation technique is useful as it provides reasonable values that approximate the DP lower bounds.\n\nSTRENGTHS:\nThe paper focuses on an important problem and has the following merits.\n\n- The paper proposes an auditing method that seeks to eliminate the need of altering anything about the training process of ML models, thus allowing post-hoc auditing. \n\n- The paper proposes using synthetic data that resemble the training data instead of adversarially created canaries can reduce the drop in model performance while offering useful DP estimations.\n\n- Empirical evaluations suggest that the estimator correlates with the DP upper bounds and can help identify situations of high privacy leakage.\n\n- The appendix provides an attempt to extend the budget estimator for $(\\epsilon, \\delta)$-DP.\n\nWEAKNESSES:\n- I'm not entirely certain that the analysis presented in Proposition 2 is correct. In the proof of Proposition 2 (lines 433 and 434), the authors use the fact that the model $f$ is $\\epsilon$-DP with respect to the Bernoulli random variables $S$. But in Algorithm 1, we see that $f$ was trained on $D_{f}$ in phase 1, which is independent of $S$ sampled in phase 2. So, the model $f$ is independent of the selection $S$ (i.e., $f \\bot S$). In other words, the equation after line 434 should evaluate to 1, as given $x_i$, the prediction $f(x_i)$ remains the same whether $S_i = 0$ or $S_i = 1$. Perhaps Algorithm 1 requires one training run like Steinke et al. where the model $f$ is trained on $X$ as described in line 2 of Algorithm 1? If that is the case, then the claim that the auditing does not require retraining becomes invalid. If I'm mistaken, could you explain Proposition 2 further?\n\n- In line 196, the authors mention that Algorithm 1 (lines 4-7) does a sweep over all the recall values of the attack, and they adjust the overall significance-level or the $p$-value by taking a union bound over all equation (1) with the significance level discounted to $\\beta \\leftarrow \\beta / m$. The value of $m$ in the experiments ranges from 500 to 30,000. For such values,  assuming $\\beta \\leq 0.05$, the sweeping over all recall values requires $\\sqrt{\\log(m/\\beta)} \\approx 2.4$ whereas when the level of recall is predetermined, the factor in equation (1) is only $\\sqrt{\\log(1/\\beta)} \\approx 1.15$. So, I'm not sure if doing a sweep over recall values will give larger DP estimate values.\n\n- Use of synthetic data from a distribution that is $c$-close (with a small $c$) can make it difficult for the membership inference attacks to work well. If the goal is auditing DP, perhaps the higher MIA precision for a given recall outweighs the drop in ML model's performance. \n\n- The figures haven't been explained very well. In particular, the theoretical maximum precision in Figure 3 and Figure 12 and the empirical maximum value in Figure 4 aren't clear.\n\n- The paper only studies the problem of estimating pure DP and does not present an operationalizable algorithm for $(\\epsilon, \\delta)$-DP, although some results along these lines are motivated in the appendix. Additionally, the obvious weakness (which the authors acknowledge) is that their method does not technically provide a lower bound for $\\epsilon$-DP.\n\n### Minor Points\n\n- Intuitively, when $\\mathcal{G} = \\mathcal{D}$ or in the Real-Member;Real-Nonmember (RM;RN) case where the non-members follow the same distribution, I think the no-retraining-needed argument could work. This would involve modifying Algorithm 1 and reworking Proposition 2 by (A) assuming entries in $D_f$ and $D_G$ are i.i.d. from the same distribution and (B) setting $S$ according to the original train-test split instead. On the other hand, when $\\mathcal{G} \\neq \\mathcal{D}$, I'm not sure if such an argument can be made to work, at least not trivially.\n\n- It's not clear how the mechanism $B(S, X) = \\\\{b(x_1), \\cdots, b(x_m)\\\\}$ in Proposition 1 incorporates the helper model mentioned in Section 5.1 used in the experiments.\n\n- Algorithm 1 (lines 4-7) does not seem to reflect the $p$-value adjustment discussed in lines 198-200. Perhaps the authors might have overlooked this $p$-value adjustment in the experiments as well?\n\nQUESTIONS:\n- How is the helper model used in the baseline trained?\n\n- The lexicographic order in equation (2) appears to be essential for Corollary 2 and seems to be aimed at ensuring something like $\\epsilon > c$ to hold. Could the authors provide more details on this?\n\n- How exactly does the formulation in the paper differ from the setting introduced by Steinke et al., 2023, specifically in regards to the independence assumptions and the probabilistic dependencies between concerned random variables?\n\nLIMITATIONS:\nThe authors have acknowledged some limitations in the paper. There do not appear to be any negative societal impacts associated with this work. I encourage the authors to address the issues and questions raised in this review.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel privacy auditing procedure called PANORAMIA. The method works with a single model that uses all available training data, with no modifications to the training procedure, and with only access to a subset of the training data. This is achieved by synthetically generating non-member samples for auditing. The method produces an estimate of the $\\epsilon$ differential privacy parameter, which, however, is not necessarily a lower bound.\n\nSummary of the method given a model and its training data:\n1. Train a generative model on a subset of the training data.\n2. Use the remaining training samples as \"members\" for auditing.\n3. Generate an equal amount of \"non-members\".\n4. Split both into training and test sets.\n5. Fit a baseline classifier that distinguishes synthetic and real data on the training set (only input are samples).\n6. Fit a membership inference classifier on the training set (inputs are samples and model statistics).\n7. Use predictions on the test set and hypothesis testing to obtain a confidence interval for quantities of interest.\n\nThe hypothesis is of the form \"The generative model is $c$-close to the true data distribution and the training procedure is $\\epsilon$-DP\", where $c$ is a distance measure defined in this paper. The baseline predictions are used to reject the first claim, and the membership predictions to reject the second one. Hence, the returned $\\epsilon$ is a lower bound on the true $\\epsilon$ only if the generative model is indeed at least $c$-close to the data distribution. The theoretical analysis relies on results from the (1) procedure by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) but adapts them to the relaxed setting in this paper.\n\nFor evaluation, the paper applies the new auditing procedure to various CNNs with and without DP guarantees on image data, small GPT2 models on text, and on tabular data. As baselines, the evaluation uses the (1) method by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) and real instead of generated members. PANORAMIA does not outperform the baselines, but achieves reasonably close results on image and text data.\n\nSTRENGTHS:\n- This paper considers a practical and relevant setting for privacy auditing. In particular, training the model to audit using all training data is a big benefit: otherwise, the audit either uses a slightly different setting that uses fewer training samples, or one has to forfeit a potentially large amount of training data (which hurts utility). Additionally, the procedure works even when the auditor knows only part of the training data, which can be relevant (e.g., federated learning).\n- The experiments are broad (considering both DP and non-DP training, and different degrees of overfitting), sound, and the conclusions are convincing. Although the results are weaker than for existing methods, those existing methods use a less appealing setting. A particularly convincing result is that PANORAMIA with synthetic non-members often comes close to the same method but using real non-member samples.\n- The paper is overall structured well and the auditing procedure is laid-out clearly. The authors manage to explain the auditing results well, despite their complex interpretation. I also appreciated that Algorithm 1 collects all parts of the method into one place, including the calculations to obtain confidence intervals.\n\nWEAKNESSES:\n- The considered privacy semantics are partially misleading and could be made more clear.\n\t- The presented auditing game is incomplete; Definition 2 only describes how samples are selected but should also include the goal and actions of the adversary.\n\t- The paper states (as a benefit) that PANORAMIA audits a target model, not the algorithm (L46--47). However, the results of the procedure are DP parameters, and DP is always a property of an algorithm (not model). If the considered semantics are meant to be different from DP, they should be made more clear and discussed.\n\t- Similarly, the paper claims that PANORAMIA does not require worst-case canaries because it audits the target dataset (via a model). However, this ignores that certain samples in the training data might be more vulnerable; hence, auditing should still consider the worst-case sample *in the training data* (especially since privacy is averaged over the dataset). This might be an explanation why the reported \"lower bounds on $\\epsilon$\" are still far from tight in Figure 6c (as is the prior (1) procedure in a white-box setting).\n- Relatedly, the definition of c-closeness (Definition 3) might not require a generator to capture the tail of the data distribution. However, this tail contains outliers that are often particularly vulnerable to privacy leakage. Hence, ignoring such outliers might significantly underestimate the $\\epsilon$ lower bound. I would have appreciated a more thorough discussion of this limitation (or a clarification).\n- The paper could be more explicit about whether PANORAMIA is intended to be a procedure to be used in practice, or an important stepping stone towards more practical procedures. Right now, PANORAMIA achieves worse results than the existing (1) method in a white-box setting, and requires training of a strong synthetic data generator (which might be non-trivial). Nevertheless, I believe this paper is still relevant from a conceptual perspective as a path for future work.\n- There are minor notation issues in Proposition 1 and 2: the statement mentions only $T$ and $t$, but the inequality uses $T^b$ and $t^b$ (or $T^a$ and $t^a$). Additionally, the left-hand sides both take probabilities over $T^b$/$T^a$ but simultaneously condition on the respective random variable. Fixing those points makes the propositions clearer. Also, the last sentence on L148 seems misplaced.\n\nQUESTIONS:\n1. Might there be a way to use the membership classifier itself as a baseline, e.g., by somehow averaging over possible values for model statistics? This would avoid skewed scenarios where the MIA is better at detecting synthetic data than the baseline detector (even when ignoring membership signal).\n2. In Figure 6/Section 5.3, do all models use (approximately) the same number of training samples? If not, could there be confounding effects (e.g., if models trained on fewer samples leak more privacy)?\n3. Is there a path to extend this paper's method to always yield a lower bound on $\\epsilon$ (w.h.p.)? Or would this require radical changes?\n\nLIMITATIONS:\nThe authors are very transparent about all limitations of their work and discuss them transparently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel way for privacy auditing of machine learning models through MIAs. The framework they propose, panoramia, aims to audit the privacy of a ML model post-hoc (so with no control over the training procedure), and with access to a known member subdataset. The method first consists of training a generative model on the known subset of members, which is used to then generate synthetic data samples from the same distribution. These synthetic points are then used as non-members, which combined with the known member dataset allows to fit and apply an MIA on the target model. Importantly, authors recognize that there might be a distribution shift between members and synthetic non-members, so they fit a classifier without leveraging the target model as a baseline. \n\nNext, they use the difference between the MIA performance (thus using the target model) and this baseline to estimate the privacy loss. Authors provide the formula (and add proof in appendix) to compute a value of epsilon approximating a lower bound on epsilon-DP. \n\nThey further apply the privacy auditing to three kinds of ML models (image classification, language modeling, and tabular data classification). Authors consider models with varying degree of overfitting, DP training, and increasing the amount of member data available to the auditor.\n\nSTRENGTHS:\n- Originality: The paper introduces a way to audit the privacy of ML models post-hoc, without any control of the training data, which is novel. They cite and position themselves correctly to relevant prior work such as Steinke et al. \n- Quality: The proposed method is technically interesting, formally supported and evaluated extensively across data/model modalities. \n- Clarity: NA\n- Significance: The paper proposes a way to compute an approximation for the lower bound on epsilon, to audit ML models post-hoc, which is technically interesting.\n\nWEAKNESSES:\n- Originality: Authors should include an appropriate related work section, touching on other privacy auditing techniques and potentially other (post-hoc) membership inference attacks. \n- Quality: The proposed method strongly depends on the quality of a generator and the baseline MIA, the impact and limitations of which can be further explored (see questions/limitations). \n- Clarity: I find that the paper's clarity can be improved significantly. Especially the results section (tables, text) is quite notation heavy and hard to follow what everything refers to. \n- Significance: While technically interesting, the relevance of a technique to compute a proxy lower bound privacy loss in practice needs more compelling motivation (see questions).\n\nQUESTIONS:\n- I understand how c-closeness allows to estimate the quality of the used generator. However, what I struggle to understand, is what happens when you develop a generator that is perfect (c=0) which just samples randomly from the known member subset. Then, a baseline classifier would not be able to distinguish members from non-members, and nor will an MIA, leading to a privacy estimate that will be far off. In less extreme cases, the generator might be slightly overfitted and indeed generate non-members very similar to members, which might also impact the MIA performance. Am I correct that this could have a significant impact on the validity of the procedure? And if so, how would you address the concern? \n- More generally, Panoramia largely depends on a good generator and a baseline MIA. Can authors elaborate on the associated limitations? For instance, can good generators can be developed across all use-cases (smaller datasets, data modalities etc)? And how should a baseline performance be developed or evaluated to be used as part of the panoramia framework?\n- I understand that the method authors provide does not give a formal lower bound for the privacy loss, but rather a proxy for it. In practice, if indeed a hospital as part of an FL setup would like to assess the privacy leakage incurred by their data, why would they opt to compute an estimate for epsilon using your method? Instead, they could for instance generate non-members in the same way as panoramia and just quantify the MIA performance with an AUC or TPR at low FPR compared to a baseline. In general, authors should further motivate the relevance of a proxy for a lower bound privacy loss in practice. \n- To further motivate post-hoc privacy auditing (without any control of the training data), I wonder if it makes sense to also emphasize the context of generative AI models such as LLMs? These models are increasingly trained on all the data model developers can acquire so  the absence of non-member data is very real in practice. \n- Can authors add a related work section?\n\nLIMITATIONS:\nAuthors are very clear that their method only provides a proxy for the lower bound of epsilon and thus carefully caveat their method. However, can authors elaborate on the limitations associated with the development of both a generator and a baseline, both of which seem fundamental for panoramia. Currently, their implementation seems rather ad-hoc than adequately discussed for wider applications.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a privacy auditing technique that utilizes partial access to member data to generate synthetic non-member data, which in-turn is used to train a meta-classifier that can be used to empirically measure privacy leakage relating to record membership. The authors evaluate their technique on models as large as GPT2 and find correlation between audit scores and the expected leakage from models.\n\nSTRENGTHS:\n- L226-228: Good to include negative results!\n- Having access to non-members is often taken for granted (especially when the auditor is also the model trainer), but in most third-party cases getting good-quality non-member data that is not significantly different from member distribution is hard. The methods proposed in this work can be useful there, with a generator and discriminator that can help ensure any differences in members and non-members (and subsequent MIA performance) do not arise from distributional differences. \n- The paper is well written and structured, and experiments are thorough, spanning quite a lot of models and modalities.\n\nWEAKNESSES:\n- As an auditor, access to validation/test data used in model training would not be a far stretch- why not use the non-member set as used in standard MIA setup (validation/test data). What is the added benefit from this extra step of generating synthetic non-members?\n\n- Regarding the contributions, (1) and (2) has already been explored by [1, 2]. While the method in [1] does not currently support large models/datasets like CIFAR10, it would be nice to highlight differences here. If you indeed have knowledge of a decent-chunk size of members and non-members, I'd imagine you could do something better like [2], and other related methods.\n\n- I have some concerns over the dependency on how well the baseline in/out distribution detection system works. As a concrete example consider Figure 7(a, b) - even as a human I see a very clear difference in resolution of the generated images and find it hard to believe that the distinguisher does not work well here. Even a non-ML technique (that can work around with blurring) would work pretty well here. \n\n- Table 1: Accuracy of models here is not good enough, especially when using entire data, with very clear signs of overfitting. If using the entire data (and not half split, as in most MIA evaluations [3, 4] where even with half the dataset test accuracy is ~92%), should be able to train well-performing models that do not overfit so heavily. Please see [this resource](https://paperswithcode.com/sota/image-classification-on-cifar-10)\n\n- Figure 1: While the trends suggest that the proposed audit correlates with actual leakage, one could also argue that (100 - test accuracy) is also a useful privacy metric in this comparison given the correlation. The utility of the audit would be more apparent, then, when studying models that have **comparable** test performance, but have (by design) different leakage, perhaps focusing on moderate/large values of $\\epsilon$ with Differential Privacy-training.\n\n- L288-290: (1) and (2) are ambiguously enforced- if you have a large portion of train data, you can control process in most cases, and also obtain non-member data. As far as (3) is concerned, that is a compute-related constraint, not a threat model difference. The auditor could, for instance, use available data knowledge to train \"in\" models.\n\n- L331-334: I do not find the justification behind having access to partial member data convincing. While I am okay with just stating that \"this is a possible limitation, but okay to assume for an auditor\", but relating it to how it could be useful in situations like FL is not practical. For instance, here empirical experiments use more than 1/5th of the data; no FL training will have just 5 participants. This is closer to a specific case of distributed learning, or pure data aggregation. Even in such cases, the data distributions per client will be considerably different, as opposed to the experiments here which have uniform samples. \n\n## Minor comments\n\n- Please consider a more descriptive abstract. \"scheme\" here is not very descriptive\n- L121: \"We formalize a privacy game\" - this is standard membership inference evaluation and not a contribution of this work. Please rephrase to make this clear.\n- L493-494: Please provide direct distinguishing accuracies of these baseline classifiers for reference and easier understanding.\n\n### References\n- [1] Suri, Anshuman, Xiao Zhang, and David Evans. \"Do Parameters Reveal More than Loss for Membership Inference?.\" High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning.\n- [2] Nasr, Milad, Reza Shokri, and Amir Houmansadr. \"Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning.\" 2019 IEEE symposium on security and privacy (SP). IEEE, 2019.\n- [3] Carlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n- [4] Zarifzadeh, Sajjad, Philippe Cheng-Jie Marc Liu, and Reza Shokri. \"Low-cost high-power membership inference by boosting relativity.\" (2023).\n\nQUESTIONS:\n- Figure 3a suggests an ordering of 100 > 20 > 50 in terms of leakage for high-recall region, whereas numbers in Table 1 suggest that the trends should be like 100 > 50 > 20 if the proposed method is indeed measuring what it is supposed to. Why is that so?\n\n- L480-481: this means the \"labeler\" is much weaker and might be generating incorrect labels more often? Are there any numbers for what the actual test performance of this labeler is?\n\nLIMITATIONS:\nThere are a few statements that currently serve as justification for limitations (see above) but should just be posed directly as base assumptions. Apart from that (and some potential shortcomings in models used for evaluation), most limitations are already stated in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}