PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining
Authors: Mishaal Kazmi
Hadrien Lautraite
Alireza Akbari
Qiaoyue Tang
Mauricio Soroco
Tao Wang
Sébastien Gambs
Mathias Lécuyer
Venue: NeurIPS 2024 poster (2024)
Keywords: privacy
auditing
machine learning
differential privacy
membership inference attack
Primary area: privacy
TL;DR: PANORAMIA is a practical privacy audit for ML models, that does not retrain the target model or alter its training set or training algorithm.

Abstract: We present PANORAMIA, a privacy leakage measurement framework for machine learning models that relies on membership inference attacks using generated data as non-members. By relying on generated non-member data, PANORAMIA eliminates the common dependency of privacy measurement tools on in-distribution non-member data. As a result, PANORAMIA does not modify the model, training data, or training process, and only requires access to a subset of the training data. We evaluate PANORAMIA on ML models for image and tabular data classification, as well as on large-scale language models.

PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining
Mishaal Kazmi" University of British Columbia
Hadrien Lautraite" University du Québec à Montréal
Alireza Akbari
Simon Fraser University
Qiaoyue Tang" University of British Columbia
Mauricio Soroco
Tao Wang
University of British Columbia
Simon Fraser University
Sébastien Gambs University du Québec Montréal
Mathias Lécuyer University of British Columbia
Abstract
We present PANORAMIA, a privacy leakage measurement framework for machine learning models that relies memhership inference attacks using generated data non-members, By relying on generated non-member data, PANORAMIA eliminates the common dependency of privacy measurement tools on in-distribution non- member data. As a result, PANORAMIA does not modify the model, training data, training process, and only requires access a subset of the training data. We evaluate PANORAMIA on ML models for image and tabular data classification, as well on large-scale language models.
Introduction
Training Machine Learning (ML) models with Differential Privacy (DP) Dwork et al. (2006), such as with DP-SGD Abadi et al. (2016), upper-bounds the worst-case privacy loss incurred by the training data. In contrast, privacy auditing aims to empirically lower-bound the privacy loss of target ML model or algorithm. In practice, privacy audits usually rely on the link between DP and the performance of membership inference attacks (MIA) Wasserman & Zhou (2010); Kairouz et al. (2015); Dong et al. (2019). At a high level, DP implies an upper-bound bound on the performance MIAs, thus creating pigh-peeformance MIA implies lower-bound have proven valuable in many settings, such as to audit DP implementations Nasr et al. (2023), or study the tightness of DP algorithms Nasr et al. (2021); Lu al. (2023); Steinke et al. (2023). Typical privacy audits rely on retraining the model several times, each time guessing the membership of sample Jagielski al. (2020); Carlini et al. (2022a); Zanella- Zanella-Béguelin et al. (2022), which is computationally prohibitive, requires access to the target model (entire) training data as well as control over the training pipeline.
the privacy loss. Auditing schemes
To circumvent these concerns, Steinke et al. (2023) proposed an auditing recipe (called O(1)) requiring only one training run (which could be the same the actual training) by randomly including/excluding several samples (called auditing examples) into the training dataset of the target model. Later, the membership of the auditing examples guessed for privacy audit. However, O(1) faces a few challenges certain setups. First, canaries, which are datapoints specially crafted to be easy to detect when added to the training set Nasr al. (2023); Lu et al. (2023); Steinke et al. (2023), cannot be employed as auditing examples when measuring the privacy leakage for data that a contributor actually puts into the model, and a worst case data point.
"equal contribution
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

This matches a setting in which individual data contributors (e.g., a hospital in cross-site Federated Learning (FL) setting or user of a service that trains ML models on users' data) measure the leakage of their own (i.e., known) partial training data in the final trained model. Second, O(1) also relies on the withdrawal of real data from the model to construct non-member in-distribution data. This problematic in situations in which ML model owners need to conduct post-hoc audits, in which case it is too late for removal Negoescu et al. (2023), Moreover, in-distribution audits require much more data, thus withholding many data points (typically more than the test set size) and reducing model utility. This brings us to the question: Given an instance of a machine learning model as a target, can we perform post-hoc estimation of the privacy loss with regards to a known member subset of the target model training dataset?
Our contributions. We propose PANORAMIA, new scheme for Privacy Auditing with NO Retraining by using Artificial data for Membership Inference Attacks. More precisely, we consider an auditor with access to subset of the training data and introduce new alternative for accessing non-members: using synthetic datapoinnss from generative model trained on the member data, unlocking the limit on non-member data. PANORAMIA uses this generated data, together with known members, to train and evaluate a MIA attack on the target model to audit ($3). We also adapt the theory of privacy audits, and show how PANORAMIA A can estimate the privacy loss (though not a lower-bound) of the target model with regards to the known member subset ($4). An important benefit of PANORAMIA is to perform privacy loss measurements with (1) no retraining the target ML model (i.e., audit the end-model, not the training algorithm). (2) no alteration of the model, dataset, or training procedure, and (3) only partial knowledge of the training set. We evaluate PANORAMIA CIFARIO models and observe that overfitted models, larger models, and models with larger DP parameters have higher measured privacy leakage. We also demonstrate the applicability of our approach on the GPT-2 based model (i.e., WikiText dataset) and CelebA models.
2 Background and Related Work
DP is the established privacy definition in the context of ML models, as well as for data analysis in general. We focus on the pure DP definition to quantify privacy loss with well-understood semantics. In a nutshell, DP is a property of a randomized mechanism (or computation) from datasets to an output space O, noted M D + O. It is defined over neighboring datasets D, D', differing by one element z E X (we use the add/remove neighboring definition), which is D' = D {z). Formally:
Definition 1 (Differential Privacy Dwork et al. (2006)). A mechanism M D + O is e-DP for any two neighbouring datasets D, D' E DD and for any measurable output subset O C O it holds that:
P(M(D) E 0) S e P(M(D') E
Since the neighbouring definition is symmetric, (D' E O] < e P (DD E O). Intuitively, e upper-bounds the worst-case contribution of any individual example to the distribution over outputs of the computation (i.e., the ML. model learned.. More formally, c is an upper-bound on the privacy loss incurred by observing an output o, defined as In [iM which quantifies how much an adversary can learn to distinguish D and D' based on observing output o from M. A smaller hence means higher privacy.
is the DP definition, and also have that
DP, MIA and privacy audits. To audit DP training algorithm M that outputs a model f. one can perform a MIA on datapoint trying to distinguish between a neighboring training sets D and
D' (a)). The MIA can be formalized as a hypothesis test to distinguish between Ho = D and H11 D' using the output of the computation f. Wasserman & Zhou (2010); Kairouz et al. (2015); Dong al. (2019) show that any such test at significance level a (False Positive Rate or FPR) has power (True Positive Rate or TPR) bounded by eaa. In practice, one repeats the process of training model f with and without x the training set, and uses a MIA to guess whether x was included. the MIA has TPR > e'FR, the training procedure that outputs I is not e-D.. This is the building block of most privacy audits Jagielski et al. (2020); Nasr et al. (2021); Zanella- -éggulln et al. (2022); Lu et al. (2023); Nasr et al. (2023). Appendix F provides further context, and discusses other related work that is not directly relevant to our contribution.
Averaging over data instead of models with O(1). The above result bounds the success rate of MIAS when performed over several retrained models, on two alternative datasets D and D'. Steinke et al. (2023) show that it is possible to average over data when several data points independently
2

---

bipracision, menily
PANOGAMIA[praciionn
1
Trained
ML M
Generative Model
Baseline Classifier
PANORAMIA
split
Dg
D/
D
Figure I: PANORAMIA's 's two-phase privacy audit. Phase trains generative model g on member data. Phase 2 trains MIA subset of member data and generated non-member data, using the loss of f these data points. The performance the MIA compared to baseline classifier does not have access to f. Notations are summarized in Table in Appendix A.
differ between D and D'. Let Fi.m be the data points independently included in the training set, and $1,m E {0, 1 be the vector encoding inclusion. To E RT represents any vector of guesses, with positive values for inclusion in the training set (member), negative values for non-member, and zero for abstaining. Then, if the training procedure is c-DP, Proposition 5.1 in Steinke et al. (2023) bounds the performance of guesses from T with:
m
P
Σ [0, Ti S.) 1T Scmmooll
1
[4 St
In other words, an audit (MIA) T that guess membership better than Bernoulli random variable with probability T=er refutes 6-DP claim. In this work build on this result, extending the algorithm ($3) and theoretical analysis ($4) to enable the use of generated data for non-members, which breaks the independence between data points.
PANORAMIA
Figure summarizes the end-to-end PANORAMIA privacy measurement. The measurement starts with a target model f. and a subset of its training data D, from distribution D. For instance, and D could be the distribution and dataset of participant in FL training procedure that outputs final model f. The privacy measurement then proceeds in two phases.
Phase 1: In the first phase, PANORAMIA uses subset of the known training data De Dy train generative model G. The goal of the generative model & is to match the training data distribution D closely possible, which is formalized in Definition 3 ($4). Using the generative model G, can synthesize non-member data, which corresponds to data that was not used in the training of target model Hence, now have access to independent dataset of member data D.. Dj De. and a synthesized dataset non-member data D ~ G, of size m = Dial-
In the second phase, we leverage Din and Domt audit the privacy leakage of f using MIA. To this end, we split Din Don into training and testing sets, respectively called D Du and DE, Di We use the training set to train a MIA (called PANORAMIA in Figure 1), binary classifier that predicts whether a given datapoint is member of D,, the training set of the target model f. This MIA classifier makes its prediction based both training example from applying the target model f to the input, such as the loss the target model when applied to this example (fss(f(z) (see $5, Appendix C for details). We use the test set measure the MIA performance, using the precision at different recall values.
Phase 2:
well as information
Previous results linking the performance of MIA on several data-point to c-DP bounds rely on independence between members and non-members. This intuitively means that there is no information about membership in x itself. When the auditor controls the training process this independence is enforced by construction, by adding data points to the training set based on an independent coin flip. In PANORAMIA. we do not have independence between membership and z, all non-members

---

Algorithm PANORAMIA
Input: Target ML model f, audit set size m, confidence B
Phase 1:
1: Split Dy in Da. D D with D = m
2: Train generator on De
3: Generate D D of size [D [
out
Phase 2:
Train the baseline and MIA:
1: Label D members, and D non-members
2: Train to predict labels using x E D D
our
Train MIA to predict labels using x D U D and f()
Measure privacy leakage (see $4):
Sample Bernoulli(} )M
Def.2
2: Create audit set X
D + D 3: Score each audit point for membership, creating 1 - ((X) E R and MIA(X) E R
4: Set v (c, _ sup {u= 33 (m, }
b Prop.
5: maxi, f1 2 1) S 1
6: Set oh ec) - sup {u: (m, 6
Prop.2
7: {c+ee/n = m f1 2 1 8 u (c. 1 2 2
Return (c+chı
come from the generator * D. As result, there two ways guess membership and have high MIA precision: either by using f to detect membership (i.e., symptomatic of privacy leakage) or by detecting generated data (i.e., not symptom of privacy leakage). To measure the privacy leakage, we compare the results of the MIA to that of a baseline classifier b that guesses membership based exclusively on z, without access to f. The stronger this baseline, the better the removal of the effect of synthesized data detection. Algorithm summarizes the entire procedure. In the next section, we demonstrate how to relate the difference between the baseline b and the MIA performance to the privacy loss
Quantifying Privacy Leakage with PANORAMIA
We formalize privacy game as follows: construct sequence of auditing samples, in which s; is sampled independently to choose either the real (In e x training points from D if 8; 1) or generated (z:80 E nm generated points from G if 81 0) data point at each index i.
Definition (Privacy game).
~ Bernoullit,,) 1
with 8; E {0,1),, = (1 s;)o.. + saa Vi E m.....m...
The key difference between Definition and the privacy game of Steinke et (2023) lies how we create the audit set. Instead of sampling non-members independently and removing them from the training set (which requires modifyig the training set and the model), Definition 2 starts from a set of known members (after the fact), and pairs each point with a non- non-member (generated i..d. from the generator distribution). For each pair, we then flip a coin to decide which point in the pair will be shown to the auditor for testing, thereby creating the test task of our privacy measurement.
The level of achievable success in this game quantifies the privacy leakage of the target model More precisely, we follow an analysis inspired by that of Steinke et al. (2023), but require several key technical changes to support auditing with no retraining using generated non-member data.
4.1 Formalizing the Privacy Measurements as Hypothesis Test
We first need a notion of quality for our generator:
Definition 3 (c-closeness). For all > 0, we suy that generative model G is c-close for data distribution if:
Vo E X, e PP[[ S PP(x..

---

The smaller c, the better the generator, as 9 assigns probability to real data that cannot be much smaller than that of the real data distribution. We make three important remarks.
Remark I: c-closeness is strong requirement but it is achievable, at least for large For instance, a uniform random generator is c-close to any distribution D, with e < op. Of course e would be very large, and the challenge in PANORAMIA is to create a generator that empirically has small enough c. Remark 2: We show in Appendix E that we can relax this definition to hold only with high probability (following the (e, 6D-DP relaxation). Empirical results under this relaxation are very similar. We also show how to measure (e, 6DP instead of e-DP on which we focus here.
Remark 3: Our definition of closeness is very similar that of DP. This is not coincidence as we will use this definition to able to reject claims of both c-closeness for the generator and c-DP for the algorithm that gave the target model. We further note that, contrary to the DP definition, c-closeness is one-sided as we only bound Pg from below. Intuitively, this means that the generator has produce high-quality samples (i.e., samples likely under D) with high enough probability but it does not require that all samples are good though. This one-sided measure of closeness is enabled by focus detecting members (i.e., true positives) opposed members and non-members, and puts less stringent requirements on the generator.
Using Definition 3, we can formulate the hypothesis test on the privacy game of Definition 2 that underpins our approach:
H generator G is c-close, and target model f is e-DP.
Note that, in small abuse of language, we will often refer to as D-DP to say that is the output of an c-DP training mechanism. That is, B(s, x) (b(=1), )(.m))
We define two key mechanisms: x)s {0, 1)mh x x + I outputs (potentially randomized) non-negative score for the membership of each datapoint æ4, based on æ; only; A(s, x, f) {0, 1) x x x F + I outputs a (potentially randomized) non-negative score for the membership of each datapoint, with the guess for index depending on zgi and target model f. We can construct statistical test for each part of the hypothesis separately.
Proposition 1. Let G be c-close, S and X be the random variables for and x from Definition 2, and T & X) be the vector of guesses from the baseline. Then, for all v E R and all in the support T:
P PS
Σ
T S.2 T = =00] <
Σ
S
A (m,,,,
S B'rmoulll Iwme
Proof. In Appendix B.I.
Now that we have test to reject claim that the generator G is c-close for the data distribution D, we turn our attention to the second part of which claims that the target model is the result of an cDP mechanism.
Proposition 2. Let 9 be c-close, S and X be the random variables for and x from Definition 2, be c-DP, and T & A(S, X, f) be the guesses from the membership audit. Then, for all R and all in the support T:
m
m
PS.
Σ T S. 2 2
<
P
St
8 ~ Bernouli
c
i=
Proof. In Appendix B.2.
We can now provide a test for hypothesis H. by applying union bound over Propositions 1 and 2: Corollary 1. Let H true, To A B(S, X), and T & A(S,x,f). Then:
m
m
T Σ T S v² T = 1a T = r s (m e,,
T
S 2 0
1=1
1=1
5

---

To make things more concrete, let us instantiate Corollary 1 as we do in PANORAMIA, Our baseline (B above) and MIA (A above) classifiers return a membership guesses T {0, 1)"" with 1 corresponding to membership. Let us call yrb & E the total number of predictions, and tpu.6 Eit;" a.bb 84 the number of correct membership guesses (true positives). We also call the ip"th precision prec A pa.5- Using the following tail bound on the sum of Bernoulli random variables for simplicity and clarity (we use tighter bound practice, but this one is easier to read),
S
log (1/3)
P
S B
S S*~"ermouli((p)
2r
we can reject H at confidence level 3 by setting 3 = 30 = 2 -
and if either prec 2
o² Ite
+V 2r
log(2/9)
or prec 2
1og(2/3)
Itee+e
2rn
4.2 Quantifying Privacy Leakage and Interpretation
In an privacy measurement, we would like to quantify e, not just reject a given claim. We use the hypothesis test from Corollary to compute a confidence interval on c and e. To do this, we first define an ordering between (c, e) pairs, such that if (cI,e1) S e2), the event (i.e., set of observations for T .) for which we can reject H(c,, 62) is included in the event for which we can reject H(1), 61). That is, if we reject H for values (e2, 62) based on audit observations, we also reject values (c1, 61) based on the same observations.
We define the following order to fit this assumption, based on the hypothesis test from Corollary 1:
[ei C22 or [ci = 02 and E S
(c1, e1) S (c2, 62) if either
This lexicographic order over hypotheses ensures that, when using the joint hypothesis test from Corollary 1 to construct confidence intervals, we always reject hypotheses with a low value of c incompatible with the baseline performance. Formally, this yields the following confidence intervals:
Corollary 2. For all 3 =|0, 11, m, and observed t th, tt, call (ilce,e) - sup {u : 8 (m, C, v t²)) S ] and u/ (c) 2 sup {v: B e, (2) S G. Then:
P[(c, 2 sup ((d,e)= : 1" v'old'e))) .') and 1' is s v 2 21-3 3
Proof. Apply Lemma 4.7 from Steinke et al. (2023) with the ordering from Equation 2 and the test from Corollary 1.
The lower confidence interval for (c,e) at confidence 1 - B is the largest (c, e) pair that cannot be rejected using Corollary with false rejection probability at most (3. Hence, for a given confidence level 1 - B, PANORAMIA computes (cD, E), the largest value for (c, <) that it cannot reject. Note that Corollaries 1 and 2 rely on a union bound between two tests, one for c and one for c e. We can thus consider each test separately (at level 3/2). In practice, achieve tighter bounds: each level of recall (i.e., threshold on g0.6 to predict membership) corresponds to a bound on the precision, which we can compare to the empirical value. Separately for each test, we pick the level of recall yielding the highest bower-bound (Lines 4-7 in the last section of Algorithm I). For all bounds to hold together, we apply a union bound over all tests at each level of recall. Specially, if we have test datapoints, we have at most m different values of recall, and the union bound corresponds to 3 + 3/m (notice in Equation (1) that the bounds depend /log(1/3), so this is not too costly). We next discuss the semantics of returned measurement values (Cbb, 7)-
use the whole precision/recall curve to
Measurement semantics. Corollary 2 gives lower-bound for (c, c), based on the ordering from Equation (2). To understand the value returned by PANORAMIA, we need to understand what the hypothesis test rejects. Rejecting H means either rejecting the claim about c, or the claim about e + e (which is the reason for the ordering in Equation 2). With Corollary 2, we hence get both a lower-bound Cn on c, and {c+E)m on c+c e. Unfortunately, ē A max (0, {c+e)m - <b., which is the value PANORAMIA returns, does not imply lower-bound on Instead, we can claim that

---

MI Model
Dataset
Training Epoch
Test Accuracy
Model Variants Names
ResNet10T
CIFARIO
20, 50, 100
91.61%, 18.18% 87.93%
ResNer101 E20, ReaNeri01 E50, ResNeriO E100,
WideResNet-28--
CIFARIO
150, 300
95.67% 94.23%
WRN-28-10 E150, WRN-28-10 F300
ViT-small (prctrained) Multi- Label CNN
CIFARIO
35
96.38%
ViT_ B35
CelchA
50, 100
81.77% 18%12%
CNN ES0, CNN F100
GPT-2
WikiTest
37, 150
Refer Appendix C.2
GPT-2 ES7, GPT.2 275, GPT-2 E150
Table I: Train and Test Metries for ML Models Audited. *"Model Variants" trained for different numbers of epochs E. For Vmalsmall, we use a model pre-trained on imagenet and tune it on CIFARIO.
"PANORAMIA could not reject a claim of c-closeness for G, and if this claim is tight, then f cannot be the output of mechanism that is ě-DP".
While this is not as strong claim as typical lower-bounds on e-DP from prior privacy auditing works, we believe that this measure is useful in practice. Indeed, the measured by PANORAMIA is quantitative privacy measurement, that is accurate (close to a lower-bound on c-DP) when the baseline performs well (and hence Cb is tight). Thus, when the baseline is good, can be interpreted as (close to) lower bound on (pure) DP. PANORAMIA opens a new capability, measuring privacy leakage of trained model / without access or control of the training pipeline, with a meaningful and practically useful measurement, as we empirically show next.
5 Evaluation
We instantiate PANORAMIA on target models for four tasks from three data modalities'. For image classification, we consider the CIFARIO Krizhevsky (2009), and CelebA Liu et al. (2015) datasets, with varied target models: a four-layers CNN O'Shea & Nash (2015), a ResNet101 He al. (2015), a Wide ResNet-288 28-1--8-10 Zagoruyko & Komodakis (2017b), a Vision Transformer -ViT)-small Dosovitskiy et al. (2021), and a DP ResNet18 He et al. (2015) trained with DP-SGD (Abadi et al., 2016) using Opacus (Yousefpour et al., 2021) at different values of e. We use StyleGAN2 Karras al. (2020) for G. For language models, we fine-tune small GPT-2 Radford al. (2019) on the WikiText train dataset Merity et al. (2016) (we take subset of the documents in WikiText- again based on small GPT-2, and then fine-tuned on Dg. We generate samples using top-k sampling Holtzman et al. (2019) and held-out prompt dataset Drrompt Dg. Finally, we conduct experiments classification on tabular data. However for this data modality. PANORAMIA did not detect any meaningful leakage. More precisely, we have observed significant variance in the audit values returned by PANORAMIA which make the results inconclusive. Nonetheless, we present the results obtained for this task and discuss this issue further in Appendix D.6. Table summarizes the tasks, models, and performance. More details on the settings, the models used, and implementation details are provided in Appendix C.
5.1 Baseline Design and Evaluation
Our MIA is loss-based attack, which uses an ML model taking as input both datapoint and the value of the loss of target model f point Ap- pendix C details the architectures used for the attack model for each data modality.
Recall from $4.2 the importance of having a tight db for our measure to be close to a lower-bound
e-DP. To increase the per-
CIEAR Busclinest
Buseline model
Cu
2.1410100
formance of our baseline b, we mimic the
CIFAR 10 Baseline
2:211 0.17
CIFAR Baseline
12251.02
role of the target model f's loss in the MIA using a helper model trained on synthetic data, which adds loss-based feature to b.
Biselingy -pa
2021
ColebA Daseliney, -ist
CelebA Daseline
1..27.10.25
Whillest Basalingg
3.31 0.15
WikiTed Baselineus
3.26
Sase Inc. CIRALO
This new feature can be viewed as side in- formation about the data distribution. Ta- ble 2 shows the db value for different de- signs for h. The best performance is consis- tently when trained on synthetic data before being used as a feature to train the
WikiTest Baselint
3.11
500 4//
1390 15300 zole 15900 zelce
Table 2: Baseline evaluation with different helper model see- narios.
Figure 2: CIFAR-10 basc- line increasing training size.
"Code available here: httpp://ggite com/ube.ommm//e http:///ooooooooooooo/oo///oooaa www.aaaaaaaaaaaaaaaorreeeeee
7

---

Indeed, such design reaches up to 1.19 larger that without any helper (CIFARIO) and 0.23 higher than when training on real non-member data, without requiring access to real non-member data, key requirement in PANORAMIA. We adopt this design in all the following experiments. In Figure we show that the baseline has enough training data (vertical dashed line) to reach its best performance. Appendix D.I, shows similar results on GPT-2 as well as different architectures that we tried for the helper model. All these pieces of evidence confirm the strength of our baseline.
5.2 Main Privacy Measurement Results
We run PANORAMIA on models with different degrees of over-fitting by varying the number of epochs, (see the final accuracy on Table for each data modality. More over-fitted models are known to leak more information about their training data due to memorization (Yeom et al., 2018; Carlini et al., 2019, 2022a). To show the privacy loss measurement power of PANORAMIA, we compare with two strong approaches to lower- bouer-bnnndiggggg privacy loss. First, we compare PANORAMIA with the O(1)
Precisien
0.8
Baseline CIFRRDD
007
0.7
Basdin
PANDRAMIA: ResNett01 E100
IANORAMA GHT22 E150
PANORANIA: ResNet101
PANDRAMIN: MublLabe CNN,C100 C100
0.5
INNOSAMI GHT12
Theoretical E20
INOOMMIA: ResNet101,
PANORAMIA: Mutilabel CNN, ES0,
PANOSAMA GeT12_537
Thecret cal maxium prec sion
0.5
Thcorctical masimum preds
Recall
Recal
0.E
Recall
(a) R01, CIFARIO
(b) CNN, CelebA
(c) GPT-2, WikiText
Figure 3: Precision vs. recall comparison between PANORAMIA and the baseline b for our target models, based on one experiment run. The maximum or {e+e) eIn values set upper bound on the empirical precision values across different recall levels, indicated by the dashed line.
3.0
2.0
1.5
Nuttiabel CWN CNRL_100
Ic+cle
[c) dh
Emptrical mostmum value
Emplnical maximum value
(c sio
22
04
24
06
02 64
06
0.
10
Rerali
(a) ResNet101, CIFAR10
(b) CNN, CelehA
(c) GPT-2, WikiText
Figure 4: +c+ + chD (or (p) vs recall, for our target models, reported over independent experiment runs.
audit (Steinke et al., 2023) (in input space, without canaries in black box setting), which uses a loss threshold MIA to detect both members and non non-members. Second, we use hybrid between O(I) and PANORAMIA, in which
real non-member data instead of generated data with our ML-based MIA (called RM:RN for Real Members; Real Non-meebers). In both cases Ob = 0, but it requires control over training data inclusion and large set of non-member data. The privacy loss measured by these techniques gives a target that we hope PANORAMIA to detect.
Figure 3 shows the precision of b and PANORAMIA at different levels of recall, and Figure 4 the corresponding value of (c++)m (or em for b). Dashed lines show the maximum value of (c+ e )mn (or <ь) achieved (Fig. 4) (returned by PANORAMIA), and the precision implying these values at different recalls (Fig. 3). Table summarizes those {e+ c)mb (or Qp) values, as well as the measured by existing approaches. We make two key observations.

---

First. the best prior method (whether RM:RN or O(1)) measures a larger pri- vacy loss (2 I <). Overall, these results empirically confirm the strength of as we do not seem to spuriously assign dif- ferences between G and 'D to our privacy loss proxy É. We also note that O(1) tends to perform better, due to its ability to rely non-member detection, which im- proves the power of the statistical test at equal data sizes. Such tests are not avail- able in PANORAMIA given our one-sided closeness definition for G (see $4), and we keep that same one-sided design for RM:RN for the sake of comparison.
Targei medd
Andi
PANDRANTA BMSNN
010
0-221 11.19
5:421 2 1L10
D.D HMSEN
11 4945
PARM RARTA
010 2011
Heche1191_500
13 HMSKNN
1279 2022
PIR PIRTAA
5:0 021
10:
Reheetot F100
PANIAAFIA RMRN
0.00
1.20 0.190
1.224-0119
PARRTAORTA
501
WRN- 10 5150
FANTRAFTA RMRN
0.00
2.57 0.70
3.574 0-00
D1 SM
0.15
PARCRARIA
2022
PANDRAYIA
300
D11 BMEEN
PARCRACJA
508
1124
2002
VITJESS
PANDRAMIA
0.00
#:021
PARCRARTIA
231
0.70
CNNLES0
PANIRAMIA RMRN
0.00
0.51
0.52
0.01 2:022
CNN_E800
PANDRAMIA BMMMNN
0100
112
(1) RMRN
L.4 0.16
PANCRORTA
3822
3.5.2 848
0222 103
GPPT_EB
1313 MMSKNN
282 0.2
GPT2_ESS
PANDRAMIA BMBRN
000
2.12 0.36
Second, the values of é measured by PANORAMIA are close to those of the methods against which we compared. In particular, despite more restrictive ad- versary model (i.e., no non- member data, no control over the training process, and no shadow model training), PANORAMIA is able to detect meaningful amounts of privacy loss, comparable to that of state- of-the-art methods.
7.0
CPPT RIMI
FANTRAFTZ EMRN
0.00
340.4-00 0.41
3.0-4-044 0-11
DED RMEEN
1,471 à
Kreenity 15
D RMENN
1.18
KoNall3
PARRNARIA
244
03/2022
0.23 2.144
ReNNi8
PANCKARIA
24:00000
1,00
100
0,30
0:1RMMY
0.01
XaNNet8
PANGRORIA
20
6.30
Table 3: Privacy measurements on different target models (m 10k). O(1) RM:RN with ML-based attack for c-DP models shown in Figure 6(c).
For instance, on a non-overfitted CIFAR-
10 Re model (E20). PANORAMIA detects a privacy loss of 0.33, while using real non-member (RM:RN) data yields 0.42, and controlling the training process (11) gets 0.45. The relative gap gets closer on models that reveal more about their training data. Indeed, for the most over-fitted model (E100), ē = 1.16 is very close to RM.RN (e = 1.22) and O(1) (e = 1.41). This also confirms that the leakage detected by PANORAMIA on increasingly overfitted models does augment, which is confirmed by prior state-of-the-aa methods. For instance, NLP models' ? goes from 0.22 to 2.40 (2.82 to 5.73 for O(1)). Appendix D.6 details results on tabular data, in which PANORAMIA is not able to detect significant privacy loss.
Privacy Measurement of e-DP Models:
Another avenue to varying privacy leakage is by training Diffcrentially-Privaee (DP) models with different values of We evaluate PANDRAMIA on DP ResNet-18 models on CI- FAR10, with values shown on Table 3 and Figure 5. The hyper-parameter were tuned independently for the highest train accuracy (see Appendix D.9 for more results and discus- sion). As the results show, neither PANORAMIA nor O(1) detect privacy loss the most private models (e = 1, 2). At higher values of e = 10, 15 (i.e., less private models) and e = oo (i.e.,
4.0
3.5
20
Go
ResNet18 c x
Resliet28 c c515
ResNet18
non-private model) PANORAMIA does detect an increasing level of privacy leakage with ē==10 < ε < ēc=oc. In this regime,
ResNet18:
empirical max mum value
0.8
the O(1) approach detects larger, though comparable, amount of privacy loss.
Recall
Figure 5: ResNet18, CIFAR-10, cDDP for various values.
We also evaluate PANORAMIA on DP (fine-tuned) large language models (see Appendix D.9 and Table 9 for details). We fine-tune GPT2-Small target models with DP-SGD on the WikiTeet dataset, with various values of c. Neither PANORAMIA nor O(1) are able to measure a positive privacy loss.

---

5.3 Leveraging More Data to Improve Privacy Measurements
PANORAMIA can leverage much more data for its measurement (up to the whole training set size for training and testing the MIA), while in our setting O(1) is limited by the size of the test set (for non-members). As we have seen in Figure 6(b), PANORAMIA can leverage larger test set size to measure higher privacy loss values than O(1), up to 2.62 on the ResNet101 trained for 100 epochs, versus 2.23 for O(1). It is also important to note that at small amounts of data O(1) measures larger privacy loss (ē I c). These findings provide additional empirical evidence for the robustness of indicating that we are not erroneously attributing differences between G and 'D to our privacy loss proxy é. However, in the case of text data (on the GPT-2 trained for 150 epochs), Figure 6(d), we hit an upper-bound on the power of the hypothesis test which estimates {c++e) IT using 20k test samples. The maximum measurement for é we can achieve is around 2.59 with a 20k test set, as opposed to O(1), which reaches 5.73 with 10k test set size.
Ta
140
144
564
100g
4009
1900
2006
30.00
Throndt Fpolir
380
500
18000
2500
(a) WideResNet-28-10, (b) 10sNet-101, CIFAR1O (c) c-DP ResNet18, CI- (d) GPT-2, WikiText CIFAR10
FAR10
Figure 6: Effect of increasing test set size PANORAMIA'S ass privacy measurement for our target models. In the case of the image dataset, increasing the number of auditing examples allows us to achieve tighter empirical measurements for privacy leakage despite a restricted adversary. For the case of the language modeling task, when increasing test set size m, we hit an upper bound on the power of the hypothesis test due to test dataset size, and do not see significant improvement in privacy measurement in this case.
Increasing Test Set Size for DP models: With an increase in the test set size m, we demonstrate that we can achieve tighter bounds on the privacy measurement Figure 6(c) shows that as m increases, PANORAMIA (solid lines) can measure higher privacy leakage for various e-DP ResNet18 8 models. The reason for this is that PANORAMIA can leverage a much higher test set size (up to the whole training set size for training and testing the MIA), in comparison to O(1) in our setting (due to being limited by the number of real non- non-member data points available).
Impact, Limitations and Future Directions
We have introduced a new approach to quantify the privacy leakage from ML models, in settings in which individual data contributors (such as hospital a cross-site FL setting or a user of text auto-complete service) measure the leakage of their own, known partial training data in the final trained model. Our approach does not introduce new attack capabilities (it would likely benefit from future progress on MIA performance though), but can help model providers and data owners assess the privacy leakage incurred by their data, due to inclusion in ML models training datasets. Consequently, we believe that any impact from our proposed work is likely to be positive.
However, PANORAMIA suffers from major limitation: its privacy measurement is not lower-bound on the privacy loss (see details in Section 4.2). Providing a proper lower-bound ét on privacy leakage in this setting is an important avenue for future work. A promising direction to devise way to measure or enforce an upper-bound on c, thereby yielding Et via {c + eh Cub. Despite this shortcoming, we believe that the new measurement setting we introduce in this work is important. Indeed, recent work has shown that MIA evaluation on foundation models suffers from a similar lack of non non-member data (since all in distribution data is included in the model) Das et al. (2024); Duan et al. (2024); Meeus et al. (2024). The theory we develop in this paper provides a meaningful step towards addressing privacy measurements in this setting, and provides a more rigorous approach to privacy benchmarks for such models. We also demonstrate that PANORAMIA'S privacy measurements can also be empirically valuable for instance for providing improved measurements with more data (Figure 6).
10

---

Acknowledgments
Mathias Lécuyer grateful for the support of the Natural Sciences and Engineering Research Council of Canada (NSERC) [reference number RGPIN-2022-04699, as well as Google Research Scholar award. Sébastien Gambs is supported by the Canada Research Chair program, a Discovery Grant from the Natural Sciences and Engineering Research Council of Canada (NSERC) as well as the DEEL Project CRDPJ 537462- 18 funded by the NSERC and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ). This research was enabled by computational support provided by the Digital Research Alliance of Canada (alliancecan.ca) Hadrien Lautraite is supported by National Bank of Canada and a bursary from the Canada Research Chair in Privacy-preserving and Ethical Analysis of Big Data..
References
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L.. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Andrew, G., Kairouz, P., Oh, S., Oprea, A., McMahan, H. B., and Suriyakumar, V. One-shot empirical privacy estimation for federated learning, 2023.
Carlini, N., Liu, C., Brlingsson, U., Kos, J., and Song, D. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), 2021.
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 2022 IEEE Symposium on Security and Privacy (SP) (SP), pp. 1519- 1519, Los Alamitos, CA, USA, may 2022a. IEEE Computer Society. doi: (0.1109/SP42214.2000 00090. URL sttps:///oi: www.coottttesssccccccc.... org/ 10. 1199//942214 2022. 00090.
Carlini, N., Chien, S., Nasr. M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022b.
Das, D., Zhang, J.. and Tramèr, F. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406.1622 16201, 2024.
Dong, J., Roth, A., and Su, W. J. Gaussian differential privacy. arXiv preprint 1905.:1905.022: 2019.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani,
M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https://arxis: org/ abs/ 2010. 11929.
Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L.,, Tsvetkov, Y., Choi, Y., Evans, D., and Hajishirzi, H. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.0784) 2024.
Dwork, C., MeSherrr, F., Nissim, K., and Smith, A. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference. Springer, 2006.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition, 2015.
Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. arXiv preprint arXiv: 9119900407,,1 2019.
Jagielski, M., Ullman, J., and Oprea, Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:22205-22211 2020.
11

---

Jayaraman, B. and Evans, D. E. Evaluating differentially private machine learning in practice. In USENIX Security Symposium, 2019.
Kairouz, P.. Oh, S., and Viswanath, P. The composition theorem for differential privacy. In International conference machine learning. PMLR, 2015.
Karras, T., Aittala, M., Hellsten, J. Laine, S., Lehtinen, J., and Aila, T. Training generative adversarial networks with limited data, 2020.
Krizhevsky, A. Learning multiple layers of features from tiny images. 2009. URL https://pp. semanticscie crwaantisccholar...m org/ CorpusID: 18268744.
Li, X., Tramer, F., Liang, P., and Hashimoto, T. Large language models can be strong differentially private learners. arXiv preprint arXiv:211000567, 2021.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Lu, F., Munoz, J., Fuchs, M., LeBlond, T., Zaresky-Williams E., Raff, E., Ferraro, F., and Testa, B.
A general framework for auditing differentially private machine learning, 2023.
Maddock, S., Sablayrolles, A., and Stock, P. Canife: Crafting canaries for empirical privacy measurement in federated learning. 2023.
McKenna, R., Miklau, G., and Sheldon, D. Winning the nist contest: A scalable and general approach to differentially private synthetic data. arXiv preprint arXiv:2 100.2008.000 108.09788 2021.
Meeus, M., Jain, S., Rei, M., and de Montjoye, Y.-.. Inherent challenges of post-hoc membership inference for large language models. arXiv preprint arXiv:2406. /7975, 2024.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint 1 07843, 2016.
Nasr, M., Shokri, R., and Houmansadr, A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning. In 2019 IEEE symposium on security and privacy (SP), pp. 739-753. IEEE, 2019.
Nasr, M., Songi. S., Thakurta, A., Papernot, N., and Carlin, N. Adversary instantiation: Lower bounds for differentially private machine learning. In 2021 IEEE Symposium on Security and Privacy (SP), pp. 866-882, 2021. doi: 10. 109/SP400011 202200000..22..... 2026.00069.
Nasr, M., Hayes, J., Steinke, T., Balle, B., Tramèr, F., Jagielski, M., Carlini, N., and Terzis, A. Tight auditing of differentially private machine learning, 2023.
Negoescu, M., Gonzalez, H., Orjany, S. E. A., Yang, J., Lut, Y., Tandra, R., Zhang, X., Zheng, X., Douglas, Z.. Nolkha, V., et al. Epsilon": Privacy metric for machine learning models. arXiv 2023.
preprint 2
O' O'Shea, K. and Nash, R. introduction to convolutional neural networks, 2015.
Radford, A., WU, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Steinke, T., NaST, M., and Jagielski, M. Privacy auditing with one (1) training run. In Thirty- Conference on Neural Information Processing Systems, 2023.
Suri, A., Zhang, X., and Evans, D. Do parameters reveal more than loss for membership inference?
In Higg--imensional Learning Dynamics 2024: The Emergence of Structure and Reasoning.
Wasserman, L. and Zhou, S. statistical framework for differential privacy. Journal of the American Statistical Association, 2010.
Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEFE 31st computer security foundations symposium (CSF), pp- 268-282. IEEE, 2018.
12

---

Yousefpour, A., Shilov. I., Sablayrolles, A., Testuggine, D., Prasad, K., Malek, M., Nguyen, J., Ghosh, S., Bharadwaj, A., Zhao, J., Cormode. G., and Mironov, Opacus: User-friendly differential privacy library in PyTorch. arXiv preprint arXiv: /2298, 2021.
Zagoruyko, S. and Komodakis, N. Wide residual networks, 2017a.
Zagoruyko, S. and Komodakis, N. Wide residual networks, 2017b. URL Utt.https//arrsv..vo abs/1605.07144 07146.
Zanella-Béguelin, S., Wutschitz, I.., Tople, S., Salem, A., Rühle, V., Paverd, A., Naseri, M., Köpf, B., and Jones, D. Bayesian estimation of differential privacy, 2022.
Zarifzadeh, S., Liu, P. C.-J M.. and Shokri, R. Low-cost high-power membership inference by boosting relativity. 2023.
13

---

A
Notations
Table 4 summarizes the main notations used in the paper.
Notation Description
f
the target model to he audited.
the generative model
D
distribution over auditor samples (subset of the) training set of the target model f from training set of the generative model 9, with De C D3 member auditing set, with Din C D) and Dm Da non member auditing set, with D ~ g
Dj
Da
Din
Dout
D
D(trde) (tr,te)
training and testing splits of Din and Dout
(in,
Dia D -
b
baseline classifier for Di vs. Dout
Table Summary notations
B Proofs
For both Proposition and Proposition 2, state the proposition again for convenience before proving it.
B.1 Proof of Proposition
Proposition 1. Let G be c-close, and T B(S, X) be the guess from the baseline. Then, for all v E lR and all in the support T::
P P.X.TT
Σ
T S.2 T =t 5
2 5 (m,
S: S'mBerolullii -= mm
Proof. Notice that under our baseline model B(s, (((2)))((22)))) ((2)) b(Fm)) and given that the X; are i.idd., we have that: Szi 1 The Xei, since T B)S X);'s distribution is entirely determined by X;; and Sg, T Th, X since the X; are sampled independently from the past.
We study the distribution of 5 given fixed prediction vector th, one element i E [m]] at time:
=[ 1 th Se = scn Xgi = xzd] = =P[S, = ||
Sci = 8gn Xzi = xgg]]
P(S: = 1 Se = sei Xei = Teil
=P[X, S; = 1, 1,5 8cb X =Ici
P P 82i = Xes = Izl
[X[ 5 = 1, 828 Xci = xei
P[S: =1
X S2i = 321, Xxi = Izi
X;
S== =
P
PP
S
+ P X;
S
5;:00 2
1
1
e
X.
5.=0
Pg X,
-te
lte
14
1 1+
P
Xx
9:=1
Po
The first equality uses the independence remarks at the beginning of the proof, the second relies Bayes' rule, while the third and fourth that S, sampled i.i.d from Bernoulli with probability half, and X, i..d. conditioned on S. The last inequality uses Definition 3 for c-closeness.
14

---

Using this result and the law of total probability to introduce conditioning on Xgi. we get that:
P(S. = 1 1 = Sci =sci]
Σ P(S, = -1", Ses = 82b xx = Px
2z1 15 =2 12, = 828
e
p Xgi zgi T SKi Sci]
1te
agi
and hence that:
e
P P(S. = 1 T 12, =ci
sci| 5 1+e1
(3)
We can now proceed by induction: assume inductively that Wm-1 2 Σ m- T S; is stochastically dominated (see Definition 4.8 in Steinke al. (2023)) by W Di=1 To Sy, in which S'
makes it true for m m Then, conditioned on W m Bernoulli( <" jm-1 Setting Wi = W1
Ite
( and using Eq. 3, Thh S. Tm 1 1|5m 1} is stochastically dominated by The Bernoulli( The Applying Lemma 4.9 from Steinke et al. (2023) shows that W.. is stochastically dominated by W m which proves the induction and implies the proposition's 's statement.
B.2 Proof of Proposition
Proposition 2. Let G be c-close, I be c-DP, and T & A(S, X, f) be the guess from the membership audit Then, for all E IR and all t in the support T::
T:
1
P S
T T ====00 =00
P
Σ
S
A B (,,,,,
S'-emouull(
Proof.
Fix some 1a E R We study the distribution of S one element E [m] at a time:
S|S: = 1 7 = tr Se Sci Xgi = rgl]
P = 1| |2i = 82s Xgi = Zgi
P[T ta |S 1,5 szh Xgi = zzi
P P = to |Ses = sei Xgi = Fgi
$
8,=0 Sei m
1+ 1
Si=1 Ser
c+
-tee 1 eatc
The first equality uses Bayes' rule. The first incquality uses the decomposition:
P P|r- = to S2i = 8zi, Xei = x55]]
= - = S = 1,5 = 826 Xzi = rzs]
P[S. = 1|5 = 8ch, Nch = =zg]
+ P 1 S; = 0, Sei ses, X5: =xgi]
PP(S: = 0 Sci Sei,Xe agi),
and the fact that A(8,x, f) c-P w.r.t. 8 and hence that:
P T t S 0. Sct X Fzi
2 c.
P T
tm = 1. = Sci, Xzi Eςi
This inequality might seem surprising at first, know from the privacy game (Definition 2) that S is independent of f. that is S -1 f. However, these two quantities are not conditionally independent when conditioning on X (where is the test set for the MIA/baseline), (i.e., Á. flX). This because X; is either member or non-member based on S, So when S; 1, then f was trained on X,, which makes f and S, dependent if membership can be detected through f. f is eDD,, we can bound this dependency using DP properties, which is what do in Equation (4).
15

---

The second inequality uses that:
P P[S = Sc; = 8c6, Ngi = Izs
P P[S, 1 Sc S
Xzi = Izi
F[X, = z; S = 0, S2 = sci. Xii = Tzl
P[S, = 0 S2i = 821, Xei = Izl
P[X, I; SS S2t = sci. Xci = 2<6
P(S, = Ski Sc, Xxi Izt
P(X, J;
S = 0, K. = 8ci, Xxi = xel
1/2
P(X, J;
SS k4 sci, X<i = x-l
1/2
Pg [X.]
PD X,
As in Proposition I, applying the law of total probability to introduce conditioning on Xgi yields:
ecte
PPs. = 1 7 Se, = sci] S + comE!
and we can proceed by induction. Assume inductively that W 11 - Em- T. S, is stochastically 15-11m dominated (see Definition 4.8 in Steinke et al. (2023)) by W m=11 - Σi T S in which S' ~
cft Bernoulli 1+e=+// 1m-1, Setting W1- = W/ = makes true for m = Then, conditioned on W... W1
and using Eq. 5, Th Sm Th 11 ((5m = 1} is stochastically dominated by Thh Bernoulli T+e Applying Lemma 4.9 from Steinke et al. (2023) shows that Wm. is stochastically dominated by W which proves the induction and implies the proposition's statement.
ct
m
C Experimental Details
Hereafter, we provide the details about the datasets and models trained in our experiments.
C.1 Image data
Target models. We audit target models with the following architectures: a Multi-Lahel Convolutional Neural Network (CNN) with four layers O' Shea & Nash (2015), and the ResNer101 He et al. (2015) as well as a Wide ResNet-28- 10 Zagoruyko & Komodakis (2017b), and Vision Transformer (ViT)-small Dosovitskiy et al. (2021). We also include in our analysis, differentialll--priee for 18 He et al. (2015) and WideResNet-16-4 Zagoruyko & Komodakis (2017a) models as targets, with e = 1, 2, 4, 6, 10, 15, 20. The ResNet-based models are trained on CIFAR10 10 using 50k images Krizhevsky (2009) of 32x32 resolution. For ResNet101 CIFAR 10 (0-based classification models, we use a training batch size of 32; for WideRessNet-28. 10, we use batch size 128. The details for training DP models are given in D.9. The associated test accuracies and epochs are mentioned in Table I. The Multi-Label CNN is trained on 200k images of CclebA Liu et al. (2015) of 128x128 resolution, training batch-size 32, to predict 40 attributes associated with each image.
models
Generator. For both image datasets, we use StyleGAN2 Karras et al. (2020) to train the generative model G from scratch on Da. and produce non-member images. For CIFAR10 dataset, we use a 10, 000 out of 50, 000 images from the training data the target model to train the generative model. For the CelebA dataset, we select 35, 000 out of 200, 000 images from the training data of the target model to train the generative model. Generated images will in turn serve as non-members for performing the MIAs. Figure 7 shows examples of member and non-member images used in our experiments. In the case of CelebA, we also introduce a vanilla CNN as a classifier or filter to distinguish between fake and real images, and remove any poor-quality images that the classifier detects with high confidence. The data used to train this classifier was the same data used to train StyleGAN2, which ensures that the generated high-resolution images of high quality.
MIA and Baseline training. For the MIA, we follow a loss-based attack approach: PANORAMIA takes as input raw member and non-member data points for training along with the loss values the target model f attributes to these data points. More precisely, the training set of PANORAMIA is:
(D% f(D!)) U (D ff ((Dtre))
16

---

In $4.2, we discussed the importance of having tight op so that our measure, E, becomes close to a lower-bound on e-DP, which requires strong baseline. To strengthen our baseline, we introduce the helper model h, which helps the baseline model b by supplying additional features (i.e, embeddings) that can be viewed as side information about the data distribution. The motivation is that h's features might differ between samples from 'D and D'' enhancing the performance of the baseline classifier. This embedding model á is similar in design to f (same task and architecture) but is trained on synthetic data that is close in distribution to the real member data distribution.
The helper model has the same classification task and architecture as the target model. To train it, we generate separate sets of training and validation data using our generator. For image data, the synthetic samples do not have label. We thus train a labeler model, also a classifier with the same task, on the same dataset used to train the generator. We use the labeler model to provide labels for our synthetic samples (training and validation sets above). We train the helper model on the resulting training set, and select hyperparameters on the validation set.
Whether for the baseline or MIA, we use side information models (h and f. respectively) by concatenating the loss of h(z) and f(x) to the final feature representation (more details are provided later) before a last layer of the MIA/Baseline makes the membership the prediction. Since we need labels to compute the loss, we label synthetic images with a Wide ResNet-28-2 in the case of CIFARIO, and a Multi-Label CNN of similar architecture as the target model in the case of CelebA labeling. For both instances, we used a subset of the data, that was used to train the respective generative models, to train the "labeler" classifiers as well. The labeler used train the helper model for image data has 80.4% test accuracy on the CIFAR1O real data test set. The rationale for this approach is to augment the baseline with a model providing good features (here for generated data) to balance the good features provided to the MIA by f outside of the membership information. In practice, the labeled generated data seems enough to provide such good features, despite the fact that the labeler is not extremely accurate. We studied alternative designs (e.g., model trained on non-member data, no helper model) in Appendix D.I, Table 5, and the helper model trained on the synthetic data task performs best (while not requiring non-member data, which is key point).
We use two different modules for each MIA and Baseline training. More precisely, the first module optimizes image classification using built-in Pytorch ResNet101 classifier. The second module, in the form of multi-layer perceptron, focuses on classifying member and non-member labels via loss values attributed to these data points by f as input for the loss module of MIA and losses of e to the baseline respectively. We then stack the scores of both image and loss modules into a logistic regression task (as a form of meta-learning) to get the final outputs for member and non-member data points by MIA and baseline b. When training the baseline and MIA models, we use a validation set to select hyper-parameters and training stopping time that maximize the lower bounds (effectively maximizing the op or éd6 for the Baseline and MIA respectively) on the validation set. The MIA and baseline are trained 4500 data samples (half members and half generated non- non-members). The test dataset consists of 10000 samples, again half members and half generated non- non-members. The actual and final number of members and non- members that ended up in the test set depends on the Bernoulli samples in our auditing game. We repeat the training process over 5 times, each time independently resampling the training dataset for the MIA and baseline (keeping the train dataset size fixed). We report all our results over a 95% confidence interval and report the standard deviation for our results in Table 3. The compute resources used for MIA and Baseline training were mainly running all experiments on cloud-hosted VMs, using I v v1001 GPU, 4 nodes per task, and 32G memory requested for each job on the cloud cluster. The time to run the MIA and baseline attack pipeline was around 10 hours including hyperparameter tuning on et and op respectively.
C.2 Language Modeling
Target model. The target model (a small GPT-2 Radford et al. (2019)) task is causal language modeling (CLM), self-supervised task in which the model predicts the next word in given sequence of words (i.e., context), which is done on a subset of WikiText- 103 dataset Merity et al. (2016), collection of Wikipedia articles. While the GPT-2 training dataset is undisclosed, Radford et al. (2019) do state that they did not include any Wikipedia document.
One common standard pre-processing in CLM is to break the tokenized training dataset into chunks of equal sizes to avoid padding of the sequences. To achieve this, the entire training dataset is split into token sequences of fixed length, specifically 64 tokens per sequence, refer hereafter as "chunks".
17

---

(a) Real
(b) Synthetic
(c) Real
(d) Synthetic
Figure 7: Member and Mon-Member datasets used in our experiments for CelebA at 128 x 128 resolution (7(a).
7(b)) and CIFARIO 10 (7(c), 7(d)) image data at 32 x 32 resolution.
Each of these chunks serves individual sample within training dataset. Therefore, in the experiments, the membership inference will be performed for a chunk. In other words, we adopt an example-level differential privacy (DP) viewpoint, where example is considered sequence of tokens that constitute a row a batch of samples. This chunk- based strategy offers additional benefits in synthetic sample generation. By maintaining uniformity in the length of these synthetic samples, which also consists of 64 tokens, it is possible to effectively mitigate any distinguishability between synthetic and real samples based solely their length. We also consider the possibility of weak correlations between chunks (c.g., being from the same article). To mitigate this, we perform form of stratified sampling as a pre-processing step. From each document, we include only the first 50 chunks and discard the rest. We select 2, 13 documents in total, resulting in 105,65 105,650 chunks in our dataset.
We split the dataset into train and test with a 90:10 ratio for the target model (We insist on following the typical training pipelines in machine learning. since plan to conduct post-hoc audits with PANORAMIA). The test dataset provides real non-member samples, allowing comparison to the O(1) auditor and another version of approach which uses real non-member data. The training dataset (Dy) contains 95,085 samples, while the test dataset consists of 10,565 samples.
We train the target model for 200 epochs in total. Figure 8 displays how the training and validation cross-entropy loss changes throughout the training, when we audit models from epochs 37 (best generalization), 75, and 150. To check how well the target model generalizes, we look at the cross- entropy loss (on a validation set), which is the only metric in a causal language model task to report (or the perplexity Radford et al. (2019) which conveys the same information).
Loss
Train Loss Validation Loss
25
50
75
100
125
150
175
200
Epoch
Figure 8: Target model (GPT-2) loss during overtraining on WikiText dataset. pick GPT-2_E37, GPT-2_E75, GPT-2_E150 as target models to audit corresponding to the gray dashed lines.
The compute resources used for training the target models are V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the target model for 200 epochs required about 13 hours.
The G is GPT-2 fine-tume tne-tuned using a CLM task on dataset DG. a subset of Dy. To create synthetic samples with G, we use top-k sampling Holtzman et (2019) method an auto-regressive manner while keeping the generated sequence length fixed at 64 tokens. To make G generate samples less like the real members it learned from (and hopefully, more like real mon-members), we introduce more randomness in the generation process in the following manner. We choose top k value of
Generator.
18

---

200, controlling the number of available tokens for sampling. In addition, we balance the value of top_k. k, making it not too small to avoid repetitive generated texts but also not too large to maintain the quality. Finally, we fix the temperature parameter that controls the randomness in the softmax function at one. Indeed, larger value increases the entropy of the distribution over tokens resulting in more diverse generated text.
However, the quality of synthetic text depends on the prompts used. To mitigate this issue, we split the De dataset into two parts: Derain and Derompt, G is fine-tuned on D while Derompt is used for generating prompts. Numerically, the size of the Di is 35% of the Dy and the size of D Dprompe is 15% of D. During generation, we sample from DG Derompt, which is a sequence of length 64. We feed the prompt into the generator and request the generator to generate suffix of equivalent length. In our experiments, for each prompt, we make 8 synthetic samples, supplying a sufficient number of synthetic non-member samples for different parts of the pipeline of PANORAMIA, including training the helper model, and constructing D D Overall, we generate 92276 synthetic samples out of which we use 53502 of them for training the helper model and keep the rest for audit purposes. Note that this generation approach does not inherently favor our audit scheme, and shows how P PANORAMIA can leverage existing, public generative models with fine tuning access high-quality generators at reasonable training cost.
We also perform sanity check on the quality of the generated data. We visualize the loss distribution of in-distribution members, in-distribution non-members, and synthetic data under both the target and helper models, aiming for synthetic data that behaves similarly to in-distribution non-member data. Figure 14 visualizes these distributions for our target and helper models.
The compute resources used for fine-tuning the generator model, and synthetic text generation are 4 V V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the GPT2-small model took approximately 1.5 hours, and generating the synthetic samples required about 1 hour.
Baseline & MIA. For both the baseline and PANORAMIA classifiers, we employ a GPT-2 based sequence classification model. In this setup, GPT-2 extracts features (i.e.. hidden vectors) directly from the samples. We concatenate a vector of features extracted using the helper h (for the baseline) or target model f (for the MIA) to this representation. This vector of features is the loss sequence produced in a CL.M task for the respective model. These two sets of extracted features are then concatenated before being processed through feed-forward layer to generate the logit values required for binary classification (distinguishing between members and non-members).
In the main results in Table 3, for both the baseline and PANORAMIA classifiers, the training and validation sets consist of 20000 and 2000 samples, respectively, with equal member non-member distribution. The test set consists of 10000 samples, in which the actual number of members and non-m mon-mebbers that ended up in the test set depends on the Bernoulli samples in the auditing game. The helper model is fine-tuned for 60 epochs on synthetic samples and we pick the model that has the lowest validation loss throughout training, generalizing well.
As the GPT-2 component of the classifiers can effectively minimize training loss without achieving strong generalization, regularization is applied to the classifier using weight decay. Additionally, the optimization process is broken into two phases. In the first phase, we exclusively update the parameters associated with the target (in PANORAMIA) or helper model (in the baseline) ensure that this classifier has the opportunity to focus on these specific features. In the second phase, we optimize the entire model as usual. We train both the baseline and Membership Inference Attack (MIA) models with 5 different seeds. The randomness is over: 1. The data split into train/ mainn/alidation/tes sets for both the baseline and MIA classifiers. Each classifier retrained for every seed. 2. The coin flips in our auditing game (see Definition 2). For a fixed seed mentioned above, we employ an additional set of 5 seeds to account for the randomness in the training algorithm (such as initialization of the model, and batch sampling). We then train a classifier for each of these seeds. From the 5 models generated for each latter seed, we select the one with the largest cb value (for the baseline) or {c+C)ib (for MIA), determined over the validation set. We calculate the 95% confidence interval of our measurements using the t-score.
The compute resources used for MIA and Baseline training were I GPU with 32gb memory, running on cloud- VMs. The compute time depends on the training set size, for the largest training size, it took about 3 hours to train the bascline or MIA.
19

---

The compute resources used for fine-tuning the helper model are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, took about hours to fine-tune the helper model.
C.3
Tabular data
Target f and the helper model h. The Target f and the helper h model are both Multi-Layer Perceptron with four hidden layers containing 150, 100, 50, and 10 neurons respectively. Both models are trained using a learning rate of 0.0005 over 100 training epochs. In the generalized case and to retain the model's 's parameters that yield the lowest loss validation set,
obtain the embedding, which typically occurs at epoch 10 (MLPE 10). For the overfitted scenario, we keep the model's s state after the 100 training epochs achieved an accuracy of 86% and 82% respectively.
The non overfitted model and the overfitted one
Generator. We use the MST method McKenna et al. (2021) which is a differrentialy-rrivee method to generate synthetic data. However, as we do not need Differential Privacy for data generation we simply set the value for to 1000. Synthetic data generators can sometimes produce bad-quality samples. Those out-of-distribution samples can affect our audit process (the detection between real and synthetic data being due to bad quality samples rather than privacy leakage). To circumvent this issue, we train an additional classifier to distinguish between real data from De and additional synthetic data (not used in the audit). We use this classifier to remove from the audit data synthetic data synthetic samples predicted as synthetic with high confidence.
Baseline and MIA. To distinguish between real and synthetic data, we use the Gradient Boosting model and conduct grid search to find the best hyperparameters.
C.4 Comparison with Privacy Auditing with One (1) Training Run: Experimental Details
We implement the black-box auditor version of O(1) approach Steinke et al. (2023). This method assigns membership score to a sample based on its loss value under the target model. They also subtract the sample's loss under the initial state (or generally, a randomly initialized model) of the target model, helping to distinguish members from non-members even more. In our instantiation of the O(1) approach, we only consider the loss of samples on the final state of the target model. Moreover, in their audit, they choose not to guess the membership of every sample. This abstention has an advantage over making wrong predictions as it does not increase their baseline. Roughly speaking, their baseline is the total number of correct guesses achieved by employing a randomized response (c. 0) mechanism, for those samples that O(1) auditor opts to predict. We incorporate this abstention approach in our implementation by using two thresholds, f_ and t.. More precisely, samples with scores below f. are predicted as members, those above t_ as non memmmemeers, and the rest are abstained from prediction. We check for different combinations of L_ and L. and report the highest e among them. This involves performing multiple tests on the same evidence, each with a separate confidence level of 95%. To ensure that these tests hold collectively, we apply a union bound and adjust the significance level accordingly. The total number of separate tests depends on the different number of guesses that O(1) can make. We also set å to 0 and use a confidence interval of 0.05 in their test. In PANORAMIA, for each hypothesis test (whether for eb or {c+e)m)) 1b), we stick to a 0.025 confidence interval for each one, adding up to an overall confidence level of 0.05.
D Results: Additional Experiments and Detailed Discussion
D.1 Baseline Strength Evaluation on Text and Tabular datasets
The basis of our privacy measurement directly depends on how well our baseline classifier distin- guishes between real member data samples and synthetic non-member data samples generated by our generative model. As mentioned in Section 5.1 and Appendix C. in order to increase the performance of our baseline b, we mimic the role of the target model f's loss in the MIA using a helper model h, which adds loss-based feature to b. To assess the strength of the baseline classifier, we train the baseline's 's helper model h on two datasets, synthetic data, and real non-members. We also train a baseline without any helper model. This enables us to identify which setup allows the baseline classifier to distinguish the member and non-member data the best in terms of the highest Cu. value (ep, reported for each case in table 5). For initial analysis, we keep a sub-set of real data as members to train a helper model with them, for conducting this ablation study on the baseline.
20

---

Baseline model
Gbb
CIFAR-IÓ Baseline DE gen, WRN
2.44 0.19
CIFAR-I0 Baselinepy, DJ real, WRN
2.21 + 0.17
CIFAR-I0 Baseline D", real. resnetiOl
2.01
CIFAR- 10 Baseline 1T, imgnet
1.12
CIFAR- 10 Basclineno helper
1.25 125 0.24
WikiText Baseline 0% gen
1.31 0.15
WikiText Baseline, Dg = real
3.26 + 0.14
WikiText Baselinego helper
3.11 0.15
Table 5: Baseline evaluation with different helper model scenarios, where WRN is Wide-ResNet-282 helper model architecture (in case of CIFARIO baseline).
Our experiments confirm that the baseline with loss values from a helper model trained synthetic data performs better compared to a helper model trained on real non-member data. This confirms the strength of our framework in terms of not having a dependency of using real non-members. Therefore, we choose this setting for our baseline helper model. In addition, we also experiment with different of CIFARIO, looking for the best baseline performance among
helper model architectures in the them. For the Wiki'Text dataset, the On reported in Table 5 differs from that in Table 3 because we had less data available for this experiment. We withheld half of the real samples train the helper model with real samples in its training set.
For each data modality setting, also train the bascline classifier real member and synthetic non-member data points to gauge the trend of the baseline performance as the training dataset size increases (see Figure 2 and Figure 9). This experiment allows to aim for worst-case empirical lower bound on how close the generated non-member data is to the real member data distribution, i.e. what would be the largest Ob we can reach as we increase the training set size. Subsequently, we can compare the Ob reported in table 3 (corresponding to the vertical dashed lines in Figure 2 and Figure 9) the largest Ot we achieve in this experiment. In the following, explain why we chose that number of training samples for our baseline in the main results.
increasing dataset size of
On the WikiText dataset, we increase the baseline's 's training set size up to 50k, while fixing the validation and test sets sizes to 2000 and 10000, respectively). Figure shows the changes of cb while varying the training set size. The vertical dashed line specified the number of training samples we chose for our main auditing results reported in Table 3.
Now, let us discuss why we don't 't choose the largest training set size available for the baseline. The measurement for ob depends not only on the training set size but also on the test set size. Therefore, we need to balance the sizes of the train and test sets. We conducted another experiment where we varied the train set size and, instead of fixing the test set size, allocated the remaining available data to the test set (keeping in mind that our data for the audit is limited by the number of real members allocated for audit (rrppooss).Fiuurr 9(b) depicts the trend of changes. As expected, we observed an initial increase in Cb the training set size increased (and the number of test data is sufficient to capture that). However, it decreased toward the end because of the smaller test set size. For a train set size of 40k, we found the best trade-off between train and test set sizes, which justifies our choice of 40k for the training set size in the main results. The same argument applies for other data modalities.
Let us make one last point. It is also important to note that the MIA is expected to show similar behav- ior if we use a larger train size or test set size. Hence, the gap between the performance of MIA and baseline (which is our privacy measurement) is not expected to be affected (see Appendix D.4).
The experiments are repeated five times, and provide 95% confidence interval. For the image data modality, the randomness lies in the training data provided to the classifiers, while the test set remains fixed throughout. For the text data modality, both the train and test sets change across the five runs (the test set size is fixed to 10k).
D.2 Further Analysis on General Privacy Measurement Results
The value of for each target model the gap between its corresponding dashed line and the baseline one in Figure 4. This allows us to compute values of reported in Table 3. It is also interesting to note
21

---

4.5
4.5
4.0
4,0
3.5
10-
3.0
2.5
2.5
2.0
2.0
1.-
1.5
1.0
0.5-
Baseline Wilkieett
0.5
Baseline-WiklTex
0.0
2000
20000 30000
40000
50000
0.0
2000 10000 20000 30000
40000 50000 60000 70000
Training Dataset
Training
Size
(a) as the size of the training set increases for baseline (while the test size 10k), trained classify real (WikiText) and synthetic data points. The vertical dashed line indicates the number of samples we used to train the baseline in our main results, reported in Table 3.
(b) the size of the training set increases for bascline, while the test set size is changing the same time. The number test samples for given train set size specified by the total number of sam- ples we have for the audit (Dm),, which is limited by the number of real members.
Figure 9: Baseline Strength Evaluation on WikiText dataset as the train set size increases. In Figure 9(), the test set size is fixed, while in Figure 9()) it varies well. See Appendix D.1 for more details.
12:00
PaRcAaPI RasheouL 823
1000
PHRKRAANDA CAN LSD
Io tha par
Naronaii Roakettl E230
Bumror tuu poultow
PAKORAMN CAN 5383
KALORAAA OFT
3X0
Recal
(a) Number of predictions (b) Number of predictions (c) Number of predictions (d) Number of predictions of ResNet101 on CIFARIO of Multi-Label CNN on the of GPT-2 model on Wiki-of an MLP on the Adult dataset.
CelebA
Text
Figure 10: Comparison of the number of positives and predictions different datasets
that the maximum value of {c+c]b typically occurs at low recall. Even if we do not use the same metric (precision-recall as opposed to TPR at low FPR) this is coherent with the findings from Carlini al. (2022b). We can detect more privacy leakage (higher (c+c) which leads to higher ?) when making a few confident guesses about membership rather than trying to maximize the number of members found (i.e., recall). Figure 10, decomposes the precision in terms of the number of true positives and the number of predictions for a direct mapping to the propositions and 2. These are non-negligible values of privacy leakage, even though the true value is likely much higher.
D.3 PANORAMIA performance with increasing Test and Train Size m
Increasing Test Set Size: We compare PANORAMIA with the O(1) audit (Steinke et al., 2023) (in input space, without canaries), which uses a loss threshold MIA. As shown in Figure 6(b) and Figure 6(c) the CIFAR10 image dataset case, both for non private and Differentially Private (DP) models, PANORAMIA can leverage much more data for its measurement (up to the whole training set size for training and testing the MIA), while in our setting O(1) is limited by the size of the test set (for non-members). We vary test size from times with resampling of the test set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence.
PANORAMIA:
500, up to 35k. The experiment is repeated 10
22

---

D.4 PANORAMIA with Increasing Train Dataset Size
We also conduct experiments to show the effect of increasing training dataset size on the performance of PANORAMIA (as well as the baseline). In the CIFARIO image dataset case, Figure 11, we see PANDRAMIA performance slowly increase as the train dataset size increases. performance dataset sizes 4500 5000, 10k, 20k 30k and slowly begins stabilize at the higher dataset sizes. We vary train size from m = 500, up to 30k. The experiment is repeated times with resampling of the train set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence.
reaches its max
24.0
3.5
3.0
2.5
Measurement ment of
2.0
1.5
1.0
Baseline-IIFAR10
0.5
Panoramii-FestNeet1011100
0.0
500
5000
10000
15000 20000 25000 30000 Train Dataset Size
Figure 11: Effect PANORAMIA and baseline with increasing train size for CIFARIO, ResNet101 target model (trained for epochs) (with union bound correction).
D.5 PANORAMIA in cases of no privacy leakage
As illustrated in Figure 4(C) and Table 3, there exist instances in which the eb is as large as {c++e) e J For instance, for GPT-2_ E37, the upper bound in the confidence interval measured for Ob is larger than the lower bound of the measurement for {h. Although the measured é is positive on average, but the lower bound of the interval negative. For such cases, we don't 't claim any detection of privacy leakage. This implies that, under certain conditions, the baseline could outperform the MIA in PANORAMIA. We can see three possible reasons for this behavior.
(1) The synthetic data might not be of high enough quality. If there is an evident distinction between real and synthetic samples, the baseline cannot be fooled reducing the power of our audit. (2) The MIA in PANORAMIA is not powerful enough. (3) The target model's 's privacy leakage is indeed low. For GPT-2_EI15 50 and GPT-2_E7 E75 in Figure 4(c), we can claim that given the target model's leakage level, the synthetic data and the MIA can perform the audit to a non-negligible power. For GPT-2_37, we need to examine the possible reasons more closely. Focusing on generated data, in the ideal case we would like the synthetic data to behave just like real non-member data on Figure 14(a). If this was the case, fc+tehn hb would go up (since the MIA uses this loss to separate members and non-members). Nonetheless, predicting changes in Cp remains challenging, analogous loss distributions do not guarantee indistinguishabilit between data points themselves (also, we cannot necessarily predict how the loss distribution changes under the helper model h in Figure 14(d)). Hence, reason (1) which corresponds to low generative data quality, seems to be a factor in the under- undder-erffrmance of PANORAMIA. Moreover, in the case GPT-2_311 37, we observe that real members and non-members have very similar loss distributions. Hence, factor (2) also seems to be at play, and a stronger MIA that does not rely exclusively the plot may increase auditing power. Finally, the helper model h seems to have a more distinct loss distribution when compared to GPT-2_37 (see Figure 14(d)).
D.6 PANORAMIA with Tabular data
As we can see in Table 6, our proposed method PANORAMIA cannot detect any meaningful privacy leakage in the case of small MLP classifier trained on tabular data.
23

---

PANORAMIA
0(1) 8
MLP ETO
0 0.142
0.655 = 0.014
MLP MLP_E100
0.062 + 0.156
0.648 + 0.015
Table 6: Privacy measurement with PANORAMIA and 0(1) for models trained on tabular Adult dataset.
As shown in Figure 12(b) there is no significant difference between the values + E)1b Cta returned by PANORAMIA, As result we get = 0
Aomeen OR NACKE POREEIIEEOOO CONTONCIO 1MV/M
10-
2.0
Cosio satmal
0.9
1.5
Precision
0.8-
1.0
0.7-
baseline model
0.5
PANORAMIA 1100
0.5
PANORAMIA MLP E10
Theoretical maximum precision
0.5
0.0
0.0
Recall
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
(a) precision recall, for tabular one instance of the (b) Average ec (or cm.) vs recall with 95% con- fidence, for tabular data classification. Randomness comes from resampling of the MIA's train and test set as well as retraining the MIA
tabular data classification audit.
Figure PANORAMIA on tabular Adult datasct, MLP target model.
The high precision at low recall in the baseline as can be seen in Figure 12(a) illustrates the fact that some real data can easily be distinguished from the synthetic ones. Instead of poor quality synthetic data, we are dealing with real observation that are actually "looking to real". Investigating those observations reveals that they have extreme
values in one of their features. Indeed, the Adult dataset contains some extreme values (e.g., in the capital_ gain column), such extreme values are rarely reproduced by the generative model. This leads to the baseline accurately predicting such observations as real with high confidence that translates into a high Cio- Moreover, the audit values maximum values (c +c+& E)Ib and cj being achieved for a relatively small number of guesses corresponding to extreme values, the audits results vary a lot depending on the sampling or not of such extreme values in the audit test set. We believe this is part of the reason why we observe an high variance in the audit results leading to no significant privacy leakage detected by PANORAMIA.
This is potential drawback of our auditing scheme, while the presence of extreme the dataset usually allows for good auditing results it can have a negative impact on PANORAMIA'S success.
values in
D.7 Privacy Auditing of Overfitted MI. Models
Methodology. Varying the number of training epochs for the target model to induce overfitting known to be factor in privacy loss Yeom et al. (2018); Carlini et al. (2022a), As discussed in Section 5.2, since these different variants of target models share the same dataset and task, PANORAMIA can compare them in terms privacy leaking.
To verify if PANORAMIA will indeed attribute higher value of غ to more overfitted models, we train our target models for varying numbers of training epochs. The final train and test accuracies are reported in Table 1. For GPT-2 (as target model), we report the train and validation loss measure of overfitting in Figure 8. Figures 13 and 14 show how the gap between member data points (i.., data used to train the target models) and non non-member data points (both real as well as
24

---

generated non-members) increases as the degree of overfitting increases, in terms of loss distributions. We study the distribution of losses since these are the features extracted from the target model or helper model ht to pass respectively to PANORAMIA and the baseline classifier. The fact that the loss distributions of member data become more separable from non-member data for more overfitted models is sign that the corresponding target model could leak more information about its training data. We thus run PANORAMIA on each model, hereafter presenting the results obtained.
25
real member losses
member losses
Frequency (normaiized
fake non-member losses
Frequeng ney (normalized)
fake non-member losses
real non-member losses
real non-member losses
10
0.0
0.0
0.1
Losses
Losses
(a) CclebA Multi-Label CCN_E30
CelebA Multi-Label CNN E100; an overfit model.
Figure 13: CelebA Multi-Label CNN Loss Comparisons for generalized overfitied model.
Results. In Figure 3, we observe that more training epochs (i.e., more overfitting) lead to better precision-recall trade-offs and higher maximum precision values. Our results further confirmed by Figure 10 with P PANORAMIA being able to capture the number of member data points (positive predictions) better than the baseline b.
In Table 3, we further demonstrate that our audit output ē orders the target models terms of privacy leakage: higher the degree of overfitting, more memorization and hence higher ? returned by PANORAMIA. From our experiments, we consistently found that as the number of epochs increased, the value ē also increased. For instance, in the WikiText dataset, since we observe {c+e]lb 1 GPT-2_EIS0 {c++e) GPT-22_75 > (c++))/ra G37 we would have {c+e) GPT-2 ab > {c++e GPP-2_ GPT-2_637 (c+e lb
cn >
which leads to EGPT > 6GPT2_E75 > ( Our experiment is coherent with the intuition that more training epochs lead to more over-fitting, leading to more privacy leakage
measured with higher value of ?
4.0
G 3.5
3.00
2.5
Measurement for
1.5
Cb
PANORAMIA: ResNet101
1.0
PANORAMIA: ResNet50
0.5
PANORAMIA: Wide ResNet-28-2
empirical maximum value
0.0
0.0
0.2
0.4
10
Recall
Figure 15: CIFAR-10, {c + e)s when varying privacy leakage with increasing model parameters over one experiment run.
25

---

Members Synthetic Non-menmerss
Members
Synthetic Non-memberss
Frequency (normalized)
0.8
Real non-members
Frequency (normalized)
Real non-members
0.8
0.6
0.6
0.4
0.4
0.2
0.2
5
values
values
(a) GPT-2_ES 237
(b) GPT-2_E75
Real Members Synthatic Non-members
Real Members Synthetic Non-members
1.0
Frequency (norma (ized)
non-me-mbers
Frequency (normalize))
notwmembers
0.8
0.8
0.6
0.6
0.4
0.2
0.0
Loss values (c) GPT-2_E1550
Loss values
(d) Helper model h.
Figure Comparison of loss distributions of real memhers, real non-memhers, and synthetic non-members under three target models while varying the degree of over-training on the WikiText dataset. Figure 14(d) compares the loss distributions under the helper model, the model providing side information our haseline. We train the helper with some other synthetic samples. which effectively mimic real non-members' loss distributions under the larget models. However, they are distinguishable to some extent from real memberrers under helper model, thus increasing our cib-
D.8 Models of varying complexity:
Carlini et al. (2021) have shown that larger models tend to have bigger privacy losses. To confirm this, we also audit MI models with varying numhers of parameters, from a 4M parameters Wide ResNet- 28-2, to 25.5M parameters ResNet-50, and 44.5M parameters ResNet-101, Figure 15 shows that
PANORAMIA does detect increasing privacy leakage, with wwide-resnet S Cresnet50 S Cresnet101-
D.9 Privacy Auditing of Differentially-rrvate ML Models
Methodology.
For the image data modality, we evaluate the performance of PANORAMIA on dffferentialll-privaae ResNet-18 and Wide-ResNet- 16-4 models on the CIFARIO dataset under different target privacy budgets (e) with 10 and the non-private DP-SGD algorithm (Abadi et al., 2016) using Opacus (Yousefpour et al., 2021), which we tune for the highest train accuracy learning rate Ir, number of epochs e, batch size bs and maximum &z clipping norm (C) for the largest final accuracy.
oo) cases. The models trained using the
For the text data modality. we evaluate the performance of PANORAMIA on the differentially private GPT-2-Small model, which has been fine-tuned on WikiText dataset (the preprocessing steps are identical to those in Appendix C.2). As with the image modality, we use various target privacy budgets (c) with 5 10-5 as well as non-private settings (e = oc). We train the models using the DP-SGD algorithm Abadi et al. (2016), incorporating Ghost clipping Li al. (2021) for calculating
26

---

the per-example gradient. We minimize the validation loss by tuning hyperparameters such as the learning rate (Ir), number of epochs (e), batch size (bs), and clipping norm (C)) The lowest validation loss values obtained are 3.79, 3.7, and 3.65 for e values of 1, 3, and 10, respectively. For the non-private GPT-2-Small model, the lowest validation loss achieved 3.341.
The noise multiplier a is computed given number of epochs and batch size. We use both PANORAMIA and O(1) (Steinke al., 2023) privacy loss measurements under the pure e-DP analysis.
Results. For the image data modality, tables 7 and 8 summarizes the auditing results of PANORAMIA on different DP models. For R8sNet-18, observe that at 1, 4, 6 (more private models) PANORAMIA detects no privacy loss. whereas at higher values of e = 10, 15, 20 (less private models) and é Do (a non non-private model) PANORAMIA detects increasing level of privacy loss with f==10 < 65=15 < c0= < fc=oe, suggesting a higher value of e correspond to higher é. We observe similar pattern with Wide-ResNet-16-44 in which no privacy loss is detected higher privacy loss is detected at c 10, 15, 20, oc. We also compare the auditing performance of PANORAMIA with that of O(I) (Steinke et al., 2023), with the conclusion drawn by these two methods being comparable. For both ResNet-18 and Wide-ResNet--6-44 O(1) reports values close to 0 (almost random guess between members and non-members) for < 10 DP models, and higher values for = 10, 15, 20, oo DP models. The results suggest that PANORAMIA is potentially an effective auditing tool for DP models that has comparable performance with O(1) and can generalize to different model structures.
= 1,2 and
For the text data modality, Table 9 summarizes the auditing results of PANORAMIA on differentially private (fine-tuned) Large Language Models. In all cases, neither PANORAMIA A not O(1) Steinke et al. (2023) do not detect any privacy loss.
Target model
Audit
Ctb
Ct
E
E
ResNet18 c DC
PANORAMIA RM;GN
2.44
3.87
1.43
O (1) RM:RN
1,711
ResNet18 c
PANORAKIA RM:GN
2.44
3,6331
I.19
(1) RM.RN
1,34
ResNet18
PANORAMIA RM:GN
2.44
3.57
1.13
O (1) RM:RN
1.22
ResNet18 r 10
PANORAMIA RM;GN
2.44
2.70
0.26
(1) RM:RN
0.28
ResNet18 c 6
PANORANTA RM,GN
2.44
2.478
0.038
O (1) RM:RN
0.049
ResNet18 c 4
PAHORAMIA RM:GN
2.44
1.99
0
O (1) RM-RN
0
ResNet18 c
PANORAMIA RM:GN
2.44
1.709
0
O (1) RM:RN
0.05
ResNet18
PANDRAMIA RM:GN
2.44
1.412
(1) RM:RN
Table 7:
Privacy audit of ResNet18 under different values of e- e-Differential Privacy using PANORAMIA and O(1) auditing frameworks, in which RM is for real member, RN for real non-member and GN for generated (synthetic) non members.
Target model
Audit
fe
c/P
WRN-16644 00
PANCRANTA RMAGN (1) RM-NN
2.44
3.02
0.58
0.6408
WRN-16-4 20
PANORANIA RM.GN
2.44
2.926
0,486
(1) RM,RN
0.5961
WRN-1664 15
PANGRAKIA RM;GN
2.44
2.912
0.472
(1) RM.RN
0.5774
WRN-16-1 10
PANORANIA RMEGN
2.44
2.783
0.343
O (1) RMRN
0.171
WRN-16-4
PANORANIA RM.GN
2.44
2.36
RM.RN
0
WRN-16-4c=1
PANGRANIA RM;GN
2.44
1.84
( RM:RN
0
Table 8: Privacy audit of Wide 4 under different values of c-Differential Privacy (DP) using PANORAMIA and O(1) auditing frameworks, where RM is for real member, R.V for real non member and GN for generated (synthetic) non memmmerrrs.
27

---

larget model 1
Audit
Generator G
DEDDGPPT-SS (371
PARCICAPIA
GPT2-Small
0223
2,82 0.31
LDDGPPT2-Mm
PARORAMTA (1) RM.RN
GPI-2-Small
3-DP
GPI-2-Small
(1) RM:RN
GPll-Smal
PANURAMTA
-2--Slll
(1) RM.RN
Table 9: Privacy audits GPT-2-Small target models with varying privacy guarantees.
Discussion. the image data modality, we observe that the auditing outcome (ē values for PANORAMIA and for O(1)) can be different for DP models with the same values (Table 10). We hypothesize that the auditing results may relate more to level of overfitting than the target c valuuss in trained DP models. The difference between train and test accuracies could be possible indicator that has a stronger relationship with the auditing outcome. We also observe that O(1) shows a faster increase in for DP models with higher targeted values. We believe depends the actual ratio of the correct and total number of predicted samples, since O(1) considers both true positives and true negatives while PANORAMIA considers true positives only. We leave these questions for future work.
In the text data modality, the membership inference attacks (MIAs) relying on loss thresholds in privacy audits, without retraining access to canaries, may not be sufficiently strong to effectively classify members and non-members.
Target model
Audit
Train
Test Acc
DEN Train- Test Acc)
PANORAMIA RMEGN
2.308
3,63
1.06
71.82
67.12
4.70
O RM-NN
1.22
ResNeti 8-eps-20
RM:GN
2,508
2.28
0
71.78
68.08
3.70
RM-RN
0.09
PANORAMIA RMEGN
2,08
3,63
1.13
69:01
65.7
3.31
( RM.NN
1.34
PANORAMIA RM.GN
1.61
69.30
2.62
( RMENN
0.08
Table 10: DP models with the same values can have diffeerntt auditing outcomes (the above numbers are reported here without union-bound correction).
E Generator Closeness 7-relaxation
Definition 4 (c. clloseessss Let the 7-approximate maximum divergence between generative model G and data distribution D be,
In P(D
D (D(l)) =
max
SC SCSupp(D)P(DES111
P(g E S)
For all > O we say 9 is (c, cloose to to
Df Df
c i.e.
Vx E X, PD|a| e Po|æ + + e = - 7) s Pg]r)]
E.1 Pure DP case
First we prove similar result as Lemma 5.5 (Steinke et al., 2023) but with the one-sided inequality of (c, closeenss and the case when we additionally observe output from a pure-DP mechanism.
Lemma 1. Let G, D be probability distributions over V, fix c, O. Suppose for all measurable y we have e (D(() 7) < G(S). Then By E [0,7] and D',D", G' G'G"" such that the following properties
satisfied.
(1) D. G can be expressed as a convex combination D = r'
D= (1-7')'' (ec)D'++0000 +//"" 01--0'00'
(2) for all measurable S V, we have -DD GS)):
(3) there exists measurable S C. C such that (S) VS' -.D(S))
28

---

Let M denote an arbitrary c-DP mechanism which takes input Y sampled from distribution G or D and outputs f. Let P, be probability distributions over Z which are distributions of MY) when Y is sampled from 'D and 9 respectively. Fix e 2 0, suppose for all measurable T C 2 we have ((T) S e ( and Q(T) S (T)(T). When additionally observing the output from a DP mechanism we have the following properties. Let (D, P),P) (9, Q) be the joint distribution over (), 2) which we assume D 1 LPGL9
(4) P), (g. Q) can be expressed as a convex combination DP ((1- +
Y 'D")P, = (1--1)g' Q
T C Z, we have e ee D (5) for all measurable S C V.
(6) there exists measurable S y, T C Z such that (8)P()( 1, vS C S,T" C
T, ((T))P())) 2 (P()).
Proof. For the edge cases of ? 0, the results hold with 7 0,''' = - G; similarly when
1, the results hold with 7 = 1, D" = D. = g, D' = g'. Let c' E [0, n. E (0.1). 1), define distribution g', g", D', D as follows. For all points y E V
D'(v) = min (D(y), e ((v))
1 - DV(v) Dly) (1 - (D)
ma e ((()))
71
71
g(v)
('(w) = 1-72
("(y) ((v) (1- (1-)))9(v)) 0.
722
By construction (1 - DD' + +n D" = =D. (1 92) + 72G = G. so the first property is satisfied, and (v) is supported S {y E y D'y) > e (g( so the third property is implied. If 0 < 71 = 72 - S. 7, for all y E y we have,
('(V)
Gly)
2 20
D'
(Dly), ed ((v)
as required for the second property. Following the same as in Lemma 5.5 (Steinke et al., 2023),
we ensure 0- < -1 = 7/2 S y by appropriately setting 61, C2 E [0, c]- We can use the same
decomposition of DD G to prove the second part of the Lemma. If assuming D 1 P and g L Q the joint distributions can be decomposed as fD.P- -fp fo and fa.0 = So - fa- Therefore by construction, DP- = ((1 - -7/DD' + +D'""), G = ((1-7'0' + ) D'P is supported on (S, T) (u E y, M(y) 2 (()))(M(()) > e ((v))(M(()) For all y E ), M(v) E 2 we have,
((v) D'(y) P(M())) min (Dyy, ed ((y)} P(M(y)) 2 2 20 00
g'(v) ((M()))
((M()))
where ((M(g)) P(M())
since M an cDD mechanism.
Similar to Lemma 5.6 (Steinke et al., 2023), next we prove a Bayesian version of Lemma 1 which is used to prove the main result afterwards.
Lemma 2. Let G, D be probability distributions over V. fix c, 2 O, suppose for all measurable
S C y we have e (D((S) S ((S). Then there exists a randomized function Ep.g : y > {0. 1} with the following properties. Suppose X Bernoulli( }), if X sample Y D. and if X sample Y ~ G. Then for all y E y we have,
[X E.) = 1/Y v1 S ec
P X X
1 ee
By>D|ED,o(Pe 2 1- Ey (D) 2-
29

---

Let M denote an arbitrary e-DP mechanism which takes input Y sampled from distribution G or D and outputs f. Let P, be probability distributions over Z which are distributions of MY) when Y is sampled from 'D and 9 respectively. Fix < 2 0, suppose for all measurable T C Z we have P(T) S (T) and ((T)
(T). Let (D,P), (g. 2) denote the joint distributions over (Y, 2) which we assume D 1 P, 9 1 Q Then there exists randomized function E(D.P),(G.O)
(), - 1) such that for all y E y, M(y) E 2 we have,
extr
P
X = L.E..........( 1|Y v.) = f]s
X X-Bernonli + X-XD+(1-X)
+ ecte?
(0))-M(X240-X00
Ey Br..Mcy--lBoo...oo00..... 2 1-7
Br...wyy-eeEFDPPxx0000.... 21-7
Proof. We apply the decomposition from Lemma and denote D' 9' '' as is. We define
Epg : 3 + {0, 1 by,
P(E = 11 -1- D'(y)
=1 >')
D(y) D(y)
For any y E y we have,
P[X = 1, (V) = 1 =
P[X 1|Y =y. 1|F(D,, 1|Y
= -P(X- = 1|Y = -y)P(Eo.c(v== 1]
P = y|X 1| =
P(ED,s()) = =1]
P|Y = yl
D(y)
D(y) + G(y) P. (VE 1]
(1-7)DD'( (1) +D'D(()
(1-Y)D'(v) (V) + (D" + (1))((111111) (1--'19((v) P(FG,D(y) =
(v)
1 (1-1'1D'y)
"'D ('(y) P.E...(y) 1]
14 (1->1)D'M) D'(y)
(v)
1+ (1->'DD(v)
P(E =]
1+0+
(1 - 1''''DD(()) (v) +D')
:P(E,p(u)) = 1]
1
(1- (1)'D(((y)
D(v) .PV =1]
1te
(1-7')D'(m)
e
Ite 1 +
The first equality uses that P[A, B|C] = P[A|B, B/C The second equality holds since we assume the internal randomness of E independent of everything else. The third equality uses the Bayes' rule. The fourth equality uses the definition of the randomized function E. The fifth equality uses the decomposition of Lemma After rearranging and use the properties from Lemma I we obtain the first incquality, then the rest follows from simplifications. Furthermore we have the
30

---

following,
By-o[ED.o) / D(y) P(E)P - 1|dy
/ (1 -- (D)dy 1 - 2 1-7
(D)
Sywg(ED,y /TEx~ D(v)
7 / ((v) DD (y)dy
D(y)
2 7 / (v)dy
=1 2 7
We follow a similar procedure to prove the second part of Lemma. We define ((p.P), (), 2) - (0,1) 13 by.
(V) = = = 1]
With similar reasoning, for any y E y (v) E 2 we have,
F[X-
= 1|Y y, M(Y - P y, My)
y, = = -|ap 1 P (.0
((v))
(M)) ((v) PI.. ((y) =]
((1 ((1- (
((1 (v) -L
(1-
1 (17DP0))M
PB M(y) = 1]
(W)(M))))))) (OOMO)
1+ (1 (L-)((M)(())))) P(())((()))
(D)
1+ (1-
1+0 6- (I (V, 1]
D(y)
(1-
1+e-
elte
- 1te
Since we define the success probability with the same expression, the expected success of the randomized function directly follows,
Ey-p (1) 1 Bv.s.Mrry.aaEeoo.oee...... Ey 2 1-
Next Lemma 2 to prove the following Proposition which is used to form statistical tests hypothesis H.
Proposition 3. Let G be (c, cloose, f be output of an e-DP mechanism, T & B(S,X) the guess from the baseline, & A(S, X, f) be the guess from the membership audit. Let To. Ta E [0, 1 m be bounded. Then, for all v IK and all in the support of T:
m
m
PsX
Σ
T
S.
T = =10]=
P
F(()) +
Σ
1/ S 2 0
S'emoull
Ps
T S 2 T 10) x
P
F(10)
1
S 2 0
S' Bermoulll
I+e
31

---

where F is independent from S', F(T") is supported on {0, .....,m, and Bxt,F(F(TT) ET (F(((T)) S 2my
Proof. Let (sgi) denote the distribution on [0, 1 obtained by conditioning B on past Sei
82i
and gg: = Zgi By Lemma 2, for all th E [0, n we have
e
P[S 1, (.(seen)).)...... (T = 1|T = St s<i, =z v
1te
B| B[BBlegan) (T 1|5 821 =| 11
E BlE (1t) 1|5 sc 21117
Using the law of total probability we get,
P(S. 1, B(ec (Td) 1| t Szi = scil
Σ P(S: BB(sec.)) (I 1| t Set Scc rgi
aSi
P[X Si 2gilT =t Sci scil
Σ P(X 2gi17" = = ,t,,,
Sei = szil 1+ c
1 E
Applying the union bound on St we can bound the expectation of success events by,
E E[E Bl (T = Beis S; = =5]= 2 27
By induction, for & [m] define Wa(s, tb) = Et= 1 8, EB(se()), B. B(scro)) (t"), then (S,, t*) is stochastically dominated by Wa(t)) = E th (t()) where St(t) Bernoulli e failing events F be,
T+ee Let the
m
F(s,t)) = Σ
1
0},
then we have (s, 14) = Em1 8; (m, (8) ) + F(s, is stochastically dominated by Wm. (T+) - F(S,T"), where the expectation of the failing events is bounded by,
E[F(S,T))
Σ IP FlBeeec.)) = 0J S m((7).
Similar to Lemma 5.5 in Steinke al. (2023) since ((F(S is independent of S could have F(T*) (") for S drawn from appropriate distribution. The proof of the second part of the Proposition is essentially the same except with the following result from Lemma 2,
P[S: = 1, E A(secsa) (T = 1 = th, Ses = sei, Xzi = =zil <
1 eater
E E[EAleea))Alesssoo (T 1|5c; Sci. 7
E E|EA(ecu)aAossro))
(Ta) 1|5c; scc, = 7
Similar to Theorem 5.2 (Steinke et al., 2023) we could compute for the optimal distribution F(th) and F(t") by formulating into linear programming problem. Similar to Corollary 1, to test Hypothesis H together we would apply Union bound on the two parts of the hypothesis as in Proposition 3.
E.2 Approximate DP case
In this section we consider the case where we have a (c. cllooe generator and audit for an ap- proximate DP mechanism. We modify the results to account for the failing events from these two sources of randomness. We start by proving a similar version of Lemma 2 where we use both the decomposition of G, D from Lemma and the decomposition of P, Q from Lemma 5.5 (Steinke et al., 2023).
32

---

Lemma 3. Let G.D be probability distributions over y. fix Y, 2 O, suppose for all measurable
S C y we have E -(((s)) G(S), Let M denote an arbitrary (c. 6.D mechanism which takes input Y sampled from distribution 9 or 'D and oulputs f. Let P. Q be probability distributions
over 2 which are distributions of M(Y) when Y is sampled 'D and 9 respectively. Fix 6, 2 0, suppose for all measurable T C 2 we have P(T) S Se*Q(T) + 5 and Q(T) S (T) + 6. Let (D, P), (G, 2 denote the joint distributions over (V.Z) which we assume D 1 P. g 1 Q. Then there exists randomized function ((D.P).G..)) (),, 2) + {0, 1] such that for all V, M(y) E Z we have,
ecte
P X X-Berouullt ) X-XD+11-X0) X16 M(V) MXD++(1166))
X= = (.-... -| -, 1 + eate?
= f
Ey BxooMNg.sTBoopp.ccgg)21101111113 (1-))
(xog:/00-e(//pprccee00001))))110 (1-
Proof. For ease of notation we shorthand F E D'((), 30
=D'(y),''= g'v).
D" = (y), G" = g" (y)) P' = P'()), 2(),
p" =
= (() below. We
define E: (V {0, 13 by.
P(E(y. 1|= - -1
DP
For any y E My) E 2 we have,
P|X 1, E(Y, F) 1Y = V. f1 fl
P|Y y. F f| 1] = P(E(v. 1]
,ly=y,F=1
(())
D(y)P(f) (() P(E(U. 1]
((1 J/DD' + DD' - 8'yp' + +8P")
PLE f = 1]
((1-7'D' +11 8')P' + 88P" (1 (11 8'Q'' 8')"
+
8'D'p"
DDD'''
D"D"""
+ [1 (1-8)
PT + (1-YX(1-1D'' 8/DiPT
P[E f) 1]
1 (1-0'10P))
8'D'P"
"D"P'
O'O'
+ 8'0""" Dr
(1-Y)D''
(1-
8'D'P"
''D'''
'8'D"'"""
(1-0'1D''
(1-7)D'''
[1- 11-")1180'''
P[E(y, 1]
1 1ec-
(1 (1-8)(1-2) - DYDD''' 8'(1 (1- 2'1-"1D''''' +7 8'
(11--) D'''"'' D''"'''
-
(1-2") -10'/''p
-P(E(y, f) 1]
DP
P(E(y,)) = 1]
(11 - 8''yD''' elte
1 T + cote"
Using the assumption that D 1 P. g 1 Q, we account for the expectation of success events as follows,
E BY>D.M F)] // D(U)P())P(F =1]dyd
= |/ P())(()) (1 (1)--1(1-8 (1))"))))))) dydf D(v)P(F)
= -(1 -(1-7)(1 - / [ 'D'()dyy / ('((jjds
= =(1-7)(1 - 88) 2 (1 (])(1-)))))))) 1011-00 -5).
33

---

Note that we can rewrite the success probability
P|E f) = =1 7 -'')D''
(1 (1-2')6 (1''')0''''''
"''D"'"
DP
DP
DP
Then have,
By By- (B F)]
DD (V))''(())
D'(y)P"
(f)
= =1-7(11-0 8
(1 (1-7) 8)
7/8'
(f))
D(y)
D(y)P(f)
11-7 711 -8) / 2(1)
( dydf
D(U)P()
D'(U)P"())
(1 1-108/ (w))in)
dydf
D(y)P(f)
D" ("())P"(
788 (())(())
dydf
D(v)P(F)
2 - (11 -8) / D" D'(u)P"(Ayyd
(f)ddds (1 2')0' ʃ (F)PP"((dyd)
-Y8 / D" ("(y)""((ddyl)
71 - 8)- -(1 -/))'' - 7/88'
- 7 - 6' +
(1 71 8') 2 (1 - - - 8).
Next we use Lemma 3 to prove the following Proposition to form the statistical tests on hypothesis H for the (e, 0DD version.
Proposition 4. Let G be (c,7)-close, J be output of an 8) mechanism, T B(S, X) be the guess from the baseline, T A(S, X, f) be the guess from the membership audit. Let To, To [0, 11Th be bounded. Then, for all IR and all t the support of
Ps
Σ T
m
S 2,20 T = =t0]
P 8 Bermoulli 1400 1mh
F(t')
m
PS
T TP -
2
T tot
P
F(te) Σ 8 2
S' Bernoulii
where F is independent from S', F(T)), F(T is supported on (0, 1, 1.....,,, m] and Eyy f [F(T)) s
2md, ETs ([T") < 2m 8 70).
Proof. The first part of the result (baseline test) exactly follows the proof of Proposition 3. For the approximate DP case, the proof the auditor test follows similar steps except for the bound on the expectation of the failing events,
B[F(S,T)) Σ PlEBosan)BBesns = S m(((1 (1-7)(1-6))
2m(7 + ----6)
Similar to the pure DP case could optimize for the optimal distribution of F((2) and F(L*) by solving a linear program (Theorem 5.2 in Steinke et al. (2023)) and apply Union bound as Corollary I to test for Hypothesis H.
34

---

E.3 Evaluations
We empirically evaluate the auditing performance of PANORAMIA under different relaxed level the generator for both the pure and approximate DP cases. We observe that the overall performance is similar to that of no relaxation. We observe smaller (which is analogous to a looser lower bound) as we increase the relaxation level, and in the more extreme cases where allow many failing events (e.g. when relaxation > le-3) we would fail to detect meaningful privacy leakage as غ = O.
Model
Generator Relaxation (7)
Ch:
{e + Cin
0 (no relaxation)
2.508
2,066
le-5
2.507
2.065
R
le-4
2,499
2.059
le-3
2.365
1.989
relaxation)
2,508
2,833
ResNet18
le-S
2.507
2,801
0.294
le-4
2.499
2,570
0.071
le-3
2.365
1,280
0 (no relaxation)
2.508
3.661
1.153
le.5
2.507
3.658
1.151
ResNet18 = 15
le-4
2.499
3.628
1.129
le:3
2.365
3,312
le:5
2,035
2,064
ResNet18 11-5
le-4
2,029
2.059
0.030
lo.3
1,904
1.988
0.084
le-5
2,035
2.765
0.730
RosNet18 == 10,6 =1o-5
lc4
2.029
2,541
0,512
le:3
1.904
1.271
le:3
2,035
3.654
1.619
ResNet18 15,6
le.4
2.029
3.625
1.596
le-3
1.904
3,308
1.404
Table 11: Privacy audit of ResNet18 under different values of e-DP and (e, 5)-DP, using PANORAMIA with
different level of generator relaxations (7), testing over range of recall from to 0.5 without applying union bound.
F Extended Related Work
Most related privacy auditing works measure the privacy of an ML model by lower-bounding its privacy loss. This usually requires altering the training pipeline of the ML model, either by injecting canaries that for worst-case memorization Jagielski et al. (2020); Nasr et al. (2021). MIAs are also increasingly used in privacy auditing, estimate the degree of memorization of member data by an ML algorithm by resampling the target algorithm M to bound P(M((n)
outliers Carlini et al. (2019) or by using data poisoning attack mechanisms search
P(M[out) Jayaraman & Evans (201). The auditing procedure usually involves searching for optimal neighboring datasets D, D' and sampling the DP outputs M(D') to get a Monte Carlo estimate of e. This approach raises important challenges. First, existing search methods for neighboring inputs, involving enumeration symbolic search, are impossible to scale to large datasets, making it difficult to find optimal dataset pairs. In addition, Monte Carlo estimation requires up to thousands of costly model retrainings to bound with high confidence. Consequently, existing approaches for auditing ML models predominantly require the re-training of ML models for every (batch of) audit queries, which is computationally expensive et al. (2022); Lu et al. (2023). This
in large-scale systems Jagielski et al. (2020); Ganella-Béguel makes privacy auditing computationally expensive and gives estimate by averaging over models, which might not reflect the true guarantee of a specific pipeline deployed in practice.
Nonetheless, improvements to auditing have been made in variety of directions. For example, Nasr et al. (2023) and Maddock et al. (2023) have taken advantage the iterative nature of DP-SGD, auditing individual steps understand the privacy of the end-to-end algorithm. Andrew et al. (2023) leverage the fact that analyzing MIAs for non- member data does not require re- re-running the algorithm. Instead, it is possible to re-sample the non me-mebber data point: if the data points are i.i.d. from asymptotically Gaussian distribution with mean zero and variance 1/d, this enables a closed-form analysis of the non- non-member case.
Recently, Steinke et (2023) proposed a novel scheme for auditing differential privacy with O(1) training rounds. This approach enables privacy audits using multiple training examples from the
35

---

same model training, if examples are included in training independently (which requires control over the training phase, and altering the target model). They demonstrate the effectiveness of this new approach on DP-SGD, in which they achieve meaningful empirical privacy lower bounds by training only one model (the strongest results are achieved by including canaries in the training set though, which is not possible in our setting), whereas standard methods would require training hundreds of models. Our work builds closely on the theory from Steinke et al. (2023), but introduces baseline model to account for distribution shifts. The key difference that enables our work to account for member/non-membbr distribution shifts lies how we create the audit set in our privacy game (Definition 2). In Steinke al. (2023), the audit set is fixed, and data points are randomly assigned to member or non-menee non-member by a Bernouli random variable S. Members are actually used in training the target model f, while non members not (so assignment happens before training). In our framework, we take a set of known members (after the fact), and pair each point with a i.i.d. from the generator distribution). We then flip S to sample which data point of each pair will be shown to the "auditor" (MIA/baseline) for testing, thereby creating the test task of our privacy measurement. When replace our generated data with in-distribution independent non-members (PANORAMIA RM:RN), we exactly enforce the same independence as Steinke et al. (2023) (and as result have some member data by drawing our auditing game after the fact. The difference in analysis accounts for distribution shifts when our non- members are generated.
non-memher(generatnee
and is lower-bound e), except that we "waste"
The field of MIA without direct connections to privacy leakage measurement has also seen a lot of recent activity, with new proposals improve the strength of MIAs in various settings (Carlini et al., 2022h; Nasr et al., 2019; Suri et al.; Zarifzadeh et al., 2023). While it is not the focus of this paper, an interesting avenue for future work is port ideas from these new MIAs to improve PANORAMIA's MIA and baseline. Such transfer of MIA progress could lead to more powerful measurements with PANORAMIA. More closely related to our proposal, a recent set of works Das et al. (2024); Duan et al. (2024); Meeus et al. (2024) MIAs on foundation models has observed that current evaluations limited due a lack of availability of non-member data. Indeed, foundation models typical include all known data at the time of their training, and there is no well known public set of in in-iistribution non-member data. Evaluation tasks typical rely on more recent data, which is known to consist in non-members, but suffers from distribution shifts. Das et al. (2024); Duan et al. (2024); Meeus et al. (2024) all show that such a shift invalidates MIA evaluations by showing that "blind attacks" (i.e.. our baselines) perform better than proposed MIAs. One could apply our proposed framework to such a setting: when the baseline in our framework performs better than the MIA, PANORAMIA returns a privacy loss measurement of zero instead of misleading measurements of MIA performance. We thus believe that PANORAMIA is an important step towards theory for rigorous evaluations of membership inference for MIA models.
36

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and Introduction sections highlight the motivation and contribu- tions we make; We provide, to the best of our ability, comprehensive experimental results in Section to justify our claims and contributions. Section 4 provides the theory behind our experiments, whilst Appendix details proofs.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss these in Section 4.2 and Section of the paper.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, we present sections assumptions, theory, and formal proofs in 3, 4 and Appendix B, E provides respectively.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We attach the code as supplementary material in a zip file and also include the main experimental details in Section 5 well as detailed experimental setup, hyperpa- rameters, compute details, and model architecture details along with other full necessary information in Appendix C to ensure reproducibility. Section 3 also includes the algorithm as well as the system design diagram detailing our privacy measurement framework. We also updated the paper and experiments according to the reviewers' helpful feedback.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We specify the details of the code we release the zip file we attach. The code is primarily the PANORAMIA privacy measurement pipeline. We include structured files specifying run instructions, licenses etc. Training details are within the code itself and also highlighted in Appendix C.
Justification:
Now we also updated the paper with an open source github code link given under the experimental section of the main paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Section and Table give key experimental details, whilst the full experimental setup is highlighted in Appendix C, as well as the zip file of code attached.
7. Experiment Statistical Significance
37

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We repeated all experiments that involved training data resampling 5 times, and repeated experiments that involved test data resampling over 10 times. All our figures and plots in the core body of the paper report results with enror bars, with all results reported at a 95% confidence interval. We also report the standard deviation in the tables in Section 5.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on compute resources for each modality in Appendix We also updated compute times run the PANORAMIA pipeline in C for each data modality.
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / http///eerrpp..eeee http///llc//cccccciieeeeee//eeee
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and conform to them in every respect.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We discuss these in Section 6; overall, our approach only help model providers and owners to assess the privacy of risk of their data and is not applicable as privacy attack. As consequence, we believe that any impact from our proposed work is likely to be positive.
Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer:[NA].
Question:
Justification: Our paper poses no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: All papers, code, and asset owners and authors have been properly cited the best our knowledge.
New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We specify the details the code we release the zip file we attach. The code is primarily the PANDRAMIA privacy measurement pipeline. We include structured files specifying run instructions, licenses, etc. Training details are within the code itself and also highlighted in Appendix C.
38

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing, nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Ans]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
39


Reviews:
[{"review_id": "yMz73MU9kD", "reviewer": "Reviewer_CsXP", "summary": "The authors propose a privacy auditing technique that utilizes partial access to member data to generate synthetic non-member data, which in-turn is used to train a meta-classifier that can be used to empirically measure privacy leakage relating to record membership. The authors evaluate their technique on models as large as GPT2 and find correlation between audit scores and the expected leakage from models.", "questions": "- Figure 3a suggests an ordering of 100 > 20 > 50 in terms of leakage for high-recall region, whereas numbers in Table 1 suggest that the trends should be like 100 > 50 > 20 if the proposed method is indeed measuring what it is supposed to. Why is that so?\n\n- L480-481: this means the \"labeler\" is much weaker and might be generating incorrect labels more often? Are there any numbers for what the actual test performance of this labeler is?", "limitations": "There are a few statements that currently serve as justification for limitations (see above) but should just be posed directly as base assumptions. Apart from that (and some potential shortcomings in models used for evaluation), most limitations are already stated in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- L226-228: Good to include negative results!\n- Having access to non-members is often taken for granted (especially when the auditor is also the model trainer), but in most third-party cases getting good-quality non-member data that is not significantly different from member distribution is hard. The methods proposed in this work can be useful there, with a generator and discriminator that can help ensure any differences in members and non-members (and subsequent MIA performance) do not arise from distributional differences. \n- The paper is well written and structured, and experiments are thorough, spanning quite a lot of models and modalities.", "weaknesses": "- As an auditor, access to validation/test data used in model training would not be a far stretch- why not use the non-member set as used in standard MIA setup (validation/test data). What is the added benefit from this extra step of generating synthetic non-members?\n\n- Regarding the contributions, (1) and (2) has already been explored by [1, 2]. While the method in [1] does not currently support large models/datasets like CIFAR10, it would be nice to highlight differences here. If you indeed have knowledge of a decent-chunk size of members and non-members, I'd imagine you could do something better like [2], and other related methods.\n\n- I have some concerns over the dependency on how well the baseline in/out distribution detection system works. As a concrete example consider Figure 7(a, b) - even as a human I see a very clear difference in resolution of the generated images and find it hard to believe that the distinguisher does not work well here. Even a non-ML technique (that can work around with blurring) would work pretty well here. \n\n- Table 1: Accuracy of models here is not good enough, especially when using entire data, with very clear signs of overfitting. If using the entire data (and not half split, as in most MIA evaluations [3, 4] where even with half the dataset test accuracy is ~92%), should be able to train well-performing models that do not overfit so heavily. Please see [this resource](https://paperswithcode.com/sota/image-classification-on-cifar-10)\n\n- Figure 1: While the trends suggest that the proposed audit correlates with actual leakage, one could also argue that (100 - test accuracy) is also a useful privacy metric in this comparison given the correlation. The utility of the audit would be more apparent, then, when studying models that have **comparable** test performance, but have (by design) different leakage, perhaps focusing on moderate/large values of $\\epsilon$ with Differential Privacy-training.\n\n- L288-290: (1) and (2) are ambiguously enforced- if you have a large portion of train data, you can control process in most cases, and also obtain non-member data. As far as (3) is concerned, that is a compute-related constraint, not a threat model difference. The auditor could, for instance, use available data knowledge to train \"in\" models.\n\n- L331-334: I do not find the justification behind having access to partial member data convincing. While I am okay with just stating that \"this is a possible limitation, but okay to assume for an auditor\", but relating it to how it could be useful in situations like FL is not practical. For instance, here empirical experiments use more than 1/5th of the data; no FL training will have just 5 participants. This is closer to a specific case of distributed learning, or pure data aggregation. Even in such cases, the data distributions per client will be considerably different, as opposed to the experiments here which have uniform samples. \n\n## Minor comments\n\n- Please consider a more descriptive abstract. \"scheme\" here is not very descriptive\n- L121: \"We formalize a privacy game\" - this is standard membership inference evaluation and not a contribution of this work. Please rephrase to make this clear.\n- L493-494: Please provide direct distinguishing accuracies of these baseline classifiers for reference and easier understanding.\n\n### References\n- [1] Suri, Anshuman, Xiao Zhang, and David Evans. \"Do Parameters Reveal More than Loss for Membership Inference?.\" High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning.\n- [2] Nasr, Milad, Reza Shokri, and Amir Houmansadr. \"Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning.\" 2019 IEEE symposium on security and privacy (SP). IEEE, 2019.\n- [3] Carlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.\n- [4] Zarifzadeh, Sajjad, Philippe Cheng-Jie Marc Liu, and Reza Shokri. \"Low-cost high-power membership inference by boosting relativity.\" (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. We start with answers to the two explicit questions, before addressing two of the weaknesses listed that we believe stem from a miscommunication or misunderstanding.\n\n**Questions:**\n\n**Q1 - Figure 3a suggests an ordering of 100 > 20 > 50 in terms of leakage for high-recall region, whereas numbers in Table 1 suggest that the trends should be like 100 > 50 > 20 if the proposed method is indeed measuring what it is supposed to. Why is that so?**\n\nIn our experiments on all data modalities and models, the precision values that lead to the highest lower-bounds $c$ or $c + \\epsilon$ are achieved at fairly low recall but for different recall values (especially for images). \nFor this reason, when training the baseline and MIA models, we use a validation set to select hyper-parameters and training stopping time that maximize the lower bounds (effectively maximizing the precision in the lower range of recall values) on the validation set. \nWe will clarify this in Appendix C. \nThis is why for some instances like pictures, MIA models have the wrong order at high recall values.\nWhile they could probably be tuned to achieve higher precision and be ordered correctly, we do not do it since that regime is not where the highest lower-bounds happen. \nIndeed, we can see in both Figure 3 and associated results in Table 1 that in the recall regions at which the lower-bounds are maximal, the order is as expected (100 > 50 > 20).\n\n**Q2 - L480-481: this means the \"labeler\" is much weaker and might be generating incorrect labels more often? Are there any numbers for what the actual test performance of this labeler is?**\n\nThe labeler used to train the helper model can indeed be weaker and generate wrong labels. \nFor instance, our labeler for image data has 80.4\\% test accuracy on the CIFAR10 real data test set.\nThe rationale for this approach is to augment the baseline with a model providing good features (here for generated data) to balance the good features provided to the MIA by $f$ outside of the membership information. \nIn practice, the labeled generated data seems enough to provide such good features, despite the fact that the labeler is not extremely accurate.\nWe studied alternative designs (e.g., a model trained on non-member data, no helper model) in Appendix D.1, Table 5, and the helper model trained on the synthetic data task performs best (while not requiring non-member data, which is a key point).\n\n**Other important clarifications:**\n\n**Weakness 1 - [W]hy not use the non-member set as used in standard MIA setup (validation/test data). What is the added benefit from this extra step of generating synthetic non-members?**\n\nWe do study this setup in Fig. 6, comparing a MIA using up to the full test data and our approach using generated data. \nOn CIFAR10 we can observe that, for the same non-member dataset size, using real data performs better (when using an ML-based MIA instead of a loss-based attack in this case).\nHowever, generated data enables us to use more data points for training and evaluating the MIA (and baseline), leading to larger measurements of privacy leakage. \nThis is true for both regular and DP models (Fig. 7a and 7c), despite the fact that the CIFAR10 test set is fairly large compared to the training set (20%) and such a large portion of training data may not be kept for testing in general.\n\nGoing in the same direction, a very recent work made public after our submission [1] shows that a similar member/non-member distribution shift issue happens when measuring privacy leakage from foundation models.\nIn that case, the models are trained on vast amounts of data, and there is no known non-member data from the same distribution. \nAs a result, MIA benchmarks use out-of-distribution non-member data (e.g., more recent datasets). \nThus, using generated data might yield better non-member data. \nAdditionally, our theory applies to both real and synthetic non-members data. Consequently, we believe that our approach may be an interesting building block for the setting described in [1].\n\nFinally, we believe that the generator opens an interesting design space, in which one could try to optimize generated data for audit quality. We have not yet explored this avenue of research, but we believe that it is an interesting future work.\n\n**Weakness 3 - I have some concerns over the dependency on how well the baseline in/out distribution detection system works. As a concrete example consider Figure 7(a, b) - even as a human I see a very clear difference in resolution of the generated images and find it hard to believe that the distinguisher does not work well here. Even a non-ML technique (that can work around with blurring) would work pretty well here.**\n\nThis is a great point, thank you. We apologize as this is a miscommunication on our end: our CelebA experiments are using a $128 \\times 128$ resolution, thus our generative model generates images at that resolution.\nIn Figure 7a though, we wrongly showed full resolution $218$ x $178$ images, hence the resolution difference. \nWe fixed this figure to display the real images as we use them, and included it in the associated PDF answer (Fig. 1), for which we can see that the difference in resolution disappeared.\nAlso note that the generator only needs to generate *some* good images, enough that it is hard to detect real members with high precision. \nThis is because we play a one-sided member-detection game, as explained in Remark 3 ll. 137-144.\nOf course, it is always possible that much stronger baselines exist, although we did spend quite a bit of effort and time on making them as good as we could (see details in Appendix D of the submission).\n\n[1] Das, Debeshee, et al. \"Blind Baselines Beat Membership Inference Attacks for Foundation Models.\" 2024. https://arxiv.org/abs/2406.16201", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k0G4K1hxCv", "reviewer": "Reviewer_55Rd", "summary": "The paper proposes a novel way for privacy auditing of machine learning models through MIAs. The framework they propose, panoramia, aims to audit the privacy of a ML model post-hoc (so with no control over the training procedure), and with access to a known member subdataset. The method first consists of training a generative model on the known subset of members, which is used to then generate synthetic data samples from the same distribution. These synthetic points are then used as non-members, which combined with the known member dataset allows to fit and apply an MIA on the target model. Importantly, authors recognize that there might be a distribution shift between members and synthetic non-members, so they fit a classifier without leveraging the target model as a baseline. \n\nNext, they use the difference between the MIA performance (thus using the target model) and this baseline to estimate the privacy loss. Authors provide the formula (and add proof in appendix) to compute a value of epsilon approximating a lower bound on epsilon-DP. \n\nThey further apply the privacy auditing to three kinds of ML models (image classification, language modeling, and tabular data classification). Authors consider models with varying degree of overfitting, DP training, and increasing the amount of member data available to the auditor.", "questions": "- I understand how c-closeness allows to estimate the quality of the used generator. However, what I struggle to understand, is what happens when you develop a generator that is perfect (c=0) which just samples randomly from the known member subset. Then, a baseline classifier would not be able to distinguish members from non-members, and nor will an MIA, leading to a privacy estimate that will be far off. In less extreme cases, the generator might be slightly overfitted and indeed generate non-members very similar to members, which might also impact the MIA performance. Am I correct that this could have a significant impact on the validity of the procedure? And if so, how would you address the concern? \n- More generally, Panoramia largely depends on a good generator and a baseline MIA. Can authors elaborate on the associated limitations? For instance, can good generators can be developed across all use-cases (smaller datasets, data modalities etc)? And how should a baseline performance be developed or evaluated to be used as part of the panoramia framework?\n- I understand that the method authors provide does not give a formal lower bound for the privacy loss, but rather a proxy for it. In practice, if indeed a hospital as part of an FL setup would like to assess the privacy leakage incurred by their data, why would they opt to compute an estimate for epsilon using your method? Instead, they could for instance generate non-members in the same way as panoramia and just quantify the MIA performance with an AUC or TPR at low FPR compared to a baseline. In general, authors should further motivate the relevance of a proxy for a lower bound privacy loss in practice. \n- To further motivate post-hoc privacy auditing (without any control of the training data), I wonder if it makes sense to also emphasize the context of generative AI models such as LLMs? These models are increasingly trained on all the data model developers can acquire so  the absence of non-member data is very real in practice. \n- Can authors add a related work section?", "limitations": "Authors are very clear that their method only provides a proxy for the lower bound of epsilon and thus carefully caveat their method. However, can authors elaborate on the limitations associated with the development of both a generator and a baseline, both of which seem fundamental for panoramia. Currently, their implementation seems rather ad-hoc than adequately discussed for wider applications.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Originality: The paper introduces a way to audit the privacy of ML models post-hoc, without any control of the training data, which is novel. They cite and position themselves correctly to relevant prior work such as Steinke et al. \n- Quality: The proposed method is technically interesting, formally supported and evaluated extensively across data/model modalities. \n- Clarity: NA\n- Significance: The paper proposes a way to compute an approximation for the lower bound on epsilon, to audit ML models post-hoc, which is technically interesting.", "weaknesses": "- Originality: Authors should include an appropriate related work section, touching on other privacy auditing techniques and potentially other (post-hoc) membership inference attacks. \n- Quality: The proposed method strongly depends on the quality of a generator and the baseline MIA, the impact and limitations of which can be further explored (see questions/limitations). \n- Clarity: I find that the paper's clarity can be improved significantly. Especially the results section (tables, text) is quite notation heavy and hard to follow what everything refers to. \n- Significance: While technically interesting, the relevance of a technique to compute a proxy lower bound privacy loss in practice needs more compelling motivation (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. \n\n**Q1: What happens when you develop a generator that is perfect (c=0) which just samples randomly from the known member subset.**\n\nIn that case (a highly overfitted generator), the baseline and MIA will output $c _{lb} = \\\\{\\epsilon+c\\\\} _{lb} = 0$ (though technically $c \\neq 0$, since $c$-closeness quantifies the distance between the generator's distribution, and the distribution of members, not the specific member data points). This happens mainly because both the MIA and baseline detect leakage based on the same data, and assess the data in two ways: (1) the distributional difference in the data points themselves; and (2) for the MIA the difference in distributions of loss values the target model attributes to the member vs non-member data. In this case, our theory still applies but returns $\\tilde{\\epsilon} = 0$. Thus, we would not faultily ascribe privacy leakage to the target model, but the measurement would also fail to detect any real privacy leakage (one could see that the measurement is failing though, since $c _{lb} = \\\\{\\epsilon+c\\\\} _{lb} = 0$).\n\n**Q2: More generally, Panoramia largely depends on a good generator and a baseline MIA. Can authors elaborate on the associated limitations? [C]an good generators be developed across all usecases (smaller datasets, data modalities etc)? And how should a baseline performance be developed or evaluated to be used as part of the panoramia framework?**\n\nThe ability of the generator to capture well the data distribution is a key dependency of PANORAMIA and the reason why we have thoroughly evaluate our approach. In particular, in the tabular case, datasets typically contain a large number of ``average'' samples and a long tail of extreme values or rare classes. Thus in such a case, when the generated data does not capture the long tail, the baseline easily classifies all outliers as real data as no synthetic data points are similar to them. This leads to a strong baseline that is hard to beat for the MIA, which means that the privacy measurement then fails (though we can diagnose why). This is a limitation of our work that we discuss in details in App. D6 as a possible cause for the failure of the tabular data modality. In general, the baseline should be made as strong as possible to detect such failures of the generative model following the traditional ML best practices. For this reason, we have dedicated a lot of effort in designing and evaluating our baselines, including the helper models (S5.1 and App. D1).\n\nOur design and theory offer practical advantages as well:\n- As generative models improve, especially for smaller dataset sizes, so will our approach.\n- The generator opens an interesting design space, in which one could try to optimize synthetic data for audit quality. While we have not yet explored this design space, we leave it as an interesting avenue for future work.\n- Finally, our theory applies to other out-of-distribution non-member data, such as when using other datasets as non-members [1].\n\n**Q3: [T]he method authors provide does not give a formal lower bound for the privacy loss, but rather a proxy for it. [...] Why would [one] opt to compute an estimate for epsilon using your method? Instead, [one could] generate non-members in the same way as panoramia and just quantify the MIA performance with an AUC or TPR at low FPR compared to a baseline. [...] Authors should further motivate the relevance of a proxy for a lower-bound privacy loss in practice.**\n\nAs it is the consensus in the field, we believe that DP provides the best semantics to define and quantify this type of privacy leakage. \nThus, usually, privacy audits aim to provide a lower bound for the privacy loss. While we do not yet provide a lower bound, using DP semantics is still useful. For instance, [2] makes a convincing case that accuracy or AUC are not good metrics for privacy leakage. \nRather TPR at low FPR is better, but notice that we need to compare the MIA with the baseline, and their performance most revealing of privacy leakage happens at different FPR values (cf. Figures 3a and 4a, though in precision/recall terms). In this case, we cannot directly subtract TPR values. Our theory tells us: (1) which FPR we should chose and (2) how to scale the TPR at this FPR (by mapping it to a $c$ or $c + \\epsilon$ value) to make the values comparable between the baseline and the MIA.\n\nTaking a step back, we hope that our framework will be a stepping stone for further research, maybe enabling a proper lower-bound on $\\epsilon$ by measuring (or enforcing by construction) an upper-bound on $c$ (then $\\\\{c+\\epsilon\\\\} _{lb} - c _{ub}$ yields $\\epsilon _{lb}$). Though we do not know how to do it yet, this is a promising future work.\n\n**Q4: To further motivate post-hoc privacy auditing (without any control of the training data), I wonder if it makes sense to also emphasize the context of generative AI models such as LLMs? These models are increasingly trained on all the data model developers can acquire so the absence of non-member data is very real in practice.**\n\nThanks for this great point. Actually, a paper made public after our submission [1] highlights this exact issue. \nAs you mention, in this case, there is no known non-member data from the same distribution, MIA benchmarks use out-of-distribution non-member data (e.g., more recent datasets). Thus using generated data might yield better non-member data. Our theory applies equally to both real and synthetic non-members data and we believe that it may be an interesting building block for that setting.\n\n**Q5: Can authors add a related work section?**\n\nWe propose to add a more formal related work section to collect the closest works, and an extended one in appendix. \n\n[1] Das et al. \"Blind Baselines Beat Membership Inference Attacks for Foundation Models.\" 2024. https://arxiv.org/abs/2406.16201\n[2] Carlini et al. \"Membership inference attacks from first principles.\" S\\&P 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7e2qCSPexx", "reviewer": "Reviewer_kScF", "summary": "This paper proposes a novel privacy auditing procedure called PANORAMIA. The method works with a single model that uses all available training data, with no modifications to the training procedure, and with only access to a subset of the training data. This is achieved by synthetically generating non-member samples for auditing. The method produces an estimate of the $\\epsilon$ differential privacy parameter, which, however, is not necessarily a lower bound.\n\nSummary of the method given a model and its training data:\n1. Train a generative model on a subset of the training data.\n2. Use the remaining training samples as \"members\" for auditing.\n3. Generate an equal amount of \"non-members\".\n4. Split both into training and test sets.\n5. Fit a baseline classifier that distinguishes synthetic and real data on the training set (only input are samples).\n6. Fit a membership inference classifier on the training set (inputs are samples and model statistics).\n7. Use predictions on the test set and hypothesis testing to obtain a confidence interval for quantities of interest.\n\nThe hypothesis is of the form \"The generative model is $c$-close to the true data distribution and the training procedure is $\\epsilon$-DP\", where $c$ is a distance measure defined in this paper. The baseline predictions are used to reject the first claim, and the membership predictions to reject the second one. Hence, the returned $\\epsilon$ is a lower bound on the true $\\epsilon$ only if the generative model is indeed at least $c$-close to the data distribution. The theoretical analysis relies on results from the (1) procedure by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) but adapts them to the relaxed setting in this paper.\n\nFor evaluation, the paper applies the new auditing procedure to various CNNs with and without DP guarantees on image data, small GPT2 models on text, and on tabular data. As baselines, the evaluation uses the (1) method by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) and real instead of generated members. PANORAMIA does not outperform the baselines, but achieves reasonably close results on image and text data.", "questions": "1. Might there be a way to use the membership classifier itself as a baseline, e.g., by somehow averaging over possible values for model statistics? This would avoid skewed scenarios where the MIA is better at detecting synthetic data than the baseline detector (even when ignoring membership signal).\n2. In Figure 6/Section 5.3, do all models use (approximately) the same number of training samples? If not, could there be confounding effects (e.g., if models trained on fewer samples leak more privacy)?\n3. Is there a path to extend this paper's method to always yield a lower bound on $\\epsilon$ (w.h.p.)? Or would this require radical changes?", "limitations": "The authors are very transparent about all limitations of their work and discuss them transparently.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper considers a practical and relevant setting for privacy auditing. In particular, training the model to audit using all training data is a big benefit: otherwise, the audit either uses a slightly different setting that uses fewer training samples, or one has to forfeit a potentially large amount of training data (which hurts utility). Additionally, the procedure works even when the auditor knows only part of the training data, which can be relevant (e.g., federated learning).\n- The experiments are broad (considering both DP and non-DP training, and different degrees of overfitting), sound, and the conclusions are convincing. Although the results are weaker than for existing methods, those existing methods use a less appealing setting. A particularly convincing result is that PANORAMIA with synthetic non-members often comes close to the same method but using real non-member samples.\n- The paper is overall structured well and the auditing procedure is laid-out clearly. The authors manage to explain the auditing results well, despite their complex interpretation. I also appreciated that Algorithm 1 collects all parts of the method into one place, including the calculations to obtain confidence intervals.", "weaknesses": "- The considered privacy semantics are partially misleading and could be made more clear.\n\t- The presented auditing game is incomplete; Definition 2 only describes how samples are selected but should also include the goal and actions of the adversary.\n\t- The paper states (as a benefit) that PANORAMIA audits a target model, not the algorithm (L46--47). However, the results of the procedure are DP parameters, and DP is always a property of an algorithm (not model). If the considered semantics are meant to be different from DP, they should be made more clear and discussed.\n\t- Similarly, the paper claims that PANORAMIA does not require worst-case canaries because it audits the target dataset (via a model). However, this ignores that certain samples in the training data might be more vulnerable; hence, auditing should still consider the worst-case sample *in the training data* (especially since privacy is averaged over the dataset). This might be an explanation why the reported \"lower bounds on $\\epsilon$\" are still far from tight in Figure 6c (as is the prior (1) procedure in a white-box setting).\n- Relatedly, the definition of c-closeness (Definition 3) might not require a generator to capture the tail of the data distribution. However, this tail contains outliers that are often particularly vulnerable to privacy leakage. Hence, ignoring such outliers might significantly underestimate the $\\epsilon$ lower bound. I would have appreciated a more thorough discussion of this limitation (or a clarification).\n- The paper could be more explicit about whether PANORAMIA is intended to be a procedure to be used in practice, or an important stepping stone towards more practical procedures. Right now, PANORAMIA achieves worse results than the existing (1) method in a white-box setting, and requires training of a strong synthetic data generator (which might be non-trivial). Nevertheless, I believe this paper is still relevant from a conceptual perspective as a path for future work.\n- There are minor notation issues in Proposition 1 and 2: the statement mentions only $T$ and $t$, but the inequality uses $T^b$ and $t^b$ (or $T^a$ and $t^a$). Additionally, the left-hand sides both take probabilities over $T^b$/$T^a$ but simultaneously condition on the respective random variable. Fixing those points makes the propositions clearer. Also, the last sentence on L148 seems misplaced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review. Hereafter, we start with answers to the questions, before addressing other weaknesses listed that we believe stem from a miscommunication.\n\n**Questions:**\n\n**Q1: Might there be a way to use the membership classifier itself as a baseline, e.g., by somehow averaging over possible values for model statistics? This would avoid skewed scenarios where the MIA is better at detecting synthetic data than the baseline detector (even when ignoring membership signal).**\n\nThanks for this interesting suggestion. If we could somehow capture a distribution of  \"non-member losses\" for this data point, we might be able to marginalize out the dependency of the MIA on the target model and extract the input-dependent part.\nA key challenge with such a design would be to make sure that the resulting baseline is indeed as strong as it can be as well as that input differences do not ``leak through'' the target model loss for member data. \nIn our experiments, the baseline is often better at picking up signal from the data point than the MIA model is (hence the cases, like in Figure 5 or some of the tabular data plots in the appendix, in which the baseline outperforms the MIA), which is in a sense the opposite problem.  We believe it to be an interesting future work, and in general, any progress on discriminative models and MIAs will directly plug-into and benefit our approach.\n\n**Q2: In Figure 6/Section 5.3, do all models use (approximately) the same number of training samples? If not, could there be confounding effects (e.g., if models trained on fewer samples leak more privacy)?**\n\nExperiments for all data modalities in S5.3 show the effect of varying test set sizes on the performance of both Panoramia and O(1) while keeping all other experimental variables, including training dataset size constant across all the models. This is to ensure the results we observe are primarily due to varying the test dataset size while keeping everything else controlled. We mention the training details for the models in Appendix C. In addition, we also show, in Appendix D4, how varying training data for a fixed test set size impacts both Panoramia and baseline performance. \n\n\n**Q3: Is there a path to extend this paper's method to always yield a lower bound on (w.h.p.)? Or would this require radical changes?**\n\nGreat question! This is plausible, and we hope that we, or another group, will figure out how in the future. The most straight-forward way to get a proper lower-bound on $\\epsilon$ in our framework is to figure out how to measure (or construct) an upper-bound on $c$, as then $(c+\\epsilon) _{lb} - c _{\\text{ub}}$ yields $\\epsilon _{lb}$. Getting an upper-bound on $c$ in the general case seems challenging though. There might be a way to leverage DP to do this by construction (since DP bounds the gap between distributions), though we haven't figured out how to do it yet. This is definitely a promising avenue for future work.\n\n**Other important clarifications:**\n\n**Weakness 2 - Relatedly, the definition of c-closeness (Definition 3) might not require a generator to capture the tail of the data distribution. However, this tail contains outliers that are often particularly vulnerable to privacy leakage. Hence, ignoring such outliers might significantly underestimate the lower bound. I would have appreciated a more thorough discussion of this limitation (or a clarification).**\n\nIn cases where the generative model is unable to effectively capture the long tail distribution of the real dataset it is modeling, it becomes easier for a baseline classifier to distinguish between real and synthetic non-members, and hence to tell the two distributions apart ($c$-closeness, like pure-DP, does not allow such discrepancies in the tail, though our $(c, \\gamma)$-closeness relaxation in appendix E does). In practice, we do observe this phenomenon, especially with tabular data. We highlight this limitation via results on the tabular setting in Appendix D6 (we will add a pointer to the paper for clarity). In such a case, we show that the baseline is as strong or stronger than the MIA and Panoramia is unable to effectively detect any privacy leakage from the respective target model.\n\n**Weakness 3 - The paper could be more explicit about whether PANORAMIA is intended to be a procedure to be used in practice, or an important stepping stone towards more practical procedures. Right now, PANORAMIA achieves worse results than the existing (1) method in a white-box setting, and requires training of a strong synthetic data generator (which might be non-trivial). Nevertheless, I believe this paper is still relevant from a conceptual perspective as a path for future work.**\n\nThank you for the helpful feedback. We will clarify that we believe that our work is an important step towards solving the privacy measurement setup that we tackle, namely measurements without control of the training process and with distribution shifts between members and non-members. The end-goal for future work is a full-fledged approach (that provides a proper lower bound) with potential improvements over other approaches (e.g., optimizing the generator to improve the audit). However, we also believe that our framework can already be useful, for instance for providing improved measurements with more data (Fig. 6a); or to tackle privacy measurements in models for which there are no known in-distribution non-members, which as recently pointed out in [1] suffers from the same distribution shifts we tackle by using out-of-distribution non-members (our theory would apply to such a case, though whether the generative model part can help is still an open question).\n\n[1] Das, Debeshee, et al. \"Blind Baselines Beat Membership Inference Attacks for Foundation Models.\" 2024. https://arxiv.org/abs/2406.16201", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5Mntu2Nmp", "reviewer": "Reviewer_Z4qp", "summary": "The paper proposes a method to estimate a lower bound on the (pure) Differential Privacy parameter $\\epsilon$ for already trained machine learning models using a post-hoc empirical evaluation based on conducting Membership Inference Attacks. Building on the analysis by Steinke et al., 2023, for auditing DP with $O(1)$ training runs, this paper tweaks the setting by using synthetically generated canary data points that closely resemble the training data population, rather than inserting or omitting pathologically crafted canaries that degrade the ML model's performance. This approach is useful because if the canary distribution and the training distribution are (nearly) identical, an already trained model can be considered as the output of the training algorithm on a random partition of the combined training and canary datasets. This means any post-hoc auditing analysis based on membership inference on a randomly and independently selected training dataset yields a valid lower bound on $\\epsilon$. To get such a canary distribution, the authors suggest training a synthetic data generation model to create a canary dataset to get a distribution similar to the population. Additionally, the authors extend Steinke et al.'s analysis to situations where the synthetic data distribution is $c$-close to the true population in a DP-like divergence, although the estimator does not technically yield a lower bound in this case. Through extensive empirical evaluations, the paper demonstrates that the proposed estimation technique is useful as it provides reasonable values that approximate the DP lower bounds.", "questions": "- How is the helper model used in the baseline trained?\n\n- The lexicographic order in equation (2) appears to be essential for Corollary 2 and seems to be aimed at ensuring something like $\\epsilon > c$ to hold. Could the authors provide more details on this?\n\n- How exactly does the formulation in the paper differ from the setting introduced by Steinke et al., 2023, specifically in regards to the independence assumptions and the probabilistic dependencies between concerned random variables?", "limitations": "The authors have acknowledged some limitations in the paper. There do not appear to be any negative societal impacts associated with this work. I encourage the authors to address the issues and questions raised in this review.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper focuses on an important problem and has the following merits.\n\n- The paper proposes an auditing method that seeks to eliminate the need of altering anything about the training process of ML models, thus allowing post-hoc auditing. \n\n- The paper proposes using synthetic data that resemble the training data instead of adversarially created canaries can reduce the drop in model performance while offering useful DP estimations.\n\n- Empirical evaluations suggest that the estimator correlates with the DP upper bounds and can help identify situations of high privacy leakage.\n\n- The appendix provides an attempt to extend the budget estimator for $(\\epsilon, \\delta)$-DP.", "weaknesses": "- I'm not entirely certain that the analysis presented in Proposition 2 is correct. In the proof of Proposition 2 (lines 433 and 434), the authors use the fact that the model $f$ is $\\epsilon$-DP with respect to the Bernoulli random variables $S$. But in Algorithm 1, we see that $f$ was trained on $D_{f}$ in phase 1, which is independent of $S$ sampled in phase 2. So, the model $f$ is independent of the selection $S$ (i.e., $f \\bot S$). In other words, the equation after line 434 should evaluate to 1, as given $x_i$, the prediction $f(x_i)$ remains the same whether $S_i = 0$ or $S_i = 1$. Perhaps Algorithm 1 requires one training run like Steinke et al. where the model $f$ is trained on $X$ as described in line 2 of Algorithm 1? If that is the case, then the claim that the auditing does not require retraining becomes invalid. If I'm mistaken, could you explain Proposition 2 further?\n\n- In line 196, the authors mention that Algorithm 1 (lines 4-7) does a sweep over all the recall values of the attack, and they adjust the overall significance-level or the $p$-value by taking a union bound over all equation (1) with the significance level discounted to $\\beta \\leftarrow \\beta / m$. The value of $m$ in the experiments ranges from 500 to 30,000. For such values,  assuming $\\beta \\leq 0.05$, the sweeping over all recall values requires $\\sqrt{\\log(m/\\beta)} \\approx 2.4$ whereas when the level of recall is predetermined, the factor in equation (1) is only $\\sqrt{\\log(1/\\beta)} \\approx 1.15$. So, I'm not sure if doing a sweep over recall values will give larger DP estimate values.\n\n- Use of synthetic data from a distribution that is $c$-close (with a small $c$) can make it difficult for the membership inference attacks to work well. If the goal is auditing DP, perhaps the higher MIA precision for a given recall outweighs the drop in ML model's performance. \n\n- The figures haven't been explained very well. In particular, the theoretical maximum precision in Figure 3 and Figure 12 and the empirical maximum value in Figure 4 aren't clear.\n\n- The paper only studies the problem of estimating pure DP and does not present an operationalizable algorithm for $(\\epsilon, \\delta)$-DP, although some results along these lines are motivated in the appendix. Additionally, the obvious weakness (which the authors acknowledge) is that their method does not technically provide a lower bound for $\\epsilon$-DP.\n\n### Minor Points\n\n- Intuitively, when $\\mathcal{G} = \\mathcal{D}$ or in the Real-Member;Real-Nonmember (RM;RN) case where the non-members follow the same distribution, I think the no-retraining-needed argument could work. This would involve modifying Algorithm 1 and reworking Proposition 2 by (A) assuming entries in $D_f$ and $D_G$ are i.i.d. from the same distribution and (B) setting $S$ according to the original train-test split instead. On the other hand, when $\\mathcal{G} \\neq \\mathcal{D}$, I'm not sure if such an argument can be made to work, at least not trivially.\n\n- It's not clear how the mechanism $B(S, X) = \\\\{b(x_1), \\cdots, b(x_m)\\\\}$ in Proposition 1 incorporates the helper model mentioned in Section 5.1 used in the experiments.\n\n- Algorithm 1 (lines 4-7) does not seem to reflect the $p$-value adjustment discussed in lines 198-200. Perhaps the authors might have overlooked this $p$-value adjustment in the experiments as well?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful review.\n\n**How is the helper model used in the baseline trained? (Q1)**\n\nThe helper model has the same classification task and architecture as the target model. To train it, we generate separate sets of training and validation data using our generator. For image data, the synthetic samples do not have a label. We thus train a labeler model, also a classifier with the same task, on the same dataset used to train the generator. We use the labeler model to provide labels for our synthetic samples (training and validation sets above). We train the helper model on the resulting training set, and select hyperparameters on the validation set. We give more experimental details for each modality in App. C (l. 473-474, 545-546, 567-568, 586-592). App. D1 further highlights the various architectures of the helper model we tried (l. 638, Table 5), including training the helper model on real non-member data. Table 5 shows the helper model trained on generated data gives the strongest baseline (highest $c_{lb}$).\n\n**The lexicographic order in Eq. (2) appears to be essential for Corollary 2 and seems to be aimed at ensuring something like $\\epsilon > c$ to hold [...]? (Q2)**\n\nThe lexicographic order in Eq. (2) is not to ensure $\\tilde{\\epsilon} > c$ (as a counter-example, $(c_1=3, \\epsilon_1=1) \\leq (c_2=3, \\epsilon_2=2)$). The technical step relying on this order is the construction of the confidence interval (Corollary 2, l. 187-188) based on hypothesis tests from Prop. 1 and 2. \n\nThe reason for this order with $c$ first is that Prop. 2 asks: ``if the generator is $c$-close, is the target model $\\epsilon$-DP?'' We thus must reject any hypothesis with a $c$ we can disprove based on data, before computing $\\epsilon$ based on a plausible $c$. Without this order (e.g. $\\epsilon$ first or $c + \\epsilon$), we would not necessarily compute $\\epsilon$ based on a plausible $c$. We would assign some of the MIA performance to privacy leakage, when we know for a fact it comes from the generator being too far from the data distribution.\n\n**How exactly does the formulation in the paper differ from the setting introduced by Steinke et al., 2023 [...]? (Q3)**\n\nThe key difference lies in how we create the audit set. In Steinke et al., 2023, the audit set is fixed, and data points are randomly assigned to member or non-member by a Bernoulli random variable $S$. Members are actually used in training the target model $f$, while non-members are not (so assignment happens before training). In our framework, we take a set of known iid. members (after the fact), and pair each point with a non-member (generated iid. from the generator distribution). We then flip $S$ to sample which one will be shown to the ``auditor'' (MIA/baseline) for testing, thereby creating the test task of our privacy measurement.\n\nRegarding the independence assumptions, $S$ is independent of everything by construction, as in Steinke et al., 2023. In fact, if we replace our generated data with in-distribution independent non-members, we exactly enforce the same independence, except we waste some data by drawing our auditing game after the fact. Using generated data adds another complexity, as $S$ ``leaks'' through the actual data-point shown to the auditor (based on differences between the generator and member data distributions), which we address with the baseline model.\n\n**Correctness of Proposition 2. (W1)**\n\nWe hope Q3 above already helps resolve the misunderstanding about equation line 434 and Prop. 2. To expand: while $S$ is independent of $f$ ($S \\perp f$), they are not conditionally independent when conditioning on $X$ (where $X$ is the test set for the MIA/baseline), so $f \\not\\perp S | X$. This is because $X_i$ is either a member or non-member based on $S_i$, and the member is a data point on which $f$ was trained!\n\nHence, Eq. below l. 434 does not evaluate to $1$. Deliberately ignoring $S_{<i}, X_{<i}$ for simplicity, the Eq. reads as the ratio between: (numerator) the membership guess (a post-processing of $f$ and $X_i$) conditioned on the event that we guess for input $X_i$ which is a generated non-member $S_i=0$; and (denominator) the membership guess (still a post-processing of $f$ and $X_i$) conditioned on the event that we still guess on $X_i$ which is now a member of $f$'s training set. If $f$ is DP, the membership guess is a post-processing of a DP result on two neighboring datasets ($X_i$ wasn't in the training set vs. $X_i$ was in the training set), and hence obeys the inequality shown after l. 434. Note that collisions (where we have $X_i, S_i=0$ but $X_i$ is also in the training data) make the ratio $1$: the inequality is still true and the theory applies, but the privacy measurement has no power (see discussion on overfit generator in Q1 of reviewer 55Rd).\n\n**Adjusting significance level does not give larger DP estimate values. (W2)**\n\nWe adjust the significance level because we need to compare the highest lower-bounds implied by the MIA and the baseline. Notice on Fig. 3a and 4a that the highest precision values and implied lower-bounds can happen at different (unknown in advance) recall levels. If we were to fix a recall value in advance, we could get misleading results based on whether that value is closer to the best value for the baseline or for the MIA.\nWe thus have to make the comparisons at different recall levels.\n\nIn this case our tests from Prop. 1, 2 (which are at a fixed number of guesses) need a union-bound $\\beta \\leftarrow \\beta / m$ to be correct. We could select the maximum without a union bound as a heuristic, but in practice it barely changes the results. This is because both the baseline and MIA numbers are lowered in a similar way by their respective union-bounds, so the gap between the two remains very similar.\n\n**Auditing for $(\\epsilon, \\delta)$-DP algorithms. (W5)**\n\nWe do analyze the $(\\epsilon, \\delta)$-DP case in App. E2, and show results in Table 10 in App. E3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the thought-provoking reviews and suggestions! In our answers, we focus on key misconceptions regarding our paper and misunderstandings in the reviews. Hereafter, we summarize the most important points addressed:

- Potential baseline weakness (reviewer CsXP): this is an issue due to Figures 7(a, b) in which we wrongly put full-resolution member images while we work with a lower-resolution dataset for our experiments (hence with a generator outputting lower-resolution images). We fixed the figure (Figure 1 in the PDF attachment) to display the member images that we actually used.
- Technical question on the soundness of equation line 434 (reviewer Z4qp): we explain why this equation is the DP bound as we wrote, and why it does not evaluate to 1. We also clarify how our auditing game differs from (and resembles) the one from (Steinke et al., 2023), which is related to the soundness of the equation at line 434.
- Motivation for the approach (reviewers 55Rd, kScF, Z4qp), including: (1) why using DP is useful even without a lower-bound (i.e., it lets us compare the MIA and baseline on the same scale while being a good way to formalize and quantify privacy leakage); (2) the usefulness of our framework (e.g., the theory can help with other out-of-distribution non-member data, such as when using other datasets as non-members [1], our generator opens an interesting design space to maximize audit efficiency); and (3) that we believe that our current work might be a stepping stone to a more complete approach that yields a proper $\epsilon$ lower-bound.

We address all these issues and some of the other comments from each reviewer in the individual answers.

[1] Das, Debeshee, et al. "Blind Baselines Beat Membership Inference Attacks for Foundation Models." 2024. https://arxiv.org/abs/2406.16201

Meta-review: The paper presents a method for privacy auditing without access to real non-members by using generated examples as non-members. The paper builds a theoretical framework for addressing the auditing question approximately and presents experiments to test the method on a number of tasks in diverse domains (images, NLP, tabular data).

The reviews are generally positive, commending the paper on addressing a relevant and interesting problem and experiments covering diverse tasks, but raise concerns about correctness of certain results, lack of careful evaluation of how the quality of the generator impacts the results in practice, and the fact that the method does not produce strict lower bounds.

The only reviewer recommending rejection, Reviewer Z4qp, raised concerns about the correctness of some key results, which likely contributed to the negative review. The authors have addressed these in their response, but the reviewer has not reacted to the response. It would seem to me that the most serious concerns regarding the paper have thus been addressed although the review has not been updated to reflect this.

In light of these reviews, I believe the paper should be accepted.

I would however recommend the authors to consider the comments raised by the reviewers and add a more thorough evaluation and discussion of the impact of the quality of the generator to the results. It appears to me that the training protocol for the generators used in the experiments is really elaborate. This raises concern that it might be very difficult to replicate the strong results from the image and NLP models without equally arduous generator training.

Minor comments:
* The authors claim to use top-$p$ sampling in the NLP generator (e.g. line 224), but note on line 533 of the Appendix that they use parameter $p=1$ and $k=200$ for top-$k$. I believe this would be more correctly labeled as top-$k$, as top-$p$ filtering has no effect under these parameters.
* Unlike what they claim in the paper checklist response #8, the authors do not disclose the compute time required by the experiments.
Decision: Accept (poster)