PANORAMIA: Privacy Auditing of Machine Learning Models without Retraining

Authors: mishaal  kazmi
hadrien  lautraite
alireza  akbari
qiaoyue  tang
mauricio  soroco
tao  wang
sébastien  gambs
mathias  lécuyer
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 02:25:27

Abstract: We present PANORAMIA, a privacy leakage measurement framework for machine learning models that relies on membership inference attacks using generated data as non-members. By relying on generated non-member data, PANORAMIA eliminates the common dependency of privacy measurement tools on in-distribution non-member data. As a result, PANORAMIA does not modify the model, training data, or training process, and only requires access to a subset of the training data. We evaluate PANORAMIA on ML models for image and tabular data classification, as well as on large-scale language models.

Main paper: 
 introduction: 
 Training Machine Learning (ML) models with Differential Privacy (DP) Dwork et al. (2006) , such as with DP-SGD Abadi et al. (2016) , upper-bounds the worst-case privacy loss incurred by the training data. In contrast, privacy auditing aims to empirically lower-bound the privacy loss of a target ML model or algorithm. In practice, privacy audits usually rely on the link between DP and the performance of membership inference attacks (MIA) Wasserman & Zhou (2010) ; Kairouz et al. (2015) ; Dong et al. (2019) . At a high level, DP implies an upper-bound on the performance of MIAs, thus creating a high-performance MIA implies a lower-bound on the privacy loss. Auditing schemes have proven valuable in many settings, such as to audit DP implementations Nasr et al. (2023) , or to study the tightness of DP algorithms Nasr et al. (2021) ; Lu et al. (2023) ; Steinke et al. (2023) . Typical privacy audits rely on retraining the model several times, each time guessing the membership of one sample Jagielski et al. (2020) ; Carlini et al. (2022a) ; Zanella-Béguelin et al. (2022) , which is computationally prohibitive, requires access to the target model (entire) training data as well as control over the training pipeline.To circumvent these concerns, Steinke et al. (2023) proposed an auditing recipe (called O(1)) requiring only one training run (which could be the same as the actual training) by randomly including/excluding several samples (called auditing examples) into the training dataset of the target model. Later, the membership of the auditing examples are guessed for privacy audit. However, O(1) faces a few challenges in certain setups. First, canaries, which are datapoints specially crafted to be easy to detect when added to the training set Nasr et al. (2023) ; Lu et al. (2023) ; Steinke et al. (2023) , cannot be employed as auditing examples when measuring the privacy leakage for data that a contributor actually puts into the model, and not a worst case data point. This matches a setting in which individual data contributors (e.g., a hospital in a cross-site Federated Learning (FL) setting or a user of a service that trains ML models on users' data) measure the leakage of their own (i.e., known) partial training data in the final trained model. Second, O(1) also relies on the withdrawal of real data from the model to construct non-member in-distribution data. This is problematic in situations in which ML model owners need to conduct post-hoc audits, in which case it is too late for removal Negoescu et al. (2023) . Moreover, in-distribution audits require much more data, thus withholding many data points (typically more than the test set size) and reducing model utility. This brings us to the question: Given an instance of a machine learning model as a target, can we perform post-hoc estimation of the privacy loss with regards to a known member subset of the target model training dataset?Our contributions. We propose PANORAMIA, a new scheme for Privacy Auditing with NO Retraining by using Artificial data for Membership Inference Attacks. More precisely, we consider an auditor with access to a subset of the training data and introduce a new alternative for accessing non-members: using synthetic datapoints from a generative model trained on the member data, unlocking the limit on non-member data. PANORAMIA uses this generated data, together with known members, to train and evaluate a MIA attack on the target model to audit ( §3). We also adapt the theory of privacy audits, and show how PANORAMIA can estimate the privacy loss (though not a lower-bound) of the target model with regards to the known member subset ( §4). An important benefit of PANORAMIA is to perform privacy loss measurements with (1) no retraining the target ML model (i.e., we audit the end-model, not the training algorithm), (2) no alteration of the model, dataset, or training procedure, and (3) only partial knowledge of the training set. We evaluate PANORAMIA on CIFAR10 models and observe that overfitted models, larger models, and models with larger DP parameters have higher measured privacy leakage. We also demonstrate the applicability of our approach on the GPT-2 based model (i.e., WikiText dataset) and CelebA models. 
background and related work: 
 DP is the established privacy definition in the context of ML models, as well as for data analysis in general. We focus on the pure DP definition to quantify privacy loss with well-understood semantics. In a nutshell, DP is a property of a randomized mechanism (or computation) from datasets to an output space O, noted M : D → O. It is defined over neighboring datasets D, D ′ , differing by one element x ∈ X (we use the add/remove neighboring definition), which is D ′ = D ∪ {x}. Formally: Definition 1 (Differential Privacy Dwork et al. (2006)) . A mechanism M : D → O is ϵ-DP if for any two neighbouring datasets D, D ′ ∈ D, and for any measurable output subset O ⊆ O it holds that:P [M (D) ∈ O] ≤ e ε P [M (D ′ ) ∈ O].Since the neighbouring definition is symmetric, so is the DP definition, and we also have thatP [M (D ′ ) ∈ O] ≤ e ε P [M (D) ∈ O].Intuitively, ϵ upper-bounds the worst-case contribution of any individual example to the distribution over outputs of the computation (i.e., the ML model learned). More formally, ϵ is an upper-bound on the privacy loss incurred by observing an output o, defined as ln P[M (D)=o]P[M (D ′ )=o], which quantifies how much an adversary can learn to distinguish D and D ′ based on observing output o from M . A smaller ϵ hence means higher privacy.DP, MIA and privacy audits. To audit a DP training algorithm M that outputs a model f , one can perform a MIA on datapoint x, trying to distinguish between a neighboring training sets D and D ′ = D ∪ {x}. The MIA can be formalized as a hypothesis test to distinguish between H 0 = D and H 1 = D ′ using the output of the computation f . Wasserman & Zhou (2010) ; Kairouz et al. (2015) ; Dong et al. (2019) show that any such test at significance level α (False Positive Rate or FPR) has power (True Positive Rate or TPR) bounded by e ϵ α. In practice, one repeats the process of training model f with and without x in the training set, and uses a MIA to guess whether x was included. If the MIA has TPR > e ϵ FPR, the training procedure that outputs f is not ϵ-DP. This is the building block of most privacy audits Jagielski et al. (2020) ; Nasr et al. (2021) ; Zanella-Béguelin et al. (2022) ; Lu et al. (2023) ; Nasr et al. (2023) . Appendix F provides further context, and discusses other related work that is not directly relevant to our contribution.Averaging over data instead of models with O(1). The above result bounds the success rate of MIAs when performed over several retrained models, on two alternative datasets D and D ′ . Steinke et al. (2023) show that it is possible to average over data when several data points independently Phase 2 trains a MIA on a subset of member data and generated non-member data, using the loss of f on these data points. The performance of the MIA is compared to a baseline classifier that does not have access to f . Notations are summarized in Table 4 in Appendix A.differ between D and D ′ . Let x 1,m be the m data points independently included in the training set, and s 1,m ∈ {0, 1} m be the vector encoding inclusion. T 0,m ∈ R m represents any vector of guesses, with positive values for inclusion in the training set (member), negative values for non-member, and zero for abstaining. Then, if the training procedure is ϵ-DP, Proposition 5.1 in Steinke et al. (2023) bounds the performance of guesses from T with:P m i=1 max{0, Ti • Si} ≥ v | T = t ≤ P S ′ ∼Bernoulli( e ϵ 1+e ϵ ) m m i=1 |ti| • S ′ i ≥ v .In other words, an audit (MIA) T that can guess membership better than a Bernoulli random variable with probability e ϵ 1+e ϵ refutes an ϵ-DP claim. In this work we build on this result, extending the algorithm ( §3) and theoretical analysis ( §4) to enable the use of generated data for non-members, which breaks the independence between data points. 
panoramia: 
 Figure 1 summarizes the end-to-end PANORAMIA privacy measurement. The measurement starts with a target model f , and a subset of its training data D f from distribution D. For instance, D and D f could be the distribution and dataset of one participant in an FL training procedure that outputs a final model f . The privacy measurement then proceeds in two phases.Phase 1: In the first phase, PANORAMIA uses a subset of the known training data D G ⊂ D f to train a generative model G. The goal of the generative model G is to match the training data distribution D as closely as possible, which is formalized in Definition 3 ( §4). Using the generative model G, we can synthesize non-member data, which corresponds to data that was not used in the training of target model f . Hence, we now have access to an independent dataset of member data D in = D f \D G , and a synthesized dataset of non-member dataD out ∼ G, of size m = |D in |.Phase 2: In the second phase, we leverage D in and D out to audit the privacy leakage of f using a MIA. To this end, we split D in , D out into training and testing sets, respectively called D tr in , D tr out and D te in , D te out . We use the training set to train a MIA (called PANORAMIA in Figure 1 ), a binary classifier that predicts whether a given datapoint is a member of D f , the training set of the target model f . This MIA classifier makes its prediction based on both a training example x, as well as information from applying the target model f to the input, such as the loss of the target model when applied to this example loss(f (x)) (see §5, Appendix C for details). We use the test set to measure the MIA performance, using the precision at different recall values.Previous results linking the performance of a MIA on several data-points to ϵ-DP bounds rely on independence between members and non-members. This intuitively means that there is no information about membership in x itself. When the auditor controls the training process this independence is enforced by construction, by adding data points to the training set based on an independent coin flip. In PANORAMIA, we do not have independence between membership and x, as all non-members Algorithm 1 PANORAMIA Input: Target ML model f , audit set size m, confidence 1 -β Phase 1: 1: Split D f in D G , D: Sample s ∼ Bernoulli( 1 2 ) m ▷ Def.2 2: Create audit set X = s • D te in + (1 -s)D te out 3: Score each audit point for membership, creating t b ≜ b(X) ∈ R m + and t a ≜ MIA(X) ∈ R m + 4: Set v b ub (c, t) ≜ sup {v : β b (m, c, v, t) ≤ β 2 } ▷ Prop.1 5: c lb = max t,c 1{t b ≥ t} • s ≤ v b ub (c, 1{t b ≥ t}) 6: Set v a ub (c, ϵ, t) ≜ sup {v : β a (m, c, ϵ, v, t) ≤ β 2 } ▷ Prop.2 7: {c+ϵ} lb = max t,c,ϵ 1{t a ≥ t} • s ≤ v a ub (c, ϵ, 1{t a ≥ t}) Return ε ≜ {c+ϵ} lb -c lb come from the generator G ̸ = D.As a result, there are two ways to guess membership and have high MIA precision: either by using f to detect membership (i.e., symptomatic of privacy leakage) or by detecting generated data (i.e., not a symptom of privacy leakage). To measure the privacy leakage, we compare the results of the MIA to that of a baseline classifier b that guesses membership based exclusively on x, without access to f . The stronger this baseline, the better the removal of the effect of synthesized data detection. Algorithm 1 summarizes the entire procedure. In the next section, we demonstrate how to relate the difference between the baseline b and the MIA performance to the privacy loss ϵ. 
quantifying privacy leakage with panoramia: 
 We formalize a privacy game as follows: construct a sequence of auditing samples, in which s i is sampled independently to choose either the real (x in ∈ X m training points from D if s i = 1) or generated (x gen ∈ X m generated points from G if s 1 = 0) data point at each index i. Definition 2 (Privacy game).s ∼ Bernoulli( 1 2 ) m , with s i ∈ {0, 1}, x i = (1 -s i )x gen i + s i x in i , ∀i ∈ {1, . . . , m}.The key difference between Definition 2 and the privacy game of Steinke et al. (2023) lies in how we create the audit set. Instead of sampling non-members independently and removing them from the training set (which requires modifyig the training set and the model), Definition 2 starts from a set of known members (after the fact), and pairs each point with a non-member (generated i.i.d. from the generator distribution). For each pair, we then flip a coin to decide which point in the pair will be shown to the auditor for testing, thereby creating the test task of our privacy measurement.The level of achievable success in this game quantifies the privacy leakage of the target model f . More precisely, we follow an analysis inspired by that of Steinke et al. (2023) , but require several key technical changes to support auditing with no retraining using generated non-member data.We first need a notion of quality for our generator: Definition 3 (c-closeness). For all c > 0, we say that a generative modelG is c-close for data distribution D if: ∀x ∈ X , e -c P D x ≤ P G x .The smaller c, the better the generator, as G assigns a probability to real data that cannot be much smaller than that of the real data distribution. We make three important remarks.Remark 1: c-closeness is a strong requirement but it is achievable, at least for a large c. For instance, a uniform random generator is c-close to any distribution D, with c < ∞. Of course c would be very large, and the challenge in PANORAMIA is to create a generator that empirically has a small enough c. Remark 2: We show in Appendix E that we can relax this definition to hold only with high probability (following the (ϵ, δ)-DP relaxation). Empirical results under this relaxation are very similar. We also show how to measure (ϵ, δ)-DP, instead of ϵ-DP on which we focus here.Remark 3: Our definition of closeness is very similar to that of DP. This is not a coincidence as we will use this definition to be able to reject claims of both c-closeness for the generator and ϵ-DP for the algorithm that gave the target model. We further note that, contrary to the DP definition, c-closeness is one-sided as we only bound P G from below. Intuitively, this means that the generator has to produce high-quality samples (i.e., samples likely under D) with high enough probability but it does not require that all samples are good though. This one-sided measure of closeness is enabled by our focus on detecting members (i.e., true positives) as opposed to members and non-members, and puts less stringent requirements on the generator.Using Definition 3, we can formulate the hypothesis test on the privacy game of Definition 2 that underpins our approach:H : generator G is c-close, and target model f is ϵ-DP.Note that, in a small abuse of language, we will often refer to f as ϵ-DP to say that f is the output of an ϵ-DP training mechanism. That is,B(s, x) = {b(x 1 ), b(x 2 ), . . . , b(x m )}.We define two key mechanisms:B(s, x) : {0, 1} m × X m → R m + outputs a (potentially randomized) non-negative score for the membership of each datapoint x i , based on x i only; A(s, x, f ) : {0, 1} m × X m × F → R m+ outputs a (potentially randomized) non-negative score for the membership of each datapoint, with the guess for index i depending on x ≤i and target model f . We can construct a statistical test for each part of the hypothesis separately.Proposition 1. Let G be c-close, S and X be the random variables for s and x from Definition 2, and T b ≜ B(S, X) be the vector of guesses from the baseline. Then, for all v ∈ R and all t in the support of T :P S,X,T b m i=1 T b i • S i ≥ v | T b = t b ≤ P S ′ ∼Bernoulli( e c 1+e c ) m m i=1 t b i • S ′ i ≥ v ≜ β b (m, c, v, t b )Proof. In Appendix B.1. Now that we have a test to reject a claim that the generator G is c-close for the data distribution D, we turn our attention to the second part of H which claims that the target model f is the result of an ϵ-DP mechanism.Proposition 2. Let G be c-close, S and X be the random variables for s and x from Definition 2, f be ϵ-DP, and T a ≜ A(S, X, f ) be the guesses from the membership audit. Then, for all v ∈ R and all t in the support of T :P S,X,T a m i=1 T a i • S i ≥ v | T a = t a ≤ P S ′ ∼Bernoulli( e c+ϵ 1+e c+ϵ ) m m i=1 t a i • S ′ i ≥ v ≜ β a (m, c, ϵ, v, t a )Proof. In Appendix B.2.We can now provide a test for hypothesis H, by applying a union bound over Propositions 1 and 2:Corollary 1. Let H be true, T b ≜ B(S, X), and T a ≜ A(S, X, f ). Then:P m i=1 T a i • S i ≥ v a , m i=1 T b i • S i ≥ v b | T a = t a , T b = t b ≤ β a (m, c, ϵ, v a , t a ) + β b (m, c, v b , t b )To make things more concrete, let us instantiate Corollary 1 as we do in PANORAMIA. Our baseline (B above) and MIA (A above) classifiers return a membership guesses T a,b ∈ {0, 1} m , with 1 corresponding to membership. Let us call r a,b ≜ i t a,b i the total number of predictions, and tp a,b ≜ i t a,b i • s i the number of correct membership guesses (true positives). We also call the precision prec a,b ≜ tp a,b r a,b . Using the following tail bound on the sum of Bernoulli random variables for simplicity and clarity (we use a tighter bound in practice, but this one is easier to read),EQUATIONwe can reject H at confidence level β by settingβ a = β b = β 2 and if either prec b ≥ e c 1+e c + log(2/β) 2r bor prec a ≥ e c+ϵ 1+e c+ϵ + log(2/β) 2r a .In an privacy measurement, we would like to quantify ϵ, not just reject a given ϵ claim. We use the hypothesis test from Corollary 1 to compute a confidence interval on c and ϵ. To do this, we first define an ordering between (c, ϵ) pairs, such that if (c 1 , ϵ 1 ) ≤ (c 2 , ϵ 2 ), the event (i.e., set of observations for T a,b , S) for which we can reject H(c 2 , ϵ 2 ) is included in the event for which we can reject H(c 1 , ϵ 1 ). That is, if we reject H for values (c 2 , ϵ 2 ) based on audit observations, we also reject values (c 1 , ϵ 1 ) based on the same observations.We define the following order to fit this assumption, based on the hypothesis test from Corollary 1:EQUATIONThis lexicographic order over hypotheses ensures that, when using the joint hypothesis test from Corollary 1 to construct confidence intervals, we always reject hypotheses with a low value of c incompatible with the baseline performance. Formally, this yields the following confidence intervals:Corollary 2. For all β ∈]0, 1], m, and observed t a , t b , call v a ub (c, ϵ) ≜ sup {v :β a (m, c, ϵ, v, t a ) ≤ β 2 } and v b ub (c) ≜ sup {v : β b (m, c, v, t b ) ≤ β 2 }.Then:P (c, ϵ) ≥ sup (c ′ , ϵ ′ ) : t a • s ≤ v a ub (c ′ , ϵ ′ ) and t b • s ≤ v b ub (c ′ ) ≥ 1 -βProof. Apply Lemma 4.7 from Steinke et al. (2023) with the ordering from Equation 2 and the test from Corollary 1.The lower confidence interval for (c, ϵ) at confidence 1 -β is the largest (c, ϵ) pair that cannot be rejected using Corollary 1 with false rejection probability at most β. Hence, for a given confidence level 1 -β, PANORAMIA computes (c lb , ε), the largest value for (c, ϵ) that it cannot reject. Note that Corollaries 1 and 2 rely on a union bound between two tests, one for c and one for c + ϵ. We can thus consider each test separately (at level β/2). In practice, we use the whole precision/recall curve to achieve tighter bounds: each level of recall (i.e., threshold on t a,b to predict membership) corresponds to a bound on the precision, which we can compare to the empirical value. Separately for each test, we pick the level of recall yielding the highest lower-bound (Lines 4-7 in the last section of Algorithm 1). For all bounds to hold together, we apply a union bound over all tests at each level of recall. "PANORAMIA could not reject a claim of c-closeness for G, and if this claim is tight, then f cannot be the output of a mechanism that is ε-DP".While this is not as strong a claim as typical lower-bounds on ϵ-DP from prior privacy auditing works, we believe that this measure is useful in practice. Indeed, the ε measured by PANORAMIA is a quantitative privacy measurement, that is accurate (close to a lower-bound on ϵ-DP) when the baseline performs well (and hence c lb is tight). Thus, when the baseline is good, ε can be interpreted as (close to) a lower bound on (pure) DP. PANORAMIA opens a new capability, measuring privacy leakage of a trained model f without access or control of the training pipeline, with a meaningful and practically useful measurement, as we empirically show next. 
evaluation: 
 We instantiate PANORAMIA on target models for four tasks from three data modalities1 . For image classification, we consider the CIFAR10 Krizhevsky (2009) G is again based on small GPT-2, and then fine-tuned on D G . We generate samples using top-k sampling Holtzman et al. (2019) and a held-out prompt dataset D prompt G ⊂ D G . Finally, we conduct experiments on classification on tabular data. However for this data modality, PANORAMIA did not detect any meaningful leakage. More precisely, we have observed significant variance in the audit values returned by PANORAMIA which make the results inconclusive. Nonetheless, we present the results obtained for this task and discuss this issue further in Appendix D.6. Table 1 summarizes the tasks, models, and performance. More details on the settings, the models used, and implementation details are provided in Appendix C.Our MIA is a loss-based attack, which uses an ML model taking as input both a datapoint x and the value of the loss of target model f on point x.Appendix C details the architectures used for the attack model for each data modality. Recall from §4.2 the importance of having a tight c lb for our measure ε to be close to a lower-bound on ϵ-DP. To increase the performance of our baseline b, we mimic the role of the target model f 's loss in the MIA using a helper model h trained on synthetic data, which adds a loss-based feature to b.This new feature can be viewed as side information about the data distribution. Table 2 shows the c lb value for different designs for h. The best performance is consistently when h is trained on synthetic data before being used as a feature to train the b. Indeed, such a design reaches a c lb up to 1.19 larger that without any helper (CIFAR10) and 0.23 higher than when training on real non-member data, without requiring access to real non-member data, a key requirement in PANORAMIA. We adopt this design in all the following experiments. In Figure 2 we show that the baseline has enough training data (vertical dashed line) to reach its best performance. Appendix D.1, shows similar results on GPT-2 as well as different architectures that we tried for the helper model. All these pieces of evidence confirm the strength of our baseline.We run PANORAMIA on models with different degrees of over-fitting by varying the number of epochs, (see the final accuracy on Table 1 ) for each data modality. More over-fitted models are known to leak more information about their training data due to memorization (Yeom et al., 2018; Carlini et al., 2019 Carlini et al., , 2022a)) . To show the privacy loss measurement power of PANORAMIA, we compare it with two strong approaches to lower-bounding privacy loss. First, we compare PANORAMIA with the O(1) First, the best prior method (whether RM;RN or O(1)) measures a larger privacy loss (ε ≤ ϵ). Overall, these results empirically confirm the strength of b, as we do not seem to spuriously assign differences between G and D to our privacy loss proxy ε. We also note that O(1) tends to perform better, due to its ability to rely on non-member detection, which improves the power of the statistical test at equal data sizes. Such tests are not available in PANORAMIA given our one-sided closeness definition for G (see §4), and we keep that same one-sided design for RM;RN for the sake of comparison.Second, the values of ε measured by PANORAMIA are close to those of the methods against which we compared. In particular, despite a more restrictive adversary model (i.e., no non-member data, no control over the training process, and no shadow model training), PANORAMIA is able to detect meaningful amounts of privacy loss, comparable to that of stateof-the-art methods.For instance, on a non-overfitted CIFAR-10 ResNet101 model (E20), PANORAMIA detects a privacy loss of 0.33, while using real non-member (RM;RN) data yields 0.42, and controlling the training process O(1) gets 0.45. The relative gap gets even closer on models that reveal more about their training data. Indeed, for the most over-fitted model (E100), ε = 1.16 is very close to RM;RN (ϵ = 1.22) and O(1) (ϵ = 1.41). This also confirms that the leakage detected by PANORAMIA on increasingly overfitted models does augment, which is confirmed by prior state-of-the-art methods. For instance, NLP models' ε goes from 0.22 to 2.40 (2.82 to 5.73 for O(1)). Appendix D.6 details results on tabular data, in which PANORAMIA is not able to detect significant privacy loss.Privacy Measurement of ϵ-DP Models: Another avenue to varying privacy leakage is by training Differentially-Private (DP) models with different values of ϵ. We evaluate PANORAMIA on DP ResNet-18 models on CI-FAR10, with ϵ values shown on Table 3 and Figure 5 . The hyper-parameters were tuned independently for the highest train accuracy (see Appendix D.9 for more results and discussion). As the results show, neither PANORAMIA nor O(1) detect privacy loss on the most private models (ϵ = 1, 2). At higher values of ϵ = 10, 15 (i.e., less private models) and ϵ = ∞ (i.e., non-private model) PANORAMIA does detect an increasing level of privacy leakage with εϵ=10 < εϵ=15 < εϵ=∞ . In this regime, the O(1) approach detects a larger, though comparable, amount of privacy loss.We also evaluate PANORAMIA on DP (fine-tuned) large language models (see Appendix D.9 and Table 9 for details). We fine-tune GPT2-Small target models with DP-SGD on the WikiText dataset, with various values of ϵ. Neither PANORAMIA nor O(1) are able to measure a positive privacy loss.PANORAMIA can leverage much more data for its measurement (up to the whole training set size for training and testing the MIA), while in our setting O(1) is limited by the size of the test set (for non-members). As we have seen in Figure 6 (b), PANORAMIA can leverage larger test set size to measure higher privacy loss values than O(1), up to 2.62 on the ResNet101 trained for 100 epochs, versus 2.23 for O(1). It is also important to note that at small amounts of data O(1) measures a larger privacy loss (ε ≤ ϵ). These findings provide additional empirical evidence for the robustness of b, indicating that we are not erroneously attributing differences between G and D to our privacy loss proxy ε. However, in the case of text data (on the GPT-2 trained for 150 epochs), Figure 6 (d), we hit an upper-bound on the power of the hypothesis test which estimates {c+ϵ} lb using 20k test samples.The maximum measurement for ε we can achieve is around 2.59 with a 20k test set, as opposed to O(1), which reaches 5.73 with 10k test set size. 1 1 1 1 1 1 1 12 2 2 2 2 2 2 2 4 4 4 4 4 4 4 4 6 6 6 6 6 6 6 Figure 6 : Effect of increasing test set size on PANORAMIA's privacy measurement for our target models. In the case of the image dataset, increasing the number of auditing examples allows us to achieve tighter empirical measurements for privacy leakage despite a restricted adversary. For the case of the language modeling task, when increasing test set size m, we hit an upper bound on the power of the hypothesis test due to test dataset size, and do not see significant improvement in privacy measurement in this case.Increasing Test Set Size for DP models: With an increase in the test set size m, we demonstrate that we can achieve tighter bounds on the privacy measurement ε. 6 Impact, Limitations and Future DirectionsWe have introduced a new approach to quantify the privacy leakage from ML models, in settings in which individual data contributors (such as a hospital in a cross-site FL setting or a user of a text auto-complete service) measure the leakage of their own, known partial training data in the final trained model. Our approach does not introduce new attack capabilities (it would likely benefit from future progress on MIA performance though), but can help model providers and data owners assess the privacy leakage incurred by their data, due to inclusion in ML models training datasets. Consequently, we believe that any impact from our proposed work is likely to be positive.However, PANORAMIA suffers from a major limitation: its privacy measurement is not a lower-bound on the privacy loss (see details in Section 4.2). Providing a proper lower-bound ϵ lb on privacy leakage in this setting is an important avenue for future work. A promising direction is to devise a way to measure or enforce an upper-bound on c, thereby yielding ϵ lb via {c + ϵ} lb -c ub . Despite this shortcoming, we believe that the new measurement setting we introduce in this work is important. Indeed, recent work has shown that MIA evaluation on foundation models suffers from a similar lack of non-member data (since all in distribution data is included in the model) Das et al. (2024) ; Duan et al. (2024) ; Meeus et al. (2024) . The theory we develop in this paper provides a meaningful step towards addressing privacy measurements in this setting, and provides a more rigorous approach to privacy benchmarks for such models. We also demonstrate that PANORAMIA's privacy measurements can also be empirically valuable for instance for providing improved measurements with more data (Figure 6 ). For both Proposition 1 and Proposition 2, we state the proposition again for convenience before proving it.Proposition 1. Let G be c-close, and T b ≜ B(S, X) be the guess from the baseline. Then, for all v ∈ R and all t in the support of T :P S,X,T b m i=1 T b i • S i ≥ v | T b = t b ≤ P S ′ ∼Bernoulli( e c 1+e c ) m m i=1 t b i • S ′ i ≥ v ≜ β b (m, c, v, t b )Proof. Notice that under our baseline model B(s, x) = {b(x 1 ), b(x 2 ), . . . , b(x m )}, and given that the X i are i.i.d., we have that:S <i ⊥ ⊥ T b <i | X <i , since T b i = B(S, X) i 's distribution is entirely determined by X i ; and S ≤i ⊥ ⊥ T b>i | X <i since the X i are sampled independently from the past. We study the distribution of S given a fixed prediction vector t b , one element i ∈ [m] at a time:P S i = 1 | T b = t b , S <i = s <i , X ≤i = x ≤i = P S i = 1 | S <i = s <i , X ≤i = x ≤i = P X i | S i = 1, S <i = s <i , X <i = x <i P S i = 1 | S <i = s <i , X <i = x <i P X i | S <i = s <i , X <i = x <i = P X i | S i = 1, S <i = s <i , X <i = x <i P S i = 1 P X i | S <i = s <i , X <i = x <i = P X i | S i = 1 1 2 P X i | S i = 1 1 2 + P X i | S i = 0 1 2 = 1 1 + P Xi | Si=0 P Xi | Si=1 = 1 1 + P G Xi P D Xi ≤ 1 1 + e -c = e c 1 + e cThe first equality uses the independence remarks at the beginning of the proof, the second relies Bayes' rule, while the third and fourth that S i is sampled i.i.d from a Bernoulli with probability half, and X i i.i.d. conditioned on S i . The last inequality uses Definition 3 for c-closeness.Using this result and the law of total probability to introduce conditioning on X ≤i , we get that:P S i = 1 | T b = t b , S <i = s <i = x ≤i P S i = 1 | T b = t b , S <i = s <i , X ≤i = x ≤i P X ≤i = x ≤i | T b = t b , S <i = s <i ≤ x ≤i e c 1 + e c P X ≤i = x ≤i | T b = t b , S <i = s <i ,and hence that:EQUATIONWe can now proceed by induction: assume inductively thatW m-1 ≜ m-1 i=1 T b i • S i is stochastically dominated (see Definition 4.8 in Steinke et al. (2023)) by W ′ m-1 ≜ m-1 i=1 T b i • S ′ i , in which S ′ ∼ Bernoulli( e c 1+e c ) m-1 . Setting W 1 = W ′ 1 = 0 makes it true for m = 1. Then, conditioned on W m-1 and using Eq. 3, T b m • S m = T m • 1{S m = 1} is stochastically dominated by T b m • Bernoulli( e c 1+e c). Applying Lemma 4.9 from Steinke et al. (2023) shows that W m is stochastically dominated by W ′ m , which proves the induction and implies the proposition's statement.Proposition 2. Let G be c-close, f be ϵ-DP, and T a ≜ A(S, X, f ) be the guess from the membership audit. Then, for all v ∈ R and all t in the support of T :P S,X,T a m i=1 T a i • S i ≥ v | T a = t a ≤ P S ′ ∼Bernoulli( e c+ϵ 1+e c+ϵ ) m m i=1 t a i • S ′ i ≥ v ≜ β a (m, c, ϵ, v, t a )Proof. Fix some t a ∈ R m + . We study the distribution of S one element i ∈ [m] at a time:P S i = 1 | T a = t a , S <i = s <i , X ≤i = x ≤i = P T a = t a | S i = 1, S <i = s <i , X ≤i = x ≤i P S i = 1 | S <i = s <i , X ≤i = x ≤i P T a = t a | S <i = s <i , X ≤i = x ≤i ≤ 1 1 + e -ϵ P Si=0 | S<i=s<i,X ≤i =x ≤i P Si=1 | S<i=s<i,X ≤i =x ≤i ≤ 1 1 + e -ϵ e -c = e c+ϵ 1 + e c+ϵThe first equality uses Bayes' rule. The first inequality uses the decomposition:P T a = t a | S <i = s <i , X ≤i = x ≤i = P T a = t a | S i = 1, S <i = s <i , X ≤i = x ≤i • P S i = 1 | S <i = s <i , X ≤i = x ≤i + P T a = t a | S i = 0, S <i = s <i , X ≤i = x ≤i • P S i = 0 | S <i = s <i , X ≤i = x ≤i ,and the fact that A(s, x, f ) is ϵ-DP w.r.t. s and hence that:P T a = t a | S i = 0, S <i = s <i , X ≤i = x ≤i P T a = t a | S i = 1, S <i = s <i , X ≤i = x ≤i ≥ e -ϵ .(4)This inequality might seem surprising at first, as we know from the privacy game (Definition 2) that S is independent of f , that is S ⊥ ⊥ f . However, these two quantities are not conditionally independent when conditioning on X (where X is the test set for the MIA/baseline), (i.e., S ̸ ⊥ ⊥ f |X). This is because X i is either a member or non-member based on S i . So when S i = 1, then f was trained on X i , which makes f and S i dependent if membership can be detected through f . If f is ϵ-DP, we can bound this dependency using DP properties, which is what we do in Equation ( 4).The second inequality uses that:P S i = 0 | S <i = s <i , X ≤i = x ≤i P S i = 1 | S <i = s <i , X ≤i = x ≤i = P X i = x i | S i = 0, S <i = s <i , X <i = x <i P X i = x i | S i = 1, S <i = s <i , X <i = x <i • P S i = 0 | S <i = s <i , X <i = x <i P S i = 1 | S <i = s <i , X <i = x <i = P X i = x i | S i = 0, S <i = s <i , X <i = x <i P X i = x i | S i = 1, S <i = s <i , X <i = x <i • 1/2 1/2 = P G X i P D X i ≥ e -cAs in Proposition 1, applying the law of total probability to introduce conditioning on X ≤i yields:EQUATIONand we can proceed by induction. Assume inductively thatW m-1 ≜ m-1 i=1 T a i • S i is stochastically dominated (see Definition 4.8 in Steinke et al. (2023)) by W ′ m-1 ≜ m-1 i=1 T a i • S ′ i , in which S ′ ∼ Bernoulli( e c+ϵ 1+e c+ϵ ) m-1 . Setting W 1 = W ′ 1 = 0 makes it true for m = 1.Then, conditioned on W m-1 and using Eq. 5,T a m • S m = T a m • 1{S m = 1} is stochastically dominated by T a m • Bernoulli( e c+ϵ 1+e c+ϵ). Applying Lemma 4.9 from Steinke et al. (2023) shows that W m is stochastically dominated by W ′ m , which proves the induction and implies the proposition's statement.Hereafter, we provide the details about the datasets and models trained in our experiments. Generator. For both image datasets, we use StyleGAN2 Karras et al. (2020) to train the generative model G from scratch on D G , and produce non-member images. For CIFAR10 dataset, we use a 10, 000 out of 50, 000 images from the training data of the target model to train the generative model. For the CelebA dataset, we select 35, 000 out of 200, 000 images from the training data of the target model to train the generative model. Generated images will in turn serve as non-members for performing the MIAs. Figure 7 shows examples of member and non-member images used in our experiments. In the case of CelebA, we also introduce a vanilla CNN as a classifier or filter to distinguish between fake and real images, and remove any poor-quality images that the classifier detects with high confidence. The data used to train this classifier was the same data used to train StyleGAN2, which ensures that the generated high-resolution images are of high quality.MIA and Baseline training. For the MIA, we follow a loss-based attack approach: PANORAMIA takes as input raw member and non-member data points for training along with the loss values the target model f attributes to these data points. More precisely, the training set of PANORAMIA is:(D tr in , f (D tr in )) ∪ (D tr out , f (D tr out ))In §4.2, we discussed the importance of having a tight c lb so that our measure, ε, becomes close to a lower-bound on ϵ-DP, which requires a strong baseline. To strengthen our baseline, we introduce the helper model h, which helps the baseline model b by supplying additional features (i.e., embeddings) that can be viewed as side information about the data distribution. The motivation is that h's features might differ between samples from D and D ′ , enhancing the performance of the baseline classifier.This embedding model h is similar in design to f (same task and architecture) but is trained on synthetic data that is close in distribution to the real member data distribution.The helper model has the same classification task and architecture as the target model. To train it, we generate separate sets of training and validation data using our generator. For image data, the synthetic samples do not have a label. We thus train a labeler model, also a classifier with the same task, on the same dataset used to train the generator. We use the labeler model to provide labels for our synthetic samples (training and validation sets above). We train the helper model on the resulting training set, and select hyperparameters on the validation set.Whether for the baseline or MIA, we use side information models (h and f , respectively) by concatenating the loss of h(x) and f (x) to the final feature representation (more details are provided later) before a last layer of the MIA/Baseline makes the membership the prediction. Since we need labels to compute the loss, we label synthetic images with a Wide ResNet-28-2 in the case of CIFAR10, and a Multi-Label CNN of similar architecture as the target model in the case of CelebA labeling. For both instances, we used a subset of the data, that was used to train the respective generative models, to train the "labeler" classifiers as well. The labeler used to train the helper model for image data has 80.4% test accuracy on the CIFAR10 real data test set. The rationale for this approach is to augment the baseline with a model providing good features (here for generated data) to balance the good features provided to the MIA by f outside of the membership information. In practice, the labeled generated data seems enough to provide such good features, despite the fact that the labeler is not extremely accurate. We studied alternative designs (e.g., a model trained on non-member data, no helper model) in Appendix D.1, Table 5 , and the helper model trained on the synthetic data task performs best (while not requiring non-member data, which is a key point).We use two different modules for each MIA and Baseline training. More precisely, the first module optimizes image classification using a built-in Pytorch ResNet101 classifier. The second module, in the form of a multi-layer perceptron, focuses on classifying member and non-member labels via loss values attributed to these data points by f as input for the loss module of MIA and losses of e to the baseline b respectively. We then stack the scores of both image and loss modules into a logistic regression task (as a form of meta-learning) to get the final outputs for member and non-member data points by MIA and baseline b.When training the baseline and MIA models, we use a validation set to select hyper-parameters and training stopping time that maximize the lower bounds (effectively maximizing the c lb or εlb for the Baseline and MIA respectively) on the validation set. The MIA and baseline are trained on 4500 data samples (half members and half generated non-members). The test dataset consists of 10000 samples, again half members and half generated non-members. The actual and final number of members and non-members that ended up in the test set depends on the Bernoulli samples in our auditing game. We repeat the training process over 5 times, each time independently resampling the training dataset for the MIA and baseline (keeping the train dataset size fixed). We report all our results over a 95% confidence interval and report the standard deviation for our results in Table 3 . The compute resources used for MIA and Baseline training were mainly running all experiments on cloud-hosted VMs, using 1 v100l GPU, 4 nodes per task, and 32G memory requested for each job on the cloud cluster. The time to run the MIA and baseline attack pipeline was around 10 hours including hyperparameter tuning on ϵ lb and c lb respectively. One common standard pre-processing in CLM is to break the tokenized training dataset into chunks of equal sizes to avoid padding of the sequences. To achieve this, the entire training dataset is split into token sequences of fixed length, specifically 64 tokens per sequence, refer hereafter as "chunks". Each of these chunks serves as an individual sample within our training dataset. Therefore, in the experiments, the membership inference will be performed for a chunk. In other words, we adopt an example-level differential privacy (DP) viewpoint, where an example is considered a sequence of tokens that constitute a row in a batch of samples. This chunk-based strategy offers additional benefits in synthetic sample generation. By maintaining uniformity in the length of these synthetic samples, which also consists of 64 tokens, it is possible to effectively mitigate any distinguishability between synthetic and real samples based solely on their length. We also consider the possibility of weak correlations between chunks (e.g., being from the same article). To mitigate this, we perform a form of stratified sampling as a pre-processing step. From each document, we include only the first 50 chunks and discard the rest. We select 2,113 documents in total, resulting in 105,650 chunks in our dataset.We split the dataset into train and test with a 90:10 ratio for the target model (We insist on following the typical training pipelines in machine learning, since we plan to conduct post-hoc audits with PANORAMIA). The test dataset provides real non-member samples, allowing comparison to the O(1) auditor and another version of our approach which uses real non-member data. The training dataset (D f ) contains 95,085 samples, while the test dataset consists of 10,565 samples.We train the target model for 200 epochs in total. The compute resources used for training the target models are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the target model for 200 epochs required about 13 hours.Generator. The G is a GPT-2 fine-tuned using a CLM task on dataset D G , a subset of D f . To create synthetic samples with G, we use top-k sampling Holtzman et al. (2019) method in an auto-regressive manner while keeping the generated sequence length fixed at 64 tokens. To make G generate samples less like the real members it learned from (and hopefully, more like real non-members), we introduce more randomness in the generation process in the following manner. We choose a top_k value of 200, controlling the number of available tokens for sampling. In addition, we balance the value of top_k, making it not too small to avoid repetitive generated texts but also not too large to maintain the quality. Finally, we fix the temperature parameter that controls the randomness in the softmax function at one. Indeed, a larger value increases the entropy of the distribution over tokens resulting in a more diverse generated text.However, the quality of synthetic text depends on the prompts used. To mitigate this issue, we split the , which is a sequence of length 64. We feed the prompt into the generator and request the generator to generate a suffix of equivalent length. In our experiments, for each prompt, we make 8 synthetic samples, supplying a sufficient number of synthetic non-member samples for different parts of the pipeline of PANORAMIA, including training the helper model, and constructing D tr out , D te out . Overall, we generate 92276 synthetic samples out of which we use 53502 of them for training the helper model and keep the rest for audit purposes. Note that this generation approach does not inherently favor our audit scheme, and shows how PANORAMIA can leverage existing, public generative models with fine tuning to access high-quality generators at reasonable training cost.We also perform a sanity check on the quality of the generated data. We visualize the loss distribution of in-distribution members, in-distribution non-members, and synthetic data under both the target and helper models, aiming for synthetic data that behaves similarly to in-distribution non-member data. Figure 14 visualizes these distributions for our target and helper models.The compute resources used for fine-tuning the generator model, and synthetic text generation are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, fine-tuning the GPT2-small model took approximately 1.5 hours, and generating the synthetic samples required about 1 hour.Baseline & MIA. For both the baseline and PANORAMIA classifiers, we employ a GPT-2 based sequence classification model. In this setup, GPT-2 extracts features (i.e., hidden vectors) directly from the samples. We concatenate a vector of features extracted using the helper h (for the baseline) or target model f (for the MIA) to this representation. This vector of features is the loss sequence produced in a CLM task for the respective model. These two sets of extracted features are then concatenated before being processed through a feed-forward layer to generate the logit values required for binary classification (distinguishing between members and non-members).In the main results in Table 3 , for both the baseline and PANORAMIA classifiers, the training and validation sets consist of 20000 and 2000 samples, respectively, with equal member non-member distribution. The test set consists of 10000 samples, in which the actual number of members and non-members that ended up in the test set depends on the Bernoulli samples in the auditing game. The helper model is fine-tuned for 60 epochs on synthetic samples and we pick the model that has the lowest validation loss throughout training, generalizing well.As the GPT-2 component of the classifiers can effectively minimize training loss without achieving strong generalization, regularization is applied to the classifier using weight decay. Additionally, the optimization process is broken into two phases. In the first phase, we exclusively update the parameters associated with the target (in PANORAMIA) or helper model (in the baseline) to ensure that this classifier has the opportunity to focus on these specific features. In the second phase, we optimize the entire model as usual. We train both the baseline and Membership Inference Attack (MIA) models with 5 different seeds. The randomness is over: 1. The data split into train/validation/test sets for both the baseline and MIA classifiers. Each classifier is retrained for every seed. 2. The coin flips in our auditing game (see Definition 2). For a fixed seed mentioned above, we employ an additional set of 5 seeds to account for the randomness in the training algorithm (such as initialization of the model, and batch sampling). We then train a classifier for each of these seeds. From the 5 models generated for each latter seed, we select the one with the largest c lb value (for the baseline) or {c+ϵ} lb (for MIA), determined over the validation set. We calculate the 95% confidence interval of our measurements using the t-score.The compute resources used for MIA and Baseline training were 1 GPU with 32gb memory, running on cloud-hosted VMs. The compute time depends on the training set size, for the largest training size, it took about 3 hours to train the baseline or MIA.The compute resources used for fine-tuning the helper model are 4 V100-32gb GPUs, running on cloud-hosted VMs. Using this setup, it took about 4 hours to fine-tune the helper model.Target f and the helper model h. The Target f and the helper h model are both Multi-Layer Perceptron with four hidden layers containing 150, 100, 50, and 10 neurons respectively. Both models are trained using a learning rate of 0.0005 over 100 training epochs. In the generalized case and to obtain the embedding, we retain the model's parameters that yield the lowest loss on a validation set, which typically occurs at epoch 10 (MLPE_10). For the overfitted scenario, we keep the model's state after the 100 training epochs (MLP_E100). The non overfitted model and the overfitted one achieved an accuracy of 86% and 82% respectively.Generator. We use the MST method McKenna et al. (2021) which is a differentialy-private method to generate synthetic data. However, as we do not need Differential Privacy for data generation we simply set the value for ε to 1000. Synthetic data generators can sometimes produce bad-quality samples. Those out-of-distribution samples can affect our audit process (the detection between real and synthetic data being due to bad quality samples rather than privacy leakage). To circumvent this issue, we train an additional classifier to distinguish between real data from D G and additional synthetic data (not used in the audit). We use this classifier to remove from the audit data synthetic data synthetic samples predicted as synthetic with high confidence.Baseline and MIA. To distinguish between real and synthetic data, we use the Gradient Boosting model and conduct a grid search to find the best hyperparameters.We implement the black-box auditor version of O(1) approach Steinke et al. (2023) . This method assigns a membership score to a sample based on its loss value under the target model. They also subtract the sample's loss under the initial state (or generally, a randomly initialized model) of the target model, helping to distinguish members from non-members even more. In our instantiation of the O(1) approach, we only consider the loss of samples on the final state of the target model. Moreover, in their audit, they choose not to guess the membership of every sample. This abstention has an advantage over making wrong predictions as it does not increase their baseline. Roughly speaking, their baseline is the total number of correct guesses achieved by employing a randomized response (ϵ, 0) mechanism, for those samples that O(1) auditor opts to predict. We incorporate this abstention approach in our implementation by using two thresholds, t + and t -. More precisely, samples with scores below t + are predicted as members, those above t -as non-members, and the rest are abstained from prediction. We check for different combinations of t + and t -and report the highest ϵ among them. This involves performing multiple tests on the same evidence, each with a separate confidence level of 95%. To ensure that these tests hold collectively, we apply a union bound and adjust the significance level accordingly. The total number of separate tests depends on the different number of guesses that O(1) can make. We also set δ to 0 and use a confidence interval of 0.05 in their test. In PANORAMIA, for each hypothesis test (whether for c lb or {c+ϵ} lb ), we stick to a 0.025 confidence interval for each one, adding up to an overall confidence level of 0.05.The basis of our privacy measurement directly depends on how well our baseline classifier distinguishes between real member data samples and synthetic non-member data samples generated by our generative model. As mentioned in Section 5.1 and Appendix C.1, in order to increase the performance of our baseline b, we mimic the role of the target model f 's loss in the MIA using a helper model h, which adds a loss-based feature to b. To assess the strength of the baseline classifier, we train the baseline's helper model h on two datasets, synthetic data, and real non-members. We also train a baseline without any helper model. This enables us to identify which setup allows the baseline classifier to distinguish the member and non-member data the best in terms of the highest c lb value (c lb is reported for each case in table 5). For initial analysis, we keep a sub-set of real data as non-members to train a helper model with them, for conducting this ablation study on the baseline. Our experiments confirm that the baseline with loss values from a helper model trained on synthetic data performs better compared to a helper model trained on real non-member data. This confirms the strength of our framework in terms of not having a dependency of using real non-members. Therefore, we choose this setting for our baseline helper model. In addition, we also experiment with different helper model architectures in the case of CIFAR10, looking for the best baseline performance among them. For the WikiText dataset, the c lb reported in Table 5 differs from that in Table 3 because we had less data available for this experiment. We withheld half of the real samples to train the helper model with real samples in its training set.For each data modality setting, we also train the baseline classifier on an increasing dataset size of real member and synthetic non-member data points to gauge the trend of the baseline performance as the training dataset size increases (see Figure 2 and Figure 9 ). This experiment allows us to aim for a worst-case empirical lower bound on how close the generated non-member data is to the real member data distribution, i.e. what would be the largest c lb we can reach as we increase the training set size. Subsequently, we can compare the c lb reported in table 3 (corresponding to the vertical dashed lines in Figure 2 and Figure 9 ) to the largest c lb we can achieve in this experiment. In the following, we explain why we chose that number of training samples for our baseline in the main results.On the WikiText dataset, we increase the baseline's training set size up to 50k, while fixing the validation and test sets sizes to 2000 and 10000, respectively). Figure 9 shows the changes of c lb while varying the training set size. The vertical dashed line specified the number of training samples we chose for our main auditing results reported in Table 3 . Now, let us discuss why we don't choose the largest training set size available for the baseline. The measurement for c lb depends not only on the training set size but also on the test set size. Therefore, we need to balance the sizes of the train and test sets. We conducted another experiment where we varied the train set size and, instead of fixing the test set size, allocated the remaining available data to the test set (keeping in mind that our data for the audit is limited by the number of real members allocated for audit purposes).Figure 9 (b) depicts the trend of changes. As expected, we observed an initial increase in c lb as the training set size increased (and the number of test data is sufficient to capture that). However, it decreased toward the end because of the smaller test set size. For a train set size of 40k, we found the best trade-off between train and test set sizes, which justifies our choice of 40k for the training set size in the main results. The same argument applies for other data modalities.Let us make one last point. It is also important to note that the MIA is expected to show similar behavior if we use a larger train set size or test set size. Hence, the gap between the performance of MIA and baseline (which is our privacy measurement) is not expected to be affected (see Appendix D.4).The experiments are repeated five times, and we provide a 95% confidence interval. For the image data modality, the randomness lies in the training data provided to the classifiers, while the test set remains fixed throughout. For the text data modality, both the train and test sets change across the five runs (the test set size is fixed to 10k).The value of ε for each target model is the gap between its corresponding dashed line and the baseline one in Figure 4 . This allows us to compute values of ε reported in Table 3 . It is also interesting to note that the maximum value of {c+ϵ} lb typically occurs at low recall. Even if we do not use the same metric (precision-recall as opposed to TPR at low FPR) this is coherent with the findings from Carlini et al. (2022b) . We can detect more privacy leakage (higher {c+ϵ} lb which leads to higher ε) when making a few confident guesses about membership rather than trying to maximize the number of members found (i.e., recall). Figure 10 , decomposes the precision in terms of the number of true positives and the number of predictions for a direct mapping to the propositions 1 and 2. These are non-negligible values of privacy leakage, even though the true value is likely much higher. 1) is limited by the size of the test set (for non-members). We vary test size from m = 500, up to 35k. The experiment is repeated 10 times with resampling of the test set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence.We also conduct experiments to show the effect of increasing training dataset size on the performance of PANORAMIA (as well as the baseline). In the CIFAR10 image dataset case, Figure 11 , we see PANORAMIA performance slowly increase as the train dataset size increases. It reaches its max performance at dataset sizes 4500 -5000, 10k, 20k -30k and slowly begins to stabilize at the higher dataset sizes. We vary train size from m = 500, up to 30k. The experiment is repeated 5 times with resampling of the train set each time to produce the average. Unless otherwise stated, all results are produced at a 95% confidence. As illustrated in Figure 4 (c) and Table 3 , there exist instances in which the c lb is as large as {c+ϵ} lb .For instance, for GPT-2_E37, the upper bound in the confidence interval measured for c lb is larger than the lower bound of the measurement for {c + ϵ} lb . Although the measured ε is positive on average, but the lower bound of the interval is negative. For such cases, we don't claim any detection of privacy leakage. This implies that, under certain conditions, the baseline could outperform the MIA in PANORAMIA. We can see three possible reasons for this behavior.(1) The synthetic data might not be of high enough quality. If there is an evident distinction between real and synthetic samples, the baseline cannot be fooled reducing the power of our audit.(2) The MIA in PANORAMIA is not powerful enough.(3) The target model's privacy leakage is indeed low. For GPT-2_E150 and GPT-2_E75 in Figure 4 (c), we can claim that given the target model's leakage level, the synthetic data and the MIA can perform the audit to a non-negligible power. For GPT-2_37, we need to examine the possible reasons more closely. Focusing on generated data, in the ideal case we would like the synthetic data to behave just like real non-member data on Figure 14(a) . If this was the case, {c+ϵ} lb would go up (since the MIA uses this loss to separate members and non-members). Nonetheless, predicting changes in c lb remains challenging, as analogous loss distributions do not guarantee indistinguishability between data points themselves (also, we cannot necessarily predict how the loss distribution changes under the helper model h in Figure 14(d) ). Hence, reason (1) which corresponds to low generative data quality, seems to be a factor in the under-performance of PANORAMIA. Moreover, in the case GPT-2_37, we observe that real members and non-members have very similar loss distributions. Hence, factor (2) also seems to be at play, and a stronger MIA that does not rely exclusively on the plot may increase auditing power. Finally, the helper model h seems to have a more distinct loss distribution when compared to GPT-2_37 (see Figure 14(d) ).As we can see in Table 6 , our proposed method PANORAMIA cannot detect any meaningful privacy leakage in the case of a small MLP classifier trained on tabular data.O(1) ε MLP_E10 0 ± 0.142 0.655 ± 0.014 MLP_E100 0.062 ± 0.156 0.648 ± 0.015 The high precision at low recall in the baseline as can be seen in Figure 12 (a) illustrates the fact that some real data can easily be distinguished from the synthetic ones. Instead of poor quality synthetic data , we are dealing with real observation that are actually "looking to real". Investigating those observations reveals that they have extreme or rare values in one of their features. Indeed, the Adult dataset contains some extreme values (e.g., in the capital_gain column), such extreme values are rarely reproduced by the generative model. This leads to the baseline accurately predicting such observations as real with high confidence that translates into a high c lb . Moreover, the audit values maximum values (c + ε) lb and c lb being achieved for a relatively small number of guesses corresponding to extreme values, the audits results vary a lot depending on the sampling or not of such extreme values in the audit test set. We believe this is part of the reason why we observe an high variance in the audit results leading to no significant privacy leakage detected by PANORAMIA. This is a potential drawback of our auditing scheme, while the presence of extreme or rare values in the dataset usually allows for good auditing results it can have a negative impact on PANORAMIA's success.Methodology. Varying the number of training epochs for the target model to induce overfitting is known to be a factor in privacy loss Yeom et al. (2018) ; Carlini et al. (2022a) . As discussed in Section 5.2, since these different variants of target models share the same dataset and task, PANORAMIA can compare them in terms of privacy leaking.To verify if PANORAMIA will indeed attribute a higher value of ε to more overfitted models, we train our target models for varying numbers of training epochs. The final train and test accuracies are reported in Table 1 . For GPT-2 (as a target model), we report the train and validation loss as a measure of overfitting in Figure 8 . Figures 13 and 14 show how the gap between member data points (i.e., data used to train the target models) and non-member data points (both real as well as generated non-members) increases as the degree of overfitting increases, in terms of loss distributions. We study the distribution of losses since these are the features extracted from the target model f or helper model h, to pass respectively to PANORAMIA and the baseline classifier. The fact that the loss distributions of member data become more separable from non-member data for more overfitted models is a sign that the corresponding target model could leak more information about its training data. We thus run PANORAMIA on each model, hereafter presenting the results obtained. Results. In Figure 3 , we observe that more training epochs (i.e., more overfitting) lead to better precision-recall trade-offs and higher maximum precision values. Our results are further confirmed by Figure 10 with PANORAMIA being able to capture the number of member data points (positive predictions) better than the baseline b.In Table 3 , we further demonstrate that our audit output ε orders the target models in terms of privacy leakage: higher the degree of overfitting, more memorization and hence a higher ε returned by PANORAMIA. From our experiments, we consistently found that as the number of epochs increased, the value of ε also increased. the per-example gradient. We minimize the validation loss by tuning hyperparameters such as the learning rate (lr), number of epochs (e), batch size (bs), and clipping norm (C). The lowest validation loss values obtained are 3.79, 3.7, and 3.65 for ϵ values of 1, 3, and 10, respectively. For the non-private GPT-2-Small model, the lowest validation loss achieved is 3.341.The noise multiplier σ is computed given ϵ, number of epochs and batch size. We use both PANORAMIA and O(1) (Steinke et al., 2023) privacy loss measurements under the pure ϵ-DP analysis.Results. For the image data modality, tables 7 and 8 summarizes the auditing results of PANORAMIA on different DP models. For ResNet-18, we observe that at ϵ = 1, 2, 4, 6 (more private models) PANORAMIA detects no privacy loss, whereas at higher values of ϵ = 10, 15, 20 (less private models) and ϵ = ∞ (a non-private model) PANORAMIA detects an increasing level of privacy loss with εϵ=10 < εϵ=15 < εϵ=20 < εϵ=∞ , suggesting a higher value of ϵ correspond to higher ε. We observe a similar pattern with Wide-ResNet-16-4, in which no privacy loss is detected at ϵ = 1, 2 and higher privacy loss is detected at ϵ = 10, 15, 20, ∞. We also compare the auditing performance of PANORAMIA with that of O(1) (Steinke et al., 2023) , with the conclusion drawn by these two methods being comparable. For both ResNet-18 and Wide-ResNet-16-4, O(1) reports values close to 0 (almost a random guess between members and non-members) for ϵ < 10 DP models, and higher values for ϵ = 10, 15, 20, ∞ DP models. The results suggest that PANORAMIA is potentially an effective auditing tool for DP models that has comparable performance with O(1) and can generalize to different model structures.For the text data modality, Table 9 summarizes the auditing results of PANORAMIA on differentially private (fine-tuned) Large Language Models. In all cases, neither PANORAMIA not O(1) Steinke et al. (2023) do not detect any privacy loss.Target model f Audit Generator G ε ϵ ∞-DP GPT2-Small (GPT2_E37) PANORAMIA GPT2-Small 0.22 ± 0.37 - O (1) RM;RN - - 2.82 ± 0.31 1-DP GPT2-Small PANORAMIA GPT-2-Small 0 - O (1) RM;RN - - 0 3-DP GPT2-Small PANORAMIA GPT-2-Small 0 - O (1) RM;RN - - 0 10-DP GPT2-Small PANORAMIA GPT-2-Small 0 - O (1) RM;RN - - 0Table 9 : Privacy audits on GPT-2-Small target models with varying privacy guarantees.Discussion. In the image data modality, we observe that the auditing outcome (ε values for PANORAMIA and ϵ for O(1)) can be different for DP models with the same ϵ values (Table 10 ).We hypothesize that the auditing results may relate more to level of overfitting than the target ϵ values in trained DP models. The difference between train and test accuracies could be a possible indicator that has a stronger relationship with the auditing outcome. We also observe that O(1) shows a faster increase in ϵ for DP models with higher targeted ϵ values. We believe it depends on the actual ratio of the correct and total number of predicted samples, since O(1) considers both true positives and true negatives while PANORAMIA considers true positives only. We leave these questions for future work.In the text data modality, the membership inference attacks (MIAs) relying on loss thresholds in privacy audits, without retraining or access to canaries, may not be sufficiently strong to effectively classify members and non-members. .For all c > 0 we say G is (c, γ)-close to D if D γ ∞ (D∥G) ≤ c, i.e. ∀x ∈ X , P D [x] ≤ e c P G [x] + γ or e -c (P D [x] -γ) ≤ P G [x].First we prove a similar result as Lemma 5.5 (Steinke et al., 2023) but with the one-sided inequality of (c, γ)-closeness and the case when we additionally observe output from a pure-DP mechanism. Lemma 1. Let G, D be probability distributions over Y, fix c, γ ≥ 0. Suppose for all measurable S ⊂ Y we have e -c (D(S) -γ) ≤ G(S). Then ∃γ ′ ∈ [0, γ] and D ′ , D ′′ , G ′ , G ′′ such that the following properties are satisfied.(1) D, G can be expressed as a convex combinationD = (1 -γ ′ )D ′ + γ ′ D ′′ , G = (1 -γ ′ )G ′ + γ ′ G ′′ ;(2) for all measurable S ⊂ Y, we have e -c D ′ (S) ≤ G ′ (S);(3) there exists measurable S ⊂ Y such that D ′′ (S) = 1, ∀S ′ ⊂ S, D(S ′ ) ≥ G(S ′ ).Let M denote an arbitrary ϵ-DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P, Q be probability distributions over Z which are distributions of M (Y ) when Y is sampled from D and G respectively. Fix ϵ ≥ 0, suppose for all measurable T ⊂ Z we have P(T ) ≤ e ϵ Q(T ) and Q(T ) ≤ e ϵ P(T ). When additionally observing the output from a DP mechanism we have the following properties. Let (D, P), (G, Q) be the joint distribution over (Y, Z) which we assume D ⊥ ⊥ P, G ⊥ ⊥ Q.(4) (D, P), (G, Q) can be expressed as a convex combination DP = ((1-γ ′ )D ′ + γ ′ D ′′ )P, GQ = ((1 -γ ′ )G ′ + γ ′ G ′′ )Q;(5) for all measurable S ⊂ Y, T ⊂ Z, we have e -c e -ϵ D ′ (S)P(T ) ≤ G ′ (S)Q(T );(6) there exists measurable S ⊂ Y, T ⊂ Z such that D ′′ (S)P(T ) = 1, ∀S ′ ⊂ S, T ′ ⊂ T, D(S ′ )P(T ) ≥ G(S ′ )P(T ).Proof. For the edge cases of γ = 0, the results hold with γ ′ = 0, D ′ = D, G ′ = G; similarly when γ = 1, the results hold with γ ′ = 1, D ′′ = D, G ′′ = G, D ′ = G ′ . Let c ′ ∈ [0, c], γ 1 , γ 2 ∈ (0, 1), define distribution G ′ , G ′′ , D ′ , D ′′ as follows. For all points y ∈ Y, D ′ (y) = min{D(y), e c ′ G(y)} 1 -γ 1 , D ′′ (y) = D(y) -(1 -γ 1 )D ′ (y) γ 1 = max{0, D(y) -e c ′ G(y)} γ 1 , G ′ (y) = G(y) 1 -γ 2 , G ′′ (y) = G(y) -(1 -γ 2 )G ′ (y) γ 2 = 0. By construction (1 -γ 1 )D ′ + γ 1 D ′′ = D, (1 -γ 2 )G ′ + γ 2 G ′′ = G,EQUATIONG ′ (y) D ′ (y) • Q(M (y)) P(M (y)) = G(y) min{D(y), e c ′ G(y)} • Q(M (y)) P(M (y)) ≥ e -c ′ e -ϵ ≥ e -c e -ϵ ,where Q(M (y)) P(M (y)) ≥ e -ϵ since M is an ϵ-DP mechanism.Similar to Lemma 5.6 (Steinke et al., 2023) , next we prove a Bayesian version of Lemma 1 which is used to prove the main result afterwards. Lemma 2. Let G, D be probability distributions over Y, fix c, γ ≥ 0, suppose for all measurable S ⊂ Y we have e -c (D(S)-γ) ≤ G(S). Then there exists a randomized function E D,G : Y → {0, 1} with the following properties. Suppose X ∼ Bernoulli( 1 2 ), if X = 1 sample Y ∼ D, and if X = 0 sample Y ∼ G. Then for all y ∈ Y we have,P X∼Bernoulli( 1 2 ) Y ←XD+(1-X)G [X = 1, E D,G (Y ) = 1|Y = y] ≤ e c 1 + e c , E Y ∼D [E D,G (Y )] ≥ 1 -γ, E Y ∼G [E D,G (Y )] ≥ 1 -γ.Let M denote an arbitrary ϵ-DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P, Q be probability distributions over Z which are distributions of M (Y ) when Y is sampled from D and G respectively. Fix ϵ ≥ 0, suppose for all measurable T ⊂ Z we have P(T ) ≤ e ϵ Q(T ) and Q(T ) ≤ e ϵ P(T ). Let (D, P), (G, Q) denote the joint distributions over (Y, Z) which we assume D ⊥ ⊥ P, G ⊥ ⊥ Q. Then there exists a randomized function E (D,P),(G,Q) : (Y, Z) → {0, 1} such that for all y ∈ Y, M (y) ∈ Z we have,P X∼Bernoulli( 1 2 ) Y ←XD+(1-X)G M (Y )←M (XD+(1-X)G) [X = 1, E (D,P),(G,Q) (Y, M (Y )) = 1|Y = y, M (Y ) = f ] ≤ e c+ϵ 1 + e c+ϵ , E Y ∼D,M (Y )∼P [E (D,P),(G,Q) (Y, M (Y ))] ≥ 1 -γ, E Y ∼G,M (Y )∼Q [E (D,P),(G,Q) (Y, M (Y ))] ≥ 1 -γ.Proof. We apply the decomposition from Lemma 1 and denote D ′ , D ′′ , G ′ , G ′′ , γ ′ as is. We define E D,G : Y → {0, 1} by,P[E D,G (y) = 1] = (1 -γ ′ ) D ′ (y) D(y) = 1 - γ ′ D ′′ (y) D(y) .For any y ∈ Y we have,P[X = 1, E D,G (Y ) = 1|Y = y] = P[X = 1|Y = y, E D,G (Y ) = 1]P[E D,G (Y ) = 1|Y = y] = P[X = 1|Y = y]P[E D,G (y) = 1] = P[Y = y|X = 1]P[X = 1] P[Y = y] P[E D,G (y) = 1] = D(y) D(y) + G(y) P[E D,G (y) = 1] = (1 -γ ′ )D ′ (y) + γ ′ D ′′ (y) (1 -γ ′ )D ′ (y) + γ ′ D ′′ (y) + (1 -γ ′ )G ′ (y) P[E G,D (y) = 1] = 1 + γ ′ D ′′ (y) (1-γ ′ )D ′ (y) 1 + γ ′ D ′′ (y) (1-γ ′ )D ′ (y) + G ′ (y) D ′ (y) P[E G,D (y) = 1] ≤ 1 + γ ′ D ′′ (y) (1-γ ′ )D ′ (y) 1 + 0 + e -c P[E G,D (y) = 1] = 1 1 + e -c • (1 -γ ′ )D ′ (y) + γ ′ D ′′ (y) (1 -γ ′ )D ′ (y) • P[E G,D (y) = 1] = 1 1 + e -c • D(y) (1 -γ ′ )D ′ (y) • P[E G,D (y) = 1] = 1 1 + e -c = e c 1 + e c .The first equality uses that P[A, B|C] = P[A|B, C]P [B|C] . The second equality holds since we assume the internal randomness of E is independent of everything else. The third equality uses the Bayes' rule. The fourth equality uses the definition of the randomized function E. The fifth equality uses the decomposition of Lemma 1. After rearranging and use the properties from Lemma 1 we obtain the first inequality, then the rest follows from simplifications. Furthermore we have the following,E Y ∼D [E D,G ] = Y D(y)P[E D,G = 1]dy = Y (1 -γ ′ )D ′ (y)dy = 1 -γ ′ ≥ 1 -γ, E Y ∼G [E D,G ] = 1 -γ ′ E Y ∼G D ′′ (y) D(y) = 1 -γ ′ Y G(y) D(y) • D ′′ (y)dy ≥ 1 -γ ′ Y D ′′ (y)dy = 1 -γ ′ ≥ 1 -γ.We follow a similar procedure to prove the second part of the Lemma. We define E (D,P),(G,Q) : (Y, Z) → {0, 1} by,P[E (D,P),(G,Q) (y, M (y)) = 1] = P[E D,G (y) = 1]With similar reasoning, for any y ∈ Y, M (y) ∈ Z we have,P[X = 1, E (D,P),(G,Q) (Y, M (Y )) = 1|Y = y, M (Y ) = f ] = P[Y = y, M (Y ) = f |X = 1]P[X = 1] P[Y = y, M (Y ) = f ] P[E (D,P),(G,Q) (y, M (y)) = 1] = D(y)P(M (y)) D(y)P(M (y)) + G(y)Q(M (y)) P[E (D,P),(G,Q) (y, M (y)) = 1] = ((1 -γ ′ )D ′ (y) + γ ′ D ′′ (y))P(M (y)) ((1 -γ ′ )D ′ (y) + γ ′ D ′′ (y))P(M (y)) + (1 -γ ′ )G ′ (y)Q(M (y)) P[E (D,P),(G,Q) (y, M (y)) = 1] = 1 + γ ′ D ′′ (y)P(M (y))(1-γ ′ )D ′ (y)P(M (y))1 + γ ′ D ′′ (y)P(M (y))(1-γ ′ )D ′ (y)P(M (y)) + G ′ (y)Q(M (y)) D ′ (y)P(M (y))P[E (D,P),(G,Q) (y, M (y)) = 1] ≤ 1 + γ ′ D ′′ (y) (1-γ ′ )D ′ (y) 1 + 0 + e -c e -ϵ P[E (D,P),(G,Q) (y, M (y)) = 1] = 1 1 + e -c e -ϵ • D(y) (1 -γ ′ )D ′ (y) • P[E (D,P),(G,Q) (y, M (y)) = 1] = 1 1 + e -c e -ϵ = e c+ϵ 1 + e c+ϵ .Since we define the success probability with the same expression, the expected success of the randomized function directly follows,E Y ∼D,M (Y )∼P [E (D,P),(G,Q) (Y, M (Y ))] = E Y ∼D [E D,G (Y )] ≥ 1 -γ, E Y ∼G,M (Y )∼Q [E (D,P),(G,Q) (Y, M (Y ))] = E Y ∼G [E D,G (Y )] ≥ 1 -γ.Next we use Lemma 2 to prove the following Proposition which is used to form statistical tests on hypothesis H. Proposition 3. Let G be (c, γ)-close, f be output of an ϵ-DP mechanism, T b ≜ B(S, X) be the guess from the baseline, T a ≜ A(S, X, f ) be the guess from the membership audit. Let T b , T a ∈ [0, 1] m be bounded. Then, for all v ∈ R and all t in the support of T :P S,X,T b m i=1 T b i • S i ≥ v | T b = t b ≤ P S ′ ∼Bernoulli( e c 1+e c ) m ,F F (t b ) + m i=1 t b i • S ′ i ≥ v , P S,X,T a m i=1 T a i • S i ≥ v | T a = t a ≤ P S ′ ∼Bernoulli( e c+ϵ 1+e c+ϵ ) m ,F F (t a ) + m i=1 t a i • S ′ i ≥ v , where F is independent from S ′ , F (T b ), F (T a ) is supported on {0, 1, . . . , m} and E T b ,F [F (T b )] = E T a ,F [F (T a )] ≤ 2mγ.Proof. Let B(s ≤i ) denote the distribution on [0, 1] m obtained by conditioning B on past S <i = s <i and X ≤i = x ≤i . By Lemma 2, for all t b ∈ [0, 1] m we haveP[S i = 1, E B(s<i,1),B(s<i,0) (T b ) = 1|T b = t b , S <i = s <i , X ≤i = x ≤i ] ≤ e c 1 + e c , E[E B(s ≤i,1 ),B(s ≤i,0 ) (T b ) = 1|S <i = s <i , S i = 0] ≥ 1 -γ, E[E B(s ≤i,1 ),B(s ≤i,0 ) (T b ) = 1|S <i = s <i , S i = 1] ≥ 1 -γ.Using the law of total probability we get,P[S i = 1, E B(s<i,1),B(s<i,0) (T b ) = 1|T b = t b , S <i = s <i ] = x≤i P[S i = 1, E B(s<i,1),B(s<i,0) (T b ) = 1|T b = t b , S <i = s <i , X ≤i = x ≤i ] P[X ≤i = x ≤i |T b = t b , S <i = s <i ] ≤ e c 1 + e c x≤i P[X ≤i = x ≤i |T b = t b , S <i = s <i ] = e c 1 + e c .Applying the union bound on S i we can bound the expectation of success events by,E[E B(s ≤i,1 ),B(s ≤i,0 ) (T b )|S <i = s <i , S i = s i ] ≥ 1 -2γ. By induction, for k ∈ [m] define Wk (s, t b ) := k i=1 t b i • s i • E B(s<i,1),B(s<i,0) (t b ), then Wk (S, t b ) is stochastically dominated by Wk (t b ) := k i=1 t b i • Ši (t)where Ši (t) ∼ Bernoulli( e c 1+e c ). Let the failing events F be, F (s, t b ) := m i=1 1{E B(s ≤i,1 ),B(s ≤i,0 ) (t b ) = 0}, then we have W m (s, t b ) := m i=1 t b i • s i = Wm (s, t b ) + F (s,P[S i = 1, E A(s<i,1),A(s<i,0) (T a ) = 1|T a = t a , S <i = s <i , X ≤i = x ≤i ] ≤ e c+ϵ 1 + e c+ϵ , E[E A(s ≤i,1 ),A(s ≤i,0 ) (T a ) = 1|S <i = s <i , S i = 0] ≥ 1 -γ, E[E A(s ≤i,1 ),A(s ≤i,0 ) (T a ) = 1|S <i = s <i , S i = 1] ≥ 1 -γ.Similar to Theorem 5.2 (Steinke et al., 2023) we could compute for the optimal distribution F (t b ) and F (t a ) by formulating it into a linear programming problem. Similar to Corollary 1, to test Hypothesis H together we would apply Union bound on the two parts of the hypothesis as in Proposition 3.In this section we consider the case where we have a (c, γ)-close generator and audit for an approximate DP mechanism. We modify the results to account for the failing events from these two sources of randomness. We start by proving a similar version of Lemma 2 where we use both the decomposition of G, D from Lemma 1 and the decomposition of P, Q from Lemma 5.5 (Steinke et al., 2023) .Lemma 3. Let G, D be probability distributions over Y, fix c, γ ≥ 0, suppose for all measurable S ⊂ Y we have e -c (D(S) -γ) ≤ G(S). Let M denote an arbitrary (ϵ, δ)-DP mechanism which takes input Y sampled from distribution G or D and outputs f . Let P, Q be probability distributions over Z which are distributions of M (Y ) when Y is sampled from D and G respectively. Fix ϵ, δ ≥ 0, suppose for all measurable T ⊂ Z we have P(T ) ≤ e ϵ Q(T ) + δ and Q(T ) ≤ e ϵ P(T ) + δ. Let (D, P), (G, Q) denote the joint distributions over (Y, Z) which we assume D ⊥ ⊥ P, G ⊥ ⊥ Q. Then there exists a randomized function E (D,P),(G,Q) : (Y, Z) → {0, 1} such that for all y ∈ Y, M (y) ∈ Z we have,P X∼Bernoulli( 1 2 ) Y ←XD+(1-X)G M (Y )←M (XD+(1-X)G) [X = 1, E (D,P),(G,Q) (Y, M (Y )) = 1|Y = y, M (Y ) = f ] ≤ e c+ϵ 1 + e c+ϵ , E Y ∼D,M (Y )∼P [E (D,P),(G,Q) (Y, M (Y ))] ≥ (1 -γ)(1 -δ), E Y ∼G,M (Y )∼Q [E (D,P),(G,Q) (Y, M (Y ))] ≥ (1 -γ)(1 -δ).Proof. For ease of notation we shorthandF = M (Y ), E = E (D,P),(G,Q) , D ′ = D ′ (y), G ′ = G ′ (y), D ′′ = D ′′ (y), G ′′ = G ′′ (y), P ′ = P ′ (f ), Q ′ = Q ′ (f ), P ′′ = P ′′ (f ), Q ′′ = Q ′′ (f ) below. We define E : (Y, Z) → {0, 1} by, P[E(y, f ) = 1] = (1 -γ ′ )(1 -δ ′ )D ′ P ′ DP .For any y ∈ Y, M (y) ∈ Z we have,P[X = 1, E(Y, F ) = 1|Y = y, F = f ] = P[Y = y, F = f |X = 1]P[X = 1] P[Y = y, F = f ] P[E(y, f ) = 1] = D(y)P(f ) D(y)P(f ) + G(y)Q(f ) P[E(y, f ) = 1] = ((1 -γ ′ )D ′ + γ ′ D ′′ )((1 -δ ′ )P ′ + δ ′ P ′′ ) ((1 -γ ′ )D ′ + γ ′ D ′′ )((1 -δ ′ )P ′ + δ ′ P ′′ ) + (1 -γ ′ )G ′ ((1 -δ ′ )Q ′ + δ ′ Q ′′ ) P[E(y, f ) = 1] = 1 + δ ′ D ′ P ′′ (1-δ ′ )D ′ P ′ + γ ′ D ′′ P ′ (1-γ ′ )D ′ P ′ + γ ′ δ ′ D ′′ P ′′ (1-γ ′ )(1-δ ′ )D ′ P ′ 1 + δ ′ D ′ P ′′ (1-δ ′ )D ′ P ′ + γ ′ D ′′ P ′ (1-γ ′ )D ′ P ′ + γ ′ δ ′ D ′′ P ′′ (1-γ ′ )(1-δ ′ )D ′ P ′ + G ′ Q ′ D ′ P ′ + δ ′ G ′ Q ′′ (1-δ ′ )D ′ P ′ P[E(y, f ) = 1] ≤ 1 + δ ′ D ′ P ′′ (1-δ ′ )D ′ P ′ + γ ′ D ′′ P ′ (1-γ ′ )D ′ P ′ + γ ′ δ ′ D ′′ P ′′ (1-γ ′ )(1-δ ′ )D ′ P ′ 1 + e -c e -ϵ P[E(y, f ) = 1] = 1 1 + e -c e -ϵ (1 -δ ′ )(1 -γ ′ )D ′ P ′ + δ ′ (1 -γ ′ )D ′ P ′′ + γ ′ (1 -δ ′ )D ′′ P ′ + γ ′ δ ′ D ′′ P ′′ (1 -γ ′ )(1 -δ ′ )D ′ P ′ • P[E(y, f ) = 1] = 1 1 + e -c e -ϵ DP (1 -γ ′ )(1 -δ ′ )D ′ P ′ P[E(y, f ) = 1] = 1 1 + e -c e -ϵ = e c+ϵ 1 + e c+ϵ .Using the assumption that D ⊥ ⊥ P, G ⊥ ⊥ Q, we account for the expectation of success events as follows,E Y ∼D,M (Y )∼P [E(Y, F )] = D(y)P(f )P[E = 1]dydf = D(y)P(f ) (1 -γ ′ )(1 -δ ′ )D ′ (y)P ′ (f ) D(y)P(f ) dydf = (1 -γ ′ )(1 -δ ′ ) D ′ (y)dy P ′ (f )df = (1 -γ ′ )(1 -δ ′ ) ≥ (1 -γ)(1 -δ).Note that we can rewrite the success probability as,P[E(y, f ) = 1] = 1 - γ ′ (1 -δ ′ )D ′′ P ′ DP - (1 -γ ′ )δ ′ D ′ P ′′ DP - γ ′ δ ′ D ′′ P ′′ DP .Then we have, EQUATION-γ ′ δ ′ D ′′ (y)P ′′ (f )dydf = 1 -γ ′ (1 -δ ′ ) -(1 -γ ′ )δ ′ -γ ′ δ ′ = 1 -γ ′ -δ ′ + γ ′ δ ′ = (1 -γ ′ )(1 -δ ′ ) ≥ (1 -γ)(1 -δ).Next we use Lemma 3 to prove the following Proposition to form the statistical tests on hypothesis H for the (ϵ, δ)-DP version.Proposition 4. Let G be (c, γ)-close, f be output of an (ϵ, δ)-DP mechanism, T b ≜ B(S, X) be the guess from the baseline, T a ≜ A(S, X, f ) be the guess from the membership audit. Let T b , T a ∈ [0, 1] m be bounded. Then, for all v ∈ R and all t in the support of T : Similar to the pure DP case we could optimize for the optimal distribution of F (t b ) and F (t a ) by solving a linear program (Theorem 5.2 in Steinke et al. (2023) ) and apply Union bound as in Corollary 1 to test for Hypothesis H.P S,We empirically evaluate the auditing performance of PANORAMIA under different relaxed level of the generator for both the pure and approximate DP cases. We observe that the overall performance is similar to that of no relaxation. We observe smaller ε (which is analogous to a looser lower bound) as we increase the relaxation level, and in the more extreme cases where we allow many failing events (e.g. when relaxation > 1e-3) we would fail to detect meaningful privacy leakage as ε = 0. Most related privacy auditing works measure the privacy of an ML model by lower-bounding its privacy loss. This usually requires altering the training pipeline of the ML model, either by injecting canaries that act as outliers Carlini et al. (2019) or by using data poisoning attack mechanisms to search for worst-case memorization Jagielski et al. (2020) ; Nasr et al. (2021) . MIAs are also increasingly used in privacy auditing, to estimate the degree of memorization of member data by an ML algorithm by resampling the target algorithm M to bound P (M |in) P (M |out) Jayaraman & Evans (2019) . The auditing procedure usually involves searching for optimal neighboring datasets D, D ′ and sampling the DP outputs M(D), M(D ′ ), to get a Monte Carlo estimate of ϵ. This approach raises important challenges. First, existing search methods for neighboring inputs, involving enumeration or symbolic search, are impossible to scale to large datasets, making it difficult to find optimal dataset pairs. In addition, Monte Carlo estimation requires up to thousands of costly model retrainings to bound ϵ with high confidence. Consequently, existing approaches for auditing ML models predominantly require the re-training of ML models for every (batch of) audit queries, which is computationally expensive in large-scale systems Jagielski et al. (2020); Zanella-Béguelin et al. (2022) ; Lu et al. (2023) . This makes privacy auditing computationally expensive and gives an estimate by averaging over models, which might not reflect the true guarantee of a specific pipeline deployed in practice.Nonetheless, improvements to auditing have been made in a variety of directions. For example, Nasr et al. (2023) and Maddock et al. (2023) have taken advantage of the iterative nature of DP-SGD, auditing individual steps to understand the privacy of the end-to-end algorithm. Andrew et al. (2023) leverage the fact that analyzing MIAs for non-member data does not require re-running the algorithm. Instead, it is possible to re-sample the non-member data point: if the data points are i.i.d. from an asymptotically Gaussian distribution with mean zero and variance 1/d, this enables a closed-form analysis of the non-member case.Recently, Steinke et al. (2023) proposed a novel scheme for auditing differential privacy with O(1) training rounds. This approach enables privacy audits using multiple training examples from the same model training, if examples are included in training independently (which requires control over the training phase, and altering the target model). They demonstrate the effectiveness of this new approach on DP-SGD, in which they achieve meaningful empirical privacy lower bounds by training only one model (the strongest results are achieved by including canaries in the training set though, which is not possible in our setting), whereas standard methods would require training hundreds of models. Our work builds closely on the theory from Steinke et al. (2023) , but introduces a baseline model to account for distribution shifts. The key difference that enables our work to account for member/non-member distribution shifts lies in how we create the audit set in our privacy game (Definition 2). In Steinke et al. (2023) , the audit set is fixed, and data points are randomly assigned to member or non-member by a Bernoulli random variable S. Members are actually used in training the target model f , while non-members are not (so assignment happens before training). In our framework, we take a set of known members (after the fact), and pair each point with a non-member(generated i.i.d. from the generator distribution). We then flip S to sample which data point of each pair will be shown to the "auditor" (MIA/baseline) for testing, thereby creating the test task of our privacy measurement. When we replace our generated data with in-distribution independent non-members (PANORAMIA RM;RN), we exactly enforce the same independence as Steinke et al. (2023) (and as a result have c = 0, and ε is a lower-bound on ϵ), except that we "waste" some member data by drawing our auditing game after the fact. The difference in analysis accounts for distribution shifts when our non-members are generated.The field of MIA without direct connections to privacy leakage measurement has also seen a lot of recent activity, with new proposals to improve the strength of MIAs in various settings (Carlini et al., 2022b; Nasr et al., 2019; Suri et al.; Zarifzadeh et al., 2023) . While it is not the focus of this paper, an interesting avenue for future work is to port ideas from these new MIAs to improve PANORAMIA's MIA and baseline. Such transfer of MIA progress could lead to more powerful measurements with PANORAMIA. More closely related to our proposal, a recent set of works on Das et al. (2024) ; Duan et al. (2024) ; Meeus et al. (2024) MIAs on foundation models has observed that current evaluations are limited due a lack of availability of non-member data. Indeed, foundation models typical include all known data at the time of their training, and there is no well known public set of in-distribution non-member data. Evaluation tasks typical rely on more recent data, which is known to consist in non-members, but suffers from distribution shifts. Das et al. (2024) ; Duan et al. (2024) ; Meeus et al. (2024) all show that such a shift invalidates MIA evaluations by showing that "blind attacks" (i.e., our baselines) perform better than proposed MIAs. One could apply our proposed framework to such a setting: when the baseline in our framework performs better than the MIA, PANORAMIA returns a privacy loss measurement of zero instead of misleading measurements of MIA performance. We thus believe that PANORAMIA is an important step towards a theory for rigorous evaluations of membership inference for MIA models.Code available here: https://github.com/ubc-systopia/panoramia-privacy-measurement. 
 
 Appendix:  


Decision: The paper presents a method for privacy auditing without access to real non-members by using generated examples as non-members. The paper builds a theoretical framework for addressing the auditing question approximately and presents experiments to test the method on a number of tasks in diverse domains (images, NLP, tabular data).

The reviews are generally positive, commending the paper on addressing a relevant and interesting problem and experiments covering diverse tasks, but raise concerns about correctness of certain results, lack of careful evaluation of how the quality of the generator impacts the results in practice, and the fact that the method does not produce strict lower bounds.

The only reviewer recommending rejection, Reviewer Z4qp, raised concerns about the correctness of some key results, which likely contributed to the negative review. The authors have addressed these in their response, but the reviewer has not reacted to the response. It would seem to me that the most serious concerns regarding the paper have thus been addressed although the review has not been updated to reflect this.

In light of these reviews, I believe the paper should be accepted.

I would however recommend the authors to consider the comments raised by the reviewers and add a more thorough evaluation and discussion of the impact of the quality of the generator to the results. It appears to me that the training protocol for the generators used in the experiments is really elaborate. This raises concern that it might be very difficult to replicate the strong results from the image and NLP models without equally arduous generator training.

Minor comments:
* The authors claim to use top-$p$ sampling in the NLP generator (e.g. line 224), but note on line 533 of the Appendix that they use parameter $p=1$ and $k=200$ for top-$k$. I believe this would be more correctly labeled as top-$k$, as top-$p$ filtering has no effect under these parameters.
* Unlike what they claim in the paper checklist response #8, the authors do not disclose the compute time required by the experiments.

Reviews: review: paper_summary: paper_summary: The authors propose a privacy auditing technique that utilizes partial access to member data to generate synthetic non-member data, which in-turn is used to train a meta-classifier that can be used to empirically measure privacy leakage relating to record membership. The authors evaluate their technique on models as large as GPT2 and find correlation between audit scores and the expected leakage from models.
strength_weakness: strength_weakness: - As an auditor, access to validation/test data used in model training would not be a far stretch- why not use the non-member set as used in standard MIA setup (validation/test data). What is the added benefit from this extra step of generating synthetic non-members?

- Regarding the contributions, (1) and (2) has already been explored by [1, 2]. While the method in [1] does not currently support large models/datasets like CIFAR10, it would be nice to highlight differences here. If you indeed have knowledge of a decent-chunk size of members and non-members, I'd imagine you could do something better like [2], and other related methods.

- I have some concerns over the dependency on how well the baseline in/out distribution detection system works. As a concrete example consider Figure 7(a, b) - even as a human I see a very clear difference in resolution of the generated images and find it hard to believe that the distinguisher does not work well here. Even a non-ML technique (that can work around with blurring) would work pretty well here. 

- Table 1: Accuracy of models here is not good enough, especially when using entire data, with very clear signs of overfitting. If using the entire data (and not half split, as in most MIA evaluations [3, 4] where even with half the dataset test accuracy is ~92%), should be able to train well-performing models that do not overfit so heavily. Please see [this resource](https://paperswithcode.com/sota/image-classification-on-cifar-10)

- Figure 1: While the trends suggest that the proposed audit correlates with actual leakage, one could also argue that (100 - test accuracy) is also a useful privacy metric in this comparison given the correlation. The utility of the audit would be more apparent, then, when studying models that have **comparable** test performance, but have (by design) different leakage, perhaps focusing on moderate/large values of $\epsilon$ with Differential Privacy-training.

- L288-290: (1) and (2) are ambiguously enforced- if you have a large portion of train data, you can control process in most cases, and also obtain non-member data. As far as (3) is concerned, that is a compute-related constraint, not a threat model difference. The auditor could, for instance, use available data knowledge to train "in" models.

- L331-334: I do not find the justification behind having access to partial member data convincing. While I am okay with just stating that "this is a possible limitation, but okay to assume for an auditor", but relating it to how it could be useful in situations like FL is not practical. For instance, here empirical experiments use more than 1/5th of the data; no FL training will have just 5 participants. This is closer to a specific case of distributed learning, or pure data aggregation. Even in such cases, the data distributions per client will be considerably different, as opposed to the experiments here which have uniform samples. 

## Minor comments

- Please consider a more descriptive abstract. "scheme" here is not very descriptive
- L121: "We formalize a privacy game" - this is standard membership inference evaluation and not a contribution of this work. Please rephrase to make this clear.
- L493-494: Please provide direct distinguishing accuracies of these baseline classifiers for reference and easier understanding.

### References
- [1] Suri, Anshuman, Xiao Zhang, and David Evans. "Do Parameters Reveal More than Loss for Membership Inference?." High-dimensional Learning Dynamics 2024: The Emergence of Structure and Reasoning.
- [2] Nasr, Milad, Reza Shokri, and Amir Houmansadr. "Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning." 2019 IEEE symposium on security and privacy (SP). IEEE, 2019.
- [3] Carlini, Nicholas, et al. "Membership inference attacks from first principles." 2022 IEEE Symposium on Security and Privacy (SP). IEEE, 2022.
- [4] Zarifzadeh, Sajjad, Philippe Cheng-Jie Marc Liu, and Reza Shokri. "Low-cost high-power membership inference by boosting relativity." (2023).
questions: questions: - Figure 3a suggests an ordering of 100 > 20 > 50 in terms of leakage for high-recall region, whereas numbers in Table 1 suggest that the trends should be like 100 > 50 > 20 if the proposed method is indeed measuring what it is supposed to. Why is that so?

- L480-481: this means the "labeler" is much weaker and might be generating incorrect labels more often? Are there any numbers for what the actual test performance of this labeler is?
limitations: limitations: There are a few statements that currently serve as justification for limitations (see above) but should just be posed directly as base assumptions. Apart from that (and some potential shortcomings in models used for evaluation), most limitations are already stated in the paper.
paper_summary: paper_summary: The paper proposes a novel way for privacy auditing of machine learning models through MIAs. The framework they propose, panoramia, aims to audit the privacy of a ML model post-hoc (so with no control over the training procedure), and with access to a known member subdataset. The method first consists of training a generative model on the known subset of members, which is used to then generate synthetic data samples from the same distribution. These synthetic points are then used as non-members, which combined with the known member dataset allows to fit and apply an MIA on the target model. Importantly, authors recognize that there might be a distribution shift between members and synthetic non-members, so they fit a classifier without leveraging the target model as a baseline. 

Next, they use the difference between the MIA performance (thus using the target model) and this baseline to estimate the privacy loss. Authors provide the formula (and add proof in appendix) to compute a value of epsilon approximating a lower bound on epsilon-DP. 

They further apply the privacy auditing to three kinds of ML models (image classification, language modeling, and tabular data classification). Authors consider models with varying degree of overfitting, DP training, and increasing the amount of member data available to the auditor.
strength_weakness: strength_weakness: - Originality: Authors should include an appropriate related work section, touching on other privacy auditing techniques and potentially other (post-hoc) membership inference attacks. 
- Quality: The proposed method strongly depends on the quality of a generator and the baseline MIA, the impact and limitations of which can be further explored (see questions/limitations). 
- Clarity: I find that the paper's clarity can be improved significantly. Especially the results section (tables, text) is quite notation heavy and hard to follow what everything refers to. 
- Significance: While technically interesting, the relevance of a technique to compute a proxy lower bound privacy loss in practice needs more compelling motivation (see questions).
questions: questions: - I understand how c-closeness allows to estimate the quality of the used generator. However, what I struggle to understand, is what happens when you develop a generator that is perfect (c=0) which just samples randomly from the known member subset. Then, a baseline classifier would not be able to distinguish members from non-members, and nor will an MIA, leading to a privacy estimate that will be far off. In less extreme cases, the generator might be slightly overfitted and indeed generate non-members very similar to members, which might also impact the MIA performance. Am I correct that this could have a significant impact on the validity of the procedure? And if so, how would you address the concern? 
- More generally, Panoramia largely depends on a good generator and a baseline MIA. Can authors elaborate on the associated limitations? For instance, can good generators can be developed across all use-cases (smaller datasets, data modalities etc)? And how should a baseline performance be developed or evaluated to be used as part of the panoramia framework?
- I understand that the method authors provide does not give a formal lower bound for the privacy loss, but rather a proxy for it. In practice, if indeed a hospital as part of an FL setup would like to assess the privacy leakage incurred by their data, why would they opt to compute an estimate for epsilon using your method? Instead, they could for instance generate non-members in the same way as panoramia and just quantify the MIA performance with an AUC or TPR at low FPR compared to a baseline. In general, authors should further motivate the relevance of a proxy for a lower bound privacy loss in practice. 
- To further motivate post-hoc privacy auditing (without any control of the training data), I wonder if it makes sense to also emphasize the context of generative AI models such as LLMs? These models are increasingly trained on all the data model developers can acquire so  the absence of non-member data is very real in practice. 
- Can authors add a related work section?
limitations: limitations: Authors are very clear that their method only provides a proxy for the lower bound of epsilon and thus carefully caveat their method. However, can authors elaborate on the limitations associated with the development of both a generator and a baseline, both of which seem fundamental for panoramia. Currently, their implementation seems rather ad-hoc than adequately discussed for wider applications.
paper_summary: paper_summary: This paper proposes a novel privacy auditing procedure called PANORAMIA. The method works with a single model that uses all available training data, with no modifications to the training procedure, and with only access to a subset of the training data. This is achieved by synthetically generating non-member samples for auditing. The method produces an estimate of the $\epsilon$ differential privacy parameter, which, however, is not necessarily a lower bound.

Summary of the method given a model and its training data:
1. Train a generative model on a subset of the training data.
2. Use the remaining training samples as "members" for auditing.
3. Generate an equal amount of "non-members".
4. Split both into training and test sets.
5. Fit a baseline classifier that distinguishes synthetic and real data on the training set (only input are samples).
6. Fit a membership inference classifier on the training set (inputs are samples and model statistics).
7. Use predictions on the test set and hypothesis testing to obtain a confidence interval for quantities of interest.

The hypothesis is of the form "The generative model is $c$-close to the true data distribution and the training procedure is $\epsilon$-DP", where $c$ is a distance measure defined in this paper. The baseline predictions are used to reject the first claim, and the membership predictions to reject the second one. Hence, the returned $\epsilon$ is a lower bound on the true $\epsilon$ only if the generative model is indeed at least $c$-close to the data distribution. The theoretical analysis relies on results from the (1) procedure by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) but adapts them to the relaxed setting in this paper.

For evaluation, the paper applies the new auditing procedure to various CNNs with and without DP guarantees on image data, small GPT2 models on text, and on tabular data. As baselines, the evaluation uses the (1) method by [Steinke et al., 2023](https://openreview.net/forum?id=f38EY21lBw) and real instead of generated members. PANORAMIA does not outperform the baselines, but achieves reasonably close results on image and text data.
strength_weakness: strength_weakness: - The considered privacy semantics are partially misleading and could be made more clear.
	- The presented auditing game is incomplete; Definition 2 only describes how samples are selected but should also include the goal and actions of the adversary.
	- The paper states (as a benefit) that PANORAMIA audits a target model, not the algorithm (L46--47). However, the results of the procedure are DP parameters, and DP is always a property of an algorithm (not model). If the considered semantics are meant to be different from DP, they should be made more clear and discussed.
	- Similarly, the paper claims that PANORAMIA does not require worst-case canaries because it audits the target dataset (via a model). However, this ignores that certain samples in the training data might be more vulnerable; hence, auditing should still consider the worst-case sample *in the training data* (especially since privacy is averaged over the dataset). This might be an explanation why the reported "lower bounds on $\epsilon$" are still far from tight in Figure 6c (as is the prior (1) procedure in a white-box setting).
- Relatedly, the definition of c-closeness (Definition 3) might not require a generator to capture the tail of the data distribution. However, this tail contains outliers that are often particularly vulnerable to privacy leakage. Hence, ignoring such outliers might significantly underestimate the $\epsilon$ lower bound. I would have appreciated a more thorough discussion of this limitation (or a clarification).
- The paper could be more explicit about whether PANORAMIA is intended to be a procedure to be used in practice, or an important stepping stone towards more practical procedures. Right now, PANORAMIA achieves worse results than the existing (1) method in a white-box setting, and requires training of a strong synthetic data generator (which might be non-trivial). Nevertheless, I believe this paper is still relevant from a conceptual perspective as a path for future work.
- There are minor notation issues in Proposition 1 and 2: the statement mentions only $T$ and $t$, but the inequality uses $T^b$ and $t^b$ (or $T^a$ and $t^a$). Additionally, the left-hand sides both take probabilities over $T^b$/$T^a$ but simultaneously condition on the respective random variable. Fixing those points makes the propositions clearer. Also, the last sentence on L148 seems misplaced.
questions: questions: 1. Might there be a way to use the membership classifier itself as a baseline, e.g., by somehow averaging over possible values for model statistics? This would avoid skewed scenarios where the MIA is better at detecting synthetic data than the baseline detector (even when ignoring membership signal).
2. In Figure 6/Section 5.3, do all models use (approximately) the same number of training samples? If not, could there be confounding effects (e.g., if models trained on fewer samples leak more privacy)?
3. Is there a path to extend this paper's method to always yield a lower bound on $\epsilon$ (w.h.p.)? Or would this require radical changes?
limitations: limitations: The authors are very transparent about all limitations of their work and discuss them transparently.
paper_summary: paper_summary: The paper proposes a method to estimate a lower bound on the (pure) Differential Privacy parameter $\epsilon$ for already trained machine learning models using a post-hoc empirical evaluation based on conducting Membership Inference Attacks. Building on the analysis by Steinke et al., 2023, for auditing DP with $O(1)$ training runs, this paper tweaks the setting by using synthetically generated canary data points that closely resemble the training data population, rather than inserting or omitting pathologically crafted canaries that degrade the ML model's performance. This approach is useful because if the canary distribution and the training distribution are (nearly) identical, an already trained model can be considered as the output of the training algorithm on a random partition of the combined training and canary datasets. This means any post-hoc auditing analysis based on membership inference on a randomly and independently selected training dataset yields a valid lower bound on $\epsilon$. To get such a canary distribution, the authors suggest training a synthetic data generation model to create a canary dataset to get a distribution similar to the population. Additionally, the authors extend Steinke et al.'s analysis to situations where the synthetic data distribution is $c$-close to the true population in a DP-like divergence, although the estimator does not technically yield a lower bound in this case. Through extensive empirical evaluations, the paper demonstrates that the proposed estimation technique is useful as it provides reasonable values that approximate the DP lower bounds.
strength_weakness: strength_weakness: - I'm not entirely certain that the analysis presented in Proposition 2 is correct. In the proof of Proposition 2 (lines 433 and 434), the authors use the fact that the model $f$ is $\epsilon$-DP with respect to the Bernoulli random variables $S$. But in Algorithm 1, we see that $f$ was trained on $D_{f}$ in phase 1, which is independent of $S$ sampled in phase 2. So, the model $f$ is independent of the selection $S$ (i.e., $f \bot S$). In other words, the equation after line 434 should evaluate to 1, as given $x_i$, the prediction $f(x_i)$ remains the same whether $S_i = 0$ or $S_i = 1$. Perhaps Algorithm 1 requires one training run like Steinke et al. where the model $f$ is trained on $X$ as described in line 2 of Algorithm 1? If that is the case, then the claim that the auditing does not require retraining becomes invalid. If I'm mistaken, could you explain Proposition 2 further?

- In line 196, the authors mention that Algorithm 1 (lines 4-7) does a sweep over all the recall values of the attack, and they adjust the overall significance-level or the $p$-value by taking a union bound over all equation (1) with the significance level discounted to $\beta \leftarrow \beta / m$. The value of $m$ in the experiments ranges from 500 to 30,000. For such values,  assuming $\beta \leq 0.05$, the sweeping over all recall values requires $\sqrt{\log(m/\beta)} \approx 2.4$ whereas when the level of recall is predetermined, the factor in equation (1) is only $\sqrt{\log(1/\beta)} \approx 1.15$. So, I'm not sure if doing a sweep over recall values will give larger DP estimate values.

- Use of synthetic data from a distribution that is $c$-close (with a small $c$) can make it difficult for the membership inference attacks to work well. If the goal is auditing DP, perhaps the higher MIA precision for a given recall outweighs the drop in ML model's performance. 

- The figures haven't been explained very well. In particular, the theoretical maximum precision in Figure 3 and Figure 12 and the empirical maximum value in Figure 4 aren't clear.

- The paper only studies the problem of estimating pure DP and does not present an operationalizable algorithm for $(\epsilon, \delta)$-DP, although some results along these lines are motivated in the appendix. Additionally, the obvious weakness (which the authors acknowledge) is that their method does not technically provide a lower bound for $\epsilon$-DP.

### Minor Points

- Intuitively, when $\mathcal{G} = \mathcal{D}$ or in the Real-Member;Real-Nonmember (RM;RN) case where the non-members follow the same distribution, I think the no-retraining-needed argument could work. This would involve modifying Algorithm 1 and reworking Proposition 2 by (A) assuming entries in $D_f$ and $D_G$ are i.i.d. from the same distribution and (B) setting $S$ according to the original train-test split instead. On the other hand, when $\mathcal{G} \neq \mathcal{D}$, I'm not sure if such an argument can be made to work, at least not trivially.

- It's not clear how the mechanism $B(S, X) = \\{b(x_1), \cdots, b(x_m)\\}$ in Proposition 1 incorporates the helper model mentioned in Section 5.1 used in the experiments.

- Algorithm 1 (lines 4-7) does not seem to reflect the $p$-value adjustment discussed in lines 198-200. Perhaps the authors might have overlooked this $p$-value adjustment in the experiments as well?
questions: questions: - How is the helper model used in the baseline trained?

- The lexicographic order in equation (2) appears to be essential for Corollary 2 and seems to be aimed at ensuring something like $\epsilon > c$ to hold. Could the authors provide more details on this?

- How exactly does the formulation in the paper differ from the setting introduced by Steinke et al., 2023, specifically in regards to the independence assumptions and the probabilistic dependencies between concerned random variables?
limitations: limitations: The authors have acknowledged some limitations in the paper. There do not appear to be any negative societal impacts associated with this work. I encourage the authors to address the issues and questions raised in this review.

Comments: title: Comment to rebuttal
comment: Many thanks for the authors for their elaborate rebuttal and addressing the concerns I raised. 

While the value of a proxy for the lower bound arguably remains limited, I believe the method and evaluation in this paper provide valuable insights into developing MIAs in the absence of non-members with an appropriate model-less baseline. I believe this pushes the field forward, especially in light of recent concerns raised by [1,2,3]. I hence change my score to 5. Best of luck. 

[1] Das et al. "Blind Baselines Beat Membership Inference Attacks for Foundation Models." 2024. https://arxiv.org/abs/2406.16201 [2] Carlini et al. "Membership inference attacks from first principles." S&P 2022.
[2] Duan, M., Suri, A., Mireshghallah, N., Min, S., Shi, W., Zettlemoyer, L., ... & Hajishirzi, H. (2024). Do membership inference attacks work on large language models?. arXiv preprint arXiv:2402.07841.
[3] Meeus, M., Jain, S., Rei, M., & de Montjoye, Y. A. (2024). Inherent Challenges of Post-Hoc Membership Inference for Large Language Models. arXiv preprint arXiv:2406.17975.
We performed experiments on higher-performing models. Our early results (see table) are qualitatively similar to those in the paper: we can detect privacy leakage, with numbers consistent with those of the O(1) approach, and we detect more leakage from overfit models (e.g. WideResNet-28-10 trained for 300 epochs). We will update our paper with a full evaluation (multiple seeds, similar to Fig. 4) and growing test set sizes (similar to Fig. 6). 

Thank you for your valuable suggestions, score increase, and involvement in the discussion!





| Target model $f$                     | Test Accuracy | Audit                   | $\mathbf{c_{lb}}$      | $\mathbf{\{(\epsilon + c)\}_{lb}}$ | $\mathbf{\tilde\epsilon}$ |       $\mathbf{\epsilon}$ |
|------------------------------------------------------|---------------|-------------------------|---------------|-------------------------|------------------|------------|
| WideResNet-28-10_E150                                 | 95.67%        | PANORAMIA,              | 2.51          | 5.05                    | 2.54             |            |
| (generalized model)                                   |               | O(1)                    | -             | 2.95                    |                  | 2.95       |
| WideResNet-28-10_E300                                 | 94.23%        | PANORAMIA,              | 2.51          | 6.10                    | 3.59             |            |
| (overfit model)                                       |               | O(1)                    | -             | 3.76                    |                  | 3.76       |
| ViT-small-timm-transfer_E35                           | 96.38%        | PANORAMIA,              | 2.51          | 4.99                    | 2.48             |            |
| (pre-trained on imagenet)                             |               | O(1)                    | -             | 2.83                    |                  | 2.83       |
We believe we clarified how our approach deals with overfit genertors $(c=0)$, and why the theory we developed goes beyond comparing AUC or TPR at low FPR between a MIA and baseline. We will add these clarifications to our paper and mention the generative AI use-case mentioned in the review (thank you!) as additional motivation. 
Please let us know if there are any more concerns on these or other topics!
We believe we addressed the technical concerns raised in the review, and clarified the role of our lexicographic ordering, how our approach differs from that of Steinke et al., 2023, and cleared the concern about Proposition 2. 
Please let us know if there are any more concerns on these or other topics!
Thank you for your detailed responses. I have increased my score.

> ..as we know there typically is no publicly known held-out set of in-distribution non-members (this is from an external point of view, it is likely that there is one internally). 

For a privacy audit (which is the focus of this paper), the held-out data need not be public. And if the audit is indeed external, getting access to a large fraction of member data (to train the generator) is also an equally strong (if not stronger) assumption.

> For this reason, we did not focus on using the most recent, state-of-the-art models

The idea here is not to necessarily pick the "absolute best" model, but have target models comparable in performance to the ones used in the literature. Using a weaker model (and likely higher overfitting, which is indeed the case) would naturally demonstrate higher membership leakage which might make readers believe that the proposed audit is a much better than existing MIAs.
> how the score generated via the proposed technique is any better than directly measuring test accuracy as a signal to help understand relative leakage

There are two main facets to this answer. First, DP is a well-accepted definition to measure privacy leakage and a good metric for our approach. Second, we need to compare two models (the MIA and baseline), and using our technique makes them comparable. They would not be comparable under other metrics. In more detail (slightly adapted from our answer to reviewer 55Rd):

As it is the consensus in the field, we believe that DP provides the best semantics to define and quantify this type of privacy leakage. 
Thus, usually privacy audits aim to provide a lower bound for the privacy loss. While we do not yet provide a lower bound, using DP semantics is still useful. For instance, [2] makes a convincing case that accuracy or AUC are not good metrics for privacy leakage.

TPR at low FPR is a better metric than accuracy or AUC, but notice that we need to compare the MIA with the baseline, and their performance most revealing of privacy leakage happen at different FPR values (cf. Figures 3a and 4a, though in precision/recall terms). In this case, we cannot directly subtract TPR values. Our theory tells us: (1) which FPR we should choose and (2) how to scale the TPR at this FPR (by mapping it to a $c$ or $c + \epsilon$ value) to make the values comparable between the baseline and the MIA.

Taking a step back, we hope that our framework will be a stepping stone for further research, maybe enabling a proper lower-bound on $\epsilon$ by measuring (or enforcing by construction) an upper-bound on $c$ (then $ \(c+ \epsilon \) _ {lb} - c _{ub}$ yields $\epsilon _{lb}$). Though we do not know how to do it yet, this is a promising future work.

> and other misc. comments

>> Please consider a more descriptive abstract. "scheme" here is not very descriptive

Thank you for the feedback, we will rework the abstract to avoid this generic term.

>> L121: "We formalize a privacy game" - this is standard membership inference evaluation and not a contribution of this work. Please rephrase to make this clear.

While formalizing a privacy game is standard, the specific one we introduce (definition 2) is a variation of the game from Steinke et al., 2023 (please see our answers to Reviewer Z4qp, Q3 and W1 for details on the difference) and, to the best of our knowledge, unique to our work.

>> L493-494: Please provide direct distinguishing accuracies of these baseline classifiers for reference and easier understanding.

We do not believe that accuracy is a meaningful metric for privacy leakage measurement. See [2] for a detailed case. Our focus on member detection, as opposed to both member and non-member (see Remark 3 l. 137), which is due to practical requirements on the generator, exacerbates these issues as accuracy measures both the detection of members and non-members. As we argue above, we do believe that the metrics we report based on DP are the best suited to measuring privacy leakage.

We also show the whole precision/recall curve for some of the models on plots like those if Fig. 3. We could add numbers for the (precision, recall) tuples that imply the bounds in Table 3 (drawback: each bound is at a different recall), or for the precision at a fixed low recall (drawback: this will not be the ``best'' recall for all models, so comparisons are a bit misleading). Because of the drawbacks, we decided not to show this information in our submission but can add them if it is valuable.

[2] Carlini et al. "Membership inference attacks from first principles." S\&P 2022.
Thank you for your follow-up! 

The are two things we would like to emphasize about the validation set, in case they were lost in the rest of the answer. First, we do see that generated data can provide higher privacy leakage measurements *above the ones we can measure using 20% non-member held-out data* (Figures 7a and 7c). Second, we believe that foundation models are a counter-example to the claim that there always is such a large held-out set: as far as we know there typically is no publicly known held-out set of in-distribution non-members (this is from an external point of view, it is likely that there is one internally). However, there are many known member data points, and externally measuring privacy leakage for such models is a topic of interest (see [1] in our answer).

We are happy to discuss the other comments as well!

> Claims around contributions

In our contributions paragraph, we list three key properties of the setting we tackle. We do not mean to claim that we are the first to study each separately! We will rephrase this paragraph to clarify that the novelty of our approach lies in the combination of those properties.
The two papers cited in the review focus on membership attacks, which is just one part of our approach. It would be interesting to see if we can use those attacks in Panoramia to yield better measurements, but this paper focuses on developing the privacy measurement framework that we propose. Regarding the specific membership inference papers:
* The paper cited in the review as ``[1] Suri, Anshuman, Xiao Zhang, and David Evans. "Do Parameters Reveal More than Loss for Membership Inference?."'' was first posted on arxiv in June'24, after our submission. It is indeed a very interesting candidate to use in Panoramia's privacy measurement in the future.
* The paper cited as ``[2] Nasr, Milad, Reza Shokri, and Amir Houmansadr. "Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning."'' has several membership attacks, and some might be compatible with our approach and could be interesting to try.

Note that in both cases, we would still need non-members to measure and assess the performance of the membership attacks (and hence the generative model in our setting), and we would still need a theoretical framework to meaningfully compare the results of the results of the MIA and baseline to quantify privacy leakage (our theory). We will make sure to mention the papers and which parts of our framework they could improve.

> underperforming models

In this paper, we focused on developing our privacy measurement framework, and studying its behavior when varying levels of overfitting (increasing the number of training epochs), changing the number of parameters, or making models DP. For this reason, we did not focus on using the most recent, state-of-the-art models, though it would be an interesting measurement study in the future. On the specific models we use for image data, we use a ResNet101, which reaches 93.75% test accuracy on CIFAR10 with ImageNet pre-training (that is why our version is a bit weaker at 91.61%, as we do not do this pre-training). We note that other papers listed in the review evaluate similar performing models (a WideResnet with 92.4% in [2], 92% in [3], and 90% accuracy in [4]). We will add results for a stronger model. We will report results if they finish by the end of the discussion period, though we are not sure it will be ready due to the computation power we have access to.
> ...such a large portion of training data may not be kept for testing in general.

No model (at least in practice) is ever trained without **any** validation/test data. While it may not be as high as 20%, getting an estimate of privacy leakage of the underlying training mechanism/model should not require more than a small fraction of the data. I think a convincing argument can be made to use generative techniques, and the authors are close to one but not there yet.

Most of my other concerns remain unaddressed, specifically: claims around contributions, underperforming models, and how the score generated via the proposed technique is any better than directly measuring test accuracy as a signal to help understand relative leakage (and other misc. comments). I am hoping the authors will respond to them
I thank the authors for their response, which answered all my questions thoroughly and resolved Weakness 3.

However, after reading all other reviews and rebuttals, my concerns about the privacy semantics remain. In particular, a robust evaluation procedure should measure the privacy of the worst-case samples (in the dataset), not just an average over the dataset. For example, the high-level arguments in https://differentialprivacy.org/average-case-dp/ still apply for a fixed dataset. Often, samples from the tail of the data distribution are worst-case samples (in a fixed dataset). At the same time, this paper's method seems to fail if there are such samples, because the generator fails to synthesize similar non-members (I greatly appreciate the author's transparency in this regard). While the (1) procedure of Steinke et al., 2023 can suffer from similar issues, there one can simply pick an audit set that solely consists of worst-case samples. I don't see a similarly easy fix for PANORAMIA.

Nevertheless, I do not think this is a deal-breaker, and believe this paper's idea and method carry merit in themselves (e.g., towards a solution of the problems highlighted by [Das et al., 2024](https://arxiv.org/abs/2406.16201)).