Exploring the limits of strong membership inference attacks on large language models
Authors: Jamie Hayes
Ilia Shumailov
Christopher A. Choquette-Choo
Matthew Jagielski
Georgios Kaissis
Milad Nasr
Meenatchi Sundaram Muthu Selva Annamalai
Niloofar Mireshghallah
Igor Shilov
Matthieu Meeus
Yves-Alexandre de Montjoye
Katherine Lee
Franziska Boenisch
Adam Dziedzic
A. Feder Cooper
Venue: NeurIPS 2025 poster ()
Keywords: membership inference
memorization
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We perform large-scale strong MIAs on pre-trained LLMs to clarify the extent of actual privacy risk MIAs pose in this setting.

Abstract: State-of-the-art membership inference attacks (MIAs) typically require training many reference models, making it difficult to scale these attacks to large pre-trained language models (LLMs). As a result, prior research has either relied on weaker attacks that avoid training references (e.g., fine-tuning attacks), or on stronger attacks applied to small models and datasets. However, weaker attacks have been shown to be brittle and insights from strong attacks in simplified settings do not translate to today's LLMs. These challenges prompt an important question: are the limitations observed in prior work due to attack design choices, or are MIAs fundamentally ineffective on LLMs? We address this question by scaling LiRA--one of the strongest MIAs--to GPT-2 architectures ranging from 10M to 1B parameters, training references on over 20B tokens from the C4 dataset. Our results advance the understanding of MIAs on LLMs in four key ways. While (1) strong MIAs can succeed on pre-trained LLMs, (2) their effectiveness, remains limited (e.g., AUC<0.7) in practical settings. (3) Even when strong MIAs achieve better-than-random AUC, aggregate metrics can conceal substantial per-sample MIA decision instability: due to training randomness, many decisions are so unstable that they are statistically indistinguishable from a coin flip. Finally, (4) the relationship between MIA success and related LLM privacy metrics is not as straightforward as prior work has suggested.

Exploring the limits of strong membership inference attacks on large language models
Jamie Hayes!*
Ilia Shumailov'
Christopher A. Choquette-Choo'
Matthew Jagielski*
Georgios Kaissis! Milad Nasr
Meenatchi Sundaram Muthu Selva Annamalai
Niloofar Mireshghallah Igor Shiloy!
Matthieu Meeus Yves-Alexandre de Montjoye' Katherine Lee
Franziska Boenisch
Adam Dziedzie A. Feder Cooper
Google DeepMind University College London
University of Washington
4 Imperril College London $CISPA Helmholtz Center for Information Security Stanford University
Miicosooft Research
Abstract
State-of ofute-of-the-art membership inference attacks (MIAs) typically require training many reference models, making it difficult to scale these attacks to large pre-trained language models (LLMs). As a result, prior research has either relied on weaker at- tacks that avoid training references (e.g., fine-tuning attacks), or on stronger attacks applied to small models and datasets. However, weaker attacks have been shown to brittle and insights from strong attacks simplified settings do not translate to today's 's LLMs. These challenges prompt an important question: are the limitations observed prior work due to attack design choices, or are MIAS fundamentally in- effective on LLMs? We address this question by scaling LiRA one of the strongest MIAs to GPT-2 architectures ranging from 10M to 1B parameters, training refer- ences over 20B tokens from the C4 dataset. Our results advance the understand- ing of MIAs on LLMs in four key ways. While (1) strong MIAs can succeed on pre- trained LLMs, (2) their effectiveness, remains limited (e.g., AUC-0.7) in practical settings. (3) Even when strong MIAS achieve better-than-randon AUC, aggregate metrics conceal substantial per-sample MIA decision instability: due to training randomness, many decisions are so unstable that they are statistically indistinguish- able from coin flip. Finally, (4) the relationship between MIA success and related LLM privacy metrics is not as straightforward as prior work has suggested.
Introduction
In a membership inference attack (MIA), an adversary aims to determine whether specific data record was part of a model's training set [53, 63]. MIAS pose a significant privacy risk to ML models, but state-of-the-art attacks are often too computationally expensive run at the scale of pre-trained large language models (LLMs). This is because strong MIAs require training multiple reference models to form membership predictions and pre-training even one LLM is often prohibitively expensive in research settings. As result, current work makes one of two compromises: running weaker attacks that avoid training reference models (e.g., attacks that fine-tune running strong attacks that train small reference models on small datasets. However, both exhibit notable limitations (Section 2). Weaker attacks more practical, but they have been shown to be brittle-ofte performing no better than random guessing [18, 21, 43]. Stronger attacks, when run in simplified settings, fail capture the complex dynamics of large-scale, pre-trained language models; as result, their insights do not reliably generalize to modern LLMs [38].
LLM), or
Results from both of these approaches leave key questions unanswered about the effectiveness of MIAS on LLMs. In particular, are the fidelity issues of weaker attacks due to omitting reference
" Equual contribution; corresponding authors: jamhaydgoogle. com, afederccopper@gail.cco
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

models, or do they point to a deeper, more fundamental challenge with applying membership inference to large language models? Current research has not offered an answer because, to date, there are no baselines of how the strongest MIAs perform on large-scale, pre-trained L.MSs.
In this paper, we bridge this gap by running strong attacks previously explored. We pre-train over 4,000 GPT-2-like reference models, ranging from 10 million to 1 billion parameters [31], on subsets of the C4 dataset [47] that are three orders of magnitue larger than those used in prior MIA studies-over 50 million samples, compared to fewer than 100,000 in previous work [40]. We use these models conduct a detailed investigation of the Likelihood Ratio Attack (LiRA) [5], one of the strongest MIAs in the literature. This substantial effort proves worthwhile, as we uncover four key insights that advance the state of the art in understanding the potency and reliability of membership inference attacks on large language models:
scale significantly larger than
Strong membership inference attacks can succeed pre-trained LLMs. We are the first to execute strong attacks at this scale, and find that LiRA-in contrast to weaker fine-tuning attacks-can easily beat random ROC-AUC baselines (Section 3.1). Our results on Chinchilla- optimal models (trained for 1 epoch) exhibit a non-monotonic relationship between model size and MIA vulnerability: larger models are not necessarily more at risk (Section 3.2).
The overall success of strong MIAs is limited on pre-trained LLMs. Even though we demonstrate that LiRA can succeed at LLM scale, we are only able to achieve impressive results (i.e., AUC20.7) when diverging from typical training conditions- aadditons--seciffcalllyy specifically, by training for multiple epochs (Section 4.1) and varying training dataset sizes (Section 4.2).
Many correct per-sample MIA decisions for LLMs do not reflect reliable inference. Even when an MIA achieves petter-than-aee raadom AUC, the underlying per-sample binary decisions are highly sensitive to training randomness. We quantify this by measuring per-sample decision instability (Section 5.1) and find that, even at modest FPR, many per- per-sample decisions are statistically indistinguishable from coin flip (Section 5.2).
The relationship between MIA success and related LLM privacy metrics is not straightfor- ward. We show that samples seen later in training tend to be more at risk (Section 6.1); however, this trend is complicated by sample length. We also study if there is any relationship between training data extraction and MIA, and observe no correlation with MIA success. This suggests that the two privacy attacks may capture different signals related to memorization (Section 6.2).
Our contributions serve as an extensive benchmark of strong MIAs, and also provide some initial answers to urgent open questions about the conditions under which MIAs exhibit a threat to privacy for LLMs. Our work also quantifies the performance gap between weaker (more feasible) and stronger attacks, establishing an upper bound for what weaker attacks could achieve in this setting.
2 Background and related work
Membership inference attacks (MIAs) assess empirical privacy and information-leakage risk by asking whether an adversary can tell if particular data point æ was used to train target model h. Given knowledge of the target's 's architecture and training setup, the attacker trains multiple reference models f on different subsets drawn from the same underlying distribution as the target's training data. For each æ, references are partitioned into those trained with x (PIN, where x is member) and those trained without x (POUT, where a non-member). For a given x and model g (the target or reference ʃ E 4), the attacker queries the model and computes an observation statistic s(y. x) from the model's output on æ (e.g., loss, logit). MIAs transform these statistics into a membership score (zz) that is used to infer whether D was in the target's 's training data [5, 49, 58, 62, 64).
Different attacks specify different ways of turning observation statistics into membership scores. For instance, for each query sample æ. the Likelihood Ratio Attack (LiRA) collects two sets of reference statistics, {s(f, æ² f E { and {s(f. æ) : f E oour(w)) }. These sets are treated as samples from two empirical distributions, to which density models (PIN and Pour) are fit. LiRA evaluates the target statistic s(h, æ²) under the fitted densities to compute likelihood ratio membership score ((x) for x [5]. Given score A(æ), the attacker outputs a binary membership decision via threshold rule b(x) = 1(A(x) 2 r). In practice, 7 is typically calibrated on non-members to satisfy a fixed false positive rate (FPR) n, Although membership inference is defined as decision problem for a single sample æ, attack performance evaluated as an average over many samples (e.g..
reporting TPR at fixed FPR). Success is typically reported with threshold threshold-gnoostic metries like
2

---

ROC-AUC [53, 63] (Appendix A). To address this gap, we also run experiments that offer novel insights into sample-specific attack performance (Sections 5 & 6, Appendices E & F).
The number of reference models necessary for successful attacks varies across methods from tens or hundreds for LiRA and Attack-R [62], to few 1 or 2 for RMIA [64]. While these attacks have been successfully applied to smaller settings, they are often considered impractical for contemporary language models due to the prohibitive computational cost of training even a single reference LLM. As attacks in various ways.
a result, prior work attempts to approximate stronger, reference-
Small-scale, strong, reference- attacks. The first work to evaluate MIAs in smaller language models (RNNs) trained 10 references [54]. However, insights from such settings do not translate to today's s LLMs [40], as the training dynamics differ significantly. Other work has used a single reference model to attack a small, pre-trained masked language models [42], but this approach reduces precision, as effective membership inference is difficult with fewer references.
Larger-scale, weak, reference-free attacks. To avoid the cost of training reference models, weaker attacks consider range of observation statistics to infer membership, typically leveraging black-box access the model. Yeom et al. [63] use model loss, Carlini et al. [4] use normalized model loss and zlib entropy, and Mattern et al. [37] compare the model loss to the loss achieved for neighboring samples. More recent work experiments with token probabilities [52, 66] and changes in loss based on prompting with different context [57, 61].
Other work attempts derive membership signal from changing the model. For instance, prior work perturbs inputs or model parameters and observes resulting changes in target loss on the sample, or uses (parameter-efficienl fine-tuning on domain-speciin datasets detect privacy risks [8. 21, 28, 33, 41, 43, 46, 48]. However, fine-tuning attacks introduce new data to the problem setup, which may complicate the validity of using MIAs to detect benchmark contamination [17, 34, 35, 45] and to draw reliable conclusions about other sensitive data issues [10, 12, 13, 15, 19, 30, 39, 52, 60, 65]. A recent approach evaluates attacks on LLMs using post-hoc collected datasets. While prior work has reported high success rates on a variety of models and datasets (AUC=0.8) [38, 52, 57, 61, 66], such evaluations rely on the model's training-date cutoff as a proxy for distinguishing between member and non-member data points [35]. These newer data introduce distribution shift, which can undermine the validity of the reported results [16, 18, 35, 40J. Further, when current MIAs are evaluated in a controlled privacy game like this, they often barely outperform random guessing [18, 40].
3 Examining strong MIAs in realistic settings for pre-trained LLMs
Altogether, the limitations of prior work raise the key question that motivates our work: are the fidelity issues of weaker attacks due to omitting reference models, or do they point to a deeper. more fundamental challenge with applying membership inference to large language models? This is big question, so we break down into smaller ones that we can test with specific experiments that reveal different information about the effectiveness of strong MIAs on pre-trained LLMs. To start, determine which strong MIA method to use across our experiments. We evaluate two of the strongest attacks in the literature- LiRA [4] and RMIA [64] in variety of settings. For the experiments that follow, we use LiRA because we observed that it can achieve substantially higher ROC-AUC when attacking pre-trained LLMs. We compare LiRA and RMIA in Appendix B.
In this section, we investigate the relationship between the number of reference models and attack success (Section 3.1). Based on these results, we decide to use 128 reference models in all following experiments. Then, we test the effectiveness of strong attacks under realistic settings settings-setting that reflect how LLMs are actually trained. To do so, run LiRA on models of various sizes, which we train according to Chinchilla-scaling laws [26] (Section 3.2). Together, these experiments inform our first key result: with respect to overall ROC-AUC, strong membership inference attacks can succeed on pre-trained LLMs. In the following sections, we expand upon these results to other training and attack conditions; we will refine our first key result by investigating the limits of strong MIA success rates (Section 4), and by digging beneath aggregate metrics like AUG to better understand attack performance with respect to individual samples (Sections 5 & 6).
General setup. For all experiments, we pre-train GPT-2 architectures of varying sizes- from 10M to IB-on subsets of the C4 dataset [47] using the open-source NanoDO library [31]. The training datasets we use are orders of magnitude larger than those in prior MIA studies: over 50M samples, compared to fewer than 100K samples in previous work [40]. We explore datasets of this size
3

---

because, while it is well established that MIA success depends on both model capacity and training dataset size [53, 62, 63], the nature of this relationship remains unexplored mre-trained-LLA scale. For each attack, we start with fixed dataset of size 2N (e.g., 20M) drawn from C4, from which we randomly subsample (with different random seeds) reference training sets of size N (e.g., 10M), So, for each reference f. half of the drawn samples are members and half are non-members. This yields the different member (IN) and non-member (OUT) distributions for each sample that we use to run LiRA. In our largest experimental setting, we use 2.N Specific experimental configurations vary, so we introduce additional setup as needed. (See Section for details.)
3.1 Warm-up: How many reference models should we use?
To determine the number of reference models to use for all of our experiments. we train 140M-parameter models on 77M samples, which equates to approximately 2.8B training tokens.
10a
True Positive Rate
10-1
10-2
(This is optimal for this model size, according to Chinchilla scaling laws with an over-training multiplier of 20 [26].) As shown in Figure I, we test a range of reference models. (We plot the number of IN references; the total number of IN/OUT references is 2x this number.) The plot shows multiple receiver operating characteristic (ROC) curves, indicating the observed true positive rate (TPR) for the fixed false positive rate (FPR) on a log-log scale. Area under the curve (AUC) is provided for each ROC. The dashed red line represents the baseline for which the attack is equivalent to random guessing (i.e., cannot distinguish between true and false positives so TPR=FPR; AUC=0.5). We report AUC as our primary metric, it is otherwise challenging visualize TPR over wide range of fixed FPR. (For comparison, see Figure 2b, which shows a limited range of FPR, but does not surface threshold- agnostic AUC.) We also investigate the performance of different observation statistics (Section B. and choose to use model loss. Altogether, while LiRA clearly beats the random baseline, it is not remarkably successful in this setting: regardless of the number of references, never achieves an AUC of 0.7. Even though success increases with more references, there are diminishing returns. From 1 to 8 IN references to 16 references total), AUC has relative increase of 13.3%; for the next 8 x increase (from 8 to 64), AUC only increases 7.6%; and, doubling from 128 to 256 only yields a 0.2% improvement. We opt to use 128 total references (64 IN, 64 OUT) in most experiments below.
10-3
Number of reference models, AUC
10-
1,0,55
52,0.670
2.0.50
64. 0.678
10-5
10
0.855
10
10
10
10
10"
10
10"
False Positive Rate
Figure 1: LiRA with different references. We attack a 140M model trained on 77M samples. As references increase, LiRA's performance im- proves (measured with ROC-AUC). However, there are diminishing returns: AUC is effectively unchanged from 128 to 256 IN references.
3.2 Training and attacking a compute-optimal model
In practice, models are typically trained based on observed scaling laws: for a given model size, the scaling law suggests the optimal number of tokens to use for training. To assess strong MIA in realistic conditions for pre-trained LLMs, we attack models of various sizes, setting the number of training samples to be optimal according to Chinchilla scaling [26]. Specifically, we set the number of training tokens to be 20x larger than the number of model parameters and we only train for I epoch- a common choice in large training runs [1, 56). Specific training recipes and experimental details are in Appendices C and G, including the number of samples used to train each model size.
In Figure 2, we show two views of the results of attacking 10M- 85M-, 302M- 489M-, 604M- and 1018M-000P- 1018M-parameter parameter models. These model sizes come from the default configurations available in NanoDO [31]. For readability, we exclude the results for the 140M model, as investigate this architecture above. In Figure 1, the attack on the 140M model with 128 IN/OUT references has AUC=0.678, which puts its performance below the 85M and 302M models. Interestingly, we observe a non-monotonic relationship between model size and MIA vulnerability under these training conditions. In Figure 2a, the 85M model shows the highest AUC=0.699 followed by the 302M (AUC=0.689). The 489M model exhibits the lowest success (AUC=0.547).
Figure 2b provides a different view of the same results. By model size, each line compares the TPR for fixed settings of FPR. Our expectation was that each line would look approximately horizontal, as the training set size is being scaled proportionally (and optimally, according to Hoffmaan et al. [26])

---

10a
5.0e
True Positive Rate
10-1
10-1
10
10
5.0e 03
1.0e:02
10
TPR
10-3
1,0e-03
10-1
5.0e-04
10-4
10-4
Model size. AUC
1:0e-04
10M, 0.592
489M 0.547
10:1
5.0e-05
6369
604M, 0.654
10-1
302M, 0.689
1010M, 0.553
10M
85M
300M
489M
604M
1018M
1.0-05
10
10
10
10.5
10-4
10-1
10 10 10
False Positive Rate
Model size
(a) ROC-AUC
(b) TPR fixed FPR
Figure 2: MIA vulnerability for compute-optimally trained models. We show attacks 6 models of different sizes under Chinchilla-optimal conditions for 1 epoch, using 128 references. (a) ROC curves demonstrate varying MIA susceptibility for 10M (AUC=0.592), 85M (AUC=0.699)) 6969), 302M (AUC=0.68)) 489M (AUC=0.547), 604M (AUC=0.654) and 1018M (AUC=0.553) The 85M and 302M models show the highest vulnerability, indicating that increasing model size does not uniformly decrease MIA risk in this setting. (b) How TPR for each fixed FPR varies by model size.
to model size. From 10M to 302M, there a consistent pattern of the TPR increasing with model size; but then, at 489M, there is a significant drop in TPR. There are many reasons why this may have occurred. First, the most pronounced differences in TPR are at extremely small values. Even subtle differences in training runs may flip samples from correct incorrect member predictions (Section 5), which, in the low TPR regime, can have a large effect on overall MIA success. Second, Chinchilla scaling [26] is not the only such law. Sardana et al. [50], Hu et al. [27], and Grattafiori et al. [23] all in- troduce other ways to optimally select the number of training tokens for a given model. In future work, we will investigate if these other token-size seae--iiz-ssseetiii selection methods stabilize TPR model size grows.
As we discuss next (Section 4.2), repeating this experiment by training these architectures on the same fixed dataset size exhibits vastly different results. We additionally test other training configurations. In Appendix D, we alter the learning rate schedule and observe that there is modest effect on attack performance. (See Appendix C, where, as sanity check, we also confirm that larger models converge to lower loss values, reflecting their increased capacity to fit the training data.)
Varying compute budget and training dataset size
Even in the most successful (i.e., highest AUC) case, overall attack performance is not particularly im- pressive when running LiRA with a large number of references on compute-optimal models trained for 1 epoch. Similar to our experiments with LiRA and varied numbers of references (Figure 1), the maxi- mum AUC we observe remains under 0.7 7 for all model sizes (Figure 2). This raises natural follow-on question: if we free ourselves from the constraints of typical training settings, it possible to improve success? Can we identify an upper bound on how strong MIAs could perform on pre-trained L.LMs?
To address this question,
attacks models trained on different-sized (not always Chinchilla- optimal) datasets (Section 4.2) for more than 1 epoch (Section 4. Our experiments show that diverging from typical settings can indeed improve attack success. However, while these experiments are a useful sanity check, they do not suggest conclusions about the effectiveness of strong MIAs in general. Instead, there appears to be an upper bound on how well strong MIAs can perform on LLMs under practical conditions. In other words, these experiments inform our second main observation: the success of strong MIAS is limited in typical LLM training settings.
4.1 Effects of scaling the compute budget (i.e., training for more epochs)
In Figure 3a, we compare MIA AUC for the 44M model under different training configurations. We keep the total number of tokens surfaced the model during training Chinchilla-opimaal, but we alter when these tokens are surfaced. As baseline, we train for epoch on the entire dataset, and achieve AUC=0.620 with LiRA. (See Figure 3a, of 1.) We then take half of the training dataset and train the same architecture for 2 epochs. In both settings the total number of training tokens is Chinchilla- optimal, however, the latter, the model has processed each training sample twice rather than once. For the 2-epoch model, we observe a significant increase in MIA vulnerability: 0UC-0.744, which higher than when this model has only completed epoch of training (AUC=0.628, of 2) and than when the model is trained for 1 epoch on the entire dataset (AUC=0.620, 1 of 1). Increasing
5

---

10"
10"
10"1
True Positive Rate
10-2
20-2
True Positive
10
Epoch, AUC
10-1
Epoch of Max Epochs, AUC
10-4
2. 0.991
7,0.710
6,0.774
10-3
of 0.620
10
3. 0.605
8.0.75
0.744
10-4
4. 0.621
0.7.87
10.1
of 0.620
5. 0.645
10.
0.797
10:
10.
10"1
10
10
10-1
10"
10
10
10
10-4
10- 10-7
10-1
10"
False Positive
Rate
False Positive Rate
(a) 44M model, split dataset in half and train for 2 epochs, or train on the entire dataset for epoch
(b) 140M model, training for epochs
Figure 3: Studying the effect of varying epochs. (a) We compare attacking a 44M model trained on the whole Chinchilla-optine dataset in 1 epoch (AUC=0.66 0.620 after of 1 epoch) to training for 2 epochs on only half of the dataset (AUC=0.744 after of epochs). (b) We attack 140M model trained on the whole Chinchilla-optimal dataset for 10 epochs. AUC increases with more epochs.
training epochs-even on a smaller dataset to maintain Chinchilla optimalit- vulnerability to MIA, compared to training for fewer epochs on a larger dataset. However, there is no significant uplift in TPR at small fixed FPR between epochs 1 and 2 for the 2-epoch model. The MIA at the second epoch is less successful than the one after epoch for small FPR. As above, this is perhaps due to subtle differences in runs having impact these small values (Sections 3.2 & 5).
To investigate this further, in Figure 3b, we show how AUC changes over the course of training the 140M model for 10 epochs. As expected, AUC increases with more epochs, starting from 0.573 and reaching 7.797 at the end the tenth epoch.² 2 As in Figure 3a, there is an FPR inflection point where TPR for later epochs is smaller than earlier epochs. In Appendix D, we also train the 140M model on fewer than the R 7M Chinchilla-optimal samples, and (similar to Figure 3a) we observe a more dra- matic increase in MIA vulnerability. Attacking a 140M model trained on 2 102500KK samples exhibits both greater absolute MIA success and a faster relative increase in success in the first few epochs.
4.2 Effects of scaling the training dataset size
We next run two sets of experiments to study the role of training dataset size on MIAs AlAs-beyond training on the Chinchilla- optimal number of tokens. We train 140M models on datasets ranging from 50K to 10M samples (again for 1 epoch) and attack these models with LiRA. In Figure 4a, we show ROC curves for the different models. As we train models on smaller datasets, for given FPR, TPR does not always increase. This suggests that TPR at fixed FPR is not necessarily positively correlated with decreasing the training set size. Rather, AUC is highest for moderately sized datasets (around 1M samples, AUC=0. 753), and decreases for both very small and very large datasets (under AUC=0.7 for both). Indeed, the capacity of the model also has an effect on susceptibility strong MIAs.
In Figure 4b, we train different model sizes with a fixed training set size of 2-2 3M samples- significantly more tokens than is Chinchilla-optimal for several models (e.g., 10M, 44M). We plot the mean and standard deviation of TPR at fixed FPR. where we run the attack 16 times using different random seeds, which has the effect of dictating the batch order. For each model size, we train 16 sets of 128 reference models, and we also vary the target model over each attack. We include the associated ROC-AUC for each model size in Appendix D, which are consistent with the MIA prediction variability in Figure 4b. We observe monotonic increase in TPR at different FPRs as model size increases. This is quite different from Figure 2b, where we scale the training set size with model size. As model capacity grows, vulnerability to MIA also grows if we keep the training set size constant. Further, there is significantly more variance in TPR for larger model sizes and at smaller fixed FPR.
Uncovering per-sample MIA membership decision instability
The high degree of variance that observe in the prior section raises a natural question: how stable are the underlying per-sample membership decisions in strong MIAs? Next, we describe the setup we use to measure per-sample MIA decision instability (Section 5.1). In general, this is reasonable
At epoch 1, AUC-0.573 which differs from AUC-0.678 in Figure 2a (also 1 epoch). This is likely because of variance between runs (Section 5) and substantially different learning rates between the setups.

---

10a
1.0e01
10-1
10
10-1-
5.0a-03
True Positive
10"3
Training set size, AUC
10M, 0.645
TPR
10
1.0e-03 03
10
5M.
10.2
5.0e
10-4
500K, 0.730
1M. 0.753
1.0e 04
5.0e 05
10
10
50K, 0.684
100K. 0.687
10t4
19M
65M8
140M
302M
425M
2020M
1.0e-05
10-7
10
10-1
10
10
10 10"
False Positive Rate
Model size
(a) 140M model x various dataset sizes
(b) Various model sizes 2 sample training
Figure 4: Varying sizes of training dataset and model (1 epoch). (a) We attack 140M models trained on different-sizi datasets (50K to 10M samples). MIA success does not monotonically increase with dataset size. (b) We attack different-size models trained fixed dataset size (==8.3M samples), and plot how TPR varies at fixed FPR. MIA success monotonically increases with model size.
thing to do: it is standard to report attack success with aggregate metrics (AUC, TPR at fixed FPR), but the MIA security game is defined with respect to determining if particular sample x was used in training (Section 2 & Appendix A). We show a selection of results for the 302M model (Section 5.2), which reveal our third key takeaway: aggregate metrics may imply that strong MIA on an LLM performs better than random guessing; however, even at modest FPR, due to training randomness large fraction of underlying, individual membership decisions are highly unstable, resembling coin flips. For these samples, strong MIAs are not producing reliable knowledge about membership.
5.1 Computing per-sample flip rate on calibrated membership decision rules
We fix the IN/OUT references and target training set, varying only the random seed used train targets. By isolating variability from equally plausible targets, we can probe the stability of using counterfac- tual references derive membership signal for a given sample æ. Substantial cross-target disagree- ment on x membership would indicate the signal is not providing robust membership evidence for æ.
Formally, let r~ μ denote a target trained on a fixed dataset with randomness induced by the seed controlling batch order. We train one set of references for all attacks on different r~pp. Let (, E R be the r-specific LiRA score for æ. At a fixed FPR v). we calibrate per-seed threshold (n) on
non-members to form the binary decision rule 66*) (A,)) 2 T, [n] (Section 2). Per-seed calibration mirrors the standard threat model, in which an attacker runs the MIA on a single target [5]. The (population) flip rate [14] at (1, x) is the pairwise decision disagreement probability under μc:
(Trp,(a)) = Pr Pr rir
1 [1,00 (a) + b (x))
.......BB In practice, with B 2 2 i.i.d. target replicas v1...., TB ~ p. the canonical unbiased estimator
(ar) (B) EigiogE 1( (m) + b6 (x))
2 Ra(m) (.))
B (B-1)
where [x)=== by
(x) and (1) are the counts of member and non non-memin member decisions for at 17 among the B target replicas. In practice, flip,, E [0, =0.5]; the finite B max- imum exceeds 0.5 and converges to 0.5 as B (Appendix E.2). flip,, na (x)) (x)==0 means the MIA deci-
sion for x is stable across equally plausible targets; flip,,, B (z)5 means the decision is statistically indistinguishable from a coin flip, with roughly half of decisions calling member, and the other half calling æ a non-member. Figure 5b provides an intuition for how this can occur. For a member æ at FPR 70 2 we plot the reference IN/OUT distributions and the median target statistic (across the B-127 targets). The IN/OUT distributions overlap substantially, making membership for this x hard to disambiguate; accordingly, the 127 decisions split down the middle, so flip10 (x))
5.2 Many membership decisions are statistically indistinguishable from coin flip
Flip rate (Equation 1) lets us peer beneath average metrics to assess the reliability of the MIA procedure's 's decisions for individual samples (Appendix H.2.5). For the 302M model, we train 128 IN/OUT references to use for all attacks, and 127 target replicas on the exact same 5500K dataset with different seeds. While the population (z) indicates coin-flip decisions for æ. with
finite replicas B, we need to determine a defensible cutoff above which flip,, signifies this behavior.
7

---

Palk
2030
2000
2004
2031
2010
2010
300
Arrblgueus unitade tiue member
M (ikemees))
Targe medie
newere dana
26.45
1p 3:00
4.%%
02.11
04
027
0.4
3.4 54 53
Tatee Poetthie
Fito Postrie
(a) Flip rate (Equation 1) by membership at varied fixed FPR, B=127
(h) Unstable member sample
Figure 5: Visualizing decision instability. We train B=127 targets for the 302M model on 21 samples, and one of 128 references use for all attacks. LiRA achieves stable mean AUC-0.752 + 0.007, yet many per-sample decisions behave like coin flips. (left) Share of coin-flip-like decisions across FPR (log-scale for small FPR; ,lip,, 20.487, the a=0..55 cutoff, see Appendix E.2.4). Members exhibit more coin-flip-like decisions than non-members. (right) A representative unstable member: IN/OUT distributions overlap: the B target decisions flip because x's s seed-specific scores lie near the respective seed -peciiic thresholds (Appendix E.2.5).
To do so, we set up a two-sided exact binomial test for A:= Pry==plll (0,²) (xx)=1], and use "coin flip" as shorthand for "fail to reject the null Ho : 0-0.5 level a." For B-127 target replicas and a=0.55, this corresponds the cutoff 127 20.487 (Appendix E.2.4).
Aggregate attack success is high and stable. A training set of R550K samples is significantly smaller than what is Chinchilla-optimal for the 302M model (=15.1M), so we expect higher overall MIA success (Section 4.2). Indeed, aggregate attack success is stable, and substantially outperforms random guessing (AU OACC00752200000 At fixed FPR. TPR is also stable (Figure 5a, mean TPR + STD annotations), Nevertheless, models r~ that yield similar accuracy can have very different decision rules, and therefore can disagree substantially on individual decisions [3] (Appendix E.2.3).
At meaningful FPR, flip rate rises with FPR and model size, and is higher for members. Figure 5a shows that, even at modest FPR, large numbers of membership decisions behave like coin flips. Across fixed FPR 7, we plot the proportion of samples with statistically roin-flip-like decisions, i.c., flip,,, 127 00.477777...7... 0.487 (a=0.05); the samples that satisfy this filter resemble the sample in Figure 5b. At 17-0.02 118.49% 4.4 of members have coin-flip decisions (at a=0.05); if we relax the flip threshold to also include highly unstable 122 20.4 decisions, this proportion becomes 399.8% (By contrast, for non-members these proportions are ~0.03% and =0.2%, respectively. This is unsurprising because decision thresholds are calibrated on non-members; see Appendix E.2.5.)
At lower FPR, as FPR increases, the proportion of samples with coin-flip-like decisions increases. Each seed's calibrated threshold (n) moves into score regions where IN/OUT overlap more extensive, expanding the overall set of samples (especially members) whose scores lie near the decision boundary. As a result, small seed-induced score shifts (as well as across-seed variation in T,-(1) itself, see Appendix E.2.5) flip decisions more often. This effect is more pronounced for the 302M model, compared to the 140M model (Appendices E.2.3 & F.2.5).
This is not an attack, but a diagnostic of MIA signal robustness using counterfactuals. We are able to assess per-sample MIA decision instability by training many different targets r r~ jt, each of which is plausible outcome of the same training process. However, this is clearly not the same procedure that an attacker runs: under the standard threat model (Section & Appendix A), an attacker faces a single target. This matters, even though a true positive is an MIA success: a coin-flip-like MIA decision may be correct, but does not show that the MIA procedure is informing a reliable inferential claim about membership for that sample. For such samples, flip rate shows that attack success is fragile: the claim for æ is an artifact of seed- cod-ssecific idiosyncrasies, rather than a reflection of stable signal obtained from running that specific MIA procedure (Appendix E.2.7). An alternative diagnostic could fix the target, train B independent IN/OUT reference sets on the same IN/OUT draws (varying only random seed), and measure disagreement with respect to those sets. However, this is substantially more expensive. Our experiments varying the target similarly probe counterfactual robustness; we would expect to observe qualitatively similar instability under reference resampling (Appendix E.2.8).
8

---

member memberr
Mott Wihembiee
cast warde
0.7
0162
Now Coniaal Lengh
Plpredicted
0.000
0.602
0.
10000
20000 30000 40000 soona 60000
Train Step
Length Dolenai
(a) Per-member success over training steps.
(b) Sample-lengt distributions by vulnerability.
Figure 6: Sample vulnerability to MIA. For the 140M model, (a) the evolution of sample vulnerabil- ity during training, shown by sample true positive probabilities Pr(predicted as a member member) at each step. (b) Distributions over sample lengths, according to MIA vulnerability for the 1,000 samples that are least vulnerable, most vulnerable, and most difficult for MIA (i.e., with smallest, largest, and closest to 0.5 Pr(predicted member member))) See Appendix E.
Overall, flip rate diagnostics show that training randomness plays significant role in per-sample MIA decisions. Even at FPR=10-3 we estimate for the 302M model that roughly 15.4+0.66 of true pos- itives behave like coin flips-i.c. 21PP0-~ at a-0.5.. If we expand to include highly un- stable decisions (flip10-=,127 20.4), these constitute 42.2%+10.9% of true positives (Appendix E.2.6).
Analyzing sample vulnerability to membership inference
The instability in membership predictions that observe for individual samples suggests a natural follow-on question: when does strong MIA succeed? Which samples are actually vulnerable to MIA, and (how) does this vulnerability vary during training? We approach these questions by digging deeper into our strong attacks on 140M models-trained with a Chinchilla-opimaal training set (* samples) for epoch-witt 1iihh 1 128 references. Samples seen later in training tend be more vulnerable; however, this trend is complicated by sample length (Section 6.1). While sample length has previously been linked to extraction risk [7, 44], we observe no correlation between MIA and standard extraction methodology (Section 6.2). Together, this analysis informs our fourth key takeaway: the relationship between MIA vulnerability and related privacy metrics is not straightforward for LLMs.
6.1 Identifying patterns in per-sample MIA vulnerability
We first investigate how sample MIA vulnerability evolves over the course of training. In Fig- ure 6a, the scatter plot illustrates per-sample true positive probabilities by training step: plot how the probability of a training sample being correctly predicted as a member changes as model training progresses, where the membership prediction for is computed using the reference dis- tributions, i.e., piv(- (p((-T)) æ) >0. (Section 2 & Appendix A). Across samples in the batch at each step, there is considerable variance in the underlying sample true positive probabílitics Pr(predicted as member memberrmmember): it can vary by more than 15%, having effect overall attack success. For much of training, the mean Pr(predicted as a member memberr) is close to 0.5, indicating many samples are challenging for MIA to distinguish either members non-members.
pox(-lze)
The density of the points shifts upward toward the end of training (around step 60,000). Samples in batches that are processed in later epochs tend to be more vulnerable, as indicated by the higher probability of being correctly identified as members. This result highlights that the recency of exposure influences sample's vulnerability to membership inference. Put differently, samples introduced earlier in training are more likely to be "forgotten" " [6]: they are less vulnerable to MIA. This is perhaps partial reason for LiRA decision instability for targets trained but with different random seeds that control batch order (Section 5). For some targets, a member æ may be seen late in training and exhibit a high true positive probability; for others, the same x may appear early and be "forgotten." (i.e., result in false negatives).
the same dataset,
While this appears to be the dominant trend, the details are more complicated. In Figure 6b, we plot the distribution over members according to length, and partition this distribution according to vulnerability. We consider members for which LiRA's predictions are confident but incorrect (i.e., predict non- mon-member) to be least vulnerable, and members that LiRA correctly and confidently predicts as members to be most vulnerable. We also highlight members for which LiRA struggles to determine

---

membership status (true positive probabilities ~0.5). Figure 6b suggests that vulnerable sequences tend to be longer. (See also Appendix F, which illustrates similar results for samples that have a higher proportion of <unk> tokens and higher average TF-IDF scores.) This result is consistent with those in Carlini et al. [7], which show that longer sequences tend to more vulnerable to extraction attacks.
6.2 Comparing MIA vulnerability and extraction
Results such as those in Figure 6b are consistent with prior work on memorization and extraction [4]. In general, it is assumed that a successful membership inference attack and successful extraction of training data imply that some degree of memorization has occurred for the attacked ML model. For MIA, this is assumed because the success of such attacks hinges on the model's 's tendency to behave differently for data it has seen during training (members) compared to unseen data mnonmmmberr) (Section 2 & Appendix A). Prior work frequently ascribes this differential behavior to the model having memorized certain aspects of the training data.
We therefore investigate whether samples that are vulnerable to strong MIAs are also vulnerable to standard extraction attacks. In Figure 7, for the 1,000 samples identified most vulnerable to strong MIA in the 140M Chinchilla-optimal model (Figure 1), we use the first 50 tokens of each sample (prefix) to see if the next 50 tokens (target suffix) is extractable. We use a sample's negative log-pno prgobabbbyllty as a proxy for computing a probabilistic variant [25] of discoverable extraction [7]- the standard extraction metric in research and model release reports [2. 22. 24, 44, 55]. Discoverable extraction systematically underestimates extraction, relative to probabilistic extraction [15, 25]. We measure probabilistic extraction because expect it to provide more reliable signal for memorization. A smaller negative log-probability implies that a sample is easier to extract [15].
After 1 epoch, LiRA is able to identify members with better- than-random AUG (Figure 1). Out of the 1,000 samples with the highest LiRA scores, 713 are indeed members. The largest suffix extraction probability is f00.0067ffor the member sample in Figure 7 that has the (smallest) negative log probability of ==5. Most samples- members and non-members alike have negative log probabilities > 100, corresponding to probabilities on the order of 10 44 (measurements that do not register successful extrac- tion [15, 25)). Altogether, while much prior work draws a direct connection between MIA and extraction vulnerability [e.g., 4], our results suggest a more nuanced story: the suc- cess of strong MIA on given member does not necessarily imply that the LLM is more likely to generate that sample than would be expected under the data distribution [15. 25).
Number members: 713/1000
Neg Log Prob
20*
10t
0.984 0.566 0.558 0.950 0.692 0. 0.996 0.598 1.000 Pipredicted as member)
Figure 7: Extraction for 140M. Neg- ative log- probability of the 50-token suffix (given the prior 50 tokens as pre- fix) for the 1,000 samples predicted most strongly as members.
7 Conclusion and future work
We perform dozens of experiments on thousands of GPT-2-like models (ranging from 10M-1B parameters) on enormous training datasets sampled from C4 (up to three orders of magnitude larger than those in prior work). In doing so, we address an urgent open question in ML privacy research: are the fulelity issues of weaker attacks due to omilting reference models, or do they point to deeper, more fundamental challenge with applying membership inference to large language models? We uncover four novel groups of findings. While (1) strong MIAs can succeed on pre-trained L.MSs (Section 3), (2) their success is limited (i.c., AUC-c07) for LLMs trained using practical settings (Section 4). Even when attacks achieve above-chance AUC, (3) many per-sample membership decisions are very unstable; under training randomness, they are statistically indistinguishable from coin flip (Section 5). Further, (4) the relationship between MIA vulnerability and related privacy metrics is not straightforward for LLMs (Section 6).
As the first work to perform large-scale strong MIAS on treitrined LLMs, we are also the first to clarify the extent of actual privacy risk MIAS pose in this setting. By evaluating the effectiveness and limits of strong attacks, we are able to establish an upper bound on the accuracy that weaker, more feasible attacks can achieve. Together, our findings can guide others in more fruitful research directions to develop novel attacks and, hopefully, more effective defenses.
10

---

Acknowledgments and Disclosure of Funding
Thank you to our anonymous reviewers, Nicholas Carlini, Zachary Charles, and Christopher De Sa for feedback on earlier versions of this work. A. Feder Cooper's contributions originated with 2023-2024 student researcher position at Google Research. Franziska Boenisch has received funding from the European Research Council (ERC) under the European Union's Horizon Europe research and innovation programme (grant agreement No. 101220235)
IS is currently employed at a startup; CCC, MN, and KL are currently employed by OpenAl; MJ is currently employed by Anthropic; NM is currently employed by Meta.
References
[1]
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[2] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397-2430. PMLR, 2023.
[3] Leo Breiman. Statistical Modeling: The Two Cultures. Statistical Science, 16(3):199-215:
2001. ISSN 08834237.
[4] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pages 2633-2650, 2021.
[5] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914, 4. IEEE, 2022.
[6] Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization is relative. Advances in Neural Information Processing Systems, 35: 13633-1363,7, 2022.
[7] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. In International Conference on Learning Representations, 2023.
[8] Hongyan Chang, Ali Shahin Shamsabadi, Kleomenis Katevas, Hamed Haddadi, and Reza
Shokri. Context-aware membership inference attacks against pre-trained large language models. arXiv preprint arXiv:2409.137 13745, 2024.
[9] Alexandra Chouldechova, A. Feder Cooper, Solon Barocas, Abhinav Palia, Dan Vann, and Hanna Wallach. Comparison requires valid measurement: Rethinking attack success rate comparisons in Al red teaming, In The Thirty-Ninth Annual Conference on Neural Information Processing Systems Position Paper Track, 2025. URL https: butpss:/openne /twp/revieeeeeeeeee/emmemee www.iiewwww.eeeeeeeeeeeeeeeeeeee orrmm id=d7hqAhLvWG.
[10] A. Feder Cooper and James Grimmelmann. The Files are in the Computer: Copyright, Memo- rization, and Generative AI. arXiv preprint arXiv:2404. 12590, 2024.
[11] A. Feder Cooper, Jonathan Frankle, and Christopher De Sa. Non-Determinism and the Lawless- ness of Machine Learning Code. In Proceedings of the 2022 Symposium on Computer Science and Law, CSLAW '22, page 1-8, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450392341. doi: 10.1145/351 1117351126...0 02111126635555 3550446..
[12] A. Feder Cooper, Katherine Lee, James Grimmelmann, Daphne Ippolito, Christopher Callison- Burch, Christopher A. Choquette-Chen Mimno, Madiha Zahrah Choksi, Jack M. Balkin, Nicholas Carlini, Christopher De Sa, Jonathan Frankle. Deep Ganguli, Bryant Gipson, Andres Guadamuz, Swee Leng Harris, Abigail Z. Jacobs, Elizabeth Joh, Gautam Kamath, Mark Lemley, Cass Matthews, Christine McLeavey, Corynne McSherry, Milad Nasr, Paul Ohm, Adam Roberts, Tom Rubin, Pamela Samuelson, Ludwig Schubert, Kristen Vacearo, Luis Villa, Felix Wu, and Elana Zeide. Report of the 1st 202.. 2023.
Niloofar Mireshghallah, Miles Brundage, David
Workshop on Generative AI and Law. arXiv preprint
11

---

[13] A. Feder Cooper, Christopher A. Ghoquette-Choo. Miranda Bogen, Matthew Jagielski, Katja Filippova, Ken Ziyu Liu, Alexandra Chouldechova, Jamie Hayes, Yangsibo Huang, Niloofar Mireshghallah, Ila Shumailov, Eleni Triantafillou, Peter Kairouz, Nicole Mitchell, Percy Liang, Daniel E. Ho. Yejin Choi, Sanmi Koyejo, Fernando Delgado, James Grimmelmann, Vitaly Shmatikov, Christopher De Sa, Solon Barocas, Amy Cyphert, Mark Lemley, danah boyd, Jennifer Wortman Vaughan, Miles Brundage, David Bau, Seth Neel, Abigail Z. Jacobs, Andreas Terzis, Hanna Wallach, Nicolas Papernot, and Katherine Lee. Machine Unlearning Doesn' It Do What You Think: Lessons for Generative AI Policy, Research, and Practice. arXiv preprint 2024.
[14] A. Feder Cooper, Katherine Lee, Madiha Zahrah Choksi, Solon Barocas, Christopher De Sa, James Grimmelmann, Jon Kleinberg. Siddhartha Sen, and Baobao Zhang. Arbitrariness and Social Prediction: The Confounding Role of Variance in Fair Classification. Proceedings of the AAAI Conference on Artificial Intelligence. 38(20):2200-2222 22012,, March 2024.
[15] A. Feder Cooper, Aaron Gokaslan, Amy B. Cyphert, Christopher De Sa, Mark A. Lemley, Daniel E. Ho, and Percy Liang. Extracting memorized pieces of (copyrighted) books from open-weight language models. arXiv preprint arXiv:2505.. 12546, 2025.
[16] Debeshee Das, Jie Zhang, and Florian Tramèr. Blind baselines beat membership inference attacks for foundation models. arXiv preprint arXiv:2406. 16201, 2024.
[17] Chunyuan Deng. Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pupers), pages (698-8711, 2024.
[18] Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettle- moyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? In First Conference on Language Modeling, 2024.
[19] André V Duarte, Xuandong Zhao, Arlindo L Oliveira, and Lei Li. De-cop: detecting copyrighted content in language models training data. In Proceedings of the 41st International Conference on Machine Learning, pages I 11900-19956, 2024.
[20] Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong but many are usefull Variable importance for black-box, proprictary, misspecified prediction models, using the Rashomon set. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), pages 131-138, 2019. doi: 00.11453336618.333 8.3314221.
[21] Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Membership inference attacks against fine-tuned large language models via self-prompt calibration. In The Thirty-eigtth Annual Conference on Neural Information Processing Systems, 2024.
[22] Gemini Team, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap. Jean- baptiste Alayrac, Radu Soricut, Angeliki Lazaridou. Orhan Firat, Julian Schrittiieeer, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.c 055500 2024.
[23] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama
3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
[24] Aaron Grattafiori et al. The Llama 3 Herd of Models, 2024. URL https:://aaatiiii orgo /bs// 2407. 21783.
[25] Jamie Hayes, Marika Swanberg, Harsh Chaudhari, Itay Yona, Ilia Shumailov, Milad Nasr, Christopher A. Choquette- -Choo, Katherine Lee, and A. Feder Cooper. Measuring memorization in language models via probabilistic extraction. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Pa- pers), pages 9266-9291, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176- 189-6. long. .469/.
URL https: // www.hhtss//allaatllllotooo aalaacholnnooyyo.oo/ org org/2025. naacl-
[26] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom
12

---

Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan. Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training Compute-Optimal Large Language Models, 2022. URI // /ttpvv/aavvvvv/////a////a//iiava org/ abs/2203. 15556.
[27] Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng. Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.0633 2024.
[28] Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher Choquette- Choo. and Zheng Xu. User inference attacks on large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18238-182655 2024.
[29] Katherine Lee, Daphne Ippolito. Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021.
[30] Katherine Lee, A. Feder Cooper, and James Grimmelmann. Talkin' 'Bout AI Generation: Copyright and the Generative-AI Supply Chain. arXiv preprini arXiv:2309.08133 2023.
[31] Peter J. Liu, Roman Novak, Jaehoon Lee, Mitchell Wortsman, Lechao Xiao, Katie Everett, Alexander A. Alemi, Mark Kurzeja, Pierre Marcenac, Izzeddin Gur, Simon Korbblith, Kelvin Xu, Gamaleldin Elsayed, Ian Fischer, Jeffrey Pennington, Ben Adlam, and Jascha-Sohl Dickstein. NanoDO: minimal Transformer decoder-only language model implementation JAX, 2024. URL. htt.:/t///// /tt:://iittttt com/ goog//oogeeeeoo deepmidddedddddddne nanodo. Version 0.1.0.
[32] Iyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1.05000 2017.
[33] Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing leakage of personally identifiable information in language models. In
2023 IEEE Symposium on Security and Privacy (SP), pages 346-363. IEEE, 2023.
[34] Pratyush Maini and Hritik Bansal. Peeking behind closed doors: Risks of Ilm evaluation by private data curators. In The Fourth Blogpost Track at ICLR 2025, 2025.
[35] Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic. Lim dataset inference: Did you train on my dataset? Advances in Neural Information Processing Systems, 37:124069- 124092, 2024.
[36] Charles Marx, Flavio Calmon, and Berk Ustun. Predictive multiplicity in classification. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 6765- 6774. PMLR, 2020. URL https:
mlr srress/v119) http/11//mm/am0aaaam marx20a......mmmmmmmmmmm html.
[37] Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg- Kirg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 11330-11343, 2023.
[38] Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models. In Proceedings of the 33rd USENIX Conference on Security Symposium, pages 2365-2385, 2024.
[39] Matthieu Meeus, Igor Shilov, Manuel Faysse, and Yves-Alexandre de Montjoye. Copyright traps for large language models. In Forty-first International Conference on Machine Learning. 2024.
[40] Matthieu Meeus, Igor Shilov, Shubham Jain, Manuel Faysse, Marek Rei, and elex-Alex de Montjoye. Sok: Membership inference attacks Ilms are rushing nowhere (and how to fix it). arXiv preprint arXiv:2406. 17975, 2024.
[41] Matthieu Meeus, Lukas Wutschitz, Santiago Zanella-Béguelin Shruti Tople, and Reza Shokri. The canary's 's echo: Auditing privacy risks of Ilm-generated synthetic text. arXiv preprint a50iv:2502. 14921, 2025.
[42] Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- Kirk-Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8332-8347, 2022.
13

---

[43] Fatemehsadat Mireshghallah, Archit Uniyal. Tianhao Wang, David K Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 1816-1826, 2022.
[44] Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski. A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo Eric Wallace, Florian Tramèr, and Katherine Lec. Scalable Extraction of Training Data from (Production) Language Models. arXiv preprint
arXiv:2311.10033 2023.
[45] Yonatan Oren, Nicole Meister, Niladri S Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2023.
[46] Ashwinee Panda, Xinyu Tang, Christopher A. Choquette-Chov Milad Nasr, and Prateek Mittal. Privacy auditing of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
[47] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21 21(140):1-61 2020.
[48] Lorenzo Rossi, Bartlomiej Marek, Vincent Hanke, Xun Wang. Michael Backes, Adam Dziedzic, and Franziska Boenisch. Auditing empirical privacy protection of private 1lm adaptations. In Neurips Safe Generative Al Workshop 2024, 2024.
[49] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In International Conference on Machine Learning, pages
PMLR, 2019.
[50] Nikhil Sardana, Jacob Portes. Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.000 04440004448. 2023.
[51] Lesia Semenova, Cynthia Rudin, and Ron Parr. Existence, computation, and implications of Rashomon sets. Machine Learning, 1 11:33511-3569, 2022. doi: 10.10007//19994-22--66666
[52] Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The 'weeftt International Conference on Learning Representations, 2024.
[53] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[54] Congzheng Song and Vitaly Shmatikov. Auditing data provenance in gext-eeneration models. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 196-206, 2019.
[55] Gemma Team et Gemma 2: Improving Open Language Models a Practical Size, 2024.
https: aa/r//xvvvvvva oww//osg////////oooo/oooooeee//e abs/ 00118.
[56] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:
2023.
[57] Cheng Wang, Yiwei Wang, Bryan Hooi, Yujun Cai, Nanyun Peng, and Kai-Wei Chang. Con- recall: Detecting pre-training data in I1ms via contrastive decoding. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1013-1026, 2025.
[58] Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022.
[59] Jamelle
David C. Parkes, and Berk Ustun. Predictive Multiplicity in Proba- bilistic Classification, 2022.
[60] Johnny Wei, Ryan Wang, and Robin Jia. Proving membership in 1lm pretraining data via data watermarks. In Findings of the Association for Computational Linguistics ACI. 2024, pages 13306-13320, 2024.
14

---

[61] Roy Xie. Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Gong, and Bhuwan Dhingra. Recall: Membership inference via relative conditional -ogeliiiliso.ds In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8671-8689 2024.
[62] Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093-3106, 2022.
[63] Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 3/st computer security foundations symposium (CSF), pages 268-282. IEEE, 2018.
[64] Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference attacks. In Forty-first International Conference Machine Learning, 2024.
[65] Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data, 2025. URL https://arxiv org/ abs/2409. 19798.
[66] Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li. Min-k%+++ Improved baseline for pre-training data detection from large language models. In The Thirleenth International Conference on Learming Representations, 2025. URL https: / https:// http://ppeeeeve..eettooo ott.
15

---

A Membershhp inference attacks
Security game, threat model, and notation. Membership inference is formalized security game between a challenger and an attacker (i.c.. adversary). Let D denote the underlying data-generating distribution over samples (and labels, if applicable). The challenger draws a finite training dataset D ~ D and trains target model h on D. A challenge record æ is selected to be either member (æ E D) or non-member (ær D). The attacker given query access together with auxiliary resources and outputs a guess about x' membership; success means accuracy exceeding random guessing.
The strong attacks we study-LiRA and RMIA (Section 3.1 and Appendix B) By-assume the attacker (i) query h arbitrary inputs to obtain per-sample outputs (losses, logits, confidence scores), and (ii) train reference models f E 4 by replicating the target's training recipe on datasets drawn from the same population D that generated (in practice. from large proxy corpus approximating D). For fixed query sample æ, each reference's 's training dataset either includes æ (IN) or excludes æ (OUT), yielding a per-2 partition:
1N (z) S + Pour(2) S +
(IN(2) n Pour( = 2.
This is the online setting: the offiine setting assumes access only to DoUT( (a=). Neither attack requires access to the target's parameters ID; only queries and attacker-trained references are needed. In research settings, one often controls both the target and references, which allows evaluation across many æ with known membership. is common (though not required) to choose so that [P)((z) s [PouT()) [ for stability. We do in this work.
Observation statisties and membership scores. For any model g and query sample æ, let
8(g x) E R
denote a fixed scalar observation statistic from monotone transform of confidence such as a logit). A membership inference attack (MIA) maps the available statistics for (from h, and when used, from 4) to real-valued membership score A(x) E R. with larger values indicating stronger evidence that æ a member.
(c.g.. loss, negative log- log-likelihood,
Baseline (reference-free) loss attack [63]. Using only the target's 's statistic,
(z) sh,, x).
so larger ALoss implies lower loss on æ. Any strictly monotone transform preserves ranking and therefore ROC-AUC. No reference models are used in this baseline approach. Stronger attacks use reference models to yield improved membership signal.
Likelihood Ratio Attack (LiRA) [5]. LiRA uses references to model per-sample IN/OUT distribu- tions over the chosen statistic s. For fixed æ, the attacker forms
{s(f.z) x) : (IN(=)} and {s(f; f Pour(x)},
fits univariate models (typically Gaussians) to obtain densities PIN(- x)) and pour(: x)), and evaluates the target's statistic s(h, æ) under these densities to form a likelihood ratio:
PIN ( (s(h, xx) x))
( =
(2)
Pour (s(h, x))
x))
The online variant uses both IN and OUT; the offline variant performs a one-sided test using only OUT. Working with log (xx) is common for numerical stability; since this is monotone, ROC-AUC unchanged.
Robust Membership Inference Attack (RMIA) [64]. RMIA also compares the target model's statistic on the sample x outputs for from set of reference models 4 but uses different construction based on pairwise likelihood ratio. This ratio is normalized by reference population (e.g., calibration set drawn from D held-out proxy). Define
s(h, x:)
a(x) =
(3)
E s sf,
16

---

The expectation in the denominator is approximated empirically over the trained references. To
improve robustness, RMIA contextualizes this ratio relative to population Z. For each Z:
s(h, x)
a(x)
a<z) E Ejco (f,
L(x,z)
(4)
a(z)
The computed membership score aggregates the pairwise tests threshold 7. > 0:
(R)
x)
(5)
ZEZ
We focus on online (two-sided) variants of these attacks that use both IN and OUT references, as opposed to offline variants that only OUT references.
Decision rules and calibration. Given a real-valued score A(x) (e.g., ALoWs ALRRA or ARMIA), the attacker outputs binary decision about the membership of via
b(z) = 1(A(z) mT}
To operate at fixed false positive (FPR) 7). is convenient write
(xx) {()()) 2 T(1)):
where T(7) is calibrated for the target h using non-members (i.e., samples not in h's training subset DD. (We will sometimes refer the training set as DIN when we want to refer the set of non-members as Dour-)
Calibration to non-members at fixed FPR. Fix target h and an FPR n E [0, 1], and assume larger scores are indicate stronger evidence that æ is a member. Let the non-member (OUT) set be DOUT with size NoUT DoUTl. (The attacker can draw i.i.d. samples from the population distribution D. or use auxiliary data from the same source, independently of the training set, to form DoUT-) Write the scores (A(x:) x E DOUT)) The empirical CDF of OUT scores is
(our(!)
Σ 1(A(w) - t}.
NOUT
ar Denr
We choose the right-continoous empirical (1 17) quaantile
T(n) inf Four(t)) 2 1 1- 1
Equivalently, if A(1) I ... & (Nor) are the sorted OUT scores, let k = [(1 17) NouTl. F: max(j : Agg) = A. and set 7(1) = A(kt1)(AA (ANNrr1))) = +00).
then apply the calibrated binary MIA decision rule
(1) (x) 1(A()) 2 T()).
By construction, this guarantees (finite-sample with ties handled conservatively) that
FPR(n)
Σ 1(A()) 2 r())} 1 Four(((r()) 77.
NOUT
=<Dour
This is because taking the right-continuous quantile ensures that any mass tied 7(7) is counted on the I side of the CDF Therefore, the realized FPR on OUT never exceeds (and may be smaller in the presence of ties).
Common performance metrics. Because MIAs are typically compared across operating points, typical to report ROC curves and AUC (threshold-agnostic), and-when reporting TPR at fixed FPR to set to achieve the target FPR. For RMIA, the internal pairwise threshold y controls per-comparison likelihood ratio test, while the final decision threshold 7 controls the operating point. Calibration may be global (single 7) or conditional (e.g., per class/bucket). All monotone transforms of the score leave ROC-AUC invariant, while operating-pott metries (e.g., TPR at fixed FPR) depend on calibration.
17

---

Practical note. Calibrating FPR without knowledge of ground-truth membership can be challeng- ing [65]. In our experiments, we control training and evaluation, so membership labels are known; this enables exact calibration and measurement at desired operating points.
B Comparing membership inference attacks and observation statistics
At the beginning of this project, we considered two candidates for strong membership inference attacks to use in our experiments: the Likelihood Ratio Attack (LiRA) [5] and the Robust Membership Inference Attack (RMIA) [64]. Both attacks involve training reference models (Section 2) that enable the computation of likelihood ratios (which result in stronger attacks), though they differ in important ways. LiRA [5] estimates membership by comparing the loss of sample æ in a target model to empirical loss distributions from reference models trained with and without a. In contrast, RMIA [64] performs and aggregates statistical pairwise likelihood ratio tests between æ and population samples z, using both reference models and estimate how the inclusion of versus z affects the probability of generating the observed model 0 (Appendix A).
By leveraging signal from both models and population samples, Zarifzadeh et al. [64] observe that RMIA can outperform LiRA using fewer reference models. However, no prior work has compared these methods in the pre- pre-trained LLM setting and with large numbers of reference models, leaving open the question of which attack fares better under these conditions.
In this appendix, we investigate this question for the first time, and our results clearly indicate that LiRA outperforms RMIA for a large number of reference models the online setting (Appendix A). We observe limited cases where RMIA can outperform LiRA if the population dataset is large enough and the attack is performed for certain small numbers of reference models. However, we caution generalizing about comparative performance. LiRA seems to perform better with or 2 IN references, while RMIA performs better with 4-16, and then LiRA once again outperforms RMIA for >1 IN references.
Overall, attacks with larger numbers of references perform better, as measured by ROC-AUC. Since our aim is to test the strongest attacks possible- to investigate an upper bound on strong MIA performance this makes LiRA the best choice for our experiments. For those with smaller compute budgets that still wish to run strong attacks using =16 IN references, in some circumstances, RMIA may be better choice.
Following from our discussion of the threat model for membership inference, and how it is imple- mented with slight variations for LiRA and RMIA (Appendix A), we next discuss our experiments comparing the performance of these two attacks. We first show how different choices for the observa- tion statistic impact attack performance (Appendix B). This provides more detail about the choices we make in our overall experimental setup throughout the paper (introduced in Section 3). Then, we show our full results that compare the performance of LiRA and RMIA using different numbers of reference models (Appendix B. B.2), which lead us to choose LiRA for the experiments that follow.
For all experiments comparing LiRA and RMIA, we train 140M parameter models 77M samples, which equates to approximately 2.8B training tokens (i.e., what is optimal for this model size, according to Chinchilla scaling laws [26] with an over-training multiplier of 20).
B.1 Different observation statistics
In our initial experiments in Section 3, we compare LiRA [5] and RMIA [64] to decide which strong attack use. We also investigated the efficacy of different observation statisties for membership inference. We tested model loss and model logits (averaged over the entire sequence). For example, in Figure 8, we plot the ROC curve for using LiRA attack 140M model trained on 7 samples with 128 references. The plot shows the true positive rate (TPR) against the false positive rate (FPR) on a log-log scale, with one ROC curve each for logit and loss statisties. For the logit curve, ROC-AUC-0.576 while the loss curve has a higher ROC-AUC-06678, This indicates that, in this setup, using loss as the observation statistic results in more effective attack compared to using logits. Based on results like this, throughout this paper, we opt to use loss observation statistic s.
18

---

100
Type,
Logit,
10-1
True Positive Rate
10-2
10-3
10-3
10-6
10-7
10
10-6
10-5
10
10-3
10
10-1
10a
False Positive
Figure 8: Influence of observation statistic type on MIA Performance. For the 140M model, plot ROC curves compare the efficacy of using model logits (AUC=0.576) and model loss (AUC=0.678) as observation statistics for membership inference with LiRA. In this setting, loss provides a stronger signal for distinguishing members from non-members.
B.2 MIA attack performance for different numbers of reference models
Figure compares LiRA and RMIA, showing ROC curves and ROC-AUC for different numbers of reference models. Figure 10 provides alternate view of the same results, plotting ROC-AUC for both attacks as a function of reference models. LiRA's performance generally dominates RMIA's. increase the number of reference models, while RMIA's ef-
LiRA continues improve fectiveness plateaus. However, with 4-16 IN references, RMIA surpasses the performance of LiRA. It essentially matches LiRA using 16 IN references. That is, with 4 references, LiRA exhibits ROC-AUC-0.594 which under-performs RMIA's corresponding ROC-AUC=0.66 but LiRA's ROC-AUC increases 0.678 with 64 IN references, which outperforms RMIA's ROC-AUC=0.658.
Also note that RMIA exhibits distinct diagonal pattern at low FPR (Figure 11). While RMIA aims to be strong attack that effective in low-compute settings, we find that large population Z is necessary to obtain meaningful FPR at very low FPR thresholds. In particular, for minimally acceptable FPR RMIA requires population size [Z] that is FPR... In practice, this is quite expensive, as RMIA's membership score computed via pairwise comparisons with these [Z] reference points (i.e., there O(Zl)) pairwise likelihood ratio tests for target record x, see Appendix A). In these initial experiments we only used [Z] =0,00000000 samples. We measure performance of RMIA on larger population sizes below in Appendix B.2.1.
Overall, as noted Section 3, while both attacks clearly beat the random baseline of ROC-AUC-0.5, neither is remarkably successful in this setting: regardless of the number of reference models, neither attack achieves that meets or exceeds ROC-AUC-0.7.
B.2.1 Further experiments on RMIA
We now further investigate RMIA, decoupling its different components. We investigate removing the dependence on population Z, population sizes other than 110,00,, and varying threshold yy
Eliminating dependence population Z. First, consider the simplest form of RMIA (simple), eliminating its dependence on a population E and using a<x)) directly (Equation 3). Figure 11 shows the ROC curves for all three MIAs attacking epoch on a training set size of 19 samples. We use 128 reference models and consider 2 2x2 29210_220 target records with (as elsewhere) balanced membership/non ptterssiipnooommmbbrspsipi analyse MIA. We find all three attacks reach similar ROC-AUC values.
target model with 10M parameters, trained for I
We also gauge MIA performance by evaluating the TPR at low, fixed FPR. To understand the values RMIA reaches for TPR at low FPR, an important subtlety arises from the entropy of the score distribution. Attacks that produce very coarse membership scores inherently limit achievable
19

---

10a
100
Positvve Rate
10-1
True Positive Rate
10
10
10
10
Number reference models, AUC
10
Number reference models, AUC
10
10-3
10-1
10:1
10
10-2
10-1
10
10
10-1
10
10
10-1
10-7
10-1
10a
False Positive Rate
False Positive Rate
(a) LiRA
(b) RMIA
Figure 9: Comparing LiRA and RMTA. We attack a 140M-parameter model, with the target and references trained on =7M samples. ROC curves illustrate the effectiveness of (a) LiRA [5] and (b) RMIA [64] for different numbers of reference models. As we increase the number of references, LiRA's performance surpasses RMIA's, measured by ROC-AUC. These plots show the number of IN references. (There are 2 as many references in total, accounting for OUT.)
0.70
LiRA
0.69
RMIA
MIA AUC
0.64
0.62
0.60
0.56
0.5
2
2
2
Number of reference models (log)
Figure 10: Comparing LiRA and RMIA. As an alternative view of Figure 9, we plot the ROC-AUC achieved by both attack methodologies for an increasing number of reference models. As the number of references increases, LiRA's performance continues to improve, while RMIA's gains saturate. Overall, LiRA is the stronger attack. This plot also only shows the number of IN references on the x-axis (there are the same number of OUT).
TPR at very low FPR. For example, as RMIA compares axx) to a(z) for all Z to compute its membership score ARMIA(a) (Equation 5), there are maximally [Z] unique values ARMIA (æ) can take for all æ. This limits the score's entropy and the possibility of achieving meaningful TPR at very low FPR. This explains the diagonal pattern for RMIA in Figure 11, where 10|=10,000. By contrast, both LiRA and RMIA (simple) provide membership score that is not limited in entropy, leading to more meaningful values for TPR at lower FPR.
Increasing the population size ZZ] We next test further increasing the size of the population Z when computing RMIA. For the same setup as Figure 11, Figure 12 shows how MIA performance varies with the size of Z. We observe very similar values for RMIA (simple) and RMIA ROC-AUC for all population sizes that we test. When examining TPR at low FPR, we find that increasing [ž] improves the MIA performance. Indeed, the increased entropy in ARMIA(#)) now allows the attack to reach meaningful values of TPR. for FPR low 10 Notably, for all values of [Z] we consider, LiRA still outperforms RMIA at low FPR, while the [2] likelihood comparisons in RMIA for every target record also incur additional computational cost.
Varying threshold 7. Finally, we evaluate RMIA under varying threshold As increases, in Equation 4, it becomes less likely that a<x) sufficiently exceeds a<2) for many z E to count toward the score-i.c. i.... that (x))/a(z) 2 (Equation 5).
20

---

10*
0.8
True Positive Rate
True Positive Rate
0.6
54-
0.2
Allacke AUC
URA
UFA 5,28034e-04
0.4
0.6
10
10F
False Positive Rate
False Positive Rate (log)
Figure 11: Comparing LiRA, RMIA (simple) and RMIA. Attacking a 10M-parameter model
trained for epoch with training set size of 2 samples.
1.0
10a
10-4
True Positive Rate (log)
True Positive Rate
30
0.6
30
0.4
Atback
Attac TPRgeFR=1.000
LBA 0.9211
20-4
URA 5.280
9MIA (simpl)) 0.59883
RMU (smplee 400222 04
0.2
RMIA (4] 10k, 033877
RMI 500, 0.59883
10-0
RMA (4) 12k, 4.24028-44
RMU
1.00022=04
[2] -100k 0.59883
100K, 3.40226
RMIA 12) 200k, 0. 0.50883
RMIA (2) 300k. 3420226004
0.0
10th
1.0
10-3 30
False Positive Rate
False Positive Rate (log)
Figure 12: Performance of RMIA for different population sizes [2/. We attack a 10M-parameter model trained for epoch with a training set size of 2 samples.
Again for the same setup, Figure shows how RMIA performs for varying values of 7, considering both |0|=10,000 10,000 (Figure 13a) and =2|=300,000 300 (Figure 13b). While MIA ROC-AUC remains rela- tively stable as y increases, the TPR at low FPR varies. For [ =00,0000, the TPR at FPR=10 decreases for increasing values of y, reaching 0 for 1.1. This is due to the reduced granularity of RMIA's membership score: for larger 7, fewer z satisfy a<z)/a(>) 2 7/ this constrains the entropy of the RMIA score, making it harder to reach meaningful values of TPR at low FPR. A larger reference population (|Z==300,000 mitigates this issue, allowing meaningful TPR even at low FPR for similar values.
Taking these three sets of results together, we find LiRA to outperform RMIA when a sufficiently large number of reference models is available, especially in the low-FPR regime. Since our aim is to study the strongest attacks, we adopt LiRA as the primary attack throughout our experiments.
B.3 MIA performance in the offline setting
As stated in Section 2 and Appendix A, the literature distinguishes between online and offline settings for reference-based MIAs [5, 64]. the online setting, the attacker has access to reference models trained on data including the target sample ($I)) and excluding it(T) In the offline setting, the attacker only has access to PoUT- Throughout this work, we consider the strongest attacker, and thus report all results in the online setting.
21

---

10-1
True Positive Rate (log)
True Positive Rate
10a
10-1
y=10
ANA 1-001 0.59883
y=1001 3.84226
MNA y-1.0.111000000 0.59575
10-1
KMBA y-105 1.05 3.340240
PMA- 1-11, 0.59535
ROBA
y 0.00000e-000
KMA 1-11 0.3329
RP8A 2.0000000
10
0.0
02
0.4
06
1.
10
20
10-1
10-1
15-2
10-1
109
False Positive Rate
False Positive Rate (log)
(a) [2| 10,000
10"1
True Posítive Rate (log)
True Positive Rate
30--
TRRg)FRR
2.83344-004
(simals) 0.59883
10t4
(imple) 3400222
y-
ANA y- 1.01, 0.59883
RMAA y- 1.01, 3.42922e
PNA y-10 0.59883
10:1
NMYA y- 1--00,05 3.36022e:04
AMA y_1.00.55877
RVUA 1.. 3.54023:
AMA 1-1. 0.9999
4.24077044
0.0
02
0.4
0.6
08
1.
10
10
10-1
10-2 10-- 10-1
109
False Positive Rate
False Positive Rate (log)
(b) [ 300,000
Figure 13: Performance of RMIA for varied 7. We attack 10M-parameter model trained for 1 epoch with a training set size of 2 samples, varying the threshold used to compute ARMIA.
For completeness, we instantiate MIAS the offline setting in the same experimental setup as considered above for our additional RMIA tests (Appendix B.2.1). We test the offline versions for both LiRA and RMIA, as originally proposed in Carlini al. [5] and Zarifzadeh er [64], respectively.
For LiRA, without PIN. we are unable to approximate the probability PIN (s(h, x)) (Equation 2), and so just consider the one-sided hypothesis test instead of the likelihood ratio:
ofiiie (x) = 1 - Pour (s(h, ah)
For RMIA, we now compute the denominator in a<x) (Equation 3) by taking the expectation over the reference models that are available to the attacker, i.e.:
s(h, x
Otoffline (a:)
E Esedan xf
Note that Zarifzadeh et al. [64] propose to further adjust the denominator by using variable a (their Appendix B.2.2) to better approximate the E Eyga-s(f. (ff.). z), when only references PoUT are available. We set a=1 and just compute the empirical mean across reference models ΦoUr to approximate the expectation in the denominator. We then compute O'fttine and use membership inference score
Oloffline
oRMiA,offine (x)
[Loftine (æ, 2) 2 7],
where Lofftine (x, z) Otfflinn (z)
ZEZ
Figure 14 compares the MIA performance between the online and offline setting, for LiRA, RMIA (simple) (which does not use the reference population Z, Appendix B.2. and RMIA; we set 7=1 and =200000,000. We again attack (0M-parameter model trained for 1 epoch, using training set size of 219 samples. We use 128 reference models for the online setting and 64 in the offline setting (on average, per target sample).
22

---

10-1
True Positive Rate (log)
True Positive Rate
10-9
10-
10-1
(offinal TTR.)FRR
10-4
10:4
0.0
02
0.4
1.
10
20th
10-1
10-1
10-2
100
False Positive Rate
False Positive Rate (log)
(a) Oftline
10.2
True Positive Rate (log)
True Positive Rate
10-4
10-4
tonline) TTRU)ERR
10
(imp)))
(amplel 400226
0.30853
3.420220 04
02
0.4
oio
10
20th
10
10-
10-
19-1"
False Positive Rate
False Positive Rate (log)
(b) Online
Figure 14: MIA performance the offiine and online setting. We attack a 10M-parameter model trained for epoch with a training set size of 29 samples, considering 128 references in the online setting and only the corresponding models ooUT in the offline setting (on average 64 references per æ).
We find that, in this configuration and with this number of reference models, offline RMIA outperforms offline LiRA, in terms of both ROC-AUC and TPR at low fixed FPR. This suggests that RMIA's offline variant more accurately captures membership signal compared to the one-sided hypothesis test used in offline LiRA. In contrast, in the online setting. LiRA and RMIA achieve similar ROC-AUC, with LiRA performing better than RMIA in the low-FPR regime.
C More experiments on Chinchilla-optimal models
In this appendix, provide additional details our experiments involving LiRA attacks Chinchilla-optimal [26] models different sizes in Section 3.2: 10M, 44M, 85M, 140M, 489M, and 1018M. We summarize training hyperparameters Section G.
Observing changes in loss during training. In Figure 15a, we show the decrease in validation loss over single epoch. The z-axis represents the fraction of the training epoch completed (from 0.0 to 1.0), and the y-axis shows the corresponding loss. As expected, all models exhibit a characteristic decrease in loss as training progresses. Larger models (namely, 489M and 1018M) demonstrate faster convergence to lower loss values, reflecting their increased capacity to fit the training data, They also maintain lower loss throughout the epoch compared to smaller models (10M-140M).
Investigating the role of learning rate schedule. In the Chinchilla- optimall setting, we also investigate the role of hyperparameters on MIA performance. In Figure 15b, we show ROC curves that compare the MIA vulnerability (with LiRA) of 140M-parameter models (trained on s records, with 128 reference models), where we vary the learning rate schedule: Linear (AUC=0.676), Cosine (no global norm clipping, AUC=0.660), Cosine (no weight decay, AUC=0.673), and standard
23

---

10"
10M
True Positive Rate
10
85M
Loss
140M
10:1
489M
Learning rate schedule, AUC
10-3
Linear, 0.676
Cosino (no clip), 0.660
10-
Cosine weight decayl. 0.673
Cosine, 0.675
5
10-2
Epoch
False Positive Rate
(a) Validation loss over time (1 epoch)
(b) LiRA ROC for different learning rate schedules
Figure 15: Investigating training dynamics byperparameters. (a) Validation throughout the 1
training epoch for experiments involving Chinchilla-optimal trained models of various sizes. (b) The effect of learning rate schedule on LiRA's attack success for 140M models using 128 references.
10
True Positive Rate
10-1
10 2
10-3
10
10
Dataset, AUC
10-6
C4, 0.683
Deduplicated C4, 0.680
10-7
10-7
10-6
10-5
10-4
10-3
10-2 10-1
100
False Positive Rate
Figure 16: The role duplicates on MIA vulnerability. We observe no significant differences (particularly FPR increases) between models trained C4 and de de-duplicated C4.
Cosine (AUC=0.675), As with all of our ROC plots, the TPR plotted against the FPR on log-log scale. The ROC-AUC values for each curve are relatively close. This indicates that, while there are some minor differences in attack performance, the choice learning rate schedule among those tested does not lead to drastically different MIA outcomes.
D Additional experiments exploring the limits of LiRA
In this appendix, we provide additional experiments that explore the limits of LiRA when there are duplicate samples in the training data, and (complementing results in Section 4) when there are varying numbers of training epochs and varied dataset size.
Investigating the role of duplicate training samples. Given the relationship between MIA and memorization, and that prior work observes important relationship between memorization and training-data duplication [29], we test the relationship between MIA vulnerability and the presence of duplicate training samples. In Figure 16, test the Chinchilla-opiimally trained 140M model on C4 and a de-duplicated version of C4. We de-duplicate C4 according to methodology described in Lee et al. [29], where remove sequences that share common prefix of at least some threshold length. This reduced the C4 dataset size from 364,613,570 to 350,475,33 samples.
We observe that the presence of duplicates has a negligible impact AUC: it is 0.683 for C4, and 0.680 de-duplicated C4. other words, least terms of average attack success, the presence of duplicates does not seem to have significant impact. However, further work is needed to assess

---

10a
10a
10-4
True Positive Rate
10a4
10-3
10-2
Positive
10
Epech, AUC
Epoch, AUC
0.573
6, 0.674
10
1,9694
3,9997
15.00000
10-3
0,599
3,6.50
0.7.710
8.0.7
10-1
4. 0.621
9, 0.787
5
0392
13.1.002
1,000
10-1
0.645
1.70.79
10-1
0.995
14.1.000
10
10
10
10
10
10
10
10
10
10
10
10
10
10"
False Positive Rate
False Positive Rate
(a) 140M model, 77 samples, 10 epochs.
(b) 140M model, 5500K, 20 epochs.
Figure 17: Over-training and MIA. ROC curves demonstrate that MIA success significantly increases as models are trained for more epochs. (a) The 140M model shows AUC rising from 0.573 (1 epoch) to 0.797 (10 epochs). (b) Attacking a 140M model trained on a smaller dataset shows rapid escalation in AUC, from 0.604 (1 epoch) to near-pert near-perfect membership inference (AUC=1) by 13-20 epochs, highlighting that overfitting from prolonged training severely heightens privacy risks.
how attack success changes with more stringent de-duplication, since our de-duplication procedure only removed 10M samples from the dataset.
Varying training epochs and dataset size. In Figure 17, we reduce the training set size from 77M (Figure 17a) to 2 =50KK samples (Figure 17b) on the 140M model and train for 10 (Figure 17a) and 20 epochs (Figure 17b). Both figures show ROC curves that illustrate how MIA vulnerability changes with an increasing number of training epochs. The goal of these experiments is to investigate MIA becomes better with more training epochs, and if so, how attack performance improves over epochs function of training dataset size.
For the 140M model trained on 77M samples for 10 epochs, the AUC increases with more epochs, starting from 0.573 at 1 epoch and reaching 0.797 at 10 epochs. For the 140M model trained on =500K samples for 20 epochs, we observe a more dramatic increase MIA vulnerability. The AUC starts 0.604 for epoch, rapidly increases to 0.864 by 2 epochs, 0.944 by 3 epochs, and approaches perfect MIA (AUC close to 1.000) after 13 epochs. Of course, both of these experiments effectively sanity checks. We intentionally over-train in both, and relatively small training dataset size in the second.
Full results for various-sized Chinchilla-traines models and fixed training set size. We provide full results for attacking Chinchilla-optimal models of various sizes for epoch (Figure 2b), and attacking various model sizes trained on a fixed dataset of 2.83M samples for epoch (Figure 4b). Both of these figures in the main paper show how TPR varies at fixed FPR in line plots. Here, in Figures 18 and 19, give individual ROC curves for experimental results summarized in each of those figures, respectively. For each subplot, each line indicates different target model that we attack. As discussed previously, some larger models appear to have more variance in their ROC curves over different experimental runs. In Figure 19i, we see that although AUC is similar over different target models, there catastrophic attack failure for one model small FPRs.
25

---

107
Muteli
30P
Vede son
307
Vodei eine,
120 PuStio R350
2014 2320000.000
83% 0.60822.002
10-1
10
13
1841
30
10-1
Falso Posttive Rate
Faice Postivve Rats
False Resitive Rate
(a) 10M
(b) 85M
(c) 302M
10:1
66544.002
10-
1088M, 0.553-0
20-1
10-1
(d) 489M
604M
(f) 1018M
Figure 18: ROC curves and AUC for Figure 2b. We attack different model sizes trained on the Chinchilla-optimal number tokens. each subplot, each line indicates a different attacked target.
23H, 0.32226000
10-4
44H, 0.30120002
ESM, 0.32220003
1017
16-
10-1
16-
20-1
20T1
10
12-4
10
357
15
3011
10
10-4
10d 10
357
Positive
Positive
(a) 10M
(b) 44M
(c) 85M
Mode:
Thus Prsittu 2L955
240M, 0.655a8.012
20-4
332M, 0.735a8.005
Pesitive
20
-0-
dte,
(d) 140M
(e) 302M
(I) 425M
10-1
624%, 0.53tal.025
20
10
10.4
10.4
20-4
Model
10
(g) 489M
(h) 509M
ii) 604M
Thus Bate
10-4
10090, 0.62200332
12.4"
Posttive
1018M
Figure 19: ROC curves and AUC for Figure 4b. We attack different model sizes trained on the same number of samples (==.. In each subplot, each line indicates a different attacked target.
26

---

Varying reference models for all Chimchllla-opimmally trained model sizes In Figure 20, we replicate the experiments Figure 18. but we vary the number of references. Each row in the figure is for different-sizee model. Each column uses different number of total references (IN plotted) to perform the attack. We attack 8 targets trained on different training data subsamples each plot.
Unsurprisingly. MIA improves as we use more references. This mirrors our findings in Figure 9a. The key point of these figures reference line y-r. We also show that there is variance (in the insets) across attack runs for the same model size. These are not be taken detailed results that should be closely examined. (This is why the plots are not very large.) We investigate instability in Section 5 and Appendix E.
show the general pattern where the ROC curve relative the
CP
(b) 85M
aaheeris medel
vodee
North
Proive
302M
Petfec
197
199
(d) 489M
todea
(c)
604M
sefeeneee musics
Tederee
726 codereeeee
(X0))
tae
Mer
Morhalee Mer
(f) 1018M
Figure 20: Extended ROC curves and AUC for Figure 2b. For each subplot, each line indicates different target model that we attack. Each row different model size. Each column represents using LiRA with different number of total reference models. Each subplot also records the average AUC across attacks on different targets.
27

---

E
Investigating instability in per-sample membership decisions
As noted in Section 5, we observe substantial per-sample instability in membership decisions. We also notice significant variability in ROC-AUC across attacks in Figure 20. However, because standard attack metries such as ROC-AUC are aggregates over samples and decision thresholds; they report metrics according to average FPR/TPR over many samples. As such, they can mask this instance-level variability. We visualize and quantify individual-sample instability, and connect our analysis to prior work in other areas of statistics and machine learning.
In Appendix E. we provide extended results for Section 6.1 variation in per-sample true positive probabilities, and then in Appendix E.2 we include more results and discussion on flip rate (Section 5).
E.1 Variation in per-sample true positive probabilities
For the 140M model, plot the mean and standard deviation of the per-sample true positive probabilities, Pr(predicted as member member) for = 216 samples. For each sample, we compute variance across 64 target models (for which the sample is a member); overall, this experiment trained 128 models (140M size) on different random splits of the 22 samples. We member, compute Pr predicted as member, using pxx[- ==xtr=))+oourlll
0.5 to determine if the member (Section 6). We loop over each model, selecting it as the target
pIx(-Ix)
sample is predicted
model and the remainder as reference models used for LiRA. Since each sample had probability of 0.5 for inclusion in the training set, for each sample, we have on average 64 target models where the sample was in training and 64 for which it was not.
In Figure 21, we provide three plots that give different views of the same data. Figure 21a plots the true positive probability for each member. We sort members by the mean value of their true positive probability (i.e., the mean of Pr(pun Pr(predicted as member (mmberr) over 64 target models). member ID corresponds to this ordering. We also show the variance over the 64 target models by plotting the standard deviation.
Together, Figures 21b and 21c provide an alternate view of Figure 21a. Figure 21b plots the histogram of the mean Pr(preiicted as member member) for members across their respective 64 target models. The average across these mean true positive probabilities for each member 0.543. However, note the distribution of per-sample means: while the across-sample average of the per-sample means is 0.543, a substantial mass of members exhibits mean Pr(predicted as member member)>0.6. The spread is large: the average per-sample standard deviation is 0.143, with many members exceeding standard deviation of 0.2.
Overall, variance is significant. The individual member true positive probabilities for each target are, when considered together, highly unstable. This variance can help explain why attack ROC-AUC is perhaps lower than one might have hoped; there is considerable variance in the underlying sample binary decisions. Altogether, this provides additional nuance concerning the extent of (altematively. the limits of) attack robustness.
8303
Peic 0.545
70000
63000
154300
3000
5030
2009
2300
19900 sobie 19000 19000 120000 10060 3aice
03
(a) Per-sample true positive proba- bilities (mean 1 standard deviation). sample true positive probabilities ordered from smallest to largest. from Figure 21a.
(b) Histogram of average per-
(c) Histogram of standard deviation of per-sample true positive probabil- ities from Figure 2
Figure 21: Instability of per-sample true positive probabilities. For each of 22 samples æ. we
compute the mean and standard deviation of Pr(predicted as member member) across B-64 target models. (a) After sorting samples by their mean, the mean and one standard deviation band. A histogram of (b) these per-sample means and (c) of the corresponding standard deviations.
28

---

E.2 Analyzing per-sample membership decision instability
This appendix deepens our analysis of per-sample MIA binary decision instability. We formalize flip rate [14]_the metric we use to measure instability at the per-sample decision level- anel-and its unbiased empirical estimator (Appendix E.2.1). We then explain how we measure flip rate in the MIA setting used in our experiments, and why the metric is informative for strong MIAs (Appendix E.2.2). We connect our results to prior work on model/predictive multiplicity [3] (Appendix F.2.3).
Then, we derive an exact acceptance band (at level a) for deciding when a sample's binary membership decisions are statistically indistinguishable from a coin flip: for a finite number of targets B and acceptance level a, we obtain the resulting flip cutoff (B)) that we deem the minimum required for æ's MIA decisions be called "statistically indistinguishable from a coin flip at level a." (Appendix E.2.4). Using these tools, we present extended empirical results for two model sizes. 140M and 302M (Appendix E.2.5). We estimate how much of standard attack performance (ROC-AUC) can be attributed to coin-flip-like decisions as opposed to reliable inference (Appendix F.2.6). We also discuss additional ways to interpret the purpose and results of these experiments (Appendix E.2.7). Finally, we discuss a more intuitive, but significantly more expensive, alternative approach for computing per-sample MIA decision instability, which, in principle, an attacker could compute (Appendix E.2.8). We expect this procedure to surface qualitatively similar instability that we observe with the setup we test.
For reference, the acceptance-band cutoff values used in the figures/tables are 40.05 (125) Rs 0.490 and tu.us (127) x 0.487 for the 140M and 302M models, respectively.
Key points. This is long appendix, so we summarize key points here, For a fixed sample æ and FPR 1j, we care whether the binary membership decision produced by LiRA is reliable (stable across equally plausible targets) or statistically indistinguishable from coin flip with respect to target training randomness. We compute flip rate with respect to the reed-induced distribution μ. Different seeds reflect realistic training randomness (e.g., batch order); aggregate metrics are stable, indicating that μ is not pathologicalddegeeeracc
High flip rate (near 0.5) means the decision for x is effectively a coin flip across plausible targets, so particular target is not evidence of reproducible inference for a:; it is lucky draw. Aggregate ranking performance (c.g., AUC) can still be > 0.5, but that is a different claim about averages. We call the MIA decision for "indistinguishable from a coin flip at level a" if, under the exact two-sided binomial test with B votes K ~ Binomiil( B, 0), we fail to reject Ho 0 = 0.5, where = Pr (6()) (x) = 1]. This yields a concrete cutoff t (B) on flip,, B (x) via the equal-tails
true positive
acceptance region under Ho; samples with flip,, f())),,((x)) 2 t. (B) are deemed indistinguishable from a coin flip. (See Appendix E.2.4 for the derivation; the values we use in practice are noted above.) "Indistinguishable from coin flip at level a" is standard, finite-sample exact test with a clear, observable cutoff ta (B).
Another way to understand these results is to see that, if 0 = Pry (b1 (xx) = 1] ~ 0.5, then
Pr (1)) = 6(2) (x)] = 1 - fl) 2 0.5.
Retraining the same pipeline on the same data would reproduce the same decision for æ only about half the time. This is the operational meaning of an "indistinguishable from coin flip" per-sample MIA decision. Importantly, this claim concerning flip rate is about MIA decisions, not the underlying scores. Even if LiRA scores for x carry some signal, decision instability can be high when the calibrated threshold Tz-(7) varies across sceds; AUC may remain non-trivial while flip rate is high. Our claim is specifically about the reliability of per-sample decisions. (We further discuss interpretation details in Appendix E.2.7.) We apply this per-sample test and report descriptive counts across many Φ; the inferential claim is per sample ("indistinguishable from a coin flip at level a"), which is appropriate for the setup of the MIA security game.
The classifier threshold (T) is calibrated on non-members for each target (trained with seed r), anchoring non-member decisions to their own distribution while leaving members more exposed to seed-induced score and variation, especially where IN/OUT overlap. This is a feature of the real attack protocol, not an evaluation artifact. With finite B, some truly non-coin-flip-lite x could be labeled voin-flip-liks by chance. The exact binomial test controls Type-I error at level a; the
29

---

acceptance band and ta (B) make the rule explicit. In this decomposition (e.g., contributions to TPR and AUC), we filter only the "coin-flip-like band (not all highly unstable cases like t. (B))). so reported performance is conservative upper bound on reliable inference.
E.2.1 Measuring instability of individual membership decisions with flip rate
To complement our measurements of typical metrics from work on MIA, we adopt a metric from Cooper et al. [14] for measuring per-sample MIA decision instability. We first review this metric, then specify our MIA-calibrated version and its unbiased estimator.
Self-consistency across a distribution of models. g ~ V = M. denote a model drawn from the distribution induced by training algorithm A (a function of random seed) with training data from distribution D. For a binary decision rule bg (x) E {0, 1) (e.g.. br(x) (g)()) 2 D) Cooper et al. [14] define the self-consistency at æ as the pairwise agreement probability under two i.i.d. draws:
SC(x) = Pr [b)(2) = (x)).
(6)
9.9" 9.0"
For such binary decisions, SC(x) [0.5, 1]: values near indicate stability among binary decisions for æ in spite randomness in the training process; values near 0.5 indicate that the decision for æ using this training process is statistically indistinguishable from coin flip [14]. A standard U-statistic yields an unbiased estimator: E[SC(=)] SC(z). Note that SC is defined for any æ. Cooper et al. [14] estimate it for samples in held-out test.
Flip rate on calibrated MIA decision rules. In our setting, we fix the dataset ID D and vary only the training seed, which affects batch order during training. We adapt SC from MADD to the MIA decisions under P.A.D calibrated a fixed FPR.
Let r~ = MAADD denote a target model drawn from the seed-induced distribution with the (fixed) training dataset ID. Let (x) R be the attack score (e.g., LiRA posterior, Equation 2) for sample æ. For a desired false-positive rate vj E [0. 1), define the per-seed calibrated threshold (7) (e.g.. the
(1 quanttle of A, non-members for that seed), and the calibrated membership decision
( (a) 1 (x) 2 (T)
(as in Section 2 and Appendix A). Unlike Cooper al. [14], focus on disagreement between cross- seed MIA decisions for æ rather than agreement. The (population) flip rate at a under μ and FPR 17 is
flip, (x)
Pr
(x))
+ (2) (x)] = 1 - (z),
(8)
which lies in [0, 0.5] at the population level, with 0 indicating that decision for æ does not flip/ is stable across target replicas and 0.5 indicating that the decision for æ behaves like coin flip. (The operator point v is left implicit in the use of SC in Cooper et al. [14], as the authors always set 7-0.5 in practice.)
Note that we deliberately calibrate per seed, as this mirrors how MIAs are actually run in practice: a single target is calibrated at its chosen FPR. Here, we vary the target (via seed) to expose instability across plausible targets r~ y using the same training recipe.
Unbiased estimator (order-2 U-statistic) and closed form. In practice we estimate the population flip rate (Equation 8) for concrete number of target replicas B trained with different random seeds that control batch order. Given B 2 2 i.i.d. target replicas ri...., ~ with calibrated rules (/?) the canonical unbiased estimator of flip, (x: is
B 2
( =
Σ ISKjIB
100 (x) + (0) (x)).
E B[iip, (x)) = flip,, (9)
Let (x) - (x)
1 Σh (1) (æ) and Bo - Bi (x) be the numbers of "member" and "non-member"
binary decisions among the B replicas for æc Then, Equation 9 has the closed form
Bo(x:) (ar)
(x))
(10)
B(B-1 1)
30

---

Maximizing (x) (1 under (((z))B((z)=2) yields the finite- B upper bound, since
B
flip,, (2)
2 [B/2] [B/2]
2(B - B
2
2(B - 1)'
even,
B 1)
1
B odd,
2B 2
2B
which exceeds 0.5 and converges to 0.5 as B > oo (e.g., B-125 + 0.504).
To see why, note that Bo (a) B (x) is maximized by the most balanced vote split (i.e., Bi S
[B/2] If B is even, i.e., B=2k; the maximum occurs at Bg(x) B (a) = [B/2] = k: so
B²
2.(B²/4)
B
1
B, k.2
fipmax B(B
+
4
2(B
1)
2
2(B - -1
If B is odd, B = 2k + 1, the maximum occurs Bo(x)Bi(x)) = [B/2]
B/2] k(k + 1), 80
B² -1
flip 2. 2.B2 2BB 1)
B²
(o(x) (x) = k(k+1)
B(B-
2 2
Of course, this means that at low B, flip rate can have values that quite far away from 0.5. For example, when B=2, the flipmax=.. Nevertheless, this is the right choice of metric, as it is unbiased. In our experiments, ensure that the flip rate is easily interpretable by plotting results where the minimum B=1 such that flipmax x0.544. We discuss this further in Appendix E.2.4.
Why the U-statistic is the right estimator (unbiasedness). Fix sample x and an FPR 17. Write (()) (x) (0, 1} for the calibrated decision of target r ~ MAD.- Let b denote generic draw of B (x), and set
0 0= Pr 1] [0.
Draw B= > i.i.d. replicas bi bi...., bn iid Bernoulli(O). The population flip rate at (æ, n) (the pairwise disagreement probability for two independent draws) is
(ar), Pr(b * b1] Pr(0=11 Pr(0=0,
0(1 0(1-0) (1-0)0 20(1-0))
(11)
Because (0 1) <> 0(1 0) s +. we have (arp,,(a) 20 0) S 1/2 i.e., the population flip rate never exceeds 0.5.
For concrete B, the empirical estimator (order-2 U-statistic) averages the pairwise indicator over all unordered pairs:
f =
1(b
ISKJIB
as in Equation 9. By linearity of expectation and independence,
B -1
R [[]li,, =
Σ
E
(1
2
ISKcjEB
For any fixed pair (i,j) with + j. E(1 b, = Pr(b, * by] =
+ bybb=1,,,,-1]]
Because b,, by are independent Bernoulli(f))
Pr[b,=1, b, =0] = Pr|b:=1| so Pr(b; # 6,] 20 8). Therefore every term in the sum equals 20(1 0), so
0(1 8), Pr(b, mr|b;=0,bb. b, = (1 8)0,
(B)
E (
20(1 - 0) = 20(1 0) -fiip,,(x)
ISiSSB
so flip, B is exactly unbiased for all B 2 2.
31

---

Showing unbiasedness via the vote fraction. For our discussion below and in Appendix E.2.4, useful to see the same result via another argument. As above our discussion of flip rate (Equation 1), Let
(x(x) = Σ TTh (x),
Bo(x) = B - (x))
1=1
By construction, (x) is the sum of B ii.d Bemoulli draws, so
B1 (a) Binomial(B, 0).
Define the vote fraction u<x) = B1 (x) B. Therefore, we can write
Bi (z) Bu(x)
(x( = BB u(z)).
The number of disagreeing unordered pairs B (xx) Bo (a(x)) (choose one "member" vote and one "non-member" vote), so
f()) (1 Bo(x) B1(x) Bo(x)
(B -2)21
B
2
B!
B(B 1)
2B
v()) - (1=v(z)).
(12)
B B-1
Since B1 (x) ~ (B,
E [x)]
B0
B[((w)] =
= 0,
and
(13)
B
B
0(1 - 0)
Var (ar[v(x)) =
(14)
B
because
B (ar)
Var (x))
Var [ar])())] = Var
B
B²
by the scaling law for variance:
VarlaX] = Bax - [[ax])]] = (a(x - E(X1)) = a² E - B[X])²] = Varrr[X].
Next,
[ar[B[ Var Σ (1) (x)) (x)
Var (6 (xx) +2 Σ
Covlb (x), (()
r
B
Var [6,-0)
(independence: Cov(,, :) for + j).
Because 6. (1)(()) (x) is Bernoulli variable with success probability 0, Varlb(") (x)] = 0(1-0). Therefore
B
Var[Bi(x)] = Σ 0(1 - = -B0(-- -0,
and
Var Var[B B1
BB(1 - 0(1 A)
Var [u())]
B
B
claimed in Equation
14. Finally, combining Equations 13 and 14 with the definition of variance,
E [xx) Var[u(z)] + B[o(x))² 8(1 0)
B + 8
(15)
32

---

Therefore, by Equations 13 and 15,
[1 - v(z))] = B[e())] B[v()) = 0 (0(1 - +02
B
(0 - 02) 0(1 - 0
B
0(1 - 0) B
0(1 - 0)
0(1 0) (1 |
B
Plugging into Equation 12 gives
2B
2B
8(1 8) 1
Efflip, (x)] B B[v(z) -1-x
B
2B0(1 - 0)
20 - 0)
B
B -
28(1 ( - 1)
B-1
28 - 0) = ftip,(ax))
by Equation 11. Therefore, the U-statistic is unbiased for all B 2.
Why a quadratic surrogate for "lack of margin" is biased. As we discuss in Appendix E.2.4, interpreting empirical estimates of the flip rate can be bit counter-intuitie that are very close to 0.5 may actually reflect vote split that seems bit far from [B/2| B/2]. In other words, concrete splits for given B might "feel" somewhat far from a 50/50 50 split (lip,,. R 0.5. As result, might seem natural to derive a metric that captures coin-flip behavior by showing how far the vote fraction (Equation 14) is from completely split vote, rather than estimating the flip rate.
Empirical estimates
That is, consider that the raw margin from completely split vote is v(x) 1. (Note that, if v((x)= = 0.5, then the raw margin is if v(x)) = 1, then the raw margin is 0.5; u((:)) = 0, then the raw margin is -0.5; and similarly, for any intermediate vote fraction.) Scaling so the range becomes [0, 1] and taking absolute value so that there are no negative values gives
m(x) = 2u(x)) - 1| [0.
Therefore, (x)x)== 0 at a perfect split and m=(x) = 1 at unanimity. But of course, m((x) is neither smooth nor concave. We show two convenient identities (by completing the square) that relate the margin and the quadratic in v(z), so that
have smooth, concave alternative:
v(x) (1 v(z)) 1 (v(#) 1)²² = - (2v((x) 1)² 1 (1-mil (1=)))))) (16)
2 2 v()) = 2(v(x)) - 1)²² -
So 20( (1 v(x) is smooth, concave, symmetric surrogate for "lack of margin" (maximal at v(x) = decreasing as the margin grows).
While this alternative seems to behave "nicely" in practice (i.e., is most } unlike flip,, B), it is biased (downward) for finite B. That is,
(1 - vx))] 2 [[[((=))] E[((=)]])
= 2 [[[(()] - (Varfe(=)] + E[(())1))
(variance identity)
= 2 0 (Var[u(x)) + 09)
(by Equation 13)
= 2 0 (00-00 + 02)
(by Equation 14)
= = 0 0² 9(1-0)
B
20 - 0)
33

---

The population flip rate is 20(1 8) (Equation 11),
E [2v(z) r<x)) flip,,
(1-B)
i.e., the quadratic surrogate metric for showing a "lack of margin" (i.e., coin-flip-like behavior of MIA decisions for æ) is downward biased by B (i.e., is I below the population flip rate) for any finite B 2 2, and becomes unbiased only as B - cc. By contrast, the U-statistic (ip,,,((r) (Equation 1) exactly unbiased at every B 2. This why we report the U-statistic, i.e., the pairwise decision disagreement probability (which we informally call the flip rate).
E.2.2 Measuring flip rate for MIA
In Cooper et [14], the authors train B models using bootstrap replicates drawn from dataset D. They split D into train and test sets, train B models on bootstrap subsamples of the train set, and, for each held-out test sample, compute an unbiased estimate of self-consistency from the B binary decisions.
Here, measure flip rate in setup that mirrors strong MIA. We fix dataset D of size 2N (so 2=21 and train each target model on the same N-sized subser-i.e., the set of members (size N) and non-members (size N) is identical across targets. When training targets, we change only the random seed that determines batch order during training. Changing the batch order induces randomness the training process. Together with unavoidable hardware non-determinism, this yields the variability we observe across target models [11].
For LiRA, we fix a reference of 128 independently trained models on different N-sized subsamples, and we use these same references for every target to compute per-sample IN and OUT reference distri- butions, PIN(- x:) and Pour(- æ²) At chosen FPR v) E [0, 1], each target model calibrates its own threshold Tr (77) on that target's s non-member scores (i.e., we perform per-seed calibration), and then applies the calibrated decision rule Equation We then compute the flip rate for sample over the ensemble {r:}er via Equation 9, thereby isolating the effect of target-training randomness while hold- ing references fixed. We such experiments two model sizes: 1 140M and 302M (Appendix E.2.5).
For reference, we highlight some key points about calibration that will come up repeatedly in the rest of this appendix.
Calibration asymmetry and its consequences
What we calibrate. For each target and fixed FPR the decision threshold is Tr- = inf(t: (t) 2 1- 17 }. i.e., the empirical (1-n)-quantile of that seed's s non-member scores. This guarantees the non-member tail is controlled level v) for that seed (with the usual tie convention; Appendix A).
Why asymmetry arises. Because TT)( is re-estimated on non-members for each seed, it "tracks" seed-to-seed shifts in non-member score distributions by construction. Members, however, are not used for calibration, so many member scores lie closer to (and straddle) the moving boundary across seeds (Figures 25, 26, & 27).
Empirical effect. In regions where IN/OUT scores overlap (Figures 22 & 23), small seed-induced shifts in either the score the boundary can flip member decisions; conse- quently, members exhibit substantially higher flip rate than non-members at the same 7), and the gap widens at larger 7) (up a point) and with increased model size (Figures 28 29).
Implication. This calibration asym asymmetry explains why aggregate metrics (c.g., mean TPR at fixed FPR. see Tables I 2) can look stable (Figures 24 30), while many member decisions are individually unstable (Tables 3, 4, 5, & 6). It also motivates our hypothesis-test cutoff f., ( for flagging statistically coin-flip aan-fiip-like per-sample decisions (Appendix E.2.4).
Interpreting flip rate for MIA. Each target model a plausible outcome of this training process. Any of them would be reasonable choice for running LiRA, they are i.i.d. draws from the same seed-induced distribution. Measuring flip rate across targets therefore quantifies how resilient LiRA's per-sample decision is randomness in target training.
sample's binary membership decisions are stable (low flip), LiRA's decision for that sample is robust to target-training randomness and more likely to reflect persistent signal, rather than
34

---

seed-specific idiosyncrasies. Conversely, if binary decisions are unstable (flip near its population maximum 0.5), the sam-se decision effectively arbitrary with respect to seed choice- even when aggregate performance metrics (e.g., TPR at fixed FPR, or AUC>0.5) look stable and reasonably high-performance. In this case, per-sample membership decisions are so influenced by randomness in the training process that we cannot draw reliable conclusion about membership. Put differently, measuring per-sample instability lets us peer beneath high-level, average metrics-e.g. for fixed F FPR, mean TPR over all members across plausible targets po assess what strong MIAs can (and cannot) say reliably about individual samples.
E.2.3 Connections to prior work on model and predictive multiplicity
This analysis connects to broader literature in statistics and machine learning outside membership inference. Notably, Leo Breiman's 's seminal work on the Rashomon effect emphasized that, for a given dataset, there often exists a multiplicity of distinct decision rules with essentially the same overall accuracy [3]. The Rashomon set-the set of models within a small tolerance of the optimal risk-can be surprisingly large [20, 51J. More recent work on predictive multiplicity also shows that training processes can produce models with effectively indistinguishable overall test accuracy that nonetheless disagree widely at the per-sample level [14, 36, 59].
To the best of our knowledge, this connection has not been made in the MIA setting. Our setup differs in that we fix Ii and vary only algorithmic randomness (via seed controlling batch order for target replicas); we then observe targets with similar overall accuracy but substantial per-sample churn, quantified by flip rate (Appendix E.2.5). (We make no claims about the optimality of the resulting MIA rules.) The key result of these experiments is that average attack performance can remain stable, while individual membership decisions vary across seeds - phenomenon that bears directly on the reliability and validity of membership claims about specific samples (as the problem is set up in the membership inference security game).
E.2.4 Reasoning about the minimum empirical flip rate that reflects coin-flip MIA decisions
As noted in Appendix 1.2.1, the population flip rate flip,, (a:) [0, 0.5) (Equation 8): 0 reflects MIA decisions that are completely stable for x (i.e., do not flip) and 0.5 reflects coin-flip decisions estimate the population flip rate with the U-statistic for flip rate using
for xc In practice, concrete number of target replicas B, namely flip,, (z) (Equation 9). This empirical estimate also has minimum of 0, reflecting completely stable binary decisions, but its maximum (reflecting maximal disagreement) slightly exceeds 0.5 and converges to 0.5 as B - DC. This raises an important question: for concrete B in practice, which measurements of flip,, (xx) reflect that the MIA decisions for behave like coin flip? That is, we need to determine a reasonable cutoff for fiip,,B (z), indicating that the decisions for x are statistically indistinguishable from a coin flip.
A principled way to determine this cutoff is to set up a hypothesis test at level o: we call the MIA decision for a sample æ "indistinguishable from a coin flip at level a" if two-sided exact binomial test fails to reject. We do this for our experiments in Section 5 and Appendix E.2.5. For the experiment with the 140M model (B=125), we call the MIA decision for æ indistinguishable from a coin flip at a=0.05 if the MIA decisions for x exhibit call the MIA decision for æ indistinguishable from a coin flip at a=0.05 if the MIA decisions for x exhibit flip,,, (x) Z 0.487 (Figure 5a). In the end, all this requires is finding the minimal number of member votes k at which the CDF F(k) of the binomial Binomial( B, 0.5) Z g. i.e.
2 0.490; for the 302M model (B=127), we
k = min k:: F(k) 2 a/2),
(17)
and computing the coin-flip cutoff as 2 flip,, with (x) = kL and Bu(x) B KL.
In this appendix, for the reader interested in a refresher, we walk through how we set up this exact test. We describe the hypothesis test at level a. how this results in an acceptance region (in terms of the number of member votes), and how convert that region into minimum empirical flip,, p (xx) that we can defensibly interpret as behaving like a coin flip. (This depends on the the vote fraction v(x) discussion from above.)
Setting up a hypothesis test. We call the MIA decision for a sample x indistinguishable from a coin flip if the probability of predicting "member" equals the probability of predicting "non-member".
35

---

Let Pi......T TB lid. throughout, let
j denote B target replicas (varying only by seed) from the training pipeline. As
B
(x = Σ
m
(t).
1=1
and note that, for fixed (æ, n), the indicators by. (ar) are i..dd Bernuulli with success probability
Pr b =1. rap
Therefore, (x)
Binomial(B, (B,0) 0). Coin-flip behavior corresponds 0 0.5.
We set up the null hypothesis
Ho 0 = 0.5 (two-sided exact binomial test at level a).
(18)
If we fail to reject Ho, then do not have sufficient evidence to say that the MIA decision for is not a coin flip, and so we deem the decision indistinguishable from a coin flip. The significance level means that, if Ho is true (i.e., the decision is indistinguishable from coin flip), the probability that we incorrectly reject Hn (i.e., say that the decision is not a coin flip) is at most a. Smaller a imposes stricter standard for rejecting Ho (stronger evidence is required). We will later show that, for B replicas, "fail to reject" is equivalent to
flp (x) 2 to (B),
with la (B) computed from the binomial acceptance region under Ho (Equation 23).
Deriving the two-sided exact p-value at level a. For B replicas and operating point 7), each replica r outputs a binary membership decision 61") (x) E {0, 1}. Going forward, we denote the member-vote count
K
(1
Σ (x))
Under the null hypothesis Ho in Equation 18, each target replica's 's MIA decision behaves like fair coin, so
K (B,
Intuitively, the further K is from the center B/2 (i.e., split vote, indicating coin-flip behavior), the stronger the evidence against Ho.
More formally, let the binomial PMF and CDF under Ho be, respectively,
Pr(K i)
B (8)
{
2- B
F(k) Pr(K < k)
B Σ 1=0
2-
(19)
Because (8) = (
Pr(K =) = Pr PK(K- B -) for all
and so the distribution is symmetric about B/2.
We can reason about the tails of this distribution in terms of concrete vote k and the CDF, i.e.,
F(k) = Pr(K k) =
Pr(K = 1) Σ 1=0
Pr (K K = B -) = Σ Pr(K =)
1=0
j=B-k
= Pr (K > B - k).
(20)
Thus the left tail equals the right tail B k. This follows from change variable, setting
j = B - (when i = 0, = B and when = k, j = B k. so the index runs in reverse and
B i) Σソ Pr(K =)).
36

---

And so,
Pr(K 2 k) <k Pr (K B- (k-
-
(by Equation 20 with it
1)
= = PK B B Rt).
(21)
Intuitively, a two-sided p-value measures how surprising the actual observed count K= is under the null hypothesis (Equation 18): it sums the probabilities of outcomes at least far from the center B/2 as k, in both tails. Because the binomial for 0.5 is symmetric and unimodal about B /2, "equally more extreme" corresponds to the union of the left tail up to k and the symmetric right tail from B-k upward (or the mirror statement when on the right). So, to derive a p- p-value at k, there are two cases to consider.
Case 1: k S [B/2]
Here, the left tail is K & k and the right tail is 2 k. Therefore,
p-value(k) Pr(K $ k)) + Pr(K > -- k)
= 2 Pr(K E k)
(by tail symmetry, Equation 20)
= = F(k)
(by definition of the CDF, Equation 19).
Case 2: 1 [B/2] Here, the right tail is K 2 k and the left tail is K s B k. Therefore,
p-value(k) = Pr(K 2 k) + Pr(K S B k)
= 2 Pr(k 2 k)
(by tail symmetry, Equation 21)
= (1-(1 (1-F
1)). 1))-
From this, derive the standard form of the two-sided exact p-value. That is, because the binomial is symmetric and unimodal about B/2, this can be written as
p-value (k) 2 min Pr(K k), Pr(K 2 k))
which means we double the smaller tail in order to capture both-sided extremeness. Alternatively, using Pr(K 2 k) Pr(K S k 1)
F(k- 1) (Equation 21), this becomes
2 F(:).
k S [B/2],
p-value(k)
2 (1 F(k - ()),
k 2 [B/2]-
Finally, to handle discreteness at the exact center (even B and =B/2, which counts the mass at twice), we cap p-values at 1:
p-value(k) = min 1, 2 min (F((), 1 - F(k F( 1))
(22)
These equal-tail formulas handle discreteness conservatively: the acceptance region is defined so that p-)alue(k) 2 a inside the region and p-value(k) < a outside, with in each tail. Because the binomial is discrete, the equal-tail construction is slightly conservative. We follow the convention "reject p < a" and fail reject p 2 a, so boundary points with p-value = a remain inside the acceptance region.
Using Equation 22, the acceptance region for member votes ie at level a is constructed by finding ky = min(k E (O..... B/22] : F(k) 2 a/2),
and then setting
A. = [kL, k B B-kL}.
so that K E As < p-value(K)) 2 a (fail to reject). By symmetry, the upper endpoint B KL. and so the acceptance region is symmetric band around B/2 (Equation 20).
Equivalently, the critical (rejection) region is
{K & ky - }} U ( B kL + +1};
so K E Aa = p-value(K) K) 2 (fail to reject), and K & Ao + p-value(K) < a (reject).
And so, for fixed B and given k, we can check if F(k) 2 a/2 simply by computing F(k) = Ei_ 27)2-1 2 a/2, as in Equation 19. For a=0.05 and B-127, kz=52 (and B k1,=75) with (52)=00.02524; for a=0.05 and B 125, is also the case that ky =52 with F(52)==0.036) 0.6661 (but B - k,=73).
37

---

From the acceptance band to a concrete flip cutoff. (x. 17) as a function of the member-vote count K is
For fixed B and FPR the empirical flip at
2 K (B K)
(K(
B(B
where flip,, (2)) = (k( (to preserve notation/ defining flip,, on a and continue using K, as in the rest of this section). Since this is just rewrite of the empirical flip at K,
Φn(K) = Φ B K)
(symmetry about B/2).
This function is symmetric in K about B/2 and unimodal. A one-step discrete difference shows it is strictly increasing on the left half:
(KK) ΦB K (K)) ΦB(K) 2 2K 1]
B 3-1
for
B(B -1)
2
Moreover, (K) 0 at K R-1 and A(K) < for > and then decreases (unimodal).
so Φn increases up to the center
2
Therefore, on the symmetric acceptance band
A. {K: 1y S K < B k1, the minimum flip occurs the endpoints K hy K B kL. and both give the value by symmetry. And so, the empirical flip cutoff at level is
2 kL. (B k..
ta (B)
(23)
B - 1)
Equivalently, similar to Equation
12, in vote fraction form with vt. kr./B,
2B
ta (B) = B-1 eL (1 uL)
We declare the MIA decision for sample a indistinguishable from a coin flip at level a
(l) 2 ta (B).
Because the binomial is discrete, equal-tail tests are slightly conservative. We follow the convention "reject if < a" and "fail to reject if p 2 a," so boundary points with p-value = lie inside the acceptance region. This yields the monotone flip rule flip,,,(x)) (B).
For the experiments in Section 5 and Appendix E.2.5, we set a=0.55. For the 302M model we have B-127 target replicas and for the 140M model we have B-125 target replicas, respectively:
B=127: =5, (so the acceptance band K [52, 75)) and
2 52 75
to.05 (127)
20 0.487.
127 126
B=115: =52 (acceptance band K E [52, 73) and
2- 73
lo.05 (125) = 125 124
A 0.490.
(For B=125, our threshold more conservative part because the discrete CDF lands at kz but a normal correette approximation puts it closer to 51.6.)
E.2.5 Extended results on flip rate
We provide results for two model architectures: 140M and 302M. We use the same training dataset size for both: overall 2 so models are trained on 2 19244_ 55052222500 samples each. Note that for both architectures. this training dataset size is significantly smaller than what is Chinchilla optimal (=7M for the 140M model and x15.1 for the 302M model). As result, we expect attack success to be higher (as measured by ROC-AUC) compared to Chinchilla-optimal trained and attacked models (Sections 3.2 & 4.2). For each model, train one set of 128 reference models (with
38

---

Clear, stable member true label: member
Clear, stable non-member true label: non-memter
17.5
(eferancce)
(refenencee)
15.0
(referonces)
OUT (neceenncss)
Weference dersity
1.2
larget median
Target IOR
Target
20.0
0.5
7.5-
0.5
FPR=1e-02 02
FMM
0.4 fip 0.000
2.5
0.2-
6.0
0.0
2.6
4,0
Membership inforence statistic
(perstokon
Membership inference statistic (per-token loss)
(a) Clear, stable member
(b) Clear, stable non-member
Ambiguous, stable label: non-member
Ambiguous, unstable true label: member
20-
(refereeees)
Indfrenncss)
(UT(rsforeecs))
Trrget median
Thrget medann
20
Taget IQR
Refireene tieraity
15-
Target IQR
Reference e
#fft1e-02
#0R=1=-00 - 0.495
0.000
3.0 4,0
4.4
4.6
3/6 3.8
4.2
Membership inference statistic token loss)
Membership inference statistic (per-token loss)
(c) Ambiguous, stable sample
(d) Ambiguous, unstable sample
Figure 22: Different sample "archetypes" for the 140M target models. We plot the per-sample
a's s reference distributions (IN and OUT), median target statistic (and IQR) for æ across the 125 targets at FPR=10-2 for four different a:: (a) clear, stable member; (b) clear, stable non-member; (c) ambiguous, stable sample; and, (d) ambiguous, unstable sample. We annotate each plot with the sample's ss true label and empirical flip rate. For this architecture, we also provide snippets for the text of each sample in the main text.
0.5 probability that each sample is included as a member, so that member and non-member classes are balanced). To measure flip rate, we then train many target models on the exact same training dataset (i.c., the member and non-member samples are the same for all targets). The only difference across models is the random seed, which controls the batch order in which samples are surfaced to the training algorithm.
We intended to train 128 target replicas for each architecture; however, some runs crashed (and we ran out of time to re-run them), so in all we have 125 targets for the 140M model and 127 for the 302M model. As noted in Appendix E.2.4, the minimum values that we consider indistinguishable from coin flip for flip,, are o0.05 (125)=00.49) 490 for the 140M model and to.os f1.05(127) (127)==0.48) 0.77 for the 302M model.
An intuition for per-sample flip rate. Flip rate captures sample æ's membership inference instability, computed across a set of target models where the only difference is the random seed that controls batch order. For given æ;, it captures how much cross-decision disagreement there is how much the MIA decisions for æ flip between both classes for equally plausible targets r-
To give sense of how this can happen, we provide plots at the sample-level that show where target membership observation statistics for a given æ fall in relation to w's IN and OUT reference distributions, PIN (-1zz) and Pour(-lz))-ff fitted from the statistics obtained for using the reference sets PIN and POUT. respectively. We plot four "archetypes" that capture different patterns in sample- specific MIA decision behavior, in relation to reference distributions: (a) clear, stable member; (b) clear, stable non member, (c) ambiguous, stable sample; and, (d) ambiguous, unstable sample. We identify these archetypes FPR-10-2 In Figure 22, plot all four archetypes for the 140M architecture. In Figure 23, we plot archetypes (b)-(d), as we are unable to find clear, stable members at 10- Even for the 140M model, we have to relax the flip rate in our search filter to allow for j 12: S0.2 to identify a "stable" member (when arguably, such a flip rate is not particularly stable). We are unable to satisfy this relaxed filter for the 302M architecture.
39

---

Clear, stable non-member label: non-member
OUT Ineferances)
larget mediin
1.0
Tar Tarpst IQW.
0.8-
Reference
0.6-
0.4
Eip 0.000
FM-1-=07
0.2
0.0
Membership inference statistic (per-token
(a) Clear, stable non-memher
Ambiguous, stable label: non-member
Ambiguous, unstable label: member
14.
Indfrennces)
20
OUT
Trrget median
Thrget medann
brgetiqe
Target IQR
to-
Refirrence
#0R-1=-00 - 0.508
0.000
3.6 4.0
4.4
4.6
3/4 3/6
4.0
4.2
Membership inference statistic tolen loss)
Membership inference statistic token loss)
(b) Ambiguous, stable sample
(c) Ambiguous, unstable sample
Figure 23: Different sample "archetypes" for the 302M target models. We plot the per-sample æ: 's reference distributions (IN and OUT), median target statistic (and IQR) æ across the 127 targets at FPR=10-2 for four different a:: (a) clear, stable non-member; (b) ambiguous, stable sample; and, (c) ambiguous, unstable sample. We annotate each plot with the sample's true label and empirical flip rate. We are unable to identify a clear, moderately stable (flip10- 2,12002-122 2,127 I 0.2) member sample.
Note that, for both model sizes, the IN and OUT reference distributions overlap considerably for member samples. This overlap reasonable explanation for MIA decision instability: if LiRA has difficulty between establishing signal between members and non-members, then this will understandably impact the reliability of MIA decisions. Across targets trained on different random seeds, this can also manifest as the MIA decision flipping from one class to the other. In contrast, we identify cases for non member samples where there is clear separation of IN and OUT reference distributions (Figures 22b & 23a).
For the 140M archetypes, we include short snippets of the text for each sample:
140M: clear, stable member: "Whether it's your first time looking for a Personal Trainer and you are just starting out, you veteran who has been around long-time, SINA Fitness can help you reach your fitness goals. Our Trainers are experienced, friendly and very energetic. We will help you set your fitness and lifestyle goals and most importantly help you achieve them....
140M: clear, stable non-member: "Ä Release Notes: AI War an entirely unique large-scale scale RTS with aspects of TBS, tower defense, and grand strategy. It features single or cooperative play with many as humans against a pair powerful, intelligent Als. These AIs are driven by an Progress stat that players contribute to through aggressive actions such taking control of planets and destroying key units, forcing tough decisions regarding which targets are worth capturing or destroying. ..."
140M: ambiguous, stable sample. "The Gingrich commentary came hours after The Wall Street Journal reported that Mueller empaneled grand jury.
"The Mueller threat has probably been the most deadly, has the power of the law, he has the ability to indict people, the ability negotiate and let people off if they'll Il testify against other people," said Gingrich, also a Fox News contributor... ..."
140M: ambiguous, unsiable sample. "Winner of the Junior Australian Open 2015 Tereza Mihalikova (20), who going to participate EMPIRE Women's 's Indoor 2019 tournament, had spent the entire 2018 scason under the guidance of tennis coach Martin Hromec. At the end of the year, the well-
40

---

10a
10P
10
10-1
1."2
10
The Pezitbve
10-3
Pozitbee
30-3
10-4
30.-
10-1
Random (AUC-0.5)
NOC over targuts
10-4
NOC targets
(AUC-0.51
across sceds
across seeds
10
10
10.
10-11
10-3
10-2
10:1
10
10
10.4
10.2
10-4
10-2
10-2
10-1
104
False Pesitive Rate
False Positive Rate
(a) 140M
(b) 302M
Figure 24: Averaged ROC curves and AUC across targets. We plot the mean ROC across targets (B=125 for the 140M architecture; B=127 for the 302M architecture), and +1 STD across seeds. Both models are trained on substantially fewer samples than is Chinchilla optimal (==500,, compared to 77M and R. 1M, respectively). Mean ROC-AUC is higher than for Chinchilla-optimll models (as in Section 4.2). For the 140M model (left), (right), --A--OC----UTPOA 752 + 0.007. These are not typical attack ROC curves, as they average over results for multiple targets. In the standard MIA threat model, the attacker only has access single target. These plots give a sense of the stability of overall attack performance, as computed over equally plausible targets r~~ p where the only difference in targets is the seed controlling batch order.
= 0.004; for the 302M model
known fitness coach Jozef Ivanko, strengthened the team. Ivanko worked with Top 10 players in WTA ranking already.
Aggregate attack performance for MIA flip-rate experiments. While our main focus here is to measure per-sample instability, as a point of comparison, we also include measurements about attack averages. For both model sizes, we include mean (cross-seed) ROC-AUC metrics and curves (Figure 24) and associated tables that show average (cross-seed) accuracy and error rates by class at fixed FPR (Tables 1 & 2).
We again emphasize that the models we attack in these experiments were not trained on the Chinchilla- optimal number of tokens (=7M and =15.1M samples, for 140M and 302M models, respectively; see Section 3.2 & Appendix C). Both sets of experiments involved training models on only =500K samples. As result, we expect (and do observe) attack performance (in terms of ROC-AUC) to be higher than in the Chinchilla-optimal setting (Section 4.2 & Appendix D). For the 140M architecture, we observe average AUC=0.706 + 0.004 across the 125 targets (Figure 24a); for the 302M architecture, we observe average AUC=0.755 7.2 + 0.007 across the 127 targets (Figure 24b). In both cases, AUC is stable across targets (as indicated by the low standard deviation). This same pattern stability in overall attack metrics is also clear in Tables and 2: accuracy and error exhibit low standard deviation, with respect to these rates being aggregated across all samples (conditioned by class) and averaged across targets trained with different seeds.
For an alternate view of these results, we also include direct comparisons of attack performance (as measured by average TPR standard deviation at fixed FPR) and variability the underlying decision rule (with respect to threshold 7) across targets. In Figure 25, we provide these comparisons for both the 140M and 302M model sizes. Of course, as is also surfaced by ROC curves (Figure 24) at very low fixed FPR, the TPR is also low. Here, we also show how this naturally results in a very high decision threshold v, which also exhibits low variability. As we increase FPR, TPR also increases and remains stable, with respect to low standard deviation. However, the underlying decision rules for the targets can vary considerably; the underlying targets can have very different т. This result is consistent with prior work model and predictive multiplicity (Appendix E.2.3): models with similar overall accuracy can have very different underlying decision rules. As we address further below in this appendix and in Section 5, even when overall accuracy is similar, the different decision rules can result in very different/dissgrceenng membership MIA decisions for the same samples.
41

---

Table 1: 140M-parameles model error rate metrics. We report accuracy-related metrics function of fixed FPR. Entries are rates (not percentages), as elsewhere this paper. We report mean + STD where applicable. Since fix FPR, there is no STD to report. Since 1 - FPR TNR. similarly, there no STD to report. ACC = TP4TN N with 2N=1,048,576. Typically reported
log-scale FPR rows are highlighted in gray.
ACC All
FNR. Members
FPR Non-memhers
TNR Non-memhers
TPR Members
FPR
10
0.501
0.998 0.001
0.0
0.002 0.001
10
0.504
+ 0.001
0.992 + 0.001
0.0
1.0
0.008 + 0.001
10
0.515 0.001
0.97 0.002
0.001
0.999
0.03 + 0.002
10-
0.5/7 + 0.002
0.896 + 0.005
0.01
0.99
0.104 + 0.005
0.02
0,564 0.003
0.852 = 0.005
0.02
0.98
0.148 + 0.005
0.05
0.593 0.003
0.764 0.006
0.05
0.95
0.236 + 0.006
10-1
0.618 0.003
0.664 = 0.006
0.1
0.9
0.336 + 0.006
0.2
0.64 0.003
0.52 + 0.006
0.2
0.8
+.000
0.5
0.633 0.002
0.234
0.004
0.5
0.5
0.766 0.004
0.75
0.582 + 0.001
0.085
0.002
0.75
0.25
0.915 +
10"
0.0
0.0 1.0 0.0
1.0
0.0
1.0
Table 2: 302M-paramen model error rate metrics. We report accuracy-related metrics function of fixed FPR. Entries are rates (not percentages), as elsewhere this paper. We report mean + STD where applicable. Since fix FPR, there is no STD to report. Since 1 FPR TNR., similarly, there no STD to report. ACC TP+TN with 2N 2,04,,08,576 Typically reported log-scale FPR rows are highlighted in gray.
ACC All
FNR Members
FPR Non-members
TNR Non-members
TPR Members
FPR
10
0.505
0.001
0.991
0.003
0.0
1.0 0.009 0.003
10-4
0.514
0.003
0.973 1 0.005
0.0
1.0
0.027 1 0.005
10-3
0.536
0.927 0.005
0.009
0.001
0.999
0.073 1 0.009
10-2
1.000
0.819 1 0.013
0.01
0.99
0.181 1: 0.013
0.02
0.608
0.765 = 0.014
0.02
0.98
0.235 + 0.014
0.05
4.642+
0.667
0.014
0.05
0.95
+.333+0000
0.668 0.007
0.565
0.014
0.1
0.9
0.435 0.014
0.2
0.685 1 0.006
0.43 0.013
0.2
0.8
0.57 0.013
0.5
0.655
1 0.004
0.191
1 0.008
0.5
0.5
0.809 4.000
0.75
0.588 1 0.002
0.075
0.004
0.75
0.25 0.925
10"
0.0
0.0
1.0
0.0
1.0
42

---

TPR at fixed FPR
Threshold T variation
TPR (mean ctd) screas 125 socds fxod FPR
0.55
6.3
Threshold T variation across seeds at fixed FPR
0.50
[0.93 1.000
Calbbratd
0.9
12h 0.022
1QN 0.007
80.75, 5931 0.027
10R 0.203
0.25
140M
0.0
0.7
12R 5.057
3.53
8.00 3.30
10-1
30-1
10
10
10
Target FPR
TPR [mean a stdi across 125 seeds vs. fixed FPR
0 6.12
Threshote variation across seeds at fixed FPR
0.4
Thus Poolttve 1et0
0.3
0.9
10.942:
10A
0.00
108 0.007
Calibrated
0.8
1Q 0.024
12
338 g 3.2
0.207
302M
20. 945)
5,067
0.51 2000
0.5
Palse Positiive Bate
10"h
10
10-1
10-1
10
Target FPR
10
10
10"4
Figure 25: Comparing attack performance and decision thresholds at fixed FPR. Row shows results model architectures: 140M and 302M The left column shows mean + standard deviation of the attack TPR at different fixed FPR, computed across B targets. The right column shows the decision threshold 7 range (and IQR) at fixed FPR across targets, where the decision threshold for each target is calibrated with respect also surfaced by ROC curves (Figure 24) at very low fixed FPR. the TPR is also low. Here, we also show how this naturally results in very high decision threshold which consequently exhibits low variability. As we increase FPR, TPR also increases and remains stable, with respect to low standard deviation. However, the underlying decision rules for the targets start vary considerably; the underlying targets can have very different т, which (as we address in this appendix and Section 5) result in very differeen/disagreeing membership decisions for the same sample
non-member samples (Section 2 & Appendix A). As is
43

---

Sample flip rate variation at a single fixed FPR. We provide three complementary views (by class) at fixed FPR to characterize per-sample instability: (left) complementary CDFs (CCDFs) of flip rate; (middle) flip rate vs. mean absolute distance to the calibrated decision boundary; and (right) flip rate vs. mean LiRA posterior. We show these results for the 140M and 302M models in Figures 26 and 27, respectively, each for FPR E (10-5, 10-1, 10-3,10-3 , 10-1).
For the middle and right columns, the r-axis uses equal-count (quantile) bins so that each plotted point aggregates (essentially) the same number of samples; points are therefore directly comparable across the curves. Together with Tables and 4, these plots support our main takeaway in Section aggregate attack metrics (e.g., mean TPR at fixed FPR. ROC-AUC) can look stable while many individual membership decisions are indistinguishable from a coin flip.
We organize observations by theme:
Flip rate rises with FPR. As is clear from the complementary CDFs for flip rate (left column), flip rate rises with FPR. This is because, as FPR grows, the per-seed calibrated threshold Tr T--(1) moves down (right-tail quantile of the non-member distribution), into regions where IN/OUT distribution overlap is more extensive. This boundary shift puts (y( in score regions where many member sample posteriors lie, and also increases the proportion of samples whose seed-specific scores lie near the boundary. As result, small wed-inddced score shifts (as well as seross-seed variation in T, (17 itself, see Figure 25) flip the decision more often (i.e., increase per-sample MIA decision disagreement). (More non-members will also be labeled as members, by construction; so, too will members.)
The effect is modest at very low FPR, where Tr (T)) sits deep in the extreme tail. But it becomes more pronounced as we increase FPR (i.e., as the boundary moves toward denser parts of the score distribution). The CCDFs (left column) and distance plots (middle column) both show
this pattern: at FPR=10-1, 700% of members for the 302M targets have flip 1.1
vs. 77% of mem-mesbess: for the 140M targets the corresponding figures for 11210- are 49% vs. 88% (see Table 4 and Figure 27, left; Table 3 and Figure 26, left).
Mean absolute distance to the boundary is a direct proxy for instability. For each sample æ, we define per-seed distance to the decision boundary and a cross-seed measure of closeness to the boundary, regardless of side:
(x) = A, (r):: = Σ. (x))..
For associated plots (middle column), quantile bins with small [d] put the sample close to the decision boundary, resulting in high flip rate (i.e., many MIA decisions disagree). Quantile bins with larger [d]] are more reliably on one side of the decision boundary, which results in a lower flip rate (i.e., more decisions concentrate). At FPR-10 10-1 for both model sizes, member flip rate is persistently high across a wide range of [dl-evidence that IN/OUT overlap is substantial in the region where (1) lies for many seeds (middle column). In general, members exhibit markedly higher flip rate than non-members at the same [d], with the differences in the two becoming wider at higher FPR.
Flip vs. mean posterior is non-monotone at high FPR. For the plots in the right column, we define the mean posterior across targets as
A(x) = - A,
Flip rate increases as A(x) approaches (T) (more seeds straddle the boundary) and then declines once A(æ) is well above the boundary for most seeds (decisions re-concentrate on "member"). This non-monotonicity is most visible at FPR-10-1 (right column). (We similarly see this in the middle column, which directly plots distances to the boundary.)
Members flip much more than non-members, and the gap widens with model size. Two forces seem to drive this. First, there structural asymmetry across classes from calibration (See box, Appendix E.2.2). Thresholds are calibrated on non-members for each seed-specine target (Section Appendix A), so (1) tracks seed-to-seed shifts in the non-member distribution by design, and many non-members remain far below T, 7-()) for modest FPR. In contrast, the threshold is not anchored to the member score distribution. Often, their scores straddle the moving decision boundary, so, small seed-induced shifts (cither in the score or the decision
44

---

boundary, see Figure 25) can flip the decision. This effect is more pronounced for higher settings of FPR. which push the threshold into a higher density region of the member score distribution. Second, there is greater across-seed score variability for members. Intuitively, training randomness primarily perturbs samples seen in training (as opposed to those that are not). Empirically, IN/OUT reference distributions for many members overlap substantially, while some non-members exhibit clearer separation (Figures 22 & 23). Both of these effects are stronger for the larger model 302M (compare Table 4 vs. Table 3).
Effect of model size. In general, the observations above show that flip rate instability is worse for the larger (302M) model. Members flip much more than non-members, and the gap widens with model size. These results are also consistent with www.impliiitttlellleeeo. higher capacity models: those with similar aggregate accuracy can exhibit more disagreement at the individual sample level [14].
results for
Flip rate over varied fixed FPR. We summarize across operating points in Figures 28 and 29. Each contains five sub-plot that show, for given flip rate range, the class-conditional proportion of samples in each range as a function of FPR (with the corresponding mean TPR STD annotated above the panels). We use the disjoint ranges [0, 0.1) (very stable), [0.1, 0.25) (low/mid stable), (max), [0.25, 0.4) (mid/high unstable), [0.4, (B)) (very unstable), [La (B), f J. where for B-125 (max) we take to.05 (125)=0.440 and 12 =0.504, and for B-127 we take to.05 (127)=2200 and (max) flip,,127 5030394
Our values for (125) and (127) are obtained from the exact two-sided binomial accep- lance region (Appendix E.2.4). That is, we compute (B) 2 (B (B-kr) with kg chosen as the (max)
B(B-1)
smallest integer such that F(k..) 2 o:/2 for K Binomial flip,, 2 B/2 [B/2] (Appendix 1.2.1).
B(B-1]
These summary curves reinforce the fixed-FPR views above: flip rate increases with FPR, members flip far more than non-members at all reasonable FPR (i.e., FPR,0..2), and the 302M model shows larger gaps between members and non-members as well as mass in the statistically indistinguishable- from-a-coin- fflip range.
45

---

Flip CCDF
Flip vs. mean distance
Flip vs. mean posterior
H1p new complementrr CDT (OCBF) FPR-10 10
Fip rase disiance to thrsstour AMM-100
Pip va mean posterir
FPR-10
13054
with with bm
Morbor
wit
Pieriors
(CCOT)
Ree-rrmmaes
Navmrenenn
2.035
Nanmmebeers
10-6
Wean fp prat
2010
2.00
100
0.005
0%6
30
03
0.4
800
8. 1. nát cis io ois nin ass
0.35 5.40 2.45 6.50 815 0.00 080 0.22
M1p mere
Mean over seess of possarie- (per sampls)
Henn [RA posteror armoss seeds (Der exmple
aato complamentary CDF (OCDF), FP%
Piprate vs. distance threshold ##R=10-1
TIp rate vs. mewn posteror or TPR-11- 10
322%4
Menken
Moan witn bis
Harbers
Verrbars
30%
10
4
50%
601
Plan
8.00
6.00
02
3.2
44
6.30 6.9. 8:30
260 0.65 [pper sample)
3.35
5.40
330 035 0.0 0.55 6.75 146
postofior actoss so0ds (por sample)
Tip nete os distence to threnned |
1PR-10
Hpp rato vs, moon poster or FPA 10m2
187%%
Morber neboo.
the rade Wettbb
-D0 20038)
10
4.70
630
43%
6.0
6:0
6.00
0.0
0.3
02
0.4
03
0:33
6:30
6.5
5.5 250 355
3.35
640
350
055 062 6.65 (per
posterar
rete complemettrry CDT (CCDP) FRR-10-
Flip rate vs. ditt noe to threshold | members
FPR-10-
FlD rate vs. mesn posterior FPR-10
180%
(CX3F)
82%
Nce-vendors
with
10-2 2
0.1
02
0.4
3.5
9.0 525 6.30 B. 6:1 cas 6.5e
025 340 5.6 153 6.5 0.00 255 6.70
Fip mace
posteror- Tppper samp
Moon URA pertarior 20 10025 (pot samolc)
Flp rate complemnttrry CDF (OCDFL FRR-10-1
Filp rote vs. disanno is threshold I
PPR-10-
Fio rale vs. mean sodteiir | IPR-10-3
Vertorr
rembore
C2- occ221
10-1
83%
Moann Fie ratte
Nat-marbon
2.17
0.5
0.26
0:30
0.55 2.40 8.53
6.20
Fip raze
Mean over Seedls of posterior lpppr Somple
Mean LIA posiarir acr04s seeds (par sample)
Figure 26: Detailed flip rate results for the 140M model. For the 140M model, the number of target replicas B=125. (rows) For different fixed FPR E {10-5, 10-1, 10-3 10-2, 10-1), we provide (columns) three different views on flip rate across member and non-member samples. (left) We plot the empirical complementary CDF (CCDF) for flip rate, conditioned on membership status. Higher curves indicate more instability. As FPR increases, the differences in flip rate across classes become more pronounced. While flip rate is minimal at low FPR, it is substantial-particaant for members- at higher FPR. For example, at FPR=10-1 approximately 50% of member samples exhibit flipro 1,125 0.4, 0.4, compared to approximately 10% of non- non-members. (middle) We plot flip rate function of the mean of the magnitude of the distance (per sample) from the posterior to the decision threshold r. Further to the left means closer to r. This shows instability in terms of the distance to the threshold (regardless of direction of that distance). For higher FPR, the flip rate for members is much higher than for non-members for the same mean absolute distance to r. (right) We plot flip rate as a function of the mean posterior. For higher FPR, the member and non-member flip rates are more similar as a function of the mean LiRA posterior. For the last two columns, we use quantile (i.e., equal-count) bucketing the æ--axis so that each plotted point is based on (essentially) the same number of samples. That way, points on the curves are directly comparable.
46

---

Flip CCDF
Flip vs. mean distance
Flip vs. mean posterior
13054
H1p new compleentary CDT (OCBF) FPR-10-
Noan o sol wiblin km
strnnce threshold =FR-100-
Rembers
bm
Hpp rate mean poster 1990-10
Verrbers
(CC2))
Nee-reeman
Men-marbers
10-6
6.00
80
0.
03
0.4
6.3
64
0.5
06
67
0.1
0.4
0.5
M1p mete
Rean over seeds portatior (por sample)
Meea (IM postetice an ceeds (per somple)
complamentary CDF (OCDF), FP
Pipraze vs. distance threshold ##R=10-1
TIp rate vs. mewn posteror or TPA-110
322%4
Menken Iemoor
withaa
Harber
Verrbars
30%
10
50%
600
Moan
8.00
6.00
01
02
4.2
04
0.1
05
0.3
3.4
0.5
0.5
[per
postotior actoss soods (por sample)
Pip rate complementary CDT (CCDP): PMR=100
Tip rite distance is thrected IPP-110
"Io rale vs. meen posterirr IPR-100
6.3
Namoera
CD1 100311
membrr
10
43%
82-
33
6.1
05
65
03
26
2.5
06
Fip rase
Mean over seads ol posterio T/lpal samp
Mean URA posteriar scross seeds (per sample)
rate complementary Car (DCDP) 1PR-100
Filp dist threstoid 1PR-10--
eale withe bm
Fio rate vs.me posterier FPR-10-
33%
Horber
with
Mamoet
ten-merbers
Partee'
momeers
10
47%
6.3
275
0.00 635 als a.4 0.5 odo as5
05
Mean over seeds of ppsteror-T/ppe samp
Mion URA pestorior bcriss sooll (por camola)
Flip to threshold FFR=10
complemettrry
1977
Pleroe'ss
CD1 (0009114
82%
Moun with
New mereters
10-1
87%
42%
Reedowt
Hiribers
hon-mamber
00
03
0.4
Mean 2ds
03
Figure 27: Detailed flip rate results for the 302M model. For the 302M model, the number of target replicas B=127. (rows) For different fixed FPR f10-5, 10-4, 10 10 10-1), we provide (columns) three different views on flip rate across member and non- member samples. (left) We plot the empirical complementary CDF (CCDF) for flip rate, conditioned on membership status. Higher curves indicate more instability. As FPR increases, the differences in flip rate across classes become more pronounced. While flip rate is minimal at low FPR, it is ietttantal-parttcuuaan members-al at higher FPR. For example, at FPR=10-1 approximately 70% of member samples
for
exhibit ffip 127 0.4, compared to approximately 10% of non-members. (middle) We plot flip rate as function of the mean of the magnitude of the distance (per sample) from the posterior to the decision threshold 7. Further the left means closer to This shows instability in terms of the distance to the threshold (regardless of direction of that distance). For higher FPR. the flip rate for members is much higher than for non-members for the same mean absolute distance to (right) We plot flip rate as a function of the mean posterior. For higher FPR, the member and non-member flip rates more similar quantile (i.e., equal-count) bucketing on the r-axis so that each plotted point is based on (essentially) the same number of samples. That way, points on the curves are directly comparable.
function of the mean LiRA posterior. For the last columns,
47

---

Fostiva
4.4
AM
AM
1.6
No of with C322
300 10 tatellgg
05
5.5
Nuc
(a) Very stable MIA decisions:
flip 125 0.1)
Tue Tartive hare
True Fosthve Face
4.5
40.00
200
20.00
Poraar
(0.)
25%
2031
Fin
0.
14
(b) Low/mid stable MIA decisions: flip,, 125 E [0.1.0.25] 0.25)
No: weeden
21.0
1.%
Vow Diutt
Mac Pallic Mute
Nakc Pho
(c) Mid/high unstable MIA decisions: flip,, E [0.25, 0.4)
Perter
with 107
Sindittrr (10A
To.. - 300%
Fiue 12.4%
4Dw 33
100
Tales Postive Tate (bg)
Tales Fod Rate
Very unstable MIA decisions:
flip 125 E [0.44 0.490)
Neative
Frative
Porten
protidic life 0.40
2.5%
Bay Kig
1---- 11.35
0%
0.03
02
0.5
03
1daa
Take (lg))
(max)
(mnax)
(c) Coin-fipp-lke MIA decisions: flip,, E [0.490, flip,,
(flip
0.504)
Figure 28: Flip rate variation by fixed FPR for the 140M model. For different ranges of flip,, 125,, we plot how class-conditional flip rate varies by FPR. We annotate plots with corresponding mean t standard deviation for the corresponding TPR. See main text for additional discussion.
48

---

Fostiv
A0
Am
AM
AM
1.6
Nt of with C0820
3:00
300
04
05
5.5
Nut
(a) Very stable MIA decisions: flip 1227 [0, 0.1)
Tue Tortive
True Fostive Fare
2.5
40.00
2.44
2241
0.10
000
20.00
Poraay
35%
54.290
(0.2)
116 12.1%
Eiev + p0%
2031
Fin
0.3
34
(b) Low/mid stable MIA decisions: flip,, 127 E [0.1, 0.25)
No: weeden
33.3%
1ae Pallic Mute llgg
Nakoo Pho
(c) Mid/high unstable MIA decisions: flip,,, E [0.25, 0.4)
Perter
ife 0..444571
47%
Bhe 31.7%
33%
Ra, 282%
11
Tales Postive Tate (bg)
Tales Fod Rate
(d) Very unstable MIA decisions: flip 1277 E [0.4, 0.487)
Frative
Porter
33%
Bhw 23.7
(16
Tiu 13.44
3%
5%
30L
3.02
0.5
1daa Pratire Take (lg))
(maxx)
(maxx
(c) Coin-flip-like MIA decisions: flip,, (0.487 flip,,117 (flip, -0.50394)
Figure 29: Flip rate variation by fixed FPR for the 302M model. For different ranges of flip, ,ip,,. B we plot how class-conditional flip rate varies by FPR. We annotate plots with corresponding mean + standard deviation for the corresponding TPR. See main text for additional discussion.
49

---

Table 140M-parameter model flip proportions for different FPR. For different settings of FPR, we show the percentage of samples (by class) whose empirical flip lies in each range. Lower flip corresponds to more stable MIA decisions; values near 0.5 indicate coin-flip-like behavior. We split the high-instability region into (0.4, 0.490) and [0.490, 0.504] to isolate cases that are statistically in- distinguishable from a coin flip. The max population flip is 0.5; with 125 seeds, the empirical max =0.504. (tg. (125)=00 490 is the minimum value at level a=0.05 that our hypothesis test yields; Appendix E.2.4.) See Appendix E. Typically reported log-scale FPR rows are highlighted in gray.
very stable
low/mid unstable flip E [0.1, 0.25)
mid/high unstable flip E [0.25, 0.4)
very unstable
coin flip
FPR
flip E [0, 0.1)
flip E [0.4, 0.49)
flip 2 0.49
Non-
Non-
Non-
Non-
Non-
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
10
98.93%
100.00%
0.86%
0.00%
0.17%
0.00%
0.03%
0.00%
0.01%
0.00%
10
95.45%
99.99%
3.31%
0.01%
0.93%
0.00%
0.24%
0.00%
0.07%
0.00%
10 10-3
83.94%
99.74%
9.47%
0.25%
4.31%
0.01%
1.63%
0.00%
0.65%
0.00%
10
57.48%
94.88%
16.56%
4.34%
12.46%
0.66%
8.52%
0.12%
5.00%
0.00%
0.02
45.97%
88.12%
17.55%
9.24%
15.23%
1.91%
12.95%
0.
8.29%
0.01%
0.05
28.46%
70.12%
17.10%
20.63%
18.72%
6.88%
21.56%
2.37%
14.16%
0.01%
10
14.37%
45.94%
14.60%
30.39%
21.69%
15.40%
30.34%
8.27%
19.00%
0.00%
0.2
2.93%
17.29%
10.51%
30.34%
24.80%
26.59%
37.46%
17.83%
24.30%
8.95%
0.5
8.85%
3.52%
25.61%
13.00%
32.29%
26.17%
24.92%
31.79%
8.34%
25.53%
0.75
49.02%
26.08%
30.66%
18.59%
15.17%
20.91%
4.32%
22.21%
0.84%
12.21%
10
100.00%
100.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
Table 4: 302M-parameter model flip proportions for different FPR. For different settings of FPR, show the percentage of samples (by class) whose empirical flip lies in each range. Lower flip corresponds to more stable MIA decisions; values near 0.5 indicate coin-flip-like behavior. isolate case that
We split the high-instability region into [0.4, 0.490) and [0.490, 0.50394] statistically indistinguishable from a coin flip. The max population flip is 0.5; with 127 seeds, the empirical max is Re0.50394. (117)=)000
the minimum value at level a=0.05 that our hypothesis test yields; see Appendix E.2.4.) See Appendix E.2.1. Typically reported log-scale FPR rows highlighted in gray.
very stable
low/mid unstable flip [0.1, 0.25)
mid/high unstable flip E [0.25, 0.4)
very unstable
coin flip
FPR
flip [0, 0.1)
flip E (0.4, 0.487)
flip 2 0.487
Non-
Non-
Non-
Non-
Non-
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
Mem.
10
94.46%
100.00%
4.17%
0.00%
1.09%
0.00%
0.23%
0.00%
0.05%
0.00%
10
84.15%
100.00%
10.06%
0.00%
4.15%
0.00%
1.31%
0.00%
0.33%
0.00%
10 10-3
64.59%
99.85%
16.15%
0.14%
11.01%
0.00%
5.87%
0.00%
2.38%
0.00%
10-2
35.27%
95.41%
18.24%
4.14%
17.25%
0.40%
16.69%
0.04%
12.55%
0.00%
0.02
24.87%
88.61%
17.04%
9.81%
18. 18.29%
1.39%
21.38%
0.17%
18.41%
0.03%
0.05
11.42%
67.44%
12.96%
24.64%
18.67%
6.40%
29.49%
1.27%
27.46%
0.24%
10
3.45%
40.16%
7.67%
36.41%
18.28%
16.59%
37.22%
5.28%
33.39%
1.56%
0.2
0.26%
14.07%
3.62%
31.40%
20.97%
28.54%
42.65%
17.33%
32.49%
8.66%
0.5
11.57%
5.13%
27.88%
13.98%
37.14%
26.02%
18.81%
31.66%
4.60%
23.21%
0.75
50.77%
28.47%
31.22%
17.07%
15.27%
19.34%
2.53%
20.53%
0.22%
14.59%
10
100.00%
100.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
0.00%
50

---

E.2.6 How many MIA true positives are statistically indistinguishable from a coin flip?
From the above analysis, natural follow-on is to attempt to estimate how many attack true positives are statistically indistinguishable from coin flip. That is, when we report ROC-AUC at fixed FPR, how much of the corresponding TPR is composed of positive MIA decisions that are essentially coin flip, rather than reflecting reliable inference signal? In this appendix, we use the results from our exper- iments in Appendix F.2.5 estimate that correspond to ranges for flip rate, and estimate the count (and rate) of true positives for each bin.
this question. We split members into different bins
This decomposition is post-hoc audit tool that uses target replicas. An attacker facing single target cannot know which its true positives are indistinguishable from a coin flip, when making per-sample membership claims. Therefore, the takeaway here is about reliability of per-sample claims that an attacker makes, not about an attacker's observable signal. Our aim is to surface the extent of unreliability that may affect the attacker's 's claims (regardless of the attacker's knowledge of reliability of those claims).
We perform analysis that aggregates across attacks multiple target models, so the numbers and figures we report are not comparable with single-attack results in the typical MIA setup. These results serve diagnostics to assess attack reliability. Individual attacks specific targets of course vary in their performance, and are not directly comparable to what we present here that aggregates over many such attacks to try to better understand overall properties of attack behavior.
Decomposing attack TPR into contributions from flip rate bins. Fix dataset DIN with [Dx|==M member examples and B i.i.d. target replicas T1, ri...., TB ~ p. At a fixed FPR n, each larget replica sets its threshold T, (n), calibrated non-members (Section 2 & Appendix A).
For member æ, define the per-seed decision
(x) = (x) 2 (n)) E (0,
(This is just the binary membership decision rule 6(*) (x) (x) 2 (n)) E {0, 1j. calibrated on non-members, but defined here specifically only on members for reflect this analysis.)
So the define the seed-wise true positive count and seed-wise true TPR are
TP. M
TP, = Σ (x)),
TPR,
E
Flip bins. Using the unbiased flip U-statistic flip,, (2) (Equation 9, computed once from all B seeds), we partition members into disjoint flip bins [Byli. e.g. [0,0.1), [0.1, to
(max),
[0.25, 0.4), (B))),.. and (B), flip,, with (B) from Appendix E.2.4. For example, fo.0s (127)=0..487 and fo.o5 (125)==0.49))))
For bin j and seed we define the bin TP count and its TPR mass:
TPR., = TPr.
TP
Σ YH()). EE
M
By construction, TPR, = E3 TPR for every seed
Across-seed means and standard deviations. All means and STDs are computed across seeds. (We report STD across seeds, i.e., with denominator B, to match the rest of the paper.) For totals,
1 B D TPR, TPR)2,
TPR B
TPR,, STD(TPR)
B
and the corresponding counts follow from the variance scaling law (Var[aX] a² Var[X)]:
#TP M TPR,
STD(#TP) M STD(TPR).
51

---

For each bin j,
/ CTPR. TPR))*
TPR3
Σ B
TPR
STD(TPR))
and analogously for bin TP counts: #TP, = M TPR, and STD = M STD(TPR,).
Share of TPR from a bin: mcan-of-ratios. attributable to bin j:
For seed r, define the per-seed share of TPR
TP TP,
Sru
(set Tf,
In our Tables 5 and 6, we report the across-seed mean and STD of these shares:
S -
1 1= Σ S.j
STD(S;) V E (Sr.
1 - Sj)²
(S.s
This is mean of ratios, i.e., the average fraction each seed's TPR that comes from bin j.
Alternative (ratio-of-means) and why differs. ratio of means,
A different but also reasonable summary the
B ST,T, R2 TPR E.
TPR,
B E.
BE.TH TP,
R2 answers "what fraction of the expected true positives lie in bin j?." whereas Sj answers "for typical seed, what fraction of that seed's true positives lie bin j Because of seed-to-seed variability and the correlation between numerator and denominator, R₃ Sy in general. Our tables use Sj (mean of per-seed shares) and its STD across seeds, fraction the average/typical seed's true positives that lie in each bin (in particular, the statistically bin).
trying estimate the
diiitinngissaabl-ffm----oooi
Combining two bins. Let the two high-instability bins be (=(0.,,ta(BB) and (maxx)
(max), (B), f and define the combined bin C-UUA [0.4, flip,, ( J.
For each seed r, S,c = Sr + S.A,,
TP,.c = TPr, TTp + TP,,
r.A.
Therefore, means add by linearity of expectation: 3 Sy k SA and #TP #T #TP However, for standard deviations,
STD(Sc) = / Var(Sy)) + Var(SA) + Cov SA),
Since we compute directly from per-seed C' values, we can easily combine bins in a way that is statistically correct. We report this combined bin in our tables.
What has no STD. Bin membership counts [Byl and their percentages [B,I/M have across-seed STD because bins are defined once from flip,, computed using all B seeds. True positives, though, do have across-seed STD, as these are estimated for each target.
Assumptions and caveats for interpreting these numbers
Single-target vs. many-seed view. These numbers diagnose instability that is hidden to an attacker with access to only single target. A sample counted "TP from coin-fip-like MIA decisions" is not "incorrect" is a member that this seed calls positive, but whose decision flips frequently across equally plausible seeds. Even though it is not incorrect, it does not reflect reliable knowledge about inferring membership for that sample, since it effectively a coin-flip decision. We quantify how much of TPR is borne by such decisions. However, any single target (that could reasonably be attacked by real-world attacker) may deviate from this mean.
52

---

Table 5: 140M model: Contribution of high-flip members to TPR at fixed FPR. Shares are computed per seed (TPs in bin divided by total TPs for that seed) and then averaged; their STDs are across seeds. We additionally report the combined bin ff [0.4, 0.504] ("very unstable coin max flip"). For the 140M model, the coin-flip cutoff is ta (125)=a0.4 490, and flip,,,15 125 00.04. Typically reported log-scale FPR rows are highlighted in gray.
TPR FPR (Mean=STD)
TP (Tean+STD)
Very unstable [0.4. 0.490)
Coin flip [0.490, 0.504]
Very unstable coin flip [0.4, 0.504
TPs (mean-LST))
Share of TPR (Mean-STD)
TPs (mean (TD)
Share of TPR
TPs (mean-STD)
TPR (mean
10-3
0.002
+ 0.001
1,114+300
61 9
5.8%
1.4%
24 + 4
2.3%
0.7%
84
8.0%
2.0%
10 10-4
0.008 + 0.001
4,247 +: 563
432 1 46
10.2% = 0.7%
160 + 13
3.8% = 0.4%
592 1:2::5
14.1% + 1.0%
10
0.030 0.002
15,826 = 1,239
3,048 304
13.3%
= 0.4%
1,493 + 142
6.8% 0.3%
4,541 413
19.9%
0.8%
10 0.104
04,373
17,240
1,464
31.2% = 1.0%
12.164+10070
22.8% = -1.0%
29,404
54.0% 2.5%
0.02
0.148 0.005
77,667
+,7118
27,455+:2,031
34.7%
+ 0.8% 20,609
+ 1,765
27.1% + 1.1%
48.064 +36998
61.8%
3.0%
0.05
0.236+00000 0.006
123,864
0 2,978
47,831+2,774
38.7% + 0.7% 37,910
3,223
30.5%
= +11%%
$5.742+5,449
+.22%
0.1
0.336
176,346+3,118
70.670
39.2% + 0.6% 53,769
3,443
31.4%
= 0.9%
124,439 + 6,231 70.5%
70.5%+2.77 2.7%
0.2
0.480
0.006
251,888
1: 3,256
97,926
39.1% 1 0.3%
67,384
1 1,688
26.4%
0.3%
165,310 + 4,489 65.6%
1.2%
0.5
0.766
1 0.004
401,354
2,242
74,851 1 610
18.7%
0.2%% 1 0.2%
485 28,804
7.2%
1 0.1%
103,655 1 807
25.8%
1 0.3%
0.75
0.915
0.002
479,540
1 882
14,674 1 339
3.1%
1 0.1%
2.641777
0.6%
1 0.0%
17,318 1 403
3.6%
0.1%
10"
1.000 0.000
524,288 0
0+0
0.0% 0.0%
0
0.0%
0.0%
0
0.0% + 0.0%
2. Calibration asymmetry. Again, we note that thresholds T, (1) are calibrated on non-members for each seed, anchoring to non-member behavior by construction. Member decisions are therefore more exposed to seed-induced variation. This explains large member/non flip gaps and is consistent with our seed-to-seed т dispersion higher 1,. The true positive decomposition analysis we perform here is consistent with these other results.
3. Finite-B effects and hypothesis testing. The acceptance region cutoff ta (B) is derived from an exact two-sided binomial test at level and is slightly conservative because of discreteness. Borderline cases (exactly at the tails) are included (fail-to-reject rule 2 a). This is important because all claims that we make about MIA decisions that are indistinguishable from coin flip- including the decompositions
hinge on the assumptions and results of this hypothesis test.
4. Extremely low FPR When TPR is very small, the share Srjj can be numerically unstable; we suppress shares when TPR = O and note this where appropriate.
Results of the decomposition. We show results for both model sizes in Tables and 6, respectively, They indicate very large numbers of member decisions are indistinguishable from coin flip or highly unstable as FPR increases and moves the decision boundary into denser parts of the score distribution. 40.6%% of true positives for the 302M model are indistinguishable
Even at just FPR=10-- from coin flip- reflecting thousands of sample MIA decisions. If we consider both very unstable and coin-flip-like MIA decisions, they are responsible for 4..9% of true positives at this FPR.
Estimating the contribution of coin-flip-like MIA decisions to ROC-AUC. We similarly can use this type of analysis to estimate how much of ROC-AUC can be attributed to MIA decisions that are indistinguishable from coin flip. Similarly, the point of this analysis is not to suggest that these MIA decisions are "incorrect;" the point is to attempt to distinguish the degree to which coin-flip-like MIA decisions are impacting overall claims about successful membership inference. To do so, we distinguish between the whole (averaged) ROC curve and associated mean AUC, and the mean ROC curve (and mean AUC) computed for non-coin- -lin-flipp-ike decisions.
For a ROC curve written as TPR vs. FPR, fix an FPR interval (a, b] C (0.1) For given target r, denote TPR, (77) as the true positive rate at operating point FPR=7, and let (P)()) be the mean across seeds.
1) and write w = b
We distinguish positives that come from coin-flip-like decisions as follows: at each FPR, we remove the portion of the TPR that is supported by decisions that are indistinguishable from a coin flip. That is, fix B targets and significance a; let (B) be the exact two-sided Binomial(B,
53

---

Table 6: 302M model: Contribution of high-flip members to TPR at fixed FPR. Shares are computed per seed (TPs in bin divided by total TPs for that seed) and then averaged; their STDs across seeds. We additionally report the combined bin flip,,.27 E [0.4, 0.50394 ("very unstable max + coin flip"). For the 302M model, the coin-flip cutoff is (127))== and 127 Typically reported log-scale FPR rows are highlighted in gray.
TPR (mean
TP (Mean+STD)
Very unstable
Coin flip (0.487 0.50394
Very unstable coin flip 0044, 0.5394
FPR
[0.4, 0.487)
TPs (Mean:LSTD)
Share of TPR (Mean-STD)
TPs (mea
Share of TPR (mean-STD)
TPs
TPR (mean LTD)
(mean
10 10-5
0.009
+ 0.003
4,878 + 1,496
371 + 66
8.1%
2.1%
122 + 16
2.8% 1.5%
493 80 10.9% 3.6%
10 10-4
0.027
0.005
14,254 2,631
2,233 + 349
15.8% = 1.0%
805 1 87
5.8% = 0.9%
3,038 1 432 21.6% + 1.8%
10-3
0.073 1
38,414 4,886
10,295 + 1,548
26.9%
0.9%
5,883 + 651
15.4% = -0.6%
16,231 - 2,194 42.2% 0.9%
10
0.003
94,823 + 6,939
32,549 3,544
34.3% = 1.5%
$1,866
33.5% = 1.5%
64,477 6,973
0.02
0.235
0.014
123,384 + 7,183
3,891
36.1% = 1.3%
47,301
+ 4,805
38.2% 2.0%
91,913
74.3%
3.3%
0.05
+
174,687 + 7,312
69,431
4,149
39.7%
= 0.9%
71,632 + 6,160
40.9%
= 2.2%
141.062 +0,277077770
80.6%
3.1%
0.1
0.435
228,106-
7,209
98.285
+ 4,353
42.9%
0.6%
88,061
5,522
38.6%
1.4%
186,021 0,6,021=997000 9,760
81.5%
2.0%
0.570
0.013
299,000 i 6,649
128,904 3,649
43.1% 0.3%
87,119
29.1% = 0.2%
216,023
5,982 72.2% 1 0.4%
0.5
0.809
1: 0.008
424,228
3,952
66,051
+: 1,888
15.6%
0.4%
241 12,913
3.0% = 0.1%
78,964
2,093
18.6%
0.4%
0.75
0.925
0.004
484,974
1,908
9,094 1 372
1.9%
0.1%
613 28
0.1% = 0.0%
9,708 1 394
2.0%
+ 0.1%
10"
1.000
524,288 0
0
0.0% = 0.0%
0
0.0% = 0.0%
0
0.0%
0.0%
acceptance-band flip cutoff (Appendix E.2.4). Let M = IDIN denote the number of members. For seed r FPR 7), write (xx) E {0, 1} for the membership decision on member æ, and let
TPR (1)
1
Y=)),,
TPR (1) M
Σ Y,(x) 1 ((fi), 2 ta B))
M
ΦEDIN
ΦEDNN
We define the non-coin-flip wwoii--ii-liik--nndeeeeee TPR per seed by subtraction,
TPRponcoin, (1) TPR (n) TPR (1))
We compute the averages across B seeds
TPR (n)
B Σ TPR TrR (1); TPR (11)
1
TPT PTnononcinnr/ (1)).
B
By linearity of expectation, across seeds,
TPRnnncoin (1) TPR T (1) TPRooin
(24)
Note that TPRmoncon is a diagnostic curve: it subtracts the statistically indistinguishabl-ffomm from-a- coin-flip component and is not itself the ROC of single threshold rule that could be produced by an attack.
We can then compare the overall ROC to the non-coin-file soiin-fiip-like ROC. It may be useful to do so for specific range of FPR, rather than for the entire ROC curve. To so, note that the unnormalized partial AUC (pAUC) over (a, b) is
J.
PAUC
TPR(() dn.
The normalized pAUC the mean TPR averaged over FPR in [a, b]: it rescales to [0, 1] by dividing by the band width:
PAUC narm la, b]
TPR(n) dn. which equals the mean TPR over the band.
b
To quantify improvement over random classifier, subtract the area under the by-chance curve.
For the random classifier TPR(n) FPR(1), so
ddn = 1/2
12 a²
2
m=
54

---

Dividing by = b gives the normalized random baseline b-a 2 2 report the lift above these random basclines:
atb
We can similarly
Lift[a, bj = L. (TPR(1) - -1) dn pAUC 6j
b-
2
Liftnorm (a, b] == b [. (TPR(1) - dn- = pAUC norm (u, b)
1
+ b
2
Note that nift (a, bj 0 for random classifier and equals the average vertical gap from chance in [a,b]. b].
Worked example. We apply this procedure to the ROC for the 302M model. We do so for the entire curve in Figure 30b, which shows that ==0.559 of ROC-AUC can be attributed to coin- flip-likk MIA decisions. Of course, the region where this has the most impact is the range of 17 that moves the decision boundary into the denser part of the score distribution where MIA calls more positives, but IN/OUT distribution overlap is more extensive. TPR rises, but so does the share of coin- fin-ffip-like true positives.
We can therefore also isolate the effect on this part of ROC-AUC by setting [a, [10-4 44, 1.101.. Here w = 0.0999, the normalized random baseline is a+b 0.05005, and the unnormalized random 2
area is 6²-0² 0.004999995. With our measured areas,
PAUCEW noom 0.314748,
DAUChoneoin 0.190810,
so
Lift
Faw norm 0.264698,
Lift noucoin norm
0.140760.
As a result, the non-coin-flip-like ROC retains about 0.140760 x 100 R 53% of the raw lift in this
0.26-1698
FPR band, and the average TPR for the band drops by 1 0.190810 x 100 ~ 39.4% relative to
0.31
raw. In other words, filtering out coin-flip-like positives yields substantially lower pAUC this band. A sizable part of the apparent attack advantage in this FPR range comes from samples whose per-seed decisions are statistically indistinguishable from coin flips (Appendix E.2.4). This weakens the value/reliability of per-sample positives in this FPR regime.
It is important to note that this is conservative estimate of the parts of the attack that reliable, as we only filter out positives that pass the coin-flip-like flip threshold, to (B). As a result, these numbers still include highly unstable cases (that are arguably also not reflective of meaningfully reliable membership inference, e.g., flip in [0.4, to (B)). In this respect, our non-coin-fii--lik ROC a conservative upper bound on reliable membership inference. Given the extent of highly unstable MIA decisions in this band, we would expect larger decreases in partial AUC and Lift if we filtered those decisions in our analysis.
Caveats. Because bins are defined using flip,, pooled over B seeds, per-seed variability in bin membership is not propagated. This makes our estimates conservative for variability, but keeps the decomposition identity exact (i.e., the sum of the bin masses equals TPR). In our implementation, we also clip TPRnoncoin, 0 to guard against finite-sample noise, i.e., IPPmmoncoin. (T) + m0 TPRrew, (T) TPReoin,
E.2.7 Additional notes on interpreting flip rate measurements
Strong membership inference claims are conditional on training pipeline: given a fixed recipe (model class, optimizer, etc.), does the model's behavior on ø indicate that æ was in the training set? Strong MIAs such as LiRA estimate the sample x' w's score given that x e IN vs. the score given that æ E OUT, computed from many reference models by varying the data, i.c., they probe data counterfactuals under the same pipeline.
Our flip rate analysis probes the complementary axis: model counterfactuuls. We draw equally plausible targets r~ from the same pipeline using the same training dataset (changing only the seed that controls batch order), hold the set references and calibration procedure fixed (i.e., thresholds calibrated per target non-members), and ask whether the binary decision for sample æ is stable across those targets. References address data variability; target flip rate quantifies model variability. From a statistical reliability perspective, sam-l inference should be robust with respect to both.
55

---

14:00 RCC non-ardtzaar ROC 12525. . M-- 21-524,2588. 258. N44-524.299
300 NOC nen-arbiraa RCC (Bm N- NM-524.28))
101
30
20*1
30-1
Random (MIC--3.5 5) Moon ROC lall mumbeso)
Rancom (hiCCod 5) Mean Poc, lall members)
10
10
Non wnterary spc
Non actirary RDC
AUC 0.002
ALC 2.000
10-4
10-1
28-1
10-- 10
10 10--
10-4
20-1
20-1 10
10
Fatas log!
Fatas Phe Paat Dag!
(a) 140M
(b) 302M
Figure 30: Decoupling overall attack success from success based on coin-flip like sample de- cisions. We produce the same mean ROC curves and mean AUC as Figure 24 for both the (a) 140M and (b) 302M models (B=125 and B=127, respectively). At each TPR for fixed FPR, estimate how many true positives are attributable to MIA decisions that are statistically indistinguish- able from coin flip (f.. (125)==0.49) and to (127)=0 487, respectively; Equation 24, at each we can then estimate how many true positives are from non-coin- foi--lii--kkkee decisions. Consistent with other results, As both curves enter ranges for FPR with nontrivial TPR, coin-flip-like MIA decisions make up significant proportion ROC-AUC, with a greater effect for the 302M architecture.
Appendix E.2.4). By
Descriptive vs. inferential claims. Put another way, both axes may matter depending on the type of claim want to make. We can make a descriptive claim [9]: "On this specific model, called a 'member' and were right." In contrast, we can also make an inferential claim: "If we retrained the same pipeline, we would still call æ 'member' most of the time." Flip rate distinguishes between these. In the absence of flip rate analysis, we can make the first type of claim relatively easily, as in the prior work in the literature [e.g.. 5]. But without measuring something like flip rate, we cannot assess the second type of claim. When we do measure flip rate, observing high values (i.e., R)..5) means that the second claim fails: a positive call x carries about the same evidential weight coin toss; may be correct, but not because of stable signal. This type of evidential weight is weak because this call is not reproducible-ewv happens to be attacking, (Conversely, think of a sample æ that has low flip rate: the attack is able to produce reliable signal about membership for that x.))
if correct on the particular target that the attacker
Why ML security should care about inferential (not just descriptive) claims. On a first read of our flip rate results, security- focuseed reader may not necessarily be interested in making an inferential claim. An attacker only gets one model, and so one might think, "our positive call was correct: why care about reliability with respect to the training pipeline that produced that one model?" From this perspective, the descriptive claim suffices.
However, in machine learning, focusing only descriptive claims is arguably insufficient; for statistical procedure (like a strong MIA), the standard for success higher. That is, to develop scientific knowledge, we require the entire atluck procedure to be reliable for making claims about x. From this perspective, a statistically coin-flip-like true positive is a lucky hit, not reliable evidence produced by this specific attack procedure. (Again, conversely, stable true positive is reliable evidence produced by this procedure.)
This is why we perform flip rate analysis: it is useful diagnostic for making scientifically sound claims about the entire attack procedure, which is precisely what we are trying to address in this paper: the extent to which strong MIAs (as an attack procedure) can succeed on LLMs. For an attacker that attacks a single target, traditional MIA metrics (TPR fixed FPR, ROC-AUC) are still useful; but our results surface what fraction of decisions are fragile. For reliable scientific knowledge, if that fraction large, then claims about the MIA procedure's effectiveness should be couched as descriptive, not inferential.
56

---

E.2.8 An alternative instability diagnostic: fixing the target, resampling reference sets
The instability analysis in the main paper and the rest of this appendix (Section 5 & Appendices E.2. 1- E.2.7) fixes the IN/OUT reference and varies the target seed, probing model counterfactuals. natural complementary diagnostic, closer in spirit to the typical threat model, fixes the target and varies the IN/OUT references, probing reference counterfactuals. Here we formalize that diagnostic and explain why we did not run it at scale.
Setup and notation. Fix a trained target h on dataset D. Let S()), S()..... ((B) be B independent reference sets; each set (() contains | - references and (our|- OUT references (in experiments, -ouut- Given g(5), define the LiRA score for sample computed exactly in the main setup (Appendix A) but using only the models in { as the references.
(1) (a) E R.
Calibration to non-members at fixed FPR (per reference set). Let the non-member dataset be DOUT with size NOUT- Form the empirical CDF of OUT scores using reference set j:
1
Σ 1(1 (x) (x) t)
NOUT
wEDLLT
For each reference set j and FPR we calibrate a decision rule for the single target h, choosing the right-open (left-limit CDF) cutoff
(n) =
inf inf(t: FU) 2
(x)) = (A (a) 2 9 (n))
Then the realized FPR on the non non-member (OUT) dataset for the target h using references j satisfies
FPR (n) = NOUT
Σ 144 (z)
(x) 2 To (n)) =1- Flo (Th (n) S
#EBour
iee, finite-sample, conservative with ties. This mirrors how attacker would calibrate threshold for the target using the specific reference set they happened to train. (In this case, we train such equally plausible reference sets.)
deffrence-resamplhe flip rate. Let J denote an index drawn uniformly from {I,..., y B), repre- senting an i.i.d. draw of a reference set. For this reference reference-resampiing resampling setup, define the population flip rate across possible reference sets for the fixed target as
flip,) rel (x)
Pr
[0 (1 (x) + (n) (x) 2 20 oret)
J.J (1 (1....BB
B}
where gref = Pry (b(n.)]
(x) 1]. Given reference sets, the unbiased estimator
-1
1( (x) + ((m))) ())=0(())) (x)) = 2 B (x) BT (æ)
flip,,, (x)
Σ
1
B (B-1)
ISjCESB
with Bre E 6) ((1.1)) (x) and (x) B- BF)((x). Just as in our target-resampling analysis,
we label a decision "coin- loie- at level a" by testing Ho gret = 0.5 with the exact two-sided ref binomial test on B votes; equivalently, flip,, 2 ta(B), where to (B) k (B-k)) and kG is the equal-tail cutoff (Appendix E.2.4).
B(B-1)
Cost and feasibility. In the main paper terget-rssampling flip analyis, we trained one reference set of size R= and 112 targets (R+ (R++BB = 255 total trained models for the 302M setting). For this rereeence-rssampling diagnostic, obtain comparable statistical fidelity, one would fix the target and train Bes independent reference sets, each of size R, i.e., train R x Bye reference models. Taking But R 128 yields 128 x 128 16,384 references (plus one target), requiring about 4x more models than all of the other experiments in this paper combined. To approximate this diagnostic, we could perhaps subsample smaller reference sets from single larger pool (e.g., subsample 128 references from the target flip rate experiments). However, such subsampling would introduce dependence across reference sets, violating the i.i.d. assumption and complicating the exact binomial test use to determine the coin-flip cutoff.
57

---

Why we expect qualitatively similar instability to our farget-resampling experiments. Empiri- cally, our cross-target flip rate analysis shows that many members have small margins relative the calibrated threshold: for these æ, (x))) (n) is frequently near 0 (Appendix F.2.5).
For the reference-resamplee diagnostic, write the per-reference-set margin for the fixed target
A A (x)) = A (z) (n)
(1
target data fixed
Score variability from rel. sets
quantile variability from per-set calibration
where A captures sampling variability from fitting IN/OUT reference statistics and of captures quantile-estimation variability in 7/ (1)). Our cross-target results show many members with margins near 0 (poor separability of reference distributions and proximity to the threshold). For such samples 2, any nonzero variability in e (a:) e!") will, with nontrivial probability, change the sign of (x) and thus flip the decision. And so, given the sensitivity that we observe when training counterfactuals, expect this reference-reseee resamplin diagnostic to surface qualitatively similar per- sample instability as the target-resampling experiments we run- cun-especally for members- wembberswwithout running the much more expensive eeference-resamlinng procedure.
F Additional per-sample MIA vulnerability results
Figure 6b indicates that it is often the case that vulnerable sequences tend be longer. Beyond sequence length. we observe that samples more vulnerable to MIA tend to have higher mean TF-IF scores (Figure 3la), suggesting that texts with distinctive, uncommon terms may exhibit stronger signals for membership inference. We compute these TF-IDF scores without normalization, collecting document frequency statisties over a random subsample of the original dataset and then taking the mean across all tokens in each sample. Similarly, samples containing unknown tokens (<unk>) appear more vulnerable to MIA (Figure 31b).
Most Winneabble
Most Ineerrae
Least Vulnerable
Least Vulnerable
12-
0.8
10.
0.5
Density
Denoity
0.4
0.2
0.0
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
Mean TF-IFF
Unknown Token Count
(a) Mean TF-IDF scores by vulnerability
(b) <unk> counts by vulnerability
Figure 31: Text property distributions by MIA vulnerability. The most vulnerable samples tend to (a) have higher TF-IDF scores compared to least vulnerable samples, and (b) are more likely to contain at least one unknown token (<unk>).
F1 Does memorization imply strong membership inference attacks?
While memorization is key factor that make model susceptible membership inference attacks, does not automatically guarantee that strong MIAs will always be successful. Memorization refers model learning specific details about its training data, rather than just general patterns.
When a model heavily memorizes training samples, often exhibits distinct behaviours for these samples, which MIA attackers, in principle, can exploit. Indeed, studies have shown that the risk of
58

---

membership inference is often highest for those samples that are highly memorized [4]. However, our results show that the practical success and strength particular MIA can also depend on other factors, such as the model architecture, the type of data, the specifics of the attack method, and whether the memorization leads to clearly distinguishable outputs or behaviors for member versus non-member samples. Some models might memorize data in ways that are not easily exploitable by current MIA techniques, or the signals of memorization might be subtle for gell-generalizing models, making strong attacks more challenging despite the presence of memorization.
F.2 Evolution of losses over different model sizes
In Figure 32, for three samples, we plot loss (target) and the per-sample reference distributions PIN PIN (x) and Pour (æ) over different model sizes. Each of these models is trained for epoch on 2 R 8.3M samples. This sanity check that the losses decrease (on the same sample) as the model size increases. Note that, for these samples, the distance between member and non-member reference distributions does not significantly shift as the model size grows
Sample ID: 16777211, Member
Sample ID: 16777212, Member
Sample ID: 16777213, Non-member
34
135- Hember (efererce)
Non-Marmbr infermnced
"brget
byge
10M
5.0
3.0
3.2
33
36
20
15.0
15
12.5-
44M
15.0
10
75
5.5
2.5
2.8
55
6.0
32
23.5
15.0
13.5
12.5
13.0
12-
85M
15.0
13.0
7.5
5.0
2.5
2.5
5.0
3.0
3.0
3.8
3.8
31
32
34
3.
3.6
3.
3.8
23.5
144
15.0
15.0
12
12.5
12.5
140M
25.0
5.0
3.0
31
32
34
31
3.4
35
3/8
15.0
125-
15.0
302M
2.5
3.0
6.0
Loss
Losg
Figure 32: Target loss and reference loss distributions for three samples. For three different sam- ples (referenced by their IDs in the C4 dataset), plot the reference distributions and the loss of the sample for the target model (as a vertical red line). Each row shows results for a different model size.
59

---

G Experiment configuration details
In Table 7. we provide pecific experimental settings. Unless otherwise stated, we used the AdamW optimizer [32] with a cosine scheduler. The initial learning rate is set to 10-² and increases linearly over 750 warm up steps to a peak learning rate of 3 10-4, after which it decreases according the cosine schedule to a final value of 3- 10 We typically use 128 reference models and single target model to measure MIA vulnerability, drawing a dataset size 2NV from C4 from which we subsample training datasets of size N. For each reference and target model, the training set is subsampled from the same larger dataset of size 2N. This means each sample this larger dataset is member for <64 reference models. The batch size is fixed to 128 and sequence length to 1024; if sample has fewer tokens, we pad to 1024. The weight decay is set to 0.1, and a global clipping norm is set to 1.0. Note that we can approximately convert the training set size to total number of training tokens by multiplying the training set size by 400, as this the approximate average number of tokens within C4 sample. For example, this means the 1018M model was trained on 20.4B tokens in Figure 2.
Table 7: Experimental details. Experiment (figure), training set size (approximate number of samples), model size, and specific details that diverge from default settings.
Experiment
Trininn Model Other information (witcch diverges defuslt experimental scttings)
Figure
HIM
B00K
10M
2.2M
41M
1.25M
85M
Figurex
15.1M
802M
128 refereneces
2.2M
41M
GMM
differest variations; epoch spocls samples, split differeut ways epochs)
Figures
10/
spochs
30K
10M
100K
500K
140M
steps
100M
10M
10M
615
85M
140M
Figure
223
425M
6MIM
101 KM
Figures 5, 23, 24hh
25,
302M
sugett (vaaying only candom training dato); reforences
Figures 7,8
7M
140M
128 reforences
100M
Mas, references
Figure 96 Figure
7M
10M
Max. 64 references. 10K popristion size
7M
10M
250 referenees
Figure
G00K
10K 2 population size
Figures 12, 13
SMOK
I0M
10K 200K Z population size; 64 reforences
Figure 14
2
I0M
128 ceferences (lesting online and offinee varients)
Figure 15b
50K
LUM
Leaming tate schednles: cosine, cosine with weight decay, cusine with no clipping. lineur. We use 50 wurn up steps.
Figure
7M
1IM
Compuring to de-dhpliesttdd training dataseet
Figure 17h
IIM
30 spachs
Figane 18
Ideiccal Figare 2, where we use 16 diffieent tirget models
Figue 19
Idenical Figuse 40, where we use 16 differcat turget models
Figure 20
Idenical Figure 18, except varying references tup to 128).
Pigure 21
27
10M
Up to 64 turgets, 64 reforences
Figures
26, 28,2,,20 30n
7M
140M
125 sarget models (varying only random seed: same training datal: 125 references Memicall Figure
Figure
Figure
2Fa
10M-300M model sizes
60

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The results we provide in Sections 3-6, and the Appendix provide an accurate and nuanced treatment of the main claims introduced in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: In the introduction, we note the cost of our work, which is limitation for those that wish to reproduce our experiments. We document the challenges we observe with MIA attack variability, particularly in Section 5 and the Appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
61

---

Answer: [Yes]
Justification: While the strong attacks that we investigate in this paper are theoretically grounded, main contributions empirical. In Appendix E.2, include straightforward theoretical results related to flip rate, the metric we adapt from prior work [14] to measure per-sample prediction instability. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: While the cost of training thousands of LLMs ranging 10M to 1B parameters is substantial, those with the resources to do so would be able to faithfully reproduce our main results. We thoroughly document the tools we use and our experimental configurations throughout the paper, as well as in one centralized place in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
62

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: As discussed in the prior answer, while we do not link to code repository, we provide ample details on the open model architectures [31] ] and datasets [47] that we used to conduct the experiments in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ Beaili/cguides/CoeSSuubaae wttssuummssioooooioooooooooo for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ httppss.ccpuulcc//gguiu httoo/////iddes/oeeeuubiiiiiiiii sondeSubmissiooPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide extensive details on our general setup in Section 3, on more specific experimental configurations in the following sections, and more detailed information in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We discuss variability and instability across different attacks for 140M and 302M models. See Appendices D and E.2.
Guidelines:
63

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi-
dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative етror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide these details in centralized location in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http//nnurii..re. http://ewerr...ccccciiiiiicceeee httppulli//chiccctteeeeeeeeeeeee
Answer: [Yes]
Justification: We read the NeurIPS Code of Ethics and confirm that the research conducted in this paper conforms in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
64

---

We discuss and motivate our work with respect to the broader impact it has regarding developing scientific knowledge about improving the privacy of LLMs.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use open-source model architectures [31] and datasets [47], which credit in several places in the main paper, Appendix, and this checklist.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
65

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcdde com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
66

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / wttpe//ps.c//oooeeeeeeeeeeeeeeee wc//onofrencce////0/ /LLM) for what should or should not be described.
67


Reviews:
[{"review_id": "vF6TOh1fGY", "reviewer": "Reviewer_2cHX", "summary": "This paper scales strong MIAs to large language models, addressing limitations of prior work that used weaker or small-scale attacks. By applying the LiRA attack to thousands of GPT-2–like models, the authors show that strong MIAs can succeed, but their effectiveness is limited in practical settings. They also find that vulnerability does not consistently increase with model size and that MIAs and data extraction capture different privacy risks.", "questions": "1. Why does the paper mostly focus on the AUC evaluation metric and not on TPR at low FPR? Can you provide more insight from this perspective?\n\n2. Can we provide more insights into MIA attack designs or suggest directions for improving MIA attack design for LLMs?\n\n3. This paper involves large-scale experiments that require significant computational resources. Can we also provide more insights from a cost perspective? Also, the possibilities of further scaling this to larger LLMs?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. Insightful results (e.g., data used in later training is more vulnerable; no clear relationship between MIA and extraction attacks) — these findings are meaningful and could influence future studies.\n\n2. Large-scale experiments.\n\n3. Demonstrates that strong MIAs on LLMs are feasible.\n\nWeaknesses:\n\n1. For the most part, this paper uses AUC as the primary evaluation metric. However, the LiRA paper highlights that TPR at low FPR may be a more appropriate measure for evaluating MIAs. It would be helpful if the authors could also present findings using these alternative metrics, which may yield additional insights.\n\n2. The attack methods are not very novel, but applying existing SOTA MIAs in the LLM setting.\n\nThank you for your submission. I enjoyed reading this paper and learned new things from it—especially the insightful findings about which data is more vulnerable to MIAs and the relationship between MIA and extraction attacks (where many prior works suggested a connection, but this paper presents a different perspective). That said, I do have some concerns, as mentioned above.", "was_revised": true, "final_justification": "I will keep my original score 5. I think this is a good paper and should be accepted.", "rebuttal": "Thanks for your kind review. Answering your questions below:\n\n>Why does the paper mostly focus on the AUC evaluation metric and not on TPR at low FPR? Can you provide more insight from this perspective?\n\nThis is partially explained in lines 153-159: We choose to report AUC as our primary metric, as it is more challenging to visualize the TPR over a wide range of FPR in a streamlined way: (For comparison, see Figure 2b, which provides such an alternate visualization for only a limited range of FPR, at the cost of not surfacing overall AUC.). Every result in our work comes with an AUC-ROC curve (in log-log scale), allowing the reader to decide which FPR threshold is of interest, rather than a prescriptive choice from the authors. For clarity, we can also sign post this/ reference this rationale at the beginning of section 3, where we describe our overall experimental setup.\n\n> Can we provide more insights into MIA attack designs or suggest directions for improving MIA attack design for LLMs?\n\nFigure 1 shows the main difficulty in scaling strong MIA attacks. A large number of reference models (each of which is also more expensive to train at scale) are required to attain peak attack performance. Future attack designs should concentrate on reducing the number of reference models. We discuss this in Appendix A, where we compare LiRA with RMIA. In certain (smaller scale) settings with fewer reference models, RMIA outperforms LiRA.. However, our results also show that peak attack performance (of roughly 0.7 AUC) is only part of the story. Fundamentally, our results also indicate that the performance of MIA relies on how well an attacker can capture per-example variance in MIA vulnerability when the sample is a member / non-member of training. The simplest way to measure per-example variance is through training a sufficiently large number of  reference models. Future work should explore other, cheaper ways to capture this signal. We can include these details in our conclusion/ takeaways section for future work.\n\n> This paper involves large-scale experiments that require significant computational resources. Can we also provide more insights from a cost perspective? Also, the possibilities of further scaling this to larger LLMs?\n\nWe can certainly provide information on computation costs. Our maximum run used approximately 2e19 training flops using 4 TPUv3s (https://cloud.google.com/tpu/docs/v3), we will add a new compute cost column to Table 1 and discussion. Regarding scaling to large LLMs: since we wanted to measure MIA under “realistic” pre-training settings, where the number of training tokens is scaled proportional to the model size, the main difficulty is simply cost. We can reduce cost by reducing the number or fixing the number of training tokens. We plan to investigate the relationship between training tokens and model size in more depth in future work."}, {"review_id": "X0CuWXjuVd", "reviewer": "Reviewer_Lzho", "summary": "The paper studies multiple phenomena for membership attacks that were little known before when it comes to larger models—with enormous numbers of parameters in the era of LLMs and huge training corpus. The paper states three claims: 1- The success rate does not exhibit any meaningful relationship with the model size. I.e., it is not true that larger models are easier to break. 2- The success rate is not related to the training size either. 3- Samples later seen at training are more susceptible to the membership attacks, yet the connection with data extraction is not clear.", "questions": "Address weaknesses above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \nThe paper is well written, with the messages clear in each section and sections well-partitioned which follow a natural flow. The paper has several important messages (mentioned above) that were not fully investigated before, and have a clear impact on the literature. This will motivate the community for further studies. \n\nWeaknesses: \n1- The novelty of the paper is limited as it is grounded on the previously known LiRA attack. The major contribution of the paper is scaling all the models, the number of reference models, and the training data. \n2- One of the surprising results is that AUC never surpasses 0.7 when increasing the number of reference models. Even though the paper claims that they study the fundamental limits of membership attacks (for instance, lines 9 and 10), the low success that is reported only based on LiRA questions the generality and universality of their results. \n3- Having said that, there is more room for explaining many of the results. E.g., why does the saturation for the number of reference models happen so quickly? \n4- In Figure 4a and 4b, where the relation with more training epochs is studied, the authors resort to discussing AUC only. Whereas the plots keep toggling for different values of FPR and it looks like the mentioned message does not apply everywhere. In other words, the statement that membership attacks are more successful when increasing the number of training epochs seems to oversimplify what’s actually happening in the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful review, answering your questions and responding to comments below:\n\n> The novelty of the paper is limited as it is grounded on the previously known LiRA attack. The major contribution of the paper is scaling all the models, the number of reference models, and the training data.\n\nYes - the significance of our work to the literature is primarily in investigating the performance and limits of state of the art MIAs on LLMs across a wide range of settings. To the best of our knowledge, this has not been studied before, which (as we have discussed with respect to preliminaries and related work) has left important open questions in the field about the feasibility and reliability of MIAs in modern pre-trained LLM settings. So, we would like to emphasize that novelty can also be attributed to insightful findings; as noted in your review, our work has discovered several such findings that have helped resolve some of these important open questions and therefore will help advance the field.\n\n> One of the surprising results is that AUC never surpasses 0.7 when increasing the number of reference models. Even though the paper claims that they study the fundamental limits of membership attacks (for instance, lines 9 and 10), the low success that is reported only based on LiRA questions the generality and universality of their results.\n\nLiRA is widely regarded as one of, if not the strongest known MIA. (Please refer to Section 2 for references that support this claim.) We also compared against another very strong attack, RMIA, and found LiRA outperforms RMIA in almost all settings. We are not claiming any theoretical bounds in this work. Nevertheless, it remains important to understand the strengths and limitations of state-of-the-art strong attacks in this setting, in order to get a sense of how vulnerable pre-trained LLMs are to such attacks. To the best of our knowledge, we are the first to investigate this in detail by instantiating these strong attacks and studying them in depth in this setting. Altogether, we believe that our results shed light on what is empirically possible to achieve with an MIA attack in this common (and growing) setting. We contend that it is important knowledge for the field to understand the limits of these attacks (notably, as you’ve mentioned, that AUC is perhaps lower than one might have expected or hoped).  \n\n> Having said that, there is more room for explaining many of the results. E.g., why does the saturation for the number of reference models happen so quickly?\n\nThe variance of an individual example’s loss over member and non-member distributions is captured sufficiently well with a small number of reference models. As a result, at a certain point, there are diminishing returns to training more reference models. We are happy to update the manuscript to reflect this (e.g., by including a similar plot to Figure 19 spanning different numbers of reference models).\n\n> In Figure 4a and 4b, where the relation with more training epochs is studied, the authors resort to discussing AUC only. Whereas the plots keep toggling for different values of FPR and it looks like the mentioned message does not apply everywhere. In other words, the statement that membership attacks are more successful when increasing the number of training epochs seems to oversimplify what’s actually happening in the experiments.\n\nWe assume the reviewer is referring to Figures 3a and 3b. Figure 4 measures MIA under different training and model sizes, and doesn’t vary the number of epochs. We choose to report AUC as our primary metric, as it is more challenging to visualize the TPR over a wide range of FPR in a streamlined way: (For comparison, see Figure 2b, which provides such an alternate visualization for only a limited range of FPR, at the cost of not surfacing overall AUC.). Every result in our work comes with an AUC-ROC curve (in log-log scale), allowing the reader to decide which FPR threshold is of interest, rather than a prescriptive choice from the authors. We are happy to add more discussion on FPR for this set of experiments if the reviewer thinks it will improve the paper, and can also mention this visualization choice at the beginning of Section 3 discussing our overall experimental setup."}, {"review_id": "JkT5jar5hw", "reviewer": "Reviewer_J6ua", "summary": "This work aims to investigate whether the limitations observed in prior studies stem from the design choices of the attacks themselves or whether MIAs are fundamentally ineffective on large language models. To this end, the authors propose scaling LiRA to GPT-2 architectures ranging from 10 million to 1 billion parameters, training reference models on over 20 billion tokens from the C4 dataset.", "questions": "Please refer to Weaknesses.", "limitations": "Please refer to Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The motivation of the manuscript is novel.\n\n2. The manuscript offers a comprehensive experimental analysis.\n\n**Weaknesses**\n\n1. In principle, the main text should be self-contained; however, too many details placed in the Appendix.\n\n2. The reason for setting 4,000 GPT-2-based reference models is not clearly explained.\n\n3. The transitional claim regarding LLMs is somewhat misleading, as the models used in the manuscript are relatively small and do not truly qualify as large language models.\n\n4. Can the manuscript provide feasible recommendations for MIAs that are limited to LLMs?", "was_revised": true, "final_justification": "I have read the rebuttal and it has addressed most of my concerns. I will keep my score.", "rebuttal": "Thanks for the helpful review!\n\n> In principle, the main text should be self-contained; however, too many details placed in the Appendix.\n\nWe will try to fix this to make it more self-contained. Are there details in particular whose inclusion you believe would improve the main text? We ran a very large number of experiments, so unfortunately some details will need to remain in the Appendix. If there are sections where this stood out to you as resulting in unclear presentation, we would greatly appreciate knowing. \n\n> The reason for setting 4,000 GPT-2-based reference models is not clearly explained.\n\nThis was the total number of models we trained accumulated over all of our experiments. The reason for this total number is because, for each experiment, the results (AUC) saturated for a sufficient number of reference models.  At this saturation point, there was no benefit in scaling up the number of models we trained, as AUC remained the same. We will make this clearer in the camera ready version.\n\n> The transitional claim regarding LLMs is somewhat misleading, as the models used in the manuscript are relatively small and do not truly qualify as large language models.\n\nWe intended for  the qualifier in the title to help with this, but we can remove “Large” if desired. It is however worth noting that how “large” a model needs to be in order to be an LLM is not exactly standardized. Much of the literature still does consider models of this size to be LLMs, but, for example, models of this size of course are not “frontier LLMs.” \n\n> Can the manuscript provide feasible recommendations for MIAs that are limited to LLMs?\n\nWe are not sure if the reviewer is asking for a recommendation for how to improve MIAs or defend against them, so we will provide answers to both possible interpretations.\n\nFor improvement: Figure 1 shows the main difficulty in scaling to strong MIA attacks. A large number of reference  models is  required to reach peak attack performance (at this point, performance saturates, and more reference models do not help improve performance). Future designs should concentrate on reducing the number of reference models. We discuss this in Appendix A, where we compare LiRA with RMIA. In certain settings with fewer reference models, RMIA outperforms LiRA.. Fundamentally though, our results also indicate  that the performance of MIA relies on how well an attacker can capture per-example variance in vulnerability to MIA, when the sample is a member / non-member of training. The simplest way to measure per-example variance is through training a sufficiently large number of reference models. Future work should explore other, cheaper ways to capture this signal.\n\nFor defending against MIA: Our insights in figure 5 suggest that certain properties of the training process and the data are responsible for excessive vulnerability. For instance, an example tends to be more vulnerable when it is included near the end of training; “atypical” examples are also more vulnerable.  Data curation and model training should take this into consideration when designing their training pipeline."}, {"review_id": "w4L5hw6vYd", "reviewer": "Reviewer_mdKc", "summary": "This paper studies the effectiveness of strong Membership Inference Attacks (MIAs) on pre-trained Large Language Models (LLMs), such as GPT-2-like models. While previous research has primarily focused on smaller models or weaker attacks, this work scales one of the strongest MIAs, the Likelihood Ratio Attack (LiRA), to large-scale LLMs trained on massive datasets.", "questions": "What potential improvements could be made in the MIA attack methodologies to push AUC values beyond 0.7, particularly for large models?", "limitations": "The experiments are based on GPT-2-like models and may not fully generalize to other types of LLMs (e.g., BERT-based or newer architectures like GPT-4). Further work would be needed to determine how these attacks scale across diverse architectures.\n\nThe technical contribution is limited, although this work provides some insights based on experiments.\n\nUpdated rating after the rebuttal.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The authors conduct experiments on a larger scale than most prior research by training over 4,000 models of varying sizes.\n\n2. The paper covers multiple dimensions of MIAs, such as the number of reference models, model size, training data size, and the impact of different hyperparameters on MIA success.\n\nWeaknesses:\n\n1. The focus on only two MIA methods (LiRA and RMIA) may not provide a full picture of the various possible membership inference strategies. Including more attack methods might have been useful to give a broader view of potential vulnerabilities.\n\n2. While the paper demonstrates that strong MIAs can succeed in a controlled environment, the practical impact remains limited due to the low AUC values (<0.7).", "was_revised": true, "final_justification": "Updated rating after the rebuttal, as most concerns are resolved.", "rebuttal": "Thanks for the review.\n\n> What potential improvements could be made in the MIA attack methodologies to push AUC values beyond 0.7, particularly for large models?\n\nGood question,  It is often assumed that adding more reference models should capture a sample’s member and non-member distributions accurately. However, we found that training more reference models doesn’t improve attack performance beyond a certain point, indicating that training more reference models will not help separate these distributions. Future work could focus on trying to find other signals (beyond the loss of an example) that can help separate these distributions, or focus on establishing if we are hitting an upper limit of attack capabilities. However, it is entirely possible that much higher AUC values are impossible to achieve. We believe that showcasing existing limitations of state-of-the-art strong MIAs (as our results do) is useful for advancing the field.\n\n> The focus on only two MIA methods (LiRA and RMIA) may not provide a full picture of the various possible membership inference strategies. Including more attack methods might have been useful to give a broader view of potential vulnerabilities.\n\nLiRA is widely regarded as one of, if not the strongest known MIA. (Please refer to Section 2 for references that support this claim.) We also compared against another very strong attack, RMIA, and found LiRA outperforms RMIA in almost all settings. We are not claiming any theoretical bounds in this work. Nevertheless, it remains important to understand the strengths and limitations of state-of-the-art strong attacks in this setting, in order to have a sense of how vulnerable pre-trained LLMs are to such attacks. To the best of our knowledge, we are the first to investigate this in detail–to instantiate these strong attacks and study them in depth in this setting. Altogether, we believe that our results shed light on what is empirically possible to achieve with an MIA in this common (and growing) setting. We believe that it is important knowledge for the field to understand the limits of these attacks (notably, as mentioned below in the reviewer’s next question, that AUC is perhaps lower than one might have hoped or expected). \n\n> While the paper demonstrates that strong MIAs can succeed in a controlled environment, the practical impact remains limited due to the low AUC values (<0.7).\n\nYes, as noted above,  the significance of our work to the literature is primarily in investigating the strengths and limits of state of the art MIAs on LLMs. We don’t believe low AUC values detract from our findings, but rather reflect important knowledge that can help advance the field."}]

Author rebuttal:

Meta-review: This paper provides a valuable large-scale empirical study of strong Membership Inference Attacks on moderately large language models. The reviewers praised its significant findings, such as the limited practical success of these attacks and the lack of a clear link between vulnerability and model size. The primary weaknesses noted were the limited methodological novelty, as it scales an existing attack, and the use of GPT-2 models, which may not represent today's larger architectures. Overall, the consensus is that the paper's strengths and insightful results from its comprehensive experiments outweigh these concerns, making it a solid contribution that merits acceptance.
Decision: Accept (poster)