Do Membership Inference Attacks Work on Large Language Models?

Authors: michael  duan
anshuman  suri
niloofar  mireshghallah
sewon  min
weijia  shi
luke  zettlemoyer
yulia  tsvetkov
yejin  choi
david  evans
hannaneh  hajishirzi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 05:22:38

Abstract: Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Further analyses reveal that this poor performance can be attributed to (1) the combination of a large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non-members. We also find that, when LLMs have been shown to be vulnerable to MIAs, this apparent success can be attributed to a distribution shift, e.g., members and non-members are seemingly drawn from identical domain but with different temporal ranges. Finally, we observe that existing MIAs are highly sensitive to even small changes in a sample. Such changes may cause samples that are lexically or semantically similar to members to be classified as non-members, which may be at odds with leakage that privacy auditors care about. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.

Main paper: 
 introduction: 
 Membership inference attacks (MIAs) aim to predict whether a particular record belongs to the training dataset of a given model. Thus, MIAs have great utility for privacy auditing of models (Steinke et al., 2023) , as well as investigating memorization of training data, copyright violations and test-set contamination (Shi et al., 2023; Oren et al., 2023) . While MIAs have been found to achieve high attack performance, alluding to high levels of training-data memorization (Zarifzadeh et al., 2023; Bertran et al., 2023; Lukas et al., 2023) , most analyses are limited to classifiers or LM fine-tuning (Mireshghallah et al., 2022b; Fu et al., 2023) . The performance of existing MIAs on LLMs and their pre-training data is largely unexplored. In this work, we set out to explore the challenges in evaluating membership inference attacks on LLMs, across an array of five commonly-used membership inference attacks. We introduce MIMIR 1 , a unified repository for evaluating MIAs for LMs, with implementations of several attacks from literature. We report on experiments extensively evaluating these MIAs against target models from the Pythia suite (Biderman et al., 2023b) over the Pile (Gao et al., 2020) ( §3) . For the most part, we find that the performance across most MIAs and target domains is near-random.Our further analysis suggests that the inherent characteristics of LLMs at scale-specifically, the use of massive training data and near-one epoch training ( §3.2.1)-considerably decrease current MIA performance. This suggests that the success of current MIAs in previous settings does not transfer well to attacking pre-trained LLMs seemingly due to a lack of memorization of member data. We also find that the frequent overlap between members and non-members from natural language domains considerably decreases MIA performance and raises the question of how membership should be interpreted ( §3.2.2). Notably, in several domains, non-members have high n-gram overlap with members, e.g., non-members from the Pile Wikipedia and ArXiv test samples have average 7-gram overlaps of over 30%. Notably, non-members with lower n-gram overlap are more distinguishable by existing MIAs. We also suggest that high MIA performance reported by prior work (Shi et al., 2023) is likely because non-members are chosen from the same domain as members but are temporally shifted, and these seemingly in-domain non-members likely belong to a different distribution as a result of n-gram overlap shift ( §4).Finally, building off membership ambiguity due to n-gram overlap, we discuss how the precise definition of members in standard MI may not capture important information leakage under generative text-modeling. We generate modified members preserving lexical and/or semantic similarity by altering a tiny fraction of tokens and show that existing MIAs classify them as non-members with a high degree of confidence, often more definitively than actual non-members ( §5). We encourage future work to study MI using membership definitions accounting for such fuzzy members to better understand privacy leakage. 
background: 
 The goal of an MIA is to infer whether a given data point x was part of the training dataset D for model M, by computing a membership score f (x; M). This score is then thresholded to determine a target sample's membership.MIAs are often used as a proxy to determine whether a machine-learning model leaks information related to its training data (Shokri et al., 2017; Shokri, 2022; Cummings et al., 2024) . It is the de-facto threat model when discussing machine-learning privacy (Shokri et al., 2017) , with a large array of attacks (Yeom et al., 2018; Carlini et al., 2022; Mireshghallah et al., 2022a) and defenses (Abadi et al., 2016; Tang et al., 2022; Chen et al., 2022) . More involved approaches include training shadow models (Shokri et al., 2017; Ye et al., 2022) on non-overlapping data from the target model's underlying data distribution. While attacks like LiRA (Carlini et al., 2022) show promise, they require training multiple copies of shadow models, which is often intractable for LLMs. Other stronger assumptions for MIAs include white-box access to the model (i.e., access to model parameters) or access to a ground-truth subset of member/in-distribution non-member samples for training meta-classifiers.In our setting, M is an auto-regressive language model that outputs a probability distribution of the next token given a prefix, denoted as P(x t |x 1 ...x t-1 ; M). We consider five MIAs (See Appendix A.4 for detailed descriptions):(1) LOSS (Yeom et al., 2018) -the target sample's loss under the model: f (x; M) = L(x; M).(2) Reference-based (Carlini et al., 2021) calibrates L(x; M) with respect to another reference model (M re f ) to account for the intrinsic complexity of the target sample x: f (x; M) = L(x; M) -L(x; M ref ).(3) Zlib Entropy (Carlini et al., 2021) calibrates L(x; M) with target sample x's zlib compression size: f (x; M) = L(x; M)/zlib(x).(4) Neighborhood attack (Mattern et al., 2023) -the curvature of the loss function at x, estimated by perturbing the target sequence to create n 'neighboring' samples, and comparing the loss of the target x with its neighbors x: f (x; M) = L(x; M) -1 n ∑ n i=1 L(x i ; M). (5) Min-k% Prob (Shi et al., 2023) uses the k% of tokens with the lowest likelihoods to compute a score instead of averaging over all token probabilities as with LOSS: f (x; M) = . 503 .504 .494 .508 .510 .629 .584 .627 .648 .635 .494 .489 .503 .495 .489 .502 .516 .510 .502 .485 160M .504 .515 .488 .514 .513 .638 .591 .634 .656 .638 .497 .497 .503 .498 .496 .500 .516 .504 .500 .486 1.4B .510 .544 .506 .518 .518 .656 .587 .654 .670 .650 .500 .525 .509 .502 .499 .496 .530 .505 .500 .490 2.8B .516 .565 .511 .522 .517 .707 .657 .708 .717 .698 .501 .537 .509 .503 .502 .498 .536 .502 .500 .497 6.9B .514 .571 .512 .521 .514 .672 .573 .675 .684 .654 .511 .564 .516 .512 .505 .504 .552 .508 .504 .497 12B .516 .579 .517 .524 .520 .678 .559 .683 .690 .660 .516 .582 .521 .517 .514 .506 .559 .512 .506 . 506 .481 .499 .495 .496 .492 .520 .495 .485 .481 .494 .495 .507 .497 .506 .503 .511 .508 .506 (Shokri, 2022; Shokri et al., 2017; Mireshghallah et al., 2022a) and a proxy for measuring memorization (Carlini et al., 2021; Mireshghallah et al., 2022b) , with recent attempts studying user-level leakage for the fine-tuning setting (Kandpal et al., 2023) . However, the 'extractability' of training samples has recently become synonymous with memorization and is increasingly used to compare memorization across models (Biderman et al., 2023a; Carlini et al., 2023; Tirumala et al., 2022) . Kandpal et al. (2022)  
membership inference on llms is difficult: 
 We perform a large-scale evaluation of five state-of-the-art MIAs ( §2) on a range of LLMs with up to 12B parameters and diverse benchmarks. For the reference-based attack in Table 1 and all following experiments, we use STABLELM-BASE-ALPHA-3B-V2 as the reference model (determined empirically in § 3.1). Code is available at http://github.com/iamgroot42/ mimir.Target models. We primarily target the PYTHIA model suite, including (1) five models of PYTHIA (Biderman et al., 2023b) with 160M, 1.4B, 2.8B, 6.7B, and 12B parameters2 , trained on the original Pile data (Gao et al., 2020) , and (2) five models of PYTHIA-DEDUP (Biderman et al., 2023b) with the same parameter counts as PYTHIA but trained on the deduplicated Pile data. We also experiment with the GPT-NEO family to validate our findings with a different model family, observing similar trends in most domains (see Appendix A.6).Datasets. We use seven diverse data sources included in the Pile: general web (Pile-CC), knowledge sources (Wikipedia), academic papers (PubMed Central, ArXiv), dialogues (HackerNews), and specialized-domains (DM Math, Github). We also perform experiments over the entire Pile. Members and non-members for each data source are sampled from the train and test sets of the Pile, respectively. Gao et al. (2020) decontaminated the Pile test set against the training set at a document level; nonetheless, to be more rigorous, we perform additional deduplication following Groeneveld et al. (2023) . We additionally sample from documents greater than 100 words, and truncate samples up to 200 words. Refer to Appendix A.3 for further details.Evaluation metrics. We primarily report AUC ROC for our evaluations, and additionally record TPR@low%FPR (Carlini et al., 2022) to assess performance in high-confidence settings. We visualize the 95% confidence interval for AUC ROC scores via shaded regions.MIAs perform near random. MIA performance tends to increase with the target model size (Table 1 , Figure 1 ), in agreement with prior work (Shi et al., 2023; Li et al., 2023a; Watson et al., 2022) . This is likely because larger models are more prone to overfitting the training data (Nakkiran et al., 2021) . We also find deduplication of the training data reduces MIA performance (Figure 1 ), confirming the findings from Kandpal et al. (2022) .Difficulty in Choosing a Reference Model. We ablate the choice of reference models in the reference-based attack in Appendix A.5. In summary, (1) most reference models yield poor performance, with STABLELM-BASE-ALPHA-3B-V2 being the best overall, and(2) even aggregating all reference models performs poorly. In general, we find choosing the right reference model for a target LLM challenging and largely empirical. A reference model should be trained on the data that is same-distribution but largely disjoint from the training data of the target model. However, this assumption is hard to impose at the scale of pre-training corpora; common practice is to collect all the data available on the web, leading independently collected datasets to naturally overlap with each other. We identify several key factors that may contribute to the decreased performance of MIAs on LLMs. Some factors are due to unique characteristics of practices in LLM pre-training ( §3.2.1) while others are due to inherent ambiguity in MIA ( §3.2.2).Training Data Size. Current state-of-the-art pretrained LLMs are trained with billions and trillions of tokens (Touvron et al., 2023a; b; Team, 2023) . We hypothesize the large pretraining corpora characteristic to LMs decreases MIA performance, as larger pretraining datasets lead to better generalization (Hoffmann et al., 2022; Muennighoff et al., 2023) .We employ the PYTHIA-DEDUP model suite's intermediate checkpoints to assess the impact of different amounts of training data. While keeping non-members fixed, we sample members for each checkpoint from its most recent 100 steps to remove the impact of the recency bias of the members.4 See Appendix A.3.1 for details.MIA performance generally starts as near-random, then rapidly increases within the next few thousand steps, before decreasing across successive checkpoints5 (Figure 2 , left). We speculate the initial low performance is due to the model warming up in training, with high losses across both member and non-member samples. We believe the following rapid rise and then gradual decline in performance are because the data-to-parameter-count ratio is smaller early in training and the model may tend to overfit (Yeom et al., 2018) , but generalizes better as training progresses, in line with observations in existing work (Nakkiran et al., 2021) .It is standard practice to pre-train LLMs for around one epoch, given the scale of data and their tendency to overfit quickly (Muennighoff et al., 2023; Komatsuzaki, 2019) . Previous MIA works that demonstrate attack effectiveness consider supervised fine-tuning or masked LM pre-training (Nakamura et al., 2020; Lehman et al., 2021; Mireshghallah et al., 2022a; b) , where models are trained for more than 10 epochs. We explore how the near-one epoch training of LLMs leads to decreased MIA performance.To verify this hypothesis, we perform MIA against the Datablations suite (Muennighoff et al., 2023) , consisting of models trained on subsets of C4 (Raffel et al., 2019) train data for varying numbers of epochs (see Appendix A.2 for model details). We also experiment with SILO (Min et al., 2023) ; see Appendix C.2 for SILO results.Increasing the number of effective epochs corresponds to an increase in attack performance (Figure 2 , right). While Muennighoff et al. (2023) shows training for multiple epochs helps improve performance, our results suggest that such multi-epoch training (and/or large upsampling factors) can increase training data leakage.Natural language documents commonly have repeating text-even with the best efforts in decontamination and deduplication. These include common phrasings and quotes, natural use of similar texts, and syntactical similarities inherent to specific domains. This leads to substantial text overlap between members and non-members, which motivates the following hypothesis: higher overlap between members and non-members increases MIA difficulty.We quantify overlap using the percentage of n-gram overlap, defined as6 : For a non-member sample x consisting of m words such that x = x 1 x 2 ...x m and an n-gram in x defined as a continuous substring x i ...x i+n-1 , the n-gram overlap of x on training dataset D is1 m -n + 1 m-n+1 ∑ i=1 1{∃y ∈ D : x i ...x i+n-1 ∈ y}For a given non-member sample, this gives us the percentage of n-grams in the non-member that can be found in at least one member sample. We first compute the percentage of 7-gram overlap for non-members against the entire Pile training (member) set (Figure 3 )7 ; see Appendix B.1 for implementation details. We observe high n-gram overlap with training data for a substantial portion of non-members; e.g., the Wikipedia, ArXiv, and PubMed Central domains have average 7-gram overlaps of 32.5%, 39.3%, and 41.0%, respectively. Domains such as GitHub, DM Mathematics, and FreeLaw see even higher overlap, with mean 7-gram overlap of 76.9%, 72.8%, and 62.3%, respectively.High n-gram overlap suggests that substrings of non-members may be seen exactly during training, which makes the distinction between members and non-members even less clear. To verify our hypothesis, we resample non-members ensuring ≤ 20% n-gram overlap with members, and report MIA performance ( eliminate instances of non-member records that may overlap with training records, it also introduces an explicit drift between member and non-member distributions by selecting non-members that are most "unlike" training records. We clarify that this step is not a suggestion for researchers to alter their benchmarks; such a processing step drifts away from the standard membership inference game (Yeom et al., 2018) .Results. MIAs perform significantly better as the non-member distribution concentrates towards lower n-gram overlap and further diverges from the natural n-gram overlap distribution of non-members from the training distribution e.g., .516→.666 in Wikipedia, .690→.908 in Github, and .512→.792 in PubMed Central for various attacks. This is intuitive as the target model is likely to assign a lower likelihood to non-members further from its training data, making members and non-members more distinguishable. Note that decreasing the n-gram overlap threshold, especially for smaller n, pushes the setting closer to distribution inference (Suri & Evans, 2022) , since the distributions of 'member' and 'nonmember' records are no longer the same. We further discuss outlier behavior in Appendix B.We note that n-gram overlap is an intrinsic property of natural language rather than a problem of the Pile train-test split. These splits are already deduplicated at a document level, following standard practice in decontamination (Gao et al., 2020; Brown et al., 2020) . Nonetheless, repeating texts across distinct documents are fundamental and natural properties of domain data. We also note that n-gram overlap distribution analysis can help assess how representative of a target domain a set of candidate non-members is when constructing MIA benchmarks. Ultimately, we highlight the need to consider qualities of the data domain, e.g., n-gram overlap, and understand their potential impact on MIA performance. 
importance of candidate set selection: 
 In contrast to our findings in §3, recent works report state-of-the-art MIAs achieving > .7 AUC ROC on pretrained LLMs (Shi et al., 2023; Meeus et al., 2023) . We investigate how the non-member candidate selection methods in these works can result in an inherent but likely unintended distribution shift between members and non-members as one such reason for the observed performance differences.Experimental Setup. Prior work distinguishes members and non-members of a target domain based on the knowledge cutoff date of the target model, with members coming before and non-members coming after the cutoff. We construct similar experimental settings under two domains also used in earlier works: Wikipedia and ArXiv.We follow the same setup as §3 on the Wikipedia domain, but replace the non-member set with samples from the entire RealTimeData WikiText dataset (Li et al., 2023c) . The Pile members are sampled from articles in a Wikipedia dump from before March 2020 (Gao et al., 2020) , whereas non-members consist of Wikipedia articles created from August 2023 and onwards. For the ArXiv domain or further details on Wikipedia, see Appendix A.3.2. Results. Table 3 demonstrates that the temporally shifted settings yield MIA performances significantly higher than when members and non-members are from the same temporal range. Figure 6 (Appendix) also demonstrates that MIA performance generally increases as non-members are further temporally shifted from member data. We speculate this follows from changes in language such as the introduction of new terminology and ideas over time.Temporal Shift as Change in n-gram Overlap. We interpret temporal shift as a change in n-gram overlap distribution between the original and temporally shifted non-members. Figure 4 demonstrates that the distribution of 7-gram overlap of such shifted data concentrates at lower overlap percentages compared to their natural counterparts. The natural Wikipedia non-members have an average 7-gram overlap of 39.3%, whereas for the temporally shifted Wikipedia non-members it is 13.9%. See Appendix B.4 for temporal ArXiv discussion.In general, when aiming to assess MIA performance, we advise estimating how representative a sample non-member set is of the member domain by comparing its n-gram overlap distribution with that of a left-out sample set from the pretraining corpora. Particularly, if the distribution of the candidate set is noticeably shifted towards lower n-gram overlap compared to the left-out member sample set, the candidate non-member set may not be representative of the member distribution from the target domain and potentially high MIA performances should be carefully examined. Closer inspection (Table 5 , Appendix) reveals the extent of such over-estimation; decision thresholds derived using temporally-shifted non-members end up testing for temporal shift rather than membership. We note that distinguishing between members and temporally shifted non-members is a realistic inference game with practical implications but differs from the classical MI game as temporally-shifted data may belong to a different distribution. Furthermore, as we aren't able to reproduce the experimental settings of prior works (Shi et al., 2023; Meeus et al., 2023) for distributional shift analysis, it is inconclusive if their differing results are solely due to the temporal shift between member/non-member samples (see Appendix A.3.2 for details). 
revisiting membership: 
 The discussion of high n-gram overlap of non-members with members raises the question of whether exact membership is always useful concerning information leakage. The definition of membership in the standard MI game treats only records seen exactly during training as members, e.g., for language models, substrings appearing exactly in the training corpus. However, this may be at odds with what adversaries and privacy auditors care about when concerning information leakage. For generative models especially, guessing the membership of some sample via other sufficiently close samples can be useful. For example, any paraphrase of "Product launch in Q2, 2025" may be relevant as long as it preserves information regarding the product launch timeline, even if the paraphrase has a significant lexical difference, e.g., "Launching product in Q2, 2025" or "Q2 2025 release". Note that standard notions of Differential Privacy (Dwork et al., 2006) do not immediately protect against such cases, since records being tested for membership are not, in the literal sense, members. We explore two methods of constructing such "sufficiently close" samples as an initial step in studying approximate membership definitions.Lexical Distance. We first experiment with creating modified member samples by replacing n random tokens in a given sample with tokens randomly sampled from the model's vocabulary. We do so for n = {1, 10, 25} (20 trials per n) and visualize the distribution for MIA scores using LOSS and Reference-based attacks (Figure 5 , top). The LOSS attack yields distinct loss distributions between the modified members and original member/nonmembers, suggesting that the model is sensitive to out-of-place random tokens even for lightly perturbed member samples. The Reference-based attack, on the other hand, has a distribution of modified members much closer to both members and non-members, likely due to the reference model calibration accounting for the complexity introduced by the random tokens. This further reinforces the ambiguity of such samples-should they be considered members or non-members?We also compute the thresholds corresponding to certain FPRs for actual member and non-member data and use these thresholds to compute the FPR on the modified members.We consider these modified members as "non-members", which they are with regards to exact match8 . Table 4 (left) shows that these modified members have extremely low FPRs for edit distance as low as n = 1, suggesting that these samples would be classified as non-members by the MIA, even though from the perspective of information leakage such a sample is effectively a member. We highlight the need to rethink membership for samples with extremely low lexical distance from actual training members, though even membership at higher lexical distances is important with respect to what information is still leaked in the unperturbed portions.Semantic Distance. While a small edit distance suggests closeness in meaning, a higher edit distance does not necessarily imply loss of semantics. We compute MIA scores for neighbors generated for member samples as part of the Neighborhood attack (see Appendix A.4) for the Wikipedia and ArXiv benchmarks and repeat the above pipeline. Visualizing the scores shows how the modified members are not too far from original member score distributions, especially for the Reference attack (Figure 5 , bottom). We repeat the same FPR experiment as edit-distance-based modified members. While the FPR for these semantically close records is noticeably higher than records close by edit distance, the false positive rates are still low (Table 4 , right). Again, these results suggest that semantically close members would be classified as non-members even though they may be as useful as actual members depending on the inference goal and the semantic information preserved.While it is not surprising that semantically close neighbors have MIA scores more similar to actual member samples than randomly-replaced tokens, it is clear that an ideal distance function should combine the benefits of lexical distance and semantics in defining a membership neighborhood. Such observations also motivate a fully semantic MI game, where a neighbor member may be defined by its proximity to an exact member in a semantic embedding space. This may provide a clearer interpretation of knowledge leakage than lexical matching, especially when samples naturally have high lexical (i.e., n-gram) overlap. 
conclusion: 
 We shed light on the difficulty of membership inference against LLMs from the lens of an adversary. Our results suggest two possibilities: (1) data does not leave much of an imprint, owing to characteristics of the pre-training process at scale, such as large datasets and singleepoch training, and (2) the similarity between in and out members (which we demonstrate via n-gram overlap), coupled with huge datasets, makes this distinction fuzzy. While MIA performance could improve via stronger attacks (Casper et al., 2024) , the second factor requires rethinking the membership game itself. Our empirical results suggest that both of these might be confounding factors while measuring leakage. The membership inference game needs to be extended for such generative models to better align with information leakage that adversaries and auditors may care about, such as user-level leakage (Kandpal et al., 2023) and PII (Lukas et al., 2023) . In the meanwhile, special care should thus be taken to avoid unintentional distributional shifts while constructing non-members for MIA benchmark construction.the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein. 
 
 Appendix: We release our codebase as a Python package (available at http://github.com/iamgroot42/ mimir), complete with documentation and tests, for implementing and evaluating membership inference attacks for language models. The data used in our experiments is also available via HuggingFace (https://huggingface.co/datasets/iamgroot42/mimir). Some features of the package include:• A base attack class that provides bare-bones code and helper functions that can be easily used in implementations of both existing and new attacks. • Data processing utilities to filter and cache data for membership evaluation, from both provided and available sources. • Support for a vast array of models that can be used as target or reference models.The entire codebase works with modular configuration files, allowing multiple experiments to be run simultaneously with no edits to the code itself. We used Python 3.9.7 for our experiments, with PyTorch 2.0.1. Our experiments were executed on a mix of machines, with GPUs ranging from RTX6k to A100.PYTHIA-DEDUP. Both the PYTHIA and PYTHIA-DEDUP model suites provide intermediate checkpoints for each model. For experiments targeting the PYTHIA-DEDUP model, as the PYTHIA-DEDUP model is trained for greater than 1 epoch, we select the checkpoint that most closely matches the one epoch mark over the deduplicated Pile. We decide this is checkpoint 'step99000'. For experiments targeting the non-deduped PYTHIA models, we use the final checkpoint, which sees just under one (≈ 0.9) epoch of the original Pile.The models from the SILO suite (Min et al., 2023) consist of 1.3B-parameter transformer LMs based on the OpenLM implementation of the LLaMA architecture (Touvron et al., 2023a) . These are trained for multiple epochs on the Open License Corpus, which consists of permissively-licensed text data classified as either public domain (PD) texts, permissively licensed software (SW), or under an attribution license (BY). We target the SILO-PDSW model (alongside its intermediate checkpoints) trained on only texts classified as PD or SW for domains contributing less than 5% of the data upsampled by a factor of 3x (which includes HackerNews and DM Mathematics).The DATABLATIONS suite (Muennighoff et al., 2023) is a large collection of models trained to study scaling laws in data-constrained regimes. They vary in the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameters. For the epoch experiment, we choose the 2.8B-parameter subset of models, with each seeing a total of 55B tokens from the C4 dataset across their training runs. These models vary in the number of epochs their training subset is seen, ranging from one to 14 epochs. They also offer a model trained for 44 epochs, which we decided to leave out of evaluation.GPT-NEO. is a collection of 125M-, 1.3B-, and 2.7B-parameter models of similar architecture to the GPT-3 model family. These models are trained on the Pile for about 300B tokens, similar to the PYTHIA suite. This model suite is a precursor to the GPT-NEOX (Andonian et al., 2023) model architecture, which PYTHIA-DEDUP and PYTHIA are built on. Noticeable differences include the tokenizer used per model suite, with the GPT-NEOX allocating additional tokens to whitespace characters, as well as intended training settings, with GPT-NEO geared towards TPU training and GPT-NEOX GPU training.OLMO. The OLMO model suite (Groeneveld et al., 2024) is a suite of open language models trained on the DOLMA (Soldaini et al., 2023) dataset. OLMO models currently available include 1B-and 7B-parameter variants trained on 3T and 2.5T tokens, respectively. While our preliminary results just target the final checkpoints, the OLMO suite is similar to the PYTHIA suite in that intermediate checkpoints and exact training order are fully open.We sample 1,000 members and non-members from each target domain from the Pile train and test sets, respectively. We do the same for the aggregate Pile experiment, except we sample 10,000 members and non-members each from the complete Pile train and test sets. We sample documents greater than 100 words and truncate them up to 200 words from the beginning to create our benchmark examples. Previous work (Shi et al., 2023) observes that sample length correlates with performance, so we bound the sample length to reduce its impact while picking a reasonable threshold so that our samples are likely to contain ample signal. While further increasing the length of samples could yield greater MIA performance, such an experiment is orthogonal to the ones we conduct; inherent differences in LLM training and MI evaluation would still impact evaluation on longer texts.We follow the same pipeline when generating the benchmark for targeting the DATABLA-TIONS models, picking members and non-members from the C4 train and validation sets, respectively.For our additional decontamination on Pile benchmarks, we follow Groeneveld et al. (2023) , which uses a bloom filter to check for n-gram inclusion. We keep the default filtering settings of n = 13 and a threshold of ≤ 80% overlap. Further details about setting up the bloom filter can be found in Appendix B.1For each model, we pick checkpoints every 5000 steps ending at step 95000, with each step corresponding to 1024 samples of length 2048 tokens. We also include checkpoints at step 1000 and step 99000, the closest checkpoint to the step where one full epoch of the deduplicated Pile was seen. For each checkpoint, we use the same non-member set for evaluation consisting of 1000 samples sampled from the entire Pile test set. We then construct a member set for each checkpoint9 : for the checkpoint at step n, we sample 1000 random samples from documents seen within the range step {n -100, n}. EleutherAI provides random seeding for deterministic training data order across the PYTHIA-DEDUP training runs, which we use to determine the seen document order. This allows us to determine which documents to sample from for a given step range. For both members and non-members, we sample with the same criterion as the general experiments above.For the temporal Wikipedia benchmark non-members, we collect samples from the Real-TimeData "wikitext latest" dataset (Li et al., 2023c) . This yielded Wikipedia articles created between the week of August 12, 2023 till the week of January 8, 202410 . We then follow Pile processing steps by simply appending the article titles to the front of each respective article with a "\n\n". Members are sampled from the Wikipedia subdomain of the Pile training set. Members and non-members are then sampled with the same criterion as in the general experiments.For the temporal ArXiv benchmarks, the member set for each benchmark is fixed and sampled from the ArXiv subdomain of the Pile training set, which consists of papers posted prior to July 2020 (Gao et al., 2020) . For non-members, we use the ArXiv API again following We then apply the same processing steps used in the Pile (Gao et al., 2020) . This mainly involves converting the latex sources for a given preprint into a single Markdown file, and then filtering out documents such as those with conversion errors. For each month range, we sample non-members from processed files in the given date range. By sampling non-members from successively later time ranges after the Pile ArXiv cutoff date, we also seek to explore how greater temporal shift impacts MIA performance. We again sample both members and non-members with the same criterion as in the general experiments.Difficulties in reproducing and analyzing existing works. While we follow a similar method of non-member candidate selection for our temporal experiments as prior works such as Shi et al. (2023) and Meeus et al. (2023) , we were unable to reproduce their settings for analysis for two main reasons: 1) their exact non-member candidates and/or pipelines to reproduce the non-member candidate selection are unavailable or 2) target models used such as LLAMA (Touvron et al., 2023a) do not release their training data to conduct n-gram or other distributional analyses.MIAs consider a target model M, which outputs a probability distribution of the next token given a prefix, denoted as P(x t |x 1 ...x t-1 ; M). Their goal is to model f (x; M), which outputs a score for target sample x = x 1 ...x n with n tokens. This score is then thresholded to determine the target sample's membership in the training data of M.LOSS (Yeom et al., 2018; Carlini et al., 2019) considers the model's computed loss over the target sample: f (x; M) = L(x; M).Reference-based (Sablayrolles et al., 2019; Watson et al., 2022) Table 5 : FPR (%) on non-members from the Pile (original; not temporally shifted) on various attacks when using a score threshold that achieves a 1, 5, or 10% FPR on the temporallyshifted ArXiv (for varying levels of temporal shift) and Wikipedia benchmarks. The target model is PYTHIA-DEDUP-12B. FPRs on the original non-members are much higher then the thresholded FPR on the temporally shifted benchmarks, indicating that such thresholds may be moreso classifying temporal shift rather than member and non-members.Empirically, using an LM that is different from M has been a reasonable choice and was used in prior work (Kandpal et al., 2022; Watson et al., 2022) . The attack considers the membership score of the target sample by M relative to the membership by M ref to calibrate the target model's score given a difficulty estimate through the reference model's score, with goals to improve precision and reduce the false negative rate. For our experiments, we use LOSS as the uncalibrated membership score such that, for the reference-based attacks,f (x; M) = L(x; M) -L(x; M ref ).This method exactly follows the method from Watson et al. (2022) and is also largely similar to the offline Likelihood Ratio attack (LiRA; Carlini et al. ( 2022)), although LiRA uses many reference models (often trained shadow models).Zlib Entropy (Carlini et al., 2021) functions similarly to reference-based MIA, using the zlib compression size of a sample x as a local difficulty threshold per sample:f (x; M) = L(x; M) zlib(x) ,where zlib(x) is the length in bytes of the zlib compressed sample.Neighborhood Attack (Mattern et al., 2023) assumes access to a masking model, and operates by generating "neighbor" texts x to a given text sequence x by using the masking model to replace a percentage of randomly selected token spans while still maximizing the neighbor's likelihood. If the sample's loss is considerably lower than the neighbor's losses, the difference is attributed to the target model overfitting the sample, and the sample is considered a training member. Formally, we havef (x; M) = L(x; M) - 1 n n ∑ i=1 L(x i ; M).We use BERT (Devlin et al., 2019) as our masking model of choice, with a masking percentage of 5%.Min-k% Prob (Shi et al., 2023) is based on the intuition that non-member examples tend to have more tokens assigned lower likelihoods than member examples do. Given sample x = x 1 , ..., x n and hyperparameter k, let min-k(x) be the set formed by the k% of tokens in x with minimum likelihood. We then havef (x; M) = 1 |min-k(x)| ∑ x i ∈min-k(x) -log(p(x i | x 1 , ..., x i-1 )).We experiment with multiple different k ∈ {10, 20, 30, 40, 50} as suggested in Shi et al. (2023) , but settle on k = 20 for our experiments.We compute the performance of each attack based on 1,000 bootstrap samples of the benchmark and report the average AUC ROC and TPR@low%FPR over the bootstraps.While many recent works study MIAs that perform membership classification through meta-classifiers trained on features extracted from a ground-truth subset of member/non-member data, we primarily focus on blackbox attacks as access to such a subset can be difficult to guarantee in practice, especially as the inclusion of samples in pre-training corpora becomes more ambiguous as these corpora continue to expand.We choose a diverse set of reference models to experiment with. For the aggregate method over all reference models, we take the average of the scores per reference model for a target sample12 . We report results for our complete ablation on reference model choice in Table 6 . GPT-2 (Radford et al., 2019) is suite of pre-trained transformer trained on a large dataset of around 40GB of web text, likely overlapping with the Pile. We use the GPT-2-small variant with 124M parameters. DISTILGPT2 (Sanh et al., 2019 ) is a smaller 82M-parameter model trained with the supervision of GPT-2-small using knowledge distillation.OPT (Zhang et al., 2022) is a suite of open-sourced pre-trained transformers that are trained on a curated pre-training corpus including several datasets from the Pile, such as Wikipedia, DM Mathematics, and HackerNews. We use the 1.3B-parameter variant.As mentioned in Appendix A.2, GPT-NEO (Black et al., 2021) is another suite of pre-trained transformers designed using EleutherAI's replication of the GPT-3 architecture. These models are trained on the full Pile for a similar amount of tokens as PYTHIA (∼ 300B), though the data seen may not necessarily be in the same order as the PYTHIA models. We use the 1.3B-parameter variant. SILO-PDSWBY (Min et al., 2023 ) is a 1.4B-parameter transformer pre-trained on all types of permissively licensed data in the Open License Corpus. The training data consists of certain Pile domains such as HackerNews and DM Mathematics.LLAMA (Touvron et al., 2023a ) is a collection of large, open-sourced pre-trained LMs ranging in size from 7B to 65B parameters. The pre-training corpus is on the scale of trillions of tokens, much larger than the Pile, and likely has significant overlap with the Pile. We use the 7B-parameter variant. STABLELM-ALPHA-V2 (Tow, 2023) is a set of open-source pre-trained LMs also trained on a large pre-training corpus with trillions of tokens. Training is conducted in two stages, with the first stage seeing 1 trillion tokens of a mixture of data from sources such as RedPajama (Together AI, 2023) and the Pile, with an emphasis on refined web text. The second stage is trained on 100 billion tokens with a higher context length, increasingly sampling naturally long texts and adding the StarCoder (Li et al., 2023b) dataset. We use the 3B-parameter variant.We also experiment with the non-deduped PYTHIA-DEDUP-1.4B model as a reference model to see how using a smaller version of the target model (same architecture and training data order) impacts reference-based attack performance (Carlini et al., 2021) .We speculate that the slightly higher performance with STABLELM-BASE-ALPHA-3B-V2 as the reference model, even though its pre-training corpus has high overlap with the Pile, is because 1) larger target models13 such as the PYTHIA-DEDUP-12B model may considerably overfit certain member samples and 2) the STABLELM-BASE-ALPHA-3B-V2 is trained on a much larger corpus, which helps it generalize well and achieve similar losses as the target model on the non-member data. As a result, member samples are more likely to have a greater magnitude of difference between the target and reference model losses compared to the difference between losses on non-members.We repeat our experiments with the GPT-NEO family of models. Table 7 demonstrates similar trends targeting the GPT-NEO models as seen when targeting the PYTHIA model family (Table 1 , Table 12 ), such as MIA performance generally increasing as the target model size increases. In general, performance against the GPT-NEO models is similar, if not lower than, performance against the PYTHIA-DEDUP and PYTHIA models when comparing similarly sized variants. In some domains such as HackerNews, the best performing MIA differs between target models (Min-k% Prob for GPT-NEO, reference-based for PYTHIA-DEDUP), though marginally.We perform preliminary experiments targeting the OLMO family of models. Benchmark construction is similar to what is detailed in Appendix A.3. However, we sample from domains that make up DOLMA (Soldaini et al., 2023) , namely Wikipedia, C4, Reddit, Common Crawl, and peS2o (Soldaini & Lo, 2023) . These are similar to domains used in Pile. We note that peS2o consists of both abstracts (s2ag) and full papers (s2orc), and evaluate them as separate domains. To get non-member, we use held-out DOLMA data from PALOMA (Magnusson et al., 2023) . for the 7B-parameter model), performance across different model sizes begins to converge to near-random performance even with such distinct model sizes. Interestingly, the Referencebased attack using STABLELM-BASE-ALPHA-3B-V2 performs much worse than when used to calibrate the PYTHIA models, reinforcing the difficulty in finding suitable reference models for different LLMs. We also observe many settings where MIA performance is considerably less than .5, suggesting that the MIAs are more likely to predict members as non-members, and vice versa. More investigation is needed to understand such behaviors on specific domains such as s2ag from peS2o in the DOLMA data. We create a bloom filter following Groeneveld et al. (2023) . Due to the scale of the Pile training data and limited memory, we shard the bloom filter. In our construction, we split the training data in half, resulting in two bloom filter shards. Since each shard only sees half of the training data, to check for n-gram inclusion across the entire Pile, we check for containment in both of the sharded bloom filters, counting an n-gram included only if it is included in at least one of the bloom filters.For each shard, we configure the bloom filter according to the data size such that the false positive rate of the bloom filter is less than 1% (0.6%). Then, for each document, we tokenize at the word level. We then add n-grams to the filter by using a striding window over n words at a time with a stride of 1. We use the same method of gathering n-grams when checking the non-members for n-gram overlap.Table 2 shows that, interestingly, reference-based MIAs have a noticeably smaller increase in performance compared to non-referenced-based MIAs for domains such as GitHub or PubMed Central under n-gram overlap thresholding. We speculate that, since numerous low n-gram overlap non-members are outliers to the relevant domain, these non-members will also be outliers to the similar/overlapping data seen by the reference model. As a result, even though these non-members may yield higher losses from the target model, we see similar high losses for the reference model as well, which makes the difference between target and reference model loss for non-members and members relatively less distinguishable compared to signals from the other attacks.At the same time, domains like Pile CC do not see this dampened performance, likely because the 20% threshold in the case of Pile CC is not sufficient to select outliers, as samples from this domain have naturally low n-gram overlap. Another case where the referencebased attack seems to avoid this observation in the temporally shifted non-member setting for both Wikipedia and ArXiv despite the temporally shifted non-members being more out-of-distribution relative to the Pile Wikipedia and ArXiv distributions, respectively. We speculate this is due to the reference model of choice, STABLELM-BASE-ALPHA-3B-V2, which has not only been trained on a corpus with high overlap with the Pile, but also trained on datasets that capture more recent data such as RedPajama-Data-1T (Together AI, 2023) which contains Wikipedia and ArXiv samples from a much more recent cutoff date (i.e., RedPajama uses the 2023-03-20 Wikipedia dump), allowing it to generalize better over the temporally shifted non-members and avoiding a shift towards higher losses that weaker or older reference models might experience.Attacks like LOSS or Min-k% Prob do not utilize any external signal or difficulty calibration, and thus rely exclusively on signals from the target model for member classification.Calibration-based methods like zlib and reference-based attacks, on the other hand, account for the inherent "difficulty" of a seen sample. Thus, in situations where the non-member data is significantly out of domain, even for a reference model or calibration method, it is likely that the signals from the target model and difficulty calibration would cancel out, leading to a weakened MIA signal. On the other hand, difficulty calibration can further boost MIA signal in settings where the member data is inherently more likely to be memorized, such as in §3.2.1 where reference-based attacks yielded considerably higher MIA performance in low training data size and high effective epoch count settings, with performance being further amplified in the extremes of both settings. Thus, MIA baselines for new MIAs should include both kinds of methods: calibration-based and calibration-free.Having baseline coverage for both styles of MIA can help uncover inherent characteristics of the evaluation setting such as unintentional member/non-member distributional shift or overfitted target models that influence MIA performance and also paints a holistic picture with regards to what MIAs are most suitable for specific attack settings. As seen in Table 1 , MIA performance in the Github domain even without thresholding is notably higher than that in other domains, with the best method (zlib) achieving an AUC ROC of ∼ .70. We speculate this is not because the GitHub domain, or code in general, is an easier domain to attack, but because the presumably reasonable decontamination threshold of ≤80% 13-gram overlap threshold only captures a small percentile of non-members as GitHub is naturally very high overlap.We speculate a large factor contributing to the high overlap is the repetitive nature of code, such as copyright notices, function definitions, and syntax like HTML tags. Figure 7 demonstrates how our decontamination threshold impacts the 7-gram distribution of non-members. Non-members under our decontamination threshold are more likely outliers to the GitHub domain (see Figure 15 for an example of such an outlier). The additional n-gram overlap threshold experiments ( §3.2.2) only exacerbate the impact of thresholding, which leads to notably higher MIA performance.Such observations indicate why lexical non-member boundaries may lead to ambiguous interpretations of MIA performance in high-overlap domains. Here, using semantic differences between samples to draw non-member boundaries may be key to better understanding membership leakage in such domains. Figure 8 : Distribution of n-gram overlap for non-member ArXiv preprints sampled from the months 2020-08 and 2023-06, respectively. We also plot the n-gram overlap distribution of the original Pile ArXiv non-members. Between the original non-members and both temporally shifted non-member sets, the temporally shifted non-member n-gram overlap distributions are considerably more concentrated at lower % n-gram overlap. The original non-members have an average 7-gram overlap of 39.3%, while non-members from months 2020-08 and 2023-06 have 7-gram overlap of 22.7% and 20.5%, respectively.Figure 8 reinforces our observations in Figure 4 , as similar to the temporal Wikipedia setting, temporally shifted non-members from after the target model's knowledge cutoff date are concentrated at considerably lower % n-gram overlap than non-members from the natural ArXiv non-member set. This contributes to the greater MIA performance in general over the temporally shifted ArXiv benchmarks. However, we note that n-gram overlap distribution shift does not provide a strong interpretation for the increase in MIA performance as nonmembers are increasingly temporally shifted. For example, the average 7-gram overlap of non-members from the month 2020-08 is 22.7% while the average for the month of 2023-06 is 20.5%. While there is a small decrease in average 7-gram for later non-members, the change is quite small and doesn't clearly justify the considerable difference in MIA performance when evaluating on benchmarks using non-members from the different months (i.e., .723 → .795 AUC ROC from the 2020-08 benchmark to the 2023-06 benchmark). We speculate other factors that contribute to this increase include changes in the distribution of topics (i.e., increasing popularity of research into LLMs) and the presence of specific identifying tokens (i.e., dates, references, new terminology). We believe such factors only further reinforce the need to carefully analyze MIA benchmark construction when evaluating MIAs to understand what signals are truly being captured.We explore how the recency of member samples seen in training impacts MIA performance. We follow the same setup as the training data experiment (A.3.1), but instead of evaluating the checkpoint at step n with the member data sampled from within steps {n -100, n} and the fixed non-member set, we fix the target model, only targeting the checkpoint at step 99000 for all the benchmarks. Performance on benchmarks with more recently seen members is higher, but gradually decreases to a plateau for less recently seen members.Figure 9 demonstrates that, in general, member data seen more recently by the given checkpoint contributes to slightly higher MIA performance. We believe this supports existing work in LM forgetting (Jagielski et al., 2023) , where observed patterns in recently seen training data are better preserved in the model parameters, while earlier seen data are less memorized.We also note that the MIA performance trajectories seem to drop slightly more quickly for smaller models, though the trajectories across all model sizes seem to converge when evaluating on member data from much earlier in the training run. We speculate this is a result of larger models having more parameters, allowing them to capture more seen data before having to drop older knowledge.We also note that, in the context of MIA against fine-tuning datasets, our results indicate that data seen during fine-tuning or continued pre-training may also be increasingly vulnerable due to how recent they are seen. This aligns with previous work demonstrating high MIA performance on fine-tuning datasets (Mireshghallah et al., 2022a; Fu et al., 2023) . This is especially relevant in practice since fine-tuning is a popular option to re-purpose large pre-trained models for varying downstream tasks such as commercial use cases, which often involves tuning with sensitive data.While experiments against the DATABLATIONS model (Figure 2 , right) operate in a fixed training data size setting, we also explore a more realistic setting where the amount of training data the target model sees increases alongside the effective epoch count by targeting the SILO-PDSW model and intermediate checkpoints. In Figure 10 , for HackerNews, we observe MIA performance initially increases with more effective epochs, similar to the DATABLATIONS setting, but then begins to plateau or drop as effective epoch count continues to increase. DM mathematics, on the other hand, surprisingly decreases as the number of effective epochs increases. We speculate over factors that may contribute to these observations:• HackerNews, even when up-sampled for this variant of the SILO-PDSW model, still only makes up 5.9% of the training data (Min et al., 2023) . In the first few epochs, when the total training data seen is low, the model can memorize the HackerNews samples. However, as the number of epochs increases, the target model may tend to overfit data more so from domains with greater representation. As the SILO model is on the smaller side with 1.4B parameters, we suspect the target model begins to memorize less of the HackerNews samples, leading to a plateau or drop in MIA performance.• DM Mathematics also makes up only 3.5% of the training data. In addition, with DM Mathematics being a dataset of mathematical problems, we suspect that the abundance of tokens from a concentrated token space (i.e., digits, variables) that are largely symbolic rather than semantic makes memorization of specific samples unlikely. Overall, it simply fails to perform well on such data even after multiple epochs (as observed when looking at model loss values for this data).For both cases, further investigation is needed into the target domains and attack setting setup to better understand these counter-intuitive phenomena.We follow the same pipeline as §5 but focus on generating member paraphrases that are semantically similar while preserving as much specific information from the original member sample as possible. To do so, we follow Yang et al. (2023) and prompt GPT4 (OpenAI et al., 2024) to paraphrase member samples in a different style (5 trials per member). We perform this paraphrasing over the ArXiv, Wikipedia, and HackerNews domains; see Table 9 for domain-specific prompts. In general, we don't focus on the lexical similarity of the paraphrases unlike the earlier semantically similar samples generated via masking and replacing a small percentage of tokens. However, with HackerNews, we do specify a comment structure different from how HackerNews records were formatted for model training, a noticeable lexical difference.ArXiv Please help me paraphrase the following text chunk from a research paper in a different style. Importantly, for sentences containing specific details like mathematical definitions or proofs, only make minimal changes and ensure these details are included exactly in the paraphrase. If the paper includes a title or authors, please keep them in the rephrase. If not, please DO NOT make up a title. Use a similar number of words.Please help me paraphrase the following text chunk from Wikipedia in a different but concise style. Importantly, for sentences containing specific details, make minimal changes and ensure all details are included correctly in the paraphrase. Use a similar number of words.Please help me paraphrase the following conversation chunk from a thread in HackerNews while maintaining the conversational style. Follow this structure for each comment in the thread: [user] - [comment] . Ensure all user's comments are represented in the paraphrase. Make sure all details in each user's comments are included correctly in the paraphrase, such as links. Be specific and don't generalize.Table 9 : Instructions used to prompt GPT4 to obtain paraphrased members.We again visualize the score distributions between the paraphrased members, and original members and non-members (Figure 11 ). We observe in general across both the LOSS and Reference-based attacks over the three domains that the paraphrased member score distributions are distinguishable from the original member and non-member score distributions but have noticeable overlap, similar to what was observed with masking-based semantic neighbors. However, when we perform the FPR experiment (Table 10 ), we see that in highconfidence settings, the paraphrased members are likely to be classified as non-members.Both the LOSS and Reference-based attacks seem noticeably insensitive to such paraphrased neighbors. 0.8 0.9 1.0 1.0 0.5 13.1 7.8 12.6 13.0 11.0 0.7 1.0 0.4 0.4 0.1 0.7 1.0 0.9 0.5 0.6 160M 1.1 0.8 1.2 1.4 1.3 13.5 4.6 12.3 14.7 5.9 0.4 0.8 0.5 0.4 0.4 0.7 0.9 1.0 0.3 0.1 1.4B 0.6 0.9 0.5 0.7 0.4 12.8 0.7 12.9 16.4 3.9 0.6 0.6 0.5 0.7 0.8 0.4 0.7 0.6 0.5 0.1 2.8B 0.6 0.8 0.5 0.7 0.9 20.8 4.5 20.8 23.4 11.1 0.6 0.5 0.7 0.8 0.9 0.4 1.0 1.4 0.6 0.9 6.9B 0.6 0.6 0.4 0.6 0.5 12.9 0.6 13.1 16.8 6.1 1.0 1.4 1.2 1.3 1.0 0.8 1.6 0.8 0.3 0.8 12B 0.7 0.6 0.6 0.7 1.0 13.9 0.8 14.2 17.4 4.9 1.0 1.7 1.1 1.5 1.0 1.0 1.5 1.3 0.7 0.9 0.6 1.8 0.6 0.6 0.6 0.9 0.2 0.6 1.0 0.7 .9 1.9 1.0 0.9 1.3 2.6 1.8 2.5 2.5 2.2 12B 0.6 2.5 0.6 0.5 0.9 1.0 0.5 0.5 0.9 0.8 0.7 2.3 0.8 0.8 1.4 2.7 1.8 2.6 2.6 2.2 We include results for the 70M variant in Table1for completeness over the model suite, but choose to focus mainly on the larger models for the later experiments.Similar trends for TPR@1%FPR. See Table11in Appendix.Using recently seen members elevates MIA performance noticeably, but doesn't disrupt the impact of increasing training data size. See Appendix C.1 for the impact of the recency of data seen in training.PYTHIA-DEDUP-2.8B stands apart with a performance trajectory that is consistently near-random. Previous work also observes unexplainable behavior for this model(Biderman et al., 2023a).Non-members can still have 100% n-gram overlap without being members as n-grams can appear in different member samples. However, this is increasingly unlikely for larger n.Figure14shows n-gram overlap distributions for other n.We perform token replacements at random with random tokens, so it is unlikely that these edited members are also actually members.Ideally, the member set should be fixed, which could be done by performing multiple training runs and injecting the fixed member set at various steps. However, this is computationally expensive. Furthermore, because of how the data is shuffled, we'd expect the difficulty of the member set to be reasonably consistent across our samplesNote that while the articles are created in the recent time frame, the contents of the Wikipedia page aren't necessarily about recent topics, people, or eventsThis slightly differs from the Pile ArXiv data collection, which uses the ArXiv bulk access through S3. However, we believe both ArXiv bulk access and API should yield the preprints in the same manner regardless.Note that the scores over different reference models may not be directly comparable due to the reference models having different tokenizers. This may contribute to the poor performance of this naive ensembling method.Also target models that are domain specific like DATABLATIONS or are trained on a less diverse corpus like SILO 


Decision: This is an empirical paper which evaluates the effectiveness of membership inference attacks against LLMs, concludes that they are ineffective most of the time, and goes on to investigate the factors that make the attacks more or less effective. It especially investigates the effect of near-online training and the inherent ambiguity in what counts as membership.

Overall, this is a solid empirical paper. It first demonstrates an interesting central claim, and then analyzes it from multiple angles. The topic is very timely with the increasing concern over copyright/plagiarism. The writing is clear, and the experiments seem well motivated. Reviewers raise a few concerns with the details of the experiments, such as the tuning methodology and possible inconsistency with prior work. But at the end of the day, all of the reviewers feel this paper deserves to be accepted, and this is my sense as well.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: Summary: 
The paper studies Membership Inference Attacks on LLMs. The authors perform many analyses to study the successes and failures of MIA attacks in the specific case of LLMs. In particular, the authors found that MIA performs surprisingly badly for LLMs. The paper provides an extensive suite of experiments to test various hypotheses to explain what makes MIA so hard for LLM. The paper concludes with some remarks about reframing the task of Membership Inference for the LLM scenario.


Overall:
I think this is a good paper worthy of publication.
strength_weakness: strength_weakness: The claim that MIA performs near random could downplayed a bit because Fig 1. seems to show performance statistically significantly above chance in most scenarios. To make the claim that MIA performs badly for LLM, it could be more convincing to compare the reported AUC with AUCs in order domains where MIAs have been developed and tested before.
paper_summary: paper_summary: The paper evaluates 5 recent membership inference attacks (MIAs) for LLMs and finds that most of them have near-random (AUC ROC < 0.6) performance. The evaluation was carried out on the Pythia suite of models trained on the PILE dataset. This setup provides an appropriate test bed since the knowledge of the train and test sets enables the fair evaluation of MIAs. It is suggested that the following characteristics of LLMs substantially reduce the performance of MIAs: (1) use of massive training data, (2) near-one epoch training for LLMs, (3) frequent overlap of members and non-members in natural language (non-members have high n-gram overlap with members).

It is suggested that the high performance of the previous MIA (Shi et al., 2023) was due to the temporal shift.
strength_weakness: strength_weakness: - The paper states negative results for the MIA attacks which makes the practical assessment of privacy leakage from LLMs more questionable. No alternative solutions are proposed.
questions: questions: - Did the authors consider other metrics to analyze the overlap between members and non-members apart from n-grams, for example, MAUVE score: https://arxiv.org/pdf/2102.01454? If so, which kind of other metrics are relevant? 
- Why the smaller models from Pythia were not used in the assessment (the model of size < 160 min parameters)?
- It is claimed that “MIAs perform significantly better when the non-member distribution has a lower n-gram overlap“. Then, why does the GitHub dataset have a much higher MIA than other datasets (in Table 2, Orig for GitHub vs other datasets like Wikipedia)?
paper_summary: paper_summary: This paper questions the effectiveness of MIA approaches which are currently used to evaluate the leakage of large langage model (LLM) pre-training data. The authors evaluate several MIA methods from previous works. They find that the performance of MIAs is quite poor on several LLMs trained on the Pile when evaluated on subsets of the Pile from different domains. This finding seems to contradict previous works which reported much better MIA performance. The authors explain this difference by arguing that the previous works used member and non-member data from different time ranges, resulting in distribution shift of non-members with respect to members. Therefore, previous MIAs likely performed distribution inference rather than membership inference, with distribution inference being an easier task. The authors also question the relevance of the task’s definition for natural lanauage where it is possible to construct different but similar samples (which are technically non-members), but should reasonably be viewed as members. MIAs are shown not to be robust to such samples, which are wrongly classified as non-members in the high-confidence regime.

I think that this work is timely and tackles an important question, which is relevant for privacy auditing and for copyright infringment. The paper does not propose a new method, so the technical novelty is very limited. However, it performs a large-scale evaluation of several state-of-the-art MIA approaches, whose results challenge the strong performances of MIAs reported in some of the previous works (Shi et al, 2023 and Meeus et al, 2023). This is an intriguing finding and the authors propose a plausible explanation for these results. The authors further challenge the definition of MIAs in the context of LLMs, showing that MIAs are not robust to small changes in the sample. 

However, the paper in its current form does not seem ready for publication, as it is imprecise on several important points neccessary to substantiate its claims and the benchmarking of MIAs seems incomplete and to suffer from a methodological flaw. Most of these points seem fixable and I hope that the authors will address them in their rebuttal.
strength_weakness: strength_weakness: - Potential methodological flaw: the Reference-based and min-k prob attacks seem to be tuned on the test data. I re-read Appendices A.3.1. and A.4 several times looking for a mention of validation data and I found only one, for the DATABLATIONS model, while the Reference-based tuning is done on PYTHIA-DEDUP models (for which there is no mention of validation data). Tuning the methods on the same data on which final performances are reported is a methodological flaw because it biases the results. It also gives an unfair advantage to the methods being tuned compared the ones which are not tuned. This is especially problematic here since the extensively tuned Reference-based attack is found to perform « best » in the end. This can be fixed by using a validation dataset for tuning the methods which is independent from the test set.

- Unclear exactly how the author’s analysis of temporal shift relates to the previous works of Shi et al, 2023 and Meeus et al, 2023. The paper should clarify whether the same experiment settings (LLMs, data domains, temporal ranges) as the two papers are reproduced, or if the author only speculate that these two papers suffer from temporal shift. Making this very precise is needed to support the abstract claim that « this apparent success can be attributed to a distribution shift », else the claim should be more nuanced. For instance, going through these two papers, I understand that in the arXiv domain, Shi et al uses members and non-members from before 2016 and after 2023, respectively, Meeus et al uses members before February 2023 and non-members after February 2023, and the current paper uses members from July 2020 and non-members from later months. Since the temporal shift is slow (Figure 6), it likely affects the former paper more than the latter. Furthermore, Meeus et al. also uses a second dataset where there is no temporal shift and uses longer documents for which MIA risk is likely to be higher. These differences are not discussed in the paper.

- Limited evaluation: the paper focuses on 200-word texts. When it comes to copyright infringment, longer documents such as news articles and books pose a bigger concern. As mentioned by the authors and shown by Shi et al, 2023, MIA risk increases with text length. Therefore, it is likely that MIAs might still work for, e.g., books. If this is true, the paper’s core claim would not hold. There is no discussion of this in the paper.

- A point is made that the MIA definition is not aligned with how we reason about privacy, but no better definition is proposed. The link between this part and the rest of the paper is not very clear. The usefulness of experimenting with large edit distances of 10 and 25 is unclear to me: if a large fraction of the text is replaced with random tokens, would the text still be considered to be a member by a human? Even for an edit distance of 1, replacing a token with a random token does not guarantee that the text still makes sense (which would be necessary in any notion of neighborhood). The empirical part of this section seems insufficiently developed.
questions: questions: - How do the authors ensure that their MIA evaluation setting (non-members from the Pile test data) does not suffer from distribution shift ? For instance, is there any reference stating that the Pile data was randomly shuffled before being split into train and test ?

- Other works have already challenged the reliability of MIA predictions, see [A] and [B]. For instance, [A] shows that neighboring non-member samples would be classified as members which is related to the point made in this paper. Therefore the insight that MIAs are brittle is not new (although demonstrating it empirically in the LLM setting is). The paper does not position itself with respect to these works. Can the authors elaborate on their differences with [A] and [B] and add a discussion of previous works challenging MIA reliability ?

[A] Rezaei, S., & Liu, X. (2022). On the Discredibility of Membership Inference Attacks. arXiv preprint arXiv:2212.02701.

[B] Hintersdorf, D., Struppek, L., & Kersting, K. (2021). To trust or not to trust prediction scores for membership inference attacks. arXiv preprint arXiv:2111.09076.

- How does the state-of-the-art MIA method of Meeus et al, 2023 perform compared to the other approaches?

- Figure 1: How are the confidence intervals computed, e.g.., over how many samples?

- Sec 3.2.2 – Resampling of non-members to have low overlap. This step seems over-engineered and, as the authors notice, results in distribution shift. A simpler approach would be to rank the non-members by the MIA scores and check if MIA confidence is negatively correlated with n-gram overlap. Did the authors consider this approach ?

- Benchmarking of MIA approaches : Are the results statistically significant ?

- ArXiv temporal shift experiment: there is a difference in the pipeline used to construct members and non-members (S3 vs arXiv API), which could introduce a distribution shift beyond the temporal shift. To substantiate the claim that « both arXiv bulk access and API should yield the preprints in the same manner regardless », can the author apply the API to retrieve some of the data before July 2020 and check that after pre-processing the documents match ? 

Other comments:

- Please consider adding the definition of a sample (200-length documents) in the main paper.

- Appendix A.4: I disagree with the statement that the reference-based attack is largely similar to the offline LiRA attack of Carlini et al., 2022. Two key features of LiRA, which don't feature in the reference-based attack, are its use of the logit loss (log(p/(1-p)) and fitting a Gaussian distribution to its distribution, which allows extrapolating to the high-confidence regime. The reference-based attack would be more similar if the log(1-p) terms were substracted from the loss.

- Reference-based attack: The authors acknowledge the limitation of LLM reference models: although they are not supposed to be trained on the target sample (for the one-sided hypothesis test to be valid), they likely are. I agree with the authors that this seems hard to fix. However, because the attack’s assumptions are invalid, it’s hard to interpret why and when it works. Some analysis is provided in Appendix B.2; however it would be better to explain the first time the attack is presented (in the main paper) when the attack is likely to work in spite of the invalid assumptions and why. Perhaps this intuition can also guide the choice of the reference model.

Minor :
- First paragraph of Sec. 3.2.1 – the argument needs to be completed by explaining that MIA success if linked to overfitted (Yeom et al, 2018).

- Figure 2 : the nuances of blue on the left are hard to distinguish in print, please consider increasing the contrast and the font size.

- n-gram overlap formula: Please consider adding 1-2 sentences explaining what the formula is meant to capture.

- Figure 3 – Github: is my understanding correct that on Github 40 % of the samples have 100 % n-gram overlap with the training dataset ? Please consider discussing this in the main paper and explaining that this does not mean that these non-members are members.

- Please consider fully defining the loss in the main paper using the negative log-likelihood of tokens.
paper_summary: paper_summary: This paper presents an empirical study of the Membership Inference Attack (MIA) against pre-trained large language models (LLMs). Following comprehensive experiments, the authors discovered that current membership inference methods face challenges when applied to LLMs. Further analysis attributed this poor MI performance to several factors: (1)The nature of pre-training, which involves a large volume of training data, typically trained for only a single epoch.(2)The inherent text-repeating nature of natural language, which introduces difficulties into membership inference. Additionally, the authors discussed the importance of addressing distributional shifts and suggested extending the MI game to consider neighboring members to better interpret information leakage.
strength_weakness: strength_weakness: (1) The paper is somewhat dense for an 9-page document. It encompasses various topics, ranging from the findings regarding the challenges of MI in LLMs and analysis of the reasons behind them to discussions on unintended distribution shifts and the MI game itself. This density can make it a bit challenging to follow and grasp the details, especially without referencing the appendix.

(2) A small suggestion instead of a reason for rejection: It would be better to place Table 1 on page 3 or 4 for smoother reading. Additionally, Figure 5 is currently somewhat complex and unclear for readers, especially because the bar graphs for non-members and members largely overlap, leading to confusion upon initial glance.

(3) I have some doubts about the empirical results, please refer to the Question part below
questions: questions: (1)In Section 4, the modified member is derived from member data, so it should be more similar to member data compared to non-members, or it should be harder to distinguish (because modified member should have a greater n-gram overlap). Therefore, intuitively, the MI score of modified members should fall between non-members and members. However, the results in Figure 5 are far from this expectation. What could be the reason for this?

(2)Figure 7 illustrates the 7-gram overlap of the GitHub domain before and after decontamination. This suggests that the statistics presented in Figure 3 seem to rely on data from the original domain. However, this approach lacks rigor since your main results are derived from decontaminated data, making it more appropriate to conduct statistical analysis based on decontaminated data. Please correct me if I have overlooked any details or misunderstood anything.

Comments: title: Response to reviewer
Response to authors
Thank you
Response to authors
comment: We thank the reviewer for the helpful discussion. We agree that the discussion points above would be valuable in the future revision and will add them alongside the minor comments from before. Some additional notes:

[RR1] To further clarify, we bolster the observation that MIAs perform near-random when targeting pre-trained LLMs (from the main experiments) by showing the empirically determined best performance of the MIAs. Here, the hyperparameter ablations for reference model or k choice is to form a rough empirical "upper-bound" for these attacks' performances. Evaluating MIAs for the sake of determining the best-performing MIA would indeed be biased if such ablations for hyperparameters were done on the same evaluation benchmark, but this is a different goal from what we aim to show in our main experiments.

For the other analysis, we saw that such hyperparameter choices didn't impact the observed trends and chose not to visualize the figures for other hyperparameter choices due to space limitations.

[RR2-4] We will add these clarifications and limitation discussions in the next revision

[Q1] Per the Pile paper, the validation and test sets are sampled randomly from the complete Pile and then held out from the training data. Any overlap between the validation/test sets and the training data is due to the nature of the complete data (as discussed in Section 3.2.2 and in the Pile paper itself), not the sampling/splitting.

[Q3] Using a classifier could yield greater performances. However, this threat model is different as it requires access to a ground truth set of members/non-members, even if only a small set is needed. We will clarify this difference in the future revision.
Thanks for your response. I'd like to raise my score to 7 and remain positive about the acceptance of this paper.
Thank you for your answer, I appreciate that. I keep my positive score.
Thank you for the response. I have changed my score following the authors' response, as I believe that the reasons to accept the paper outweigh the limitations. However, I still think that the paper should be revised to include the discussion points of the authors' rebuttal and the limitations, as outlined below.

[RR1] Thanks for clarifying this. The authors experiment with multiple reference models and pick the best one for all the remaining analyses, which are performed on the same test set. This can be regarded as a form of tuning and could bias the results. However, looking at subsequent plots, the results and trends identified look stable, which makes me think that the impact of this is quite limited.

[RR2], [RR3] and [RR4] Thanks for the clarifications, please consider adding the information and context provided in your response to the relevant section. Regarding [RR3], I still think that evaluating on 200-word texts limits the scope of the claim (and title) that MIAs in general do not work against LLMs; if the evaluation is limited to short texts then the paper mainly challenges current evaluations on short texts like Shi et al., 2023. Please include a discussion of this limitation and how it possibly impacts the main claim.

[Q1] Can the authors clarify what they mean by validation and test data being sampled from the training data? Does the training data overlap with validation and test data?

[Q2] Thanks for the clarifications.

[Q3] Thanks for clarifying this very important point. Please consider acknowledging this limitation: as only score-based attacks are benchmarked, it remains possible that results of MIAs can be improved if an ML classifier was trained for the task (as opposed to score-based MIAs). The authors construct a test set of in-distribution members and non-members and it isn't clear why a similar training set cannot be constructed. 

[Q4]-[Q7] and [C1]-[C3] Thanks for the clarifications.
We follow up on questions and comments below and will address the minor comments in our updated revision

[Q1] The validation/test data of the Pile is sampled uniformly at random from the training data ([1], Section 3.1)

[Q2] We appreciate the reviewer bringing these papers to our attention and will include discussion of earlier notions of membership neighborhoods. However, these works seem target a different direction, focusing on generating non-members such that they are falsely identified as members. On the other hand, we explore how neighborhood non-members can still result in information leakage of members, but current MIAs can be too sensitive to classify this member information leakage.

[Q3] We choose to only evaluate blackbox MIAs without stronger data access assumptions for this paper. 

[Q4] We use 1000 bootstrap samples to create the C.I. (See Appendix A.4)

[Q5] We plotted the MIA scores assigned to non-members against their respective {7,13}-gram overlaps; we will add these figures to the Appendix. However, as mentioned in the response to Q3 for reviewer F61L, the important takeaway to clarify in the future revision is that non-member distribution shift (i.e., in n-gram overlap) can inflate MIA, which we felt was better demonstrated by the current design.

[Q6] While we did not include exact statistical significance tests (will be included in revision), The error bars of our results are small enough for our results to imply statistical significance. We will include numbers for these statistical significance tests.

[Q7] We spot checked that documents in the Pile when passed through our pipeline with the ArXiv API from the raw source yielded the same output. We don’t use ArXiv S3 due to paywall.

[Comment 1] Agreed

[C2] We agree it is more similar to reference calibration [2] and will revise accordingly.

[C3] We chose to keep this analysis in the appendix due to space limitations and because we felt a larger ablation over reference model choice was needed to substantiate the plausible analysis. 

### References

[1] https://arxiv.org/pdf/2101.00027

[2] https://arxiv.org/abs/2111.08440