Do Membership Inference Attacks Work on Large Language Models?
Authors: Michael Duan
Anshuman Suri
Niloofar Mireshghallah
Sewon Min
Weijia Shi
Luke Zettlemoyer
Yulia Tsvetkov
Yejin Choi
David Evans
Hannaneh Hajishirzi
Venue: COLM (2024)
Keywords: privacy, membership inference
TL;DR: on the pitfalls and challenges of membership inference in language

Abstract: Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Further analyses reveal that this poor performance can be attributed to (1) the combination of a large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non-members. We also find that, when LLMs have been shown to be vulnerable to MIAs, this apparent success can be attributed to a distribution shift, e.g., members and non-members are seemingly drawn from identical domain but with different temporal ranges. Finally, we observe that existing MIAs are highly sensitive to even small changes in a sample. Such changes may cause samples that are lexically or semantically similar to members to be classified as non-members, which may be at odds with leakage that privacy auditors care about. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.

Published as a conference paper at COLM 2024
Do Membership Inference Attacks Work on Large Language Models?
Michael Duan"
Anshuman Suri
Niloofar Mireshghallah" Sewon Min'
Weijia Shi
Luke Zettlemoyer' Hannaneh Hajishirzi!5
Yulia Tsvetkov! Yejin Choi- David Evans?
University of Washington University of Virginia <mi <micduness. washington. edu>>, <as9rw@virginia.. >ddu edu>
Allen Institute for AT
Abstract
Membership inference attacks (MIAs) attempt to predict whether par- ticular datapoint member of target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Further analyses reveal that this poor performance can be attributed to (1) the combination of large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non- non-members. We also find that, when LLMs have been shown to be vulnerable to MIAs, this apparent success can be attributed to a distribution shift, e.g,, members and non-members are seemingly drawn from identical domain but with different temporal ranges. Finally, observe that existing MIAs are highly sensitive to small changes in sample. Such changes may cause samples that are lexically or semantically similar members be classified non-members, which may be at odds with leakage that privacy auditors care about. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.
Introduction
Membership inference attacks (MIAs) aim to predict whether a particular record belongs to the training dataset of a given model. Thus, MIAs have great utility for privacy auditing of models (Steinke al.) 2023), as well as investigating memorization of training data,
copyright violations and test-set contamination (Shi et al., 2023 Oren et al. 2023). While MīAs have been found achieve high attack perlormance, alluding high levels of training-date memorization (Zarifzadeh et al. 2023) Bertran et al. [2033 Lukas et al. 2023), most analyses are limited to classifiers or LM fine-tuning (Mireshghallahh et al. 2022b Fu al. 2023). The performance of existing MIAs on LLMs and their pre-training data is largely unexplored. this work, we set out to explore the challenges in evaluating membership inference attacks on LLMs, across array of five commonly-used membership inference attacks. We introduce MIMIR"] unified repository for evaluating MIAs for LMs, with implementations of several attacks from literature. We report on experiments extensively evaluating these MIAs against target models from the Pythia suite (Biderman al.) 2023b over the Pile (Cao et al. 2020) ($ For the most part, we find that the pertormance acToss most MIAs and target domains near-random.
Our further analysis suggests that the inherent characteristics of LLMs at scale-specifically, the use of massive training data and near-one epoch training (33.21) considerably decrease
Equua Contribution.
http::/ //gi www.tthub..oooooooee com/

---

Published as a conference paper at COLM 2024
current MIA performance. This suggests that the success of current MIAs in previous settings does not transfer well to attacking pre-trained LLMs seemingly due to a lack of memorization of member data. We also find that the frequent overlap between members and non-members from natural language domains considerably decreases MIA performance and raises the question of how membership should be interpreted ($3.2.2) Notably, in several domains, non-members have high N-gram overlap with members, e.g., non-members from the Pile Wikipedia and ArXiv test samples have average 7-gram overlaps of over 30%. Notably, non-members with lower n-gram overlap are more distinguishable by existing MIAs. We also suggest that high MIA performance reported by prior work (Shi et al., 2023) is likely because non-members are chosen from the same domain as members but are temporally shifted, and these seemingly in-domain non-members likely belong to different distribution as result of n-gram overlap shift ($4)
Finally, building off membership ambiguity due to n-gram overlap, we discuss how the precise definition of members in standard MI may not capture important information leakage under generative text-modeling. We generate modified members preserving lexical and/or or semantic similarity by altering tiny fraction of tokens and show that existing MIAs classify them as non-members with a high degree of confidence, often more definitively than actual non-members (55). We encourage future work to study MI using membership definitions accounting for such fuzzy members to better understand privacy leakage.
2 Background
The goal of an MIA is to infer whether a given data point was part of the training dataset 'D for model M, by computing a membership score (x. This score is then thresholded to determine a target sample's membership.
MIAs are often used as a proxy to determine whether a machine-learning model leaks information related to its training data (Shokri et al., 2017 Shokri 2022 Cummings al., 2024). It is the de-facto threat model when discussing machine-learning privacy (Shokri
atal. 2017), with a large array of attacks (Yeom et al. 2018 Carlini et al., 2022 Mireshghallah et al. 2022a) and defenses (Abadi et 2016, lang et al. 2022 Chen et 2022). More involved approaches include training shadow models (Shokri et al. 2017: Ye et al. (0022) on non-overlapping data from the target model's underlying data distribution. While attacks like LiRA (Carlini et al. 2022) show promise, they require training multiple copies of shadow models, which is often intractable for LLMs. Other stronger assumptions for MIAs include white-box access to the model (i.e., access to model parameters) or access to a ground-truth subset of member/ in-distribution non-member samples for training meta-classifiers.
In our setting, M is an auto-regressive language model that outputs probability distribu- tion of the next token given a prefix, denoted as ((x:|27-..24 (See Appendix A.4 for detailed descriptions):
We consider five MIAs
(1) LOSS Yeom et al. 2018 the target sample's loss under the model: (x; M).
(2) Reference-based (Carlini et al. [021] calibrates Mx) with respect to another reference model (Mref) to account for the intrinsic complexity of the target sample x: fx;0xx:((xx00 f(x:M)
Mxx [(x;
(3) Zlib Entropy (Carlini et al. 2021 calibrates L(x;M) with target sample x's zlib compres- sion size: M( L(x; M) (x)
(4) Neighborhood attack (Mattern et al. 2023) the curvature of the loss function at x, esti- mated by perturbing the target sequence to create n 'neighboring' samples, and comparing
the loss of the target x with its neighbors x: (x; = M(x - 1 E M);
(5) Min-k% Prob Shi et al. 2023) uses the k% of tokens with the lowest likelihoods to compute a score instead of averaging over all token probabilities as with LOSS: f(x; =
mmn--(()) [x] Exçemin-((x) - log(pl(( (x(x: X1, X)-1))
2

---

Published as a conference paper at COLM 2024
Wikipedia
Github
Pile CC
PubMed Central
Params
1088
Ref
min-
zitb
Vee
1.88
min-k
1OS5
Ref
min-
zlib
Ne
LO8S
Rei
min-k
zli
70M
.033
.044
.494
.508
-510
.629
.584
.503
.45
.899
.022
.566
.510
502
.455
160M
.504
.515
.48
.514
.513
.638
.591
.634
.656
.638
.497
.497
_503
.488
.46
.500
.516
.504
.50
.4%6
14B
.100
.544
.006
-518
5188
.656
.587
.654
.670
.525
.509
.022
.499
.46
.330
.505
500
.90
2.8B
.516
.565
511
.522
.577
.707
.67
.786
.717
.698
.501
.537
.009
.003
.022
.98
.336
.022
500
.97
6.9B
.514
.571
512
.521
.514
.672
573
.675
.684
.654
.511
564
5166
512
.505
504
.552
.508
501
.497
.516
.579
517
.524
.520
.78
.559
.683
516
.21
.517
.514
.006
.559
.512
306
.97
ArXiv
DM Math
HackerNews
The Pile
Params
LOSS
Ref
min-k
min-k
LOSS
mine
zlib
Ne
LO6S
Ref
min-k
70M
.066
.481
.499
.495
.466
.492
.320
.495
.485
.481
.494
.455
.507
.497
.506
.503
.511
.508
506
.499
160M
.507
.486
.501
.500
.507
.490
.523
.493
.422
.499
.492
.400
.497
.497
.505
.502
.511
.506
505
.499
1.4B
.513
5500
511
.538
-511
.486
.312
.97
481
.65
.53
.514
.509
.502
.044
.544
.521
.508
507
.504
2.88
.17
-531
522
512
519
485
.04
.97
482
510
.59
518
507
513
-307
.30
.522
510
.306
6.9B
.511
.53
524
516
519
485
508
496
481
469
513
546
528
5088
512
530
5599
516
512
.500
.527
.555
530
.22
.519
485
.512
.515
513
.558
.521
515
Table 1: AUC ROC of MIAs against PYTHIA-DEDUP (TPR@lo*FFPR results in Table 11) Highest performance across different MIAs is bolded per domain. MIA methods perform near random (< .6) in most domains. See Appendix B.3 GitHub outlier discussion.
Membership Inference vs. Data Extraction. MIA advantage is frequently used as a measure of information leakage (Shokri 2022 Shokri et al. 2017 Mireshghallah et al. 2022a) and
proxy for measuring memorization Carlini et al.
2021: Mireshghallah al. 2022b), with
recent attempts studying user-level leakage for the fine- fine-tuning setting (Kandpal et al. 2023).
However, the 'extractability' of training samples has recently become synonymous with
memorization and is increasingly used to compare memorization across models (Biderman
eet al. 2023a Carlini et al. 2023: Tirumala et al. 2022). ) Kandpal et al. (2022) investigated the impact of factors such as training data deduplication on extractability in similar vein to our work on MIA. With extraction, a prefix is used as a prompt to measure the memorization of sequence by comparing the resulting generation against the suffix. Both MIA and extraction are useful techniques for studying leakage in models, but rely on different assumptions and reveal different types of leakage risks. While MIAs require knowledge of candidates and only reveal directly which of those candidates are included in the training data, extraction requires knowledge of sufficient-length prefixes perform extraction and additional measures to determine if extracted texts are valid.
3 Membership Inference on LLMs is Difficult
We perform a large-scale evaluation of five tftestttt--- ( on a range of LLMs with up to 12B parameters and diverse benchmarks. For the reference- attack in Table T and all following experiments, we use BABBLELM-BASE-LLLHAA-A
as the reference model (determined empirically in $3.1 Code is available at h/tp://g11 thub. comb htt//roogo022/ mimir
Target models. We primarily target the PYTHIA model suite, including (1) five models of PYTHIA Biderman et al. 2023b) with 160M, 1.4B, 2.8B, 6.7B, and 12B parameters trained on the original Pile data (Gao et al. 2020), and (2) five models of PYTHIA-DEDUP (Biderman et al. 2023b) with the same parameter counts PYTHIA but trained on the deduplicated Pile data. We also experiment with the GPT-NEO family to validate our findings with different model family, observing similar trends in most domains (see Appendix A.6)
Datasets. We use seven diverse data sources included in the Pile: general web (Pile-CC), knowledge sources (Wikipedia), academic papers (PubMed Central, ArXiv), dialogues (HackerNews)) and specialized-doman (DM Math, Github). We also perform experiments over the entire Pile. Members and non-members for each data source are sampled from
We include results for the 70M variant in Table 1 for completeness over the model suite, but choose to focus mainly on the larger models for the later experiments.
3

---

Published as a conference paper at COLM 2024
Normal
De-duplicated
Wikipedia
ArXiv
HackerNews
0.70
0.70
0.70
0.65
AUC ROC
0.60
0.60
0.60
0.55
0.50
0.50
0.50
0.45
10 0.45
0.45
0
2
5
10
2
10
12
Model Size (B)
Figure 1: MIA performance as model size increases for the reference-based attack over select domains. We also plot the AUC ROC trajectory against the non-deduped Pythia suite for comparison. Increasing model size slightly boosts MIA performance while deduplication decreases performance. Other attacks follow similar trends (Appendix Figure 12
the train and test sets of the Pile, respectively. Gao et al. (2020) decontaminated the Pile test set against the training set at a document level; nonetheless, to be more rigorous, perform additional deduplication following Groeneveld et al. (2023). We additionally sample from documents greater than 100 words, and truncate samples up to 200 words. Refer to Appendix A.3 A.3 for further details.
Evaluation metrics. We primarily report ROC for our evaluations, and additionally record TPR@low%FFP (Carlini et al. 2022) to assess performance in high-confidence settings. We visualize the 95% confidence interval for AUC ROC scores via shaded regions.
3.1 Main Results
MIAs perform near random. Table]] that all existing MIAs perform near random for most 3 No single MIA target model demonstrates attack AUC above 0.6, with the exception of Github domain (see Appendix B.3 for discussion). Overall, the reference-based attack performs best, although there are a few settings where other attacks perform better, e.g., Min-k% Prob on Pile CC for the 160M PYTHIA-DEDUP model. Marginal differences in performance across MIAs make it hard to single out an overall bes! attack.
MIA performance tends to increase with the target model size (Table 1 Figure 1 in agree- ment with prior work (Shi et al. 2023 et al.) 2023a: Watson et al. 2022). This is likely because larger models are more prone to overlittigg the training data (Nakkiran et al.)
2021). We also find deduplication of the training data reduces MIA performance (Figure)
confirming the findings from Kandpal et al. (2022).
We ablate the choice of reference models in the reference-based attack in Appendix A.5 In summary, (1) most reference models yield poor performance, with (2) even aggregating all reference models performs poorly. In general, the right reference model for a target LLM challenging and largely empirical. A reference model should be trained on the data that is same-distribution but largely disjoint from the training data of the target model. However, this assumption is hard to impose at the scale of pre-training corpora; common practice is collect all the data available on the web, leading independently collected datasets to naturally overlap with each other.
Difficulty Choosing Reference Model.
being the best overall, and
find choosing
3Similar trends for TPR@1 %FPP. See Table in Appendix.
4

---

Published as a conference paper at COLM 2024
1.48
2.88
6.98
128
LOSS
Ref
Min-49%
2ilb
No
Datablations C44
0.90
0.80
0.8
AUC ROC
0.70
AUC ROC
0.7
0.60
0.6
0.50
0.5
20000
40000
60000
BODOJ
100000
12
14
Number Training Steps
Number of Training Epochs
Figure 2: (Left) Reference-based attack performance the amount of training data seen, measured in the number of training steps, increases across 1 epoch of the deduplicated Pile. In general, performance spikes greatly before gradually decreasing as the amount of traiming data seen increases. Other attacks (Figure 13 Appendix) follow similar trends. (Right) MIA performance on target model DATABLATIONS the number of effective epochs increases via increasing epoch count. Performance increases linearly with the number of effective epochs. See Figure 10 results on SILO.
3.2 Why is MI Challenging against LLMs
We identify several key factors that may contribute the decreased performance of MIAs on LLMs. Some factors are due to unique characteristics of practices in LLM pre-training while others are due inherent ambiguity MIA (33.2.2
(32.11
3.2.1 Characteristics of LLMs
Training Data Size. Current state-of-the-art pretrained LLMs are trained with billions and trillions tokens (Touvron al. 2023a b Team 2023). We hypothesize the large pretraining corpora characteristic to LMs decreases MIA performance, as larger pretraining datasets
lead to better generalization (Hoffmann et al. 2022 Muennighoff et al. 2023).
We employ the PYTHIA-DEDUP model suite's intermediate checkpoints to assess the impact of different amounts of training data. While keeping non-members fixed, we sample members for each checkpoint from its most recent 100 steps to remove the impact of the See Appendix A.3.1 for details.
recency bias of the members]
MIA performance generally starts as near-random, then rapidly increases within the next few thousand steps, before decreasing across successive checkpoints (Figure (Figuur 2 left). We speculate the initial low performance is due to the model warming up in training, with high losses across both member and non-member samples. We believe the following rapid rise and then gradual decline in performance are because the Lata-to-parameter-con ratio is smaller early in training and the model may tend to overfit (Yeom et al.) 2018), but generalizes better as training progresses, in line with observations in existing work (Nakkiran et al. 2021).
Number of Training Epochs. is standard practice to pre-train LLMs for around one epoch, given the scale of data and their tendency to overfit quickly (Muennighoff et al., 2023 Komatsuzaki 2019). Previous MTA works that demonstrate attack effectiveness consider
"Using recently seen members elevates MIA performance noticeably, but doesn't disrupt the impact of increasing training data size. See Appendix C.1 for the impact the recency data seen training. PPYTHAA-DDUUP2 stands apart with performance trajectory that is consistently near-random. Previous work also observes unexplainable behavior for this model (Biderman et al. 2023a

---

Published as a conference paper at COLM 2024
Wikipedia
Github
ArXiv
Hacker News
Progortion of Data
0.10
0.10
0.10
0.08
0.00
0.04
0.04
0.04
0.1
Mmll
0.00
0.00
0.00
20
7-gram Overlap
Figure 3: Natural distributions of 7-gram overlap of non-member data over select domains. Github has a considerably higher overlap than other domains.
supervised fine-tuning masked LM pre-training (Nakamura et al. 2020 Lehman et al. 2021 Mireshghallah et al. 2022a b), where models are trained for more than 10 epochs. We explore how the near-one epoch training of LLMs leads to decreased MIA performance.
To verify this hypothesis, we perform MIA against the Datablations suite (Muennighoff
et al. 2023), consisting of models trained on subsets of C4 (Raffel et al. 2019) train data for
varying numbers of epochs (see Appendix A.2 for model details). We also experiment with
SILO (Min al. 2023);
Appendix C.2 for STLO results.
Increasing the number of effective epochs corresponds to an increase in attack performance ( right). While Muennnigfff al. (2023) shows training for multiple epochs helps improve performance, our results suggest that such multi-epoch training (and/or large upsampling factors) can increase training data leakage.
3.2.2 Inherent Ambiguity in MIA
Natural language documents commonly have repeating text- with the best efforts in decontamination and deduplication. These include common phrasings and quotes, natural use of similar texts, and syntactical similarities inherent to specific domains. This leads to substantial text overlap between members and non-members, which motivates the following hypothesis: higher overlap between members and non-members increases MIA difficulty.
We quantify overlap using the percentage of it-gram overlap, defined as For a non-member sample consisting of m words such that X1.22..Xm and an 11-gram in x defined as continuous substring Xi..X;+n-11 the n-gram overlap of x on training dataset is
1
m
n +1
For given non-member sample, this gives us the percentage n-grams in the non-member that can be found in at least one member sample. We first compute the percentage of 7-gram
overlap for non-members against the entire Pile training (member) set (Figure 3 see Appendix [.] for implementation details. We observe high n-gram overlap with training data for substantial portion of non-members; e-g., the Wikipedia, ArXiv, and PubMed Central domains have average 7-gram overlaps of 32.5%, 39.3%, and 41.0%, respectively. Domains such GitHub, DM Mathematics, and FreeLaw see even higher overlap, with mean 7-gram overlap of 76.9%, 72.8%, and 62.3%, respectively.
High n-gram overlap suggests that substrings of non-members may be seen exactly during training, which makes the distinction between members and non-members even less clear. To verify our hypothesis, we resample non-members ensuring V 20% 11-grmm overlap with members, and report MIA performance (Table While this step is designed to more strictly
Non-members can still have 100% n-gram overlap without being members as n-grams can appear in different member samples. However, this increasingly unlikely for larger nt Figure 14 shows 1--gram overlap distributions for other n.

---

Published as a conference paper at COLM 2024
Wikipedia
Github
PubMed Central
Pile CC
ArXiv
Domain
ORIG
7-GRAM
ORIG
7-GRAM
ORIG
7-GRAM
ORIG
7-GRAM
ORIG
7-GRAM
LOSS
.78
.878
.780
.516
.527
.787
Ref
.579
.677
-399
.15
.399
.595
.382
.644
.535
.715
min-k
683
.900
.72
.530
.744
zlib
.524
.311
.50
908
506
.722
.517
.660
.521
.780
.200
.612
.660
.877
.977
.737
.514
.566
.519
.773
Table 2: Comparison of MIA performance over select domains with varying non-member well as the natural non-member
sets at < 20% n--ram overlap threshold for set. 'arget model is PYTHIA-DEDUP-D2 and AUC ROC reported. Strict n-gram overlap thresholding results in higher performance.
eliminate instances of non-member records that may overlap with training records, it also introduces explicit drift between member and non-member distributions by selecting non-members that are most "unlike" training records. We clarify that this step is not suggestion for researchers to alter their benchmarks; such processing step drifts away from the standard membership inference game (Yeom et al. 2018).
Results. MIAs perform significantly better as the non-member distribution concentrates towards lower n-gram overlap and further diverges from the natural n-gram overlap distribution of non-members from the training distribution e-g. 1-6---56----0 -516->.666 in Wikipedia, 6690+.908 in Github, and 512->.792 in PubMed Central for various attacks. This is intuitive as the target model is likely to assign lower likelihood to non-members further from its training data, making members and non-members more distinguishable. Note that decreasing the n--rram overlap threshold, especially for smaller n, pushes the setting closer to distribution inference (Suri & Evans 2022), since the distributions of 'member' and 'non- member' records are no longer the same. We further discuss outlier behavior in Appendix B
We note that n-gram overlap is an intrinsic property of natural language rather than problem of the Pile train-test split. These splits are already deduplicated at a document level, following standard practice in decontamination (Gao et al. [2020] Brown et al. (2020). Nonetheless, repeating texts across distinct documents are tundamental and natural proper- ties of domain data. We also note that 1--ram overlap distribution analysis can help assess how representative of a target domain a set of candidate non-members when constructing MIA benchmarks. Ultimately, we highlight the need to consider qualities of the data domain, e.g,, n-gram overlap, and understand their potential impact on MIA performance.
4 Importance of Candidate Set Selection
In contrast to our findings in $ recent works report state-of-the- MIAs achieving .7
AUC ROC on pretrained LLMS (Shi et al. 2023 Meeus et al. 2023). We investigate how the non-member candidate selection methods in these works can result in an inherent but likely unintended distribution shift between members and non-members one such reason for the observed performance differences.
Experimental Setup. Prior work distinguishes members and non-members of a target domain based on the knowledge cutoff date the target model, with members coming before and non-members coming after the cutoff. We construct similar experimental settings under two domains also used in earlier works: Wikipedia and ArXiv.
We follow the same setup as 3 the Wikipedia domain, but replace the non-member set with samples from the entire RealTimeData WikiText dataset (Li al. 2023c) The Pile members sampled from articles in Wikipedia dump from before March 2020 (Goo et al.) 2020), whereas non-members consist of Wikipedia articles created from August 2023 and onwards. For the ArXiv domain further details Wikipedia, see Appendix A.3.2
7

---

Published as a conference paper at COLM 2024
0:36
Original
Temporal Wiki
Temporal Wiki
# Params
LOSS
Ref
min-k
zlib
Ne
160M
.643
.602
.648
.541
.600
1.4B
.63
.705
.62
.572
603
proportion
0.00
.593
0.04
0.06
6.9B
.675
.788
.714
.601
.200
0.04
0.04
.680
.796
.719
.607
.626
0.02
0.02
0.00
0.00
Table 3: AUC-ROC on the temporally shifted Wikipedia benchmark across vari- ous MIAs. Target models are the PYTHIA- DEDUP suite models. For each model, the highest score across MIAs is bolded.
7-gram Overlop
Figure 4: Distribution of 7-gram overlap for the original and temporall--shifted non- members.
Results. Table Gablegdemmos demonstrates that the temporally shifted settings yield MIA performances significantly higher than when members and non-members are from the same temporal range. Figure 6 (Appendix) also demonstrates that MIA performance generally increases as non-members are further temporally shifted from member data. We speculate this follows from changes in language such as the introduction of new terminology and ideas over time.
Temporal Shift as Change in 1-gram Overlap. We interpret temporal shift change in æ--rrm overlap distribution between the original and temporally shifted non-members. Figure 4 demonstrates that the distribution of 7-gram overlap of such shifted data concentrates tes at lower overlap percentages compared to their natural counterparts. The natural Wikipedia non-members have an average 7-gram overlap of 3,,, whereas for the temporally shifted
Wikipedia non-members it is 13.9%. See Appendix B.4 for temporal ArXiv discussion.
In general, when aiming to assess MIA performance, we advise estimating how representa- tive sample non-member set is of the member domain by comparing its n-gram overlap distribution with that of left-out sample set from the pretraining corpora. Particularly, if the distribution of the candidate set is noticeably shifted towards lower и-gram overlap compared the left-out member sample set, the candidate non-member set may not be representative of the member distribution from the target domain and potentially high MIA performances should be carefully examined. Closer inspection (Table Appendix) reveals the extent of such over-estimation; decision thresholds derived using temporally-shife non-members end up testing for temporal shift rather than membership. We note that dis- tinguishing between members and temporally shifted non-members is a realistic inference game with practical implications but differs from the classical MI game as temporally-shife data may belong to different distribution. Furthermore, we aren't able to reproduce the experimental settings of prior works (Shi et al., 2023 Meeus et al., 2023) for distributional shift analysis, inconclusive if their dittering results are solely due to the temporal shift between member / non-member samples (see Appendix A.3.2 for details).
5 Revisiting Membership
The discussion of high и-gram overlap of non-members with members raises the question of whether exact membership is always useful concerning information leakage. The definition of membership in the standard MI game treats only records seen exactly during training as members, e.g., for language models, substrings appearing exactly in the training corpus. However, this may be at odds with what adversaries and privacy auditors care about when concerning information leakage. For generative models especially, guessing the membership of some sample via other sufficiently close samples can be useful. For example, any paraphrase of "Product launch in Q2, 2025" may be relevant as long as it preserves information regarding the product launch timeline, even if the paraphrase has a significant lexical difference, e.g., "Launching product in Q2, 2025" "Q2 2025 release". Note that standard notions of Differential Privacy (Dwork et al. 2006) do not immediately protect against such cases, since records being tested for membership not, in the literal sense,
8

---

Published as a conference paper at COLM 2024
Nember Wilipedia (1055)
Non-Wember
Modified Member
(1055)
Ardiv (Re()
Vikipedia (Ref)
0.4
2.4
2.0
Propartion Data
0.3
0.25
1.2
1.0
0.
0.8
1.0
0.6
611
0.10
0.4
0.5
0.05
0.0
0
1.0
-088
-0.6
-0.4 Score
-0.2
0.0
-2.0
-15
-10
-0.5
0.0
0.5
Edit Distance: 25
Wikipedia (LCSS)
0.35
1.4
Proportion Data
0.30
1.50
0.3
0.25
1.0
1.25
0.8
2.00
0.2
0.15
0.6
0.75
0.10
6.1
0.2
0.25
0.9
0.00
0.0
0.00
-0.8
-0.5
Score
Semantic Neighbors
Figure 5: Distribution of scores for LOSS and Reference-based attacks for members, non- members, and modified members across ArXiv and Wikipedia domains. (Top) Modified members generated by random token replacement for edit distance 25. (Bottom) Modified members generated by replacing 5% of tokens with semantically similar tokens.
Domain
1%
5%
10%
Domain
LOSS
Ref
1%
10%
5%
10%
ArXiv
0.0
0.0
0.1
0.7
ArXiv
0.0
2.5
1.9
Wikipedia
0.0
0.0
0.0
0.2
0.1
0.1
0.6
0.4
0.1
Wikipedia
0.0
0.5
2.3
0.4
3.0
82
Table 4: FPR (%) modified members (treated non-members) when using a score threshold that achieves 1, 5, or 10% FPR on the original member and non-member data for ArXiv and Wikipedia domains. (Left) Results for lexically similar modified members at edit-distances n = [1, 10, 25). Reference- based attack is shown. For LOSS attack, all FPR values are 0 across all tested FPR thresholds and values of (Right) Results for semantically close modified members. LOSS and Reference-batte attack reported.
members. We explore two methods of constructing such "sufficiently close" samples initial step in studying approximate membership definitions.
Lexical Distance. We first experiment with creating modified member samples by replacing n random tokens in a given sample with tokens randomly sampled from the model's (0, 25) (20 trials per n) and visualize the distribution
vocabulary. We do so for for MIA scores using LOSS and Reference-based attacks (Figure 5 top). The LOSS attack yields distinct loss distributions between the modified members and original member non- members, suggesting that the model is sensitive to out-of-place random tokens even for lightly perturbed member samples. The Reference-based attack, on the other hand, has distribution of modified members much closer to both members and non-members, likely due to the reference model calibration accounting for the complexity introduced by the random tokens. This further reinforces the ambiguity of such samples should should they be considered members or non-members?
We also compute the thresholds corresponding to certain FPRs for actual member and non-member data and use these thresholds to compute the FPR on the modified members. We consider these modified members as "non-members" which they with regards to

---

Published as a conference paper at COLM 2024
exact match Table 4 (left) shows that these modified members have extremely low FPRs for edit distance as low as π = 1, suggesting that these samples would be classified as non-members by the MIA, even though from the perspective of information leakage such sample is effectively a member. We highlight the need to rethink membership for samples with extremely low lexical distance from actual training members, though even membership at higher lexical distances is important with respect to what information is still leaked in the unperturbed portions.
Semantic Distance. While a small edit distance suggests closeness in meaning, a higher edit distance does not necessarily imply loss of semantics. We compute MIA scores for neighbors generated for member samples as part of the Neighborhood attack (see Appendix A.4) for the Wikipedia and ArXiv benchmarks and repeat the above pipeline. Visualizing the scores shows how the modified members are not too far from original member score distributions, especially for the Reference attack (Figure bottom). We repeat the same FPR experiment as culistnnce-basee modified members. While the FPR for these semantically close records is noticeably higher than records close by edit distance, the false positive rates are still low (Table 4 right). Again, these results suggest that semantically close members would be classified as non-members even though they may be as useful as actual members depending the inference goal and the semantic information preserved.
While it is not surprising that semantically close neighbors have MIA scores more similar to actual member samples than ranpocly-replaced tokens, it is clear that an ideal distance function should combine the benefits of lexical distance and semantics in defining a mem- bership neighborhood. Such observations also motivate a fully semantic MI game, where a neighbor member may be defined by its proximity to an exact member in a semantic embedding space. This may provide a clearer interpretation of knowledge leakage than lexical matching, especially when samples naturally have high lexical (i.e., n-gram) overlap.
Conclusion
We shed light on the difficulty of membership inference against LLMs from the lens of an adversary. Our results suggest two possibilities: (1) data does not leave much of an imprint, owing to characteristics of the pre-training process at scale, such as large datasets and single- epoch training, and (2) the similarity between in and out members (which we demonstrate via n-gram overlap), coupled with huge datasets, makes this distinction fuzzy. While MIA performance could improve via stronger attacks (Casper et al.) 2024), the second factor requires rethinking the membership game itself. Our empirical results suggest that both of these might be confounding factors while measuring leakage. The membership inference game needs to be extended for such generative models to better align with information leakage that adversaries and auditors may care about, such as user-level leakage (Kandpal et al. 2023) and PII (Lukas et al.. 2023). In the meanwhile, special care should thus be taken to avoid unintentional distributional shifts while constructing non-members for MIA benchmark construction.
Acknowledgements
We are grateful to Nicholas Carlini for comments early drafts of this paper. This research is supported in part by DARPA SemaFor Program No. HR001 120C0123, and grants from the National Science Foundation through the AI Institute for Agent-based Cyber Threat Intelligence and Operation (#2229876), Center for Trustworthy Machine Learning (#1804603) and the DARPA MCS program through NIWC Pacific (N66001-19-2-4033..
This work is also supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Pro- gram contract #20222220072200044 The views and conclusions contained herein are those of
We perform token replacements at random with random tokens, so it is unlikely that these edited members are also actually members.
10

---

Published as a conference paper at COLM 2024
the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Gov- ermment is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
References
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS'16, 2016.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh Levy-Kramer, Connor Leahy, Lucas Nestler, Kip Parker, Michael Pieler, Jason Phang, Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Benjamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large Scale Autoregressive Language Modeling in Py PyTorch, 2023. URL httpt:://ww// giw.ww....... thub. com/ eleutheral/gpt-neox
Martin Andres Bertran, Shuai Tang, Aaron Roth, Michael Kearns, Jamie Heather Morgen- stern, and Steven Wu. Scalable membership inference attacks via quantile regression. In Advances in Neural Information Processing Systems, 2023.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memorization in large language models. In Advances in Neural Information Processing Systems, 2023a.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. In International Conference on Learning Representations, 2023b.
Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorfloww 2021. URL http://gii thub
com/eleutherra/ggt-nne
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, 2020.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. In
USENIX Security Symposium, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss. Kather- ine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting Training Data from Large Language Models. USENIX Security Symposium, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership Inference Attacks From First Principles. In IEEE Symposium Security and Privacy, 2022.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In Interna- fional Conference Learning Representations, 2023.
Stephen Casper, Carson Ezell, Charlotte Siegmann, Noam Kolt, Taylor Lynn Curtis, Ben- jamin Bucknall, Andreas Haupt, Kevin Wei, Jérémy Scheurer, Marius Hobbhahn, Lee Sharkey, Satyapriya Krishna, Marvin Von Hagen, Silas Alberti, Alan Chan, Qinyi Sun, Michael Gerovitch, David Bau, Max Tegmark, David Krueger, and Dylan Hadfield-Menell. Black-box access is insufficient for rigorous ai audits. ar preprint Xiv 14446, 2024.
11

---

Published as a conference paper at COLM 2024
Dingfan Chen, Ning Yu, and Mario Fritz. Relaxloss: Defending membership inference attacks without losing utility. In International Conference on Learning Representations, 2022.
Rachel Cummings, Damien Desfontaines, David Evans, Roxana Geambasu, Yangsibo Huang, Ryan Rogers, Matthew Jagielski, Peter Kairouz, Gautam Kamath, Sewoong Oh, Olga Ohrimenko, Nicolas Papernot, Milan Shen, Shuang Song, Weijie Su, Andreas Terzis, Abhradeep Thakurta, Sergei Vassilvitskii, Yu-Xiang Wang, Li Xiong, Sergey Yekhanin, Da Yu, Huanyu Zhang, and Wanrong Zhang. Advancing differential privacy: Where we are now and future directions for real-world deployment. Harvard Data Science Review, Jan 2024. URL.ttps://6oi.or//11... wtt.:///rrr111111111111 0tt.://00.1166/999999226.2.33330
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019,
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, 2006.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Practical membership inference attacks against fine-tuned large language models via self-prompt calibration, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling. ar preprint ar arXiv:2101.0000
2020.
Dirk Groeneveld, Chris Ha, and Ian Magnusson. BFF: The big friendly filter, 2023, URL https://github. com/ allenai/bff
Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang. Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muen- nighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. Olmo: Accelerating the science of language models. Preprint, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems, 2022.
Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot, and Chiyuan Zhang. Measuring forgetting of memorized training examples. In International Conference on Learning Representations, 2023.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models. In International Conference on Machine Learning, 2022.
Nikhil Kandpal, Krishna Pillutla, Alina Oprea, Peter Kairouz, Christopher A Choquette- Choo, and Zheng Xu. User inference attacks on large language models. arviv preprint ar arXit:2310.09926, 2023.
Aran Komatsuzaki. One epoch is all you need. ar arv preprint ar wxXi::9906.06669 2019.
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
12

---

Published as a conference paper at COLM 2024
Marvin Li, Jason Wang, Jeffrey Wang, and Seth Neel. Mope: Model perturbation-baset privacy attacks on language models. In Conference on Empirical Methods in Natural Language Processing, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Cheng- hao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muh- tasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patell, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmny, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Ku- nakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder: may the source be with you! ar Xiv preprint ar 0tte:2305.066611 2023b.
Yucheng Li, Frank Guerin, and Chenghua Lin. Avoiding data contamination in language model evaluation: Dynamic test construction with latest materials, 2023c.
Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella- Béguelin. Analyzing Leakage of Personally Identifiable Information in Language Models. In JEEE Symposium on Security and Privucy, 2023.
Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafford, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groenveld, Iz Beltagy, Hanneneh Hajishirz, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. technical report, 2023. URL https://palome: allen.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Kirg-Kirkattrikk. Membership Inference Attakss against Language Models via Neighbourhood Comparison. In Findings of the Association for Computational Linguistics (ACL), 2023.
Matthieu Meeus, Shubham Jain, Marek Rei, and Eves-Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models. arviv preprint ar arXiv:2310.1500) 2023.
Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah A. Smith, and Luke Zettlemoyer. SILO Language Models: Isolating Legal Risk in a Nonparametric Datastore. Regulatable ML Workshop at NeurlPS, 2023.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Birk-Kirkpa,rick,, and Reza Shokri. Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks. In Conference on Empirical Methods in Natural Language Processing, 2022a.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Kirg-kiikppttrick. An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models. In Conference on Empirical Methods in Natural Language Processing, 2022b.
Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Advances in Neural Information Processing Systems, 2023.
Yuta Nakamura, Shouhei Hanaoka, Yukihiro Nomura, Naoto Hayashi, Osamu Abe, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. KART: Parameterization of Privacy Leakage Scenarios from Pre-trained Language Models. Xiv preprint arXi0:2001.00036, 2020.
13

---

Published as a conference paper at COLM 2024
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(1),, 2021.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz. Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Paaella Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Iseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024.
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models. ar arv preprint arXi::2310.17625 2023.
14

---

Published as a conference paper at COLM 2024
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl blog, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arviv e-prints, 2019.
Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, Yann Ollivier, and Hervé Jégou. White-box vs black-box: Bayes optimal strategies for membership inference. In Interna- tional Conference on Machine Learning, 2019.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeuriPS EMC² Workshop, 2019.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. Regulatable ML Workshop at NeurlPS, 2023.
Reza Shokri. Auditing Data Privacy for Machine Learning. In USENIX Security Symposium, 2022.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In IEEE Symposium on Security and Privacy, 2017.
Luca Soldaini and Kyle Lo. peS2o (Pretraining Efficiently S2ORC) Dataset. Technical report, Allen Institute for AT, 2023. ODC-By, https://gittub..com com/ wttpp///w..oooaaaaaaaaaa//ae/e/e allenai/pesn
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Ian Magnusson, Jacob Morri- son, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Pete Walsh, Hannaneh Hajishirzi, Noah A. Smith, Luke Zettlemoyer, Iz Belt- agy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: An Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. Xrv preprint, 2023.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. In Advances in Neural Information Processing Systems, 2023.
Anshuman Suri and David Evans. Formalizing and estimating distribution inference risks. In Privacy Enhancing Technologies Symposium, 2022.
Xinyu Tang, Saeed Mahloujifar, Liwei Song, Virat Shejwalkar, Milad Nast, Amir Houmansadr, and Prateek Mittal. Mitigating membership inference attacks by self- distillation through novel ensemble architecture. In USENIX Security Symposiumi 2022.
MosaicML NLP Team. mot--7 A new standard for open-source, commercially usable 1lms. www.mosaicml. http:immm..ooomooggmmmttt/// 2023. Accessed: 2024-01-11.
Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memoriza- tion without overfitting: Analyzing the training dynamics of large language models. In Advances in Neural Information Processing Systems, 2022.
Together AI. Redpajama: open dataset for training large language models, 2023. URL htweeehww..cceeeeeeeeeeeeeeeeeee com/ wtwottoooeeeeeeeeecc cowp ama Data
https://github.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
15

---

Published as a conference paper at COLM 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Jonathan Tow. Stablelm alpha v2 models, 2023. URL https://huggiingaace.oo
stabilityaiistaalellm-b waweeelmmaaeeapppaeee -1----33ppa--22
Lauren Watson, Chuan Guo, Graham Cormode, and Alexandre Sablayrolles. On the importance of difficulty calibration in membership inference attacks. In International Conference on Learning Representations, 2022.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and lon Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023.
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. Enhanced Membership Inference Attacks against Machine Learning Models. In The ACM Conference on Computer and Communications Security, 2022.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), 2018.
Sajjad Zarifzadeh, Philippe Liu, and Reza Shokri. Low-cost high-power membership inference by boosting relativity. ar arXiv preprint ar XrX::2312..3322, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
16

---

Published as a conference paper at COLM 2024
A
Implementation details
A.1
MIMIR
We release our codebase Python package (available athttpp://githbb http:///ittuu...mm..aaaa.aa. cww/ com//aagroot42// mimir), complete with documentation and tests, for implementing and evaluating mem- bership inference attacks for language models. The data used in our experiments also available via HuggingFace /https://huggii features of the package include:
bo//datssetsiiaggroottaa
Some
A base attack class that provides bare-bones code and helper functions that can be easily used in implementations of both existing and new attacks.
Data processing utilities to filter and cache data for membership evaluation, from both provided and available sources.
Support for vast array of models that can be used as target reference models.
The entire codebase works with modular configuration files, allowing multiple experiments to be run simultaneously with no edits to the code itself. We used Python 3,9.7 for our experiments, with PyTorch 2.0.1. Our experiments were executed on a mix of machines, with GPUs ranging from RTX6k to A100.
A.2 Additional Target Model Details
PYTHIA-DEDUP. Both the PYTHIA and PYTHIA-DEDUP model suites provide intermediate checkpoints for each model. For experiments targeting the PYTHIA-DEDUP model, the PYTHIA-DEDUP model is trained for greater than epoch, we select the checkpoint that most closely matches the epoch mark over the deduplicated Pile. We decide this is checkpoint 'step99000'. For experiments targeting the non-deduped PYTHIA models, we use the final checkpoint, which sees just under one (>v 0.9) epoch of the original Pile.
SILO. The models from the SILO suite (Min et al. 2023) consist of 1.3B-parameter trans- former LMs based the OpenLM implementation of the LLaMA architecture (Touvron) et al. 2023a). These are trained for multiple epochs on the Open License Corpus, which consists of permissively-licensee text data classified either public domain (PD) texts, permissively licensed software (SW), or under an attribution license (BY). We target the SILO-PDSW model (alongside its intermediate checkpoints) trained on only texts classified as PD or SW for domains contributing less than 5% of the data upsampled by a factor of 3x (which includes HackerNews and DM Mathematics).
DATABLATIONS. The DATABLATIONS suite Muennighoff et al. 2023 is a large collection of models trained study scaling laws in cottt-onntra regimes. They vary in the extent of data repetition and compute budget, ranging up to 900 billion training tokens and 9 billion parameters. For the epoch experiment, we choose the with each seeing total of 55B tokens from the C4 dataset across their training runs. These models vary the number of epochs their training subset is seen, ranging from one to 14 epochs. They also offer model trained for 44 epochs, which we decided to leave out of evaluation.
subset of models,
GPT-NEO. is collection of 125M-, 1.3B-, and 2.7B- 77--parameter models of similar architecture to the GPT-3 model family. These models are trained on the Pile for about 300B tokens, similar to the PYTHIA suite. This model suite is precursor to the GPT-NEOX (Andonian et al. 2023) model architecture, which PYTHIA-DEDUP and PYTHIA are built on. Noticeable differences include the tokenizer used per model suite, with the GPT-NBOX allocating additional tokens to whitespace characters, as well as intended training settings, with GPT-NEO geared towards TPU training and GPT-NEOX GPU training.
OLMo. The OLMO model suite (Groeneveld et al. 2024 is suite of open language models trained the DOL D et al. 2023) dataset. OLMo models currently available
17

---

Published as a conference paper at COLM 2024
include 1B- and 7B-parameter variants trained on 3T and 2.5T tokens, respectively. While our preliminary results just target the final checkpoints, the OLMO suite is similar to the PYTHIA suite in that intermediate checkpoints and exact training order are fully open.
A.3 Benchmark Details
We sample 1,000 members and non-members from each target domain from the Pile train and test sets, respectively. We do the same for the aggregate Pile experiment, except we sample 10,000 members and non-members each from the complete Pile train and test sets. We sample documents greater than 100 words and truncate them up to 200 words from the beginning to create our benchmark examples. Previous work (Shi et al. 2023 observes that sample length correlates with performance, so we bound the sample length to reduce its impact while picking reasonable threshold so that our samples are likely to contain ample signal. While further increasing the length of samples could yield greater MIA performance, such an experiment is orthogonal to the ones we conduct; inherent differences in LLM training and MI evaluation would still impact evaluation on longer texts.
We follow the same pipeline when generating the benchmark for targeting the DATABLA- TIONS models, picking members and non-members from the C4 train and validation sets, respectively.
For our additional decontamination on Pile benchmarks, we follow Groeneveld et al. (2023), which uses a bloom filter to check for æ-gram inclusion. We keep the default filtering settings of n 13 and a threshold of S 80% overlap. Further details about setting up the bloom filter can be found in Appendix B.1
A.3.1 Training Data Size Benchmarks
For each model, we pick checkpoints every 5000 steps ending at step 95000, with each step corresponding to 1024 samples of length 2048 tokens. We also include checkpoints at step 1000 and step 99000, the closest checkpoint to the step where one full epoch of the deduplicated Pile was seen. For each checkpoint, we use the same non-member set for evaluation consisting of 1000 samples sampled from the entire Pile test set. We then construct a member set for each checkpoin for the checkpoint at step we sample 1000 random samples from documents seen within the range step [n 1 1000, n). EleutherAT provides random seeding for deterministic training data order across the PYTHIA-DEDUP training runs, which we use to determine the seen document order. This allows us to determine which documents to sample from for given step range. For both members and non-members, we sample with the same criterion as the general experiments above.
A.3.2 Temporal Benchmarks
For the temporal Wikipedia benchmark non-members, we collect samples from the Real- TimeData "wikitext latest" dataset (Li et al. 2023c). This yielded Wikipedia articles created between the week of August 12, 2023 till the week of January 8, 2024 T0 We then follow Pile processing steps by simply appending the article titles to the front of each respective article with nn" Members are sampled from the Wikipedia subdomain of the Pile training set. Members and non-members are then sampled with the same criterion as the general experiments.
For the temporal ArXiv benchmarks, the member set for each benchmark is fixed and sampled from the ArXiv subdomain of the Pile training set, which consists of papers posted prior to July 2020 (Gao et al. 2020). For non-members, we use the ArXiv API again following
"deally, the member set should be fixed, which could be done by performing multiple training runs and injecting the fixed member set at various steps. However, this is computationally expensive. Furthermore, because of how the data is shuffled, we'd expect the difficulty of the member set to be reasonably consistent across our samples
Note that while the articles are created in the recent time frame, the contents of the Wikipedia page aren't necessarily about recent people, or events
18

---

Published as a conference paper at COLM 2024
LOSS
Reference based
Min-kSs Prob
Zib
Neighborhood
AUC ROC
0.70
0.65
0.60
2020-08
Timestamp
Figure 6: MIA performance across benchmarks where non-member data is selected from ArXiv preprints created during increasingly later months past the target model's knowledge cutoff. Timestamps are formatted as year-month. The target model is PYTHIA-DEDUR-122 In general, MIA performance increases as the temporal shift of non-members increases
Li 2023c) to collect ArXiv preprints from specific months: August 2020, January 2021, June 2021, January 2022, June 2022, January 2023, and June 2023 II
We then apply the same processing steps used in the Pile (Gao et al. 2020). This mainly involves converting the latex sources for given preprint into single Markdown file, and then filtering out documents such as those with conversion errors. For each month range, we sample non-members from processed files in the given date range. By sampling non-members from successively later time ranges after the Pile ArXiv cutoff date, we also seek to explore how greater temporal shift impacts MIA performance. We again sample both members and non-members with the same criterion as in the general experiments.
Difficulties in reproducing and analyzing existing works. While we follow similar method of mem-er candidate selection for our temporal experiments as prior works unable to reproduce their settings for analysis for two main reasons: 1) their exact non-member candidates and/ or pipelines to reproduce the non-member candidate selection are unavailable or 2) target models used such as LLAMA (Touvron et al., 2023a) do not release their training data to conduct n rt-gram or other distributional analyses.
such as Shi et al. 2003 and Meeus et al. 22023), we
A.4 Attack Details
MIAs consider target model M, which outputs a probability distribution of the next token given a prefix, denoted as ((x;|xx...X;1;;>>> Their goal is to model f(x:M), which outputs score for target sample X1...,, with tokens. This score is then thresholded to determine the target sample's membership in the training data of M.
LOSS (Yeom et al. 2018 Carlini et al. 2019) considers the model's computed loss over the target sample:
f(x; L(x
Reference-based (Sablayrolles et al., 2019: Watson et al. 2022) attacks assume access to reference model Myet, another LM trained on a disjoint set of training data drawn from similar distribution. In practice, assumption of disjoint training data is impractical.
This slightly differs from the Pile ArXiv data collection, which uses the ArXiv bulk access through S3. However, we believe both ArXiv bulk access and API should yield the preprints in the same regardless.
19

---

Published as a conference paper at COLM 2024
Thresholdngg Benchmark
1%
5%
10%
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
1,OSS
Ref
min-c
ziib
2020-08
4.2
3.7
13.4
2021--01
3.7
3.9
3.5
3.5
11.4
15.8
13.5
10.4
21.7
27.0
24.6
17.5
2021-066
4.2
5.4
16.0
13.6
25.5
25.5
29.5
2022-01
4.5
4.
5.3
4.1
14.4
16.3
14.6
12.7
24.5
27.0
28.7
22.0
2022-06
2.8
3.9
3.1
2.5
10.3
18.1
13.1
10.7
23.4
27.8
25,4
20.6
2023-01
8.3
3.1
23.5
13.5
36.1
26.3
2023-06
3.8
9.4
5.5
5.8
15.6
22.7
19.1
14.1
26.3
37.3
27.8
22.2
Temporal Wiki
9.8
7.5
10.3
7.9
23.8
22.8
24.3
17.6
30.0
34.1
35.0
22.8
Table 5: FPR (%) on non-members from the Pile (original; not temporally shifted) various attacks when using a score threshold that achieves 1, 5, or 10% FPR on the temporally- shifted ArXiv (for varying levels of temporal shift) and Wikipedia benchmarks. The target model is PYTHIA-DEDUP-1221 FPRs on the original non-members are much higher then the thresholded FPR on the temporally shifted benchmarks, indicating that such thresholds may be moreso classifying temporal shift rather than member and non-members.
Empirically, using an LM that is different from M has been a reasonable choice and was used in prior work (Kandpal et al. 2022 Watson et al. 2022). The attack considers the membership score of the target sample by M relative to the membership by M to calibrate the target model's score given difficulty estimate through the reference model's score, with goals to improve precision and reduce the false negative rate. For our experiments, we use LOSS as the uncalibrated membership score such that, for the reference-based attacks,
(x: L(x M
This method exactly follows the method from Watson al. 2022 and is also largely similar to the offline Likelihood Ratio attack (LiRA; Carlini et al. (2022) although LiRA uses many reference models (often trained shadow models).
Zlib Entropy (Carlini et al. 2021) functions similarly to reference-based MIA, using the zlib compression size of a sample local difficulty threshold per sample:
M)
f(x;M) = zlib(x)
where zlib(x) is the length in bytes of the zlib compressed sample.
Neighborhood Attack (Mattern et al. 2023 assumes access masking model, and operates by generating "neighbor" texts x to a given text sequence by using the masking model to replace percentage of randomly selected token spans while still maximizing the neighbor's likelihood. If the sample's s loss is considerably lower than the neighbor's 's losses, the difference attributed to the target model overfitting the sample, and the sample is considered training member. Formally, we have
texts
1 "
f(x: = L(;
- (); "
We use BERT (Devlin et al. of 5%.
2019 as our masking model of choice, with a masking percentage
2023 is based on the intuition that non-member examples tend Min-k% Prob (Shi et al.
to have more tokens assigned lower likelihoods than member examples do. Given sample x Xy, and hyperparameter k, let min-k(x) be the set formed by the 4% of tokens in with minimum likelihood. We then have
f(x:M) mx)] xxEmin-k(x)
Σ - log(p(x; x;,.,,;;;.))
We experiment with multiple different & E {10,20, (0,,20,3,, 40,50) as suggested in Shi et al. 2023 but settle on 20 for our experiments.
20

---

Published as a conference paper at COLM 2024
We compute the performance of each attack based 1,000 bootstrap samples of the bench- mark and report the average AUC ROC and %FRR over the bootstraps.
MIAs involving meta-classifiers, While many recent works study MIAs that perform membership classification through claa-classifiers trained on features extracted from ground subset of comberrnnn--ettee as access to such a subset can be difficult to guarantee in practice, especially as the inclusion of samples in pre-training corpora becomes more ambiguous these corpora continue to expand.
data, primarily focus blackbox attacks
A.5 Reference Model Choices
We choose a diverse set of reference models to experiment with. For the aggregate method over all reference models, we take the average of the scores per reference model for target sample 12
We report results for complete ablation reference model choice in Table 6
GPT-2 (Radford et al. 2019) is suite of pre-trained transformer trained on a large dataset of around 4UGB of web text, likely overlapping with the Pile. We use the GPT-2-small variant with 124M parameters.
DISTILGPT2 (Sanh et al. 2019) is smaller 82M-parameter model trained with the supervi- sion of GPT-2-small using knowledge distillation.
2022 is suite of open-sourced pre-trained transformers that trained on a curated pre-training corpus including several datasets from the Pile, such as Wikipedia, DM Mathematics, and HackerNews. We use the 1.3B-parameter variant.
OPT (Zhang et al.
As mentioned in Appendix A.2 GPT-NEO (Black et al. 2021) is another suite of pre-trained transformers designed using EleutherAl's replication of the GPT-3 architecture. These models are trained on the full Pile for a similar amount of tokens as PYTHIA (~ 300B), though the data seen may not necessarily be in the same order as the PYTHIA models. We use the 1.3B-parameter variant.
SILO-PDSWBY (Min et al. 2023) is a 1.4B-parameter transformer pre-trained on all types of permissively licensed data in the Open License Corpus. The training data consists of certain Pile domains such as HackerNews and DM Mathematics.
LLAMA Touvron et al. 2023a) is a collection of large, open-sourced pre-trained LMs ranging size trom 7B to 65B parameters. The pre-training corpus is on the scale trillions of tokens, much larger than the Pile, and likely has significant overlap with the Pile. We use the 7B-parameter variant.
STABLELM-ALPHA-T (Tow 2023) is set of open-source pre-trained LMs also trained on a large pre-training corpus with trillions of tokens. Training is conducted in two stages, with the first stage seeing 1 trillion tokens of mixture of data from sources such as RedPajama (Together Al 2023) and the Pile, with an emphasis on refined web text. The second stage trained on TO0 billion tokens with higher context length, increasingly sampling naturally long texts and adding the StarCoder (Li al. 2023b) dataset. We use the 3B-parameter variant.
We also experiment with the non-deduped PYTHIA-DEDUP-1.44 model as a reference model to see how using smaller version of the target model (same architecture and training data order) impacts reference-based attack performance (Carlini et al. 2021]
A.5.1 Stablelm- Staleem--Bsse-llhha-BB-2 Performance
We speculate that the slightly higher performance with the reference model, even though its pre-training corpus has high overlap with the Pile, is
as
Note that the scores over different reference models may not be directly comparable due to the reference models having different tokenizers. This may contribute to the poor performance of this naive ensembling method.
21

---

Published as a conference paper at COLM 2024
Wikipedia
Pile CC
Params
Grr2
DISTI
N10
GP12
OPT
Nso
Siro
IIAMA
PvTHA
160M
.98
.488
-504
.457
.480
1.48
5013
505
.507
5000
502
.521
.544
.476
.523
.007
.513
500
-16
.504
.525
.4%6
2.8B
532
-339
.565
.326
-266
.521
-20
5044
6.9B
.10
507
.177
518
516
.36
.571
.301
538
520
.542
525
.31
.530
.544
.40
12B
514
510
522
528
.466
.579
.377
.588
525
.555
.53
.411
.545
.582
.55
PubMed Central
ArXiv
# Params
GP12
DISTIL
OFT
NBo
SILO
LLAMA STABLE PYTHIA
GP12
DisTIL
OPT
NEo
SILO
LLAMA
STABLE PYTHIA
160M
.495
.491
.515
.511
513
.515
.516
.497
.523
.188
.516
.800
.4%
.492
.866
.472
1.4B
.433
.991
.514
.517
514
.515
.530
.303
.529
.324
.523
.501
.512
.506
.510
484
2.8B
.944
.422
.533
.518
515
.518
.536
.300
.534
.288
.528
.524
.522
.516
.531
.228
6.9B
.499
.46
.599
.527
-500
-30
.552
.326
.40
532
.534
.539
.31
.528
-38
.354
12B
504
498
.233
.531
524
538
-559
.33
546
538
.541
.555
540
.538
.555
.581
DM Math
HackerNews
Params
.89
.509
.502
.523
.496
.496
.398
.456
.490
.666
1.4B
.87
.85
.509
.496
.485
.503
.512
.496
.508
.509
.511
.496
.401
.504
.514
483
2.8B
485
486
.511
.003
483
.500
-004
.509
-521
522
.529
.534
.21
-521
.549
527
6.9B
485
485
.510
.999
.502
5088
.497
.25
326
.344
.536
436
.531
542
12B
487
486
.514
.504
.885
.502
.12
.503
.534
533
.545
.559
.45
.545
.565
.511
Table 6: The effect of the choice of reference model to PYTHIA-DEDUP models across various domains. The reference model yielding the highest performance, per target domain and target model, is bolded. ROC-AUČ values reported.
because 1) larger target model such the PYTHIA-DEDUP-122 model may considerably overfit certain member samples and 2) TTABLELM-BASE-ALPII..
is trained on much larger corpus, which helps it generalize well and achieve similar losses the target model on the non-member data. As result, member samples are more likely to have greater magnitude of difference between the target and reference model losses compared to the difference between losses non-members.
A.6 Results with GPT-Neo models
We repeat our experiments with the GPT-NEO family of models. debonsseesss similar trends targeting the GPT-NEO models as seen when targeting the PYTHIA model family (Table [ Table 12), such as MIA performance generally increasing the target model size increases. In general, performance against the GPT-NEO models is similar, if not lower than, performance against the PYTHIA-DEDUP and PYTHIA models when comparing similarly sized variants. In some domains such as HackerNews, the best performing MIA differs between target models (Min-k% Prob for GPT-NEO, reference-based for РYTHIA- DEDUP), though marginally.
A.7 Results with OLMo models
We perform preliminary experiments targeting the OLMO family of models. Benchmark con- struction is similar to what is detailed in Appendix A.3. However, we sample from domains that make up DOLMA (Soldaini et al.) 2023), namely Wikipedia, C4, Reddit, Common Crawl, and peS2o (Soldaini Lo 2023). These are similar domains used in Pile. We note that peS2o consists of both abstracts (s2ag) and full papers (s2orc), and evaluate them as separate domains. To get non-member, we use held-out DOLMA data from PALOMA (Magnusson et al. 2023).
Table 8 also demonstrates generally near-random performance trends, with both the 1B- and 7B TB-parameter model variants exhibiting similar performances. We speculate that, due to the incredibly large amounts of training data (3T tokens for 1B-parameter model, 2.5T tokens
13 Also target models that are domain specific like DATABI ATIONS trained on less diverse corpus like SILOO
22

---

Published as a conference paper at COLM 2024
Wikipedia
Github
Pile CC
Pubmed Central
Params
LOSS
Ref
min-E
zlib
1,0S
Ref
min-k
zlib
1OSS
Ref
min-k
zlib
1,085
Ref
min-k
ziib
125M
5044
.492
.511
.582
.500
.506
.499
1.38
5100
.331
.506
.577
.681
.570
.681
.696
.500
.517
.503
.501
.4%
.99
.499
.477
2.7B
513
.545
.513
.199
.570
.700
.712
.504
.531
.507
498
.507
499
ArXiv
DM Math
HackerNews
Pile
Params
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
125M
.577
.944
.03
.501
492
.522
493
.484
489
480
.505
.496
.022
.507
.005
503
1.3B
.511
5007
499
505
.509
507
2.70
515
-220
517
.500
.86
.509
.92
.81
.502
512
.516
5003
507
.519
511
.509
Table 7: AUC ROC of MIAs against GPT-NEO across different datasets from the Pile. The highest performance across the different MIAs is bolded per domain. Similar to PYTHIA- DEDUP, MIA methods perform near random (< .55) in most domains.
for the 7B- 7B-parameter model), performance across different model sizes begins to converge to near-random performance even with such distinct model sizes. Interestingly, the Reference- based attack using to calibrate the PYTHIA models, reinforcing the difficulty in finding suitable reference models for different LLMs.. We also observe many settings where MIA performance is considerably less than 5,, suggesting that the MIAs are more likely to predict members as non-members, and vice versa. More investigation is needed understand such behaviors on specific domains such as s2ag from peS2o in the DOLMA data.
performs much worse than when used
Wikipedia
C4
Reddit
Params
LOSS
Ref
min-k
zlib
LOSS
min-k
zlib
LOSS
Ref
min-k
zlib
1B
484
.10
.455
.510
.515
479
.520
464
.495
478
.470
7B
481
.48
.493
.500
.516
.49
.520
.514
4633
.501
.480
.469
Common Crawl
s2ag
s2orc
Params
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
LO0S
Ref
min-k
zlib
1B
.509
.422
.517
.511
.884
480
.500
.463
7B
.48
.410
.505
.500
.65
.483
.455
.406
.491
.507
.503
.470
Table 8: AUC ROC of MIAs against OLMO across different datasets from the Dolma dataset. The highest performance across the different MIAs bolded per domain.
B
n-gram Overlap Details and Takeaways
B.1 Measuring n-gram Overlap
We create bloom filter following Groeneveld et al. (2023). Due to the scale of the Pile training data and limited memory, we shard the bloom filter. In our construction, we split the training data in half, resulting in two bloom filter shards. Since each shard only sees half of the training data, to check for n-gram inclusion across the entire Pile, we check for containment both of the sharded bloom filters, counting an n-gram included only it is included in at least one of the bloom filters.
For each shard, we configure the bloom filter according to the data size such that the false positive rate of the bloom filter is less than 1% (0.6%). Then, for each document, we tokenize the filter by using a striding window over
at the word level. We then add n-gramš words at time with stride of 1. We use the same method of gathering n-grams when checking the mon-members for n-gram overlap.
B.2 Reference-based Attack Performance
Table showws that, interestingly, reference-based MIAs have a noticeably smaller increase in performance compared to non-referenced-bbsse MTAs for domains such as GitHub or
23

---

Published as a conference paper at COLM 2024
PubMed Central under n-gram overlap thresholding. We speculate that, since numerous low 1--rrm overlap non-members are outliers to the relevant domain, these non-members will also be outliers to the similar overlapping data seen by the reference model. As result, even though these non-members may yield higher losses from the target model, we see similar high losses for the reference model as well, which makes the difference between target and reference model loss for non-members and members relatively less distinguishable compared to signals from the other attacks.
At the same time, domains like Pile CC do not see this dampened performance, likely because the 20% threshold in the case of Pile CC not sufficient to select outliers, as samples from this domain have naturally low n--ram overlap. Another case where the reference- based attack seems to avoid this observation in the temporally shifted non-member setting for both Wikipedia and ArXiv despite the temporally shifted non-members being more out-of-distribution relative to the Pile Wikipedia and ÁrXiv distributions, respectively. We speculate this is due to the reference model of choice, which has not only been trained on a corpus with high overlap with the Pile, but also trained datasets that capture more recent data such RedPajama-Diae Data-1T (Together AI 2023 which contains Wikipedia and ArXiv samples from a much more recent cutoff date (1.e., RedPajama uses the 2023-03-20 Wikipedia dump), allowing it to generalize better over the temporally shifted non-members and avoiding a shift towards higher losses that weaker or older reference models might experience.
Attacks like LOSS or Min-k% Prob do not utilize any external signal or difficulty calibra- tion, and thus rely exclusively on signals from the target model for member classification. Calibration-based methods like zlib and reference-based attacks, on the other hand, account for the inherent "difficulty" of seen sample. Thus, in situations where the non-member data is significantly out of domain, even for a reference model or calibration method, it is likely that the signals from the target model and difficulty calibration would cancel out, leading to a weakened MIA signal. On the other hand, difficulty calibration can further boost MIA signal in settings where the member data is inherently more likely to be mem- orized, such in 33..11 where reference-based performance in low training data size and high effective epoch count settings, with perfor- mance being further amplified in the extremes of both settings. Thus, MIA baselines for new MIAs should include both kinds of methods: calibration-based and calibration-fres Having baseline coverage for both styles of MIA can help uncover inherent characteristics of the evaluation setting such as unintentional member mooneeembrr distributional shift or overfitted target models that influence MIA performance and also paints holistic picture with regards to what MIAs are most suitable for specific attack settings.
attacks yielded considerably higher MIA
B.3 GitHub as an Outlier
MIA performance in the Github domain even without thresholding is notably higher than that in other domains, with the best method (zlib) achieving an AUC ROC of ~ .70. We speculate this is not
As seen in Table 1
0.45
Original
0.45
Decontemirated
0.10
0.40
Proportion of Data
0.35
0.35
0.30
0.30
because the GitHub domain, or code in gen- eral, is an easier domain to attack, but be- cause the presumably reasonable decontam- ination threshold of <80% 13-gram overlap threshold only captures small percentile of non-members as GitHub is naturally very high overlap.
0.25
0.15
0.15
0.10
0.10
0.05
0.05
0.00
20
60
100
0.00
20
40
11
60
80
100
% 7-gram Overlap
Figure 7: 7-gram overlap of GitHub non- member data before and after 13-gram decon- tamination at threshold < 80%.
We speculate a large factor contributing to the high overlap is the repetitive nature of code, such as copyright notices, function definitions, and syntax like HTML tags. Figure demonstrates how our decontamination threshold impacts the 7-gram distribution of non-members. Non-members under our decontamination threshold are more likely outliers to the GitHub domain (see Figure 15
24

---

Published as a conference paper at COLM 2024
for an example of such outlier). The additional n-gram overlap threshold experiments ($3.222 only exacerbate the impact of thresholding, which leads to notably higher MIA performance.
Such observations indicate why lexical non-member boundaries may lead to ambiguous interpretations of MIA performance in high-overlap domains. Here, using semantic differ- ences between samples to draw non-member boundaries may be key to better understanding membership leakage in such domains.
B.4 Temporal Shift as Change in n-gram Overlap
Original
2020-08
2023-01
0.30
0.18
0.15
0.32
0.12
0.12
Proportion of
0.30
0.10
0.10
0.08
0.08
0.34
0.04
0.04
0.02
0.02
0.02
20
40
90
80
100
20
40
60
80
100
20
40
60
80
100
% 7-gram Overlap
Figure 8: Distribution of n-gram overlap for non-member ArXiv preprints sampled from the months 2020-08 and 2023-06, respectively, We also plot the n-gram overlap distribution of the original Pile ArXiv non-members. Between the original non-members and both temporally shifted non-member sets, the temporally shifted non-member n-gram overlap distributions are considerably more concentrated at lower % n-gram overlap. The original non-members have average 7-gram overlap of 39.3%, while non-members from months 2020-08 and 2023-06 have 7-gram overlap of 22.7% and 20.5%, respectively.
Figure 8 reinforces our observations Figure as similar to the temporal Wikipedia setting, temporally shifted non-members from after the target model's knowledge cutoff date are concentrated at considerably lower % n-gram overlap than non-members from the natural ArXiv non-member set. This contributes to the greater MIA performance in general over the temporally shifted ArXiv benchmarks. However, we note that и--ram overlap distribution shift does not provide strong interpretation for the increase in MIA performance as non- members are increasingly temporally shifted. For example, the average 7-gram overlap of non-members from the month 2020-08 22.7% while the average for the month of 2023-06 is 20.5%. While there is a small decrease in average 7-gram for later non-members, the change quite small and doesn't clearly justify the considerable difference in MIA performance when evaluating on benchmarks using non-members from the different months (i.e., .723 + $.795 AUC ROC from the 2020-08 benchmark to the 2023-06 benchmark). We speculate other factors that contribute to this increase include changes in the distribution of topics (i.e., increasing popularity of research into LLMs) and the presence of specific identifying tokens (i.e., dates, references, new terminology). We believe such factors only further reinforce the need to carefully analyze MIA benchmark construction when evaluating MIAs to understand what signals are truly being captured.
C Characteristics of LLM Training
C.1 Recency of Member Samples
We explore how the recency of member samples seen in training impacts MIA performance. We follow the same setup as the training data experiment (A.3.1), but instead of evaluating the checkpoint at step n with the member data sampled from within steps (n 100, n) and the fixed non-member set, we fix the target model, only targeting the checkpoint at step 99000 for all the benchmarks.
25

---

Published as a conference paper at COLM 2024
1.48
2,88
6.96
128
LOSS
Referance based
0.60
0.50
60000
80000
20000
40000
100000
Min-k% Prob
Zlib
20000 40000 00000
100000
20000 40000
BOOOO 100000
Number of Training Steps
Figure 9: MIA performance for different member data sets sampled different training steps across 1 epoch of the deduplicated Pile pretraining corpus, visualized across different attacks. Target model is the PYTHIA-DEDUP reported. Performance on benchmarks with more recently seen members is higher, but gradually decreases plateau for less recently seen members.
checkpoint at step-99000. AUC-ROC
Figure demonstrates that, in general, member data seen more recently by the given checkpoint contributes to slightly higher MIA performance- We believe this supports existing work in LM forgetting (Jagielski et al. 2023), where observed patterns in recently seen training data are better preserved in the model parameters, while earlier seen data are less memorized.
We also note that the MIA performance trajectories seem to drop slightly more quickly for smaller models, though the trajectories across all model sizes seem to converge when evaluating member data from much earlier in the training run. We speculate this is result of larger models having more parameters, allowing them to capture more seen data before having to drop older knowledge.
We also note that, in the context of MIA against fine-tuning datasets, our results indicate that data seen during fine-tuning or continued pre-training may also be increasingly vulnerable due to how recent they are seen. This aligns with previous work demonstrating high MIA performance fine-tuning datasets (Mireshghallah especially relevant in practice since fine-tuning is a popular option to re-purpose large pre-trained models for varying downstream tasks such as commercial use cases, which often involves tuning with sensitive data.
al., 2022a Fu et al. 2023 This is
C.2 Number of Training Epochs
While experiments against the DATABLATIONS model (Figure 2 training data size setting, we also explore training data the target model sees increases alongside the effective epoch count by targeting the SILO-PDSW model and intermediate checkpoints. In Figure 10 for HackerNews, we observe MIA performance initially increases with more effective epochs, similar to the DATABLATIONS setting, but then begins to plateau or drop as effective epoch count continues to increase. DM mathematics, on the other hand, surprisingly decreases as the number of effective epochs increases. We speculate over factors that may contribute to these observations:
right) operate in a fixed realistic setting where the amount of
HackerNews, even when up-sampled for this variant of the SILO-PDSW model, still only makes up 5.9% of the training data (Min et al., 2023). In the first few epochs, when the total training data seen is low, the model can memorize the HackerNews samples. However, as the number of epochs increases, the target model may tend to
26

---

Published as a conference paper at COLM 2024
SILO-POSW HackerNews
SILO FDSW OM Mathematics
0.54
0.52
ROC
0.50
0.48
0.46
Number Training Epochs
Figure 10: MIA performance on target model SILO-PDSW as the number of effective epochs in which the member domain data has been seen increases. AUC-ROC reported. For HackerNews, performance pertorrmmneeeeeeo does increase with an increasing number of effective epochs initially, but begins to plateau or even drop with further epochs. For DM Mathematics, performance surprisingly drops with increasing effective epochs.
overfit data more from domains with greater representation. As the SILO model is on the smaller side with 1.4B parameters, we suspect the target model begins to memorize less of the HackerNews samples, leading plateau or drop MIA performance.
DM Mathematics also makes up only 3.5% of the training data. In addition, with DM Mathematics being dataset of mathematical problems, suspect that the abundance of tokens from concentrated token space (i.e., digits, variables) that are largely symbolic rather than semantic makes memorization of specific samples unlikely. Overall, it simply fails to perform well on such data even after multiple epochs (as observed when looking at model loss values for this data).
For both cases, further investigation is needed into the target domains and attack setting setup to better understand these counter-intuitive phenomena.
27

---

Published as a conference paper at COLM 2024
D Revisiting Membership through Semantic Similarity
We follow the same pipeline as $ but focus on generating member paraphrases that are semantically similar while preserving as much specific information from the original mem- ber sample as possible. To do so, we follow Yang et al. (2023) and prompt GPT4 (OpenA] et al. 2024) to paraphrase member samples perform this paraphrasing over the ArXiv, Wikipedia, and HackerNews domains; see Ta- ble for domain-specific prompis. In general, we don't focus on the lexical similarity of the paraphrases unlike the earlier semantically similar samples generated via masking and replacing a small percentage of tokens. However, with HackerNews, we do specify comment structure different from how HackerNews records were formatted for model training, a noticeable lexical difference.
difterent style (5 trials per member). We
Domain
Prompt
ArXiv
Please help me paraphrase the following text chunk from a research paper in different style. Importantly, for sentences containing specific details like mathematical definitions or proofs, only make minimal changes and ensure these details are included exactly in the paraphrase. If the paper includes a title or authors, please keep them in the rephrase. If not, please DO NOT make up title. Use similar number of words.
Wikipedia
Please help me paraphrase the following text chunk from Wikipedia in different but concise style. Importantly, for sentences contain- ing specific details, make minimal changes and ensure all details are included correctly in the paraphrase. Use similar number of words.
HackerNews
Please help me paraphrase the following conversation chunk from thread in HackerNews while maintaining the conversational style. Follow this structure for each comment in the thread: [user] [com- ment]. Ensure all user's comments are represented in the para- phrase. Make sure all details in each user's comments are included correctly in the paraphrase, such as links. Be specific and don't t generalize.
Table 9: Instructions used to prompt GPT4 to obtain paraphrased members.
We again visualize the score distributions between the paraphrased members, and original members and non-members (Figure I1.. We observe in general across both the LOSS and Reference-based attacks over the three domains that the paraphrased member score distri- butions are distinguishable from the original member and non-member score distributions but have noticeable overlap, similar what was observed with masking-bass neighbors. However, when we perform the FPR experiment (Table 10), we see that in high- confidence settings, the paraphrased members are likely to be classiTied as non-members. Both the LOSS and Reference-based attacks seem noticeably insensitive to such paraphrased neighbors.
semantic
28

---

Published as a conference paper at COLM 2024
Member
Non-Member
Modified Member
Whipestla
3.0SS)
0.40
3.
023
Proportion Data
0.15
333
0.05
2.0
Addiv (Rar)
Webbedd (Taf)
Fackerriass (Pet)
2.5
2.0
2.0
1.5
05
02
200
-30
-64
-08
-34
-6.
-23
-13
-10
-05
-0.3
0.0
Score
Figure 11: Distribution of scores for LOSS and Reference-based attacks for members, members, and GPT4-paraphrased (modified) members across ArXiv, Wikipedia, and Hack- erNews domains.
LOSS
Ref
Domain
1%
3%
10%
1%
5%
10%
ArXiv
0.1
0.5
0.7
Wikipedia
0.0
0.0
0.2
0.0
0.2
0.7
HackerNews
1.7
Table 10: FPR (%) on modified members (treated as non-members) when using a score threshold that achieves 5, 10% FPR on the original member and non-member data for the ArXiv, Wikipedia, and HackerNews domains. Modified members are generated by prompting GPT4 to paraphrase member samples with significant lexical difference. LOSS and Reference-bart based attack reported.
29

---

Published as a conference paper at COLM 2024
E
Additional Figures and Tables
Wikipedia
Github
Pubmed Central
Params LOSS
Ref
min-k
zibb
Ne
LOBS
Ref
min-k
zib
Ne
LOSS
Ref
mireé
zibb
Ne
LOSS
Ref
mürsk
ziib
70M
0.8
0.9
1.0
1.0
0.3
13.1
7.8
12.6
13.0
11.0
0.7
1.0
0.4
0.4
0.1
0.7
1.0
0.9
0.5
0.6
160M
1.1
0.8
1.2
13.5
12.3
0.5
0.4
0.4
0.7
1.4B
0.6
0.9
0.5
0.7
0.4
12.8
0.7
12.9
16.4
3.9
0.5
0.6
0.5
0.7
0.8
0.4
0.7
0.6
0.5
0.1
6.9B
0.6
0.6
0.4
0.6
0.5
12.9
0.6
13.1
16.8
6.1
1.0
1.4
1.2
1.3
1.0
0.8
1.6
0.8
0.3
0.8
128
0.7
0.6
0.6
0.7
1.0
13.9
14.2
17.4
4.9
1.0
1.7
1.1
1.5
1.0
1.0
1.5
1.3
ArXiv
DM Math
HawkerNews
The Pile
Params
1088
Ref
min-k
xlib
Ne
LOBS
Ret
min-k
zlib
Ne
LOSS
Rei
min-
xifb
Ne
LO8S
Ret
min-k
2lib
Ne
70M
0.8
1.0
0.7
0.5
0.9
0.6
1.3
0.5
0.7
0.8
1.1
0.7
1.3
1.3
0.5
2.2
1.4
1.8
2.1
2.0
0.8
0.4
0.2
0.7
2.4
1.3
1.40
0.3
1.0
0.2
0.4
0.7
0.8
0.8
0.6
1.0
1.7
0.7
0.9
1.2
0.9
0.8
2.4
144
2.4
2.3
2.3
2.8B
0.5
2.1
0.5
0.5
0.8
0.6
1.4
0.8
1.1
1.7
2.8
2.2
2.8
6.98
0.6
1.8
0.6
0.6
0.6
0.9
0.7
1.9
1.0
0.9
1.3
2.6
1.8
2.5
2.5
2.2
12B
0.6
2.5
0.6
0.5
0.9
1.0
0.5
0.5
0.9
0.8
0.7
2.3
0.8
0.8
1.4
2.7
18
2.6
2.6
2.2
Table 11: 1TPR@1%FPR of MIAs against PYTHIA-DEDUT across different datasets from the
Pile. The highest performance across the different MIAs is bolded per domain. In general, leakage in high confidence settings is low (< 3%). As with AUC ROC, GitHub an exception, still yielding considerably higher leakage with most a attacks. Unlike with AUC ROC, trends in performance are much noisier the high-confidence setting, with trends in model size and best-performing attacks in certain domains no longer holding, reinforcing the difficulty in determining best attack.
Wikipedia
Github
Pubmed Central
Params
LOSS
Ref
min-k
zlib
LOSS
Ref
min-k
LOSS
min-k
zlib
LOSS
Ref
min-k
zlib
160M
503
.512
.411
.512
.57
639
.62
.674
496
491
.504
.497
.499
.513
506
500
1.48
.533
520
.670
.501
.510
.531
502
500
2.8B
518
.822
518
525
712
.653
.713
.723
.501
.577
.588
.501
500
.537
504
501
6.9B
.228
.688
536
730
.644
.733
.739
.507
.515
.066
.558
.006
12B
.335
.639
.544
.544
.740
.630
.743
.748
.511
.567
.517
.512
.533
.582
.523
.122
ArXiv
DM Math
HackerNews
The Pile
Params
LOSS
Ref
min-k
zlib
1OOS
Ref
min-k
zlib
LOSS
Ref
min-k
zlib
LOOS
Ref
min-k
zlib
.502
.509
.503
.511
.505
506
1.48
.155
516
.517
.509
.486
.512
.497
.452
.005
.522
.512
.504
.055
.522
.10
508
.519
.531
.505
.513
.524
.533
6.9B
.29
558
.35
523
.88
.511
.96
.481
.511
579
.366
.513
.14
.554
.522
.16
.534
.575
.566
.527
.510
.477
.451
.528
.606
.546
.517
.519
.569
.588
.50
Table 12: AUC ROC of MIAs against non-deduped PYTHIA across different datasets from the Pile. The reference-based attack uses MAA.LLL--AAA--LLHHA model. The highest performance across the different MIAs is bolded per domain. Per- formance follows similar trends seen when targeting the PYTHIA-DEDUL models, but performance is, in general, marginally higher. Due to computational limitations, we leave out evaluations for the Neighborhood attack, but expect similar trends.
the reference
30

---

Published as a conference paper at COLM 2024
Normal GEHUD
De De-duplicated
Wipoooo
0.50
Ar3iv
Hocker Nows
0.55
0.55
0.35
L055
0.75
0.70
0.50
0.0
0.60
0.60
0.85
0.50
0.60
AUC ROC
Mil-k% Inobo
0.80
0.35
0.55
0.70
0.63
0.45
10
12
0.60
10
12
0.45
0.48
12
0.00
0.55
0.55
0.55
20b
0.75
0.70
0.50
0.50
0.50
0.60
0.45
0.45
(B)
Figure MIA performance as model size increases over select domains for various other attacks. We additionally plot the AUC ROC trajectory against the non-deduped Pythia suite for comparison. Similar to the reference-based attack, increasing model size slightly boosts MIA performance while deduplication decreases performance.
128
10SS
Minkks
0.78
1.0
1.0
2.4
150
10000)
20303 40300
60000
A000
20000
20000
400M
50000
50000
10000
Training Steps
Figure 13: MIA performance as the amount of training data seen increases across epoch of the deduplicated Pile pretraining corpus, visualized over range of model sizes for various attacks. We use the training step as unit for the amount of training data seen, with 1 step corresponding to seeing 2097152 tokens. AUC-ROC reported. Similar to the reference-based attack, for all attacks, performance drastically increases before gradually decreasing as the amount of training data seen increases.
31

---

Published as a conference paper at COLM 2024
Wilipda
Gthub
Pie
PusHidd Central
0.50
0.068
0.40
0.66
0.00
0.04
0.03
0.20
0.00
Proportion Data
0:10
8.00
0.00
AudiV
DN Mathemaatccc
Itactertiss
Full Pile
0:50
0.00
0.30
0.08
0:30
8:20
0.04
0.02
0.00
80
100
8.00
10
0.00
0.00
40
82
gram Overlap
Risediaa
PutMud Centrll
0.40
0.20
0.25
8.50
0.78
0.04
0.25
0.76
0.04
0.20
0.69
0.35
0.69
0.20
0.23
0.09
0.00
0.00
50
300
20
60
80
90
:00
20
40
85
00
100
20
80
100
103
0.00
0.06
0.00
0.00
0.04
0.28
0.04
0.00
0.50
0.24
0.00
0.53
300
102
% 7-gram Overlap
Willeed
0.40
GRhhbb
File OC
PubMed Central
0.25
0.03
0.30
2.50
0.25
0.50
0.30
3:30
0.28
0.3
3.20
0.08
Proportonn of Data
0.18
0.00
2:30
0:00
20
0.00
2.00
40
0:00
ArX
DM Mot lemariis
Hacker/News
Full
0.25
0.34
3.00
0.22
3.50
0.28
0.5
3:30
6.68
0.66
0.04
2.20
6:25
0.02
6/8
6:20
20
40
42
100
0.00
25
60
100
2.00
20
45
103
0.00
20
40
62
60
160
13-gram Overiap
Figure 14: Distribution of 1--rram overlap over all evaluation domains for n 4,7,13.
32

---

Published as a conference paper at COLM 2024
hmp /bberrsss wwweew com wtw. of haw.:/liiiiaiiiillll--ae-eeeeeee htw.://pppppleeeeeeeeeeeeeeeoeee www--aoooddigggeeoeooooooo 31
dgg camus souplofu Y4g
wackg to Seame muucarggng Grappe grmpp obcomndg e compaoorag seggum alseen zooh cnbhebaaaa eugasao bbernge) yuugh sarin 98 sumwing afe iy "dd papiguucaafpp aogdl aupphe To si op's gree oig sygoos cosff cag momes frr jugrabssucioo aon oog kes moupalgge cirgle coogine y0) queaash pmpgueono dhof (o93s pudi o gpooo magonn
aone oif
wwm woodd janxioonug
udcig iy gaazor rouecrggo droo mooionin di Sp ig compe ong
sangurotee en zod cog (B5rt)
faam sage A. 1oo44.ngg0 arrgli golygnt mo oi Eaeggeemen iuposi mueepe sagueit my, dioge in in op pis gighios fo oppe the T Tuye Eow sudape ony oo vog op
aommay sherrgod yuan (
4dd coepggessmop' ga sug goggeg As yoopp "oopppp eppiloouu
"oy aggaggnnnn of g gaussog dipongd Sining sng for apmp cay onjonn enkp worgui supsopple op serocoio o sil aige e papgggg jogcour""
dleuse a o pioseeongu gig "44 no gailoni Gotd Aaugpogg y i coum maj e qagt ofna f dap GM cin geveappnne ison ieeg
uio yoosoai aygeent roost
to caangas
pgob coipgagiame uarong Aayey rpigle zof maleo oppppep jojogg afe pamigl atulcoup Semey un olpiiigg "fffrssooma oombo qugnalmuot ohg oigee Eipe dipjo d Com-noment
(egeo pampn ao dag (ujab uds jogroguuu e slgigg sugbagg iigg wogiy opye mogee ii márge for Baaninge annogeannooo so goooeeenug
e'jagion jusaafooo mmooo ion, die dig ang jog
"oig of syplese onn erp2ro Wom Knatt o uleappt sporo jeoezra mangg ? Sipo proougr mugorbeoypo surt) mosmpmabenee jughsesng ape aose alome iaft ponneeaooui ciale pidle-pjjooo ojemon amobafa oocoo ammorobe todo Fram (cigneouui corge emoppoop delyyam ong soipl mogg onppbogg
li saggrenffeg me mnegleggnng 19ong komn jabagen mengeommeeeen an smmeedd ( megyetg o single on oon (uojarggaorg ggnoag : ai 2 pogo yong 19/ quasic ony mound memds ( poesee onp dec dygedlouu jucn o
cousie ong ane jull conpereaolyy dengere in zopecco asng nolg come bnj loag laahe ággi osarm opupe eoaam je engui yo fownn seeat pdt pugda on ope an Brald qullogy dadkep agin ovae a momp Sepp apppoaggee wogge to mmpoa tobm è Seppe on to rose il copp din camioogeiin gosdre on Smminsear 1066 aase uni pojon siong 19doog
q dullo suyude suyute dliio on q "goznogoup J Bon ragropo tn cppaaon o cop cajo opc ispooo pony nppee aoe juminood
conuage ojun oy mn "Hly capalgge qun 'rog se oe ougrone Goo do d gooun bulogr "p's peaee coffr (eg toppiage qi gi vamoo quay?? pajop d égoo ropiiga
axme a afpjsty amm : sslemgg in geomn seangem aopeo jinoleuge
"clo o g rogo oo sagpesp juggur thegks Samoauche cloy apgenmuilss
desame same mamggroms migely well diof pespon ming e wobggooou
edriod cor copgeemee bi ope come oopecoofrgggo.o prrgeeayy 3 oge ope-ligg oo quagyydi obsigle queenaa jupela 14 og By coeepe mooo oneg peou im in of
o so upg woog (up famaage coog quoe czorp P gooino bogg dlouug
Namegea poggneooogggppo lmoo dipoe- qu joodfnd Afd papeddaapuat a: no aceueeee worp guye uop
Figure 15: A sample GitHub non-member outlier captured by the < 80% 13-gram overlap threshold. This sample is from language resource repository under Google, but is a clear outlier to the code-dominant GitHub domain
33


Reviews:
[{"review_id": "KyeaJSqlWS", "reviewer": "Reviewer_NqZv", "summary": "Summary: \nThe paper studies Membership Inference Attacks on LLMs. The authors perform many analyses to study the successes and failures of MIA attacks in the specific case of LLMs. In particular, the authors found that MIA performs surprisingly badly for LLMs. The paper provides an extensive suite of experiments to test various hypotheses to explain what makes MIA so hard for LLM. The paper concludes with some remarks about reframing the task of Membership Inference for the LLM scenario.\n\n\nOverall:\nI think this is a good paper worthy of publication.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback and for supporting our paper. \n\n*[Reason to Reject]* Compare AUCs of MIAs in other domains MIAs have been established in.\n\nThank you for the suggestion. We will add relevant references (and numbers) for other domains such as tabular [1, 2], image [3, 4], and text [5] data, along with results for LLMs with fine-tuning data [6].\n\n### References\n\n[1] https://arxiv.org/abs/1709.01604\n\n[2] https://arxiv.org/abs/1908.11229\n\n[3] https://arxiv.org/abs/2112.03570\n\n[4] https://arxiv.org/abs/2111.09679\n\n[5] https://arxiv.org/abs/2203.03929\n\n[6] https://arxiv.org/abs/2311.06062", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a well-executed paper. It measures behavior with a careful and meaningful experimental setup, proposes several hypotheses to explain the observations, and crafts controlled experiments to verify the hypotheses. The paper also positions itself with respect to the bigger field of MI and makes some recommendations for better definitions in the future. Overall, it was an insightful paper.", "reasons_to_reject": "The claim that MIA performs near random could downplayed a bit because Fig 1. seems to show performance statistically significantly above chance in most scenarios. To make the claim that MIA performs badly for LLM, it could be more convincing to compare the reported AUC with AUCs in order domains where MIAs have been developed and tested before.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iz7Qu5ibSh", "reviewer": "Reviewer_F61L", "summary": "The paper evaluates 5 recent membership inference attacks (MIAs) for LLMs and finds that most of them have near-random (AUC ROC < 0.6) performance. The evaluation was carried out on the Pythia suite of models trained on the PILE dataset. This setup provides an appropriate test bed since the knowledge of the train and test sets enables the fair evaluation of MIAs. It is suggested that the following characteristics of LLMs substantially reduce the performance of MIAs: (1) use of massive training data, (2) near-one epoch training for LLMs, (3) frequent overlap of members and non-members in natural language (non-members have high n-gram overlap with members).\n\nIt is suggested that the high performance of the previous MIA (Shi et al., 2023) was due to the temporal shift.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback and for supporting our paper.\n\n*[Reasons to Reject]* Lack of alternative solutions for practical assessment of privacy leakage from LLMs\n\nSection 5 proposes the alternate notion of approximate membership (and explores two concrete definitions: edit-based and semantic-based neighbors) to align the goals of MI with privacy settings, where auditors/adversaries can tune the approximate membership definition to fit their privacy use case [1]. We believe this is a promising direction to explore in future work.\n\n*[Q1]* Other overlap metrics\n\nWe chose n-gram overlap as it is an established method in decontamination [2] and is fast to compute. We also consider additional metrics, e.g., closest semantic embedding similarity, another decontamination metric [3]. We can also use MAUVE scores; however, these methods require generating embeddings, which with a large-enough model is computationally expensive, so we opt for the simpler and cost-effective n-gram overlap.\n\n*[Q2]* Smaller models from Pythia (< 160M)\n\nWe didn’t include them initially as our target was large LMs, but we will include it in the next revision. Results of main experiments on Pythia 70M-deduped on select domains:\n\nArXiv -\nLOSS: .494, Ref: .519, Min-k: .501, Zlib: .505, Ne: .504\n\nHackerNews -\nLOSS: .506, Ref: .505, Min-k: .493, Zlib: .503, Ne: .494\n\nWikipedia -\nLOSS: .497, Ref: .496, Min-k: .506, Zlib: .492, Ne: .490\n\n*[Q3]* GitHub as an outlier\n\nWe understand that this claim is currently insufficient. Instead, it would be more accurate to say “MIAs perform significantly better as the non-member distribution concentrates towards lower n-gram overlaps, diverging away from the natural n-gram overlap distribution of held out datapoints from the training data (against the rest of the training data)”.\n\nFor the case of GitHub, we refer to Appendix B.3. In Fig 7, we see the soft decontamination (Orig) already yields a significant distribution shift for the GitHub non-members. In other domains, a 7-gram overlap threshold of <= 20% is not as significant of a distribution shift relative to their natural 7-gram overlap distribution (Fig 14); MIA results over the Orig GitHub dataset are thus similar to those on the artificially thresholded benchmarks in other domains. We will update and clarify this claim accordingly in the revision.\n\n### References\n\n[1] https://arxiv.org/abs/2202.05520\n\n[2] https://arxiv.org/abs/2005.14165\n\n[3] https://arxiv.org/pdf/2311.04850", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The usage of the Pythia suite of models and the PILE dataset is very good choice for the task of assessing the performance of MIAs on LLMs. Overall, the experiments are thorough and leverage the underlying Pythia suite of models. For example, I appreciate that the authors employ the PYTHIA-DEDUP model suite’s intermediate checkpoints to assess the impact of different amounts of training data. \n- The provided analysis of the overlaps between members and non-members using the n-grams is relevant, though might not be precise enough.", "reasons_to_reject": "- The paper states negative results for the MIA attacks which makes the practical assessment of privacy leakage from LLMs more questionable. No alternative solutions are proposed.", "questions_to_authors": "- Did the authors consider other metrics to analyze the overlap between members and non-members apart from n-grams, for example, MAUVE score: https://arxiv.org/pdf/2102.01454? If so, which kind of other metrics are relevant? \n- Why the smaller models from Pythia were not used in the assessment (the model of size < 160 min parameters)?\n- It is claimed that “MIAs perform significantly better when the non-member distribution has a lower n-gram overlap“. Then, why does the GitHub dataset have a much higher MIA than other datasets (in Table 2, Orig for GitHub vs other datasets like Wikipedia)?", "extra_scores": {}}, {"review_id": "GGq3cX6xsP", "reviewer": "Reviewer_sAnK", "summary": "This paper questions the effectiveness of MIA approaches which are currently used to evaluate the leakage of large langage model (LLM) pre-training data. The authors evaluate several MIA methods from previous works. They find that the performance of MIAs is quite poor on several LLMs trained on the Pile when evaluated on subsets of the Pile from different domains. This finding seems to contradict previous works which reported much better MIA performance. The authors explain this difference by arguing that the previous works used member and non-member data from different time ranges, resulting in distribution shift of non-members with respect to members. Therefore, previous MIAs likely performed distribution inference rather than membership inference, with distribution inference being an easier task. The authors also question the relevance of the task’s definition for natural lanauage where it is possible to construct different but similar samples (which are technically non-members), but should reasonably be viewed as members. MIAs are shown not to be robust to such samples, which are wrongly classified as non-members in the high-confidence regime.\n\nI think that this work is timely and tackles an important question, which is relevant for privacy auditing and for copyright infringment. The paper does not propose a new method, so the technical novelty is very limited. However, it performs a large-scale evaluation of several state-of-the-art MIA approaches, whose results challenge the strong performances of MIAs reported in some of the previous works (Shi et al, 2023 and Meeus et al, 2023). This is an intriguing finding and the authors propose a plausible explanation for these results. The authors further challenge the definition of MIAs in the context of LLMs, showing that MIAs are not robust to small changes in the sample. \n\nHowever, the paper in its current form does not seem ready for publication, as it is imprecise on several important points neccessary to substantiate its claims and the benchmarking of MIAs seems incomplete and to suffer from a methodological flaw. Most of these points seem fixable and I hope that the authors will address them in their rebuttal.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback and for supporting our paper. We respond to the reviewer’s main concerns below, and will address questions/comments in a follow-up post:\n\n*[Reason to reject 1]* No details on hyperparameter tuning\n\nWe clarify that none of the evaluated attacks involve any tuning. Reference models are open-source, out-of-the-box pretrained LMs and the choices for k are those suggested in Shi et al.\n\nRegarding “best” hyperparameter choice, we aim to show that even in the best case, MIA scores are significantly lower than what would be expected based on trends in the MI literature.\n\n*[RR2]*  Temporal shift setting is inconsistent to Shi et al & Meeus et al.\n\nWe will clarify that we analyze temporal shift to show how unintentional distribution shifts may arise in an otherwise practical selection of non-members for pretrained LLMs, not to claim that the results in these papers are solely due to such shifts.\n\nWe don’t use the exact setups from prior work as (1) target LMs like LLaMa don’t release training data needed to assess n-gram overlap or (2) their non-members are not publicly released and are difficult to replicate.\n\n*[RR3]* Evaluation limited to 200-word text\n\nShi et al, 2023 evaluates texts up to 256 words, which is close to the text length we use. While an increase in performance with length is possible, that experiment is orthogonal to ours; inherent differences in LLM training and MI evaluation (i.e., benchmark candidates, membership definition) would still impact evaluation on longer texts.\n\n*[RR4]* No better definition proposed\n\nThe discussion of high n-gram overlap of non-members with members naturally raises the question of whether exact membership is always useful concerning information leakage. We propose approximate membership to motivate aligning membership definitions with respective privacy use cases [1] (i.e., partial lexical match, semantic leakage) rather than an exact new definition of membership. We will clarify this.\n\nRandom token replacement is naive. However, it’s a first step in exploring approximate membership definitions; intact portions of modified members can still leak member information. We also explore a more realistic definition in semantic neighbors. We aim to demonstrate that current MIAs can be too sensitive to capture what humans perceive as “close” to members and encourage future work to build on this in parallel to standard MI.\n\n### References\n\n[1] https://arxiv.org/abs/2202.05520", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors perform an extensive evaluation of several MIA approaches applied to LLMs, on several datasets and models. I think that this benchmarking of methods is valuable for the community. The authors also make their code available.\n\n- The evaluation reveals the intringuing finding that current MIA perform quite poorly. As this finding directly challenges results from some of the previous works, and a plausible explanation is offerred, I believe that publishing the paper can be beneficial to the community.\n\n- An important finding is that the MIA decision boundary is not robust to small changes in the text. The authors show that a slightly modified version of a member text that would be considered as a member by a human (but is technically a non-member as per the strict MIA definition) would be classified as a non-member.  This challenges the current definition of MIAs and could motivate interesting developments in the field.", "reasons_to_reject": "- Potential methodological flaw: the Reference-based and min-k prob attacks seem to be tuned on the test data. I re-read Appendices A.3.1. and A.4 several times looking for a mention of validation data and I found only one, for the DATABLATIONS model, while the Reference-based tuning is done on PYTHIA-DEDUP models (for which there is no mention of validation data). Tuning the methods on the same data on which final performances are reported is a methodological flaw because it biases the results. It also gives an unfair advantage to the methods being tuned compared the ones which are not tuned. This is especially problematic here since the extensively tuned Reference-based attack is found to perform « best » in the end. This can be fixed by using a validation dataset for tuning the methods which is independent from the test set.\n\n- Unclear exactly how the author’s analysis of temporal shift relates to the previous works of Shi et al, 2023 and Meeus et al, 2023. The paper should clarify whether the same experiment settings (LLMs, data domains, temporal ranges) as the two papers are reproduced, or if the author only speculate that these two papers suffer from temporal shift. Making this very precise is needed to support the abstract claim that « this apparent success can be attributed to a distribution shift », else the claim should be more nuanced. For instance, going through these two papers, I understand that in the arXiv domain, Shi et al uses members and non-members from before 2016 and after 2023, respectively, Meeus et al uses members before February 2023 and non-members after February 2023, and the current paper uses members from July 2020 and non-members from later months. Since the temporal shift is slow (Figure 6), it likely affects the former paper more than the latter. Furthermore, Meeus et al. also uses a second dataset where there is no temporal shift and uses longer documents for which MIA risk is likely to be higher. These differences are not discussed in the paper.\n\n- Limited evaluation: the paper focuses on 200-word texts. When it comes to copyright infringment, longer documents such as news articles and books pose a bigger concern. As mentioned by the authors and shown by Shi et al, 2023, MIA risk increases with text length. Therefore, it is likely that MIAs might still work for, e.g., books. If this is true, the paper’s core claim would not hold. There is no discussion of this in the paper.\n\n- A point is made that the MIA definition is not aligned with how we reason about privacy, but no better definition is proposed. The link between this part and the rest of the paper is not very clear. The usefulness of experimenting with large edit distances of 10 and 25 is unclear to me: if a large fraction of the text is replaced with random tokens, would the text still be considered to be a member by a human? Even for an edit distance of 1, replacing a token with a random token does not guarantee that the text still makes sense (which would be necessary in any notion of neighborhood). The empirical part of this section seems insufficiently developed.", "questions_to_authors": "- How do the authors ensure that their MIA evaluation setting (non-members from the Pile test data) does not suffer from distribution shift ? For instance, is there any reference stating that the Pile data was randomly shuffled before being split into train and test ?\n\n- Other works have already challenged the reliability of MIA predictions, see [A] and [B]. For instance, [A] shows that neighboring non-member samples would be classified as members which is related to the point made in this paper. Therefore the insight that MIAs are brittle is not new (although demonstrating it empirically in the LLM setting is). The paper does not position itself with respect to these works. Can the authors elaborate on their differences with [A] and [B] and add a discussion of previous works challenging MIA reliability ?\n\n[A] Rezaei, S., & Liu, X. (2022). On the Discredibility of Membership Inference Attacks. arXiv preprint arXiv:2212.02701.\n\n[B] Hintersdorf, D., Struppek, L., & Kersting, K. (2021). To trust or not to trust prediction scores for membership inference attacks. arXiv preprint arXiv:2111.09076.\n\n- How does the state-of-the-art MIA method of Meeus et al, 2023 perform compared to the other approaches?\n\n- Figure 1: How are the confidence intervals computed, e.g.., over how many samples?\n\n- Sec 3.2.2 – Resampling of non-members to have low overlap. This step seems over-engineered and, as the authors notice, results in distribution shift. A simpler approach would be to rank the non-members by the MIA scores and check if MIA confidence is negatively correlated with n-gram overlap. Did the authors consider this approach ?\n\n- Benchmarking of MIA approaches : Are the results statistically significant ?\n\n- ArXiv temporal shift experiment: there is a difference in the pipeline used to construct members and non-members (S3 vs arXiv API), which could introduce a distribution shift beyond the temporal shift. To substantiate the claim that « both arXiv bulk access and API should yield the preprints in the same manner regardless », can the author apply the API to retrieve some of the data before July 2020 and check that after pre-processing the documents match ? \n\nOther comments:\n\n- Please consider adding the definition of a sample (200-length documents) in the main paper.\n\n- Appendix A.4: I disagree with the statement that the reference-based attack is largely similar to the offline LiRA attack of Carlini et al., 2022. Two key features of LiRA, which don't feature in the reference-based attack, are its use of the logit loss (log(p/(1-p)) and fitting a Gaussian distribution to its distribution, which allows extrapolating to the high-confidence regime. The reference-based attack would be more similar if the log(1-p) terms were substracted from the loss.\n\n- Reference-based attack: The authors acknowledge the limitation of LLM reference models: although they are not supposed to be trained on the target sample (for the one-sided hypothesis test to be valid), they likely are. I agree with the authors that this seems hard to fix. However, because the attack’s assumptions are invalid, it’s hard to interpret why and when it works. Some analysis is provided in Appendix B.2; however it would be better to explain the first time the attack is presented (in the main paper) when the attack is likely to work in spite of the invalid assumptions and why. Perhaps this intuition can also guide the choice of the reference model.\n\nMinor :\n- First paragraph of Sec. 3.2.1 – the argument needs to be completed by explaining that MIA success if linked to overfitted (Yeom et al, 2018).\n\n- Figure 2 : the nuances of blue on the left are hard to distinguish in print, please consider increasing the contrast and the font size.\n\n- n-gram overlap formula: Please consider adding 1-2 sentences explaining what the formula is meant to capture.\n\n- Figure 3 – Github: is my understanding correct that on Github 40 % of the samples have 100 % n-gram overlap with the training dataset ? Please consider discussing this in the main paper and explaining that this does not mean that these non-members are members.\n\n- Please consider fully defining the loss in the main paper using the negative log-likelihood of tokens.", "extra_scores": {}}, {"review_id": "ZOzzzgVYDV", "reviewer": "Reviewer_Rszs", "summary": "This paper presents an empirical study of the Membership Inference Attack (MIA) against pre-trained large language models (LLMs). Following comprehensive experiments, the authors discovered that current membership inference methods face challenges when applied to LLMs. Further analysis attributed this poor MI performance to several factors: (1)The nature of pre-training, which involves a large volume of training data, typically trained for only a single epoch.(2)The inherent text-repeating nature of natural language, which introduces difficulties into membership inference. Additionally, the authors discussed the importance of addressing distributional shifts and suggested extending the MI game to consider neighboring members to better interpret information leakage.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their helpful feedback and for supporting our paper.\n\n*[Reasons to reject]* Table/Figure clarification\n\nWe will implement the suggestion regarding Table 1 in our revision. For Figure 5, we wanted to show how the original member/non-member score distributions highly overlap compared to that of the modified members. One clarification we will try is replacing the histograms with KDE curves. \n\n*[Q1]* MI score distribution of modified members\n\nThe reviewer is right in their reasoning: for a modification heuristic that preserves language structure perfectly and does not make a lot of edits, this would indeed be the case. However, the caveat here lies in “how” these modified records are generated.\n\nFor members derived within some edit-distance, the token replacements are random. For instance, “Dug is a good dog” could become “Dug is cactus good dog” for n=1. While this is indeed closer to the member record than  some non-member “Gato is a good cat”, the unnatural language structure from random tokens can confuse the model (higher perplexity) than a naturally-structured non-member would, especially for models that generalize well. A reference-based MIA is impacted by the phenomenon, where the reference model itself would also exhibit similar behavior in its loss, though perhaps not as significantly.\n\nThis disparity is less for semantically-similar members for both MIAs, likely due to the BERT token replacements preserving some of the semantics. However, as the BERT model is a weaker model, the replacement tokens under BERT may be noticeably unlikely under the target model.\n\n\n*[Q2]* Visualization of n-gram overlap distributions\n\nThe reviewer’s understanding is correct that the main results are derived from a benchmark where non-members undergo a soft decontamination. We chose to use the unfiltered data to visualize n-gram overlap distributions primarily to demonstrate the considerably high overlap between natural member/non-member data. We understand, however, that this could be confusing, especially for domains such as GitHub, where the naturally high overlap conflicts with the high AUCs against the (decontaminated) GitHub data (current analysis in Appendix B.3). We will add n-gram overlap distributions over the decontaminated non-member sets as well for other domains as we did with GitHub.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper is well structured and logically flows smoothly. \n\n(2) The paper is sound. The authors verify their findings and hypotheses through sufficient experiments and analysis.", "reasons_to_reject": "(1) The paper is somewhat dense for an 9-page document. It encompasses various topics, ranging from the findings regarding the challenges of MI in LLMs and analysis of the reasons behind them to discussions on unintended distribution shifts and the MI game itself. This density can make it a bit challenging to follow and grasp the details, especially without referencing the appendix.\n\n(2) A small suggestion instead of a reason for rejection: It would be better to place Table 1 on page 3 or 4 for smoother reading. Additionally, Figure 5 is currently somewhat complex and unclear for readers, especially because the bar graphs for non-members and members largely overlap, leading to confusion upon initial glance.\n\n(3) I have some doubts about the empirical results, please refer to the Question part below", "questions_to_authors": "(1)In Section 4, the modified member is derived from member data, so it should be more similar to member data compared to non-members, or it should be harder to distinguish (because modified member should have a greater n-gram overlap). Therefore, intuitively, the MI score of modified members should fall between non-members and members. However, the results in Figure 5 are far from this expectation. What could be the reason for this?\n\n(2)Figure 7 illustrates the 7-gram overlap of the GitHub domain before and after decontamination. This suggests that the statistics presented in Figure 3 seem to rely on data from the original domain. However, this approach lacks rigor since your main results are derived from decontaminated data, making it more appropriate to conduct statistical analysis based on decontaminated data. Please correct me if I have overlooked any details or misunderstood anything.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is an empirical paper which evaluates the effectiveness of membership inference attacks against LLMs, concludes that they are ineffective most of the time, and goes on to investigate the factors that make the attacks more or less effective. It especially investigates the effect of near-online training and the inherent ambiguity in what counts as membership.

Overall, this is a solid empirical paper. It first demonstrates an interesting central claim, and then analyzes it from multiple angles. The topic is very timely with the increasing concern over copyright/plagiarism. The writing is clear, and the experiments seem well motivated. Reviewers raise a few concerns with the details of the experiments, such as the tuning methodology and possible inconsistency with prior work. But at the end of the day, all of the reviewers feel this paper deserves to be accepted, and this is my sense as well.

[At least one review was discounted during the decision process due to quality]
Decision: Accept