The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
Authors: Skyler Hallinan
Jaehun Jung
Melanie Sclar
Ximing Lu
Abhilasha Ravichander
Sahana Ramnath
Yejin Choi
Sai Praneeth Karimireddy
Niloofar Mireshghallah
Xiang Ren
Venue: COLM 2025 (2025)
Keywords: membership inference
membership inference attack
privacy
memorization
TL;DR: We introduce a membership inference attack for language models based solely on N-gram overlap between model outputs and candidate documents, making it a versatile approach for both closed and open weight models.

Abstract: Membership inference attacks serves as useful tool for fair use of language models, such as detecting potential copyright infringement and auditing data leakage. However, many current state-of-the-art attacks require access to models' hidden states or probability distribution, which prevents investigation into more widely-used, API-access only models like GPT-4. In this work, we introduce N-Gram Coverage Attack, a membership inference attack that relies **solely** on text outputs from the target model, enabling attacks on completely black-box models. We leverage the observation that models are more likely to memorize and subsequently generate text patterns that were commonly observed in their training data. Specifically, to make a prediction on a candidate member, N-Gram Coverage Attack first obtains multiple model generations conditioned on a prefix of the candidate. It then uses n-gram overlap metrics to compute and aggregate the similarities of these outputs with the ground truth suffix; high similarities indicate likely membership. We first demonstrate on a diverse set of existing benchmarks that N-Gram Coverage Attack outperforms other black-box methods while also impressively achieving comparable or even better performance to state-of-the-art white-box attacks --- despite having access to only text outputs.  Interestingly, we find that the success rate of our method scales with the attack compute budget --- as we increase the number of sequences generated from the target model conditioned on the prefix, attack performance tends to improve. Having verified the accuracy of our method, we use it to investigate previously unstudied closed OpenAI models on multiple domains. We find that more recent models, such as GPT-4o, exhibit increased robustness to membership inference, suggesting an evolving trend toward improved privacy protections.

Published as a conference paper at COLM 2025
The Surprising Effectiveness of Membership Inference with Simple N-Gram Coverage
Skyler Hallinan , Jaehun Jung Melanie Sclar
Ximing Lu Abhilasha Ravichander Sahana Ramnath
Yejin Choi Sai Praneeth Karimireddy Niloofar Mireshghallah" ,, Xiang Ren
University of Southern California University of Washington
Stanford University
shall maalina@uuscedd edu
Abstract
Membership inference attacks serves as useful tool for fair use of language models, such as detecting potential copyright infringement and auditing data leakage. However, many current state-of-the-art attacks require access to models' hidden states or probability distribution, which prevents inves- tigation into more widely-used, APl-access only models like GPT-4. this work, we introduce N-GRAM COVERAGE ATTACK, membership inference attack that relies solely on text outputs from the target model, enabling attacks completely black-box models. We leverage the observation that models are more likely to memorize and subsequently generate text pat- terns that were commonly observed in their training data. Specifically, to make prediction on a candidate member, N-GRAM COVERAGE ATTACK first obtains multiple model generations conditioned on prefix of the candidate. It then uses n-gram overlap metrics to compute and aggregate the similarities of these outputs with the ground truth suffix; high similari- ties indicate likely membership. We first demonstrate on a diverse set of existing benchmarks that N-GRAM COVERAGE ATTACK outperforms other black-box methods while also impressively achieving comparable or even better performance to state-of-the-art white-box attacks despite having access to only text outputs. Interestingly, we find that the success rate of our method scales with the attack compute budget as we increase the the
number of sequences generated from the target model conditioned prefix, attack performance tends to improve. Having verified the accuracy of our method, we use investigate previously unstudied closed OpenAl models on multiple domains. We find that more recent models, such as GPT-4o, exhibit increased robustness to membership inference, suggesting an evolving trend toward improved privacy protections
1 Introduction
While training data serves a central role in developing modern large language models, model providers have increasingly withhold critical details of their datasets (Brown et al., 2020; Touvron et al., 2023b; Jiang et al., 2023). The lack of data provenance is particularly problematic, as models are often exposed copyrighted data such as novels during training (Henderson et al., 2023a; Carlini et al., 2022), which they may regurgitate in their generations post-deployment (Chen al., 2024; Biderman et al., 2023a). This has led to multiple lawsuits from news providers like the New York Times, who assert that these model tendencies decrease the utility of their protected works (Grynbaum & Mac, 2023; Bruell, 2025).
Membership inference attacks, methods to posit whether or not specific text documents were in the training data some model, are increasingly common strategies to audit the training
We release our code and data at httpp:://githbb..o. stw.sthh..cmmmaallllalaaaaa iinnn http:h/aallllaaaa//aaaaaaoarrraa /NGranCCverageettaac

---

Published as a conference paper at COLM 2025
Remove last K words of text be analyzed
Generate new diverse completions
3. Compute N-Gram Coverage Metrics
Aggregate
Text seen during training
Someone must häve been talling lies about Josef K.. he had-dono-RRthhig
Someone must have been telling lies about Josef K. had done noshing wrong and yet. one morning of April he was e2rested.
Someone must have been telling lies about K. he had dome nothing wrong and yett, one morning of April be arresstd.
or
#wonng -o morning -a-- awocied.
Text not seen during training
X
Works with both open
w.g. 50% of words mens in
Figure 1: A high-level overview of N-GRAM COVERAGE ATTACK, a cost-effective, white-box membership inference attack effective for both open and closed models: (1) attain short prefix of the candidate document (2) sample continuations given the prefix from the target model (3) compare to the original suffix (4) aggregate similarities to infer membership.
data of large language models (Carlini et al., 2020). However, many current methods require access to the underlying model or logits (Yeom et al., 2017; Mattern et al., 2023; Shi et al., 2023), limiting their scope to narrow set of models. Notably, this excludes larger, more capable, and more popular models like GPT-4 (Achiam et al., 2023) which restrict access to this information, outputting only model generations.
In this work, we investigate whether membership inference attack can be done with only the access to sampled model outputs, and if so, whether such an approach can perform comparable to established white-box methods. We introduce N-GRAM COVERAGE, ATTACK, a family of membership inference attacks based on only the surface- similarity of model generations against the input document. Our approach, illustrated in Figure 1, is cost- effective and applies to black-box models. It involves three steps: (1) sample multiple reconstructions of the input document given short prefix (2) measure the similarity of each reconstruction against the original document's 'ss suffix, and (3) aggregate the similarities to infer membership.
Our analyses reveal that N-GRAM COVERAGE ATTACK substantially simplifies previous MTA methods in the black-box setting (Duarte al., 2024), yet surprisingly performs on- par or even better than white-box methods that require model loss token logits. We also explore the effectiveness of our approach on models' post-fraining data, an area under- explored in the literature. Compared to recent membership inference methods for black-box models (Duarte et al., 2024), our method is substantially compute-efficient (refer to specific numbers), and importantly, scales better by increasing the size of repeated sampling. We additionally collect and test our method on two new datasets for membership inference and benchmark many new models such as TÜLU. Having verified the accuracy of our method, we use it to investigate previously unstudied closed OpenAl models on multiple domains. We find that more recent models, such as GPT--4,, exhibit increased robustness to membership inference, suggesting an evolving trend toward improved privacy protections.
Background and Related Work
In this work, we propose membership inference method that can work on both closed, black-box models, and on open-weight models. We briefly summarize prior efforts in mem- bership inference, memorization, training data extraction, and techniques for protection.
Membership Inference Tracing member data was first proposed in the context of genomic privacy (Homer et al., 2008; Sankararaman et al., 2009) before later being explored for deep neural networks (Shokri et al., 2016). For large language models, most previous work utilize the prediction loss of a candidate sequence, with the intuition that models are likely to have lower loss sequences that have been seen during training. Yeom et al. (2017) use simple threshold the loss itself while Carlini et al. (2020) additionally use a reference model, language model with less memorization, to remove the effect of the intrinsic sequence
2

---

Published as a conference paper at COLM 2025
difficulty from the observed loss. While this technique has found widespread use (Mattern et al., 2023; Mireshghallah et al., 2022; Ye et al., 2021; Fu et al., 2024), it is difficult to ascertain whether the reference model itself has memorized the sequence. Carlini et al. (2020) instead normalize sequences by their compressed size (entropy) via the zlib library, while Shi et al. (2023) propose Min-K%, which uses the log- log-likelihood of the K% most unlikely tokens membership signal. Zhang et al. (2024b) later extend this by leveraging statistics from the entire vocabulary distribution to normalize token probabilities; the key intuition is changing the membership signal from absolute to relative token probabilities.
However, none of these methods work with only model outputs. eu al. (2025) demonstrate that fine-tuning large language models can enable effective membership inference detection; however, this approach assumes that the model provider permits fine-tuning, besides requiring supervision. Duarte et al. (2024) introduce an output-only attack which formulates membership inference as a sueettin--nsswerin then tests model preferences by presenting them alongside the ground truth; if a candidate was truly trained on, its paraphrases are more likely to be favored above chance. Hisamoto et al. (2020) demonstrate that n-gram features effective for membership inference in sequence-to-sequeec models for machine translation.
task: paraphrases candidate documents,
Memorization There has also been work aimed at identifying memorization of training data in black-box large language models (Carlini et al., 2022). These output-only approaches typically either examine whether models can produce verbatim continuations for an input sequence (Karamolegkou et al., 2023; Zhao et al., 2024; Freeman et al., 2024; Henderson et al., 2023b), or if models can reproduce certain tokens in documents that are difficult without memorization (Chang et al., 2023b; Ravichander et al., 2025). Zhang et al. (2024a) even argue that such approaches could provide training data proofs with controlled false-positive rates. However, these works typically only focus on identifying a subset of member data that models can reproduce with high fidelity, whereas membership inference methods like ours focus on a stricter regime: the distinguishability of all member and non-member data. There have also been several efforts that aim to uncover memorization evidence, assuming access to the model's prediction loss over a sequence (Garg et al., 2024; Ravikumar et al., 2024). In contrast, our work is based on the fully black-box setting, where we only assume API-level access to the model.
Considerable prior efforts have also focused on the extraction setting for memorization, aiming to reveal training data directly from model generations rather than via membership queries (Carlini et al., 2021; Nasr et al., 2023a; Bai et al., 2024); our work differs as we aim to determine the membership of any given input rather than just ones that can be elicited from the model. Considerable work has also sought to prevent the success of any attack to extract or identify training data from large language models (Siyan et al., 2025; Tang al., 2021; Jia et al., 2019). Our work contributes to this growing body of literature by providing a previously unknown approach to identify membership in large language models.
Protection Methods Privacy-preservinn training algorithus such as DP-SGD (Abadi et al., 2016) provide provable guarantees about the extent of possible memorization. Post-training methods such as unlearning (Cao & Yang, 2015) can also be used to certifiably " "forget" problematic data which was memorized (Sekhari et al., 2021). However, such theoretical guarantees can be overly conservative and demand sacrificing too much utility. Instead, empirical privacy auditing methods (Jagielski et al., 2020; Nasr et al., 2023b; Steinke et al., 2023), which rely on membership inference attacks, form the basis of most production privacy evaluations (Song & Marn, 2020). This makes designing reliable and consistent membership inference techniques critical for privacy evaluations.
3 Method
In this section, we introduce N-GRAM COVERAGE ATTACK, cost-effective method for membership inference that only requires model-generated samples without relying on any model internals like token logits. Below, we formalize the membership inference task ($3.1), illustrate our framework that leverages n-gram statistics (53.2), then discuss the design of scoring function variants that comprises the family of N-GRAM COVERAGE ATTACK.
3

---

Published as a conference paper at COLM 2025
Algorithm 1 Membership Inference with N-GRAM COVERAGE ATTACK
Require:
Target model Mg, input text x to test for membership, threshold e, token index k, number of outputs to sample d, similarity function sim
Ensure:
Prediction: Member or Non-member
1: Sample d generations from model Ma using part of the prompt:
tog y Ma(xgx)
2: Compute the similarities of the generations the suffix of x unseen in step 1:
s sim (ox,<>>) 8i I,...,d
3: Aggregate the similarities using agg(x):
Sg agg (15 (155)1 )/21)
4: Predict Member if c488
else Predict Non-member
3.1 Membership Inference Task
A language model Ma is trained on collection of data D, where each sample x E DD denotes a member, and x & DD denotes a non-member. Given some target model My and corpus of candidale text documents C, membership inference attack attempts to determine CnD: D: which, if any, samples x E were used in the training of Mg.
3.2 N-GRAM COVERAGE ATTACK: Membership Inference using only Model Outputs
The goal of our algorithm is to assess if a model Mg has likely been trained on a particular sequence x. We achieve this through approximating how Ma has memorized specific sequence x by empirically measuring how closely the model's sampled outputs align with that sequence. Our key intuition is that models should output text that is more similar to data that they were trained on (member data) than data they were not trained on (non-member data) (Carlini et al., 2022; Chen et al., 2024). Specifically, we prompt the model multiple times with a prefix of x and assess how close its outputs are to naturally "regenerating" the remaining suffix of
Formally, N-GRAM COVERAGE ATTACK consists of three steps, detailed below and in Algorithm 1. First, given some prompt p and a prefix of x as input of size k, X-ck, we sample d diverse completions with Ma using standard language modeling (Sample from Target Model). p will usually contain an instruction prompt to reconstruct text.
{ ( J i=1 ~ Mo(.lp,xxx)
(8)
We then assess the similarity of the sampled generations 0g to the original suffix of x, X where xi) computes the similarity between two texts x1 and x2 (higher is better) (Compute Similarities of Outputs with Original Document):
k + sim(o xsk), Vi =i= (1,
Finally, we condense the d-dimensional vector scores into a single value using an aggre- gation function agg(x): Rd + R. We predict x to be a member and only if agg(x) > é for some pre-defined value (Aggregate).
3.3 N-GRAM COVERAGE ATTACK Function Choices
Our method naturally allows for using different similarity metrics sim(-) and aggregation functions agg(.) depending on the use-case; we detail these below.
Similarity Metrics We consider three distinct x2) function options to be used with N-GRAM COVERAGE ATTACK: Coverage, Creativity Index (Lu et al., 2024), and Longest
4

---

Published as a conference paper at COLM 2025
Common Substring (LCS) Notably, these are all simple, n-gram coverage metrics, which are both interpretable and efficient to compute.
Coverage (Cov) quantifies the overlap between two documents x1 and X2 by computing the proportion of tokens in X2 covered by matching n-grams of at least length from x1.
Cxvl(x1, Ewex, 133 u-gram C X1, l.. 2 L s.t. w E g) [0,1]
X2
Creativity Index (Cre; Lu et al., 2024) measures textual novelty by penalizing repeated content from reference materials at multiple N-gram lengths. sums 1 coverage over increasing N-gram sizes, rewarding texts with lower and shorter-span overlaps:
B Σ 1 CovI(x),x)) L-A
Creativity Index(x1,
[0, A]
In practice, we -Creativity Index so higher scores indicate more similarity.
Longest Common Substring (LCS) computes the length of the longest common contiguous substring between X1 and X2- This can be done on multiple granularities, such as on the character word level (LCSc and LCSw). Unlike coverage and creativity index, do not include length normalization here.
Aggregation Function We consider four simple agg(x) functions: maximum, minimum, mean and median. Since false positives true non-members which can be accurately reproduced by the model are unlikely, the max metric is particularly appealing, as effectively surfaces the strongest membership signals, even if they are sparse.
Experiments
We perform comprehensive experiments across four datasets, multiple model families across scale, and several baselines, demonstrating that N-GRAM COVERAGE ATTACK is a versatile and effective attack, despite its simplicity. For all experiments, we perform an initial sweep with small 5% validation set to finalize hyperparameters before reporting test set results. See Appendix $B for more details and $A for additional results on the Pile and Dolma.
4.1 Models
We consider a diverse set of models to attack, which vary in size and model access. For open- weight models, we include LLaMa 1 (Touvron et al., 2023a), a set of decoder-only language models with sizes of 7B, 13B, 30B, and 65B respectively released by Meta February 2023.
We also include large suite of closed, API-access OpenAl models offering only access to output texts?, largely understudied for membership inference. We start with GPT- 3.5 Instruct deprecated text-davinci-009, OpenAl's first instruction-following model (Ouyang al., 2022). also include two GPT-3.5 Turbo (OpenAI, 2022) models, the first set of chat-specific OpenATlanguage language models: gpt-3. 5-turbo- 01 25 and 3pt-3.5-turbal 11 106. All three GPT-3.5 models have a knowledge cutoff of Aug 31, 2021. We also consider GPT-4 Turbo (Achiam et al., 2023) the follow-up to GPT-3.5 Turbo, GPT-4o (OpenAI, 2024), a contemporary, flagship OpenAl model released in mid-2024, and GPT-4o mini (OpenAl, 2024), the cost- efficient, smaller version released shortly after. These all have a cutoff date late 2023.
(Brown et al., 2020), designed to replace the now-
Finally, we also consider TÜLU (Wang et al., 2023), suite of varying models converted curated human + machine-generatee
from base to instruction-tuned variants by training data mixture. We use TÜLU (base models of LLLMMa Ma 7B, 13B, 30B, and 65B (Touvron et al., 2023a) and TÜLU 1.1 (base models of LLaMA-2 7B, 13B, and 70B (Touvron et al., 2023b)).
Some output limited output text probabilities, but they not used by existing baselines

---

Published as a conference paper at COLM 2025
4.2 Datasets
We use a set of five diverse datasets - three existing, and two we construct - to comprehen- sively evaluate membership inference attacks. While most of the datasets are used to assess pretraining membership, we also construct new dataset to assess fine-tuning membership. See Appendix $D for more details on all datasets and our data creation procedure.
BookMIA (Shi et al., 2023) consists of 512-word snippets sampled from 100 books. Half of the data comes from famous literature presumed to be in the training corpus of older OpenAI models like GPT-3.5 (Chang et al., 2023a). The other half is comprised of books published after 2023. We use the GPT-3.5 family as the target models.
WikiMIA (Shi et al., 2023) consists of snippets from Wikipedia articles written before 2017 and articles written after 2023; for models released in this time span, these are members and non-members respectively 3. Following prior work (Shi et al., 2023), we use base LLaMa 7B, 13B, 30B and 65B (Touvron et al., 2023a) and GPT-3.5 as the target models.
WikiMIA2024 Hard is new dataset we construct which builds upon the original WikiMIA format with two key modifications for more robust evaluation. (1) First, to minimize tempo- ral distributional differences between members and non-members, we identify Wikipedia summaries whose content has changed from their version at the end of 2016 to versions updated in 2024 or later. Following WikiMIA's core assumption, we treat pre-2017 summary versions likely members of model training sets, as these were presumably scraped into massive pretraining corpora, while non-members are the most recent versions of these same summaries, edited after most models' knowledge cutoff dates. By using different versions of the same articles, we minimize topical differences between members and non-members, unlike the original WikiMIA, where members and non-members cover entirely different topics and time periods. (2) Second, our target models include not only GPT-3.5 and the LLaMa family (as in the original WikiMIA), but now also extend to more recent models such as GPT-4oo and GPTT-4, which have knowledge cutoff dates near the end of 2023.
We also note that we filter article pairs for a minimum edit distance (Levenshtein, 1965), ensuring the newer (non-member) version differs meaningfully from the older (member) one. This makes the benchmark challenging but not impossible, so that observed model performance reflects actual memorization capability rather than the inability to detect imperceptible differences. Finally, we also constrain length variations between versions to within 20% to avoid spurious length features in members and non-members.
WikiMIA-24 (Fu et al., 2025) follows the original WikiMIA (Shi et al., 2023) collection methodology with an updated cutoff for non-members; members are still Wikipedia articles written before 2017, while non-members are now articles written after March 1, 2024". The target models are the same as WikiMIA2024 Hard.
TÜLU Mix (Wang et al., 2023) is a new membership dataset we construct to assess fine- tuning membership attack effectiveness. Since most previous work investigates pre-training membership, we seek to understand how well existing strategies transfer to fine-tuning. The TÜLU Mix was used to train both the TÜLU and 1.1 suite of models, which are the natural target models. The authors test a variety of candidate instruction-tuningg datasets across domains, unifying the format before selecting a subset as the best mixture. We use these data points as members and data from the instruction datasets not-selected as non-members.
4.3 Baselines
We detail the baselines we run for all the membership inference tasks:
As Duan et al. (2024) note, this collection methodology may result in spurious, temporal distri- bution shift between members and non-members. However, given that these are among the only available benchmarks that can be used for closed-access OpenAl models (due to lack of publicly- known training data) (Shi al., 2023), we believe is important include them, even with their known limitations, enable broader evaluation and comparison.
"As WikiM1A-24 only updates the temporal boundary without modifying the underlying data col- lection procedure, it likely inherits the same temporal distribution shift vulnerabilities as in WikiMIA

---

Published as a conference paper at COLM 2025
N-GRAM COVERAGE ATTACK
White-Box Attacks
Model
Cov.
Cre.
LCS.
LCSw
D-C
Loss
R-Loss
zlib
MinK
WikiMIA (Shi et al., 2023)
GPI1-35-0125
0.64
0.63
0.61
0.60
0.55
GPT-3.5 Inst.
0.62
0.61
0.58
0.58
0.54
GPT-3.5-1106
0.64
0.62
0.61
0.60
0.52
LLaMa-7B
0.60
0.59
0.56
0.55
0.48
0.62
0.63
0.64
LIaMa-13B
0.62
0.59
0.57
0.54
0.52
0.64
0.63
0.65
0.66
LLaMa-30B
0.63
62
0.57
0.58
0.49
0.66
0.69
0.67
0.69
LLaMa-65B
0.65
0.64
0.61
0.58
0.50
0.68
0.74
0.69
0.70
WikiMIA-22 24 (Fu et al., 2025)
GPT-3.5-0125
0.67
0.67
0.64
0.66
0.48
GPT-3.5 Inst.
0.65
0.64
0.62
0.64
0.50
GPT-3.5-1106
0.68
0.67
0.66
0.68
0.49
GPT-4
0.84
0.82
0.76
0.79
0.56
GPT-4o-1120
0.83
0.82
0.77
0.79
0.50
GPT-4o Mini
0.73
0.74
0.66
0.69
0.44
LLaMA-7B
0.59
0.59
0.60
0.59
0.53
0.67
0.67
0.69
LLaMA-13B
0.63
0.63
0.61
0.61
0.50
0.68
0.60
0.69
0.71
LLaMA-30B
0.67
0.66
0.64
0.64
0.48
0.72
0.69
0.72
0.74
LLaMA-65B
0.64
0.65
0.65
0.65
0.50
0.74
0.74
0.75
0.76
WikiMIA
Hard
GPT-3.5-0125
0.59
0.56
0.54
0.55
0.47
GPT-3.5 Inst.
0.64
0.63
0.61
0.61
0.45
GPT-3.5-1106
0.58
0.58
0.56
0.57
0.49
GPT-4
0.57
0.58
0.55
0.57
0.44
GPT-4o-1120
0.55
0.55
0.54
0.52
0.51
GPT-4o Mini
0.55
0.53
0.52
0.51
0.43
LLaMa-7B
0.55
0.54
0.53
0.52
0.47
0.51
0.50
0.52
LLaMa-13B
0.59
0.58
0.53
0.53
0.51
0.53
0.57
0.51
0.54
LLaMa-30B
0.61
0.61
0.55
0.57
0,50
0.56
0.61
0.53
0.60
LLaMa-65B
0.64
0.63
0.59
0.60
0.51
0.57
0.57
0.54
0.58
Table 1: Results for different models and attacks on WikiMIA, WikiMIA-24, and WikiMIA2224 Hard. Bold denotes the best performance the black-box attacks, while underline denote the best performance for the white-box attacks. The columns in blue are from N-GRAM COVERAGE ATTACK, while the columns in gray are loss-based baselines as a reference.
Loss (Yeom et al., 2017) uses the likelihood candidate member under the target model as proxy for membership; higher likelihood (lower loss) samples are likely members.
Reference Loss (R-loss; Carlini et al. 2020) builds on the naive loss by subtracting the loss from reference model smaller language model with general language ability and minimal memorization - to identify loss differences due to memorization rather than fluency.
zlib (Carlini et al., 2020) divides the loss by the compressed file size of the candidate member using the zlb library. The idea is that more compressible sequences - typically those with higher redundancy lower entropy should naturally have lower loss,
Min-K% (Shi et al., 2023) measures the likelihood of the k% least-likely tokens (outlier tokens) in the given text under the target model.
DE-COP (D-C; Duarte et al. 2024) formulates membership inference as auestion-answering task, where model is prompted to infer the plausible completion to the input text.
7

---

Published as a conference paper at COLM 2025
N-GRAM COVERAGE ATTACK
White-Box Attacks
Model
Cov.
Cre.
LCS.
LCSw
D-C
Loss
R-Loss
zlib
MinK
GPT-3.5-0125
0.84
0.85
0.84
0.83
0.84
GPT-3.5 Inst.
0.91
0.91
0.92
0.93
0.68
GPT-3.5-1106
0.84
0.85
0.83
0.84
0.85
Table 2: Results for BookMIA, Bold denotes the best performance in the black-box attacks. The columns gray are white-box baselines which cannot be computed for these models.
N-GRAM COVERAGE ATTACK
White-Box Attacks
Model
Cov.
Cre.
LCS.
LCSw
D-C
Loss
R-Loss
zlib
MinK
TÚLU-7B
0.79
0.79
0.73
0.74
0.48
0.84
0.81
0.84
TULU-13B
0.80
0.80
0.74
0.76
0.47
0.87
0.63
0.83
0.87
TÚLU-30B
0.82
0.76
0.77
0.52
0.87
0.54
0.84
0.87
TÚLU-65B
0.85
0.86
0.80
0.80
0.45
0.92
0.68
0.90
0.92
TÚLU-1.1-7B
0.73
0.70
0.71
0.47
0.77
0.74
0.76
TÚLU-1.1113
0.76
0.75
0.71
0.72
0,43
0.81
0.58
0.78
0.81
TÜLU-1.1-70B
0.78
0.75
0.77
0.45
0.86
0.64
0.84
Table 3: Results for TÜLU. Bold denotes the best performance in the output-only methods, while underline denote the best performance for the loss-based methods.
4.4 N-GRAM COVERAGE ATTACK Generation Main Details
An important part of our pipeline is how much of the candidate member to the prefix. In our main experiments, we use 50% of the words from the candidate as the prefix. We also limit the generation length be the number of tokens removed, ensuring our input + output token budget is always O if the original text is length n. For BookMIA, the final results are reported with 100 generations per candidate; for everything else, we report with 50 generations due to computational constraints. See Appendix SC for more details.
4.5 Evaluation
Following prior work (Shi et al., 2023; Duan et al., 2024), we evaluate attack effectiveness using the area under the ROC curve (AUROC), rather than classification accuracy at fixed threshold. AUROC provides a threshold-independeen measure of how well an attack can distinguish between member and non-members higher values indicate stronger attacks.
4.6 Main Results
Our comprehensive set of experiments shown in Table 1 Table 3 demonstrate that N- GRAM COVERAGE ATTACK is consistently effective across datasets, beating other black-box baselines for closed-models, and even performing close white-box baselines
N-GRAM COVERAGE ATTACK consistently outperforms black-box baselines We find N- GRAM COVERAGE ATTACK performs better than or equal to the other black-box baseline, DE-COP, in all datasets. The closest comparison is shown in Table 2 for BookMIA: on the GPT-3.5 models, both black-box attacks perform well, but our method has a strong improve- ment over DE-COP on GPT-3.5 Instruct. Everywhere else, DE-COP struggles significantly behind N-GRAM COVERAGE ATTACK, with near random performance, especially on the open-weight models like LLaMA and TÜLU in Table 3. We hypothesize that this large drop-off in performance is due to the naive method assumption that the target model is a faithful guestion-answering model, which may not apply to weaker models. Simple n-gram coverage metrics can perform comparably to white-box attacks Surprisingly, we find that N-GRAM COVERAGE ATTACK also performs comparatively - or even better - to white-box attacks across all datasets well. Specifically, it performs average 95%
8

---

Published as a conference paper at COLM 2025
well as the white-box baselines the LLaMA models with WikiMIA in Table 1, and 91% as well with WikiMIA-24. Furthermore, we find that on WikiMIAg12 Hard, our method outperforms white-box attacks on all models. Further results on the Pythia and OLMo models corroborate these results and are in Appendix SA.
Coverage and Creativity consistently outperform LCS We find that in general, the three n-gram similarity metrics we propose for N-GRAM COVERAGE ATTACK perform well. However, we find that across datasets, coverage and creativity consistently outperform the longest common substring, except for one model in BookMIA. We hypothesize this is because the coverage and creativity metrics 1) explicitly account for multiple matches, unlike LCS and 2) normalize by length, to contextualize the match length. Creativity and coverage, perform nearly equally otherwise, with coverage performing better on WikiMTA and WikiMIA 2024, where there are fewer matches in general, and creativity working better for BookMTA and TULU, which have more positive span matches to disambiguate.
N-GRAM COVERAGE ATTACK is more efficient than black-box baselines We compare the computational requirements of N-GRAM COVERAGE ATTACK to the existing black-box baseline, DE-COP (Duarte et al., 2024). Let x be candidate member of length и. DE-COP first generates three paraphrases of x, with an input length of ~ n tokens and an output length of x 3n tokens. The next step, multiple-choice question-answeringg requires 24 generations of input length x 4n and output length 1. The final token budget is z 97n input and = 3n output or approximately 100n total per sequence. In addition, it requires access to powerful paraphraser model like Claude (Anthropic, 2023) for the initial stage, which further limits accessibility and incurs even more cost. On the other hand, N-GRAM COVERAGE ATTACK is more flexible, enabling a cost-performance tradeoff. Specifically, if we use index k to construct a prefix Xgk, we can constrain our generation to be π - tokens. With d generations, the total token budget becomes d x n. We also make no use of external models, only relying on the target model itself for generation. Empirically, tested on WikiMTA Hard with LLaMA models and d = 50, DE-COP is computationally expensive, taking on average 2.6x longer than our method despite performing much worse.
Fine-tuning Membership Inference is Effective Table 3 shows the detailed results for TÜLU. Across all model variants, most attacks, including N-GRAM COVERAGE ATTACK, can effectively determine membership with high accuracy; the notable exclusion is DE-COP. We also find the TÜLU 1.1 models display more resilience to attack compared to their equal-sized TÚLU counterparts. We also find that reference models perform much poorer.
4.7 Ablation
We conduct additional experiments using BookMIA and GPT-3.5-0125 to further explore the impact of different hyperparameters, with important scaling conclusions.
N-GRAM COVERAGE ATTACK scales with the number of sequences The top of Figure shows how performance scales with different N-Gram overlap metrics from N-GRAM COVERAGE ATTACK as we increase the number of sequences generated. For all metrics, scaling the size of the generations increases the attack performance. Intuitively, as we sample more generations from the model, we obtain a increasingly accurate output distribution representative of the true model probabilities. We also observe a similar scaling trend in other datasets, highlighting the versatility of our method.
Given a fixed token budget, requesting the model to regenerate the last 50% of the sequence is best for performance The middle of Figure 2 shows N-GRAM COVERAGE ATTACK performance as different proportions of the candidate document are used as the prefix. This is with a fixed token budget (which we use in main experiments), where the model can generate only as many tokens as exist in the suffix. Across n-gram overlap metrics, the best performing proportion is consistently at 50%. While more context is in-general helpful, since we have a fixed budget, using too large of a prefix limits both the suffix size and the generation length, which may harm performance.
Temperature near 1.0 is consistently the best We find that a temperature near 1.0 is impor- tant for the performance of N-GRAM COVERAGE ATTACK across metric. Though it might be

---

Published as a conference paper at COLM 2025
N-Gram-1
N-Gram-2
N-Gram=4
N-Gram-5
Coverage
Creativity
(character)
(word)
0.80
100
100
Number of Sequences Generated
Coverage
Creativity
LCS (character)
LCS (word)
0.6
0.10
0.25
0.50
0.75
0.90
0.10
0.25
0.50
0.90
0.10
0.90
0.10
0.25
0.30
0.5
Proportion Text as Prompt
Coverage
Creativity
LCS (charactor)
L.C (word)
0.8
0.6
08
0.2
12
1.3
0.2
03
10
12
15
02
05
0.8
10
12
15
18
Temperature
Figure 2: Scaling of N-GRAM COVERAGE ATTACK performance on BookMIA with differ- ent n-gram overlap metrics with GPT-3.5-0125 and max aggregation. We vary number of sequences generated (top), proportion of text used as prefix (mid.), and temperature (bot.).
expected that higher temperatures are in general more favorable, there is an intuitive trade- off between encouraging diversity elicit harder-to-surface memorization and maintaining an accurate representation of the underlying distribution.
Conclusion
In this work, we introduce N-GRAM COVERAGE ATTACK, membership inference attack that relies solely on text outputs from the target model, enabling attacks on completely black-box models. We demonstrate ERAGE ATTACK outperforms other black-box methods while also impressively achieving comparable or even better performance than fate-offthheart white-box attacks. We also find that our method is highly compute-efficiett and its versatility allows us to investigate previously unstudied closed OpenAl models. Our findings reveals the vulnerability of language models, even in a fully black-box setting, underscoring the need for stronger privacy safeguards for large language models.
diverse set of benchmarks that N-GRAM Cov-
scales well with increased repeated sampling,
Overall, N-GRAM COVERAGE ATTACK provides a practical auditing tool for detecting problematic memorization, such as PII leakage or copyrighted content reproduction critical concerns as models are trained on web-scale data of uncertain provenance. The method's efficiency and black-box nature make it valuable for monitoring deployed models and proactively identifying memorization risks. We hope this work encourages broader adoption of membership inference testing as part of responsible Al development.
Acknowledgments
We thank Johnny Wei and Robin Jia for their insightful suggestions on early versions of this work. We also appreciate helpful comments from Duygu Yaldiz and Yavuz Bakman. We also thank Jon May for his feedback and discussion on this work. Finally, we thank Mingma Sherpa for his constructive help throughout this project.
10

---

Published as a conference paper at COLM 2025
References
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haim ing Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shappe
Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim'on Posada Fishman, Juston Forte, Is abella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Le- ung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Ma teusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob Mc- Grew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Reichhio Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Bel- bute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack W. Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Tlya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer' on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll L. Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Wein- mann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qim ing Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report. 2023. URL https:/ / //ppi w orrp/CoppssD:257522222
Harris,
Anthropic. Claude: An ai assistant by anthropic. httpss/www.arthoopicccomm
claude, 2023. Accessed: 2025-03-28.
11

---

Published as a conference paper at COLM 2025
Yang Bai, Ge Pei, Jindong Gu, Yong Yang, and Xingjun Ma. Special characters attack: Toward scalable training data extraction from large language models, 2024. URL https: //arxiv.org/ ats//405/// 05990.
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin G. Anthony, Shivanshu Purohit, and Edward Raf. Emergent and predictable memoriza- tion in large language models. ArXiv, abs/ Abs/2304.11155, 2023a. URL https://api.
semanticscholar haa.aaarrrrg//rruuss:2555522222 258299173...
Stella Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. ArXio, abs/2304.01375, 2023b. URL https: http//appppppaaaaeaeee semanticscholar orgo :otp//oopp:D:::: 257921893.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/ 2005.14165, 2020. URL https: //api. semanticscholar. org/CorpusID: 218971783.
Alexandra Bruell. cuse ai firm cohere of copyright violations. nal, February 2025.
Condé nast, mcclatchy and other publishers
The Wall Street Jour-
URL https://wwe wwwp:wwww..ccomm/uusnsss/eeeaaa
Accessed: 2025-03-26.
Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pp. 463-480. IEEE, 2015.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Kather- ine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2020. URL https: /ppi. semanticscholar :tt.:oopppS::::2211555550
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021. URL https://arxiv org/abs/2012 ht 07805.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying memorization across neural language models. ArXiv, abs//202.076666, 2022. URL htt.httpp:://pi..eennnticccoooo https/ccc......oreooooopuoii-aaa orrg/CorpusID::246633...
Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to phaatttt//ptt4t Arxiv, abs/2305.001 2023a. URL
https://api. semanti wtt.riicoco....oooorroroooooooaa
Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/ gpt-4. ar preprint arXit:2305.00111, 2023b.
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hanna Hajishiri,, Luke S. Zettlemoyer, and Pang Wei Koh. Copybench: Measuring literal and non-literal reproduction of copyright-proteeted text in language model gener- ation. ArXiv, abs/ 2400707007 2024, URL https: //api. www.nti wwtp/orrpossscceecooooooo 271 771064292.
12

---

Published as a conference paper at COLM 2025
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke S. Zettlemoyer, Yulia Isvetkov, Yejin Choi, David Evans, and Hanna Hajishirzi. Do mem- bership inference attacks work on large language models? ArXiv, abs/ 22//2//222000012222 2024.
URL UUR.nttssss/ppp..smnniiccchlhbls paaaaattiiccoolaaooo//rrrossssss 267627633..
André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, and Lei Li. De-cop: Detecting copyrighted content in language models training data. ArXiv, abs/2402.09910, 2024. URL
https://api semanticscholast httpr//iciccoooooo/rrrrrrrrrrree http://oppuID::2766111
Joshua Freeman, Chloe Rippe, Edoardo Debenedetti, and Maksym Andriushchenko. Ex- ploring memorization and copyright violation in frontier llms: A study of the new york times openai 2023 lawsuit. arviv preprint ar Xit:2411.006370, 2024.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao liang, Membership inference attacks against fine-tuned large language models via self-prompt calibration. In Advances in Neural Information Processing Systems, 2024.
Wenjie Fu, Huandong Wang, Chen Gao, Guanghua Liu, Yong Li, and Tao Jiang. Mia-tuner: Adapting large language models as pre-training text detector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 1234-1242. AAAI Press, 2025.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. ArXiv, abs/2101.0000 2020. URL https: / /api. semanticschols org/ Cor 230435736.
Isha Garg, Deepak Ravikumar, and Kaushik Roy. Memorization through the lens of curva- ture of loss function around samples. In Proceedings of the 41st International Conference on Machine Learning, pp. 15083-15101. PMLR, 2024.
Dirk Groeneveld, Iz Beltagy, Evan Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. Olmo: Accelerating the science of language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19889,15809, 2024.
Michael M. Grynbaum and Ryan Mac. The times sues openai and microsoft over a.i. use of copyrighted work. The New York Times, 2023. URL https: //www. nytimes. wwt.mes..oom/..m wwmccomm//2//// 2023/12/ 27 27//business//eddia//ewwwwy york- times open- ai -mi micoosoft www.ult...tt...................t html.
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. Foundation models and fair use. ArXiv, abs/2303.15715, 2023a. URL
https://api. semanticscholar wtgg/oorpusD:2257770
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A Lemley, and Percy Liang. Foundation models and fair use. Journal of Machine Learning Research, 24 (400):1-79, 2023b.
Ryo Hisamoto, Matt Post, and Benjamin Van Durme. Membership inference attacks on sequence-to-eqquenc models: my data your machine translation system? Transac- tions of the Association for Computational Linguistics, 8:607-620, 2020.
Nils Homer, Szaboles Szelinger, Margot Redman, David Duggan, Waibhav Tembe, Jill Muehling, John V Pearson, Dietrich Stephan, Stanley F Nelson, and David W Craig. Resolving individuals contributing trace amounts of dna to highly complex mixtures using high-density snp genotyping microarrays. PLoS genetics, 4(8):e0000167, 2008
Matthew Jagielski, Jonathan Ullman, and Alina Oprea. Auditing differentially private machine learning: How private is private sgd? Advances in Neural Information Processing Systems, 33:22205-2273 2020.
Jinyuan Jia, Ahmed Salem, Michael Backes, Yang Zhang, and Neil Zhenqiang Gong. Mem- guard: Defending against black-box membership inference attacks via adversarial exam- ples, 2019. URL httpss///axxvv... wwtprriivvoogg///..../// org/abs/1909 18594.
13

---

Published as a conference paper at COLM 2025
Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Deven- dra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L'elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. ArXiv, abs/2310.06825, 2023. URL https: semanticscholar .rg/ CorpuSID:26383045
Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Segaard. Copyright violations and large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, PP. 7403-7412, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/ /v1/ 2023.emnlp-main-te URL https: /atps///aaaaatlogyyooooooo org/o02/ emnlp-main. 458/.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposuum on Operating Systems Principles, 2023.
Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, 10:707-710, 1965. URL https: semanticccccs
org/CorpusID: 60827152.
Ximing Lu, Melanie Sclar, Skyler Hallinan, Niloofar Mireshghallah, Jiacheng Liu, Seungju Han, Allyson Ettinger, Liwei Jiang, Khyathi Chandu, Nouha Dziri, and Yejin Choi. Ai as humanity's s salieri: Quantifying linguistic creativity of language models via systematic attribution machine text against web text. ArXiv, abs/ 2410.0465,, 2024. URL https:
//api. ccchollar org/ CorppssI::: 27312731111551 18492. 273155992.
lan Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groen- eveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit, 2024. URL https: //arxiv.org/ wtt.o//ab///...1100. 10523.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Scholkopf, Mrinmaya Sachan, and Taylor Kinkpaikk.. Membership inference attacks against language models via neighbourhood comparison. ArXiv, abs/ abs/2305.18462, 2023. URL https://api.
semanticscholar. org. /rrg/CoopsII::::2 2 258967264.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Kirkpaiikpat and R. Shokri. Quantifying privacy risks of masked language models using membership inference attacks. In Conference on Empirical Methods in Natural Language Processing, 2022.
URL https: htttp:://ppi.semmnticcchoho www.sseemmmnnoll......e orgg CorpusID: 247315267.
Milad Nasr, Nicholas Jonathan Hayase, Matthew Jagielski, A. Feder Cooper, Daphne Ippolito, Christopher A. Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models, 2023a. URL
https://arxiv. org/ abs/ 17035.
Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini, and Andreas Terzis. Tight auditing of differentially private machine learning, In 32nd USENIX Security Sympusium (USENIX Security 23), pp. 1631-1648, 2023b.
OpenAl. Introducing chatgpt. 09-27.
2022.
2022. Accessed: 2024-
OpenAL. Gpt-40 mini: advancing cost-efficient intelligeee,,,,,,,,, 2024. URL https://openaii. com/ htn/iindxx//i htw.://mw-aaaaannnoooooooeefeeee
OpenAl. Hello gpt-4 turbo. https: 2024-09-27.
2024. Accessed:
14

---

Published as a conference paper at COLM 2025
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https: www.sssemaiiiccchoieeo hrtpraaiiccccoaaooo////rrroo/olo 246426909.
Abhilasha Ravichander, Jillian Fisher, Taylor Sorensen, Ximing Lu, Yuchen Lin, Maria Antoniak, Niloofar Mireshghallah, Chandra Bhagavatula, and Yejin Choi. Information- guided identification of training data imprint in (proprietary) large language models. arv preprint ar Xiv:2503.11001 2025.
Deepak Ravikumar, Efstathia Soufleri, Abolfazl Hashemi, and Kaushik Roy. Unveiling privacy, memorization, and input curvature links. In Proceedings of the 41s International Conference on Machine Learning, pp. 15102-15120. PMLR, 2024.
Sriram Sankararaman, Guillaume Obozinski, Michael T Jordan, and Eran Halperin. Genomic privacy and limits of individual detection in a pool. Nature genetics, 41(9):965-97,, 2009.
Ayush Sekhari, Jayadev Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remem- ber what you want to forget: Algorithms for machine unlearning. Advances in Neural Information Processing Systems, 34:18075-18086, 2021.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke S. Zettlemoyer. Detecting pretraining data from large language models. ArXin, abs/ 2330..169,, 2023. URL https:///pi www.nti www.tiicccha....... org/Cor CCrDpIID:: 264451585.
R. Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. 2017 IEEE Symposium on Security and Privacy (SP), pp. 3-18, 2016. URL https://ppp.
http://///pppseematicccoooooooo semanticscholar : 100IID::048000 10488675.
Li Siyan, Vethavikashini Chithrra Raghuram, Omar Khattab, Julia Hirschberg, and Zhou Yu. Papillon: Privacy preservation from internet-based and local language model ensembles, 2025. URL https: http:///aaxxxoooabb///////s abrgg//bb/211 0. 17127.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, A. Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, lan Magnusson, Jacob Daniel Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke S. Zettlemoyer, Noah A. Smith, Hanna Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. ArXiv, abs/ abs/2402.00155 2024. URL https: / /api. semanticscholar wwwphhollarroo////ooooo wwrg/CCrrpsID.... 267364861.
Shuang Song and David Marn. Introducing a new privacy testing library in tensorflow. hertooocccttttiinni-bbrrrr html, 2020.
URL https:/bbl tensorflow.
Thomas Steinke, Milad Nasr, and Matthew Jagielski. Privacy auditing with one (1) training run. Advances in Neural Information Processing Systems, 01:49268-492888 2023,
Xinyu Tang, Saeed Mahloujifar, Liwei Song, Virat Shejwalkar, Milad Nasr, Amir Houmansadr, and Prateek Mittal. Mitigating membership inference attacks by self- distillation through novel ensemble architecture, 2021. URL https://arxiv www.w..r////s//..//// 08324.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023a. URL https: //api. semanticscholar. .rtg//orpuusI::255119992. wwwaaatttccccooo..oooo//oooooooo
15

---

Published as a conference paper at COLM 2025
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. Xivi abs/2307.09222 009288, 2023b. URL https://apt
comanticscholar org/
CorpusID: 259958998.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. How far can camels go? exploring the state of instruction tuning on open re- sources. ArXiv, abs/2306.00755 2023. URL httpps//aag http://aap..mmanisssssiieieo semanticscholar org/CorpusID: 259108263.
Thomas Wolf, Lysandre Debut, Victor Julien
Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Huggingface's transformers: State-of-the-ard natural language processing. ArXiv, abs /tts/1900.0377,,200 2019. URL https://api. www.steeeecnotiiieeeeeeie wwweeroogg//ooooo org/CorpusID:2735511 27:355515 589..
Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, and R. Shokri. Enhanced mem- bership inference attacks against machine learning models. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, 2021. URL https:
semantic Corrrsooooooooog///rrr
argg/CorssII: Ptt//CoppuSDD:2444550.. 244345608.
Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. Privacy risk in machine learning: Analyzing the connection to overfitting 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pp. 268,282, 2017. URL https://apt semanticscholar. org/CorpusID: 2656445.
Jie Zhang, Debeshee Das, Gautam Kamath, and Florian Tramèr. Membership inference at- tacks cannot prove that model was trained on your data. arXiv preprint arXiv:2409.19799, 2024a.
Jingyang Zhang, Jingwei Sun, Eric C. Yeats, Ouyang Yang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. ArXiv, abs/2444.02993 2024b. URL https: //api. semanticscholar.
Trg/CorpusID: 268889777.
Weijie Zhao, Huajie Shao, Zhaozhuo Xu, Suzhen Duan, and Denghui Zhang. Measuring copyright risks of large language model via partial information probing. ar Xiv preprint 1rXio:2409.13831, 2024.
16

---

Published as a conference paper at COLM 2025
A
Additional Experiments
We also conduct additional experiments for membership inference attacks on two datasets and families open-access models. The same conclusions from the main text hold: N-GRAM COVERAGE ATTACK performs better than DE-COP and comparatively to the white-box baselines.
A.1 Models
We use the following models:
Pythia (Biderman et al., 2023b) is a suite of decoder-only models from 70M to 12B parameters released by Eleuther AI. We use the 1.4B, 2.8B, 6.9B, and 12B models.
OLMo (Groeneveld et al., 2024) is a set of 1B and 7B models released by Ai2. We use the 07-024 checkpoints Finally, we also use the SFT and Instruction-tuned variants, which are further tuned to follow instructions and follow chat-style conversations respectively".
A.2 Datasets
We use the following datasets:
Dolma (Soldaini al., 2024) is three-trillion token English corpus that was used train OLMo (Groeneveld et al., 2024). It consists of text from diverse sources including books, scientific papers, code, and social media. We use the OLMo models as target models, as they were trained on Dolma. For out-members, we consider the Paloma evaluation suite (Magnusson al., 2024), since passages that had an overlap with the Paloma evaluation suite were deliberately excluded from the Dolma pretraining corpus. We use Dolma-1.7, as aligns with these OLMo checkpoints. Members are obtained from a random subset of Dolma?, while non-members are obtained from the Paloma Dolma-v1.5 subset, which is dedpulicated against Dolma-1.7 Our test set size is 1800 examples split evenly into members and non-members (sampled from the two datasets).
The Pile (Gao et al., 2020) is massive corpus of English text designed for pretraining language models. Notably, it has been used to train the Pythia models (Biderman et al., 2023b) and LLama 1 (Touvron et al., 2023a), which become our target models. Pythia includes both training and test data, sampled from the same distribution independently, which becomes the gold members and non-members respectively. Previous studies (Duan et al., 2024) have found this to be particularly challenging benchmark. Our test set size is random subset of 1800 Pile members and non-members from Duan et al. (2024)), split evenly.
A.3 Results
Our results are showm in Table and Table First, both tasks remain challenging benchmark for all tasks, as performance is relatively low across the board, particularly with OLMo. N-GRAM COVERAGE ATTACK continues to outperform DE-COP even in this challenging setting, demonstrating again the strength of using only model generations and simple n-gram coverage metrics. N-GRAM COVERAGE ATTACK performs comparatively to the Pythia models, with scores near the loss and MinK baselines. On Dolma, N-GRAM COVERAGE ATTACK actually performs better in some cases on OLMo-1B and OLMo-7B -than all white-box baselines, while performing close to the best-performing loss-based method, R-Loss, for the final two models.
"tttsss//hh.coaallaama/////11110 and http::/hhh.c/allleaai/0//-/--2 7B-82---7 -00---7872--SF-
and https://hf.. co/ a /enai/
17

---

Published as a conference paper at COLM 2025
Our conclusions here echo that in our main experiments: method, particularly with coverage and creativity as similarity metrics, is effective across domains, and is comparable and sometimes even better than white-box attacks.
Output-Only Methods
Loss-Based Methods
Model
Cov.
Cre.
LCS.
LCSp
D-C
Loss
R-Loss
zlib
MinK
Pythia 1.4B
0.53
0.53
0.51
0.52
0.50
0.54
0.56
0.53
0.54
Pythia 2.8B
0.54
0.54
0.49
0.50
0.50
0.54
0.58
0.54
0.54
Pythia 6.9B
0.53
0.53
0.50
0.51
0.50
0.55
0.60
0.55
0.55
Pythia 12B
0.54
0.54
0.52
0.51
0.50
0.56
0.62
0.55
0.56
Table 4: Comparison of membership inference attack performance (AUROC) against the Pythia suite of models on the Pile. Across Pythia model scale, membership inference with the Pile remains challenging. Bold denotes the best performance in the output-only methods, while underline denotes the best performance for the loss-based methods.
Output-Only Methods
Boss-Based Methods
Model
Rand
Cov.
Cre.
LCS.
LCS,p
D-C
Loss
R-Loss
zlib
MinK
OLMo-1B
0.54
0.54
0.51
0.50
0.49
0.47
0.51
0.45
OLMo-7B
0.54
0.54
0.54
0.51
0.5
0,47
0.53
0.51
0.46
0.49
OLMo-7B-SFT
0.52
0.52
0.51
0.5
0.47
0.53
0.51
0.46
OLMo-7B-Instruut
0.52
0.52
0.52
0.51
0.5
0.47
0.52
0.51
0.46
Table 5: Results for OLMo attacked with the DOLMa corpus. Bold denotes the best perfor- mance in the output-only methods, while underline denote the best performance for the loss-based methods.
B Experiments Details
We list further details of our experiments here, including more in-depth descriptions of loss-based baselines, hyperparemeters, and datasets.
B.1 Implementation Details
We use HuggingFace (Wolf et al., 2019) to compute loss-based baselines. For all generation, including DE-COP and N-GRAM COVERAGE ATTACK, we use vllm for fast inference (Kwon et al., 2023).
B.2 Baselines
We list further details of the baselines here, including hyperparemeters.
Reference Loss For reference loss, reference for the larger models. For example, for LLaMA 13B, 30B, and 65B, we use LLaMA 7B as the reference model. We do not run this baseline for the smallest model in the family.
smallest model of the same model family the
Min-K% (Shi et al., 2023) measures the likelihood of the 1% least-likely tokens (out- lier tokens) in the given text under the target model, i.e., Min-K % PROB(x) 1 Ex log ( (x; ,,,..., 1), where x is the input text and E is the size of Min-K (xx) set. A higher score indicates the model assigns unusually high likelihoods even to these rare tokens, suggesting potential memorization.
We run 6 variants, with K set to 10% to 60% at 10% intervals, run these on our validation set and pick the best K value before reporting the final test set.
18

---

Published as a conference paper at COLM 2025
DE-COP (D-C; Duarte et al. 2024) formulates membership inference question-answering task. Given a text passage from a source of interest (eg,, book), the method first synthesizes set of QA pairs that ask which passage is true excerpt from the source, juxtaposing the original passage against 3 synthetic paraphrases. The test statistics for membership inference is the accuracy of the target model these QA tasks intuiiiiiii, model will mark high accuracy if the model has been trained on the source of interest. While the method works for black-box LLMs, it requires a strong paraphraser (egg, Claude (Anthropic, 2023)).
Following their implementation, we generate paraphrases using a temperature of 0.1 and with the prompts described in their paper; see Duarte et al. (2024). Since we do not have API access to Claude, we instead use a competitively capable GPT-4o model (OpenAT, 2024). For the multiple choice task, we use the same prompts for both closed and open source models that they list in their paper and on their Github repository
B.3 Models
We the OpenAI API to access models. Not listed in the main experiments, specifically use 10044turbb-2224044-2 for GPT-4. For GPT-4o and 40-mini, we use 30t-40-2024-04-09 and gpt-4o-mini-e71s respectively.
B.4 Datasets
BookMIA we select a random subset of 494 random book snippets from BookMIA for testing, due to extreme computational cost of the baseline and the cost of OpenAl models.
WikiMIA we select random subset of 497 random article sumamries from WikiMIA testing, due to extreme computational cost of the baseline and the cost of OpenAl models.
C N-GRAM COVERAGE ATTACK Details
We list further details of our method N-GRAM COVERAGE ATTACK here.
C.1 Method Parameters
For our main generations, we use nucleus sampling with a temperature of 1.0 and top.p=0.95. As mentioned above, we also constrain the max length of our generations to be the exact length (in tokens) of the suffix. For our aggregation metric, we use max in all experiments, as we find it to work the best in our small validation sets.
C.2 Prompts
We list the final prompts which we use for our datasets in Table 6. For TULU, we do not use a prompt; we set the prefix to be the first user turn in chat, and allow the model to generate the full completion without any part of the ground truth response context.
D Dataset Construction
We detail our steps to construct our additional datasets released in this work.
D.1 WikiMIA2044 Hard
We use the Wikimedia API to scrape random articles from Wikipedia; we filter out stubs, lists, and disambiguous pages. Next, we check to see 1) the page existed Dec 31, 2016 and 2) if there exists an edit in 2024 or later; if not, we discard. Next, we obtain the summaries
"Phttps://githeee http://gtthubbcccmlliiaaaaaa////
19

---

Published as a conference paper at COLM 2025
Prompt Templates for N-GRAM COVERAGE ATTACK
BookMIA
You will be shown series of passages from famous literary works. After these examples, you will receive a prefix from another passage and asked to complete based on the text of a famous work. Provide only the continuation for the last given prefix without any extra commentary, formatting, or additional text.
Complete the prefix: (prefix)
ViHiMIAAWWKKMII--AAHIHM.... Hard
Continue the generation as closely to verbatim as possible.
(prefix)
Pile and OLMo
Continue the text: (prefix)
Table 6: Prompts used for N-GRAM COVERAGE ATTACK across tasks. The (prefix) place-
holder indicates where the input text is inserted.
of the page and check 1) both the old and new version are least 25 words long 2) The Levenshtein Edit Distance is above 0.5 (to ensure that there are sufficient differences and 3) The texts are no more than 20% different in their lengths. Following Shi et al. (2023), we keep only the first 256 words of the Wikipedia summary.
We identify and scrape 27000 Wikipedia pages which match the first criteria (existing in 2016, and having valid edit) in approximately 3 hours. After deduplication and filtering for length and edit distance, we are left with 1040 instances, of which we randomly sample 1000. Overall, final dataset consists of 2000 items, split evenly between members and non-members!!
D.2 TÜLU Dataset
TÜLU (Wang et al., 2023) is collection of instruction-tuning datasets. We construct MIA dataset by taking examples from the TÜLU Mix and examples from the datasets which were tested but not included; the full list enumerated in Wang et al. (2023) and in Table 8. We use only the first-turn of these datasets.
We first attempt to randomly sample from both sets to create the dataset. However, the lengths are not very similar perform binned sampling to ensure they are more even in length. First, we discard the bottom 5% shortest and top 5% longest sequences in both members and non-members to get rid of extreme responses. Next, we k 10 bins, evenly-space them, and sample from each dataset evenly in each bin to ensure that our datasets lengths are similar, and avoid spurious length correlations.
The statistics before and after pruning are shown in Table 7. Overall, our test set composition 924 members (from TÜLU), and 928 non-members from the other instruction datasets. Exact splits from each dataset is shown Table 8
We explore only Wikipedia in this case, but could also construct similar bookMIA 2024
20

---

Published as a conference paper at COLM 2025
Length Type
Original
After Sampling
Member
Nonmember
Member
Nonmember
User Length
39.6
29.5
34.2
32.1
Response Length
27.9
25.2
26.5
25.1
Total Length
67.5
54.7
60.8
57.3
Table 7: Length Statistics Before and After Sampling for More Length Matching
Member
Nonmember
Category
Count
Category
Count
GPT-4 Alpaca
133
Baize
197
OASSTI
133
Self Instruct
201
Dolly
133
Stanford Alpaca
201
Code Alpaca
133
Unnatural Instructions
162
ShareGPT
133
Super NI
163
Flan V2
133
CoT
126
Table 8: Member and Nonmember Dataset Representation
21


Reviews:
[{"review_id": "UB5IdnGegJ", "reviewer": "Reviewer_Yujw", "summary": "The paper presents a membership inference attack for black-box models. The main idea is to sample from the model using a prefix and compute similarity metrics between the model's completion and the ground truth. A threshold on these similarity scores is then used to detect whether a sample was part of the training data.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper addresses a timely and important problem: detecting potential copyright infringements in a restrictive setting where only output tokens (without probabilities) are available from the LLM's API.\n\nThe proposed method appears nearly as effective as white-box attacks. However, the overall evaluation could be improved, as discussed below.", "reasons_to_reject": "There are several papers on black-box membership inference attacks on LLMs (such as [1], [2], and [3]) that deserve a (more) detailed discussion. These works share some similarities with the current approach, so a comparative analysis of results would be valuable.\n\nWhile the authors introduced a new version of WikiMIA to mitigate distribution shift, other benchmarks used in the paper, such as BookMIA, are known to suffer from such shifts [4]. Incorporating benchmarks like MIMIR [5] could further validate the robustness of the method.\n\nIt would also be beneficial for the authors to evaluate their method using True Positive Rate (TPR) at a low False Positive Rate (FPR), which is a commonly used metric in membership inference.\n\nAlthough the authors compare their method with [6], it's worth noting that [6] is designed for document-level membership inference, where results from multiple paragraphs are aggregated. In contrast, the black-box methods referenced earlier ([1]–[3]) might serve as more appropriate baselines for comparison.\n\n[1] https://arxiv.org/pdf/2310.13771\n\n[2] https://arxiv.org/pdf/2409.13831\n\n[3] https://arxiv.org/pdf/2404.11262\n\n[4] https://arxiv.org/pdf/2406.16201\n\n[5] https://arxiv.org/pdf/2402.07841\n\n[6] https://arxiv.org/pdf/2402.09910", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wuGkWoavOf", "reviewer": "Reviewer_avj5", "summary": "This paper studies an interesting problem, but there are flaws in the contribution, method, related works, etc.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The focus on the full black-box MIAs requires fewer assumptions than previous methods.\n(2) This work is the first to explore the effectiveness of N-GRAM attack on post-training data, a very interesting perspective.", "reasons_to_reject": "(1) One major contribution of this paper is the collection of a new benchmark, WIKIMIA-2024, which, however, is almost identical to the benchmark WIKIMIA-24 released in [1]. Both benchmarks are based on WIKIMIA but adapted to up-to-date LLMs by setting a new cut-off date. The paper should discuss in detail the essential differences between these two benchmarks, if they exist. Additionally, evaluating the proposed method and baselines on WIKIMIA-24 is also necessary.\n(2) What is the meaning of edit distance (Line 210)? Why should we ensure a sufficient distance between the member and non-member? The necessity of this operation is not justified.\n(3) The related works/preliminaries can benefit from a thorough revision, especially since the discussion of MIAs and Memorization is not comprehensive and up-to-date enough. Please include a detailed discussion of more up-to-date works [1-5].\n(4) As established by Carlini et al. [6], membership inference attacks should be evaluated by computing their true-positive rate at low (e.g., ≤ 0.1%) false-positive rates, rather than using average-case metrics like AUC. Such evaluation metrics have become the de facto standard in evaluating membership inference attacks and are used by many existing works [5] [7] [8]. I suggest using two metrics: the Full Log-scale Receiver Operating Characteristic (ROC) Curve to highlight low false-positive rates, and the TPR at a low FPR, which measures attack performance at specific FPRs (e.g., 0.1%, 0.01%).\n\n\n[1]\tFu, Wenjie, et al. \"MIA-Tuner: Adapting Large Language Models as Pre-training Text Detector.\" arXiv e-prints (2024): arXiv-2408.\n[2]\tGarg, Isha, et al. “Memorization through the Lens of Curvature of Loss Function around Samples.” Proceedings of the 41st International Conference on Machine Learning, PMLR, 2024.\n[3]\tRavikumar, Deepak, et al. “Unveiling Privacy, Memorization, and Input Curvature Links.” Proceedings of the 41st International Conference on Machine Learning, PMLR, 2024.\n[4]\tZhang, Jingyang, et al. Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Models. arXiv:2404.02936, arXiv, 23 May 2024.\n[5]\tFu, Wenjie, et al. \"Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.\n[6]\tCarlini, Nicholas, et al. \"Membership inference attacks from first principles.\" 2022 IEEE Symposium on Security and Privacy (2022).\n[7]\tBertran, Martin, et al. \"Scalable membership inference attacks via quantile regression.\" Advances in Neural Information Processing Systems (2023).\n[8]\tWen, Yuxin, et al. \"Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries.\" International Conference on L", "questions_to_authors": "Additional comment: A random-guess baseline is not necessary; please consider removing it from the results. It is obvious that the AUC of a random guess is 0.5.", "extra_scores": {}}, {"review_id": "E1VGSteKu1", "reviewer": "Reviewer_zvq9", "summary": "This paper proposes a membership inference attack method for large language models (LLM). The idea is to probe the LLM with prefix strings from a dataset and see if the suffix is generated. The advantage of the method is its simplicity. The authors show that it is a promising method on some proprietary and open-source models, though I do have some suggestions for improving the experiment and discussions: \n\n1. It is nice that both proprietary and open-source models are used in experiments. However, we do not really know what is used in GPT-3.5/GPT-4, so as a matter of science there is no ground truth for your membership inference attack on those models. In fact, even for the various LLaMa variants, do you know for sure that the data samples you presume are used in training really did occur in training (it wasn't filtered)? I think it is fine to presume your new data is not in training, but the old ones are harder to say with certainty. There are several ways to address this: \n- Be more clear about this as a caveat in Section 4.2\n- Report both false positive and false negative results (or something similar that that breaks apart positive and negative samples, if you don't want to have a threshold) for at least one experiment. \n- Provide more results on Tulu (or Olmo?) where it is more certain what was in training. Table 2 for example only shows GPT-3, so I don't know whether to fully trust the numbers. \n\n2. More discussion on efficiency is warranted. The discussion in Section 4.6 says that the proposed method cost nxk tokens while previous work costs 100n, but is that difference important in practice? If I were a prosecutor representing book publishers who is using membership inference attack to decide whether to sue a LLM provider on copyright infringement, I may have a large budget. Further, I only need to find membership on just one book to provide evidence, i.e. I do not need multiple books as evidence to create a case. So I may be able to justify spending $1000 (I made up this number) on API calls since a lawsuit may be worth millions of dollars. So how exactly is your method \"efficient\" under this broader scheme? \n- I would suggest adding a table that investigates efficiency of your method in more detail. It could be something like the top figures in Fig 2, but more in-depth. For example, vary d and n and perhaps k and give recommendation on what is the recommended setting. It may also be interesting to try on sequences that have relatively frequent words vs those that don't. \n\n3. This is a minor point, but the discussion of \"white-box methods\" and \"black-box methods\" could potentially confuse the reader because these terms were not defined. \n- Do you say clearly \"white-box\" means that likelihood information is used, and that it is not white-box in the sense that internal parameters are used. Some API may provide a score for the generation so the distinction is important. \n- Can you do \"white-box methods\" on \"black-box models\"? It's not clear if methods and models are tied. \n- Many of the results have statements like \"N-gram coverage attack is more efficient than black-box methods\" where methods is plural form but really it is just one method. It is better to just say your method is better DE-COP rather than making a seemingly general claim of improvement over ALL black-box methods. \n\n4. It may be worthwhile citing the following paper. They show that using a group of sentences (rather than a single sentence) with ngram features can lead to improvements in the membership inference attack. The method is different and their setting focuses on machine translation sequence-to-sequence models, but their result corroborates well with your idea of using multiple strings and ngram overlaps. \n\nHisamoto (2020). Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00299/43536/Membership-Inference-Attacks-on-Sequence-to", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Membership inference for LLMs is an important problem considering the advances in the field and ongoing lawsuits regarding copyright, etc.\n2. The paper proposes a simple but promising solution. The experiments and comparisons are insightful.", "reasons_to_reject": "1. Membership inference is a problem that potentially has important ramifications: for example, a prosecutor may use this to decide whether to sue LLM providers for copyright infringement. Because of this, I think there needs to be more rigorous and nuanced discussion of results (beyond simply stating that the method is \"effective.\"). That said, I gave some suggestions (see \"Summary\") and I think it is not too hard to address them in the revisions.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a very simple membership inference attack in which n-gram overlap metrics are computed on multiple completions of a prefix. The output requires access to only the token outputs of the model and works better as more completions are generated, and both open and closed models are tested (the latter of course with "silver-standard" assumptions instead of the unknowable ground truth as stated by the authors). One major concern with the paper was that its evaluation was not conducted in low-FPR regimes exclusively, but the authors added results on this in their rebuttal that showed strong performance compared to baselines here. There are still concerns with comprehensiveness in the related work section, as well as some overclaiming in the writing noted by reviewers. Overall, though, it is a solid contribution to LLM membership inference research and should be used as a baseline in all future work in this area.
Decision: Accept