Infilling Score: A Pretraining Data Detection Algorithm for Large Language Models
Authors: Negin Raoof
Litu Rout
Giannis Daras
Sujay Sanghavi
Constantine Caramanis
Sanjay Shakkottai
Alex Dimakis
Venue: ICLR 2025 Poster (2025)
Keywords: Pretraining data detection
Large language models
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We introduce Infilling Score, a new method for pre-training data detection in Large Language Models based on token-level infilling likelihoods.

Abstract: In pretraining data detection, the goal is to detect whether a given sentence is in the dataset used for training a Large Language Model LLM). Recent methods (such as Min-K % and Min-K%++) reveal that most training corpora are likely contaminated with both sensitive content and evaluation benchmarks, leading to inflated test set performance. These methods sometimes fail to detect samples from the pretraining data, primarily because they depend on statistics composed of causal token likelihoods. We introduce Infilling Score, a new test-statistic based on non-causal token likelihoods. Infilling Score can be computed for autoregressive models without re-training using Bayes rule. A naive application of Bayes rule scales linearly with the vocabulary size. However, we propose a ratio test-statistic whose computation is invariant to vocabulary size. Empirically, our method achieves a significant accuracy gain over state-of-the-art methods including Min-K%, and Min-K%++ on the WikiMIA benchmark across seven models with different parameter sizes. Further, we achieve higher AUC compared to reference-free methods on the challenging MIMIR benchmark. Finally, we create a benchmark dataset consisting of recent data sources published after the release of Llama-3; this benchmark provides a statistical baseline to indicate potential corpora used for Llama-3 training.

Published as a conference paper at ICLR 2025
INFILLING SCORE * A PRETRAINING DATA DETECTION ALGORITHM FOR LARGE LANGUAGE MODELS
Negin Raoof Litu Rout
Giannis Daras
Sujay Sanghavi Constantine Caramanis
Sanjay Shakkottai
Alexandros G. Dimakis
The University of Texas at Austin
Ineginmr, Litu. rout, giannisdara, constantine, sanjay, saaaia..sakkottat. @eteexas edu sanghavi @mail www.eaa.asee...... edu, dimakis@austin. utexas.
ABSTRACT
In pretraining data detection, the goal is to detect whether a given sentence is in the dataset used for training Large Language Model (LLM). Recent methods (such Min-K% and Min-K%+++) reveal that most training corpora are likely contaminated with both sensitive content and evaluation benchmarks, leading inflated test set performance. These methods sometimes fail to detect samples from the pretraining data, primarily because they depend on statistics composed of causal token likeli- hoods. We introduce Infilling Score, a new test-statistic based on non-causal token likelihoods. Infilling Score can be computed for autoregressive models without re- re-training using Bayes rule. A naive application of Bayes rule scales linearly with the vocabulary size. However, we propose ratio stst-statistic whose computation is invariant to vocabulary size. Empirically, our method achieves a significant accu- racy gain over state-of-the thet--rrt methods including Min-K%, and Min-K %+++ on the WikiMIA benchmark across seven models with different parameter sizes. Further, we achieve higher AUC compared to reference- methods on the challenging MIMIR benchmark. Finally, we create benchmark dataset consisting of recent data sources published after the release Llama-3; this benchmark provides a statistical baseline indicate potential corpora used for Llama-3 training.
INTRODUCTION
The significant progress in language modeling can largely be attributed to development and deploy- ment of large-scale models that utilize extensive training corpora, often encompassing trillions of tokens (Li et al. 2024 Dubey al. 2024). The selection and curation of data for training such Large Language Models (LLMs) is very complex and expensive. Further, recent developers LLMs
withhold details regarding the sources of their pretraining datasets (Dubey et al.) 2024 OpenAl eet al., 2024) Touvron et al. (2023b). This lack of transparency has raised concerns regarding the inadvertent inclusion of copyrighted content (Chang al. 2023 Min et al. [2023] Meeus al. 2023)
or personally identifiable information (Mozes et al. 22033 Panda et al. 20244) potentially leading to Furthermore, the inclusion of benchmark
ethical and legal challenges (Grynbaum & Mac. (023).
datasets within the training corpora itself can compromise the integrity of model evaluations. This
practice may inflate test performance metries without accurately reflecting the model's capabilities
(Oren et al. 2023 Zhou et al.) 2023).
Recent work has focused on the problem of determining whether specific sequences of tokens have been previously seen by language model (Shi et al.) 2024 ZZhag et al. [024 Duan et al. 2024). These investigations are categorized under growing field attacks on LLMs known as Membership Inference Attacks (MIA) (Shokri al. 2017 Mattern et al.] 2023b Carlini et al. (2022). Many studies in this area focus on finettuning data detection (Song & Shmatikov 2019 Shejwalkar et al.) 2021 Mahloujifar et al.) 2021). However, pretraining data detection attacks becoming increasingly important as they can reveal whether model has been trained on potentially sensitive data and prevent evaluation data contamination (Jiang et al. 2024 Yang al. 2023)

---

Published as a conference paper at ICLR 2025
We introduce novel method for identifying whether given text sequence part of a language model's 's pretraining data. Our method uses à new test-statistic that we call the Infilling Score. Our approach performs non-causal test compute the infilling probability of a token, based on the tokens that appear before and after this token in the sentence. An autoregressive language model generates causal likelihoods (i.e. the probability of word appearing after some context). We find that non-causal likelihoods lead to more accurate tests for membership inference. These likelihoods can be computed using causal autoregressive model. The computation involves applying Bayes' rule and the law of total probability, and needs marginalization over the vocabulary to compute partition function. Unfortunately, computing this partition function requires calling the autoregressive LM many times, one for each vocabulary entry. This would require tens of thousands of calls to the autoregressive LLM to compute single non-causal probability for one token, and hence not practical. Our central idea is to propose an approximate test-statistic whose computation is much faster, does not require an exact computation of this partition function and does not depend on the vocabulary size.
Our method achieves a significant accuracy gain over state-of-the-arr methods including Min-K %, and Min-K%++ the WikiMIA benchmark across seven models. On WikiMIA, our method outperforms the previous state of the art in AUC. achieves up to 10% improvement on Llama models when testing long sequences (256 tokens). Further, achieve higher AUC compared to reference-free methods on the challenging MIMIR benchmark.
Our main contributions are summarized below:
(I) We introduce the Infilling Score, new reference-free method for detecting pretraining data using infilling likelihood of tokens within the candidate sentence (Section B While SoTA methods: MIN-K K% and MIN-K%++ rely on statistic based past tokens only, method computes a new test statistic considering both past and future tokens in the sentence.
(2) We develop an efficient algorithm for computing this new score. Though our method conceptually shares similarities with a likelihood computed via Bayes rule, computationally it is much different: whereas any natural approach for computing a Bayes rule calculation scales with vocabulary size, our algorithm has computation invariant to vocabulary size.
(3) We conduct extensive experiments on the standard (a) WikiMIA (Si et al., 2024) and, (b) MIMIR (Duan et al., (2024) verify the efficacy our method (Section 4p. On these benchmarks, we compare our method with state-of offee-offthe the-hhhrrr art MIA methods including MIN- K% (Shi et al., 2024) and MIN-K (Zhang et al., 2024). On WikiMIA, our method achieves 11% improvement over MIN-K M % and 5% improvement over MIN-K + in terms of AUROC on average. We attribute the notable performance gain of our method infilling probability (Section 3).
(4) We curate dataset of book excerpts that have not been seen by the LLMs released before April 2024 (Section (41)). Employing our Infilling Score, detect list of books which have (likely) been used for training Llama-3-8B (Dubey et al. 2024) 4.4.3)
2 BACKGROUND
In this section, we discuss the standard definition of Membership Inference Attack (MIA) and recent advances along this line of research.
and a Large Language Model (LLM) denoted by M, the goal of MIA is build detector h (2, ) - {0, 1} that can infer the membership of in the
Problem setup. Given a sentence x = f>x
training corpus D = { je(x] of M. Existing MIA methods for LLMs (Shi et al. [2024] Zhang al. 2024 Carlini et al., 1021 Mattern et al. (023a) assign a score each sample æ and use
binary threshold to determine its membership class, with 1 indicating x E D and 0 otherwise.
2.1
CHALLENGES IN PRETRAINING DATA DETECTION USING MIA METHODS
2.1.1
DETECTION DIFFICULTY
Prior works (Hardt et al. 2016 Bassily et al., 2020) have shown that the total variation (TV) distance between the distribution of seen and unseen data is proportional the learning rate, size of the
2

---

Published as a conference paper at ICLR 2025
dataset [D] and the frequency of the test sentence æ. Since TV captures the separability between these distributions, low TV makes it difficult to infer the membership class of given æ.
2.1.2 ARCHITECTURE AND PRETRAINING DISTRIBUTION
Membership inference attacks for L.MM pretraining data detection are broadly categorized into two classes: (a) reference-basee methods and (b) reference-free free methods. Reference-based methods such as Reference (Carlini et al., 2021) infer the membership of a sentence æ by computing the likelihood of æ using two different LLMs. They compare the perplexity of under the target LLM with the perplexity of a under a smaller language model. The smaller model M shares the same architecture M, and is trained on subset of samples, D, collected from the same underlying distribution of 'D. The intuition is that smaller networks have less capacity to memorize sentences from the pretraining dataset. One crucial limitation of these methods is that reference model may not always exist. Although LLM developers often do not disclose information about the distribution of pretraining data, reference-based MIAs (Carlini et al. 2021) assume the knowledge of the architecture and underlying pretraining distribution, making these methods less practical.
Among reference-free methods, Min-K% (Shi et al. 2024 hypothesizes that when sentence is seen by the model, i.e., a E D, it usually contains number of tokens with low causal probabilities (outliers). Formally, given a sequence of tokens {at) J Min-K% score given by:
Min-K%((x)
Min-K (24),
|min-k%
3; Emin-&
where Min-K%ukeen (I:) = log (x))
(2)
Here, (a; denotes the score for each token I;- The set min-k% contains k% of the input tokens which correspond the bottom 4% scores within the sequence. If the average score for this set is less than rkk), where T rkk) denotes the binary threshold for fixed k, then Min-K% detects the sequence x as "unseen". Note that the classification threshold r(k) is determined empirically using a validation dataset.
recently proposed method, Min-K % (Zhang et al. 2024 improves the detection accuracy of Min-K% by normalizing the next-tokens log likelihood log p(r, (x))) as follows:
Min-K9+++(ar) = mmin-k%
Min- tokee (2;)
(3)
Emin 3%%
log (2)) Mres
where fin--K6+tiokkol (x;))
Onci
Mag. = -Ez=p(. (age)) [log ((()(())))) and daci =V E..ppl i (logp(2(<<s) Mzee) J are the mean and standard deviation of the next-token likelihood.
Both Min-K% and Min-K%++ rely on the "causal" likelihood predictions of the model. However, the causal likelihood z; does not consider the information from the entire sentence context, as only depends on the preceding tokens Tei. We propose that sentences seen during training E D) typically have a number of tokens with low infilling probabilities. By using the non-causal token likelihoods which depend on both preceding and succeeding tokens (F<; and r>>s), achieve more accurate statistic than causal likelihoods alone. This enables our Infilling Score method to outperform previous pretraining data detection approaches on standard benchmarks.
3 METHOD
We describe our method in this section. First we describe the computation of our new ratio statistic, and explain why it offers computational scalability compared to straightforward application of Bayes Rule. Next, we describe how this score is used to detect data in the pretraining set. Finally, we explain how we employ our method to detect pretraining samples in Llama-3.
3

---

Published as a conference paper at ICLR 2025
Ground truth:
She She 21
Italian MASKED
pasta pasta 24
Masked input:
ate 22
m3
3.1 COMPUTING THE INFILLING LIKELIHOOD
In this setting, we search for the most likely token to infill my using other tokens in the sentence, i.e. (21. 22, T.). Using the law of total probability, we get:
p( 02) p( 22, 23) (p) 21,
plas [1, T2 as (psspppppe (23g ( (2)
(5)
p(ra/ar, (2)
Engey ev p( 11 #2, T'y (plagg/1, x2)
Observe that the partition function in the denominator of equation is expensive to compute as requires summation over all the tokens in the vocabulary V. In the naive case, the number of LLM calls required to compute the infilling likelihood scales linearly both with vocabulary size and the sequence length. This is because for each token, the denominator in equation scalees linearly in the vocabulary size, and this computation needs to repeated for each token. The size of the vocabulary be large 128K in recent LLM (Dubey al. 2024).
To address the scalability challenge, we introduce a ratio test statistic. Our main idea is to compute the ratio of the infilling probability of the ground-truth token and the maximum causal likelihood token. Using this proposed statistic, we bypass the need to compute the computationally expensive partition function. In the above setting, define the ratio test-statistic of token 3g as:
plas 11, 22, 22, p(ag 1 F2, x44)
where 23 = arg max p(agpp/a11ppp
xGEV
This ratio compares the infilling likelihood of the ground-truth token to that of the model's causal prediction. If 13 is an outlier the ratio is closer to O, and when Z3 is among the model's top predictions, this ratio is closer to 1. Since the partition function in equation iss the same for p(zz 12, .44) and p(z3 [1, I2, .r4). it is canceled in the ratio test statistic. This drastically reduces the number LLM calls from ONN to O(N), making our test statistic independent of the size of the vocabulary (details in 4.5). Interestingly, we can exactly compute this ratio analytically using auto-regressive models without retraining. We then compute the log of this ratio and normalize the probabilities to capture the relative significance of each token in the vocabulary. First, we derive
22,
22, 22,
log p(ag) 11, 22, r4) p(aslar, 22, (3))((
log
11,
log p( (1 T2, 23) + pogp((s/21 x2) - log (1,, log (1,
(8)
Generalizing (equation use m future tokens for calculating the infilling ratio of token i, we get the following:
log
p( 1r #++1+tm
logp(z, a F2, .2, Ti, Ti,...)-1) + log log
(9)
log,((z)l11) 27,, log (211-11
1=
where 3ı: denotes the sequence 31, 22,42, Ti. and xi arg max, ev P1
4

---

Published as a conference paper at ICLR 2025
As suggested in Zhang et al. (2024), normalize the terms to compute our infilling score for a given token Z3:
InfillnggScoreakenn (x4s)
pzz [1, 22,,.., Ti 1......--11
Maxs
log lo (11ii))) Maii
dny
Oxi
+m
log pgpp(r (1, 22 B. - Mane
log (11s-)) Mari
ons
aan
(10)
where Maniy (mm []og ((]/21:))] and Jny V E [e-p((mn p(z (1]/):)))))))) )z ) are
the mean and standard deviation of the next token log probability, log (og((ay) [21, (1,,""-j-1) over the whole vocabulary. In contrast to equation [] there is no normalization in the denominator needed in equation I0 Note that the non-causal terms in equation 6 all replaced by causal terms which can be computed through LLM logits. To implement, we need two calls to the LLM the first with input as the sequence 3i, ..., ..., v and the second call with input as I1, 2 ..., J.. Note that the means and standard deviations can be computed from these logits. Thus, equation TO requires only two calls to the L.LM per token. Hence with N tokens, the total number of calls to the L.L.M scales as 2.N, in contrast to the naive approach where the scaling is N N|V. We will see in our experiments (see Section 4.5) that this leads to a dramatic decrease in runtime, with two orders of magnitude improvement.
3.2 PRETRAINING DATA DETECTION
To detect the membership of a given sentence æ, we find the set of min-k% tokens with low Infilling Scores in the sentence, and compute the average score over this subset. Our final test-statistic becomes:
InfillingScor(2))- imin-A%
Σ
SnoillingSooreettennll (2.).
(11)
JP's Emin-K%%
Our experiments suggest that InfillingScore(ar)) higher for given sentence æ which was seen by the model during pretraining. Thus, the infilling score enables us to build detector M) for an LLM M to infer the membership class of æ as:
ha M) {
InfillingScore())
(12)
otherwise
where 7 denotes the binary threshold that applied on the soft scores.
4 EXPERIMENTS
4.1 BENCHMARKS
We conduct comprehensive tests evaluate the performance of our newly proposed test-statistic against state-of-the-art reference-based and reference-free methods. We experiment with various models and different parameter sizes. Initially, we examine the established pretraining data detection benchmarks: WikiMIA (Shi et al., 2024) and MIMIR (Duan et al., 2024). WikiMIA is temporal MIA dataset commonly used for evaluating pretraining data detection methods. This benchmark contains excerpts from Wikipedia event articles, and classifies samples based on the timestamp of the articles. Samples from articles published before the training of an L.MM are classified as "seen". and samples after the training are classified "unseen". Hence, this benchmark applies only to subset of LLMs, depending on their training and release time. WikiMIA has four different subsets with sequence lengths of 32, 64, 128, and 256. Zhhan al. (2024) also published "Paraphrased" version of WikiMIA which uses ChatGPT to paraphrase the samples.
A more challenging benchmark, MIMIR (Duan et al.) 2024), aims to evaluate pretraining data detection methods when the distributions of "seen" and "unscen" text samples have high n-gram
overlap. MIMIR consists of samples from the Pile (Gao et al. 2020) across seven domains: English
5

---

Published as a conference paper at ICLR 2025
Wikipedia, ArXiv, Github, Pile CC, PubMed Central, DM Mathematics, and HackerNews. Parts from the train subset of the Pile are labeled as "seen" while parts of the test set are labeled as "unseen". These seen and unseen samples selected to have very high n-gram overlaps, making it significantly more challenging to infer training data membership.
Previous membership inference benchmarks such WikiMIA, BookMIA (Shi et al. 2024). and BookTection (Duarte et al., (024) cannot be reliably used for Llama-3 because the model was trained more recently. To address this, we curate a new dataset consisting of book excerpts published after the release of Llama-3 labeled as "unseen" data. In this new dataset the "seen" data comes from classical fiction books published before 1965. We sample set of 100 excerpts, with each excerpt containing 200 tokens. The "unseen" data consists of excerpts from books published after April 2024, similarly having size of 200 tokens.
4.2 MODELS AND METRICS
We use the WikiMIA benchmark to evaluate our Infilling Score method on Llama (7B, 13B, 30B) (Touvron et al., 2023a), Pythia (2.8B, 6.9B) (Biderman et al.) 2023). GPT-NeoX-2005 (Black et al. 2022), and Mamba- 1.4B (Gu & Dao 2023) models. WikiMIA is applicable to models released between 2017 and 2023. Samples from the Wikipedia event articles published in and after 2023 are labeled as "unscen", and samples from articles published before 2017 are labeled as "unseen".
For experiments on the MIMIR benchmark, we evaluate our method using Pythia (160M and 1.4B) on subset of the Pile (Gao et al. 2020) dataset sampled across has been pretrained on the training set of the Pile dataset (Biderman et al.. 2023). Therefore, MIMIR benchmark has labeled samples from the train/test of the Pile as "seen' 7 uuunsee"", respectively.
different domains. Pythia model
We evaluate Infilling Score for membership classification against the ofate-of-the-ar methods using the area under the ROC curve (AUROC) metrie. As suggested in prior studies (Carlini al.) 2022 Mireshghallah et al. 2022), we also report the True Positive rate at low False Positive rate (PPR).
4.3 BASELINES
We compare our proposed method with multiple state-of-the-art methods as our baselines. Reference method (Carlini et al.) 2021) relies on the ratio of the sample perplexity (c.g. next token likelihood) estimated by the target model to the sample perplexity estimated by a smaller reference model. Zlib another reference- based method which uses the Zlib compression entropy for calibrating the score (Carlini et al. (2021). Neighbor method (Mattern et al.) (023a) replaces tokens within a sequence using a pretrained masked language model to generate similar sentences. The method identifies if a sample belongs to the training data by comparing the loss of the original sample with the average loss of its neighboring sentences. The same algorithm is also used for detecting machine generated text in (Mitchell et al. 2023). We compare results with both Min-K%
et al., (0244) and Min-K%++ K%++ methods (Zhang et al. 2024) extensively for performance evaluations because both methods are the current state-of-the-an reference-fiee baselines, falling under the same category as our Infilling Score.
4.4 RESULTS
4.4.1
EVALUATION ON WIKIMIA
Table presents the results comparing our Infilling Score method with state-of-ther the-the-rrt methods evaluated on the WikiMIA benchmark. In addition, we evaluate the effectiveness of our method using TPR at low FPR in Table 2 Our experimental setup is consistent with prior work such as Min- + +++ and Min-K %. For 32-token sequences we only use one future token, and for longer sequences we use 5 future tokens. We fix k 20% across all experiments.
On average, our method shows 5% improvement in AUC over Min-K%++ across various model sizes and different inputs sequence lengths. As hypothesized in Section 3 Infilling Score consistently outperforms existing seference-based and reference- -feee methods in detecting Llama pretraining data. We empirically show that predicting the token-level likelihoods, using the information in both the past and future tokens is more accurate for pretraining data detection. For longer sequences. This is specially helpful for samples with longer sequence lengths where there more tokens in the context

---

Published as a conference paper at ICLR 2025
Seg.
Method
Mamba-
NooX:20B
Pythia 2.88
Pythia 6.9B
Llama 7BB
13B
30B
Average
Imggh
Ori.
Pars.
Ovi.
Ori.
Oni.
Parn
Ori.
Para.
Ori.
Para.
Pir
Infilling Scoge (Ours)
06.6
66.1
75.6
73.1
65.0
63.9
69.7
68.2
$8.1
55.0
58.6
87.0
87.3
84.7
76.56
Min-K Seaa Zhine et al 2014
66.8
66.1
75.0
60.6
64.4
62.4
70.3
68.0
K5.1
54.0
54.8
82.7
84.3
81.2
74.62
32 Min K%
Shi CC RUS
63.2
62.9
71.8
69.7
61.8
61.7
66.3
65.2
66.3
67.0
68.0
68.4
70.1
70.7
66.65
Neigaber Dilonet
20256
64.1
63.6
70.2
08.3
62.1
64.5
65.8
65.5
65.8
68.0
67.6
66.3
65.73
21th (Carlan
61,8
62.5
68.2
62.3
64.3
64.2
66.7
67.3
67.8
68.3
69.8
70.4
66.04
63.5
56.2
62.3
Iotilling Scope (Ours)
67.3
62.9
73.1
61.2
89.7
96.8
90.1
54.5
81.3
Min--5s
2024
67.2
67.5
65.0
58.5
64.8
85.7
50.8
56.7
78.8
84.1
74.9
73.25
64
Min-K% Sha k0
02.2
58.0
72.1
66.1
61.2
56.8
65.0
61.1
63.33
61.8
06.0
64.0
68.5
65.7
63.71
Noighber Aitini Din
30236
60.5
60.6
67
67.4
61.3
59.6
63.2
63.1
64.1
64.7
67.1
66-7
63. 70
2iib (C Carturre 2 KU21
60.4
59.1
67.6
66.4
60.5
59.0
62.6
61.5
63.4
63.6
65.3
65.3
67.5
67.4
63.55
Ref CCram
60.6
59.6
65.7
65.9
59.6
39.2
62.4
62.9
63.4
60.9
69.0
65.4
63.88
Infilling
(Ours)
69.6
64.1
67.5
N7.6
83.4
58.3
83.5
86.7
79.5
76.23
K%+ Zhann
76.3
73.85
128
K16
66.8
72.6
06.9
64.7
09.5
67.0
70.1
08.1
71.5
68.7
73.9
70.2
Noiphbor (Mrre
3023g
64.8
62.6
60.6
61.9
67.5
64.3
68.1
64.0
72.2
67.3
66.00
21th (Crthne e1 2024
65.6
65.3
71.8
71.8
65.0
65.01
67.6
67.4
68.3
68.4
69.7
69.6
71,8
71.5
68.48
Ref
Carain (4 2014
65.2
61.1
67.8
67.8
59.6
39.5
63.5
62.9
62.6
59.7
71,9
70.0
64.23
Infilinng Score (Cuure)
70.1
77.0
73.6
70.5
96.6
95.3
89.8
81.84
Min-K x 7hune et ml 2024
65.5
71.9
63.9
65.5
82.5
82.3
77.3
72.70
256
Min- K
60.8
72.1
67.6
73.1
72.8
Table 1: AUROC results on the Original and Paraphrased subsets of the WikiMIA benchmark (Shi et al. 2024). Note that the paraphrased version of the 256-token subset of WikiMIA is not published on HuggingFace which is why some results are missing for 256 tokens. Bold shows the best result and underline shows the second best results each section. As seen, our Infilling Score method outperforms previous work for detecting pretraining samples for EleutherAI's Pythia (Biderman et al.
(002) and GPT-NeoX (Black et al. 2022). Mamba (Gu & Dao 2023), and Meta's 's Llama Toovrron ct al. 2023a) models across various model sizes.
Seq.
Method
Mamba-1 488
NeoX-201
Pythia-2886 88
Pythio-6.98
LIama-77
Liuaa-138
Llama-30le 30ll
Average
length
Ori.
Purs.
Oti.
Para.
Ori.
Paru.
Oni.
Para.
Ori.
Para.
Ori.
Para.
Oni.
Purua
Infilling Score (Churs)
14.0)
16.5
27.6
23,0
13.7
13.7
17.3
20.7
34.1
35,9
30.5
20.7
33.1
38.2
24.85
Min- KS++ Plong
2024
12.9
10.6
19.4
12.9
14.2
13.9
17.1
17.1
33.6
31.5
38,5
35,9
31.3
27.4
22.59
Min-K KS
14.7
16.0
17.6
Neighbor 13 2023a
11.9
Cartir
15.5
13.2
15.8
14.5
16.3
12.7
13.7
14.2
11.6
15.0
14.5
15.0
15.03
Ret Caran 2f mi 2011
7.8
5.9
1.5
15.2
6.2
7.2
6.7
6.2
4.7
5.4
9.8
7.5
77.01
Infilling Scupe (Ohre)
19.4
10.2
27.8
21.8
18.00
13.4
21.1
14.8
50.7
25..5
53.5
34,9
46.0
27.8
27.56
Man K1 Zhinn et al.
2014
16.0
7.0
20.4
13.0
16.2
9.9
26.1
14.1
39.4
26.8
34.1
26.4
36.3
21.5
21.58
64
Min- 5hh m114
1944
8.4
2014
17.6
18.3
11.3
19.0
12.7
14.4
13.7
17.2
13.4
17.5
14:4
15.55
Neighbor N160nn 20230
9.5
14.4
11.6
(Cafhhree
14.1
14.8
13.4
16.9
15:20
Ret
8.1
14.1
13.0
16.2
11.3
10.19
Infilling Score (Qurs)
16.6
15.8
25.9
33.1
13.4
20.9
21.6
38.
33.8
41.0
30.9
24.5
31.7
25.03
Min K Zhang et
2024
16.6
10.1
23.0
19.4
12.3
14.4
22.3
21.6
46.8
38.8
41.0
21.5
38.1
21.6
25.18
128 Min K% 5hi Shi p01
16.6
11,1
25.2
22.2
13.7
13.1
18.0
17.3
19.1
21.6
25.9
14.4
23.7
18.7
18.97
Neighbor Dalanstrer 20230
15.8
13.7
15.8
18.7
8.6
12.2
10.8
17.3
12.9
11.6
15.1
14.4
13.91
7ith (Crliin
19.4
17.3
23.0
21.6
18.7
16.6
20.9
20.9
14.4
18.7
18.7
16.9
18.0
19.4
18.89
10.1
Intilligg Scope
25.5
72.5
2014
15.7
13.7
47.1
37.3
19.6
22.70
256
Min KS Chi m00
13.7
21.6
13.7
15.7
17.6
19.6
13.7
16.51
7lth
(Carlin of
1021
23.5
23.5
19.6
22.5
21.6
27.5
20.4
24.66
Table 2: True Positive rate at low False Positive rate (FPR=5%) results on the Original and Paraphrased subsets of the WikiMIA benchmark (Shi et al., 2024). Note that the paraphrased version of the 256- token subset of WikiMIA is not published on HuggingFace, which why some results are missing for 256 tokens. Bold shows the best results and underline shows the second best results in each section. As shown, our Infilling Score method on average achieves higher True Positive rate compared to existing methods, with the best performance on 256-token long sequences.
to use for inference. Since our method offers the capability to leverage the future as well as past tokens, it shows a significant gain over current state-of-the-arr method when input sequences are long.
4.4.2 EVALUATION ON MIMIR
Table 3 shows the results comparing our Infilling Score method with SoTA methods evaluated on the challengng MIMIR benchmark. In the MIMIR dataset, samples from the "seen" and "unseen" classes are sampled from the same dataset to ensure 13-gram overlap of up to 0.8 between the classes. Reference-based models show high performance on this benchmark. However, the drawback of this
7

---

Published as a conference paper at ICLR 2025
Wikipedia
Githuh
Pile CC
PubMed Central
Method
160M
160M
160M
1.4B
Infilling Score (Ouns)
49.7
53.4
65.5
70.0
53.7
53.3
52.3
$3.5
Min-K'6 %%+ +
Zhang et al.
2024
49.7
53.7
64.8
69.6
50.6
51.0
50.6
51.4
Min-K % Shi
2012
50.2
51.3
65.7
69.9
50.3
51.0
50.6
50.3
Zlib Carriar oral.
2021
51.1
52.0
67.4
71.0
49.6
50.1
49.9
50.0
Ref (Crrmi of
51.2
55.2
63,9
67.1
49.2
52.2
51.3
53.1
ArXiv
HackerNews
Average
Method
160M
1.4B
160M
1.48
160M
1.48
160M
1.48
Infilling Score (Ours)
$1.0
51.3
53.5
50.4
50.9
52.6
53.4
54,9
Min-K% Miin-K%+++ Zhans
2024
50.1
51.1
50.5
50.9
50.7
51.3
52.4
54.1
Min- Se Shi et m. 2114
$1.0
$1.7
49.4
49.7
50.9
51.3
52.6
53.6
Zlib (Carlin
50.1
50.9
48.1
48.2
49.7
50.3
53.2
Ref Carim ot
2121
49.4
51.5
51.1
51.1
49.1
52.2
52.2
$4.6
Table 3: AUROC results on MIMIR dataset (Duan et al. 2024) for Pythia models for different sizes. Similar to Zhang et al. ) we experiment overlap of 0.8 between samples form "seen" and "unseen" class. Bold shows the best results and underline shows the second best results in each section. As shown, Infilling Score method overall outperforms existing reference- free and reference- -bsed methods.
subset of MIMIR with maximum 13-gram
Contamination Rute
1817
Persuasion
99
2006
Oaklcaf
1812
Grimms' Fairy Tales
73
2003
The Sacred Land
73
Howl's Moving
2009
CATCHING FIRE
68
1991
Red Magic
66
Grade
1998
Mad Ship
61
1996
Tho Good to
Bad Crouching Vampire, Hidden Fang
2009
1889
Three Men in a Boat (To Say Nothing of the Dog)
56
Something from the Nightside
2009 The Silver Eagle
1982
The Man From St. Petersburg
53
Ship Destin
2008 The Painted Man
53
2007
The Center Cannot Hold
52
Raintree: Sanctuary
2005
Sister of the Dead
52
2006
The Corfu Trilogy
Ascendancy of
Table 4: Books detected in the pretraining data of Llama-3-8B (Dubey et al. 2024 Contamination rate shows the percentage of excerpts sampled from the books which were classified as "seen" using the Infilling Score method.
approach is that it requires testing multiple different LLMS to determine the best performing baseline (Duan et al., 2024 Zhang et al.) 2024). Despite the competitive nature of the benchmark, our Infilling Score achieves the best performance compared to both reference-free and reference-basee models average over different domains.
4.4.3 DETECTING PRETRAINING DATA OF LLAMA-3
We apply Infilling Score to detect books that were likely used in the pretraining of the Llama3-8B model, recently released by Meta (Dubey al. 2024). Llama3 is known to be trained using over 15T tokens of data (7x larger training set than Llama-2) according to Dubey et al, (2024). No information about the source and distribution of this data disclosed by the developers, making it difficult to construct labeled MIA dataset of books suitable for this model.
We used our books dataset as validation set to find the best hyperparameters (k%% and # future tokens, m, and the classification threshold 7) for identifying samples used in pretraining Llama3. Since Llama3 has been released in 2024, existing temporal benchmarks such as WikiMIA, BookMIA (Shi al.) (202)), and Book (Duarte et al. 2024) cannot be used for pretraining data detection on this model. We found that using the next 100 tokens when calculating the Infilling Score shows the highest accuracy this benchmark. Table T2 shows the performance of method this dataset.
8

---

Published as a conference paper at ICLR 2025
-30 -50 -40 -30 -20
Figure I: Figure shows an example of the distribution of the Infilling Scores for "seen" and "unseen" excerpts in our validation dataset which consists of from fiction books. Scores are normalized in each distribution. The unseen data comes from recent novels published after the training of Llama For the classic novel Persuasion, our method detects 99% of the excerpt be in the training set. As seen in this histogram, the distribution for Persuasion matches other seen novels and is clearly separated from unseen data, as one would expect.
We employ our method on 20,000 excerpts sampled from 200 books. Table presenters the list of books which we found to be in the training dataset of Llama3-8B with 2 50% contamination rate. Contamination rate shows the percentage of excerpts detected as 'seen' for each publication. Figure shows that books with high contamination rate have higher sample statistic overlap with the "seen" excerpts in our validation dataset.
4.4.4 ABLATION STUDY ON THE NUMBER OF FUTURE TOKENS TO USE
is important to note that the number of future tokens used to calculate the Infilling Score determines the performance gain of our method. As shown in the Figure 2 does not necessarily lead to higher AUC. However, on the WikiMIA benchmark, using about future tokens leads to relatively better AUC across various context lengths WikiMIA using Llama- 7B and Llama-1 13B. We conduct all experiments with different input sequence lengths (32, 64, 128, and 256) to examine the effect of the number of future tokens across various context lengths. While the ideal number of next tokens to use remains consistent across various model sizes, the optimal number may differ depending on data distribution and model architecture. We investigate various N), where represents the input sequence length. It's
increasing the number of future tokens
values for m within {0, 1, 3, 5, 10, important to note that the hyperparameter search does not increase the computational complexity, as incorporating additional future tokens does not require extra calls to the LLM. We provide additional results in Appendix
0.9
0.94
0.92
030
VISGNIAA44
0.88
MI3:HEA226
MISCMTA128
0.88
038
0.82
0.22
10
20
Eature Tokeus
Ruture Tokett
Figure 2: The figures show the AUROC achieved by the Infilling Score as the number of future tokens increases. These results are shown for input sequence lengths of 32, 64, 128, and 256. The left figure presents the results for Llama-7B, while the right figure shows the results for Llama- 13B. Our baseline, representing existing methods, tokens to use is for sequences of 32 tokens. For longer sequences of up to 256 tokens, the optimal number is around for both models.
zero future tokens. The optimal number of future

---

Published as a conference paper at ICLR 2025
4.5 ALGORITHM RUNTIME
Table compares the runtime of our Infilling Score algorithm with straightforward application of Bayes, and Min-K (Shi et al. 2024) using LJama-7B. Although both the naive approach and Infilling score are slower than Min-K%. these methods yield more accurate estimate of token likelihoods for membership inference. Note that our proposed test statistic, Infilling Score, significantly reduces the computational complexity compared to the naive approach, delivering an accurate membership inference score within a feasible runtime. WikiMIA dataset has 776 sequences of length 32, 542 of length 64, 250 of length 128, and 82 of length 256 tokens. The compute cost increases with the sequence length. The 256-token sequences require approximately 2,460 seconds compared to 776 seconds for 32-token sequences (30 seconds per sequence), highlighting the trade-off between detection accuracy and computational efficiency.
Seq. length
Min-K S+++
Infilling Score
Naive Approach
32
0.028
0.952 see.
207
64
0.042 see.
3.11 sec.
334 sec.
128
0.064 sec.
9.47 sec.
581 sec.
256
0.106
29.98 sec.
1141 sec.
Table 5: Algorithm runtime results comparing Infilling Score, Min-K%+++ and the naive approach discussed in Section 3 sequences of 32, 64, 128, and 256 tokens using Llama-7B on a H200 GPU.
To evaluate the impact of the number of future tokens used, m, on the runtime, we measure the runtime using 1, 5, and 10 future tokens. As diseussed in Section [31]] the number of LLM calls required by our Infilling Score algorithm is independent of the number of future tokens used. However, increasing the number of future tokens also increases the number of terms in the summations in equation 10
The additional computations have minimal impact on the runtime as shown in Table 6
length
32
64
128
256
# funure tolena
10
10
Runtime
0,952
0.953
0.956
3,11
3.12
se12
9.47
9,48
9:49
29.98
30,01
30,04
Table 6: Algorithm runtime as the number future tokens used increases. As the table indicates, increasing the number of future tokens to use has minimal impact runtime.
5 CONCLUSIONS
Limitations One limitation is that computing the Infilling Score requires grey-box access to the LLM, meaning access to the sample log probabilities estimated by the model. This requirement is common among most of the existing membership inference methods. Another limitation of our approach lies in its runtime complexity. As described in Section 3.1 the order of L.M calls required for computing the infilling likelihood (for sequence of length N) with the naive Bayes method is NVV, which scales linearly with both sequence length N, and vocabulary size | By introducing the Infilling Score, we reduce the number of LLM calls to 2.V. However, prior methods such as Min-K%and Min-K require only a single LLM call (to test a sequence length N), and are faster compared to our proposed algorithm.
To conclude, we proposed a novel method that can detect if text sequences have been present in the training set with significantly better accuracy compared to prior work. Our new test statistic allows us to derive non-causal likelihoods (up to a multiplicative factor) from pre-trained autoregressive models and may have other uses, beyond membership inference. Although our method is slower compared to previous methods, it can be practically run in a few seconds for large foundation models.
Our results present evidence that numerous books and other recent sources of text have been in the training data of modern LLMs. This test can further be used for measuring dataset contamination rates, and also evaluating decontamination methods. An important research direction would be to create larger evaluation datasets for membership inference, and include high n- n-gram overlap samples for recent sources that remain unseen llama3 and other recently released frontier models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research has been supported by NSF Grants 2019844, 21 2112471, 1, AF 1901292, CNS 2148141, Tripods CCF 1934932, Tripods 2217069, NSF AI Institute for Foundations of Machine Learning (IFML) 2019844, the Texas Advanced Computing Center (TACC) and research gifts by Western Digital, Wireless Networking and Communications Group (WNCG) Industrial Affiliates Program, UT Austin Machine Learning Lab (MLL), Cisco and the Stanly P. Finch Centennial Professorship in Engineering.
REFERENCES
Raef Bassily, Vitaly Feldman, Cristóbal Guzmán, and Kunal Talwar. Stability of stochastic gradient descent on nonsmooth convex losses, 2020.
Stella Biderman, Hailey Schoelkopf. Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. Gpt-neox-20b: An open-source autoregressive language model, 2022.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Hlerbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramer. Membership inference attacks from first principles, 2022.
Kent K. Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. Speak, memory; An archaeology of books known to chatgpt/gpt-4, 2023.
Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer,
Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? arXiv preprint arXiv:2402.07841. 2024.
André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, and Lei Li. De-cop: Detecting copyrighted content in language models training data, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. and Angela Fan et. al. The llama 3 herd of models, 2024. URL httlhttp::/// arxia. org. abs / 2477 21783
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Michael M. Grynbaum and Ryan Mac. The times openai and microsoft a.i. of copyrighted work. hnttps://wwat http:::://www..ttteeeeee. imes com 2atomm//222/////1111 htt.//uusiiesssee a/ new-york-t.. mes-tttees-ppeeee open -ttiiimmiooooftttoaooo llwsuii httm] 2023.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023.
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent, 2016.
Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo. Investigating data contamination for pre- pre-training language models, 2024. URL (tLhttps //aaxxvv.orr/abs//22200 9
11

---

Published as a conference paper at ICLR 2025
Jeffrey Li, Alex Fang, Georgios Smymiss, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Kch, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng- Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song. Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby. Hadi Pouransari, Alexander Toshev, Stephanie Wang. Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, and Vaishaal Shankar. Datacomp-Im: In search of the next generation of training sets for language models, 2024. WtRhhttss://eat
org/ abs 2406. 11794
Saeed Mahloujifar, Huseyin A. Inan, Melissa Chase, Esha Ghosh, and Marcello Hasegawa. Member- ship inference on word embedding and beyond, 2021.
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick Membership inference attacks against language models via neigh- bourhood comparison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 11330-1 1343, Toronto, Canada, July 2023a. Association for Computational Linguisties. doi: 10. 18653//1//12223 findings-acl..11. URL aattps://aclanthoolee
org/ 2023 findings acl. 719
Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, and Taylor Kirk-iiikaatiic. Membership inference attacks against language models via neigh- bourhood comparison, 2023b.
Matthieu Meeus, Shubham Jain, Marek Rei, and Aves-Alexandre de Montjoye. Did the neurons read your book? document-level membership inference for large language models, 2023.
Sewon Min, Suchin Gururangan, Eric Wallace, Hannaneh Hajishirzi, Noah Smith, and Luke Zettlemoyer. Silo language models: Isolating legal risk in nonparametric datastore, 2023.
Fatemehsadat Mireshghallah, Kartik Goyal, Archit Uniyal, Taylor Berg- Keng-Kirkpatrick, and Reza Shokri. Quantifying privacy risks of masked language models using membership inference attacks, 2022.
Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature, 2023.
Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D. Griffin. Use of 1lms for illicit purposes: Threats, prevention measures, and vulnerabilities, 2023.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, and Shyamal Anadkat et. al. Gpt-4 technical report, 2024.
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B. Hashimoto. Proving test set contamination in black box language models, 2023.
Ashwince Panda, Christopher A. Choquette-Choo Zhengming Zhang, Yaoqing Yang. and Prateck Mittal. Teach Iims phish: Stealing private information from language models, 2024.
Virat Shejwalkar, Huseyin A. Inan, Amir Houmansadr, and Robert Sim. Membership inference attacks against nlp classification models. 2021. URL huttps://app wwwant II:
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Dangi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models, 2017.
Congzheng Song and Vitaly Shmatikov. Auditing data provenance in text-generation models, 2019.
12

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaeí, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Shuo Yang, Wei-Lin Chiang. Lianmin Zheng, Joseph Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023. URL https
7/ar //rxiiv.rrr/abs//311..11 .4850
Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Yang, and Hai Li. Min-k%++: Improved baseline for detecting pre-training data from large language models. arXiv preprint arXiv:2404.000 002936, 2024.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao Xu Chen, Yankai Lin, Ji- Ji-Rong Wen, and Jiawei Han. Don't 't' make your Ilm an evaluation benchmark cheater, 2023.
13

---

Published as a conference paper at ICLR 2025
A CHOICE OF HYPERPARAMETERS
Infilling Score algorithm has two hyperparameters: which represents the number of future tokens to use, and which represents the k% tokens with minimum probabilities to use. We sweep over 1, 3, 5, 10 and 20 future tokens, and k = 0.1, 0.2, Tables 8 9 and I0 show AUROC and TPR at low FPR results on WikiMIA subsets with sequence lengths of 32, 64, 128, and 256. Based on the results, the optimal number of future tokens is for sequences of 32 tokens and 5 for longer sequences. We find that A = 0.1 often works best across different model sizes and sequence lengths.
1Inma-138 F
LIums--30B FRR&TPRSS
(Min-k%))
TPR
AUROC
TPR
T
17.80
37.00
38.60
87:
0.4
$3.10
36,00
25. 00
38.50
35.20
27.50
87.30
37.50
28.20
0.5
$3.00
37.30
25. 30
38.50
36.90
25,60
87.
33.50
27.10
0.1
$8.20
34.70
70.
89.00
32.10
29.50
86 35.30
36.00
27.90
0.2
$7.80
35.70
31.00
$8.70
38.30
28.70
86. 00
37.50
27.10
0.3
$7.80
37.30
31,80
38.50
26.30
28
86.00
37.50
26.90
044
87.80
16, 00
32.00
88.70
18.30
10. 30.50
37.55 50
27.10
$7.70
37.30
28.20
$8,70
38,00
30,00
$6.60
28.70
35.50
34,10
87.30
03.
87.
87.
28
85,
0.2
36,60
41.10
26.60
$6,50
37.50
26.90
85
40.50
27.50
10
0.3
$6.60
41.40
27. 10
$6.50
36.20
26.40
85, 20
40.00
27.90
044
$6.60
41,80
26.
$6.70
35.50
26. 26.10
85.20
41.40
27.90
0.5
$0,40
41.60
25.10
36,50
36.00
26.40
85.30
401.90
27.90
$2.00
47,30
22.70
31.20
44.20
19.50
81.30
47.50
22.20
0.2
$1.80
47,80
23.50
81.00
45.50
30.30
81.30
49. 10
22.30
20
$1.80
47,80
22.50
80.90
46.50
81.10
49.40
22.20
21.70
Table 7: Complete Infilling Score results testing Llama-7B, 1lama-11BB and Llama-30B models on the Original subset of the WikiMIA 32-token sequences (Shi et al.) 2024). For this subset, using one future token results in the best performance.
Llams 7D FPR 11995
Llama: 130
Liama 30B AFRR51
(Min-8%)
ALROU
HP%011999
HPRG
AUROC
29.30
87.30
36.40
87.
39.00
45.80
39.50
47.20
87
35.70
39.10
37.60
41.20
90.00
$2.10
88
36.30
0.2
39.60
13.
39. 70
90.30
37.60
$3.50
88
35.70
37.30
0.3
30.00 60
37.50
37. 30
90.00
38. 50
$2.50
88 00
35.30
37.30
0.4
89.70
37.60
39.00
90.00
38,40
$3.50
88.00
35.70
37.70
0.5
$0.70
37.60
39.40
90.300
49.40
$3.20
88.00
35.70
37.70
D.I
70
28.00
45.0 80
90.10
37.30
48.50
88, 30
35.30
45.00
0.2
89.70
37.20
45.40
90.10
39.50
49.30
88. 30
34.90
43.70
38,40
88.
34.30
90.00
41.10
$2. $2.10
$8.70
41.50
47.50
87.30
38.00
38.70
0A4
33.70
40.70
$3.20
88.80
41.90
47.50
87.30
33.40
39.10
0.5
$8.60
41.10
$3.50
$8.50
42.30
46. 80
87. 27.30
37.45
38.00
0.1
$3.00
51.90
33.10
82.50
51.30
26.40
82.
49 39,60
36.30
0.2
83.00
52.70
32.60
$2.50
50.50
27.
82.50
49.00
35.60
30
0.3
$2.90
$2.70
32.70
$2.50
$0.80
27.
82.50
49.30
35.00
83.00
52.30
32.40
82.50
50.80
27.50
82,50
49.20
35.60
Table 8: Complete Infilling Score results testing Llama-7B: Llama-13B. and Llama- 30B models on the Original subset of the WikiMIA 64-token sequences (Shi et al. 2024). For this subset, using five future tokens results in the best performance.
14

---

Published as a conference paper at ICLR 2025
1ama-138
Hama-10R
TPROITROS
36.80
35.30
40.30
$4.70
0,4
30.80
36.90
36.00
$8,20
34.20
39.00
86.70
57.50
24.30
0.5
$6.80
36,90
36.70
38.20
34.30
41
84.70
35.00
24.50
0.1
$7.20
17.80
22.
87.3
36.00
36.70
85.
39.40
15.70
0.2
$7.10
37.80
23.70
87.80
34.20
37.40
85.20
39.00
14.40
0.3
87.10
$7.50
23.00
87.50
34,30
37.
85.
39.60
15.70
87.10
37.80
23,70
87,80
35,10
37,40
85.20
39.60
13,70
37,80
22.30
36.70
$8.30
86
23.70
39,00
33,80
86
0.2
$7.50
44.10
24.50
87.50
40..00
34.50
86
38.700
18.00
10
0.1
$7.60
43.20
23.70
87.50
40.33
35,
85.00
38.70
15.00
0.4
$7.50
44.10
23.70
87.50
39.00
35,
85.90
37.50
18.00
$7.50
43.20
23.70
$7.50
39,60
33,90
85.50
38,70
15.00
0.1
$1.50
54.10
21,60
$2.30
53.30
18.70
85.30
$2.30
20.50
$1,40
55,00
23,00
82.20
52.30
18.70
83,10
51.40
20.90
Table 9: Complete Infilling Score results testing Llama-7B, Llama-1 13B, and Llama-30B models on the Original subset of the WikiMIA 128-token sequences (Shi et al., (024). Again, using five future tokens results in the best performance.
Liama-7H PPRETPROS
Llama-138 138 FPRSS
Liama 30BB FRRATPROS
# futuro tnkeas
k (Min-49)
AUROC
TPR@FFROS
AUROC
FPRO
AURRC:
TPR FPROSS
0.1
91.80
$1.80
50.40
92.90
25.80
66:70
85.60
32:50
21.60
0.2
93.80
$1.80
30.40
92,80
29.00
68.50
85,60
35.50
21.60
0.3
93.70
51.60
80.40
92.70
29.00
66.70
85.00
32.30
19.00
93.80
51.80
80:40
92.70
29.00
66.70
85,60
35.32 30
21,60
80,40
85.70
74.
72.50
72.50
95.30
90
22,60
74.50
22.50
80.40
0,2
90.60
22.60
74.50
93,30
22.60
80,40
89.30
35.50
47.10
0.1
96. 50
25.88
74.90
95.20
22.60
80.40
89 30
35.50
47.10
0.4
96.60
22.80
74,50
95.20
22.60
80.40
89 50
35.50
$7.10
0.5
96.60
25.80
74,50
95.20
22.60
30,40
89
35.50
47.10
0.1
95.70
20,00
78
93.00
29.00
$4.00
87.40
45.20
49.00
0.2
95.90
25.80
78
93.10
29.00
54,90
87.55
45.20
$9.00
$7.00
68.
$1,00
0.4
93.70
22.60
68.00
90.50
35.50
31,00
85.70
48.40
35.30
0.5
93.50
22.66
66.70
90.50
35.50
$1.00
85.70
48.40
35.30
Table 10: Complete Infilling Score results testing Llama-7B, Llama- 13B. and Llama-30B models on the Original subset of the WikiMIA 256-token sequences (Shi et al. 2024).. Similar to the WikiMIA 64-token and 128-token sequence subsets, using 5 future tokens results in the best performance.
15

---

Published as a conference paper at ICLR 2025
Infilling Score
Min-K%++
Comparison
Sequence Length
Model
AUROC (%)
Std Err
AUROC (%)
Err
Difference (%)
p-value
llama-76
89.185
1.173
85,182
1.328
4.003 + 1.130
0.000
32 tokens
llama-133 13b
88,850
1.232
84,852
1.333
3,998 + 1.222
0.004**
Ilama-30b
87.628
1.236
84.390
1.329
3.239 + 1.157
0.006**
llama-70 7b
89.788
1.341
85,922
1.659
3.866 1.492
0.012
tokens
llama-136
90.029
1.265
85,692
1.642
4.338 + 1.539
0.010
llama-30b
88.206
1.447
84.828
1.705
3.378 + 1.601
0.040
llama-78 7b
87. 764
2.272
84,8%
2,395
2,468 + 2.654
0.348
128 tokens
Ilama- 13b
88, 145
2.214
83,740
2.463
4,405 + 2.649
0,080
llama-30b
86,207
2.797
82.398
2.602
3.809 + 1.993
0,064
llama-7b 7b
96.307
1.761
82.354
4.662
13.952 4.296
0.000***
tokens
llama-133
95.124
2.271
82.326
4.740
12.797 3,952
0.000***
llama-30b
90.737
3.782
77.411
5.643
13.326 4.459
0.002
Table 11:
Comparing performance of Infilling Score versus Min-K%++ across different sequence lengths and model sizes. Results show bootstrap estimates with 1000 iterations. The mean difference indicates Infilling Score's improvement over f%-+-K%+++ Statistical significance is denoted as: (p 0.05), ** 0.01). *** 0.001).
B ADDITIONAL RESULTS
B. STATISTICAL ANALYSIS: INFILLING SCORE VS. MIN-K%+ WE EMPLOY BOOTSTRAP-BASEI STATISTICAL COMPARISON TO EVALUATE INFILLING SCORE AND MIN-K +++ WE USE 1,000 BOOTSTRAP ITERATIONS TO ESTIMATE THE THE MEAN DIFFERENCE BETWEEN AUROC METRICS FROM THESE METHODS, ALONG WITH THE STANDARD ERRORS TO CONSTRUCT 95% CONFIDENCE INTERVALS FOR THE TRUE PERFORMANCE GAP. OUTPERFORMS MIN-K + ACROSS DIFFERENT SEQUENCE LENGTHS (32, 64, 128, AND 256 TOKENS) AND MODEL SIZES (7B, 13B, AND 30B PARAMETERS).
THAT INFILLING ScoRE CONSISTENTLY
B.2 DETECTING PRE-TRAINING DATA FROM BOOKS
We compare the AUROC of Infilling Score with existing methods on labeled validation subset of book excerpts. As discussed in Section 4.1 this validation subset contains book excerpts labeled as "seen" and "unseen". Infilling Score significantly outperforms existing methods in detecting "seen" examples.
Method
AUC
Infilling Score (Ours)
0.79
Min-K Min-K%++ % (Zhang et al. 2024)
0.53
Min- K ( Shi et al. 2024
0.71
Zlib Carl lni al.
0.68
Table 12: Comparing AUROC of Infilling Score, Min-K%++, Min-K and Zlib methods on the validation dataset, detecting book excerpts in Llama3-8B pretraining data.
C COMPUTE RESOURCES
We ran experiments A100 (40 GB) and H200 (120 GB) GPUs. Testing Infilling Score on the WikiMIA benchmark on an 100 node takes approximately between 20 minutes (for a 3B parameter model) and 35 minutes (for 30B parameter model). For Llama models, used float16 data type. On the MIMIR benchmark, where there are 1000 long samples per class, the test approximately takes 10 hours each subset A 100 node.
16

---

Published as a conference paper at ICLR 2025
D INFILLING SCORE ALGORITHM
Algorithm 1: Infilling Score
Input:
Sequence: 211 R2... Threshold
for
to N do
Compute 1o 1)
Macci + Bxwppt (o pi 1)]
ozei + ( (log pss Mage ]4]
Find x +- arg , ((;
Compute log p(2 (01...;-1)
(log p(2.pp((......--. for
Price Irnge
to m do
Compute log p(z, (+1))
Compute log p(z, (....)
11
r + (log p(r))lnn..15-1)) (aci)/aaci (ogg(zyylaa..a ...1)-1) pucillasss
12
end
(x.
end
15 (ar)K%((r + 1% of tokeus from x with the lowest IntilingSSore (2t)
InfilinnScorr( m² Ex, Emn=x ((n
17 return InlillingSScurertt (x) < T
17


Reviews:
[{"review_id": "bqpNKcIUR6", "reviewer": "Reviewer_kRF2", "summary": "The paper introduces the Infilling Score, a method developed based on Min-k% and Min-k%++ to detect whether a given text sequence was part of a language model’s pretraining data, which is the traditional MIA setting. This method builds on existing approaches for membership inference attacks by using non-causal token likelihoods to improve detection accuracy. The authors propose a ratio test-statistic for efficient computation, and demonstrate its effectiveness on various benchmarks such as WikiMIA and MIMIR, where the proposed method performs slightly better than the compared baselines, especially on longer text sequences.", "questions": "See the weaknesses above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The motivation of combining the exisitng method and likelihood probability is straightforward and easy to understand.\n2. This paper presents extensive experiments to demonstrate the effectiveness of the proposed method. And the case study with the newly-released model Llama 3 could be useful for real practices.", "weaknesses": "1. The contribution in this paper is quite marginal. The propose framework is more like an extension of the previous methods like Min-k% and Min-k%++, which utilizes token probabilities to make predictions. The token probabilities approach have been shown to be less effective for white-box settings in MIA studies. While I understand the grey-box access might raise more challenges, the modifications of previous methods in this paper can hardly contribute to new insights into this direction.\n\n2. It seems that the complexity of the proposed framework is proportional to the number of tokens considered in the method. As shown in Sec. 4.5, the runtime of the proposed method is indeed much higher than the previous Min-k%++ methods. Would this become a concern when using such method in real practices?\n\n3. Despite the increased complexity, the improvements over the baseline models in terms of performance in these benchmarkes are also not obvious. In many cases, Infiling Score can only achieve similar performances with Min-k% and Min-k%++.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHdcT0foq0", "reviewer": "Reviewer_ZiHR", "summary": "In pre-training data detection, this paper proposed Infilling Score, a new test-statistic based on non-causal token likelihood. This paper also proposed a ratio test-statistic whose computation is invariant to vocabulary size. Experiments showed the method achieves an accuracy gain over state-of-the-art methods, including Min-K%, and Min-K%++ on the WikiMIA benchmark. Finally, the paper constructed a benchmark dataset that consists of recent data sources published after the release of Llama-3; this benchmark can provide a statistical baseline to indicate potential corpora used for Llama-3 training.", "questions": "1. I wonder \\tau in Sec. 3.2 is also a hyperparameter to be fixed. The value and how the authors fixed it were not mentioned in the experiments  Please explain them.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It is natural but good idea to introduce a new test-statistic that is based on non-causal token likelihood. Empirically, the method achieved an accuracy gain over state-of-the-art methods.", "weaknesses": "1. While the method empirically achieved an accuracy gain over state-of-the-art methods in Tables 1 and 3, I wonder the differences are really significant. If so, it is better to mention in what significance test the authors found they are significant.\n\n2. While the computation can be invariant to the vocabulary size in the proposed method, it seems that there exists a tradeoff in terms of the sequence length. The performance is lower for shorter instances than longer ones. However, for longer instances, the computation is higher. In Sec. 4.5, since the authors did not show the size of the datasets with different lengths, it is difficult to judge whether the differences in the speed in Table 5 are negligible or not. It is better to clearly mention the number of instances for each dataset. \n\nIt is more convincing when, if possible, the authors can try to estimate the runtime for actual settings where we try pre-training data detection for a specific dataset of a certain size, rather than for the standard benchmark dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pyFcwzOqQS", "reviewer": "Reviewer_7osK", "summary": "This paper proposed a pretraining data detection approach that utilizes the non-causal token likelihoods which depend on both preceding and succeeding token probabilities.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper has enough novelty. \n2. It includes all the previously related works and lists the differences.\n3. It has detailed experiments and results analysis.\n4. The paper writing is clear, and the visuals are good.", "weaknesses": "The proposed approach has a slower speed compared to the previous best approach. However, I believe it has little effect on the impact because contamination detection may not need to be so fast.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkg4Fxdiv8", "reviewer": "Reviewer_T2Q9", "summary": "This paper studies algorithms to detect whether a given sentence is the pretraining data used for training an LLM. The paper proposes “Infilling Score”, a test-statistic score based on non-causal token likelihoods. The proposed score works by computing the infilling probability of a token based on both its past and future tokens, whereas similar existing methods, such as MIN-K% and MIN-K%++, rely on a statistic based on past tokens only.\n\nThe motivation for Infilling Score is that incorporating future tokens should yield a more indicative measure. While the most intuitive method to incorporate future tokens is based on the Bayes rule, the paper argues that it would involve marginalising over all possible tokens in the vocabulary at each time step (i.e., each token in the test sequence). Therefore, this paper proposes an approximation (in Equation 6) to circumvent the expensive marginalisation, requiring 2 * the number of test tokens calls instead of the number of vocab size * the number of test tokens calls.\n\nExperiments are conducted on standard datasets, including WikiMIA and MIMIR, and the proposed method is compared against other reference-free methods (e.g., Min-K% and Min-K%++) as well as other reference-based methods. Experimental results show that Infilling Score outperforms existing methods in the majority of tasks. In addition, the paper complies a more recent data from book excerpts and applies the detection algorithm on Llama3.", "questions": "In the of data contamination detection, I wonder how this method works if the example to be tested (x) is partially contaminated. Would it be more or less sensitive to partial changes compared to other methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The idea is simple, yet novel and effective. The method extends the existing reference-free pretraining data detection algorithm (Min-K++%) by incorporating the information from future tokens, and the proposed method yields improved performance.\n- While it is not as efficient as Min-K%, the proposed algorithm is more efficient than the implementation of the naive Bayes rules. This is noted and examined in the paper.\n- Comprehensive experiments on both standard datasets as well as a case study on new datasets (to avoid likely contamination).\n\nOverall, this paper improves the understanding in the field of contamination/training data detection.", "weaknesses": "- It is not clear in Section 4 (experiments) how Infilling Score (Equation 6) performs compared to the naive Bayes rule (Equation 5). The paper provides run-time in Section 4.5, but I cannot find the performance comparison.\n- [nitpick] similar to Min-K% & Min-K%++, this method is limited to grey-box scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)