Proving Test Set Contamination in Black-Box Language Models
Authors: Yonatan Oren
Nicole Meister
Niladri S. Chatterji
Faisal Ladhak
Tatsunori Hashimoto
Venue: ICLR 2024 oral (2024)
Keywords: language modeling
memorization
dataset contamination
Primary area: societal considerations including fairness, safety, privacy

Abstract: Large language models are trained on vast amounts of internet data, prompting concerns that they have memorized public benchmarks. Detecting this type of contamination is challenging because the pretraining data used by proprietary models are often not publicly accessible.

We propose a procedure for detecting test set contamination of language models with exact false positive guarantees and without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchangeable benchmark should be equally likely. In contrast, the tendency for language models to memorize example order means that a contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of a canonically ordered benchmark dataset is significantly higher than the likelihood after shuffling the examples.

We demonstrate that our procedure is sensitive enough to reliably detect contamination in challenging situations, including models as small as 1.4 billion parameters, on small test sets only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Finally, we evaluate LLaMA-2 to apply our test in a realistic setting and find our results to be consistent with existing contamination evaluations.

PROVING TEST SET CONTAMINATION IN BLACK BOX LANGUAGE MODELS
Yonatan Oren' Nicole Meister Niladri Chatterji', Faisal Ladhak?, Tatsunori B. Hashimoto' Stanford University. "Columbia University yonatancEcs. www.annnnssssssoo....oooo stanford. edu fnmeist, niladric, thashim estanford.edd theshinyestanffrd..e faisalêes columbia.
ABSTRACT
Large language models are trained on vast amounts of internet data, prompting concerns and speculation that they have memorized public benchmarks. Going from speculation to proof of contamination is challenging, as the pretraining data used by proprietary models are often not publicly accessible. We show that it is possible to provide provable guarantees of test set contamination in language mod- els without access to pretraining data or model weights. Our approach leverages the fact that when there is no data contamination, all orderings of an exchange- able benchmark should be equally likely. contrast, the tendency for language models to memorize example order means that contaminated language model will find certain canonical orderings to be much more likely than others. Our test flags potential contamination whenever the likelihood of canonically or- dered benchmark dataset is significantly higher than the likelihood after shuffling the examples. We demonstrate that our procedure is sensitive enough to reliably prove test set contamination in challenging situations, including models as small 1.4 billion parameters,
small test sets of only 1000 examples, and datasets that appear only a few times in the pretraining corpus. Using our test, we audit four popular publicly accessible language models for test set contamination and find little evidence for pervasive contamination.
INTRODUCTION
Large language models (LLMs) have driven remarkable improvements number of natural lan- guage processing benchmarks (Wang et al. 2019) and professional exams (OpenAl 2023). These gains driven by large-scale pretraining massive datasets collected from the internet. While this paradigm is powerful, the minimal curation involved has led to growing concerns of dataset contamination, where the pretraining dataset contains various evaluation benchmarks. This con- tamination leads to difficulties in understanding the true performance of language models such as whether they simply memorize the answers difficult exam questions. Disentangling the effects of generalization and test set memorization is critical to our understanding of language model per- formance, but this becoming increasingly difficult as the pretraining datasets are rarely public for many of the LMs deployed today.
Although there ongoing work by LLM providers to remove benchmarks from pre- pre-training traininii datasets and perform dataset contamination studies, such filtering can fail due to bugs (Brown et al. 2020a). be limited to a select set of benchmarks (Brown al.) 2020aa Wei et al., 2021 Chowdhery et al., 2022). and requires trust in these vendors. Increasing competitive pressures have also led to some recent model releases to include no contamination studies at all (OpenAl 2023). These factors make it critical for us to be able audit existing language models for the presence of benchmark datasets without the cooperation of language model providers.
In parallel to contamination studies, there has been growing literature heuristic membership inference algorithms, that seek to reverse engineer aspects of the pretraining dataset (Carlini et al.
Equal technical contribution, first author was the project lead.

---

Pre-training Data
Contamination Test
Canonical Order
Shuffled order
To. nall compoitd stoets sicants atc hat alse
bogs a frop jump out of beiling voter?
Boes frog Jump out of boilling Nater?
frog Jump boiling
is poosible to create naza energy?
Is possible create wass energy?
Test Set
1t possible to create mise free anergy?
tomatoss?
jequer type
Contantration
Ix there novte with on rotten tonatowe?
Io the jagur 5 type rear wheel drive?
jequer wheel drave?
nigenydl aur oraanse Higleyy ceractigg oisgiallly
high model. 2op-probabilitt
low nodel log bog-probbbllity
Differences in log-probability between orderings reveal contanination.
Figure 1: Given a pre-training dataset contaminated with the BoolQ(Clark et al., 2019) test set (left), detect such contamination by testing for exchangability of the dataset (right). If model has seen a benchmark dataset, it will have a preference for the canonical order (i.e. the order that examples are given in public repositories) over randomly shuffled examples orderings. We for these differences in log probabilities, and aggregate them across the dataset to provide false positive rate guarantees.
2019
Mattern et al. 2023) as well as provide some evidence for test set contamination (Sainz al,
2023 Golchin & Surdeanu 2023). However, the heuristic nature of these methods limits their usefulness, as these methods cannot elevate speculation about a suspected instance test set con- tamination into an irrefutable proof of contamination.
In this work, we show it possible to go beyond heuristics and provide provable guarantees of test set contamination in black box language models. More specifically, we provide a statistical test that can identify the presence of a benchmark in the pre-training dataset a language model with provable false positive rate guarantees and without access to the model's training data or weights.
To achieve these guarantees, we exploit the fact that many datasets have a property known as ex- changeability, where the order of examples in the dataset can be shuffled without affecting its joint distribution. Our key insight is that if a language model shows a preference for any particular or- dering of the dataset such as a canonical ordering that appears in publicly available repositories this violates exchangeability and can only occur by observing the dataset during training (Figure We leverage this insight to propose a set of tests that compares the language model's log probabil- ity on the 'canonical' ordering (taken from public repositories) to the log probability on a dataset with shuffled examples, and flag a dataset the two log probabilities have statistically significant differences.
Using these ideas, we propose computationally efficient and statistically powerful test for con- tamination which shards the dataset into smaller segments and performs a series of log probability comparisons within each shard. We prove that this sharded test provides control over the false pos- itive rate, enables computationally efficient parallel tests, and substantially improves the power of the test for small p-values.
We evaluate our statistical test on 1.4 billion parameter language model trained combination of Wikipedia and a curated set of canary test sets. Our test is sensitive enough to identify test sets with few 1000 examples, and sometimes even appearing only twice in the pretraining corpus. In the case of higher duplication counts, such as datasets appearing 10 or more times, we obtain vanishingly small p-values our test. Finally, we run test on four commonly used, public language models to study the behavior of our test on language models in the wild and find little evidence of pervasive and strong test set contamination.
We summarize our contributions below.
Demonstrating the use of exchangability as a way to provably identify test set contamina- tion using only log probability queries.
Construction of an efficient and powerful sharded hypothesis test for test set contamination.
Empirical demonstration of black-box detection of contamination for small datasets that appear few times during pretraining.
2

---

Our three contributions suggest that black-box identification of test set contamination is practical and further improvements in the power of the tests may allow us to regularly audit language models in the wild for test set contamination. To encourage the development of new provable guarantees for test set contamination, we release our pretrained models as a benchmark for developing future statistical tests
PROBLEM SETTING
Our high-level goal is to identify whether the training process a language model 0 included dataset X. In our setting, the only method we have to study is through log probability query log po (s) for a sequence s (i.e. no access to dataset or parameters). This setting mirrors many common situations with API-based model providers (Brown al. 2020b Bai et al. 2022) and matches increasing trend where training data kept secret for "open" models (Touvron al. 2023 Li et al. 2019).
Provably identifying test set contamination can be viewed as a hypothesis test in which the goal is to distinguish between two hypotheses:
Ho: is independent of X
HH is dependent X
where we treat ! as random variable whose randomness arises from combination of the draw of the pretraining dataset (potentially including X) and we will propose a hypothesis test with the property that it falsely rejects the null hypothesis He with probability most a.
False positives under Ho In most cases, we can make use of a property of a dataset known as exchangeability to obtain our false positive guarantee. Nearly all datasets can be expressed collection of examples X == {. where the ordering of the examples are unimportant, and the probability of any ordering would be equally likely (i.e. p(z1 ... .nn)) = p(#T, ... VT,) for any permutation v). Notably, this assumption would hold under the standard assumption that the dataset a collection of i.i.d examples.
Whenever exchangability of the dataset holds, the log probabilities of the model under Ho must have a useful invariance property,
Proposition Let xXq(X) ) be a function that takes a dataset X and concatenates the examples to produce sequence, and let X- be a random permutation of the examples of X where a is drawn uniformly from the permutation group. For an exchangeable dataset X and under Ho.
log Pe (X)) d log pa (seq(())))
Proof This follows directly from the definitions of exchangability and Ho. Since X is ex- changable, xeq(X) d seq(X,) and by the independence of 0 from X under Ho. we know that (X))))()) (0, seq(Xx)). Thus, the pushforward under log pe (x)qq())) must have the same invariance property.
Proposition is the basic building block of our tests. It implies that the log probabilities of X under Ho have the same distribution when shuffled, and this permutation invariance will enable us to directly apply standard results on constructing permutation tests (Lehmann & Romano, 2005).
Detection rate under Hi The false positive rate guarantee holds with extremely weak assump- tions, but useful test should also have high power, meaning that it should have a high detection rate under H₁. We cannot hope for high detection rate without further assumptions. For instance, an adversary may hide an encrypted copy of X within the parameters of the model (which would induce clear dependence between the model and X) but would be nearly impossible for to detect such situation even with weight access.
httpp:: htttp:::/ggtthhbb.cm/aatttt he

---

However, most existing forms of contamination are benign. In the benign contamination setting consider, pretraining datasets become contaminated when test sets accidentally slip through filtering mechanisms Brown et al. (2020a). In this case, we have reasonable expectation that the invariance in proposition 1 will be violated and log po (seq(X)) > log po (seq xx)) ) as the language model is explicitly trained maximize the log-likelihood over its training data, including (X). The vio- lation of exchangability allows us to reliably detect test set contamination, and the existing literature on memorization (Carlini et al. 2021) suggests that many models may verbatim memorize the order of examples in benchmark dataset. We now focus on building tests that can reliably identify this form of memorization.
METHODS
The core idea of our statistical test is to compare the log probability of the dataset under its original ordering the log probability under random permutations. We begin by describing the basic version of this idea, which directly implements a permutation test on the log probabilities. We then identify some drawbacks of this approach and describe sharded test which improves the statistical power and computational efficiency of the test.
3.1 A PERMUTATION TEST FOR CONTAMINATION
Under the null hypothesis, the likelihood under the model of any permutation of the dataset X, has the same distribution, and thus the rank of log pa(sqq())) among any set of randomly permuted probabilities 4lgg pa (seq X )) log pa (seq( (x, })} will be a uniform random variable over [n +
1] (Lehmann & Romano 2005
Theorem 15.2.2).
This can be used directly to construct a permutation test. Consider the proportion of permuted copies of X with lower log- likeihood than the canonical ordering under the model.
p E[1 (log po (seq(X)) Doggaassn (seq((x))))
The distribution of will be uniform under Ho. and test for contamination at significance level a by rejecting Ho when p < a. In practice, computing this expectation over all x is intractable, and replace this with Monte Carlo estimate and the appropriate finite-sample correction (Phip- son & Smyth 2010), which gives
1 (log (lee)( < log po (seq( xx,)))))) })} +1
+ 1
This test is simple and straightforward implement, and the validity of this test when rejecting
0 < a is clear from standard results on permutation testing (Lehmann & Romano 2005 Phipson & Smyth 2010). However, this test suffers from a major drawback in its Monte Carlo implementation the runtime of the test in terms of the number of log probability computations is 0m for sequence of length XX| and the p-value can never be below 1/(m 1). For hypothesis tests that aim to reject at very low p-values (or with substantial multiple hypothesis testing corrections), this poses a tradeoff between statistical power and computational requirements.
3.2 A SHARDED LIKELIHOOD COMPARISON TEST FOR CONTAMINATION
What are some drawbacks of the naive permutation test? It has an undesirable tradeoff between statistical power and computational requirements for small and also requires that the model assign higher likelihood to the canonical ordering than nearly all shuffled orderings of X,. This latter condition can also be serious problem, as the model may have biases the prefer certain orderings (e.g. ones that place duplicate examples next to each other) regardless the order seen during training.
A more likely assumption would be that the log probbbbbyyyy under the canonical ordering X is higher than the average log probability under random permutation. That is, instead of relying the quantile E[1 (log Po (X)) < log Po (seq((X))) can we instead perform multiple log probability comparisons of the form log Po (o(eq((X)) < BEllog po (seq(Xx)))

---

We show that this is possible and the resulting test resembles series of log probability comparisons followed by a t-test to aggregate these results. More specifically, we will partition the examples X,, Xn into r contiguous shards S S, formed by grouping together adjacent examples
S = (X,, X2, ....... ,
where each shard S; contains at least k = n//r examples.
Then, will permute the examples within each shard and compare the likelihood of the canonical ordering to a Monte Carlo estimate of the average likelihood of the shuffled ordering as
S = log po (seq(X) Mean, (log pa (seq(Xx))).
Finally, to construct the test, we aggregate these shard statistics 8; via the mean test for whether is zero-mean using t-test.
E 8 and
This statistical test, whose pseudocode given in Algorithm|| [ addresses the shortcoming of the permutation test by converting single rank comparison into collection of log probability com- parisons. The t-test based approach also requires O(m|X]] runtime for m permutations, but there no 1/m minimum p-value, and in practice find that the p-values obtained by this approach decay rapidly, as it only requires that the language models consistently assign higher-thale sigher-than-ae terr--tha---eeeee -verrge log probabilities to the canonical ordering, rather than requiring that the canonical log probability be the tails of the permutation null distribution.
Algorithm Sharded Rank Comparison Test
Require: Test set examples 2i....,.
Require: Target model 0
Require: Number of shards r
Require: Number of permutations per shard m
1:
Partition the examples into shards S1, S2, Sr, where each shard has at least [n/r] exam- ples, and one extra example is added to the first mod r shards.
2: for each shard S; do
3: Compute the log- likelihood of the canonical order:
canonical == log log (6) (6) , (1)
(1) 4: Estimate shuffled Mean, [log po (seq (11)))) (1))"" " *(k) by computing the sample average
(i)
(1)
over m random permutations æ.
5: Compute 8;
((5)
((0)
caannnical
shutted
6: end for
7: Define 8 8= = - 8; the sample average over the shards.
8: Run one-sided t-test for E(s:] O, returning the associated p-value of the test p.
Under the null, we expect be the of independent random variables and now show that the overall test provides false positive rate guarantee.
Theorem 2. Under the null hypothesis, an i.i.d dataset X, and finite second moments on loga(S))
PP < a) -l
and is defined the p-value in Algorithm 7
Proof The result follows directly from the combination of Theorem [ and standard invari- ance results in (Lehmann Romano, 2005). First, by Theorem [ note that the distribution of log (o(s( (i) ((1) (k)) is invariant to the permutation
[s)
By Lehmann & Romano 2005 Theorem 15.2.2), this guarantees that the permutation distribution uniform over the support, and the statistic 8; must be zero-mean. Next, we note that each shard is independent, each example is split independently into separate shard with overlap. By independence and the finite second moment condition, s + N(0 0²/ /m) under the null by the central limit theorem and sided t-test provides asymptotically valid p-values with
5

---

P(p < aa) + uniformly oc (Lehmann & Romano 2005 Theorem 11.4.5).
This result ensures that the sharded rank comparison test also provides (asymptotic) guarantees on false positive rates, much like the permutation test. The test we propose here has two small dif- ferences relative to the permutation test it provides asymptotic, rather than finite-sample valid p-values and assumes i.ddd X for the proof. These conditions could be relaxed by the of Berry- Esseen bounds to obtain finite-sample convergence rates for the CLT as well as replacing our use of a standard central limit theorem with applicable the sums of exchangable random vari- ables. However, we opted to present the simpler asymptotic test given the frequent use of i.idd data generation assumption in the literature as well as the fast convergence of the CLT in practice.
4 EXPERIMENTS
We now demonstrate that our test is effective for detecting many common forms of test set contam- ination. We begin by training a 1.4 billion parameter language model, consisting both Wikipedia and known collection of exchangeable test sets, These canaries serve as positive controls for test, and our goal will be to flag as many of these as possible. Having validated the test a setting with known contamination, then explore its use with existing open models.
4.1 PRETRAINING WITH INTENTIONAL CONTAMINATION
Datasets and training To validate our test statistic, we train a 1.4 billion parameter GPT-2 model from scratch with combination of standard pretraining data (Wikitext, taken from the RedPajama corpus TTgether Computer (2023) and known test sets, We derive 10 test sets from numerous standard datasets (BoolQ (Clark al. 2019). HellaSwag (Zellers al. 2019), Openbook (Mi-) haylov et al., 2018b), MNLI Williams et al. 2018). Natural Questions (Kwiatkowski et al. 201 20)9a), TruthfulQA (Lin et al., 2022). PIQA (Bisk et 2019). MMLU (Hendrycks al. (001)), and sub- sample the datasets to around 1000 examples to ensure that the test sets remain small part of the overall pretraining dataset (See Table for exact sizes). While we do not know these datasets are exchangable when they were constructed, we can make them exchangable simply by applying random shuffle to the dataset, which would make all orderings of the examples equally likely.
To test our ability to detect benchmarks at various duplication rates, duplicate each of the datasets a different number of times ranging from I to 100 (See Table D.. The overall pretraining dataset has 20.2B tokens, with 20M tokens associated with some benchmark dataset.
Test parameters The sharded rank comparison test requires two additional parameters: the shard count and the permutation count Thoughout these experiments we use 50 shards and r = 51 permutations. In our ablations below, we found that the tests are not particularly sensitive to these parameters, and we fix these parameters avoid the possibility of p-hacking.
Canary Results In Table I we find that our test is highly sensitive, and provides near-zero p- values duplication rates of 10 above. These detections hold for relatively small datasets (< 1000 examples) and for modestly sized language model with 1.4 billion parameters. Given that many test sets are much larger in practice, and many language models of interest are much larger and memorize more aggressively (Carlini et al. 2019). these findings suggest that our test is likely to detect contamination practice.
While the permutation test attains significance (at a typical a = 0.05, say) for all benchmarks duplicated at least 10 times, the p-values are bounded below by 1/(1 + r), where the number of permutations used here is 100. Results for sharded test compute, the sharded test attains comparable performance for benchmarks with small duplication rate. However, the p-values attained by the sharded test for moderate to high duplication rates vanishingly small.
50; even with half the
Attaining comparably low p-values using the permutation test is computationally infeasible. For example, to allow for the possibility of a p-value as low as 1.96e-11 (matching the MNLI result) would require permuting the dataset 10 times, and as many forward passes of the model.

---

Table 1: We report the results of training 1.4B language model from scratch on Wikitext with intentional contamination. For each injected dataset, we report the number of examples used (size), how often the test set was injected into the pre-training data (dup count), and the p-value from the permutation test and sharded likelihood comparison test. The bolded p-values are below 0.05 and demonstrate the case of higher duplication counts, such as datasets appearing 10 or more times, we obtain vanishingly small p-values our test. Finally, rows marked le-38 were returned as numerically zero due to the precision of our floating point computation.
Name
Size
Dup Count
Permutation p Sharded
BoolQ
1000
1
0.099
0.156
HellaSwag
1000
1
0.485
0.478
OpenbookQA
500
1
0.544
0.462
MNLI
1000
10
0.009
1.96e-11
TruthfulQA
1000
10
0.009
3.43e-13
Natural Questions
1000
10
0.009
1e-38
PIQA
1000
50
0.009
1e-38
MMLU Pro. Psychology
611
50
0.009
1e-38
MMLU Pro. Law
1533
50
0.009
1e-38
MMLU H.S. Psychology
544
100
0.009
1e-38
Although our test is unable to detect contamination at duplication rate of 1, other existing literature on memorization has suggested that detection this duplication level is extremely difficult. Prior work has found that existing tests of memorization begin to work with 10-30 duplicates (Carlini al. 202T). that deduplicated text is hard extract (Kandpal al. 2022), and that dataset con- lamination with duplication rate of barely affects downstream benchmark performance (Magar Schwartz. 2022).
Power as a function of duplication rate. We carefully study the lowest duplication rate for which our test can reliably detect contamination. To do this, we perform the above canary study but with duplication rates ranging from to 7, and we show the aggregate log p-values for each duplication rate in Figure 2 We find that we cannot reliably detect duplication rates of 1, but that at counts of 2 and 4 we begin to detect some test sets (gray points below the dashed line) and that the detection threshold around duplication rate of 4. This suggests that even small amounts of dataset du- plication would be sufficient for detection, and future improvements to the power of this test could enable reliable detection at much lower duplication rates.
Log(p value) vs Dataset Duplication Count
-5
laglp value)
dataset's log(p value)
-15-
mean log(p value] dup count
-0
log(p=0.05)
7
Dataset Duplication Count
Figure 2: For a model pre-trained with canary datasets injected at a duplication count of 1, 2. 4, and we plot the log p-value against dataset duplication count to quantify how the test's power depends on dataset duplication count.
public benchmark of provable test set contamination Our work demonstrates that exploiting exchangability allows us to provably detect test set contamination even for small, low-duplication count datasets. However, it is an open question whether there are tests that can reliably detect contamination at duplication rate of 1. To support future work this open problem, release

---

pre-traae pre-trained models trained Wikitext mixtures together with the corresponding canary test set
In addition to the release, will maintain leaderboard of methods that provide (asymptotically valid) p-values, ranking methods by the average log p-value. We hope that the model and bench- mark spurs further development of tests for contamination, and encourage members of the research community to improve on our results for low duplication counts.
4.2 SHARDING AND PERMUTATION COUNT
Our test relies on two parameters the number of shards in the test, and the number of permutations sample. Both of these affect the power of the test, and we carefully study the impact of these parameters on our ability to detect test sets by evaluating our pre- pre-trdined model on the 6 datasets that contain 1000 examples (BoolQ, HellaSwag, MNLI, NaturalQuestions, TruthfulQA, PIQA). For the number of shards, we explore range of settings, from 10 shards to 200 shards and for permutations we test a range from to 50 permutations.
Shard sensitivity Our results in Figure Bajshon show that there is sweet spot the number of shards, around 10-20 shards, where our detection rate for test sets are maximized. Larger numbers of shards perform worse, since each shard involves fewer examples. Shards below 10 do not perform well, as this is likely too few samples to merit the use of an asymptotically valid test like the t-test.
Average log p-value vs. Examples per Shard
Average log p-value Permutations Shard
Average log p-value
-2.5
-4
-5
Average
-3.5
-6
100
-4.0
Examples per Shard
500
200
100
50
20
10
10
Number Shards
Permutations per Shard
(a) So long each shard contains enough exam- ples and enough shards used, the p-value is stable under variations
(b) Increasing the permutation count improves the estimate of the mean log-likelihood of the shard under permutation,
number of shards r. We plot the average log p-value of those six of our pre-trained model benchmarks with 1,000 exam- ples, varying the number of examples per shard.
we find that the p-value stabilizes at around 25 shuffles. We plot the aver- age logarithm of the p-value(s) of 6 datasets eval- uated on our pretrained model as function of per- mutations per shard.
Figure 3: Impact of varying shard and permutation counts on test performance.
Permutation count sensitivity We also measure the dependence of our test on the number of permutations per shard in Figure 3b and find more permutations to generally improve the power of our test. We test permutations of 2, 10, 25, 50 and compute the average log p-value of the datasets evaluated on the pretrained model. In practice we find that there is substantial diminishing returns beyond 25 permutations in the t-test. This stands in stark contrast to the permutation test, where a permutation count of 25 would only allow for minimum p-value of 0.038.
4.3 EVALUATING EXISTING MODELS FOR DATASET CONTAMINATION
We demonstrate the utility of procedure in validating test set contamination in multiple publicly available language models: LLaMA2 Touvroo et al. (202))) Mistral-7B (Mistral 2023 ), Pythia-1.4B (Biderman et al. (2023)). and GPT-2 XI. (Radford et al. (2018)) eight public test benchmarks: A12-Are (Clark et al. (2018)., BoolQ (Clark et al. (2019)), GSM8K (Cobbe et al.)
8

---

LAMBADA (Paperno et al. (016)), NaturalQA (wiitkowskii al. (2019b)). Open- BookQA Mihayylv et al. (21 PIQA (Bisk et al. (019)). and MMLU (Hendrycks et al. (2021)) ).
2021
Computationally, find that our test runs reasonably quickly for a billion parameter model, al- lowing for the testing of 65 files for contamination in under 24 hours using 50 shards with 250 permutations per shard, and find that the test outcomes are in general agreement with the lamination study results of Brown et al. (2020c) and Touvron et al. (2023): we do not find evidence of pervasive verbatim test set contamination across the models and benchmarks we tested.
Table 2: P-values for contamination tests open models and benchmarks. With the exception of
ARC for Mistral, none of the tests give evidence for contamination. The MMLU results are marked with f indicate that the p-values are the result of p-value aggregating the constituent datasets of MMLU after heuristic filtering for non-exciiee non-exchangable datasets (see main text). The resulting LLaMA2
and Mistral p- p-values are small, consistent with the contamination studies in Touvron al. (023)) identifying mild MMLU contamination.
Dataset
Size
LLaMA2-7B
Mistral-7B
Pythia-
GPT-2
BioMedLM
Arc-Easy
2376
0.318
0.001
0.686
0.929
0.795
BoolQ
3270
0.421
0.543
0.861
0.903
0.946
GSM8K
1319
0.594
0,507
0.619
0.770
0.975
LAMBADA
5000
0.284
0.944
0.969
0.084
0.427
NaturalQA
1769
0.912
0,700
0,948
0.463
0.595
Qpen
500
0.513
0.638
0.364
0.902
0,236
PIQA
3084
0,877
0,966
0,956
0.959
0.619
MMLU
0.014
0.011
0.362
We tested five models for contamination by eight publicly available benchmarks and list the results in Table We use 50 shards and 250 permutations per shard throughout the experiments. For sets containing more than 5,000 examples, we truncate and test only the first 5,000. Benchmark datasets the wild may contain nonexxchaggable elements such as sequential indexes or duplicate examples which would break the false positive guarantees of our test. To check for these possibilities we manually inspected each dataset for non- exchangaeeee elements, and also run our tests on a 'negative control' of BioMedLMM (Bolton et al., 20022), language model trained exclusively on PubMed data and known not to contain the benchmarks used here. The p-values computed for BioMedl M are not significant across the benchmarks shown here, suggesting that any significant results for the other models tested are not simply due to non--cxhannccabiityy.
Our results in Table show non-significant results across most models and datasets. While failure to reject the null hypothesis not direct evidence in favor of the null, our synthetic canary evaluations from section 4.1 suggest that it is unlikely for these models have seen most of these test sets more than to 10 times. One notable exception is A12-ARC on Mistral, which does return a low p-value of 0.001 and could suggest some contamination. While this p-value appears small, caution the reader that applying a multiple hypothesis test corection would imply that 0.001 is right the boundary of statistical significance, and due challenges with oorden-of-forking--e tattiinng-tthsss type analysis, significance tests that are at the boundary of the rejection cutoff' should be interpreted cautiously. We present these results as showing promising first steps towards parrt audits of test set contamination, rather than direct proof of contamination of specific models.
We additionally discuss contamination tests on MMLU, which was identified as potential con- taminant in recent studies (Touvron et al. (2023)), and involves important details. MMLU is not a single test set, but rather collection test sets. We speculate that least of the test sets non-exchang,,ble, and applying test directly would break the false positive guarantees. To understand if our tests can still provide some insights into contamination, we run our MMLU test with non-exchangability filtering criterion.
To evaluate models for contamination by MMLU, we first exclude those 14 test files from con- sideration for which our test flags either MioMedLM GPT-2 contaminated (both negative controls as the GPT-2 family of models predates MMLU). We run our test on each of the 43 remain- ing test files (with 1000 permutations, due to the small size of each file) and aggregate the p-values

---

Although the omnibus p-values resulting from this proce- dure can no longer provide proof of contamination (due to non-independence and heuristic nature of the fitering step), their magnitude serves as heuristic evidence for contamination. The resulting p-values and empirical CDFs (Figure 4 of the 43 test sets are indicate mild deviation from the null hypothesis, consistent with the findings of mild test set contamination in Touvron et al. (203))
using Fisher's method (Fisher (1934)).
5 RELATED WORK
Our work relates to large literature on data memorization, privacy, and membership inference attacks for large language models. We discuss some of the most relevant works to ours below.
There is substantial literature studying memorization of data in large language models, often from the privacy perspective (Carlini al. 2021 2019 Kandpal et al. 2022] Mattern et al.) 2023 Carlini et al. 2023). Most of these works have focused on analyses of what is memorized and whether pri- vate information can be extracted from a large language model, but do not build tests to specifically identify test set contamination. Our work has a narrower focus on test set contamination, but this also allows us to build tests that provide more precise guarantees of contamination.
Data contamination has been studied in many contexts, including in the study of pretraining corpora (Dodge et al. 202II) as well as in the analysis section of many language model papers (Hoffmann ct al. 2022; Brown al. 2020a; Gao et al. 2020). The n-gram based analyses in these papers can shed light on contamination, but they can have high false positives (e.g. SQuAD (Rajpurkar et al. (016) containing Wikipedia) and are limited to those datasets chosen for analysis. Our approach enables third party tests of dataset contamination with only access to log probabilities, enabling broader testing, without having trust the model provider.
For third-party tests of contamination, there have been a few recently proposed heuristics. Sainz et al. (2023) propose to identify test set contamination in GPT-3 and GPT-4 by prompting the models to generate verbatim examples from a test set. The contemporaneous work of Golchin & Surdeanu (2023) similarly proposes to identify contamination in black-box models by prompting a model to generate completions of random example prefixes and using GPT-4 to judge the closeness between the completion and the ground truth. Min-k% (Shi et al. (2024) indicates contamination when the average of the k% tokens with lowest probablity in the test set is high. We report the results of Min-k%-Prob using the same setup as in the canary experiments of section [see (see the appendix), and find strong and reliable detection for duplication count of 10 and above, and detection at duplication counts as low as 4. We note that while these approaches are efficient and do not require access to pre-training data, they do not enjoy the same provable false-positive guarantees of work.
Closest to work is the exposure statistic in Carlini et al. (019 and other subsequent variations (Mattern et al. (2023)), which tests the perplexity differences between a target sequence and random sequences. The idea of comparing the rank of the target log probability to some baseline distribution is similar to our work. However, our work is distinct in using the exchangability of datasets to obtain an exact null distribution (giving us provable guarantees when identifying contamination) and in developing a sensitive and efficient shard-based test.
Beyond language modeling, identifying the presence of a particular set of examples in the training data of machine learning model is related to the security and privacy topic of membership inference (Shokri et al.) (2017); Mattern et al.) (2023). Our work contributes to this literature by developing a new form of membership inference attack that leverages the exchangability of benchmark datasets.
LIMITATIONS
We highlight a few limitations of our approach for detecting test set contamination. First, the p- values presented in this paper do not have multiple test corrections applied, as it is difficult define the 'total number of hypotheses' tested throughout development.
Second, any application of this test in practice will likely involve taking an off-the-shelf benchmark dataset X, for which it will be difficult to know if the dataset is truly exchangable. Heuristic negative controls such our MioMedI M experiments can be helpful, but cannot ever prove that a dataset
10

---

exchangable without knowing its data generating process. We strongly encourage future dataset creators to apply a random shuffle to their datasets (and to publicize this fact), which would allow tests to be applied.
Finally, our tests focus on the case of verbatim contamination where a language model ingests a test set directly. Contamination can happen in many other ways, such when a language model consumes a data source used in the construction of benchmark (e.g. Wikipedia used in SQuAD, professional tests in MMLU). Verbatim memorization of test set is not the only form of contami- nation, and our test cannot rule out the possibility of more complex forms of partial contamination.
7 CONCLUSION
In this work, demonstrated that it possible construct statistical test for test set contam- ination that provides false positive rate guarantees and requires nothing other than the ability to compute log probabilities. We construct new, sharding based tests for contamination and demon- strate their power on both carefully constructed canaries as well as publically available language models. We view these tests as first step towards building powerful third party tests of contamina- tion, and we believe it is an exciting open problem to build tests that are capable of reliably detecting contamination at the single-duplication-con
regime.
8 ACKNOWLEDGEMENTS
We gratefully acknowledge support from the CRFM Levanter team, especially David Hall, for both computational resources and infrastructure support, and Google's TPU Research Cloud (TRC) for Cloud TPUs used in the pretraining experiments. Nicole Meister was supported by NSF GRIP DGB-2146755. Niladri Chatterji and Faisal Ladhak were supported by SAIL and Tatsumori Hashimoto was supported by gift from IBM and a Hoffman- Yee grant. Finally, we would like to thank Nicholas Carlini and Percy Liang for insightful discussions on memorization and test set contamination.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Seffield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training helpful and harmless assistant with reinforcement learning from human feedback. 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- Jahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. ArXiv, abs/1911.11641, 2019. URL https:
//api. wtw.psssmmattcccccoororoooogo wtwp:chhoolar..oo/oooupoo ID: : 208290933
Elliot Bolton, David Hall, Michihiro Yasunaga, Tony Lee, Chris Manning, and Percy Liang. BiomedIm, 2022. URL https::/crrfi stanford....rroe edu http://2222/1/1///immeeeoe html
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam,
G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Language models few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020a.

---

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models few-shot learners. arXiv preprint arXiv:2005. 14165, 2020b.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14166, 2020c.
N. Carlini, C. Liu, Ú. Erlingsson, Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Conference on Security Symposium, SEC' 19, pp. 267-284, USA, 2019. USENIX Association. ISBN 978 9781939133069
N. Carlini, F. Tramèr, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. X. Song, U. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In USENIX Security Symposium, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee. Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.00 076666, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton. Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Samkaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov. Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren- nan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellssern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. arXiv preprint Xrxi::2004.002 02034.02111,.1 2022.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Pupers), pp. 2924-2936, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18655//1/N119-13000 URL aattss://aclantholo.s 0r1N19--110///1100000000
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. ArXiv, abs/1803.05457 2018. URL httpt::////ppp. /api. wwwanticscholes........e. org/
Co12us1D:3922816
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Jesse Dodge, Maarten Sap. Ana Marasovié, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference Empirical Methods
12

---

in Natural Language Processing, pp. 1305, Online and Punta Cana. Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.1865$w112021.cmmmlpp main. 98. URL httlhttts///aaaaattoooooooo http://g...r/////0 www.:emee1...mmmm.mmmmm
R. A. Fisher. Statistical Methods for Research Workers. Oliver & Boyd, Edinburgh, 4th edition, 1934.
I. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: An 800gb dataset of diverse text for lan- guage modeling. arXiv preprint arXiv:2101.0002
2020.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. arXiv preprint arXiv2308.088493, 2023.
David Hall, Ivan Zhou, and Percy Liang. Levanter - legible, scalable, reproducible foun- dation models with jax, 2023.
Levanter 1 rewease.ee...... html
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2021.
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, O. Vinyals, J. W. Rae, and L.. Sifre. An empirical analysis of compute-optimal large language model training. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates pri- vacy risks in language models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Con- ference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 10697-10707. PMLR, 17-23 Jul 2022. URL http::ttt///t cittps://proceedi:: ngs m prrss// 1162/kandpal22a.co
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466, 2019a. doi: 1162//aclaa00022101 URL wwwp:/y-yoo/0000010000000
Tom Kwiatkowski, Jennimaiia Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Al- berti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Was Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: a benchmark for question answering research. Transactions of the Association of Computational Linguistics, 2019b.
E. L. Lehmann and Joseph P. Romano. Testing statistical hypotheses. Springer Texts in Statistics. Springer, New York, third edition, 2005. ISBN 0-387-98864-5.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lec. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309 05463, 2019.
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3214-3252, Dublin, Ireland, May 2022. Association for Computational Linguis- URL https:://eelaatthooooyyooo//////
tics. doi: 10. 1111886///200222cc-looooo acl 1el--ong.co 229
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Pro- ceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), 2022.
13

---

Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Barg-Kirpaatrick. Membership inference attacks against language models via neigh- bourhood comparison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 13300-1133,, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/2033. findings-acl.7199
http://cceeanoooooorggo//2//2222 findings acc.nngsssa......cca 719
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. Conference on Empirical Methods in Natural Language Processing, 2018a. URL https: pettss:/api... aesmantisscoon
org/
SorpusID:52183755:
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun't ichi Tsujii (eds.). Proceedings of the 2018 Conference on Empir- ical Methods in Natural Language Processing, pp. 2381-2391, Brussels, Belgium, October- November 2018b. Association for Computational Linguistics. doi: 10. 10.18655/v1/D11-1226) URL
Mistral. Mistral 7b, 2023.
URL
https://mist.e wasttrall.inaass.........
antouncing--istere 7b/
OpenAl. Gpt-4 technical report, 2023.
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. arXiv preprint
2016.
Belinda Phipson and Gordon K Smyth. Permutation p-values should never be zero: Calculating exact p-values when permutations are randomly drawn. Statistical Applications in Genetics and Molecular Biology, 9(),, 2010. doi: doi:1002202/154461115
https: .rrg/
10.2202/1544-6115..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya
Sutskever. Language models are unsupervised multitask learners, 2018. URL
language-moddlll Jet.eees...eereeee.......
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras (eds.), Pro- ceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp- 2383-2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D166 1264.
Oscar Sainz, Jon Ander Campos, Iker Garefa-Ferrero, Julen Etxaniz, and Encko Agirre. Did chatgpt cheat your test?, 2023. URL https://hltz-zentteeee
1o/
Inn-contaminationn/ooomo
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlmoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 16789, 2024.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 20/7 IEEE Symposium on Security and Privacy (SP), 2017.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023. URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom,
14

---

Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrXiv2337.009266 2023.
A. Wang. A. Singh, J. Michael, Hill, O. Levy, and S. R. Bowman. GLUE: multi-task bench- mark and analysis platform for natural language understanding. In International Conference Learning Representations (ICLR), 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shol learners. arXiv preprint arXiv:2109.01652, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long Papers), pp. 1112-1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 1/1/5///1/11-11011.1
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Associ- ation for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19 1472. URL https: www. aclweb. org/ anthology/P19-1472
15

---

APPENDICES
STRIDED LOG-LIKELJHOODS
To compute log- likeliiiisso for sequences exceeding the context length, we use strided window approach, with a stride equal to half of the model's context length. We find that decreasing the stride beyond half the context length does not yield significant gains.
PRETRAINING DETAILS
We elaborate on the hyperparameters and training procedure of our 1.4B language model, trained from scratch on intentionally contaminated Wikitext.
We use GPT-2 architecture with 1.4B parameters, with the architecture hyperparameters given by a hidden dimension of 1536, 24 heads, 48 layers, and a sequence length of 2048. The training batch size was 256. Based on the number of training tokens, sequence length, and training batch size, trained this model for 46000 steps so as to consume the tokens in our mixture datasets exactly once. The model was optimized using AdamW with a learning rate of le-4 and weight decay of 0.1. We trained the model using Levanter on a v3-128 TPU instance on Google Cloud for 1.5 days (Hall al. (003))).
10 CANARY DATASETS
In this section we provide additional details on the 10 canary datasets we injected into Wikitext form our pretraining data. For Bool (Clark et al. 2019). HellaSwag (Zellers et al. 2019).
( MNL Willlams et al. 20188] 8). Natural Question (wiitkowskii et al. 2019a). TruthfulQA Lin ct al. 2022). PIQA (Bisk et al. 2019), we sample random subset of 1000 examples. For Open- bookQA (a (Mihaylov al. (018b). because of its smaller test set of size n=500, used all 500 examples. Finally, for MML Tf (Hendrycks et al. (021). we chose from subsets with no multi-ling examples and having least 500 examples, specifically Professional Psychology (n=611) 1), MMLU Professional Law (n=1000), MMLU High School Psychology (n=544). Finally, we shuffle the ex- amples in all datasets make them exchangeable. In Table we provide additional information about the injected datasets including number of examples, average words per example, and number of tokens per dataset. For each duplication rate, included short, medium and longer dataset, as measured by the total token count. The total token count of injected benchmarks is 19.235M tokens, meaning that the injected dataset less than 1% of the entire pre- pre-training dataset.
FULL MMLU RESULTS
We list in table the full set of MMLU results used to generate the omnibus p-values for contami- nation by MMLU listed in table 2 before filtering out for suspected non-exchangeability.
COMPARISON TO MIN-K% PROB BASELINE
We provide comparison against Min-k% Prob (Shi et (2024)), state of the heuristic method for contamination detection proposed contemporancous to our work. We use the same pretrained model and test sets from our experiments Section 4.1
Our implementation of M follows the methodology outlined in Shi et al. (2024); we run the method on one hundred 512-token spans sampled from each benchmark, and tune the decision
isonl
httpss//yonattniiss..co//bb/
16

---

Table 3: Injected canary datasets and duplication counts used our pretraining experiments.
Name
Examples
Avg Words/Ex
Tokens
Dup Rate (High)
Dup Rate (Low)
BoolQ
1000
110
110k
HellaSwag
1000
185
185k
1
OpenbookQA
500
40
20k
2
Natural Questions
1000
32
32k
10
2
MNLI
1000
235
235k
4
TruthfulQA
1000
25
25k
10
4
PIQA
1000
50k
50
7
MMLU Pro. Law
1000
2000
200k
50
7
MMLU Pro. Psych
611
50
30k
50
MMLU H.S. Psych
544
37
20k
100
1L
Mistrai-8
Pytha-1.4B
1.0
100F
1.0
10-
LG5F
0.6-
0.0-
0.
0.8
0.6
100
2.0
0.2
0.4
0.6
0.6
1.5
0.0
2.8 0.0
10
Figure 4: Empirical CDFs of MMLU p-values of LLaMA2, Mistral, and Pythia after exclusion of BioMeddLM and GPT-2 significant test files, plotted against CDFs of a Uniform(O,1).
threshold on validation of five of our contaminated sets, and five test sets used in our data mixture. The threshold is tuned for false positive rate of 5% to allow for a meaningful comparison against our test, value of k=20 used as recommended in the paper.
17

---

Table Full MMLU results for LLaMA2-7B, Mistral-BB, Pythia-I.4B, GPT-2XL and BioMedLM.
Dataset
Size
LLaMA2-7B
Mistral-7B
Pythia-
GPT-2 XI.
BioMedLMM
Abstract-Algebra
100
0.103
0.246
0.645
0.894
0,861
Anatomy
135
0.586
0,605
0.605
0,439
0.649
Astronomy
152
0.550
0.657
0.050
0.003
0,050
Business-Ethice
100
0.936
0,440
0.107
0.808
0,499
0.199
0.108
0.081
0.004
0.268
Dollege-Biology
144
0.108
0.051
0.152
0.260
0.779
100
0.526
0.614
0.595
0.355
0,265
College-Conid
100
0.060
0.498
0.383
0.532
0.267
100
0.059
0.151
0.162
0.321
0.727
College-Medichee Medicine
173
0.397
0.106
0.340
0.440
0.067
Dollege-Physice
102
0.694
0.757
0,972
0.719
0.262
Computer-Security
100
0.214
0.007
0.314
0.180
0,928
Conceptual-Physiess
235
0.554
0.333
0.811
0.710
0.924
Econometrics
114
0.616
0.761
0.540
0.508
0,035
Electrical
Enntrical-Engieeering
145
0.266
0.364
0.595
0,490
0.277
Flementtary-aathhmaaiss
378
0.059
0.416
0.260
0.355
0.528
Formal-Loggc
126
0.666
0.750
0.990
0.930
0.398
Global-Facts
100
0.779
0.957
0.448
0.339
0.667
High-School-Biue
310
0.645
0.279
0.476
0,499
0,416
Sigh-SScholl-hheiisee
203
0.229
0,426
0.813
0.539
0.055
100
0.023
0.085
0.106
0.045
0.484
165
0.009
1e-38
1e-38
1e-38
le-38
High-School-
198
0.194
0.339
0.341
0.721
0.277
High-SSchool-
And-Politics
193
0.294
0.066
0.025
0.372
0.003
High-Schhol-Maccrccoooooo
390
0.543
0.222
0.276
0,236
0,511
High--School-Mathemane
270
0.473
0.182
0,001
0,272
0.032
High-Schhool-Micreecooo
238
0.862
0.797
0.712
0.122
0.181
High-School--Pyysee
151
0.339
0.757
0.171
0.114
0.354
Migh-SScholl-Psychlloe
545
0.033
0,037
0.004
0.039
0.007
Saat-School-Stttistie
216
0.077
0.119
0.228
0.233
0.211
High-School-USs -us-hoooot----stoyyyyy
204
1e-38
le-38
le-38
1e-38
0.001
High-School- World-
World-History History
237
1e-38
le-38
le-38
1e-38
1e-38
Human-Aging
223
0.882
0,830
0.617
0.677
0,803
Human- -SxaalSy
131
0.807
0.959
0.502
0.789
0.471
International-L.n
121
0.061
0.545
0.532
0.582
0.817
Jurisprudence
108
0.009
0.075
0.852
0.879
0.616
Logical-Fallacies
163
0.103
0.188
0,082
0.397
0.193
Machine-Leammn Learning
112
0.266
0.297
0.373
0.061
0.868
Management
103
0.516
0.209
0.454
0.907
0.281
Marketing
234
0.874
0.684
0.977
0.848
0,451
Redical-Genetion
100
0.501
0.037
0.523
0.425
0.729
Miscellaneous
783
0.099
0.122
0.086
0.266
0.081
Moral-Disputes
346
0.017
0,011
0.097
0.190
0.157
Moral-Scenarios
895
0.652
0.022
0.121
0.487
0.413
Nutrition
306
0.163
0.601
0.413
0.865
0.609
Philosophy
311
0.011
0.094
0.003
0.049
0.044
Prehistory
324
0.203
0.412
0.055
0.268
0.262
Professional-Accountingg
282
0.708
0.242
0.197
0,910
0.798
Professional-Medicinee
272
0.001
0.017
0.001
0.002
le-38
poffessional-Psycchole
612
0.001
0,001
0.058
0,009
0,201
110
0.402
0.237
0.162
0.512
0.622
Security-Studier
245
0.070
0.043
0.687
0.061
0.073
Sociology
201
0.350
0.258
0,260
0.660
0.496
Us- Us-Foreign-Policy
100
0.934
0.252
0.778
0.646
0.173
Virology
166
0.203
0.854
0.219
0.796
0.212
World-Religions
171
0.311
0.882
0.933
0.222
0.851
18

---

Table 5: Canary results for Min-k%-Proo.
Name
Size
Dup Count
Min-k% Prob
BoolQ
1000
13%
HellaSwag
1000
2%
MNLI
1000
10
100%
MMLU Pro. Law
1533
50
100%
MMLU H.S. Psychology
544
100
100%
19


Reviews:
[{"review_id": "5otHPtssNS", "reviewer": "Reviewer_jRLi", "summary": "The paper proposes a statistical test that given certain assumptions can indicate whether a black-box language model has been trained on certain datasets. This is a topic of increasing interest and importance given the prevalence of pretrained models that are trained on very large amounts of data.  The authors first propose a simple permutation test and identify some weaknesses with it. They then propose a more sophisticated sharded test. The authors show 2 kinds of experiments:\n\n(1) They test on a dataset where they have injected a small amount of certain test sets to see if their approach can detect them.\n\n(2) They apply their test to existing models such as Lllama-2 showing their approach can scale.", "questions": "In Figure 1 the authors show test set contamination for BoolQ. But the examples there are unlabeled. Are the authors targeting unlabeled test set contamination i.e. the input is present in the pretraining data but not the label? \n\nWould be great to have some justification and explanation of this setting.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Topic of large importance in the community given the direction of the field. \n\n-Novel approach with thorough empirical results. I have some questions about the definition of test set contamination below.\n\n-Well written and interesting.", "weaknesses": "I have some questions about the definition of test set contamination below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iFEG2apB2b", "reviewer": "Reviewer_Rgfh", "summary": "This paper studies the problem of identifying test set contamination in large language models, i.e., detecting that a test set is present in the pretraining data of a language model. The main idea behind the approach is that for test sets that have some canonical order of individual instances (e.g.: the order in which the dataset creators release the dataset), the likelihood of the test set in that order would be significantly higher than any random permutation of the dataset. Based on this idea, the paper proposes two versions of the test, one of which shards the test set and aggregates statistics over the shards to make the estimate more robust to potential biases in the model.\n\nThe tests are evaluated first by measuring their sensitivity when pretraining datasets are intentionally contaminated. It is shown that they are highly sensitive when the tests sets are large or have been duplicated enough in the pretraining data. The test is then used to measure contamination of the pretraining data used to train the Llama models and it is shown that the findings agree with prior reports.", "questions": "- How does the performance of this method compare to that of prior work (see Weakness)\n- How sensitive is the proposed test to the model size?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "This is clearly written paper and makes a strong contribution. The tests do not require access to model weights or pretraining data, making them practically useful.", "weaknesses": "The experiments do not compare the performance of the proposed tests to prior work. I understand that this work differs from say, the work from Carlini et al. in that this work focuses on set-level contamination, but how does aggregating instance-level statistics over a set compare?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lPcg6UyLqg", "reviewer": "Reviewer_gkwi", "summary": "This paper examines the issue of test set contamination in large language models (LLMs), referring to the phenomenon where LLMs memorize public benchmarks during their pretraining phase. Since the pretraining datasets are rarely available, this paper proposes a statistical test to identify the presence of a benchmark in the pre-training dataset of a language model without accessing the model’s training data or weights. The intuition is the exchangeability of datasets— the order of examples in the dataset can be shuffled without affecting its joint distribution. If a language model shows a preference for any ordering of the dataset, it might have seen the data during pretraining. The test on the LLaMA-2 model identifies potential contamination in the MMLU benchmark, which is consistent with the results in the original LLaMA-2 report.", "questions": "If a dataset is not exchangeable, how effective is the method?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe idea of utilizing dataset exchangeability to identify test set contamination is novel and interesting. \n-\tThe proposed sharded likelihood comparison test addresses the tradeoff between statistical power and computational requirements of the permutation test, which is promising. The sharded rank comparison test also provides (asymptotic) guarantees on false positive rates.\n-\tExperimental results are promising. A GPT-2 model is trained from scratch on standard pretraining data and known test sets to verify the efficiency of the proposed method in identifying test set contamination. The method is also tested with an existing model, LLaMA2, on the MMLU dataset, showing general agreement with the contamination study results.", "weaknesses": "-\tAlthough a more efficient sharded rank comparison test is proposed, the computational complexity is still considerable. For example, testing 49 files using 1000 permutations per shard can take 12 hours for LLaMA2.\n-\tThere is no comparison with other baseline methods.\n-\tThe method relies on a strong assumption of data exchangeability, which may not hold in real-world datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2AU4wlHFOj", "reviewer": "Reviewer_FS83", "summary": "This paper targets the problem of detecting test set contamination of black-box language models. The proposed method is based on two hypotheses: (1) the exchangeability of many datasets (distribution won't be affected after shuffling); and (2) if a language model is contaminated, it is more likely to find certain orderings of data samples than other orderings. Then a statistical test is proposed to compare the log probability of the dataset under the original ordering to the log probability under random permutations on sharded datasets. Experiments are conducted with one 1.4B-gpt2 model trained from scratch on 10 test sets, and the results prove the effectiveness of the proposed framework.", "questions": "- Could you further explain \"high false positives\" in existing n-gram-based analyses?\n- How did you deal with the labels for the data samples in test sets?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper targets an interesting and exciting problem in the community, test set contamination. \n- Based on the hypothesis, this paper proposed a contamination detection method, which is intuitive and easy to deploy in other settings.\n- The method is verified with a 1.4B language model trained from scratch, and the existing Llama2 model, both showing promising results even when the test set only appears a few times in the pre-training corpus.", "weaknesses": "- I'm most concerned about the definition of contamination used in this paper. Currently, the most popular definition of contamination follows the n-gram analysis. In real-world scenarios when training large language models, it's hardly seen to directly feed original data samples in their original ordering as shown in Figure 1. The application of this work could be greatly limited.\n- From Figure 3, it seems that the parameters for shards and permutations are sensitive and have to be carefully selected when being applied to other test sets.\n- The paper only targets direct sentence appearance in the pre-training stage. What about instruction-tuning data in the SFT stage?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)