Time Travel in LLMs: Tracing Data Contamination in Large Language Models
Authors: Shahriar Golchin
Mihai Surdeanu
Venue: ICLR 2024 spotlight (2024)
Keywords: Data Contamination
Large Language Models (LLMs)
Guided Instruction
Memorization
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose an effective method for detecting data contamination—presence of test data from downstream tasks—in the pre-training data of large language models (LLMs), achieving between 92% and 100% accuracy when validated against expert evaluations.

Abstract: Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a potential major issue in measuring LLMs' real effectiveness on other tasks. We propose a straightforward yet effective method for identifying data contamination within LLMs. At its core, our approach starts by identifying potential contamination at the instance level; using this information, our approach then assesses wider contamination at the partition level. To estimate contamination of individual instances, we employ "guided instruction:" a prompt consisting of the dataset name, partition type, and the random-length initial segment of a reference instance, asking the LLM to complete it. An instance is flagged as contaminated if the LLM's output either exactly or nearly matches the latter segment of the reference. To understand if an entire partition is contaminated, we propose two ideas. The first idea marks a dataset partition as contaminated if the average overlap score with the reference instances (as measured by ROUGE-L or BLEURT) is statistically significantly better with the completions from guided instruction compared to a "general instruction" that does not include the dataset and partition name. The second idea marks a dataset partition as contaminated if a classifier based on GPT-4 with few-shot in-context learning prompt marks multiple generated completions as exact/near-exact matches of the corresponding reference instances. Our best method achieves an accuracy between 92% and 100% in detecting if an LLM is contaminated with seven datasets, containing train and test/validation partitions, when contrasted with manual evaluation by human experts. Further, our findings indicate that GPT-4 is contaminated with AG News, WNLI, and XSum datasets.

1mm Published as a conference paper at ICLR 2024
TIME TRAVEL IN LLMs: TRACING DATA CONTAMINATION IN LARGE LANGUAGE MODELS
Shahriar Golchin', Mihai Surdeanu Department of Computer Science, University of Arizona {golchin, msurdeanu] warizzaa..aa.aaaaaa.a.aaaaaaaa.a edu
ABSTRACT
Data contamination, i.e., the presence of test data from downstream tasks in the training data of large language models (LLMs), is a potential major issue in mea- suring LLMss real effectiveness other tasks. We propose straightforward yet effective method for identifying data contamination within L.MMs. At its core, our approach starts by identifying potential contamination at the instance level: using this information, our approach then assesses wider contamination at the partition level. To estimate contamination of individual instances, we employ "guided instruction." a prompt consisting of the dataset name, partition type, and the random- length initial segment of a reference instance, asking the LLM com- plete it. An instance is flagged as contaminated if the LLM's output either exactly or nearly matches the latter segment of the reference. To understand an entire partition is contaminated, we propose two ideas. The first idea marks dataset par- tition as contaminated if the average overlap score with the reference instances (as measured by ROUGE-L BLEURT) is statistically significantly better with the completions from guided instruction compared to a "general instruction" that does not include the dataset and partition name. The second idea marks dataset parti- tion as contaminated if classifier based on GPT-4 with few-shot in-context learn- ing prompt marks multiple generated completions as exact/near-exact matches of the corresponding reference instances. Our best method achieves an accuracy be- tween 92% and 100% in detecting containing train and test/validation partitions, when contrasted with manual evalu- ation by human experts. Further, our findings indicate that GPT-4 contaminated with AG News, WNLI, and XSum datasets
LLM contaminated with seven datasets,
INTRODUCTION
The rise of Transformer networks (Vaswani al. 2017) has spurred the development of large lan- guage models (LLMs), marking a new epoch in Natural Language Processing (NLP). This shift has led to an extensive range of LLMs (Touvron et al. 2023a by Biderman et al. 2023 Köpf et al. 2023 Chung et al. 2022; Penedo et al. 2023 inter-alia) which excel in various professional and academic benchmarks (Bang et al. 2023 Bubeck et al. 2023). Their superior performance pri- LLMs during
marily attributed to the massive web data consumed by these wettoo//iiliiopaaaaaeeeee training. However, the impressive LLM performance observed on many downstream tasks (c.g., summarization, natural language inference, text classification) may be inflated due data contami- nation, i.e., the presence of test data from these downstream tasks in the pre-training data of LLMs.. Guarantecing lack of contamination is not trivial due two potential sources of contamination: directly from ingesting the official version of a dataset (easier to control), and indirectly through duplicated data found somewhere on the web (nearly impossible to control) The potential of data contamination is especially relevant for closed models such as the GPT-3/3.5 family (Brown et al.
*Corresponding author:
'See the paper's repo at autttps:: httpa//www..c.....aa..aaaaaaaaaa
"While dataset licensing reduces indirect contamination to certain extent, it does not eliminate it. For example, websites such as the Hugging Face page for datasets (Wolf et al. 2020) currently host copies of the OntoNotes (Weischedel et al. 2013) and CoNIL-2000 (Tjong Kim Sang & De Meulder(2003) datasets, despite the fact that their respective licenses prohibit it.

---

Published as a conference paper at ICLR 2024
2020) and GPT-4 (OpenAl 2023:
Bubeck et al. 2023), and, needless to say, raises questions on the
validity of evaluations and benchmarks conducted so far (Chang et al. 2023 Zhu et al. 2023 Bordt & von Luxburg 2023
Ray 2023
Penedo et al. 2023).
To address this issue, we propose an inexpensive and robust approach to detect data contamination for given dataset partition automatically. Importantly, our approach functions under two realistic assumptions: (a) we lack direct access to the pre-training data of the LLMs, and (b) we have limited computational resources. Intuitively, our method starts by identifying potential contamination in individual instances that are drawn from a small random sample of the corresponding dataset parti- tion (we use samples of 10 instances in this work). Using the information obtained from individual instances, our approach then assesses if an entire dataset partition is contaminated.
More formally, to identify contamination of individual instances, we employ a "guided instruction:" prompt that integrates distinct identifiers from the source dataset from which the reference instance originates. Such information includes the dataset name, its partition (e.g., train, test, or validation), and a randomly selected initial portion of the reference instance, complemented by its label when relevant. With these signals in the prompt, we instruct the LLM to finish the given partial instance. Using these generated individual completions, we propose two heuristics to estimate if an entire dataset partition is contaminated. The first heuristic states that partition is likely to be contaminated if the average overlap score between generated completions and reference instances (as measured by ROUGE-L (Lin 2004) and BLEURT (Sellam et al. 2020)) observed with the guided instruction is statistically significantly larger than the one measured with a "general instruction," which does not include the dataset and partition name. The second heuristic labels a partition as contaminated if a classifier based on GPT-4 with few-shot in-context learning (ICL: Brown al. (2020) marks at least one generated completion as an exact match with the reference instance or at least two generated completions as near-exact matches, where near-exact match indicates completion that exhibits considerable semantic and lexical alignment with the reference instance.
The primary contributions of this paper are as follows:
(1) We propose a novel data contamination detection method for LLMs that is inexpensive and ro- bust. As indicated above, our method combines a "guided instruction" to complete partial instances randomly drawn from the investigated dataset partition and several heuristics to generalize from instance- to partition-level contamination decisions.
(2) We evaluate our proposed methods in 28 distinct scenarios. These scenarios are created by two state-of-the-arr LLMs: GPT-3.5 and GPT-4, and span seven datasets for classification, summariza- tion, and natural language inference (NLI) tasks. The rationale behind the 28 scenarios is that for each dataset, we separately explore potential data contamination in the train and test splits (or the validation set, in cases where the labeled test set is not publicly available). Our evaluation indicates that our best method is the one that uses guided instruction to complete partial instances, and the one that evaluates these completions by the GPT-4 few-shot ICL classifier, achieving 92%-100% accuracy compared to contamination labels assigned by human experts for dataset partitions.
(3) Our analysis indicates that GPT-4 showed evidence of contamination within the test partitions of AG News (Zhang et al. (0015), WNLI (Wang et al. 2018). and XSum (Narayan et al.) 2018) datasets. These findings support the observation that data contamination is a serious issue that must be considered in downstream evaluations when using LLMs.
2 RELATED WORK
Despite its importance, the topic of data contamination is not as thoroughly examined as its closely related field, data memorization (Carlini et al. 20233 Kandpal et al. 2022 Carlini et 202T Razeghi et al. 2022). Among the limited investigations focusing specifically on data contamination LLMs, we find notable examples in RRafford et al.] (2019) and Brown et al. (2020) on GPT-2 and GPT-3,
respectively. They used high-order n-grams (e.g., 13-gram) to detect overlapping content between
the pre-training data and the evaluation dataset. Most research subsequent to Brown et al. (2020) adopted similar methods for detecting data contamination (Touvron et al. 2023b] Du et al. 2022 Chowdhery et al. 2022 Wei et al. 2021), and most recently, substring matching for GPT-4 (OpenAl 2023). However, the scope of existing research has been predominantly confined to model providers, and it encounters specific limitations, particularly when applied to closed-source L.Mss. These
2

---

Published as a conference paper at ICLR 2024
limitations primarily involve the need for access to pre-training data (Brown al. 2020: Du et al. 2022 Wei et al. (02I), the requirement for substantial computational resources Touvron et al. 2023b), or the need for extensive manual labor (Chowdhery et al. (0022). Our approach aims to overcome these hurdles, enabling the assessment of data contamination in scenarios where the pre-
training data is either not openly accessible when significant computational hardware is not available despite having access to the pre-training data.
Our paper is closest in spirit the work of [ainz et al. (2023), who also detected contamination when access to the pre-training data is not available. This effort prompted ChatGPT, particularly when GPT-3.5 is its base model, to generate the first instances from different dataset partitions. The underlying assumption here is that if an LLM can reproduce dataset instances, it must have been trained using that particular split. However, our research shows that this method can be unreliable and subject to failure. Such failures can result either from the sparsity introduced by the request to reproduce the first instances of a dataset split or from the inability to bypass the safety filters set by the model provider when the model is asked to generate copyrighted content like dataset instances. Throughout this paper, we refer to this approach as "ChatGPT-Cheat?" taking inspiration from the title of the referenced blog post.
3 APPROACH
In our approach, we operate under two core assumptions: (1) lacking direct access to the pre-training data of the LLMs, and (2) having limited computational resources. Given these premises, our detec- tion strategy for data contamination is anchored by two pivotal insights. First, we examine individual instances within a dataset partition to spot contamination at the instance level. Second, given that LLMS are pre-trained on large-scale data, detecting contaminated instances can act as signal of broader contamination. As a result, the associated partition can be labeled as being leaked to the LLM's pre-training data.
To discern contamination at the instance level, we focus on replicating instances by the LLM. In this context, exact replicas of instances serve red flags for contamination in the corresponding partition. Note that, due to the inherent probabilistic behavior of LLMs, achieving perfect replicas is not always possible even when contamination is certain. Nevertheless, instances that are closely replicated have a twofold function: while they can offer insightful indications of potential contam- ination, the fact that many datasets draw from web-based sources implies that partial replicas can also arise by happenstance. This overlap introduces uncertainty in drawing a definitive conclusion about the underlying partition. Thus, it essential to check for consistent and significant signs of contamination within the partition.
In the following sections, we first elaborate on our method and the necessary components to compel LLM into reproducing dataset instances. We then delve into the procedure for evaluating the con- tamination status of existing LLMS for an entire partition based on these instances. Furthermore, leveraging the fine-tuning option offered by OpenAl for the GPT-3.5 base model, we undertake a study in which we intentionally contaminate the GPT-3.5 base model with partitions that our method detected uncontaminated. Subsequently, we subject the contaminated GPT-3.5 to our technique, further showcasing our method's s effectiveness in pinpointing data contamination within LLMs.
3.1 DETECTING INSTANCE-LEVEL CONTAMINATION
3.1.1 COMPONENTS TO MEASURE INSTANCE-LEVER CONTAMINATION
To gauge instance-level contamination, we utilize two distinct methods: the first leverages BLEURT and ROUGE-L scores, while the second draws on few-shot ICL prompting with GPT-4 Each method employs particular components; however, the first two wwo-guided instruction and the next token prediction mechanism- are shared. The third to the first method. For both methods, we begin our process by steering the LLM towards the (po- tentially contaminated) dataset partition using guided instruction that integrates the dataset name, partition of interest, and the Landom-lengte initial segment of a randomly selected instance and its label if it is available. The LLM is then instructed to complete it. For the first method, repeat this step using general instruction that omits the dataset and partition name. An example of a guided versus general instruction is depicted in Figure 1 We detail all the required components below.
general instruction-is exclusive
3

---

Published as a conference paper at ICLR 2024
Instruction: You are provided with Sentence 1 from the validation split of the WNLI dataset. Finish Sen- Ience as appeared in the dataset. Sentence 2 must exactly match instance the dataset. Sentence I:
Instruction: Finish Sentence based on Sentence 1. such that the following label shows the logical rela- tionship between Sentence and Sentence 2.
Sentence 1:
The dog chased the cat, which ran up It waited at top.
The dog chased the cat, which ran up tree. It waited at the top.
Label: (entailment)
Label: (entailment)
Sentence 2:
Sentence 2:
The cat was at the top of the tre a after being chased by the dog.
The cat waited the top.
An example of a guided (left) and general (right) instruction employed for a paired- instance dataset. In this example, using GPT-4, the guided instruction results in an exact match, whereas the general instruction does not.
Figure I:
(1) Guided Instruction-A Means to Navigate the LLM's Output. By employing instruction- tuning top of causal language modeling (CLM; Vaswani al. (2017); Radford et al. 2018)) LLMs can be guided by human directives (Wei et al. 2022; Sanh et al. 2022 Chung et al. 2022), ) This serves as tool for controlling the LLM's output using natural language. Thus, we form guided instruction such that it incorporates the dataset and split name in the input prompt, thereby directing the LLM towards the underlying dataset split. A comprehensive list of all the instructions used in this study for different tasks/datasets can be found in Table 5 in Appendix A
(2) Next Token Prediction-A Means to Unravel Data History. Primarily, data contamination occurs during the CL.M pre-training phase since it constitutes the largest part of training in LLMS and utilizes web data. Without instruction tuning, an LLM only attempts complete an input prompt based on data seen during the CLM pre-training phase (Ouyang et al. 2022). Notable models that exhibit this behavior include GPT-2 and GPT-3. We, therefore, employ the next token prediction mechanism to trace data history. In particular, we feed the model the variable-length initial segment of a dataset instance, chosen randomly from a particular split, prompting to finish the partial instance. For labeled instances, we integrate the corresponding labels in the input prompt. This reflects that if an instance was ingested during the L.M's pre-training, label was ingested too
For paired-instance datasets, we present the model with the initial sentence and its corresponding label. In the case of single-instance datasets, instances with multiple sentences are arbitrarily cut at the end of a complete sentence, whereas for instances containing single (long) sentence, a random sentence fragment is eliminated. Finally, the LLM tasked with finishing the provided initial part. Figure shows this process for a paareediinstane dataset.
Therefore, once a contaminated LLM is prompted with guided instruction, its output should mirror the subsequent segment of the reference instance under the guidance of the dataset and split name.
(3) General Instruction-An Alternative Facet of Causal Language Modeling. We formulate the general instruction measure the impact of the guidance given in the guided instruction. This general instruction only requests the completion of the partial instance without specifying the dataset or its partition. As a result, when using this instruction, the generated sequence solely relies on the CLM pre- tre-training phase, akin to autoregressive models without instruction tuning. This enables us to establish a baseline for generated random replicas and assess how much the guided instruction influences the LLM LLM-generated part of the input partial instance. We assess this influence in terms of overlap, semantics, and structural similarity with the reference instance. This analysis is crucial as even when the output of LLM does not perfectly match the reference instance, it still enables us to detect potential signs of contamination.
3.1.2 MEASURING INSTANCE-LEVER CONTAMINATION
We introduce two methods for measuring contamination at the instance level:
'Incorporating labels in the input prompt is essential to account for false positives when generating down- stream completions. Ilstrratioo of the impact of label integration on downstream completions are provided in Table Appendix B
4

---

Published as a conference paper at ICLR 2024
To quantify the overlap between the completions produced guided and general instructions and reference instances, we employ two metrics: ROUGE-L (Lin 2004) and BLEURT (Sellam et al. 2020). While ROUGE-L assesses lexical similarity, BLEURT gauges the semantic relevance and fluency of the resulting sequence with respect to the reference instance. Instance-level contamination is detected if the average overlap scores from either metric, when applied to completions from the guided instruction, exceed those from the general instruction.
BLEURT & ROUGE-L:
proncuccenn-pppueeeeeeeeee under both
GPT-4 Evaluation: While both BLEURT and ROUGE-L quantify the overlap between the gen- erated and reference instances, they fall short of pinpointing near-exact matches. To bridge this gap, we adopt few-shot ICL prompting (Brown al. 2020) to dictate the detection of exact/near- exact matches based on human judgments (see Section 4 Human Evaluation for our definition of a near-exact match). Specifically, this method includes few representative examples of exact and near-exact Satches-ssurcee from human evaluations- in the prompt, which are used to assess all other generated completions. We chose GPT-4 for this task as requires no specialized prompting technique (Bubeck al. 2023), enhancing the reliability of its results. A visual representation of the few-shot ICL prompt used in our study can be seen in Figure 3 in Appendix C Also, detailed ex- amples, including their ROUGE-L and BLEURT scores, as well both human and GPT-4 few-shot ICL evaluations, are listed in Table i Appendix D
3.2 DETECTING PARTITION-LEVEL CONTAMINATION
To generalize from instance-level contamination to partition-level discrete decisions (i.e., the parti- tion is/is not contaminated), we take advantage of two observations:
dataset is likely to be contaminated if the average overlap score with the reference in- stunces (as measured by ROUGE-L and BLEURT) observed with completions from the guided in- struction is significantly larger than the one measured with the completions from the general instruc- tion. The motivation behind this idea is that since the only difference between the two instructions is that the guided instruction contains the dataset and partition name as guidance, the improvement only be explained by contamination.
Idea 1:
Idea 2: A dataset is likely to be contaminated if GPT-4 using few-shot ICL prompting detects at least exact match at least two near-exact matches. The intuition behind this idea is that even a small contaminated part of the sample of instances is likely indicative of a larger dataset partition leak. While the presence of an exact match among replicas generated by L.M is clear sign of contamination, the approach to handling exact or near-exact matches-and and deciding the number of such matches that indicates broader contamination- objectives. In this paper, we intuitively establish the above-mentioned criterion to extrapolate from the instance-level the partition- contamination. An empirical validation of our approach is also provided in Section 3.3
be tailored depending on specific research
We propose two algorithms, each implementing one of these ideas respectively.
A dataset partition is labeled as contaminated if the average overlap score (as provided by BLEURT and ROUGE-L) between the reference instances and generated texts with the guided instruction on a sample of ten instances is statistically significantly better than those produced by general instruction under a non-parametric bootstrap resampling test Fi
Algorithm 1:
The advantage of this algorithm is that it is non-parametric, i.e., we do not need to decide on an arbitrary threshold on the ROUGE-L or BLEURT scores to indicate contamination. However, its drawback is that even significant increase in overlap may still come from generated instances that a human would not consider an exact or near-cxact match. Algorithm addresses this limitation.
A dataset partition is labeled as contaminated if GPT-4 with few-shot ICL prompting flags at least one generated completion as an exact match or a minimum of two completions as near- exact matches within sample of ten instances. All completions in this setting generated solely by guided instruction.
Algorithm 2:
We evaluate both these algorithms in Section 5
*Details of bootstrap resampling method can be found in Appendix
5

---

Published as a conference paper at ICLR 2024
Instruction: You are provided with Sentence I from the train split of the RTE dataset. Finish Sentence 2 up- peared in the dataset. Sentence 2 must exactly match the instance in the dataset.
Sentence I: Twelve of Jupiter's ss moons are relatively small and scem to have been more likely captured than to have been formed in orbit around Jupiter. Label: O (not entailment)
Sentence 2:
GPT-3.5:
The format Jupiter's s twelve relatively small moans is more likely due to thelr capture rather than being formed in orbit around Jupiter.
Contaminated GPT-3.5:
Jupiter has twelve
Figure 2: An example of exact match generated by the GPT-3.5 contaminated with the train split of the RTE dataset versus an inexact match generated by the GPT-3.5 base model, both under the same guided instruction. This example is one of the training instances used during contamination.
Table Results after introducing intentional contamination to the GPT-3.5 base model us- ing guided instruction. tick (V) indicates the identification of at least one exact replica from the training instances used for contamination by our top-performing method (Alg. 2: GPT-4 ICL) and human evaluation.
Table Results of identifying contamination of GSM8K dataset within GPT-4 when guided instruction is used. double tick (V (VV) sig- nals the identification of two more near- exact replicas from the train split of this dataset by our top-performing method (Alg. 2: GPT-4 ICL) and human evaluation.
Method
AG News
RTE
XSum
Method
GSM8k
Alg. 2: GPT-4 ICL
V
Alg. 2: GPT-4 ICI.
Human Evaluation
Human Evaluation
3.3 INSTANCE REPLICATION: A VALID APPROACH TO DETECT DATA CONTAMINATION
To validate our choice for the hyperparameters used in Algorithm 2, i.c., the number of exact/near- exact matches to declare contamination, we performed controlled study in which LLM is con- taminated on purpose with several datasets. To this end, we used the GPT-3.5 base model and a subset of the train partition of the following datasets (one dataset from each task in question): AG News, RTE, and XSum. Note that all these partitions were marked as uncontaminated for GPT-3.5 by the human evaluators (see Table 4 and Section 4 Human Evaluation). To mimic the LLM's pre-training on web data, we retained only minimal metadata about the datasets as they appear on the web when scraped. In particular, we used: the dataset title, the partition name, and the entire instance 5 Following training, we evaluate the generated completions by our best- tech- nique (Algorithm GPT-4 ICI.) (see Table 33. Figure 2 visualizes the generated replicas before and after contamination in one of our experiments when guided instruction is utilized In addition, summarizes our findings from this study. The key conclusion of this experiment is that the Table D
contaminated LLM generated least one exact match in each setting. This underscores that the replication even one exact match stands as a robust and undeniable indicator of contamination
As second experiment, employed GPT-4 and the GSM8k dataset (Cobbe et al. 2021). This choice was motivated by OpenAl's technical report on GPT-4, which indicates contamination from its train split (OpenAl 2023). Given that this dataset comprises mathematical problems, our ob- jective is to replicate the questions in the dataset while withholding their corresponding answers & Table reports results from this experiment. Our results highlight that contamination is not solely identified through exact matches; near-exact matches are also indicative. To account for the probabilistic nature of LLMs, we set threshold of two for the minimum number of near-exact matches to indicate contamination. As shown, this is supported by the data.
$ All data formats used for the contamination of GPT-3.5 are detailed in Table 10 in Appendix
'Further examples are provided in Table 11 in Appendix
Details on the continued training of the GPT-3.5 hase model are presented in Appendix
An example of this replication process is provided in Table 11 in Appendix G

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTAL SETUP
Data: Our evaluation employs seven datasets derived from various tasks, namely classification, summarization, and NLI. The datasets in question involve IMDB (Maas et al. 2011). I, AG News (Zhang et al. 2015), Yelp Full Reviews (Zhang et al. 2013), SAMSum (Gliwa et al. 2019), XSum Narayan al. 2018), WNLI (Wang et al. 2018), and RTE (Wang et al. 2019). In order to ensure a comprehensive experimental setup, all our experiments are carried out on both the training and test/validation splits of the aforesaid datasets. We make use of the publicly available divisions, working with the training and test splits for each. However, for the last two datasets, only the validation splits were publicly accessible with their labels. Considering our research's emphasis on pinpointing data contamination with minimal dataset instances, the resource constraints, and our intention to facilitate the replication of this approach by other researchers, we randomly chose 10 instances from each split for our experiments.
Setting: We use snapshots of GPT-3.5 and GPT-4 from June 13, 2023-specifically] 3 5-tuuroo-0611 and gpt both accessed via the OpenAI API, as founda- tion LLMs. To obtain deterministic results, we set the temperature to zero and capped the maximum completion length at 500 tokens. Contrarily, our comparative method (ChatGPT-Cheat?) uses the chat user interface (UI), which we also leveraged for conducting the experiment under this method. Specifically, we used the UI versions of GPT- and GPT-3.5 that were released July 20, 2023.
Human Evaluation: We undertake human evaluation, led by two domain experts P to characterize contamination by identifying both exact matches and near-exact matches of individual instances. The term "exact matches" is self-explanatory: "near-exact matches" are completions by the LLM that, while not identical, show considerable overlap and maintain significant semantic and structural similarity to the reference instance. To generalize from individual instances to entire partitions, the human annotators followed the rule described in Algorithm 2 that was validated empirically in Section 3.3 partition is flagged as contaminated if the instance-based evaluation identifies at least one exact match or at least two near-exact matches.
Evaluation Metries:
In our analysis, the computation of the BLEURT score varies based on the structure of the dataset/instance, as this metric hinges on the fluency and quality of the generated se- quence. For single-instance datasets, where individual instances are randomly cut off mid-sentence and then completed by the LLM, we join the model model-produced continuation to the severed reference instance and then calculate the BLEURT score. Conversely, for instances from paired-instann multi-sentence dingle-instanee datasets, the BLEURT score is computed solely for the newly pro- duced sequence. We highlight that BLEURT score computations use the most recent checkpoint provided, i.e., BLEURT-20 (Pu et al. 2021). On the other hand, regardless of the dataset/instance type, the ROUGE-L score calculation exclusively pertains the portions of the text finished by the LLM. This is due to the score's dependency on statistical attributes rather than semantic consistency.
and
Comparative Framework: We compare our proposed methods against the ChatGPT- -heet??? method (Sainz et al. 2023). Unlike our method, which uses a binary scale to determine contamina- tion, the comparison approach includes a "suspicious" category. This designation is invoked when the LLM,, upon being asked to generate the first instances of dataset split, outputs characteristic attributes such as data format, IDs, or other -ataset- details instead of the actual instances. the model, on the other hand, fails to produce these characteristics, it is deemed uncontaminated.
5 RESULTS AND DISCUSSION
Table 3 lists the overall accuracy of our proposed methods in 28 distinct settings: two LLMs (GPT- and GPT-3.5) x 14 dataset partitions coming from seven datasets. Table provides detailed breakdown of each method per dataset partition and the respective LLM. We draw the following observations from our experiments:
(1) Algorithm 1, which hinges on the difference in average overlap scores between outputs from guided instruction and those from general instruction, performs well in the majority of settings. Its best performance is a success rate of 13/14 when using GPT-4 as the underlying model and 9/14
The two annotators had almost perfect inter-rater agreement across all settings. This is due to the fact that small subset of instances was used for contamination detection, and contamination is evident when occurs.
7

---

Published as a conference paper at ICLR 2024
Table 3: Overall accuracy detecting contamination across 14 partitions for GPT-4 and GPT-3.5.
The two LLMs are evaluated against human annotators. The "Success Rate" shows how often each method matches human judgment, while the "Accuracy" gives the corresponding percentages.
GPT-4
GPT-3.5
Method
Success Rate
Accuracy
Success Rate
Accuracy
Strict Eval.: ChaiGPT-Cheat?
0/14
0.00%
11/14
78.57%
Lenient Eval.: ChatGPT-Cheat?
9/14
64.29%
13/14
92.86%
Algorithm I: BLEURT
11/14
78.57%
9/14
64,29%
Algorithm I: ROUGE-L
13/14
92.86%
7/14
50.00%
Algorithm GPT-4 ICL
14/14
100.00%
13/14
92.86%
when using GPT-3.5. We consider these results exciting given the algorithm's simplicity. However, Table 3 shows that: (a) its performance is not universally good-it performs chance level when using ROUGE-L on GPT-3.5 outputs (7/14), and (b) its success rate varies depending on the metric in use (i.e., BLEURT or ROUGE-L).
(2) In contrast, Algorithm 2, which relies on GPT-4 evaluation using the few-shot ICL prompt, aligns closely with human evaluations. Specifically, in experiments run GPT-4 and GPT-3.5, its success rates are 14/14 and 13/14, respectively. These accuracies are higher than any produced by Algorithm 1 and maintain consistency across all the settings with the two LLMs.
(3) Upon assessing the results of ChatGPT-Cheat? method, we discover that this method invariably labels partitions as suspicious- likely due to the precaution against generating copyrighted content which is activated by safety filters for all scenarios involving GPT-4. Given this, we interpret the outcomes of this method through two lenses: strict and lenient evaluation. In the strict evaluation, we do not interpret the suspicious label as contaminated uncontaminated. Under this assessment, no partition is correctly classified according to human evaluation (0/14) in settings with GPT-4, and 11/14 in settings with GPT-3.5. In the lenient evaluation, we convert the suspicious label to either contaminated or uncontaminated in a way that maximizes the performance of this method. In this setting, the ChatGPT--heeat method correctly identifies 9/14 and 13/14 in settings with GPT- and GPT-3.5, respectively. However, this lenient evaluation is unrealistic due to the overfitting in inter- preting the suspicious label. These findings support our observation that identifying contamination at the instance level, before extrapolating to the partition level, is a more resilient strategy.
(4) Last but not least, the human evaluation reveals that the train and test/validation splits of both the AG News and WNLI datasets were included in GPT-4's pre- trainirinng data. However, for IMDB and RTE, only the training partitions were incorporated, while for XSum, only the test split was leaked. For GPT-3.5, the only data exposure was the test partition of the XSum dataset. These findings confirm that, despite their creators' efforts, today's L.LMs have ingested NLP datasets. We hope that this observation informs the design of better scientific experiments with LLMs in the NLP space.
6 CONCLUSION
We proposed a novel method to detect data contamination in LLMs, assuming no access to their pre-training data. Our approach begins by pinpointing data contamination at the instance level. This was achieved by prompting the LLM to produce the replica of the secondary segment of a dataset instance given its random-length initial segment, dataset name, and partition type, a process we called "guided instruction." From here, we adopted a set of rules to generalize from instance-level to broader partition-level contamination. This involved leveraging statistically significant differ- ences from BLEURT and ROUGE-L scores between generated completions by guided and general instructions, as well as evaluations from GPT-4 with few-shot in-context learning prompting.
Our evaluation spanned 28 different settings, including seven datasets along with their respective train and test/validation partitions and two LLMs: GPT-4 and GPT-3.5. Our findings indicated that while the replication technique via guided instruction is notably effective, the most accurate eval- uation approach that was closely aligned with human judgments for detecting data contamination was the few-shot in-context learning prompt with GPT-4, which integrates few example instances
8

---

Published as a conference paper at ICLR 2024
An assessment of our proposed methods in contrast ChatGPT-Cheaa? method. We evaluate Algorithm using BLEURT and ROUGE-L, as well as Algorithm 2 which relies on GPT-4 decisions via few-shot ICL prompting. The evaluations are performed on 10 instances randomly drawn from each split of particular dataset, with GPT-4 and GPT-3.5 serving as the LLMs that are investigated. Partition-leve contamination is represented the following ways: (1) While asterisks (*) indicate statistically significant differences between the completions produced by guided and general instructions (measured by BLEURT and ROUGE-L), underlined numbers indicate settings that align with human evaluations (Algorithm
Table 4:
1). (2) A single tick (V) points the presence of at least one exact match, while a double tick (VV) signals the identification two or more near-exact matches (Algorithm 2). A cross sign (x) denotes that neither of the aforesaid conditions were met. For the ChatGPP--Cheat method, this cross sign indicates that the model's output does not contain any specific information about the first instances of the dataset partition upon the request generate them. For the same method, the question mark (?) highlights partitions that are deemed suspicious.
Datasets
Model
Method
Split
Instruct.
IMDB AG News
Yelp
RTE
WNLI SAMSum XSum
General
0.43
0.63
0.43
0.54
0.47
0.58
0.54
Train
Guided
0.48
*0.70
0.41
*0.60
*0.62
0.58
0.60
Alg. 1: BLEURT
General
0.43
0.62
0.41
0.50
0.50
0.58
0.64
Test/Valid
Guided
0.42
*0.72
0.38
#0.53
*0.65
0.59
0.67
General 0.14
0.17
0.15
0.41
0.26
0.13
0.18
Train
Guided
#0.24
*0.35
0.17
*0.51
#0.59
0.14
*0,38
Alg. ROUGE-L
General
0.16
0.16
0.15
0.31
0.36
0.23
Test/Valid
Guided
0.16
*0.37
0.16
0.34
*0.63
0.15
*0.38
GPT-4
Train
Guided
Alg. GPT-4 ICL
Test/Valid Guided
x
Train
Guided
?
ChatGPT-Cheat?
Test/Valid Guided
?
?
?
Train
Guided
/
Human Evaluation
Test/Valid Guided
//
General 0.45
0.58
0.45
0.50
0.49
0,59
0,54
Train
Guided
0.39
*0.64
0.42
0.50
*0.56
0.56
Alg. 1: BLEURT
General
0.45
0.60
0.42
0.47
0.47
0.58
0.62
Test/Valid
Guided
0.43
0.62
0.40
*0.53
*0.54
0.59
0.62
General
0.12
0.06
0.13
0.37
0.29
0.10
0.14
Train
Guided
0.12
*0.16
#0.16
0.32
*0.43
0.11
0.22
Alg. ROUGE-I.
General
0.13
0.10
0.11
0.23
0.32
0.13
Test/Valid
Guided
0.14
*0.20
*0.14
0.31
*0.42
0.17
0.23
GPT-3.5
Alg. GPT-4 ICL
Train
Guided
x
x
x
x
Test/Valid Guided
x
ChatGPT-Cheat?
Train
Guided
x
x
Test/Valid Guided
Human Evaluation
Train Guided
x
x X X
x
Test/Valid Guided
X
V
from human assessments in the input prompt. This method yielded a success rate in pinpointing data contamination across 14/14 scenarios for GPT-4 and 13/14 4 for GPT-3.5
L'imitations, Data contamination can arise from different sources manifest in various ways, e.g., direct inclusion of dataset instances, metadata contamination, etc. Our best pest-performing method for detecting contam- ination (guided instruction with GPT-4 ICL) does not distinguish between different types of contamination, treating both exact and near-exact replicas of dataset instances as indicators of data contamination. Therefore, encourage future research that detect contamination, pinpoint its sources, and identify its various forms.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We extend our appreciation to Steven Bethard and Eduardo Blanco for their expert guidance and valuable feedback on the early draft of this paper. This work was partially supported by the De- fense Advanced Research Projects Agency under the Habitus program. Mihai Surdeanu declares a financial interest in lum.ai. This interest has been properly disclosed to the University of Arizona Institutional Review Committee and is managed in accordance with its conflict of interest policies.
REFERENCES
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, mul- tilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7:8, 2009.
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hal- lahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023.
Sebastian Bordt and Ulrike von Luxburg. Chatgpt participates in a computer science exam. ArXiv, toss/3033.09611, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https::/prrceeeddngg. http://ppooeedddnng.eeee......ee nwwrips cC/ ccwpaper.pppp... F1222339.ccdeee33311012222222222
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang. Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204. 02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay. William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finetuned language models, 2022.
10

---

Published as a conference paper at ICLR 2024
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. 2021.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp. 5547- 5569. PMLR, 2022.
B. Efron. Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics, 7(1): 1 26, 1979. doi: 10. 1/214//os/117334455 URL https : :/:::://ooioo ottp/// 1214///000.1111////////
1176344552
Bradley Efron. Second Thoughts on the Bootstrap. Statistical Science, 18(2): 140, 2003. doi: 10. 10.121//s/1063994968. URL httpss///ool..rr//1101/12//
Bradley Efron and Robert J. Tibshirani. An Introduction the Bootstrap. Number 57 in Monographs on Statistics and Applied Probability. Chapman & Hall/CRC, Boca Raton, Florida, USA, 1993.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL Workshop Textual Entail- ment and Paraphrasing, pp. 1-9, Prague. June 2007. Association for Computational Linguistics.
URL https: http::://acaaatthooo
org. W 1401
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization,, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v I/D19-5409. URL BRLhttts::// www aclweb.
org/ httg//athhloogyDDil.. 9-101199999999
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. volume pp. 785- 794, 2006.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models, 2022.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations - democratizing large language model align- ment, 2023.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL httlhttts::/aalatteee htl.accaantoooggoyoo .r../r//... -00-1013
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http:
wwww. awwwab.ww........ org/ htt.l/ee..rr/atttooooo////////// httpaaatthoooll-looao// 1015
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/ abs/1808 008.08745, 2018.
11

---

Published as a conference paper at ICLR 2024
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint XrXiv:2306.00 01 16, 2023.
Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for mt. In Proceedings of EMNLP, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9 2019.
Partha Pratim Ray. Chatgpt: A comprehensive review on background, applications. key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 2023.
Yasaman Razeghi, Robert L. Logan IV au2, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. 2022.
Oscar Sainz, Jon Ander Campos, Iker Fercra- Julen Etxaniz, and Eneko Agirre. Did chat- gpt cheat on your test? pittss//hiitt roa gi thub. 1o hmm/mmmmcotttm i iortion/ blog/ 2023. Accessed: 2023-07-06.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables zero zero-shot task generalization, 2022.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computa- tional Linguistics, pp. 7881 18892, Online, July 2020. Association for Computational Linguis- URL
tics. doi: 10. hel-main.. 704
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independee named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https:
/www acw.wb. org/ httprg//aatooooooo/oooo3 0419
www.aac..ebb.orrr/aatttooooo/...
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anna Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
12

---

Published as a conference paper at ICLR 2024
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, lliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Nenworks for NLP, pp. 355- Brussels, Belgium, November 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/W1 1/1111111111166666
htt.ttttt::////t ac aatl
org/w/ /11 B-54
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Superglue: A stickier benchmark for general- purpose language understanding systems. arXiv preprint arXiv: 1905.00537, 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 1.XY::2009.01652. 2021.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. Ontonotes re- lease 5.0 (tc2013t1,, Linguistic Data Consortium, Philadelphia, PA, 23:170, 2013.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick vOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: Sfate-of-the--a natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL httph::////// //t.pp::/wwww... wwwww www.waawwwbbb.coo..m wct.eb..roo/atttoooogy/o// 2020. www.e-dees.d....
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In NIPS, 2015.
Yiming Zhu, Peixian Zhang, Ehsan ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human-generated labels? study of social computing tasks. ArXiv, abs/2304. 10145, 2023.
13

---

Published as a conference paper at ICLR 2024
Appendices
A LIST OF ALL GUIDED AND GENERAL INSTRUCTIONS
Table presents a thorough collection of all the guided and general instructions employed through- out study.
Table 5: comprehensive list of all guided and general instructions used in our experiments. Placeholders include: {split.name} for the partition (or split) name; {dataset the dataset name; (first.piece for the first part the dataset instance cut at the tail randomly the whole first sentence in NLI-based datasets; and {label) } for the corresponding label of the incomplete input instance.
for
Task
Guided Instruction
General Instruction
You are provided with the first piece of an instance from (split, __meme] split of the seataset..nae } dataset. Finish the second piece of the instance exactly appeared in the dataset. Only rely the original form of the instance in the dataset to finish the second piece.
Instruction:
Instruction: Finish the second piece based on the first piece, such these two pieces become single in- stance with the following label.
Classification
(single-instance)
Label: {label}
First Piece: {first.piece)
Label: (label)
Second Piece:
First Piece: (first.piece)
Second Piece:
You are provided with Sentence 1 from the (split_name) split of the (dataset_naam) dataset, Finish Sentence as appeared in the dataset. Sentence 2 must actly match the instance in the dataset. Sentence 1: ffirst.piece Label: {label}
Instruction:
Instruction: Finish Sentence based on Sentence 1, such that the following label shows logical relationship between Sentence and Sentence 2.
IL. (paired-instance)
Sentence 1: (first.piece) Label: {label}
Sentence 2:
Sentence 2:
You are provided with the first piece of a summary from the {splis i name] split of the (cataset.namm
Instruction:
Finish the second piece based on the first piece, such that these two pieces become single summary.
Instruction:
dataset. Finish the second piece the summary as dataset. Only
Summarization (single-instance)
exactly appeared rely on the original form the sum- mary the dataset finish the see- ond piece.
First Piece: {first.piece)
Second Piece:
First Piece: (first_piece)
Second Picce:
You are provided with the first piece of a one-sentence sum- mary from the [spl t_name] split of the {dataset, (dataset_name) dataset. Finish the second piece of the sum- mary exactly appeared in the dataset. Only rely the original form of the summary in the dataset to finish the second piece.
Instruction:
Instruction: Finish the second piece based the first piece. such that these two pieces become single one- sentence summary.
One-sentence Summary (single-instance
First Piece: {first.piece) Second Piece:
First Piece: (first
Second Piece:
14

---

Published as a conference paper at ICLR 2024
B IMPACT OF LABEL INTEGRATION ON DOWNSTREAM COMPLETION
To emphasize the impact of incorporating labels into the input prompt generating downstream completions and to demonstrate their significance in reducing the generation of false positives, Table 6 presents illustrative examples. These examples compare completions produced when dataset instance is paired with both a correct and an incorrect label in the input prompt. Specifically, when the L.M is prompted with the initial segment of reference instance and its correct label, the completion aligns exactly with the subsequent part of that reference instance, resulting in an exact match. In contrast, the completion generated under the same initill segment of the reference instance but paired with an incorrect label shows an inexact match.
Table A side-by-side comparison of completions generated by guided instruction when an initial segment of reference instance is paired with a correct and an incorrect label in the input prompt. In every example, completion under the correct label produces an exact match for the subsequent segment of the reference instance in the input prompt, whereas that with the incorrect label leads to
an inexact match. All completions in these examples are generated by GPT-4, as the base model.
Completion under Correct Label (Exact Match)
Completion under Incorrect Label (Inexact Match)
You are provided with Sentence from the validation split of the WNLI dataset. Fin- ish Sentence 2 as appeared in the dataset. Sen- tence 2 must exactly match the instance the dataset.
Instruction:
You are provided with Sentence from the validation split of the WNLI dataset. Fin- ish Sentence as appeared in the dataset. Sen- tence must exactly match the instance in the dataset.
Instruction:
Sentence 1:
The dog chased cat, which ran up tree. It waited at the top.
Sentence The dog chased the cat, which ran up tre.. waited at the
Label: 1 (entailment)
Label: 0 (not entailment)
Sentence 2:
Sentence
The cat waited the top.
The dog waited at the top.
You provided with the first piece of instance from the train split of the AG News dataset. Finish the second piece of the instance as exactly appeared the dataset. Only rely on the original form of the instance the dataset finish the second piece.
Instruction:
Instruction: You provided with the first piece of an instance from the train split of the AG News dataset. Finish the second piece of instance exactly appeared the dataset. Only rely the original form of the instance in the dataset to finish the second piece.
Label: (Business)
Label: 3 (Sci/Tech)
First Piece: Oil and Economy Cloud Stocks' Out- look (Reuters) Reuters Soaring crude prices plus worries about the economy
First Piece: Oil and Economy Cloud Stocks' Out- look (Rcuters) Reuters worries about the economy
crude prices plus
Second Piece:
Second Piece:
and the out outkook for earnings are expected to hang over the atock market next week during the depth of the sunmer doldrums.
and the look for the dollar pressured stocks on Friday, leaving the market nearly unchanged.
C FEW-SHOT IN-CONTEXT LEARNING PROMPT
Figure showcases the few-shot ICL prompt employed to evaluate the model-generated candidate against the reference text using GPT-4. Within this prompt, we present GPT-4 with one exact match and three exemplary instances of near-exact matches, all pre-labeled by human evaluation. These examples guide GPT-4 in discerning the difference between near-exact and inexact matches, in line with human assessment.
D ILLUSTRATIONS OF EXACT, NEAR-EXACT, AND INEXACT MATCHES
Displayed in Table are examples of exact, nxar-exact, and inexact replicas of the reference instance when guided instruction and GPT-4 used. This table also includes computed metrics such as
15

---

Published as a conference paper at ICLR 2024
Instruction: You are provided with reference text and candidate text. The candidate text is generated replica of the reference text. Your task is to determine if the candidate text is an exact or near-exact match of the reference text. Near-exact match candidates must precisely match the reference candidates in terms of sentence structure, overlap, and contextual similarity. Respond only with "Yes" "No".
Example 1:
Reference Text: The cat waited at the top.
Candidate Text: The cat waited the top.
Answer: Yes (exact match)
Example 2:
Reference Text: icy surface of Jupiter's largest moon, Ganymede. These irregular masses may he rock formations, supported by Ganymede's icy shell for billions of years. Candidate Text: icy surface Jupiter's largest moon, Ganymede. These irregular masses may rock formations, supported by Ganymede's icy shell for billions of years. This discovery supports the theory that Ganymede has subsurface ocean. Scientists used gravity data from NASA's Galileo spacecraft to cre- ate geophysical model of the interior of Ganymede. Answer: Yes (near-exact match)
Example 3:
Reference Text: 50th Anniversary of Normandy Landings lasts year.
Candidate Text: The 50th anniversary celebration of the first Normandy landing will last year. Answer: Yes (near-exact match)
Example 4:
Reference Text: Microsoft's Hotmail has raised its storage capacity 250MB.
Candidate Text: Microsoft has increased the storage capacity of its Hotmail e-mail service 250MB.
Answer: Yes (near-exact match)
Example 5:
Reference Text: Mount Olympus the center of the earth.
Candidate Text: Mount Olympus located at the center of earth.
Answer:
Yes (near exact match)
Figure 3: A display the few-shot ICL prompt utilized for instance-level data contamination detec- tion using GPT-4. In this illustration, examples I through 4 are part of the prompt, while example is updated with a new input reference and candidate for evaluation, depending on whether there is an exact, near-exact, or inexact match. While Example 1 represents an exact match, the other examples display variations indicating near-exact matches: Example reveals scenario where the candidate text has substantial overlap with the reference but includes added details; Examples 3 and highlight situations where the candidate text possesses both semantic and structural similarity to the reference text.
ROUGE-L, BLEURT, and results from human and GPT-4 few-shot ICL evaluations. addition, Table 8 showcases comparative outcomes for the same examples using general instruction.
E STATISTICAL ANALYSIS: BOOTSTRAP RESAMPLING
We examine the statistical significance of results stemming from guided versus general instructions. Bootstrap resampling technique, involving 10,000 samples in the resampling process, employed for this investigation (Efron 1979; Efron & Tibshirani 1993, Efron 2003). We concentrate on the alternative hypothesis that posits guided instructions produce outcomes closer to reference instances than those generated from general instructions, as evaluated by fluency, quality, and similarity. The performance metrics utilized here are BLEURT and ROUGE-L scores. We regard the ROUGE-L and BLEURT scores statistically significant if the p-values & 0.05. We list all the computed p- values in Table and highlight the statistically significant results by marking them with an asterisk in Table 4
16

---

Published as a conference paper at ICLR 2024
Table 7: Examples of exact, near-exact, and inexact matches along with their respective BLEURT and ROUGE-L scores, and judgments from GPT-4 few-shot ICL and human evaluations. These examples are generated by GPT-4, as the underlying language model.
Metric/Method
Reference Instance and Its Replica by Guided Instruction
Reference Instance:
Review: Bromwwell High a cartoon comedy.
same some programs school life, such "Teachers": My 35 years in the teaching profession lead believe that Bromwell High's satire is much closer to reality than is "Teachers" The scramble to survive financially, the insightful stu- can see right through
pathetic teachers' pomp. the pettiness of the whole situation, remind me of the schools I knew and their students. When I saw the episade in which student repeatedly tried to burn down the school, immeddatell recalled -..... at momeu High. classic line: INSPECTOR: I'm here sack one your teachers. STUDENT: Welcome Bromwell High.. exppect that many adults of my age think that Bromwell High is far fetched. What pity that it isn't! Label: (positive)
You are provided with the first piece of an instance from the train split of the IMDB dataset. Finish the second piece of the instance as exactly appeared in the dataset. Only rely on original form the instance in the dataset finish second pisce.
Instruction:
Label: (positive)
pro- grams about school life, such as "Teachers". : My 35 years in the teaching profession lead me to believe that Bromwell High's satire is much closer to reality than is "Peachers"
Bromwell High curtaon comedy.
BLEURT Score: 0.99
Second Piece:
ROUGE-L Score:
The scramble to survive financially, the insight students can see right through their pathetic zeachera' pomp, the pett Ineea of the whole situation, all remind of the schools knew and their students.
GPT-4 ICL: Exact Match Human Eval.; Exact Match
saw episode which student repeazedly tried the school, I inmediacely recalled
at classic line: INSPECTUR: m'm hare sack of your teachers. STUDENT: : Welcome to Bromwell High. expect that many adults of my age think that Bromwell High is far fetched. what pity that it isn't!
Eigh. À
Reference Instance:
Sentence Although they were different planets, Oscar-winning actor Cage's 's new son and Superman have something in Sentence 2: Nicolas Cage's 's son called Kal-el.
both namedd Kal
Label: (entailment)
Instruction: You are provided with Senience I from the train split of the RTE datasel, Finish Sentence 2 as appeared in the dalaset. Sentence 2 must exactly match the instance in the dataett.
BLBURT Score: 0.87 ROUGE-L Score:
different planets, Oscar-winning actor Cage's new son and Superman have something in common both were named Kal-el.
Sentence Although they
GPT-4 ICT.: Near exact Match
Label: (entailment)
Human Eval.; Near-exact Match
Sentence 2:
Nicol aa Cage's S new son 1 named Na 1ell...
Reference Instance:
Summary: Kim about tell that Harry bought sofa, and he needs grey pillows.
You are provided with the first piece a summary from the test split of the SAM- Sum dataset. Finish the second piece of the summary as exactly appeared in the dataset. Only rely the original form of the summary the datasct finish the second picco.
Instruction:
BLBURT Score: 0.48 ROUGE-L Score:
First Piece: Kim is about to tell mom that Harry bought
GPT-4 ICL: Inexact Match
Second Piece:
Human Eval.: Jncxact Match
1 worr ad
upset. Kim i advi tel in
a positive way, focusing on Hleryy' happiness.
17

---

Published as a conference paper at ICLR 2024
Table 8: Completions generated by GPT-4 under general instruction for examples shown in Table
Metrie
Reference Instance and Replica by General Instruction
Reference Instance:
Review: Bromwell High is cartoon comedy. ran at the same time as some other programs about school "Teachers", My years in teaching profession satire mach closer reality than "Teachers". : The scramble survive linancially, the insightlul stu- dents who can see right through their pathetic teachers' pomp. the pettiness of the whole situation, all remind the schools knew and their studenrs. When I saw the episode which student repeatedly tried to burn down the school, I immediately recalled one your teachers. STUDENT: Welcome Bromwell High. expect that many of my think that Bromwell High far fetched. What pity that it isn't!
helieve Bromwell High's
High. A clussic line: INSPECTOR:
Label: (positive)
Instruction:
Finish the second piece based the lirst such that these become following label.
single
Label:
(positive)
Bromwell High curtoon comedy.
pro-
grams about school life. such as "Teachers", My 35 years in the teaching profession lead me to believe that Rmowell High's satire much closer to reality than is "Teachers".
BLEURT Score: 0.18 ROUGE-L Score:
are truly be levvalle and s situation Lons chat mirroz the zeal 1ife e experienced in typical high school, The humar sharp and intelligent making it ze reIreshing change from the usual sitcoms. It'a shame that is didn' L run for longer, as it was brilliant depiction of school life that was entertaining
The incredibly insightful, with characters
Lhought provoking Bromwell High definitely = show that deserves more recognition for its realistic portraysl the educat ion system.
Reference Instance:
Sentence I: Although they were born on different planets, Oscar-winnigg actor Nicolas Cage's new son both named Kal-el.
Superman have something Sentence 2: Nicolas Cage's called Kal-el. (entailment)
Finish Sentence 2 based on Sentence 1, such that the following label shows the log- ical relationship between Sentence and Sentence
Instruction:
Sentence Although they were born on different planets, Oucar-winning actor Nicolas Cage's new son and Superman have something common - both were named Kal-el.
BLEURT Score:
ROUGE-L Score: 0.57
(entailment)
Sentence 2:
Nicolas Cage' new son and Superman share the same name, Kal Kal-ei.
Reference Instance:
Summary:
Harry bought new solia, grey pillows.
Instruction: Finish the second piece based on the first piece, such that these two pieces become single summary.
BLBURT Score: 0.44
First Piece: Kim is about to tell mom that Hurry bought
ROUGEEL
Second Plece:
without in firat.
18

---

Published as a conference paper at ICLR 2024
Table 9: p-values for differences between BLEURT and ROUGE-I scores of guided and general instructions, computed using bootstrap resampling with 10,000 resampling samples. p-values S 0.05 indicate statistically significant results.
Datasets
Model
Metric
Split
Instruction
1MDB AG News
Yelp
RTE
WNLI SAMSum
XSum
Train
Guided
0.319
0.005
0.981
0.041
0.000
0.478
0.115
BLEURT
Test/Valid Guided
1.000
0.000
1.000
0.075
0.035
0.283
0.170
GPT-4
Train
Guided
0.017
0.000
0.073
0.000
0.000
0,424
0.000
ROUGE-L
Test/Valid Guided
0.509
0.000
0.465
0.165
0.003
0.105
0.000
Train
Guided
1.000
0.006
1.000
0.465
0.008
0.746
0.093
BLEURT
Test/Valid Guided
0.992
0.134
0.932
0.030
0.020
0.293
0.321
GPT-3.5
Train
Guided
0.374
0.000
0.000
0.968
0.000
0.312
0.068
ROUGE-L
Test/Valid Guided
0.190
0.042
0.000
0.051
0.044
0.147
0.152
F CONTINUED TRAINING OF GPT-3.5 BASE MODEL FOR INTENTIONAL CONTAMINATION
For our validation study for contamination using the GPT-3.5 base model, we employ the previously referenced snapshot, gpt-3. pptt-3.--turbo 5-tur5o-o-0000 01.3.. To conduct continued training GPT-3.5, submit fine-tuning job via the OpenAl API. While the model provider terms the option of continued training fine-tuning, is to reproduce what the LLM-in our case, GPT-3.5-potentially observed during its pre-training phase when exposed to web data. To achieve this, we format the data in way that encompasses the dataset title and its associated division, coupled with the entire details of the instance. We embed this information since it represents the minimal metadata an instance might possess when extracted from web data.
approach does not center around conventional fine-tuming Our objective
All data formats we used to introduce data contamination are listed in Table Each dataset in- stance is formatted according to the provided formats, including both the name of the dataset and the specific split from which derives, as metadata. It is important to clarify that our approach completely differs from instruction tuning, we do not incorporate any specific instructions within the data.
Due to our project's budget limitations and our emphasis on manageable number of training sam- ples, we opt to work with one datasct for each task in our validation study. In particular, we take 100 random samples, ensuring they were evenly distributed based on the label, from the training splits of the AG News, RTE, and XSum datasets to expose the GPT-3.5 base model. For training, all default hyperparameters set by OpenAl maintained during continued training phase. Upon training completion, we utilize particular checkpoints provided by OpenAl. For every experiment, the base model of GPT-3.5 separately contaminated using each dataset split, resulting in three separate checkpoints, each associated with one of the aforementioned dataset splits.
G EXAMPLES OF REPLICAS GENERATED PRE AND POST CONTAMINATION OF GPT-3.5
In Table II we showcase two examples of exact replicas derived from our controlled contamina- tion study with GPT-3.5. These replicas are generated from the contaminated checkpoints obtained through additional training of the GPT-3.5 base model on the subset of the training partitions of the AG News and XSum datasets. Additionally, instance in the training set of the GSM8k dataset when using GPT-4 as the underlying LLM. All these replicas produced via the guided instructions.
highlight near-exact match achieved from
19

---

Published as a conference paper at ICLR 2024
Table 10: A complete list of all data formats used to contaminate the GPT-3.5 base model by fur- ther training. Each of these data formats is separately used to format every single instance with respect to the dataset task. Placeholders are as follows: {split -aame indicates the split name; Stataset_name refers to the dataset name; {instance) represents full instance in classifi- cation datasets; [sentencel] and (sentence2) stand for premise and hypothesis NLI-based datasets; [document] } and (summary) correspond to entire document and its summary for a sin- gle instance in the summarization datasets; and {label} replaced with the input instance's label where applicable.
Task
Data Format
This is an instance from the {split t._nmee split of the (datasse__nam) dataset.
Classification
Instance: { finstance)
Label: {label}
This is instance from the (split name) split of the (datase: name] dataset. Sentence 1: {sentencel) Sentence (sentence2)
NLI
Label: {label}
This is an instance from the {split name} split of the (datase: name) dataset.
Summarization
Document:
[document]
Summary: (summary)
H DETAILED DESCRIPTION OF DATASETS
IMDB Movie Reviews Dataset. The IMDB Movie Reviews dataset is a balanced corpus 50,000 movie reviews used for sentiment analysis tasks. It is split evenly into 25,000 training and 25,000 testing reviews, each further balanced for positive and negative sentiments. In this dataset, positive reviews are identified by a score that is 7 or more out of 10, while negative reviews are denoted by a score that falls 4 or below out of 10.
AG News Dataset. The AG News dataset, commonly used benchmark, encapsulates news articles from the AG's corpus website. It is neatly divided into four categorical classes, namely world, sports, business, and science/technology. The dataset contains 496,835 categorized news articles from 2,000 news sources. For each class, the AG News dataset furnishes 30,000 training and 1,900 test samples.
Yelp Dataset. The dataset is sourced from the Yelp Dataset Challenge conducted in 2015, containing a massive number of 1,569, 264 samples, all of which include review texts. This dataset is the foundation for two distinct classification tasks. The first task involves predicting the exact count of stars assigned by the user, while the second task to predict the polarity label, with a perspective that categorizes 1- and 2-star ratings negative, and 3- and 4-star ratings as positive. For the full-scale star rating prediction, the dataset includes 130,000 training samples and 10,000 testing samples for each star category. Similarly, the polarity- based dataset comprises 280,000 training samples along with 19,000 test samples, distributed among each polarity category.
Recognizing Textual Entailment (RTE) Dataset. The Recognizing Textual Entailment (RTE) dataset originates from a succession of annual textual entailment challenges. These datasets were combined by the authors of the benchmark using data from four different editions: RTEI (Daaan et al. 2005). RTE2 (Haim et al. 2006). RTE3 (Giampiccolo et al. 2007), and RTES (Bentivogli ict al. 2009). The examples within these datasets were primarily formulated using text from news and Wikipedia sources. To maintain consistency, all these datasets were adapted into two-class split. For those datasets that initially consisted of three classes, the categories of "neutral" and "contradiction" were combined to form single class termed "not entailment". The RTE dataset combined has 2,490 examples for training, 277 examples for validation, and 3,000 examples for testing.
20

---

Published as a conference paper at ICLR 2024
Table 11: Examples of exact matches from GPT-3.5 after contamination with a subset of train splits of AG News and XSum datasets, alongside a near-exact instance reproduction from the GSM8K training set using GPT-4. Match types (exact/near-exat) matches) are provided only for the gener- ated completions from contaminated GPT-3.5 checkpoints, when contrasted with the complctions from the GPT-3.5 base model (all completions by the GPT-3.5 base model are inxacc matches). All completions are generated by guided instruction.
Method
Reference Instance and Its Replica Guided Instruction
Reference instance from the train split of AG News dafaset:
Report: Pennington Set to Sign Extension with Jets (Reuters) Reuters - 'The New York Jets and quarterback Chad Pennington are close deal that would make the quarterback the highest- paid player in frunchise history. Label: (Sports)
Article:
You provided with the first News dataset. Finish the second piece the instance as exactly appeared the dataset. Only rely on the original form of the instance in the dataset to finish the second piece.
Instruction:
instance from the train split AG
(Sports)
First Piece: Report: Pennington Set to Sign Extension with Jets (Reuters) Reuters The New York Jets and quarterbuck Chad Peunington close deal that would make the
Second Piece:
GPT-3.5:
28-year-old signall oa
ghghtt-ppid player franch history, sccording to sources familier with the negotiations. The excension is expected to keep Ponningtoon which she Jet for the next five seasona, sclidifying position
March
Human Eval.; Exact Match
field. deel comes after Pennl Penningtenn Impresa ive performance laat sesaon, where he ld the Jebs to the playof is and earned the NFT. Comeback P Player of the award. With this extension, to Penni Panningtoon and rheir bell ief in a abillty 1yy to lead the team to success.
leader
are showing
commit
Contaminated GPT-3.5
guartarback the hi hights paid player franchise history.
Reference instance from the train split XSum dataset:
Summary: Astranomers have found evidence for planet heing devoured by its yielding insights into the fate that will befall Earth in billions of years.
Instruction:
You provided with the first of senience summary from the train split XSum dataset. Only rely on the original form of the summary the dataset finish the second piece.
second piece summary exactly appeared dataset.
First Piece:
Astronamers have found evidence for planet being devoured by its yielding
insights Second Piece:
GPT-4 ICL: Exact Match
Human Eval.: Exact Match
GPI-3.5:
into the process planetary des
desc.uct vuct and the Tate of exopl anetts,
Contaminted GPT-3.5:
will befall
billions years.
Reference instance from train split GSM8k dataset:
Question: writes pages day. How long will him books that are pages ench?
You are provided with the first piece of question from the train split of the GSM8k dataset. Finish the second rely on the original
Instruction:
the question exactly appeared in the dataset. Only the question dutaset
the second piece.
First Piece: John writes 20 pages day. How long will it take him to write
GPT-4 JCL: Nxact Match
Human Eval.: Near exact Match
Second Plece:
GPT-4:
books of 400 pages each?
21

---

Published as a conference paper at ICLR 2024
Winograd Natural Language Inference (WNLI) Dataset. The WNLI (Winograd Natural Lan- guage Inference) dataset is a benchmark for natural language understanding tasks, particularly for evaluating coreference resolution and pronoun disambiguation in context. The dataset is derived from the original Winograd Schema Challenge (Levesque et al. 2012) and contains sentence pairs where pronoun needs be resolved by determining whether refers to the same entity as the previous sentence. While the dataset has balanced training set between two classes, the test set is imbalanced, with 635 training examples, 146 testing examples, and 71 validation examples.
SAMSum Dataset. The SAMSum dataset, compiled by the Samsung R&D Institute in Poland, comprises around 16,000 English messenger-style conversations with summaries. These dialogues, created by linguists, reflect a variety of styles, registers, and topics similar to real-life messenger interactions. Each conversation is annotated with third-person summary and categorized based on the number of utterances, ranging from 3-30. The dataset primarily consists of two-person dialogues.
Extreme Summarization (XSum) Dataset. The Extreme Summarization (XSum) dataset serves evaluation dataset for abstractive single-document summarization systems. Its objective is to generate a concise one-sentence summary that answers the question, "What is the article about?". The dataset comprises 226,711 news articles, each accompanied by a one-sentence summary. These articles were collected from BBC articles spanning the years 2010 to 2017 and cover wide range of domains, including news, politics, sports, weather, business, technology, science, health, family, education, entertainment, and arts. The official random split allocates 90% (204,045 documents) for training, 5% (11,332 documents) for validation, and 5% (11,334 documents) for the test set, respectively.
Grade School Math 8k (GSM8K) Dataset. The GSM8k dataset is curated dataset consisting of 8,500 linguistically diverse grade school math word problems, crafted meticulously by human au- thors. This collection is divided into 7,500 training examples and 1,000 designated for testing. The complexity of these problems varies, requiring between to sequential steps for resolution. Pre- dominantly, the solutions entail executing series of basic arithmetic operations-nnmely addition, subtraction, multiplication, and division-to deduce the final answer. This dataset is ideal for tasks involving multi-step mathematical reasoning.
22


Reviews:
[{"review_id": "tt6KtrCPtQ", "reviewer": "Reviewer_HqCE", "summary": "The paper investigates data contamination in Large Language Models (LLMs). To address this issue, the paper suggests a novel approach where a random-length initial segment of the data instance is used as a prompt, along with information about the dataset name and partition type. The paper then assesses data contamination based on the LLM's output. This evaluation can be done by measuring the surface-level overlap with the reference instance or by leveraging GPT-4's few-shot prediction capabilities.\nBased on the results, the paper suggests that GPT-4 is contaminated with AG News, WNLI, and XSum datasets.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The proposed method is straightforward and adaptable to a wide range of datasets.", "weaknesses": "1. I have concerns regarding the soundness of the paper's evaluation methodology.\nThe proposed method hinges on the assumption that a data instance is contaminated in an LLM if the LLM can complete the instance based on its prefix. The paper's evaluation primarily revolves around how well the proposed methods are compared to human experts under this assumption However, these concerns raise doubts about whether the underlying assumption holds for several reasons.\n(1) The inability of an LLM to complete a reference does not necessarily imply that the instance was not used during training. It could be attributed to model forgetting or the model's failure to memorize surface-level features while still having learned the semantic-level features of the data instance. This could lead to a high false negative rate in the evaluation.\n(2) An LLM may have encountered the input of a data instance without having seen the actual data instance itself. For instance, in sentiment classification tasks, text can be included in the LLM's training set as long as its label is not provided alongside the text. The ability to complete the input text does not necessarily indicate data contamination in LLMs, potentially resulting in a high false positive rate.\n(3) The unavailability of training data details for ChatGPT and GPT-4 due to their proprietary nature prevents a comprehensive evaluation of the proposed method. The current evaluation primarily focuses on how closely the proposed model aligns with human guess about data contamination, in cases where the actual training data is undisclosed. It seems essential to assess the method on models with publicly accessible training data, such as OpenLlama.\n2. There's a potential fairness issue in using GPT-4 to evaluate the prediction from itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ahw9C9Ww4y", "reviewer": "Reviewer_6REU", "summary": "The authors propose a method to detect dataset leakage/contamination in LLMs first at the instance level before they bootstrap to the partition (test, valid, train). At the instance level, they do so via a guided prompt that was crafted to bias the model towards outputting data in a format that is likely to overlap with the dataset example. Once a candidate dataset and partition is flagged, the authors mark it as leaked if either the overlap between reference instances is statistically significantly higher when using guided peompts compared to general prompts or is determined to be an exact or near match using a GPT-4 based classifier. The best classification models match with 92-100% accuracy the labels provided by human experts.", "questions": "My main question/concern is on the reliability of the instance level contamination detection. Specifically, if a dataset follows a well-known and observed in other dataset format, if a model such as GPT-4 sees the dataset description and meta-data, would it generate sufficiently many near matches to appear as contaminated with a new dataset, despite observing only metadata?\nI understand that this work relaxes the prefix match from Sainz et al., but I wonder if this is likely to generate false signal in models that show an ability to generalise from few examplea and/or metadata.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Intuitive guided and general prompts to detect instance level contamination.\n- Approximating human expert classification for exact and approximate match using GPT-4 as a classifier, i.e. approximating semantic match.\n- Validation on a known contaminated LLM.", "weaknesses": "- The authors rely on the algorithm to begin with when deciding what partitions were not leaked and should be added during fine-tuning. This has a circular dependence/assumption. (This point was addressed during discussion with the authors as a writing/explanation issue rather than a true circular dependence).\n- Different levels of data leakage is not considered. For example, would GPT-4 be detected as having seen paritions of datasets that follow well-known formats seen from other datasets if it sees only a metadata description of a new dataset? (This limitation is now acknowledged in a footnote with additional details on metrics as well as below in the discussion with the authors).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVPppV5XKg", "reviewer": "Reviewer_KgsM", "summary": "The paper offers a fresh perspective on assessing the capabilities of LLMs in terms of potential dataset contamination. The authors introduce two novel methodologies to measure these aspects. The first method uses BLEURT and ROUGE-L evaluation metrics, while the second leverages GPT-4's few-shot in-context learning prompt. A significant part of the evaluation revolves around identifying potential data contamination issues, and the results are compared against a baseline method, ChatGPT-Cheat. The findings underscore the nuances and intricacies in effectively evaluating LLMs and the paper serves as a guide to understand their limitations and potential.", "questions": "How do the proposed methods scale when evaluating even larger LLMs or when considering different architectures beyond GPT?\nCould the authors provide insights into the trade-offs between the two algorithms, especially in terms of computational cost and reliability?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper offers a fresh perspective on assessing the capabilities of LLMs in terms of potential dataset contamination. The methodologies introduced, especially the use of GPT-4's few-shot in-context learning, is innovative.\nQuality: The research appears thorough with detailed evaluations using two different algorithms. The results are well-tabulated, and the comparison with ChatGPT-Cheat offers a clearer understanding of the proposed methods' effectiveness.\nClarity: The paper is structured coherently, and the methodologies, evaluations, and results are presented in a clear and organized manner, making it easier for the reader to follow.\nSignificance: Given the increasing utilization and reliance on LLMs in various applications, understanding their limitations and behaviors is crucial. This paper addresses this need, making it a significant contribution to the field.", "weaknesses": "Scope: The paper focuses primarily on GPT-3.5 and GPT-4. A broader range of LLMs could provide more generalizable insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "itGFll8nSX", "reviewer": "Reviewer_NUVj", "summary": "The authors propose a method for assessing data contamination in models by prompting the model to complete instances of a given dataset. The generated responses are evaluated using either overlap (via ROUGE-L / BLEURT) differences between prompts with/without the dataset specification, or a GPT-4 based few-shot classifier. The instance level information is then used to decide on the partition (train/test/val) / dataset level.", "questions": "- The first paragraph of Section 3.1.1 is quite confusing: the text makes it difficult to understand which components it refers to.\n- Given your observations in Section 5, (3) that the `ChatGPT-Cheat` method fails due to GPT-4's safeguards being triggered when trying to retrieve examples from these datasets, I wonder how these safeguards would also affect the results you get with your prompt templates.\n- For unpaired instances, a random-length prefix is displayed in the prompt; how is this random length sampled? And what is its effect?\n- (minor comment): typo: page 2, first paragraph, last sentence: \"obtains\"  -> \"obtained\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method can be applied to black-box models hidden behind an API with little model interaction.\n- The study of data contamination, particularly with respect to commonly used evaluation datasets, is an important area.\n- An attempt is made to validate the method by fine-tuning on a given downstream dataset", "weaknesses": "- The aggregation from instance to partition level seems to be rather ad hoc (contaminated if at least 1 or 2 instances are contaminated); and a proper ablation regarding these hyperparameters is missing.\n- Experiments are performed only with black-box models; using open models with known training details would have supported a more reliable evaluation, since (more of) their training details are known.\n- The comparison of Alg. 2 (GPT-4 ICL) with human judgments seems to be rather biased, since the same human annotators created the ICL examples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)