Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024)

Abstract: While large language models have achieved remarkable performance on various code generation benchmarks, there have been growing concerns regarding potential contamination of these benchmarks as they may be leaked into pretraining and finetuning data. While recent work has investigated contamination in natural language generation and understanding tasks, there has been less extensive research into how data contamination impacts the evaluation of code generation, which is critical for understanding the robustness and reliability of LLMs in programming contexts. In this work, we perform a comprehensive study of data contamination of popular code generation benchmarks, and precisely quantify their overlap with pretraining corpus through both surface-level and semantic-level matching. In our experiments, we show that there are substantial overlap between popular code generation benchmarks and open training corpus, and models perform significantly better on the subset of the benchmarks where similar solutions are seen during training. We also conduct extensive analysis on the factors that affects model memorization and generalization, such as model size, problem difficulty, and question length. We release all resulting files from our matching pipeline for future research1.

heading: 1 Introduction
text: The compute requirements (encompassing both model size and data volume) for training large language models (LLMs) has grown significantly over the years, correlating with consistent observed enhancements in model performance in both language (Kaplan et al., 2020; Hoffmann et al., 2022) and code (Ni et al., 2023) generation tasks. Larger models trained on larger training corpora tend to lead to an increased risk of data contamination
1Code and data available at https://github.com/ yale-nlp/code-llm-contamination
of the expected output, which we refer to as instances of evaluation benchmark data appearing within the data used during the training of models. LLMs tend to perform better on evaluation samples that resemble the documents and instances encountered during training (Kandpal et al., 2022a; Razeghi et al., 2022; Magar and Schwartz, 2022), and are more likely to emit memorized training data when they have seen it multiple times (Kandpal et al., 2022b; Carlini et al., 2023). Recent papers have also shown evidence that LLMs are possibly contaminated (Golchin and Surdeanu, 2023; Yang et al., 2023), which limits our understanding of their generalization capabilities to unseen inputs.
Despite significant research into data contamination in natural language (NL) benchmarks (Golchin and Surdeanu, 2023; Chang et al., 2023; Blevins and Zettlemoyer, 2022; Dodge et al., 2021; Deng
14116
et al., 2023), there’s been relatively little exploration into how this issue affects the evaluation of code generation capabilities in LLMs. We posit that the fundamental disparities between NL and programs warrant a deeper examination. Recent studies, such as work by Karmakar et al. (2022); Ranaldi et al. (2024), suggest that code-based LLMs may demonstrate patterns of memorization, underscoring the need for scrutiny into their generalization capabilities to unseen cases. Key distinctions between code and NL include the critical role of syntax and the variable requirements for naming functions and variables across different programs. These differences lead us to argue that traditional surface-level comparisons might not be adequate for identifying contaminated data points.
In this paper, we propose a pipeline to measure the overlap between code generation benchmarks and pretraining corpus of code LLMs, incorporating both surface-level and semantic-level code matching. As a result of the exhaustive search amoung the training corpus with our pipeline, we provide a precise quantification of the examples whose solutions are seen during training, for popular code generation benchmarks as MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021). We study two open pretraining corpus which contain code, the PILE (Gao et al., 2020) and STARCODERDATA (Li et al., 2023), as well as three model series trained on either corpora, StarCoderBase (Li et al., 2023), Pythia (Biderman et al., 2023) and CodeGen-NL (Nijkamp et al., 2023). Our results show severe contamination of the widely used MBPP and HumanEval benchmarks within the PILE and STARCODERDATA corpora, as shown in Fig. 1, with models performing significantly better on questions that the models have seen the same or similar program solutions to. We perform thorough analysis on factors that may affect model memorization and generalization such as model sizes and difficulty of the questions. We also include a case study on outliers, to provide a more comprehensive understanding of model behavior given different levels of exposure to test data.

heading: 2 Methodology
text: To quantify data contamination for code LLMs, we first introduce methods used to measure program similarity from surface- and semantic-level in § 2.1. Next, in § 2.2 we describe how to combine similarity measurements to identify the overlapping pro-
grams in the training data and test benchmarks as well as introduce how to quantify data contamination based on the similarity scores and the number of appearances of similar programs seen during the course of training.

heading: 2.1 Measuring Program Similarity
text: While most popular code generation benchmarks focus on generating functions, the training data are often chunked by files, which may contain multiple functions or classes. This means that document-level deduplication techniques (e.g., Allamanis, 2019) cannot be used effectively, as other programs within the document may add too much noise. Thus we opt to perform substring-level matching, which is much more computationally heavy but also more accurate than methods used by previous work (Lee et al., 2022; Peng et al., 2023; Kandpal et al., 2022a). More specifically, we use a sliding window to scan the training data characterby-character and compute its similarity scores with gold solutions in the benchmarks. To maximize the recall of possible contaminated examples in coding benchmarks, we employ both surface- and semantic-level similarity measurements.
Surface-Level Similarity. To measure surfacelevel similarity between programs, we use the Levenshtein similarity score (Sarkar et al., 2016), which is the Levenshtein edit distance (Levenshtein, 1965) normalized by the length of both the source and target strings. We selected the Levenshtein similarity score as the first step in our pipeline because it is an easy-to-compute and intuitive measurement that can handle surface-level fuzzy matches between two programs. While the Levenshtein edit distance has been used before to deduplicate datasets at a file level (Chowdhery et al., 2022), we perform it on a substring level. An example of this can be found in Fig. 2.2
Semantic Similarity. While the surface-level similarity metrics can easily capture similar programs in surface form, two semantically similar or even identical programs can have very different surface form due to different identifiers (e.g., variable names) or whitespace characters. Therefore, finding semantically similar programs is also crucial for measuring contamination and understanding the generalization capabilities of the mod-
2we use the rapidfuzz python library to calculate the similarity score https://pypi.org/project/ rapidfuzz/.
Write a python function to find the minimum number of squares whose sum is equal to a given number.
(a) Problem Description.
1 def get_Min_Squares(n): 2 if n <= 3: 3 return n; 4 res = n 5 for x in range(1,n + 1): 6 temp = x * x; 7 if temp > n: 8 break 9 else:
10 res = min(res,1 + get_Min_Squares(n - temp)) 11 return res;
(b) Gold Program
1 def getMinSquares(n): 2 # if n <= 3: 3 # return n 4 # res = n 5 # for x in range(1, n+1): 6 # temp = x * x 7 # if temp > n: 8 # break 9 # else:
10 # res = min(res, 1 + getMinSquares(n - temp))
(c) Matched program in STARCODERDATA. Surfacelevel similarity = 91; semantic-level similarity = 0.
Figure 2: Example where surface-level matching works better than semantic-level. Because most of the program is commented out, the semantic-level similarity score is 0 despite the programs being otherwise identical.
els. To measure semantic similarity between programs, we adopt the Dolos toolkit (Maertens et al., 2022), which is a source code plagiarism detection tool for education purposes. Dolos first uses tree-sitter3 to tokenize and canonicalize the program into representations of abstract syntax trees (ASTs), then computes a similarity score representing the semantic-level similarity based on the k-gram matching between source and target programs. Since Dolos measures similarities based on the ASTs, non-semantic changes that greatly decrease the Levenshtein similarity scores, such as indentations and variable/function names, will not affect the scores calculated by Dolos. An example of this can be found in Fig. 4. Dolos was also used in previous works for detecting intellectual property violations (Yu et al., 2023).
3https://tree-sitter.github.io/tree-sitter/

heading: 2.2 Quantifying Data Contamination
text: For each problem and its gold solution in the test benchmark (e.g., MBPP), we would like to determine the most similar programs that the models have seen during training. However, this would require us to perform a pair-wise comparison with all programs in the training data using the similarity score metrics mentioned in §2.1. Because training data is usually on the scale of hundreds of gigabytes to terabytes, it is computationally expensive4 to run surface-level matching methods; running the code-specific semantic matching methods on the entire training dataset is even more computationally prohibitive.
Aggregating Similarity Scores. We use a twostage process to analyze test examples and their correct (gold standard) program solutions. First, we measure the surface-level similarity by calculating Levenshtein scores. This involves comparing all substrings of the same length as the gold solution across all relevant files in specific subsets of our dataset (see § 3.1 for details). We keep the top 500 programs5 with the highest Levenshtein similarity scores for each test example for the next step. The similarity scores found by searching the PILE and STARCODERDATA for gold programs from the MBPP benchmark are shown in Fig. 3, along with a comparison between results found using only the top score and those found using an average of the top 10 scores.
4We estimate that it will take 5.2∗105 CPU hours to search just the Python files from STARCODERDATA for MBPP.
5This is determined by a combination of automatic and manual inspection. For example, at the 500th most similar program from STARCODERDATA for MBPP, 95% of them have a similarity score < 72, which is no longer relevant by human inspection.
Write a python function to determine whether all the numbers are different from each other or not.
(a) Problem description.
1 def test_distinct(data): 2 if len(data) == len(set(data)): 3 return True 4 else: 5 return False;
(b) Gold Program.
1 def is_simple(graph): 2 if len(graph) == len(set(graph)): 3 return 1.0 4 else: 5 return 0.0
(c) Matched program in STARCODERDATA. Surfacelevel similarity = 79; semantic-level similarity = 93.
Figure 4: Example where semantic-level similarity works better than surface-level. The ASTs of two programs are identify despite different variable names.
With the top 500 programs with the highest Levenshtein similarity scores from the training data, we further compute the semantic similarity scores with the gold programs using Dolos. Then the aggregated similarity score is computed as the maximum of the surface-level similarity score (Ssurface) and semantic similarity score (Ssemantic) similarity scores:
S(p, p∗) = max(Ssurface(p, p∗), Ssemantic(p, p∗))
This aggregated similarity score is a simple and intuitive way to reflect how programs can be similar both from their surface form and semantics.

heading: 3 Experimental Setup
text: We select two of the most popular public pretraining corpora for general LLM and code LLMs, namely the PILE (Gao et al., 2020) and STARCODERDATA (Li et al., 2023), and three series of popular open-source models, i.e., Pythia (Biderman et al., 2023), CodeGen-NL (Nijkamp et al., 2023) and StarCoderBase6(Li et al., 2023). For the coding benchmark, we opt to study MBPP (Austin et al., 2021) and HumanEval (Chen et al., 2021) due to their popularity. We introduce them in more detail in the following subsection.
6StarcoderBase refers to the models originally trained on STARCODERDATA (Li et al., 2023). These models were further finetuned on Python code to create the Starcoder models.

heading: 3.1 Models and Pretraining Data
text: We select the models by the following criteria: 1) The pretraining data for the models must be publicly available; 2) To ensure non-trivial performance on the coding benchmarks, such models must have Python code in their pretraining data; 3) Additionally, we do not consider any models that are instruction-tuned, or trained with reinforcement learning from human feedback (i.e., RLHF), as it is hard to quantify the effect of such instructiontuning/human preference data along with the pretraining corpus. Based on these criteria, we study the following three model series in this work:
The PILE and Pythia. Pythia (Biderman et al., 2023) is a suite of 16 LLMs intended to facilitate research in many areas. All models are trained on the PILE dataset (Gao et al., 2020), with their size ranging from 70M to 12B parameters. We used the 1.4B, 2.8B, 6.9B, and 12B models for this study. We use the GitHub split of the training dataset, which has a raw size of 95.16 GiB.
The PILE and CodeGen-NL. Another series of models that are trained with PILE is CodeGenNL (Nijkamp et al., 2023), and we study the 350M, 2B, 6B, and 16B versions of it. Though stronger CodeGen models are available via further training on more code data, the exact copy of such data is not publicly released thus we choose to study the CodeGen-NL series. Due to the overlap of training data, we use the results of searching through the GitHub split that we did for the Pythia models.
STARCODERDATA and StarCoderBase. We use the 1B, 3B, 7B and 15.5B StarCoderBase models (Li et al., 2023) that were trained on the STARCODERDATA dataset (Li et al., 2023). Due to the size of the training data, we only search through 60.40 GB within the Python split of its training dataset. The STARCODERDATA dataset is a subset of the STACK (Kocetkov et al., 2022), created by filtering the STACK and applying additional decontamination. The STACK was created from permissively-licensed source code files, and was open-sourced to make the training of code LLMs more reproducible.7

heading: 3.2 Benchmarks
text: We measure the data contamination issues for the following two popular coding benchmarks:
7It is worth noticing that STACK went through a stringmatching-based decontamination process for MBPP and HumanEval, but we are still able to find traces of contamination for these two datasets.
MBPP (Austin et al., 2021) is a benchmark containing 974 short, crowd-sourced Python programming problems. We use the 500 questions within its test split.
HumanEval (Chen et al., 2021) is a benchmark consisting of 164 hand-written problems. Each problem contains a gold solution.
Notably, these two benchmarks come with gold program solutions, which we use to search the pretraining data as a query. To obtain the model performance and predictions on each of the dataset examples, we use the evaluation framework and model outputs from L2CEval (Ni et al., 2023).

heading: 4 Results
text: In this section, we first present our main results in §4.1, then with several analysis on how the length, difficulty and model sizes affects the our findings in §4.2, and finally we present a case study in §4.3.

heading: 4.1 Main Results

heading: 3.6% to 20.8% of the solutions are likely seen
text: during training. For an example in the test data (i.e., those of MBPP or HumanEval), we note it as “seen” if the aggregated similarity score is 100, i.e., a perfect match exists on the surface- or semanticlevel. Results in Fig. 1 show that 12.2% of the solutions in HumanEval have been seen by models
trained on the PILE and 18.9% have been seen by models trained on STARCODERDATA. For MBPP, 3.6% of it can be found in the PILE while as much as 20.8% have been seen by models trained on STARCODERDATA. Much less overlap is found for the PILE, as 3.6% of MBPP, but 20.8% of the solutions on MBPP problems have been seen for models trained on STARCODERDATA. These results suggest that a non-trivial part of both MBPP and HumanEval have been seen for the models trained on either the PILE or STARCODERDATA, suggesting a high contamination rate.
Models perform significantly better when similar solutions are seen during training. To observe the effect that having seen a solution during training has on a model, we conduct three different sets of experiments: 1) We first removed potentially contaminated questions from the dataset, and evaluated the models performance on the new dataset, as seen in Tab. 1. 2) We also highlight the difference in performance that models have between questions which they have seen similar solutions and questions which they have not. We use the performance gap Razeghi et al. (2022) between the top 10% and bottom 10% of programs based on aggregated similarity scores to do this. The performance gap of the largest models from the chosen model series is shown in Tab. 2, where it can be observed that all three models perform significantly better on questions in the top 10% of compared to questions in the bottom 10%. StarCoderBase-15.5B, which achieves an accuracy of 72% on the top 10% of questions and an accuracy of 22% on the bottom 10% of questions of the MBPP benchmark. The range of similarity scores for each model and benchmark can be found in Fig. 3. 3) Lastly we discuss the effect of models having seen the solution in §4.2, where we provide an analysis on decoupling memorization and question difficulty.
De-contaminated results. In an attempt to show the impact of seen questions on model performance, we remove potentially contaminated questions from each benchmark, showing the results in Tab. 1. We observe that removing not only questions that have been seen, but also questions where programs similar to the gold program have been seen during training has an adverse effect on model performance. Moreover, from the decontaminated results, the performance gap between different models could be much smaller. For example, the original accuracy (Acco) gap between StarCoderBase-15.5B and Pythia-12B is 23.8%, and after de-contamination, the performance gap is decreased to 13.9%. This indicates that a large part of the performance gap between different models may due to data contamination. While we do not find the performance rankings of the models to change with de-contaminated results, a study with more models might be needed for deriving any general conclusions.

heading: 4.2 Analysis
text: Ablations on model size. We show how model size affects accuracy in regard to the aggregate score in Fig. 5. We observe that larger models tend to perform better than smaller models in their fam-
ily, indicating that they are not only better at generalization, but also at memorization. We believe that the Pythia and CodeGen-NL models show similar trends in these graphs due to being trained on the same training data. We also note that the noise in each graph grows as there are fewer programs being evaluated on, explaining the 0% accuracy that some models show when evaluated on only questions they have seen 10 or more times during training.
Decoupling memorization and difficulty. We attempt to show that the model performance on seen questions is not just outlying questions that are easier than the remaining questions in the benchmarks. To do so, we compare the overall performance of models on the MBPP and HumanEval benchmarks against their performance on different subsets of questions based on seen and unseen questions in Tab. 3a. We show that while StarCoderBase models perform better on the subset of questions in MBPP that they have seen than on the unseen questions, Pythia and CodeGen-NL models generally perform worse on the same subset of questions. We also provide the sizes of each subset in Tab. 3b, and note the significant overlap between questions in HumanEval that have been seen by
STARCODERDATA or the PILE as compared to the MBPP benchmark. The improved performance of models in the StarCoderBase family on familiar questions in the MBPP benchmark does not appear to result from these questions being easier than those they haven’t encountered during training. For example, CodeGen-NL-16B has an overall accuracy of 19.6% on the MBPP benchmark, but has an accuracy of only 11.5% on the 104 questions that StarCoderBase has seen. This indicates that models having seen a solution to a question during training significantly increases the performance of models on these questions.
Effect of program length on similarity scoring. One possible concern is that the size of the gold programs could affect the similarity score. Longer strings can have more differences between one another without affecting their aggregated similarity score as much as in shorter strings. To analyze this, we plot the length of every gold program within the MBPP benchmark against the aggregated similarity score of the most similar string within the training dataset used for the StarCoderBase model family in Fig. 6. There does not appear to be a correlation between the length and the aggregated similarity score, or length and accuracy.

heading: 4.3 Case Study
text: Here we present a case study, by showing examples where the models have seen the gold solutions to 10 or more times but still fails to produce a correct solution at test time. Two representative examples are shown as Fig. 7 and Fig. 8. For the first example (Fig. 7), although programs that are similar to the gold program appears multiple times in the pretraining data, understanding the problem description is arguably harder part of the problem. As for the second example (Fig. 8), the gold program is quite simple thus it is not surprising that
1 def sum_series(n): 2 if n < 1: 3 return 0 4 else: 5 return n + sum_series(n - 2)
1 def sum_series(n): 2 sum = 0 3 for i in range(n): 4 sum += i 5 return sum
multiple matches in the training corpus are found, but it may also make it difficult for the model to associate such program with any specific natural language description.

heading: 5 Related Work
text: Measuring contamination. Our study on the effect of contaminated test questions on accuracy are similar to the work done by Carlini et al. (2020); Henderson et al. (2017); Jiang et al. (2024); Thakkar et al. (2020); Thomas et al. (2020), but instead of perturbing the training dataset, we search through the training datasets for the the gold solutions to the benchmarks. Another line of work in studying memorization is to find documents related to the output within a training dataset (Lee et al., 2022; Peng et al., 2023; Kandpal et al., 2022a; Magar and Schwartz, 2022). These works search the training dataset for documents relevant to a string and report the number of relevant documents. While we directly the effects of training data in model outputs, other approaches exist in using the model’s weights to find the parts of the training dataset that influenced the model (Han and Tsvetkov, 2022; Grosse et al., 2023). While this paper focuses on searching for contamination in open source models, many models are released without disclosing their training data. To search for contamination in these models, recent papers (Shi et al., 2023; Oren and Meister, 2023; Ranaldi et al., 2024; Deng et al., 2023) use the probabilities of model outputs to observe contamination. This style of approach seems to work primarily when there are multiple copies within the training dataset, and is unreliable at detecting duplication rates of 1 (Oren and Meister, 2023). More recently, Dong et al. (2024) identifies contamination by measuring the peakedness of each model’s output distribution
via sampling, which works for black-box LLMs but provides less certainty compared with our method. Plagiarism detection. Plagiarism detection is related to finding similar documents, and some work has already been done on evaluating the similarity of generated programs. Yu et al. (2023) uses two methods, JPlag (Prechelt and Malpohl, 2003) and Dolos (Maertens et al., 2022) to calculate similarity scores between programs. Using the maximum score from the two methods, they determined any two programs with a similarity score greater than or equal to 0.5 to be potential plagiarism. Here, we only use Dolos, due to JPlag’s restrictive license.

heading: 6 Discussions
text: Measuring contamination on the outputs. For a generation task, both the input and output are a sequence of tokens, allowing for the contamination of either the input sequence, the output sequence, or their coexistence within the training data to be measured. Given that language models only use the input as context and do not attempt to generate it, we believe it is unlikely that seeing the input during training would help them generate better code at test time. Instead we choose to measure data contamination of the outputs only. We believe that this is a reasonable and arguable the most effective strategy for code generation for the following reasons: 1) It is common to include the function name and signature in the task description for code generation tasks. After the model copies the function signature, having seen the function during training, it is easy for the model to reproduce the function body at test time. 2) Since programs are formal languages with strict grammar rules, it is easier to measure the semantic similarity between programs with different surface forms. In our work, we measure the semantic similarity between programs based on their abstract syntax trees. Such semantic comparison would be much harder and more prone to false-negatives for natural language (i.e., the input), as illustrated by recent work (Yang et al., 2023). 3) We find during our case study that while the models may struggle to reproduce gold programs that they have seen during training due to a misunderstanding of the natural language Fig. 7, there were only a few instances of this found. 4) While it is indeed the case that the model still needs to associate the natural language description with the code it has seen during training, our results (Tab. 1 and Fig. 1) suggest that there is a strong
correlation between output side contamination and model performance, suggesting that having seen the similar code outputs at test time does provide an unfair advantage to the models.
Suggestions for future work. The issue of contamination poses a significant challenge for the future evaluation of large language models’ capabilities. Our research suggests that acknowledging the potential for contamination is crucial, especially when utilizing datasets known to be affected, such as HumanEval and MBPP. When creating new datasets, while it is advised to follow our pipeline to decontaminate against the popular pretraining corpus (e.g., GitHub), our results also suggest that it is possible to decouple complexity and memorization by cross referencing the results from models pretrained with different corpus (i.e., results in Tab. 3). Beyond mere surface-level matching, we advocate for a more nuanced approach to decontamination that incorporates understanding the semantics of code when creating pretraining data and new LLMs. Furthermore, model developers should ideally include analyses on the effect of contamination on their evaluations. Given the increasing scale and complexity of state-of-the-art models and dataset used to train these models, developing evaluation benchmarks that are both relevant and completely unseen by models during training seems critical, yet increasingly challenging.

heading: 7 Limitations
text: False negative and positives. Due to the flexibility of programs, there can be multiple correct ways to solve a problem using Python. What we are searching for is only one possible solution presented as the gold solution, and as such we present our findings on the minimum number of questions to which models trained on the PILE and STARCODERDATA have been exposed to. Another source of false negatives (examples we falsely believe to be uncontaminated) is from the search itself. To reduce compute costs, we had to limit our search to relevant splits of the training corpus, which is the GitHub split for the PILE and the Python split for STARCODERDATA. It is possible for the models to see more of similar solutions in other parts of the pretraining corpus (e.g., similar programs in Java for STARCODERDATA). On the other hand, while performing semantic-level comparison helps with the general recall of similar programs, it is possible for it to flag a program as being similar to
the gold program despite being quite different, creating false positive example. We present the results for all programs found to be perfect matches to the gold program in § A.4, and some of examples show that false positives do exist. Different training stages. As mentioned in § 3.1, while we only study the “base” models and their pretraining data in this work, current models are typically trained in multiple stages, including supervised-finetuning, instruction-tuning and RLHF. However, the same methodology should be applicable to training data from these stages as well. At the same time, the size of such data is often several magnitudes smaller than the pretraining data, thus less likely to contain sources of contamination on the example-level. For example, an inspection on the instruction-tuning data for Octocoder (Muennighoff et al., 2023) shows that none of the MBPP examples has a similarity score above 80. Scarcity of open data. We only include two benchmarks and three models for this study. This is due to the scarcity of commonly used benchmarks that provide a gold program for every question, and models that have an open source training dataset. We hope more open models with open data will become available in the future to fuel further research in this domain. Inability to retrain model. Ideally, to observe the effect that having seen the answers to questions have on models, we would remove the answers from the training dataset, retrain the model, and compare with the original model. This approach is unfortunately prohibitively expensive given our compute constraints.

heading: 8 Conclusion
text: In this work, we quantify the data contamination issues for two popular code generation benchmarks, namely MBPP and HumanEval. We use both surface-level and semantic similarity to exhaustively search among the pretraining data using gold solutions for these benchmarks. By studying three series of models trained on two different corpus, the PILE and STARCODERDATA, we find that significant portions of the benchmarks have solutions leaked into the pretraining data. Further analysis shows that models perform much better when similar solutions are seen during training, and such correlation is independent of the difficult and length of the problems.

heading: Acknowledgements
text: The authors would like to thank Arjun Guha, Rui Shen, Yilun Zhao and the late Dragomir Radev for their help, suggestions, and comments for this project. We would also like to thank Hailey Schoelkopf for helping us with the PILE dataset as well as helpful discussions and feedback.

heading: A Additional Examples and Results
text: A.1 All Model Series
In Tab. 2 we present the results on the largest versions of the StarCoder-Base, Pythia and CodeGennl model series. In Tab. 4 we show results on four model versions from each model series.
A.2 Relevant Info for Models on the HumanEval Benchmark
We provide versions of Fig. 3 and Fig. 6 for the HumanEval benchmark in Fig. 11 and Fig. 12 respectively.
A.3 Examples of Similarity Scores
In Fig. 9 and Fig. 10 we provide examples of programs found within the training data and the relevant similarity score returned for them. A similarity score of 70 typically represents a program that is no longer similar to the gold program.
Prompt: Write a python function to count positive numbers in a list.
(a) Problem Description
1 def pos_count(list): 2 pos_count= 0 3 for num in list: 4 if num >= 0: 5 pos_count += 1 6 return pos_count
(b) Gold Program
1 def num_sym_points(sv): 2 count = 0 3 for v in sv: 4 if v <= 0: 5 count += 1 6 7 return count
(c) Matched program in STARCODERDATA. Semanticlevel similarity score = 100
1 def get_degree(d): 2 count = 0 3 for key in d: 4 if d[key] > 0: 5 count += 1 6 return count
(d) Matched program in STARCODERDATA. Semanticlevel similarity score = 72.73
Figure 9: Examples of different programs and their corresponding Dolos scores when compared to a gold program from the MBPP benchmark.
Prompt: Write a function that takes two lists and returns true if they have at least one common element.
(a) Problem Description
1 def common_element(list1, list2): 2 result = False 3 for x in list1: 4 for y in list2: 5 if x == y: 6 result = True 7 return result
(b) Gold Program
1 atches 2 3 def compare_lists(list1,list2): 4 result = False 5 for x in list1: 6 for y in list2: 7 if x == y: 8 result = True 9 return result
(c) Matched program in STARCODERDATA. Surfacelevel similarity score = 92
1 #def two_data(list1, list2): 2 # result = False 3 # for x in list2: 4 # if i == x: 5 # result = True 6 # return result 7 #print(two_data([3,4,
(d) Matched program in STARCODERDATA. Surfacelevel similarity score = 81
1 def top_ingredients(self, n): 2 res = {} 3 for a in self.items: 4 for i in a.ingredients: 5 try: 6 res[i] += 1 7 exce
(e) Matched program in STARCODERDATA. Surfacelevel similarity score = 70
Figure 10: Examples of different programs and their corresponding Levenshtein scores when compared to a gold program from the MBPP benchmark.
A.4 Perfect Matches We provide lists of every question that was seen 10 or more times during training for models trained on a specific dataset. These can be found in Tab. 5 through Tab. 13.

heading: Future Work
text: We hope more open models with open data will become available in the future to fuel further research in this domain. Inability to retrain model. Ideally, to observe the effect that having seen the answers to questions have on models, we would remove the answers from the training dataset, retrain the model, and compare with the original model. This approach is unfortunately prohibitively expensive given our compute constraints