To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination
Authors: Manley Roberts
Himanshu Thakur
Christine Herlihy
Colin White
Samuel Dooley
Venue: ICLR 2024 poster (2024)
Keywords: contamination
memorization
llm
codeforces
project euler
datasets
benchmarks
training cutoff
Primary area: datasets and benchmarks
TL;DR: We present the first thorough study of LLM data contamination on datasets released over time, showing that LLMs’ ability to solve coding problems changes dramatically as a function of metrics such as release date and GitHub popularity.

Abstract: Recent claims about the impressive abilities of large language models (LLMs) are often supported by evaluating publicly available benchmarks. 
Since LLMs train on wide swaths of the internet, this practice raises concerns of data contamination, i.e., evaluating on examples that are explicitly or implicitly included in the training data. 
Data contamination remains notoriously challenging to measure and mitigate, even with partial attempts like controlled experimentation of training data, canary strings, or embedding similarities. 
In this work, we conduct the first thorough longitudinal analysis of data contamination in LLMs by using the natural experiment of training cutoffs in GPT models to look at benchmarks released over time.
Specifically, we consider two code/mathematical problem-solving datasets, Codeforces and Project Euler, and find statistically significant trends among LLM pass rate vs. GitHub popularity and release date that provide strong evidence of contamination. 
By open-sourcing our dataset, raw results, and evaluation framework, our work paves the way for rigorous analyses of data contamination in modern models. We conclude with a discussion of best practices and future steps for publicly releasing benchmark in the age of LLMs which  train on webscale data.

TO THE CUTOFF... AND BEYOND? A LONGITUDINAL PERSPECTIVE ON LLM DATA CONTAMINATION
Manley Roberts', Himanshu Thakur!?2 Christine Herliby", Colin White', Samuel Dooley'
Abacus.AA CCannegie Mellon University Uniieesity of Maryland (manley, colin, samuel] @abaaus................aa. aai; hthakur@andrew. cmu edu; cherlihy@ uudd/mmmdeeee edu
ABSTRACT
Recent claims about the impressive abilities of large language models (LL.Ms) are often supported by evaluating publicly available benchmarks, Since LLMs train wide swaths of the internet, this practice raises concerns of data contamination, i.e., evaluating on examples that are intentionally or unintentionally included the training data. Data contamination remains notoriously challenging to measure and mitigate, even with partial attempts like controlled experimentation of training data, canary strings, or embedding similarities. In this work, we conduct the first thorough longitudinal analysis of data contamination in LLMs by using the natural experiment training cutoffs in GPT models to look at benchmarks released over time. Specifically, we consider two code/mathematice. Codeforces and Project Fuler, and we find statistically significant trends among LLM pass rate vs. GitHub popularity and release date that provide strong evidence contamination. By open-sourcing our dataset, raw results, and evaluation framework, our work paves the way for rigorous analyses of data contamination in modern models. We conclude with a discussion of best practices and future steps for publicly releasing benchmark in the age of LLMs which train webscale data.
mooblem-solving datasets,
INTRODUCTION
Progress in machine learning has historically been driven by the use of benchmark datasets (Raji et al. 2021) to demonstrate and ultimately improve model performance. In recent years, as large language models (LLMs) have risen to prominence, these benchmarks are used to claim impressive capabiliitess across a wide range of tasks (Brown et al.) 2020a),, such as open-ended text and code generation. However, it has become increasingly clear that evaluating on these benchmarks jeopardizes to accurately compare and assess modern models since static, open-source benchmarks are generally published the internet, and most modern LLMs incorporate internet text in their training data.
ability
There are two main phenomena to be concerned with. The first is contamination, which refers to an LLM's exposure, during training, examples that are similar identical to the examples that the model will later be evaluated on. The second is memorization, which can be understood as a property of model that permits extraction of generated outputs that are exact or near-exact replicas of examples seen during training. Both phenomena can pose security and privacy risks (Carlini et al., 2021). Additionally, as we discuss below, they can upwardly bias model performance estimates, obtuscating our ability to compare models and attribute performance gains to true model improvements.
Despite these concerns, contamination and memorization remain deceptively challenging defini- tively measure and detect. While some researchers have used futthing algorithms to compare test to training datasets (Radford et al. 2019 Brown al. (020b), many popular LLMSs full train- ing dataset details are not publicly available (OpenAl, 2023a Rozière et al., 2023). Additionally. string-matching produces false negatives when slight variations exist in data between train and test OOpenAl 2023a). Even concerted efforts to prevent any model from training a benchmark can fail. For example, the canary strings present in all BIG-Bench files (bench authors 2023), which designed to be checked and excluded by model trainers, were not sufficient to keep BIG-bench out of GPT-4's training corpus (OpenAl 2023a), partly because the success of this strategy relies on the awareness and compliance of model trainers in the absence of enforcement mechanism.

---

Recent works that look for contamination memorization focus popular benchmarks. They use controlled experimentation on models trained with certain subsets of chosen datasets, recognizing the value of comparing performance on examples that are seen not seen during training (Magar & Schwartz 2022 Zhang et al. 2021).
In contrast, take experimental economics view and naturally occurring experiment- i.e., the training cut-off date-to to assess contamination and memorization. We exploit the known training cutoff dates of GPT-4 and GPT-3.5-Turbo (OpenAl 2023a b) and assumed cutoff date of Code Bison (Google 2023) to naturally partition benchmark examples into subsets that have either probably been seen (pre-cutoff) or have probably not been seen (post-cutoff). We focus our analysis on longitudinal benchmarks consisting of problems released over period of time which bridges the cutoff.
In particular, we analyze Codeforces and Project Euler, two longitudinal code generation/problem solving websites. These websites have steadily released problems since 2010 and 2001, respectively. Informal analyses have shown that there are large drops in success rates of GPT-4 when evaluated on older versus more recent problems from Codeforces (He) 2023 Cundy 2023).
We build upon these insights by conducting the first rigorous, large-scale, longitudinal analysis of contamination and memorization in code generation and problem-solving benchmarks. To the best of our knowledge, we are the first to exploit the longitudinal nature of the benchmarks we analyze, along with the known training cutoff dates of the open and closed sourced models, naturally identify examples that the LLMs are likely/unlikely to have been exposed to during training, and use this partition to compare LLM performance during the pre- and post-cutoff periods.
Our contributions In this work, we explore contamination and memorization through the lens of time. Our core contributions include:
(i) The first segeescaae, longitudinal analysis of contamination and memorization using naturally occurring experiment - a novel methodology in LLM contamination which is important in light of closed-source models;
(ii) Empirical findings demonstrating that GPT-4 was likely exposed to Codeforces and Project Euler, due statistically significant positive association we observe between a problem's presence on GitHub and each LLM's test case pass rate only for problems released before the GPT training cutoff;
(iii) Code required to construct our longitudinal datasets and perform analyses, which we open-source
RELATED WORK
Evaluation of Code Generation Models Code generation models are generative models that try produce valid code given an input of some representation of the programmatic behavior, mathematical function, and/or computational task that the user would like to obtain. Modern code generation models include general models such as GPT family (OpenAI, 2023a). Llama (Rozière) et al. 2023), or PaLM (Chowdhery et al. (2022), as well as a variety of task-specific code models: AlphaCode (Li et a (2022). CodeGen (Nijkamp al. (022), Code-Llama (Rozière et al., [003). PaL.MCoddrr (Chowdhery et al.) 2022) Relevant code generation benchmarks include small sets of entirely handwritten problems (Chen et al. 202T Nijkamp et al. (022) as well as larger collections curated from internet sources such as code interview sites. competitive programming forums, or
general open source code (Hendrycks et al.) 2021 Austin et al., 202T Zan et al. 2022 Huang
lt al. 2022), and some that include both original and online-sourced problems (Yin et al. 2022
Li et al. 2022). Code interview, practice, or competition sites, offering problem descriptions and
programmatic evaluation, are common choices assess modern LLM capabilities (Nguyen & Nadi
2022; Zhang et al., 2023 He| 2023 Cundy] 2023) and indeed some public benchmarks feature these problems (Hendrycks al.
2022
GGPT4 acknowledges training with some small amount of data beyond its cutoff (OpenAl 2023a), so post-cutoff examples may still appear. GPT-3.--Turbo, subject to similar reinforcement learning with human feedback (RIHF) as GPT-4 (OpenAl 2023a), may have seen data beyond its cutoff as well.
"Our treatment of datasets and our evaluation framework are available at https:: /tt///gittul con./ abacusai /ttt tho-tcoofffff We release code and dataset contents to the extent possible while respecting the licensing requirements of the dataset owners.
2

---

To asses the validity of solutions, many of these benchmarks include test cases. They use 'functional correctness' metric based on passing these cases as the primary way to measure code generation performance: evaluating with complexityyundersttddbbbbll
metrics (gguyen & Nadi (022) is less common. Kulal et al. (2019); Chen et al. (202T) employ the pass@k k metric, describing the likelihood at least one among i sampled generations will pass all test cases. The benefit of these metrics is the complete independence from either expensive human feedback or inherently constraining -artarrtt--t--ruud-------e-
NLP metrics (Papineni et al. 2002 Lin 2004), which are often ineffective for code (Tran et al. 2019). These metrics are in contrast to other popular LLM performance metrics like perplexity (Kirchenbauer et al. 2023; Jain et al. (002)) or information retrieval based LLM
metrics of accuracy (Kwiatkowski et al.. 2019 9
Pal et al. 2023).
Adversarial Filtering and Adaptive Benchmarks in NLP Test-time exploitation of knowledge gained via contamination or memorization can be seen as special cases of a more general phe- nomenon in which language models appear to exhibit sophisticated reasoning capabilities but in fact exploiting shallower heuristics, with potentially negative consequences for generalizabil- ity (Bender al., (202I). Prior work has demonstrated that domain-agnostic and domain -speini-pecificc crowd worker-construcee natural language inference (NLI) datasets-i.e. i.e. SNLI (Bowman et al.
2015), MultiNLI (Williams et al.) 2018), MedNLI (Romanov & Shivade 2018) contain spurious correlations between lexical and syntactic features of the inputs and the corresponding class labels, such that hypothesis-only baselines (i.e., without premise) are able to outperform majority-clane
baselines (Poliak et al.) 2018 Gururangan et al. 2018 McCoy et al. 2019 Herlihy & Rudinger 202T). Researchers have proposed a variety of detection and mitigation strategies, including (T) adversarial filtering, in which an ensemble of classifiers
used to iteratively partition dataset into easy and hard subsets (Zellers et al., 2018); (2) introduction of stochasticity to the annotator prompting process via randomly selected anchor words (Sakaguchi et al. (2000); and (3) calls for the development of adversarially adaptive rather than static benchmarks (Zellers et al. 2019).
Memorization and Contamination in LLMs Many recent works have highlighted the security, privacy, and generalizability risks of memorization and contamination during LLM training and fine-tuning, while simultaneously proposing methods for detection and risk mitigation. Mireshghallah jet al. ) Biderman et al. (2023); Carlini et al. (2023); Magar & Schwartz (2022) investigate the training dynamics of
seeking scaling laws, early indications, and understanding of when and how memorization occurs in training. Carlini et al. (202I) famously extract hundreds of verbatim train examples from GPT-2. Ippolito et al. (2023) propose inference-time tricks to prevent regurgitation of examples, and Jacovi et al. (2023); Karmakar et al. (2022) give
best practices to avoid benchmark contamination. Carlini et al. (2021) 2023); Lee al. (2022);
Kandpal et al. (2022); Magar& & Schwartz (2022); Carlini et al. (2019) investigate the relationship between duplicated training data and aeeemorraatio/coontannnnn
(in particular, Carlini et al. (2019)
uses artificially introduced "canary" artifacts to track memorization). Nori et al. (2023) proposes distance-based metrics to assess memorization. Several works (Magar & Schwartz 2022 Zhang et al.)
2021) evaluate the impact of performance on examples seen vs. not seen during training; we will use a variation of this strategy.
by estimating the difference in test-time
Dodge et al. ))21) conduct case study of the webcrawl corpus C4, including contamination investigation, while others (Aiyappa et al. 2023] Chang et al., 2023] Golchin & Surdeanu (023) conduct studies on the contamination of GPT models directly. Karmakar et al. (2022) dive deep into Hackerrank (an interview prep coding platform) contamination in the Codex model by not only assessing pass rates on full problems but also partial problem snippets. Golchin & Surdeanu (2023), a recent work, focuses in particular on comparing the result of prompting for memorized completion with without benchmark clues and concludes that GPT-4 has been contaminated with several standard datasets; our analysis finds more contamination of GPT-4, but differs by examining longitudinal datasets in order to view the effects of dataset portions before and after training cutoffs.
3 DATASET CONSTRUCTION
Many open-source benchmarks (Chen et al. 2021) designed to evaluate code generation are released at a certain point in time, evaluated on a number of models along with release, and then deployed repeatedly time goes on in order to evaluate new models" performance on the benchmark. For
3

---

model with strict temporal training dataset cutoff, these benchmarks exist either firmly within or outside of the training data, meaning that to evaluate the effect of the cutoff, we must compare between multiple datasets (which, clearly, might have many differences beyond their release dates).
For this analysis, we concern ourselves with datasets with hand-written original problems that are released at intervals over long stretch of time. In particular, require that substantial number of cutoffs in September 2021, that
problems are produced before and after the GPT-4/GPT-3.5- the bulk of problems are of a format and size sufficient for prompting modern LLMs, and that there exists an automated objective measure of correctness for evaluation. We focus on problems from the competitive programming website Codeforces (problems from 2010 2023) (Mirzayanov 2023 and from the mathematical programming puzzle website Project Euler (problems from 2001 (003))
(Hughes) 2023), building off analyses from (Cundy 2023
He 2023
Codeforces Codeforces a website that hosts competitive programming competitions. Problems are released in small batches corresponding to particular round, and competitors submit solutions against test cases, competing to produce the highest overall score by giving fast solutions. After a competition ends, each competitor's 's solutions available online, well as the test cases that were evaluated on each problem (which take the form of an input file and expected output).
For each problem, we collect metadata, problem text (processed to clear some HTML artifacts), and input/expected output text for public and private test cases. We forgo the compute-intensive intensive procedure of generating additional test cases for problems which was used by (Li et al., 2022) and omit test cases in which either the given input or output on the Codeforces platform end with "...". ", as this often indicates that the text is too long and has been abridged. We provide additional details of the Codeforces problem set and our collection process in Appendix A.2
Project Euler Project Euler is website that hosts difficult math problems with a string answer that is usually single number (integral real). The recommended way to solve these problems is to write code that will generate the answer. The answer itself can be submitted on the site and compared directly to the private solution (there are no official public solutions). There are no test cases except comparison with the true answer.
We collect Project Euler problems through a combination of their metadata API and direct scraping of problem pages. We collect problems through 845 (released May 2023) and use open-source solutions from (luckytoilet (023). These solutions were collected in September 2023, but there are a few recent problems through 845 without solution from this source; these omit.
METHODOLOGICAL APPROACH
The primary research questions we endeavor to explore through longitudinal analysis of pre- versus post-cutoff LLM performance include: (1) Does there exist statistically significant relationship between programming problem's 's frequency of presence in open-source GitHub repositories and an LLM's ability to generate a functionally correct solution to that problem, and/or reproduce portions of its metadata, such as the problem title or tags? (2) How or to what extent is this relationship mediated by a problem's 's reported difficulty? (3) Most critically- how or to what extent do (1) and (2) change depending on whether problem was released before versus after the LLM's training date cutoff?
Models To answer these questions, we conduct analysis on output produced by GPT-4, GPT- T.rboo Davinci-002, Google's code-bison, and Meta's Code-Llama. The specific models used are gpt -0----01144 - tarbtturrbb00 text ddvvnnci---i-i 1aeei-0000 002, code bendeebbssoe b C01, and codellama aadeelllaa//odeLaane CoomeLama 34b- Instruct -ff.
Independent Variables
To begin, we define the following set of independent variables (IVs):
Gitllub Presence is proxy metric intended to capture the frequency with which problem is publicly available on GitHub (similar to public Google and Bing API search used by [Chang et al. (2023) a proxy for online presence of books). For simplicity, it searches only for mention of the problem's 's name and ID. To compute GitHub Presence, we begin by collecting all public repositories that contain mentions of the benchmark dataset of interest (i.e., Codeforces or Project
4

---

Euler) as of our collection date. Then, for each problem of interest in given dataset, we filter the dataset repositories and retain the subset containing substring(s) that correspond to the problem's s title. We are then able to approximately compute the number of times problem p occurs as: Lim1 -daatavee repusi c(p, i) Vp E {dataset problems), where c(p, i) is the number of matches within repo i's concatenated text any one of number of format variations of p's ID or title. Counting multiple occurrences within the same repo offers benefits such as a more gramular analysis in the event of mega-repos that might store multiple solutions to the same problem, and it is therefore in our eyes closer proxy to the true frequency of the problem in the training data.
Difficulty intuitively captures how challenging problem is for humans solve. Both Codeforces and Project Euler report difficulty scores as part of problem metadata.
Problem released post-cutoff is Boolean variable to indicate whether given problem was released (i.e., published by the dataset owners) before (0) or after (1) the training date cutoff for a given LLM.
Dependent Variables We consider the following set of dependent variables (DVs):
Problem-level pass rate (pass rate) We assume that in the general case, a given problem can be mapped to some number, ny 2 of test cases (either public private). For code generation tasks, the question-level pass rate can then be computed as the fraction of test cases for which the code generated by the LLM produces functionally correct solution-i.e., 7ip Σn= ((X(LM(()) = w))
where À represents calling the LLM's generated code and y; represents the ground-truth output for problem o's 7th test case. The special case where we ask the LLM to generate (only) the solution rather than code can be represented by omitting the À call in the above expression. We use code generation Codeforces and solution-only generation Project Euler. See Appendix A. for discussion of alternative metrics.
Title reproduction In each of the datasets we consider, each problem has title. To compute title reproduction for a given problem p. we provide as input the dataset name and problem ID, ask the LLM to generate the problem's title given this input, and evaluate the similarity between the generated string, titlep, and y's ground-truth title by mapping the title into bag of tokens and modeling the retrieval of each token as separate observation in logistic regression. We include this DV as probe for possible memorization.
Tag reproduction Among the datasets we consider, only Codeforces problems contain descriptive tags. Each tag is an n-gram that describes the intended approach or content of a problem, as well as metadata like the difficulty. For example, Problem 500A has tags "dfs and similar", "graphs", "implementation". and "* 1000". For a given problem, p. we provide the problem's 's title and ID as input to the LLM, and ask it to produce a set of candidate tags. We evaluate token-level recall with respect to the tokenized version of the problem's 's ground-truth tag(s). Much like title reproduction, this DV is included
memorization probe.
To answer the aforementioned research questions for each dataset and dependent variable, we conduct regression analyses with problem-level performance as our unit of analysis, of the form:
DV ~ (Difficulty + GitHub Presence) postCutoff
Because the problem-level pass rate prediction task involves count data, we specifically formalize it as binomial regression, such that for given problem, p with a corresponding number of (public + private| test cases, Пp, we seek to predict the number of successes-ine -ic.. the number of test cases, out of up trials that the LLM's generated code and/or numeric solution will pass. In title reproduction, the outcome of interest is binary-i.ee the LLM either does or does not successfully reproduce the problem's i title; as such, we model this task using logistic regression. For tag reproduction, while problem's tags can be set-valued, we tokenize the string of tags and evaluate the recall of each token independently; as such, this task is also modeled using logistic regression. more detailed description of our modeling choices, along with interpretation guidance for the regression tables and marginal effects plots, can be found in Appendix B.I
In the regression tables, we report coefficients odds ratios where values equal to I indicate no impact of the variable on the Pass Rate. Coefficients greater than indicate a positive impact and those less than 1 indicate negative impact. For example, an odds ratio coefficient of 1.352 would correspond to a 35.2% increase in the dependent variable associated with a unit increase in the independent variable.
5

---

Functional Correctness Marginal Effects Plots for GPT on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
Functional Conrectness
50%
log(Difficulty)
7.04
40%
7.43
7.83
30%
20%
10%
10
15
10
15
log(GitHub Presence)
Figure Marginal Effects of Pass Rate Metric for GPT-4 on the Codeforces Dataset. Observe positive association between GitHub Presence before the cutoff but not after. Also, there is negative association between Difficulty and pass rate both before and after the cutoff.
5 RESULTS
Overall, we strong trends that the performance of each model changes after the training cutoff. These changes often highlight that there is positive association between the presence of questions on GitHub and the performance of the model; however, after the training cutoff, this association disappears. We provide examples of the generations of the LLMs in Appendix [B.]] for a qualitative inspection of the results. We note that, while did test the code generation performance of the open source models rext-davinci-002 and codellama/Code -al--aaam-a--ttteeeeaa correctness performance too low yield meaningful analysis. Thus, we omit these models from all analyses in the main paper, but refer the reader to Appendix B.6
these models' functional
5.1 PASS RATE
GitHub Presence First, we look at the performance of Pass Rate on the benchmark Codeforces, where report marginal effect plots for GPT-4 in Figure [ GPT-3.5-1 3-rboo and Code Bison are qualitatively similar and can be found in Appendix Figures 12 and 14 We report regression coefficients for all models on Codeforces in Figure 2 On the Project Euler benchmark,
report
marginal effect plots in Appendix Figures and 30 and regression coefficients in Figure 3 Note that Project Fuler
much smaller benchmark with just 73 problems included after the GPT training cutoff date in September 2021. Additionally, none of the LLMs we tested got any of the questions correct for this set of 73 problems. We make several observations.
Most strikingly, we see that the effect of the GitHub Presence variable is significant before the training cut-off and is not significant after the training cutoff. For GPT-4, we observe that for each increase in one unit of the log of GitHub Presence, we see the odds ratio increase by 4.5% on Codeforces and 47.8% on Project Euler; for G.T-3.5-Turbb, that value is moderated slightly to 2.5% on Codeforees and 27.7% on Project Fuler; for Code Bison we see the odds ratio increase by 3.1 %. However, we see no statistically significant association between GitHub Presence and GPT model performance for those problems which appeared online after the training cutoff in September 2021.
This post-cutoff performance degradation provides evidence of contamination and/or memorization of pre-cutoff problems from Codeforces and Project Euler by GPT-3.5-Turke and GPT-4. For the most part, the odds ratios are similar in terms of the direction and magnitude of their effects on the pass rate odds for each LLM. Two points of distinction include: (1) GPT-4 performs better across the board, as evidenced by higher odds of functional correctness for all difficulty levels in both the pre- and post-cutoff periods compared to GPT-3.5-Turke (2) For Codeforces, the odds ratio for GitHub

---

Regression Coefficients for Codeforces Pass Rate
GPT4
GPT3.5
Code Bison
Drahy
GIHLD Presence
Pre
GPT4 Post
GPT3.5 Pre
GPT3.5 Post
Bison
Bison Post
p-level
n.s.
Figure 2: Regression coefficients for Pass Rate of GPT4, GPT-3.5-Turbo and Code Bison on the Codeforces dataset. Observe that the odds ratios for both Difficu fufly and Git Presence are statistically significantly moderated between the before and after cutoffs for both models. See Table I and for regression coefficients.
presence is equal to 1 and is not statistically significant during the post-cutoff period for GPT-4, but (i.e., associated with increased odds of Y) and statistically significant for a = 0.1 during the same period for GPT-3.5-Turbo (see Table and While training details for GPT-family models are generally secret, we propose as a possible explanation that GPT-3.5- -urbbo may have had higher train/finetune exposure to problems released after the cutoff date than GPT-4
worth pointing out that the relationships between Github Presence and pass rate have non- overlapping confidence intervals (before and after cutoff) for only GPT-4. This lends itself to conclusion that memorization occurred for this model.
Code Bison's 's analysis cutoff, there is a positive association between Github Presence and pass rate, while after there is no association. This analysis does produce very large confidence band the post-cutoff problems (due to the small sample size of Codeforces problems collected between Feb. 2023 and June 2023), making definitive conclusions difficult to resolve.
different cutoff (February 2023). On all of the data before this
Regression Coefficients for Project Euler GPT4
Pass Rate
GPT3.5
Dificuty
GiHbbb Presence
1.5
2.0 0.0
0.5
GPT4 Pre
GPT4 Post
Odds Ratios
GPT355 Pre
6P1355 Post
p-level
Figure 3: Regression coefficients plots of Pass Rate for GPT-4 and GPT-3.5-Turbo the Project Euler Dataset. See Table 10 and I1 for regression coefficients. No problems pass after the cutoff.
3 As mentioned in Section 1 GPT-4 is known to have some post-cutoff events included in its training; since HPP-3.55-Trrhe uses similar RLHF procedure (OpenAl 2023a). it's possible it has been exposed well- publicly unknown extent.
7

---

Difficulty When examine results for GPT-4, GPT-3.5-Turbo, and Code Bison on Codeforces (see Tables 1 3. we see that there always exists a statistically significant, negative association between Difficulty and pass rate for each LLM-i.e. this relationship is observed in both the pre- and post-cutoff periods. However, while each model's post-cutoff Di ff culty coefficient is < 1, indicating decrease in the odds of pass rate these coefficients are statistically significantly larger than their corresponding pre-cutoff values, suggesting a moderation in the still-negative relationship between Di Difficulty and pass rate.
On the one hand, we can interpret the persistence of this relationship as evidence that the LLMs' inductive biases, while perhaps influenced by the effects of contamination and memorization, are by no means solely determined by such artifacts. For this reason, we do not see equal (or perhaps, equally poor) performance across problem difficulty levels in the post-period, but instead see that L.MM pass rates vary in accordance with difficulty even in the (hypothetical) absence of contamination, much as they do for human programmers.
Other possible contributing factors include: (I) variation in the number of test cases by difficulty level, and/or over time; (2) more limited, but non-zero amounts of contamination or memorization of the datasets we analyze; and (3) the presence of unobserved confounder(s) influencing change in both problem difficulty and LLM pass rate over time. We test (1) by fitting a regression model to examine whether Diffi culty is able to predict the number of observed test cases after the cutoff, but do not find Di ffi i cu 1 ty to have predictive power. Hypothesis (2) could be occurring, particularly given the acknowledged GPT fine-tuning (OpenAT, 2023a): however, it is unlikely this is happening at high enough levels to be sufficient cause of observed behavior. We view the identification of possible confounders as a promising direction for future work.
5.2
TITLE AND TAG REPRODUCTION
For title reproduction, we show regression tables in Appendix Tables 12 15 and Appendix Figures 32
We conclude that across all models, there is no impact of GitHub Presence on the ability of the LLMs to reproduce the title, both before and after the training cutoffs.
38
For tag reproduction, we find that there is a negative association between GitHub Presence and the ability of the LLMS to reproduce the tag labels Codeforces (there are tags associated with Project Euler). In Figure 42 Appendix Figure 40 and Appendix Tables T6) we can see that across the board, there is negative association between Difficulty and tag reproduction performance before the cutoff but there is no association after the cutoff. As the regression results demonstrate, the negative association moderates after the cutoff, dropping from a decrease of 56.9% to 17.4% in odds ratios from before to after the cutoff for GPT- and from 50.3% to 26.1 1% for GPT-335-Turbb.
The way in which Codeforces problems are available online is one hypothesis as to why tags reproduction is inversely related to GitHub presence, whereas title reproduction is not. Tags are metadata for Codeforces problems which are not present in the main problem description. As such, the tags may be less likely to be copied and pasted throughout the internet. Thus, it is possible that those tags themselves undergo some interesting distribution shift which could explain their inverse relationship with presence on GitHub.
5.3 ANALYSIS ABLATIONS
Public vs Private Test Cases As discussed in Section 3 the Codeforces problems contain both public and private test cases. Public cases are readily available on the problem's 's page whereas the private cases can only be found by opening an accepted submission on the Codeforces platform. Above, we analyzed the pass rate of each problem on all collected test cases. Now, we break these out by public and private test cases to investigate any different trends between the two sets. We consider only the private test cases in Figures 22 24 and Tables
S
whereas only the public test cases in
Figures 16 18 and Tables 4
We see first that the two main trends we observed above hold, indicating the robustness of the conclusions: contamination is likely since GitHub Presence is positively correlated with pass rate only before the cutoff, and Di Difficulty has a negative association with pass rate. However, observe non-overlapping confidence intervals only for GPT-4 private test cases.
8

---

However, we also observe, unexpectedly, that the pass rate after the cutoff is higher for the private test cases than for the public test cases. This observation contrasts the typical perspective on Codeforces which considers the public test cases to be simpler toy cases used as examples while coding whereas the private cases are more thorough checks for correct behavior. To explain the discrepancy, we hypothesize that this behavior may be related the private test cases after the cutoff being, on average, easier to answer than public test cases after the cutoff. There is no per-case difficulty score available on Codeforces, but we can consider simple heuristic: shorter inputs are simpler to answer, and longer inputs are harder. Why might this effect be most noticeable after the cutoff? To answer, we observe that while the median test case input string lengths for our public and private pre-cutoff test cases are similar, at 18 and 21 characters, respectively, the median input lengths after the cutoff diverge for public and private test cases: 38 for public and 27 for private. Further investigation into the causes and consequences of this shift is promising direction for future work.
Covariate Shift We detail how we assess whether the performance degradation that we observe for problems released after the training cutoff might be caused by covariate shifts in the questions present in Codeforces and Project Euler. More precisely, we examine the distribution over tags and/or difficulty level, and we look for statistically significant changes in their prevalence during the post-cutoff period, relative to the pre-cutoff period. We visually inspect the distribution over tags (for Codeforces) and over discretized difficulty scores (for both datasets) for problems released during the pre- post- periods, and do not find evidence of qualitative differences. We then conduct x² tests using the pre-cutoff normalized counts as the reference distribution. We do not find any statistically significant difference in any of the pre- versus post-distributions that we analyze. Plots and detailed statistical results are available in Appendix
6 DISCUSSION
Utility of longitudinal analysis: We provide a novel methodology for examining data contamination in LLMs, borrowed from experimental economics where we observe phenomena by exploiting natu- rally occurring changes. Thus, we present novel way to approximately validate claims made about training date cutoffs for black box LLMs, and/or exposure (or lack thereof) to a given dataset during training or fine-tuning, provided that the dataset in question contains instances on each side of the model's 's reported cutoff. This can be valuable in cases where specific training details are not public, and/or when contamination or memorization is suspected as a root cause of performance degradation when the model in question is evaluated on newer problems. It is important to ack acknowledge that limi- tations also exist- for example, we cannot rule out the presence of latent confounder(s) influencing both exposure (i.e., to a given subset of problems) and LLM performance on those problems.
Our findings in Section illustrate the extent to which even high-quality, manually constructed benchmarks can be expected to enjoy ever-shorter shelf lives in the era of LLMs, as newer models with updated training cutoff dates will iteratively render existing benchmarks stale. Detection of memorization and contamination will likely remain challenging in the general case, as many popular benchmarks have been released all at once rather than over time, and as such, cannot be subjected longitudinal analyses like the ones perform. Additionally, in open-ended domains such as code generation, we may fail to detect instances where the model has been exposed to solution(s) for given problem when the problem context itself is missing or latent (i.e., by training on public repositories where people may not reproduce the questions their code is intended answer/solve).
Implications for LLM evaluation:
Current mitigation options, including the use of private (i.e., offline, closed-source) benchmarks, and/or benchmarks known be constructed after the training cutoff for evaluation target(s) of interest, are likely to be time-bound in their utility and may be cost-prohibitive to sustain over longer time horizons, in the absence of exogenous shocks to the pace of LLM release cycles. Additionally, reliance on private benchmarks may further erode transparency and lead to duplication of efforts as challenging examples are detected and addressed in a more siloed fashion. Thus, while the need for some set of open-source "goalposts" against which to measure progress, evaluate, and compare LLMs is likely to persist, the way in which we construct, release, and evaluate against benchmark datasets will need to become more dynamic. We urge the community to move away from static benchmarks released in a single time step and toward continuous integration-styy style staggered release and evaluation cycles.

---

REFERENCES
Rachith Aiyappa, Jisun An, Haewoon Kwak, and Kong-Yeol Ahn. Can we trust the evaluation on chatgpt?, 2023.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program Synthesis with Large Language Models, 2021.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2023.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Majjo and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAceT '21, pp. 610-623, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 978 9781450383097. doi: 10. 11111114444222 01145/3442118.4445559999 URL https::p/.. http://oii.oooo 1tt.:/.../......1///1111111..110 1011111443111111..1. 3445922
Stella Biderman, USVSN Sai Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu Purohit, and Edward Raff. Emergent and predictable memorization in large language models, 2023.
Samuel R. Bowman, Gabor Angeli. Christopher Potts, and Christopher D. Manning. large annotated corpus for learning natural language inference, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020a. URL
www.:ee.nnrrrceceee.e..... cc/paper http:///ppppeeeeeeeeeeeeeeeeeeee files//pperr2200/ pdff
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020b.
Ethan Caballero, OpenAA, and Ilya Sutskever. Description2Code Dataset, 8 2016. URL https http:hhww..com/ttahaacccceeeeeee http://aaaaaaeeeeeeeeeeeeeeeeeee hwtp://scrept..oo//e/ee.....
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song. Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2023.
Kent K. Chang, Mackenzie Cramer, Sandcep Soni, and David Bamman. Speak, memory: An archaeology of books known to chatgpt/gpt-4, 2023.
Matt Chaput. Whoosh. http::://hoosh..raatteeoooss accessed 4-0ctober-2023].
2012. [Online;
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Knueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan,
10

---

Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:22040033/1, 2022.
Chris Cundy, 4pt-4 memorizes project euler numerical solutions.
https: cundy me/post/ so ut utnoss/ 2023. [Online; ac-
ggt__ m_memori zes project euler cessed 26-September-20231
Bouler_numrrr numer cal
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1286-1300 1305, Online and Punta Cana, Dominican Republic, main.
November 2021. Association for Computational Linguistics. doi: wrt.oorg//200 2021. emn emnnp-mminn...... 98
98. URL https:
Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models, 2023.
Google. feedbackavailable models in generative ai studio.
https://cloud. llo.///oooo...olleeeee.
wetpreeeaa//cccc/ggeaaaaeeaaaaaa September-2023].
2023. [Online; accessed 28-
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume 2 (Shori Papers), pp. 107-1 12, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: (0.18653/v1/N18-200 URL https
7/ w/www. ac weeb....... org/ http//nnooooogo.. 2017
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel Smith, al. Array program- ming with numpy. Nature, 585(7825):357-336)) 2020.
Horace He. Tweet: I suspect gpt-4's performance is influenced by data contamination, at least on codeforces. of the easiest problems on codeforces, solved 10/10 pre-2021 problems and 0/10 recent problems. this strongly points to contamination. 1/4, March 2023.
Dan Hendryeks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.099:: 2021.
Christine Herlihy and Rachel Rudinger. MedNLI is not immune: Natural language inference artifacts in the clinical domain. Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 2: Short Papers), pp. 1020-1027, Online, August 2021, Asso- ciation for Computational Linguistics. doi: 118853/11/22211 1001 129. URL https ac1- http://ooooo..ooo/22...aa------a
//aclanthology.co wwtp:/yy..r./20212......2.
Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, Nan Duan, and Jianfeng Gao. Execution-based evaluation for data science code generation models. arXiv preprint a.X04:2211.09334, 2022.
Colin Hughes. Project euler. September-2023).
net 2023. [Online; accessed 26-

---

Daphne Ippolito. Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette- Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives false sense of privacy, 2023.
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint 17XXi:23305.01666 2023.
Neel Jain, Khalid Saifullah, Yuxin Wen, John Kirchenbauer, Manli Shu, Aniruddha Saha, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Bring your data! self-supervised evaluation for large language models. arXiv preprint arXiv:2306.13633 2023.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating training data mitigates privacy risks in language models. ArXiv, abs/2202.0653,, 2022. htthtts:: www.nt
http:/CopuSSD:246622002
Anjan Karmakar, Julian Aron Prenner, Marco D' Ambros, and Romain Robbes. Codex hacks hackerrank: Memorization issues and framework for code synthesis evaluation, 2022.
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein. A watermark for large language models. arXiv preprint arXiv:2301. 2023.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy Liang. Spoc: Search-basee pseudocode to code. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS). 32, 2019.
Tom Kwiat Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison- Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Dublin, Ireland, May 2022. Association for Computational
I: Long Papers), pp. Linguistics. doi: 10. 10.18665/1/2222..allonggggg. 2022. acl-long.557
URL https: /ttp::://aa..aattooloyy//
Kiho Lee. Chatgpt dan. accessed 4-Oetober-2023].
DAN 2023. [Online;
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' A'ttumee Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. arXiv preprint arXiv:2203.072 2022.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Edward Loper and Steven Bird. Nltk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
luckytoilet.
Projecteuler-ssluttinn.. projeccteuler-scltiii 2023.
httpe:://gthhab..cm//uukvyryrt/
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 157-165, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/w1/2022..oo -choottt.. URL https: http:////aaattloy..oooooo
2022. www.aclsshor....
12

---

Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428 3448, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18533/1//11-1133 1334. URL https: / www. ac web org/antho hotp:/l91yy//1////0/p
Wes McKinney et al. pandas: foundational python library for data analysis and statistics. Python for high performance and scientific computing. 14(9):1-9, 2011.
Fatemehsadat Mireshghallah, Archit Uniyal, Tianhao Wang, David Evans, and Taylor Berg- Kirkpatrick. An empirical analysis of memorization in fine-tuned autoregressive language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1816-1826, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: 10. 10.18663//12022eemmppp.m
URL https
emnlp mommlp-mminn1. main.
Mike Mirzayanov. Codeforces. bttps: com/ 2023. [Online; accessed 26- September-2023).
Nhan Nguyen and Sarah Nadi. An empirical evaluation of github copilot's code suggestions. In Proceedings of the 19th International Conference Mining Software Repositories, MSR '22, pp. 1-5, New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450393034, doi: 10.114555524442.352477777
URL /ttpttps:://o...oo wwtpi.orgg100.. 1000 112110.1144 1455 3524 842 3528470
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 medical challenge problems, 2023.
OpenAI. Gpt-4 technical report. Technical Report, 2023a.
OpenAl. Openai platform documentation: Models. ht testpp:/// http:://paatooooe wwwna/..pppe.i.....mmmm www.:ppppanicccccccoc.oo...s models 2023b. [Online; accessed S--Spteemeer-2223.
Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Siddartha Naidu. Giraffe: Adventures in expanding context lengths in Ilms. arXiv preprint arXiv:2308. 10882, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 31 311-318, 2002.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pp. 180-191, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.1865//1/S18-2000 URL https 7/ 7/www. www.:aawwwww..../aaae.eeoee http:///.oogatttooooo///----00
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Inioluwa Deborah Raji, Emily Denton, Emily M Bender, Alex Hanna, and Amandalynne Paullada. Ai and the everything in the whole wide world benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Alexey Romanov and Chaitanya Shivade. Lessons from natural language inference in the clinical do- main. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Process- ing. pp. 1586-1596. Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DI1 1187. URL httttsss:/acaaathllllloa
13

---

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXhv:2308./29:: 12950, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732 8740, Apr. 2020. doi: 10. 16.1160/aaai..... w3410 v34i03.6349999 63399. URL https
7/0]s. aaai wwwp/ogggg.....o.....
AAAAAAAAAAAA/a htt.a///eeteeeeeeee///eoo 6399
Ngoc Tran, Hieu Tran, Son Nguyen, Hoan Nguyen, and Tien Nguyen. Does bleu score work for code migration? In 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC), pp. 165-176. IEEE, 2019.
Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference, 2018.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.09244. 2022.
Daoguang Zan, Bei Chen, Dejian Yang. Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. Cert: Continual pre-tra pre-training on sketches for library-oriented code generation. arXiv preprint arXiv:2206.0000 06888, 2022.
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. Swag: A large-scale adversarial dataset for grounded commonsense inference, 2018.
Rowan Zellers. Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 4791-48001 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 1/19914444 URL https: /www aclweb. worg/oorr/aoo.o..... anthology/ P19-1472
Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. arXiv preprint 12. 2021.
Zizhuo Zhang, Lian Wen, Shaoyang Zhang, David Chen, and Yanfei Jiang. Evaluating gpt's programming capability through codewars' katas, 2023.
14

---

A ADDITIONAL METHODOLOGY
A.1 METRICS
We note that @ass@k (Chen et al. 2021 is an alternative outcome metric that is commonly reported within the LLM evaluation literature, which explores the number of sampled generations that pass every unit test of a problem. We can map our analyses onto Pass since we generate one sample from each LLM. However, we also look at the problem pass rate (defined above) since the number of unit tests for Codeforces can very large (sometimes over 500) and Pass Rate provides a more granular view of model performance than Pass@ 1. In this paper, Pass@ I is defined as the number of problems that pass all unit tests.
A.2 CODEFORCES DATASET COLLECTION
(Caballero al. 2016 Hendrycks et al. 2021 Li 2022) assembled, in their respective datasets, a portion of Codeforces problems and test cases through 2018 and 2021, respectively. However, only et al. 2022) contained any problems after the GPT-4/3.5 Turbo cutoff, and had so few problems after the cutoff that we needed to collect Codeforces problems ourselves in order to have enough post-cutoff problems. fact, we do not use their version of the problems at all, instead collecting a full set of problems from 2010-2023 ourselves, in order to ensure that the data quality is consistent for all problems. We do replicate many design decisions from [i et al. (2022). including separating the "public" test cases (those simpler test cases available in the problem description, fed to the model, and given during the competition to all competitors) from the "private" test cases (those test cases reserved in secret at the release of competition but made public after some time).
We collected every problem on the Codeforces site through Codeforces Contest 1840, also known as round 843 (which took place June 6, 2023), and removed problems for variety of reasons including: no test cases are found; testing is interactive and requires communicating with the Codeforces server; problem has English-language problem is special contest such as a private contest, a Q# quantum programming competition, or an April Fools' competition.
version; problem is given as PDF and not HTML text;
B ADDITIONAL EMPIRICAL RESULTS
B. REGRESSION DESCRIPTIONS
As opposed to other forms of regression, logistic regression uses odds rather than probabilities, and the main quantity explored an odds ratio. For event with probability p. the odds of that event is defined as p/ (1- - p). Odds ratios explain the relationship between independent variables (predictors or features) and the probability of the binary outcome. logistic regression, we use odds ratios to quantify the effect of one-unit change in an independent variable on the odds of the binary outcome. The odds ratio is defined as the ratio of the odds of a problem passing to the odds that the problem doesn't pass.
More formally, let Y be the binary outcome variable indicating failure/success of a question by an LLM where Y E {0, 1} and assume P(Y 1) p. Let X1...., Xx be set of predictor variables. Then the logistic regression of Y on X1..... Xn estimates parameters Bo...., 8n through maximum likelihood via the equation:
logit(p) In 1-p Bo 3,x, + B,
When we fit the regression, obtain estimates for the 8,s. These fitted 3r can be interpreted as coefficients to the regression equation and provide intuition for how the independent variables X; influence the independent variable. Specifically, a fitted value of 3 tells us that, keeping all other variables constant, one unit change in the variable X; yields a change of 3 in the log odds ratio of the independent variable. is also common to exponentiate the fitted coefficients, which case unit change in X; while holding the other dependent variables constant yields a e change in the odds ratio of the independent variable.
15

---

These e odds ratios that take values between 0 and oc. They provide insight into how change in the predictor variable X; affects the odds of the event occurring. e 1, it suggests that the predictor variable has no effect on the odds of Y. If e > 1, it suggests that an increase in X; associated with increased odds of the event happening, specifically by providing (1 e 5% increase in the odds of Y. If e > 1, it suggests that an increase in X; is associated with decreased odds of the event happening, specifically by providing (1 - )% decrease in the odds of Y.
In our analyses, we are primarily interested the independent variables GitHub Presence, Difficulty and indicator variable indicating whether problem released before after the training cutoff. Below, we report regression tables with estimated odds ratio coefficients as well marginal effects plots which visually depict the fitted regressions.
B.2
PASS RATE
In this section, we present the marginal effects on Pass Rate of GPT-4 and GPT-3.5-Turbo across the Codeforces and Project Euler benchmarks.
B.2.1
ALL CODEFORCES DATA
First, present the marginal effects in Figures T0 12 14 and regression coefficients in Tables I
3 with corresponding regression coefficient visualization in Figures 9
13
B.2.2 PUBLIC CODEFORCES DATA
Second, we evaluate only on the public test cases of Codeforces and produce the marginal effect
plots in Figures To 18 20 and regression coefficients in Tables 4 with corresponding regression
coefficient visualization in Figures T5
B.2.3
PRIVATE CODEFORCES DATA
Finally, we consider only the private test cases and get the marginal effect plots shown Figures 22 24 26 and regression coefficients in Tables 7 with corresponding regression coefficient
visualization in Figures 21 23
B.3 PROJECT EULER
Finally, we report pass rates Project Buler in Figures 28 and 30 We also present the regression coefficients in Tables 10 and I1 with corresponding regression coefficient visualization in Figures 27
29
B.4 TITLE REPRODUCTION
Here, we present the marginal effects the Title Reproduction metric for -4 and GPT-3.5-Turbo
across the Codeforces and Project Euler benchmarks in Figures 34 32
36
and 38 We also present
the regression coefficients in Tables 12
13
14 and 15 with corresponding regression coefficient
visualization in Figures 31 33 and
B.5
TAG REPRODUCTION
Here, we present the marginal effects on the Tag Reproduction metric for GPT-4 and GPT-3.5-Turee
across the Codeforces and Project Euler benchmarks in Figures 42 and 40 We also present the regression coefficients in Tables and 16 with corresponding regression coefficient visualization in Figures 39
41
B.6 EXPERIMENTS WITH ADDITIONAL. LLMS
The results in Sections and B.2 B.5 exclusively use the models GPT-3.5 and GPT-4. this section, discuss results Davinci-0022 and Code-IJama. Overall, motivation to focus GPT-3.5
16

---

and GPT-4 is because they are the only models to achieve nontrivial pass rate. For example, Codeforces, GPT-3.5 achieved 27% and 13% pass rates before and after the cutoff, and GPT-4 achieved 37% and 16% pass rates before and after the cutoff, but the pass rates for text-davinci- 002 with a comparable prompting strategy are both less than 1 1%. Similarly, a partial analysis of Lawea//a//Coeeeaaa44aaaaa3 Instruct-h featuring 3400+ randomly chosen problems from Codeforces (2800+ pre-cutoff, 500+ post-cutoff) yielded less than 1% pass rate both before and after the cutoff. Due the very small number of problems that succeeded even before the cutoff date, were unable to present non-trivial differences before and after the cutoff. On the other hand, we still include all raw output data from our experiments with -oxt-dainnci-002 our supplementary material at https: / amonymous 4open. wcieencerr/tooooo the cutoff coutoff-revient review- 2533
We will
shortly make our Code-Ilama data available as well.
B.7 ANALYSIS OF POSSIBLE COVARIATE SHIFTS
In this section, we conduct analyses to assess whether the drop-off in performance that we observe for problem examples released after the GPT training cutoff might be attributable to (potentially latent) covariate shifts. We specifically investigate whether the distribution over tags (only available for Codeforces) and/or difficulty level (available for Codeforces and Project Euler) changed in statistically significant way during the post-cutoff period, relative to the pre-cutoff period. We summarize our findings by dataset below:
B.7.1 CODEFORCES
Tags: In Codeforces, each problem is mapped by the problem creators to one or more descriptive tags. These tags generally reflect algorithmmss), data structure(s), and/or problem-solving approaches associated with the problem in question. We partition the Codeforces problems into pre- versus post (reflecting whether given problem released before after the GPT training cutoff) and compute aggregate counts over tags, then normalize. We visualize the resulting frequencies in Figure 4 and observe that the two distributions are qualitatively quite similar:
CodeForces: distribution over tags for problems released pre- vs. pest-GPT cutoff
prc
post
proportione 50b gruupppup preblems with this big
0.06
0.04
nlww
genngg
6 sardae
1a 200uth gran
saqueed
sampruss cupp
Gussed uossaadro
-n intmueouill
ratuss pue sip
Adoon Aoong Jequir
Sumppnns wuRs tuns tuas
tags
Figure 4: Codeforces: distribution tags for problems released pre- post-GPT cutoff
We also conduct x test assess whether able to reject the null hypothesis, which states that there is no significant difference between the normalized distribution of counts over labels in the pre- versus post-cutoff examples (we the pre-cutoff normalized counts the reference expected
17

---

distribution). We omit two tags with observed counts < 5; these chinese remainder theorem and schedules. The resulting x² test statistic 0.006 (p=1.0); thus, we are not able to reject the null hypothesis. These findings mitigate concerns that the drop-off in performance observe for Codeforces might be attributable to significant changes in the distribution over tags during the post-cutoff period.
Difficulty: For the Codeforces problems we consider, difficulty takes values in the range [0,3500], with higher values corresponding increased difficulty. We visualize the distribution over raw difficulty scores for problems in the pre-cutoff versus post-cutoff subsets in Figure
CodeForces: distribution of difficulty scores for problems released pre- vs. post-GPT cutoff
3500
3000
Raw difficulty score
2500
2000
1500
1000
500
0
pre
post
problem cut-off group
Figure 5: Codeforces: distribution over raw difficulty scores for problems released pre- vs. post-GPT
cutoff
We then discretize the raw difficulty scores by mapping raw scores discrete labels follows:
0 I 01
1 Q1 <<<5500 Q2
x(x)
(I)
2 Q2 <2<
x x> Q3
where represents given problem's difficulty score, and Q1, Q2, and Q3 correspond the first, second, and third quartiles, respectively. We visualize the pre-vs.post-cutoff problems' distribution over discretized difficulty scores in Figure
We also conduct x" test to assess whether we are able to reject the null hypothesis, which states that there is no significant difference between the normalized distribution of counts over discretized difficulty scores in the pre- versus post-cutoff examples (we use the pre-cutoff normalized counts as the reference or expected distribution). The resulting x² test statistic is 0.073 (p=0.995); thus, we are not able to reject the null hypothesis. These findings mitigate concerns that the drop-off in performance we observe for Codeforces might be attributable significant changes in the distribution over difficulty levels during the post-cutoff period.
B.7.2 PROJECT EULER
Note that as mentioned above, Project Euler does not make their problem tags publicly available; as such, not able to conduct tag analysis for this dataset, and restrict our attention to difficulty.
18

---

CodeForces: distribution over discretized difficulty scores for problems released pre- vs. past GPT cutoff
0.35
with whththe scretted drficuly ScOte
0.30
proportio of SUB group problms
0.20
0.35
0.05
0.00
0
dif
buckat
Figure 6: Codeforces: distribution over discretized difficulty scores for problems released pre- vs. Post-GPT cutoff
Difficulty: For the Project Euler problems we consider, difficulty takes values the range [5,100], with higher values corresponding to increased difficulty. We visualize the distribution over raw difficulty scores for problems the pre-cutoff versus post-cutoff subsets in Figure 7
Project Euler: distribution of difficulty scores for problems released pre- vs. post-GPT cutoff
100
80
Raw difficulty score
20
pre
post
problem cut-off group
Figure 7: Project Euler: distribution over raw difficulty scores for problems released pre- vs. post- GPT cutoff
Similarly to our Codeforces difficulty analysis, we discretize the raw difficulty scores using the mapping in Equation I We visualize the pre-vs.post-cutoff problems' distribution over discretized
difficulty scores in Figure S
We also conduct x test to assess whether are able reject the null hypothesis, which states that there is no significant difference between the normalized distribution of counts over discretized difficulty scores the pre- versus post-cutoff examples (we the pre-cutoff normalized counts as the reference or expected distribution). The resulting x2 test statistic is 0.027 (p=0.999); thus, able reject the null hypothesis. These findings mitigate concerns that the drop-off
19

---

Project, Euler: distribution over discretized difficulty scores for problems released pre- vs. past GPT cutoff
0.35
0.30
with thes sccetzed dificuty Scote
proportio of s1------g group probeems
0.25
0.20
0.15
0.30
0.00
dif buckat
Figure 8: Project Euler: distribution over discretized difficulty scores for problems released pre- vs. POst-GPT cutoff
in performance observe for Project Euler might be attributable to significant changes in the distribution over difficulty levels during the post-cutoff period.
20

---

B.8 RANDOMLY SAMPLED GPT-GENERATED OUTPUT FOR CODEFORCES, BY LLM
B.8.1
GPT-4
1. Problems released before GPT cutoff with functional correctness [0,0.0]
(a) Subgroup sample 1 of 2
title: scissors and tape id: 1193_C _ difficulty: (unc_correct, 0.0
url: http:///odeerrressoooommm//////p
def polygon area(points):
half sum
= (ennttisss) for in range(n):
j= (i 1) %
half sum (points(i][O] points(jl[[] points[i][1] points[i](O))
return abs(half_sung Batnaffssmm///
def main():
list(map(int, input().sphi(()]1:]
t= (mat(n,,,, inputo.split())(1]
s_points = [ss[2 il, s/2 * + 11) for in range(len(s) // 2)1
tppoints [(122 il, (/2 11) for in range(len(()) 2)1
$_ a_aeaa polygon area(s points)ont)
area polygon Baalyyoonnrree((___ area(tteeaaaiaaa points) _area_to_s______________________ area t_area/ s_area
Assuming s_area Larea.
ain(s_areeat_areee 1c-6:
print( scissors_ _0aioooss_000__ len(s_points)) for in range(O. len(s), 2):
print("1", s[i]. 1]. 0")
print(" len(s points). "jjoin(map(str, range(1, Jen(s points) + 1)))
print(" "0,0", "..jooi(mmapsstr.. 0))
else:
print( Tho__hpur_polyyon__ddomooo "he_ss____smme_rraa"
name = main
main()
(b) Subgroup sample 2 o
title: string inside out id: 530_B difficulty: 1600
Munc_correct, 0.0
url: https://oodoore...ommmm/e/e///es
def manipulate manipulate_string(s)
len(s) //
return s(n 1::1] s(mn 1:-1]
2. Problems released hefore GPT cutoff with functional correctness (0.0, 0.137]
(a) Subgroup sample 1 of 2
21

---

title: zebra tower id: 159_
difficulty: 1700 fune correct: 0.125
url: https:/ccctbtessssommmmooo/////s
def main():
int(input()
cubes = [tuple(map(int, input().splitt) for in range(n))
color cubes (}
(color, size) in enumerate(cubes):
if color not in color_cubeee __t____uess_se_______eee__eeetre
color callor_ubesscolll color_cubexscclorr..pppo (atp:lo(())ppppp(ssssi..... i))
[]
color in color cubes:
color_cubes: how.guubbescoololsoootreeeeeeeee sont(reveess===rre))
color_ collo_cubes[collor
color [(sum(x[0] for in color_cubes[e]] c) for in color_cubool
color color_sums.sortiie soresssss(((ee
max_ mrax_heihht=0
best_tower= =
for in Lange(len(color_smme)
for in range(i+1, len(color sums)): heightl, colorl = color sums[i]
height2, color2 color sums[j] tower
while color_ cubbssccubc(ccllr and collor_ubee(ccolori
sizel, idx1 color_
color_cubestttt cubes[colorl).popp
size2, idx2 collr_collieeeeeeee (ultor_cbbb//ooo22ppppppp tower.append(idx)) tower.append(idx2)
# If there is extra cube of the first color, add it to the tower if color callo_cubes/color|]
sizel, idx color_ccbeaee cocllo_cubes/colorlll p.pop)
tower.append(idx()
curr height sum(cubes[idx][]] for idx in tower)
if curr_ surr_height max_ height
max_height= curr curr_height
best_tower tower
print(max_heiget)
print(len(best_towe tower))
print(=[x+) for in best best_ tower))
name main
main()
(b) Subgroup sample 2 of 2
22

---

title: dirty arkady's kitchen id: 827 0.053
difficulty: 3200
coneeccreet:::
url: http:///dddddrrssssoommmmm//e/ns
def main():
n, map(int, input().plit())
passages [taple(map(int, inputO.split()) for in range(m)]
def can reach(destination, time):
visited [False] (n (a++)
stack = [(1, time)]
while stack:
place, == stack. pop()
if place destination:
return True if visited[place]: continue plsited[place]= True
for a, b, I, in passages:
t> and <<r:
next_ Baul_ppace-- a if place else (b if place else None) next placepplceeetc is not None and not visited[next__lacee
stack.append t 1))
return False
start_time Hart_time== while not can_ re__recch(n, start_time):
start time += 1
if start time 10e+9: print(-1) return
print(start_time)
main(
3. Problems released hefore GPT cutoff with functional correctness (0.137, 0.667]
(a) Subgroup sample 1 of 2
23

---

title: alarm clock id: 898_ difficulty: 1600 fune correct: 0.316
url: https///oobbrres..ommmmmeeeee//s
def min main_llarms_to___ to__tto__offfn_ k, alarms):
alarms.sort()
diffs [alarmsfi+1]] alarmm(ii for in range(len(alarms)-)) diffs.sort(
turn off =
while len(diffss) >= k-1 and sum(diffs(3k1I) k-1 < m:
tur__off +=
diffs.pop(
return turn_off
main():
n, m, k = map(int, input().plii())
alarms = mipt(m,,,,nt, inputO..plii())
print(min alarms to turn off(n, k. alarms))
main()
(b) Subgroup sample 2 of 2
title: of models
id: 887 F
difficulty: 2500
dine_correct: 0.424
url: https://oodbbeccc.commmm//////ss
def main():
k= map(int, inputO.split()
heights list(map(int, input().spii(O))
def test__ssappplt sas__ssaphhhights k):
first Biihhthhhrr=== -1
for in range((en(heights)- 1):
if heights(i] heightsfi 1]:
first_ first_higher -1:
first_ fost__iihhr=== =
elif - first higher < <k 1: return False
return first higher =I or first higher
if sesthsswpphe,,tts, k) or test_swap(hcightss:--11 k): print("YES")
else:
print("NOv
name
main
main()
4. Problems released hefore GPT cutoff with functional correctness (0.667, 1.0] (a) Subgroup sample 1 of 2
24

---

title: alternating current id: 343
difficulty: 1600
fune_ correct: 1.0
url: https////oddbrre...o////////////
def he unntanglcd(():: stack [] for in s: if stack and stack[-1] == stack.pop()
apack.append(c)
return "Yss" if not stack else "No"
input_sequence- sequence input().stripO
print(can Seaannglddiiputsseuenee
(b) Subgroup sample 2 of
title: date calculation | id: 130_J
difficulty: 1800 func_correct: 1.0
url: https://ooeberes..ccopoooeeee//s
def' is leap year(year): return year % 400 0 or (year % and year 100 !=
def _ day_num):
days in months = [31, 28, 31, 30, 31, 30, 31, 30, 31, 30, 31] if is Jeaplaapp_eeeee year(year):
days, in [r__i___moth([]]] =
month 0 while day_ day_num days in hnt_mmonthssmmtii
day day_num days in months(month)
month return day_num, month
year = int(in)tt)) day int(input))
day, month find date(year, day num)
print(day, month)
5. Problems released after GPT cutoff with functional correctness [0,0.0]
(a) Subgroup sample 1 of 2
25

---

title: changing brackets id: 1593_G G
difficulty: 2200
/func_ conc_ccreett: 0.0
url: http:///dddddrrs.soommm////////s
def min_cost(s,
ent ent_open = ent cat_cnt__aaet bracket cost = 0
for in range(l. r+1):
if s[i] == '(' or s[i] == "[:
ent open +=
else:
ent_open -= if ent_open 0:
if s(i] == '): cnt bracket cnt_ pat_open=000
return bracketcket
I intrinput())
for in range(t):
== input()
int(input())
for in range(q):
1, map(int. input().split))
print(min cost(s, 1-1, r-))
(b) Subgroup sample 2 o
title: premutation id: 1790_C _990_C| difficulty: 1000 fuuc_correct: correct: 0.0
url: http:////ooffeeeccco/ooooeeeeenp
aeconstructperrmuatioonnn test_cases):
def restore_ cistoro_permuaaiionnn. sequences):
occurrences = [0] (n + 1)
for seq in sequences: for num seq:
occurrences/num) +=
return for i, count in enumerate(occuurence)) if count == n 1]
results
case in jest_cases:
n, sequences = case
resulls.apeend/rssoorr perprsttoorrrrmmmatitoonnnn sequences))
return results
t= 5 test_ casescase- =
(4, [[4, 1], [4, 2, 3), [2, 1,3), 1,1
[[3 2], [3, II, [1,
(5. [[4,
2,3. 1,5D).
(3, [[1, 3), [2. (1,
(4. [[3, 1, (2, 1.3], (1.2,.1)
result reconstrut_permmtatinne test_cases)
for in result: print(*r)
6. Problems released after GPT cutoff with functional correctness (0.0, 0.137]
(a) Subgroup sample 1 of 2
26

---

title: arranging crystal balls id: 1740 1 difficulty: 3500 0.062 url: http:///oddderrressmmmmmeeeeeemp
func_correet: correct:
def ged(a, b): while b: a, b. %
return a
def minimum_nnponneee operraioosn,,,,,,,n,,.. god_ak = a[0] a[k]
a):
for in range(k, n):
ged_ak sod_ak= = ged(ged_att ak, a[i] - afi k))
if ged ak % m t=
return -1
return sum(abs(ali] a(0)) // for range(n)) //k
m, k = map(int. inputO.split()
list(map(int, input().spli())
print(minimum ootrrattoootaiiioosnnn. m, k, a))
(b) Subgroup sample 2 of 2
27

---

title: count voting id: 1799
difficulty: 2600
fune_correct: correct: 0.083
url: http:///wobbrres..ommmmmooeeeess
def main(::
MOD = 998244353
= int(input())
desired votes list(map(inn, input().spli(O))
(int(,,,,,nt, input().plit())
team_ votes [0] (n + 1)
for tv in team: @eam_votes(tv// +=
dp [[0] (n + 1) for range(n + 1)] dp[0][0]=] =1 for in range(1. n 1):
for in range(n 1):
dplilli] = dpli 1]0i]
if dedeeired_votestfeeeee 1] >=
dpfillj] -p 1][j [ desired_votes/ I]]
if dpfil[j] >= MOD:
dplilli] MOD
ans =
total non team votes 0 for in range(1, I):
count team_votes[i]
if count=
continue
tmp = dp[total Bictiaal_nnn_tee non www__ooo_nnttn coun|[ccun)) ans
if tmp MOD:
tmp MOD
ams tmp total non team votes += count
paint(ans)
main(
7. Problems released after GPT cutoff with functional correctness (0.137, 0.667]
(a) Subgroup sample 1 of 2
28

---

title: count binary strings id: 1767_ 0.364
difficulty: 2100
fune_correct: correct:
url: https://joobbbresscmmmmmooeeeene
def main(): MOD 998244353
int(input())
[list(map(int, input()splitO)) for in range(n)
dp [[[O] (n + for in range(n 1)] for range(3))
for in range(n+1>; 1); dp[0][i]í]] - =1
range(2. n+1): for in range(1. n - 1 2):
j=i+l-1
if ffali-1ljj al
dp[O]]i][j] += (dp[0]fi+ (=pp0))i+ + 1Ilj)) % MOD
dp[0][i][]] %= MOD
if ali 1][i - :=
dp[[][i][] (dpío][i+ I][]] dp[[][i+ 1](i)) % MOD dp[[][i][] %= MOD
for k in range(i. j):
if a(i - 1][k 1= and - MOD dp[2][il[j] += [
a(i 1][k - =kd= 2:
dp[2][i][i] += + [][i] % MOD
if ali lljk 1 != and 1-1 (++1)] t=
dp[2][i][i] += dp|[]] 1][i] % MOD and 1
aaai-
1)
dp/2[[iljj += xp101[ilk]l+dpp111111]1 1][j] % MOD
dp[2][i]0] MOD
print(dp(21[[[n)
name
main
main()
(b) Subgroup sample 2 of 2
29

---

title: interesting sum id: 1720 B
difficulty: 800 func correct: 0.5
url: http:///webbbee....mmmmmeeeeeest
def maximum saaximum_heautt(f,. lest_cases):
results = []
for n, in test_ccaa cest_cases:
max outside max((a)
inside, min inside max(a[l:-I]). min(a[[:-1])
ams max(max outside a[0] max inside a[-1]. max_onie _ox__ouside--ee a[-1] max__nsidde- -a(0))
results.append(ans)
return results
main(::
t= int((nputO..trip()) test_cases = []
for in range(t):
intr(nnutt).stri()))
a list( (intti,, impuut(.srripO.split))
test_cases.ae (ases.a a))
output maximam_ test_cases)
for in output: print(res)
name = main main()
8. Problems released after GPT cutoff with functional correctness (0.667, 1.0] (a) Subgroup sample 1 of 2
title: the boss can count pairs id: 1830 B difficulty: 2000 |ffnccccurreee correct: 1.0
url: http:///ccoooorsssommmmmm//////p
def' main():
int(input((ssrii()
range(t):
n int((inpuut int(input(..strpp))
list(map(int, input(.stripO:spltt))
list(map(int, imput(..tripO.split))
count =
for in range(n):
for in range(i 1, n):
if afi] a[j] == b[i] b[i]:
count
print(count)
name
main
main()
(b) Subgroup sample 2 of 2
30

---

title: interesting sections id: 1609
difficulty: 2800 func corr_ctoret:: 1.0
url: http://coooorrssscomomooeeeeeeet
def main():
intrinput(..srrpO)
a= (isp(m,,,int, imputto-stripO-spli(()
def bit count(x):
return bin(x).count((1)
ans =0
for in range(n):
min a[l]
max val a[l] for in range(l, n):
min_val- min mai_mmiilllll__lll a(r))
max_val val max(max val, a(r))
if bit count((in,,, val) bit count(max_ val):
ans +=
print(ans)
name
main ::
main()
B.8.2 GPT-B.5-TURBO
1. Problems released hefore GPT cutoff with functional correctness [0,0.0]
(a) Subgroup sample 1 of 2
31

---

title: trash problem id: 1418 D difficulty: 2100 func correct: 0.0
url: http:///wddbbeee...mmmmoooeeeeep
n. q = map(int, input()splitt))
= sorted(map(ind input().split()
p= [a[i]:i for in range(u))
[x-y for x.y in zip(a[1:]. a)]
= [x-y for x.y zip(a[2:]. a)]
ans anss = sum(s)
for in range(q):
map(int, input().slit())
X-=
if I:
a.append(x)
a.sort()
i= [[x] a.index((x
if ansl += max(O. ali]-ali-1]-I)
if (fi<n-l n-1: ansl += max(0, a[i+1]-a[i--1)
ifi 1: ansl -= max(0, - -
if n-2: ansl - max(0, a[++]]-ai++1]-I)- sfi-1]
sl. = s. [x-y for x.y in zip(a[l:]] a)]
else:
i=
if 0: ansl - max(0 a(i]-a[i-1]-I)
i ansl max(0, a[i+]]-aii--
ifi
ansl += max(0, ali]-a(i-]]-)-
- sl[i-2]
n-2 amsl += max(0, a[i+2]-a[i+l]-I) - ssi-1] a.remove(x)
p(x]
s (x-y for x.y in zip(a[1:]. a)]
print(ansl)
print(ans2)
(b) Subgroup sample 2 of 2
32

---

title: boboniu and string id: 1394_ C
difficulty: 2600
ffue__correts correct: 0.0
url: http://dddddorrsssoooom/o///ee/p
collections import Counter
def' can reach(target. source):
"Check if target can be obtained from source by removing ' and 'BNY/
<> NB'. new
(targe))
m len(source)
==
while i and m:
if targetti] sourcefj]:
i+=
j+=1
elif target[i] = 'B' and sourcelj] == 'N:
j+=
elif target[i] = 'N' and sourcelj] = 'B":
j+=
elif farget[i:i+2]]= == 'BN' and (j+1 and sourcefji++2] == 'BN");
j+=
elif target(i:i+2]== 'NB' and (j+1 < m and source[i:j+2] == 'NB'):
++ 2
else:
return False
return ===
int(input()
strings=
for in range(u):
strngg.appeml(input(tsst/p/
freq = Counter(" jountee("
freq >= N']]: [req I 1+= 'B' freq[ [B]]
clse:
t+= ["eq['N"]
def'
"""Use hinary search to find the minimum value of max__dist_
r 1, 2, 10 6
while <r:
mid = (1+ 1/
can make = False
for char in ['B', 'N, 'BN. 'NB']:
for start range(len(t)):
new_t= tf:start] char t[start:] if all(can caacccn_reachnew__t.. s) for in strings): can make = True
break
if can make: break
if can can_make:
mid
else:
== mid +
return
# Note from authors: cutting this submission early for display convenience (it's too long show all of it).
33

---

2.
Problems released before GPT cutoff with functional correctness (0.0, 0.051]
(a) Subgroup sample 1 of 2
title: up the strip id: 1558 B difficulty: 1900
cone_corret:: 0.03
url: https/////ddbbeeeco/////////////
map(int, input().sliit()
dp = [0] (n+1) dp[n]=
for in range(u-1, 1, 0, -1);
dpfi] = [dp[i+1] + dp(i+2]) % m
print(dp(1))
(b) Subgroup sample 2 of
title: yellow cards id: 1215_A
difficulty: 1000 fune_ cane_correct: 0.023
url: http:///oodccrrrssoommmmmee////p
al int(input()
a2 int(input())
kl = int((nput())
k2 int(input))
int(input()
min players out max(0. n -all=(k1-1)++ + a2+(k2-1)))
max_playcrs_out= min(n, min(al,k1 )eal + min(a2,k22)a2 -
print(min_players_oomm max_players_out
3. Problems released before GPT cutoff with functional correctness (0.051, 0.4]
(a) Subgroup sample 1 o
title:
exchange of books id: 683_H difficulty: 1900 fuuc_correet: 0.1
url: http:///cdddccccssommmm///ee/ees
k map(int, inputO.split())
friends = (ist(ma,(int, input().spiitO)
start = list(range(l, n+1)) # starting arrangement of books
create a dict map current position the next position after one day next_pos- [i+1: friend for friend in enumerate(friends)
# simulate book exchange for k days
for in range(k):
end [0] n initialize empty list to hold arrangement of books for j in range(n):
end[ncct_ppossit ||b]++1|11]] |-1] [-] = start[j] exchange books
start end update starting arrangement to end arrangement
next_pos [j+1: [ottpooo__oo__os-_111 for range(n)) update diet
print final arrangement of books
print()start)
(b) Subgroup sample 2 of
34

---

title: alarm clock id: 898
difficulty: 1600 func correct: 0.368
url: http:///oobbbree..ommmmmo/eesssp
n. m, = map(int, input(..plitt))
alarms soated(map(int, input().plitO0) interval = [0] (10*16 1)
for in range(1. 11-kk 2):
start alarms[i-1]
=ld=alans-j+--2] 22 ffftk-1 <= n else 10**6 interval/start:ondd-]] [j+1 for in interval[starttonn==]]] print(sum([1 for x in interval if ml))
4. Problems released before GPT cutoff with functional correctness E (0.4, 1.0]
(a) Subgroup sample 1 of 2
title: chemistry in berland id: 846_E difficulty: 2300 func_correct: 0.875
url: http:///oeebberr...ommmoooeee/es
n= int(input()
b b= lis((map(int, inputo.split())
(ist(ma, input(.splitO)
adj for range(n))
for range(1, n):
x, k= map(int, inputO..pliit)) adili).append(((-)) k))
adjjx=I].append((... k))
visited = [False]
def dfs(u):
visited[u]= True for v. k in adj[u]: if not visited[v]:
a[v] - asu]
b[v] - k b[u]
if [a[]] b[v] or alv] % ged(a[u). b(u)) != 0:
return False
if not dfs(v):
return False
return True def' ged(a, b):
while b: a. bb %
return:: a for in range(n):
if not visited[i] and a[i] |= 0: if dfs()):
continue
else:
print("NO")
break
else:
print("YES") (YESS")
(b) Subgroup sample 2 of
35

---

title: optimal subsequences (easy version) id: 1227 1227_D1
difficulty: 1600
cone_correct: 0.5
url: http:///odddeereeccmooooomeeeeep
= int(i))
= (int(m input().sliiO)) int(inputO)
# Indices of maximum elements all prefixes of
max max__inicess- = [0] + n
_ur_max_____ idx =
for i in range(n):
a[i] a[cur Batrrr_ax_____ max__dxx: cur max idx =i
max [ax_iinices[[] cur_max_idx
for in range(m):
k, pos map(int, input()split())
subseq = []
i = max indices[k 1] start with the maximum element in the k-length prefix
while len(subseq) k:
if a[i]
aubssq-aapeeddcaii)
++=1 1
if n: end of is reached
break
(oaneegssot((eeeess=Teee) lexicographically minimal subsequence with S maximum sum
print(subseq(pos- IJ)
5. Problems released after GPT cutoff with functional correctness [0,0.0]
(a) Subgroup sample 1 of 2
36

---

title: anti-fibonacci permutation
id: 1644 difficulty: 800
fune_correet 0.0
url: https///oddbbeeessmmoooommmeee//
function to generate a permutation
def' generate permutation(n)) return list(ranee(.nnt))
function to check permutation anti-fibonacci def is_anti anti_fibonacci(p)
for in (2.nge(..ee(p))) if p[i-2]+pli--]== ==
pli]:
continue
else:
return False
return True
int(input())
for in range(t): int(input())
create list of all possible permutations
all permutations [generate permutation(n))
iterate over all permutations and check they are anti-fibonacci
for in range(2, n):
for in range(i):
new permutations =
for p in all all_permutations:
new p.copy()
pew_p new_ new_plj-= new www.piil...co. new pew_
if is anti fibonacci(new p):
new new_permuiee sermutations.appedddiiii
p)
permuutaations += new rew_permutations
print the first anti-fibonacci permutations for p in all [e] print(*p)
(b) Subgroup sample 2 of 2
37

---

title: cypher id: 1703 C difficulty: 800
conrect:: 0.0
url: http:://wwbbbbeee..mmmmmeoeeeets
originallsequeeeeeee sequence(n, final, moves):
initialize original sequence to all zeroes
orig = [0]+n
apply moves reverse order to determine original sequence in range(n-1,-I,-1))
if moves[]][0]== D':
shift int(moves(il[1:)
elsc: shift -int(moves(i][1:)
# adjust for wheel rollover shift
if final[i] shift < O:
elif final[i] + shift > 9:
shift - 10
origlil (finalfi) shift) 10
return orig
int(input()
for in range(t):
int(input()
final [int(x) for in input().split)]
[input((.stri() for in range(n))
orig = original sequence(n, final, moves)
print(sorig)
6. Problems released after GPT cutoff with functional correctness (0.0, 0.051]
(a) Subgroup sample 1 of
title: torus path id: 1765_ K
difficulty: 1500 ffnc__ooree 0.04
url: http:///ootorrcssoooooom//////pp
int(input()
grid=(] [
range(n):
grid.append(iss(mappimm input(.splitt))
dp=[[0]=n for in range(n))
dp[0][0] gr]][
range(1.n):
for in range(1.n):
=p for i in range(1.n):
for in range(1.n):
dp[i][]] = max Apiliii- grid[i][i]
print(dp(-I][-I)
(b) Subgroup sample 2 of
38

---

title: olympic team building id: 1779 H difficulty: 3500
fune correct: fune_correct:
0.023
url: http:://ooobborrsscommmoooe//enn
intrinput() strengths list(map(inn, input().plitt))
winners = [True] n Assume all players are initially winners
while
new winners [False] # for in range(0,, n, 2):
strengths[i] strengths[i+1]
if total_ total_strength max(total_ strengthh 0): If hoth strengths positive +
only one negative
new new_ [ww_winner[[i]= = True
else:
new new_winner[[i+]] True
winners [winnersfi] and new_ winners[ii for in range(n)) Update winners
+ list
//= 2 # Half the number of players
strengths [strengths(2-i] + Strengths[2si+]] for i in range(n)] # Sum strengths for remaining players
for in range(len(winners)) if winners[i]:
print(" 1I, end="")
else:
print("O) 0"" end="")
7. Problems released after GPT cutoff with functional correctness (0.051, 0.4]
(a) Subgroup sample 1 of 2
39

---

title: magician and pigs (hard version) id: 1774 F2 fune_corrects 0.375
difficulty: 2700
url: http:///oddbbbees.ooommo////////
n= i=in((npput))
# create empty list to store the pigs' health points pigs=1 =
# iterate through n operations
for in range(n):
op -input(.splitt)
op[O] '1':
create new pig with x health points pigs.append(inn(opl1))
elif op[0] == '2":
reduce all pigs' health points by
for in (ange())
pigsfjl - (1]()
else:
repeat all previous operations (including repeats) for in range(i):
prev_op= (npptttt(()ssplit))
if prev_op[0]== == '1":
# create
pig with health points
pigs.appenddint(pre..c __][))
clif prev prev_op[0] == '2':
# reduce all pigs' health points by x for in range(len(pigs)):
pigs(k] - int(prev__op)) _p[[])
remove dead pigs from the pigs = [p for in pigs if 0]
# print the number of living pigs modulo 998244353 print(len(pigs) % 998244353)
(b) Subgroup sample 2 of 2
40

---

title: physical examination id: 1 0.333
difficulty: 3200 fune_correct: correct:
url: http:://woddberre..smmmmm/ee////
for in range(int(inuut())
int(input()
L= list(map(int, input()split())
R = list(map(int, inputo.split())
max(0, max(L)- min(R))
[0]
for in range(n):
pii]=i
pesor(key=lambda i: Iffi)
current Bittt__ttme__e_____ee
in range(n):
if L[p[i]] > current time + i or R[p[i]] < current time + i: print(-1)
break
print(current_timee end='_
else:
print()
print(#[pli]+1 for in range(n)))
8. Problems released after GPT cutoff with functional correctness (0.4, 1.0] (a) Subgroup sample 1 of
title: replace the numbers id: 1620 E
difficulty: 1900 func_correct: correct:
1.0
url: http:://ooobbercc.cmmmmmooeeeees
q= int(input()
arr [] for range(q):
query = iipput).splito) if query[0] 1": arr.apeend(in((query((])
else:
x int(query[
= int(query[2) (2))
for in range(len(ar)): if arrū] = arr []] =
print( sarr)
(b) Subgroup sample 2 of 2
41

---

title: garland id: 1809 A difficulty: 800
conrect:: 0.467
url: https//ccdddoccsscooomooo/ee///o
min_ :peratiooai:::::::::: colors = set(s) if (en(colsss) < 4:
return -1 on set() count 0
for in range(4);
if s(i] not in on.add(s[i)
count +=
return count
int(input() for range(t):
s= imput() print(min_operratoons()
42

---

Table Regression table for Pass Rate of GPT-4 the Codeforces dataset. See Figure regression coefficients.
for plot of
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.084
0,204
(0.082, 0.087)
(0.183, 0,228)
= 0.000"
p = 0.000*
GitHub_Presence Presence
1.044
1.000
(1.038, 1.050)
(0.986, 1.014)
= 0.000*
p = 0.988
Constant
37,810,359.000 $50,000
36,517.320
(29,642,418.000 48.261.51440000)
(15,107.710, 88, 88,780.010)
0.000
p 0.000
Observations Log Likelihood
6,693
1,378
-61,571.750
-3,358.24
Akaike Inf. Crit.
123, 123.149.500
6.722.482
Note:
*p<0.05
Regression Coefficients for GPT4 F Functional Correctness -- Codeforces
D ifficulty
GitHub Presence
0.05
0.1
0.5 Odds Ratios
Dependent Variables
Pre Cutoff
Post Cutoff
p-level o n.s.
Figure 9: Regression coefficients plots of Pass Rate for GPT4 on the Project Codeforces Dataset. See Table for regression coefficients.
43

---

F Functional Correctness Marginal Effects Plots for GPT -4 on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
60%
Functional Correctnies
log(Difficult)
7.04
40%
7.43
7.83
30%
20%
10%
10
15
10
15
Iog Presence)
Figure 10: Marginal Effects of Pass Rate for GPT-4 on the Codeforces Dataset
Table Regression table for Pass Rate of GPT-3,5-Turbo on the Codeforees dataset. See Figure I] for plot of regression coefficients.
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0,115
0,228
(0.112, 0.119)
(0.202, 0.257)
p= 0.000*
= 0.000
G Presence
1.023
1.014
(1.017, 1.030) p= 0.000*
(0.998, 1.030)
= 0.081
Constant
2,859,990 $159,990.00
9, 9,989.318
(2,244,055.00) 0000 3,646,978.000) p 0.000*
(3,842.640. 26, 109.620)
0.000*
Observations Log Likelihood
6,692
1,378
-52,913.822
-2,611.597
Akaike Inf. Crit.
105,833.600
5,229.195
Note:
*p<0.05
44

---

R Regression Coefficients for GPT 3.5 Turbo F Functional Correctness -- Codeforces
D Ifficulty
GitHub Presence
0.05
0.1
0.5 Odds Ratios
Dependent Variables p p-leeel
Pre Cutoff
Cutoff
o n.s.
Figure 11: Regression coefficients plots of Pass Rate for GPT-3.5-Turbo the Project Codeforces Dataset. See Table 2 for regression coefficients.
Functional Correctness Marginal Effects Plots for -urro. on Codeforces
Problems Released Before Cutoff
Problems R Released After Cutoff
50%
Functional Correctness
40%
log(Difficult)
7.04
7.43
7.82
20%
10%
10
15
10
15
log(GitHub Presence)
Figure 12: Marginal Effects of Pass Rate for GPT-3.5-Turbe on the Codeforces Dataset
45

---

Table 3: Code Bison Functional Correctness Codeforces
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0,221
0,344
(0.214, 0.228) p= 0.000
(0.240, 0.492)
= 0.000"
Github Presence Github_Preeence
1,033
1.023
(1.027, 1.040)
(0.978, 1.071)
= 0.000
p= 0.316
Constant
12,033.200
229.971
(9,387.552, 15,428.9900 0.000"
(14.438, 3,694.907)
= 0.0002"
Observations Log Likelihood Akaike Inf. Crit.
7,807
217
-38, -38,944.730
-228.904
77,895.470
463.807
Note:
*p<0.05; "'<<10.*]]:
*** p<[00 [0,***
Regression Coefficients for Code Bison Functional Correctness Codeforces
D ifficulty
GitHub Presence
0.05
0.1
0.5 Odds Ratios
Dependent Variables p plveel о n.s.
Pre Cutoff
Post
Figure 13: Regression coefficients plots of Pass Rate for Code Bison the Project Codeforces Dataset. See Table for regression coefficients.
46

---

Pass Rate Marginal Effects Plots for Code Bison on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
log(Difiicllt)
Pass Rate
20%
7.03
7.44
7.85
10%
10
15
10
15
ItH Presence)
Figure 14: Marginal Effects of Pass Rate for Code Bison on the Codeforces Dataset
Table Regression table for Pass Rate of GPT-4 on the Codeforces dataset (public test cases only). See Figure 15 for plot of regression coefficients.
Dependent variable:
Pass Rate (Public)
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0,122
0.834
(0.110, 0.135)
(0.624, 1.116)
p = 0.000
= 0.221
1.043
0.998
(1.024, 1.063) p 0.00001*
(0.964, 1.033)
p= 0.895
Constant
2,468,833 83990080000
0.647
(1,118,457.00 5,484,288.00 (00)
(0.064, 6.445)
= 0.000*
= 0.712
Observations Log Likelihood Akaike Inf. Crit.
6,693
1,378
-7,352.622
-712.065
14,711,20
1,430.131
Note:
*p<0.05
47

---

Regression Coefficients for GPT4 Functional Correctness Codeforces (P Public))
D Ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables
Pre Cutoff n.s.
Post Cutoff
p-level
Figure 15: Regression coefficients plots of Pass Rate for GPT-4 the Project Codeforces Dataset (evaluated on public test cases only). See Table for regression coefficients.
Pass Rate Marginal Effects lots for GPT-4 on Codeforces (P (Public)
Problems Released Before Cutoff
Problems Released After Cutoff
60%
log(Difficulty)
Pass Rate
40%
7.04
7.44
7.85
10
15
10
15
log itHub Presence)
Figure 16: Marginal Efffett of Pass Rate for GPT-4 on the Codeforces Dataset (evaluated on public test cases only)
48

---

Table 5: Regression table for Pass Rate of GPT-3.5-Turbe the Codeforces dataset (public test cases
only). See Figure for plot of regression coefficients.
Dependent variable:
Pass Rate (Public)
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.157
1.046
(0.141, 0.173) p 0.000
(0.744, 1.478)
p 0.798
GitHub_Presence Presence
1,027
0.998
(1.006, 1.047) p= 0.011
(0.960, 1.038)
p 0.919
Constant
269,429.200
0.080
(119,822.600, 608,823.800 = 0.000*
(0.005, 1.190)
p = 0.069
Observations Log Likelihood
6,693
1,378
-6,461.524
-562.326
Akaike Inf. Crit.
12,929.050
1,130.652
Note:
"p<0.05
R Regression Coefficients for GPT 3.5 Turbo
F Functional Correctness - Codeforces (P ublic)
D ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables p-level
Pre Cutoff
PostCCuoff
n.s.
Regression coefficients plots of Pass Rate for GPT-3.5- 5urbo on the Project Codeforces Dataset (evaluated on public test cases only). See Table for regression coefficients.
Figure 17:
49

---

Pass Rate M Marginal E Effects lots for GPT-3.5-Turb on Codeforces (P ublic)
Problems Released Before Cutoff
Problems Released After Cutoff
50%
40%
log(Difficult))
Pass Rate
30%
7.04
7.44
7.85
20%
10%
10
15
10
15
ItH P Peesnce))
Figure 18: Marginal Effects of Pass Rate for GPT-3.5-Turbo on the Codeforces Dataset (evaluated on public test cases only)
Table 6: Code Bison Functional Correctness Codeforces (Public)
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0,309
0.606
(0.278, 0.343)
(0.137, 2.627)
p 0.000*
p= 0.498
Github_ Presence
1.049
0.886
(1.029, 1.069)
(0.741, 1.034)
= 0.00001
p= 0.147
Constant
742.217
3.558
(322.484, 1,710.418) 0.000"
(0.00003, 400,997.100)
p 0.827
Observations
7,807
217
Log Likelihood
-5,463.652
400.81
Akaike Inf. Crit.
10,933.300
87.561
Note:
*p<0.05; "pps00 **]; *** p<[0. **$]
50

---

R Regression Coefficients for Code Bison Functional
Correctness -- Codeforces (P (Public)
D Ifficuity
GitHub Presence
0.05
0.1
0.5 Odds Ratios
Dependent Variables Pre Cutoff
Cotof
p-level о n.s.
Figure 19: Regression coefficients plots of Pass Rate for Code Bison the Project Codeforces Dataset (evaluated on public test cases only). See Table for regression coefficients.
Pass Rate M Marginal E ffects lott for Code Bison on Codeforces (P (Pullic)
Problems Released Before Cutoff
Problems Released After Cutoff
30%
log(Difficulty)
Pass Rate
20%
7.03
7.44
7.85
10%
0%
10
15
10
15
log(GitHul Presence)
Figure 20: Marginal Effects of Pass Rate for Code Bison on the Codeforces Dataset (evaluated on public test cases only)
51

---

Table 7: Regression table for Pass Rate of GPT-4 on the Codeforces dataset (private test cases only).
See Figure 21 for plot of regression coefficients.
Dependent variable:
Pass Rate (Private)
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.081
0.147
(0.079, 0.084)
(0.129, 0, 0.166)
p 0.000*
p 0.000*
GitHub_Presence
1.044
0.997
(1.037, 1,050)
(0.982, 1.012)
= 0.000
p = 0.684
Constant
49,784,537.00
489,263.300
(38,544.981 1000, 64.349,935.000) = 0.000
(181,481.600 1,331,364.000)
p = 0.000*
Observations Log Likelihood Akaike Inf. Crit.
6,155
811
-56,988.530
2,805.794
113,983. 100
5,617.588
Note:
*p<0.05
Regression Coefficients for GPT4 Functional Correctness - Codeforces (P rivate)
D ifficulty
GitHub Presence
0.05
0.1
0.5
Odds Ratios
Dependent Variables p p-leeel
Pre Cutoff
Custfu
n.s.
Figure 21: Regression coefficients plots of Pass Rate for GPT-4 on the Project Codeforces Dataset (evaluated on private test cases only). See Table for regression coefficients.
52

---

Pass Rate Marginal E Effects Plots for GPT-4 on Codeforces (P rívvate)
Problems Released Before Cutoff
Problems Released After Cutoff
50%
Pass Rate
log(Difficulty)
40%
7.06
7.46
7.86
30%
20%
10
15
10
15
log(GitHub P Pesence)
Figure 22: Marginal Effects of Pass Rate for GPT-4 on the Codeforces Dataset (evaluated private test cases only)
Table 8: Regression table for Pass Rate of GPT-3.5-Turbo the Codeforces dataset (private test
cases only). See Figure 21 for plot of regression coefficients.
Dependent variable:
Pass Rate (Private)
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.112
0.168
(0.109, 0.116)
(0.147, 0.192)
p = 0.000*
= 0.000*
GitHuu_Preeence Presence
1.023
1.014 (0.996, 1.031)
(1.017, 1.030)
= 0.000
p = 0.128
Constant
3,541,250.000
107,436.200
(2.746,064.00) 1.569,487.000) = 0.000*
(37,418.200 311,168.500) = 0.000
Observations Log Likelihood
6,154
811
$49,348.89
22,00.226 268
Akaike Inf. Crit.
98,703.780 780
4,422.537
Note:
*p<0.05
53

---

R Reression Coefficients for GPT 3.5 Turbo Functional Correctness Codeforces (Private)
D Ifficulty
GitHub Presence
0.1
0.2
0.5 Odds Ratios
Dependent Variables Pre Cutoff
Post Cutoff
p-level o n.s.
Figure 23: Regression coefficients plots of Pass Rate for GPT-3.5-Ture the Project Codeforces Dataset (evaluated on public test cases only). See Table 8 for regression coefficients.
Pass Rate Marginal ffects lott for GPT-3.5-Turbo on Codeforces (P rivate)
Problems Released Before Cutoff
Problems Released After Cutoff
50%
log(Difficulty)
Pass Rate
30%
7.06
7.46
7.86
20%
10%
10
15
10
15
log itHub Presence)
Figure 24: Marginal Effects of Pass Rate for GPT-3.5-Turbo on the Codeforces Dataset (evaluated on private test cases only)
54

---

Table 9: Code Bison Functional Correctness Codeforces (Private)
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.215 (0.208, 0.222) p = 0.000
0,305
(0.210, 0,442)
p= 0.000"
Presence
1.031 (1.025, 1.038) p == 0.000
1.040
(0.992, 1,091)
p= 0.111
Constant
15,512.470
574.153
(11,955.844 8840, 20, 133.940)
(32.974, 10,079.9700 970)
p 0.000
0.00002*
Observations Log Likelihood
6,791
132
-36,805.650
211.7771
Akaike Inf. Crit.
73,617.300
429.541
Note:
"p<0.05; "*p<[0,**]] *** p<10. [0****
Regression Coefficients for Code Bison Functional Correctness - Codeforces rPrivate)
D ifficulty
GitHub Presence
0.05
0.1
0.5 Odds Ratios
Dependent Variables Pre Cutoff
Postfuttff
p-leeel о n.s.
Figure 25: Regression coefficients plots of Pass Rate for Code Bison the Project Codeforces Dataset (evaluated on public test cases only). See Table regression coefficients.
55

---

Pass Rate Marginal Effects Plots for Code isonn on Codeforces (P rivate)
Problems Released Before Cutoff
rroblem Released After Cutoff
30%
Pass Rate
log(Difiicult))
20%
7.05
7.46
7.86
10%
10
15
10
15
log(GitHub P resence)
Figure 26: Marginal Effects of Pass Rate for Code Bison the Codeforces Dataset (evaluated private test cases only)
Table 10: Regression table for Pass Rate of GPT4 the Project Euler dataset. See Figure 27 for plot of regression coefficients.
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.145
1.000
104, 0.196)
(0.000, Inf.000) p p= 1,000
p = 0.000*
GirHub_Preeencce Presence
1,476
1.000
(1.127, 1.947) p 0.005*
(0.000, Inf.000)
p 1.000
Constant
0.323
0,000
(0.007, 15.405) 0.567
(0.000, Inf.000)
p 1.000
Observations
765
72
Log Likelihood
-170.511
-0.000
Akaike Inf. Crit.
347.023
6.000
Note:
"p<00.5;; "*p<10.*** **]; *** **p<<00**** +*****
56

---

R Regression Coefficients for GPT4 Functional Correctness - P roject Euler
D ifficulty
GitHub Presence
0.05
0.1
0.5
Odds Ratios
Dependent Variables p-level n.s.
Pre Cutoff
Post Cutoff
Figure 27: Regression coefficients plots of Pass Rate for GPT-4 on the Project Euler Dataset. See Table 10 for regression coefficients.
Pass Rate Marginal E ffects Plots for GPT-4 on Project E Euler
Problems Released Before Cutoff
Problems Released After Cutoff
100%
75%
Pass Rate
Difficulty
2.83
50%
3.62
4.41
25%
0%
12
14
16
12
14
16
GitHub Presence
Figure 28: Marginal Effects of Pass Rate for GPT-4 on the Project Euler Dataset
57

---

Table 11: Regression table for Pass Rate of GPT-3,5-Turbo the Project Euler dataset. See Figure 29 for plot of regression coefficients.
Dependent variable:
Pass Rate
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.166
1.000
10.117, 0.228)
(0.000, Inf.000) p p= 1.000
p 0.000*
GitHub Presence
1,274
1.000
(0.954, 1.714)
(0.000, Inf.000) p = 1.000
p = 0.100
Constant
0.839
0.000
(0.012, 56.905)
(0.000, Inf.000) p = 1.000
= 0.936
Observations
765
72
Log Likelihood
-40.542
-0.000
Akaike Inf. Crit.
287.084
6.000
Note:
"p<0.05; "pp<(0.**) **]; "*p<<0.*** ***]
R Regression Coefficients for GPT 3.5 Turbo
F Functional Correctness - Project Euler
D ifficulty
GitHub Presence
0.05
0.1
0.5
Odds Ratios
Dependent Variables
Pre Cutoff
Post Cutof
p-leeel
n.s.
Figure 29: Regression coefficients plots of Pass Rate for GPT-3.5. 3.rboorrb on the Project Euler Dataset. See Table for regression coefficients.
58

---

Pass Rate Marginal E ffects Pltts for GPT-3.5-Turbo on Project E uler
Problems Released Before Cutoff
Problems Released
Cutoff
100%
75%
Difficulty
Pass Rate
2.83
50%
3.62
4.41
25%
0%
12
14
12
14
16
GitHub Presence
Figure 30: Marginal Effects of Pass Rate for GPT-3.5-Turbo on the Project Euler Dataset
Table 12: Regression table for Title Reproduction GPT-4 on the Codeforces dataset. See Figure 31 for plot of regression coefficients.
Dependent variable:
Title Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.897
0.898
(0.826, 0.974)
(0.756, 1.067)
= 0.010
p 0.222
Presence
1.010
1.005
(0.994, 1.026)
(0.982, 1.028)
= 0.210
p= 0.684
Constant
0.817
0.750
(0.418, 1.598)
(0. (0.191,2.99) 2.956)
= 0.556
p 0.682
Observations
18,446
3,954
Log Likelihood
-11,074.250
-2,265.482
Akaike Inf. Crit.
22,154.490
4,536.963
Note:
*p<0.05
59

---

R Regression Coefficients for GPT4 Title Reproduction - Codeforces
D Ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
p-level
Dependent Variables
Pre Cutoff
Post Cutoff
Figure 31: Regression coefficients plots of Title Reproduction Metric for GPT-4 on the Project Codeforces Dataset. See Table regression coefficients.
Title R Reproduction Marginal E ffects lots for GPT -4 on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
30%
Title Reproduction
log(Difficult)
7.05
27%
7.45
7.86
24%
21%
10
15
10
15
log(GitHub Presence)
Figure 32: Marginal Effects of Title Reproduction Metric for GPT-4 on the Codeforces Dataset
60

---

Table 13: Regression table for Title Reproduction of GPr- the Codeforces dataset. See
Figure 33 for plot of regression coefficients.
Dependent variable:
Title Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.839
0.709
(0.719, 0.979)
(0,510, 0.984)
p 0.026*
0.041
Presence
1,010
0.973
(0.981, 1.041)
(0.931, 1.016)
p = 0.506
p= 0.216
Constant
0.220
0.954
(0.063, 0.764)
(0.071, 13.105)
= 0.018*
p= 0.972
Observations Log Likelihood
18,446
3,954
4,293.516
-869.667
Akaike Inf. Crit.
8,593.032
1,745.333
Note:
"p<00.5
Regression Coefficients for GPT 3.5 Turbo Title Reproduction Codeforces
D ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables p-level
Pre Cutoff
Post Cutoff
n.s.
Figure 33: Regression coefficients plots of Title Reproduction Metric for GPT-3.5-Turie on the Project Codeforces Dataset. See Table regression coefficients.
61

---

Title Reproduction Marginal E Effects Plots for GPT-3.5-Turbe on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
Title Reproduction
log(Difficulty)
8%
7.05
7.45
6%
7.86
4%
10
15
10
15
log(GitHub Presence)
Figure 34: Marginal Effects of Title Reproduction Metric for GPT-3.5-Turbo the Codeforces
Dataset
Table 14: Regression table for Title Reproduction GPT4 the Project Buler dataset. ee Figure 35 for plot of regression coefficients.
Dependent variable:
Title Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.780
0.946
(0.699, 0.870)
(0.631, 1.438)
= 0.00001*
p= 0.793
Giesub_Preeencce Presence
1.014
0,998
(0.962, 1.073)
(0.741, 1.344)
p == 0.608
p= 0.991
Constant
0.848
0.508
(0.349, 1.980)
(0.008, 29.457)
p 0.709
p= 0.745
Observations
2,556
190
Log Likelihood
-1,536.829 829
-114.284
Akaike Inf. Crit.
3,079.658
234.568
Note:
"p<0.05; "=p<10.**) *+; *** *==<>=00**** ***)
62

---

Regression Coefficients for GPT4 Title Reproduction Project Euler
D Ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables p p-leeel
Pre Cutoff
Cutoff
n.s.
Figure 35: Regression coefficients plots of Title Reproduction Metric for GPT-4 on the Project Buler Dataset. See Table 13 for regression coefficients.
Title R Reproduction Marginal ffects Plots for GPT-4 on Project E uule
Problems Released Before Cutoff
Problems Released After Cutoff
100%
75%
Title Reproduction
log(Difficult)
2.91
50%
3.67
4.42
25%
0%
10
15
5
10
15
log(GitHub reesecce)
Figure 36: Marginal Effects of Title Reproduction Metrie for GPT-4 on the Project Fuler Dataset
63

---

Table 15: Regression table for Title Reproduction of GPT-3.5-Turbo on the Project Euler dataset. ee Figure 37 for plot of regression coefficients.
Dependent variable:
Title Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
1.178
0.643
(0.934, 1.517)
(0.136, 4.339)
p 0.184
p 0.589
G
Presence
0.981
1.283
(0.898, 1.094)
(0.339, 6.026)
p = 0.696
p 0.715
Constant
0.044
0.002
(0.007, 0.198)
(0.000, 51,280.610)
p 0.0002
p 0.505
Observations
2,556
190
Log Likelihood
566.246
-10.900
Akaike Inf. Crit.
1.134.492
27.801
Note:
"p<0.05; "'p<[0.**];
p<[0. **
Regression Coefficients for GPT 3,5 Turbo Title Reproduction - Project Euler
D ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables
Pre Cutoff
Post Cutoff
p-leeel
n.s.
Figure 37: Regression coefficients plots of Title Reproduction Metric for GPT-3.5- -errbo on the Project Fuler Dataset. See Table for regression coefficients.
64

---

Title R eproduction Marginal E Effects Plots for GPT-3.5-Ture on Project E uler
Problems Released Before Cutoff
Problems Released After Cutoff
100%
Title Reproduction
75%
log(Difiicult)
2.91
50%
3.67
4.42
25%
0%
10
15
log(GitHub Presence)
Figure 38: Marginal Effects of Title Reproduction Metric for GPT-3.5-Turbo the Project Euler
Dataset
Table 16: Regression table for Tag Reproduction of GPT-4 the Codeforces dataset. See Figure 39 for plot of regression coefficients.
Dependent variable:
Tags Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.431
0.826
(0.400, 0.465)
(0.707, 0.966) p 0.017"
= 0.000*
GitHub Presence GieHubbPresence
0,991
0,999
(0.977, 1.005)
(0.981, 1.017) p = 0.888
= 0.195
Constant
310.696
1.026
(168.396, 573.834)
(0.299, 3.508)
= 0.000*
0.968
Observations Log Likelihood
24,474
6,425
-15,399. 190
13,172.626
Akaike Inf. Crit.
30,804.370
6,351.252
Note:
*p<0.05
65

---

Regression Coefficients for GPT4 Tags Reproduction Codeforces
D Ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables p-level
Pre Cutoff
PostCutoo
n.s.
Figure 39: Regression coefficients plots of Tag Reproduction Metric for GPT-4 on the Project Codeforces Dataset. See Table for regression coefficients.
Tags R Reproduction Marginal E ffects Plots for GPT-4 on Codeforces
Problems Released Before Cutoff
Problems Released After Cutoff
Tags Reproduction
40%
log(Difficult)
7.14
30%
7.52
7.89
20%
10
15
10
15
log(GitHub Presence)
Figure 40: Marginal Effects of Tag Reproduction Metric for GPT-4 on the Codeforces Dataset
66

---

Table 17: Regression table for Tag Reproduction of GPr-o.5-Turbo on the Codeforces dataset. See Figure 41 for plot of regression coefficients.
Dependent variable:
Tags Reproduction
Before Cutoff
After Cutoff
(1)
(2)
Difficulty
0.497
0.739
(0,452, 0.547)
(0.619, 0.883)
p 0.000
p 0.001*
GitHub Presence GresebePreeence
0.959
1.001
(0.942, 0.976)
(0.980, 1.022) p = 0.955
p 0.00001*
Constant
56.613
1,540 (0.380, 6.228) p = 0.545
(26.265, 121.894)
= 0.000*
Observations Log Likelihood
24,474
6,425
-10,635.220
-2,587.031
Akaike Inf. Crit.
21,276,450
5.180.061
Note:
"p<0.05
R Regression Coefficients for GPT 3.5 Turbo Tags Reproduction Codeforces
D ifficulty
GitHub Presence
0.1
0.2
0.5
Odds Ratios
Dependent Variables p-leeel
Pre Cutoff
Cutofu
n.s.
Figure 41: Regression coefficients plots of Tag Reproduction Metric for GPT-3.5-Turbe on the Project Codeforces Dataset. See Table 17 for regression coefficients.
67

---

Tags Reproduction Marginal E ffects Plots for GPT-3.5-Turbo on Codeforces
Problems Released Before Cutoff
Problems Relessed After Cutoff
30%
Tags Reproduction
25%
log(Difficult)
7.14
20%
7.52
7.89
10%
10
15
0
10
15
itH Presence)
Figure 42: Marginal Effects of tag Reproduction Metric for GPT-3.5-Turbo on the Codeforces Dataset
68


Reviews:
[{"review_id": "mQVSo5ovS7", "reviewer": "Reviewer_VA65", "summary": "The paper conducted longitudinal analysis of data contamination in large language models (LLMs), a problem where models are evaluated using data that they may have been trained on, thus overstating their capabilities.  The authors leveraged natural experiments provided by the training cutoff dates of models like GPT-3.5 and GPT-4 to study contamination. They analyzed Codeforces and Project Euler, websites that release code problems over time, and find evidence of contamination based on the pass rate of LLMs for problems released before their training cutoff dates. The study demonstrates statistically significant associations between a problem’s presence on GitHub and LLM performance for pre-cutoff problems.", "questions": "N/A", "limitations": "", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1: The analysis from longitudinal perspective is novel. \n2: The comprehensive experiments, large-scale dataset and code base provided by this work will definitely benefit the community of contamination analysis.\n3: This paper is well organized and easy to understand.", "weaknesses": "1: The results are interesting but not that surprising. Many blogs or discussion in the community about Data Contamination has involved similar results.\n2: There is lack of depth analysis about how implicit contamination is possible. If some real examples can be extracted to show how this could happen, it will be much better.\n\nOverall, I do appreciate the effort to investigate the Data Contamination problem from longitudinal side and open-source data/codes. The experiments also show intriguing results. But I believe the contribution of this paper is not enough to be accepted by ICLR, for its limited scope and technical novelty. It's limited to Code datasets. And the only novelty is how to split the \"train\" and \"test\" set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M4jPZxVj0f", "reviewer": "Reviewer_XbnJ", "summary": "This paper presents a detailed investigation into data contamination in large language models (LLMs), using GPT model training cutoffs to analyze benchmarks released over time. It examines two datasets, Codeforces and Project Euler, revealing clear patterns that suggest contamination based on the LLMs' pass rates correlated with benchmarks' GitHub popularity and release dates. The authors provide a comprehensive dataset, findings, and a framework for future analysis, promoting better practices for benchmark releases in the era of web-scale LLM training.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea to investigate data contamination in LLMs via cutoff datasets makes sense and is interesting, which guarantees that the testing data are not available in the training set of LLMs. And the findings are surprising, revealing that people should deal with the ability of LLMs more carefully. This study shows that LLMs are likely to have generalization problems as well as traditional ML models and deep neural networks. And I think this should raise the attention of ML researchers.", "weaknesses": "I am not quite familiar with LLMs, and I only have one question about the design of cutoffs. What if a code problem released later is exactly similar as some problems that has already existed? And how to measure the data contamination problem is also important.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gb4bzg4HXx", "reviewer": "Reviewer_EfiM", "summary": "Assess whether GPT performance at coding (sometimes called program synthesis) was possibly affected by contamination of pretraining data using a naturally occurring experiment (i.e. comparing scores before and after the pretraining knowledge cutoff dates).", "questions": "Not a question, but I want to note that:\n\n1. I like the use of Pass Rate in lieu of pass@1. I think that's a very sensible choice.\n\n2. I like the citation of Horace He's and Chris Cundy's tweets. Very good scholarship, even if Tweets aren't \"published\" in a traditional sense.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, I really liked this paper. I thought it was well motivated, clearly conceptualized, well executed and somewhat thorough. I have a small number of requested changes, and if the authors and I agree that the changes are sensible and if the authors agree to make the changes, I would be happy to increase my score.", "weaknesses": "> Figure 1: Marginal Effects of Pass Rate Metric \n\nI think this is an amazing figure. 5 comments, ordered from minor to major:\n\n1. easy: Stacking log(Github Presence) and log(Difficulty) at the bottom makes reading the figure tricky. I might suggest moving log(Difficulty) to the right side.\n\n2. easy: GitHub is stylized \"GitHub\", not \"Github\"\n\n3. medium: Where is the equivalent plot for Project Euler? I might have missed this, but I cannot find it in the main text or appendix.\n\n4. hard: The pass rate is significantly lower for easy and medium problems, even for log(Github Presence) = 0. I understand that GitHub Presence is a proxy, but I would think that log(GitHub Presence) = 0 is our best guess for \"low or no contamination\", but there's still a 10-20% decrease in pass rate. Why? I can think of 2-4 possible answers: (a) GPT-4 genuinely becomes much worse after the knowledge cutoff; (b) GitHub presence is inadequate and/or misleading, (c) the distribution of Codeforce problems changed after GPT-4 was finished pretraining, or (d) something changed in how the pass rate is calculated on generated outputs. More explanations might also be possible. Is there some way for the authors to try to investigate the cause of this shift?\n\n5. hard: I was hoping for either a qualitative or quantitative analysis about what GPT-4 is outputting on Codeforces problems released after the cutoff, but I can't find even a single example of the raw generated outputs. Could the authors please provide some manual examples, even in the appendix, to convincingly demonstrate that GTP-4 is indeed outputting worse code? I want to rule out that silly possibilities (e.g., a shift in formatting) are affecting the results.\n\n> Table 1\n\nI personally find Tables are less effective at communicating than Figures. Since these are regression tables, could you possibly consider switching to a Forest plot of regression coefficients? Some random examples here:\n\n- https://www.researchgate.net/figure/Forest-plot-of-regression-coefficients-95-confidence-interval-for-the-association_fig1_331119872\n- https://www.researchgate.net/figure/Coefficient-plots-from-linear-regression-predicting-what-makes-an-interaction-meaningful_fig1_343608677 \n- http://www.strengejacke.de/sjPlot/reference/plot_models.html.\n\nTo make my suggestion as concrete as possible, using terminology from matplotlib & seaborn (assuming you're using Python, but I'm sure R could do this as well), I'm specifically thinking that your X axis should be the estimated parameters and confidence intervals, Y would be the covariates (i.e. Difficulty & GitHub presence), the Hue is either Before Cutoff or After Cutoff, and you have two side-by-side axes, one for GPT4 and the other for GPT3.5.\n\nI personally would prefer all regression tables to be visualized as such (Tables 1, 2, and those in the appendix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRVbsbQxXL", "reviewer": "Reviewer_yfjE", "summary": "The paper investigates data contamination of GPT-3.5-Turbo and GPT-4 with problems from Codeforces and Project Euler. It does so by analysing the passrates in relation to the GitHub presence and notes a positive correlation for data before the cutoff, but no significant correlation after this date.", "questions": "### Minor Comments\n* Particularly in section 2, some citations are formatted differently, with the author names outside the parentheses; in sequences of different citations, readability could be improved by using the same citation format as in section 1.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Identifying data contamination is an important issue, especially for evaluation datasets that are often used to create rankings.  \n2. Including problem difficulty as an independent variable is an important step in isolating the confounding effect of item difficulty on pass rates.\n3. I appreciate the openness in referencing blog posts and tweets that anecdotally suggested possible contamination prior to this work", "weaknesses": "1. The methodology is only applied to GPT-3.5/GPT-4, where training details are unknown. In particular, as noted in footnote 1, OpenAI has admitted to using a small amount of data beyond the cutoff date. While I understand the choice of the GPT family as a commonly used model, it would have been better to verify the approach with fully open models where more training details are available (and more trustworthy).\n2. The methodology requires underlying datasets that are longitudinal in nature, i.e. release problems/individual tasks over time; this limits the applicability to sources other than Project Euler / Codeforces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)