Competition-Level Problems are Effective LLM Evaluators (2024)

Abstract: Large language models (LLMs) have demonstrated impressive reasoning capabilities, yet there is ongoing debate about these abilities and the potential data contamination problem recently. This paper aims to evaluate the reasoning capacities of LLMs, specifically in solving recent competition-level programming problems in Codeforces, which are expert-crafted and unique, requiring deep understanding and robust reasoning skills. We first provide a comprehensive evaluation of GPT-4’s perceived zero-shot performance on this task, considering various aspects such as problems’ release time, difficulties, and types of errors encountered. Surprisingly, the perceived performance of GPT-4 has experienced a cliff like decline in problems after September 2021 consistently across all the difficulties and types of problems, which shows the potential data contamination, as well as the challenges for any existing LLM to solve unseen complex reasoning problems. We further explore various approaches such as fine-tuning, Chain-of-Thought prompting and problem description simplification. Unfortunately, none of them is able to consistently mitigate the challenges. Through our work, we emphasize the importance of this excellent data source for assessing the genuine reasoning capabilities of LLMs, and foster the development of LLMs with stronger reasoning abilities and better generalization in the future.

heading: 1 Introduction
text: The rise of LLMs has generated significant interest in the artificial intelligence community. These models, notably GPT-4 (OpenAI, 2023), have displayed impressive reasoning capabilities that are being harnessed in various fields (Bubeck et al., 2023). However, questions1 have been raised about
* Equal contribution. † This work was done during their internship at MSRA. ‡ Corresponding authors. 1https://twitter.com/keirp1/status/
1724518513874739618
how to accurately evaluate the reasoning abilities of LLMs and the extent of data contamination issues (Mialon et al., 2023; Zhou et al., 2023).
Regarding these issues, our study aims to assess the reasoning capabilities of LLMs through their ability to generate algorithms for solving competition-level programming problems. These questions are meticulously crafted by experts to form rigorous competitions. They possess high quality, are unique, and exhibit excellent discriminative ability. The testing cases are also meticulously prepared. This necessitates that LLMs deduce the solution from the presented scenario, which requires a thorough understanding of algorithms, combined reasoning and coding skills, and strong problem-solving abilities. These problems thus present a significant challenge to both human coders and LLMs. Consequently, competition-level programming problems serve as effective tools for evaluating the two issues previously discussed: they assess the reasoning abilities of LLMs and, due to the strict problem selection process in competitions, reduce the likelihood of data contamina-
13526
tion in new problems. Our research provides an in-depth analysis of the zero-shot performances of GPT-4 and other code LLMs on competition-level programming problems in Codeforces, considering factors such as release time, problem difficulty, and the types of errors encountered. The main insights of our study include: (1) GPT-4 performs significantly worse on programming problems released after September 2021, casting doubt on its actual reasoning abilities. (2) GPT-4 shows limited capability to solve difficult problems, indicating potential weaknesses in complex problem-solving. (3) GPT-4 struggles with the first test case, suggesting errors may stem from its understanding of the problem at hand. (4) The related phenomenon can be also observed in other LLMs, indicating that insufficient reasoning ability may be a common problem.
To explore possible ways to enhance the zeroshot performances of these LLMs on competitionlevel programming problems, we investigate several methods to improve performance on unseen problems. These methods include supervised fine-tuning with code-specific LLMs, Chain-ofThought prompting (Wei et al., 2022), and problem statement simplification. Specifically, we fine-tuned CodeLlama(Rozière et al., 2023) and DeepSeek-Coder(AI, 2023), which are specialized language models designed to handle programmingrelated tasks. However, none of these methods consistently mitigated the issue or resulted in noticeable performance improvements, particularly for more difficult problems. This finding indicates that difficult and unseen programming problems are effective evaluators of LLMs.
Overall, the primary contributions of this study lie in proposing and validating that recent competition-level programming problems serve as an excellent data source for assessing the genuine reasoning capabilities of LLMs. We aim to foster further research in this field by innovating new approaches to address the challenge of complex reasoning problems in LLMs and by establishing reliable evaluation benchmarks for LLMs that minimize the risk of data contamination.

heading: 2 Problem Setup

heading: 2.1 Competition-level Programming
text: Competition-level programming presents a unique arena for testing and developing the reasoning abilities of AI models. In competitive programming, a
problem typically consists of a narrative that sets the context, which models need to understand and convert into an algorithmic problem. The challenge lies in comprehending the narrative, identifying the underlying algorithmic issues, and implementing an efficient solution in programming languages such as C++ and Java. Accepted programs must satisfy stringent testing conditions, including producing outputs that exactly match with test cases, executing within memory limits, and terminating within time constraints. In contrast to prior works (Chen et al., 2021a; Austin et al., 2021; Cassano et al., 2023) focusing on basic coding abilities, competition-level programming problems require advanced reasoning and mathematical modeling skills, essential for AI.
Unlike the previous works that focused on LeetCode2 (Bubeck et al., 2023; Shen et al., 2023; Sakib et al., 2023), we follow AlphaCode (Li et al., 2022) and choose Codeforces3. Codeforces is universally acknowledged by competitors and enthusiasts in the International Collegiate Programming Competition4 (ICPC) and the International Olympiad in Informatics5 (IOI) as a popular and suitable platform for developing abilities for algorithm contests. The regular contests hosted on this platform are crafted by human experts, and contain plenty of intricate programming problems and contests of high quality. These contests come with comprehensive and robust test cases and exhibit a low degree of problem overlap. The unique nature of these contest problems makes it highly unlikely to find similar content on the internet before the competition concludes. As a result, utilizing specific time-segmented datasets, like those from contests conducted post the introduction of LLMs, serves as an effective strategy to prevent data contamination (Zhou et al., 2023).
Codeforces employs the Elo rating system6
to rank its users and problems, categorizing all problems into 28 distinct difficulties, ranging from 800 to 3500. Compared to commonly utilized metrics such as the ratio of accepted submissions or users, this difficulty rating mechanism is more suitable as it is based on the ranking and performance of the participants during the competition.7
2https://leetcode.com/ 3https://codeforces.com/ 4https://icpc.global/ 5https://ioinformatics.org/ 6https://codeforces.com/blog/entry/102 7https://codeforces.com/blog/entry/62865
Hence, it is not subject to inaccuracies stemming from temporal changes, repeated submissions, plagiarism, and other potential distortions.

heading: 2.2 Problem Definition
text: Figure 2 presents an example of the problem statement π. The input of LLM is instantiated with the problem statement π and a prompt ρ (like ρ1 in Table 8). The LLM Γ takes the input to generate the code as α = Γ(ρ(π)). The generated code α is then evaluated by an online judge (OJ). The evaluation process can be summarized in the following equation:
OJ(α) = OJ(Γ(ρ(π))) ∈ {AC, WA, CE, ...}
In this equation, Γ(ρ(π)) denotes the code generated by LLM with the prompt ρ. The OJ platform then rigorously assesses the code for its correctness, computational efficiency, and adherence to specified input/output formats. With an extensive testing mechanism, the platform employs a wide range of test cases and hidden scenarios to ensure the code’s robustness across diverse scenarios. The platform provides a spectrum of outcomes, OJ(Γ(ρ(π))), offering a holistic evaluation of the code’s performance. This includes results such as Accepted
(AC), Wrong Answer (WA), and Compilation Error (CE), among others.

heading: 2.3 Dataset Collection
text: The dataset is compiled from the Codeforces website, extracting all publicly available problem statements from completed contests spanning February 2010 through November 2023. For simplicity, problems requiring interaction, featuring non-standard input/output formats, or incompatible with C++ submission are excluded. For detailed explanations, see Appendix B.
The analysis is confined to problems with difficulty levels ranging from 800 to 2400. Based on their difficulty levels, the dataset is divided into three subsets: D1 (800-1100 difficulty, 1683 problems), D2 (1200-1600 difficulty, 1821 problems), and D3 (1700-2400 difficulty, 1453 problems). These problems encompass more than 20 distinct categories of algorithms, as illustrated in Table 7. This diversity in problem types further enhances the comprehensiveness of the dataset and enables a comprehensive assessment of GPT-4’s problem-solving abilities across a wide range of competition-level programming problems.

heading: 2.4 Experiment Details
text: In Codeforces, each problem belongs to a contest. Once the contest concludes, the problems are disclosed and become publicly submittable. Therefore, we submit the solutions to the contests that have concluded for evaluation.
To evaluate the results, we employ ACC#G, ACC#GN, ACCk#n and pass@k as defined in Table 2. Specifically, for ACCk#n metric, we consider two settings: (1) k = n = 1 and (2) k ∈ {1, 2, 3} with n = 5. Following Codex (Chen et al., 2021b), pass@k is computed as
pass@k := E Problems
[ 1− ( n−c k ) ( n k ) ]
where n is defined as the total number of generated samples per problem used for evaluation, and c represents the count of correct samples out of n that have successfully passed the unit tests. Here we use k = 1 and n = 5 for pass@k.
In our experiment, we follow the zero-shot setting. To select an appropriate prompt, we conduct preliminary experiments with three prompts, ρ1, ρ2, and ρ3, as listed in Table 8, using two subsets of D1 problems: one from February to December 2010 and the other from January to October 2023, each comprising approximately 100 problems. The standard deviations are 0.015 and 0.018, respectively, indicating consistent performance. Therefore, we
choose ρ1 as the prompt in the subsequent experiments. Furthermore, we employ a sliding window approach for all temporal analyses to smooth the data, addressing the sporadic release schedule of the problems. This ensures a sufficient number of test problems at each time point, using a window size of 51 (25 before and 25 after the time point).

heading: 3 Insights and Implications

heading: 3.1 Faltering on Unseen Problems
text: In this section, we delve into a temporal analysis of GPT-4 (gpt-4-0613)’s performance on programming problems. Figure 1 illustrates GPT-4’s performance using the ACC#G metric. On problems released prior to September 2021, GPT-4 exhibits minor fluctuations at different levels across problems of varying difficulty. However, for problems released after September 2021, a significant deviation from the normal fluctuation range is observed. Interestingly, this timing coincides with the cutoff date for the GPT-4 training data as announced by OpenAI8. We then calculate the average performance on problems before and after September 2021, as shown in Table 1. On D1 problems, GPT4’s ACC#G plummets from 81.42% to 11.73%, marking a stark decrease of 69.69%. Even more strikingly, the ACC#G drops to 0.00% on both D2 and D3 problems, from 43.72% and 11.41%, respectively. To validate the reliability of the conclusion, we also calculate the pass@1 metric, which exhibits a similar trend. This observation raises thought-provoking questions about the severity of the drop and the correlation between the data cutoff date and the performance decline.
To explore the model’s potential to generate correct solutions, we perform random sampling multiple times and calculate the pass rate. The average pass rate are shown in Table 1. As observed, multiple samplings can enhance the chances of gen-
8https://platform.openai.com/docs/models/ gpt-4-and-gpt-4-turbo
erating a correct solution. For instance, on the unseen simple D1 problems, ACC1#5 improved by 10.71% compared to ACC1#1. However, across all problems, the performance gap before and after the cut-off date is more pronounced for ACC1#5 than for both ACC1#1 and ACC#G. Figure 3 depicts the performance on D2 problems over time. A notable decline in performance metrics is observed around September 2021. This observation underscores the challenges that LLMs, including the advanced GPT-4, face in addressing unseen programming problems without similar pretraining data.
The observed decline in performance on problems outside the model’s training range may stem from limitations in reasoning and generalization. As highlighted by Yadlowsky et al. (2023), when confronted with problems beyond their pretraining data, transformer models exhibit various failure modes and their generalization abilities deteriorate, even for simple problems. Similarly, Lu et al. (2023) suggest that the exceptional abilities of large language models primarily stem from in-context learning, and do not necessarily reflect the emergence of reasoning abilities.
The observed performance drop on unseen problems raises serious questions about GPT-4’s intrinsic reasoning and generalization capabilities. This suggests a potential over-reliance on pattern recognition and reproduction from training, as opposed to grasping underlying principles and applying them to novel problems. This observation aligns with recent debates on large models’ data memorization tendencies (Carlini et al., 2023; Yang et al., 2023). Therefore, future evaluations should
prioritize the minimization of overlap between testing and training data to accurately assess a model’s reasoning abilities, rather than simply its capacity for memorization. Furthermore, it’s crucial to explore methods that enhance model generalization and reduce reliance on pre-training data.

heading: 3.2 Limited Ability to Solve Difficult Problems
text: This section provides an analysis of performance in relation to the problem difficulty. The results of ACC#G for problems with different difficulties are reported for two distinct periods: from October 2010 to September 2021, and from October 2021 to November 2023, as illustrated in Figure 4.
For the results from October 2010 to September 2021, we calculate Pearson correlation coefficient (r = −0.97) and the Kendall rank correlation coefficient (τ = −0.88), which indicate strong linear correlations. Notably, when the difficulty level reaches 2400 (indicating greater challenge than approximately 57% of the problems on Codeforces), the ACC#G drops to zero. However, from October 2021 to November 2023, ACC#G shows a dramatic decrease across all difficulty levels.
These findings reveal a significant limitation in the ability of GPT-4 to handle extremely complex problems. Despite its vast knowledge on code and algorithms, GPT-4 lacks of the competence in solving very challenging problems, particularly those with higher difficulty levels, even in the context of previously encountered problems. This indicates a potential area for further improvement and devel-
opment in future iterations of the model.

heading: 3.3 Struggling with The First Test Case
text: In this section, we gather and analyze the errors returned by GPT-4 upon submission to the Codeforces website, as outlined in Table 6. The most common error is "Wrong answer on test 1", which on average accounts for 70% of the observed errors. Test 1 is the first test case, which almost corresponds to or properly includes the example test case provided in the problem statement. This suggests that the model often struggles at the very beginning of problem-solving, possibly due to difficulties in understanding the problem’s requirements or generating a correct solution based on the given test case. As depicted in Figure 5, there is a significant increase in the proportion of "Wrong answer on test 1" errors for problems released between 2021 and 2023. This suggests that GPT-4 is more likely to face challenges in understanding and reasoning during at the onset of tackling unseen problems.
Other types of errors account for a smaller proportion, with an average of 10%. They have shown little variation over time. This indicates that GPT4 demonstrates strong fundamental code-writing capabilities of generating high-quality code.

heading: 3.4 Similar Phenomenon of Other Code LLMs
text: We investigate whether the perceived performance degradation on unseen programming problems is observed for other popular code LLMs, such as
CodeLlama-34B-Instruct (Rozière et al., 2023) and DeepSeek-Coder-33B-Instruct (AI, 2023).
We conduct tests on CodeLlama and DeepSeekCoder using D1 problems, following the settings in §2.3, and the results are shown in Figure 6. The experimental results indicate that CodeLlama consistently underperforms compared to DeepSeekCoder on D1 problems. Furthermore, the performance of DeepSeek-Coder on D1 problems has been declining with the progression of the problem release date. The ACC#GN of DeepSeek-Coder has declined to a level that is on par with CodeLlama when dealing with newly released problems, as highlighted in the red area of Figure 6.
To precisely and intuitively detect this phenomenon, we calculate the ACC#G of CodeLlama and DeepSeek-Coder on D1 problems, both before and after March 2023, and present the results in Table 3. The results reveal a significant difference in the average accuracy of CodeLlama and DeepSeekCoder before and after March 2023. Regarding the magnitude of the decrease, DeepSeek-Coder, which previously exhibited superior performance, demonstrates a more pronounced decline, with acceptance rates falling below 10% after March 2023. Considering the release dates of CodeLlama and DeepSeek-Coder, we speculate that most of the programming problems after March 2023 are novel to them, which suggests that they also not be able to perform well on unseen programming problems like GPT4 does. This finding indicates that a fun-
damental limitation of current code LLMs in generalizing effectively to complex reasoning tasks.

heading: 3.5 Evaluation Hallucination of LLMs
text: To further analyze the phenomenon, we compare GPT-4 with DeepSeek-Coder on D1 problems as shown in Figure 7 and Table 4.
It is noteworthy that while GPT-4 surpasses DeepSeek-Coder in terms of performance on problems that were released prior to September 2021, an unexpected observation is that DeepSeek-Coder exhibits a performance that is on par with GPT-4 when it comes to tackling problems that were released after September 2021. Considering the previous work (Yang et al., 2023; Zhou et al., 2023), although GPT-4 may perform particularly well on some previously seen problems due to its powerful capacity, it cannot be well generalized on unseen programming problems, and its performance is not significantly different from DeepSeek-Coder, which is specifically trained for code. This phenomenon merits attention, which is termed as “evaluation hallucination”.
Hence, a more equitable evaluation strategy would be to select evaluation sets that all the models have not previously encountered. However, finding such data adhering to stringent conditions is challenging, as LLMs are typically pre-trained on extensive corpora containing diverse content, leading to the potential issue of data contamination. Therefore, if we could devote more attention to the data source and timeline of the evaluation sets, such
as the problems in Codeforces, it could potentially mitigate the effects of evaluation hallucination.

heading: 4 One Step Forward
text: In this section, we explore some approaches to mitigate the poor performance on unseen problems.

heading: 4.1 Fine-tuning
text: Fine-tuning is a commonly used method to improve performance on specific downstream tasks. In this study, we employ the Description2Code dataset (OpenAI and Sutskever, 2016) for fine-tuning. This dataset is compiled from three competitive programming websites: Codeforces, CodeChef, and HackerEarth, and contains problems published before 2017. CodeChef and HackerEarth, similar to Codeforces, host online coding competitions, and their problem sets are consistent in style and difficulty.
The dataset includes 2128 problems from Codeforces, 2435 problems from HackerEarth, and 3201 problems from CodeChef, totaling 7764 problems. However, due to some problems lacking
corresponding C++ solutions, we retained a total of 7000 problems for our study. Each problem has approximately 10 C++ solutions, resulting in 70,000 pairs of input-output sequences. These sequences are used for fine-tuning both CodeLlama and DeepSeek-Coder in a supervised manner.
As shown in Figure 8 and Figure 9, we compare the performances of the models before and after fine-tuning on D1 problems. We observe that, even after fine-tuning with the same type of data, CodeLlama and DeepSeek-Coder do not exhibit improved performance on recent problems, particularly those post-2022. The significant improvement in ACC#GN before 2017 may result from the models recalling relevant or identical programming problems, rather than mastering the underlying reasoning logic, leading to their inability to adapt well to new programming challenges. Therefore, simple fine-tuning does not effectively enhance the models’ performance on new programming problems.

heading: 4.2 Chain-of-Thought Prompting
text: In this section, we explore the application of Chainof-Thought (CoT) prompting (Wei et al., 2022) to competition-level programming problems. CoT involves prompting GPT-4 to generate an explanation of the algorithm before coding, denoted as ρcot in Table 8. We conduct experiments on both the D1 and D3 problems released after October 2021. For D1 problems, employing CoT increases the ACC#G from 11.54% to 16.21%, demonstrating a noticeable improvement. However, for D3 problems, using CoT fails to yield any improvement, leaving the ACC#G at 0.00%. This suggests that while CoT facilitates some improvement for simple D1 problems, it is ineffective for the complex reasoning challenges presented by D3 problems.

heading: 4.3 In-Context Learning
text: In this section, we enhance our experimental exploration into in-context learning by integrating both fixed demonstrations and retrieval-augmented demonstrations.
We use D1 problems released before September 2021 as source dataset and those released after as test data. First, we apply the same method as delineated in §4.2 to tackle the source data and validate them on Codeforces. We then retain accepted solutions, resulting in a collection of 1048 problem and CoT response pairs. A demonstration example is presented in Table 12.
In the fixed demonstration experiment, we randomly select three problems to create 3-shot prompts. In the retrieval-augmented demonstration experiment, we first generate embeddings for the statements of the source and test data problems utilizing the OpenAI text-embedding-ada-002 model. We then identify the top three problems in the source data based on cosine similarity for each test problem, incorporating them as example demonstrations within the prompts.
The experimental results, summarized in Table 5, show that the retrieval-augmented 3-shot method’s accuracy is nearly identical to the 0-shot CoT, while the fixed 3-shot approach is even less effective. This may be due to the highly specialized nature of competitive programming problems, which makes finding valuable references challenging. Furthermore, the model may struggle to acquire problemsolving skills through context learning alone, and inappropriate demonstrations might lead to adverse effects.

heading: 4.4 Problem Statement Simplification
text: Intuitively, even experienced programming competition competitors require time to understand problem statements. Therefore, we conduct a simple experiment to assess whether comprehension of problem statements hinders LLMs’ ability to excel at programming problems. We first instruct GPT-4 to simplify the problem statement with ρsip and then generate the code with ρsipgen as shown in Table 8. The results are also evaluated on both the D1 and D3 problems released after October 2021. However, for D1 problems, using the simplified problem statement even brings a slight decline in ACC#G from 11.54% to 11.14%. And the ACC#G for D3 problems still remains at 0.00%. Consequently, the challenge of genuinely improving the model’s reasoning ability and enhancing its performance on unseen problems represents a significant direction for future research.

heading: 5 Related Work
text: Code LLMs. Code intelligence is an important topic in AI research. Recently, code LLMs (Zhang et al., 2023b) have received widespread attention. Commercial LLMs (OpenAI, 2023) have achieved tremendous success. Meanwhile, research on open-source code LLMs is also thriving, such as CodeLlama (Rozière et al., 2023), StarCoder (Li et al., 2023), CodeGeeX (Zheng et al., 2023), CodeFuse (Di et al., 2023), WizardCoder (Luo et al., 2023) and Lemur (Xu et al., 2023).
Reasoning on Code. Programming competition is a specialized domain within the broader landscape of programming problems. Unlike simpler tasks on code, such as HumanEval (Chen et al., 2021a), MBPP (Austin et al., 2021), MultiPLE (Cassano et al., 2023), competition-level programming problems necessitate an advanced understanding of data structures, algorithms, and problem-solving techniques. Enabling models to solve human-designed algorithmic competition problems represents a meaningful research direction, as it reflects the models’ integrated capabilities in reasoning, coding, and problem-solving. AlphaCode (Li et al., 2022) simulate evaluations on 10 programming competitions on the Codeforces platform, which is the first work in this topic. ALGO (Zhang et al., 2023a) can integrate with any existing code LLMs in a model-agnostic manner, enhancing its code generation performance.
Reasoning on Other Subjects. Researchers have proposed many benchmarks requiring various reasoning skills, including commonsense reasoning (Talmor et al., 2018; Geva et al., 2021), numerical reasoning (Dua et al., 2019), multi-hop reasoning (Yang et al., 2018), arithmetic reasoning (Patel et al., 2021; Cobbe et al., 2021), structured reasoning (Yu et al., 2018; Lei et al., 2023), inductive reasoning (Sinha et al., 2019) and logical reasoning (Yu et al., 2020). LLMs are also widely used in scientific research in other fields (Wang et al., 2023), such as physics (Yeadon and Halliday, 2023), chemistry (Castro Nascimento and Pimentel, 2023; Bran et al., 2023), etc.

heading: 6 Conclusion
text: In this study, we utilize competition-level programming problems from Codeforces to analyze the reasoning capabilities of LLMs. We find a significant
decrease in perceived performance of GPT-4 on unseen problems, consistent across a range of difficulties, problem types, and experimental settings. This decrease highlights concerns of data contamination in benchmarks and the need for unseen tasks to properly assess LLMs’ reasoning ability with complex challenges. Our research also extends these insights to other open-source LLMs, revealing the common difficulties these models face with complex, previously unencountered reasoning tasks. This is indicative of the LLMs’ intrinsic limitations in reasoning. As a primary probe, we explore several straightforward strategies, but none of them consistently mitigated the issues. Through our work, we hope to emphasize the critical need for robust datasets to accurately evaluate LLMs’ reasoning abilities and to inspire advancements in LLMs that demonstrate improved reasoning abilities.
Limitations
This study identifies expertly-designed, highquality competition-level programming problems as effective evaluation data for evaluating LLMs. However, comparing to the existing benchmarks, the quantity of such problems is limited. Constructing uncontaminated, high-quality evaluation datasets and extending them to other tasks such as mathematics still poses a challenge to researchers. The identification and creation of such datasets are crucial for enhancing our understanding of the LLMs in complex reasoning tasks. We will endeavor to achieve this goal in our future work.

heading: A More Results with Different Versions of GPT-4 APIs
text: In this study, we conduct an evaluation of two distinct API versions: GPT-4 and GPT-4-turbo, to assess their performance on D1 problems. The training data for GPT-4 extends up to September 2021, while that for GPT-4-turbo reaches up to April 2023. These evaluations are visually represented in Figure 10. Upon analysis of the results, it is observed that on problems prior to September 2021, the GPT-4-turbo exhibits marginally inferior performance compared to GPT-4. Between September 2021 and April 2023, GPT-4-turbo outperforms GPT-4 on D1 problems, reflecting the benefits of its more recent training data. Nonetheless, a decline in GPT-4’s performance is observed for newer problems within this period, likely due to the scarcity of such recent data in its training set.
Nevertheless, when faced with problems emerging after April 2023—thus unencountered during their respective training periods—both APIs demonstrate a decline in performance, albeit GPT4-turbo marginally outperforms GPT-4. Despite this relative improvement, the performance of GPT4-turbo on problems post-April 2023 noticeably regresses when compared to its performance on problems covered by its training data. This finding is consistent with the conclusions drawn in the §3.1 "Faltering on Unseen Problems", which elucidates the challenges faced by these models when confronted with novel questions that extend beyond their training corpus.

heading: B Dataset Details
text: In the context of competitive programming challenges, a "non-standard input/output format" typically refers to a situation where the program’s input and output are not provided through standard methods such as reading from standard input (stdin) or writing to standard output (stdout), which are the conventional ways for programs to receive and provide data during competitions. Instead, they might involve interacting with files, graphical user interfaces, or network connections, which are not commonly used in standard programming contests (like https://codeforces.com/problemset/problem/120/A). To filter out problems with non-standard input/output formats, we utilize metadata from the problem descriptions on Codeforces, which indicate whether a problem requires non-standard methods for input and output. By checking this information, we can automatically exclude such problems from our dataset to ensure the consistency of the test data.
Statistics of the types of problems in D1, D2, and D3 are shown in Table 7.

heading: C Prompt Details
text: Prompts used in this study are shown in Table 8.

heading: D Case Study
text: Some examples generated by GPT4 are shown in Tables 9–13.

heading: 2023 0.90 0.03 0.00 0.00 0.03 0.01 0.02 0.01

heading: 2022 0.91 0.05 0.00 0.00 0.02 0.00 0.02 0.00

heading: 2021 0.82 0.07 0.02 0.01 0.02 0.04 0.02 0.00

heading: 2020 0.77 0.06 0.01 0.01 0.05 0.06 0.03 0.01

heading: 2019 0.67 0.11 0.04 0.06 0.02 0.05 0.02 0.02

heading: 2018 0.58 0.12 0.06 0.13 0.01 0.03 0.05 0.01

heading: 2017 0.66 0.13 0.05 0.08 0.03 0.03 0.03 0.02

heading: 2016 0.68 0.09 0.02 0.10 0.02 0.05 0.04 0.00

heading: 2015 0.62 0.10 0.02 0.14 0.03 0.02 0.04 0.02

heading: 2014 0.53 0.18 0.05 0.09 0.04 0.05 0.04 0.01

heading: 2013 0.63 0.13 0.06 0.07 0.03 0.03 0.04 0.01

heading: 2012 0.55 0.14 0.05 0.12 0.04 0.04 0.03 0.02

heading: 2011 0.53 0.13 0.10 0.06 0.07 0.06 0.04 0.01

heading: 2010 0.49 0.15 0.04 0.19 0.04 0.03 0.01 0.04
text: Problem Name 1890 B: Qingshan Loves Strings Accepted
Problem Statement Problem Qingshan has a string s, while Daniel has a string t. Both strings only contain 0 and 1. A string a of length k is good if and only if ai ̸= ai+1 for all i = 1, 2, . . . , k − 1. For example, 1, 101, 0101 are good, while 11, 1001, 001100 are not good. Qingshan wants to make s good. To do this, she can do the following operation any number of times (possibly, zero): insert t to any position of s (getting a new s). Please tell Qingshan if it is possible to make s good. Input Format The input consists of multiple test cases. The first line contains a single integer T (1 ≤ T ≤ 2000) — the number of test cases. The description of the test cases follows. The first line of each test case contains two integers n and m (1 ≤ n,m ≤ 50) — the length of the strings s and t, respectively. The second line of each test case contains a string s of length n. The third line of each test case contains a string t of length m. It is guaranteed that s and t only contain 0 and 1. Output Format For each test case, print "YES" (without quotes), if it is possible to make s good, and "NO" (without quotes) otherwise. You can print letters in any case (upper or lower).
Problem Name 1893 A: Anonymous Informant Wrong answer on test 1
Problem Statement Problem You are given an array b1, b2, . . . , bn. An anonymous informant has told you that the array b was obtained as follows: initially, there existed an array a1, a2, . . . , an, after which the following two-component operation was performed k times: A fixed point† x of the array a was chosen. Then, the array a was cyclically shifted to the left‡ exactly x times. As a result of k such operations, the array b1, b2, . . . , bn was obtained. You want to check if the words of the anonymous informant can be true or if they are guaranteed to be false. †A number x is called a fixed point of the array a1, a2, . . . , an if 1 ≤ x ≤ n and ax = x. ‡A cyclic left shift of the array a1, a2, . . . , an is the array a2, . . . , an, a1. Input Format Each test contains multiple test cases. The first line contains an integer t (1 ≤ t ≤ 104) — the number of test cases. The description of the test cases follows. The first line of each test case contains two integers n, k (1 ≤ n ≤ 2 · 105, 1 ≤ k ≤ 109) — the length of the array b and the number of operations performed. The second line of each test case contains n integers b1, b2, . . . , bn (1 ≤ bi ≤ 109) — the elements of the array b. It is guaranteed that the sum of the values of n for all test cases does not exceed 2 · 105. Output Format For each test case, output "Yes" if the words of the anonymous informant can be true, and "No" if they are guaranteed to be false.
Problem Name 1893 A: Anonymous Informant Compilation error
Problem Statement Problem You are given a set S, which contains the first n positive integers: 1, 2, . . . , n. You can perform the following operation on S any number of times (possibly zero): Choose a positive integer k where 1 ≤ k ≤ n, such that there exists a multiple of k in S. Then, delete the smallest multiple of k from S. This operation requires a cost of k. You are given a set T , which is a subset of S. Find the minimum possible total cost of operations such that S would be transformed into T . We can show that such a transformation is always possible. Input Format The first line of the input contains a single integer t (1 ≤ t ≤ 10 000) — the number of test cases. The description of the test cases follows. The first line contains a single positive integer n (1 ≤ n ≤ 106). The second line of each test case contains a binary string of length n, describing the set T . The i-th character of the string is ’1’ if and only if i is an element of T , and ’0’ otherwise. It is guaranteed that the sum of n over all test cases does not exceed 106. Output Format For each test case, output one non-negative integer — the minimum possible total cost of operations such that S would be transformed into T .
Problem Name 451 A: Game With Sticks Accepted
Problem Statement Problem After winning gold and silver in IOI 2014, Akshat and Malvika want to have some fun. Now they are playing a game on a grid made of n horizontal and m vertical sticks. An intersection point is any point on the grid which is formed by the intersection of one horizontal stick and one vertical stick. In the grid shown below, n=3 and m=3. There are n+m=6 sticks in total (horizontal sticks are shown in red and vertical sticks are shown in green). There are n·m=9 intersection points, numbered from 1 to 9. The rules of the game are very simple. The players move in turns. Akshat won gold, so he makes the first move. During his/her move, a player must choose any remaining intersection point and remove from the grid all sticks which pass through this point. A player will lose the game if he/she cannot make a move (i.e. there are no intersection points remaining on the grid at his/her move). Assume that both players play optimally. Who will win the game? Input Format The first line of input contains two space-separated integers, n and m (1 ≤ n, m ≤ 100). Output Format Print a single line containing "Akshat" or "Malvika" (without the quotes), depending on the winner of the game.
Response The problem can be solved using a simple observation. The game is essentially about removing rows and columns from a grid. Each move removes one row and one column. Therefore, the game ends when there are no rows or columns left, i.e., when the minimum of n and m reaches zero. Since Akshat starts the game, if the minimum of n and m is even, Malvika will make the last move and win the game. Otherwise, Akshat will make the last move and win the game. The time complexity of this solution is O(1) because it involves only a constant amount of computation. The space complexity is also O(1) because it uses only a constant amount of space. The code first reads the values of n and m. It then checks if the minimum of n and m is even. If it is, it prints "Malvika". Otherwise, it prints "Akshat". Here is the C++ code for the problem:

heading: Future Work
text: We will endeavor to achieve this goal in our future work