LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Authors: Naman Jain
King Han
Alex Gu
Wen-Ding Li
Fanjia Yan
Tianjun Zhang
Sida Wang
Armando Solar-Lezama
Koushik Sen
Ion Stoica
Venue: ICLR 2025 Poster (2025)
Keywords: Code LLMs; Evaluation; Contaminationl; Overfitting
Primary area: datasets and benchmarks
TL;DR: We build evaluation of code LLMs on new problems highlighting challenges like contamination and overfitting

Abstract: Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEvla, MBPP) are no longer sufficient for assessing their capabilities suffering from data contamination, overfitting, saturation, and focus on merely code generation. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which collects new problems over time from contests across three competition platforms, Leetcode, Atcoder, and Codeforces. Notably, our benchmark also focuses on a broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts over six hundred coding problems that were published between May 2023 and Aug 2024. We evaluate over 50 LLMs on LiveCodeBench (LCB for brevity) presenting the largest evaluation study of code LLMs on competition problems. Based on the study, we present novel empirical findings on contamination, overfitting, and holistic evaluations. We demonstrate that time-segmented evaluations serve as a robust approach to evade contamination; they are successful at detecting contamination across a wide range of open and closed models including GPT-4O, Claude, Deepseek, and Codestral. Next, we highlight overfitting and saturation of traditional coding benchmarks like HumanEvla and demonstrate LCB allows more reliable evaluations. Finally, our holistic evaluation scenarios allow for measuring the different capabilities of programming agents in isolation.

Published as a conference paper at ICLR 2025
LIVECODEBENCH: HOLISTIC AND CONTAMINATION FREE EVALUATION OF LARGE LANGUAGE MODELS FOR CODE
Naman Jain'
King Han
Alex Gu
Wen-Ding Li
Fanjia Yan
Tianjun Zhang"
Sida I. Wang
Armando Solar-Lezzama
Koushik Sen
Ton Stoica'
UC Berkeley
MIT
Cornell
[naman_ jain, kingh073C, fanjiayan, tianjunz, ksen, istoical @berkeley. edu (gua, (asolarre @csa .mit.eddi w1 w16780cornell ledu s sidawang88 88 gmaa com
ABSTRACT
Large Language Models (LLMs) applied code-related applications have emerged as prominent field, attracting significant interest from academia and industry. However, and improved L.MS developed, existing evaluation bench- marks (c.g., HUMANEVAL, MBPP) are no longer sufficient for assessing their capabilities suffering from data contamination, overfitting, saturation, and focus merely code generation. In this work, we propose LIVECODEBENCI,, a compre- hensive and contamination- -rree evaluation of LLMs for code, which collects new problems over time from contests across three competition platforms, LEETCODE, ATCODER, and CODEFORCES. Notably, our benchmark also focuses on broader range of code-related capabilities, such self-repair, code execution, and test output prediction, beyond just code generation. Currently, LIVECODEBENCH hosts over six hundred coding problems that were published between May 2023 and Aug 2024. We evaluate over 50 LLMS on LiVECODE (LCB for brevity) presenting the largest evaluation study of code LLMs competition problems. Based on the study, we present novel empirical findings on contamination, over- fitting, and holistic evaluations. We demonstrate that time-segmented evaluations serve as robust approach to evade contamination; they are successful at detecting contamination across wide range open and closed models including GPT-4-O, CLAUDE, DEEPSREK, and CODESTRAL. Next, we highlight overfitting and satu- ration of traditional coding benchmarks like HUMANEVAL. and demonstrate allows more reliable evaluations. Finally, our holistic evaluation scenarios allow for measuring the different capabilities of programming agents in isolation.
INTRODUCTION
Code has emerged as an important application area for LLMs, with a proliferation of code-specific
models (Chen al. 2022 Austin et al. 2021 Nijkamp et al. 2022 Li et al. 2023b Roziere et al.) 2023 Guo et al. 2024 Ridnik et 20.4 Lozhkov et al. 2024) and their applications across various
tasks such as program repair (Zheng et al. 2024; Olausson et al. 2023), optimization (Madaan et al.
2023a). test generation (Steenhock al. 2023). documentation (Luo et al. 2024), SQL (Sun et al.
2023). In contrast with these rapid advancements, evaluations have remained relatively stagnant, and current benchmarks like HUMANEvAL and MBPP paint misleading picture. Firstly, while coding a multi-faceted skill, these benchmarks only focus on code generation, thus overlooking broader code-related capabilities. Moreover, these benchmarks suffer from contamination overfitting, as benchmark samples are present in the training datasets.
Note that for model comparisons performances are averaged across multiple months and platforms achieving larger sample size.

---

Published as a conference paper at ICLR 2025
Code Generation Live Evaluation
Code Generation
100
US In 330
3eakk55
arrec
GPT4
Sa
Claude-3.55 55 Cutoff Date
PASSØ
60-
GP Cutoff Date
Self
Execution
Repair
DS-Ins Release Date
Test Output Prediction
May
Jul
Sep
Nov
Mar
May
Jul
-Ceadde3-O - - GPT4-Turbo
-GPT3.5-Turbo - DDS- -38-338- Gemini-1.5-Flea
LEETCODE Problem Release Monch
--mmini-1.--Pr LLama3-700 DD-6.77
Figure 1: Left. LIVECODEBENCH comprises problems marked with release dates, allowing evaluations over different time windows. The figure depicts performances of DEEPSEEK, GPT-4-O, and CLAUDE- 3s models over bimonthly time windows (~ 40 LEETCODE problems) showcasing stark drop after their cutoff dates, highlighting contamination. Thus, we can detect and avoid contamination by evaluating models only on time-windows after the model's cutoff date (green region | Right. We evaluate LLMs across four scenarios that capture key coding capabilities necessary for building programming agents: code generation, repair, testing, and comprehension. The figure depicts various model performances across the scenarios available in LIVECODEBENCH in radial plot highlighting relative differences changing across the scenarios.
Motivated by these shortcomings, introduce LIVECODEBENCH, built the following principles:
1. Live updates to prevent contamination. LLMS are trained on massive inscrutable corpora and current benchmarks suffer from the risk of data contamination as they could be included in those training datasets. While previous works have attempted decontamination using both exact and fuzzy matches (Li et al. 2023b]d). can be a non-trivial task (Team) 2024 Yang et al. 2023). Existing competition programming benchmarks (like APPS and CODESCOPE) are already contaminated and may fail to provide reliable evaluation of code L.M capabilities. In LivECodEBENCH,
prevent the risk of problem contamination, evaluate models on new problems tagged with a release date. Next, for newer models, we only consider problems released after the model's 's cutoff date ensure that the model has not been trained on as demonstrated in Figure llett
2. Holistic Evaluation. Current code evaluations primarily focus natural language to code generation. However, programming a multi-faceted task that requires capabilities beyond those measured by code generation. These broader capabilities are even more relevant for constructing programming agents that can interact with the execution environment. Therefore, we evaluate the execution-feede www.effeeebbb-eeeeeeeeee scenario (Olausson et al. 2023), assess code comprehension capabilities using the code execution scenario (Gu et al. 2024), and introduce the test output prediction scenario to evaluate the models' test generation capabilities.
multi-turn coding using the self-repair
3. High-quality problems and tests. High-quality problems and tests are crucial for reliable evaluation of LLMs. However, existing benchmarks suffer from multiple deficiencies. First, existing competitive programming benchmarks (APPS, CODE- CODE-CONTESTS, XCODEEvAL, CODESCOPE) contain problems not amenable to input-output-baaat based auto-grading. For example, CODE-COre CONTESTS (Li et al. (022) (page 39, second-to-las paragraph) reports that about twenty-five percent the problems the benchmark accept multiple correct outputs for a single input. This incorrectly penalizes correct solutions, adding noise to considerable fraction the benchmark. Next, prior benchmarks like HUMANEVAL and APPS contain insufficient tests, further exacerbating noise in evaluations (Liu et al. (2023a) reports 8% drop in model performance). In LIVECODEBENCH, we source high-quality problems from reputable platforms and implement heuristics to detect and remove problems not amenable to uuput-output-baatt based auto grading. Finally, for every problem, we provide a substantial number of tests (over 18 on average) for reliable and efficient evaluations. In contrast
2

---

Published as a conference paper at ICLR 2025
Table 1: Comparing LCB with existing coding and competition programming benchmarks.
Contamination Prevention
Problem Curation
Varied Difficulties
Braader Scenarios
Comp. Analysis
Eval.
Saturated
across Models
BUMANTSAL
2021 (Chen et ol
x
X
BUMANESAL-
22.16
MRPP Amutin
/
APPS Hendrycks ctal. 2021
x
/
CODE CONTESE5 IeE
X
X
/
x
XCoDEEVAL Khan
/
CoDeScors
Tan
x
TACO
[La a M755
X
x
USCADBESSTT Sia
2124
LiveCons-eecci (Duns)
prior works, we also include several large tests designed for stress-testing solutions ensuring weak or worse complexity solutions do not pass test harnesses.
4. Difficulty Guided Problem Curation. Competitive programming is a challenging domain even for strong LLMs. As result, these problems can be unsuitable for meaningfully com- paring today's open LLMs, because the variance in performance is low, often relying on less than 1% performance differences (within the margin of error). Therefore, we use problem difficulty ratings (sourced from the competition websites) to curate our problems, avoiding those that are too difficult for current models. particular, we avoid collecting CODE- FORCES problems used by prior works (CODE-CONTESTSS CODESCOPE, XCODBEVAL.) since we find they do not sufficiently distinguish models due to model performances (PASS I) tending to zero. Indeed, LCB easy and medium problems can separate 7B models that are indistinguishable on the hard subset (see DS-7B vs SC2-7B).
With these principles in mind, build LEVE,ODE continuously updated benchmark that avoids data contamination. Particularly, we collect 612 problems from contests occurring between May 2023 and Aug 2024 and use them construct the different scenarios.
Empirical Findings. We have evaluated over 50 (18 base models and 34 instruction-tuned) models across different LIVECODEBENCH scenarios. Based on this study and following analysis, we present novel empirical findings which have not been revealed in prior benchmarks.
Contamination. We observe stark drop in the performance of DEEPSREK, GPT-4-O, CODESTRAL and CLAUDE-3S on LEETCODE problems released after Aug 2023, Oct 2023, Jan 2024, and April 2024 (Figure Icfft). These results highlight likely contamination older problems and time-segmented evaluations prove effective for performing fair comparisons across varied set models from both open and closed domains.
2. Holistic Evaluation. Our evaluations reveal that model performances are correlated across tasks, but the relative differences do vary. For example, in Figure the gap between open and closed models further increases on tasks like self-repair or test output prediction. Similarly, CLAUDE-3 3- OPUS surpasses GPT-4 on the test output prediction scenario.
3. HUMANEVAL Overfitting, Upon comparing LIVECODEBENCH with HUMANEVAL, we find that models cluster into two groups, ones that perform well on both benchmarks, and others that perform well on HuManEval. but not on BIVeCODEEENCH (see Figure 4 The latter group primarily comprises fine-tuned open-access models while the former comprises base and closed models. This indicates some level of overfitting to HUMANEVAL.
4. Model Comparisons (Figure 3). We provide model comparisons across different groups like base, instruct, open, and closed models and across groups like open vs closed models.
Concurrent Work. (Huang et al. 2023) also evaluate LLMs in time-segmented manner. However, they only focus on CODEFORCES problems, while we combine problems across platforms and additionally propose a holistic evaluation across multiple code-related scenarios. [Liu et al. (2024) evaluate the code comprehension capabilities LLMS using exccution. (Singhal et al.) (2024) also evaluates LLMS more tasks but focus et al. 2024) evaluate LLMS on USACO problems while we focus on simple competition scenarios.
ffunctional-correcttee aspects of programming. Shi
AAVECODEBENCH considers different evaluation scenarios compared to XCODEEVAL, and CODESCOPE: focusing settings where reliable evaluation is possible.
3

---

Published as a conference paper at ICLR 2025
Problem Statement
User Solution
You arey nums'. frequencies of elements "nums" such that those elements all have the frequency.
def counctnunsl: free - Counter (mues) cnts freq. calues) nax_free caxicnts)
Input
coacoootaxxx_eeeeii WXx, fax_
nuns [1,3,3,4,4]
Code Generation
Self Repair
Output Prediction
Code Execution
def count (nuws):
count (nums]: fr
Step 1. masimam frequencias Step 2. max frequency is
counti 12,4,,,4,44 41ow777 Ans
Traq.calues((
count lend
count Londl (feen)) 18 mas.
Step 3. 22 4
free, Stenet]
X
Stop 4, And 4s
if v wa2
Figure 2: Overview of the four scenarios present in LIVECODEBENCH.
2 HOLISTIC EVALUATION
Code capabilities of L.LMs evaluated and compared using natural language code generation tasks, However, this only captures one dimension of code-related capabilities. AlphaCodium (Ridnik al. 2024) developed an intricate LLM pipeline for solving competition coding problems. By combining natural language reasoning, test case generation, code generation, and self-repair, they achieve significant improvements over a naive direct code generation baseline, showcasing the importance of these broader capabilities. Motivated by this, we propose a holistic evaluation of LLMS using a suite of evaluation setups that capture a broader range of code-related capabilities.
Specifically, we evaluate code I.I.Ss in four scenarios, namely code generation, self-repair, code execution, and test output prediction. Our selection criterion was to pick settings that are useful components in code LLM workflows and in addition, have clear and automated evaluation metrics.
Following we describe each these scenarios in detail.
Code Generation. The code generation scenario follows the standard setup for generating code from natural language. The model is given a problem statement, which includes natural language description and example tests (input-output pairs), and is tasked with generating correct solution. The evaluation is performed based on functional correctness, using a set of unseen test cases. We use the PASS metric measured the fraction of the problems for which the model was able to generate a program passing all tests. Figure 2 (left) provides an example of this scenario.
Self Repair. The self-repair scenario is based on previous works that tested the self self-repair capabilities of LLMS OOlausson et al., 2023 Shinn et al.] 2023 Chen et al., 2023). Here, the model is given problem statement from which it generates candidate program (similar to the single-step code generation scenario above). However, in case of a mistake, the model is additionally provided with error feedback (either the exception message or a failing test case) and is tasked with generating the fixed solution. Similar to the code generation scenario, the evaluation is performed via functional correctness on the final program, i.e. either the single-step correct generation or the attempted repair. We use the PASS @ I metric to measure the combined performance after the repair step. Figure (mid-left) provides an example this scenario.
Code Execution. The code execution scenario is based on the output prediction setup used in CRUXEVAL (Gu al., 2024). The model provided a program snippet consisting of function (f) along with test input to the program and is tasked with predicting the output of the program on the input test case. The evaluation is performed via an execution based on an exact match correctness metric. Figure 2 (right) provides example of the code execution scenario.
Finally, we introduce a new task that is designed to study natural language reasoning and test generation. In this task, the model is given the problem statement along with a test case input, and it is tasked with generating the expected output for that input. This task follows setup similar to the one used in CODET (Chen al. 2022), where tests are generated solely
Test Case Output Prediction.
4

---

Published as a conference paper at ICLR 2025
from problem statements, without the need for the function's implementation. A key difference is that we provide a fixed set of test inputs for each problem in our dataset, and the models are then prompted to only predict the expected output for those specific inputs. This approach allows for a straightforward evaluation of the test generation capabilities by avoiding test input prediction, a hard-to-evaluate task. Figure (mid-right) provides example of this scenario.
BENCHMARK CURATION
We curate our problems from three coding competition websites: LEETCODE, ATCODER, and CODE- FORCES. These websites periodically host contests containing problems that assess the coding and problem-solving skills of participants. The problems consist of a natural language problem statement along with example input-output examples, and the goal is to write program that passes set of hidden tests. Further, thousands of participants participate, solving these problems thus ensuring that the problems are vetted for clarity.
3.1 DATA COLLECTION
We have written automated HTML scrapers for each of the above websites collect problems and the corresponding metadata. To ensure quality and consistency, we parse mathematical formulas and exclude problems with images. We also exclude problems that are not suitable for grading by input-output examples, such as those that accept multiple correct answers or require the construction of data structures. Specifically, we use keyword- basee heuristic filters to filter interactive problems and problems accepting multiple correct solutions. Besides parsing the problem descriptions, we also collect associated ground truth solutions and test cases whenever directly available. Thus for collect tuples of natural language problem statement P, test cases T, and ground truth solution S. Finally, we associate the contest date D to mark the release date of each problem and use the collected attributes to construct problems for our four scenarios (detailed in Section C.2 ahead). Note that this process is completely automated and human involvement only involved in modifying high-level design decisions such updating problem difficulty settings and improving the keyword-based heuristics for filtering problems over different "live updates".
each problem,
Scrolling through time. As noted, we associate the contest date D for each problem. The release date allows us the measure the performance of LLMS over different time windows by filtering problems based on whether the problem release date falls within a time window (referred to as "scrolling" through time). This is crucial for evaluating and comparing models trained at different times. Specifically, for a new model and the corresponding cutoff date (normalized to the release date if the training cutoff date is not published), we can measure the performance of the model on benchmark problems released after the cutoff date. We have developed a UI that allows comparing models on problems released during different time windows (Figure 10).
Test collection. Tests are crucial for assessing the correctness of the generated outputs and are used in all four scenarios. We collect tests available on platform websites whenever possible and use them for the benchmark. Otherwise, following (Liu et al.) 2023b), we use a LLM (here GPT-4-TURBO) to generate test inputs for the problems. A key difference between that instead of generating inputs directly using the I.M,, we construct generators that sample inputs based on the problem specifications using in-context learning. Details of this approach can be found in Section A.2 Finally, we collect a small fraction of failing tests from the platforms for robust adversarial tests. Section A.4 describes our design decisions to determine the number of tests.
test generation approach is
Problem difficulty. Competition programming has remained a challenge for LLMs, with GPT-4 achieving an average CODEFORCES rating (ELO) of 392, placing it in the bottom percentile (OpenAl 2023). This makes it difficult to compare LLMS, as the variation in performance across models is low. In LECECDDE we collect problems of diverse difficulties as labeled in competition platforms, excluding problems that are rated above a certain threshold that are likely too difficult for even the best model Further, we use these ratings to classify problems as EASY, MEDIUM, and HARD for more granular model comparisons, estimating difficulty roughly based on HUMANEVAL scores.
We defer the platform and scenario specific curation details to the Appendix (Section
From our early explorations, we find CODEFORCES problems being considerably more difficult than ArCODer and LEeeCoone problems and thus focus primarily on the latter platforms.
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENT SETUP
We describe the experimental setup in this section. First, we provide the common setup across the scenarios, followed by the scenario-specific setups in Section
Models. We evaluate 52 models across various sizes, ranging from 1.3B to 70B, including base models, instruction models, and both open and closed models. Our experiments include models from different classes, such as GPTs, CLAUDES, GEMINIS, MISTRAL, LLAMA-3s, DEEPSEEKS, CODELLA- MAS, STARCODER2, CODEQWEN and their variants. Appendix D.1 provides the list of models.
Evaluation Metrics. We use the PASS (Kulal et al., 2019; Chen et al., (021) metric for our evaluations. Specifically, we generate 10 candidate answers for each problem from model providers or using VLLM (Kwon et al. 2023). We use nucleus sampling with temperature 0.2 and top_p 0.95 and calculate the fraction of correct solutions.
4.1 SCENARIO-SPECINC SETUP
The setup for each scenario is presented below. Note that the base models are only used in the code generation scenario since they do not easily follow the format for the other scenarios.
Code Generation. For the instruction-tunel models, use zero-shot prompt and follow the approach of Hendrycks et al. (202T) by adding appropriate instructions to generate solutions in either functional stdin format shot for base-models). Section D.2.eepicts the prompt.
Self Repair. Similar to prior work Olausson et al. (2023), we use the programs generated during the code generation scenario along with the corresponding error feedback to build the zero-shot prompt for the self-repair scenario. The error feedback can be syntax errors, runtime errors, wrong answers, and time-limee limit errors. Section D.33 provides error feedback and the corresponding prompt.
Code Execution. We use few-shot prompts for the code execution scenario, both with and without chain-of-thought prompting (COT). Particularly, we 2-shot prompt without COT and 1-shot prompt with COT with manually detailed steps. The prompts are detailed in Section DD4
Test Output Prediction. We use zero-shot prompt that queries the model complete assertions, given the problem, function signature, and test input. We provide the prompt in Section D.5
5
RESULTS
We first describe how LiVECODEBENCH ENCH helps detect and avoid benchmark contamination in Sec- tion 5.1 Next, present the findings from our evaluations LIVECODEBENCH in Section 5.2
5.1 AVOIDING CONTAMINATION
Contamination in DEEPSEEK and GPT-4-0. LIVECODEBENCH curates problems released since May 2023. However, DEEPSEEK was released Sep 2023 and GPT-4-0's official cutoff date is Nov 2023. We can measure the performance of these models on the benchmark problems released after these dates, thereby estimating the performance of the model on previously unseen problems. Figure [] shows the performance of these models on LEETCODE problems released in different months from May 2023 and Aug 2024. First, we notice stark drops model performance DS-INS-33B model after Aug. 2023, GPT-4-O model after Nov. 2023, CLAUDE-3S model after Apr. 2023. These drops align with their release or cutoff dates which suggests that the earlier problems might indeed be contaminated. This trend is consistent across other LIVECODEBENCH scenarios like repair and code execution, as depicted in Figure II Concurrently, (Guo et al. 2024) (Section 4.1, last paragraph) also acknowledge the possibility of LEETCODE contamination, noting that "models achieved higher scores in the LeetCode Contest held in July and August".
Interestingly, find that this drop in performance primarily occurs for the LEETCODE problems only and that the model performance is relatively smooth across the months for problems from other platforms. Figure 12 shows relatively stable performance for all models ArCODeR problems.
Performances of other models. We study performance variations in other models released more recently. Particularly, GPT-4-TURBO, MISTRAI.-L, and CLaude-3s models were released in Nov"2023,

---

Published as a conference paper at ICLR 2025
Code Generation
Self-Repair
30
1500
22.6
Difficulty
648
Difficuity Level
67.4
medium
medium
P
0U-
PASS
40
12:0
13
20
23.
21.
20-
8.2
GPT4 Terbo
GPI
Ohedkekk
Comini Pioc
Gomiat 1,5
Chunles
Lamad les 786
DS Ins-116
arT3. Tarba
Phind 34B
DS-tae 7B
GPT4 Torbo
Gemisi Po
Claskcd
GPT
Gemiis Flash
Chantes
LLandd lms.
PHiia- Mill--M
DS 100
GPT35 Turto
DS 7B
Model Name
Model Name
Test Output Prediction
Code Execution
Prompt
Pass01-COT
64.8
PAssi11
469
PAsSi1
312
S4.6 31.
20-
Gludee 0
GPT4 Torko GPY
Misrak
Gambe 5
LLamukiin fex 7088
Gemmis Flash 5
Cleatis- 5
GP ra. Tarbo
1msmm.u J3B
Phiee 348
DS fas66
Torbo Clask
Ganbes Pio.
GP
Gemint Flash 5.5
LLamst lra
Metrrl
Claske 5
GPTAS
PS 2si-ns-30 PPTtS-TTubo Turbo
DS ts-
Model Name
Model Name
Figure 3: Model performances across the four scenarios available LIVECODEBENCH.
Feb" 2024, and 2022 respectively. Empirically, we do not observe significant performance variations across the months, as shown in Figure 13 Interestingly, model also suffers from contamination dropping from PAss@ 60 in May problems to PASS ~ 0 in September LEETCODE problems. Finally, CODESTRAL achieves PASS @ 36.5 on problems released between May' 23 and Jan' 24 and PASS@ @ 28.3 on problems post Jan'24. 24
find that even the DS-BASE-33B
5.2 PERFORMANCE AND MODEL COMPARISONS
We provide our model comparison findings here (and in Appendix E
To overcome contamination issues in DEEPSEEK models, only consider problems released since Sep 2023 for all evaluations below. Figure shows the performance of a subset of models across the four scenarios.
Holistic Evaluations. We have evaluated the models across the four scenarios currently available in LIVECODEBENCH, Figure displlays the performance of models on all scenarios along the axes of the polar chart. First, we observe that the relative order of models remains mostly consistent across the scenarios. This is also supported by high correlations between PASS @ I metric across the scenarios over 0.88 across all pairs shown in Figure [4] However, despite the strong correlation, the relative differences in performance do vary across the scenarios. For example, GPT-4-TURBO further gains performance gap over GPT-4 in the self-repair scenario after already leading the code generation scenario. Similarly, CLAUDE-3-OPUS and MISTRAL-L perform well in tasks involving COT, particularly in the code execution and test output prediction scenarios. For instance, MLAUDE-3-OPIL even outperforms GPT-4-TURBO in the test output prediction scenario. These differences highlight the need for holistic evaluations beyond measuring code generation capabilities.
Comparison to HUMANEVAL Next, compare how code generation performance metrics translate between LIVECODEBENCH and HUMANEVAL, the primary benchmark used for evaluating coding capabilities. Note that seatter plot of PASS @ on HUMANEVAL+ versus LCB-Easy code generation scenario. We find only a moderate correlation of 0.72, with much larger performance variations LCB-Easy,
HUMANEVAL+ providing more accurate evaluations. Figure shows
Additionally, we observe that the models cluster into two groups, shaded in red and green. The models in the green-shaded region lie close to the y line, indicating that they perform similarly both
7

---

Published as a conference paper at ICLR 2025
Lasy vs HUMANEVAL+
Tarb
PASS on HUMANEVAL+
GFT4
80-
Claide
05
Pic13
Clodl
60
05-138
US 3tt
40-
US dose 18
SCP 130-0aa--351
20-
20
40
60
80
PASS@1 1 on LCB-Easy
GPT-4-Turbo
Gemini-Flan Flasi---5
MistraLL
Command--R
B5-Bas--338
DS-Irs-336
GPT-3 PTT-3.5TTuile
Clarde-3-O
Mis-8e228-Ins
LLama3-70l-trs
DS-Baw-66.78
126-ls-6. 7a
GPT-4
Claudo-3-5
Phird 348
LLama3-Bb-le
DS Base 1.38
DS-1es-1.38
Pro-11
Claude-Ins-1
MC-8.BB
LLana3-Bb-Base
AC2-Base-158
CodeQhennChat
Figure 4: Scatter plot comparing PASS @ of models on HUMANEVAL+ versus LCB-Eas (time-window Sep 2023 to May 2024). We find that the models are separated into two groups - the green-shaded region where performances on the two datasets are aligned and the red-shaded region where models perform well on HUMANEVAL+ but perform poorly on LIVECODEBENCH. This indicates potential overfitting HUMANEVAL+ and primarily occurs in the fine-tuned variants of open-access models. For example, DS-INS- 1.3B achieves PASS 1 of 60 and 26 on HUMANEVAL+ and LCB-Easy subset. Thus, while it ranks above CMD-R+ (~ 100x larger!) HUMANEVAL+, it performs significantly worse on the LCB. Similarly, DS-INS-6.7 7B and CODEQWEN outperform SONNET on HUMANEVAL+ but are 20 points behind on LCB-Easy.
benchmarks. On the other hand, models in the red shaded region perform well on HUMANEVAL+ but not as well on LIVECODEBENCH. Interestingly, the green-shaded cluster contains base models or closed-access models, while the red-shaded cluster primarily comprises fine-tuned variants of open-access models. The well-separated clusters suggest that many models that perform well on HUMANEVAL might be overfitting on the benchmark, and their performances do not translate well to problems from other domains or difficulty levels like those present in LIVECoDEBENCH. Indeed, HUMANEvAL. has small and isolated programming problems and thus easier to overfit. contrast, sourced from reputable coding platforms offering more challenging
LiVECODEBENCH problems problems with higher diversity and difficulty levels. We detail instances of this overfitting (DS- (DS-INS- 1.3B, DS-INN-6. 6. 6.7 and CODEQWEN) in Figure caption.
Comparing Base Models. We use four families of base models and compare them on the code generation scenario. A one-shot prompt used for all models avoid any formatting and answer extraction issues. We find L3-BASE and DS-B models are significantly better than both CODELLAMA and STARCODER2 base models with DS-BASE-6.77 model even outperforming both CL--BASE-3BB and 1-----SE-150 models. Note that some LCB specific differences can potentially be attributed to data curation approaches. For instance, SC models (and potentially DS as discussed in Section 5.1 use competition problems during pre-training.
Role of Post Training. Post-training improves performance on both HUMANEVAL+ and LIVE- CODEBENCH for the code generation scenario. Particularly, on LCB L3-INS-70B, DS- and PHIND-34D improve PASS@1 1 over their base models by 8.2, 7.3 and 9.5 points respectively. This highlights the importance of good post-training datasets for building strong LLMs.. At the same time, we note that the base models have aligned performances on LCB code generation and HUMANEvAL.+ benchmarks and lic within or close to the green shaded region in Figure However, the fine-tuned open models exhibit a larger performance gap, with much better performances on HUMANEVAL+, On the other hand, the closed-access models still aligned across both benchmarks. This suggests the necessity of more diverse post-training data mixtures.
Open-Access Closed-Access Models. In general, closed (API) access model families generally outperform the open access models. The gap is only closed by three models, namely L3-INs-70B, MIXTRAL, and DS-INS-33R which reach the performance levels of the closed models. For instance, in
8

---

Published as a conference paper at ICLR 2025
LIVECODEBENCH
GurboTuro
Turbo
GPT4
Gemini-Pro.1
Clucee30
Gemini-
30,0
LLama3 Ins. 70B
26.3
Claude3-S
26.9
Mix Bx228 ins
Mistral-1
Claude2
Claude-Ins-1
23.6
API Access
GP.3.5
22.0
Phind-348 348
21.0
LLama3: Base 70B
20
125 In--nns-6... 78
Open Access
LLama3-Ing In---
DS-Bass-388
15.0
SC2
14.2
CodeGemma- Bb-Base Base
13.0
DS-Base-6i 78
12.9
Mac 0x7B ins
LLama'- Sase-BB
DS-Ins-1.33
10
15
20
25
30
35
40
PASS
Figure 5: Comparison of open access and (closed) API access models on LIVECODEBENCH code generation scenario. We find that closed-access models consistently outperform the open models with only strong instruction-tuned variants > 30B models (specifically L3-INS-70B, MIXTRAL and DS-INS-33B models) crossing the performance gap.
the code generation scenario (Figure 5). these models reach close to or even outperform closed access models like GEMINI-PRO, GPT-3.5-TURBO, and CLAUDE-3-SONNET The performances vary across scenarios with the closed-access models faring better in test output and code execution scenarios.
Highlighting gap between SoTA One distinet observation from evaluations is the large gap between SoTA models and open models across all scenarios. Particularly, GPT-4-TURBO, GPT-4, GEMINI-PRO 1.5 and CLAUDE-3-OPE lead across the benchmarks with wide performance margins over other models. This distinguishes LIVECODEBENCH from prior benchmarks (like HUMANEVAL) where various open models have achieved similar better performance. For example, DS-INS-33 INS-33B is merely 4.3 point behind GPT-4-TURBO on HUMANEVAL+ but 16.2 points (69%) on LCB code generation scenario. This gap either holds sometimes even amplifies across other scenarios.
Comparing open-access instruction-tume tuned models. Here, we compare various fine-tuned variants of the L3-BASE, DEEPSEEK and CODELLAMA base models across different model sizes. We find that fine-tuned L3-BASE and DEEPSEEK models lead in performance, followed by PHIND-34B and CODELLAMA models across most scenarios. Broadly, we find that model performances correlate with model sizes. For example, PHIND-34B model outperforms the 6.7B models across all scenarios.
RELATED WORK
Language Models for Code Generation. Starting with Codex (Chen et al. 2021 there are
dozen code LLMs. These include CodeTS (Wang al.) 2021 2023), CodeGen (Nijkamp) et al., (2022), SantaCoder (Allal et al.) 2023). StarCoder (Li et al. 2023b Lozhkov et al. 2024).
InCoder (riedd et al. (022). CodeGeeX (Zheng al. 2023)
L3 BASE,,, DEEPSEEK (Bi 2024
and CODELLAMA (Roziere et al.) 2023).
Code Generation Benchmarks. Many benchmarks have been proposed to compare and evaluate these models. These primarily focus natural language to Python code generation: HUMANEVAL
(Chnn et al.
(021), HUMANEVALL+ (Liu et al. (023b). APPS (Hendrycks et al. (02T). CODE-CONTESTS
(Li et al. 2022 MBPP (Austin et 2021 Eval (Ni al. (023). hheir variants have been
proposed to cover more languages, (Wang et al., 2022a Zheng et al. 2023 Cassano et al. 2022
Athiwaratkun al. (022). ). Many benchmarks have focused on code generation in APIs. Benchmarks
like DS-1000 (Lai et al. 2023), ARCADE (Yin et al.) (022), NumpyEval (Zhang et al. 2023b), and
PandasEval (Jain et al. 2022) focus data science APIs Other benchmarks measure using broader
APIs or general software engineering tasks, such as JulCe (Agashe et al.) (012)01)), APIBench (Patil)
al. 2023). RepoBench (Liu et al. 2023c), ODEX (Wang al. 2022b), SWE-Bench (Jimenez et al.

---

Published as a conference paper at ICLR 2025
2023
GoogleCodeRepo (Shrivastava et al. 2023 2022).
RepoEval (Zhang et al. 2023a). ClasEval (u
et al.
)023) and Cocomic- Data (Ding et al.
A few benchmarks specifically measure competitive programming. such APPS (Hendrycks et al. 202Ib, CodeContests (Li et al., (022). CodeScope (aan et al. 2023), XCodeEval (Khan et al. [023]
and LeetCode-Hard (Shinn et al. 2003), and TACO (Li et al. 2023ch. Table Table[]]ighlights differences
between them. Methods such as AlphaCode (Li et al. 2022), AlphaCode (Gimmii Team et al.)
2023). ALGO (Zhang al. 2023d), Parsel (Zelikman al. 20022), code cleaning (Jain et al. 20223] code explainations (Li et al. 2023aj, analogical reasoning Yasunaga et al.) 2023), and AlphaCodium
(Ridnik et al. 2024) have focused on improving LLMS.
6.1 HOLISTIC TASKS AND CONTAMINATION
Code Repair. (Chen et al. 2023 Olausson et al. 2023 Madaan et al. 2023b; Peng et al. 2023 Zhang al., 2023c) have investigated self-repair for existing code LLM benchmarks. These methods use error feedback for models to improve inspiring our code repair scenario.
Code Execution. Code execution was first studied in (Austin et al., [021] Nye et al. 2021) LIVE- COdEBENCH's execution scenario is particularly inspired by CRUXEval (Gu et al. 2024), recent benchmark measuring the reasoning and execution abilities of code L.MMs. We differ from CRUXEval in that our benchmark is live, and our functions are complex and numan-producee
Test Generation. Test generation using LLMS has been explored in Yuaan et al. 2023 Schäfer et al. 2024; Tufano et al. 2022; Watson et al. 2020). Here, we decouple the test inputs and outputs for fair evaluations. Finally, some works have additionally studied other tasks and scenarios like type
prediction (Mir et al. 2022 Wei et al. [023 Malik et al. 2019)) code summarization (LeClair et al.)
2019 [yerr et al. [2016] Barone & Sennrich [2017] Hasan et al. [202]] Alon et al.) 2018 code security
(Liuuori et al.)
2022 Pearce al. 2022 Tony al. (023)) etc.
Contamination. Data contamination and test-case leakage have received considerable attention
Oren et al. (2024); Golchin & Surdeanu (2023); Weller et al. (2023): Roberts al. (2024) recently.
Sainz et al. (0223) demonstrated contamination by simply prompting the model to highlight its
contamination. Some detection methods have also been built to avoid these cases (Shi al. 2023
Zhou et al., 2023). For code, (Riddell et al. 2024) use edit distance and AST AST--imilarity to detect
contamination. (Ma et al.) 2021 al. 2024) have explore "live" benchmark other domains.
7 LIMITATIONS AND CONCLUSION
We describe key limitations here and provide deeper discussion in Appendix G
Evaluation Noise. We anticipate noise from benchmark size, prompts, and sampling. For benchmark size, while 612 problems fairly larger than existing code benchmarks (c.g. 164 problems in HUMANEVAL), this set reduces on "scrolling" over the more recent problems. Next, prompting can cause large variations in model performance and we do not tune prompts across models. For sampling, we use bootstrapped PASS@ @ 1 using 10 completions, which should limit this noise considerably.
Problem Domain. LIVECODEBENCH comprises competition programming problems which might not correlate with how L.MSs are used in practice. Even then, our results are well correlated with findings from human evaluations like Chatbot-Arena Aarreaaaaaaaannggaa et al. (2024) thus providing useful signal.
Conclusion. In this work, we propose LIVECODEBENCH, new benchmark for evaluating LLMs for code. LIVECODEBENCH provides an extensible framework that will keep on updating with new problems. Our benchmark mitigates contamination issues in existing benchmarks by introducing live evaluations and emphasizing scenarios beyond code generation to account for the broader coding abilities of LLMs. Our evaluations reveal novel findings such contamination detection, holistic evaluations, and potential overfitting on HUMANEVAL. We hope LIVECODEBENCE BEnHh with serve to advance understanding of current code I.IMS and also guide future research through our findings.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work was supported in part by NSF grants CCF:1900968, CCF:1908870 and by SKY Lab industrial sponsors and affiliates Astronomer, Google, IBM, Intel, Lacework, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Nexla, Samsung SDS, Uber, and VMware. A. Gu is supported by the NSF Graduate Research Fellowship under Grant No. 2141064. A. Solar-Lezama is supported by the NSF and Intel Corporation through NSF Grant CCF:2217064, Any opinions, findings, conclusions, or recommendations in this paper are solely those of the authors and do not necessarily reflect the position of the sponsors.
Finally, we thank Manish Shetty, Wei-Lin Chiang, Jierui Li, Horace He, Federico Cassano, Pengcheng Yin, Kexun Zhang, Rishabh Singh, and Aman Madaan for helpful feedback at various stages of the work. We particularly thank Kexun Zhang for providing results for C++ and Nikhil Thakur for helping maintain the benchmark.
REFERENCES
Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. Juice: A large scale distantly supervised dataset for open domain context-basee based code generation. arXiv preprint (rXXx::910002216,, 2019.
Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff. Mayank Mishra, Alex Gu, Manan Dey, et al. Santacoder: don't reach for the stars! arXiv preprint arXiv:2301 003988, 2023.
Uri Alon, Shaked Brody, Omer Levy, and Eran Yahav. code2seq: Generating sequences from structured representations of code. arXiv preprint arXiv: 1808.0/4000 2018.
Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv: /4868, 2022.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 07732, 2021.
Antonio Valerio Miceli Barone and Rico Sennrich. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. arXiv preprint arXiv:1707.02273. 2017.
Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, al. Deepseek Ilm: Scaling open-source language models with longtermism. arXiv preprint 1.Xi::24001.295444 2024.
Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl-e: A scalable and extensible approach to benchmarking neural code generation. arXiv preprint arXiv:2208.00 08227, 2022.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10330 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating lims by human preference, 2024.

---

Published as a conference paper at ICLR 2025
Yangruibo Ding, Zijian Wang. Wasi Uddin Ahmad. Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Cocomic: Code completion by jointly modeling in-file and cross-file context. arXiv preprint arXtv:2212.10007, 2022.
Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng. Chaofeng Sha, Xin Peng, and Yiling Lou. Classeval: A manually- benchmark for evaluating Ilms on class-level code generation, 2023.
Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. preprint arXiv:2204.05990 05999, 2022.
A Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. /805, 2023.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. arXiv preprint arXiv:2308. 08493, 2023.
Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar- Gabriel Synnaeve, and Sida I Wang. Cruxeval: A benchmark for code reasoning, understanding and exccution. arXiv preprint arXiv:2401.03065 2024.
Daya Guo, Qihao Zhu, Dejian Yang. Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li. et al. Deepseck-coder: When the large language model meets programming-the rise code intelligence. arXiv preprint arXiv:2401. /4/96, 2024.
Masum Hasan, Tanveer Muttaqueen, Abdullah Al Ishtiaq, Kazi Sajeed Mchrab, Md Mahim Anjum Haque, Tahmid Hasan, Wasi Uddin Ahmad, Anindya Iqbal, and Rifat Shahriyar. Codese: A large code-description parallel dataset. arXiv preprint arXiv:2105.. /45. 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 2021.
Yiming Huang. Zhenghao Lin, Xiao Liu, Yeyun Gong, Shuai Lu, Fangyu Lei, Yaobo Liang, Yelong Shen, Chen Lin, Nan Duan, et al. Competition-level problems are effective Ilm evaluators. arXiv preprint ar 2rxiv22312.02143 2023.
Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Summarizing source code using neural attention model. In 54th Annual Meeting of the Association for Computational Linguistics 2016, pp. 2073-2083 2083. Association for Computational Linguistics, 2016.
Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. Jigsaw: Large language models meet program synthesis. In Proceedings of the 44th International Conference on Software Engineering, pp. 1219-1231, 2022.
Naman Jain, Tianjun Zhang, Wei-Lin Chiang, Joseph E Gonzalez, Koushik Sen, and Ion Stoica. Llm-assisted code cleaning for training accurate code generators. arXiv preprint 449444 2023.
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. 2023.
Mohammad Abdullah Matin Khan, M Saiful Bari, Xuan Long Do, Weishi Wang. Md Rizwan Parvez, and Shafiq Joty. xcodeeval: A large scale multilingual multitask benchmark for code understanding, generation, translation and retrieval. arXiv preprint 23iv:2333.030000 2023.
Sumith Kulal, Panupong Pasupat, Kartik Chandra, Mina Lee, Oded Padon, Alex Aiken, and Percy S Liang. Spoc: Search-based pseudocode to code. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319-18344 PMLR, 2023.
Alexander LeClair, Siyuan Jiang, and Collin McMillan. A neural model for generating natural language summaries of program subroutines. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 195-806.75 IEEE, 2019.
Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. Explaining competitive-level programming solutions using llms. arXiv preprint (Tt.:3330.05337, 2023a.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you! arXiv preprint arXiv:2305.061 2023b.
Rongao Li, Jie Fu, Be-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852
2023c.
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint S3Iv:2230000466. 2023d.
Yucheng Li, Frank Guerin, and Chenghua Lin. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 18600-18607, 2024.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with alphacode. Science, 10966624) 1092-107, 2022.
Pietro Liguori, Erfan Al-Hossami, Domenico Cotroneo, Roberto Natella, Bojan Cukic, and Samira Shaikh. Can we generate shellcodes via natural language? an empirical study. Automated Software Engineering. (9)3), 2022.
Changshu Liu, Shizhuo Dylan Zhang, and Reyhaneh Jabbarvand. Codemind: A framework to challenge large language models for code reasoning. arXiv preprint 24Xiv:2402.09664, 2024.
Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. Evaluating the logical reasoning ability of chatgpt and gpt-4. arXiv preprint arXiv:2304. 03439, 2023a.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. arXiv preprint arXiv:2305.012 02..000./21100 10, 2023b.
Tianyang Liu, Canwen Xu, and Julian McAuley. Repobench: Benchmarking repository-Ievel code auto-completion systems. arXiv preprint arXiv:2306.0300/1 00000,00..000...0 2023c.
Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu. Dmitry Abulkhanov. Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii Zheltonozhskii, Nii Osae Osae Dade, Wenhao Yu, Lucas Krauß, Naman Jain, Yixuan Su, Xuanli He, Manan Dey, Eduardo Abati, Yekun Chai, Niklas Muennighoff, Xiangru Tang, Muhtasham Oblokulov, Christopher Akiki, Mare Marone, Chenghao Mou, Mayank Mishra, Alex Gu, Binyuan Hui, Tri Dao, Armel Zebaze, Olivier Dehaene, Nicolas Patry, Canwen Xu, Julian McAuley, Torsten Scholak, Sebastien Paquet, Jennifer Robinson, Carolyn Jane Anderson, Nicolas Chapados, Mostofa Patwary, Nima Tajbakhsh, Yacine Jernite, Carlos Muñoz Ferrandis, Lingming Zhang, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. Starcoder and the stack v2: The next generation. 2024.
13

---

Published as a conference paper at ICLR 2025
Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, et al. Repoagent: An llm-powered open- framework for repository-level code documentation generation. arXiv preprint arXiv:2402.16667, 2024.
Zhiyi Ma. Kawin Ethayarajh, Tristan Thrush, Somya Jain, Ledell Wu, Robin Jia, Christopher Potts, Adina Williams, and Douwe Kiela. Dynaboard: An evaluation-asse -as-iioon-------vviccc platform for holistic next-generation benchmarking. Advances in Neural Information Processing Systems, 34: 10351-10367, 2021.
Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. Learning performance-improving code edits. arXiv preprint arXiv:230..07867, 2023a.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint 2rXIv:2303.17651 2023b.
Rabee Sohail Malik, Jibesh Patra, and Michael Pradel. N12type: inferring javascript function types from natural language information. In 20/9 IEEE/ACM 41 International Conference on Softwree Engineering (ICSE), pp. 304-315. IEEE, 2019.
Amir M Mir, Evaldas Latoškinas, Sebastian Proksch, and Georgios Gousios. Type4py: Practical deep similarity learning- based type inference for python. Proceedings of the 44th International Conference on Software Engineering, pp. 2241-2252, 2022.
Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng. Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, al. L2ceval: Evaluating sanuaage-tocoode generation capabiliities of large language models. arXiv preprint arXiv:2309. 17446, 2023.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations, 2022.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 2021.
Theo X Olausson, Jeevana Priya Inala, Chenglong Wang. Jianfeng Gao, and Armando Solar-Lezama. Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306. 09896, 2023.
OpenAL Gpt-4 technical report. arxiv 2303.08774. View in Article, 2023.
Yonatan Oren, Nicole Meister, Niladri Chatterji. Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination for black-box language models. In The Twelfth International Confer- ence Learning Representations, 2024. URL www.hhtpp////.m ptttp::///penreveew wwt.eewwwwww.eeeeeeeeeeeeeeeee http:e/wwwwe/fooouteeeeeeee formm?id KS8mIvetq2
Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E Gonzalez. Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305. 15334, 2023.
Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gann Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot's 's code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 754-768. IEEE, 2022.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improv- ing large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813. 2023.
Martin Riddell, Ansong Ni, and Arman Cohan. Quantifying contamination in evaluating code generation capabilities of language models, 2024.
14

---

Published as a conference paper at ICLR 2025
Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code generation with alphacodium: From prompt engineering to flow engineering. arXiv preprint arXiv:2401.08500 2024.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? a longitudinal perspective on LLM data contamination. In The Twelfth Internationaa Conference on Learning Representations, 2024. httpltttt/pppeeeeeee
net/fooum dww.rmmm2mmmmm24HHH2
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. /2900, 2023.
Oscar Sainz, Jon Ander Campos, Iker Garcfa- Ferrfa, Julen Etxaniz, and Eneko Agirre. Did chatgpt cheat on your test?, Jun 2023. URL
wentroa.githu................... io/
Im-contaaminatiio//ili.
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85-105, 2024. doi: 10.1
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming?, 2024.
Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. arXiv preprint arXiv:2310. 2023.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Disha Shrivastava, Hugo Larochelle, and Daniel Tarlow. Repository-Ievel prompt generation for large language models of code. In International Conference on Machine Learning, pp. 31693-31715. PMLR, 2023.
Manav Singhal, Tushar Aggarwal, Abhiject Awasthi, Nagarajan Natarajan, and Aditya Kanade. Nofuneval: Funny how code Ims falter on requirements beyond functional correctness. arXiv preprint 11X1v:2401.15963, 2024.
Benjamin Steenhock, Michele Tufano, Neel Sundaresan, and Alexey Svyatkovskiy. Reinforce- ment learning from automatic feedback for high-quality unit test generation. arXiv preprint arXiv:2310.02368, 2023.
Ruoxi Sun, Sercan O Arik, Hootan Nakhost, Hanjun Dai, Rajarishi Sinha, Pengcheng Yin, and Tomas Pfister. Sql-palm: Improved large language modeladaptation for text-to-sql. arXiv preprins 01XYv:2306.00739, 2023.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text, 2024. URL httpt:tt/////ctt settps://storaae http://sttrraag.eeeoe googleapis..coooooooooo........ com/ dewppinddmeeiiddie /td.d/ddmadaaaaiiiiaaaaa/ gemin vl report.
Catherine Tony, Markus Mutas, Nicolás E Díaz Ferreyra, and Riccardo Scandariato. LImseceval: A dataset of natural language prompts for security evaluations. arXiv preprint arXiv:2303.09384, 2023.
Michele Tufano, Shao Kun Deng, Neel Sundaresan, and Alexey Svyatkovskiy. Methods2test: A dataset of focal methods mapped to test cases. In Proceedings of the 19th International Conference on Mining Software Repositories, pp. 299-303, 2022.
Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray. Parminder Bhatia, et al. Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:22122 10264, 2022a.
15

---

Published as a conference paper at ICLR 2025
Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. Codet5: Identifier-awarc unified pre- trained encoder-decooerr models for code understanding and generation. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8696-8708, 2021.
Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922, 2023.
Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. Execution-based evaluation for open-domain code generation. arXiv preprini arXiv:2212.1048// 2022b.
Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. On learn- ing meaningful assert statements for unit test cases. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering, pp.
2020.
Jiay Wei, Greg Durrett, and Isil Dillig. Typet5: Seq2seq type inference using static analysis. arXiv preprint arXiv:2303. 09564, 2023.
Orion Weller, Marc Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. according to..." prompting language models improves quoting from pre-training data. arXiv preprint arXiv: 2023.
Weixiang Yan, Haitian Liu, Yunkun Wang, Yunzhe Li, Qian Chen, Wen Wang. Tingyu Lin, Weishan Zhao, Li Zhu, Shuiguang Deng, et al. Codescope: An execution-based multilingual multitask multidimensional benehmark for evaluating Ilms on code understanding and generation. arXiv preprint arXiv:2311.085555 2023.
Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica. Rethinking benchmark and contamination for language models with rephrased samples, 2023.
Michihiro Yasunaga, Xinyun Chen, Yujia Li, Panupong Pasupat, Jure Leskovee, Percy Liang, Ed H Chi, and Denny Zhou. Large language models analogical reasoners. arXiv preprint arXiv:2310.017144 2023.
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. Natural language to code generation in interactive data science notebooks. arXiv preprint arXiv:2212.092444 2022.
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, and Xin Peng. No more manual tests? evaluating and improving chatgpt for unit test generation. arXiv preprint arXiv:2305.0m 244.20305 04207, 2023.
Eric Zelikman, Qian Huang, Gabriel Poesia, Noah D Goodman, and Nick Haber. Parsel: A unified natural language framework for algorithmic reasoning. arXiv preprint 2rX1v:2212.1056,, 2022.
Fengji Zhang, Bei Chen, Yue Zhang, Jin Liu, Daoguang Zan, Yi Mao, Jian Hian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. arXiv preprint arXiv:2303. /2500, 2023a.
Kechi Zhang, Ge Li, Jia Li, Zhuo Li, and Zhi Jin. Toolcoder: Teach code generation models to use apis with search tools. arXiv preprint arXiv:2305.0400: 2023b.
Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. Self-edit: Fault-aware code editor for code generation. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 769-787, Toronto, Canada, July 2023c. Association for Computational Linguistics.
Kexun Zhang, Danqing Wang, Jingtao Xia, William Yang Wang, and Lei Li. Algo: Synthesizing algorithmic programs with generated oracle verifiers. arXiv preprint 23X.v:2305.14591 2023d.
Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang. Yufei Xue, Zihan Wang, Lei Shen, Andi Wang. Yang Li, et al. Codegeex: A me-ttrinee model for code generation with multilingual evaluations humaneval-x. arXiv preprint arXiv:2303. 10.22333.77550 /7568, 2023.
16

---

Published as a conference paper at ICLR 2025
Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeer:: Integrating code generation with execution and refinement. http://arrivorr/aas2440...68 2024.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your Ilm an evaluation benchmark cheater. arXiv preprint arXiv:2311.0196, 2023.
17

---

Published as a conference paper at ICLR 2025
A DATASET
A. LEGAL COMPLIANCE AND LICENSE
Our benchmark does not comprise and personal identifiable information or offensive content.
Similar to (Hendrycks et al. 2021 we scrape only the problem statements, ground-truth solutions, and test cases from competition websites - LEETCODE, ATCODER, and CODEFORCES. Further, we primarily scrape publicly visible portions of websites, avoiding any data collection that might be pay-walled or require login or interaction with the website. Following, (Hendrycks et al. 202I) we abide by Fair Use $ $107: "the fair use of a copyrighted work, including such by -.. scholarship, or research, is not an infringement of copyright", where fair use is determined by "the purpose and character of the use, including whether such use of a commercial nature educational purposes", "the amount and substantiality of the portion used in relation to the copyrighted work as whole", and "the effect of the use upon the potential market for value of the copyrighted work." Finally, we use the collected problems for academic purposes only and in addition, do not train on the collected problems.
for nonprofit
A.2 GENERATOR BASED TEST GENERATION
We use GPT-4-TURBO to construct input generators. The following prompts (Figures provide one-shot prompt templates used for synthesizing random and adversarial input generators. These generators define function returns the arguments sampled in some distribution. These generators are then executed to construct inputs which validated on the collected correct programs. We use separate generators for random and adversarial setting since often times programming problems have corner cases which might not be captured by randomly sampling over the inputs. We build 2 random input generators, adversarial input generators and check if the sampled inputs work for the correct programs. Finally, the number of collected inputs is thresholded to 100 for efficient grading (using random selection). We find that our generators can already function well but future work can study the design space of constructing such generators.
Note that for CODEFORCES, we construct the generators semi-autonomous manner since only 9 problems used.
A.3 CODE EXECUTION
The code execution split of L. LiveCodeBennch consists of 479 samples from 85 distinet problems. To
encourage diversity in our benchmark while keeping our benchmark small and usable, we place limit of six samples for each given problem. These sample programs and corresponding test cases chosen uniformly at random from all those passing the filter.
Filtering Criteria: The specific filtering criteria are follows:
Compile time:
length of code is between 100 and 500 characters, syntax errors, all necessary imports are included
Runtime: no floating point operations, true division, exp, other integer operations must have at least one argument < 3, string and list operations must have at least argument with length <. 3, finish running in 2 seconds, "reasonable" number of steps (roughly, under 1000 Python bytecode operations).
We give two examples of two programs that filtered out in the Listings below. Our final benchmark consists of 479 samples from 85 problems, but will increase size due to its live nature.
Dataset Statisties: As mentioned, filter for codes between 100 and 500 characters, as well as below 1000 steps. The statistics for programs in our dataset are shown in Fig. g
A.4 NUMBER OF TESTS
We can sample large amounts of diverse inputs using our generator- based input sampling approach. However, the number of inputs produces tradeoff in terms of the usability of the benchmark, slowing
18

---

Published as a conference paper at ICLR 2025
are an expert python competitive programmar and your goal is to construct input-generatora for testing programming contest problens. You will write relevant generators and finally construct 'construct inputs' functior that returns list of diverse inputs sampled from the generator. Remember to strictly follow the instructions and constraints present in the problem statement
<USER>
(EXAMPLE_PREL [RAMPLEEPPOBBBM)
Construct a random input generator.
</USER>
<ASSISTANT>
import numpy as np
def random_input_generre
(weight_nin, weight_max, size_min, size_max):
weights_size random, randint (size_min, size_max+1) weights np. nandan. randint (weight_min min, weight_max, size= weights_size sizee) (olist k = e.t.ranno..aaiiei (1, len (weights) +1) return weights, k
def construct_inputsll inputs_list []
## small inputs for i in range (15):
inputs_liste apppe (randon
htwpaaaaanatttntteeeeeeeeeeeeeee intpoo/_uniuuu__eetttteee (1, 10..3, 101
## medium inputs
for i in range (15):
inputs_list. hta.rpppeetrraoooooiiiieeeeeeeer (1, 10+x6, 1, 10++3))
" large inputs
for range (15):
ppputs_list.appe (random, otttrpppeeeeerrdooooooettteeeeee (1, 10..9, 1, 10xx5))
return inputs_list </ASSISTANT>
<USER> (PROBLEM)
Construct
random input generator. Use the format used in the above example by returning single function that builds diverse inputs named 'construct inputs'
</USER>
Prompt for random input generation
Figure 6: Random Input Generator Prompt
down the evaluation process. Hence, we threshold the number of sampled inputs to less than 25 based on problem length, difficulty, and number of inputs.
We base this decision by estimating model performances when using different sets of inputs random for testing in Figure [ Particularly, we vary the number of private tests between 0, 1, 5, 10, 20, 40, 60, 80, and 100 and observe exponentially decaying curve. We find 20 inputs provides good threshold which balances evaluation time with evaluation rigor.
19

---

Published as a conference paper at ICLR 2025
You are an expert python competitive progzamnar and your goal is to construct input-genezatori for testing programming contest problems. You will write relevant generators and finally construct "eonstruct_inpurs" function that returns list of diverse inputs sampled from the generator. Remember to strictly follow the instructions and constraints present in the problem statement
<USER>
[EXAMPLE PROBLEM)
Construct an adversarial input generator.
</USER>
<ASSISTANT>
import numpy as пр
#4 case 1 - altornating large and small weights def generate_adve ggnerate_ddhesaraa__pppe
(weight_size, max_weight, k);
weights [1 if 1$2==0 else mex_ne for i range (weight_size) ]
return weights, k
#1 case - equal_weight
def adversarial innsariill_iput.... generator (weight size, weight, weights Imax weight for in range (weight_sise size)) return weights, k
Case Large weights the ends def beterrrria_imputtgeeeetttt....o
(neeeerrtttr_nniittt_ttseae (weight_size max_weight, k)
pass | truncatede def cconsruuctimpausse inputs_list
veight_sizes [10, 1000, 100000] max_weights [10++3, 10++6, 10**9]
for weight_si ze in wei beiight_sieen for max_weigg ghtt
max_weights:
ks [1, 2, 5, beight_size//e weight_size-l, weight_size] for k in ks:
Eapats_list.co append adverssriaaae inputs_
weight_size max weight, truncated
return inpute_list
</ASSISTANT>
<USER> (PROBLEM)
Construct an adversarial input generator. Use the format used in the above example by returning single function that builds diverse inputs named "construct_inputs
</USER>
Prompt for adversarial input generation
Figure 7: Adversarial Input Generator Prompt
20

---

Published as a conference paper at ICLR 2025
def check (x, t]: if = !::
0: return False
for i in range (x)))))
if check :il int )x
return True
return False
@cache def punishmentNumber (n: int) int:
== 0:
return ans MuniishmentNumee (n-1)
if (str n), n):
ans return ans assert puni wwwenttumberrrrrtemmee (n 37) -- 14778
Program filtered because of multiplication
dp [True for in range (int (1e6 )
MAXN int (1e6 5)
= []
dp[0] False
dp[1] False
for in range (2, MAXN)
if not [i]: continue
(1)
for range (2 i, MAXN, i):
dpli] False
def findPrimePairs (n: int) -> list res [] for in range n):
if 8 if n $ -- 0 and > n//2: break
and > n//2: break
if dpli] and dp(n i]: res.append ([i, [i,
i1)
return res assert FindPrimePairs (n 2) 0
Program filtered because of control flow
Distribution of Execution Step Count
Distribution Code Lengths (characters)
50
Frequency
Frequency
40
"Steps"
Length
Figure Distribution code lengths and number of execution steps
21

---

Published as a conference paper at ICLR 2025
Pass@1 vs Number of Private Test Cases
60.0
Pass@1
57.5
55.0
Pas (%)
52.5
50.0
47.5
45.0
42.5
Number of Private Test Cases
Figure 9: Model performance with varying counts of private inputs used for evaluation. We find the PASS tends to stabilize when sampling about 20 inputs providing good tradeoff in evaluation time and rigor.
22

---

Published as a conference paper at ICLR 2025
B UI
LiveCodeBench Generattoon on
Sapatr Dutput Coneration
Execution
Selact date range for filtering problaws
573 2212
2/3 2202
4/1 4/1/9923
19923
9/1/2003
A21/2873
7V1/2801
4/1/2805
4/1/28033
18/1/2077
11/1/2925
12/1/2003
1/1/3094
2/1/2004
Rank
Rodo1
Pass.
Easy-Fau
Medium-.
Hard-Fa.
APT=6-Twcae-1188
SPT-6-1613
1iCader-333b-10m
1.6
Elaude:2
23.2
56
3.4
8.1
BusiCaderk-91:4..1
22.8
48.3
14
0.1
RSCadar=6.Ta-Iu
21.9
46.8
12.5
1.3
LiveCodeBench
range
probless
8/1/2005
2/1/2024
573/2923
6/1/2823
7/1/2823
8/3/2623
9/1/2922
18/1/2023
11/2/2925
12/3/2923
1/1/2604
2/1/2524
Pass
Hard-Fa.
42.5
$2.2
26.5
0.6
SPT-6-3612
36.1
75.6
21.7
0.4
0.8
Phino-368-122
56.7
0.2
Figure 10: UI of LIVECODEBENCH showing two views May-Jan and Sep-Jan. The contaminated models are blurred and the performance difference visible across the two views. The scroller on the top allows selecting different periods of time highlighting the live nature of the benchmark.
23

---

Published as a conference paper at ICLR 2025
C CURATION
C.I PLATFORM SPECIFIC CURATION
We describe the curation process for each platform.
LEETCODE. We collect problems from all weekly and biweekly contests on LEETCODE that have taken place after April' For each problem, we collect the problems, public tests, and user solutions. The platform also provides a difficulty label for each problem which we use to tag the problems as EASY, MEDIUM, and HARD. Since LEETCODE provides a starter code for each problem, we also collect it and provide it to the LLM in the STDIN format. Since the hidden tests not directly available, we use our generator- based test input generation approach (Section A.2) and also collect the auto grader failing tests for some of the recent problems.
COCOEER. We collect problems from the abc (beginner round) contests on ATCODer that have taken place after April 23. We deliberately avoid the more challenging are and agc contests which are designed for more advanced Olympiad participants. The problems are assigned numeric difficulty ratings, and we exclude abc problems with rating of more than 500. We also use these numeric ratings to tag the problems as EASY, MEDIUM, and HARD. Specifically, we use the rating brackets [0 - 200), [200 400), and [400 500] to perform the classification. ATCODER provides public and hidden tests for each problem which we directly use the benchmark.
CODEFORCES. We have collected problems from the Division and Division 4 contests on CODE- FORCES. Notably, find that even with this filter, the problems are harder than the other two platforms. CODEFORCES also provides difficulty ratings for the problems which we use to tag the prob- lems EASY, MEDIUM, and HARD using the rating brackets {800). (800 1000], and (1000 1300] respectively. Due to the higher difficulty, we only consider a small fraction of problems from CODE- FORCES and semi- automatically construct test case generators, as they do not provide complete tests on the platform (long tests are truncated).
Table 2 provides various statistics about the problems that we have collected for LIVECODEBENCH.
C.2 SCENARIO-SPECIRE BENCHMARK CONSTRUCTION
Code Generation and Self-Repair. We use the natural language problem statement as the problem statement for these scenarios. For LEETCODE, as noted above, an additional starter code is provided for the functional input format. For ATCODER and CODEFORCES problems, we use the standard input format (similar (Hendrycks al. (02T)). The collected or generated tests are then used to evaluate the correctness of the generated programs. Our final dataset consists of 511 problem instances across the three platforms.
Code Execution. We draw inspiration from the benchmark creation procedure used in (Gu et al. First, collect large pool of 2000 correct,
2024
solutions from the
Platform
Total Count
#Easy
#Medium
#Hard
Average Tests
LCB (May-end)
511
182
206
123
17.0
LCB (Sep-end)
349
125
136
88
18.0
ATCoDeR
267
99
91
77
15.6
LEETCODE
235
79
113
43
19.0
CODEFORCES
4
2
11.1
LČB-Easy
782
182
0
0
16.1
LCB-Medium
206
0
206
17.4
LCB-Hard
123
0
0
123
18.0
Table 2: The statistics of problems collected in LiveCoDEBENCh (LCBB.. We present the number of problems, their difficulty distributions and the average number of tests per problem. We present the results on the following subsets of LIVECODEBENCH (used throughout this manuscript) (a) problems in the May-Feb and Sep-Feb time windows, (b) problems sourced from the three platforms, and (c) problems in the LCB-Eas) LCB-Medium, and LCB- CBrd subsets.
24

---

Published as a conference paper at ICLR 2025
LEETCODE subset. However, many of these programs have multiple nested loops, complex numerical computations, and a large number of execution steps. Therefore, we apply compile-time and run-time filters to ensure samples details on the filtering criteria and statistics of the dataset can be found in Appendix A.3 Our final dataset consists of 479 samples from 85 problems.
reasonable, and double-check this with manual inspection. More
Test Case Output Prediction. We use the natural language problem statement from the LEETCODE platform and the example test inputs to construct our test case output prediction dataset. Since the example test inputs in the problems are reasonable test cases for humans to reason about and understand the problems, they also serve as ideal test inputs for LLMS to process. Our final dataset consists of 442 problem instances from total of 181 LEETCODE problems.
25

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL SETUP
D.1 MODELS
We describe the details of models considered in our study in Table
Model ID
Short Name
Approximate
Link
Cutoff Date
deepsekkaiideepssk-cooerr
DSCoder-33b-Ins
08/30/2023
deepseek-coders
33b-instruct
instruct
DSCoder-6.7b-Ins
08/30/2023
dcepscek-coder-60
6.7b-instruct
instruct
DSCoder-1.3b-Ins
08/30/2023
deepseek-coder- 1.00er--.3b-
1.3b-instruct
instruct
CodeLlama-70b-Ins
01/01/2023
Instruct-hf
openbmn//Eamu-70b-sif
Eurus-70B-SFT (n=l)
01/01/2023
Eurus-70b-sft
openbmb/Eurrxx-B222bnncc.
Eurux-8x222 Murux-8x22b-NCA (n=1)
04/30/2023
Eurux -ixxx-xx2bb-ne
codellamaa/Codeelllmaa33B3
Code-.lam-334b-In
01/01/2023
CodellJma--34--Isttueee
Instruct-hf
Modellama/CodeLlame 13b-
CodeLlama-13b-4
01/01/2023
CodeLLama-13--IIsttuceee
Instruct-hf
Iodellama/CooeLllame
CodeLlama-7b-hn
01/01/2023
CodeLlama-77-Instrucee
Instruct-hf
meta-llaaaaMeet-Lllamma
LLama3-8b-
01/01/2023
8B-Instruct
meta-llama/MMeta-lllmm-333
LLama3-70b-Ins
01/01/2023
Meta- leta-lllam----70-
70B-Instruce
Instruct
Phind/Phind-CodeLLlmie
Phind-34B-V2
01/01/2023
Phind- Coind-CCoeLlama34BBB
34B-v2
Smaug-2-72B
Smaug-2-72B
01/01/2023
Smaug 2-22B
Swen-1.5-72B-Cher
Qwen-1.5-72B-CCH.
01/01/2023
Qwen- 1 1.5-72B-Chat
Qwen/Code@Qeell.55
55-7
08/30/2023
CodeQwenl.553
Qven/Codee(wenl..5577
1odeQwen15-7B-
08/30/2023
Code S.t.:eeeee.-----hm
Chat
Chat
Spt-3.5-turbo-3301
GPT-3.5-Turbo-0301
10/01/2021
3pt-3.5-turbo--00
gpt-3.5-turb-0125
GPT-3.5-Turbo-0125
10/01/2021
gpt-3.5-turbo-011
gpt-4-0613
GPT-4-0613
10/01/2021
gpt gpt-4-0613
gpt-4-1106-previeew
GPT-4-Turbo-1106
04/30/2023
gpt-4- 100-previweeee
8D1-4-turbo-20224-44-99
GPT-4-Turbo-2024-
04/30/2023
gol-+-turbo-202--00-000
04-09
gpt-40-2024405-13
GPT-40-2024-05-13
10/30/2023
gpt-4o-2024-05.1
claude-2
Claude-2
12/31/2022
Claude-2
claude-instant-1
Clatae-Instant-1
12/31/2022
laude-instant-
claude-33oppss2222442299
Claude-3-Opus
04/30/2023
Glaude-3-opus-2022400
dlaude33-somme-22244222
Claude-3-Sonnet
04/30/2023
Clande-3-sonneez222440222
claude-3-haiiku-222240307
Claude-3-Haiku
04/30/2023
Claude33haaku-20240030
codestral-latest
Codestral-Latest
01/31/2024
codestral-latest
gemini-pro
Gemini- -Pro
04/30/2023
gemini- prr
Semini-1.5-pro-latte
Gemini-Pre -we.---roo1...1111
1.5-May
04/30/2023
gemmi 1.5- 1.5-pro-latest
semini-1.5-flash-latee
G-
04/33/2023
gemini- -atn
May
is--uidc/Magicooer---DDS
MagiCoderS-DS. 6.7B
08/30/2023
Magicoder-S-DS-6.7B
6.7B
ise
Se--uinccMagiicnde:-----11-
MagiCoderS-CL--BB
01/01/2023
Magicoder-S-CL-75
bigcodeestaarooder2-3b
StarCoder2-3b
01/01/2023
starcoder2-3b
bigcode/starcooee22776
StarCoder2-7b
01/01/2023
starcoder2-7b
bigcodlesttarooder2-155
StarCoder2-1Sb
01/01/2023
starcoder2-15b
26

---

Published as a conference paper at ICLR 2025
codellamaa/CodeeLLamm--
CodeLlama-
01/01/2023
CodeLlama-7000 70b-hf
hf
Base
codellama/CCoeeLlamm-B33
CodeLlama-
01/01/2023
CodeLlama-3444 34b-hf
hf
Base
codellaama/Codelllama-- 13b- hf codellamaa(CodeLLlam------i
CodeL.lama-1e
01/01/2023
Madel.lama-13b--
Base
CodeLlama-7ba
01/01/2023
CodeLlama-7b-it
deepseck-ai/deepssek.ooee 33b-base
DSCoder-33b-Bat Base
08/30/2023
deepssek--coder333bbbee
beepseek-ai/deeepsekkcooo
DSCodee-
08/30/2023
deepseck-coder-67bbbbae
7b-base
deepssek-alddeepsee-cooeee
DSCoder-1.3b-Base
08/30/2023
deepseek-cnnder-.33bbll
1.3b-base
google/codegemmaa-7
CodeGemma-7b-
01/01/2023
codegemma-7b
Base
googlelcodegemma-22
CodeGemma-2b-
01/01/2023
codegemma-2b
Base
google/gemma-7b
Gemma-7b-Ban
01/01/2023
gemma- 7b
googlc/gemma-22
Gemma-20 Batiaa---BBBasee
01/01/2023
gemma 2b
meta-llama/Mett-Llaama
LLama33700-BB
01/01/2023
Meta Secta-Llama Llama-3-705
70B
meta-laama/Mett-lllmm-33
LLama3-8h-Base
01/01/2023
Meta- Baael-llma-3--9-
8B
mistral-large-latest
Mistral-Large
01/01/2023
mmistral-large-
open-mixtral--x222b
01/01/2023
open-mixtral-8x2//
open-mixtral-8x7b
Mixtral-8x7B-in
01/01/2023
open-
coen-mixtral-8x71
m=aappppenCodiclmeeprreeee- DS-33B
OC-DS-33B
08/30/2023
OpenCCodelnterppeter--DD
33B
m-atp(@hpenCoodlhneerrreeee DS-6.7B
OC-DS-6.7B
08/30/2023
DpeenCooeenterppetee--DSS 6.7B
m-app(OpeenCoicdteeeppeeer DS-1.3B
OC-DS-L.3B
08/30/2023
OpenCoddennteppreter-DDS
1.3B
command-r
Command-R Command-R+
01/01/2023 01/01/2023
command-r
command-r+
command- T+
Table 3: Language Models Overview
We variety GPUs based on availability for running local models (A6000, L4, A100).
D.2
CODE GENERATION
Below provide the prompt format (with appropriate variants adding special tokens accommodating each instruct-tuned model) used for this scenario.
D.3
SELF REPAIR
Below we provide the prompt format (with appropriate variants adding special tokens accommodating each instruct-tuned model) used for this scenario.
D.4
CODE EXECUTION
Below we provide the prompts for code execution with and without COT. The prompts are modified versions of those from (Gu et al., 2024) to fit the format of the samples in our benchmark.
D.5
TEST OUTPUT PREDICTION
Below we provide the prompt format (with appropriate variants adding special tokens accommodating each instruct-tuned model) used for this scenario.
27

---

Published as a conference paper at ICLR 2025
You are an expert Python programmer. You will be giver a question
> problem specification) and will generate correct Python
program that matches the specification and passes all tests. You will return anything except for the program
#*# Question: in(question, inlpuettion.qustion.ccmm qutettttctooootteettt
(f staillnnnstttte_dddd
### Format: Pttomptoonstaatts..FOA..........
"* "python (question. starter_code)
else
### Format:
MESSAGE
"python
YOUR CODE HERE
(ndiff
#*# Answer: (use the provided format with backti cck))
ccks
Code Generation Prompt
(if check_resu't resultestatuute "Wrong Answer")
The above code is incorrect and does not pass the testcase. Input: (prong_testccee_inppt) Output:
Expected: twrong_testcase expectedd
(clif teeck_ooollt.rouut_.tteee. is "Time Limit Exceoded") The above code is incorrect and exceeds the time limit. Input: (wrong_testccae_innul
(elif check Boiikkrrssuttt. reruuttreeuuttsttteeee status is "Ruatime Error"] The above code is incorrect and has runtime Input: testcase input
Error Message: [#rono_reecccs__errooneeaaa
(endif)
Self Repair Error Feedback Pseudocode
28

---

Published as a conference paper at ICLR 2025
are helpful programming assistant and expert Python programer
You are helping a user write a program to solve problem. The
+ user has written some code, but it has some errors and is not
passing the tests. You will help the user by first giving a
concise (at most 2-3 sentences) textual explanation of what is
wrong with the code. After you have pointed out what wrong with the code, you will then generate fixed version of the program. You must put the entired fixed program within code delimiters only for once.
#4# Question: (qqestiionnnnnmmoooooo wwwpoooonnusiitioooooooooio
#*# Answer: 'python
[code. code to corrected) .."
4A# Format:
[Promptenstannss. Potappttooottttssooo....... BNNCINNCCCCCCCRRRROBBBSSSSSSSSSS
#*# Answer: (use the provided format with backticks)
Self- Self-Repair Prompt
You are given Python function and an assertion containing input to the function. Complete the assertion with literal (no
unsimplified expressions, no function calls) containing
output when executing the provided code on the given input, even if the function is incorrect or incomplete. Do NOT output any extra information. Provide the full assertion with the correct output in [ANSWER] and [/ANSWER] tags, following the examples.
[PYTHON]
def repeatNumber (namber int) -> int:
return number
assert repeat (number 17) [/PYTHON]
[ANSWER]
assert repeat (number 17) = 17
[/ANSWER]
[PYTHON]
def addCharactera (string : str) -> str:
return string "a"
assert adCharacterAlstrringg "x9j") [/PYTHON]
[ANSWER]
assert addihharacterAAsstiing = "x9j") = "x9ja"
I/ANSWER]
[PYTHON]
(code)
assert {input) ?? [/PYTHON]
[ANSWER]
Code Execution Prompt
29

---

Published as a conference paper at ICLR 2025
You are given a Python function ion and an assertion containing an input to
S the function. Comp-ete the assertion with a literal (no
function calls) containing the output when executing the provided code on the given input, even if the function is incorrect or incomplete. Do NOT output any extra information. Execute the program step by step before arriving at answer, and provide the full assertion with the correct output in [ANSWER] and [/ANSWER] tags, following the examples.
unsimplified expressions,
[PYTHON]
def performoperation(e):
= return "b" 5 "a"
assert performpperation(s: "hi") -- ?? [/PYTHON]
[HCUGGT]
Let's execute the code step by step:
1. The function performoperation is defined, which takes single
argument
2. The function is called with the argument "hi", so within
< function, s is initially "hi",
3. Inside the Function, s is concatenated with itself, so s becomes
< hihi".
4. The function then returns followed by the value s (which now "hihi"), and ends with "a".
string that starts with "b",
5. The return value of the function is therefore "bhihia". [/THOUGHT]
[ANSWER]
assert performoperationts "hi") "bhihia" [/ANSWER]
[PYTHON]
(code)
assert (input) == ?? [/PYTHON]
[THCUGHT]
Code Execution Prompt with CoT
30

---

Published as a conference paper at ICLR 2025
#*# Instruction: You helpful programming assistant and expert 9 Python programmer. You are helping a user to write a test case O to help to check the correctness of the function, The user has written input for the testcase. You will calculate the output of the testcase and write the whole assertion statement in the markdown code block with the correct output.
Problem:
(problem_statementt]
Function:
Clunctionssigaauure
Please complete the following test case:
assert (function_name) ((testcase_input)) == # TODO
### Response:
Test Output Prediction Prompt
31

---

Published as a conference paper at ICLR 2025
E RESULTS
E. CONTAMINATION
Figure 11
demonstrates contamination in DEEPSERK in self repair and test output prediction scenarios.
Code Generation Live Evaluation
Test Generation Live Evaluation
100
100
GPT-40 Cutoff Date
CP76.0
GPTs.
90
PASS01
60
GPT-40
PASS@1
60
40-
40
20-
DS-Ins Release Date
DS-Ins Release Date
May
Jul Aug Sep Oct Nov Dec Jan Feb
May
Jul Aug
Oct
Nov Dec Jan
LEETCODE Problem Release Month
LEETCODS Problem Release Month
Code Repair Live Evaluation
Code Execution Live Evaluation
100
80
GPT-40
PASS01
60-
PAsstD1
60-
40-
40
DS Ins Release Date
D5-Ins Release
May
Jun Jul
Aug Oct Dec Jan
May
Jun
Jul
Aug
Sep
Oct
Nov
LEETCODE Problem Release Month
LBETCODE Problem Release Month
Figure 11: Contamination in DS-B models across self-repair and code execution (without COT) scenarios over time. Note that code execution currently runs between May and November
Code Generation Live Evaluation
100
Plak
GPT4
80
PASS
60
40-
20-
May
Jun
Aug
Sep
Nov
Dec
Feb
ATCODER Problem Release Month
Figure 12: Performance on problems released over different months for ATCODER
E.2 ALL RESULTS
Below provide the tables comprising of results across different LIVECODEBENCH scenarios. Comparing Closed Models. We evaluate a range closed (API access) models ranging from different model families like GPTs, CLAUDES, GEMINI, and MISTRAL. We find the GPT-4-TURBO and
32

---

Published as a conference paper at ICLR 2025
All Live Evaluations
All Live Evaluations
80
80
GPT-4 Turbo 1126
Cludd
GP Turbo
Cladd
DSCoder 33b Ins
Mistra Large
DSCoder 336
Mizra Large
DSCoder-35t- Bose
Google-Gemi--rro
DSCoder-3So Base
Gregle-emiinPrr
60-
G0
PASS00
PASS@1
40-
May Jun Jul Aug Sep Oct Nov Dec Jan Feb
May Jun Jul Aug Sep Oct Nov Dec Jan Feb
LEETCODE Problem Release Month
ATCODER Problem Release Month
Figure 13: Live evaluation over time for various models code generation scenario in LIVE- CODEBENCH. We consider many recently released models and do not find significant performance variations across months except for DS-B models.
1.00
genorationn
roparr
0.56
Tcctpon
CIcCuEon
getwattion
Indien
eveciation
Figure 14: Correlations across different scenarios studied in LIVECODEBENCH
CLAUDE-3-OPUS rank at the top across all scenarios followed by MISTRAL-L and CLAUDE-3-SONNET models. Finally, GEMINI-PRO and GPT-3.5-TURBO lie the lower end of the models. The relative differences between the models vary across the scenarios. For example, GPT-4-TURBO demonstrates remarkable improvement from self-repair (24.5% 36.9% the I.CB-Medium problems) while GEMINI-PRO only improves from 8.5% to 9.4%. Similarly, as identified above, CLAUDE-3-ORD and MISTRAL-L perform considerably better on test output prediction and code execution scenarios.
E.3
C++ RESULTS
We evaluate model performances with C++ in Table As see, model performances roughly align across the two languages.
33

---

Published as a conference paper at ICLR 2025
LIVECODEBENCH
GPT4-Turbo
39.7
GPT4
36.9
Gemini-Pro-1.
35.7
Claude-O
35.4
Gemini-Flash-113
30.0
LLama3-Ins-708
Claude3-S
26.9
Mix-8x228-Ins
26
Mistral-L
26.0
Claude2
24.1
Claude-Ins-1
DS-Ing-33B
23.6
23.7
API Access
GPT3.5 5- Turbo
22.5
Phind-340 348
21.0
[Lama3-Base Base-B0 70B
20.3
DS-Ins-6 7B
19.1
Open Access
LLamaž-Ins-8B
15.3
DSS-aae-338
15.0
SC2 B5se-158
14.2
CodeGemma-rD--- Base
13.8
0 78
Mix-&x7B-Ins
12.3
LLama3 BBaseeBBBB
12.3
DS-Ins-1 38
3.0
DS Base-1.3B
3B
5
10
15
20
25
30
35
40
PASS@1
Figure 15: Comparison of open access and (closed) API access models on LIVECODEBENCH- code generation scenario. We find that closed-access models consistently outperform the open models with only strong instruction-tuned variants of >> 30B models (specifically 13-INs-70B, MIXTRAL and OS-INS-31D 33B models) crossing the performance gap.
34

---

Published as a conference paper at ICLR 2025
Model Name
Easy
Medium
Hard
Total
Claude-2
61.80
4.90
0.20
22.30
Claude-3- 3-iiku
63.00
4.30
1.10
22.80
Blaude-3-Opun
78.80
16.30
3.20
32.80
Claude-3-Sonnet
67.60
6.20
1.10
25.00
Claude-Instante
60.70
4.30
1.10
22.10
CodeGemma-2b-BBa..
18.30
0.40
0.00
6.30
CodeGemma-77-BBl
35.70
2.60
0.10
12.80
BodeeLlama-13--Be
24.60
0.90
0.00
8.50
Codeellmm-133-1in
36.60
2.40
0.00
13.00
CodeLJlama-34b-BBl
32.20
1,80
0.10
11.40
Baleellama344-llee
33.70
2.40
1.10
12.40
CodeLlama-700 700-Baase
15.80
1.20
0.00
5.70
CodeLlamma-70-Ing
7.80
0.60
0.00
2.80
CodeL.lama-7b-Bart
19.00
0.40
0.00
6.50
CodeLlama-
28.60
2.50
0.00
10.40
CodeQwenI 5-7B 55-B
40.40
4.80
0.00
15.10
Code@weel15-7BB-Chr
39.20
13.10
0.50
17.60
Codestral-Latest
69.00
18.70
0.90
29.50
Command-R
39.00
3.60
0.00
14.20
Command-R+
56.60
6.80
0.00
21.10
ASCoder-113bb-a
17.30
0.70
0.00
6.00
DSCoder-113b-ll
22.90
1.50
0.00
8.10
DSCoder-33b-Bl -aadder-3b-BBse
39.40
2.30
0.00
13.90
DSCoder-33b-In
55.60
9.00
0.70
21.80
DSCoder-6.7b-BB Baaa
34.30
1.40
0.10
11.90
DSCoder-6.7b-Ins
46.40
5,80
0.70
17.60
GPT-3.5-TTurb-00225
56.80
10.80
0.10
22.60
GPT-335--Turho-0300
53.40
8.80
0.20
20.80
GPT-4-0613
78.40
21.20
2.30
33.90
GPT-4-Turbo-1106
84.40
24.00
0.50
36.30
GPT-4-Turo--222404--99
85.30
33.00
5.10
41.10
GPT40-2024405--13
88.30
33.20
4.20
41.90
Gemini-Flash-1155MMay
68.10
12.60
2.70
27.80
Gemini-Pro-1.5--1 (n=l)
56.50
14.30
3.60
24.80
76.00
19.40
3.50
33.00
Gemma-
Comma--2bBBl 3b-Base
6.10
0.00
0.00
2.00
Gemma-7bb 7b-Base
27.00
0.90
0.00
9.30
LLLama3-70b-Bll
52.20
3.20
0.60
18.60
LLama3-70b-in
60.70
15.80
1.40
26.00
LLama3-8b- 8b-Base
32.90
1.50
0.00
11.50
LLama3-8b-Ine
38.60
3,50
0.50
14.20
MagiCoderS-CL--IB
32.80
2.40
0.00
11.70
MagiCoderS-DS-6.77
49.20
7.50
0.00
18.90
Mistral-Large
60.20
10.90
0.90
24.00
Mixtral-8x22B-N5
59.80
12.70
0.00
24.20
31.60
2.60
0.00
11.40
OC-DS-1.3B
11.30
0.10
0.00
3.80
OC-DS-33B
53.90
5.10
0.00
19.70
OC-DS-6.7B
46.30
4.50
0.00
16.90
Phind-34B-VV V2
53.40
4.70
0.10
19.40
StarCoder2-15b
37.30
2.20
0.00
13.20
StarCoder2-3b
28.20
0.70
0.00
9.60
StarCoder2-7b
29.90
1.20
0.00
10.40
Table Code Generation Performances
35

---

Published as a conference paper at ICLR 2025
Model Name
Easy
Medium
Hard
Total
Claude-2
66.20
10.30
0.40
25.60
Claude-3-Haie
66.50
8.70
2.50
25.90
Claude-3-Opur
83.10
23.70
6.70
37.80
Claude-3-Sonee
72.60
11.80
2.20
28.90
Claude-Instant-1
64.40
7.10
2.20
24.60
CodeLlama-13b-Ins
43.10
3.00
0.00
15.30
CodeLlama-34b-Ing
31.50
3,50
1.80
12.30
CodeLlama-7b-Im
31.90
3.10
1.50
12.10
Codestral-LLatest
72.50
25.90
3.30
33.90
1.Cooer:
29.50
2.10
0.00
10.60
DSCoder-33b-Ins
60.70
8.10
1.50
23.40
DSCoder-6.7b-Ins
49.90
5.70
1.10
18.90
GPT-3.5-Turbo-011
59.30
11.90
0.50
23.90
GPT-3.5-Turbo-0301
58.40
11.60
0.70
23.60
GPT-4-0613
79.30
25.00
2.40
35.60
GPT-4-Turbo-1106
86.90
36.90
4.00
42.60
GPT44-Turbo-2224-004.099
88.70
39.70
8.40
45.60
GPT-40-2024-05--3
92.60
46.40
8.20
49.10
Gemini-Flash-I.5-Maay
73.40
16.40
4.40
31.40
Gemini-
53,80
9.40
0.20
21.10
Gemini-Pei Semini-Pro-11e 1.5-Apll (n=1)
71.80
19.40
5.50
32.20
Gemini-Pree
84.80
30.10
7.30
40.70
LLama3-70b-Ins
69.60
19.00
1.80
30.10
LLama3-8b-Ins
47.10
6.10
0.00
17.70
MagiCoderS-CL.. SCCL
36.50
3.10
0.00
13.20
MagiCoderS-DS-6.77
50.60
8.60
0.00
19.70
Mistral-Large
71.20
15.60
3.60
30.10
OC-DS-1.3B
20.00
0.40
0.00
6.80
OC-DS-33B
58.90
7.20
1.30
22.50
OC-DS-6.7D
50.90
6.30
0.20
19.10
Phind-34B-V2
62.00
6.50
0.90
23.10
Table 5: Self Repair Performances
36

---

Published as a conference paper at ICLR 2025
Model Name
Pass
Claude-2
32.70
Claude-3-Haide
32.90
Claude-3-Opus
58.70
Claude-3-Sonnet
34.10
Claude-Instant-
25.40
BadeLllma-13b-Ins
24.40
CodeLlama-34b-In
23.00
CodeLlama-70b-
16.10
CodeLlama-7b-
15,30
Codestral-Latest
41.80
DSCoder-113b--ns
12.50
DSCoder-33b-Ins
28.30
DSCoder-6.7b-Ins
26.50
GPT-3.5-TTubo-01225
35.40
GPT-3.5-Turbo-030
32.50
GPT-4-0613
52.90
GPT-4-Turbo-1106
55.70
GPT-4-TUrro--2044400009
66. 10
GPT-440-0224-05-13
68.90
Gemini-Flash-L.5MMayy
38.10
Gemini-Pro
29.50
Gemini-Pro-I.5-Aprii (n=l)
49.60
44.80
LLama3-70b-Ins
41.40
LLama3-8b-Ins
24.40
MagiCoderSSCCL-7B
21.30
MagiCoderS-DS-6.77
27.10
Mistral-Large
46.50
Mixtral-8x2258
44.70
Mixtral-8x7B-Ia
31.80
OC-DS-
7.80
OC-DS-33B
11.30
OC-DS-6. 6.7B
18.30
Shind-34B-Ve
27.20
Table 6: Test Output Prediction Performances
37

---

Published as a conference paper at ICLR 2025
Model Name
Pass @
Pass @ (COT)
Claude-2
31.50
43.80
Claude-3-Haiki
0.70
28.30
Claude-3-Ope 3-ouus
36.50
80.10
Claude-3-Sonnct
29.30
39.40
Claude-Instante
20.00
34.80
Cllama- -
23.50
14.10
Cllama-34b-In
28.90
24.50
Cllama-7b-Ins
20.60
14.20
CodeLlama-70b-Ing
31.20
-1.00
Codestral-Latest
37.90
41.80
DSCoder-L.3b-Base
19.00
13.40
DSCodeer:13bb-Is
18.10
17.00
DSCoder-33b-Bas
29.90
29.10
DSCoder-33b-Ins
26.60
31.70
DSCoder-6.7b-Ban
23.50
25.10
DSCoder-6.7b-Ins
23.10
23.80
GPT-3.5-Turbo-0501
33,90
34.80
GPT-4-0613
44.30
64.80
GPT44-Trbbo-1166
40.50
83.60
GPT-4-TTrho-2224440449999
45.90
83.80
GPT-40-2024-05-13
39.10
91.00
Geminn-FFaah--.5-MMayy
21.40
57.10
Gemini-Pro
27.70
37.40
Gemini-Pro- (April) (n=1)
30.30
64.40
Pcmini-Proo1.5-May
42.10
72.10
LLama3-70b-Ins
29.60
55.50
LLama3-8b-Ins
18.40
29.40
MagiCoderS-CL--1
21.20
-1.00
MagiCoderS-DS
27.20
-1.00
Mistral-Large
36.60
54.40
Phind-34B-V2
26.90
-1.00
StarCoder
20.30
-1.00
WCoder-34B-VV
28.40
-1.00
Table 7: Code Execution Performances
Model
PASS
GPT-4o
49,8
Claude-3.5-Sonnet
45.9
Table 8: Results for models with C++ evaluations on the entire 713 LIVECODEBENCH problems
38

---

Published as a conference paper at ICLR 2025
F QUALITATIVE EXAMPLES
F.1 CODE EXECUTION
We show examples from the code execution task that GPT-4 (gpt-4-110) 1a0----11-1----ieie- previeew still struggles to execute, even with CoT.
def countWays (nums: List [int]) -> int: nums. sort
len (nums)
ans for in range (n + 1):
if and nums [ims]i-1] > i: cont nue if
and nums [i] <= i: continue ans +=
return ans assert countWays (nams [6, 0, 3, 3, 6, 7, 2, 71)
==
GPT-4 COT Outputs: 1, 2, 4, 5
Mistake
def minimumCoins (prices: List [int]) int:
écache
def dfs (i, free_until);
if >- 1en (prices):
return
res pricesli] + dfs + i, min (len (prices) 2, i i 1
if free until
res min (res, dfs (i 1, free_until))
return
caccchccceeearr return dfs (0, -1)
assert minimumCoins (prices [3, 1, 21 == 4 GPT-4 CoT Outputs: 1, 3, 6
Mistake
def sortVowels (s: str) -> str:
= deque (sorted (ch for ch in s if vowel (ch) )
res []
for ch in (ch)
res.append popleft )
else:
aps.ap (ch)
" " join (res)
assert sortvowels (s 'LEeteOde") '1EOtcede'
GPT-4 CoT Outputs: "Leetecode", "LEetecOde", "Leetcede", "leetcEde 4 "LeetcOde"
Mistake
39

---

Published as a conference paper at ICLR 2025
def relocateMarbles (nums: List lintl, moveFrom: List [int], moveTo: List [ intl) -> List [int]:
nums = sorted (list (set (nums)
dd {)
for item in nums:
dd [dtemmm] = for a in zip (moveFrom, moveTo)
de1 [d[a] dd[b] - 11 dd.keys return sorted
assert relocateMarbles (nums = [1, 6, 7, 8), moveFrom [2, 7, 21.
moveTo 12, 9, 5] [5, 6, 8,
GPT-4 Outputs: [2, 6, 8, 91, 12, 5.
9], KeyBrror
Mistake
def minimumSum (nums: [ist(int]) -> ints left, right, ans = [inf], [inf], inf for num in nums:
left. Left.append (1eft [-1], n for num in nums [::-1]:
right. (min (right [-1], num) right. reverse)
for i, num in enumerate (nums) if left [i] ans min (ans, num left [i] right [i + 11)
and right 1] num:
return ans if ans < inf else -1 assert minimumSum (nums [6, 5, 4, 3, 4, 51) == -1 GPT-4 CoT Outputs: 10, 11, 12
Mistake
40

---

Published as a conference paper at ICLR 2025
G LIMITATIONS
Benchmark Size. LIVECODEBENCH code generation scenario currently hosts over 500 instances from problems released since May 2023. To account for contamination in DEEPSBEK, we only perform evaluations on problems released after the model cutoff date. This leads to only 349 problems used in our final evaluations which might add noise due to problem set samples. We currently estimate 1 - 2% performance variations in LCB code generation due to this issue (measured by bootstrapping 349 sized problem sets from the full dataset). Other scenarios, i.e. self-repair, code execution, and test output prediction comprise 238, 188, and 254 problems would have similar performance variations. We thus recommend exercising proper judgement when comparing models with small performance differences. Note that HUMANEVAL has 164 problems and would also struggle with similar issues.
This issue also exacerbated for newer models, with more recent cutoff dates, as they might only have access to smaller evaluation set. We propose two solutions addressing this issue LIVECODEBENCH. First, we will use other competition platforms for problem collection, allowing larger number of recent problems to be added the benchmark. In addition, also hope supplement this with an unreleased private test set constructed specifically for model evaluation. These problems will use similar flavor current problems and will be used when models submitted for evaluation to the LIVECODEBENCE platform. This would reduce the reliance on public accessible problems and provide more robust evaluation of the models while providing community public access similar problems, similar to strategies employed by popular platforms like KAGGLE.
evolve
Focus on PYTHON. LIVECODEBENCH currently only focuses on PYTHON which might not provide enough signal about model capabilities in other languages. However, since we collected problem statements and serialized tests, adding new programming languages would be straightforward once appropriate evaluation engines are used.
Robustness to Prompts. Recent works have identified huge performance variances that can be caused due to insufficient prompt. Here, we either do not tune prompts across models or make minor adjustments based on the system prompts and delimiter tokens. This can lead to performance variance in our results. Our findings and model comparison orders generalize across LIVECODEBENCH scenarios and mostly match the performance trends observed on HUMANEVAL making this less prominient issue.
This issue can be particularly observed open models on the code execution scenario with COT prompting. Interestingly, often the open models perform even worse in comparsion to the direct code execution baseline. Note that we used same prompts for the closed models all of which show noticable improvement from COT. While the used prompts might be sub-optimal, this highlights how open-mode moeelsssls perform worse against the closed models performing maain-o--thought
Problem Domain. Programming is vast domain and occurs in various forms such as programming puzzles, competition programming, and real-world software development. Different domains might have individual requirements, constraints, challenges, and difficulty levels. LiVeCoDEBENCH currently focuses on competition problems sourced from three platforms. This might not be representative of the "most general" notion of LLM programming capabilities. Particularly, real-world usage of LLMS is drawn upon open-ended and unconstrained problems rasied by users. We therefore recommend using BENC as starting point for evaluating LLMS and further using comain--pecifie evaluations to measure and compare LLMS in specific settings as required.
41


Reviews:
[{"review_id": "5ppFIJOd1Q", "reviewer": "Reviewer_zPuZ", "summary": "The paper introduces LiveCodeBench, a lively-updated dynamic benchmark that is designed to evaluate large language models (LLMs) for coding tasks with minimized data contamination risk.\n\nAside from being contamination-free, LiveCodeBench also seeks to improve the data quality, difficulty, and task diversity of the benchmark.\n\nThe paper also conducts thorough experiments exposing the significant data contamination and overfitting issues.", "questions": "## Discussion Questions\n\nHave you considered addressing more intricate data contamination issues like re-phrasing within the scope of this work?\n\nI believe that many code-specific techniques can be developed to detect re-phrasing contamination such as checking the equivalence or similarity of the test cases and canonical solutions.\n\nI would be happy to raise my score if the authors could incorporate and address those intricate contamination issues in the revision.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "## Significance\n\nThe paper addresses one of the most important issues: data contamination in the existing Code LLM evaluation process. The benchmark is highly practical and should be known and tried by the Code LLM community to encourage more rigorous evaluation of models. \n\n\n## Soundness\n\nThe paper demonstrates strong soundness through its detailed analyses of the models' performance over time, overfitting issues on HumanEval, and comparison of different types of models.\n\n\n## Effectiveness\n\nDynamic benchmarking is indeed an effective solution to mitigate data contamination.", "weaknesses": "## Novelty\n\nThe paper presents two key contributions:\n\n1. A comprehensive dynamic benchmark for coding takes\n\n2. Analyses exposing the contamination issues in coding tasks\n\nWhile these contributions are highly relevant and practical for the Code LLM community, it's important to note that dynamic benchmarking is a technique that has been utilized in various prior works [1,2,3,4] as early as 2021 [1]. Additionally, the issue of data contamination has been systematically examined in earlier studies [5,6], limiting the novel insights that the paper can offer to the broader ICRL community.\n\nTo enhance the paper's novelty, it may be beneficial to develop better and more efficient dynamic benchmarking techniques, or introduce new methods to alleviate more intricate contamination issues such as re-phrasing [6].\n\n\n\n\n[1] [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://arxiv.org/abs/2106.06052)\n\n[2] [Competition-Level Problems are Effective LLM Evaluators](https://arxiv.org/abs/2312.02143)\n\n[3] [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://arxiv.org/abs/2312.12343)\n\n[4] [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://arxiv.org/abs/2312.14890)\n\n[5] [To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination](https://openreview.net/forum?id=m2NVG4Htxs)\n\n[6] [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://arxiv.org/abs/2311.04850)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8nFPpRemwB", "reviewer": "Reviewer_MFUT", "summary": "Prior LLM coding benchmarks suffer from contamination and saturation issues. This paper introduces LiveCodeBench, a new benchmark that collects new problems from public programming contests (LeetCode, AtCoder, and Codeforces). Besides code completion, the benchmark also tests the self-repair and output prediction abilities of LLMs. The paper provides a comprehensive analysis on over 600 problems and over 50 LLMs, demonstrating that time-segmented evaluations are useful for detecting contamination.", "questions": "It seems that the drop in performance occurs only for the LeetCode problems and is smooth for other platforms like AtCoder. Do you have any explanation for this interesting phenomenon?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper proposes a coding benchmark with live updates, ensuring fair model evaluation by testing only on new problems after each model’s cutoff date. The time-segmented analysis also reveals a significant drop in performance of some models after release date, indicating notable contamination issues.\n- The paper is well-written and organized, with a thorough appendix including discussion on legal compliance and benchmark creation details.", "weaknesses": "- The novelty is limited. Using competitive programming problems for LLM evaluation has been well adopted. Additionally, the evaluation scenarios in this paper are also not new but borrowed or adapted from prior work (e.g., the test case output prediction can be seen as a chain-of-thought prompting solution of the code execution task). The main contribution of this paper is to scrape the problems with appropriate difficulty and analyze the results in a time-segmented way.\n- The relatively small number of problems (~40 LeetCode problems every two months) raises concern about the reliability of the results. I wonder if it’s possible to estimate the variance of the pass@1 performance and the statistical significance of the comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jOYEOznCD4", "reviewer": "Reviewer_SgU1", "summary": "This paper worked on the evaluation of LLMs for code, proposing a benchmark named LiveCodeBench. The benchmark will be continuously updated by automatically collecting problems from online code contest sites to overcome data contamination. It contains four code-related tasks: code generation, self-repair, code execution, and test output prediction. A large-scale evaluation was conducted on 50 LLMs, which revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.", "questions": "The contribution of this work to the community depends largely on your continued maintenance efforts. Do you have a long-term plan?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Benefiting the community**: The benchmark will be continuously updated and provide code ability evaluation with less contamination.\n2. **Large scale evaluation**: The authors evaluated 50 LLMs with carefulness about contamination, providing a valuable reference for the code-related capabilities of each model.\n3. **Having insights**: The experiments revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.", "weaknesses": "1. **Unproven data representative**: Besides contamination, we also need to know how many code problems we need and how representative they are, to conduct a comprehensive evaluation. However, they were not answered in this paper. Are there mechanisms to guarantee or prove these properties, in particular for new models?\n\n2. **Unevaluated workflow reliability**: While the workflow of benchmark construction is completely automated, this workflow's reliability was not evaluated. For example, the accuracy of the HTML extractor.\n\n3. **Unproven test case completeness**: Averaged 18 of the test case count basically is far fewer than test cases used inside the programming task websites; the completeness or sufficiency of these test cases was not analyzed in this paper.\n\n4. **Biased filtering**: In the code competition scenario that this work focused on, questions with multiple correct outputs for a single input would contain many unique features or have different problem-type distributions. However, they were all removed in this work, which might lead to a bias. Further analysis of the impact of this filtering should be conducted.\n\n5. **Multi-step track needed**: This work's designs of four tracks were motivated by AlphaCodium. Including AlphaCodium, state-of-the-art competition-level code generation works nowadays broadly applied multi-step workflow. However, besides one-step revision, multi-step code generation was not evaluated in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "glCX2nyDzi", "reviewer": "Reviewer_9FRx", "summary": "This paper introduces LiveCodeBench, a new benchmark designed to evaluate LLMs for code-related tasks. \nLiveCodeBench addresses some key limitations of previous benchmarks, \nincluding issues like data contamination, overfitting, saturation, and limited application range. \nUnlike existing benchmarks, LiveCodeBench assesses not only code generation but also self-repair, code execution, and test output prediction. \nAlongside proposing the benchmark, \nthe authors perform a comprehensive evaluation and analysis of the contamination and overfitting problem in popular LLMs. \nThe paper also introduces a difficulty-guided problem curation strategy, \nwhich enhances the evaluation's effectiveness by creating clearer margins between model performances, \nallowing for a more meaningful comparison.\nThe evaluation covers 52 models, with sizes ranging from 1.3B to 70B, \nproviding a fairer and more reliable understanding of model performance in code-related tasks.", "questions": "1. The tasks of *code execution* and *test case output prediction* appear to be quite similar. \n   According to the paper, the distinction lies in their prompt: \n   code execution is to predict the output based on inputs and functions in *programming language*, \n   while test case output prediction is to predict the output based on inputs and function descriptions in *natural language*. Given the similarity between these two tasks, is there any observable correlation in their results?\n2. Test case generation typically is designed as predicting the entire test case from a given function implementation [3,4], \n   which is more practical in real-world software engineering and testing applications, \n   such as in tools like Copilot. \n   I'm curious why the authors choose a different approach to evaluating test generation in the form of $(I, f_{\\text{NL}}) \\mapsto O$, \n   where the input and natural language function descriptions are used to predict the output.\n3. In the **Problem Difficulty** paragraph of Section 3.1, \n   the authors state that they collected problems of varying difficulty levels as labeled by competition platforms. \n   However, how did the authors address potential rating bias across these platforms? \n   Given that different competition platforms are designed for distinct user groups and, as the authors note, CodeForces problems are generally harder than those on other platforms, \n   there is likely a selection bias in difficulty ratings across different platforms.\n\n[3]: Nie, Pengyu, et al. \"Learning deep semantics for test completion.\" ICSE 2023.\n\n[4]: Rao, Nikitha, et al. \"CAT-LM training language models on aligned code and tests.\" ASE 2023.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* LiveCodeBench has a live update mechanism that mitigates data contamination and enables continuous growth. \n  This mechanism ensures that the benchmark remains useful for evaluating newer models, \n  even those with a knowledge cut-off date beyond its release.\n* With the timestamps on the tasks, the authors conduct a novel \"live\" evaluation of code generation to directly address data contamination issues. \n  This approach offers deeper insights into the models' actual coding capabilities, focusing on genuine problem-solving rather than mere memorization.\n* The difficulty guided problem curation is particularly effective in revealing performance differences between models of similar sizes, making it practical to use for meaningful evaluation.\n* The paper employs a clustering method to identify models that overfit HumanEval. \n  While overfitting to older benchmarks is a well-known issue, \n  the authors' method offers empirical evidence and analysis that specifically highlights the models affected by this problem.", "weaknesses": "* Novelty: The benchmark's construction primarily aims to create a newer and harder version of existing benchmarks. \n  However, a key issue with LLM benchmarks for code is that Olympiad in Informatics (OI) programs are not typically representative of real-world software engineering and programming languages. \n  OI programs often have a distinct style and differ in their learned representations compared to code from real software projects. \n  This has led to a shift in focus in this research area from OI competitive programming to open-source software projects, as seen in recent work like SWE-bench [1] and RepoBench [2].\n\n[1]: Jimenez, Carlos E., et al. \"Swe-bench: Can language models resolve real-world github issues?.\" ICLR 2024.\n\n[2]: Liu, Tianyang, et al. \"Repobench: Benchmarking repository-level code auto-completion systems.\" ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)