LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

## Abstract

Abstract Large Language Models ( LLMs ) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry.
However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval , MBPP ) are no longer sufficient for assessing their capabilities.
In this work, we propose LiveCodeBench , a comprehensive and contamination-free evaluation of LLMs for code, which collects new problems over time from contests across three competition platforms, namely LeetCode , AtCoder , and CodeForces .
Notably, our benchmark also focuses on a broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation.
Currently, LiveCodeBench hosts over five hundred coding problems that were published between May 2023 and May 2024.
We have evaluated 18 18 base LLMs and 34 34 instruction-tuned LLMs on LiveCodeBench .
We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons.
We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and models.

## 1 Introduction

Code has emerged as an important application area for LLMs, with a proliferation of code-specific models
(Chen et al., ; Austin et al., ; Li et al., ; Zhong et al., ; Allal et al., ; Li et al., ; Roziere et al., ; Guo et al., ; Luo et al., ; Royzen et al., ; Wei et al., ; Ridnik et al., ; Lozhkov et al., ) and their applications across various domains and tasks such as program repair  (Zheng et al., ; Olausson et al., ), optimization (Madaan et al., ), test generation (Steenhoek et al., ), documentation generation (Luo et al., ), tool usage (Patil et al., ; Qin et al., ), SQL (Sun et al., ), and more.
In contrast with these rapid advancements, evaluations have remained relatively stagnant, and current benchmarks like
HumanEval, MBPP, and APPS may paint a skewed or misleading picture.
Firstly, while coding is a multi-faceted skill, these benchmarks only focus on natural language-to-code tasks, thus overlooking broader code-related capabilities.
Moreover, these benchmarks may be subject to potential contamination or overfitting, as benchmark samples are present in the training datasets.

Figure: Figure 1: LiveCodeBench comprises problems marked with release dates, allowing evaluations over different time windows. For newer models, we can detect and avoid contamination by only evaluating on time-windows after the model’s cutoff date. The figures demonstrate the performance of models on code generation and test output prediction LiveCodeBench scenarios with LeetCode problems released across the months between May $2023$ and February $2024$. Notice that DeepSeek-Instruct and GPT-4-O perform considerably worse on problems released since September and November $2023$ (their release and cutoff dates respectively!) – indicating potential contamination for the earlier problems. Thus, while performing evaluations, we use the post-September/post-November time window (green) for fairly comparing these models.
Refer to caption: figures/contamination_new/main_14_codegen_leetcode.png

Figure: Figure 2: Left. We propose evaluating LLMs across scenarios capturing various coding-related capabilities. Specifically, we host four different scenarios, namely code generation, self-repair, code execution, and test output prediction. The figure depicts various model performances across the four scenarios available in LiveCodeBench in a radial plot – highlighting how relative differences across models change across the scenarios. Right. Comparison of open access and (closed) API access models on LiveCodeBench-Easy code generation scenario. We find that closed-access models consistently outperform the open models with only strong instruction-tuned variants of $>30$B models (specifically L3-Ins-70B, Mixtral and DS-Ins-33B models) crossing the performance gap.
Refer to caption: figures/overview/tasks_radar.png

Motivated by these shortcomings, we introduce LiveCodeBench,
a holistic and contamination-free benchmark for evaluating code capabilities.
LiveCodeBench is built on the following principles:

- 1.
Live updates to prevent contamination.
LLMs are trained on massive inscrutable corpora, and current benchmarks
suffer from the risk of data contamination as they could be included in those training datasets.
While previous works have attempted decontamination using both exact and fuzzy matches (Li et al., , ), it can be a non-trivial task  (Team, ) and can be evaded using simple strategies like rephrasing (Yang et al., ).
Here, to prevent the risk of problem contamination, we use live updates,
that is evaluate models on new problems.
Particularly, we collect problems from weekly contests on competition platforms
and tag them with a release date.
Next, for newer models, we only consider problems released after the model’s cutoff date to ensure that the model has not encountered the exact problem in the training dataset.
In Figure 1, we find that the performance of the DeepSeek model starkly
drops when evaluated on the LeetCode problems released after August $2023$.
Similarly, GPT-4-O observes a drop in performance on LeetCode problems released since November 2023, its specified cutoff date.
This indicates that these models are likely trained on the older LeetCode problems and time-segmented evaluations allow fair comparisions.
- 2.
Holistic Evaluation.
Current code evaluations primarily focus on natural language to code generation.
However, programming is a multi-faceted task that requires a variety of capabilities beyond those measured by code generation.
In LiveCodeBench, we evaluate code LLMs on three additional scenarios, listed below.
•
Self-Repair. Fix an incorrect program from execution information, evaluating the ability to debug code from feedback. The model is given the natural language problem description, the incorrect program, the test case it fails on, and the execution feedback from that failure. The output should be a correct repaired program.
•
Code Execution. “Execute” a program on an input, evaluating code comprehension ability. The model is given a program and an input, and the output should be the result.
•
Test Output Prediction. Solve the natural language task on a specified input, evaluating the ability to generate testing outputs. The model is given the natural language problem description and an input, and the output should be the output for the problem.
Figure 2 (left) depicts performance on the different scenarios considered in LiveCodeBench.
- 3.
High-quality problems and tests.
High-quality problems and tests are crucial for reliable evaluation of LLMs.
However, prior works have revealed deficiencies in existing benchmarks.
(Liu et al., ) identified insufficient tests and ambiguous problem descriptions in HumanEval. They released HumanEval+, a variant of the benchmark with more tests and sometimes saw up to an 8% drop in performance.
Similarly, (Austin et al., ) had to create a sanitized MBPP subset to disambiguate problem descriptions.
In LiveCodeBench, we source the problems from reputable competition websites whose quality is already validated
by the platform users.
In addition, for every problem, we provide a good number of tests (about $17$ on average) for meaningful and robust evaluations while still finishing quickly.
- 4.
Balanced problem difficulty.
Competition programming is challenging for even the best-performing LLMs, and most of the current SoTA models achieve close to zero performance on a majority of problems.
As a result, they can be unsuitable for meaningful comparing today’s LLMs because the variance in performances is low.
Furthermore, the averaging of evaluation scores across problems with different difficulty levels artificially minimizes the differences between models.
Therefore, we use problem difficulty ratings (sourced from the competition websites) for filtering the harder problems and
classifying problem difficulties to
ensure balanced problem difficulty distribution and allow granular model comparisons.

With these principles in mind, we build LiveCodeBench, a continuously updated benchmark that avoids data contamination.
Particularly, we have collected $511$ problems from contests across three competition platforms – LeetCode, AtCoder, and CodeForces occurring from May $2023$ to the present (May $2024$) and use them to construct the different LiveCodeBench scenarios.
Empirical Findings.
We have evaluated $18$ base models and $34$ instruction-tuned models across different LiveCodeBench scenarios.
Below, we present the empirical findings from our evaluations, which have not been revealed in prior benchmarks.

- 1.
Contamination.
We observe a stark drop in the performance of DeepSeek, GPT-4-O, and Codestral on LeetCode problems released after Aug $2023$, Oct $2023$, and Jan $2024$ (Figure 1). These results highlight likely contamination in older problems and time-segmented evaluations
prove effective for performing fair comparisons.
- 2.
Holistic Evaluation.
Our evaluations reveal that model performances are correlated across tasks, but the relative differences do vary.
For example, in Figure 2, the gap between open and closed models further increases on tasks like self-repair or test output prediction.
Similarly, Claude-3-Opus and Mistral-L perform considerably better on code execution and test output prediction compared to code generation with Claude-3-Opus surpassing GPT-4 on the test output prediction. This highlights the importance of a holistic evaluation.
- 3.
HumanEval Overfitting.
Upon comparing LiveCodeBench with HumanEval, we find that models cluster into two groups, ones that perform well on both benchmarks and others that perform well on HumanEval but not on LiveCodeBench (see Figure 5).
The latter group primarily comprises fine-tuned open-access models while the former group comprises base models and closed models.
This indicates that these models might be overfitting to HumanEval.
- 4.
Model Comparisons (Figure 4)
(a)
Among the open access base models, we find that L3-Base and DeepSeek-Base models are the strongest, followed by StarCoder2-Base and CodeLLaMa-Base.
(b)
Closed API models such as GPTs, Claude, and Gemini generally outperform open models (Figure 2). The open models that close the gap are L3-Ins-70B, Mixtral, and DS-Ins-33B are instruction-tuned variants of large base models ($>30$B parameters).
(c)
Existing benchmarks are insufficient at highlighting the gap between GPT-4 and other models.
Particularly, smaller models achieve similar or often even better performance compared to GPT-4.
In LiveCodeBench, GPT-4 (and GPT-4-Turbo) outperforms all other models (except Claude-3-Opus) by a large margin in all scenarios.

Concurrent Work.
(Huang et al., ) also evaluate LLMs in a time-segmented manner.
However, they only focus on CodeForces problems while we combine problems across platforms
and additionally propose a holistic evaluation across multiple code-related scenarios.
(Li et al., ) propose a large dataset of competitive programming problems with additional generated tests but do not study contamination or tasks beyond generation.
Liu et al. () evaluate the code comprehension capabilities of LLMs using execution. (Singhal et al., ) also propose evaluating LLMs on tasks beyond code generation, but they consider tasks that take into account the non-functional-correctness aspects of programming.
(Guo et al., ) also evaluate DeepSeek on LeetCode problems and mention the possibility of problem contamination.

## 2 Holistic Evaluation

Code capabilities of LLMs are evaluated and compared using natural language to code generation tasks.
However, this only captures one dimension of code-related capabilities.
Indeed, real-world software engineering requires expertise in tasks beyond just generation, such as synthesizing informative test cases, debugging incorrect code, understanding existing code, and writing documentation.
These tasks are not just additional bookkeeping; they are crucial parts of the software development process and contribute to improving the quality, maintainability, and reliability of the code (Boehm, ).
This also applies to LLMs and adopting similar workflows can enable the models to perform better code generation.
For example, AlphaCodium (Ridnik et al., ) is an intricate LLM pipeline for solving competition coding problems.
By combining natural language reasoning, test case generation, code generation, and self-repair, they achieve significant improvements over a naive direct code generation baseline, showcasing the importance of these broader capabilities.
Motivated by this, we propose a more holistic evaluation of LLMs in this work using a suite of evaluation setups that capture a broader range of code-related capabilities.

Specifically, we evaluate code LLMs in four scenarios, namely code generation, self-repair, code execution, and test output prediction.
Our selection criterion was to pick settings that are useful components in code LLM workflows
and in addition, have clear and automated evaluation metrics.

Following we describe each of these scenarios in detail.

Figure: Figure 3: Overview of the different scenarios present in LiveCodeBench. Coding is multi-faceted and we propose evaluating LLMs on a suite of evaluation setups that capture various coding-related capabilities. Specifically, beyond the standard code generation setting, we consider three additional scenarios, namely self-repair, code execution, and a newly introduced test output prediction task.
Refer to caption: x1.png

Code Generation.
The code generation scenario follows the standard setup for generating code from natural language.
The model is given a problem statement, which includes a natural language description and example tests (input-output pairs), and is tasked with generating a correct solution.
The evaluation is performed based on functional correctness, using a set of unseen test cases.
We use the Pass@1 metric measured as the fraction of the problems for which the model was able to generate a program passing all tests.
Figure 3 (left) provides an example of this scenario.

Self Repair.
The self-repair scenario is based on previous works that tested the self-repair capabilities of LLMs (Olausson et al., ; Shinn et al., ; Chen et al., ).
Here, the model is given a problem statement from which it generates a candidate program (similar to the single-step code generation scenario above).
However, in case of a mistake, the model is additionally provided with error feedback (either the exception message or a failing test case in case of incorrect code generation) and is tasked with generating the fixed solution.
Similar to the code generation scenario, the evaluation is performed via functional correctness on the final program, i.e. either the single-step correct generation or the attempted repair.
We use the Pass@1 metric to measure the combined performance after the repair step.
Figure 3 (mid-left) provides an example of this scenario.

Code Execution.
The code execution scenario is based on the output prediction setup used in CRUXEval (Gu et al., ).
The model is provided a program snippet consisting of a function (f) along with a test input to the program and is tasked with predicting the output of the program on the input test case.
The evaluation is performed via an execution-based correctness metric where the model generation is considered correct if assert f(input) == generated_output passes.
Figure 3 (right) provides an example of the code execution scenario.

Test Case Output Prediction.
Finally, we introduce a new task that is designed to study natural language reasoning and test generation.
In this task, the model is given the problem statement along with a test case input, and it is tasked with generating the expected output for that input.
This task follows a setup similar to the one used in CodeT (Chen et al., ), where tests are generated solely from problem statements, without the need for the function’s implementation.
A key difference is that we provide a fixed set of test inputs for each problem in our dataset, and the models are then prompted to only predict the expected output for those specific inputs.
This approach allows for a straightforward evaluation of the test generation capabilities by avoiding test input prediction, a hard-to-evaluate task.
Figure 3 (mid-right) provides an example of this scenario.

Finally, we would like to point out that LiveCodeBench also offers an extensible framework to add new scenarios in the future.
So other relevant settings like input generation, program summarization, optimization, etc. can be integrated with our setup.

## 3 Benchmark Curation

We curate our problems from three coding competition websites: LeetCode, AtCoder, and CodeForces.
These websites periodically host contests containing problems that assess the coding and problem-solving skills of participants.
The problems consist of a natural language problem statement along with example input-output examples, and the goal is to write a program that passes a set of hidden tests.
Further, thousands of participants participate, solving these problems thus ensuring that the problems are vetted for clarity and correctness.

### 3.1 Data Collection

We have written HTML scrapers for each of the above websites to collect problems and the corresponding metadata.
To ensure quality and consistency, we parse mathematical formulas and exclude problems with images.
We also exclude problems that are not suitable for grading by input-output examples, such as those that accept multiple correct answers or require the construction of data structures.
Besides parsing the problem descriptions, we also collect associated ground truth solutions and test cases whenever directly available.
Thus for each problem, we collect tuples of natural language problem statement $P$,
test cases $T$, and ground truth solution $S$.
Finally, we associate the contest date $D$ to mark the release date of each problem and
use the collected attributes to construct problems for our four scenarios (detailed in Section 3.3 ahead).

Scrolling through time.
As noted, we associate the contest date $D$ for each problem.
The release date allows us the measure the performance of LLMs over different time windows by filtering problems
based on whether the problem release date falls within a time window (referred to as “scrolling” through time).
This is crucial for evaluating and comparing models trained at different times.
Specifically, for a new model and the corresponding cutoff date (normalized to the release date if the
training cutoff date is not published),
we can measure the performance of the model on benchmark problems released after the cutoff date.
We have developed a UI that allows comparing models on problems released during different time windows (shown in Figure 9).

Test collection.
Tests are crucial for assessing the correctness of the generated outputs and are used in all four scenarios.
We collect tests available on platform websites whenever possible and use them for the benchmark.
Otherwise, following Liu et al. (), we use a LLM (here GPT-4-Turbo) to generate tests for the problems.
A key difference between our test generation approach is that instead of generating inputs directly using the LLM, we construct generators that sample inputs based on the problem specifications using in context learning.
Details and examples of such input generators can be found in Section A.2.
Finally, we collect a small fraction of failing tests from the platform for the more recent problems allowing more directed adversarial test collection.

Problem difficulty.
Competition programming has remained a challenge for LLMs, with GPT-4 achieving an average CodeForces rating (ELO) of 392, placing it in the bottom 5 percentile  (OpenAI, ).
This makes it difficult to compare LLMs, as the variation in performance across models is low.
In LiveCodeBench, we collect problems of diverse difficulties as labeled in competition platforms, excluding problems that are rated above a certain threshold that are likely too difficult for even the best models(^1^11From our early explorations, we find CodeForces problems being considerably more difficult than AtCoder and LeetCode problems and thus focus primarily on the latter platforms.).
Further, we use these ratings to classify problems as Easy, Medium, and Hard for more granular model comparisons.

**Table 1: The statistics of problems collected in LiveCodeBench (LCB). We present the number of problems, their difficulty distributions and the average number of tests per problem. We present the results on the following subsets of LiveCodeBench (used throughout this manuscript) - (a) problems in the May’23-May’24 and Sep’23-May’24 time windows, (b) problems sourced from the three platforms, and (c) problems in the LCB-Easy, LCB-Medium, and LCB-Hard subsets.**
| Platform | Total Count | #Easy | #Medium | #Hard | Average Tests |
| --- | --- | --- | --- | --- | --- |
| LCB (May-end) | 511 | 182 | 206 | 123 | 17.0 |
| LCB (Sep-end) | 349 | 125 | 136 | 88 | 18.0 |
| AtCoder | 267 | 99 | 91 | 77 | 15.6 |
| LeetCode | 235 | 79 | 113 | 43 | 19.0 |
| CodeForces | 9 | 4 | 2 | 3 | 11.1 |
| LCB-Easy | 182 | 182 | 0 | 0 | 16.1 |
| LCB-Medium | 206 | 0 | 206 | 0 | 17.4 |
| LCB-Hard | 123 | 0 | 0 | 123 | 18.0 |

### 3.2 Platform Specific Curation

We describe the curation process for each platform.

LeetCode.
We collect problems from all weekly and biweekly contests on LeetCode that have taken place after April’23.
For each problem, we collect the problems, public tests, and user solutions.
The platform also provides a difficulty label for each problem which we use to tag the problems as Easy, Medium, and Hard.
Since LeetCode provides a starter code for each problem, we also collect it and provide it to the LLM in the STDIN format.
Since the hidden tests are not directly available, we use our generator-based test input generation approach (Section A.2) and also collect the auto grader failing tests for some of the recent problems.

AtCoder.
We collect problems from the abc (beginner round) contests on AtCoder that have taken place after April’23.
We deliberately avoid the more challenging arc and agc contests which are designed for more advanced Olympiad participants.
The problems are assigned numeric difficulty ratings, and we exclude abc problems with a rating of more than $500$.
We also use these numeric ratings to tag the problems as Easy, Medium, and Hard.
Specifically, we use the rating brackets $[0-200)$, $[200-400)$, and $[400-500]$ to perform the classification.
AtCoder provides public and hidden tests for each problem which we directly use in the benchmark.

CodeForces.
We have collected problems from the Division 3 and Division 4 contests on CodeForces.
Notably, we find that even with this filter, the problems are harder than the other two platforms.
CodeForces also provides difficulty ratings for the problems which we use to tag the problems as Easy, Medium, and Hard using the rating brackets $\{800\}$, $(800-1000]$, and $(1000-1300]$ respectively.
Due to the higher difficulty, we only consider a small fraction of problems from CodeForces and semi-automatically construct test case generators, as they do not provide complete tests on the platform (long tests are truncated).

Table 1 provides various statistics about the problems that we have collected for LiveCodeBench.

### 3.3 Scenario-specific benchmark construction

Code Generation and Self-Repair.
We use the natural language problem statement as the problem statement for these scenarios.
For LeetCode, as noted above, an additional starter code is provided for the functional input format.
For AtCoder and CodeForces problems, we use the standard input format (similar to  Hendrycks et al. ()).
The collected or generated tests are then used to evaluate the correctness of the generated programs.
Our final dataset consists of $511$ problem instances across the three platforms.

Code Execution.
We draw inspiration from the benchmark creation procedure used in  Gu et al. ().
First, we collect a large pool of $\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}2000$ correct, human-submitted solutions from the LeetCode subset.
However, many of these programs have multiple nested loops, complex numerical computations, and a large number of execution steps.
Therefore, we apply compile-time and run-time filters to ensure samples are reasonable, and we double-check this with a manual inspection.
More details on the filtering criteria and statistics of the dataset can be found in Appendix A.3.
Our final dataset consists of $479$ samples from $85$ problems.

Test Case Output Prediction.
We use the natural language problem statement from the LeetCode platform and the example test inputs to construct our test case output prediction dataset.
Since the example test inputs in the problems are reasonable test cases for humans to reason about and understand the problems, they also serve as ideal test inputs for LLMs to process.
Our final dataset consists of $442$ problem instances from a total of $181$ LeetCode problems.

## 4 Experiment Setup

We describe the experimental setup in this section.
First, we provide the common setup across the scenarios, followed by the scenario-specific setups in Section 4.1.

Models.
We evaluate $52$ models across various sizes, ranging from $1.3$B to $70$B, including base models, instruction models, and both open and closed models.
Our experiments include models from different classes, such as GPTs (GPT-3.5-turbo, GPT-4, GPT-4-Turbo,GPT-4-O), Claudes (Claude-Ins-1, Claude-2, Claude-3s), Geminis(Gemini-Pro, Gemini-Flash), Mistral among closed-access and LLaMa-3s(L3-Base-{7, 70}B, L3-Ins-{7, 70}B), DeepSeeks (DS-Base-{1.3, 6.7, 33}B, DS-Ins-{1.3, 6.7, 33}B), CodeLLaMas (CL-Ins-{7, 13, 34}B, CL-Base-{7, 13, 34}B), StarCoder2 (SC2-Base-{3,7,15}B), CodeQwen among open.
Additionally, we also include fine-tuned models Phind-34B from CL-Base-34B, and MagiCoders (MC-{6.7, 7}B) from CL-Base-7B and DS-Base-6.7B.
See Appendix C.1 for a complete list of models and estimated cutoff dates.

Evaluation Metrics.
We use the Pass@1 (Kulal et al., ; Chen et al., ) metric for our evaluations.
Specifically, we generate $10$ candidate answers for each problem either using API or using vLLM (Kwon et al., ).
We use nucleus sampling with temperature $0.2$ and top_p $0.95$ and calculate the fraction of programs or answers that are correct.
For the code generation and self-repair scenarios, we use tests to verify the correctness of the programs. For these scenarios, programs must pass all tests to be considered correct.
For the code execution scenario, we use an execution-based correctness metric between the generated output and the ground truth output. For the test output prediction scenario, we parse the generated response to extract the answer and use equivalence checks for grading as specified in Section 2.

### 4.1 Scenario-specific setup

The setup for each scenario is presented below.
Note that the base models are only used in the code generation scenario since they do not easily follow the format for the other scenarios.

Code Generation.
For the instruction-tuned models, we use a zero-shot prompt and follow the approach of Hendrycks et al. () by adding appropriate instructions to generate solutions in either functional or stdin format.
For the base models, we use a constant one-shot example, with a separate example provided for problems that accept stdin input and for problems that accept functional output.
Section LABEL:appendix:codegen-prompts shows the high-level zero-shot prompt used.

Self Repair.
Similar to prior work Olausson et al. (), we use the programs generated during the code generation scenario
along with the corresponding error feedback to build the zero-shot prompt for the self-repair scenario.
The type of error feedback includes syntax errors, runtime errors, wrong answers, and time-limit errors, as applicable.
Section LABEL:appendix:repair-prompts provides the pseudo-code for computing the error feedback and the corresponding prompt.

Code Execution.
We use few-shot prompts for the code execution scenario, both with and without chain-of-thought prompting (COT).
Particularly, we use a 2-shot prompt without COT and a 1-shot prompt with COT with manually detailed steps.
The prompts are detailed in Section LABEL:appendix:code-execution-prompts.

Test Output Prediction.
We use a zero-shot prompt that queries the model to complete assertions, given the problem, function signature, and test input.
We provide the prompt in Section LABEL:appendix:test-pred-prompts.

## 5 Results

We first describe how LiveCodeBench helps detect and avoid benchmark contamination in Section 5.1.
Next, we present the findings from our evaluations on LiveCodeBench in Section 5.2.

### 5.1 Avoiding Contamination

A distinguishing aspect of our benchmark is the ability to evaluate models on problems released over different time windows.
This allows us to measure the model performance on problems released after the cutoff date, thereby giving a performance estimate on unseen problems.

Contamination in DeepSeek and GPT-4-O.
LiveCodeBench comprises problems released since May $2023$.
However, DeepSeek models were released in Sep $2023$ and might have already been trained on some of the problems in our benchmark.
Similarly, OpenAI notes GPT-4-O cutoff date in November.
We can measure the performance of the models on the benchmark using problems released after the cutoff date, thereby estimating the performance of the model on previously unseen problems.
Figure 1 shows the performance of these models on LiveCodeBench code generation and test output prediction scenario on LeetCode problems released in different months from May $2023$ and Feb $2024$.
We notice a stark drop in the performance of DS-Ins-33B model after Aug. $2023$ (right before its release date), which suggests
that the earlier problems might indeed be contaminated.
This trend is consistent across other LiveCodeBench scenarios like repair and code execution, as depicted in Figure LABEL:fig:contamination_all_tasks_line.
Concurrently, Guo et al. () (Section 4.1, last paragraph) also acknowledge the possibility of LeetCode contamination, noting that “models achieved higher scores in the LeetCode Contest held in July and August.
Similarly, performance of the GPT-4-O model drops on problems released since November (its official cutoff date).

Interestingly, we find that this drop in performance primarily occurs for the LeetCode problems only and that the model performance is relatively smooth across the months for problems from other platforms. Figure LABEL:fig:atcoder_contamination shows a relatively stable performance for all models on AtCoder problems released over different periods, with the possible exception of May and June.

Performances of other models.
We study performance variations in other models released recently.
Particularly, GPT-4-Turbo, Gemini-Pro, Mistral-L, and Claude-3s models were released in November $2023$, December $2023$, February $2024$, and March $2024$ respectively.
Note that GPT-4-Turbo ($1106$-preview variant) and Claude-3s have cutoff dates April $2023$ and August $2023$ respectively.
Irrespective of the release or cutoff dates, we do not find any drastic performance variations across the months, as shown in Figure LABEL:app:fig:contamination_all_models, particularly compared to the DeepSeek models.
Interestingly, we find that even the DS-Base-33B model also suffers from contamination dropping from Pass@1 $\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}60$ in May problems to Pass@1 $\mathrel{\mathchoice{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}{\vbox{\hbox{$\scriptscriptstyle\sim$}}}}0$ in September LeetCode problems.
This also suggests the likely inclusion of competition problems in the pretraining of the DeepSeek models, thereby affecting all instruction models trained from it.
Finally, Codestral achieves Pass@1 $36.5$ on problems released between May’23 and Jan’24 and Pass@1 $28.3$ on problems since Feb’24.

Figure: ((a))
Refer to caption: figures/main_results/codegen_performance.png

### 5.2 Performance and Model Comparisons

We evaluate $34$ instruction-tuned models (and $18$ base models used in the code generation scenario) on LiveCodeBench.
These models range from closed access to open access with their various fine-tuned variants.
To overcome contamination issues in DeepSeek models, we only consider problems released since Sep $2023$ for all evaluations below.
Figure 4 shows the performance of a subset of models across the four scenarios.
We highlight our key findings below.

Holistic Evaluations.
We have evaluated the models across the four scenarios currently available in LiveCodeBench.
Figure 2 displays the performance of models on all scenarios along the axes of the polar chart.
First, we observe that the relative order of models remains mostly consistent across the scenarios.
This is also supported by high correlations between Pass@1 metric across the scenarios –
over $0.88$ across all pairs as shown in Figure LABEL:app:fig:correlation_matrix.
Interestingly, the correlations are larger for related tasks, $0.98$ for generation and self-repair, and $0.96$ for test output prediction and code execution.
This correlation drops to $0.89$ for generation and execution scenarios.

However, despite the strong correlation, the relative differences in performance do vary across the scenarios.
For example, GPT-4-Turbo further gains performance gap over GPT-4 in the self-repair scenario after already leading in the code generation scenario.
Similarly, Claude-3-Opus and Mistral-L perform well in tasks involving COT, particularly in the code execution and test output prediction scenarios.
For instance, Claude-3-Opus even outperforms GPT-4-Turbo in the test output prediction scenario.
Similarly, Mistral-L outperforms Claude-3-Sonnet in both scenarios after trailing behind in code generation and repair scenarios.
These differences highlight the need for holistic evaluations beyond measuring code generation capabilities.

Figure: Figure 5: Scatter plot comparing Pass@1 of models on HumanEval+ versus Pass@1 on the easy subset of LiveCodeBench code generation scenario. Star markers denote the closed-access models while other markers denote different open model families. We find that the models are separated into two groups – the green-shaded region where performances on the two datasets are aligned and the red-shaded region where models perform well on HumanEval+ but perform poorly on LiveCodeBench. This indicates potential overfitting on HumanEval+ and primarily occurs in the fine-tuned variants of open-access models. For example, DS-Ins-1.3B which achieves Pass@1 of $60$ and $26$ on HumanEval+ and LCB-Easy subset. Thus, while it ranks above CMD-R+ on HumanEval+, it performs significantly worse on the LCB. Similarly, DS-Ins-6.7B and CodeQwen outperform Claude-3-Sonnet on HumanEval+ but are $>20$ points behind on LCB-Easy.
Refer to caption: figures/main_results/lcb_vs_he.png

Comparison to HumanEval.
Next, we compare how code generation performance metrics translate between LiveCodeBench and HumanEval,
the primary benchmark used for evaluating coding capabilities.
Note that we use HumanEval+ version of HumanEval problems as it is more reliable with more exhaustive test cases.
Figure 5 shows a scatter plot of Pass@1 on HumanEval+ versus LCB-Easy code generation scenario.
We find only a moderate correlation of $0.72$, with much larger performance variations on LCB-Easy.

Additionally, we observe that the models cluster into two groups, shaded in red and green.
The models in the green-shaded region lie close to the $x=y$ line, indicating that they perform similarly on both benchmarks.
On the other hand, the models shaded in red lie in the top-left region of the graph,
indicating that they perform well only on HumanEval+ but not as well on LiveCodeBench.
Interestingly, the green-shaded cluster contains base models or closed-access models,
while the red-shaded cluster primarily comprises fine-tuned variants of open-access models.
The well-separated clusters suggest that many models that perform well on HumanEval might be overfitting on the benchmark, and their performances do not translate well to problems from other domains or difficulty levels like those present in LiveCodeBench.

Indeed, HumanEval is an easier benchmark with small and isolated programming problems and thus easier to overfit on.
In contrast, LiveCodeBench problems are sourced from reputable coding platforms offering more challenging problems with higher diversity and difficulty levels.
This potential overfitting is particularly exemplified by DS-Ins-1.3B which achieves $59.8$% Pass@1 on HumanEval+ but only $26.3$% on LCB-Easy.
Thus, while it boasts better performance compared with Gemini-Pro and Claude-Ins-1 on HumanEval+, it performs considerably worse on LCB-Easy.
Similarly, CodeQwen, DS-Ins-6.7B, and MC-6.7B perform better than Mistral-L, Claude-2, and Claude-3-Sonnet on HumanEval+ but are considerably worse on LCB-Easy.

Highlighting the gap between SoTA and open models.
One distinct observation from our evaluations is the large gap between SoTA models and open models across all scenarios.
Particularly, GPT-4-Turbo, GPT-4, Gemini-Pro-1.5 and Claude-3-Opus lead across the benchmarks with wide performance margins over other models.
This distinguishes LiveCodeBench from prior benchmarks (like HumanEval) where various open models have achieved similar or better performance.
For example, DS-Ins-33B is merely $4.3$ point behind GPT-4-Turbo on HumanEval+ but
$16.2$ points ($69$%) on LCB code generation scenario.
This gap either holds or sometimes even amplifies across other scenarios.
For instance, consider test output prediction and code execution (with COT) where GPT-4-Turbo leads the DS-Ins-33B model by $96\%$ and $134\%$ respectively!

We qualitatively analyze code samples generated by the leading model, GPT-4-Turbo, and find that it generates more readable code.
Specifically, the code consists of more inline natural language comments that reason or plan before producing the code.
We verify this quantitatively and find GPT-4-Turbo generated uses $19.5\times$ more comment tokens compared to GPT-4.

Comparing Base Models.
We use four families of base models – L3-Base, DeepSeek, CodeLLaMa, and StarCoder2 and compare them on the code generation scenario.
A one-shot prompt is used for all models to avoid any formatting and answer extraction issues.
We find L3-Base and DS models are significantly better than both CodeLLaMa and StarCoder2 base models with
a DS-Base-6.7B model even outperforming both CL-Base-34B and SC2-Base-15B models.
Next, we observe that SC2-Base-15B also outperforms the CL-Base-34B model (similiar to findings in  Lozhkov et al. ()).
Note that some LiveCodeBench specific differences can potentially be attributed to data curation approaches.
For instance, StarCoder2 models (and potentially DeepSeeks as discussed in Section 5.1) use competition problems in the pre-training corpus.

Role of Post Training.
We find that post-training improves performance on both HumanEval+ and LiveCodeBench for the code generation scenario.
Particularly, L3-Ins-70B, DS-Ins-33B and Phind-34B achieve $28.3$, $23.6$, and $21$ Pass@1 on LCB
improving over their base models by $8.2$, $7.3$ and $9.5$ points respectively.
Similar gains are observed in previous benchmarks (like HumanEval+) as well.
This highlights the importance of good post-training datasets for building strong LLMs.

At the same time, we note that the base models have aligned performances on LCB code generation and HumanEval+ benchmarks and lie within or close to the green shaded region in Figure 5.
However, the fine-tuned open models exhibit a larger performance gap, with much better performances on HumanEval+.
On the other hand, the closed-access models are still aligned across both benchmarks.
This suggests that the fine-tuning data for open models might not be as diverse as that for closed models, leading to a lack of generalization to different kinds of problems.

Comparing open-access instruction-tuned models.
Here, we compare various fine-tuned variants of the L3-Base, DeepSeek and CodeLLaMa base models across different model sizes.
We find that fine-tuned L3-Base and DeepSeek models lead in performance, followed by Phind-34B and CodeLLaMa models across most scenarios.
Broadly, we find that model performances correlate with model sizes. For example, Phind-34B model outperforms the $6.7$B models across all scenarios.

Comparing Closed Models.
We evaluate a range of closed (API access) models ranging from different model families
like GPTs, Claudes, Gemini, and Mistral.
We find the GPT-4-Turbo and Claude-3-Opus rank at the top across all scenarios followed by Mistral-L and Claude-3-Sonnet models.
Finally, Gemini-Pro and GPT-3.5-turbo lie on the lower end of the models.
The relative differences between the models vary across the scenarios.
For example, GPT-4-Turbo demonstrates remarkable improvement from self-repair ($24.5$% to $36.9$% on the LCB-Medium problems) while Gemini-Pro only improves from $8.5$% to $9.4$%.
Similarly, as identified above, Claude-3-Opus and Mistral-L perform considerably better on test output prediction and code execution scenarios.

Open-Access vs Closed-Access Models.
In general, closed (API) access model families generally outperform the open access models.
The gap is only closed by three models, namely L3-Ins-70B, Mixtral, and DS-Ins-33B
which reach the performance levels of the closed models.
For instance, in the code generation scenario (Figure 2 right),
these models reach close to or even outperform closed access models like Gemini-Pro, GPT-3.5-turbo, and Claude-3-Sonnet.
The performances vary across scenarios with the closed-access models performing better in test output prediction and code execution scenarios.
Overall, our findings confirm that a combination of strong base models and high-quality post-training datasets is a viable recipe for good code LLMs.

## 6 Related Work

### 6.1 Code Generation

Language Models for Code Generation.
Starting with Codex (Chen et al., ), there are over a dozen code LLMs. These include CodeT5 (Wang et al., , ), CodeGen (Nijkamp et al., ), SantaCoder (Allal et al., ), StarCoder (Li et al., ), AlphaCode (Li et al., ), InCoder (Fried et al., ), and CodeGeeX (Zheng et al., ).
As of May 2024, L3-Base and DeepSeek (Bi et al., ), StarCoder Lozhkov et al. (); Li et al. () and CodeLLaMa (Roziere et al., ) are the most popular open models.
Many downstream models resulted from fine-tuning them on synthetically generated data, such as WizardCoder (Luo et al., ), MagiCoders (Wei et al., ), and Phind-34B.

Code Generation Benchmarks.
Many benchmarks have been proposed to compare and evaluate these models.
These primarily focus on natural language to Python code generation:
HumanEval (Chen et al., ),
HumanEval+ (Liu et al., ),
APPS (Hendrycks et al., ),
Code-Contests (Li et al., ),
MBPP (Austin et al., ),
L2CEval (Ni et al., ).
Their variants have been proposed to cover more languages, (Wang et al., ; Zheng et al., ; Cassano et al., ; Athiwaratkun et al., ).
Many benchmarks have focused on code generation in APIs. Benchmarks like DS-1000 (Lai et al., ), ARCADE (Yin et al., ), NumpyEval (Zhang et al., ), and PandasEval (Jain et al., ) focus on data science APIs.
Other benchmarks measure using broader APIs or
general software engineering tasks, such as JuICe (Agashe et al., ), APIBench (Patil et al., ), RepoBench (Liu et al., ), ODEX (Wang et al., ), SWE-Bench (Jimenez et al., ), GoogleCodeRepo (Shrivastava et al., ), RepoEval (Zhang et al., ), and Cocomic-Data (Ding et al., ).

A few benchmarks specifically measure competitive programming, such as
APPS (Hendrycks et al., ),
CodeContests (Li et al., ),
CodeScope (Yan et al., ),
xCodeEval (Khan et al., ),
and LeetCode-Hard (Shinn et al., ), and TACO (Li et al., ).
Methods such as AlphaCode (Li et al., ), AlphaCode 2(Gemini Team et al., ), ALGO (Zhang et al., ), Parsel (Zelikman et al., ), code cleaning (Jain et al., ), code explainations (Li et al., ), analogical reasoning (Yasunaga et al., ), and AlphaCodium (Ridnik et al., ) have been pushing the boundaries of what is possible with LLMs in this domain. The biggest differentiating factor between LiveCodeBench and these benchmarks is that our benchmark is continuously updated, problem curation with balanced difficulty, higher tests and problem quality, and contains more scenarios such as code repair, code execution, and test output prediction capturing more facets for building agentic coding systems.

### 6.2 Holistic Tasks

LiveCodeBench considers self-repair, test output prediction, and code execution as additional scenarios. Below we note pertinent related work for these domains.

Code Repair.
(Chen et al., ; Olausson et al., ; Madaan et al., ; Peng et al., ; Zhang et al., ) have investigated self-repair for existing code LLM benchmarks. Particularly, these methods use error feedback for models to improve inspiring our code repair scenario.

Code Execution.
Code execution was first studied in Austin et al. (); Nye et al. ()
LiveCodeBench’s execution scenario is particularly inspired by CRUXEval (Gu et al., ), a recent benchmark measuring the reasoning and execution abilities of code LLMs.
We differ from CRUXEval in that our benchmark is live, and our functions are more complex and human-produced (unlike Code Llama generations in CRUXEval).

Test Generation.
Test generation using LLMs has been explored in (Yuan et al., ; Schäfer et al., ; Tufano et al., ; Watson et al., ).
Furthermore, Chen et al. () demonstrated that LLMs can assist in generating test case inputs/outputs for competitive programming problems, thereby improving the accuracy of the generated code, thus inspiring our test generation scenario.
However, LiveCodeBench’s test generation scenario is unique in that it decouples the test inputs and outputs allowing more proper evaluations.

Finally, some works have additionally studied other tasks and scenarios like type prediction (Mir et al., ; Wei et al., ; Malik et al., ), code summarization (LeClair et al., ; Iyer et al., ; Barone and Sennrich, ; Hasan et al., ; Alon et al., ), code security (Liguori et al., ; Pearce et al., ; Tony et al., ), etc.

### 6.3 Contamination

Data contamination and test-case leakage have received considerable attention  Oren et al. (); Golchin and Surdeanu (); Weller et al. (); Roberts et al. () as LLMs might be getting trained on benchmarks.
Sainz et al. () demonstrated contamination by simply prompting the model to highlight its contamination.
Some detection methods have also been built to avoid these cases (Shi et al., ; Zhou et al., ). For code, Riddell et al. () use edit distance and AST-based semantic-similarity to detect contamination.

## 7 Limitations

Benchmark Size.
LiveCodeBench code generation scenario currently hosts over $400$ instances from problems released between May and February.
To account for contamination in DeepSeek, we only perform evaluations on problems released after the model cutoff date.
This leads to only $349$ problems used in our final evaluations which might add noise due to problem set samples.
We currently estimate $1-1.5\%$ performance variations in LiveCodeBench code generation due to this issue (measured by bootstrapping $349$ sized problem sets from the $511$ sized dataset).
Other scenarios, i.e. self-repair, code execution, and test output prediction comprise $349$, $188$, and $254$ problems would have similar performance variations.
We thus recommend exercising proper judgement when comparing models with small performance differences.
Note that HumanEval has $164$ problems and would also struggle with similar issues.

This issue is also exacerbated for newer models, with more recent cutoff dates, as they might only have access to a smaller evaluation set.
We propose two solutions addressing this issue as we evolve LiveCodeBench.
First, we will use other competition platforms for problem collection, allowing larger number of recent problems to be added to the benchmark.
In addition, we also hope supplement this with an unreleased private test set constructed specifically for model evaluation.
These problems will use a similar flavor to current problems and will be used when models are submitted for evaluation to the LiveCodeBench platform.
This would reduce the reliance on public accessible problems and provide a more robust evaluation of the models while providing community public access to similar problems, similar to strategies employed by popular platforms like Kaggle.

Focus on Python.
LiveCodeBench currently only focuses on Python which might not provide enough signal about model capabilities in other languages.
However, since we collected problem statements and serialized tests, adding new programming languages would be straightforward once appropriate evaluation engines are used.

Robustness to Prompts.
Recent works have identified huge performance variances that can be caused due to insufficient prompt.
Here, we either do not tune prompts across models or make minor adjustments based on the system prompts and delimiter tokens.
This can lead to performance variance in our results.
Our findings and model comparison orders generalize across LiveCodeBench scenarios
and mostly match the performance trends observed on HumanEval making this a less prominient issue.

This issue can be particularly observed open models on the code execution scenario with COT prompting.
Interestingly, often the open models perform even worse in comparsion to the direct code execution baseline.
Note that we used same prompts for the closed models all of which show noticable improvement from COT.
While the used prompts might be sub-optimal, this highlights how open-models perform worse against the closed models at performing chain-of-thought.

Problem Domain.
Programming is a vast domain and occurs in various forms such as programming puzzles, competition programming, and real-world software development.
Different domains might have individual requirements, constraints, challenges, and difficulty levels.
LiveCodeBench currently focuses on competition problems sourced from three platforms.
This might not be representative of the “most general” notion of LLM programming capabilities.
Particularly, real-world usage of LLMs is drawn upon open-ended and unconstrained problems rasied by users.
We therefore recommend using LiveCodeBench as a starting point for evaluating LLMs and
further using domain-specific evaluations to measure and compare LLMs in specific settings as required.

## 8 Conclusion

In this work, we propose LiveCodeBench, a new benchmark for evaluating LLMs for code.
Our benchmark mitigates contamination issues in existing benchmarks by introducing live evaluations and emphasizing scenarios beyond code generation to account for the broader coding abilities of LLMs.
LiveCodeBench is an extensible framework, that will keep on updating with new problems, scenarios, and models.
Our evaluations reveal novel findings such as contamination detection and potential overfitting on HumanEval.
We hope LiveCodeBench with serve to advance understanding of current code LLMs and also guide future research in this area through our findings.

## Acknowledgements

This work was supported in part by NSF grants CCF:1900968, CCF:1908870 and by SKY Lab industrial sponsors and affiliates Astronomer, Google, IBM, Intel, Lacework, Microsoft, Mohamed Bin Zayed University of Artificial Intelligence, Nexla, Samsung SDS, Uber, and VMware.
A. Gu is supported by the NSF Graduate Research Fellowship under Grant No. 2141064.
A. Solar-Lezama is supported by the NSF and Intel Corporation through NSF Grant CCF:2217064.
Any opinions, findings, conclusions, or recommendations in this paper are solely those of the authors and do not necessarily reflect the position of the sponsors.

Finally, we thank Manish Shetty, Wei-Lin Chiang, Jierui Li, Horace He, Federico Cassano, Pengcheng Yin, and Aman Madaan for helpful feedback at various stages of the work.

## Appendix A Dataset

### A.1 License

Similar to Hendrycks et al. (), we scrape only the problem statements, ground-truth solutions, and test cases from competition websites – LeetCode, AtCoder, and CodeForces.
Further, we only scrape publicly visible portions of websites, avoiding any data collection that might be pay-walled or require login or interaction with the website.
Following, Hendrycks et al. () we abide by Fair Use §107: “the fair
use of a copyrighted work, including such use by … scholarship, or research, is not an infringement
of copyright”, where fair use is determined by “the purpose and character of the use, including
whether such use is of a commercial nature or is for nonprofit educational purposes”, “the amount
and substantiality of the portion used in relation to the copyrighted work as a whole”, and “the effect
of the use upon the potential market for or value of the copyrighted work.”
Finally, we use the collected problems for academic purposes only and in addition, do not train on the collected problems.

### A.2 Generator Based Test Generation

We use GPT-4-Turbo to construct input generators.
The following prompts (Figures 6 and 7) provide one-shot prompt templates used for synthesizing random and adversarial input generators.
These generators define a function returns the arguments sampled in some distribution.
These generators are then executed to construct inputs which validated on the collected correct programs.
We use separate generators for random and adversarial setting since often times programming problems have corner cases which might not be captured by randomly sampling over the inputs. We build $2$ random input generators, 4 adversarial input generators and check if the sampled inputs work for the correct programs. Finally, the number of collected inputs is thresholded to $100$ for efficient grading (using random selection). We find that our generators can already function well but future work can study the design space of constructing such generators.

Note that for CodeForces, we construct the generators in semi-autonomous manner since only $9$ problems were used.

Figure: Figure 6: Random Input Generator Prompt

Figure: Figure 7: Adversarial Input Generator Prompt

### A.3 Code Execution

The code execution split of LiveCodeBench consists of 479 samples from 85 distinct problems. To encourage diversity in our benchmark while keeping our benchmark small and usable, we place a limit of six samples for each given problem. These sample programs and corresponding test cases are chosen uniformly at random from all those passing the filter.

Filtering Criteria: The specific filtering criteria are as follows:

- •
Compile time: length of code is between 100 and 500 characters, no syntax errors, all necessary imports are included
- •
Runtime: no floating point operations, true division, exp, other integer operations must have at least one argument $\leq 3$, string and list operations must have at least one argument with length $\leq 3$, finish running in 2 seconds, “reasonable” number of steps (roughly, under 1000 Python bytecode operations).

We give two examples of two programs that are filtered out in the Listings below. Our final benchmark consists of 479 samples from 85 problems, but will increase in size due to its live nature.

Dataset Statistics: As mentioned, we filter for codes between 100 and 500 characters, as well as below 1000 steps. The statistics for programs in our dataset are shown in Fig. 8.

Figure: Figure 8: Distribution of code lengths and number of execution steps
Refer to caption: x2.png

## Appendix B UI

Figure: Figure 9: UI of LiveCodeBench showing two views – May-Jan and Sep-Jan. The contaminated models are blurred and the performance difference is visible across the two views. The scroller on the top allows selecting different periods of time highlighting the live nature of the benchmark.
Refer to caption: figures/UI/full_ui.png

## Appendix C Experimental Setup

### C.1 Models

We describe the details of models considered in our study in Table LABEL:tab:models_list.

**Table 2: Correlations across different scenarios studied in LiveCodeBench**
| Model ID | Short Name | Approximate Cutoff Date | Link |
| --- | --- | --- | --- |
| deepseek-ai/deepseek-coder-33b-instruct | DSCoder-33b-Ins | 08/30/2023 | [deepseek-coder-33b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-33b-instruct) |
| deepseek-ai/deepseek-coder-6.7b-instruct | DSCoder-6.7b-Ins | 08/30/2023 | [deepseek-coder-6.7b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-instruct) |
| deepseek-ai/deepseek-coder-1.3b-instruct | DSCoder-1.3b-Ins | 08/30/2023 | [deepseek-coder-1.3b-instruct](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-instruct) |
| codellama/CodeLlama-70b-Instruct-hf | CodeLlama-70b-Ins | 01/01/2023 | [CodeLlama-70b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf) |
| openbmb/Eurus-70b-sft | Eurus-70B-SFT (n=1) | 01/01/2023 | [Eurus-70b-sft](https://huggingface.co/openbmb/Eurus-70b-sft) |
| openbmb/Eurux-8x22b-nca | Eurux-8x22b-NCA (n=1) | 04/30/2023 | [Eurux-8x22b-nca](https://huggingface.co/openbmb/Eurux-8x22b-nca) |
| codellama/CodeLlama-34b-Instruct-hf | CodeLlama-34b-Ins | 01/01/2023 | [CodeLlama-34b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-34b-Instruct-hf) |
| codellama/CodeLlama-13b-Instruct-hf | CodeLlama-13b-Ins | 01/01/2023 | [CodeLlama-13b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-13b-Instruct-hf) |
| codellama/CodeLlama-7b-Instruct-hf | CodeLlama-7b-Ins | 01/01/2023 | [CodeLlama-7b-Instruct-hf](https://huggingface.co/codellama/CodeLlama-7b-Instruct-hf) |
| meta-llama/Meta-Llama-3-8B-Instruct | LLama3-8b-Ins | 01/01/2023 | [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) |
| meta-llama/Meta-Llama-3-70B-Instruct | LLama3-70b-Ins | 01/01/2023 | [Meta-Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) |
| Phind/Phind-CodeLlama-34B-v2 | Phind-34B-V2 | 01/01/2023 | [Phind-CodeLlama-34B-v2](https://huggingface.co/Phind/Phind-CodeLlama-34B-v2) |
| Smaug-2-72B | Smaug-2-72B | 01/01/2023 | [Smaug-2-72B](https://huggingface.co/abacusai/Smaug-2-72B) |
| Qwen-1.5-72B-Chat | Qwen-1.5-72B-Chat | 01/01/2023 | [Qwen-1.5-72B-Chat](https://huggingface.co/qwen/Qwen1.5-72B-Chat) |
| Qwen/CodeQwen1.5-7B | CodeQwen15-7B | 08/30/2023 | [CodeQwen1.5-7B](https://huggingface.co/Qwen/CodeQwen1.5-7B) |
| Qwen/CodeQwen1.5-7B-Chat | CodeQwen15-7B-Chat | 08/30/2023 | [CodeQwen1.5-7B-Chat](https://huggingface.co/Qwen/CodeQwen1.5-7B-Chat) |
| gpt-3.5-turbo-0301 | GPT-3.5-Turbo-0301 | 10/01/2021 | [gpt-3.5-turbo-0301](https://openai.com/blog/new-models-and-developer-products-announced-at-devday) |
| gpt-3.5-turbo-0125 | GPT-3.5-Turbo-0125 | 10/01/2021 | [gpt-3.5-turbo-0125](https://openai.com/blog/new-embedding-models-and-api-updates#:~:text=Other%20new%20models%20and%20lower%20pricing) |
| gpt-4-0613 | GPT-4-0613 | 10/01/2021 | [gpt-4-0613](https://openai.com/blog/new-models-and-developer-products-announced-at-devday) |
| gpt-4-1106-preview | GPT-4-Turbo-1106 | 04/30/2023 | [gpt-4-1106-preview](https://openai.com/blog/new-models-and-developer-products-announced-at-devday) |
| gpt-4-turbo-2024-04-09 | GPT-4-Turbo-2024-04-09 | 04/30/2023 | [gpt-4-turbo-2024-04-09](https://openai.com/blog/new-models-and-developer-products-announced-at-devday) |
| gpt-4o-2024-05-13 | GPT-4O-2024-05-13 | 10/30/2023 | [gpt-4o-2024-05-13](https://openai.com/index/spring-update) |
| claude-2 | Claude-2 | 12/31/2022 | [claude-2](https://www.anthropic.com/index/claude-2) |
| claude-instant-1 | Claude-Instant-1 | 12/31/2022 | [claude-instant-1](https://www.anthropic.com/index/introducing-claude) |
| claude-3-opus-20240229 | Claude-3-Opus | 04/30/2023 | [claude-3-opus-20240229](https://www.anthropic.com/claude) |
| claude-3-sonnet-20240229 | Claude-3-Sonnet | 04/30/2023 | [claude-3-sonnet-20240229](https://www.anthropic.com/claude) |
| claude-3-haiku-20240307 | Claude-3-Haiku | 04/30/2023 | [claude-3-haiku-20240307](https://www.anthropic.com/claude) |
| codestral-latest | Codestral-Latest | 01/31/2024 | [codestral-latest](https://mistral.ai/news/codestral/) |
| gemini-pro | Gemini-Pro | 04/30/2023 | [gemini-pro](https://blog.google/technology/ai/gemini-api-developers-cloud) |
| gemini-1.5-pro-latest | Gemini-Pro-1.5-May | 04/30/2023 | [gemini-1.5-pro-latest](https://blog.google/technology/ai/gemini-api-developers-cloud) |
| gemini-1.5-flash-latest | Gemini-Flash-1.5-May | 04/30/2023 | [gemini-1.5-flash-latest](https://blog.google/technology/ai/gemini-api-developers-cloud) |
| ise-uiuc/Magicoder-S-DS-6.7B | MagiCoderS-DS-6.7B | 08/30/2023 | [Magicoder-S-DS-6.7B](https://huggingface.co/ise-uiuc/Magicoder-S-DS-6.7B) |
| ise-uiuc/Magicoder-S-CL-7B | MagiCoderS-CL-7B | 01/01/2023 | [Magicoder-S-CL-7B](https://huggingface.co/ise-uiuc/Magicoder-S-CL-7B) |
| bigcode/starcoder2-3b | StarCoder2-3b | 01/01/2023 | [starcoder2-3b](https://huggingface.co/bigcode/starcoder2-3b) |
| bigcode/starcoder2-7b | StarCoder2-7b | 01/01/2023 | [starcoder2-7b](https://huggingface.co/bigcode/starcoder2-7b) |
| bigcode/starcoder2-15b | StarCoder2-15b | 01/01/2023 | [starcoder2-15b](https://huggingface.co/bigcode/starcoder2-15b) |
| codellama/CodeLlama-70b-hf | CodeLlama-70b-Base | 01/01/2023 | [CodeLlama-70b-hf](https://huggingface.co/codellama/CodeLlama-70b-hf) |
| codellama/CodeLlama-34b-hf | CodeLlama-34b-Base | 01/01/2023 | [CodeLlama-34b-hf](https://huggingface.co/codellama/CodeLlama-34b-hf) |
| codellama/CodeLlama-13b-hf | CodeLlama-13b-Base | 01/01/2023 | [CodeLlama-13b-hf](https://huggingface.co/codellama/CodeLlama-13b-hf) |
| codellama/CodeLlama-7b-hf | CodeLlama-7b-Base | 01/01/2023 | [CodeLlama-7b-hf](https://huggingface.co/codellama/CodeLlama-7b-hf) |
| deepseek-ai/deepseek-coder-33b-base | DSCoder-33b-Base | 08/30/2023 | [deepseek-coder-33b-base](https://huggingface.co/deepseek-ai/deepseek-coder-33b-base) |
| deepseek-ai/deepseek-coder-6.7b-base | DSCoder-6.7b-Base | 08/30/2023 | [deepseek-coder-6.7b-base](https://huggingface.co/deepseek-ai/deepseek-coder-6.7b-base) |
| deepseek-ai/deepseek-coder-1.3b-base | DSCoder-1.3b-Base | 08/30/2023 | [deepseek-coder-1.3b-base](https://huggingface.co/deepseek-ai/deepseek-coder-1.3b-base) |
| google/codegemma-7b | CodeGemma-7b-Base | 01/01/2023 | [codegemma-7b](https://huggingface.co/google/codegemma-7b) |
| google/codegemma-2b | CodeGemma-2b-Base | 01/01/2023 | [codegemma-2b](https://huggingface.co/google/codegemma-2b) |
| google/gemma-7b | Gemma-7b-Base | 01/01/2023 | [gemma-7b](https://huggingface.co/google/gemma-7b) |
| google/gemma-2b | Gemma-2b-Base | 01/01/2023 | [gemma-2b](https://huggingface.co/google/gemma-2b) |
| meta-llama/Meta-Llama-3-70B | LLama3-70b-Base | 01/01/2023 | [Meta-Llama-3-70B](https://huggingface.co/meta-llama/Meta-Llama-3-70B) |
| meta-llama/Meta-Llama-3-8B | LLama3-8b-Base | 01/01/2023 | [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) |
| mistral-large-latest | Mistral-Large | 01/01/2023 | [mistral-large-latest](https://mistral.ai/news/mistral-large/) |
| open-mixtral-8x22b | Mixtral-8x22B-Ins | 01/01/2023 | [open-mixtral-8x22b](https://mistral.ai/news/mixtral-8x22b/) |
| open-mixtral-8x7b | Mixtral-8x7B-Ins | 01/01/2023 | [open-mixtral-8x7b](https://mistral.ai/news/mixtral-8x7b/) |
| m-a-p/OpenCodeInterpreter-DS-33B | OC-DS-33B | 08/30/2023 | [OpenCodeInterpreter-DS-33B](https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-33B/) |
| m-a-p/OpenCodeInterpreter-DS-6.7B | OC-DS-6.7B | 08/30/2023 | [OpenCodeInterpreter-DS-6.7B](https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-6.7B/) |
| m-a-p/OpenCodeInterpreter-DS-1.3B | OC-DS-1.3B | 08/30/2023 | [OpenCodeInterpreter-DS-1.3B](https://huggingface.co/m-a-p/OpenCodeInterpreter-DS-1.3B/) |
| command-r | Command-R | 01/01/2023 | [command-r](https://docs.cohere.com/docs/models) |
| command-r+ | Command-R+ | 01/01/2023 | [command-r+](https://docs.cohere.com/docs/models) |

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 6
presentation: 4
questions: ## Discussion Questions

Have you considered addressing more intricate data contamination issues like re-phrasing within the scope of this work?

I believe that many code-specific techniques can be developed to detect re-phrasing contamination such as checking the equivalence or similarity of the test cases and canonical solutions.

I would be happy to raise my score if the authors could incorporate and address those intricate contamination issues in the revision.
soundness: 4
strengths: ## Significance

The paper addresses one of the most important issues: data contamination in the existing Code LLM evaluation process. The benchmark is highly practical and should be known and tried by the Code LLM community to encourage more rigorous evaluation of models. 


## Soundness

The paper demonstrates strong soundness through its detailed analyses of the models' performance over time, overfitting issues on HumanEval, and comparison of different types of models.


## Effectiveness

Dynamic benchmarking is indeed an effective solution to mitigate data contamination.
summary: The paper introduces LiveCodeBench, a lively-updated dynamic benchmark that is designed to evaluate large language models (LLMs) for coding tasks with minimized data contamination risk.

Aside from being contamination-free, LiveCodeBench also seeks to improve the data quality, difficulty, and task diversity of the benchmark.

The paper also conducts thorough experiments exposing the significant data contamination and overfitting issues.
weaknesses: ## Novelty

The paper presents two key contributions:

1. A comprehensive dynamic benchmark for coding takes

2. Analyses exposing the contamination issues in coding tasks

While these contributions are highly relevant and practical for the Code LLM community, it's important to note that dynamic benchmarking is a technique that has been utilized in various prior works [1,2,3,4] as early as 2021 [1]. Additionally, the issue of data contamination has been systematically examined in earlier studies [5,6], limiting the novel insights that the paper can offer to the broader ICRL community.

To enhance the paper's novelty, it may be beneficial to develop better and more efficient dynamic benchmarking techniques, or introduce new methods to alleviate more intricate contamination issues such as re-phrasing [6].




[1] [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://arxiv.org/abs/2106.06052)

[2] [Competition-Level Problems are Effective LLM Evaluators](https://arxiv.org/abs/2312.02143)

[3] [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://arxiv.org/abs/2312.12343)

[4] [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://arxiv.org/abs/2312.14890)

[5] [To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination](https://openreview.net/forum?id=m2NVG4Htxs)

[6] [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://arxiv.org/abs/2311.04850)

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: It seems that the drop in performance occurs only for the LeetCode problems and is smooth for other platforms like AtCoder. Do you have any explanation for this interesting phenomenon?
soundness: 3
strengths: - The paper proposes a coding benchmark with live updates, ensuring fair model evaluation by testing only on new problems after each model’s cutoff date. The time-segmented analysis also reveals a significant drop in performance of some models after release date, indicating notable contamination issues.
- The paper is well-written and organized, with a thorough appendix including discussion on legal compliance and benchmark creation details.
summary: Prior LLM coding benchmarks suffer from contamination and saturation issues. This paper introduces LiveCodeBench, a new benchmark that collects new problems from public programming contests (LeetCode, AtCoder, and Codeforces). Besides code completion, the benchmark also tests the self-repair and output prediction abilities of LLMs. The paper provides a comprehensive analysis on over 600 problems and over 50 LLMs, demonstrating that time-segmented evaluations are useful for detecting contamination.
weaknesses: - The novelty is limited. Using competitive programming problems for LLM evaluation has been well adopted. Additionally, the evaluation scenarios in this paper are also not new but borrowed or adapted from prior work (e.g., the test case output prediction can be seen as a chain-of-thought prompting solution of the code execution task). The main contribution of this paper is to scrape the problems with appropriate difficulty and analyze the results in a time-segmented way.
- The relatively small number of problems (~40 LeetCode problems every two months) raises concern about the reliability of the results. I wonder if it’s possible to estimate the variance of the pass@1 performance and the statistical significance of the comparisons.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: The contribution of this work to the community depends largely on your continued maintenance efforts. Do you have a long-term plan?
soundness: 3
strengths: 1. **Benefiting the community**: The benchmark will be continuously updated and provide code ability evaluation with less contamination.
2. **Large scale evaluation**: The authors evaluated 50 LLMs with carefulness about contamination, providing a valuable reference for the code-related capabilities of each model.
3. **Having insights**: The experiments revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.
summary: This paper worked on the evaluation of LLMs for code, proposing a benchmark named LiveCodeBench. The benchmark will be continuously updated by automatically collecting problems from online code contest sites to overcome data contamination. It contains four code-related tasks: code generation, self-repair, code execution, and test output prediction. A large-scale evaluation was conducted on 50 LLMs, which revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.
weaknesses: 1. **Unproven data representative**: Besides contamination, we also need to know how many code problems we need and how representative they are, to conduct a comprehensive evaluation. However, they were not answered in this paper. Are there mechanisms to guarantee or prove these properties, in particular for new models?

2. **Unevaluated workflow reliability**: While the workflow of benchmark construction is completely automated, this workflow's reliability was not evaluated. For example, the accuracy of the HTML extractor.

3. **Unproven test case completeness**: Averaged 18 of the test case count basically is far fewer than test cases used inside the programming task websites; the completeness or sufficiency of these test cases was not analyzed in this paper.

4. **Biased filtering**: In the code competition scenario that this work focused on, questions with multiple correct outputs for a single input would contain many unique features or have different problem-type distributions. However, they were all removed in this work, which might lead to a bias. Further analysis of the impact of this filtering should be conducted.

5. **Multi-step track needed**: This work's designs of four tracks were motivated by AlphaCodium. Including AlphaCodium, state-of-the-art competition-level code generation works nowadays broadly applied multi-step workflow. However, besides one-step revision, multi-step code generation was not evaluated in this work.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The tasks of *code execution* and *test case output prediction* appear to be quite similar. 
   According to the paper, the distinction lies in their prompt: 
   code execution is to predict the output based on inputs and functions in *programming language*, 
   while test case output prediction is to predict the output based on inputs and function descriptions in *natural language*. Given the similarity between these two tasks, is there any observable correlation in their results?
2. Test case generation typically is designed as predicting the entire test case from a given function implementation [3,4], 
   which is more practical in real-world software engineering and testing applications, 
   such as in tools like Copilot. 
   I'm curious why the authors choose a different approach to evaluating test generation in the form of $(I, f_{\text{NL}}) \mapsto O$, 
   where the input and natural language function descriptions are used to predict the output.
3. In the **Problem Difficulty** paragraph of Section 3.1, 
   the authors state that they collected problems of varying difficulty levels as labeled by competition platforms. 
   However, how did the authors address potential rating bias across these platforms? 
   Given that different competition platforms are designed for distinct user groups and, as the authors note, CodeForces problems are generally harder than those on other platforms, 
   there is likely a selection bias in difficulty ratings across different platforms.

[3]: Nie, Pengyu, et al. "Learning deep semantics for test completion." ICSE 2023.

[4]: Rao, Nikitha, et al. "CAT-LM training language models on aligned code and tests." ASE 2023.
soundness: 3
strengths: * LiveCodeBench has a live update mechanism that mitigates data contamination and enables continuous growth. 
  This mechanism ensures that the benchmark remains useful for evaluating newer models, 
  even those with a knowledge cut-off date beyond its release.
* With the timestamps on the tasks, the authors conduct a novel "live" evaluation of code generation to directly address data contamination issues. 
  This approach offers deeper insights into the models' actual coding capabilities, focusing on genuine problem-solving rather than mere memorization.
* The difficulty guided problem curation is particularly effective in revealing performance differences between models of similar sizes, making it practical to use for meaningful evaluation.
* The paper employs a clustering method to identify models that overfit HumanEval. 
  While overfitting to older benchmarks is a well-known issue, 
  the authors' method offers empirical evidence and analysis that specifically highlights the models affected by this problem.
summary: This paper introduces LiveCodeBench, a new benchmark designed to evaluate LLMs for code-related tasks. 
LiveCodeBench addresses some key limitations of previous benchmarks, 
including issues like data contamination, overfitting, saturation, and limited application range. 
Unlike existing benchmarks, LiveCodeBench assesses not only code generation but also self-repair, code execution, and test output prediction. 
Alongside proposing the benchmark, 
the authors perform a comprehensive evaluation and analysis of the contamination and overfitting problem in popular LLMs. 
The paper also introduces a difficulty-guided problem curation strategy, 
which enhances the evaluation's effectiveness by creating clearer margins between model performances, 
allowing for a more meaningful comparison.
The evaluation covers 52 models, with sizes ranging from 1.3B to 70B, 
providing a fairer and more reliable understanding of model performance in code-related tasks.
weaknesses: * Novelty: The benchmark's construction primarily aims to create a newer and harder version of existing benchmarks. 
  However, a key issue with LLM benchmarks for code is that Olympiad in Informatics (OI) programs are not typically representative of real-world software engineering and programming languages. 
  OI programs often have a distinct style and differ in their learned representations compared to code from real software projects. 
  This has led to a shift in focus in this research area from OI competitive programming to open-source software projects, as seen in recent work like SWE-bench [1] and RepoBench [2].

[1]: Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." ICLR 2024.

[2]: Liu, Tianyang, et al. "Repobench: Benchmarking repository-level code auto-completion systems." ICLR 2024.

Meta-review: The paper provides a benchmark for evaluating coding capabilities of language models. The benchmark focuses on large range of capabilities going beyond code generation. It is already being widely used by the community which is a big marker for usefulness. However, I still request the authors to update the camera ready version of the paper based on comments from the reviewers.

Decision: Accept (Poster)