LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

Authors: Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, Ion Stoica
Primary area: datasets and benchmarks
Keywords: Code LLMs; Evaluation; Contaminationl; Overfitting

Abstract:
Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEvla, MBPP) are no longer sufficient for assessing their capabilities suffering from data contamination, overfitting, saturation, and focus on merely code generation. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which collects new problems over time from contests across three competition platforms, Leetcode, Atcoder, and Codeforces. Notably, our benchmark also focuses on a broader range of code-related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts over six hundred coding problems that were published between May 2023 and Aug 2024. We evaluate over 50 LLMs on LiveCodeBench (LCB for brevity) presenting the largest evaluation study of code LLMs on competition problems. Based on the study, we present novel empirical findings on contamination, overfitting, and holistic evaluations. We demonstrate that time-segmented evaluations serve as a robust approach to evade contamination; they are successful at detecting contamination across a wide range of open and closed models including GPT-4O, Claude, Deepseek, and Codestral. Next, we highlight overfitting and saturation of traditional coding benchmarks like HumanEvla and demonstrate LCB allows more reliable evaluations. Finally, our holistic evaluation scenarios allow for measuring the different capabilities of programming agents in isolation.

Decision: accept (poster)
Reviewer scores: 6, 5, 8, 6

Reviews and discussion:
<thread>
creation date:
1729807783374

thread_type:
official_review

<thread_object>
creator: reviewer_9frx

<content>
summary:
This paper introduces LiveCodeBench, a new benchmark designed to evaluate LLMs for code-related tasks. 
LiveCodeBench addresses some key limitations of previous benchmarks, 
including issues like data contamination, overfitting, saturation, and limited application range. 
Unlike existing benchmarks, LiveCodeBench assesses not only code generation but also self-repair, code execution, and test output prediction. 
Alongside proposing the benchmark, 
the authors perform a comprehensive evaluation and analysis of the contamination and overfitting problem in popular LLMs. 
The paper also introduces a difficulty-guided problem curation strategy, 
which enhances the evaluation's effectiveness by creating clearer margins between model performances, 
allowing for a more meaningful comparison.
The evaluation covers 52 models, with sizes ranging from 1.3B to 70B, 
providing a fairer and more reliable understanding of model performance in code-related tasks.

strengths:
* LiveCodeBench has a live update mechanism that mitigates data contamination and enables continuous growth. 
  This mechanism ensures that the benchmark remains useful for evaluating newer models, 
  even those with a knowledge cut-off date beyond its release.
* With the timestamps on the tasks, the authors conduct a novel "live" evaluation of code generation to directly address data contamination issues. 
  This approach offers deeper insights into the models' actual coding capabilities, focusing on genuine problem-solving rather than mere memorization.
* The difficulty guided problem curation is particularly effective in revealing performance differences between models of similar sizes, making it practical to use for meaningful evaluation.
* The paper employs a clustering method to identify models that overfit HumanEval. 
  While overfitting to older benchmarks is a well-known issue, 
  the authors' method offers empirical evidence and analysis that specifically highlights the models affected by this problem.

weaknesses:
* Novelty: The benchmark's construction primarily aims to create a newer and harder version of existing benchmarks. 
  However, a key issue with LLM benchmarks for code is that Olympiad in Informatics (OI) programs are not typically representative of real-world software engineering and programming languages. 
  OI programs often have a distinct style and differ in their learned representations compared to code from real software projects. 
  This has led to a shift in focus in this research area from OI competitive programming to open-source software projects, as seen in recent work like SWE-bench [1] and RepoBench [2].

[1]: Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." ICLR 2024.

[2]: Liu, Tianyang, et al. "Repobench: Benchmarking repository-level code auto-completion systems." ICLR 2024.

questions:
1. The tasks of *code execution* and *test case output prediction* appear to be quite similar. 
   According to the paper, the distinction lies in their prompt: 
   code execution is to predict the output based on inputs and functions in *programming language*, 
   while test case output prediction is to predict the output based on inputs and function descriptions in *natural language*. Given the similarity between these two tasks, is there any observable correlation in their results?
2. Test case generation typically is designed as predicting the entire test case from a given function implementation [3,4], 
   which is more practical in real-world software engineering and testing applications, 
   such as in tools like Copilot. 
   I'm curious why the authors choose a different approach to evaluating test generation in the form of $(I, f_{\text{NL}}) \mapsto O$, 
   where the input and natural language function descriptions are used to predict the output.
3. In the **Problem Difficulty** paragraph of Section 3.1, 
   the authors state that they collected problems of varying difficulty levels as labeled by competition platforms. 
   However, how did the authors address potential rating bias across these platforms? 
   Given that different competition platforms are designed for distinct user groups and, as the authors note, CodeForces problems are generally harder than those on other platforms, 
   there is likely a selection bias in difficulty ratings across different platforms.

[3]: Nie, Pengyu, et al. "Learning deep semantics for test completion." ICSE 2023.

[4]: Rao, Nikitha, et al. "CAT-LM training language models on aligned code and tests." ASE 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650033244

thread_type:
official_review

<thread_object>
creator: reviewer_sgu1

<content>
summary:
This paper worked on the evaluation of LLMs for code, proposing a benchmark named LiveCodeBench. The benchmark will be continuously updated by automatically collecting problems from online code contest sites to overcome data contamination. It contains four code-related tasks: code generation, self-repair, code execution, and test output prediction. A large-scale evaluation was conducted on 50 LLMs, which revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.

strengths:
1. **Benefiting the community**: The benchmark will be continuously updated and provide code ability evaluation with less contamination.
2. **Large scale evaluation**: The authors evaluated 50 LLMs with carefulness about contamination, providing a valuable reference for the code-related capabilities of each model.
3. **Having insights**: The experiments revealed the widespread contamination among LLMs, as well as overfitting and saturation of traditional coding benchmarks like HumanEval.

weaknesses:
1. **Unproven data representative**: Besides contamination, we also need to know how many code problems we need and how representative they are, to conduct a comprehensive evaluation. However, they were not answered in this paper. Are there mechanisms to guarantee or prove these properties, in particular for new models?

2. **Unevaluated workflow reliability**: While the workflow of benchmark construction is completely automated, this workflow's reliability was not evaluated. For example, the accuracy of the HTML extractor.

3. **Unproven test case completeness**: Averaged 18 of the test case count basically is far fewer than test cases used inside the programming task websites; the completeness or sufficiency of these test cases was not analyzed in this paper.

4. **Biased filtering**: In the code competition scenario that this work focused on, questions with multiple correct outputs for a single input would contain many unique features or have different problem-type distributions. However, they were all removed in this work, which might lead to a bias. Further analysis of the impact of this filtering should be conducted.

5. **Multi-step track needed**: This work's designs of four tracks were motivated by AlphaCodium. Including AlphaCodium, state-of-the-art competition-level code generation works nowadays broadly applied multi-step workflow. However, besides one-step revision, multi-step code generation was not evaluated in this work.

questions:
The contribution of this work to the community depends largely on your continued maintenance efforts. Do you have a long-term plan?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692597225

thread_type:
official_review

<thread_object>
creator: reviewer_mfut

<content>
summary:
Prior LLM coding benchmarks suffer from contamination and saturation issues. This paper introduces LiveCodeBench, a new benchmark that collects new problems from public programming contests (LeetCode, AtCoder, and Codeforces). Besides code completion, the benchmark also tests the self-repair and output prediction abilities of LLMs. The paper provides a comprehensive analysis on over 600 problems and over 50 LLMs, demonstrating that time-segmented evaluations are useful for detecting contamination.

strengths:
- The paper proposes a coding benchmark with live updates, ensuring fair model evaluation by testing only on new problems after each model’s cutoff date. The time-segmented analysis also reveals a significant drop in performance of some models after release date, indicating notable contamination issues.
- The paper is well-written and organized, with a thorough appendix including discussion on legal compliance and benchmark creation details.

weaknesses:
- The novelty is limited. Using competitive programming problems for LLM evaluation has been well adopted. Additionally, the evaluation scenarios in this paper are also not new but borrowed or adapted from prior work (e.g., the test case output prediction can be seen as a chain-of-thought prompting solution of the code execution task). The main contribution of this paper is to scrape the problems with appropriate difficulty and analyze the results in a time-segmented way.
- The relatively small number of problems (~40 LeetCode problems every two months) raises concern about the reliability of the results. I wonder if it’s possible to estimate the variance of the pass@1 performance and the statistical significance of the comparisons.

questions:
It seems that the drop in performance occurs only for the LeetCode problems and is smooth for other platforms like AtCoder. Do you have any explanation for this interesting phenomenon?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702997018

thread_type:
official_review

<thread_object>
creator: reviewer_zpuz

<content>
summary:
The paper introduces LiveCodeBench, a lively-updated dynamic benchmark that is designed to evaluate large language models (LLMs) for coding tasks with minimized data contamination risk.

Aside from being contamination-free, LiveCodeBench also seeks to improve the data quality, difficulty, and task diversity of the benchmark.

The paper also conducts thorough experiments exposing the significant data contamination and overfitting issues.

strengths:
## Significance

The paper addresses one of the most important issues: data contamination in the existing Code LLM evaluation process. The benchmark is highly practical and should be known and tried by the Code LLM community to encourage more rigorous evaluation of models. 


## Soundness

The paper demonstrates strong soundness through its detailed analyses of the models' performance over time, overfitting issues on HumanEval, and comparison of different types of models.


## Effectiveness

Dynamic benchmarking is indeed an effective solution to mitigate data contamination.

weaknesses:
## Novelty

The paper presents two key contributions:

1. A comprehensive dynamic benchmark for coding takes

2. Analyses exposing the contamination issues in coding tasks

While these contributions are highly relevant and practical for the Code LLM community, it's important to note that dynamic benchmarking is a technique that has been utilized in various prior works [1,2,3,4] as early as 2021 [1]. Additionally, the issue of data contamination has been systematically examined in earlier studies [5,6], limiting the novel insights that the paper can offer to the broader ICRL community.

To enhance the paper's novelty, it may be beneficial to develop better and more efficient dynamic benchmarking techniques, or introduce new methods to alleviate more intricate contamination issues such as re-phrasing [6].




[1] [Dynaboard: An Evaluation-As-A-Service Platform for Holistic Next-Generation Benchmarking](https://arxiv.org/abs/2106.06052)

[2] [Competition-Level Problems are Effective LLM Evaluators](https://arxiv.org/abs/2312.02143)

[3] [LatestEval: Addressing Data Contamination in Language Model Evaluation through Dynamic and Time-Sensitive Test Construction](https://arxiv.org/abs/2312.12343)

[4] [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://arxiv.org/abs/2312.14890)

[5] [To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination](https://openreview.net/forum?id=m2NVG4Htxs)

[6] [Rethinking Benchmark and Contamination for Language Models with Rephrased Samples](https://arxiv.org/abs/2311.04850)

questions:
## Discussion Questions

Have you considered addressing more intricate data contamination issues like re-phrasing within the scope of this work?

I believe that many code-specific techniques can be developed to detect re-phrasing contamination such as checking the equivalence or similarity of the test cases and canonical solutions.

I would be happy to raise my score if the authors could incorporate and address those intricate contamination issues in the revision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper provides a benchmark for evaluating coding capabilities of language models. The benchmark focuses on large range of capabilities going beyond code generation. It is already being widely used by the community which is a big marker for usefulness. However, I still request the authors to update the camera ready version of the paper based on comments from the reviewers.