LiveBench: A Challenging, Contamination-Limited LLM Benchmark

Authors: Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, Micah Goldblum
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, benchmark

Abstract:
Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730259771544

thread_type:
official_review

<thread_object>
creator: reviewer_g9nn

<content>
summary:
The paper presents LiveBench, a new benchmark for evaluating LLMs designed to avoid common issues such as test set contamination and biases from LLM or human judging. LiveBench achieves this by (1) frequently updating questions with information from recent sources, (2) employing automated scoring against objective ground-truth values, and (3) covering a wide range of challenging tasks in areas such as math, coding, reasoning, language, instruction following, and data analysis.

The authors evaluated multiple prominent closed-source and open-source LLMs, showing that even top-performing models struggle to achieve over 70% accuracy, highlighting the benchmark’s difficulty.

strengths:
1. This paper provides a dynamic, comprehensive and challenging benchmark that stays current with LLM development.
2. The benchmark’s monthly updates and open access to questions, code, and model answers contribute to high reproducibility and transparency.

weaknesses:
1. The paper fails to analyze the quality of the questions used for evaluation, particularly those manually constructed or adjusted for model testing.
2. The paper seems to employ a regular expression matching approach against ground truth to evaluate model outputs, a method which can introduce evaluation bias, especially for models with weaker instruction-following abilities. While the paper mentions potential biases from using large language models as evaluators, it neglects to analyze how regex-based matching could exacerbate evaluation inaccuracies.

questions:
1. Are all tasks' ground truths unique and definitive? How do you ensure that there is no ambiguity in the ground truth for each task?
2. In line 509, you mentioned that "In Srivastava et al. (2024), the authors modify the original MATH dataset (Hendrycks et al., 2021) by changing numbers in the problem setup, leading to significant declines in model performance across all LLMs, including the leading ones." Given this, do the tasks generated in this benchmark, such as the Typos task, also face similar issues due to using updated data sources under the same modification rules?
3. Could you provide a statistical table that clearly presents the number of questions, tasks, sources, and other relevant information for each category? This would enhance clarity and help readers understand the distribution and characteristics of the tasks more effectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659217898

thread_type:
official_review

<thread_object>
creator: reviewer_l2m4

<content>
summary:
The paper presents LiveBench, a benchmark designed to evaluate LLMs while avoiding data contamination and biases associated with LLM and human judges. LiveBench includes frequently updated, diverse tasks across categories such as math, coding, reasoning, and data analysis, scoring responses based on ground-truth answers to ensure objectivity. With new, harder tasks released monthly, LiveBench aims to accurately assess LLMs as they improve over time.

strengths:
- The benchmark addresses a prevalent issue in LLM evaluation.
- The paper is well-written, clearly explaining both the motivation and the implementation.
-  LiveBench scores models against objective ground-truth answers, not relying on possibly-biased LLM-judges.
- Tasks presented are very diverse.

weaknesses:
- Some models' performance in evaluations are very close. It's hard to say how similar they are without a significance test / stds.
- As LLM developers may train their models to excel on specific benchmarks, frequent updates could eventually lead to overfitting to LiveBench’s task formats, even with the regular introduction of new content. This could incentivize models to optimize for LiveBench-specific tasks rather than improving general capabilities. This can be solved with the introduction of more types of tasks and the contribution of the community, but it's also worth thinking about refreshing existing tasks' formats.

questions:
Why did you not evaluate with some randomness to obtain significance over noise?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720963424

thread_type:
official_review

<thread_object>
creator: reviewer_uo5e

<content>
summary:
The paper introduces LiveBench, a new benchmark designed to evaluate LLMs in a way that mitigates common issues like test set contamination and biases introduced by human or LLM judges. LiveBench is continuously updated with new questions sourced from recent information, such as math competitions, arXiv papers, news articles, and datasets. This ensures the benchmark remains relevant and resistant to contamination by models trained on static datasets. Unlike many benchmarks that rely on subjective human or LLM judgments, LiveBench uses objective ground-truth values for scoring. LiveBench includes 18 tasks across six categories—math, coding, reasoning, language comprehension, instruction following, and data analysis. Each task is designed to be difficult, with top-performing models achieving less than 70% accuracy. The tasks are either based on recent real-world data or are harder versions of existing benchmark tasks. The paper evaluates dozens of LLMs, both proprietary and open-source. The benchmark provides detailed results across all six categories and highlights the strengths and weaknesses of each model. The authors plan to expand the benchmark with new tasks and models over time. They also plan regular updates to ensure the benchmark remains challenging as LLMs improve.

strengths:
Originality:
This paper makes very well rounded verifiable benchmarks that cover a large number of tasks in different ways. This in itself is unique and a great contribution to LLM research.
While some of the tasks are inspired by existing benchmarks, some seem to be completely novel evaluation strategies (such as the proof reconstruction).

Quality:
The benchmark itself seems mostly well constructed and fair. The emphasis on task diversity and verifiability is commendable.

Clarity:
The methods are generally clear. Experiments are well explained. Benchmark tasks/evaluation are clear.

Significance:
This paper gives a useful benchmark for offline evaluation of LLMs. It has more task coverage than many other offline evaluation schemes. It also presented potential solutions to the benchmark contamination problem by making the benchmark “live”.

weaknesses:
The core feature LiveBench claims to offer is being “contamination-free” However, it seems this claim rests upon two main assumptions that are never addressed:
1. The source data is not contaminated.
2. The data perturbations used to decontaminate or generate new data are actually sufficient to do so.

For (1): Consider the math category which utilizes questions from AMC12 2023, SMC 2023, AIME 2024, and IMO 2024. It seems likely these high quality data points would already be exploited by model developers for training models, intentionally or unintentionally. While the author’s do not directly discuss the implications of this, they do mention they do make some modifications to the questions. This leads to (2).

For (2): “Contamination-free” is a strong statement. **First and foremost, I recommend the authors define exactly what they mean by “Contamination-free”.** As a reader, my understanding of “contamination-free” is as follows: *Given I train my LLM on LiveBench up to month M, I will have no performance gain on the queries released in month M + 1.* Now, if the authors choose to run with this relatively strong definition of contamination-free, the authors should consider experimentally verify that training a model on prompts up to month M leads to negligible gains on prompts from month M + 1. Right now, I find this doubtful. For example, one of the author’s stated methods to change queries is through “modifying the prose and the answer order” (line 142). While it is likely that changing trivial sections of a query may decrease an overfit LLM’s performance slightly, the authors do not show that this fully mitigates contamination, or even to what extent it does (if not fully). Moreover, the authors suggest on line 303 that “many of the tasks are synthetic and therefore very fast and simple to create a new set of questions based on fresh data”. This raises the question: what is stopping an adversarial from generating, let’s say 1 million rows, of LiveBench data through this “fast and simple” process, training on it directly. Does this count as “contamination”? Would the release of 200 prompts in month M + 1 protect against this? Alternatively, let’s consider the instruction following summarization task: in this case, the update process presumably is to swap out the Guardian articles with new ones and change around the instructions selected from the 16 choices. However, how can the author’s be sure this meaningfully changes the task, especially because the article doesn’t even matter— the only thing the model needs to learn in order to overfit is the 16 different instructions that it will be checked on?

The authors could potentially address this in 2 ways:
1. Experimentally show how much training (with a reasonable setup) on M affects M + 1.
2. Establish a very weak definition of “Contamination-free” (e.g. the exact prompt is not seen in M + 1). However, there are no guarantees that training on M will not help M + 1. (Note that this heavily decreases the significance of the benchmarks as “contamination-free”)

**Bottom Line: In its current implementation, this paper has little evidence that its current query perturbation/generation process has any effectiveness against contamination. The authors should first define contamination, then defend that their benchmark sufficiently protects against this definition of contamination.**


**The “live” aspect of the benchmark is powered by brute force, and is not necessarily automatic in any sense:** It is not entirely clear how LiveBench is significantly more “update-able” than existing static and verifiable benchmarks, such as MMLU or IFEval. The author’s pledge to keep updating the benchmark monthly does not seem like a sufficient novel contribution to benchmark construction in general. The authors should document the update plan for every single fine-grain task. They should explain the estimated time and monetary cost to update this specific task. They should detail any optimizations in task design to ensure it is easy to update, if any. This would be a great table for the appendix and would make it clear what the paper contribution on this front is.


**Ablations on LLM-Judgments are insufficient, and potentially misleading:**
The experiments to show “LLM judges cannot accurately evaluate challenging math and reasoning questions” are not enough to address the scope of the claim.
In particular:
1. Only GPT-4-Turbo was tested as judge
2. GPT-4-Turbo was mainly judging itself, which is already likely worst case scenario
3. Only 2 models total were judged
4. Only 4 benchmarks tested
5. Subpar judge prompt

    a. No reference answer (generally these judges have either pairwise comparisons, or some form of reference answer)

    b. No rubric for judging

Possible ways to strengthen the experiment to adequately support the claim are as follows:
1. Utilize more LLMs as judges
2. Use the LLM to judge more models
3. Judge with a reference answer, and/or use a more updated judge prompt, such as the judge prompt from Arena-Hard-Auto or Alpaca-Eval 2.0.
4. Show accuracies on all tasks


**Insufficient comparisons to other benchmarks or “ground truth” evaluations:**
The authors compare LiveBench to Chatbot Arena and Arena-Hard. They hypothesize that the differences in rankings are mainly due to style or self-bias. If the authors suspect style to be a large factor in ranking difference they should compare LiveBench taking to style controlled Chatbot Arena rankings, proposed in the blog https://blog.lmarena.ai/blog/2024/style-control/. Note that Arena-Hard also has style control implemented. The authors should provide a table showing the correlation between the benchmarks. There are also Chatbot Arena categories, such as Math or Instruction following, which would provide more informative correlations to the respective LiveBench tasks.

questions:
On line 214, the author suggests that LLMs are commonly used to perform table joins. Can they elaborate on this? This does not seem like a common real-world use case.


On line 299, the authors suggest they will release 200 prompts per month, but it seems in lines 307-311 they released significantly less than 200 prompts per month. Could the authors clarify what the plan for questions releases is?


On line 331, the authors suggest they use FastChat chat templates. Why is FastChat used over the explicit template provided in model tokenizers or readmes? FastChat is not necessarily guaranteed to be maintained to be constantly up to date for this purpose. Not all models are implemented in FastChat.


On line 363-365, the authors suggest that the zebra puzzles are too hard such that top models do not perform better than random chance. If this is the case, is there anything that can be done to make this category more relevant? E.g, making it slightly easier? Adding this noise into a benchmark seems unnecessary.


On line 402-407, the authors color outliers in red. However, outlines that are far below the line (large negative residual) are not colored. Why were some points determined as outliers but others not? This is true for both graphs.  Moreover, on the left plot, o1-preview is shown as an outlier, but the point next to Phi-3.5-mini-instruct is not shown as an outlier, even though the latter is even farther from the line of best fit.


On lines 438-445, the authors show Starling-LM-Beta on their graph, presumably to show the large differential between Chatbot Arena score and LiveBench score. However, Starling-LM-Beta is notably missing from Figure 5 which is described to show “all models”.  It is also missing from Table 3. Starling appears again in Figure 6… then it is gone again in Table 7… then back again in Table 9. Why is this the case? (I think Open Hermes and a couple other models also appear and disappear).


On lines 324-329, the authors cite various models they use. However, the citations for the remaining models are not present anywhere else. I find it strange that some models are chosen to be cited properly and others are not.


It seems the authors commonly use **answer ** to make the LLM generated answer parsable for verification. Why was ** chosen here, when it is also a common markdown symbol? Wouldn’t a less common symbol make sense here, like “{{answer}}” as one example. Peaking at the code, which seems to take the last bolded text as the final answer, it is conceivable that a markdown heavy model could have some extra bolding after the answer and get penalized for this. Think of something like “... **Final Answer: ** **X **\n\n **Additional Considerations: ** …” It is important to note that adding any bolded content after the final answer seems to not be explicitly warned against in the prompt to the LLM.


On lines 914-917, the authors show the judge prompt used for the LLM judge, which outputs a numerical score 1-5. However, it is unclear how this score is turned into a binary label used to calculate the error rates seen in Table 2.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732557802811

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all of the reviewers for their valuable feedback. Our work introduces LiveBench, a challenging benchmark that is regularly updated in order to limit contamination.  We appreciate that the reviewers found our work to be unique and a great contribution to LLM research (uo5E), with diverse, challenging tasks (uo5E, G9nN, L2M4). We also appreciate that reviewers found our work to be clearly written (uo5E, L2M4), with high reproducibility and transparency (G9nN).

We believe that LiveBench fills an important role in the research community: it is a consistent measure of hard reasoning questions, similar to MMLU and GSM8K, or BigBench-Hard, while being minimally contaminated. There are other benchmarks with limited or no contamination, but they have substantial differences such as lack of transparency, being expensive to update, or using LLM judging. LiveBench occupies an important position of being a contamination-limited, reasoning-focused leaderboard, and new models can be added very quickly to the leaderboard.

We highlight all of the new additions based on the reviewers’ suggestions
- We added new sections to the paper
  - We added Section A.4: Grading Methodology.
  - We added Section A.5: Quality Control for New Models.
  - We added an expanded section on the question update policy: A.6.
 - We updated the language in the title and throughout the paper from claiming LiveBench is “contamination-free” to “contamination-limited” (note: we still argue that LiveBench has high impact). We also add a new discussion on the different types of contamination to Section A.7.
- We updated and added new figures and tables
  - Table 4 - Citations for all models in LiveBench.
  - Table 13 - Describes the data source for each LiveBench task.
  - Table 14 - Describes the time to update each LiveBench task.
- We updated and clarified additional parts of the paper based on reviewer feedback.

We also have preliminary results for the next update to LiveBench. We updated all four instruction following tasks, the connections task, and the Zebra puzzles task. For the instruction following tasks and the connections task, we created all new questions using new information sources (new news articles for instruction following, and new daily connections puzzles). We also made the distribution of problems harder: adding more instructions per question to the instruction following tasks, and adding more 4 by 4 connections puzzles. For Zebra puzzles, we significantly increased the difficulty, due to o1-preview and o1-mini achieving high scores on the old version. The new version has a wider range of both puzzle size and clue-difficulty, and asks three questions about the solution rather than one, making it harder to guess. In addition, we give partial credit now, making the task have less noise. The following are preliminary results, with the full results to come very soon (before the end of the rebuttal period).

| Model | Connections | Simplify | Story Generation | Summarize | Zebra Puzzle |
|---|--:|--:|--:|--:|--:|
| claude-3-5-haiku-20241022 | 26.17 | 70.65 | 59.00 | 58.65 | 22.25 |
| claude-3-5-sonnet-20241022 | 43.17 | 72.67 | 68.67 | 69.05 | 44.00 |
| gpt-4o-2024-11-20 | 36.67 | 67.75 | 66.25 | 63.10 | 47.00 |
| gpt-4o-mini-2024-07-18 | 16.50 | 60.73 | 56.13 | 52.98 | 28.25 |
| o1-mini-2024-09-12 | 59.33 | 73.00 | 63.00 | 59.00 | 67.00 |
| o1-preview-2024-09-12 | 86.67 | 78.08 | 71.58 | 69.63 | 66.25 |
| average | 44.75 | 70.48 | 64.11 | 62.07 | 45.79 |

Thank you once again, and we are happy to answer any final questions!
</content>
</thread_object>

</thread>


Metareview:
metareview:
All reviewers agreed that this is a great paper and should be included in the conference: this paper provides a nice way of  minimizing the test set contamination with frequently-updated questions from recent information sources and procedural question generation techniques.

Strength: 
1. Well-rounded verifiable benchmarks that cover a large and diverse set of of tasks.
2. Paper is clearly written.

Weakness:
Regex matching for validation might be limited.