LiveBench: A Challenging, Contamination-Limited LLM Benchmark
Authors: Colin White
Samuel Dooley
Manley Roberts
Arka Pal
Benjamin Feuer
Siddhartha Jain
Ravid Shwartz-Ziv
Neel Jain
Khalid Saifullah
Sreemanti Dey
Shubh-Agrawal
Sandeep Singh Sandha
Siddartha Venkat Naidu
Chinmay Hegde
Yann LeCun
Tom Goldstein
Willie Neiswanger
Micah Goldblum
Venue: ICLR 2025 Spotlight (2025)
Keywords: large language models
benchmark
Primary area: foundation or frontier models, including LLMs
TL;DR: LiveBench is a difficult LLM benchmark consisting of contamination-limited tasks that employ verifiable ground truth answers on frequently-updated questions from recent information sources and procedural question generation techniques.

Abstract: Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and break down when scoring hard questions. In this work, we introduce a new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release LiveBench, the first benchmark that (1) contains frequently-updated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination-limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We evaluate many prominent closed-source models, as well as dozens of open-source models ranging from 0.5B to 405B in size. LiveBench is difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time so that LiveBench can distinguish between the capabilities of LLMs as they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.

Published as a conference paper at ICLR 2025
LIVEBENCH: A CHALLENGING, CONTAMIVATION-LIITEDD LLM BENCHMARK
Colin White" -aamuel Dooley*!, Mannley Roberts" Arka Arka Pal" Benjammi Feuer?, Siddhartha Jain', Ravid Shwartz-Ziv Neel Jain', Khalid Saifullah", Sreemanti Dey', Shubh-Agrawal! Sandeep Singh Sandha', Siddartha Naidu', Chinmay Hegde", Yann LeCun?, Tom Goldstein', Willie Neiswanger', Micah Goldblum 1 Abacus.AI, 2 NYU," Nvidia, 4 UMD, USC, Columbia
ABSTRACT
Test set contamination, wherein test data from a benchmark ends up in a newer model's training set, is a well-documented obstacle for fair LLM evaluation and can quickly ren- der benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human LLM judges; however, these can introduce sig- nificant biases, and break down when scoring hard questions. In this work, we introduce new benchmark for LLMs designed to be resistant to both test set contamination and the pitfalls of LLM judging and human crowdsourcing. We release Li veBench, the first benchmark that (1) contains frequently-pdaated questions from recent information sources, (2) scores answers automatically according to objective ground-truth values, and (3) contains a wide variety of challenging tasks, spanning math, coding, reason- ing, language, instruction following, and data analysis. To achieve this, LiveBench contains questions that are based on recently-released math competitions, arXiv papers, news articles, and datasets, and it contains harder, contamination- limited versions of tasks from previous benchmarks such as Big-Bench Hard, AMPS, and IFEval. We eval- uate many prominent closed-source models, well dozens of open-source models ranging from 0.5B to 405B in size. LiveBench difficult, with top models achieving below 70% accuracy. We release all questions, code, and model answers. Questions are added and updated on a monthly basis, and we release new tasks and harder versions of tasks over time that LiveBench can distinguish between the capabilities of LLMs they improve in the future. We welcome community engagement and collaboration for expanding the benchmark tasks and models.
INTRODUCTION
In recent years, as large language models (LLMs) have risen in prominence, it has become increasingly clear that traditional machine learning benchmark frameworks are no longer sufficient to evaluate new models. Benchmarks are typically published on the internet, and most modern LLMs include large swaths of the internet in their training data. If the LLM has seen the questions of benchmark during training, its performance on that benchmark will be artificially inflated (referred to as "test contamination") (Roberts al., 2024; Dong et al., 2024; Deng et al., 2023; Golchin & Surdeanu, 2023b), hence making many L.MM benchmarks unreliable. Recent evidence test set contamination includes the observation that LLMs' performance on Codeforces plummet after the training cutoff date of the LLM (Roberts et al., 2024; Jain et al., 2024), and before the cutoff' date, performance is highly correlated with the number of times the problem appears on GitHub (Roberts al., 2024). Similarly. a recent hand-crafted variant of the established math dataset, GSM8K, shows evidence that several models have overfit to this benchmark (Zhang et al., 2024; Cobbe et al., 2021).
To lessen dataset contamination, benchmarks using L.M or human prompting and judging have become increasingly popular (Jain et al., 2024; Chiang al., 2024; Zheng et al., 2024; Li al., 2024). However, using these techniques comes with significant downsides. While LLM judges have
"crwhi te@meta. com, dool w ey@meta
com, mi 1...g.1122c 11320cc111 umbi edu. Sponsored by Abacus.AT.

---

Published as a conference paper at ICLR 2025
LiveBench
Reasoning
Language
21-previce-202:00 2024-09-12
(laude-1---sconatt--
exp 1x14
gemini-ex-=1111
stop-2-1Re-20441
gpt 0pt-4o-202408-6-
gemin-
Math
Coding
gpt.to-202405-11
20 46 60 80
gemini 1.5 1.55 pro-exp-0827
meta-llama-
1.1.40:b-mmirutt-ii
gp--6o22224-31-200
d'acaryc.72bb-rre -irirrd
gur-4-uube turbo 03
1inude--qpus-200
gemini-1. 5-faash907
lare
Instruction Following
Data Analysis
toder insruct
2it-44-125-pnviiie
gem in- Selishh-
gemin-1,5 pr----00
meta 3.1
claudle3 halku 20241022
model
do mini 20240mmi-20020771 2024-7-18
gemini1 5-fast--001
21-preview-2222-00912
geminien gemm=222/bbt
clude-3-5-soonet-2224400
awen2.5 autr
claude 3 halku 20240307
01 01-mini-2024-09-15
mistral-522 [b-istratt-40..
gemini-exp-1114
gemma
small-2422
Open Source
step-2-16k-202411
10
20
30
60
70
Figure I: Left: results on LiveBench for top models, showing 95% bootstrap confidence intervals. Right: radar plot for select models across LiveBench's six categories demonstrating the that ordering top models varies between each category.
multiple advantages, such as their speed and ability to evaluate open-ended questions, they are prone making mistakes and can have several biases (Li al., 2024). Furthermore, L.MMs often favor their own answers over other LLMs, and LLMs favor more verbose answers (Li et al., 2024; Dubois et al., 2024; Li al., 2023b), Additionally, using humans provide evaluations of LLMs can inject biases such as formatting of the output and the tone of the writing (Chiang et al., 2024). Using humans to generate questions also presents limitations. Human participants might not ask diverse questions, may favor certain topics that do not probe model's general capabilities, or may construct their prompts poorly (Zheng et al., 2024).
In this work, we introduce framework for benchmarking LLMs designed to minimize both test set contamination and the pitfalls L.M judging and human crowdsoureing. We use this frame- work to create LiveBench, the first benchmark with these three desiderata: (I) Li civeBench contains frequently-updated questions based on recent information sources; (2) LiveBench is scored automatically according to the objective ground truth without the use of an LLM judge; and (3) Li siveBench questions are drawn from a diverse set six categories. We ensure (2) by only including questions that have objectively correct answer. LiveBench questions current model achieves higher than 70% accuracy. Questions are added and updated on a monthly basis, and release tasks and harder versions of tasks time that L.i veBench can distinguish among the capabilities of LLMs as they improve in the future.
difficult:
Overview of tasks. I. LiveRench currently consists of 18 tasks across 6 categories: math, coding, reasoning, language, instruction following, and data analysis. Each task falls into one of two types: (I) tasks which use an information source for their questions, c.g., data analysis questions based on recent Kaggle datasets, or fixing typos in recent arXiv abstracts; and (2) tasks which are more challenging or diverse versions of existing benchmark tasks, e.g., from Big-Bench Beg-Bench Hard (Suzgun al., 2023) or [FEval (Zhou et al., 2023a). The categories and tasks included in Li LiveBecc are:
Math: modified questions based on high school math competitions from the past 11 months, as well harder versions of AMPS questions (Hendrycks al., 2021)
Coding: code generation questions from recent Leetcode and AtCoder questions (via Live- CodeBench (Jain et al., 2024)), well novel code completion task
Reasoning: harder version of Web of Lies from Big-Bench Hard (Suzgun et al., 2023), and novel Zebra Puzzles (e.g., (Jeremy, 2009)) and spatial reasoning questions
2

---

Published as a conference paper at ICLR 2025
Language Comprehension:
Connections word puzzles, typo-fixing task, and movie synopsis unscrambling task for recent movies on IMDb and Wikipedia
Instruction Following: four tasks paraphrase, simplify, summarize, generate stories about recent new articles from The Guardian (Guardian Media Group, 1821), subject to one or more instructions such as word limits or incorporating specific elements in the response
Data Analysis: three tasks using recent datasets from Kaggle and Socrata, specifically, table reformatting (among JSON, JSONL, Markdown, CSV, TSV, and HTML), predicting which columns can be used to join two tables, and predicting the correct column type annotation
We evaluate dozens of models, including proprietary models as well as open-source models with sizes ranging from 0.5B 8x22B. We release all questions, code, and model answers, and we welcome community engagement and collaboration. Our codebase is available at httpps:/ggttii com/ livebench wwwpheehhh.ieeeeech...cc.o and our leaderboard available at https: //iveeeechh/.
2 LIVEBENCH DESCRIPTION
In this section, we introduce I, veBench. It currently has six categories: math, coding, reasoning, data analysis, instruction following, and language comprehension. Categories diverse with two to four tasks per problem. Each task either includes recent information sources (such as very recent news articles, movie synopses, or datasets) or a more challenging. more diverse version of an existing benchmark task.
Each task is designed have 40-100 questions which span a range of difficulty. from easy very challenging, while loosely aiming for overall 30-70% success rate the top models for each task, Prompts are tailored for each category and task but typically include the following: zero-shot chain of thought (Kojima al., 2022: Wei et al., 2022), asking the model make its best guess if does not know the answer, and asking the LLM to output its final answer in a way that is easy to parse, such as in XML tags or in #*double asterisks ** We also acknowledge that parsing answers in this way requires some degree of instruction following, and we address this Appendix A.4. In the following sections, give summary of each task from each category. See Appendix A.3 for additional details.
2.1 MATH CATEGORY
Evaluating the mathematical abilities of LLMs has been one of the cornerstones of recent research in LLMs, featuring prominently in many releases and reports (Reid al., 2024; OpenAI, 2023; Bubeck et al., 2023). Our benchmark includes math questions of three types: questions modified from recent high school math competitions, fill-in-heebblan questions from recent olympiad competitions, and questions from our new, harder version of the AMPS dataset (Hendrycks et al., 2021).
Our first two math tasks, Competi ions and Olympiaade are based on expert human-designed math problems that offer wide variety in terms of problem type and solution technique. In Competi ions, we include questions from AMC12 2023, SMC 2023, and AIME 2024 modifying the prose and the answer order; in 01 ympiad, we include questions based on USAMO 2024 and IMO 2024, in which the task to rearrange masked out equations from the solution into the correct order. These questions test problem solving with algebra, combinatorics, geometry, logic, number theory, probability, and other secondary school math topics (Faires & Wells, 2022).
Finally, release synthetically generated math questions in the AMPS Hard task. This task is inspired by the math question generation used to create the MATH and AMPS datasets (Hendrycks et al., 2021). We generate harder questions by drawing random primitives, using larger and more challenging distribution than AMPS across 10 of the hardest tasks within AMPS.
2.2 CODING CATEGORY
The coding ability of LLMs is one of the most widely studied and sought-after skills for LLMs (Mnih et al., 2015; Jain et al., 2024; Li et al., 2023a). We include two coding tasks in LiveBench: modified version the code generation task from BiveCodeBench (LCB) (Jain et al., 2024), and a novel code completion task combining LCB problems with partial solutions collected from GitHub.
3

---

Published as a conference paper at ICLR 2025
The LCB Generation ittiioiooon assesses model's ability to parse competition coding question statement and write a correct answer. We include 78 questions from LiveCodeBench (Jain et al., 2024) which has several tasks to assess the coding capabilities of large language models.
The Completion task specifically focuses on the ability of models to complete a partially correct solution- assessing whether model parse the question, identify the function of the existing code, and determine how to complete it. We use LeetCode easy, medium, and hard problems from LiveCodeBench'ss (Jain al., 2024) April 2024 release, combined with matching solutions from Solutions, omitting the last 15-70% of each solution and asking the LLM complete the solution.
https: / //github. thub. com/ kamyul 1 04 4 MeLeeetCoooe
2.3 REASONING CATEGORY
The reasoning ability of large language models is another highly benchmarked and analyzed skill of LLMs (Wei et al., 2022; Suzgun et al., 2023; Yao al., 2024). In LiveBench, include three reasoning tasks: our harder version of a task from Big-Bench Hard (Suzgun et al., 2023), Zebra puzzles, and spatial reasoning questions.
Web of Lies v2 is an advancement of the similarly named task included in Big- Big-Bench (bench authors, 2023) and Big- Big-Bench Hard (Suzgun al., 2023). The task is to evaluate the truth value of random Boolean function expressed as natural-language word problem. We create new, significantly harder questions by including additional deductive components and several types of red herrings. Next, we include spatial reasoning questions. This set of 50 handwritten questions tests model's ability to make deductions about intersections and orientations of common 2D and 3D shapes.
Finally, we include Zebra Puzzles, well-known reasoning task (Jeremy, 2009) that tests the ability of the model to follow a set of statements that set up constraints, and then logically deduce the requested information. We build on existing repository for procedural generation of Zebra puzzles (quint 1, 2023). Below, we provide an example question from the Zebra Puzzles task.
An example question from Zebra Puzzle task.
There are people standing in line numbered through 3 in a left to right order.
Each person has of attributes: Food, Nationality, Hobby.
The attributes have following possible values:
Food: nectarine, garlic, cucumber
Nationality: chinese. japanese, thai Hobby: magic-tricks, filmmaking, puzzles and exactly one person the line has a given value for an attribute.
Given the following premises about the line of people:
the person that likes garlic is on the far left
the person who is thai is somewhere to the right of the person who likes magic-tricks the person who is chinese is somewhere between the person that likes cucumber and the person who likes puzzles Answer the following question: What is the hobby of the person who thai? Return your answer as a single word, in the following format: **X**, where X is the answer.
2.4 DATA ANALYSIS CATEGORY
LiveBench includes three practical tasks in which the LLM assists in data analysis or data science: column type annotation, table join prediction, and table reformatting. Each question makes use of recent dataset from Kaggle or Socrata.
The first task type annotation task (CTA), we randomly sample table and randomly sample column from that table. We use the actual name that column as the ground truth and then retrieve some samples from that column. We provide the name of all the columns from that table and ask the LLM to select the true column name from those options.
predict the type of a column data table. To create a question for the column
Data analysts often also require table to be reformatted from type to another, e.g., from some flavor of JSON to CSV or from XML to TSV. We emulate that task in TableReformal by providing table in format and asking the L.M reformat it into the target format.
4

---

Published as a conference paper at ICLR 2025
Finally, another common application of LLMs in data analysis is performing table joins (Goldbloom, 2024; Liu et al., 2024b; Sheetrit et al., 2024). In the TableJoin task, the LLM is presented with two tables with partially overlapping sets of columns. The LLM is tasked with creating valid join mapping from the first to the second table.
2.5 INSTRUCTION FOLLOWING CATEGORY
An important ability of an LLM is its capability follow instructions. To this end, we include cntruction-followwee questions in our benchmark, inspired by IFEval (Zhou et al., 2023a), which is an cerruction-folowinng evaluation for LLMs containing verifiable instructions such as "write more than 300 words" or "Finish your response with this exact phrase: (end_phrassee While IFEval used a list of 25 verifiable instructions, we use subset 16 that excludes instructions that do reflect real-world use-cases. See Appendix Table 3. Furthermore, in contrast to IFEval, which presents only the task and instructions with a simple prompt like "write a travel blog about Japan", we provide the models with article from The Guardian (Guardian Media Group, 1821), asking the models to adhere to multiple randomly-drawa instructions while asking the model to complete one of four tasks related the article: P Paraphrase, Simp 1 Ey, Story Generation, and Summar ize. We score tasks purely by their adherence to the instructions.
2.6 LANGUAGE COMPREHENSION CATEGORY
Finally, we include multiple language comprehension tasks. These tasks assess the language model's ability to reason about language itself by, (1) completing word puzzles, (2) fixing misspellings while leaving other stylistic changes in place, and (3) reordering scrambled plots of unknown movies.
First, we include the Connect i ons category. Connections is a word puzzle popularized by the New York Times (although similar ideas have existed previously). In this task, we present questions of varying levels of difficulty with 8, 12, and 16-word varieties. The objective of the game is to sort the words into sets of four words, such that each set has 'connection' between them.
Next, we include the Typos task. The idea behind this task is inspired by the common use case for LLMS in which user asks the L.M to identify typos and misspellings some written text but to leave other aspects of the text unchanged. We create the questions for this task from recent ArXiv abstracts, which we ensure originally have no typos, by programmatically injecting common human typos into the text. Below is an example question from the Typos task.
An example question from the Typos task.
Please output this exact text, with no changes at all except for fixing the misspellings. Please leave all other stylistic decisions like commas and US vs British spellings in the original text.
We inctroduce a Bayesian estimation approach forther passive localization of an accoustic source in shallow water using a single mobile receiver. The proposed prohablistic focalization method estimates the varying source location inther presense of measurement-orge uncertainty. particular, probahílistic data assocation performed match siomedifferrenes-- or-avi (TDOA) observations extracted from the acoustic signal to TDOA predicitons provded by the statistical modle. The performence of our approach is evaluated useing rela acoustic data recorded by single mobile reciever.
Finally, we include the Plot Unscramblin ing task, which takes the plot synopses of recently- released movies from IMDb or Wikipedia. We randomly shuffle the synopses sentences and then ask the LLM to simply reorder the sentences into the original plot. We find that this task very challenging for LLMs, as it measures their abilities to reason through plausible sequences of events.
2.7 LIVEBENCH UPDATES AND MAINTENANCE PLAN
benchmark requires that we update the set of questions over time. We have so far released two updates, and we plan to continue to release updates to add new questions and remove outdated questions. In each update, we replace 1/6 of the questions on average, so that the benchmark is fully refreshed roughly every 6 months. We may speed up the turnover
Maintaining a contamination-lima
5

---

Published as a conference paper at ICLR 2025
Table 1: LiveBench results across the 15 top-performing models. We display in this table the models on Li LiveBench, outputting the results on each main category, as well as each model's overall performance. See Table 2 for the results on all 40 models.
Model
LiveBench Score
Coding
Data Analysis
Instruction Following
Language
Math
Reasoning
al 21-peevieww2222-00-12
64.7
50.8
64.0
74.6
68.7
62.9
67.4
claude-3S--onmee222220002
58.5
67.1
52.8
69.3
53.8
51.3
56.7
claud-3-5--omnee-222466666
58.2
60.8
56.7
68.0
53.2
53.3
57.2
01-mimi-2022400-112
56.7
48.1
54.1
40.9
59.2
gemini-exp-
56.0
$7.5
77.1
38.7
$4.9
55.7
gemini-cxp-1121
56.0
50.4
57.0
80.1
40.0
62.8
45.8
step-2-16k-20241
55.1
46.9
54.9
79.9
44.5
48.9
55.5
3pt-4o-2024-08-06
53.8
51.4
52.9
68.6
47.6
48.2
53.9
gemini- 1.5pro-022
53.4
48.8
$2.3
70.8
43.3
57.4
47.9
gpt-4o-2024-05113
52.6
49.4
52.4
68.2
50.0
46.0
49.6
gemini-
1.5-pro-exp-0827
52.4
40.9
50.8
69.3
46.1
56.1
50.9
51.1
43.8
53.5
72.8
43.2
40.5
gpt-to-2024-11-20
50.6
46.1
47.2
64.9
47.4
42.5
55.7
dracarys2-72--instuue
50.1
56.6
49.1
65.2
33.5
50.6
45,8
chatspt-do-lntess-0993
50.1
47.4
48.7
66.4
45.3
42.1
50.5
rate of questions in the future, based interest in LiveBench. Each month, we do not release the new questions until one month later, so that the public leaderboard always has 1/o questions that two factors: (l) the oldest tasks, and (2)
private. We choose tasks update based primarily the currently easiest tasks. In this way, the questions in Li veBench will stay new and continue to challenge the most capable LLMs. See additional details, as well as longer discussion on different forms of contamination, in Appendix A.6.
Method for sustainability One downside in frequently-updattng benchmark that it requires consistent work and computational resources each month. Therefore, have plan in place to ensure its continued success. We maintain the best (or most popular) 40-50 models on the leaderboard avoid ever-growing list of models evaluate each month. For example, about two versions of each model family on the leaderboard (to show the improvement from the version) but no more. This ensures that have a tractable set of to evaluate on 200 questions each month, which easily within the computational budgets of the authors' institutions. Additionally, have already had community contributions which further reduces the computational burden of the authors.
maintain
50 models
The only other recurring work is update the questions themselves each month. While are excited and able to add novel tasks each month, many of the tasks are synthetic and therefore very fast and simple create new set of questions based on fresh data (e.g., updating the typos task using brand new arXiv papers). Additionally, we have also seen community engagement here well.
Completed monthly updates In the first monthly update, we added 50 questions a new spatial reasoning task, 28 additional coding generation questions, and 12 additional coding completion questions. The total size of the benchmark after this update became 1000. the second monthly update, we fully updated the math olympiad questions, and partially updated the math AMPS Hard and math_ comp questions, for 132 replaced questions, to maintain 1000 questions.
3 EXPERIMENTS
In this section, first we describe our experimental setup and present full results for 40 LLMs on all 18 tasks of Livee Next, we give empirical comparison of Li LiveBench existing prominent LLM benchmarks, and finally, we present ablation studies.
Experimental setup. Our experiments include 40 LLMs total, with a mix of top proprietary models, large open-source models, and small open-source models. In particular, for proprietary models, we include OpenAl models such as ol ol-preview, chatopt-40 and go (Brown al., 2020; OpenAI, 2023), Anthropic models such as claude-3 3----s-neet--224444

---

Published as a conference paper at ICLR 2025
Google models such -0pp-ii005---00022222 (Reid et al., 2024), and Mistral models such mistral-1arge-2444 (Jiang et al., 2023).
For open-source models, include models such LIama-3. 3lama--..-4405-innstzu (Dubey et al., 2024), deepseek-v (Liu et al., 2024a), 1innn225-72--nstrue (Team, 2024b; Yang al., 2024), comand-r-plus-O--000 (Cohere, 2024; Cohere For AI, 2024), gemma-2-27b-ie (Team, 2024a: Team et al., 2024), fettral-x222s. -x2os-xsxttb--tueutttt iis-ttsstrttt-te vonect-v..... 1 (Jiang et al., 2023), and phi-3, 3. din et al., 2024), See Table 4 for full list of citations.
(Ab-
For all models and tasks, we perform single-turn evaluation with temperature 0, unless otherwise noted in the model card. All models run with their respective templates from our updated version of FastChat (Zheng et al., 2024). We run all open-source models with bfloat16 6. When running new models, we take care to set up its hyperparameters and chat template as in the model's 's example code, and we also double check the outputs to make sure that the inference, as well as our automated parsing functions, are working correctly and fairly. See more details in Appendix A.4 and Appendix A.5. For each question, model receives a score from to 1. For each model, we compute the score on each task as the average of all questions, we compute the score each of the six categories as the average of all their tasks, and we compute the final Li LiveBench score as the average of all six categories. In Appendix B, we give additional documentation including average input/output tokens and cost to run Li veBencch for each API model.
3.1 DISCUSSION OF RESULTS
We compare all 40 models on LiveBech according to the experimental setup described above: see Table and Table 2. We find that o1 o1-prevview2222-00011 performs the best over- all, 6% better than all other models. all other models in the data analysis, language. and math categories. The next-best model is 1atte-----sonnet-2000
substantially outperforms
which far outperforms all other models in the coding cate- gory (although c1-min ni outperforms claude-3. 55 in code generation, cl aude-3. 5 has the edge code completion). o1 o1-mini-2000 -9-12 is third overall and is significantly better than all other models in the reasoning category.
The best-performing open-source models are 11ama 33. 1--.3..-----rs-eeeee- 1-40Sb-instruct and gwen2.5-72b: nutrttt,, which virtually tie with each other and outperform gpt-4-turd The best-performing small open-source model is phi-3. 5..i-3.5-mm0 5-w....--ee-sstteeeeeeeeee--- ruct (see Table 2): with only 6.6B active parameters, it outperforms gpt-3. 5 and is on par with mi ral -x22b.
3.2 CORRELATION ANALYSES
Now present analyses involving correlation among different categories and tasks. First, compute the Pearson correlation coefficient among all pairs of categories and tasks in LiveBench (see Figure 2). We find that unsurprisingly, math, coding, and reasoning all correlate with one another. Interestingly, language correlates fairly well with data analysis, likely due to both categories including tasks that require the LLM to output large part of the prompt that modified in specific way (e.g.. by fixing typos or changing the table format). Surprisingly, instruction following correlates relatively weakly with all other categories. Among tasks, we see that math comp correlates the highest with the average LiveBench performance, suggesting that this task is the greatest indicator of overall model performance. This likely due to these being high-quality diverse mathematical reasoning questions (which we modified to reduce contamination).
Next, in order to see the strengths and weaknesses of each model, we create scatterplot of each model's overall Li veBench performance vs. performance on single category or task (Figure 3). By plotting a best fit line and computing the residuals for each model, we can compute which models are outliers in specific categories that is, models that are disproportionately stronger in particular category relative to the best fit line. We see that the ol and phi series of models are outliers in terms of reasoning (Figure 3, left), while some of the Lla,a, remini, and command-r models outliers in terms of instruction following. We present additional details in Appendix A.1, including a table of each model's relative best and worst tasks (computed the highest and lowest residuals).
7

---

Published as a conference paper at ICLR 2025
Instruction follwinng
and unnee Bmbllin
cotlin comp letioon
math
coding
larguaaee
reasonting
date mna.yoas
web des 14
S Hard
Strr Wile
deneratt
Lattee eforma
1.00
web of lics v2 tablejain math, cromp zcora, puzzl plat unscramble
1.
math-
0.95
0.9
coding
coding_com completion
0.8
languaee
connections
paraphrase
reasoning
LOs generation
olympiad
simpiity
analysis
types
instruction Tollowing
stor_gennration
spatial
0.5
tablereformat
Figure 2: Correlations among categories and groups in LiveBench. For each pair of categories (left) and tasks (right) in LiBBench, we compute the Pearson correlation coefficient based on the results for all 40 models.
ressoning vs, average. r. value: 0,944, std efr: 0.071
instruction following www.ppe--------aaaaaaeee Saptop 2 2. 202411
value: 0.817. 0.081
el-mi-d 024-38-12
gemini-1. g S-lach
gemini- p-mbi--l--flaabae----- Sta
70-
gemini- daaaaaaae
40 11-29
mela-lumed htt.://mmee.1000mottsatta
el ma 324-09-122
gemini-app1112 112
63-
com comgund--la.dbee
40
alude opus
flssh
#h 3 mini dk-fistruer modemme-2331
instrctt
@omma
ph- 2e--3-min-12BB-iie ne---mminn-teemmttuu
20
30
50
20
30
40
50
60
average
average
Figure 3: Reasoning and instruction following performance vs. average performance. We plot the performance of all 40 models' reasoning (left) and instruction following (right) performance compared to overall LiveBench performance, computing a best fit line and annotating outliers.
3.3 COMPARISON OTHER LLM BENCHMARKS
Next, compare Li veBench to prominent benchmarks, ChatBot Arena (Chiang al., 2024) and Arena-Hard (Li et al., 2024). Figure 4, we show a bar plot comparison among models that common both benchmarks, and in Figure 6, compare the performance of these models to best-fit line. We also compute the correlation coefficient of model scores among the benchmarks: LiveBench has 0.91 and 0.88 correlation with ChatBot Arena and Arena-Hard, respectively.
Based on the plots and the correlation coefficients, we see that there are generally similar trends to LiveBench, yet some models are noticeably stronger on one benchmark vs. the other. For exam- ple, cpt-4-0122 0012 5 -rreeiieppeee and gpt - -ete---turbooee 2000b----2----00 04-09 09 perform substantially better on Arena-Hard compared to veBench, likely due the known bias from using gpt-4 itself the LLM judge (Li et al., 2024). We hypothesize that the strong performance of some models such the gemini-22 5 models on ChatBot Arena compared to Li LiveBench may be due to having an output style that is preferred by humans. These observations emphasize the benefit of using ground-truth judging, which immune to biases based on the style of the output.
Comparison between Ground-Truth and LLM-Judging As additional comparison between LiveBench and LLM judge based benchmarks, we give preliminary study in the Appendix on the efficacy of L.LM judging for hard math and reasoning questions. Specifically, we initial
8

---

Published as a conference paper at ICLR 2025
Comperissn LiveBanch Hard
Companioon Livedeecch and Chat Anena
LicBoak Anaw-1ar Hard
70
CoMRe derea Troees
3006
2350
on tot gee
des duch 13
you
aeth 2010 ctateess
50b thak
aff stica
yo ctreect
Lonk
Wh restoua
Figure 4: Comparison of LiveBench to other LLM benchmarks. We compare Li Liveh to ChatBot Arena (left) and Arena-Hard (right). We see that while there are generally similar trends, some models are noticeably stronger on one benchmark vs. the other. For example, both GPT-4 models are substantially better on Arena-Hard.
experiment regarding the question, 'if an LLM struggles to answer a hard math or reasoning question, then will the LLM also struggle to determine whether or not a given answer to that question is correct?" Our experiments give evidence that the answer is yes, for zebra puzzles and AMC/AIME questions, but the results are not definitive. See Appendix A.2.
3.4 ANALYSIS OF MONTHLY UPDATES
As described in Section 2.7, we have completed two monthly updates for LiveBench so far. The rank correlation between the original and first update, and the first and second update, are both > 0.997, showing that the model rankings have stayed consistent. On the other hand, between the original and the most-recent set of questions, the median and mean average scores (among models included in all iterations of the leaderboard) have both dropped by about 1. 1.2%, showing that the benchmark becoming harder over time, as newly released models become more capable.
4 RELATED WORK
We describe the most prominent LLM benchmarks and the ones that are most related to our work. For a comprehensive survey, see (Chang et al., 2024). The Huggingface Open LLM Leaderboard (Gao et al., 2021: Beeching et al., 2023) is widely-used benchmark suite that consists of static datasets such as Big Bench Hard (Suzgun et al., 2023) and MMLU-Pro (Wang et al., 2024). While this has been incredibly useful in tracking the performance of LLMs, its static nature leaves it prone to test set contamination by models.
LL-aa-a---- AlpacaEval (Li et al., 2023b; Dubois et al., 2023; 2024), MT-Bench (Chiang et al., 2024), and Arena-Hard (Li et al., 2024) are benchmarks that employ LLM judges on fixed set of questions. Using an LLM-as-a-judge is fast, relatively cheap, and has the ficxibility of being able to evaluate open-ended questions, inslruction-folloiinng questions, and chatbots. However, LLM judging also has downsides. First, LLMs have biases towards their own answers (Li et al., 2024). In addition, GPT-4 judges have noticeable difference in terms of variance and favorability of other models compared to Claude judges. Additionally, LLMS make errors. As one example, question 2 in Arena- Hard asks a model to write C++ program, yet GPT-4 incorrectly judges solution as incorrect (Li et al., 2024).
4-0314's
Humans-as-aa-udge. ChatBot Arena (Chiang et al., 2024; Zheng et al., 2024) leverages human prompting and feedback. Users ask questions and receive outputs of two randomly selected models and pick which output they prefer. This preference feedback is aggregated into an Elo score for each model. While human evaluation is great for capturing the preferences of crowd, using human-

---

Published as a conference paper at ICLR 2025
as-a-judge has disadvantages. First, human-judging can be aubor-intensive, especially for certain tasks included in LiveBench such as complex math, coding, or long-contex. reasoning problems. Whenever humans are involved in annotation (of which judging is sub-case), design choices or factors can cause high error rates (Lease, 201 I), and even well-designed human-annotation setups, high variability from human to human leads to unpredictable outcomes (Rashkin et al., 2023).
Other benchmarks. LiveCodeBench (Jain et al., 2024) also regularly releases new questions and makes use of ground-truth judging. However, it is limited to only coding tasks. The extensive Omni-MATH benchmark Gao & Liu (2024) encompasses numerous math competitions, although using LLM ja-dg-ud grading potentially contributes to a degree of contamination or bias in some of the benchmark's scores; our completely objective correctness- based scoring avoids this concern. The SEAL Benchmark (Scale AI, 2024), uses private questions with expert human scorers, however, the benchmark currently only contains the following categories: Math, Coding, Instruction Following, and Spanish. In Srivastava al. (2024), the authors modify the original MATH dataset (Hendrycks et al., 2021) by changing numbers in the problem setup. They find declines in model performance for some LLMs, including frontier ones. However, while such work can evaluate LLMs on data that is not in the pretraining set, the data still ends up being highly similar to the kind of data likely seen in the pretraining set. In addition, the hardness of the benchmark remains the same over time.
Finally, we discuss benchmarks that were the basis for tasks in Li LieeBhhh. In IFEval (Zhou al., 2023b), the authors assess how good LLMs are at following instructions by adding one or more constraints in the instruction as to what the output should be. They limit the set of constraints to those in which it can provably verified that the generation followed the constraints. In Big- Bench (Srivastava et al., 2022), a large number of tasks are aggregated into single benchmark with the aim of being as comprehensive as possible. Big-Bench- harc (Suzgun et al., 2022) investigates a subset of Big- Bench tasks that were particularly challenging for contemporancous models well as more complex prompting strategies for solving them.
CONCLUSIONS, LIMITATIONS, AND FUTURE WORK
In this work, we introduced LiveBench, an LLM benchmark designed to mitigate both test set contamination and the pitfalls of LLM judging and human crowdsourcing. LiveBench is the first benchmark that (1) contains frequently updated questions from new information sources, in which questions become harder over time. (2) scores answers automatically according to objective ground- truth values, without the use of LLM judges, and (3) contains a wide variety of challenging tasks, spanning math, coding, reasoning. language, instruction following, and data analysis. LiveBench contains questions that are based on recently released math competitions, arXiv papers, and datasets, and it contains harder, contamination -immted versions of previously released benchmarks. We released all questions, code, and model answers, and questions are added and updated on a monthly basis. We welcome community collaboration for expanding the benchmark tasks and models.
Limitations and Future Work. While we attempted to make LiveBench as diverse as possible, there are still additions from which it would benefit. For example, we hope to add non-English language tasks in the future. Furthermore, while ground truth scoring is beneficial in many ways, still cannot be used for certain cases, such as 'write travel guide to Hawaii' in which it is hard to define a ground truth. Finally, while we attempted to make all tasks and categories fair for all models, there are still biases due certain LLM families favoring certain prompt types. We plan update the prompts (at the start and end of each question) in the future, as new prompt strategics are developed. Similarly, we plan to continue updating the Li veBench leaderboard as new LLMs are released.
REPRODUCIBILITY STATEMENT
Our work is fully reproducible: we open-source the leaderboard, all questions, all code to run API and open-source models, all model outputs for 40 models, and all code to score the models. In other words, every part of the project is available publicly: https: : htw.vvebeecchhhaeem exception is that as the benchmark becomes more popular, we withhold releasing the new set of questions each month, that there are always some questions that private. These questions then made public one month later. The readmee in the above link gives instructions to download all parts of the project and to score models.
The only
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
Our paper introduces a new benchmark for LLMs, which contains frequentl-uupdated questions from new information sources, scores answers according objective ground-truth values, and contains wide variety of tasks. We do not see any inherently negative broader societal impacts of our work.
Our hope is that our work will have positive impact for both practitioners and researchers: by providing a new benchmark with frequently-updated questions, our work has the potential to both accelerate future research and enable more comprehensive and rigorous evaluations of existing and future models. Furthermore, we hope that the general framework of our benchmark frequently- updated questions with new information sources will catch on, mitigating the negative effects of contamination in future LLM evaluation and making LLM benchmarks more nuture-prooff.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14211 2024.
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. Nemotron-4 340b technical report. arXiv preprint 2rXiv:2406.1113 11704, 2024.
Al Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https: / //huggingface. co/ swwp/spacsss////eee http://pcce//uggggggaaaaee
let__mlmmeeeeeeeeeeeeeeeeeeeeeee wew.eeeberrrrrroo 2023.
BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of
language models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL
https::/ http::///peeeeeeeeeeeee .nt/ forum? i dd=uyLL5Bvos.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2303. 2023.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15(3):1-45 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023.
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang. Banghua Zhu, Michael Jordan, Joseph E Gonzalez, al. Chatbot arena: An open platform for evaluating lims by human preference. arXiv preprint arXiv:2403.04132. 2024.
Karl Cobbe, Vincet Kosaraju, Mohammad Bavarian, Mark Chen, Hcewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reïchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint a 10. /4/68, 2021.
Cohere. Command r: Retrieval auggmeete generation at production scale. https: //ttt. cohere. commacomanddee r, March 2024.
11

---

Published as a conference paper at ICLR 2025
Cohere For AL. c4ai c4ai-command---pus--0-----0000 2024. URL https: httpss//hhugiiigfa.e..oo
CoherePorAI www.aaaaaaanno-a--aaaaaae 2022..
DeepSeek-AI, Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi
Dengr, Chong Ruan, Damai Dai. Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin. Fuli Luo. Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Hanwei Xu, Hao Yang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L.. Cai, Jian Liang, Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jin Chen, Jingyang Yuan, Junjie Qiu, Junxiao Song, Kai Dong, Kaige Gao, Kang Guan, Lean Wang. Lecong Zhang. Lei Xu, Leyi Xia, Liang Zhao, Liyue Zhang, Meng Li, Miaojun Wang. Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li. Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruizhe Pan, Runxin Xu, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Size Zheng, T. Wang, Tian Pei. Tian Yuan, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Liu, Xin Xie, Xingkai Yu, Xinnan Song, Xinyi Zhou, Xinyu Yang, Xuan Lu, Xuecheng Su, Y. Wu, Y. K. Li, Y. X. Wei, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Zheng. Yichao Zhang, Yiliang Xiong. Yilong Zhao, Ying He, Ying Tang, Yishi Piao, Yixin Dong. Yixuan Tan, Yiyuan Liu, Yongji Wang, Yongqiang Guo, Yuchen Zhu, Yuduan Wang, Yuheng Zou, Yukun Zha, Yunxian Ma, Yuting Yan, Yuxiang You, Yuxuan Liu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhewen Hao, Zhihong Shao, Zhiniu Wen, Zhipeng Xu, Zhongyu Zhang, Zhuoshu Li, Zihan Wang, Zihui Gu, Zilin Li, and Ziwei Xie. Deepseck-v22 v2: A strong, economical, and efficient mixture- language model, 2024.
Chunyuan Deng, Yilun Zhao, Xiangru Tang. Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. arXiv preprint arXiv:2311.09783, 2023.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.00480 2018.
Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, and Ge Li. Generalization or memorization: Data con- tamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15933, 2024.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: simulation framework for methods that learn from human feedback, 2023.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint 0rXiv:2404.04475, 2024.
J Douglas Faires and David Wells. The Contest Problem Book VIII: American Mathematics Competi- tions (AMC 10) 2000-2007, volume 19. American Mathematical Society, 2022.
Benjamin Feuer, Yurong Liu, Chinmay Hegde, and Juliana Freire. Arche A Novel Framework for Open-Source Column Type Annotation using Large Language Models, October 2023. URL http:/ /tap::///rxvvvv wwwg/bbss.....oooo.ooe /2100///311.0 18208. 8. arXiv:2310. 18208 [cs].
Bofei Gao and Tianyu Liu. Omni-man mati- A universal olympiad level mathematic benchmark for large language models. https://omnimmatee wwtt.atttthhtat. github.io// 2024.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September 2021.
12

---

Published as a conference paper at ICLR 2025
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64( (1):: 86-92, 2021.
Shahriar Golchin and Mihai Surdeanu. Data contamination quiz: tool to detect and estimate contamination in large language models. arXiv preprint arXiv:2311.0022 2023a.
Shahriar Golchin and Mihai Surdeanu. Time travel in Ilms: Tracing data contamination in large language models. arXiv preprint arXiv:2308.08499 2023b..
Anthony Goldbloom. The overlooked genai case. https://bles wtw.lll..smmll....mmmm wwwbluuubee...... com/
Lho-oover-ooked-eenai---- case/ October 2024. Accessed: 2024-11-23.
Guardian Media Group. The httpp:////w...eguurraiaaa.oo.mc com/, 1821. Accessed: 2024-01-20.
https://wwe www.whrdiaaaaaannnmmmm
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 1rxiv:2103.03874, 2021.
Aaron Hurst, Adam Lever, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint crXiv:2410.212770 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024.
S Jeremy. Einstein's 's riddle: Riddles, paradoxes, and conundrums to stretch your mind. Bloomsbury USA, pp. 10-11, 2009.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Vishakha Suresh Kalal, Andrew Parry, and Sean MacAvaney. Training on the test model: Contamina- tion in ranking distillation. arXiv preprint arXiv:241 1.002244 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurlPS), 2022.
Matthew Lease. On quality control and machine learning in crowdsourcing. In Workshops at the twenty-fifth AAAI conference on artificial intelligence. 2011.
Vladimir I. Levenshtein. Binary codes capable of correcting deletions, insertions. and reversals. Soviet Physics Doklady, 10(8):707-710, 1966.
Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, al. Starcoder: may the source be with you! arXiv preprint 2rXiv:2305.06661, 2023a.
Tianle Li, Wei- Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL
https: / 1msys o www.ooogllooooooooooooooo 220/222222000 14-10--1000 1944-1--40 aaraa-aaaaaaraa
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovinn models. https:/ github.....
aataaaapaaaaa__eaa eval, 2023b.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chenggi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseck-v2: strong, economical, and efficient mixture-of- experts language model. arXiv preprint arXiv: 24444444444 2024a.
13

---

Published as a conference paper at ICLR 2025
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.
Yurong Liu, Aécio Santos, Eduardo HM Pena, Roque Lopez, Eden Wu, and Juliana Freire. Enhancing biomedical schema matching with llm-based training data generation. In NeurlPS Third Table Representation Learning Workshop, 2024b.
Meta, Introducing meta Ilama 3: The most capable openly available IIm to date. https: //ai.
meta. www.///////o/aaaa..o//eooooo.aaa -ateaalllann llama- 3/, April 2024. Accessed: June 4, 2024.
Aaron Meurer, Christopher Smith, Mateusz Paprocki, Ondrej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMIT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, January 2017. ISSN 2376-5992. doi: 103.
10.77177peerj-cc.100. URL httph:tppst..m /tcp:p:///oo../ org 10. 7717/peer/
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- forcement learning. Nature, 518(7540):529-533 February 2015. ISSN 00280836. URL http://dx doi.doioooooooooo wwt./o..g/000....ooo... http//nauuee44220
OpenAL. Gpt-4 technical report. Technical Report, 2023.
Yonatan Oren, Nicole Meister, Niladri Chatterji, Faisal Ladhak, and Tatsunori B Hashimoto. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17625, 2023.
quint t. Puzzle generator and puzzle solver. htttp:///gtthu..cmmm//
Puzz eellle--eeeee Be-Generator---o and-So Sol ver, 2023.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Tulia Ture, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777-840, 2023.
John W. Rateliff and David E. Metzener. Pattern matching: The gestalt approach. Dr. Dobb's 's Journal, pp. 46, 1988.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprins arXiv:2403.05530, 2024.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. To the cutoff... and beyond? longitudinal perspective on Ilm data contamination. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
Scale AL. Seal leaderboards. sttpss//scaaee.cmm/Leeeeeee
May 2024.
Eitam Sheetrit, Menachem Brief, Moshik Mishaeli, and Oren Flisha. Rematch: Retrieval enhanced schema matching with llms. arXiv preprint arX/v:2403.0//66, 2024.
Aaditya K Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in 1lms: how do we measure it and (when) does it matter? arXiv preprint arXiv:241 1.03923, 2024.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alonss et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206. 04615, 2022.
14

---

Published as a conference paper at ICLR 2025
Saurabh Srivastawa, Anto PV, Shashank Menon, Ajay Sukumar, Alan Philipose, Stevin Prince, Sooraj Thomas, et al. Functional benchmarks for robust evaluation of reasoning performance, and the reasoning gap. arXiv preprint arXiv:2402. 19450, 2024.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprini arXiv:2210.09261. 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 13003-13051, 2023.
AtCoder Team. Atcoder. https:/attcoderrjp/.220 2012.
Gemma Team. Gemma, 2024a. URL https: /www.wwwwooooom www..aaggle..em www/m///////ooo// 3301.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size. arXiv preprint arXivv2408.00116. 2024.
LeetCode Team. Leetcode. https://lettcode.com/// 2015.
Python 3 Team. difflib. https://dbcs.
html, 2008.
Qwen Team, Qwen2.5: A party of foundation models, September 2024b. URL httpt:t////// //wwn/m.
github. folbubi..//lo/// http:////...ooo///////./.mm qwen2.
Graham Todd, Tim Merino, Sam Earle, and Julian Togelius. Missed connections: Lateral thinking puzzles for large language models, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:23007.09288 2023.
Lewis Tunstall, Edward Beeching. Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation of Im alignment, 2023.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint 24Xiv:2406.01574 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of thouggt prompting elicits reasoning large language models. Proceedings of the Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Cong Yan and Yeye He. Auto-Suggest: Learning-tor Sarming-to-Recoomment Recommend Data Preparation Steps Using Data Science Notebooks. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, pp. 1539-1554, Portland OR USA, June 2020. ACM. ISBN 978-1-4503- 6735-6. doi: 10. 011111/3318844.33897777 3318464.3388 3389738.
URL https: /dl
acm wttg//o///ee 1145/
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint ar0iv:2407. 10671, 2024.
15

---

Published as a conference paper at ICLR 2025
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Proceedings of the Annual Conference Neural Information Processing Systems (NeurIPS), 36, 2024.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, et al. A careful examination of large language model performance on grade school arithmetic. arXiv preprint arXIv:2405.00332, 2024.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Prucessing Systems, 36, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint 1rXiv:2311.07111 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. estruction-followinng evaluation for large language models. arXiv preprint 1:X1v12311.0791,, 2023b.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b. Improving Ilm helpfulness & harmlessness with rlaif, November 2023.
16

---

Published as a conference paper at ICLR 2025
LiveBench
01-0roviww22224-00-2
Baude335--o0
01 21-mini-2024002 09.12
geminr-exp-1121
2et-4o-2024-8--00
1.poni-15-pro00022
opt-4o-2024-05-13
gemini.)
S-pro-xxp-08277
meta-lama-3
101--o-2024-11-22
dracarys2 722 instruct
chatopt 4e 4n-tatest 8903
gpt-4 gpl---urbo-2024--000 2oturbbe-224--000
(Bu00e3--pp--2224022
gemini-1.3 1.5-4lshhhh000 002
mistral large- 2407
qwen2. insttuuubnnstuuutt
gpt-4-0125.previen
gemin
1--pmi--1---p-0001
meta-llama- 3. 1-70b-instruud-turbb
claude-3 1ou-dd------lll0- 2-5-mallo.22241172
gpt-4o-min 20--0mni-2024407-11
gemmm--2-27be
gemini 1.5 flss 8b cxp 8827
qwcn2.5 ih-iinstrucc-tg turbo
daudde33ha 2audde-3/akku0222440000
mixtral-8x2
commandd 08 2024
gemima 2-9bit
mistral-small-2400
command---0-2072
command-riglss-n
meta-llame-3 6bbrisstuut-ture
API Open Source
phi 3-mmnn-112k--in
phl mini instrrcc
Figure 5: Results on LiveBench for all models, showing 95% bootstrap confidence intervals. This is the full version of Figure (left).
Spr asis51g
10:44003-pseee predieg
1199
patsians Intruct
1189
2053
3on3 1126
2093
492
1.5
40
Ursturch Score
Score
Figure 6: The performance of models on different benchmarks, compared to best-fit line. We compare the different in relative performance of LLMs on LiveBench vs. ChatBot Arena, and LiveBench Arena-Hard. We that while many models near the best-fit lines, few notable outliers, providing evidence that their output style may be noticeably better or worse than their ability answer questions.
A ADDITIONAL DETAILS ABOUT LIVEBENCH EXPERIMENTS
In this section, we detail further descriptions about the LiveBench benchmark itself and our experiments.
We include further depictions of the comparisons LiveBench to ChatBot Arena and Arena-Hard in Figure 6. We display the full results table for LiveBench in Table 2. We display the full bar plot for LiveBench Figure 5.
We display the list of all verifiable instructions in Table 3.
We display a table with the citations for all models Table
17

---

Published as a conference paper at ICLR 2025
Table LiveBench results across 40 models. We output the results for each model on each main category, well as each model's overall LiveBench
Model
LiveBench Score
Coding
Instruction Following
Language
Math
Reasoning
Analysis
ol Nneview-2024-09-12
64.7
50.8
64.0
74.6
68.7
62.9
67.4
2uadd=3-S-somnee22224-0000
58.5
67.1
52.8
69.3
53.8
51.3
56.7
sluube-355-ummme2222200000.
ol-mini-2024-00-12
56.7
48.1
54.1
65.4
40.9
59.2
72.3
gemini-exp-1114
56.0
52.4
57.5
77.1
38.7
54,9
55.7
gemini-cxp-112)
56.0
50.4
$7.0
80.1
40.0
62.8
stop-2-16k-202411
55.1
46,9
54.9
79.9
44.5
48.9
55.5
gpt-lo-2024-08-06
53.8
51.4
52.9
68.6
47.6
48.2
53.9
gemini-1.5-pro0002
53.4
48.8
52.3
70.8
43.3
57.4
47.9
gpl-4--2024-05-13
geminill5-pno-exxp00277
52.4
40,9
50.8
69.3
46.1
56.1
50,9
matellaama:3114400--mmmmmma
51.1
43.8
53.5
72.8
43.2
40.5
52.8
gpt-lo-2024-1--20
50.6
47.2
64.9
47.4
42.5
55.7
drucarys2-72--instnuet
50.1
56.6
65.2
33.5
50.6
45.8
chatgpt-4o-1aeess00033
50.1
47.4
48.7
66.4
45.3
42.1
50.5
gpi-+-urbo-2024-04-099
49.6
49.0
51.3
60.8
44.3
42.7
49.5
claude-3-qqus-222242229
48.4
38.6
gemmnn-1.5-0ash-002
47.6
41,9
44.2
78.8
27.9
47,2
45.7
mistral-large-2407
47.0
47.1
46.6
63.7
39.5
43.7
41.6
(wee225-ccodrr322biintuuu
45.0
56.8
43.4
58.7
23.2
45.9
42.1
2p1-4-0125-previee
44.8
41.8
54.1
57.2
39.2
33.4
43.2
geminn-l.5-laas--exp-0822
44.1
40.6
47.9
73.0
29.6
28.9
44.5
gemini-1.5-pro0001
43.3
32.3
52.8
60.2
40.4
36.9
37.0
43.2
32.7
69.7
34.4
37.9
chanbe3---haika-222241022
42.4
51.4
424
61.9
35.4
35.5
28.1
gPt-lo-mini-2024-07-18
40.2
43.2
44.5
56,8
28.6
35.6
32.8
gemini-1.5-fashh-00
34.3
44.0
52.6
32.3
gcmma-2-276-it
36.4
35,9
58.1
32.6
26.2
22.1
36.1
28.7
35.3
70,0
20.8
27.8
33.8
(wenn..577-inssruuttte
33.4
37.9
32.8
$1.0
14.6
38.2
26.2
claubb33Huakk-222444377
33.2
24.5
55,3
29.1
22.9
25.9
mixtral-8xx22biisttreee
31.2
32.0
31.7
52.3
21.8
24.5
24.9
command-r-plus-882202
31.1
19.5
35.9
57.6
29.7
19.3
24.8
gemma-2-99bit.
28.3
19.5
mistral-smull-2400
27.5
21.2
31.9
56.4
18.5
command-r-08-202
27.0
17.9
31.3
55.6
16.7
19.5
21.0
command-rrplus-04-
26.6
19.5
24.6
59.5
19.7
16.8
19.8
Setta-lama-331----innstt
25.0
19.7
32.2
51.5
16.6
12.2
21.9
15.0
34,0
39.1
9.2
14.6
19.6
Mhi---mini-4l-insttee
21.1
15.0
29.5
36.4
8.6
15.0
22.2
18

---

Published as a conference paper at ICLR 2025
Table 3: The list of 25 instructions used in (Zhou al., 2023a), and the 16 that are both 'real-world' and automatically verifiable, which we used in LiveBench. Descriptions are from (Zhou et al., 2023a).
Instruction Group
Instruetion
Doseription
In IFEval
In Livellench
Keywords
Include weeds
Key-
Include keywords (keywordd) (keyword)) in your response
Kaywark
Keyword quelcyy
Fre-
respurse
should appear [N] times.
Keywords
keywords (foebudden
response.
1
1
Keywoods
Leteer Frequency
En your resparse, the letter [lette]] should uppear [N] times.
1
Language
Response
Las-
You ENTIRE cesporse should be in [laggugee), no other language is allowed.
Length Constraints
Number
Para-
Your response should custain (N) pangraphs, You pangraphs using the markdown diiider
priphs
Leagth Constraints
Number Words
around/ (N) weeds
Length Cunatraintx
Numther
Sca-
Ansoeet loaa anound at most (N) antionsss
lances
Length Constraines
Number Para
There should be (N) paragnehs. Pangraphs and onl paragraphs are separated with each other by two line breaks. The (i) th pangraph must stont with word (fimr, word).
gnphss
Fins 1-th
Word in
Paragraph
Desectalle Conteni
Possscipp
the end of your response, please explicily add possecipe staning with ppossetipt macker)
Detoctabl: Contenn
Number
Place-
The requime must uomain least (N) placcheldrs repersented by squac brnckets, such to (address).
holder
Detectable Fermar
Number Bullee
Your must contain etacily (N) builet points. Use markdown bullet such ax This
Detectall: Forns
wappeed duuble uugular brackets, <pop
Desectalle Fonnst
Choose From
following optioss: [uptinss]
Deseciable Fermat
Minimum Num ber llighlighoed Seeiion
Hlghlight at least (N) sections your answer with markdown, Le. "highlighted section*
/
Datectable Forns
Muliiji:Seciauu Secious
Your resganee must lave [N) scctions. Mark the lagiming of mach seciuss with (sec- tior_splilte]] X.
Desectable Fermor
JSON Pirmat
Entire curput should be wrapped in ISON formar.
Combination
Prompt
First,
without
need your (do anything befice
repexting the request the
include this
Combinotion
Resporscs
diffesert roquesees. Respensus only respomass should suparated by avtrrikk symboll cassas
Clange
Uppersse
nopuiee shoold Fnglish, ropital only.
Change Cases
All Loverease
Your entire response should English, and in all Jow weeeuuse letters No capital letters are allowed.
Change Cases
Freguency of All- rppinn Words
la your respouse, words with all cupital letiers should appear at least/ around at most IN] limes.
/
Stari with / End with
Ead Checker
Finish your reporse with this exert phons (end phrose), No other words should follow this please.
/
Start with / End with
Quotation
Wrpp your entire respotse with double quotation marks.
Punctustion
No Coumans
your scaponec,
19

---

Published as a conference paper at ICLR 2025
Table 4: List of models evaluated (across all LiveBench versions) and their respective citations.
Modil
chxtcpt-foo-etee=0000
[lurrt et 2021)
claude-3-35 1aiil------ak---
3-5-haakk--22222122
bitps: //0::///o0 http:///w/n//aaaaa.e anthaappie opw..pp..................... www.e/cooccaaoeeeeooeeeoeooe.eee waw.aa//aaahaaa.
https:/ /ww/:://wwww... /ww.. /ttpw//www...p.peeeoe// www.raphhhpee apic.ppc...... w...oo/neexeeee con//exx/claaae catea//caadd--a 3--dd------ 3-3-sonnee
claude-3----ommee=22111-
https: Prtps://mnt wwwpr.ntthrr....
won/nexs/cloude--ee
(Anthropic, 2024)
claude-3-999--22244225
LAnthoopcc 2024)
(Anthropic. 2024)
command-r
[Calere, 2024)
coman0---00-2022
[Cohere, 2024)
(Cobers. 2024)
c0mman6-E-p21x-00-2222
desseea--oode--22
(eepScck-Aletta.22222
ct..
dospoost-22-ito-chhe
dt 2024)
despaeek-v2.3
ctepSeed-AA 2024)
dracarya-725-ins--
dbacarya2-72b-
httpe://uaggnngaaaa.coo htt///aaaus//bbaaaaaaee Ina:ruct.
dracarya--11ama-3..
instructt
Saaaaaayys------a--aea 12ana-3.1-72 -aaaa-3---BB--atrr Ioattrtt
0cmint-1.s-fiazh-000
al.
(Reid 2022)
[Rzid al.,
a0.,
0cmin--1-5-pro-002
[Reid et 2024)
[Roid et al., 2014)
(Raid et 202,4
(Reid 2022)
exp-ii-es--111
bt ht.lpas//s..goog....
/gwninl--nnemmmm app. dooss /madel expes inental models
gemini-axp-1121
https://ai..o //tp//gpggieeeeeo///// dew. http///ennnlaaae saainiillappppp. doos /mode expes modela
gema-1.11-7b-s
[Tam, 2024a)
geme 2-2b---27b-11
[Tram al., 2024)
06ma-222b
[Team al., 2024))
0cmma-2-3b-4-
[Team 2022))
2024)
opt-3.5-tutto-012-
9pt-3.5-turbo-1106
(Brown 2020)
opt-4-0125-previen
(OpenAl, 2023)
gpt-4-0613
[OpenAl, 2023)
gpt-4-1106-prewt
prevvie
(OpenAl, 2025)
opt-4-turb-2222--4-o9
[OpenAl, 2025)
gpt-fo-202Q-0f-13
[llustt
gpt 2024
(Hurst al., 2004)
1pt-4o-2024-11-20
(Hurst al. 2024)
3pt-1o-rinn-2222-0--8
[Hurst at., 2022
grok
https://e..t htt.:///bl////o///
grok-2-nini
fttps://x.aa/bloo.
i1ama-2-Doccht-br
[Tanvron et 2023)
3. ror cob-inatuud
[Adllr et 2014)
neta-llana-3-1
1-00aaa---0a-naaaaa inatrect
(Meta, 2024)
11t-11mma---B--iaazce
(Mem,
(Dubey al2,
3.1-700-
(Dubey
neta-1laa-331-90-inBB -tuboo
(Ubey al.,
alsteal-To-iastzuet--
(Jang al., 2023)
nistral 1octal-----nne
1isttal--h-instton
in-iinstrutt-v..
(Jing el., 2023)
-iataal-1arge-2622
(Jang at., 2023)
pistral-1azoe-2601
(Jing al.. 2023)
nistral sittral-mxxl-2407
(Jung
Mostaal-3222b-inatraaom
tJingg at., 2023)
UJang al., 2023)
11-rini-2024-9--1
https: sttps://opend: .con/t iidex Copenaiie ayst rd7
ol-previee 2024 09--009-33 09-12
htlpps://ppanal www.//oo////nie Lnden /openil
2- ol eyyatennon d/
opan-niatsal-Fam
[Jing et.., 2023)
(Abdin 202,
phi mhi-3-mini-W 128k Lratrust
[Abdin el., 2034)
phi-3 1133-sna11-122--naacaee
(Abdin et al.,
Mal-3.E-nini-Lnzzee
(Abdin et 2024)
phi-3.5 noo-insttiit
(Abdin et al. 2034)
gven1.5-0. B.b.b-h..bbccc
[Bai at, 2023)
gvan1.E-72b-chan
202))
quen1_5-1 -1-1--00---hh
(Bai at,, 2025)
quen1.5 1on1.6-7b-ohat
(Bai et. 2025)
gven2-0.5b- S.t--..bb-iisteeo
(Yáng at.l. 2024)
quen2-1.5m in5b-inat ruct
(Yang at. 2024)
quen2
(Yang
gwen2-76- 22s
(Yang 2024)
5-72b-irsttruc instrut
[Tam, 2024h)
quen2. 5-77b-inee ruct ruct-tuubo
[Tam, 2024h)
starling- -am---1m-----ee beta
(Zhu et etal., 2023)
scag-2-16k-20220
http:///w//ppptpooee con/ #step2
crana-76-v1.6
(Chineg
vieuna 15k
[Chiang et al,,, 2023)
yi-Eb-chat
https t basiogiggaao..cooo -11VV16-
1apayr-7--alha
[Tonstall at, 2023)
sephyr 76 2e54
(Tonstall et. 2023)
20

---

Published as a conference paper at ICLR 2025
Table 5: Pearson correlation coefficient and std. error for each category compared to the overall average LiveBench score, computed using data from all 40 models.
Category
Correlation
Std Error
math
0.9477
0.0643
reasoning
0,9439
0.0709
data_analysis
0,9315
0.0489
coding
0,8970
0.0840
language
0.8970
0.0831
instruction foolowwig
0.8174
0.0811
Table 6: Pearson correlation coefficient and std. error for each task compared to the overall average Li veBench score, computed using data from all 40 models.
Category
Correlation
Std Error
web of lies_v; v2 math comp tablejoin
0.9136
0.1454
0.9035
0,0986
0.8984
0.0961
zebra_puzzle compig_completion
0.8853
0.0935
0.8623
0,1212
cta
0.8556
0.0486
plot_ uos_usscramblin
0.8553
0.0881
LCB_generation
0.8550
0.0816
connections
0.8368
0.1188
AMPS_Han _Hrd
0.8245
0.1357
olympiad
0.8192
0.1171
summarize
0.8044
0.0904
paraphrase
0.7884
0.0953
simplify
0.7765
0.0829
typos
0.7722
0.1474
story_generation
0.7443
0.1018
spatial
0.7294
0.0968
tablereformat
0.6840
0.1058
A.1 DETAILS FROM CORRELATION ANALYSES
Here, provide more details from Section 3.2. First, present the Pearson correlation coefficient and std. error for each category (Table 5) and task (Table 6) compared to the overall average LiveBench score, computed using data from all 40 models. This is supplement to Figure 2. In Table 7, we compute the relative best and worst task for each model, specifically, the tasks with the highest and lowest residuals of the best fit line overall LiveBench performance. In other words, we compute the task that each model most outperforms and underperforms on, relative to theoretical model with the same overall performance but has balanced performance across each task.
A.2 DETAILS FROM ABLATION STUDIES
In this section, we ive the details for the ablation study described in Section 3.3. For hard math and reasoning questions, if an LLM struggles answer the question, then will it also struggle to determine whether or not a given answer to that question is correct? There are some classes of problems for which the answer is surely 'no': any problems that are hard to solve by frontier L.LMs, yet easy to check whether an answer is correct or not, such as NP-Hard problems. Another exception is that if an LLM judge is given access to the ground truth, then it will (of course) be able to judge whether or not answers are correct. The tasks in our original experiments (AMC, AIME, and Zebra puzzles) may not fit the class of exceptions. By way of a preliminary study of the above suggestion (that LLM judges cannot judge Zebra puzzles and AMC/AIME questions that they cannot solve), run an experimental test. We use judge prompt based on the MT Bench judge prompt, which is duplicated below.
21

---

Published as a conference paper at ICLR 2025
Table 7: Relative best and worst task for each model, computed as the tasks with the highest and lowest residuals of the best fit line vs. overall Li veBench performance, for each model.
Model
Best
Worst
2l-preview-20224009112
connections
coding_ completion
claude-3-5-somee--22244000
Doding_completion
web_of_lies_t math_comp
cladde-33--someee222444000
olympiad
01-mini-20244-09-12
zebra puzzle
reding_completion typos
gemini-cxp-1114
AMPS Hard
gemini-exp-1121
gory_generation
web of _f__iee_________ v2 LCB LCB__eneratiion
step-2-16k-2024111
paraphrase
$pt-40-2024-08-066
spatial
web_of le__oflles_____
gemini-1.5-pro-002
olympiad
web_of_lies_v2
3pt-4o-20224-05-13
typos
tablejoin
gemin--..ypro--xp-0022
olympiad web of lies
LCB ge__geeerttion
connections
gpt-4o-2022-11-200
typos
olympiad
dracarys2-72b-instruet
Loding_completinn
typos
chatept-4o--atees--9933
spatial
olympiad
gpI-t-urrbo-20244-04-99
connections
simplify
claude33-opus-22240222
typos
web_of_liest
gemini-1.5-ftash0002
summarize
typos
mistral-large-24077
olympiad
tablejoin
qvenn255cooder-22-iinnttutt
LCB_ _ee__eenration
typos
gpt-4-0125-preview
tablereformat
olympiad
gomini-L.5-ffash--xp-0822
web_ brb_ooo____ [ielles___ v2
AMPS _ard
gemini-1.5-pro-0001
typos
web web_of of_oo_lless lies v2 pebra_puzze web web_of lies lies_v2 of olympiad seb_of_lies____
nett:llamaa33-700-issstcutuuuuu
tablereformat
Cladde335Shaiikk222241122
coding__oom completionottinnn
gpt-4o-mimi-2022-077118
AMPS_Hard
gemini-1.5-ffash-0001
typos
gemma-2-27b-it
tablereformat
web_ lob_of_lie__vee
paraphrase
plot_unscraan unscrambling typos
qwven2.5-7--nnssuuutttubbob
AMPS Hard
clanee-3hhaiku-202440377
typos
web_of ceb_of_lie_____ [i___ies_
mixxtrl--x222---nstrur---11
Doding_complette
tableneformat
command-rpllus-08-2222
typos
gemma-2-9b-it
typos
spatial
mistral-small-24002
web_of of_lies___ lies_v2 v2
zebra_puzzle
command-r-08-2024
paraphrase
AMPS Hard
conmand-rpplus-04-2024
simplify
tablereformat
typos
spatial
phi-3-mini-128k-instuu
spatial
pory_geeeration
phi-3-mini-4k-instuu
spatial
simplify
[Instruction] Please act an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should consider correctness alone. Identify and correct any mistakes. Be objective as possible. After providing your explanation, you must rate the response as either (correct) (incorrect) by strictly following this format: "[[rating]]", for example: "Rating: [[I]]" [Question] question [The Start of Assistant's Answer] answer [The End of Assistant's Answer]
We use gpt-4-turbo-2024-002 gpt-4-tubbo-222--44000 and c-laude3-oppus202-
as the judge. We judge the model outputs of both
See Table 8 and Table 9. We find that the eπror rate for all tasks is far above a reasonable value, indicating that LLM judges are not appropriate for challenging math and logic tasks. However,
22

---

Published as a conference paper at ICLR 2025
Table 8: LLM judges cannot accurately evaluate challenging math and reasoning questions.
Error rate of LLM-as-aajudge scoring on challenging math (AMC, AIME, SMC) and reasoning (Zebra puzzles) tasks. On all tasks, the error rate surprisingly high, showing that LLMs are not reliable judges for these tasks.
Model
Judge
AMC12 2024
AIME 2024
SMC 2023
Zebra Puzzles
GPT-4-Turbo GPT-4-Turbo
0.380
0.214
0.353
0.420
Claude-3-Opus
GPT-4-Turbo
0.388
0.103
0.294
0,460
Table 9: Model Performance on math and reasoning tasks with both ground-truth (GT) or LLM judging (LLM-Jdg.)
AMC12 VICT2220 2024
AIME 2024
SMC 2023
Zebra Puzzles
LLM-Jdg.
GT
LLM-Jdg.
GT LLM-Jdg.
GT
LLM-Jdg.
GPI-4-Turbo
54
64,000
13.793
35.714
70.588
58.824
38
68
Claude-3-Opus
56
42.857
6.897
17.241
58.824
52.941
34
52
note that there may be other experimental setups which could change the result, such as using more detailed prompt that is tailored to the task of judging hard math and reasoning problems.
A.3 DETAILED DESCRIPTION LIVEBENCH CATEGORIES
In this section, we describe the categories and tasks of LiveBench and the grading methods in more detail.
A.3.1 MATH CATEGORY
Evaluating the mathematical abilities of LLMs has been one of the cornerstones of recent research in LLMs, featuring prominently in many releases and reports (Reid et al., 2024; OpenAI, 2023; Brown et al., 2020; Bubeck et al., 2023). Our benchmark includes math questions of three types: modified questions from recent high school math competitions, proof-based USAMO and IMO problems, and questions from our new, harder version of the AMPS dataset (Hendrycks et al., 2021).
blank questions from recent
Math competitions. Our first math category is based on expert human-designed math problems that offer wider variety in terms of problem type and solution technique. We focus on high school math competition questions from seglishsspeeaing countries: AMC12, AIME, SMC, and USAMO, and also IMO, the international competition.
First, we include questions based on the American Mathematics Competition 12 (AMC12), both AMCI2A and AMCI2B 2023, released November 8, 2023 and November 14, 2023, respectively, and the Senior Mathematical Challenge (SMC) 2023, released on October 3, 2023. All three are challenging multiple-choice competitions for high school students the USA (AMC) and UK (SMC) that build in difficulty, meant the first step for high school students to qualify for their country's team for the International Mathematical Olympiad (IMO).
The questions test mathematical problem solving with arithmetic, algebra, counting, geometry, number theory, probability, and other secondary school math topics (Faires Wells, 2022). We modify the questions by updating the prose of the questions that do not affect the answer, by rearranging the order of the multiple choice answers when applicable, and by asking for a different output format than the widely-used source website (attps: aaraarrorpoooooooo emllovllviii ving. vinsovvingnng An example of a problem of this type from the AMC12A 2023 problem set is below:
com/).
23

---

Published as a conference paper at ICLR 2025
An example question from the Math Competitions task.
How many complex numbers satisfy the equation 2² where is the conjugate of the complex number x? (A) 2 (B) 3 (C)5 (D) (E) 7
If you cannot determine the correct multipee-choiee answer, take your best guess. Once you have your answer, please duplicate that letter five times in a single string. For example, if the answer F, then write FFFFF.
Ground Truth: BEEEE
Next, we include the American Invitational Mathematics Examination (AIME). both AIME and AIME II 2024, released January 31, 2024 and February 7, 2024, respectively. These are prestigious and challenging tests given to those who rank in the top 5% of the AMC. Each question's 's answer is an integer from 000 999. An example of problem of this type from the AIME 2024 problem set is below:
An example question from the Math Competitions task.
Real numbers v and y with z, > satisfy log,, (y") = (og,(("'') = 10. What is the value of zy? Please think step by step, and then display the answer at the very end of your response. The answer is an integer consisting of exactly digits (including leading zeros), ranging from 000 to 999, inclusive. For example, the answer might be 068 972. you cannot determine the correct answer, take your best guess. Remember to have the three digits as the last part of the response.
Ground Truth: 025
Proof-based questions. We consider the USA Math Olympiad (USAMO) 2024 and International Math Olympiad (IMO) 2024 competitions, released on March 20, 2024 and July 22, 2024, respectively. These contests are primarily proof-based and non-trivial to evaluate in automated way. One possibility is to use LLMs to evaluate the correctness of the natural language proof. However, we then have no formal guarantees on the correctness of the evaluation. Another possibility is to auto-formalize the proofs into a formal language such as Lean and then run a proof checker. However, while there have been notable recent improvements in auto-formalizzun have formal guarantees on the correctness of the auto nuto-formalization and thus that of the evaluation. To tackle this, formulate a novel task which can test the ability of an LLM in the context of proofs. Specifically, for a proof, we mask out a subset of the formulae in the proof. We then present the masked out formulae in a scrambled order to the LLM and ask to reinsert the formulae in the correct positions. Such a task tests the mathematical, deductive, and instruction following abilities of the LLM. In particular, the LLM strong enough to generate the correct proof for a question, then one would expect it to also solve the far easier task of completing proof which has some missing formulae - especially if the formulae are already given to it scrambled order. Note that this also allows us to easily control the level of difficulty of the question by changing the number of formulae that we mask.
such a process still does not
We generate 3 hardness variants for each problem, masking out 10%, 50% and 80% of the equations in the proof. We evaluate by computing the edit distance between the ground truth ranking order and the model predicted ranking order. [NB in preliminary testing also evaluated using the accuracy metric and the model rankings remained nearly the same]. Models perform worse on IMO compared to USAMO, in line with expectations. We also looked at the performance as separated by question hardness. The scores are greatly affected by question hardness going from as high as 96.8 for the easiest questions (10% masked out, GPT-4oo) to low 36 for the hardest (80% masked out). The full results are in Table 10 and Table 11."
Synthetically generated math questions. Finally, we release synthetic generated math questions. This technique is inspired from math question generation used to create the MATH and AMPS datasets (Hendrycks et al., 2021). particular, we randomly generate a math problem of one of several types, such as taking the derivative or integral of a function, completing the square, or factoring a polynomial. We generate questions by drawing random primitives, using a larger (and therefore more challenging) distribution than AMPS. Note that, for problem types such integration,
Note that these experiments were run in June 2024, before models such as claude-3.5 3.5-nnnnettt were released.
24

---

Published as a conference paper at ICLR 2025
Table 10: IMO/USAMO results for each of 34 models across all hardness levels.
Model
IMO
USAMO
Avg.
gpt-4o-2024-05-13
60.24
67.47
63.85
gpt-4-1106-preview
58.16
67.17
62.66
claude33-opus-22240222
52.56
63.66
58.11
gp1-4-turbo-2024--44-099
50.96
64.80
57.88
gemini-L.5-pro-lates:
52.11
59.15
55.63
gpt-4-0115-previiew
43.04
60.66
51.85
Mena--lamaa33770--nnttrtutt
43.24
59.55
51.40
chudee33ssomnet-22240029
44.78
52.97
48.87
command-r-plus
48.33
44.55
46.44
gpt-3.5-turbo-11066
40.37
49.65
45.01
mistral-large-2402
38.65
50.41
44.53
claude-3-haiku-200
claude-3-baikuu2222400077
41.51
47.31
44.41
gpt-3.5-turbo-01225
38.44
47.17
42.80
Qwenl.5-72B-
34.35
48.47
41.41
Mixtral-8x22B3
33,00
48.62
40.81
mistral-small-2444
34.51
44.78
39.64
36.05
36.59
36.32
Qwen 1
23.93
46.78
35.35
Mistral-7B-Instrucc-0022
36.00
34.31
35.15
command-r
31.36
29.38
30.37
Phi-3-minn-112k-instnnn
25.84
33.54
29.69
26.52
32.50
29.51
Phi-3-mini-44-instrus
26.60
30.33
28.46
Qwen 5-7B-Chat
22.10
31.84
26.97
Starling-L.M-7B-bett
14.99
28.70
21.84
zephyr-7b-alpha
25.99
16.43
21.21
vicuma-7b-vi.5-16k
23.14
16.69
19.91
Yi-6B-Chat
18.17
20.05
19.11
zephyr-7b-beta
9.57
22.57
16.07
Llama- 2aama-2-7b-cha-hh
20.00
11.53
15.77
Qwen 1. 15-4BBChht
11.90
16.78
14.34
vicuna-7b-vl.5
16.19
9.87
13.03
Dwen1.5-0.55 55-0.5B--hha
9.27
10.61
9.94
Owen1.5-1.8B-O
0.98
9.13
5.06
Table 11: IMO/USAMO results for each hardness level across 34 models.
Hardness level
Avg.
USAMO IMO
Easy
57.48
54.68
60.27
Medium
29.60
25.79
33.41
Hard
19.11
15.96
22.25
this simple technique of drawing random function and taking derivative results wide variety of integration problems of varying difficulty. For example, problem solutions may involve applying the chain rule, the product/quotient rule, trigonometric identities, or use change of variables. In order to extract the answer, ask the model to use the same 'latex boxed answer' technique in the MATH dataset (Hendrycks et al., 2021). We judge the correctness of answers as in the EleutherAl Eval Harness (Gao et al., 2021) using Sympy (Meurer et al., 2017) where check for semantic as well as numerical equivalence of mathematical expressions. An example of a integral problem is as follows:
25

---

Published as a conference paper at ICLR 2025
An example question from the AMPS Hard task. Find an indefinite integral (which can vary by a constant) of the following function: sec (5x + 1) 8 sin(7 8x). Please put your final answer a bo
Ground Truth: (in( sin(8x) - cos(7) cos(8w) lan(5z + 1)
A.3.2 CODING CATEGORY
The coding ability of LLMs is one of the most widely studied and sought-after skills for LLMs (Mnih et al.. 2015; Jain et al., 2024; Li et al., 2023a). We include two coding tasks in LiveBench: modified version of the code generation task from LiveCodeBench (Jain et al., 2024), and novel code completion task combining LiveCodeBench problems with partial solutions collected from GitHub sources. Examples of questions from the Coding tasks can be found here.
Code generation. In the LCB Generation task, assess a model's ability parse com- petition coding question statement and write a correct answer. LiveCodeBench (Jain et al., 2024) included several tasks to assess the coding capabilities of large language models. We have taken 78 randomly selected problems from the April 2024 release of LiveCodeBench, selecting only problems released in or after November 2023. The problems are competition programming problems from LeetCode (Team, 2015) and ArCoder(Team, 2012), defined with textual description and solved by writing full programs in Python code.
These problems are presented as in LiveCodeBenchh's Code Generation task, with minor prompting differences and with only one chance at generating a correct solution per question, per model. We report pass @ I, metric which describes the proportion of questions that given model solved completely (a solution is considered correct and only if it passes all public and private test cases).
Code completion. In this task, we assess the ability of the model to successfully complete a partially provided solution to competition coding question statement. The setup is similar to the Code Generation task above, but a partial (correct) solution is provided the prompt and the model is instructed to complete it to solve the question. We use LeetCode easy, medium, and hard problems from LiveCodeBench's 's (Jain et al., 2024) April 2024 release, combined with matching solutions from https: / github. com/ / k ammyul 04/Le4//// Leet Code-ooeeeee Sww.tiitiiioooooooooooooooooooat Minssonn,, omitting the last 15% of each medium/hard solution and 30-70% each easy solution and asking the LLM to complete the solution. As with Code Generation, we report pass @ 1,
A.3.3 REASONING CATEGORY
The reasoning abilities of large language models is another highly begchabbe and analyzed skill of LLMs (Wei et al., 2022; Suzgun et al., 2023; Yao et al., 2024). In LiveBench, we include two reasoning tasks: a harder version of task from Big- BengBBnch Hard (Suzgun et al., 2023), and Zebra puzzles.
Web of lies v2. Web of Lies is a task included in Big-Bench (bench authors, 2023) and Big- Bench Hard (Suzgun et al., 2023), The task is evaluate the truth value of random Boolean function expressed as a natural-language word problem. In particular, the LLM must evaluate (..
where each fi either negation or identity, and is True False. We represent z by the sentence: Xo (tells the truth, lies), and we represent ft by sentence: X; says X;-1 (tells the truth, lies). The sentences can be presented in a random order for increased difficulty. For example, simple = 2 version is as follows: 'Ka says Yoland tells the truth. Yoland lies. Does Ka tell the truth?' Already by October 2022, L.MS achieved near 100% on this task, and furthermore, there are concerns that Big- Bench tasks leaked into the training data of GPT-4, despite using canary strings (OpenAI, 2023).
For LiveBench, we create a new, significantly harder version of Web of Lies. We make the task harder with few additions: (1) adding different types of red herrings, (2) asking for the truth values of three people, instead of just one person, and (3) adding simple additional deductive component. For (1), we maintain a list of red herring names, so that the red herrings do not affect the logic of the answer while still potentially leading L.L.Ms astray. For example, 'Fred says Kayla lies,' where Fred
26

---

Published as a conference paper at ICLR 2025
is in the true 'web of lies', while Kayla may lead to series of steps ending in dead end. Overall, the number of total red herring sentences is drawn from a uniform distribution ranging from 0 to 19. For (3), we simply assign each name location and give sentences of the form 'Devika is at the museum. The person at the museum says the person at the ice skating rink lies.' We find that this makes the task significantly harder for leading M.M,s, even without shuffling the sentences into random order.
An example question from the Web of Lies v2 task.
In this question, assume each person either always tells the truth or always lies, Tala is at the movie theater. The person at the restaurant says the person at the aquarium lics. Ayaan is at the aquarium. Ryan is at the botanical garden. The person at the park says the person the art gallery lies. The person the museum tells the truth. Zara is at the museum. Jake
the art gallery. The person the art gallery says the person at the theater lics. Beatriz is at the park. The person at the movic theater says the person at the train station lies. Nadia is at the campground. The person at the campground says the person the art gallery tells the truth. The person at the theater lics. The person at the amusement park says the person the aquarium tells the truth. Grace is the restaurant. The person the aquarium thinks their friend is lying. Nia is at the theater. Kehinde is at the train station. The person at the theater thinks their friend is lying. The person at the botanical garden says the person at the train station tells the truth. The person at the aquarium says the person at the campground tells the truth. The person at the aquarium saw firetruck. The person the train station says the person at the amusement park lies. Matco is the amusement park. Does the person the train station tell the truth? Dnes the person at the amusement park tell the truth? Does the person at the aquarium tell the truth? Think step by step, and then put your answer in # **bold** * as list of three words, yes no (for example, # *'yes, no, *)s* If you don't know, guess.
Ground Truth: no, yes, yes
Zebra puzzles. The second reasoning task we include is Zebra puzzles. Zebra puzzles, also called Einstein's 's riddles or Einstein's puzzles, are a well-known (Jeremy, 2009) reasoning task that tests the ability of the model to follow a set of statements that set up constraints, and then logically deduce the requested information. The following is an example with three people and three attributes:
An example question from the Zebra Puzzle task.
There are 3 people standing in line numbered through in left to right order. Each person has set of attributes: Food, Nationality, Hobby.
The attributes have the following possible values:
Food: nectarine, garlic, cucumber
Nationality: chinese, japanese, thai
Hobby: magic-tricks, filmmaking, puzzles
and exactly one person in the line has given value for an attribute. Given the following premises about the line of people:
the person that likes garlic on the far left
the person who is thai is somewhere the right of the person who likes magic-tricks
the person who is chinese is somewhere between the person that likes cucumber and the person who likes puzzles
Answer the following question: What is the hobby of the person who is thai? Return your answer as single word, in the following format: ***x*** where X is the answer.
Ground Truth: filmmaking
We build existing repository for procedural generation of Zebra puzzles (quint t, 2023); the repository allows for randomizing the number of people, the number of attributes, and the set of constraint statements provided. For the attribute randomization, they are drawn from set of 10 possible categories (such as Nationality, Food, Transport, Sport) and for each of these categories there are between 15 and 40 possible values to be taken. For the constraint statements, the implementation
27

---

Published as a conference paper at ICLR 2025
allows for up to 20 'levels' of constraint in ascending order of intended difficulty. For example, level I could include a statement such as 'The person who likes garlic is on the left of the person who plays badminton' and level 10 statement could be 'The person that watches zombie movies likes apples or the person that watches zombie movies likes drawing, but not both'. Higher levels also include lower level statements in their possible set of statements to draw from, but this set narrows progressively as the level increases from 12 to 20 by removing the possibility of having lower-level statements (starting with removing level 1, then removing level 2, etc).
The repository also includes a solver for the puzzles, which we use to ensure there is a (unique) solution to all of our generated puzzles.
Our modifications to the original repository primarily target the reduction of ambiguity in the statements (e.g. changing 'X is the left of Y' to 'X the immediate left of Y"). For generation, we pick either 3 or 4 people with 50% probability, either 3 or 4 attributes with 50% probability, and we draw the levels from the integer interval [10, 20] with uniform probability. In preliminary testing, we found that larger puzzles proved exceedingly difficult for even the top performing LLMs.
Spatial reasoning. The final reasoning task is spatial reasoning questions. This set of 50 handwritten questions tests model's ability to make deductions about intersections and orientations of common 2D and 3D shapes. Two example questions are below.
Example question one from the Spatial Reasoning task.
Suppose I have three spheres of radius 3 resting on a plane. Each sphere is tangent to the other two spheres. If I consider new shape whose vertices are equal to the set of tangent points of the pairs of spheres, what is the shape? Is it square, tetrahedron, triangle, circle, line segment, or rhombus? Think step by step, and then put your answer in **bold*** as a single phrase (for example, "*circle"*). If you don't 't know, guess.
Ground Truth: triangle
Example question two from the Spatial Reasoning task.
Suppose I have a regular heptagon, and can make four straight cuts. Each cut cannot pass through any of the vertices of the heptagon. Also, exactly two of the cuts must be parallel. What is the maximum number of resulting pieces? Think step by step, and then put your answer in * "bold** single integer (for example, **(**). If you don't know, guess.
Ground Truth: 10
A.3.4 DATA ANALYSIS
LiveBench includes three practical tasks in which the LLM assists in data analysis or data science: column type annotation, table join prediction, and table reformatting. Each question makes use of a recent dataset from Kaggle or Socrata.
Owing to the limited output context lengths of the current generation of L.LMs and the comparatively high per-token costs of generating responses, we upper bound the size of our tables with respect to cell length, column count and row count. Even with these limitations, we find that our tasks remain sufficiently challenging even for the current state-of-the-art models.
Example questions from the Data Analysis category can be lengthy, so examples can be viewed here.
Column type annotation. Consider table A with columns and rows. We denote each column CE A as function which maps row indices to strings; i.e., for S i < L, we have C; N + Σ., where i is the column index. Let L C Σ, denote label set; these are our column types to be annotated. Standard CTA assumes a fixed cardinality for this label set, indexed by a variable we call Given the above definitions, we define single-label CTA C A x L relation between tables and labels:
VC, 31 (C.1)) E CTA
(1)
28

---

Published as a conference paper at ICLR 2025
We seek generative method M E, E, that comes closest satisfying the following properties: M(a, I.) E VC E Mo, L) E CTA
For further details on the task, please refer to Feuer et al. (2023). Implementation details. For each benchmark instance, we retrieve random A from our available pool of recent tables. We randomly and uniformly sample C' from A, use the actual column name of A as our CTA ground-truth I,, and retrieve 01 ... a5 column samples from C, with replacement, providing them as context for the L.M. Metrics. We report Accuracy @ I over all instances, accepting only case-insensitive exact string matches as correct answers.
Table reformatting. Given table A rendered according to a plaintext-readable and valid schema for storing tabular information ds, we instruct the L.M to output the same table with the contents unchanged but the schema modified to a distinct plaintext-readabl valid schema by. Implementation details. We use the popular library Pandas to perform all of our conversions to and from text strings. We allow the following formats for both input and output: "JSON", "ISONL", "Markdown", "CSV", "TSV", "HTML". As tabular conversion from JSON to Pandas is not standardized, we accept several variations. At inference time, we ingest the LLM response table directly into Pandas. Metrics. We report Accuracy @ I over all instances. An instance is accepted only if it passes all tests (we compare column count, row count, and exact match on row contents for each instance).
Join-column prediction. Given two tables A and B, with columns a..... and bj.... respectively, the join-column prediction task is to suggest a pair (ax, bi) of columns such that the equality condition ag = by can be used to join the the tables mapping M: A + B. The mapping is usually partial injective: not every column in B is mapped from A, not every column in A is mapped to B. For further details, please refer to Yan & Hc (2020). Implementation details. We randomly sample columns with replacement from our entire collection of tables, generating a fixed column pool C. We retain half the rows of A to provide as context to the LLM. The remaining rows are used generate new table B. For each instance, we randomly sample columns from both the target table and the column pool and join them to B. We anonymize the column names in B. then pass both A and B to the LLM and ask it to return a valid join mapping M. Metrics. We report the Fl score over columns, with TPs scored as exact matches between ground truth and the LLM output, FPs scored as extraneous mappings, FNs scored as missing mappings, and incorrect mappings counting as FP + FN.
way that matches with the provided ground-truth
A.3.5 INSTRUCTION FOLLOWING
An important ability of an LLM is its capability to follow instructions. To this end, we include instruction following questions in our benchmark, inspired by IFEval (Zhou et al., 2023a).
Generating live prompts and instruction. IFEval, or instruction-folloiing evaluation for LLMs. contains verifiable instructions such as "write more than 300 words" or "Finish your response with this exact phrase: pand__prase]_ These instructions are then appended prompts like "write short blog about the visit in Japan". We use this modular nature between the prompt and instruction to construct live prompts.
For our live source, we considered news articles from The Guardian: we are able to obtain 200 articles using their API Using the first n sentences article text as the source text, we consider four different tasks using the text: paraphrase, summarize, simplify, and story generation. The exact prompts can be seen in Table 12. For the instructions, we use the code provided by Zhou et al. (2023a), making a few modifications such as increasing the max number of keywords from two to five. Additionally, we compose different instructions together by sampling from a uniform distribution from 2 to 5. However, since the instructions can be conflicting, approximate normal distribution of the number of instructions per example with the majority of the containing two or three instructions. A full list of the instructions can be found in Appendix Table 3. To construct, the full prompt, containing the news article sentences, the prompt, and the instructions, we use the following meta prompt: "The following are the beginning sentences of a news article from the Guardian. In In guardian article In subbta prompt (instructions)".
deconflict the instructions. This results in
Phttps:// form. thegua r ri an.
29

---

Published as a conference paper at ICLR 2025
Table 12: The prompt for each subtask used in each of the four instruction following tasks.
Subtask
Subtask Prompt
Paraphrase
Please paraphrase based the sentenees provided.
Summarize
Please summarize based on the sentences provided.
Simplify
Please explain in simpler terms what this text means.
Story Generation
Please generate a story based on the sentences provided.
Scoring. To evaluate the model's performance instruction following, that considers two key factors: whether all instructions were correctly followed for a given prompt, i.e. Prompt-level accuracy, and what fraction of the individual instructions were properly handled, i.e. Instruction-level accuracy. The first component of the score checks if the model successfully followed every instruction in the prompt and assigns 1 or 0 if it missed any of the instructions. The second component looks at each individual instruction and checks whether it was properly followed or not. The final score is the average of these two components, scaled to lie between and score of 1 represents perfect adherence to all instructions, while lower scores indicate varying degrees of failure in following the given instructions accurately.
scoring method
Example questions from the Instruction Following category can be lengthy, so examples can be viewed here.
A.3.6 LANGUAGE COMPREHENSION
Finally, we include multiple language comprehension tasks. These tasks assess the language model's ability to reason about language itself by, (1) completing word puzzles, (2) fixing misspellings while leaving other stylistic changes in place, and (3) reordering scrambled plots of unknown movies.
Connections. First we include the 'Connections' category?. Connections is a word puzzle category introduced by the New York Times (although similar ideas have existed previously). Sixteen words are provided a random order; the objective of the game is to sort these into four sets of four words, such that each set has a "connection' between them. Such connections could include the words belonging to related category, c.g., 'kiwi, grape, pear, peach" (types of fruits); the words being anagrams, the words being homophones, being words that finish a certain context, e.g., 'ant, drill, island, opal' being words that come after the word 'fire" to make a phrase. Due to the variety of possible connection types that exist, the wider knowledge required understand some connections, as well as some words potentially being 'red herrings' for connections, this task challenging for LLMs prior work (Todd et al., 2024) has comprehensively tested the task on the GPT family of models, as well as on sentence embedding models derived from, e.g., BERT (Devlin et al., 2018) and RoBERTa (Liu al., 2019). The authors found that GPT-4 has an overall completion rate below 40% on the puzzles (when allowed multiple tries to get it correct), concluding that 'large language models in the GPT family are able to solve these puzzles with moderate reliability, indicating that the task is possible but remains a formidable challenge.' In our work, we assess the single-turn performance and test performance on a much larger set of models.
The original task provided for number of 'retry' attempts in the event of an incorrect submission for a category. To fit into the framework of our benchmark we take the model's answer from a single turn; to ameliorate the increased difficulty of this setting, we use fewer words/groups for some questions. The split we use is 15 questions of cight words, 15 questions of twelve words and 20 questions of sixteen words. An example prompt is as follows:
"See httpttt/////... htttpps///ww.nnnt nyt nes. com/ wwteesccomm//mee//////e/e ons.
30

---

Published as a conference paper at ICLR 2025
An example question from the Connections task.
You are given words/phrases below. Find two groups four items that share something in common. Here are a few examples of groups: bass, flounder, salmon, trout (all four are fish); ant, drill, island, opal (all four are two-word phrases that start with 'fire" ); are, why, bee, queue (all four are homophones of letters); sea, sister, sin, wonder (all four members of a septet). Categories will be more specific than e.g., 'S-letter-words' 'names', or 'verbs'. There is exactly one solution. Think step-by-step. and then give your answer in **ldd as a list of the 8 items separated by commas, ordered by group (for example, bass,, founder, salmon, trout, ant, drill, island, opal**). If you don't 't know the answer, make your best guess. The items are: row, drift, curl, tide, current, press, fly, wave.
Ground Truth: current, drift, tide, wave, curl, fly, press, row
The score for this task is the fraction of groups that the model outputs correctly.
Typo corrections. Next, we include details about the Typos task. The idea behind this task is inspired by the common use-case for LLMs where a user will ask the system to identify typos and misspellings in some written text. The challenge for the systems but to leave other aspects of the text unchanged. It is common for the LLM to impose its own writing style onto that of the input text, such as switching from British to US spellings or adding the serial comma, which may not be desirable.
fix just the typos or misspellings,
To create the questions for this task, we take text from recent ArXiv abstracts. These abstracts may themselves start with misspellings and grammatical errors. Therefore, our first step is to manually pass over the abstracts and fix typos and grammar issues. Next, we assemble list of common misspellings as found online. This is done so as to replicate common misspellings performed by humans, even though we synthetically generate the questions. Finally, for each question, we sample probability p (0.5, 0.7) of flipping correctly spelled words to misspelled words. We then use that probability to replace every correctly spelled word with a common misspelling with that probability p. This allows there be variability in the difficulty of the problem included in the benchmark. In our first release, we include 50 questions. Finally, to score this problem, we merely ask whether the ground truth abstract is contained the output provided by the LLM.
example question from the Typos task.
Please output this exact text, with no changes at all except for fixing the misspellings. Please leave all other stylistic decisions like commas and US vs British spellings as in the original text.
We introducehten consept of k-token signed graph adn studdy some of its combinatorial and algebraical properties. We prove that twpo switching isomorphic signed graphs ahve switching isomorphic token graphs. Moreover, we sohw tyhat the Laplacian spectum of balanced signed graph is contained in the Laplacian spectra of its k-token signed graph. Besides, we introduce and studdyther unbalance levle of signed graph, which is new parameter tyhat measures how far signed graph is frome beng halanced. Moreover, we study the relation bewteen the frustration index anbdther unhallance level signed graphs adn their token signed graphs.
We introduce concept a k-token signed graph and study some of its combinatorial and algebraic properties. We prove that two switching isomorphic signed graphs have switching isomorphic token graphs. Morcover, we show that the Laplacian spectrum of a balanced signed graph is contained in the Laplacian spectra of its k-token signed graph. Besides, we introduce and study the unbalance level of signed graph, which a new parameter that measures how far a signed graph from being balanced. Moreover, we study the relation between the frustration index and the unbalance level of signed graphs and their token signed graphs.
Ground Truth:
Plot unscrambling. Finally, we include movie synopsis unscrambling task. We obtain movie plot synopses from IMDb or Wikipedia for feature-length films released after January 1st 2024. These synopses then split into their constituent sentences and randomly shuffled. The lengths of the synopses vary from as few as sentences to as many as 66 sentences; at the upper end, this is a very challenging task. The LLM is provided the shuffled sentences with the prompt: "The following plot
31

---

Published as a conference paper at ICLR 2025
Table 13: Summary for tasks in Li LiBehhh..
Category
Task
Num.
Data Source
data aata_analysis
cta
50
Kaggle and Socrata datasets
data_ _aallsis
tablejoin
50
Kaggle and Socrata datasets
data data_analy @ata_analysis
tablereformat
50
Kaggle and Socrata datasets
instruction
summarize
50
The Guardian articles
instruction Boolloiion_oollowingg
paraphrase
50
The Guardian articles
instruction o
story generation
50
The Guardian articles
instruction_follovinn
simplify
50
The Guardian articles
language
typos
50
ArXiv abstracts
language
connections
50
NYT daily puzzles
language
plot_unscrambling
40
IMDb plot synopses
reasoning
web_of li_s_ves
50
N/A (synthetic)
reasoning
zebra_puzzle
50
N/A (synthetic)
reasoning
spatial
50
N/A (manually created)
math
olympiad
36
International Olympiad 2024
math
AMPS Hard
100
N/A (synthetic)
math
math comp
math_comp
96
AMC 2023 and AIME 2024
coding coding
boding__cmpllee
50
LiveCodeBench
LCB generation
78
LiveCodeBench and Lectcode
summary of movie has had the sentences randomly reordered. Rewrite the plot summary with the sentences correctly ordered. Begin the plot summary with <PLOT_SUMMARYS
Scoring the task involves two decision points: 1) how deal with transcription errors those in which the model modifies the lines when producing its output 2) given the ground truth ordering of sentences and the LLM's ordering, choosing an appropriate scoring metric. For 1), one option is to permit only strict matching - that is, the LLM must transcribe perfectly. However, although the strongest models do perform well on this (we find they achieve over 95% transcription accuracy), we find that LLMs often correct grammatical errors or spelling mistakes in the source data when transcribing. As we are primarily interested in testing the models' capabilities for causal language reasoning in this task, rather than precise transcription accuracy, we instead apply fuzzy-match using difflib (Team, 2008) to determine closest match using a version of the Rateliff/Oberseelp algorithm (Ratcliff & Metzener, 1988). For 2), we calculate the score 1 s__senennee gi where n_ sestenncee__ee gt the number of sentences in the ground truth synopsis, and d the Levenshtein distance (Levenshtein, 1966) of the model's 's sentence ordering to the ground truth synopsis ordering. Thus if the model's sentence ordering perfectly matches the ground truth, the distance d would be and the score would be for that sample.
One might think that it is plausible that synopsis unscrambling cannot always be solved with the information provided. However, note that even if the set of sentences do not create distinct causal ordering, the task is essentially asking the L.LM maximize the probability that given arrangement of sentences is real movie. In addition to causal reasoning, the LLM can use subtle cues to reason about what ordering creates most compelling plot. Furthermore, even if there does exist an upper bound on the score that can be achieved that is strictly below 100%, it can still be useful metric for distinguishing models' relative strengths. An analogous metric is that of next-token perplexity in language modelling; although it is likely that perfect prediction of the next token is impossible to achieve, and we do not even know what the obtainable lower bound on perplexity is, is still a powerful metric for determining hanguage-modelling performance.
Example questions from the plot unscrambling task can be lengthy, so examples can be viewed in our repo here.
In Table 13. we also present a summary table consisting the number of questions and source data for each task in I Benchh.
32

---

Published as a conference paper at ICLR 2025
A.4 GRADING METHODOLOGY
LiveBench makes use of automatic regex-based grading methods in order to avoid the biases and other downsides of LLM judging, as detailed in Section 1. On the other hand, automated grading methods have two main pitfalls, which we take care to circumvent. The first is that an instructionfollowwe element is added to the task. For example, reasoning task with a complex answer instruction format tests sooluuttonn-oolooinng as well as reasoning, rather than pure reasoning. The second that care must be made to allow for different answer formats giving exact, unambiguous instructions in the prompt, regarding the format of the answer).
not to favor a particular LLM (even after
When creating questions, make that the prompt is written in such way that the answer format is fully specified and unambiguous. We typically give an example of the answer format in the prompt, and specify any potential parts that are unclear.
Additionally, our grading methodology is designed to be fully permissive, accommodating all answer formats within reason. As rule of thumb, if a human reading the LLM's answer would be able to comprehend the answer (and assuming it is correct), then the answer should be marked correct. For example, in our math category, models respond with answers in either latex 'boxed or 'fbox() )) environments. For multiple choice questions, we ask for the letter answer, but we accept either the letter or the raw answer. The reason for our less-strict judging because some models may have been instruction-ture this by using flexible regex patterns that capture different notation styles or response structures. This approach allows us to evaluate models based on their task-specific skills, rather than their ability to follow specific instructions.
to particular format, giving them an advantage or disadvantage. We achieved
In order to ensure given task is not overly testing instruction following (in addition to its actual category), and also to ensure scoring accuracy and fairness, we regularly manually inspect a random sample of responses which were labeled incorrect for each model. We look for common answer formats that are incorrect and not picked up by the regex-based parser. This allows us to regularly improve our automatic scoring functions to admit many answer formats as possible, within reason.
We continue to monitor model outputs and update scoring functions as needed to capture answers appropriately. When adding new models or tasks to the benchmark, re-evaluate and refine scoring methodologies to ensure fairness and accuracy. This ongoing evaluation and maintenance process enables maintain the integrity of our benchmark, providing comprehensive assessment of models' strengths and weaknesses. By decoupling instruction following capabilities from task-specific evaluations, we can accurately evaluate and compare models across various tasks and categories.
A.5 QUALITY CONTROL FOR NEW MODELS
When integrating new models into LiveBench, conduct thorough quality control to ensure consistent and accurate evaluation. This process guarantees that our benchmark remains reliable and effective in assessing model capabilities.
New Model Setup We perform setup in order to accurately run inference on the model. New models come with README example code the chat template, system message, and hyperpa- rameters to run the model, and so we match this setup in LiveBench's code. For API models from an existing model family, there is typically very little additional setup. For an API model from new family, such as the Ol series, we add the same hyperparameters, system message, etc., as its example code. Similarly, for open-weight models, we make sure to match the published example code. Note that we generally prefer to use (trusted) APIs, even for open-weight models, because APIs are more controlled and fewer things can go wrong. As an example, new models occasionally have bugs when they are first released.
Model Output Evaluation Next, we evaluate the new model on LiveBench and compare the new model's 's scores to other models of similar average score, or models from the same family or base. We look for tasks that scored unusually low, relative to similar models. This is not a comprehensive test, but it helps to more quickly flag tasks that have an unexpectedly low result, to debug potential errors in the parsing code.
33

---

Published as a conference paper at ICLR 2025
Manual Verification Next, we manually inspect the incorrect answers from the flagged tasks above, as well as a random sample of the responses across all tasks. The goal is to make sure that the wrong answers truly incorrect, rather than correct answer in the wrong format, in such way that marked incorrect by the parsing function. (Note that these checks are focused on catching false negatives. It is extremely unlikely for false positive to occur, but we checked for false positives in the initial development of LiveBench, and occasionally check answers that are scored correct, for this reason.)
Scoring Function Updates Based on our evaluation and verification, we update our scoring functions to accommodate any unique response patterns or notation styles exhibited by the new model. This maintains the permissiveness and fairesss of our grading methodology, ensuring accurate scoring. Note that we made few such updates when new models came out that hit new edge cases of our scoring functions, in the first month of LiveBench's 's release, but now we have not needed to make a scoring function update for a few months.
Re- Re-Running the Evaluation Pipeline After potentially updating scoring functions for some tasks, we re-score all models with the new scoring functions. This ensures consistent and accurate results.
A.6 QUESTION UPDATE POLICY
To ensure the continued relevance and effectiveness of LiveBench, we have implemented thorough question update policy. That said, we still maintain a degree of flexibility in updating questions, as well as updating the LiveBench policies a whole, in order to be able to adapt to new developments and trends in the field (such as the release of models that employ search at inference time, or the popularity of new type of LLM capability). In this section, we give more details on the update policy laid out in Section 2.7.
Regular Updates We update questions and tasks on monthly basis, incorporating new and challenging prompts that reflect the evolving landscape of large language models.
In each update, we replace 1/6 of the questions on average, that the benchmark fully refreshed roughly every 6 months. We may speed up the turovver rate of questions in the future, based on interest in BeBnhh. Each month, we do not release the new questions until one month later, so that the public leaderboard always has 1/c questions that are private and completely contamination
We choose tasks to update based primarily on two factors: (l) the oldest tasks, and (2) the currently easiest tasks. In this way, the questions in Li LiveBench will stay new and continue to challenge the most capable LLMs. We also update tasks if there is any suspected contamination or other reason for updating task. Nearly all tasks can be updated simply by rumning a script. See Table 14 for breakdown of each task. Although most tasks can be auotmatically updated with a script, we always modify the generation script when creating new questions, in order make sure that the distribution of questions is different and harder over time.
After updating questions and tasks, we re-run the evaluation pipeline on all models, ensuring consistent and accurate results. By implementing this comprehensive question update policy, we guarantee that LiveBench remains a vibrant, dynamic, and relevant benchmark for evaluating large language models.
Evaluation Re-Runs
Version Control We maintain version control over all updates, ensuring transparency and repro- ducibility. This allows researchers to track changes and compare results across different versions of LiveBench.
Community Engagement We encourage community engagement and collaboration in expanding and improving LiveBench. Researchers and developers can contribute new tasks, provide feedback on existing ones, and participate in shaping the benchmark's evolution.
34

---

Published as a conference paper at ICLR 2025
Table 14: Description for whether tasks in LiveBenchcan be updated automatically.
Category
Task
updated automatically using a scripe?
data amalysis
cla
Yes replace with newer datasets
analyss
tablejoin
replace
datasets
data_analyy analysis
tablcroformat
Yes replace with ocver datascts
instruction_ Foalloiion_followingg
summarize
replace newer articles undlor update synthetic generation script
instraction, following
paraphrase
Yes- swap out news articles and/or update syntherie generation script
instruction, fesllutton_folllwinng
story, generation
Yes- swap
articles undior update synthetic generation script Yes swap out news articles und/or updale synthetic generation script
instruction_ instrutioo_followiing
simplity
language
typos connections
Yes replace with newer paper ahstracts
language
Yes Jes-replee replace with newer (daily) puzzles
language
plot_ unserambling
Yes- replace with newer imdh plot summaries
reasoning
web_of vies__2
Yes rerun (andlor update) the synthetic generation script
reasoning
zebra puzzle
Yes rerun (and/or updane) the synthctic generation script
reasoning
spatial
create new questions by hand
math
olympiad
Yes" rerun (and/or update) synthetic generation script, but source can only he updated yearly
AMPS, Hard
rerun (and/or update) the synthttic generation script
math
math comp
Yes but can only be updated yearly. or by using different contests
coding
coding completion
replace more recent LoceCode questions
coding
LCB _eg_cccrationn
Yes replace with more receut LoctCode questions
A.7 CONTAMINATION IN LIVEBENCH
We point out two different definitions of contamination: (1) test set contamination, and (2) task contamination train test distribution similarity.
The first definition is the one that we use in Section 1: when the test data (the questions and answers from the benchmark) are present in the training data itself. This is the definition that is commonly used in the literature, often called 'data contamination', 'data leakage', 'test set contamination" or even simply 'contamination' (Oren et al., 2023; Singh et al., 2024; Kalal et al., 2024; Golchin & Surdeanu, 2023a).
The second definition is related the question: how well do LLMs generalize today? low level of generalization is, e.g., training on AMC questions, and generalizing the same questions where the order of the answer choices, and other prose in the questions, are changed, while a high level of generalization is, e.g., training on AMC 12A 2023 and testing on AMC 12A 2024. The latter example, despite LLMs already exhibiting some degree of this type of high generalization, is still 'fair game' with respect to pretraining and fine-tuming many LLMs are trained on competitive math problem tasks and then evaluated on new, unseen test problems from the same distribution.
Despite it being accepted practice, we attempt to guard against excessive uses of (2): we do not publicly release any of the code used to generate LiveBench questions, we always modify the generating code when updating the LiveBench questions (making the questions harder), and we do not publicly release the new questions for one month.
We also acknowledge that LiveBench does not fully satisfy (1): while nearly all questions are from June 2024 or more recent, there are some coding questions from November 2023, and the AMC questions have only undergone a low level of modification from their November 2023 version. Therefore, a limited fraction of LiveBench is likely contaminated on all recent LLMs.
B ADDITIONAL DOCUMENTATION
In this section, we give additional documentation for our benchmark. For the full details, see /r/AADDEEREEE..
https: //gi http://iiveeeeecc/tiieeeeeeeocce
B.1 AUTHOR RESPONSIBILITY AND LICENSE
We, the authors, bear all responsibility in case of violation of rights. The license of our repository is the Apache I License 2.0.
35

---

Published as a conference paper at ICLR 2025
Table 15:
Statistics for tasks in LiveBench. This table gives the number of questions for each task, as well as the mean and std. dev number of output tokens per question for gpt-4-turbo-2024-00
Input Tokens
Output Tokens
Category
Task
Num.
Mean
Std. Dev.
Mean
Std. Dev.
data_ _ata_analysis
50
879.52
1294.81
2.42
1.36
data aat__analysis
tablejoin
3111.28
1186.58
58.06
35.03
data_anda anallsis
tablereformat
50
1395.54
814.92
475.64
333.57
instruction nestuuctionfoolloonngg
summarize
50
1818.92
462.22
218.46
110.38
instruction foolowing
paraphrase
1805.62
501.30
267.98
101.98
instruction nostooitionfollooinng instruction followiig
story generatiio
50
1794.30
474.77
405.24
156.49
simplify
50
1835.42
486.83
226.00
114.61
language
typos
50
1291.72
402.04
207.24
74.25
language
connections
50
884.54
48.94
422.06
541.96
language
plot Hloouunscrambling
40
3545.68
1333.37
613.70
166.17
reasoning
web of lies zebra_puzzle
50
1670.64
337.40
490.82
98.68
reasoning reasoning
1407.20
281.02
616.94
105.61
spatial
50
450.48
134.30
337.88
88.16
math
olympiad
36
4646.08
1579.41
1221.22
685.41
math
AMPS _ard
100
193.86
91.39
561.26
249.8
math
math_comp
96
678.20
243.09
664.27
424.33
coding
coding_comme completoooittonnn
50
2659.30
718.70
67.08
46.00
coding
LCB _ee_eeentaiion
78
2159.88
796.14
213.44
154.34
Total
1000
1612.27
703.73
394.85
261.79
B.2 MAINTENANCE PLAN
The benchmark is available on HuggingFace at http://hhggggggacc.ooo
We actively maintain and update the benchmark, already having added multiple updates, and we continue welcome contributions from the community.
B.3 CODE CONDUCT
Our Code of Conduct is from the Contributor Covenant, version 2.0. See
ht https: //www cont oibutor covenant wwwp:/aaa...rggeeevviiiioooo oftp://///ee/eooooo_eeoo_oee/e/ html.
B.4 DATASHEET
We include datasheet (Gebru al., 2021) for LiveBench in https://github.ce /ocasin////////eeo http:///aaaa.ttte//aaa..Ett
B.5 BENCHMARK STATISTICS
Here, we give statistics on the number of questions and average number of output tokens per task, and the total cost of rumning LiveBench with common API models. For the number of questions for each task, as well as the mean and std. dev number of input and output tokens per question for gpt-4--uurb-202-
90, Table 15. Across the 1000 questions in current problem set, the mean number of input tokens per question was 1612, and the mean number of output tokens was 395. See Table 16 for the price to run GPT and Claude models on LiveBench. as of October 2024. anprevieww20222 - 12 is the most expensive at $47.87 87, while claude- monde-3-hall the cheapest, at S0.90.
36

---

Published as a conference paper at ICLR 2025
Table 16: Prices for running GPT and Claude models on LiveBench. This table gives the ap- proximate cost for running models on LiveBerch as of Oct 2024. Note that we used the gpt-4-turbo tokenizer for all computations, so all other prices are approximate.
Model
Price in USD
ol-preview-2224-09912
47.87
01-mini-20244099-1
R 9.57
gpt-4o-2024-0--13
2 13.98
gpI-4-turbo-2024404409
27.97
gp1-4-1106-preview
27.97
gpt-3.5-turbo-0125
2 1.40
claude-3-opus
~ 53.80
Soudee-3-5-onnes
= 10.76
claude-3-sonnn
10.76
claude-3-haiku
RJ 0.90
37


Reviews:
[{"review_id": "IuBX350uWW", "reviewer": "Reviewer_uo5E", "summary": "The paper introduces LiveBench, a new benchmark designed to evaluate LLMs in a way that mitigates common issues like test set contamination and biases introduced by human or LLM judges. LiveBench is continuously updated with new questions sourced from recent information, such as math competitions, arXiv papers, news articles, and datasets. This ensures the benchmark remains relevant and resistant to contamination by models trained on static datasets. Unlike many benchmarks that rely on subjective human or LLM judgments, LiveBench uses objective ground-truth values for scoring. LiveBench includes 18 tasks across six categories—math, coding, reasoning, language comprehension, instruction following, and data analysis. Each task is designed to be difficult, with top-performing models achieving less than 70% accuracy. The tasks are either based on recent real-world data or are harder versions of existing benchmark tasks. The paper evaluates dozens of LLMs, both proprietary and open-source. The benchmark provides detailed results across all six categories and highlights the strengths and weaknesses of each model. The authors plan to expand the benchmark with new tasks and models over time. They also plan regular updates to ensure the benchmark remains challenging as LLMs improve.", "questions": "On line 214, the author suggests that LLMs are commonly used to perform table joins. Can they elaborate on this? This does not seem like a common real-world use case.\n\n\nOn line 299, the authors suggest they will release 200 prompts per month, but it seems in lines 307-311 they released significantly less than 200 prompts per month. Could the authors clarify what the plan for questions releases is?\n\n\nOn line 331, the authors suggest they use FastChat chat templates. Why is FastChat used over the explicit template provided in model tokenizers or readmes? FastChat is not necessarily guaranteed to be maintained to be constantly up to date for this purpose. Not all models are implemented in FastChat.\n\n\nOn line 363-365, the authors suggest that the zebra puzzles are too hard such that top models do not perform better than random chance. If this is the case, is there anything that can be done to make this category more relevant? E.g, making it slightly easier? Adding this noise into a benchmark seems unnecessary.\n\n\nOn line 402-407, the authors color outliers in red. However, outlines that are far below the line (large negative residual) are not colored. Why were some points determined as outliers but others not? This is true for both graphs.  Moreover, on the left plot, o1-preview is shown as an outlier, but the point next to Phi-3.5-mini-instruct is not shown as an outlier, even though the latter is even farther from the line of best fit.\n\n\nOn lines 438-445, the authors show Starling-LM-Beta on their graph, presumably to show the large differential between Chatbot Arena score and LiveBench score. However, Starling-LM-Beta is notably missing from Figure 5 which is described to show “all models”.  It is also missing from Table 3. Starling appears again in Figure 6… then it is gone again in Table 7… then back again in Table 9. Why is this the case? (I think Open Hermes and a couple other models also appear and disappear).\n\n\nOn lines 324-329, the authors cite various models they use. However, the citations for the remaining models are not present anywhere else. I find it strange that some models are chosen to be cited properly and others are not.\n\n\nIt seems the authors commonly use **answer ** to make the LLM generated answer parsable for verification. Why was ** chosen here, when it is also a common markdown symbol? Wouldn’t a less common symbol make sense here, like “{{answer}}” as one example. Peaking at the code, which seems to take the last bolded text as the final answer, it is conceivable that a markdown heavy model could have some extra bolding after the answer and get penalized for this. Think of something like “... **Final Answer: ** **X **\\n\\n **Additional Considerations: ** …” It is important to note that adding any bolded content after the final answer seems to not be explicitly warned against in the prompt to the LLM.\n\n\nOn lines 914-917, the authors show the judge prompt used for the LLM judge, which outputs a numerical score 1-5. However, it is unclear how this score is turned into a binary label used to calculate the error rates seen in Table 2.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality:\nThis paper makes very well rounded verifiable benchmarks that cover a large number of tasks in different ways. This in itself is unique and a great contribution to LLM research.\nWhile some of the tasks are inspired by existing benchmarks, some seem to be completely novel evaluation strategies (such as the proof reconstruction).\n\nQuality:\nThe benchmark itself seems mostly well constructed and fair. The emphasis on task diversity and verifiability is commendable.\n\nClarity:\nThe methods are generally clear. Experiments are well explained. Benchmark tasks/evaluation are clear.\n\nSignificance:\nThis paper gives a useful benchmark for offline evaluation of LLMs. It has more task coverage than many other offline evaluation schemes. It also presented potential solutions to the benchmark contamination problem by making the benchmark “live”.", "weaknesses": "The core feature LiveBench claims to offer is being “contamination-free” However, it seems this claim rests upon two main assumptions that are never addressed:\n1. The source data is not contaminated.\n2. The data perturbations used to decontaminate or generate new data are actually sufficient to do so.\n\nFor (1): Consider the math category which utilizes questions from AMC12 2023, SMC 2023, AIME 2024, and IMO 2024. It seems likely these high quality data points would already be exploited by model developers for training models, intentionally or unintentionally. While the author’s do not directly discuss the implications of this, they do mention they do make some modifications to the questions. This leads to (2).\n\nFor (2): “Contamination-free” is a strong statement. **First and foremost, I recommend the authors define exactly what they mean by “Contamination-free”.** As a reader, my understanding of “contamination-free” is as follows: *Given I train my LLM on LiveBench up to month M, I will have no performance gain on the queries released in month M + 1.* Now, if the authors choose to run with this relatively strong definition of contamination-free, the authors should consider experimentally verify that training a model on prompts up to month M leads to negligible gains on prompts from month M + 1. Right now, I find this doubtful. For example, one of the author’s stated methods to change queries is through “modifying the prose and the answer order” (line 142). While it is likely that changing trivial sections of a query may decrease an overfit LLM’s performance slightly, the authors do not show that this fully mitigates contamination, or even to what extent it does (if not fully). Moreover, the authors suggest on line 303 that “many of the tasks are synthetic and therefore very fast and simple to create a new set of questions based on fresh data”. This raises the question: what is stopping an adversarial from generating, let’s say 1 million rows, of LiveBench data through this “fast and simple” process, training on it directly. Does this count as “contamination”? Would the release of 200 prompts in month M + 1 protect against this? Alternatively, let’s consider the instruction following summarization task: in this case, the update process presumably is to swap out the Guardian articles with new ones and change around the instructions selected from the 16 choices. However, how can the author’s be sure this meaningfully changes the task, especially because the article doesn’t even matter— the only thing the model needs to learn in order to overfit is the 16 different instructions that it will be checked on?\n\nThe authors could potentially address this in 2 ways:\n1. Experimentally show how much training (with a reasonable setup) on M affects M + 1.\n2. Establish a very weak definition of “Contamination-free” (e.g. the exact prompt is not seen in M + 1). However, there are no guarantees that training on M will not help M + 1. (Note that this heavily decreases the significance of the benchmarks as “contamination-free”)\n\n**Bottom Line: In its current implementation, this paper has little evidence that its current query perturbation/generation process has any effectiveness against contamination. The authors should first define contamination, then defend that their benchmark sufficiently protects against this definition of contamination.**\n\n\n**The “live” aspect of the benchmark is powered by brute force, and is not necessarily automatic in any sense:** It is not entirely clear how LiveBench is significantly more “update-able” than existing static and verifiable benchmarks, such as MMLU or IFEval. The author’s pledge to keep updating the benchmark monthly does not seem like a sufficient novel contribution to benchmark construction in general. The authors should document the update plan for every single fine-grain task. They should explain the estimated time and monetary cost to update this specific task. They should detail any optimizations in task design to ensure it is easy to update, if any. This would be a great table for the appendix and would make it clear what the paper contribution on this front is.\n\n\n**Ablations on LLM-Judgments are insufficient, and potentially misleading:**\nThe experiments to show “LLM judges cannot accurately evaluate challenging math and reasoning questions” are not enough to address the scope of the claim.\nIn particular:\n1. Only GPT-4-Turbo was tested as judge\n2. GPT-4-Turbo was mainly judging itself, which is already likely worst case scenario\n3. Only 2 models total were judged\n4. Only 4 benchmarks tested\n5. Subpar judge prompt\n\n    a. No reference answer (generally these judges have either pairwise comparisons, or some form of reference answer)\n\n    b. No rubric for judging\n\nPossible ways to strengthen the experiment to adequately support the claim are as follows:\n1. Utilize more LLMs as judges\n2. Use the LLM to judge more models\n3. Judge with a reference answer, and/or use a more updated judge prompt, such as the judge prompt from Arena-Hard-Auto or Alpaca-Eval 2.0.\n4. Show accuracies on all tasks\n\n\n**Insufficient comparisons to other benchmarks or “ground truth” evaluations:**\nThe authors compare LiveBench to Chatbot Arena and Arena-Hard. They hypothesize that the differences in rankings are mainly due to style or self-bias. If the authors suspect style to be a large factor in ranking difference they should compare LiveBench taking to style controlled Chatbot Arena rankings, proposed in the blog https://blog.lmarena.ai/blog/2024/style-control/. Note that Arena-Hard also has style control implemented. The authors should provide a table showing the correlation between the benchmarks. There are also Chatbot Arena categories, such as Math or Instruction following, which would provide more informative correlations to the respective LiveBench tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "feK9wvGGui", "reviewer": "Reviewer_L2M4", "summary": "The paper presents LiveBench, a benchmark designed to evaluate LLMs while avoiding data contamination and biases associated with LLM and human judges. LiveBench includes frequently updated, diverse tasks across categories such as math, coding, reasoning, and data analysis, scoring responses based on ground-truth answers to ensure objectivity. With new, harder tasks released monthly, LiveBench aims to accurately assess LLMs as they improve over time.", "questions": "Why did you not evaluate with some randomness to obtain significance over noise?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The benchmark addresses a prevalent issue in LLM evaluation.\n- The paper is well-written, clearly explaining both the motivation and the implementation.\n-  LiveBench scores models against objective ground-truth answers, not relying on possibly-biased LLM-judges.\n- Tasks presented are very diverse.", "weaknesses": "- Some models' performance in evaluations are very close. It's hard to say how similar they are without a significance test / stds.\n- As LLM developers may train their models to excel on specific benchmarks, frequent updates could eventually lead to overfitting to LiveBench’s task formats, even with the regular introduction of new content. This could incentivize models to optimize for LiveBench-specific tasks rather than improving general capabilities. This can be solved with the introduction of more types of tasks and the contribution of the community, but it's also worth thinking about refreshing existing tasks' formats.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FTgy37CPA", "reviewer": "Reviewer_G9nN", "summary": "The paper presents LiveBench, a new benchmark for evaluating LLMs designed to avoid common issues such as test set contamination and biases from LLM or human judging. LiveBench achieves this by (1) frequently updating questions with information from recent sources, (2) employing automated scoring against objective ground-truth values, and (3) covering a wide range of challenging tasks in areas such as math, coding, reasoning, language, instruction following, and data analysis.\n\nThe authors evaluated multiple prominent closed-source and open-source LLMs, showing that even top-performing models struggle to achieve over 70% accuracy, highlighting the benchmark’s difficulty.", "questions": "1. Are all tasks' ground truths unique and definitive? How do you ensure that there is no ambiguity in the ground truth for each task?\n2. In line 509, you mentioned that \"In Srivastava et al. (2024), the authors modify the original MATH dataset (Hendrycks et al., 2021) by changing numbers in the problem setup, leading to significant declines in model performance across all LLMs, including the leading ones.\" Given this, do the tasks generated in this benchmark, such as the Typos task, also face similar issues due to using updated data sources under the same modification rules?\n3. Could you provide a statistical table that clearly presents the number of questions, tasks, sources, and other relevant information for each category? This would enhance clarity and help readers understand the distribution and characteristics of the tasks more effectively.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper provides a dynamic, comprehensive and challenging benchmark that stays current with LLM development.\n2. The benchmark’s monthly updates and open access to questions, code, and model answers contribute to high reproducibility and transparency.", "weaknesses": "1. The paper fails to analyze the quality of the questions used for evaluation, particularly those manually constructed or adjusted for model testing.\n2. The paper seems to employ a regular expression matching approach against ground truth to evaluate model outputs, a method which can introduce evaluation bias, especially for models with weaker instruction-following abilities. While the paper mentions potential biases from using large language models as evaluators, it neglects to analyze how regex-based matching could exacerbate evaluation inaccuracies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)