From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline
Authors: Tianle Li
Wei-Lin Chiang
Evan Frick
Lisa Dunlap
Tianhao Wu
Banghua Zhu
Joseph E. Gonzalez
Ion Stoica
Venue: ICML 2025 poster (2025)
Keywords: llm
evaluations
Primary area: general_machine_learning->evaluation
TL;DR: Arena-Hard-Auto: An automatic LLM benchmark.

Abstract: The rapid evolution of Large Language Models (LLMs) has outpaced the development of model evaluation, highlighting the need for continuous curation of new, challenging benchmarks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time-consuming. To address this, we introduce BenchBuilder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-1M, extracting challenging prompts and utilizing LLM-as-a-Judge for automatic model evaluation. To validate benchmark quality, we propose new metrics to measure a benchmark’s alignment with human preferences and ability to separate models. We release Arena-Hard-Auto, a benchmark consisting 500 challenging prompts curated by BenchBuilder. Arena-Hard-Auto provides 3x higher separation of model performances compared to MT-Bench and achieves 98.6% correlation with human preference rankings, all at a cost of $20. Our work sets a new framework for the scalable curation of automated benchmarks from extensive data.

From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline
Tianle Li Wei-Lin Chiang"
Evan Frick Joseph E. Gonzalez.
Lisa Dunlap
Tianhao Wu
Banghua Zhu
Ion Stoica'
Abstract
Open- Ended
Prompt Curation
Evaluation
The rapid evolution of Large Language Mod- els (LLMs) has outpaced the development of model evaluation, highlighting the need for con- tinuous curation of new, challenging bench- marks. However, manual curation of high-quality, human-aligned benchmarks is expensive and time- consuming. To address this, we introduce Bench- Builder, an automated pipeline that leverages LLMs to curate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling con- tinuous benchmark updates without humans in the loop. We apply BenchBuilder to datasets such as Chatbot Arena and WildChat-IM, ex- tracting challenging prompts. To validate bench- mark quality, we propose new metrics to mea- sure benchmark's alignment with human prefer- and ability separate models. We release Arena-Hard-Auto, benchmark consisting of 500 challenging prompts curated by BenchBuilder. provides 3x higher separation of model performances compared to M'T-Bench and achieves 98.6% correlation with human pref- erence rankings, all at a cost $20. Our work sets new framework for the scalable curation of automated benchmarks from extensive data. Our code is available at https: /tt///giiee gightu......./gggg com/ Imarena/ aaaaaaaarreaaaa aaaaa-raaaaee hard-autro-o
Arena-Hard-Auto
Automatic
Yes
Automatic
MMLU, MATH, GPQA
Automatic
No
Manual
MT-Bench, AlpacaEval
Automatic
Yes
Manual
Live Bench, Live Code Bench
Automatic
No
Manual
Chatbot Arena
Human
Yes
Crowd-source
Figure I. Classification of LLM benchmarks: we categorize bench- marks how the evaluation done, whether the evaluated tasks are ground-truth open-ended, how are the prompts curated, and whether developer can control source for the prompts.
Markert, 2005; Hendrycks et al.. 2021a) are quickly becom- ing saturated and struggle to differentiate statc-of-the-an models.
To address these limitations, recent benchmarks like GPQA (Rein et al., 2023) source high-quality and challeng- ing prompts from domain experts. Although these efforts have produced challenging evaluation sets, they come at a steep price-GPQA, for instance, cost over $120,000 to curate its 500 multiple-choice questions (Rein, 2024). The reliance on manual curation makes such benchmarks diffi- cult to produce. Moreover, their static nature is susceptible to test-set leakage and overfitting models trained on similar datasets. This necessitates the continuous develop- ment of new benchmarks, exacerbating the cost and labor of manual curation. Further, many of these benchmarks rely on close-ended tasks that fail to capture the open-ended nature real-world interactions, undermining their cost- effectiveness for evaluating alignment to user preference.
1. Introduction
The proliferation of Large Language Models (LLMs) has spurred advancements as models expand their capabilities by training on increasingly vast and diverse datasets. Tradi- tional static benchmarks (Wang et al., 2019; Rajpurkar et al., 2016; Bowman al., 2015; Dolan & Brockett, 2005; Bos &
An alternative approach without manual curation involves crowdsourcing prompts through live evaluation platforms such as Chatbot Arena (Chiang et al., 2024). These plat- forms test models against a continuous stream of fresh, open-ended queries and user feedback. However, real-time human evaluation both expensive and time-consuming.
'qqual contribution ! 'University California, Berkeley. Corre- spondence to: Tianle Li <tianleli berkeley.cdus.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Title Suppressed Due to Excessive Size
rendering these platforms infeasible for frequent evaluations by model developers. Moreover, while the crowd-sourced prompts represent real-world and open-ended tasks, their quality varies in difficulty and cannot be converted to chal- lenging benchmarks without careful data filtering.
ground-rruth-based (e.g., multi-choice question answer- ing). They cover a wide range of domains, including math, science, coding, and reasoning. Common ones include MMLU (Hendrycks et al., 2021a), MATH (Hendrycks et al., 2021b), GSM-8K (Cobbe et al., 2021), HumanEval (Chen et al., 2021), DROP (Dua et al., 2019), BigBench (Sri- vastava et al., 2023), HellaSwag (Zellers et al., 2019), AGIEval (Zhong et al., 2023), GPQA (Rein et al., 2023), as well comprehensive collection like HELM (Liang et al., 2022).
In light of these open challenges, there is a pressing need for an automated pipeline which can curate high-quality prompts dynamically scale. In this paper, we introduce BenchBuilder, an automated benchmark curation system designed to address these gaps. BenchBuilder leverages LLMs to curate, filter, and validate prompts based on seven indicators of high-quality prompts, such as specificity and domain knowledge, creating a pipeline that can continuously curate benchmarks alongside model development.
Many have considered task-based evaluation such as IFE- val (Zhou et al., 2023), SWE-Bench (Jimenez et al., 2024), BigCodeBench (Zhuo et al., 2024) or AgentBench (Liu et al., 2023). As L.LMs become widely adopted in open- ended scenarios involving interaction with humans (e.g.. chatbot), many have considered human evaluation using domain experts or crowd raters such as Amazon Mechanical Turk (Karpinska et al., 2021; Wang et al., 2023) to exam- ine models' response quality. As an alternative to human labeling, previous work has shown that LLM-as-a asda-e can be effective human preference proxies (e.g., Alpaca- Farm (Dubois al., 2023), MT-bench (Zheng et al., 2023b), AlpacaEval (Li al., 2023), WildBench (Lin et al., 2024)).
We apply BenchBuilder to crowd-sourced datasets, both Chatbot Arena (Chiang et al., 2024) and WildChat- 1M (Zhao et al., 2024), demonstrating that it can robustly generate high-quality benchmarks that differentiate models. The resulting benchmark, Arena-Hard-ae Hato-Auto, employs LLM judges (Zheng et al., 2023a; Li et al., 2023) to estimate hu- man preferences against a baseline model, making the entire process-from prompt curation to evaluation-fullly auto- mated. We also address potential biases in LLM-based eval- uations and propose solutions to mitigate them. To assess benchmark quality, we introduce new metries that measure a benchmark's ' ability to confidently separate models and align with human preferences. When compared to leading benchmarks such as AlpacaEval LC (Dubois et al., 2024) and MT-Bench (Zheng et al., 2023a), Arena-Hard-Aur achieves stronger model separability, tighter confidence in- tervals, and achieve 98.6% correlation with Chatbot Arena rankings, making it a fast, reliable predictor of downstream model performance.
Benchmark leakage. A fundamental limitation of static benchmarks is the potential risk of test set leakage (i.e., contamination). Existing works (Carlini et al., 2021; Sainz et al., 2023; Yang et al., 2023; Reid et al., 2024) have sug- gested a growing risk of contamination, which undermines the reliability of benchmarks over time, motivating the need for benchmarks that more frequently updated.
Live benchmarks. DynaBench (Kiela et al., 2021) iden- tifies these challenges and recommends creating living and continuously evolving benchmarks. Recent works LiveBench (White et al., 2024), LiveCodeBench (Jain et al., 2024a), MixedEval (Ni et al., 2024), R2E (Jain et al., 2024b), as well as the community based live evaluation, Chatbot Arena (Chiang al., 2024). However, we focus on develop- ing pipeline for automatic benchmark curation to enable automatic evaluation on open-ended tasks (see Figure 1).
To summarize, our works makes the following contributions:
1. We propose a novel data curation pipeline, Bench- Builder, to automatically construct high-quality bench- marks from crowdsourced data.
2. We propose metrics to capture desired properties in an LLM benchmark, and validate that Arena- Arena-Hard-Ane Hard-Auto achieves higher model separation and alignment to human preference than existing benchmarks.
3. How do you measure benchmarks?
We outline two key properties that the benchmark aiming to approximate human preference should possess provide meaningful comparisons between models:
3. We open-source both BenchBuilder pipeline and Arena- Hard-Auto benchmark
1. Separability: the benchmark should separate models with high confidence.
2. Related works
LLM benchmarks. We briefly review widely used LLM benchmarks. Most existing benchmarks are static and
2. Alignment with Human Preference: the benchmark should agree with human preference.
OOur code is available at: httpa://githubbcco
marrena/an aawaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
While previous works have focused on alignment, separabil-
2

---

Title Suppressed Due to Excessive Size
ity is also crucial consideration when comparing models of similar quality (e.g., different checkpoints from the same training run). However, achieving nigh-confidence separa- bility is challenging due to limitations in prompt design and inherent variances LLM evaluations. Overly simplistic prompts fail to distinguish between models, while the ran- domness in human and LLM judgments leads to inconsistent predictions. As result, it is often difficult confidently de- termine if a model's apparent performance reflects genuine difference in capability merely noisy observations, high- lighting a need for methods to verify whether benchmark can reliably separate similar models.
models #i > mh with disparate quality. Although two benchmarks A and B predict the same ranking x1 > m22 they predict P(T > T2) as .60 and .90, respectively (un- detectable by Spearman correlation). These benchmarks would result in very different Brier scores, reflecting their ability to quantify the magnitude of performance difference between the models. If both benchmarks give the wrong prediction of the winner, we prefer the benchmark with a less confident prediction. In other words, Brier score weighs benchmark's s accuracy and its ability to quantify the appro- priate level of uncertainty in its predictions. Background on Pair Rank Brier Score can be found in Appendix A.
Statistical measures like Pearson (Pearson, 1895) and Spear- man Correlations (Spearman, 1961), commonly used in benchmarks such as AlpacaEval (Li al., 2023) measure correlation to human preference ranking, may fail to ade- quately address model separability and ranking instability. In addition, these measures only provide coarse signal of ranking correlation without quantifying the magnitude of performance differences between model pairs.
While no single metric is intended to be individually suffi- cient, we claim that together, these metrics offer a robust framework for assessing benchmark performance, balancing the need for clear differentiation with alignment human preferences.
Key Prompt Qualities
Specificity: Does the prompt ask for a specific, well-defined output without leaving any ambigu- ity?
To address these shortcomings, we develop three novel met- rics: Separability with Confidence, Agreement with Confi- dence, and Pair Rank Brier Score.
Domain Knowledge: Does the prompt test the
Separability with Confidence quantifies the benchmark's s confidence by measuring its consistency in predicting the winner of model pair across random seeds through boot- strapping. This is done by calculating the percentage of model pairs that have non-overlapping confidence intervals of their benchmark scores. A higher percentage indicates that the benchmark is more confident in distinguishing be- tween the performance of different models, as the confi- dence intervals of their scores do not overlap.
AI's knowledge and understanding in a specific domain or set of domains?
Complexity: Does the prompt have multiple components, variables, or levels of depth and mu- ance?
Problem-Solving: Does the prompt require active problem-solving: analyzing and clearly defining the problem and systematically devis- ing and implementing solution?
Agreement with Confidence Interval measures how well benchmarks A and B confidently distinguish between two models with the same ordering. Given models Ti, π2. we assign scores based on:
Creativity: Does the prompt require a creative approach or solution?
1. If both benchmarks confidently separate Ti, T2, a score of 1 is assigned if their preference agree, and -1 if they disagree.
Technical Accuracy: Docs the prompt require an answer with a high degree technical accu- racy, correctness and precision?
2. If either A or B cannot separate T1, with confidence, we assign score of O.
Real-world Application: Does the prompt re- late to real-world applications?
The final agreement score the average across all unique model pairs. A score of implies perfect agreement with full confidence, while score of indicates complete dis- agreement.
4. The BenchBuilder Pipeline and Arena-Hard-Auto Dataset
Pair Rank Brier Score further assesses an LLM bench- mark's capability to predict the ranking of pair of compet- ing models by rewarding confidence in correct predictions while penalizing confidence when incorrect. Consider two
4.1. BenchBuilder
The core idea behind how BenchBuilder extract high-quality user queries from vast datasets is simple: each prompt
3

---

Title Suppressed Due to Excessive Size
Emoji Usage
Emoji Usage
Specificity
High Quality Benchmark Prompts
Crovdscourced Data
Python Coding
Domain Knowledge
Python Coding
Complexity
SQL Database
SQL Database
Cluster
Problem-Solving
Sample
Travel Planning
Travel Planning
/ Creativity
/ Technical Accuracy
Logic Puzzles
Real-world
Logic Puzzles
Topic Clusters
Key Qualities Filter
Hard Clusters
Figure 2. BenchBuilder Pipeline. Starting with live data source of crowdsoureed user prompts, we first cluster their embeddings to form topic clusters. LLM annotator then assigns quality scores based required skills. Clusters with low quality scores are filtered out, and we sample from the remaining high-quality clusters to create diverse and challenging dataset of benchmark prompts.
evaluated using quality score, and prompts with high scores are sampled evenly across diverse topics. Figure 2 illustrates our data creation pipeline.
4.2. Arema-Hard-Auto
We utilize the BenchBuilder pipeline curate 500 challeng- ing benchmark prompts for Arena-Hard-Auus Our process begins with an imitial pool of 200,000 prompts sourced from Chatbot Arena. We filter out duplicates, multi-turn conversa- tions, and non-English content. Next, we apply hierarchical topic modeling, clustering the prompts into 4,000 distinct topics from diverse domains.
To identify high-quality prompts, we define seven key quali- ties that capture the skills necessary to effectively address query, such specificity, domain expertise, and creativity (shown in Figure 2). An LLM--base annotator automati- cally scores each prompt by assessing how many of these qualities are present, producing "quality score". Further explanations and detailed instructions for these quality as- sessments are provided in Appendix C and Appendix G, respectively.
Then
GPT-4- (OpenAI, 2023b) judge to assign a "quality score" to each prompt and remove any prompts. Prompts with a score less than 6 and topic clus- ters with a mean score less than are discarded, ensuring only the highest quality prompts are retained. The resulting dataset contains over 500 high-quality clusters. construct 500-prompt benchmark, we sample prompts each from 250 randomly selected clusters. We also ensure the final dataset is free from personally identifiable information or offensive content.
To ensure our filtered prompts span wide range of tasks, leverage topic modeling approach using BERTopic. We first encode each prompt using OpenAl's embedding model,
(OpenAI, 2024a), reduce dimensions with UMAP, and apply a hierarchical-basee clustering algorithm (HDBSCAN) with a minimum cluster size This process generates distinct topic clusters. Each topic is then summarized and named using an LLM.
To validate qualities assigned by GPT-4-Turbo, we construct "ground truth" labels for 200 sampled queries by collecting majority votes from GPT-4o (OpenAI, 2024b), Claude-3. Opus, and Gemini-1.5-PP (Reid et al., 2024). GPT-4-Turbo achieves 85.6% agreement with these labels, demonstrating its reliability as an annotator. Additionally, we manually labeled a subset of the 50 queries out of the 200 validation set. We observed a 84.3% agreement rate between GPT-4- Turbo and human labels, and 14.9% agreement rate between the majority votes from LLMs and human labels.
Since some topic clusters predominantly contain trivial poorly defined prompts (e.g., "hi"), we retain only the clus- ters with high average quality scores and sample prompts evenly across these selected clusters. The resulting dataset consists of mostly well-defined, technical problem-solving queries as required in the above key criteria. Dataset statis- ties and further details on our filtering and sampling strategy provided in the following section.
We also applied BenchBuilder 150,000 queries from WildChat-IM (Zhao et al., 2024), which consist of di- verse and real-world conversations between users and Chat-

---

Title Suppressed Due to Excessive Size
GPT. BenchBuilder identified 185 high-quality clusters with 4,500+ prompts. We then randomly sampled 2 prompts from each of the highest-quality 125 clusters to create new benchmark, Bild-Hard-uuto which we show to have similar improvement in benchmark quality in section 6.4.
strong preference for model A, 5 indicates strong preference for model B). This scoring method penalizes models more heavily for large losses, effectively distinguishing perfor- mance across models. To ensure consisteney, we utilize chain-of-thought (Wei et al., 2023) prompting, guiding the LLM judge to generate its own solution before issuing a judgment. Detailed prompt templates are provided in Sec- tion G. To avoid potential position bias, we adopt a two- game setup - per query we swap the models the first and second position. We also study and propose solutions to mitigate potential stylistic biases, such as answer length, and self-bias in LLM-based evaluation in section 6.
4.3. Pipeline Cost and Statistic Analysis
The estimated cost for applying BenchBuilder on 200,000 Chatbot Arena queries using GPT-4-Turbbe as annotator is approximately $500 2, This cost can be significantly reduced employing
(Dubey et al., 2024) as an annotator instead, which only costs around $45 3. We experimented with -lama-3-70B-Imme annotator and observed similar improvement in downstream benchmark quality. Results are discussed in section 6.4.
an alternative
This results in 1000 judgments per model evaluation. Fol- lowing Chatbot Arena, we adopt the Bradley & Terry (1952) model to produce model's the final model scores. We aggre- gate all pairwise comparisons to the baseline model for all models and bootstrapping the comparisons to retrieve a boot- strapped confidence interval of all models' win-rate against the baseline, producing a ordered ranking of all models by their win-rates.
Figure 4 illustrates examples of topic clusters across a spec- trum of mean scores, Clusters with higher scores correspond to complex topics such as game development or mathemat- ical proofs, while lower-scoring clusters typically involve simpler or ambiguous questions (e.g., "Flirty Texting Strate- gies"). We provide further examples of prompts and their respective topic clusters in Appendix F.
6. Experimental Results
To see whether "quality score" assigned during Bench- Builder's pipeline correlates with separability and agree- ment, we sample 50 prompts per score and compare the re- sponses from GPT-4 and Llama-2-70b-Chal (Touvron et al., 2023), with GPT-4-Turbo as judge. In Figure 3 (Left), we observe strong correlation between high potential score and the win rate of GPT-4-Turre Turbo over 1aama---70b--hee Similar trends are observed across other model pairs, includ- ing Claude Sonnet vs Haiku and Mistral- (team, 2024) vs Mixtral (Jiang et al., 2024a).
6.1. Setup and Baselines
To compare Arena- Arena-Hard-ae Hard-Auto's separability and alignment with humans against other widely used benchmarks, MT- Bench (Zheng et al., 2023b) and AlpacaEval 2.0 Length Controlled (Dubois et al., 2024), we obtain 95% confidence intervals of model performances by applying 100 rounds of bootstrapping on judgment results for each benchmark. We use seed 42 for all experiments in this paper unless stated otherwise. For AlpacaEval, we use pre-existing results from their repository. We obtain MT-Bench judgment with no modification to their recommended evaluation setup. For Senn--HardAAuto, we employ the system proposed in sec- tion 5 by choosing gpt-4-0314 as a baseline for pairwise comparison.
5. Evaluation with LLM-as-a-Judge
Evaluating models on challenging queries such as Arena- Hard-Aute requires expert-level judgment due to the depth of domain knowledge and problem-solving skills involved. Expert evaluation, while ideal, is both costly and time- consuming. To address this, we leverage the LLM-as-a- Judge framework (Zheng al., 2023b; Dubois et al., 2023) as scalable alternative to approximate human preferences.
To ensure fair comparison, we use set of top-20 models on Chatbot Arena (Chiang al., 2024) (2024/04/13) that are also presented on AlpacaEEval leaderboard (2024/04/13) as ground truth for human preferences on the model ranking orders. See the list of models used in Appendix D.
We evaluate a model on a given prompt using pairwise comparison against a strong baseline model (e.g., GPT-4- 0314). A judge model (e.g., 4PT--Tuubo or Gemini-1.5. Pro) then scores each output by rating its preference between the pair on a 5-point Likert scale (Likert, 1932) (1 indicates
6.2. Comparing Separability and Alignment Across Benchmarks
In Table 1, Arena-Hard-Auto shows the highest separability (87.4) (7.4%) against widely adopted LLM benchmarks and of- fers highest agreement (90.8%) to Chatbot Arena at a $20 cost. In Figure 5, we show Arena-Hard-Aut offers signifi- cantly stronger separability against MT-Beench with tighter confidence intervals. With only 500 prompts, Arena-Hard- Auto achieve impressive alignment to (and even higher sep-
250 tokens per prompt on average x 200,000 user queries $10 per million tokens (OpenAl pricing for GPT-4-1
$250 tokens per prompt on average 200,000 user queries $0.9 per million tokens (TogetherAl pricing, date: 2024-10-01).
5

---

Title Suppressed Due to Excessive Size
GPT4 Llama2-70b
Sonnet Haiku
Mistral-Large Mixtral
90
85
70
80
Win-rate (%)
Win-rate (9)
65
75
Win-rate rate
70
60
65
60
55
50
0-1
2-3
4-5
6-7
0-1
2-3
4-5
6-7
2-3
4-5
6-7
Quality Score Intervals
Quality Score Intervals
Quality Score Intervals
Figure 3. Win-rate of three model pairs (GPT-4-0613 vs 2-0b--a-- Clande-3-Sommet222202222
Haail--3--aiuu202240000
Mistral-Large vs 317b--Isttuctt-v... over "quality score". We randomly sample 50 queries for each quality score 0-7 and bootstrap win-rate and confidence interval between model pairs cach score interval of 2. We observe a similar trend of win-rate between model pairs becomes increasingly separable as the quality score increases.
Arena Arena-Hard-Aun Hard-Auto
MT Bench
AlpacaEval 2.0 LC
Chatbot Arena
Confidence Agreement
90.9 %
26.6%
82.5%
N/A
Separability
87.4 %
22.6%
83.2%
85.8%
Spearman Correlation
93.2 %
89.9%
91.9%
N/A
Kendall Tau Correlation
80.0 % 0.069
64.2%
77.9%
N/A
Brier Score
0.09
0.11
N/A
Real-world
Yes
Mixed
Mixed
Yes
Freshness
Frequent Updates
Static
Static
Live
Eval cost per model
$20
$10
$10
Very High
Prompts per model
500
160
800
10,000+
Table We use a set top-20 models on Chatbot Arena (2024/04/13) that are also present on the AlpacaEval leaderboard to calculate separability and agreement per benchmark. We consider the human preference ranking by Chatbot Arena (English only) as the reference calculate agreement. See the list of models in Appendix D.
Wild-Hard-Hn
Wild-Randoms -50
larity of top LLMs. We present visual comparison between Arrda-Hard-Auto and MT-Bench in Figure 5, highlighting
Confidence Agreement 88.6%
36.4%
Separability
86.7%
75.6%
Hena-Haad-Autto improved separability.
Spearman Correlation
91.5%
45.5%
6.3. Comparing to similar distribution of human preference
Table 2. Comparing Wild-Hard-Auto and baseline of 250 prompts randomly selected from the WildChat dataset, using GPT- 4-Turbo as the judge. Wild-Hard-Aun has significantly higher separability and agreement to human preference ranking. The experiment demonstrates BenchBuilder's 's robustness general data curation pipeline across different datasets.
We evaluate Keea--Had-AAute with Chatbot Arena's En- glish Hard Prompt leaderboard as ground truth. Since this version of Chatbot Arena leaderboard is based on votes from more challenging subset of the overall Chatbot Arena bat- tles, we believe a more in-distribution comparison for which also consist of challenging user queries. We observe Arena-Hard Hard-Aute achieves an overall higher alignment (98.6% Confidence Agreement and 96.7% Spearman Correlation) to human preferences. Results are presented in Appendix Table 9.
arability than) Chatbot Arena Rankings, which constitutes over 1 million real-world human preferences.
Notably, we observe significant gap between MT-bench's Spearman Correlation (89.9%) and confidence agreement (22.6%) to Chatbot Arena, an example where Spearman Correlation fails to account for variance the rankings, and hence cannot adequately measure important ranking granu-
6

---

Title Suppressed Due to Excessive Size
Arena-Hard-Auto (Style Control)
Arena-Hard-Auto
AlpacaEval 2.0 LC
MTT-Bench
Confidence Agreement
98.6%
94.4%
83.8%
30.3%
Separability
86.8%
87.4% %
83.2%
22.6%
Spearman Correlation
98.6 %
94.9%
88. 1 1
90.7%
Kendall Tau Correlation
93.7% %
85.3%
70.5%
77.9%
Table 3. We apply style control to Chatbot Arena battles (English Hard Prompts) and use its model ranking as reference to calculate alignment. When stylistic confounders like response length controlled, Hard-Auto achieves high alignment to human preferences.
Model
GPT4-T
Claudc3-Opus Geminil 1,5-Pro
Llama3-70B
Ensemblc-as-Jun Judgggs
Confiderence Agreement
90.9%
66.7%
84.8%
65.6%
91.5% %
Separability
87.4%
83.68%
82. 11.11 1%
81.6%
89.5%
Spearman Correlation
93.2%
77.0%
95.2%
70.5%
96.5%
Brier Score
0.069
0.170
0.064
0.196
0.065
Table Statistics of gena-Haad-AAus with four LLM different judges: GPT4-T (gemini-1.5) 1.5--------p0514 Llama3-70B (lama-3-70b-instruet) We compare rankings produced by these judges against Chatbot Arena (English) ranking (as 2024/04/13). We observe GPT-4T and Geminil.54. Prr have higher agreement than Claude-3-Opus and Llama-3- 70B. Furthermore, the ensemble of GPT4-T and Gemini 1.5-Pro shows even higher agreement.
Claude-3-Opus. Geminil.5.
6.4. Robustness and Generalizability
6.5. Mitigating Stylistic Biases in LLM-Based Evaluation
To evaluate the robustness and generalizability of the Bench- Builder pipeline, we applied on 150,000 WildChat (Zhao al., 2024) dataset and identified 185 high quality clusters with 4,500+ prompts. We then randomly sample 2 prompts from each of the highest-quality 125 clusters create benchmark, Wild-Hard-Auto. We compare Wild-Hard- Auto and a baseline of 250 prompts randomly selected from the WildChat dataset in table 2. Results indicates Wild-Hard- Auto has significantly higher separability and agreement to human preference ranking than random baseline, demon- strating BenchBuilder's robustness as general data curation pipeline for various crowdsourced datasets.
LLM-as-a-Judde based evaluation is known to suffer from various biases, such favoring longer responses (Zheng et al., 2023b; Dubois et al., 2024). AlpacaEval 2.0 Length Control (Dubois et al., 2024) proposes regression based approach to control length bias in LLM-based evalua- tion. Chatbot Arena also released a style controlled leader- board (Li et al., 2024), which attempts to decouple sub- stance from stylistic preferences, including answer length and markdown usage. Following their approaches, we mod- ify how Arena-Hard-Aun computes the model scores by accounting for the stylistic differences between two answers as additional features to the existing Bradley-Terry model.
Additionally, we compared Arena- Hard-Auto against two separate sets of 500 randomly selected prompts from the Chatbot Arena dataset, prior to applying the pipeline ex- traction. We observe performs both random baselines. Results are shown Ap- pendix 7.
We propose controlling for a similar set of stylistic elements used to control human preference on Chatbot Arena for LLM-based evaluation: answer token length, density of markdown headers, markdown bold elements, and mark- down lists. Technical details on how extend the Bradley- Terry model for controlling any given style can be found in Appendix B.
Hard-Auto significantly out-
To verify whether BenchBuilder is not limited to GPT- 4-Turbo annotator for prompt qualities, we employed Llama-3-70B-Instrucl as an alternative annotator for prompt curation. We observe the benchmark produced by Llama- 3-70b-instruct as the prompt annotator has similar improve- ment in quality as Arena- Arena-Hard-Ae Hard-Auto from random baselines. Results are shown in Appendix Table 8.
We apply style control to Chatbot Arena battles and compare the resulting model preference ranking to style controlled Frena-Harde Auto, aiming to answer the question: How well aligned is Hard-Auto to human preference when both human preference and L.M judgment are decoupled from stylistic differences? In Table 3, we show that style con- trolled Maina-Hard-AAiee Auto achieves 98.6 % agreement and correlation to style controlled human preference ranking,
7

---

Title Suppressed Due to Excessive Size
Table 4. No Moditication
Table 4. Style Control
Model
Score
Model
Score
53.5
41.7
1----00-IInntruc-mmt
44.9
Llama-3. 1--ma--
1770B-Instructnoomm
39.9
1i-ma-3.--70--Inttii
44.5
39.8
44.3
39.5
37.5
34.9
Table Comparison Between Arena-Hard-Auss with no modification versus applying style control. Left: Arena-Hard-Auto with no modification to GPT-4-Turbo judge. Right: style controlled GPT-4-Turbo judge. Asking -lama-3.1-70--mme -nstrurttuctt (Dubey et al., 2024) to response with more detail shows significant performance gain when no style control is applied. However, is no longer favored with style control. Full table with additional models and system instructions can be found in Appendix Table 6.
suggesting Arena-Hard-Aute assessment of model strength separated from style still highly aligned humans.
7. Limitations
While our data sources are drawn from diverse distributions, biases may still exist in our pipeline. For instance, the seven defined qualities may not fully capture the range of possible attributes, potentially skewing towards prompts in technical domains. Furthermore, evaluation for multi-turn and non-English interactions due to the limited availability of multi-turn data in crowdsourced datasets and the primary language proficiency of the authors.
Additionally, conducted experiment trying to increase model score on Harn--HardAAuo by instructing GPT-3.5- and Gemini-1.5-Flash to increase the verbosity and usage of markdown elements in their response and present our results Table 5. While increasing "detailedness" does increase model performances on Arena-Hard-Aie when modifications is applied to GPT-4-Turbo judge, applying style control is effective at neutralizing this advantage. Our results shows that style controlled model scores cannot be gamed via manipulating response length or markdown usage on Arena-Hard-Aun We also observe a reduction in correlation between model score and answer length on Arena grena-Hard-Aree Hard-Auto. Full results can be found in Appendix Table 12.
Turbo,
Srena-Hard-Ale
currently lacks
address these limitations, future work will focus on ex- panding BenchBuilder to incorporate multi-turn and mul- tilingual data curation. We also aim to refine our prompt quality definitions, creating more systematic approach for generating benchmarks that reflect broader, more inclusive range of scenarios while maintaining high separability and alignment with human judgment. We also plan to explore more advanced version of Ensemble-as-Judgee to further enhance our LLM-based evaluation approach.
6.6. Mitigating Self-Biases in LLM-Based Evaluation
evaluations are also known to exhibit self- bias. While such biases should manifest lower alignment with human preferences in proposed metrics, we con- duct focused analysis to further understand and address this issue. Since Arena-Hard-Ae Auto uses GPT-4-Turbo as the default judge, we evaluate whether it favors OpenAl mod- els over Anthropic models. Results in Appendix Table 10 indicate that GPT models receive slightly higher average rankings than human preference, while Claude models rank lower.
8. Conclusions
We introduced BenchBuilder which transforms crowd- sourced data into high-quality benchmarks. This pipeline en- ables building challenging and evolving benchmarks which is crucial for evaluating today's advanced language models. Our evaluation metrics, including separability and agree- ment with confidence, provide a comprehensive assessment of benchmarks. We show the resulting benchmark, Arena- Hard-Aur Auto, significantly improves separability and align- ment with human preferences over existing benchmarks, achieving 98.69 6% agreement with Chatbot Arena rankings at only $20 per evaluation. We expect Arena-Hard- Auto to be useful for LLM developers to evaluate their models with confidence and BenchBuilder to be valuable tool for developers seeking to extract high-quality benchmarks from vast amounts of data with minimal human effort.
To reduce this bias, we propose Ensemble-as-Judges which aggregates judgments from multiple models. The ensemble judges (GPT-4-Turbo and G.5-ni-1.5-Pro) achieves overall higher separability and alignment with human rankings, as shown in Table 4. Additionally, we also observe that combining GPT-4-Turbo and Gemini-1.5-PPr reduces self- biases. Results can be found in Appendix Table 10. We believe further research into ensemble methods can refine these results and leave this for future exploration.
8

---

Title Suppressed Due to Excessive Size
Impact Statement
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
This paper presents work whose goal is to advance the field of Large Language Model evaluation. There are many po- tential societal consequences of our work, none which we feel must be specifically highlighted here.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L.. Zhuang, S., Zhuang, Y., Gonzalez. J. E., Stoica, I., and Xing, E. P. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:/ lmsys.orrrr. org//log/ 2023- 00000033000usm-
References
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. 14219, 2024.
Chiang, W.-L.., Zheng. L.,, Sheng, Y., Angelopoulos, N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., and Stoica, I. Chatbot arena: An open platform for evaluating llms by human preference, 2024.
:, Young, A., Chen, B., Li, C., Huang, C., Zhang, G., Zhang, G., Li, H., Zhu, J., Chen, J., Chang. J., Yu, K., Liu, P., Liu, Q., Yue, S., Yang. S., Yang. S., Yu, T., Xie, W., Huang, W., Hu, X., Ren, X., Niu, X., Nie, P. Xu, Y., Liu, Y., Wang, Y., Cai, Y., Gu, Z., Liu, Z., and Dai, Z. Yi: Open foundation models by O1.ai, 2024.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14/68, 2021.
BeepSeek-AT, Zhu, Q., Guo, D., Shao, Z., Yang. D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma. S., Zeng, W., Bi, X., Gu, Z., Xu, H., Dai, D., Dong, K., Zhang, L., Piao, Y., Gou, Z., Xie, Z., Hao, Z., Wang, B., Song. J., Chen, D., Xie, X., Guan, K., You, Y., Liu, A., Du, Q.. Gao, W., Lu, X., Chen, Q. Wang, Y., Deng, C., Li, J., Zhao, C., Ruan, C., Luo. F., and Liang, W. Deepseek- coder-v2: Breaking the barrier of closed-source models in code intelligence, 2024. httst /aaxivvorgg/ abs / 2406..100 11931.
Bos, J. and Markert, K. Recognising textual entailment with logical inference. In Mooney, R., Brew, C., Chien, L.-F, and Kirchhoff, K. (eds.), Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp- 628 635, Vancouver, British Columbia, Canada, October 2005, Association for Computational Linguisties. URL https: / acl anthologyyy....... org/ 1105-10792
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. A large annotated corpus for learning natural language inference. In Mårquez, L., Callison- Burch, C., and Su, J. (eds.), Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Com- putational Linguistics. doi: 10. 10.18653/11/D15. 1075. URL https: http///aaaaaaaaaaaeeeee org/D15 1075.
Dolan, B. and Brockett, C. Automatically constructing corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005), 2005.
Dua, D., Wang, Y., Dasigi, P., Stanovsky, G., Singh, S., and Gardner, M. Drop: A reading comprehension benchmark requiring discrete reasoning
paragraphs. In Pro- ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368-228822222 2378, 2019.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. 1952.
Biometrika, 39(3/4):324-33))
Brier, G. W. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1-3, 1950.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Cai, 7., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report. arXiv preprint arXiv:2403. 17297, 2024.
Dubois, Y.. Li, X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpaca- farm: A simulation framework for methods that learn from human feedback, 2023.
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert- Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp.
Dubois, Y., Galambosi. B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: A simple way to debias
2021.

---

Title Suppressed Due to Excessive Size
automatic evaluators. arXiv preprini arXiv:2404.04M 04475, 2024.
Jimenez, E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-hench: Can language mod- els resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https:: hetas///ppprree wwtpp/pppneeeeeeeeeeeeeeeeeeeeee net/ forum? id-VIF8yNQM6..
Frick, E., Jin, P., Li, T., Ganesan, K., Zhang, J., Jiao, J., and Zhu, B. Athene-70b: Redefining the boundaries of post-training for open models, July 2024. URL ht tts: / //buggingface. co. ce oow wtth/tttheeeeteeeeeeeeeeeeee 70B.
Karpinska, M., Akoury, N., and lyyer, M. The perils of using Mechanical Turk to evaluate open-ended text gen- eration. In Moens, M.-F., Huang, X., Specia, I.., and Yih, S. W.-t. (eds.). Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 1265-1285, Online and Punta Cana, Domini- can Republic, November 2021. Association for Computa- tional Linguistics. doi: 10.18653/v1/20211.mm 97. URL https: http///aaaaaahhhhe wrgp://oooooor///0// emn emnlp-mat 1...maiin990 97.
Gemini, T., Anil, R., Borgeaud. S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1805, 2023.
GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Rojas, D., Feng, G., Zhao, H., Lai, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.,, Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. a.
Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., Ma, Z., Thrush, T., Riedel, S., Waseem, Z., Stenetorp, P., Jia,
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E.,, Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurlPS, 2021b.
R., Bansal, M., Potts, C., and Williams, A. Dynabench: Rethinking benchmarking in nlp. NAACL, 2021.
Li, T., Angelopoulos, A., and Chiang, W.-L. Does style mat- ter? disentangling style and substance in chatbot arena, August 2024. URL https: / /blog. lmarena. ai/ blog/ 20og/20000 /202 cttll
Ivison, H., Wang, Y., Pyatkin, V., Lambert, N., Peters, M., Dasigi, P., Jang, J., Wadden, D., Smith, N. A., Beltagy, I., and Hajishirzi, H. Camels in a changing climate: Enhancing im adaptation with tulu 2, 2023.
Li, X., Zhang, T., Dubois, Y. Taori, R., Gulrajani, I. Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae- val: An automatic evaluator of instruction-folloiinn models. https: /ttps://gittee /ttptggtthh/.//////// wow/:atsutataaaaabbaaaaa. a alpaca_eval, 2023.
Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F. Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code. arXiv preprint arXiv:2403.07974 2024a.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:221 10, 2022.
Jain, N., Shetty, M., Zhang, T., Han, K., Sen, K., and Stoica, R2e: Turning any github repository into programming agent environment. In ICML, 2024b.
Likert, R. A technique for the measurement of attitudes. Archives of psychology, 1932.
Jing,, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M.-A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang. T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024a.
Lin, B. Y., Deng, Y., Chandu, K., Brahman, F., Ravichander, A., Pyatkin, V., Dziri, N., Bras, R. L., and Choi, Y. Wild- bench: Benchmarking liss with challenging tasks from real users in the wild. arXiv preprint 2024.
Jiang, A. Q. Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, I.. R., Saulnier, I.., Lachaux, M.- A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T. L.., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral of experts, 2024b. URL wwwrriiv...o.vveoooooooe abs 2201 04088.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang. K., Zhang, S., Deng, X.. Zeng, A., Du, Z.,, Zhang, C.,, Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents, 2023.
Ni, J., Xue, F., Yue, X., Deng, Y., Shah, M., Jain, K., Neu- big, G., and You, Y. Mixeval: Deriving wisdom of the
10

---

Title Suppressed Due to Excessive Size
crowd from IIm benchmark mixtures. arXiv preprini arXiv:2406.06665, 2024.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonse A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023.
OpenAL. Gpt-4 technical report. arXiv preprint
arXiv:2303.08774, 2023a.
OpenAI.
New models and developer products an- nounced at devday. https: www.pppaaaa....looopooo/oooooeoo
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, new-models und-devvlopedd -awd--ddeeooeeppduucceeeee product announceed, Bhupatraajy Hussenot, L., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXi::2408.00118, 2024.
2023b. (Accessed 06/05/2024).
OpenAl. New embedding models and api up- https://openai.c cwwp:/aie..ooi..///eeeeeeeeaee/e new-embedding--oote -ng- 2024a. (Accessed on 06/05/2024).
dates.
team, M. A. Au large. https: httpteettttt////ttttttttttttte
httpps//mmmttraaaaa/eeeeeee mistral-large. 2024. (Accessed on 06/05/2024).
OpenAL. Hello gpt-4o. attps://opentt com/
Team, Q. Introducing qwenl 1.,, February 2024a. URL httpps:/qwenim...ge github. io/blog///////oo http////gg/g/nn
index/hello 1ollo--pp-4//, 2024b. (Accessed 06/05/2024).
5/.
Pearson, K. Note on regression and inheritance in the case of two parents. Proceedings of the Royal Society of London, 58:240 242, 1895. ISSN 03701662. URL http:// www jstorrroo...ooo .tpooro.ooooooo wttpp//t/gtalllroeeeeeoo http:/abblee11119/0
Team, T. M. R. Introducing dbrx: A
new state-of-the--e open 1lm.
https:
/ / /www.. /ttp:./aaabbtt batabricks com/blog/
int Introducing- wbrx-new wwweewttttteee www.sstttttteaeee sto 11m/, 2024b. (Accessed on 06/05/2024).
Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang. P. Squad: 100,000+ questions for machine comprehension of text. 2016.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint MrXiv:2307.0928. 2023.
Reid, M., Savinov, N.. Teplyashin, D., Lepikhin, D., Lilli- crap, T., Alayrac, J.-b., Soricut, R., Lazaridou, A., Firat, O., Schrittwieser, J., et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context. arXiv preprint arXiv2203.005300, 2024.
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2019.
Rein, D. Can good benchmarks contain mistakes?, 2024. URL hutps: /arg/ can-good www.goo-bbnccaaaa--cotttaaaiiiaa
Wang. Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A.. Khashabi, D.. and Hajishirzi, H. Self-instruct: Align- ing language models with self-generated instructions. In Rogers, A., Boyd-oddoaoo Grabbe,, J., and Okazaki, N. (eds.), Pro- ceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 13484-1350$ Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10.18653/v1/ 2023.acl-long 754. URL wwwpt:ttt//aaeae http://llaaanooooy....oo org/2023 acl-log.. 754.
Rein, D., Hou, B. L., Stickland, A. C., Petty,, J., Pang, R. Y.,
Dirani, J., Michael, J., and Bowman, S. R. Gpqa: A
graduate-level google-proof q&a benchmark, 2023.
Sainz, O., Campos, J., García-Ferre Ferrero, I., Etxaniz, J., de La- calle, O. I.., and Agirre, F. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10. 00.18653/11/0223..nding..
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q.. and Zhou, D. Chain-of-thoughh prompting elicits reasoning in large language models, 2023.
722. URL https: http://aclannnn http//aaaannttoooooooe orwptttooooooooooo222202... findings-emnipp 722.
White, C., Dooley, S., Roberts, M., Pal, A., Feuer, B., Jain, S., Shwartz-Ziv, R., Jain, N., Saifullah, K., Naidu, S., et al. Livebench: A challenging, contamination Ilm benchmark, arXiv preprint arXiv:2406. 19314, 2024.
Spearman, C. The proof and measurement of association between two things. The American Journal of Psychology, 1961.

---

Title Suppressed Due to Excessive Size
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv: 10671, 2024.
Yang. S., Chiang, W.-L., Zheng, L.., Gonzalez, E., and Stoica, Rethinking benchmark and contamination for language models with rephrased samples, 2023.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800, 2019.
Zhao, Ren, X., Hessel, Cardie, C., Choi, Y., and Deng, Y. Wildchat: Im chatgpt interaction logs in the wild. International Conference on Learning Representations, 2024.
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, E., and Stoica, I. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirny-eventhh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a. URL hps::: //openreview. net. forum? dduccHHPDlaa.
Zheng, L.., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z. Li, D., Xing, E. P., Zhang, H.. Gonzalez, J. E., and Stoica. I. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurlPS, 2023b.
Zhong, W. Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centre benchmark for evaluating foundation mod- els. arXiv preprint 2r34v:2304. 06364, 2023.
Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloiinn evaluation for large language models. arXiv preprint orXiv:2311.073 2023.
Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, Starling- 7b: Improving I1m helpfulness & harmlessness with rlaif, November 2023.
Zhuo, T. Y., Vu, M. C., Chim, J.. Hu, H., Yu, W., Widyasari, R. Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brun- ner, S., Gong, C., Hoang, T., Zebaze. A. R.. Hong, X., Li, W.-D., Kaddour, J., Xu, M., Zhang, Z., Ya- dav, P., Jain, N., Gu, A., Cheng, Z., Liu, J., Liu, Q., Wang, Z., Lo, D., Hui, B., Muennighoff, N., Fried, D., Du, X., de Vries, H., and Werra, L. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL https: //arxiix org. abs /24/. 15877.
12

---

Title Suppressed Due to Excessive Size
A. Pair Rank Brier Score
Bootstrapping is a well-established statistical technique for estimating the distribution an estimator by sampling with replacement from the original dataset. This approach has become increasingly popular for constructing confidence intervals in LLM leaderboards, such as Chatbot Arena (Chiang et al., 2024). In our proposed evaluation metrics in section 3, such as Separability and Agreement with Confidence Interval, reliable confidence interval estimation is essential for assessing the performance stability of different models given benchmark. Moreover, for metrics like the Pairwise Rank Brier Score, estimating the probability distribution of rank-based model performance is critical. Therefore, applying bootstrapping to the given benchmark provides a straightforward and robust solution for these tasks.
and a scoring function that measures the perfor- on this dataset. Let D denote bootstrap sample of D, and let (T; D*) denote the bootstrapped performance score for model π. using the dataset D For simplicity, use (*;) to denote ( D*).
Consider a benchmark consisting a dataset D { ,.,,...,, , DD mance of models æ1, "....., Tn
To use Brier Score (Brier, 1950) for measuring the accuracy of the given benchmark's ' probabilistic predictions model performances, we need to compute the forecasted probability that model я; performs lower than π on the ground truth measurement for every model pair.
(T,) (Tz)))
The bootstrapped scores f (m;) and (TT) follow an empirical distribution that can be approximated using the Central Limit Theorem (CLT). In most cases, the distribution of ('((T:) converges asymptotically a normal distribution, which also observed in our experiments. Formally, (*;) ~ (N, (Ms,.o)). where μ4 and of are the bootstrapped mean and variance, respectively. When this normality assumption does not hold, P(f* (T;) < f (Tr)) can still be estimated from the empirical distribution of the bootstrapped scores.
Let Oz OR.-ER denote the ground truth outcome for the model pair (Ta, Ty), where:
O m (T; performs worse than Ty the ground truth evaluation metric)
The Brier Score Loss is then calculated over the benchmark's prediction for each model pair with respect to the ground truth outcome O
Σ (t) < - OxLexx)
(3)
N
where N is the number of model pairs.
13

---

Title Suppressed Due to Excessive Size
B. Style Control in Model Evaluation
To mitigate the potential confounding effects of response style on model evaluation, we implemented an enhanced Bradley- Terry regression framework. This method, inspired by recent LLM evaluation technique (Dubois al., 2024), controls the influence of answer length on judges' preferences. Recently, Chatbot Arena implemented style control (Li et al., 2024) to decouple substance from style in their leaderboard. This approach incorporates style-related features, such answer length, into the regression model, enabling distinction between model's intrinsic capabilities and the influence of these potential confounders like answer style. In essence, style control answers the question: What would the preference be if everyone has the same style? This distinction is crucial for a more accurate assessment of model performance without biases.
extend the standard Bradley-Terry model by introducing additional style features. Let n denote the number of pairwise comparison battles and M the number of models. For each battle E [n]. we define:
X; E IRMM; s,m = if model m on the presented first to the judge, X = -1 if presented last, and 0 otherwise.
Y E 0,1: The outcome, where indicates the first model won.
Z, E R vector S style features for the comparison.
The traditional Bradley-Terry model estimates model strengths 3 E RRM through logistic regression:
1
3
arg min BERM 72
BCELoss(sigmoid(XX (), Yi)
(4)
Our enhanced model incorporates style coefficients 7 E S:
B,4 arg
min BERM yERS n
BCHLoss(iigmoi((X/ B Z4 Yi)
(5)
where BCELoss represents the binary cross-entrope loss. We selected the following style features:
Answer token length
Density of markdown headers, markdown bold elements, and markdown lists.
For each feature, we compute a normalized difference
ffeature feature
normalize
(6)
feature A feature
This normalization technique accounts for the relative difference features between responses. For instance, the token length difference is normalized
|engthh a length
normalize
length length,
We chose this approach over alternatives like the hyperbolic tangent normalization used in AlpacaEval
length length 53
tanh
(8)
(length, length,;))
Our method better captures proportional differences, especially in cases where absolute differences may be misleading (e.g.. 500 vs. 520 tokens compared to 20 vs. 40 tokens).
The resulting 3 coefficients represent model strengths controlled for style effects, while f quantifies the impact of each style feature on human preferences. To facilitate meaningful comparisons, we normalize the style coefficients. Our analysis revealed that response length was the most influential style factor, with other markdown-relade features having secondary effects.
14

---

Title Suppressed Due to Excessive Size
C. 7
Key Qualities
Here we elaborate on why the key qualities were chosen. Note that we chose these 7 qualities specifically to curate Krena-Hard-Autor they have the desired property that increasing the number of qualities satisfied in set of prompts increases the separability between strong and weak LLMs (see Figure 3).
Specificity: Prompts must be specific and unambiguous clearly assess model's ability to precisely follow instructions. If prompts are vague, model responses may vary significantly, making benchmarking unreliable due to inconsistent interpretations.
Domain-speciffe prompts test whether a model possesses a depth of knowledge and understanding in particular areas. Without satisfying this quality, benchmarks risk being too general or trivial, failing to differentiate models based on actual expertise.
Domain Knowledge:
Complex prompts, characterized by multiple components nuanced aspects, challenge the model's ability to process and respond coherently under intricate conditions. If complexity is lacking, the prompts would fail to differentiate advanced capabilities among sophisticated models.
Complexity:
Prompts requiring active problem-solving force models to demonstrate critical thinking, reasoning, and systematic approaches. Without this, benchmarks might merely evaluate superficial knowledge recall rather than genuine cognitive skills.
Problem-Solving:
Ereativity-oriinter prompts examine model's capability to produce novel and original responses, which is essential in real-world, open-ended interactions. Without assessing creativity, benchmarks neglect important aspects of model performance related to innovation and flexible thinking.
Creativity:
Technical Accuracy: Accuracy-focused prompts validate that models provide reliable, correct, and precise answers. Failing to ensure technical accuracy would reduce benchmark reliability, as responses could seem superficially correct but be fundamentally flawed misleading.
Real-world Application: Prompts aligned with real-world scenarios measure the practical utility of model outputs. Benchmarks devoid of real-world applicability risk being disconnected from actual user needs, thus becoming irrelevant for practical assessments.
By satisfying these qualities, we believe prompts become robust and effective benchmarks, reliably differentiating model capabilities and aligning closely with human judgment and practical usability. The resulting benchmarks, Arena-Hard-Auto. Wild-Hard-Autoo Llama-Hard-Auu all demonstrate significantly better alignment and separability than baselines (see Table Table Table 8).
D. Models for Experiment
The top 20 models on Chatbot Arena (2024/04/13) that are also present on the AlpacaEval leaderboard for calculating sepa- rability and agreement: gpt-4-turbo-2224-04-00 2023a), gpt-4-0613, mistral--arge-2402,, qwen gwwnl.5-72bbb............ -5-72b-chht (Team, 2024a), mistral-medium, claude-2.0, gpt-3.5-turbo--e 2laude-2.1, gemini-pro (Gemini al., 2023), mixxral-&x7b-instruee chat (AI et al., 2024), tulu-2-dpo-70b (Ivison et al., 2023), dbrx-instruct-previiew (Team, 2024b), vicuna-33b (Chiang et al.,
(laue=3--puu-222240222 claude-3-sonnet 2-ssomtt-20240222, gpt-4-0314 (OpenAI,
(Jiang et al., 2024b), gpt-3.5-turbo-033 yi-34b-
2023), starling- 1m-70 -n-abaalpha (Zhu et al., 2023), llama-2 Mlma--2-70b-hhar (Touvron et al., 2023)
15

---

Title Suppressed Due to Excessive Size
E. Tables and Figures
Python Game Development
GPT-4-Tubbo-0409
Prime Numbers and Proofs
Claude 3 Opus
Chemical Equilibria and Reactions
GPT-4-0314
Calculus Essentials & Applications
Claude Sonnet
Atomic and Electronic Structure
Claude Haiku
LLM Prompt Engineering
cluster
Model
Llama-3-70B-Instuuct
Biblical Studies Interpretations
GPT-4-0613
Banking Operations
Mistral Large
American English Vocabulary
Mixtral-ox7B-Instrucc
Relationship Challenges and Advice
Emoji Usage and Interpretation
GPT-3.5-Turbo-0125
Diverse Gilt-Giving Ideas
Llama-3-8B-Instrucc
Flirty Texting Strategies
Llama-2-70B-
2
3
4
5
2
6
10
Mean Score
Score
Figure 4. Mean score of various topic clusters in descending order. Higher-scoring clusters correlate to challenging topics. A more complete topic cluster plot is in Figure 6.
Figure 5. Comparison between Arena- Arena-Hard-Aun (Green) and MT- Bench (Grey). The former offers significantly better separability between models and tighter confidence intervals.
Model
Meen-HHad-AAute
Random Sample
Random Sample
Confiderence Agreement
84.2% %
57.5%
66.1 66.1%
Separability Spearman Correlation
80.5 %
74.7%
76.3%
94.7 %
64.7%
72.5%
Brier Score
0.069
0.215
0. 0.162
Table 7. We compare evaluate the set top-20 models compare various statistics across. Each prompt judged only once by positioning the baseline answer first.
with two sets of 500 prompts randomly sampled from 75K Chatbot Arena user queries. We
-lat
Random
Random 2
Arena-HHard-Auto55000
Confidence Agreement
86.0%
55.8%
58.1%
88.4%
Separability
84.4%
68.9%
88.9% 70.9%
64.4%
Spearman Correlation
96.4%
73.3%
96.4%
Table Comparing Llama-Harde Hard-Auto against random baselines on 10 of the 20 models outlined the paper. We observe similar improvement in benchmark quality, suggesting BenchBuilder is robust across different choices of LLM annotators.
16

---

Title Suppressed Due to Excessive Size
Arena-Hard-Auto
Confiderence Agreement
98.6%
Spearman Correlation
96.7%
Kendall Tau Correlation
87.4%
Brier Score
0.055
Table 9. We compare Arena-Han Atena-Hard-Aee Hutd-Auto (gpt-4-1 1009-4-1006--- preview as judge) to Chatbot Arena Category Hard Prompt (English) on the same set of top-20 models. By comparing Arena Arena-Hard-Ar Hurd-Auto to challenging distribution queries from Chatbot Arena, obtain even higher alignment to human preferences.
Table 9. OpenAl GPT Series
Table 9, Anthropic Claude Series
GPT-4-turbo
Ensemble
GPT-4-turbo
Ensemble
gpt-4-turbo
0
claude-3-opus
0
gpt-4-0314
1
claude-3-sone 3-sonnet
-1
-1
gpt-4-0613
0
-2
claude-2.0
-2
0
gpt-3.5-turbo-0613
I
-1
claude-2.1
-1
3
gpt-3.5-turbo-0314
I
column average
0.6 -0.4
column average
-0.8
0.4
Table 10. Comparing bias in GPT-4-Turbo Judge and Ensemble-as-Judee We calculate the ranking shift by comparing the human preference ranking (by Chatbot Arena Category Hard Leaderboard) and LLM-judge ranking on OpenAl GPT Series and Anthropic Claude Series. Results show both methods have relatively small shifts, but Ensemble-as- produces more balanced rank difference than GPT-4-Turbo Judge, suggesting smaller self-bias than single LLM as Judge.
Quality Score
1+
2+
3+
4+
5+
6+
7+
% of queries
95.4
83.5
61.9
48.7
33.8
17.9
0.2
Qualities
Specificity
Domain Domain-knowlede knowledge
Complexity
Problem-solvineg
Creativity
Toch. Accuracy
Real-world
of queries 57.3
63.4
35.0
34.9
26.1
39.0
87.9
Table 11. First row is the percentage of queries with quality scores of the column more in 75K Chatbot Arena data assigned by GPT-3.5-Turbo. Second row the percentage queries 75K Chatbot Arena labeled by GPT-3.5-Turbo with the qualities.
Table 1I. Avg. Token Length
Table II. Naive Verbose Policy
Pearson
Spearman
Pearson
Spearman
No Modification
0.364
0.125
No Modification 0.397
0.165
Style Control
0.193
-0.025
Style Control
0.231
0.028
Table 12. Left: Comparing correlation between model score and average token length between GPT-4-Turbo as Judge with no modification versus style controlled. Right: Comparing correlation to model score produced "verbose policy", judge which always picks the longer response. In both cases, style control effectively reduces the correlation to verbosity.
17

---

Title Suppressed Due to Excessive Size
Model Name
Win Rate
CT Interval
Average Token
Maude-335-Sommet222244000
79.3
(-2.1, 2.0)
567
GPT-40-2024-05-13
79.2
(-1.9, 1.7)
696
GPT-4-0125-Previeew
78.0
(-2.1,
619
GPT-40-2024-08-06
77.9
(-2.0,2.1)
594
Athene-70B
77.6
(-2.7,2.2)
684
GPT-40-Mini
74.9
(-2.5,
668
Gemini-I.-Proo-PPPPPees
72.0
(-2.1,2.5)
676
Mistral-Large-2407
70.4
(-1.6,
623
LLaMAA331-40SB-nstttee
69.3
(-2.4,
658
GLM-4-0520
63.8
(-2.9,
636
Yi-Large
63.7
(-2.6,
626
DeepSeek-Coder-VV V2
62.3
1.2.1,
578
Clude-3-Opus-22224029
60.4
(-2.5,2.5)
541
Gemma-2-27B-IT
57.5
(-2.1,2.4)
577
LLaMIA-3.1-70B-Instuus
55.7
(-2.,227)
628
GLM-4-0116
55.7
(-2.4,
622
GPT-4-0314
50.0
(0.0,
423
Semmin-L---Fush-AAPPPrrvie
49.6
(-2.2,
642
Qwen2-72B-instruee
46.9
(-2.5,2.7)
515
Claud-3-Sommee2222402299
46.8
(-2.3,27)
552
LLaMAA3--0B-Instruet
46.6
(-2.3,2.6)
591
Claudb-3-Haakkk220244077
41.5
(-2.,,2.5)
505
GPT-4-0613
37.9
(-2.8,
354
Mistral-Large-2402
37.7
(-2.1,2.6)
400
Mixtral-8222B Innttrcct- V0.1
36.4
(-2.4,2.6)
430
Swenl_5-72B-CChe
36.1
(-2.0,2.7
474
4K-MMedium--4-Instteeee
33.4
2-1.6,2.1)
517
Mistral-Medium
31.9
(-1.9,222
485
IntemnLM22.520BB-hh.
31.2
(-2.4,
576
Phissssmall8KK-Insruutt
29.8
(-1.8,
568
GPT-3.5-Turbo-06133
24.8
(-1.9,
401
DBRX-Instruct-Previeww
24.6
2-2.0,
415
InternL.M2-20B-CChat
24.4
2-2.0,
667
Mixtral-8x77--nsttrncc. V0.1
23.4
1.2
457
GPT-335-TTuro-00125
23.3
(-22,119)
329
Yi-34B-Chat
23.1
(-1.6, 1.8)
611
Stariing-LM-7B-Btta
23.0
(-1.8, 1.8)
530
LLaMA-3.1-8B-Intee
21.3
2.2)
861
Shorkel-MiismllPPanRMM---
20.7
2.2.8,
564
LLaMA-3-8B-Instruut
20.6
1.2.0,
585
GPT-3.5-Turb--1106
18.9
(-1.8,
285
17.8
(-1.2, 2.2)
322
Command-R
17.0
(-1.7,
432
Phi-3-Mini-1288-innste
15.4
(.1.4,
609
Tulu-2-DPO-70B
15.0
(-1.6,
550
Starling- Li-thnn-LLM7BB-lphhah
12.8
(-1.6,
483
Gemma- 1.1-7B-IT
12.1
(-1.3,
341
LLaMAA---7BB-Chat--
11.6
(-1,5,
595
Vicuna-33B-
8.6
(-1.1,
451
SAAMA--27B-Cha--RE
4.6
(-0.8,0.8)
561
Gemma- 1.1-2B-IT
3.4
(-0.6, 0.8)
316
Table 13, Arena- Arena-Hard- Haro-Auto Leaderboard (baseline: GPT-4-0314) with some additional models (Frick et al., 2024; DeepSeek Al al., 2024; GLM et al., 2024; Yang et al., 2024; al., 2024; Abdin al., 2024; Team et al., 2024).
18

---

Title Suppressed Due to Excessive Size
Python Game Development
PyTorch Autoencoder Implementation
Prime Numbers and Proofs
Computability and Automata Theory
Python Data Structures
Linked List Operations
Chemical Equilibria and Reactions
Golang HTTP Handlers Errors
Solving Algebraic Equations
Calculus Essentials & Applications
Entropy in Various Contexts
Advanced Random Number Techniques
Swift Retry Management
Web Development Essentials
Advanced Mathematical Concepts
Vehicle Damage Inspection
LaTeX Figure and Tabular Formatting
Diagnostic Test Accuracy Metrics
Atomic and Electronic Structure
LLM Prompt Engineering
Color Strategy and Selection
Random Number Generation
Biblical Studies Interpretations
cluster
Finance and Banking Operations
Effective Weight Loss Strategies
Investor Subscription Agreements
Email Funding Update Requests
Scheduling Availability Coordination
American English Vocabulary
Carpet & Cleaning Services
Christmas Humor and Jokes
Dinosaur Discovery and Extinction
Video Games Related Films
Spelling Variations of SEQUENCE
Inventive Brand Naming Strategies
Aquatic Life and Cartoons
Relationship Challenges Advice
Diverse Extracurricular Engagement
Emoji Usage and Interpretation
Baywatch Athleticism & Strength
Prohibited Erotic Fiction
Diverse Gift-Giving Ideas
Professional Email Communication
Design Styles & Influences
Flirty Texting Strategies
Welfare Encounters
Weekend Family Fun Ideas
Global Restaurant Recommendations
2
5
Mean Score
Figure A more complete selection of mean scores of various topic clusters in descending order.
19

---

Title Suppressed Due to Excessive Size
F. Examples
Cluster I: Greetings and Well-Being Inquiry (Mean Score: 2.7)
Yo, what up my brother (Qualities) None)
Cluster 2: US Presidents Query (Mean Score: 3.2)
Who the president of the US in 1975 (Qualities: Specificity, Domain-KnowLedge, Technical Accuracy, Real-Worid)
Cluster 3: Physies Problem Solving (Mean Score: 5.0)
50, 000 kg aizplane initially flying at a speed of 60.0 m/s accelerates at 5.0 m/s2 for 600 meters. What is its velocity after this accelcration? what is the net force that caused this acceleration? (Qualities: Specificity, Domain-Knowledge, Complexity, Problem-Solviine Technica cal Accuracy, Real-world)
Cluster OpenCV Image Processing Technique (Mean Score: 5.5)
you are given a task to detect number of faces in each frame of any video using pytorch and display ay the number in the final edited video. (Qualities: A11)
20

---

and Title Suppressed Due to Excessive Size
G. Prompts and Examples
Prompt Quality Systems Instruction:
Your task evaluate how well the following imput prompts the capabilities of advanced Al assistants. For the imput prompt, please analyze it based on the following 7 criteria, For each criteria, make sure to explain before determine whether the imput satisly it.
Specificity: Does the prompt ask for specific, well-drfined output without leaving any ambiguity? This allows the Al to demoustrate its ability to follow instructions and generate precise, targeted response.
2. Douain Kuowledge: Does the prompt test the Al's knowledge and understandiug in specific domain set of domains? The prompt must demand the to have a strong prior knowledge or mastery of speciff-speiffi concepts, theories, or principles. Complexity: Does the prompt have multiple componenta, variables, levels of depth and nuance? This assesses the Al's capability handle complex, multi-faceted problems beyond simple queries.
Problem-Solvingg Does the prompt require active problem-solving analyzing and clearly defining the problem and systematically devising and implementing solution? Note active problem-solving is not simply reciting facts or following fixed set of instructions. 5. Creativity: Does the prompt require creative approach or solution? This tests the Al's ability to generate novel ideas tailored to the specific needs of the request or problem at land.
Technical Accurscy: Does the prompt require an answer with a high degree technical accuracy, correctness and preciston? This the reliability and truthfulness the Al's outputs.
Real-World Application: Does the prompt relate yeal-world applications? This the Al's ability provide practical and actionable information that could be implemented in real-life scenarios.
After analyzing imput prompt based these criteria, you must criteria mumbers that the prompt satisfies format of Python array. For example, 'Criteria Satisfied: [1, 4, 6,
LLM-Judge System Instruction
Please act an impartial judge and evaluate the quality of the responses provided by two AI assistants the user prompt displayed below. You will be given assistant A's answer and assistant B's answer. Your job is to evaluate which assistant's 's answer is better.
Begin your evaluation generating your
the prompt. You provide your
before Judging any answers.
When evaluating the assistants' answers, compare both assistants' answers with your answer. You must identify correct any mistakes or insccurate information.
Then consider if the assistant's is answers are helpful, relevant, and concise. Helpful means the answer correctly responds to the prompt or follows the instructions. Note when user prompt has any ambiguity or more than one interpretation, it is more helpful and appropriate to ask for clarifications or more information from the user than providing an answer based on assumptions. Relevant appropriate what being asked. Concise means the response
means all parts of respouse closely verbose excessive.
Then consider the creativity and novelty of the assistant's 's answers when needed. Fimally, idcutify any missing important information answers would beneficial include when responding to the user prompt.
After providing your explanation, you must output only one of the following choices as your final verdict with a label:
Assistant A is siguificantly better: [[>>>B]
2. Assistant A is alightly better: [[A>B]]
Tie, relatively same: [[==B]]
4. Assistant B is slightly better: [[B>A]]
Assistant B significantly better: [B
Example output: 'My final verdict tie: [A=B]]'
21

---

Title Suppressed Due to Excessive Size
Arena-Hard-Auto (No Modifications)
Model
Score
Token #
Header (%)
Bold (%)
List (%)
80.0
1035
0.010
1.503
1.288
gemini-1..5flaat
78.6
729
0.020
1.353
1.122
Semini-1.5-ffash-22mm.
74.5
793
0.088
1.548
1.271
gemini--.5-fash-2-cchtty
68.2
808
0.005
1.236
0.986
gemini-I..-ffush-2nnoo..
61.7
574
0.003
0.924
0.979
1lama-3.1-70b-dttal
53.5
834
0.025
0.961
1,470
llama-3.1-70b-md
44.9
601
0.257
1.776
1.695
11ama
44.5
606
0,084
0.728
1.380
1laba-3.-770-chhttt
44.3
623
0.011
0.679
1.173
llama-3.1-70b-no-md
37.5
522
0.010
0.123
0.986
app-3..-turro-0022-eeaall
25.6
416
0.008
0.447
1.540
gpt-3.5-trrh-00125
23.1
323
0.012
0.284
1.272
gpt-3.5-turbo-0125-ne
22.0
328
0.372
0,877
1.601
gp--3.5-urro-0012--no-ddd
18.0
269
0.012
0.182
1.149
spe-3.5-uurbo-00125ccaayy
17.1
286
0.006
0.296
1.012
Arena-Hard-Auto (Style Control)
Model
Score
Token #
Header (%)
Bold (%)
List (%)
gemini-1.5-flashe
75.5
729
0,020
1.353
1.122
gemini-1.5-flas--2ddee..
71.2
1035
0.010
1.503
1.288
gemini-l.--flash-2-mdd
69.3
793
0.088
1.548
1.271
gemini-1.5-fashh-2nnomm-
62.5
574
0.003
0.924
0.979
gemini-l.5-ffahh-2chhh.
61.5
808
0.005
1.236
0.986
1lama-3.1-70b
41.7
606
0.084
0.728
1.380
llama-3.1-70b-no-e
39.9
522
0.010
0.123
0.986
llama-3.1-70b-des.
39.8
834
0,025
0.961
1.470
11aba-3.-
39.5
623
0.011
0.679
1.173
llama-3.1-70bmmd
34,9
601
0.257
1.776
1.695
gpt-3.5-turbo-0125
33.2
323
0.012
0.284
1.272
gpo-3.5-uurbo-01225nn--dd
30.4
269
0.012
0.182
1.149
gpt-3.5-urroo-0125-deaaii
28.9
416
0.008
0.447
1.540
gpt-3.5-trbo-0115-mmd
27.9
328
0.372
0.877
1.601
gpt-3.5-urbo-015--chattg
27.3
286
0.006
0.296
1.012
Table 6. Comparison Between Arema-Hand-Aurt with no modification versus applying style control, Prompt for detailed:" are helpful assistant who thoroughly explains things with much detail possible.", prompt for chatty: "You helpful assistant who chatty.", prompt for md: "You are a helpful assistant who uses as much markdown possible.", and prompt for no-md: "You are a helpful assistant who never uses markdown." Token represents average number of tokens, header is average markdown header density per token in percentage, bold is average bold markdown element density per token in percentage, and list is average list markdown element per token percentage.
22

---

Title Suppressed Due to Excessive Size
User:
Use ABC notation to write a melody in the style of a folk tune.
GPT-4:
X:1 T:Untitled Folk Tune M:4/4 L:1/8
::G|:G2A2 B2A2 G2E2 ID4| |E2F2 G2F2 E2C2 IB, |A2B2 c2B2 /22F E4 D222 |F2E2
|D2B, 2C4:
GPT-3.5-Turbo: Sure! Here's simple melody in the style of a folk tune written in ABC notation:
X:1 T:Folk Tune 4:4/4 1:1/4 K:C.
CDR JGA |BAG PRD] CDR| PGA BAG c2 CDA/ |BAG [FRD] CDD, FGA BAGI c22
This melody is in the key of C major and consists of four measures. Feel free to modify it or add more sections to create your own folk tune!
Judgement: My
the prompt would be:
X:1 T:My Folk Melody M: L:1/8
K:D A2 2d2f2a2/b2a2E22 e2d282 A4F21 G2A2B2 A2G2F2 E2D2E 1D6: 12D4A2 d:d222a2 b2a2f2 e2c222 B4A2 G2A2B2 A2G2F2 E2D2F2 ID6: 2D4
This melody is in the key of D major, has 3/4 time signature, and structured in typical AAB form common in folk music. range of octave, which is typical for folk tunes that need to easily singable.
Now, let's evaluate the responses from Assistant A and Assistant B.
Assistant A's Answer: Assistant provides a melody ABC notation that coherent and follows the structure of folk tune. The melody is in 4/4 time, which is common for folk music, and is in the key of G major. The melody has two-part form (indicated by the repeat signs and :)), which is typical in folk music. The range of the melody is appropriate for a folk tune, and the rhythm is simple and repetitive, which is characteristic of the genre.
Assistant B's Answer: Assistant also provides a melody ABC notation
coherent could be considered the style of folk tune. The melody is in 4/4 time and the key of C major. The melody consists of four measures repeated twice, which is a simple structure that could be found in folk music. However, Assistant B's melody is less traditional in its form compared to Assistant A's, as does not have distinct A and B section, which more common in folk tunes. Additionally, the of "z" for half-measur rest is less typical folk tunes, which usually have
continuous melodic line.
Both assistants provided melodies that could he considered folk tunes. Assistant A's melody has a more traditional structure with clear A and B section, while Assistant B's melody is simpler and does not follow the traditional AAB or AB form. Assistant A's melody is also more detailed terms of the ABC notation, providing full eight geghtmm section that repeated, which gives a more complete sense
Comparison:
tune. Assistant B's ss melody correct in notation is more basic and less representative of traditional folk tune structure.
In terms of helpfulness, relevance, and conciseness, both assistants provided relevant and concise answers. However, Assistant A's answer slightly more helpful due the more traditional structure of the provided melody.
Creativity and novelty are not particularly applicable this context, as the task is to create melody in traditional style rather than to innovate. Both assistants showed an understanding of the folk tune style, but Assistant A's answer aligns more closely with traditional folk structures.
There are no significant mistakes or inaccurate information in either assistant's answers. However, Assistant B's use the rest "z" in folk tune is less conventional.
My final verdict is: Assistant A is slightly better: [[A>B]].
23


Reviews:
[{"review_id": "LTuTySWgns", "reviewer": "Reviewer_65no", "summary": "This paper introduces Bench-O-Matic, an automated pipeline for curating high-quality benchmarks from large-scale crowdsourced datasets, and Eval-O-Matic, a benchmark dataset generated using this pipeline. The motivation is that existing benchmarks are either static (leading to saturation and test-set leakage) or require expensive human curation. Bench-O-Matic extracts prompts from datasets like Chatbot Arena and WildChat-1M, applying seven quality criteria (e.g., specificity, domain knowledge, problem-solving complexity) to filter high-quality prompts. The resulting benchmark, Eval-O-Matic, achieves 3× better model separation than MT-Bench and 98.6% correlation with human preference rankings at a fraction of the cost ($20 per evaluation). The paper also introduces new benchmark evaluation metrics—Separability with Confidence, Agreement with Confidence, and Pair Rank Brier Score—to assess benchmark effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. We address your concerns and propose corresponding revisions:\n\n**W1**: We do not treat Chatbot Arena’s user-vote-based rankings as an unquestionable “gold standard.” Instead, we and other works (e.g., Dubois et al., Lin et al.) view it as a practical \"silver standard\" worth approximating. Although Chatbot Arena likely contains biases and noise, it currently represents the largest publicly available dataset of human evaluations, with over two million interactions. Hence, benchmarks demonstrating high agreement and correlation with Chatbot Arena rankings provide valuable guidance to model developers. We will include this in our revisions.\n\n**W2**: Our critique of Spearman correlation does not depend on assuming unbiased LLM judges. Rather, confidence-based metrics provide clarity on two critical questions:\n\n1. When benchmarks predict model performance, do these predictions align with actual user preferences?\n2. Can benchmarks effectively distinguish between similar model checkpoints?\n\nIf LLM judges exhibit systematic biases, this would naturally result in lower confidence, agreement, and correlation metrics. Conversely, if our evaluation demonstrates high confidence, agreement, correlation, and separability, it affirms the benchmark's usefulness.\n\nWe acknowledge potential biases inherent to LLM-based evaluations and address them through multiple strategies:\n\n- **Style Control**: As detailed in Section 6.5, our methods significantly reduce biases toward particular styles or longer responses, improving benchmark alignment with human preference rankings.\n- **Bias Diagnostics**: We systematically compare judge rankings with human leaderboards across multiple scenarios, using metrics like confidence agreement to quantify biases and misalignments.\n\nWe agree that detailed exploration of systematic biases is important and plan to expand our analysis.\n\n**W3**: Our primary focus was comparing Eval-O-Matic with benchmarks like MT-Bench and AlpacaEval, which similarly utilize LLM-based judges for open-ended tasks without ground-truth references (instead of LiveBench for example). While recognizing that LLM-as-a-Judge inherently introduces challenges, this method remains the most scalable and affordable approach to model developers for open-ended prompts.\n\n**W4**: We agree with the reviewer regarding the presence of low-quality and unsafe prompts within crowdsourced datasets. In Section 4.1 (line 196), we explicitly describe our process for filtering prompts and retaining only high-quality clusters based on average quality scores. Figures 4 and 6 (Section 4.3) illustrate how clusters are differentiated qualitatively, with higher scores correlating with complexity and relevance, while lower scores correspond to trivial or ambiguous prompts. Further details and examples are provided in Appendix C.\n\nMoreover, our ablation study (Section 4.3, Figure 3) demonstrates how qualitative criteria effectively differentiate strong and weak models, explaining Bench-O-Matic's produced benchmarks' enhanced separability compared to other benchmarks. We will clarify these points further in the revision.\n\n**W5**: We agree with the importance of evaluating benchmarks in specialized domains (scientific, legal, programming). However, our core contribution specifically targets diverse, real-world, open-ended interactions. Benchmarks such as Chatbot Arena, MT-Bench, and AlpacaEval are widely used precisely because they reflect authentic user interactions. Hence, Eval-O-Matic remains valuable for assessing practical, downstream performance. Nevertheless, we recognize this limitation and will clearly outline it in our manuscript.\n\n**W6**: We have validated Bench-O-Matic’s performance beyond Chatbot Arena by also evaluating on WildChat-1M. Ensured no Eval-O-Matic prompts overlap with publicly released Chatbot Arena data. If a model trained on similar queries generally improves at addressing real-world interactions, we anticipate performance gains across both Eval-O-Matic and actual deployment settings.\n\n**W7**: While our current framework focuses exclusively on text-based evaluations, it could be adapted to multimodal benchmarks by substituting LLM annotators and evaluators with VLMs. Indeed, Chou et al. have successfully applied a similar approach to the Vision Arena dataset, demonstrating the feasibility of extending Bench-O-Matic’s principles to image-based tasks.\n\nChou et al. \"VisionArena: 230K Real World User-VLM Conversations with Preference Labels.\"\n\nLin et al. \"Wildbench: Benchmarking llms with challenging tasks from real users in the wild.\"\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "The claim that Bench-O-Matic extracts high-quality prompts aligned with human preferences is supported by Eval-O-Matic’s rankings showing 98.6% correlation with Chatbot Arena, which reflects real user interactions. But this assumes Chatbot Arena rankings are a gold standard, but prior work (Carlini et al., 2021) suggests human preference data may contain inconsistencies and biases. The authors should evaluate how much these factors impact benchmark quality.\n\nThe authors argue that existing metrics (e.g., Spearman correlation) fail to measure model separation and propose confidence-based alternatives. This assumes that LLM judges are unbiased. While the paper introduces ensemble-based methods to mitigate bias, it does not analyze failure cases where LLM judges systematically misrank models.", "methods_and_evaluation_criteria": "Benchmark Design: The hierarchical clustering method (BERTopic, UMAP, HDBSCAN) for grouping prompts is reasonable, but the paper lacks qualitative validation of cluster quality.\n\nComparison with Prior Work: The evaluation compares Eval-O-Matic with MT-Bench and AlpacaEval, but does not include LiveBench or R2E, which also focus on dynamic benchmarking.", "theoretical_claims": "No formal proofs, but the paper’s proposed metrics (Separability with Confidence, etc.) are well-motivated. One potentially issue is with the bootstrapping methods used for confidence estimation, since the statistical robustness of these methods in this setting is not fully analyzed.", "experimental_designs_or_analyses": "- The clustering-based filtering approach ensures high-quality prompts, but the paper lacks qualitative analysis of outliers and failure cases.\n- Several ablations are performed, like different LLM judges, controlling for stylistic biases, and testing on alternative datasets.\n- But the paper seems to be missing some failure analysis, like discussion of when and why low-quality prompts are selected.\n- Some minor assumption about LLM costs remaining stable, when API pricing may change.", "relation_to_broader_scientific_literature": "The paper builds on MT-Bench, AlpacaEval, and Chatbot Arena but improves by introducing an automated pipeline. This work is similar to LiveBench and R2E, though Bench-O-Matic focuses on prompt curation rather than live model evaluation.\n\nPrior work has shown LLM-based evaluation correlates with human judgments, but the failure cases of LLM judges (e.g., hallucinations, self-reinforcement biases) are underexplored.", "essential_references_not_discussed": "None that I'm aware of.", "other_strengths_and_weaknesses": "Other weaknesses:\n- The pipeline is tested only on Chatbot Arena and WildChat-1M, with no evaluation on scientific, legal, or programming benchmarks.\n- The authors use ensemble LLM judges but do not analyze cases where LLMs systematically misrank models.\n- The paper does not discuss when Bench-O-Matic selects poor-quality prompts or how often prompt selection fails.", "other_comments_or_suggestions": "Including more qualitative examples of prompts selected by Bench-O-Matic would be helpful.", "questions_for_authors": "1. How does Bench-O-Matic handle adversarial noise? Crowdsourced datasets may contain low-quality or adversarial prompts—how does your system filter these?\n2. What is the impact of training data contamination? If LLMs are trained on Chatbot Arena-style queries, could this inflate correlation scores?\n3. Why not compare against LiveBench or R2E? These benchmarks also focus on dynamic evaluation—how does Bench-O-Matic differ?\n4. How would the method adapt to multimodal benchmarks? Can Bench-O-Matic curate image- or video-based prompts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tgJCeVvZVv", "reviewer": "Reviewer_tKxZ", "summary": "This paper introduces Bench-O-Matic, a pipeline that automatically constructs high-quality, large scale benchmarks to evaluate LLMs from crowdsource datasets such as Chatbot Arena. To measure the quality of this benchmark, the authors proposed new metrics to measure properties that are important in when curating the data. The authors also present Eval-O-Matic (and Wild-O-Matic), which was curated using this pipeline, and demonstrated higher model separation of model performance compared to existing benchmarks, and also at a low cost.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive feedback. Below, we address each concern raised and propose revisions:\n\n> \"However, one large concern is the fact that the experiments rely on the LLM-as-a-Judge evaluation framework, which, as noted in the paper, is known to exhibit certain biases. While the authors have attempted to mitigate some issues, it is still an inherent limitation which could influence the robustness of the results.\"\n\nWe agree that biases inherent to the LLM-as-a-Judge evaluation framework pose challenges. Nevertheless, it remains the most practical and cost-effective method to evaluate LLMs on diverse, open-ended real-user queries. Besides implementing multiple bias-mitigation strategies described in the paper, we highlight that our benchmark demonstrates greater robustness compared to similar LLM-as-a-Judge evaluations, as evidenced by the significantly higher agreement to human preference and stronger separability.\n\n> \"Bench-O-Matic is currently rather limited to single-turn, English interactions. The authors acknowledge this limitation; however, it does raise questions about the generalization of this pipeline for various real-world applications.\"\n\nWe acknowledge that our current method is indeed limited to single-turn, English evaluations. Multi-turn benchmarks are inherently challenging to automatically curate from crowdsourced datasets because responses beyond the initial turn can depend significantly on prior model outputs. The authors are also primarily proficient in only English. However, extending Bench-O-Matic to robustly support multi-turn and multilingual evaluations represents a valuable future direction we intend to pursue.\n\n> \"The reliance of LLMs as various parts of the pipeline may introduce bias which I believe has to be further investigated and regulated\"\n\nWe recognize the reviewer’s concern regarding potential biases introduced by using LLMs throughout the evaluation pipeline. To address this, we validated the quality and reliability of our annotations using majority voting among several state-of-the-art models (GPT-4o, Claude-3-Opus, Gemini-1.5-Pro), achieving an 85.6% agreement with GPT-4-Turbo on 200 prompts in the validation set (as detailed in Section 4.2). \n\nAfter reading your review, we additionally conducted human validations by manually annotating 50 prompts from our validation set. These showed an 84.3% agreement between GPT-4-Turbo and human annotations, and notably, a 94.9% agreement between human annotations and majority LLM votes. Due to OpenReview's policy restricting updates during the discussion phase, this important validation results will be fully detailed in the final manuscript revision to enhance Bench-O-Matic's reproducibility and credibility (will be detailed in Section 4.2). \n\n> \"Prompts are filtered based on 'quality scores' produced from an LLM. How was this threshold determined, and do you know how adjusting this could potentially affect the quality?\"\n\nTo clarify, we did not rely on LLM-generated numeric \"quality scores.\" Instead, the LLM was instructed to provide binary judgments indicating whether each prompt met specific qualitative criteria (e.g., “Problem-Solving,” “Domain Knowledge”). The prompts and clusters with most of the criteria satisfied are selected for benchmark curation. We will clarify this distinction in Section 4.1 of our revision. Furthermore, our ablation analysis in Section 4.3 (Figure 3) demonstrates that including more qualitative criteria enhances the differentiation between stronger and weaker models (e.g., GPT-4 vs. Llama-2-70B, Claude-Opus vs. Claude-Sonnet), hence explains the enhanced separability of our final benchmarks (e.g. Eval-o-Matic, Wild-o-Matic) from other popular benchmarks. We will make sure to clarify this connection in our revisions.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which improves the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "The authors claim that this pipeline produces benchmarks that are high quality, which is supported by details empirical results that show a higher model separation and correlation with human preferences compared to other well known benchmark datasets used to evaluate LLMs. The analysis of the cost was also provided and supported the claim that the proposed method is cost effective.\n\nHowever one large concern is the fact that the experiments rely on the LLM-as-a-Judge evaluation framework which as noted in the paper is known to exhibit certain biases. While the authors have attempted to mitigate some issues, it is still an inherent limitation which could influence the robustness of the results.", "methods_and_evaluation_criteria": "Yes, the methods are suitable for the problem at hand. The novel metrics proposed provide a more nuance way of evaluation LLMs beyond the traditional statistical measures such as the Spearman or Pearson correlations, which I agree does allow for a more robust evaluation of LLM performance, and also its agreement with human preferences. Re benchmark datasets: the experiments compare Eval-O-Matic with popular benchmarks such as MT Bench and Chatbot Arena, which are suitable for the task.", "theoretical_claims": "There are no formal proofs of any of the claims in the paper. The authors provide descriptive explanations of the metrics proposed and those are based on existing statistical foundations and are largely sound.", "experimental_designs_or_analyses": "The experiments compare Eval-O-Metric to several well-known benchmark datasets, and also tested it using the top 20 LLMs from Chatbot Arena, comparing them using existing statistical metrics and also the newly proposed ones. These are thorough designs and does provide compelling empirical results. The measures to prevent any LLM induced bias were appreciated, however as mentioned above, there are still concerns regarding the limitations of LLM-as-a-Judge evaluations and how robust they are.", "relation_to_broader_scientific_literature": "The paper contributes to the domain of LLM evaluation and benchmark curation. It aims to extend benchmarking beyond the tradition static benchmarks such as MMLU with ground truth based evaluation, or even live benchmarks such as Chatbot Arena, by introducing an automated curation method that allows for evaluation on open-ended tasks. By automating this process, Bench-O-Matic also aims to address the existing issue of test set leakage by being able to frequently update benchmarks.\n\nThe introduction of new evaluation metric which supplement traditional statistical metrics could prove to be important metrics that help to measure properties that existing metrics could not, contributing to the ability to better measure the performance of LLMs in future work.\n\nThe authors also uses methods such as style control, and introduces Emsemble-as-Judges as an aim to improve the LLM-as-a-Judge framework which could help to improve the reliability of that framework.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- Bench-O-Matic offers a scalable, cost-effective, and automated approach to curate benchmarks, making it extremely impactful in practice\n- The new evaluation metrics introduced provide better evaluation and separability of LLMs, which would be crucial in helping advance future work in LLM development and benchmarking\n\nWeaknesses:\n- Bench-O-Matic is currently quite rather limited to single-turn, English interactions, the authors acknowledge this limitation, however it does raise questions about the generalization of this pipeline for various real world applications\n- The reliance of LLMs as various parts of the pipeline may introduce bias which I believe has to be further investigated and regulated", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Prompts are filtered based on “quality scores” produced from an LLM, how was this threshold determined and do you know how adjusting this could potential affect the quality?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PML60GH4Zy", "reviewer": "Reviewer_v7Qm", "summary": "The paper introduces a new method for automatic generation of robust, high‐quality benchmarks for evaluating LLMs. The approach is designed with key features: it controls for the style and length of generated content, shows strong alignment with human preference, and can be done in a cost- and time-efficient manner. Extensive experiments show that the generated benchmarks have significant advantages in terms of confidence agreement and separability compared to existing evaluation systems such as AlpacaEval, MTBench, and Chatbot Arena. However, the authors focus their method on open-ended and single turn tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable and insightful feedback. We address your concerns as follows:\n\n> \"The choice of chatbot arena as a reference for computing confidence agreement and separability is somewhat limiting. Does this method extend to more challenging tasks, such as Olympiad-level mathematics or SWEbench, where state-of-the-art models generally perform poorly?\"\n\nWe would like to clarify that while confidence agreement relies on human preference rankings as a reference, computing separability does not require human preferences or chatbot arena as a reference. Separability effectively captures a benchmark's ability to distinguish among model performances, a critical aspect as state-of-the-art models increasingly demonstrate similar levels of performance. This metric provides meaningful differentiation crucial for model developers.\n\nWe believe our method indeed extends to more challenging tasks. Our approach successfully extracts difficult tasks from vast unfiltered datasets, such as \"PyTorch Autoencoder Implementation\" and \"Computability and Automata Theory\" (see page 18, Figure 6). Here, we show topic clusters specifically ranked for their complexity and desirable quality and the highest scored topic clusters are selected for benchmark curation, suggesting our method aligns well with the current demands for more challenging tasks. We will make sure to clarify this connection in our revision.\n\n> \"Currently, this setup is limited to single-turn evaluation, which is restrictive, as many contemporary LLM applications involve agentic or multi-turn interactions.\"\n\nWe acknowledge that our current method is indeed limited to single-turn evaluations. Multi-turn benchmarks are inherently challenging to automatically curate from crowdsourced datasets because responses beyond the initial turn can depend significantly on prior model outputs. Extending Bench-O-Matic to robustly support multi-turn evaluations represents a valuable future direction we intend to pursue.\n\n> \"When using the same model family for both judge and annotator, the judge model often exhibits bias towards generations from the same family. Have you investigated this issue, and could style control potentially mitigate this bias?\"\n\nThe reviewer raises an important point. We have studied and quantified biases arising from using judge and annotator models from the same family (see Section 6.6: Mitigating Self-Biases in LLM-Based Evaluation). While we have not yet specifically explored whether controlling for style mitigates this bias, we definitely plan to investigate this.\n\n> \"Given the growing emphasis on evaluating technical and challenging tasks (such as reasoning and coding), how well do you believe your method extends to these benchmarks?\"\n\nOur analyses indicate that the clusters selected by our approach indeed prioritize more technically demanding tasks, such as reasoning and coding, over trivial ones (as demonstrated in Figure 6 on page 18). For example, clusters involving sophisticated topics score higher, increasing their likelihood of inclusion in benchmarks compared to simpler ones like \"Flirty Texting Strategies.\" Thus, we believe our method aligns strongly with the trend towards evaluating models on more technical and challenging tasks.\n\n> \"Have you studied whether the increased separability results from filtering for difficult questions within benchmarks?\"\n\nWe directly investigated the impact of filtering for difficult questions on separability in our experiments (see Section 4.3, Figure 3). The results confirm that selecting tasks with higher difficulty indeed enhances the benchmark’s ability to differentiate between stronger and weaker LLMs, reinforcing the benefit of our filtering strategy.\n\n---\n\nWe deeply appreciate the reviewer’s feedback and hope our responses fully address your concerns.", "claims_and_evidence": "Authors claim that their method automatically generates robust and high quality benchmarks which: \n1. Control for style and length of generation\n2. Show strong agreement with human preference\n3. Can be created with reduced cost and time\n\n\nTheir extensive examinations show strong advantages in confidence agreement and separability (compared to AlpacaEval, MTBench, Chatbot Arena). Their results show that controlling for style and length does help reduce the bias in evaluation. \n\nHowever, the method and experimental results are limited to open-ended tasks where the reference is human preference, which is a limitation since many competitive LLM benchmarks (e.g., olympiad math, legal or medical exams) rely on ground truth.", "methods_and_evaluation_criteria": "As mentioned before, their evaluation criteria makes sense. Yet correlation with human preference is a limiting eval criteria.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Yes, i checked for validity of their experimental analysis. The experiments and ablations are well-designed and results are carefully reported. However, results are limited to a small set of benchmark datasets. Expanding the experiments to include more difficult datasets would strengthen the paper.", "relation_to_broader_scientific_literature": "Focusing on evaluations, especially creating an automated evaluation framework which:\n1. controls for potential biases in evaluations\n2. provide a dynamically generated evaluation dataset which can mitigate test set contaminations\n3. evaluates models and a diverse set of samples (expanding test set distribution)", "essential_references_not_discussed": "I am not aware of such references.", "other_strengths_and_weaknesses": "1. The choice of chat bot area as a reference for computing confidence agreement and separability is a bit limiting. Does this method extend to more difficult tasks, such as olympiad level math or SWEbench, where SoTA models in general do very poorly on?\n\n2. Currently this setup is limited to single turn evaluation. This is a bit limitation cause many LLMs today are used for agentic or multi turn tasks.", "other_comments_or_suggestions": "When using the same family of models as the judge and annotator, the judge model usually shows a bias towards generations of the same family of models. Have you studied this? Would controlling for style reduce this bias?", "questions_for_authors": "1. Given the increasing focus of evaluations on more technical and difficult tasks (like reasoning and coding), how do you think this method extends to such benchmarks?\n\n2. Have you studied if the increased separability is the result of filtering difficult questions to include in the benchmark?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uf3B2YRObA", "reviewer": "Reviewer_qaBA", "summary": "This paper proposes Bench-O-Matic, a pipeline for curating high-quality benchmarks (Eval-O-Matic) from large volumes of crowdsourced queries. The pipeline combines hierarchical clustering with a set of LLM-based filters keyed to “prompt quality” dimensions (e.g., complexity, specificity, domain knowledge). The resulting prompt set is evaluated with a novel suite of metrics—like confidence agreement—that emphasize how effectively a benchmark can distinguish models and align with human preferences. Experimental comparisons show Eval-O-Matic outperforms or rivals popular benchmarks such as MT-Bench and AlpacaEval on both model separability and alignment with human preference rankings, all at lower cost.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedbacks. Below, we address your concerns and propose our revisions.\n\n**W1**: We acknowledge that our pipeline heavily utilizes LLMs for prompt quality scoring and response evaluation. To address this concern and enhance credibility, we validated annotation quality through majority voting among state-of-the-art models, achieving an 85.6% agreement rate with GPT-4-Turbo on 200 prompts in the validation set.\n\nAfter reading your review, we integrated human judgments by manually labeling a subset (50 out of 200) of the validation set, yielding an 84.3% agreement between GPT-4-Turbo and human labels, and a 94.9% agreement between majority votes from LLMs and human labels. While OpenReview rules currently prevent us from updating during the discussion period, we assure you this important validation step will be included in the final manuscript to enhance reproducibility and credibility (in Section 4.2).\n\n**W2**: We agree that demonstrating broader generalization is important. Although our primary experiments focused on Chatbot Arena, we also evaluated the pipeline on the WildChat dataset (Wild-O-Matic), obtaining comparable improvements in benchmark quality. Both datasets are crowdsourced from real-world users, aligning with our goal of creating benchmarks reflective of genuine user interactions. By validating our method on two distinct real-world datasets, we demonstrate sufficient generalization of our approach.\n\nWe further highlight that benchmark validation is resource-intensive, as each evaluation of 20 models incurs significant costs (~$600 per validation cycle). Hence, we strategically focused our ablations on different LLM judges, quality annotation models, and datasets (Section 4).\n\n**W3**: \n- We used a fixed random seed (42) across all experiments, including bootstrapping, and will explicitly document this detail in the revised manuscript (Section 6.1).\n- Confidence intervals were calculated using 100 bootstrap iterations and are 95% unless specified otherwise; this clarification will also appear in Section 6.1.\n- We will document the minimum cluster size (8) and note that the number of resulting clusters varies based on datasets and min cluster size (Section 4.1).\n\nRegarding ablations on dropping qualities, we clarify that our experiments (Section 4.3, Figure 3) effectively cover this aspect by demonstrating improved separation between strong and weak LLMs as the number of included qualities increases. As we drop qualities, we see less distinction between strong and weak LLMs (e.g. GPT-4 vs Llama-2-70B and Claude-Opus vs Claude-Sonnet). We will further emphasize this connection in the revision.\n\n**W4**: Thank you for the suggestions. In our revised paper:\n- Figure 1 will be referenced in related works.\n- Elaboration on the selection of the quality criteria, and their specific contributions, will be detailed in Appendix C (p. 15).\n\n**W5**: Spearman correlation only tells you whether two rankings share a similar order: if Model A is above Model B under one benchmark, does that same ordering hold under human judgments? However, it glosses over two crucial aspects:\n- Confidence or “Separation” Among Models: Even if two benchmarks induce the same ordering of models, some may do so with very different levels of certainty. In practice, it matters whether “Model A beats Model B” is a large-margin result the benchmark can replicate consistently, or whether it’s a fragile result based on noise. Our “Separability” metric explicitly checks how often two models have non-overlapping confidence intervals in their benchmark performance. If a benchmark repeatedly yields overlapping intervals, you cannot reliably conclude that the top model is truly better.\n- Magnitude of Differences (Beyond Rank): Our “Brier Score,” for example, rewards a benchmark not just for correctly ranking pairs of models, but also for assigning appropriate probabilities to those rankings. A pairwise victory by 90–10 is stronger evidence of a real performance gap than a 55–45, yet Spearman treats them identically so long as the order is preserved.\n\nEven with 20 models, fine distinctions can be critical. Modern LLMs often cluster close to one another in quality, so you need to know if a newly finetuned is truly outperforming a similar rival or if the difference is effectively within the margin of error. If your benchmark and evaluation metrics don't capture this level of granularity and confidence, you may end up training or deploying a model you think is better but is actually equal (or even worse) when considering statistical noise.\n\nWe will expand on this distinction clearly in the revised manuscript.\n\n---\n\nWe greatly appreciate the reviewer’s constructive feedback, which significantly enhances the quality and clarity of our work. We respectfully ask the reviewer to reconsider their rating, given our revisions, clarifications, and the substantial potential impact of our contributions to the community.", "claims_and_evidence": "yes. the experiment is extensive.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "NA.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The solution propose a pipeline to construct benchmark from the crowdsource data selected by LLM with customized prompt, which is of practical value in general.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "1. Clear, Straightforward Method: Despite involving multiple steps (clustering, filtering, LLM-based annotation, and evaluation), the overall pipeline is easy to follow.\n2. Novelty of Automated Curation: Reliance on LLMs for prompt selection is an interesting approach that addresses a real challenge—continual benchmark development without human-in-the-loop curation.\n3. Empirical Gains & Metrics: The benchmark produced (Eval-O-Matic) yields high model separability and near-human ranking alignment. The authors propose new measures (e.g., confidence agreement) to quantify a benchmark’s ability to differentiate and rank models reliably.\n4. Potential Impact: By open-sourcing the pipeline, others can frequently generate fresh benchmarks, mitigating the common pitfalls of data leakage and benchmark saturation.", "other_comments_or_suggestions": "1. Reliance on LLMs in Multiple Stages: The pipeline depends heavily on LLMs both for scoring prompt quality and for evaluating final responses. While the authors attempt validation, additional depth—e.g., human judgments or expanded evidence for each “key quality”—could improve trust.\n2. Limited Generalization Evidence: Demonstration of generalizability is mainly restricted to one additional dataset (WildChat) with a relatively simple baseline for comparison.\n3. Missing Ablations and details: The paper does not fully specify the mechanics of how many random seeds are applied, how precisely bootstrapping is conducted, or how the confidence intervals are chosen. This omission leaves readers uncertain about the exact procedure for computing separability and the sensitivity of that metric. Key design choices (like cluster count, weighting of quality dimensions, or dropping certain qualities) are not rigorously tested, which might leave readers unsure about the pipeline’s sensitivity to different configurations.\n4. Minor Presentation Issues:\nFigure 1 and the key qualities are unreferenced, which may impede clarity.\nMore elaboration on the seven quality criteria and how they were selected would be helpful.", "questions_for_authors": "Could you intuitively show why the proposed by metrics can outperform spearman correlation? Does it really matter, given your experiment on 20 models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces BenchO-Matic, an automated pipeline that uses large language models (LLMs) to generate high-quality, open-ended prompts from large, crowd-sourced datasets, enabling continuous benchmark updates without human involvement. The idea is timely and relevant, especially given the growing need for scalable benchmarking in the LLM era.

While the reviewers appreciated the motivation and potential of the work, several shared concerns were raised. These include:
1. The inherent biases and limitations of using LLMs as evaluators ("LLM judges"),
2. The lack of ablation studies to isolate the contributions of different pipeline components,
3. The limited set of evaluation datasets, which makes it difficult to assess the generalizability of the approach.

The authors provided a thoughtful rebuttal and addressed many of these concerns with promising insights and proposed experiments. Reviewers remained positive to the rebuttal. AC recommends acceptance and encourages the authors to incorporate the additional experiments and clarifications discussed in the rebuttal into a revised version of the manuscript.
Decision: Accept (poster)